Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Raoofian, Erfan
Master of Science - MSc
University of British Columbia
Open Source Software (OSS) ecosystems function as the digital infrastructure upon which numerous applications and services are built. The role of human factors like diversity, inclusion, and teamwork is well-known to be essential for the growth and longevity of OSS communities in Software Engineering (SE). Central to the sustainability and growth of OSS ecosystems is the effective integration and retention of new contributors. This thesis examines the role-specific dynamics of mentors and mentees in the Google Summer of Code (GSoC) program through the analysis of the OpenMent dataset. OpenMent is a comprehensive compilation of over 500 thousand issue and pull request comments and commit messages from GitHub, that is curated as part of this thesis. By applying sentiment and emotion analysis, topic modeling, and temporal analysis of communication data between mentors and mentees, this study examines how these interactions evolve over the program's duration and influence mentees' contributions to OSS. Significant findings from this research highlight the key role of early engagement in technical aspects and how mentor-mentee communications impact mentee contributions on GitHub projects. Differences in the communication patterns between mentors and mentees across various stages of GSoC indicates the importance of targeted support to mentors and mentees. By providing insights into the dynamics of mentorship in OSS, this thesis contributes valuable recommendations for improving mentorship frameworks, with the aim of stimulating a more robust and inclusive OSS community.
The Alchemy of OSS: TransformingChats into Insights Using Sentiment,Emotion, and Topic Analysis ofMentor-mentee Communications inGoogle Summer of CodebyErfan RaoofianB.Sc., Isfahan University of Technology, 2021A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe College of Graduate Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)September 2024© Erfan Raoofian 2024The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:The Alchemy of OSS: Transforming Chats into Insights Using Sentiment,Emotion, and Topic Analysis of Mentor-mentee Communications in GoogleSummer of Codesubmitted by Erfan Raoofian in partial fulfillment of the requirements forthe degree of Master of Sciencein Computer ScienceExamining Committee:Dr. Fatemeh Hendijani Fard, Irving K. Barber Faculty of ScienceSupervisorGema Rodr´ıguez-Pe´rez, Irving K. Barber Faculty of ScienceCo-supervisorIfeoma Adaji, Irving K. Barber Faculty of ScienceSupervisory Committee MemberYves Lucet, Irving K. Barber Faculty of ScienceSupervisory Committee MemberAmir Ardestani-Jaafari, Faculty of ManagementUniversity ExamineriiAbstractOpen Source Software (OSS) ecosystems function as the digital infrastruc-ture upon which numerous applications and services are built. The role ofhuman factors like diversity, inclusion, and teamwork is well-known to beessential for the growth and longevity of OSS communities in Software En-gineering (SE). Central to the sustainability and growth of OSS ecosystemsis the effective integration and retention of new contributors. This thesisexamines the role-specific dynamics of mentors and mentees in the GoogleSummer of Code (GSoC) program through the analysis of the OpenMentdataset. OpenMent is a comprehensive compilation of over 500 thousandissue and pull request comments and commit messages from GitHub, that iscurated as part of this thesis. By applying sentiment and emotion analysis,topic modeling, and temporal analysis of communication data between men-tors and mentees, this study examines how these interactions evolve over theprogram’s duration and influence mentees’ contributions to OSS. Signifi-cant findings from this research highlight the key role of early engagement intechnical aspects and how mentor-mentee communications impact menteecontributions on GitHub projects. Differences in the communication pat-terns between mentors and mentees across various stages of GSoC indicatesthe importance of targeted support to mentors and mentees. By provid-ing insights into the dynamics of mentorship in OSS, this thesis contributesvaluable recommendations for improving mentorship frameworks, with theaim of stimulating a more robust and inclusive OSS community.iiiLay SummaryIn the world of computer software development, open source projects-wheresoftware is openly developed and shared-are important and influential. Theseprojects rely on bringing in and training new developers. This thesis uses ar-tificial intelligence to identify and analyze the emotions and topics discussedin mentor-mentee communications in a mentorship program in open sourcedomain. We explore how these factors evolve throughout the program andrelate to contributions of new developers to open source projects. Findingsreveal that the nature of discussions and emotions expressed by mentorsand mentees are different and they are related to the newcomers’ contribu-tions to open source projects after the mentorship program ends. Successfulprojects tend to focus on core project tasks over preliminary setups earlyin the mentorship program. We suggest tailored support for mentors andmentees adjusted to different stages of the project, to improve the mentor-ship experience and enhance the longevity of Open Source Software (OSS)development.ivPrefaceThis thesis represents an original, unpublished, and independent work bythe author, Erfan Raoofian. Under the guidance and direction of my su-pervisor, Dr. Fatemeh Hendijani Fard, and my co-supervisor, Dr. GemaRodr´ıguez-Pe´rez, I was primarily responsible for all major areas of studydesign, data collection, analysis, and the majority of the manuscript writ-ing. No generative artificial intelligence (AI) tools were used in research,drafting, or preparation of this thesis.The journey of this thesis has been intellectually enriching and challeng-ing, pushing the boundaries of my understanding and skills. The findingsof this research are intended to contribute significantly to the field, and assuch, we are planning to submit this work for publication at a relevant aca-demic conference or in a scholarly journal, aiming to share our insights andcontributions with the scientific community.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xviDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Research Questions . . . . . . . . . . . . . . . . . . . . . . . 21.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 31.5 Structure of the Thesis . . . . . . . . . . . . . . . . . . . . . 42 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1 Mentorship in OSS Projects . . . . . . . . . . . . . . . . . . 62.1.1 Why OSS Needs Mentoring . . . . . . . . . . . . . . . 72.1.2 Types of Mentorship in OSS . . . . . . . . . . . . . . 82.1.3 Challenges in Mentorship . . . . . . . . . . . . . . . . 82.1.4 Proposed Solutions in Literature . . . . . . . . . . . . 92.2 Google Summer of Code (GSoC) . . . . . . . . . . . . . . . . 122.3 Sentiment and Emotion Analysis in the Existence of LargeLanguage Models . . . . . . . . . . . . . . . . . . . . . . . . 13viTable of Contents2.3.1 Sentiment and Emotion Analysis in Software Engi-neering . . . . . . . . . . . . . . . . . . . . . . . . . . 142.4 Topic Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 152.5 Identification of the Literature Gap . . . . . . . . . . . . . . 173 OpenMent: A Dataset of Developers Interactions in theGoogle Summer of Code . . . . . . . . . . . . . . . . . . . . . 193.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2 The OpenMent Dataset . . . . . . . . . . . . . . . . . . . . . 213.2.1 Scale and Statistics . . . . . . . . . . . . . . . . . . . 213.2.2 How to Read the OpenMent Dataset . . . . . . . . . 243.3 Related Datasets . . . . . . . . . . . . . . . . . . . . . . . . . 263.3.1 OpenMent Differentiation . . . . . . . . . . . . . . . . 273.4 Construction of OpenMent . . . . . . . . . . . . . . . . . . . 283.4.1 Collecting Projects’ Data From GSoC . . . . . . . . . 293.4.2 Repository URL Verification and Extraction . . . . . 303.4.3 Participant Identification and Verification . . . . . . . 303.4.4 Collecting GitHub Data . . . . . . . . . . . . . . . . . 313.4.5 Normalization of GSoC Program Milestones . . . . . 323.4.6 Gender Identification . . . . . . . . . . . . . . . . . . 333.4.7 Anonymizing Data . . . . . . . . . . . . . . . . . . . 333.4.8 Analysis Potential . . . . . . . . . . . . . . . . . . . . 344 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.2 RQ1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 394.2.1 Sentiment Extraction . . . . . . . . . . . . . . . . . . 394.2.2 Emotion Extraction . . . . . . . . . . . . . . . . . . . 414.2.3 Analysis of Sentiments and Emotions Across Roles . 424.2.4 Analysis of Sentiments and Emotions Over Time . . . 434.2.5 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 444.3 RQ2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 464.3.1 Topic Modeling . . . . . . . . . . . . . . . . . . . . . 474.3.2 Topic Analysis . . . . . . . . . . . . . . . . . . . . . . 484.3.3 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 494.4 RQ3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . 504.4.1 Analysis Procedure . . . . . . . . . . . . . . . . . . . 514.4.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 514.5 Statistical Methods . . . . . . . . . . . . . . . . . . . . . . . 55viiTable of Contents5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615.1 Sentiment Analysis (RQ1 Results) . . . . . . . . . . . . . . . 615.1.1 Distribution of Sentiments . . . . . . . . . . . . . . . 625.1.2 Statistical Analysis of Sentiments . . . . . . . . . . . 635.1.3 Weekly Analysis of Sentiments . . . . . . . . . . . . . 655.1.4 Milestone-based Analysis of Sentiment . . . . . . . . 69Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.2 Emotion Analysis (RQ1 Results) . . . . . . . . . . . . . . . . 715.2.1 Distribution of Emotions . . . . . . . . . . . . . . . . 715.2.2 Statistical Analysis of Emotions . . . . . . . . . . . . 725.2.3 Weekly Analysis of Emotion . . . . . . . . . . . . . . 735.2.4 Milestone-Based Analysis of Emotions . . . . . . . . . 80Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825.3 Sentiment and Emotion Analysis of Transitioning Roles (Menteeto Mentor) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825.3.1 Sentiment Analysis Before and After Transition . . . 825.3.2 Emotion Analysis Before and After Transition . . . . 835.3.3 Statistical Analysis . . . . . . . . . . . . . . . . . . . 84Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 855.4 Topic Modeling (RQ2 Results) . . . . . . . . . . . . . . . . . 855.4.1 Main Themes and Topics in Mentor-Mentee Commu-nications . . . . . . . . . . . . . . . . . . . . . . . . . 865.4.2 Evolution of Topics Throughout the GSoC Program . 895.4.3 Comparison of Roles . . . . . . . . . . . . . . . . . . 92Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.5 Effectiveness of GSoC (RQ3 Results) . . . . . . . . . . . . . 945.5.1 Analysis of Sentiment, Comments, and Commits . . . 955.5.2 Analysis of Emotion and Themes . . . . . . . . . . . 100Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1066 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1076.1 What Do Improved Projects Do Differently? . . . . . . . . . 1076.2 Role Differences and Milestone Reactions . . . . . . . . . . . 1086.3 Week 13 as a Potential Critical Point . . . . . . . . . . . . . 1086.4 Recommendations for Enhancing Mentorship in GSoC . . . . 1096.4.1 Program Structure and Phasing. . . . . . . . . . . . . 1096.4.2 Mentee Engagement Strategies. . . . . . . . . . . . . 1096.4.3 Mentor Engagement Strategies. . . . . . . . . . . . . 1106.5 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . 110viiiTable of Contents7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114AppendicesA First Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . 131B Second Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . 136ixList of Tables2.1 Categories of studies related to OSS mentoring. . . . . . . . . 72.2 Examples of challenges and proposed solutions in OSS men-torship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.1 Gender distribution within the GSoC program duration. . . . 225.1 Overview of the data used for analysis of RQ1. . . . . . . . . 625.2 CD({role}, {sentiment}): Distribution of sentiments amongdifferent roles. . . . . . . . . . . . . . . . . . . . . . . . . . . . 625.3 Mean, standard deviation, and variance of sentiment scoresin mentees’ and mentors’ comments. . . . . . . . . . . . . . . 635.4 T-test results comparing sentiment scores between mentorsand mentees. . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.5 ANOVA results for weekly sentiment profiles. . . . . . . . . . 675.6 Pairwise t-test results for consecutive weeks. Significance lev-els: * p < 0.05, ** p < 0.01, *** p < 0.001. . . . . . . . . . . 685.7 ANOVA results for sentiment profiles across GSoC’s majorproject periods. . . . . . . . . . . . . . . . . . . . . . . . . . . 705.8 Sentiment profiles across GSoC’s major periods. . . . . . . . 705.9 Counts of comments with different emotion and role variablesand CD({role}, {emotion}) in parentheses. . . . . . . . . . . 725.10 Chi-square test results checking the relation between role andemotion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 735.11 Chi-square test results for weekly emotion trends. . . . . . . . 775.12 Pairwise chi-square test results for consecutive weeks. Signif-icance levels: * p <0.05, ** p <0.01, *** p <0.001. . . . . . . 785.13 Emotion distributions across major project periods. . . . . . 815.14 Chi-square test results for emotions across major project mile-stones. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 815.15 Proportions of different emotions expressed before and aftertransitioning from mentee to mentor. . . . . . . . . . . . . . . 84xList of Tables5.16 T-test results comparing average sentiment scores before andafter transitioning. . . . . . . . . . . . . . . . . . . . . . . . . 845.17 Chi-square test results comparing emotion proportions beforeand after the role transition. . . . . . . . . . . . . . . . . . . . 855.18 Overview of the themes and their associated topics identifiedin the project communications. . . . . . . . . . . . . . . . . . 875.19 The main themes along with their sharemetric (Equation 4.4),the percentage of comments written by each role, and thesparkline of the impact metric (Equation 4.5) over the 20weeks of the GSoC. . . . . . . . . . . . . . . . . . . . . . . . . 905.20 Basic statistics for improved and unimproved groups. . . . . . 955.21 Comparing improved and unimproved projects using senti-ment scores (Defined in Section 4.2.1), number of commentsper project (PPCcomment, see Equation 4.9), and number ofcommits per project (PPCcommit, see Equation 4.10). Themeans of PPC metrics are defined in Equations 4.11, and 4.12. 965.22 T-test results, improved vs. unimproved. . . . . . . . . . . . . 975.23 Weekly mean number of comments and commits per projectcomparing improved and unimproved projects over time inGSoC. MentorMPPCcomment is not ploted as the differenceswere not statistically significant between improved and unim-proved. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005.24 Counts of comments having different emotion label and projectstatus (CD({role, status}, {emotion}) in parentheses). . . . . 1015.25 Chi2 Tests of Independence between improved and unim-proved groups. . . . . . . . . . . . . . . . . . . . . . . . . . . 1015.26 Improved projects vs unimproved projects - theme difference. 1025.27 Weekly impact sparklines of top six discussed themes’ com-paring improved and unimproved projects over time. Inter-pret with care: 1. The Y-axis range is different from one plot toanother. 2. The Y-axis does not include the full range of possiblevalues, which may visually exaggerate small differences. . . . . . . 105A.1 users: This table lists all participants with anonymized iden-tifiers and their corresponding gender. . . . . . . . . . . . . . 131A.2 This table contains key details of each project, including itsunique identifier, year, and mentor detection status. . . . . . 131A.3 Outlines the roles of users within projects, specifying whethereach user serves as a mentee or mentor. . . . . . . . . . . . . 132xiList of TablesA.4 Records details of issues and pull requests, including type,status, and dates relative to program start. . . . . . . . . . . 133A.5 This table maps issues and pull requests to the projects theyare associated with. . . . . . . . . . . . . . . . . . . . . . . . 134A.6 Contains all comments made on issues and pull requests,linked to specific GSoC projects. . . . . . . . . . . . . . . . . 134A.7 Documents all commits made by GSoC users on the GSoCprojects’ repo. . . . . . . . . . . . . . . . . . . . . . . . . . . . 135B.1 Discovered topics and their top words. . . . . . . . . . . . . . 136B.2 The discovered topics, their share metric (Equation 4.4), roledistribution, and impact (Equation 4.5) trend. . . . . . . . . 139xiiList of Figures3.1 Distribution of commit messages and issue/PR commentswithin the GSoC program duration, showcasing the involve-ment levels of mentors and mentees. . . . . . . . . . . . . . . 223.2 Yearly distribution of communications in the GSoC program,indicating the total number of commit messages and issue/PRcomments from 2016 to 2023. . . . . . . . . . . . . . . . . . . 243.3 GSoC website showing the list of projects. Elements of inter-est for our dataset are: Contributor (mentee), Mentor, the“View Code”, and “View project details” buttons. Note:This is an image of a reconstruction of the GSoC website,not the original. . . . . . . . . . . . . . . . . . . . . . . . . . . 294.1 Overview of our methodology. . . . . . . . . . . . . . . . . . . 374.2 Overview of sentiment, emotion, and topic assignment to tecomments of our dataset. The task-specific instruction is con-catenated with the GitHub comment text and fed into theE5instruct model. This process generates embeddings that en-capsulate the semantic meaning of the comments, which arethen used for downstream classification (sentiment and emo-tion) and clustering (topic modeling) tasks, ensuring high per-formance and accurate representation of the data. . . . . . . 415.1 Distribution of differences in sentiment means from 10,000resamples of the pooled sentiment scores. . . . . . . . . . . . 645.2 Distribution of sentiment means (mentors vs. mentees). . . . 655.3 Sentiment profile (weekly mean sentiment score) of mentorsand mentees over weeks of the GSoC. Interpret with care: TheY-axis does not include the full range of possible values, which mayvisually exaggerate small differences. . . . . . . . . . . . . . . . . 66xiiiList of Figures5.4 CD({emotion}, {role}): Distribution of roles in each emo-tion category. Each bar represents an emotion label, and hastwo sections distinguishing mentors and mentees portion indifferent colors. . . . . . . . . . . . . . . . . . . . . . . . . . . 725.5 Impact of sadness throughout the program in mentors andmentees. Interpret with care: The Y-axis does not include thefull range of possible values, which may visually exaggerate smalldifferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745.6 Impact of anger throughout the program in mentors and mentees.Interpret with care: The Y-axis does not include the full range ofpossible values, which may visually exaggerate small differences. . . 755.7 Impact of fear throughout the program in mentors and mentees.Interpret with care: The Y-axis does not include the full range ofpossible values, which may visually exaggerate small differences. . . 765.8 Impact of surprise throughout the program in mentors andmentees. Interpret with care: The Y-axis does not include thefull range of possible values, which may visually exaggerate smalldifferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 765.9 Impact of joy throughout the program in mentors and mentees.Interpret with care: The Y-axis does not include the full range ofpossible values, which may visually exaggerate small differences. . . 775.10 Analysis of sentiments and emotions before and after tran-sitioning from mentee to mentor. The figure is divided intothree parts: Left: Percentage of positive and negative sen-timents expressed before and after transitioning, showing adecrease in positive sentiments and a slight increase in nega-tive sentiments post-transition. Middle: Average sentimentscores before and after the transition, illustrating a declinein overall sentiment positivity when users become mentors.Right: Proportions of different emotions expressed beforeand after transitioning, highlighting noticeable increases inanger and fear, and a decrease in sadness after becoming men-tors. Interpret with care: The Y-axis does not include the fullrange of possible values, which may visually exaggerate small dif-ferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835.11 Heat map showing the prominence of the identified themesover time. Each cell represents the NI metric (Equation 4.6)of one theme in a specific week of the program. . . . . . . . . 91xivList of Figures5.12 Weekly Normalized Impact(NI, Equation 4.6) of the 6 mostdiscussed themes in discussions. Each plot has two trendlines, one calculated on the mentor comments, and the othercalculated on the mentee comments. The normalized natureof NI keeps the comparison focused on the rate of changesopposed to only focusing on the number of comments createdby each role in that theme category. . . . . . . . . . . . . . . 925.13 Histogram with KDE plots of bootstrap means for senti-ment scores comments per project, and commits per project.Greater overlap of the distributions indicates a higher p-value,suggesting less significant differences between improved andunimproved projects. A larger distance between the centersof distributions on the other hand, highlights a greater magni-tude of difference, visually representing both the significanceand magnitude of difference for each metric. . . . . . . . . . . 98xvAcknowledgementsI am grateful to a number of individuals whose guidance and support havebeen essential throughout the course of my Master of Science program.First, I would like to express my heartfelt appreciation to my supervi-sor, Fatemeh Hendijani Fard. Her mentorship covered a broad spectrumincluding research, technical expertise, and invaluable life advice during myacademic journey. Without doubt, her support has been fundamental to myacademic and personal growth.I also wish to extend my thanks to my co-supervisor, Gema Rodr´ıguez-Pe´rez, who guided and supported me through out my M.Sc. program, es-pecially during the first year. Her guidance helped me overcome the initialchallenges of graduate studies.I appreciate my supervisory committee members, Ifeoma Adaji and YvesLucet for reviewing my work. Ifeoma has been a great source of feedback. Iam also grateful to Yves, whose knowledge, patience, and modest demeanorare qualities I greatly admire and look up to.I have been part of an amazing research group. I thank my lab mates,Amir Reza, Jie, Jumar, Meghdad, Mohammad, Nikta, Ramin, and Shawn,for making my study period enjoyable and enriching.A special and profound thanks to my brother, Ali, for being a constantsource of support and inspiration throughout my Master’s studies and inlife. His encouragement has been indispensable to me.Each of these individuals has made a significant impact on my journey,and I am thankful for their contributions to my life.xviTo my parents, who taught me that greatness is rooted in love.Although the science within these pages may not directly speak of love’s language,trust me, every word and every graph pulses with the love you’ve embedded within me.xviiChapter 1Introduction1.1 ContextCurrently, Open Source Software (OSS) is one of the most important parts ofglobal software development [35]. The vitality of OSS ecosystems is deeplyintertwined with their ability to continuously evolve and adapt through con-tributions from global community [78]. This depends not merely on theinnovations but crucially on the integration of new developers into ongoingprojects [113].The path for newcomers into OSS projects is filled with challenges,from navigating complex project architectures [150] to integrating into well-established community cultures [92]. In this setting, mentorship is rec-ognized as a critical infrastructure that ensures the sustained health andgrowth of these ecosystems [114]. Effective mentorship bridges the formidablegap between novice enthusiasm and productive, long-term contributions,thereby fortifying the foundational structures of open source projects [36].Previous studies highlight how structured programs like GSoC1 andspontaneous, community-driven mentorship efforts significantly enhance new-comer retention, and skill acquisition. Yet, despite its recognized benefits,mentorship in OSS faces numerous challenges ranging from social and com-munication barriers [5, 18, 109] to technical complexities and recognitiondeficits [67, 77, 96, 150].GSoC represents a significant initiative in OSS domain, providing a plat-form where student developers collaborate with established OSS projectsunder the guidance of experienced mentors. Since its inception, GSoC hasfacilitated the engagement of thousands of participants from diverse back-grounds in meaningful software development, contributing significantly tothe OSS ecosystem [103, 124]. This program enhances the educational ex-periences of student participants by immersing them in real-world softwaredevelopment scenarios [38, 102].Central to this thesis is the creation and analysis of the OpenMent1https://developers.google.com/open-source/gsoc/11.2. Goaldataset, a thoroughly compiled repository of over 400 thousand issue andpull request comments and commits from GitHub, capturing the interactionsbetween mentors and mentees during GSoC from 2016 to 2023. This datasethelps in understanding the mentor-mentee relationships and opens a newway to assess the efficacy of mentorship practices within OSS projects.1.2 GoalBy conducting a nuanced analysis of mentor-mentee interactions withinGSoC, this thesis aims to identify the differences in different roles, andidentify factors that contribute to successful mentorship in OSS projects.The OpenMent dataset, which provides a unique window into the mentor-ing relationships, is central to achieving this goal.Despite the recognized importance of mentorship in OSS, several areascrucial for understanding and improving these relationships remain under-explored. Current literature lacks an analysis of how different mentoringapproaches is related to the outcomes of OSS projects, and improvementsin mentees. There is a need to identify mentorship dynamics and analyzeits relation to project success metrics to understand what works and whatdoes not. Additionally, while some understanding of the general benefits ofmentorship exists, there is insufficient detailed analysis of the interactionsbetween mentors and mentees. Furthermore, to the best of our knowledge,the sequence in which knowledge and skills are imparted in OSS mentorshiphas not been studied, leaving it unclear what is the best order for mentoringtopics that could potentially lead to more efficient learning and integration ofnewcomers. A focused analysis on the topics of these interactions and howthey evolve over the mentorship period is missing. Understanding thesedynamics can lead to more tailored and timely interventions that supportmentees’ needs. Additionally, much of the existing research relies on self-reported data from questionnaires, which can be biased or incomplete [128].OpenMent enhances survey-based mentorship studies by providing mentor-mentee interaction data. This offers an unfiltered view of interactions duringmentorship relationships.1.3 Research QuestionsThe research questions formulated for this thesis are designed to delve intothe mentor-mentee communications within the context of OSS, particularlythrough the lens of the GSoC program.21.4. ContributionsRQ1: Does the sentiment and emotion expressed in commu-nications differ significantly between mentors and mentees, andacross different times in the GSoC program?This question seeks to explore the emotions expressed in the mentorshipexperience. By analyzing the sentiment and emotional tone of communi-cations, this research aims to identify patterns in the mentoring process.The findings could inform about the communication tones of both roles inmentor-mentee relationships, and how they evolve over time and in responseto main milestones in the program.RQ2: How do the topics in mentor-mentee conversations evolvethroughout the GSoC program?The evolution of discussion topics can shed light on the progression oflearning and integration of mentees into the OSS community. This ques-tion investigates whether there are distinct phases in the mentoring process,characterized by shifts in topic focus. Understanding these shifts could leadto more structured and phase-appropriate mentorship strategies that caterto the needs of mentors and newcomers at different stages of the mentorship.RQ3: How does participation in the GSoC program affect thelevel of mentees’ contributions to OSS on GitHub, and are theredifferences in the sentiments, emotions, or topics discussed bythose with increased OSS contribution versus those who do not?This question explores the direct outcomes of mentorship on the produc-tive engagement of mentees in OSS projects. By comparing the contribu-tions of mentees who show marked improvement after the GSoC program tothose who do not, this research aims to relate successful mentorship prac-tices with specific communication topics, and emotion expressions. Thiscould potentially highlight effective mentoring techniques for better integra-tion of mentees into the OSS community.By examining the emotions and topics of mentor-mentee interactions,we aim to contribute actionable insights that could enhance the mentorshipmodels within the OSS community.1.4 ContributionsWe introduce different contributions to the field of software engineering,specifically focusing on the mentorship within GSoC through the creationand analysis of the “OpenMent” dataset. We outline the contributions de-rived from our study below.• The creation of the OpenMent dataset is one of the primary contribu-31.5. Structure of the Thesistions of this thesis. It includes GitHub communications such as issues,pull requests, and commits of GitHub repositories of GSoC programs.There is not a straight forward way to identify the GSoC reposito-ries in GitHub and identify the roles in each repository which makesit nonviable to use existing GitHub Datasets for this kind of analy-sis. We address this gap in existing datasets by concentrating on thementor-mentee relationship, providing a basis for a detailed analysisof mentorship effectiveness and outcomes. The dataset allows for anuanced analysis of communication styles, mentorship patterns, anddevelopment practices within open source projects, thereby facilitatingfuture research in mentorship and collaboration.• We employed temporal and topic modeling analyses to understand howcommunication and topics evolve throughout the mentorship program.This methodology help in providing insights into the impact of differ-ent factors on project success and mentee development.• The analysis of sentiments and emotions shows the differences in howmentors and mentees experience the GSoC program. We recommendstrategies to enhance the effectiveness of mentorship programs likeGSoC. These include focusing on core programming skills early, im-plementing a structured mentorship approach, and enhancing menteeautonomy.• Other than the recommended strategies, we provide a wide range ofplots, and analysis data that can be used to understand the emotions,sentiments, and topics of the communications in GSoC in each week ofthe program, grouped by different roles, timeframes, and the successof the projects.1.5 Structure of the ThesisThis thesis is structured into seven chapters. After Chapter 1, we delveinto the previous research relevant to the study in Chapter 2 reviewingexisting findings from prior studies that have influenced the current researchapproach. In Chapter 3, we present the primary empirical tool used inthis research; this chapter describes the development and structure of theOpenMent dataset establishing the basis for the subsequent analysis. InChapter 4, we outline the analytical methods employed to examine andanalyze the OpenMent dataset. This includes the statistical techniques,41.5. Structure of the Thesissentiment and emotion analysis model, and topic modeling approach used toanalyze the communications between mentors and mentees. In Chapter 5,we present the findings of the research by addressing each research questionposed in the introduction. Drawing from the results, Chapter 6 interpretsthe findings within the broader context of OSS and mentorship research. Itcompares these results with existing literature, discusses the theoretical andpractical implications, and identifies the limitations of the study. Finally, inChapter 7, we synthesize the entire study, summarizing the key findingsand their implications for improving mentorship in OSS.5Chapter 2Related Works2.1 Mentorship in OSS ProjectsOSS has become a crucial component of modern software development.Analogous to the essential “roads” and “bridges” in our physical world, OSSconstitutes the foundational infrastructure of our digital landscape [35]. Thisinfrastructure supports a wide range of applications across various domains,mirroring the indispensable role of physical infrastructure. Consequently,maintaining and sustaining this digital infrastructure is of paramount soci-etal importance.One of the challenges in OSS sustainability is attracting and retain-ing new developers [111]. Fagerholm et al. [36] elaborates on how mentor-ing maintains a contributor’s motivation and enhances their learning curve,which is essential for sustained contribution.Mentorship plays an important role in the sustainability and growth ofOSS projects [36]. Feng et al. [40] shows that nearly 90% of contributors inthe Apache Project underscore the essential nature of efficient mentorshipfor managing significant contributions. Similarly, Silva et al. [104] highlightshow mentees participating in GSoC benefit immensely from mentorship, byfacilitating their contributions, networking with experienced developers, andsignificantly enhancing their skills.In the following, we provide an overview of mentoring practices withinOSS as portrayed in the literature. We delve into the need for mentoring inOSS, identify the typical mentors and mentees, define mentoring activities,and examine the platforms where mentoring occurs. The papers can bebroadly categorized into one or several of the topics: “analysis of mentoringin OSS”, “collaboration dynamics in OSS”, or “barriers of newcomers inOSS”, as provided in Table 2.1.62.1. Mentorship in OSS ProjectsCategory CitationAnalysis of mentoring in OSS. [12, 17, 18, 36, 54, 67, 96, 102, 106, 110,112, 114, 117, 118, 125, 150]Collaboration dynamics in OSS. [12, 17, 18, 36, 40, 52, 67, 89, 114, 125,150]Barriers of newcomers in OSS. [5, 6, 16, 37, 54, 77, 92, 96, 102, 106,108, 109, 111–113, 117, 118]Table 2.1: Categories of studies related to OSS mentoring.Six studies exclusively used contribution data from issue tracking soft-ware such as GitHub [36, 37, 40, 108, 117, 150]. Meanwhile, 13 studiesfocused solely on interviews, surveys, or questionnaires [5, 6, 52, 54, 67, 77,89, 102, 104, 109, 111, 114, 125]. Two studies relied on mailing lists, [16, 106],and one study used existing literature as its sole source [110]. Seven studiescombined multiple sources [12, 17, 92, 96, 112, 113, 118]. Additionally, onestudy included a review of existing literature along with personal experiencesand student interactions [18].2.1.1 Why OSS Needs MentoringMentorship within OSS projects plays multiple crucial roles, including theseamless integration of newcomers and the enhancement of their coding skillsand project involvement [104, 108]. Stanik et al. [108] found that newcom-ers often select complex initial tasks from the issue tracker system withoutknowing the level of complexity of the chosen task. Furthermore, many OSSprojects leave newcomers to navigate these tasks on their own [37]. Thesestudies highlight a lack of support for newcomers, which can significantlydiminish their motivation within the OSS community.According to Balali et al. [5] and Steinmacher et al. [111] mentoringhelps newcomers join OSS projects by addressing the various challengesand significant barriers they face and facilitating their integration into thecommunity. Prana et al. [92] address the inclusion issues and the signifi-cant role mentorship plays in making OSS communities more accessible tonewcomers.The benefits of mentorship, particularly in initiatives like GSoC, are sig-nificant. Silva et al. [104] note that mentorship enhances mentee skills andimproves project outcomes, showcasing the tangible advantages of structured72.1. Mentorship in OSS Projectsguidance. Empirical data provided by Rehman et al. [96] shows that mentor-ship significantly increases newcomer retention and satisfaction, underliningits importance in the OSS ecosystem.2.1.2 Types of Mentorship in OSSFeng et al. [40] categorize mentorship in OSS into “formal”, and “informal”.They define the informal mentoring as when a developer reaches out toothers to seek/give guidance to solve a problem. Formal mentoring on theother hand, involves structured programs that provide specific guidelines andmechanisms to ensure effective interactions between mentors and mentees.Carillo et al. [17] highlight GSoC as a prime example of a formal mentorshipprogram. In GSoC, experienced OSS developers are paired with newcomers.This pairing provides one-on-one guidance and project-specific training.2.1.3 Challenges in MentorshipMentorship in OSS projects faces a range of challenges that can hinder theeffectiveness of these programs. Addressing these challenges is necessary forthe improvement and success of mentorship initiatives and the integrationof newcomers into the community. We categorize different challenges foundand pointed out by the studies into five groups as described below.Expectation Management and Engagement. Balali et al. [5] dis-cuss the difficulties that arise when mentees do not meet the expectationsset by mentors, emphasizing the critical need for thorough mentor trainingand careful mentee selection processes. Similarly, Steinmacher et al. [114]detail the challenges faced by mentors in large-scale projects such as thosewithin the Apache Software Foundation, where managing expectations andcommitments can be particularly taxing. These challenges can diminishmentees’ confidence and cause anxiety [111, 112]. If mentors are unable toalign with the expectations and thought processes of mentees, the menteesmay begin to believe that their abilities and skills are fixed, feeling that theycannot learn and improve [18]. A significant factor that can lead menteesto abandon their efforts is receiving unfriendly feedback [5]. As stated bySingh [106], such negative experiences can decrease the mentees’ desire tolearn, leading them to prefer having others resolve the challenges they face.Social and Communication Barriers. Steinmacher et al. [111] pointout social barriers such as lack of feedback and adequate social interaction,which are significant challenges for newcomers. Tan et al. [118] address thenecessity of ensuring that mentees are actively engaged and have access to82.1. Mentorship in OSS Projectsthe resources they need to succeed in their projects. An unsupportive projectenvironment, and limited availability of mentors are some of the examplesof this kind of barriers [5, 109].Complexity and Technical Difficulties. Cereceda and Quinn [18]point out that mentees can feel overwhelmed by the complexity of theirprojects, which means mentors need to be supportive and patient. Thedifference in work experience and project seniority between mentors andmentees can create big gaps in practical knowledge and technical skills [5].Mentees often find it hard to understand advanced concepts when giventasks beyond their abilities [114]. Without good guidance, mentees maystruggle to find the right tasks to work on and often give up because theycannot find tasks they can handle [112].Recognition and Motivation. Trinkenreich et al. [125] highlight thatmentoring, often being a non-code contribution, receives less recognition,potentially leading to mentor demotivation. This is specially more often seenin informal mentoring, where mentors efforts may not be recognized [40].Gender-specific Barriers. Gender-specific issues are also prevalent, asnoted by [77], who found that women in OSS projects face multiple barriersthat require targeted support through mentorship. The complexity of thedevelopment environment setup is another challenge, discussed by Guizaniet al. [52], which can be daunting for newcomers without proper guidance.These challenges underscore the need for thoughtful, inclusive, and well-structured mentorship programs that can adapt to the diverse needs of OSScommunities and their contributors.2.1.4 Proposed Solutions in LiteratureThese solutions focus on enhancing mentee engagement, improving mentorsatisfaction, and ensuring that contributions are acknowledged and rewardedappropriately.Balali et al. [5] highlight the effectiveness of using a “newcomer tag” inprojects, which signals mentors to be more patient and provide the necessarysupport. This approach helps in recognizing the efforts of new contributorsand ensures they receive the guidance they need.Prana et al. [92] discuss the benefits of using varied communication toolsin OSS projects. These tools facilitate better interaction between mentorsand mentees and are essential for acknowledging mentees’ contributions ef-fectively. To motivate mentors and ensure their efforts are recognized, Fenget al. [40] propose using a simple tagging system with “@mentor” that al-lows contributors to publicly acknowledge mentoring efforts. They suggest92.1. Mentorship in OSS Projectsthat OSS projects should acknowledge implicit mentoring to sustain andencourage mentoring activities within the community. Furthermore, Fager-holm et al. [36] and Trinkenreich et al. [125] also emphasize the importanceof recognizing and rewarding mentors. Providing mentors with recognitionor rewards can sustain their motivation and enhance their commitment tothe mentoring process.Silva et al. [104] suggest incorporating elements such as achievementsand leaderboards in OSS projects. These features can increase engagementand provide a gamified aspect to the contribution process, making it moreappealing to newcomers.Publicly displaying mentees’ names on project web sites as contribu-tors is another motivational strategy discussed by Rehman et al. [96]. Thisrecognition enhances mentee satisfaction and promotes a sense of belongingwithin the community.102.1. Mentorship in OSS ProjectsChallenge SolutionExpectation Management and EngagementMentees not meeting expectations [5]. Use of a “newcomer tag” [5].Managing expectations in largeprojects [114].Public recognition systems like “@men-tion” [40].Social and Communication BarriersLack of feedback and adequate social in-teraction [111].Enhancing interaction through variedcommunication tools [92].Communication and language chal-lenges.Providing communication guidelines formentees.Complexity and Technical DifficultiesOverwhelmed by project complex-ity [18].Supportive mentoring approaches [18].Development environment setup com-plexity [52].Simplification and guidance in technicalsetup [52].Recognition and MotivationMentor does not receive recogni-tion [125].Recognizing and rewarding the effortthat mentors provide to help mentees inthe projects [36, 125].Gender-specific BarriersBiased treatment from others [12]. Targeted support and creating a moreinclusive and diverse community [77,106].Table 2.2: Examples of challenges and proposed solutions in OSS mentor-ship.The previous literature have explored the challenges of mentorship inOSS, but a detailed analyses of the direct and temporal interactions be-tween mentors and mentees ramains insufficiently addressed. Most existingresearch relies on self-reported data, missing the details of evolution of dis-cussions and emotional shifts during mentorship. Also, the role of emotionsand sentiments in OSS mentorship is underexplored. In this thesis, we ad-dress these gaps by analyzing communications, emotions, and sentimentsover time, and their relation with post-GSoC engagement improvements.One of our primary goals is to provide new insights into how mentorshipcould potentially be optimized in OSS environments.112.2. Google Summer of Code (GSoC)2.2 Google Summer of Code (GSoC)Google Summer of Code (GSoC)2 is a global initiative led by Google, de-signed to engage students in OSS development during the summer months.Launched in 2005, the program encourages student participation in OSSprojects by offering stipends, thereby fostering a deeper understanding andcommitment to OSS communities [104]. According to Trainer et al. [124]and Silva et al. [103], GSoC stands out as one of the most well-known in-ternships globally due to its long-standing operation and the vast numberof students it attracts from around the world.The program benefits students by providing an opportunity to partici-pate in a prestigious company’s project, enhancing their skills, and enablingpersonal and career development. Moreover, it offers peer recognition, sta-tus, and monetary incentives ranging from 3, 000 to 6, 600 USD, dependingon the participant’s country [104]. To apply, students must submit detailedproject proposals to open source organizations approved by Google, such asthe Apache Software Foundation and Debian. These organizations, guidedby mentors who are usually regular contributors, evaluate and select pro-posals [103].The application process for organizations also requires detailed docu-mentation, including prior participation, communication strategies, projectmanagement plans, and an “idea list” that outlines potential projects andneeded skills [118]. Once an organization is approved, it assigns mentors tostudents to facilitate the development of proposed projects. These mentorsoffer technical support throughout the project.GSoC encompasses several phases: organization application, studentproposal submission, student selection, community bonding, and the cod-ing period. The coding period lasts three months, followed by mid-termand final evaluations. Successful students receive payments at three criticalmilestones: the beginning, middle, and end of the program [103].Since its inception, GSoC has successfully connected over 19,000 studentswith more than 18,000 mentors from 133 countries, producing over 43 millionlines of code across more than 800 open source organizations3. The programnot only aims to enhance students’ coding skills but also strives to promote along-term interest in OSS development. Google emphasizes this objective onits official Summer of Code webpage4, highlighting the program’s intent toencourage participants to continue contributing to OSS projects even after2https://summerofcode.withgoogle.com/3https://summerofcode.withgoogle.com/archive4https://summerofcode.withgoogle.com/122.3. Sentiment and Emotion Analysis in the Existence of Large Language Modelstheir GSoC journey ends.52.3 Sentiment and Emotion Analysis in theExistence of Large Language ModelsSentiment analysis (SA), also known as opinion mining, has been a signif-icant area of research in natural language processing (NLP) since its earlystages [58, 126]. It focuses on extracting subjective information from textualdata, providing insights into the sentiments and emotions expressed in textentities [90, 147]. The practical applications of sentiment analysis are exten-sive, ranging from analyzing customer reviews [65, 146] to monitoring socialmedia opinions [7, 141], and analyzing developers interactions [136]. One ofthe fundamental tasks within sentiment analysis is sentiment classification,which involves determining the overall sentiment polarity of a text entity,typically categorized as either as a binary form of positive or negative, or asa ternary form of positive, negative, or neutral [65, 126].Recent advancements have led to more granular tasks such as aspect-based sentiment analysis (ABSA), which extracts detailed sentiment in-formation about specific aspects of an entity rather than the entity as awhole [146]. For example for a sentence like “The pizza is delicious, butthe service is terrible.”, an ABSA tool can be used to identify that thesentiment of the sentence is positive about pizza and negative about theservice. Additionally, multifaceted analysis of subjective texts (MAST) hasemerged, focusing on specialized tasks such as hate speech detection [100],irony detection [143], and emotion detection [98], each addressing differentdimensions of sentiment in text.In this thesis, we refer to the simple form of sentiment analysis used toidentify binary or ternary sentiments as sentiment analysis, and the formthat identifies different emotions in text entities as emotion analysis.The Role of Large Language Models. The development of sentimentand emotion analysis tools has evolved significantly with the advent of largelanguage models (LLMs). These models, such as GPT-3 [14], PaLM [21],and ChatGPT [84], have demonstrated impressive capabilities in zero-shotor few-shot learning settings, shifting the focus from fine-tuning paradigmsto prompting strategies [134]. Studies have shown that LLMs perform sat-isfactorily on sentiment and emotion analysis tasks, without the need fortask-specific training [133, 149]. As a result of impressive performance of5https://buildyourfuture.withgoogle.com/programs/summer-of-code132.3. Sentiment and Emotion Analysis in the Existence of Large Language ModelsLLMs in these tasks, Zhang et al. [146] suggests a benchmark frameworkbased on LLMs to evaluate the effectiveness of sentiment analysis tools.However, the performance of LLMs on more complex tasks requiring deeperunderstanding or structured sentiment information, such as ABSA, remainsless effective compared to smaller language models like RoBERTa [71] whentrained on domain-specific datasets [26]. Despite this, LLMs show signifi-cant potential in few-shot learning settings, where they outperform smallerlanguage models when annotation resources are limited [149].While significant progress has been made in sentiment and emotion anal-ysis, several challenges remain. Understanding complex linguistic nuancesand cultural specificity is a major challenge, as sentiment is often shadedwith subtlety and context-dependent meanings [100]. Additionally, extract-ing fine-grained and structured sentiment information, particularly in taskslike ABSA, is challenging but useful for summarizing large-scale data effi-ciently [146].2.3.1 Sentiment and Emotion Analysis in SoftwareEngineeringSentiment analysis has become a significant area of interest within the do-main of software engineering, particularly for understanding and improvingthe collaborative dynamics and productivity of development teams. Thesetools are utilized in various domains of software engineering. Whiting et al.[136] analyzes developers communications at Amazon MTurK, they applysentiment analysis on participants’ chats. Ortu et al. [85] demonstratedthat developer emotions significantly impact productivity. Gkontzis et al.[46] uses sentiment analysis tools to analyze interactions of university stu-dents. Islam and Zibran [61] studies the sentiments of bug-fixing commitsin different times of a day.Pre-LLM. Before larger language models became popular, general SA toolshad shown limitations when applied to SE datasets [53, 80]. This has ledto the development of SE-specific tools over the past decade [1]. Severalstudies focus on developing new sentiment and emotion analysis tools tai-lored specifically for software engineering. For example, SentiCR [2] is oneof the most popular sentiment analysis tools in SE which is designed forcode review interactions. Senti4SD [15] and SentiStrength-SE [62] are twoof the other most used sentiment analysis tools in SE domain. Novielli et al.[82] compare and assess the performance of these tools in different softwareengineering contexts. Among these, Stanik et al. [108] used SentiStrengthto extract the sentiment of the text of issue descriptions to help predict142.4. Topic Modelingwhich issues newcomers are likely to select and resolve. They assumed thatsentiment can impact a newcomer’s decision on which issues to work on.For training and evaluation, studies use the existing datasets of senti-ment and emotion in SE domain. For sentiment analysis, some of the exist-ing dataset are: a dataset of Stack Overflow posts [15], a Jira dataset [86]composed of issue comments, a GitHub dataset [83] of pull request commentsand commit messages, and a Gerrit dataset [2] of code review comments.For emotion analysis, Murgia et al. [81] provides a dataset of issue commentslabeled for emotion extraction.Post-LLM. Similar to other domains, smaller language models like BERT[27], RoBERTa [71], and large language models show promise in improvingSA accuracy. Models typically containing fewer than one billion parameters,are fine-tuned with domain-specific data to achieve better performance inSE tasks [71, 144]. These models have been utilized to encode SE artifacts toenhance various SE applications [135, 144]. Despite the potential of smallerlanguage models, their performance can degrade significantly when there isnot enough labeled data for fine-tuning [144]. LLMs on the other hand,are trained on massive corpora of text and contain billions of parameters,such as GPT-3 [14] and LLaMA [123]. These language models facilitate SAtasks simply by providing a prompt without the need for domain-specifictraining data [11]. This adaptability has the potential to address the chal-lenges associated with fine-tuning models, particularly in contexts wheregeneralizability is an issue [145].2.4 Topic ModelingTopic models, which aim to discover topics within document collectionsbased on word co-occurrences, semantics, etc. have seen wide applicationsin various domains due to their ability to uncover underlying topics in textdata. Traditional topic models like Latent Dirichlet Allocation (LDA) havebeen effectively used for document retrieval, content recommendation, opin-ion mining, event detection, and trend analysis [10, 13, 22, 33, 131]. Thesemodels are particularly noted for their interpretability and ability to handlelarge text corpora. One significant application of topic models is in the realmof text analysis. For instance, Hoyle et al. [57] and Laureate et al. [66] high-light the use of topic models in analyzing large datasets to identify prevalentthemes and trends. Bai et al. [4] utilize topic models to analyze scientificarticles, incorporating citation graphs to recommend related articles. Baruaet al. [8] uses topic modeling to identify technology trends in Stack Overflow152.4. Topic Modelingposts over time. Furthermore, topic models have been employed to classifyshort texts, as demonstrated by Zeng et al. [142], who combined a neuraltopic model with a memory network for short text classification and topicdiscovery.Neural Topic Modeling Algorithms. In recent years, neural topicmodels (NTM) have advanced the field of topic modeling by addressing lim-itations of traditional models, such as scalability, and flexibility. ProdLDAis one such model that employs a variational autoencoder (VAE) frame-work to produce topics by directly modeling the document-topic distributionthrough a SoftMax function [107]. Another notable model, ETM (EmbeddedTopic Model), integrates word embeddings into the existing topic modelingprocesses that previously did not utilize text embeddings, thereby captur-ing semantic similarities [30]. Graphbtm [151] captures document topicsthrough a graph structure of word co-occurrences, also known as biterms.These examples illustrate the advancements in neural network architecturesand learning techniques that have enhanced the performance and applica-bility of topic models.Tools for Topic Modeling. Several tools have been developed to fa-cilitate topic modeling, each with its own set of features and capabilities.Early tools such as MALLET [75], gensim [95], STTM6, ToModAPI [70],and tomotopy [68] provide foundational functionalities for topic modelingbut lack support for NTMs. More sophisticated toolkits like OCTIS [120]and TopMost [137] are developed recently. OCTIS includes several NTMmethods, and evaluation techniques, making it a useful tool for research.TopMost also covers a wide range of topic modeling scenarios. It also de-couples model implementations from training processes, thereby simplifyingthe extension and adaptation of new models. Additionally, BERTopic [50]is a well known algorithm and tool that has gained attention for its abilityto generate topics using transformer-based embeddings. BERTopic clustersdocument representations, and is very effective for discovering topics in largeand complex datasets. This tool is particularly useful for those looking toleverage state-of-the-art language models to enhance topic modeling perfor-mance.BERTopic has been used in several studies in software engineering. Dia-mantopoulos et al. [29] apply BERTopic to semantically extract issue textualdata from 656 Apache Software Foundation projects. They identified 10,510distinct topics to use for bug assessment, and role extraction. A study onself-admitted library migrations across software ecosystems is done by Gu6https://github.com/qiang2100/STTM162.5. Identification of the Literature Gapet al. [51]. They utilize BERTopic for an analysis of migration patterns anddomain-specific trends. Tao et al. [119] designed a recommendation systemto suggest relevant open source libraries to developers based on the contextof their existing code.2.5 Identification of the Literature GapBy exploring the body of literature on mentorship in OSS, we identify somegaps that have not been a point of focus by previous studies, which we aimto explore in this thesis.Analysis of Mentor-Mentee Interactions. Previous research has fo-cused on the outcomes of mentorship or the challenges faced by participantswithin these programs. There is a notable gap in the literature regardinga detailed, interaction-based analysis that considers the actual communi-cations between mentors and mentees. Most studies rely on self-reporteddata or observational insights without delving into the details of interac-tion patterns, the evolution of discussion topics, or the specific emotion andsentiment changes over the mentorship period. This thesis aims to providean examination of these interactions to understand the interactions betweenmentors and mentees in GSoC.Temporal Analysis of Mentorship Processes. While some studieshave touched upon the phases of mentorship, there is a lack of detailedtemporal analysis that tracks the progression of mentorship interactionsthroughout the duration mentorship. Understanding how mentorship needsand effectiveness evolve over time is useful for designing phase-appropriatestrategies that can be more responsive to the changing needs of mentees andmentors.Role of Emotion and Sentiment Analysis in Mentorship. Al-though the significance of emotions and sentiment in educational and orga-nizational settings is recognized, their specific impact within OSS mentor-ship has been underexplored. This landscape of mentor-mentee interactionscould offer deeper insights into the relations that contribute to successfulmentorship. We seek to fill this gap by employing sentiment and emotionanalysis techniques to quantify and analyze these measures within mentor-ship communications.Impact of Mentorship on Mentees’ OSS Engagement. There is ascarcity of studies that link mentorship during specific programs like GSoCto engagement in OSS projects post program. Little is known about howthese mentorship experiences improve the contributions of mentees. This172.5. Identification of the Literature Gapresearch aims to quantify the mentees’ engagement post-GSoC comparedto their engagement pre-GSoC to assess the impact of these initiatives onmentees’ improvement in contributing to OSS. By addressing these gaps,we aim to contribute to the existing knowledge about OSS mentorship andmentor-mentee interactions, and provide actionable insights and recommen-dations to potentially enhancing mentorship frameworks in OSS environ-ments.Two of the works most closely related to this thesis are the studiesby Silva et al. [104], which examined student motivations and contributionsin Google Summer of Code, and Fronchetti et al. [43], which investigatedfactors attracting newcomers to onboard on OSS projects. While Silva et al.[104] provided valuable insights into the GSoC program based primarily onsurveys and interviews, Fronchetti et al. [43] used a data driven approachto analyze newcomer joining rates and project characteristics. However, ourwork differs significantly from both studies in several aspects.The main contributions of this thesis, distinguishing it from current lit-erature including the aforementioned studies, are threefold. First, we in-troduce the OpenMent dataset, a novel compilation of about half a mil-lion GitHub interactions specifically focused on GSoC projects, providing aunique resource for studying mentor-mentee dynamics in OSS. This datasetallows for a more comprehensive analysis than previous studies. Second,we employ sentiment analysis, emotion detection, and topic modeling, toanalyze these interactions at a depth not previously explored in OSS men-torship studies. Our approach provides a nuanced understanding of theactual content of mentor-mentee communications. Finally, our analysis ofhow communication patterns and themes evolve throughout the GSoC pro-gram, and their relation to post-program engagement, offers new insightsinto the effectiveness of structured mentorship in OSS. These contributionscollectively provide a more detailed and data-driven understanding of men-torship in OSS, addressing gaps in the current literature which has largelyrelied on either self-reported data or high-level project metrics, and lackeddetailed temporal analysis of mentor-mentee interactions.18Chapter 3OpenMent: A Dataset ofDevelopers Interactions inthe Google Summer of Code3.1 IntroductionArtificial Intelligence (AI) has become critical in enhancing various aspectsof software engineering, from automated code generation [20] to sophisti-cated bug detection [64, 91]. Recent advancements in AI for text analysis,particularly with the development of more sophisticated language models,have significantly enhanced the precision of textual data examinations, asevidenced by numerous studies [55, 87, 132, 139]. However, as highlightedby Di Franco and Santurro [28], the application of AI in exploring the socialdynamics of human interactions remains relatively underexplored. Althoughresearch has frequently addressed collaboration in OSS, focusing on areassuch as demographic impacts [93] and the effects of varying coding experi-ence levels [140], investigations into how developer roles like mentors andmentees influence interactions are sparse. Utilizing AI to analyze the exten-sive communicative data within these collaborative settings could explain ef-fective mentorship practices, enhance project outcomes, and provide deeperinsights into the communications and engagements of successful softwaredevelopment teams.Advances in deep learning models have demonstrated promise in improv-ing the analysis of textual data and when used on comprehensive datasets,these models can identify patterns, and infer communication styles [55, 87,132]. However, a critical limitation is the scarcity of datasets specifically fo-cusing on the textual data of a specific kind like the interactions of mentorsand mentees within OSS development. Recognizing this, while numerousdatasets have been curated primarily to train or tune language models forbroad linguistic capabilities [32, 47], datasets explicitly crafted to under-stand specific types of interactions [69], such as mentor-mentee dynamics193.1. Introductionin software development, remain scarce. This gap underscores the need forspecialized datasets that not only enrich our understanding of these inter-actions but also support targeted analysis within the context of real-worlddevelopmental processes.GSoC program stands as a popular example of an initiative that fostersmentorship relationships between seasoned developers and aspiring contrib-utors [103, 118]. This work introduces a novel dataset compiled from GSoCprojects and is distinguished by its focus on the interactions among mentorsand mentees engaged in the GSoC program. By carefully curating GitHub is-sues, pull requests, comments, and commits generated within GSoC projects,this dataset offers a unique lens through which to study the dynamics of men-torship and contribution within a guided open source context. This datasetis intended to support research into several potential areas of analysis, asoutlined below, though these represent only a subset of possible researchdirections:• Understanding mentorship practices: Patterns in communicationto identify effective mentorship strategies, common challenges, andareas where support can be improved.• Evaluating collaborative dynamics: Uncover patterns of team-work between mentors and mentees to determine factors contributingto successful project outcomes and participant satisfaction.• Assessing the impact of the GSoC program: Analyze data overmultiple years to measure the long-term impact of the program onmentees’ skill development and career paths.• Optimizing future mentorship programs: Use insights derivedfrom the dataset to refine the design and execution of GSoC and similarinitiatives.The remainder of this chapter is structured as follows: Section 3.2 pro-vides a detailed overview of the dataset, including its scale and statistics,and guidance on how to effectively utilize the dataset. Section 3.3, exploresexisting similar datasets and details how OpenMent differentiates itself. Fi-nally, Section 3.4, describes the methodology employed in assembling theOpenMent dataset, ensuring a comprehensive understanding of its utility.203.2. The OpenMent Dataset3.2 The OpenMent DatasetThe OpenMent dataset comprises a comprehensive collection of GitHubcommunications in the form of issue and pull request comments, as well ascommit messages, representing mentor-mentee interactions. Alongside themain focus of the dataset which is the textual communications, this datasetincludes extensive metadata from GitHub repositories, aimed at supportingresearch in text and data analysis within OSS. This dataset draws from his-torical data of GSoC projects. GSoC’s structure is marked by four criticalmilestones-start of the program, coding begins, midterm evaluation, and thelast day of the program. OpenMent makes it very easy to study participants’interactions relative to these milestones.3.2.1 Scale and StatisticsThe OpenMent dataset compiles an extensive collection of over 2 millionmessages and comments, drawn from GitHub communications on issues andpull requests (PRs) between developers who participated in GSoC from 2016to 2023. This dataset includes communications from both within and outsidethe official GSoC program durations, providing a comprehensive view ofmentor-mentee interactions before, during, and after the program, therebyenhancing our understanding of these dynamics throughout the duration ofthe projects. The rest of this chapter only focuses on the subset of the dataconfined to the official GSoC period. Further details on dataset constructionare provided in Section 3.4. This subset of OpenMent has more than 500thousand messages and comments.Distribution of Contributions by RoleThe OpenMent dataset provides a detailed view into the distribution ofcommits and comments within GSoC program duration.The dataset categorizes commit messages based on the role of the con-tributor: mentors and mentees. As shown in Figure 3.1, mentors contributed114,614 commits, which represents approximately half of the total commits,and mentees were responsible for 101,748 commits. This nearly balanced dis-tribution underscores the active engagement of both mentors and menteesin the coding aspects of the projects, reflecting a collaborative environmentwhere learning and guidance are dynamically exchanged through hands-oninvolvement.213.2. The OpenMent Dataset47.0%53.0%Commit Distribution by RoleTotal: 216,362Mentee: 101,748Mentor: 114,61428.3%30.4%41.2%Comment Distribution by RoleTotal: 346,630Mentee: 98,165Mentor: 105,529Other: 142,936Figure 3.1: Distribution of commit messages and issue/PR comments withinthe GSoC program duration, showcasing the involvement levels of mentorsand mentees.The distribution of comments on issues and pull requests is also de-picted in Figure 3.1 presenting a similar picture. Mentees contributed 98,165comments, compared to mentors who contributed 105,529 times, indicatingmentors’ role in providing feedback, clarifications, and direction. The cate-gory labeled “Other” encompasses 142,936 comments. These are from othermembers or contributors not directly categorized as mentors or mentees buthave commented on the issues and PRs related to GSoC projects.These statistics highlight the intensive communicative exchange that oc-curs within the GSoC, with mentors and mentees actively engaging in bothcode contributions and discussions. This balanced participation betweencommits and comments suggests the possibility of a robust mentorship en-vironment where mentees are not only guided but also encouraged to con-tribute to technical discussions.Table 3.1 presents gender distribution among participants, reflectingbroader trends in the open source community and the tech sector.Role Male Female UnidentifiedMentees 1521( 78%) 207 ( 11%) 211 ( 11%)Mentors 1856 ( 83%) 216 ( 10%) 176 ( 8%)Both Roles 175 ( 76%) 36 ( 16%) 18 ( 8%)Table 3.1: Gender distribution within the GSoC program duration.223.2. The OpenMent DatasetKey Points::• The balance of commits by mentees (47%) and mentors (53%) demon-strates substantial mentee engagement in coding activities within theprogram.• The prevailing gender disparity within GSoC, with male participantssignificantly outnumbering females, echoes established patterns in theOSS community and the technology industry [77, 93, 129].• The GSoC program attracts mentors from a wide range of companies,ranging from large industry leaders, to small and medium enterprises.7GSoC projects themselves span a broad spectrum of domains and ap-plications, providing valuable exposure to varying software develop-ment mentorship practices employed by different industry segments.Distribution of Contributions by YearFigure 3.2 shows the distribution of these communications within differentyears of the GSoC programs, showing a significant volume of exchanges eachyear. In 2020, the total number of communications reached its peak, com-prising 34.1K commit messages and 63.5K issue/PR comments. Generally,the dataset shows a healthy balance between commit messages and issue/PRcomments. While there is a noticeable decline in communications from 2021onwards, the overall trend indicates a robust interaction level throughout theprogram’s history. The fluctuations in the volume of communications couldbe attributed to various factors such as changes in the program’s structure,the global economic environment, or the number of projects and partici-pants each year. The temporal distribution of these communications offersvaluable insights into how communication trends within the GSoC programevolve in response to external and internal program changes. This under-standing can help in planning future iterations of the program, ensuringthey are better tailored to facilitate effective mentorship and collaboration.7https://www.gsocorganizations.dev/233.2. The OpenMent Dataset20162017201820192020202120222023GSoC Program Year020,00040,00060,00080,000100,000Count32.2K 30.7K 25.8K 28.9K34.1K26.1K 22.4K 16.3K37.3K 37.8K55.8K 42.7K63.5K47.1K34.1K28.3KCommit MessagesIssue/PR CommentsFigure 3.2: Yearly distribution of communications in the GSoC program,indicating the total number of commit messages and issue/PR commentsfrom 2016 to 2023.3.2.2 How to Read the OpenMent DatasetThis section provides guidance on navigating the dataset, which consistsof several relations (tables) structured to offer a comprehensive view of theparticipant roles, communications, and their GitHub activities within GSoCprojects. The detailed structure of these relations are shown in Appendix Aof this thesis. Here is a brief explanation of each relation:• users: This relation serves as the foundational index of participants.Each row represents an individual participant of the GSoC, identifiedby a unique, anonymized user id. This table also specifies the probablegender of each user, allowing for demographic analysis. The detailedstructure of the users relation is shown in Table A.1.• projects: Each project within GSoC is represented here with a uniqueproject id. It includes metadata such as the program year and aboolean field named mentors fully detected indicating the complete-ness of mentor data for each project. Each project in GSoC mayofficially have one or several mentors, and although all the projectsin OpenMent have at least one of their mentors identified, not all ofthem have their mentors fully identified. The detailed structure of theprojects relation is shown in Table A.2.243.2. The OpenMent Dataset• project roles: This relation links users to projects and specifies theirroles (either mentor or mentee). It uses foreign keys project id anduser id from the projects and users relations, respectively, indicatingthe structure and roles within each project. The detailed structure ofthe project roles relation is shown in Table A.3.• issues pull requests: This records the issues and pull requests. It in-cludes fields such as type (issue or pull request), title, body, and status(open or closed), as well as timing data (creation day and close day).The detailed structure of the issues pull requests relation is shown inTable A.4.• project issue pr map: A mapping relation that associates issuesand pull requests with the projects they pertain to.The detailed struc-ture of this relation is shown in Table A.5.• comments: Captures comments made on issues and pull requests,linked by project id and issue pr id. It includes the comment text andthe creation day. The detailed structure of the comments relation isshown in Table A.6.• commits: Documents commits made to GSoC projects’ repositories.Each record includes the SHA, the project and user it is associatedwith, the commit message, and the creation day. The detailed struc-ture of the commits relation is shown in Table A.7.Many analyses will require joining relations on their key fields. For example,to explore the contributions of mentors versus mentees, one might join theusers, project roles, and commits relations. Structured queries can be usedto extract complex patterns and relationships. Each relation is stored in acomma-separated value (CSV) file for easy access, and can be imported andused in different tools, programming environments and database systems.Understanding the Dataset’s Date FieldsIn order to facilitate comparisons across different years of GSoC, it is crucialto comprehend the relativity of the date fields within the dataset. OpenMentis structured so that the start of the program is designated as day 0 in allprogram years. Any activity occurring on this initial day is assigned acreation day in the range of [0:1), meaning they took place within the firstday of the program but do not reach the second day, which is denoted as day0. Conversely, activities on the day before the program starts are tagged253.3. Related Datasetswith a creation day in the range of [-1:0), indicating they happened withinone day prior to the program’s commencement.The GSoC timeline within the dataset is normalized such that each pe-riod between major milestones (program start to coding, coding to midtermevaluation deadline, and midterm evaluation deadline to end of the pro-gram) is uniformly represented across all years, irrespective of the actualnumber of days these periods may have spanned in any given year in re-ality. Specifically, the normalized durations are 28 days from the programstart to the start of coding, 42 days from coding to the midterm evaluationdeadline, and 70 days from midterm evaluation to the program’s conclusion.This normalization results in a total of 140 days, with the program’s lastday marked as day 139.Due to this normalized and relative approach to the dataset’s tempo-ral structure, it is important to note that the creation day or close day fortwo corresponding activities from different program years are not directlycomparable. A difference in creation day of 1, for example, could repre-sent activities that are chronologically much further apart than a single day,potentially spanning different years of the program. This normalized struc-ture ensures consistency in analysis but requires careful interpretation whencomparing temporal data points across different iterations of the GSoC.3.3 Related DatasetsWhile there are lots of datasets in the domain of OSS development [47,74, 79], those focused on a specific matter within GitHub projects remainless common. Our dataset provides a unique perspective on the mentorshipdynamics within a structured open source program, emphasizing the com-municative fabric that tries to support software development. We examinerelated datasets and highlight the distinctive value of our work.Open Source Collaboration Datasets: Datasets such as GHTor-rent [48] and GH Archive8 provide comprehensive records of GitHub events(issues, pull requests, commits). They offer a broad perspective on opensource collaboration but lack the fine-grained selection of repositories relatedto a specific program like GSoC. The GitHub Torrent and GitHub Archiveoffer functionalities similar to the GitHub API, with the added benefits ofsimplicity and potentially access to larger repositories which are not acces-sible by GitHub API. While the GitHub API provides direct programmatic8https://www.gharchive.org/263.3. Related Datasetsaccess to GitHub’s features and data, GHTorrent and GH Archive can beeasier to use and may offer access to more extensive historical data.Code-Centric Datasets: Datasets such as “the big code project”9, andCodeSearchNet [59] emphasize the code itself, others focus on some specifictasks like code clone identification [115], or code repair [63]. While theymight include some natural language data, their core focus remains on sourcecode rather than interpersonal communication dynamics. Our dataset offersa complementary perspective, highlighting the social and guidance aspectsof code development.Educational Datasets: Datasets extracted from Q&A platforms orMOOC forums [97] share some similarities in terms of knowledge exchange.However, they lack the project-oriented context, software engineering focus,and the specific OSS development interactions that characterizes our GSoCdata.3.3.1 OpenMent DifferentiationOur dataset presents distinct advantages and addresses limitations presentin comparable datasets within the open source communication domain.Our dataset possesses the following distinctive attributes:Mentor-Mentee Focus: Our primary strength lies in the explicit iden-tification of mentor and mentee roles. This enables analyses focused on guid-ance patterns, knowledge transfer, and the evolution of the mentee’s skillswithin the project scope.GSoC Context: Our dataset adheres to the boundaries of GSoC, offer-ing a well-defined framework for project timelines, mentorship expectations,and potential success indicators, a context absent from more general opensource datasets.Communication-Centric: We place a strong emphasis on textual in-teractions over raw code and contribution statistics. This focus allows forthe analysis of communication style, sentiment, responsiveness, and the col-laborative processes inherent in OSS.Data Privacy: In contrast to other datasets that may include iden-tifiable information, the OpenMent dataset ensures the privacy of GSoCparticipants by anonymizing all user and project identifiers. This approachaddresses concerns raised in other platforms, such as GHTorrent and GHArchive, where the presence of user IDs, emails, and repository names hasled to individuals being repeatedly contacted for surveys and other inquiries.9https://www.bigcode-project.org/273.4. Construction of OpenMent3.4 Construction of OpenMentThe main focus of the OpenMent is to construct a well organized collec-tion of GitHub textual interactions between mentees and mentors in GSoC.There are specific requirements and challenges associated with gatheringdata from the GSoC program for the purpose of analyzing interactions be-tween mentors and mentees on GitHub.To create a dataset conducive to this analysis, two key pieces of infor-mation were necessary to be gathered from the GSoC website:• GitHub Repository URLs: This is essential as it provides accessto the codebase where the mentee and mentors interact. Without this,we could not extract the GitHub data crucial for our dataset.• Identification of Participants: Distinguishing between menteesand mentors within the GitHub repository is vital. Knowing whoeach participant is allows for precise analysis of their interactions andcontributions.283.4. Construction of OpenMent3.4.1 Collecting Projects’ Data From GSoCFigure 3.3: GSoC website showing the list of projects. Elements of interestfor our dataset are: Contributor (mentee), Mentor, the “View Code”, and“View project details” buttons. Note: This is an image of a reconstructionof the GSoC website, not the original.As illustrated in Figure 3.3, projects are listed on the GSoC website. Foreach project, we are interested in 1. Contributor (Mentee): Identified as theindividual primarily responsible for the project, 2. Mentor: One or moreindividuals listed as providing guidance and support to the mentee, 3. CodeRepository URL (“View Code” button): Ideally, this should link directly toa repository, although it varied in practice, sometimes pointing to personalor organizational web pages instead. It is important to note that the nameslisted for mentees and mentors on the GSoC website were not standardized;they could be full names, nicknames, usernames, or anything that the par-ticipant wanted. This inconsistency posed a challenge for accurately linkingGitHub profiles to the names listed on the GSoC website. Additionally,the URL provided by the “View Code” button did not consistently lead to293.4. Construction of OpenMenta GitHub repository, necessitating further verification steps to ensure therelevance and accuracy of the links collected.Each project also has a unique project ID, captured via the “Viewproject details” button’s URL, which aided in distinguishing GSoC projectsthroughout our data processing. We collected these details for all the 8, 633projects listed on the GSoC website from 2016 to 2023, the years for whichinformation was available at the time of data collection (Dec. 2023).3.4.2 Repository URL Verification and ExtractionAs mentioned earlier, not all “View code” links directly pointed to a coderepository. For the ones not directly pointing to a GitHub repository, weanalyzed the linked page to extract any potential GitHub repository URLspresent, and if a unique repository URL was found, considered as the projectrepository URL. The projects without an identified repository URL werefiltered out at this stage. This step resulted in a curated list of 3,937 projectswith verified GitHub repository URLs, enabling further analysis.3.4.3 Participant Identification and VerificationTo accurately attribute project contributions and interactions to specificmentees and mentors, we needed to resolve the challenge of diverse identifica-tion methods on GSoC website (real names, nicknames, GitHub usernames,etc.). Our approach was as follows:• GitHub User Data Collection: Each repository on GitHub has anofficial list of contributors; we fetched this official contributor list ineach identified repository and also gathered a list of any users withrecorded activity (commit, comment, issue creation, pull request) inthat repository within the GSoC timeline. For every individual ineither of these two lists, GitHub username, their name on the GitHubtheir account and their name in their commits details were recorded.This way, for each of our identified GitHub repositories, we had a listof contributors with their GitHub username, and two potentials fortheir name.• Name Matching: Following a methodology similar to Bird et al.[9], we utilized the Levenshtein distance similarity score to match thenames obtained from the GSoC website with the GitHub user data.Each participant name obtained from GSoC was compared to its cor-responding GitHub repository’s contributors info gathered in previous303.4. Construction of OpenMentstep. The highest similarity score was considered as a match. A matchwas considered valid only if the similarity score exceeded 70, ensuringa high confidence level in the identification process.• Data Filtering: One of the main objectives of our dataset construc-tion is to have mentees and mentors identified. Projects for whichthe mentee and at least one mentor could not be accurately identifiedwere excluded from the dataset. The remaining 2, 272 projects eachhad their mentee and at least one mentor identified. The identificationstatus of mentors (whether all mentors were identified or not) is storedas a metadata for each project in the dataset.3.4.4 Collecting GitHub DataHaving identified GitHub repository URLs for each GSoC project along-side its identified mentee and mentor GitHub usernames, we progress to theGitHub data collection phase. This phase is done using only the GitHub’s of-ficial API, and is designed to add participants’ interactions from each GSoCproject’s GitHub repository, specifically focusing on issues, pull requests(PRs), and commits.Collecting Issues and Pull RequestsIssues and pull requests are central to collaborative software development,serving as primary channels for proposing changes, reporting bugs, and dis-cussing enhancements [19]. In GSoC program, the mentee is the primaryfocus of each project, with a goal of improving their skills and potentiallypreparing them for the future projects. A mentee may be asked to work on ahuge GitHub repository owned by the company with thousands of developersworking and collaborating on. The main person that distinguishes contribu-tions related to the GSoC project on a repository is the mentee. Collectingissues and PRs authored by or mentioning the mentee can provide a com-prehensive view of their involvement and impact within the project. Thisapproach not only captures the contributions of the mentee but also high-lights their engagement and interaction with the wider project community.By focusing on these interactions, we can analyze the mentee’s influence onproject development, their responsiveness to community feedback, and theirability to work within a team.Process: For each project, we retrieved all issues and pull requests wherethe mentee was either the author or mentioned. This included the full313.4. Construction of OpenMentcontent of the issue/PR, as well as all associated comments, and reviewcomments. This approach ensured that we captured a wide range of inter-actions, from initial steps to final resolution or the potential merge of thepull requests, providing a detailed narrative of the mentee’s contributionsand the collaborative dynamics of the project team.Collecting CommitsCommits are the backbone of version control history, representing discretechanges to the project’s codebase [19]. Collecting commits authored bymentees or mentors offers direct evidence of their technical contributions tothe project. This data helps quantifying the extent of each participant’sinvolvement in the development process. Furthermore, analyzing commitpatterns can shed light on mentorship practices, and the frequency of con-tributions.Process We collected all commits made to the project repository by thementee or mentors during the GSoC timeline. This data allows for a detailedanalysis of each participant’s contribution to the project. By examiningcommits, we could assess the complexity and significance of contributions,and the evolution of the project over time.3.4.5 Normalization of GSoC Program MilestonesThe GSoC’s timeline is defined by four critical milestones: start of the pro-gram, coding begins, midterm evaluation, and the last day of the program.However, the duration between these milestones vary from year to year,presenting a challenge for direct comparison. To address this challenge, wefirst align the start of each program year to a common baseline (day 0).Following this alignment, we undertake a resampling process to normalizethe duration of periods between milestones across all years. This methodinvolves adjusting the actual number of days into a fixed number of seg-ments or “normalized days” ensuring that the periods pre-coding, codingto midterm, and midterm to program end are comparable across differentprogram iterations.This approach is akin to resampling techniques used in time series anal-ysis, where data points are interpolated or aggregated to fit a uniform timescale [31, 60]. Additionally, it draws upon the concept of event history anal-ysis, where time-to-event data are adjusted to facilitate comparisons acrossdiverse temporal frameworks [138].323.4. Construction of OpenMentBy aligning the milestones and creating a uniform timeline, patternsand trends can be identified more effectively [105]. The duration of periodsbetween the milestones were determined based on the observed average du-rations, reflecting the relative importance and typical length of each phasewithin the GSoC timeline. Through the alignment of milestones and the re-sampling of time periods, OpenMent establishes a methodologically soundbasis for analysis of GSoC programs across different years.3.4.6 Gender IdentificationWe use the NamSor API10 for gender identification of GitHub users partici-pating in the GSoC programs. NamSor categorizes individuals as either maleor female, and provides a probability score for each classification. To en-hance the reliability of gender identification, we adopted a confidence thresh-old approach. Specifically, NamSor assigns a probability score to its genderpredictions, quantifying the confidence in each inference. Although Sebo[101] suggests using all gender assingments from NamSor regardless of theirconfidence scores, to decrease the probability of misclassification, followingthe methodology established by Zhao [148] and considering the classificationerrors reported by Sebo [101], a confidence threshold of 0.6 was set for gen-der identification. Gender classifications with a probability score below thisthreshold were categorized as “unknown” to ensure that only high-confidencegender identifications were included in the study.Selection of NamSor for gender identification in OpenMent is justifiedby its linguistic diversity coverage, high accuracy in gender inference, andprior successful applications in similar research contexts [39, 41, 94, 148].3.4.7 Anonymizing DataOpenly accessible data has tremendous potential for science, but, in re-searches with human subjects, there is always a trade off between privacyand preserving high-quality open data [24]. The goal of the anonymizationis to preserve participant confidentiality [3]. We want to achieve this inthe OpenMent dataset while maintaining the dataset’s integrity and utilityfor research. This section outlines the methods used to anonymize personalidentifiers and sensitive information.• Hashing Identifiers: We applied a cryptographic hash function (SHA-3 256-bit)11 to all unique identifiers, such as GitHub usernames, issue10https://www.namsor.com/11https://csrc.nist.gov/projects/hash-functions/sha-3-project333.4. Construction of OpenMentIDs, pull request IDs, and commit SHA values. A randomly generatedbut consistent hashing salt was used across all operations to ensuredata cleanliness and enable linkage of activities by the same user un-der their anonymized ID. This approach prevents easy tracing of databack to individuals and protects against de-anonymization throughhash matching [45].• Removing Usernames and URLs: We eliminated all mentions ofusernames (indicated by “@”) and URLs from textual data includingcomments, pull requests, and commit messages. This step is to preventany direct or indirect identification of participants or linkage to theiractivities on GitHub or external sites.While there is no single correct way of anonymization of data [24], thetechniques used allow for the secure study of mentor-mentee interactionswithin the GSoC context. The OpenMent’s approach contrasts with manyexisting GitHub datasets that do not incorporate anonymization measures.The absence of such measures has led to privacy concerns, exemplified bythe creation of online lists where individuals request not to be contacted forresearch purposes12. These lists highlight the growing need for privacy indatasets derived from public platforms like GitHub.While our methods do not guarantee complete anonymization (since thedata can still be compared with public GitHub data), they mitigate the riskof direct or easy bulk identification. This approach protects participants pri-vacy and reduces the unsolicited direct contacts that stem from inadequatedata anonymization. We hope that this leads to a more ethical frameworkfor conducting research on open source collaborations.3.4.8 Analysis PotentialWith the collection of issues, pull requests, their respective comments andreview comments, alongside commits, our dataset now provides a rich collec-tion of project interactions and contributions. This data allows for analyses,including but not limited to:• Contribution Analysis: Evaluating the quantity and quality of con-tributions from mentees and mentors.• Collaboration Patterns: Understanding how project participantsinteract through issues and pull requests.12https://github.com/notslang/do-not-survey-list343.4. Construction of OpenMent• Community Engagement: Assessing the mentee’s engagement withthe wider project community through mentions in issues and PRs.• Development Practices: Analyzing commit patterns to infer aboutcoding standards, contribution frequency, and areas of active develop-ment.35Chapter 4MethodologyIn this chapter, we will discuss the methodology to conduct this research.Figure 4.1 shows an overview of the process. We first collect the dataset,followed by filtering and pre-processing steps to prepare it for the analysis.To answer RQ1, we filter the dataset to extract the comments of mentorsand mentees involved in the project. Then, we apply sentiment and emotionanalysis and compare the results using statistical tests. To answer RQ2,we apply topic modeling and analyze the main topics and their evolutionover time. For RQ3, we investigate the impact of the GSoC program onmentees by analyzing the improvement in their GitHub contributions afterparticipating in GSoC compared to their contributions before participation.In the following, we elaborate on each part.364.1. DatasetOpenMentEmbedding ModelTask Speci c Instructions K MeansThematicAnalysisSuccessAnalysisRole-basedComparisonTemporalAnalysisTOPICS & THEMESManual topic&theme namingSentimentEmotionCountVectorizerTF - IDFLogiatic RegressionMTEB DatasetsTrainStatistical testst-testANOVAchi2-testMetricsImpactShareSucsessRQ1RQ2RQ3RQ1RQ2RQ3RQ1RQ2RQ3RQ1RQ2RQ3Figure 4.1: Overview of our methodology.4.1 DatasetAs the process of collecting dataset is complicated, we have dedicated aseparate chapter to explain how the dataset is collected and built, and whatkind of records and features it includes in the next chapter, i.e. we nameour collected dataset OpenMent. In this section, we therefore, only explainthe necessary information for understanding the rest of the methodology.OpenMent Composition The dataset consists of GitHub communi-cations in the form of issue and pull request comments, and commit374.1. Datasetmessages, representing interactions between mentors and mentees withinthe GSoC programs from 2016 to 2023.Each communication entry in the dataset includes:• Role: Identifies whether the author is a mentor or mentee.• Creation Day: The date the comment or commit was made, normal-ized to the GSoC timeline (explained below). For example, a creationday of 0 means the first day of the GSoC program for that year.• Project ID: A unique identifier for the GSoC project associated withthe communication.• Author ID: A unique identifier for the author of the communication.• GSoC Program Year: The specific year of the GSoC program towhich the communication belongs.The duration of the GSoC program slightly varies each year. To en-sure comparability across different program years in our analysis, we havenormalized the GSoC timeline to a consistent 140-day framework with fourmain milestones. These milestones are described below.• Day 0: Start of the program.• Day 28: Coding officially begins• Day 70: Midterm evaluation deadline• Day 139: Final day of the programThis normalization of the timeline enables comparing the analysis resultsover similar periods for all the records in our dataset.To answer the research questions, we also added the following metadatato the records.• Week Number: Each communication entry is also assigned a weeknumber, calculated from the creation day.• Period: Each communication entry is categorized into one of the threemain periods based on the creation day:– Community Bonding Period: Day 0 to Day 27.– First Work Period: Day 28 to Day 69.– Second Work Period: Day 70 to Day 139.384.2. RQ1 Methodology4.2 RQ1 MethodologyFor RQ1, we should identify the sentiment and emotions expressed in thecommunications by mentors and mentees, and analyze whether those differbased on the roles. We also need to analyze the differences of expressedsentiments and emotions across different times in the GSoC program. Toachieve this, we use the issue and pull request comments from the Open-Ment dataset. Each record includes metadata that identifies the role ofits author. Therefore, we can separate the communication records of men-tors and mentees. As the OpenMent dataset does not include predefinedlabels for sentiment and emotion, we apply a text classification strategyto determine the sentiments and emotions. We identify the sentiments as“negative”, “positive”, or “neutral”; and the emotions as “sadness”, “joy”,“love”, “anger”, “fear”, and “surprise”. These sentimenets and emotionsare used in different metrics we define for comparison against each of thetwo roles, and also across different timelines, i.e., the periods between thekey milestones of the GSoC program, as mentioned above. To compare thedifferences, we apply several statistical tests. The details of each process isexplained below.4.2.1 Sentiment ExtractionFor sentiment extraction, we convert the text to vector representations,known as embeddings. For this purpose, we choose the E5instruct model.This model excels in generating text embeddings for classification and clus-tering tasks. This model was chosen based on its top performance in theMassive Text Embedding Benchmark (MTEB) [79], evaluated for its abilityto handle diverse text classification scenarios. Because of the model’s out-standing performance on MTEB, we decided to use the same methodologythat was employed to evaluate the model for our purposes.In order to use the Mistral model, our dataset should be changed toinclude task-specific instructions to be used as input of the model. Thetask-specific instructions are then fed into the model and the Mistral modelgenerates embeddings tailored towards the given instruction. For example,the embeddings generated for the same text could be different if instructionsare asking the model to generate the embeddings for sentiment versus foremotion classification tasks. Figure 4.2 illustrates this process. The processof generating text embeddings for our dataset involves:• Task Instruction: Crafting specific instructions for sentiment analysis,similar to those used in the MTEB evaluation [79].394.2. RQ1 Methodology• Model Input: Combining the task instruction with the GitHub com-ment text.• Embedding Generation: Feeding the concatenated input into the modelto generate a vector representation of the comment, capturing its se-mantic meaning.We use the following instruction to prompt the model to generate em-beddings for sentiment classifcation task.Instruction: Classify sentiment of the github comment into positive,neutral, or negativeAfter this step, the sentiment embeddings are ready for our classifier.Following MTEB [79], we utilize a logistic regression model from the scikit-learn library [88], configured with its default parameters, to classify textembeddings into sentiment categories. We train the classifier on 27,500 ex-amples and evaluate on 3,500 examples, taken from the train and test splitsof the sentiment extraction dataset used in MTEB [73]. For the Negativeclass, evaluation metrics show a precision of 0.71 (the percentage of cor-rectly identified negative comments among those labeled as negative), recallof 0.74 (the percentage of actual negative comments correctly identified bythe model), and an F1-score of 0.72 (calculated as 2 × precision×recallprecision+recall); allmetrics stand at 0.68 for the Neutral class; and for the Positive class, preci-sion reaches 0.78, recall is 0.75, and F1-score is 0.76. The overall accuracyof the classifier is 0.72 (the percentage of total test examples correctly clas-sified across all categories). After training, we apply this logistic regressionmodel to assign a Sentiment Label of either positive, negative, or neutral toeach comment in our dataset. Following this, we introduce a correspondingsentiment score to enable numerical analysis. The sentiment score (S(d))quantifies the sentiment of comment d in our dataset as either -1 (negative),0 (neutral), or 1 (positive).404.2. RQ1 Methodologye5 - mistral - 7b - instructSadness Negative Topic 1Topic 2Topic 3NeutralPositiveJoyLoveAngerFearSurpriseClassify sentiment of the github comment into positive, neutral, or negativeClassify the emotion expressed in the given github comment into one of the six emotions: anger, fear, joy, love, sadness, and surpriseIdentify the topic or theme of the given github commentClassi er Classi er ClusterGitHubCommentsTopicembeddingsSentiment embeddingsEmotionembeddingsFigure 4.2: Overview of sentiment, emotion, and topic assignment to tecomments of our dataset. The task-specific instruction is concatenated withthe GitHub comment text and fed into the E5instruct model. This processgenerates embeddings that encapsulate the semantic meaning of the com-ments, which are then used for downstream classification (sentiment andemotion) and clustering (topic modeling) tasks, ensuring high performanceand accurate representation of the data.4.2.2 Emotion ExtractionFor emotion analysis, we use the same technique as for sentiment analysis.The Mistral model is used, as at the time of conducting the experiments, ithas the best performance for classification and clustering tasks. We promptthe model the same way for sentiment analysis, but the prompt is as follows.Instruction: Classify the emotion expressed in the given github com-414.2. RQ1 Methodologyment into one of the six emotions: anger, fear, joy, love, sadness, andsurpriseAfter this step, the embeddings of the data are generated. These em-beddings are specific to the emotion task.Similar to our sentiment classification process, we train a logistic re-gression classifier to assign one of six predefined emotion labels to eachembedded comment. The classifier is trained on 16,000 examples and vali-dated on 2,000 test examples from the emotion extraction dataset [99] usedin MTEB. Its performance on the test set is summarized as follows: sad-ness (precision 0.66, recall 0.75, F1-score 0.70), joy (precision 0.68, recall0.78, F1-score 0.73), love (precision 0.48, recall 0.23, F1-score 0.31), anger(precision 0.73, recall 0.56, F1-score 0.64), fear (precision 0.61, recall 0.59,F1-score 0.60), and surprise (precision 0.62, recall 0.35, F1-score 0.45). Theoverall accuracy of the classifier on the test set is 0.66 (the percentage oftotal test examples correctly classified across all categories). We apply theclassifier to the embeddings to assign one of six predefined emotion labelsto each embedded comment.After this step, each record in our data has an Emotion Label assignedto it.4.2.3 Analysis of Sentiments and Emotions Across RolesIn this section, we explore the potential differences in sentiment and emotionexpression between mentors and mentees in GSoC. We first categorize thecomments in the OpenMent dataset by the role of their author (mentoror mentee). This role-based categorization serves as a fundamental step incomparing the emotion and sentiment expressions of the two groups.For our sentiment analysis, we utilize the Sentiment Labels (negative,neutral, positive) assigned to comments. The distribution of these labels isquantified through the Contextual Distribution (CD) metric (Equation 4.1).We analyze the percentages of each sentiment label and also calculate andcompare the mean sentiment scores for comments from each role.The same CD metric is applied to assess the emotion content of the com-ments, capturing the Emotion Labels distributions among the comments ofeach role. This analysis reveals the proportion of comments that exhibitspecific emotions, thus highlighting emotional discrepancies between men-tors and mentees.424.2. RQ1 MethodologyStatistical SignificanceHaving sentiment scores as a numerical representation of sentiments, toassess if the observed differences between mentors and mentees are statis-tically significant, we employ the Independent Samples t Test. This test iscommonly used in software engineering studies to evaluate differences in nu-merical data across different groups, as evidenced by Upadhya et al. [127],who utilized it to compare the programming languages used by GitHubusers among users with different numbers of followers. Using this method,we can directly test the hypothesis that mentors and mentees may expresssignificantly different levels of sentiment in their communications.For the Emotion Labels, which are categorical data, we employ the chi-square test of independence. This test is well-regarded for evaluating theassociation between categorical variables and has been applied in numer-ous studies, such as the work by Codabux et al. [23], who have used thechi-square test to investigate the association between different user roles (au-thors, editors, and reviewers) and the types of technical debt identified inthe peer-review documentation of R packages. We investigate whether thereis a significant association between two categorical variables, Role (mentoror mentee) and Emotion Label.By applying these tests, our objective is to verify whether the roles withinthe GSoC program influence the emotion and sentiment expressions in com-munication, providing quantitative support for answering our research ques-tion. The details of each of the statistical tests can be found in Section 4.5.4.2.4 Analysis of Sentiments and Emotions Over TimeWe aim to identify patterns and significant shifts in sentiment and emotionacross different program phases and weekly timelines, analyzing how thesentiments and emotions fluctuate over time.Using the sentiment profile metric 4.2) introduced in Section 4.2.5, wetrack the average sentiment scores across the 20-week program timeline.This metric aggregates sentiment scores within each week, allowing us tovisualize how sentiments evolve throughout the GSoC program timeline.For emotions, the emotion impact metric (Equation 4.3), detailed inSection 4.2.5, quantifies the prevalence of each specific emotion during dif-ferent timeframes in the program (e.g. different weeks). For each of thesix emotions classified, we plot the proportion of each emotion over time.This visualisation allows us to observe if and when certain emotions peakduring the GSoC in each role. For instance, we might find that “joy” peaks434.2. RQ1 Methodologyduring early weeks due to initial excitement, while “anger” might increaseas deadlines approach.Statistical SignificanceTo assess the significance of observed temporal patterns in sentiments andemotions, we employ statistical tests.Similar to our methodology for role comparison, we apply the chi-squaretest of independence to the Emotion Labels, but this time instead of roles,we compare different weeks or periods in GSoC. This test determines if theimpact of specific emotions are associated with particular a timeframe.We can not use an independent samples t-test to test the significanceof differences between multiple weeks. We apply the Analysis of Variance(ANOVA) to determine if there are statistically significant differences in sen-timent scores across various weeks (or periods) during the GSoC program.This test generalizes the t-test beyond two means. Although ANOVA in-forms us if at least one week is significantly different from the others, it doesnot specify which specific weeks significantly differ from each other.To further investigate the exact weeks where significant shifts in sen-timent occur, we employ t-tests to compare the mean of sentiment scoresbetween consecutive week pairs. By doing so, we aim to pinpoint specificweeks where substantial changes in sentiment are evident, suggesting poten-tial points in time within the GSoC timeline that may influence participantsentiments.These statistical methods identify if and when significant shifts occur,aiding in answering our first research question regarding the emotion andsentiment differences over time. The details of these tests are elaborated inSection 4.5 of this thesis.4.2.5 MetricsContextual Distribution (CD)This metric examines how specific attributes, such as sentiments or emo-tions, are distributed within particular segments of our dataset. Consideran analysis aimed at understanding the distribution of emotions expressedby mentees. Specifically, we want to know the proportion of comments thatexpress each emotion among all comments made by mentees. To achieve this,we employ a two-tiered filtering approach. Initially, we filter the dataset toselect all comments made by mentees, establishing the base subset for anal-ysis. Then, from this base subset, we further filter to identify comments444.2. RQ1 Methodologyexpressing a particular emotion. By dividing the number of comments ex-pressing this emotion by the total number of comments made by mentees, weobtain the proportion of comments expressing that emotion among mentees.This process is repeated for each emotion to derive the distribution of dif-ferent emotions among comments made by mentees. If we do the same formentors, we can compare the distribution of emotions between mentors andmentees.The formula for Contextual Distribution (CD) is applied as follows:CD(F,G) =|D(F ) ∩D(G)||D(F )| × 100, (4.1)where:• D(F ) is the set of all data points satisfying the base criteria F , whichdefines the foundational subset of the dataset for analysis.• D(G) is the set of all data points satisfying the focus criteria G, whichrefines the specific attributes of interest such as particular sentimentsor emotions.• |D(F )| is the cardinality of the set D(F ), indicating the total numberof data points that satisfy the base criteria F .• D(F ) ∩D(G) represents the intersection of sets D(F ) and D(G), i.e.,the data points that satisfy both criteria F and G. The cardinality ofthis intersection indicates the number of data points that satisfy bothbase and focus criteria.• Example: CD({Mentee}, {Joy}) determines the percentage of com-ments expressing joy among all comments made by mentees.Sentiment ProfileThe sentiment profile metric is designed to aggregate sentiment scores acrossspecified time intervals or milestones, providing a measure of sentimenttrends throughout the program. This metric examines how sentimentsevolve during the program, potentially highlighting points in time whenthe sentiment of comments change significantly.The sentiment profile of timeframe t is defined as the average sentimentscore of comments made within that specific timeframe:454.3. RQ2 MethodologySentiment Profile(t) =1|D(t)|∑d∈D(t)S(d), (4.2)where:• D(t) represents the set of all comments within a specific timeframe t.• |D(t)| is the total number of comments in D(t).• S(d) is the sentiment score of comment d.Emotion ImpactWe adapt the impact metric from Barua et al. [8] to our specific context;this metric quantifies the proportion of each specific emotion within thecommunications over defined timeframes or during particular project phases,illustrating how emotion responses vary at different times.The impact of emotion e in a timeframe t is defined as:impact(e, t) =1|D(t)|∑d∈D(t)I(E(d) = e), (4.3)where:• D(t) is the set of all comments made during the timeframe t.• |D(t)| is the total number of comments in D(t).• E(d) is the emotion label of comment d.• I(E(d) = e) is an indicator function that equals 1 if the emotion ofcomment d is e and 0 otherwise.4.3 RQ2 MethodologyTo answer RQ2, we use the issue and pull request comments from the Open-Ment dataset. We begin by extracting the topics of the communications.Next, we perform a temporal analysis to identify the peak focus periods foreach topic. This allows us to determine if the emphasis on specific topicsare aligned with some key milestones of the GSoC program. Additionally,we will compare the topic trends between mentees and mentors to assesswhether the patterns align or differ for each topic.464.3. RQ2 Methodology4.3.1 Topic ModelingTopic modeling is an information retrieval technique that identifies abstract“topics” within a collection of documents by clustering similar documentsand providing representations that explain the topic of each cluster withoutthe need for training data [49]. This approach enables us to recognize com-munication topics in GSoC. Our methodology is inspired by BERTopic [50].To extract topics, BERTopic first leverages language models to generate textembeddings for each data entry in the dataset. These embeddings are thenclustered, to group semantically similar comments together. BERTopic fur-ther analyzes these clusters to extract the most representative words of eachcluster. These top representative words in each cluster define its core topic.Following is the steps we took and the adaptations we made for our study.Document Embeddings. To generate embeddings specific to topicmodeling from the comments in our dataset, we employ the same tech-nique used for sentiment and emotion analysis. We provide task-specificinstructions combined with the comments to the Mistral model, which thengenerates embeddings suitable for topic modeling. The instruction used tocreate these embeddings is is described below.Instruction: Identify the topic or theme of the given GitHub comment.Unlike BERTopic, which reduces the dimensionality of embeddings, ourmethod preserves the original high-dimensional space. This approach issupported by MTEB [79], which demonstrated great clustering performancein high-dimensional spaces for embeddings created using the Mistral model.After this step, each comment in our dataset has an embedding representingthe semantics related to the topic it potentially encompasses.Clustering To cluster the embeddings, we use the k-means algorithm [42],deviating from the HDBSCAN [76] approach used in BERTopic. This choiceis motivated by the consistent performance of k-means with high-dimensionaldata, as evidenced by MTEB benchmarks. The optimal number of clusterswas determined using the Elbow method [121]. In our context, this involvesplotting the sum of square distances from each point to its assigned cen-ter against the number of clusters and identifying the “elbow” point wherefurther improvements become marginal. This visual heuristic helped us toselect a k-value of 36. After this step, each comment in our dataset is as-signed to one of 36 clusters, with each cluster representing a potential topic.Topic Representation To identify the topic represented by each clus-ter, we focus on determining the top words present in each cluster that bestdescribe the cluster. We begin this process by employing the CountVector-izer from the Scikit-learn library [88], which converts textual content into a474.3. RQ2 Methodologynumerical matrix of token counts. We adjust the CountVectorizer settingsto extract both unigrams (single tokens) and bigrams (two-token combina-tions). For example, from the sentence “good quality code” it would extractunigrams [“good”, “quality”, “code”] and bigrams [“good quality”, “qualitycode”]. This decision follows the approach of [8] and is supported by [116],who show that using bigrams can enhance the quality of text analysis.Next, we apply the TF-IDF (Term Frequency-Inverse Document Fre-quency) transformation, which measures the relative importance of a termwithin a document against its frequency across all documents. This transfor-mation downplays common words across the corpus while highlighting thoseunique to specific documents. For instance, if the word “code” appearscommonly across many documents, its significance is scaled down unlessit is particularly prevalent in a specific document. We treat all commentsassigned to a single cluster as one document during the TF-IDF transforma-tion to identify terms that are uniquely significant to each cluster. Followingthis step, each of the 36 clusters generates a list of top terms, providing asemantic signature that aids in the manual naming of the topics.Topic Naming and Merging Each of the 36 clusters is manuallynamed based on its top terms to reflect the underlying discussion themewithin that cluster. Following the approach of Barua et al. [8], after nam-ing the clusters, we merge similar topics to into themes, which are broadercategories covering multiple related subjects. This is to reduce the numberof distinct topic classes. In our study, these new merged topics are referredto as “themes” to distinguish them from the initial 36 topics. From theoriginal 36 named topics, we merged similar ones based on their semanticrelatedness, resulting in 11 broader themes that more effectively encapsulatethe primary discussion points in the dataset. The decision to merge into 11themes was guided by the sets of topics representing similar subjects, deter-mined after naming the initial 36 extracted topics.These themes are then used for further analysis, as mentioned below.4.3.2 Topic AnalysisThe topic analysis involves examining the communication themes extractedfrom the GSoC mentor-mentee interactions to understand their prevalence,distribution, and evolution over time. The goal of this analysis is to un-cover patterns in the interactions, identify focal points of discussions, andcomprehend how these discussions change in response to various weeks andmilestones within the GSoC program.484.3. RQ2 MethodologyDistribution of Topics. After identifying the topics, we analyzedtheir distribution across the comments in our dataset. The distribution wascalculated using the theme’s share metric (Equation 4.4), which representsthe proportion of comments within each theme relative to the total numberof comments. Additionally, the role-based distribution provides insights intowhich themes are predominantly discussed by mentors or mentees.Temporal Analysis of Themes. To understand how the focus ondifferent themes evolves throughout the GSoC program, we conducted atemporal analysis. We examined the weekly impact of each theme using ourimpact (Equation 4.5) and normalized impact (NI) (Equation 4.6) metrics.The theme impact metric measures the prominence of the themes in differ-ent timeframes, while the NI metric (Equation 4.6) provides a normalizedimpact measure over the entire length of the program, enabling comparativeanalysis across time. We also explored the role-based dynamics by compar-ing the weekly Normalized Impact (NI) of the themes between mentors andmentees. The normalized nature of the NI metric ensures that the com-parison focuses on the relative change in impact rather than the absolutevolume of contributions, highlighting how the emphasis on various themesshifts over time and differs between roles.4.3.3 MetricsShareWe adapt the share metric from Barua et al. [8] to our specific context; theshare of a topic or theme z is defined as:share(z) =1|D|∑d∈DI(Z(d) = z)× 100 (4.4)where:• D is the set of all comments in the dataset.• Z(d) represents the topic/theme assigned to comment d.• I(Z(d) = z) is an indicator function that equals 1 if comment d isassigned to topic/theme z, and 0 otherwise.Topic impactWe adapt the impact metric from Barua et al. [8]; the impact of topic (ortheme) z in week w is defined as:494.4. RQ3 Methodologyimpact(z, w) =1|D(w)|∑d∈D(w)I(Z(d) = z) (4.5)Normalized Topic Impact (NI)By normalizing the topic impact using Min-Max normalization, we obtainthe Normalized Impact (NI) of topic (or theme) z in week w as:NI(z, w) =impact(z, w)−min(impact(z))max(impact(z))−min(impact(z)) (4.6)where:• impact(z, w) is the impact of topic (or theme) z in week w.• min(impact(z)) is the minimum impact of topic (or theme) z acrossall weeks.• max(impact(z)) is the maximum impact of topic (or theme) z acrossall weeks.4.4 RQ3 MethodologyRQ3 aims to uncover differences linked to successful GSoC projects. Byidentifying significant differences in sentiments and emotions expressed inthe communications, themes of discussion, and interaction and commit fre-quencies, we seek to provide actionable insights that could help optimize theGSoC program to better achieve its objectives of promoting lasting engage-ment in OSS development.In RQ3, we investigate the effects of participating in the GSoC programon the contributions of mentees to OSS. The primary goal is to understand ifthe GSoC experience influences mentees’ subsequent engagement with OSSdevelopment. This is to examine the effectiveness of GSoC in promotinga sustained interest in OSS among mentees. For this analysis, we leveragethe issue and pull request comments from our OpenMent dataset. Unlike inRQ1 and RQ2, we also incorporates the commit data, examining the numberof commits made by each role per project.504.4. RQ3 Methodology4.4.1 Analysis ProcedureWe classify projects into two categories, “improved” and “unimproved”,based on a predefined success metric (Equation 4.7) that quantifies changesin the GitHub contributions of mentees pre- and post-GSoC. The metneesin “improved” projects, see an improvement in their contribution to GitHubafter the GSoC program ends, compared to their pre program contributions;and the mentees in “unimproved” projects, either see no improvement orreduced contributions to GitHub post-program compared to their level ofcontribution pre program. The details of the success metric (Equation 4.7),and how the projects are categorized into “improved” and “unimproved”are explained in Section 4.4.2. This classification allows us to:Comparison of Sentiments and Emotions. We analyze the sen-timents and emotions expressed in the communications of mentors andmentees across different project categories. This helps determine if emo-tion and sentiment expression correlates with project outcomes.Number of Comments and Commits. By evaluating the number ofcomments and commits made by each role per project, we aim to identifywhether more or less frequent communication or commits correlate withproject success.Comparison of Themes of Discussion. We explore the topics ofdiscussion in the communications to see if certain themes are more prevalentor notably different between the improved and unimproved project groups.This aspect examines whether the focus of discussion impacts the success ofthe projects in GSoC.Statistical Significance. To validate the observed differences betweenthe “improved” and “unimproved” projects, similar to RQ1, we employ in-dependent samples t-tests and chi-square test of independence. T-tests areused to assess the significance of differences in numerical variables, i.e. sen-timent scores and counts of comments and commits per project. Chi-squaretests are applied to categorical data, i.e. sentiment and emotion categories.4.4.2 MetricsProject successGSoC aims to promote a long-term interest in OSS development amongparticipants. One of its primary goals is to encourage mentees to continuecontributing to OSS after their participation ends, as stated by Google on514.4. RQ3 Methodologytheir Summer of Code webpage13 and FAQ section14. To evaluate the suc-cess of this objective, we developed a metric that quantifies each mentee’simprovement in engaging with OSS before and after the GSoC program.Data Collection. We measure the mentees’ contributions by countingtheir activities on GitHub in the six months preceding and following theirGSoC participation. This period was chosen to maximize the observationwindow while ensuring that the data collected does not overlap with otherGSoC cycles. Contributions are retrieved via GitHub’s GraphQL API15with the following query structure:{user(login: $mentee_handle) {contributionsCollection(from: $start, to: $end) {contributionCalendar {totalContributions}}}}This query captures the total number of contributions made by a mentee($mentee handle) within a specified period ($start to $end).Metric Formulation. The “success” metric for each GSoC project (p)is defined as the relative change in the total number of GitHub contributionsby the project’s mentee, comparing the six months before and after theprogram. It is calculated as:success(p) =PostTC(p)−PreTC(p)PreTC(p) , if PostTC(p) > PreTC(p) > 0;PostTC(p)−PreTC(p)PostTC(p) , if PreTC(p) > PostTC(p) > 0;undefined, otherwise,(4.7)where:• PreTC(p) represents the total number of contributions made by thementee of project p in the six months before the GSoC start.• PostTC(p) denotes the total contributions in the six months after theprogram’s conclusion.13https://buildyourfuture.withgoogle.com/programs/summer-of-code14https://developers.google.com/open-source/gsoc/faq#what_is_the_goal_of_google_summer_of_code15https://docs.github.com/en/graphql524.4. RQ3 Methodology• This metric quantifies the change in mentee’s engagement in OSS de-velopment.• This metric is not directly reflecting the quantity of contributions, itmeasures how much the contribution has changed post-program com-pared to pre-program.Projects with mentees who did not make any GitHub contributions ei-ther before or after the GSoC are marked as having an “undefined” successmetric. These cases are excluded from the analysis to avoid issues withdivision by zero.This structured approach to defining and calculating the “success” met-ric provides a clear, quantifiable measure of improvement in OSS engagementof its participants, directly aligning with the program’s key objective.We then categorize each GSoC project into one of the “improved” or“unimproved” categories. For each GSoC project p, status(p) indicateswhether a project is considered “improved” or “unimproved”:For any p ∈ P :status(p) =improved, if success(p) > 0;unimproved, if success(p) ≤ 0;undefined, if success(p) = undefined,(4.8)where:• P represents the set of all the GSoC projects in OpenMent.By measuring the relative change in contributions, we aim to objectivelyassess whether the mentee’s have increased their engagement in OSS de-velopment. The “improved” and “unimproved” project categories, allowscomparative analysis and comparison of factors that may contribute to suc-cessful projects.Per Project Count of Comments (PPCcomment)The (PPCcomment) is defined as the total number of comments made by roler per project over the duration of the GSoC program. This metric providesthe total communication activity of each role within each project.PPCcomment(r, p) = |Dr(p)|, (4.9)where:534.4. RQ3 Methodology• p is a specific GSoC project in OpenMent.• r can be mentee or mentor.• Dr(p) is the set of all comments by role r and associated with projectp.Per Project Count of Commits (PPCcommit)The (PPCcommit) is defined as the total number of commits made by role rper project over the duration of the GSoC program. This metric providesthe total development activity of each role within each project.PPCcommit(r, p) = |Cr(p)|, (4.10)where:• p is a specific GSoC project in OpenMent.• r can be mentee or mentor.• Cr(p) is the set of all commits by role r and associated with project p.Mean Per Project Count of Comments (MPPCcomment)The (MPPCcomment) is defined as the average number of comments madeby role r per project over the duration of the GSoC program. This metricprovides the average of per project communication activity of each role.MPPCcomment(r) =1|P |∑p∈P|Dr(p)|, (4.11)where:• P is the set of all GSoC projects in OpenMent.• r can be mentee or mentor.• Dr(p) is the set of all comments by role r and associated with projectp.544.5. Statistical MethodsMean Per Project Count of Commits (MPPCcommit)The (MPPCcommit) is defined as the average number of commits made byrole r per project over the duration of the GSoC program. This metricprovides the average of per project development activity of each role.MPPCcommit(r) =1|P |∑p∈P|Cr(p)|, (4.12)where:• P is the set of all GSoC projects in OpenMent.• r can be mentee or mentor.• Cr(p) is the set of all commits by role r and associated with project p.4.5 Statistical MethodsThis section outlines the statistical methods employed to analyze the resultsto answer the research questions. We use these tests to determine meaningfuldifferences between groups. The comparisons are made on either categoricalor numerical variables. Categorical data in our study include sentimentlabels (positive, negative, neutral), emotion labels (joy, anger, fear, etc), andidentified topics and themes of discussion, derived from text classification.Numerical variables in our study include sentiment scores (-1, 0, and 1),comment counts and commit counts.Parametric and non-parametric statistical tests are two broad categoriesof methods used to analyze data based on different assumptions about thedata’s distribution [44]. Parametric tests assume that the data follows aknown distribution or has specific characteristics (limit on the number ofobservations, etc.). These tests provide precise results if the assumptionshold true. From these tests, we use the independent samples t-test, chi-square test of independence, and ANOVA. Non-parametric tests, on the otherhand, do not require any assumptions about the distribution of the data.These tests are more flexible and can be used on ordinal data or when theassumptions of parametric tests are not met. We use bootstrap resampling,which is a non-parametric test as a backup for our t-test. We describe eachof these tests below.554.5. Statistical MethodsIndependent samples t-testWe used the independent samples t-test (ttest ind from SciPy [130]) to com-pare the means of two groups and determine if they are significantly differentfrom each other. The steps involved in performing this test are [72]:1. Formulating hypotheses:• Null hypothesis (H0): There is no significant difference betweenthe means of the two groups.• Alternative hypothesis (H1): There is a significant difference be-tween the means of the two groups.2. Calculating the test statistic: The t-statistic is calculated basedon the means, standard deviations, and sample sizes of the two groups.3. Determining the p-value: The p-value indicates the probability ofobserving a t-statistic as extreme as, or more extreme than, the onecalculated from the data, assuming the null hypothesis is true. Weconsider a p-value of less than 0.05 as statistically significant.We use this test to analyze the differences in sentiment scores in RQ1,and RQ3, and to analyze the differences in per project counts of commentsand commits in RQ3.ANOVAAnalysis of Variance (ANOVA) (more specifically, one-way ANOVA) is usedwhen comparing the means of three or more groups [44]. We applied ANOVA(f oneway from SciPy [130]) to assess differences in sentiment scores acrossmultiple project milestones and different periods of the GSoC program inRQ1.The steps involved in performing ANOVA are:1. Formulating hypotheses:• Null hypothesis (H0): There is no significant difference betweenthe means of the groups.• Alternative hypothesis (H1): At least one group mean is signifi-cantly different from the others.564.5. Statistical Methods2. Calculating the F-statistic: The F-statistic is calculated by com-paring the variance between the group means to the variance withinthe groups.3. Determining the p-value:The p-value in an ANOVA test indicatesthe probability of observing an F-statistic as extreme as, or more ex-treme than, the one calculated from the data, under the assumptionthat the null hypothesis is true. This test assesses whether any of thegroup means are statistically significantly different from each other.A p-value less than 0.05 is considered statistically significant in ourstudy.Chi-square Test of IndependenceThe chi-square test of independence is used to determine if there is a signifi-cant association between two categorical variables [44]. To answer RQ1, Weused this test (chi2 contingency from SciPy [130]) to examine the relation-ship between categorical communication features (e.g., emotion labels) androles (mentor vs. mentee), as well as the relationship between categoricalcommunication features and temporal aspects (e.g., week of the program,main periods of the GSoC program). In RQ3, we use this test to exam-ine the relationship between status 4.8 of GSoC projects and categoricalcommunication features (i.e. emotion labels, and themes)The steps involved in performing the chi-square test of independence are:1. Formulating hypotheses:• Null hypothesis (H0): There is no association between the twocategorical variables.• Alternative hypothesis (H1): There is an association between thetwo categorical variables.2. Calculating the chi-square statistic: The chi-square statistic iscalculated based on the observed and expected frequencies of the cat-egorical variables.3. Determining the p-value: The p-value indicates the probability ofobserving a chi-square statistic as extreme as, or more extreme than,the one calculated from the data, assuming the null hypothesis is true.We consider a p-value of less than 0.05 as statistically significant.574.5. Statistical MethodsBootstrap ResamplingBootstrapping is a statistical method used to estimate the distribution of astatistic by resampling with replacement from the original data. This tech-nique was introduced by Efron [34] in 1979 and is particularly useful whenthe underlying distribution of the data is unknown or when traditional para-metric assumptions are difficult to justify. By generating multiple resampleddatasets, bootstrapping allows for the estimation of the sampling distributionof almost any statistic and can provide measures such as confidence inter-vals, standard errors, and p-values. In chapter four of their book, Davisonand Hinkley [25] emphasize that in many practical situations, it is benefi-cial to employ nonparametric bootstrap resampling as a backup to paramet-ric methods because nonparametric tests, in general, compute significancewithout assuming specific forms for the data distributions. This approachis especially useful when the assumptions required by parametric tests, suchas normality, are not met by the data.Given that our data do not always meet the normality assumption re-quired by the independent samples t-test, we have opted to use bootstrap re-sampling methods as suggested by Tibshirani and Efron [122] as a backup forour results. Bootstrap resampling provides a robust alternative to validatethe results of our t-tests, ensuring the reliability of our tests of independence.Bootstrap resampling was employed for two primary purposes:Validation of Statistical Significance. The first objective of applyingbootstrap resampling was to validate the results of t-tests. Specifically, weaimed to test the null hypothesis that the means of two different groups(Group A and Group B) are equal. Whenever the parametric two-samplet-test 4.5 suggested a significant difference, bootstrap resampling was usedto ensure the validity of the observed statistical significance. We use thistest to validate our t-test results in RQ1 when comparing sentiment scoresbetween mentors and mentees. Following Davison and Hinkley [25], thisprocess involved:1. Pooling the Data: The data from both groups (Group A and GroupB) were combined into a single dataset.2. Resampling with Replacement: 10,000 bootstrap samples weregenerated by randomly resampling with replacement from the pooleddataset. Each bootstrap sample was the same size as the originalcombined dataset.584.5. Statistical Methods3. Splitting the Bootstrap Samples: Each of the 10,000 bootstrapsamples was split into two groups, maintaining the original samplesizes of Group A and Group B.4. Calculating the Test Statistic: The test statistic (e.g., differencein means) was calculated for each of the 10,000 bootstrap samples.5. Constructing the Distribution: The distribution of the test statis-tic across all 10,000 bootstrap samples was constructed to approximatethe sampling distribution under the null hypothesis.6. Determining the p-value: The p-value was calculated based on theproportion of the 10,000 bootstrap samples where the test statistic wasas extreme as or more extreme than the observed test statistic fromthe original data.Visualization of the sampling distribution of means The second ob-jective was to provide a robust visualization of the magnitude of differencesbetween the groups. Bootstrap resampling was used to generate distribu-tions of a particular statistic (e.g., means of sentiment scores ) for Group Aand Group B separately. This process involved:1. Generating Bootstrap Samples: For each group (Group A andGroup B), 10,000 bootstrap samples were generated by randomly re-sampling with replacement from the group data.2. Calculating the Statistic: The statistic of interest (e.g., means ofsentiment scores ) was calculated for each of the 10,000 bootstrapsamples.3. Visualizing the Distributions: The distribution of the statistic wasplotted in a single graph, with each group’s histogram displayed in adistinct color. This visualization allows for an intuitive understandingof the differences between the groups, the magnitude of the differ-ence (as indicated by the distance between the two histograms), andthe significance of this difference (as shown by the amount of overlapbetween the two histograms).The resulting plot provides insight into the magnitude of the differ-ence between the groups, as indicated by the distance between the twohistograms, and the significance of this difference, as shown by the amountof overlap between the two histograms. When two histograms have overlap,594.5. Statistical Methodsit means that the ranges of data they represent share some common val-ues. Thus, more overlap between the distributions indicates less differencein the distribution of the bootstrap resampled means, and a smaller practi-cal difference between the groups, conversely, less overlap implies a greaterpractical difference, reinforcing the significance of the observed differences.We use this visualization in RQ1, and RQ3 to show the distribution of thebootstrap resampled means of sentiment scores and the distribution of thebootstrap resampled means of per project counts of comments and commitsin RQ3.60Chapter 5ResultsIn this chapter, we present the findings from the analysis of mentor-menteeinteractions in the GSoC program. The study focuses on understandingthese interactions by examining sentiments, emotions (RQ1), and topics(RQ2) in their communications. The results are structured around thethree main research questions of the study. Sections 5.1, 5.2, and 5.3 ad-dress RQ1, which investigates differences in sentiment and emotion betweenmentors and mentees. Sentiment analysis was performed over time and atkey project milestones, comparing the proportion of messages expressingvarious emotions for both roles. Additionally, the sentiment and emotionsexpressed in comments of individuals who transitioned from mentees to men-tors were analyzed. Section 5.4 explores RQ2, which examines the mainthemes of discussion between mentors and mentees, and how these themesevolve throughout the GSoC program. Section 5.5 covers RQ3, which an-alyzes changes in open source contributions (i.e., commits, pull requests,project involvements) by mentees before and after GSoC participation.5.1 Sentiment Analysis (RQ1 Results)We conducted a comprehensive sentiment analysis of the communicationsbetween mentors and mentees. We conducted this analysis on the issueand pull request comments from the OpenMent dataset, necessitating theexclusion of projects with zero comments. Table 5.1 is an overview of thedata used for this analysis.615.1. Sentiment Analysis (RQ1 Results)Statistic ValueNumber of comments 203,694Number of mentees’ comments 98,165 (48%)Number of mentors’ comments 105,529 (52%)Number of projects in OpenMent 2,272Projects with at least one comment 1,638Minimum number of comments in a project 1Maximum number of comments in a project 2,406Average number of comments per project 124Table 5.1: Overview of the data used for analysis of RQ1.5.1.1 Distribution of SentimentsEach comment in our dataset is assigned a sentiment score as defined inSection 4.2.1, with values of -1 indicating Negative, 0 indicating Neutral,and 1 indicating Positive sentiment. We delve into the sentiments by ex-amining their distribution among different roles. Utilizing our CD metric(Equation 4.1), we have calculated and presented the sentiment distributionfor both mentors and mentees, as illustrated in Table 5.2. The analysisreveals that the majority of comments, approximately 85%, are neutral insentiment. Mentors tend to have about 2% fewer positive comments and aslightly higher percentage of negative comments compared to mentees.RoleSentiment Distribution (%)Negative Neutral PositiveMentee 3.6 84 12.4Mentor 3.8 86 10.1Table 5.2: CD({role}, {sentiment}): Distribution of sentiments among dif-ferent roles.We computed the mean, standard deviation, and variance of these senti-ment scores. The results are summarized in Table 5.3. The mean sentimentscore is slightly higher for mentees (0.089) compared to mentors (0.063),625.1. Sentiment Analysis (RQ1 Results)which is expected based on our earlier observation of a more positive senti-ment among mentees’ comments. The magnitude of difference is quantifiedby the difference in mean sentiment scores between the two roles. The stan-dard deviation and variance also reflect the distribution of sentiment scores,with mentees showing slightly higher values than mentors. These statisticshighlight the greater variability and slightly more positive tendency in thesentiments expressed by mentees.StatisticRoleMentor MenteeMean 0.06 0.09Standard Deviation 0.37 0.39Variance 0.14 0.15Table 5.3: Mean, standard deviation, and variance of sentiment scores inmentees’ and mentors’ comments.5.1.2 Statistical Analysis of SentimentsBeyond examining the mean of sentiment scores, to determine whether thesedifferences are statistically significant, we conducted an independent samplest-test. The results of the t-test, as presented in Table 5.4, reveal a statisticallysignificant difference in sentiment scores between mentors and mentees witha p-value < 0.001. This extremely low p-value indicates that the likelihoodof the observed differences arising by chance is negligible, thereby confirminga real disparity in how mentors and mentees communicate within the GSoCprogram.Statistic ValueT-Statistic −15.1P-Value < 0.001Table 5.4: T-test results comparing sentiment scores between mentors andmentees.Bootstrap Resampling To validate the t-test results comparing the meansof sentiment scores between mentors and mentees, we employed bootstrap635.1. Sentiment Analysis (RQ1 Results)resampling. Figure 5.1 presents the distribution of the differences in meansentiment scores from 10, 000 bootstrap resamples of the pooled sentimentscores . The x-axis shows the difference in mean sentiment scores of thebootstrap resamples and the y-axis is the count of the samples. The ob-served difference in mean sentiment scores between mentors and menteesis 0.025, as indicated by the dashed vertical line. This line falls outsidethe bootstrap distribution, suggesting that the observed difference is veryunlikely to have occurred by random chance alone.Figure 5.2 illustrates the bootstrap distributions of mean sentiment scoresfor mentors and mentees, generated from 10, 000 resamples each. The clearseparation between the two distributions further highlights the significantdifference in mean sentiment scores between the two groups.-0.005 0.0 0.005 0.01 0.015 0.02 0.025Difference Between Means of Sentiment Scores0100200300400500600700800CountObserved Difference = 0.025Distribution of Resampled DifferencesFigure 5.1: Distribution of differences in sentiment means from 10,000 re-samples of the pooled sentiment scores.645.1. Sentiment Analysis (RQ1 Results)0.06 0.065 0.07 0.075 0.08 0.085 0.09 0.095Mean of Sentiment Scores02004006008001000CountMentorsMenteesFigure 5.2: Distribution of sentiment means (mentors vs. mentees).5.1.3 Weekly Analysis of SentimentsTo understand the dynamic nature of sentiments expressed by mentors andmentees over the course of the GSoC program, we conducted a weekly senti-ment analysis using the sentiment profile metric (Equation 4.2). This met-ric provides a clear picture of sentiment trends by averaging the sentimentscores of comments created in a specific time interval.Weekly TrendFirst, we visualize the sentiment profile for mentors and mentees over the20-week duration of the program. Figure 5.3 shows the weekly sentimentprofile, which helps us understand the overall trend and changes in sentimentover time. The x-axis shows the number of weeks of the GSoC program, andthe y-axis represents the sentiment scores . The yellow line shows the scoresof mentees and the blue line represents the mentors’ scores. The two dashedlines distinguish the GSoC milestones, at week 3, where the coding starts,and at week 9, the midterm evaluation deadline.655.1. Sentiment Analysis (RQ1 Results)0 3 5 7 9 11 13 15 17 19Week0.060.080.100.120.14Sentiment profileMentorMenteeCoding StartsMid-term EvaluationFigure 5.3: Sentiment profile (weekly mean sentiment score) of mentorsand mentees over weeks of the GSoC. Interpret with care: The Y-axis doesnot include the full range of possible values, which may visually exaggerate smalldifferences.From the plot, we observe the following trends:• Initial Weeks (0-3): Mentors start with relatively high sentimentprofile, which decline sharply, likely due to initial challenges and ad-justments. Mentees also start high but decline more gradually.• Weeks 4-8: When coding officially starts, mentors’ sentiment profiledrop more significantly than mentees’. Mentors reach their lowestpoint during this time. Mentees’ sentiment profile remains relativelystable during this period.• Mid-Term Evaluation (Week 9): There is a slight increase in men-tor sentiment profile possibly reacting to midterm evaluation deadline,while mentees’ sentiment remains stable.• Weeks 11-18: Both mentors’ and mentees’ sentiment profile gradu-ally increase, with noticeable spikes in Week 12 for mentees and Week15 for mentors.• End of Program (Week 19): During the final three weeks, senti-ment profile continues to rise in both roles, peaking towards the endof the project.665.1. Sentiment Analysis (RQ1 Results)Analysis of Variance (ANOVA)To statistically verify whether there are significant differences in the meanof sentiment scores over the program duration, we conducted an ANOVA(Analysis of Variance) test. This test evaluates if the sentiment profilesdiffer significantly across the 20 weeks. The ANOVA results are presentedin Table 5.5, revealing a significant difference in the sentiment profile ofdifferent weeks of the program in both mentors and mentees with a p-value< 0.001.Role F-statistic P-valueMentor 10.6 < 0.001Mentee 2.6 < 0.001Table 5.5: ANOVA results for weekly sentiment profiles.Pairwise weekly t-testsTo identify specific points in time in which the sentiment profiles changesignificantly, we conducted pairwise t-tests between consecutive weeks. Foreach week w in the range 0 to 18, we used an independent samples t-test to test the significance of difference between sentimentprofile(w) andsentimentprofile(w + 1). This helps pinpoint specific weeks where signifi-cant sentiment changes occur. The pairwise t-test results are presented inTable 5.6.675.1. Sentiment Analysis (RQ1 Results)WeeksT-statisticMentor Mentee0 vs 1 3.30*** 0.971 vs 2 −0.50 1.502 vs 3 3.26*** −0.313 vs 4 −0.27 −0.194 vs 5 2.25* 0.595 vs 6 0.72 0.156 vs 7 0.03 0.347 vs 8 1.33 0.998 vs 9 −2.69** 0.189 vs 10 −0.60 −0.0310 vs 11 0.97 0.4311 vs 12 1.48 −1.6212 vs 13 0.61 1.9013 vs 14 −0.67 −0.3814 vs 15 −2.38* −0.8515 vs 16 2.20* 0.2916 vs 17 0.05 0.4617 vs 18 −1.06 −0.1718 vs 19 −1.47 −2.14*Table 5.6: Pairwise t-test results for consecutive weeks. Significance levels:* p < 0.05, ** p < 0.01, *** p < 0.001.Key findings from the pairwise t-tests include:• Week 0 vs 1: A significant decrease in mentor sentiment (p < 0.001),likely reflecting initial challenges and adjustments.• Week 2 vs 3: Another significant decrease for mentors (p = 0.0011),indicating continued initial challenges.• Week 4 vs 5: A significant decrease for mentors (p = 0.025), possiblydue to the onset of the main coding phase and associated pressures.685.1. Sentiment Analysis (RQ1 Results)• Week 8 vs 9: A significant increase for mentors (p = 0.007), possiblyrelated to the midterm evaluation.• Week 14 vs 15: A significant increase for mentors (p = 0.018).• Week 15 vs 16: A significant decrease for mentors (p = 0.028).• Week 18 vs 19: A significant increase in mentee sentiment (p =0.032), possibly due to end-of-program excitement and project com-pletion.By integrating the sentiment profile metric (Equation 4.2) and structur-ing the analysis around significant points of change, this section providesa clear and comprehensive understanding of the weekly sentiment trendsamong mentors and mentees. The use of ANOVA and pairwise t-tests helpsidentify and explain key changes in sentiment over the program duration.5.1.4 Milestone-based Analysis of SentimentThe milestone-based analysis examines sentiment profiles (Equation 4.2)across three key periods of the GSoC program: the Community BondingPeriod (weeks 0-3), the First Work Period (weeks 4-9), and the SecondWork Period (weeks 10-19). The goal is to determine if these milestonessignificantly affect the sentiments of mentors and mentees.Community Bonding Period (Weeks 0-3) During the CommunityBonding Period, participants get acquainted with the community, set uptheir development environments, and plan their projects. We calculated themean sentiment profiles for this period and compared them with subsequentperiods to understand the initial sentiment trends.First Work Period (Weeks 4-9) The First Work Period marks theofficial start of coding. Participants begin implementing their project plans.This period is crucial as it includes the ramp-up phase where initial projectchallenges are encountered. We analyzed the sentiment profiles to see howthe start of actual coding affects the sentiment.Second Work Period (Weeks 10-19) The Second Work Period extendsfrom the midterm evaluation to the end of the program. This period includescritical project milestones such as midterm evaluations, final coding, test-ing, and project completion. We examined the sentiment profiles during695.1. Sentiment Analysis (RQ1 Results)this extended period to observe how ongoing development and evaluationsimpact sentiment.The results of the ANOVA tests for these periods are shown in Table 5.7.These results suggest significant differences in sentiment profiles across themajor project periods with p-values < 0.001 for both mentors and mentees.Role F-statistic P-valueMentor 14.8 < 0.001Mentee 8.7 < 0.001Table 5.7: ANOVA results for sentiment profiles across GSoC’s majorproject periods.The sentiment profiles for each period are summarized in Table 5.8.These means highlight how sentiment changes as participants move throughdifferent phases of the program.Role CommunityBonding PeriodFirst WorkPeriodSecond WorkPeriodMentor 0.07 0.08 0.04Mentee 0.09 0.09 0.07Table 5.8: Sentiment profiles across GSoC’s major periods.From the data, we observe the following trends:• Community Bonding Period: Both mentors and mentees exhibitrelatively high sentiment profiles (mean sentiment score ), reflectinginitial enthusiasm and positive engagement.• First Work Period: Sentiment profiles for both groups slightly in-crease as the coding phase begins, indicating a positive start despitepotential initial challenges.• Second Work Period: There is a noticeable decrease in sentimentprofiles for both mentors and mentees during this period, potentiallydue to the increased workload and pressure of nearing project comple-tion.705.2. Emotion Analysis (RQ1 Results)Summary- Approximately 85% of comments were neutral and mentors havea more positive sentiment score compared to mentees (0.09 vs 0.06approximately).- The weekly sentiment profiles of both roles follow a trend that re-sembles an exponential decay curve, sharply decreasing from weekzero to week 3, followed by a relative stability with minor changes.This pattern likely reflects the initial excitement of participants atthe start of the program, followed by a more realistic engagementas they encounter the actual workload of their projects when codingstarts. Irrespective of the size of the observed differences, they arestatistically significant reflecting a distinction between roles which isunlikely to be due to chance.5.2 Emotion Analysis (RQ1 Results)We conduct a comprehensive emotion analysis of the interactions betweenmentors and mentees using the issue and pull request comments data in ourdataset.The initial data used for this analysis is identical to the sentiment anal-ysis we did. Table 5.1 is an overview of this data.5.2.1 Distribution of EmotionsWe start emotion analysis by examining the distribution of emotions ex-pressed by mentors and mentees. Comments are classified into six emotioncategories: Sadness, Joy, Love, Anger, Fear, and Surprise. We use our con-textual distribution metric (CD, Equation 4.1) to explore the proportion ofemotions in comments made by each role.Table 5.9 presents the distribution of emotion labels in mentee and men-tor comments. Joy is the most frequently expressed emotion in both roles,constituting approximately 90% of all the comments. Sadness follows, whileAnger, Fear, and Surprise appear less frequently. Love is the least expressedemotion in the dataset representing less than 0.02% of the comments.715.2. Emotion Analysis (RQ1 Results)RoleEmotionSadness Joy Love Anger Fear SurpriseMentees6,177 88,731 15 1,201 1,269 772(6.3%) (90.4%) (0%) (1.2%) (1.3%) (0.8%)Mentors4,565 94,992 14 3,067 1,530 1,361(4.3%) (90.0%) (0%) (2.9%) (1.5%) (1.3%)Table 5.9: Counts of comments with different emotion and role variablesand CD({role}, {emotion}) in parentheses.By switching the filters in the CD metric, we can calculate the distri-bution of roles in each emotion category. Figure 5.4 visually illustrates theproportion of roles within each emotion category. This figure highlights thatwhile Joy is almost equally expressed by both roles, mentors are much morelikely to express Anger and Surprise, whereas mentees express Sadness more.0 20 40 60 80 100PercentageSadnessJoyLoveAngerFearSurpriseEmotion Label57.5%48.3%51.7%28.1%45.3%36.2%42.5%51.7%48.3%71.9%54.7%63.8% RolementeementorFigure 5.4: CD({emotion}, {role}): Distribution of roles in each emotioncategory. Each bar represents an emotion label, and has two sections dis-tinguishing mentors and mentees portion in different colors.5.2.2 Statistical Analysis of EmotionsTo investigate the significance of these differences, a chi-square test of inde-pendence was used. Table 5.9 shows the observed counts of each emotion formentors and mentees (contingency table), serving as the input for our chi-square test to examine if there exists any relation between role and emotionlabel. Table 5.10 presents the results of the chi-square test.725.2. Emotion Analysis (RQ1 Results)Statistic ValueChi-square statistic 1,193P-value <0.001Table 5.10: Chi-square test results checking the relation between role andemotion.The p-value < 0.001 indicates that the observed association between roleand emotion is unlikely to be due to chance. This result reveals a statisticallysignificant difference in emotion expression between mentors and mentees,highlighting a genuine disparity in how these two groups express emotionsin their GitHub communications within the GSoC program.5.2.3 Weekly Analysis of EmotionIn this section, we analyze the expressed emotions in comments made ineach week of the GSoC. We use the emotion impact metric (Equation 4.3)to quantify the impact of each emotion over time. This metric calculatesthe proportion of comments expressing a specific emotion within a giventimeframe, allowing us to observe trends and changes in emotion expressionsthroughout the program.Weekly TrendWe calculated the weekly emotion impact of each emotion label, and theresults are visualized in Figures 5.5 to 5.9. These figures show the pro-portion of comments expressing each emotion, highlighting key trends andfluctuations over the 20-week duration of the GSoC program.Sadness Figure 5.5 illustrates the weekly impact of sadness for mentorsand mentees. The impact of sadness in mentees comments generally fluc-tuates between 5.5% and 7.5%, with notable peaks in Week 9 right at themidterm evaluation deadline and week 13. For mentors, the proportion ofsadness remains relatively stable around 3.5% and 4% before the midtermevaluation, and then experiences a noticeable increase one week prior tothe midterm evaluation and keeps its heightened impact for the rest of theprogram fluctuating around 4.5%.735.2. Emotion Analysis (RQ1 Results)0 3 5 7 9 11 13 15 17 19Week3.64.24.95.56.26.87.5ImpactMentorMenteeCoding StartsMid-term EvaluationFigure 5.5: Impact of sadness throughout the program in mentors andmentees. Interpret with care: The Y-axis does not include the full range ofpossible values, which may visually exaggerate small differences.Anger Figure 5.6 presents the weekly impact of anger. Impact of angerfluctuates between 0.5% and 1.5% in mentees comments, with an initialrise peaking around Week 2 right before coding starts. For mentors, theimpact is two to three times larger than mentees in the whole duration ofthe program. Its most notable peaks are in Week 3 right when coding startsand in week 17.745.2. Emotion Analysis (RQ1 Results)0 3 5 7 9 11 13 15 17 19Week0.51.01.52.02.53.03.5Impact MentorMenteeCoding StartsMid-term EvaluationFigure 5.6: Impact of anger throughout the program in mentors andmentees. Interpret with care: The Y-axis does not include the full range ofpossible values, which may visually exaggerate small differences.Fear The weekly impact of fear is illustrated in Figure 5.7. The trends formentees and mentors exhibit distinct patterns, with noticeable differencesin their peaks and troughs. For mentees, the most significant peaks occur inWeeks 13 and 18, suggesting heightened anxiety during these periods. Men-tors on the other hand, experience substantial peaks in fear during Weeks3 and 8, interestingly corresponding to some of the lowest levels of fear formentees. This inverse relationship between mentors and mentees is partic-ularly noteworthy; unlike sadness and anger, which show similar trends forboth roles, fear’s impact demonstrates an opposing behavior. When theimpact of fear increases for one role, it tends to decrease for the other inmany cases, highlighting the differing fear dynamics between mentors andmentees throughout the program.755.2. Emotion Analysis (RQ1 Results)0 3 5 7 9 11 13 15 17 19Week0.91.11.21.41.51.71.8ImpactMentorMenteeCoding StartsMid-term EvaluationFigure 5.7: Impact of fear throughout the program in mentors and mentees.Interpret with care: The Y-axis does not include the full range of possible values,which may visually exaggerate small differences.Surprise Figure 5.8 depicts the weekly impact of surprise. The impact ofsurprise expressed by mentees is low and is fairly stable in different weekswith a noticeable decrease in week 18. For mentors, the impact is roughlydoubled of mentees surprise impact, with peaks in Weeks 2, and 10.0 3 5 7 9 11 13 15 17 19Week0.50.70.91.01.21.41.6ImpactMentorMenteeCoding StartsMid-term EvaluationFigure 5.8: Impact of surprise throughout the program in mentors andmentees. Interpret with care: The Y-axis does not include the full range ofpossible values, which may visually exaggerate small differences.765.2. Emotion Analysis (RQ1 Results)Joy Figure 5.9 illustrates the weekly trend of joy, which remains consis-tently high for both mentors and mentees. Minor fluctuations are observed,with drops around midterm evaluation deadline. The trend lines of mentorsand mentees tend too agree in majority of the duration of the GSoC.0 3 5 7 9 11 13 15 17 19Week88.689.289.890.491.091.692.2ImpactMentorMenteeCoding StartsMid-term EvaluationFigure 5.9: Impact of joy throughout the program in mentors and mentees.Interpret with care: The Y-axis does not include the full range of possible values,which may visually exaggerate small differences.Statistical Analysis of Weekly TrendsTo statistically verify the observed trends, we conducted chi-square testsof independence for weekly emotion distributions. Table 5.11 shows theresults, indicating significant differences in weekly emotion proportions forboth mentors and mentees.Role Chi-square statistic P-valueMentor 177 < 0.001Mentee 159 < 0.001Table 5.11: Chi-square test results for weekly emotion trends.The p-values < 0.001 for mentors’ and mentees’ comments confirm thestatistically significant differences in weekly emotion impacts across the 20weeks. These p-values underscore the significant fluctuations in emotionsthroughout the program.775.2. Emotion Analysis (RQ1 Results)Similar to our analysis of sentiments, now we compare the consecutiveweeks to identify the most significant week pairs in which the emotion impacthas the most statistically significant change. Table 5.12, highlight key pointsof emotion fluctuations throughout the program identifying specific weekswith significant emotional changes, particularly for mentors.WeeksChi-square statisticMentor Mentee0 vs 1 12.47* 3.861 vs 2 3.99 7.912 vs 3 4.44 4.833 vs 4 15.08* 4.674 vs 5 10.27 5.415 vs 6 5.56 2.816 vs 7 4.92 7.607 vs 8 23.40*** 4.278 vs 9 14.42* 11.009 vs 10 4.27 7.7210 vs 11 2.20 3.9811 vs 12 5.98 1.7212 vs 13 6.82 20.92***13 vs 14 5.78 8.7814 vs 15 12.68* 7.5615 vs 16 13.68* 3.7316 vs 17 6.14 4.4317 vs 18 6.04 8.3918 vs 19 2.96 13.08*Table 5.12: Pairwise chi-square test results for consecutive weeks. Signifi-cance levels: * p <0.05, ** p <0.01, *** p <0.001.The weekly analysis of emotions reveal significant trends and fluctuationsin emotion expressions among mentors and mentees throughout the GSoC785.2. Emotion Analysis (RQ1 Results)program. By examining the emotion impact metric (Equation 4.3) and con-ducting statistical tests, we identified critical periods of emotional change,providing insights into the dynamic nature of mentor-mentee interactionsduring the program.Revisiting emotion impactIntegrating the analysis of weekly emotion impacts with the statisticallysignificant week pairs identified, we revisit the emotion impacts highlightingcritical phases in the GSoC program.For mentors, shifts in emotions were statistically significant in a coupleof consecutive weeks, focusing on these week pairs, we take another look atthe emotion impact trend plots.• Week 0 to 1: Notable increase were observed in the impact of anger.• Week 3 to 4: This period corresponds to the beginning of the codingphase, where notable declines in anger and fear were seen in the impactplots.• Week 7 to 8: This period precedes the midterm evaluation and ismarked by the most significant p-value from our chi-square test. Dur-ing this time, we observe notable increases in the impacts of sadnessand fear, along with a sharp decrease in the impact of joy among men-tors. The rise in negative emotions, coupled with the drop in joy, likelyreflects the heightened levels of stress and challenges faced by mentorsas they approach the critical midterm evaluation deadline.• Week 8 to 9: The last day of Week 9 marks the midterm evaluationdeadline. During this period, we observe a sharp decrease in the impactof fear and a notable increase in joy among mentors. This reductionin fear, coupled with the rise in joy, likely indicates a sense of reliefand satisfaction in mentors as they reach this critical milestone.• Week 14 to 15: We see a large decrease in the impact of anger, cou-pled with an evident increase in joy, reflecting a possible reduction infrustration and a boost in positive emotions. This shift likely corre-sponds to the successful progress or resolution of challenges within theproject, contributing to a more positive outlook among mentors.• Week 15 to 16: We observe a decrease in the impacts of joy and fear,alongside an increase in the impacts of sadness and anger. Notably, the795.2. Emotion Analysis (RQ1 Results)decrease in fear and the increase in anger are more pronounced. Thisshift may indicate the onset of new challenges following the previousweek’s boost in positive emotions. The heightened anger suggestsgrowing frustration, while the decreased fear may imply mentors areconfronting these challenges head-on.Formentees, the identified significant week pairs are: (12, 13), and (18,19). These pairs highlight key moments of emotion fluctuations in mentees:• Week 12 to 13: This period showed significant peaks in sadness andfear. A large decline in joy is also present during this period. Thisperiod can be an interesting point, as it does not coincide with themain milestones or other data we have.• Week 18 to 19: As the program approached its conclusion in thisperiod, we see huge and sharp decrease in fear and a large increase injoy in mentees, suggesting a possible sense of relief and accomplishmentas mentees completed their projects.5.2.4 Milestone-Based Analysis of EmotionsFor a milestone-based emotion analysis, we focus on three key periods: theCommunity Bonding Period (weeks 0-3), the First Work Period (weeks 4-9), and the Second Work Period (weeks 10-19). The emotion impact metric(Equation 4.3), is used to quantify the presence of each emotion during theseperiods.Table 5.13 shows the distribution of emotions across these periods. Bothmentors and mentees exhibit high levels of Joy throughout the program, in-dicating overall positive engagement. However, there are notable differencesin the impact of other emotions at each milestone.During the Community Bonding Period, mentees show a higher impact ofsadness compared to mentors, possibly due to initial adjustment challenges.In the First Work Period, which marks the start of coding, there is anincrease in Anger and Fear among both mentors and mentees, indicatingthe pressures and challenges of beginning project implementation.The Second Work Period extends from the midterm evaluation to theprogram’s end. During this phase, the impact of Fear and Anger decreasestowards the end, while Joy increases, indicating relief and satisfaction asprojects near completion. Mentors exhibit a more stable emotion profile,while mentees show more fluctuation, particularly the midterm evaluation.To statistically verify differences in emotion impact across the three peri-ods, we conducted chi-square test of independence for the datas of each role.805.2. Emotion Analysis (RQ1 Results)Table 5.14 presents the results, indicating significant differences in emotionexpressions among mentors across the milestones (p-value < 0.001), butnot among mentees (p-value = 0.52). These findings suggest that mentors’emotion show a statistically significant variation among these three periods,while the observed differences in mentees’ emotions among these three peri-ods is probably just a random thing explained by the very large p-value of0.52.Bonding Period (weeks 0-3), the First Work Period (weeks 4- 9), and theSecond Work PeriodRole PeriodEmotion (%)Sadness Joy Love Anger Fear SurpriseMenteeBonding Period 6.1 90.9 0.02 1.1 1.2 0.7First Work Period 6.3 90.4 0.02 1.2 1.3 0.8Second Work Period 6.3 90.3 0.01 1.3 1.3 0.8MentorBonding Period 4 90.6 0.01 2.7 1.5 1.2First Work Period 4 90.5 0.02 2.9 1.4 1.2Second Work Period 4.6 89.6 0.01 2.9 1.5 1.4Table 5.13: Emotion distributions across major project periods.Role Chi-square statistic p-valueMentor 34.264 < 0.001Mentee 9.131 0.52Table 5.14: Chi-square test results for emotions across major project mile-stones.The analysis highlights how mentors and mentees experience and expressemotions differently at various stages of the program. These insights providea nuanced understanding of the emotions expressed in mentor-mentee inter-actions, emphasizing the importance of considering emotional well-being instructured programs like GSoC.815.3. Sentiment and Emotion Analysis of Transitioning Roles (Mentee to Mentor)Summary- Joy is the dominant emotion for both mentors and mentees (About90% of all comments). Sadness follows, comprising of about 6% ofmentee comments, and 4% of mentor comments. Love was scarcelyrepresented at less than 0.02% for both roles.- Mentors expressed Anger 2.9% and Surprise 1.3% which is almostdouble of these percentages in mentees who express 1.2% Anger and0.8% Surprise.- Emotion trends showed variations in to time. For instance, menteesexperienced peaks in Sadness during the midterm evaluation, whereasmentors see an increase in Anger at the onset of first coding phase.- Irrespective of the amount, the differences in emotions between men-tors and mentees and between different times of the program are sta-tistically significant and most probably non-random.5.3 Sentiment and Emotion Analysis ofTransitioning Roles (Mentee to Mentor)In the GSoC program, some users transition from being mentees in oneyear to becoming mentors in subsequent years. In this section, we explorehow users who transitioned from mentees to mentors in the GSoC programchanged their expression of sentiments and emotions. This transition pro-vides a unique perspective on how assuming a mentor role impacts communi-cation styles and emotion expressions. By examining the same individuals inboth roles, we aim to conduct an in depth analysis of having different roles,compared to the previous results showing the differences between differentgroups of people.5.3.1 Sentiment Analysis Before and After TransitionTo examine the changes in sentiment, we calculated the average sentimentscores of users’ comments before and after they transitioned from mentees tomentors. Additionally, we analyzed the proportion of positive and negativecomments from them in both roles. Figure 5.10 shows three plots, comparingthe sentiments and emotions before (i.e., as a mentee) and after (i.e., as amentor).825.3. Sentiment and Emotion Analysis of Transitioning Roles (Mentee to Mentor)Before After0246810Comment PercentagePositiveNegativeBefore After0.000.010.020.030.040.050.060.070.08Average Sentiment ScoreBefore After0.000.020.040.060.08ProportionNote: Joy proportion values are divided by 10EmotionsangerfearjoysadnesssurpriseFigure 5.10: Analysis of sentiments and emotions before and after transi-tioning from mentee to mentor. The figure is divided into three parts: Left:Percentage of positive and negative sentiments expressed before and aftertransitioning, showing a decrease in positive sentiments and a slight increasein negative sentiments post-transition. Middle: Average sentiment scoresbefore and after the transition, illustrating a decline in overall sentimentpositivity when users become mentors. Right: Proportions of differentemotions expressed before and after transitioning, highlighting noticeableincreases in anger and fear, and a decrease in sadness after becoming men-tors. Interpret with care: The Y-axis does not include the full range of possiblevalues, which may visually exaggerate small differences.Figure 5.10 illustrates three key aspects of this analysis:• Positive and Negative Sentiments:– Mentees expressed 11.1% positive sentiments and 3.1% negativesentiments.– After becoming mentors, percentage of positive sentiments de-creased to 7.4%, while negative sentiments slightly increased to3.3%.• The average sentiment score decreased by 50% when transitioningfrom being a mentee to a mentor (from 0.08 to 0.04).5.3.2 Emotion Analysis Before and After TransitionNext, we analyzed the proportions of different emotions expressed by usersbefore and after their transition. The right plot in Figure 5.10 shows the835.3. Sentiment and Emotion Analysis of Transitioning Roles (Mentee to Mentor)differences among the emotions before and after becoming a mentor.EmotionDistribution(%)Mentee MentorJoy 92.3 90.7Sadness 4.9 4.1Fear 1.1 1.6Anger 0.9 2.7Surprise 0.7 0.8Love 0 0Table 5.15: Proportions of different emotions expressed before and aftertransitioning from mentee to mentor.As presented in Table 5.15, the proportions of joy, sadness and surpriseshow minor changes. The changes in fear and anger are more pronounced,the percentage of fear increases from 1% to 1.6%, and anger increases to2.73% after the transition, from being 0.93% before the transition.5.3.3 Statistical AnalysisTo determine the significance of these changes, we performed a t-test forthe sentiment scores and a chi-square test for the emotion proportions. Thet-test results (see Table 5.16) indicate a statistically significant reductionin sentiment scores . Similarly, the chi-square test results (see Table 5.17)confirm significant differences in emotion proportions before and after thetransition.Statistic ValueT-Statistic −7.08P-Value < 0.001Table 5.16: T-test results comparing average sentiment scores before andafter transitioning.845.4. Topic Modeling (RQ2 Results)Statistic ValueChi-square Statistic 107.8P-Value < 0.001Table 5.17: Chi-square test results comparing emotion proportions beforeand after the role transition.Transitioning from a mentee to a mentor in the GSoC program impactshow users express sentiments and emotions. The decrease in positive sen-timent and increase in negative emotions like anger and fear suggest thatthe mentor role introduces new challenges and responsibilities that affectcommunication dynamics. These findings provide valuable insights into theemotional experiences of participants as they advance in their roles withinthe program.Summary- Focusing on participants who transition from mentees to mentors,sentiment analysis indicates a notable decrease in positive sentiments,from 11% to 7% as individuals transition from mentees to mentors.The average sentiment score was reduced by half, from 0.08 to 0.04.- Emotion analysis indicates that Anger tripled, rising from 0.93%to 2.73%, and fear rises from 1.08% to 1.62%. Changes in joy andsadness were minor.- These results closely align with previous analyses we did comparingmentors and mentees, reinforcing the observed differences betweenthese roles. The comparison of the same individuals in different roleshelps to isolate the effect of roles on sentiment and emotion expres-sions.5.4 Topic Modeling (RQ2 Results)We identify and analyze the main themes and topics in the communica-tions between mentors and mentees within the GSoC program. Our topicmodeling approach uncovers 36 topics from the textual data of mentor andmentee interactions. These topics were then manually labeled to facilitateinterpretation and grouped into broader themes of discussion for better un-derstanding. These themes cover various aspects of software development855.4. Topic Modeling (RQ2 Results)and project management, including version control, programming founda-tions, data management, and quality and testing among others. We presentthe distribution of comments across these themes, comparing the contribu-tions of mentors and mentees. We also examined how the prevalence ofthese themes changed over time and in relation to key milestones in theGSoC program. This temporal analysis helps in understanding the dynam-ics of mentor-mentee interactions and the shifting focus areas throughoutthe project lifecycle. To present the findings from our topic modeling anal-ysis, we start with the identification of topics which then lead to the mainthemes of discussion by merging the similar topics.5.4.1 Main Themes and Topics in Mentor-MenteeCommunicationsWe discovered 36 topics using our topic modeling methodology. The topicsare presented in Table B.1 (in appendix) along with a manually-createdtopic names to briefly describe each topic. We also show the distributionof roles in each topic in Table B.2 (in appendix), indicating the proportionof comments from mentees, and mentors. The topics are sorted by theirshare metric (Equation 4.4) in descending order. We grouped similar topicsforming 11 themes of discussion between GSoC users. As a result, each ofthe 36 initially discovered topics are mapped into one of the themes. Thesethemes and their respective topics mapped to them are shown in Table 5.18.865.4. Topic Modeling (RQ2 Results)Theme Merged TopicsProgrammingFoundationsVariable Naming Conventions, Coding Conventions, DevelopmentEnhancements, Python Imports and Data Structures, Code In-dentation and Continuation, Constants and Static Variables, Syn-tax Specifics, Mathematical ComputationsVersion Control &CollaborationReview Comments and Acceptance, Pull Request Management,Code Updates and Commits, Review Rework, Review and Feed-back, Conflict Resolution, Comment JustificationsData Management Package Management and Errors, Kubernetes Operations,Stream Loading and RANSAC, Database Schema and KeysTask Completion Gratitude and Acknowledgements, Task ClosureQuality & Testing Build Failures, Testing Quality, Test Results and Expectations,Punctuation and Proofreading, Coverage and Diff AnalysisWeb & UI/UX UX and UI Elements, Web App UI ElementsMaintenance &EvolutionCode Cleanup, Code RefactoringDebugging & Error Fatal Errors and Allocators, Fixes and ResolutionsSoftware Design Code Structure and DesignProject Planning Sprint Planning and Issue AssignmentDeployment Documentation and Comments, Licensing and CLATable 5.18: Overview of the themes and their associated topics identified inthe project communications.To provide a better sense of the themes, we present a subset of them andan example16 comment for a couple of the topics in that theme.16The examples are chosen randomly from all the comments with more than 10 words;names and usernames are removed from the raw comment content.875.4. Topic Modeling (RQ2 Results)Theme: Programming FoundationsFrom the 36 topics extracted, Eight are about Programming Founda-tions. This theme focuses on basic programming concepts and bestpractices, including naming conventions, code structure, and funda-mental coding principles.Topic: Variable Naming ConventionsExample: No need for the extra casing in here, we can call it ‘Autocomplete’.Topic: Coding ConventionsExample: If you wanted, you could rewrite this like this:“ const double pointAccumErrorTol = accumError(queryIndex) / ((already-DidRefPoint0) ? (refNumDesc - 1) : refNumDesc); ”It’s up to you which you think is more readable. :+1:Topic: Python Imports and Data StructuresExample: Can ‘float(x)’ return ‘f32’ or ‘f64’ depending on the kind of ‘x’? Ordoes it always return ‘f64’?Topic: Code Indentation and ContinuationExample: Line should be indented 4 spaces, but was indented 5 spacesTheme: Task CompletionOut of 36 topics, two are specifically about Task Completion. Thistheme includes discussions about completing tasks, acknowledgingcontributions, and celebrating milestones. It emphasizes recognizingthe hard work of contributors and the progress made on the project.Topic: Gratitude and AcknowledgementsExample: Apart from the problems with getting this to run, this is looking reallypositive! Thanks, NAME! And thanks NAME for the review - I didn’t havetime until tonight.Topic: Task ClosureExample: @USER @USER Since I have added these changes in #42 . Henceclosing this.885.4. Topic Modeling (RQ2 Results)Theme: Web & UI/UXTwo of the extracted topics are specifically about Web & UI/UX.Discussions in this theme revolve around web development and userinterface/user experience design.Topic: UX and UI ElementsExample: I guess the following inline CSS will be removed once the design isfinalized? :thinking:Topic: Web App UI ElementsExample: **1. Infinite Scroll on speakers page**Features such as this can be implemented only when the data is fetched from thefront-end side. @USER what’s your suggestion ?Theme: Project PlanningAmong all the 36 topic discovered by our topic modeling methodology,only one is about Project Planning. This theme involves discussionsaround planning and managing tasks.Topic: Sprint Planning and Issue AssignmentExample: @USER what’s your status on this? This would be a good first issuefor you to fix/implement as an official GSoC student.5.4.2 Evolution of Topics Throughout the GSoC ProgramTo investigate the main themes of discussion over time, we analyzed themon a weekly basis using our impact (Equation 4.5) and normalized impactNI(Equation 4.6) metrics on the identified topics and themes. Table 5.19presents an overview of the identified themes. For each theme, we can seethe share (Equation 4.4), role distribution, and a sparkline of the trend ofimpact throughout the program timeline.895.4. Topic Modeling (RQ2 Results)Theme Share(%)Distribution (%)Impact trendMentee MentorProgramming Foundations 25 37.5 62.5Version Control & Collaboration 21.2 54 46Data Management 12.4 45.9 54.1Task Completion 7.8 81.3 18.7Quality & Testing 7.4 44.4 55.6Web & UI/UX 6.8 48.9 51.1Maintenance & Evolution 6.4 38 62Debugging & Error 5.4 67.2 32.8Software Design 3 26.7 73.3Project Planning 2.7 50 50Deployment 2 28.2 71.8Table 5.19: The main themes along with their share metric (Equation 4.4),the percentage of comments written by each role, and the sparkline of theimpact metric (Equation 4.5) over the 20 weeks of the GSoC.The top three themes based on the share metric (Equation 4.4) are Pro-gramming Foundations, Version Control & Collaboration, and Data Man-agement, covering more than half of the discussions collectively. In termsof role distribution, Task Completion is the top theme with the majority ofcomments from mentees, with mentees accounting for 81% of the comments.Conversely, Deployment and Software Design are the top themes with themajority of comments from mentors, with mentors representing 72% and69% of the comments in Deployment and Software Design, respectively.For a better investigation over the variation of themes over time, aheatmap of the NI metric (Equation 4.6) is shown in Figure 5.11. Theheatmap provides a valuable view of the temporal dynamics of discussionthemes throughout the GSoC program timeline, highlighting how much fo-cus is on different themes over time in the program.As one would expect, Programming Foundations has more domi-nance in the weeks of active development and coding. It has a high NI fromweek 3 when the coding starts all the way to week 17, only two weeks be-fore the end of program. In the first three weeks and the final 2 weeks ofthe program, there is very low level of focus on this theme relative to otherweeks. Version Control & Collaboration has a high impact in weeks 0905.4. Topic Modeling (RQ2 Results)to 4, week 9, 14, and the final two weeks of the program. Conversations inthis theme are mainly about repository structure and management; and itis not surprising that it has its most impact in the early weeks to set up theproject and the very last weeks to conclude and end the program. DataManagement shows moderate fluctuations with its peak in weeks 2. TaskCompletion peaks in week 5, maintains high impact until week 8, followedby a moderate impact in weeks 9 to 15. It starts to rise again from week15 and continues to increase with its peak in the two final weeks. Quality& Testing is mostly prominent in weeks 0 to 17 having its highest point inweek 8, two weeks prior to the midterm evaluation deadline. It has has verylow relative impact in weeks 18, and 19, the 2 final weeks. Web & UI/UXis highly impactful in the first three weeks. Then it has a peak in week 10and keeps its high impact until week 16. Debugging & Error has a lowimpact in the first eight weeks, it rises a little bit around week 9, right atthe midterm evaluation deadline, followed by a gradual decrease until week14, then rises again, leading to its highest impact in the final week. ProjectPlanning is highly impactful in weeks 0-4, then significantly drops, havingvery low NI for the rest of the program. Deployment shows low impactuntil week 11 which is the first week after the midterm evaluation, thenincreases, peaking in week 16, having a high impact period from week 15 toweek 18 corresponding to final project wrap-up activities.0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19weekProgrammingFoundationsVersion Control& CollaborationData ManagementTask CompletionQuality & TestingWeb & UI/UXMaintenance& EvolutionDebugging& ErrorSoftware DesignProject PlanningDeployment0.13 0.16 0.39 0.73 0.61 0.95 1.00 0.93 0.78 0.66 0.87 0.40 0.80 0.75 0.35 0.66 0.61 0.55 0.11 0.001.00 0.64 0.48 0.67 0.63 0.36 0.23 0.00 0.05 0.46 0.18 0.31 0.12 0.19 0.63 0.22 0.11 0.32 0.49 0.590.00 0.85 1.00 0.41 0.37 0.40 0.49 0.54 0.69 0.49 0.46 0.64 0.36 0.71 0.28 0.58 0.71 0.48 0.71 0.610.11 0.26 0.00 0.38 0.26 0.67 0.26 0.63 0.58 0.25 0.25 0.20 0.33 0.39 0.13 0.39 0.55 0.45 0.90 1.000.71 0.88 0.81 0.57 0.54 0.47 0.81 0.67 1.00 0.45 0.49 0.65 0.86 0.37 0.82 0.51 0.65 0.76 0.20 0.001.00 0.46 0.77 0.18 0.59 0.19 0.16 0.66 0.00 0.65 0.94 0.90 0.62 0.75 0.88 1.00 0.51 0.35 0.19 0.190.00 0.13 0.28 0.42 0.64 0.64 1.00 0.66 0.89 0.73 0.63 0.78 0.92 0.56 0.69 0.27 0.43 0.53 0.49 0.520.48 0.30 0.26 0.13 0.00 0.11 0.15 0.13 0.37 0.45 0.47 0.32 0.23 0.34 0.47 0.32 0.46 0.44 0.51 1.000.00 0.70 0.51 0.64 0.55 0.77 0.81 0.56 0.60 0.73 0.71 0.84 0.77 0.82 0.71 0.87 0.77 1.00 0.86 0.501.00 0.46 0.43 0.37 0.43 0.18 0.16 0.10 0.07 0.08 0.14 0.12 0.09 0.07 0.02 0.08 0.01 0.00 0.11 0.200.00 0.25 0.23 0.52 0.43 0.29 0.16 0.62 0.36 0.23 0.11 0.35 0.44 0.56 0.34 0.73 1.00 0.83 0.84 0.220.00.20.40.60.81.0Figure 5.11: Heat map showing the prominence of the identified themes overtime. Each cell represents the NI metric (Equation 4.6) of one theme in aspecific week of the program.915.4. Topic Modeling (RQ2 Results)5.4.3 Comparison of RolesThe percentage of comments posted by each role is provided in Table 5.19.We also analysed and discussed the most impactful weeks for each themeby presenting a heat map of weekly NI metric (Equation 4.6) of the themesin Figure 5.11. Now we compare the thematic trends of comments made byeach role.0 3 9 19Week0.00.51.0NIMentorMenteeCoding StartsMid-term Evaluation(a) Programming Foundations0 3 9 19Week0.00.51.0NIMentorMenteeCoding StartsMid-term Evaluation(b) Version Control & Collaboration0 3 9 19Week0.00.51.0NIMentorMenteeCoding StartsMid-term Evaluation(c) Data Management0 3 9 19Week0.00.51.0NIMentorMenteeCoding StartsMid-term Evaluation(d) Task Completion0 3 9 19Week0.00.51.0NIMentorMenteeCoding StartsMid-term Evaluation(e) Quality & Testing0 3 9 19Week0.00.51.0NIMentorMenteeCoding StartsMid-term Evaluation(f) Web & UI/UXFigure 5.12: Weekly Normalized Impact(NI, Equation 4.6) of the 6 mostdiscussed themes in discussions. Each plot has two trend lines, one cal-culated on the mentor comments, and the other calculated on the menteecomments. The normalized nature of NI keeps the comparison focused onthe rate of changes opposed to only focusing on the number of commentscreated by each role in that theme category.925.4. Topic Modeling (RQ2 Results)Figure 5.12 shows the role-based dynamics by illustrating the weeklynormalized impact (NI, Equation 4.6) of the six most discussed themes. Eachplot presents two trend lines, one for each role, providing a clear comparisonof how mentors and mentees contribute to discussions over time. By usingNI (Equation 4.6), we aim to observe the rate at which the focus on eachtheme changes over time, rather than simply presenting the actual changesthat occurred. Aligned NI curves between mentors and mentees in a themeshow similar level of concern about that subject as both roles are similarlychanging their level of focus in that theme. This is similar to methods usedin other fields such as environmental science, where researchers assess therate of global temperature changes to understand climate trends.In some themes, the trend lines are mostly synchronized (e.g., VersionControl & Collaboration) indicating a collaborative approach, where the NItrend line of both mentors and mentees have similar slopes and are mostlymoving in the same direction. Conversely, in some themes (e.g., Program-ming Foundations), the behaviour of the trend lines of mentors and menteesare somewhat opposite, where the trend lines are mostly either diverging orconverging. The NI trend lines of Programming Foundations, for example,show opposite behaviors initially, but later, start to have matching slopesin trend lines suggesting phased focus and eventual alignment in efforts asthe program progresses towards its end. For Data Management, the trendsmatch closely until week 4, but become somewhat opposite in direction af-ter week 4, reflecting a shift in focus over time. Similar to strategies inhealthcare where researchers try to align the rate of allocation of specifichealthcare resources with a disease spread rate [56], this concept has thepotential to lead to better support and enhancements in mentorship.935.5. Effectiveness of GSoC (RQ3 Results)Summary- We identified 36 distinct topics grouped into 11 themes.- Programming Foundations, Version Control & Collaboration, andData Management were the most discussed themes.- Task Completion was predominantly discussed by mentees; Deploy-ment by mentors.- Programming Foundations had higher impact during the coding pe-riod. Version Control & Collaboration was more prominent in initialand final weeks. Quality & Testing peaked around midterm evalua-tion.- NI trends during GSoC showed synchronized focus between mentorsand mentees in some themes, e.g., Version Control & Collaborationand Data Management, and phased focus in some others, e.g., Pro-gramming Foundations and Web & UI/UX.5.5 Effectiveness of GSoC (RQ3 Results)In this section, we shift our focus to the overall effectiveness of GSoC. Ourgoal is to evaluate the impact of GSoC on mentees’ improvement in con-tributions to OSS post-participation compared to their contributions pre-participation.Using the success metric (Equation 4.7), which is detailed in Chap-ter 4, we classify GSoC projects into two categories: “improved” and “unim-proved.” Projects classified as “improved” are those where the success metricis positive, indicating an increase in mentee contributions after GSoC. Con-versely, “unimproved” projects are those with a non-positive success metric,suggesting no improvement or a decline in contributions.In this final section, we will compare the “improved” and “unimproved”projects to identify behaviors and patterns during the GSoC duration thatcorrelate with these outcomes. Our aim is to uncover significant factors asso-ciated with successful projects and to highlight any key differences betweenthe two groups. To achieve this, we will use various metrics introduced inprevious sections, and their significance will be evaluated using t-tests fornumerical metrics and Chi-square tests for categorical variables.Before delving into detailed comparisons, we present some basic statis-tics for the “improved” and “unimproved” projects in Table 5.20. Thesestatistics provide an overview of the data.945.5. Effectiveness of GSoC (RQ3 Results)Statistic Improved Unimproved TotalNumber of projects 577 901 1,478Number of comments 68,247 123,972 192,219Number of commits 74,368 114,285 188,653Mean success (Equation 4.7) 3 −4.7 −1.7Table 5.20: Basic statistics for improved and unimproved groups.These statistics reveal that a significant portion of projects did not seean improvement in mentee contributions post-GSoC. However, there is asubstantial number of projects that were successful.In the following, we will analyze differences in sentiment, emotion, themesof discussion, the number of comments per project, and the number of com-mits per project between the “improved” and “unimproved” groups. Ourobjective is to identify factors that significantly correlate with project suc-cess, thereby providing valuable insights into optimizing GSoC’s impact onopen source contributions.5.5.1 Analysis of Sentiment, Comments, and CommitsWe evaluate the differences in sentiment scores comments per project(PPCcomment - Equation 4.9), and commits per project (PPCcommit - Equa-tion 4.10) between improved and unimproved projects.Comparison of MeansAs detailed in Table 5.21, the mean sentiment scores for mentors andmentees are very close between improved and unimproved projects indicatingthat sentiment alone does not substantially differ between improved andunimproved projects.The mean PPCcomment for mentors is 71.1 in improved projects and76.9 in unimproved projects, resulting in a difference of 5.8. For mentees, themean PPCcomment is about 58 in improved projects and 72.2 in unimprovedprojects, indicating a difference of 14.2. These results show a higher averagenumber of comments per project in unimproved projects, particularly formentees.For PPCcommit, mentors have a mean of 54.4 in improved projectsand 69.8 in unimproved projects, with a difference of 15.4. For mentees,955.5. Effectiveness of GSoC (RQ3 Results)the mean PPCcommit is 62.5 in improved projects and 49.7 in unimprovedprojects, with a difference of 12.8. These results show that improved projectshave a higher average number of commits from mentees, while unimprovedprojects have more commits from mentors.Measure Role Statistic Improved UnimprovedSentiment scoreMentorMean 0.06 0.06Std 0.37 0.37MenteeMean 0.09 0.09Std 0.40 0.39PPCcommentMentorMean 71 77Std 120 116MenteeMean 58 72Std 100 145PPCcommitMentorMean 54 70Std 84 125MenteeMean 62 50Std 91 71Table 5.21: Comparing improved and unimproved projects using senti-ment scores (Defined in Section 4.2.1), number of comments per project(PPCcomment, see Equation 4.9), and number of commits per project(PPCcommit, see Equation 4.10). The means of PPC metrics are definedin Equations 4.11, and 4.12.T-testsTo determine the statistical significance of the observed differences in sen-timent scores comments per project, and commits per project between im-proved and unimproved projects, we conducted t-tests. The results are pre-sented in Table 5.22.The t-tests for sentiment scores indicate no significant differences be-tween improved and unimproved projects. The high p-value of 0.87 for965.5. Effectiveness of GSoC (RQ3 Results)mentors, and 0.93 for mentees, suggest that there is no statistically signif-icant difference in sentiment scores between “improved” and ‘unimproved”projects.The t-tests for PPCcomment reveal a significant difference for menteesbut not for mentors with p-values of 0.04 and 0.39 respectively. This resultsuggests that the number of mentees comments per project is significantlyhigher in unimproved projects compared to improved projects.The t-tests for PPCcommit indicate significant differences for both men-tors and mentees. For mentors, the p-value is 0.008, showing a significantdifference between the groups. For mentees, the p-value is 0.0012, also in-dicating a significant difference. These results suggest that the number ofcommits per project is significantly higher in improved projects for menteesand significantly higher in unimproved projects for mentors.Measure Role T-statistic P-valueSentiment scoreMentor −0.16 0.87Mentee 0.09 0.93PPC comment (Equation 4.9)Mentor −0.86 0.39Mentee −2.03 0.04PPC commit (Equation 4.10)Mentor −2.64 0.008Mentee 3.24 0.001Table 5.22: T-test results, improved vs. unimproved.In summary, while sentiment scores do not differ between improved andunimproved projects, the number of comments and commits per projectshow significant variations. Specifically, unimproved projects have signifi-cantly more comments from mentees, and improved projects have signifi-cantly more commits from mentees and fewer commits from mentors.Bootstrap ResamplingTo further validate our findings and provide a better visual comparisonbetween improved and unimproved projects, we employed bootstrap resam-pling to estimate the distribution of means for sentiment scores commentsper project (PPCcomment), and commits per project (PPCcommit).Figures 5.13 presents the bootstrap distributions of mean sentimentscores comments per project, and commits per project comparing “im-975.5. Effectiveness of GSoC (RQ3 Results)proved” and “unimproved” projects. Each distribution is generated from10,000 bootstrap samples.0.058 0.060 0.062 0.064 0.066 0.068 0.070Mean of Sentiment Scores04080120CountImprovedUnimproved(a) Mentor sentiment scores0.082 0.084 0.086 0.088 0.090 0.092 0.094Mean of Sentiment Scores04080120CountImprovedUnimproved(b) Mentee sentiment scores55 60 65 70 75 80 85 90MPPCcomment04080120CountImprovedUnimproved(c) Mentor PPCcomment45 50 55 60 65 70 75 80 85MPPCcomment04080120CountImprovedUnimproved(d) Mentee PPCcomment45 50 55 60 65 70 75 80 85MPPCcommit04080120CountImprovedUnimproved(e) Mentor PPCcommit45 50 55 60 65 70 75 80MPPCcommit04080120CountImprovedUnimproved(f) Mentee PPCcommitFigure 5.13: Histogram with KDE plots of bootstrap means for sentimentscores comments per project, and commits per project. Greater overlapof the distributions indicates a higher p-value, suggesting less significantdifferences between improved and unimproved projects. A larger distancebetween the centers of distributions on the other hand, highlights a greatermagnitude of difference, visually representing both the significance and mag-nitude of difference for each metric.Figures 5.13(a) and 5.13(b) show the bootstrap distributions of meansentiment scores for mentors and mentees in improved and unimprovedprojects. For both mentors and mentees, the bootstrap distributions of mean985.5. Effectiveness of GSoC (RQ3 Results)sentiment scores for improved and unimproved projects overlap significantly,reinforcing the results of the t-tests which indicated no significant differencesin sentiment scores between the two groups.Figures 5.13(c) and 5.13(d) show the bootstrap distributions of meanPPCcomment for mentors and mentees, respectively. The bootstrap dis-tributions for mentors’ comments per project show considerable overlap,aligning with the t-test results that found no significant difference. How-ever, for mentees, the bootstrap distributions has a much smaller overlapindicating the significant difference identified by the t-test.Figures 5.13(e) and 5.13(f) present the bootstrap distributions of meanPPCcommit for mentors and mentees, respectively. The bootstrap distribu-tions for mentors’ commits per project show a clear difference between im-proved and unimproved projects, supporting the significant difference iden-tified by the t-test. The bootstrap distributions also highlight a significantdifference for mentees, with improved projects having a higher mean numberof commits per project.Comparison Over TimeComparing improved with unimproved projects revealed significant differ-ence in number of mentee comments per project, and number of commitsper project for both roles. Now, we plot these measures over time comparingimproved and unimproved projects as shown in Figure 5.23. Mentees in im-proved projects exhibited fewer comments yet a higher number of commitscompared to their counterparts in unimproved projects. This suggests thatmentees in improved projects were more engaged in the technical contribu-tions rather than in discussions.Confirming our non temporal comparison results, mentors in unimprovedprojects tend to commit more frequently than mentees, whereas in improvedprojects, the opposite is true, indicating a more active role of mentees inthe technical development of the projects in the improved group. Interest-ingly, following the midterm evaluation, we can see a notable increase inmentee commits in improved projects. Conversely, in unimproved projects,a similar increase was observed among mentors rather than mentees. Thisdisparity suggests a potential shift in project contributions post-evaluation,with mentees in improved projects possibly carrying greater load in technicalcontributions.After week 5, the number of commits by mentees in improved projectsaligns closely with the number of mentor commits in unimproved projects.This alignment suggests a relative decrease in autonomy among mentees in995.5. Effectiveness of GSoC (RQ3 Results)unimproved projects as the program progresses.MetricRoleMentee Mentor3 9 13 194.67.410.1MPPCcomment3 9 13 193.46.69.83 9 13 193.46.69.8MPPCcommitLegendImprovedUnimprovedCoding StartsMid-term EvaluationTable 5.23: Weekly mean number of comments and commits per projectcomparing improved and unimproved projects over time in GSoC. MentorMPPCcomment is not ploted as the differences were not statistically signifi-cant between improved and unimproved.5.5.2 Analysis of Emotion and ThemesIn this subsection, we delve into the differences in emotion expression andthemes of discussion between improved and unimproved projects. Our ob-jective is to understand how these factors correlate with projects’ status 4.8and to identify any significant patterns or behaviors that might contributeto improved outcomes.Emotion AnalysisTable 5.24 presents the distribution of emotions expressed in comments bymentors and mentees, categorized by project status (improved vs. unim-proved).1005.5. Effectiveness of GSoC (RQ3 Results)Role StatusEmotionSadness Joy Love Anger Fear SurpriseMentorImproved1,482 32,708 3 1,070 497 437(4.1%) (90.4%) (0%) (3%) (1.4%) (1.2%)Unimproved2,780 55,739 9 1,817 937 841(4.5%) (89.7%) (0%) (2.9%) (1.5%) (1.4%)MenteeImproved2,018 28,904 5 427 408 288(6.3%) (90.2%) (0%) (1.3%) (1.3%) (0.9%)Unimproved3,859 56,021 10 729 794 436(6.2%) (90.6%) (0%) (1.2%) (1.3%) (0.7%)Table 5.24: Counts of comments having different emotion label and projectstatus (CD({role, status}, {emotion}) in parentheses).The chi-square tests of independence (Table 5.25) reveal significant dif-ferences in emotion expressions between improved and unimproved projectsfor both mentors (Chi2 = 16.1214, p = 0.006) and mentees (Chi2 = 14.8076,p = 0.011)17. Despite the significance, the distributions of emotions betweenimproved and unimproved are almost identical, showing negligible differencein the percentage of each emotion.Measure Role Chi2statistic P-valueEmotion distributionMentor 15.4 0.004Mentee 14.8 0.005Theme distributionMentor 31.2 < 0.001Mentee 359 < 0.001Table 5.25: Chi2 Tests of Independence between improved and unimprovedgroups.Theme AnalysisTable 5.26 presents the distribution of discussion themes for mentors andmentees, categorized by project status (improved vs. unimproved).17Love is not used for this test due to the chi-square requirement that each value fedinto the test and the expected values should be greater than 5 for a valid result.1015.5. Effectiveness of GSoC (RQ3 Results)Theme RoleComment count CDDifference(%)(CD % [Equation 4.1])Improved UnimprovedTaskCompletionmentee 3,597 ( 11.2) 9,016 ( 14.6) 3.4mentor 1,101 ( 3.0) 1,696 ( 2.7) 0.3DataManagementmentee 4,287 ( 13.4) 6,779 ( 11.0) 2.4mentor 4,796 ( 13.2) 7,982 ( 12.8) 0.4Version Control &Collaborationmentee 7,172 ( 22.4) 14,997 ( 24.2) 1.9mentor 6,724 ( 18.6) 11,845 ( 19.1) 0.5Debugging &Errormentee 2,583 ( 8.1) 4,440 ( 7.2) 0.9mentor 1,165 ( 3.2) 2,221 ( 3.6) 0.4ProgrammingFoundationsmentee 6,368 ( 19.9) 11,834 ( 19.1) 0.7mentor 10,741 ( 29.7) 18,677 ( 30.1) 0.4Maintenance& Evolutionmentee 1,696 ( 5.3) 3,028 ( 4.9) 0.4mentor 2,846 ( 7.9) 4,715 ( 7.6) 0.3Web & UI/UXmentee 2,256 ( 7.0) 4,203 ( 6.8) 0.2mentor 2,470 ( 6.8) 4,090 ( 6.6) 0.2SoftwareDesignmentee 587 ( 1.8) 992 ( 1.6) 0.2mentor 1,556 ( 4.3) 2,625 ( 4.2) 0.1ProjectPlanningmentee 921 ( 2.9) 1,671 ( 2.7) 0.2mentor 921 ( 2.5) 1,667 ( 2.7) 0.1Deploymentmentee 397 ( 1.2) 713 ( 1.2) 0.1mentor 989 ( 2.7) 1,763 ( 2.8) 0.1Quality &Testingmentee 2,186 ( 6.8) 4,176 ( 6.8) 0.1mentor 2,888 ( 8.0) 4,842 ( 7.8) 0.2Table 5.26: Counts of themes in improved and unimproved groups(CD({role, status}, {theme}) in parentheses).1025.5. Effectiveness of GSoC (RQ3 Results)The chi-square test of independence (Table 5.25) reveal significant differ-ences in the themes of discussion between improved and unimproved projectsfor both mentors (Chi2 ≈ 31,p-value = 0.0005) and mentees (Chi2 ≈383,p-value < 0.0001). This indicates that the themes discussed duringthe GSoC period are significantly associated with project success.For mentees, themes such as task completion, version control & col-laboration, and maintenance & evolution are more prevalent in unimprovedprojects compared to improved ones. Conversely, themes related to softwaredesign, data management, debugging & error, programming foundations,and project planning are more prominent in improved projects compared tounimproved ones. This suggests that mentees in successful projects might bemore focused on foundational programming and design aspects, which couldcontribute to better long-term outcomes. For mentors, improved projectssee a higher focus on themes like data management, task completion, andmaintenance and evolution, while unimproved projects have more emphasison version control & collaboration, and debugging & error.Themes that are more prevalent in improved projects for both mentorsand mentees include maintenance & evolution, data management, and soft-ware design. On the other hand, task completion is more emphasized bymentees in unimproved projects but is more prevalent for mentors in im-proved projects. Similarly, debugging & error is more common for menteesin improved projects but is more emphasized by mentors in unimprovedprojects. This indicates different areas of focus between mentors and menteesthat correlate with project outcomes.Comparison Over TimeThe results from the chi-square test indicated a statistically significant dif-ference in the themes and emotion expressions between the two project cat-egories. Although the difference in emotion was statistically significant, itsize of difference was negligible, suggesting minimal practical impact.Figure 5.27 presents the evolution of themes focus across the durationof the program, showing the distinctions between improved and unimprovedgroups. The plots show a general similarity in thematic focus over timebetween the two groups. However, notable differences exist, particularly inthe early weeks of the program.In the improved groups, there was a pronounced emphasis on technicalthemes such as Programming Foundations, and Quality & Testing. Thesethemes are critical for the technical development of projects and suggest anearly active approach to addressing project tasks technically. Conversely,1035.5. Effectiveness of GSoC (RQ3 Results)mentors and mentees in the unimproved groups were more engaged in dis-cussions with Version Control & Collaboration theme. This indicates afocus on setup and organizational aspects rather than technical aspects inthe initial stages.The thematic comparison highlights a strategic divergence in the initialfocus of project groups, with improved groups concentrating on technicalthemes from the outset, whereas unimproved groups prioritized preliminarytasks and setup. This early divergence in focus could be a point of inter-est in understanding the factors that contribute to the overall success ofprojects. The findings emphasize the importance of technical preparednessand problem-solving early in the project development.1045.5. Effectiveness of GSoC (RQ3 Results)ThemeRoleMentee Mentor3 9 13 190.150.273 9 13 190.240.33ProgrammingFoundations3 9 13 190.190.33 9 13 190.150.24Version Control &Collaboration3 9 13 190.080.163 9 13 190.110.16Data Management3 9 13 190.070.23 9 13 190.020.05Task Completion3 9 13 190.050.13 9 13 190.060.12Quality & Testing3 9 13 190.050.13 9 13 190.040.08Web & UI/UXLegendImprovedUnimprovedCoding StartsMid-term EvaluationTable 5.27: Weekly impact sparklines of top six discussed themes’ comparingimproved and unimproved projects over time. Interpret with care: 1. TheY-axis range is different from one plot to another. 2. The Y-axis does not includethe full range of possible values, which may visually exaggerate small differences.1055.5. Effectiveness of GSoC (RQ3 Results)Summary- The majority of GSoC projects did not see improvement in menteecontributions post-GSoC.- While sentiment scores between improved and unimproved projectsdid not differ significantly, we observed statistically significant dif-ferences in other measures. Unimproved projects had more com-ments from mentees, while improved projects had more commits frommentees and fewer from mentors.- The difference in themes of discussion were statistically significantbetween improved and unimproved; improved projects focused moreon programming foundations, while unimproved projects emphasizedversion control and collaboration.- A divergence in themes impact (Equation 5.19) was observed inearly weeks of the program, with improved projects concentrating ontechnical aspects and unimproved projects on preliminary tasks.- Despite a significant difference in emotion between improved andunimproved groups, the size of this difference was negligible.106Chapter 6DiscussionThis study evaluates communications between mentors and mentees in theGSoC programs from 2016 to 2023, focusing on sentiment analysis, emotionanalysis, and topic modeling. The insights are organized and presented inthis chapter aiming to provide valuable insights and actionable guidancefor future GSoC programs and similar mentoring programs. This chapterexplores the implications of these findings for future research and practicalapplications in mentoring.6.1 What Do Improved Projects Do Differently?Importance of Initial Engagement. Our analysis shows that the earlyweeks of the GSoC play an important role in the successful trajectory ofthe mentees and their projects. We found that in this phase, mentees andmentors in improved projects display a stronger emphasis on core program-ming and software skills, as opposed to those in less successful groups whoput more focus on discussing version control systems, such as pull requestmanagement and initial project setups on collaborative platforms. Theseobservations align with previous research that suggests contributors can be-come discouraged by the lengthy and complex initial processes of projectsetups [12, 52, 109]. Clear, early engagement and understanding of theproject are shown to be beneficial for maintaining mentee motivation andcontribution levels [92, 114], which is supported by our findings demonstrat-ing that a reduction in focus on initial setup tasks and an increase in earlydiscussions about programming foundations are related to overall contribu-tions improvement in mentees of the GSoC projects.Adaptability as the Project Progresses. As the program advances,we observed a shift in the mentees’ focus towards more complex, project-specific challenges, such as data management, starting around the fourthweek. This suggests a well-rounded developmental approach where initialfoundational skills are built upon with more advanced topics introducedlater in the program. This progression is consistent with recommendationsfor structured and established mentorship processes [114].1076.2. Role Differences and Milestone ReactionsChanges in Number of Commits. The analysis of commit patternsreveals an interesting role dynamics within the improved and unimprovedprojects. Mentees in more successful projects increasingly take on moreresponsibilities, as shown by their higher number of commits. Conversely,mentors in these groups tend to reduce their direct contributions over time,which indicates effective empowerment of the mentees and a transition to-ward mentee autonomy. This contrast in improved and unimproved projectswas especially notable after the midterm evaluations, where mentees inunimproved projects showed less contributions, possibly due to diminish-ing motivation.Emotion and Sentiment Dynamics. Despite statistically significantdifferences, the emotion and sentiment in communications between men-tors and mentees did not greatly vary between improved and unimprovedprojects. This observation shows that while the mentees in improved projectsshow higher motivation for contributing and committing, participants inboth groups have similar sentiment and emotion response to different mile-stones. This can be a hint that less engagement doesn’t mean less amountof stress or pressure in participants or vice versa.6.2 Role Differences and Milestone ReactionsStatistical analyses in RQ1 reveal significant differences in sentiment scoresand emotions expressed by mentors and mentees and how they react tovarious milestones in the program. For instance, mentors show increasedFear at the start of coding in week 3, and show an increased joy coupledwith a reduction in Fear following midterm evaluation deadline, suggestingrelief and satisfaction at this critical point. We observe increased sadnessin both roles as the midterm evaluation gets closer. Mentees have highermean sentiment score in the very first weeks of the program coupled withhigh joy levels, which could be indicative of their high levels of appreciation,and excitement to be part of the program. Understanding these emotiondynamics can help design support structures at various program phases.6.3 Week 13 as a Potential Critical PointWeek 13 emerged as a noteworthy period in our analysis, showing signifi-cant shifts in communication, commit patterns, and emotions. We noteda statistically significant change in the emotions of mentees from week 12to week 13, and a statistically significant shift in mentor sentiment scores1086.4. Recommendations for Enhancing Mentorship in GSoCin week 14. Additionally, the impact scores of various discussion themesalso change noticeable during weeks 12, 13, and 14. Concurrently, therewas a notable increase in the number of commits per project for menteesand mentors during weeks 12 and 13, respectively. Although this point intime is not recognized as an official milestone within the GSoC framework,the observed trends during this period warrant further exploration to fullyunderstand their implications on project dynamics.6.4 Recommendations for Enhancing Mentorshipin GSoC6.4.1 Program Structure and Phasing.Early Focus on Fundamentals We suggest an emphasize on core pro-gramming skills early in the program to better prepare mentees forthe technical aspects of the project. Working on the main projectright at the beginning of the program helps mentees get familiar withthe tasks and goals of their project and can reduce the likelihood ofthem becoming discouraged and uninterested about the project.Phased Project Development We recommend implementing awell-defined and structured mentorship approach for each project in-dividually. Initially, this should focus on improving individual skillsand covering simple, but technical aspects of the project. As partici-pants gain proficiency and get familiar with the project, the mentor-ship should then shift towards more advanced topics. Additionally, weadvise treating collaborative skills similarly to other skills—developingthem steadily over time rather than concentrating heavily on them atthe project’s outset. This balanced progression evolves with the com-plexities of the project over time.6.4.2 Mentee Engagement Strategies.Continuous Engagement We encourage mentees to consistently engagewith the project and make regular commits, to get more familiar withthe project over time.Utilize Mentor Expertise We suggest that mentees try to fully utilizetheir mentors’ expertise and knowledge, particularly in areas they findchallenging. The rationale for this recommendation stems from obser-vations that improved projects tend to emphasize critical areas such as1096.5. Threats to Validitydebugging, and error handlin more consistently throughout the pro-gram compared to the unimproved ones. Actively seeking guidanceand resources can enrich the learning experience of mentees and im-prove their overall project outcomes.6.4.3 Mentor Engagement Strategies.Reduction of Direct Involvement Mentors should gradually reduce theirdirect coding contributions to promote mentee autonomy.Responsive Mentorship Programs should provide support based on thechanging needs of mentees. We recommend a distributed learning ap-proach where various skills are integrated throughout the duration ofthe program rather than concentrating on one skill at a time. By do-ing so, mentees can immediately apply new knowledge, maintainingtheir engagement and reducing the likelihood of boredom or discour-agement. This recommendation is based on the observed themes ofdiscussions in improved projects, where we can see a more distributedfocus on skills over time compared to unimproved projects. Previousstudies also suggest focusing on the mentee’s needs in designing tasksand challenges in mentorship [6, 114].Being Open to Answering Questions To help mentees focus on codingchallenges and improve, mentors should be welcoming and supportive.Although not directly identified in our study, previous researches haveidentified various barriers that prevent mentees from seeking necessaryassistance. These include feelings of low self-efficacy [5, 6, 108, 111,112], and the frustration of their efforts going unrecognized [52, 54, 92].By implementing these recommendations and thoroughly reviewing thedetailed charts and analyses provided in this study, GSoC can potentiallyenhance its mentorship model, leading to better preparedness and successrates of mentees, thus strengthening the overall impact of the program.6.5 Threats to ValidityGSoC has a unique environment and structure that may not align with othermentorship programs or other forms of mentorship in OSS. This specificitymeans that GSoC participants might not reflect the wider population ofprogrammers or students, and the mentorship and project selection methodsused in GSoC may not be relevant to other programs or settings. Therefore,1106.5. Threats to Validityour findings are specific to GSoC, although the metrics we developed couldpotentially be adapted for different settings.For each research question, we established and quantified various con-structs. To address construct validity threats, we utilized statistical testsand, where feasible, validated them by a second test.A potential validity threat identified during data collection was the accu-racy of matching GSoC participants’ names from the program’s website totheir GitHub profiles. To mitigate this, we employed a Levenshtein distancesimilarity score with a minimum threshold of 70%, a method previouslyutilized in similar research [9]. Our study only analyzes the GitHub inter-actions, not covering other forms of communication between mentors andmentees, such as video calls or direct messages, which may also influenceoutcomes.There could be various reasons why mentees stop contributing to projectson GitHub immediately after their GSoC program ends, such as time con-straints, shifts in career focus, and change in interests that our study doesnot capture. In the future, we plan to study unimproved projects to deter-mine why there is a higher number of such projects from mentees.In our analysis, we reported observations and tested for associationsbetween various measures and outcomes—for example, the relationship be-tween participant roles and the sentiment of their comments, or the con-nection between project success and factors like themes, or number of com-ments. It is crucial to note that identifying these relationships does notimply causality. For instance, while we observed differing sentiments amongroles, we cannot conclusively assert that the differences in sentiment arecaused by the roles themselves. To elucidate, consider the classic exam-ple of ice cream sales and shark attacks. Both are higher in the summermonths, thus displaying a correlation. However, this does not mean that icecream sales cause shark attacks or vice versa; instead, both are related toa third factor—warmer weather. Similarly, in our study, while correlationsexist between certain variables, they do not necessarily indicate a directcausal relationship. Further research would be required to explore causativemechanisms.111Chapter 7ConclusionIn this thesis, we analyzed the mentor-mentee interactions within GoogleSummer of Code (GSoC), supported by our curated dataset, OpenMent,which details GitHub interactions during the program. Our research pro-vided a nuanced view of mentorship’s role in OSS development, emphasizingsentiment and emotion, communication topics, and mentee contributions.The OpenMent dataset is a significant contribution of our work, provid-ing a basis for detailed analysis of OSS mentorship. Our methodology, com-bining sentiment and emotion analysis with topic modeling, coupled with anormalized timeframe that treat different years similarly offer a comprehen-sive approach for studying interpersonal communications within mentorshipcontexts.We identify statistically significant variations in sentiments and emo-tions between mentors and mentees, with mentees generally showing slightlyhigher sentiment scores. Mentors displayed more anger and surprise, whilementees more frequently expressed sadness. The analysis of communicationtopics revealed that Programming Foundations was predominant during ac-tive coding weeks. A high rate of reduced OSS engagement post-GSoCwas noted, with over half of the projects’ mentees having reduced contri-butions on GitHub post-GSoC. Successful projects were characterized by ahigher average number of mentee commits, lower average number of men-tor commits, and more focus on fundamental programming topics early inthe project. Less successful projects were more focused on topics relatedto collaborative environments, and issue assignment early in the project.The early emphasis on critical programming skills can be helpful for menteeintegration and project success, and too much focus on project setup andprelimenery tasks early in the project can cause demotivation in mentees.For OSS projects and programs like GSoC, adopting structured mentor-ship strategies that evolve over time and focus on critical early skills couldenhance long-term mentee engagement and project success. Our findingsdemonstrate the critical impact of structured mentorship in OSS environ-ments, suggesting pathways for enhancing project outcomes and potentiallythe long-term sustainability of the OSS ecosystem.112Chapter 7. ConclusionSustaining mentee engagement post-GSoC remains as a key challenge forfuture mentorship programs. There is a need for more research exploringthe causes of mentee disengagement post-GSoC and the influence of differentmentoring approaches on long-term mentee contributions. This thesis is justa start to textual analysis of mentorship communications. Future researchcan study additional measures that have the potential of being related tomentorship success including but not limited to the amount of time it takesfor issues to close, the responsiveness of mentors in answering questions, andthe quality or complexity of mentees’ contributions over time.Another important area for future research is examining the impact ofthe COVID-19 pandemic and related industry layoffs on the GSoC program.This could include analyzing how these events affected participant numbers,sentiment in communications, and long-term engagement in open source.A comparative study of pre-pandemic, pandemic, and post-pandemic yearscould provide valuable insights into how this global disruption influencedopen source mentorship programs and contribute to more resilient programdesign in the future.113Bibliography[1] Md Ahasanuzzaman, Muhammad Asaduzzaman, Chanchal K Roy,and Kevin A Schneider. Caps: a supervised technique for classify-ing stack overflow posts concerning api issues. Empirical SoftwareEngineering, 25:1493–1532, 2020.[2] Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and ShahramRahimi. Senticr: A customized sentiment analysis tool for code reviewinteractions. In 2017 32nd IEEE/ACM International Conference onAutomated Software Engineering (ASE), pages 106–111. IEEE, 2017.[3] Olivia Angiuli, Joe Blitzstein, and Jim Waldo. How to de-identifyyour data. Commun. ACM, 58(12):48–55, nov 2015. ISSN 0001-0782.doi: 10.1145/2814340. URL https://doi.org/10.1145/2814340.[4] Haoli Bai, Zhuangbin Chen, Michael R Lyu, Irwin King, and ZenglinXu. Neural relational topic models for scientific article analysis. InProceedings of the 27th ACM International Conference on Informationand Knowledge Management, pages 27–36, 2018.[5] Sogol Balali, Igor Steinmacher, Umayal Annamalai, Anita Sarma, andMarco Aurelio Gerosa. Newcomers’ barriers... is that all? an anal-ysis of mentors’ and newcomers’ barriers in oss projects. ComputerSupported Cooperative Work (CSCW), 27:679–714, 2018.[6] Sogol Balali, Umayal Annamalai, Hema Susmita Padala, BiancaTrinkenreich, Marco A Gerosa, Igor Steinmacher, and Anita Sarma.Recommending tasks to newcomers in oss projects: How do mentorshandle it? In Proceedings of the 16th International Symposium onOpen Collaboration, pages 1–14, 2020.[7] Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, and LuisEspinosa-Anke. Tweeteval: Unified benchmark and comparative eval-uation for tweet classification. arXiv preprint arXiv:2010.12421, 2020.114Bibliography[8] Anton Barua, Stephen W Thomas, and Ahmed E Hassan. What aredevelopers talking about? an analysis of topics and trends in stackoverflow. Empirical software engineering, 19:619–654, 2014.[9] Christian Bird, Alex Gourley, Prem Devanbu, Michael Gertz, andAnand Swaminathan. Mining email social networks. In Proceedingsof the 2006 international workshop on Mining software repositories,pages 137–143, 2006.[10] David M Blei and John D Lafferty. Dynamic topic models. InProceedings of the 23rd international conference on Machine learning,pages 113–120, 2006.[11] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Sim-ran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg,Antoine Bosselut, Emma Brunskill, et al. On the opportunities andrisks of foundation models. arXiv preprint arXiv:2108.07258, 2021.[12] Amiangshu Bosu and Kazi Zakia Sultana. Diversity and inclusionin open source software (oss) projects: Where do we stand? In2019 ACM/IEEE International Symposium on Empirical SoftwareEngineering and Measurement (ESEM), pages 1–11. IEEE, 2019.[13] Jordan Boyd-Graber, Yuening Hu, David Mimno, et al. Applicationsof topic models. Foundations and Trends® in Information Retrieval,11(2-3):143–296, 2017.[14] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared DKaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam,Girish Sastry, Amanda Askell, et al. Language models are few-shotlearners. Advances in neural information processing systems, 33:1877–1901, 2020.[15] Fabio Calefato, Filippo Lanubile, Federico Maiorano, and NicoleNovielli. Sentiment polarity detection for software development.In Proceedings of the 40th International Conference on SoftwareEngineering, pages 128–128, 2018.[16] Gerardo Canfora, Massimiliano Di Penta, Rocco Oliveto, and Sebas-tiano Panichella. Who is going to mentor newcomers in open sourceprojects? In Proceedings of the ACM SIGSOFT 20th internationalsymposium on the foundations of software engineering, pages 1–11,2012.115Bibliography[17] Kevin Carillo, Sid Huff, and Brenda Chawner. What makes agood contributor? understanding contributor behavior within largefree/open source software projects–a socialization perspective. TheJournal of Strategic Information Systems, 26(4):322–359, 2017.[18] Oihane Cereceda and Danielle EA Quinn. A graduate student perspec-tive on overcoming barriers to interacting with open-source software.FACETS, 5(1):289–303, 2020.[19] Scott Chacon and Ben Straub. Pro git. Springer Nature, 2014.[20] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, HenriquePonde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda,Nicholas Joseph, Greg Brockman, et al. Evaluating large languagemodels trained on code. arXiv preprint arXiv:2107.03374, 2021.[21] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, MaartenBosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung WonChung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scal-ing language modeling with pathways. Journal of Machine LearningResearch, 24(240):1–113, 2023.[22] Rob Churchill and Lisa Singh. The evolution of topic modeling. ACMComputing Surveys, 54(10s):1–35, 2022.[23] Zadia Codabux, Melina Vidoni, and Fatemeh H. Fard. Technical debtin the peer-review documentation of r packages: a ropensci case study.In 2021 IEEE/ACM 18th International Conference on Mining SoftwareRepositories (MSR), pages 195–206, 2021. doi: 10.1109/MSR52588.2021.00032.[24] Jon P Daries, Justin Reich, Jim Waldo, Elise M Young, JonathanWhittinghill, Andrew Dean Ho, Daniel Thomas Seaton, and IsaacChuang. Privacy, anonymity, and big data in the social sciences.Communications of the ACM, 57(9):56–63, 2014.[25] A. C. Davison and D. V. Hinkley. Bootstrap Methods and theirApplication. Cambridge Series in Statistical and Probabilistic Math-ematics. Cambridge University Press, 1997.[26] Xiang Deng, Vasilisa Bashlovkina, Feng Han, Simon Baumgartner,and Michael Bendersky. Llms to the moon? reddit market sentimentanalysis with large language models. In Companion Proceedings ofthe ACM Web Conference 2023, pages 1014–1019, 2023.116Bibliography[27] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.BERT: pre-training of deep bidirectional transformers for languageunderstanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805.[28] Giovanni Di Franco and Michele Santurro. Machine learning, artificialneural networks and social research. Quality & quantity, 55(3):1007–1025, 2021.[29] Themistoklis Diamantopoulos, Dimitrios-Nikitas Nastos, and An-dreas Symeonidis. Semantically-enriched jira issue tracking data. In2023 IEEE/ACM 20th International Conference on Mining SoftwareRepositories (MSR), pages 218–222. IEEE, 2023.[30] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling inembedding spaces. Transactions of the Association for ComputationalLinguistics, 8:439–453, 2020.[31] Anton Digno¨s, Michael H. Bo¨hlen, and Johann Gamper. Temporalalignment. In Proceedings of the 2012 ACM SIGMOD InternationalConference on Management of Data, SIGMOD ’12, page 433–444, NewYork, NY, USA, 2012. Association for Computing Machinery. ISBN9781450312479. doi: 10.1145/2213836.2213886. URL https://doi.org/10.1145/2213836.2213886.[32] Jesse Dodge, Maarten Sap, Ana Marasovic´, William Agnew, GabrielIlharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Doc-umenting large webtext corpora: A case study on the colossal cleancrawled corpus. arXiv preprint arXiv:2104.08758, 2021.[33] Cuc Duong, Qian Liu, Rui Mao, and Erik Cambria. Saving earthone tweet at a time through the lens of artificial intelligence. In 2022International Joint Conference on Neural Networks (IJCNN), pages1–9. IEEE, 2022.[34] Bradley Efron. Bootstrap methods: another look at the jackknife.In Breakthroughs in statistics: Methodology and distribution, pages569–593. Springer, 1992.[35] Nadia Eghbal. Roads and bridges: The unseen labor behind our digitalinfrastructure. Ford Foundation, 2016.117Bibliography[36] Fabian Fagerholm, Alejandro S Guinea, Ju¨rgen Mu¨nch, and JayBorenstein. The role of mentoring and project characteristics foronboarding in open source software projects. In Proceedings ofthe 8th ACM/IEEE international symposium on empirical softwareengineering and measurement, pages 1–10, 2014.[37] Fabian Fagerholm, Alejandro Sanchez Guinea, Jay Borenstein, andJu¨rgen Mu¨nch. Onboarding in open source projects. IEEE Software,31(6):54–61, 2014.[38] Joseph Feller and Brian Fitzgerald. Understanding open sourcesoftware development. Addison-Wesley Longman Publishing Co., Inc.,2002.[39] Zixuan Feng. The state of survival in oss: The impact of diver-sity. In Proceedings of the 31st ACM Joint European SoftwareEngineering Conference and Symposium on the Foundations ofSoftware Engineering, pages 2213–2215, 2023.[40] Zixuan Feng, Amreeta Chatterjee, Anita Sarma, and Iftekhar Ahmed.A case study of implicit mentoring, its prevalence, and impact inapache. In Proceedings of the 30th ACM Joint European SoftwareEngineering Conference and Symposium on the Foundations ofSoftware Engineering, pages 797–809, 2022.[41] Zixuan Feng, Amreeta Chatterjee, Anita Sarma, and Iftekhar Ahmed.Implicit mentoring: The unacknowledged developer efforts in opensource. arXiv preprint arXiv:2202.11300, 2022.[42] Evelyn Fix. Discriminatory analysis: nonparametric discrimination,consistency properties, volume 1. USAF school of Aviation Medicine,1985.[43] Felipe Fronchetti, Igor Wiese, Gustavo Pinto, and Igor Steinmacher.What attracts newcomers to onboard on oss projects? tl; dr: Popu-larity. In Open Source Systems: 15th IFIP WG 2.13 InternationalConference, OSS 2019, Montreal, QC, Canada, May 26–27, 2019,Proceedings 15, pages 91–103. Springer, 2019.[44] James Gareth, Witten Daniela, Hastie Trevor, and Tibshirani Robert.An introduction to statistical learning: with applications in R. Sp-inger, 2013.118Bibliography[45] Praveen Gauravaram. Security analysis of salt—— password hashes.In 2012 International Conference on Advanced Computer ScienceApplications and Technologies (ACSAT), pages 25–30. IEEE, 2012.[46] Andreas F Gkontzis, Christoforos V Karachristos, Chris T Panagio-takopoulos, Elias C Stavropoulos, and Vassilios S Verykios. Sentimentanalysis to track emotion and polarity in student fora. In Proceedingsof the 21st Pan-Hellenic Conference on Informatics, pages 1–6, 2017.[47] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, 2019. URLhttp://Skylion007.github.io/OpenWebTextCorpus.[48] Georgios Gousios and Diomidis Spinellis. Ghtorrent: Github’s datafrom a firehose. In 2012 9th IEEE Working Conference on MiningSoftware Repositories (MSR), pages 12–21. IEEE, 2012.[49] Thomas L Griffiths, Mark Steyvers, and Joshua B Tenenbaum. Topicsin semantic representation. Psychological review, 114(2):211, 2007.[50] Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794, 2022.[51] Haiqiao Gu, Hao He, and Minghui Zhou. Self-admitted library migra-tions in java, javascript, and python packaging ecosystems: A com-parative study. In 2023 IEEE International Conference on SoftwareAnalysis, Evolution and Reengineering (SANER), pages 627–638.IEEE, 2023.[52] Mariam Guizani, Amreeta Chatterjee, Bianca Trinkenreich,Mary Evelyn May, Geraldine J Noa-Guevara, Liam James Rus-sell, Griselda G Cuevas Zambrano, Daniel Izquierdo-Cortazar, IgorSteinmacher, Marco A Gerosa, et al. The long road ahead: Ongoingchallenges in contributing to large oss organizations and what todo. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–30, 2021.[53] Emitza Guzman, Rana Alkadhi, and Norbert Seyff. An exploratorystudy of twitter messages about software applications. RequirementsEngineering, 22:387–412, 2017.[54] Christoph Hannebauer and Volker Gruhn. On the relationship be-tween newcomer motivations and contribution barriers in open sourceprojects. In Proceedings of the 13th International Symposium on OpenCollaboration, pages 1–10, 2017.119Bibliography[55] Nicholas Harris, Anand Butani, and Syed Hashmy. Enhancing embed-ding performance through large language model-based text enrichmentand rewriting. arXiv preprint arXiv:2404.12283, 2024.[56] Susanne Hempel, Rita V Burke, Michael Hochman, Gina Thomp-son, Annie Brothers, Jennifer Shin, Aneesa Motala, Jody Larkin, andJeanne Ringel. Resource allocation and pandemic response: An evi-dence synthesis to inform decision making, 2020.[57] Alexander Hoyle, Pranav Goel, Rupak Sarkar, and Philip Resnik. Areneural topic models broken? arXiv preprint arXiv:2210.16162, 2022.[58] Minqing Hu and Bing Liu. Mining and summarizing customer reviews.In Proceedings of the tenth ACM SIGKDD international conferenceon Knowledge discovery and data mining, pages 168–177, 2004.[59] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, andMarc Brockschmidt. Codesearchnet challenge: Evaluating the state ofsemantic code search. arXiv preprint arXiv:1909.09436, 2019.[60] Rob J Hyndman and George Athanasopoulos. Forecasting: principlesand practice. OTexts, 2018.[61] Md Rakibul Islam and Minhaz F Zibran. Sentiment analysis of soft-ware bug related commit messages. Network, 740(740):88, 2018.[62] Md Rakibul Islam and Minhaz F Zibran. Sentistrength-se: Exploitingdomain specificity for improved sentiment analysis in software engi-neering text. Journal of Systems and Software, 145:125–146, 2018.[63] Rene´ Just, Darioush Jalali, and Michael D Ernst. Defects4j: Adatabase of existing faults to enable controlled testing studies for javaprograms. In Proceedings of the 2014 international symposium onsoftware testing and analysis, pages 437–440, 2014.[64] Hossein Keshavarz and Gema Rodr´ıguez-Pe´rez. Jitgnn: A deep graphneural network framework for just-in-time bug prediction. Journal ofSystems and Software, 210:111984, 2024.[65] Phillip Keung, Yichao Lu, Gyo¨rgy Szarvas, and Noah A Smith. Themultilingual amazon reviews corpus. arXiv preprint arXiv:2010.02573,2020.120Bibliography[66] Caitlin Doogan Poet Laureate, Wray Buntine, and Henry Linger. Asystematic review of the use of topic models for short text social mediaanalysis. Artificial Intelligence Review, 56(12):14223–14255, 2023.[67] Amanda Lee, Jeffrey C Carver, and Amiangshu Bosu. Understandingthe impressions, motivations, and barriers of one time code contribu-tors to floss projects: a survey. In 2017 IEEE/ACM 39th InternationalConference on Software Engineering (ICSE), pages 187–197. IEEE,2017.[68] Minchul Lee. bab2min/tomotopy: 0.12.3, July 2022. URL https://doi.org/10.5281/zenodo.6868418.[69] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and ShuziNiu. Dailydialog: A manually labelled multi-turn dialogue dataset.arXiv preprint arXiv:1710.03957, 2017.[70] Pasquale Lisena, Ismail Harrando, Oussama Kandakji, and RaphaelTroncy. Tomodapi: A topic modeling api to train, use and comparetopic models. In Proceedings of second workshop for NLP open sourcesoftware (NLP-OSS), pages 132–140, 2020.[71] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, DanqiChen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoy-anov. Roberta: A robustly optimized bert pretraining approach. arXivpreprint arXiv:1907.11692, 2019.[72] Thomas Lumley, Paula Diehr, Scott Emerson, and Lu Chen. Theimportance of the normality assumption in large public health datasets. Annual review of public health, 23(1):151–169, 2002.[73] Wei Chen Maggie, Phil Culliton. Tweet sentiment ex-traction, 2020. URL https://kaggle.com/competitions/tweet-sentiment-extraction.[74] Vadim Markovtsev and Waren Long. Public git archive: a big codedataset for all. In Proceedings of the 15th International Conferenceon Mining Software Repositories, pages 34–37, 2018.[75] Andrew Kachites McCallum. Mallet: A machine learning for lan-guagetoolkit. http://mallet. cs. umass. edu, 2002.121Bibliography[76] Leland McInnes, John Healy, and Steve Astels. hdbscan: Hierarchicaldensity based clustering. Journal of Open Source Software, 2(11):205,2017. doi: 10.21105/joss.00205. URL https://doi.org/10.21105/joss.00205.[77] Christopher Mendez, Hema Susmita Padala, Zoe Steine-Hanson,Claudia Hilderbrand, Amber Horvath, Charles Hill, Logan Simpson,Nupoor Patil, Anita Sarma, and Margaret Burnett. Open source bar-riers to entry, revisited: A sociotechnical perspective. In Proceedingsof the 40th International conference on software engineering, pages1004–1015, 2018.[78] Tom Mens, Maelick Claes, and Philippe Grosjean. Ecos: Ecologicalstudies of open source software ecosystems. In 2014 Software EvolutionWeek-IEEE Conference on Software Maintenance, Reengineering, andReverse Engineering (CSMR-WCRE), pages 403–406. IEEE, 2014.[79] Niklas Muennighoff, Nouamane Tazi, Lo¨ıc Magne, and Nils Reimers.Mteb: Massive text embedding benchmark. arXiv preprintarXiv:2210.07316, 2022.[80] Alessandro Murgia, Parastou Tourani, Bram Adams, and Marco Ortu.Do developers feel emotions? an exploratory analysis of emotions insoftware artifacts. In Proceedings of the 11th working conference onmining software repositories, pages 262–271, 2014.[81] Alessandro Murgia, Marco Ortu, Parastou Tourani, Bram Adams,and Serge Demeyer. An exploratory qualitative and quantitativeanalysis of emotions in issue report comments of open source sys-tems. Empirical Software Engineering, 23(1):521–564, February 2018.ISSN 1573-7616. doi: 10.1007/s10664-017-9526-0. URL https://doi.org/10.1007/s10664-017-9526-0.[82] Nicole Novielli, Fabio Calefato, Davide Dongiovanni, Daniela Girardi,and Filippo Lanubile. Can we use se-specific sentiment analysis toolsin a cross-platform setting? In Proceedings of the 17th InternationalConference on Mining Software Repositories, pages 158–168, 2020.[83] Nicole Novielli, Fabio Calefato, Davide Dongiovanni, Daniela Girardi,and Filippo Lanubile. A gold standard for polarity of emotions ofsoftware developers in github. In Proceedings of the 17th InternationalConference on Mining Software Repositories, volume 3, 2020.122Bibliography[84] OpenAI. Chatgpt (mar 3 version) [large language model]. https://chat.openai.com/chat, 2024.[85] Marco Ortu, Bram Adams, Giuseppe Destefanis, Parastou Tourani,Michele Marchesi, and Roberto Tonelli. Are bullies more produc-tive? empirical study of affectiveness vs. issue fixing time. In2015 IEEE/ACM 12th Working Conference on Mining SoftwareRepositories, pages 303–313, 2015. doi: 10.1109/MSR.2015.35.[86] Marco Ortu, Alessandro Murgia, Giuseppe Destefanis, ParastouTourani, Roberto Tonelli, Michele Marchesi, and Bram Adams. Theemotional side of software developers in jira. In Proceedings of the 13thInternational Conference on Mining Software Repositories, MSR ’16,page 480–483, New York, NY, USA, 2016. Association for Comput-ing Machinery. ISBN 9781450341868. doi: 10.1145/2901739.2903505.URL https://doi.org/10.1145/2901739.2903505.[87] Rajvardhan Patil, Sorio Boit, Venkat Gudivada, and JagadeeshNandigam. A survey of text representation and embedding techniquesin nlp. IEEE Access, 2023.[88] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion,O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Van-derplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, andE. Duchesnay. Scikit-learn: Machine learning in Python. Journalof Machine Learning Research, 12:2825–2830, 2011.[89] Heather Pon-Barry, Becky Wai-Ling Packard, and Audrey St. John.Expanding capacity and promoting inclusion in introductory computerscience: a focus on near-peer mentor preparation and code review.Computer Science Education, 27(1):54–77, 2017.[90] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and RadaMihalcea. Beneath the tip of the iceberg: Current challenges andnew directions in sentiment analysis research. IEEE transactions onaffective computing, 14(1):108–132, 2020.[91] Chanathip Pornprasit and Chakkrit Kla Tantithamthavorn. Jitline: Asimpler, better, faster, finer-grained just-in-time defect prediction. In2021 IEEE/ACM 18th International Conference on Mining SoftwareRepositories (MSR), pages 369–379. IEEE, 2021.123Bibliography[92] Gede Artha Azriadi Prana, Denae Ford, Ayushi Rastogi, David Lo,Rahul Purandare, and Nachiappan Nagappan. Including everyone, ev-erywhere: Understanding opportunities and challenges of geographicgender-inclusion in oss. IEEE Transactions on Software Engineering,48(9):3394–3409, 2021.[93] Huilian Sophie Qiu, Alexander Nolte, Anita Brown, Alexander Sere-brenik, and Bogdan Vasilescu. Going farther together: The impactof social capital on sustained participation in open source. In 2019ieee/acm 41st international conference on software engineering (icse),pages 688–699. IEEE, 2019.[94] Huilian Sophie Qiu, Zihe H Zhao, Tielin Katy Yu, Justin Wang,Alexander Ma, Hongbo Fang, Laura Dabbish, and Bogdan Vasilescu.Gender representation among contributors to open-source infrastruc-ture: An analysis of 20 package manager ecosystems. In 2023IEEE/ACM 45th International Conference on Software Engineering:Software Engineering in Society (ICSE-SEIS), pages 180–187. IEEE,2023.[95] Rehurek Radim and Petr Sojka. Gensim-statistical semantics inpython. statistical semantics: gensim: Python: LDA: SVD, 2011.[96] Ifraz Rehman, Dong Wang, Raula Gaikovina Kula, Takashi Ishio,and Kenichi Matsumoto. Newcomer oss-candidates: Characteriz-ing contributions of novice developers to github. Empirical SoftwareEngineering, 27(5):109, 2022.[97] Lorenzo A. Rossi and Omprakash Gnawali. Language IndependentAnalysis and Classification of Discussion Threads in Coursera MOOCForums. In Proceedings of the IEEE International Conference onInformation Reuse and Integration (IRI 2014), August 2014.[98] Kashfia Sailunaz, Manmeet Dhaliwal, Jon Rokne, and Reda Alhajj.Emotion detection from text and speech: a survey. Social NetworkAnalysis and Mining, 8(1):28, 2018.[99] Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, andYi-Shin Chen. CARER: Contextualized affect representations for emo-tion recognition. In Ellen Riloff, David Chiang, Julia Hockenmaier,and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference onEmpirical Methods in Natural Language Processing, pages 3687–3697,124BibliographyBrussels, Belgium, October-November 2018. Association for Com-putational Linguistics. doi: 10.18653/v1/D18-1404. URL https://aclanthology.org/D18-1404.[100] Anna Schmidt and Michael Wiegand. A survey on hate speech de-tection using natural language processing. In Proceedings of thefifth international workshop on natural language processing for socialmedia, pages 1–10, 2017.[101] Paul Sebo. Are accuracy parameters useful for improving the perfor-mance of gender detection tools? a comparative study with westernand chinese names. Journal of general internal medicine, 37(15):4024–4027, 2022.[102] Jefferson Silva, Igor Wiese, Daniel M German, Christoph Treude,Marco Aure´lio Gerosa, and Igor Steinmacher. A theory of the en-gagement in open source projects via summer of code programs. InProceedings of the 28th ACM Joint Meeting on European SoftwareEngineering Conference and Symposium on the Foundations ofSoftware Engineering, pages 421–431, 2020.[103] Jefferson De Oliveira Silva, Igor Scaliante Wiese, Daniel M German,Igor Fabio Steinmacher, and Marco Aure´lio Gerosa. How long andhow much: What to expect from summer of code participants? In2017 IEEE International Conference on Software Maintenance andEvolution (ICSME), pages 69–79. IEEE, 2017.[104] Jefferson O Silva, Igor Wiese, Daniel M German, Christoph Treude,Marco A Gerosa, and Igor Steinmacher. Google summer of code: Stu-dent motivations and contributions. Journal of Systems and Software,162:110487, 2020.[105] Judith D Singer and John B Willett. Applied longitudinal dataanalysis: Modeling change and event occurrence. Oxford universitypress, 2003.[106] Vandana Singh. Newcomer integration and learning in technical sup-port communities for open source software. In Proceedings of the 2012ACM International Conference on Supporting Group Work, pages 65–74, 2012.[107] Akash Srivastava and Charles Sutton. Autoencoding variational infer-ence for topic models. arXiv preprint arXiv:1703.01488, 2017.125Bibliography[108] Christoph Stanik, Lloyd Montgomery, Daniel Martens, Davide Fucci,andWalid Maalej. A simple nlp-based approach to support onboardingand retention in open source communities. In 2018 IEEE internationalconference on software maintenance and evolution (ICSME), pages172–182. IEEE, 2018.[109] Igor Steinmacher, Ana Paula Chaves, Tayana Uchoa Conte, andMarco Aure´lio Gerosa. Preliminary empirical identification of bar-riers faced by newcomers to open source software projects. In 2014Brazilian Symposium on Software Engineering, pages 51–60. IEEE,2014.[110] Igor Steinmacher, Marco Aure´lio Graciotto Silva, and Marco Aure´lioGerosa. Barriers faced by newcomers to open source projects: asystematic review. In Open Source Software: Mobile Open SourceTechnologies: 10th IFIP WG 2.13 International Conference on OpenSource Systems, OSS 2014, San Jose´, Costa Rica, May 6-9, 2014.Proceedings 10, pages 153–163. Springer, 2014.[111] Igor Steinmacher, Tayana Conte, Marco Aure´lio Gerosa, and DavidRedmiles. Social barriers faced by newcomers placing their first con-tribution in open source software projects. In Proceedings of the 18thACM conference on Computer supported cooperative work & socialcomputing, pages 1379–1392, 2015.[112] Igor Steinmacher, Tayana Uchoˆa Conte, and Marco Aure´lio Gerosa.Understanding and supporting the choice of an appropriate task tostart with in open source software communities. In 2015 48th HawaiiInternational Conference on System Sciences, pages 5299–5308. IEEE,2015.[113] Igor Steinmacher, Christoph Treude, and Marco Aurelio Gerosa. Letme in: Guidelines for the successful onboarding of newcomers to opensource projects. IEEE Software, 36(4):41–49, 2018.[114] Igor Steinmacher, Sogol Balali, Bianca Trinkenreich, Mariam Guizani,Daniel Izquierdo-Cortazar, Griselda G Cuevas Zambrano, Marco Au-relio Gerosa, and Anita Sarma. Being a mentor in open source projects.Journal of Internet Services and Applications, 12:1–33, 2021.[115] Jeffrey Svajlenko and Chanchal K Roy. Bigclonebench. Code CloneAnalysis: Research, Tools, and Practices, pages 93–105, 2021.126Bibliography[116] Chade-Meng Tan, Yuan-Fang Wang, and Chan-Do Lee. The use ofbigrams to enhance text categorization. Information processing &management, 38(4):529–546, 2002.[117] Xin Tan, Minghui Zhou, and Zeyu Sun. A first look at good firstissues on github. In Proceedings of the 28th ACM Joint Meeting onEuropean Software Engineering Conference and Symposium on theFoundations of Software Engineering, pages 398–409, 2020.[118] Xin Tan, Minghui Zhou, and Li Zhang. Understanding mentors’engagement in oss communities via google summer of code. IEEETransactions on Software Engineering, 49(5):3106–3130, 2023.[119] Lili Tao, Alexandru-Petre Cazan, Senad Ibraimoski, and Sean Moran.Code librarian: A software package recommendation system. In 2023IEEE/ACM 45th International Conference on Software Engineering:Software Engineering in Practice (ICSE-SEIP), pages 196–198. IEEE,2023.[120] Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, PietroTropeano, and Antonio Candelieri. Octis: Comparing and optimizingtopic models is simple! In Proceedings of the 16th Conference of theEuropean Chapter of the Association for Computational Linguistics:System Demonstrations, pages 263–270, 2021.[121] Robert L Thorndike. Who belongs in the family? Psychometrika, 18(4):267–276, 1953.[122] Robert J Tibshirani and Bradley Efron. An introduction to the boot-strap. Monographs on statistics and applied probability, 57(1):1–436,1993.[123] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet,Marie-Anne Lachaux, Timothe´e Lacroix, Baptiste Rozie`re, NamanGoyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficientfoundation language models. arXiv preprint arXiv:2302.13971, 2023.[124] Erik H Trainer, Chalalai Chaihirunkarn, Arun Kalyanasundaram,and James D Herbsleb. Community code engagements: summer ofcode & hackathons for community building in scientific software. InProceedings of the 2014 ACM International Conference on SupportingGroup Work, pages 111–121, 2014.127Bibliography[125] Bianca Trinkenreich, Mariam Guizani, Igor Wiese, Anita Sarma, andIgor Steinmacher. Hidden figures: Roles and pathways of success-ful oss contributors. Proceedings of the ACM on human-computerinteraction, 4(CSCW2):1–22, 2020.[126] Peter D Turney. Thumbs up or thumbs down? semantic orienta-tion applied to unsupervised classification of reviews. arXiv preprintcs/0212032, 2002.[127] K. Jyothi Upadhya, B. Dinesh Rao, and M. Geetha. Discovery ofpopular languages from github repository: A data mining approach. InV. Sivakumar Reddy, V. Kamakshi Prasad, Jiacun Wang, and K.T.V.Reddy, editors, Soft Computing and Signal Processing, pages 165–173,Singapore, 2022. Springer Nature Singapore. ISBN 978-981-16-7088-6.[128] Thea F Van de Mortel. Faking it: social desirability response bias inself-report research. Australian Journal of Advanced Nursing, The, 25(4):40–48, 2008.[129] Bogdan Vasilescu, Vladimir Filkov, and Alexander Serebrenik. Per-ceptions of diversity on git hub: A user survey. In 2015 IEEE/ACM8th International Workshop on Cooperative and Human Aspects ofSoftware Engineering, pages 50–56. IEEE, 2015.[130] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland,Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson,Warren Weckesser, Jonathan Bright, Ste´fan J. van der Walt, MatthewBrett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, AndrewR. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, I˙lhanPolat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, JosefPerktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R.Harris, Anne M. Archibald, Antoˆnio H. Ribeiro, Fabian Pedregosa,Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamen-tal Algorithms for Scientific Computing in Python. Nature Methods,17:261–272, 2020. doi: 10.1038/s41592-019-0686-2.[131] Chong Wang and David M Blei. Collaborative topic modeling forrecommending scientific articles. In Proceedings of the 17th ACMSIGKDD international conference on Knowledge discovery and datamining, pages 448–456, 2011.128Bibliography[132] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Ma-jumder, and FuruWei. Improving text embeddings with large languagemodels. arXiv preprint arXiv:2401.00368, 2023.[133] Zengzhi Wang, Qiming Xie, Yi Feng, Zixiang Ding, Zinong Yang, andRui Xia. Is chatgpt a good sentiment analyzer? a preliminary study.arXiv preprint arXiv:2304.04339, 2023.[134] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams WeiYu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.Finetuned language models are zero-shot learners. arXiv preprintarXiv:2109.01652, 2021.[135] Moshi Wei, Nima Shiri Harzevili, Yuchao Huang, Junjie Wang, andSong Wang. Clear: contrastive learning for api recommendation.In Proceedings of the 44th International Conference on SoftwareEngineering, pages 376–387, 2022.[136] Mark E Whiting, Irena Gao, Michelle Xing, N’godjigui Junior Diar-rassouba, Tonya Nguyen, and Michael S Bernstein. Parallel worlds:Repeated initializations of the same team to improve team viabil-ity. proceedings of the ACM on Human-Computer Interaction, 4(CSCW1):1–22, 2020.[137] Xiaobao Wu, Fengjun Pan, and Anh Tuan Luu. Towards the topmost:A topic modeling system toolkit. arXiv preprint arXiv:2309.06908,2023.[138] Kazuo Yamaguchi. Event history analysis. Sage, 1991.[139] Yanming Yang, Xin Xia, David Lo, and John Grundy. A surveyon deep learning for software engineering. ACM Computing Surveys(CSUR), 54(10s):1–73, 2022.[140] Likang Yin and Vladimir Filkov. Team discussions and dynamicsduring devops tool adoptions in oss projects. In Proceedings of the35th IEEE/ACM International Conference on Automated SoftwareEngineering, pages 697–708, 2020.[141] Lin Yue, Weitong Chen, Xue Li, Wanli Zuo, and Minghao Yin. A sur-vey of sentiment analysis in social media. Knowledge and InformationSystems, 60:617–663, 2019.129[142] Jichuan Zeng, Jing Li, Yan Song, Cuiyun Gao, Michael R Lyu, andIrwin King. Topic memory networks for short text classification. arXivpreprint arXiv:1809.03664, 2018.[143] Qingcheng Zeng and An-Ran Li. A survey in automatic irony process-ing: Linguistic, cognitive, and multi-x perspectives. arXiv preprintarXiv:2209.04712, 2022.[144] Ting Zhang, Bowen Xu, Ferdian Thung, Stefanus Agus Haryono,David Lo, and Lingxiao Jiang. Sentiment analysis for software en-gineering: How far can pre-trained transformer models go? In2020 IEEE International Conference on Software Maintenance andEvolution (ICSME), pages 70–80. IEEE, 2020.[145] Ting Zhang, Ivana Clairine Irsan, Ferdian Thung, and David Lo. Re-visiting sentiment analysis for software engineering in the era of largelanguage models. arXiv preprint arXiv:2310.11113, 2023.[146] Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam.A survey on aspect-based sentiment analysis: Tasks, methods, andchallenges. IEEE Transactions on Knowledge and Data Engineering,35(11):11019–11038, 2022.[147] Jun Zhao, Kang Liu, and Liheng Xu. Sentiment analysis: miningopinions, sentiments, and emotions, 2016.[148] Zihe H Zhao. The distribution and disengagement of womencontributors in open-source: 2008–2021. In 2023 IEEE/ACM45th International Conference on Software Engineering: CompanionProceedings (ICSE-Companion), pages 305–307. IEEE, 2023.[149] Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao.Can chatgpt understand too? a comparative study on chatgpt andfine-tuned bert. arXiv preprint arXiv:2302.10198, 2023.[150] Minghui Zhou and Audris Mockus. What make long term contrib-utors: Willingness and opportunity in oss community. In 2012 34thInternational Conference on Software Engineering (ICSE), pages 518–528. IEEE, 2012.[151] Qile Zhu, Zheng Feng, and Xiaolin Li. Graphbtm: Graph enhanced au-toencoded variational inference for biterm topic model. In Proceedingsof the 2018 conference on empirical methods in natural languageprocessing, pages 4663–4672, 2018.130Appendix AFirst AppendixFieldNameData Type Unit/PossibleValuesDescriptionuser id String none A unique, anonymized identifierfor each usergender String male,female,unknownUser’s probable genderPrimary Key: user idTable A.1: users: This table lists all participants with anonymized identifiersand their corresponding gender.FieldNameDataTypeUnit/PossibleValuesDescriptionproject id String none A unique, anonymized identifierfor each GSoC projectmentorsfullydetectedBoolean True, False Indicates if all mentors withinthe project are present in thedatasetprogram year Integer [2016:2023] The year in which the projectparticipated in the GSoC pro-gramPrimary Key: project idTable A.2: This table contains key details of each project, including itsunique identifier, year, and mentor detection status.131Appendix A. First AppendixFieldNameData Type Unit/PossibleValuesDescriptionproject id String none Foreign key referencing theprojects tableuser id String none Foreign key referencing the userstablerole String mentor,menteeThe user’s role within theprojectPrimary Key (composite): project id AND user idTable A.3: Outlines the roles of users within projects, specifying whethereach user serves as a mentee or mentor.132Appendix A. First AppendixFieldNameDataTypeUnit/PossibleValuesDescriptionissue pr id String none A unique identifier for each issueor pull requestauthor id String none If the author is a mentor, or amentee, this id is present in userstable’s user idtype String issue, pullrequestCategorizes the entry as an “is-sue” or “pull request”title String none The title of the issue or pull re-questbody String none The main descriptive text(body) of the issue or pullrequeststatus String open,closedIndicates whether the issue/PRis “open” or “closed”, closedmeaning merged for pull requestscreation day Float day On which day of the GSoC pro-gram, the item was createdclose day Float day On which day of the GSoCprogram, the item wasclosed/mergedPrimary Key: issue pr idTable A.4: Records details of issues and pull requests, including type, status,and dates relative to program start.133Appendix A. First AppendixField Name Data Type Unit/PossibleValuesDescriptionproject id String none Foreign Key referenc-ing the projects tableissue pr id String none Foreign Key ref-erencing the is-sues pull requeststablePrimary Key (composite): project id AND issue pr idTable A.5: This table maps issues and pull requests to the projects they areassociated with.FieldNameDataTypeUnit/PossibleValuesDescriptioncomment id String none A unique identifier for each com-mentproject id String none Foreign key referencing theprojects tableissue pr id String none Foreign key referencing the is-sues pull requests tableauthor id String none If the author is a mentor, or amentee, this id is present in userstable’s user idtext String none The comment text contentcreation day Float day On which day of the GSoC pro-gram, the comment was createdPrimary Key (composite): comment id AND project idTable A.6: Contains all comments made on issues and pull requests, linkedto specific GSoC projects.134Appendix A. First AppendixFieldNameData Type Unit/PossibleValuesDescriptionsha String none Securely hashed SHA of the com-mit on GitHubproject id String none Foreign key referencing theprojects tableuser id String none Foreign key referencing the userstablemessage String none Commit messagecreation day Float day On which day of the GSoC pro-gram, the commit was createdPrimary Key (composite): sha AND project idTable A.7: Documents all commits made by GSoC users on the GSoCprojects’ repo.135Appendix BSecond AppendixTable B.1: Discovered topics and their top words.Topic name Top wordsTask Closure published done, done published, done closing, pub-lished published, done closed, closed done, closingdone, done ptal, ptal done, yeah doneDevelopment Enhancements func call, view new, new alerts, request intro-duces, line raw, codeformat, introduces alerts,strokewidth, hq, bpmReview and Feedback ptal ptal, ditto ditto, comparison job, job queued,thanks ptal, ptal ditto, hey ptal, ditto ptal, ptalyes, ptal okWeb App UI Elements pane, lighthouse, classfas, modals, web app,popover, angularjs, breadcrumb, safari, user clicksPull Request Management unassigning, unassigning since, approved pr, al-ready approved, changelog label, pr unassigning,ask reviewers, label merged, prs without, label prsSyntax Specifics noundef, defined noundef, javac, noundef defined,console statement, instead saw, unexpected con-sole, noconsole, statement noconsole, note candi-dateKubernetes Operations ksbuild, ksresource, manifests, ksapply, ksunde-ploy, create image, applied resources, delete ap-plied, pod, pod serviceCode Refactoring consider refactoring, separate class, blocks code,locations consider, found locations, code found, do-main layer, selector depth, applicability greater,depth applicabilityReview Comments and Acceptance comments ptal, review accept, us review, trtd, warnings messages, were tr, messages tbody,datakindmessage messages, wip alignright, mes-sages thankConstants and Static Variables suggestion const, static const, const int, es use,available es, constexpr, typename, mapstring, std-string, suggestion privateContinued on next page136Appendix B. Second AppendixTable B.1 – continued from previous pageTopic name Top wordsCode Updates and Commits updated updated, changed changed, changes ptal,code committed, committed pull, heads pr, mod-ifies following, pr modifies, changes updated, fol-lowing filesConflict Resolution changes develop, conflict please, branch pr, followneed, conflict pr, help resolving, resolving conflict,due recent, hi due, branch latestCoding Conventions ulli, cpp inline, nodiscard, nonvoidmethodcall-mutator, error clangtidy, reported error, trail-ing return, modernizeusetrailingreturntype, warn-ing use, membervariablemutatorPackage Management and Errors ccextractor, li ccextractor, package filename,notimplementederror pypi, filename notimple-mentederror, pypi package, pypi, notimplemented-error, autoprogram, ccextractor autoprogramFatal Errors and Allocators iovlen, fatal, fatal error, tinydnnalignedallocator-float, tinydnnalignedallocatorfloat ul, cve cve, er-ror matching, matching program, program output,iovbaseSprint Planning and Issue Assignment hrs, sprint, first issue, ive assigned, claim issue,planning issue, days please, public lab, issue as-signed, assign issueCode Structure and Design eqm, warnnonsnakecase, warnnonsnakecase de-fault, case name, snake case, snake, eqm eqm, lo-calname, warning variable, default pubUX and UI Elements ux, hover, properties ordered, tooltips, font size,px px, navbar, grey, scrollbar, user experienceDatabase Schema and Keys foreign, varchar, primary key, graphql, foreign key,schema version, firebase, usertags, mongo, varcharnullReview Rework rereview, addressed review, rereview requested,since rereview, sure addressed, requested please,proof changes, unassigning, unassigning since,complete followingTesting Quality styletextdecoration, tda, tda width, quality gate,gate, tr tr, styletextdecoration tda, height ti-tletests, titletests, security securityStream Loading and RANSAC loaded bytes, seconds loaded, stream load, bytesstream, ransac, lmeds, lmeds ms, ransac ms,estimateaffinedestimateaffine, estimateaffinepar-tialdestimateaffineCode Indentation and Continuation continuation line, long characters, line missing,characters line, missing indentation, indentationoutdented, outdented, line continuation, visual in-dent, underindentedContinued on next page137Appendix B. Second AppendixTable B.1 – continued from previous pageTopic name Top wordsCoverage and Diff Analysis coverage diff, diff coverage, impacted, arrowdown,files lines, arrowup, hits misses, powered last, pow-ered, report mergingTest Results and Expectations itshould, col col, suggestion itshould, pass ex-pected, junit junit, expected fail, subtest, resultpass, subtest result, unexpected subtestComment Justifications understand comment, explain comment, needs jus-tification, public transport, ditto needs, neededneeded, transport, whats reason, justificationditto, explain changeCode Cleanup removed removed, removed remove, remove com-mented, done removed, remove removed, removeddone, line removed, imported unused, needed re-moved, line removePython Imports and Data Structures dictstr, python import, liststr, list comprehension,python class, import numpy, initself, import top,def initself, valueerrorGratitude and Acknowledgements thanks thank, tada tada, congratulations, thankthanks, thanks nice, thanks awesome, good thank,thank thank, job thanks, thanks soundsFixes and Resolutions fixed fixed, fixup commit, fixed fixup, commitfixed, fix fixed, thanks fixed, fixed resolved, re-solved fixed, fixed yes, resolved resolvedMathematical Computations coefficients, positivetrue, piecewise, polynomial, oooo, rational, derivative, yy, sage, polynomialsBuild Failures build failed, failing develop, build pr, stale could,result tests, request latest, develop please, developthanks, update pull, could resultVariable Naming Conventions camel case, maybe rename, rename variable, snake-case, maybe call, lets rename, name variable, de-scriptive name, think name, descriptive variableDocumentation and Comments documentation changes, preview documentation,add docstring, jsdoc, javadocs, documentationadd, pr preview, javadoc javadoc, header comment,please documentPunctuation and Proofreading misspelled, period end, fullstop, missing comma,full stop, punctuation, translatable, proofread, addperiod, please proofreadLicensing and CLA cla, license header, gpl, signed cla, cla assistant, as-sistant, committers, committers signed, suggestioncopyright, assistant committers138Appendix B. Second AppendixTable B.2: The discovered topics, their share metric (Equation 4.4), role dis-tribution, and impact (Equation 4.5) trend.Topic share (%)Distribution (%) impactMentee Mentor trendTask Closure 4.9 95.2 4.8Development Enhancements 4.5 51.8 48.2Review and Feedback 4.5 56.3 43.7Web App UI Elements 4.2 49.1 50.9Pull Request Management 4.0 50.7 49.3Syntax Specifics 4.0 43.8 56.2Kubernetes Operations 3.5 43.4 56.6Code Refactoring 3.4 34.8 65.2Review Comments and Acceptance 3.4 50.8 49.2Constants and Static Variables 3.3 34.1 65.9Code Updates and Commits 3.3 68.1 31.9Conflict Resolution 3.2 48.5 51.5Coding Conventions 3.2 29.1 70.9Package Management and Errors 3.1 52.6 47.4Fatal Errors and Allocators 3.0 58.0 42.0Sprint Planning and Issue Assignment 2.9 50.0 50.0Code Structure and Design 2.7 26.7 73.3UX and UI Elements 2.6 48.5 51.5Database Schema and Keys 2.6 41.3 58.7Review Rework 2.6 64.9 35.1Testing Quality 2.6 47.5 52.5Stream Loading and RANSAC 2.5 46.6 53.4Code Indentation and Continuation 2.5 20.6 79.4Coverage and Diff Analysis 2.5 85.6 14.4Test Results and Expectations 2.4 43.0 57.0Comment Justifications 2.3 32.2 67.8Continued on next page139Appendix B. Second AppendixTable B.2 – continued from previous pageTopic share (%)Distribution (%) impactMentee Mentor trendCode Cleanup 2.2 42.6 57.4Python Imports and Data Structures 2.2 34.6 65.4Gratitude and Acknowledgements 2.0 45.4 54.6Fixes and Resolutions 1.9 79.8 20.2Mathematical Computations 1.8 42.7 57.3Build Failures 1.8 46.2 53.8Variable Naming Conventions 1.7 24.0 76.0Documentation and Comments 1.7 28.6 71.4Punctuation and Proofreading 1.0 29.1 70.9Licensing and CLA 0.2 24.9 75.1140
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Graduate
Fard, Fatemeh Hendijani
Rodríguez-Pérez, Gema
The alchemy of OSS : transforming chats into insights using sentiment, emotion, and topic analysis of mentor-mentee communications in Google Summer of Code
Text