Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Augmenting source code editors with external information Liu, Xinhong 2019

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2019_november_liu_xinhong.pdf [ 9.22MB ]
Metadata
JSON: 24-1.0381006.json
JSON-LD: 24-1.0381006-ld.json
RDF/XML (Pretty): 24-1.0381006-rdf.xml
RDF/JSON: 24-1.0381006-rdf.json
Turtle: 24-1.0381006-turtle.txt
N-Triples: 24-1.0381006-rdf-ntriples.txt
Original Record: 24-1.0381006-source.json
Full Text
24-1.0381006-fulltext.txt
Citation
24-1.0381006.ris

Full Text

Augmenting Source Code Editorswith External InformationbyXinhong LiuB.Eng, City University of Hong Kong, 2012A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)September 2019c© Xinhong Liu, 2019The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Augmenting Source Code Editorswith External Informationsubmitted by Xinhong Liu in partial fulfillment of the requirements for the degreeof Master of Science in Computer Science.Examining Committee:Reid Holmes, Computer ScienceSupervisorRonald Garcia, Computer ScienceSupervisory Committee MemberiiAbstractDevelopers use many tools and services to acquire code-centric information in theirdevelopment processes. These tools, such as continuous integration and coverageplatforms, make the development process fluid and help developers make betterdecisions, but developers often need to leave their code editors to retrieve the in-formation from them. In this thesis, I present External Information in Code Editor(EICE), an external information integration technology probe, to understand howto help developers acquire and present information from external sources. In thisthesis, I explore the kinds of external information developers need in their devel-opment process and how to present it. As visually changing code editors can affectdevelopers’ interaction and productivity, I conduct an online survey to understanddevelopers’ preferences on visual representations of external information. Finally,I investigate the benefits and disadvantages of EICE plugins in a user study. Theresults demonstrate that with careful design, the integration of external informationinto source code editors can be beneficial to developers.iiiLay SummaryDo your projects use any external tools like Continuous Integration (CI), coverage,or performance benchmarking? Do you find yourself having to visit different web-sites to find out information about the files you’re looking at in your IDE? Thisthesis investigates how to best represent information from these services and toolsmore directly in source code editors, so you can focus more on your tasks andspend less time hunting through cloud-based analysis tools looking for the datayou need.ivPrefaceThe work presented in this thesis was conducted in the Software Practices Lab-oratory at the University of British Columbia, Point Grey campus. All projectsand associated methods were approved by the University of British Columbias Re-search Ethics Board [certificate #H18-02971].This material is the result of ongoing research at the Software Practices Lab-oratory. I was the lead investigator, responsible for all major areas of conceptformation, data collection and analysis, as well as manuscript composition. ReidHolmes was the supervisory author on this project and was involved throughoutthe project in concept formation and manuscript composition. The material has notbeen published prior to this thesis.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 External Information in Code Editor (EICE) . . . . . . . . . . . . 32 External Information . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1.1 Awareness . . . . . . . . . . . . . . . . . . . . . . . . . 82.1.2 Comprehension . . . . . . . . . . . . . . . . . . . . . . . 92.1.3 Traceability . . . . . . . . . . . . . . . . . . . . . . . . . 102.1.4 Tracking Changes to Source Code . . . . . . . . . . . . . 10vi2.1.5 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.6 Navigation . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.7 Debugging and Analysis . . . . . . . . . . . . . . . . . . 112.2 Associated Element Type . . . . . . . . . . . . . . . . . . . . . . 122.3 Frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.4 Temporal Aspect . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Visual Representation . . . . . . . . . . . . . . . . . . . . . . . . . . 163.1 Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.1.1 Non-intrusiveness . . . . . . . . . . . . . . . . . . . . . . 163.1.2 Intuitive Expressiveness . . . . . . . . . . . . . . . . . . 173.1.3 Expandability . . . . . . . . . . . . . . . . . . . . . . . . 173.1.4 Uniqueness . . . . . . . . . . . . . . . . . . . . . . . . . 173.2 Presentation Views . . . . . . . . . . . . . . . . . . . . . . . . . 173.3 Usage of Presentation Views . . . . . . . . . . . . . . . . . . . . 204 EICE Prototyping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.1 Selection of Plugins . . . . . . . . . . . . . . . . . . . . . . . . . 214.1.1 Build-Break . . . . . . . . . . . . . . . . . . . . . . . . . 214.1.2 Test-History . . . . . . . . . . . . . . . . . . . . . . . . . 224.1.3 Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . 224.1.4 Runtime-Value . . . . . . . . . . . . . . . . . . . . . . . 224.1.5 Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . 224.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.2.1 Editor of Choice . . . . . . . . . . . . . . . . . . . . . . 224.2.2 EICE Architecture . . . . . . . . . . . . . . . . . . . . . 234.2.3 Data Source . . . . . . . . . . . . . . . . . . . . . . . . . 255 Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.1 Participants and Distribution . . . . . . . . . . . . . . . . . . . . 275.2 Survey Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.2.1 Background Information . . . . . . . . . . . . . . . . . . 275.2.2 Visual Representations of Individual Plugins . . . . . . . 285.2.3 Persistence versus Toggling . . . . . . . . . . . . . . . . 30vii5.2.4 Management of Multiple Plugins . . . . . . . . . . . . . 305.3 Survey Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.3.1 Which representation do you prefer? Gutters or badges? . 325.3.2 Would you prefer to display the information in an extrapanel? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.3.3 Why / how can external information visualizations be im-proved? . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.3.4 How do you prefer to show persistent visual elements? . . 445.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456 User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476.1 Selection of Participants . . . . . . . . . . . . . . . . . . . . . . 486.2 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486.2.1 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 486.2.2 Experimental: Baseline Group . . . . . . . . . . . . . . . 496.2.3 Experimental: EICE Group . . . . . . . . . . . . . . . . 506.2.4 Project Preparation . . . . . . . . . . . . . . . . . . . . . 506.2.5 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516.2.6 Follow-up Questions . . . . . . . . . . . . . . . . . . . . 526.3 User Study Results . . . . . . . . . . . . . . . . . . . . . . . . . 536.3.1 EICE Benefits . . . . . . . . . . . . . . . . . . . . . . . 536.3.2 EICE Disadvantages . . . . . . . . . . . . . . . . . . . . 576.3.3 Would Developers Use EICE in Practice? . . . . . . . . . 596.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 597 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617.1 Internal Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . 617.2 External Validity . . . . . . . . . . . . . . . . . . . . . . . . . . 627.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 628 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 648.1 Software Visualization . . . . . . . . . . . . . . . . . . . . . . . 648.2 Adding Separate Display to Code Editors . . . . . . . . . . . . . 648.3 In-line Augmenting . . . . . . . . . . . . . . . . . . . . . . . . . 65viii9 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68A Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71A.1 Page 1: Background . . . . . . . . . . . . . . . . . . . . . . . . . 71A.1.1 How many years have you been programming? . . . . . . 71A.1.2 Have you ever worked as a professional programmer/de-veloper? . . . . . . . . . . . . . . . . . . . . . . . . . . . 71A.1.3 How many years have you been a professional program-mer/developer? . . . . . . . . . . . . . . . . . . . . . . . 72A.1.4 Which IDEs or code editors do you use? . . . . . . . . . . 72A.2 Page 2: Scenario 1 . . . . . . . . . . . . . . . . . . . . . . . . . 72A.2.1 Which representation do you prefer? . . . . . . . . . . . . 73A.2.2 Which of the following reasons explain your choice? . . . 73A.2.3 Why / how can this visualization be improved? . . . . . . 74A.2.4 Would you prefer to display the information in an extrapanel, away from the code area, like the one shown in thescreenshot below? . . . . . . . . . . . . . . . . . . . . . 74A.2.5 Which of the following reasons explain your choice? . . . 74A.3 Page 3: Scenario 2 . . . . . . . . . . . . . . . . . . . . . . . . . 75A.3.1 Which representation do you prefer? . . . . . . . . . . . . 75A.3.2 Which of the following reasons explain your choice? . . . 76A.3.3 Why / how can this visualization be improved? . . . . . . 76A.3.4 Would you prefer to display the information in an extrapanel, away from the code area, like the one shown in thescreenshot below? . . . . . . . . . . . . . . . . . . . . . 76A.3.5 Which of the following reasons explain your choice? . . . 77A.4 Page 4: Scenario 3 . . . . . . . . . . . . . . . . . . . . . . . . . 77A.4.1 Which representation do you prefer? . . . . . . . . . . . . 78A.4.2 Which of the following reasons explain your choice? . . . 78A.4.3 Why / how can this visualization be improved? . . . . . . 78ixA.4.4 Would you prefer to display the information in an extrapanel, away from the code area, like the one shown in thescreenshot below? . . . . . . . . . . . . . . . . . . . . . 79A.4.5 Which of the following reasons explain your choice? . . . 79A.5 Page 5: Scenario 4 . . . . . . . . . . . . . . . . . . . . . . . . . 80A.5.1 Which representation do you prefer? . . . . . . . . . . . . 80A.5.2 Which of the following reasons explain your choice? . . . 80A.5.3 Why / how can this visualization be improved? . . . . . . 81A.5.4 Would you prefer to display the information in an extrapanel, away from the code area, like the one shown in thescreenshot below? . . . . . . . . . . . . . . . . . . . . . 81A.5.5 Which of the following reasons explain your choice? . . . 82A.6 Page 6: Non-Scenario Preferences . . . . . . . . . . . . . . . . . 82A.6.1 How do you prefer to show these “persistent visual ele-ments”? . . . . . . . . . . . . . . . . . . . . . . . . . . . 82A.6.2 Which approach do you prefer to show multiple visual el-ements on the same line? . . . . . . . . . . . . . . . . . . 82A.6.3 Can you tell us why you made this choice? . . . . . . . . 83A.7 Page 7: Follow-up Study . . . . . . . . . . . . . . . . . . . . . . 83A.7.1 Are you interested in participating the follow-up study? . . 84A.7.2 Please leave your email address below so we can get in touch. 84A.8 Page 8: Completion . . . . . . . . . . . . . . . . . . . . . . . . . 84B Proposed Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . 85B.1 Task 1: Code Comprehension . . . . . . . . . . . . . . . . . . . . 85B.2 Task 2: Fix Bug that Failed the Recent Builds . . . . . . . . . . . 86B.3 Task 3: Improve Coverage . . . . . . . . . . . . . . . . . . . . . 86B.4 Task 4: Optimize Performance . . . . . . . . . . . . . . . . . . . 87xList of TablesTable 2.1 Types of external information grouped by purpose. Possible ap-proaches to extract the information and how frequent I thinkthey are referred to are also presented. Frequency are rated be-tween “+” (least frequent) to “+++++” (most frequent) by mesubjectively based on my development experience . . . . . . . 13Table 5.1 Provided reason options for gutters and badges. . . . . . . . . 29Table 5.2 Options for Q6 based on answers in Q5. . . . . . . . . . . . . 29Table 5.3 Why participants prefer gutters over badges for each scenario. . 34Table 5.4 Reasons participants “prefer the separate panel.” . . . . . . . . 37Table 5.5 Reasons participants “prefer in the code area while keeping thepanel.” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39Table 5.6 Reasons participants “prefer not to have a separate panel.” . . . 40Table 6.1 Task descriptions and plugins that can help with the tasks. . . . 51Table 6.2 The workflow recorded in Task 2, as well as action and contextswitch count. Supplemental actions are marked with “*.” . . . 54Table B.1 The proposal workflow in Task 1, as well as action and contextswitch counts. Supplemental actions are marked with “*.” . . . 85Table B.2 The proposal workflow in Task 2, as well as action and contextswitch counts. Supplemental actions are marked with “*.” . . . 86Table B.3 The proposal workflow in Task 3, as well as action and contextswitch counts. Supplemental actions are marked with “*.” . . . 86xiTable B.4 The proposal workflow in Task 4, as well as action and contextswitch counts. Supplemental actions are marked with “*.” . . . 87xiiList of FiguresFigure 1.1 Typical flow of information between the source code, externaltools, and the developer. . . . . . . . . . . . . . . . . . . . . 2Figure 1.2 The Atom code editor with plugin lcov-info installed. The cov-erage is reflected on individual line of code. . . . . . . . . . . 4Figure 1.3 The flow of information between the source code, externaltools, and the developer when using EICE. . . . . . . . . . . 4Figure 3.1 Five inline presentation views. . . . . . . . . . . . . . . . . . 18Figure 3.2 Notification items (in the notification area) and the notificationpanel after the developer clicks the notification item. . . . . . 20Figure 4.1 The architecture of EICE system. . . . . . . . . . . . . . . . 24Figure 5.1 The survey flowchart. . . . . . . . . . . . . . . . . . . . . . . 27Figure 5.2 The flowchart of a scenario. . . . . . . . . . . . . . . . . . . 28Figure 5.3 Survey screenshot for showing information in a separate panel(Question 5). . . . . . . . . . . . . . . . . . . . . . . . . . . 30Figure 5.4 Management of multiple pieces of information presented asbadges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Figure 5.5 Management of multiple pieces of information presented asgutters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31Figure 5.6 Numbers of responses for each question group. . . . . . . . . 32Figure 5.7 Participant experience. . . . . . . . . . . . . . . . . . . . . . 32Figure 5.8 Preferences of visual representation for all scenarios. . . . . . 33xiiiFigure 5.9 Preferences on management of multiple plugins, badges (A) orgutters (B). . . . . . . . . . . . . . . . . . . . . . . . . . . . 36Figure 5.10 Preferences for showing information in a separate panel foreach scenario. . . . . . . . . . . . . . . . . . . . . . . . . . . 38Figure 5.11 Preferences on when the persistent visual elements should bedisplayed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Figure 6.1 Randomization process of tasks and grouping. . . . . . . . . . 52Figure 6.2 Comparison of the number of context switches (NCS) betweenthe baseline group and the EICE group for all tasks. . . . . . . 55Figure 6.3 Comparison of the number of actions (NA) between the base-line group and the EICE group for all tasks. . . . . . . . . . . 56Figure 6.4 The interface of Runtime-Value plugin, where values of vari-able of interest is listed for different executions. . . . . . . . . 57Figure A.1 Management of multiple pieces of information presented asbadges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83Figure A.2 Management of multiple pieces of information presented asgutters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83xivGlossaryCD Continuous DeliveryCI Continuous IntegrationEICE External Information in Code EditorIDE Integrated Development EnvironmentxvAcknowledgmentsThanks to the pals at the Software Practices Lab of the Computer Science Depart-ment of the University of British Columbia for my past two years in the lab.Reid helped me in every single stage, from forming the project idea, planningthe methodologies, the executions and analyzing the responses and data. Thanksto Reid for helping me figure it out with his countless pieces of advice.Also, thank you to Jessie Jiang, for all her love and support. Most importantly,for my mom and dad who have been backing me up, encouraging me, and support-ing whatever choices I make, big thanks to them.xviChapter 1IntroductionDevelopers use external information during the software development process (e.g.,[7, 13, 16, 19, 21]). According to a survey conducted by JetBrains in 20181, inaddition to Integrated Development Environments (IDEs), where developers writecode, 77% of developers use source code collaboration tools. Nearly 50% of devel-opers use issue trackers, Continuous Integration (CI) or Continuous Delivery (CD)tools. Other categories such as static analysis tools and code review tools also havesignificant industrial buy-in. Integrated Development Environment (IDE)Developers are likely to rely on multiple tools for a project, and since these ex-ternal tools usually run in several different places, such as developers’ computers orremote servers, information gathering can become a non-trivial task as developersneed to use the interfaces from tools to retrieve information relevant to their task.For example, to increase test coverage, developers need to first open the coveragereport generated by the test runner (or they need to visit the cloud service that com-putes their coverage metrics), locate the uncovered lines, open the correspondingsource file, and finally develop new unit tests for the uncovered section of code.Figure 1.1 shows the typical flow of external information among the sourcecode, external tools/services, and the developer. These external tools/services canbe hosted locally but increasingly reside on cloud-based services. The reports or re-sults are independent from the source code to which they pertain. When developers1The State of Developer Ecosystem in 2018 https://www.jetbrains.com/research/devecosystem-2018/1%QFGNKPM6GUVU%QFGCPCN[UKU4GRQTVU9GDUKVGU2TQHKNGT&GXGNQRGT4GUWNVUFigure 1.1: Typical flow of information between the source code, externaltools, and the developer.inspect these external results, the link between the information and the source codeis implicit and developers need to do the linking work (e.g., via code navigation)themselves.1.1 MotivationsIn this thesis, I define any information computed by a tool or service not di-rectly integrated into the IDE as external source code information. In orderto acquire external information, developers may need to conduct static/dy-namic analysis or invoke online services, which I define as external informa-tion sources.This kind of information has the following disadvantages:It requires developers to constantly switch contexts. External tools usually comewith their own interfaces, and external web services are hosted on independentwebsites. Developers need to interact with these interfaces located in differentplaces (console, different websites or dedicated desktop apps). When developers2switch between interfaces, their cognitive load increases, which impacts their pro-ductivity.The information presented by these tools or services usually does not have di-rect links back to source code. Despite tools and services providing informationclosely related to the source code, when developers see a piece of information, theyneed to navigate to the corresponding source code file and jump to the line of codemanually. For Code-centric information, I believe it is better for the informationto be linked to or displayed next to the code, to eliminate this step.Notifications can be distracting. In a typical scenario, when the developer makesa change to the codebase, the CI compiles and runs the tests. When somethinggoes wrong, developers will receive a notification. Notifications can be distracting,especially when they are sent through multiple channels. A Slack message, or anemail notification, prevents developers from concentrating on their programmingtasks [14]. When developers mute notifications, they will not be aware of the issueswhen anything goes wrong. Developers should be notified in a non-intrusive way.1.2 External Information in Code Editor (EICE)IDEs and code editors are extensible, and communities have made thousands ofplugins for them. One type of plugin can augment source code with external in-formation, saving developers from leaving the code editor to acquire informationrelevant to them. For example, a plugin named lcov-info 2 written for the popu-lar code editor Atom, is capable of displaying code coverage information directlyinside Atom (see Figure 1.2).In this thesis, I push this idea further and describe a technology probe to explorehow to extract external information and present it in code editors, which I callExternal Information in Code Editor (EICE). Using EICE, developers do not needto leave their code editor to acquire information from external information sources.The code editors (or the plugins) fetch information from corresponding externalsources and present the information directly alongside the code area within the2https://atom.io/packages/lcov-info3Figure 1.2: The Atom code editor with plugin lcov-info installed. The cover-age is reflected on individual line of code.%QFGNKPMNKPMNKPM6GUVU%QFGCPCN[UKU4GRQTVU9GDUKVGU2TQHKNGT&GXGNQRGT4GUWNVUFigure 1.3: The flow of information between the source code, external tools,and the developer when using EICE.4code editor. As a result, developers only need to interact with the code editor.Figure 1.3 demonstrates how information flows in this model.The primary advantage of this approach is that developers can stay within theircode editor and avoid switching contexts, spending more time on the importantparts of their tasks. All information is directly linked to the source code. Theinformation that is code-centric is presented next to the code, or has an explicitlink to the code element. Developers do not need to manually navigate to thecorresponding code location.Notifications can be non-intrusive and grouped. Because the code editor can ac-cess the information directly, it can present the notifications within the developer’scurrent context. A small area of the code editor can be used for notification pur-poses. When something goes wrong, developers can glance at that area and noticethe issue without interrupting their tasks. Despite adding new user interface (UI)elements, grouping notifications together maintains a low interface complexity [2].Notifications can be instantly actionable. Since notifications are placed in thecode editor, when the developer examines a notification it will be visible along-side the code to which it pertains, which makes this kind of notification instantlyactionable.The thesis is organized as follows. I first explore the kinds of external informa-tion developers refer to in their development process in Chapter 2. Then I discussvisual representations of external information in Chapter 3 and develop the EICEplugin prototypes in Chapter 4. To understand developers’ preferences on visualrepresentations, I design an online survey with EICE plugin mockups (Chapter 5).Finally, I investigate EICE’s functional benefits and how developers interact withEICE plugin prototypes in an empirical user study (Chapter 6).Below are five research questions I aim to answer in this thesis:RQ1 What kinds of external information do developers refer to during the softwaredevelopment process?5RQ2 What preferences do developers have for integrating data from external infor-mation sources into their source code editors?RQ3 Can developers successfully use EICE to reduce effort acquiring external in-formation?RQ4 Are there negative consequences to integrating external information sourcesinto the IDE?RQ5 Will developers use EICE over traditional methods for external informationacquisition in practice?6Chapter 2External InformationDuring the development process, developers frequently refer to external informa-tion such as build status, coverage, source control, etc. To systematically studywhat external information developers refer to and to answer RQ1, I categorize ex-ternal information sources according to their purpose. Each type of information hasan associated element type, where the associated element is where the informationlinks to in the code editor. Table 2.1 shows a collection of external informationfrom work related to my hypothesis. Some of the related work studied the infor-mation developers need instead of refer to. I assume that if developers need a pieceof information, they refer to it in practice. I subjectively set these frequency ratingsand list preliminary data sources in Table 2.1.2.1 PurposeI find external information can mostly be categorized to one of the following pur-poses:• Awareness• Comprehension• Requirement traceability• Tracking changes to source code7• Test cases• Navigation• Debugging and analysis2.1.1 AwarenessAwareness information supports collaboration during the software developmentprocess [6]. Lack of awareness could negatively affect developers’ performance[12]. Questions related to awareness developers might ask during software devel-opment include:• Who is working on what? [13]• What are they [coworkers] working on right now? [13]• Who changed this [code], focused on person? [13]• Who to assign a code review to? / Who has the knowledge to do the codereview? [13]• What classes has my team been working on? [13]• Who is working on the same classes as I am and for which work item? [13]• Who owns this piece of code? Who modified it most recently? Who modi-fied it most? [13]• Which class has been changed most? [13]• How do recently delivered changes affect changes that I am working on?[13]Many of the answers to these questions can be acquired from Git. vscode-gitlens1 is a tool that integrates some Git features directly into the code editor.vscode-gitlens provides a quick glimpse into by whom, why, and when a line waschanged, with inline annotations and more advanced insights via its interface.1https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens82.1.2 ComprehensionComprehending computer programs is one of the core software engineering activi-ties, and it is required in many software development phases [20]. Comprehendingsource code can be time-consuming [1] and involves finding answers to questionslike:• Where is this method called or type referenced? [21]• When during execution is this method called? [21]• Where are instances of this class created? [21]• Where is this variable or data structure being accessed? [21]• What are the arguments to this function? [21]• What are the values of these arguments at runtime? [21]• What data is being modified in this code? [21]• How are instances of these types created and assembled? [21]• How are these types or objects related? [21]• What is the “correct” way to use or access this data structure? [21]• How does this data structure look at runtime? [21]• Where is it declared and initialized?• Where is this variable used afterwards?• Does it change any states, or mutate any data?• Is it a rear exception catch block?• What is the common call path of its statements?• Is it likely to be a utility function?• Does it update UI in its thread (does it need to be in the main thread)?9• What is the super/subclass?• What is the mapping between these UI types and these model types? [21]• What are the common execution routine of this code?2.1.3 TraceabilityTraces are valuable in the area of validation and verification, and are concernedwith program comprehension [25]. Regarding traceability, developers sometimesneed to know “What is the intention behind this piece of source code?” [17] Recentwork demonstrated a combination of approaches to tackle the “concern locationproblem” (identify source code to the features) [11].• How is this feature or concern (object ownership, UI control, etc.) imple-mented? [21]• What is the requirement behind this code?• What code is most relevant to this code (for satisfying the requirement)?2.1.4 Tracking Changes to Source CodeDevelopers may ask questions about one or more changes to the codebase. Change-related information includes change metadata (who and when a change was made),the code being changed, and the impact code changes can cause.• What is the evolution of the code? [13]• What caused this build to break? [13]• What will be (or has been) the direct impact of this change? [21]• When is the recent modification?102.1.5 Test CasesTest cases are also code, and can benefit from annotations. Similar to code changes,metadata information such as authorship helps to find the developer responsible forit.• Who owns a test case? [13]• How do test cases relate to packages/classes? [13]• Is this statement covered by tests?• Has this test ever failed? How often does this test fail?• What are related tests (cover similar codes)?• What are correlated tests (having similar pass or fail patterns)?2.1.6 NavigationExternal information can also assists code navigation so that developers can quicklylocate the place where the existing code is or new code is to be added.• Where in the UI should this functionality be added? [21]• Where should I insert this code?2.1.7 Debugging and AnalysisWhen programs have correctness or performance issues, developers need to lookinto the source codes for them.• How can I know that this object has been created and initialized correctly?[21]• Why is [variable] equal to [value]? [16]• What is average execution time of this statement? [7]• Where the value being changed?112.2 Associated Element TypeExternal information could be about a variable, statement, line(s), code block,method/function, or class/type/interface. These are the elements in the code ed-itor, and in this thesis, I am only interested in information that can be associatedto any element(s). The link between the information and the associated elementimpacts how I present it.2.3 FrequencyMoving external information into the code editor saves time and effort. If a devel-oper refers to one piece of information once a month, the effort of integrating itinto the code editor is hard to justify. Conversely, if developers refer to somethinga dozen times a day, even a tiny bit of seconds add up.2.4 Temporal AspectSome information describes the temporal aspects of the associated elements. Forexample, past states of the source code need to be aggregated to answer the ques-tion “What is the evolution of the code?” Usually this type of information requiresthe version control system (e.g., Git) as its data source.12Table 2.1: Types of external information grouped by purpose. Possible approaches to extract the information andhow frequent I think they are referred to are also presented. Frequency are rated between “+” (least frequent) to“+++++” (most frequent) by me subjectively based on my development experiencePurpose Information Need Attached Element Type Information Source Frequency Frequency CommentAwarenessWho is working on the same classes as I am and for which workitem? [13]Class/type/interface, state-mentgit remote +++Who owns this piece of code? Who modified it most recently?Who modified it most? [13]Line(s) git blame +++ When I have questions about thiscode, I can ask him/herWhich class has been changed most? [13] Class/type/interface git log + Out of curiosityHow do recently delivered changes affect changes that I am work-ing on? [13]Statement git log, static analysis ++++ Important to know there is no con-flictWho is responsible for this statement? Statement git blame +++ When I have questions about thiscode, I can ask him/herComprehensionWhere is this method called or type referenced? [21] Method/function Static analysis ++++ During comprehensionWhen during execution is this method called? [21] Method/function Dynamic analysis (test run) ++++ During comprehensionWhere are instances of this class created? [21] Class/type/interface Static analysis ++++ During comprehensionWhere is this variable or data structure being accessed? [21] Variable Static analysis ++++ During comprehensionWhat are the arguments to this function? [21] Method/function Static analysis ++++ Debugging and codingWhat are the values of these arguments at runtime? [21] Variable Dynamic analysis (test run) ++ During comprehensionWhat data is being modified in this code? [21] Statement Static analysis ++ During comprehension or debug-gingHow are instances of these types created and assembled? [21] Class/type/interface Static analysis ++ During comprehensionHow are these types or objects related? [21] Class/type/interface, vari-ableStatic analysis +++ During comprehension, to get a bigpictureWhat is the ”correct” way to use or access this data structure? [21] Class/type/interface Static analysis +++++ During codingHow does this data structure look at runtime? [21] Class/type/interface Dynamic analysis (test run) ++ During comprehensionWhere is it declared and initialized? Variable Static analysis +++ I want to know its initial valueWhere is this variable used afterwards? Variable Static analysis or Dynamic anal-ysis++++ Understand data flowDoes it change any states, or mutate any data? Statement, Method/function Static analysis +++Is it a rear exception catch block? Code block Dynamic analysis (test run) ++++ I wish it does not appearWhat is the common call path of its statements? Code block Dynamic analysis (test run) +++ Helps to understand the logicsIs it likely to be a utility function? Method/function Static analysis ++ Code comprehensionDoes it update UI in its thread (does it need to be in the mainthread)?Method/function Static analysis ++++ Frequent if dealing with GUI devel-opment13Purpose Information Need Attached Element Type Information Source Frequency Frequency CommentWhat is the super/subclass? Class/type/interface Static analysis +++ Looking for methods in superclassor subclassWhat is the mapping between these UI types and these modeltypes? [21]Class/type/interface Static analysis ++ Newbies try to understandWhat are the common execution routine of this code? Code block Dynamic analysis (test run) +++ During comprehension and refac-toringRequirementTraceabilityHow is this feature or concern (object ownership UI control etc.)implemented? [21]Method/function CERBERUS [11] + RareWhat is the requirement behind this code? Code block Analysis based on CERBERUS[11]++ Code comprehensionWhat code is most relevant to this code (for satisfying the require-ment)?Code block Analysis based on CERBERUS[11]+++ When changing code, I also need tofind relevant locations where codeneed to be changed accordingly.ChangesWhat is the evolution of the code? [13] git log ++ Newbies try to understandWhat caused this build to break? [13] Dynamic analysis (roll back andbuild)What will be (or has been) the direct impact of this change? [21] Line(s) Static analysis ++++ Good to know whenever I am aboutto change somethingWhen is the recent modification? Statement git log + Wanna know if it is actively devel-oped.Test casesWho owns a test case? [13] Test git blame + Know who to askHow do test cases relate to packages/classes? [13] Test Static analysisIs this statement covered by tests? Statement, Method/function Dynamic analysis (test run) +++++ If testing matters, this is very fre-quentHas this test ever failed? How often does this test fail? Test Dynamic analysis (test run) ++++ When scrolling through many tests,one need for each testWhat are related tests (cover similar codes)? Test Dynamic analysis (test run) ++What are correlated tests (having similar pass or fail patterns)? Test Dynamic analysis (test run) ++NavigationWhich segment of this source code should this functionality beadded?Line(s) Static analysis +++ Useful when I get a bootstrap codeand want to insert my codesWhere should I insert this code? Line(s) Static analysis +++ Useful when I get a bootstrap codeand want to insert my codesDebuggingand analysisHow can I know that this object has been created and initializedcorrectly? [21]Variable Dynamic analysis (test run) +Why is [variable] equal to [value]? [16] Variable Dynamic analysis (trace) +++ When I debug and see some oddvalues14Purpose Information Need Attached Element Type Information Source Frequency Frequency CommentWhat is average execution time of this statement? [7] Statement Dynamic analysis (profiling) +++ Only when I find the executionslow, but when I care about the per-formance, it is frequentWhere the value being changed? Variable Static analysis or Dynamic anal-ysis+++++ Understand data flow15Chapter 3Visual RepresentationThis chapter discusses the visual representation of external information. I herestudy alternatives for how a piece of information can be presented, and what prin-ciples are important for keeping a proper interplay among multiple informationfragments.It is worth mentioning that some external information can be associated witha specific source code element (e.g., coverage), and some external information isabout a project in general (e.g., list of contributors). This thesis primarily illustratesthe former. That is, all information discussed can be associated to an element ofsource code (e.g., a variable, a statement, or a code block).3.1 PrinciplesIt is useful to have extra information in the code editor, but it could overwhelmthe interface or distract developers. These presentation principles are based onmaximizing informativeness of external information while maintaining a low over-whelmingness. Plug-ins for EICE should follow these presentation principles.3.1.1 Non-intrusivenessExternal information should not distract developers. The main focus within codeeditors is always the source code. The presentation needs to be minimal. If therepresentation is an overlay spanning across a large area of the code editor, it should16be hidden until it is contextually activated.Similar colors used in syntax highlighting should be avoided, as should eye-distracting (bright) colors.3.1.2 Intuitive ExpressivenessIcons should be preferred over texts, and colors used should match their commonmeanings. For example, a green check represents “covered by passed tests,” a redcheck represents “covered by failed tests,” and a broken gray check represents “notcovered by any tests.”3.1.3 ExpandabilityAlthough the information provided should be minimal so that developers are notdistracted from the source code, in case developers want to know more, the infor-mation presentation should be able to expand with more details while keeping thedeveloper within their IDE.3.1.4 UniquenessThe presentation of information should be distinguishable from other information.For instance, a brown solid circle has an impression as a breakpoint, so other plug-ins should avoid using this presentation.3.2 Presentation ViewsModern code editors such as Atom and VSCode are highly customizable and itis easier to customize the visual look of the code editors with web technologies(HTML/CSS/JS). Communities have been creating many code editor plugins; somecommon techniques for presenting information used by these plugins includes:1. The plugin has a separate panel and presents information in that panel.2. The plugin highlights code segments.3. The plugin adds gutters.4. The plugin adds icons or text to the end (below/above) of a line.175. The plugin adds overlays or floating windows.6. The plugin adds items in context menus.Options 2-6 add visual elements that are directly associated to elements withinthe source code. From 2-6, I can summarize the following five kinds of inlinepresentation views. These are shown in Figure 3.1:1234567x%QPVGZV/GPWKVGO1XGTNC[$CFIG)WVVGT *KIJNKIJVFigure 3.1: Five inline presentation views.Inline Presentation ViewsGutter. Icon/label at the beginning of a line. A small piece of informationabout a line, a function/method, or a class. Gutters can accommodate a smallchart, single word, or icon. Usually multiple icons/labels do not work welltogether due to space constraints.Badges. Similar to gutters, but with more space to accommodate more itemsbecause they appear at the end of the line. Multiple badges can appear togetherprovided enough space, when the line is short.18Context menu item. When information is not suitable to be represented byeither gutters or badges, it can be hidden in a context menu.Highlight. Highlighting variables or an area guides developers’ attention tothe subject(s). They must be manually triggered by the developer.Overlay. An overlay can display complex and rich information. They mustbe manually triggered by the developer.These views can be interactive. If the developer clicks an icon, more informa-tion can be shown, usually in the form of another kind of view. For example, if thedeveloper clicks a context menu item, an overlay can be displayed.Option 1 above presents information outside the code area. Although I wantto integrate all information within the source code elements, an overview is help-ful for many tasks, especially when developers do not know which segment ofsource code they are about to work with. For example, when the task is to im-prove coverage, developers do not know which file contains the most uncoveredlines. A ranked list of source files can deliver this information, but this rankedlist only exists in a full coverage report, and is not suitable to put inside the codearea for a specific element. The Windows Modern operating system has an areanamed “Notification area1” which is located at the lower right corner for notifica-tions and status. Similarly, a dedicated area within the code editor can be allocatedto accommodate notification items shown in Figure 3.2.Isolated Presentation ViewsNotification Item. An icon in a dedicated area of the code editor, which servesa notification purpose. When developers click on the icon, a notification panelappears with detailed information.1https://docs.microsoft.com/en-us/windows/win32/uxguide/winenv-notification191234567x0QVKHKECVKQP2CPGN0QVKHKECVKQP#TGC0QVKHKECVKQP+VGOFigure 3.2: Notification items (in the notification area) and the notificationpanel after the developer clicks the notification item.Notification Panel. An isolated panel, can be displayed adjacent to or abovethe code area. It shows an overview, which gathers individual pieces of inlineinformation that can be associated to a code element, in an organized manner(e.g. sorted, hierarchically).3.3 Usage of Presentation ViewsA single type of external information can utilize multiple of the presentation viewsmentioned above. Take test coverage information for instance: gutters can be dis-played at each line to indicate whether that line of code is covered. When the de-veloper clicks a gutter, an overlay can pop up showing the tests that cover the line.An isolated presentation view is also useful for coverage information, where a no-tification item can be displayed to indicate overall coverage condition, and whenthe developer clicks it, a notification panel appears with a full coverage report.20Chapter 4EICE PrototypingTo investigate the benefits of EICE, I developed a technology probe to evaluate acollection of selected plugins.4.1 Selection of PluginsI chose to implement the following five plugins:1. Debug: Build-Break2. Test: Test-History, Coverage3. Comprehension: Runtime-Value4. Optimization: ProfilingThese plugins are chosen based on them covering different types of develop-ment tasks and being feasible to implement.4.1.1 Build-BreakThe plugin adds gutter items to lines that belongs to commits that directly or in-directly cause the build to fail. When users click the gutter item the commit SHAand related metadata will be displayed in an overlay window.214.1.2 Test-HistoryThe plugin adds a gutter item to each unit test, displaying the latest result of the test.When the developer clicks a gutter, the detailed historical behaviors are displayed.4.1.3 CoverageThe coverage plugin pulls coverage data from the latest coverage report and addsgutter items to lines captured in the report. Green gutter items indicate the lines arecovered and how many times times they are covered. Red gutter items indicate thelines are not covered by unit tests.4.1.4 Runtime-ValueFor values covered by unit tests, this plugin will display the value before and afterthe line executes. Its overlay window is triggered by clicking “Show the runtimevalues of XXX.”4.1.5 ProfilingThis plugin extracts profiling results of functions and shows a badge next to eachfunction’s declaration, showing how long it takes to run along with the runtime ofthe entire program.4.2 ImplementationThese EICE plugins were created to empirically answer RQ3 to RQ5.4.2.1 Editor of ChoiceMost code editors and IDEs are extensible and offer different levels of customiz-ability. VSCode is one of the most popular code editors1; however, it has a restric-tions on DOM access which is essential for the overlays and other complex inlineUIs. After researching the plugin capabilities of different code editors, I choseAtom due to its available APIs. It uses web technologies to render code and pro-1https://code.visualstudio.com22vides full DOM access, which means EICE can flexibly modify, remove and addvisual elements.Early prototypes had a considerable impact on the responsiveness of the editor.In the next section, I talk about how the design of the EICE architecture aims tominimize the performance issues underlying these kinds of plugins.4.2.2 EICE ArchitectureThe EICE system uses the master/slave model, and it consists of an EICE con-troller (master) and multiple EICE plugins (slaves). Figure 4.1 shows the overallEICE architecture.The EICE controller is the central computer of EICE system, and it is respon-sible for the communication between Atom and the plugins. The EICE controlleris in charge of the following tasks:• Parsing ASTs.• Listening to events.• Communicating with EICE plugins.• Rendering presentation views.Each EICE plugin fulfills a single information need. The plugin has accessto the latest AST tree via the EICE controller and prepares its information in thebackground. When a new piece of information is available, it will notify the EICEcontroller which will invalidate old information and update the view.Common external information is code-centric, associated to a source code en-tity. For example, whether a statement is covered or not is associated to thatstatement, and the diff of a commit is associated to that commit, so is indirectlyassociated to the code changes of that commit. A source code entity can be a lineof code, variable, statement, class/type/interface, block or method/function (Fig-ure 4.1). When the EICE controller queries information about a code entity, eachEICE plugin passes a piece of information to the EICE controller to render, in theform of a special data structure ExInfo, which contains the information itself andhow it should be presented.23EICE ControllerListen events Parse AST TreeFile modifiedModifier key keydown/keyupEICE PluginRequest ExInfo about an entityExInfoEntityPresentation Viewof:EntityVariableClass/type/interfaceMethod/functioonStatement BlockPush AST to PluginsPush AST UpdatesSend ExInfo about an entityNotify EICE controller the update of ExInfoResponse ExInfo of an entity upon requestPull ExInfo from PluginsLineRender presentation viewsFigure 4.1: The architecture of EICE system.Performance ConcernsAs I mentioned above, it is easy to undermine Atom’s responsiveness. In thissection, I describe several optimization techniques EICE adopts to reduce theseproblems.Preprocessing the shared resources. To reduce the response time between theEICE controller making the request and the EICE plugin responding the ExInfo, itis helpful to shift the heavy computations before the request is made to process theresources shared by multiple entities. For example, in the Runtime-Value plugin,the values are stored in a large JSON file (around 100 MB), so parsing it is acomputational intense task. As all variables share the same JSON file, it is naturalto preprocess the file. This creates a little lag after the AST updates, in exchangefor responsive querying of individual variables, especially during quick scrolling.Lazy loading. For information that is not likely to be frequently queried, like theone provided by Runtime-Value (which requires developers to explicitly trigger theplugin for every single code entity), EICE delays loading to speed up the startup24time.Updating AST when idle. The EICE controller updates the AST tree and noti-fies plugins of the change only when the user is not actively typing. The guttersand other presentation views still follow the entities’ locations, but plugins are notaware of the code being changed until the user stops typing for several seconds.4.2.3 Data SourceEICE plugins fetch data and post-process it into presentable ExInfo objects.Having a single source of information is easy to manage, and because continu-ous integration services like CircleCI allow developers to keep build/test artifacts,they become a great fit for that “single source.” In addition, each build has its ownartifact space, it is self-contained, easy to index and stored in a single place.To make data generation follow a standard and reproducible process, EICE plu-gins follow a protocol that specifies how the data should be generated and stored.Each CI runs each plugin’s special script to generate and store the artifacts once itdetects a new commit, so the plugins can access this data when it is required.25Chapter 5SurveyTo understand developers’ preferences for the visual representation of external in-formation (RQ2), I conducted a survey and distributed it online among softwaredevelopers. The main format of survey asks developers to pick the option theyprefer from several given options and to explain the reasons.For presentation views, gutters and badges are the main candidates, for theirability to be presented persistently, and not be mandatorily triggered by shortcutsor context menu items. Survey participants were presented with these two presen-tation views and asked to answer several related questions. Adding visual elementsto the code editor has the risk of distracting developers when the information is notuseful. Participants’ preferences about clutter were also investigated.When the information is presented next to the related code entity, it is an in-line representation. There is also an option to display the information in an areaseparate from the source code, but still inside the code editor. For tasks that arenot associated to a specific code entity, such as locating bugs, and these separatedisplays let developers access information from a centralized place.In the survey, most questions have an “other” option, to give participants theoption to supplement their own thoughts.The four sub-questions of RQ2 are:RQ2.1 Do developers prefer gutters or badges?RQ2.2 Does information need to be inline or in a separate area?26RQ2.3 Do developers prefer persisting or toggle-able visual elements?RQ2.4 What other visual preferences do developers have?5.1 Participants and DistributionRQ2 focuses on visual representation, therefore, the functional aspects of the plu-gins are decoupled from the survey. The survey shows several mockups of EICEplugins for participants to have a visual idea of what these plugins do. Althoughthe mockups are in JavaScript projects, participants do not need to be familiar withJavaScript to participate in the survey.The online survey is distributed via email (to CS students and Industry con-tacts) and on social media (e.g., Reddit, HackerNews, etc.) to reach broader audi-ences.5.2 Survey DesignThe estimated time to complete the survey is 10 minutes and participants are al-lowed to quit in the middle of the survey without providing a reason. When aparticipant leaves without finishing the survey, the finished part is still recorded tobe analyzed because sections are independent.Figure 5.1shows the flowchart of the survey. The entire questionnaire can befound in Appendix A.Start Consent FormDigital SignatureBackground InformationScenarios 1-4(Shuffled)Persistence? A or B?Multi-plugin approaches A/B presentedExplain whyFollow-up? EndFigure 5.1: The survey flowchart.5.2.1 Background InformationBefore asking questions about EICE plugins, Participants were first asked abouttheir background.27Experienced developers might have different preferences from junior develop-ers, so there are questions on how much experience they have as a software de-veloper, and how professional they are and whether they are used to full-featuredIDEs or lightweight code editors.5.2.2 Visual Representations of Individual PluginsTo answer RQ2.1, I present participants the same information with these two typesof representation, and ask participants to choose their preferred view and why theychose it.A or B?A/B presentedScenario StartAgree reason B1?Agree reason B2?Agree reason B3?Agree reason A1?Agree reason A2?Other reason?ABHow to improve?Opinion on TopView?Agree reason TP?Agree reason TB?Agree reason TA?PreferAgainstBothOther reason?ScenarioEndTopView presentedFigure 5.2: The flowchart of a scenario.Participants are presented with four independent scenarios (Hist, Fail, Profand Cover), and for each scenario, several questions follow. Figure 5.2 shows aflowchart of a scenario. Although there are many visual representations availableto choose from, gutter and badge are two common yet distinct persisting forms.Gutters, which are outside the actual source code area, are less likely to interferewith the source code; however, because of their limited size, they can only presenta small amount of information. The gutter area is also for traditional functions likebreakpoints, causing visual conflicts. Badges reside inside the source code area,to the right of the source code, and have more room for more details, but couldbe more distracting. Each scenario starts with a brief introduction of the scenario,then presents a mockup of two visual representations (A: badge, B: gutter) with thefirst question “Q1: Which representation do you prefer?” as shown in Section A.2.Once participants choose the form they prefer, the next question asks why they28made that choice in “Q2: Which of the following reasons explain your choice?”.The options are adaptive, based on their choice (Table 5.1 shows the options pro-vided in the survey).Preference WhyBadges (A)A1 “A” provides more information at a glance and saves mefrom the extra step (click)A2 “B” occupies gutter area, which is used for setting break-pointsGutters (B)B1 “A” is distracting as it shares the same space as the sourcecodeB2 “B” is easier to spot since they align wellB3 “B” already provides enough information for most of thescenario, it is good for me while keep visually minimalTable 5.1: Provided reason options for gutters and badges.Next comes an optional open question “Q4: Why / how can this visualizationbe improved?” to gather additional ideas about what developers want (RQ2.4).To better understand developers preferences on inline representation vs repre-sentation in isolated areas, I ask the next question “Q5: Would you prefer to displaythe information in an extra panel, away from the code area, like the one shown inthe screenshot below?” (Figure 5.3 shows an example screenshot of showing in-formation in a separate panel). Depends on the answer in Q5, an adaptive question“Q6: Which of the following reasons explain your choice?” as shown in Table 5.2.Participants choice in Q5 Options of Q6“Yes, I prefer the extra panel.” “I would like to keep the code area clean.”“Yes, I prefer the extra panel” or “I want theinline representation with the option to showthe extra panel.”“The extra panel is better at summarizing thisinformation.”“No, I prefer the inline representation.” “I don’t want too many panels.”Table 5.2: Options for Q6 based on answers in Q5.29Figure 5.3: Survey screenshot for showing information in a separate panel(Question 5).5.2.3 Persistence versus TogglingEICE can be seen as an information platform and allows multiple pieces of infor-mation to co-exist. When the developer works on a specific task, only a few piecesof information are helpful and others maybe unnecessary. Making all informationvisible at all times is good for discovery but might be distracting. On the contrary,if the information is not always visible, developers may not be aware of its exis-tence. To answer RQ2.3, below are three options participants can choose from inQuestion 6.• Always display.• Display when explicitly toggled (e.g., via shortcuts or context menu items).• Display when the cursor enters the area of interest (line of code).5.2.4 Management of Multiple PluginsTo answer RQ2.1 in a multi-plugin setting, I present two management approachesfor plugins adding multiple pieces of information to a single line of code; one for30Figure 5.4: Management of multiple pieces of information presented asbadges.gutters (Figure 5.4) and the other for badges (Figure 5.5).Figure 5.5: Management of multiple pieces of information presented as gut-ters.Participants were asked to pick their preference and explain their choices byanswering these two questions: “Which approach do you prefer to show multiplevisual elements on the same line?” and “Can you tell us why you made this choice?”5.3 Survey Results29 developers answered the survey; Figure 5.6 shows numbers of responses foreach question group, which indicates the completion rate for each survey sectionranges from 72.4% to 82.8%.Figure 5.7 shows the distributions of programming experience and number ofyears as a professional programmer. Among participants, most of them have be-tween 5 to 10 years of experience.31S1 S2 S3 S4 Persist Multi-Plugin051015202530Number of participantsFinish vs Skip for each question groupFinishSkipFigure 5.6: Numbers of responses for each question group.0 5 10 15n of years0123456n of participantsProgramming0 5 10 15n of yearsProfessionalFigure 5.7: Participant experience.5.3.1 Which representation do you prefer? Gutters or badges?RQ2.1: Do developers prefer gutters or badges?Figure 5.8 shows the visual representation preferences. The comparison betweengutter and badge receives a mixed result and it varies from scenario to scenario.For example in S1 (History), badges are preferred by about 2/3 participants,32whose popularity drops to 1/3 in S4. Overall badges are slightly more popular thanthe gutter counterparts. Below is the discussion on why participants prefer oneover the other.65.2%34.8%Scenario 1BadgesGuttersHistory45.8%54.2%Scenario 2BadgesGuttersTest73.9%26.1%Scenario 3BadgesGuttersProfiles27.3%72.7%Scenario 4BadgesGuttersCoverageFigure 5.8: Preferences of visual representation for all scenarios.Why do you prefer badges over gutters?The top reason for those who chose badges over gutters is, “‘A’ provides moreinformation at a glance and saves me from the extra step (A1)”, which is partic-ipants’ main selections across all scenarios (S1: 93.3%, S2: 81.8%, S2: 76.5%,S4: 83.3%). It means participants choose badges mostly because of the higherinformation density. “[The gutter] occupies gutter area, which is used for settingbreakpoints (A2)” is a less popular reason compared to A1, and it makes up for13.3%, 27.3%, 41.2% and 50.0% selections in each scenario respectively. Theseparticipants think adding items to the gutter area is distracting.As for reasons not provided in the survey, I find they are mostly reflection ongutters’ readability issue:“The number 4 is too small to see [in the gutter]. The icons look likecompiler errors.” — P11 in Scenario 1“[The badge] is visually easier to read.” — P2 in Scenario 2“’Failed’ text is hard to read.” — P11 in Scenario 3“Percentage is too hard to read. And icon is unclear.” — P11 in Sce-nario 3“The gutter icon seems hard to read.” — P28 in Scenario 333Reason option S1 (History) S2 (Test) S3 (Profiles) S4 (Coverage)The badge is distracting as itshares the same space as thesource code50.0% 30.8% 50.0% 25.0%The gutter is easier to spot sincethey align well62.5% 23.1% 33.3% 75.0%The gutter already providesenough information for most ofthe scenario, it is good for mewhile keep visually minimal50.0% 69.2% 33.3% 68.8%Table 5.3: Why participants prefer gutters over badges for each scenario.Why do you prefer gutters over badges?From Table 5.3, the overall reasons participants preferring gutters pick are “Gutteris easier to spot since they align well (B2)” and “Gutter already provides enoughinformation for most of the scenario, it is good for me while keep visually minimal(B3).” Participants prefer gutter because of B2 and because an aligned user inter-face reduces UI complexity [2]. B3 indicates those participants do not want moreinformation than needed.“The badge is distracting as it shares the same space as the source code (B1)”,despite its lower popularity among the three provided reason options, received aconsiderable number of selections. It indicates that badges occupying source codearea could become an issue.Below are some reasons provided by participants themselves:“when you test hundreds of time, the bars area might get too long.”— P7 in Scenario 1. P7’s concern is valid, and reveal the challengewhen there is too much information to put in a badge. I discuss this inChapter 7.“I like the icon in [the gutter].” — P27 in Scenario 3. This is anaesthetic issue I can address easily with a better icon.“this gutter info is clear and easy to read. Also, [...] there is a lotmore inline text in A that can really clutter up the text area.” — P11in Scenario 4.34Variance among scenariosAs shown in Figure 5.8, there are large variances among scenarios.First, the information density developers need varies. In the coverage scenario,the information developers need is binary (either covered or not covered), whichmakes it suitable for gutters. However, in the profiling scenario, the informationdevelopers need is more detailed and textual, making badges a better choice.My design choices for individual plugin mock-ups also have effect. For exam-ple, if the percentage is not shown as text but as a progress bar, P11 could not havecomplained “Percentage is too hard to read. And icon is unclear.”Which approach do you prefer to show multiple visual elements on the sameline?When multiple plugins add information to the same place, 76% of participantschoose badges over gutters (Figure 5.9).When asked about why they prefer badges1, many says it is cleaner, and hasroom for more information and better visualizations:“Visually clearer. All info displayed, compared to [gutters], whichonly displays icons.” — P2“It is easier to read on the right side.” — P11“[It is] better to visualize [in badges].” — P16“[Badges] seems more intuitive and less messy when there are un-indented code/comments/notes in the code” — P18“More information is displayed (and in a pleasing manner) making iteasier to draw accurate conclusions.” — P19“[Badges are] more intuitive and prominent. It does not feel annoyingdespite always there.” — P21“[Badges] seems more clean.” — P25“I think that there is already a lot of content in the left-hand side ofIDEs. The right-hand side is not used as much and could benefit from1Some comments are removed if they are only about aesthetics.35the badges.” — P26On the other hand, 24% participants prefer gutters as they are less likely tooverload information compared to badges:“[Gutters] feel less visually jarring; there’s only one line that expandshorizontally. [Gutters] also avoid an “explosion” of information andreminds me of a context menu (feels more familiar).” — P20“[Badges] really seem like an overload of information and would hardlyfit anymore if code lines are long and take up a big part of the screen.”— P28165Multi-plugin ManagementA (Badges)B (Gutters)Figure 5.9: Preferences on management of multiple plugins, badges (A) orgutters (B).When there are multiple pieces of information for the same code element,badges, despite occupying more space, are more popular because they can con-vey more information with better visualization.36RQ2.1: Ultimately, badges are slightly more popular than gutters, becausethey can display more information, and the space constraints in gutters couldresult in readability issue. In scenarios when the information needed for a taskcan be fitted into the gutters, participants prefer to use gutters, and they donot want more information than needed. These shows the tendency of moreinformation, but not superfluous amounts. Besides, badges are better at show-ing multiple pieces of information at the same time, which makes badges theoverall winner.5.3.2 Would you prefer to display the information in an extra panel?RQ2.2: Does information need to be inline or in a separate area?Figure 5.10 shows the preferences between showing the information in a separatepanel or inline with code. For all scenarios, most developers prefer to have theinformation shown in the code area (inline), but when there is the option, many ofthem also want a separate panel.Three rationales for this were provided: “[R1] I would like to keep the codearea clean”, “[R2] The extra panel is better at summarizing this information”, and“[R3]I don’t want too many panels.”In scenario 1 (History), 3 (Profiles), and 4 (Coverage), the sample size of pre-ferring the separate panel rather than inline presentation is too small to be statisti-cally significant.Table 5.4: Reasons participants “prefer the separate panel.”Reason option S1 (History) S2 (Test) S3 (Profiles) S4 (Coverage)[R1] I would like to keep thecode area clean3 (100%) 4 (57%) 2 (67%) 2 (100%)[R2] The extra panel is better atsummarizing this information1 (33%) 3 (43%) 3 (100%) 1 (50%)Other reasons 0 (0%) 1 (14%) 1 (33%) 0 (0%)Number of participants 3 7 3 2373128S1YesNo, but I want to keep itNoHistory7125S2YesNo, but I want to keep itNoTest3128S3YesNo, but I want to keep itNoProfiles2146S4YesNo, but I want to keep itNoCoverageFigure 5.10: Preferences for showing information in a separate panel for eachscenario.For scenario 2 (Test), P2 says only the separate panel has room for the usefulinformation in the circumstance: “Commit information + Diff information is moreimportant at diagnosing ’why’ than just knowing a specific line of code likely brokethe tests.”Participants do not find themselves frequently profiling (Scenario 3), P29 thinksa separate panel is easier to toggle off when it is not needed: “It can be seen onlyyou need the information.”38Why do you prefer about answering the code area while keeping the panel?While more than half of participants prefer the inline presentation but want to keepthe separate panel, most agree the later is better at summarizing the informationwith a top-down manner (67% to 93%, see Table 5.5.) Although P20 did notchoose [R2] in Scenario 4 (Coverage), P20 said: “Shows whole project coverageby file, while the gutter shows coverage by line.”Table 5.5: Reasons participants “prefer in the code area while keeping thepanel.”Reason option S1 (History) S2 (Test) S3 (Profiles) S4 (Coverage)[R2] The extra panel is better atsummarizing this information8 (67%) 9 (75%) 10 (83%) 13 (93%)Other reasons 4 (33%) 3 (25%) 3 (25%) 5 (36%)Number of participants 12 12 12 14Some participants think the separate panel has some capabilities that inlinepresentations do not have:“Extra panel can be sorted” — P25 in Scenario 3“It may be easier to jump across files with the panel. The bottom partof the panel also helps” — P26 in Scenario 4“Extra panel could in theory show extra information about each rele-vant commit in the bar chart.” — P6 in Scenario 1“Extra panel scrolls with text area” — P25 in Scenario 2. It is aninteresting presentation idea to make the extra panel display differentinformation based on the scroll position.A popular rationale is that the inline presentation is best for its minimalism,and a toggleable separate panel is best for more information only when needed:“Ability to hide the information when not required.” — P2 in Scenario1“Stay minimal until I ask for more” — P4 in Scenario 1“Keep it minimal and inline maybe like the ones offered in VSCode”39— P18 in Scenario 1“Ability to show the additional data when required.” — P2 in Scenario2“Stay minimal until I ask for more information” — P4 in Scenario 2“I prefer inline with the option to explore in an extra panel” — P18 inScenario 3“Ability to hide information until required” — P2 in Scenario 4“Both the inline tagging and the extra panel should be toggleable on& off ” — P19 in Scenario 4A few participants want to keep the separate panel because they like to haveoptions:“Nice to have options.” — P1 in Scenario 3“Good to have the option” — P1 in Scenario 4Why you prefer not to have a separate panel?29.3% of participants prefer not to have the separate panel at all, mostly becausethey do not want too many panels in their code editors (see Table 5.6.) P10 inScenario 3 (Profiles) does not want an additional panel because “[the information]is not lined up vertically with [the source code]. P6 in Scenario 4 (Coverage) thinksinline presentation is functionally more useful: “inline with the editor panel makesfor faster REPL workflow.”Table 5.6: Reasons participants “prefer not to have a separate panel.”Reason option S1 (History) S2 (Test) S3 (Profiles) S4 (Coverage)[R3] I don’t want too many panels 8 (100%) 5 (100%) 6 (75%) 5 (83%)Other reasons 1 (12%) 0 (0%) 2 (25%) 1 (17%)Number of participants 8 5 8 640RQ2.2: In conclusion, inline representations are much more popular be-cause developers are not fond of too many panels in their code editors. Butseparate panels are better at presenting summary data and have some capabil-ities inline representations do not have. The option to toggle on the separatepanels can be helpful when warranted.5.3.3 Why / how can external information visualizations beimproved?RQ2.4: What other visual preferences do developers have?Participants made many suggestions and comments from participants, and hereclassified into the following categories2:UI OptimizationsParticipants come up with interesting ideas to better present the information. Awell aligned user interface is less complex [2], and it is why P2, P13 and P29suggest “[badges is] too close (P29)”, “align [the badges] to the right (P2)”. P11concern that the badges might be “... conflict[ing] with ... other information (e.g.,Git blame/status in VS Code).” P28 has similar concern, and says “The info boxesshould absolutely not overlap the code [when there is] no white space on the right.”Some propose to use more visualization than texts. P28 suggest the profilingplugin to “use color or icons to encode the run-time percentage in the gutter insteadof the number [...]”, and P10 suggest to use “histogram, defaulting to log scaling[...].”Some participants prefer to have them be more eye-catching so it does notconfuse itself with comments or source code. They are more likely to find theinformation since it looks different from the surroundings:“Make it more prominent.” — P25“Make it more noticeable than a comment. ” — P232Aesthetic suggestions and ones that do not contain useful information have been removed.41“More distinct coloration. ” — P22“[...] I wish it had bigger font.” — P19“Maybe a different font-color for quick catch.” — P18On the other hand, some participants prefer the representation to be faded intothe background so that it is less intrusive:“B can use dull colors.” — P7“remove the background.” — P16“More subtle color and using an indicator beyond just color [...].” —P1Some participants want only the errors to be prominent. These participantswant to be alerted when there is something wrong, but left alone when there is noissue:“using a more contrast color for the potential error line ” — P5“Lightly color code the entire line that a test covers, making it reallyeasy to distinguish what has successful and failing test coverage.” —P19 in Scenario 4As the information is “only relevant at certain times during the development(P27)”, some think the ability to hide certain plugins is necessary:“[...] it should be possible to hide or replace [the information] withother task-relevant information.” — P27 in Scenario 4“Only show on hover maybe. This information is not needed all thetime. Also, it should include information about when the profiling wastaken.” — P1 in Scenario 3For large amounts of information, P26 proposes a method that “[...] some[...] bars that merge failed builds and you could click them to expand the bars [inScenario 1 (History)].” P1 suggest to “... add a bit more emphasis on the latestresult.”42At present, participants can only choose one of the gutter and the badge. P20visions a dynamic solution: “For a few per screen, A or B are fine, but if there werelots, a less detailed version might be preferable.” P5 also gave similar suggestions.Furthermore, P26 suggests a single plugin can utilize both badges and gutters:“[put] the failed [gutter] flag at the function level and the comments [badges] atthe line level.”Functional ImprovementsAlthough the online survey aims to investigate the visual presentational aspect ofEICE plugins, there are a few responses about functional aspects that are worthmentioning.“Instead of showing the full SHA, I’d like to see a popup with the diffand/or the build errors/test failures.” — P20 wanted more detailedinformation displayed in Scenario 2 (Test).“[...] it should include information about when the profiling was taken.”— P1 in Scenario 3. This suggestion is about transparency, partic-ipants in the follow-up study also raise this problem. For any infor-mation given by the plugins, developers should be able to know whenand how the information is generated.“[The number] of times covered is likely irrelevant” — P6 in Scenario4. P26 suggest the same: “These numbers can be really big in largeenough projects. Perhaps just some color coding? Green covered andpass. Red covered and failing.” It is true when the task is to improvethe coverage. I think the number of times covered is still useful in othercircumstances. For example, when the developer spots an abnormallarge number, there might be a bug.“Do it on the assembly, with awareness of conditional moves and com-parison results.” — P10 in Scenario 4. P10 suggest to make the cov-erage plugin work for assembly language. It is a bit off-topic here, buta language with little readability can really benefit from plugins likethis.43Participants have suggestions about what alternative information is more suit-able for the tasks. Some information might require more advanced processing (e.g.,complicated dynamic analysis), but it is feasible in theory:“[...] If there was a code error, I would want syntax-error like redsquiggles. If it is a test that fails, it should have information on thetest.” — P1 in Scenario 2.“[...] Did the covered code change, or is the test flaky. Having thiscontextual information available would be helpful.” — P20 in Sce-nario 1.“* annotate every line of code / every instruction with sample/timingcount * use [the number of ] samples or seconds of time, not %time.”— P6 in Scenario 3.RQ2.4: Participants made many suggestions to improve external informa-tion visualizations, including UI Optimizations (e.g., using visualization overtexts), functional improvements (e.g., more transparency) and proposing al-ternative information. These suggestions are very useful for the EICE futuredevelopment.5.3.4 How do you prefer to show persistent visual elements?RQ2.3: Do developers prefer persisting or toggleable visual elements?Few participants like intrusiveness or being “informed” all the time, as Figure 5.11shows participants’ preferences on the visibility of “persistent visual elements.”Most participants prefer toggleable. Only 10% of participants would like themalways visible. 48% prefer to only display them when actively turned on (by tog-gling) and 43% prefer a passive triggering mechanism (e.g., when the cursor entersthe area of interest).442109Persistent visual elementsAlways displayDisplay when explicitly toggled Display when the cursor enters the area of interest Figure 5.11: Preferences on when the persistent visual elements should bedisplayed.5.4 SummaryRQ2: What preferences do developers have for integrating data from ex-ternal information sources into their source code editors?By analyzing survey responses from 29 participants, I gathered developers’ visualpreferences on how the external information should be presented within their codeeditors. For the common presentation views (gutter and badges), 29 participants donot form a clear answer. From the responses of the follow-up questions, the choiceis insight-driven rather than a simple gutter-or-badge decision: participants preferhigher information density to reduce number of actions, but not higher than neededfor the task to avoid being overwhelming. Despite participants preference for theinline representation, they want to have the option to summon a separate panelto show the information in a top-down (or summary) manner. Most participants45would like the plugins to stay hidden by default, and be toggled on either explicitlyor implicitly when needed. The diversity of ideas made by participants implicitlyshows their interest in the idea of external information integration. This chapterprovides guidelines about how to best integrate external information into sourcecode editors.46Chapter 6User StudyThe online survey focused on visual aspects of EICE. To better understand de-velopers needs and investigate the functional aspect of EICE plugins, I conducteda follow-up in-person study. In this study, I aim to evaluate the benefits (RQ3),downsides (RQ4) and developers’ opinions on whether they would use EICE inpractice (RQ5). During the study, I can be engaged in in-depth conversation withdevelopers, taking the opportunity to discuss more about developers opinions onvisual representations (RQ2).The study performed was a controlled experiment. My hypothesis is that EICEbridges the gap between external information and the related source code, there-fore, developers can gain the same information in less time and with less effort. AsEICE plugins aggregate external information, it is a natural implication that theyreduce context switches. The three sub-questions of RQ3 are:RQ3.1 Can developers avoid context switching with EICE plugins?RQ3.2 Can developers save effort with EICE plugins?RQ3.3 What other benefits does EICE bring to developers?The study is a comparison between the traditional approach and the EICE ap-proach. Participants are asked to perform several tasks either in their traditionalway (the baseline group), or with EICE plugins (the EICE group). I will discussmore details in the next few sections.476.1 Selection of ParticipantsAt the end of the online survey, participants had the option to opt-in to the follow-up user study, and I invited them through email. In order to participate in thefollow-up study, participants needed to have experience with JavaScript for at leastone year.I conducted the follow-up study with 8 developers, 5 out of whom were meton-site in Vancouver. The average number of years of programming experienceis 7.5. Participants were entered into a draw for one of two CAD 50 Amazon orStarbucks gift cards.6.2 Study DesignThis section describes the study design and the metrics used to measure along withthe project participants work on and the details about the tasks.6.2.1 MetricsThe intuition is that EICE plugins can save developers time, but in the study, thetotal time to finish the task is not a reliable metric. Participants are not familiar withthe plugins, and are likely to spend time to figure out how to use them, which isa disadvantage to the EICE group. So without measuring the time, the experiencefocuses on the “effort.” These two metrics are used to reflect effort: the Numberof Actions (NA) and the Number of Context Switches (NCS). The more actionsa developer needs to take to complete a task, the more effort the task requires.The more context switches and programs involved, the more complex the processbecomes, the heavier cognitive load is.The most straightforward way to measure NA and NCS is to ask developers todo a task and count them during the experience. Due to unfamiliarity with the studyenvironment or the project, participants might ask for help during the tasks. Since Idon’t use the time for comparison, The flexibility to pause and have conversationsis possible and counteracts the effect of the unfamiliarity.In the process of comparing the performance when developers use EICE plu-gins against when they do the tasks without EICE plugins, I treat the two groupsdifferently. For the EICE group, I want to study the interactions participants expe-48rience with the plugins, with minimal guidance. For the the baseline group, thegoal, however, is to form consistent baselines, so it is more of a conversational pro-cess where the participants have the freedom to change their mind to form a moreoptimal workflow.6.2.2 Experimental: Baseline GroupIn this group, participants are asked to complete the task using a traditional ap-proach. My hypothesis is that the EICE group outperforms the baseline group,which means fewer context switches (NCS) (ideally 0) and fewer actions (NA) in-volved. As mentioned in the last paragraph, I aim to find the participants optimalsteps (lower bound) in the baseline group. The lower bound is determined through-out the user study, where I prepare an initial standard instruction list to completethe task, and participants can follow the instructions and then optimize workflow ifthey can. I provide another two options to find the optimal steps for a participant:1. Participants can choose to attempt the task without reading the provided stan-dard instructions, then they are given the opportunity to optimize their steps.I guide this process by asking them whether they can combine skills fromtheir own work experience. They can also talk about the tools they use ontheir tasks but not yet available on the study environment. Finding unneces-sary steps introduced by stress or unfamiliarity is also a goal of this step.2. Similar to the previous option, participants can directly skip actually per-forming the task, but instead simulate their steps in the interview. They canoptimize their steps at any time in this process.The improved workflow is the participant’s final baseline, which will be usedto compare against the EICE group.For inspecting coverage, a common traditional tool is Coveralls1. For profiling,a similar website does not exist. I find acquiring the profiling report is a non-trivial task, and not practical in a study which only has 60 minutes. I developeda web service, ProfileWell, for viewing a profiling report. ProfileWell is only usedin the baseline group and participants can inspect a simple profiling report in a1https://coveralls.io49traditional website form. The website starts with the list of builds and each’s status.ProfileWell displays the runtime information of functions in a bullet list, with themost time-consuming functions at the top.6.2.3 Experimental: EICE GroupIn this group, participants are asked to perform the task with the EICE plugins in-stalled. If the participants cannot finish the task within 10 minutes, it is considereda failure. With the help of EICE plugins, the number of switches (NCS) is expectedto be zero.First, to make sure the EICE plugins work as intended, I ask participants “Doyou have any confusion about EICE plugins?” I can also use the answers here toexplain the quantitative results, in case they find frictions when interacting with theplugins.I then ask participants “Do you feel the interface overwhelming or get in yourway?” for RQ4.Participants might have some ideas to improve the plugin design, and I canunderstand what they actually need when they express their ideal versions of theplugins. They might suggest some modifications to the previous two questions, Ido not want to miss any. So, I ask “Do you have any other suggestions to presentthe information you need in this task better?”Last, to form an answer for RQ5, I ask them “Will you use EICE plugins overtraditional approaches for this task?”6.2.4 Project PreparationBelow are 5 criteria for project selection:1. It is written in JavaScript because the plugins are developed for JavaScriptprojects,2. It has at least one-year development history with at least 200 commits fromat least 10 contributors, to indicate a good code quality,3. It is covered by a decent number of unit tests,50Task Task Name Task Description Plugins that are likely to beinvolved1 Code Comprehension Can you describe the meaningof encoded and what doeslinearizeEncodings() do to it?Runtime-Value2 Fix the bug that failed therecent buildsSome tests started to fail atsome point. Can you locatewhich changes result in thefails and describe a fix?Test-History, Build-Break3 Improve coverage There are lines of code thatare not covered by tests, canyou describe a unit test toimprove the coverage?Coverage4 Optimize performance The program has a perfor-mance issue that results in avery slow test run. Can youlocate the issue and describe afix?ProfilingTable 6.1: Task descriptions and plugins that can help with the tasks.4. It uses common build and test routines, with minimal customization config-uration,5. It has a reasonable directory hierarchy with depth between 3 to 5, and num-ber of files between 20 to 40.In the end, I picked the Barcode generation library, JsBarcode 2 as the foun-dation project for my study. JsBarcode has 24 source files, in 3-layer hierarchicaldirectories. It is well tested by 102 unit tests. 24 developers have been involved inthis project contributing 615 commits from 2012 to 20193.6.2.5 TasksThere are 4 tasks in total, each independent from the others. Each task has somefollow-up questions, and after participants complete all 4 tasks, the study is con-cluded with two final questions. Table 6.1 shows four tasks, as well as the pluginsdesigned to be evaluated. The tasks cover code comprehension, bug fixing, testing,and optimization. Participants do not need to actually write any source code.I compare the performance of participants with EICE plugins (the EICE group)against completing the tasks without EICE plugins (baseline). The approximate2https://github.com/lindell/JsBarcode3The project and contribution data was collected on August 5, 2019.51Task 1Task 2Task 3Task 4Task ATask BTask CTask DShuffleEICEBASEEICEBASEEICEBASEFigure 6.1: Randomization process of tasks and grouping.time of completing a task with follow-up questions is 10 minutes. To reduce learn-ing effects, I randomly assign 2 tasks to the baseline group and the other 2 to theEICE group. Combined with the introduction section and end-study interview, thetotal duration of the study is concluded within 60 minutes.Randomization & Counter-balancingThere are two levels of randomness: (1) randomness of the task order, and (2)randomness of the treatment (EICE or BASE). For each participant, there are twoEICE tasks and two BASE tasks. To treat tasks fairly, there is counter-balancing tomaintain the total number of EICE and BASE close for each task (see Figure 6.1).6.2.6 Follow-up QuestionsAt this point, participants should have a good sense of what EICE plugins canbring to them. I continue with an open question “Do you think you will benefitfrom this code editor augmentation? How about the opposite, do you think in somecircumstance, it will decrease your productivity?” to form collective answers forRQ3.3 and RQ4.Finally, I wrap the study up with the last open question “Do you have anyother thoughts about augmenting source code with external information in codeeditors?”526.3 User Study ResultsHere, I summarize the results of the experimental study.6.3.1 EICE BenefitsThe screen-recorder records participants’ workflow in completing the tasks, and bywatching the playback, I count the number of context switches (NCS) and actions(NA).Counting context switches. I define a context switch as the developer switchingfrom one type of interface to another different type of interface, because developersneed to process the visual differences. When developers switch between sourcecode files in the code editor, the contexts do not differ much. When developersswitch tabs in the web browser, the contexts can be quite different. Therefore,when developers switch between files, switches count. When developers switchbetween different windows, switches count. When they switch between browsertabs, if the two pages are from different websites, switches count, otherwise, onlythe action is counted. Table 6.2 is an example of how the actions and contextswitches are counted. More examples can be found in Appendix B of the proposedinstructions.Supplemental actions As mentioned above, participants do not need to code inthe study to make the duration of the study within a reasonable range. To make theworkflow complete, some actions are appended to the end, such as “Fix the bug.”Besides, a task should start and end inside the Atom editor, if the workflow endsoutside Atom, an supplemental action named “Switch back to Atom” is added.RQ3.1: Can developers avoid context switching with EICE plugins?Figure 6.2 shows the comparison of the number of context switches betweenthe baseline group and the EICE group.EICE plugins are designed to satisfy developers’ external information needswithin the code editor, and to be context switch free. Ideally the number of context53Description Destination Action CS1 Click the EICE notification view item Editor (Overlay) ◦2 Click the latest commit Browser (GitHub) ◦ ◦3 Scroll to the diff Browser (GitHub) ◦4 Switch back to Atom Editor (Overlay) ◦ ◦5 Click the oldest commit Browser (GitHub) ◦ ◦6 Scroll to the diff Browser (GitHub) ◦*7 (Bug found) switch back to Atom Editor (Overlay) ◦ ◦*8 Close the overlay Editor ◦*9 Navigate to the file Editor ◦*10 Navigate to the line Editor ◦*11 Fix the bug Editor ◦Total 11 4Table 6.2: The workflow recorded in Task 2, as well as action and contextswitch count. Supplemental actions are marked with “*.”switches in the EICE group should be zero, which is the case for task 1, 3 and 4,where the baseline group shows 2 or even 4 context switches.“If I’m already working on something I don’t need to worry aboutswitching to all different applications.” (P4)However, for Task 2, the EICE group does not achieve zero context switches,and on the contrary, it has more context switches than the baseline group. It ismainly caused by a design flaw in the interface of Build-Break plugin. The Build-Break plugin shows possible commits and file changes that related to the buildfailures, and I add external links that link to commit webpage in case developerswant to check to information source. The external link that opens commit in thebrowser looks similar to the internal link that jump to the source files, and theformer is more prominent and accessible. Therefore, participants clicked on theexternal link to look at the diff, which results in a context switch, but this is not howthe plugin was designed to be used. This problem can be avoided by annotatingexternal links with an icon “ ” and making it less prominent compared to internallinks.54Task 1 (Baseline)Task 1 (EICE)Task 2 (Baseline)Task 2 (EICE)Task 3 (Baseline)Task 3 (EICE)Task 4 (Baseline)Task 4 (EICE)01234Number of context switches (NCS)Figure 6.2: Comparison of the number of context switches (NCS) betweenthe baseline group and the EICE group for all tasks.RQ3.2: Can developers save effort with EICE plugins?Figure 6.3 shows the comparison of the number of actions between the baselinegroup and the EICE group.The ability to integrate the external information to the source code saves de-velopers the effort of code navigation, letting them directly jump from the problemto the source of the problem with one click. For Task 1, 3 and 4, the EICE groupreduces about 30-60% actions.Due to the same reason mentioned in the previous section, the number of ac-tions to complete Task 2 is comparative or higher in the EICE group. These actionsresulted from participants clicking the external links, switching between Atom andthe browser.RQ3.3: What other benefits does EICE bring to developers?“I think that they add value [...]” (P1) Most developers agree EICE adds value,55Task 1 (Baseline)Task 1 (EICE)Task 2 (Baseline)Task 2 (EICE)Task 3 (Baseline)Task 3 (EICE)Task 4 (Baseline)Task 4 (EICE)4681012Number of actions (NA)Figure 6.3: Comparison of the number of actions (NA) between the baselinegroup and the EICE group for all tasks.depending on the tasks. Below I summarize the benefits participants think they canreceive from EICE plugins.First, EICE can reduce the number of applications developers need to install(have) for a task. “as soon as [I need to] run a test, pressing [the Runtime-Valuebutton] would be valuable. [And ] it is good not have to install profiler or otherthings, because it is really handy.” (P1)EICE could keep developers up to date in a non-intrusive way. “[...] as I’mworking, if [EICE] notices these problems, it might give me a cue to there’s gonnabe problems, so I need to go and look at this. [Even] I’ve disabled communications,and I’m working, but I still see that a really relevant information because it wouldbe showing me in the code that I’m working on. So that can be quite helpful.” (P4)Another advantage of EICE is the information is continuously processed. Be-cause of the processing happens when a commit is pushed, the information is pre-processed when participants need it. “It is faster and more convenient than thedebugger,” said P6 about the Task 1 (Code Comprehension), “the runtime valuesshow up instantly, which saves a lot time.”56Sometimes an EICE plugin creates a new solution for some tasks. For ex-ample, for the Task 1 (code comprehension), compared to a debugging methodwhere developers can only inspect one execution of the line at a time, P5 finds “[Ican] parallel debug multiple [executions] because I [can] see the values of multipleiterations.” (see Figure 6.4 for the interface of the EICE plugins P5 is describ-ing.) The Test-History plugin, which lets developers peek historical behaviors ofindividual unit tests, is another example. Normally, developers need to manuallyiterate through the entire build history to acquire such information, and it makesdevelopers not want to even try to acquire it. The Test-History plugin that makesthis insight accessible.Figure 6.4: The interface of Runtime-Value plugin, where values of variableof interest is listed for different executions.6.3.2 EICE DisadvantagesRQ4: Are there negative consequences to integrating external informationsources into the IDE?One of the concerns with EICE is that the visual elements it adds to the editormight distract developers. In the interview, many agree those visual elements are57noticeable but they (participants) are mostly ok with them, especially when “havingthe option to toggle on and off [the visual elements] (P1).”“If I was using it for a really long time? I mean, I guess Yeah,” said P4,“it’s not really that in your face, nothing’s like, bouncing or anything.So okay. It’s not too bad. [...] Usually, I expect these kinds of badgesto show up and I’m running something like a debugger or a profilingtool, and then disappear when I’m done running it. So maybe havinga toggle on will be helpful, but certainly it’s not.”In the current EICE design, when there is information that represents an error,it adds items in both the notification view and inline. Although the way EICEdelivers notification is not disruptive, it can still distract developers in some way.Some participants prefer to remove the inline representation for error scenarios.“[the inline representation] might be too distracting in some cases”, said P3, “I getkind of notified, but in a non disruptive way, [... but] if I’m actually working onsomething, and it keeps telling me that it has broken build, I might get annoyed.”Another concern participants express is that the information might be out-of-date. This is one disadvantage of having the information preprocessed. Because theprocessing happens in the cloud, triggered by a commit event, the information isonly guaranteed to be valid for the commit triggering it. When developers modifythe code, the information becomes instantly outdated4. P8 suggested “[EICE] needa local CI that tries to generate the information whenever a change is made tokeep [the information] up to date.” This is not practical because the build, test, andanalysis take a considerable amount of time to complete. The current workaroundis when developers change a line of code, the information associated with that lineof code is invalidated and hidden. It is a compromise, and there should be a betterway to handle it. P4 envisions a mechanism that can “magically compute thesevalues quickly [by] not need to run the whole thing.” Theoretically, static analysisshould be able to figure out what information is invalid and EICE only needs togenerate that part of the information to save time.4Although some pieces of information are still valid because they are not relevant to the modifi-cations, developers cannot trust them any more.586.3.3 Would Developers Use EICE in Practice?RQ5: Will developers use EICE over traditional methods for external in-formation acquisition in practice?Most participants say they want to use it when the EICE plugins exist, because ofthe aforementioned benefits, such as fewer tool installation, instant inspection andnon-intrusive notification.There are some concerns, mainly due to the questionable matches betweenplugins and tasks. Many of these can be easily resolved by presenting suitableinformation.“ I probably won’t take this shortcut (runtime-value plugin) to under-stand a variable. I would rather read [the code] because some casesmaybe not covered by tests, and [the information provided by EICEplugins] may not be complete. ” — P8 on the code comprehensiontask“Ideally, you want to get profiling data from real life users, that mightbe more realistic, unlike a cloud or on a server.” — P3 on profilingusing the unit testTransparency is one of the factors that affects participants’ decisions. In theplugin designing process, the source of the information is usually presented alongwith the information itself, like the links to the online coverage report, the profilingreport and the commit page. Participants think it needs to be more transparent togain their trust. “It was a little unclear to me how it pick these [commits],” said P6,“I think sometimes developers don’t trust these kind of tools, [...] at least it has tobe very clear how it’s processed.”6.4 SummaryIn this chapter, I investigate the functional aspects of the EICE plugins. From theinteractions with my 8 participants, the number of context switches can be reduced59down to zero with EICE plugins. The number of actions, which used to reflectdevelopers’ effort, is also cut to around half for the evaluated tasks. With EICE’sbenefits such as fewer tool installations, notifying ability and preprocessing, someproblems also emerged. The potential distraction and whether the information isup-to-date and trustworthy need to be addressed. Overall, participants would liketo use EICE plugins rather than their traditional counterparts, which demonstratesthe functional benefits of EICE plugins.60Chapter 7DiscussionIn this section I discuss threats to validity and future work.7.1 Internal ValidityI used a number of actions to infer developers’ effort in the controlled study. Someactions are more difficult than others and invisible actions (like thinking) could notbe captured.The follow-up study has a certain degree of moderator bias, since participantswere in the author’s personal networks. I believe it is very unlikely that participantsresponded in favor of the author just to influence the results positively.Some of the EICE plugins went through minor interface changes after I con-ducted the follow-up study with the first 2 participants. The interface changes wereto address some confusion and make it faster for participants to get the information.Since time is not one of the metrics, the impact is minimal.The task goals were not clarified when I conducted the follow-up study withthe first 2 participants, as they got stuck in the code comprehension process, whichresulted in timeout and a task fail. Code comprehension is not relevant in thefollow-up study at all, so I made the change that if code comprehension is the finalstep, it can be skipped, and if the code comprehension is in the middle, adequatehints would be given to reduce the time and effort of code comprehension to beminimal. For example, in Task 4, when participants start to inspect the buggy61function, the task is completed assuming they are able to find the bug after effective“comprehension.”There is a question for each scenario in the online survey asking “Q3: Willyou change your mind if the following modifications were made?” This questionwas designed to consolidate participants’ preference by eliminating some irrele-vant factors. However, a follow-up question like “Q3.5: If you change your mind,what are the reasons you prefer [option after changing mind]?” should be added.Because the follow-up question (Q3.5) was missing, the responses of Q3 are notused for analysis. Although the analysis did not utilize all the data available, I didnot selectively omit the responses of Q3 because of they might negatively influencethe result.7.2 External ValidityThe primary threat to external validity is the sample size. Also only a single systemis evaluated for selected tasks.In the survey and follow-up study, the participants I recruited may not be repre-sentative of all developers in practice. The participants of the online survey mainlycome from social network or university and few from industry contacts. From theexperience distribution in Figure 5.7, I managed to have a fairly full range of ex-perience from junior to senior (15+ years) developers. For the follow-up study,participants have the experience of up to 12 years.The base project (JsBarcode.js) might not represent projects of all types andscales. The tasks selected, although covering a variety of different type of tasks,might not represent all tasks developers perform in their workflows.7.3 Future WorkIn this section, I discuss the future directions.Plugin Improvements. There is work that can be done to make EICE plugins moreusable for real-life scenarios. The transparency issue was raised by several partic-ipants in the survey and the follow-up study. It is essential to build trust betweendevelopers and the plugins, therefore, when and how the information is generated62should be presented wherever possible.Run Local Copy of EICE Backends. CI, as EICE’s backend, serves as the datasource and generates the raw information for plugins to retrieve. The raw informa-tion is only generated when there is a new commit detected, and study participantsfind it may be an issue. Having a local version of the backend to generate the rawinformation more frequently and efficiently can alleviate the problem.Bundle Plugins according to Tasks. A task only requires a subset of EICE plu-gins, so being able to automatically toggle on those plugins that are useful to thetask and hide those unneeded is a handy feature. It can be done either by build-ing a collection of tasks and their associated plugins, or using machine learning todecide which plugins are useful based on developers’ past toggling actions.63Chapter 8Related WorkPrior work has explored various methods to deliver external information to de-velopers, including (1) software visulization, (2) tools adding separate displays tocode editors, and (3) in-line code editor augmentation.8.1 Software VisualizationSoftware visualization provides another perspective to aid code understanding.Prior work such as Software Landscapes [3], Software World [15], ComponentCity [5], and CodeCity [23, 24] visualizes software, allowing for data explorationand analysis. Such approaches replace normal code editor views, asking develop-ers to completely concentrate on the visualization. I believe allowing developers tonot leave their code editors can bring numerous benefits, such as reducing contextswitches.8.2 Adding Separate Display to Code EditorsThere are previous studies that attempt to adding “side views” to code editors topresenting external information. Reacher, an Eclipse plug-in, shows the call graphof methods of interest next to the code editor view [18]. Software terrain maps,based on the metaphor of cartographic maps, provides an additional visualizationdisplay for developers to navigate around the source code in the editor [8].Code bubbles [4], Code canvas [9] and Debug canvas [10] work on a differ-64ent path: instead of adding a “size view,” they create a larger view where sourcecode views are enclosed and become parts of it. They can further display rela-tions among these small source code parts, easing comprehension and debuggingprocess. Despite developers perceived significant value in such interfaces, theyintroduce extra scalability issues I do not have [4].8.3 In-line AugmentingTools such as HATARI [22], GitLens1, CodeMetrics2, Whyline [16] and Perfor-manceHat [7] integrate external information by adding visual changes (e.g., an-notations) directly to the source code, without (necessarily) providing a separateindependent display. HATARI plug-in for Eclipse adds annotations to source codeelements, using colors to indicate risk levels of code changing [22]. GitLens usesannotations including highlight and heat-map to seamlessly integrate Git core fea-tures in the code editor itself. CodeMetrics shows the complexity informationabove the function declarations. Whyline is a tool developed recently to allowdevelopers to select questions about a program’s output and work backwards fromoutput to its causes [16]. PerformanceHat creates operational awareness of per-formance problems and integrates runtime performance traces into source codes,displaying the performance statistics/impacts in the form of overlays [7].This type of tool is closest to this study, where developers do not have to leavethe interface of the editor they are familiar with. I take a step further to use onesingle tool to satisfy multiple external information needs, which drastically reducesthe number of tools developers need to find and master to one. With proper centralmanagement of external information, developers can find the information they needwithout too much effort, in the meantime, not overwhelmed by the clutter resultingfrom likely visual conflicts among multiple plug-ins.A VSCode feature called CodeLens aids the development of GitLens and Code-Metrics. The essence of CodeLens is “actionable contextual information inter-spersed” in the source code, in the form of clickable links located throughout the1a plug-in for VSCode that help visualize code authorship at a glance via Git blame annotationsand code lens. The project is available at https://github.com/eamodio/vscode-gitlens2a plug-in for VSCode that computes and shows the code complexity next to the function. Theproject is available at https://github.com/kisstkondoros/codemetrics65source code. CodeLens contains a subset of EICE features.66Chapter 9ConclusionsIn this thesis, I describe a technology probe to explore how to integrate externalinformation into source code editors. An online survey with 29 responses showsdevelopers’ visual preferences: they prefer more information, but not superfluousamounts. They prefer to place information next to the code, while also havingthe option to show a top-down presentation in a separate view. Developers pre-fer the information to only show up when they need them, so it does not distractthem when they are working on something else. Five EICE plugin prototypes areimplemented to provide hands-on experience for developers in a follow-up study.Through 4 different tasks, the follow-up study demonstrates that EICE plugins re-duce developers’ effort to acquire external information, simplifies the toolset, andkeeps developers up to date. EICE plugins also have disadvantages such as po-tentially distracting developers in some cases. Ultimately, this thesis demonstratedthat with careful design, external information could be integrated into source codeeditors in a way that developers found useful and that has the potential to decreasethe number of actions required to complete development.67Bibliography[1] N. A. Al-Saiyd. Source code comprehension analysis in softwaremaintenance. In 2017 2nd International Conference on Computer andCommunication Systems (ICCCS), pages 1–5, 2017.doi:10.1109/CCOMS.2017.8075175. → page 9[2] K. Alemerien and K. Magel. GUIEvaluator: A Metric-tool for Evaluatingthe Complexity of Graphical User Interfaces. page 6. → pages 5, 34, 41[3] M. Balzer, A. Noack, O. Deussen, and C. Lewerentz. Software landscapes:Visualizing the structure of large software systems. In IEEE TCVG, 2004.→ page 64[4] A. Bragdon, S. P. Reiss, R. Zeleznik, S. Karumuri, W. Cheung, J. Kaplan,C. Coleman, F. Adeputra, and J. J. LaViola Jr. Code bubbles: rethinking theuser interface paradigm of integrated development environments. InProceedings of the 32nd ACM/IEEE International Conference on SoftwareEngineering-Volume 1, pages 455–464, 2010. → pages 64, 65[5] S. M. Charters, C. Knight, N. Thomas, and M. Munro. Visualisation forinformed decision making; from code to components. In Proceedings of the14th international conference on Software engineering and knowledgeengineering, pages 765–772, 2002. → page 64[6] J. Chisan. Towards a model of awareness support of software developmentin GSD. In ”Third International Workshop on Global Software Development(GSD 2004)” W12S Workshop - 26th International Conference on SoftwareEngineering, volume 2004, pages 28–33, 2004. doi:10.1049/ic:20040309.→ page 8[7] J. Cito, P. Leitner, C. Bosshard, M. Knecht, G. Mazlami, and H. C. Gall.PerformanceHat: augmenting source code with runtime performance tracesin the IDE. pages 41–44, 2018. doi:10.1145/3183440.3183481. → pages1, 11, 15, 6568[8] R. DeLine. Staying Oriented with Software Terrain Maps. In DMS, pages309–314, 2005. → page 64[9] R. DeLine and K. Rowan. Code canvas: zooming towards betterdevelopment environments. In Proceedings of the 32nd InternationalConference on Software Engineering - ICSE ’10, volume 2, page 207, 2010.doi:10.1145/1810295.1810331. → page 64[10] R. DeLine, A. Bragdon, K. Rowan, J. Jacobsen, and S. P. Reiss. DebuggerCanvas: Industrial experience with the code bubbles paradigm. pages1064–1073, 2012. doi:10.1109/ICSE.2012.6227113. → page 64[11] M. Eaddy, A. Aho, G. Antoniol, and Y.-G. Gueheneuc. CERBERUS:Tracing Requirements to Source Code Using Information Retrieval,Dynamic Analysis, and Program Analysis. In 2008 InternationalConference on Program Comprehension, pages 53–62, 2008.doi:10.1109/ICPC.2008.39. → pages 10, 14[12] H. C. Estler, M. Nordio, C. A. Furia, and B. Meyer. Awareness and MergeConflicts in Distributed Software Development. In International Conferenceon Global Software Engineering, pages 26–35, 2014.doi:10.1109/ICGSE.2014.17. → page 8[13] T. Fritz and G. C. Murphy. Using information fragments to answer thequestions developers ask. In Proceedings of the International Conference onSoftware Engineering, pages 175–184, 2010.doi:10.1145/1806799.1806828. → pages 1, 8, 10, 11, 13, 14[14] T. Jackson, R. Dawson, and D. Wilson. The cost of email interruption.Journal of systems and information technology, 5(1):81–92, 2001. → page 3[15] C. Knight and M. Munro. Virtual but visible software. In Conference onInformation Visualization. An International Conference on ComputerVisualization and Graphics, pages 198–205, 2000.doi:10.1109/IV.2000.859756. → page 64[16] A. J. Ko and B. A. Myers. Finding causes of program output with the JavaWhyline. In Proceedings of the international conference on Human factorsin computing systems, pages 1569–1578, 2009.doi:10.1145/1518701.1518942. → pages 1, 11, 14, 65[17] A. J. Ko, R. DeLine, and G. Venolia. Information needs in collocatedsoftware development teams. In Software Engineering, 2007. ICSE 2007.29th International Conference on, pages 344–353, 2007. → page 1069[18] T. D. LaToza and B. A. Myers. Visualizing call graphs. In Symposium onVisual Languages and Human-Centric Computing (VL/HCC), pages117–124, 2011. doi:10.1109/VLHCC.2011.6070388. → page 64[19] E. Murphy-Hill, R. Jiresal, and G. C. Murphy. Improving softwaredevelopers’ fluency by recommending development environmentcommands. In Proceedings of the International Symposium on theFoundations of Software Engineering - FSE ’12, pages 1–11, 2012.doi:10.1145/2393596.2393645. → page 1[20] M. P. O’Brien. Software Comprehension – A Review & Research Direction.page 29. → page 9[21] J. Sillito, G. Murphy, and K. De Volder. Asking and Answering Questionsduring a Programming Change Task. Transactions on Software Engineering,34(4):434–451, 2008. ISSN 0098-5589. doi:10.1109/TSE.2008.26. URLhttp://ieeexplore.ieee.org/document/4497212/. → pages 1, 9, 10, 11, 13, 14[22] J. S´liwerski, T. Zimmermann, and A. Zeller. HATARI: raising riskawareness. In Proceedings of the European software engineering conferenceheld jointly with international symposium on Foundations of softwareengineering - ESEC/FSE-13, page 107, 2005.doi:10.1145/1081706.1081725. → page 65[23] R. Wettel. Scripting 3d visualizations with codecity. In FAMOOSr’08:Proceedings of the 2nd Workshop on FAMIX and Moose in Reengineering,2008. → page 64[24] R. Wettel and M. Lanza. CodeCity: 3d visualization of large-scale software.In Companion of the international conference on Software engineering,pages 921–922, 2008. doi:10.1145/1370175.1370188. → page 64[25] S. Winkler and J. von Pilgrim. A survey of traceability in requirementsengineering and model-driven development. Software & Systems Modeling,9(4):529–565, 2010. ISSN 1619-1366, 1619-1374.doi:10.1007/s10270-009-0145-0. URLhttp://link.springer.com/10.1007/s10270-009-0145-0. → page 1070Appendix AQuestionnaireBelow is the full online survey questionnaire.A.1 Page 1: BackgroundA.1.1 How many years have you been programming?# 1# 2# ...# 14# 15+A.1.2 Have you ever worked as a professionalprogrammer/developer?# Yes# No71A.1.3 How many years have you been a professionalprogrammer/developer?# 1# 2# ...# 14# 15+A.1.4 Which IDEs or code editors do you use? Eclipse IntelliJ IDEA or JetBrains’ other IDEs VSCode Atom Sublime Text OtherA.2 Page 2: Scenario 1Scenario description:You are facing some test case failures in a well-tested project. To un-derstand the importance of these failures, you are trying to understand thehistorical behavior of failing tests. Below are two representations (A and B)of historical test results.72* By clicking the barcharts in (A) or the gutter icons in (B), the pop-up shown below will appear:A.2.1 Which representation do you prefer?# A (→ A.2.2: Case 1)# B (→ A.2.2: Case 2)A.2.2 Which of the following reasons explain your choice?Case 1 “A” provides more information at a glance and saves me from the extra step (click) “B” occupies gutter area, which is used for setting breakpoints Other (please describe):Case 2 “A” is distracting as it shares the same space as the source code “B” is easier to spot since they align well73 “B” already provides enough information for most of the scenario, it is good for me whilekeep visually minimal Other (please describe):A.2.3 Why / how can this visualization be improved?A.2.4 Would you prefer to display the information in an extra panel,away from the code area, like the one shown in the screenshotbelow?# Yes, I prefer the extra panel (→ A.2.5: Case 1)# I want the inline representation with the option to show the extra panel (→ A.2.5: Case 2)# No, I prefer the inline representation (→ A.2.5: Case 3)A.2.5 Which of the following reasons explain your choice?Case 1 I would like to keep the code area clean The extra panel is better at summarizing this information Other (please describe):74Case 2 The extra panel is better at summarizing this information Other (please describe):Case 3 I don’t want too many panels Other (please describe):A.3 Page 3: Scenario 2Scenario description:You are working on a project with version control and continuous integration.After you make a commit, or you pull commits, you want to know whether some changebreaks the build or fails some tests. Below are two representations (A and B) of locatingchanges that are likely to cause builds to break.* By clicking the badge in (A) or the gutter icon in (B), the pop-up shown below will appear:A.3.1 Which representation do you prefer?# A (→ A.3.2: Case 1)# B (→ A.3.2: Case 2)75A.3.2 Which of the following reasons explain your choice?Case 1 “A” provides more information at a glance and saves me from the extra step (click) “B” occupies gutter area, which is used for setting breakpoints Other (please describe):Case 2 “A” is distracting as it shares the same space as the source code “B” is easier to spot since they align well “B” already provides enough information for most of the scenario, it is good for me whilekeep visually minimal Other (please describe):A.3.3 Why / how can this visualization be improved?A.3.4 Would you prefer to display the information in an extra panel,away from the code area, like the one shown in the screenshotbelow?76# Yes, I prefer the extra panel (→ A.3.5: Case 1)# I want the inline representation with the option to show the extra panel (→ A.3.5: Case 2)# No, I prefer the inline representation (→ A.3.5: Case 3)A.3.5 Which of the following reasons explain your choice?Case 1 I would like to keep the code area clean The extra panel is better at summarizing this information Other (please describe):Case 2 The extra panel is better at summarizing this information Other (please describe):Case 3 I don’t want too many panels Other (please describe):A.4 Page 4: Scenario 3Scenario description:While working on a source file in your project, you are trying to figure out what part of thecode is executing slowly. Below are two representations (A and B) of profiling information.77* By clicking the badge in (A) or the gutter icon in (B), the pop-up shown below will appear:A.4.1 Which representation do you prefer?# A (→ A.4.2: Case 1)# B (→ A.4.2: Case 2)A.4.2 Which of the following reasons explain your choice?Case 1 “A” provides more information at a glance and saves me from the extra step (click) “B” occupies gutter area, which is used for setting breakpoints Other (please describe):Case 2 “A” is distracting as it shares the same space as the source code “B” is easier to spot since they align well “B” already provides enough information for most of the scenario, it is good for me whilekeep visually minimal Other (please describe):A.4.3 Why / how can this visualization be improved?78A.4.4 Would you prefer to display the information in an extra panel,away from the code area, like the one shown in the screenshotbelow?# Yes, I prefer the extra panel (→ A.4.5: Case 1)# I want the inline representation with the option to show the extra panel (→ A.4.5: Case 2)# No, I prefer the inline representation (→ A.4.5: Case 3)A.4.5 Which of the following reasons explain your choice?Case 1 I would like to keep the code area clean The extra panel is better at summarizing this information Other (please describe):Case 2 The extra panel is better at summarizing this information Other (please describe):Case 3 I don’t want too many panels Other (please describe):79A.5 Page 5: Scenario 4Scenario description:While working on a source file in your project, you want to know which lines are coveredby test cases and the current status of those tests. Assuming we developed a tool to extractthe coverage status of the current build.* By clicking the badge in (A) or the gutter icon in (B), the pop-up shown below will appear:A.5.1 Which representation do you prefer?# A (→ A.5.2: Case 1)# B (→ A.5.2: Case 2)A.5.2 Which of the following reasons explain your choice?Case 1 “A” provides more information at a glance and saves me from the extra step (click) “B” occupies gutter area, which is used for setting breakpoints Other (please describe):80Case 2 “A” is distracting as it shares the same space as the source code “B” is easier to spot since they align well “B” already provides enough information for most of the scenario, it is good for me whilekeep visually minimal Other (please describe):A.5.3 Why / how can this visualization be improved?A.5.4 Would you prefer to display the information in an extra panel,away from the code area, like the one shown in the screenshotbelow?# Yes, I prefer the extra panel (→ A.5.5: Case 1)# I want the inline representation with the option to show the extra panel (→ A.5.5: Case 2)# No, I prefer the inline representation (→ A.5.5: Case 3)81A.5.5 Which of the following reasons explain your choice?Case 1 I would like to keep the code area clean The extra panel is better at summarizing this information Other (please describe):Case 2 The extra panel is better at summarizing this information Other (please describe):Case 3 I don’t want too many panels Other (please describe):A.6 Page 6: Non-Scenario PreferencesThe gutter or inline badges in the previous questions are “persistent visual elements”.A.6.1 How do you prefer to show these “persistent visual elements”?# Always display# Display when explicitly toggled (e.g., via shortcuts or context menu items)# Display when the cursor enters the area of interest (line of code)One challenge with these visual elements is how to show multiple of them at the same time.Below are two proposed approaches for managing multiple visual elements on the same line.A.6.2 Which approach do you prefer to show multiple visualelements on the same line?# A (Badges)# B (Gutters)82Figure A.1: Management of multiple pieces of information presented asbadges.Figure A.2: Management of multiple pieces of information presented as gut-ters.A.6.3 Can you tell us why you made this choice?A.7 Page 7: Follow-up StudyBefore we end the survey, here is a few quick question about the follow-up study.The follow-up study contains three parts:(a) a 10-minute long introduction session in which we will explain the study to you, askfor your consent and install Atom code editor along with some plug-ins including EICEplug-ins and a screen recording tool. If you can do the study on-site, we will provide a83MacBook Air with all software pre-installed.(b) a one-hour long task session during which we will ask you to complete 4 tasks. Aftereach task we will ask you several questions.(c) a 10-minute long follow-up interview after the task session, in which we will ask youabout the experience you have with the tasks and other feedback related to EICE plug-ins.All follow-up participants will be entered in a draw for one of two CAD 50 Amazonor Starbucks gift cards after the follow-up study.A.7.1 Are you interested in participating the follow-up study?* The follow-up study will be conducted in person (in Vancouver, Canada) or via Skype. YesA.7.2 Please leave your email address below so we can get in touch.A.8 Page 8: CompletionThank you so much for your time!Want to know more about our research projects?Software Practices LabUniversity of British Columbia84Appendix BProposed InstructionsB.1 Task 1: Code ComprehensionDescription Destination Action CS1 Add “print” code before line 99 Editor ◦2 Add “print” code after line 99 Editor ◦3 Open a Terminal Window Terminal ◦ ◦4 Navigate to project directory Terminal ◦5 Run the program with commands Terminal ◦6 Inspect the output (and comprehend) Terminal ◦*7 Switch back to Atom Atom ◦ ◦Total 7 2Table B.1: The proposal workflow in Task 1, as well as action and contextswitch counts. Supplemental actions are marked with “*.”85B.2 Task 2: Fix Bug that Failed the Recent BuildsDescription Destination Action CS1 Open browser Browser ◦ ◦2 Open project CI page Browser (CI) ◦3 Open oldest build since fail Browser (CI) ◦4 Click commit SHA Browser (GitHub) ◦ ◦5 Scroll to the diff Browser (GitHub) ◦6 (Issue found) *switch to atom Editor ◦ ◦*7 Navigate to the file Editor ◦*8 Navigate to the line Editor ◦*9 Fix the bug Editor ◦Total 9 3Table B.2: The proposal workflow in Task 2, as well as action and contextswitch counts. Supplemental actions are marked with “*.”B.3 Task 3: Improve CoverageDescription Destination Action CS1 Open browser Browser ◦ ◦2 Open project ProfileWell.io page Browser (ProfileWell) ◦3 Click “coverage” of the first build Browser (ProfileWell) ◦4 Click the first file not 100% covered Browser (ProfileWell) ◦5 Scroll to an uncovered line Browser (ProfileWell) ◦*6 Switch to atom Editor ◦ ◦*7 Navigate to the test file Editor ◦*8 Navigate to the line where test should be placed Editor ◦*9 Write test Editor ◦Total 9 2Table B.3: The proposal workflow in Task 3, as well as action and contextswitch counts. Supplemental actions are marked with “*.”86B.4 Task 4: Optimize PerformanceDescription Destination Action CS1 Open browser Browser ◦ ◦2 Open project ProfileWell.io page Browser (ProfileWell) ◦3 Click the first build Browser (ProfileWell) ◦4 Copy the path to the longest function Browser (ProfileWell) ◦6 Switch to atom Editor ◦ ◦7 Navigate to the function using the path copied Editor ◦8 (After inspection) switch to ProfileWell.io Browser (ProfileWell) ◦ ◦9 Copy the path to the 2nd longest function Browser (ProfileWell) ◦10 Switch to atom Editor ◦ ◦11 Navigate to the function using the path copied Editor ◦*12 (Issue found) Fix the issue Editor ◦Total 12 4Table B.4: The proposal workflow in Task 4, as well as action and contextswitch counts. Supplemental actions are marked with “*.”87

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.24.1-0381006/manifest

Comment

Related Items