Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Explanations for command recommendations : an experimental study Jiresal, Rahul 2013

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


24-ubc_2014_spring_jiresal_rahul.pdf [ 3.43MB ]
JSON: 24-1.0165690.json
JSON-LD: 24-1.0165690-ld.json
RDF/XML (Pretty): 24-1.0165690-rdf.xml
RDF/JSON: 24-1.0165690-rdf.json
Turtle: 24-1.0165690-turtle.txt
N-Triples: 24-1.0165690-rdf-ntriples.txt
Original Record: 24-1.0165690-source.json
Full Text

Full Text

Explanations for CommandRecommendationsAn Experimental StudybyRahul JiresalBachelor of Engineering, University of Pune, India, 2008A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)December 2013c Rahul Jiresal 2013AbstractRecently, evaluation of a recommender system has been beyond evaluatingjust the algorithm. In addition to accuracy of algorithms, user-centric ap-proaches evaluate a system?s e?ectiveness in presenting recommendations,explaining recommendations and gaining users? confidence in the system.Existing research focuses on explaining recommendations that are relatedto user?s current task. However, explaining recommendations can proveuseful even when recommendations are not directly related to user?s currenttask. Recommendations of development environment commands to soft-ware developers is an example of recommendations that are not related tothe user?s current task, which is primarily focussed on programming, ratherthan inspecting recommendations.In this dissertation, we study three di?erent kinds of explanations forIDE commands recommended to software developers. These explanationsare inspired by the common approaches based on literature in the domain.We describe a lab-based experimental study with 24 participants where theyperformed programming tasks on an open source project. Our results sug-gest that explanations a?ect users? trust of recommendations, and expla-nations reporting the system?s confidence in recommendation a?ects theirtrust more. The explanation with system?s confidence rating of the recom-mendations resulted in more recommendations being investigated. However,explanations did not a?ect the uptake of the commands. Our qualitative re-sults suggest that recommendations, when not user?s primary focus, shouldbe in context of his task to be accepted more readily.iiPrefaceThe work presented in this dissertation was conducted in the Software Prac-tices Lab under supervision of Prof. Gail Murphy.The experiment described in this dissertation were approved by the UBCBehavioural Research Ethics Board under the certificate H13-00703.iiiList of Tables3.1 Total number of recommendations shown to each participant. 173.2 Summary table of recommendations for each explanation con-dition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.1 Summary Table ? One Way ANOVA for percentage of rec-ommendations investigated. . . . . . . . . . . . . . . . . . . . 204.2 Summary Table ? Tukey?s HSD Test for multiple comparisonsbetween conditions. . . . . . . . . . . . . . . . . . . . . . . . . 234.3 Summary Table ? One Way ANOVA for uptake as a percent-age of investigated recommendations. . . . . . . . . . . . . . . 245.1 Pearson?s correlation coecients between participants? expe-rience and the total number of recommendations shown tothem, total number of recommendations they investigated,and uptake, respectively. . . . . . . . . . . . . . . . . . . . . . 29viList of Figures3.1 Experiment Setup ? Screens aligned side-by-side. Left: TaskScreen, 13-inch display at 1280x800 screen resolution; Right:IDE Screen, 23-inch display at 2048x1152 screen resolution. . 93.2 Task Screen. Colour coding of the boxes: Blue ? task, Green? the ?Show recommendations? button, Red ? the area of therecommendations pane, Yellow ? text input area. . . . . . . . 103.3 Details Page with more information about the command . . . 103.4 Recommendations with their conditions as numbered in Sec-tion 3.3 ? 1. None, 2. Friend, 3. Community, 4. Confidence . 114.1 Between-conditions percentage of recommendations investi-gated by participants. . . . . . . . . . . . . . . . . . . . . . . 214.2 Between conditions uptake in terms of percentage of investi-gated recommendations. A command is considered for uptakeif it is used two or more times after the recommendation isinvestigated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.1 Scatterplot representing the correlation between participants?experience and the number of recommendations generated forthem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295.2 Scatterplot representing the correlation between participants?experience and the percentage of recommendations they in-vestigated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.3 Scatterplot representing the correlation between participants?experience and the uptake of the investigated recommendations. 31B.1 Registration Form used for the experiment. . . . . . . . . . . 45viiAcknowledgementsI am grateful to my supervisor Prof. Gail Murphy for her encouragement toapply to UBC, guidance towards an interesting topic for my Masters Thesis,and pointing me to the right path along the way.I thank Dr. Emerson Murphy-Hill for his initial work in IDE CommandRecommendations on which this work is based on.Many thanks to the students who took time to participate in the exper-imenr documented here.Finally, I would like to acknowledge NSERC and IBM Canada for fund-ing this research.viiiDedicationTo my beautiful girlfriend . . .. . . for all her support.ixChapter 1IntroductionSoftware developers use integrated development environments and executecommands to accomplish their programming tasks. These commands rangefrom simple text editing functions, such as copying text, to sophisticatedprogramming actions, such as refactoring code or synchronizing with repos-itories. However, compared to the community, a given developer often usesonly a small subset of all available commands. This happens due to reasonslike reaching a personal fixed-point, or lack of awareness of other commandsamong other reasons. Our previous work [15] discusses this command dis-covery problem and addresses it by making custom command recommen-dations to the developer. The work in [15] improves on a similar solutionto the command discovery problem in AutoCAD [11] presented by Matejkaand colleagues, and the results suggest that it is feasible to recommend in-tegrated development environment commands to developers. We evaluatedour work by comparing the accuracy of our novel algorithms with the onespresented by Matejka et al.In practice, the e?ectiveness of a recommender system depends on morethan the accuracy of the algorithm used in the recommender. In addition toaccuracy, the e?ectiveness of a recommender is known to be a?ected by howthe system presents and explains recommendations to the user [9, 10, 18, 21].For instance, Pu and colleagues argue that the system?s ability to explaina recommendation is highly correlated to a user gaining trust to accept arecommendation [18].Existing research about how users perceive recommendations based onexplanations focuses on recommendations directly related to the user?s cur-rent task. For example, Herlocker et al. [7] evaluated explanations for rec-ommendations in the context of movie recommender system, MovieLens.Users browse through a system like MovieLens with the intention of watch-ing a movie, their main task at that moment, with the system then makingrecommendations, movies to watch, directly related to that task. Recom-mendations can also play a role when the recommendations are indirectlyrelated to the user?s current task. For example, Matejka et al. [11] havedemonstrated that there is value in recommending commands to users of1Chapter 1. Introductioncomplex software applications to improve the users? use of those applica-tions. Recommendations in this context may help a user perform a similartask more e?ectively in the future or may introduce the user to completelynew functionality to be used at a later point in time. It may be dicult fora user to judge the e?ectiveness of such a recommendation as the recom-mendation may not be applied or evaluated immediately. Gaining a user?strust in these indirect recommendations is critical as interviews we have con-ducted of software developers related to command recommendations haveshown that they will consider such a recommendation only when they havetrust in the human or system suggesting a command [15]. How then can weexplain a recommendation indirectly related to a user?s current task suchthat the user trusts the recommendation? We consider this question in thecontext of command recommendations.In this dissertation, we study the e?ectiveness of three di?erent kindsof explanations for commands recommended to software developers using acomplex integrated development environment (IDE) that provides hundredsof commands to a user. We selected the explanation approaches based oncommon approaches in the literature [4, 7, 21]. The first explanation ap-proach we considered is the friend condition where the explanation is basedon one of the user?s aquaintances using the command [21], as in ?Alicia Mar-tin uses this command?. The second approach is the community explanationwhich describes how many people are reported to use the command [7], asin ?54 people use this command?. The third approach, confidence, is thesystem?s confidence in the recommendation [4] as in ?the confidence ratingof this command is 4.5 out of 5 stars?. To evaluate these conditions, we con-ducted a laboratory experiment involving 24 participants who were askedto perform a series of simple programming tasks and who were shown rec-ommendations about commands they might have used that were randomlyassigned to have an explanation according to one of the conditions or noexplanation. We found that explanations do a?ect a user?s trust of a recom-mendation and that explanations reporting the system?s confidence in therecommendation a?ects their trust more than the other two kinds of socialexplanations.This dissertation makes three contributions:? we highlight the need for empirical investigation of users? perception ofexplanations when recommendations are indirectly related to a user?stask-at-hand,? we report on an experiment about di?erent explanation styles for onestyle of indirect recommendation, command recommendations, that2Chapter 1. Introductionindicated users? prefer explanations related to a system?s confidence ina recommendation, and? we present directions for improving the explanation of recommenda-tions when a recommendation is indirectly related to a user?s task.We begin by reviewing the research that has been performed regardinguser perceptions of recommendation explanations (Chapter 2). We thendescribe the study we performed (Chapter 3) and its results (Chapter 4).We conclude with discussion and future work in Chapter 6.3Chapter 2Related Work2.1 Why Explain Recommendations?Traditionally, recommenders have beeen evaluated based on the accuracyof the algorithms used by the recommender (e.g., [2, 5, 8, 19]. However,McNee et al. [12] argue that higher accuracy in a recommendation algo-rithm does not always mean higher user satisfaction. As a result, evaluationof a recommender system should not depend solely on evaluating the algo-rithm. More recently, there has been a movement to adopting user-centricapproaches to evaluate a recommender system (e.g., [9, 10, 18, 21]). In ad-dition to accuracy, these user-centric approaches consider a system?s e?ec-tiveness in presenting recommendations, explaining recommendations andgaining users? confidence on making decisions based on recommendations.For recommenders based on e-commerce systems, studies have shown thatcustomers? tendency to inspect and buy a product is associated with theirtrust of the e-commerce system [6].Several studies (e.g., [1, 13, 17]) have shown that explanations play animportant role in improving the e?ectiveness of recommender systems. Puet al. [18], based on a survey of existing research done on evaluating rec-ommender systems from the user experience point of view, discuss that arecommender system?s ability to explain recommended results is highly cor-related to gaining user trust to accept the items recommended to them.Tintarev and Mashto? [22] described severals aims for explaining recom-mendations, such as transparency, scrutability, trust, e?ectiveness, persua-siveness, eciency and satisfaction. One means of improving a user?s trustand satisfaction with a recommender system is to provide good explanationsof the recommendations (e.g., [7, 20, 22]). While there is substantial agree-ment that explanations play an important role in recommender systems, awide variety of explanation styles are possible. For example, ?this item wasrecommended to you because you showed interest in ?x? ?, ?this movie wassuggested to you because your friend rated this movie highly?, ?we think youwill rate this item 9 out of 10?, and so on. Not very surprisingly, di?erentkinds of explanations have di?erent e?ects on users? trust of the system [7].42.2. Di?erent Kinds of Explanations2.2 Di?erent Kinds of ExplanationsA variety of studies have considered the e?ectiveness of di?erent explana-tions on a user?s trust of the recommendations.Cramer et al. [4] present the e?ects of transparent explanations on users?trust and acceptance of a content-based art recommender. They conductedan experiment to evaluate three versions of this recommender. The firstversion did not explain the recommendations, the second version explainedwhy a recommendation had been made and the third version showed a rat-ing of how confident the system was in the recommendation. Their resultsshow that the transparency achieved by explaining to the user why a rec-ommendation was made increased the acceptance of the recommendations.Herlocker et al. [7] explore explanations in automated collborative filter-ing systems. They performed an experiment with the MovieLens web-basedmovie recommender to identify which explanation component is most e?ec-tive for users. They used 23 di?erent graphical explanations and found thatthe histogram with grouping of neighbour ratings was the most compellingone. This histogram was ?good? ratings clustered together and ?bad? rat-ings clustered together in a bar graph. As a result, the user only had todo a visual comparison to understand the explanation. An analogy to com-mand recommendations would be ?how many people use the command?.Herlocker et al. also demonstrated that 86% of their survey respondentsvalued explanations and would like to see them added to their recommen-dations. They asserted that explanations can increase the acceptance of therecommendations.Sinha et al. [20] argue that the human analogue of a recommendationsystem is a transparent social process. The collaborative filering algorithmsare very similar to the social process of asking friends for recommenda-tions of movies, books or products they like. The very social nature ofcollaborative filtering recommendation algorithms makes it obvious for therecommendation to include an explanation of who else it applies to. In[21], the authors showed that users? friends provided a higher percentage of?good? and ?useful? recommendations compared to a system that doesn?tprovide explanations. In our previous work [15], we discussed that one ofthe most natural and e?ective ways of learning commands is from a peer[3, 14, 23], hence one can reason that the recommendations will be acceptedmore easily if they seem to be coming from a peer, friend or a colleague.Murphy-Hill et al. [16] speculate that letting the developers know ?who?uses the recommended commands would be a key factor for the developerto accept the recommendation. When recommended by friends, a user may52.3. Command Recommenderstrust the recommendation based on the similarity between the friend andhim or herself.2.3 Command RecommendersUsers of sophisticated applications often plateau in their use of commandsavailable in the application. To help users discover and make use of a greaterpercentage of commands available in AutoCAD, Matejka and colleagues [11]presented CommunityCommands. In CommunityCommands, they used col-laborative filtering algorithms to recommend commands that users shouldbe using, but are not aware of, accelerating their learning process. Softwaredevelopers also use sophisticated applications like the integrated develop-ment environments (IDEs) and execute commands to accomplish their pro-gramming tasks. These commands range from simple text editing functions,such as copying text, to sophisticated programming actions, such as refac-toring code or synchronizing with repositories. However, compared to thecommunity, a given developer often uses only a small subset of all availablecommands. This happens due to reasons like reaching a personal fixed-point,or lack of awareness of other commands among other reasons. Improving onMatejka?s work, our previous work [15] discusses this command discoveryproblem for developers and addresses it by making custom command rec-ommendations. We evaluated our work by comparing the accuracy of ournovel algorithms with the ones presented by Matejka et al. in [11]. Ourresults also showed that the simple algorithms like ?Most Popular?, ?MostWidely Used?, and vanilla Collaborative Filtering algorithms did very wellfor novices compared to the more sophisticated algorithms.A widely used method to evaluate trust in the recommender systemis to ask users whether the recommendations were useful to them [4, 7,17]. However, this may not always be true as users are often mistakenin expressing what they want. Another way of evaluating users? trust inrecommendations is in an online test scenario, with an assumption thathigher trust translates to higher number of recommendations being used.In this dissertation for command recommendations, we assume that trustin the recommendations is related to investigation of the recommendation.Whether the user learns the command and continues to use it still dependson various other factors such as diculty of the command, context of thedeveloper?s current tasks, the willingness of the developer to learn a newcommand, etc.The existing literature studies explanations on recommendations that62.3. Command Recommendersare directly related to the user?s task. To our knowledge, the e?ect of ex-planations on recommendations that are indirectly related to the user?s taskhave not been studied previously. Indirect recommendations are di?erentbecause they may not help user perform the task at hand right away, butthey may be helpful in the future. In this dissertation, we study explana-tions for recommendtions that are indirectly related to the users? task. Wepresent results of di?erent explanations on both investigation of the recom-mendation and uptake of the command where the recommendations are notthe primary focus of user?s work.7Chapter 3Study DesignTo evaluate the e?ect of three di?erent kinds of explanations for IDE com-mand recommendations we conducted a laboratory based user experiment.The experiment consisted of participants working on common programmingtasks in Java using an open source IDE called Eclipse1.The experiment consisted of 16 small programming tasks, followed byinterviews. During the programming tasks, a participant was shown com-mand recommendations that were indirectly related to his or her tasks. Thecommands would allow a user to complete a task more eciently. The par-ticipant was told that his primary goal was to complete the tasks, but thathe was free to investigate the recommendations to learn new commandsthat might be useful when programming. As the purpose of the experi-ment was to investigate explanations, we ensured that all recommendationsgiven to the users were 100% applicable and accurate. Even though the rec-ommendations were applicable and accurate, completing the programmingtasks was the primary goal for the participant, rather than investigatingthe recommendations and learning new commands. Recommendations werepresented with randomly assigned explanations (Chapter 3.3). The systemrecorded the participant?s command usage and his clicks on the recommen-dations provided. After the programming tasks were finished, we conducteda semi-structured interview with the participant.3.1 ParticipantsWe recruited 24 students (15 undergraduate and 9 graduate) from the Uni-versity of British Columbia as our participants. The participants? experiencewith Java and Eclipse varied from 1 month to 20 months, with a mean of7 months, median of 6 months and a standard deviation of 5.75 months.We chose to recruit only novices because there was more room to makerecommendations to them.As part of registration, a participant was asked to provide the full, verified December 10, 201383.1. ParticipantsFigure 3.1: Experiment Setup ? Screens aligned side-by-side. Left: TaskScreen, 13-inch display at 1280x800 screen resolution; Right: IDE Screen,23-inch display at 2048x1152 screen resolution.93.1. ParticipantsFigure 3.2: Task Screen. Colour coding of the boxes: Blue ? task, Green ?the ?Show recommendations? button, Red ? the area of the recommendationspane, Yellow ? text input area.Figure 3.3: Details Page with more information about the command103.2. SetupFigure 3.4: Recommendations with their conditions as numbered in Sec-tion 3.3 ? 1. None, 2. Friend, 3. Community, 4. Confidenceof at least three friends with whom they have worked with on a program-ming project, or know to be a programmer, studying at UBC or locatedin Vancouver. We used these names to make explanations from friends berealistic.3.2 SetupAfter registration, participants were scheduled for a lab session in which theywould perform programming tasks on an open source Android project named?Comic Reader?2. To get familiar with the project, when they came to thesession, the participants were shown a video3 showing a person navigatingthrough the project and explaining di?erent components of the softwarecomprising the project. This video included the narrator?s name which waslater used during the experiment to induce trust of recommendations.Participants were then asked to complete 16 programming tasks4 that re-flected common everyday programming activities, such as navigating throughthe code, making changes and refactoring the code, debugging etc. Thesetasks were performed on the Comic Reader project. The programming, verified December 10,, verified December10,, verified December 10,2013113.3. Experimental Conditionsguage for the project was Java and the participants were asked to work withthe Eclipse IDE. While performing these tasks, participants were allowed touse their existing knowledge of Eclipse commands.The experiment was set up using two screens aligned side-by-side asshown in Figure 3.1. The IDE was presented on one screem, while thesecond screen ? Task Screen ? showed the tasks and command recommen-dations. The IDE screen was on a 23-inch monitor with 2048x1152 screenresolution, and the Task Screen was shown on a 13-inch laptop display witha 1280x800 screen resolution. Figure 3.2 shows the screenshot of the TaskScreen. The recommendations pane (area in the red box) showed the gener-ated recommendations after the user clicks on the ?Show Recommendations?button. Explanations were shown under the names of the commands on therecommendations. No restrictions were placed on when a user could clickon the ?Show Recommendations? button. Participants were advised beforethe study that clicking on the button at least once per task would ?makethe algorithm learn, and generate more and better recommendations?. Wechose to state this advise so that participants would be more encouraged toask for recommendations. Since most tasks were in the form of a question,the Task Screen provided them with a text area to enter the answer to thequestion. The answers were not considered to make recommendations ordata analysis.A recommendation, when clicked, opened a new page that showed detailsabout the command (Figure 3.3). The details included information aboutwhat the command does, how to invoke the command, what are the keyboardshortcuts, along with sample screenshots of the command usage.The experiment was about an hour long, and the time taken by theparticipants was between 50 minutes to 70 minutes.3.3 Experimental ConditionsFrom the related work, it is clear that di?erent types of explanations influ-ence a user?s trust di?erently [7]. In our study, we explored the e?ect ofthese explanations in following four conditions:1. None ? No explanation2. Friend ? ?Who uses this? condition3. Community ? ?How many people use this? condition4. Confidence ? ?Confidence rating? condition123.3. Experimental ConditionsNone ? This was the control condition. In this condition, no explanationwas provided for the recommendation.Friend ? Below the name of the recommended command, this conditionincluded an explanation like ?Alicia Martin uses this command?. As dis-cussed in Chapter 2, a recommendation may be more readily accepted andtrusted if it appears to be coming from a peer, friend or a colleague [20, 21].This condition provided the participants an impression that the recommen-dation was coming from one of their friends/colleagues. For this condi-tion, we randomly selected the friends? names that they provided when theysigned up for the study and added it to the explanation. We also includedthe narrator of the video the students watched for background on Comi-cReader in the list of friends the participants provided.Community ? Below the name of the recommended command, this con-dition included an explanation like ?27 people uses this command?. The?community? condition was intended to quantify the usefulness of the com-mand. Unlike ?# of neighbors? explanation mentioned by Herlocker et al. [7],the ?community? condition is a raw number of people using the command.Though the people using the command may not be the user?s friends, thereasoning behind it ? ?So many people use it, it should be good!? ? isreasonable, similar to what Herlocker et al?s histogram grouped by ratingsrepresented ? ?So many people rated it 5, it should be good?. The ?commu-nity? condition included a randomly generated number between 13 and 39.We chose this range to be representative of the size of a community becauseit gives a sense of the data being real, since the number of participants insuch an experiment is not generally known to be very high.Confidence ? Below the name of the recommended command, this con-dition included an explanation ?Confidence rating of this recommendation:4.5/5 stars?. We define confidence as the recommender system?s trust in thegiven recommendation. The work in [7] points out that users may trust thesystem more by observing the confidence ratings of the recommendations.For example, a user may choose a recommendation with high confidenceover a recommendation with a lower confidence rating. The ?confidence?condition was designed to give participants information on how sure therecommender system is in recommending this command to them. The ?con-fidence? condition included a star rating (as shown in Figure 3.4) with 3.5to 5 stars. The reasoning behind not showing lower ratings is that a rec-ommender system should filter out recommendations if it is not confidentabout them.Figure 3.4 shows recommendations with these explanations in the ordermentioned above.133.3. Experimental ConditionsThe purpose of the experiment was to evaluate the e?ect of explanationson command recommendations. Hence, we made sure all recommendationswere 100% accurate and applicable to the context of the user?s tasks. Toachieve that, instead of generating the recommendations using an algorithm,they were generated from a list of commands that we prepared that couldpossibly be used for each task. If a participant did not use a command listedfor a task, that command was recommended to them with an assumptionthat they did not know the command. This strategy ensured that the rec-ommendations were in the context of the task. Participants could use therecommended commands in the subsequent tasks.The recommendations were randomly assigned an experimental condi-tion before being presented to the participants. If a recommendation wasrepeated to a user, we ensured that it always gets the same condition. Tomake sure that one condition is not favoured over another, all conditionswere equally distributed. Table 3.2 shows the total number of recommenda-tions generated (for all participants together) for each condition with theirmeans and standard deviations.For each task, the number of recommendations shown varied from 1 to6 depending on the commands the participant used while performing thetasks. If they used more commands that are applicable to the task, theywould get fewer recommendations. If a recommendation was applicable tomore than one task, it was shown to them unless they used it 3 or more timessince the last time they investigated it. The total number of recommenda-tions shown to each participant throughout the experiment ranged from 16to 57, with a mean of 27.58, median of 27 and standard deviation of 7.2recommendations. Table 3.1 shows the total number of recommendationsshown to each participant.To explain how recommendations are generated, lets consider that ahypothetical participant (lets name him Alex) arrives for the experiment.The first two tasks Alex performs do not have any commands applicable tothem; they are warmup tasks to get him familiar with the project. Whenhe starts Task#3, he clicks on the ?Show Recommendations? button toget the recommendations. The recommendations get populated from thelist of commands listed in Appendix A for each task by subtracting thecommands Alex has already used. After populating the recommendations,they get assigned an explanation condition such that each condition getsequal weightage at the end of the experiment. Since he hasn?t used anycommand till now, he receives the recommendation for the ?Open Resource?command with the randomly assigned Friend condition. If Alex has used?Open Resource? before it was recommended to him for the first time, it143.4. Hypotheseswould not have been recommended to him, because he already knew thecommand. When Alex investigates the recommendation, the system recordsthat he clicked on the recommendation to view the details of the command.Alex tries out the ?Open Resource? command, finishes the task and thenmoves on to Task#4. On Task#4, when Alex asks for recommendations, heonly recieves ?Open Type? and not ?Open Resource? because he alreadyused it more than 3 times since it was last recommended to him. Thesystem assumes that Alex has learned that command and does not need torecommend that anymore. Appendix A lists all the tasks and lists all thecommands that can be recommended for each task.Even though the recommendations participants received from the systemwere related to the programming tasks they were performing, their primarygoal was to complete the task, rather than investigating the recommenda-tions and learning new commands. This is why the recommendations wereindirectly related to their primary objective.3.4 HypothesesWe were interested in both how often recommendations were investigatedand how often they were used after investigation. Our rationale is thatrecommendations can not be helpful to a user if not investigated and onlyif commands are used are the recommendations e?ective. In the context ofthis study, we defined two terms ?? Investigation ? We said the recommendation was investigated bythe user if the user clicks on the recommendation to open the pagecontaining detailed information about the command.? Uptake ? We defined uptake as ?the recommended command beingused two or more times after investigating the recommenation?.We designed our experiment to investigate two hypotheses.Hypothesis 1 (H1): Adding explanations to recommendationsresults in more recommendations being investigated.Null Hypothesis 1 (H10): Explanations did not a?ect the investigation of therecommendations.We hypothesize that explanations improve the users? intent to investigaterecommendations on the systems where recommendations are indirectly re-lated to the user?s task. In particular, since Sinha et al. [20] argue thatpeople more readily take recommendations from their friends, and we previ-ously point out that people learn more e?ectively from their peers [15, 16],153.4. Hypotheseswe hypothesize that the Friend condition will be most e?ective for improv-ing the investigation of the recommendations. Since the study is done onnovices to intermediate developers, we also predicted that participants willnot be interested in the system?s confidence in the recommendations becauseof lack of understanding on how the recommender system works. Hence wehypothesized that the Confidence condition will not be as e?ective as theother conditions.Hypothesis 2 (H2): Adding explanations to recommendationsimproves the uptake of the recommended commands.Null Hypothesis 2 (H20): Explanations did not a?ect the uptake of the rec-ommended commands.Identical to the investigation, we hypothesized that explanations would im-prove the uptake of recommended commands. To be specific, we predictedthat the participants will learn more commands that are recommended tothem by a friend.163.4. HypothesesParticipant #recommendations shownParticipant#1 22Participant#2 23Participant#3 27Participant#4 29Participant#5 27Participant#6 23Participant#7 28Participant#8 28Participant#9 28Participant#10 30Participant#11 24Participant#12 27Participant#13 16Participant#14 26Participant#15 30Participant#16 30Participant#17 29Participant#18 27Participant#19 20Participant#20 27Participant#21 30Participant#22 57Participant#23 24Participant#24 30Mean 27.58Median 27Standard Deviation 7.2Table 3.1: Total number of recommendations shown to each participant.173.4. HypothesesCondition None Friend Community ConfidenceTotal 166 167 166 164Mean 6.92 6.96 6.92 6.79Median 7 7 7 7SD 1.79 1.81 1.79 1.98Table 3.2: Summary table of recommendations for each explanation con-dition.18Chapter 4ResultsWe divide the results into quantitative5 and qualitative based on data wecollected from the experiment and the subsequent interviews.4.1 Quantitative ResultsWe present the quantitative results of the experiment by examining ourhypotheses.Hypothesis 1 (H1): Adding explanations to recommendationsresults in more recommendations being investigated.Figure 4.1 shows the di?erences in percentages of recommendations investi-gated by participants, grouped by explanation types. Each box correspondsto one condition and the Y-axis represents the percentage of recommenda-tions investigated by participants. The data supports the first hypothesisthat adding explanations results in higher investigation of the recommen-dations. A one-way ANOVA comparing the results reports a statisticallysignificant di?erence between the explanations with a p-value of 0.027 (Ta-ble 4.1).While the ANOVA test revealed that the conditions di?ered, it cannottell which conditions di?er from each other. To clarify further, we per-formed Tukey?s HSD Test to find out which conditions di?er significantly.According to the summary table for multiple comparison between conditions(Table 4.2), only the conditions None and Confidence di?er significantly (p= 0.02).Hypothesis 2 (H2): Adding explanations to recommendationsimproves the uptake of the recommended commands.We expected the uptake of commands to be correlated with the investigationof the recommendations. However, the results were contradictory to ourexpectations. There seemed to be no e?ect on the uptake of the commandswith or without the explanations. Figure 4.2 shows the uptake as percentage5All statistical tests performed using R version 3.0.2 (,verified December 10, 2013) on Mac OS X Quantitative Resultsdf SS Mean SS F-value Pr(>F)Conditions 3 5460 1820 3.209 0.027Residuals 92 52168 567Table 4.1: Summary Table ? One Way ANOVA for percentage of recom-mendations investigated.of investigated recommendations, grouped by explanation types. A one wayANOVA test for comparison reported no statistically significant di?erencebetween the conditions (Table 4.3).204.1. Quantitative ResultsFigure 4.1: Between-conditions percentage of recommendations investigatedby participants.214.1. Quantitative ResultsFigure 4.2: Between conditions uptake in terms of percentage of investigatedrecommendations. A command is considered for uptake if it is used two ormore times after the recommendation is investigated.224.1.QuantitativeResultsPair of conditions Di? in Means Lower End Point Upper End Point P ValueFriend-None 12.72 -5.27 30.71 0.26Community-None 14.21 -3.78 32.19 0.17Confidence-None 20.84 2.85 38.83 0.02Community-Friend 1.49 -16.50 19.47 0.99Confidence-Friend 8.12 -9.87 26.10 0.64Confidence-Community 6.63 -11.35 24.62 0.77Table 4.2: Summary Table ? Tukey?s HSD Test for multiple comparisons between conditions.234.2. Qualitative Resultsdf SS Mean SS F-value Pr(>F)Conditions 3 2296 765.3 0.832 0.48Residuals 876 79984 919.4Table 4.3: Summary Table ? One Way ANOVA for uptake as a percentageof investigated recommendations.4.2 Qualitative ResultsAfter completing the programming tasks, we interviewed the participant.The participant was asked a few questions about the recommendations, andthe experiment in general.The semi-structured interview was guided by the following set of ques-tions.1. How would you rate the diculty of the tasks on the scale of very easyto very dicult?2. How useful did you find the recommendations?3. Which explanations a?ected the most on what recommendations youinvestigated during the experiment?4. For the recommendations with your friends? names, how do you thinkthe system inserted those names there?5. How would you rate each of your friends from the recommendationsfor their programming ability on the scale of 1 to 5?We asked each participant about the diculty of the tasks. Almost allparticipants said that the tasks were easy to moderate in diculty, exceptfor the participant who had only one month of experience, who thought thetasks were dicult. We asked the participants to rate the overall usefulnessof the recommendations. All participants said that the recommendationswere useful, and that they could see use for a command recommender sys-tem in their IDE. Participant #9 said ?Tasks were (in) increasing (orderof) diculty, but the recommendations were helpful?. More than one par-ticipants mentioned that the recommender system should be a part of theIDE and should be shipped as an Eclipse plug-in.Although most (all except one) participants investigated the recommen-dations, there were a couple of instances where the participants did not244.2. Qualitative Resultsinvestigate many recommendations. For instance, participant #13 said ?Iknew the commands before it was recommended, but I mostly had eitherforgotten them or didn?t see them fit in the context. But after I saw therecommendations, I remembered the command and used it without havingto investigate the recommendation.? This comment shows that the recom-mendations can also be used as a reminder of existence of already-knowncommands.Two participants mentioned that they were users of Visual Studio7 andXcode8. From their experience with those IDEs, they imagined that thecommands analogous to the ones they use in Visual Studio or Xcode existin Eclipse, but didn?t know how to invoke them, or where to find them. Theymentioned that the recommendations helped them in that concern. Thesecomments emphasize that the command recommendations are not only use-ful to learn new commands, but there is a also a use case as a mapping ofalready-known commands from other similar development environments toa new environment.We asked the participants if the conditions influenced which recommen-dations they investigated. Only 3 (12.5%) participants mentioned that thenames of their friends a?ected which recommendations they investigated.Participant #4 said ?Something my friend used is not neccessarily applica-ble to me?. Despite that, there were some cases when the participant justwanted to know more about the command his friend used ? Participant #8said ?I was curious to see what person-X used!?. Most number of partic-ipants mentioned that they preferred the community condition over otherconditions. The common reasoning behind it was ? ?I trusted the one thatsaid ?50 people are using it? because 1 person is not better than 50 people?,as mentioned by participant #6. Several participants ? 7 out of 24 (30%) ?mentioned that the conditions did not a?ect which recommendations theyinvestigated. They mentioned that they only skimmed through the recom-mendation and tried to make a judgement on how it fits the context of theirtask.The answers to the question ?what conditions influenced which recom-mendations they investigated? brings up an interesting observation. Con-trary to the quantitative results presented in Chapter 4.1, the conditionconfidence ? preferred by only 6 people in the interviews ? was mentionedfewer than the community condition or the none condition. For participants7An IDE for development on Microsoft platforms ?, verified December 10, 20138An IDE by Apple for OS X and iOS development ?, verified December 10, 2013254.3. Summary of Resultswho mentioned that the explanations did not a?ect their choice, they saidthat recommendations were investigated based on their understanding ofhow relevant recommendations seemed to their task9. Most of them men-tioned that they made a decision based on the names of the commands.However, it may have also been the order in which the recommendationswere presented. One participant (#20) said ?I used the first recommenda-tion that I found?. Even though participants mentioned that they did notinvestigate the recommendations based on the explanations, we believe thatfor most of them, the explanations a?ected their decision subconciously. Forinstance, participant #20 mentioned that ?maybe subconsciously the otherinformation might?ve a?ected my decision?. This draws our attention to oneof the basic HCI principles for usability, that one should pay attention towhat users actually do, not what they say they do.Some participants were concerned about the way details about the com-mands were presented on the details page. They indicated the diculty inreading the description of the commands and mentioned that a short videotutorial would be more e?ective in learning the command to increase theuptake. For more experienced participants, they mentioned inertia as a fac-tor to a?ect the uptake of the recommended commands ? participant #13said ?I am comfortable using the command I already know, I don?t want tolearn a new command.?4.3 Summary of ResultsCompared to most literature [7, 20] where recommendations were user?sprimary focus, the results di?ered significantly when they were indirectlyrelated to the user?s primary task. Herlocker et al. [7] found the histogramwith user ratings to be most e?ective. But the similar explanation ?23people use this command? did not fare well in our results. Recommendationscoming from friends, contrary to the results in Sinha?s work [20], were notpreferred by users either. Our quantitative results show that the users?tend to investigate recommendations more readily with an explanation thatprovides confidence rating of the recommender system. Also, contrary to ourbelief and initial hypothesis, users do not neccessarily take recommendationsthat seem to be coming from friends. It turns out that they are more likely to9Note that even though the recommendations were made in the context of users? tasks,they were still indirectly related to the task, as in the primary focus of the participantswas to complete the tasks, rather than investigating recommendations and learning morecommands.264.3. Summary of Resultsinvestigate recommendations that show commands used by a larger numberof anonymous people. However, the explanations did not a?ect the uptakeof the commands.During our interviews, participants mentioned that they found the rec-ommendations useful and suggested that it should be shipped as an Eclipseplug-in. We also learned that the recommendations can be used as a re-minder of already known commands, or migrating from one developmentenvironment to other. Even though many participants mentioned that theirdecisions were not a?ected by the explanations, it may have a?ected theusers subconsciously. Participants mentioned diculty in learning the rec-ommended commands due to inertia and/or the way detailed informationabout the commands was presented.27Chapter 5Threats to ValidityA possible threat to the experiment was the varied experience of the partic-ipants. To see if experience was a factor, we conducted Pearson?s product-moment correlation tests10 to find out the correlation of participants? experi-ence with the total number of commands recommended to them, percentageof recommendations they investigated, and the percentage of commands theyused, respectively. Table 5.1 presents the Pearson?s coecients for all threefactors with the participants? experience. The low coecients pointed outthat there was no correlation between the experience and the other factors.The scatterplots in Figures 5.1, 5.2 and 5.3 corroborate the Pearson?s coef-ficients and show that the plotted points are not along the regression lines.This suggests that the experiment was fair, even though the participants?experience varied.One threat to the experiment was that the participants would be ableto determine that the data was not gathered from their friends. We min-imized this threat by not letting them take the study within one week oftheir registration. This increased the likelihood that they forget about giv-ing the names during registration; or had them believe that their friendsparticipated in the study during that one week. During the post-study in-terviews, we asked the participants ?How do you think the system insertedthose names in the recommendations??. All the answers to this questionwere either ?I imagine those people participated in the study before me? (18out of 24) or ?Those people also use this recommender system? (4 out of24). Two participants even mentioned that they completely forgot thatthey provided us with these names. This mitigated the possible threat tothe experiment.Another threat to the experiment was that participants would be biasednegatively by knowing that their friends are not good programmers. Toremove that threat, in the post-study interviews, we asked the participantsto rate each friends they provided on ?How good a programmer you thinkthey are? on a scale of 1 to 5. We also included the ?person from the video?10, verified December 10, 201328Chapter 5. Threats to ValidityFigure 5.1: Scatterplot representing the correlation between participants?experience and the number of recommendations generated for them.Recommendations Investigations UptakeExperience -0.22 -0.35 0.43Table 5.1: Pearson?s correlation coecients between participants? experi-ence and the total number of recommendations shown to them, total numberof recommendations they investigated, and uptake, respectively.29Chapter 5. Threats to ValidityFigure 5.2: Scatterplot representing the correlation between participants?experience and the percentage of recommendations they investigated.30Chapter 5. Threats to ValidityFigure 5.3: Scatterplot representing the correlation between participants?experience and the uptake of the investigated recommendations.31Chapter 5. Threats to Validityto this list. Except for 2 instances (rated 3 out of 5) in the whole study,all participants rated their friends either a 4 or 5. This meant that if arecommendation was coming from that friend, they would not disregard itbecause it came from a ?non-credible source?. This removed the threat thatthe recommendations with ?friend? condition weighed less compared to otherconditions.The confidence condition showed star rating of up to 5 stars. The graph-ical nature of this condition, as opposed to textual nature of the other con-ditions, may have a?ected the participants? decisions. Further study needsto be conducted to validate or refute this threat.A threat to evaluation of explanations on the uptake of commands isthat uptake may have a?ected by other factors such as diculty of thecommand, medium used to present details of the command, and the user?sintent to switch from his current known command to a new one. We tried tocontrol the factors as much as possible such as being consistent in presentingcommand details. But it was impossible to control factors such as dicultyof the commands, and the users? intent to learn a new command and switchto it.The participants recruited for our experiment were students who hadlittle expertise in using the IDEs. An external threat to the experiment wasthat we may not be able to generalize the results to the whole population.The results might be di?erent if we had recruited seasoned developers, orhad conducted a field study instead of a lab-based experiment.Might we have had di?erent results if the experiment was done in users?workplace rather than a lab-based experiment? It is possible that the par-ticipants behave di?erently in a lab-based scenario compared to their work-place environment. The best way to gather more accurate data would be toconduct field tests of the recommendations with real-time recommendationdelivery and context learning algorithms.32Chapter 6Future WorkBuilding on what we presented in this dissertation, we feel that there areseveral promising areas for future work.One area for future work is to find the context of the user?s current task.Throughout this dissertation, we presented recommendations such that theywere 100% accurate. This means that the recommendations were applica-ble11 to the programming tasks performed by the participants. Even then,from the recommendations shown to them, a few participants mentionedthat they tried to filter out recommendations that they thought to be notapplicable for their current task. One possible explanation that could helpimprove the acceptance of recommendation is to tell users how much therecommendation fits their context. For example ? ?89% applicable to whatyou?re doing right now?. It would be interesting to evaluate how this expla-nation fares compared to the ones we evaluated. To be able to use an IDEcommand recommender system to its full extent, automated learning of theuser?s current task context is needed. Although it could be a hard problemto solve, though one can start with simple contexts such as ?browsing thecode?, ?refactoring code?, ?debugging? or ?writing code? based on what viewsand windows they have open on the screen, keyboard and mouse activityand the recent history of their command usage.In this dissertation, we evaluated the e?ect of explanations on the up-take of commands. However, uptake also depends on other factors such asdiculty of the command, medium used to explain details of the command,and the user?s intent to switch from his current known command to a newone. The details window explaining the command (Figure 3.3) used in ourexperiment contained textual description and screenshots. However, thereare di?erent ways to convey the same information, like videos or step-by-step guides. Evaluating these interfaces would give us a perspective on whata?ects the uptake.We mentioned earlier that participants reported inertia as a factor for11We verified in the semi-structured interviews that the recommendations were related(Chapter 4.2). Also, completing the tasks was their primary goal, rather than checkingthe recommendations.33Chapter 6. Future Worknot switching to a di?erent command than what they already use. For usersto readily make the switch, we need to provide an incentive for them toovercome the inertia. Explanations that tell users what they will gain byusing the new command could be used in this case. For example, ?You willsave 16 minutes by using X rather than Y?, or ?This command is reportedly28% faster than Y that you already use?.We imagine the future of an IDE command recommender system as aplatform where developers share their command usages along with videos,tutorials and usage guides, and intelligent machine learning algorithms de-liver them to appropriate users who would need them, when they need them.34Chapter 7ConclusionIn this dissertation, we pointed out the need for evaluation of explaning rec-ommendations when recommendations are indirectly related to user?s taskat hand. We undertook an experiment to compare three di?erent expla-nations for IDE command recommendations. We looked into the e?ect ofthese explanations on number of recommendations being investigated, andthe uptake of the recommended commands they investigate. We had 24 par-ticipants for our lab-based experiment where they performed programmingtasks on a software project. Our results demonstrate that confidence ratingof the recommendation was most e?ective on promoting the recommenda-tion so they are investigated more by the users. However, we saw no e?ectof the explanations on the uptake of the commands.Although this kind of comparison of explanations is done previously,it has never been done in the context of IDE command recommendations,where recommendations are not the user?s primary focus. Finally, we discussfuture work that needs to be done towards making IDE command recom-mendations useful in a developer?s daily life.35Bibliography[1] Mustafa Bilgic. Explaining recommendations: Satisfaction vs. promo-tion. In In Proceedings of Beyond Personalization 2005, the Workshopon the Next Stage of Recommender Systems Research(IUI2005, pages13?18, 2005.[2] John S. Breese, David Heckerman, and Carl Kadie. Empirical analysisof predictive algorithms for collaborative filtering. In Proceedings of theFourteenth conference on Uncertainty in artificial intelligence, UAI?98,pages 43?52, San Francisco, CA, USA, 1998. Morgan Kaufmann Pub-lishers Inc.[3] Alistair Cockburn and Laurie Williams. Extreme programming ex-amined. chapter The costs and benefits of pair programming, pages223?243. Addison-Wesley Longman Publishing Co., Inc., Boston, MA,USA, 2001.[4] Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten Someren,Lloyd Rutledge, Natalia Stash, Lora Aroyo, and Bob Wielinga. Thee?ects of transparency on trust in and acceptance of a content-based artrecommender. User Modeling and User-Adapted Interaction, 18(5):455?496, November 2008.[5] Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. Performance ofrecommender algorithms on top-n recommendation tasks. In Proceed-ings of the fourth ACM conference on Recommender systems, RecSys?10, pages 39?46, New York, NY, USA, 2010. ACM.[6] Sonja Grabner-Kra?uter and Ewald A. Kaluscha. Empirical research inon-line trust: a review and critical assessment. Int. J. Hum.-Comput.Stud., 58(6):783?812, June 2003.[7] Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. Explainingcollaborative filtering recommendations. In Proceedings of the 2000ACM conference on Computer supported cooperative work, CSCW ?00,pages 241?250, New York, NY, USA, 2000. ACM.36Bibliography[8] Alexandros Karatzoglou, Xavier Amatriain, Linas Baltrunas, and NuriaOliver. Multiverse recommendation: n-dimensional tensor factorizationfor context-aware collaborative filtering. In Proceedings of the fourthACM conference on Recommender systems, RecSys ?10, pages 79?86,New York, NY, USA, 2010. ACM.[9] Bart P. Knijnenburg, Lars Schmidt-Thieme, and Dirk G.F.M. Bollen.Workshop on user-centric evaluation of recommender systems and theirinterfaces. In Proceedings of the fourth ACM conference on Recom-mender systems, RecSys ?10, pages 383?384, New York, NY, USA, 2010.ACM.[10] Joseph A. Konstan and John Riedl. Recommender systems: from algo-rithms to user experience. User Modeling and User-Adapted Interaction,22(1-2):101?123, April 2012.[11] Justin Matejka, Wei Li, Tovi Grossman, and George Fitzmaurice. Com-munitycommands: command recommendations for software applica-tions. In Proceedings of the 22nd annual ACM symposium on Userinterface software and technology, UIST ?09, pages 193?202, New York,NY, USA, 2009. ACM.[12] Sean M. McNee, John Riedl, and Joseph A. Konstan. Being accurate isnot enough: how accuracy metrics have hurt recommender systems. InCHI ?06 Extended Abstracts on Human Factors in Computing Systems,CHI EA ?06, pages 1097?1101, New York, NY, USA, 2006. ACM.[13] David McSherry. Explanation in recommender systems. Artif. Intell.Rev., 24(2):179?197, October 2005.[14] M.M. Muller and W.F. Tichy. Case study: extreme programming ina university environment. In Software Engineering, 2001. ICSE 2001.Proceedings of the 23rd International Conference on, pages 537?544,2001.[15] Emerson Murphy-Hill, Rahul Jiresal, and Gail C. Murphy. Improvingsoftware developers? fluency by recommending development environ-ment commands. In Proceedings of the ACM SIGSOFT 20th Inter-national Symposium on the Foundations of Software Engineering, FSE?12, pages 42:1?42:11, New York, NY, USA, 2012. ACM.37[16] Emerson Murphy-Hill and Gail C. Murphy. Peer interaction e?ectively,yet infrequently, enables programmers to discover new tools. In Proceed-ings of the ACM 2011 conference on Computer supported cooperativework, CSCW ?11, pages 405?414, New York, NY, USA, 2011. ACM.[17] Pearl Pu and Li Chen. Trust building with explanation interfaces.In Proceedings of the 11th international conference on Intelligent userinterfaces, IUI ?06, pages 93?100, New York, NY, USA, 2006. ACM.[18] Pearl Pu, Li Chen, and Rong Hu. Evaluating recommender systemsfrom the user?s perspective: survey of the state of the art. User Modelingand User-Adapted Interaction, 22(4-5):317?355, October 2012.[19] Guy Shani and Asela Gunawardana. Evaluating recommendation sys-tems. In Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B.Kantor, editors, Recommender Systems Handbook, pages 257?297.Springer US, 2011.[20] Rashmi Sinha and Kirsten Swearingen. The role of transparency in rec-ommender systems. In CHI ?02 Extended Abstracts on Human Factorsin Computing Systems, CHI EA ?02, pages 830?831, New York, NY,USA, 2002. ACM.[21] Swearingen Sinha, Kirsten Swearingen Rashmi, and Rashmi Sinha. Be-yond algorithms: An hci perspective on recommender systems, 2001.[22] Nava Tintarev and Judith Mastho?. E?ective explanations of recom-mendations: user-centered design. In Proceedings of the 2007 ACMconference on Recommender systems, RecSys ?07, pages 153?156, NewYork, NY, USA, 2007. ACM.[23] Michael B. Twidale. Over the shoulder learning: Supporting brief in-formal learning. Comput. Supported Coop. Work, 14(6):505?547, De-cember 2005.38Appendix AStudy TasksDuring our experiment, the following programming tasks were given to theparticipants. Each task below has a list of possible commands12 that couldbe recommended for that task. For each task, the commands that the par-ticipants have not yet used (within the scope of the whole experiment) wererecommended to them. A command could be recommended more than onceif the participant did not investigate it previously, or he has not used thecommand more than 2 times since he investigated it the first time. Theparticipants were told that the first two tasks were ?warm-up? tasks and norecommendations will be shown during those tasks because ?the system willstill be learning from their interactions?.1. What components (files/packages etc.) in the project seem responsiblefor updating content in the ComicReader app? Try guessing just bygoing through the file names.2. Some comics are released daily, some are released weekly, and so on.What components (files/packages etc.) represent the di?erent typesof comics in the application? Answer just by going through the filenames13.3. In an Android application, the window you see on the phone screenis an Activity. Each screen is implemented as a separate Activity.Go through the project files and name three classes that seem to beimplementing an Activity in the ComicReader project.Commands ?? org.eclipse.ui.navigate.openResource4. Open the file ArchivedComic.javaCommands ?12The commands are presented using the command-ids by which Eclipse recognizesthem.13No commands were listed for the first two tasks. No recommendations were shownbecause these were warm-up tasks.39Appendix A. Study Tasks? org.eclipse.ui.navigate.openResource? What is the relationship between the classes ArchivedComic and Comic?Commands ?? Does the class DailyComic have a sibling? If yes, name it.Commands ?? org.eclipse.ui.navigate.openResource? Open the file Find a location where an object of the classStrip is created. (Hint: The constructor is usually called to create anobject)Commands ?? org.eclipse.ui.navigate.openResource? Open the file Navigate to the method parse(). No-tice that parse() is an overridden method. Find out what class it isoverridden from.Commands ?? org.eclipse.ui.navigate.openResource? org.eclipse.ui.navigate.showIn40Appendix A. Study Tasks9. Search for the text Go to github. This message is a part of a dialogbox shown in the app. Comment the code that opens this dialog box.Also add a comment line explaining why the code was commented.Commands ?? org.eclipse.ui.edit.text.hippieCompletion10. Open the file Add an overridden method getComic-Name() to this class. This method should return the string Cyanideand Happiness Comic.Commands ?? org.eclipse.ui.navigate.openResource? org.eclipse.ui.edit.text.contentAssist.proposals? org.eclipse.ui.edit.text.hippieCompletion11. Name 3 classes defined inside the class ActivityComicReader.Commands ?? org.eclipse.ui.navigate.openResource? org.eclipse.ui.navigate.showIn12. Open the file Navigate to the method downloadCur-rentStrip(). Name one method from the Downloader class that getsinvoked during the execution of downloadCurrentStrip().Commands ?? org.eclipse.ui.navigate.openResource? A. Study Tasks13. Open the file and navigate to line 622. You willfind the method ignoreArticle(String) with 3 if-conditions. Move thesecond if-condition below the third if-condition.Commands ?? org.eclipse.ui.navigate.openResource? org.eclipse.ui.edit.text.goto.line? org.eclipse.ui.editors.lineNumberToggle? org.eclipse.ui.edit.text.moveLineDown14. Open the file and navigate to line 51. You willfind the declaration of the variable mClasses. Find out where a newobject gets created and gets assigned to the variable mClasses.Commands ?? org.eclipse.ui.navigate.openResource? org.eclipse.ui.edit.text.goto.line? org.eclipse.ui.editors.lineNumberToggle? This is a debugging task. Debugging on Android emulator is a pain.To make debugging easier, we will use a di?erent project named Comi-cReaderDebugging. This project will run the classes as Java applica-tion on the computer instead of Android emulator. Switch to thisproject.Open the file Notice that this file has a main method.This allows you to run/debug this file as a Java application. Themethod getAllComicUrls() returns a list of URL strings for all thecomic strips of The Bug Comic. Find out the URL string for 212thitem in the list.Commands ?? org.eclipse.ui.navigate.openResource? A. Study Tasks? org.eclipse.debug.ui.commands.ToggleBreakpoint? org.eclipse.jdt.debug.ui.commands.Inspect? org.eclipse.debug.ui.commands.RunToLine? org.eclipse.jdt.debug.ui.commands.Display? org.eclipse.debug.ui.commands.StepInto? org.eclipse.debug.ui.commands.StepOver? org.eclipse.debug.ui.commands.StepReturn16. This is also a debugging task. Open the file GoGetARoomie.javaand notice that this file has also a main method. This allows youto run/debug this file as a Java application. Run it as a Java applica-tion. The application crashes and throws a NumberFormatExceptionon line 39. This exception is thrown if the string to be converted tointeger contains characters that are not numbers. Fix the code so theexception is not thrown anymore.Commands ?? org.eclipse.ui.navigate.openResource? org.eclipse.debug.ui.commands.ToggleBreakpoint? org.eclipse.jdt.debug.ui.commands.Inspect? org.eclipse.debug.ui.commands.RunToLine? org.eclipse.jdt.debug.ui.commands.Display? org.eclipse.debug.ui.commands.StepInto? org.eclipse.debug.ui.commands.StepOver? org.eclipse.debug.ui.commands.StepReturn43Appendix BRegistration FormFigure B.1 (shown on the next page for legibility reasons) shows the formgiven to the participants at the time of registration for the experiment.44Appendix B. Registration FormFigure B.1: Registration Form used for the experiment.45


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items