"Arts, Faculty of"@en . "Library, Archival and Information Studies (SLAIS), School of"@en . "DSpace"@en . "Library hi tech 17 (2), 197-206."@en . "Morrison, Heather"@en . "2008-06-26T23:29:59Z"@en . "1999"@en . "The verbal protocol method is used extensively in computer usability studies. This study was designed to test the feasibility of using the verbal protocol method as a means of conducting field research on the online catalogue. Ten undergraduate students conducted their own research on DRA\u00E2\u0080\u0099s Infogate. As they searched, they talked aloud about what keys they were pressing, what was happening on the screen,\r\nand their reactions. Transcripts of sessions were analyzed. A total of 65 different problems or comments were noted during search sessions, with 11 problems or comments noted in three or more sessions. This supports the hypothesis that the verbal protocol method is a valuable means of identifying common problems for users. It is suggested that the verbal protocol method offers great potential as a research tool in librarianship, for example in the area of interface design."@en . "https://circle.library.ubc.ca/rest/handle/2429/957?expand=metadata"@en . "95805 bytes"@en . "application/pdf"@en . "197 Introduction Usability is an important feature of any successful computer program. The online catalogue is one example of an extremely important interface for library users that ought to be as easy to use as possible. There is an extensive body of online catalogue research literature. Yet Borgman\u00E2\u0080\u0099s research (1996, 1986) reveals that, in spite of all this research, users continue to find online cata- logues difficult to use. A number of different approaches have been taken to research the online catalogue. Seymour (1991) reviewed research methodologies used in online public access catalogue user studies from March 1986-November 1989. Surveys were found to be the most frequently used method. Interviews, observation, controlled experiment, and transaction log analysis are reported as well. One \u00E2\u0080\u009Cthink aloud\u00E2\u0080\u009D (verbal protocol) study is mentioned, but details are not reported. Henderson et al. (1995) conducted a major study to evaluate the efficiency of the four most prominent user-based methods in computer usability studies: logged data, questionnaire, interview, and verbal protocol analysis. They found verbal protocol analysis to be the most efficient method of gathering data. Even combin- ing verbal protocol analysis with other techniques did not result in a statistically significant improve- ment in the quantity of data obtained. Originally developed as a research tool in the field of cognitive psychology, the verbal protocol method was initially used as a means of studying human problem-solving processes. For example, subjects would solve arithmetic problems while talking aloud. This method is also used in the field of expert system design, to gather data about how experts go about solving problems in their field of expertise. Verbal protocol is now used extensively in computer usability studies. Major advantages of the verbal protocol method are the quality and quantity of data obtained. One group of authors (Wiedenbeck et al., 1989) suggests that protocol analysis is par- ticularly useful in studying areas where little is currently known. The major disadvantage of the verbal protocol method is that it is time-consum- ing. Another disadvantage is the likelihood that the method of study has some impact on the behavior under observation. Following are the results of a study designed to examine the feasibility of using verbal protocol as a method of field research on the online catalogue. The quality and quantity of data produced by a Online catalogue research and the verbal protocol method Heather G. Morrison The author Heather G. Morrison is Circulation Services Coordinator, Arnold Guebert Library, Concordia University College of Alberta, Edmonton, Canada. Keywords Catalogues, Field research, Libraries, Library users, Online computing, User studies Abstract The verbal protocol method is used extensively in computer usability studies. This study was designed to test the feasibili- ty of using the verbal protocol method as a means of con- ducting field research on the online catalogue. Ten under- graduate students conducted their own research on DRA\u00E2\u0080\u0099s Infogate. As they searched, they talked aloud about what keys they were pressing, what was happening on the screen, and their reactions. Transcripts of sessions were analyzed. A total of 65 different problems or comments were noted during search sessions, with 11 problems or comments noted in three or more sessions. This supports the hypothesis that the verbal protocol method is a valuable means of identifying common problems for users. It is suggested that the verbal protocol method offers great potential as a research tool in librarianship, for example in the area of interface design. Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 pp. 197\u00E2\u0080\u0093206 \u00C2\u00A9 MCB University Press \u00C2\u00B7 ISSN 0737-8831 study using ten subjects was analyzed. The poten- tial usefulness of the method in online catalogue research is assessed, both as a development tool and as a tool to assist reference librarians in deter- mining the most common problems experienced by users. Methodology The study was conducted at Concordia Universi- ty College of Alberta, a small undergraduate liberal arts college located in Edmonton, Alberta. Concordia is a member of the NEOS Library Consortium, a group of about 20 academic, government, and health libraries. The consor- tium shares one online catalogue, which is main- tained at the University of Alberta. The NEOS Libraries\u00E2\u0080\u0099 Catalogue (sometimes referred to as The GATE) contains over three million titles, of which about 60,000 (or about 2 percent ) belong to Concordia. The automation platform for the NEOS Libraries\u00E2\u0080\u0099 Catalogue is DRA Version 2.5, and the DRA interface used at Concordia is Infogate. DRA offers three options for display of the online catalogue: search everything, search everything but highlight materials in the local collection, and limit to materials in the local collection. The option selected by Concordia is search every- thing with highlighting of materials in the local collection. The reason for this choice was to balance the needs of Concordia clients to find materials immediately available on the shelves with the needs of Concordia clients to access the wider range of resources available through the NEOS Consortium. Until recently Concordia\u00E2\u0080\u0099s collection was classified using the Dewey Decimal Classifica- tion System. About two years ago, new acquisi- tions began to be catalogued using the Library of Congress Classification system. The library also uses the CODOC classification system for gov- ernment documents. The majority of the collec- tion is classified with Dewey. Users of online catalogue stations were asked if they would participate in a study to test the feasibility of the talk aloud method as a means of researching the online catalogue. An attempt was made to minimize bias in the sample by selecting times to approach potential participants in advance, numbering online cata- logue stations, and selecting the order of online catalogue stations at which to approach people on a rotating basis. Later, the rotating approach was abandoned. The researcher was approaching potential participants after leaving the researcher\u00E2\u0080\u0099s office, and it appeared awkward to walk past certain online catalogue stations in order to approach participants at more distant computers. Recruitment attempts also appeared to be less successful when people at closer com- puters were skipped. At this point (about halfway through the sessions), the researcher decided to follow the same order of approach on each occa- sion. Participants conducted their own searches in the privacy of the researcher\u00E2\u0080\u0099s office, with the researcher present, while talking aloud about their searches, what keys they were pressing and how they reacted to information on the screen. Search sessions were recorded on audiotape. Brief interviews followed the search sessions. For a detailed script including recruitment proce- dures, instructions, and questions, see Appendix. A total of 25 minutes was spent recruiting for eight sessions, or an average of three minutes per session. The number of sessions included in recruitment data is eight rather than ten, because the first two sessions were pilot sessions. A total of 14 people were asked to participate for a total participation rate of 57 percent . One participant approached the researcher at the request of someone who had declined to participate because they had finished their searching for the day. On ten occasions, there was no one available at GATE stations. Time per session averaged approximately 171 minutes, including recruitment and data analy- sis: \u00E2\u0080\u00A2 recruitment \u00E2\u0080\u0093 3 minutes; \u00E2\u0080\u00A2 session \u00E2\u0080\u0093 23 minutes; \u00E2\u0080\u00A2 transcription \u00E2\u0080\u0093 69 minutes; \u00E2\u0080\u00A2 data analysis (initial) \u00E2\u0080\u0093 16 minutes; \u00E2\u0080\u00A2 collating data (est.) \u00E2\u0080\u0093 60 minutes; \u00E2\u0080\u00A2 total time/session: 171 minutes/session. Characteristics of participants: 60 percent of the participants were female, 40 percent male. At least seven of the ten participants were senior students (second year of university or higher); one was a first year student. Data analysis involved reviewing transcripts and grouping together similar problems or com- ments. Problems or comments noted in tran- scripts of search sessions and in response to interview questions were tabulated separately for comparison purposes. 198 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206 Results Problems and comments Table I shows the problems and comments noted by researcher in decreasing order of frequency. Interview questions In order to compare the number of comments obtained through the search sessions and in response to interview questions, answers to the first four questions relating to the online catalogue have been collated (see Table II). These four questions were: (1) During this session, what aspects of the online catalogue did you find were the most problematic? (2) During this session, what aspects of the online catalogue did you find were the most attractive? (3) Based on your past experiences with the online catalogue, what aspects have you found problematic or attractive? (4) Do you have any other comments or ques- tions about the online catalogue? 199 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206 Table I Problems and comments Obser- Number of vation sessions Comment / problem 1 5 Prefer materials available onsite 2 5 Difficulty determining which term to use in a subject search 3 5 Find menu or subject index coming down over search results 4 4 Arrowing past where you want to go 5 3 Difficulty moving from screen with locations to call number and availability information 6 3 Subject search results in \u00E2\u0080\u009Cno matches\u00E2\u0080\u009D message, participant finds no helpful information 7 3 Subject search results in a long hit list (161 \u00E2\u0080\u0093 980 hits); one participant satisfied with results, others frustrated 8 3 Hot keys / letters (appreciated once participant knew about them) 9 3 Call number questions and comments: \u00E2\u0080\u00A2 what does it mean when there are letters in front of the call numbers? \u00E2\u0080\u00A2 man, these numbers are long, eh! \u00E2\u0080\u00A2 \u00E2\u0080\u009CLB\u00E2\u0080\u009D in call number interpreted as \u00E2\u0080\u009Clibrary\u00E2\u0080\u009D 10 3 Typing problems 11 3 Periodicals \u00E2\u0080\u00A2 wonders whether to search for books or periodicals on a topic \u00E2\u0080\u00A2 is it possible to go into periodicals from here? \u00E2\u0080\u00A2 I have no idea what [the professor] means by the journals 12 2 Not sure how to move from title list to locations 13 2 Difficulty locating items known to be at Concordia 14 2 Needed help using \u00E2\u0080\u009Climit by location\u00E2\u0080\u009D 15 2 Difficulty identifying libraries 16 2 List of subject headings and subheadings (or related terms) confuses participant 17 2 Uses related term or term that appeared in prior search as a subject search term 18 2 Losing information/getting lost when trying to back up 19 2 Spelling problems 20 2 Infogate request function (thought it was turned on, but it wasn\u00E2\u0080\u0099t) 21 2 Tried to use mouse in Infogate 22 2 Interpreting citations 23 1 Excited at seeing something highlighted (would be available at Concordia) 24 1 Using highlighting to identify availability at Concordia, even in a long list (Continued) 200 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206 Table I Obser- Number of vation sessions Comment/problem 25 1 Will inquire at the circulation desk of an off-site library regarding \u00E2\u0080\u009Con order\u00E2\u0080\u009D, \u00E2\u0080\u009Corder received\u00E2\u0080\u009D and overdue (date due is past) statuses 26 1 Loses interest in item on seeing \u00E2\u0080\u009Con order\u00E2\u0080\u009D status 27 1 \u00E2\u0080\u009CPress down and select locations to give you holdings at other locations\u00E2\u0080\u009D doesn\u00E2\u0080\u0099t work (worked when researcher tried later) 28 1 Pressing line number doesn\u00E2\u0080\u0099t work to see locations of an \u00E2\u0080\u009Corder received\u00E2\u0080\u009D item 29 1 Difficulty trying to figure out if other locations have an item 30 1 Wonders if \u00E2\u0080\u009Cno holdings\u00E2\u0080\u009D message means no holdings in NEOS, not just Concordia 31 1 Found a number of titles, but none at Concordia 32 1 Not sure how to figure out which location has a particular volume of a journal 33 1 Not sure how to interpret volume/issue information 34 1 Mentioned not having materials available onsite as a shortcoming 35 1 Missed periodical holdings 36 1 Using subject search when subject keyword would work 37 1 Assumed a limit of two words in a subject keyword search because three didn\u00E2\u0080\u0099t work 38 1 Subject search results in a lot of irrelevant information 39 1 Not sure whether to use keyword or subject search 40 1 Using expert keyword strategy in a keyword search 41 1 Expert keyword search results in no hits although terms used are in the database 42 1 Finding a call number to browse around on the shelves 43 1 Quick, successful subject search (subject keyword) 44 1 Backing up from location to main subject heading didn\u00E2\u0080\u0099t work (did work later when researcher tried this) 45 1 Does page down work? 46 1 \u00E2\u0080\u009CArrowing\u00E2\u0080\u009D is very good for one-finger typists 47 1 Not sure of printing procedures 48 1 Printing locations is cool 49 1 Press a key, that\u00E2\u0080\u0099s good, it\u00E2\u0080\u0099s highlighted, it makes you look there first 50 1 Disappointing to find books that may be relevant but in a language you cannot understand 51 1 Found name of an author in a previous search but did not get the call number; came back and searched by author, but did not find the item; searched by other means and found the item; why did it not come up by author? 52 1 Question about putting a book belonging to another library on hold 53 1 Confused when item is not found at all, appears to assume anything in a particular index should be available through the NEOS consortium 54 1 Uses author search because title is so long (although title is more unique than author) 55 1 May have abandoned difficult title search due to the influence of the research project 56 1 Author search, last name only (common name) leads to lots of paging down to find individual author 57 1 Using publisher information (and keywords, e.g. Congressional in title) to determine which is Canadian information 58 1 Frustrated at amount of American information found when looking only for Canadian information 59 1 Two author records found for a single author 60 1 Connecting to the GATE is so slow 61 1 NEOS online catalogue icon: would never have guessed that this is what it would be called 62 1 Help: only one participant attempted to use DRA\u00E2\u0080\u0099s online help (to figure out how to limit by location), with no success 63 1 Options: one participant tried options, which was irrelevant to the search 64 1 I\u00E2\u0080\u0099ve used microfiche before, but I\u00E2\u0080\u0099m not too sure how to use them 65 1 Usually goes to the public library, but prof won\u00E2\u0080\u0099t allow for this assignment 105 Total problems and comments Post-search questions Question: Do you have any other comments or questions about this research project? Four partici- pants had no comments. In three sessions, the question was not asked. Two participants had positive comments (pretty good, very interest- ing). One suggested that the researcher mention at the beginning that you\u00E2\u0080\u0099d like the person to kind of talk out their actions, and explain what\u00E2\u0080\u0099s 201 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206 Table II Interview questions Obser- Number of vation sessions In decreasing order of frequency Problems 1 6 Subject terminology \u00E2\u0080\u0093 not knowing what terms to use and/or lack of computer assistance, e.g.: \u00E2\u0080\u00A2 no matches on subject search and no assistance given \u00E2\u0080\u00A2 screen with list of subject headings and subheadings is confusing 2 5 Material not available locally 3 2 Instructions at bottom of screen are: \u00E2\u0080\u00A2 hard to read \u00E2\u0080\u00A2 easy to ignore 4 2 Bringing menu down over results 5 2 Overshooting with arrow keys 6 2 Interpreting citations: \u00E2\u0080\u00A2 cannot tell if it is an anthology or journal, etc. \u00E2\u0080\u00A2 does the word \u00E2\u0080\u009Creferences\u00E2\u0080\u009D in a citation mean material not available for loan? 7 2 Whether to use subject or keyword 8 2 Moving around (up/down, between menus) 9 1 Cluttered screen 10 1 Request function (on OPAC) is not available 11 1 Limiting to Canadian information takes time to learn 12 1 Retrieving items in different languages 13 1 Not finding an item you have found before when you have author information 14 1 Trying to figure out which icon to click on to start a search 15 1 Limiting by location could be easier Attractive features 16 7 Location and/or availability information 17 5 Fast 18 3 I like the OPAC/it\u00E2\u0080\u0099s really good 19 2 Selection 20 2 Limiting 21 2 Keywords 22 2 Printing: like being able to print results 23 2 Improvement over older OPAC or card catalogue 24 1 Highlighting 25 1 Are you sure you really want to quit option 26 1 Backup and forward 27 1 The fact that I could access the book from so many directions 28 1 Arrow keys 29 1 Subject keyword 30 1 The actual information you get is pretty good Other comments 31 3 Shortcuts: would like help identifying shortcuts 32 1 Question about how to find a CODOC call number 66 Total problems, attractive features, and other comments about the online catalogue on the screen. One participant said that you could manage without a practice session. Question: Did you find the information you needed? Five participants answered yes; one participant had found some of the information needed. The question was not asked in four sessions. One participant was definitely having difficulty finding information and was referred to the reference desk. One participant found a few citations, not available at Concordia, but remarked after the session that they knew we had five books on the subject, and where to find them on the shelves. Question: Based on the online catalogue the way it is now, is there anything that the library staff could do to help people find what they need in terms of orientation or written guides or\u00E2\u0080\u00A6 Comments on orientation or written guides Comments on orientation or written guides in decreasing order of frequency are shown in Table III. Discussion Participants seemed to find the verbal protocol instructions easy to follow. In the two pilot sessions, participants were given a practice search, but this was abandoned because partici- pants did not find it necessary. Initially, the plan was to obtain logged data during the sessions as well, in order to compare comments with what was happening on the screen at the time. This was abandoned after the first pilot session. The log from one search session (QVT Terminal) was over 700 pages long. Timing data appeared only at the begin- ning and end of the session, which would have made it very difficult to match the log and audiotape. Also, the data recorded on audiotape appeared to provide sufficient content to deter- mine what was happening on the screen at any given time. A greater quantity of data was obtained from the search sessions using the verbal protocol method than from the interview questions. A total of 65 different problems and comments 202 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206 Table III Comments on orientation or written guides Obser Number of -vation sessions Comment 1 3 Forgetting information is a problem (orientation) 2 2 Never taken one (orientation) 3 2 List of subject headings, keywords (written guide) 4 2 Quick information beside computer: \u00E2\u0080\u00A2 short note about a number of the convenient options on the online catalogue placed by the computer \u00E2\u0080\u00A2 step-by-step information, with examples, beside the computers 5 2 Important to have reference service available 6 1 Orientations are the best here 7 1 Orientations are good for the first-year students 8 1 Orientations are too long 9 1 Orientations are theoretical / observational rather than hands-on 10 1 Orientation was disconnected with anything students were doing at the time 11 1 How to limit by location (written guide) 12 1 Everyone here is available to help you out (so orientation and written guides are not necessary) 13 1 Standardized terms among different catalogues/universal system 14 1 Catalogue that the public library used before the GATE was more user-friendly, although it had less room to expand 15 1 Some people can figure out the GATE without instruction or written guides 16 1 What does a call number starting with HV mean? 17 1 Proquest is fantastic, and pretty booked up\u00E2\u0080\u00A6another of those, or for different subjects, like education (At the time of the study, Concordia had a subscription to Proquest Social Sciences only, with two workstations. This subscription has since been canceled and replaced with the broader subject coverage of Infotrac\u00E2\u0080\u0099s Expanded Academic ASAP) 18 1 Which computer has what? 24 Total comments were noted based on the search sessions, with a total of 105 problems and comments including repetitions. In contrast, a total of 32 different problems, attractive features, and other com- ments were counted in response to interview questions about the online catalogue, with a total of 66 problems, attractive features, and comments. This suggests that the verbal proto- col method is a richer source of data than inter- view questions. Even with this small sample size, 11 prob- lems or comments were noted in three or more of the ten search sessions, suggesting that this method can be useful in determining some of the more common problems experienced by system users. In contrast, only five problems or attractive features were noted in three or more of the ten sessions in response to interview questions on the online catalogue. There was overlap in the data obtained by both methods. For example, the top two prob- lems or comments noted both during search sessions and in response to interview questions about problems with the online catalogue were the same. These top two comments were a preference for material available onsite and that users found subject searching to be difficult. Data obtained in the verbal protocol search sessions tended to be more technically detailed than answers to interview questions. This is not surprising. It seems unlikely that people would remember the technical details of a complex problem encountered some time ago. Explain- ing what is appearing on the screen right at the moment is relatively easy. Another advantage of this method is that it is possible to determine the effect of a problem on an actual search. Some problems may be minor nuisances while others can be significant obsta- cles that many individuals could not overcome without help. The frequency with which a particular prob- lem occurs may or may not be an indication of its importance. For example, the fact that only one participant in this study attempted to use DRA\u00E2\u0080\u0099s online Help, and did not find any helpful information, may be significant in itself. Some of the information obtained through asking interview questions did not overlap data obtained through the verbal protocol search sessions. This was particularly true of attractive features. This suggests that combining both types of information gathering in this type of study would be more valuable than the verbal protocol method alone. One potential problem with the approach to recruitment used in this study, approaching people already at PAC stations, is that many of the resulting sessions may have been only partial search sessions. As a result, common problems generally encountered at the beginning of a search may have been missed. This may also have distorted the search session times. Another difficulty with recruitment encoun- tered in this study could have been avoided. The study was conducted relatively early in the academic semester, when the OPAC stations were less busy. A study conducted during a busier period may have had a lower rate of no one being available to ask to participate. Results of this study cannot be generalized to the population of Concordia library users as a whole, because of the small and non-random sample. The presence of the researcher during search sessions and the talk-aloud method per se may have had an impact on the searches. For ethical reasons, subjects experiencing difficulties with searches were allowed to ask for assistance. In some cases the researcher provided direction during searches when assistance appeared to be needed, but was not requested. The researcher also asked some probing questions. The pres- ence of the researcher and the researcher\u00E2\u0080\u0099s interactions with participants make this a hybrid research method, part verbal protocol, part interview. The talk aloud method itself may have had some impact on the participants\u00E2\u0080\u0099 search strate- gy. For example, participants may have tried to demonstrate what they considered to be the \u00E2\u0080\u009Ccorrect\u00E2\u0080\u009D method of searching rather than searching in their usual manner. Further research would be needed to deter- mine the frequency with which problems noted during this study occur. Results of this study may be useful in indicating areas which merit further study. Most of the problems appeared to fall into one of three categories: (1) availability and limiting by location; (2) subject searching; and (3) arrow and keystroke. 203 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206 One finding that could be relevant to develop- ment of online catalogues is that it appears to be very important, at least to this group of subjects, for users to identify easily those materials that are available locally. Although half of the partici- pants indicated a preference for materials avail- able in the local collection, none were aware of DRA\u00E2\u0080\u0099s \u00E2\u0080\u009Climit by location\u00E2\u0080\u009D feature. Even those who requested assistance found it difficult to limit by location. Simplifying this task would improve the usefulness of the online catalogue. Difficulties with subject searching have been noted by a number of researchers. Clear online instructions for users when either no matches or long hit lists are retrieved could be helpful. Signage in the reference area clearly pointing to subject help guides, i.e. The Library of Con- gress Subject Headings, might alleviate some problems for some users. Arrowing and keystroke problems might disappear with the next generation of online catalogues based on graphical user interfaces. Recommendations for Concordia Half the participants in this study indicated a preference for materials available onsite, but determining availability and limiting by location were the most common problems identified. This supports the prior belief of Concordia library staff that identifying materials available locally is difficult. One strategy which staff has implemented to minimize this problem is to enhance the browsability of the collection. Signage has been added at the end of ranges listing subject headings for all areas where collection size is significant (more than 50 titles). A collection of pathfinders covers all major subject areas taught at Concordia. Library of Congress major subject headings are posted prominently in the reference area and a list of Dewey subject headings is readily avail- able as well. While this approach may be useful in providing clients with alternative methods of finding materials in the collection, it does not address the main problem, which is that clients want to be able to use the online catalogue to easily identify materials in the local collection. Setting DRA to limit materials available locally would solve the problems clients have in locating materials available locally, but would make it somewhat more difficult for clients to identify materials available through the NEOS Consortium. The importance of having materials available locally may be decreasing as interlibrary loan activity increases. A few years ago, interlibrary loan activity was negligible at Concordia. This was due partly to the lack of delivery mecha- nisms fast enough to meet the needs of an undergraduate population, and partly due to the availability of a number of high quality research and special libraries in the Edmonton area (e.g. the University of Alberta, government depart- ment libraries). Interlibrary loan activity has increased dra- matically in the past few years, primarily as a result of the document delivery agreement within NEOS that guarantees a three-day turn- around time. The merger of circulation and interlibrary loan services and faster delivery service (because of the courier service and the use of Ariel for articles) have both helped to improve the availability of interlibrary loan services during the library\u00E2\u0080\u0099s open hours. This trend is likely to continue to increase as Concordia becomes involved in more resource- sharing agreements, particularly The Alberta Library, which involves public, academic, and regional library systems throughout the province. For example, one of the projects of The Alberta Library is involved in is ensuring that all post-secondary libraries throughout the province have either Relais (large institutions) or Ariel (smaller institutions) workstations for document delivery, thanks to funding provided by Alberta Advanced Education and Career Development. This will make requesting of articles much more convenient and desirable for Concordia\u00E2\u0080\u0099s undergraduates. Library instruc- tion sessions now place more emphasis both on teaching students to identify materials available locally and how to place interlibrary loan requests. It is hoped that DRA\u00E2\u0080\u0099s new TAOS operating system, currently under development, will address the difficulty of identifying materials available locally. From the point of view of the client of the small library in a big consortium, it would be ideal if the main screen would include the option of searching the current location only (i.e. Concordia only), the whole NEOS 204 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206 database, or other options using Z39.50, as determined by Concordia. Permitting individ- ual locations to set up their own search options would be ideal because even within a consor- tium, different institutions or groups of institu- tions, have different licensing arrangements for databases, and different resource-sharing agree- ments with other agencies. Arrow and keystroke problems will eventual- ly be eliminated by moving to a Web-based interface. Currently, DRA Web is available to the NEOS Consortium. However, Infogate is preferred at Concordia, because DRA Web does not allow for limiting by location, and Borrower Information has not been made available due to concerns that it does not work correctly. Emphasis has been placed on Infogate only, because although some kinds of searching would be easier on DRA Web, it is believed that the complications of teaching students two different interfaces for the online catalogue would outweigh the benefits. The DRA Web2 interface is expected to be available to the NEOS Consortium by fall of 1999. Concordia staff will evaluate this interface and determine whether to continue emphasizing Infogate or switch to DRA Web2. Suggestions for further research The verbal protocol method offers great poten- tial as a research method in a number of areas of librarianship. When designing interfaces, whether for intranets, or customized interfaces for various library databases or products, the verbal protocol method can be used to identify problems for users before the interface is made available to clientele. The verbal protocol method also has potential as a tool for analyzing the research process or the reference interview. Wiedenbeck et al.(1989) provide a concise, useful description of how to conduct a verbal protocol study. Conclusion The verbal protocol method is a useful means of obtaining data on the online catalogue. A total of 11 problems or comments were noted in three or more of the ten search sessions, in contrast to five problems or attractive features noted in three or more of the ten sessions in response to interview questions. Participants find it easy to follow instructions to comment on what keys they are pressing, what they see on the screen, and how they react. These comments make it easy to determine what was happening on the screen when tran- scribing sessions. When planning a verbal proto- col study, allow about three times as much time for the average search session to allow for recruitment, transcription, and data analysis. This method has a lot of potential applica- tions in librarianship, including interface design, the research process, and the reference interview. Librarians might find this a natural method to use, as it is somewhat similar to conducting a reference interview. Unlike Henderson et al.\u00E2\u0080\u0099s (1995) study, it was found that interview questions provided addi- tional information not obtained through the verbal protocol search sessions. Answers to interview questions could also help to indicate the relative importance of problems to users, i.e. sort out the difference between a common nuisance and a genuine obstacle to research. The results of this study (remembering the small size of the sample) indicate three main areas of concern for users: identifying what is held locally and what is available, subject searching, and arrowing and keystrokes. References Borgman, C.L. (1986), \u00E2\u0080\u009CWhy are online catalogs hard to use? Lessons learned from information retrieval studies\u00E2\u0080\u009D, Journal of the American Society for Information Science, Vol. 37 No. 6, pp. 387-400. Borgman, C.L. (1996), \u00E2\u0080\u009CWhy are online catalogs still hard to use?\u00E2\u0080\u009D, Journal of the American Society for Information Science, Vol. 47 No. 7, pp. 493-503. Henderson, R.D., Smith, M.C., Podd, J. and Varela-Alvarez, H. (1995), \u00E2\u0080\u009CA comparison of the tour prominent user- based methods for evaluating the usability of computer software\u00E2\u0080\u009D, Ergonomics, Vol. 38 No.10, pp. 2030-44. Seymour, S. (1991), \u00E2\u0080\u009COnline public access catalog user studies: a review of research methodologies, March 1986-November 1989\u00E2\u0080\u009D, Library and Information Science Research, Vol. 13, April-June, pp. 89-102. Wiedenbeck, S., Lampert, R. and Scholtz, J. (1989), \u00E2\u0080\u009CUsing protocol analysis to study the user interface\u00E2\u0080\u009D, Bulletin of the American Society for Information Science, Vol. 15 No. 5, pp. 25-6. 205 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206 Appendix Detailed script including recruitment procedures, instructions, and questions Verbal protocol script Try for some randomization of recruitment, e.g. pre-select times on a random basis to approach people; number computer terminals, approach first person to go to that terminal at that partic- ular time and say something like this: Hi. I\u00E2\u0080\u0099m conducting a research study on the online catalogue. May I take about a minute of your time to briefly explain my project and ask for your participation? My name is Heather Morrison. As you may know, I am the Circulation Services Coordinator here at Concordia. I am also working on a Master\u00E2\u0080\u0099s degree in Library and Information Studies at the University of Alberta. As part of my work towards this degree, I am conducting a study to test the talk aloud method as a way of studying the online catalogue, or GATE. What I am asking people to do is to conduct the searches they came to the library to do, while talking aloud about what they\u00E2\u0080\u0099re doing and how they react to the computer interface. This would be done in the privacy of my office, with myself present as an observer. Participation is purely voluntary; please feel free to say no if you would rather not participate. The search session will be recorded on audiotape and is completely confi- dential. After the search session, there will be a brief interview. No information that would identi- fy you as an individual would be collected. The time required for this study is about the same as your search is likely to take, plus about five extra minutes for talking and questions, including this time. Are you willing to participate in this study? If no, then say: \u00E2\u0080\u009Cthanks for your time, and good luck with your search.\u00E2\u0080\u009D If yes, say: I\u00E2\u0080\u0099d like to give you a copy of this brief summary of this research project, including the name and phone number of my instructor, should you have any concerns or questions about the study. Talk aloud study Say: The main focus of my study is evaluating the computer interface of the online catalogue, that is, does it help people to find the information or materials they need, or does it need improvement? So, what I\u00E2\u0080\u0099d like you to do is to conduct your search as you usually would, but talk aloud about what you\u00E2\u0080\u0099re doing, that is, what keys you\u00E2\u0080\u0099re press- ing, what you\u00E2\u0080\u0099re seeing on the screen, what you\u00E2\u0080\u0099re looking for on the screen, that type of thing, as well as how you feel about the information the computer is giving you. Do you have any questions before we start? Any comments to participants, if needed, both at this time and during session, are restricted to: \u00E2\u0080\u00A2 talking about what they are doing; \u00E2\u0080\u00A2 commenting on the reasons for what they\u00E2\u0080\u0099re doing; \u00E2\u0080\u00A2 comments about how they feel about what\u00E2\u0080\u0099s on the computer. After the session, ask: \u00E2\u0080\u00A2 About any specific questions noted during the session. \u00E2\u0080\u00A2 What aspects of the online catalogue did you find were the most problematic during the session? \u00E2\u0080\u00A2 What aspects of the online catalogue did you find were the most attractive during the session? \u00E2\u0080\u00A2 Based on past experiences with the online catalogue, what aspects have you found problematic or attractive? \u00E2\u0080\u00A2 Do you have any other comments or ques- tions about the online catalogue? \u00E2\u0080\u00A2 Given the online catalogue as it is now, what can library staff do to help people find the things they need, in terms of orientation or written guides? \u00E2\u0080\u00A2 Do you have any other comments or ques- tions about this research project? \u00E2\u0080\u00A2 Did you find the information you needed? (If not, provide reference assistance or refer subject to the reference desk.) Finally, say: I\u00E2\u0080\u0099d like to thank you very much for participating. If you\u00E2\u0080\u0099d like a copy of the final report of my research, it should be available in early April; you can pick up a copy here at the Circulation Desk. 206 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 \u00C2\u00B7 Number 2 \u00C2\u00B7 1999 \u00C2\u00B7 197\u00E2\u0080\u0093206"@en . "Article"@en . "10.14288/1.0107431"@en . "eng"@en . "Reviewed"@en . "Vancouver : University of British Columbia Library"@en . "MCB University Press"@en . "All rights reserved"@en . "Faculty"@en . "Usability testing"@en . "Online library catalogues"@en . "Interface design"@en . "Verbal protocol method"@en . "Online catalogue research and the verbal protocol method"@en . "Text"@en . "http://hdl.handle.net/2429/957"@en .