197 Introduction Usability is an important feature of any successful computer program. The online catalogue is one example of an extremely important interface for library users that ought to be as easy to use as possible. There is an extensive body of online catalogue research literature. Yet Borgman’s research (1996, 1986) reveals that, in spite of all this research, users continue to find online cata- logues difficult to use. A number of different approaches have been taken to research the online catalogue. Seymour (1991) reviewed research methodologies used in online public access catalogue user studies from March 1986-November 1989. Surveys were found to be the most frequently used method. Interviews, observation, controlled experiment, and transaction log analysis are reported as well. One “think aloud” (verbal protocol) study is mentioned, but details are not reported. Henderson et al. (1995) conducted a major study to evaluate the efficiency of the four most prominent user-based methods in computer usability studies: logged data, questionnaire, interview, and verbal protocol analysis. They found verbal protocol analysis to be the most efficient method of gathering data. Even combin- ing verbal protocol analysis with other techniques did not result in a statistically significant improve- ment in the quantity of data obtained. Originally developed as a research tool in the field of cognitive psychology, the verbal protocol method was initially used as a means of studying human problem-solving processes. For example, subjects would solve arithmetic problems while talking aloud. This method is also used in the field of expert system design, to gather data about how experts go about solving problems in their field of expertise. Verbal protocol is now used extensively in computer usability studies. Major advantages of the verbal protocol method are the quality and quantity of data obtained. One group of authors (Wiedenbeck et al., 1989) suggests that protocol analysis is par- ticularly useful in studying areas where little is currently known. The major disadvantage of the verbal protocol method is that it is time-consum- ing. Another disadvantage is the likelihood that the method of study has some impact on the behavior under observation. Following are the results of a study designed to examine the feasibility of using verbal protocol as a method of field research on the online catalogue. The quality and quantity of data produced by a Online catalogue research and the verbal protocol method Heather G. Morrison The author Heather G. Morrison is Circulation Services Coordinator, Arnold Guebert Library, Concordia University College of Alberta, Edmonton, Canada. Keywords Catalogues, Field research, Libraries, Library users, Online computing, User studies Abstract The verbal protocol method is used extensively in computer usability studies. This study was designed to test the feasibili- ty of using the verbal protocol method as a means of con- ducting field research on the online catalogue. Ten under- graduate students conducted their own research on DRA’s Infogate. As they searched, they talked aloud about what keys they were pressing, what was happening on the screen, and their reactions. Transcripts of sessions were analyzed. A total of 65 different problems or comments were noted during search sessions, with 11 problems or comments noted in three or more sessions. This supports the hypothesis that the verbal protocol method is a valuable means of identifying common problems for users. It is suggested that the verbal protocol method offers great potential as a research tool in librarianship, for example in the area of interface design. Library Hi Tech Volume 17 · Number 2 · 1999 · pp. 197–206 © MCB University Press · ISSN 0737-8831 study using ten subjects was analyzed. The poten- tial usefulness of the method in online catalogue research is assessed, both as a development tool and as a tool to assist reference librarians in deter- mining the most common problems experienced by users. Methodology The study was conducted at Concordia Universi- ty College of Alberta, a small undergraduate liberal arts college located in Edmonton, Alberta. Concordia is a member of the NEOS Library Consortium, a group of about 20 academic, government, and health libraries. The consor- tium shares one online catalogue, which is main- tained at the University of Alberta. The NEOS Libraries’ Catalogue (sometimes referred to as The GATE) contains over three million titles, of which about 60,000 (or about 2 percent ) belong to Concordia. The automation platform for the NEOS Libraries’ Catalogue is DRA Version 2.5, and the DRA interface used at Concordia is Infogate. DRA offers three options for display of the online catalogue: search everything, search everything but highlight materials in the local collection, and limit to materials in the local collection. The option selected by Concordia is search every- thing with highlighting of materials in the local collection. The reason for this choice was to balance the needs of Concordia clients to find materials immediately available on the shelves with the needs of Concordia clients to access the wider range of resources available through the NEOS Consortium. Until recently Concordia’s collection was classified using the Dewey Decimal Classifica- tion System. About two years ago, new acquisi- tions began to be catalogued using the Library of Congress Classification system. The library also uses the CODOC classification system for gov- ernment documents. The majority of the collec- tion is classified with Dewey. Users of online catalogue stations were asked if they would participate in a study to test the feasibility of the talk aloud method as a means of researching the online catalogue. An attempt was made to minimize bias in the sample by selecting times to approach potential participants in advance, numbering online cata- logue stations, and selecting the order of online catalogue stations at which to approach people on a rotating basis. Later, the rotating approach was abandoned. The researcher was approaching potential participants after leaving the researcher’s office, and it appeared awkward to walk past certain online catalogue stations in order to approach participants at more distant computers. Recruitment attempts also appeared to be less successful when people at closer com- puters were skipped. At this point (about halfway through the sessions), the researcher decided to follow the same order of approach on each occa- sion. Participants conducted their own searches in the privacy of the researcher’s office, with the researcher present, while talking aloud about their searches, what keys they were pressing and how they reacted to information on the screen. Search sessions were recorded on audiotape. Brief interviews followed the search sessions. For a detailed script including recruitment proce- dures, instructions, and questions, see Appendix. A total of 25 minutes was spent recruiting for eight sessions, or an average of three minutes per session. The number of sessions included in recruitment data is eight rather than ten, because the first two sessions were pilot sessions. A total of 14 people were asked to participate for a total participation rate of 57 percent . One participant approached the researcher at the request of someone who had declined to participate because they had finished their searching for the day. On ten occasions, there was no one available at GATE stations. Time per session averaged approximately 171 minutes, including recruitment and data analy- sis: • recruitment – 3 minutes; • session – 23 minutes; • transcription – 69 minutes; • data analysis (initial) – 16 minutes; • collating data (est.) – 60 minutes; • total time/session: 171 minutes/session. Characteristics of participants: 60 percent of the participants were female, 40 percent male. At least seven of the ten participants were senior students (second year of university or higher); one was a first year student. Data analysis involved reviewing transcripts and grouping together similar problems or com- ments. Problems or comments noted in tran- scripts of search sessions and in response to interview questions were tabulated separately for comparison purposes. 198 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206 Results Problems and comments Table I shows the problems and comments noted by researcher in decreasing order of frequency. Interview questions In order to compare the number of comments obtained through the search sessions and in response to interview questions, answers to the first four questions relating to the online catalogue have been collated (see Table II). These four questions were: (1) During this session, what aspects of the online catalogue did you find were the most problematic? (2) During this session, what aspects of the online catalogue did you find were the most attractive? (3) Based on your past experiences with the online catalogue, what aspects have you found problematic or attractive? (4) Do you have any other comments or ques- tions about the online catalogue? 199 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206 Table I Problems and comments Obser- Number of vation sessions Comment / problem 1 5 Prefer materials available onsite 2 5 Difficulty determining which term to use in a subject search 3 5 Find menu or subject index coming down over search results 4 4 Arrowing past where you want to go 5 3 Difficulty moving from screen with locations to call number and availability information 6 3 Subject search results in “no matches” message, participant finds no helpful information 7 3 Subject search results in a long hit list (161 – 980 hits); one participant satisfied with results, others frustrated 8 3 Hot keys / letters (appreciated once participant knew about them) 9 3 Call number questions and comments: • what does it mean when there are letters in front of the call numbers? • man, these numbers are long, eh! • “LB” in call number interpreted as “library” 10 3 Typing problems 11 3 Periodicals • wonders whether to search for books or periodicals on a topic • is it possible to go into periodicals from here? • I have no idea what [the professor] means by the journals 12 2 Not sure how to move from title list to locations 13 2 Difficulty locating items known to be at Concordia 14 2 Needed help using “limit by location” 15 2 Difficulty identifying libraries 16 2 List of subject headings and subheadings (or related terms) confuses participant 17 2 Uses related term or term that appeared in prior search as a subject search term 18 2 Losing information/getting lost when trying to back up 19 2 Spelling problems 20 2 Infogate request function (thought it was turned on, but it wasn’t) 21 2 Tried to use mouse in Infogate 22 2 Interpreting citations 23 1 Excited at seeing something highlighted (would be available at Concordia) 24 1 Using highlighting to identify availability at Concordia, even in a long list (Continued) 200 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206 Table I Obser- Number of vation sessions Comment/problem 25 1 Will inquire at the circulation desk of an off-site library regarding “on order”, “order received” and overdue (date due is past) statuses 26 1 Loses interest in item on seeing “on order” status 27 1 “Press down and select locations to give you holdings at other locations” doesn’t work (worked when researcher tried later) 28 1 Pressing line number doesn’t work to see locations of an “order received” item 29 1 Difficulty trying to figure out if other locations have an item 30 1 Wonders if “no holdings” message means no holdings in NEOS, not just Concordia 31 1 Found a number of titles, but none at Concordia 32 1 Not sure how to figure out which location has a particular volume of a journal 33 1 Not sure how to interpret volume/issue information 34 1 Mentioned not having materials available onsite as a shortcoming 35 1 Missed periodical holdings 36 1 Using subject search when subject keyword would work 37 1 Assumed a limit of two words in a subject keyword search because three didn’t work 38 1 Subject search results in a lot of irrelevant information 39 1 Not sure whether to use keyword or subject search 40 1 Using expert keyword strategy in a keyword search 41 1 Expert keyword search results in no hits although terms used are in the database 42 1 Finding a call number to browse around on the shelves 43 1 Quick, successful subject search (subject keyword) 44 1 Backing up from location to main subject heading didn’t work (did work later when researcher tried this) 45 1 Does page down work? 46 1 “Arrowing” is very good for one-finger typists 47 1 Not sure of printing procedures 48 1 Printing locations is cool 49 1 Press a key, that’s good, it’s highlighted, it makes you look there first 50 1 Disappointing to find books that may be relevant but in a language you cannot understand 51 1 Found name of an author in a previous search but did not get the call number; came back and searched by author, but did not find the item; searched by other means and found the item; why did it not come up by author? 52 1 Question about putting a book belonging to another library on hold 53 1 Confused when item is not found at all, appears to assume anything in a particular index should be available through the NEOS consortium 54 1 Uses author search because title is so long (although title is more unique than author) 55 1 May have abandoned difficult title search due to the influence of the research project 56 1 Author search, last name only (common name) leads to lots of paging down to find individual author 57 1 Using publisher information (and keywords, e.g. Congressional in title) to determine which is Canadian information 58 1 Frustrated at amount of American information found when looking only for Canadian information 59 1 Two author records found for a single author 60 1 Connecting to the GATE is so slow 61 1 NEOS online catalogue icon: would never have guessed that this is what it would be called 62 1 Help: only one participant attempted to use DRA’s online help (to figure out how to limit by location), with no success 63 1 Options: one participant tried options, which was irrelevant to the search 64 1 I’ve used microfiche before, but I’m not too sure how to use them 65 1 Usually goes to the public library, but prof won’t allow for this assignment 105 Total problems and comments Post-search questions Question: Do you have any other comments or questions about this research project? Four partici- pants had no comments. In three sessions, the question was not asked. Two participants had positive comments (pretty good, very interest- ing). One suggested that the researcher mention at the beginning that you’d like the person to kind of talk out their actions, and explain what’s 201 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206 Table II Interview questions Obser- Number of vation sessions In decreasing order of frequency Problems 1 6 Subject terminology – not knowing what terms to use and/or lack of computer assistance, e.g.: • no matches on subject search and no assistance given • screen with list of subject headings and subheadings is confusing 2 5 Material not available locally 3 2 Instructions at bottom of screen are: • hard to read • easy to ignore 4 2 Bringing menu down over results 5 2 Overshooting with arrow keys 6 2 Interpreting citations: • cannot tell if it is an anthology or journal, etc. • does the word “references” in a citation mean material not available for loan? 7 2 Whether to use subject or keyword 8 2 Moving around (up/down, between menus) 9 1 Cluttered screen 10 1 Request function (on OPAC) is not available 11 1 Limiting to Canadian information takes time to learn 12 1 Retrieving items in different languages 13 1 Not finding an item you have found before when you have author information 14 1 Trying to figure out which icon to click on to start a search 15 1 Limiting by location could be easier Attractive features 16 7 Location and/or availability information 17 5 Fast 18 3 I like the OPAC/it’s really good 19 2 Selection 20 2 Limiting 21 2 Keywords 22 2 Printing: like being able to print results 23 2 Improvement over older OPAC or card catalogue 24 1 Highlighting 25 1 Are you sure you really want to quit option 26 1 Backup and forward 27 1 The fact that I could access the book from so many directions 28 1 Arrow keys 29 1 Subject keyword 30 1 The actual information you get is pretty good Other comments 31 3 Shortcuts: would like help identifying shortcuts 32 1 Question about how to find a CODOC call number 66 Total problems, attractive features, and other comments about the online catalogue on the screen. One participant said that you could manage without a practice session. Question: Did you find the information you needed? Five participants answered yes; one participant had found some of the information needed. The question was not asked in four sessions. One participant was definitely having difficulty finding information and was referred to the reference desk. One participant found a few citations, not available at Concordia, but remarked after the session that they knew we had five books on the subject, and where to find them on the shelves. Question: Based on the online catalogue the way it is now, is there anything that the library staff could do to help people find what they need in terms of orientation or written guides or… Comments on orientation or written guides Comments on orientation or written guides in decreasing order of frequency are shown in Table III. Discussion Participants seemed to find the verbal protocol instructions easy to follow. In the two pilot sessions, participants were given a practice search, but this was abandoned because partici- pants did not find it necessary. Initially, the plan was to obtain logged data during the sessions as well, in order to compare comments with what was happening on the screen at the time. This was abandoned after the first pilot session. The log from one search session (QVT Terminal) was over 700 pages long. Timing data appeared only at the begin- ning and end of the session, which would have made it very difficult to match the log and audiotape. Also, the data recorded on audiotape appeared to provide sufficient content to deter- mine what was happening on the screen at any given time. A greater quantity of data was obtained from the search sessions using the verbal protocol method than from the interview questions. A total of 65 different problems and comments 202 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206 Table III Comments on orientation or written guides Obser Number of -vation sessions Comment 1 3 Forgetting information is a problem (orientation) 2 2 Never taken one (orientation) 3 2 List of subject headings, keywords (written guide) 4 2 Quick information beside computer: • short note about a number of the convenient options on the online catalogue placed by the computer • step-by-step information, with examples, beside the computers 5 2 Important to have reference service available 6 1 Orientations are the best here 7 1 Orientations are good for the first-year students 8 1 Orientations are too long 9 1 Orientations are theoretical / observational rather than hands-on 10 1 Orientation was disconnected with anything students were doing at the time 11 1 How to limit by location (written guide) 12 1 Everyone here is available to help you out (so orientation and written guides are not necessary) 13 1 Standardized terms among different catalogues/universal system 14 1 Catalogue that the public library used before the GATE was more user-friendly, although it had less room to expand 15 1 Some people can figure out the GATE without instruction or written guides 16 1 What does a call number starting with HV mean? 17 1 Proquest is fantastic, and pretty booked up…another of those, or for different subjects, like education (At the time of the study, Concordia had a subscription to Proquest Social Sciences only, with two workstations. This subscription has since been canceled and replaced with the broader subject coverage of Infotrac’s Expanded Academic ASAP) 18 1 Which computer has what? 24 Total comments were noted based on the search sessions, with a total of 105 problems and comments including repetitions. In contrast, a total of 32 different problems, attractive features, and other com- ments were counted in response to interview questions about the online catalogue, with a total of 66 problems, attractive features, and comments. This suggests that the verbal proto- col method is a richer source of data than inter- view questions. Even with this small sample size, 11 prob- lems or comments were noted in three or more of the ten search sessions, suggesting that this method can be useful in determining some of the more common problems experienced by system users. In contrast, only five problems or attractive features were noted in three or more of the ten sessions in response to interview questions on the online catalogue. There was overlap in the data obtained by both methods. For example, the top two prob- lems or comments noted both during search sessions and in response to interview questions about problems with the online catalogue were the same. These top two comments were a preference for material available onsite and that users found subject searching to be difficult. Data obtained in the verbal protocol search sessions tended to be more technically detailed than answers to interview questions. This is not surprising. It seems unlikely that people would remember the technical details of a complex problem encountered some time ago. Explain- ing what is appearing on the screen right at the moment is relatively easy. Another advantage of this method is that it is possible to determine the effect of a problem on an actual search. Some problems may be minor nuisances while others can be significant obsta- cles that many individuals could not overcome without help. The frequency with which a particular prob- lem occurs may or may not be an indication of its importance. For example, the fact that only one participant in this study attempted to use DRA’s online Help, and did not find any helpful information, may be significant in itself. Some of the information obtained through asking interview questions did not overlap data obtained through the verbal protocol search sessions. This was particularly true of attractive features. This suggests that combining both types of information gathering in this type of study would be more valuable than the verbal protocol method alone. One potential problem with the approach to recruitment used in this study, approaching people already at PAC stations, is that many of the resulting sessions may have been only partial search sessions. As a result, common problems generally encountered at the beginning of a search may have been missed. This may also have distorted the search session times. Another difficulty with recruitment encoun- tered in this study could have been avoided. The study was conducted relatively early in the academic semester, when the OPAC stations were less busy. A study conducted during a busier period may have had a lower rate of no one being available to ask to participate. Results of this study cannot be generalized to the population of Concordia library users as a whole, because of the small and non-random sample. The presence of the researcher during search sessions and the talk-aloud method per se may have had an impact on the searches. For ethical reasons, subjects experiencing difficulties with searches were allowed to ask for assistance. In some cases the researcher provided direction during searches when assistance appeared to be needed, but was not requested. The researcher also asked some probing questions. The pres- ence of the researcher and the researcher’s interactions with participants make this a hybrid research method, part verbal protocol, part interview. The talk aloud method itself may have had some impact on the participants’ search strate- gy. For example, participants may have tried to demonstrate what they considered to be the “correct” method of searching rather than searching in their usual manner. Further research would be needed to deter- mine the frequency with which problems noted during this study occur. Results of this study may be useful in indicating areas which merit further study. Most of the problems appeared to fall into one of three categories: (1) availability and limiting by location; (2) subject searching; and (3) arrow and keystroke. 203 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206 One finding that could be relevant to develop- ment of online catalogues is that it appears to be very important, at least to this group of subjects, for users to identify easily those materials that are available locally. Although half of the partici- pants indicated a preference for materials avail- able in the local collection, none were aware of DRA’s “limit by location” feature. Even those who requested assistance found it difficult to limit by location. Simplifying this task would improve the usefulness of the online catalogue. Difficulties with subject searching have been noted by a number of researchers. Clear online instructions for users when either no matches or long hit lists are retrieved could be helpful. Signage in the reference area clearly pointing to subject help guides, i.e. The Library of Con- gress Subject Headings, might alleviate some problems for some users. Arrowing and keystroke problems might disappear with the next generation of online catalogues based on graphical user interfaces. Recommendations for Concordia Half the participants in this study indicated a preference for materials available onsite, but determining availability and limiting by location were the most common problems identified. This supports the prior belief of Concordia library staff that identifying materials available locally is difficult. One strategy which staff has implemented to minimize this problem is to enhance the browsability of the collection. Signage has been added at the end of ranges listing subject headings for all areas where collection size is significant (more than 50 titles). A collection of pathfinders covers all major subject areas taught at Concordia. Library of Congress major subject headings are posted prominently in the reference area and a list of Dewey subject headings is readily avail- able as well. While this approach may be useful in providing clients with alternative methods of finding materials in the collection, it does not address the main problem, which is that clients want to be able to use the online catalogue to easily identify materials in the local collection. Setting DRA to limit materials available locally would solve the problems clients have in locating materials available locally, but would make it somewhat more difficult for clients to identify materials available through the NEOS Consortium. The importance of having materials available locally may be decreasing as interlibrary loan activity increases. A few years ago, interlibrary loan activity was negligible at Concordia. This was due partly to the lack of delivery mecha- nisms fast enough to meet the needs of an undergraduate population, and partly due to the availability of a number of high quality research and special libraries in the Edmonton area (e.g. the University of Alberta, government depart- ment libraries). Interlibrary loan activity has increased dra- matically in the past few years, primarily as a result of the document delivery agreement within NEOS that guarantees a three-day turn- around time. The merger of circulation and interlibrary loan services and faster delivery service (because of the courier service and the use of Ariel for articles) have both helped to improve the availability of interlibrary loan services during the library’s open hours. This trend is likely to continue to increase as Concordia becomes involved in more resource- sharing agreements, particularly The Alberta Library, which involves public, academic, and regional library systems throughout the province. For example, one of the projects of The Alberta Library is involved in is ensuring that all post-secondary libraries throughout the province have either Relais (large institutions) or Ariel (smaller institutions) workstations for document delivery, thanks to funding provided by Alberta Advanced Education and Career Development. This will make requesting of articles much more convenient and desirable for Concordia’s undergraduates. Library instruc- tion sessions now place more emphasis both on teaching students to identify materials available locally and how to place interlibrary loan requests. It is hoped that DRA’s new TAOS operating system, currently under development, will address the difficulty of identifying materials available locally. From the point of view of the client of the small library in a big consortium, it would be ideal if the main screen would include the option of searching the current location only (i.e. Concordia only), the whole NEOS 204 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206 database, or other options using Z39.50, as determined by Concordia. Permitting individ- ual locations to set up their own search options would be ideal because even within a consor- tium, different institutions or groups of institu- tions, have different licensing arrangements for databases, and different resource-sharing agree- ments with other agencies. Arrow and keystroke problems will eventual- ly be eliminated by moving to a Web-based interface. Currently, DRA Web is available to the NEOS Consortium. However, Infogate is preferred at Concordia, because DRA Web does not allow for limiting by location, and Borrower Information has not been made available due to concerns that it does not work correctly. Emphasis has been placed on Infogate only, because although some kinds of searching would be easier on DRA Web, it is believed that the complications of teaching students two different interfaces for the online catalogue would outweigh the benefits. The DRA Web2 interface is expected to be available to the NEOS Consortium by fall of 1999. Concordia staff will evaluate this interface and determine whether to continue emphasizing Infogate or switch to DRA Web2. Suggestions for further research The verbal protocol method offers great poten- tial as a research method in a number of areas of librarianship. When designing interfaces, whether for intranets, or customized interfaces for various library databases or products, the verbal protocol method can be used to identify problems for users before the interface is made available to clientele. The verbal protocol method also has potential as a tool for analyzing the research process or the reference interview. Wiedenbeck et al.(1989) provide a concise, useful description of how to conduct a verbal protocol study. Conclusion The verbal protocol method is a useful means of obtaining data on the online catalogue. A total of 11 problems or comments were noted in three or more of the ten search sessions, in contrast to five problems or attractive features noted in three or more of the ten sessions in response to interview questions. Participants find it easy to follow instructions to comment on what keys they are pressing, what they see on the screen, and how they react. These comments make it easy to determine what was happening on the screen when tran- scribing sessions. When planning a verbal proto- col study, allow about three times as much time for the average search session to allow for recruitment, transcription, and data analysis. This method has a lot of potential applica- tions in librarianship, including interface design, the research process, and the reference interview. Librarians might find this a natural method to use, as it is somewhat similar to conducting a reference interview. Unlike Henderson et al.’s (1995) study, it was found that interview questions provided addi- tional information not obtained through the verbal protocol search sessions. Answers to interview questions could also help to indicate the relative importance of problems to users, i.e. sort out the difference between a common nuisance and a genuine obstacle to research. The results of this study (remembering the small size of the sample) indicate three main areas of concern for users: identifying what is held locally and what is available, subject searching, and arrowing and keystrokes. References Borgman, C.L. (1986), “Why are online catalogs hard to use? Lessons learned from information retrieval studies”, Journal of the American Society for Information Science, Vol. 37 No. 6, pp. 387-400. Borgman, C.L. (1996), “Why are online catalogs still hard to use?”, Journal of the American Society for Information Science, Vol. 47 No. 7, pp. 493-503. Henderson, R.D., Smith, M.C., Podd, J. and Varela-Alvarez, H. (1995), “A comparison of the tour prominent user- based methods for evaluating the usability of computer software”, Ergonomics, Vol. 38 No.10, pp. 2030-44. Seymour, S. (1991), “Online public access catalog user studies: a review of research methodologies, March 1986-November 1989”, Library and Information Science Research, Vol. 13, April-June, pp. 89-102. Wiedenbeck, S., Lampert, R. and Scholtz, J. (1989), “Using protocol analysis to study the user interface”, Bulletin of the American Society for Information Science, Vol. 15 No. 5, pp. 25-6. 205 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206 Appendix Detailed script including recruitment procedures, instructions, and questions Verbal protocol script Try for some randomization of recruitment, e.g. pre-select times on a random basis to approach people; number computer terminals, approach first person to go to that terminal at that partic- ular time and say something like this: Hi. I’m conducting a research study on the online catalogue. May I take about a minute of your time to briefly explain my project and ask for your participation? My name is Heather Morrison. As you may know, I am the Circulation Services Coordinator here at Concordia. I am also working on a Master’s degree in Library and Information Studies at the University of Alberta. As part of my work towards this degree, I am conducting a study to test the talk aloud method as a way of studying the online catalogue, or GATE. What I am asking people to do is to conduct the searches they came to the library to do, while talking aloud about what they’re doing and how they react to the computer interface. This would be done in the privacy of my office, with myself present as an observer. Participation is purely voluntary; please feel free to say no if you would rather not participate. The search session will be recorded on audiotape and is completely confi- dential. After the search session, there will be a brief interview. No information that would identi- fy you as an individual would be collected. The time required for this study is about the same as your search is likely to take, plus about five extra minutes for talking and questions, including this time. Are you willing to participate in this study? If no, then say: “thanks for your time, and good luck with your search.” If yes, say: I’d like to give you a copy of this brief summary of this research project, including the name and phone number of my instructor, should you have any concerns or questions about the study. Talk aloud study Say: The main focus of my study is evaluating the computer interface of the online catalogue, that is, does it help people to find the information or materials they need, or does it need improvement? So, what I’d like you to do is to conduct your search as you usually would, but talk aloud about what you’re doing, that is, what keys you’re press- ing, what you’re seeing on the screen, what you’re looking for on the screen, that type of thing, as well as how you feel about the information the computer is giving you. Do you have any questions before we start? Any comments to participants, if needed, both at this time and during session, are restricted to: • talking about what they are doing; • commenting on the reasons for what they’re doing; • comments about how they feel about what’s on the computer. After the session, ask: • About any specific questions noted during the session. • What aspects of the online catalogue did you find were the most problematic during the session? • What aspects of the online catalogue did you find were the most attractive during the session? • Based on past experiences with the online catalogue, what aspects have you found problematic or attractive? • Do you have any other comments or ques- tions about the online catalogue? • Given the online catalogue as it is now, what can library staff do to help people find the things they need, in terms of orientation or written guides? • Do you have any other comments or ques- tions about this research project? • Did you find the information you needed? (If not, provide reference assistance or refer subject to the reference desk.) Finally, say: I’d like to thank you very much for participating. If you’d like a copy of the final report of my research, it should be available in early April; you can pick up a copy here at the Circulation Desk. 206 Online catalogue research and the verbal protocol method Heather G. Morrison Library Hi Tech Volume 17 · Number 2 · 1999 · 197–206