UBC Faculty Research and Publications

Online catalogue research and the verbal protocol method Morrison, Heather 1999

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-online_catalogue_morrison.pdf [ 93.56kB ]
52383-license.html [ 7B ]
JSON: 52383-1.0107431.json
JSON-LD: 52383-1.0107431-ld.json
RDF/XML (Pretty): 52383-1.0107431-rdf.xml
RDF/JSON: 52383-1.0107431-rdf.json
Turtle: 52383-1.0107431-turtle.txt
N-Triples: 52383-1.0107431-rdf-ntriples.txt
Original Record: 52383-1.0107431-source.json
Full Text

Full Text

Introduction  Online catalogue research and the verbal protocol method Heather G. Morrison  The author Heather G. Morrison is Circulation Services Coordinator, Arnold Guebert Library, Concordia University College of Alberta, Edmonton, Canada. <hmorrison@concordia.ab.ca> Keywords Catalogues, Field research, Libraries, Library users, Online computing, User studies Abstract The verbal protocol method is used extensively in computer usability studies. This study was designed to test the feasibility of using the verbal protocol method as a means of conducting field research on the online catalogue. Ten undergraduate students conducted their own research on DRA’s Infogate. As they searched, they talked aloud about what keys they were pressing, what was happening on the screen, and their reactions. Transcripts of sessions were analyzed. A total of 65 different problems or comments were noted during search sessions, with 11 problems or comments noted in three or more sessions. This supports the hypothesis that the verbal protocol method is a valuable means of identifying common problems for users. It is suggested that the verbal protocol method offers great potential as a research tool in librarianship, for example in the area of interface design.  Library Hi Tech Volume 17 · Number 2 · 1999 · pp. 197–206 © MCB University Press · ISSN 0737-8831  Usability is an important feature of any successful computer program. The online catalogue is one example of an extremely important interface for library users that ought to be as easy to use as possible. There is an extensive body of online catalogue research literature. Yet Borgman’s research (1996, 1986) reveals that, in spite of all this research, users continue to find online catalogues difficult to use. A number of different approaches have been taken to research the online catalogue. Seymour (1991) reviewed research methodologies used in online public access catalogue user studies from March 1986-November 1989. Surveys were found to be the most frequently used method. Interviews, observation, controlled experiment, and transaction log analysis are reported as well. One “think aloud” (verbal protocol) study is mentioned, but details are not reported. Henderson et al. (1995) conducted a major study to evaluate the efficiency of the four most prominent user-based methods in computer usability studies: logged data, questionnaire, interview, and verbal protocol analysis. They found verbal protocol analysis to be the most efficient method of gathering data. Even combining verbal protocol analysis with other techniques did not result in a statistically significant improvement in the quantity of data obtained. Originally developed as a research tool in the field of cognitive psychology, the verbal protocol method was initially used as a means of studying human problem-solving processes. For example, subjects would solve arithmetic problems while talking aloud. This method is also used in the field of expert system design, to gather data about how experts go about solving problems in their field of expertise. Verbal protocol is now used extensively in computer usability studies. Major advantages of the verbal protocol method are the quality and quantity of data obtained. One group of authors (Wiedenbeck et al., 1989) suggests that protocol analysis is particularly useful in studying areas where little is currently known. The major disadvantage of the verbal protocol method is that it is time-consuming. Another disadvantage is the likelihood that the method of study has some impact on the behavior under observation. Following are the results of a study designed to examine the feasibility of using verbal protocol as a method of field research on the online catalogue. The quality and quantity of data produced by a  197  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  study using ten subjects was analyzed. The potential usefulness of the method in online catalogue research is assessed, both as a development tool and as a tool to assist reference librarians in determining the most common problems experienced by users.  on a rotating basis. Later, the rotating approach was abandoned. The researcher was approaching potential participants after leaving the researcher’s office, and it appeared awkward to walk past certain online catalogue stations in order to approach participants at more distant computers. Recruitment attempts also appeared to be less successful when people at closer computers were skipped. At this point (about halfway through the sessions), the researcher decided to follow the same order of approach on each occasion. Participants conducted their own searches in the privacy of the researcher’s office, with the researcher present, while talking aloud about their searches, what keys they were pressing and how they reacted to information on the screen. Search sessions were recorded on audiotape. Brief interviews followed the search sessions. For a detailed script including recruitment procedures, instructions, and questions, see Appendix. A total of 25 minutes was spent recruiting for eight sessions, or an average of three minutes per session. The number of sessions included in recruitment data is eight rather than ten, because the first two sessions were pilot sessions. A total of 14 people were asked to participate for a total participation rate of 57 percent . One participant approached the researcher at the request of someone who had declined to participate because they had finished their searching for the day. On ten occasions, there was no one available at GATE stations. Time per session averaged approximately 171 minutes, including recruitment and data analysis: • recruitment – 3 minutes; • session – 23 minutes; • transcription – 69 minutes; • data analysis (initial) – 16 minutes; • collating data (est.) – 60 minutes; • total time/session: 171 minutes/session.  Methodology The study was conducted at Concordia University College of Alberta, a small undergraduate liberal arts college located in Edmonton, Alberta. Concordia is a member of the NEOS Library Consortium, a group of about 20 academic, government, and health libraries. The consortium shares one online catalogue, which is maintained at the University of Alberta. The NEOS Libraries’ Catalogue (sometimes referred to as The GATE) contains over three million titles, of which about 60,000 (or about 2 percent ) belong to Concordia. The automation platform for the NEOS Libraries’ Catalogue is DRA Version 2.5, and the DRA interface used at Concordia is Infogate. DRA offers three options for display of the online catalogue: search everything, search everything but highlight materials in the local collection, and limit to materials in the local collection. The option selected by Concordia is search everything with highlighting of materials in the local collection. The reason for this choice was to balance the needs of Concordia clients to find materials immediately available on the shelves with the needs of Concordia clients to access the wider range of resources available through the NEOS Consortium. Until recently Concordia’s collection was classified using the Dewey Decimal Classification System. About two years ago, new acquisitions began to be catalogued using the Library of Congress Classification system. The library also uses the CODOC classification system for government documents. The majority of the collection is classified with Dewey. Users of online catalogue stations were asked if they would participate in a study to test the feasibility of the talk aloud method as a means of researching the online catalogue. An attempt was made to minimize bias in the sample by selecting times to approach potential participants in advance, numbering online catalogue stations, and selecting the order of online catalogue stations at which to approach people  Characteristics of participants: 60 percent of the participants were female, 40 percent male. At least seven of the ten participants were senior students (second year of university or higher); one was a first year student. Data analysis involved reviewing transcripts and grouping together similar problems or comments. Problems or comments noted in transcripts of search sessions and in response to interview questions were tabulated separately for comparison purposes.  198  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  Results Problems and comments Table I shows the problems and comments noted by researcher in decreasing order of frequency.  Interview questions In order to compare the number of comments obtained through the search sessions and in response to interview questions, answers to the first four questions relating to the online catalogue have been collated (see Table II).  These four questions were: (1) During this session, what aspects of the online catalogue did you find were the most problematic? (2) During this session, what aspects of the online catalogue did you find were the most attractive? (3) Based on your past experiences with the online catalogue, what aspects have you found problematic or attractive? (4) Do you have any other comments or questions about the online catalogue?  Table I Problems and comments  Observation  Number of sessions  1 2 3 4 5 6 7 8 9  5 5 5 4 3 3 3 3 3  10 11  3 3  12 13 14 15 16 17 18 19 20 21 22 23 24  2 2 2 2 2 2 2 2 2 2 2 1 1  Comment / problem Prefer materials available onsite Difficulty determining which term to use in a subject search Find menu or subject index coming down over search results Arrowing past where you want to go Difficulty moving from screen with locations to call number and availability information Subject search results in “no matches” message, participant finds no helpful information Subject search results in a long hit list (161 – 980 hits); one participant satisfied with results, others frustrated Hot keys / letters (appreciated once participant knew about them) Call number questions and comments: • what does it mean when there are letters in front of the call numbers? • man, these numbers are long, eh! • “LB” in call number interpreted as “library” Typing problems Periodicals • wonders whether to search for books or periodicals on a topic • is it possible to go into periodicals from here? • I have no idea what [the professor] means by the journals Not sure how to move from title list to locations Difficulty locating items known to be at Concordia Needed help using “limit by location” Difficulty identifying libraries List of subject headings and subheadings (or related terms) confuses participant Uses related term or term that appeared in prior search as a subject search term Losing information/getting lost when trying to back up Spelling problems Infogate request function (thought it was turned on, but it wasn’t) Tried to use mouse in Infogate Interpreting citations Excited at seeing something highlighted (would be available at Concordia) Using highlighting to identify availability at Concordia, even in a long list  (Continued) 199  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  Table I  Observation  Number of sessions  25  1  26 27  1 1  28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51  1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1  52 53  1 1  54 55 56 57 58 59 60 61 62 63 64 65  1 1 1 1 1 1 1 1 1 1 1 1 105  Comment/problem Will inquire at the circulation desk of an off-site library regarding “on order”, “order received” and overdue (date due is past) statuses Loses interest in item on seeing “on order” status “Press down and select locations to give you holdings at other locations” doesn’t work (worked when researcher tried later) Pressing line number doesn’t work to see locations of an “order received” item Difficulty trying to figure out if other locations have an item Wonders if “no holdings” message means no holdings in NEOS, not just Concordia Found a number of titles, but none at Concordia Not sure how to figure out which location has a particular volume of a journal Not sure how to interpret volume/issue information Mentioned not having materials available onsite as a shortcoming Missed periodical holdings Using subject search when subject keyword would work Assumed a limit of two words in a subject keyword search because three didn’t work Subject search results in a lot of irrelevant information Not sure whether to use keyword or subject search Using expert keyword strategy in a keyword search Expert keyword search results in no hits although terms used are in the database Finding a call number to browse around on the shelves Quick, successful subject search (subject keyword) Backing up from location to main subject heading didn’t work (did work later when researcher tried this) Does page down work? “Arrowing” is very good for one-finger typists Not sure of printing procedures Printing locations is cool Press a key, that’s good, it’s highlighted, it makes you look there first Disappointing to find books that may be relevant but in a language you cannot understand Found name of an author in a previous search but did not get the call number; came back and searched by author, but did not find the item; searched by other means and found the item; why did it not come up by author? Question about putting a book belonging to another library on hold Confused when item is not found at all, appears to assume anything in a particular index should be available through the NEOS consortium Uses author search because title is so long (although title is more unique than author) May have abandoned difficult title search due to the influence of the research project Author search, last name only (common name) leads to lots of paging down to find individual author Using publisher information (and keywords, e.g. Congressional in title) to determine which is Canadian information Frustrated at amount of American information found when looking only for Canadian information Two author records found for a single author Connecting to the GATE is so slow NEOS online catalogue icon: would never have guessed that this is what it would be called Help: only one participant attempted to use DRA’s online help (to figure out how to limit by location), with no success Options: one participant tried options, which was irrelevant to the search I’ve used microfiche before, but I’m not too sure how to use them Usually goes to the public library, but prof won’t allow for this assignment Total problems and comments 200  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  Table II Interview questions  Observation  Number of sessions  In decreasing order of frequency  1  6  2 3  5 2  4 5 6  2 2 2  7 8 9 10 11 12 13 14 15  2 2 1 1 1 1 1 1 1  Problems Subject terminology – not knowing what terms to use and/or lack of computer assistance, e.g.: • no matches on subject search and no assistance given • screen with list of subject headings and subheadings is confusing Material not available locally Instructions at bottom of screen are: • hard to read • easy to ignore Bringing menu down over results Overshooting with arrow keys Interpreting citations: • cannot tell if it is an anthology or journal, etc. • does the word “references” in a citation mean material not available for loan? Whether to use subject or keyword Moving around (up/down, between menus) Cluttered screen Request function (on OPAC) is not available Limiting to Canadian information takes time to learn Retrieving items in different languages Not finding an item you have found before when you have author information Trying to figure out which icon to click on to start a search Limiting by location could be easier  7 5 3 2 2 2 2 2 1 1 1 1 1 1 1  Attractive features Location and/or availability information Fast I like the OPAC/it’s really good Selection Limiting Keywords Printing: like being able to print results Improvement over older OPAC or card catalogue Highlighting Are you sure you really want to quit option Backup and forward The fact that I could access the book from so many directions Arrow keys Subject keyword The actual information you get is pretty good  16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32  3 1 66  Other comments Shortcuts: would like help identifying shortcuts Question about how to find a CODOC call number Total problems, attractive features, and other comments about the online catalogue  Post-search questions Question: Do you have any other comments or questions about this research project? Four participants had no comments. In three sessions, the  question was not asked. Two participants had positive comments (pretty good, very interesting). One suggested that the researcher mention at the beginning that you’d like the person to kind of talk out their actions, and explain what’s 201  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  on the screen. One participant said that you could manage without a practice session. Question: Did you find the information you needed? Five participants answered yes; one participant had found some of the information needed. The question was not asked in four sessions. One participant was definitely having difficulty finding information and was referred to the reference desk. One participant found a few citations, not available at Concordia, but remarked after the session that they knew we had five books on the subject, and where to find them on the shelves. Question: Based on the online catalogue the way it is now, is there anything that the library staff could do to help people find what they need in terms of orientation or written guides or…  Comments on orientation or written guides Comments on orientation or written guides in decreasing order of frequency are shown in Table III.  Discussion Participants seemed to find the verbal protocol instructions easy to follow. In the two pilot sessions, participants were given a practice search, but this was abandoned because participants did not find it necessary. Initially, the plan was to obtain logged data during the sessions as well, in order to compare comments with what was happening on the screen at the time. This was abandoned after the first pilot session. The log from one search session (QVT Terminal) was over 700 pages long. Timing data appeared only at the beginning and end of the session, which would have made it very difficult to match the log and audiotape. Also, the data recorded on audiotape appeared to provide sufficient content to determine what was happening on the screen at any given time. A greater quantity of data was obtained from the search sessions using the verbal protocol method than from the interview questions. A total of 65 different problems and comments  Table III Comments on orientation or written guides  Obser Number of -vation sessions 1 2 3 4  3 2 2 2  5 6 7 8 9 10 11 12 13 14 15 16 17  2 1 1 1 1 1 1 1 1 1 1 1 1  18  1 24  Comment Forgetting information is a problem (orientation) Never taken one (orientation) List of subject headings, keywords (written guide) Quick information beside computer: • short note about a number of the convenient options on the online catalogue placed by the computer • step-by-step information, with examples, beside the computers Important to have reference service available Orientations are the best here Orientations are good for the first-year students Orientations are too long Orientations are theoretical / observational rather than hands-on Orientation was disconnected with anything students were doing at the time How to limit by location (written guide) Everyone here is available to help you out (so orientation and written guides are not necessary) Standardized terms among different catalogues/universal system Catalogue that the public library used before the GATE was more user-friendly, although it had less room to expand Some people can figure out the GATE without instruction or written guides What does a call number starting with HV mean? Proquest is fantastic, and pretty booked up…another of those, or for different subjects, like education (At the time of the study, Concordia had a subscription to Proquest Social Sciences only, with two workstations. This subscription has since been canceled and replaced with the broader subject coverage of Infotrac’s Expanded Academic ASAP) Which computer has what? Total comments 202  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  were noted based on the search sessions, with a total of 105 problems and comments including repetitions. In contrast, a total of 32 different problems, attractive features, and other comments were counted in response to interview questions about the online catalogue, with a total of 66 problems, attractive features, and comments. This suggests that the verbal protocol method is a richer source of data than interview questions. Even with this small sample size, 11 problems or comments were noted in three or more of the ten search sessions, suggesting that this method can be useful in determining some of the more common problems experienced by system users. In contrast, only five problems or attractive features were noted in three or more of the ten sessions in response to interview questions on the online catalogue. There was overlap in the data obtained by both methods. For example, the top two problems or comments noted both during search sessions and in response to interview questions about problems with the online catalogue were the same. These top two comments were a preference for material available onsite and that users found subject searching to be difficult. Data obtained in the verbal protocol search sessions tended to be more technically detailed than answers to interview questions. This is not surprising. It seems unlikely that people would remember the technical details of a complex problem encountered some time ago. Explaining what is appearing on the screen right at the moment is relatively easy. Another advantage of this method is that it is possible to determine the effect of a problem on an actual search. Some problems may be minor nuisances while others can be significant obstacles that many individuals could not overcome without help. The frequency with which a particular problem occurs may or may not be an indication of its importance. For example, the fact that only one participant in this study attempted to use DRA’s online Help, and did not find any helpful information, may be significant in itself. Some of the information obtained through asking interview questions did not overlap data obtained through the verbal protocol search sessions. This was particularly true of attractive features. This suggests that combining both  types of information gathering in this type of study would be more valuable than the verbal protocol method alone. One potential problem with the approach to recruitment used in this study, approaching people already at PAC stations, is that many of the resulting sessions may have been only partial search sessions. As a result, common problems generally encountered at the beginning of a search may have been missed. This may also have distorted the search session times. Another difficulty with recruitment encountered in this study could have been avoided. The study was conducted relatively early in the academic semester, when the OPAC stations were less busy. A study conducted during a busier period may have had a lower rate of no one being available to ask to participate. Results of this study cannot be generalized to the population of Concordia library users as a whole, because of the small and non-random sample. The presence of the researcher during search sessions and the talk-aloud method per se may have had an impact on the searches. For ethical reasons, subjects experiencing difficulties with searches were allowed to ask for assistance. In some cases the researcher provided direction during searches when assistance appeared to be needed, but was not requested. The researcher also asked some probing questions. The presence of the researcher and the researcher’s interactions with participants make this a hybrid research method, part verbal protocol, part interview. The talk aloud method itself may have had some impact on the participants’ search strategy. For example, participants may have tried to demonstrate what they considered to be the “correct” method of searching rather than searching in their usual manner. Further research would be needed to determine the frequency with which problems noted during this study occur. Results of this study may be useful in indicating areas which merit further study. Most of the problems appeared to fall into one of three categories: (1) availability and limiting by location; (2) subject searching; and (3) arrow and keystroke.  203  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  One finding that could be relevant to development of online catalogues is that it appears to be very important, at least to this group of subjects, for users to identify easily those materials that are available locally. Although half of the participants indicated a preference for materials available in the local collection, none were aware of DRA’s “limit by location” feature. Even those who requested assistance found it difficult to limit by location. Simplifying this task would improve the usefulness of the online catalogue. Difficulties with subject searching have been noted by a number of researchers. Clear online instructions for users when either no matches or long hit lists are retrieved could be helpful. Signage in the reference area clearly pointing to subject help guides, i.e. The Library of Congress Subject Headings, might alleviate some problems for some users. Arrowing and keystroke problems might disappear with the next generation of online catalogues based on graphical user interfaces.  make it somewhat more difficult for clients to identify materials available through the NEOS Consortium. The importance of having materials available locally may be decreasing as interlibrary loan activity increases. A few years ago, interlibrary loan activity was negligible at Concordia. This was due partly to the lack of delivery mechanisms fast enough to meet the needs of an undergraduate population, and partly due to the availability of a number of high quality research and special libraries in the Edmonton area (e.g. the University of Alberta, government department libraries). Interlibrary loan activity has increased dramatically in the past few years, primarily as a result of the document delivery agreement within NEOS that guarantees a three-day turnaround time. The merger of circulation and interlibrary loan services and faster delivery service (because of the courier service and the use of Ariel for articles) have both helped to improve the availability of interlibrary loan services during the library’s open hours. This trend is likely to continue to increase as Concordia becomes involved in more resourcesharing agreements, particularly The Alberta Library, which involves public, academic, and regional library systems throughout the province. For example, one of the projects of The Alberta Library is involved in is ensuring that all post-secondary libraries throughout the province have either Relais (large institutions) or Ariel (smaller institutions) workstations for document delivery, thanks to funding provided by Alberta Advanced Education and Career Development. This will make requesting of articles much more convenient and desirable for Concordia’s undergraduates. Library instruction sessions now place more emphasis both on teaching students to identify materials available locally and how to place interlibrary loan requests. It is hoped that DRA’s new TAOS operating system, currently under development, will address the difficulty of identifying materials available locally. From the point of view of the client of the small library in a big consortium, it would be ideal if the main screen would include the option of searching the current location only (i.e. Concordia only), the whole NEOS  Recommendations for Concordia Half the participants in this study indicated a preference for materials available onsite, but determining availability and limiting by location were the most common problems identified. This supports the prior belief of Concordia library staff that identifying materials available locally is difficult. One strategy which staff has implemented to minimize this problem is to enhance the browsability of the collection. Signage has been added at the end of ranges listing subject headings for all areas where collection size is significant (more than 50 titles). A collection of pathfinders covers all major subject areas taught at Concordia. Library of Congress major subject headings are posted prominently in the reference area and a list of Dewey subject headings is readily available as well. While this approach may be useful in providing clients with alternative methods of finding materials in the collection, it does not address the main problem, which is that clients want to be able to use the online catalogue to easily identify materials in the local collection. Setting DRA to limit materials available locally would solve the problems clients have in locating materials available locally, but would  204  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  database, or other options using Z39.50, as determined by Concordia. Permitting individual locations to set up their own search options would be ideal because even within a consortium, different institutions or groups of institutions, have different licensing arrangements for databases, and different resource-sharing agreements with other agencies. Arrow and keystroke problems will eventually be eliminated by moving to a Web-based interface. Currently, DRA Web is available to the NEOS Consortium. However, Infogate is preferred at Concordia, because DRA Web does not allow for limiting by location, and Borrower Information has not been made available due to concerns that it does not work correctly. Emphasis has been placed on Infogate only, because although some kinds of searching would be easier on DRA Web, it is believed that the complications of teaching students two different interfaces for the online catalogue would outweigh the benefits. The DRA Web2 interface is expected to be available to the NEOS Consortium by fall of 1999. Concordia staff will evaluate this interface and determine whether to continue emphasizing Infogate or switch to DRA Web2.  noted in three or more of the ten sessions in response to interview questions. Participants find it easy to follow instructions to comment on what keys they are pressing, what they see on the screen, and how they react. These comments make it easy to determine what was happening on the screen when transcribing sessions. When planning a verbal protocol study, allow about three times as much time for the average search session to allow for recruitment, transcription, and data analysis. This method has a lot of potential applications in librarianship, including interface design, the research process, and the reference interview. Librarians might find this a natural method to use, as it is somewhat similar to conducting a reference interview. Unlike Henderson et al.’s (1995) study, it was found that interview questions provided additional information not obtained through the verbal protocol search sessions. Answers to interview questions could also help to indicate the relative importance of problems to users, i.e. sort out the difference between a common nuisance and a genuine obstacle to research. The results of this study (remembering the small size of the sample) indicate three main areas of concern for users: identifying what is held locally and what is available, subject searching, and arrowing and keystrokes.  Suggestions for further research The verbal protocol method offers great potential as a research method in a number of areas of librarianship. When designing interfaces, whether for intranets, or customized interfaces for various library databases or products, the verbal protocol method can be used to identify problems for users before the interface is made available to clientele. The verbal protocol method also has potential as a tool for analyzing the research process or the reference interview. Wiedenbeck et al.(1989) provide a concise, useful description of how to conduct a verbal protocol study.  Conclusion The verbal protocol method is a useful means of obtaining data on the online catalogue. A total of 11 problems or comments were noted in three or more of the ten search sessions, in contrast to five problems or attractive features  References Borgman, C.L. (1986), “Why are online catalogs hard to use? Lessons learned from information retrieval studies”, Journal of the American Society for Information Science, Vol. 37 No. 6, pp. 387-400. Borgman, C.L. (1996), “Why are online catalogs still hard to use?”, Journal of the American Society for Information Science, Vol. 47 No. 7, pp. 493-503. Henderson, R.D., Smith, M.C., Podd, J. and Varela-Alvarez, H. (1995), “A comparison of the tour prominent userbased methods for evaluating the usability of computer software”, Ergonomics, Vol. 38 No.10, pp. 2030-44. Seymour, S. (1991), “Online public access catalog user studies: a review of research methodologies, March 1986-November 1989”, Library and Information Science Research, Vol. 13, April-June, pp. 89-102. Wiedenbeck, S., Lampert, R. and Scholtz, J. (1989), “Using protocol analysis to study the user interface”, Bulletin of the American Society for Information Science, Vol. 15 No. 5, pp. 25-6.  205  Online catalogue research and the verbal protocol method  Library Hi Tech  Heather G. Morrison  Volume 17 · Number 2 · 1999 · 197–206  Appendix  does it help people to find the information or materials they need, or does it need improvement? So, what I’d like you to do is to conduct your search as you usually would, but talk aloud about what you’re doing, that is, what keys you’re pressing, what you’re seeing on the screen, what you’re looking for on the screen, that type of thing, as well as how you feel about the information the computer is giving you. Do you have any questions before we start?  Detailed script including recruitment procedures, instructions, and questions Verbal protocol script Try for some randomization of recruitment, e.g. pre-select times on a random basis to approach people; number computer terminals, approach first person to go to that terminal at that particular time and say something like this: Hi. I’m conducting a research study on the online catalogue. May I take about a minute of your time to briefly explain my project and ask for your participation?  Any comments to participants, if needed, both at this time and during session, are restricted to: • talking about what they are doing; • commenting on the reasons for what they’re doing; • comments about how they feel about what’s on the computer.  My name is Heather Morrison. As you may know, I am the Circulation Services Coordinator here at Concordia. I am also working on a Master’s degree in Library and Information Studies at the University of Alberta. As part of my work towards this degree, I am conducting a study to test the talk aloud method as a way of studying the online catalogue, or GATE. What I am asking people to do is to conduct the searches they came to the library to do, while talking aloud about what they’re doing and how they react to the computer interface. This would be done in the privacy of my office, with myself present as an observer. Participation is purely voluntary; please feel free to say no if you would rather not participate. The search session will be recorded on audiotape and is completely confidential. After the search session, there will be a brief interview. No information that would identify you as an individual would be collected. The time required for this study is about the same as your search is likely to take, plus about five extra minutes for talking and questions, including this time. Are you willing to participate in this study?  If no, then say: “thanks for your time, and good luck with your search.” If yes, say: I’d like to give you a copy of this brief summary of this research project, including the name and phone number of my instructor, should you have any concerns or questions about the study.  After the session, ask: • About any specific questions noted during the session. • What aspects of the online catalogue did you find were the most problematic during the session? • What aspects of the online catalogue did you find were the most attractive during the session? • Based on past experiences with the online catalogue, what aspects have you found problematic or attractive? • Do you have any other comments or questions about the online catalogue? • Given the online catalogue as it is now, what can library staff do to help people find the things they need, in terms of orientation or written guides? • Do you have any other comments or questions about this research project? • Did you find the information you needed? (If not, provide reference assistance or refer subject to the reference desk.) Finally, say: I’d like to thank you very much for participating.  Talk aloud study Say:  If you’d like a copy of the final report of my  The main focus of my study is evaluating the computer interface of the online catalogue, that is,  206  research, it should be available in early April; you can pick up a copy here at the Circulation Desk.  


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items