UBC Faculty Research and Publications

Online catalogue research and the verbal protocol method Morrison, Heather 1999

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata


online_catalogue_morrison.pdf [ 93.56kB ]
license.html [ 7B ]
JSON: 1.0107431.json
JSON-LD: 1.0107431+ld.json
RDF/XML (Pretty): 1.0107431.xml
RDF/JSON: 1.0107431+rdf.json
Turtle: 1.0107431+rdf-turtle.txt
N-Triples: 1.0107431+rdf-ntriples.txt
Original Record: 1.0107431 +original-record.json
Full Text

Full Text

197IntroductionUsability is an important feature of any successfulcomputer program. The online catalogue is oneexample of an extremely important interface forlibrary users that ought to be as easy to use aspossible. There is an extensive body of onlinecatalogue research literature. Yet Borgman?sresearch (1996, 1986) reveals that, in spite of allthis research, users continue to find online cata-logues difficult to use.A number of different approaches have beentaken to research the online catalogue. Seymour(1991) reviewed research methodologies used inonline public access catalogue user studies fromMarch 1986-November 1989. Surveys werefound to be the most frequently used method.Interviews, observation, controlled experiment,and transaction log analysis are reported as well.One ?think aloud? (verbal protocol) study ismentioned, but details are not reported.Henderson et al. (1995) conducted a majorstudy to evaluate the efficiency of the four mostprominent user-based methods in computerusability studies: logged data, questionnaire,interview, and verbal protocol analysis. Theyfound verbal protocol analysis to be the mostefficient method of gathering data. Even combin-ing verbal protocol analysis with other techniquesdid not result in a statistically significant improve-ment in the quantity of data obtained. Originally developed as a research tool in thefield of cognitive psychology, the verbal protocolmethod was initially used as a means of studyinghuman problem-solving processes. For example,subjects would solve arithmetic problems whiletalking aloud. This method is also used in thefield of expert system design, to gather data abouthow experts go about solving problems in theirfield of expertise. Verbal protocol is now usedextensively in computer usability studies.Major advantages of the verbal protocolmethod are the quality and quantity of dataobtained. One group of authors (Wiedenbeck etal., 1989) suggests that protocol analysis is par-ticularly useful in studying areas where little iscurrently known. The major disadvantage of theverbal protocol method is that it is time-consum-ing. Another disadvantage is the likelihood thatthe method of study has some impact on thebehavior under observation.Following are the results of a study designed toexamine the feasibility of using verbal protocol as amethod of field research on the online catalogue.The quality and quantity of data produced by aOnline catalogueresearch and the verbalprotocol methodHeather G. MorrisonThe authorHeather G. Morrison is Circulation Services Coordinator,Arnold Guebert Library, Concordia University College ofAlberta, Edmonton, Canada. <hmorrison@concordia.ab.ca>KeywordsCatalogues, Field research, Libraries, Library users, Onlinecomputing, User studiesAbstractThe verbal protocol method is used extensively in computerusability studies. This study was designed to test the feasibili-ty of using the verbal protocol method as a means of con-ducting field research on the online catalogue. Ten under-graduate students conducted their own research on DRA?sInfogate. As they searched, they talked aloud about whatkeys they were pressing, what was happening on the screen,and their reactions. Transcripts of sessions were analyzed. Atotal of 65 different problems or comments were notedduring search sessions, with 11 problems or comments notedin three or more sessions. This supports the hypothesis thatthe verbal protocol method is a valuable means of identifyingcommon problems for users. It is suggested that the verbalprotocol method offers great potential as a research tool inlibrarianship, for example in the area of interface design.Library Hi TechVolume 17 ? Number 2 ? 1999 ? pp. 197?206? MCB University Press ? ISSN 0737-8831study using ten subjects was analyzed. The poten-tial usefulness of the method in online catalogueresearch is assessed, both as a development tooland as a tool to assist reference librarians in deter-mining the most common problems experiencedby users.MethodologyThe study was conducted at Concordia Universi-ty College of Alberta, a small undergraduateliberal arts college located in Edmonton, Alberta.Concordia is a member of the NEOS LibraryConsortium, a group of about 20 academic,government, and health libraries. The consor-tium shares one online catalogue, which is main-tained at the University of Alberta. The NEOSLibraries? Catalogue (sometimes referred to asThe GATE) contains over three million titles, ofwhich about 60,000 (or about 2 percent ) belongto Concordia.The automation platform for the NEOSLibraries? Catalogue is DRA Version 2.5, and theDRA interface used at Concordia is Infogate.DRA offers three options for display of the onlinecatalogue: search everything, search everythingbut highlight materials in the local collection,and limit to materials in the local collection. Theoption selected by Concordia is search every-thing with highlighting of materials in the localcollection. The reason for this choice was tobalance the needs of Concordia clients to findmaterials immediately available on the shelveswith the needs of Concordia clients to access thewider range of resources available through theNEOS Consortium. Until recently Concordia?s collection wasclassified using the Dewey Decimal Classifica-tion System. About two years ago, new acquisi-tions began to be catalogued using the Library ofCongress Classification system. The library alsouses the CODOC classification system for gov-ernment documents. The majority of the collec-tion is classified with Dewey. Users of online catalogue stations were askedif they would participate in a study to test thefeasibility of the talk aloud method as a means ofresearching the online catalogue. An attempt was made to minimize bias in thesample by selecting times to approach potentialparticipants in advance, numbering online cata-logue stations, and selecting the order of onlinecatalogue stations at which to approach peopleon a rotating basis. Later, the rotating approachwas abandoned. The researcher was approachingpotential participants after leaving theresearcher?s office, and it appeared awkward towalk past certain online catalogue stations inorder to approach participants at more distantcomputers. Recruitment attempts also appearedto be less successful when people at closer com-puters were skipped. At this point (about halfwaythrough the sessions), the researcher decided tofollow the same order of approach on each occa-sion. Participants conducted their own searches inthe privacy of the researcher?s office, with theresearcher present, while talking aloud abouttheir searches, what keys they were pressing andhow they reacted to information on the screen.Search sessions were recorded on audiotape.Brief interviews followed the search sessions. Fora detailed script including recruitment proce-dures, instructions, and questions, see Appendix.A total of 25 minutes was spent recruiting foreight sessions, or an average of three minutes persession. The number of sessions included inrecruitment data is eight rather than ten, becausethe first two sessions were pilot sessions. A totalof 14 people were asked to participate for a totalparticipation rate of 57 percent . One participantapproached the researcher at the request ofsomeone who had declined to participatebecause they had finished their searching for theday. On ten occasions, there was no one availableat GATE stations.Time per session averaged approximately 171minutes, including recruitment and data analy-sis:? recruitment ? 3 minutes;? session ? 23 minutes;? transcription ? 69 minutes;? data analysis (initial) ? 16 minutes;? collating data (est.) ? 60 minutes;? total time/session: 171 minutes/session.Characteristics of participants: 60 percent of theparticipants were female, 40 percent male. Atleast seven of the ten participants were seniorstudents (second year of university or higher);one was a first year student.Data analysis involved reviewing transcriptsand grouping together similar problems or com-ments. Problems or comments noted in tran-scripts of search sessions and in response tointerview questions were tabulated separately forcomparison purposes.198Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206ResultsProblems and commentsTable I shows the problems and comments notedby researcher in decreasing order of frequency.Interview questionsIn order to compare the number of commentsobtained through the search sessions and inresponse to interview questions, answers to thefirst four questions relating to the online catalogue have been collated (see Table II).These four questions were:(1) During this session, what aspects of theonline catalogue did you find were the mostproblematic?(2) During this session, what aspects of theonline catalogue did you find were the mostattractive?(3) Based on your past experiences with theonline catalogue, what aspects have youfound problematic or attractive? (4) Do you have any other comments or ques-tions about the online catalogue?199Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206Table I Problems and comments Obser- Number ofvation sessions Comment / problem1 5 Prefer materials available onsite2 5 Difficulty determining which term to use in a subject search3 5 Find menu or subject index coming down over search results4 4 Arrowing past where you want to go5 3 Difficulty moving from screen with locations to call number and availability information6 3 Subject search results in ?no matches? message, participant finds no helpful information7 3 Subject search results in a long hit list (161 ? 980 hits); one participant satisfied with results, others frustrated8 3 Hot keys / letters (appreciated once participant knew about them)9 3 Call number questions and comments: ? what does it mean when there are letters in front of the call numbers?? man, these numbers are long, eh!? ?LB? in call number interpreted as ?library?10 3 Typing problems11 3 Periodicals? wonders whether to search for books or periodicals on a topic? is it possible to go into periodicals from here?? I have no idea what [the professor] means by the journals12 2 Not sure how to move from title list to locations13 2 Difficulty locating items known to be at Concordia14 2 Needed help using ?limit by location?15 2 Difficulty identifying libraries 16 2 List of subject headings and subheadings (or related terms) confuses participant17 2 Uses related term or term that appeared in prior search as a subject search term18 2 Losing information/getting lost when trying to back up19 2 Spelling problems20 2 Infogate request function (thought it was turned on, but it wasn?t)21 2 Tried to use mouse in Infogate22 2 Interpreting citations23 1 Excited at seeing something highlighted (would be available at Concordia)24 1 Using highlighting to identify availability at Concordia, even in a long list(Continued)  200Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206Table I Obser- Number ofvation sessions Comment/problem25 1 Will inquire at the circulation desk of an off-site library regarding ?on order?, ?order received? and overdue (date due is past) statuses26 1 Loses interest in item on seeing ?on order? status27 1 ?Press down and select locations to give you holdings at other locations? doesn?t work (worked when researcher tried later)28 1 Pressing line number doesn?t work to see locations of an ?order received? item29 1 Difficulty trying to figure out if other locations have an item 30 1 Wonders if ?no holdings? message means no holdings in NEOS, not just Concordia31 1 Found a number of titles, but none at Concordia32 1 Not sure how to figure out which location has a particular volume of a journal33 1 Not sure how to interpret volume/issue information34 1 Mentioned not having materials available onsite as a shortcoming35 1 Missed periodical holdings36 1 Using subject search when subject keyword would work37 1 Assumed a limit of two words in a subject keyword search because three didn?t work38 1 Subject search results in a lot of irrelevant information39 1 Not sure whether to use keyword or subject search40 1 Using expert keyword strategy in a keyword search41 1 Expert keyword search results in no hits although terms used are in the database42 1 Finding a call number to browse around on the shelves43 1 Quick, successful subject search (subject keyword)44 1 Backing up from location to main subject heading didn?t work (did work later when researcher tried this)45 1 Does page down work?46 1 ?Arrowing? is very good for one-finger typists47 1 Not sure of printing procedures48 1 Printing locations is cool49 1 Press a key, that?s good, it?s highlighted, it makes you look there first50 1 Disappointing to find books that may be relevant but in a language you cannot understand51 1 Found name of an author in a previous search but did not get the call number; came back and searched by author, but did not find the item; searched by other means and found the item; why did it not come up by author?52 1 Question about putting a book belonging to another library on hold53 1 Confused when item is not found at all, appears to assume anything in a particular index should be available through theNEOS consortium54 1 Uses author search because title is so long (although title is more unique than author)55 1 May have abandoned difficult title search due to the influence of the research project56 1 Author search, last name only (common name) leads to lots of paging down to find individual author57 1 Using publisher information (and keywords, e.g. Congressional in title) to determine which is Canadian information58 1 Frustrated at amount of American information found when looking only for Canadian information59 1 Two author records found for a single author60 1 Connecting to the GATE is so slow61 1 NEOS online catalogue icon: would never have guessed that this is what it would be called62 1 Help: only one participant attempted to use DRA?s online help (to figure out how to limit by location), with no success63 1 Options: one participant tried options, which was irrelevant to the search64 1 I?ve used microfiche before, but I?m not too sure how to use them65 1 Usually goes to the public library, but prof won?t allow for this assignment105 Total problems and commentsPost-search questionsQuestion: Do you have any other comments orquestions about this research project? Four partici-pants had no comments. In three sessions, thequestion was not asked. Two participants hadpositive comments (pretty good, very interest-ing). One suggested that the researcher mentionat the beginning that you?d like the person tokind of talk out their actions, and explain what?s201Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206Table II Interview  questionsObser- Number ofvation sessions In decreasing order of frequencyProblems 1 6 Subject terminology ? not knowing what terms to use and/or lack of computer assistance, e.g.: ? no matches on subject search and no assistance given? screen with list of subject headings and subheadings is confusing2 5 Material not available locally3 2 Instructions at bottom of screen are:? hard to read? easy to ignore4 2 Bringing menu down over results5 2 Overshooting with arrow keys6 2 Interpreting citations:? cannot tell if it is an anthology or journal, etc.? does the word ?references? in a citation mean material not available for loan?7 2 Whether to use subject or keyword8 2 Moving around (up/down, between menus)9 1 Cluttered screen10 1 Request function (on OPAC) is not available11 1 Limiting to Canadian information takes time to learn12 1 Retrieving items in different languages13 1 Not finding an item you have found before when you have author information14 1 Trying to figure out which icon to click on to start a search15 1 Limiting by location could be easierAttractive features 16 7 Location and/or availability information17 5 Fast18 3 I like the OPAC/it?s really good19 2 Selection20 2 Limiting21 2 Keywords22 2 Printing: like being able to print results23 2 Improvement over older OPAC or card catalogue24 1 Highlighting25 1 Are you sure you really want to quit option26 1 Backup and forward27 1 The fact that I could access the book from so many directions28 1 Arrow keys29 1 Subject keyword30 1 The actual information you get is pretty goodOther comments31 3 Shortcuts: would like help identifying shortcuts32 1 Question about how to find a CODOC call number66 Total problems, attractive features, and other comments about the online catalogueon the screen. One participant said that youcould manage without a practice session.Question: Did you find the information youneeded? Five participants answered yes; oneparticipant had found some of the informationneeded. The question was not asked in foursessions. One participant was definitely havingdifficulty finding information and was referredto the reference desk. One participant found afew citations, not available at Concordia, butremarked after the session that they knew wehad five books on the subject, and where to findthem on the shelves.Question: Based on the online catalogue the wayit is now, is there anything that the library staffcould do to help people find what they need in termsof orientation or written guides or?Comments on orientation or writtenguidesComments on orientation or written guides indecreasing order of frequency are shown inTable III.DiscussionParticipants seemed to find the verbal protocolinstructions easy to follow. In the two pilotsessions, participants were given a practicesearch, but this was abandoned because partici-pants did not find it necessary. Initially, the plan was to obtain logged dataduring the sessions as well, in order to comparecomments with what was happening on thescreen at the time. This was abandoned after thefirst pilot session. The log from one searchsession (QVT Terminal) was over 700 pageslong. Timing data appeared only at the begin-ning and end of the session, which would havemade it very difficult to match the log andaudiotape. Also, the data recorded on audiotapeappeared to provide sufficient content to deter-mine what was happening on the screen at anygiven time. A greater quantity of data was obtained fromthe search sessions using the verbal protocolmethod than from the interview questions. Atotal of 65 different problems and comments202Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206Table III Comments on orientation or written guidesObser Number of-vation sessions Comment1 3 Forgetting information is a problem (orientation)2 2 Never taken one (orientation)3 2 List of subject headings, keywords (written guide)4 2 Quick information beside computer:? short note about a number of the convenient options on the online catalogue placed by the computer? step-by-step information, with examples, beside the computers5 2 Important to have reference service available6 1 Orientations are the best here7 1 Orientations are good for the first-year students8 1 Orientations are too long9 1 Orientations are theoretical / observational rather than hands-on10 1 Orientation was disconnected with anything students were doing at the time11 1 How to limit by location (written guide)12 1 Everyone here is available to help you out (so orientation and written guides are not necessary)13 1 Standardized terms among different catalogues/universal system14 1 Catalogue that the public library used before the GATE was more user-friendly, although it had less room to expand15 1 Some people can figure out the GATE without instruction or written guides16 1 What does a call number starting with HV mean?17 1 Proquest is fantastic, and pretty booked up?another of those, or for different subjects, like education (At the time of thestudy, Concordia had a subscription to Proquest Social Sciences only, with two workstations. This subscription hassince been canceled and replaced with the broader subject coverage of Infotrac?s Expanded Academic ASAP)18 1 Which computer has what?24 Total commentswere noted based on the search sessions, with atotal of 105 problems and comments includingrepetitions. In contrast, a total of 32 differentproblems, attractive features, and other com-ments were counted in response to interviewquestions about the online catalogue, with atotal of 66 problems, attractive features, andcomments. This suggests that the verbal proto-col method is a richer source of data than inter-view questions.Even with this small sample size, 11 prob-lems or comments were noted in three or moreof the ten search sessions, suggesting that thismethod can be useful in determining some ofthe more common problems experienced bysystem users. In contrast, only five problems orattractive features were noted in three or moreof the ten sessions in response to interviewquestions on the online catalogue.There was overlap in the data obtained byboth methods. For example, the top two prob-lems or comments noted both during searchsessions and in response to interview questionsabout problems with the online catalogue werethe same. These top two comments were apreference for material available onsite and thatusers found subject searching to be difficult.Data obtained in the verbal protocol searchsessions tended to be more technically detailedthan answers to interview questions. This is notsurprising. It seems unlikely that people wouldremember the technical details of a complexproblem encountered some time ago. Explain-ing what is appearing on the screen right at themoment is relatively easy. Another advantage of this method is that it ispossible to determine the effect of a problem onan actual search. Some problems may be minornuisances while others can be significant obsta-cles that many individuals could not overcomewithout help.The frequency with which a particular prob-lem occurs may or may not be an indication ofits importance. For example, the fact that onlyone participant in this study attempted to useDRA?s online Help, and did not find any helpfulinformation, may be significant in itself.Some of the information obtained throughasking interview questions did not overlap dataobtained through the verbal protocol searchsessions. This was particularly true of attractivefeatures. This suggests that combining bothtypes of information gathering in this type ofstudy would be more valuable than the verbalprotocol method alone.One potential problem with the approach torecruitment used in this study, approachingpeople already at PAC stations, is that many ofthe resulting sessions may have been only partialsearch sessions. As a result, common problemsgenerally encountered at the beginning of asearch may have been missed. This may alsohave distorted the search session times. Another difficulty with recruitment encoun-tered in this study could have been avoided. Thestudy was conducted relatively early in theacademic semester, when the OPAC stationswere less busy. A study conducted during abusier period may have had a lower rate of noone being available to ask to participate.Results of this study cannot be generalized tothe population of Concordia library users as awhole, because of the small and non-randomsample. The presence of the researcher during searchsessions and the talk-aloud method per se mayhave had an impact on the searches. For ethicalreasons, subjects experiencing difficulties withsearches were allowed to ask for assistance. Insome cases the researcher provided directionduring searches when assistance appeared to beneeded, but was not requested. The researcheralso asked some probing questions. The pres-ence of the researcher and the researcher?sinteractions with participants make this a hybridresearch method, part verbal protocol, partinterview. The talk aloud method itself may have hadsome impact on the participants? search strate-gy. For example, participants may have tried todemonstrate what they considered to be the?correct? method of searching rather thansearching in their usual manner.Further research would be needed to deter-mine the frequency with which problems notedduring this study occur. Results of this studymay be useful in indicating areas which meritfurther study. Most of the problems appeared tofall into one of three categories:(1) availability and limiting by location; (2) subject searching; and (3) arrow and keystroke. 203Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206One finding that could be relevant to develop-ment of online catalogues is that it appears to bevery important, at least to this group of subjects,for users to identify easily those materials thatare available locally. Although half of the partici-pants indicated a preference for materials avail-able in the local collection, none were aware ofDRA?s ?limit by location? feature. Even thosewho requested assistance found it difficult tolimit by location. Simplifying this task wouldimprove the usefulness of the online catalogue.Difficulties with subject searching have beennoted by a number of researchers. Clear onlineinstructions for users when either no matches orlong hit lists are retrieved could be helpful.Signage in the reference area clearly pointing tosubject help guides, i.e. The Library of Con-gress Subject Headings, might alleviate someproblems for some users. Arrowing and keystroke problems mightdisappear with the next generation of onlinecatalogues based on graphical user interfaces.Recommendations for ConcordiaHalf the participants in this study indicated apreference for materials available onsite, butdetermining availability and limiting by locationwere the most common problems identified.This supports the prior belief of Concordialibrary staff that identifying materials availablelocally is difficult. One strategy which staff hasimplemented to minimize this problem is toenhance the browsability of the collection.Signage has been added at the end of rangeslisting subject headings for all areas wherecollection size is significant (more than 50titles). A collection of pathfinders covers allmajor subject areas taught at Concordia.Library of Congress major subject headings areposted prominently in the reference area and alist of Dewey subject headings is readily avail-able as well. While this approach may be usefulin providing clients with alternative methods offinding materials in the collection, it does notaddress the main problem, which is that clientswant to be able to use the online catalogue toeasily identify materials in the local collection.Setting DRA to limit materials availablelocally would solve the problems clients have inlocating materials available locally, but wouldmake it somewhat more difficult for clients toidentify materials available through the NEOSConsortium. The importance of having materials availablelocally may be decreasing as interlibrary loanactivity increases. A few years ago, interlibraryloan activity was negligible at Concordia. Thiswas due partly to the lack of delivery mecha-nisms fast enough to meet the needs of anundergraduate population, and partly due to theavailability of a number of high quality researchand special libraries in the Edmonton area (e.g.the University of Alberta, government depart-ment libraries). Interlibrary loan activity has increased dra-matically in the past few years, primarily as aresult of the document delivery agreementwithin NEOS that guarantees a three-day turn-around time. The merger of circulation andinterlibrary loan services and faster deliveryservice (because of the courier service and theuse of Ariel for articles) have both helped toimprove the availability of interlibrary loanservices during the library?s open hours. This trend is likely to continue to increase asConcordia becomes involved in more resource-sharing agreements, particularly The AlbertaLibrary, which involves public, academic, andregional library systems throughout theprovince. For example, one of the projects ofThe Alberta Library is involved in is ensuringthat all post-secondary libraries throughout theprovince have either Relais (large institutions)or Ariel (smaller institutions) workstations fordocument delivery, thanks to funding providedby Alberta Advanced Education and CareerDevelopment. This will make requesting ofarticles much more convenient and desirable forConcordia?s undergraduates. Library instruc-tion sessions now place more emphasis both onteaching students to identify materials availablelocally and how to place interlibrary loanrequests. It is hoped that DRA?s new TAOS operatingsystem, currently under development, willaddress the difficulty of identifying materialsavailable locally. From the point of view of theclient of the small library in a big consortium, itwould be ideal if the main screen would includethe option of searching the current location only(i.e. Concordia only), the whole NEOS 204Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206database, or other options using Z39.50, asdetermined by Concordia. Permitting individ-ual locations to set up their own search optionswould be ideal because even within a consor-tium, different institutions or groups of institu-tions, have different licensing arrangements fordatabases, and different resource-sharing agree-ments with other agencies. Arrow and keystroke problems will eventual-ly be eliminated by moving to a Web-basedinterface. Currently, DRA Web is available tothe NEOS Consortium. However, Infogate ispreferred at Concordia, because DRA Web doesnot allow for limiting by location, and BorrowerInformation has not been made available due toconcerns that it does not work correctly.Emphasis has been placed on Infogate only,because although some kinds of searchingwould be easier on DRA Web, it is believed thatthe complications of teaching students twodifferent interfaces for the online cataloguewould outweigh the benefits. The DRA Web2interface is expected to be available to theNEOS Consortium by fall of 1999. Concordiastaff will evaluate this interface and determinewhether to continue emphasizing Infogate orswitch to DRA Web2.Suggestions for further researchThe verbal protocol method offers great poten-tial as a research method in a number of areas oflibrarianship. When designing interfaces,whether for intranets, or customized interfacesfor various library databases or products, theverbal protocol method can be used to identifyproblems for users before the interface is madeavailable to clientele. The verbal protocolmethod also has potential as a tool for analyzingthe research process or the reference interview.Wiedenbeck et al.(1989) provide a concise,useful description of how to conduct a verbalprotocol study.ConclusionThe verbal protocol method is a useful means ofobtaining data on the online catalogue. A totalof 11 problems or comments were noted inthree or more of the ten search sessions, incontrast to five problems or attractive featuresnoted in three or more of the ten sessions inresponse to interview questions.Participants find it easy to follow instructionsto comment on what keys they are pressing,what they see on the screen, and how they react.These comments make it easy to determinewhat was happening on the screen when tran-scribing sessions. When planning a verbal proto-col study, allow about three times as much timefor the average search session to allow forrecruitment, transcription, and data analysis. This method has a lot of potential applica-tions in librarianship, including interfacedesign, the research process, and the referenceinterview. Librarians might find this a naturalmethod to use, as it is somewhat similar toconducting a reference interview.Unlike Henderson et al.?s (1995) study, it wasfound that interview questions provided addi-tional information not obtained through theverbal protocol search sessions. Answers tointerview questions could also help to indicatethe relative importance of problems to users, i.e.sort out the difference between a commonnuisance and a genuine obstacle to research.The results of this study (remembering thesmall size of the sample) indicate three mainareas of concern for users: identifying what isheld locally and what is available, subjectsearching, and arrowing and keystrokes.ReferencesBorgman, C.L. (1986), ?Why are online catalogs hard to use?Lessons learned from information retrieval studies?,Journal of the American Society for InformationScience ol. 37 No. 6, pp. 387-400.Borgman, C.L. (1996), ?Why are online catalogs still hard touse??, Journal of the American Society for InformationScience ol. 47 No. 7, pp. 493-503.Henderson, R.D., Smith, M.C., Podd, J. and Varela-Alvarez, H.(1995), ?A comparison of the tour prominent user-based methods for evaluating the usability of computersoftware?, Ergonomics, Vol. 38 No.10, pp. 2030-44.Seymour, S. (1991), ?Online public access catalog userstudies: a review of research methodologies, March1986-November 1989?, Library and InformationScience Research ol. 13, April-June, pp. 89-102.Wiedenbeck, S., Lampert, R. and Scholtz, J. (1989), ?Usingprotocol analysis to study the user interface?, Bulletinof the American Society for Information Science,Vol. 15 No. 5, pp. 25-6.205Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206AppendixDetailed script including recruitmentprocedures, instructions, and questionsVerbal protocol scriptTry for some randomization of recruitment, e.g.pre-select times on a random basis to approachpeople; number computer terminals, approachfirst person to go to that terminal at that partic-ular time and say something like this: Hi. I?m conducting a research study on the onlinecatalogue. May I take about a minute of your timeto briefly explain my project and ask for yourparticipation?My name is Heather Morrison. As you may know,I am the Circulation Services Coordinator here atConcordia. I am also working on a Master?sdegree in Library and Information Studies at theUniversity of Alberta. As part of my work towardsthis degree, I am conducting a study to test thetalk aloud method as a way of studying the onlinecatalogue, or GATE.What I am asking people to do is to conduct thesearches they came to the library to do, whiletalking aloud about what they?re doing and howthey react to the computer interface. This wouldbe done in the privacy of my office, with myselfpresent as an observer. Participation is purelyvoluntary; please feel free to say no if you wouldrather not participate. The search session will berecorded on audiotape and is completely confi-dential. After the search session, there will be abrief interview. No information that would identi-fy you as an individual would be collected.The time required for this study is about the sameas your search is likely to take, plus about five extraminutes for talking and questions, including thistime.Are you willing to participate in this study?If no, then say: ?thanks for your time, and goodluck with your search.?If yes, say: I?d like to give you a copy of this brief summary ofthis research project, including the name andphone number of my instructor, should you haveany concerns or questions about the study. Talk aloud studySay: The main focus of my study is evaluating thecomputer interface of the online catalogue, that is,does it help people to find the information ormaterials they need, or does it need improvement?So, what I?d like you to do is to conduct yoursearch as you usually would, but talk aloud aboutwhat you?re doing, that is, what keys you?re press-ing, what you?re seeing on the screen, what you?relooking for on the screen, that type of thing, aswell as how you feel about the information thecomputer is giving you. Do you have any questionsbefore we start?Any comments to participants, if needed, bothat this time and during session, are restrictedto:? talking about what they are doing;? commenting on the reasons for what they?redoing;? comments about how they feel about what?son the computer.After the session, ask:? About any specific questions noted duringthe session.? What aspects of the online catalogue did youfind were the most problematic during thesession?? What aspects of the online catalogue did youfind were the most attractive during thesession?? Based on past experiences with the onlinecatalogue, what aspects have you foundproblematic or attractive?? Do you have any other comments or ques-tions about the online catalogue?? Given the online catalogue as it is now, whatcan library staff do to help people find thethings they need, in terms of orientation orwritten guides?? Do you have any other comments or ques-tions about this research project?? Did you find the information you needed? (Ifnot, provide reference assistance or refersubject to the reference desk.)Finally, say:I?d like to thank you very much for participating.If you?d like a copy of the final report of myresearch, it should be available in early April; youcan pick up a copy here at the Circulation Desk.206Online catalogue research and the verbal protocol methodHeather G. MorrisonLibrary Hi TechVolume 17 ? Number 2 ? 1999 ? 197?206


Citation Scheme:


Usage Statistics

Country Views Downloads
United States 9 4
France 6 0
China 5 0
Canada 5 0
Japan 3 3
Germany 1 0
City Views Downloads
Unknown 13 0
Ashburn 6 1
Abbotsford 5 0
Beijing 3 0
Shenzhen 2 0

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}
Download Stats



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items

Admin Tools

To re-ingest this item use button below, on average re-ingesting will take 5 minutes per item.


To clear this item from the cache, please use the button below;

Clear Item cache