UBC Faculty Research and Publications

An empirical evaluation of computerized databases for emergency care Pugh, George E.; Tan, Joseph K.H. Nov 30, 1994

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-Pugh_G_et_al_Computerized_databases.pdf [ 13.95MB ]
JSON: 52383-1.0048387.json
JSON-LD: 52383-1.0048387-ld.json
RDF/XML (Pretty): 52383-1.0048387-rdf.xml
RDF/JSON: 52383-1.0048387-rdf.json
Turtle: 52383-1.0048387-turtle.txt
N-Triples: 52383-1.0048387-rdf-ntriples.txt
Original Record: 52383-1.0048387-source.json
Full Text

Full Text

CD0 ——B=CDCD..(1)0—H—aC,oCDCl)1jBHCbr—“C“C.‘10HThe Centrefor HealthServices and PolicyResearchwas established by the BoardofGovernors of the University of BritishColumbia in December 1990. It was officially openedin July 1991. The Centre's primaryobjectiveis to co-ordinate, facilitate, and undertakemultidisciplinary researchin the areas of healthpolicy,health servicesresearch, populationhealth, and healthhumanresources. It brings togetherresearchers in a varietyof disciplineswhoarecommitted to a multidisciplinary approach to researchandto promoting wide dissem­Ination anddiscussi.on o! researchresults, in these areas.The Centreaimsto contribute tothe improvement of population healthby being responsiveto theresearchneedsof thoseresp­onsible for healthpolicy. To thisend, it providesa researchresourcefor graduate students;develops and facilitates access to healthandhealth care databases; sponsorsseminars, workshops,conferences andpolicyconsultations; and distributes Discussion papers,Research Reports andpublication reprints'resulting fromthe researchprogramsof Centrefaculty.The Centre's HealthPolicyResearch UnitDiscussionPaper seriesprovidesa vehiclefor thecirculation of preliminary (pre-publication) worle of CentreFacultyand associates. It is intendedlOpromote discussion and to elicitcomments and suggestions thatmightbe incorporated withintheworkprior to publication. While theCentreprintsand distributes thesepaperS for thispurpose,theviews in thepapers are those of theauthor(s).A complete list of available Health Policy Research UnitDiscussion Papers andReprints, alongwith an address to which requests for copies shouldbe sent,appears at thebackof each paper.AN EMPIRICAL EVALUTION OF COMPUTERIZED DATABASES FOREMERGENCY CARE*byGeorge E. Pugh, MD, MHSc.,FRCPC(EM)Staff, Department of EmergencyMedicine,Mount St. Joseph Hospital,3080 Prince Edward StreetVancouver, British Columbia, CanadaTelephone: (604) 874-1141Fax: (604) 875-8733Joseph K. H. Tan, PhD,"Associate Professor,Division of Health Policy andManagementDepartment of Health Care andEpidemiology,Faculty of medicine,University of British ColumbiaTelephone: (604) 822-2737Fax: (604) 822-4994E-mail: josepht@unixg.ubc.ca"Please send all correspondence to Dr. J.Tan. This work is partly supported by funds received by theauthors from the University Hospital Foundation, the Social Sciences and Humanities ResearchCouncil (SSHRC), and the National Health Research and Development Program (NHRDP)2AN EMPIRICAL EVALUTION OF COMPUTERIZED DATABASES FOREMERGENCY CAREABSTRACTA field-based evaluation is conducted of a Clinical Computerized InformationSystemTM (CCISTM), a medical database product of Micromedex, Inc. Followingtraining, the use of the CCISTM database, word processing and other programs bythirteen full-time practicing emergency physicians in two urban emergencydepartments of a University-associated teaching hospital was studied over a l-yearperiod. A tracking program automatically logged frequency and duration of use bythe physicians and user satisfaction was assessed by a reliable and validatedquestionnaire instrument. Based on utilization data and verbal reports of thesephysicians, CCISTM database searching was not only found to be easy-to-learn butwas readily accessible during emergency shifts. Individual physician was found toperform an average of 3.5 searches per month lasting a mean search time of 8minutes. Positive notes about the CCISTM system included ease-of-use, accuracy ofdata, accessibility of system and value of output while negative perceptions includeda lack of integration with other systems, a lack of system completeness and a highsubscription cost. It was suggested that a less costly telephone link to a high-volumeCentre would be desirable in actual implementation of the system.Key Words: Clinical Computerized Information Systems, ComputerDatabase, User Satisfaction, Software Evaluation, Managementof Technology3AN EMPIRICAL EVALUTION OF COMPUTERIZED DATABASES FOREMERGENCY CAREINTRODUCTIONManagement of emergency patients demands a breadth of knowledge, and theability to recall and utilize that knowledge under stress. Emergency physicians havetraditionally used protocols and guidelines that are either memorized orsummarized on wall or pocket charts. Further information to solve clinicalproblems is retrieved at some time expense from medical texts, notes, andcolleagues. These additional sources are often incomplete and may slow downresponse time. Although complex decision making that deals with ill-structuredgoals and relationships is typically best handled by man, the recent introduction ofPersonal Computers (PCs) and on-line databases has provided an innovativesolution to the information overload that must be managed by the emergencyphysician.On-Line Databases For Emergency PhysiciansThe recent availability of on-line databases designed specifically for emergencyphysicians has gained popularity among some users. There are currently fifteensuch database implementations within eleven institutions in British Columbia(BC), Canada. Both teaching and community hospitals have subscribed to thistechnology for their Emergency departments. Indeed, each of the 68 GeneralHospitals in BC and the 649 General Hospital facilities within Canada is a potential4subscriber to this technology. The effectiveness, quality and cost of current healthcare delivery may be altered by the use of automated databases. Nonetheless, thequestion: "How attractive does a new technology have to be to warrant adoption andutilization?" [1] is often asked. Further, as Emery [2] noted, "cost/benefit analysis ofinformation systems faces some extremely complex issues - we are a long way fromdeveloping fully satisfactory approaches". It is our hope that this report will add toour knowledge on the evaluative benefits of introducing a CD-ROM basedinformation system to a hospital emergency department.The 1970's saw the introduction of on-line databases such as MEDLINE(National Library of Medicine) that are accessible by PCs via modems and telephonedata links. A previous study by Marshall [3] in a medical office setting showed thatan average search took 22 minutes, and that these searches were plagued byconsiderable initial difficulties when interfacing between the end-users and thehardware and software svstern. User interface programs such as Grateful MedR asdescribed by Lindberg [.1.] has made MEDLINETM searches easier and less expensive.Nevertheless, time and expense remain significant barriers. Most of the 24participants in the Marshall project indicated that they were unwilling to pay a$85.00 monthly subscription fee to continue the service after the project wasterminated.Since 1974, stand-alone databases were promoted by Micromedex under thename of Computerized Clinical Information System™ (CCISTM) [5] , and thisproduct is beginning to gain attention, especially in Poison Control Centers and forteaching hospital Emergency departments. The database was originally provided onmicrofiche, but is now available on PCs with CD-ROM or it can be accessed through5mainframe systems such as IBM PCS/ADS or DEC VAX VM3.The CCISTM database is updated every 3 months and includes the followingmodules:1. Toxicology Information (POISINDEX, TOMES, IDENTIDEX)2. Drug Information (DRUGDEX, MARTINDALE, IDENTIDEX)3. Disease and Trauma Information for Acute Care (EMERGINDEX)4. Aftercare Instructions5. Teratogen Information System (TERIS)The CCISTM system is designed to provide information that is supportiveand applicable for emergency clinical decisions and thus one of its main features isan interface that is both easy-to-use and easy-to-learn. Yearly software licensesapproximate $13,000 (Canadian) depending on the range of products subscribed toand the number of subscriptions. Despite the 'hype' [6] surrounding the use ofMicromedex CCISTM, there remains a lack of published empirical reports thatobjectively evaluates the usefulness of this product. Goldman [7], in summarizingcomputer use in medical education, noted that electronic tutorials, multiple choicetesting, and patient simulations have become commonplace and are regarded aseffective for training medical students and residents. It is hoped that this initialattempt will fuel interest in the need for more critical reviews and evaluations ofsimilar type products.Yamamoto [8] reported that EMERGINDEX is an extensive computer databaseof emergency management information that displays and calculates drug dosages for6pediatric and adult patients. However, the drug table does not guide a practitionerin the management of a specific condition. He described the development andtesting of a computerized 'laptop' system that can be used at the bedside, andshowed that for residents in training, there was a reduction of time required for thepreparation and administration of emergency treatment and that managementerrors were minimized.Dependent VariablesThis study examines the usefulness of Micromedex CCISTM as perceived andreported by experienced certified emergency physicians. The following dependentvariables are used to guide the interpretation of the study results:1. Reported satisfaction with the CCISTM databases2. Actual utilization of the database3. Expressed willingness to pay for the serviceMATERIALS AND METHODSSubjectsAll emergency physicians of University Hospital at both the UBC and Shaughnessysites were invited to participate. Both departments have students of emergencymedicine rotating through for varying periods of time, and have active monthlyjournal clubs and Clinical Rounds that these physicians and students are required toparticipate in during the study (evaluation) period. Neither department is equipped7as a major trauma Centre nor handles a large number of pediatric admissions.All thirteen experienced emergency physicians (excluding the principalinvestigator, who is one of the co-authors of this paper) consented to participate inthe study. Table llays out key characteristics of the participating physicians and theemergency departments where they practice.[PLACE TABLE 1 ABOUT HERE]At the UBC Site, the CCISTM system was implemented at the nursing station,close to where clinical notes were written by both nursing and medical staff. At theShaughnessy Site, space limitations required the system to be installed 15 metersaway from the nursing station. Staff must therefore leave the patient care area andcross a corridor to the emergency doctor's office-library to access the database.ProceduresInitial contact with the participants was made to outline the purpose of the studyand assure the confidentiality of data to be collected. The microcomputer-basedCCISTM system was then introduced (July, 1991) at the two University Hospital sitesand each staff member was given initial training of up to two hours. Training wasprovided by a company representative as well as by one of the investigators of thisstudy, who was then also a staff member of the two hospital sites.Use of the CCISTM system by students was actively promoted and encouragedthrough a letter at the time of orientation. A brief introduction to the system was8also given whenever new students attended the department. Use of the database bystudents and other staff doctors was logged as "resident" mode. Satisfaction scoresfor "resident" and "registered nurse" were not assessed by this study.The hardware which was used included stand-alone IBM clones (IPC Systems,Vancouver), which are 386 CPU computers (25-Mhz, 2-MB memory, VGA colourmonitor) running on DOS 5.0 (Microsoft Corporation, Redmond, Washington,USA). These machines were wired with Hitachi 1700 CD-ROM players (Hitachi,Canada), and connected to Quietjet printers (Hewlett Packard, Canada). For thisstudy, we chose to use a stand-alone microcomputer environment in preference to amainframe environment to avoid the problem of shared CPU resources (i.e.,computer processing time associated delays) and, more importantly, to ensure thatthe timing data for information searches would not be affected by system overloads.In addition, the decision not to use mainframe computers at both hospital siteseliminated high initial setup and hardware costs.A simple log-on procedure using Saber Menu SystemR (Saber SoftwareCorporation, Dallas, Texas), and an individual's three initials permitted thefollowing information to be traced: (1) identification of individuals using thecomputer, and (2) the application that was being used. Saber MeterR, using astandard dBASE file format, recorded the time and method of logging in and out ofan application. The program Saber SecureR, was used to force logout from theMicromedex database following a period of 10-minute keyboard inactivity. Validelapsed time measurements for the use of the database were calculated by analyzingthe dBASE file with dBASE III PLUS command language (Borland International,Scotts Valley, California) and statistical summaries performed with LOTUS9algorithms as described in the Data Analysis section. This is not to suggest thatlinking spreadsheet with DBASE may serve as a means for all statisticalcomputations except that in this instance these software provided for fileinformation sharing, thus gaining the convenience which cannot be achieved withinterfacing DBASE with most standard statistical software, for example, SAS.Moreover, It should be noted that the analysis done for the study with thesesoftware pertains only to easy-to-generate statistical measures such as means, modes,cross-tabulations and correlational analysis.In general, it was found that staff did not built up the habit of exitingprograms after use; care was therefore taken to subtract 10 minutes of keyboardinactivity time from forced logouts each time they occurred. Access duration's ofone minute or less were however regarded as insignificant since a minimum of 65seconds were required to obtain even a quick reference from the CCISTM database.Following 6 months of exposure to the Micromedex system, logging on systemutilization commenced (December 1991) and this system evaluation processcontinued until the end of June 1992.CCISTM System EvaluationTen months following the introduction of the CCISTM system, a questionnaire wasadministered and completed by all the participants with the full assurance ofconfidentiality on individual responses. The instrument was adapted from Dr. J.E.Bailey [9], who developed it specifically for measuring user satisfaction with variousaspects of computer systems. The choice of this instrument was based on aconsideration of its reliability and validity ratings as well as on published results of10its testing on 860 computer users in 22 different hospitals (comparison group). Theobserved scale reliability, that is the ability to measure the same thing withconsistent results, was adequate and acceptable as supported by test-retest result.Considerations were also given to the content and predictive validity of theinstrument; that is, indications of its validity for measuring overall satisfaction andthe causes for those feelings.Several items in the questionnaire were simply re-worded to fit in with theMicromedex database evaluation. Measurement of user satisfaction employed the"semantic differential technique" which involves presenting pairs of bipolaradjectives (i.e., fast - slow) describing the factor to be measured. The adjectives wereseparated by a 7-interval rating scale that allows for quantification of the intensity ofindividual responses; each of the 41 factors assessed was evaluated by 2 adjectivepairs. Another scale was used to measure the importance of each factor with 7intervals ranging from extremely unimportant to extremely important.The following equation calculates the sample group's feeling about aparticular factor [9]:S(j) = 1 I KL W(i,j) .. K LI(i,j,k)K i=! k=!where: S(J)= Satisfaction of all users with factor jW(i,j)= Importance value of factor j for user iI(i,j,k)= Score of user i for adjective pair k of factor jJ= Number of factors in the study (= 41)K= Number of adjective pairs per factor (= 2)1= Number of subjects in the study (= 13)11When subjects indicated that a particular factor is "not applicable" (N / A), scores forthat factor were subsequently excluded from the computation of an overall factorscore (see the Appendix). The Appendix gives a copy of the questionnaire that isused in the study reported here.Data AnalysisThe composite raw score was computed according to the proposed model and thennormalized to a range of -100 to +100 by dividing by the appropriate scale factortimes the number of subjects who considered a particular factor (j) applicable.Individual satisfaction scores, although calculated, are not reported here. Previouswork by Bailey suggests that composite user satisfaction scores of less than 20 forindividual factors (see results presented on comparison group) indicate thatproblems may exist. Negative factor scores were regarded as a significant sign ofproblems. Support from the "most liked" and "least liked" comments given by therespondents provides additional information for interpreting the satisfaction scores.Questionnaire data was entered directly onto a Lotus V2.01 spreadsheet foranalysis; utilization data were gathered and stored in DBASE files and wereabstracted accordingly for analysis. Descriptive statistics and regression analyseswere computed using Lotus V2.01 algorithms (Lotus Development Corporation,Cambridge, Massachusetts). The results are tabulated in Tables 2 and 3 and furthersummarized and presented graphically in Figures 2 and 3.[PLACE TABLE 2 ABOUT HERE]12RESULTSCCISTM UtilizationTable 2 summarizes the utilization for the Micromedex database and otherprograms by the participants. After training, 12 of 13 physicians (92%) elected toperform information searches on the CCISTM system. Searches by emergencyphysicians on the system took an average time of 8.1 minutes with a median time of5 minutes and 9 seconds. Figure 1 shows a histogram plot of the frequency ofutilization time.[PLACE FIGURE 1 ABOUT HERE]Figure 1 also showed the convex pattern of cumulative frequency of CCISTMutilization and accesses for emergency physicians. For example, 50% of all theaccesses took between 4 to 6 minutes of information retrieval time to complete andthe longest searches took between 50 to 60 minutes (= 1 hour). Search frequency byall emergency personnel (i.e., including emergency physicians, nurses, and students)was found to average 57 accesses per month at the 2 sites combined. Utilization ratefor 'active' (i.e., 12 out of 13) emergency physicians averaged 3.5 searches per month;one physician although trained, was not personally motivated to use the system. Inparticular, emergency physicians did not feel that access was limited in theirdepartments. Multiple logistic regression suggested that owning or having regularaccess to a home computer did not influence the rate of searching on CCISTM inthis study. Search frequency was found to be positively correlated to physiciancertification in emergency medicine, early date of graduation, older age, and fewer13full-time years in emergency medicine.Figure 2 depicts that searches were initiated uniformly over the 24 hours forevery 2 hours of the day with no particular clustering evident.[PLACE FIGURE 2 ABOUT HERE]User Satisfaction ScoresTable 3 summarizes the views of the 13 respondents concerning their feelingstowards the computerized database.[PLACE TABLE 3 ABOUT HERE]The application software (i.e., Micromedex CCISTM database) was noted to beeasy-to-use, perceived as providing accurate data, and seen as being placed inaccessible and convenient locations. The value of the retrieved information wasrated as high . "Most liked" comments included references to "quick access" (6),ability to print out information (3), timeliness (currency) of information (4), andcompleteness of retrieved information (4).In general, users did not find that the database saved time (user satisfactionscore for "time saving" was 11). The need for the database was felt to be low(difference score of -27). Participants felt excluded from planning the introduction ofthe database. Perceived weaknesses of the automated database included a lack ofsystem completeness (Satisfaction score of 8), relevancy (difference score of -22), and14low volume of information (Score of 0). "Most disliked" comments amongrespondents included references to "many important topics missing" (8),"repetitious, platitudes" (4), "time consuming/difficult to locate information" (4).Not one participant suggested that his clinical practice had changed as a result of theL-year exposure to the computerized database. Participants indicated the service wasworth individually an average of $107. (range $SO-SOO)/year.The 386 DOS-based computer system generally invoked positive feelings. Yet,the stand-alone nature of the terminal prevented use of the system from home orfrom other terminals within the hospital, and thereby resulted in a general feelingof concern for its lack of integration with other systems in the hospital. Ratings oncomputer staff factors were similar to those reported of large well-managed systemsat other hospitals, as shown in Table 3 (comparison group). Organization factorsreceived three low satisfaction scores, reflecting the fact that ManagementInformation Services (MIS) had no visible role in introducing or supporting thestand-alone computer system.DISCUSSIONOur investigation has focused on studying the frequency and usefulness of on-sitecomputer access to a state-of-the-art database CD-ROM system designed forEmergency Department physicians. While the opinions of other users (e.g.,residents and registered nurses) have been tracked, these were not explored furthersince they were not the subject population of interest. Limitations of the studytherefore include the limited number of experienced emergency physicians locatedat the two different sites. As well, the results are generally interpreted as not15applicable to hospital personnel other than experienced emergency physicians, andmay only apply to sites involved in regular teaching of residents and students.Perhaps, this is one reason why the findings have little association with the resultsof previous use of computer in addition to results of studies cited below.In general, it was found that system accessibility impacts significantly on itsutilization. The UBC site, with the computer physically at the nursing station, was aprime example. Further investigation of the full utilization data set revealed thatUBC Registered nurse searches totaled 73 compared to 10 accesses by Shaughnessynurses. There is however a lack of evidence that prior exposure to and interactionswith computers (i.e., owning or having regular access to a home computer) willinfluence office use of computers although Hayes [10] reported a positive correlationbetween the use of MEDLINE system and that of home computer systems.Silver [11] reporting library utilization, noted that interactive searches onMEDLINE took 29 minutes, and PsycLIT took 44 minutes while Hayes reportedMEDLINE searches averaged 23.6 minutes. In the case of the CCISTM system, wefound that the average search time was only 8 minutes. This may be due to thenature of the database system which is designed specifically for use of emergencystaff in hospital emergency departments. While the number of participants in ourstudy may be limited, we recorded a participation rate of 92% (Table 2) in contrast toonly 81% reported by Hayes even after adequate training in on-line searching ofMEDLINE had been provided to 188 doctors. In our study, only one emergencyphysician (although he consented to participate in the study) did not use the on-linesystem and admitted that he had no personal interest or motivation in interactingwith the system.16Tobias [12], reporting on the use of the same CCISTM database in anEmergency Department serving a 615-bed teaching hospital, noted that an average of97 searches were conducted per month by all emergency staff while our finding was57 searches/month (Table 2). Physicians and other certified health professionalswere noted to access the "Library Information System-Drug Information Service"System (CCIS, MEDLINE, and Library Information) at a rate of 0.8 times per monthin that study; in contrast, we found emergency physicians accessing an average of3.5 searches/month. Hayes noted a monthly search frequency on MEDLINE of 0.9times for staff doctors while medical students performed an average 3.2 searches permonth. Apparently, different professions appear to valuate different informationretrieval systems on a contingency basis depending on its perceived capacity ofcontributing to and supporting their respective needs for information.It is clear therefore that accessible, accurate, relevant and comprehensiveinformation is required in all clinical education and work-site settings. Laupacis etal. [1] published guidelines for clinical and economic evaluations of newtechnologies. The costs of computerized databases for information retrieval can beeasily measured and quantified. The benefits of quality-adjusted life-years (QALY's)gained are however not so easily assessed following the use of reference material.Laupacis et al. suggest the alternative approach of determining "willingness to pay"on the part of the user. For participants in our study, an average of $107.00 wasdetermined, indicating that they valued the system much lower than the yearlysubscription price. This value may understate the benefits to students, registerednurses, and potentially to the patients themselves, all of whom are beyond the scopeof the applicability of the findings reported here.17CONCLUSIONMicromedex, Inc. has produced a product that is easily accessed, is focused (8.1minute search time compared to 23.6 minutes for MEDLINE) and gives accuratedata. Unfortunately, as Dvorak [13], a keen observer of the computer field, notes:"data is almost worthless -like a heap of dirt laced with gold dust. How much of thedirt has any real value to the miner? Not much. The bigger the pile of dirt, themore work that has to be done to extract the necessary information". Thissentiment underlies the low satisfaction scores for "system completeness","relevancy", and "low volume of information" and multiple comments concerninginability to find the topic desired.While there are limitations with this study as the focus has been on theemergency physician's interaction and use of CD-ROM information systems, theresults will still be applicable to other practicing physicians in community hospitalswho are planning to acquire such systems. Participants accessed the database at arate higher than previously reported in the literature: information seeking habitshave certainly been improved by the system. Usage by the emergency staff wasenhanced by physical proximity to the system. However, further work is needed toassess the cost-benefit of computerized database technology on the effectiveness ofphysician performance and ultimately on patient care. Consideration of benefitssuch as having MEDLINE on an interactive CD-ROM, computerized simulations ofvarious clinical procedures and other software packages such as the Utilization CarePlan (UCP) system advocated by Tan et al. [14] may thus be seen as potential areas forfuture clinical database evaluation research.18Finally, some of the recommendations made by the survey respondents as aresult of the questionnaire administration include:1. Computerized logging to provide a means of evaluating new system features2. Recognition of the low volume and high cost of searches generated by ourtwo sites suggests a formal agreement for dial-up access by modem to a largervolume user would be more cost effective3. Integration of the database to other hospital computer systems is seen asdesirable4. Location of the computer system at the work site will encourage greaterutilization1 9ACKNO~EDGEMENTSWe are indebted to the University Hospital Foundation and other granting agencies,in particular, SSHRC and NHRDP for supporting this work. We are also indebted toMr. Scott Bryson: Assistant Director, Clinical, Education and Research Programs,Department of Pharmacy, University Hospital; to Dr. Victor Wood: Head,Department of Emergency Medicine, University Site; to Dr. Bruce Fleming: Head,Department of Emergency Medicine, UBC Site; and to all our Colleagues atUniversity Hospital, both nursing and medical, for their encouragement andparticipation, without which this project would not have been successfullycompleted. We thank also Dr. James E. Bailey, Arizona State University, for hiskind permission to adapt and use his "Computer User's Satisfaction" questionnaire.Finally, we thank the three anonymous review~rs who have been very helpful inproviding further insight to the study and in improving the presentation of thisreport.20REFERENCES[1] Laupacis A, Feeny D, Detsky AS, Tugwell PX: How Attractive does a NewTechnology have to be to Warrant Adoption and Utilization? TentativeGuidelines for using Clinical and Economic Evaluations. Can Med Assoc J.1992; 146(4):473-481.[2] Emery J: Cost/Benefit Analysis of Information Systems, in Cougar JD, ColterMA, Knapp RW(eds): Advanced System Development /Feasibility Techniques,J Wiley & Sons, 1982:459-488.[3] Marshall JG: On-line Databases for Health Professionals: Are They ReallyUseful? Can. Fam. Physician. 1987;33:887-893.[4] Lindberg DAB: Information Systems to Support Medical Practice and ScientificDiscovery. Methods of Information in Medicine. 1989; 28: 202-206.[5] Micromedex Computerized Clinical Information System, Vol. 73, Expires Aug31th, 1992,600 Grant St., Denver, CO., 80203-3527.[6] Rann LS, Winokur, :\1G, Kuchta, NE: The Computerized Clinical InformationSystem on CD-ROyL Optical Information Systems 1986;6(4):313-316.[7] Yamamoto LG, \Viebe RA: Pediatric and Adult Emergency ManagementAssistance Using Computerized Guidelines. American Journal of EmergencyMedicine. 1989;7(1):91-96.[8] Goldman B: Computers in Primary Care Teaching. Canadian Family Physician .1990;30:1983-1986.[9] Bailey, JE: Development of an Instrument for the Management of ComputerUser Attitudes in Hospitals. Methods of Information in Medicine 1990;29:51-56.[10] Hayes RB, McKibbon KA, Walker CJ, Ryan N, Fitzgerald D, Ramsden MF: On­line Access to MEDLINE in Clinical Settings. Annals of Internal Medicine .211990;112:78-84.[11] Silver H, Dennis S: Monitoring patron use of CD-ROM Databases using Signln-Stat.Bull Med Libr Assoc. 1990;78(3):252-257.[12] Tobias R, Bierschenk NF, Knodel LC, Bowden VM: Improving Access toComputer-Based Library and Drug Information Services in Patient-Care Areas.Am J of Hospital Pharmacy. 1990;47:137-141.[13] Dvorak JC: Data has no Worth. PC Magazine. Apr 1991:81.[14] Tan JKH, McCormick E. and Sheps S. Utilization Care Plans and EffectivePatient Data Management. Hospital & Health Services Administration.1993;38(1):81-99.TABLE 1: Characteristics of Emergency Departments and PhysiciansNumber surveyed:Age:SexYear of MD graduation:CertificationfTraining:Full-time emergencyexperience:Computer expertise:Home computer:13 emergency physicians41.5 years average (range: 33-49)2 females, 11 maleswithin span of 1969-19857 Fellows of royal College of Physicians and Surgeons (FRCPC)1 American Board of Emergency Medicine (ABEM)1 Canadian College of family Practice--Emergency Medicine(CCFP--EM)4 MDts averaging 12years 00-14 range) full time emergencyexperience.12.5 years (range: 4 to 22)5 with no prior exposure4 with passing acquaintance4 who are comfortable with personal computers4 possessed computers at study onset3 purchased computers during course of studyEmergency Departments of University Hospital:Number of PhysiciansVisits/yearNo. acute bedsAdmissions: RateTypeShaughnessy71860034322%Non pediatricuac62260026515%large student populationTABLE 2: Utilization of Micromedex CCIS™ DatabaseLength of Survey:• Computer malfunction:• Available time:Total access of Database:• 12 Emerg. Physicians:• Residents and nurses:Participation Rate (*) by MD's:Used by:• 'active' MD's (12):• all emergency staff:Search time (See Figure 1):• Average• MedianSoftware licenses (Two):Cost/search by MD's:Physician's willing to pay:Total usage of systems• Educational programs:• Word processor• Administrative programs:199 days5 days average for each site.194 days366 searches265 searches101searches92% (12 of 13)3.5 searches/month (range .3-12.7)57 searches/month8 minutes (range 2-60)5 minutes 9 seconds$8300 .()O/ycar$16.65 (Cdn)$107/year854 accesses229 accesses236 accesses23 accesses(*) One physician although trained, was not personally motivated to use the systemTABLE 3: Factor Satisfaction Scores for Emergency Physicians and Bailey'sComparison Group of 860 Hospital Computer UsersClassification Sample Comparisonof Factors GroupC*) GroupC*) Dill.Application Software Factors 40 44 41. Ease of Use 56 47 92. Availability of answers 18 33 -153. Completeness of CCIS database 8 37 -294. Accuracy of data 59 54 55. Format of output screens 32 53 216. Relevancy of output screens 36 58 -227. Reliability of output 0 (41)8. Computer's response time 47 26 219. Volume of information 0 28 -2810. Convenience of computer access 60 62 211. Procedure for receiving output 50 56 -612. Currency of data 32 58 -2613. Value ofCCIS output 52 20 3214. Convenience of printer access (50) ()15. Procedure for logging on (66) 0Organization Factors 13 24 -II1. University Hospital MIS polices -8 21 -292. Administration involvement w. MIS 8 19 -II3. Your control over the system 17 9 84. Goal congruence with MIS 34 36 -25. Error recovery procedures 20 19 16. Power of MIS in hospital 8 38 -307. Payment policy for MIS 0 (22)8. Competition with MIS 0 (23)9. Authority of Chief Investigator (67) 0(*) Score range: -1Olli;Score~+100TABLE 3 (cont'd)Classification Sample Comparisonof Factors Group(*) Group(*) Jlli[.Computer Staff (Investigator) Factors 54 48 61. Relationship to Staff 57 50 72. Interpersonal skills of Staff 47 49 -23. Technical skills of Staff 56 47 94. Attitude of Staff 60 58 25. Healthcare knowledge of Staff 50 36 14Computer System Factors 30 12 181. Flexihility of system 9 -13 222. Time-to-process changes 40 -5 453. Database integration with other sys. -29 15 -444. Capabilities of Computer System 30 19 115. Ease of introduction of system 53 19 346. Software support by Investigator 62 19 437. Documentation erects Database 24 40 - I6H. Security of data 40 X ,')--User Factors 28 ~8 -10I. Your training with system 30 18 122. Effects on your role as an ~D 36 41 -53. Participation in planning 12 20 -84. The CCIS Database saves you time 11 49 -385. Need for the CCIS Database 40 67 -276. Your confidence in the Database 49 50 -17. Your understanding of the Database 49 24 25Notes: 0 indicates that questions were not comparable between the two questionnaires. Satisfaction scores less than20 or less than the Comparison Group score by 20 are regarded as significant(*) Score range: - 100 S Score S +100....s.,FIGURE 1: Accumulated frequency of utilization Time ofEmergency Physicians on CCISM.- ,--- -- I --I I -- I - ---- 1 I I IUtilization Time (Min)6-8 8-1010-1515-2020-3030-4040-5050-60+Mean = 8 Min.. ~~ __ . I ~ I __ I I I I I I2-4 4-6100. 60.0(%) .010.00.00-2Median = 5 Min. 9 Sec.0>. 0..eu ~C N....0(1)E.- 0I- 0..0en COT'"">C0.-.... 0eu 0.~ 00N ~-T'""+:i nsC::J""'"0(1) CI)en Eeu i=.ceu 0....0eu 00CDC 0:::Een-0 00 0••0N 0W 0LJ") 0 LJ") 0 LI') 0('t') N Na:::JC' .......- ~LL '-'"t:0.~CON.....~APPENDIX; INSTRUMENT USED IN THE EVALUTION OF CLINICALCOMPUTERIZED INFORMATION SYSTEM (CCISTM)1. Questions are worded so that answers can be given on a scale with a neutral mid-point betweenpoles labelled "strongly agree" and "strongly disagree". Please answer each question by placing acircle about the number which best describes your evaluation of the factor being judged.2. Evaluate every scale; do not omit any.3. If the question is not applicable to you, write NIA to the left of the scales.Definitions:CCIS: Micromedex's "Computerized Clinical Information System" for toxicology, disease, andtrauma problems presenting to Emergency.MIS: Management Information Services, including the Chief Investigator.WP: Word Perfect, the word processing package installed on the microcomputer.The questions in Part I are grouped into five categories:1. Input/Output factors2. Organization factors3. MIS Staff factors4. Computer System and eCIS Database factors5. User factors.You will have a chance in Part II to indicate whether a given question in Part I is important ingauging your satisfaction with the System!The following factors deal with the inputs and outputs that you see when using the CCIS Database.1. EASE OF USE: How easy or hard the CCIS Database is to use.UnforgivingEasy to use1_2_3_4_5_6_71_2_3_4_5_6_7UserfriendlyHard to use2. COMPLETENESS OF THE CCIS DATABASE: The degree of detail and thecomprehensiveness of the output screens of the CCIS Database relative to your needs.InadequatedetailThoroughcoverage1_2_3_4_5_6_71_2_3_4_5_6_7AdequatedetailIncompletecoverage3. ACCURACY OF DATA: Your confidence that the data on the output screens are correct.InconsistentaccuracySufficientaccuracy1_2_3_4_5_6_71_2_3_4_5_6_7ConsistentaccuracyInsufficientaccuracy4. FORMAT OF OUTPUT SCREENS: The design and appearance of the output screens.Confusing layoutEasy to read1_2_3_4_5_6_71_2_3_4_5_6_7Simple layoutHard to read5. RELEVANCY OF OUTPUT SCREENS: The compatibility of the CCIS output screens to yourneeds.Useless information 1__2__3__4__5__6__7Relevant 1__2__3__4__5__6__7Usefu1 informationIrrelevant6. AVAILABILITY OF ANSWERS: Your confidence that the answers you require will beavailable when needed.Low confidenceAlways available1_2_3_4_5_6_71_2_3_4_5_6_7High confidenceNever available7. COMPUTER'S RESPONSE TIME: The response time of your computer terminal,SlowConsistent1_2_3_4_5_6_71_2_3_4_5_6_7FastInconsistent8. VOLUME OF INFORMAnON: The amount of information you get from the CCIS Databaserelative to your needs.Too much or toolittle informationPointed1_2_3_4_5_6_71_2_3_4_5_6_7AppropriateinformationRepetitive9. CONVENIENCE OF ACCESS TO THE TERMINAL: The location and availability of thecomputer terminal is convenientHard to get atAvailable1_2_3_4_5_6_71_2_3_4_5_6_7Easy to get toUnavailable10. CONVENIENCE OF ACCESS TO THE PRINTER: The location and availability of the printeris convenient.Hard to get atAvailable1_2_3_4_5_6_71_2_3_4_5_6_7easy to get toUnavailable11.PROCEDURES FOR LOGGING ON: The way you log onto the CCIS Database.InconvenientSimple1_2_3_4_5_6_71_2_3_4_5_6_7ConvenientComplex12. PROCEDURES TO RECENE OUTPUT: The way you receive output from the CCISDatabase.InconvenientSimple1_2_3_4_5_6_71_2_3_4_5_6_7ConvenientComplex13. CURRENCY OF DATA: The age of the information relative to your need for up-to-date facts.Out-of-dateAdequate1_2_3_4_5_6_71_2_3_4_5_6_7Up-to-dateInadequate14. VALUE OF THE OUTPUT: Your feeling about the worth of the CCIS Database's outputrelative to the effort involved in obtaining itUselessWorth it1_2_3_4_5_6_71_2_3_4_5_6_7BeneficialNot worth itThe following factors deal with the way the Hospital and your Department operate computerservices.15. UNIVERSITY HOSPITAL POUCIES FOR MIS (Management Information Services): TheHospital's policies and priorities regarding allocationof computer resources to users.VaguepoliciesFair priorities1_2_3_4_5_6_71_2_3_4_5_6_7Precise policiesUnfair priorities16. YOUR HOSPITAL ADMINISTRATION'S INVOLVEMENT WITH MIS: The interest andinvolvement toward computers shown by hospital managers.Frustratingconsistentinvolvement1_2_3_4_5_6_71_2_3_4_5_6_7HelpfulInconsistentinvolvement17. YOUR FEELING OF CONTROL OVER THE COMPUTER EQUIPMENT: Your power tocontrol use and maintenance/modifications of the computer system you are evaluating.IgnoredConsistent1_2_3_4_5_6_71_2_3_4_5_6_7DominantInconsistent18. GOAL AGREEMENT: the extent to which the CCIS Database and computer system'sobjectives agree with your objectives (e.g. increased productivity, increased quality of care,cost containment,etc.)DisagreeConsistent1_2_3_4_5_6_71_2_3_4_5_6_7AgreeInconsistent19. NECESSARY AUTIlORITY OF THE MIS WITHIN THE HOSPITAL: the position andauthority of your hospital'sMIS Department over your computer support.Inappropriatepower 1__2__3__4__5__6__7Used well 1_2_3_4_5_6_7Appropriate powerMisused20. NECESSARY AUTIlORITY OF CHIEF INVESTIGATOR (DR. G. PUGH) OVERSOFTWARE: The power and authority of the chief investigator over your computersupport.Inappropriatepower 1__2__3__4__5__6__7Used well 1_2_3_4_5_6_7Appropriate powerMisused21. ERROR RECOVERYPROCEDURE: The methods and policies for correctingerrors noted inthe computer system.SlowSimpleprocedure1_2_3_4_5_6_71_2_3_4_5_6_7FastComplex procedureThese factors deal with the MIS Staff, (including the Chief Investigator) in your Hospital.22. INTERPERSONAL SKILLS OF THE MIS STAFF: The ability of your hospital MIS staff tounderstandand communicatewith users.Inferior skillsCompetentskills1_2_3_4_5_6_71_2_3_4_5_6_7Superior skillsIncompetent skills23. TECHNICAL SKILLS OF THE MIS STAFF: The computer and systems analysis skillsexhibited by your MIS staff.Inferior skillsCurrentunderstanding1_2_3_4_5_6_71_2_3_4_5_6_7Superior skillsObsoleteunderstanding24. ATTITUDE OF TIIE MIS STAFF: The attitude and behaviour of the Hospital MIS personneltoward users.ImpersonalCooperative1_2_3_4_5_6_71_2_3_4_5_6_7PersonableUnhelpful25. HEALTIl CARE KNOWLEDGE OF MIS STAFF: The MIS personnel's understanding ofthe healthcare environment and special requirements.InadequateunderstandingCorrect1_2_3_4_5_6_71_2_3_4_5_6_7CompleteunderstandingMistaken26. YOUR RELATIONSHIP WITH TIIE MIS STAFF: The interaction process betweenusersand your MIS personnel.Harmful 1_2_3_4_5_6_7Healthcare oriented 1__2__3__4__5__6__7HelpfulSelf serving27. FLEXIBll..ITYOF THE COMPUTER SYSTEMAND CCIS DATABASE: The ability of thesystem to change and adapt to fit new circumstances.RigidVersatile1_2_3_4_5_6_71_2_3_4_5_6_7FlexibleLimited28. TIME AND PROCESS FOR CHANGES OR NEW DEVELOPMENTS TO THE SYSTEM:The time and process to accomplish a change or incorporate a new development to thesystem.Unacceptably long 1__2__3__4__5__6__7Simple 1_2_3_4_5_6_7ReasonablyfastComplex29. INTEGRAnON OF THE CCIS DATABASEand COMPUTERSYSTEM: The ability of thedatabase to link with other computers and programs.InsufficientEasy1_2_3_4_5_6_71_2_3_4_5_6_7SufficientDifficult30. CAPABll..ITIES OF THE CCISDATABASE and COMPUTERSYSTEM: The capabilities ofthe CCISDatabase and computersystemrelative to what you feel is reasonable.InsufficientcapabilitiesConsistent1_2_3_4_5_6_71_2_3_4_5_6_7SufficientcapabilitiesInconsistent31. EASE OF INTRODUCING THE CCIS DATABASE AND COMPUTER SYSTEM:Introduction of the system into the Department was smooth and readily accomplished.Slow intrcxiuction 1__2__3__4__5__6__7Simple introduction 1__2__3__4__5__6__7Quick introductionComplexintroduction32. SOFTWARE SUPPORT: The variety and quality of service given to you by ChiefInvestigator.Limited coverageSuperior quality1_2_3_4_5_6_71_2_3_4_5_6_7Broad coverageInsufficient quality33. WRITfEN INSTRUCTIONS FOR THE CCIS DATABASE AND COMPUTER SYSTEM:The formal description/documentation of the system and how to use itUselessAvailable1_2_3_4_5_6_71_2_3_4_5_6_7UsefulUnavailable34. SECURITY OF DATA: The safety of the data on the computer system from tampering andloss.VulnerableRecoverable1_2_3_4_5_6_71_2_3_4_5_6_7Totally secureIrretrievableThese factors deal with you and your relationship to the CCIS Database.35. YOUR DEGREE OF 1RAINING: The quantity and quality of training you received about theCCIS Database.Insufficientquantity 1__2__3__4__5__6__7Consistentquality 1__2__3__4__5__6__7SufficientquantityInferiorquality36. EFFECTS OF THE CCIS DATABASE ON YOUR FUNCTIONING IN THEDEPARTMENT: The effects the database has on enhancing your job and yourperformance.InhibitingImprovesProductivity1_2_3_4_5_6_71_2_3_4_5_6_7LiberatingReducesProductivity37. YOUR PARTICIPATION IN DIRECTING THE CCIS OPERATION: Your feeling ofinvolvementand commitment in operating and maintaining the CCIS Database.UninvolvedCommitted1_2_3_4_5_6_71_2_3_4_5_6_7InvolvedUncommitted38. THE CCIS DATABASE SAVES YOU TIME: The extent to which the use of the CCISDatabase saves you time.Time costlyOften1_2_3_4_5_6_71_2_3_4_5_6_7Time savingSeldom39. NEED FOR THE CCIS DATABASE: the extent to which the CCIS Database fulfils animportant need.UnnecessaryConsistent1_2_3_4_5_6_71_2_3_4_5_6_7NeededInconsistent40. YOUR CONFIDENCE IN THE CCIS DATABASE: Your feelings of and certainty about theDatabase.Doubtful confidence 1_2__3__4__5__6__7 Certain confidenceConsistent 1_2_3_4_5_6_7 Inconsistent41. YOUR UNDERSTANDING OF THE CCIS DATABASE: Your knowledge of andunderstanding about the Database.Limited knowledge 1_2_3_4_5__6_7Deep 1_2_3_4_5_6_7understandingBroad knowledgeShallowunderstandingIn this part of the questionnaire. you are askedto score the 41 factors on a relative importancescalewhich goes from extremely unimportant on the left to extremely important on the right. Mark thescales realizing that some factors are more important than others as they affect your satisfactionwith the eCls Database.Try to give your feelings concerning the relative importanceof each factor within each of the fivegroups (Input/outputfactors. Organizationfactors. etc.).ExtremelyunimportantExtremelyimportant1.Ease of Use2. Completenessof Screens3. Accuracy of Data4. Format of Output Screens5. Relevancy of Output Screens6. Availabilityof Answers7. Terminal's Response time8. Volume of Information9. Convenience of Access to Terminal10. Convenience of access to Printer11. Procedures for Logging On12.Procedures to receive output13. Currencyof Data14. Value of the Output1_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2__3_4_5_6__71_2_3_4_5_6_71_2_3_4_5_6__71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_7o.RGANIZATIOlS FACTORSExtremelyunimportantExtremelyimportant15.Hospital policies for MIS16. Hospital MIS involvement1_2_3_4_5_6_71_2_3_4_5_6_717.Feeling control over system18.Goal agreement19. Authority MIS within Hospital1_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_720. Authority of Chief Invest over software 1__2__3__4__5__6_721. Error Recovery Procedure 1_2_3_4_5_6_7ExtremelyunimportantExtremelyimportant22. Staff Interpersonal Skills23. Staff Technical Skills24. Attitudeof MIS Staff25. Staff health Care Knowledge26. Your relationship with Staff1_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5__6__71_2_3_4_5_6_7CQ~~UTER SYSTEM.,.FACTORSExtremelyunimportantExtremelyimportant27. Flexibility of System 1__2__3__4__5__6__728. Time and Process for Changes or New 1__2__3__4__5__6__7Developments to System29. Integration of the System 1__2__3__4__5__6__730. Capabilities of the System 1__2__3__4__5__6__731. Ease of Introducing System 1__2__3__4__5__6__732. Software support 1__2__3__4__5__6__733. Written Instructions on system use 1__2__3__4__5__6__734. Securityof Data 1__2__3__4__5__6__7ExtremelyunimportantExtremelyimportant35. Your degree of training36. Effects of CCIS Database on your work37. Yourpart in directing CCISDatabase38. CCIS Database saves you time39. Need for CCIS Database40. Your confidence in CCISDatabase41. Your understandingof CCIS Database1_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_71_2_3_4_5_6_7Describewhat you like most about the Computersystemand CCIS Database:Describewhat you like least about the Systemand CCIS Database:FINALLY, A FEW PERSONAL QUESTIONS TO IDENTIFY CHARAcrERISTICS OFINDIVIDUALS.1. Your present age?2. Your sex?__ years.__ MorF3. Year of medical school graduation? 19__4. Type of training in Emergency:FRCPC or CCFP (EM) Practice eligible (Y or N)Residency trainedOther Formal Training:5. Years of full time emergency practice: years.6. Site of your Emergency Department:7. Your prior familiarity with operating personal computers:No prior knowledge 1__2__3__4__5__6__7Access to home 1__2__3__4__5__6__7computerCompletely at easeNo contact withcomputersOnce again, thank you for your perseverance. I trust that the questions provoke thoughtfulreflection for you.THE END! THANK YOU!


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items