UBC Faculty Research and Publications

Perceptions of frontline staff regarding data collection methodologies used during the 2009 A H1N1 influenza… Foisy, Julie; Quach, Susan; Heidebrecht, Christine L.; Pereira, Jennifer A.; Quan, Sherman D.; Guay, Maryse; Bettinger, Julie A.; Deeks, Shelley L.; Brien, Stephanie; Kwong, Jeffrey C. 2010

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-Foisy_J_et_al_Perceptions_frontline_staff.pdf [ 154.37kB ]
JSON: 52383-1.0228247.json
JSON-LD: 52383-1.0228247-ld.json
RDF/XML (Pretty): 52383-1.0228247-rdf.xml
RDF/JSON: 52383-1.0228247-rdf.json
Turtle: 52383-1.0228247-turtle.txt
N-Triples: 52383-1.0228247-rdf-ntriples.txt
Original Record: 52383-1.0228247-source.json
Full Text

Full Text

RESEARCH ARTICLE Open AccessPerceptions of frontline staff regarding datacollection methodologies used during the 2009A H1N1 influenza immunization campaign inCanadaJulie Foisy1*, Susan Quach1, Christine L Heidebrecht1, Jennifer A Pereira1, Sherman D Quan2,Maryse Guay3,4,5,6, Julie A Bettinger7, Shelley L Deeks1,8, Stephanie Brien9, Jeffrey C Kwong1,8,10,11,for the Public Health Agency of Canada/Canadian Institutes of Health Research Influenza Research Network(PCIRN) Vaccine Coverage Theme Group*AbstractBackground: During the 2009 H1N1 immunization campaign, electronic and hybrid (comprising both electronicand paper components) systems were employed to collect client-level vaccination data in clinics across Canada.Because different systems were used across the country, the 2009 immunization campaign offered an opportunityto study the usability of the various data collection methods.Methods: A convenience sample of clinic staff working in public health agencies and hospitals in 9 provinces/territories across Canada completed a questionnaire in which they indicated their level of agreement with sevenstatements regarding the usability of the data collection system employed at their vaccination clinic. Questionsincluded overall ease of use, effectiveness of the method utilized, efficiency at completing tasks, comfort using themethod, ability to recover from mistakes, ease of learning the method and overall satisfaction with the method.A 5-point Likert-type scale was used to measure responses.Results: Most respondents (96%) were employed in sites run by public health. Respondents included 186 nursesand 114 administrative staff, among whom 90% and 47%, respectively, used a paper-based method for datacollection. Approximately half the respondents had a year or less of experience with immunization-related tasksduring seasonal influenza campaigns. Over 90% of all frontline staff found their data collection method easy to use,perceived it to be effective in helping them complete their tasks, felt quick and comfortable using the method,and found the method easy to learn, regardless of whether a hybrid or electronic system was used.Conclusions: This study demonstrates that there may be a greater willingness of frontline immunization staff toadapt to new technologies than previously perceived by decision-makers. The public health community shouldrecognize that usability may not be a barrier to implementing electronic methods for collecting individual-levelimmunization data.* Correspondence: julie.foisy@oahpp.ca1Department of Surveillance and Epidemiology, Ontario Agency for HealthProtection and Promotion, Toronto, CanadaFull list of author information is available at the end of the articleFoisy et al. BMC Public Health 2010, 10:796http://www.biomedcentral.com/1471-2458/10/796© 2010 Foisy et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative CommonsAttribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction inany medium, provided the original work is properly cited.BackgroundCollecting individual-level information at the point ofvaccination enables timely assessment of vaccination cov-erage, effectiveness and safety at the population level [1].Vaccine providers must consider many factors for opti-mizing data collection, including available financial andhuman resources, and the impact that the approach willhave on users at all levels of the system. As technologyhas evolved and the value of electronic patient records isincreasingly recognized, electronic means of collectingand storing health data are becoming more viable. How-ever, usability constraints have been identified as a poten-tial obstacle to implementing novel technologies[2].During the 2009 A H1N1 influenza immunizationcampaign, paper, electronic, and hybrid systems wereemployed in clinics across Canada. Because differentsystems were used across the country, the 2009 immuni-zation campaign offered an opportunity to study theefficiency and usability of these data collection methods.The objective of this study was to determine the percep-tions held by frontline clinic staff of H1N1 immuniza-tion data collection methods used in Canada.MethodsSettingThe Public Health Agency of Canada/Canadian Institutesof Health Research Influenza Research Network (PCIRN)Vaccine Coverage Theme conducted an on-site assess-ment of pandemic immunization data collection among asample of public health agencies and hospitals that pro-vided influenza immunization across Canada from Octo-ber to December 2009. All public health jurisdictions inCanada were contacted by email and invited to partici-pate. The front-line staff perceptions survey was adminis-tered as part of a larger on-site assessment that includedobserving data collection methodologies, and measuringthe time spent by front-line immunization staff to recordpandemic immunization data. Results from the time andmotion study, a part of the larger on-site assessment,have been published elsewhere[3]. This report sum-marizes the questionnaire that was administered to clinicstaff to assess usability of the data collection methods.Details of the methods and results for the on-site assess-ment objectives have been described elsewhere[4]. Ethicsapproval was obtained from the University of Toronto’sHealth Sciences Research Ethics Board (REB) and otherjurisdictions’ REBs as required.Questionnaire DevelopmentThe IBM Computer System Usability Questionnaire(CSUQ) is a validated 19-item instrument that was ori-ginally developed to measure users’ satisfaction with theusability of computer systems in non-laboratory settings[5]. We modified the questionnaire and retained 7 ques-tions that were directly applicable to both paper andelectronic immunization collection systems. Questionsincluded overall ease of use, effectiveness of the methodutilized, efficiency at completing tasks, comfort usingthe method, ability to recover from mistakes, ease oflearning the method and overall satisfaction with themethod. Using a Likert-type scale, response choicesincluded strongly disagree, disagree, neither agree nordisagree, agree, strongly agree and not applicable. Addi-tional questions addressed respondent’s position,responsibilities, number of years with said responsibil-ities, and location of clinic. A copy of the questionnairecan be found in the additional files.Administration of the QuestionnaireThe questionnaire was completed by a conveniencesample of frontline staff at immunization clinics thatwere visited between October 27 and December 17,2009. Participants were approached by a member of theresearch team and asked to provide informed consentprior to being observed, as part of the on-site assess-ment. Participants were also asked to complete thequestionnaire at any point during the day, with assur-ance of the anonymity of their responses. The question-naire was then given to the participants and collectedfrom them by the end of the day.Statistical AnalysisSince participants generally performed a limited range oftasks in the immunization process (e.g., administrativestaff may have only registered clients using an electronicmethod, while nurses performed immunization andrelated documentation using a paper method), question-naire responses were dichotomised according to thedata collection method used for the task(s) (i.e., electro-nic vs. paper). Analyses consisted of frequency distribu-tions. Statistical significance testing was conducted usingFisher’s exact test to identify differences between thoseusing electronic and paper methods (all staff combined,nurses and administrative clerks separately). Exact 95%confidence limits were calculated. Analyses were per-formed using STATA version 10.0[6].ResultsOf the 165 organizations contacted across Canada, 38(23%) (with 79 physically distinct immunization clinicsites across Canada) agreed to participate and 300 front-line staff responded to the survey. The number of staffwho refused to respond to the survey was not tracked;however it is believed to be very few.The characteristics of the participating frontline staffcan be found in Table 1. The majority of respondentsFoisy et al. BMC Public Health 2010, 10:796http://www.biomedcentral.com/1471-2458/10/796Page 2 of 5were employed in sites operated by public health (96%).Respondents included 186 nurses and 114 administrativestaff, among whom 90% and 47% respectively used apaper method to perform data collection tasks. Over52% of respondents had a year or less of experiencewith immunization-related tasks during seasonal influ-enza campaigns; the number of years of experience ran-ged from 0 to 32.Frontline staff found the data collection system usedin their clinic, whether electronic or paper, to be highlyacceptable (Table 2). Among all frontline staff using anelectronic method, 96% felt comfortable with theapproach being used. Ninety-six percent of frontlinestaff using an electronic method and 94% of staff usinga paper method found the method easy to learn.Ninety-eight percent of electronic method users and91% of paper method users strongly agreed/agreed withthe statement: “Overall I am satisfied with this method”.No statistically significant differences were foundbetween users of electronic and paper methods (nurses,administrative clerks or all staff combined), however, weobserved a trend favouring electronic over paper meth-ods in the overall satisfaction question for all staff com-bined (p = 0.08).Table 1 Characteristics of respondentsCharacteristics n (%)Total Respondents 300 (100)Type of SitePublic Health 287 (95.7)Hospital 13 (4.3)Method UsedElectronic 80 (26.7)Paper 220 (73.3)Nurses 186 (100)Using electronic methods 19 (10.2)Using paper methods 167 (89.8)Median years of experience with immunization task (range) 6 (0 - 32)Administrative clerks 114 (100)Using electronic methods 61 (53.5)Using paper methods 53 (46.5)Median years of experience (range) 1 (0 - 29)Table 2 Perceptions of frontline staff by position and methodNurses Administrative clerks All staff combinedStrongly agree/agree %(95% CI)Strongly agree/agree %(95% CI)Strongly agree/agree %(95% CI)Statement ElectronicMethod(n = 19)PaperMethod(n = 167)ElectronicMethod(n = 61)PaperMethod(n = 53)ElectronicMethod(n = 80)PaperMethod(n = 220)It was easy to use this data collection method. 94.7%(74.0-99.9)91.0%(85.6-94.9)96.7%(88.7-99.6)84.9%(72.4-93.3)96.3%(89.4-99.2)89.5%(84.7-93.3)I could effectively complete my tasks using this method. 100%(82.4-100)94.0%(89.3-97.1)96.7%(88.7-99.6)92.5%(81.8-97.9)97.5%(91.3-99.7)93.6%(89.6-96.5)I was able to complete my tasks quickly using this method. 94.7%(74.0-99.9)89.2%(83.5-93.5)93.4%(84.1-98.2)84.9%(72.4-93.3)93.8% 86.0-97.9)88.2%(83.2-92.1)I felt comfortable using this method. 94.7%(74.0-99.9)95.2%(90.8-97.9)96.7%(88.7-99.6)90.6%(79.3-96.9)96.3%(89.4-99.2)94.1%(90.1-96.8)It was easy to learn to use this method. 89.5%(66.9-98.7)93.4%(88.5-96.7)98.4%(91.2-100)96.2%(87.0-99.5)96.3%(89.4-99.2)94.1%(90.1-96.8)Whenever I make a mistake using this method, I canrecover easily and quickly.84.2%(60.4-96.6)87.4%(81.4-92.0)93.4%(84.1-98.2)90.6%(79.3-96.9)91.3%(82.8-96.4)88.2%(83.2-92.1)Overall, I am satisfied with this method. 100%(82.4-100)91.6%(86.3-95.3)96.7%(88.7-99.6)86.8%(74.7-94.5)97.5%(91.3-99.7)90.5%(85.8-94.0)Foisy et al. BMC Public Health 2010, 10:796http://www.biomedcentral.com/1471-2458/10/796Page 3 of 5DiscussionOur study indicates that frontline workers are highlysatisfied with the data collection methods used at vacci-nation clinics regardless of whether it was paper-basedor electronic. Over 90% of all frontline staff found theirdata collection method quick and easy to use, perceivedit to be effective in helping them complete their tasks,felt comfortable using the method, and found it easy tolearn.In a national study conducted prior to the H1N1 cam-paign[7] decision makers identified training of frontlinestaff as a perceived barrier to implementing electronicmethods of collecting immunization data. Familiaritywith a system and increased training raise users’ accept-ability of a novel system[8]. One of the electronic sys-tems used in Canada - and the one used by the majorityof observed organizations - was developed and rolledout in a very short period of time. Although usersreceived training for this new system, it may not havebeen as extensive as it would have been during a regularinfluenza season due to time constraints associated withthe urgent vaccine delivery schedule. Our results showthat although most users had minimal experience withthis electronic system, they nonetheless found the com-puter-based tasks highly acceptable. In a separate survey,69% of nurses indicated that they had received adequatetraining prior to use of this novel system[9].This study had several limitations. First, although theIBM survey has been validated for new computer usersit may not be directly applicable to users of paper-basedsystems. It has also not been validated specifically toassess usability of immunization data collection methodsby healthcare staff. Many may not have used both paperand electronic methods and therefore would not havebeen in a position to directly compare them. Finally, thequestionnaire was only completed by frontline staff whowere observed for the time and motion study, whichmainly consisted of those using a paper method.Because of this limited sample, our comparisons mayhave inadequate power to detect a true differencebetween methods.ConclusionThis study suggests that there may be a greater willing-ness of frontline immunization staff to adapt to new tech-nologies than previously perceived by decision-makers.The high acceptability for both electronic and paper-based methods illustrates that frontline staff are contentwith either method, regardless of the novelty. The publichealth community should recognize that usability maynot be a barrier to implementing electronic methods forcollecting individual-level immunization data. Utilizingelectronic methods for collecting individual-level dataoffers the possibility for data to be analyzed and appliedquickly for decision-making purposes, which could resultin timely assessment of vaccine coverage, effectivenessand safety.Additional materialAdditional file 1: User Perceptions Questionnaire. A copy of the “UserPerceptions Questionnaire” that was used for this study.AcknowledgementsPCIRN Vaccine Coverage Theme Group members are: David Allison, JulieBettinger, Nicole Boulianne, Stephanie Brien, David Buckeridge, LarryChambers, Natasha Crowcroft, Shelley Deeks, Michael Finkelstein, Julie Foisy,Effie Gournis, Maryse Guay, Jemila Hamid, Christine Heidebrecht, DonnaKalailieff, Faron Kolbe, Jeff Kwong, Allison McGeer, Jane Nassif, JenniferPereira, Susan Quach, Sherman Quan, Beate Sander, Chris Sikora, and DonWillison.This study was supported by an operating grant from the Public HealthAgency of Canada and the Canadian Institutes of Health Research. TheCanadian Association for Immunization Research and Evaluation providednetworking assistance. We are grateful for the contributions and support ofthe individuals at participating immunization clinics, and thank Angela Maskand Radha Shah for their assistance with data collection.Author details1Department of Surveillance and Epidemiology, Ontario Agency for HealthProtection and Promotion, Toronto, Canada. 2University Health Network,Toronto, Canada. 3Département des sciences de la santé communautaire,Université de Sherbrooke, Longueuil, Canada. 4Institut national de santépublique du Québec, Montréal, Canada. 5Agence de la santé et des servicessociaux de la Montérégie, Longueuil, Canada. 6Centre de recherche del’Hôpital Charles LeMoyne, Longueuil, Canada. 7University of BritishColumbia, Vaccine Evaluation Center, BC Children’s Hospital, Vancouver,British Columbia, Canada. 8Dalla Lana School of Public Health, University ofToronto, Toronto, Canada. 9Department of Epidemiology, Biostatistics, andOccupational Health, McGill University, Montreal, Canada. 10Institute forClinical Evaluative Sciences, Toronto, Canada. 11Department of Family andCommunity Medicine, University of Toronto, Toronto, Canada.Authors’ contributionsAll authors were involved with the study design. JF, SQ, CLH and JAPcollected the data. JF and SQ analyzed the data. JF drafted the manuscriptwith contributions from CLH, JAP, SQ, SDQ, SLD, MG, JAB, SB, and JCK. Allauthors read and approved the manuscript.Competing interestsThe authors declare that they have no competing interests.Received: 30 August 2010 Accepted: 30 December 2010Published: 30 December 2010References1. Writing team for the Public Health Agency of Canada/Canadian Institutes ofHealth Research Influenza Research Network Vaccine Coverage ThemeGroup: Why collect individual-level vaccination data? CMAJ 2010,182:273-275.2. Welker JA: Implementation of electronic data capture systems: Barriersand solutions. Contemporary Clinical Trials 2007, 28(3):329-336.3. Quach S, Hamid JS, Pereira JA, Heidebrecht CL, Foisy J, Bettinger JA,Rosella L, Crowcroft NS, Deeks SL, Quan SD, Finkelstein M, Guay M,Buckeridge DL, Sikora CA, Kwong JC: Time and motion study to compareelectronic and hybrid data collection systems during the pandemic(H1N1) 2009 influenza vaccination campaign. Vaccine 2010.4. Pereira JA, Quach S, Heidebrecht C, Foisy J, Quan S, Finkelstein M,Sikora CA, Bettinger JA, Buckeridge DL, McCarthy A, Deeks SL, Kwong JC:Foisy et al. BMC Public Health 2010, 10:796http://www.biomedcentral.com/1471-2458/10/796Page 4 of 5Pan-Canadian Assessment of Pandemic Immunization Data Collection:Study Methodology. BMC Medical Research Methodology 2010, 10:51.5. Lewis JR: IBM Computer Usability Satisfaction Questionnaires:Psychometric Evaluation and Instructions for Use. IBM Corporation; 1993[http://drjim.0catch.com/usabqtr.pdf], Retrieved March 19, 2010.6. StataCorp: Stata Statistical Software: Release 10 College Station, TX:StataCorp LP; 2007.7. Heidebrecht CL, Foisy J, Pereira JA, Quan S, Willison DJ, Deeks SL,Finkelstein M, Crowcroft NS, Buckeridge DL, Guay M, Sikora CA, Kwong JC,for the Public Health Agency of Canada/Canadian Institutes of HealthResearch Influenza Research Network (PCIRN) Vaccine Coverage ThemeGroup: Perceptions of Immunization Information Systems for CollectingPandemic H1N1 Immunization Data within Canada’s Public HealthCommunity: A Qualitative Study. BMC Public Health 10:523.8. Yen PY, Gorman P: Usability testing of digital pen and paper system innursing documentation. AMIA Annual Symposium Proceedings 2005, 844-8.9. Nassif J, Gentry A, Biletchi J: Protocol for Electronic Clinic Systems SurveyResults. Niagara Region Health Department 2010.Pre-publication historyThe pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/10/796/prepubdoi:10.1186/1471-2458-10-796Cite this article as: Foisy et al.: Perceptions of frontline staff regardingdata collection methodologies used during the 2009 A H1N1 influenzaimmunization campaign in Canada. BMC Public Health 2010 10:796.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionSubmit your manuscript at www.biomedcentral.com/submitFoisy et al. BMC Public Health 2010, 10:796http://www.biomedcentral.com/1471-2458/10/796Page 5 of 5


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items