Medicine, Faculty of
Pediatrics, Department of
DSpace
B.C. Injury Research and Prevention Unit
Raina, Parminder
Soubhi, Hassan
Turcotte, Kate
Brussoni, Mariana
Janssen, Patricia A.
Hotte, Alan
This report evaluates the status of the Injury Surveillance Pilot Project (ISPP) based upon the framework established in the first ISPP Evaluation Report. Brief background information is provided and project goals related to data collection, electronic data storage, data analysis, information dissemination, and information use are addressed through a series of evaluation questions. Methods and/or steps are described and findings are presented and discussed. Finally, highlights from this evaluation are summarized, quality of the data discussed, limitations of the evaluation presented, and next steps for Emergency Department Injury Surveillance System (EDISS) outlined.
INJURY SURVEILLANCE PILOT PROJECT Evaluation Report Prepared by: Team Members: Parminder Raina Hassan Soubhi Kate Turcotte Mariana Brussoni Patti Janssen Alan Hotte March 2002 2Acknowledgements ISPP Evaluation Report: Christa Gross - Evaluation Report Lise Olsen - Key Informant Survey Herb Chan - Data Report Evaluation & Needs Assessment, Key Informant Survey Clarissa Tufts - Key Informant Survey Douglas College Health Information Students - Chart Audits Collaborating Centres & Contacts: South Fraser Health Region - Kerry Yoshitomi Langley Memorial Hospital - Karen Hatchwell Delta Hospital - Eloise Irving Peace Arch Hospital (White Rock) - Moira Riddell Surrey Memorial Hospital - Rosalie Chin North West Health Services Society - Ester Brisch Prince Rupert Regional Hospital - Carol Lindsay Mills Memorial Hospital (Terrace) - Teresa Ward Kitimat General Hospital - Grace O'Connor Bulkley Valley District Hospital (Smithers) - Kelly Abrams Thompson Health Region - James Lu Royal Inland Hospital (Kamloops) - Brenda Nadeau Lillooet District Hospital - Brenda Nadeau ISPP Steering Committee: Susan Mackenzie - Senior Epidemiologist, Injury Section, Health Surveillance and Epidemiology Division, Health Canada Mariana Brussoni - Associate Director, BCIRPU Caroline da Silva - Project Officer, Centre for Surveillance Coordination, Health Canada Al Erlenbusch - (retired) Senior Manager, Emergency Health Services Branch, Ontario Ministry of Health Alan Hotte - Project Manager, Centre for Surveillance Coordination, Health Canada Parminder Raina - Director, BCIRPU Hassan Soubhi - Post-Doctorate Fellow, BCIRPU Bryon Taylor - Office of Injury Prevention, BC Ministry of Health Planning Kate Turcotte - Project Co-ordinator, ISPP, BCIRPU Yau Yip - consultant, Emergency Health Services Branch, Ontario Ministry of Health Kerry Yoshitomi - Regional Co-ordinator, Quality Improvement & Clinical Information, South Fraser Health Region, Peace Arch Hospital Funding for the ISPP Evaluation provided by: Injury Surveillance, Health Canada Office of Injury Prevention, BC Ministry of Health Planning Participating Hospitals and Health Regions 3Table of Contents LIST OF ABBREVIATIONS................................................................................................. 4 INTRODUCTION.................................................................................................................. 5 BACKGROUND ..................................................................................................................................................................7 Injury Surveillance Pilot Project (ISPP).....................................................................................................................7 Emergency Department Injury Surveillance System (EDISS).....................................................................................8 ISPP Pilot Sites ...........................................................................................................................................................9 ISPP Process Details ................................................................................................................................................11 ISPP EVALUATION........................................................................................................... 17 METHODS AND FINDINGS ...............................................................................................................................................18 1.0 Data Collection Questions ................................................................................................................................20 2.0 Data Integration Questions...............................................................................................................................47 3.0 Data Analysis & Interpretation Questions........................................................................................................54 4.0 Dissemination Questions...................................................................................................................................56 DISCUSSION....................................................................................................................................................................81 Support for the ISPP .................................................................................................................................................81 Data Quality..............................................................................................................................................................81 Summary of the Obstacles Encountered....................................................................................................................83 Limitations of the Evaluation ....................................................................................................................................84 Recommendations for the ISPP.................................................................................................................................84 REFERENCES................................................................................................................... 87 APPENDIX I....................................................................................................................... 88 ISPP OPERATING BUDGET PERIODS 1 & 2......................................................................................................................88 APPENDIX II ...................................................................................................................... 89 EVALUATION QUESTIONS DEVELOPMENT AND WORKSHEETS........................................................................................89 APPENDIX III ..................................................................................................................... 99 FINANCIAL SUMMARY ....................................................................................................................................................99 APPENDIX IV .................................................................................................................. 103 HEALTH RECORDS PERSONNEL FORM ..........................................................................................................................103 APPENDIX V ................................................................................................................... 106 CHART AUDIT FORM ....................................................................................................................................................106 APPENDIX VI .................................................................................................................. 108 KEY INFORMANT SURVEYS: PARTS I & II ....................................................................................................................108 APPENDIX VII ................................................................................................................. 113 COMPLETENESS OF CHARTING CHART AUDIT RESULTS ...............................................................................................113 APPENDIX VIII ................................................................................................................ 118 SENSITIVITY AND SPECIFICITY CHART AUDIT RESULTS ...............................................................................................118 4LIST OF ABBREVIATIONS Abbreviation Meaning BC British Columbia BC EDISS (= EDISS) British Columbia Emergency Department Injury Surveillance System BCIRPU BC Injury Research and Prevention Unit BCIRS British Columbia Injury Reporting System BVDH Bulkley Valley District Hospital CEO Chief Executive Officer (for hospitals) CHSPR Centre for Health Services and Policy Research (UBC) CI Confidence Interval (95%) CIHI Canadian Institute for Health Information DH Delta Hospital ED Emergency Department EDISS (= BC EDISS) Emergency Department Injury Surveillance System FTE Full Time Equivalent HDW Health Data Warehouse (Ministry of Health Planning) HR Health Records HRA Health Records Administrator HRT Health Records Technician ICD-9 International Classification of Disease, version 9 ICD-10 International Classification of Disease, version 10 IP In Progress IS Information Systems ISPP Injury Surveillance Pilot Project KGH Kitimat General Hospital LDH Lillooet District Hospital LHA Local Health Area LMH Langley Memorial Hospital MMH Mills Memorial Hospital (Terrace) NACRS National Ambulatory Care Reporting System NWHSS North West Health Services Society PAH Peace Arch Hospital (White Rock) PHN Personal Health Number PPA Percentage Positive Agreement PRRH Prince Rupert Regional Hospital PYLL Potential-Years-Life-Lost RIH Royal Inland Hospital (Kamloops) SFHR South Fraser Health Region SMH Surrey Memorial Hospital THR Thompson Health Region UBC University of British Columbia 5INTRODUCTION Injuries are costly to society and to the health care system. In 1993, the total economic burden to Canadians due to intentional and unintentional injury was 11.1 percent of the total costs of illness, or $14.3 billion (Health Canada, 1997). Only cardiovascular and musculo-skeletal disease exceeded injury in total cost. Even costs associated with cancer were smaller than the economic burden of injury. The total cost of unintentional injuries in Canada in 1994 was estimated to be over $8.7 billion (Angus et. al., 1998). These numbers do not reflect the additional human and emotional costs for the injured individuals and their families. In British Columbia (BC), unintentional injuries alone cost $2.1 billion per year in direct and indirect costs (Cloutier & Albert, 2001). On average almost 200 young people aged 0 – 24 years die from unintentional injuries each year (BC Vital Statistics Agency, 1999). Another 11,000 are hospitalized as a result of injuries and tens of thousands more are treated at Emergency Departments (EDs) and released. Unintentional injuries were the sixth leading cause of death for all age groups in 2000 (BC Vital Statistics Agency, 2001). Further, the average potential-years-life-lost (PYLL)1 due to injury was 34.1 years for all of BC, compared to 6.1 years for those who died of coronary and circulatory diseases and 12.3 years for those who died of cancer. Injury research funding accounted for 1.2 percent of the total health science research expenditures in 1993 (Health Canada, 1997). While injury is the third largest contributor to the total burden of illness, it ranks last in research support dollars. The Injury Surveillance Pilot Project (ISPP) is a joint federal and provincial initiative that begins to address the problem of injuries. At the provincial level, the BC Injury Research and Prevention Unit (BCIRPU), a research unit partly funded by the BC Ministry of Health Planning, developed the Emergency Department Injury Surveillance System (EDISS) 2 as part of its broader mandate of injury surveillance as illustrated by the Injury Prevention and Evaluation Cycle. The Injury Prevention and Evaluation Cycle is a framework which describes the steps required to answer the following questions as well as highlighting the role of data:  What is the link between data and prevention?  What information is needed before planning, implementing or evaluating an injury prevention program?  What kind of data is needed? 1 PYLL denotes the total number of years of life lost from an established life expectancy of 75 years of age. 2 BCIRPU was established in 1997 and is accountable to the British Columbian Injury Prevention Leadership Council and the Ministry of Health Planning’s Office for Injury Prevention. BCIRPU is housed within the BC Children’s Hospital, and is supported by the BC Ministry of Health Planning, the BC Research Institute for Children’s and Women’s Health, the Centre for Community Child Health Research, and external grants. BCIRPU is meant to bring together the many organizations, communities, and individuals who work towards reducing the burden of injuries in BC under a common mission. 6 Where can these data be found and is it reliable information?  How is success monitored? At the federal level, Health Canada was similarly interested in gathering and understanding injury data to direct injury prevention activities. Thus, Health Canada and BCIRPU collaboratively initiated the ISPP with EDISS at its core. The ISPP began implementation in December 2000. It builds on BCIRPU’s developmental work with the EDISS program. BCIRPU, the participating Hospitals and Health Regions, and Health Canada jointly contribute resources for the implementation of the ISPP. This report evaluates the status of the ISPP.3 Background information is provided about the EDISS program. Project goals related to data collection, electronic data storage, data analysis, information dissemination, and information use are addressed through a series of evaluation questions. Methods and/or steps are described and findings are presented and discussed. Finally, support for the ISPP is highlighted, quality of the data summarized, limitations of the evaluation presented, and recommendations for action based on the evaluation are listed. 3 The ISPP evaluation is based on an evaluation model developed by the Ottawa Carleton Health Department, called the Program Evaluation Tool Kit (Porteous, Sheldrick, & Stewart, 1997). Burden ofInjuryRisk Factors &Conditions Effectiveness ff tiEfficiencyffi iPlanning &Implementationl i I l t tiMonitoringit riReassessmenttDATAInjury Prevention & Evaluation Cycle 7Background Injury Surveillance Pilot Project (ISPP) Historically, the majority of information regarding injuries is derived from mortality and hospitalization data. This information does not reflect the full extent of the injury problem because most people who are injured do not die from their injuries or become inpatients in hospitals, as demonstrated by the Injury Pyramid. Several population-based studies indicate that 78 to 87 percent of all medically attended injuries are treated in EDs (Survey Research Center, 1993; Ribbeck et. al., 1992). For example, in BC's Thompson Health Region (THR) there were 53 injury-related deaths, 1,792 injury-related hospitalizations, and 14,389 injury-related ED visits reporting to two of the regions hospitals from April 1999 to the end of March 2000 (BCIRPU, 2001). The majority of these injuries are preventable. Since ambulatory treatment accounts for a large proportion of health care services, and can provide local data needed to set local priorities, a critical step in the prevention of injuries is to collect injury data in hospital EDs. The goals of ISPP are to:  Implement data collection, including provision of training and support to Emergency Department Admitting, Nursing and Physician staff, and to Health Records personnel to chart, abstract and code a minimum data set for Emergency Department patients for which the visit is the first visit for treatment of that injury;  Establish and manage electronic storage of injury data;  Analyze and interpret data on a regular basis;  Disseminate information; and  Use information leading towards injury prevention. Injuries treated at home or injuries that were not treatedInjuries requiring hospitalizationInjuries resulting in deathInjuries resulting in treatment in emergency depts. & physicians office 8Emergency Department Injury Surveillance System (EDISS) BCIRPU’s Emergency Department Injury Surveillance System (EDISS) is a multifaceted program that serves as the core of the ISPP. EDISS was conceived as a two-stage surveillance system. The first stage is to provide general injury surveillance of the population to quantify the burden of injury presenting to the EDs, and to identify trends. The second stage is to conduct detailed resource intensive surveillance of a specific trend as identified in the first stage. BCIRPU did the early development of the EDISS initiative from 1997 to 1999. Regionally, this involved working with local champions of injury prevention and establishing working groups. BCIRPU developed a minimum data set, aimed at collecting the maximum amount of useful information with the minimum amount of burden to ED and Health Records (HR) staff. Needs assessments were conducted at each of the participating hospitals to evaluate the resources required to chart, abstract, and code the minimum data set. In collaboration with the working groups, commitment from the Health Regions was secured and internal and external funding was obtained. During 1999 and 2000, Information Systems (IS) and resource requirements specific to each hospital were outlined. Data availability and access was determined, and operating agreements were outlined. In order to establish an integrated data collection system, the requirements, resources and challenges of each institution were considered. The EDISS initiative standardizes abstraction and coding of injury data in all participating hospitals. The EDISS database supplements existing, but limited, electronic information regarding injuries that is routinely collected from mortality and hospitalization data in BC. Additionally, it is designed to be compatible with the emerging National Ambulatory Care Reporting System (NACRS)4. By using EDISS to add ED data to the existing data set, a more comprehensive assessment of injury and its causes is possible. 4 The Canadian Institute of Health Information (CIHI) developed the National Ambulatory Care Reporting System (NACRS) to abstract and report ambulatory care and emergency department data. NACRS was adopted by Prince Rupert Regional Hospital in BC in 1997; was mandated in Ontario in 1999; and received letters of intent from the North Okanagan Health Region in BC and from Prince Edward Island in 2000. NACRS is scheduled to undergo revision and conversion from ICD-9 to ICD-10 in 2002. Many institutions in BC have expressed interest in adopting NACRS. 9ISPP Pilot Sites Three Health Regions in BC committed to participating in the ISPP and implementing EDISS5. They include the South Fraser Health Region (SFHR), the North West Health Services Society (NWHSS), and the Thompson Health Region (THR). Within these three regions, ten hospitals participated in the project. The SFHR includes Langley Memorial Hospital (LMH), Delta Hospital (DH), Peace Arch Hospital (PAH) in White Rock, and Surrey Memorial Hospital (SMH). Hospitals from the NWHSS are Prince Rupert Regional Hospital (PRRH), Mills Memorial Hospital (MMH) in Terrace, Kitimat General Hospital (KGH), and Bulkley Valley District Hospital (BVDH) in Smithers. Finally, Royal Inland Hospital (RIH) in Kamloops and Lillooet District Hospital (LDH) comprise the THR’s participants. 5 Health Regions underwent restructuring and realignment in December 2001. The newly defined Health Authorities are not reflected in this report. 10As regions in the province increasingly take responsibility for local heath care resources, including health promotion and illness prevention, these three regions anticipate using the information from this pilot project to plan and evaluate their injury prevention initiatives. Regionalization implies coordination of resources and services within communities that have previously been dependent on provincial mandates. Regions across the province, however, differ vastly on important determinants of injury such as population demographics, occupations, climate, and geography. Initiatives that are focused on local issues have the potential to be more relevant, timely, and cost-effective. One of the drivers of the ISPP is that pre-event injury information is needed to direct prevention activities (Mackenzie, 1998). Information is also needed regarding injuries needing medical attention yet less severe than those leading to death or hospitalization. Ambulatory treatment accounts for a large proportion of health care services, and can provide local data needed to set local priorities. The following illustrates the approximate total number of ED visits per year for each participating hospital, the number of ED injury visits per hospital, and the percentage of ED injury visits each hospital is expected to contribute to the EDISS database during 2001.6 Hospital – Region # of ED visits / year # of ED injuries / year % of ED visits / EDISS Langley – SFHR 43,113 10,452 Delta – SFHR 21,809 6,404 Peace Arch – SFHR 32,285 5,546 Surrey – SFHR 71,483 17,156 SFHR Total Prince Rupert – NWHSS 17,566 3,622 Mills Memorial – NWHSS 2,220 Kitimat – NWHSS 5,997 1,445 Buckley Val. – NWHSS 13,000 2,353 NWHSS Total Royal Inland – THR 38,363 11,348 Lillooet Dist. – THR 2,692 924 THR Total 6 The following numbers provided by the Health Regions are only an estimate of the total numbers of ED visits and ED injury visits per year as submitted to BCIRPU. 11ISPP Implementation Plan The ISPP implementation plan includes five stages, the last of which has been extended until the end of March 2003. The stages are listed in the table below. The operating budget for the implementation is detailed in Appendix I. Stage Timing Details Stage 1 Dec 2000 – Mar 2001  Implement the minimum data set within the data collection system of Emergency Departments  Standardize the process of data collection, coding and extraction among hospitals.  Provide training and support to Emergency Department Admitting, Nursing and Physician staff and Health Records personnel  Receive test data from all sites Stage 2 Apr 2001 – Mar 2002  Collect pilot data  Analyze and interpret data Stage 3 Sep 2001 onwards  Disseminate descriptive analyses and interpretation to a wide variety of community stakeholders and injury prevention professionals  Collaborate with communities to target injury prevention initiatives and ensure that data is adequately informing program planning and prevention Stage 4 Sep 2001 – Mar 2002  Evaluate short-term processes and outcomes – disseminate findings to all partners including community partners, provincial and national audiences Stage 5 Apr 2002 – Mar 2003  Continue to collect Emergency Department injury surveillance data  Develop protocol for long-term information dissemination (implementation and uptake plan)  Monitor uses of information generated by Emergency Department injury surveillance data  Evaluate longer-term processes and outcomes ISPP Process Details Minimum Data Set Mackenzie (1998) defines a minimum data set for injury surveillance as "a process that monitors injury occurrence in a population using the smallest possible number of variables". A minimum data set must have enough variables and enough detail in coding to capture useful information. Thirteen data elements were selected for the EDISS data collection. 12The EDISS minimum data set includes both demographic and injury-event data. Demographic data elements are comprised of: Chart Number, Personal Health Number (PHN), Postal Code, Sex, and Date of Birth. Injury-event data includes: Date of Visit, Main Diagnosis, Other Diagnosis, Cause of Injury, Place of Occurrence, Activity When Injured, a Narrative Statement describing the injury event and Visit Disposition. Case Definition A case is identified if the reason for accessing the ED is any type of injury, intentional or unintentional, for all age groups, and for which the visit is the first visit for treatment of that injury. Examples of injuries include: blunt or penetrating, electrical or radiation, aspiration of foreign objects, drowning, blast or explosion injuries, adverse drug reactions, ingestion of foreign objects, adverse effects of medical care, burns, stings, bites, or acute poisonings. Exclusions include: visits for acute illnesses or disease, a medical check-up or a follow-up treatment, or non-acute poisoning. Examples of non-injuries include: pain with no evidence of acute injury, chronic alcoholism or drug abuse, anxiety or chronic depression, fever attributed to cold or viral infection, plaster/cast control, food poisoning, emotional trauma, infection, chronic back pain, epilepsy, heart attack, suture removal, contact dermatitis, chronic poisoning, and repetitive strain injury. Data Sources Patient identification and injury-event information is self-reported by patients presenting to the ED for treatment. The ED physicians determine main and other diagnoses, as well as the visit disposition at the end of the ED visit. Quality Control Chart audits are done to ensure reliable abstraction and coding of data during the pilot project. Logbooks kept in the HR Departments record the amount of time invested in data coding, as well as any problems or concerns. Data Management Data are transferred to BCIRPU by email, cleaned by the Database Manager and merged with data from other hospitals into a relational database (Microsoft Access). 13Data Collection Flow Chart Patients admitted to hospital from the ED may or may not be part of this database, depending on hospital protocol. Patient presents to Emergency Department Is this an injury? No Emergency Department nurses and physicians chart details and diagnoses Health Records identify/verify injury cases Admitting staff record personal identifiers Emergency Department nurses and physicians chart details of injury-event and diagnoses Charts transferred to Health Records department Minimum data set abstracted & coded into hospital softwareMinimum data set downloaded to a spread-sheet, Chart Number and Personal Health Number are scrambled, and data set transmitted to BCIRPU Yes Are injury charts flagged? Yes No First presentation for this injury? No Yes 14Database Data Entry Flow Chart *The Accumulating Formula table contains a list of formulas that code raw data elements into an aggregated form. By implementing this table and restricting the formulas, the process is run on the Import Database, analyzing only the new data as they are entered into the Main Database. As the Main Database grows, this table will maintain a statistical analysis of the data without decreasing performance of the database. Hospital data file received at BCIRPU File loaded into Import DatabaseAutomatic data testing for errors & omissions Failed records transferred to Error Database Manager can review records and correct or override errors, and transfer back to Import databaseClean data passed through Accumulating Formula* Bad records stored for future correctionsData transferred to Main Database Records corrected at hospital Bad records returned to hospital for correction 15Data Analysis Data are analyzed at BCIRPU by the ISPP project staff, consisting of three researchers (PhD level), the Project Co-ordinator (Masters level) and a Data Analyst (Masters level). Data analysis is descriptive. The analysis is automated within the database using Access and Excel to produce charts such as:  Injury by age group and sex  Leading Causes of Injury by age group and sex  Leading Natures of Injury by age group and sex  Place of Occurrence by age group and sex (using ICD-10 coding as standard or using the BCIRPU coding if available)  Activity When Injured by age group and sex  Visit Disposition by age group and sex  Cross tabulations  Distribution of injuries by time  Distribution of injuries by geographical region Privacy and Confidentiality The information is collected pursuant to sections 26(c) and sections 27(1)(b) of the Freedom of Information (FOI) Act. The information relates directly to and is necessary for an operating program or activity of the public body. The data are necessary for BCIRPU to conduct appropriate injury monitoring to guide injury prevention activities. BCIRPU collects information from hospitals, which already report hospital separation data for injury prevention. Authorization of data collection was obtained from the hospitals through a signed agreement that complies with the requirements of FOI Act rules regarding disclosure of information. The participating hospitals are releasing data under section 33(c) to promote overall population health and inform public health interventions. Data collected in the ED are sent to BCIRPU with the Chart Number and the Personal Health Number scrambled for confidentiality. Patients involved in this project are all cases of injuries presenting for the first time for treatment to EDs of the participating hospitals. At the level of the hospitals, current and accepted privacy procedures are maintained as for all routinely collected data. Data sent from the hospitals are accessed by the ISPP project staff including three BCIRPU researchers, the Project Co-ordinator, the Database Manager, and the Data Analyst. The database is password protected and stored on a single computer in a locked office. Passwords are renewed every three months. Raw data are not released by BCIRPU. Results of all descriptive analyses are presented at aggregate levels. Any geographical analyses reporting on fewer than five individuals are not disclosed to avoid any potential for case identification, in compliance with policies of the BC Privacy Commission. 16Legal Authority Legal authority with respect to the data resides with the participating hospitals and their corresponding regions, the Regional Health Boards, who in turn report to the BC Ministry of Health Services. The data are owned by the hospitals. Surveillance activities are limited to the EDs and the HR departments. Dissemination Reports are produced at both the hospital and regional levels. Hospital-level reports provide local injury data and may assist hospitals to assess the sustainability of injury surveillance and identify required system changes. Regional-level reports provide the regions with an introduction to the data available from EDISS. From this, more targeted injury events can be explored to support injury prevention initiatives. Injury surveillance activities that can be supported by the EDISS data include:  Measuring the local burden of injury  Identifying populations at high risk for injury  Identifying new and emerging causes of injury  Making regional comparisons of injury profiles  Assisting in setting priorities for injury prevention  Informing the development of injury prevention programs  Monitoring progress in injury prevention  Supporting planning of trauma care and rehabilitation  Evaluating measures designed to prevent and limit injury-related morbidity  Identifying opportunities for partnership/collaboration  Evaluating public health policy  Prioritizing allocation of health resources  Providing the basis for epidemiologic research  Providing data for use in advocacy  Allowing cost estimates  Identifying opportunities for cost saving Current updates regarding ISPP and EDISS are also disseminated via bimonthly newsletters as well as on the BCIRPU web-site (www.injuryresearch.bc.ca). Reports are disseminated to stakeholders and target groups including:  Health Regions (Medical Health Officers and Public Health Nurses)  Hospital strategic planning teams  ED Admitting, Nursing and Physician management  HR departments  Organizations delivering injury prevention programs 17ISPP EVALUATION An evaluation of the ISPP was undertaken from January to March 2002. The evaluation was performed approximately one year into the project. The next section details the evaluation purpose and the stakeholders involved. The purpose of the evaluation was to assess the implementation of the ISPP in its mission to describe the burden of injury among persons presenting at EDs in selected hospitals in British Columbia. Specifically, the evaluation assessed the following project goals:  Data collection implementation, including provision of training and support to Emergency Department Admitting, Nursing and Physician staff and Health Records personnel to chart, abstract and code a minimum data set for ED patients presenting for the first time for treatment of an injury;  Establishment and management of electronic injury data storage;  Regular analysis and interpretation of data;  Dissemination of information; and  Use of information for injury prevention. The stakeholders in the ISPP evaluation are the groups or individuals who are interested in the results or who will be using the evaluation information. Internal and external stakeholders are listed below: Internal Stakeholders External Stakeholders  Hospital EDs  Hospital HR Departments  Hospital IS Departments  BCIRPU  Health Canada  Ministry of Health Planning – Deputy Medical Officer of Health  Ministry of Health Planning – Office of Injury Prevention  Ministry of Health Planning – Regional Interface for Health Data  Health Region – Medical Health Officers  Health Region – Injury Prevention Coordinators  Hospital CEOs  Injury Prevention organizations and advocates 18Methods and Findings The ISPP evaluation is structured around a series of evaluation questions. The evaluation process began with the compilation of a list of evaluation questions related to data collection, data integration, data analysis and interpretation, and data dissemination. See Appendix II for details of the process for determining evaluation questions. In addition to the evaluation questions, specific evaluation tools were designed and tested to support various aspects of data collection. These include:  Financial Summary form for delineating start-up costs and annual costs (Appendix III);  Health Records Personnel form to track the number and type of staff hired for the project (Appendix IV);  Chart Audit Form to determine the sensitivity and specificity of EDISS, charting completeness, and database reliability (Appendix V); and  Key Informant Survey to gather general information about respondents’ knowledge of EDISS and to address issues raised in certain ISPP evaluation questions (Appendix VI). Key Informant Survey The Key Informant Survey, in particular, was central to gathering answers to a number of the evaluation questions. For this reason, a synopsis of the Key Informant Survey is presented at this point. The Key Informant Survey was comprised of two parts. Part I was completed by “first level” Key Informants including ED Admitting, Nursing and Physician staff and HR personnel. Fifty-six percent (33 of 59) of Key Informants completed Part I. “Second level” Key Informants included all Admitting Managers, Nursing Managers, Chiefs of ED, as well as informants at the regional level. This group was asked to complete both Part I and Part II of the Key Informant Survey. Forty-nine percent (21 of 43) of second level Key Informants responded accordingly. The overall response rate for the Key Informant surveys was 53 percent (54 of 102). Of these, 50 (93%) indicated that they had heard of the Emergency Department Injury Surveillance System, and 53 (98%) were aware that injury data are being collected from their hospital or region. Of all “first level” Key Informants, 27 of 33 (82%) identified themselves as playing a role in this data collection system. Overall, 48 of 54 (89%) Key Informants were familiar with the data elements being collected. 19Finally, 52 percent (28 of 54) Key Informants indicated that they were familiar with the Narrative Statement element of the data set, and 59 percent (32 0f 54) thought that it could provide information for injury prevention initiatives. Other responses from the Key Informant Surveys address Evaluation Questions posed in the following sections. Responses from the Key Informant Survey are presented in: Data Collection Questions 1.7.1 page 45 1.7.2 page 46 Dissemination Questions 4.6 page 72 4.7 page 74 4.8 page 76 4.10 page 79 Evaluation Questions There are a total of 26 questions considered in the ISPP evaluation. They address Data Collection, Data Integration, Data Analysis & Interpretation, and Dissemination. For each question, an expected outcome statement is presented; the methods and/or steps taken to answer the query are outlined; and the actual outcomes or findings are delineated. To decrease complexity, methods, findings and an optional discussion are combined under each question. 201.0 Data Collection Questions 1.1 Are hospitals able to break down the cost of implementing the system, including in-kind support, such that BCIRPU can accurately describe the resources and time involved for other agencies to join the Emergency Department Injury Surveillance System? Expected Outcome  A list of annual expenses for data collection, including ongoing training sessions and hiring of additional staff. Methods / Steps 1. Finalize Financial Summary form developed by SFHR Regional Director of Health Records. This covers both a start-up budget (three months) and an annual budget for running the system. 2. Distribute forms via e-mail to the Health Records Administrators of EDISS at SFHR, THR, PRRH, KGH, MMH, and BVDH. 3. Follow-up (1-week post dissemination) on the completion and submission of these forms via e-mail. 4. Follow-up (2-weeks post dissemination) on the completion and submission of these forms via telephone. 5. Enter data from forms into Excel software for summary. 6. Summarize information from forms. Findings Five of six Financial Summary forms were returned, from two regions and three hospitals respectively, for a response rate of 83 percent. A summary of start-up costs and ongoing annual costs are shown in the table on the following pages. The HR Manager at MMH was unable to provide Financial Summary information at this time because of current responsibilities and workload. This information will be provided at a later date. 21 ISSP Financial Summary Report March 2002 Start-Up Costs Ongoing Annual Costs SFHR – Langley Memorial, Surrey Memorial, Peace Arch, and Delta hospitals Training  HRA trainer – grade 3 1.0 FTE @ $62,261 / FTE $ 62,261- - Health Records Personnel  HRA – grade 1 abstracting, data analysis, submission, training, & quality checks LMH: 1.7 FTE DH: 1.0 FTE PAH: 0.8 FTE SMH: 2.4 FTE $ 23,658$ 13,916$ 11,133$ 33,399 HRA – grade 1 abstracting, data analysis, submission, training, & quality checks LMH: 1.7 FTE DH: 1.0 FTE PAH: 0.8 FTE SMH: 2.4 FTE  Administrative Assistant 0.6 FTE @ $44577 / FTE $ 94,629 $ 55,664 $ 44,532 $133,594 $ 26,746Information Systems (IS) IS setup of databases, routine for data submission  Hardware (procurement & installation) x 20 hrs  Extraction routines from admitting to abstracting systems x 30 hrs  Design / program encryption routines for data exchange x 30 hrs  General administration (meetings, telephone) x 10 hrs Total 90 hrs @ $40 / hr $ 3600  IS support of databases @ $40 / hr x 4 hr / mon $ 1,920Equipment / Other  Hardware / software - - - Total Start-Up Costs $147,967 Total Annual Costs $357,085 NWHSS – Prince Rupert Regional Hospital Training - - - - Health Records Personnel  HR level HRA x 0.3 FTE @ $58,383 / yr x 3 mon  HR Clerk x 0.2 FTE @ $48,238 / yr x 3 mon $ 4,379$ 2,412 HR level HRA x 0.3 FTE @ $58,383 / yr  HR Clerk x 0.2 FTE @ $48,238 / yr $ 17,515 $ 9,648 Information Systems (IS)  Hardware (procurement & installation) x 4 hrs  IS support of databases @ $35 / hr x 2 hr / mon $ 840 x 3 mon @ $55664 / FTEx12 mon @ $55664 / FTE 22ISSP Financial Summary Report March 2002 Start-Up Costs Ongoing Annual Costs  Extraction routines from admitting to abstracting systems x 4 hrs  Design / program encryption routines for data exchange x 4 hrs  General administration (meetings, telephone) x 4 hrs Total: 16 hrs @ $35 / hr $ 560Equipment / Other  Hardware & software – software upgrade $ 1,500- - Total Start-Up Costs $ 8,851 Total Annual Costs $ 28,003 NWHSS – Kitimat General Hospital Training  HR level HRA @ 0.05 FTE @ $54,000/yr x 3 mon $ 675 - - Health Records Personnel  HR level HRA x 0.5 FTE @ $54,000 / yr x 3 mon $ 675  HR level HRA x 0.05 FTE @ $54,000 / yr  HR level HRT x 0.1 FTE @ $48,000 / yr $ 2,700 $ 4,800 Information Systems (IS)  Hardware (procurement & installation) x 5 hrs  Design / program encryption routines for data exchange x 20 hrs  General administration (meetings, telephone) x 15 hrs Total: 40 hrs @ $28 / hr $ 1,120 IS support of databases @ $28 / hr x 2 hr / mon $ 672Equipment / Other  Hardware & software $ 1,070 - - Total Start-Up Costs $ 3,540 Total Annual Costs $ 8,172 NWHSS – Bulkley Valley District Hospital Training  HR level HRA Grade IV x 72 hrs @ $36 / hr $ 2,591 - - Health Records Personnel  HR level HRA x 0.6 FTE @ $__/ yr x 3 mon $ 2,888 HR Level HRT x 0.6 FTE @ 43,353 / yr $ 26,012 Information Systems (IS) IS functions done by Health Record / Information professional:  Hardware (procurement &  IS support to HR and Admitting is done by HRA Grade IV and is included in costing for HRT above - 23ISSP Financial Summary Report March 2002 Start-Up Costs Ongoing Annual Costs installation) x 3 hrs  Design / program encryption routines for data exchange x 3 hrs  General administration (meetings, telephone) x 5 hrs Total: 11 hrs @ $30 / hr + 20% benefits $ 396Equipment / Other  Hardware & software - - - Total Start-Up Costs $ 5,875 Total Annual Costs $ 26,012 NWHSS – Mills Memorial Hospital Training Health Records Personnel Information Systems (IS) Equipment / Other THR – Royal Inland Hospital & Lillooet District Hospital Training - - - - Health Records Personnel - - RIH:  HR level 11 x 0.43 FTE @ $39,000 / yr +20% benefits LDH:  300 hrs @ $20 / hr $ 20,100 $ 6,000Information Systems (IS) - -  IS support of databases @ $25 / hr x 3 hrs / mon x 12 mon $ 900 Equipment / Other - - - - Total Start-Up Costs - Total Annual Costs $ 27,000 Start-up costs were not relevant at participating hospitals in the THR because these hospitals have been collecting ED injury data prior to the inception of the ISPP. 24Financial costing for the ISPP also involves “In-Kind” support where work done on the project was part of a larger job portfolio and not costed separately. Details of In-Kind support provided by the ISPP Project Management Team are outlined in the following table: In-Kind Support Time Commitment Pay Rate Regional Level Medical Health Officer 5 – 6 hrs / month @ $100 / hr Chair of Injury Surveillance Task Group 16 hrs / month @ $100 / hr Regional Director of Health Records 16 hrs / month @ $ 40 / hr Information Systems Manager 6 hrs / month @ $ 40 / hr Injury Prevention Co-ordinator 8 hrs / month @ $ 35 / hr Hospital Level Chief of Emergency 2 – 5 hrs / month @ $100 / hr ED Nurse Managers 2 – 5 hrs / month @ $ 35 / hr Health Records Managers 5 – 12 hrs / month @ $ 39 / hr Admissions Managers 5 – 8 hrs / month @ $ 30 / hr 251.2 Have participating hospitals filled all staff positions required for data abstraction and coding, and if not, what were the barriers? Expected Outcomes  A list of staff positions that were required and filled for implementation of data collection.  A statement of problems and barriers associated with inability to fill positions Methods / Steps 1. Develop Health Records Personnel form describing staff positions. 2. Distribute forms to Health Records managers at SFHR, THR, PRRH, KGH, MMH and BVDH. 3. Follow-up (1- week post dissemination) on the completion and submission of these forms via e-mail. 4. Follow-up (2- weeks post dissemination) on the completion and submission of these forms via telephone. 5. Summarize information from forms. Findings Five of six Health Records Personnel forms were returned, from two regions and three hospitals respectively, for a response rate of 83 percent. The table below outlines the type of HR position, the portion of a Full Time Equivalent (FTE) staff member, and if it was filled internally to the HR department using part-time or casual staff, or if external hiring was required. The HR Manager at MMH was unable to provide Health Records staffing information at this time because of current responsibilities and workload. This information will be provided at a later date. Hospital Health Records Position FTE Filled South Fraser Health Region Langley Memorial Hospital HR Technician 1.0 Internally with external backup Delta Hospital HR Technician 0.5 Internally with external backup Peace Arch Hospital HR Technician 1.0 Internally with external backup Surrey Memorial Hospital HR Technician 2.4 Internally with external backup 26Hospital Health Records Position FTE Filled North West Health Services Society Prince Rupert Regional Hospital HR Administrator 0.3 Internally Mills Memorial Hospital (Terrace) Not available Kitimat General Hospital HR Technician HR Administrator 0.2 0.05 Internally Internally Bulkley Valley District Hospital (Smithers) HR Technician HR Administrator 0.6 for both Externally Internally Thompson Health Region Royal Inland Hospital (Kamloops) R7 Clerk 0.3 Internally Lillooet and District Hospital R7 Clerk 0.1 Not filled – charts sent to RIH for coding Three sites reported difficulties in hiring HR staff. SMH encountered a lack of qualified Health Records Technicians (HRTs) and so recruited second-year Douglas College students. Unfortunately, they returned to school in September 2001 and were no longer available. Similarly, BVDH in Smithers was unable to recruit until mid-August 2001. Another barrier encountered by SMH in filling HR positions for the project included higher-than-projected costs because Health Record Administrators (HRAs), rather than HRTs, were staffed, as they were more available. Discussion The shortage of available HR staff is a current problem not only in BC but also in Canada as a whole. Despite this shortage, HR managers have been able to fill the positions created by the ISPP. Douglas College in New Westminster will be graduating a class of HRAs in the spring of 2002. This will help ease the shortage of HR staff but will also influence the HR resource requirements, as HRAs are more costly than HRTs. 271.3.1 Is the training program resulting in more complete charting in the ED? Expected Outcome  A review of random samples of hospital records pre- and post-training sessions demonstrating at least a 20 percent absolute improvement in completeness. Methods / Steps 1. Design Chart Audit form, first in Word and then also in Access as an e-form (see Appendix V). 2. For each hospital, describe, by date, the training sessions undertaken to improve data capture for Minimum Data Set. 3. Sample size calculations: 150 charts required (50 per hospital).7 4. From the date of the training session, randomly select 25 pre- and 25 post-training charts per hospital from the following hospitals:  Surrey Memorial Hospital: Training dates of July 18-24, 2001  25 charts from April - June, 2001  25 charts from Aug - Oct, 2001  see Data Collection Question 1.6 for sampling details  Royal Inland Hospital: Training date of February 14, 2001  25 charts from Dec 2000 - Feb 13, 2001  25 charts from Feb 15, - March 2001  selected using computer randomizing software (Research Randomizer Shareware http://www.randomizer.org/)  Prince Rupert Regional Hospital: Training date of February 21, 2001  25 charts from Dec 2000 - Feb 15, 2001  25 charts from Feb 25 - May 2001  starting at Feb 15, selecting every other day going backwards in time for 50 days, with all ED forms per day shuffled in no particular order, selecting the third identified injury for each day  starting at Feb 25, selecting every day going forwards in time for 25 days, with all ED forms per day shuffled in no particular order, selecting the third identified injury for each day  the post-training forms can be augmented by supplementing forms from the reliability chart audit 7 Software used: PEPI, Version 3.01, PM Gahlinger & JH Abramson, 1993-2000 Calculation of the sample size was made to provide the number of subjects required for detecting a difference of 20% in the rates of completion before and after training with an alpha error of 0.05, and a power of 80%. This approach indicated a sample size of n=114 to n=134 subjects. Calculations for 150 subjects indicated Power to be 82-87%. 285. Retrieve selected ED charts in Health Records Department for chart audit. 6. Hire Health Records students from Douglas College to perform chart audits. Health Records staff also required at Surrey Memorial Hospital to co-ordinate audit. 7. Complete one long form per chart directly into Access e-form or on hardcopy. 8. Enter any hardcopy audits into Access database. 9. Assess the completeness of charting according to BCIRPU criteria, where:  "0" indicates not available  "1" indicates not sufficient for coding (from BCIRPU perspective)8  "2" indicates sufficient for coding (from BCIRPU perspective) 10. Reassess the completeness of charting according to ICD-9/ICD-10 criteria for Diagnosis and External Cause of Injury where:  "0" indicates not available  "1" indicates not sufficient for coding (from ICD perspective)9  "2" indicates sufficient for coding (from ICD perspective) 11. For each data element, compute the proportions of charting completion (indicated by a "2") by ED staff for both the pre-training and post-training periods.10 12. Assess the Difference of Proportions using a Z-test (Fleiss, 1981). Findings a. Chart Audit Site Overview Chart audits to determine changes in charting practices pre- and post-training were undertaken in three of the participating hospitals, one per region. Hospitals of the THR received training in mid-February, 2001. RIH (Kamloops), representative of a mid-size ED, was selected for the chart audit in this region. ED injury charts during the December 2000 - March 2001 time period were readily identifiable, as this site had been abstracting injury ED charts for a related project for the past three years. Hospitals of the NWHSS received training in late February 2001. PRRH was selected for the chart audit in this region and is representative of a small ED. ED injury charts are identifiable by manual searching only for the December 2000 - March 2001 period. Injury charts from this period are not accessible from any of the other NWHSS hospitals. Injury charts were not routinely abstracted prior to April 1, 2001 in this region. Hospitals in the SFHR received training in July 2001. SMH was selected for the chart audit as it represents a large ED. Identifying ED injury charts is not an issue, as the time 8 BCIRPU perspective refers to the level detail required to appropriately code a data element according to the level of aggregation used to report the EDISS data. 9 ICD perspective refers to the level of detail required to appropriately code a data element (Diagnosis or External Cause of Injury) according to the ICD coding scheme. This system codes to a much finer detail than the level of aggregation used to report the EDISS data. 10 The Narrative Statement was not assessed in this evaluation. Currently, six hospitals are consistently collecting a Narrative Statement in at least 50% of their injury cases, and 57% of all charts in the EDISS database include a Narrative Statement. Further work with the narrative data will be addressed in a separate project. 29frame for this chart audit is during the one-year data collection period and data from this time period have been received at BCIRPU. b. Description of Training Sessions Thompson Health Region: Royal Inland Hospital (Kamloops) February 14, 2001: Hassan Soubhi  Met with Admitting Staff  Met with Nurse Manager and ED nurses  The purpose of the project was outlined and material reviewed concerning the roles of charting the Minimum Data Set on the ED forms.  Materials left for distribution to other staff. North West Health Services Society: Prince Rupert Regional Hospital February 21, 2001: Hassan Soubhi  Met with Admitting Staff  Met with Nurse Manager and ED nurses  The purpose of the project was outlined and material was reviewed concerning the roles of charting the Minimum Data Set on the ED forms.  Materials left for distribution to other staff. South Fraser Health Region: Surrey Memorial Hospital Problems were encountered in trying to meet with the ED Admitting and Nursing staff and HR staff for the training sessions scheduled during February 2001. Nurse Managers suggested repeated visits during shift change (e.g. 0730 hours shift change from night to day staff) would allow for personal contact with a greater number of nurses. July 18, 20 & 24, 2001: Kate Turcotte & Hassan Soubhi  The Nurse Manager co-ordinated all three training visits during the 0730 hours shift change and ensured maximum participation.  Small group sessions were held in order for a portion of the nurses to attend (while the remaining nurses covered their patients for them).  The purpose of the project was outlined and material was reviewed concerning the charting of the Minimum Data Set on the ED forms.  Handouts were distributed and extras left for distribution to other staff.  The first morning included Admitting, Nursing and Health Records staff and provided an excellent opportunity to facilitate ISPP-related communication between their departments.  All three visits were well attended.  The nurses were very concerned about limited space on the charts to include information, let alone finding time for more detailed charting.  One nurse in particular was very frustrated with the provincial government allocation of health resources, specifically regarding the BC Nurses' Union contract negotiations, and was not supportive of this project or the idea of prevention. Much discussion was generated around this topic.  At this site, the Admitting staff does most of the injury-event charting, but their allotted space for a narrative statement in the current software is limited. These software issues were to be discussed further with the regional IS Department. 30 There is good communication between Admitting and HR staff, which facilitates bringing these issues to light. In addition to the training described above, ED posters and pocket booklets for Admitting, Nursing and Physician staff were developed and distributed by June 2001. The goal of these products is to increase awareness of the project among the ED staff, highlight the 13 elements of the minimum data set, and encourage complete and accurate charting. When asked if they had seen an EDISS poster in their departments, 18 of 33 (54%) “first level” Key Informants and 12 of 21 (57%) “second level” Key Informants responded positively. Of the 30 respondents who had seen the poster, 26 (87%) indicated that the poster raised or somewhat raised their awareness of EDISS. When asked if they had received the EDISS Pocket Booklet, 16 of 33 (48%) “first level” Key Informants and 11 of 21 (52%) “second level” Key Informants responded positively. Of the 27 respondents who had seen the pocket booklet, 22 (81%) indicated that it was useful or somewhat useful for charting the data elements. c. Chart Audit Results The results of the 150 chart audits from the three sites combined—RIH, SMH, and PRRH—are depicted in the table below. The completeness rate refers to category “2” of the coding scale from the BCIRPU perspective. Completeness was high for the demographic elements. Completeness Coding: 0, 1 & 2* All Sites n = 75 pre- + 75 post-training = 150 Number of Cases Data Elements 0 Pre- / Post- 1 Pre- / Post- 2 Pre- / Post- PHN Postal Code Date of Visit Date of Birth Sex 1 3 0 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 74 72 75 75 74 73 75 75 75 75 Place of Occurrence Activity When Injured Cause of Injury-BCIRPU Diagnosis-BCIRPU Cause of Injury-ICD Diagnosis-ICD Visit Disposition 6 18 3 0 3 0 11 13 29 4 1 4 1 13 6 21 3 8 8 18 0 3 12 4 5 10 16 0 63 36 69 67 64 57 64 59 34 67 69 61 58 62 * 0=not available, 1=incomplete, 2=sufficinet for coding 31Statistical analysis of the chart audit focused on four aspects of injury documentation done in the ED pre- and post-training. They include Place of Occurrence, Activity When Injured, Cause of Injury, and Diagnosis. The specific focus of the training was to increase the completeness and quality of the injury-event information charted. Note that although physicians document diagnosis they were not targeted by these training sessions. None of the differences found attained the 20 percent absolute improvement as stipulated in the expected outcome. Therefore a larger sample size would be required to provide the power needed to determine statistical significance of the changes in charting completeness pre- or post-training. Despite this limitation, the only data element shown to improve in charting completeness post-training was Diagnosis. Charting Completeness - Pre & Post Training All Sites n = 75 pre- + 75 post-training = 150 Data Elements & Statistics Values Place of Occurrence Pre-training proportion = Post-training proportion = Absolute Difference = Standard error of difference = z = p = 95% CI of difference: 0.84 0.79 0.05 (negative) 0.06 0.62 0.53 -0.08 to 0.19 Activity When Injured Pre-training proportion = Post-training proportion = Absolute Difference = Standard error of difference = z = p = 95% CI of difference: 0.48 0.45 0.03 (negative) 0.08 0.17 0.87 -0.14 to 0.20 Cause of Injury Pre-training proportion = Post-training proportion = Absolute Difference = Standard error of difference = z = p = 95% CI of difference: 0.92 0.89 0.03 (negative) 0.048 0.29 0.77 -0.08 to 0.13 32Charting Completeness - Pre & Post Training All Sites n = 75 pre- + 75 post-training = 150 Data Elements & Statistics Values Diagnosis Pre-training proportion = Post-training proportion = Absolute Difference = Standard error of difference = z = p = 95% CI of difference: 0.89 0.92 0.03 0.048 0.29 0.77 -0.08 to 0.13 Using a stricter rating of completeness for External Cause and Diagnosis, based on the ability to code for ICD-9/ICD-10, the following results were produced: Cause of Injury Pre-training proportion = Post-training proportion = Absolute Difference = Standard error of difference = z = p = 95% CI of difference: 0.85 0.81 0.04 (negative) 0.06 0.44 0.66 -0.09 to 0.17 Diagnosis Pre-training proportion = Post-training proportion = Absolute Difference = Standard error of difference = z = p = 95% CI of difference: 0.76 0.77 0.01 0.07 0.00 1.00 -0.14 to 0.16 While sample size calculations do not support statistical analysis of these data elements by site, the individual site results are included for comparison in Appendix VII. 33Discussion Analyses of the pre- and post-training periods showed slight differences in charting completeness. None of these differences reached the 20 percent threshold stipulated in the expected outcomes. None of the observed differences reached statistical significance. As the sample size calculations were based on detecting a difference of 20 percent, any lack of statistical significance could be due to insufficient power of the analysis. Training primarily addressed Admitting and Nursing staff. It was not expected that the training sessions would influence the charting of diagnosis information, as this falls under the responsibility of the physicians. Other factors that may have influenced the charting practices of Admitting and Nursing staff included ongoing job actions and union contract negotiations, and the changing political environment in BC at this time. From the analysis by site presented in Appendix VII, Place of Occurrence is seen to increase post-training at both RIH and PRRH, while it decreases at SMH. Activity When Injured decreased at RIH and SMH, while showing a higher level of completeness and increasing post-training at PRRH. Completeness of Cause of Injury as judged by the BCIRPU perspective was 100 percent both pre- and post-training at PRRH, and 96 percent both pre- and post-training using the ICD perspective. Cause of Injury decreased at both RIH and SMH using the BCIRPU perspective, but held at 84 percent at SMH using the ICD perspective. Diagnosis as judged by the BCIRPU perspective increased post-training at RIH and SMH, but using the ICD perspective only increased at SMH. SMH is the largest of the three sites, and therefore may not have been as influenced by the training as the smaller sites. Training at the smaller sites was provided in a seminar atmosphere with longer duration, whereas at SMH small groups were taken aside and addressed for shorter periods of time. Another difference between sites is their past experience in collecting ED data. RIH has abstracted and coded several years' worth of ED injury data prior to the ISPP, and therefore the ED staff has received ongoing encouragement in their charting efforts. Similarly, PRRH routinely abstracts and codes all ED data to submit to CIHI, while only SMH is new to the collection of ED data. 341.3.2 What is the reliability of the Health Records departments in abstracting and coding injury data? Expected Outcome  A review of random samples of hospital records demonstrating a 90 percent level of reliability for the following elements: Date of Visit, Date of Birth, Sex, Postal Code, Diagnosis, Cause of Injury, Place of Occurrence, Activity When Injured, and Visit Disposition. Methods / Steps 1. Design Chart Audit form as per Data Collection Question 1.3.1. 2. Sample size calculations: minimum 35 charts per hospital required.11 3. See sampling strategy in Data Collection Question 1.6  Surrey Memorial Hospital - these charts will overlap with Data Collection Questions 1.3.1 & 1.6  Royal Inland Hospital - these charts will overlap with Data Collection Question 1.6  Prince Rupert Regional Hospital - these charts will overlap with Data Collection Question 1.6 4. Retrieve selected ED charts from Health Records department for chart audit. 5. Hire Health Records students from Douglas College to perform chart audits. Health Records staff are also required at Surrey Memorial Hospital to co-ordinate audit. 6. Complete one long form per chart and enter data directly into Access e-form or on hardcopy. 7. Enter any hardcopy audits into Access database. 8. Compare Health Records codes as submitted to BCIRPU with the information from the chart audit (agreement: Yes/No). 9. For each hospital, calculate Percent Positive Agreement (PPA)12 for each data element. 10. Calculate 95 percent confidence intervals (CI).13 11 Software used: PEPI, Version 3.01, PM Gahlinger & JH Abramson, 1993-2000 The sample size for the determination of 90% agreement with 10% acceptable difference with a confidence level of 95% was determined to be 35 cases. 12 The Chamberlain's Percent Positive Agreement (PPA) is the number of occurrences for which both observers report a positive result out of the total number of observations for which at least one observer reports a positive result. Adapted to our analyses, it is the number of occurrences where information from the database agree with information from the chart audit agree (a) out of the total number of occurrences where they agree (a) and the number of occurrences where they disagree (b+c): (a/ (a+b+c))*100. Cicchetti & Feinstein (1990) as cited in: Szklo, M., & Nieto, F.J. (2000). Epidemiology. Beyond the Basics. Gaithersburg, Maryland: Aspen Publishers, Inc. 13 95% CI = Proportion ± 1.96 * standard error, where standard error = square root of [P * (1-P)]/n 35Findings Chart audits to measure the reliability of coded injury data were done at three sites. One site from each region was represented. The audited sites also represented examples of small (PRRH), medium (RIH, Kamloops), and large (SMH) EDs. The Percent Positive Agreement (PPA) for each data element and related 95 percent Confidence Intervals (95% CIs) are depicted in the tables below. Activity When Injured was coded least reliably in both RIH (73.8%) and SMH (70.7%). Place of Occurrence scores were ranked second lowest at all three sites. Visit Disposition (69.4%) was least reliable at PRRH. When all sites were combined, the least reliable coding pertained to the Activity When Injured, Visit Disposition and Place of Occurrence data elements. Coding Reliability - Percent Positive Agreement (PPA) Thompson Health Region: Royal Inland Hospital (Kamloops) n=76 Data Element PPA 95% CI Date of Visit 95.2 88.7-100.0 Date of Birth 97.6 92.9-100.0 Sex 100.0 - Postal Code 90.5 81.6-99.3 Diagnosis 92.9 85.1-100.0 Cause of Injury 90.5 81.6-99.3 Place of Occurrence 83.3 72.0-94.5 Activity When Injured 73.8 60.5-87.1 Visit Disposition 92.9 85.1-100.0 Coding Reliability - Percent Positive Agreement (PPA) North West Health Services Society: Prince Rupert Regional Hospital n=42 Data Element PPA 95% CI Date of Visit 98.7 96.1 - 100.0 Date of Birth 100.0 - Sex 100.0 - Postal Code 100.0 - Diagnosis 94.7 89.6 - 99.7 Cause of Injury 96.1 91.7 - 100.0 Place of Occurrence 81.6 72.8 - 90.3 Activity When Injured 94.7 89.6 - 99.7 Visit Disposition 69.4 57.9 - 78.8 36Coding Reliability - Percent Positive Agreement (PPA) South Fraser Health Region: Surrey Memorial Hospital n=82 Data Element PPA 95% CI Date of Visit 96.3 92.2 - 100.0 Date of Birth 92.7 87.1 - 98.3 Sex 98.8 96.4 - 100.0 Postal Code 87.8 80.7 - 94.8 Diagnosis 89.0 82.2 - 95.7 Cause of Injury 87.8 80.7 - 94.8 Place of Occurrence 86.6 79.2 - 93.9 Activity When Injured 70.7 60.8 - 80.5 Visit Disposition 92.7 87.0 - 98.3 Coding Reliability - Percent Positive Agreement (PPA) All Sites n=200 Data Element PPA 95% CI Date of Visit 97.0 94.6 - 99.3 Date of Birth 96.5 93.9 - 99.0 Sex 99.5 98.5 - 100.0 Postal Code 93.0 89.4 - 96.5 Diagnosis 92.0 88.2 - 95.7 Cause of Injury 91.5 87.6 - 95.3 Place of Occurrence 84.0 78.9 - 89.0 Activity When Injured 80.5 75.0 - 85.9 Visit Disposition 83.5 78.3 - 88.6 Discussion The reliability of data abstraction and coding was judged using the PPA between the coding performed by the HR staff and the information documented in the chart audits. Reliability was seen to be highest for the routinely collected data elements and to decrease among the elements requiring high levels of detail for coding purposes, such as Diagnosis, Cause of Injury, Place of Occurrence and Activity When Injured. The highest reliability was found for the socio-demographic variables: Date of Visit, Date of Birth and Sex. Reliability of Postal Code was not significantly lower than the reliability for Date of Visit or Date of Birth, as indicated by the confidence intervals. The lowest levels of reliability were found for Place of Occurrence, Activity When Injured, and Visit Disposition. Place of Occurrence and Activity When Injured are the two injury-event data elements that are not routinely collected by all EDs. Therefore it is to be expected that these are the elements with the lowest PPAs. While Visit Disposition also had a low PPA, this is considered a limitation of the chart audit, as described in the next section. 37Analysis by site demonstrated similarities in reliability among the three hospitals. Lower levels of reliability were observed at SMH, although these levels were not statistically different from RIH or PRRH. It is notable that Activity When Injured had the highest reliability at RIH. Site differences may have influenced the reliability of the coding. Specifically, SMH has been coding directly into Excel, originally in order to avoid the backlog of abstracting and coding that would have resulted if they had waited for the new hospital ambulatory software required for the ICD-9 to ICD-10 switchover. Therefore, this site did not have any systematic prompts or checks for coding that the other two sites did have. Alternatively, PRRH has continued coding in ICD-9 as per their requirement for submitting ambulatory data to CIHI. Therefore their coding skills have not suffered from learning ICD-10 for this data collection. 38 1.4 Is the Minimum Data Set more efficient to abstract and code than the full NACRS? Expected Outcome  Documentation that time required by Health Records personnel to abstract and code the minimum data set is 20 percent less than that required to code all of NACRS. Methods / Steps  Report to be written and submitted by Carol Lindsay at Prince Rupert Regional Hospital based on the Health Records Department's experiences abstracting and coding the complete NACRS and the BCIRPU Minimum Data Set. Findings PRRH currently abstracts and codes all ambulatory data using the National Ambulatory Care Reporting System (NACRS), as developed by the Canadian Institute for Health Information (CIHI). NACRS is broken into mandatory and optional data elements. Approximately 25 mandatory data elements are collected for NACRS. Some variation exists depending on the type of case (i.e. blood transfusions: the blood component and number of units given are collected). Some of the data elements are combined into one field (i.e. date of visit/registration time and date visit completed/disposition time). The demographic data elements are collected on registration directly into the admissions system and are subsequently downloaded to the 3M coding and abstracting system. The ten clinical data elements are entered manually in the HR Department. For all injury ED charts, five data elements are also collected specifically for BCIRPU in regards to the ISPP and EDISS (Chart Number, Activity When Injured, Place of Occurrence, Visit Disposition and a Narrative Statement). Determining whether or not the Minimum Data Set is more efficient to abstract and code than the full NACRS is not simple. Fewer elements are collected for the EDISS database (13 versus 25) but some of these elements overlap. PRRH submits ambulatory data (NACRS) to CIHI and then modifies its existing product to collect the Minimum Data Set to submit to BCIRPU. Therefore it actually takes longer to code an injury chart because of the five elements collected in addition to the 25 NACRS elements. Similarly, if an organization was collecting EDISS data for BCIRPU and was to modify their program to adapt NACRS (going from 13 to 25), it would then take longer to collect data for NACRS than the Minimum Data Set. 39Comparing only the time requirements to code the data, the Minimum Data Set takes approximately three minutes per case to collect versus five minutes per case for NACRS data collection. In reality, coding and abstracting data for the Minimum Data Set is not more efficient than for NACRS, they are the same. The time HR staff spends on coding a chart is not in the collection of demographic information, as Admitting Staff has already done this at the time of registration. HR staff codes the diagnoses and interventions. Time is required to review the nurse and physician documentation to determine the reason for visit and/or treatment, as well as in selecting appropriate codes. Depending on the amount and quality of the documentation, some charts take much longer than five minutes to code while others take less. NACRS costs about 0.5 FTE of an HRA ($25,000) plus vendor maintenance costs per year ($1,500). New products are required for April 1, 2002 as NACRS is switching from ICD-9 to ICD-10. The cost of this software is not available at this time. To purchase the DAD abstract and ICD-10 coding software last year was $10,000. Expected upgrades to the new NACRS product is estimated to cost approximately $2,000. 401.5 Is quality of data received from hospitals acceptable, with 90 percent or more of records being error-free? Expected Outcome  Ninety percent of records received by BCIRPU are free of errors according to the computing algorithm. Methods / Steps  Determine the total number of cases submitted from the ten hospitals and calculate the percentage of these cases that were submitted with errors, by reviewing the BCIRPU database.  Errors are identified by the computing algorithm when:  There are one or more missing elements with Chart Number, Sex, Date of Birth, Date of Visit, Main Diagnosis, Cause of Injury, or Visit Disposition. The other elements always have the potential to not be present, such as Postal Code (Non-BC Resident), PHN or Other Diagnosis. Or the element has a code for "unknown" (e.g. Place of Occurrence or Activity When Injured) that hasn’t been entered but is assumed.  The code is not a valid ICD-9 or ICD-10 code or is not a valid BCIRPU code. If the code is not contained within a specified range (Diagnosis code not an injury, Date of Visit before or beyond the specified period of submission). If the code does not conform to the specified size or structure (Postal Code or PHN) or has missing or extra characters.  There is an error in the layout of the structure in some of the records (e.g. Narrative Statement appears in Visit Disposition). Or one or more elements are completely missing from the data set. Findings The numbers of injury cases, submitted by hospitals and imported by BCIRPU, are listed below. Data submission was in progress from several sites at the time of the data run for this evaluation, therefore not all submitted cases had been imported into the EDISS database. The percentages of cases that are free of errors are also shown. All hospitals, except KGH, show error-free rates above 94 percent. No cases have been imported for KGH, so the error-free rate has yet to be determined. 41 Error-Free Data Rates Hospital Cases Submitted Cases Imported % Error Free* South Fraser Health Region Langley Memorial Hospital 6,393 2,894 99.5 Delta Hospital 5,022 2,346 99.8 Peace Arch Hospital 4,317 2,141 98.2 Surrey Memorial Hospital 11,652 11,652 99.6 North West Health Services Society Prince Rupert Regional Hospital 1,760 1,746 99.2 Mills Memorial Hospital (Terrace) 2,146 2,036 94.9 Kitimat General Hospital 735 0 - Bulkley Valley District Hospital (Smithers) 1,278 1,256 98.2 Thompson Health Region Royal Inland Hospital (Kamloops) 1,865 1,858 96.7 Lillooet and District Hospital 150 148 96.7 * % Error Free based on Imported Data. Not all submitted cases have been imported to date. There were some instances of incomplete data submission. KGH provided data for April to August 2001 coded in ICD-9, however Place of Occurrence was not coded using the requisite BCIRPU codes for this time period. Also, data from September 2001 to February 2002, coded in ICD-10, was missing BCIRPU Activity When Injured codes. Both of these errors are the result of miscommunication between the hospital and their software vendor. The hospital is currently recoding the data. Discussion The quality of data received from hospitals was found to be very high at nine of the ten sites, ranging between 94.9 and 99.8 percent error-free records. One site has been unable to submit the full Minimum Data Set to date, but is currently addressing the coding problems they have encountered. 42 1.6 What is the proportion of injury cases seen in the ED that are being reported to EDISS? Expected Outcome  Sensitivity and specificity for injury cases are maintained within an acceptable range, 90 percent or higher. Methods / Steps 1. Design Chart Audit form as per Data Collection Question 1.3.1. 2. Sample size calculations: 600 charts required (200 per hospital).14 3. Randomly select 200 charts (both injury and non-injury charts) for audit using computer randomizing software (Research Randomizer Shareware http://www.randomizer.org/), within the dates of data submission to BCIRPU for each of:  Surrey Memorial Hospital: April 1- June 30, 2001  Royal Inland Hospital: April 1 - May 26, 2001  Prince Rupert Regional Hospital: April 1 - August 17, 2001 4. Determine which of the randomly selected charts are contained in the EDISS database. These charts will be audited using the long form. The remaining charts will be audited using the short form. 5. Retrieve selected ED charts in the Health Records Department for chart audit. 6. Hire Health Records students from Douglas College to perform chart audits. Health Records staff also required at Surrey Memorial Hospital to co-ordinate audit. 7. Complete one audit per chart and enter directly into Access e-form or on hardcopy. 8. Enter any hardcopy audits into Access database. 9. Identify injury cases and non-injury cases based on BCRIPU criteria using documented Diagnosis and Cause of Injury information. The BCIRPU definition of injury, as used in EDISS, is the gold standard for identifying injury cases. 10. Tabulate the results: True Positive, True Negative, False Positive (non-injury sent to BCIRPU) and False Negative (injury not sent to BCIRPU). 11. Calculate sensitivity & specificity.15 12. Calculate 95 percent confidence intervals.16 14 Software used: PEPI, Version 3.01, PM Gahlinger & JH Abramson, 1993-2000 Sample size for determination of a 90% sensitivity or specificity with a 95% confidence level, and estimated true proportion of 90% and a maximum acceptable difference of 5% was 139 cases. To ensure 139 non-injury cases with a 95% confidence level and an estimated proportion of 70%, a sample size of 215 cases is required. To ensure 139 injury cases with a 95% confidence level and an estimated proportion of 30%, a sample of 521 cases is required. This was brought up to 600 cases, 200 per site. 15 Sensitivity: True Positive / (True Positive + False Negative) Specificity: True Negative / (False Positive + True Negative) 16 95% CI = Proportion ± 1.96 * standard error, where standard error = square root of [P * (1-P)]/n 43Findings A sample size of 521 is needed to determine a 90 percent sensitivity or specificity; based on the assumption that 30 percent of ED visits are injury cases. At least 200 charts were audited at each of the three sites. Therefore, it is necessary to consider the results of the chart audit from all sites combined. There were a total of 688 charts reviewed at the three chart audit sites. Overall, 82.6 percent of true injury cases are being correctly classified by hospitals, and 1.1 percent of non-injury cases are being misclassified. Reported Injury Cases All Sites Injuries Identified by BCIRPU Audit Injury Non-Injury Total Injury 199 5 204 Non-Injury 42 442 484 Injuries Identified by Health Records Total 241 447 688 Sensitivity = 82.6% (77.8% - 87.3%) Specificity = 98.9% (97.9% - 99.8%) Although the sample size by site was insufficient to determine 90 percent sensitivity or specificity by site, analyses for each hospital revealed high specificity ranging between 96 and 100 percent with no significant differences between sites (Appendix VII). Lower values for sensitivity were observed in all the sites, ranging between 79.4 percent (lowest in SMH) and 87.2 percent (highest in RIH). However, none of the differences between sites were significant. Discussion For all sites combined, the sensitivity of the system was found to be 82.6 percent (95% CI: 77.8-87.3). Therefore, 82.6 percent of the all injury cases presenting to the EDs are imported into the EDISS database. Hospitals are therefore under reporting ED injuries by 17.4 percent. Specificity of the system was found to be 98.9 percent (95% CI: 97.9-99.8). Only 1.1 percent of all non-injury cases presenting to the EDs are mistakenly sent to BCIRPU to be imported into the EDISS database. Sample sizes were inadequate to determine 90 percent sensitivity or specificity by site. The range of sensitivity by site was 79.4 percent (lowest at SMH) to 87.2 percent (highest at RIH). These differences were not statistically significant. Specificity ranged between 96 and 100 percent, with no significant differences between sites. 44RIH has collected injury ED data for several years in conjunction with BCIRPU, and is therefore experienced at identifying injury cases using the BCIRPU criteria. SMH is the largest of all participating EDs, and appears to have the biggest challenge with the classification of injury cases. These hospitals currently identify 29 percent of all ED cases as injuries. If the BCIRPU identification of injury is indeed correct, injuries may constitute up to 35 percent of all ED cases. For the chart audit, the BCIRPU criteria of injury were applied to the information documented on the ED form alone. Should more information from the chart be available, it is possible that details concerning the patient or the injury event would prove some of the cases identified as injuries to be excluded from this database (e.g. repeat visits). Therefore the injuries identified by the BCIRPU audit may be artificially high and may be driving down the sensitivity. 45 1.7.1 If resources are available, will the Hospitals/Regions continue to collect ED injury data and submit to BCIRPU? Expected Outcome  Willingness of ED staff, HR staff and Hospital/Regional management to continue data collection. Methods / Steps 1. Design Key Informant Survey, Parts I & II. (Appendix IV) 2. Complete list of Key Informants from each of the three regions (SFHR, THR, & NWHSS). This list includes ED Admitting, Nursing and Physician staff, Health Records staff, Hospital Management and other regional personnel involved with this project. 3. Send surveys (Parts I and II) out to all Hospital Management and involved Health Region personnel. A response rate of 70 percent or better is desired. 4. Send survey (Part I only) out to two ED physicians, two ED nurses, one ED admitting staff and one Health Records staff involved in the data abstraction per hospital. These individuals are accessed through their managers via e-mail or fax. 5. Follow-up (1-week post dissemination) on the completion and submission of these surveys via e-mail or telephone. 6. Follow-up (2-weeks post dissemination) on the completion and submission of these surveys via telephone. 7. Enter data from surveys into spreadsheet for summary. 8. Summarize information from surveys. Findings In discussion with BCIRPU, all three participating regions expressed the desire for continued data collection should resources continue to be available. In the NWHSS, one of the four hospitals has not been able to ensure continued data collection even if with continued resources. MMH has stated that insufficient HR staffing levels has made the abstraction and coding of data for EDISS a challenging project. The Key Informant Survey results indicate that 77.8 percent (42 of 54) of all Key informants would like to see the EDISS continue. Thirteen percent (7 of 54) do not want to continue data collection and 9.2 percent (5 of 54) did not respond. 461.7.2 Have any plans been made to secure internal or external funding for this project? Expected Outcome  Report efforts to secure internal or external funding for this project. Methods / Steps • Same methods as Data Collection Question 1.7.1: Key Informant Survey, page 45 Findings The Key Informant Survey showed that 66.7 percent (14 of 21) of Management Key Informants were unaware of any efforts undertaken to secure funding either internally or externally for the continuation of the ED injury data collection. In the SFHR, Dr. K. Yoshitomi and Dr. R. Gasparini, SFHR’s Public Health Officer, have been lobbying for regional funding. As well, efforts have been undertaken to work with the Red Cross to obtain external funding. Meanwhile, the THR is not actively seeking internal funding at this time as resources are limited and major service cuts may be up-coming. 472.0 Data Integration Questions 2.1 What is the mechanism in place in hospitals to scramble/unscramble chart numbers and personal health numbers, and transmit data securely to BCIRPU? Expected Outcomes  Documentation of standards and procedures to scramble, archive, protect, and transmit data.  Ability to unscramble data to link charts. Methods / Steps • Database Manager and the SFHR IS department will prepare a report outlining and explaining the steps to scramble and transmit data to BCIRPU. Findings Documentation of standards and procedures to scramble and transmit data, as well as the storage of data, is presented in the BC EDISS Database Manual from Data Analysis & Interpretation Question 3.1, page 54. Discussion The procedure to scramble and unscramble Chart Number and PHN was developed by a member of the SFHR IS department. This procedure was developed based on the requirements of the hospitals of the SFHR. Some adaptation has been required for other sites external to the SFHR. 482.2 Does the system demonstrate availability such that timelines for reporting are being met? Expected Outcomes  Amount of time for system management, including data transfer, entry, editing, storage, backup, analysis, and production of data reports to be within 20 percent of desired or proposed time.  Timelines for data reports by BCIRPU as outlined in the contract with Health Canada are met. Methods / Steps 1. Database Manager tracks data as it is:  Received by BCIRPU (Sept 2001 & Jan/Feb 2002)  Passed through the database to identify coding errors or missing codes  Error charts returned to the hospital for correction/confirmation  Error charts received back from hospitals and passed through software  Analysis of data available for reports 2. Time required to produce analyzed data will be determined, both for the first round of reports as well as the second round, at which time the data analysis will have been automated. Findings The original plan for data submission was for the hospitals to submit four months’ worth of data electronically to BCIRPU at the following time periods: Submission Period Injuries Seen in ED Submission Dates 1 April - July 2001 late Aug / early Sept 2001 2 Aug - Nov 2001 late Dec 2001 / early Jan 2002 3 Dec 2001 - March 2002 Not specified Reports were scheduled to be ready for distribution October 9, 2001 and February 4, 2002, approximately six months from the time the earliest patient was seen in the ED. Only the four hospitals of the SFHR were able to submit data for the first round of Hospital Data Reports. The SFHR submitted data files between September 19 and 49October 9, 2001. All errors were returned from the hospitals by October 18, 2001. Data reports for Period One were based on three months' worth of data rather than four, and were distributed November 22, 2001. Details are shown in the table below. Data Submission to BCIRPU Period 1 Key: N/A = Not Applicable Hospital Transmission Date Errors Sent to Hospital Errors Returned Period when Data Report Compiled Report Sent Out LMH Oct 9, 2001 Oct 12, 2001 Oct 18, 2001 Oct 6- Nov 10, 2001 Nov 22, 2001 DH Sept 19, 2001 N/A N/A Oct 6- Nov 10, 2001 Nov 22, 2001 PAH Sept 19, 2001 Sept 24, 2001 Oct 1, 2001 Oct 6- Nov 10, 2001 Nov 22, 2001 SMH Sept 11-19, 2001 Sept 24, 2001 Oct 11, 2001 Oct 6- Nov 10, 2001 Nov 22, 2001 PRRH - - - - - MMH - - - - - KGH Aug 29, 2001 Entire data set was missing BCIRPU Place codes. Corrections pending.* BVDH - - - - - RIH - - - - - LDH - - - - - * See explanation of Kitimat data in Data Collection Question 1.5, page 40. All hospitals were able to submit data for the second round of Hospital Data Reports (Period 2). Data files were submitted between December 14, 2001 and March 13, 2002. The correction of errors is still in progress at several hospitals. For this reason, Period 2 encompasses broader time frames for some regions and data reports will differ accordingly. The anticipated distribution timeframe for Period 2 Data Reports is April 2002. Period 2 details are outlined in the following table: Data Submission Period 2 Key: IP = In Progress Hospital Transmission Date Errors Sent to Hospital Errors Returned Period when Data Report Compiled Report Sent Out LMH Feb 6-27, 2002 IP IP IP IP DH Feb 15 - March 8, 2002 IP IP IP IP PAH Jan 30 - Feb 15, 2002 IP IP IP IP SMH Jan 28-31, 2002 Feb 4, 2002 Feb 11, 2002 March 6, 2002 IP 50 Data Submission Period 2 Key: IP = In Progress Hospital Transmission Date Errors Sent to Hospital Errors Returned Period when Data Report Compiled Report Sent Out PRRH Jan 17- Feb 5, 2002 Feb 6, 2002 Feb 8, 2002 March 4, 2002 IP MMH March 8, 2002 IP IP IP IP KGH March 6, 2002 Missing BCIRPU Activity Codes. Corrections pending.* BVDH March 13, 2002 IP IP IP IP RIH Dec 14, 2001-Jan 7, 2002 Jan 8, 2002 Jan 24, 2002 Feb 28, 2002 IP LDH Dec 14, 2001-Jan 7, 2002 Jan 8, 2002 Jan 24, 2002 Feb 28, 2002 IP * See explanation of Kitimat data in Data Collection Question 1.5, page 40. Discussion Delays in data submission were the cumulative result of the obstacles to ED injury data collection that have been experienced by all participating sites over the past year. The obstacle that most affected this data collection was the switchover from the ICD-9 to the ICD-10 coding system. This switchover required new hospital abstracting software with which to code the 2001/2002 fiscal year hospital data, however the software was not available until the end of the summer. The four hospitals of the SFHR circumvented this obstacle by coding the EDISS data directly into Excel software. Other obstacles include a province-wide shortage of HR staff as well as job actions and union negotiations. The first round of data reports produced required building, testing and revising the framework for the data report tables. Initially this process required over one month to produce the final tables for the four hospital data reports for the SFHR. This process has now been finalized and automated to a large extent, and requires substantially less time. Also as hospitals submit data for the first time, formatting and transmission issues are encountered and solved. Once a hospital has completed this process, subsequent submissions will be less onerous. 512.3 Can the data link with other data systems related to injury mortality and hospitalization such as the BC Linked Database (UBC), hospital discharge information, and Vital Statistics mortality data? Expected Outcome  Requirements for data linkage sought and adhered to such that the data will have the ability to link with other health data systems in BC. Methods / Steps • Contact the BC Linked Database at UBC to determine and summarize the requirements of the data for the potential of linkage. Findings The processes surrounding an agreement for data linkages between BCIRPU and the Centre for Health Services and Policy Research were investigated. They would include the following: 1. An agreement would be signed between BCIRPU and the UBC Centre for Health Services and Policy Research (CHSPR). CHSPR would act as the custodian of the personal information contained in BCIRPU's data files. CHSPR would also act as the agent of BCIRPU, linking information to the BC Linked Health Database. CHSPR would retain information (Minimum Data Set) from the BCIRPU data files in the CHSPR database. 2. BCIRPU would abide by the principles expressed in the Ministry of Health Planning/ CHSPR “Access Policy for Research Uses of Linked Health Data” agreement. BCIRPU would agree to participate in the BC Linked Health Database by providing CHSPR with information from the BCIRPU data files, beginning from a specified date and including all files, by year, to the present, and ongoing over time. 3. The Access Policy would set the following rules:  Each record in each file in the BCIRPU database has a scrambled unique identifier attached to it, which allows it to be linked to records in other files, or within a file over time, on a person-specific basis. However, the files are stored separately, and not in a linked form. Linkages are made for research projects only, on a project-by-project basis, for approved projects under the Access Policy. No other uses of the data are allowed. In particular, CHSPR is prohibited from participating in projects that involve administrative uses of linked data.  Chart Numbers and Personal Health Numbers (PHNs) contained in the BCIRPU database would be unscrambled (using the EDISS scrambling process), and 52transmitted to the Ministry of Health Planning where they would be scrambled in a manner that matches the current scrambling process used by CHSPR in linking their data.  Access to linked data is made available to bona fide researchers whose projects have been approved through a process outlined in the Access Policy. The process begins with researchers filling out access forms, as shown in the back of the Access Policy document. These identify the files needed for the research project; the variables needed in each file; the purposes to which the data will be put; and documentation that the project has been given ethical approval at the sponsoring institution and methodological approval through peer review.  These forms are submitted to the access co-ordinator in the Ministry of Health Planning, who circulates them to each of the relevant data stewards for approval. Each data file within the BC Linked Health Database has a ‘steward’, which is the agency or branch that generated the data and retains responsibility for them. For example, the steward of the birth and death files within the BC Linked Health Database is Vital Statistics and the steward of the Medical Services Plan file is the Medical Services Commission. Each of the stewardships has identified an individual who reviews and approves projects on its behalf. For instance, if a research project involves linking death files to Pharmacare files, then approval is needed from both Vital Statistics and Pharmacare before the project can proceed. BCIRPU would have the right of refusal of any research project involving the disclosure of information and/or linkage of information from BCIRPU data files.  The basis of decision-making for approvals is the ‘hierarchy of ethical challenge’ laid out in the Access Policy. It identifies five different uses of linked data, which range from the ethically trivial (e.g. the production of regional reports) to the highly challenging (e.g. using information from the linked system to contact individuals). Each of the data stewards has the right of refusal in relation to each project involving its data.  Difficult access issues, policies for access, and other ongoing issues are referred to the Data Access Committee. This committee, chaired by the Ministry of Health Planning access co-ordinator, includes representatives from the CHSPR and each of the data stewardships, as well as the Ministry’s Freedom of Information and Privacy Protection officer. It meets periodically when sufficient issues have accumulated. Within one month of the signing of the Memorandum of Agreement, BCIRPU would appoint a steward to act as a member of the Data Access Committee. Partnership and agency means that BCIRPU would agree to enter the BC Linked Health Database as a new steward, and CHSPR would agree to act as agent and custodian of BCIRPU data files. Applications for research projects requesting access to BCIRPU information and involving linkage of that information to other data held in the BC Linked Health Database would be submitted, first to the access co-ordinator in the Ministry of Health Planning who would then circulate them to the BCIRPU data steward. The BCIRPU steward would then review the application. 53 The Memorandum of Agreement would be valid until such time as one party requests its review. 543.0 Data Analysis & Interpretation Questions 3.1 Are aspects of data management and analysis at BCIRPU made transparent though written guidelines and through communication among staff? Expected Outcome  A manual outlining data management and analysis procedures Methods / Steps  Prepare a manual outlining and explaining the functioning of the database. Findings Please refer to the BC EDISS Database Manual (accompanying document). 553.2 Has the surveillance system, as described by the Minimum Data Set, accurately described the scope of injury related to patients presenting to EDs in participating health regions? Expected Outcome  Use of the ISPP data by the BCIRPU to describe at least 90 percent of the burden of injury presenting to Emergency Departments and to report new trends within eight months of their appearance in the participating Health Regions. Methods / Steps This question could not be addressed at this time, as sufficient data had not been submitted by the three regions to determine the burden of injury presented to EDs or for the identification of new trends. 564.0 Dissemination Questions 4.1 Does the system report in a timely fashion? (Follow-up on Data Integration Question 2.2) Expected Outcome  Time intervals between reporting by patient to Emergency Department, transfer of data from hospital to BCIRPU, and dissemination of information in reports within eight months. Methods Steps  Review first series of hospital data reports and release dates.  Review second series of hospital data reports and release dates. Findings Data Report Timelines Data Submitted Report Produced Time from ED to Report SFHR April-June, 2001 July-Sept, 2001 Nov 22, 2001 April 2002 4.5-7.5 months 9 months LMH April-June, 2001 July- Sept 2001 Nov 22, 2001 April 2002 4.5-7.5 months 6-9 months DH April-June, 2001 July-Dec 2001 Nov 22, 2001 April 2002 4.5-7.5 months 3-9 months PAH April-June, 2001 July-Nov 2001 Nov 22, 2001 April 2002 4.5-7.5 months 4-9 months SMH April-June, 2001 July-Nov 2001 Nov 22, 2001 April 2002 4.5-7.5 months 4-9 months NWHSS Incomplete Pending NA PRRH April-July 2001 April 2002 9-12 months MMH April 2001-Feb 2002 April 2002 1-12 months KGH Sept 2001-Feb 2002 April 2002 1-6 months BVDH Sept 2001-Feb 2002 April 2002 1-6 months THR April-May 26 2001 (8 weeks) April 2002 10-12 months RIH April- More data pending >12 months LDH April- More data pending >12 months 57The time frames from data submission to report production are specified in the previous table. Only the four SFHR hospitals submitted data for the first round of Hospital Data Reports. For these reports, the maximum time interval from the patient presenting to the ED to the dissemination of the data report was 7.5 months. For the second round of reports for the SFHR, the maximum time interval was 9 months. Two hospitals of the NWHSS submitted data originating in April 2001, PRRH and MMH. KGH and BVDH were unable to submit data for the April to August 2001 period. Both of these hospitals have submitted data from September 2001 onwards. The hospitals of the THR have submitted eight weeks’ worth of data to date. A Regional Report has been produced with these data. Once three or more months’ worth of data is received, individual Hospital Data Reports will be produced. Discussion As per Data Integration Question 2.2, delays in the reporting of the EDISS data has for the most part been the result of the backlog of chart abstracting and coding caused by the adoption of ICD-10 and its associated delay in hospital ambulatory software. This hurtle has now been overcome by most of the participating hospitals and should continue to improve over the next year of data collection. It is anticipated that the targeted interval of no more than eight months from ED visit to data dissemination will be achieved for all sites in the upcoming year. 584.2 What plans have been made for academic presentations (i.e. abstracts submitted)? Expected Outcomes  Abstracts submitted to academic organizations. Methods / Steps  Outline the abstracts that BCIRPU or any of the Hospitals/Regions have submitted for presentation.  Historical presentations will also be included. Findings 1. The following abstract was accepted for poster presentation at the 6th World Conference Injury Prevention and Control, Montreal Quebec, May 2002. Turcotte K, Raina P, Soubhi H & Brussoni M. The Injury Surveillance Pilot Project. Poster Presentation, 6th World Conference Injury Prevention and Control, Montreal Quebec, May 2002. Submitted Abstract: The Injury Surveillance Pilot Project Problem Under Study Current injury surveillance activities are based on data that are available in mortality and hospitalization databases. These data are limited in detail and do not reflect the full extent of the injury problem as the majority of people injured do not die or become hospitalized. EDs treat a large number of patients who sustain relatively severe injuries. This information is an invaluable source of injury-event information. Objectives The purpose of the British Columbia Emergency Department Injury Surveillance System is to collect standard injury data in a systematic fashion from all EDs throughout the province. The goals of this system are to: (1) Reduce the effort needed to determine the burden of injury in local areas; (2) Improve programming to reduce injuries in the community; (3) Provide the information base necessary for evaluating the impact of new laws, regulations and programs on community injury rates; (4) Coordinate injury prevention efforts; and (5) Reduce financial and societal costs of injuries. 59Method or Approach Steps taken to achieve these goals have included recruiting hospitals and implementing a minimum data set consistent with the injury portion of the National Ambulatory Care Reporting System. Data collection, coding and extraction have been standardized among ten hospitals in three Health Regions, with training and support provided to ED and HR staff. An electronic injury database has been established at the BC Injury Research and Prevention Unit. Data are received from the hospitals, cleaned, analyzed and interpreted. Information regarding this system is disseminated to hospitals and regions participating in this endeavor, community stakeholders and injury prevention professionals via a bimonthly newsletter and web-based updates. Information generated by these data is disseminated via data reports. Evaluation of this system has commenced, with long-term plans also in place. Results The first year of data collection for the ten hospitals participating in the British Columbia Emergency Department Injury Surveillance System occurred during the fiscal year April 1, 2001 to March 31, 2002. Hospital and regional reports were produced based on these data, as well as a comparison report describing the ten hospitals and three regions. An evaluation report was produced in March 2002, focusing primarily on the process and impact evaluations. These reports have been used to stimulate active collaboration with communities in the planning and evaluation of injury prevention initiatives. Conclusion The minimum data set is designed to include routinely recorded information, thereby minimizing the burden on the ED staff. Building on current hospital infrastructure as well as committing to a minimum data set ensures both the utility and sustainability of this project. Important considerations are: (1) Data must be useful; (2) The system must be simple while providing valid and reliable information; (3) Data collected must be accessible in an electronic format and have the capacity to link with other health care databases; (4) Dissemination strategies must meet community users needs. This system has been funded nationally as the Injury Surveillance Pilot Project by Health Canada (December 1999 - March 2002) and is supported provincially by the BC Ministry of Health. This initiative is in collaboration with community-partners at both the regional and provincial levels. Limits Challenges to this system have included the adoption of ICD-10 CA (from ICD-9 CA) along with the corresponding required software updates, job actions by the BC Nurses' Union and the Health Sciences Association, and a chronic understaffing of HR personnel. Sustainability of this initiative is a priority. Contribution of the Project to the Field An ED injury surveillance system will fill the gap in surveillance activities, currently limited to mortality and hospitalization data. Furthermore, the minimum data set 60promotes consistent collection of the injury-event information missing from these data sources. 2. Past Presentations  Turcotte K & Hotte A. Emergency Department Injury Surveillance. BC Injury Research and Prevention Unit, Teleconference 2001-2002 series, November 13, 2001.  Lu J, Nadeau B, Soubhi H, Raina P, Scanlan A, & Turcotte K. Injury Surveillance in the Emergency Department. Report of the second year experience of data collection in the Thompson Health Region. Poster presentation, British Columbia Injury Prevention Conference - 2001, Sidney, BC, May 2001.  Turcotte K, Raina P, Scanlan A & Soubhi H. Surveillance: Process and Overview. Poster presentation, Canadian Conference on Injury Prevention and Control - 2000, Kananaskis Alberta, October 2000. Also presented at the British Columbia Injury Prevention Conference - 2001, Sidney, BC, May 2001.  Yoshitomi K, McIntyre K, Scanlan A & Raina P. Journey in Development of a Community Injury Surveillance Program: From Inspiration to Implementation. Poster presentation, Canadian Conference on Injury Prevention and Control - 2000, Kananaskis Alberta, October 2000. Also presented at the British Columbia Injury Prevention Conference - 2001, Sidney, BC, May 2001.  Lu J, Nadeau B, Soubhi H, Raina P, Scanlan A & Turcotte K. Injury Surveillance in the Emergency Department. Report of a one-year experience of data collection in the Thompson Health Region. Poster presentation, Canadian Conference on Injury Prevention and Control - 2000, Kananaskis Alberta, October 2000. 614.3 Has a process been developed to address when and in what format aggregated data will become available? Expected Outcome  A plan for BCIRPU to provide access to aggregated data for participating hospitals and regions (e.g. Internet access via a web interface). Methods / Steps  BCIRPU & Bryon Taylor (Ministry of Health Planning, Office of Injury Prevention) to communicate with the Ministry of Health Planning, Health Data Warehouse administrators. Findings Availability of data in an aggregated form on the web has received a positive response from the Health Data Warehouse (HDW), Ministry of Health Planning. The following issues are currently being examined: 1. What are the legal/privacy requirements? (Ministry of Health Planning) 2. In what workable format would the data be presented for the user? (BCIRPU) 3. How much funding is needed to support this data access, and who will fund the costs? Ultimately, the data interface with the user would likely be added to the BC Injury Reporting System (BCIRS) interface of the HDW. Depersonalized Report Based on the data structure provided, HDW could provide a depersonalized report similar in structure to the Injury Hospitalization reports: Standard Parameters: Region of Patient Residence Year and Month of Visit Age Group (20 age groups and 3 age groups) Sex Additional Filters: Primary Diagnosis Secondary Diagnosis External Cause of Injury Place of Occurrence Activity When Injured Visit Disposition Notes 62 Month of visit was added to the standard year parameter because there is not a full fiscal year of data yet.  Hospital code could be added to the list of additional filters.  Possible statistics would be number of cases. Average cases and age-standardized cases could also be provided if it makes sense and data volumes are sufficient to make the information reliable.  Postal Code would be used to determine Local Health Area (LHA) of patient residence and other regions would be aggregates of LHA. If the Health Authority is to be supported, then depending on the outcome of the regional boundary discussions, a separate translation might be needed to support these regions.  Patient’s age would be calculated using Date of Birth and Date of Visit.  Given that this report is depersonalized, it is assumed that the audience for the report would be the same as the audience for any HDW report. That is, existing HDW clients would be allowed to run queries. The effort involved in providing a report of this nature should be about two weeks of technical time and one week of testing (data verification and application functionality). Personalized Reports Enhanced Security It is assumed that not all HDW users would be able to query at the individual case level. Enhanced security functionality (as described in the Ministry of Children and Families project) would be required to ensure that only HDW clients authorized to query at the case level were allowed access to the personalized report(s). The estimate for enhanced security has already been determined as two weeks for development and one week for testing. Reports The simplest personalized report would be a list of the individual cases meeting the parameters and filters criteria described for the depersonalized report. Other reports might allow the user to select a specific PHN or Chart Number. The development effort for these types of reports would be small – about two days per report (one-day technical, one-day testing). The depersonalized report could have an additional export button that would allow the user to download the individual cases (personalized) as well as the summarized (depersonalized) report. Of course, the additional download button would only be provided to authorized users. If this kind of functionality is required, then an additional day of technical time and a half day of testing time is required. 63Summary The effort needed for report production is delineated in the table below. Summary of Effort for Reports Effort in Days Task Technical Testing Total Depersonalized Report 10.0 5.0 15.0 Enhanced Security 10.0 5.0 15.0 Personalized Report – List of cases with same parameters/filters 1.0 1.0 2.0 Personalized Report – Chart Number query 1.0 1.0 2.0 Personalized Report – PHN query 1.0 1.0 2.0 Personalized Report – Download button on summary report 1.0 0.5 1.5 Total Effort 24.0 13.5 37.5 If all HDW clients could query the individual cases, then enhanced security would not be required and could be dropped from the task list. Otherwise, personalized reports cannot be done without the enhanced security feature. The first release of this data source could support depersonalized reports only. 644.4.1 Have hospital and regional participants as identified in the dissemination plan received data reports? Expected Outcome  Data sent to hospitals and regions identified in the dissemination plan at time intervals specified in the contract with Health Canada. Methods / Steps  Describe dissemination of data reports to participating hospitals and regions. Findings Hospital and Regional Data Reports have been disseminated to the SFHR Project Management group only. Reports and summaries of the report have been made available to the ED Physician, Nursing and Admitting staff, and HR staff personnel. Data Reports for the hospitals of the NWHSS and THR have been delayed due to delays in the data submission from these hospitals. These reports are currently in production and will be completed by the end of March 2002. A data report evaluation and needs assessment was done in January / February 2002 (see Evaluation Question 4.4.2 page 65). Discussion As per Data Integration Question 2.2 (page 48) and Dissemination Question 4.1 (page 56), the dissemination of both hospital and regional data reports has not met the dissemination plan at time intervals specified in the contract with Health Canada. 654.4.2 Has the feedback from data report evaluation forms been incorporated into the reporting structure? Expected Outcome  Feedback from data report evaluation has been used to improve the reporting structure. Methods / Steps 1. Design Data Report Evaluation form and Data Report Needs Assessment form. 2. Distribute both forms to all SFHR recipients of the first round of data reports. 3. Follow-up (1-week post dissemination) on the completion and submission of these forms via e-mail. 4. Follow-up (2-weeks post dissemination) on the completion and submission of these forms via telephone. 5. Enter data from forms into spreadsheet for summary. 6. Summarize information from evaluation and needs assessment forms. 7. Restructure Data Report Format to incorporate suggestions from evaluation forms. 8. Plan Data Reports based on needs assessment forms. Findings Synopsis Data Report Evaluation and Needs Assessment Forms were sent out to 26 members of the SFHR and one member of the Red Cross who received and likely read the Data Reports. The forms elicited responses regarding their rating of the Data Reports in terms of ease of reading and usefulness, and collected suggestions for improvement. Five (18%) replied that they were too busy to respond because of recent Health Region restructuring imposed by the provincial government, and eight (30%) did not respond after both email and telephone reminders. Fourteen (52%) completed and returned the forms. The representation of different disciplines was well balanced among the respondents: two physicians and two nurses from the EDs of two different hospitals; one Regional Medical Health Officer; three managers of HR and Admitting departments; one Communications Specialist; and one Red Cross Injury Prevention Co-ordinator. Key Findings The main message is that the first series of Data Reports for the SFHR was well received. The key findings are as follows: 66  The first series of Hospital/Regional Data Reports for the SFHR is easy to read. - Over 90 percent (13) of those who responded indicated that the report is easy to read. n=14 Not easy Somewhat Easy Very easy 0 1 (7.1%) 9 (64.3%) 4 (28.6%)  A great majority of the key informants was satisfied with the present Data Report format. - 86 percent (12) of respondents were satisfied (of these approximately 20% indicated very satisfied) with the content, presentation and graphics of the data report. n=14 Not satisfied Somewhat Satisfied Very Satisfied Content 1 (7.1%) 1 (7.1%) 10 (71.4%) 2 (14.3%) Presentation 1 (7.1%) 1 (7.1%) 10 (71.4%) 2 (14.3%) Graphics 1 (7.1%) 1 (7.1%) 9 (64.3%) 3 (21.4%)  Most informants found the breakdown of injury-event information useful. - The majority of the respondents found that Nature of Injury, Cause of Injury, Activity When Injured, Place of Occurrence, cross tabulations, and time and region are useful to them. n=11 of 14 Not Useful Somewhat Useful Very useful Nature of Injury 0 2 (18.2%) 8 (72.7%) 1 (9.0%) Cause of Injury 0 2 (18.2%) 7 (63.6%) 2 (18.2%) Activity When Injured 0 3 (27.3%) 6 (54.5) 2 (18.2%) Place of Occurrence 0 3 (27.3%) 7 (63.6%) 1 (9.0%) Cross Tabulations 0 3 (27.3%) 7 (63.6%) 1 (9.0%) Injuries by Time 1 (9.0%) 2 (18.2%) 7 (63.6%) 1 (9.0%) Injuries by Region 2 (18.2%) 2 (18.2%) 6 (54.5) 1 (9.0%) 67  64 percent (9) of the informants would recommend this injury information to other colleagues. - Most informants agreed that the Data Reports are useful in planning for prevention program, budget for emergency care and education to targeted high-risk groups. Some thought the report could be used to monitor the temporal changes in injury rates. - Most informants liked the current Data Report format. Some thought the reports should add narrative summaries and might be more detailed. Yet some would like to see the report concentrating on top 10 (or major) injuries with cross tabulations for the major injuries such as where, how, why and what.  Over 60 percent (7) of the respondents thought the information collected by the Emergency Department Injury Surveillance System was relevant for injury prevention. n=11 of 14 Not Relevant Somewhat relevant Relevant Very Relevant 1 (9.1%) 3 (27.3%) 3 (27.3%) 4 (36.4%) Comments No suggestions were made for changes to the current format. Suggestions for future reports included:  Include a commentary co-authored by BCIRPU and an ED physician or the Regional Medical Health Officer.  Recommendation of injury prevention strategies (especially if they have been found successful in other jurisdictions) for the communities and regions should be added to the reports.  Include information on how injuries occurred, exact locations (other than home) where injuries occurred and what activities.  The denominator should be clearly stated for each community population to calculate the injury rate.  Include focused reports for:  elderly  sport injuries  work-related injuries  youth injuries  detailed comparisons across regions and time  review of narrative summaries to reveal clues as of how and what injuries could be prevented 684.5.1 How are new regions engaged in EDISS? Expected Outcome  A system delineating how new regions can join the EDISS initiative.  A summary of first steps of regions currently participating in EDISS. Methods / Steps  Develop a tool outlining the process by which new regions are engaged in EDISS.  Prepare a report describing the first steps of regions currently participating in EDISS and the ISPP. Findings The process by which new regions are engaged in EDISS involves the following steps: 1. Introduction to regions by the BC Injury Research and Prevention Unit. Focus on the desire to implement injury surveillance in BC, with a data collection component. 2. One person from the interested region initiates contact with BCIRPU, stating that their region has an interest in injury prevention initiatives.  Several of our initial contacts discussed their desire to implement surveillance systems, while needing support and leadership for the process. 3. Following initial discussions with the contact person, support is secured from someone with the responsibility and authority to designate resources and pledge their Region’s commitment to injury surveillance and prevention initiatives in upcoming years.  Both BCIRPU and our initial contact person play key roles in this phase.  Presentations and reports are delivered to the Regional Health Board.  BCIRPU attains written commitment from the Regional Health Board, committing to match BCIRPU resources for the purpose of developing a population-based injury surveillance system in their hospital EDs. 4. Getting ready to collect data. Each ED, including the HR Department, must answer the follow questions as part of a Needs Assessment:  What type of injury information is presently collected in the ED?  Who is charting the information?  Is there room on the ED form for additional questions?  How is the information processed in the Health Records department?  What type of software system used in computer for data collection?  Are there hardware requirements for implementing an injury surveillance system? 69 Is the software compatible with abstracting injury specific reports? i.e. producing reports relevant to minimum data set fields.  How will the patient be identified as having an injury?  How much additional time for Health Records personnel is needed to deal with the data collection: abstraction, coding and data transmission?  What are the perceived barriers at the source of data collection?  What are your specific needs regarding surveillance?  What kind of training and human resources are required for this data collection? First Steps of Regions Participating in EDISS 1. North West Health Services Society The NWHSS has been involved in injury prevention initiatives for 10 years. They originally secured money through a Closer to Home grant in 1992/1993, and have built injury prevention into a permanent part of their budget since 1997. Their injury prevention project has funded multiple community injury prevention initiatives over the last several years. This region initiated a regional project with their four main hospitals to collect population-based ED injury data in collaboration with BCIRPU. The hospitals are Kitimat General, Mills Memorial (Terrace), Bulkley Valley District (Smithers), and Prince Rupert Regional. The HR Departments of these four hospitals process data for six communities because Prince Rupert Regional Hospital manages the Queen Charlotte Islands and Kitimat General Hospital manages Stewart. Injury surveillance data have previously been collected in one hospital in this region, however the project was not sustainable. Major barriers included human resource costs in the HR department related to extracting data. This region has several interventions in place aimed at reducing population-based injuries, but is doing little in the way of evaluation. The region's goals include setting up population-based injury surveillance systems in their EDs, implementing more intervention programs and developing evaluations for all interventions in place. 2. Thompson Health Region This region conducted a three-month pilot project comprising of the collection of injury data in two EDs (Royal Inland - Kamloops and Lillooet District Hospitals). A one-time grant of $10,000 from their Health Board enabled the project's genesis. Their goals included sustaining the project with the help of BCIRPU. Along with their interest in surveillance, the THR would like to conduct injury prevention initiatives once surveillance is operational and providing information regarding the causes, types, risks, etc. of injury in their region. 70 3. South Fraser Health Region Staff from Peace Arch Hospital set up a committee to examine injury in their region. The committee had a wide array of stakeholders interested in developing awareness, surveillance and prevention initiatives within their communities. They identified a number of priority action strategies and consulted with BCIRPU to determine the next steps. The region assessed the requirements to collect ED injury data from all four EDs of the SFHR (Langley Memorial, Delta, Peace Arch - White Rock, and Surrey Memorial Hospitals). 714.5.2 Have non-participating hospitals or regions expressed interest in joining the EDISS initiative? Expected Outcome  Requests for information regarding EDISS from non-participating Health Regions expressing interest in joining this initiative. Methods / Steps 1. Develop complete list of Key Informants from each of the three regions 2. Contact each Key Informant via telephone to elicit responses to the following two questions:  Has anyone outside of your hospital / region expressed interest in this system? (Evaluation Question 4.5.2)  Has this system stimulated any research proposals or projects relevant to injury prevention? (Evaluation Question 4.9) Findings Various degrees of interest in participating in EDISS have been shown from several regions. BCIRPU received some interest from the East Kootenay region before the ISPP was initiated. As well, the Public Health Department in Prince George initiated ED injury data collection independently of the ISPP. The North Okanagan region has expressed interest in NACRS, and the necessary adaptations to be compatible with EDISS. SFHR and NWHSS have reported receiving no external interest in this system. THR has received interest in EDISS and the data coming out of this project from Kelowna General Hospital. Interest was also expressed from non-participating hospitals of the THR. James Lu has made a formal presentation of EDISS within the THR. Resource constraints have deterred any further action in this region to date. 724.6 Did the newsletter enhance understanding of the project by Regional Health Boards, ED staff & managers, and program directors? Expected Outcome  Feedback regarding the bimonthly newsletter made available to ED and HR staff, and hospital strategic planning teams. Methods / Steps  Same methods as Data Collection Question 1.7.1: Key Informant Survey, page 45 Findings Information regarding the newsletter was gathered through the Key Informant Survey. The newsletter was seen by 61.1 percent of respondents (33 of 54). Of these respondents, 90.9 percent (30 of 33) read the newsletter and subsequently 63.3 percent (19 of 30) felt that their awareness of EDISS had definitely been raised while 30 percent (9 of 30) felt it had been somewhat raised. Positive comments included thoughts about formatting, content, and enhanced understanding. The format was described as concise, easy to read, clear, to the point, and well laid out. The graphs, tables, and electronic format were appreciated. Newsletter content was described as pertinent, informative, and detailed. It clarified the purpose and status of the data collection and made the work more meaningful for some. That the information was pertinent, hospital-specific and allowed comparisons with other hospitals on both data and population elements was especially well liked. Alternatively, there were some things that respondents did not like. Some criticized the formatting, saying that the fonts and graphs were too small or too busy. One respondent had concerns that the report did not print properly. In terms of content, there was some criticism that it was not specific enough. There was a suggestion that content be tailored to a specialized audience, rather than be left too general. Respondents suggested that the newsletter should have a Frequently Asked Question addressing the BCIRPU guideline interpretation. As well, specific data collection issues geared to ED Admitting, Nursing and Physician staff could be addressed. Other suggestions were geared toward making the information in the newsletter more relevant. For example, inclusion of more summary statistics such as the highest incidences of a particular element, or selecting a focused topic and reviewing it in more detail, such as falls in the elderly. 73A recurring request was made for community-specific information about injury prevention activities that occurred as a result of the ISPP data. Respondents also wanted to know about possible prevention programs that could be developed as a result of the research. Discussion EDISS Newsletters are disseminated as PDF files via e-mail. Recipients are instructed to try downloading a newer version of adobe acrobat if they have trouble opening the PDF file, and a web link is provided. Newsletters are also available on the BCIRPU web-site (www.injuryresearch.bc.ca). To date, the following EDISS Newsletters have been disseminated:  June/July EDIS Newsletter: Our First Newsletter (in PDF format).  August/September EDISS Newsletter: (in PDF Format).  October/November EDISS Newsletter: (in PDF Format).  Large Graphs: Larger Version of the Graphs from the Oct/Nov Newsletter (in PDF Format).  EDISS Newsletter Volume II Number 1: (in PDF Format). The October/November EDISS Newsletter contained graphs from the first SFHR data report. The format for these graphs were determined to be too small for clear legibility, and BCIRPU responded by enlarging these graphs are reissuing them as a separate PDF file. EDISS Newsletters will continue to be produced over the next year. Therefore there is ample opportunity to address some of the suggestions from the Key Informant Survey, including specific focused topics, such as injuries among the elderly (e.g. age 75+). Further work will also focus on gaining interest, readership, and stimulating prevention actions. 744.7 Has ED injury surveillance data been used to plan and evaluate injury prevention? Expected Outcome  At least one injury prevention program or program evaluation within each participating region. Methods / Findings  Same methods as Data Collection Question 1.7.1: Key Informant Survey, page 45 Findings It is too early for information from the ISPP to have been used to plan or evaluate injury prevention. The Key Informant Survey showed that two thirds of informants (36 of 54, 66.7%) believe that data collection can assist in planning and evaluating Injury Prevention programs. Another 25.9 percent (14 of 54) felt that data collection could be somewhat useful while 5.6 percent (3 of 54) did not respond. Management level Key Informants were asked to describe how their hospital or region has used surveillance data in the past to plan or evaluate injury prevention programs. Health information departments related that they have received data requests from other services and organizations such as those interested in brain injury. The data have also been used to get the local community interested in specific issues such as traffic-related deaths. Past data have been used to justify some hospitals’ participation in the present EDISS project. A minority of Management level Key Informants (8 of 21, 38.1%) anticipated that their hospital or region would use the ED injury data to plan or evaluate injury prevention programs. Moreover 42.8 percent of respondents (9 of 21) did not anticipate the use of these data in this manner and 19.0 percent (4 of 21) did not respond. One respondent asked where staff with enough time to even set up a program might be found. Another mentioned that their site had only received three months’ worth of data; and they had not used the data as the hospital data report cautioned that the sample was small and results should not be considered to be a stable estimate. Positive and negative comments included the following:  Cannot say, new administration coming in for restructuring.  Depends on changes with new Health Authority.  It is unlikely that the hospital would be doing evaluation. It is more likely that we would provide data to another group or person doing the evaluation. 75 Maybe have pamphlet in ED with suggestions as how to prevent certain injuries, articles in appropriate magazine or papers with data about injuries and how to prevent them, e.g., horse magazines regarding falls from horses.  Maybe. There is not sufficient data trending available yet. Adequate data trending would be necessary to determine the value of the information and how it may be used within the Health Regions  The outcome of different initiatives by the traffic safety coalition to decrease deaths (could be evaluated).  We may partner or collaborate with community programs to plan and evaluate injury prevention programs.  Will be using this to look at forming a Health Authority wide injury prevention initiative/plan.  Work with medical health office, public health and outside groups such as the Red Cross and specific groups such as fire, police and ambulance.  Would like to establish Team with ED and public health for community education programs based on data collected. 764.8 Are surveillance findings reviewed by hospital or regional executive boards as part of resource allocation activities? Expected Outcome  Information from the ISPP is incorporated into administrative reporting and decision-making activities. Methods / Steps  Same methods as Data Collection Question 1.7.1: Key Informant Survey, page 45 Findings It is too early for information from the ISPP to have been incorporated into administrative reporting and decision-making activities. Management level Key Informants were asked if they anticipated these reports to be to be used in determining resource allocation. One-third (7 of 21) anticipated this type of use, one-third (7 of 21) did not anticipate this type of use, and one-third (7 of 21) did not respond. Over one third (8 of 21, 38.1%) of Management level Key Informants anticipated that reports produced using the ED injury data would be reviewed by their executive board and incorporated into administrative reporting and decision-making activities. Less than one third (6 of 21, 28.6%) did not anticipate that this would happen, and one-third (7 of 21) did not respond. There was a lot of uncertainty expressed about the impact of the provincial Health Region restructuring on this (or any) initiative. Individual comments included:  At the program level, the EDs in the South Fraser will use it but our future in the Fraser Health Authority is uncertain.  Administration reporting and decision making is strongly influenced by data such as these.  Any data collected should be used for analysis and planning.  Difficult to say what the priorities will be given the current climate.  Might be used to demonstrate long-term effectiveness of injury prevention programs to justify their existence. More likely to be used by specific groups within the organization to assess existing programs and develop new programs on injury reduction.  Probably, but with current climate unsure at this time.  The executive board is in Prince George. I expect that the board would be making regional decisions.  To plan community based injury prevention education programs once target populations are identified. 77 Will be using this to look at forming a Health Authority wide injury prevention initiative/plan.  With community injury prevention program. 784.9 Has EDISS stimulated research proposals / projects relevant to injury prevention? Expected Outcome  At least one research proposal / project relevant to injury prevention is planned within each participating region. Methods / Steps 1. Develop complete list of core Key Informants from each of the three regions 2. Contact each core Key Informant via telephone to elicit responses to the following two questions:  Has anyone outside of your hospital or region expressed interest in this system? (Evaluation Question 4.5.2)  Has this system stimulated any research proposals or projects relevant to injury prevention? (Evaluation Question 4.9) Findings BCIRPU received a request from Walter Schneider in relation to his work with Fathers for Life and the Men's Defense Association. He has indicated interest in the data coming from EDISS, particularly those pertaining to Intentional Injury and domestic violence. The SFHR has had discussions with a regional group focusing on the health of seniors and falls prevention. This group is very interested in the EDISS data, and look forward to seeing a specialized report on falls among the elderly. No proposal had been presented as yet. Jerry Wong, of the BC Men's Resource Centre in Vancouver, has also expressed interest in the data coming from EDISS in a direct communication with Kerry Yoshitomi. The NWHSS felt that the project needed increased promotion to external groups involved with injury prevention initiative. The Public Health Unit is very eager to see the EDISS data. The THR was able to contribute ED injury data towards an ICBC Project that overlapped with the beginning of ISPP. Also the Public Health Unit hired an Injury Prevention Coordinator last summer. To date she has requested and received ED data concerning bicycling and skiing injuries. 794.10 Who are the champions of EDISS within participating health regions and what are their roles? Expected Outcome  A champion for the EDISS is identified within each participating health region. Methods / Steps  Same methods as Data Collection Question 1.7.1: Key Informant Survey, page 45 Findings SFHR  Admitting staff for their efforts to obtain all relevant injury details (LMH)  Darlene Sinkovits, Health Records (LMH)  Tracy Parkinson, ED Nurse Manager (LMH)  Anne Clarke, ED Director who really likes to see outcomes (LMH)  Adriana Devries, Admissions Manager, and Admitting staff (LMH)  Dr Vroom, ED Physician (SMH)  Joy Avasthi, Health Records trainer for coding (SMH)  Dr Yoshitomi, Regional Medical Coordinator Quality Improvement, Chair of the Clinical Health Informatics Committee (PAH/SFHR)  Kathy McIntyre, ED Nurse Manger (PAH)  Jill Buchan, Regional Health Records Manager (PAH/SFHR)  Roberta Morris, Admissions Manager (PAH)  Emergency Reception/Admissions staff (PAH)  Emergency RN (PAH)  Amy Setterington, Health Records Technician (PAH)  Marlene Walker, Health Records Technician (PAH) NWHSS  Ester Brisch, Public Health Unit  Andrea Lorette, Health Records Manger (PRRH)  Beth Adeshynski, Management & Administration (PRRH)  Doris Gemmell, Health Records Administrator (PRRH)  Gail Eliot, Nurse Manager (PRRH)  Grace O'Connor, Health Records Manager (KGH)  Kathi Nelson - Nurse Manager - assists in educating nurses to data element requirement (KGH)  Kelly Abrams, Director of Health Information-this department provides the data (BVDH) 80THR  Dr James Lu, Medical Health Officer - helped get the project started (RIH)  Vicky Weiler, Injury prevention coordinator - very anxious to start using the data  Maxine Mucha, Director Health Records Services (RIH)  Brenda Nadeau, HR Resource Information Analyst (RIH)  Dr. Ciastko, Physician, Dept. Pediatrics (RIH)  Dr Routley, Physician, provides information regarding EDISS and encourages physicians to fill in the ED sheet correctly (LDH)  Admitting clerks who ask initial questions and enter original information on ED admission charts  ED RNs who record the input data from ED information  ED - we use this data in conjunction with the trauma registry for data collection  HR data clerks who input the reported data 81Discussion Support for the ISPP This evaluation of the ISPP has demonstrated that most of the participating hospitals and regions desire to continue the data collection for EDISS, and that the limiting factor of this data collection beyond funding is the availability of HR staff. Key Informants have identified multiple champions of the ISPP and the data collection for EDISS at both the ED and HR levels as well as the management level. Indeed, this list of champions exceeds those in direct contact with BCIRPU, illustrating the growing interest in and commitment to this project from within the hospitals and regions. One means of promoting the ISPP within participating hospitals and regions has been the EDISS Newsletter. As indicated by the respondents of the Key Informant Survey, the newsletters have been well received and are helping to raise awareness around injury prevention and the role of data collection. Through this medium, future support for EDISS will be attained. EDISS and the ISPP have been presented multiple times over the last few years at both provincial and national injury conferences. The ISPP is currently scheduled for presentation internationally at the 6th World Conference on Injury Prevention and Control being held in Montreal, Canada from May 12-15, 2002. Overall, findings of this evaluation indicate that there is the potential to expand EDISS within BC, as indicated by the interest reported from other hospitals and regions. Further work would be required regarding the changing political climate and structure of the health systems to determine where EDISS might fall within the management of the new Health Authorities. The sustainability and future expansion of EDISS beyond the ISPP will be determined primarily by the availability of resources dedicated to injury surveillance activities. Data Quality Assessment of sensitivity and specificity of EDISS was conducted to examine the ability of the system to capture the injury cases and exclude non-injury cases presenting to the participating EDs. Results of this evaluation indicate an 82.6 percent (77.8% - 87.3%) capture rate. Inappropriate inclusion of non-injuries in the database was negligible with system specificity of approximately 99 percent, a result comparable to other studies (Macarthur & Pless, 1999). Improvements in the level of completeness of Main Diagnosis and Cause of Injury would have to be addressed relative to the ICD coding requirements for these data elements. There is some indication that completeness of charting may be easier to achieve at smaller sites. This could be influenced by fewer injury cases seen over the same period of time as compared to larger sites, and therefore the perception of more time to document injury- 82event information. Smaller sites also have fewer ED staff members, therefore it may be easier to achieve commitment from the ED staff to chart more injury-event details. Finally, budgetary pressures may be felt differently at the smaller sites. Reliability was reasonably high across data elements with the lowest boundary of the 95 percent CI at 75 percent and an upper boundary of 99 percent. Analyses of reliability for each hospital revealed high similarities between the selected sites. Overall, analyses of reliability for all sites combined showed that it decreased with the level of detail required for proper coding of a given data element. Alternatively, it could be concluded that reliability decreased for the injury-event data elements, which the ED staff may not have been routinely documenting prior to implementing EDISS. Two of the three sites participating in the chart audit adopted ICD-10 for the 2001/2002 fiscal year, therefore reliability could have been affected by the learning curve for this new coding system of Diagnosis and Cause of Injury. Place of Occurrence and Activity When Injured are injury-event data elements that ED staff are not as likely to chart in great detail. Furthermore, the coding scheme for Activity When Injured is particularly stringent, and requires a high level of detail to code appropriately. Completeness proportions were found to be between 96 and 100 percent for the socio-demographic variables. Diagnosis and Cause of Injury reached a level of completeness17 of 89 and 92 percent respectively. The lowest level of completion was for Activity When Injured, where only 48 percent of the audited charts had complete information. Analyses of the differences in the completion rates between the pre- and post-training periods showed slight differences that did not reach statistical significance. As none of these differences reached the 20 percent threshold assumed, a larger sample size would be required to provide adequate power for assessing the statistical significance of the differences found. These results clearly indicate the need for increased attention towards improving the completeness of the Activity When Injured data element. Difficulties with this data element stem from the level of detail required for the coding system. Activity When Injured requires not only the literal activity, such as riding a bicycle, but also an explanation of the motivation for that activity (e.g. traveling to work). Taking into account the combined results of sensitivity, reliability and completeness analyses, the injury information collected by the EDISS Minimum Data Set provides an accurate assessment of the injury events seen in the participating EDs. Increased attention must be paid to the data elements that describe Place of Occurrence and Activity When Injured, which could be addressed by increasing the focus on the Narrative Statement. Potential improvements to sensitivity may be expected with a review of the protocol for identifying injury ED charts and targeted training on the inclusion and exclusion criteria used in EDISS. In short, this evaluation has demonstrated that the data being collected and submitted by the participating hospitals are error-free, and that the system has achieved good sensitivity 17 Based on the BCIRPU perspective, referring to the level detail required to appropriately code a data element according to the level of aggregation used to report the EDISS data. 83and very high specificity. Charting by ED staff is complete for the socio-demographic data elements, Diagnosis and Cause of Injury, while needing more attention for the injury-specific data elements, such as Place of Occurrence and particularly Activity When Injured. Abstraction and coding by the HR staff is reliable, again with future attention directed at Place of Occurrence and Activity When Injured. Summary of the Obstacles Encountered The implementation of the ISPP involving ten Hospitals within three Health Regions of BC has encountered several obstacles over the past 16 months. Specifically these include:  Provincial shortage of HR staff for external hiring  Mandated adoption of ICD-10 for inpatient charts at all sites  Optional adoption of ICD-10 for ambulatory charts at nine of the ten sites  Delay in receiving hospital software updated for ICD-10  Job actions involving the BC Nurses' Union, the Hospital Employees' Union and the BC Health Sciences Association  Contract negotiations between the three unions and the provincial government The impact of these obstacles on the ISPP has been varied. The obstacle with the greatest impact has been the adoption of ICD-10 and the resulting delay in receiving the new hospital software. This created a backlog of not only the EDISS data abstraction and coding, but also of all the hospital charts processed through the HR departments. Furthermore, ICD-10 is a completely different system for coding than the ICD-9 version, therefore all HR employees were required to under go training for ICD-10. Adoption of a new coding system increased the time required per chart for abstraction and coding, which has slowly decreased over time as the HR staff's learning curve improved. With all of these changes in the HR departments, scheduling of the EDISS training and awareness sessions was delayed. Also, targeting the different groups involved in EDISS activities within the hospitals proved challenging, particularly in the larger hospitals and for all Physician staff. The provincial shortage of HR staff has been a chronic challenge throughout the implementation of the ISPP, and has also played a role in delaying the abstraction and coding of injury ED charts for EDISS at some of the sites. This problem has also made it more difficult to catch-up with the large backlog. Fortunately Douglas College in New Westminster, BC, will be graduating a class of HRAs shortly, which should help ease this staffing shortage. Despite all of these obstacles, data is coming out of the system. Furthermore, many of these obstacles are one-time events, and their effect on the ISPP will be diminished over time. The timeliness for data abstraction, coding and submission will improve as these challenges are addressed. 84Limitations of the Evaluation The one all-encompassing limitation of the current evaluation of the ISPP and EDISS is the timing of the evaluation in relation to the progress attained by the participating hospitals. As described throughout this evaluation, many obstacles were encountered over the past year resulting in the delay of data abstraction, coding and submission to BCIRPU. Therefore, many of the evaluation questions could not be fully addressed, as hospitals are just now finalizing their internal procedures for data submission and the data has yet to be widely disseminated. MMH in Terrace was unable to submit both Financial Summary and HR Staffing information at this time. These pieces will be submitted later and this evaluation amended. Low response rates to both the Key Informant Survey and the Data Report Evaluation and Needs Assessment are considered to be a result of the recent reorganization of BC Health Regions into new Health Authorities, and the related budget cuts imposed by the provincial government. Regional management is in the process of reorganizing and downsizing, with many positions undergoing changes in responsibilities. New hospital CEOs as yet unfamiliar with the EDISS activities in their own hospital were unable to answer the survey questions regarding the ISPP. Other factors influencing the response rate of these tools include the workload related to the fiscal year-end activities and vacation times spanning the course of the Key Informant Survey and the Data Report Evaluation and Needs Assessment. Further investigation regarding who did not respond will aid in determining future changes to the implementation of such evaluation tools. Finally, an identified limitation of this evaluation is that chart audits were based on the information documented on the ED forms only. These forms potentially contain all of the demographic and injury-event information concerning the ED visit. However, when abstracting and coding the ED charts, HR staff may or may not have access to other documents in the patient's chart, and may or may not take demographic or injury-event information from these sources. An example of this is the Visit Disposition. For the chart audit at PRRH, Visit Disposition was not always indicated on the ED form. If the Visit Disposition was missing from the Chart Audit documentation, but was coded as "Visit Completed" in the EDISS database, this element was not considered to be in agreement. This limitation may have lead to low estimates of charting completeness, reliability and the sensitivity of EDISS. Recommendations for the ISPP In response to the potential uses of the EDISS data by hospitals and regions, it is recommended that each of the three participating regions create committees to address the use of these data and explore the role of community partners in injury prevention. In response to the need for continued funding, it is recommended that each of the three participating regions create committees to address these funding issues if the data 85collection for EDISS is to be sustained beyond the scope of the ISPP. Specific focus should be placed on the role of ED data collection within the newly formed Health Authorities. In response to the completeness of charting by the ED staff, it is recommended that the structure and content of the training sessions be reviewed with a focus on Place of Occurrence and Activity When Injured. Increased support from hospital management may aid in individually targeting the Admitting, Nursing and Physician staff in the ED. Periodic follow-up of the training sessions may be warranted to maintain high levels of completeness. In response to the reliability of data submitted to BCIRPU, further investigation may reveal that reliability is directly influenced by the completeness of charting, particularly for the Place of Occurrence and Activity When Injured data elements. Training sessions for both ED and HR staff may affect the reliability of the data. Furthermore there is the potential to increase reliability by reviewing and adjusting the coding schemes for both Place of Occurrence and Activity When Injured. In response to the sensitivity of EDISS, strategies to reduce the number of real injury cases missed by the HR staff should be explored. The first step is to review the False Negative cases as identified in the chart audit to ensure that these cases are indeed missed injuries. Further exploration of the patient charts beyond the ED may provide reasons for excluding these cases from EDISS, thus revealing a higher sensitivity than is reported here. The second step is to explore the different systems currently in use at the different sites for identifying injury cases, along with a comparison based on the size of EDs. The third step is to address ongoing training of HR staff in the identification of injuries. In response to the current scrambling and transmission procedures in place for submitting EDISS data to BCIRPU, it is recommended that a committed individual external to BCIRPU is required, with time and resources to deal with each hospital or region one-on-one when adjustments are required. In response to the data analysis and production of data reports, continued refining of the automation of the EDISS database is recommended. The specific focus of this automation will be directed by the specialized topics selected for in-depth investigation. In response to the dissemination of data reports, an updated Dissemination Plan is warranted for Period 3 of the ISPP, with a focus on specialized data reports and systematic evaluations of these reports. The list of recipients should be reviewed, along with their roles in the new political climate and creation of Health Authorities. Furthermore the means of distribution (electronic vs. hard copy) should also be reviewed. In response to the current EDISS Newsletter, content for upcoming newsletters for Period 3 of the ISPP should be drafted with themes identified, taking into account suggestions from this evaluation. The distribution list should be reviewed and expanded, as well as the means of distribution. 86In response to the plans for upcoming academic presentations regarding the ISPP and EDISS, it is recommended that an abstract be planned for submission to the BC Injury Conference scheduled for April 2003. Furthermore, a plan for the production of papers for academic publication should be developed. Furthermore, it is recommended that resources be identified to conduct a second stage surveillance of a particular injury trend as identified by the first year of EDISS data with a six month component for data collection and a three month component for data entry and analysis. 87REFERENCES Angus DE, Cloutier E, Alber T, Chenard D, Shariatmadar A, Picket W, & Hartling, L. (1998). The economic burden of unintentional Injury in Canada. Toronto, ONT: SMARTRISK. BCIRPU. (2001). British Columbia Injury Research and Prevention Unit Newsletter, June/July. Vancouver, BC: BCIRPU. BC Vital Statistics Agency. (1999). Selected vital statistics and health status indicators. Annual Report 1998. Victoria, BC: British Columbia Vital Statistics Agency. BC Vital Statistics Agency. (2001). Selected vital statistics and health status indicators. Annual Report 2000 (ISSN 1188-3642). Victoria, BC: British Columbia Vital Statistics Agency. Cloutier E & Albert T. (2001). Economic Burden of Unintentional Injury in British Columbia. Vancouver, BC: BC Injury Research and Prevention Unit, and SMARTRISK. Fleiss, J. (1981). Statistical methods for rates and proportions (2nd ed.). New York, NY: John Wiley & Sons. Health Canada. (1997). Economic burden of illness in Canada, 1993. Ottawa, ONT: Minister of Public Works and Government Services Canada. Macarthur, C., & Pless, I. (1999). Evaluation of the quality of an injury surveillance system. American Journal of Epidemiology, 149, 586-592. Mackenzie, S. (1998) Proceedings of the International Collaborative Effort on Injury Minimum Data set. Injury Surveillance Meeting Committee Room, Government Conference Centre. Ottawa, November 3-4. Minister’s Injury Prevention Advisory Committee. (1997). The provincial injury prevention plan for children and youth (ages birth to 24). BC Injury Free: A Framework For Action. Victoria, BC: Ministry of Health, Office for Injury Prevention. Porteous, N., Sheldrick, B., & Stewart, P. (1997). Program evaluation toolkit: A blueprint for Public Health management. Ottawa, ONT: Ottawa-Carleton Health Department. Ribbeck, B., Runge, J., Thomason, M., & Baker, J. (1992). Injury surveillance: A method for recording E-codes for injured emergency department patients. Annals of Emergency Medicine, 21(1), 37-40. Survey Research Center. (1993). West Virginia Social Indicators Survey, 1992. Morgantown, WV: Department of Sociology and Anthropology, West Virginia University. 88APPENDIX I ISPP Operating Budget Periods 1 & 218 Period 1 Period 2 BCIRPU Labour: $115,328 $259,470 BCIRPU Researchers Project Co-coordinator Database Manager Data Analyst Evaluator Secretary Research Assistant (year 1 only) Overhead $23,010 $39,719 Equipment and Software $ 7,760 $ 0 Materials $ 8,132 $ 9,830 Travel $ 8,061 $18,760 ________________________________________________________________________ Total $162,291 $327,779 Period 1 Period 2 Hospitals Labour $347,805 $563,201 Regional Management Health Records IS Support Equipment & Software $ 35,598 $ 30,500 Travel $ 2,260 $ 2,894 ________________________________________________________________________ Total $385,663 $565,595 Period Total $547,954.00 $924,374.00 Project Total: $1,472,328.00 18 Financial commitment varies from region to region, as well as across hospitals within a region. The greatest factor influencing the financial outlay depends on the existing system for data collection. Significant steps in the process that impact new financial costs include: number of data elements currently being collected and coded; computer hardware and software; and volume of presentations in the emergency department. Once data collection systems are in place, the primary costs in maintaining the system are the cost of Health Records staff abstracting the charts and software licensing. 89APPENDIX II Evaluation Questions Development and Worksheets The starting point of the evaluation process involved the compilation of a list of evaluation questions related to data collection, data integration, data analysis and interpretation, and data dissemination. The grouped questions were then further divided according to program components including: activities, target groups, short term outcomes, and long term outcomes. The Evaluation Logic Model, depicted below, shows the linkages between IPSS components and evaluation questions. Questions were next designated as high or low priority for three stakeholder groups: Health Canada (program manager), internal stakeholders, and external stakeholders. Only those questions rated as being high priority for at least two of the three stakeholder groups were retained for evaluation. For each retained question, expectations were developed in terms of outcomes, how outcomes would be measured, and what level of outcome would be deemed acceptable. Along with each evaluation question and corresponding outcome, a data collection plan was devised and a decision about feasibility for achievement within the designated time frame was determined. This information is summarized in the Methods Worksheets on the next pages. 90 Recruitment Data Collection Data Integration Data Analysis & Interpretation Dissemination ↓ ↓ ↓ ↓ ↓ Activities Staff & program deliverables BCIRPU -regional buy-in -hospital buy-in -needs assessment -signing agreements -ethics, privacy Hospitals -deliver training sessions -purchase/install hardware/software -create staff positions Hospitals -download data into Excel -clean & save as a text file -scramble & transfer BCIRPU -load data into import database -test & triage errors -transfer clean data to main database BCIRPU -create variables -report on data quality -present a descriptive analysis of injury findings -interpret findings, including context and explanation BCIRPU -develop a distribution plan -create & update website -distribute bimonthly newsletters -conduct presentations & workshops -distribute data reports ↓ ↓→ ↓ ←↓ ↓ Target Groups For whom the program is designed -BCIRPU -regional hospital CEOs and Medical Health Officers, Injury Prevention Coordinators -Hospital staff (ED, HR, IS) -Health Canada -Health Records staff -Admitting staff -Hospital IS department -Hospital IS department -BCIRPU -BCIRPU -MOH Office of Injury Prevention -MOH Deputy Medical Officer and Director of Office for Injury Prevention -Regional hospital CEOs & Medical Health Officers -Injury Prevention Coordinators -Advocacy ↓→ ↓ ↓→ ↓ ↓ Short Term Outcomes Direct impact of program on participants -initiate data collection in selected health regions -initiate data collection -preparation of clean data by hospitals into standard format -confidential & secure transfer of data files -accrual of clean data into main database -reports to stakeholders -increased awareness of data reports among all health regions in BC ↓ ↓ ↓ ↓ ↓ Long Term Outcomes Consequences of program in community -sustain participation from initial health regions -expand to other health regions / hospitals in BC -sustain data collection -ongoing accrual of secure, confidential, and clean data -development of injury surveillance reports at regular intervals -inclusion of surveillance data in strategic planning initiatives -enhancement of evidence-based decision-making re injury prevention programs -use as a basis for research activity 91 Methods Worksheet Evaluation Questions 2a Expectations of the Program (based on Expectations Worksheet) 2b Data Collection Plan 2c Logistics (based on Logistics Worksheet) 1.0 Data Collection “I expect to have…” Does Data Exist? Type of Tool Who could Provide the Data? (Source) Who Can Get the Data? (Collector) Design How Many? Timeframe Is This Feasible? 1.1 Are hospitals able to break down the cost of implementing the system, including in kind support, such that BCIRPU can accurately describe the resources and time involved for other agencies to join the Emergency Department Injury Surveillance System?  A list of annual expenses for data collection, including ongoing training sessions, and hiring of additional staff. X Yes  No Financial Summary Form Administrator for the surveillance system, designated by each participating hospital/health region. BCIRPU Descriptive 10 hospitals March 2002 X Yes  No 1.2 Have participating hospitals filled all staff positions required for data abstraction and coding, and if not, what were the barriers?  A list of staff positions that were required and filled for implementation of data collection.  A statement of problems and barriers associated with inability to fill positions. X Yes  No Health Records Personnel Form Administrator for the surveillance system, designated by each participating hospital/health region. BCIRPU Descriptive 10 hospitals March 2002 X Yes  No 92 1.0 Data Collection “I expect to have…” Does Data Exist? Type of Tool Who could Provide the Data? (Source) Who Can Get the Data? (Collector) Design How Many? Timeframe Is This Feasible? 1.3.1 Is the training program resulting in more complete charting in the ED? 1.3.2 What is the reliability of the health records departments in abstracting and coding injury data?  A review of a random samples of hospital records pre-and post-training sessions demonstrating at least a 20 percent absolute improvement in completeness.  A review of a random samples of hospital records demonstrating a 90percent level of reliability for the following elements: Date of Visit, Date of Birth, Sex, Postal Code, Diagnosis, Cause of Injury, Place of Occurrence, Activity When Injured, and Visit Disposition. X Yes  No Chart Audit Form Health Records Department BCIRPU Evaluator Pre- Post-test of proportions (z-test) Percent Positive Agreement Random sample of records March 2001 X Yes  No 1.4 Is the Minimum Data Set more efficient to abstract and code than the full NACRS?  Documentation that time required by Health Records personnel to abstract and code the Minimum Data Set is 20percent less than that required to code all of NACRS. X Yes  No Report Health Records Staff BCIRPU Descriptive One hospital (PRRH) Completed X Yes  No 1.5 Is quality of data received from hospitals acceptable, with 90 percent or more of records being error-free?  Ninety percent of records received at BCIRPU are free of errors according to the computing algorithm. X Yes  No Data checking software at BCIRPU BCIRPU Data Manager BCIRPU Data Manager Descriptive 10 hospitals Feb 2002 X Yes  No 1.6 What is the proportion of injury cases seen in the ED that are being reported to EDISS?  Sensitivity & specificity for injury cases are maintained within an acceptable range, 90 percent or higher. X Yes  No Chart Audit Form Health Records Department BCIRPU Evaluator Sensitivity & specificity Random sample of records Feb 2002 X Yes  No 93 1.0 Data Collection “I expect to have…” Does Data Exist? Type of Tool Who could Provide the Data? (Source) Who Can Get the Data? (Collector) Design How Many? Timeframe Is This Feasible? 1.7.1 If resources are available, will the Hospitals/Regions continue to collect ED injury data and submit to BCIRPU? 1.7.2 Have plans been made to secure internal or external funding for this project?  Willingness of ED staff, HR staff and Hospital/Regional management to continue data collection.  Report efforts to secure internal or external funding for this project.  Yes X No Key Informant Survey Key Informants BCIRPU Descriptive 10 hospitals Mar 2002 X Yes  No 94 Methods Worksheet Evaluation Questions 2a Expectations of the Program (based on Expectations Worksheet) 2b Data Collection Plan 2c Logistics (based on Logistics Worksheet) 2.0 Data Integration “I expect to have…” Does Data Exist? Type of Tool Who could Provide the Data? (Source) Who Can Get the Data? (Collector) Design How Many? Timeframe Is This Feasible? 2.1 What is the mechanism in place in hospitals to scramble/unscramble chart numbers and personal health numbers, and transmit data securely to BCIRPU?  Documentation of standards and procedures to scramble, archive, protect, and transmit data.  Ability to unscramble data to link charts. X Yes  No Report BCIRPU Data Manager & South Fraser Health Region IS Department BCIRPU Descriptive SFHR Feb 2002 X Yes  No 2.2 Does the system demonstrate availability such that timelines for reporting are being met?  Amount of time for system management, including data transfer, entry, editing, storage, backup, analysis and production of data reports to be within 20 percent of desired or proposed time.  Timelines for data reports by BCIRPU as outlined in the contract with Health Canada are met. X Yes  No Report Data Manager, BCIRPU Data Manager Descriptive BCIRPU March 2002 X Yes  No 2.3 Can the data link with other data systems related to injury mortality and hospitalization such as the BC Linked Database (UBC), hospital discharge information, and Vital Statistics mortality data?  Requirements for data linkage sought and adhered to such that the data will have the ability to link with other health data systems in BC.  Yes X No Report Custodians of BC Linked Database BCIRPU Descriptive BCIRPU 2002 + X Yes  No Beyond March, 2002 95 Methods Worksheet Evaluation Questions 2a Expectations of the Program (based on Expectations Worksheet) 2b Data Collection Plan 2c Logistics (based on Logistics Worksheet) 3.0 Data Analysis “I expect to have…” Does Data Exist? Type of Tool Who could Provide the Data? (Source) Who Can Get the Data? (Collector) Design How Many? Timeframe Is This Feasible? 3.1 Are aspects of data management and analysis at BCIRPU made transparent through written guidelines and through communication among staff?  A manual outlining data management and analysis procedures. X Yes  No Report Data Manager, BCIRPU BCIRPU Descriptive BCIRPU February 2002 X Yes  No 3.2 Has the surveillance system, as described by the Minimum Data Set, accurately described the scope of injury related to patients presenting to EDs in participating health regions?  Use of the ISPP data by the BCIRPU to describe at least 90 percent of the burden of injury presenting to Emergency Departments and to report new trends within eight months of their appearance in the participating Health Regions. X Yes  No A summary report with conclusions based on answers to questions 3.1, 3.2, 5 & 6 under data collection, question 2 under data integration. BCIRPU BCIRPU Summary BCIRPU March 2002 X Yes  No 96 Methods Worksheet Evaluation Questions 2a Expectations of the Program (based on Expectations Worksheet) 2b Data Collection Plan 2c Logistics (based on Logistics Worksheet) 4.0 Dissemination “I expect to have…” Does Data Exist? Type of Tool Who could Provide the Data? (Source) Who Can Get the Data? (Collector) Design How Many? Timeframe Is This Feasible? 4.1 Does the system report in a timely fashion?  Time intervals between reporting by patient to Emergency Department, transfer of data from hospital to BCIRPU, and dissemination of information in reports within eight months.  Yes X No Report on time sequence BCIRPU BCIRPU Descriptive BCIRPU 2002+ X Yes  No Beyond March, 2002 4.2 What plans have been made for academic presentations (i.e. abstracts submitted)?  Abstracts submitted to academic organizations. X Yes  No Abstracts BCIRPU BCIRPU Descriptive BCIRPU March 2002 X Yes  No 4.3 Has a process been developed to address when and in what format aggregated data will become available?  A plan for BCIRPU to provide access to aggregated data for participating hospitals and regions (e.g. Internet access via a web interface).  Yes X No Report on process Health Data Warehouse BCIRPU Descriptive BCIRPU March 2002 X Yes  No 4.4.1 Have hospital and regional participants as identified in the dissemination plan received data reports? 4.4.2 Has the feedback from data report evaluation forms been incorporated into the reporting structure?  Data sent to hospitals and regions identified in the dissemination plan at time intervals specified in the contract with Health Canada.  Feedback from data report evaluation has been used to improve the reporting structure.  Yes X No BCIRPU Data Report Evaluation and Needs Assessment Forms BCIRPU Recipients of Data Reports BCIRPU Descriptive BCIRPU SFHR Nov 2001 X Yes  No 97 4.0 Dissemination “I expect to have…” Does Data Exist? Type of Tool Who could Provide the Data? (Source) Who Can Get the Data? (Collector) Design How Many? Timeframe Is This Feasible? 4.5.1 How are new regions engaged in EDISS? 4.5.2 Have non-participating hospitals or regions expressed interest in joining the EDISS initiative?  A system delineating how new regions can join the EDISS initiative.  A summary of first steps of regions currently participating in EDISS.  Requests for information regarding EDISS from non-participating Health Regions expressing interest in joining this initiative. X Yes  No Report from BCIRPU Telephone survey BCIRPU Key Informants BCIRPU Descriptive BCIRPU March 2002 X Yes  No 4.6 Did the newsletter enhance understanding of the project by Regional Health Boards, ED staff & managers, and program directors? Feedback regarding the bimonthly newsletter made available to ED and HR staff, hospital strategic planning teams.  Yes X No Key Informant Survey Hospital Mangers, Regional Medical Health Officers BCIRPU Survey Key informants from each hospital, regional health board Nov 2001 X Yes  No 4.7 Has ED injury surveillance data been used to plan and evaluate injury prevention? At least one injury prevention program or program evaluation within each participating region.  Yes X No Key Informant Survey Hospital Mangers, Regional Medical Health Officers BCIRPU Survey Key informants from each hospital, regional health board March 2002 X Yes  No Beyond March, 2002 4.8 Are surveillance findings reviewed by hospital or regional executive boards as part of resource allocation activities? Information from the ISPP is incorporated into administrative reporting and decision-making activities.  Yes X No Key Informant Survey Hospital Mangers, Regional Medical Health Officers BCIRPU Survey Key informants from each hospital, regional health board Feb 2002 X Yes  No 4.9 Has the EDISS stimulated research proposals/projects relevant to injury prevention?  At least one research proposal/project relevant to injury prevention is planned within each participating region.  Yes X No Telephone Survey Hospital Mangers, Regional Medical Health Officers BCIRPU Descriptive Key informants from each hospital, regional health board 2002+ X Yes  No Beyond March, 2002 98 4.0 Dissemination “I expect to have…” Does Data Exist? Type of Tool Who could Provide the Data? (Source) Who Can Get the Data? (Collector) Design How Many? Timeframe Is This Feasible? 4.10 Who are the champions of EDISS within participating Health Regions and what are their roles?  A champion for EDISS identified within each participating Health Region.  Yes X No BCIRPU Key Informant Survey Hospital Mangers, Regional Medical Health Officers BCIRPU Survey 10 hospitals March 2002 X Yes  No 99APPENDIX III Financial Summary 100 EMERGENCY DEPARTMENT INJURY SURVEILLANCE SYSTEM INJURY SURVEILLANCE PILOT PROJECT FINANCIAL SUMMARY Start-up Costs: Annual Ongoing Costs: Please specify (where applicable): • Level of Health Records positions • Portion of FTE required • Hourly wage or annual salary • Number of hours required Health Records Personnel task include: • Abstracting & coding • Data scrambling & submission • Data quality checks IS Support tasks include • Setup of databases • Routine for data submission • Support of Databases 101 Emergency Department Injury Surveillance Project Startup Costs1 Health Records Trainer (optional)2 HR level _____ at _____ FTE @ $_____/year for 3 months $_______Health Records Personnel: HR level _____ at _____ FTE @ $_____/year for 3 months $_______HR level _____ at _____ FTE @ $_____/year for 3 months $_______HR level _____ at _____ FTE @ $_____/year for 3 months $_______IS Support: Hardware (procurement, installation) - _____ hrs.Extraction routines from admitting to abstracting systems - _____ hrs.Design/Program encryption routines for data exchange - _____ hrs.General administration (meetings, telephone) - _____ hrs.Total: _____ hours @ $___/hr $_______Equipment and Other Charges3 (Please specify) Hardware/Software: $_______(must be based on needs of each site) Total for Startup Costs $______1. Startup Costs cover 3 months of activity. 2. Training is for EDISS abstraction on NACRS abstracting system effective April 2002. • Assumes abstracters trained already in ICD-10 CA inpatient abstracting system. • Gives increased training time. 3. Licensing costs of NACRS abstract not included in this budget. 102 Emergency Department Injury Surveillance Project Annual Ongoing Costs1 Health Records Personnel: HR level _____ at _____ FTE @ $_____/year $_______HR level _____ at _____ FTE @ $_____/yearHR level _____ at _____ FTE @ $_____/yearIS Support IS Support of databases @ $_____/hr for _____ hrs/month for 12 months $_______ Total Annual Costs $______ 1. Annual costs cover 12 months of activities. Annual ongoing costs need to be pro-rated to 9 months and added to 3-month setup costs to give correct cost for first year of project 103APPENDIX IV Health Records Personnel Form The Health Records Personnel Form collects information regarding the specific personnel actually recruited to work on the ISPP at the hospital level. 104 Health Records Personnel We would appreciate your help in evaluating the Emergency Department Injury Surveillance System pilot project. The purpose of this form is to document Health Records positions required for data collection and abstraction for the minimum data set. Please fax your completed form to the BC Injury Research and Prevention Unit at (604) 875-3569. Name ______________________________________ Date ________________ Organization______________________________________________________ Position__________________________________________________________ For each hospital (if more than one), please specify all Health Records positions required for the purpose of this pilot project. Hospital Level of Health Records Position FTE Filled • internally • externally • not filled 105If positions were not filled, please explain why: _____________________________ ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ Please describe any additional barriers to filling positions encountered for the EDISS project. ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ ___________________________________________________________________ Thank you for your participation in this evaluation. 106 APPENDIX V Chart Audit Form Two versions of the Chart Audit Form exist:  Electronic E-form  Manual Form E-Form Version 107Manual Version Emergency Department Injury Surveillance System CHART AUDIT Chart Number PHN Postal Code Gender Date of Birth dd/mm/yyyy Date of Visit dd/mm/yyyy Data Element Documentation in Chart Diagnosis External Cause Long Audits Only Place of Occurrence Activity When Injured Visit Disposition Any other information pertaining to injury-event Auditor's Comments __________________________ Auditor Name 108APPENDIX VI Key Informant Surveys: Parts I & II 109 Emergency Department Injury Surveillance System Key Informant Survey We would appreciate your help in evaluating the Emergency Department Injury Surveillance System pilot project. Your feedback is valuable and will be kept confidential. Please fax your completed form to the BC Injury Research and Prevention Unit at (604) 875-3569. Name _________________________________________ Date __________________ Organization___________________________________________________________ Position_______________________________________________________________ (Circle one) 1. Have you heard of the Emergency Department Injury Surveillance System (EDISS)? Yes / No2. Are you aware that injury data are being collected from your hospital/regional Emergency Department(s)? Yes / No3. Do you play a role in this data collection system? Yes / No4. Are you familiar with the data elements being collected? Yes / No5. Have you seen the EDISS Poster in your Emergency Department? Yes / No6. If Yes, did the Poster raise your awareness regarding the EDISS data collection? No Somewhat Yes 1 2 3 7. Did you receive the EDISS Pocket Booklet? Yes / No8. If Yes, was the Pocket Booklet useful to you for charting the data elements? Not Useful Somewhat Useful 1 2 3 9. Have you seen the EDISS Newsletter? Yes / No10. If Yes, have you read the Newsletter? Yes / No11. If Yes, did the Newsletter raise your awareness regarding the EDISS data collection? No Somewhat Yes 1 2 3 Next Page 110 12. Can you describe what you liked about the Newsletter? ______________________________________________________________________ ______________________________________________________________________ 13. Can you describe what you did not like about the Newsletter? ______________________________________________________________________ ______________________________________________________________________ 14. Can you describe any issues you would like covered in the Newsletter? ______________________________________________________________________ ______________________________________________________________________ 15. Do you think that Injury Prevention should be a priority for Health Regions? No Somewhat Yes 1 2 3 16. Do you think that data collection can assist in planning and evaluating Injury Prevention programs? No Somewhat Yes 1 2 3 17. Would you like to see the Emergency Department Injury data continue to be collected? Yes / No 18. Are you familiar with the Narrative Statement element of the data set? Yes / No 19. Do you think that the Narrative Statement can provide information for injury prevention initiatives? No Somewhat Yes 1 2 3 20. Can you name any of the champions (strong supporters) of this surveillance system within your hospital/region, and the role that they play? ______________________________________________________________________ ______________________________________________________________________ Thank you for your participation in this evaluation. 111 Key Informant Survey - Part II 21. Can you describe how surveillance data has been used in the past by your hospital/region to plan or evaluate injury prevention programs? ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ 22. Do you anticipate your hospital/region to use the Emergency Department injury data to plan or evaluate injury prevention programs? Yes / No If Yes, please describe. ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ 23. Do you anticipate reports produced using the Emergency Department Injury data to be reviewed by your executive board and incorporated into administrative reporting and decision making activities? Yes / No If Yes, please describe. ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ Next Page 112 24. Do you anticipate these reports to be used in determining resource allocation? Yes / No 25. Can you describe how injury surveillance activities are or might be integrated into strategic plans? ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ 26. Are you aware of any efforts undertaken to secure funding either internally or externally for the continuation of this Emergency Department Injury data collection? Yes / No If Yes, please describe. ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ 27. Do you have any further comments regarding the Emergency Department Injury Surveillance System? ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ Thank you for your participation in this evaluation. 113 APPENDIX VII Completeness of Charting Chart Audit Results19 Charting Completeness – Pre & Post Training Thompson Health Region: Royal Inland Hospital (Kamloops) n = 25 Place of Occurrence Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.72 0.76 0.04 0.1239 0.00 1.00 -0.24 to 0.32 Activity When Injured Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.40 0.32 0.08 (negative) 0.1353 0.29 0.768 -0.22 to 0.38 External Cause of Injury Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.80 0.76 0.04 (negative) 0.1170 0.00 1.00 -0.23 to 0.31 Diagnosis Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.84 0.88 0.04 0.0980 0.00 1.00 -0.19 to 0.27 19 The number of charts audited at each individual site is insufficient to provide the power to assess if the results are statistically significant. Combined results are provided with Evaluation Question 1.3.1. 114 Charting Completeness – Pre & Post Training Thompson Health Region: Royal Inland Hospital (Kamloops) n = 25 Using a stricter rating of completeness for External Cause and Diagnosis, based on the ability to code for ICD-9/ICD-10, the following results were produced: External Cause of Injury Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.76 0.64 0.12 (negative) 0.1285 0.62 0.537 -0.17 to 0.41 Diagnosis Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.72 0.64 0.08 (negative) 0.1353 0.30 0.762 -0.22 to 0.38 Charting Completeness – Pre & Post Training North West Health Services Society: Prince Rupert Regional Hospital n = 25 Place of Occurrence Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.88 0.92 0.04 0.0847 0.00 1.00 -0.16 to 0.24 Activity When Injured Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.64 0.68 0.04 0.1339 0.00 1.000 -0.26 to 0.34 115 Charting Completeness – Pre & Post Training North West Health Services Society: Prince Rupert Regional Hospital n = 25 Cause of Injury Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 1.00 1.00 Diagnosis Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.96 0.92 0.04 (negative) 0.0669 0.00 1.00 -0.13 to 0.21 Using a stricter rating of completeness for External Cause and Diagnosis, based on the ability to code for ICD-9/ICD-10, the following results were produced: Cause of Injury Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.96 0.96 0.00 0.0554 0.00 1.00 -0.15 to 0.15 Diagnosis Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.88 0.80 0.08 (negative) 0.1031 0.39 0.700 -0.16 to 0.32 116 Charting Completeness – Pre & Post Training South Fraser Health Region: Surrey Memorial Hospital n = 25 Place of Occurrence Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.92 0.68 0.24 (negative) 0.1079 1.77 0.077 -0.01 to 0.49 Activity When Injured Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.44 0.36 0.08 (negative) 0.1381 0.29 0.773 -0.23 to 0.39 Cause of Injury Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.96 0.92 0.04 (negative) 0.0669 0.00 1.000 -0.13 to 0.21 Diagnosis Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.88 0.96 0.08 0.0759 0.52 0.602 -0.11 to 0.27 Using a stricter rating of completeness for External Cause and Diagnosis, based on the ability to code for ICD-9/ICD-10, the following results were produced: Cause of Injury Pre-training proportion Post-training proportion 0.84 0.84 117 Charting Completeness – Pre & Post Training South Fraser Health Region: Surrey Memorial Hospital n = 25 Absolute Difference Standard error of difference z = p = 95% CI of difference 0.00 0.1037 0.00 1.000 -0.24 to 0.24 Diagnosis Pre-training proportion Post-training proportion Absolute Difference Standard error of difference z = p = 95% CI of difference 0.68 0.88 0.20 0.1137 1.37 0.172 -0.06 to 0.46 118 APPENDIX VIII Sensitivity and Specificity Chart Audit Results20 Reported Injury Cases THR: Royal Inland Hospital Injuries identified by BCIRPU Audit Injury Non-Injury Total Injury 75 1 76 Non-Injury 11 117 128 Injuries Identified By Health Records Total 86 118 204 Sensitivity = 87.2% (80.1% - 94.2%) Specificity = 99.2% (97.5% - 100.0%) Reported Injury Cases NWHSS: Prince Rupert Regional Hospital Injuries identified by BCIRPU Audit Injury Non-Injury Total Injury 43 3 46 Non-Injury 10 144 154 Injuries Identified By Health Records Total 53 147 200 Sensitivity = 81.1% (70.5% - 91.6%) Specificity = 98.0% (95.7% - 100.0%) Reported Injury Cases SFHR: Surrey Memorial Hospital Injuries identified by BCIRPU Audit Injury Non-Injury Total Injury 81 1 82 Non-Injury 21 181 202 Injuries Identified By Health Records Total 102 182 284 Sensitivity = 79.4% (71.5% - 87.2%) Specificity = 99.4% (98.2% - 100.0%) 20 The number of charts audited at each individual site is insufficient to provide the power to assess if the results are statistically significant. Combined results are provided with Evaluation Question 1.6.
Report
eng
Unreviewed
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Faculty
Researcher
Injury policy and prevention
Injury Surveillance Pilot Project : Evaluation Report
Text