IDENTIFYING HIGH-RISK C L A I M S WITHIN THE W O R K E R S ' COMPENSATION B O A R D OF BRITISH COLUMBIA'S C L A I M I N V E N T O R Y B Y USING LOGISTIC REGRESSION M O D E L I N G by , ERNEST U R B A N O V I C H Ph.D. (Chemistry) University of Bucharest 1992 A THESIS SUBMITTED IN P A R T I A L F U L F I L M E N T OF THE REQUIREMENTS FOR THE D E G R E E OF M A S T E R OF SCIENCE IN BUSINESS ADMINSITRATION in THE F A C U L T Y OF G R A D U A T E STUDIES F A C U L T Y OF C O M M E R C E A N D BUSINESS ADMINISTRATION We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH C O L U M B I A December 1999 © Ernest Urbanovich, 1999 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for copying of this thesis for scholarly purposes department or by his or her may be granted by the head of my representatives. It is understood that publication of this thesis for financial gain shall not be allowed without permission. Department of N-C - 0?hKWCt The University of British Columbia Vancouver, Canada Date DE-6 (2/88) DEC 2.1, *W extensive O U f f ^ f #0/% copying or my written 11 ABSTRACT The goal of the project was to use the data in the Workers' Compensation Board (WCB) of British Columbia's data warehouse to develop a statistical model that could predict on an ongoing basis those short-term disability (STD) claims that posed a potential high financial risk to the W C B . We were especially interested in identifying factors that could be used to model the transition process of claims from the STD stratum to the vocational rehabilitation (VR) and long term disability (LTD) strata, and forecast their financial impact on the W C B . The reason for this focus is that claims experiencing these transitions represent a much higher financial risk to the W C B than claims that only progress to the health care ( H Q and/or the short term disability (STD) strata. The sample used to investigate the conversion processes of claims consists of all STD claims (323,098) that had injury dates between January 1, 1989 and December 31, 1992. Although highrisk claims represent only 4.2 % of all STD claims, they have received 64.3% ($1.2 billion) of the total payments and awards ($1.8 billion) made to July 1999. Low-risk claims make up 95.8% of all the claims but only receive 35.7% ($651 million) of the payments and awards. Moreover, the average cost of high-risk claims ($86,200) is 41 times higher than the average cost of lowrisk claims ($2,100). The main objective of the project was to build a reliable statistical model to identify high-risk claims that can be readily implemented at the W C B and thereby improve business decisions. To identify high-risk claims early on, we used logistic regression modeling. Since ten of the most frequently observed injury types make up 95.72% of all the claims, separate logistic regression models were built for each of them. Besides injury type, we also identified STD days paid and age of claimant as statistically significant predictors. The logistic regression models can be used to identify high-risk claims prior to or at the First Final STD payment date provided we know the injury type, STD days paid and age of claimant. The investigation showed that the more STD days paid and the older the injured worker, the higher the probability of the claim being highrisk. T A B L E OF CONTENTS ABSTRACT LIST OF T A B L E S LIST OF FIGURES ACKNOWLEDGEMENTS I. INTRODUCTION 1.1 The Workers' Compensation Board of British Columbia 1.2 Risk Management at the W C B 1.3 Project Background 1.4 Literature Review II. W C B PROCESSES A N D F I N A N C I A L ASPECTS II. 1 Claim Processing at the W C B 11.2 Converted, Active, and Inactive STD Claims 11.3 Common Paths for FFSTD Claims 11.4 Classification of STD Claims by Injury Type 11.5 Financial Aspects III. M E T H O D O L O G Y III. 1 Data Sources and Collection 111.2 Logistic Regression 111.3 Stratification of Data IV. A P P L I C A T I O N IV. 1 Model Building IV.2 The Cutoff Point IV. 3 Cost Analysis of the Optimal Cutoff Point IV.4 Accuracy of the Models IV. 5 Cross-validation of the Models IV.6 Critical STD Days Paid V. IMPLEMENTATION VI. A R E A S FOR FURTHER INVESTIGATION VII. C O N C L U S I O N REFERENCES APPENDIX Histograms of the transition times of paths 1, 2, 3 and 4 LIST OF T A B L E S II. 1 Descriptive statistics of the transition times of paths 1,2,3 and 4 11.2 Fitted theoretical distributions of the transition times involved in paths 13 1,2, 3 and 4 14 11.3 Distribution of claims by injury type 15 11.4 Description of the most frequently observed injury types 16 II. 5 Distribution of payments and awards by injury type (up to July 1999) 18 II.6 Cost per claim by injury type 19 IV. 1 The logistic regression models 30 IV.2 The coefficient of determination between age and STD days paid 32 IV.3 Estimated parameters of the logistic regression models 33 IV.4 Percentages of converted, non-converted and all claims correctly classified by the Fracture model, and the corresponding workload for various values of the cutoff point 37 IV. 5 Cutoff points for the logistic regression models 40 IV.6 The number of converted claims incorrectly predicted, the number of non-converted claims incorrectly predicted, and the expected total cost of incorrect predictions for various values of the cutoff point for the Fracture model (A = 6 and B = 1) 43 IV. 7 The optimal cutoff points for various values of the A / B cost ratio 44 IV. 8 Accuracy of the regression models when the CF is the cutoff point (1989-1992 sample) IV.9 46 Accuracy of the regression models when ECP is the cutoff point (1989-1992 sample) 46 IV. 10 Model cross-validation (CF is the cutoff point) 48 IV. 11 Model cross-validation (ECP is the cutoff point) 48 IV. 12 Critical STD days paid (CF is the cutoff point) 50 IV. 13 Critical STD days paid (ECP is the cutoff point) 50 VI LIST OF FIGURES II. 1 Common paths for FFSTD claims 12 II. 2 Distribution of all claims and converted claims by injury type 16 III. 1 Converting Crystal Reports files to SPPS 21 IV. 1 Time line for decision making IV.2 Probability of conversion as a function of STD days paid for the Contusion, files Laceration and Fracture models (age of claimant is 40) IV. 3 34 Probability of conversion as a function of STD days paid and age of claimant for the Sprain/Strain model IV.4 31 35 Correct classification percentages for the Fracture model for various values of the cutoff point 38 IV.5 The workload for the Fracture model as a function of the Cutoff Point 39 IV. 6 The expected total costs of incorrect predictions for B=l and four specific values of the A cost 44 IV.7 The optimal cutoff point as a function of the A / B cost ratio 45 A. 1 Path 1 - Histogram of the Injury Date to FFSTD payment date Transition 59 A.2 Path 1 -Histogram of the FFSTD to F V R Transition Time 59 A.3 Path 1 - Histogram of the F V R to F L T D Transition Time 60 A.4 Path 2 - Histogram of the Injury Date to FFSTD payment date Transition Time 60 A.5 Path 2 - Histogram of the FFSTD to F V R Transition Time 61 A.6 Path 3 - Histogram of the Injury Date to FFSTD payment date Transition Time 61 A.7 Path 3 - Histogram of the FFSTD to F L T D Transition Time 62 A. 8 Path 4 - Histogram of the Injury Date to FFSTD payment date Transition Time 62 A.9 Path 4 - Histogram of the FFSTD to F L T D Transition Time 63 A . 10 Path 4-Histogram of the F L T D to F V R Transition Time 63 ACKNOWLEDGEMENTS I take pleasure in expressing my gratitude to those who have helped me with this project: Ella Young - Risk Manager, W C B Martin Puterman - Professor, U B C Sidney Fattedad - V P Finance and Information Services, W C B Brian Van Snellenberg - Risk Manager, W C B Jonathan Berkowitz - Professor, U B C Shelby Brumelle - Professor, U B C 1 I. INTRODUCTION 1.1 The Workers' Compensation Board of British Columbia The Workers' Compensation Board (WCB) of British Columbia is a statutory agency responsible for representing the occupational health and safety, rehabilitation, and compensation interests of the province's workers and employers. Created in 1917, the W C B ' s main objective is to assist workers and employers to ensure safe workplaces, income security and safe return to work for injured workers. The workers' compensation system was founded on what is known as the "historic compromise", in which the risk of economic loss through personal injury or occupational disease resulting from employment should be borne by industry, and the cost considered as part of the costs of production. Accordingly, the funds that the Board needs to make compensation payments and meet its other obligations are provided from assessments levied on employers by the Assessment Department of the Board. In return, the employers receive protection from lawsuits arising from work-related injuries and diseases. Moreover, as a part of this historic compromise, injured workers receive the right to benefits on a no-fault basis. The Workers Compensation Act is the legal document that guides the W C B ' s operations. The Act gives the Board the official authority to set and enforce occupational safety and health standards, provide compensation and rehabilitation to injured workers or their dependants, and collect funds from business to operate the workers' compensation system. In 1998, the W C B served approximately 160,000 employers who employed about 1.8 million workers in British Columbia. The WCB's main objectives are: • preventing workplace injuries, diseases, and fatalities, • rehabilitating injured workers and returning them to work, • providing fair compensation for workers suffering from an occupational injury or disease, • providing sound financial management for a viable workers' compensation system, and • protecting the public interest! 2 The W C B ' s incomes are composed of premiums paid by employers and investment income. O f the W C B ' s 1998 total income of $1.6 billion, the employers paid $917 million in premiums. The investment portfolio of stocks and bonds of the W C B had a market value of $7.7 billion at yearend 1998, and provided an average market return of 11 percent. As regards specific costs, in 1998, the W C B spent approximately $1.1 billion on compensation and rehabilitation, up $101 million from the previous year due to an increase in claim duration. During the same period operating expenses totaled $226 million, an increase of 12.1 percent over 1997 due to increased investment in prevention services and technology. Despite the increased costs and expenses, in 1998, the W C B achieved an operating surplus of $289 million. See the 1998 Annual Report of the W C B for more financial highlights. 1.2 Risk Management at the W C B The Risk Management Group within the Finance Department of the W C B was created in 1998 to protect the W C B against underwriting exposure and loss. This means the Risk Management Group's main objective is to protect the Board from both existing and emerging risks just over the horizon that need to be identified and quantified. Accordingly, the Group works closely with other corporate departments to identify, assess, and help resolve long-term threats to the W C B ' s financial stability. In particular, the Risk Management Group focuses on potential threats posed by funding and cost trends within individual industry classes (e.g., Logging or Building). Most of the projects the Risk Management Group has been working on since its inception have involved operational research, and statistical analysis, and have came up with suggested actions to minimize the risk exposure of the Board. Another key responsibility of the Group has been to collaborate with the Information Services Division and other business units to develop new tools the W C B can use to improve the business decision making process. One of these is the Data Warehouse, which is like a central data warehouse for decision-makers that includes a compilation of finance, claim, assessment, and other relevant decision-support data from all W C B sources (departments and services). Risk Management analyzes the data in search of trends and performance indicators to comprehend better the middle and long-term financial implications that might lie within. 3 1.3 Project Background The project was initiated by Risk Manager Ella Young from the Risk Management Group and Sidney Fattedad, Vice-President of Finance and Information Services of the W C B . In May 1999, Ella Young contacted Professor Martin L . Puterman from the Faculty of Commerce of the University of British Columbia (UBC) in order to explore a joint effort between the W C B and the U B C to develop a model to identify financially high risk claims within the W C B ' s inventory. During the initial meeting with the Risk Management Group, Professor Puterman suggested that a M S c student in Management Science associated with the Centre for Operations Excellence (COE) at the University of British Columbia could work on the project. They also agreed that, i f successfully completed, the.MSc student would use the most important results of the project to develop his or her Masters' thesis. Created in January 1998, the C O E supports education and research at U B C through affiliations with leading Canadian private and public sector organizations and its extensive international linkages with leading applied research programs. Subsequently to a meeting between Professor Puterman and the Risk Management Group, M S c student Ernest Urbanovich joined the W C B on May 19, 1999, and started working on the project under Ella Young's direct coordination. Professor Puterman, the C O E director, advised Ernest Urbanovich on the scientific components of the project and supervised his MSc thesis. The goal of the project was to use the data in the W C B ' s data warehouse to develop a model that could predict on an ongoing basis those claims that posed a potential risk of being reopened at any time in the future. Initially, the term reopening was used to describe the process in which a claim received any type of additional payments after the first final short-term disability (FFSTD) payment. In addition, there was a need to identify attributes and risk characteristics of claims in the system that, after being inactive for some time, could become a large financial risk for the W C B . The model should have allowed the W C B to achieve a better understanding of where financial risk exposure exists, and the magnitude of those risks due to reopened claims. The approach suggested was to analyze the population of claims that have had reopenings in the past, and develop a regression model to describe the reopening of claims. 4 The thorough investigation of the claim population showed that reopening was an inadequate term to describe financially high risk claims since the costs associated with almost 95% of the reopened claims proved to be relatively low (see the Financial Aspect section later). Consequently, we introduced "conversion" as a new expression to distinguish between low-risk and high-risk claims. We define a converted claim as one that received vocational rehabilitation and/or long term disability payments after the FFSTD payment. On the other hand, a nonconverted claim is one that receives at most health care and/or short-term disability payments after the FFSTD payment. See the Claim processing at the W C B section in the next chapter for details about different types of payments a claim may receive. The average cost per converted claim is $86,223, which is about 41 times higher than the average cost per non-converted claims ($2,101). It was straightforward thus to consider converted claims as financially high-risk, and non-converted claims as financially low-risk claims. Since the outcome of any FFSTD claim is binary, that is, it can be either converted (high-risk) or not converted (low-risk), we used logistic regression to model the conversion process of claims. We found that the most significant predictors are nature of injury, age of claimant, and number of short term disability days paid. The logistic regression models allow one to classify a given claim as either high-risk or low-risk, and thus one of the most important objectives of the project became the use of these models at the W C B to improve the business decision making process. 1.4 Literature Review In this section we first review three previous W C B studies related to our study in that they also addressed the problem of high-risk claims but from a different perspective and using different approaches. Then, we describe some relevant applications of logistic regression found in literature. Although the literature concerning logistic regression and its applications is considerable and still growing, we restricted our review to a few studies that are relevant to our project. In a recent study, Jessup and Gallie (1996) focused on identifying the characteristics of workers who made 20 or more W C B - B C claims in their working lifetime. The study showed that as of 5 November 1, 1995, there were 15,042 workers who had made 20 or more W C B injury reports. These high-risk workers had made a total of 382,151 injury reports, a number approximately equal to the total number of W C B claims first reported in 1994 and 1995. Even more intriguing was that while a few of the 20+ claims were made as early as 1917, over l / 3 had been made in rd the last 10 years. The authors identified age of worker, gender, nature injury, body part and occupation (industry) as the most important factors in profiling these high-risk claims, but developed no quantitative model that could be used to quantify the risk associated with them. Fattedad and Charron (1998) studied the claims inventory control at the W C B . The study focused on categorization of inventory, distribution of claims cost by benefit type, claims conversion from short term disability (STD) to long term disability (LTD) and/or vocational rehabilitation (VR), and reopening of STD claims. The investigation showed that the number of STD claims that are converted to V R and/or L T D is relatively low, but they are extremely costly. That is, only 7.8% of the claims included in the study are converted, but they account for approximately 65% of costs. To identify high-risk converted claims up front on the business process, the authors suggested using STD days paid as an indicator of conversion, but provided no mathematical model to determine the likelihood of conversion as a function of STD days paid. However, Fattedad and Charron conclude that once a claim goes beyond 70 STD days paid, it is likely to be converted. In a study focused on claim duration, Mason (1999) developed a statistical model for claim duration. The study showed that the WCB's Data Warehouse had information by which a claim duration model could be developed. To determine the factors that are likely to affect claim duration, Mason used analysis of variance. The analysis identified several factors including nature of injury, industry subclass (e.g., Logging), age of claimant, gender, the year of the claim, and the type of the accident. We now describe some related applications of logistic regression. Wiginton (1980) was one of the firsts to describe the results of using logistic regression in credit scoring. The model allowed Wiginton to classify potential applicants for credit into two groups, 6 i.e., good credit risk and bad credit risk applicants. Although Wiginton was not very impressed with the performance of the logistic regression model, it has subsequently become the main approach to the classification step in credit scoring. Johnson (1998) developed a logistic regression model to determine whether local college students should be given credit for future purchases at a campus department store. The data consisted of information collected from students who were given credit during the preceding two years. Some of the variables collected included the students' gender, age, grade point average, college major, and hours worked per week. Then, based on each student's past credit history at the store, each student was classified according to whether the student was a good credit risk or low credit risk, and a logistic regression model was built to describe the data. The investigation showed that age and gender were not statistically significant (a = 0.10), while grade point average, college major, and hours worked proved to be statistically significant predictors. In a survey of credit and behavioral scoring, Thomas (1999) identified logistic regression modeling as one of the most powerful statistical techniques to be used by organizations to decide whether or not to grant credit to consumers who apply to them. As regards the sample used to build the logistic regression models, Thomas emphasizes that usually it can vary from a few thousand to as high as hundreds of thousands. He also recommends that the proportion of good credit risk and bad credit risk applicants in the sample should reflect the proportions in the populations. Thompson (1985) used stepwise logistic regression to study the outcome (success or failure) in a community mental health program. To build the logistic regression model, Thompson used information on 519 client admissions with data on 17 client characteristics such as demographic data, referral data, mental health history, intelligence scores, and follow-up treatment. O f the predictors investigated, age at admission to the program was statistically the most significant with a p-value lower than 0.01. Tabachnick and Fidell (1996) used logistic regression analysis to model and predict work status (employed versus unemployed) of women. The study employed four continuous attitudinal 7 variables: locus of control, attitude toward current marital status, attitude toward women's right, and attitude toward housework. Of the 440 women surveyed, 205 were housewives and 235 were women who worked outside the home more than 20 hours a week. The investigation showed that all four predictors were statistically significant (p-value <0.001) and thus incorporated into the model. The prediction accuracy of the model was rather poor, with 56% of the working women and 49% of the housewives correctly predicted, for an overall prediction rate of 53%. In a landmark study, Lemeshow et al (1988) investigated the survival of patients following admission to an adult intensive care unit. The major goal of the study was to develop a logistic regression model to predict the probability of survival to hospital discharge of the patients. The study employed more than 20 predictor variables such as age, sex, race, service (medical or surgical) at admission, history of chronic renal failure, blood pressure at admission, heart rate at admission, P H from initial blood gases, etc. 8 II. W C B PROCESSES A N D FINANCIAL ASPECTS II. 1 Claim Processing at the W C B A request for compensation under the Workers Compensation Act is called "claim". Not all injuries and diseases are compensable. That is, a compensation request should be on behalf of a worker (known as the injured worker or the claimant) who may have been injured in a workrelated accident, or suffers from an occupational disease, which may be a result of job-related factors. Not everyone is entitled to compensation under the Act, even i f injured at work. A person qualifies for compensation i f he or she is a worker employed by an employer covered by the Act. A n adjudication process determines i f a claim is valid and, i f so, to what compensation benefits the injured worker is entitled. Whenever an injury or disease resulting from a person's employment causes a period of temporary disability from work, the W C B pays wage-loss benefits to the injured worker. Wageloss benefits are also known as short-term disability (STD) benefits; we will further use the second term. Usually, STD benefits commence shortly after the initial acceptance of a claim, and they cease when the injured worker recovers from the injury or the condition becomes a permanent one. Permanent disability awards, also called long-term disability (LTD) awards, are payable when a worker fails to recover completely from a work-related accident or an occupational disease, and is left with a permanent total disability or permanent partial disability. If a worker has a permanent total disability, such as blindness, paraplegia, hemiplegia, and severe loss of cerebral powers, he or she is awarded a periodic payment equal in amount to 75% of his or her average earnings. This amount must be payable during the lifetime of the worker. Where permanent partial disability results from the injury, the compensation must be a periodic payment to the injured worker of a sum equal to 75% of the estimated loss of average earnings resulting from the impairment, and must be payable during the lifetime of the worker. 9 Vocational rehabilitation (VR) is a service provided by the W C B to assist workers in their effort to return to their pre-injury employment or to an occupational category comparable in terms of earning capacity to the pre-injury occupation. V R assistance may be provided in cases where it appears to a V R consultant that such assistance may be of value. Injuries that are likely to be referred immediately to a V R consultant for further consideration are: spinal cord injuries resulting in paraplegia or qudruplegia, major extremity amputations, severe crush injuries, severe brain or brain stem injuries, significant burns, and significant loss of vision. In addition to STD, L T D , and V R compensations, the W C B is responsible for the cost of health care (HC) benefits such as necessary hospitalization, treatment provided by recognized health care professional, nursing and other care or treatment, prescription drugs, and necessary medical appliances. Any claim that receives only health care benefits is called a health-care-only (HCO) claim, while any claim that is entitled to short term disability benefits and/or long term disability benefits is called a non-health-care-only (non-HCO) claim. In 1998, there were 207,019 claims first reported at the W C B . Of the total number of claims first reported, 153,545 (74.17%) claims were actually accepted for HCO and/or non-HCO benefits, 6,100 (2.95%) claims were disallowed, and 2,824 (1.36%) claims were rejected. Disallowed claims are those that fall within the scope of the Workers Compensation Act, but are not payable because they are not work related. Rejected claims are those that do not fall within the scope of the Workers Compensation Act since they represent claims from workers employed in industries not covered under the Act, claims from self-employed workers without optional protection, and accounts from physicians submitted in error to the W C B . Notice that the number of accepted, disallowed, and rejected claims (162,469) only make up 78.48% of the total number of claims (207,019) first reported. This is due to the fact that claims are not necessarily disallowed, rejected, or accepted in the year in which they are reported. As mentioned previously, the focus of this study is STD claims and their movement through various benefit type strata of the W C B ' s claim inventory. The transition from the H C stratum to the STD stratum takes place whenever a claim receives a first STD (FSTD) payment. After spending some time in the STD stratum (and possibly receiving additional STD payments), a 10 claim usually receives a first final STD payment. In spite of the term "final" used here, we should keep in mind that claims are never closed, and that after the first "final" payment has been made, a claim may subsequently be re-opened for additional compensation benefits. The transition from the STD stratum to the V R or L T D strata takes place when the claim receives either a first vocational rehabilitation payment or a first long term disability payment. 11.2 Converted, Active, and Inactive STD Claims The definition of converted, active and inactive STD claims is directly related to their movement through the system from one stratum to another one. A n active STD claim is defined as one that had a First Final STD (FFSTD) payment and then received additional H C and/or STD payments but no V R and/or L T D payments and awards. Similarly, we define a converted claim as one that had a FFSTD payment and then subsequently received either a First V R (FVR) or a First L T D (FLTD) payment. This definition also includes those claims that after the FFSTD payment received additional H C and/or STD payments, but later on received either a F V R or F L T D type of payment. A third category of claims is represented by inactive claims, that is, claims that after the FFSTD payment date receive no additional payments and consequently are called inactive. 11.3 Common Paths for FFSTD Claims The sample used to investigate the conversion process of claims consists of all the claims (323,098) that had injury dates between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment. The reason we have not used more recent data (e.g., claims that had injury dates between 1994 - 1997) is that we wanted to capture as much information as possible regarding the claims investigated. For instance, i f we had used to recent samples (claims with injury date after 1993), we would have missed valuable information such as F V R and/or F L T D payments for a significant number of claims since the life cycle of these types of claims is usually higher than 6 years. See Fattedad and Charron (1998) and Mason (1999) for more details regarding STD claim duration. Thus by using the 1989-1992 sample we tried to avoid the bias that could have been induced into our analysis by using more recent samples. 11 The flowchart presented in Figure II. 1 below shows the six most likely paths the FFSTD claims take through the system and the corresponding average transition times (in brackets) between various benefit type strata. The flowchart is based on the sample of all the claims that received a FFSTD payment and had their injury dates between January 1, 1989 and December 31, 1992. Notice that four paths (1, 2, 3 and 4) actually represent conversions since they lead to V R and/or L T D payments and awards after the FFSTD payment. Although only 4.2 % of the FFSTD claims move through these paths, the payments made for them represent 64.3% ($1,173 million) of the total payments ($1,824 million) made up to July 1999 (see the Financial Aspects section later). Claims that move through path 5 are called active claims, and they represent 79.1% of all the claims from the sample. They receive additional H C and STD payments after the FFSTD payment, but no V R or L T D payments. Path 6 corresponds to inactive claims that after the FFSTD payment date have not received any additional payments. The financial impact on the W C B ' s reserves of active and inactive claims is less significant than the financial impact of converted claims since they make up 95.8% of all the claims but only receive 35.7% ($651 million) of the payments made up to July 1999. Since the significant financial impact of converted claims on the W C B ' s reserves, we will focus further on developing a statistical model to identify them as early as possible in the decision making process. Transition times between various benefit type strata are important for understanding the dynamic nature of the movement of claims through the system. We define the transition time from the STD stratum to the V R stratum as the time between the First Final STD payment date and the First V R payment date, given that there are no L T D payments between the two dates. Similarly, the transition time from the STD stratum to the L T D stratum is defined as the time between the First Final STD payment date and the First L T D payment date, given that there are no V R payments between the two dates. That is, the claim moves directly from the STD stratum into the L T D stratum. Also, we define the transition time from the L T D stratum to the V R stratum as the time between the F L T D payment date and the F V R payment date, given that the F L T D payment occurred before the F V R payment date. Finally, we define the transition time from the injury date to the FFSTD payment date as the time between the two dates. Table II. 1 below gives the descriptive statistics of all the transition times related to converted claims. Figure II.1 Common paths for FFSTD claims 1.34% FLTD FVR Path 1 [17.3 months] [10.6 months] 0.65% NO F L T D FVR Path 2 [8.8 months] 2.09% NO F V R FLTD Path 3 [20.1 months] FFSTD Claim 0.12% FVR FLTD Path 4 [20.8 months] [20.9 months] 71.64% HC O N L Y 79.08% 4.98% STD O N L Y Path 5 2.46% B O T H STD & H C 16.72% Path 6 STD INACTIVE 13 Table II. 1 Descriptive statistics of the transition times of paths 1, 2, 3 and 4 Path 1 Transition Injury Date to Mean Standard Median Minimum Maximum Number of Claims (months) Deviation (months) (months) (months) in Sample 10.28 8.97 8 0 95 4,344 FFSTD 1 FFSTD to FVR 10.57 14.14 4 0 99 4,344 1 FVR to FLTD 17.31 14.03 13 0 99 4,344 2 Injury Date to 7.03 8.62 5 0 121 2,101 . 2,101 FFSTD 2 FFSTD to FVR 8.82 15.04 3 0 113 3 Injury Date to 6.19 7.56 4 0 89 6,766 20.14 17.26 14 0 119 6,766 8.66 8.35 7 0 81 389 20.94 17.86 15 0 101 389 20.85 18.64 17 0 86 389 FFSTD 3 FFSTD to FLTD 4 Injury Date to FFSTD 4 FFSTD to FLTD 4 FLTD to FVR The minimum value of the transition time is zero for all paths. The reason for this is that in each path there were claims for which the injury date and the FFSTD payment date were in the same month, and/or the FFSTD and F L T D (or FVR) payment dates were in the same month. Notice that the highest average transition times correspond to the FFSTD to F L T D and F L T D to F V R transitions respectively, while the lowest average transition times are those corresponding to the injury date to the FFSTD payment date transitions. None o f the transition times are normally distributed (bell shaped). To illustrate, the Appendix presents all the histograms of the transition times involved in paths 1,2,3 and 4. We have also used the Arena simulation software package to fit several theoretical distributions to the data, and determined those that provided the best fit; see Kelton et al (1998) for more details regarding Arena and its theoretical distributions available for fitting. As regards the results obtained, they are summarized in Table II.2 below. 14 Table II.2 Fitted theoretical distributions of the transition times involved in paths 1, 2, 3 and 4 Path Transition Expression* Distribution Mean Squared Error 1 Injury Date to Weibull WEIB(10.7, 1.17) 0.003984 FFSTD 1 FFSTD to FVR Beta 99xBETA(0.289, 2.55) 0.006844 1 FVR to FLTD Erlang ERLA(8.65, 2) 0.004874 2 Injury Date to Weibull WEIB(6.26, 1.01) 0.002660 FFSTD 2 FFSTD to FVR Beta 113xBETA(0.239, 2.83) 0.002251 3 Injury Date to Exponential EXPO(6.19) 0.002424 FFSTD to FLTD Erlang ERLA(10.1,2) 0.010328 Injury Date to Weibull WEIB(8.93, 1.09) 0.002729 FFSTD 3 . 4 FFSTD 4 FFSTD to FLTD Weibull WEIB(21.9, 1.15) 0.009204 4 FLTD to FVR Beta 86xBETA(0.706, 2.21) 0.006290 *SeeKeltonetai (1998) br notation The Injury Date to FFSTD transition times for paths 1, 2, and 4 are best described by the Weibull distribution, while for path 3 the corresponding transition time is described by the exponential distribution. The FFSTD to F V R transition times are best described by the Beta distribution. As regards the FFSTD to F L T D transition times, for path 3 Erlang is the best distribution, while for path 4 Weibull is the most appropriate distribution. II.4 Classification of STD Claims by Injury Type Nature of injury is a classification of the injury or illness in terms of its principal physical characteristics. Nature of injury (NOI) classifications are provided by, the common coding system that was developed by the National Work Injuries Statistics Program (NWISP) of Canada. Since nature of injury appeared to be the most useful variable for discriminating and 15 grouping claims, we used it as the primary classification variable to model the conversion process of FFSTD claims; see the Stratification of Data section later for more details. Consequently, we investigated the distribution of claims that move through paths 1, 2, 3, 4, 5 and 6 as a function of injury type. Table II.3 below summarizes the results by providing the distribution of all claims and converted claims respectively by ten of the most frequently observed injury types. Notice they make up 95.72% of all claims from our sample. See also Figure II.2 for a graphical illustration of the information presented in Table II.3. The description of the nature of injury type codes is presented in Table II.4. Table II.3 Distribution of claims by injury type Nature of A l l claims Distribution of Converted claims Distribution injury (all paths) all claims (%) (paths 1-4) of converted Conversion Factor(%) claims (%) type code 00100 701 0.22 583 4.28 83.17 00120 7,193 2.23 142 1.04 1.97 00160 51,069 15.81 1,405 10.32 2.75 00170 44,617 13.81 1,852 13.60 4.15 00210 16,265 5.03 2,575 18.92 15.83 00261 4,903 1.52 385 2.83 7.85 00262 11,158 3.45 645 4.74 5.78 00264 1,718 0.53 197 1.45 11.47 00300 11,690 3.62 63 0.46 0.54 00310 159,947 49.50 5,102 37.48 3.19 Other 13,837 4.28 665 4.88 4.81 TOTALS 323,098 100 13,614 100 4.21 16 Figure II.2 Distribution of all claims and converted claims by injury type Distribution of Claims by Injury Type 90 • Percent of all 80 H Percent of converted • Conversion Factor (%) I 70 60 170 210 261 262 264 Nature of injury type code Table II.4 Description of the most frequently observed injury types Nature of Injury Type Description of the Nature of Injury Type Code 00100 A M P U T A T I O N OR E N U C L E A T I O N 00120 B U R N OR SCALD(HEAT) (HOT SUBSTANCES) 00160 CONTUSION, CRUSHING, BRUISE(SOFT TISSUE) 00170 CUT, L A C E R A T I O N , P U N C T U R E - OPEN W O U N D 00210 FRACTURE 00261 BURSITIS (EPICONDYLITIS, TENNIS ELBOW) 00262 TENOSYNOVITIS, SYNOVITIS, TENDONITIS 00264 CARPAL TUNNEL SYNDROME 00300 SCRATCHES, ABRASIONS (SUPERFICIAL WOUND) 00310 SPRAINS, STRAINS Other 17 To have a qualitative measure of the extent of conversion within each subset of claims determined by injury type, we introduced the terminology conversion factor. The conversion factor (CF) is the proportion of claims in a given category that have been converted: CF = Number of converted claims/Total number of claims or CF(%) = CFxlOO Table II.3 and Figure II.2 also show the conversion factors for each injury type. Notice that the average conversion factor is 4.2% while for scratches and abrasions (nature of injury type code 00300) the CF is only 0.54%. On the other hand, for amputations or enucleations (nature of injury type code 00100) the CF is very high at 83.19%. This means that scratches and abrasions are less likely to be converted than the average claim, whereas amputations or enucleations have a much higher likelihood of being converted than the average claim. The CF is a key indicator for comparing various subsets of claims determined by nature of injury type or other potential classification variable (e.g. industry), and will be used later as a cutoff point for the logistic regression models. II.5 Financial Aspects To study the financial impact of converted claims on the W C B ' s reserves, we determined the payments and awards received up to July 1999 by the set of claims (323,098) that had injury date between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment. Table II.5 below summarizes the cost of all claims versus the cost of converted claims broken down by the most frequently observed injury types. Then, using the information presented in Tables II.3 and II.5 we evaluated the average costs per claim by injury type. The appropriate results are given in Table II.6. Notice that although converted claims only make up 4.2 % of all the claims, they incur 64.3% ($1,173 million) of the total payments and awards ($1,824 million) up to July 1999. On the other 18 hand, active and inactive claims represent 95.8% of all claims from the sample, but they only incur 35.7% ($651 million) of the payments up to July 1999. Thus, from a financial point of view, converted claims represent high-risk claims while active and inactive claims that move through paths 5 and 6 can be grouped together and categorized as low-risk claims. The costs per claim also support the significant differences between the two types of claims. For instance, the average cost per low-risk claim is about $2,101 whereas the average cost per high-risk claim is about 41 times higher ($86,223). Table II. 5 Distribution of payments and awards by injury type (up to July 1999) Nature of Payments and Payments and awards Distribution of Distribution of injury type awards ($) for ($) for payments and awards payments and code converted claims all claims for converted claims awards for all (Paths 1-4) (Paths 1-6) (%) claims (%) 00100 24,072,419.38 24,463,117.25 2.05 1.34 00120 8,834,650.62 15,663,582.83 0.75 0.86 00160 140,330,268.72 219,223,971.31 11.97 12.02 00170 75,612,215.80 121,556,802.03 6.45 6.67 00210 208,689,899.40 264,153,250.08 17.80 14.48 00261 27,751,981.11 44,721,521.47 2.37 2.45 00262 57,046,331.10 84,465,146.10 4.86 4.63 00264 13,320,066.64 22,472,430.00 1.14 1.23 00300 5,972,470.79 11,644,754.22 0.51 0.64 00310 539,983,562.51 911,368,465.38 46.05 49.97 Other 71,017,601.67 103,952,340.03 6.05 5.71 1,172,631,467.74 1,823,685,380.70 100 100 TOTALS 19 Table II.6 Cost per claim by injury type Nature of Cost per claim ($) for injury type low-risk claims code (Paths 5 and 6) Cost per claim ($) for Cost per claim ($) for high-risk claims (Paths 1, all claims (Paths 1, 2, 3, 2, 3 and 4) 4, 5, and 6) 00100 3,311.00 41,219.90 34,847.75 00120 967.27 62,215.85 2,174.89 00160 1,587.24 99,666.38 4,289.01 00170 1,073.30 40,827.33 2,721.89 00210 4,047.24 80,981.72 16,224.63 00261 3,751.83 72,270.78 9,113.82 00262 2,604.37 8.8,857.21 7,561.79 00264 6,017.33 69,015.89 13,111.10 00300 487.44 94,801.12 995.28 00310 2,395.37 106,066.31 5,691.29 Other 2,496.02 106,957.83 7,500.54 Average Cost 2,101.25 86,222.90 5,638.39 20 III. M E T H O D O L O G Y III. 1 Data Sources and Collection The data concerning STD claims are available in the Data Warehouse (DW) of the W C B . The Data Warehouse is a single integrated source of information formed by collecting data from multiple sources, and then transforming and summarizing this information to enable improved decision making at the W C B . It is important to stress that the D W is not an on-line transaction system, and typically does not generate data for other applications. The data come into the D W from several sources (databases) maintained by various departments and services within the W C B (e.g., Statistics Department and Compensation Services), and are refreshed monthly. To access the data stored in the D W we used a specialized analytical tool known as Crystal Reports. Crystal Reports is a report writing tool produced by Seagate Software, and is considered by many the world standard for desktop reporting and design; see Peck (1999) for more details. Crystal Reports is produced in a stand alone version or as a component of Crystal Info, the reporting tool chosen by the W C B for accessing the Data Warehouse. As a report writing tool, Crystal Reports enables one to extract the data from the Data Warehouse, and then format, summarize and present the extracted data into a meaningful and easy to use manner. Once the data extraction is finished, one can distribute the report by exporting it to popular formats including Microsoft Word and Excel, Text, H T M L or even e-mail. Since Crystal Reports cannot be used to perform advanced statistical analyses such as logistic regression, we used SPSS to analyze the data extracted through Crystal Reports. To create the appropriate SPSS files, we first exported the appropriate Crystal Reports files (.rpt) to tabseparated text (.ttx) format, and then used SPSS to read and convert the text files into SPSS files. Figure III. 1 below illustrates the main steps from Crystal Reports to SPSS. 21 Figure III. 1 Converting Crystal Reports files to SPPS files Crystal Reports File (.rpt) Tab-Separated Text File (.ttx) w w SPSS File (.sav) The sample used to investigate the conversion process of claims consisted of all the claims that had injury dates between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment. During the data collection process we focused on extracting information from the Data Warehouse that was relevant to understanding the conversion process of STD claims. Taking into consideration that the data incorporated into the Data Warehouse were previously checked for possible errors by the departments and services that are the primary providers of the data, we assumed that the data extracted from the Data Warehouse were accurate. Next we present a list with the most important pieces of information (fields) we collected from the D W and the corresponding definitions. (1) Claim Number: A system generated 8-digit number that uniquely identifies a claim. (2) Claim Injury Date: The date on which a worker was injured in an accident or an exposure. (3) Reporting C L S B I N : the identifier (CLSBIN code) of the Assessment Classification of the employer who submitted the claim to the W C B . This code indicates the industry activity in which a worker was engaged at the time of the accident or injury. For example, the C L S B I N for Logging is 010200. (4) Injured Worker Age Quantity: The age in years of the injured worker as of the date of injury. (5) Injured Worker Gender Code: A code identifying whether the injured worker is a male (M) or female (F). (6) Claim First STD payment date: The date on which the first short term disability (FSTD) payment was made for the claim. It indicates the transition from the H C stratum to the STD stratum. (7) Claim First Final STD payment date: The date on which the first final short term disability (FFSTD) payment was made for the claim. Even though this is called a final payment, a claim may subsequently be re-opened for additional compensation benefits. 22 (8) Claim First V R payment date: The date on which the first vocational rehabilitation payment was made for the claim. If this payment occurs after a FFSTD payment, and i f no first L T D payment was made before the F V R payment date, it indicates the transition from the STD stratum to the V R stratum. (9) Claim First L T D payment date: The first date on which a long term disability reserve was set up for the claim, or the date on which the first lump sum L T D payment was made for the claim, whichever is earlier. If this payment occurs after a FFSTD payment, and if no first V R payment was made before the F L T D payment date, it indicates the transition from the STD stratum to the L T D stratum. (10) Claim Cost Summary Amount: The total cost of the claim for a specific year and month. Claim costs are year-to date costs, not the total cost of the claim. (11) Claim Cost Summary STD Days Paid Quantity: The number of short-term disability days that have occurred to date. (12) Claim Cost Summary Month: The month in which the claim costs were charged. (13) Claim Cost Summary Year: The year in which the claim costs were charged. (14) Nature of Injury Type Code: A classification of the injury or illness in terms of its physical characteristics. (15) Body Part Type Code: The number that uniquely identifies the injured body part. (16) ICD9 medical diagnosis code: The code that uniquely identifies the "International Classification of Diseases 9 th (ICD9) Revision Clinical Modification" Medical Diagnosis Type. It is important to emphasize that the numerical codes associated with nature of injury, body part type, ICD9 medical diagnosis code, and C L S B I N are not adequate measures of the severity of an injury or disease or industry, and thus these predictors should be considered nominal categorical variables. III.2 Logistic Regression Regression methods are a fundamental component of any data analysis concerned with describing the relationship that might exist between a response variable and one or more 23 predictor, variables. The essential difference between logistic regression and linear regression is that the outcome variable in logistic regression is binary or dichotomous, while the outcome variable in linear regression is continuous; see Hosmer and Lemeshow (1989) for more details. A n alternative approach to logistic regression is discriminant analysis. Similar to logistic regression, discriminant analysis is also concerned with classifying distinct sets of objects into well-defined groups. Logistic regression is however more flexible than discrimination since it has no assumptions about the distribution of the predictor variables. That is, in logistic regression the predictors do not have to be normally distributed, linearly related, or of equal variance within each group. See Tabachnick and Fidell (1996) for further details regarding discriminant analysis and the differences between this technique and logistic regression. We define the binary variable of our model, Y , so that Y = 1 corresponds to a conversion (highrisk claim) and Y = 0 represents a non-conversion (low-risk claim). Consider p independent predictor variables which will be denoted by the vector x' = (xi, X2, x ). Let the conditional p probability that Y = 1 be denoted by P(Y = 1 | x) = 7i(x). The probability that the response variable equals 0 is P(Y = 0 | x) = 1 - 7i(x). The odds (O) favoring Y = 1 versus Y = 0 are 0(Y= l) = 7t(x)/[l -7i(x)] (III.l) . The logit transformation (L) is defined in terms of 7i(x) and expressed as the natural logarithm of 0(Y=1) L = ln[0(Y=l)] (logit transformation) (III.2) Logistic regression refers to models with the logit L as a linear function of the predictor variables, i.e. L = g(x) = Po + P i x + p X 2 + . . . + PpXp l 2 In this case 7i(x) can be expressed as (III.3) 24 7i(x) = e g(x) /[l+e g(x) ] (III.4) The P's are referred to as the parameters of the model. Observe that logistic regression is different from linear regression in that linear regression expresses a linear relationship between the response variable and its predictors, whereas logistic regression expresses a linear relationship between the natural logarithm of the odds and the predictor variables. The parameters of a logistic regression model are most commonly estimated by using the method of maximum likelihood. See Hosmer and Lemeshow (1989) and McCullagh and Nelder (1989) for a detailed description of this method. In order to apply this method we must first construct the likelihood function, that is, the function that expresses the probability of the observed data as a function of the unknown parameters. Assume that the data available for study consist of n pairs (x;, Y{), where i = 1,2, ..., n. Since each Y, observation is an ordinary Bernoulli random variable, we can represent its probability distribution as follows: f (Y ) = 7T(x ) [l -TTto)] -* Yi i i 1 i Yi = 0, 1; i = l , 2 , ....,n (III.5) Since the Y ; observations are independent, their joint probability function is obtained as the product of the terms given in expression (III.5): L(P)=J7 fi(Yi) (III.6) 1=1 The principle of maximum likelihood states that the parameters of the model are those values of the P's that maximize the expression in the equation (III.6). However, it is easier mathematically to work with the log transform of equation (III.6). Thus one can define the log likelihood as n ln[L(P)] =X {Yiln[7i(xi)] + (1 - Y ) l n [ l - Tift)]} (III.7) 25 The estimates of the parameters are obtained by differentiating equation (III.7) with respect to Pi (i = 1,2, ..., n), and then solving the following system of equations simultaneously: 51n[L(P)]/api - 0 (i= 1,2, ...,n) (III.8) The expressions in equation (III.8) are nonlinear in P's, and thus require special iterative methods for their solution. Details regarding these methods can be found in Hosmer and Lemeshow (1989) and McCullagh and Nelder (1989). After estimating the parameters of the model, one should be interested in assessing the significance of the parameters and the significance of the model. The deviance, D, of a fitted model compares the log-likelihood of the fitted model to the log-likelihood of a model with n parameters that fits the n observations perfectly. Such a perfectly fitting model is called a saturated model; see Hosmer and Lemeshow (1989). The comparison of observed to predicted values using the likelihood function is based on the following expression: D = -2 ln[(likelihood of the current model)/(likelihood of the saturated model)] (HI.9) Notice that the smaller the difference in the two log-likelihood values, the smaller is the deviance and the closer is the fitted model to the saturated model. Hence, the model deviance can be used as a goodness of fit criterion, that is, the larger the model deviance, the poorer the fit. Using equation (III.7), equation (III.9) can be re-expressed in the form D =-2^ where 7tj* {Yiln(7tj*/Yi) + (1 - Y01n[(l - 7t,*)/(l - Yi)]} (III. 10) denotes the estimated value of 7t(xj). To assess the significance of a particular model we compare the value of D with and without the predictors in the equation. The change in D due to including the predictor variables in the logistic regression model is expressed as follows: 26 G = D(for the model without predictors) - D(for the model with predictors) (III. 11) Taking into account expression (III.9), equation (III. 11) becomes G = -2 ln[(likelihood without predictors)/(likelihood with predictors)] (III. 12) Since the G statistic asymptotically follows a chi-square distribution, it can be used to assess the goodness of fit of the model. Degrees of freedom' are the difference between the degrees of freedom for the bigger models and the smaller models. The constant only model has 1 df and the full model with n predictors has n+1 df (1 df for each individual predictor and one for the constant). See Hosmer and Lemeshow (1989) for more details regarding the G statistic. A commonly used method for assessing the fit of an estimated logistic regression model is the Hosmer-Lemeshow test. To perform the test, we divide the observations into ten approximately equal groups based on the estimated probability of the event occurring (deciles of risk), and see how the observed and expected number of Y = 1 events and Y = 0 events compare. To assess the difference between the observed and expected number of events Hosmer and Lemeshow used the chi-square test. To calculate the Hosmer-Lemeshow goodness-of-fit chi-square, we compute the predicted (Ej) number of observations in group j , and then we calculate (Oj - Ej) /Ej, where Oj represents the number of observations within group j . The chi-square value is the sum of this quantity over all groups. The degrees of freedom are calculated as the number of groups minus two. A n additional criterion for evaluating model performance in logistic regression is the correct classification rate (CCR), that is, the proportion of subjects in the data set that are classified correctly. To determine the C C R of a given model, we need a cutoff point for the model, i.e., that value of the P(Y = 1) that allow us to classify a subject as either 1 or 0. For a detailed discussion regarding the usefulness of C C R in logistic regression and the selection of the cutoff value see Ryan (1996), Neter et al (1996), McCullagh and Nelder (1989), and The Cutoff Point section later. 27 To assess the statistical significance of each of the estimated parameters of the model, we can use the Wald test. This test is obtained by comparing the maximum likelihood estimate of each parameter (bj) to an estimate of its standard error. The resulting ratio, under the hypothesis that p; = 0, will asymptotically follow a standard normal distribution. For instance, if we had W = bi/SE(bj) = 3.46 (III. 13) the two tailed p-value would be P( | Z | > 3.46), where Z denotes a random variable following the standard normal distribution. SE(bj) denotes the estimate of b/s standard error; see Hosmer and Lemeshow (1989) and Ryan (1996) for further details regarding its estimation. So far, we have focused on estimating and testing the parameters of the logistic regression model, without being concerned about actually how many variables our model might include. When the number of predictors is too high, the traditional approach to statistical model building involves seeking the most parsimonious model that still explains reasonably accurate the data. The logic behind minimizing the number of predictor variables in the model is that the resultant model is more likely to be numerically stable, and it can be more easily generalized. The most important model building strategies for identifying the minimum number of predictors are forward stepwise selection and backward stepwise elimination. At each step, the Wald statistic and/or the change in (-2 log-likelihood) is employed to determine which predictor will be eliminated from or introduced into the model. The two selection methods are similar to those used in linear regression. Whenever we build a regression model, it is also important to examine the adequacy of the resulting model. In linear regression we look at a variety of residuals and indicators of collinearity. To study the adequacy of the fitted model in logistic regression, we are using comparable diagnostic methods. However, in logistic regression, the evaluation of diagnostics is more complicated than in linear regression. Of the several proposed residuals for logistic regression, we chose the deviance residuals since according to McCullagh and Nelder (1989) they are closer tp being normally distributed than are other type of residuals such as the Pearson residuals. A n even more compelling reason for preferring the deviance residual was provided by 28 Pregibon (1981), who noted that the Pearson residuals are unstable when the estimated value of TC(XJ) is close to either 1 or 0. That is, the Pearson residual significantly changes even for small changes in the values of the Xj predictors. III.3 Stratification of Data The first step in building the logistic regression models was to stratify (divide) the population of all claims into smaller groups that would allow us to build smaller but more accurate regression models within each of these groups. O f the potential predictors available for stratification, we primarily focused on injury type, body part type, and ICD9 medical diagnosis. The reason for using only categorical variables was • Previous studies within the W C B showed that these variables, and especially injury type, were the most useful in describing high-risk claims, and • Incorporating these categorical variables directly into a single model would have been a very difficult task due to the high number of dummy variables that should have been created, and moreover the resulting model would have been difficult to use. Subsequent to our investigation, we decided to use injury type as the primary stratification variable since it appeared to be the most suitable in our attempt to cluster claims by their severity of injury. Moreover, we also knew that the ten most frequent natures of injuries make up about 96% of the claims from our sample, and thus the number of models to be built would not have been too high. See the Classification of STD Claims by Injury Type section above. A second choice for the primary classification variable was ICD9, but we eliminated it since its frequency distribution table showed that: (1) Ten of the most frequent ICD9s only make up 44.4% of all the claims, (2) Twenty of the most frequent ICD9s only make up 58.7% of all the claims, and (3) 127 of the most frequent ICD9s make up 96% of all the claims. 29 As regards body part type, our investigation showed that the best way to incorporate it into the logistic regression models would be to cross-classify it with nature of injury and consequently perform the stratification of claims by using the cells generated during the cross-classification. Although this approach appeared straightforward, we decided not to follow it since the crossclassification would have created too many cells (over 200), and the significant work effort required by this approach would not have allowed us to complete the project in time. We consider however this approach as a potential follow-up of the present study. 30 IV. APPLICATION IV. 1 Model Building Since ten of the most frequent injury types make up 95.72% of all the claims, we decided to use injury type as the primary classification variable, and thus built separate logistic regression models for each of the corresponding subset of claims. Table IV. 1 below shows the abbreviations used for the regression models and the appropriate description of the nature of injury type codes. Table IV. 1 The logistic regression models Abbreviation of model Nature of injury type code Description of the injury Overall All - Amputation Model 00100 Amputation or Enucleation Burn Model 00120 Burn or Scald (Heat) (Hot Substances) Contusion Model 00160 Contusion, Crushing, Bruise (soft tissue) Laceration Model 00170 Cut, Laceration, Puncture - Open wound Fracture Model 00210 Fracture Bursitis Model 00261 Bursitis (Epicondylitis, Tennis elbow) Joint Inflammation 00262 Tenosynovitis, Synovitis, Tendonitis Carpal Model 00264 Carpal Tunnel Syndrome Scratch/Abrasion 00300 Scratches, Abrasions (Superficial Model Wound) Model Sprain/Strain Model 00310 Sprains, Strains Next we focused on identifying beyond nature of injury other predictor variables that could be incorporated into the regression models. Since we had decided not to use body part type and ICD9 medical diagnosis code (see the Stratification of Data section before) in our models, we investigated further whether or not gender of claimant, age of claimant as of the date of injury, and STD days paid could be used as predictor variables in the logistic regression models. 31 Gender of claimant might have been a significant predictor, but we decided not to incorporate it into the model since the initial examination of the data set showed that approximately 67.9% of the cases from our sample were missing information regarding the gender of claimant. It seems that this information was not recorded at all for the cases that were missing it. On the other hand, approximately 99.8% of the cases had information regarding age and STD days paid. Consequently, we ended up using only three predictors to model the conversion process of FFSTD claims. Of the three, nature of injury was used as the primary predictor/stratification variable, and then within each model age and STD days were directly used as quantitative predictor variables. As regards STD days paid, for each individual claim we used only STD days paid up to and including the First Final STD (FFSTD) payment date. The reason for this was that we wanted to identify converted claims as early as possible in the decision making process but at the same time acquire a reasonable accuracy for our models. If we had used the First STD (FSTD) payment, we could have identified converted claims earlier but with lower accuracy since at the FSTD payment date less information about claims would have been available than was at the FFSTD payment date. On the other hand, i f we had used the F V R or F L T D payment date (whichever came first after the FFSTD payment date), we might have ended up with more accurate models, but would have identified converted claims too late. Figure IV. 1 below shows the appropriate time line for decision making. Figure IV. 1 Time line for decision making Injury Date FSTD w FFSTD w F V R or F L T D The initial examination of the data set also revealed that there were some errors, outliers, and missing observations. Thus, we eliminated from our analysis all the cases for which we identified at least one of the problems mentioned above. Accordingly, the "cleanup" of the data consisted of: (1) Eliminating very unusual cases for which age was under 14 or age was higher than 75, 32 (2) Eliminating cases with negative or zero values of STD days paid, and (3) Eliminating cases that had at least one missing predictor variable. As a result of the cleanup, we actually eliminated only 2,125 cases (0.66%) out of the total number of 323,098 cases that made up the initial set of data. The next step was to find out i f collinearity existed among age of claimant and STD days paid. Consequently we checked the coefficient of determination between age and STD days paid for all the models, including the overall model that actually incorporates all claims. Table IV.2 below shows the appropriate results. Table IV.2 The coefficient of determination between age and STD days paid Model R-squared Overall 0.020 Amputation Model 0.010 Burn Model 0.011 Contusion Model 0.019 Laceration Model 0.012 Fracture Model 0.023 Bursitis Model 0.011 Joint Inflammation Model 0.025 Carpal Model 0.008 Scratch/Abrasion Model 0.004 Sprain/Strain Model 0.014 Notice the relatively low values of the coefficient of determination for all models. Thus we concluded that collinearity was not a significant issue to deal with during the model building process. 33 To fit the logistic regression models and determine the statistically significant (a = 0.10) predictors, we used forward stepwise selection and employed the Wald statistic to eliminate the statistically insignificant variables; see the previous chapter for details. For each model, we performed the usual goodness of fit tests such as the Hosmer-Lemeshow test and the -2LL test, and the analysis of the deviance residuals. We also used an additional criterion for evaluating model performance in logistic regression known as model discrimination. Model discrimination evaluates the ability of the model to distinguish between the two groups of cases, based on the estimated probability of the event occurring (see the Accuracy of the Models section later for more details). Table IV.3 below shows the estimated parameters of the logistic regression models. Table IV.3 Estimated parameters of the logistic regression models Model Estimate of Po Estimate of Pi (predictor: Estimate of P2 (predictor: STD days paid) age of claimant) Overall -4.8899 0.0213 0.0188 Burn Model -5.1627 0.0455 - Contusion Model -5.5536 0.0247 0.0236 Laceration Model -4.8297 0.0473 0.0124 Fracture Model -3.6756 0.0212 0.0088 Bursitis Model -3.7105 0.0138 0.0103 Joint Inflammation -4.6126 0.0172 0.0224 Carpal Model -2.9705 0.0109 - Sprain/Strain Model -5.7373 0.0197 0.0311 Model Except for the Burn and Carpal models that have only STD days paid as predictor, the other models include both STD days paid and age of claimant as statistically significant predictors at an alpha level of a = 0.10. Notice that we have not provided regression models for the Amputation and Scratch/Abrasion models. The main reason is that for the Amputation model the 34 conversion factor is so high (83.2%) that we decided to classify all claims within this category as likely conversion. As regards the Scratch/Abrasion model, its conversion factor of 0.54% is much smaller than the overall CF (4.2%), so that we decided to classify all claims within this category as likely non-conversion. Having determined the estimates of the parameters of the models, we can calculate the probability of conversion of any claim prior to or at the FFSTD payment date provided we know the injury type, STD days paid, and age of claimant. Figure IV.2 below illustrates the estimated probabilities of conversion as a function of STD days paid for the Contusion, Laceration and Fracture models for a 40-year old claimant, while Figure IV.3 gives the estimated probabilities of conversion for the Sprain/Strain model for various ages of the claimant. Figure IV.2 Probability of conversion as a function of STD days paid for the Contusion, Laceration and Fracture models (age of claimant is 40) Estimated Probabilities of Conversion for the Contusion, Laceration and Fracture Models u ^ o i n o i o o i o o i o o i n o t o o t o o i o o i o o i o o i n o i o o i o C N I « i n ( O O O C 3 5 T - C ^ ^ t U ) t ^ O O O ' r - C O T r C D r ^ O > O C N m L O O C O C J ) T - T - - ^ T - T - r - C N C N C S I C N C N C S l C N C O C O C O r O C O t » 5 C O STD days paid 35 Figure IV. 3 The probability of conversion as a function of STD days paid and age of claimant for the Sprain/Strain model Estimated Probabilities of Conversion for the Sprain/Strain Model 1.2 • STD days paid From the results presented in Table IV.3 and Figures IV.2 and IV.3, we can easily infer that for the Overall, Contusion, Laceration, Fracture, Bursitis, Joint Inflammation and Sprain/Strain models, the more STD days paid and the older the injured worker, the higher the probability of conversion. As regards the Burn and Carpal models, the more STD days paid the higher the probability of conversion, and the probability of conversion does not significantly depend upon the age of claimant. IV.2 The Cutoff Point The logistic regression models are primarily used to predict prior to the FFSTD payment date the probability of conversion of any claim provided we know the nature of injury, STD days paid and age of claimant. Since the estimated probability is a direct measure of a claim's risk of being converted, we would like to know a specific value of it, called the cutoff point that allows one to classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). Given that 36 we have chosen a cutoff point, all claims that have an estimated probability of conversion higher or equal to the cutoff value are classified as potential conversions, while all claims that have an estimated probability of conversion smaller than the cutoff value are classified as likely nonconversions. Determining the value of the cutoff point is not a straightforward or easy task. Ryan (1996) and Neter et al (1996) highlighted a variety of possible approaches to determine where the cutoff point should be located. We next present four standard approaches to deal with the problem: (1) Use 0.5 as the cutoff point. This approach is reasonable when it is equally likely in the population of interest that outcomes 0 and 1 will occur. Since in our populations the proportion of l's is much lower than the proportion O's (see the conversion factors), we decided not to use this approach. (2) Use the conversion factor (CF), that is, the proportion of l's as the initial cutoff value. This approach is reasonable whenever (a) the data set is a random sample from the population, and thus reflects the proper proportion of O's and l's in the population, and (b) the sample is unbalanced, i.e., the proportion of l's is significantly lower than the proportion of O's. (3) Find the cutoff point for which the proportion of converted claims correctly classified equals the proportion of non-converted claims correctly classified, and consequently they are equal to the proportion of all the claims correctly classified; we will call this value equal classification percentages (ECP). This approach is reasonable when (a) the data set is a random sample from the population, and thus reflects the proper proportion of O's and l's in the population, and (b) the population is highly unbalanced. (4) Use prior probabilities and costs of incorrect predictions to determine the optimal cutoff point so that the expected total cost of incorrect predictions will be minimized. This approach is reasonable when prior information is available about the likelihood of l's and O's in the population, and we know the costs of incorrectly predicting outcome 1 and 0 respectively. The downside of this approach is that in most of the cases it is difficult to assess costs to incorrect predictions. See the next section for more on this point. 37 In addition to the four standard approaches presented above, we can also perform a more direct analysis of the data to better understand the problems behind evaluating a suitable cutoff point. We will illustrate this by considering the Fracture model and evaluating for various values of the cutoff point the percent of converted claims correctly classified, the percent of non-converted claims correctly classified, the total percent correctly classified, and the total number of claims identified as likely conversions. The total number of claims identified as likely conversions actually represents the workload of claims that require special care (examination). Usually, an entitlement officer or a case manager will examine each of these claims; see Table IV.4 below for results. Table IV.4 Percentages of converted, non-converted and all claims correctly classified by the Fracture model, and the corresponding workload for various values of the cutoff point Cutoff Point Converted Correctly Predicted (%) Non-Converted Correctly Predicted (%) A l l Claims Correctly Predicted (%) 0.07 92.06 59.79 64.90 Total Number of Claims Identified as Likely Conversions* 7,825 0.08 88.54 66.51 70.00 6,820 0.09 86.27 71.74 74.04 6,051 0.10 83.85 75.92 77.17 5,421 0.11 81.58 78.70 79.16 4,984 0.12 79.27 81.10 80.81 4,598 0.13 77.40 83.15 82.24 4,272 0.14 75.24 84.60 83.12 4,019 0.15 73.91 86.18 84.24 3,771 0.16 72.12 87.39 84.97 3,560 0.17 70.55 88.21 85.42 3,408 0.18 69.00 89.17 85.97 3,238 0.19 67.62 89.89 86.37 3,104 0.20 66.33 90.52 86.69 2,986 * Workload (number of claims to ?e examined) 1 , 38 Observe that as the value of the cutoff point increases, the percent of actual non-converted claims correctly classified and the percent of all claims correctly classified increases, but the percent of actual converted claims correctly classified decreases. On the other hand, i f the value of the cutoff point decreases, the percent of actual non-converted claims correctly classified and the percent of all claims correctly classified decreases, but the percent of actual converted claims correctly classified increases. Notice as well that i f the value of the cutoff point decreases, the workload of claims that require special examination significantly increases. Thus i f we wanted to increase the accuracy of correctly predicting converted claims, we would at the same time significantly increase the workload. Figures IV.4 and IV.5 below illustrate graphically the results presented in Table IV.4. Figures IV.4 Correct classification percentages for the Fracture model for various values of the cutoff point Accuracy of the Fracture Model versus the Cutoff Point 0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 Cutoff Point 0.15 0.16 0.17 0.18 0.19 0.20 39 Figures IV. 5 The workload for the Fracture model as a function of the Cutoff Point Workload versus the Cutoff Point 2,500 I . 0.07 . 0.08 0.09 0.10 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 — 0.20 Cutoff Point The specific analysis focused on Fractures clearly shows that the "optimal" value of the cutoff point should be primarily determined by the trade-off between predicting converted claims more accurately versus predicting the set of all claims more accurately. At the same time, we should take into account the workload determined by the cutoff point. Besides, at least three additional factors should also be considered: cost of processing converted claims versus cost of processing non-converted claims, number of entitlement officers and case managers available to process claims that have been classified as potential conversions, and the level at which entitlement officers and case managers' subjective use of the logistic regression models can influence their decision-making. Table IV.5 below shows the cutoff points for each model when using approaches 2 and 3 respectively. Notice that ECP is slightly lower than CF for each model. As regards approach 4, the next section provides a cost analysis of the optimal cutoff point for the Fracture model. 40 Table IV.5 Cutoff points for the logistic regression models Model CutoffPoint = CF CutoffPoint = ECP Overall 0.04 0.0292 Burn Model 0.02 0.0163 Contusion Model 0.03 0.0189 Laceration Model 0.04 0.0292 Fracture Model 0,16 0.1160 Bursitis Model 0.08 0.0585 Joint Inflammation Model 0.06 0.0396 Carpal Model 0.11 0.0984 Sprain/Strain Model 0.03 0.0223 IV.3 Cost Analysis of the Optimal Cutoff Point In section IV.2 we mentioned that one standard approach to determine the cutoff point is to use the costs of incorrect predictions, and determine that value of the cutoff point that minimizes the expected total cost of incorrect predictions. The decision tree corresponding to this particular problem is presented on the next page. C is the average cost of claims that are classified as likely conversions and will be actually converted, while A is the average excess cost incurred by claims that will be actually converted but are classified as non-conversion. The excess cost is due to the delayed preventive intervention on these claims. On the other hand, B is the average extra-cost (case management cost) induced by claims that are classified as likely conversions but do not convert. 41 To determine the optimal cutoff point, we express the expected total cost of incorrect predictions (TC) and minimize it over the value of the cutoff point. Since the costs of incorrect predictions are A and B respectively, the expected total cost of incorrect predictions is expressed as follows: TC = (Number of converted claims classified as non-conversions) x A + (Number of nonconverted claims classified as conversions) xB We illustrate this approach using the Fracture model. Table IV.6 below shows the appropriate number of converted claims incorrectly predicted, number of non-converted claims incorrectly predicted, and the expected total cost of incorrect predictions for various values of the cutoff point in the 0.04 to 0.30 range for A = 6 and B = 1. The optimal value of the cutoff point was 42 determined through fitting an interpolation polynomial function to the data and then determining that value of the cutoff point for which the interpolation function reaches its minimum. Figure IV.6 shows the expected total costs of incorrect predictions for B = 1 and four specific values of A (2, 4, 6, and 10). Table IV.7 presents the optimal cutoff point for B = 1 and twelve different values of A ranging from 2 to 30, while Figure IV.7 provides a graphical illustration of the results presented in Table IV.7. Observe that the value of the optimal cutoff point is decreasing as the value of the A / B cost ratio is increasing. Table IV.7 also provides the values of the A / B cost ratio corresponding to the two cases for which the optimal cutoff point would have been equal to the two cutoff points used in this study (CF and ECP respectively). When the conversion factor (CF) is used as the cutoff point, A / B is approximately 3.3, whereas when the equal classification percentages (ECP) is used as the cutoff point, the value of the A / B cost ratio is approximately 5.5. This last result shows that i f we used the ECP as the cutoff point and assumed a cost of $2,000 for B , we would implicitly assume a cost of $ 11,000 for A . Results presented in this section clearly indicate that the proposed approach to evaluate the optimal cutoff point might be worthwhile to investigate further. Most important would be to assess with reasonable accuracy the actual values of A and B , and then to determine the optimal cutoff point subject to some restrictive conditions such as the number of cases that case managers can handle in a given time period. 43 Table IV.6 The number of converted claims incorrectly predicted, the number of non-converted claims incorrectly predicted, and the expected total cost of incorrect predictions for various values of the cutoff point for the Fracture model (A = 6 and B = 1) Cutoff Point 0.04 0.05 0.06 0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 0.21 0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 0.30 Number of Converted Claims Incorrectly Predicted 38 93 151 203 293 351 413 471 530 578 633 667 713 753 793 828 861 892 920 946 985 1,003 1,031 1,057 1,079 1,100 1,116 Number of Non-converted Claims Incorrectly Predicted 10,868 8,435 6,772 5,471 4,556 3,845 3,277 2,898 2,571 2,293 2,095 1,881 1,716 1,604 1,474 1,375 1,290 1,215 1,148 1,079 1,017 974 922 881 836 800. 764 Total Cost 11,096 8,993 7,678 6,689 6,314 5,951 5,755 5,724 5,751 5,761 5,893 5,883 5,994 6,122 6,232 6,343 6,456 6,567 6,668 6,755 6,927 6,992 7,108 7,223 7,310 7,400 7,460 44 Figure IV.6 The expected total costs of incorrect predictions for B=l and four specific values of the A cost Expected Total Cost of Incorrect Predictions vs Cutoff Cutoff Point Table IV.7 The optimal cutoff points for various values of the A / B cost ratio A/B Optimal Cutoff Point 2 0.232 3 0.173 3.3 0.160 = C F 4 0.148 5 0.135 5.5 0.116 = E C P 6 0.108 8 0.099 10 0.092 15 0.070 20 0.065 30 0.053 45 IV.4 Accuracy of the Models To assess the accuracy of the regression models we determined the classification accuracy of each of the models. That is, we evaluated for each model the percent of claims correctly and incorrectly classified for the 1989-1992 sample, which was actually used to build the regression models. Table IV.8 shows the appropriate results when the CF was used as the cutoff point, while Table IV.9 shows similar results for the alternative case when ECP was used as the cutoff point. Observe that when we use the CF, the percentage of converted claims correctly predicted is lower than the percentage of not-converted correctly predicted and the percentage all claims correctly predicted respectively. Also notice the high accuracy of most of the models as measured by the percentage of all claims correctly predicted, that is, eight out of nine models have over 80% accuracy. 46 Table IV.8 Accuracy of the regression models when the CF is the cutoff point (1989-1992 sample) Model Overall 320,973 Burn Model 7,095 Contusion 50,717 Model 44,234 Laceration Model Fracture 16,163 Model 4,884 Bursitis Model 11,096 Joint Inflammation Model Carpal Model 1,706 Sprain/Strain 159,100 Model NonConverted incorrectly predicted (%) 10.98 Total correctly classified (%) 13,512 141 1,395 307,461 6,954 73.70 . 87.23 22.30 12.77 NonConverted correctly predicted (%) 89.02 93.77 49,322 74.70 25.30 92.68 6.23 7.32 93.64 92.18 1,846 42,388 80.77 19.23 91.03 8.97 90.60 2,557 13,606 72.12 27.88 87.39 12.61 84.97 383 4,501 60.57 39.43 85.80 14.20 83.83 641 10,455 64.74 35.26 88.92 11.08 87.53 197 1,509 64.47 35.53 76.41 23.59 75.03 5,059 154,041 74.34 25.66 88.71 11.29 88.25 Number Number of Converted Total Non-. . of claims correctly analyzed Converted Converted predicted claims claims (%) Converted incorrectly predicted (%) 88.38 Table IV.9 Accuracy of the regression models when ECP is the cutoff point (1989-1992 sample) 18.26 90.07 9.93 90.07 Overall 320,973 Burn Model 7,095 Contusion 50,717 Model 44,234 Laceration Model 16,163 Fracture Model 4,884 Bursitis Model Joint 11,096 Inflammation Model Carpal Model 1,706 Sprain/Strain 159,100 Model 307,461 Total correctly classified (%) 81.74 Number Number of Converted Total of Nonclaims correctly analyzed Converted Converted predicted claims claims (%) 13,512 NonConverted incorrectly predicted (%) 18.26 NonConverted correctly predicted (%) 81.74 Model Converted incorrectly predicted (%) 81.74 141 6,954 90.07 9.93 1,395 49,322 84.16 15.84 84.16 15.84 84.16 1,846 42,388 86.54 13.46 86.54 13.46 86.54 2,557 13,606 80.18 19.82 80.18 19.82 80.18 383 4,501 74.59 25.41 74.59 25.41 74.59 641 10,455 77.32 22.68 77.32 22.68 77.32 197 1,509 69.22 30.78 69.22 30.78 69.22 5,059 154,041 80.58 19.42 80.58 19.42 80.58 47 On the other hand, when the ECP is employed, the percent of converted claims correctly predicted is equal to the percentage of not-converted claims correctly predicted and the percentage all claims correctly predicted, but the percentage of claims correctly predicted is slightly lower in this case. The percent of converted claims correctly predicted is however considerably higher than for the alternative case that employs the CF as the cutoff point. As regards the overall model (recall that this model does not require knowledge of the nature of injury), notice its remarkable high accuracy that goes beyond 80%. We concluded therefore that this model can be used with high confidence in early stages of a claim (first 6 months) when information regarding the nature of injury might be missing from the data warehouse. IV.5 Cross-validation of the Models The final step in the model building process is the cross-validation or out of sample testing of the selected models. Cross-validation usually involves checking the model against a set of independent data. In our study, the cross-validation sample consisted of all FFSTD claims (78,471) that had injury dates in 1993. To validate the logistic regression models, we employed them to predict the likely outcome (converted or non-converted) of all claims from the crossvalidation sample (1993 claims). The results are presented in Tables IV. 10 and IV. 11 below for the two different cutoff points used in the study. The percentages of claims correctly classified in the cross-validation sample (1993 claims) are very close to the percentages of claims correctly classified in the 1989-1992 sample used to build the regression models. For the Carpal model the accuracy is even higher for the cross-validation sample. The results presented in Tables IV. 10 and IV. 11 clearly indicate that the validation of the regression models was successfully completed, and thus we conclude that the logistic regression models can be used to predict the outcome of other FFSTD claims that were not included in the 1989-1992 sample. 48 Table IV. 10 Model cross-validation (CF is the cutoff point) Model Overall Burn Model 26.68 24.00 NonConverted correctly predicted (%) 88.79 93.07 Total Number of Number of Converted Converted Nonclaims Converted correctly • incorrectly Converted predicted claims predicted claims (%) (%) 78,471 1,568 11,323 Contusion Model Laceration 10,005 ' Model 4,041 Fracture • Model 1,316 Bursitis Model 3,120 Joint Inflammation Model 562 Carpal Model Sprain/Strain 40,164 Model Total correctly classified (%) 88.01 25 1,543 73.33 76.00 315 11,008 78.73 21.27 91.02 8.98 90.68 448 9,557 79.46 20.54 91.00 9.00 90.49 683 3,358 71.74 28.26 84.90 15.10 82.68 105 1,211 69.52 30.48 83.57 16.43 82.45 203 2,917 76.85 23.15 85.43 14.57 84.87 57 505 71.93 28.07 77.62 22.38 77.05 1,319 38,845 76.57 23.43 88.06 11.94 87.68 NonConverted incorrectly predicted (%) 18.61 Total correctly classified (%) 3,297 75,174 NonConverted incorrectly predicted (%) 11.21 6.93 93.45 Table IV. 11 Model cross-validation (ECP is the cutoff point) Converted correctly predicted (%) Converted incorrectly predicted (%) 75,174 81.39 18.61 NonConverted correctly predicted (%) 81.39 10.11 89.89 10.11 89.89 Model Total Number of Number of Nonclaims Converted claims Converted claims Overall 78,471 3,297 81.39 Burn Model 1,568 25 1,543 89.89 Contusion Model Laceration Model Fracture Model Bursitis Model Joint Inflammation Model Carpal Model 11,323 315 11,008 82.79 17.21 82.79 17.21 82.79 10,005 448 9,557 86.43 13.57 86.43 13.57 86.43 4,041 683 3,358 78.34 21.66 78.34 21.66 78.34 1,316 105 1,211 73.36 26.64 73.36 26.64 73.36 3,120 203 2,917 75.07 24.93 75.07 24.93 75.07 57 505 71.08 28.92 71.08 28.92 71.08 1,319 38,845 80.06 19.94 88.06 19.94 80.06 562 Sprain/Strain 40,164 Model 49 IV.6 Critical STD Days Paid We now know that the logistic regression models can be employed to predict the probability of conversion of any claim provided we know the nature of injury, STD days paid and age of claimant. The cutoff point is a specific value of the probability of conversion that allows one to classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). Accordingly, all claims that have an estimated probability of conversion higher or equal to the cutoff point are classified as potential conversions, while all claims that have an estimated probability of conversion smaller than the cutoff point are classified as likely non-conversions. For each logistic regression model the cutoff point translates into critical values of the predictor variables incorporated into the models. Since for each claim age of claimant at injury is a fixed quantity, we are going to determine the critical STD days paid for each subset of claims determined by the age of claimant. For instance, for the Contusion model the critical STD days paid is 46 for a 40-year old claimant and a cutoff point of 0.03. Thus all 40-year old claimants that have contusion will be classified as likely conversion i f they reach or exceed 46 STD days paid. Similar to the cutoff point, Critical STD days paid allows the decision-maker to classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). Any claim that has accumulated a number of STD days paid higher or equal to the critical value of STD days paid is classified as potential conversion. Similarly, any claim that has accumulated a number of STD days paid lower than the critical value of STD days paid is classified as potential non-conversion. Tables IV. 12 and IV. 13 below provides for all regression models the values of the critical STD days paid for various ages of claimant and using the CF and ECP respectively as the cutoff points. Observe that for all models that incorporate age as a predictor the value of critical STD days paid decreases as the age of claimant increases. On the other hand, for the Burn and Carpal models, which do not include age as predictor, the critical value of STD days paid does not significantly 50 vary with age of claimant. Also notice that the critical STD days paid are lower when we use ECP as the cutoff point. Table IV. 12 Critical STD days paid (CF is the cutoff point) Model Overall Burn Model Contusion Model Laceration Model Fracture Model Bursitis Model Joint Inflammation Model Carpal Model Sprain/Strain Model CF 0.04 0.02 0.03 0.04 0.16 0.08 0.06 20 63 28 65 30 87 77 82 25 58 28 60 28 85 73 76 30 54 28 55 27 83 69 69 35 49 28 51 26 81 66 63 Age 40 45 28 46 24 79 62 56 0.11 0.03 80 83 80 75 80 67 80 60 80 52 80 44 80 36 80 28 80 20 45 25 23 22 16 59 34 24 50 21 23 17 15 57 30 18 55 17 23 12 14 55 26 11 60 12 23 8 12 53 23 5 69 28 69 21 69 13 69 5 45 41 28 41 23 76 58 50 50 36 28 36 22 74 55 43 55 32 28 32 20 72 51 37 60 27 28 27 19 70 47 30 Table IV. 13 Critical STD days paid (ECP is the cutoff point) Model Overall Burn Model Contusion Model Laceration Model Fracture Model Bursitis Model Joint Inflammation Model Carpal Model Sprain/Strain Model ECP 0.0292 0.0163 0.0189 0.0292 0.1160 0.0585 0.0396 20 47 23 46 23 69 53 57 25 43 23 41 21 67 49 50 30 39 23 36 20 65 45 44 35 34 23 32 19 63 41 37 Age 40 30 23 27 18 61 38 31 0.0984 0.0224 69 68 69 60 69 52 69 44 69 36 51 V. IMPLEMENTATION The main objective of the project is to use the results obtained - especially the tables with critical STD days paid - for improving the decision making process at the Workers' Compensation Board of British Columbia. The issue of implementing this model into the W C B ' s decisionmaking system was raised during several meetings with key W C B executives. The suggestion was to do the implementation through the following four phases. Phase 1: Informing potential users within the WCB. The purpose was to present the new decision making model through meetings held in various departments of the W C B . During September and October we made several presentations in the Prevention and Compensation Services divisions of the W C B . The presentation in the Compensation Services Department proved to be especially useful since it was attended by more than 80 entitlement officers and case managers, who will be the actual users of the new decision making model. The subsequent discussion clarified a lot of problems, and it seemed that the vast majority of the entitlement officers and case managers were willing to use the model. Phase 2: Pilot study. The next step towards implementation of the results of the Converted Claims Project should be a pilot study. The goal is to start using the new decision making tool on a subset of claims, involving only a limited number of customer service representatives (CSRs), entitlement officers (EOs), and case managers (CMs). The format of this study could take on several different forms. For instance: - have a given number of staff at various levels of the process (CSRs, EOs, and CMs) use the model results to trial all claims they receive; have all staff members use the results to trial claims of a few specified injury types only (e.g., fractures); Combination of these two. That is, have selected staff members use the results to trial only specific injuries. Our expectation is that the combination format would be the most likely choice. Under this framework, we suggest that at least all the fracture claims should be in the pilot. Some of the reasons are as follows 52 • Fractures make up over 5% of all the claims (approximately 4,000 per year); • Approximately 19% of all the converted claims are fractures, so we would be addressing a significant cost issue; • Fractures' conversion factor of 16% is significantly higher than the average CF of 4.2%, so we would be potentially averting more future costs than with another injury type; • The pilot study does not require the involvement of too many CSRs, EOs, and CMs. • The average number of fractures to be considered is approximately 1,200 per year (see Table IV.4), or 5 per day. This phase will likely commence at the end of November 1999, and will be closely monitored by the Risk Management Group of the W C B , who will provide the necessary assistance. Phase 3. Using feedback to improve the models. The feedback received from entitlement officers and case managers will be used to improve the new decision making model through (1) Readjusting the cutoff points to improve accuracy of predictions , but at the same time not to exceed the claim handling capacity of decision makers , and (2) Building new regression models by taking into consideration other potential predictors such as assessment classification (CLSBIN) and body part type. Phase 4. Full implementation. The full implementation of the critical STD days paid in the decision making process of the W C B will be considered i f the pilot study performed on fractures with a reduced number of decision makers is successful. The annual number of claims to be analyzed during this phase is approximately 18,000, that is, the expected annual number of claims that will be classified as likely conversion. This means an average of 75 claims per day that require special treatment since there is approximately 240 working days in a year. 53 VI. A R E A S FOR FURTHER INVESTIGATION This section presents some possible refinements of the existing methodology and potential new areas/directions for further research on converted claims. (1) Improving the selection method of the Cutoff Point. The goal is to improve the selection method of the cutoff point by using prior probabilities and costs of incorrect predictions to determine the optimal cutoff so that the expected cost of incorrect predictions will be minimized. The major problem is to assess the costs of incorrect predictions. Since no study has been performed at the W C B on this issue, we will raise the possibility of an appropriate study during meetings with key people of the W C B . (2) Use body part as an additional predictor. Body part type might be an additional predictor that could be used to improve the predictive accuracy of the regression models. The main difficulty to be addressed is how to reduce the number of cells created i f nature of injury would be cross-classified with body part type; see the Stratification of the Data section earlier for details. One approach to solve this problem would be to group the body parts into a meaningful way so that body parts within the same group have a comparable level of severity. For instance, we might want to create two separate groups; one group for finger fractures and another group for arm and leg fractures respectively since the severity of arm or leg fractures is usually much higher than the severity of finger fractures. (3) Use both assessment classification (CLSBIN) and nature of injury as primary classification variables. The purpose of this investigation should be to evaluate whether or not assessment classification could be used together with nature of injury to build specific model applicable to some of the high risk industries such as Logging, Wood manufacturing, and Building. The essence of this investigation would be to cross-classify C L S B I N with nature of injury, and then for each particular model to determine whether or not age and STD days paid are statistically significant predictors. (4) Build models that incorporate only STD days paid up to and including the FSTD payment. The logistic regression models presented in this study use STD days paid up to and including the FFSTD payment; see time line for decision making presented in Figure IV. 1. The goal of this new study would be to develop logistic regression models that would use STD days paid 54 at the FSTD payment. The new models would allow one to identify earlier claims that are likely to convert, that is, at the time of the FSTD payment. The downside of this approach would be the lower accuracy of the models since at the FSTD payment date less information regarding STD claims would be available. (5) Multinomial Regression. Multinomial regression is similar to logistic regression, but is more general because the response variable is not restricted to two categories; see Hosmer and Lemeshow (1989) and Agresti (1990) for more details. If we decided to use this method, we would be able to build a model that could be used to model not only two types of STD claims (converted and reopened or inactive) but all three types of STD claims as defined previously (converted, active and inactive). (6) .Proportional Hazards Regression. Proportional hazards regression is a method for modeling time-to-event data in the presence of censored cases; see Kalbfleisch and Prentice (1980) and Cox and Oakes (1984) for a detailed presentation of this method. In our particular case the conversion is the event and the time between injury date and F V R or F L T D payment date is the survival time. A l l claims that have not been converted up to the time when the study ends are considered censored cases. The main advantage of using this model is that it would allow the investigation of more recent STD claims such as claims that had injury dates after 1994. 55 VII. C O N C L U S I O N The project's main objective was to investigate the conversion process of short-term disability claims that had injury dates between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment. The investigation showed that: 1) It is possible to group FFSTD claims into three categories based on the types of payments and amounts they receive. The three categories are inactive, active and converted FFSTD claims respectively. 2) Converted claims make up only about 4.2 % of all the FFSTD claims of our sample, but they received about 64.3% ($1,173 million) of the total payments and awards ($1,824 million) to July 1999. 3) Active and inactive claims have a less significant impact on the W C B ' s reserves since they make up about 95.8% of all the claims, but receive only about 35.7% ($651 million) of the payments made to July 1999. 4) Because of the significant financial impact on the W C B ' s reserves of converted claims, they were classified as high risk claims, while active and inactive (not converted) claims that pose a much lower financial threat are grouped together and classified as low risk claims. 5) The average cost per converted claim is $86,223, which is about 41 times higher than the average cost per not-converted claim ($2,101). 6) Since the outcome of any FFSTD claim is either converted or not converted, we used logistic regression to model the conversion process of FFSTD claims. 7) Since ten of the most frequent injury types make up 95.72% of all the claims (323,098) of our sample, we built separate regression models for each of them. 8) For each subset of claims determined by nature of injury we defined the conversion factor as the proportion of converted claims with respect to all the claims. 9) Besides nature of injury, used as a primary stratification variable, we identified STD days paid up to and including the FFSTD payment date and age of claimant as statistically significant predictors. 56 10) We also built an overall regression model that does not require the knowledge of nature of injury. This model can be used in earlier stages (first 6 months) of a claim, when nature of injury is not available in the Data Warehouse. 1 l)The logistic regression methods allow one to determine the probability of conversion of any FFSTD claim provided one knows the age of claimant and STD days paid. 12) As regards the probability of conversion, the more STD days paid and the older the claimant, the higher its value. 13) The cutoff point is a specific value of the probability of conversion that allows one to classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). 14) To evaluate the optimal value of the cutoff point we need to know the costs of incorrect predictions. Since these costs were not available, we used the conversion factor and ECP respectively as alternative cutoff points. The ECP is that value of the cutoff for which the proportion of converted claims correctly classified equals the proportion of not-converted claims correctly classified. 15) The accuracy of the models is quite remarkable, with most of the model correctly classifying over 80% of all the claims. 16) The cutoff point determines critical values of STD days paid. Critical STD days paid allows the decision-makers to classify a given claim as likely conversion or non-conversion. 17) The implementation of the project will commence with a pilot study performed on fractures and with a limited number of entitlement officers and case managers. 18) C L S B I N and body part type might be use in a further study as additional predictors to improve the accuracy of the models. 19) Multinomial logistic regression and proportional hazards regression were identified as alternative methods to model the conversion process of claims. 20) The methodology developed in this study can be applied to solve similar problems at the WCB. 57 REFERENCES Agresti, A. (1990). Categorical Data Analysis, New York, Wiley-Interscience. Chatfield, C. (1995). Problem Solving - A Statistician's Guide, 2nd edn, Boca Raton, Chapman & Hall/CRC. Cox, D. R. and Oakes, D. O. (1984). Analysis of Survival Data, London, Chapman and Hall. Draper, N . R. and Smith, H. (1998). Applied Regression Analysis, 3rd edn, New York, WileyInterscience. Fattedad, S. and Charron, M . (1998). Claims Inventory Control, The Workers' Compensation Board of British Columbia, Vancouver. Hosmer, D. W. and Lemeshow, S. (1989). Applied Logistic Regression, New York, John Wiley & Sons. Jessup, B . and Gallie, K. (1996). High Risk Project Report, The Workers' Compensation Board of British Columbia, Vancouver. Johnson, D. E., (1998). Applied Multivariate Methods for Data Analysis, Pacific Grove, Duxbury Press. Kalbfleisch, J. D. and Prentice, R. L . (1980). The Statistical Analysis of Failure Time Data, New York, John Wiley and Sons. Kelton, D. W., Sadowski, R. P. and Sadowski, D. A. (1998). Simulation with Arena, Boston, Massachusetts, WCB/McGraw-Hill. Lemeshow, S., Teres, D., Avrunin, J. S. and Pastides, H . (1988). Predicting the Outcome of Intensive Care Unit Patients, Journal of the American Statistical Association 83, 348-356. Mason, K . (1999). Claim Duration Model, The Workers' Compensation Board of British Columbia, Vancouver. McCullagh, P. andNelder, J. A . (1989). Generalized Linear Models, 2nd edn, London, Chapman and Hall. Neter, J., Kutner, M . H., Nachtsheim, C. J. and Wasserman, W. (1996). Applied Linear Statistical Models, 4 edn, Chicago, Irwin. th Ostle, B . and Malone, L. C. (1988). Statistics in Research, 4th edn, Iowa State University Press, Ames, Iowa. Peck, G. (1999). Crystal Reports 7: The Complete Reference, Boston, 58 Massachusetts, Osborne McGraw-Hill. Pregibon, D. (1981). Logistic Regression Diagnostics, Annals of Statistics 9, 705-724. Ryan, T. P. (1996). Modern Regression Methods, New York, Wiley-Interscience. Snedecor, G. W. and Cochran, W. G (1989). Statistical Methods, 8th edn, Iowa State University Press, Ames, Iowa. Tabachnick, B . G. and Fidell, L . S., (1996). Using Multivariate Statistics, 3rd edn, New York, Harper Collins College Publishers. Thomas, L . C , (1999). A Survey of Credit and Behavioral Scoring; Forecasting Financial Risk of Lending to Consumers, in preparation. Thompson, C. M . , (1985). Characteristics Associated with Outcome in a Community Mental Health Partial Hospitalization Program, Community Mental Health Journal 21, 179-188. W C B (1996). The Workers' Compensation Act. The Workers' Compensation Board of British Columbia,^ ancouvev. W C B (1998). 1998 Annual Report, The Workers' Compensation Board of British Columbia, Vancouver. Wiginton, J. C , (1980). A Note on the Comparison of Logit and Discriminant Models of Consumer Credit Behavior, Journal of Financial and Quantitative Analysis 15, 757-770. 59 Appendix Histograms of the transition times of paths 1, 2, 3 and 4 Figure A.1 Path 1 - Histogram of the Injury Date to FFSTD payment date Transition Time Histogram - Injury Date to FFSTD Transition Time (Path 1) Transition Time (months) Figure A.2 Path 1 - Histogram of the FFSTD to F V R Transition Time Histogram - FFSTD to FVR Transition Time (Path 1) 60 Figure A.3 Path 1 - Histogram of the F V R to F L T D Transition Time Histogram - FVR to FLTD Transition Time (Path 1) 600 500 400 o c <x> 300 13 CT CD LL 200 ! 100 - O <o n> <V *§> >§> & & & # <y Transition Time (months) A* ^ # c£ Figure A.4 Path 2 - Histogram of the Injury Date to FFSTD payment date Transition Time Histogram - Injury Date to FFSTD Transition Time (Path 2) 600 500 | >, 400 o ^ 300 I cr £ 200 100 0 * * O & o> # fc> # # # ^ A* <b* Transition Time (months) # ^ N # ^ o$> N 61 Figure A.5 Path 2 - Histogram of the FFSTD to F V R Transition Time Histogram - FFSTD to FVR Transition Time (Path 2) Transition Time (months) Figure A.6 Path 3 - Histogram of the Injury Date to FFSTD payment date Transition Time Histogram - Injury Date to FFSTD Transition Time (Path 3) c Transition Time (months) 62 Figure A.7 Path 3 - Histogram of the FFSTD to FLTD Transition Time Histogram - FFSTD to FLTD Transition Time (Path 3) * * <y <v* >§> & & & # <o° # A* Transition Time (months) # # ^ d> ^ N N Figure A.8 Path 4 - Histogram of the Injury Date to FFSTD payment date Transition Time Histogram - Injury Date to FFSTD Transition Time (Path 4) 63 Figure A.9 Path 4 - Histogram of the FFSTD to FLTD Transition Time Histogram - FFSTD to FLTD Transition Time (Path 4) Transition Time (months) Figure A . 10 Path 4 - Histogram of the F L T D to F V R Transition Time Histogram - FLTD to FVR Transition Time (Path 4)
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- Identifying high-risk claims within the Workers' Compensation...
Open Collections
UBC Theses and Dissertations
Featured Collection
UBC Theses and Dissertations
Identifying high-risk claims within the Workers' Compensation Board of British Columbia's claim inventory… Urbanovich, Ernest 1999
pdf
Page Metadata
Item Metadata
Title | Identifying high-risk claims within the Workers' Compensation Board of British Columbia's claim inventory by using logistic regression modeling |
Creator |
Urbanovich, Ernest |
Date Issued | 1999 |
Description | The goal of the project was to use the data in the Workers' Compensation Board (WCB) of British Columbia's data warehouse to develop a statistical model that could predict on an ongoing basis those short-term disability (STD) claims that posed a potential high financial risk to the WCB. We were especially interested in identifying factors that could be used to model the transition process of claims from the STD stratum to the vocational rehabilitation (VR) and long term disability (LTD) strata, and forecast their financial impact on the WCB. The reason for this focus is that claims experiencing these transitions represent a much higher financial risk to the WCB than claims that only progress to the health care (HC) and/or the short term disability (STD) strata. The sample used to investigate the conversion processes of claims consists of all STD claims (323,098) that had injury dates between January 1, 1989 and December 31, 1992. Although high-risk claims represent only 4.2 % of all STD claims, they have received 64.3% ($1.2 billion) of the total payments and awards ($1.8 billion) made to July 1999. Low-risk claims make up 95.8% of all the claims but only receive 35.7% ($651 million) of the payments and awards. Moreover, the average cost of high-risk claims ($86,200) is 41 times higher than the average cost of low-risk claims ($2,100). The main objective of the project was to build a reliable statistical model to identify high-risk claims that can be readily implemented at the WCB and thereby improve business decisions. To identify high-risk claims early on, we used logistic regression modeling. Since ten of the most frequently observed injury types make up 95.72% of all the claims, separate logistic regression models were built for each of them. Besides injury type, we also identified STD days paid and age of claimant as statistically significant predictors. The logistic regression models can be used to identify high-risk claims prior to or at the First Final STD payment date provided we know the injury type, STD days paid and age of claimant. The investigation showed that the more STD days paid and the older the injured worker, the higher the probability of the claim being high-risk. |
Extent | 10472686 bytes |
Subject |
Workers' Compensation Board of British Columbia. Insurance claims. Risk (Insurance) |
Genre |
Thesis/Dissertation |
Type |
Text |
File Format | application/pdf |
Language | eng |
Date Available | 2009-07-09 |
Provider | Vancouver : University of British Columbia Library |
Rights | For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use. |
DOI | 10.14288/1.0089506 |
URI | http://hdl.handle.net/2429/10515 |
Degree |
Master of Science in Business - MScB |
Program |
Business Administration |
Affiliation |
Business, Sauder School of |
Degree Grantor | University of British Columbia |
Graduation Date | 2000-05 |
Campus |
UBCV |
Scholarly Level | Graduate |
Aggregated Source Repository | DSpace |
Download
- Media
- 831-ubc_2000-0298.pdf [ 9.99MB ]
- Metadata
- JSON: 831-1.0089506.json
- JSON-LD: 831-1.0089506-ld.json
- RDF/XML (Pretty): 831-1.0089506-rdf.xml
- RDF/JSON: 831-1.0089506-rdf.json
- Turtle: 831-1.0089506-turtle.txt
- N-Triples: 831-1.0089506-rdf-ntriples.txt
- Original Record: 831-1.0089506-source.json
- Full Text
- 831-1.0089506-fulltext.txt
- Citation
- 831-1.0089506.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0089506/manifest