Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Identifying high-risk claims within the Workers' Compensation Board of British Columbia's claim inventory… Urbanovich, Ernest 1999

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2000-0298.pdf [ 9.99MB ]
Metadata
JSON: 831-1.0089506.json
JSON-LD: 831-1.0089506-ld.json
RDF/XML (Pretty): 831-1.0089506-rdf.xml
RDF/JSON: 831-1.0089506-rdf.json
Turtle: 831-1.0089506-turtle.txt
N-Triples: 831-1.0089506-rdf-ntriples.txt
Original Record: 831-1.0089506-source.json
Full Text
831-1.0089506-fulltext.txt
Citation
831-1.0089506.ris

Full Text

IDENTIFYING HIGH-RISK C L A I M S WITHIN THE W O R K E R S ' COMPENSATION B O A R D OF BRITISH COLUMBIA'S C L A I M I N V E N T O R Y B Y USING LOGISTIC REGRESSION M O D E L I N G by  ,  ERNEST U R B A N O V I C H  Ph.D. (Chemistry) University of Bucharest 1992  A THESIS SUBMITTED IN P A R T I A L F U L F I L M E N T OF THE REQUIREMENTS FOR THE D E G R E E OF  M A S T E R OF SCIENCE IN BUSINESS ADMINSITRATION  in  THE F A C U L T Y OF G R A D U A T E STUDIES F A C U L T Y OF C O M M E R C E A N D BUSINESS ADMINISTRATION  We accept this thesis as conforming to the required standard  THE UNIVERSITY OF BRITISH C O L U M B I A December 1999 © Ernest Urbanovich, 1999  In  presenting  this thesis  in partial fulfilment  of  the  requirements  for  an  advanced  degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for copying  of this thesis for scholarly purposes  department  or  by  his  or  her  may be granted by the head of my  representatives.  It  is  understood  that  publication of this thesis for financial gain shall not be allowed without permission.  Department of  N-C -  0?hKWCt  The University of British Columbia Vancouver, Canada Date  DE-6 (2/88)  DEC  2.1,  *W  extensive  O U f f ^ f  #0/%  copying  or  my written  11  ABSTRACT The goal of the project was to use the data in the Workers' Compensation Board (WCB) of British Columbia's data warehouse to develop a statistical model that could predict on an ongoing basis those short-term disability (STD) claims that posed a potential high financial risk to the W C B . We were especially interested in identifying factors that could be used to model the transition process of claims from the STD stratum to the vocational rehabilitation (VR) and long term disability (LTD) strata, and forecast their financial impact on the W C B . The reason for this focus is that claims experiencing these transitions represent a much higher financial risk to the W C B than claims that only progress to the health care ( H Q and/or the short term disability (STD) strata.  The sample used to investigate the conversion processes of claims consists of all STD claims (323,098) that had injury dates between January 1, 1989 and December 31, 1992. Although highrisk claims represent only 4.2 % of all STD claims, they have received 64.3% ($1.2 billion) of the total payments and awards ($1.8 billion) made to July 1999. Low-risk claims make up 95.8% of all the claims but only receive 35.7% ($651 million) of the payments and awards. Moreover, the average cost of high-risk claims ($86,200) is 41 times higher than the average cost of lowrisk claims ($2,100).  The main objective of the project was to build a reliable statistical model to identify high-risk claims that can be readily implemented at the W C B and thereby improve business decisions. To identify high-risk claims early on, we used logistic regression modeling. Since ten of the most frequently observed injury types make up 95.72% of all the claims, separate logistic regression models were built for each of them. Besides injury type, we also identified STD days paid and age of claimant as statistically significant predictors. The logistic regression models can be used to identify high-risk claims prior to or at the First Final STD payment date provided we know the injury type, STD days paid and age of claimant. The investigation showed that the more STD days paid and the older the injured worker, the higher the probability of the claim being highrisk.  T A B L E OF CONTENTS  ABSTRACT LIST OF T A B L E S LIST OF FIGURES ACKNOWLEDGEMENTS I. INTRODUCTION 1.1 The Workers' Compensation Board of British Columbia 1.2 Risk Management at the W C B 1.3 Project Background 1.4 Literature Review II. W C B PROCESSES A N D F I N A N C I A L ASPECTS II. 1 Claim Processing at the W C B 11.2 Converted, Active, and Inactive STD Claims 11.3 Common Paths for FFSTD Claims 11.4 Classification of STD Claims by Injury Type 11.5 Financial Aspects III. M E T H O D O L O G Y III. 1 Data Sources and Collection 111.2 Logistic Regression 111.3 Stratification of Data IV. A P P L I C A T I O N IV. 1 Model Building IV.2 The Cutoff Point IV. 3 Cost Analysis of the Optimal Cutoff Point IV.4 Accuracy of the Models IV. 5 Cross-validation of the Models IV.6 Critical STD Days Paid V. IMPLEMENTATION  VI. A R E A S FOR FURTHER INVESTIGATION VII. C O N C L U S I O N REFERENCES APPENDIX  Histograms of the transition times of paths 1, 2, 3 and 4  LIST OF T A B L E S  II. 1  Descriptive statistics of the transition times of paths 1,2,3 and 4  11.2  Fitted theoretical distributions of the transition times involved in paths  13  1,2, 3 and 4  14  11.3  Distribution of claims by injury type  15  11.4  Description of the most frequently observed injury types  16  II. 5  Distribution of payments and awards by injury type (up to July 1999)  18  II.6  Cost per claim by injury type  19  IV. 1  The logistic regression models  30  IV.2  The coefficient of determination between age and STD days paid  32  IV.3  Estimated parameters of the logistic regression models  33  IV.4  Percentages of converted, non-converted and all claims correctly classified by the Fracture model, and the corresponding workload for various values of the cutoff point  37  IV. 5  Cutoff points for the logistic regression models  40  IV.6  The number of converted claims incorrectly predicted, the number of non-converted claims incorrectly predicted, and the expected total cost of incorrect predictions for various values of the cutoff point for the Fracture model (A = 6 and B = 1)  43  IV. 7  The optimal cutoff points for various values of the A / B cost ratio  44  IV. 8  Accuracy of the regression models when the CF is the cutoff point (1989-1992 sample)  IV.9  46  Accuracy of the regression models when ECP is the cutoff point (1989-1992 sample)  46  IV. 10 Model cross-validation (CF is the cutoff point)  48  IV. 11 Model cross-validation (ECP is the cutoff point)  48  IV. 12 Critical STD days paid (CF is the cutoff point)  50  IV. 13 Critical STD days paid (ECP is the cutoff point)  50  VI  LIST OF FIGURES  II. 1  Common paths for FFSTD claims  12  II. 2  Distribution of all claims and converted claims by injury type  16  III. 1  Converting Crystal Reports files to SPPS  21  IV. 1  Time line for decision making  IV.2  Probability of conversion as a function of STD days paid for the Contusion,  files  Laceration and Fracture models (age of claimant is 40) IV. 3  34  Probability of conversion as a function of STD days paid and age of claimant for the Sprain/Strain model  IV.4  31  35  Correct classification percentages for the Fracture model for various values of the cutoff point  38  IV.5  The workload for the Fracture model as a function of the Cutoff Point  39  IV. 6  The expected total costs of incorrect predictions for B=l and four specific values of the A cost  44  IV.7  The optimal cutoff point as a function of the A / B cost ratio  45  A. 1  Path 1 - Histogram of the Injury Date to FFSTD payment date Transition  59  A.2  Path 1 -Histogram of the FFSTD to F V R Transition Time  59  A.3  Path 1 - Histogram of the F V R to F L T D Transition Time  60  A.4  Path 2 - Histogram of the Injury Date to FFSTD payment date Transition Time  60  A.5  Path 2 - Histogram of the FFSTD to F V R Transition Time  61  A.6  Path 3 - Histogram of the Injury Date to FFSTD payment date Transition Time  61  A.7  Path 3 - Histogram of the FFSTD to F L T D Transition Time  62  A. 8  Path 4 - Histogram of the Injury Date to FFSTD payment date Transition Time  62  A.9  Path 4 - Histogram of the FFSTD to F L T D Transition Time  63  A . 10  Path 4-Histogram of the F L T D to F V R Transition Time  63  ACKNOWLEDGEMENTS  I take pleasure in expressing my gratitude to those who have helped me with this project:  Ella Young - Risk Manager, W C B Martin Puterman - Professor, U B C Sidney Fattedad - V P Finance and Information Services, W C B Brian Van Snellenberg - Risk Manager, W C B Jonathan Berkowitz - Professor, U B C Shelby Brumelle - Professor, U B C  1  I. INTRODUCTION 1.1 The Workers' Compensation Board of British Columbia  The Workers' Compensation Board (WCB) of British Columbia is a statutory agency responsible for representing the occupational health and safety, rehabilitation, and compensation interests of the province's workers and employers. Created in 1917, the W C B ' s main objective is to assist workers and employers to ensure safe workplaces, income security and safe return to work for injured workers.  The workers' compensation system was founded on what is known as the "historic compromise", in which the risk of economic loss through personal injury or occupational disease resulting from employment should be borne by industry, and the cost considered as part of the costs of production. Accordingly, the funds that the Board needs to make compensation payments and meet its other obligations are provided from assessments levied on employers by the Assessment Department of the Board. In return, the employers receive protection from lawsuits arising from work-related injuries and diseases. Moreover, as a part of this historic compromise, injured workers receive the right to benefits on a no-fault basis. The Workers Compensation Act is the legal document that guides the W C B ' s operations. The Act gives the Board the official authority to set and enforce occupational safety and health standards, provide compensation and rehabilitation to injured workers or their dependants, and collect funds from business to operate the workers' compensation system.  In 1998, the W C B served approximately 160,000 employers who employed about 1.8 million workers in British Columbia. The WCB's main objectives are: •  preventing workplace injuries, diseases, and fatalities,  •  rehabilitating injured workers and returning them to work,  •  providing fair compensation for workers suffering from an occupational injury or disease,  •  providing sound financial management for a viable workers' compensation system, and  •  protecting the public interest!  2 The W C B ' s incomes are composed of premiums paid by employers and investment income. O f the W C B ' s 1998 total income of $1.6 billion, the employers paid $917 million in premiums. The investment portfolio of stocks and bonds of the W C B had a market value of $7.7 billion at yearend 1998, and provided an average market return of 11 percent. As regards specific costs, in 1998, the W C B spent approximately $1.1 billion on compensation and rehabilitation, up $101 million from the previous year due to an increase in claim duration. During the same period operating expenses totaled $226 million, an increase of 12.1 percent over 1997 due to increased investment in prevention services and technology. Despite the increased costs and expenses, in 1998, the W C B achieved an operating surplus of $289 million. See the 1998 Annual Report of the W C B for more financial highlights.  1.2 Risk Management at the W C B  The Risk Management Group within the Finance Department of the W C B was created in 1998 to protect the W C B against underwriting exposure and loss. This means the Risk Management Group's main objective is to protect the Board from both existing and emerging risks just over the horizon that need to be identified and quantified. Accordingly, the Group works closely with other corporate departments to identify, assess, and help resolve long-term threats to the W C B ' s financial stability. In particular, the Risk Management Group focuses on potential threats posed by funding and cost trends within individual industry classes (e.g., Logging or Building).  Most of the projects the Risk Management Group has been working on since its inception have involved operational research, and statistical analysis, and have came up with suggested actions to minimize the risk exposure of the Board. Another key responsibility of the Group has been to collaborate with the Information Services Division and other business units to develop new tools the W C B can use to improve the business decision making process. One of these is the Data Warehouse, which is like a central data warehouse for decision-makers that includes a compilation of finance, claim, assessment, and other relevant decision-support data from all W C B sources (departments and services). Risk Management analyzes the data in search of trends and performance  indicators to comprehend better the middle and long-term financial  implications that might lie within.  3 1.3 Project Background  The project was initiated by Risk Manager Ella Young from the Risk Management Group and Sidney Fattedad, Vice-President of Finance and Information Services of the W C B . In May 1999, Ella Young contacted Professor Martin L . Puterman from the Faculty of Commerce of the University of British Columbia (UBC) in order to explore a joint effort between the W C B and the U B C to develop a model to identify financially high risk claims within the W C B ' s inventory.  During the initial meeting with the Risk Management Group, Professor Puterman suggested that a M S c student in Management Science associated with the Centre for Operations Excellence (COE) at the University of British Columbia could work on the project. They also agreed that, i f successfully completed, the.MSc student would use the most important results of the project to develop his or her Masters' thesis. Created in January 1998, the C O E supports education and research at U B C through affiliations with leading Canadian private and public sector organizations and its extensive international linkages with leading applied research programs.  Subsequently to a meeting between Professor Puterman and the Risk Management Group, M S c student Ernest Urbanovich joined the W C B on May 19, 1999, and started working on the project under Ella Young's direct coordination. Professor Puterman, the C O E director, advised Ernest Urbanovich on the scientific components of the project and supervised his MSc thesis.  The goal of the project was to use the data in the W C B ' s data warehouse to develop a model that could predict on an ongoing basis those claims that posed a potential risk of being reopened at any time in the future. Initially, the term reopening was used to describe the process in which a claim received any type of additional payments after the first final short-term disability (FFSTD) payment. In addition, there was a need to identify attributes and risk characteristics of claims in the system that, after being inactive for some time, could become a large financial risk for the W C B . The model should have allowed the W C B to achieve a better understanding of where financial risk exposure exists, and the magnitude of those risks due to reopened claims. The approach suggested was to analyze the population of claims that have had reopenings in the past, and develop a regression model to describe the reopening of claims.  4 The thorough investigation of the claim population showed that reopening was an inadequate term to describe financially high risk claims since the costs associated with almost 95% of the reopened claims proved to be relatively low (see the Financial Aspect section later). Consequently, we introduced "conversion" as a new expression to distinguish between low-risk and high-risk claims. We define a converted claim as one that received vocational rehabilitation and/or long term disability payments after the FFSTD payment. On the other hand, a nonconverted claim is one that receives at most health care and/or short-term disability payments after the FFSTD payment. See the Claim processing at the W C B section in the next chapter for details about different types of payments a claim may receive. The average cost per converted claim is $86,223, which is about 41 times higher than the average cost per non-converted claims ($2,101). It was straightforward thus to consider converted claims as financially high-risk, and non-converted claims as financially low-risk claims.  Since the outcome of any FFSTD claim is binary, that is, it can be either converted (high-risk) or not converted (low-risk), we used logistic regression to model the conversion process of claims. We found that the most significant predictors are nature of injury, age of claimant, and number of short term disability days paid. The logistic regression models allow one to classify a given claim as either high-risk or low-risk, and thus one of the most important objectives of the project became the use of these models at the W C B to improve the business decision making process.  1.4 Literature Review  In this section we first review three previous W C B studies related to our study in that they also addressed the problem of high-risk claims but from a different perspective and using different approaches. Then, we describe some relevant applications of logistic regression found in literature. Although the literature concerning logistic regression and its applications is considerable and still growing, we restricted our review to a few studies that are relevant to our project.  In a recent study, Jessup and Gallie (1996) focused on identifying the characteristics of workers who made 20 or more W C B - B C claims in their working lifetime. The study showed that as of  5 November 1, 1995, there were 15,042 workers who had made 20 or more W C B injury reports. These high-risk workers had made a total of 382,151 injury reports, a number approximately equal to the total number of W C B claims first reported in 1994 and 1995. Even more intriguing was that while a few of the 20+ claims were made as early as 1917, over l / 3 had been made in rd  the last 10 years. The authors identified age of worker, gender, nature injury, body part and occupation (industry) as the most important factors in profiling these high-risk claims, but developed no quantitative model that could be used to quantify the risk associated with them.  Fattedad and Charron (1998) studied the claims inventory control at the W C B . The study focused on categorization of inventory, distribution of claims cost by benefit type, claims conversion from short term disability (STD) to long term disability (LTD) and/or vocational rehabilitation (VR), and reopening of STD claims. The investigation showed that the number of STD claims that are converted to V R and/or L T D is relatively low, but they are extremely costly. That is, only 7.8% of the claims included in the study are converted, but they account for approximately 65% of costs. To identify high-risk converted claims up front on the business process, the authors suggested using STD days paid as an indicator of conversion, but provided no mathematical model to determine the likelihood of conversion as a function of STD days paid. However, Fattedad and Charron conclude that once a claim goes beyond 70 STD days paid, it is likely to be converted.  In a study focused on claim duration, Mason (1999) developed a statistical model for claim duration. The study showed that the WCB's Data Warehouse had information by which a claim duration model could be developed. To determine the factors that are likely to affect claim duration, Mason used analysis of variance. The analysis identified several factors including nature of injury, industry subclass (e.g., Logging), age of claimant, gender, the year of the claim, and the type of the accident.  We now describe some related applications of logistic regression.  Wiginton (1980) was one of the firsts to describe the results of using logistic regression in credit scoring. The model allowed Wiginton to classify potential applicants for credit into two groups,  6 i.e., good credit risk and bad credit risk applicants. Although Wiginton was not very impressed with the performance of the logistic regression model, it has subsequently become the main approach to the classification step in credit scoring.  Johnson (1998) developed a logistic regression model to determine whether local college students should be given credit for future purchases at a campus department store. The data consisted of information collected from students who were given credit during the preceding two years. Some of the variables collected included the students' gender, age, grade point average, college major, and hours worked per week. Then, based on each student's past credit history at the store, each student was classified according to whether the student was a good credit risk or low credit risk, and a logistic regression model was built to describe the data. The investigation showed that age and gender were not statistically significant (a = 0.10), while grade point average, college major, and hours worked proved to be statistically significant predictors.  In a survey of credit and behavioral scoring, Thomas (1999) identified logistic regression modeling as one of the most powerful statistical techniques to be used by organizations to decide whether or not to grant credit to consumers who apply to them. As regards the sample used to build the logistic regression models, Thomas emphasizes that usually it can vary from a few thousand to as high as hundreds of thousands. He also recommends that the proportion of good credit risk and bad credit risk applicants in the sample should reflect the proportions in the populations.  Thompson (1985) used stepwise logistic regression to study the outcome (success or failure) in a community mental health program. To build the logistic regression model, Thompson used information on 519 client admissions with data on 17 client characteristics such as demographic data, referral data, mental health history, intelligence scores, and follow-up treatment. O f the predictors investigated, age at admission to the program was statistically the most significant with a p-value lower than 0.01.  Tabachnick and Fidell (1996) used logistic regression analysis to model and predict work status (employed versus unemployed) of women. The study employed four continuous attitudinal  7 variables: locus of control, attitude toward current marital status, attitude toward women's right, and attitude toward housework. Of the 440 women surveyed, 205 were housewives and 235 were women who worked outside the home more than 20 hours a week. The investigation showed that all four predictors were statistically significant (p-value <0.001) and thus incorporated into the model. The prediction accuracy of the model was rather poor, with 56% of the working women and 49% of the housewives correctly predicted, for an overall prediction rate of 53%.  In a landmark study, Lemeshow et al (1988) investigated the survival of patients following admission to an adult intensive care unit. The major goal of the study was to develop a logistic regression model to predict the probability of survival to hospital discharge of the patients. The study employed more than 20 predictor variables such as age, sex, race, service (medical or surgical) at admission, history of chronic renal failure, blood pressure at admission, heart rate at admission, P H from initial blood gases, etc.  8  II. W C B PROCESSES A N D FINANCIAL ASPECTS II. 1 Claim Processing at the W C B  A request for compensation under the Workers Compensation Act is called "claim". Not all injuries and diseases are compensable. That is, a compensation request should be on behalf of a worker (known as the injured worker or the claimant) who may have been injured in a workrelated accident, or suffers from an occupational disease, which may be a result of job-related factors. Not everyone is entitled to compensation under the Act, even i f injured at work. A person qualifies for compensation i f he or she is a worker employed by an employer covered by the Act. A n adjudication process determines i f a claim is valid and, i f so, to what compensation benefits the injured worker is entitled.  Whenever an injury or disease resulting from a person's employment causes a period of temporary disability from work, the W C B pays wage-loss benefits to the injured worker. Wageloss benefits are also known as short-term disability (STD) benefits; we will further use the second term. Usually, STD benefits commence shortly after the initial acceptance of a claim, and they cease when the injured worker recovers from the injury or the condition becomes a permanent one.  Permanent disability awards, also called long-term disability (LTD) awards, are payable when a worker fails to recover completely from a work-related accident or an occupational disease, and is left with a permanent total disability or permanent partial disability. If a worker has a permanent total disability, such as blindness, paraplegia, hemiplegia, and severe loss of cerebral powers, he or she is awarded a periodic payment equal in amount to 75% of his or her average earnings. This amount must be payable during the lifetime of the worker. Where permanent partial disability results from the injury, the compensation must be a periodic payment to the injured worker of a sum equal to 75% of the estimated loss of average earnings resulting from the impairment, and must be payable during the lifetime of the worker.  9 Vocational rehabilitation (VR) is a service provided by the W C B to assist workers in their effort to return to their pre-injury employment or to an occupational category comparable in terms of earning capacity to the pre-injury occupation. V R assistance may be provided in cases where it appears to a V R consultant that such assistance may be of value. Injuries that are likely to be referred immediately to a V R consultant for further consideration are: spinal cord injuries resulting in paraplegia or qudruplegia, major extremity amputations, severe crush injuries, severe brain or brain stem injuries, significant burns, and significant loss of vision.  In addition to STD, L T D , and V R compensations, the W C B is responsible for the cost of health care (HC) benefits such as necessary hospitalization, treatment provided by recognized health care professional, nursing and other care or treatment, prescription drugs, and necessary medical appliances. Any claim that receives only health care benefits is called a health-care-only (HCO) claim, while any claim that is entitled to short term disability benefits and/or long term disability benefits is called a non-health-care-only (non-HCO) claim.  In 1998, there were 207,019 claims first reported at the W C B . Of the total number of claims first reported, 153,545 (74.17%) claims were actually accepted for HCO and/or non-HCO benefits, 6,100 (2.95%) claims were disallowed, and 2,824 (1.36%) claims were rejected. Disallowed claims are those that fall within the scope of the Workers Compensation Act, but are not payable because they are not work related. Rejected claims are those that do not fall within the scope of the Workers Compensation Act since they represent claims from workers employed in industries not covered under the Act, claims from self-employed workers without optional protection, and accounts from physicians submitted in error to the W C B . Notice that the number of accepted, disallowed, and rejected claims (162,469) only make up 78.48% of the total number of claims (207,019) first reported. This is due to the fact that claims are not necessarily disallowed, rejected, or accepted in the year in which they are reported.  As mentioned previously, the focus of this study is STD claims and their movement through various benefit type strata of the W C B ' s claim inventory. The transition from the H C stratum to the STD stratum takes place whenever a claim receives a first STD (FSTD) payment. After spending some time in the STD stratum (and possibly receiving additional STD payments), a  10  claim usually receives a first final STD payment. In spite of the term "final" used here, we should keep in mind that claims are never closed, and that after the first "final" payment has been made, a claim may subsequently be re-opened for additional compensation benefits. The transition from the STD stratum to the V R or L T D strata takes place when the claim receives either a first vocational rehabilitation payment or a first long term disability payment.  11.2 Converted, Active, and Inactive STD Claims  The definition of converted, active and inactive STD claims is directly related to their movement through the system from one stratum to another one. A n active STD claim is defined as one that had a First Final STD (FFSTD) payment and then received additional H C and/or STD payments but no V R and/or L T D payments and awards. Similarly, we define a converted claim as one that had a FFSTD payment and then subsequently received either a First V R (FVR) or a First L T D (FLTD) payment. This definition also includes those claims that after the FFSTD payment received additional H C and/or STD payments, but later on received either a F V R or F L T D type of payment. A third category of claims is represented by inactive claims, that is, claims that after the FFSTD payment date receive no additional payments and consequently are called inactive.  11.3 Common Paths for FFSTD Claims  The sample used to investigate the conversion process of claims consists of all the claims (323,098) that had injury dates between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment. The reason we have not used more recent data (e.g., claims that had injury dates between 1994 - 1997) is that we wanted to capture as much information as possible regarding the claims investigated. For instance, i f we had used to recent samples (claims with injury date after 1993), we would have missed valuable information such as F V R and/or F L T D payments for a significant number of claims since the life cycle of these types of claims is usually higher than 6 years. See Fattedad and Charron (1998) and Mason (1999) for more details regarding STD claim duration. Thus by using the 1989-1992 sample we tried to avoid the bias that could have been induced into our analysis by using more recent samples.  11 The flowchart presented in Figure II. 1 below shows the six most likely paths the FFSTD claims take through the system and the corresponding average transition times (in brackets) between various benefit type strata. The flowchart is based on the sample of all the claims that received a FFSTD payment and had their injury dates between January 1, 1989 and December 31, 1992.  Notice that four paths (1, 2, 3 and 4) actually represent conversions since they lead to V R and/or L T D payments and awards after the FFSTD payment. Although only 4.2 % of the FFSTD claims move through these paths, the payments made for them represent 64.3% ($1,173 million) of the total payments ($1,824 million) made up to July 1999 (see the Financial Aspects section later). Claims that move through path 5 are called active claims, and they represent 79.1% of all the claims from the sample. They receive additional H C and STD payments after the FFSTD payment, but no V R or L T D payments. Path 6 corresponds to inactive claims that after the FFSTD payment date have not received any additional payments. The financial impact on the W C B ' s reserves of active and inactive claims is less significant than the financial impact of converted claims since they make up 95.8% of all the claims but only receive 35.7% ($651 million) of the payments made up to July 1999. Since the significant financial impact of converted claims on the W C B ' s reserves, we will focus further on developing a statistical model to identify them as early as possible in the decision making process.  Transition times between various benefit type strata are important for understanding the dynamic nature of the movement of claims through the system. We define the transition time from the STD stratum to the V R stratum as the time between the First Final STD payment date and the First V R payment date, given that there are no L T D payments between the two dates. Similarly, the transition time from the STD stratum to the L T D stratum is defined as the time between the First Final STD payment date and the First L T D payment date, given that there are no V R payments between the two dates. That is, the claim moves directly from the STD stratum into the L T D stratum. Also, we define the transition time from the L T D stratum to the V R stratum as the time between the F L T D payment date and the F V R payment date, given that the F L T D payment occurred before the F V R payment date. Finally, we define the transition time from the injury date to the FFSTD payment date as the time between the two dates. Table II. 1 below gives the descriptive statistics of all the transition times related to converted claims.  Figure II.1 Common paths for FFSTD claims  1.34% FLTD  FVR  Path 1  [17.3 months]  [10.6 months]  0.65% NO F L T D  FVR  Path 2  [8.8 months] 2.09% NO F V R  FLTD  Path 3  [20.1 months]  FFSTD Claim  0.12% FVR  FLTD  Path 4  [20.8 months]  [20.9 months] 71.64%  HC O N L Y 79.08%  4.98% STD O N L Y  Path 5  2.46% B O T H STD & H C  16.72% Path 6  STD INACTIVE  13 Table II. 1 Descriptive statistics of the transition times of paths 1, 2, 3 and 4  Path  1  Transition  Injury Date to  Mean  Standard  Median  Minimum  Maximum  Number of Claims  (months)  Deviation  (months)  (months)  (months)  in Sample  10.28  8.97  8  0  95  4,344  FFSTD 1  FFSTD to FVR  10.57  14.14  4  0  99  4,344  1  FVR to FLTD  17.31  14.03  13  0  99  4,344  2  Injury Date to  7.03  8.62  5  0  121  2,101  . 2,101  FFSTD 2  FFSTD to FVR  8.82  15.04  3  0  113  3  Injury Date to  6.19  7.56  4  0  89  6,766  20.14  17.26  14  0  119  6,766  8.66  8.35  7  0  81  389  20.94  17.86  15  0  101  389  20.85  18.64  17  0  86  389  FFSTD 3  FFSTD to FLTD  4  Injury Date to FFSTD  4  FFSTD to FLTD  4  FLTD to FVR  The minimum value of the transition time is zero for all paths. The reason for this is that in each path there were claims for which the injury date and the FFSTD payment date were in the same month, and/or the FFSTD and F L T D (or FVR) payment dates were in the same month. Notice that the highest average transition times correspond to the FFSTD to F L T D and F L T D to F V R transitions respectively, while the lowest average transition times are those corresponding to the injury date to the FFSTD payment date transitions. None o f the transition times are normally distributed (bell shaped). To illustrate, the Appendix presents all the histograms of the transition times involved in paths 1,2,3 and 4. We have also used the Arena simulation software package to fit several theoretical distributions to the data, and determined those that provided the best fit; see Kelton et al (1998) for more details regarding Arena and its theoretical distributions available for fitting. As regards the results obtained, they are summarized in Table II.2 below.  14 Table II.2 Fitted theoretical distributions of the transition times involved in paths 1, 2, 3 and 4  Path  Transition  Expression*  Distribution  Mean Squared Error  1  Injury Date to  Weibull  WEIB(10.7, 1.17)  0.003984  FFSTD 1  FFSTD to FVR  Beta  99xBETA(0.289, 2.55)  0.006844  1  FVR to FLTD  Erlang  ERLA(8.65, 2)  0.004874  2  Injury Date to  Weibull  WEIB(6.26, 1.01)  0.002660  FFSTD 2  FFSTD to FVR  Beta  113xBETA(0.239, 2.83)  0.002251  3  Injury Date to  Exponential  EXPO(6.19)  0.002424  FFSTD to FLTD  Erlang  ERLA(10.1,2)  0.010328  Injury Date to  Weibull  WEIB(8.93, 1.09)  0.002729  FFSTD 3 .  4  FFSTD 4  FFSTD to FLTD  Weibull  WEIB(21.9, 1.15)  0.009204  4  FLTD to FVR  Beta  86xBETA(0.706, 2.21)  0.006290  *SeeKeltonetai (1998) br notation  The Injury Date to FFSTD transition times for paths 1, 2, and 4 are best described by the Weibull distribution, while for path 3 the corresponding transition time is described by the exponential distribution. The FFSTD to F V R transition times are best described by the Beta distribution. As regards the FFSTD to F L T D transition times, for path 3 Erlang is the best distribution, while for path 4 Weibull is the most appropriate distribution.  II.4 Classification of STD Claims by Injury Type  Nature of injury is a classification of the injury or illness in terms of its principal physical characteristics. Nature of injury (NOI) classifications are provided by, the common coding system that was developed by the National Work Injuries Statistics Program (NWISP) of Canada. Since nature of injury appeared to be the most useful variable for discriminating and  15 grouping claims, we used it as the primary classification variable to model the conversion process of FFSTD claims; see the Stratification of Data section later for more details.  Consequently, we investigated the distribution of claims that move through paths 1, 2, 3, 4, 5 and 6 as a function of injury type. Table II.3 below summarizes the results by providing the distribution of all claims and converted claims respectively by ten of the most frequently observed injury types. Notice they make up 95.72% of all claims from our sample. See also Figure II.2 for a graphical illustration of the information presented in Table II.3. The description of the nature of injury type codes is presented in Table II.4.  Table II.3 Distribution of claims by injury type  Nature of  A l l claims  Distribution of Converted claims Distribution  injury  (all paths)  all claims (%)  (paths 1-4)  of converted  Conversion Factor(%)  claims (%)  type code 00100  701  0.22  583  4.28  83.17  00120  7,193  2.23  142  1.04  1.97  00160  51,069  15.81  1,405  10.32  2.75  00170  44,617  13.81  1,852  13.60  4.15  00210  16,265  5.03  2,575  18.92  15.83  00261  4,903  1.52  385  2.83  7.85  00262  11,158  3.45  645  4.74  5.78  00264  1,718  0.53  197  1.45  11.47  00300  11,690  3.62  63  0.46  0.54  00310  159,947  49.50  5,102  37.48  3.19  Other  13,837  4.28  665  4.88  4.81  TOTALS  323,098  100  13,614  100  4.21  16  Figure II.2 Distribution of all claims and converted claims by injury type  Distribution of Claims by Injury Type  90  • Percent of all  80  H Percent of converted • Conversion Factor (%) I  70 60  170  210  261  262  264  Nature of injury type code  Table II.4 Description of the most frequently observed injury types  Nature of Injury Type  Description of the Nature of Injury Type  Code 00100  A M P U T A T I O N OR E N U C L E A T I O N  00120  B U R N OR SCALD(HEAT) (HOT SUBSTANCES)  00160  CONTUSION, CRUSHING, BRUISE(SOFT TISSUE)  00170  CUT, L A C E R A T I O N , P U N C T U R E - OPEN W O U N D  00210  FRACTURE  00261  BURSITIS (EPICONDYLITIS, TENNIS ELBOW)  00262  TENOSYNOVITIS, SYNOVITIS, TENDONITIS  00264  CARPAL TUNNEL SYNDROME  00300  SCRATCHES, ABRASIONS (SUPERFICIAL WOUND)  00310  SPRAINS, STRAINS  Other  17 To have a qualitative measure of the extent of conversion within each subset of claims determined by injury type, we introduced the terminology conversion factor. The conversion factor (CF) is the proportion of claims in a given category that have been converted:  CF = Number of converted claims/Total number of claims or  CF(%) = CFxlOO Table II.3 and Figure II.2 also show the conversion factors for each injury type. Notice that the average conversion factor is 4.2% while for scratches and abrasions (nature of injury type code 00300) the CF is only 0.54%. On the other hand, for amputations or enucleations (nature of injury type code 00100) the CF is very high at 83.19%. This means that scratches and abrasions are less likely to be converted than the average claim, whereas amputations or enucleations have a much higher likelihood of being converted than the average claim. The CF is a key indicator for comparing various subsets of claims determined by nature of injury type or other potential classification variable (e.g. industry), and will be used later as a cutoff point for the logistic regression models.  II.5 Financial Aspects  To study the financial impact of converted claims on the W C B ' s reserves, we determined the payments and awards received up to July 1999 by the set of claims (323,098) that had injury date between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment. Table II.5 below summarizes the cost of all claims versus the cost of converted claims broken down by the most frequently observed injury types. Then, using the information presented in Tables II.3 and II.5 we evaluated the average costs per claim by injury type. The appropriate results are given in Table II.6.  Notice that although converted claims only make up 4.2 % of all the claims, they incur 64.3% ($1,173 million) of the total payments and awards ($1,824 million) up to July 1999. On the other  18 hand, active and inactive claims represent 95.8% of all claims from the sample, but they only incur 35.7% ($651 million) of the payments up to July 1999. Thus, from a financial point of view, converted claims represent high-risk claims while active and inactive claims that move through paths 5 and 6 can be grouped together and categorized as low-risk claims. The costs per claim also support the significant differences between the two types of claims. For instance, the average cost per low-risk claim is about $2,101 whereas the average cost per high-risk claim is about 41 times higher ($86,223).  Table II. 5 Distribution of payments and awards by injury type (up to July 1999)  Nature of  Payments and  Payments and awards  Distribution of  Distribution of  injury type  awards ($) for  ($) for  payments and awards  payments and  code  converted claims  all claims  for converted claims  awards for all  (Paths 1-4)  (Paths 1-6)  (%)  claims (%)  00100  24,072,419.38  24,463,117.25  2.05  1.34  00120  8,834,650.62  15,663,582.83  0.75  0.86  00160  140,330,268.72  219,223,971.31  11.97  12.02  00170  75,612,215.80  121,556,802.03  6.45  6.67  00210  208,689,899.40  264,153,250.08  17.80  14.48  00261  27,751,981.11  44,721,521.47  2.37  2.45  00262  57,046,331.10  84,465,146.10  4.86  4.63  00264  13,320,066.64  22,472,430.00  1.14  1.23  00300  5,972,470.79  11,644,754.22  0.51  0.64  00310  539,983,562.51  911,368,465.38  46.05  49.97  Other  71,017,601.67  103,952,340.03  6.05  5.71  1,172,631,467.74  1,823,685,380.70  100  100  TOTALS  19 Table II.6 Cost per claim by injury type  Nature of  Cost per claim ($) for  injury type  low-risk claims  code  (Paths 5 and 6)  Cost per claim ($) for  Cost per claim ($) for  high-risk claims (Paths 1, all claims (Paths 1, 2, 3, 2, 3 and 4)  4, 5, and 6)  00100  3,311.00  41,219.90  34,847.75  00120  967.27  62,215.85  2,174.89  00160  1,587.24  99,666.38  4,289.01  00170  1,073.30  40,827.33  2,721.89  00210  4,047.24  80,981.72  16,224.63  00261  3,751.83  72,270.78  9,113.82  00262  2,604.37  8.8,857.21  7,561.79  00264  6,017.33  69,015.89  13,111.10  00300  487.44  94,801.12  995.28  00310  2,395.37  106,066.31  5,691.29  Other  2,496.02  106,957.83  7,500.54  Average Cost  2,101.25  86,222.90  5,638.39  20  III. M E T H O D O L O G Y III. 1 Data Sources and Collection  The data concerning STD claims are available in the Data Warehouse (DW) of the W C B . The Data Warehouse is a single integrated source of information formed by collecting data from multiple sources, and then transforming and summarizing this information to enable improved decision making at the W C B . It is important to stress that the D W is not an on-line transaction system, and typically does not generate data for other applications. The data come into the D W from several sources (databases) maintained by various departments and services within the W C B (e.g., Statistics Department and Compensation Services), and are refreshed monthly.  To access the data stored in the D W we used a specialized analytical tool known as Crystal Reports. Crystal Reports is a report writing tool produced by Seagate Software, and is considered by many the world standard for desktop reporting and design; see Peck (1999) for more details. Crystal Reports is produced in a stand alone version or as a component of Crystal Info, the reporting tool chosen by the W C B for accessing the Data Warehouse. As a report writing tool, Crystal Reports enables one to extract the data from the Data Warehouse, and then format, summarize and present the extracted data into a meaningful and easy to use manner. Once the data extraction is finished, one can distribute the report by exporting it to popular formats including Microsoft Word and Excel, Text, H T M L or even e-mail.  Since Crystal Reports cannot be used to perform advanced statistical analyses such as logistic regression, we used SPSS to analyze the data extracted through Crystal Reports. To create the appropriate SPSS files, we first exported the appropriate Crystal Reports files (.rpt) to tabseparated text (.ttx) format, and then used SPSS to read and convert the text files into SPSS files. Figure III. 1 below illustrates the main steps from Crystal Reports to SPSS.  21 Figure III. 1 Converting Crystal Reports files to SPPS files  Crystal Reports File (.rpt)  Tab-Separated Text File (.ttx) w  w  SPSS File (.sav)  The sample used to investigate the conversion process of claims consisted of all the claims that had injury dates between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment. During the data collection process we focused on extracting information from the Data Warehouse that was relevant to understanding the conversion process of STD claims. Taking into consideration that the data incorporated into the Data Warehouse were previously checked for possible errors by the departments and services that are the primary providers of the data, we assumed that the data extracted from the Data Warehouse were accurate. Next we present a list with the most important pieces of information (fields) we collected from the D W and the corresponding definitions.  (1) Claim Number: A system generated 8-digit number that uniquely identifies a claim. (2) Claim Injury Date: The date on which a worker was injured in an accident or an exposure. (3) Reporting C L S B I N : the identifier (CLSBIN code) of the Assessment Classification of the employer who submitted the claim to the W C B . This code indicates the industry activity in which a worker was engaged at the time of the accident or injury. For example, the C L S B I N for Logging is 010200. (4) Injured Worker Age Quantity: The age in years of the injured worker as of the date of injury. (5) Injured Worker Gender Code: A code identifying whether the injured worker is a male (M) or female (F). (6) Claim First STD payment date: The date on which the first short term disability (FSTD) payment was made for the claim. It indicates the transition from the H C stratum to the STD stratum. (7) Claim First Final STD payment date: The date on which the first final short term disability (FFSTD) payment was made for the claim. Even though this is called a final payment, a claim may subsequently be re-opened for additional compensation benefits.  22 (8) Claim First V R payment date: The date on which the first vocational rehabilitation payment was made for the claim. If this payment occurs after a FFSTD payment, and i f no first L T D payment was made before the F V R payment date, it indicates the transition from the STD stratum to the V R stratum. (9) Claim First L T D payment date: The first date on which a long term disability reserve was set up for the claim, or the date on which the first lump sum L T D payment was made for the claim, whichever is earlier. If this payment occurs after a FFSTD payment, and if no first V R payment was made before the F L T D payment date, it indicates the transition from the STD stratum to the L T D stratum. (10) Claim Cost Summary Amount: The total cost of the claim for a specific year and month. Claim costs are year-to date costs, not the total cost of the claim. (11) Claim Cost Summary STD Days Paid Quantity: The number of short-term disability days that have occurred to date. (12) Claim Cost Summary Month: The month in which the claim costs were charged. (13) Claim Cost Summary Year: The year in which the claim costs were charged. (14) Nature of Injury Type Code: A classification of the injury or illness in terms of its physical characteristics. (15) Body Part Type Code: The number that uniquely identifies the injured body part. (16) ICD9 medical diagnosis code: The code that uniquely identifies the "International Classification of Diseases 9  th  (ICD9) Revision Clinical Modification" Medical Diagnosis  Type.  It is important to emphasize that the numerical codes associated with nature of injury, body part type, ICD9 medical diagnosis code, and C L S B I N are not adequate measures of the severity of an injury or disease or industry, and thus these predictors should be considered nominal categorical variables.  III.2 Logistic Regression  Regression methods are a fundamental component of any data analysis concerned with describing the relationship that might exist between a response variable and one or more  23 predictor, variables. The essential difference between logistic regression and linear regression is that the outcome variable in logistic regression is binary or dichotomous, while the outcome variable in linear regression is continuous; see Hosmer and Lemeshow (1989) for more details.  A n alternative approach to logistic regression is discriminant analysis. Similar to logistic regression, discriminant analysis is also concerned with classifying distinct sets of objects into well-defined groups. Logistic regression is however more flexible than discrimination since it has no assumptions about the distribution of the predictor variables. That is, in logistic regression the predictors do not have to be normally distributed, linearly related, or of equal variance within each group. See Tabachnick and Fidell (1996) for further details regarding discriminant analysis and the differences between this technique and logistic regression.  We define the binary variable of our model, Y , so that Y = 1 corresponds to a conversion (highrisk claim) and Y = 0 represents a non-conversion (low-risk claim). Consider p independent predictor variables which will be denoted by the vector x' = (xi, X2,  x ). Let the conditional p  probability that Y = 1 be denoted by P(Y = 1 | x) = 7i(x). The probability that the response variable equals 0 is P(Y = 0 | x) = 1 - 7i(x). The odds (O) favoring Y = 1 versus Y = 0 are  0(Y=  l) = 7t(x)/[l -7i(x)]  (III.l) .  The logit transformation (L) is defined in terms of 7i(x) and expressed as the natural logarithm of 0(Y=1) L = ln[0(Y=l)]  (logit transformation)  (III.2)  Logistic regression refers to models with the logit L as a linear function of the predictor variables, i.e.  L = g(x) = Po + P i x + p X 2 + . . . + PpXp l  2  In this case 7i(x) can be expressed as  (III.3)  24  7i(x) = e  g(x)  /[l+e  g(x)  ]  (III.4)  The P's are referred to as the parameters of the model. Observe that logistic regression is different from linear regression in that linear regression expresses a linear relationship between the response variable and its predictors, whereas logistic regression expresses a linear relationship between the natural logarithm of the odds and the predictor variables.  The parameters of a logistic regression model are most commonly estimated by using the method of maximum likelihood. See Hosmer and Lemeshow (1989) and McCullagh and Nelder (1989) for a detailed description of this method. In order to apply this method we must first construct the likelihood function, that is, the function that expresses the probability of the observed data as a function of the unknown parameters. Assume that the data available for study consist of n pairs (x;, Y{), where i = 1,2, ..., n. Since each Y, observation is an ordinary Bernoulli random variable, we can represent its probability distribution as follows: f (Y ) = 7T(x ) [l -TTto)] -* Yi  i  i  1  i  Yi = 0, 1;  i = l , 2 , ....,n  (III.5)  Since the Y ; observations are independent, their joint probability function is obtained as the product of the terms given in expression (III.5):  L(P)=J7  fi(Yi)  (III.6)  1=1  The principle of maximum likelihood states that the parameters of the model are those values of the P's that maximize the expression in the equation (III.6). However, it is easier mathematically to work with the log transform of equation (III.6). Thus one can define the log likelihood as  n  ln[L(P)]  =X  {Yiln[7i(xi)] + (1 - Y ) l n [ l  - Tift)]}  (III.7)  25  The estimates of the parameters are obtained by differentiating equation (III.7) with respect to Pi (i = 1,2, ..., n), and then solving the following system of equations simultaneously:  51n[L(P)]/api - 0  (i= 1,2, ...,n)  (III.8)  The expressions in equation (III.8) are nonlinear in P's, and thus require special iterative methods for their solution. Details regarding these methods can be found in Hosmer and Lemeshow (1989) and McCullagh and Nelder (1989).  After estimating the parameters of the model, one should be interested in assessing the significance of the parameters and the significance of the model. The deviance, D, of a fitted model compares the log-likelihood of the fitted model to the log-likelihood of a model with n parameters that fits the n observations perfectly. Such a perfectly fitting model is called a saturated model; see Hosmer and Lemeshow (1989). The comparison of observed to predicted values using the likelihood function is based on the following expression:  D = -2 ln[(likelihood of the current model)/(likelihood of the saturated model)]  (HI.9)  Notice that the smaller the difference in the two log-likelihood values, the smaller is the deviance and the closer is the fitted model to the saturated model. Hence, the model deviance can be used as a goodness of fit criterion, that is, the larger the model deviance, the poorer the fit. Using equation (III.7), equation (III.9) can be re-expressed in the form  D =-2^  where  7tj*  {Yiln(7tj*/Yi) + (1 - Y01n[(l - 7t,*)/(l - Yi)]}  (III. 10)  denotes the estimated value of 7t(xj).  To assess the significance of a particular model we compare the value of D with and without the predictors in the equation. The change in D due to including the predictor variables in the logistic regression model is expressed as follows:  26 G = D(for the model without predictors) - D(for the model with predictors)  (III. 11)  Taking into account expression (III.9), equation (III. 11) becomes  G = -2 ln[(likelihood without predictors)/(likelihood with predictors)]  (III. 12)  Since the G statistic asymptotically follows a chi-square distribution, it can be used to assess the goodness of fit of the model. Degrees of freedom' are the difference between the degrees of freedom for the bigger models and the smaller models. The constant only model has 1 df and the full model with n predictors has n+1 df (1 df for each individual predictor and one for the constant). See Hosmer and Lemeshow (1989) for more details regarding the G statistic.  A commonly used method for assessing the fit of an estimated logistic regression model is the Hosmer-Lemeshow test. To perform the test, we divide the observations into ten approximately equal groups based on the estimated probability of the event occurring (deciles of risk), and see how the observed and expected number of Y = 1 events and Y = 0 events compare. To assess the difference between the observed and expected number of events Hosmer and Lemeshow used the chi-square test. To calculate the Hosmer-Lemeshow goodness-of-fit chi-square, we compute the predicted (Ej) number of observations in group j , and then we calculate (Oj - Ej) /Ej, where Oj represents the number of observations within group j . The chi-square value is the sum of this quantity over all groups. The degrees of freedom are calculated as the number of groups minus two.  A n additional criterion for evaluating model performance in logistic regression is the correct classification rate (CCR), that is, the proportion of subjects in the data set that are classified correctly. To determine the C C R of a given model, we need a cutoff point for the model, i.e., that value of the P(Y = 1) that allow us to classify a subject as either 1 or 0. For a detailed discussion regarding the usefulness of C C R in logistic regression and the selection of the cutoff value see Ryan (1996), Neter et al (1996), McCullagh and Nelder (1989), and The Cutoff Point section later.  27 To assess the statistical significance of each of the estimated parameters of the model, we can use the Wald test. This test is obtained by comparing the maximum likelihood estimate of each parameter (bj) to an estimate of its standard error. The resulting ratio, under the hypothesis that p; = 0, will asymptotically follow a standard normal distribution. For instance, if we had  W = bi/SE(bj) = 3.46  (III. 13)  the two tailed p-value would be P( | Z | > 3.46), where Z denotes a random variable following the standard normal distribution. SE(bj) denotes the estimate of b/s standard error; see Hosmer and Lemeshow (1989) and Ryan (1996) for further details regarding its estimation.  So far, we have focused on estimating and testing the parameters of the logistic regression model, without being concerned about actually how many variables our model might include. When the number of predictors is too high, the traditional approach to statistical model building involves seeking the most parsimonious model that still explains reasonably accurate the data. The logic behind minimizing the number of predictor variables in the model is that the resultant model is more likely to be numerically stable, and it can be more easily generalized. The most important model building strategies for identifying the minimum number of predictors are forward stepwise selection and backward stepwise elimination. At each step, the Wald statistic and/or the change in (-2 log-likelihood) is employed to determine which predictor will be eliminated from or introduced into the model. The two selection methods are similar to those used in linear regression.  Whenever we build a regression model, it is also important to examine the adequacy of the resulting model. In linear regression we look at a variety of residuals and indicators of collinearity. To study the adequacy of the fitted model in logistic regression, we are using comparable diagnostic methods. However, in logistic regression, the evaluation of diagnostics is more complicated than in linear regression. Of the several proposed residuals for logistic regression, we chose the deviance residuals since according to McCullagh and Nelder (1989) they are closer tp being normally distributed than are other type of residuals such as the Pearson residuals. A n even more compelling reason for preferring the deviance residual was provided by  28 Pregibon (1981), who noted that the Pearson residuals are unstable when the estimated value of TC(XJ)  is close to either 1 or 0. That is, the Pearson residual significantly changes even for small  changes in the values of the Xj predictors.  III.3 Stratification of Data  The first step in building the logistic regression models was to stratify (divide) the population of all claims into smaller groups that would allow us to build smaller but more accurate regression models within each of these groups. O f the potential predictors available for stratification, we primarily focused on injury type, body part type, and ICD9 medical diagnosis. The reason for using only categorical variables was •  Previous studies within the W C B showed that these variables, and especially injury type, were the most useful in describing high-risk claims, and  •  Incorporating these categorical variables directly into a single model would have been a very difficult task due to the high number of dummy variables that should have been created, and moreover the resulting model would have been difficult to use.  Subsequent to our investigation, we decided to use injury type as the primary stratification variable since it appeared to be the most suitable in our attempt to cluster claims by their severity of injury. Moreover, we also knew that the ten most frequent natures of injuries make up about 96% of the claims from our sample, and thus the number of models to be built would not have been too high. See the Classification of STD Claims by Injury Type section above.  A second choice for the primary classification variable was ICD9, but we eliminated it since its frequency distribution table showed that:  (1) Ten of the most frequent ICD9s only make up 44.4% of all the claims, (2) Twenty of the most frequent ICD9s only make up 58.7% of all the claims, and (3) 127 of the most frequent ICD9s make up 96% of all the claims.  29 As regards body part type, our investigation showed that the best way to incorporate it into the logistic regression models would be to cross-classify it with nature of injury and consequently perform the stratification of claims by using the cells generated during the cross-classification. Although this approach appeared straightforward, we decided not to follow it since the crossclassification would have created too many cells (over 200), and the significant work effort required by this approach would not have allowed us to complete the project in time. We consider however this approach as a potential follow-up of the present study.  30  IV. APPLICATION IV. 1 Model Building  Since ten of the most frequent injury types make up 95.72% of all the claims, we decided to use injury type as the primary classification variable, and thus built separate logistic regression models for each of the corresponding subset of claims. Table IV. 1 below shows the abbreviations used for the regression models and the appropriate description of the nature of injury type codes.  Table IV. 1 The logistic regression models  Abbreviation of model  Nature of injury type code  Description of the injury  Overall  All  -  Amputation Model  00100  Amputation or Enucleation  Burn Model  00120  Burn or Scald (Heat) (Hot Substances)  Contusion Model  00160  Contusion, Crushing, Bruise (soft tissue)  Laceration Model  00170  Cut, Laceration, Puncture - Open wound  Fracture Model  00210  Fracture  Bursitis Model  00261  Bursitis (Epicondylitis, Tennis elbow)  Joint Inflammation  00262  Tenosynovitis, Synovitis, Tendonitis  Carpal Model  00264  Carpal Tunnel Syndrome  Scratch/Abrasion  00300  Scratches, Abrasions (Superficial  Model  Wound)  Model Sprain/Strain Model  00310  Sprains, Strains  Next we focused on identifying beyond nature of injury other predictor variables that could be incorporated into the regression models. Since we had decided not to use body part type and ICD9 medical diagnosis code (see the Stratification of Data section before) in our models, we investigated further whether or not gender of claimant, age of claimant as of the date of injury, and STD days paid could be used as predictor variables in the logistic regression models.  31 Gender of claimant might have been a significant predictor, but we decided not to incorporate it into the model since the initial examination of the data set showed that approximately 67.9% of the cases from our sample were missing information regarding the gender of claimant. It seems that this information was not recorded at all for the cases that were missing it. On the other hand, approximately 99.8% of the cases had information regarding age and STD days paid. Consequently, we ended up using only three predictors to model the conversion process of FFSTD claims. Of the three, nature of injury was used as the primary predictor/stratification variable, and then within each model age and STD days were directly used as quantitative predictor variables.  As regards STD days paid, for each individual claim we used only STD days paid up to and including the First Final STD (FFSTD) payment date. The reason for this was that we wanted to identify converted claims as early as possible in the decision making process but at the same time acquire a reasonable accuracy for our models. If we had used the First STD (FSTD) payment, we could have identified converted claims earlier but with lower accuracy since at the FSTD payment date less information about claims would have been available than was at the FFSTD payment date. On the other hand, i f we had used the F V R or F L T D payment date (whichever came first after the FFSTD payment date), we might have ended up with more accurate models, but would have identified converted claims too late. Figure IV. 1 below shows the appropriate time line for decision making.  Figure IV. 1 Time line for decision making  Injury Date  FSTD  w  FFSTD  w  F V R or F L T D  The initial examination of the data set also revealed that there were some errors, outliers, and missing observations. Thus, we eliminated from our analysis all the cases for which we identified at least one of the problems mentioned above. Accordingly, the "cleanup" of the data consisted of: (1) Eliminating very unusual cases for which age was under 14 or age was higher than 75,  32 (2) Eliminating cases with negative or zero values of STD days paid, and (3) Eliminating cases that had at least one missing predictor variable.  As a result of the cleanup, we actually eliminated only 2,125 cases (0.66%) out of the total number of 323,098 cases that made up the initial set of data.  The next step was to find out i f collinearity existed among age of claimant and STD days paid. Consequently we checked the coefficient of determination between age and STD days paid for all the models, including the overall model that actually incorporates all claims. Table IV.2 below shows the appropriate results.  Table IV.2 The coefficient of determination between age and STD days paid  Model  R-squared  Overall  0.020  Amputation Model  0.010  Burn Model  0.011  Contusion Model  0.019  Laceration Model  0.012  Fracture Model  0.023  Bursitis Model  0.011  Joint Inflammation Model  0.025  Carpal Model  0.008  Scratch/Abrasion Model  0.004  Sprain/Strain Model  0.014  Notice the relatively low values of the coefficient of determination for all models. Thus we concluded that collinearity was not a significant issue to deal with during the model building process.  33 To fit the logistic regression models and determine the statistically significant (a = 0.10) predictors, we used forward stepwise selection and employed the Wald statistic to eliminate the statistically insignificant variables; see the previous chapter for details. For each model, we performed the usual goodness of fit tests such as the Hosmer-Lemeshow test and the -2LL test, and the analysis of the deviance residuals. We also used an additional criterion for evaluating model performance in logistic regression known as model discrimination. Model discrimination evaluates the ability of the model to distinguish between the two groups of cases, based on the estimated probability of the event occurring (see the Accuracy of the Models section later for more details). Table IV.3 below shows the estimated parameters of the logistic regression models.  Table IV.3 Estimated parameters of the logistic regression models  Model  Estimate of Po  Estimate of Pi (predictor:  Estimate of P2 (predictor:  STD days paid)  age of claimant)  Overall  -4.8899  0.0213  0.0188  Burn Model  -5.1627  0.0455  -  Contusion Model  -5.5536  0.0247  0.0236  Laceration Model  -4.8297  0.0473  0.0124  Fracture Model  -3.6756  0.0212  0.0088  Bursitis Model  -3.7105  0.0138  0.0103  Joint Inflammation  -4.6126  0.0172  0.0224  Carpal Model  -2.9705  0.0109  -  Sprain/Strain Model  -5.7373  0.0197  0.0311  Model  Except for the Burn and Carpal models that have only STD days paid as predictor, the other models include both STD days paid and age of claimant as statistically significant predictors at an alpha level of a = 0.10. Notice that we have not provided regression models for the Amputation and Scratch/Abrasion models. The main reason is that for the Amputation model the  34 conversion factor is so high (83.2%) that we decided to classify all claims within this category as likely conversion. As regards the Scratch/Abrasion model, its conversion factor of 0.54% is much smaller than the overall CF (4.2%), so that we decided to classify all claims within this category as likely non-conversion.  Having determined the estimates of the parameters of the models, we can calculate the probability of conversion of any claim prior to or at the FFSTD payment date provided we know the injury type, STD days paid, and age of claimant. Figure IV.2 below illustrates the estimated probabilities of conversion as a function of STD days paid for the Contusion, Laceration and Fracture models for a 40-year old claimant, while Figure IV.3 gives the estimated probabilities of conversion for the Sprain/Strain model for various ages of the claimant.  Figure IV.2 Probability of conversion as a function of STD days paid for the Contusion, Laceration and Fracture models (age of claimant is 40)  Estimated Probabilities of Conversion for the Contusion, Laceration and Fracture Models  u ^ o i n o i o o i o o i o o i n o t o o t o o i o o i o o i o o i n o i o o i o  C N I « i n ( O O O C 3 5 T - C ^ ^ t U ) t ^ O O O ' r - C O T r C D r ^ O > O C N m L O O C O C J ) T - T - - ^ T - T - r - C N C N C S I C N C N C S l C N C O C O C O r O C O t » 5 C O  STD days paid  35  Figure IV. 3 The probability of conversion as a function of STD days paid and age of claimant for the Sprain/Strain model  Estimated Probabilities of Conversion for the Sprain/Strain Model 1.2  •  STD days paid  From the results presented in Table IV.3 and Figures IV.2 and IV.3, we can easily infer that for the Overall, Contusion, Laceration, Fracture, Bursitis, Joint Inflammation and Sprain/Strain models, the more STD days paid and the older the injured worker, the higher the probability of conversion. As regards the Burn and Carpal models, the more STD days paid the higher the probability of conversion, and the probability of conversion does not significantly depend upon the age of claimant.  IV.2 The Cutoff Point  The logistic regression models are primarily used to predict prior to the FFSTD payment date the probability of conversion of any claim provided we know the nature of injury, STD days paid and age of claimant. Since the estimated probability is a direct measure of a claim's risk of being converted, we would like to know a specific value of it, called the cutoff point that allows one to classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). Given that  36 we have chosen a cutoff point, all claims that have an estimated probability of conversion higher or equal to the cutoff value are classified as potential conversions, while all claims that have an estimated probability of conversion smaller than the cutoff value are classified as likely nonconversions.  Determining the value of the cutoff point is not a straightforward or easy task. Ryan (1996) and Neter et al (1996) highlighted a variety of possible approaches to determine where the cutoff point should be located. We next present four standard approaches to deal with the problem:  (1) Use 0.5 as the cutoff point. This approach is reasonable when it is equally likely in the population of interest that outcomes 0 and 1 will occur. Since in our populations the proportion of l's is much lower than the proportion O's (see the conversion factors), we decided not to use this approach. (2) Use the conversion factor (CF), that is, the proportion of l's as the initial cutoff value. This approach is reasonable whenever (a) the data set is a random sample from the population, and thus reflects the proper proportion of O's and l's in the population, and (b) the sample is unbalanced, i.e., the proportion of l's is significantly lower than the proportion of O's. (3) Find the cutoff point for which the proportion of converted claims correctly classified equals the proportion of non-converted claims correctly classified, and consequently they are equal to the proportion of all the claims correctly classified; we will call this value equal classification percentages (ECP). This approach is reasonable when (a) the data set is a random sample from the population, and thus reflects the proper proportion of O's and l's in the population, and (b) the population is highly unbalanced. (4) Use prior probabilities and costs of incorrect predictions to determine the optimal cutoff point so that the expected total cost of incorrect predictions will be minimized. This approach is reasonable when prior information is available about the likelihood of l's and O's in the population, and we know the costs of incorrectly predicting outcome 1 and 0 respectively. The downside of this approach is that in most of the cases it is difficult to assess costs to incorrect predictions. See the next section for more on this point.  37 In addition to the four standard approaches presented above, we can also perform a more direct analysis of the data to better understand the problems behind evaluating a suitable cutoff point. We will illustrate this by considering the Fracture model and evaluating for various values of the cutoff point the percent of converted claims correctly classified, the percent of non-converted claims correctly classified, the total percent correctly classified, and the total number of claims identified as likely conversions. The total number of claims identified as likely conversions actually represents the workload of claims that require special care (examination). Usually, an entitlement officer or a case manager will examine each of these claims; see Table IV.4 below for results.  Table IV.4 Percentages of converted, non-converted and all claims correctly classified by the Fracture model, and the corresponding workload for various values of the cutoff point  Cutoff Point  Converted Correctly Predicted (%)  Non-Converted Correctly Predicted (%)  A l l Claims Correctly Predicted (%)  0.07  92.06  59.79  64.90  Total Number of Claims Identified as Likely Conversions* 7,825  0.08  88.54  66.51  70.00  6,820  0.09  86.27  71.74  74.04  6,051  0.10  83.85  75.92  77.17  5,421  0.11  81.58  78.70  79.16  4,984  0.12  79.27  81.10  80.81  4,598  0.13  77.40  83.15  82.24  4,272  0.14  75.24  84.60  83.12  4,019  0.15  73.91  86.18  84.24  3,771  0.16  72.12  87.39  84.97  3,560  0.17  70.55  88.21  85.42  3,408  0.18  69.00  89.17  85.97  3,238  0.19  67.62  89.89  86.37  3,104  0.20  66.33  90.52  86.69  2,986  * Workload (number of claims to ?e examined)  1  ,  38  Observe that as the value of the cutoff point increases, the percent of actual non-converted claims correctly classified and the percent of all claims correctly classified increases, but the percent of actual converted claims correctly classified decreases. On the other hand, i f the value of the cutoff point decreases, the percent of actual non-converted claims correctly classified and the percent of all claims correctly classified decreases, but the percent of actual converted claims correctly classified increases. Notice as well that i f the value of the cutoff point decreases, the workload of claims that require special examination significantly increases. Thus i f we wanted to increase the accuracy of correctly predicting converted claims, we would at the same time significantly increase the workload. Figures IV.4 and IV.5 below illustrate graphically the results presented in Table IV.4.  Figures IV.4 Correct classification percentages for the Fracture model for various values of the cutoff point  Accuracy of the Fracture Model versus the Cutoff Point  0.07  0.08  0.09  0.10  0.11  0.12  0.13  0.14  Cutoff Point  0.15  0.16  0.17  0.18  0.19  0.20  39  Figures IV. 5 The workload for the Fracture model as a function of the Cutoff Point  Workload versus the Cutoff Point  2,500 I  . 0.07  . 0.08  0.09  0.10  0.11  0.12  0.13  0.14  0.15  0.16  0.17  0.18  0.19  — 0.20  Cutoff Point  The specific analysis focused on Fractures clearly shows that the "optimal" value of the cutoff point should be primarily determined by the trade-off between predicting converted claims more accurately versus predicting the set of all claims more accurately. At the same time, we should take into account the workload determined by the cutoff point. Besides, at least three additional factors should also be considered: cost of processing converted claims versus cost of processing non-converted claims, number of entitlement officers and case managers available to process claims that have been classified as potential conversions, and the level at which entitlement officers and case managers' subjective use of the logistic regression models can influence their decision-making.  Table IV.5 below shows the cutoff points for each model when using approaches 2 and 3 respectively. Notice that ECP is slightly lower than CF for each model. As regards approach 4, the next section provides a cost analysis of the optimal cutoff point for the Fracture model.  40 Table IV.5 Cutoff points for the logistic regression models  Model  CutoffPoint = CF  CutoffPoint = ECP  Overall  0.04  0.0292  Burn Model  0.02  0.0163  Contusion Model  0.03  0.0189  Laceration Model  0.04  0.0292  Fracture Model  0,16  0.1160  Bursitis Model  0.08  0.0585  Joint Inflammation Model  0.06  0.0396  Carpal Model  0.11  0.0984  Sprain/Strain Model  0.03  0.0223  IV.3 Cost Analysis of the Optimal Cutoff Point  In section IV.2 we mentioned that one standard approach to determine the cutoff point is to use the costs of incorrect predictions, and determine that value of the cutoff point that minimizes the expected total cost of incorrect predictions. The decision tree corresponding to this particular problem is presented on the next page.  C is the average cost of claims that are classified as likely conversions and will be actually converted, while A is the average excess cost incurred by claims that will be actually converted but are classified as non-conversion. The excess cost is due to the delayed preventive intervention on these claims. On the other hand, B is the average extra-cost (case management cost) induced by claims that are classified as likely conversions but do not convert.  41  To determine the optimal cutoff point, we express the expected total cost of incorrect predictions (TC) and minimize it over the value of the cutoff point. Since the costs of incorrect predictions are A and B respectively, the expected total cost of incorrect predictions is expressed as follows:  TC = (Number of converted claims classified as non-conversions) x A + (Number of nonconverted claims classified as conversions) xB  We illustrate this approach using the Fracture model. Table IV.6 below shows the appropriate number of converted claims incorrectly predicted, number of non-converted claims incorrectly predicted, and the expected total cost of incorrect predictions for various values of the cutoff point in the 0.04 to 0.30 range for A = 6 and B = 1. The optimal value of the cutoff point was  42 determined through fitting an interpolation polynomial function to the data and then determining that value of the cutoff point for which the interpolation function reaches its minimum. Figure IV.6 shows the expected total costs of incorrect predictions for B = 1 and four specific values of A (2, 4, 6, and 10). Table IV.7 presents the optimal cutoff point for B = 1 and twelve different values of A ranging from 2 to 30, while Figure IV.7 provides a graphical illustration of the results presented in Table IV.7.  Observe that the value of the optimal cutoff point is decreasing as the value of the A / B cost ratio is increasing. Table IV.7 also provides the values of the A / B cost ratio corresponding to the two cases for which the optimal cutoff point would have been equal to the two cutoff points used in this study (CF and ECP respectively). When the conversion factor (CF) is used as the cutoff point, A / B is approximately 3.3, whereas when the equal classification percentages (ECP) is used as the cutoff point, the value of the A / B cost ratio is approximately 5.5. This last result shows that i f we used the ECP as the cutoff point and assumed a cost of $2,000 for B , we would implicitly assume a cost of $ 11,000 for A .  Results presented in this section clearly indicate that the proposed approach to evaluate the optimal cutoff point might be worthwhile to investigate further. Most important would be to assess with reasonable accuracy the actual values of A and B , and then to determine the optimal cutoff point subject to some restrictive conditions such as the number of cases that case managers can handle in a given time period.  43 Table IV.6 The number of converted claims incorrectly predicted, the number of non-converted claims incorrectly predicted, and the expected total cost of incorrect predictions for various values of the cutoff point for the Fracture model (A = 6 and B = 1)  Cutoff Point 0.04 0.05 0.06 0.07 0.08 0.09 0.10 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 0.21 0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 0.30  Number of Converted Claims Incorrectly Predicted 38 93 151 203 293 351 413 471 530 578 633 667 713 753 793 828 861 892 920 946 985 1,003 1,031 1,057 1,079 1,100 1,116  Number of Non-converted Claims Incorrectly Predicted 10,868 8,435 6,772 5,471 4,556 3,845 3,277 2,898 2,571 2,293 2,095 1,881 1,716 1,604 1,474 1,375 1,290 1,215 1,148 1,079 1,017 974 922 881 836 800. 764  Total Cost  11,096 8,993 7,678 6,689 6,314 5,951 5,755 5,724 5,751 5,761 5,893 5,883 5,994 6,122 6,232 6,343 6,456 6,567 6,668 6,755 6,927 6,992 7,108 7,223 7,310 7,400 7,460  44 Figure IV.6 The expected total costs of incorrect predictions for B=l and four specific values of the A cost Expected Total Cost of Incorrect Predictions vs Cutoff  Cutoff Point  Table IV.7 The optimal cutoff points for various values of the A / B cost ratio  A/B  Optimal Cutoff Point  2  0.232  3  0.173  3.3  0.160 = C F  4  0.148  5  0.135  5.5  0.116 = E C P  6  0.108  8  0.099  10  0.092  15  0.070  20  0.065  30  0.053  45  IV.4 Accuracy of the Models  To assess the accuracy of the regression models we determined the classification accuracy of each of the models. That is, we evaluated for each model the percent of claims correctly and incorrectly classified for the 1989-1992 sample, which was actually used to build the regression models. Table IV.8 shows the appropriate results when the CF was used as the cutoff point, while Table IV.9 shows similar results for the alternative case when ECP was used as the cutoff point.  Observe that when we use the CF, the percentage of converted claims correctly predicted is lower than the percentage of not-converted correctly predicted and the percentage all claims correctly predicted respectively. Also notice the high accuracy of most of the models as measured by the percentage of all claims correctly predicted, that is, eight out of nine models have over 80% accuracy.  46 Table IV.8 Accuracy of the regression models when the CF is the cutoff point (1989-1992 sample)  Model  Overall 320,973 Burn Model 7,095 Contusion 50,717 Model 44,234 Laceration Model Fracture 16,163 Model 4,884 Bursitis Model 11,096 Joint Inflammation Model Carpal Model 1,706 Sprain/Strain 159,100 Model  NonConverted incorrectly predicted (%) 10.98  Total correctly classified (%)  13,512 141 1,395  307,461 6,954  73.70 . 87.23  22.30 12.77  NonConverted correctly predicted (%) 89.02 93.77  49,322  74.70  25.30  92.68  6.23 7.32  93.64 92.18  1,846  42,388  80.77  19.23  91.03  8.97  90.60  2,557  13,606  72.12  27.88  87.39  12.61  84.97  383  4,501  60.57  39.43  85.80  14.20  83.83  641  10,455  64.74  35.26  88.92  11.08  87.53  197  1,509  64.47  35.53  76.41  23.59  75.03  5,059  154,041  74.34  25.66  88.71  11.29  88.25  Number Number of Converted Total Non-. . of claims correctly analyzed Converted Converted predicted claims claims (%)  Converted incorrectly predicted (%)  88.38  Table IV.9 Accuracy of the regression models when ECP is the cutoff point (1989-1992 sample)  18.26  90.07  9.93  90.07  Overall  320,973  Burn Model  7,095  Contusion 50,717 Model 44,234 Laceration Model 16,163 Fracture Model 4,884 Bursitis Model Joint 11,096 Inflammation Model Carpal Model 1,706 Sprain/Strain 159,100 Model  307,461  Total correctly classified (%)  81.74  Number Number of Converted Total of Nonclaims correctly analyzed Converted Converted predicted claims claims (%) 13,512  NonConverted incorrectly predicted (%) 18.26  NonConverted correctly predicted (%) 81.74  Model  Converted incorrectly predicted (%)  81.74  141  6,954  90.07  9.93  1,395  49,322  84.16  15.84  84.16  15.84  84.16  1,846  42,388  86.54  13.46  86.54  13.46  86.54  2,557  13,606  80.18  19.82  80.18  19.82  80.18  383  4,501  74.59  25.41  74.59  25.41  74.59  641  10,455  77.32  22.68  77.32  22.68  77.32  197  1,509  69.22  30.78  69.22  30.78  69.22  5,059  154,041  80.58  19.42  80.58  19.42  80.58  47 On the other hand, when the ECP is employed, the percent of converted claims correctly predicted is equal to the percentage of not-converted claims correctly predicted and the percentage all claims correctly predicted, but the percentage of claims correctly predicted is slightly lower in this case. The percent of converted claims correctly predicted is however considerably higher than for the alternative case that employs the CF as the cutoff point.  As regards the overall model (recall that this model does not require knowledge of the nature of injury), notice its remarkable high accuracy that goes beyond 80%. We concluded therefore that this model can be used with high confidence in early stages of a claim (first 6 months) when information regarding the nature of injury might be missing from the data warehouse.  IV.5 Cross-validation of the Models  The final step in the model building process is the cross-validation or out of sample testing of the selected models. Cross-validation usually involves checking the model against a set of independent data. In our study, the cross-validation sample consisted of all FFSTD claims (78,471) that had injury dates in 1993. To validate the logistic regression models, we employed them to predict the likely outcome (converted or non-converted) of all claims from the crossvalidation sample (1993 claims). The results are presented in Tables IV. 10 and IV. 11 below for the two different cutoff points used in the study.  The percentages of claims correctly classified in the cross-validation sample (1993 claims) are very close to the percentages of claims correctly classified in the 1989-1992 sample used to build the regression models. For the Carpal model the accuracy is even higher for the cross-validation sample. The results presented in Tables IV. 10 and IV. 11 clearly indicate that the validation of the regression models was successfully completed, and thus we conclude that the logistic regression models can be used to predict the outcome of other FFSTD claims that were not included in the 1989-1992 sample.  48  Table IV. 10 Model cross-validation (CF is the cutoff point)  Model  Overall Burn Model  26.68 24.00  NonConverted correctly predicted (%) 88.79 93.07  Total Number of Number of Converted Converted Nonclaims Converted correctly • incorrectly Converted predicted claims predicted claims (%) (%) 78,471 1,568  11,323 Contusion Model Laceration 10,005 ' Model 4,041 Fracture • Model 1,316 Bursitis Model 3,120 Joint Inflammation Model 562 Carpal Model Sprain/Strain 40,164 Model  Total correctly classified (%) 88.01  25  1,543  73.33 76.00  315  11,008  78.73  21.27  91.02  8.98  90.68  448  9,557  79.46  20.54  91.00  9.00  90.49  683  3,358  71.74  28.26  84.90  15.10  82.68  105  1,211  69.52  30.48  83.57  16.43  82.45  203  2,917  76.85  23.15  85.43  14.57  84.87  57  505  71.93  28.07  77.62  22.38  77.05  1,319  38,845  76.57  23.43  88.06  11.94  87.68  NonConverted incorrectly predicted (%) 18.61  Total correctly classified (%)  3,297  75,174  NonConverted incorrectly predicted (%) 11.21 6.93  93.45  Table IV. 11 Model cross-validation (ECP is the cutoff point)  Converted correctly predicted (%)  Converted incorrectly predicted (%)  75,174  81.39  18.61  NonConverted correctly predicted (%) 81.39  10.11  89.89  10.11  89.89  Model  Total Number of Number of Nonclaims Converted claims Converted claims  Overall  78,471  3,297  81.39  Burn Model  1,568  25  1,543  89.89  Contusion Model Laceration Model Fracture Model Bursitis Model Joint Inflammation Model Carpal Model  11,323  315  11,008  82.79  17.21  82.79  17.21  82.79  10,005  448  9,557  86.43  13.57  86.43  13.57  86.43  4,041  683  3,358  78.34  21.66  78.34  21.66  78.34  1,316  105  1,211  73.36  26.64  73.36  26.64  73.36  3,120  203  2,917  75.07  24.93  75.07  24.93  75.07  57  505  71.08  28.92  71.08  28.92  71.08  1,319  38,845  80.06  19.94  88.06  19.94  80.06  562  Sprain/Strain 40,164 Model  49 IV.6 Critical STD Days Paid  We now know that the logistic regression models can be employed to predict the probability of conversion of any claim provided we know the nature of injury, STD days paid and age of claimant. The cutoff point is a specific value of the probability of conversion that allows one to classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). Accordingly, all claims that have an estimated probability of conversion higher or equal to the cutoff point are classified as potential conversions, while all claims that have an estimated probability of conversion smaller than the cutoff point are classified as likely non-conversions.  For each logistic regression model the cutoff point translates into critical values of the predictor variables incorporated into the models. Since for each claim age of claimant at injury is a fixed quantity, we are going to determine the critical STD days paid for each subset of claims determined by the age of claimant. For instance, for the Contusion model the critical STD days paid is 46 for a 40-year old claimant and a cutoff point of 0.03. Thus all 40-year old claimants that have contusion will be classified as likely conversion i f they reach or exceed 46 STD days paid.  Similar to the cutoff point, Critical STD days paid allows the decision-maker to classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). Any claim that has accumulated a number of STD days paid higher or equal to the critical value of STD days paid is classified as potential conversion. Similarly, any claim that has accumulated a number of STD days paid lower than the critical value of STD days paid is classified as potential non-conversion. Tables IV. 12 and IV. 13 below provides for all regression models the values of the critical STD days paid for various ages of claimant and using the CF and ECP respectively as the cutoff points.  Observe that for all models that incorporate age as a predictor the value of critical STD days paid decreases as the age of claimant increases. On the other hand, for the Burn and Carpal models, which do not include age as predictor, the critical value of STD days paid does not significantly  50 vary with age of claimant. Also notice that the critical STD days paid are lower when we use ECP as the cutoff point.  Table IV. 12 Critical STD days paid (CF is the cutoff point)  Model Overall Burn Model Contusion Model Laceration Model Fracture Model Bursitis Model Joint Inflammation Model Carpal Model Sprain/Strain Model  CF 0.04 0.02 0.03 0.04 0.16 0.08 0.06  20 63 28 65 30 87 77 82  25 58 28 60 28 85 73 76  30 54 28 55 27 83 69 69  35 49 28 51 26 81 66 63  Age 40 45 28 46 24 79 62 56  0.11 0.03  80 83  80 75  80 67  80 60  80 52  80 44  80 36  80 28  80 20  45 25 23 22 16 59 34 24  50 21 23 17 15 57 30 18  55 17 23 12 14 55 26 11  60 12 23 8 12 53 23 5  69 28  69 21  69 13  69 5  45 41 28 41 23 76 58 50  50 36 28 36 22 74 55 43  55 32 28 32 20 72 51 37  60 27 28 27 19 70 47 30  Table IV. 13 Critical STD days paid (ECP is the cutoff point)  Model Overall Burn Model Contusion Model Laceration Model Fracture Model Bursitis Model Joint Inflammation Model Carpal Model Sprain/Strain Model  ECP 0.0292 0.0163 0.0189 0.0292 0.1160 0.0585 0.0396  20 47 23 46 23 69 53 57  25 43 23 41 21 67 49 50  30 39 23 36 20 65 45 44  35 34 23 32 19 63 41 37  Age 40 30 23 27 18 61 38 31  0.0984 0.0224  69 68  69 60  69 52  69 44  69 36  51  V. IMPLEMENTATION The main objective of the project is to use the results obtained - especially the tables with critical STD days paid - for improving the decision making process at the Workers' Compensation Board of British Columbia. The issue of implementing this model into the W C B ' s decisionmaking system was raised during several meetings with key W C B executives. The suggestion was to do the implementation through the following four phases.  Phase 1: Informing potential users within the WCB. The purpose was to present the new decision making model through meetings held in various departments of the W C B . During September and October we made several presentations in the Prevention and Compensation Services divisions of the W C B . The presentation in the Compensation Services Department proved to be especially useful since it was attended by more than 80 entitlement officers and case managers, who will be the actual users of the new decision making model. The subsequent discussion clarified a lot of problems, and it seemed that the vast majority of the entitlement officers and case managers were willing to use the model.  Phase 2: Pilot study. The next step towards implementation of the results of the Converted Claims Project should be a pilot study. The goal is to start using the new decision making tool on a subset of claims, involving only a limited number of customer service representatives (CSRs), entitlement officers (EOs), and case managers (CMs). The format of this study could take on several different forms. For instance: -  have a given number of staff at various levels of the process (CSRs, EOs, and CMs) use the model results to trial all claims they receive; have all staff members use the results to trial claims of a few specified injury types only (e.g., fractures); Combination of these two. That is, have selected staff members use the results to trial only specific injuries.  Our expectation is that the combination format would be the most likely choice. Under this framework, we suggest that at least all the fracture claims should be in the pilot. Some of the reasons are as follows  52 •  Fractures make up over 5% of all the claims (approximately 4,000 per year);  •  Approximately 19% of all the converted claims are fractures, so we would be addressing a significant cost issue;  •  Fractures' conversion factor of 16% is significantly higher than the average CF of 4.2%, so we would be potentially averting more future costs than with another injury type;  •  The pilot study does not require the involvement of too many CSRs, EOs, and CMs.  •  The average number of fractures to be considered is approximately 1,200 per year (see Table IV.4), or 5 per day.  This phase will likely commence at the end of November 1999, and will be closely monitored by the Risk Management Group of the W C B , who will provide the necessary assistance.  Phase 3. Using feedback to improve the models. The feedback received from entitlement officers and case managers will be used to improve the new decision making model through (1) Readjusting the cutoff points to improve accuracy of predictions , but at the same time not to exceed the claim handling capacity of decision makers , and (2) Building new regression models by taking into consideration other potential predictors such as assessment classification (CLSBIN) and body part type.  Phase 4. Full implementation. The full implementation of the critical STD days paid in the decision making process of the W C B will be considered i f the pilot study performed on fractures with a reduced number of decision makers is successful. The annual number of claims to be analyzed during this phase is approximately 18,000, that is, the expected annual number of claims that will be classified as likely conversion. This means an average of 75 claims per day that require special treatment since there is approximately 240 working days in a year.  53  VI. A R E A S FOR FURTHER INVESTIGATION This section presents some possible refinements of the existing methodology and potential new areas/directions for further research on converted claims.  (1) Improving the selection method of the Cutoff Point. The goal is to improve the selection method of the cutoff point by using prior probabilities and costs of incorrect predictions to determine the optimal cutoff so that the expected cost of incorrect predictions will be minimized. The major problem is to assess the costs of incorrect predictions. Since no study has been performed at the W C B on this issue, we will raise the possibility of an appropriate study during meetings with key people of the W C B . (2) Use body part as an additional predictor. Body part type might be an additional predictor that could be used to improve the predictive accuracy of the regression models. The main difficulty to be addressed is how to reduce the number of cells created i f nature of injury would be cross-classified with body part type; see the Stratification of the Data section earlier for details. One approach to solve this problem would be to group the body parts into a meaningful way so that body parts within the same group have a comparable level of severity. For instance, we might want to create two separate groups; one group for finger fractures and another group for arm and leg fractures respectively since the severity of arm or leg fractures is usually much higher than the severity of finger fractures. (3) Use both assessment classification (CLSBIN) and nature of injury as primary classification variables. The purpose of this investigation should be to evaluate whether or not assessment classification could be used together with nature of injury to build specific model applicable to some of the high risk industries such as Logging, Wood manufacturing, and Building. The essence of this investigation would be to cross-classify C L S B I N with nature of injury, and then for each particular model to determine whether or not age and STD days paid are statistically significant predictors. (4) Build models that incorporate only STD days paid up to and including the FSTD payment. The logistic regression models presented in this study use STD days paid up to and including the FFSTD payment; see time line for decision making presented in Figure IV. 1. The goal of this new study would be to develop logistic regression models that would use STD days paid  54 at the FSTD payment. The new models would allow one to identify earlier claims that are likely to convert, that is, at the time of the FSTD payment. The downside of this approach would be the lower accuracy of the models since at the FSTD payment date less information regarding STD claims would be available. (5) Multinomial Regression. Multinomial regression is similar to logistic regression, but is more general because the response variable is not restricted to two categories; see Hosmer and Lemeshow (1989) and Agresti (1990) for more details. If we decided to use this method, we would be able to build a model that could be used to model not only two types of STD claims (converted and reopened or inactive) but all three types of STD claims as defined previously (converted, active and inactive). (6) .Proportional Hazards Regression. Proportional hazards regression is a method for modeling time-to-event data in the presence of censored cases; see Kalbfleisch and Prentice (1980) and Cox and Oakes (1984) for a detailed presentation of this method. In our particular case the conversion is the event and the time between injury date and F V R or F L T D payment date is the survival time. A l l claims that have not been converted up to the time when the study ends are considered censored cases. The main advantage of using this model is that it would allow the investigation of more recent STD claims such as claims that had injury dates after 1994.  55  VII. C O N C L U S I O N The project's main objective was to investigate the conversion process of short-term disability claims that had injury dates between January 1, 1989 and December 31, 1992 and subsequently received a FFSTD payment. The investigation showed that:  1) It is possible to group FFSTD claims into three categories based on the types of payments and amounts they receive. The three categories are inactive, active and converted FFSTD claims respectively. 2) Converted claims make up only about 4.2 % of all the FFSTD claims of our sample, but they received about 64.3% ($1,173 million) of the total payments and awards ($1,824 million) to July 1999. 3) Active and inactive claims have a less significant impact on the W C B ' s reserves since they make up about 95.8% of all the claims, but receive only about 35.7% ($651 million) of the payments made to July 1999. 4) Because of the significant financial impact on the W C B ' s reserves of converted claims, they were classified as high risk claims, while active and inactive (not converted) claims that pose a much lower financial threat are grouped together and classified as low risk claims. 5) The average cost per converted claim is $86,223, which is about 41 times higher than the average cost per not-converted claim ($2,101). 6) Since the outcome of any FFSTD claim is either converted or not converted, we used logistic regression to model the conversion process of FFSTD claims. 7) Since ten of the most frequent injury types make up 95.72% of all the claims (323,098) of our sample, we built separate regression models for each of them. 8) For each subset of claims determined by nature of injury we defined the conversion factor as the proportion of converted claims with respect to all the claims. 9) Besides nature of injury, used as a primary stratification variable, we identified STD days paid up to and including the FFSTD payment date and age of claimant as statistically significant predictors.  56 10) We also built an overall regression model that does not require the knowledge of nature of injury. This model can be used in earlier stages (first 6 months) of a claim, when nature of injury is not available in the Data Warehouse. 1 l)The logistic regression methods allow one to determine the probability of conversion of any FFSTD claim provided one knows the age of claimant and STD days paid. 12) As regards the probability of conversion, the more STD days paid and the older the claimant, the higher its value. 13) The cutoff point is a specific value of the probability of conversion that allows one to classify a given claim as a likely conversion (high-risk) or non-conversion (low-risk). 14) To evaluate the optimal value of the cutoff point we need to know the costs of incorrect predictions. Since these costs were not available, we used the conversion factor and ECP respectively as alternative cutoff points. The ECP is that value of the cutoff for which the proportion of converted claims correctly classified equals the proportion of not-converted claims correctly classified. 15) The accuracy of the models is quite remarkable, with most of the model correctly classifying over 80% of all the claims. 16) The cutoff point determines critical values of STD days paid. Critical STD days paid allows the decision-makers to classify a given claim as likely conversion or non-conversion. 17) The implementation of the project will commence with a pilot study performed on fractures and with a limited number of entitlement officers and case managers. 18) C L S B I N and body part type might be use in a further study as additional predictors to improve the accuracy of the models. 19) Multinomial logistic regression and proportional hazards regression were identified as alternative methods to model the conversion process of claims. 20) The methodology developed in this study can be applied to solve similar problems at the WCB.  57  REFERENCES Agresti, A. (1990). Categorical Data Analysis, New York, Wiley-Interscience. Chatfield, C. (1995). Problem Solving - A Statistician's Guide, 2nd edn, Boca Raton, Chapman & Hall/CRC. Cox, D. R. and Oakes, D. O. (1984). Analysis of Survival Data, London, Chapman and Hall. Draper, N . R. and Smith, H. (1998). Applied Regression Analysis, 3rd edn, New York, WileyInterscience. Fattedad, S. and Charron, M . (1998). Claims Inventory Control, The Workers' Compensation Board of British Columbia, Vancouver. Hosmer, D. W. and Lemeshow, S. (1989). Applied Logistic Regression, New York, John Wiley & Sons. Jessup, B . and Gallie, K. (1996). High Risk Project Report, The Workers' Compensation Board of British Columbia, Vancouver. Johnson, D. E., (1998). Applied Multivariate Methods for Data Analysis, Pacific Grove, Duxbury Press. Kalbfleisch, J. D. and Prentice, R. L . (1980). The Statistical Analysis of Failure Time Data, New York, John Wiley and Sons. Kelton, D. W., Sadowski, R. P. and Sadowski, D. A. (1998). Simulation with Arena, Boston, Massachusetts, WCB/McGraw-Hill. Lemeshow, S., Teres, D., Avrunin, J. S. and Pastides, H . (1988). Predicting the Outcome of Intensive Care Unit Patients, Journal of the American Statistical Association 83, 348-356. Mason, K . (1999). Claim Duration Model, The Workers' Compensation Board of British Columbia, Vancouver. McCullagh, P. andNelder, J. A . (1989). Generalized Linear Models, 2nd edn, London, Chapman and Hall. Neter, J., Kutner, M . H., Nachtsheim, C. J. and Wasserman, W. (1996). Applied Linear Statistical Models, 4 edn, Chicago, Irwin. th  Ostle, B . and Malone, L. C. (1988). Statistics in Research, 4th edn, Iowa State University Press, Ames, Iowa. Peck, G. (1999). Crystal Reports 7: The Complete Reference, Boston,  58 Massachusetts, Osborne McGraw-Hill. Pregibon, D. (1981). Logistic Regression Diagnostics, Annals of Statistics 9, 705-724. Ryan, T. P. (1996). Modern Regression Methods, New York, Wiley-Interscience. Snedecor, G. W. and Cochran, W. G (1989). Statistical Methods, 8th edn, Iowa State University Press, Ames, Iowa. Tabachnick, B . G. and Fidell, L . S., (1996). Using Multivariate Statistics, 3rd edn, New York, Harper Collins College Publishers. Thomas, L . C , (1999). A Survey of Credit and Behavioral Scoring; Forecasting Financial Risk of Lending to Consumers, in preparation. Thompson, C. M . , (1985). Characteristics Associated with Outcome in a Community Mental Health Partial Hospitalization Program, Community Mental Health Journal 21, 179-188. W C B (1996). The Workers' Compensation Act. The Workers' Compensation Board of British Columbia,^ ancouvev. W C B (1998). 1998 Annual Report, The Workers' Compensation Board of British Columbia, Vancouver. Wiginton, J. C , (1980). A Note on the Comparison of Logit and Discriminant Models of Consumer Credit Behavior, Journal of Financial and Quantitative Analysis 15, 757-770.  59  Appendix Histograms of the transition times of paths 1, 2, 3 and 4 Figure A.1 Path 1 - Histogram of the Injury Date to FFSTD payment date Transition Time  Histogram - Injury Date to FFSTD Transition Time (Path 1)  Transition Time (months)  Figure A.2 Path 1 - Histogram of the FFSTD to F V R Transition Time  Histogram - FFSTD to FVR Transition Time (Path 1)  60 Figure A.3 Path 1 - Histogram of the F V R to F L T D Transition Time  Histogram - FVR to FLTD Transition Time (Path 1) 600 500 400 o c <x> 300 13 CT CD  LL  200 ! 100 -  O  <o  n>  <V  *§> >§> & & & # <y Transition Time (months)  A*  ^  #  c£  Figure A.4 Path 2 - Histogram of the Injury Date to FFSTD payment date Transition Time  Histogram - Injury Date to FFSTD Transition Time (Path 2) 600 500 | >, 400 o ^ 300 I cr £  200  100 0 *  *  O  &  o>  #  fc>  #  #  #  ^  A* <b*  Transition Time (months)  #  ^  N  #  ^  o$>  N  61 Figure A.5 Path 2 - Histogram of the FFSTD to F V R Transition Time  Histogram - FFSTD to FVR Transition Time (Path 2)  Transition Time (months)  Figure A.6 Path 3 - Histogram of the Injury Date to FFSTD payment date Transition Time  Histogram - Injury Date to FFSTD Transition Time (Path 3)  c  Transition Time (months)  62  Figure A.7 Path 3 - Histogram of the FFSTD to FLTD Transition Time  Histogram - FFSTD to FLTD Transition Time (Path 3)  *  * <y  <v*  >§> &  & & # <o° # A* Transition Time (months)  # # ^ d> ^ N  N  Figure A.8 Path 4 - Histogram of the Injury Date to FFSTD payment date Transition Time  Histogram - Injury Date to FFSTD Transition Time (Path 4)  63  Figure A.9  Path 4 - Histogram of the FFSTD to FLTD Transition Time  Histogram - FFSTD to FLTD Transition Time (Path 4)  Transition Time (months)  Figure A .  10 Path 4 - Histogram of the F L T D to F V R Transition Time  Histogram - FLTD to FVR Transition Time (Path 4)  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0089506/manifest

Comment

Related Items