TWO-STAGE MAXIMUMLIKELIHOOD APPROACH FORITEM-LEVEL MISSING DATA INREGRESSIONbyLihan ChenB.A., Simon Fraser University, 2012A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF ARTSinThe Faculty of Graduate Studies(Psychology)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)August 2017c© Lihan Chen 2017AbstractPsychologists often use scales composed of multiple items to measure un-derlying constructs, such as well-being, depression, and personality traits.Missing data often occurs at the item-level. For example, participants mayskip items on a questionnaire for various reasons. If variables in the datasetcan account for the missingness, the data is missing at random (MAR). Mod-ern missing data approaches can deal with MAR missing data effectively,but existing analytical approaches cannot accommodate item-level missingdata. A very common practice in psychology is to average all available itemsto produce scale means when there is missing data. This approach, calledavailable-case maximum likelihood (ACML) may produce biased results inaddition to incorrect standard errors. Another approach is scale-level fullinformation maximum likelihood (SL-FIML), which treats the whole scaleas missing if even one item is missing. SL-FIML is inefficient and proneto bias. A new analytical approach, called the two-stage maximum likeli-hood approach (TSML), was recently developed as an alternative (Savalei& Rhemtulla, 2017b). The original work showed that the method outper-formed ACML and SL-FIML in structural equation models with parcels.The current simulation study examined the performance of ACML, SL-FIML, and TSML in the context of bivariate regression. It was shownthat when item loadings or item means are unequal within the composite,ACML and SL-FIML produced biased estimates on regression coefficientsunder MAR. Outside of convergence issues when the sample size is smalland the number of variables is large, TSML performed well in all simulatedconditions, showing little bias, high efficiency, and good coverage. Addi-tionally, the current study investigated how changing the strength of theMAR mechanism may lead to drastically different conclusions in simulationstudies. A preliminary definition of MAR strength is provided in order todemonstrate its impact. Recommendations are made to future simulationstudies on missing data.iiLay SummaryPsychologists often use scales made up of many items to measure underlyingconstructs, such as well-being, depression, and personality traits. Missingdata often occurs for some of the items. For example, participants mayskip items on a questionnaire for various reasons. A very common practicein psychology is to average all available items to calculate the final scorewhen there is missing data at the item level. The current study investigatedthe performance of this approach when applied to a common data analysistechnique called regression. It was found that the method was only ableto produce accurate results under the unrealistic assumptions that all itemsmeasure the target construct equally well, and the items all have the sameaverage scores. In contrast, a new approach, called the two-stage maximumlikelihood approach (Savalei & Rhemtulla, 2017b) was shown to producecorrect estimates in a large array of conditions for regression.iiiPrefaceThe code used in the simulation study is based on the earlier work of Dr.V. Savalei, and Mijke Rhemtulla. Specifically, the code used for the two-stage approach is directly adapted by Dr. Savalei from the previous studyto be used in the current study. The mathematical theory behind the two-stage approach, as well as the details of the method, have been published inSavalei and Rhemtulla (2017b).ivTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . vList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Missing Data Mechanisms . . . . . . . . . . . . . . . . . . . . 42.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Generating Missing Data . . . . . . . . . . . . . . . . . . . . 62.3 MAR Strength . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Missing Data Techniques . . . . . . . . . . . . . . . . . . . . . 123.1 General Missing Data Techniques . . . . . . . . . . . . . . . 123.1.1 Listwise and Pairwise Deletion . . . . . . . . . . . . . 123.1.2 Mean Imputation . . . . . . . . . . . . . . . . . . . . 133.1.3 Full-information Maximum Likelihood . . . . . . . . . 133.2 Item-level Missing Data Techniques . . . . . . . . . . . . . . 143.2.1 Available-case Maximum Likelihood . . . . . . . . . . 143.2.2 Scale-level Full-information Maximum Likelihood . . 153.2.3 Two-stage Approach . . . . . . . . . . . . . . . . . . 163.2.4 Multiple Imputation . . . . . . . . . . . . . . . . . . . 18vTable of Contents4 Simulation Studies in Missing Data . . . . . . . . . . . . . . 204.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.2 Measuring Performance . . . . . . . . . . . . . . . . . . . . . 204.3 Number of Replications . . . . . . . . . . . . . . . . . . . . . 224.4 Previous Literature . . . . . . . . . . . . . . . . . . . . . . . 225 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255.1 Design Overview . . . . . . . . . . . . . . . . . . . . . . . . . 255.2 Model Description . . . . . . . . . . . . . . . . . . . . . . . . 265.3 Missing Data Conditions . . . . . . . . . . . . . . . . . . . . 305.4 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326.1 ML Convergence . . . . . . . . . . . . . . . . . . . . . . . . . 326.2 Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366.3 Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456.4 Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637.1 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . 637.1.1 For Analyzing Empirical Data . . . . . . . . . . . . . 637.1.2 For Missing Data Simulation Studies . . . . . . . . . 647.2 Limitations and Future Directions . . . . . . . . . . . . . . . 68References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69viList of Tables5.1 Parameters of population models. . . . . . . . . . . . . . . . . 285.2 Derived parameters of population models. . . . . . . . . . . . 305.3 Parameters of missing data mechanisms. . . . . . . . . . . . . 316.1 Nonconvergence in TS. . . . . . . . . . . . . . . . . . . . . . . 336.2 Nonconvergence in TS. . . . . . . . . . . . . . . . . . . . . . . 346.3 Nonconvergence in TS. . . . . . . . . . . . . . . . . . . . . . . 356.4 Raw bias under equal loadings and equal means in the 8-itemconditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376.5 Raw bias under equal loadings and unequal means in the 8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 386.6 Raw bias under unequal loadings and equal means in the 8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 396.7 Raw bias under unequal loadings and unequal means in the8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . 406.8 Raw bias under equal loadings and equal means in the 14-itemconditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416.9 Raw bias under equal loadings and unequal means in the 14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 426.10 Raw bias under unequal loadings and equal means in the 14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 436.11 Raw bias under unequal loadings and unequal means in the14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . 446.12 ESE ratio under equal loadings and equal means in the 8-itemconditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466.13 ESE ratio under equal loadings and unequal means in the8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . 476.14 ESE ratio under unequal loadings and equal means in the8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . 486.15 ESE ratio under unequal loadings and unequal means in the8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . 49viiList of Tables6.16 ESE ratio under equal loadings and equal means in the 14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 506.17 ESE ratio under equal loadings and unequal means in the14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . 516.18 ESE ratio under unequal loadings and equal means in the14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . 526.19 ESE ratio under unequal loadings and unequal means in the14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . 536.20 Coverage under equal loadings and equal means in the 8-itemconditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.21 Coverage under equal loadings and unequal means in the 8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 566.22 Coverage under unequal loadings and equal means in the 8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 576.23 Coverage under unequal loadings and unequal means in the8-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . 586.24 Coverage under equal loadings and equal means in the 14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 596.25 Coverage under equal loadings and unequal means in the 14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 606.26 Coverage under unequal loadings and equal means in the 14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . 616.27 Coverage under unequal loadings and unequal means in the14-item conditions. . . . . . . . . . . . . . . . . . . . . . . . . 62viiiList of Figures2.1 Examples of MAR . . . . . . . . . . . . . . . . . . . . . . . . 82.2 More examples of MAR . . . . . . . . . . . . . . . . . . . . . 92.3 Examples of a perfect MAR and an MCAR . . . . . . . . . . 115.1 Scale illustration: composite scale as a latent variable model. 267.1 Coverage under Strong MAR showing a large difference be-tween methods . . . . . . . . . . . . . . . . . . . . . . . . . . 667.2 Coverage under Weak MAR showing a much smaller differ-ence between methods . . . . . . . . . . . . . . . . . . . . . . 67ixAcknowledgementsI would like to thank my parents, without whose loving support I wouldnot have been able to set upon the academic path; my friends, who arethe source of my strength; and my colleagues, who are the source of myinspiration. A special thank to Victoria Savalei, my supervisor, for beingsupportive, patient, as well as immensely illuminating on complex topics.I would also like to thank other faculty members in the quantitative area,Dr. Jeremy Biesanz, and Professor Emeritus Dr. Ralph Hakstian, for theirincredible knowledge and passion in the subject. I would like to express mygratitude to Dr. Mark Blair, who showed me the path to academic researchwhen I was an undergraduate. Finally, a big thank to Social Sciences andHumanities Research Council (SSHRC), for the financial support during myresearch.xDedicationTo Mom and Dad, who are pretty freaking cool.xiChapter 1IntroductionFacts do not cease to exist because they are ignored.—Aldous HuxleyMissing data is an important issue in statistics. It is a common occurrence inpsychological research that, when handled incorrectly, can cause researchersto draw incorrect conclusions from the data. Unfortunately, poor meth-ods of handling missing data, due to their convenience and intuitiveness,are prevalent in the field of psychology. Since the seminal work by Rubin(1976), the field of statistics has made significant progress in techniques thatcan deal with missing data in a scientifically satisfactory manner. Drasticimprovements in computing power of modern home computers, and otheradvancements in computing science, have enabled programmers to packageinnovations and discoveries in quantitative research into tools that are rea-sonably easy to use for the general psychologists. Programs for structuralequation modelling (SEM)1, such as MPlus (Muthe´n & Muthe´n, 2010), havebuilt in functionalities to deal with missing data. For the more technicallysavvy psychologists, R (R Core Team, 2016) packages like norm (Schafer &Graham, 2002) and lavaan (Rosseel, 2012) provide programming tools formissing data handling. Many authors of these statistical packages improveand expand their tools based on results from the field of statistics. The cur-rent study aims to become part of that literature by examining a prevalentyet not often studied phenomenon: missing data at the item level.Psychologists, especially social, personality, and clinical psychologists,often use scales that are composed of multiple items. For example, theUCLA Loneliness Scale (Russell, 1996) has 20 items, and the Big Five Inven-tory has 8-10 items on each personality dimension. Participants answeringan inventory questionnaire may refuse to answer questions that they deemtoo sensitive, leave items blank when they do not apply, or simply skip items1The nature of structural equation modelling is outside of the scope of the currentpaper. In short, SEM takes a model that contains paths and fits the model-impliedparameters (typically the variance-covariance matrix; sometimes variable means are alsoincluded) to the observed sample values in order to estimate path coefficients and residuals.1Chapter 1. Introductiondue to carelessness. In any of these situations, as well as others, item-levelmissing data occurs. Item-level missing data is not unique to questionnairestudies. For example, when a confirmatory factor analysis (CFA) containstoo many variables in a factor, parcelling is recommended (e.g., Little et al.,2013). Item-level missing data occurs when at least one of the variables ina parcel is missing (Savalei & Rhemtulla, 2017b). In cognitive psychologyexperiments, participants may fail to complete some of the trials, or datamay be lost due to equipment failure under systematic circumstances (Chenet al., 2012). Item-level missing data occurs when the mean score of thosetrials are used as composite variables in the analysis.It is common practice for psychologists to ignore item-level missing databy simply taking the means of available items. This conventional person-mean imputation, sometimes known as proration (Mazza et al., 2015), isalso known as the available-case maximum likelihood approach, or ACML(Savalei & Rhemtulla, 2017b). When the number of missing items is deemedtoo high, the scale is treated as missing entirely, with all the available itemsdeleted. The threshold for choosing between ACML and deletion can varygreatly from one researcher to another. For example, Culbert et al. (2013)applied ACML for scales missing 10% of its items or less, and treated thewhole scale as missing when more than 10% of the items were missing,whereas Beebe et al. (2007) applied the same procedure at 50% of the itemsmissing. While quantitative researchers have long known that such a pro-cedure was theoretically unsound (Schafer & Graham, 2002), simulationstudies that investigate the exact extent of the bias and conditions where itarises have been scarce.More recently, quantitative researchers began to show interest in item-level missing data, and multiple studies comparing the performance of dif-ferent missing data techniques have been published (Orcan, 2013; Parent,2013; Mazza et al., 2015; Savalei & Rhemtulla, 2017b). Yet, until veryrecently, multiple imputation (MI) remained the only statistically sound ap-proach that can deal with item-level missing data. Multiple imputation is asimulation-based numerical technique for handling missing data. As such, ithas qualities that are not desired by some researchers. For example, when MIis performed on the same dataset multiple times, it may produce a slightlydifferent result every time. These results may also be sensitive to parame-ters in the method such as the number of imputations performed. Typically,for researchers who favour a more analytic approach, a method known asthe full-information maximum likelihood (FIML) is available. However, thisapproach cannot handle item-level missing data when the model is at thecomposite level. A more flexible analytic alternative, called the two-stage2Chapter 1. Introductionapproach (TS; Yuan & Bentler, 2000; Savalei & Bentler, 2009), was recentlyextended to address item-level missing data (Savalei & Rhemtulla, 2017b).The original work on TS showed good performance of the method in thecontext of item-level missing data that occur when SEM parcels containvariables with missing data using a simulation study. The current study ex-tends the previous study on TS and exam the performance of the approachin the context of regression under a large array of simulation conditions,including much smaller sample sizes.3Chapter 2Missing Data Mechanisms2.1 OverviewBased on the seminal work of Rubin (1976), and further elaborations ofLittle and Rubin (2002), a popular classification of missing data mecha-nisms contains the following three categories: missing completely at random(MCAR), missing at random (MAR), and missing not at random (MNAR).While missing data mechanisms can have drastic impacts on the results ofthe analyses, it is often impossible to statistically determine which types ofmissing data mechanisms underlies any particular dataset: Some tests forMCAR exist (Little, 1988; Kim & Bentler, 2002), but it is impossible totest for MAR. Researchers must decide if the assumption of a missing datamechanism is tenable before proceeding to handle missing data using anytechnique. Ideally, researchers should be prepared for missing data prior toconducting their research, and plan their research accordingly in producemissingness that is easier to handle; however, in reality, predicting whattype missing data will occur is difficult. For example, in longitudinal stud-ies, dropout is common, but the reasons for dropout are often unknown.Unfortunately, the terms and acronyms for these mechanisms can be con-fusing and not widely understood by researchers outside of the field of statis-tics. For example, while MCAR can be considered a special case of MAR,when MAR is used, it more often refers to all types of MAR except MCAR.While MAR stands for missing at random, most common explanations forMAR describe it as missing systematically based on observed variables inthe dataset.It is important to distinguish between the causes of missing data andmissing data mechanisms. Missing data mechanisms are agnostic to thechain of events which lead to the occurrence of the missing data. They onlydescribe the mathematical relationship between the probability of missingdata and the variables in the dataset. In natural language, the definition ofMAR can be interpreted as the following: After observed variables withinthe dataset have been factored out, the probability of missing data and anyvariable outside of the dataset are independent of each other. In mathemat-42.1. Overviewical terms, if we define the matrix of missingness M which indicates wherethe data is missing, and let the complete dataset be Y , the observed portionof Y be Yobs, the missing portion of Y be Ymis, and θ be unknown param-eters, missing at random is defined based on the relationship between theconditional distribution of M given complete data and unknown parameters,f(M |Y, θ), and the conditional distribution given observed data instead ofcomplete data,f(M |Y, θ) = f(M |Yobs, θ) ∀Ymis, θ. (2.1)It should be clear from this definition why it is impossible to test whetherMAR holds. In order to do so, one would require a sample of Ymis, whichis by definition never available. Missing completely at random, or MCAR,is the case where the above is true regardless of the distribution of Ymis,namely,f(M |Y, θ) = f(M |θ) ∀Y, θ. (2.2)In natural language, MCAR describes the case where the probability of miss-ing data is independent of any data, missing or otherwise. It necessarilyfollows that the probability of missing data and any variable outside of thedataset are independent of each other after factoring in observed variableswithin the dataset. That is to say, MCAR can be considered a special case ofMAR. In empirical research, MCAR is a very restrictive and often unrealis-tic assumption. It is perhaps for this reason that it is typically distinguishedfrom MAR as a separate category of missing data mechanism. Once we haveestablished that for the missing data mechanism to be MAR, the missingdata probability can only be dependent on observed data, it is straightfor-ward to define missing not at random, MNAR, as its opposite, which is caseswhere the probability of missingness is dependent on missing data. With anadditional mild assumption of the independence between model parametersand the parameters guarding missing data, MCAR and MAR are known asignorable missing data mechanisms (Little & Rubin, 2002), because existingmissing data techniques can generally deal with these types of missing datavery well. MNAR, on the other hand, is always nonignorable, which is muchmore difficult to handle. Because dealing with MNAR often require explicitmodelling of the missing data mechanism on a case by case basis, studies of52.2. Generating Missing Datageneral techniques for dealing with missing data are typically limited to thestudy of the ignorable mechanisms, MCAR and MAR.2.2 Generating Missing DataWhen quantitative researchers investigate the performance of missing datatechniques through simulation studies, they must first generate datasets withmissing data that satisfy the definition of missing data mechanisms. It iscommon to start with a complete dataset, and then delete data followinga specific algorithm. The algorithm for generating MCAR missingness isrelatively straightforward: If each data point on the variable with missing-ness is given a missing probability equal to the intended overall populationmissing probability for that variable, the population missing probability willarise naturally. Alternatively, if the researcher wishes to fix the percentageof missing data at the sample level, a proportion of the variable or set ofvariables equal to that percentage is randomly selected and deleted.2 Inmost cases, researchers simply generate missingness in each variable in theset independently. It has been suggested, however, that special missingnessrules should be used to ensure the same number of missing patterns acrossdifferent missingness conditions (Savalei & Bentler, 2005).There are many algorithms to generate data that satisfy the definitionof a MAR mechanism in the literature. The simplest is the single cutoffapproach. For example, the missing probability in one variable is 100% ifthe corresponding data point in the conditioning variable is below the meanof the entire conditioning variable. In mathematical terms, let N be thesample size, u be the conditioning variable, ui be the ith data point in thatvariable, y be the variable with missing data, and yi be the ith data point inthat variable, then yi is missing if ui <∑u/N . This approach presents thestrongest selection mechanism that may not be entirely realistic. Outsideof an earlier paper by Yuan and Bentler (2000) which removed values in avariable when the corresponding variable is above than the population me-dian value (resulting in a 50% population missing rate), it has not appearedin recent missing data literature. In reality, missing probability is unlikelyto follow such a strict cutoff. Typically, more sophisticated single cutoffapproaches are used. Savalei and Rhemtulla (2017b) applied the cutoff at2As is common in simulation studies, researchers may select certain population pa-rameters for the simulation models, and generate samples based on those parameters.The sample statistics corresponding to those population parameters can vary greatly fromsample to sample. Some model attributes, such as the rate of missing data, are sometimesfixed at the sample level instead.62.2. Generating Missing Datarandom until a specific percentage of missing data at the sample level isreached. As long as the percentage of data above the cutoff is larger thanoverall variable missing percentage, this procedure also leads to the deletionof data points above the cutoff of at a probability lower than 100%.A natural extension of the single cutoff approach is the multiple cutoffapproach. For example, in Savalei and Rhemtulla (2017b), nonlinear MARwas created through the deletion of data both above a cutoff based on atop percentile and a cutoff below a bottom percentile, resulting in missingdata at both ends of the distribution. A further extension of the multiplecutoff leads to the continuous approach. For example, in studies by Enders(Enders, 2001, 2003), MAR was created by assigning a missing probabilityto yi based on the one minus the percentile of ui. For example, if u1 wasat the 25th percentile, y1 would then be assigned a missing probability of1−.25 = .75. Data would be deleted from the top percentile until the desiredmissing rate is reached. While probability of missingness assigned this wayis linearly related to the rank of the conditioning variable, it has a sigmoidalrelationship with the variable itself when it has a normal distribution. In-stead of relying on the distribution of the conditioning variable to achievethe sigmoidal relationship, Mazza et al. (2015) used a logistic regressionto generate the probability of missingness based on the conditioning vari-able. An advantage of the logistic regression approach is that the strengthof the selection mechanism can be controlled via the resulting R2 betweenthe conditioning variable and the missing probabilities. However, the logis-tic regression approach may not be as intuitive. The shape of the logisticregression is controlled by the coefficients a and b in its underlying linear re-gression aX + b. While psychologists are used to encountering standardizedregression coefficients smaller than 1, the a in a logistic regression MARoften needs to be much higher (e.g. greater than 10) if the conditioningvariable X is distributed as N(0, 1).72.2. Generating Missing Data0.000.250.500.751.000 25 50 75 100Conditioning VariableMissing Probability0.000.250.500.751.000 25 50 75 100Conditioning VariableMissing ProbabilityFigure 2.1: The left panel shows an example of a single cutoff MAR. Theright panel shows an example of a multiple cutoff MAR, specifically thenonlinear MAR in Savalei and Rhemtulla (2017b).82.2. Generating Missing Data0.000.250.500.751.000 25 50 75 100Conditioning VariableMissing Probability0.000.250.500.751.000 25 50 75 100Conditioning VariableMissing ProbabilityFigure 2.2: The left panel shows an example of a continuous ranked basedMAR that relies on the normal distribution of the conditioning variable forits sigmoidal shape, along with its cutoff hybrid. The right panel shows alogistic regression based MAR.92.3. MAR Strength2.3 MAR StrengthThe MAR mechanism is defined by the relationship between the probabilityof missing and the conditioning variable, but the exact type of relation-ship is not specified as long as the probability of missing is independent ofeverything else. There is little discussion on the potential impact on thedifferent types of MAR. Mazza et al. (2015) used a logistic regression withan R2 = .4 between the conditioning variable and the missing probabili-ties to define what they called a “strong selection mechanism.” When therelationship between the probability of missing value and the conditioningvariable is weak, in order to satisfy MAR, it is necessary that f(M |θ) andf(M |Yobs, θ) are sufficiently similar. That is to say, a weaker MAR is moresimilar to MCAR. If a method shows low bias for a dataset with strongMAR missing data, then in theory, it should always work just as well, if notbetter, for a dataset with weak MAR missing data. The opposite, however,is likely not to be the case. In the most extreme case, methods the workfor the weakest MAR, namely MCAR, may not be applicable to MAR miss-ingness at all. It is therefore important for simulation studies to ensure amechanism strong enough for comparison.There is no established measurement of the strength of MAR in theliterature. R2 may be used as a loose indication in the case of a logisticregression, but it does not reflect the intuitive concepts of MAR strength.For example, in the case of a perfect MAR with a single cutoff such asthe one shown in the left panel of Figure 2.3, R2 does not indicate thefact that there is no way for such a MAR mechanism to be any stronger.The current study takes a preliminary step to explore potential consequencesstrong versus weak MAR mechanisms by looking at MAR strength under thesingle cutoff case. Intuitively, under the single cutoff case, the definition ofa perfect MAR is straightforward: the missing probability above (or below)the cutoff must be exactly 100%. Formally, let y ∼ N(µ, σ2) be a variablewith MAR missingness, and u be the conditioning variable that is relatedto the missingness in y, and pmis be the overall percentage of missingnessin y, and the strength of the MAR mechanism be ξ. Then, when ξ = 1, yiis always missing whenever ui is above a certain cutoff value.With perfect MAR, it is necessarily the case that the cutoff value mustbe the 1−pmis quantile of N(µ, σ2), in order to ensure the correct populationmissingness percentage. As such, ξ is bounded by [pmis, 1]. When ξ = pmis,the missing mechanism is simply MCAR. It is trivial to deduce that fora MAR strength of below 1, the corresponding percentile cutoff value is1 − pmisξ . For example, if pmis = .15 and MAR strength is .8, and Xm ∼102.3. MAR StrengthN(1.5, 1), the cutoff value is the top .8125 percentile of N(1.5, 1), which is2.387. That means, for every data point in Xm that is greater than 2.387,the corresponding Xi is assigned an 80% chance of missing. In Enders(2003), the missing data rate was controlled by stopping the procedure oncethe deletion of data from the highest percentile achieves the target samplemissing rate. This essentially creates a hybrid between continuous MARand cutoff MAR (see Figure 2.2). With the logistic regression approach, itis possible to control the missing data rate by adjusting the coefficients inthe underlying linear regression function. However, both of these approachesare considerably more complex when it comes to quantifying MAR strengthin an intuitive manner. Furthermore, more complex rules for MAR mayinvolve using a combination of conditioning variables for the deletion ofeach missing variable (e.g. Savalei & Bentler, 2005; Savalei & Bentler, 2009.The combined strength of the selection mechanism will be more difficultto quantify for those scenarios. The current study will focus on adjustingMAR strength under the single cutoff MAR approach. The aim of theexercise is to provide preliminary evidence of the impact of MAR strength,in order to motivate future investigations into quantifying MAR strengthunder different simulation methods.0.000.250.500.751.000 25 50 75 100Conditioning VariableMissing Probability0.000.250.500.751.000 25 50 75 100Conditioning VariableMissing ProbabilityFigure 2.3: The left panel shows an example of a perfect MAR with sin-gle cutoff. The right panel shows MCAR when per variable the missingprobability is 50%.11Chapter 3Missing Data Techniques3.1 General Missing Data TechniquesMissing data mechanisms are crucial because they affect which kind of miss-ing data technique is appropriate. In order to understand missing datatechniques at the item level, it is helpful to first cover general missing datatechniques. For example, available-case ML in item-level missing data hasthe same effect as applying mean imputation, a more general technique, ona person level. Most issues facing the general techniques also occur at theitem-level. In addition, item-level missing data has its own unique set ofchallenges that prevent the direct application of certain general techniques.3.1.1 Listwise and Pairwise DeletionDeletion is the most common technique used by psychologists when handlingmissing data. Listwise deletion is also known as the complete case analysis.It involves deleting the entire row of data whenever at least one variable ismissing. The approach is often deemed acceptable when less than 5% ofthe rows have at least one variable missing (Parent, 2013), but it usually re-sults in an unjustifiable loss of power. In the most drastic scenario, if thereis at least one participant is missing for each variable, the entire datasetis deleted under listwise deletion, despite low overall missingness. Pairwisedeletion tries to get around this problem by performing independent listwisedeletion on the relevant subset of data for each parameter estimate. For ex-ample, for a covariance matrix between variables X1, X2, and X3 where onlyX1 contains missingness, cov(X2, X3) will be computed using all observeddata in X2, X3 which contains no missingness, whereas cov(X1, X3) is com-puted based on observed data after a “listwise” deletion performed based onX1 and X3. An obvious issue with this procedure is that each cell in the co-variance matrix is not estimated under the same sample size, which can leadto incorrect standard error. Furthermore, a covariance matrix estimated thisway may not be positive-definite, which may render the subsequent compu-tations in the analysis impossible. While both listwise and pairwise deletion123.1. General Missing Data Techniquesapproaches may produce unbiased estimates under MCAR, they are biasedunder MAR (Graham, 2009).3.1.2 Mean ImputationMean imputation is another conventional missing data handling technique.It involves replacing missing data in each variable with the mean of theavailable data for that variable. While this approach provides an unbiasedestimate of the mean of the variable under MCAR, it underestimates thestandard deviation. Intuitively, because standard deviation is calculatedby combining the difference between the individual value and the overallmean at each data point, replacing missing values with the mean wouldsimply introduce a lot of zeroes into the calculation. Under MAR, meanimputation produces biased estimates not only for the means, but also thestrength of the relationships between variables under MAR. For example,for variable X and variable Y to have a strong linear relationship, Yi shouldbe close Y when Xi is close to X. If Xi = X due to mean imputation, Yimay not be anywhere close to Y . As such, mean imputation obfuscates therelationship between variables. While it is intuitively very straightforwardto understand why mean imputation is bad, and social science researcherstend to avoid this approach, its equivalent under item-level missing data, theavailable-case maximum likelihood, is often used. The perils of the item-levelequivalent of mean imputation will be discussed in detail in a later section.3.1.3 Full-information Maximum LikelihoodIn contrast to the conventional methods, full-information maximum like-lihood (FIML) is a modern approach that produces consistent estimatesunder the multivariate normality assumption for MAR missing data (Alli-son, 2003). Intuitively, the approach uses the relationship between variablesto infer what the missing values are most likely to be. For example, if twovariables, X and Y , are positively related, then if, for some i, Xi is thehighest value in the variable, then the missing value Yi is also likely to bea high value. FIML utilizes this information not by imputing, or filling in,the missing values. Instead, the approach proceeds with the analysis usingthe estimated most likely population parameters. Mathematically, the like-lihood function of the population parameters of a dataset with missing datacan, under multivariate normality, be written as133.2. Item-level Missing Data TechniquesL(µ,Σ) =∏if(xi|µi,Σi), (3.1)where xi is the observed information, and µi, Σi are the population param-eters with the elements corresponding to missingness in xi deleted. The µand Σ that maximizes the likelihood. In software, this is typically achievedby maximizing the sum logL =∑Ni=1 logLi, wherelogLi = Ki − 12log|Σi| − 12(xi − µi)′Σi−1(xi − µi). (3.2)Ki is a constant that depends on the number of complete data points. Thestandard errors can be subsequently be obtained by inverting the associ-ated information matrix, which is given by the negative Hessian of the log-likelihood in (3.2).While FIML has the advantage of being consistent and asymptoticallyefficient, it is not possible to use it directly when the model involves compos-ites of the original items without estimating the item structure explicitly.The standard FIML approach does not contain an explicit model to dealwith item-level missing data. The na¨ıve application of FIML to item-levelmissing data turns out to be problematic, as we will explore in more detailin the following section on item-level missing data techniques.3.2 Item-level Missing Data Techniques3.2.1 Available-case Maximum LikelihoodAvailable-case maximum likelihood (ACML), sometimes known as prora-tion, is a popular technique for dealing with item-level missing data (Mazzaet al., 2015). When a scale has missing items, the ACML method simplytakes the mean of all available item for each participant as their personalscale score. Obtaining the scale mean this way produces the same resultsas performing person-mean imputation, that is to say, each participant’smissing values within each scale are replaced by the mean of the scores ofthe available items in that scale. Just like mean imputation, ACML pro-duces incorrect standard errors by essentially pretending there is no missingdata. Also similar to mean imputation, ACML tends to produce biasedmean estimates and underestimate relationship between scales under MARmissingness. This is because, if data is missing for items with more extreme143.2. Item-level Missing Data Techniquesvalues due to MAR, the ACML scale scores will drift closer to the mean andexhibit a lower variance as a result. In some cases, ACML produces biasedresults even under MCAR (Mazza et al., 2015). While it is intuitive to real-ize that the mean imputation approach is problematic, the similar problemwith ACML may not be as apparent. As long as the loadings of all the itemsare the same within each scale, i.e. , they have the same weight in termsof measuring the underlying construct, and the item means are the samewithin the scale, at least one simulation study showed that the ACML pa-rameter estimates may be unbiased under MAR (Mazza et al., 2015). Evenin that ideal scenario, the ACML standard errors are still off, because theydon’t take into account the varying precision of different composite scoresdue to different levels of missing data. More importantly, it is often the casethat the assumptions of equal item means and equal item correlations areuntenable. Despite its disadvantages, ACML is undoubtedly convenient. Asa result, researchers may be tempted to assume equal loadings, equal itemmeans, and even MCAR in order to apply the approach. However, it shouldbe noted that previous research under these particular scenarios only rec-ommends the approach for small amounts of missing data (< 5%) (Parent,2013). The current study will examine potential ACML bias under bivariateregression in a wide range of conditions.3.2.2 Scale-level Full-information Maximum LikelihoodFull-information ML, as previously mentioned, is a favourable option foranalyzing missing data. FIML uses the ML estimates of population param-eters instead of imputing the missing values. Because models for compositescores require population parameter estimates (such as the sample variance-covariance matrix) at the scale level, an extra step needs to be taken toproduce the scale-level parameter estimates from the item-level data. Un-der the most naive approach, this would mean listwise deletion at the itemlevel. That is to say, for each participant, if any item is missing, the wholeassociated scale is treated as missing. This approach, called scale-level FIML(SL-FIML; Savalei & Rhemtulla, 2017b) would clearly result in a significantpower loss. However, there is a far more insidious problem: If the itemsthat the MAR mechanism depend on (i.e. items that predict missingness)are incidentally deleted during the SL-FIML process, the missing mecha-nism becomes MNAR. Under MNAR, SL-FIML is likely to produce biasedestimates. It has been shown that SL-FIML is sensitive to simulation con-ditions. For example, SL-FIML may produce unbiased estimates when allitems within each scale have the same means across all participants, but153.2. Item-level Missing Data Techniquesbiased estimates when the means are different between items (Mazza et al.,2015). Mazza et al. (2015) aimed to address this issue by using some ofthe items as auxiliary variables in the model. However, all items used asauxiliary variables in this way cannot be also used in the composite scales,e.g., if a scale has 5-items, only 4 can be used as auxiliary variables. As aresult, this approach does not in fact use all the items for full-informationML missing data handling. Its consistency and asymptotic efficiency is notguaranteed (Savalei & Rhemtulla, 2017a).3.2.3 Two-stage ApproachIn order to utilize the ML approach correctly, we need a better way to ob-tain the population parameter estimates than the naive item-level deletionapproach. An analytical approach, known as the two-stage maximum like-lihood approach (TSML, or simply TS) has been extended to handle item-level missing data (Savalei & Rhemtulla, 2017b). The original TS approachis a more flexible, as well as easier to understand, alternative to FIML. TheTS approach has been improved and modified to handle nonnormal miss-ing data (Yuan & Bentler, 2000) and to utilize auxiliary variables (Savalei& Bentler, 2009). The most recent paper (Savalei & Rhemtulla, 2017b),which is a direct predecessor of the current study, introduced the methodin the context of handling parcelling in SEM. The following description isreproduced from that paper.Let X = (X1, X2, ..., Xp1) be the predictor scale variable and Y =(Y1, Y2, ..., Yp2) be the outcome scale variable. All variables are then rep-resented as p×1 vector, Z = (X,Y ), where p = p1 +p2. Let Xc =∑XipmandYc =∑Yipn, where pm, pn are the number of items in Xc and Yc, respectively,then Zc = (Xc, Yc). When the data contains m scales, p =∑mi=1 pi, andZc is a m× 1 vector containing m vectors of composite scale scores. Whilethe current study only involves the case of m = 2, the generalized descrip-tion from Savalei and Rhemtulla (2017b) is preserved for clarity. While thisstudy uses regression models, in order to apply the TS approach, they willbe conducted as path analyses.During Stage 1, the TSML approach fits the saturated model to thewhole data set to obtain the maximum likelihood estimates of the populationparameter, in this case the estimates of means and the variance-covariancematrix, µˆ and Σˆ. Let γˆ = (vechΣˆ, µˆ) (Magnus & Neudecker, 1989), whichis the (p∗ + p) × 1 vector where p∗ = 12p(p + 1), which is the horizontallyvectorized form of Σˆ containing all of its nonredundant elements, as well163.2. Item-level Missing Data Techniquesas µˆ. Let Aˆγ be the associated observed information matrix.3, then theestimated asymptotic covariance matrix of γ is given by Ωˆγ = Aˆ−1γ .Stage 1a is the extension for item-level missing data. In this stage, Ωˆγ isconverted from item-level to the corresponding scale-level matrix, denoted asΩˆδ, where δ the scale-level corresponding vector to γ, and δˆ′ = ((vechΣˆ)′c, µˆ′c)for the scale-level mean and covariance matrix estimates µˆc and Σˆc. Inorder to perform the conversion, a m × p selection matrix, C, is definedsuch that Zc = CZ, where Zc contains the scale scores, and Z contains theitems scores. The corresponding saturated estimates for the means and thecovariance matrix are therefore µˆc = Cµˆ and Σˆc = CΣˆC. These will beused during Stage 2 to produce the ML fit for the composite scale model.For example, for two composites containing 3 items each (i.e. m = 2,p = p1+p2 = 3+3 = 6), C =(1 1 1 0 0 00 0 0 1 1 1). The scale-level estimatesis obtained byδˆ′ =(D+m(C ⊗ C)Dp 00 C)γˆ= Cbigγˆ, (3.3)where Dp is the duplication of order p, and D+m is the Moore-Penrose inverseof the duplication matrix of order m (Magnus & Neudecker, 1989). Theasymptotic covariance matrix of δˆ is Ωˆδ˜ = CbigΩˆγCbig.During Stage 2, the composite scale model is fit to the population pa-rameter estimates of Zc. Let the model representation be µc = µc(θ) andΣc = Σc(θ), where θ is the q×1 vector of model parameters. The fit processminimizes the complete data ML function,FML(θ) = tr{ΣˆcΣ−1c (θ)} − log|ΣˆcΣ−1c (θ)|+ (µˆc − µc(θ))′Σ−1c (θ)(µˆc − µc(θ))−m. (3.4)The resulting TSML estimates are θ˜, and the corresponding estimates ofmeans and covariances are µ˜c = µc(θ˜) and Σ˜c = Σc(θ˜). Let δ˜ = ((vechΣ˜c), µ˜c),then the correct estimates of the asymptotic covariance matrix of θ˜, account-ing for missing data, is given by the “sandwich” estimator,Ω˜θ˜ = (∆˜′H˜∆˜)−1∆˜′H˜ΩˆδH˜∆˜(∆˜′H˜∆˜)−1. (3.5)3An explicit asymptotic expression under MAR is given by Savalei (2010).173.2. Item-level Missing Data Techniqueswhere∆˜ =∂δ(θ)∂θ′∣∣∣∣θ=θ˜(3.6)is the matrix of model derivatives evaluated at the TSML estimates θ˜, andH˜ =(.5D′m(Σ˜−1c ⊗ Σ˜−1c )Dm 00 Σ˜−1c)(3.7)is the normal theory weight matrix from Stage 2. (∆˜′H˜∆˜)−1 is the “naivecovariance matrix of parameter estimates that would be produced by de-fault under complete-data ML estimation (Yuan & Bentler, 2000). Sinceregression models are saturated, and ∆˜ is a square matrix, in a bivariateregression, (3.5) can be simplified intoΩ˜θ˜ = ∆˜−1H˜−1(∆˜′)−1∆˜′H˜ΩˆδH˜∆˜∆˜−1H˜−1(∆˜′)−1= (∆˜′)−1Ωˆδ∆˜−1. (3.8)3.2.4 Multiple ImputationWhile this chapter has focused on analytical approaches in dealing withmissing data, there is a popular numerical alternative, namely multiple im-putation (MI) (Rubin, 1987). To understand the process of multiple impu-tation intuitively, one can imagine aggregating a large number of stochasticimputations. During each stochastic imputation, an ML estimate is obtainedfor each missing value based on the model and the observed data. A certainamount of noise, congruent with the rest of the data, is added to this esti-mate. While each stochastic imputation does not capture the uncertaintyof missing data, by repeating the process many times, the asymptoticallycorrect standard error can be derived, by combining the variance acrossall imputations, a kind of uncertainty indicator, with the average variancewithin each imputation. Traditionally, a number of 3 to 5 imputations wasrecommended. However, research has since shown that this can be highlyinadequate in many circumstances. For example, Graham et al. (2007) rec-ommended at least 40 imputations for 50% missing information.Mathematically, MI is a Bayesian approximation approach based on theposterior distribution. Based on the original work of Rubin (1987), underMCAR or MAR, p(θ|Zobs), the posterior distribution under observed data,and p(θ|Zobs, Zmis), the posterior distribution under complete data, are re-lated by standard probability theory as183.2. Item-level Missing Data Techniquesp(θ|Zobs) =∫p(θ, Zmis|Zobs)dZmis=∫p(θ|Zmis, Zobs)dZmisp(Zmis|Zobs)dZmis. (3.9)Intuitively, the posterior distribution of model parameters θ given the ob-served data, p(θ|Zobs), can be simulated based on the relationship betweenthe missing and the observed data, by drawing missing values from theirjoint distribution p(Zmis|Zobs). By imputing the missing values with thedrawn values, Z(d)mis, θ can be drawn from the posterior distribution basedon the imputed data set p(θ|Zobs, Z(d)mis). Over some large number of imputa-tions M , the model parameters can be estimated by integrating the averageposterior distributions over multiple imputations,E(θ|Zobs) ≈∫θ1M∑p(θ|Z(d)mis, Zobs)dθ. (3.10)The variance of the resulting parameter estimate can be approximated bycombining the average variance V = 1M∑Mm=1 Vm and the variance amongthe estimates across imputations, B = 1M−1∑Mm=1(θˆm − θˆ)2,Var(θ|Yobs) ≈ V +B. (3.11)Savalei and Rhemtulla (2017b) observed that item-level MI and TSML arelikely asymptotically equivalent, although no proof was provided. The twoapproaches had similar performance in the SEM simulations performed inthat study. While MI has the advantage of being more generally applicablewithout custom written code, it tends to produce different results every timethe analysis runs, and the process itself can be sensitive to method-relatedparameter adjustments. In contrast, TSML is an analytical approach, andthus may be deemed more desirable by researchers. Furthermore, it wasshown in Savalei and Rhemtulla (2017b) that while the performance of MIand TSML are virtually identical at high sample size, TSML outperformsMI when the sample size is relatively small and the rate of missing data ishigh. The current study will focus on developing and investigating TSML.19Chapter 4Simulation Studies inMissing Data4.1 BackgroundAn important part of a statistical method is its estimator of the populationparameter. As the method is used to estimate the population parameterfrom different random samples, the estimates from every sample start toform a distribution. The theoretical distribution under an infinite amountof random samples is known as the sampling distribution, which can beused to evaluate the performance of an estimator. For example, the meanstatistics for a random variable X ∼ N(µ, σ2) with a sample size of N is∑Ni=1 xi/N , which has a sampling distribution X ∼ N(µ, σ2/N). We caninfer that the estimator is unbiased, and that it becomes more precise asthe sample size grows (i.e., consistency). Most conventional procedures aredesigned in ways such that, when the assumptions are met, the samplingdistributions of relevant estimators and test statistics can be derived ana-lytically. However, real life data is messy, and they rarely fall neatly withinthe confines of assumptions necessary for analytical derivations. It is of-ten the case that the sampling distributions are asymptotic, i.e. they arecorrect under an infinite sample size. As a result, quantitative researchersturn instead to simulation studies as a way to assess the performance ofanalytical procedures under finite sample sizes. By simulating drawing ran-dom samples from the population over a long period of time, the samplingdistribution of the estimators and test statistics can be estimated, which isin turn used to assess the performance of the procedure.4.2 Measuring PerformanceFirst and foremost, for any statistical method, its estimators of populationparameters should be unbiased, or at least asymptotically unbiased. Anunbiased estimator is one where the mean of the sampling distribution cor-204.2. Measuring Performanceresponds to the population parameter. As long as a estimator is unbiased,the estimates will reflect the true population parameter on average. This isnot the case with biased statistics. In simulation studies, the calculation ofthe bias in the parameter estimation in a procedure is straightforward. LetM stand for the number of replications for each model , β be the populationparameter, and βˆi be the parameter estimate on the ith replication, thenβˆi =∑Mi=1 βi is the mean of all the parameter estimates. Then, the esti-mated bias is simply the difference between the population parameter andthe empirical run average β − βˆi.Secondly, it is also important that the estimator is efficient, i.e. it doesnot tend to produce wildly different results across a large number of randomsamples. In other words, the sampling distribution of the estimator has asmall standard deviation. In a simulation study with M replications, this isestimated using the empirical standard error (ESE), given asESE =√√√√ 1M − 1M∑i=1(βˆi − βˆ)2. (4.1)There is a potential trade off between bias and efficiency. While an unbiasedestimator is always better in the long run and for large sample sizes, amore efficient estimator with a small bias may be more likely to produce anestimate closer to the population parameter in small samples. A commonway to combine the measurement of bias and efficiency is to simply replacethe run mean βˆ with the population parameter β. This leads to the rootmean squared error (RMSE),RMSE =√√√√ 1M − 1M∑i=1(βˆi − β)2. (4.2)The measurements cover critical properties of the sampling distribution it-self. However, statistical methods provide not only parameter estimates butalso statistical inferences. With assumption violations, the procedure mayproduce incorrect standard errors. It is therefore important to not only ex-amine the empirical standard error over the replication runs, but also thestandard error produced by the method in each run. A common way to ex-amine the performance of the inference procedure is to look at its coverage,the percentage of replication runs in which the 95% confidence interval cov-ers the population parameter. The confidence interval for the ith replicationrun is simply the range [βˆi − 1.96SEi, βˆi + 1.96SEi].214.3. Number of Replications4.3 Number of ReplicationsWhile some older simulation studies on missing data use 250 or 500 repli-cations (Enders, 2001; Yuan & Bentler, 2000), most simulation studies onthe subject perform 1,000 replications on each case scenario (e.g., Enders,2003; Savalei & Bentler, 2009; Mazza et al., 2015), and occasionally 2,000replications (Gottschall et al., 2012). The minimum required number ofreplications, MLB, can be computed (Burton et al., 2006) asMLB =Z1−α/2σβδβ, (4.3)where σβ is the standard deviation of the estimator, Z1−α/2 the z-scorecorresponding to the 1− α/2 percentile of a standard distribution, with αbeing the Type I error rate of the inference procedure. δβ sets the permissibledifference from β, which controls for the precision of the calculation of MLB.While the standard 1,000 replications is typically larger than MLB, it maynot always be true with a very small sample size N . As σβ typically increaseswith a smaller N , MLB will also become larger. The current study examinescases in sample sizes of 50, which is more realistic for bivariate regression,but also much smaller than most similar studies. The smaller sample sizemay necessitate a larger number of replications. For example, although someprevious work on the TS approach (e.g., Yuan & Bentler, 2000) used only 500replications, they were run on much larger sample sizes, such as N = 1000and 2000. The more recent study by Savalei and Bentler (2009) whichimproved upon the method used sample sizes of N = 200, 400, and 600.While using Nrep = 500 and Nrep = 1000 were sufficient for those studies, itis not immediately clear that it is enough for N = 50. Fortunately, becauseNrep = 1000 replications is already more than sufficient for most studies,it appears reasonable to adopt the same. For example, a simulation studyby Enders (2001) on the use of FIML in a three predictor regression wasperformed with a smallest sample size of N = 100 with only Nrep = 250replications.4.4 Previous LiteratureRecent development of the TS approach can be traced back to Yuan andBentler (2000). Motivated by the need to deal with nonnormal missingdata in SEM, the authors developed asymptotically correct test statisticsfor FIML and TS under MCAR. The MCAR assumption was chosen be-cause earlier work suggested that under nonnormality, ML estimates may224.4. Previous Literaturelose consistency if the missing mechanism is MAR. In order to examine theperformance of the ML approaches, a simulation study of 500 replicationswas conducted using sample sizes of 1,000 and 2,000. A perfect MAR wassimulated by removing all values corresponding to the top 50% of the condi-tioning variable, with a 50% population missing rate. MCAR was generatedby removing data on even sample numbers, with a 50% sample missing rate.It was discovered that for the medium sample size, at least, the ML ap-proaches did not show serious bias under MAR with nonnormality (Yuan &Bentler, 2000).Savalei and Bentler (2009) extended the TS approach to incomplete non-normal data and studied its empirical performance relative to FIML. Basedon the observation by Yuan (2009) that normal theory MLE can be consis-tent under MAR for certain misspecified (i.e. nonnormal) population mod-els, the study provided TS estimates under MAR. The authors favoured TSover FIML due to its intuitiveness as well as versatility. A simulation studywas conducted under smaller sample sizes of 200, 400, and 600, as well asmore realistic missing data rate of 15% and 30% per variable under MARsimilar to Savalei and Bentler (2005). A two-factor CFA with 9 variables perfactor. In a set of conditions, 6 additional variables were added as auxiliaryvariables. Loadings were set to be similar, with three sets of .6, .65, and .7.The correlation between factors was set to .4. Auxiliary variables were setto correlate with variables in the factors at .1 and .3. The simulation showeda small loss of efficiency of TS compared to FIML, but recommended TSdue to it being more conservative, and therefore less prone to Type I error.Savalei and Rhemtulla (2017b) expanded the TSML approach to item-level missing data in the context of SEM parcelling. An additional step wasadded to Stage 1 to convert the item covariance matrix and mean estimatesto the composite scale level. The study compared ACML, SL-FIML, andMI to TSML. The simulation study investigated a three factor CFA modelwith a first order factor and a secondary order factor. Each factor contained9 variables which were separated into three parcels. The factor loadingswere .3, .4, .5 on one set of conditions, and .6, .7, .8 on another. Samplesizes of 200, 400, and 600 were used. 1,000 replications were run on eachcondition combination. 14 variables contained missing data with 5%, 15%,or 30% missing per variable based on 6 conditioning variables. The studyfound biased factor loadings under nonlinear MAR for both ACML and SL-FIML. However, latent regression coefficient estimates for all methods wererelatively unbiased, especially for ACML. The study also found that thetwo-stage approach was overall the most efficient for estimating the latentregression coefficients, as well as their asymptotic covariance matrix, when234.4. Previous Literaturethe loadings were relatively low, outperforming MI in virtually every samplesize and missing rate condition.Mazza et al. (2015) showed that the lack of bias in regression coefficientfrom ACML may arise due to equal means and equal loadings. The studyinvestigated bivariate regression with two composite scales, with 8 or 16items each, with sample sizes of 200 and 500. MAR was created usinglogistic regression with R2 = .4, for 5%, 15%, and 25% per variable missingrate. In the equal loading conditions, all loadings are .75. In the unequalloading conditions, loadings for items without missing data were set to .5instead. For the unequal mean conditions, the measurement intercepts onthe items with missing values were set to .5. The study showed that wheneither the item loadings or item means varied within the same scale, ACMLresulted in biased estimates of the regression coefficients. The study did notexamine the TSML approach.24Chapter 5Method5.1 Design OverviewThe current study investigates available-case ML (ACML), scale-level full-information ML (SL-FIML), and two-stage ML (TSML) approach in thecontext of a bivariate regression. Both variables are composite scales com-prised of k items where k = 8 or 14. Because this study generated data fromsmall samples, the TSML approach encountered a high number of runs wherethe estimation of the saturated model for the components in Stage 1 did notconverge, even in the 8-item condition. Because the issue was caused by thecombination of small sample size and large number of variables, it was de-cided that instead of matching the 16-item condition in Mazza et al. (2015),14 items would be used instead in the second set of conditions. The miss-ing mechanisms studied are MCAR, strong linear MAR, weak linear MAR,strong nonlinear MAR, and weak nonlinear MAR. Strong linear MAR is de-fined as 80% missing when above a certain cutoff, and weak linear MAR isdefined as 30% missing when above a certain cutoff. Strong nonlinear MARis defined as 80% missing when above an upper cutoff or below a lower cutoff,and weak nonlinear MAR is defined as 30% missing when above an uppercutoff or below a lower cutoff. The exact cutoff depends on overall missing-ness. Sample sizes of 50, 100, and 200 are studied. Either 15% or 25% ofthe first half of the items in the predictor scale are missing on every scale,resulting in an overall missingness of 7.5% and 15.5% in the predictor scale.In total, there are 5 mechanisms×3 sample sizes×2 missing rates = 60 sim-ulation conditions. These simulation conditions are applied to each of the 8population models described in the following section. 1,000 replications arerun for each combination.255.2. Model Description5.2 Model DescriptionThe regression model is simply a bivariate regression where the predictorand outcome variable are both composite scales with k items each,Yc = βXc + α, (5.1)Xc =∑ki=1Xik, (5.2)Yc =∑ki=1 Yik. (5.3)where k is the number of items in the composite scale.The relationship between items and scales is specified under a factormodel. As seen in Figure 5.1, each item has a certain loading onto theunderlying structure it aims to measure. The first half of the items haveloadings equal to λ1, and the second half have loadings equal to λ2. Themodel is set up this way in order to compare conditions where λ1 = λ2 andλ1 6= λ2. Furthermore, Xi ∼ N(µ1, 1) for i = 1, 2, ..., k/2, and Xi ∼ N(µ2, 1)for i = k/2+1, k/2+2, ..., k/2. In each condition, items in Yc have the samepopulation model as items in Xc.FX1 X2 ... Xk/2 X k2+1X k2+2 ... Xkλ1 λ1 ... λ1 λ2 λ2 ... λ2Figure 5.1: The composite scale as a latent variable model.Let Xc be a scale composed of X with k items, where k is an evennumber. Each item has a variance of 1. The loadings for the first k2 itemsare λ1, and the loadings for the lastk2 items are λ2. For each item of X,Xi ∼ N(µ, 1). Similarly, Yi ∼ N(µ, 1). The loadings for the first k2 itemsare λ3, and the loadings for the lastk2 items are λ4. Let ΣXY be a k × kmatrix filled with the same element σxy, and265.2. Model Descriptionσxy = cov(Xc, Yc)= cov(1kl′X,1kl′Y )=1k2l′Cov(X,Y )l (5.4)Let ΣXX be the k × k covariance matrix of all items in X,ΣXX =1λ21 1λ21 λ21 1... ... ... ...λ1λ2 λ1λ2 λ1λ2 ... 1λ1λ2 λ1λ2 λ1λ2 ... λ22 1λ1λ2 λ1λ2 λ1λ2 ... λ22 λ22 1, (5.5)and ΣY Y be the k × k covariance matrix of all items in Y ,ΣY Y =1λ23 1λ23 λ23 1... ... ... ...λ3λ4 λ3λ4 λ3λ4 ... 1λ3λ4 λ3λ4 λ3λ4 ... λ24 1λ3λ4 λ3λ4 λ3λ4 ... λ24 λ24 1, (5.6)then for every model during each replication, complete data was drawn fromthe multivariate normal distribution with the covarianceΣA =(ΣXXΣXY ΣY Y). (5.7)For the population models, scales with equal and unequal item loadingsare simulated. In equal conditions, λm = .7 ∀m ∈ {1, 2, 3, 4}. In un-equal conditions, λ1 = λ2 = .5, and λ3 = λ4 = .7. These values devi-ate from Mazza et al. (2015) in that the overall scale reliability stays thesame under the equal vs unequal conditions. The loadings are also low-ered somewhat so that the overall scale reliability does not become unreal-istically high. Scales with equal and unequal item means are simulated.275.2. Model DescriptionIn equal conditions, µm = .0 ∀m ∈ {1, 2, 3, 4}. In unequal conditions,µ1 = µ2 = 0, and µ3 = µ4 = .5. High and medium correlation be-tween the predictor and outcome scales are simulated. In equal loadingconditions and high correlation conditions, σxy = .37. In order to retainroughly equivalent βs between equal vs unequal loadings, for both 8 vs 14items conditions, σxy = .32 for unequal loadings and high correlation con-ditions. Similarly, for medium correlation conditions, σxy = .22 when load-ings are equal and σxy = .18 when loadings are unequal. In total, there are2 loading conditions×2 mean conditions×2 covariance conditions = 8 pop-ulation models. These model condition parameters are summarized Table5.1.Model λ1= λ3 λ2= λ4 µ1= µ3 µ2= µ4 σxy1EEH .7 .7 0 0 .372UEH .5 .8 0 0 .323EUH .7 .7 0 .5 .374UUH .5 .8 0 .5 .325EEM .7 .7 0 0 .226UEM .5 .8 0 0 .187EUM .7 .7 0 .5 .228UUM .5 .8 0 .5 .18Table 5.1: Parameters of each population model. The models were namedafter the following conventions: The number at the start of the name is forthe sole purpose of indexing and sorting. The first letter “E” or “U” standsfor “equal” or “unequal” loadings; the second letter “E” or “U” stands for“equal” or “unequal” means, and the third letter “H” and “M” stands for“high” or “medium” regression coefficients. As such, “7EUH” indicates “the7th model, which has equal loadings, unequal means, and medium regressioncoefficient.’From these 8 population models, other parameters such as the scalereliability, ρscale, as well as the regression coefficients, β and α, are derived.Following (5.5), the sum of elements of ΣXX isl′ΣXX l = (k2− 1)2(λ21 + λ22) +k22λ1λ2 + k, (5.8)and the scale reliability ω2 is285.2. Model Descriptionω2 =(∑ki=1 λi)2V ar(φ)l′Σ̂XX l=(k2λ1 +k2λ2)2(k2 − 1)2(λ21 + λ22) + k22 λ1λ2 + k. (5.9)In order to calculate the regression coefficients, we will need to derive thepopulation variance of the predictor composite scale,var(Xc) =1k2l′ΣXX l=(k2 − 1)2(λ21 + λ22) + k22 λ1λ2 + kk2. (5.10)Then, the regression coefficient can be calculated asβY ·X =cov(Xc, Yc)var(Xc)=k2σxy(k2 − 1)2(λ21 + λ22) + k22 λ1λ2 + k. (5.11)αY ·X = µYc − βY ·XµXc=µ3 + µ42− k2σxy(µ1 + µ2)(k2 − 1)2(λ21 + λ22) + k22 λ1λ2 + k. (5.12)The numerical results of these derived parameters are summarized in Table5.2.295.3. Missing Data ConditionsModel ρitem ρscale (8, 14 items) β (8, 14 items) α (8, 14 items)1EEH .49 .88, .94 .67, .70 0, 02UEH .25, .64 .86, .92 .65, .69 0, 03EUH .49 .88, .94 .67, .70 .08, .074UUH .25, .64 .86, .92 .65, .69 .09, .085EEM .49 .88, .94 .40, .42 0, 06UEM .25, .64 .86, .92 .37, .39 0, 07EUM .49 .88, .94 .40, .42 .15, .148UUM .25, .64 .86, .92 .37, .39 .16, .15Table 5.2: Derived population model parameters. These important prop-erties of the model condition are fixed given the model and the parametersdefined in Table 1.5.3 Missing Data ConditionsFor all missing data mechanisms, Xu = X k2+1 is the conditioning variable.Let µu be the population mean of that variable, and the strength of theMAR mechanism be ξ. In the strong MAR condition, MAR strength isξ = .8. In the weak MAR condition, ξ = .3. For MCAR, as previouslydiscussed, ξ = pmis where pmis is the missing rate. For the linear MARconditions, ξ is the population missing rate above a certain cutoff z. Thecutoff z is the value corresponding to the pcut = 1 − pmisξ percentile ofthe population distribution of the conditioning variable Xu ∼ N(µu, 1),where pmis is the population missing rate per variable with missingness.For strong linear MAR with pmis = .25, the cutoff is the pcut = 1 − .25.8 =.6875 percentile, with a corresponding z = 0.4888 + µu. For every value inXi ∀i ∈ Z : i ∈ [1, k] where the corresponding Xu is above z, a randomnumber is generated between 0 and 1. If that number is greater than or equalto ξ, the data point is deleted. For pmis = .15, the 1− .15.8 = .8125 percentileis the cutoff. Similarly, the cutoff for weak linear MAR on pmis = .25 and.15, are pcut = .1667 and .5, respectively. For nonlinear MAR, the sameprocedure is applied, except values above 1− (1− pcut2 ) and below 1− pcut2are affected instead. The relevant parameters for these missing mechanismsare summarized in Table 5.3.305.4. AnalysisMechanism pmis ξ pcut zMCAR .25 .25 0 −∞MCAR .15 .15 0 −∞Strong linear MAR .25 .8 .6875 .4888 + µStrong linear MAR .15 .8 8125 −.0281 + µWeak linear MAR .25 .3 .5000 µWeak linear MAR .15 .3 .1667 −.9673 + µStrong nonlinear MAR .25 .8 .3438 −.4021 + µStrong nonlinear MAR .15 .8 .4063 −.2371 + µWeak nonlinear MAR .25 .3 .2500 −.6745 + µWeak nonlinear MAR .15 .3 .0833 −1.3832 + µTable 5.3: Missing data mechanism parameters. For nonlinear MAR, onlythe upper cutoff z is shown. The lower cutoff is simply 1− z.5.4 AnalysisAll analyses were conducted in R (R Core Team, 2016) using the lavaan(v0.5-23.1097) Rosseel (2012) package and custom code adapted from Savaleiand Rhemtulla (2017b). In order to compare ACML, FIML, and TSML ineach condition, bias, ESE, and coverage were computed based on the equa-tions previously given in the simulation background section. Because thecurrent study aims to examine the performance of the new TSML approach,efficiency is given as the ratio of the ESE of each approach over the ESE ofTSML. A ratio greater than 1 implies TSML estimates are more efficient.31Chapter 6Result6.1 ML ConvergenceThe TSML approach has previously encountered issues during Stage 1,where the saturated item-model covariance matrix did not converge underthe EM algorithm in the lavaan package (Savalei & Rhemtulla, 2017b). Here,due to a much smaller sample size, the problem was more pronounced. Over-all in the 8-variable condition, nonconvergent runs only occurred when thesample size was 50 and the per-item missing rate on items with missingnesswas 25%, i.e. 12.5% scale missing rate. The highest rate of nonconvergenceoccurred in the weak linear MAR condition with high regression coefficient,at 20% (see Table 6.1). However, in the 14-item condition with 28 variablesin total, this problem became somewhat dramatic.The EM algorithm usedin this study frequently failed to converge on all 1,000 replications when thesample size was 50 (see Table 6.2). The problem was alleviated by a stronglinear MAR mechanism when the missing rate was low, and to a much lesserdegree, by a strong nonlinear MAR or the MAR mechanisms. At the sam-ple size of 100, the situation significantly improved, with no nonconvergencewhen the per variable missing rate was at 15% (7.5% overall; see Table 6.3).These results suggest that practical applications of TSML may require theuse of more sophisticated implementations of the EM algorithm than theone used in the study. The small sample size of 50 also lead to some no-ticeable outliers, but they had no impact on the conclusions of the studyand were kept in the tables. Whenever the nonconvergence issue results inunavailable data points, “NA” will be used to mark that the results are “notavailable.”326.1. ML Convergencemiss N loadings means b ≈ .7 b ≈ .4MCAR 0.25 50 equal equal 81 81unequal 66 81unequal equal 95 82unequal 62 57S.MAR.Ln equal equal 2 3unequal 2 2unequal equal 1 0unequal 0 1S.MAR.Nln equal equal 10 13unequal 20 14unequal equal 1 4unequal 5 5W.MAR.Ln equal equal 62 70unequal 200 56unequal equal 53 54unequal 46 52W.MAR.Nln equal equal 57 60unequal 73 54unequal equal 86 77unequal 73 63Table 6.1: The number of runs where the EM algorithm did not convergence(out of 1,000 replications) during Stage 1 of TSML for the 8-item scenarioswhen N = 50.336.1. ML Convergencemiss N loadings means b ≈ .7 b ≈ .4MCAR 0.15 50 equal equal 930 911unequal 917 912unequal equal 940 923unequal 943 7090.25 equal equal 1000 1000unequal 1000 1000unequal equal 1000 1000unequal 1000 1000S.MAR.Ln 0.15 equal equal 347 145unequal 310 143unequal equal 174 338unequal 147 9170.25 equal equal 981 918unequal 983 903unequal equal 925 976unequal 914 1000S.MAR.Nln 0.15 equal equal 625 651unequal 638 852unequal equal 405 367unequal 414 3330.25 equal equal 1000 998unequal 1000 1000unequal equal 999 995unequal 996 981W.MAR.Ln 0.15 equal equal 789 761unequal 770 782unequal equal 753 688unequal 737 6370.25 equal equal 1000 1000unequal 1000 1000unequal equal 1000 1000unequal 1000 1000W.MAR.Nln 0.15 equal equal 870 872unequal 888 888unequal equal 868 839unequal 865 3580.25 equal equal 1000 1000unequal 1000 1000unequal equal 1000 1000unequal 1000 998Table 6.2: The number of runs where the EM algorithm did not convergence(out of 1,000 replications) during Stage 1 of TSML for the 14-item scenarioswhen N = 50.346.1. ML Convergencemiss N loadings means b ≈ .7 b ≈ .4MCAR 0.15 100 equal equal 0 0unequal 0 0unequal equal 0 0unequal 0 00.25 equal equal 107 92unequal 90 108unequal equal 298 106unequal 294 69S.MAR.Ln 0.15 equal equal 0 0unequal 0 0unequal equal 0 0unequal 0 00.25 equal equal 0 0unequal 0 0unequal equal 0 0unequal 0 104S.MAR.Nln 0.15 equal equal 0 0unequal 0 0unequal equal 0 0unequal 0 00.25 equal equal 16 18unequal 18 76unequal equal 17 1unequal 8 0W.MAR.Ln 0.15 equal equal 0 0unequal 0 0unequal equal 0 0unequal 0 00.25 equal equal 79 77unequal 80 82unequal equal 217 63unequal 205 23W.MAR.Nln 0.15 equal equal 0 0unequal 0 0unequal equal 0 0unequal 0 00.25 equal equal 112 102unequal 96 107unequal equal 282 62unequal 249 0Table 6.3: The number of runs where the EM algorithm did not convergence(out of 1,000 replications) during Stage 1 of TSML for the 14-item scenarioswhen N = 100.356.2. Bias6.2 BiasFor the regression coefficient estimate, TSML showed negligible bias acrossall 8-item conditions. In 14-item conditions, TSML showed virtually no biasexcept when the sample size was 50. Based on its excellent performance inthe 8-item conditions even at N = 50, it is possible that TSML would haveperformed well had it been able to obtain converged covariance matrices. Infact, in the strong linear MAR conditions with 15% per item missing rate,where the convergence rate was high, TSML also performed reasonably well.ACML underestimated the regression coefficient in all conditions, but thebias tended to be small when the loadings were equal. Unlike Mazza et al.(2015), this study showed a much smaller impact of unequal mean comparedto unequal loadings. This is likely due to the fact the strength of manipula-tion differed between the two studies. When both the loadings and meanswere unequal, ACML showed notable bias across all strong MAR condi-tions, for both 8 items and 14 items. The bias was less noticeable, however,when the MAR mechanism was weak. When the coefficient regression wasmedium (β ≈ .4) instead of high (β ≈ .7), the bias was predictably smaller.Nevertheless, the bias frequently exceeded 10% under strong MAR whenloadings and means were unequal. SL-FIML generally showed positive biasin all conditions, with the exception of MCAR or weak linear MAR, whereSL-FIML showed little bias. See Tables 6.4-6.11. Relative biases higher than10% are highlighted in bold fonts in the tables.366.2. Biasmiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 -.009 .014 -.001 -.004 .012 -.002100 -.006 .006 .003 -.002 .004 .002200 -.008 .002 .001 -.002 .001 .0010.25 50 -.015 .026 .004 .001 .018 .004100 -.008 .016 .001 -.007 .008 .002200 -.009 .012 .002 -.005 .005 .001S.MAR.Ln 0.15 50 -.015 .097 -.005 -.004 .032 .003100 -.016 .095 .001 -.016 .019 .001200 -.018 .089 .002 -.008 .029 .0000.25 50 -.013 .138 .001 -.012 .044 -.008100 -.018 .121 .003 -.008 .031 .004200 -.019 .122 .000 -.013 .020 -.002S.MAR.Nln 0.15 50 -.017 .158 .001 -.018 .029 -.004100 -.017 .162 -.001 -.009 .043 -.002200 -.019 .153 -.002 -.013 .032 -.0030.25 50 -.026 .200 .000 -.019 .061 -.001100 -.025 .206 -.002 -.013 .064 .007200 -.027 .201 -.001 -.016 .039 .000W.MAR.Ln 0.15 50 -.012 .021 -.002 -.011 .006 -.009100 -.007 .023 .000 -.004 .011 .003200 -.007 .021 .000 -.007 .007 .0000.25 50 -.005 .010 .003 .001 .017 .004100 -.011 -.014 .000 -.004 .001 .003200 -.010 -.024 .002 -.006 -.002 -.002W.MAR.Nln 0.15 50 -.012 .074 .007 -.008 .017 -.002100 -.008 .068 -.005 -.004 .017 .000200 -.013 .062 -.002 -.006 .020 -.0020.25 50 -.014 .047 .001 -.001 .035 .008100 -.012 .044 -.001 -.007 .012 .001200 -.015 .034 -.001 -.005 .011 .002Table 6.4: Raw bias under equal loadings and equal means in the 8-itemconditions.376.2. Biasmiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 -.010 .014 .009 -.003 .011 .000100 -.009 .005 .001 -.001 .009 .005200 -.008 .002 .001 -.007 .000 -.0030.25 50 -.012 .031 -.001 -.011 .015 -.008100 -.009 .021 .000 -.009 .007 .000200 -.012 .009 .000 -.008 .007 -.003S.MAR.Ln 0.15 50 -.043 .094 .005 -.018 .032 .003100 -.040 .094 -.002 -.020 .036 .002200 -.044 .090 -.003 -.023 .028 .0010.25 50 -.054 .127 .000 -.035 .022 -.001100 -.059 .119 .003 -.031 .034 .003200 -.058 .114 .001 -.035 .033 -.003S.MAR.Nln 0.15 50 -.020 .149 .005 -.009 .048 .000100 -.023 .153 -.004 -.021 .023 -.008200 -.021 .152 .001 -.013 .036 -.0010.25 50 -.030 .188 .002 -.018 .031 -.001100 -.027 .198 .000 -.020 .040 -.002200 -.032 .202 .001 -.018 .048 .000W.MAR.Ln 0.15 50 -.018 .015 .000 -.010 .017 .003100 -.020 .004 -.005 -.018 .000 -.002200 -.023 -.005 .001 -.013 .005 -.0020.25 50 -.019 .010 .005 -.008 .007 .000100 -.022 -.012 .000 -.016 -.006 -.003200 -.021 -.024 -.001 -.011 -.003 -.001W.MAR.Nln 0.15 50 -.011 .070 .006 -.009 .026 -.004100 -.011 .070 .001 -.010 .020 -.002200 -.008 .068 .001 -.005 .028 .0010.25 50 -.016 .057 .004 -.011 .003 -.004100 -.018 .038 .001 -.008 .012 -.002200 -.014 .037 -.002 -.008 .007 .001Table 6.5: Raw bias under equal loadings and unequal means in the 8-itemconditions.386.2. Biasmiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 -.028 .011 .005 -.021 .006 -.006100 -.027 .005 .002 -.017 .000 -.004200 -.025 .004 .003 -.014 .000 .0020.25 50 -.044 .033 .003 -.030 -.013 .001100 -.045 .018 .000 -.026 .016 .000200 -.049 .007 -.001 -.025 .008 .000S.MAR.Ln 0.15 50 -.052 .135 .000 -.031 .053 .001100 -.057 .118 -.005 -.034 .049 -.002200 -.058 .124 .000 -.032 .048 -.0010.25 50 -.066 .193 .001 -.036 .076 -.001100 -.072 .171 .001 -.040 .056 .002200 -.075 .163 .001 -.042 .070 .001S.MAR.Nln 0.15 50 -.064 .237 -.001 -.037 .086 .004100 -.064 .236 -.002 -.035 .097 .004200 -.071 .226 -.002 -.037 .093 -.0010.25 50 -.091 .347 .001 -.049 .161 .002100 -.089 .351 .001 -.049 .142 .005200 -.088 .360 .001 -.049 .139 .001W.MAR.Ln 0.15 50 -.029 .042 -.006 -.020 .026 .001100 -.031 .036 -.005 -.018 .006 -.003200 -.027 .033 .001 -.015 .015 -.0010.25 50 -.043 -.018 .004 -.019 .005 .006100 -.045 -.023 .000 -.027 -.009 -.004200 -.045 -.032 .001 -.023 -.010 .000W.MAR.Nln 0.15 50 -.034 .116 .001 -.021 .041 .002100 -.042 .099 -.003 -.022 .042 .000200 -.041 .099 .000 -.019 .045 .0010.25 50 -.052 .070 -.003 -.022 .033 .001100 -.054 .055 -.001 -.032 .020 -.003200 -.051 .051 -.003 -.031 .021 .000Table 6.6: Raw bias under unequal loadings and equal means in the 8-itemconditions.396.2. Biasmiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 -.022 .019 -.002 -.020 .000 .000100 -.031 .004 .001 -.017 .001 -.002200 -.026 .005 .002 -.017 .002 .0010.25 50 -.051 .027 -.003 -.021 .007 .001100 -.047 .018 .005 -.027 .005 -.004200 -.050 .008 .001 -.031 .005 -.002S.MAR.Ln 0.15 50 -.082 .128 .000 -.049 .039 .002100 -.088 .125 .001 -.050 .051 .002200 -.087 .128 -.001 -.049 .047 .0010.25 50 -.112 .174 -.005 -.068 .058 -.004100 -.113 .174 .002 -.064 .072 -.006200 -.115 .171 .000 -.062 .078 .000S.MAR.Nln 0.15 50 -.064 .241 .002 -.038 .090 -.002100 -.071 .235 .000 -.038 .090 .000200 -.067 .241 -.001 -.037 .094 .0010.25 50 -.088 .364 -.002 -.048 .139 .004100 -.092 .362 -.002 -.051 .139 .001200 -.093 .354 -.001 -.054 .136 -.001W.MAR.Ln 0.15 50 -.051 .029 -.003 -.026 .014 -.003100 -.046 .033 .000 -.028 .014 -.001200 -.047 .028 -.002 -.026 .016 -.0010.25 50 -.052 -.009 .001 -.037 -.007 -.007100 -.057 -.029 .002 -.030 -.004 -.002200 -.056 -.032 .002 -.032 -.011 -.002W.MAR.Nln 0.15 50 -.039 .113 .004 -.019 .049 .000100 -.043 .101 -.001 -.024 .042 .002200 -.040 .099 .004 -.025 .034 -.0020.25 50 -.060 .058 -.007 -.026 .028 .000100 -.057 .057 -.001 -.030 .030 .004200 -.053 .053 .003 -.028 .019 .003Table 6.7: Raw bias under unequal loadings and unequal means in the8-item conditions.406.2. Biasmiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 -.004 .029 -.030 .003 .018 -.027100 -.004 .015 -.002 -.007 .002 -.004200 -.003 .008 .000 -.001 .003 .0010.25 50 -.009 .044 NA -.005 -.002 NA100 -.007 .039 -.006 -.008 .016 -.005200 -.007 .026 .003 -.004 .017 -.003S.MAR.Ln 0.15 50 -.010 .124 -.015 -.008 .050 -.028100 -.008 .132 .003 .000 .048 .006200 -.008 .128 .002 -.007 .032 -.0020.25 50 -.012 .174 -.081 -.012 .062 -.047100 -.010 .164 .001 -.008 .045 .003200 -.011 .171 -.001 -.011 .043 -.002S.MAR.Nln 0.15 50 -.013 .234 -.070 -.002 .070 -.057100 -.012 .236 .001 -.005 .069 .000200 -.010 .244 .002 -.006 .056 .0000.25 50 -.011 .263 NA -.008 .049 NA100 -.015 .329 -.002 -.014 .040 -.005200 -.016 .311 -.002 -.009 .052 -.004W.MAR.Ln 0.15 50 -.004 .066 -.038 .003 .036 -.038100 -.006 .046 .001 -.002 .015 .000200 -.003 .048 -.001 -.002 .013 -.0010.25 50 -.010 .043 NA -.011 .011 NA100 -.006 .005 -.002 .000 .016 -.009200 -.007 -.025 -.001 -.003 -.006 .000W.MAR.Nln 0.15 50 -.009 .123 -.042 -.001 .028 -.049100 -.006 .123 .006 -.002 .039 .004200 -.007 .117 .000 -.005 .032 .0010.25 50 -.009 .101 NA -.007 .022 NA100 -.007 .096 -.005 -.002 .037 .003200 -.012 .066 .001 -.004 .024 .000Table 6.8: Raw bias under equal loadings and equal means in the 14-itemconditions.416.2. Biasmiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 -.001 .031 -.044 .002 .021 -.050100 -.006 .013 .002 .000 .013 .004200 -.004 .007 .000 -.001 .005 .0010.25 50 -.002 .066 NA -.005 -1.461 NA100 -.009 .045 -.004 -.004 .009 -.002200 -.010 .021 .000 -.006 .011 -.002S.MAR.Ln 0.15 50 -.037 .136 -.019 -.018 .040 -.021100 -.034 .135 -.002 -.023 .027 .001200 -.039 .124 .000 -.022 .037 -.0010.25 50 -.053 .161 -.011 -.033 .013 -.064100 -.048 .177 .000 -.031 .030 -.004200 -.048 .166 .000 -.030 .049 .001S.MAR.Nln 0.15 50 -.011 .221 -.062 -.002 .057 -.044100 -.008 .249 .002 -.006 .054 .001200 -.013 .239 .002 -.005 .055 .0030.25 50 -.016 .251 NA -.016 .027 -.101100 -.016 .296 -.005 -.008 .060 -.002200 -.018 .328 .000 -.011 .045 .002W.MAR.Ln 0.15 50 -.018 .069 -.033 -.005 .040 -.019100 -.018 .053 .002 -.020 .010 -.008200 -.017 .046 .001 -.012 .014 .0000.25 50 -.019 .045 NA -.007 .001 NA100 -.019 -.006 -.003 -.009 -.008 -.003200 -.015 -.018 -.001 -.009 .002 .000W.MAR.Nln 0.15 50 -.006 .128 -.058 -.007 .033 -.068100 -.007 .124 .003 -.007 .040 -.001200 -.008 .120 .002 -.004 .033 -.0020.25 50 -.013 .111 NA -.004 .023 NA100 -.006 .101 -.001 -.002 .034 -.001200 -.010 .063 .004 -.006 .028 .001Table 6.9: Raw bias under equal loadings and unequal means in the 14-itemconditions.426.2. Biasmiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 -.031 .027 -.023 -.020 .016 -.019100 -.024 .018 -.001 -.019 .006 -.008200 -.029 .004 -.001 -.015 -.002 .0000.25 50 -.045 -.011 NA -.032 -.071 NA100 -.051 .039 -.008 -.024 .019 .000200 -.047 .022 .003 -.028 .004 -.001S.MAR.Ln 0.15 50 -.049 .203 -.009 -.030 .087 -.010100 -.052 .199 .000 -.032 .075 -.005200 -.052 .193 -.001 -.033 .082 -.0020.25 50 -.070 .267 -.045 -.042 .095 -.055100 -.074 .265 .001 -.037 .117 .004200 -.072 .258 .000 -.043 .114 .001S.MAR.Nln 0.15 50 -.059 .428 -.038 -.033 .191 -.030100 -.065 .420 -.004 -.027 .172 .006200 -.066 .420 -.001 -.037 .178 .0020.25 50 -.083 .662 -.041 -.049 .299 -.144100 -.089 .669 -.005 -.047 .286 .002200 -.087 .667 .000 -.047 .290 .004W.MAR.Ln 0.15 50 -.030 .114 -.007 -.010 .069 -.008100 -.026 .093 -.002 -.018 .043 -.006200 -.027 .086 .001 -.016 .041 .0010.25 50 -.043 .016 NA -.028 -.006 NA100 -.045 -.021 -.008 -.018 .004 .004200 -.042 -.035 -.005 -.022 -.011 .002W.MAR.Nln 0.15 50 -.030 .247 -.072 -.020 .115 -.046100 -.038 .228 -.001 -.025 .098 -.001200 -.039 .219 -.001 -.023 .086 .0000.25 50 -.052 .159 NA -.031 .106 NA100 -.053 .125 -.016 -.031 .059 -.006200 -.050 .104 -.001 -.033 .038 -.001Table 6.10: Raw bias under unequal loadings and equal means in the 14-item conditions.436.2. Biasmiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 -.029 .031 -.040 -.017 .001 -.025100 -.029 .010 .000 -.019 .008 -.004200 -.026 .010 .001 -.012 .009 .0040.25 50 -.047 -.070 NA -.033 -.013 NA100 -.047 .043 -.009 -.020 .020 .001200 -.047 .023 .000 -.031 .005 -.003S.MAR.Ln 0.15 50 -.088 .193 -.014 -.050 .079 -.004100 -.090 .190 .003 -.048 .085 .000200 -.088 .192 .001 -.051 .069 .0000.25 50 -.123 .269 -.031 -.061 .132 -.038100 -.118 .265 -.003 -.066 .110 .000200 -.117 .261 .002 -.063 .110 .000S.MAR.Nln 0.15 50 -.054 .430 -.035 -.034 .174 -.033100 -.065 .416 -.002 -.035 .185 .003200 -.064 .421 .001 -.039 .171 -.0030.25 50 -.079 .672 -.140 -.054 .300 -.084100 -.087 .677 .003 -.046 .290 .000200 -.093 .661 -.001 -.049 .298 .001W.MAR.Ln 0.15 50 -.051 .092 -.042 -.025 .042 -.014100 -.050 .090 .002 -.027 .039 -.001200 -.047 .084 .001 -.027 .037 -.0010.25 50 -.056 .028 NA -.036 -.010 NA100 -.054 -.022 -.008 -.023 .001 .010200 -.054 -.036 -.002 -.030 -.009 .000W.MAR.Nln 0.15 50 -.040 .242 -.046 -.026 .104 -.047100 -.039 .220 .002 -.024 .103 .001200 -.040 .212 .001 -.022 .092 .0000.25 50 -.051 .220 NA -.035 .058 NA100 -.052 .150 -.011 -.031 .058 -.007200 -.053 .113 .002 -.029 .046 .003Table 6.11: Raw bias under unequal loadings and unequal means in the14-item conditions.446.3. Efficiency6.3 EfficiencyAs expected, the SL-FIML estimate was always inefficient, showing as highas 5 times the ESE compared to TSML. ACML tended to be slightly moreefficient in all conditions, but seldom drastically more so: the lowest ESEratio was .868 in the 8-item conditions. For 14-item conditions, it is onceagain difficult to determine the ESE ratio due to nonconvergence in TSML.After excluding the 14-item conditions where N = 50, ACML obtained alowest ESE ratio of .819. Interestingly, in many cases, such as when theloadings were equal but the means were unequal, the high efficiency wouldwork against ACML: Even with a relatively small bias that was less than10%, ACML’s unwarranted high efficiency could lead to as low as 78.3%coverage. When an approach produces biased estimates, high efficiency canbecome a drawback rather than an advantage. Tables 6.12-6.19 contain allESE ratio results. Relative efficiencies lower than 90% are highlighted inbold fonts in the tables.456.3. Efficiencymiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 .917 1.182 1.000 .992 1.414 1.000100 .984 1.243 1.000 .970 1.327 1.000200 .957 1.186 1.000 .965 1.266 1.0000.25 50 .960 1.676 1.000 .955 1.918 1.000100 .998 1.515 1.000 .971 1.728 1.000200 1.006 1.512 1.000 .997 1.783 1.000S.MAR.Ln 0.15 50 .985 1.622 1.000 .979 1.725 1.000100 .956 1.525 1.000 .917 1.574 1.000200 .992 1.517 1.000 .993 1.607 1.0000.25 50 .982 2.129 1.000 .979 2.299 1.000100 .940 1.862 1.000 .990 2.166 1.000200 .997 1.926 1.000 .963 2.157 1.000S.MAR.Nln 0.15 50 .959 1.977 1.000 .970 2.201 1.000100 1.001 1.975 1.000 1.004 2.132 1.000200 .970 1.909 1.000 .960 2.104 1.0000.25 50 .926 3.436 1.000 .904 3.151 1.000100 1.004 3.193 1.000 .943 3.032 1.000200 .953 2.982 1.000 .965 3.055 1.000W.MAR.Ln 0.15 50 .990 1.291 1.000 .979 1.398 1.000100 .965 1.219 1.000 .982 1.406 1.000200 .983 1.286 1.000 .972 1.356 1.0000.25 50 .978 1.434 1.000 1.014 1.659 1.000100 .966 1.294 1.000 .975 1.448 1.000200 .930 1.255 1.000 1.039 1.471 1.000W.MAR.Nln 0.15 50 .993 1.649 1.000 1.027 1.794 1.000100 1.026 1.522 1.000 .982 1.668 1.000200 .973 1.416 1.000 .961 1.620 1.0000.25 50 1.027 1.893 1.000 .949 2.098 1.000100 .993 1.663 1.000 .954 1.860 1.000200 .980 1.646 1.000 .975 1.917 1.000Table 6.12: ESE ratio under equal loadings and equal means in the 8-itemconditions.466.3. Efficiencymiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 .951 1.224 1.000 1.002 1.400 1.000100 .982 1.202 1.000 1.011 1.370 1.000200 .988 1.219 1.000 1.005 1.305 1.0000.25 50 .986 1.639 1.000 .984 2.015 1.000100 .998 1.542 1.000 .983 1.775 1.000200 .976 1.366 1.000 .996 1.699 1.000S.MAR.Ln 0.15 50 .896 1.514 1.000 .941 1.737 1.000100 .988 1.577 1.000 .954 1.631 1.000200 .969 1.506 1.000 .951 1.754 1.0000.25 50 .942 2.164 1.000 .919 2.230 1.000100 .868 1.810 1.000 .902 2.041 1.000200 .906 1.804 1.000 .931 2.003 1.000S.MAR.Nln 0.15 50 .947 1.437 1.000 .954 1.470 1.000100 .979 1.470 1.000 1.001 1.572 1.000200 1.022 1.515 1.000 .968 1.526 1.0000.25 50 1.000 1.883 1.000 .973 2.023 1.000100 .948 1.719 1.000 .933 1.880 1.000200 .931 1.686 1.000 .968 1.766 1.000W.MAR.Ln 0.15 50 .956 1.306 1.000 .992 1.463 1.000100 .953 1.273 1.000 .945 1.340 1.000200 .969 1.319 1.000 1.010 1.373 1.0000.25 50 .947 1.718 1.000 .979 1.674 1.000100 .926 1.327 1.000 .985 1.522 1.000200 1.067 1.497 1.000 .995 1.506 1.000W.MAR.Nln 0.15 50 .970 1.557 1.000 .983 1.412 1.000100 .993 1.519 1.000 .971 1.480 1.000200 .978 1.438 1.000 .959 1.360 1.0000.25 50 1.003 1.879 1.000 1.002 1.725 1.000100 .993 1.707 1.000 .980 1.638 1.000200 1.008 1.613 1.000 .972 1.502 1.000Table 6.13: ESE ratio under equal loadings and unequal means in the8-item conditions.476.3. Efficiencymiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 .952 1.221 1.000 .962 1.340 1.000100 .963 1.221 1.000 .983 1.326 1.000200 .935 1.150 1.000 .967 1.271 1.0000.25 50 .923 1.701 1.000 .918 2.014 1.000100 .950 1.494 1.000 .942 1.783 1.000200 .943 1.452 1.000 .989 1.772 1.000S.MAR.Ln 0.15 50 .997 1.567 1.000 .939 1.577 1.000100 1.016 1.551 1.000 .955 1.601 1.000200 .957 1.460 1.000 .970 1.604 1.0000.25 50 .945 1.846 1.000 .925 2.266 1.000100 .974 1.785 1.000 .908 2.095 1.000200 .959 1.794 1.000 .897 1.934 1.000S.MAR.Nln 0.15 50 .883 1.731 1.000 .863 2.064 1.000100 .961 1.813 1.000 .897 1.935 1.000200 .999 1.882 1.000 .937 2.137 1.0000.25 50 .890 2.699 1.000 .873 2.987 1.000100 .937 2.675 1.000 .921 2.990 1.000200 .923 2.535 1.000 .891 2.869 1.000W.MAR.Ln 0.15 50 .943 1.324 1.000 .985 1.387 1.000100 1.006 1.323 1.000 .966 1.431 1.000200 .953 1.254 1.000 .968 1.383 1.0000.25 50 .922 1.377 1.000 .974 1.651 1.000100 .948 1.321 1.000 .946 1.466 1.000200 .930 1.235 1.000 .970 1.458 1.000W.MAR.Nln 0.15 50 .958 1.574 1.000 .960 1.806 1.000100 .956 1.507 1.000 .946 1.698 1.000200 .969 1.468 1.000 .978 1.661 1.0000.25 50 .956 2.010 1.000 .963 2.175 1.000100 .945 1.683 1.000 .941 1.989 1.000200 .944 1.632 1.000 .942 1.806 1.000Table 6.14: ESE ratio under unequal loadings and equal means in the8-item conditions.486.3. Efficiencymiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 .965 1.250 1.000 .976 1.439 1.000100 .965 1.184 1.000 .974 1.328 1.000200 .975 1.144 1.000 .930 1.331 1.0000.25 50 .904 1.683 1.000 .937 1.997 1.000100 .920 1.373 1.000 .955 1.701 1.000200 .979 1.470 1.000 .916 1.639 1.000S.MAR.Ln 0.15 50 .934 1.487 1.000 .893 1.649 1.000100 .912 1.411 1.000 .877 1.574 1.000200 .936 1.514 1.000 .860 1.558 1.0000.25 50 .882 1.881 1.000 .866 2.093 1.000100 .916 1.742 1.000 .881 2.140 1.000200 .885 1.763 1.000 .911 2.019 1.000S.MAR.Nln 0.15 50 .996 1.455 1.000 .963 1.562 1.000100 .970 1.405 1.000 .885 1.443 1.000200 .978 1.401 1.000 .935 1.469 1.0000.25 50 .943 1.791 1.000 .918 1.823 1.000100 .988 1.672 1.000 .913 1.774 1.000200 .922 1.559 1.000 .934 1.808 1.000W.MAR.Ln 0.15 50 .905 1.254 1.000 .928 1.450 1.000100 .985 1.336 1.000 .935 1.348 1.000200 .955 1.246 1.000 .939 1.342 1.0000.25 50 .909 1.325 1.000 .885 1.501 1.000100 .898 1.219 1.000 .924 1.530 1.000200 .972 1.243 1.000 .929 1.387 1.000W.MAR.Nln 0.15 50 .948 1.351 1.000 .977 1.468 1.000100 .981 1.359 1.000 .932 1.325 1.000200 .978 1.285 1.000 .926 1.409 1.0000.25 50 .982 1.529 1.000 .931 1.689 1.000100 .948 1.509 1.000 .939 1.570 1.000200 1.022 1.539 1.000 .921 1.549 1.000Table 6.15: ESE ratio under unequal loadings and unequal means in the8-item conditions.496.3. Efficiencymiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 .780 1.474 1.000 .893 1.818 1.000100 .975 1.516 1.000 1.037 1.741 1.000200 .972 1.467 1.000 1.056 1.722 1.0000.25 50 NA NA NA NA NA NA100 .930 2.835 1.000 .926 3.169 1.000200 1.010 2.349 1.000 1.013 2.622 1.000S.MAR.Ln 0.15 50 .817 1.564 1.000 .893 1.881 1.000100 .963 1.711 1.000 .979 2.065 1.000200 .956 1.692 1.000 1.021 2.009 1.0000.25 50 .593 1.688 1.000 .853 2.633 1.000100 .962 2.466 1.000 .966 2.814 1.000200 .904 2.268 1.000 .978 2.733 1.000S.MAR.Nln 0.15 50 .825 2.559 1.000 .950 3.095 1.000100 .982 2.807 1.000 1.027 3.097 1.000200 .952 2.639 1.000 .960 2.853 1.0000.25 50 NA NA NA NA NA NA100 .945 5.763 1.000 .952 5.715 1.000200 .927 5.121 1.000 .959 5.356 1.000W.MAR.Ln 0.15 50 .746 1.250 1.000 .919 1.786 1.000100 1.003 1.544 1.000 .990 1.810 1.000200 .975 1.489 1.000 1.025 1.775 1.0000.25 50 NA NA NA NA NA NA100 1.000 2.000 1.000 1.032 2.490 1.000200 1.031 1.927 1.000 .991 2.034 1.000W.MAR.Nln 0.15 50 .744 1.994 1.000 .855 2.371 1.000100 .935 2.128 1.000 .982 2.391 1.000200 .966 2.104 1.000 .950 2.215 1.0000.25 50 NA NA NA NA NA NA100 .954 3.488 1.000 .995 3.738 1.000200 .924 2.476 1.000 1.018 3.255 1.000Table 6.16: ESE ratio under equal loadings and equal means in the 14-itemconditions.506.3. Efficiencymiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 .701 1.193 1.000 .921 1.772 1.000100 .979 1.431 1.000 .961 1.658 1.000200 .978 1.419 1.000 1.002 1.617 1.0000.25 50 NA NA NA NA NA NA100 .925 2.589 1.000 .920 3.081 1.000200 .948 2.258 1.000 .996 2.925 1.000S.MAR.Ln 0.15 50 .810 1.488 1.000 .885 1.777 1.000100 .944 1.734 1.000 .926 1.934 1.000200 .925 1.765 1.000 .874 1.874 1.0000.25 50 .687 2.048 1.000 .708 2.353 1.000100 .905 2.344 1.000 .932 2.835 1.000200 .917 2.374 1.000 .896 2.560 1.000S.MAR.Nln 0.15 50 .783 2.614 1.000 .913 2.929 1.000100 .985 2.762 1.000 .970 2.841 1.000200 1.000 2.781 1.000 .998 3.076 1.0000.25 50 NA NA NA 3.773 27.453 1.000100 .899 5.578 1.000 1.038 6.015 1.000200 1.003 5.323 1.000 .979 5.816 1.000W.MAR.Ln 0.15 50 .775 1.297 1.000 .875 1.781 1.000100 .974 1.564 1.000 .932 1.744 1.000200 .932 1.512 1.000 1.032 1.837 1.0000.25 50 NA NA NA NA NA NA100 .891 1.968 1.000 .948 2.394 1.000200 1.000 1.849 1.000 .989 1.990 1.000W.MAR.Nln 0.15 50 .869 2.053 1.000 .844 2.364 1.000100 .968 2.125 1.000 .978 2.495 1.000200 .960 1.877 1.000 1.001 2.293 1.0000.25 50 NA NA NA NA NA NA100 .916 3.411 1.000 .993 3.899 1.000200 .968 2.606 1.000 1.006 3.424 1.000Table 6.17: ESE ratio under equal loadings and unequal means in the14-item conditions.516.3. Efficiencymiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 .761 1.432 1.000 .829 1.672 1.000100 .939 1.469 1.000 .974 1.769 1.000200 .988 1.428 1.000 .993 1.743 1.0000.25 50 NA NA NA NA NA NA100 .868 2.554 1.000 .959 3.348 1.000200 .964 2.325 1.000 .976 2.785 1.000S.MAR.Ln 0.15 50 .946 1.621 1.000 .960 2.018 1.000100 .951 1.588 1.000 .979 1.915 1.000200 1.012 1.672 1.000 .922 1.703 1.0000.25 50 .719 1.689 1.000 .824 2.438 1.000100 1.025 2.242 1.000 .920 2.491 1.000200 .987 2.082 1.000 .908 2.303 1.000S.MAR.Nln 0.15 50 .909 2.180 1.000 .946 2.727 1.000100 .971 2.176 1.000 .949 2.858 1.000200 .978 2.278 1.000 .960 2.602 1.0000.25 50 Inf Inf NA 23.000 136.903 1.000100 .971 3.767 1.000 .895 4.650 1.000200 1.024 3.660 1.000 .905 4.264 1.000W.MAR.Ln 0.15 50 .884 1.570 1.000 .858 1.694 1.000100 1.038 1.688 1.000 1.004 1.814 1.000200 .974 1.546 1.000 .916 1.700 1.0000.25 50 NA NA NA NA NA NA100 .819 1.634 1.000 .977 2.416 1.000200 1.057 1.714 1.000 .967 1.991 1.000W.MAR.Nln 0.15 50 .850 2.175 1.000 1.027 3.103 1.000100 .972 2.173 1.000 .941 2.414 1.000200 1.001 2.129 1.000 .912 2.239 1.0000.25 50 NA NA NA NA NA NA100 .847 2.870 1.000 .941 3.926 1.000200 .989 2.767 1.000 .931 3.478 1.000Table 6.18: ESE ratio under unequal loadings and equal means in the14-item conditions.526.3. Efficiencymiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 .768 1.290 1.000 .907 1.854 1.000100 1.029 1.568 1.000 .966 1.759 1.000200 1.024 1.489 1.000 1.005 1.716 1.0000.25 50 NA NA NA NA NA NA100 .817 2.385 1.000 .933 3.405 1.000200 1.002 2.301 1.000 .965 2.814 1.000S.MAR.Ln 0.15 50 .859 1.437 1.000 .889 1.921 1.000100 .913 1.566 1.000 .951 1.882 1.000200 .959 1.563 1.000 .960 1.818 1.0000.25 50 .694 1.786 1.000 .806 2.426 1.000100 .948 2.155 1.000 .885 2.557 1.000200 .953 2.091 1.000 .885 2.475 1.000S.MAR.Nln 0.15 50 .956 2.205 1.000 .902 2.747 1.000100 1.001 2.288 1.000 .944 2.613 1.000200 .985 2.158 1.000 .953 2.675 1.0000.25 50 .895 4.170 1.000 1.337 7.818 1.000100 .999 3.847 1.000 .917 4.495 1.000200 .984 3.465 1.000 .921 4.293 1.000W.MAR.Ln 0.15 50 .795 1.438 1.000 .881 1.785 1.000100 .944 1.607 1.000 .925 1.805 1.000200 .928 1.603 1.000 .929 1.696 1.0000.25 50 NA NA NA NA NA NA100 .906 1.800 1.000 .880 2.246 1.000200 .964 1.745 1.000 .920 1.955 1.000W.MAR.Nln 0.15 50 .913 2.306 1.000 .977 2.902 1.000100 .971 2.302 1.000 .970 2.407 1.000200 .985 2.192 1.000 .996 2.520 1.0000.25 50 NA NA NA NA NA NA100 .873 3.101 1.000 .947 3.752 1.000200 .978 2.781 1.000 .947 3.240 1.000Table 6.19: ESE ratio under unequal loadings and unequal means in the14-item conditions.536.4. Coverage6.4 CoverageTSML showed close to 95% coverage across all conditions, again with theexception of nonconvergent runs in the 14-item conditions when N = 50.ACML suffered from decreasing coverage as sample size increased, under allunequal loading conditions. As long as the loadings were equal, unequalmeans only became a significant problem when the population regressioncoefficient was high and the MAR mechanism was strong. However, a com-bination of unequal loadings and unequal means had serious consequenceson the ACML coverage, showing poor coverage even under MCAR when thesample size was large. In the 14-item condition, ACML coverage droppedto 30.4% under strong linear MAR at the sample size of 200 with 25% pervariable missing rate. When item loadings and means were both unequal,ACML coverage sometimes underperformed compared to even SL-FIML.When only the loadings were unequal, ACML performed almost just aspoorly, only showing reasonable coverage in MCAR and weak linear MARconditions. It should be noted that the similar coverage rates of SL-FIMLin MCAR and weak linear MAR were produced slightly differently. In weaklinear MAR, SL-FIML generally showed a larger bias, but also lower effi-ciency. As a result, the low efficiency offset the bias in weak linear MAR,producing seemingly acceptable coverage rates. The recurrent differencesbetween strong MCAR and weak MAR in these results should serve as acaution to the potential danger of using an insufficiently strong MAR mech-anism. See Tables 6.20-6.27. Coverages lower than 90% are highlighted inbold fonts in the tables.546.4. Coveragemiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 95.7 93.5 94.5 93.8 90.4 94.2100 94.0 93.4 95.2 94.7 94.4 94.5200 94.1 94.5 95.1 94.8 94.8 95.20.25 50 93.1 89.9 95.6 94.5 88.6 94.3100 94.8 91.7 96.1 93.8 90.9 94.1200 94.4 92.4 95.1 95.7 92.4 94.6S.MAR.Ln 0.15 50 94.4 86.8 95.9 94.9 91.8 95.4100 93.9 84.1 95.2 95.2 93.9 95.7200 93.8 78.3 96.4 94.2 92.5 95.20.25 50 95.3 82.3 96.9 94.3 89.3 95.8100 94.5 80.4 95.5 92.9 92.0 94.4200 93.2 71.5 96.1 94.7 92.3 95.2S.MAR.Nln 0.15 50 93.1 84.2 96.5 94.7 90.2 94.5100 92.6 76.3 95.8 94.0 93.1 95.2200 91.9 65.3 95.1 95.0 94.2 95.90.25 50 93.8 75.6 97.3 95.0 86.5 94.5100 92.3 75.2 96.7 93.8 90.0 94.6200 90.4 68.5 94.6 94.0 91.5 94.4W.MAR.Ln 0.15 50 93.7 92.6 96.7 93.4 93.3 95.0100 94.0 93.1 95.2 95.1 92.1 95.4200 93.9 91.6 95.7 94.2 93.1 94.80.25 50 94.0 91.3 95.5 93.0 89.0 93.7100 94.8 93.3 94.8 94.5 93.3 95.6200 94.1 91.1 93.8 94.4 94.3 95.2W.MAR.Nln 0.15 50 94.3 87.4 96.7 94.2 91.3 95.6100 93.1 87.8 95.6 94.8 92.2 94.0200 94.0 86.9 95.1 96.1 93.6 94.90.25 50 93.2 88.6 96.7 95.5 87.3 94.1100 93.7 89.9 95.1 94.5 91.3 94.9200 94.4 91.3 95.4 94.4 92.2 94.8Table 6.20: Coverage under equal loadings and equal means in the 8-itemconditions.556.4. Coveragemiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 94.2 92.1 95.9 95.3 91.0 94.8100 95.2 94.5 94.9 95.3 92.7 95.5200 94.2 93.3 95.5 94.0 94.2 93.70.25 50 94.0 88.3 95.7 93.4 86.2 94.2100 93.0 91.4 96.0 94.1 90.8 94.5200 95.6 94.6 95.1 94.3 93.0 94.7S.MAR.Ln 0.15 50 91.7 87.8 95.6 94.6 91.0 95.3100 90.6 83.6 95.9 95.0 93.9 95.0200 84.9 79.2 95.6 94.0 90.4 95.50.25 50 89.2 83.4 96.8 93.3 89.8 94.9100 84.9 80.9 94.8 92.1 91.8 93.6200 78.3 74.9 95.1 89.8 91.7 94.5S.MAR.Nln 0.15 50 94.6 89.3 95.4 94.3 92.4 94.3100 92.5 83.9 95.1 93.9 92.3 94.0200 90.2 78.5 96.2 94.5 94.0 94.80.25 50 92.1 84.8 96.3 94.1 90.7 95.3100 90.7 80.4 95.7 93.0 91.7 94.7200 88.0 67.2 95.2 92.7 94.7 95.1W.MAR.Ln 0.15 50 94.8 93.1 97.0 94.5 91.3 95.0100 94.5 92.9 94.9 94.1 94.2 95.1200 93.1 93.5 96.1 94.0 93.5 94.20.25 50 93.7 88.7 96.2 94.0 89.1 95.0100 92.8 93.4 94.7 94.6 92.4 95.1200 91.3 91.6 95.9 94.8 93.5 95.5W.MAR.Nln 0.15 50 93.5 89.7 95.9 94.8 93.5 95.0100 94.4 89.1 95.5 95.1 91.2 95.3200 92.9 88.7 94.9 95.3 94.2 96.00.25 50 93.7 89.6 96.5 93.6 89.9 94.5100 93.5 91.3 95.9 94.3 92.4 94.1200 92.7 90.2 95.2 94.3 94.8 94.2Table 6.21: Coverage under equal loadings and unequal means in the 8-itemconditions.566.4. Coveragemiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 93.6 92.7 95.7 93.7 93.4 94.5100 93.4 94.4 95.3 94.7 94.8 94.7200 93.0 94.8 95.3 94.0 94.3 95.20.25 50 92.7 89.4 94.7 94.0 87.2 94.3100 90.1 91.4 95.2 94.5 90.2 94.0200 86.2 93.5 95.1 92.8 92.8 94.9S.MAR.Ln 0.15 50 91.8 83.0 96.4 94.3 91.7 95.4100 86.2 79.9 96.6 92.1 92.6 95.0200 78.3 64.0 94.7 90.4 90.1 94.30.25 50 89.4 78.8 96.8 93.9 87.7 95.7100 82.4 73.8 96.2 93.1 90.5 95.6200 68.5 58.3 94.7 89.5 88.4 95.1S.MAR.Nln 0.15 50 87.6 72.7 95.3 94.4 90.3 95.7100 84.9 58.6 95.6 92.0 89.9 95.1200 69.6 38.6 95.8 90.6 85.4 94.90.25 50 82.7 64.2 96.8 91.4 83.5 94.4100 73.9 52.5 97.6 89.9 87.4 95.4200 57.8 28.5 95.8 86.3 83.2 93.9W.MAR.Ln 0.15 50 94.7 91.3 96.0 92.8 91.4 95.4100 92.0 91.3 95.9 93.3 92.4 95.1200 93.0 91.4 95.7 94.9 93.6 95.40.25 50 94.1 92.9 96.2 94.3 88.7 94.5100 90.9 91.6 95.5 93.7 92.2 94.7200 87.7 92.6 94.8 93.7 94.2 95.3W.MAR.Nln 0.15 50 93.1 85.7 95.5 93.9 90.9 95.9100 90.6 84.5 95.6 94.0 91.3 94.3200 86.3 75.2 95.0 92.5 91.7 95.00.25 50 91.7 88.8 96.6 94.1 86.8 93.9100 88.4 88.9 94.9 93.5 91.0 95.5200 83.8 88.8 95.3 91.4 92.5 94.2Table 6.22: Coverage under unequal loadings and equal means in the 8-itemconditions.576.4. Coveragemiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 94.9 93.4 95.9 93.8 91.8 96.6100 92.3 94.4 94.8 93.6 94.0 93.6200 92.1 94.6 95.9 95.0 94.1 95.00.25 50 91.6 89.4 95.3 95.9 88.2 95.4100 89.3 92.8 94.2 93.1 92.7 94.4200 83.6 92.2 95.0 92.7 94.7 95.6S.MAR.Ln 0.15 50 85.5 85.9 95.9 92.7 92.2 94.8100 74.2 77.5 95.7 91.8 92.9 95.3200 59.5 62.1 95.8 85.7 91.0 92.80.25 50 78.9 80.4 95.4 88.8 87.8 94.5100 60.7 70.1 95.6 87.1 90.0 95.3200 34.5 54.2 95.5 80.9 87.0 95.7S.MAR.Nln 0.15 50 88.1 86.5 96.9 92.3 91.0 95.9100 85.2 78.8 95.7 93.5 92.9 94.4200 77.0 65.0 96.8 91.3 89.6 95.40.25 50 84.0 78.5 97.0 92.4 91.4 95.9100 77.4 66.8 95.9 89.9 91.4 95.1200 59.9 45.1 95.3 86.7 88.7 95.0W.MAR.Ln 0.15 50 91.2 91.9 95.2 94.0 91.8 94.2100 89.6 91.2 96.2 93.1 92.1 94.2200 85.5 92.3 94.8 92.8 93.9 94.70.25 50 92.5 93.0 95.8 93.8 90.7 94.2100 88.5 92.9 94.3 94.9 92.2 95.6200 80.7 92.5 95.5 92.3 94.0 93.8W.MAR.Nln 0.15 50 91.6 89.4 95.9 93.9 91.8 94.9100 92.1 85.3 96.0 93.9 93.7 93.7200 88.0 77.2 94.8 94.7 92.5 95.20.25 50 91.8 86.7 95.4 94.2 90.2 93.6100 88.2 84.9 95.2 93.2 93.1 94.9200 84.5 79.9 96.5 92.4 93.3 95.2Table 6.23: Coverage under unequal loadings and unequal means in the8-item conditions.586.4. Coveragemiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 95.4 90.8 95.7 93.8 85.7 91.6100 94.2 91.5 94.8 94.7 92.1 96.5200 95.0 92.2 94.4 94.4 94.2 96.80.25 50 93.7 70.8 NA 93.5 69.9 NA100 94.9 81.9 93.6 95.5 83.5 93.8200 93.5 87.4 94.7 93.0 90.3 94.1S.MAR.Ln 0.15 50 95.0 84.3 95.3 94.0 88.7 91.6100 94.3 78.5 95.9 94.0 89.0 94.9200 94.8 66.2 95.0 93.7 91.8 94.50.25 50 93.3 75.9 73.7 94.3 83.8 100.0100 95.2 74.2 95.3 94.7 88.7 94.9200 95.5 61.7 95.0 96.0 90.1 95.4S.MAR.Nln 0.15 50 93.1 75.5 89.3 93.9 86.9 90.6100 94.3 70.1 96.4 93.7 90.9 95.6200 94.5 53.3 94.9 94.2 90.6 93.90.25 50 93.3 62.3 NA 93.7 77.2 NA100 93.3 68.7 95.9 94.0 85.2 94.5200 94.5 63.4 94.9 94.2 88.1 94.2W.MAR.Ln 0.15 50 95.2 90.6 93.8 94.6 88.4 91.6100 94.7 91.2 95.8 95.2 92.8 95.7200 96.0 88.8 95.1 93.8 91.7 94.70.25 50 95.5 77.5 NA 93.9 73.6 NA100 93.5 86.3 95.3 94.1 85.0 95.2200 94.4 88.8 96.3 93.7 92.0 94.9W.MAR.Nln 0.15 50 95.5 81.0 89.2 94.4 87.5 86.7100 95.0 77.8 94.7 93.4 89.3 93.8200 95.6 73.9 96.0 94.8 92.5 95.10.25 50 93.4 69.1 NA 93.5 66.8 NA100 95.2 82.1 95.2 94.4 82.4 95.0200 95.5 85.3 93.8 95.4 87.3 96.2Table 6.24: Coverage under equal loadings and equal means in the 14-itemconditions.596.4. Coveragemiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 94.2 90.5 89.2 95.2 89.4 87.0100 94.8 92.1 95.9 95.6 92.4 94.0200 94.2 93.8 95.2 95.6 94.7 94.50.25 50 95.4 72.8 NA 94.5 71.1 NA100 94.9 82.5 94.7 93.8 83.9 92.5200 94.6 87.9 95.9 96.1 90.0 95.4S.MAR.Ln 0.15 50 92.0 84.1 93.9 93.2 90.0 91.8100 92.4 78.1 96.0 92.6 92.9 93.9200 86.4 66.7 94.8 94.9 91.6 94.00.25 50 92.3 77.6 88.2 93.5 84.8 87.5100 88.8 73.3 95.6 93.3 89.0 94.7200 82.3 61.2 95.3 92.2 90.6 95.3S.MAR.Nln 0.15 50 94.6 75.8 90.3 95.4 90.4 92.8100 93.4 66.4 95.8 92.5 90.0 93.1200 93.2 55.3 95.7 95.3 91.9 95.70.25 50 94.6 66.6 NA 93.1 75.2 100.0100 93.9 69.6 94.4 94.1 85.6 96.0200 92.5 62.0 97.0 94.9 89.4 95.9W.MAR.Ln 0.15 50 93.8 88.2 90.9 94.9 87.4 93.1100 93.2 88.4 96.0 95.7 92.7 94.3200 93.9 87.1 94.7 94.0 92.3 96.40.25 50 94.8 79.3 NA 93.6 75.7 NA100 93.8 86.0 94.6 94.9 87.5 94.8200 92.7 88.1 96.2 94.1 91.3 95.3W.MAR.Nln 0.15 50 94.9 80.9 91.1 93.8 86.7 88.4100 94.1 80.7 94.8 94.7 89.0 95.0200 94.6 73.3 95.0 94.6 92.4 95.40.25 50 93.2 70.3 NA 93.9 66.5 NA100 95.8 78.2 95.5 94.3 80.8 93.3200 93.9 85.5 94.7 95.1 87.7 96.0Table 6.25: Coverage under equal loadings and unequal means in the 14-item conditions.606.4. Coveragemiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 93.9 89.7 88.3 94.4 89.1 89.9100 93.8 91.2 95.4 94.2 90.7 94.8200 90.3 94.0 95.3 93.8 93.3 94.60.25 50 92.8 72.6 NA 92.5 68.9 NA100 88.7 84.3 93.9 94.4 82.0 94.7200 85.1 87.3 95.3 91.9 89.7 94.9S.MAR.Ln 0.15 50 90.7 72.2 95.5 93.6 86.4 94.2100 87.9 59.6 96.1 92.8 89.9 95.3200 80.8 36.6 96.0 91.2 87.2 94.00.25 50 87.8 67.0 86.7 93.4 86.7 85.4100 79.2 53.4 96.4 92.9 85.8 95.5200 69.2 29.2 95.9 89.7 85.0 94.9S.MAR.Nln 0.15 50 88.6 50.2 93.4 92.8 85.5 94.4100 81.6 28.7 94.3 94.3 83.0 95.2200 70.7 6.8 96.2 89.1 79.7 94.90.25 50 84.5 45.2 100.0 92.3 76.6 100.0100 72.3 26.4 95.6 89.7 78.3 94.2200 56.8 7.3 96.4 86.8 76.2 94.4W.MAR.Ln 0.15 50 94.1 84.0 95.1 96.1 86.3 93.6100 92.8 81.8 95.1 94.9 91.4 94.4200 91.6 76.5 94.5 95.2 91.2 94.30.25 50 92.3 79.7 NA 93.2 77.9 NA100 91.1 88.5 93.4 94.1 86.0 96.5200 85.8 89.2 95.3 93.4 92.3 94.0W.MAR.Nln 0.15 50 92.7 69.0 84.1 93.6 83.8 93.2100 90.3 62.0 95.5 94.5 88.1 94.9200 88.0 40.3 95.7 92.9 87.0 93.50.25 50 89.5 68.4 NA 93.0 68.5 NA100 88.5 76.8 92.9 92.3 80.2 94.8200 81.7 78.7 93.8 92.5 88.3 95.3Table 6.26: Coverage under unequal loadings and equal means in the 14-item conditions.616.4. Coveragemiss N ACML FIML TS ACML FIML TSb ≈ .7 b ≈ .4MCAR 0.15 50 93.6 90.5 87.7 94.3 89.3 90.9100 91.5 90.4 96.1 93.7 90.9 95.0200 92.2 93.1 96.2 95.0 93.7 95.80.25 50 91.3 71.5 NA 93.6 66.7 NA100 90.5 83.8 92.5 94.8 84.5 94.3200 84.8 88.9 95.2 92.6 91.2 95.7S.MAR.Ln 0.15 50 83.2 74.6 94.3 92.3 87.7 94.9100 69.8 62.5 94.9 91.2 88.9 95.4200 53.9 37.1 94.9 85.1 88.3 94.80.25 50 73.0 66.8 89.5 92.6 84.7 91.8100 57.2 52.4 95.7 87.3 86.5 95.0200 30.4 29.1 95.9 80.4 84.3 95.1S.MAR.Nln 0.15 50 87.1 50.0 93.9 93.6 85.7 92.4100 81.3 31.1 96.5 92.5 83.3 94.1200 72.1 6.6 95.4 89.4 81.1 94.90.25 50 82.1 41.2 50.0 90.6 76.0 80.0100 73.7 28.5 97.2 88.9 78.1 94.4200 50.4 8.1 94.0 85.2 75.2 92.8W.MAR.Ln 0.15 50 91.3 86.1 91.3 93.5 89.3 91.8100 87.6 82.9 94.9 94.7 92.0 94.8200 84.1 78.7 95.4 91.9 89.8 92.80.25 50 92.1 84.0 NA 93.5 76.4 NA100 87.3 88.8 93.7 94.3 85.6 93.6200 82.2 89.8 95.8 92.9 92.9 95.1W.MAR.Nln 0.15 50 92.5 71.3 89.6 93.2 83.2 93.9100 91.5 62.5 96.4 93.3 88.3 94.2200 87.1 44.4 95.8 92.9 87.0 95.80.25 50 91.9 65.6 NA 93.6 67.0 NA100 88.9 75.0 93.7 92.9 84.7 94.5200 81.4 74.7 95.1 91.7 86.9 93.3Table 6.27: Coverage under unequal loadings and unequal means in the14-item conditions.62Chapter 7Discussion7.1 Recommendations7.1.1 For Analyzing Empirical DataWhen researchers encounter item-level missing data, it is often tempting touse the ACML approach due to its convenience. However, the current studyhas shown that, in the context of bivariate regression, the ACML approachis likely to underestimate the regression coefficient. More critically, becausethe ACML standard error is uncorrected, a larger sample size often leads toworse coverage. The ACML estimate is unbiased under the assumption thatitem loadings and item means are equal, but such an assumption is oftenunrealistic. It can also be observed that when the regression coefficient issmaller, the underestimate is naturally smaller as well. However, in order todetect a smaller effect size, a larger sample size is required, which would inturn result in bad coverage. It is worth noting that, while the current studyshowed the impact of unequal loadings to be larger than that of unequalmeans, Mazza et al. (2015) showed the opposite. This is likely attributableto the difference in the degree of manipulation on each dimension betweenthe two studies. These findings suggest that a smaller within-scale itemmean or loading variation is likely to have a smaller negative effect. Thatbeing said, the current study showed that, even when one of the assumptionviolation is relatively small, the two assumption violations combined canlead to significantly worse results. Overall, as long as it is suspected thatthe item means or loadings may vary within each composite score, ACMLis not recommended.In contrast, TSML is highly recommended for bivariate regression withitem-level missing data. It has the property of being unbiased, having goodcoverage across the board, and it is only slightly less efficient than ACMLin some cases. However, when the sample size is small and the number ofvariables is high, it is possible that the EM algorithm available in the soft-ware will not converge when estimating the saturated ML covariance matrix.There are a few considerations in this case. Adjusting the starting values of637.1. Recommendationsthe EM algorithm may help it converge more easily. It may also be worthtrying different software to see if one of the EM algorithm implementationsis more effective at converging for the particular dataset. Finally, in the caseof 14- or 16-item scores, parcelling may be a good idea. It should be notedthat incomplete items cannot be used in the parcels in this context, becausethey are required to be directly present in the saturated model during Stage1. However, items with complete data may be parcelled to reduce the sizeof the overall covariance matrix.Finally, based on the theoretical discussion and the simulation results, itis clear that SL-FIML should not be used in the case of bivariate regression.Not only is the method likely to overestimate the regression coefficient, it isalso highly inefficient. The resulting combination of a large effect size andwide confidence interval is likely to be more misleading than informativeregarding the research question.7.1.2 For Missing Data Simulation StudiesThere are many different ways to generate MAR missing data. When choos-ing the appropriate method, there are some major factors to consider, suchas, the number of missing data patterns, the complexity of the mechanism,and how well the method corresponds to what may happen with real data.The current study suggests that the strength of the MAR mechanism shouldbe a more conscious consideration in simulation studies as well. In caseswhere large differences can be observed between different missing data meth-ods under strong MAR, the same observation cannot be made under weakMAR (see Figure 7.1 and Figure 7.2). An insufficiently strong MAR mech-anism may show good performance for a method, when the method couldpoor for the real data due to stronger MAR. Conversely, a perfect MAR,e.g., a single cutoff with 100% missing above the cutoff and 0% missingbelow the cutoff, may show poorer performance than one would expect toobserved in real data.Critically, researchers should be more conscious that MAR strength maychange when other aspects of the simulation studies are manipulated. Forexample, in a hybrid cutoff/linear MAR such as the one in Enders (2003)(Figure 2.2), the point where the sigmoidal function becomes a step functionis dependent on the per variable missing rate in the data. This meansthe shape of the function that defines missing probabilities based on theconditioning variable changes as the result of missing rate manipulation.Although the current paper does not provide a formal definition of MARstrength under such a case, it can be speculated that MAR strength is in647.1. Recommendationsfact changing along with the rate of missing data as well. Because the rate ofmissing data and MAR strength may have independent effects, this methodof MAR creation may confound the two effects. As for logistic linear MAR,to what extent it maintains the same strength of MAR selection mechanismacross different missing rate is less intuitive, and an important subject offuture research, should the method become more popular.657.1. Recommendations025507510050 100 150 200CoverageEqual Means, Equal Loadings025507510050 100 150 200Sample SizeCoverageDifferent Means, Equal Loadings025507510050 100 150 200Equal Means, Different Loadings025507510050 100 150 200Sample SizeMethodACMLSL−FIMLTSDifferent Means, Different LoadingsFigure 7.1: With strong linear MAR and a high regression coefficient,different population models showed large performance differences betweenACML, SL-FIML, and TSML.667.1. Recommendations025507510050 100 150 200CoverageEqual Means, Equal Loadings025507510050 100 150 200Sample SizeCoverageDifferent Means, Equal Loadings025507510050 100 150 200Equal Means, Different Loadings025507510050 100 150 200Sample SizeMethodACMLSL−FIMLTSDifferent Means, Different LoadingsFigure 7.2: With weak linear MAR and a high regression coefficient, dif-ferent population models showed drastically smaller performance differencesbetween ACML, SL-FIML, and TSML.677.2. Limitations and Future Directions7.2 Limitations and Future DirectionsThe current study only examined bivariate regression, and it should be ex-panded to multiple regression in future studies. While many properties ofthe methods are likely to hold, the degree to which they manifest may differgreatly. In particular, bivariate regression is the worst case scenario for SL-FIML. At the scale level, the outcome variable is the only extra informationthat can be used to fill in for the missing information in the predictor. Itis no surprise that the regression coefficient is overestimated in such a case.While efficiency would still be a problem for SL-FIML regardless of the num-ber of predictor scales, the degree of bias may be different for more complexmodels. More importantly, in a bivariate regression, the conditioning vari-able is always deleted at the item-level when data is missing, resulting inMNAR missingness. If all conditioning variables happen to be outside ofcomposite scores with missing data, SL-FIML should provide an unbiasedestimate, with only a loss in efficiency. More complex models will providethe opportunity to investigate the performance of the method between theworst case scenario and the ideal scenario.The current study described a basic approach of quantifying MAR strengthin the single cutoff case for a single conditioning variable. While it serves todemonstrate the potential problem with not controlling for MAR strength,and the possibility of looking into the performance of different methodson the continuum between perfect MAR to MCAR, the broader applica-tion of MAR strength requires more sophisticated definitions. As previouslypointed out, R2 (Mazza et al., 2015) does not capture the intuitive idea ofthe strength of the MAR mechanism, because a single-cutoff perfect MARwould not have a R2 of 1. Furthermore, when simulation studies use complexrules, for example to ensure the number of missing patterns (e.g., Savalei& Bentler, 2009, the MAR strength may be different for each conditioningvariable, and it may not be trivial to combine into a single overall number.Finally, if MAR strength is to be defined solely based on the relationshipbetween missing probabilities and the conditioning variables, it is likely tobe sensitive to nonnormality in the conditioning variable. For example, thesigmoidal part of the Enders (2003) MAR mechanism (Left panel in Figure2.2) only takes on the sigmoidal curve because the conditioning variable isthe rank transformed normally distributed variable. If the underlying vari-able was uniformly distributed, the missing probability function takes onthe shape of a straight line. It is likely that a more viable definition of MARwould involve metrics dealing directly with the missing information itself,such as the fraction of missing information.68ReferencesBeebe, D. W., Lewin, D., Zeller, M., McCabe, M., MacLeod, K., & Daniels,S. R. (2007). Sleep in overweight adolescents: shorter sleep, poorer sleepquality, sleepiness, and sleep-disordered breathing. Journal of PediatricPsychology, 32(1), 69–79.Burton, A., Altman, D. G., Royston, P., & Holder, R. L. (2006). The designof simulation studies in medical statistics. Statistics in Medicine, 25(24),4279–4292.Chen, L., Meier, K. M., Blair, M. R., Watson, M. R., & Wood, M. J. (2012).Temporal characteristics of overt attentional behavior during categorylearning. Attention, perception & psychophysics, 75(2), 244–56.Culbert, K. M., Breedlove, S. M., Sisk, C. L., & Burt, S. A. (2013). Theemergence of sex differences in risk for disordered eating attitudes duringpuberty: a role for prenatal testosterone exposure. Journal of AbnormalPsychology, 122(2), 420–432.Enders, C. K. (2001). The performance of the full information maximumlikelihood estimator in multiple regression models with missing data. Ed-ucational and Psychological Measurement, 61(5), 713–740.Enders, C. K. (2003). Using the expectation maximization algorithm toestimate coefficient alpha for scales with item-level missing data. Psycho-logical Methods, 8(3), 322–337.Gottschall, A. C., West, S. G., & Enders, K. (2012). A comparison ofitem-level and scale-level multiple imputation for questionnaire batteries.Multivariate Behavioral Research, 47(1), 1–25.Graham, J. W. (2009). Missing data analysis: making it work in the realworld. Annual Review of Psychology, 60(1), 549–576.Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007). How manyimputations are really needed? Some practical clarifications of multipleimputation theory. Prevention Science, 8(3), 206–213.69ReferencesKim, K. H. & Bentler, P. M. (2002). Tests of homogeneity of means and co-variance matrices for multivariate incomplete data. Psychometrika, 67(4),609–623.Little, R. J. A. (1988). A test of missing completely at random for multi-variate data with missing values.Little, R. J. A. & Rubin, D. B. (2002). Statistical analysis with missing data.New York: John Wiley & Sons.Little, T. D., Rhemtulla, M., Gibson, K., & Schoemann, A. M. (2013).why the items versus parcels controversy needn’t be one. PsychologicalMethods, 18(3), 285–300.Magnus, J. & Neudecker, H. (1989). Matrix Differential Calculus with Ap-plications in Statistics and Econometrics. John Wiley & Sons.Mazza, G. L., Enders, C. K., & Ruehlman, L. S. (2015). Addressing item-level missing data: a comparison of proration and full information maxi-mum likelihood estimation. Multivariate Behavioral Research, 50(5), 504–519.Muthe´n, L. K. & Muthe´n, B. O. (2010). Mplus User’s Guide. Sixth Edition.Los Angeles: CA: Muthe´n & Muthe´n.Orcan, F. (2013). Use of item parceling in structural equation modelingwith missing data. Eitimde ve Psikolojide O¨lc¸me ve Deerlendirme Dergisi,7(November), 1–109.Parent, M. C. (2013). Handling item-level missing data : simpler is just asgood. The Counseling Psychologist, 41(4), 568–600.R Core Team (2016). R: A Language and Environment for Statistical Com-puting. R Foundation for Statistical Computing, Vienna, Austria.Rosseel, Y. (2012). lavaan: An R package for structural equation modeling.Journal of Statistical Software, 48(2), 1–36.Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592.Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. JohnWiley & Sons.70ReferencesRussell, D. (1996). UCLA Loneliness Scale Version 3. Journal of Personalityand Social Psychology, 39, 3–4.Savalei, V. (2010). Expected versus observed information in SEM withincomplete normal and nonnormal data. Psychological Methods, 15(4),352–367.Savalei, V. & Bentler, P. M. (2005). A statistically justified pairwise MLmethod for incomplete nonnormal data: a comparison with direct MLand pairwise ADF. Structural Equation Modeling: A MultidisciplinaryJournal, 12(2), 183–214.Savalei, V. & Bentler, P. M. (2009). A two-stage approach to missing data:theory and application to auxiliary variables. Structural Equation Model-ing: A Multidisciplinary Journal, 16(3), 477–497.Savalei, V. & Rhemtulla, M. (2017a). Normal theory GLS estimator formissing data: an application to item-level missing data and a comparisonto two-stage ML. Frontiers in Psychology, 8(May).Savalei, V. & Rhemtulla, M. (2017b). Normal theory two-stage ML estimatorwhen data are missing at the item level. Journal of Educational andBehavioral Statistics, XX(X), 1–27.Schafer, J. L. & Graham, J. W. (2002). Missing data: Our view of the stateof the art. Psychological Methods, 7(2), 147–177.Yuan, K. H. (2009). Normal distribution based pseudo ML for missing data: With applications to mean and covariance structure analysis. Journalof Multivariate Analysis, 100(9), 1900–1918.Yuan, K. H. & Bentler, P. M. (2000). Three likelihoodbased methods formean and covariance structure analysis with nonnormal missing data.Sociological methodology, 30(1), 165–200.71
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- Two-stage maximum likelihood approach for item-level...
Open Collections
UBC Theses and Dissertations
Featured Collection
UBC Theses and Dissertations
Two-stage maximum likelihood approach for item-level missing data in regression Chen, Lihan 2017
pdf
Page Metadata
Item Metadata
Title | Two-stage maximum likelihood approach for item-level missing data in regression |
Creator |
Chen, Lihan |
Publisher | University of British Columbia |
Date Issued | 2017 |
Description | Psychologists often use scales composed of multiple items to measure underlying constructs, such as well-being, depression, and personality traits. Missing data often occurs at the item-level. For example, participants may skip items on a questionnaire for various reasons. If variables in the dataset can account for the missingness, the data is missing at random (MAR). Modern missing data approaches can deal with MAR missing data effectively, but existing analytical approaches cannot accommodate item-level missing data. A very common practice in psychology is to average all available items to produce scale means when there is missing data. This approach, called available-case maximum likelihood (ACML) may produce biased results in addition to incorrect standard errors. Another approach is scale-level full information maximum likelihood (SL-FIML), which treats the whole scale as missing if even one item is missing. SL-FIML is inefficient and prone to bias. A new analytical approach, called the two-stage maximum likelihood approach (TSML), was recently developed as an alternative (Savalei & Rhemtulla, 2017b). The original work showed that the method outperformed ACML and SL-FIML in structural equation models with parcels. The current simulation study examined the performance of ACML, SL- FIML, and TSML in the context of bivariate regression. It was shown that when item loadings or item means are unequal within the composite, ACML and SL-FIML produced biased estimates on regression coefficients under MAR. Outside of convergence issues when the sample size is small and the number of variables is large, TSML performed well in all simulated conditions, showing little bias, high efficiency, and good coverage. Additionally, the current study investigated how changing the strength of the MAR mechanism may lead to drastically different conclusions in simulation studies. A preliminary definition of MAR strength is provided in order to demonstrate its impact. Recommendations are made to future simulation studies on missing data. |
Genre |
Thesis/Dissertation |
Type |
Text |
Language | eng |
Date Available | 2017-08-18 |
Provider | Vancouver : University of British Columbia Library |
Rights | Attribution-NonCommercial-NoDerivatives 4.0 International |
DOI | 10.14288/1.0354497 |
URI | http://hdl.handle.net/2429/62724 |
Degree |
Master of Arts - MA |
Program |
Psychology |
Affiliation |
Arts, Faculty of Psychology, Department of |
Degree Grantor | University of British Columbia |
Graduation Date | 2017-09 |
Campus |
UBCV |
Scholarly Level | Graduate |
Rights URI | http://creativecommons.org/licenses/by-nc-nd/4.0/ |
Aggregated Source Repository | DSpace |
Download
- Media
- 24-ubc_2017_september_chen_lihan.pdf [ 849.94kB ]
- Metadata
- JSON: 24-1.0354497.json
- JSON-LD: 24-1.0354497-ld.json
- RDF/XML (Pretty): 24-1.0354497-rdf.xml
- RDF/JSON: 24-1.0354497-rdf.json
- Turtle: 24-1.0354497-turtle.txt
- N-Triples: 24-1.0354497-rdf-ntriples.txt
- Original Record: 24-1.0354497-source.json
- Full Text
- 24-1.0354497-fulltext.txt
- Citation
- 24-1.0354497.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.24.1-0354497/manifest