Robustness of Multivariate Mixed Model A N O V A by-Robert James Prosser B.Sc, The University of British Columbia, 1971 A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of M A S T E R OF ARTS in The Faculty of Graduate Studies Department of Educational Psychology and Special Education We accept this thesis as conforming to the reauired standard The University of British Columbia August 1985 © Robert James Prosser, 1985 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make i t freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department of /jy^^-^^^ ? a f ^f^ax/tav The University of British Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 DE-6 (3/81) ii Abstract In experimental or quasi-experimental studies in which a repeated measures design is used, it is common to obtain scores on several dependent variables on each measurement occasion. Multivariate mixed model ( M M M ) analysis of variance (Thomas, 1983) is a recently developed alternative to the M A N O V A procedure (Bock, 1975; T imm, 1980) for testing multivariate hypotheses concerning effects of a repeated factor (called occasions in this study) and interaction between repeated and non-repeated factors (termed group-by-occasion interaction here). If a condition derived by Thomas (1983), multivariate multi-sample sphericity (MMS), regarding the equality and structure of orthonormalized popu-lation covariance matrices is satisfied (given multivariate normality and independence for distributions of subjects' scores), valid likelihood-ratio M M M tests of group-by-occasion interaction and occasions hypotheses are possible. To date, no information has been available concerning actual (empirical) levels of significance of such tests when the M M S condition is violated. This study was conducted to begin to provide such information. Departure from the M M S condition can be classified into three types— termed departures of types A , B, and C respectively: (A) the covariance matrix for population g (g = 1,...G), when orthonormalized, has an equal-diagonal-block form but the resulting matrix for population g is unequal to the resulting matrix for population g' (g / g'); (B) the G populations' orthonormalized covariance matrices are equal, but the matrix common to the populations does not have equal-diagonal-block structure; or (C) one or more populations has an orthonormalized covariance matrix which does not have equal-diagonal-block structure and two or more populations have unequal orthonormalized matrices. In this study, Monte Carlo procedures were used to examine the effect of each type of violation in turn on the Type I error rates of multivariate mixed model tests of group-by-occasion interaction and occasions null hypotheses. For each form of violation, experiments modelling several levels of severity were simulated. In these experiments: (a) the number of measured variables was two; (b) the number of measurement occasions was three; (c) Abstract iii the number of populations sampled was two or three; (d) the ratio of average sample size to number of measured variables was six or 12; and (e) the sample size ratios were 1:1 and 1:2 when G was two, and 1:1:1 and 1:1:2 when G was three. In experiments modelling violations of types A and C, the effects of negative and positive sampling were studied. When type A violations were modelled and samples were equal in size, actual Type I error rates did not differ significantly from nominal levels for tests of either hypoth-esis except under the most severe level of violation. In type A experiments using unequal groups in which the largest sample was drawn from the population whose orthogonalized covariance matrix has the smallest determinant (negative sampling), actual Type I error rates were significantly higher than nominal rates for tests of both hypotheses and for all levels of violation. In contrast, empirical levels of significance were significantly lower than nominal rates in type A experiments in which the largest sample was drawn from the population whose orthonormalized covariance matrix had the largest determinant (positive sampling). Tests of both hypotheses tended to be liberal in experiments which modelled type B violations. No strong relationships were observed between actual Type I error rates and any of: severity of violation, number of groups, ratio of average sample size to number of variables, and relative sizes of samples. In equal-groups experiments modelling type C violations in which the or-thonormalized pooled covariance matrix departed at the more severe level from equal-diagonal-block form, actual Type I error rates for tests of both hypotheses tended to be liberal. Findings were more complex under the less severe level of structural departure. Empirical significance levels did not vary with the degree of interpopulation heterogeneity of orthonormalized covariance matrices. In type C experiments modelling negative sampling, tests of both hypothe-ses tended to be liberal. Degree of structural departure did not appear to influence actual Type I error rates but degree of interpopulation heterogeneity did. Actual Type I error rates in type C experiments modelling positive sampling were apparently related to the number of groups. When two populations were sampled, both tests tended to be conser-vative, while for three groups, the results were more complex. In general, under all types of violation the ratio of average group size to number of variables did not greatly affect actual Type I error rates. The report concludes with suggestions for practitioners considering use of the M M M procedure based upon the findings and recommends four avenues for future research on Type I error robustness of M M M analysis of variance. The matrix pool and computer programs used in the simulations are included in appendices. Contents iv Acknowledgement xi 1 Introduction 1 1.1 Hypothesis testing: P = 1 case 1 1.2 Hypothesis testing: P > 2 case 6 2 Multivariate Mixed Model A N O V A 9 2.1 The model 9 2.2 G by T and T null hypotheses 14 2.3 Likelihood ratio testing of multivariate hypotheses . . . 16 2.4 M M M hypothesis testing 19 2.5 Use of the M M M testing procedure 24 2.6 Covariance matrices which satisfy the M M S condition . . 25 2.7 Violations of the M M S condition 26 2.8 Review of related research 27 3 Methodology 46 3.1 Conditions 46 3.2 Violations of the M M S condition modelled in the study 47 3.3 The simulation process 51 4 Results 54 4.1 Empirical alpha values under type A violations of M M S . 55 4.2 Empirical alpha values under type B violations of M M S . 58 4.3 Empirical alpha values under type C violations of M M S . 61 5 Conclusions and Recommendations 67 5.1 Summary of effects on Type I error rates 68 5.2 Practical implications of this study 72 5.3 Limitations of the study 74 References 78 Contents v A P P E N D I C E S A Matrices Used in the Simulations 80 A . l Generating and testing matrices 80 A . 2 The matrix pool 82 B Computer Programs for the Monte Carlo Simulations . . . 86 B. l The preparation program 86 B.2 The simulation program 93 B.3 Benchmark tests 100 vi Tables 1 Number of Type I errors in 1000 replications for mixed model tests, related to matrix e values 29 2 Number of Type I errors per 1000 replications for multivariate and mixed model ANOVA tests, related to sampling arrangement and matrix e value 33 3 Number of Type I errors per 1000 replications for multivariate and mixed model tests, related to form of matrices 37 4 Power of tests of H 0 g t and H 0 x related to form of matrices . . . . 39 5 Number of Type I errors per 1000 replications for multivariate and mixed model tests, related to score distribution form and matrix e value 41 6 Power of multivariate and mixed model tests, related to score dis-tribution form and matrix e value 43 7 Number of Type I errors in 1000 replications (1000 x a n o m = 50) : Type A violations 56 8 Number of Type I errors in 1000 replications (1000 x o t n o m = 10) : Type A violations 57 9 Number of Type I errors in 1000 replications (1000 x a n o m = 50) : Type B violations 59 10 Number of Type I errors in 1000 replications (1000 x or n o m = 10) : Type B violations 60 11 Number of Type I errors in 1000 replications (1000 x a n o m = 50) : Type C violations 62 12 Number of Type I errors in 1000 replications (1000 x a n o m = 10) : Type C violations 63 13 Number of Type I errors in 1000 replications for benchmark tests of the first type 101 14 Number of Type I errors in 1000 replications for benchmark tests of the second type 102 Figures vii Figures 1 Operations involved in conducting a study using a groups by occa-sions repeated measures design 2 2 Data set obtained from a study based upon a groups by occasions repeated measures design 3 3 Structure of the data matrix in MMM analysis 10 4 Expanded display of X B + E 11 viii Notation The following list identifies symbols used in the thesis and the number of the page on which each symbol first appears. N the total number of subjects in a given experiment, p. 1. G the number of populations sampled in an experiment, p. 1. P the number of variables on which each subject provides scores, p. 1. T the number of measurement occasions on which each subject provides P scores, p. 1. ng the number of subjects in the gth. sample, p. 1. Yp the pth dependent variable, p. 1. Y?t the observed score on variable p for subject i of group g on the tth. occasion, p. 1. Yigt the vector of observed scores for subject i of group g on the tth occa-sion, p. 1. Jjg the PT x PT covariance matrix for the gth population, p. 4. ~N(ng,'Eg) a multivariate normal distribution with centroid fig and covariance matrix p. 4. C a T x T — 1 orthonormal contrast matrix, p. 4. A a positive constant, p. 4. Ij-i an identity matrix of order T — 1, p. 4. e Box's (1954) parameter, p. 6. S e an error sums of squares and cross products (SSCP) matrix, p. 6. S / , T an SSCP matrix for an occasions hypothesis, p. 6. S ^ Q T an SSCP matrix for a parallelism hypothesis, p. 6. Ip an identity matrix of order P, p. 7. ® the Kronecker product operation on a pair of matrices, p. 7. E a P x P positive definite symmetric matrix, p. 7. Y a data matrix, p. 9. X a design matrix, p. 9. B a matrix of parameters, p. 9. E a residual matrix, p. 9. Notation ix the direct sum of G matrices, p. 12. <7=1 lng a vector of ng ones, p. 12. E,-£ the residuals for subject t in the gth group, p. 14. vec the vec operator, p. 12. H 0 G T the (omnibus) parallelism null hypothesis, p. 14. H 0 T the (omnibus) occasions null hypothesis, p. 14. F a G x G — 1 contrast matrix of full column rank, p. 15. Wp(JV ,E , A) a Wishart distribution function with N degrees of freedom and non-centrality parameter A , p 14. Up,(/2,i/i the U distribution function with parameters u2 and uit p. 17. i/h degrees of freedom for the hypothesis SSCP, p. 18. ue degrees of freedom for the residual SSCP, p. 18. A Wilks likelihood ratio criterion, p.18. T an SSCP matrix for the parallelism hypothesis, p. 19. U an SSCP matrix for the occasions hypothesis, p. 19. Cq a T x q (q < T — 1) orthonormal contrast matrix of rank q, p. 20. Tq an SSCP matrix for the occasions hypothesis formed using as the contrast matrix, p. 20. F r a G x r (r < G — 1) contrast matrix of rank r, p. 20. U g r an SSCP matrix for the parallelism hypothesis formed using Cq and F r , p. 20. Hg an ng x ng — 1 orthonormal contrast matrix for the gth. group, p. 20. W an SSCP residual matrix for testing within-subjects hypotheses, p. 20. M Box's (1949) M statistic, p. 24. S a matrix constant for G populations, p. 24. Sg an unbiased estimate of Jjg, p. 24. S the pooled sample covariance matrix, p. 24. M a transformation of the generalized Mauchly criterion, p. 25. a P x P matrix for the gth. population, p. 26. a n o m the nominal Type I error rate for a given hypothesis test, p 28. N r e p s the number of replications of a simulated experiment, p. 28. E^> • • • > E \y population covariance matrices used in the simulations in this study, p. 48. Notation x M* a modified form of Box's M, p. 48. AT a modified form of the transformed generalized Mauchly criterion, p. 49. Mlo M* evaluated with N = 40, p. 49. xi Acknowledgement At all stages of this study I was able to call on the advice and support of my thesis supervisor, Dr. Todd Rogers. His willingness to share of his expertise and time is greatly appreciated. I am also thankful for the assistance and encouragement of Dr. J im Steiger and Dr. Robert Conry. This project received computing support from the Department of Educa-tional Psychology and Special Education. Special thanks go to Dr. Bryan Clarke, Depart-ment Head. Special thanks also go to Dr. Larry Roberts, Mr . Eric Ledoxix, and Mr . Feng Chen for their assistance in the typesetting of this thesis. I deeply appreciate the encouragement and support of my family, friends, and colleagues throughout my graduate program. CHAPTER 1 Introduction In an experiment in which a groups by occasions repeated measures design is used, each of N subjects provides a set of scores on P (> 1) variables of interest, Y1,..., Yp on T (> 2) measurement occasions. The N subjects are divided among G (> 1) groups; ng subjects belong to group g (g = 1, . . . , G) and Y^=i ng — N. Figure 1 diagrams the experimental operations involved in such a study, employing notation similar to that of Cook and Campbell (1979). Using the symbol Yigt to represent the vector of P scores which subject t of group g provides at time t, Figure 2 schematically portrays the data set obtained from an experiment based on this design. 1.1 H Y P O T H E S I S T E S T I N G : P = l C A S E In analyzing data from an experiment using this design with P = 1, a statistician has a choice among several procedures with which to test hypotheses concerning group-by-occasion (G by T) interaction effects, occasion (T) main effects and/ or group (G) main effects (Rogan, Keselman, & Mendoza, 1979). The two primary techniques are univari-ate mixed model analysis of variance (ANOVA) and multivariate analysis of variance ( M A N O V A ) . These are well described in Meyers (1979) and T imm (1975) respectively. Each technique is founded upon assumptions about the data-generating process operating in the experimental situation, and the choice between techniques depends, in part, on the tenability of the assumptions for each. If the assumptions for a hypothesis testing proce-1 Introduction / I.J 2 G R O U P O P E R A T I O N S X „ O 11 ••it O it o 9 G o 5 LG1 o G l o Gt GT o X is the administration of a treatment condition of type t to subjects in group g O is the recording of P scores, on completion of treatment condition t for group g subjects F I G U R E 1 Operations Involved in Conducting a Study Using a Groups by Occasions Repeated Measures Design. Introduction / 11 3 G R O U P Subjects OCCASIONS 111 lit n, nilT lgt igT nggl nggt ^nggT IG1 IGt IGT nG nGGl nGGt nGGT FIGURE 2 Data Set Obtained from a Study Based Upon a Groups by Occasions Repeated Measures Design. introduction / 1.1 4 dure are not met, the Type I error rate and power values for tests using the procedure will be affected. In the univariate mixed model, an observed score Yigt for subject t of group g on occasion t can be expressed as the sum of a fixed parameter figl and a random component eigt: Yigt — Pgt + eigt> 1 = • • •»ng't 9 = 1| • • • > * = 1, • • •, ^ • (These terms are discussed in detail in the first section of the following chapter.) Let E g be the covariance matrix for population g subjects' T scores. The necessary and sufficient conditions for validity of G by T and T hypothesis tests with univariate mixed model A N O V A are: (a) multivariate normality of the distribution of the T scores for subject t of group g, » = !>•••»ng\ 9 = that is, [YigU...,YigT\ ~ N( /x f f ,E i / ) where fig is the centroid for population g\ (b) independence of the distributions of the scores of subjects i g and i ' g1 for i, i' = ly->ng', gg' = 1,...,G; (with i' and g ^ g'); (c) a multisample sphericity condition on the G population covariance matrices. This condition—established by Huynh and Feldt (1970)—can be written C ' E ^ = ••• = C ' E G C = A I r _ ! (1.1) where C is a T x T - 1 orthonormal contrast matrix, I^-i i3 the identity matrix of order T — 1, and A is a positive constant. The assumptions underlying the univariate mixed model A N O V A test of the G hypothesis are those of the univariate fixed effect one factor A N O V A ; a valid test of the G hypothesis thus does not require that the multisample sphericity condition hold. Effects of violation of assumptions for this type of A N O V A are well documented (see, for example, Glass, Peckham, & Sanders (1972)). The assumptions for valid use of M A N O V A in testing G by T and T hy-potheses are: (a) multivariate normality of the distribution of the T scores for subject i of group g, i = 1 , . . . , ng\ g = 1 , . . . , G\ Introduction / 1.1 5 (b) independence of the distributions of the scores of subjects t g and »' g' for t = 1, . . . , ng) i' = 1 , . . . , rig,-, g, g' = 1 , . . . , G; (with t / i' and g ^ g')\ (c) simple equality of the covariance matrices for the G populations—Ex = • • • = (Huynh and Mandeville, 1979). Although the assumptions underlying the M A N O V A method are less re-strictive than those of the univariate mixed model approach, M A N O V A is not always the method of choice for testing G by T and T hypotheses. Rogan et al. (1979, p. 274) give two reasons for this: (a) the multivariate procedure cannot be used when the number of subjects is less than G + T — 1 and, more importantly, (b) when the conditions are satisfied for valid use of the mixed model A N O V A technique, this technique is more powerful than the M A N O V A procedure (See also Mendoza, Toothaker, & Nicewander, 1974). Behavioral scientists wishing to use the mixed model A N O V A technique to analyze data from experiments in which P = 1 should bear in mind that covariance ma-trices for T (> 2) measurement occasions in typical behavioral science experiments will often fail to satisfy the multisample sphericity condition (Rogan et al., 1979). In studies examining the effect of treatments on subjects' performance over time, for example, co-variance matrices may have a simplex form (Humphreys, 1960), since two measurements obtained over a short time interval will generally correlate more highly than two measure-ments between which there is a longer interval. The effect of such departures from the multisample sphericity assumption underlying univariate mixed model A N O V A testing (in situations in which all samples are equal in size) is to increase the Type I error rate associated with testing the omnibus G by T and T hypotheses above the nominal level (Collier, Baker, Mandeville, &; Hayes, Introduction / 1.2 6 1967; Mendoza, Toothaker, & Nicewander, 1974). An empirical study by Collier et al. (1967) of the relationship of Type I error rate to degree of departure (of one type) from multisample sphericity—as indexed by Box's (1954) parameter e—suggests that when small samples are used, the problem of excessive test size becomes serious for e values less than approximately 0.64. 1.2 H Y P O T H E S I S T E S T I N G : P > 2 C A S E Bock (1975) and T imm (1980) described a M A N O V A procedure to test G by T and T hypotheses for experiments in which each subject supplies two or more scores on each measurement occasion. In this procedure each subject is considered to have scores on PT dependent variables. Results from the fundamental multivariate least squares theorem (Timm, 1975, p. 189) are used in developing the hypothesis tests and in computing the sums of squares and cross product (SSCP) matrices required for the tests. Recently, Thomas (1983) proposed a multivariate mixed model (MMM) tech-nique as an alternative to the M A N O V A procedure. This technique is analogous to the mixed model A N O V A technique used in hypothesis testing for the P = 1 case. The three PxP SSCP matrices used in the M M M procedure—S/tG T, for the G by T hypothesis, S / , T , for the T hypothesis, and S e for the within-cells error—correspond to the familiar sums of squares for G by T, T , and subjects by occasions within groups, respectively. A multivari-ate test in M M M analysis based on the characteristic roots of the matrix S e ( S e + S ^ G T ) 1 and a similar test based on the roots of S e ( S e + S/, ) 1 correspond to the F tests of the G by T and T hypotheses respectively in the univariate mixed model A N O V A procedure. Thomas has derived a sufficient condition which, if satisfied, permits valid use of the M M M tests. The condition, which he called multivariate multisample sphericity (MMS), corresponds to the Huynh and Feldt (1970) condition—Equation (1.1)—for valid univariate mixed model A N O V A testing. The latter is both sufficient and necessary. If S s , g = 1 , . . . , G, are the G groups' PT x PT population covariance matrices, S is some Introduction / 1.2 7 P x P positive definite symmetric matrix, and Ip and I T - 1 are identity matrices of order P and T — 1 respectively, and ® is the Kronecker product (Timm, 1975, pp. 30-31), the condition can be stated as follows: ( C ' ® I p ) 2 0 ( C ® I p ) = I r _ i ® S for all g= 1 , . . . , G . (1.2) Thomas (1983, p. 451) suggested that the M M M procedure is possibly of value for two reasons: (a) the process and results of testing "within subjects" hypotheses with the M M M tech-nique parallel those of the mixed model A N O V A technique familiar to behavioral scientists and, more importantly, (b) there may be a power advantage for the M M M technique over the Bock/ T imm M A N O V A technique when conditions guaranteeing valid tests under both proce-dures are satisfied. To date, no theoretical or empirical investigations of the Type I error rate and/ or power of the M M M tests of G by T or T hypotheses have appeared in the literature. Since it is common in behavioral experiments to obtain measurements on more than one dependent variable, an evaluation of the M M M procedure would appear to be warranted. Olson (1974) pointed out that "very high Type I error rates make a test dangerous; low power merely makes it less useful" (p.907). Following this philosophy, an assessment of the M M M procedure should begin with an investigation of the Type I error rates of M M M tests under a variety of conditions. The purpose of this study, then, was to answer the following preliminary question about Thomas' (1983) M M M testing procedure in experiments in which P > 2 : What effect does departure from the MMS condition—Equation (1.2)—have on the Type I error rates of MMM tests of omnibus G by T and T hypotheses? Introduction / 1.2 8 The following more specific questions were addressed in the study: (a) How does alteration of G modify the effect of departure from M M S on the Type I error rate of M M M tests of G by T and T hypotheses? (b) How does alteration of N/(GP) modify the effect of departure from M M S on the Type I error rate of M M M tests of G by T and T hypotheses? (c) How does alteration of the relative sizes of the samples modify the effect of departure from M M S on the Type I error rate of M M M tests of G by T and T hypotheses? The sequel describes the theory underlying M M M analysis and the proce-dures used to provide information concerning these questions. The second describes the multivariate mixed model and discusses the multivariate multisample sphericity condition. Chapter 3 outlines the process for the computer simulation used in the present study. The fourth and fifth chapters present the results and conclusions from the investigation. CHAPTER 2 Multivariate Mixed Model ANOVA This chapter first outlines the theory underlying the multivariate mixed model analysis of variance procedure for testing hypotheses on group by occasion inter-action and occasion main effects in studies for which P > 2 . Included are discussions of Thomas' ( 1 9 8 3 ) sufficient condition under which the M M M procedure can be validly used—Equation (1 .2) in the previous chapter—and violations of this condition. The chap-ther concludes with a brief review of robustness studies concerned with techniques for analysis of repeated measures data in experiments in which P = 1. The description given here closely follows the presentations of Thomas ( 1 9 8 1 , 1 9 8 3 ) , Rogan et al. ( 1 9 7 9 ) , Meyers ( 1 9 7 9 ) , and Timm ( 1 9 7 5 , 1 9 8 0 ) . 2.1 T H E M O D E L The form of the data collected in an experiment having a G by T repeated measures design was illustrated in Figure 2 . The data vectors, Y t f f t can be arranged to produce an NT x P data matrix Y . The structure of Y is shown in Figure 3 . This matrix of observed scores may be fitted using the general linear model Y = X B + E . (2.1) This expression is written in an expanded form in Figure 4 to show the structure of the design, parameter and residual matrices—X, B , and E respectively. 9 Multivariate Mixed Model ANOVA / 2.1 yl M i l Y P Yl M I T Yp M i r yP M i r Y1 Y1 yP J n ! l l IT yP J n . l l yP Y1 101 yP 'lGl Y? \G1 V1 1GT Y\GT yP '1GT Y1 yP yP *nGGl Yl .nGGT YnGGT •' yP nGGT. F I G U R E 3 Structure of the Data. Matrix in MMM Analysis. Multivariate Mixed Model ANOVA / 2.1 1 0 0 1 0 0 0 0 0 00 0 0 0 0' 0 0 0 1 0 0 00 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 Mil 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 00 0 1 0 0 0 1 0 0 0 0 0 A*G1 P I T A*SI + 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 00 F I G U R E 4 Expanded Display of X B + E . Multivariate Mixed Model ANOVA / 2.1 12 The element Ypgl of Y stands for the observed score for subject t of group g on variable p on measurement occasion t. The NT x GT design matrix specifies the grouping structure for the N subjects, taking into account that each subject has T sets of scores. It can be written in a shorthand notation using the direct sum (Searle, 1971, p. 231) and Kronecker product operations on matrices. G x = E + ( 1 n , ® I r ) (2-2) g=l In the NT x P residual matrix, element represents a sum of random components of observed score Ypgl: <. = •&,+(")£/,+«&. <2-3) In this expansion, irPjg is an effect attributable to randomly-selected person i in group g, (WT)it/g 1 3 a n m t e r a c t i o n between that person and occasion t, and e^igt is random measure-ment error. Element pFgt of the GT x P parameter matrix is the expected value of Ypt— that is, (tgf = E(F£ t ) . Each element can be expressed as a sum of a group effect parameter, 7^, an occasion effect parameter, rf, and, when an additive model is not assumed, a group by occasion interaction parameter, i?gt: »Pgt = 1P9 + r[ + cpgt (2.4) Part of Meyers' (1979) discussion of mixed model A N O V A provides a good framework for interpreting Equation (2.4). Six preliminary definitions are needed. (1) fiPgl is the expected value of Ypgl. It is the true score of subject i of group g for variable p on occasion t. (2) fip — E ( l / T ^ ^ _ 1 Yfgt) is the true average score (across all levels of the occasion factor) of subject i of group g for variable p. (3) fip is the expected value of variable p over all occasions for the gth. population—that is, the population from which subjects in group g were randomly selected. Multivariate Mixed Model ANOVA / 2.1 13 (4) fip is the expected value—over all G populations—of variable p on occasion t. (5) nPt = E(Ypt), as noted earlier. (6) fip is the grand mean for the pth variable—(ip — l/T YlJ=i The terms included in Equation (2.4) can now be fully defined: (a) if = fip — fip is the occasions effect on occasion t for variable p; (b) = nPg — fip is the group effect for variable p; and (c) igl = (igl - if - T£ + fj,p is the interaction of group g with occasion t for variable p. Further, definitions of the three random components of t which appear in Equation (2.3) can also be stated in terms of the six expected values above: (a) n?/g = fipig - fipg. ( b ) W i t / g = tigt - Tl - ^ - * i / g ~ 'Jo ^ (c) ep - Y? - up In the G by T repeated measures design discussed here, the grouping and occasions factors are both considered to be fixed. The model is mixed because subjects-within-groups is considered to be a random factor. Side conditions expressing the fixed nature of the G and T factors can be incorporated into the model: G ]r 7£ = 0 for all p (2.5) 47=1 T ^2 rf = 0 for all p (2.6) i=i G Lpgt = 0 for all * and p (2.7) 9=1 T ^2 ijt = 0 f o r a U 9 and P- (2-8) t=i In fitting the model X B + E to Y , two basic assumptions are made. These assumptions are also made for the Bock/ T imm multivariate analysis of the data. Let E,„ be the submatrix eigl •' • Si" A J> Multivariate Mixed Model ANOVA / 2.2 14 of E . Using the vec operator (Muirhead, 1982, p. 17) , these assumptions can be written: (1) for 1 < i < ng, 1 < g < G, vec(E^) has a multivariate normal distribution N(0, Hg) where S g is a PT x PT covariance matrix for population g; and (2) vec(E^) and vec(E^y) are independent whenever t ^ i' and/ or g ^ g1. 2.2 G B Y T A N D T N U L L H Y P O T H E S E S Expressions for the omnibus group by occasion interaction null hypothesis—sometimes termed the parallelism hypothesis—and the omnibus occasion null hypothesis can be writ-ten in several equivalent ways using (a) vectors composed of the fipt elements of B ; (b) vectors of rf and igl effect parameters; or (c) matrix products—contrast matrices pre-multiplying B . A l l three formats are given here for completeness. The latter is utilized in the discussion of hypothesis testing procedures. The parallelism hypothesis H Q g t is: Mi l A*12 M i ( r - i ) ~ M i r M n A*f 2 p P M i ( r - i ) M i r MGI M r ? r r -G{T-1) MGI M G 2 MGT MG2 p _ p MG(T-I) MGT Use of Equation (2.4) and inclusion of side conditions (2.7) and (2.8) in the model permit a restatement of this omnibus hypothesis as: H 0 G T : tpgt = 0 for g = 1 , . . . , G; t = 1 , . . . , T; and p = 1, . . . , P. The omnibus occasion null hypothesis H 0 t can be expressed as: (2.9) " M i l " M i r MGI MGT . . P , p M n M i r -MGI. .MGT-Multivariate Mixed Model ANOVA / 2.2 15 In situations in which H 0 q t as expressed in Equation (2.9) is tenable, the omnibus occasion null hypothesis can be written Hox : 1 i = z ' = T T forp = 1,...,P. (2.11) When B is structured as in Figure 4, the matrix product format of the parallelism hypothesis is H 0 g t : (F' ® C')B = 0. (2.12) In this expression, F is a G x G - 1 contrast matrix of full column rank such as 1 .. o-- 1 0 0 0 0 0 0 1 0 .. . - 1. and C i s a T x T - l orthonormal contrast matrix of rank T — 1—for example this Helmert type matrix (Basilevsky, 1983, pp. 155-157): C' = V2 - 1 v/2 0 0 - l i - ( r - i ) y/T2-T y/T2-T y/T*-T " ' y/T^-T Multivariate Mixed Model ANOVA / 2.3 16 When H 0 g t is true, the omnibus occasion null hypothesis can be written H 0 X : ( l ' G ® C ' ) B = 0 (2.13) where 1G is a vector of G ones. 2.3 L I K E L I H O O D R A T I O T E S T I N G O F M U L T I V A R I A T E H Y P O T H E S E S Multivariate mixed model tests of null hypotheses concerning group by occasion interaction and occasion effects are patterned after the mixed model A N O V A tests of H 0 g t and H 0 T used when P = 1. Several elements of the general theory of multivariate likelihood ratio hypothesis testing are presented here as background for discussion of the M M M tests in Section (2.4): (a) the definitions of the Wishart (W) and U distributions; (b) a theorem about the distribution of the likelihood ratio criterion, A; (c) theorems about the independence and distribution of quadratic forms; (d) the Fundamental Multivariate Least Squares Theorem (Timm, 1975, p. 189). Comparisons are noted between the P = 1 case and the P > 2 case. Def in i t ion of the W i sha r t D i s t r i bu t i on . If . . . , Y ^ are N indepen-dently distributed vectors of P random variables such that Yt- is distributed as Np(/z,-, E ) for i = 1,...,N then S = YliLi Y,-Y't- has a noncentral Wishart distribution with N degrees of freedom and noncentrality parameter A = Ylf=i ViV-'i- This is written S ~ Wp(N, E , A ) . If ft; — 0 for t = 1, . . . , N then S has a central Wishart distribution. If P > N the Wishart distribution does not exist. When P = 1, E becomes cr2 and S becomes a scalar random variable, s 2 , whose x2 distribution has N degrees of freedom— s2~X2(N,6) where5 = E£i^?-A Desc r ip t ion of the U D i s t r i bu t i on . When the null hypothesis C B = 0 is true and when V\ , v 2 > P, the distribution of U p ^ Vl is the same as the distribution of Multivariate Mixed Model ANOVA / 2.3 17 Ylp=i Yp where the Yp p = 1,..., P are independent random variables with beta density functions—that is fp(y) = Pr{Yp < y} = \ . \ ,\ y-^'1 ( l - y ) ^ " 1 forO<y<l; and = 0 elsewhere, where Y{z) (= /0°° t*'1^1 dt , when z > 0) is the gamma function. Define a as - ( ^ -/ >~2 2 + I). Then the cumulative distribution function of —o In Up„2)I/1 is approximated as follows: Pr{-a In U < u) = Pr{x2(Pi/2) < u} + b (PT{X2(PV2 + 4) < u} - Pr{x2(P"2) < u}) where 6 = Fv^Pi&*i2~^ a n d °( a) - 3 1 8 a remainder whose size is in the order of a - 3 (An-derson, 1958, p. 208; Muirhead, 1982, p. 459). When P = 1, there is a simple relationship between the U distribution and the central F distribution: U l l / 2 I / 1 = vxl{yx + v2FV2l,l) (Timm, 1975, p. 147). Theorem 2.1 If matrix quadratic forms Sx and S2 have independent central Wishart distributions with ux and u2 degrees of freedom respectively—Sx ~ Wp(^!,S,0) and S 2 ~ Wp(i/2,E,0)—and if ux > P, likelihood ratio criterion A = I S ^ S j + S 2 ) - 1 | has a Up^ V l distribution with parameters u2 and ux (Timm, 1975, p. 147). Theorem 2.2 IfY = [ Y l r .., YN}' with Y , ~ Np(//t-, E) independently for i = 1,..., N and if A is a symmetric matrix of rank r then Y ' A Y ~ Wp(r, E, A) with A = E ( Y ' ) A E ( Y ) if and only if A2 = A (Timm, 1975, p. 138). Theorem 2.3 Let Y = [Y^ . . . , Y ^ ] ' with Y , ~ NP(/^,E) independently for i = 1,..., N and let A x and A 2 be two symmetric matrices of ranks rx and r 2 respec-tively. Suppose Y ' A . Y ~ Wp^E, A x ) and Y ' A 2 Y ~ Wp(r2, E, A 2). Then Y ' A X Y and Y ' A 2 Y are distributed independently if and only if A X A 2 :~~ 0. Multivariate Mixed Model ANOVA / 2.3 18 Theorem 2.4: Fundamenta l Mu l t i v a r i a t e Least Squares Theorem Let Y = [Yi,YJV]' where Y,- t = 1,...,N are N independently distributed vectors of P random variables with Y,- ~ Np(/z,-,E). Fit Y with X B + E where X is an N x q design matrix of rank r, (r < q < N), B is a q x P matrix of unknown fixed parameters and E is an N x P matrix of random residuals. Say N — r > P. Suppose E ( Y ) = X B and Var(Y) = 1N ® S where S is a P x P positive definite covariance matrix. Let C* be a vh x r contrast matrix of rank uh, with Vh<r such that all of the uh contrasts in C * B A f A A are estimable and independent. Define St to be ( Y — X B W ) (Y — X B W ) where B W is an estimate o/B when the constraint of null hypothesis H 0 : C * B = M 0 is applied in mini-mization of Tr(Y - X B ) ' ( Y - X B ) . Also, define S e as ( Y - X B N ) ' ( Y - X B N ) where B N is a solution to the normal equations: ( X ' X ) B = X ' Y . One solution is B Q = ( X ' X ) ~ X ' Y where ( X ' X ) - is a generalized inverse of ( X ' X ) . Define Sh to be S t — S e . Then: (a) Se= Y ' [ I - X ( X ' X ) - X ' ] Y ; (b) S h = [ C ^ n - M 0 ] , [ C * ( X , X ) - C * ' ] - 1 [ C * A N - M 0 ] ; (c) Se and S/, are independently distributed; (d) Se ~ WF(JV- r ,E,0) and S/, ~ Wp(^, E ,M^) where the noncentrality parameter M/ t is [ X B - X C * ' ( C * C * ' ) - 1 M 0 ] ' [ X ( X ' X ) - C * ' ( C * ( X ' X ) - C * ' ) _ 1 C * ( X ' X ) " X ' ] x [ X B - X C * ' ( C * C * ' ) - 1 M 0 ] When the null hypothesis H 0 : C * B = M 0 is true, M A = 0 and A, which is given by l ( Y - X B N ) ' ( Y - X B N ) | | ( Y - X B J ' ( Y - X B J | is equal to |S e (S e + S/,)-1| and is distributed as Up5j/e „h where ve = N — r. Theorem 2.4 holds when P = 1. SSCP matrices S/, and S e become hypothesis and residual sums of squares—SS/, and SSe respectively. A ~ Uj Ue^Uh. In practice, H 0 is rejected when F, equal to (SS/l/i'/l)(SSe/i/e)-1, is larger than F^ V e. Multivariate Mixed Model ANOVA / 2.4 19 2.4 M M M H Y P O T H E S I S T E S T I N G To test null hypothesis H 0 : C * B = MQ when a multivariate mixed model is being fitted to data matrix Y, the following general steps are performed: a) two matrix quadratic forms are computed—SSCP matrices § h and Se. b) the value of the test statistic A is computed using A = |S e(S e + S^) - 1| c) the sample value of A is compared to a critical value (for a test of a given size) of the Up „ e i „ f c distribution with H 0 being rejected if A < U*Tta. Matrix S/, is computed from Y using the general formulae provided in the statement of Theorem 2.4: Sh = [ C * B N -M 0]'[C*(X 'X)-C*']" 1[C*A n - M 0 ] = [C*(X'X)- IX'Y-M 0]'[C*(X'X)- 1C*']~ 1[C*(X'X)- 1X'Y-Mo] (2.14) In the second line of Equation (2.14), one particular solution to the normal Equations has been inserted for B Q . Also, (X'X) - 1 has been used for (X'X) - since, in the statement of the model given earlier, X has full column rank. (See Figure 4.) If an omnibus test of the hypothesis of no occasion effects is to be conducted, ( l ' ® C') from (2.13) is used for C* in Equation (2.14). Similarly for an omnibus test of the group by occasion interaction null hypothesis, the contrast matrix (F' ® C') used in Equation (2.12) is substituted for the general contrast matrix C* in Equation (2.14). The SSCP matrices produced when M 0 = 0—denoted as T for the omnibus occasion hypothesis and U for the parallelism hypothesis—are thus: T = Y'[X(X'X) - 1(1 G ® C)] [(l' G ® C ) ( X ' X ) - 1 ( 1 G ® C)] " 1 x [(l'G®C')(X'X)-1X']Y (2.15) and U = Y'[X(X'X)~ 1(F ® C)] [(F' ® C'XX'X)- 1^ ® C ) ] - 1 x [(F'sC'KX'XJ-^X'jY (2.16) Multivariate Mixed Model ANOVA / 2.4 20 To test a null hypothesis involving q (< T — 1) specific independent comparisons among the T occasion effects—a hypothesis which is "less than " the omnibus null hypothesis when q <T — 1—one can replace matrix C in Equation (2.15) with a T x q orthonormal contrast matrix of rank q. Thomas pointed out that without loss of generality, Cq could be taken to be the first q columns of C . The resulting S S C P matrix Tq is: T , = Y ' l x p r t q - ^ i o ® Cf)] [(!'« ® c;)(x'x)-1(iG ® cj]"1 x [ ( l ' G ® C ; ) ( X ' X ) - 1 X ' ] Y (2.17) Likewise, to test a partial interaction null hypothesis, one can substitute a contrast matrix F r of rank r (< G) and /or matrix Cq into Equation (2.15) for F and C respectively giving S S C P matrix U 5 r : U 9 r = Y ' [ X ( X ' X ) - 1 ( F r ® C,)] [(F' r ® C ; ) ( X ' X ) " 1 ( F r ® C , ) ] " 1 x [(F' r ® C ; ) ( X ' X ) - 1 X ' ] Y (2.18) The residual S S C P matrix used in M M M tests of parallelism and occasion hypotheses— denoted hereafter as W — i s different from the S E described in Theorem 2.4. Matr ix W is the multivariate extension of SST-T/FF, the sum of squares used in computing the error term for tests of H 0 g t and H 0 t in mixed model A N O V A when P = 1, and thus represents "interaction variability for subjects and [T] that remains after we remove the contribution due to the interaction variability of [G] and [T] " (Meyers, 1979, p. 203). The formula for computing W is: W = Y ' X ; + ( H X ) ® ( C C ) 9=1 (2.19) where H f f is an ng x ng — 1 orthonormal contrast matrix. Given assumptions (1) and (2) in Section 2.1 about the normality and inde-pendence of distributions of subjects' residual vectors, vec(Eig), if it can be shown that U , T , and W have independent Wishart distributions, then the A test statistic and the U distribution can b e used in conducting valid tests of H 0 g t and H 0 t . Thomas (1983) Multivariate Mixed Model ANOVA / 2.4 21 demonstrated that U 9 r and W are always independent under (1) and (2) as are Tq and W . He further demonstrates that when population covariance matrices for the G popu-lations satisfy a multivariate multisample sphericity condition—Equation (1.2)—Ugr, Tq, and W have Wishart distributions. The details of this argument are now stated as theo-rems. The proofs of these theorems draw upon two lemmas which correspond to theorems 2.2 and 2.3 above. L e m m a 2.5 Let Y be a J x K matrix of random variables with the distribu-tion vec(Y') ~ (vec(M'), V ) where V is a nonsingular JK x JK covariance matrix. Let Ai and A 2 be two symmetric matrices. Then Y'A{Y and Y ' A 2 Y are independent if and only if ( A 1 ® I j ) V ( A 2 ® I J ) =0 (2.20) L e m m a 2.6 Again let Y be a J x K matrix of random variables with the distribution vec(Y') ~ Nj#(vec(M'), V ) , where V is a JK x JK covariance matrix. Let A be a symmetric matrix of rank K. If ( A ® I J ) V ( A ® I J ) = A ® S (2.21) then Y ' A Y has a V/P(K, S , A ) distribution. The distribution is central if and only if A M = 0. Theorem 2.7: Independence of Tq, U g r, and W . Let Y = X B + E as described in Section 2.1 with E t g as the residual submatrix for person i of group g. Suppose vec(E,3) and vec(E,-y) are distributed independently for i ^ t' or g ^ g', and vec(EtJ/) ~ Npj(0, for i = 1,..., ng and g = 1,..., G. DeGne matrix quadratic forms Tq,Vqr, and W as in Equations (2.17), (2.18), and (2.19) respectively. Then: (a) U q r and W are distributed independently for each E f f g = 1,..., G; and (b) Tq and W are distributed independently; Multivariate Mixed Model ANOVA / 2.4 22 If (C'q ® I p ) S i 7 ( C g ® I p) = Vq, a constant qP x qP matrix (q < T - 1) for all g = 1 , . . . , G and if ng = N/G for all g = 1 , . . . , G then Tq and Uqr are distributed independently. Theorem 2.8: Mu l t i v a r i a t e Mu l t i s amp l e S p h e r i c i t y — A Sufficient C o n d i t i o n for the D i s t r ibu t ions of T ? , U g r , and W to be W i sha r t . Let C be an orthonormal contrast matrix of rank T — 1 and let S beaPxP positive definite symmetric matrix. Given the definitions and conditions of Theorem 2.7, if ( C ' ® I P ) E y ( C ® I p ) = Ir_! ® S for g=l,...,G (2.22) then (a) Vqr ~ W p ( g r , E , AGT) and A G T = 0 if H 0 G T is true; (b) Tq ~ W p ( g , E , A r ) and AT = 0 if H 0. R is true; and (c) W ~ W p ( ( r - l ) ( i V - G ) , E ) Proof: A proof of the first statement is given here. Demonstrations of the validity of the second and third conclusions use very similar reasoning. Lemma 2.6 states a sufficient condition for a matrix quadratic form Y ' A Y to have a Wishart distribution. The present proof links fulfillment of the multivariate multisample sphericity condition to satisfaction of the Lemma 2.6 condition. Matrix Cq is a T x q orthonormal matrix of rank q and F r is a G x r contrast matrix of rank r. Equation (2.17) defines the A matrix for matrix quadratic form \Jgr, namely [X{X!X)-l{Fr ® C f ) ] [(F'r ® C j ) ( X ' X ) - 1 ( F r ® C , ) ]- 1 [ (# ® C ; ) ( X ' X ) " 1 X ' ] . The covariance matrix for Y , called V in Lemma 2.6, is Z ) + f f = 1 ( l „ f l ® E g ) . Dimension J in the lemma is P. It is helpful at the outset to simplify the expression for A using Equation Multivariate Mixed Model ANOVA / 2.4 23 (2.2), the definition for design matrix X. One component of A, (X'X) 1 becomes: ((EX»*)(E+= (E+K-'])®ir - D"1 ® I r where D _ 1 = diag[nlf... ,nff]. In the more compact notation, A is [Fr(Fj.D_1Fr)-1F|.] ® C^C'j. Substituting for A and V in the left side of Equation (2.21) produces { [ F ^ D - * , ) - ^ ® ( C f C y ® I P J C D - 1 ® E , ) x {[F ^ F ' . D - ^J-^;] ® (cg;) ® I P } after several straightforward simplifications using the properties of the direct sum and Kronecker product. This can be transformed to give: ® { [ ( C g C y®Ip ] E j ( C g C i)®Ip]} which reduces to [ F ^ D - ^ r V j ® {[(C . c y ® I P ] E f f[(C , c y ® Ip]}. (2.23) Assume that the multivariate multisample sphericity condition given in Equation (2.22) holds. Then the component on the right side of expression (2.23) can be rewritten: [ ( c , c y ® i P ] E , [ ( c , c y ® ip] = ( c f ® ip) [ ( c ; ® i P ) E f l ( c , ® i P)] ( c ; ® i P) = (cg®ip)(i3®s)(c;®iP) = ( C , C ; ) ® E So when the MMS condition holds, (A ® Ip)V(A ® Ip) = [ F , ( F ^ F , ) " ^ ® (C fC f ) ® E = A ® E . Therefore, TJqr has a Wishart distribution. The rank of A is rank[Fr(F'rD~1FP)~1F'r] x rank(CgC^) = rq so the distribution is WP(rq, E , A G r ) . The expected value, M , of Y Multivariate Mixed Model ANOVA / 2.5 24 is X B . Recall that the Lemma 2.6 condition for centrality is A M = 0. Substituting the earlier unsimplified expression for A into this condition gives [ x p r x r ^ F , ® c y ] [(F'R ® c j ) ( x ' x ) - 1 ( F R ® c , ) ] " 1 [(F; ® C;) (X'X)- 1 X']XB = 0 Using the fact that (XX' ) _ 1 (XX' ) = I, it becomes clear that if (Pj. ® C^)B = 0 -that is, if H 0 g t as expressed in Equation (2.12), is true-then the Wishart distribution for TJqr is central. The fact, just established, that Vqr, Tq, and W have Wishart distributions together with Theorem 2.1 implies that the U distribution can be used in testing H 0 G T and H 0 T . When the MMS condition is satisfied, the omnibus tests of H 0 g t and H 0 T are valid. Tests of occasions null hypotheses involving q < T — 1 orthonormal contrasts are also valid, as are tests of partial interaction hypotheses using fewer than {G — 1)(T — 1) contrasts. 2.5 U S E O F T H E M M M T E S T I N G P R O C E D U R E The MMS condition is actually a pair of specifications concerning the forms of G population covariance matrices: (a) (C ® Ip)E S (C ® Ip) = B for all $ = 1,... G; and (b) S = I r_! ® E Thomas (1981, 1983) suggested that when it cannot be assumed a priori that Ej = • • • = E G , a preliminary test of the equality in (a) using Box's M criterion be performed with the matrices ( C ® Ip)S ?(C ® Ip) g = 1,..., G where is the unbiased estimator of E f f for group g. Define S as ^ l _ G Y ^ = i ( n g ~ *)S?- Box's M, for this purpose, is given by G (N -G) In | (C'®Ip)S(C®I P)| ~ X X l n \ {C ®lp)Sg{C®IP)\. Multivariate Mixed Model ANOVA / 2.6 25 Define the constant m as G 2(PqY + SPq - 1 (y. 1 _ 1 \ 6(Pq + 1)(G - 1) ng - 1 N - G j ' 1 q p g + l j ^ - l j \ g = i When the population matrices are indeed equal, mM has a x2{d) distribution with d = {G - l)Pq{Pq + l)/2 (Huynh & Mandeville, 1979). If the (a) equality is tenable, a test concerning the structure of S can be performed. Thomas (1981, 1983) developed a likelihood ratio test of the hypothesis as-sociated with (b) above using a generalized form of Mauchly's (1940) criterion. With S denned as before, let sk be the A;th (k = 1,.. .q < T — 1) P x P diagonal submatrix of (N - G) (C'q ® I P)S(C q ® The statistic M given by ^ ( ^ I E T -ln|(C q®I P)S(C q®I P)|) has a x2 distribution when the hypothesis for (b) is true, with P(q — l)(Pq + P + l)/2 degrees of freedom. When both (a) and (b) are found to be satisfied, it is safe to perform MMM tests of the parallelism and occasions hypotheses. Thomas (1981, 1983) pointed out that in the event of either part of the MMS condition not being satisfied, the Bock/ Timm MANOVA procedure cannot be used as a replacement if the Sff's are not equal. Note that M and M can be used as descriptive statistics to roughly index degree of departure from specifications (a) and (b) respectively in a given experiment. 2.6 T H E S T R U C T U R E O F C O V A R I A N C E M A T R I C E S W H I C H S A T I S F Y T H E M M S C O N D I T I O N If assumptions (1) and (2) hold and the covariance matrix of Y has the property that (C' ® Ip)Sj7(C ® Ip) = I T_! ® S for all g = 1,... G (where S is a P x P positive definite matrix), then MMM tests of omnibus parallelism and occasion null hypotheses are valid. What is the form of a covariance matrix which has this property? Multivariate Mixed Model ANOVA / 2.7 26 Thomas (1981,1983) demonstrated that a covariance matr ix for G groups satisfies this mult ivar iate mult isample sphericity (MMS) condit ion when each S 3 (g = l,...,G) has a structure related to that of a Huynh and Feldt (1970) Type H matr ix . Let R and R ' be factors of S in the above expression. Define V f f as ( I r ® R ~ 1 ) S f f ( I r ® R - 1 ) . The MMS condit ion can then be expressed as: ( C C ® I P ) V , ( C C ® I P ) = ( C C ® Ip) for al l g = 1,..., G. (2.24) That is, when the G V f f matrices are al l generalized inverses of ( C C ® Ip ) , the MMS condit ion holds. The fol lowing theorem presents the general form of a Jjg which makes ( I j ® 'R~i)Jjg(Ix ® R - 1 ) such a generalized inverse. Theorem 2.9 Let 9 be a PT x PT covariance matrix such that ( C ® I p ) ^ ( C ® Ip) = I T - I ® S where S is a positive definite mat r ix which can be expressed as R R ' . Then 9 = ( I r ® R ) V ( I r ® R ' ) where V i s a positive definite matrix whose P x P submatrices k, I = 1 , . . . T have the form IP + Kk + K'{ for k = I, and Kk + K't for k ^ I with Kk k = l,...,T being arbitrary P x P matrices. If K.j. = /Cfclp for k = 1..., T, V has the form O ® Ip where 9> is a mat r ix of the type Huynh and Feldt (1970) cal l Type H . 2.7 V I O L A T I O N S O F T H E M M S C O N D I T I O N Departures f rom MMS can take three basic forms and, wi th in each, var iat ion in degree is possible. For a given set of G covariance matrices, suppose each matr ix ( C ® I p ) S f f ( C ® I p ) has the form I T - I ® E£ g = 1,... , G , but that at least one of the S * ' s is different f rom the others. Th i s situation could be called a v io lat ion of type A . O n the other hand, the matrices ( C ® I p ) S 1 ( C ® I p ) , . . . , ( C ® I p ) S G ( C ® Ip ) may al l be equal to some H, but without E having form I T - I ® ^« Such a violat ion could be called type B . The most Multivariate Mixed Model ANOVA / 2.8 27 likely occurence in real world experiments—a type C violation—is that at least one of the matrices ( C ® Ip)E f f(C ® Ip) does not have the equal-diagonal-block form and at least one member of the set { ( C ® I p j E ^ C ® Ip),.. . , ( C ® Ip)S G (C ® IP)} is different from others in the set. 2.8 R E V I E W O F R E L A T E D R E S E A R C H The four studies discussed here provide information about the effect of departures from the multisample sphericity condition, Equation (1.1), on the Type I error rates of mixed model ANOVA tests of omnibus parallelism and occasions null hypotheses in experiments which use a groups by occasions repeated measures design with P = 1. In addition, some empirical results are presented concerning the relative power of mixed model ANOVA and MANOVA tests of H 0 g t and H 0 T . Collier, Baker, Mandeville, and Hayes (1967) studied the actual Type I error rates for tests of H 0 g t and H 0 t by conducting Monte Carlo simulations of 15 pairs of experiments with the following specifications: a) in all experiments, subjects in three groups of equal size provided scores on four measurement occasions; b) the group size was five in one experiment of each pair, and 15 in the other; and c) in all experiments, both H 0 g t and H 0 t were true and subjects' scores were dis-tributed independently with multivariate normal distributions. A pool of fifteen covariance matrices was used in the study. In any given experiment, all populations sampled had the same covariance matrix. All violations were type B in the terminology of the previous section. Each matrix was generated using one of three sets of variances (homogeneous, and heterogeneous, to two degrees) and one of five patterns of interscore correlations. Box's (1954) e parameter was calculated for each matrix Ey in the pool: [tr C 'EyC] 2 € ~ (T - l)tr (C'EyC) 2 Multivariate Mixed Model ANOVA / 2.8 28 The e value for Ey serves as an index of the degree to which C'EyC departs from the form A I T - I - The e values for the matrices ranged from 1.00 to 0.45. Table 1 presents some findings of Collier et al. for mixed model tests. Entries in the table are numbers of Type I errors in N r e p s = 1000 replications of an experiment. Figures marked with an asterisk exceed N r e p s x a n o m by twice the standard error for that proportion, 2 Y / N r e p s Q ! n o m ( l — a n o m ) - (Values are 13.8 and 6.2 for nominal alphas of 0.05 and 0.01 respectively.) It is apparent that when a type B violation is present, mixed model testing procedures for the omnibus parallelism and occasion null hypotheses are quite liberal, particularly for experiments in which e < 0.64. Generally, it appears that Type I error rates increase for tests of H Q g t and H 0 x as e decreases. Tests of H 0 g t appear to be somewhat more liberal than the corresponding tests of H0x. Type I error rate appears to have no consistent relationship to group size. Noe (1976) extended the investigations of Collier et al. (1967) in three di-rections of particular interest for the present study by: (a) modelling violations of types A and C; (b) drawing samples of unequal sizes in a given experiment; and (c) examining performance of multivariate tests—Hotelling's T 2 for H 0 x and Rao's F approximation statistic for H 0 g t —as well as mixed model ANOVA tests. Noe used a three group by four occasion design, as Collier et al. had done, and set total sample size at 15 in all experiments. Matrices used in the simulations were multiples of 12 of the 15 matrices Collier et al. had used. A condition of equality of matrices across groups was modelled as were two types of intergroup heterogeneity: (a) E 2 = 2Ej and E 3 = 3Ei; and (b) E x = E 2 and E 3 = 3E X Multivariate Mixed Model ANOVA / 2.8 29 e values of covariance matrices 1.00 1.00 0.90 0.74 0.72 0.64 0.64 0.62 H0 1000 x nom. a Subjects per sample = 5 GT T 50 52 44 65* 75* 78* 74* 78* 98* 10 14 6 12 31* 28* 25* 22* 28* 50 46 51 56 59 67* 70* 64* 73* 10 10 10 11 14 23* 20* 15 30* Subjects per sample = 15 GT T 50 54 50 62 66* 88* 72* 75* 81* 10 7 5 17* 27* 32* 19* 26* 33* 50 65* 58 62 65* 56 86* 72* 73* 10 15 9 13 18* 19* 28* 30* 27* Results from Collier et al. (1967, pp. 346, 348). T A B L E 1 Number of Type I Errors in 1000 Replications for Mixed Model Tests, Related to Matrix e Values. Multivariate Mixed Model ANOVA / 2.8 30 e values of covariance matrices 0.62 0.60 0.59 0.55 0.53 0.49 0.45 H0 1000 x nom. a Subjects per sample = 5 GT T 50 84* 88* 98* 92* 79* 93* 104* 10 34* 31* 32* 33* 35* 36* 44* 50 65* 77* 84* 29* 73* 101* 105* 10 18* 32* 25* 43* 29* 38* 56* Subjects per sample =15 GT T 50 85* 101* 87* 101* 92* 104* 115* 10 33* 37* 37* 44* 33* 42* 61* 50 74* 83* 84* 83* 79* 92* 95* 10 18* 23* 33* 23* 24* 33* 37* Results from Collier et al. (1967, pp. 346, 348). T A B L E 1 continued. Multivariate Mixed Model ANOVA / 2.8 31 where Hlt E 2, and E 3 are the matrices used in a given experiment. Matrices were con-strained by the condition 1/15 ^ 3 = 1 ng^g = ^ where E is a matrix in the basic pool. (It should be noted that Noe discussed his heterogeneity violations in terms of inequality of these "raw" population matrices, but constructed his matrix sets in such a way that the true homogeneity condition—Equation (1.1)—was violated in the same ways.) Noe used five sample size arrangements: (a) nx = n 2 = n 3 = 5; (b) nx = 7, n 2 = 5, n 3 = 3; (c) nx = 3, n 2 = 5, n 3 = 7; (d) nx = 1, n 2 = 5, n 3 = 9; and (e) nx =9, n 2 = 5, n 3 = 1. Only eight of the 15 possible matrix-sampling arrangement combinations were used in the study, however. For example, type B violations were not simulated with unequal sized samples. A portion of the data most relevant to the present study is reported in Table 2. The results shown here are for experiments in which heterogeneity condition (1) above and sampling arrangements (a), (b), and (c) were used. Entries are numbers of Type I errors per 1000 replications; a t t o m for the tests reported was 0.05. Figures marked with an asterisk indicate that the corresponding a a c t u a i levels are outside the confidence interval 0.05 ± 2 (0.0049). This confidence band is shorter than that of the Collier et al. study because N r e p s was 2000 in Noe's study. To simplify discussion of Noe's results in experiments in which sample sizes were unequal and violations were of types A or C, the terms positive sampling and negative sampling are introduced. (These terms will be used in discussion of such experiments in the present study also.) Suppose populations 1,..., G, sampled in an experiment using a repeated measures design, have covariance matrices Ex,...,E(-r. Calculate determinants |(C'®Ip)E1(C®IP)|,..., |(C'®Ip)EG(C®Ip)|. Positive sampling is defined as drawing Multivariate Mixed Model ANOVA / 2.8 32 the largest sample from the population giving the largest of these determinants. Negative sampling involves selecting the largest sample from the population giving the smallest determinant. Noe found that when a violation of type A was modelled and negative sam-pling was used, mixed model ANOVA tests of H 0 g t were very liberal—more liberal than when samples were equal in size. Multivariate tests of this hypothesis were also liberal but less so. Mixed model and multivariate tests of H 0 x gave similar Type I error rates—above a n o m — b u t were less liberal than multivariate tests of H 0 c T. When type A violations and positive sampling were modelled, mixed model and multivariate tests of HoGT became conservative, to approximately the same degree. Values of |a a c t u a l — a n o m l w e r e small for mixed model and multivariate tests of H0x. Values of a a c t u a i — a n o m w e r e v e r v positive for mixed model tests of H 0 G J when type C violations and negative sampling were modelled. As e decreased, a a c t u a i — ^ n o m increased. Multivariate tests of H 0 g t were also liberal but less so. Decreasing e did not lead to higher Type I error rates for the latter tests. When type C violations and negative sampling were modelled, mixed model tests of H 0 t were liberal but gave lower Type I error rates than mixed model tests of the parallelism hypothesis. Again, increases in ar a c t I i a i were associated with decreases in e. Multivariate tests of the occasions hypothesis were liberal as well, but a a c t u a i tended to be closer to oenom than for mixed model tests of H0x. Changes in e appear not to have altered ^ a c t u a l f°r multivariate tests. With positive sampling and type C violations, mixed model tests of HoGX were slightly conservative for larger values of e (0.90 and 0.72) and somewhat liberal for c = 0.45. For other values of e, values of a a c t u a i were close to a n o m. Multivariate tests of H 0 g t were conservative in all experiments with e < 0.90. The value of or a c t u a i was not affected by decreasing e. Mixed model tests of H 0 x were liberal for e < 0.74, and, as e decreased, a a c t U a l — a n o m increased. Differences between actual and nominal Multivariate Mixed Model ANOVA / 2.8 33 e values of covariance matrices 1.00 1.00 0.90 0.74 0.72 0.64 V. T. A A C C C C H0 Test GT MM 55 66* 55 74* 64* 88* Rao F 65* 66* 46 61* 63* 64* T MM 50 53 58 77* 58 75* T2 62* 58 58 58 54 58 nx = 7 : n2 — 5 : n 3 = 3 GT MM Rao F 120* 104* 126* 116* 128* 110* 143* 111* 142* 104* 155* 126* MM JI2 60* 66* 62* 66* 62* 67* 80* 70* 86* 63* 82* 64* = 3 : n 2 = 5 : n 3 = 7 GT MM 26* 20* 34* 46 38* 45 Rao F 28* 28* 30* 28* 29* 25* T MM 42* 51* 56* 77* 60* 78* rp2 40* 52* 46* 60* 48 47 V. T. refers to violation type. Results from Noe (1976, pp. 34-36 and pp. 39-41). T A B L E 2 Number of Type I Errors per 1000 Replications for Multivariate and Mixed Model ANOVA Tests, Related to Sampling Arrangement and Matrix e Value. Multivariate Mixed Model ANOVA / 2.8 34 e values of covariance matrices 0.62 0.62 0.60 0.53 0.49 0.45 V. T. C C C C C C Hft Test rii = n-2 = n$ = 5 GT MM 86* 98* 102* 100* 116* 104* Rao F 59 60* 60* 70* 62* 56 T MM 78* 74* 90* 89* 97* 110* 59 56 62* 57 61* 52 = 7: n 2 = 5 : n 3 = 3 GT MM 160* 148* 143* 175* 169* 182* Rao F 124* 112* 122* 124* 124* 113* T MM 90* 82* 96* 91* 105* 112* rp2 70* 52 64* 62* 64* 64* n2 — 5 : ri3 = 7 GT MM Rao F 46 28* 50 24* 61* 27* 48 26* 58 27* 80* 32* MM rp2 76* 52 74* 40* 70* 50 86* 51 95* 51 103* 50 V. T. refers to violation type. 1000 x a n o m = 50. Results from Noe (1976, pp. 34-36 and pp. 39-41). T A B L E 2 continued. Multivariate Mixed Model ANOVA / 2.8 35 Type I error rates were greater for mixed model tests of H 0 x than for mixed model tests of HoGT. Multivariate tests of H 0 t gave aactual values which were close to their respective a n o m values.This was true for all values of e. Values of |a a c t n ai — a n o m| were smaller for multivariate tests of H 0 x than for mixed model tests of this hypothesis. Noe's results for mixed model tests in experiments in which type B violations were modelled with equal sample sizes agreed in part with those of Collier et al. As e decreased, a a c t u a i — anom increased for tests of both within-subjects hypotheses, but tests of the parallelism hypothesis were not much more liberal than tests of the occasions hypothesis. As expected, multivariate tests of H 0 g t and H 0 x were robust. When violations of type A were modelled with samples of equal size, Type I error rates for mixed model and multivariate tests of both hypotheses were not very dif-ferent from a n o m. When violations of type C were modelled and sample sizes were equal, mixed model tests of H 0 g x were somewhat liberal and Type I error rates increased as e decreased. Mixed model tests of H 0 x exhibited the same trend but were generally not as liberal. Multivariate tests of both hypotheses were liberal but less so than their mixed model counterparts. Decreases in e did not greatly alter Type I error rates for the multi-variate tests. A study by Rogan, Keselman, and Mendoza (1979) provides a comparison of mixed model ANOVA and multivariate tests with respect to Type I error rate and power. Twenty variations were simulated of an experiment in which 39 subjects in three equal-sized groups were measured on four occasions. Ten sets of three population covariance matrices, {Si, E 2, S3}, were generated to represent four conditions with respect to departure from multisample sphericity requirements: a) no departure; b) diagonal forms Ajl, A2I, and A3I for the matrices C'EiC,C'E2C, and C'E3C respectively, with Xlf A2, and A3 unequal—a type A violation in the terminology of Section 2.7; Multivariate Mixed Model ANOVA / 2.8 36 c) equality of the three matrices C'E^C'SaC, and C'E3C—that is C'EjC = C'E2C = C'E3C = E, coupled with departure of E from the diagonal form of AI — a type B violation; and d) inequality of the matrices C'EjC, C'E2C, and C'E3C together with departure of each from a diagonal structure, a violation of type C. Degree of departure from the equality requirement ( C'E^C = S for g = 1,..., G) was indexed with a modified form of Box's (1949) criterion M given by G Af * = G In |C'EpooledC| - |C'SffC|, ! / = l where Sp,,^ is the pooled covariance matrix for the populations involved—Epooie(j = l/GY^=i Eg (See Rogan et al., 1979, p. 279.) When the equality requirement is satisfied, M* = 0 regardless of the form of the pooled matrix. The index e was used to quantify pooled matrices' degrees of departure from a diagonal form. Values of 0 and 1.37 were used for Af*, and values of 1.00, 0.96, 0.75, 0.57, and 0.48 were used for e. Two distributions of subjects' scores were used—multivariate normal and multivariate chi-squared, giving the 20 conditions. Table 3 summarizes findings of Rogan et al. on actual alpha levels for four sets of tests—mixed model ANOVA tests of H 0 g t and H 0 t , Hotelling T2 tests of H0x, and a likelihood ratio test using Wilks' A for H 0 G T — f o r the nine experiments in which a violation was modelled and the subjects' scores had a multivariate normal distribution. The entries in the table are numbers of Type I errors per 1000 replications when the tests were conducted with a n o m = 0.05. Again, the findings suggest that with B and C violations, when scores have a multivariate normal distribution and when sample sizes are equal, mixed model ANOVA tests of H 0 g t and H 0 t become increasingly liberal as £ decreases below approximately 0.75. Multivariate tests of H 0 g t and H 0 x appear to have provided smaller values of | o t a c t u a l —a n o m | than mixed model tests in all experiments with more extreme B and C violations ( e < Multivariate Mixed Model ANOVA / 2.8 37 Form of covariance matrices M* 0.00 0.00 0.00 0.00 1.37 1.37 1.37 1.37 1.37 € 0.96 0.75 0.57 0.48 1.00 0.96 0.75 0.57 0.48 V. T. B B B B A C C C C Test Parallelism hypothesis MM 57 74 81 110 58 56 75 90 110 A 50 48 36 62 54 55 58 50 64 Occasions hypothesis MM 49 59 66 98 50 51 76 82 76 T2 46 48 44 46 57 55 65 55 49 V. T. = violation type (See text.) 1000 x a n o m = 50 Results from Rogan et al. (1979, pp. 381-382). T A B L E 3 Number of Type I Errors per 1000 Replications for Multivariate and Mixed Model Tests, Related to Form of Matrices. Multivariate Mixed Model ANOVA / 2.8 38 0.75). As in the Collier et al. study, mixed model tests of H 0 q t appear to be more liberal than tests of H 0 X when a violation of type B occurs. The same ordering for test size appears to hold for C violations. The type A violation modelled by Rogan et al. did not seem to affect Type I error rate adversely for either mixed model (or multivariate) test. Rogan et al. (1979) compared the power of mixed model ANOVA to that of multivariate procedures in tests of H 0 g t and H 0 t . One set of population mean vectors representing presence of interaction and an occasion main effect was used for tests of H 0 G T , while a second set representing on occasion effect only (of the same size as for the former set) was used for examining tests of H 0 X . Table 4 summarizes the most relevant findings. When the multisample sphericity requirement was satisfied for a set of population covari-ance matrices, and subjects' sets of scores had multivariate normal distributions, mixed model ANOVA tests of H 0 g t and H 0 X had larger power values than multivariate tests of these hypotheses. Further, in all but two of the experiments in which the matrices departed in some way from multisample sphericity, the mixed model tests were more powerful than their multivariate competitors for both within-subjects hypotheses. A consistent decrease in power for mixed model ANOVA tests occurs as e decreases regardless of the value of M. The general trend is true for multivariate tests except for a consistent anomaly when matrices were used which gave an e value of 0.57. Data tables were not presented for the experiments using the skewed distri-bution. Rogan et al. report, however, that the effects of the non-normality were: (1) slight increases in Type I error rates and power in the mixed model and T2 tests of H o T ; (2) slight decreases in Type I error rates and slight increases in power for mixed model and multivariate tests of H 0 G T . In addition, Rogan et al. examined the efficacy of using sequential testing strategies. One such strategy was to test for violation of multisample sphericity using Box's Multivariate Mixed Model ANOVA / 2.8 39 Form of covariance matrices M* 0.00 0.00 0.00 0.00 1.37 1.37 1.37 1.37 1.37 € 0.96 0.75 0.57 0.48 1.00 0.96 0.75 0.57 0.48 V. T. B B B B A C C C C Test Parallelism hypothesis MM 41 41 40 40 41 42 40 39 39 A 34 31 33 30 38 37 31 34 31 Occasions hypothesis MM 72 65 63 61 72 73 66 62 59 T2 66 59 67 58 70 68 61 66 56 V. T. = violation type (See text.) Power values are table entries divided by 100. Results from Rogan et al. (1979, pp. 381-382). TABLE 4 Power of Testa of H 0 q t and H 0 X Related to Form of Matrices. Multivariate Mixed Model ANOVA / 2.8 40 M and the Mauchly (1940) criterion, and to follow a rejection of the multisample sphericity hypothesis with a multivariate test of the parallelism/ occasions hypothesis rather than a mixed model ANOVA test. In general, sequential testing provided little improvement in Type I error rates and power over uniform adoption of a particular form of test of H Q G T or HoT because of high sensitivity of the preliminary tests to non-normality as well as to departures from multisample sphericity—particularly when e < 0.75 or M ^ 0. Mendoza, Toothaker, and Nicewander (1974) examined the Type I error rates and power of multivariate and mixed model ANOVA tests of H Q g t and H 0 t by simulating 18 similar experiments in which 27 subjects in three equal-sized groups provide scores on four measurement occasions. Three different covariance matrices with e values of 1.0, 0.54, and 0.51 were used. (The latter two matrices had simplex structures.) In each simulated experiment, the three groups' covariance matrices were identical; type B violations were modelled. In half the experiments the distribution of subjects' scores was multivariate normal and in the other half the distribution was skewed. The Roy largest root criterion was the test statistic used in the multivariate test of H 0 g t . The multivariate procedure used with H U x was the Hotelling one sample T2 test. Nominal alpha was 0.05 in all simulations. Table 5 summarizes selected results of this study pertaining to Type I error rates. The figures are numbers of Type I errors per 1000 replications and the values marked with an asterisk are those which are more than twice the standard error of the proportion above or below nominal alpha. For both multivariate normal and skewed score distributions, the mixed model ANOVA tests of H 0 g t and H 0 x are liberal for the type B violations modelled. Skewness in the score distribution appears to affect ar^^ai levels for mixed model tests. Tests of HoGX, as in the Rogan et al. study, appear to be less liberal when the score distribution is skewed than when it is normal. In contrast to the Rogan et al. findings, mixed model tests of H 0 x in the Mendoza et al. study are also less liberal when the distribution is skewed. Multivariate Mixed ModeJ ANOVA / 2.8 41 e values for matrices 1.00 0.54 0.51 H0 Test Normal distribution GT MM Roy 45 49 103* 47 92* 56 T MM <T>2 42 50 89* 44 90* 49 N " reps 3000 1000 2000 Skewed distribution GT MM Roy 39* 32* 83* 45 81* 47 T MM 51 92* 80* 90* 80* 55 N A'reps 1000 1000 1000 Results from Mendoza et al. (1974, p. 173) TABLE 5 Number of Type I Errors per 1000 Replications for Multivariate and Mixed Model Tests, Related to Score Distribution Form and Matrix e Vaiue. Multivariate Mixed Model ANOVA / 2.8 42 The T 2 test for H 0 X becomes liberal when the multivariate normality assump-tion is violated. The multivariate test of H Q g t has a slight tendency to be conservative particularly when the scores have a skewed distribution. In the power studies, two sets of group centroids were used to represent large and small interoccasion differences. Table 6 presents results concerning test power from the six experiments in which interoccasion differences were relatively small. Violation of the multivariate normality assumption does not greatly affect any test's power. Departures from multisample sphericity requirement of type B lead to a decrease in power for mixed model ANOVA tests but for multivariate tests, an increase in power can occur when c < 1. When the requirement of multisample sphericity is met, the mixed model ANOVA tests of H 0 g t and H 0. R are more powerful than the multivariate tests used here. When the requirement is not met, the latter can be more powerful than the conventional F tests for these hypotheses. In all six experiments using centroids which represented larger interoccasion differences, the mixed model ANOVA test of H 0 g t was more powerful than the multivariate test of this hypothesis. Thus the Rogan et al. and Mendoza et al. studies both suggest that when the multisample sphericity condition is met (in experiments in which P = 1), mixed model ANOVA tests of within-subjects hypotheses are more powerful than corresponding multivariate tests of these hypotheses. A parallel result for experiments with P > 2 would mean that MMM testing of parallelism and occasions hypotheses would be more powerful than MANOVA testing when the MMS condition holds. If, however, there is a carryover to experiments with P > 2 of Noe's Type I error rate comparisons between mixed model tests of H 0 g t and H 0 X and their multivariate competitors, the MMM testing procedure would seem less desirable. The results of the studies reviewed suggest several conjectures about Type I error rates for MMM tests of the parallelism and occasion null hypotheses in experiments in which P > 2. a) When a violation of type A occurs and sample sizes are equal, Type I error rates Multivariate Mixed Model ANOVA / 2.8 43 e values for matrices Hn Test 1.00 0.54 0.51 GT MM Roy 55 49 Normal distribution 42 74 48 58 MM rp2 N r e p s 48 44 3000 37 66 1000 43 50 2000 GT MM Roy 54 50 Skewed distribution 46 77 48 64 MM j>2 N reps 48 36 1000 41 71 1000 Results from Mendoza et al. (1974, p. 173) Power values are table entries divided by 100. 40 51 1000 T A B L E 6 Power of Multivariate and Mixed Model Tests Related to Score Distribu-tion Form and Matrix e Value. Multivariate Mixed Model ANOVA / 2.8 44 for tests of the parallelism and occasions hypotheses will not depart greatly from a„om values. b) When a violation is type B, and samples are equal in size, tests of H 0 g t and H 0 T will be liberal. As the structure of (C ® Ip)S p o oi e (j(C ®Ip) becomes more different from the form Ij-i ® S (for some S), Qr a c t n ai — cr n o m will increase for tests of both hypotheses. Tests of the parallelism hypothesis will generally be more liberal than tests of the occasions hypothesis. For tests of within-subjects hypotheses, a a c t u a i will not be affected greatly by changes in sample size. c) With type C violations and equal sample sizes, a a c t n aj will exceed a n o m for tests of H 0 g t and H 0 t and or a c t n ai - a n o m will increase for each as the structure of (C ® Ip)S p o oi e (j(C ® Ip) becomes more different from the form ® £ (for some S). Tests of the parallelism hypothesis will be slightly more liberal than tests of the occasions hypothesis, particularly with severe departures from the form I ® S. d) When covariance matrices exhibit a type A violation and negative sampling occurs, M M M tests of H 0 g t will be very liberal—more so than when samples are of equal size. M M M tests of H 0 T will be liberal also, but a a c t u a i values will be lower than for corresponding tests of parallelism. e) When a type A violation occurs and sampling is positive, M M M tests of H 0 g t will be conservative. For tests of H 0 t , values of |ccactuai — a n o m| will be small. f) When a violation is type B, and sampling is negative, tests of H 0 q t will again be very liberal—more liberal than with samples of equal size. When (C' ® Ip)E p o oj e d(C ® Ip) has a structure that is very dissimilar to that of Ir-i ® S (for some S), oractuai — ctnom will be greater than when it closely resembles the equal-diagonal-blocks form. Multivariate Mixed Model ANOVA / 2.8 45 Tests of H 0 x will also be liberal, but less so than tests of H Q G T . T h e above-mentioned trend for a a c t n a l — anom w i U ^ e exhibited for tests of the occasions hypothesis. g) When a type C violation occurs and sampling is positive, tests of H o G T will be conservative if ( C ® I p ) E p o o i e < i ( C ® I p ) has a structure that is quite similar in form to I x - i ® S (for some S ) . Tests of H 0 x will be liberal when ( C ® I p ) E p o o i e d ( C ® Ip) has a structure that is very dissimilar to that of I T - I ® S (for some S ) . Differences between a a c t u a i and a n o m will be greater for the latter tests under the same conditions. T h e following chapter outlines procedures used to provide information related to these conjectures. CHAPTER 3 Methodology The effect of three types of violation of the MMS condition on Type I error rates for tests of H 0 g t and H 0 X was assessed in this study. This chapter contains a description of the Monte Carlo simulation procedures used, covering (a) the background conditions and limitations of the investigation, (b) types of violation used, and (c) details of the Monte Carlo data generation algorithms. Each Monte Carlo replication involved (a) simulating the selection of G in-dependent samples of size ng, g = 1,..., G with each of the G matrices containing scores for P variables on T occasions; and (b) performance of omnibus MMM tests of the null hypotheses on the resulting data. 3.1 C O N D I T I O N S U N D E R W H I C H E F F E C T S O F D E P A R T U R E F R O M M M S W E R E S T U D I E D The following factors, among others, could shape the effects of violation of the MMS condition: (a) the degree to which assumptions of multivariate normality and independence are met for distributions of subjects' scores; (b) the number of variables on which subjects provide scores on each measurement occasion; (c) the number of measurement occasions; 46 Methodology / 3.2 47 (d) the number of populations from which samples of subjects are drawn; (e) the ratio of mean sample size to number of variables, N/(GP), where N is the total number of subjects in the G samples drawn; and (f) the relative sizes of the G samples. Taking into account available computing resources, it was decided to confine attention here to points (d), (e), and (f). In all simulations , subjects' data matrices— composed of scores on P = 2 variables on each of T = 3 measurement occasions—were independently distributed with multivariate normal distributions. Eight background conditions were used in the study. A background condition is defined here as the product of selecting a value or level for each of three factors over which a researcher planning a real world experiment would have some control: (a) G—values of two and three were used; (b) N/(GP)—values of six and 12 were used; and (c) relative sizes of groups—sample size ratios of 1:1 and 1:2 were used when G was two and 1:1:1 and 1:1:2 were used when G was three. The sample size combinations used were 12:12, 8:16, 24:24, and 16:32 when G was two, and 12:12:12, 9:9:18, 24:24:24, and 18:18:36 when G was three. The modelling of a type A or C violation involved use of two different covariance matrices, say XJT and S N , in generating data for different groups. Under background conditions in which samples were of unequal size, pairs of experiments were conducted to determine whether or not Type I error rate was different under positive and negative sampling. (See Section 2.8 for definitions of these terms.) 3.2 V I O L A T I O N S O F T H E M M S C O N D I T I O N M O D E L L E D I N T H E S T U D Y As indicated in Section 2.7, violations of the MMS condition can be classified as type A, B, or C. Violations can be roughly indexed as to relative severity using modified forms of Methodology / 3.2 48 Box's M and the transformed generalized Mauchly criterion, M. Pairs and triples of pop-ulation covariance matrices were generated to model violations of each type and, within each type, different levels of severity. The set of 23 matrices used in forming these pairs and triples, {EA,..., Ew}, 1S displayed in Appendix A. Also included in this appendix is a brief discussion of the procedures used in producing and selecting them. Type A Violations: Two Group Case Matrix pairs {ED,EE}, {SF,SG}, and {EH,ET} were used to provide the following situation: ( C a l j E ^ C a l ) = 1 ® EJ (C'®I)E2(C®I) = I®S£ E * ^ E ; A modified form of Box's M, M*, was used to index the degree of difference of I® EJ and I ® E*>, for each of these pairs. (See Section 2.8.) The pooled population matrix, I ® E*, given by I®EJ + --- + I ® E G _ was formed (with G = 2) and the formula G M* = G In |I ® E*| - ^ 2ln I 1 ® m 3=1 was evaluated. M* values for the three pairs were 0.3, 1.3, and 3.9 respectively. Each of ED,...,Ej had an identical set of three 2x2 submatrices for di-agonal elements. In each matrix pair, the Eff giving the smaller value of |I ® E*| could be characterized as indicating (a) correlations larger than 0.6 across occasions for each dependent variable considered individually; and (b) intervariable transoccasion correla-tions in the 0.2 to 0.3 range. The matrices giving the larger determinants had (a) lower within-variable transoccasion correlations (< 0.4); and (b) very low (< 0.1) intervariable interoccasion correlations. Methodology / 3.2 49 Type A V io l a t i ons : Three G r o u p Case Matrix triples {Ej,ED,EE}, {Ej,EF,EG}, and {Sj,^,^} with M* val-ues of 0.3, 1.3, and 3.9 respectively were used to provide the following situation: ( C a i j E ^ c ® ! ) = I ® E ; ( C ® l ) E 2 ( C ® l ) = I®E?, ( C ® l ) £ 3 ( C ® l ) = I®E*. EJ, E2, and E 3 are unequal. The matrix common to all triples, Ej, is the pooled matrix for each pair of matrices listed for the G = 2 case. In each simulated experiment in which sample sizes were in the proportion 1:1:2, a smaller sized sample was drawn from the population whose covariance matrix was E p o oi e (]. This permitted simulation of sampling methods parallel to those of two-group experiments in which samples were of different sizes. Type B V io l a t i ons : T w o G r o u p Case Matrix pairs {EK, EK},..., {E0, E 0} were used to provide examples of the following situation: ( C ^ ^ E ^ C ® ! " ) = ( C ' ® I ) E 2 ( C ® I ) = S H ^ I ® E A modified form of M (see Section 2.5) was used to index the departure of S from the equal-diagonal-block structure. If & is the fcth diagonal block of S, k = 1,..., q < T — 1 (with q = T — 1 here), define M* as j v / g l n l ^ ^ - H B l ) Note that the total of the sample sizes is a factor in this formula. Degrees of departure can be compared for a set of matrices by choosing an arbitrary value of N and using it in all computations of Al*. Here, the value of N used was 40—a middle value between the smallest and largest total sample sizes (24 and 72) used in the study. These matrices were chosen to provide three levels of AQQ, and, for the two lower levels, different correlational Methodology / 3.2 50 structures. The M\Q value for pairs and triples involving E K was 8.7 and the value was 9.3 when E L was used. All off-diagonal correlations for E R were in the range 0.32 to 0.60. All same-variable interoccasion correlations for E L were higher (0.67 to 0.78) but intervariable interoccasion correlations were lower (0.16 to 0.21) and all intervariable same-occasion correlations were 0.316. Matrices E ^ and E N provided M\Q values of 27.8 and 27.5 respectively. Correlations for E M ranged from 0.32 to 0.80, while E^ had very small intervariable same-occasion correlations (0.02 to 0.14) and a maximum off-diagonal correlation of 0.68. Intervariable same-occasion correlations were lower for E Q than for the other four matrices (0.28 to 0.32) but other correlations were similar to those of E^- The pair and triple using Eo had an M\Q value of 45.4. Type B V i o l a t i ons : Three G r o u p Case Matrix triples {ER, E R , ER}, •.., {Eo, E G, Eo} were used to provide exam-ples of a situation parallel to the previous one: (C ' ® l)Si(C® l ) = 2 ( C ® l ) E 2 ( C ® l ) =S (C'®I)E 3 (C®I) = 5 H ^ I ® E M\Q values were the same as in the two-group experiments. Note that only one of two subclasses of type B violation was modelled in this study. All members of each pair/ triple used here were identical. This need not be the case. Type C V io l a t i ons : T w o G r o u p Case Matrices ST, and E^ were used as "parents" in generating pairs that provided four forms/ degrees of type C violations: Set Sp 0 0i e d M* M\0 {E P > E Q} E L 0.20 9.3 Methodology / 3.3 51 {E R, Es} E L 0.95 9.3 {E T, Eu) E N 0.20 27.5 {E v, E w} E N 0.92 27.5 That is, (C ® I)E X(C ® I) is diferent from (C ® I)E 2(C ® I) to a greater or lesser extent, and the pooled matrices in each case depart from equal-diagonal-block form to a larger or smaller degree. Type C Violations: Three Group Case Matrix triples, each of which included a pooled matrix, were used to provide violations corresponding to those in the two-group case in terms of M* and AQQ : Set Spooled M* M*i0 { E L , E P, E Q} E L 0.20 9.3 {EL, E R, Es} E L 0.95 9.3 {E N, E T, Eu} E N 0.20 27.5 {E N, E v, E w} E N 0.92 27.5 As in the three-group type A experiments, when the sample sizes were in the proportion 1:1:2, a smaller sized sample was drawn from the population whose covariance matrix was E p o o i e ( i to permit simulation of sampling methods parallel to those of two-group experiments in which ng s were unequal. 3.3 T H E S I M U L A T I O N P R O C E S S The outcomes of repeated simulations of each experiment were: (a) an estimate—for each of two levels of nominal or, namely 0.05 and 0.01—of the rate of occurence of values of the test statistic AQX smaller than the appropriate critical values of the Up,„et„fc distribution with ve = (N-G){T-1) and vh = (G- 1){T-1); (b) an estimate, for each of the two levels of a n o m, of the rate of occurence of smaller than the appropriate critical values of the U distribution with ue as above and vh equal to T — 1. Methodology / 3.3 5 2 For each experiment modelled, three basic operations were repeated 1000 times to provide the above estimates: (1) a matrix of simulated scores, Y , was produced using a random number generator and a program, described below, to transform the random numbers into data with specified characteristics; (2) SSCP matrices T , U, and W were computed from Y using Equations (2.15), (2.16) and (2.19); (3) values for the statistics AQT a n d were computed— AQX = |W(W + U) - 1| and Ax = | W ( W + T ) - 1 | —and compared with critical values obtained from Timm (1975, pp. 626-627). When a comparison supporting rejection of a null hypothesis at a particular level occured, a Type I error counter was incremented. Data matrix Y was generated as G submatrices. Each Yg g = 1,..., G was produced using a four step process. (1) Prior to the experiment, a lower triangular factor lig was produced for us-ing Cholesky factorization (Harman, 1976): E ? = kffL'ff. This was done with the DREDUC subroutine in EISPACK version 2 (Smith, Boyle, Dongarra, Garbow, Ikebe, Klema, & Moler, 1976). This type of factor was computed—as opposed to one obtained from eigenvalue eigenvector decomposition—so that during replica-tions of Step 2, the number of multiplications and additions performed would be minimal. (2) A set of PTng (6ng here) pseudorandom numbers were generated using the library subroutine RANDN (Nichol, 1981). This subroutine uses a multiplicative congru-ential generator to produce pseudorandom variates uniformly distributed on the interval [0,1). This generator, which has passed extensive testing (Nichol, 1981, p. 7), uses the formula 2 n + 1 = (78125005z„)mod 232 where zn and z n + i are the nth and n + 1 th integers in the sequence, respectively, (floated to obtain numbers in the interval). A transformation routine based upon Marsaglia's Rectangle Wedge Methodology / 3.3 53 Tail method (Knuth, 1968) outputs pseudorandom variates whose distributions are N(0,1). (3) Vectors Z,ff of length PT (6) were formed from this set of pseudorandom num-bers. Vectors produced in this way have independent distributions, each of which is Npy(0,Ipr). (See Anderson, 1958, pp. 19-27 for a discussion of this point.) Each Zig was transformed to a PT-tuple, Yig, using Y,? = I*</ZtJ/. The Y t ? vectors rep-resent sets of scores drawn independently from the multivariate normal distribution Npr^S,). (4) Each Yig was split into three segments of length two which were arranged as in Figure 3 to form Yg. All simulations were performed on the University of British Columbia's Am-dahl 470 V7 computer, a machine with a 32 bit word size. Programs for the simulations were written in VS FORTRAN (International Business Machines, 1984) and specified dou-ble precision calculations. Listings for these programs are included in Appendix B, as are results of "benchmark" tests of the operation of the programs. CHAPTER 4 Results The focal question of this study was: When there is in fact no group-by-occasion interaction and no occasion effect present for G populations from which observa-tions are drawn, how frequently will multivariate mixed model ANOVA tests of parallelism and occasions hypotheses indicate presence of effects under various violations of the mul-tivariate multisample sphericity condition? The results of a Monte Carlo investigation of this question are presented here, organized by type of violation—A, B, and C. Within each section, the effects of departure from MMS are discussed first for experiments in which samples were equal in size and then for experiments in which sample sizes were unequal. At each stage in the discussion, apparent influences of (a) degree of departure from MMS, (b) number of groups (G), and (c) ratio of average sample size to number of variables (N/(GP)) are noted. Finally, Type I error rates for tests of H 0 g t are compared with those of tests ofH0x. In the discussion of results for violations of type B, comparisons are made with results for type A violations. Similarly, comparisons are drawn between Type I error rates observed under type C violations and those noted for the other types of violation. When there is a departure from MMS in a simulated experiment, a dis-crepancy between an empirical value for a and a n o m may be due to sampling error, the departure, or both. Approximate 0.95 confidence intervals around each value of a n o m can be constructed using the standard error of proportion p (that is, ^ N r e p sp(l — p)/Nreps where Nreps, here, is 1000 and p = a n o m) and normal curve probabilities (Hakstian et al., Results / 4.1 55 1979). This gives 0.05 ± 1.96(0.0069) and 0.01 ± 1.96(0.0031). In the tables in this chapter, each entry is the number of Type I errors for a given a n o m in 1000 replications of an experi-ment. The confidence interval for 1000 x a n o m is [37,63] when a n o m is 0.05, and [4,16] when a n o m is 0.01. Each entry representing an actual a value outside the appropriate confidence interval is marked with an asterisk. Such an a is considered to be reliably different from 4.1 E M P I R I C A L A L P H A V A L U E S U N D E R T Y P E A V I O L A T I O N S O F M M S Three levels of type A violation were modelled with associated M* values of 0.3, 1.3, and 3.9. In all experiments, M*i0 was zero. Tables 7 and 8 contain data from tests conducted at anom levels of 0.05 and 0.01 respectively. The — and + signs refer, respectively, to negative and positive sampling. E q u a l Sample Sizes When samples were equal in size and N/(GP) was six, Type I error rates for tests of H 0 Q T and H 0 t were not significantly different from nominal levels at any level of Af*. When Nj{GP) was 12, tests of the parallelism hypothesis tended to become liberal while tests of the occasions hypothesis had actual Type I error rates within sampling variability of nominal values. It appears that when sample sizes are equal, interpopulation heterogene-ity of orthonormalized covariance matrices has little effect upon the level of significance when the pooled matrix has equal-diagonal-block structure, except, perhaps, in the case of extreme heterogeneity (M* = 3.9). Unequa l Sample Sizes (a) Negat ive Samp l ing In all experiments in which the largest sample was drawn from the population having the orthonormalized covariance matrix with the smallest determinant, empirical Type I error rates for tests of the parallelism and occasions hypotheses were significantly larger than nominal levels. As M* increased, actual Type I error rates tended to depart from nominal Results / 4.1 56 Sample specifications G = 2 G = 3 H 0 N/{GP) 1:1 1:2 (-) 2:1 (+) 1:1:1 1:1:2 (") 1:1:2 (+) M* = 0.30 GT 6 12 53 56 90* 98* 24* 35* 55 57 86* 90* 35* 41 T 6 12 44 53 82* 77* 21* 28* 43 56 73* 76* 36* 28* Af* = 1.3 GT 6 12 57 61 147* 174* 9* 18* 55 69* 149* 139* 24* 30* T 6 12 52 53 139* 149* 13* 11* 48 54 107* 115* 25* 21* M* = 3.9 GT 6 12 60 68* 244* 261* 0* 5* 62 75* 210* 197* 19* 20* T 6 12 59 66 241* 239* 4* 4* 53 48 167* 164* 15* 11* (—) refers to negative sampling (+) refers to positive sampling TABLE 7 Number of Type I Errors in 1000 Replications (1000 x a n o m = 50) : Type A Violations. Results / 4.1 57 Sample specifications G = 2 G = 3 1:1 1:2 2:1 1:1:1 1:1:2 1:1:2 (-) (+) (-) (+) H0 N/(GP) M* = 0.30 GT 6 11 18* 4 6 26* 4 12 12 28* 6 21* 35* 5 T 50 6 19* 6 14 19* 5 10 7 17* 2* 11 14 2* M* = 1.3 GT 6 13 49* 0* 11 45* 3* 12 8 58* 1* 25* 58* 6 T 6 10 45* 3* 12 31* 1* 12 11 49* 1* 11 27* 2* M* = 3.9 GT 6 16 104* 0* 15 95* 2* 12 13 120* 0* 30* 84* 6 T 6 13 99* 0* 12 53* 0* 12 14 109* 1* 10 51* 1* (—) refers to negative sampling (+) refers to positive sampling TABLE 8 Number of Type I Errors in 1000 Replications (1000 x a n o m = 10) : Type A Violations. Results / 4.2 58 rates to a greater degree for tests of both hypotheses. Tests of each hypothesis tended to be more liberal when G was two than when G was three. At each level of M* and value of G, changing N/(GP) made no difference to actual Type I error rates, for tests of H 0 G T and H 0 t , When G was three, tests of H 0 g t tended to be more liberal than tests of H 0 T . (b) Positive Sampling In experiments in which the largest sample was drawn from the population whose or-thogonalized covariance matrix had the largest determinant, actual Type I error rates were significantly smaller than nominal levels. As M* increased, empirical Type I error rates tended to differ to a greater extent from nominal rates for tests of both hypotheses. Alteration of G did not affect actual Type I error rates for tests of either hypothesis. 4.2 E M P I R I C A L A L P H A V A L U E S U N D E R T Y P E B V I O L A T I O N S O F M M S Five matrices, which when orthonormalized had different covariance structures, were used to model three levels of type B violation: low (AQQ = 8.7 and 9.3), medium (MlQ = 27.8 and 27.5), and high {M\Q = 45.4). In all experiments, M* was zero. Table 9 contains results for tests of H 0 g t and H 0 X conducted at the 0.05 level of significance; Table 10 contains the corresponding results for tests conducted at the 0.01 level. Generally, data from type B experiments was less patterned than data from type A experiments. Equal Sample Sizes In experiments in which sample sizes were equal, tests of H 0 g t and H 0 X tended to be liberal, particularly when the most extreme violation (AQQ = 45.4) was modelled. For lower values of M*i0, the relationship between actual level of significance and level of violation was less clear, but with one exception, actual Type I error rates did not differ significantly from nominal levels when M\Q was at the lowest level (8.7). The largest empirical Type I error rates observed in experiments involving equal-sized groups were of the same "order of magnitude" as the largest rates in equal-groups experiments modelling the most severe type A violations. Results / 4.2 59 Relative sizes of samples G = 2 G ~ 3 1:1 1:2 1:1:1 1:1:2 H0 S N/ (GP) = 6 GT J 8.7 49 49 59 65* K 9.3 61 52 68* 68* L 27.8 59 61 51 70* M 27.5 49 70* 72* 71* N 45.4 70* 60 67* 70* T J 8.7 66* 61 46 57 K 9.3 41 55 46 58 L 27.8 69* 68* 55 51 M 27.5 60 54 51 78* N 45.4 72* 69* 59 80* N/ (GP) = 12 GT J 8.7 56 46 48 59 K 9.3 63 55 56 67* L 27.8 73* 63 58 66* M 27.5 56 62 60 90* N 45.4 76* 66* 63 78* T J 8.7 61 61 53 59 K 9.3 55 53 64* 48 L 27.8 68* 70* 68* 59 M 27.5 71* 67* 70* 65* N 45.4 72* 77* 66* 74* TABLE 9 Number of Type I Errors in 1000 Replications (1000 x a n o m = 50) : Type B Violations. Results / 4.2 60 Relative sizes of samples G = 2 G = 3 1:1 1:2 1:1:1 1:1:2 H 0 E **40 N/ (GP) = 6 GT J 8.7 9 15 12 12 K 9.3 11 10 19* 16 L 27.8 18* 17* 14 13 M 27.5 9 17* 20* 19* N 45.4 21* 22* 21* 22* T J 8.7 17* 18* 14 12 K 9.3 14 13 14 12 L 27.8 21* 19* 19* 16 M 27.5 20 13 13 24* N 45.4 19* 21* 16 21* N/ (GP) = 12 GT J 8.7 13 11 7 21* K 9.3 15 16 8 14 L 27.8 22* 14 19* 17* M 27.5 12 20* 14 26* N 45.4 18* 13 18* 25* T J 8.7 16 7 12 8 K 9.3 18* 17* 16 10 L 27.8 26* 16 18* 15 M 27.5 21* 17* 16 17* N 45.4 30* 18* 18* 21* TABLE 10 Number of Type I Errors in 1000 Replications (1000 x a n o m = 10) : Type B Violations. Results I 4.3 61 Unequal Sample Sizes Again the empirical Type I error rates tended to be significantly greater than their paired nominal values when the most extreme violation was modelled. Further, tests of H 0 G T tended to be liberal (for a n o m = 0.05) when G was three and when N/(GP) was 12. The patterns of results for the remaining conditions and tests were less clear. 4.3 E M P I R I C A L A L P H A V A L U E S U N D E R T Y P E C V I O L A T I O N S O F M M S Four forms of type C violation were modelled using combinations of low and high values of M* (0.2 and 0.9) with low and high values of M*i0 (9.3 and 27.5). In Table 11 the results for tests conducted at the 0.05 level of significance are presented and in Table 12, results for the 0.01 level. Equal Sample Sizes In experiments in which sample sizes were equal and At^ was 27.5, tests of H 0 g t and H 0 t tended to be liberal. In contrast, actual Type I error rates for tests of H 0 t were not significantly different from nominal rates in any experiment in which M\0 was 9.3. Findings for tests of H 0 g t were more complex. With the lower value of M\Q, actual Type I error rates tended to be significantly larger than or n o m levels when G was three but not when G was two. It would appear that increasing M* from 0.2 to 0.9 for a given level of M\0 and combination of G and N/(GP) levels did not affect actual Type I error rates. Actual Type I error rates for tests of H 0 q t were similar to those for tests of H 0 T . In an effort to further understand results from type C experiments, empirical significance levels for equal-groups type C experiments were compared with those for equal-groups experiments modelling the other types of violation. Actual Type I error rates for type C experiments in which M\0 had the maximum value—for each type of test and for each combination of levels of G and N/(GP)—were similar to rates for type A experiments in which M* was 0.3 or 1.3. This suggests that there is not a strong relationship between Results / 4.3 62 Sample specifications G = 2 G = 3 1:1 1:2 (") 2:1 (+) 1:1 1:1:2 (") 1:1:2 (+) H0 M* N/(GP] (=6 GT 9.3 0.20 0.95 55 53 57 94* 36* 29* 57 62 75* 98* 57 44 27.5 0.20 0.92 65* 54 72* 99* 56 39 76* 72* 76* 111* 80* 44 T 9.3 0.20 0.95 51 49 68* 88* 28* 34* 53 58 61 77* 44 35* 27.5 0.20 0.92 64* 64* 71* 88* 39 36* 72* 74* 93* 88* 62 58 N/(GP) = 12 GT 9.3 0.20 0.95 61 57 64* 108* 46 30* 69* 77* 71* 102* 50 47 27.5 0.20 0.92 63 64* 76* 97* 49 35* 69* 74* 74* 100* 55 50 T 9.3 0.20 0.95 60 56 67* 82* 37 25* 60 60 65* 96* 60 42 27.5 0.20 0.92 55 57 69* 77* 47 32* 80* 76* 72* 88* 61 56 (—) refers to negative sampling (+) refers to positive sampling TABLE 11 Number of Type I Errors in 1000 Replications (1000 x a n o m = 50) : Type C Violations. Results / 4.3 63 Sample specifications G = 2 G = 3 1:1 1:2 (") 2:1 (+) 1:1 1:1:2 (-) 1:1:2 (+) H 0 M" N/{GP) 1=6 GT 9.3 0.20 0.95 14 14 15 29* 6 7 21* 22* 18* 35* 15 10 27.5 0.20 0.92 14 12 23* 34* 9 11 24* 22* 20* 39* 22* 17* T 9.3 0.20 0.95 14 12 15 18* 3* 8 10 12 20* 23* 14 12 27.5 0.20 0.92 13 16 21* 21* 6 12 12 12 21* 28* 17* 13 N/{GP) = 12 GT 9.3 0.20 0.95 15 15 16 32* 10 3* 19* 20* 23* 39* 22* 10 27.5 0.20 0.92 21* 23* 12 29* 13 8 19* 27* 30* 32* 16 15 T 9.3 0.20 0.95 13 15 14 22* 7 4 10 11 22* 29* 13 13 27.5 0.20 0.92 18* 18* 14 30* 15 4 23* 23* 32* 24* 10 21* (—) refers to negative sampling (+) refers to positive sampling TABLE 12 Number of Type I Errors in 1000 Replications (1000 x a n o m = 10) : Type C Violations. Results / 4.3 64 actual Type I error rate and M\0 when sample sizes are equal. Comparison of data from equal-groups type B experiments (Af* = 0) in which M\Q values were 9.3 and 27.5 with data from type C experiments using Af * = 0.95 supported the above finding of an apparent lack of relationship between actual Type I error rate and Af* when samples are equal in size. Unequal Sample Sizes As in experiments modelling type A violations, tests of the parallelism and occasions hypotheses tended to become liberal when sampling was negative and conservative when sampling was positive, (a) Negative Sampling For each level of Ml0 and combination of G and N/(GP), empirical Type I error rates for tests of H 0 g t and H 0 x tended to be significantly greater than nominal levels when sampling was negative. For tests of each hypothesis, actual significance levels when .M40 was 27.5 were generally similar to rates when M\0 was 9.3. Tests of each hypothesis tended, however, to become more liberal as Af* increased. This pattern mirrors the relationship between actual Type I error rates and Af* found with type A experiments using negative sampling. There was no apparent relationship, for tests of each hypothesis, between Type I error rate and either G or N/(GP). However, for a given combination of G, N/(GP), Af*, and Ml0, tests of H 0 g t tended to be more liberal than tests of H 0 T. Experiments in which sampling was negative and violations of type C were modelled gave larger Type I error rates for tests of H 0 g t than comparable experiments (in terms of G and N/(GP)) in which violations were of type B. The same held for tests of HQ t . The combination of increasing Af * from 0 to 0.95 and sampling negatively appears to have enhanced the effect of departure (in the pooled matrix) from equal-diagonal-block structure—for each level of H\0. While violations modelled in two-group type C experiments are not directly comparable, in terms of Af*, with violations in any of the three sets of type A experiments, Results / 4.3 65 it is interesting to examine together data from type C experiments in which M* = 0.95 and data from type A experiments in which Af* = 0.3. For each sampling situation, tests of H 0 g t and H 0 x gave actual levels in the same "order of magnitude" for the two types of violation. Empirical a levels for type A experiments with M* values of 1.3 or 3.9 were increasingly greater, suggesting that type A experiments using a matrix set with an M* value of 0.95 would have given Type I error rates between those of the experiments in which M* was 0.3 and 1.3. Possibly, when sampling was negative, the a-elevating effect of interpopulation matrix heterogeneity was somehow "moderated" by departure of the pooled matrix from equal-diagonal-block form, (b) Positive Sampling When sampling was positive, empirical Type I error rates for tests of HoGT and H 0 t appear to be related to G . When G was two, actual Type I error rates tended to be lower than nominal values at each level of M * I ( I and at each level of N / ( G P ) . When G was three, actual Type I error rates for tests of each hypothesis tended not to be significantly different from the nominal value (when a n o m was 0.05) however, at the 0.01 level actual Type I error rates were significantly larger than the nominal level. For tests of the two types of hypothesis under each combination of levels of G and N / ( G P ) , actual Type I error rates when M \ Q was 27.5 were similar to those when M\0 was 9.3. Also there was no strong relationship between actual Type I error rate and M* and actual level of significance and N / { G P ) for either type of test. Empirical Type I error rates for tests of H 0 g t were similar to rates for tests of H 0 < at each level of M \ 0 and for each combination of levels of G and N / { G P ) . The type A violations (M\Q = 0) modelled with positive sampling produced significantly lower actual significance levels than nominal levels as reported earlier. It would appear, given the results for the type C experiments, that under positive sampling, as the departure of the pooled matrix from equal-diagonal-block form became more severe, the a-depressing effect of interpopulation matrix heterogeneity became less pronounced. This apparent trend—and, as with all apparent trends discussed here, more data is needed Results / 4.3 66 for confirmation—parallels the situation reported with negative sampling. The following chapter synthesizes results concerning the three types of vi-olation of the MMS condition by responding to the conjectures at the conclusion of the review of the literature (Section 2.8). CHAPTER 5 Conclusions and Recommendations In experimental or quasi-experimental studies in which a repeated measures design is used, it is common to obtain scores on several dependent variables on each mea-surement occasion. Multivariate mixed model (MMM) analysis of variance (Thomas, 1983) is a recently developed alternative to the MANOVA procedure (Bock, 1975; Timm, 1980) for testing multivariate hypotheses concerning effects of a repeated factor (called occasions in this study) and interaction between repeated and non-repeated factors (termed group-by-occasion interaction here). If a condition derived by Thomas (1983), multivariate mul-tisample sphericity (MMS), regarding the equality and structure of orthonormalized popu-lation covariance matrices is satisfied, (given multivariate normality and independence for distributions of subjects' scores), valid likelihood-ratio MMM tests of group-by-occasion interaction and occasions hypotheses are possible. To date, no information has been available concerning actual (empirical) levels of significance of such tests when the MMS condition is violated. This study was conducted to begin to provide such information. Departure from the MMS condition can be classified into three types— termed departures of types A, B, and C respectively: (A) the covariance matrix for population g (g = 1,...G), when orthonormalized, has an equal-diagonal-block form but the resulting matrix for population g is unequal to the resulting matrix for population g' (g ^ g'); 67 Conclusions and Recommendations / 5.1 68 (B) the G populations' orthonormalized covariance matrices are equal, but the matrix common to the populations does not have equal-diagonal-block structure; or (C) one or more populations has an orthonormalized covariance matrix which does not have equal-diagonal-block structure and two or more populations have unequal orthonormalized matrices. In this study, Monte Carlo procedures were used to examine the effect of each type of violation in turn on the Type I error rates of multivariate mixed model tests of group-by-occasion interaction and occasions null hypotheses. For each form of violation, experiments modelling several levels of severity were simulated. Each experiment was replicated 1000 times and Type I errors for tests conducted at nominal a levels of 0.05 and 0.01 were counted. In these experiments: (a) the number of measured variables (P) was two; (b) the number of measurement occasions (T) was three; (c) the number of populations sampled (G) was two or three; (d) the ratio of average sample size to number of measured variables (N/(GP)) was six or 12; and (e) the sample size ratios were 1:1 and 1:2 when G was two, and 1:1:1 and 1:1:2 when G was three. In experiments modelling violations of types A and C, the effects of negative and positive sampling were studied. 5.1 S U M M A R Y O F E F F E C T S O N T Y P E I E R R O R R A T E S The organization of this section is similar to that of the previous chapter. Effects of each type of violation are summarized in turn with robustness with sample sizes equal discussed first, followed by discussion of robustness when samples are unequal in size. Type A Violations When type A violations of the lowest and intermediate levels of severity were modelled Conclusions and Recommendations / 5.1 69 in experiments using samples of equal sizes, actual Type I error rates for tests of each hypothesis were not significantly different from the corresponding nominal levels. This result parallels findings of Noe (1976) and Rogan et al. (1979) for equal-groups experiments modelling type A violations in the case of one dependent variable. However, with a more severe type A violation, tests of of both hypotheses tended to become liberal, in particular, when samples of the larger size were drawn. Turning to the case of samples of unequal sizes, when the largest sample was drawn from the population whose orthonormalized covariance matrix has the smallest determinant (negative sampling) actual Type I error rates were significantly higher than nominal rates for both hypothesis tests and for all levels of violation. Further, this dis-crepancy increased for tests of both hypotheses as orthonormalized covariance matrices for populations sampled became more dissimilar. Actual Type I error rates were not related to the number of samples or the ratio of average sample size to number of variables when the least severe violation was modelled. While the Type I error rate was not influenced by the ratio of average sample size to number of variables when violations of greater severity were modelled, the actual Type I error rates for tests of both hypotheses decreased when the number of groups increased. The fact that tests of both hypotheses became liberal under negative sam-pling matches a finding by Noe (1976). Noe, however, observed that Type I error rates for tests of the interaction hypothesis were larger than those for the occasions hypothesis. No such difference was found in the present study. In type A experiments in which the largest sample was drawn from the pop-ulation whose orthonormalized covariance matrix had the smallest determinant (positive sampling), empirical Type I error rates for tests of both hypotheses were significantly smaller than nominal levels for all degrees of violation. As the orthonormalized covariance matrices became more dissimilar, the empirical level of significance decreased. Further, as Conclusions and Recommendations / 5.1 70 the number of groups increased, actual Type I error rates approached the nominal levels at 0.05 level of significance; these rates, however, were not related to N/{GP). The findings for positive sampling differ slightly from those of Noe. Here, tests of both hypotheses were equally conservative (for all levels of violation severity) whereas Noe observed conservatism in the tests of interaction only. Type B Violations In equal-groups experiments modelling type B violations, actual Type I error rates tended to exceed nominal rates. There was not a strong relationship, however, between actual level of significance (for both hypothesis tests) and the degree to which the orthonormal-ized covariance matrix common to all populations departed from equal-diagonal-block structure. This result contrasts with findings by Collier et al. (1967), Noe (1976), and Ro-gan et al. (1979) who concluded from experiments involving one dependent variable, that actual Type I error rates increased as degree of violation (indexed with Box's e) increased. Suggestions concerning this discrepancy are discussed in Section 5.3. There was no relationship between actual Type I error rate (for either hy-pothesis test) and either G or N/(GP). This finding is in agreement with a result from the Collier et al. (1967) study. Actual Type I error rates for tests of the interaction hypothesis were similar to rates for tests of the occasions hypothesis. This result matches a finding in Noe's study, but contrasts with observations of differences by Collier et al. and Rogan et al. There researchers found that actual Type I error rates were generally larger for tests of the interaction hypothesis. Empirical levels of significance in type B experiments using unequal sized groups again tended to exceed nominal rates. Actual Type I error rates were in the same range for each level of a n o m as that found in type B experiments using equal groups. As in the latter experiments, there were no clear relationships between actual level of significance and severity of violation, G, or N/(GP). Conclusions and Recommendations / 5.1 71 Type C V io l a t i ons In equal-groups experiments modelling type C violations, actual Type I error rates for tests of both hypotheses tended to exceed the nominal level when departure in pooled orthonor-malized covariance matrices from equal-diagonal-block form was at the more severe level. When this structural departure was at the less severe level, actual Type I error rates for tests of the occasions hypothesis were not significantly different from nominal levels, but tests of the interaction hypothesis tended to be liberal when three groups were sampled. It was concluded that for the levels of type C violation modelled in this study, there was not a strong relationship between actual Type I error rate and degree of departure from equal-diagonal-blocks form. Similarly, in equal-groups type C experiments, empirical levels of significance (for each type of test) were not related strongly to degree of interpopulation heterogeneity in orthonormalized covariance matrices. Actual Type I error rates for tests of the interaction hypothesis were similar to rates for tests of the occasions hypothesis for a given set of covariance matrices and combination of levels of G and N/{GP). As in the case of type A violations, tests of both hypotheses became lib-eral under negative sampling. Degree of departure from equal-diagonal-block form in the pooled orthonormalized covariance matrices did not influence actual levels of significance. This finding is in contrast to a trend, in Noe's study, towards greater liberalism as depar-ture from diagonal structure became more severe. Degree of interpopulation heterogeneity of orthonormalized covariance matrices apparently affected actual Type I error rates. Tests of each hypothesis were more liberal in experiments modelling the greater degree of hetero-geneity than those modelling the lesser degree. Tests of the interaction hypothesis tended to be more liberal than tests of the occasions hypothesis for a given set of matrices and combination of levels of G and N/{GP). Empirical significance levels in type C experiments using positive sampling were related to the number of populations sampled. In two-population experiments, ac-tual Type I error rates tended to be below nominal levels for tests of each type. Actual Conclusions and Recommendations / 5.2 72 significance levels for tests conducted at the 0.05 level of a n o m tended not to be signifi-cantly different from this nominal level when G was three, but rates for tests conducted at the 0.01 level tended to exceed the nominal level. There did not appear to be a strong relationship between empirical Type I error rates and degree of departure of the pooled orthonormalized covariance matrix from equal-diagonal-block form although, when two populations were sampled, lower actual Type I error rates tended to be associated with the lower degree of structural departure. Further, actual Type I error rates did not appear to be strongly affected by alteration of the level of interpopulation heterogeneity of orthonormalized covariance matrices. Lower empirical significance levels tended to occur, however, in experiments modelling the higher level of heterogeneity. Empirical Type I error rates for tests of the interaction hypothesis were similar to rates for tests of the occasions hypothesis. The findings just summarized suggest several implications for potential users of multivariate mixed model analysis, to be discussed next. 5.2 P R A C T I C A L I M P L I C A T I O N S O F T H I S S T U D Y Conclusions from this study are considered as implications for research practice and are presented in the form of suggestions to the following practitioners: (a) a designer of a study in which subjects in several samples are to supply two or more scores on each of three or more measurement occasions; and (b) a statistician called upon to determine whether or not data from such a study suggest the presence of a group-by-occasion interaction, and, if not, an occasions main effect. Design Suggestions (1) Use samples of equal size. Regardless of the type of violation of MMS which could be present in the populations under study, omnibus tests of within-subjects hypotheses will at worst be slightly liberal. Conclusions and Recommendations / 5.2 73 (2) Use as many subjects as possible in the study to optimize power, without concern that Type I error rates would be adversely affected by so doing. (3) Divide subjects into as many groups as are appropriate for the research questions. Using more or fewer groups will have no major effect on Type I error rates. An Analysis Strategy If sample sizes are equal or approximately equal, conduct MMM tests of omnibus interac-tion and occasions hypotheses confidently, being aware that tests may be slightly liberal. If a null hypothesis is considered tenable following an MMM test, it may be assumed that rejection would be no more likely in the absence of any violation which may exist (Noe, 1976). If sample sizes differ considerably (say if the largest sample is twice the size of the smallest sample), perform the following sequence of operations: a) compute the unbiased estimate for the covariance matrix Sg for each group g = 1,..., G, and compute the pooled covariance matrix S; b) orthonormalize each of these matrices; c) perform a Box M test (See Section 2.5.) to determine whether or not the hypothesis of equality among the orthonormalized population covariance matrices is tenable. If this hypothesis is tenable, the "worst case" situation is the presence of a type B violation and the positive prospect is the satisfaction of the MMS condition. It is unlikely, however, that the MMS condition is ever fully satisfied. Since (within the range of Ml0 values used in the present study) type B violations in the presence of unequal sample sizes can lead to slightly liberal testing of within-subjects hy-potheses, it would be prudent to conduct the tests of interest at a slightly reduced nominal significance level. If more precise information about the degree of B viola-tion is desired, the value of the transformed generalized Mauchly criterion, M, can be computed (See Section 2.5). Conclusions and Recommendations / 5.3 74 If the value of M is significant, presence of a type A or C violation is indi-cated. The important question becomes, "Which type of sampling—negative or positive— is likely to have occurred?" Assuming one group is much larger than the other(s), compute the determinant of each orthonormalized sample covariance matrix and make a decision about the form of sampling. If positive sampling appears more likely to have occurred than negative sam-pling, actual levels of significance for tests of H 0 q t and H 0 X could be less than or equal to nominal levels. Tests would tend to be conservative if the violation is A-like, that is, if orthonormalized covariance matrices have almost perfect equal-diagonal-block structure. In such a situation, tests of within-subjects hypotheses should be conducted at a nominal level above that which would normally be used, say 0.075 instead of 0.05. If examination of the structure of the orthonormalized matrices and results from a generalized Mauchly test suggest the presence of even mild departures from equal-diagonal-block form, C-like violations are indicated and the actual level of significance may be near the nominal level (particularly if G > 2). Tests of H0(_,T and H 0 t could then be conducted at the nominal level usually used. If it appears probable that negative sampling occurred, tests will be liberal to a greater or lesser extent. Further testing of the orthonormalized sample covariance matrices would not be very helpful in deciding how liberal the tests could be. In conducting tests of the interaction and occasions hypotheses, use a smaller nominal level of significance than would normally be considered acceptable—for example, 0.025 instead of 0.05. 5.3 L I M I T A T I O N S O F T H E S T U D Y A N D D I R E C T I O N S F O R F U R T H E R R E S E A R C H It is possible, in a single simulation study of this type, to model only a selection of the matrix forms one could encounter in the real world. It could be argued, however, that to maximize utility of the findings, extensive research should have been conducted prior to creation of the matrix pool to "map" the domain of real world covariance structures (from Conclusions and Recommendations / 5.3 75 repeated measures experiments)—determining real ranges for intergroup heterogeneity and degree of departure from equal-diagonal-block form. (See, however, Appendix A.) It is not known, at present, whether the violations modelled here were more severe than one would generally find in real world studies or less severe. The task of creating an "ecologically valid" matrix pool for testing the MMM procedure remains one for future research. On the other hand, it can be contended that in an exploratory study such as this, a new technique should be quickly "pushed to its limits", permitting an early discovery of the boundaries of its domain of usefulness. Viewed from this perspective, the present study is possibly quite incomplete. An example will illustrate this point. Examination of the results for experiments modelling type B violations un-covered no clear relationship between actual Type I error rates and At\Q, the index used to summarize departure from equal-diagonal-block form in the pooled orthonormalized covariance matrix. This finding was surprising in light of trends reported by Collier et al. (1967) and Rogan et al. (1979) for studies in which P was one. (See Tables 1 and 4.) It is possible that there really is no relationship to be found. It is possible, too, that even the most severe violation modelled here {M\Q = 45.4) was not sufficiently strong departure in form to cause a major elevation in Type I error rates. It may be, however, that M\0 is not a sensitive index for the purpose of describing relative violation severity. In the studies cited earlier in which a trend was observed, the index of departure from the P = 1 equivalent of equal-diagonal-block form was Box's e, not a modified form of the original Mauchly (1940) criterion. Unfortunately, no multivariate extension of e was available. Exploration of these possibilities remains a task for future research. Due to limitations on resources, the following restrictions were made: a) N r e p s was limited to 1000 and only one set of N r e p s replications was conducted for each simulated experiment; b) a single value for each of P and T was used; Conclusions and Recommendations / 5.3 76 c) a limited range of values was used for each of G and N/(GP); d) only a multivariate normal distribution was used in generating data sets for MMM testing; and e) estimates of actual Type I error rates for tests of H 0 g t and H 0 x using MANOVA (Bock, 1975; Timm, 1980), the MMM procedure's competitor, were not obtained under the conditions modelled in the study. The first of the above restrictions limited the possibility of clean mapping of all but the strongest of trends. It is possible that relationships do exist between actual Type I error rate and G or N/(GP) but that patterns were masked by "noise" in this study. Increasing N r e p s would have permitted use of shorter confidence intervals for estimates of levels of significance. Replicating the entire set of experiments would make it possible to calculate an estimate of internal variability among the data points—a "within-cells" variance estimate in ANOVA terminology—thus allowing the use of statistical inference in determining whether or not effects were present for the independent variables in this study such as G. The second, third, and fourth restrictions limited the generalizability of the findings. While the MMM procedure has an inherent limitation on P—it may be no larger than T — 1 in order for the matrix quadratic form for the occasions hypothesis to have a Wishart distribution—it is common, say, in pharmacology experiments, to measure several physiological variables at frequent intervals, giving a broad range for T and thus for P. It would be useful to researchers in this field to have information on the robustness of MMM ANOVA for values of T and P larger than three and two respectively. Further, it is not uncommon to involve more than three subpopulations and fewer than 12 subjects per group in educational and psychological studies using repeated measures designs. Hence, it would be helpful to examine the robustness of MMM tests for a wider range of values of G and N/(GP). Similarly, it would be useful to examine other distributions of sample sizes across groups. Conclusions and Recommendations / 5.3 77 Although many variables of interest in the behavioral sciences have been found to have normal distributions, variables studied in other fields may not. It could be of use to know how the interaction of departure from normality and violation of the MMS condition affects Type I error rates. A data analyst faced with a set of multivariate repeated measures data would need to be aware of the strengths and weaknesses of all available procedures in order to accurately optimize the balance between Type I and Type II error rates in tests of interaction and occasions hypotheses. Data on actual Type I error rates (and power) for MMM and MANOVA tests under each type of violation would facilitate informed decision-making regarding analysis strategies. In summary, tasks for future research concerning Type I error robustness of MMM ANOVA include: a) developing a matrix pool which better represents the range of violations of MMS found in real world studies and testing the MMM procedure with wider ranges of violations of MMS; b) examining the possibility of developing a multivariate extension of e; c) simulating experiments with other values of T, P, G, N/(GP), and sample size ratios, in each case using skewed distributions as well as the multivariate normal distribution; d) comparing actual Type I error rates for MMM and MANOVA tests. 78 References Anderson, T. W. (1958). An introduction to multivariate statistical analysis. New York: Wiley. Atkinson, T., Blishen, B., Ornstein, M. , ic Stevenson, H. M . (1977). Social change in Canada: Trends in attitudes, values, and perceptions. Downsview, Ontario: York University, Institute for Behavioral Research. Basilevsky, A. (1983). Applied matrix algebra in the statistical sciences. New York: North-Holland. Bock, R. D. (1975). Multivariate statistical methods in behavioral research. New York: McGraw-Hill. Box, G. E . P. (1949). A general distribution theory for a class of likelihood criteria. Biometrika, 86, 317-346. Box, G. E . P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems, II. Effects of inequality of variance and of correlation between errors in the two-way classification. The Annals of Mathematical Statistics, 25, 484-498. Collier, R. O., Jr., Baker, F. B., Mandeville, G. K., ic Hayes, T. F . (1967). Estimates of test size for several test procedures based on conventional variance ratios in the repeated measures design. Psychometrika, 82, 339-353. Cook, T. D., ic Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin. Glass, G. V., Peckham, P. D., ic Sanders, J. R. (1972). Consequences of failure to meet assumptions under-lying the fixed effects analysis of variance and covariance. Review of Educational Research, 42, 237-288. Hakstian, A. R., Roed, C , ic Lind, J. (1979). Two-Sample T2 procedure and the assumption of homo-geneous covariance matrices. Psychological Bulletin, 86, 1255-1263. Harman, H. H. (1976). Modem factor analysis. (3rd ed.). Chicago: University of Chicago Press. Humphreys, L. G. (1960). Investigations of the simplex. Psychometrika, 25, 313-323. Huynh, H., ic Feldt, L. S. (1970). Conditions under which mean square ratios in repeated measurements designs have exact F-distributions. Journal of the American Statistical Association, 65, 1582-1589. Huynh, H., ic Mandeville, G. K. (1979). Validity conditions in repeated measures designs. Psycho-logical Bulletin, 86, 964-973. International Business Machines Corp. (1984). VS FORTRAN Language and library reference. San Jose, CA: Author. Knuth, D. E. (1968). The art of computer programming (Vol. 2): Semi-numerical algo-rithms. Reading, MA: Addison-Wesley. 79 Mauchly, J. W. (1940). Significance test for sphericity of a normal n-variate distribution. Annals of Mathematical Statistics, 29, 204-209. Mendoza, J. L. , Toothaker, L. E., ic Nicewander, W. A. (1974). A Monte Carlo comparison of the univari-ate and multivariate methods for the groups by trials repeated measures design. Multivariate Behavioral Research, 9, 165-178. Meyers, J . (1979). Fundamentals of experimental design (Srd ed.). Boston: Allyn ic Bacon. Muirhead, R. J. (1982). Some aspects of multivariate statistical theory. New York: Wiley. Nichol, T. (1981). UBC RANDOM. Vancouver, B. C : University of British Columbia, Computing Center. Noe, M. J. (1976, April). A Monte Carlo study of several test procedures in the repeated measures design. Paper presented at the meeting of the American Educational Research As-sociation, San Francisco. Olson, C. L. (1974). Comparative robustness of six tests in multivariate analysis of variance. Journal of the American Statistical Association, 69, 894-908. Rogan, J . C , Keselman, H. J. , ic Mendoza, J. L. (1979). Analysis of repeated measurements. British Journal of Mathematical and Statistical Psychology, 82, 269-286. Searle, S. R. (1971). Linear models. New York: Wiley. Smith, B. T., Boyle, J. M., Dongarra, J. J., Garbow, B. S., Ikebe, Y., Klema, V. C , ic Moler, C. B. (1976). Matrix eigensystem routines—EISPACK guide. (2nd ed.). In G. Goos ic J. Hartmanis (Eds.), Lecture notes in computer science (Vol. 6). New York: Springer-Verlag. Thomas, D. R. (1975). Covariance matrices for univariate repeated measures designs. Ot-tawa: Carleton University, Department of Mathematics. Thomas, D. R. (1981). On the extension of univariate repeated measures ANOVA techniques to multivariate data. Ottawa: Carleton University, Computing Services. Thomas, D. R. (1983). Univariate repeated measures techniques applied to multivariate data. Psy-chometrika, 48, 451-464. Timm, N. H. (1975). Multivariate analysis with applications in education and psychology. Monterey, CA: Brooks-Cole. Timm, N. H. (1980). Multivariate analysis of variance of repeated measurements. In P. R. Krishnaiah (Ed.), Handbook of statistics (Vol. 1). New York: North Holland. A P P E N D I X A Matrices Used in the Simulations A . l G E N E R A T I N G A N D T E S T I N G M A T R I C E S Two considerations directed the the process of generating and selecting population covari-ance matrices for use in the study. (a) When possible, matrices were to reflect covariance structures found in real-world studies using repeated measures designs with P > 2. (b) Pairs and triples formed from selected matrices were to represent violations of types A, B, and C. Within each category, violations of greater and lesser severity were to be modelled. Data from a recent panel study, Social Trends in Canada, (Atkinson, Blishin, Ornstein, & Stevenson, 1977) were used in an examination of relationships among real variables over time. This study was "designed to measure respondents' ... perceptions of their quality of life, emphasizing measurement of the degree of satisfaction with their lives in general and with particular aspects of them as well as their expectations and aspirations in these areas. [Subjects] covered by the survey included employment, income and family relationships" (Atkinson et al., 1977, p. 2). Panel members (N = 1665) answered questions on these topics on three occasions—in 1977, 1979, and 1981. 80 Matrices Used in the Simulations / A.l 81 Questions of interest, for the purposes of generating covariance matrices, were those which (a) were used in the same form on all occasions and (b) had a Likert scale type of response format. Responses from a subsample of 81 people on five 10-point scale items (a global quality of life index, the "Ladder of Life") were used to generate a covariance matrix. Variances and covariances for two of the questions—"rating of present life" and "life deserved now"—were chosen arbitrarily for a 6 x 6 exemplar matrix. All elements of this matrix were rescaled by a factor of lOO/s^^ and the resulting values which were not multiples of five were raised or lowered to a close multiple. The resulting matrix is: "100 30 40 10 40 10" 30 90 5 20 20 25 40 5 65 20 30 20 10 20 20 50 15 25 40 20 30 15 70 25 10 25 20 25 25 50. The corresponding correlation matrix is: "1.00 .32 .50 .14 .48 .14" .32 1.00 .07 .30 .25 .37 .50 .07 1.00 .35 .45 .35 .14 .30 .35 1.00 .25 .50 .48 .25 .45 .25 1.00 .42 . .14 .37 .35 .50 .42 1.00. It was noted that all covariances/ correlations were positive. This feature was retained for convenience in the matrices used in the study. An initial pool of matrices was produced by systematically altering diagonal and off-diagonal submatrices of this exemplar. For each matrix generated: (a) eigenvalues were calculated and positive definiteness was checked; (b) the corresponding correlation matrix was obtained; (c) the structure of the matrix resulting from premultiplication by C'® I and postmul-tiplication by C ® I was examined; and (d) M\Q was calculated. (The matrix was considered as if it was a pooled matrix.) Matrices Used in the Simulations / A.2 82 The values of M* and AQo were calculated for each pair and triple considered for use in the simulations—see Sections 2.5 and 2.7—and, in some cases, adjustments to the matrices and additions to the pool were made. A.2 T H E M A T R I X P O O L The 23 matrices used in the study (with all entries multiplied by 10) are as follows: 1000 300 400 400 400 400" 1000 300 200 200 200 200 300 900 400 400 400 400 300 900 200 200 200 200 400 400 1000 300 400 400 200 200 1000 300 200 200 400 400 300 900 400 400 200 200 300 900 200 200 400 400 400 400 1000 300 200 200 200 200 1000 300 400 400 400 400 300 900. . 200 200 200 200 300 900 E A S B "900 200 100 100 100 100" "1000 300 375 75 350 50" 200 800 100 100 100 100 300 900 75 350 50 325 100 100 900 200 100 100 375 75 950 250 325 25 100 100 200 800 100 100 75 350 250 800 25 275 100 100 100 100 900 200 350 50 325 25 900 200 .100 100 100 100 200 800. 50 325 25 275 200 750. 23 c E D "1000 300 675 225 650 200" "1000 300 275 75 250 50" 300 900 225 550 200 525 300 900 75 250 50 225 675 225 950 250 625 175 275 75 950 250 225 25 225 550 250 800 175 475 75 250 250 800 25 175 650 200 625 175 900 200 250 50 225 25 900 200 . 200 525 175 475 200 750. . 50 225 25 175 200 750. S E S F Matrices Used in the Simulations / A.2 "1000 300 775 225 750 200" 300 900 225 650 200 625 775 225 950 250 725 175 225 650 250 800 175 575 750 200 725 175 900 200 . 200 625 175 575 200 750. "1000 300 875 225 850 200' 300 900 225 750 200 725 875 225 950 250 825 175 225 750 250 800 175 675 850 200 825 175 900 200 200 725 175 675 200 750. Si "1000 300 400 400 400 400" 300 900 400 400 400 400 400 400 1000 300 400 400 400 400 300 900 400 400 400 400 400 400 700 300 400 400 400 400 300 500. "1000 300 400 400 400 400' 300 900 400 400 400 400 400 400 700 300 400 400 400 400 300 500 400 400 400 400 400 400 700 300 . 400 400 400 400 300 500. E M "1000 300 175 75 150 50" 300 900 75 150 50 125 175 75 950 250 125 25 75 150 250 800 25 75 150 50 125 25 900 200 . 50 125 25 75 200 750. "1000 300 525 150 500 125" 300 900 150 450 125 425 525 150 950 250 475 100 150 450 250 800 100 375 500 125 475 100 900 200 125 425 100 375 200 750 "1000 300 750 200 700 200" 300 900 200 700 200 600 750 200 1000 300 750 150 200 700 300 900 150 650 700 200 750 150 1000 300 200 600 150 650 300 900 "1000 300 680 130 260 90 300 900 130 600 90 260 680 130 1000 300 100 20 130 600 300 900 20 340 260 90 100 20 1000 300 . 90 260 20 340 300 900 Matrices Used in the Simulations / A.2 84 "1000 300 400 400 400 400" 300 900 400 400 400 400 400 400 1000 200 400 400 400 400 200 500 400 400 400 400 400 400 1000 200 . 400 400 400 400 200 500. S 0 "1000 300 770 230 750 250" 300 900 230 720 250 650 770 230 1000 300 800 150 230 720 300 900 150 650 750 250 800 150 1000 300 . 250 650 150 650 300 900. "1000 300 660 200 640 200" 300 900 200 610 200 540 660 200 1000 300 730 150 200 610 300 900 150 650 640 200 730 150 1000 300 200 540 150 650 300 900. "1000 300 740 130 320 90 300 900 130 650 90 320 740 130 1000 300 150 20 130 650 300 900 20 380 320 90 150 20 1000 300 90 320 20 380 300 900 "1000 300 730 170 650 150" 300 900 170 680 150 550 730 170 1000 300 700 150 170 680 300 900 150 650 650 150 700 150 1000 300 . 150 550 150 650 300 900. "1000 300 840 200 760 200" 300 900 200 790 200 660 840 200 1000 300 770 150 200 790 300 900 150 650 760 200 770 150 1000 300 . 200 660 150 650 300 900. S R "1000 300 620 130 200 90 300 900 130 550 90 200 620 130 1000 300 50 20 130 550 300 900 20 300 200 90 50 20 1000 300 90 200 20 300 300 900 "1000 300 780 130 380 90 300 900 130 700 90 380 780 130 1000 300 150 20 130 700 300 900 20 420 380 90 150 20 1000 300 90 380 20 420 300 900 Matrices Used in the Simulations / A.2 "1000 300 300 900 400 400 400 400 400 400 400 400 400 400 400 400 1000 300 300 900 400 400 400 400 400 400' 400 400 400 400 400 400 1000 300 300 900 A P P E N D I X B Computer Programs for the Monte Carlo Simulations In the first two sections of this appendix complete listings are provided of the two programs used in conducting the simulations. The first reads in the specifications for a given set of experiments (G, P, T, Nreps, and sample sizes), the covariance matrices, and the critical values for the U distribution. It performs the Cholesky fatorization of the covariance matrices and generates the projection matrices needed for computing matrix quadratic forms U , T , and W . These inputs are passed to a second program which generates data (calling the library subroutine RANDN), transforms it using the lower triangular factors, computes test statistic values, and compares them to critical values, counting exceedences as Type I errors. Operations were divided between two programs to avoid tying up the com-puter memory needed for the first tasks for the duration of a set of 1000 replications. All input information was printed to permit checking. In the third section of the appendix, results of benchmark tests of the oper-ation of the simulation program are presented. B.l T H E P R E P A R A T I O N P R O G R A M INTEGER GRP. OCC, VAR, REPS. OCCVAR. G. T , P INTEGER OCCGRP, RUNS, DAY DIMENSION N(6,3) 86 Computer Programs tor the Monte Carlo Simulations / B.l DOUBLE PRECISION LOWTRI(3,21), SIG(3,6,6), UCRIT(6.4) DIMENSION TPR0J(9.9). GTPR0J(9,9), TSPR0J(3.72.72) DIMENSION TXCCT(3.3), FMIDFT(3,3), UNITYG(3,3) REAL NGXHHT(3,24.24) C WRITE (6.10) 10 FORMAT (/* ****************************************•) WRITE (6,20) 20 FORMAT (/' MMM ROBUSTNESS STUDY: TYPE I ERROR RATES') WRITE (6.30) 30 FORMAT (/' ****************************************") READ (5.40) MONTH. DAY 40 FORMAT (212) WRITE (6.50) MONTH. DAY 50 FORMAT (//' DATE OF RUN: 85 \ 12. 2X. 12/) C C READS PARAMETERS AND POPULATION COVARIANCE MATRICES C FOR A SET OF SIMULATED EXPERIMENTS C C READS NUMBER OF GROUPS. OCCASIONS. VARIABLES AND REPS C READ (5.60) GRP. OCC. VAR, REPS. SEED. RUNS 60 FORMAT (3(11.IX), 14, IX, F4.0, IX, II) WRITE (6,70) GRP. OCC, VAR 70 FORMAT (/' GRP=\ II, ' OCC=", II, ' VAR=\ II/) WRITE (6,80) SEED 80 FORMAT (/' SEED=\ F6.1///) WRITE (7.90) GRP. OCC, VAR, REPS. SEED, RUNS 90 FORMAT (3(II,IX). 14. IX. F6.1, IX. II) C OCCGRP = OCC * GRP OCCVAR = OCC * VAR LAST = (OCCVAR*(OCCVAR + 1)) / 2 C C READS THE POPULATION COVARIANCE MATRIX, SIG(G). FOR C EACH GROUP AND PERFORMS CHOLESKY FACTORIZATIONS C DO 160 G = 1, GRP DO 100 I = 1. OCCVAR 100 READ (5.110) (SIG(G,I,J),J=1.OCCVAR) 110 FORMAT (6(F5.0,1X)) DO 120 I = 1. OCCVAR 120 WRITE (6.130) (SIG(G,I,J),J=l.OCCVAR) 130 FORMAT (6(F9.2,1X)) WRITE (6,140) G 140 FORMAT (/' COVARIANCE MATRIX FOR POPULATION \ II///) Computer Programs for the Monte Carlo Simulations / B.l CALL CH0L(G. SIG. OCCVAR. LOWTRI) C WRITE (7.150) (LOWTRI(G.L),L=1.LAST) 150 FORMAT (21F12.9) 160 CONTINUE C C READS GROUP SIZES AND U DISTRIBUTION CRITICAL VALUES C DO 999 NRUN = 1. RUNS READ (5.170) (N(NRUN.G),G=1.GRP) WRITE (7,170) (N(NRUN.G),G=1.GRP) 170 FORMAT (3(I3.3X)) 999 CONTINUE C DO 199 NRUN = 1. RUNS READ (5.198) (UCRIT(NRUN.J).J=l.4) WRITE (7.198) (UCRIT(NRUN.J),J=1,4) 198 FORMAT (4(F6.4.1X)) 199 CONTINUE C c ***************************************************** C CREATES CENTER MATRICES NEEDED TO PRODUCE THE MATRIX C QUADRATIC FORMS C C PRODUCES TXCCT C DO 230 I = 1. OCC DO 230 J = 1. OCC IF (I .EQ. J) TXCCT(I.J) =2.0 IF (I .NE. J) TXCCT(I.J) = -1.0 230 CONTINUE C C PRODUCES UNITYG C DO 240 I = 1, GRP DO 240 J = 1. GRP UNITYG(I.J) = 1.0 240 CONTINUE C C DO 998 NRUN = 1. RUNS C WRITE (6.180) GRP. NRUN 180 FORMAT (//' N(G) : G=l TO G=\ II. ' RUN ', II/) WRITE (6.170) (N(NRUN,G).G=l,GRP) Computer Programs for the Monte Carlo Simulations / B.l WRITE (6.200) 200 FORMAT (///' CENTER MATRICES USED IN GENERATING') WRITE (6.210) 210 FORMAT (' HYPOTHESIS SSCP MATRICES: OCCASION') WRITE (6.220) 220 FORMAT (' AND PARALLELISM RESPECTIVELY'/) C IF (GRP .EQ. 3) GO TO 250 C C PRODUCES FMIDFT IN TWO GROUP CASE C FMIDFT(l.l) = FLOAT(N(NRUN,i)) * FLOAT(N(NRUN,2)) FMIDFT(2.2) = FMIDFT(l.l) FMIDFTU.2) = -1.0 * FMIDFT(l.l) FMIDFT(2.1) = FMIDFTU.2) C GO TO 260 C C PRODUCES FMIDFT IN THREE GROUP CASE C 250 FMIDFT(l.l) = FLOAT(N(NRUN.1)) * : (FLOAT(N(NRUN,2)) + FLOAT(N(NRUN,3))) FMIDFT(2.2) = FLOAT(N(NRUN.2)) * : (FLOAT(N(NRUN,1)) + FLOAT(N(NRUN,3))) FMIDFT(3.3) = FLOAT(N(NRUN.3)) * : (FLOAT(N(NRUN,1)) + FLOAT(N(NRUN,2))) FMIDFTU.2) « -1.0 * FLOAT(N(NRUN, 1)) * : FLOAT(N(NRUN,2)) FMIDFT(2,1) = FMIDFT(1,2) FMIDFT(1.3) = -1.0 * FLOAT(N(NRUN,1)) * : FLOAT(N(NRUN,3)) FMIDFT(3,1) = FMIDFT(1,3) FMIDFT(2.3) = -1.0 * FLOAT(N(NRUN,2)) * : FLOAT(N(NRUN,3)) FMIDFT(3.2) = FMIDFT(2,3) C C PRODUCES TPROJ MULTIPLIED BY (OCC * SINVNG) C 260 CALL KR0N2(UNITYG. TXCCT. GRP. GRP. OCC, OCC, TPROJ) DO 270 1=1. OCCGRP 270 WRITE (7.280) (TPROJ(I.J),J=l.OCCGRP) 280 FORMAT (9F7.1) C C PRODUCES GTPROJ MULTIPLIED BY (OCC * NTOT) C CALL KR0N2(FMIDFT. TXCCT. GRP. GRP. OCC. OCC. GTPROJ) DO 290 I = 1. OCCGRP 290 WRITE (7.280) (GTPROJ(I,J),J=l.OCCGRP) Computer Programs for the Monte Carlo Simulations / B.l C PRODUCES TSPROJ(G) MULTIPLIED BY NG*0CC [G=l GRP] C DO 330 G - 1. GRP NG = N(NRUN.G) DO 300 I = 1, NG DO 300 J - 1. NG IF (I .EQ. J) NGXHHT(G,I,J) = FLOAT(NG) - 1.0 IF (I .NE. J) NGXHHT(G,I,J) = -1.0 300 CONTINUE C CALL KR0N3(G. NGXHHT, TXCCT. NG. NG. OCC. OCC, TSPROJ) NGXOCC = NG * OCC DO 310 1=1, NGXOCC 310 WRITE (7,320) (TSPROJ(G,I.J),J=l.NGXOCC) 320 FORMAT (72F7.1) 330 CONTINUE C 998 CONTINUE C STOP END C *T^- "t^ t^1* 'fr* *TT^ 4^ 5 ^ 't^ *^T* -^ TT* 4^ STJC s4<i 5 ^ "t* 4^ 4^ *Hr^ C END OF MAIN PROGRAM / BEGINNING OF SUBROUTINES Q ******************************************** C C PERFORMS A CHOLESKY FACTORIZATION OF THE INPUT MATRIX: C RETURNS A VECTOR, B, CONTAINING ELEMENTS OF THE FACTOR C WHICH IS A LOWER TRIANGULAR MATRIX C SUBROUTINE CHOL(LEVEL, SG, NDIM, B) DOUBLE PRECISION B(3.21). SG(3,6.6), X. Y K = 0 DO 20 I = 1, NDIM DO 10 J = 1, I K = K + 1 B(LEVEL.K) = SG(LEVEL,I,J) 10 CONTINUE 20 CONTINUE C DO 60 I = 1. NDIM DO 60 J = I. NDIM INDEX1 = (J*J - J) / 2 + I X = B(LEVEL,INDEX1) IF (I .EQ. 1) GO TO 40 KF = I - 1 DO 30 K = 1, KF Computer Programs for the Monte Carlo Simulations / B.l 91 INDEX2 = (1*1 - I ) / 2 + K INDEX3 = ( J * J - J ) / 2 + K X = X - B ( L E V E L , I N D E X 2 ) * B ( L E V E L , I N D E X 3 ) 30 CONTINUE C 40 I F ( I . N E . J ) GO TO 50 I F (X . L E . 0 . D 0 ) GO TO 70 Y = DSQRT(X) 50 B ( L E V E L , I N D E X 1 ) = X / Y 60 CONTINUE C C GO TO 90 70 WRITE ( 6 . 8 0 ) 80 FORMAT ( / ' MATRIX NOT P . D . ' / ) 90 CONTINUE C DO 110 I = 1. NDIM L L = ( ( 1 * 1 ) - I ) / 2 + 1 KK = ( ( 1 * 1 ) + I ) / 2 WRITE ( 6 . 1 0 0 ) ( B ( L E V E L , L ) , L = L L . K K ) 100 FORMAT ( 6 ( F 9 . 2 . 1 X ) ) 110 CONTINUE C WRITE ( 6 . 1 2 0 ) L E V E L 120 FORMAT ( / ' LOWER TRIANGULAR FACTOR: S I G ( \ I I , ' ) ' / / ) RETURN END C C FORMS KRONECKER PRODUCTS: 2 - D ARRAYS C SUBROUTINE K R 0 N 2 ( A , B . I T , J T , I P , J P , AB) DIMENSION A ( 3 , 3 ) , B ( 3 , 3 ) , A B ( 9 , 9 ) DO 40 I = 1, I T IMO = I - 1 DO 30 IB = 1 . IP I I = ( IMO* IP) + IB DO 20 J = 1 , J T JMO = J - 1 DO 10 J B = 1 . J P J J - ( J M O * J P ) + J B A B ( I I . J J ) = A ( I . J ) * B ( I B . J B ) 10 CONTINUE 20 CONTINUE 30 CONTINUE 40 CONTINUE C WRITE ( 6 . 5 0 ) Computer Programs for the Monte Carlo Simulations / B.l 50 FORMAT ( / / / ) I A S I B S = I T * IP J A S J B S = J T * J P DO 60 I I = 1 , I A S I B S 60 WRITE ( 6 . 7 0 ) ( A B ( I I , J J ) . J J = 1 . J A S J B S ) 70 FORMAT ( 9 ( F 7 . 1 . 2 X ) ) C RETURN END C C FORMS KRONECKER PRODUCTS: 3 -D ARRAYS C SUBROUTINE K R 0 N 3 ( L E V E L . A . B . I T . J T . I P . J P . AB) DIMENSION A ( 3 , 2 4 . 2 4 ) . B ( 3 . 3 ) . A B ( 3 . 7 2 . 7 2 ) DO 40 I = 1 . I T IMO = 1 - 1 DO 30 IB - 1. IP I I = ( IMO* IP) + IB DO 20 J = 1. J T JMO = J - 1 DO 10 J B = 1 . J P J J = ( J M O * J P ) + J B A B ( L E V E L . I I . J J ) = A ( L E V E L . I . J ) * B ( I B . J B ) 10 CONTINUE 20 CONTINUE 30 CONTINUE 40 CONTINUE C WRITE ( 6 . 4 5 ) 45 FORMAT ( / / / ) WRITE ( 6 . 5 0 ) L E V E L 50 FORMAT (/* MATRIX USED TO PRODUCE ERROR SSCP \ I I A S I B S = I T * IP J A S J B S = J T * J P C DO 80 I I = 1 , I A S I B S WRITE ( 6 . 6 0 ) ( A B ( L E V E L . I I . J J ) , J J = 1 . J A S J B S ) 60 FORMAT ( 1 2 ( F 7 . 1 , 2 X ) ) WRITE ( 6 . 7 0 ) 70 FORMAT ( / / ) 80 CONTINUE C RETURN END Computer Programs for the Monte Carlo Simulations / B.2 B.2 T H E S I M I L A T I O N P R O G R A M DIMENSION N(6,3) INTEGER GRP. OCC. VAR. REPS. OCCVAR. G. T. P. RUNS INTEGER OCCGRP. PREVHI. GTLOW. GTHIGH. PP INTEGER CGT01. CGT05. CT01. CT05 DOUBLE PRECISION UCRIT(6.4). L0WTRI(3.21) DOUBLE PRECISION TPROJ(9,9), GTPROJ(9.9) DOUBLE PRECISION TSPR0J(3.108.108). TSSSCP(3.2.2) DOUBLE PRECISION N0K3.6.36). Y(3.2.108). Z(2.9) DOUBLE PRECISION TSSCP(2.2). GTSSCP(2,2) DOUBLE PRECISION ERR0R(2.2). HGTPE(2.2). HTPE(2.2) DOUBLE PRECISION WILKT. WILKGT. SINVNG, NTOT C C READS IN NECESSARY PARAMETERS AND THE MATRICES GENER-C ATED BY THE FAST.PREP PROGRAM. C C READ (7,10) GRP, OCC, VAR. REPS. SEED, RUNS 10 FORMAT (3(11.IX). 14. IX. F6.1. IX. II) C OCCVAR = OCC * VAR OCCGRP = OCC * GRP LAST = (OCCVAR*(OCCVAR + 1)) / 2 C DO 20 G = 1. GRP 20 READ (7.30) (LOWTRI(G.L),L=1.LAST) 30 FORMAT (21F12.9) C DO 40 NRUN = 1. RUNS 40 READ (7.50) (N(NRUN,G),G=1.GRP) 50 FORMAT (3(I3.3X)) C DO 60 NRUN = 1. RUNS 60 READ (7,70) (UCRIT(NRUN.J).J=l.4) 70 FORMAT (4(F6.4.1X)) C C C START OF LOOP 1: TO RUN VARIOUS N/ (GRP * VAR) AND C BALANCE COMBINATIONS WITH A GIVEN SET OF COVARIANCE C MATRICES. C DO 560 NRUN = 1. RUNS Computer Programs for the Monte Carlo Simulations / B.2 94 C C NTOT = O.ODO SINVNG = O.ODO C DO 80 I = 1. OCCGRP 80 READ (7.90) (TPR0J(I.J).J=l.OCCGRP) 90 FORMAT (9F7.1) DO 100 I = 1. OCCGRP 100 READ (7.90) (GTPROJ(I.J).J=l.OCCGRP) C DO 130 G = 1. GRP NG = N(NRUN.G) NGXOCC = NG * OCC DO 110 I = 1, NGXOCC 110 READ (7.120) (TSPROJ(G.I.J),J=1.NGXOCC) 120 FORMAT (72F7.1) C NTOT = NTOT + DFLOAT(NG) SINVNG = SINVNG + (1.0 / DFLOAT(NG)) 130 CONTINUE C C INITIALIZES TYPE I ERROR COUNTERS C CGT01 = 0 CGT05 = 0 CT01 = 0 CT05 = 0 C WRITE (6.165) NRUN 165 FORMAT (////' RUN '. II//) WRITE (6.170) 170 FORMAT (/' CRITICAL VALUES OF U DISTRIBUTIONS'/) WRITE (6.180) UCRIT(NRUN.l). UCRIT(NRUN.2) 180 FORMAT (/' UGT05 = '. F6.3. ' UT05 = ', F6.3/) WRITE (6.190) UCRIT(NRUN,3). UCRIT(NRUN.4) 190 FORMAT (/' UGT01 = ', F6.3. ' UT01 = '. F6.3//) WRITE (6.200) 200 FORMAT (/' ACTUAL VALUES OF TEST STATISTICS') WRITE (6.210) 210 FORMAT (/' REP. LAMBDA GT LAMBDA T'//) C C INITIALZES THE N(O.l) RANDOM NUMBER GENERATOR, RANDN C E = RANDN(SEED) C C c ************************************************ Computer Programs for the Monte Carlo Simulations / B.2 C START OF 1000 REPS: THE SAMPLE DATA MATRICES ARE GEN-C ERATED; SAMPLE VALUES OF TEST STATISTICS ARE CALC-C ULATED; COMPARISONS ARE MADE WITH CRITICAL VALUES; C AND TYPE I ERRORS ARE COUNTED. C C DO 490 NR = 1, REPS C C GENERATES OCCVAR X N(G) MATRICES. N01. OF INDEPEND-C ENT N(O.l) VARIATES (G = 1 GRP). C DO 240 G = 1. GRP NG = N(NRUN.G) DO 230 ITP = 1. OCCVAR DO 220 NSUB = 1, NG NOl(G.ITP.NSUB) = FRANDNCO.) 220 CONTINUE 230 CONTINUE 240 CONTINUE C C TRANSFORMS THE N01 MATRICES TO Y MATRICES USING THE C CHOLESKY FACTORS (LOWTRI) OF THE COVARIANCE MATRICES. C C SUBJECT I IN GROUP G HAS OCC VECTORS OF LENGTH VAR C IN THE ARRAY Y—A BLOCK WHICH CAN BE CALLED Y(I,G). C C THE VECTOR VEC[ Y(I,G) ] HAS A MULTIVARIATE NORMAL C DISTRIBUTION WITH A 0 CENTROID AND COVARIANCE MATRIX C SIG(G). C C C FOR EACH GROUP. A MEAN VALUE OF EACH OF THE OCC*VAR C MEASURES IS CALCULATED. (THESE AVERAGES ARE TAKEN C ACROSS SUBJECTS.) C C FIRST. INITIALIZES THE Z MATRIX OF VAR*OCC*GRP MEANS. C DO 250 I = 1. VAR DO 250 J = 1, OCCGRP Z(I.J) = O.ODO 250 CONTINUE C DO 310 G = 1, GRP NG = N(NRUN.G) C DO 290 NSUB = 1, NG C Computer Programs tor the Monte Carlo Simulations / B.2 DO 280 T - 1. OCC JNSUBT = (NSUB - 1) * OCC + T JGT - (G - 1) * OCC + T C DO 270 P = 1, VAR C ITP - ( T - 1) * VAR + P X = O.ODO PREVHI = (ITP*ITP - ITP) / 2 C DO 260 I = i , ITP L = PREVHI + I X - X + LOWTRI(G„L) * N01(G.I.NSUB) 260 CONTINUE C Y(G.P.JNSUBT) = X Z(P.JGT) = Z(P,JGT) + X 270 CONTINUE C 280 CONTINUE 290 CONTINUE C GTLOW = (G - 1) * OCC + 1 GTHIGH = G * OCC C DO 300 PP = 1. VAR DO 300 J = GTLOW. GTHIGH Z(PP.J) = Z(PP.J) / DFLOAT(NG) 300 CONTINUE C 310 CONTINUE C C PRODUCES SSCP MATRIX, GTSSCP, TO TEST FULL PARALLEL-C ISM NULL HYPOTHESIS. GTSSCP = Z * GTPROJ * Z* C CALL QUAD(Z, GTPROJ, VAR. OCCGRP. 2, 9, 2. GTSSCP) C C PRODUCES SSCP MATRIX, TSSCP. TO TEST FULL OCCASIONS C NULL HYPOTHESIS. TSSCP = Z * TPROJ * Z' C CALL QUAD(Z. TPROJ. VAR. OCCGRP. 2, 9, 2, TSSCP) C C PRODUCES TSSSCP(l) TO TSSSCP(GRP), THE ERROR MATRICES C FOR THE GROUPS [TSSSCP(G) = Y(G) * TSPROJ(G) * Y'(G)] C WHICH ARE SUMMED TO FORM THE ERROR SSCP MATRIX USED C TO TEST BOTH WITHIN HYPOTHESES. C DO 320 I - 1. VAR Computer Programs tor the Monte Carlo Simulations / B.2 DO 320 J - 1. VAR ERROR(I.J) = O.ODO 320 CONTINUE C DO 380 G - 1. GRP NGXOCC = N(NRUN.G) * OCC C DO 360 I = 1, VAR C DO 350 J - 1. I TSSSCP(G,I,J) = O.ODO C DO 340 K = 1. NGXOCC X = O.ODO C DO 330 L = 1, NGXOCC X = X + Y(G.I.L) * TSPROJ(G,L,K) 330 CONTINUE C TSSSCP(G,I,J) = TSSSCP(G,I,J) + X * Y(G.J.K) 340 CONTINUE C TSSSCP(G.I.J) = TSSSCP(G,I,J) / DFLOAT(N(NRUN.G)) 350 CONTINUE 360 CONTINUE C DO 370 II = 1. VAR DO 370 JJ = 1, II 370 ERROR(II,JJ) = ERROR(II.JJ) + TSSSCP(G,II,JJ) / : DFLOAT(OCC) C 380 CONTINUE C C PRODUCES THE HYPOTHESIS + ERROR MATRIX: PARALLELISM C DO 390 1 = 1 , VAR DO 390 J = 1, I 390 HGTPE(I.J) = ERROR(I.J) + GTSSCP(I.J) / : (NTOT * DFLOAT(OCC)) C C PRODUCES THE HYPOTHESIS + ERROR MATRIX: OCCASIONS C DO 400 I = 1. VAR DO 400 J = 1. I 400 HTPE(I.J) = ERROR(I.J) + TSSCP(I.J) / : (SINVNG * DFLOAT(OCO) C C CALCULATES THE TEST STATISTIC VALUES Computer Programs for tie Monte Cario Simu/ations / B.2 98 WILKT = ((ERR0R(1,1)*ERR0R(2.2)) - (ERR0R(2,1) : *ERR0R(2,1))) / : ((HTPE(1.1)*HTPE(2.2)) - (HTPE(2,1)*HTPE(2.1))) WILKGT - ((ERR0R(1,1)*ERR0R(2.2)) - (ERR0R(2,1) : *ERR0R(2.1))) / ((HGTPE(1,1)*HGTPE(2,2)) : - (HGTPE(2,1)*HGTPE(2.1))) C C COUNTS THE TYPE I ERRORS C 410 IF (WILKGT .GE. UCRIT(NRUN,3)) GO TO 420 CGT01 = CGT01 + 1 CGT05 = CGT05 + 1 IGT01 = 1 IGT05 = 1 GO TO 440 C 420 IF (WILKGT .GE. UCRIT(NRUN,1)) GO TO 430 CGT05 = CGT05 + 1 IGT01 = 0 IGT05 = 1 GO TO 440 C 430 IGT01 = 0 IGT05 = 0 440 IF (WILKT .GE. UCRIT(NRUN.4)) GO TO 450 CT01 = CTOl + 1 CT05 = CT05 + 1 ITOl = 1 IT05 = 1 GO TO 470 C 450 IF (WILKT .GE. UCRIT(NRUN,2)) GO TO 460 CT05 = CT05 + 1 ITOl = 0 IT05 = 1 GO TO 470 C 460 ITOl = 0 IT05 = 0 C 470 WRITE (6.480) NR. WILKGT, IGT01, IGT05. WILKT, ITOl, : IT05 480 FORMAT (IX. 14, 4X, F7.4. 2X. 211. 5X, F7.4. 2X. 211) C C 490 CONTINUE C Computer Programs tor the Monte Carlo Simulations / B.2 WRITE (6.500) NRUN 500 FORMAT (///' COUNTS OF ACTUAL TYPE I ERRORS: RUN '. : ID WRITE (6.510) 510 FORMAT (/' HYPOTHESIS NOM. ALPHA ERRORS') WRITE (6.520) CGT01 520 FORMAT (//' PARALLELISM 0.01 '. 4X. 14) WRITE (6.530) CGT05 530 FORMAT (//' PARALLELISM 0.05 ', 4X, 14) WRITE (6.540) CT01 540 FORMAT (//' OCCASIONS 0.01 '. 4X. 14) WRITE (6.550) CT05 550 FORMAT (//' OCCASIONS 0.05 '. 4X. 14) C C 560 CONTINUE STOP END C C C END OF MAIN PROGRAM / BEGINNING OF SUBROUTINE C C SUBROUTINE QUAD(Z. A. IZ. JZ. IDIMZ. IDIMA. IDIMQ. Q) C C PRODUCES A MATRIX QUADRATIC FORM: Q = Z * A * Z' C DOUBLE PRECISION Z(IDIMZ.l). A(IDIMA.l). Q(IDIMQ.l) DO 30 I = 1. IZ DO 30 J = 1. I Q(I.J) = O.ODO C DO 20 K = 1. JZ X = O.ODO C DO 10 L = 1. JZ X = X + Z(I.L) * A(L.K) 10 CONTINUE C Q(I.J) = Q(I.J) + X * Z(J.K) 20 CONTINUE C 30 CONTINUE RETURN END Computer Programs tor the Monte Carlo Simulations / B.3 100 B.3 B E N C H M A R K T E S T S It is expected that in simulated experiments in which the two or three populations have the same covariance matrix 53 with (C'®l)E(C®l) = I®E* for some E*, each empirical value of a should be within sampling error of a n o m. Benchmark tests of two types were conducted to check (a) the operation of the computer program used for the simulations and (b) the validity of this expectation. In four benchmark experiments of the first type, each population sampled had E A from the pool as its covariance matrix. (See Appendix A.) Matrix E A has the required structure: ( C ® I ) S A ( C ® I ) = I ® ( : « ° : » ° ) . Equal and unequal sample sizes were used as were N/(GP) values of six and 12. (Actual sample sizes used were 12:12, 8:16, 24:24, and 16:32 when G was two and 12:12:12, 9:9:18, 24:24:24, and 18:18:36 when G was three.) Table 13 presents the findings from these experiments. All empirical a values were within sampling error of their respective or n o m values. Twelve benchmark experiments of the second type were conducted using population matrices E B and E c which satisfy the MMS condition: (C'®l)E B(C®l) = (C'®I)E C(C®I) T /80 10\ It was expected that in the experiments in which sample sizes were unequal, Type I error rate would not be related to the choice of population from which the sample of larger size was drawn. This conjecture was tested. In four of these experiments, the sample of the larger size was drawn from a population having Eg as its covariance matrix. In the other four, a greater number of observation vectors was drawn from the E c population. When G was three, the experiments simulated were as follows: Expt. E ? and ng Computer Programs for the Monte Carlo Simulations / B.3 101 Sample specifications G = 2 G = 3 1:1 1:2 1:1:1 1:1:2 H0 1000 x a nom N/{GP) = 6 GT 50 10 46 7 40 7 47 4 45 9 T 50 10 52 7 37 7 48 8 46 8 N/(GP) = 12 GT 50 10 43 9 49 8 53 9 59 14 T 50 10 49 10 49 12 57 5 55 12 TABLE 13 Number of Type I Errors in 1000 Replications for Benchmark Tests of the First Type. Computer Programs for the Monte Carlo Simulations / B.3 102 Sample specifications N/(GP) = 6 N/{GP) = 12 1 : 1 1 : 2 2:1 1:1 : 1 1:1:2" 1:1:2" H 0 1000 x a nom N/(GP) = 6 GT 50 10 49 10 43 5 45 7 41 8 39 7 40 6 T 50 10 34* 8 35* 9 51 8 40 9 52 14 55 14 N/{GP) = 12 GT 50 10 48 5 44 6 36* 4 50 9 48 13 51 13 T 50 10 44 5 36* 2* 49 8 46 5 53 7 54 5 a Matrices: Eg : Eg : E Q b Matrices: E c : E c : Eg T A B L E 14 Number of Type I Errors in 1000 Replications for Benchmark Tests of the Second Type. Computer Programs for the Monte Carlo Simulations / B.3 103 Group 1 Group 2 Group 3 1 s B 12 s B 12 S c 12 2 s B 9 s B 9 £ 0 18 3 S c 9 9 s B 18 4 s B 24 s B 24 24 5 s B 18 s B 18 S 0 36 6 S c 18 18 s B 36 Table 14 displays results from these simulations. The above conjecture ap-pears to have been correct. Thirty-three percent of these experiments produced at least one empirical a value outside the 0.95 confidence band for a n o m. Each such a value was lower than the corresponding o t n o m value. It is interesting to note that four of the five low a values occurred when the sample sizes were unequal. Because a set of calculations performed by the program checked with hand calculations, and because the results for the first type of benchmark test indicated no systematic bias, these results were attributed to sampling variability.
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- Robustness of multivariate mixed model ANOVA
Open Collections
UBC Theses and Dissertations
Featured Collection
UBC Theses and Dissertations
Robustness of multivariate mixed model ANOVA Prosser, Robert James 1985
pdf
Page Metadata
Item Metadata
Title | Robustness of multivariate mixed model ANOVA |
Creator |
Prosser, Robert James |
Publisher | University of British Columbia |
Date Issued | 1985 |
Description | In experimental or quasi-experimental studies in which a repeated measures design is used, it is common to obtain scores on several dependent variables on each measurement occasion. Multivariate mixed model (MMM) analysis of variance (Thomas, 1983) is a recently developed alternative to the MANOVA procedure (Bock, 1975; Timm, 1980) for testing multivariate hypotheses concerning effects of a repeated factor (called occasions in this study) and interaction between repeated and non-repeated factors (termed group-by-occasion interaction here). If a condition derived by Thomas (1983), multivariate multi-sample sphericity (MMS), regarding the equality and structure of orthonormalized population covariance matrices is satisfied (given multivariate normality and independence for distributions of subjects' scores), valid likelihood-ratio MMM tests of group-by-occasion interaction and occasions hypotheses are possible. To date, no information has been available concerning actual (empirical) levels of significance of such tests when the MMS condition is violated. This study was conducted to begin to provide such information. Departure from the MMS condition can be classified into three types— termed departures of types A, B, and C respectively: (A) the covariance matrix for population ℊ (ℊ = 1,...G), when orthonormalized, has an equal-diagonal-block form but the resulting matrix for population ℊ is unequal to the resulting matrix for population ℊ' (ℊ ≠ ℊ'); (B) the G populations' orthonormalized covariance matrices are equal, but the matrix common to the populations does not have equal-diagonal-block structure; or (C) one or more populations has an orthonormalized covariance matrix which does not have equal-diagonal-block structure and two or more populations have unequal orthonormalized matrices. In this study, Monte Carlo procedures were used to examine the effect of each type of violation in turn on the Type I error rates of multivariate mixed model tests of group-by-occasion interaction and occasions null hypotheses. For each form of violation, experiments modelling several levels of severity were simulated. In these experiments: (a) the number of measured variables was two; (b) the number of measurement occasions was three; (c) the number of populations sampled was two or three; (d) the ratio of average sample size to number of measured variables was six or 12; and (e) the sample size ratios were 1:1 and 1:2 when G was two, and 1:1:1 and 1:1:2 when G was three. In experiments modelling violations of types A and C, the effects of negative and positive sampling were studied. When type A violations were modelled and samples were equal in size, actual Type I error rates did not differ significantly from nominal levels for tests of either hypothesis except under the most severe level of violation. In type A experiments using unequal groups in which the largest sample was drawn from the population whose orthogonalized covariance matrix has the smallest determinant (negative sampling), actual Type I error rates were significantly higher than nominal rates for tests of both hypotheses and for all levels of violation. In contrast, empirical levels of significance were significantly lower than nominal rates in type A experiments in which the largest sample was drawn from the population whose orthonormalized covariance matrix had the largest determinant (positive sampling). Tests of both hypotheses tended to be liberal in experiments which modelled type B violations. No strong relationships were observed between actual Type I error rates and any of: severity of violation, number of groups, ratio of average sample size to number of variables, and relative sizes of samples. In equal-groups experiments modelling type C violations in which the orthonormalized pooled covariance matrix departed at the more severe level from equal-diagonal-block form, actual Type I error rates for tests of both hypotheses tended to be liberal. Findings were more complex under the less severe level of structural departure. Empirical significance levels did not vary with the degree of interpopulation heterogeneity of orthonormalized covariance matrices. In type C experiments modelling negative sampling, tests of both hypotheses tended to be liberal. Degree of structural departure did not appear to influence actual Type I error rates but degree of interpopulation heterogeneity did. Actual Type I error rates in type C experiments modelling positive sampling were apparently related to the number of groups. When two populations were sampled, both tests tended to be conservative, while for three groups, the results were more complex. In general, under all types of violation the ratio of average group size to number of variables did not greatly affect actual Type I error rates. The report concludes with suggestions for practitioners considering use of the MMM procedure based upon the findings and recommends four avenues for future research on Type I error robustness of MMM analysis of variance. The matrix pool and computer programs used in the simulations are included in appendices. |
Subject |
Analysis of variance |
Genre |
Thesis/Dissertation |
Type |
Text |
Language | eng |
Date Available | 2010-06-09 |
Provider | Vancouver : University of British Columbia Library |
Rights | For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use. |
DOI | 10.14288/1.0096513 |
URI | http://hdl.handle.net/2429/25511 |
Degree |
Master of Arts - MA |
Program |
Special Education |
Affiliation |
Education, Faculty of Educational and Counselling Psychology, and Special Education (ECPS), Department of |
Degree Grantor | University of British Columbia |
Campus |
UBCV |
Scholarly Level | Graduate |
AggregatedSourceRepository | DSpace |
Download
- Media
- 831-UBC_1985_A8 P76.pdf [ 6.06MB ]
- Metadata
- JSON: 831-1.0096513.json
- JSON-LD: 831-1.0096513-ld.json
- RDF/XML (Pretty): 831-1.0096513-rdf.xml
- RDF/JSON: 831-1.0096513-rdf.json
- Turtle: 831-1.0096513-turtle.txt
- N-Triples: 831-1.0096513-rdf-ntriples.txt
- Original Record: 831-1.0096513-source.json
- Full Text
- 831-1.0096513-fulltext.txt
- Citation
- 831-1.0096513.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0096513/manifest