Education, Faculty of
Language and Literacy Education (LLED), Department of
DSpace
UBCV
Yan, Zheng
Master of Arts - MA
University of British Columbia
The Test of English as a Foreign Language (TOEFL) has been used in making admission decisions for over 30 years; however, the predictive validity of the test has been uncertain. The present study was intended to investigate the predictive validity of TOEFL scores on first term's grade point average (GPA). Participants were 97 second-year university students, 46 male and 52 female, in an international academic exchange program. Most majored in Humanities and Social Sciences. The predictor variables in the study included TOEFL total scores, TOEFL section I scores, TOEFL section II scores, TOEFL section III scores, oral proficiency interview scores, writing sample scores, and gender. First term's GPA was the criterion variable. The data were analyzed by multiple regression analysis with a hierarchical procedure. The results were interpreted on the basis of Cohen's (1988) conventional definitions on the effect size of R². The main findings of the study indicate that: (a) TOEFL total scores have a medium level of predictive validity on GPA (ΔR²=.142, ϱ<.001); (b) TOEFL section I scores have a medium level of predictive validity (ΔR²=.044, ϱ<.05); (c) TOEFL section II scores have a medium level of predictive validity (ΔR²=.112, ϱ<.001); (d) TOEFL section III scores have a negligible level of predictive validity (ΔR² =.005, ϱ>.05); (e) Oral proficiency interviews scores have a negligible level of predictive validity (ΔR² =.010, ϱ>.05); (f) Writing samples scores have a small level of predictive validity (ΔR² =.047, ϱ<.05); And (g) gender has a medium level of predictive validity (ΔR² =.130, ϱ<.001). The findings of the study thus validate the use of TOEFL scores as one of the requirements for admission in the international exchange program and provide new empirical evidence for investigation of the relationship between language proficiency and academic achievement.
4362341 bytes
application/pdf
PREDICTIVE VALIDITY. OF TOEFL SCORES ON FIRST TERM'S GPA AS THE CRITERION FOR INTERNATIONAL EXCHANGE STUDENTS by ZHENG YAN M.Ed.., N o r t h e a s t Normal U n i v e r s i t y , 1986 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS i n THE FACULTY OF GRADUATE STUDIES Depa r t m e n t o f Language E d u c a t i o n We a c c e p t t h i s t h e s i s a s c o n f o r m i n g t o t h e r e q u i r e d s t a n d a r d THE UNIVERSITY OF BRITISH COLUMBIA March, 1995 © Zheng Yan, 1995 In presenting t h i s thesis i n p a r t i a l f u l f i l l m e n t of the requirements f o r an advanced degree at the University of B r i t i s h Columbia, I agree that the Library s h a l l make i t f r e e l y available for reference and study. I further agree that permission for extensive copying of t h i s thesis for scholarly purposes may be granted by the Head of my Department or by h i s or her representatives. I t i s understood that copying or publication of t h i s t h e s i s f o r f i n a n c i a l gain s h a l l not be allowed without written permission. L Department of The University of B r i t i s h Columbia Vancouver, Canada Date Abstract The Test of English as a Foreign Language (TOEFL) has been used i n making admission decisions for over 30 years; however, the predictive v a l i d i t y of the t e s t has been uncertain. The present study was intended to investigate the pred i c t i v e v a l i d i t y of TOEFL scores on f i r s t term's grade point average (GPA). Participants were 97 second-year un i v e r s i t y students, 46 male and 52 female, i n an international academic exchange program. Most majored i n Humanities and Social Sciences. The predictor variables i n the study included TOEFL t o t a l scores, TOEFL section I scores, TOEFL section II scores, TOEFL section III scores, oral p r o f i c i e n c y interview scores, writing sample scores, and gender. F i r s t term's GPA was the c r i t e r i o n variable. The data were analyzed by multiple regression analysis with a h i e r a r c h i c a l procedure. The res u l t s were interpreted on the basis of Cohen's (1988) conventional d e f i n i t i o n s on the e f f e c t s i z e of R 2 . The main findings of the study indicate that: (a) TOEFL t o t a l scores have a medium l e v e l of pre d i c t i v e v a l i d i t y on GPA (AR 2=.142, p_<.001); (b) TOEFL section I scores have a medium l e v e l of pre d i c t i v e v a l i d i t y (AR=.044, p_<.05); (c) TOEFL section II scores have a medium l e v e l of p r e d i c t i v e v a l i d i t y (AR 2=.112, p_<.001); (d) TOEFL section III scores have a n e g l i g i b l e l e v e l of pre d i c t i v e v a l i d i t y ( A R 2 =.005, p_>.05); (e) Oral proficiency interviews scores have a n e g l i g i b l e l e v e l of pre d i c t i v e v a l i d i t y (AR =.010, p_>.05); (f) Writing samples scores have a small l e v e l of p r e d i c t i v e v a l i d i t y (AR =.047, p_<.05); And (g) gender has a medium l e v e l of pre d i c t i v e v a l i d i t y (AR2=.130, p_<.001). T n e findings of the study thus validate the use of TOEFL scores as one of the requirements for admission i n the international exchange program and provide new empirical evidence for investigation of the re l a t i o n s h i p between language proficiency and academic achievement. iv TABLE OF CONTENTS Abstract i i Tables of contents i v L i s t of tables v i L i s t of figures v i i Acknowledgments v i i i Chapter One: Introduction 1 Research problem 1 Research questions 2 D e f i n i t i o n of terms 3 Chapter Two: Literature Review 6 Background 6 Part I: Factors influencing academic achievement 7 A conceptual structure 7 A c l a s s i f i c a t i o n scheme 9 Language factors and academic achievement 12 Non-language factors and academic achievement 16 Part I I : Factors a f f e c t i n g TOEFL's pr e d i c t i v e v a l i d i t y 20 An a l y t i c a l models 21 Subject variables 26 Predictor variables 29 C r i t e r i o n variables 32 Result interpretation 34 Summary 3 6 Chapter Three: Method 38 The program se t t i n g 38 Participants 40 V The predictor variables 41 The c r i t e r i o n variable 43 A n a l y t i c a l model 44 Operational d e f i n i t i o n s of the pr e d i c t i v e v a l i d i t y 45 Research Hypotheses 46 Summary 47 Chapter Four: Results 48 Treatment of the missing data 48 Descriptive s t a t i s t i c a l analysis 49 Checking for v i o l a t i o n of assumptions 52 Hier a r c h i c a l regression analysis 58 Summary 64 Chapter Five: Discussion 65 Predictive v a l i d i t y of TOEFL t o t a l scores 65 Predictive v a l i d i t y of TOEFL sectional scores 68 Predictive v a l i d i t y of writing scores 70 Predictive v a l i d i t y of speaking scores 71 Predictive v a l i d i t y of gender 72 Implications 74 Limitations 76 Directions for future research 77 Conclusions 78 Bibliography 80 Appendix I The data f i l e 95 Appendix II The l i s t of standardized residuals and leverage values 99 vi L i s t of Tables Table 3.1 Grade c r i t e r i o n on d i f f e r e n t aspects i n the three courses 45 Table 3.2 Four l e v e l s of the pred i c t i v e v a l i d i t y 46 Table 4.1 Means and standard deviations of a l l the variables 49 Table 4.2 Pearson c o r r e l a t i o n matrix of the variables 51 Table 4.3 Summary table of the h i e r a r c h i c a l analysis with TOEFL t o t a l scores 61 Table 4.4 Summary table of the h i e r a r c h i c a l analysis with TOEFL sectional scores 63 v i i L i s t of figures Figure 2.1 A two-level conceptual structure i n a study of TOEFL's p r e d i c t i v e v a l i d i t y 7 Figure 2.2 A f i v e - l e v e l c l a s s i f i c a t i o n scheme of factors influencing academic achievement 10 Figure 4.1 Scatterplot of the d i s t r i b u t i o n of the residuals 55 Figure 4.2 D i s t r i b u t i o n of residuals 56 Figure 5.1 Scatterplot of TOEFL t o t a l scores and GPA 67 Acknowledgments The growth of a flower, no matter how small, has to appreciate the Sun's enlightenment, the Rain's refreshment, and the Mother Earth's nutrition and grounding. Here, I wish to extend my heartfelt thanks to: Dr. Lee Gunderson, my M A program advisor, for giving over three years of tireless guidance; Dr. Richard Berwick and Dr. Stephen Carey, my research committee members, for their invaluable support and advice; Dr. Areigh Reichl for providing consultation on statistics and William McMichael for providing consultation on the UBC/Ritsumeikan program. They both read the whole thesis and gave constructive criticism; Sheri Wenman, Jean Hamilton, and the UBC/Ritsumeikan program's instructors and students for their constant support and cooperation for my thesis project; Dr. Robert Kantor, Director of ETS's TOEFL Program, for providing both professional consultation on TOEFL and a long list of ETS ' free publications; Dr. Xiufeng Liu, Dr. Dean Mellow, Bingzheng Liu, and particularly Dr. Nand Kishor for their inspiration in the development of my thesis project; Victoria Dixon, Lynda Hayward, and Elizabeth Crittenden for their laborious proof-reading on the different chapters of the thesis; Dr. William Mackey, Dr. David Robitaille, Dr. Karen Armstrong, Dr. Robert Conry, Dr. Bernard Mohan, Dr. Marshall Arlin, Dr. Jon Shapiro, Dr. Marion Crowhurst, and Dr. Judith Johnston for their inestimable support, teaching, and/or encouragement; My friends, White Harvey, Cathy Galloaher, Cuhui Zhao, Elizabeth Smith, Gary and Mary Gates, Jingzi Wang and Dr. Yuan Gao, Prof. Kunwei Wang, Dr. Lianqin Wang, Jim and Katherine Yuen, Zhong Liu, Dan Zhang and Yaoyao, Dr. Glen Dixon and Victoria Dixon, Roberta Buck, and Dr. Leigh Faulkner for their immeasurable support; My parents and brother for their deep understanding of and exhaustive financial support for my study at U B C ; And my wife, Jingkai Zhang, for sharing my stress and happiness. 1 Chapter One Introduction This chapter presents the research problem under study. The s p e c i f i c research questions and detailed d e f i n i t i o n s of terms are also given. Research Problem The Test of English as a Foreign Language (TOEFL) i s the most widely used t e s t of English as a foreign language i n the world. I t was f i r s t administered i n 34 countries i n 1964 (Oiler & Spolsky, 1979). At present, as reported by the Educational Testing Service (ETS), TOEFL i s given on a monthly basis at over 1,200 t e s t centers i n 175 countries or regions around the world, with a population of approximately 700,000 examinees every year (ETS, 1994a, 1994b). The primary function of TOEFL, as stated i n the l a t e s t TOEFL Test and Score Manual, i s "to measure the English pro f i c i e n c y of international students wishing to study at colleges and u n i v e r s i t i e s i n the United States and Canada" (ETS, 1992, p. 6). Although considerable evolution of TOEFL has occurred during i t s 30 years of development, the primary function has never changed. TOEFL scores are currently required for admission into undergraduate or graduate programs by more than 2,500 colleges and u n i v e r s i t i e s i n the USA and Canada (ETS, 1994c). A great deal of research has been conducted to v a l i d a t e the use of TOEFL (Hale, Stansfield, & Duran, 1984; ETS, 1994d). A large proportion of the research has explored 2 TOEFL*s pr e d i c t i v e v a l i d i t y with grade point average (GPA) as the c r i t e r i o n . Since English proficiency i s necessary to achieve academic success i n an English environment, there should be a p o s i t i v e r e l a t i o n s h i p between English p r o f i c i e n c y and academic achievement, and consequently a p o s i t i v e r e l a t i o n s h i p between TOEFL scores as an indicator of English proficiency and GPA as an indicator of academic achievement. Accordingly, TOEFL scores should have strong p r e d i c t i v e v a l i d i t y i n predicting GPA; however, TOEFL predict i o n studies have consistently revealed widely divergent r e s u l t s (Graham, 1987; Hale, S t a n s f i e l d , & Duran, 1984) . Although researchers generally agree that English language proficiency i s important for academic achievement, they have not yet been able to reach a consensus on TOEFL*s pre d i c t i v e v a l i d i t y . The problem i s , therefore, that TOEFL has been used worldwide by thousands of i n s t i t u t i o n s to make admission decisions for 30 years, but the p r e d i c t i v e v a l i d i t y of TOEFL i s s t i l l an unsolved question for professionals i n language education. Research Questions This study was designed to estimate the p r e d i c t i v e v a l i d i t y of TOEFL scores on GPA for students i n the 1993-94 UBC/Ritsumeikan Academic Exchange Program, which was j o i n t l y administered by the University of B r i t i s h Columbia (UBC) of Canada and Ritsumeikan University of Japan. As an i n s t i t u t i o n a l v a l i d i t y study, i t was intended to provide 3 empirical evidence to investigate whether TOEFL scores predict GPA, and to explore how language pr o f i c i e n c y i s related to academic achievement. The study addressed the following s p e c i f i c research questions: 1. Do TOEFL scores predict GPA for international exchange students? 2. Do grades measuring English writing and speaking a b i l i t i e s predict GPA for international exchange students? 3. Do non-language variables predict GPA for international exchange students? De f i n i t i o n s of Terms Predictive v a l i d i t y . V a l i d i t y refers to the appropriateness of inferences from t e s t scores or other forms of assessment (American Psychological Association, 1974, pp. 25-27). Based upon the kinds of inferences one might wish to draw from te s t scores, people t r a d i t i o n a l l y r e f e r to the following types of v a l i d i t y : c r i t e r i o n - r e l a t e d v a l i d i t y , including both pre d i c t i v e v a l i d i t y and concurrent v a l i d i t y , content v a l i d i t y , and construct v a l i d i t y . Predictive v a l i d i t y indicates the extent to which one can predict future performances from p r i o r information. Predictive variables. The information that i s used to make a pred i c t i o n i s t y p i c a l l y referred to as a p r e d i c t i v e variable or simply as a predictor. 4 C r i t e r i o n variables. The event or outcome to be predicted i s t y p i c a l l y referred to as a c r i t e r i o n v a r i a b l e or simply as a c r i t e r i o n . GPA. This i s an acronym for grade point average. I t i s used as a measure of academic achievement i n subjects or courses, usually obtained by d i v i d i n g the sum of the t o t a l grade points by the t o t a l number of courses. In the current study i t i s used as an indicator of u n i v e r s i t y academic achievement. TOEFL. This i s an acronym for the Test of English as Foreign Language. The current study uses TOEFL scores as indicators of English Language proficiency. Model. A model i s a hypothesized structure used for the i n v e s t i g a t i o n of i n t e r r e l a t i o n s between variables or hypotheses. After variables have been i d e n t i f i e d , or hypotheses have been advanced i n the course of inquiry, i t may be necessary to advance a model that provides a structure for the i n t e r r e l a t i o n s between the set of variables or hypotheses. Model building and model t e s t i n g are two strategies that can be employed i n inquiry. Both c o r r e l a t i o n and regression can contribute to model bu i l d i n g (See Husen, 1994, pp. 3865-3873). Language proficiency. This term means progress towards the attainment of a high degree of knowledge and s k i l l i n English language. In the present study, t h i s i s used i n t e n t i o n a l l y to d i s t i n g u i s h i t from language competence, language performance, and language aptitude. 5 Academic achievement. In t h i s study, academic achievement refers to performance by students i n academically oriented courses. I t i s interchangeable with academic success. 6 Chapter Two Literature Review Introduction The review of l i t e r a t u r e i n t h i s chapter focuses on research findings related to the p r e d i c t i v e v a l i d i t y of TOEFL scores with GPA as a c r i t e r i o n v a r i a b l e . The review i s divided into two parts: factors that influence academic achievement, and factors that a f f e c t the estimation of TOEFL's pr e d i c t i v e v a l i d i t y . Part I examines conceptual issues i n studies of the p r e d i c t i v e v a l i d i t y of TOEFL scores, while Part II concentrates on methodological issues. Background The Test of English as a Foreign Language i s a standardized t e s t which uses a multiple-choice format to evaluate the English language proficiency of non-native speakers. Between 1963 and 1976, TOEFL contained f i v e sections: Listening Comprehension, English Structure, Vocabulary, Reading Comprehension, and Writing A b i l i t y . Since September of 1976, TOEFL has consisted of three sections: Listening Comprehension, Structure and Written Expression, and Vocabulary and Reading Comprehension. The two forms of the t e s t d i f f e r i n t e s t i n g items used and t e s t i n g time allowed, but the score scale of both t e s t s i s the same. The d i f f e r e n t sections of the t e s t were designed to measure d i f f e r e n t language s k i l l s within the general domain of language proficiency. Three decades of t e s t i n g administration and extensive research have shown that TOEFL 7 has a high degree of r e l i a b i l i t y and v a l i d i t y (ETS, 1992, pp. 30-36). Part I: Factors Influencing Academic Achievement The f i r s t part of the l i t e r a t u r e review presents both a two-level conceptual structure i n the study of TOEFL's pr e d i c t i v e v a l i d i t y and a f i v e - l e v e l c l a s s i f i c a t i o n scheme of factors influencing academic achievement. I t w i l l then examine both language factors and non-language factors related to academic achievement. A conceptual structure TOEFL scores are but one indicator of language proficiency, while GPA i s but one indicator of academic achievement. In a sense, the r e l a t i o n s h i p between TOEFL scores and GPA i s a surface-level manifestation of the p a r a l l e l but underlying r e l a t i o n s h i p e x i s t i n g between Manifest level: Indicators TOEFL scores Latent level: Constructs / GPA Language Proficiency Academic achievement Conventional measurement: Predictor & Criterion variables Underlying rationale: Independent & Dependent variables Figure 2.1. A two-level conceptual structure i n the study of TOEFL's pr e d i c t i v e v a l i d i t y . g language prof i c i e n c y and academic achievement. This r e l a t i o n s h i p i s i l l u s t r a t e d i n Figure 2.1. I t i s shown that s t r u c t u r a l l y a TOEFL pre d i c t i o n study has two portions: the manifest l e v e l and the latent l e v e l . This i s analogous i n structure to an iceberg, i t s v i s i b l e part being above sea l e v e l and the rest below sea l e v e l . The manifest-level portion i s a conventional s t a t i s t i c a l measurement of the r e l a t i o n s h i p between TOEFL scores as a predictor and GPA as a c r i t e r i o n . The l a t e n t - l e v e l portion i s a t h e o r e t i c a l assumption about the r e l a t i o n s h i p between language proficiency as an independent variable and academic achievement as a dependent variable. Figure 2.1 also demonstrates that these two portions are not separate from but harmonize with each other. In a TOEFL predic t i o n study, the underlying t h e o r e t i c a l assumption about the r e l a t i o n s h i p between language profi c i e n c y and academic achievement should j u s t i f y s t a t i s t i c a l methods used to measure TOEFL's pr e d i c t i v e v a l i d i t y , while the conventional measurement of the r e l a t i o n s h i p between TOEFL scores and GPA should f i t the underlying t h e o r e t i c a l rationale. For every study, i n fact, the research method used ought to match well with the proposed t h e o r e t i c a l assumption. For example, one might conduct a c o r r e l a t i o n study to analyze the r e l a t i o n s h i p between children's IQ and the s i z e of shoes they wear. However, t h i s study would not make any sense because the 9 s t a t i s t i c a l method i n the study, no matter how well i t would be u t i l i z e d , lacks a l o g i c a l supporting rationale. Clearly, a good TOEFL prediction study needs a proper s t a t i s t i c a l approach, but more important, i t l a r g e l y r e l i e s on a strong rationale. This i s simply because the hypothesized r e l a t i o n s h i p between language pro f i c i e n c y and academic achievement influences how the r e l a t i o n s h i p between TOEFL scores and GPA i s measured. Unfortunately, the issue of the underlying rationale for the TOEFL predic t i o n study has been repeatedly ignored. Thus, i n proposing such a two-l e v e l conceptual structure for TOEFL prediction studies, the intention i s to emphasize the importance of a comprehensive examination of factors influencing academic achievement. This examination serves to e s t a b l i s h a s o l i d r ationale underlying the measurement of the p r e d i c t i v e v a l i d i t y of TOEFL scores. A c l a s s i f i c a t i o n scheme Numerous studies have documented a great v a r i e t y of factors influencing academic achievement, such as i n t e l l i g e n c e , language, motivation, personality, i n t e r e s t , age, teacher expectation, e t h n i c i t y , learning s t y l e , teaching strategies, family involvement, classroom environment, and peer pressure. Since i t i s nearly impossible to l i s t a l l of these factors within a l i m i t e d space, the present study grouped them into a f i v e - l e v e l c l a s s i f i c a t i o n scheme sequenced from external factors to i n t e r n a l factors. This c l a s s i f i c a t i o n i s preliminary and 10 h e u r i s t i c . I t i s used to show the hierarchical- structure and complicated i n t e r r e l a t i o n s of the various factors related to academic achievement (See Figure 2 . 2 ) . External factors Social variables Educational variables Language variables $ Psychological variables Internal factors Physiological variables Academic achievement Figure 2 . 2 . A f i v e - l e v e l c l a s s i f i c a t i o n scheme of factors influencing academic achievement. The f i r s t major type of variable influencing academic achievement i s s o c i a l . This category includes s o c i a l development, socioeconomic status, c u l t u r a l background, e t h n i c i t y , s o c i a l attitude, family environment, parental involvement, morals and values, marital status, employment chances, and r e l i g i o u s b e l i e f s . The second major type of variable influencing academic achievement i s educational. Examples include curriculum implementation, educational objectives, i n s t r u c t i o n a l materials, teaching approaches, c h a r a c t e r i s t i c s of students, c h a r a c t e r i s t i c s of teachers, classroom interactions, time 11 spent on learning, p r i o r knowledge, learning s t y l e , teacher expectation, school assessment and evaluation, subject matter, students' status, and classroom environment. The t h i r d i s a l i n g u i s t i c category with such variables as f i r s t language(LI), second language(L2), bilingualism, reading, speaking, l i s t e n i n g , writing, genre, language proficiency, communicative competence, receptive s k i l l s , productive s k i l l s , vocabulary, and meta-awareness of language. The fourth category consists of psychological variables such as motivation, cognition, emotion, personality, attention, attitude, i n t e r e s t , aptitude, anxiety, creation, temperament, and self-esteem. Physiological variables represent the f i f t h category, including gender, genetic factors, maturation, f i t n e s s , brain l a t e r a l i z a t i o n , aging, health, and n u t r i t i o n . This c l a s s i f i c a t i o n scheme reveals how a large numbers of factors may contribute to academic achievement. Language factors are only one group among f i v e which influence academic achievement. Non-language factors, such as s o c i a l variables, educational variables, and phys i o l o g i c a l variables, also play important ro l e s . Oversimplification of the process of academic learning or the i n t e r r e l a t i o n s h i p among factors concerned may lead to erroneous findings. Thus, the following l i t e r a t u r e review i s organized into two sections: f i r s t , language factors and academic achievement, with a focus on the relat i o n s h i p between second language 12 p r o f i c i e n c y and un i v e r s i t y academic achievement; and second, non-language factors and academic achievement, with a hi g h l i g h t on the rela t i o n s h i p between gender differences and academic achievement. Language factors and academic achievement 1 I t has been generally recognized that language i s the major medium of learning (Mohan, 1986) and language pro f i c i e n c y i s important to academic success. For those who study i n educational i n s t i t u t i o n s where the language of i n s t r u c t i o n i s t h e i r second language, i n p a r t i c u l a r , t h e i r L2 p r o f i c i e n c y remarkably a f f e c t s , even determines, academic achievement. However, research i n L2 education shows that the strength of the rela t i o n s h i p between L2 pr o f i c i e n c y and academic achievement varies for d i f f e r e n t language s k i l l s and across content areas. L2 s k i l l s and academic achievement. Cummins (1981) described two types of language proficiency: Basic Interpersonal Communication S k i l l (BICS) and Cognitive Academic Language Proficiency (CALP). He pointed out that academic language proficiency, rather than d a i l y conversational competence, i s necessary for academic success. His findings have been supported by many empirical studies ( C o l l i e r , 1987). Other researchers have explored the 1 This review mainly focuses on studies concerning the re l a t i o n s h i p between second language pro f i c i e n c y and academic achievement. This i s because of the l i m i t a t i o n of the space, and the topic being too broad to cover. More important, i t rel a t e s d i r e c t l y to the present research questions. 13 r e l a t i o n s h i p among d i f f e r e n t language s k i l l s , l i s t e n i n g , speaking, reading, and writing, to academic achievement. Johns (1981) conducted a study involving an academic s k i l l s questionnaire with 200 faculty from a l l departments at an American un i v e r s i t y i n order to determine which language s k i l l s among reading, writing, speaking, and l i s t e n i n g were most essential to non-native speakers' success i n t h e i r u n i v e r s i t y classes. Results of the study showed that receptive s k i l l s , both reading and l i s t e n i n g , were ranked f i r s t . Ostler (1980) reported s i m i l a r findings i n a study of a group of ESL college students' assessment of what academic s k i l l s they needed to achieve academic success. The study revealed that academic reading s k i l l was ranked as the most needed among sixteen language s k i l l s . Other highly ranked s k i l l s were taking notes, asking questions i n c l a s s , reading journals, and writing research proposals. In a study of 178 u n i v e r s i t y professors' judgments of errors i n the writing of non-native speaking students, Santos (1988) reported that professors seemed to place more emphasis on language features than on content features, and l e x i c a l errors i n writing were rated as the most serious. This suggested that basic knowledge of vocabulary i n writing plays an important r o l e i n academic achievement. Magan (1986) conducted research on the r e l a t i o n s h i p between speaking proficiency and academic achievement of 40 college French students. His findings revealed a s i g n i f i c a n t 14 p o s i t i v e r e l a t i o n s h i p between speaking a b i l i t y and academic success. In a canonical c o r r e l a t i o n analysis, Ho and Spinks (1985) found that l i s t e n i n g a b i l i t y was not as p r e d i c t i v e of academic performance at the u n i v e r s i t y l e v e l as were speaking, reading, and writing a b i l i t i e s . They argued that i t was l i k e l y that l i s t e n i n g d i f f i c u l t i e s might be compensated through additional reading. The foregoing research findings suggest that d i f f e r e n t language s k i l l s have d i f f e r e n t impacts on academic achievement, although i t appears that no consensus ex i s t s yet i n terms of which language s k i l l plays the most important r o l e . L2 p r o f i c i e n c y across subject matters. Mohan (1986) analyzed the r e l a t i o n s h i p between language and content and considered the nature of language i n education as a medium of learning. Mohan's t h e o r e t i c a l perspective provided insight into the r e l a t i o n s h i p between second language profi c i e n c y and academic achievement i n d i f f e r e n t subject areas across the curriculum. Slark and Bateman (1981) studied non-native English speakers* college academic achievement. Their findings showed that there was a s i g n i f i c a n t p o s i t i v e c o r r e l a t i o n between language scores and course grades i n two courses (Anthropology and Sociology), whereas three other courses (Chemistry, Mathematics, and Music) consistently showed negative c o r r e l a t i o n c o e f f i c i e n t s . The r e s u l t s indicated 15 that courses i n s o c i a l sciences required higher l e v e l s of language pro f i c i e n c y than those i n natural sciences and music. Crandall and others (1987) analyzed the r e l a t i o n s h i p of ESL language development to academic achievement i n mathematics, science, and s o c i a l studies. They argued that although the exact rela t i o n s h i p between ESL language development and content learning of these subjects was not c l e a r l y understood, both a minimal l e v e l of language pr o f i c i e n c y with s p e c i f i c l i n g u i s t i c r e g i s t e r s and a minimal knowledge of the academic area were required for academic success. As f a r as mathematics learning i s concerned, studies with monolingual English speakers have revealed a high p o s i t i v e c o r r e l a t i o n between mathematics achievement and English reading a b i l i t y (Aiken, 1971; Duran, 1979). These r e s u l t s are i n t e r e s t i n g because mathematics uses i t s own symbolic system except for word problem solving. In MacNamara's studies (1966, 1967), b i l i n g u a l children kept pace with monolinguals i n mechanical arithmetic, but f e l l behind i n solving word problems. Several researchers have found that language minority students frequently do not understand the language used to present mathematics t e s t problems (DeAvila & Havassy, 1974; Moreno, 1970). In short, research findings show that there i s a re l a t i o n s h i p between language factors and academic achievement for d i f f e r e n t language s k i l l s and i n d i f f e r e n t 16 subject areas, but do not reveal i d e n t i f i a b l e patterns. As Vinke and Jochems (1993) pointed out, there i s no generally acknowledged theory on the precise nature of the re l a t i o n s h i p between language proficiency and academic achievement. Therefore, making conclusive statements about the r e l a t i o n s h i p i s premature. Non-language factors and academic achievement Comprehensive studies on non-language variables a f f e c t i n g academic achievement. Many researchers have examined the e f f e c t s of non-language factors, i n d i v i d u a l l y or i n combination, on academic achievement. These factors include teacher expectation (Rosenthal and Jacobson, 1968) , achievement motivation (Ames & Ames, 1984), home environment (Soto, 1990), and s o c i a l disadvantage (Ushasree, 1990). In addition, comprehensive studies on varied factors a f f e c t i n g academic achievement have been conducted i n order to i d e n t i f y factors that s i g n i f i c a n t l y and consistently influence academic achievement and to provide empirical evidence about weights and i n t e r r e l a t i o n s h i p s among these factors. Ho and Spinks (1985) examined the e f f e c t s of four variables, verbal i n t e l l i g e n c e , English language s k i l l s , personality, and attitude, on un i v e r s i t y academic performance. Their findings showed that (a) English language s k i l l s had the most predictive value, accounting for about 10% of the variance i n academic performance; (b) Verbal 17 i n t e l l i g e n c e , attitude (excepting study orientation) and personality were not pre d i c t i v e of academic performance. Walberg, S c h i l l e r , and Haertel (1979, 1982) c o l l e c t e d and analyzed the review l i t e r a t u r e of the 1970s on the e f f e c t s of i n s t r u c t i o n and related factors on cognitive, a f f e c t i v e and behavioral domains. Based on a synthesis of 2 3 major research topics addressed by thousands of studies, they found that nine variables appeared to have consistent causal influences on academic leaning: student age or developmental l e v e l , a b i l i t y , motivation, amount of i n s t r u c t i o n , q u a l i t y of i n s t r u c t i o n , the psychological environments of the class, home, peer group outside school, and exposure to the mass media. By performing a l i n e a r structure r e l a t i o n analysis (LISREL), Walberg and three other co-researchers (1984) compared f i v e causal models to examine the r e l a t i o n s h i p between achievement i n science and a combination of eight va r i a b l e s . The eight variables were students' a b i l i t y , home environment, peer group, exposure to mass media, s o c i a l environment, time on task, motivation, and i n s t r u c t i o n a l strategies. Results showed that among the eight factors students' a b i l i t y (r ranged from .72 to .75) and motivation (r ranged from .11 to .12) consistently had the largest influences on science achievement. In another research synthesis (Walberg, Pascarella, Haertel, Junker, & Boularger, 1982), 14 major variables which a f f e c t academic achievement i n science, math, s o c i a l 18 studies, and reading were l i s t e d . The 14 variables were age, achievement, attitude, socioeconomic status, q u a l i t y of i n s t r u c t i o n , quantity of i n s t r u c t i o n , education, home, peer, homework, media-TV, extracurricular, stimulation, and gender. Gender differences and academic achievement. Numerous studies have discussed gender differences and academic achievement. Maccoby and J a c k l i n (1974) i n t h e i r widely c i t e d book summarized and analyzed a large amount of research on gender differences and concluded that: (a) G i r l s have greater verbal a b i l i t y than boys; (b) Boys excel i n v i s u a l - s p a t i a l a b i l i t y ; (c) Boys excel i n mathematical a b i l i t y ; And (d) males are more aggressive. Their r e s u l t s were supported by findings of large scale studies conducted na t i o n a l l y or i n t e r n a t i o n a l l y . The National Assessment of Educational Programs (Husen, 1994, pp. 5425-5426) i n i t s large scale studies over ten years found that the g i r l s performed consistently better on both reading and writing tests than boys, but not on science. The International Association for the Evaluation of Educational Achievement (IEA) studies of mathematics and science (Keeves, 1973) showed that, while the general pattern of r e s u l t s was one of superior performance by male students i n both subjects, there was considerable v a r i a t i o n between countries i n the extent to which boys exceeded g i r l s i n performance. 19 Walker (1976, i n Husen, 1992, p. 5426) reported another IEA study on gender differences i n s i x subjects areas: reading, l i t e r a t u r e , English as a foreign language, French as a foreign language, and c i v i c education. On reading comprehension te s t s , boys showed lower performance than g i r l s i n a majority of countries, but i n general these differences were s l i g h t . On the l i t e r a t u r e t e s t s , i n a l l countries the boys did less well, and they also showed les s i n t e r e s t i n l i t e r a t u r e . Again, i n a study of the teaching of English as a foreign language, the boys scored below the g i r l s on both the reading and l i s t e n i n g tests, but the differences were small. In a study of the teaching of French as a foreign language, s t a t i s t i c a l l y s i g n i f i c a n t gender differences i n the learning of French were recorded i n English-speaking countries, with g i r l s performing better than boys. In c i v i c education achievement te s t s , the boys generally recorded higher scores than g i r l s . Several studies examined issues of gender differences i n language t e s t s . Landsheere (1994) found that boys perform marginally better than g i r l s on multiple-choice tests and problem-solving exercises. G i r l s perform better than boys on essay t e s t s i n written composition and are generally assigned higher grades i n school-based assessments. In another study (Zeidner, 1987), the researcher analyzed the English language aptitude t e s t scores of 824 f u l l time Jewish students i n I s r a e l and found that a small degree of gender differences i n t e s t scores was observed, tending to 20 overpredict the f i r s t year's GPA of males and underpredict that of females. The researcher argued that t h i s might be the r e s u l t of d i f f e r e n t i a l grading practices and unevenness i n the number of males and females i n courses, rather than as a fact of nature. In summary, much research has documented gender differences i n academic achievement i n such subject areas as mathematics, science, s o c i a l studies, language arts, and foreign languages. I t appears cl e a r that (a) there are gender differences i n academic achievement; (b) these differences should not be exaggerated; and (c) many factors contribute to gender differences. In fact, gender should not be considered as a purely b i o l o g i c a l entity, but rather, a composite variable combining physiological, psychological, and s o c i o l o g i c a l components. Gender differences i n academic achievement originate from a vari e t y of sources, such as p a r t i c i p a t i o n differences, a b i l i t i e s differences, b i o l o g i c a l differences, s o c i a l i z a t i o n differences, differences i n attitudes and t h e i r e f f e c t s , and differences i n the expectancy of success (Husen, 1982, pp. 5428-5430). Part I I : Factors Influencing TOEFL's Predictive V a l i d i t y The following part of l i t e r a t u r e review examines f i v e major methodological factors which s u b s t a n t i a l l y influence the estimation of TOEFL scores' p r e d i c t i v e v a l i d i t y . These factors are: (a) Which a n a l y t i c a l models are employed? (b) What subject variables are involved? (c) What predictor 21 variables are used? (d) What c r i t e r i o n variables are selected? And (e) how re s u l t s are computed and interpreted? , 2 A n a l y t i c a l models An a n a l y t i c a l model refers to a hypothesized structure to emulate and analyze the i n t e r r e l a t i o n s between variab l e s . There are many a n a l y t i c a l models used for pr e d i c t i o n or explanation studies (Pedhazur, 1982). I t i s important to choose and employ appropriate a n a l y t i c a l models i n conducting a study of TOEFL*s pre d i c t i v e v a l i d i t y . The model should be chosen properly i n order to f i t the data as well as the research question under study. I t should be used c o r r e c t l y i n order to meet the assumptions underlying the model. The c o r r e l a t i o n model versus the regression model. Most studies estimating TOEFL*s pre d i c t i v e v a l i d i t y have applied the c o r r e l a t i o n model as the sole a n a l y t i c a l model (Chase & S t a l l i n s , 1966; Abdzi, 1967; Kwang & Dizney, 1970; Martin, 1971; AACRAO, 1971; Pack, 1972; H e i l & Aleamoni, 1974; Shay, 1975; Harcey, 1979; Bostic, 1981; Riggs, 1982; Odunze, 1982; Light, Xu & Mossop, 1987; Johnson, 1988; Light & Wan, 1991; Ayers & Ouattlebaum, 1992). These studies usually estimated TOEFL*s pr e d i c t i v e v a l i d i t y by c a l c u l a t i n g c o r r e l a t i o n 2 The present study purposely used the term of a n a l y t i c a l models instead of s t a t i s t i c a l methods or s t a t i s t i c a l models. A s c i e n t i f i c analysis i s not i d e n t i c a l to a s t a t i s t i c a l method. Even for quantitative research i n which the s t a t i s t i c a l method i s i t s ess e n t i a l component, the s t a t i s t i c a l method cannot cover a l l the content that the a n a l y t i c a l model contains, such as model construction and model modification. 22 c o e f f i c i e n t s between TOEFL scores and GPA. The c o r r e l a t i o n model has dominated TOEFL prediction research. Some researchers (Schreder & Pitcher, 1970; Sharon, 1972; Gue & Holdaway, 1973; Stove, 1982; Hassan, 1982; Yule & Hoffman, 1990) have used the c o r r e l a t i o n model as the main a n a l y t i c a l model with the regression model as a supplement. These authors estimated c o r r e l a t i o n c o e f f i c i e n t s (r) and proportion of variance accounted for by regression (R 2), i n some cases with regression c o e f f i c i e n t s (b & P) or the regression equation). A few studies have adopted the regression model as the main a n a l y t i c a l t o o l with the c o r r e l a t i o n model as i t s integrated component (Wilcox, 1975; Andalib, 1976; Ayers & Peters, 1977; Sokari, 1981). In these studies, c o r r e l a t i o n c o e f f i c i e n t s are calculated as one of the basic d e s c r i p t i v e estimates. The main procedure i s to perform a regression analysis so that the regression equation, squared multiple c o r r e l a t i o n , and/or regression c o e f f i c i e n t s are obtained. Which a n a l y t i c a l model should be chosen for p r e d i c t i o n studies? This issue has been discussed extensively i n psychometric research since the 1950s (Kendall, 1951; Fish, 1958; Binder, 1959; Ezekiel St Fox, 1959; Fox, 1968; Warren, 1971; Thorndike, 1978;). Based on these studies, Pedhazur (1991) concluded that when the focus of the research i s on the explanation, or the prediction, of dependent variables, the regression model i s appropriate (p. 409). In TOEFL predic t i o n studies, the research purpose i s to see how well 23 TOEFL scores predict the c r i t e r i o n variable GPA, but not to describe the association between two a r b i t r a r i l y selected var i a b l e s . Thus, the regression model rather than the co r r e l a t i o n model i s the proper solution. The simple regression model versus the multiple regression model. TOEFL prediction studies using the regression model as t h e i r main or supplemental a n a l y t i c a l t o o l can be c l a s s i f i e d into three groups i n terms of the number and the vari e t y of predictor variables involved. In the f i r s t group of studies, only one predictor variable i s used i n the regression model (for instance, Hassan, 1982). In the second group, multiple predictor variables of English language proficiency are used as predictors i n the regression models (for example, GRE-V and MTELP scores, Abdzi, 1967; TOEFL's f i v e - s e c t i o n scores, Sharon, 1972; P r i o r - and post-admission TOEFL scores, and interview scores, Gue and Holdaway, 1973; TOEFL's o v e r a l l and sectional scores, Hu, 1991). Multiple predictor variables with multiple features are used i n the t h i r d group of studies (for example, TOEFL and LSAT. Schrader & Pitcher, 1970; TOEFL, ACT, SAT, high school GPA, and age. Andalib, 1976; TOEFL, ESL course grades, native language, major areas of study. Stove, 1982; TOEFL, GRE-V and GRE-Q. Yule & Hoffman, 1990). The number and the vari e t y of predictors i n the regression model are dependent upon the complexity of the 24 research problem under study. When a one-cause-one-effect re l a t i o n s h i p e x i s t s , a simple regression model should be used to predict a phenomenon completely determined by a single factor. For more complicated phenomena, more predictors are needed. For phenomena influenced by d i f f e r e n t types of factors, a multiple regression model with d i f f e r e n t types of predictors i s required. In s o c i a l and educational research, the multiple regression model i s necessary i n most cases to make the prediction study defensible. There are manifold factors a f f e c t i n g academic achievement, therefore, a multiple regression model with multiple predictors i s appropriate to predict GPA. Using only one predictor, or the language-based predictors, makes i t d i f f i c u l t to gain an accurate prediction of GPA. Many TOEFL predi c t i o n studies, as reviewed above, used language-based variables; as a r e s u l t , they frequently obtained r e l a t i v e l y smaller R2, even though more s i m i l a r language-based predictors were added into the regression equation. Thus i n TOEFL prediction studies, we should not only choose multiple predictors, but also take into account the degree of d i v e r s i t y of the predictors. More complicated models, such as path analysis model, Linear Structural Relations model, Hi e r a r c h i c a l Linear Model, canonical analysis model, and discriminant analysis model, can also be used to analyze the complex r e l a t i o n s h i p of factors a f f e c t i n g academic achievement. 25 The single-step regression c a l c u l a t i o n versus the comprehensive regression analysis package. Regression analysis should not be seen as the sole c a l c u l a t i o n of R2, or of regression c o e f f i c i e n t s . An a n a l y t i c a l process of the multiple regression t y p i c a l l y involves integrated components and relevant techniques, including the checking of assumptions, detecting o u t l i e r s (by using residual analysis and influence analysis), regression estimation, hypothesis t e s t i n g , as well as power analysis (Cohen, 1988; Husen, 1994, p. 3866; Pedhazur, 1991). The procedures and techniques mentioned above examine the f i t of regression models to data, the existence of o u t l i e r s , the weighting of the variables, and the degree to which r e s u l t s can be generalized so that the q u a l i t y of a multiple regression analysis can be optimized. Multiple regression studies of TOEFL's p r e d i c t i v e v a l i d i t y usually report R2, the regression equation (Schreder & Pitcher, 1970; Sharon, 1972; Sokari, 1981; Stove, 1982; Yule & Hoffman, 1990; Hu, 1991), r e s u l t s of the stepwise regression (Gue & Holdaway, 1973; Andalib, 1976; Ayers & Peters, 1977), and/or standard error of estimation (Sgg,.) and shrinkage (Hassan, 1982) . However, i t appears that few researchers, i f any, perform the comprehensive regression analysis mentioned above. 26 Subject variables 3 Many studies have reported that various subject variables a f f e c t the estimation of TOEFL's pr e d i c t i v e v a l i d i t y (Hale, S t a n f i e l d , & Duran, 1984). These subject variables can be grouped into four categories. (a) Personal information, such as gender, age, parents' educational l e v e l . (b) S o c i a l factors, including native language, home country or region, c i t i z e n s h i p , ethnic group, s o c i a l adjustment, and occupation i n home country. (c) Academic background, for instance, areas of study, type of degree sought, educational l e v e l , and previous grades. (d) Test-related information, such as TOEFL repeaters or non-repeaters, TOEFL scores i n the Friday program or the Saturday program, and the l i k e . The following discussion, however, focuses on two issues related to subject variables. These issues cause serious problems but were often ignored i n the estimation of TOEFL's pr e d i c t i v e v a l i d i t y . Sample s i z e . Sample s i z e i s associated with the homogeneousness of subjects under study. Differences i n 3 In some studies, some subject variables were used as predictor variables, functioning as a moderator or mediator along with TOEFL scores to predict GPA. This could be also viewed as an evidence of influences of subject variables on TOEFL/GPA r e l a t i o n . I t i s t h i s kind of unintended and easily-neglected e f f e c t of subject variables that make non-experimental research, including the TOEFL/GPA study, more complicated. 27 sample s i z e have d i f f e r e n t e f f e c t s on the pr e d i c t i v e v a l i d i t y of TOEFL scores. Although almost a l l studies on the pre d i c t i v e v a l i d i t y of TOEFL scores reported t h e i r sample sizes, the range i n sample si z e among varied from 15 to 900. Some TOEFL pr e d i c t i v e studies lacked s u f f i c i e n t sample s i z e (Bostic, 1981; Hassan, 1982; Riggs, 1982). Most studies used the cumulative sample s i z e obtained across years (e.g., Schreder & Pitcher, 1970. n=63, from 1964 to 1969; Sharon, 1972. n=973, 1964-69; Pack, 1972. n=402, 1960-72; Gue & Holdaway, 1973. n=123, 1967-70). This kind of cumulative sample s i z e might r e s u l t i n problems regarding the pre d i c t i v e v a l i d i t y of TOEFL scores. I t might confound various subject variables, ignore the differences i n the two forms of TOEFL ( i . e . , three-section and f i v e - s e c t i o n ) , or lose unique information i n sub-samples for each year. Mean TOEFL scores. Mean TOEFL scores indicate the average l e v e l of English language proficiency of the subjects under study. They subs t a n t i a l l y influence the extent to which TOEFL scores predict academic achievement. Wilcox (1975) found that one group of subjects with better ESL proficiency showed no relat i o n s h i p between TOEFL scores and GPA, whereas another group with lower English l e v e l s showed a s i g n i f i c a n t r e l a t i o n s h i p . He explained that English a b i l i t y and academic success may be related at low le v e l s of prof i c i e n c y but unrelated at l e v e l s above c e r t a i n threshold values. Wilcox's findings suggest that the existence of ce r t a i n thresholds of TOEFL scores probably 28 r e s u l t s i n a nonlinear rel a t i o n s h i p between English p r o f i c i e n c y l e v e l and academic achievement. S i m i l a r l y , Johnson (1988) found that when English p r o f i c i e n c y i s r e l a t i v e l y low, TOEFL scores can predict academic performance. With higher language proficiency, other variables such as p r i o r exposure to subject matter, motivation, study s k i l l s , c u l t u r a l adaptability, and even f i n a n c i a l security, may became more important. The TOEFL Test Manual (ETS, 1992) states that i f the standard for English language proficiency i s set at such a high l e v e l that only applicants with good English s k i l l s are admitted, there may be l i t t l e r e l a t i o n s h i p between TOEFL scores and any of the c r i t e r i o n measures. Because there w i l l be no large variance i n English proficiency among the group members, var i a t i o n s i n success on the c r i t e r i o n variables w i l l be due to other non-English causes. On the other hand, i f the standard i s set at too low a l e v e l , a large number of applicants selected with TOEFL scores may be unsuccessful i n the academic program. There w i l l be a r e l a t i v e l y high c o r r e l a t i o n between t h e i r TOEFL scores and i t s c r i t e r i o n measures. Thus, with a standard that i s neither too high nor too low, the c o r r e l a t i o n between TOEFL scores and subsequent success w i l l be only moderate. Mean TOEFL scores also involve the issue of r e s t r i c t i o n of range. R e s t r i c t i o n of range means that, as a r e s u l t of se l e c t i o n , the range of subjects i n a study i s i n e v i t a b l y r e s t r i c t e d and only those who are selected with c e r t a i n 29 standards rather than those who are randomly drawn from the true population are available for investigation. R e s t r i c t i o n of range leads to a sampling bias. In TOEFL's predi c t i o n studies, the sample under study i s r e s t r i c t e d by the minimum TOEFL requirement for admission se l e c t i o n so that an unrandomized sampling bias occurs. Based on a c r i t i c a l analysis of s i x studies of TOEFL 1s pr e d i c t i v e v a l i d i t y , Yan (1994) found that TOEFL means i n these studies ranged from 491.00 to 561.00, which were above the 50th p e r c e n t i l e rank i n the population of a l l TOEFL takers. Standard deviations i n these studies ranged from 38.80 to 66.00, which were lower than the standard deviation of the population. In most cases, sampling i n the TOEFL/GPA studies was based primarily upon a v a i l a b i l i t y of subjects instead of randomization. This e a s i l y produces a biased sample with higher homogeneity than i t s population. A homogeneous sample w i l l underestimate the pr e d i c t i v e v a l i d i t y of TOEFL scores (Pedhazur, 1982; Cohen, 1983) . Predictor variables There are d i f f e r e n t kinds of TOEFL scores used i n TOEFL predic t i o n studies. This v a r i a t i o n i n s e l e c t i o n of predictor variables influences the estimation of TOEFL's p r e d i c t i v e v a l i d i t y . Some examples are given as follows. F i r s t l y , some studies only used TOEFL t o t a l scores (Johnson, 1988; Light & Wan, 1991), some used the TOEFL sectional scores, others used t o t a l and sectional scores separately (Kwang & Dizney, 1970; Light, Xu & Mossop, 1987; 30 Hu, 1991) and a few used a combination of TOEFL t o t a l scores and sectional scores as one predictor. The pr e d i c t i v e v a l i d i t y varies on the basis of single scores or composite scores of TOEFL. Secondly, from 1963 to 1976 TOEFL consisted of f i v e subtests. The f i v e - s e c t i o n TOEFL had 200 t o t a l items and required two hours and 20 minutes of administration time. Some predic t i o n studies examined the f i v e - s e c t i o n TOEFL (Harcey, 1979; Bositc, 1981; Stover, 1982). The current three-section TOEFL consists of 150 items and requires one hour and 45 minutes of actual t e s t i n g time. Some studies explored the pr e d i c t i v e v a l i d i t y of the three-section TOEFL(Martin, 1971; Sharon, 1972; Shay, 1975; Riggs, 1982). Because of differences i n section construction items included and time allocated for the two forms of TOEFL, spec i a l caution has to be taken when one compares the pre d i c t i v e v a l i d i t y of TOEFL scores obtained over time from d i f f e r e n t t e s t administrations. However, t h i s was unfortunately ignored i n some TOEFL prediction studies (e.g., Odunze, 1982). Thirdly, English language a b i l i t y can be affected over a short period of time by additional t r a i n i n g or lack of pre-test practice (ETS, 1994a). Thus ETS set a rule that a TOEFL score report w i l l only be v a l i d for two years. However, even within two years the range of time to take TOEFL i s s t i l l important. The research l i t e r a t u r e documented the TOEFL prediction studies with a vari e t y of timings, such 31 as summer TOEFL scores a f t e r a r r i v a l i n the USA (Gue & Holdaway, 1973), pre-instruction TOEFL scores (Schrader & Pitcher, 1970), and pre-study TOEFL scores (Light & Wan, 1991). Most studies used pre-admission TOEFL scores (e.g., H e i l , & Aleamoni, 1974; Ayers & Peters, 1977), except a study using after-admission TOEFL scores (Ho & Spinks, 1985). I t i s important to note that the time lapse between the c o l l e c t i o n of predictor scores and the c o l l e c t i o n of the c r i t e r i o n scores w i l l impact the p r e d i c t i v e v a l i d i t y of TOEFL scores. Furthermore, pre- and post-admission scores a f f e c t s i g n i f i c a n t l y the degree of homogeneity of the sample. The former w i l l be much more heterogeneous, and the l a t t e r w i l l r e s u l t i n a f a i r l y s e l e c t i v e sample. Besides various forms of TOEFL scores, many TOEFL pre d i c t i o n studies used other language t e s t scores, obtained from such standardized or l o c a l tests as Lado Test B and C (Chase & S t a l l i n g s , 1966), the Pennstat (Chase & S t a l l i n g s , 1966), Test of the American Language I n s t i t u t e at Georgetown University (AACRAO, 1971), Michigan Test of English language Proficiency (MTELP) (Pack, 1972; Abadzi, 1976), the English Placement Examination (Heil & Alaemini, 1974), the GRE general t e s t ' s verb subtest (GRE-V) (Ayers & Peters, 1977), and Wechsler Adult Intelligence Scale's form R Vocabulary subtest (WAIS-R-V) (Hassen, 1982), as predictors. Other studies used writing scores and interview scores (Gue & Holdaway, 1973), ESL course average grade (Stover, 1982; South, 1992) as the predictor of academic success. 32 Quite a few studies have used non-language predictors, such as GRE-Q (Ayers & Peters, 1977; Yule & Hoffman, 1990; Ayers & Quattlebaum, 1992); high school GPA, age, years out of school, resident status, c u l t u r a l background (Andalib, 1976) ; WAIR-R-V, SAT (Wilox, 1975); native language, major area of study (Stove, 1982); ratings of qual i t y of academic performance (AACRAO, 1971); and LAST (Schrader & Pitcher, 1970) . C r i t e r i o n variables Selection of c r i t e r i o n variables i s a c r u c i a l but d i f f i c u l t task i n designing a prediction study. Although i t has always been c r i t i c i z e d (e.g., Graham, 1987), GPA i s s t i l l the most frequently used c r i t e r i o n v a r i a b l e . This i s lar g e l y because (a) i t i s the most t y p i c a l ( i f not perfect) indicator of academic success (Wimberley, McCloud, & Fl i n n , 1992) ; (b) i t i s the most re a d i l y accessible c r i t e r i a for academic achievement (Light, Xu & Mossop, 1987) ; and (c) i t i s r e l a t i v e l y well-defined and widely understood (Young, 1993) . However, d i f f e r e n t versions of GPA have been seen i n the research l i t e r a t u r e . Types of GPAs i n terms of a period of time include: • First-term GPA (Pack, 1972; Stove, 1982; Wilcox, 1975; Light & Wan, 1991; Light, Xu & Mossop, 1987; Kwang & Dizney, 1970); • F i r s t - and second-term GPA (Abdzi, 1967; Harvey, 1979; H e l l & Aleamoni, 1974; Martin, 1971; Odunze, 1982) ; 33 • F i r s t - y e a r GPA (Chase and S t a l l i n s , 1966; Riggs, 1982; AACRAC, 1971; Gue & Holdaway, 1973; Schrader & Pitch, 1970); • Fi r s t - y e a r , one-and-half-year, and two-year GPA (Yule and Hoffman, 1990); • Graduation GPA (Ayers & Peters, 197 7); Ayers & Quattlebaum, 1992); • GPA obtained from unreported or unable to i d e n t i f i e d terms (Hassen, 1982; Andalib, 1976; Sharon, 1972; Hu, 1991; Johnson, 1988). Types of GPAs i n terms of d i f f e r e n t point systems include: • Four-point GPA (AACRAO, 1971; Martin, 1971; Light & Wan, 1990; Ayers & Peters, 1977); • Five-point GPA (Andalib, 1976); • Nine-point GPA (Gue & Holdaway, 1973) ; • Percentage GPA (UBC, 1993); • Letter grade GPA (UBC, 1993). Other c r i t e r i a used include: • Numbers of c r e d i t hours (Shay, 1975; Abdzi, 1967; Johnson, 1988); • Average of 12 cr e d i t s successful completed (Light & Wan, 1991); • Verbal- and nonverbal-course GPA (Bostic, 1981); • Academic index, advisor's rating (AACRAO, 1971); • Eventual TA recombination (Yule & Hoffman, 1990); • Average accumulated c r e d i t per semester 34 (Christopher, 1993). Besides the above, as an index to academic achievement GPA also varies i n sections, courses, in s t r u c t o r s , majors, years, programs, and i n s t i t u t i o n s , as well as countries. Various versions of GPA w i l l have a s i g n i f i c a n t impact on the estimation of TOEFL's pred i c t i v e v a l i d i t y . Therefore, each TOEFL prediction study should specify and j u s t i f y what kind of GPA i s used. Result i n t e r p r e t a t i o n Various ways of interpreting r e s u l t s are another source of inconsistency i n research findings of TOEFL's p r e d i c t i v e v a l i d i t y . There were two consistent problems regarding r e s u l t i n t e r p r e t a t i o n i n the TOEFL prediction studies. F i r s t , there have been neither consistent standards nor conventional terminology used to evaluate whether a measure of the TOEFL's pred i c t i v e v a l i d i t y i n a study i s high or low. Some studies claimed that TOEFL scores were a useful, r e l i a b l e , s i g n i f i c a n t , meaningful, adequate, strong, and sa t i s f a c t o r y predictor of GPA respectively (Chase & S t a l l i n g s , 1966; ; Hwang & Dizney, 1970; Shay, 1975; Ayers & Peter, 1977; Sokari, 1981; Odunze, 1982; Ayers & Quattlebaum, 1992). On the contrary, other studies declared that TOEFL scores were of limited, doubtful, and questionable value i n predicting GPA respectively (Harvey, 1979; Bostic, 1981). Obviously here, what was meant by useful, doubtful, or other modifiers was rather vague and subjective. As a matter of fact, what i s deemed useful, 35 e f f e c t i v e , or strong by one researcher may be deemed useless, i n e f f e c t i v e , or weak by another researcher or by the same one at another context. For example, based on the research finding (r=.14, p_<.05), o n e study (Light, Xu & Mossop, 1987) asserted that the c o r r e l a t i o n was too low to have any p r a c t i c a l s i g n i f i c a n c e and therefore TOEFL was not an e f f e c t i v e predictor of academic success. However, the researchers explained neither why an r of .14 was too low nor what standards were used to r e j e c t the TOEFL's pr e d i c t i v e v a l i d i t y i n the study. Thus, t h e i r conclusion i s a r b i t r a r y . To determine the strength, importance, and meaningfulness of findings, an estimate of e f f e c t s i z e instead of t e s t i n g of s t a t i s t i c a l s i g n i f i c a n c e i s generally recommended (Cohen, 1988; Pedhazur & Schmelkin, 1991). Cohen (1988) proposed conventions for small, medium, and large e f f e c t sizes for c o r r e l a t i o n c o e f f i c i e n t s , regression c o e f f i c i e n t s , and differences between means. The r e s u l t s of TOEFL predi c t i o n studies should be interpreted according to well-established standards l i k e Cohen's conventional d e f i n i t i o n s on R2 (1988) to avoid subjectiveness and a r b i t r a r i n e s s . The second problem i s about what estimates should be used to judge the r e l a t i v e importance among predictor varia b l e s . Some studies concluded that TOEFL scores were a better, higher, best, strongest, or lower predictor by comparing the scores with other predictors i n terms of 36 c o r r e l a t i o n c o e f f i c i e n t s or regression c o e f f i c i e n t s obtained (Wilcox, 1975; Chase & S t a l l i n g s ; Ho & Spinks, 1985; AACRAO, 1971). These studies judged the predictors' r e l a t i v e importance on the basis of sign i f i c a n c e t e s t r e s u l t s on improper estimates such as r or R2. Conventionally, change i n R2 or squared semipartial c o r r e l a t i o n i s recommended to estimate the r e l a t i v e importance among predictor variables (Pedhazur, 1982; Tabachnick & F i d e l l , 1989). Summary The l i t e r a t u r e review i n t h i s chapter helps to b u i l d both a conceptual and a methodological bases for estimating the p r e d i c t i v e v a l i d i t y of TOEFL scores on G P A . Theoretically, i t was revealed that numerous factors i n s o c i a l , educational, l i n g u i s t i c , psychological, and physi o l o g i c a l domains influence academic achievement. There i s no one single factor which can f u l l y determine academic success or f a i l u r e . The unique contribution of any single factor to academic achievement should be examined i n comparison with other relevant factors. Therefore, to investigate TOEFL's pred i c t i v e v a l i d i t y , one should consider i t within a comprehensive context which includes both language factors and non-language factors. For language factors, one should further consider d i f f e r e n t language prof i c i e n c y (e.g., CALP, BICS), language s k i l l s (e.g., l i s t e n i n g , writing), or l i n g u i s t i c r e g i s t e r s i n subject areas (e.g., mathematics, science). 37 From a methodological perspective, there are f i v e major aspects which have s i g n i f i c a n t e f f e c t s on the estimation of TOEFL's pr e d i c t i v e v a l i d i t y : a n a l y t i c a l models, subject variables, predictors, c r i t e r i a , and r e s u l t i n t e r p r e t a t i o n . They deserve special attention i n the research design i n order to ensure s a t i s f a c t o r y research v a l i d i t y . 38 Chapter Three Method This chapter outlines the method of the present study, including the program setting, p a r t i c i p a n t s , the predictor variables, the c r i t e r i o n variable, the a n a l y t i c a l model, operational d e f i n i t i o n s of pre d i c t i v e v a l i d i t y , and research hypotheses. The program se t t i n g The UBC/Ritsumeikan Academic Exchange Program began i n 1991 based upon an agreement for the establishment of an international academic exchange between the University of B r i t i s h Columbia and Ritsumeikan University. I t i s the largest exchange program of t h i s type i n North America. The program operates on an eight-month basis. Each year about 100 undergraduate students from Ritsumeikan University study at UBC from September to A p r i l as a part of t h e i r four-year u n i v e r s i t y education. After that, they go back to continue t h e i r studies i n Japan. Ritsumeikan University was o r i g i n a l l y founded i n 1869 by Japanese Prince S a i o n j i Kinmochi and i s one of the private u n i v e r s i t i e s i n Japan. I t presently comprises seven Colleges and seven Graduate Schools i n Law, Economics, Business Administration, Social Sciences, International Relations, Letters, Science and Engineering. The t o t a l enrollment of students i n the 1992-93 academic year was about 25,000, of which undergraduate students were over 23,000 (Ritsumeikan University, 1993). The r a t i o of success 3 9 i n a p p l i c a t i o n for admission into the University i s about 1:20. Applicants to the UBC/Ritsumeikan Academic Exchange Program are required to submit t h e i r academic records, TOEFL o f f i c i a l score reports, as well as writing samples i n English for evaluation. To help applicants prepare for writing TOEFL, Ritsumeikan University provides TOEFL preparation workshops. Based upon both academic aptitude and English proficiency, Ritsumeikan University selects about 100 q u a l i f i e r s into the program from the pool of applicants i n second- and third-year courses. The program provides a content-oriented curriculum with an emphasis on c r o s s - c u l t u r a l communication. The in s t r u c t o r s are from the Department of Language Education at UBC. English i s used as the medium of i n s t r u c t i o n . At the beginning of the program, the students are grouped into f i v e classes. Each class included about 20 students with one teaching assistant. They are required to complete s i x three-c r e d i t courses i n one academic year, three three-credit courses for each term. A l l of them take courses offered by the Department of Language Education i n the f i r s t term. The courses offered i n 1993-94 included: I n t e r c u l t u r a l Communication i n Second Language Education, Communication S k i l l s i n Educational Settings, Academic Discourse i n Second Language Education, and Second Language Education Practicum. In the second term, those whose TOEFL t o t a l scores meet the UBC minimum requirement of 550 may attend regular UBC 40 classes i n the Faculty of Arts and other f a c u l t i e s for which they have pre-requisites. The c r e d i t s the students earn at 4 UBC are transferable to t h e i r home university. A l l the program students l i v e i n the UBC/Ritsumeikan House on the campus of UBC. Pairs share an apartment with two Canadian roommates. In addition to d a i l y l i f e experience, f i e l d studies, a buddy program, and other programs are arranged to enhance the students 1 cross-c u l t u r a l understanding of Canadian society. The students are also involved i n s o c i a l a c t i v i t i e s on and of f campus, such as a seminar series by the UBC P a c i f i c Rim Club and volunteer work at preschools. Participants The target population of the study was the UBC/Ritsumeikan Academic Exchange Program students. The sample was a t o t a l of 97 students who enrolled i n the 1993-1994 program. Among them, 46 students were male and 52 female. The range i n age was from 20 to 2 3 years old, except one senior student aged over 60. They were a l l second year undergraduate students at Ritsumeikan University. Ninety f i v e students majored i n the humanities and s o c i a l sciences such as Law, Business Administration, International Relation, Economics, and English Literature, and only two i n Engineering. Japanese i s t h e i r f i r s t language and most of According to William McMichael (W. McMichael. personal communication, A p r i l , 1995), current academic coordinator of the program, the 1994-95 program has adjusted i t s curriculum structure and the 1995-96 program w i l l have larger changes. 41 them had not yet experienced studying and/or staying i n North America before the program. Most were at the intermediate l e v e l i n English proficiency. Their TOEFL t o t a l score mean was 515.96 with a standard deviation of 26.03. I t was evident that the sample was quite homogeneous i n terms of age, native language, country of o r i g i n , c u l t u r a l background, major f i e l d s , and English p r o f i c i e n c y . The predictor variables Seven predictor variables were used i n the study based upon s u i t a b i l i t y to the research question and a v a i l a b i l i t y i n the UBC/Ritsumeikan Academic Exchange program. These predictor variables were: TOEFL t o t a l scores, TOEFL section I scores, TOEFL section II scores, TOEFL section III scores, o r a l interview scores, writing sample scores, and gender. TOEFL scores, including t o t a l scores and three subscores, served as predictor variables i n the study. These scores were obtained from the d i f f e r e n t TOEFL administrations at Ritsumeikan University through the I n s t i t u t i o n Testing Program (ITP) from January through May of 1993 when the students applied for admissions into the program. Two things should be noted. F i r s t , the highest TOEFL score for each student was used i n the study. Most students i n the program wrote the TOEFL repeatedly i n d i f f e r e n t administrations. There are three frequently seen alt e r n a t i v e s for use of TOEFL scores for admissions: the highest TOEFL score, the l a t e s t TOEFL score, or the average 42 TOEFL score. Both Ritsumeikan University and UBC consistently used applicants' highest TOEFL scores to evaluate English language proficiency for program admission. Second, TOEFL scores were obtained through the I n s t i t u t i o n Testing Program rather than the regular Friday and Saturday Testing Programs.5 ETS states that TOEFL scores under the ITP are not acceptable for o f f i c i a l admission purposes. However, the study had to use the ITP TOEFL scores because they were the only al t e r n a t i v e Ritsumeikan University administered for the program applicants. UBC and Ritsumeikan University agreed to use ITP TOEFL scores f o r program admissions purposes. According to the TOEFL Test Manual (ETS, 1992), the ITP Manual (ETS, 1994f), and discussions (Kantor, R. N., personal e-mail communications, 1994 & 1995) between the author of the current thesis and Dr. Kantor, Director of TOEFL Program Of f i c e , the ITP TOEFL scores are s t i l l considered s u b s t a n t i a l l y v a l i d and are comparable to scores earned under the regular programs. Two other English proficiency measurements were availa b l e i n the program and used as predictor variables. They were the September o r a l speaking scores and the September writing sample scores. The purpose of those two 5 There are two d i f f e r e n t kinds of TOEFL t e s t i n g programs according to the TOEFL Test and Scores Manual (ETS, 1992). The o f f i c i a l TOEFL te s t i n g programs, including Friday and Saturday t e s t i n g programs, are administrated i n t e r n a t i o n a l l y i n the TOEFL t e s t i n g centers. The I n s t i t u t i o n Testing Program, whose items were previously used i n the o f f i c i a l t e s t i n g programs, i s administrated at l o c a l i n s t i t u t e s around the world. 43 measurements was to evaluate English speaking s k i l l and writing s k i l l respectively before i n s t r u c t i o n started, while TOEFL does not provide d i r e c t information on writing and speaking s k i l l s . The v a l i d i t y and r e l i a b i l i t y of these two measurements have not been reported. The September o r a l proficiency interview took 20 minutes. Each student's o r a l performance was rated by the interviewers on a 0-5 11-point scale (including extra f i v e plus marks. See Berwick & McMichael, 1993, p. 3 & Appendix C). The interviewers received pre-interview t r a i n i n g i n o r a l p r o f i c i e n c y interview and r a t i n g procedure. The September writing sample scores were given by trained raters based on a 1-6 6-point scale (See Berwick & McMichael, 1993, p. 2 & Appendix B). Each student was required to write an essay on designated topics within 90 minutes. The assessment of the writing samples followed that of the TOEFL Test of Written English (TWE).6 In addition to the foregoing language-based predictors, Gender was used as a non-language predictor i n the study. The c r i t e r i o n variable The c r i t e r i o n variable i n the study was the f i r s t - t e r m GPA. I t was calculated on the basis of a percentage grading and c r e d i t weighting system 7 which UBC adopted i n 1991. At 6 Both scores of the September o r a l p r o f i c i e n c y interview and scores of the September writing sample w i l l be labeled as speaking scores and writing scores respectively i n the following text. The c o n v e r t i b i l i t y among d i f f e r e n t grading and c r e d i t weighting systems i n North America i s beyond the scope of 44 UBC, course weight i s expressed i n c r e d i t s . In general one c r e d i t represents one hour of i n s t r u c t i o n or two to three hours of laboratory work per week throughout one term. Courses are normally graded on a percentage basis with a corresponding l e t t e r grade assigned (UBC, 1993). The f i r s t term GPA included the average percentage grades i n three courses. They were: EDUC395A, Second Language Education Practicum; EDUC490A, Regional Studies In Second Language Education; and ENED379, Crosscultural Studies i n Second Language Education. Each was a three c r e d i t course. Instructors i n the Department of Language Education taught the courses and assessed academic achievement. The course grades were given based upon a set of s p e c i f i c c r i t e r i a outlined i n the various course s y l l a b i at the beginning of the term (Berwick & McMichael, 1992; Berwick & McMichael, 1993). Table 3.1 shows that the set of c r i t e r i a mainly placed weights on written tasks to evaluate students' academic achievement. Analytic model This study used multiple regression analysis as the a n a l y t i c model. The study focused on the estimate of the p r e d i c t i v e v a l i d i t y of TOEFL scores on GPA. Hence the regression model i s appropriate and intimately related to the primary goal of the study. Furthermore, the complexity of the research problem under study required a the present study. For detailed discussion on t h i s issue see Cohen & Cohen (1983) and Pedhazur (1982). 45 Table 3.1 Grade c r i t e r i a on d i f f e r e n t aspects i n the three courses EDUC395A (%) EDUC490A (%) ENED379 (%) F i e l d work journal 20 20 Oral presentation 20 15 10 Term paper 20 20 Lab work Fi n a l Examinations 25 15 Assignments 30 30 35 Progress evaluations 30 Bibliography 10 Lite r a t u r e review 10 P a r t i c i p a t i o n 10 TOTAL 100 100 100 powerful a n a l y t i c t o o l . As a highly general and very f l e x i b l e data-analytic system (Cohen & Cohen, 1983), the regression model, p a r t i c u l a r l y the multiple regression model, can be applied to investigate various factors related to the pr e d i c t i v e power of the TOEFL score. The data of the study was processed with SPSS for Windows (Release 6.0). Operational d e f i n i t i o n s of predictive v a l i d i t y . The present study u t i l i z e d change i n squared R (AR2) as the estimator to assess predictive v a l i d i t y . According to Cohen's conventional d e f i n i t i o n s (Cohen, 1988, pp. 412-414), .02, .13, and .26 are respectively defined as small, medium, 46 and large e f f e c t s i z e of R2. Based on Cohen's d e f i n i t i o n s , the present study defined four operational l e v e l s of pr e d i c t i v e v a l i d i t y (see Table 3.2). Table 3.2 Four l e v e l s of Predictive V a l i d i t y Level R2 Predictive v a l i d i t y 1. .000 - .019 Negligible 2. .020 - .129 Small 3 . .130 - .259 Medium 4. .260 - 1.00 Large Research Hypotheses The present study advanced the following research hypotheses for t e s t i n g : 1. TOEFL t o t a l scores have p r e d i c t i v e v a l i d i t y on f i r s t term's GPA for the UBC/Ritsumeikan Exchange Program students. 2. TOEFL sectional scores have pre d i c t i v e v a l i d i t y on f i r s t term's GPA for the UBC/Ritsumeikan Exchange Program students. 3. Writing scores have predictive v a l i d i t y on f i r s t term's GPA for the UBC/Ritsumeikan Exchange Program students. 47 4. Speaking scores have pre d i c t i v e v a l i d i t y on f i r s t term's GPA for the UBC/Ritsumeikan Exchange Program students. 5. Gender has pre d i c t i v e v a l i d i t y on f i r s t term's GPA for the UBC/Ritsumeikan Exchange Program students. Summary This chapter delineated the research design of the present study. Participants were 9 7 Japanese exchange students. The study employed a multiple l i n e a r regression model to analyze the relationships of TOEFL scores and other predictor variables to f i r s t term's GPA. Four operational l e v e l s of pr e d i c t i v e v a l i d i t y were defined for r e s u l t i n t e r p r e t a t i o n . The study tested f i v e research hypotheses. 48 Chapter Four Results This chapter summarizes treatment of the missing data, steps taken to check for v i o l a t i o n of assumptions, and an analysis of the descriptive data. The chapter presents the main findings of a multiple regression analysis. Treatment of the missing data An examination of the data f i l e used i n the present study showed that there were three cases with missing values and one case which had a suspicious value on the TOEFL section II score. Since only three missing-value cases were found from a sample of 97, there were very few chances that a systematic pattern existed among the missing-value cases. In other words, there were reasons to believe that the missing values for the variables occurred randomly. Therefore, the l i s t w i s e missing-value treatment was employed i n the study. This treatment keeps a l l variables but eliminates the missing-value cases. I t i s also the default for the missing-value treatment i n the SPSS for Window program. Three cases, two with missing values i n speaking scores and one i n GPA, were eliminated from the data f i l e . For case 40, the TOEFL t o t a l score was 57 0, with three sectional scores 50, 68, and 53 respectively. The TOEFL section II score was suspicious. Note that 68 i s the maximum score i n Section I I . I t was almost impossible to reach i t while the other sectional scores were around 50. I t was also 49 found that i n a TOEFL t e s t administrated about two months e a r l i e r than the currently discussed t e s t the same person scored only 513. I t was u n l i k e l y that t h i s student would gain about 60 points within two months. Thus the section II score of 68 might be a data entry error. Since a l l the o r i g i n a l reports of TOEFL scores were at Ritsumeikan University i n Japan, i t was impossible to check t h i s p a r t i c u l a r TOEFL score. Case 42 was, therefore, excluded from the data. Descriptive s t a t i s t i c a l analysis Means and standard deviations of a l l the variables are shown i n Table 4.1 below. The mean TOEFL t o t a l score i n the study was 515.96 and sectional scores were 49.74, 53.58, and 51.48 respectively. Standard deviation (SD) of the TOEFL t o t a l score was 2 6.03. Table 4.1 Means and standard deviations of a l l the variables GPA GENDER SPEAK WRITE TOEFL S E C 1 SEC2 SEC3 N 93 93 93 93 93 93 93 93 M 71.97 1.55 1.46 2.71 515.96 49.74 53.58 51.48 SD 7.54 .50 .83 .83 26.03 4 . 1 6 3.34 2.91 ETS reported that based on the t o t a l of 1,3 38,682 examinees tested between July 1991 to June 1993, the mean 50 TOEFL t o t a l score was 519.00 and SD was 68.00. The mean TOEFL t o t a l scores and the mean sectional scores were 490.00, 49.00, 50.00, and 48.00 respectively for the group of the t e s t takers whose native language i s Japanese,(ETS, 1993) . Means of three groups (the t o t a l group, the group of Japanese examinees, and the 1993-94 program students) were s i m i l a r (519.00, 490.00, and 515.96), but SDs of the sample under study were almost three times smaller than those of the t o t a l group. The considerable difference i n SD between the t o t a l group and the sample under study indicates that the sample was homogeneous i n terms of TOEFL scores. Obviously, t h i s i s because the sample was r e s t r i c t e d to successful applicants whose TOEFL scores met the minimum TOEFL score, rather than to applicants randomly selected from the true population. Table 4.2 below shows a Pearson Correlation matrix among the variables. When the c o r r e l a t i o n matrix of variables i s obtained, i t i s necessary to perform an omnibus te s t to make sure there i s an o v e r a l l s i g n i f i c a n t i n t e r r e l a t i o n e x i s t i n g among each p a i r of correlations i n the matrix (Cohen & Cohen, 1983, p. 85 & pp. 315-316). I f there i s no o v e r a l l s i g n i f i c a n t r e l a t i o n s h i p among the c o r r e l a t i o n , then the o v e r a l l r e l a t i o n s h i p i n the matrix r e s u l t s from random sampling error rather than from the meaningful association 51 Table 4.2 Pearson c o r r e l a t i o n matrix of the variables G P A G E N D E R S P E A K W R I T E S E C 1 S E C 2 S E C 3 T O E F L G P A 1 . 0 0 0 . 4 5 7 . 1 3 9 . 3 4 2 . 2 8 4 . 3 3 2 . 1 9 3 . 3 6 5 . 0 0 0 . 1 8 4 . 0 0 1 . 0 0 6 . 0 0 1 . 0 6 3 . 0 0 0 G E N D E R 1 . 0 0 0 . 0 0 3 . 2 8 3 . 2 5 2 . 0 6 7 - . 0 6 5 . 1 3 9 . 9 7 9 . 0 0 6 . 0 1 5 . 5 2 0 . 5 3 7 . 1 8 5 S P E A K 1 . 0 0 0 . 3 5 8 . 3 5 5 . 2 3 8 . 1 6 5 . 3 5 2 . 0 0 0 . 0 0 0 . 0 2 2 . 1 1 4 . 0 0 1 W R I T E 1 . 0 0 0 . 1 8 3 . 2 9 7 . 0 6 8 . 2 4 8 . 0 7 9 . 0 0 4 . 5 1 7 . 0 1 6 S E C 1 1 . 0 0 0 . 3 1 3 . 3 4 7 . 7 9 4 . 0 0 2 . 0 0 1 . 0 0 0 S E C 2 1 . 0 0 0 . 3 6 8 . 7 3 1 . 0 0 0 . 0 0 0 S E C 3 1 . 0 0 0 . 7 1 3 . 0 0 0 T O E F L 1 . 0 0 0 between each p a i r of variables. This i s almost the same as performing an omnibus F-test before post hoc t - t e s t s for means of each group i n ANOVA. In the present study, a B a r t l e t t Chi-square t e s t was performed to t e s t the o v e r a l l n u l l hypothesis that a l l possible sample correlations among the set of variables i n the matrix were zero. The r e s u l t rejects the n u l l hypothesis (p_<.001). This indicates that there i s a s i g n i f i c a n t 52 i n t e r r e l a t i o n among the enti r e set of Pearson c o r r e l a t i o n c o e f f i c i e n t s . Checking for v i o l a t i o n of assumptions Each a n a l y t i c a l model, such as the c o r r e l a t i o n model and the regression model, has been developed based on c e r t a i n e s s e n t i a l assumptions. I n t e l l i g e n t use of a n a l y t i c a l models must meet the assumptions underlying the models. V i o l a t i o n s of assumptions lead to estimate biases. Therefore, checking for v i o l a t i o n of underlying assumptions i s considered to be an indispensable component inherent i n a regression analysis. The following sections w i l l discuss (a) two general assumptions underlying any a n a l y t i c model, i . e . , the assumption of s p e c i f i c a t i o n errors and the assumption of measurement errors; (b) s i x s p e c i f i c assumptions underlying a regression a n a l y t i c model (Berry, 1993); (c) o u t l i e r s and i n f l u e n t i a l points. The assumption of s p e c i f i c a t i o n error. This assumption requires that an an a l y t i c model should flawlessly r e f l e c t i t s underlying rationale regarding the e f f e c t of independent variables on dependent variables. There are three types of s p e c i f i c a t i o n errors: (a) omission of relevant variables into the regression model; (b) incorrect s p e c i f i c a t i o n of the manner i n which independent variables a f f e c t the dependent variables and (c) incl u s i o n of i r r e l e v a n t variables (Pedhazur & Schmelkin, 1991, pp. 389-390). S p e c i f i c a t i o n errors are the most damaging as they pose the most serious threat to v a l i d interpretation of regression 53 r e s u l t s . However, i t i s d i f f i c u l t to t e l l whether a l l relevant variables have been included i n the model, i f a l l i r r e l e v a n t variables have been excluded, or whether the model has been c o r r e c t l y s p e c i f i e d i n the context of s o c i a l science research. The p r a c t i c a l way to avoid s p e c i f i c a t i o n errors i s to use a well-grounded theory to b u i l d an a n a l y t i c model. As Berry (1993) has pointed out, people should judge regression models by whether these models conform to t h e i r theories, and thus whether the models can be used to answer t h e i r research questions (P. 8). To reduce s p e c i f i c a t i o n errors i n the present study, the following e f f o r t s were made within data and time constraints. 1. Variables were selected for a regression analysis based upon knowledge about language and non-language factors that influence academic achievement (see chapter two). The present study used language-based predictors and also introduced an exploratory non-language variable, gender, into the regression. 2. The study focused on accurately estimating the unique contribution of TOEFL scores on GPA, rather than on measuring the e f f e c t s of a l l the variables i n the regression model. This i s because the primary research i n t e r e s t i s to know how well TOEFL scores, as a single predictor, can predict GPA, not how much variance i n GPA can be explained. Assumption of measurement errors. This assumption assumes that a l l variables under study are measured without 54 error. In r e a l i t y , t e s t scores unavoidably include measurement error. Berry (1993) has provided an extensive discussion of three types of measurement errors: random measurement errors, non-random measurement errors, and measurement errors involving the use of proxy variables (pp. 49-60) . The present study dealt with the issue of measurement error i n two ways: 1. Information about measurement of indicators, GPA, TOEFL scores, speaking scores, and writing scores, was gathered. I t i s almost impossible to perform a measurement without error i n s o c i a l science research. Information about a l l measures used i n the study were gathered i n order to i d e n t i f y possible measurement errors. In the present study, both GPA and TOEFL scores are among the most frequently used indicators i n educational practice, although the qu a l i t y of these measures have been long debated. Both speaking scores and writing scores are l o c a l l y used within the UBC/Ritsumeikan Program. Thus there i s s u f f i c i e n t information available about TOEFL and GPA, but not much about speaking scores and writing scores. 2. The findings were interpreted with special care. The regression model does not provide s u f f i c i e n t power to handle measurement errors l i k e LISREL does. Therefore, the present study c l e a r l y distinguishes the difference between TOEFL and language proficiency, and between GPA and academic achievement. With these d i s t i n c t i o n s i n mind, r e s u l t s of the 55 regression analysis were interpreted c a r e f u l l y so as to avoid overgeneralization. The assumption of l i n e a r i t y . The nature of the re l a t i o n s h i p s between predictors and c r i t e r i a , l i n e a r i t y or non-linearity, requires a proper model for regression analysis. As shown i n Figure 4.1, the residuals are randomly d i s t r i b u t e d and there are no systematic patterns e x i s t i n g between the predicted values and the residuals. This j u s t i f i e s the use of the l i n e a r regression model. Dependent V a r i a b l e : GPA - 1 0 1 2 R e g r e s s i o n S t a n d a r d i z e d P r e d i c t e d V a l u e Figure 4.1. Scatterplot of the d i s t r i b u t i o n of the residuals. The assumption that mean of the residual i s zero. This assumption means that the variance of the residuals i s constant for a l l l e v e l s of the independent variables. Figure 4.1 also shows that the spread of the residuals does not increase or decrease with the magnitude of the predicted 56 values on the X axis. This indicates that the above assumption was met. The assumption that residuals are independent. This assumption requires that residuals are independent of one another. V i o l a t i o n of t h i s assumption, often referred to as autocorrelation, a f f e c t s the v a l i d i t y of tests of s i g n i f i c a n c e . From Figure 4.1, we also can see that the residuals are randomly scattered above and below the zero horizontal band. This t e l l s us that autocorrelation does not occur and the above assumption i s met. The assumption of normal d i s t r i b u t i o n of residuals. This assumption requires that residuals should d i s t r i b u t e normally. In the histogram of Figure 4.2, the d i s t r i b u t i o n of the residuals appears approximately normal. Dependent V a r i a b l e : GPA Regression Standardized Residual Figure 4.2. D i s t r i b u t i o n of residuals. 57 The assumption of the absence of perfect m u l t i c o l l i n e a r i t y . This assumption assumes that there i s no strong i n t e r c o r r e l a t i o n s among independent variables. The existence of c e r t a i n c o r r e l a t i o n among the independent variables indicates high m u l t i c o l l i n e a r i t y . The tolerance of an independent variable i s a commonly used measure of m u l t i c o l l i n e a r i t y . In the present study, the tolerance of the predictor variables ranged from .63 to .89. This implies that the above assumption i s s a t i s f i e d . The assumption that residuals are not correlated with each of the independent variables. In the present study, a c o r r e l a t i o n analysis was performed to check t h i s assumption. The r e s u l t s showed that a l l the c o r r e l a t i o n c o e f f i c i e n t s between the independent variables and the residuals were .00 except that between gender and the residuals (r=.371). Therefore, t h i s assumption was also s a t i s f i e d . Diagnosis of o u t l i e r s and i n f l u e n t i a l points. Two frequently used measures, standardized residual and centered leverage were selected to detect o u t l i e r s and i n f l u e n t i a l points respectively. As shown i n Appendix II, a l l the standardized residuals are below 3 units from zero, and thus no o u t l i e r s are found. However, case 73 has a leverage of .241 which i s twice as large as the upper l i m i t of normal leverage values. I t turned out that t h i s case had a very low The considerable r e l a t i o n s h i p between gender and the residuals c l e a r l y indicates again that gender i s a composite var i a b l e which i n t e r r e l a t e s with other variables, known and unknown, or currently available and unavailable for research. 58 TOEFL t o t a l score, 487, but i t s GPA, 77, was f i v e points higher than the GPA mean. Case 73, therefore, was i d e n t i f i e d as an i n f l u e n t i a l point and eliminated before performing the multiple regression analysis. H i e r a r c h i c a l regression analysis The present study employs the multiple l i n e a r regression analysis with a h i e r a r c h i c a l procedure instead of a stepwise procedure that i s used most commonly. This decision i s made based on the comparison among three options i n the procedure of the multiple regression analysis. The primary purpose of the present study was to estimate the pr e d i c t i v e v a l i d i t y of TOEFL scores. That i s , the study aimed at estimating the unique contribution of TOEFL scores to GPA, rather than the o v e r a l l contribution of a l l predictors to GPA or the best l i n e a r combination of predictors to predict GPA, by p a r t i a l l i n g out the res t of the predictors under study. To accomplish t h i s , the key issue was to determine the order or sequence of entering the predictors because d i f f e r e n t entry orders y i e l d d i f f e r e n t estimates of the unique contribution of a predictor. Generally speaking, there are three a l t e r n a t i v e procedures for the multiple regression analysis: simultaneous, stepwise, and h i e r a r c h i c a l . In the simultaneous analysis, every predictor i s entered into the regression analysis simultaneously and i s p a r t i a l l e d out from every other predictor indiscriminately. This procedure can provide a regression equation and squared R for a l l 59 predictors i n the equation, but i t does not estimate the unique contributions of each variable to the t o t a l variance i n the equation. Stepwise analysis can estimate the unique contribution of predictors by obtaining p a r t i a l c o r r e l a t i o n or incremental variance. However, t h i s procedure s o l e l y r e l i e s on s t a t i s t i c a l c r i t e r i a to determine the sequence of entering predictors. When the competing predictors s u b s t a n t i a l l y connect with each other, the p a r t i a l c o r r e l a t i o n or incremental variance might vary s i g n i f i c a n t l y according to the sequence i n which predictors are entered. Thus, the procedure of stepwise analysis might create d i f f i c u l t i e s i n estimating, interpreting, comparing, and r e p l i c a t i n g the regression r e s u l t s . The h i e r a r c h i c a l procedure enters predictors i n a pre-sp e c i f i e d sequence to estimate the unique contribution of each predictor to the t o t a l variance i n the regression equation. The choice of a p a r t i c u l a r sequence of predictors i s made i n advance by the purpose and l o g i c of the research, i n contrast to the stepwise regression. The h i e r a r c h i c a l procedure leads to tests of the hypotheses that define the order and improve our understanding of the phenomena under study (Cohen, 1983, pp. 120-125). As Tabachnick and F i d e l l (1989) discussed, simultaneous, stepwise, and h i e r a r c h i c a l regression can be best used for model-estimating, model-building, and model-testing respectively (P . 150). Thus, i n 60 the present study, h i e r a r c h i c a l procedure was selected to te s t the hypotheses of the present study. The sequence of entering the predictors i n the present study was TOEFL t o t a l scores, writing scores, speaking scores, and gender. This sequence was mainly based on the research p r i o r i t y of the study because no causal r e l a t i o n s h i p among the predictors was found. Since TOEFL scores r e f l e c t e d the major goal of the research and were the primary focus of the study, they were entered into the equation f i r s t . Writing scores and speaking scores followed because they were viewed as having lesser relevance to the research than TOEFL scores. Gender was entered l a s t because i t was used as an explanatory variable to exemplify non-language predictors' pr e d i c t i v e v a l i d i t y which was not much documented i n research l i t e r a t u r e . Within the TOEFL sectional scores, the sequence for entering was from sectional I I , I I I , and I. This was based on a descending order i n terms of three scores' pr e d i c t i v e v a l i d i t y reported i n the previous research (Abdzi, 1967; Aleamoni, 1974; Harvey, 1979; H e i l & Johnson, 1988; Z i r p o l i , 1 9 8 8 ) . In the following section, the f i r s t h i e r a r c h i c a l regression analysis was performed to t e s t the primary hypothesis regarding the pred i c t i v e v a l i d i t y of TOEFL t o t a l scores. The second h i e r a r c h i c a l regression analysis then mainly served to examine the unique contribution of each TOEFL sectional scores to GPA. Note that change i n squared R 61 ( A R 2 ) was used i n SPSS as the estimator f o r unique contribution of each predictor to GPA i n a h i e r a r c h i c a l regression analysis. 9 H i e r a r c h i c a l analysis with TOEFL t o t a l scores. Table 4.3 shows that the r e s u l t s of the h i e r a r c h i c a l analysis with TOEFL t o t a l scores. Table 4.3 Summary table of the h i e r a r c h i c a l analysis with TOEFL t o t a l scores S t e p R A d j R 2 F P A R 2 A F A P V a r i a b l e 1 . 3 7 7 . 1 3 2 14 . 8 9 5 . 0 0 0 . 1 4 2 14 . 8 9 5 . 0 0 0 I n : T O E F L 2 . 4 4 8 . 1 8 3 11 . 1 8 4 . 0 0 0 . 0 5 9 6 . 5 5 5 . 0 1 2 I n : W R I T E 3 . 4 5 7 . 1 8 2 7 . 7 3 1 . 0 0 0 . 0 0 8 • 8 5 9 . 3 5 7 I n : S P E A K 4 . 5 8 4 . 3 1 1 11 . 2 5 1 . 0 0 0 . 1 3 2 1 4 . 4 7 0 . 0 0 0 I n : G E N D E R In step 1, TOEFL t o t a l scores entered into the regression equation i n order to determine the extent to 9 2 • • In the present sturdy A R i s interpreted as the amount of variance added to R by each predictor at the point that i t enters the equation i n a h i e r a r c h i c a l procedure. For the distinguished differences i n the 2meaning of unique contribution of a predictor to R among the three procedures due to the differences i n handling the overlapping among correlated predictors, see Tabachnick and F i d e l l , 1989, pp. 141-142 & pp. 150-154. 62 which the TOEFL o v e r a l l score predicted GPA. Results showed that 14.20% of the variance i n GPA was accounted for by the TOEFL t o t a l score. The rest of about 86% of the variance remained as residual or unexplained error which should not be misinterpreted as measurement error. This mainly implies that the amount of variance had not yet been explained. In step 2, writing scores entered into the regression equation. The r e s u l t showed that i t accounted for 5.90% of the variance. By adding writing scores, the squared R i n the equation increased to .183. In step 3, the addition of speaking scores to the regression equation only increased 0.70% of variance accounted for. This may indicate that speaking p r o f i c i e n c y of the students did not contribute to t h e i r academic achievement s i g n i f i c a n t l y . Step 4 was used to examine the e f f e c t of the addition of an explanatory non-language variable to the regression model. By addition of gender, the squared R increased to .31 and the squared R change was .132. This showed that gender difference placed one of the largest weight on GPA. To further analyze the e f f e c t of gender difference, an ANOVA on TOEFL t o t a l scores and gender was performed. Results showed that there was no s i g n i f i c a n t difference of TOEFL scores due to gender, F (1,90) = 1.476, p > .05. This implied that the variance of GPA was not due to gender difference i n TOEFL t o t a l scores. In other words, the difference i n academic achievement appeared not to be 63 affected s i g n i f i c a n t l y by the d i f f e r e n t l e v e l s of English proficiency, but rather, by the differences i n non-language factors among male and female students. Hi e r a r c h i c a l analysis with TOEFL sectional scores. Table 4 . 4 shows that the re s u l t s of a h i e r a r c h i c a l analysis with TOEFL sectional scores. Table 4 . 4 Summary table of the h i e r a r c h i c a l analysis with TOEFL sectional scores tep R A d j R 2 F P A R 2 A F A P Variable 1 • . 3 3 4 . 1 0 2 11 . 2 9 7 . 0 0 1 . 1 1 2 11 . 2 9 7 . 0 0 1 In: S E C 2 2 . 3 4 1 . 0 9 6 5 . 8 5 7 . 0 0 4 . 0 0 5 . 4 8 2 . 4 8 9 In: S E C 3 3 . 4 0 0 . 1 3 1 5 . 5 9 2 . 0 0 2 . 0 4 4 4 . 5 9 0 . 0 3 5 In: S E C 1 4 . 4 5 5 . 1 7 0 5 . 674 . 0 0 0 . 0 4 7 5 . 1 3 0 . 0 2 6 In: W R I T E 5 . 4 6 6 . 1 7 1 4 . 7 6 2 . 0 0 7 . 0 1 0 1 . 0 9 3 . 2 9 9 In: S P E A K 6 . 5 8 9 . 3 0 0 7 . 5 1 3 . 0 0 0 . 1 3 0 1 6 . 8 7 0 . 0 0 0 In: G E N D E R The second h i e r a r c h i c a l analysis used three TOEFL subscores as the predictors rather than a composite TOEFL score. Results showed that, while high tolerances for three sectional scores revealed low i n t e r r e l a t i o n s among the subscores, TOEFL section II had the highest squared R 64 2 2 change(AR =.112), compared with section I ( AR =.044), section III (AR =.005). In other words, section II scores among them had the most importance impact on the variance i n GPA. The unique contribution of writing scores to the equation was .047, speaking scores was .010, gender was .130. Compared with writing scores .059, speaking scores .008, and gender .132 as shown i n the f i r s t h i e r a r c h i c a l analysis, the r e s u l t s indicated that the pattern of the r e l a t i v e importance among the three predictors did not change, while TOEFL t o t a l scores were par t i t i o n e d into three sectional scores. Summary The r e s u l t s of the present study show that the pr e d i c t i v e v a l i d i t y of TOEFL t o t a l scores was .142 (p_<.001). For TOEFL sectional scores, section II scores were the most important of three sectional scores. Among a l l predictors i n the study, gender had the highest pr e d i c t i v e v a l i d i t y on GPA when TOEFL sectional scores were used. Gender, TOEFL t o t a l scores, writing scores, speaking scores accounted for 31.00% (p_<.001) of the variance i n GPA. Gender, TOEFL sectional scores, writing scores, speaking scores accounted for 3 0.04% of the variance i n GPA (p_<.001). 65 Chapter Five Discussion This chapter discusses findings pertinent to the research hypotheses, interprets the implications of these findings, and draws conclusions of the study. Predictive v a l i d i t y of TOEFL t o t a l scores on GPA The present study examined the p r e d i c t i v e v a l i d i t y of a predictor variable i n two ways: (a) evaluation of i t s unique contribution to GPA on the basis of the four operational l e v e l s of p r e d i c t i v e v a l i d i t y (see Chapter three) and (b) assessment of i t s r e l a t i v e importance i n comparison with other predictors under study. Results of the present study show that A R 2 of TOEFL t o t a l scores i s .142 (p_<.001). This r e s u l t i s comparable with r e s u l t s of a meta-analysis of 27 TOEFL predi c t i o n studies i n which the mean c o r r e l a t i o n c o e f f i c i e n t s of TOEFL t o t a l scores and f i r s t year's GPA i s .300, i . e . , R 2 i s 9% (Yan, 1994). According to the operational l e v e l s of p r e d i c t i v e v a l i d i t y , therefore, TOEFL t o t a l scores have a medium l e v e l of the p r e d i c t i v e v a l i d i t y on f i r s t term's GPA. The r e s u l t s also reveal that TOEFL t o t a l scores are ranked as the second largest among a l l the predictors under study and as the largest compared with the other two language-based predictors, writing and speaking scores. Thus, TOEFL t o t a l scores are an important predictor of GPA i n the present study. 66 Based on these two findings reported above, i t can be concluded that hypothesis I of the study i s supported. That i s , TOEFL t o t a l scores have a medium l e v e l of the p r e d i c t i v e v a l i d i t y on f i r s t term's GPA for the group of students under study. I t i s not sur p r i s i n g that TOEFL t o t a l scores only account for 14.20% of the variance i n GPA. As shown i n many studies, English language proficiency i s just one of many factors a f f e c t i n g academic achievement. I t appears that no single factor alone can completely or larg e l y determine academic achievement. As the unique contribution of a single predictor to GPA, therefore, 14.20% c l e a r l y indicates that TOEFL t o t a l scores alone do explain a s i g n i f i c a n t amount of variance i n GPA. In other words, language pr o f i c i e n c y by i t s e l f , among many other factors, does have an important e f f e c t on academic achievement. We can further analyze how TOEFL's pr e d i c t i v e v a l i d i t y on GPA i s affected by each p a i r of TOEFL scores and GPA for each student under study. As shown i n Figure 5.1, we can divide each GPA-TOEFL p a i r into four d i v i s i o n s by using the mean GPA and the mean TOEFL t o t a l scores: upper l e f t , upper r i g h t , lower l e f t , and lower r i g h t . Both upper r i g h t and lower l e f t d i v i s i o n s share one commonality: A high TOEFL score corresponds to a high GPA, or a low TOEFL score with a low GPA. However, both the upper l e f t and lower r i g h t d i v i s i o n s show that a high TOEFL 67 score goes with a low GPA, or a low TOEFL with a high GPA. Note that there are 24 cases i n the upper l e f t d i v i s i o n , whereas only 15 i n the lower r i g h t . Among these cases, there are at l e a s t 6 cases with TOEFL scores below 4 80 but t h e i r GPAs are above the mean GPA, whereas there i s only one case with a TOEFL score above 540 and a GPA below 6 0 . This indicates that i n the present study about one quarter of the students who had low l e v e l s of language pro f i c i e n c y managed to achieve academic success. The number of t h i s sub-group i s higher than that of students who have the high l e v e l of language pro f i c i e n c y but are unable to reach the high l e v e l of academic achievement. In other words, a good TOEFL score does not necessarily guarantee a good GPA, but a low TOEFL score i s often associated with a good GPA. I t i s these cases that might considerably decrease the magnitude of the 68 p r e d i c t i v e v a l i d i t y of TOEFL scores. They prove again that for those whose native language i s not English, many factors are involved i n t h e i r academic learning at u n i v e r s i t i e s and language pro f i c i e n c y does not always function predominantly as a key element. Predictive v a l i d i t y of TOEFL sectional scores on GPA Results of the study show that the combination of three sectional scores accounts for 16.10% of the variance i n GPA. This i s close to what TOEFL t o t a l scores do. However, the unique contribution and r e l a t i v e importance of three sectional scores to GPA are remarkably d i f f e r e n t . I t 2 i s shown that the changes i n R of section I, II, and III are .044 (p_>.05), .112 (P<.001), and .005 (p_>.05) which indicate they have small, medium, and n e g l i g i b l e l e v e l s of p r e d i c t i v e v a l i d i t y respectively. Section II scores have the second highest p r e d i c t i v e v a l i d i t y among the six predictors under study and the highest among three TOEFL sectional scores. This finding i s comparable with those i n previous research (Johnson, 1988; Z i r p o l i , 1988; Light, Xu, & Morris, 1989). Thus, based on the uneven contributions of three sectional scores to GPA as well as t h e i r d i f f e r e n t importance, the conclusions for hypothesis II are: Section II have a medium l e v e l of predictive v a l i d i t y , section I scores have a small l e v e l of predictive v a l i d i t y , and section III scores have a n e g l i g i b l e l e v e l of p r e d i c t i v e v a l i d i t y . 69 There i s an i n t e r e s t i n g question to ask: Among three sectional scores of TOEFL, why do section II scores tend to be so dominant i n predicting GPA? In the TOEFL te s t , section I, Listening Comprehension, measures the a b i l i t y to understanding o r a l English; Section I I , Structure and Written Expression, tests recognition of selected s t r u c t u r a l and grammatical knowledge in standard written English; And section I I I , Vocabulary and Reading Comprehension, tests the a b i l i t y to understand written English (ETS, 1992, pp. 6-7). The three sections measure 10 l i s t e n i n g s k i l l s , writing knowledge, and reading s k i l l s r espectively. Thus, the findings presented here might indicate that a good GPA may demand more written English s k i l l s than spoken English s k i l l s ( i . e . , section II scores vs. section I scores). Furthermore, for written English s k i l l s , a good GPA may require more productive s k i l l s than receptive s k i l l s of written English ( i . e . , section II scores vs. section III scores). As seen i n Table 3.1, about 85% of the course grades require written productive s k i l l s to f u l f i l l various academic tasks such as term paper, course assignments, and f i n a l examinations. When students* academic achievement i s assessed mainly based on performance i n written expression, the weight of section II scores on GPA i s greater than the other two sectional scores. To •" ETS has not explained what i s exactly meant by structure and written expression. Since section II uses both sentence correction and sentence completion to t e s t basic knowledge about written English, the present study simply labels section II as writing knowledge instead of writing s k i l l s . 70 Predictive v a l i d i t y of writing scores on GPA Results of the study reveal that the change of R2 of writing scores i s .059 (p_<.05) when TOEFL t o t a l scores are used. This means that writing scores have a small l e v e l of p r e d i c t i v e v a l i d i t y on GPA. I t i s also shown that the r e l a t i v e importance of writing scores are ranked t h i r d among four predictors, behind gender and TOEFL t o t a l scores, and before speaking scores. These findings to some extent support hypothesis III i n the study and indicate that writing scores have a small l e v e l of p r e d i c t i v e v a l i d i t y on GPA. I t i s i n t e r e s t i n g to note that p r e d i c t i v e v a l i d i t y of writing scores i s s u b s t a n t i a l l y lower than that of section II scores (.047 vs. .112 for AR2) when TOEFL sectional scores are used. Both deal with measurement of written English, but why do writing scores have so l i t t l e contribution to GPA compared with i t s counterpart? There might be tentative explanations to t h i s question. For instance, as a l o c a l l y used t e s t i n g instrument, the writing sample assessment might not possess s u f f i c i e n t r e l i a b i l i t y and v a l i d i t y i n measuring English writing s k i l l s as i t should. This may r e s u l t i n under-estimation of i t s p r e d i c t i v e v a l i d i t y on GPA. Also, Section II scores measure writing knowledge, while writing scores d i r e c t l y assess writing s k i l l s . For t h i s group of Japanese students whose English i s at the intermediate l e v e l , they might need more 71 basic writing knowledge of written English i n order to f u l f i l l t h e i r academic learning tasks successfully. Predictive v a l i d i t y of speaking scores on GPA The r e s u l t s of the study indicate that speaking scores have a n e g l i g i b l e l e v e l of p r e d i c t i v e v a l i d i t y on GPA (AR2=.O08, p_>.05 or AR2=.010, p>.05, depending on TOEFL t o t a l scores or sectional scores are used) and i s consistently ranked as the l e a s t important among a l l the predictors i n predicting GPA. Therefore, hypothesis IV regarding the p r e d i c t i v e v a l i d i t y of speaking scores i s rejected. Due to i n s u f f i c i e n c y of information about the r e l i a b i l i t y and v a l i d i t y of the o r a l interview used i n t h i s academic exchange program, the question concerning why speaking scores have a medium l e v e l of p r e d i c t i v e v a l i d i t y on GPA must be l e f t for future analysis. Writing scores, speaking scores, and TOEFL scores i n section I, I I , and III could be seen to assess four language s k i l l s : l i s t e n i n g , speaking, reading, and writing. I t i s i n t e r e s t i n g to look at weights of i n d i v i d u a l s k i l l s i n predicting GPA as well as t h e i r o v e r a l l e f f e c t s on GPA. F i r s t , the r e s u l t s of the study suggest that written s k i l l s i n English are more important than oral s k i l l s i n p r e d i c t i n g GPA, as we have seen that AR2 of section II scores i s larger than that of section I and AR2 of writing scores i s larger than that of speaking scores. Also, the r e s u l t s tend to indicate that productive s k i l l s i n written 72 English are more important than receptive s k i l l s since section II scores and writing scores have more pr e d i c t i v e power on GPA than section III scores. However, we are unable to i n t e r p r e t the r e l a t i v e importance on GPA of written comprehension and aural comprehension, although A R 2 of section I scores exceed that of section I I I . A l l i n a l l , i t seems premature to draw a conclusion on the basis of findings of the present study about the r e l a t i o n s h i p of those four language s k i l l s and academic achievement. Second, r e s u l t s show that the cumulative R 2 of f i v e language-based predictors, speaking scores, writing scores, and the three sectional scores, i s .217 (p_<.001). Thus, i t could be i n f e r r e d that the o v e r a l l p r e d i c t i v e v a l i d i t y of language factors on GPA might have an upper l i m i t . Probably R 2 of language factors i n a regression model would probably not exceed .25. In other words, among many other variables, language factors alone might optimize t h e i r contribution at about one-quarter of academic achievement assessed by GPA. Predictive v a l i d i t y of gender on GPA The r e s u l t s show that gender's A R 2 i s .13 2, (p_<.001) when TOEFL t o t a l scores are used, and .130 (p_<.001) when the TOEFL sectional scores are used. The findings indicate that gender has a medium l e v e l of p r e d i c t i v e v a l i d i t y on GPA. Gender i s consistently ranked as one of the most powerful predictors under study. Therefore, i t can be concluded that 73 gender i s a good predictor i n predicting GPA and hypothesis V i s strongly supported. Many studies have already proved that gender differences do influence academic achievement. However, i t i s s t i l l s u r p r i s i n g that gender had such a remarkable contribution to GPA i n the present study. Japanese female students as a group performed s i g n i f i c a n t l y better than male students i n the course grades. This unanticipated finding raises a question: Why do gender differences a f f e c t GPA so greatly? In other words, why do female students academically excel over t h e i r male counterparts? As reviewed i n chapter two, research on gender differences reveals that gender i s a composite factor influenced by and impacting on various factors i n s o c i a l , educational, l i n g u i s t i c , psychological, and p h y s i o l o g i c a l domains. Generally speaking, female students perform better i n language arts and male students perform better i n science. To f i n d the possible cause for the gender difference i n GPA, an F-test on gender difference i n TOEFL scores was conducted. The r e s u l t s showed that there were no s i g n i f i c a n t gender differences i n these scores, although a l l three courses from which GPA were obtained were about language education and required good language profi c i e n c y . This r e s u l t c l e a r l y indicates that gender differences i n GPA are not caused by language factors but by other non-language factors. Probably non-language factors such as learning motivation, time spent on learning, academic aptitude, 74 learning s t y l e , previous knowledge, and c u l t u r a l adaptability, might i n d i r e c t l y place e f f e c t s on GPA through gender differences. Due to lack of data to analyze, what kinds of non-language factors and how they contribute to GPA for t h i s group of students remain open for future research. Implications The findings i n the present study may have p r a c t i c a l implications for the UBC/Ritsumeikan Academic Exchange Program. 1. The main findings i n the study c l e a r l y indicate that TOEFL t o t a l scores are a good predictor of f i r s t term's GPA for the UBC/Ritsumeikan Program students. Therefore, the program should continue to use TOEFL to measure English language proficiency for program admissions. Since TOEFL section II scores have the highest p r e d i c t i v e v a l i d i t y among three sectional scores, they deserve p a r t i c u l a r attention for admission s e l e c t i o n . 2. The findings on gender differences i n GPA strongly suggest that non-language factors play an important r o l e i n 11 . . academic achievement. Thus, i t i s advisable that the This finding might lead mistakenly to another implication for program admissions: including more female students into the program and excluding more male ones from the program. In fact, t h i s p o l i c y , given i t was taken, would be not only p o l i t i c a l l y incorrect but also l o g i c a l l y oversimplified. As discussed i n the previous chapters, the true reasons for gender difference i n academic achievement are not due to sex difference, but rather, a combination of physical, cognitive, emotional, s o c i a l factors embedded i n gender difference. Therefore, for an i n t e l l i g e n t educator, he or she should always f i n d s p e c i f i c factors behind gender difference i n academic achievement i n order to help 75 program should gather as much information, p a r t i c u l a r l y non-language data, as possible i n order to select the most promising applicants. These types of information include previous GPA at Ritsumeikan University, l e t t e r s of recommendation, scores i n academic aptitude t e s t , and personal statements of i n t e r e s t s . More factors such as c u l t u r a l knowledge, LI l e v e l , motivation, i n t e l l i g e n c e , and personality, should be taken into consideration i n making admission decisions. 3. The findings reveal that TOEFL scores alone do not absolutely ensure academic success. Thus, i t i s recommended that the currently used minimum TOEFL score of 550 not be used as a requirement for r e g i s t r a t i o n i n regular UBC courses. Rather, an appropriate c r i t i c a l range of TOEFL scores should be established for program admission and management. In p a r t i c u l a r , for those who have low TOEFL scores but c l e a r l y show academic pot e n t i a l , the decision makers i n the program should have a special p o l i c y for them so as to s a t i s f y t h e i r learning needs and academic c a p a b i l i t i e s . The present study may have t h e o r e t i c a l implications. The issue of whether or not TOEFL scores can predict GPA has been debated for over 30 years. The study examined thoroughly the underlying rationale for TOEFL predi c t i o n studies and proposed a comprehensive framework for the students, no matter male or female, to achieve t h e i r academic potentials. 76 analysis of factors a f f e c t i n g academic achievement. For i t s research design, the present study c a r e f u l l y considered technical treatments i n predictor c o l l e c t i o n , c r i t e r i o n s e l e c t i o n , a n a l y t i c a l models, regression procedure, and v a l i d i t y l e v e l s i n order to ensure the correct estimation of TOEFL scores* pr e d i c t i v e v a l i d i t y . For these reasons, i t may be thought that the present study might have taken a further step i n solving the 30-year's TOEFL-GPA puzzle i n terms of i t s comprehensive rationale and i t s improved methodology. Limitations of the study 1. The present investigation did not include more relevant non-language predictors into the multiple regression analysis due to t h e i r current u n a v a i l a b i l i t y . This might cause possible s p e c i f i c a t i o n errors i n the multiple regression model used. 2. GPA used i n the study was from three "Bridge Courses" designed s p e c i f i c a l l y for the program. Compared to regular UBC courses, these courses may have d i f f e r e n t features such as course grade standards, i n s t r u c t o r and teaching assistant's a l l o c a t i o n , and communicative language environments. The uniqueness of t h i s type of GPA might make uncertain the v a l i d i t y and g e n e r a l i z a b i l i t y of the study. 3. The study did not estimate the e f f e c t of r e s t r i c t i o n of range i n TOEFL scores on the r e s u l t s of the multiple regressions analysis. Research l i t e r a t u r e has indicated that r e s t r i c t i o n of range i n admissions w i l l r e s u l t i n underestimation of the predictive v a l i d i t y , but we s t i l l 77 need empirical evidence to know to what extent and under what circumstances t h i s underestimation may occur. Directions for future research 1. The present study can be expanded into a time-series research project. On the basis of the data avai l a b l e for four years (1991-1995), we can examine the change pattern of TOEFL's pr e d i c t i v e v a l i d i t y on f i r s t term's GPA over years. I t i s also f e a s i b l e to compare the re l a t i o n s h i p of TOEFL's pr e d i c t i v e v a l i d i t y to d i f f e r e n t kinds of GPA, such as second term's GPA and f i r s t year's GPA. 2. Other a n a l y t i c a l models and s t a t i s t i c a l techniques can be used i n the study. For instance, Multivariate Analysis of Variance can be used to analyze d i f f e r e n t GPA subscores; H i e r a r c h i c a l Linear Model can be adopted to examine the s p e c i f i c e f f e c t s of d i f f e r e n t units such as in d i v i d u a l , group, course, and instructor, on TOEFL's pr e d i c t i v e v a l i d i t y ; Linear Structural Relations can be employed to di s t i n g u i s h d i r e c t and i n d i r e c t r e l a t i o n s h i p s among variables and assess the extent of measurement error that may appear. 3. A series of prediction studies can be developed to compare the pred i c t i v e v a l i d i t y of TOEFL with those of other language t e s t s such as Michigan Test of English language Proficiency (MTELP) and C e r t i f i c a t e of Proficiency i n English (CPE), and those of with aptitude tests such as Graduate Record Examinations (GRE) and Scholastic Aptitude Test (SAT). 78 4. I t should be further examined how language proficiency, i n p a r t i c u l a r , speaking, l i s t e n i n g , reading, and writing, are related to academic achievement. 5. Case studies can be conducted to explore some spec i a l issues i n depth. For instance, why are some students with good TOEFL scores unable to achieve academic success? Why do some other students eventually overcome t h e i r language problems and meet t h e i r academic challenges? What differences e x i s t between female and male students i n motivation, c u l t u r a l adaptability, IQ, previous GPA, and other domains. 6. Decision theory (see Cronbach & Glaser, 1965) should be introduced i n order to use TOEFL scores properly for admissions decision-making and program management. Conclusion The present study employed TOEFL scores as well as other predictors to predict f i r s t term's GPA with a multiple regression h i e r a r c h i c a l a n a l y t i c approach. For the UBC/Ritsumeikan Academic Exchange Program students, the following conclusions can be drawn from the findings of the present study: 1. TOEFL t o t a l scores alone have a medium l e v e l of p r e d i c t i v e v a l i d i t y on f i r s t term's GPA. 2. TOEFL Section scores II, section I scores, and section III scores have the predictive v a l i d i t y on f i r s t term's GPA at medium, small, and n e g l i g i b l e l e v e l s respectively. 3 . Writing scores alone have a small l e v e l of pr e d i c t i v e v a l i d i t y on f i r s t term's GPA. 4 . Speaking scores alone have a n e g l i g i b l e l e v e l p r e d i c t i v e v a l i d i t y on f i r s t term's GPA. 5. Gender alone has a medium l e v e l of p r e d i c t i v e v a l i d i t y on f i r s t term's GPA. 80 Bibliography Adamson, H. D. (1990). ESL students' use of academic s k i l l s i n content courses. English for S p e c i f i c Purpose, 9. 67-87. Alderman, D. L. (1982). Language Proficiency as a Moderator Variable i n Testing Academic Aptitude. Journal of Educational Psychology 74. 580-87. American Psychology Association. (1994). Publication Manual of American Psychology Association (4th ed.). Washington, DC: American Psychology Association. American Psychology Association. (1985). Standards for educational and psychological t e s t s . Washington, DC: American Psychology Association. Ames, R. & Ames, C. (1984) Research on motivation i n education (volume 1). Orlando, F l o r i d a : Academic Press, Inc. Arena, L. (ed.) (1990). Language profic i e n c y : defining, teaching, & t e s t i n g . New York: Plenum Press. Ayers, J . B. & Peters R. M. (1977). Predictive v a l i d i t y of the Test of English as a Foreign Language for Asian graduate students i n engineering, chemistry, or mathematics. Educational & Psychological Measurement, 37, 461-463. Ayers, J . B. & Quattlebaum, R. F. (1992). TOEFL Performance and Success i n a Masters Program i n Engineering. Educational and Psychological Measurement, 52, 973-75. The s t y l e of the present thesis followed consistently the four e d i t i o n of Publication Manual of the American Psychological Association (APA, 1994) throughout the t h e s i s . 81 Bachman, L. F. (1991). What does language t e s t i n g have to offer? TESOL Quarterly. 25. 671-704. Bachman, L. F. (1990). Fundamental considerations i n language t e s t i n g . Hong Kong: Oxford University Press. Bachman, L. F., Davidson, F., & Foulkes J . (1990). A comparison of the a b i l i t i e s measured by the Cambridge and educational t e s t i n g service EFL t e s t b a t t e r i e s . Issues i n Applied L i n g u i s t i c s . 1. 30-54. Berry, W. (1993). Understanding regression assumptions. Beverly H i l l s , CA: Sage Publications, Inc. Berry, W & Feldman, S. (1985). Multiple regression i n practi c e . Beverly H i l l s , CA: Sage Publications, Inc. Berwick, R. & McMichael, W. (1992). 1991-92 Ritsumeikan evaluation report. Unpublished manuscript, University of B r i t i s h Colombia, Vancouver, B r i t i s h Columbia, Canada. Berwick, R. & McMichael, W. (1993). 1992-93 Ritsumeikan evaluation report. Unpublished manuscript, University of B r i t i s h Columbia, Vancouver, B r i t i s h Columbia, Canada. Best, J . & Kahn, J . (1989). Research i n education (6th ed.). Englewood C l i f f s , NJ: Prentice-Hall. Black, J . (1991). Performance i n English s k i l l s courses and o v e r a l l academic achievement. TESL Canada Journal, 9, 42-56. Bloom, B. et a l . (1964). Taxonomy of educational objectives: Handbook II A f f e c t i v e domain. New York: David McKey Company, Inc. 82 Bloom, B. et a l . (1956). Taxonomy of educational objectives: Handbook I Cognitive domain. New York: David McKey Company, Inc. Bosher, S. & Rowekamp, J. (1992). Language prof i c i e n c y and academic success: the refugee/immigrant i n higher education. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED353914) Bruner, J . (1966). Toward a theory of i n s t r u c t i o n . Cambridge, MA: Harvard University Press. Buteyn, R. J . (1989). Gender and academic achievement i n education. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED313103) Carlson, S. B. (1985). Relationship of admission t e s t scores to writing performance of native and nonnative speakers of English. East Lansing, MI: National Center f o r Research on Teacher Learning. (ERIC Document Reproduction Service No. ED268135) Ching, R. J . & Moore, C. A. (1993). ESL assessment: What we learn when we open Pandora 1s box. Metropolitan U n i v e r s i t i e s , 3. 35-46. Choy, S. C , Davenport, B. M. (1986). The TOEFL: incomplete t e s t of English proficiency. College Teaching. 34. 108-110. Christopher, V. (1993). Direct and i n d i r e c t placement t e s t scores as measures of language proficiency and 83 predictors of academic success for ESL students. Unpublished master's thesis, University of B r i t i s h Columbia, Vancouver, B r i t i s h Columbia, Canada. Cocking, R & Messier, J . (1988). L i n g u i s t i c and c u l t u r a l influences on learning mathematics. H i l l s d a l e , NJ: Lawrence Erlbaum Associates, Publishers. Cohen, J . (1988). S t a t i s t i c a l power analysis for the behavioral sciences (2nd ed.). H i l l s d a l e , NJ: LEA. Cohen, J & Cohen, P. (1983). Applied Multiple regression/correlation analysis for the behavioral sciences (2nd ed.). H i l l s d a l e , NJ: LEA. C o l l i e r . V. P. (1989). How long? A synthesis of research on academic achievement i n a second language. TESOL Quarterly, 23, 509-531. Crandall, J . & Dale, T. (1987). ESL through content-area i n s t r u c t i o n : mathematics, science, s o c i a l studies. Eaglewood C l i f f s , NJ: Prentice-Hall. Crowhurst, M. (1994). Language and learning across the curriculum. Scarborough, Ontario: A l l y n & Bacon Canada. Cummins, J . (1992). Language proficiency, bilingualism, and academic achievement. In Richard-Clmato, P. A. & Snow, M. A. (eds.) The m u l t i c u l t u r a l classroom (pp. 16-69). London: Longman. Cummins, J . (1991). Language development and academic learning. In Malave, L & Duguette, G. (Eds.). Language, Culture and cognition, (pp. 161-189). Clevedon, England: M u l t i l i n g u a l Matters Ltd. 84 Cummins, J . (1984). Bilingualism and spe c i a l education: issues i n assessment and pedagogy. Clevedon, England: M u l t i l i n g u a l Matters Ltd. DeMauro, G. (1992). Examination of the Relationships among TSE, TWE, and TOEFL Scores. Language Testing. 9. 149-161. Educational Testing Service. (1994a). TOEFL t e s t and score manual supplement. Princeton, NJ: Educational Testing Service. Educational Testing Service. (1994b). TOEFL Update. Princeton, NJ: Educational Testing Service. Educational Testing Service. (1994c). 1994-95 B u l l e t i n of information for TOEFL. TWE. and TSE. Princeton, NJ: Educational Testing Service. Educational Testing Service. (1994d). The Researcher. Princeton, NJ: Educational Testing Service. Educational Testing Service. (1994e). Checklist for using TOEFL scores. Princeton, NJ: Educational Testing Service. Educational Testing Service. (1994f). TOEFL 1994-1995: Test of English as a foreign language I n s t i t u t i o n a l t e s t i n g program oversea e d i t i o n . Princeton, NJ: Educational Testing Service. Educational Testing Service. (1992a). Guidelines for the use of TOEFL scores. Princeton NJ: Educational Testing Service. 85 Educational Testing Service. (1992b). TOEFL t e s t and score manual (1992-93). Princeton, NJ: Educational Testing Service. Educational Testing Service. (1991). Guidelines for TOEFL i n s t i t u t i o n a l v a l i d i t y studies. Princeton, NJ: Educational Testing Service. Educational Testing Service. (1989). The uses of standardized tests i n American education. Princeton, NJ: Educational Testing Service. Fischer, K. W. & Lazerson, A. (1984). Human development. New York: W. H. Freeman & Company. Fletcher, J . & Stern, R. (1986). Language s k i l l s and adaptation: A study of foreign students i n a Canadian un i v e r s i t y . Curriculum Inquiry. 19, 293-308. Gradamn, H. L. & Hanania, E. (1991). Language learning background factors and ESL proficiency. Modern Language Journal. 75, 39-51. Graham, J . G. (1987). English language pro f i c i e n c y and the p r e d i c t i o n of academic success. TESOL Quarterly. 21. 505-521. Gue, L. R. & Holdaway, E. A. (1973). English p r o f i c i e n c y tests as predictors of success i n graduate studies i n education. Language Learning, 23, 89-103. Hackett, G. et a l . (1992). Gender, e t h n i c i t y , and s o c i a l cognitive factors predicting the academic achievement of students i n engineering. East Lansing, MI: National 86 Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. EJ454109) Hale, G. A., et a l . (1983). E f f e c t s of t e s t disclosure on performance on the Test of English as a Foreign Language. Language Learning. 33. 449-464. Hale, G.A. et a l . (1984). A Comprehensive TOEFL Bibliography, 1963-82. Modern Language Journal, 6 8 , 45-51. Hale, G. A. (1988). Student major f i e l d and text content: i n t e r a c t i v e e f f e c t s on reading comprehension i n the Test of English as a Foreign Language. Language Testing, 5, 49-61. Hale, G. A., S t a n f i e l d , C., & Duran, R. (1984). Summaries of Studies Involving the Test of English as a Foreign Language 1963-1982. Princeton, NJ: Educational Testing Service. Ho, D. Y. F. & Spinks, J . A. (1985). Multivariate pred i c t i o n of academic performance by Hong Kong University students. Contemporary Educational Psychology, 19, 249-259. Hosley, D. Meredith, K . (1979). Inter- and i n t r a - t e s t correlates of the TOEFL. TESOL Quarterly. 13, 209-217. Hu, S. P. (1991). English proficiency and academic performance of international graduate students. Dissertation Abstract International, 52, 1626-A. Hughes, A. (1989). Testing for language teachers. New York: Cambridge University Press. 87 Husen T. & Postlethwaite, N. (Eds.)(1992). The international encyclopedia of educational (Vols. 1-12). New York: Pergamon. Kishor, N. (1994). EPSE596 course notes. Unpublished manuscript, University of B r i t i s h Columbia, Vancouver, B r i t i s h Columbia, Canada. Kwang, K. & Dizney, H. F. (1970). Predictive v a l i d i t y of the TOEFL for Chinese graduate students at an American un i v e r s i t y . Educational and Psychological Measurement. 30. 475-477. Light, R. L., & Wan, T. (1991). Soviet students at U. S. Colleges: s o c i a l perceptions, language proficiency, and academic achievement. TESOL Quarterly. 25. 179-185. Light, R. L., Xu, M., & Mossop, J . (1987). English Proficiency and Academic Performance of International Students. TESOL Quarterly. 21. 251-61. Johns A. M. (1981). Necessary English: a faculty survey. TESOL Quarterly. 15. 51-57. Maccoby. E. & J a c k l i n C. (1974). The psychology of sex differences. Stanford, CA: Stanford University Press. Mestre, J . P. (1981). Predicting academic achievement among b i l i n g u a l Hispanic college technical students, Educational and Psychological Measurement, 41, 1266-1264. Mohan, B. (1986). Language and content. Reading, Mass: Addison-Wesley. gg Morgen, B. S. (1990). A comparative study of the use of standardized English language pro f i c i e n c y tests by U. S. graduate schools. College and University. 65. 295-307. Mullein, K. A. 1978). Direct evaluation of second language profi c i e n c y : the e f f e c t of rater and scale i n o r a l interviews. Language Learning. 28. 301-308. Norusis, M. (1994). SPSS-Window: introductory S t a t i s t i c s Guide (for SPSS-window release 6.0). Chicago, IL: SPSS Inc. Norusis, M. (1988). SPSS-X Introductory S t a t i s t i c s Guide (for SPSS-X release 3). Chicago, IL: SPSS Inc. Norusis, M. (1985). SPSS-X Advanced S t a t i s t i c s Guide. Chicago, IL: SPSS Inc. Nunan, D. (1992). Research methods i n language learning. New York: Cambridge University Press. Oltman, P. K., Strieker, L. J.(1988). How native language and l e v e l of English proficiency a f f e c t the structure of the Test of English as a Foreign Language (TOEFL). East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED296592) Ostler, S. (1980). A survey of academic needs for advanced ESL. TESOL Quarterly. XIV. 489-501. Othuon, L. (1993). A study of the p r e d i c t i v e v a l i d i t y of the Kenya c e r t i f i c a t e of primary education examination: Application of Hie r a r c h i c a l Linear Models. Unpublished 89 master's thesis, University of B r i t i s h Columbia, Vancouver, B r i t i s h Columbia, Canada. Pack, A. C. A (1972). Comparison between TOEFL and Michigan Test scores and student success i n (1) freshman and 2) completing a college program. TESL Reporter, 5, 1-7 & 9. Parkerson, J . A., Lomax, R. G., S c h i l l e r , D. & Walberg, H. J . (1984). Exploring causal models of educational achievement. Journal of Educational Psychology, 76, 63 8-64 6. Patkowski, M. S. (1991). Basic s k i l l s t e sts and academic success of ESL college students. TESOL Quarterly, 25. 735-738. Pedhazur, E. (1982). Multiple regression i n behavioral research. New York: CBS College Publishing. Pedhazur, E & Schmelkin. (1991). Measurement, design, and analysis: Integrated approach. H i l l s d a l e , NJ: LEA, Publishers. Peirce, B. N. (1992). Demystifying the TOEFL Reading Test. TESOL Quarterly, 26, 665-691. Perkins, K. (1988). Measuring ESL Readers' A b i l i t y to Apply Reasoning i n Reading: A V a l i d i t y Study of the TOEFL Reading Comprehension Subtest. Journal of Research i n Reading. 11. 36-49. Perkins, K. & Parish, C. (1988). What's wrong with reading comprehension tests? East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED297305) 90 Perry, W. S. (1989). The re l a t i o n s h i p of the Test of English as a Foreign language (TOEFL) and other c r i t i c a l v a riables to the academic performance of international graduate students. Dissertation Abstract International. 50. 422A. Powers, D. E. (1985). A Survey of academic demands related to l i s t e n i n g s k i l l s . Test of English as a Foreign Language Research Reports Number 20. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED3 04011) Raimes, A. (1990). The TOEFL t e s t of written English: cause for concern. TESOL Quarterly. 24. 227-243. Reed, D. J . (1992). The rela t i o n s h i p between c r i t e r i o n -based l e v e l s of oral proficiency and norm-referenced scores of general pro f i c i e n c y i n English as a second language. System. 20. 329-345. Ritsumeikan University. (1992). Ritsumeikan University (1992-1993). Kyoto, Japan: Ritsumeikan University. Rivera, C. (Ed.) (1984). Language prof i c i e n c y and academic achievement. Clevedon, England: M u l t i l i n g u a l Matters Ltd. Rosenthal, R. & Jacobson, L. (1968). Pygmalion i n the classroom; teacher expectation and pupil's i n t e l l e c t u a l development. New York: Rinehart and Winston. Santos, T. (1988). Professors' reactions to the academic writing of nonnative-speaking students. TESOL Quarterly. 22. 69-90. 91 Sharon, A. T. (1972). English proficiency, verbal aptitude, and foreign student success i n American graduate schools. Educational & Psychological Measurement, 32, 425-431. Snow, C. E., Barnes W. S., Chandler J . , Goodman I. F., & Hemphill L. (1991). U n f u l f i l l e d expectations: Home and s o c i a l influences on l i t e r a c y . Cambridge, MA: Harvard University Press. Spolsky, B. (1990). The Prehistory of TOEFL. Language Testing, 7, 98-118. Sposky, B. (1979). Some major t e s t s . Arlington, V i r g i n i a : The Center for Applied L i n g u i s t i c s . Sposky, B. (1978). Approaches to Language Testing. Arlington, V i r g i n i a : The Center for Applied L i n g u i s t i c s . S p r i n t h a l l , N. & S p r i n t h a l l , R. (1994). Educational psychology: a developmental approach (6th ed.). New York: McGraw-Hill. Swinton, S. S. & Powers, D. E. (1980). Factor analysis of the Test of English as a Foreign Language for several language groups. TOEFL Research Reports, 6. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED218921) Tabachnick, B. G. & F i d e l l , L. S. (1989). Using multivariate s t a t i s t i c s (2nd ed.). HarperCollinsPublishers, Inc. 92 Traynor, R. (1985). The TOEFL: An appraisal. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. EJ315491) University of B r i t i s h Columbia. (1993). The University of B r i t i s h Columbia Calendar 1993-94. Vancouver, Canada: University of B r i t i s h Columbia. Ushasree, S. (1990). Academic adjustment and s c h o l a s t i c achievement among the s o c i a l l y disadvantaged. T i r u p a t i , India: S r i Venkateswara University. Verhoeven, L. & Jong, J . (eds.) (1992). The construct of language proficiency: applications of psychological models to language assessment. Philadelphia: John Benjamins Publisher. Vigotsky, L. (1986). Thought and language (Kozulin A., Trans.). Cambridge, MA: MIT Press. Vinke, A. A. & Jochems, W. M. G. (1993). English p r o f i c i e n c y and academic success i n international postgraduate education. Higher Education, 26, 275-285. Walberg, H. J . & Haertel, G. D. (eds.) (1990). The international encyclopedia of educational evaluation. New York: Pergamon Press. Wan, T., Chapman. D. W., & Biggs, D. A. (1992). Academic achievement stress of international students attending U. S. u n i v e r s i t i e s . Research i n Higher Education, 33. 607-623. 9 3 Willingham, W. M. (1990). Predicting college grades: An analysis of i n s t i t u t i o n a l trends over two decades. Princeton, NJ: Educational Testing Service. Willingham, W. M. with Young, J . W., & Morris, M. M. (1985). Success i n college: the r o l e of personal q u a l i t i e s and academic a b i l i t y . New York: College Entrance Board. Wilson, K. M. (1982). A comparative analysis of TOEFL examinee c h a r a c t e r i s t i c s . 1977-1979. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED237512) Wilson, K. M. (1986). The re l a t i o n s h i p of GRE General Test scores to f i r s t - y e a r grades for foreign graduate students: Report of a Cooperative Study. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED281862) Wilson, K. M. (1987). Patterns of t e s t taking and score change for examinees who repeat the Test of English as a Foreign Language. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED283831) Wimberley, D. W., McCloud, D. & F l i n n , W. (1992). Predicting Success of Indonesian Graduate Students i n the United States. Comparative Education Review, 36. 487-508. Wolf, F. (1986). Meta-analysis: Quantitative methods for research synthesis. Beverly H i l l s , CA: Sage Publications, Inc. 94 Yan, Z. (1994). Range r e s t r i c t i o n and indicator s e l e c t i o n : a preliminary review of six prediction studies on re l a t i o n s h i p between TOEFL scores and GRE. Unpublished manuscript, University of B r i t i s h Columbia, Vancouver, B r i t i s h Columbia, Canada. Yan, Z. (1994). A meta-analysis on studies of TOEFL scores' p r e d i c t i v e v a l i d i t y on f i r s t year's GPA (1964-1994). Unpublished manuscript, University of B r i t i s h Columbia, Vancouver, B r i t i s h Columbia, Canada. Young, J . W. (1993). Grade adjustment methods. Review of Education Research. 63. 151-165. Yule, G., Hoffman, P. (1990). Predicting Success for International Teaching Assistant i n a U. S. University. TESOL Quarterly. 24. 227-43. Zeidner, M. (1987). A comparison of ethnic, sex, and age bias i n the pr e d i c t i v e v a l i d i t y of English language aptitude t e s t s : some I s r a e l i data. Language Testing. 4. 55-71. Z i r p o l i , T. J . , Halllahan, D. P., & Kneedler, R. D. (1988). The Indonesian project: correlates of student performance i n a special education teachers t r a i n i n g program. International Journal of Special Education. 3. 73-79. Zhang S., Dunkel. P. & Zubovic, Y. (1992). Regression models i n an ESL context: issues i n construction and int e r p r e t a t i o n . TESOL Quarterly. 26. 191-196. 95 Appendix I The data f i l e ,qpa ened206a ened379 educ395a t o e f l s e e l 76.00 60 .00 483.00 49.67 50 .00 43 .00 56.00 483.00 49.00 45.00 52 .00 65.00 41.00 50.00 477.00 47.00 58 .33 57 .00 58.00 60.00 543.00 54.00 58.33 66.00 42 .00 67.00 497.00 59 .33 75.00 45.00 58.00 497.00 59.33 77 .00 31.00 70.00 523.00 59 .67 50.00 57 .00 72 .00 523.00 49.00 61.33 70 .00 44.00 70 .00 530.00 10 61. 67 60 .00 55.00 70.00 483.00 11 62 .00 74 . 00 52 .00 60.00 510.00 12 62 .33 72 .00 53.0& 62.00 533.00 13 62 .33 74.00 47.00 66.00 467.00 14 62 .67 70.00 60.00 58.00 513.00 15 16 62.67 75.00 43 .00 70.00 503.00 63 . 00 72 .00 51.00 66.00 500.00 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 63 .33 63 .33 64 .00 64 . 67 65 . 00 65 . 00 65.00 65 .33 67 .33 67 .33 67.67 67 . 67 68 .00 68 . 00 68 . 00 68.67 69 .00 69.33 70 . 00 70 .00 70.33 70.33 70.33 70.67 71.33 71.33 71.67 72 .00 72 .33 72 .33 72 .67 72 . 67 73 .00 73 .00 .74.33 74.33 74.33 74 .33 74.33 74 . 67 74 .67 73 .00 57 .00 60 .00 517.00 75.00 75.00 77 . 00 69 .00 72 .00 75.00 67 .00 73 .00 74 .00 74 .00 83 .00 68 .00 74.00 76 .00 76.00 75.00 74.00 76 .00 82 .00 54 .00 79 .00 80 .00 75.00 82 .00 84.00 65.00 78 .00 75.00 79 .00 70 .00 76 .00 74.00 84 .00 74 .00 79 .00 79 .00 81.00 83 .00 73 .00 77 .00 53 .00 50.00 57 .00 63 .00 65.00 58.00 65.00 63 .00 60.00 61.00 52.00 65.00 70.00 58.00 70.00 61.00 66.00 74.00 60.00 83.00 60.00 68.00 61.00 58.00 60.00 78.00 64.00 70.00 68.00 74.00 70.00 71.00 65.00 75.00 67.00 73.00 76.00 67.00 81.00 77.00 62.00 67 .00 60.00 63 .00 58.00 62.00 64.00 66.00 68.00 68.00 68.00 71.00 60.00 70.00 60.00 71.00 68.00 60.00 68.00 74.00 72.00 63.00 76.00 74.00 70.00 72.00 74.00 72.00 70.00 74.00 72.00 74.00 70.00 74.00 77 .00 71.00 66.00 73.00 70.00 70.00 480.00 493.00 530.00 497.00 517.00 483.00 513.00 523.00 490.00 517.00 507.00 500.00 520.00 517.00 490.00 527.00 477.00 530.00 517.00 540.00 487.00 537.00 570.00 550.00 490.00 483.00 513.00 493.00 480.00 550.00 500.00 520.00 520.00 540.00 563.00 470.00 513.00 550.00 497.00 493.00 96 qpa ened206a ened379 educ3 9 5a toef 1 s e e l 58 — 74.67 77 .00 77.00 70.00 557.00 54.00 59 74.67 78.00 76.00 70.00 543.00 49 .00 60 74.67 82.00 70.00 72.00 473.00 46.00 61 74.67 84.00 67.00 73 .00 553.00 55.00 62 75.00 77.00 78.00 70.00 527.00 51.00 63 75.33 79.00 75.00 72.00 470.00 45.00 64 75.33 82.00 73.00 71.00 477.00 44.00 65 75.33 84.00 74.00 68.00 507.00 47 .00 66 75.67 77.00 78.00 72 .00 540.00 56.00 67 15.61 79.00 71.00 77 .00 553.00 55.00 68 75.67 86.00 77 .00 64.00 503.00 52.00 69 75.67 89.00 71.00 67.00 503.00 48.00 70 76.00 79.00 76.00 73 .00 510.00 48.00 71 76.00 82 .00 72.00 74.00 550.00 56.00 72 76.33 76.00 81.00 72.00 500.00 41.00 73 76.67 75.00 83.00 72 .00 497.00 47.00 74 76.67 78.00 80.00 72 .00 540.00 56.00 75 77 .00 76.00 77.00 78.00 487.00 39.00 76 77.00 82 .00 77 .00 72 .00 507.00 49.00 77 77.67 74.00 85.00 74.00 540.00 51.00 78 77.67 81.00 78.00 74.00 503.00 50.00 79 78.67 79.00 86.00 71.00 543.00 52.00 80 79.00 83 .00 77 .00 77 .00 520.00 48.00 81 79.00 90.00 77.00 70.00 540.00 50.00 . 82 79.33 82.00 78.00 78.00 533.00 52.00 83 19.61 88.00 78.00 73.00 520.00 47 .00 84 80.00 81. 00 81.00 78.00 543.00 50.00 85 80.00 81.00 83.00 76.00 513.00 52 .00 86 80.00 85.00 78.00 77.00 533.00 51.00 87 82.33 75.00 94.00 78.00 480.00 48.00 88 82 .33 86.00 80.00 81.00 530.00 50.00 89 82 .33 87 .00 83.00 77 . 00 580.00 61.00 90 82 .67 83.00 85.00 80.00 573.00 61.00 91 82 .67 85.00 83.00 80.00 573.00 56.00 92 82.67 88.00 80.00 80 . 00 513.00 52.00 93 82.67 89.00 81.00 78.00 497.00 47.00 94 83.00 82 .00 80.00 87.00 550.00 59.00 95 83.33 86.00 84.00 80.00 530.00 54.00 96 83.33 90.00 83.00 77 .00 560.00 53.00 97 87.00 90.00 87.00 84.00 543.00 56.00 97 sec2 sec3 write speak qender toeflpre 1 48.00 48.00 2.00 .90 1.00 483.00 2 50 .00 50 .00 1.00 .90 1.00 483.00 3 48 .00 48 .00 2.00 .90 1.00 477.00 4 55.00 54.00 3.00 1.90 1.00 543.00 5 47 .00 53 .00 1.00 1.00 1.00 497.00 6 51.00 • 51.00 3.00 1.90 2.00 497.00 7 53 .00 52 .00 1.00 1.90 1.00 507.00 8 54.00 53 .00 2.00 .90 1.00 523.00 9 57 .00 48.00 3 .00 1.90 1.00 530.00 10 48 .00 54 .00 3.00 1.00 483.00 11 52 .00 48.00 2.00 1.90 1.00 510.00 12 58 .00 55.00 4.00 1.90 2.00 533.00 13 50 .00 47 .00 2 .00 .90 1.00 467.00 14 52.00 53 .00 3.00 .90 2.00 513.00 15 52 .00 49 .00 2 .00 2.00 1.00 503 .00 16 52 .00 48.00 3 .00 1.90 1.00 500.00 17 48.00 54 .00 3.00 2.00 . 1.00 517.00 18 49 .00 49 .00 2.00 2.00 2.00 480.00 19 52 .00 47 .00 3.00 1.00 1.00 460.00 20 54.00 54.00 2.00 2.00 1.00 530.00 21 53 .00 49 .00 2.00 1.00 2.00 497.00 22 55.00 53 .00 2.00 1.90 1.00 517.00 23 51. 00 48 .00 3 .00 .00 2.00 483.00 24 55.00 54 .00 2.00 1.00 1.00 513.00 25 58 .00 " 51.00 3 .00 1.00 1.00 523.00 26 51.00 51.00 2.00 .90 1.00 490.00 27 49 .00 53 .00. 1.00 1.00 2.00 517.00 28 52 .00 53 .00 3.00 .90 1.00 507.00 29 51.00 49 .00 2.00 .00 2.00 500.00 30 54.00 54.00 2.00 .00 1.00 520.00 31 50.00 53.00 4.00 2.00 1.00 517.00 32 55.00 50.00 4.00 1.00 1.00 490.00 33 61.00 52.00 2.00 .90 1.00 487.00 34 47 .00 47 .00 2.00 .90 1.00 477.00 •35 52 .00 54.00 3.00 2.00 2.00 530.00 36 53 .00 53 .00 2.00 1.00 1.00 517.00 37 55,00 53 .00 4.00 1.90 2.00 540.00 38 50.00 49.00 2.00 .90 1.00 487.00 39 56 .00 55.00 3.00 1.90 1.00 523.00 40 68 .00 53 .00 3.00 1.00 2.00 513.00 41 56 .00 51.00 3.00 2.00 2.00 550.00 42 52 .00 48.00 2.00 2.00 2.00 490.00 43 46.00 52 .00 2.00 1.00 1.00 483.00 44 54 .00 51.00 3.00 1.90 2.00 483.00 45 • 54.00 48.00 2.00 .90 2.00 493.00 46 51.00 48.00 2.00 1.90 1.00 480.00 47 61.00 50.00 4.00 3.00 2.00 547.00 48 53 .00 51.00 3.00 1.90 1.00 487.00 49 58.00 53 .00 4.00 1.90 1.00 520.00 50 56.00 50 .00 5.00 2.00 2.00 520.00 51 54 .00 56.00 2.00 1.90 1.00 520.00 52 58.00 60.00 2.00 . 1.00 547.00 53 52 .00 45.00 2.00 1.00 2.00 470.00 54 54.00 49.00 3.00 1.90 2.00 493.00 55 54 .00 59 .00 3.0C 1.00 1.00 550.00 56 51.00 50 .OC 3.0C l.OO 2 .00 497.00 51 51. 0C 50.00 3.0C ) 1.0C 2.00 1 493.00 98 sec2 sec3 w r i t e speak qender t o e f I p r e 58 57 . 00 56.00 2.00 1.90 1.00 557.00 59 60.00 54.00 2.00 1.90 1.00 517.00 60 52.00 44.00 4.00 2.00 2 .00 473.00 61 55.00 56.00 3 .00 1.90 2.00 553.00 62 54.00 53.00 2.00 1.90 2.00 527.00 63 48.00 48.00 4.00 1.00 2.00 470.00 64 52.00 47.00 3 .00 1.90 1.00 477.00 65 55.00 50.00 3 .00 .00 2.00 507.00 66 53.00 53.00 3 .00 2.00 2.00 540.00 67 56.00 55.00 2 .00 1.00 2 .00 553.00 68 51.00 48.00 2.00 1.90 1.00 503.00 69 52 . 00 51.00 2.00 2.00 2.00 503.00 70 53 .00 52.00 3.00 .00 2.00 510.00 71 56.00 53 .00 4.00 1.90 2.00 550.00 72 53 .00 56.00 2.00 1.00 2.00 500.00 73 52.00 50.00 3.00 1.00 2.00 497.00 74 58.00 48.00 3 .00 .90 2.00 520.00 75 53.00 54.00 4.00 2.90 1.00 487.00 76 52 . 00 51.00 2.00 1.00 2.00 507.00 77 61.00 50.00 2 .00 1.90 2.00 540.00 78 49.00 52 .00 3 .00 1.90 2.00 503.00 79 55.00 56.00 4.00 2.00 2 .00 543.00 80 56.00 52.00 3 .00 .90 2.00 520.00 81 57 .00 55.00 4.00 1.90 1.00 520.00 82 56.00 52 .00 3 .00 2.00 2.00 493.00 83 56.00 53 .00 2 .00 1.00 2 .00 513.00 84 59.00 54.00 2 .00 1.00 1.00 543.00 85 52 .00 50.00 3 .00 1.00 2.00 513.00 86 58.00 51.00 3.00 1.90 1.00 533.00 87 49.00 47 .00 3 .00 .90 2.00 480.00 88 56.00 53.00 2.00 .90 1.00 530.00 89 57.00 56.00 3 .00 2.00 2.00 577.00 90 58.00 53 .00 4.00 2 .00 2 .00 570.00 91 58.00 58.00 3.00 1.00 2.00 573.00 92 50.00 52.00 3.00 2.00 2.00 513.00 93 52.00 50.00 2.00 .00 2.00 497.00 94 54.00 52 .00 3.00 2 .00 2 .00 550.00 95 55.00 50.00 4.00 2 .00 2.00 530.00 96 61.00 54.00 4.00 2.00 2.00 550.00 97 53 .00 54.00 3.00 3 .00 2.00 543.00 Appendix II The l i s t of standardized residuals and leverage values qpa z r e _ t o t ! l e v t o t 1 z r e sec l e v sec 1 78.67 .01185 1 .03357 .00005 .05956 2 62.33 -.26058 .04696 -.27775 .05135 3 75.33 .37326 .08908 .45587 .09910 4 75.67 -.16919 .02028 -.02946 .03603 5 67.67 -.18236 .03353 -.16574 .04249 6 74.67 .76297 .04023 .53343 .08277 7 82.67 1.65650 .06361 1.63646 .06392 8 52.00 -2.02214 .03501 -1.87327 .05101 9 7 6.67 -.05763 .03317 . 00203 . 11375 10 75.00 .14203 .03923 .09495 .04372 11 74.33 .32776 .06275 .39459 .11503 12 65.00 -.43412 .02645 -.55406 .04046 13 82.67 .26183 .06595 .41617 .09953 14 65.00 -1.09487 .02780 -1.16161 .03486 15 83.33 .91557 .03011 .98745 .04238 16 79.67 .93115 .02610 .74468 .05455 17 71.33 -.98196 .02740 -.86176 .05509 18 62.67 -.62034 .03393 -.56548 .04415 19 59.33 -2 .20797 .02870 -2.22204 . 03879 20 61.33 -1.43484 .02255 -1.36927 .09307 21 72.67 .73991 1 .02744 .68385 .03040 22 49.67 -2.21252 I .05536 -2 .22396 .05557 23 72.33 1.19734 | .05094 1.15341 .05441 24 65.33 -.36852 | .02076 -.52355 .04633 25 80.00 1.47875 .02394 1.40383 .04503 26 83 .33 .53302 .04774 .38734 .06668 27 76.00 .18139 .07434 .18329 .07680 28 76.67 .49493 .02110 .47693 .02220 29 80.00 ' .81801 .01580 .92246 .02695 30 70.33 -1.27192 .03182 -1.20212 .03535 31 69.00 .02855 .03199 -.27853 .12271 32 67.67 -.67309 .07477 -.52036 .10099 33 79.00 .97624 .06004 .92182 .06622 34 77.67 .39796 .04508 .15958 .13103 35 58.33 -2.07501 .03101 -1.98387 .03685 36 75.67 1.43117 .02962 1.56725 .06390 37 74.33 -.03998 .01664 -.04228 .02403 38 76.33 .65906 .02644 .38595 .16597 39 68.00 -.70106 .06366 -.60169 .07461 40 82.67 .47460 .08217 .47333 .09221 41 74.33 .72534 .05533 .61600 .08467 42 75.67 .55774 .05063 .48508 .05638 43 87.00 1.62708 .07591 1.71437 .09016 44 74.67 .22967 .02394 .23312 .02561 45 71.67 1.01556 .02852 1.17619 .07769 46 74.67 .58444 .05824 .57255 .05847 47 82.67 1.28205 .01993 1.39658 .04128 48 62.00 -.81926 .02711 -.67128 .07029 49 73.00 .28247 .05445 .08020 .08797 50 67.33 -.43488 .03378 -.54908 .05677 51 74.33 .74850 .03733 .76494 .04961 52 79.00 .56638 .01995 .44869 .03077 53 69.33 .71885 .03501 .94238 .07997 54 63.33 -1.09931 .07792 -1.11695 .08167 55 59.33 -1.15162 .07445 -1.11406 .07686 56 82.33 .37496 .07149 .51442 .08633 57 64.00 -.57943 .03394 -.47375 .06678 100 opa zre t o t lev t o t zre sec le v _ s e c 58 63 .00 -.78872 .02744 -.68818 .04791 59 64.67 -.64837 .03318 -.64558 .03547 60 72 .33 .11155 .03066 -.01559 .05153 61 83 .00 .86294 .02740 1.05386 .06134 62 71.33 .03825 .06359 -.03909 .07055 63 65.00 -1.21379 .07931 -1.18508 .08110 64 68.67 -.05833 .08838 -.20536 .10605 65 72.67 -.98388 .07231 -1.11125 .12319 66 68.00 -.07843 .08129 -.08717 .08378 67 72.00 -.40896 .01664 -.47273 .01968 68 68.00 -.46571 .05530 -.24876 .10158 69 70.33 .74946 .02684 .82353 .03234 70 67 .33 .23680 .02513 .21706 .02765 71 77.67 .61466 .02305 .71146 .05036 72 80.00 1.56824 .04706 1.42202 .06560 73 77 .00 1.37816 .12488 1.11990 .24115 74 75.33 .11421 .07353 .04687 .08050 75 77 .00 .67523 .02460 .67218 .02519 76 70.00 -.93777 .01693 -.85039 .03661 77 62.67 -1.92721 .01889 -1.90177 .03147 78 73.00 -.84607 .08662 -.86892 .08959 79 59.67 -1.39631 .02875 -1.37712 .02915 80 76.00 -.50334 .03817 -.41273 .04435 81 79.33 .49989 .01754 .44542 .02009 82 58.33 -1.01632 .04976 -.85125 .08982 83 74.67 .27193 .09043 .24740 .11580 84 74.67 .17866 .02110 .21727 .02329 85 75.67 -.12219 .05862 -.10667 .05977 86 70.00 .31844 .02230 .33727 .02368 87 82.33 1.60356 .03890 •1.70789 .05079 88 82.33 2 .09875 .03482 2.04925 .03743 89 70.33 -.10089 .02631 -.14970 .03334 90 74.67 -.49733 .02887 -.45606 .04251 91 . 63.33 -.94856 .02250 -.68273 .09999 92 62.33 -2.44774 .02963 -2.65863 .08243 93 75.33 1.45488 .05424 1.39603 .06382
Thesis/Dissertation
eng
Language and Literacy Education
Vancouver : University of British Columbia Library
For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use.
Graduate
Predictive validity of TOEFL scores on first term’s GPA as the criterion for international exchange students
Text