Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Predictive validity of TOEFL scores on first term’s GPA as the criterion for international exchange students Yan, Zheng 1995

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_1995-0433.pdf [ 4.16MB ]
Metadata
JSON: 831-1.0078086.json
JSON-LD: 831-1.0078086-ld.json
RDF/XML (Pretty): 831-1.0078086-rdf.xml
RDF/JSON: 831-1.0078086-rdf.json
Turtle: 831-1.0078086-turtle.txt
N-Triples: 831-1.0078086-rdf-ntriples.txt
Original Record: 831-1.0078086-source.json
Full Text
831-1.0078086-fulltext.txt
Citation
831-1.0078086.ris

Full Text

PREDICTIVE AS  V A L I D I T Y . OF T O E F L  THE CRITERION  FOR  SCORES  ON  FIRST  TERM'S  I N T E R N A T I O N A L EXCHANGE  STUDENTS  by  ZHENG M.Ed.., N o r t h e a s t  A  THESIS THE  YAN  Normal  University,  SUBMITTED IN PARTIAL REQUIREMENTS MASTER  OF  1986  FULFILLMENT  FOR T H E D E G R E E  OF  ARTS  in THE  F A C U L T Y OF GRADUATE  Department  We  accept to  THE  o f Language  this  thesis  the required  UNIVERSITY  OF  March, ©  Zheng  STUDIES Education  as  conforming  standard  BRITISH 1995  Yan, 1995  COLUMBIA  GPA  OF  In p r e s e n t i n g  t h i s t h e s i s i n p a r t i a l f u l f i l l m e n t of the  requirements f o r an advanced degree a t the U n i v e r s i t y o f B r i t i s h Columbia, I agree t h a t the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r reference  and study. I f u r t h e r agree  t h a t p e r m i s s i o n f o r e x t e n s i v e copying o f t h i s t h e s i s f o r s c h o l a r l y purposes may be granted by t h e Head o f my Department o r by h i s o r her r e p r e s e n t a t i v e s .  It is  understood t h a t copying o r p u b l i c a t i o n o f t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be allowed without permission.  Department o f  L  The U n i v e r s i t y o f B r i t i s h Columbia Vancouver, Canada Date  written  Abstract The T e s t o f E n g l i s h as a F o r e i g n Language (TOEFL) has  been used i n making admission d e c i s i o n s  f o r over 30  y e a r s ; however, t h e p r e d i c t i v e v a l i d i t y o f t h e t e s t has been u n c e r t a i n .  The p r e s e n t study was intended t o  i n v e s t i g a t e t h e p r e d i c t i v e v a l i d i t y o f TOEFL s c o r e s on f i r s t term's grade p o i n t  average (GPA).  Participants  were 97 second-year u n i v e r s i t y students, 46 male and 52 female, i n an i n t e r n a t i o n a l academic exchange program. Most majored i n Humanities and S o c i a l S c i e n c e s . The predictor variables  i n t h e study i n c l u d e d  scores,  TOEFL s e c t i o n I s c o r e s ,  scores,  TOEFL s e c t i o n I I I s c o r e s ,  interview  scores,  TOEFL t o t a l  TOEFL s e c t i o n I I oral proficiency  w r i t i n g sample s c o r e s ,  and gender.  F i r s t term's GPA was t h e c r i t e r i o n v a r i a b l e . The data were a n a l y z e d by m u l t i p l e  regression  analysis with a  h i e r a r c h i c a l procedure. The r e s u l t s were i n t e r p r e t e d  on  the b a s i s o f Cohen's (1988) c o n v e n t i o n a l d e f i n i t i o n s on the  e f f e c t s i z e of R . 2  The main f i n d i n g s o f the study i n d i c a t e t h a t :  (a) TOEFL  t o t a l s c o r e s have a medium l e v e l o f p r e d i c t i v e v a l i d i t y on GPA  (AR =.142, p_<.001); (b) TOEFL s e c t i o n I s c o r e s have a 2  medium l e v e l o f p r e d i c t i v e v a l i d i t y  (AR=.044, p_<.05); (c)  TOEFL s e c t i o n I I s c o r e s have a medium l e v e l o f p r e d i c t i v e validity  (AR =.112, p_<.001); (d) TOEFL s e c t i o n I I I s c o r e s 2  have a n e g l i g i b l e l e v e l o f p r e d i c t i v e v a l i d i t y  ( A R =.005, 2  p_>.05); (e) O r a l p r o f i c i e n c y negligible level (f) W r i t i n g validity level  (AR  i n t e r v i e w s s c o r e s have a  of p r e d i c t i v e v a l i d i t y  (AR  =.010, p_>.05);  samples s c o r e s have a small l e v e l =.047, p_<.05); And  of p r e d i c t i v e v a l i d i t y  of p r e d i c t i v e  (g) gender has a medium  (AR =.130, p_<.001). 2  T  n  e  f i n d i n g s o f the study thus v a l i d a t e the use o f TOEFL s c o r e s as one o f the requirements f o r admission i n the i n t e r n a t i o n a l exchange program and p r o v i d e new  empirical  evidence f o r i n v e s t i g a t i o n o f the r e l a t i o n s h i p between language p r o f i c i e n c y and academic achievement.  iv TABLE OF CONTENTS Abstract  i i  T a b l e s o f contents  iv  L i s t of tables  vi  L i s t of figures  v i i  Acknowledgments  viii  Chapter One: I n t r o d u c t i o n  1  Research problem  1  Research q u e s t i o n s  2  D e f i n i t i o n o f terms  3  Chapter Two: L i t e r a t u r e Review  6  Background  6  Part  7  I : F a c t o r s i n f l u e n c i n g academic achievement  A conceptual structure  7  A c l a s s i f i c a t i o n scheme  9  Language  f a c t o r s and academic achievement  Non-language Part  f a c t o r s and academic achievement  I I : F a c t o r s a f f e c t i n g TOEFL's p r e d i c t i v e v a l i d i t y  12 16 20  A n a l y t i c a l models  21  Subject v a r i a b l e s  26  Predictor variables  29  Criterion variables  32  Result interpretation  34  Summary Chapter Three: Method  36 38  The program s e t t i n g  38  Participants  40  V  The  predictor variables  41  The  criterion variable  43  A n a l y t i c a l model  44  Operational  45  d e f i n i t i o n s of the p r e d i c t i v e v a l i d i t y  Research Hypotheses  46  Summary  47  Chapter Four: R e s u l t s  48  Treatment o f t h e m i s s i n g  data  48  Descriptive s t a t i s t i c a l analysis  49  Checking f o r v i o l a t i o n  52  Hierarchical regression  o f assumptions analysis  58  Summary  64  Chapter F i v e : D i s c u s s i o n  65  P r e d i c t i v e v a l i d i t y o f TOEFL t o t a l scores  65  P r e d i c t i v e v a l i d i t y o f TOEFL s e c t i o n a l scores  68  P r e d i c t i v e v a l i d i t y o f w r i t i n g scores  70  P r e d i c t i v e v a l i d i t y o f speaking scores  71  P r e d i c t i v e v a l i d i t y o f gender  72  Implications  74  Limitations  76  Directions f o r future research  77  Conclusions  78  Bibliography  80  Appendix I  The data f i l e  Appendix I I  The l i s t o f s t a n d a r d i z e d leverage values  95 r e s i d u a l s and 99  vi  L i s t of Tables T a b l e 3.1 Grade c r i t e r i o n on d i f f e r e n t a s p e c t s i n t h e t h r e e courses  45  T a b l e 3.2 Four l e v e l s o f t h e p r e d i c t i v e v a l i d i t y T a b l e 4.1 Means and standard d e v i a t i o n s  of a l l the  variables  49  T a b l e 4.2 Pearson c o r r e l a t i o n matrix o f t h e v a r i a b l e s T a b l e 4.3 Summary t a b l e o f t h e h i e r a r c h i c a l  Table 4.4 Summary t a b l e of t h e h i e r a r c h i c a l scores  51  analysis  w i t h TOEFL t o t a l s c o r e s  w i t h TOEFL s e c t i o n a l  46  61 analysis 63  vii  L i s t of f i g u r e s Figure  2.1 A t w o - l e v e l  conceptual structure  i n a study o f  TOEFL's p r e d i c t i v e v a l i d i t y Figure  2.2 A f i v e - l e v e l c l a s s i f i c a t i o n scheme o f f a c t o r s i n f l u e n c i n g academic achievement  Figure  7  10  4.1 S c a t t e r p l o t o f the d i s t r i b u t i o n o f t h e residuals  55  Figure  4.2 D i s t r i b u t i o n o f r e s i d u a l s  56  Figure  5.1 S c a t t e r p l o t o f TOEFL t o t a l s c o r e s and GPA  67  Acknowledgments The growth of a flower, no matter how small, has to appreciate the Sun's enlightenment, the Rain's refreshment, and the Mother Earth's nutrition and grounding. Here, I wish to extend my heartfelt thanks to: Dr. Lee Gunderson, my M A program advisor, for giving over three years of tireless guidance; Dr. Richard Berwick and Dr. Stephen Carey, my research committee members, for their invaluable support and advice; Dr. Areigh Reichl for providing consultation on statistics and William McMichael for providing consultation on the UBC/Ritsumeikan program. They both read the whole thesis and gave constructive criticism; Sheri Wenman, Jean Hamilton, and the UBC/Ritsumeikan program's instructors and students for their constant support and cooperation for my thesis project; Dr. Robert Kantor, Director of ETS's TOEFL Program, for providing both professional consultation on TOEFL and a long list of E T S ' free publications; Dr. Xiufeng Liu, Dr. Dean Mellow, Bingzheng Liu, and particularly Dr. Nand Kishor for their inspiration in the development of my thesis project; Victoria Dixon, Lynda Hayward, and Elizabeth Crittenden for their laborious proof-reading on the different chapters of the thesis; Dr. William Mackey, Dr. David Robitaille, Dr. Karen Armstrong, Dr. Robert Conry, Dr. Bernard Mohan, Dr. Marshall Arlin, Dr. Jon Shapiro, Dr. Marion Crowhurst, and Dr. Judith Johnston for their inestimable support, teaching, and/or encouragement; My friends, White Harvey, Cathy Galloaher, Cuhui Zhao, Elizabeth Smith, Gary and Mary Gates, Jingzi Wang and Dr. Yuan Gao, Prof. Kunwei Wang, Dr. Lianqin Wang, Jim and Katherine Yuen, Zhong Liu, Dan Zhang and Yaoyao, Dr. Glen Dixon and Victoria Dixon, Roberta Buck, and Dr. Leigh Faulkner for their immeasurable support; My parents and brother for their deep understanding of and exhaustive financial support for my study at U B C ; And my wife, Jingkai Zhang, for sharing my stress and happiness.  1 Chapter  One  Introduction T h i s chapter p r e s e n t s the r e s e a r c h problem under The  study.  s p e c i f i c r e s e a r c h q u e s t i o n s and d e t a i l e d d e f i n i t i o n s of  terms are a l s o g i v e n . Research Problem The T e s t of E n g l i s h as a F o r e i g n Language (TOEFL) i s the most w i d e l y used t e s t of E n g l i s h as a f o r e i g n language i n the world. 1964  I t was  f i r s t administered  ( O i l e r & Spolsky,  i n 34 c o u n t r i e s i n  1979). At present, as r e p o r t e d by  the  E d u c a t i o n a l T e s t i n g S e r v i c e (ETS), TOEFL i s g i v e n on a monthly b a s i s at over 1,200  t e s t c e n t e r s i n 175  r e g i o n s around the world, with a p o p u l a t i o n of 700,000 examinees every year The primary  (ETS,  countries or approximately  1994a, 1994b).  f u n c t i o n of TOEFL, as s t a t e d i n the  latest  TOEFL T e s t and Score Manual, i s "to measure the E n g l i s h p r o f i c i e n c y of i n t e r n a t i o n a l students wishing t o study a t c o l l e g e s and u n i v e r s i t i e s i n the U n i t e d S t a t e s and (ETS,  1992,  p. 6). Although  Canada"  c o n s i d e r a b l e e v o l u t i o n of TOEFL  has o c c u r r e d d u r i n g i t s 30 years of development, the  primary  f u n c t i o n has never changed. TOEFL s c o r e s are c u r r e n t l y r e q u i r e d f o r admission  i n t o undergraduate or graduate  programs by more than 2,500 c o l l e g e s and u n i v e r s i t i e s i n the USA  and Canada (ETS,  1994c).  A g r e a t d e a l of r e s e a r c h has been conducted t o v a l i d a t e the use of TOEFL (Hale, S t a n s f i e l d , & Duran, 1984; 1994d). A l a r g e p r o p o r t i o n of the r e s e a r c h has  ETS,  explored  2  TOEFL*s p r e d i c t i v e v a l i d i t y w i t h grade p o i n t as the c r i t e r i o n .  Since English p r o f i c i e n c y  a c h i e v e academic success i n an E n g l i s h should be  academic achievement, and  i s necessary t o  English consequently a  p o s i t i v e r e l a t i o n s h i p between TOEFL s c o r e s as an of E n g l i s h p r o f i c i e n c y and  GPA  as an  (GPA)  environment, t h e r e  a p o s i t i v e r e l a t i o n s h i p between  p r o f i c i e n c y and  average  indicator  i n d i c a t o r of academic  achievement. A c c o r d i n g l y , TOEFL s c o r e s should have s t r o n g p r e d i c t i v e v a l i d i t y i n p r e d i c t i n g GPA;  however, TOEFL  p r e d i c t i o n s t u d i e s have c o n s i s t e n t l y r e v e a l e d divergent r e s u l t s  (Graham, 1987;  Hale, S t a n s f i e l d , & Duran,  1984) . Although r e s e a r c h e r s g e n e r a l l y language p r o f i c i e n c y  widely  agree t h a t  English  i s important f o r academic achievement,  they have not yet been a b l e t o reach a consensus on TOEFL*s predictive The  validity.  problem i s , t h e r e f o r e ,  t h a t TOEFL has  worldwide by thousands of i n s t i t u t i o n s decisions is s t i l l  f o r 30 y e a r s , but  been used  t o make admission  the p r e d i c t i v e v a l i d i t y of TOEFL  an unsolved q u e s t i o n f o r p r o f e s s i o n a l s  i n language  education. Research Questions This  study was  designed t o e s t i m a t e the  v a l i d i t y of TOEFL s c o r e s on GPA  predictive  f o r students i n the  UBC/Ritsumeikan Academic Exchange Program, which was a d m i n i s t e r e d by the U n i v e r s i t y Canada and  of B r i t i s h Columbia  Ritsumeikan U n i v e r s i t y  of Japan. As  i n s t i t u t i o n a l v a l i d i t y study, i t was  1993-94 jointly (UBC)  an  intended t o p r o v i d e  of  3  empirical  evidence t o i n v e s t i g a t e whether TOEFL s c o r e s  p r e d i c t GPA,  and  t o e x p l o r e how  language p r o f i c i e n c y  is  r e l a t e d t o academic achievement. The  study addressed the  following  s p e c i f i c research  questions: 1. Do  TOEFL s c o r e s p r e d i c t GPA  for  international  exchange students? 2. Do grades measuring E n g l i s h w r i t i n g and a b i l i t i e s p r e d i c t GPA  speaking  f o r i n t e r n a t i o n a l exchange  students? 3. Do  non-language v a r i a b l e s p r e d i c t GPA  for  i n t e r n a t i o n a l exchange students? D e f i n i t i o n s of Terms P r e d i c t i v e v a l i d i t y . V a l i d i t y r e f e r s to a p p r o p r i a t e n e s s of i n f e r e n c e s  from t e s t s c o r e s or  forms of assessment (American P s y c h o l o g i c a l 1974,  pp.  the other  Association,  25-27). Based upon the k i n d s of i n f e r e n c e s  one  might wish t o draw from t e s t s c o r e s , people t r a d i t i o n a l l y r e f e r t o the validity,  following  types of v a l i d i t y : c r i t e r i o n - r e l a t e d  i n c l u d i n g both p r e d i c t i v e v a l i d i t y and  v a l i d i t y , content v a l i d i t y , and  construct  P r e d i c t i v e v a l i d i t y i n d i c a t e s the predict  concurrent  validity.  extent t o which one  can  f u t u r e performances from p r i o r i n f o r m a t i o n .  P r e d i c t i v e v a r i a b l e s . The make a p r e d i c t i o n  information that  i s used t o  i s t y p i c a l l y r e f e r r e d t o as a p r e d i c t i v e  v a r i a b l e or simply as a  predictor.  4 C r i t e r i o n v a r i a b l e s . The predicted  event or outcome t o  be  i s t y p i c a l l y r e f e r r e d t o as a c r i t e r i o n v a r i a b l e  or simply as a c r i t e r i o n . GPA.  This  i s an acronym f o r grade p o i n t  average. I t i s  used as a measure of academic achievement i n s u b j e c t s courses, u s u a l l y o b t a i n e d by d i v i d i n g the grade p o i n t s  sum  of the t o t a l  by the t o t a l number of c o u r s e s . In the  study i t i s used as an  or  current  i n d i c a t o r of u n i v e r s i t y academic  achievement. TOEFL. T h i s  i s an acronym f o r the T e s t of E n g l i s h  F o r e i g n Language. The  current  study uses TOEFL s c o r e s  i n d i c a t o r s of E n g l i s h  Language p r o f i c i e n c y .  Model. A model i s a hypothesized s t r u c t u r e  as as  used f o r  i n v e s t i g a t i o n of i n t e r r e l a t i o n s between v a r i a b l e s  the  or  hypotheses. A f t e r v a r i a b l e s have been i d e n t i f i e d , or hypotheses have been advanced i n the course of i n q u i r y , i t may  be necessary t o advance a model t h a t p r o v i d e s a  structure  f o r the  variables  or hypotheses. Model b u i l d i n g and  are two  set  of  model t e s t i n g  s t r a t e g i e s t h a t can be employed i n i n q u i r y .  c o r r e l a t i o n and (See  i n t e r r e l a t i o n s between the  Husen, 1994,  regression pp.  can  contribute  Both  t o model b u i l d i n g  3865-3873).  Language p r o f i c i e n c y . T h i s term means p r o g r e s s towards the  attainment of a h i g h degree of knowledge and  English  skill  in  language. In the p r e s e n t study, t h i s i s used  i n t e n t i o n a l l y t o d i s t i n g u i s h i t from language competence, language performance, and  language  aptitude.  5  Academic achievement. In t h i s study,  academic  achievement r e f e r s t o performance by students i n academically oriented courses. I t i s interchangeable with academic success.  6  Chapter  Two  L i t e r a t u r e Review Introduction The research  review of l i t e r a t u r e i n t h i s chapter focuses f i n d i n g s r e l a t e d t o the p r e d i c t i v e v a l i d i t y  TOEFL s c o r e s w i t h GPA divided  i n t o two  achievement, and  as a c r i t e r i o n v a r i a b l e . The  parts:  factors that influence  on of  review i s  academic  f a c t o r s t h a t a f f e c t the e s t i m a t i o n  of  TOEFL's p r e d i c t i v e v a l i d i t y . P a r t I examines c o n c e p t u a l issues  i n s t u d i e s of the p r e d i c t i v e v a l i d i t y of TOEFL  scores,  w h i l e P a r t I I c o n c e n t r a t e s on m e t h o d o l o g i c a l  issues.  Background The  T e s t of E n g l i s h as a F o r e i g n  standardized  Language i s a  t e s t which uses a m u l t i p l e - c h o i c e  format t o  e v a l u a t e the E n g l i s h language p r o f i c i e n c y of non-native speakers. Between 1963  and  1976,  TOEFL c o n t a i n e d  s e c t i o n s : L i s t e n i n g Comprehension, E n g l i s h Vocabulary, Reading Comprehension, and S i n c e September of 1976,  TOEFL has  s e c t i o n s : L i s t e n i n g Comprehension, S t r u c t u r e E x p r e s s i o n , and two  Vocabulary and  Structure,  Writing  consisted  five  Ability.  of t h r e e and  Written  Reading Comprehension.  forms of the t e s t d i f f e r i n t e s t i n g items used  t e s t i n g time allowed, but the same. The  the  The  and  score s c a l e of both t e s t s i s  d i f f e r e n t s e c t i o n s of the t e s t were designed  t o measure d i f f e r e n t language s k i l l s w i t h i n  the  general  domain of language p r o f i c i e n c y . Three decades of t e s t i n g administration  and  extensive research  have shown t h a t TOEFL  7  has  a h i g h degree of r e l i a b i l i t y and  pp.  30-36). Part The  I: F a c t o r s I n f l u e n c i n g  f i r s t p a r t of the  p r e d i c t i v e v a l i d i t y and  (ETS,  1992,  Academic Achievement  literature  two-level conceptual structure  validity  i n the  a five-level  review p r e s e n t s both a study of TOEFL's c l a s s i f i c a t i o n scheme  of f a c t o r s i n f l u e n c i n g academic achievement. I t w i l l examine both language f a c t o r s and  non-language  then  factors  r e l a t e d t o academic achievement. A conceptual  structure  TOEFL s c o r e s are but p r o f i c i e n c y , w h i l e GPA  one  i s but  i n d i c a t o r of language one  i n d i c a t o r of academic  achievement. In a sense, the r e l a t i o n s h i p between TOEFL s c o r e s and  GPA  i s a s u r f a c e - l e v e l m a n i f e s t a t i o n of  the  p a r a l l e l but u n d e r l y i n g r e l a t i o n s h i p e x i s t i n g between  GPA  TOEFL scores  Conventional measurement: Predictor & Criterion variables  Manifest level: Indicators  Latent level: Constructs  Underlying rationale: Independent & Dependent variables  /  Language Proficiency  F i g u r e 2.1.  Academic achievement  A t w o - l e v e l conceptual s t r u c t u r e  TOEFL's p r e d i c t i v e  validity.  i n the  study of  g  language p r o f i c i e n c y relationship  and  i s i l l u s t r a t e d i n Figure  I t i s shown t h a t has  two  academic achievement. T h i s  portions:  s t r u c t u r a l l y a TOEFL p r e d i c t i o n  the m a n i f e s t l e v e l and  T h i s i s analogous i n s t r u c t u r e  t o an  p a r t b e i n g above sea  the  manifest-level  and  l e v e l and  portion  measurement of the predictor  2.1.  GPA  as a c r i t e r i o n . The  as an  achievement as a dependent  and  latent-level  a  portion  and  academic  variable. portions  harmonize w i t h each o t h e r . In a theoretical  r e l a t i o n s h i p between language  academic achievement should  s t a t i s t i c a l methods used t o measure TOEFL's v a l i d i t y , w h i l e the  The  r e l a t i o n s h i p between  independent v a r i a b l e  study, the u n d e r l y i n g  assumption about the proficiency  level.  statistical  a l s o demonstrates t h a t these two  separate from but  TOEFL p r e d i c t i o n  its visible  r e s t below sea  i s a conventional  level.  r e l a t i o n s h i p between TOEFL s c o r e s as  language p r o f i c i e n c y  are not  latent  iceberg,  i s a t h e o r e t i c a l assumption about the  F i g u r e 2.1  the  study  justify predictive  c o n v e n t i o n a l measurement of  r e l a t i o n s h i p between TOEFL s c o r e s and  GPA  the  should f i t the  u n d e r l y i n g t h e o r e t i c a l r a t i o n a l e . For every study, i n f a c t , the  r e s e a r c h method used ought t o match w e l l w i t h  proposed t h e o r e t i c a l assumption. For example, one conduct a c o r r e l a t i o n study t o analyze the between c h i l d r e n ' s  IQ and  the  the might  relationship  s i z e of shoes they wear.  However, t h i s study would not make any  sense because  the  9  s t a t i s t i c a l method i n the be u t i l i z e d ,  study, no matter how  well  i t would  lacks a l o g i c a l supporting r a t i o n a l e .  C l e a r l y , a good TOEFL p r e d i c t i o n study needs a proper s t a t i s t i c a l approach, but more important, i t l a r g e l y on a s t r o n g r a t i o n a l e . T h i s  i s simply because  relies  the  h y p o t h e s i z e d r e l a t i o n s h i p between language p r o f i c i e n c y academic achievement i n f l u e n c e s TOEFL s c o r e s and  the  r e l a t i o n s h i p between  i s measured. U n f o r t u n a t e l y , the  issue  of the u n d e r l y i n g r a t i o n a l e f o r the TOEFL p r e d i c t i o n  study  has  GPA  how  and  been r e p e a t e d l y ignored. Thus, i n p r o p o s i n g such a  l e v e l conceptual structure  f o r TOEFL p r e d i c t i o n  i n t e n t i o n i s t o emphasize the  two-  studies,  the  importance of a comprehensive  examination of f a c t o r s i n f l u e n c i n g academic achievement. T h i s examination serves t o e s t a b l i s h a s o l i d  rationale  u n d e r l y i n g the measurement of the p r e d i c t i v e v a l i d i t y  of  TOEFL s c o r e s . A c l a s s i f i c a t i o n scheme Numerous s t u d i e s have documented a g r e a t v a r i e t y f a c t o r s i n f l u e n c i n g academic achievement, such  as  i n t e l l i g e n c e , language, m o t i v a t i o n , p e r s o n a l i t y , age,  teacher expectation, e t h n i c i t y , learning  teaching strategies, family environment, and  of  interest,  style,  involvement, classroom  peer p r e s s u r e . S i n c e i t i s  nearly  i m p o s s i b l e t o l i s t a l l of these f a c t o r s w i t h i n space, the p r e s e n t study grouped them i n t o a c l a s s i f i c a t i o n scheme sequenced from e x t e r n a l  a limited  five-level factors  i n t e r n a l factors. This c l a s s i f i c a t i o n i s preliminary  to and  10  h e u r i s t i c . I t i s used t o show t h e h i e r a r c h i c a l - s t r u c t u r e and complicated i n t e r r e l a t i o n s of the various f a c t o r s r e l a t e d t o academic achievement  (See F i g u r e  2.2).  Social variables  External factors  Educational variables Language variables  Academic achievement  $  Psychological variables Physiological variables  Internal factors  Figure 2 . 2 .  A f i v e - l e v e l c l a s s i f i c a t i o n scheme o f f a c t o r s  i n f l u e n c i n g academic achievement.  The f i r s t major type o f v a r i a b l e  influencing  achievement i s s o c i a l . T h i s c a t e g o r y i n c l u d e s development,  socioeconomic s t a t u s ,  e t h n i c i t y , s o c i a l a t t i t u d e , family  academic  social  c u l t u r a l background, environment,  involvement, morals and v a l u e s , m a r i t a l  parental  s t a t u s , employment  chances, and r e l i g i o u s b e l i e f s . The second major type o f v a r i a b l e achievement i s e d u c a t i o n a l .  Examples  include  implementation, e d u c a t i o n a l o b j e c t i v e s , materials,  influencing  academic  curriculum  instructional  t e a c h i n g approaches, c h a r a c t e r i s t i c s o f s t u d e n t s ,  c h a r a c t e r i s t i c s o f t e a c h e r s , classroom i n t e r a c t i o n s , time  11  spent on l e a r n i n g , p r i o r knowledge, expectation,  school  learning style,  assessment and e v a l u a t i o n ,  teacher  subject  matter, s t u d e n t s ' s t a t u s , and classroom environment. The t h i r d  i s a l i n g u i s t i c category w i t h such v a r i a b l e s  as f i r s t  l a n g u a g e ( L I ) , second l a n g u a g e ( L 2 ) ,  reading,  speaking, l i s t e n i n g , w r i t i n g , genre, language  p r o f i c i e n c y , communicative competence, productive  skills,  bilingualism,  receptive  skills,  v o c a b u l a r y , and meta-awareness  of  language. The f o u r t h category c o n s i s t s o f p s y c h o l o g i c a l such as m o t i v a t i o n ,  cognition,  emotion,  attention, a t t i t u d e , i n t e r e s t , aptitude,  variables  personality, anxiety,  creation,  temperament, and s e l f - e s t e e m . P h y s i o l o g i c a l v a r i a b l e s represent the f i f t h i n c l u d i n g gender, g e n e t i c brain lateralization,  f a c t o r s , maturation,  aging, h e a l t h ,  category,  fitness,  and n u t r i t i o n .  T h i s c l a s s i f i c a t i o n scheme r e v e a l s how a l a r g e numbers of f a c t o r s may c o n t r i b u t e  t o academic achievement. Language  f a c t o r s a r e o n l y one group among f i v e which academic achievement. Non-language v a r i a b l e s , educational variables, also play  influence  f a c t o r s , such as s o c i a l  v a r i a b l e s , and p h y s i o l o g i c a l  important r o l e s . O v e r s i m p l i f i c a t i o n o f  the p r o c e s s o f academic l e a r n i n g o r t h e i n t e r r e l a t i o n s h i p among f a c t o r s concerned may l e a d t o erroneous f i n d i n g s . Thus, t h e f o l l o w i n g l i t e r a t u r e review i s o r g a n i z e d i n t o two sections:  first,  language f a c t o r s and academic  achievement,  w i t h a focus on t h e r e l a t i o n s h i p between second language  12  p r o f i c i e n c y and u n i v e r s i t y academic achievement; and second, non-language  f a c t o r s and academic achievement, w i t h a  h i g h l i g h t on the r e l a t i o n s h i p between gender d i f f e r e n c e s and academic achievement. Language f a c t o r s and academic achievement  1  I t has been g e n e r a l l y r e c o g n i z e d t h a t language i s t h e major medium of l e a r n i n g  (Mohan, 1986)  and language  p r o f i c i e n c y i s important t o academic success. For those who study i n e d u c a t i o n a l i n s t i t u t i o n s where the language o f i n s t r u c t i o n i s t h e i r second language, i n p a r t i c u l a r ,  their  L2 p r o f i c i e n c y remarkably a f f e c t s , even determines, academic achievement. However, r e s e a r c h i n L2 e d u c a t i o n shows t h a t the  s t r e n g t h o f the r e l a t i o n s h i p between L2 p r o f i c i e n c y and  academic achievement v a r i e s f o r d i f f e r e n t language  skills  and a c r o s s c o n t e n t areas. L2 s k i l l s and academic achievement. Cummins d e s c r i b e d two types o f language p r o f i c i e n c y : I n t e r p e r s o n a l Communication  Skill  Academic Language P r o f i c i e n c y  (1981)  Basic  (BICS) and C o g n i t i v e  (CALP). He p o i n t e d out t h a t  academic language p r o f i c i e n c y , r a t h e r than d a i l y c o n v e r s a t i o n a l competence,  i s necessary f o r academic  success. H i s f i n d i n g s have been supported by many e m p i r i c a l studies  (Collier,  1987). Other r e s e a r c h e r s have e x p l o r e d the  T h i s review mainly focuses on s t u d i e s c o n c e r n i n g the r e l a t i o n s h i p between second language p r o f i c i e n c y and academic achievement. T h i s i s because of the l i m i t a t i o n o f the space, and the t o p i c being too broad t o c o v e r . More important, i t r e l a t e s d i r e c t l y t o the p r e s e n t r e s e a r c h questions. 1  13  r e l a t i o n s h i p among d i f f e r e n t language s k i l l s , speaking, r e a d i n g , and  listening,  w r i t i n g , t o academic achievement.  Johns (1981) conducted a study i n v o l v i n g an skills  questionnaire  w i t h 200  academic  f a c u l t y from a l l departments  a t an American u n i v e r s i t y i n order t o determine which language s k i l l s  among r e a d i n g , w r i t i n g , speaking,  and  l i s t e n i n g were most e s s e n t i a l t o non-native speakers' success i n t h e i r u n i v e r s i t y c l a s s e s . R e s u l t s showed t h a t r e c e p t i v e were ranked Ostler  both r e a d i n g and  listening,  (1980) r e p o r t e d  s i m i l a r f i n d i n g s i n a study o f a  c o l l e g e s t u d e n t s ' assessment of what academic  they needed t o achieve academic success. The  revealed  study  first.  group of ESL skills  skills,  of the  t h a t academic r e a d i n g s k i l l was  needed among s i x t e e n language s k i l l s .  study  ranked as the most  Other h i g h l y  ranked  s k i l l s were t a k i n g notes, a s k i n g q u e s t i o n s i n c l a s s , r e a d i n g journals,  and  w r i t i n g research  In a study of 178  proposals.  university professors'  judgments of  e r r o r s i n the w r i t i n g of non-native speaking students, Santos (1988) r e p o r t e d  that professors  emphasis on language f e a t u r e s  seemed t o p l a c e more  than on content f e a t u r e s ,  l e x i c a l e r r o r s i n w r i t i n g were r a t e d as the most  and  serious.  T h i s suggested t h a t b a s i c knowledge of v o c a b u l a r y i n w r i t i n g plays  an important r o l e i n academic achievement. Magan (1986) conducted r e s e a r c h  between speaking p r o f i c i e n c y and c o l l e g e French s t u d e n t s . His  on the  relationship  academic achievement of  findings revealed  40  a significant  14  p o s i t i v e r e l a t i o n s h i p between speaking a b i l i t y and  academic  success. In a c a n o n i c a l c o r r e l a t i o n a n a l y s i s , (1985) found t h a t  l i s t e n i n g a b i l i t y was  academic performance a t the u n i v e r s i t y speaking, r e a d i n g , and i t was  l i k e l y that  not  Spinks  as p r e d i c t i v e  l i s t e n i n g d i f f i c u l t i e s might  that  be  reading.  foregoing research findings  suggest t h a t  achievement, although i t appears t h a t no  different  academic consensus e x i s t s  i n terms of which language s k i l l p l a y s the most  important r o l e . L2 p r o f i c i e n c y a n a l y z e d the  of l e a r n i n g .  and  Mohan's t h e o r e t i c a l p e r s p e c t i v e p r o v i d e d  and  r e l a t i o n s h i p between second language academic achievement i n d i f f e r e n t s u b j e c t  areas a c r o s s the Slark  content  nature of language i n e d u c a t i o n as a medium  i n s i g h t i n t o the proficiency  a c r o s s s u b j e c t matters. Mohan (1986)  r e l a t i o n s h i p between language and  c o n s i d e r e d the  and  curriculum.  Bateman (1981) s t u d i e d  speakers* c o l l e g e  a significant positive  between language s c o r e s and (Anthropology and  non-native  academic achievement. T h e i r  showed t h a t t h e r e was  (Chemistry, Mathematics, and  English  findings correlation  course grades i n two  Sociology),  of  l e v e l as were  language s k i l l s have d i f f e r e n t impacts on  yet  and  w r i t i n g a b i l i t i e s . They argued  compensated through a d d i t i o n a l The  Ho  courses  whereas t h r e e o t h e r courses Music) c o n s i s t e n t l y  n e g a t i v e c o r r e l a t i o n c o e f f i c i e n t s . The  results  showed indicated  15  that courses i n s o c i a l sciences required higher l e v e l s of language p r o f i c i e n c y than those i n n a t u r a l s c i e n c e s and music. C r a n d a l l and o t h e r s (1987) a n a l y z e d the r e l a t i o n s h i p o f ESL language development mathematics,  t o academic achievement i n  s c i e n c e , and s o c i a l s t u d i e s . They argued t h a t  although the exact r e l a t i o n s h i p between ESL language development  and c o n t e n t l e a r n i n g o f these s u b j e c t s was  not  c l e a r l y understood, both a minimal l e v e l of language p r o f i c i e n c y w i t h s p e c i f i c l i n g u i s t i c r e g i s t e r s and a minimal knowledge o f the academic area were r e q u i r e d f o r academic success. As f a r as mathematics  l e a r n i n g i s concerned, s t u d i e s  w i t h monolingual E n g l i s h speakers have r e v e a l e d a h i g h p o s i t i v e c o r r e l a t i o n between mathematics English reading a b i l i t y  achievement  (Aiken, 1971; Duran,  1979).  and These  r e s u l t s are i n t e r e s t i n g because mathematics uses i t s own symbolic system except f o r word problem s o l v i n g . In MacNamara's s t u d i e s  (1966, 1967), b i l i n g u a l c h i l d r e n kept  pace w i t h monolinguals i n mechanical a r i t h m e t i c , but  fell  behind i n s o l v i n g word problems. S e v e r a l r e s e a r c h e r s have found t h a t language m i n o r i t y students f r e q u e n t l y do not understand the language used t o p r e s e n t mathematics problems  (DeAvila & Havassy,  test  1974; Moreno, 1970).  In s h o r t , r e s e a r c h f i n d i n g s show t h a t t h e r e i s a r e l a t i o n s h i p between language f a c t o r s and  academic  achievement f o r d i f f e r e n t language s k i l l s and i n d i f f e r e n t  16  subject  areas, but do not r e v e a l  Vinke and Jochems (1993) p o i n t e d  i d e n t i f i a b l e patterns. out, t h e r e i s no  As  generally  acknowledged t h e o r y on the p r e c i s e nature o f the r e l a t i o n s h i p between language p r o f i c i e n c y and academic achievement. T h e r e f o r e , making c o n c l u s i v e  statements about  the r e l a t i o n s h i p i s premature. Non-language  f a c t o r s and academic achievement  Comprehensive  s t u d i e s on non-language  variables  a f f e c t i n g academic achievement. Many r e s e a r c h e r s have examined the e f f e c t s o f non-language f a c t o r s , i n d i v i d u a l l y or i n combination, on academic achievement. These include teacher expectation achievement m o t i v a t i o n  factors  (Rosenthal and Jacobson, 1968) ,  (Ames & Ames, 1984), home environment  (Soto, 1990), and s o c i a l disadvantage (Ushasree, 1990). In a d d i t i o n , comprehensive s t u d i e s  on v a r i e d  factors affecting  academic achievement have been conducted i n order t o i d e n t i f y f a c t o r s t h a t s i g n i f i c a n t l y and  consistently  i n f l u e n c e academic achievement and t o p r o v i d e  empirical  evidence about weights and i n t e r r e l a t i o n s h i p s among t h e s e factors. Ho and Spinks (1985) examined the e f f e c t s o f f o u r variables, verbal personality,  i n t e l l i g e n c e , E n g l i s h language  skills,  and a t t i t u d e , on u n i v e r s i t y academic  performance. T h e i r f i n d i n g s showed t h a t  (a) E n g l i s h  language  s k i l l s had the most p r e d i c t i v e v a l u e , a c c o u n t i n g f o r about 10% o f t h e v a r i a n c e  i n academic performance; (b) V e r b a l  17  i n t e l l i g e n c e , a t t i t u d e (excepting  study o r i e n t a t i o n )  p e r s o n a l i t y were not p r e d i c t i v e of academic Walberg, S c h i l l e r , and H a e r t e l  and  performance.  (1979, 1982)  collected  and analyzed the review l i t e r a t u r e o f the 1970s on the e f f e c t s o f i n s t r u c t i o n and r e l a t e d f a c t o r s on c o g n i t i v e , a f f e c t i v e and b e h a v i o r a l major r e s e a r c h  domains.  Based on a s y n t h e s i s  of 2 3  t o p i c s addressed by thousands of s t u d i e s ,  they found t h a t nine v a r i a b l e s appeared t o have c o n s i s t e n t causal  i n f l u e n c e s on academic l e a n i n g :  student age o r  developmental l e v e l , a b i l i t y , m o t i v a t i o n , i n s t r u c t i o n , q u a l i t y o f i n s t r u c t i o n , the  amount o f psychological  environments o f the c l a s s , home, peer group o u t s i d e  school,  and exposure t o the mass media. By p e r f o r m i n g a l i n e a r s t r u c t u r e r e l a t i o n a n a l y s i s (LISREL), Walberg and t h r e e o t h e r c o - r e s e a r c h e r s  (1984)  compared f i v e c a u s a l models t o examine the r e l a t i o n s h i p between achievement i n s c i e n c e and a combination o f e i g h t v a r i a b l e s . The e i g h t v a r i a b l e s were s t u d e n t s ' a b i l i t y , home environment, peer group, exposure t o mass media, environment, time on task, m o t i v a t i o n , strategies. Results students' a b i l i t y  social  and i n s t r u c t i o n a l  showed t h a t among the e i g h t  ( r ranged from .72 t o .75) and  factors motivation  (r ranged from .11 t o .12) c o n s i s t e n t l y had the l a r g e s t i n f l u e n c e s on s c i e n c e  achievement.  In another r e s e a r c h  synthesis  (Walberg, P a s c a r e l l a ,  H a e r t e l , Junker, & Boularger, 1982), 14 major  variables  which a f f e c t academic achievement i n s c i e n c e ,  math, s o c i a l  18  s t u d i e s , and  r e a d i n g were l i s t e d . The  14 v a r i a b l e s were  age,  achievement, a t t i t u d e , socioeconomic s t a t u s , q u a l i t y of i n s t r u c t i o n , q u a n t i t y of i n s t r u c t i o n , education,  home, peer,  homework, media-TV, e x t r a c u r r i c u l a r , s t i m u l a t i o n ,  and  gender. Gender d i f f e r e n c e s and s t u d i e s have d i s c u s s e d  academic achievement. Numerous  gender d i f f e r e n c e s and  academic  achievement. Maccoby and J a c k l i n (1974) i n t h e i r w i d e l y c i t e d book summarized and research  analyzed  on gender d i f f e r e n c e s and  a l a r g e amount of concluded t h a t :  have g r e a t e r v e r b a l a b i l i t y than boys; visual-spatial ability; a b i l i t y ; And  Girls  (b) Boys e x c e l i n  (c) Boys e x c e l i n mathematical  (d) males are more a g g r e s s i v e .  were supported by  (a)  Their results  f i n d i n g s of l a r g e s c a l e s t u d i e s conducted  n a t i o n a l l y or i n t e r n a t i o n a l l y . The 1994,  N a t i o n a l Assessment of E d u c a t i o n a l  pp.  Programs (Husen,  5425-5426) i n i t s l a r g e s c a l e s t u d i e s over t e n  y e a r s found t h a t the g i r l s performed c o n s i s t e n t l y b e t t e r both r e a d i n g  and w r i t i n g t e s t s than boys, but not  on  on  science. The  I n t e r n a t i o n a l A s s o c i a t i o n f o r the E v a l u a t i o n  Educational science  Achievement (IEA)  (Keeves, 1973)  p a t t e r n of r e s u l t s was  s t u d i e s of mathematics  showed t h a t , w h i l e the one  and  general  of s u p e r i o r performance by male  students i n both s u b j e c t s , t h e r e was between c o u n t r i e s i n the extent i n performance.  of  considerable v a r i a t i o n  t o which boys exceeded  girls  19  Walker (1976, i n Husen, 1992,  p. 5426) r e p o r t e d another  IEA study on gender d i f f e r e n c e s i n s i x s u b j e c t s a r e a s : r e a d i n g , l i t e r a t u r e , E n g l i s h as a f o r e i g n language, as a f o r e i g n language, comprehension girls  and c i v i c  French  e d u c a t i o n . On r e a d i n g  t e s t s , boys showed lower performance  than  i n a m a j o r i t y of c o u n t r i e s , but i n g e n e r a l these  d i f f e r e n c e s were s l i g h t . On the l i t e r a t u r e t e s t s ,  ina l l  c o u n t r i e s the boys d i d l e s s w e l l , and they a l s o showed l e s s i n t e r e s t i n l i t e r a t u r e . Again, i n a study of the t e a c h i n g of E n g l i s h as a f o r e i g n language, the boys s c o r e d below the g i r l s on both the r e a d i n g and l i s t e n i n g t e s t s , but the d i f f e r e n c e s were s m a l l . In a study of the t e a c h i n g of French as a f o r e i g n language,  statistically  s i g n i f i c a n t gender  d i f f e r e n c e s i n the l e a r n i n g of French were r e c o r d e d i n English-speaking countries, with g i r l s performing b e t t e r than boys. In c i v i c e d u c a t i o n achievement  t e s t s , the boys  g e n e r a l l y recorded h i g h e r s c o r e s than g i r l s . S e v e r a l s t u d i e s examined i s s u e s of gender d i f f e r e n c e s i n language t e s t s . Landsheere  (1994) found t h a t boys perform  m a r g i n a l l y b e t t e r than g i r l s on m u l t i p l e - c h o i c e t e s t s  and  p r o b l e m - s o l v i n g e x e r c i s e s . G i r l s perform b e t t e r than boys on essay t e s t s i n w r i t t e n composition and are g e n e r a l l y a s s i g n e d h i g h e r grades i n school-based assessments. another study  In  (Zeidner, 1987), the r e s e a r c h e r a n a l y z e d the  E n g l i s h language a p t i t u d e t e s t s c o r e s of 824  full  time  Jewish students i n I s r a e l and found t h a t a s m a l l degree of gender d i f f e r e n c e s i n t e s t s c o r e s was  observed, t e n d i n g t o  20  overpredict  t h e f i r s t y e a r ' s GPA o f males and u n d e r p r e d i c t  t h a t o f females. The r e s e a r c h e r argued t h a t t h i s might be the r e s u l t o f d i f f e r e n t i a l g r a d i n g p r a c t i c e s and unevenness i n t h e number o f males and females i n courses, r a t h e r  than  as a f a c t o f nature. In summary, much r e s e a r c h differences  has documented gender  i n academic achievement i n such s u b j e c t  mathematics, s c i e n c e ,  s o c i a l studies,  language a r t s , and  f o r e i g n languages. I t appears c l e a r t h a t gender d i f f e r e n c e s  areas as  (a) t h e r e a r e  i n academic achievement;  (b) these  d i f f e r e n c e s should not be exaggerated; and (c) many f a c t o r s contribute  t o gender d i f f e r e n c e s . In f a c t , gender should not  be c o n s i d e r e d as a p u r e l y  b i o l o g i c a l e n t i t y , but r a t h e r , a  composite v a r i a b l e combining p h y s i o l o g i c a l , and  psychological,  s o c i o l o g i c a l components. Gender d i f f e r e n c e s  i n academic  achievement o r i g i n a t e from a v a r i e t y o f sources, such as participation differences, a b i l i t i e s differences, b i o l o g i c a l differences,  s o c i a l i z a t i o n differences, differences i n  a t t i t u d e s and t h e i r e f f e c t s , and d i f f e r e n c e s  i n the  expectancy o f success (Husen, 1982, pp. 5428-5430). Part I I : Factors The  Influencing  TOEFL's P r e d i c t i v e V a l i d i t y  f o l l o w i n g p a r t o f l i t e r a t u r e review examines f i v e  major m e t h o d o l o g i c a l f a c t o r s which s u b s t a n t i a l l y the e s t i m a t i o n f a c t o r s are:  o f TOEFL s c o r e s '  influence  p r e d i c t i v e v a l i d i t y . These  (a) Which a n a l y t i c a l models a r e employed? (b)  What s u b j e c t v a r i a b l e s a r e i n v o l v e d ?  (c) What p r e d i c t o r  21  v a r i a b l e s a r e used? (d) What c r i t e r i o n v a r i a b l e s a r e s e l e c t e d ? And (e) how r e s u l t s a r e computed and i n t e r p r e t e d ? ,  2  A n a l y t i c a l models An a n a l y t i c a l model r e f e r s t o a h y p o t h e s i z e d  structure  t o emulate and a n a l y z e t h e i n t e r r e l a t i o n s between v a r i a b l e s . There a r e many a n a l y t i c a l models used f o r p r e d i c t i o n o r explanation studies  (Pedhazur, 1982). I t i s important t o  choose and employ a p p r o p r i a t e a n a l y t i c a l models i n c o n d u c t i n g a study o f TOEFL*s p r e d i c t i v e v a l i d i t y . The model should be chosen p r o p e r l y i n order t o f i t the data as w e l l as t h e r e s e a r c h q u e s t i o n under study. I t should be used c o r r e c t l y i n order t o meet t h e assumptions u n d e r l y i n g t h e model. The c o r r e l a t i o n model v e r s u s t h e r e g r e s s i o n model. Most s t u d i e s e s t i m a t i n g TOEFL*s p r e d i c t i v e v a l i d i t y have a p p l i e d the c o r r e l a t i o n model as t h e s o l e a n a l y t i c a l model Stallins,  (Chase &  1966; Abdzi, 1967; Kwang & Dizney, 1970; M a r t i n ,  1971; AACRAO, 1971; Pack, 1972; H e i l & Aleamoni, 1974; Shay, 1975; Harcey, 1979; B o s t i c , 1981; Riggs, 1982; Odunze, 1982; L i g h t , Xu & Mossop, 1987; Johnson, 1988; L i g h t & Wan, 1991; Ayers & Ouattlebaum, 1992). These s t u d i e s u s u a l l y e s t i m a t e d TOEFL*s p r e d i c t i v e v a l i d i t y by c a l c u l a t i n g  correlation  2  The p r e s e n t study p u r p o s e l y used t h e term o f a n a l y t i c a l models i n s t e a d o f s t a t i s t i c a l methods o r s t a t i s t i c a l models. A s c i e n t i f i c a n a l y s i s i s not i d e n t i c a l t o a s t a t i s t i c a l method. Even f o r q u a n t i t a t i v e r e s e a r c h i n which t h e s t a t i s t i c a l method i s i t s e s s e n t i a l component, t h e s t a t i s t i c a l method cannot cover a l l the content t h a t t h e a n a l y t i c a l model c o n t a i n s , such as model c o n s t r u c t i o n and model m o d i f i c a t i o n .  22  c o e f f i c i e n t s between TOEFL s c o r e s and GPA. The c o r r e l a t i o n model has dominated TOEFL p r e d i c t i o n Some r e s e a r c h e r s 1972;  research.  (Schreder & P i t c h e r , 1970; Sharon,  Gue & Holdaway, 1973; Stove, 1982; Hassan, 1982; Yule  & Hoffman, 1990) have used t h e c o r r e l a t i o n model as t h e main a n a l y t i c a l model w i t h t h e r e g r e s s i o n model as a supplement. These authors estimated c o r r e l a t i o n c o e f f i c i e n t s (r) and proportion  of variance  accounted f o r by r e g r e s s i o n  (R ), i n 2  some cases w i t h r e g r e s s i o n c o e f f i c i e n t s (b & P) o r t h e regression  equation).  A few s t u d i e s have adopted the r e g r e s s i o n model as t h e main a n a l y t i c a l t o o l w i t h the c o r r e l a t i o n model as i t s i n t e g r a t e d component (Wilcox, Peters,  1977; S o k a r i ,  1975; A n d a l i b ,  1976; Ayers &  1981). In these s t u d i e s , c o r r e l a t i o n  c o e f f i c i e n t s a r e c a l c u l a t e d as one o f the b a s i c d e s c r i p t i v e estimates.  The main procedure i s t o perform a r e g r e s s i o n  a n a l y s i s so t h a t t h e r e g r e s s i o n equation,  squared m u l t i p l e  c o r r e l a t i o n , and/or r e g r e s s i o n c o e f f i c i e n t s a r e obtained. Which a n a l y t i c a l model should be chosen f o r p r e d i c t i o n s t u d i e s ? T h i s i s s u e has been d i s c u s s e d psychometric r e s e a r c h  extensively i n  s i n c e t h e 1950s ( K e n d a l l , 1951; F i s h ,  1959; E z e k i e l St Fox, 1959; Fox, 1968; Warren,  1958;  Binder,  1971;  Thorndike, 1978;). Based on these s t u d i e s , Pedhazur  (1991) concluded t h a t when t h e focus o f the r e s e a r c h the e x p l a n a t i o n ,  i s on  o r t h e p r e d i c t i o n , o f dependent v a r i a b l e s ,  the r e g r e s s i o n model i s a p p r o p r i a t e  (p. 409). In TOEFL  p r e d i c t i o n s t u d i e s , t h e r e s e a r c h purpose i s t o see how w e l l  23  TOEFL s c o r e s p r e d i c t t h e c r i t e r i o n v a r i a b l e GPA, but n o t t o d e s c r i b e t h e a s s o c i a t i o n between two a r b i t r a r i l y  selected  v a r i a b l e s . Thus, t h e r e g r e s s i o n model r a t h e r than t h e c o r r e l a t i o n model i s t h e proper s o l u t i o n . The  simple r e g r e s s i o n model v e r s u s t h e m u l t i p l e  r e g r e s s i o n model. TOEFL p r e d i c t i o n s t u d i e s u s i n g t h e r e g r e s s i o n model as t h e i r main o r supplemental a n a l y t i c a l t o o l can be c l a s s i f i e d  i n t o t h r e e groups i n terms o f t h e  number and t h e v a r i e t y o f p r e d i c t o r v a r i a b l e s  involved.  In t h e f i r s t group o f s t u d i e s , only one p r e d i c t o r v a r i a b l e i s used i n t h e r e g r e s s i o n model ( f o r i n s t a n c e , Hassan, 1982). In t h e second group, m u l t i p l e p r e d i c t o r v a r i a b l e s o f E n g l i s h language p r o f i c i e n c y a r e used as p r e d i c t o r s i n t h e r e g r e s s i o n models ( f o r example, GRE-V and MTELP s c o r e s , Abdzi,  1967; TOEFL's f i v e - s e c t i o n s c o r e s ,  P r i o r - and post-admission TOEFL s c o r e s , scores,  Sharon, 1972;  and i n t e r v i e w  Gue and Holdaway, 1973; TOEFL's o v e r a l l and  s e c t i o n a l s c o r e s , Hu, 1991). M u l t i p l e p r e d i c t o r v a r i a b l e s with m u l t i p l e features are used i n t h e t h i r d group o f s t u d i e s  ( f o r example, TOEFL and  LSAT. Schrader & P i t c h e r , 1970; TOEFL, ACT, SAT, h i g h GPA, and age. A n d a l i b ,  school  1976; TOEFL, ESL course grades,  n a t i v e language, major areas o f study. Stove, 1982; TOEFL, GRE-V and GRE-Q. Yule & Hoffman, 1990). The  number and t h e v a r i e t y o f p r e d i c t o r s i n t h e  r e g r e s s i o n model a r e dependent upon t h e complexity o f t h e  24  research  problem under study. When a one-cause-one-effect  r e l a t i o n s h i p e x i s t s , a simple r e g r e s s i o n model should  be  used t o p r e d i c t a phenomenon completely determined by  a  s i n g l e f a c t o r . For more complicated  phenomena, more  p r e d i c t o r s are needed. For phenomena i n f l u e n c e d by d i f f e r e n t types of f a c t o r s , a m u l t i p l e r e g r e s s i o n model w i t h d i f f e r e n t types of p r e d i c t o r s i s r e q u i r e d . In s o c i a l and research,  educational  the m u l t i p l e r e g r e s s i o n model i s necessary i n most  cases t o make the p r e d i c t i o n study d e f e n s i b l e . There are m a n i f o l d  f a c t o r s a f f e c t i n g academic  achievement, t h e r e f o r e , a m u l t i p l e r e g r e s s i o n model w i t h multiple p r e d i c t o r s i s appropriate o n l y one  t o p r e d i c t GPA.  Using  p r e d i c t o r , or the language-based p r e d i c t o r s , makes  i t d i f f i c u l t t o g a i n an a c c u r a t e  p r e d i c t i o n of GPA.  Many  TOEFL p r e d i c t i o n s t u d i e s , as reviewed above, used languagebased v a r i a b l e s ; as a r e s u l t , they f r e q u e n t l y r e l a t i v e l y smaller R, 2  obtained  even though more s i m i l a r language-  based p r e d i c t o r s were added i n t o the r e g r e s s i o n Thus i n TOEFL p r e d i c t i o n s t u d i e s , we  should  equation.  not o n l y choose  m u l t i p l e p r e d i c t o r s , but a l s o take i n t o account the degree of d i v e r s i t y of the p r e d i c t o r s . More complicated  models, such as path a n a l y s i s model,  L i n e a r S t r u c t u r a l R e l a t i o n s model, H i e r a r c h i c a l L i n e a r Model, c a n o n i c a l a n a l y s i s model, and  discriminant  analysis  model, can a l s o be used t o analyze the complex r e l a t i o n s h i p of f a c t o r s a f f e c t i n g academic achievement.  25  The  s i n g l e - s t e p regression c a l c u l a t i o n versus  the  comprehensive r e g r e s s i o n a n a l y s i s package. R e g r e s s i o n a n a l y s i s should  not be seen as the s o l e c a l c u l a t i o n of  R,  or of r e g r e s s i o n c o e f f i c i e n t s . An a n a l y t i c a l p r o c e s s of  2  the  m u l t i p l e r e g r e s s i o n t y p i c a l l y i n v o l v e s i n t e g r a t e d components and  r e l e v a n t techniques,  i n c l u d i n g the checking of  assumptions, d e t e c t i n g o u t l i e r s and  (by u s i n g r e s i d u a l a n a l y s i s  influence a n a l y s i s ) , regression estimation,  hypothesis  t e s t i n g , as w e l l as power a n a l y s i s (Cohen, 1988; 1994,  p. 3866; Pedhazur, 1991). The  procedures  t e c h n i q u e s mentioned above examine the models t o data,  the e x i s t e n c e  the v a r i a b l e s , and generalized  f i t of  and regression  of o u t l i e r s , the w e i g h t i n g of  the degree t o which r e s u l t s can  so t h a t the q u a l i t y of a m u l t i p l e  a n a l y s i s can be  Husen,  be  regression  optimized.  M u l t i p l e r e g r e s s i o n s t u d i e s of TOEFL's p r e d i c t i v e v a l i d i t y usually report R, 2  (Schreder & P i t c h e r , 1970; Stove, 1982;  Ayers & P e t e r s , and  t h a t few  Sharon, 1972;  Yule & Hoffman, 1990;  stepwise r e g r e s s i o n  (Sgg,.)  the r e g r e s s i o n  (Gue  Hu,  Sokari,  1981;  1991), r e s u l t s o f  & Holdaway, 1973;  1977), and/or standard  equation  Andalib,  e r r o r of  1976;  estimation  shrinkage (Hassan, 1982) . However, i t appears researchers,  i f any,  perform the  r e g r e s s i o n a n a l y s i s mentioned above.  the  comprehensive  26  Subject v a r i a b l e s  3  Many s t u d i e s have r e p o r t e d t h a t v a r i o u s  subject  v a r i a b l e s a f f e c t the e s t i m a t i o n of TOEFL's p r e d i c t i v e validity  (Hale, S t a n f i e l d , & Duran, 1984). These s u b j e c t  v a r i a b l e s can be grouped i n t o f o u r c a t e g o r i e s . (a) Personal educational  information,  such as gender, age,  parents'  level.  (b) S o c i a l f a c t o r s , i n c l u d i n g n a t i v e language, home country  or r e g i o n , c i t i z e n s h i p , e t h n i c group,  adjustment, and  occupation  social  i n home country.  (c) Academic background, f o r i n s t a n c e , areas of study, type of degree sought, e d u c a t i o n a l  l e v e l , and  previous  grades. (d) T e s t - r e l a t e d i n f o r m a t i o n , or non-repeaters, TOEFL scores Saturday program, and the The  such as TOEFL  repeaters  i n the F r i d a y program or  the  like.  f o l l o w i n g d i s c u s s i o n , however, focuses  on  two  i s s u e s r e l a t e d t o s u b j e c t v a r i a b l e s . These i s s u e s cause s e r i o u s problems but were o f t e n ignored  i n the e s t i m a t i o n  of  TOEFL's p r e d i c t i v e v a l i d i t y . Sample s i z e . Sample s i z e i s a s s o c i a t e d with  the  homogeneousness of s u b j e c t s under study. D i f f e r e n c e s i n In some s t u d i e s , some s u b j e c t v a r i a b l e s were used as p r e d i c t o r v a r i a b l e s , f u n c t i o n i n g as a moderator or mediator along w i t h TOEFL scores t o p r e d i c t GPA. T h i s c o u l d be a l s o viewed as an evidence of i n f l u e n c e s of s u b j e c t v a r i a b l e s on TOEFL/GPA r e l a t i o n . I t i s t h i s k i n d of unintended and e a s i l y - n e g l e c t e d e f f e c t of s u b j e c t v a r i a b l e s t h a t make nonexperimental r e s e a r c h , i n c l u d i n g the TOEFL/GPA study, more complicated. 3  27  sample s i z e have d i f f e r e n t e f f e c t s on the p r e d i c t i v e v a l i d i t y o f TOEFL s c o r e s . Although almost a l l s t u d i e s  on t h e  p r e d i c t i v e v a l i d i t y o f TOEFL s c o r e s r e p o r t e d t h e i r sample s i z e s , t h e range i n sample s i z e among v a r i e d Some TOEFL p r e d i c t i v e s t u d i e s  lacked  from 15 t o 900.  s u f f i c i e n t sample s i z e  ( B o s t i c , 1981; Hassan, 1982; Riggs, 1982). Most s t u d i e s the  cumulative sample s i z e obtained a c r o s s y e a r s  Schreder & P i t c h e r , 1972.  used  (e.g.,  1970. n=63, from 1964 t o 1969; Sharon,  n=973, 1964-69; Pack, 1972. n=402, 1960-72; Gue &  Holdaway, 1973. n=123, 1967-70). T h i s k i n d o f cumulative sample s i z e might r e s u l t i n problems r e g a r d i n g the p r e d i c t i v e v a l i d i t y o f TOEFL s c o r e s . I t might confound various subject  variables,  i g n o r e the d i f f e r e n c e s  forms o f TOEFL ( i . e . , t h r e e - s e c t i o n  i n t h e two  and f i v e - s e c t i o n ) , o r  l o s e unique i n f o r m a t i o n i n sub-samples f o r each year. Mean TOEFL s c o r e s . Mean TOEFL s c o r e s i n d i c a t e t h e average l e v e l o f E n g l i s h  language p r o f i c i e n c y o f the  s u b j e c t s under study. They s u b s t a n t i a l l y i n f l u e n c e t h e e x t e n t t o which TOEFL s c o r e s p r e d i c t academic achievement. Wilcox  (1975) found t h a t one group o f s u b j e c t s w i t h  b e t t e r ESL p r o f i c i e n c y showed no r e l a t i o n s h i p between TOEFL s c o r e s and GPA, whereas another group with lower  English  l e v e l s showed a s i g n i f i c a n t r e l a t i o n s h i p . He e x p l a i n e d  that  E n g l i s h a b i l i t y and academic success may be r e l a t e d a t low l e v e l s o f p r o f i c i e n c y but u n r e l a t e d threshold  a t l e v e l s above c e r t a i n  v a l u e s . Wilcox's f i n d i n g s suggest t h a t t h e  e x i s t e n c e o f c e r t a i n t h r e s h o l d s o f TOEFL s c o r e s p r o b a b l y  28  r e s u l t s i n a n o n l i n e a r r e l a t i o n s h i p between E n g l i s h p r o f i c i e n c y l e v e l and academic achievement. S i m i l a r l y , Johnson (1988) found t h a t when E n g l i s h p r o f i c i e n c y i s r e l a t i v e l y low,  TOEFL s c o r e s can p r e d i c t  academic performance. With h i g h e r language p r o f i c i e n c y , o t h e r v a r i a b l e s such as p r i o r exposure t o s u b j e c t matter, m o t i v a t i o n , study s k i l l s , c u l t u r a l a d a p t a b i l i t y , and f i n a n c i a l s e c u r i t y , may  became more  The TOEFL T e s t Manual (ETS, standard  even  important.  1992)  s t a t e s t h a t i f the  f o r E n g l i s h language p r o f i c i e n c y i s s e t a t such a  h i g h l e v e l t h a t o n l y a p p l i c a n t s with good E n g l i s h s k i l l s admitted,  t h e r e may  are  be l i t t l e r e l a t i o n s h i p between TOEFL  s c o r e s and any of the c r i t e r i o n measures. Because t h e r e  will  be no l a r g e v a r i a n c e i n E n g l i s h p r o f i c i e n c y among the group members, v a r i a t i o n s i n success on the c r i t e r i o n v a r i a b l e s w i l l be due  t o other non-English  i f the standard  causes.  On the o t h e r hand,  i s s e t a t too low a l e v e l , a l a r g e number o f  a p p l i c a n t s s e l e c t e d with TOEFL s c o r e s may  be u n s u c c e s s f u l i n  the academic program. There w i l l be a r e l a t i v e l y c o r r e l a t i o n between t h e i r TOEFL s c o r e s and  high  i t s criterion  measures. Thus, with a standard t h a t i s n e i t h e r too h i g h too low,  the c o r r e l a t i o n between TOEFL s c o r e s and  nor  subsequent  success w i l l be only moderate. Mean TOEFL s c o r e s a l s o i n v o l v e the i s s u e of of  restriction  range. R e s t r i c t i o n of range means t h a t , as a r e s u l t  s e l e c t i o n , the range of s u b j e c t s i n a study r e s t r i c t e d and o n l y those who  of  i s inevitably  are s e l e c t e d with  certain  29  standards r a t h e r than those who  are randomly drawn from the  t r u e p o p u l a t i o n are a v a i l a b l e f o r i n v e s t i g a t i o n .  Restriction  of range l e a d s t o a sampling b i a s . In TOEFL's p r e d i c t i o n s t u d i e s , the sample under study i s r e s t r i c t e d by the minimum TOEFL requirement f o r admission s e l e c t i o n so t h a t an unrandomized sampling b i a s o c c u r s . Based on a c r i t i c a l a n a l y s i s of s i x s t u d i e s of TOEFL s p r e d i c t i v e v a l i d i t y ,  Yan  1  (1994) found t h a t TOEFL means i n these s t u d i e s ranged  from  491.00 t o 561.00, which were above the 50th p e r c e n t i l e i n the p o p u l a t i o n of a l l TOEFL t a k e r s . Standard  rank  deviations  i n these s t u d i e s ranged from 38.80 t o 66.00, which were lower than the standard d e v i a t i o n of the p o p u l a t i o n . In most cases, sampling i n the TOEFL/GPA s t u d i e s was  based  primarily  upon a v a i l a b i l i t y of s u b j e c t s i n s t e a d of randomization. T h i s e a s i l y produces a b i a s e d sample w i t h h i g h e r homogeneity than i t s p o p u l a t i o n . A homogeneous sample w i l l underestimate the p r e d i c t i v e v a l i d i t y of TOEFL s c o r e s (Pedhazur,  1982;  Cohen,  1983) . Predictor variables There are d i f f e r e n t k i n d s of TOEFL s c o r e s used i n TOEFL p r e d i c t i o n s t u d i e s . T h i s v a r i a t i o n i n s e l e c t i o n of p r e d i c t o r v a r i a b l e s i n f l u e n c e s the e s t i m a t i o n of TOEFL's p r e d i c t i v e v a l i d i t y . Some examples are g i v e n as f o l l o w s . Firstly,  some s t u d i e s o n l y used TOEFL t o t a l s c o r e s  (Johnson, 1988;  L i g h t & Wan,  1991),  some used the TOEFL  s e c t i o n a l s c o r e s , o t h e r s used t o t a l and s e c t i o n a l s c o r e s separately  (Kwang & Dizney, 1970;  L i g h t , Xu & Mossop,  1987;  30  Hu, 1991) and a few used a combination o f TOEFL t o t a l and  scores  s e c t i o n a l s c o r e s as one p r e d i c t o r . The p r e d i c t i v e  v a l i d i t y v a r i e s on t h e b a s i s o f s i n g l e scores  o r composite  s c o r e s o f TOEFL. Secondly, from 1963 t o 1976 TOEFL c o n s i s t e d o f f i v e s u b t e s t s . The f i v e - s e c t i o n TOEFL had 200 t o t a l  items and  r e q u i r e d two hours and 20 minutes o f a d m i n i s t r a t i o n  time.  Some p r e d i c t i o n s t u d i e s examined t h e f i v e - s e c t i o n TOEFL (Harcey, 1979; B o s i t c , 1981; Stover,  1982). The c u r r e n t  t h r e e - s e c t i o n TOEFL c o n s i s t s o f 150 items and r e q u i r e s one hour and 45 minutes o f a c t u a l t e s t i n g time. Some s t u d i e s explored  the p r e d i c t i v e v a l i d i t y of the t h r e e - s e c t i o n  TOEFL(Martin, 1971; Sharon, 1972; Shay, 1975; Riggs, 1982). Because o f d i f f e r e n c e s i n s e c t i o n c o n s t r u c t i o n  items  i n c l u d e d and time a l l o c a t e d f o r t h e two forms o f TOEFL, s p e c i a l c a u t i o n has t o be taken when one compares t h e p r e d i c t i v e v a l i d i t y o f TOEFL scores d i f f e r e n t t e s t administrations. unfortunately  ignored  obtained  over time from  However, t h i s was  i n some TOEFL p r e d i c t i o n s t u d i e s  (e.g., Odunze, 1982). T h i r d l y , E n g l i s h language a b i l i t y can be a f f e c t e d over a s h o r t p e r i o d o f time by a d d i t i o n a l t r a i n i n g o r l a c k o f p r e - t e s t p r a c t i c e (ETS, 1994a). Thus ETS s e t a r u l e t h a t a TOEFL score r e p o r t w i l l o n l y be v a l i d f o r two y e a r s . However, even w i t h i n two years t h e range o f time t o take TOEFL i s s t i l l  important. The r e s e a r c h  l i t e r a t u r e documented  the TOEFL p r e d i c t i o n s t u d i e s w i t h a v a r i e t y o f t i m i n g s ,  such  31  as summer TOEFL s c o r e s a f t e r a r r i v a l  i n the USA  Holdaway, 1973), p r e - i n s t r u c t i o n TOEFL s c o r e s P i t c h e r , 1970), and pre-study  TOEFL s c o r e s  1991). Most s t u d i e s used pre-admission Heil,  & Aleamoni, 1974;  &  (Schrader  (Light &  &  Wan,  TOEFL s c o r e s  (e.g.,  Ayers & P e t e r s , 1977), except  study u s i n g a f t e r - a d m i s s i o n TOEFL s c o r e s 1985). I t i s important  (Gue  a  (Ho & Spinks,  t o note t h a t the time l a p s e between  the c o l l e c t i o n of p r e d i c t o r s c o r e s and the c o l l e c t i o n of the c r i t e r i o n scores w i l l  impact the p r e d i c t i v e v a l i d i t y  TOEFL s c o r e s . Furthermore, p r e - and post-admission  of  scores  a f f e c t s i g n i f i c a n t l y the degree of homogeneity of the sample. The  former w i l l be much more heterogeneous, and  the  l a t t e r w i l l r e s u l t i n a f a i r l y s e l e c t i v e sample. Besides v a r i o u s forms of TOEFL s c o r e s , many TOEFL p r e d i c t i o n s t u d i e s used other language t e s t s c o r e s ,  obtained  from such s t a n d a r d i z e d or l o c a l t e s t s as Lado T e s t B and (Chase & S t a l l i n g s ,  1966), the Pennstat  C  (Chase & S t a l l i n g s ,  1966), T e s t of the American Language I n s t i t u t e a t Georgetown University Proficiency  (AACRAO, 1971), Michigan (MTELP) (Pack, 1972;  Placement Examination  T e s t of E n g l i s h language  Abadzi,  (Heil & Alaemini,  g e n e r a l t e s t ' s verb s u b t e s t  1976), the E n g l i s h 1974), the  (GRE-V) (Ayers & P e t e r s , 1977),  and Wechsler A d u l t I n t e l l i g e n c e S c a l e ' s form R subtest  GRE  Vocabulary  (WAIS-R-V) (Hassen, 1982), as p r e d i c t o r s . Other  s t u d i e s used w r i t i n g s c o r e s and  interview scores  (Gue  Holdaway, 1973), ESL course average grade (Stover, South, 1992)  as the p r e d i c t o r of academic  success.  &  1982;  32  Q u i t e a few s t u d i e s have used non-language p r e d i c t o r s , such as GRE-Q (Ayers & P e t e r s , 1977; Yule & Hoffman, 1990; Ayers & Quattlebaum, 1992); h i g h s c h o o l GPA, age, y e a r s out of s c h o o l , r e s i d e n t s t a t u s , c u l t u r a l background  (Andalib,  1976) ; WAIR-R-V, SAT (Wilox, 1975); n a t i v e language, major area o f study (Stove, 1982); r a t i n g s o f q u a l i t y o f academic performance (AACRAO, 1971); and LAST (Schrader & P i t c h e r , 1970) . Criterion variables S e l e c t i o n of c r i t e r i o n v a r i a b l e s i s a c r u c i a l but d i f f i c u l t t a s k i n d e s i g n i n g a p r e d i c t i o n study. Although i t has always been c r i t i c i z e d still  (e.g., Graham, 1987), GPA i s  t h e most f r e q u e n t l y used c r i t e r i o n v a r i a b l e . T h i s i s  l a r g e l y because (a) i t i s t h e most t y p i c a l  ( i f not p e r f e c t )  i n d i c a t o r o f academic success (Wimberley, McCloud, & F l i n n , 1992) ; (b) i t i s t h e most r e a d i l y a c c e s s i b l e c r i t e r i a f o r academic achievement ( L i g h t , Xu & Mossop, 1987) ; and (c)  i t  i s r e l a t i v e l y w e l l - d e f i n e d and w i d e l y understood (Young, 1993) . However, d i f f e r e n t v e r s i o n s o f GPA have been seen i n the r e s e a r c h  literature.  Types o f GPAs i n terms o f a p e r i o d o f time i n c l u d e : • F i r s t - t e r m GPA (Pack, 1972; Stove, 1982; Wilcox, 1975; L i g h t & Wan, 1991; L i g h t , Xu & Mossop, 1987; Kwang & Dizney, 1970); • F i r s t - and second-term GPA (Abdzi, 1967; Harvey, 1979; H e l l & Aleamoni, 1974; M a r t i n , 1971; Odunze, 1982) ;  33  • F i r s t - y e a r GPA (Chase and S t a l l i n s ,  1966; Riggs,  1982; AACRAC, 1971; Gue & Holdaway, 1973; Schrader & Pitch,  1970);  • F i r s t - y e a r , one-and-half-year, and two-year GPA (Yule and Hoffman, 1990); • G r a d u a t i o n GPA (Ayers & P e t e r s , 197 7); Ayers & Quattlebaum,  1992);  • GPA o b t a i n e d from unreported o r unable t o i d e n t i f i e d terms  (Hassen, 1982; A n d a l i b , 1976;  Sharon, 1972; Hu, 1991; Johnson,  1988).  Types o f GPAs i n terms o f d i f f e r e n t p o i n t  systems  include: • F o u r - p o i n t GPA (AACRAO, 1971; M a r t i n , 1971; L i g h t & Wan, 1990; Ayers & P e t e r s , • F i v e - p o i n t GPA (Andalib,  1977);  1976);  • N i n e - p o i n t GPA (Gue & Holdaway, 1973) ; • Percentage GPA (UBC, 1993); • L e t t e r grade GPA (UBC, 1993). Other c r i t e r i a used i n c l u d e : • Numbers o f c r e d i t hours (Shay, 1975; Abdzi, 1967; Johnson,  1988);  • Average o f 12 c r e d i t s s u c c e s s f u l completed ( L i g h t & Wan,  1991);  • V e r b a l - and nonverbal-course GPA ( B o s t i c , • Academic index, a d v i s o r ' s r a t i n g  1981);  (AACRAO, 1971);  • E v e n t u a l TA recombination (Yule & Hoffman, 1990); • Average accumulated c r e d i t p e r semester  34  (Christopher,  1993).  Besides t h e above, as an index t o academic achievement GPA a l s o v a r i e s i n s e c t i o n s ,  courses, i n s t r u c t o r s , majors,  y e a r s , programs, and i n s t i t u t i o n s , as w e l l as c o u n t r i e s . Various versions the e s t i m a t i o n  o f GPA w i l l have a s i g n i f i c a n t impact on  o f TOEFL's p r e d i c t i v e v a l i d i t y .  Therefore,  each TOEFL p r e d i c t i o n study should s p e c i f y and j u s t i f y what k i n d o f GPA i s used. Result  interpretation  V a r i o u s ways o f i n t e r p r e t i n g r e s u l t s a r e another source of i n c o n s i s t e n c y  i n research  f i n d i n g s o f TOEFL's p r e d i c t i v e  v a l i d i t y . There were two c o n s i s t e n t  problems  r e s u l t i n t e r p r e t a t i o n i n t h e TOEFL p r e d i c t i o n F i r s t , t h e r e have been n e i t h e r c o n s i s t e n t  regarding studies. standards nor  c o n v e n t i o n a l terminology used t o e v a l u a t e whether a measure of t h e TOEFL's p r e d i c t i v e v a l i d i t y i n a study i s h i g h o r low.  Some s t u d i e s claimed t h a t TOEFL s c o r e s were a u s e f u l ,  reliable,  s i g n i f i c a n t , meaningful, adequate, strong, and  s a t i s f a c t o r y p r e d i c t o r o f GPA r e s p e c t i v e l y Stallings,  (Chase &  1966; ; Hwang & Dizney, 1970; Shay, 1975; Ayers &  P e t e r , 1977; S o k a r i ,  1981; Odunze, 1982; Ayers &  Quattlebaum, 1992). On t h e c o n t r a r y ,  other studies  declared  t h a t TOEFL s c o r e s were o f l i m i t e d , d o u b t f u l , and q u e s t i o n a b l e v a l u e i n p r e d i c t i n g GPA r e s p e c t i v e l y 1979;  Bostic,  (Harvey,  1981). O b v i o u s l y here, what was meant by  useful, doubtful,  or other modifiers  was r a t h e r vague and  s u b j e c t i v e . As a matter o f f a c t , what i s deemed u s e f u l ,  35  e f f e c t i v e , or s t r o n g by one r e s e a r c h e r may u s e l e s s , i n e f f e c t i v e , or weak by another  be deemed  r e s e a r c h e r o r by  the same one a t another c o n t e x t . For example, based on the research finding Mossop, 1987)  (r=.14, p_<.05),  o  n  e  study  ( L i g h t , Xu &  a s s e r t e d t h a t the c o r r e l a t i o n was  too low t o  have any p r a c t i c a l s i g n i f i c a n c e and t h e r e f o r e TOEFL was  not  an e f f e c t i v e p r e d i c t o r of academic success. However, the r e s e a r c h e r s e x p l a i n e d n e i t h e r why  an r of .14 was  too  low  nor what standards were used t o r e j e c t the TOEFL's p r e d i c t i v e v a l i d i t y i n the study. Thus, t h e i r c o n c l u s i o n i s arbitrary. To determine the s t r e n g t h , importance, meaningfulness  and  of f i n d i n g s , an estimate of e f f e c t  size  i n s t e a d of t e s t i n g o f s t a t i s t i c a l s i g n i f i c a n c e i s g e n e r a l l y recommended  (Cohen, 1988;  Pedhazur & Schmelkin,  (1988) proposed conventions  1991). Cohen  f o r s m a l l , medium, and l a r g e  effect sizes for correlation coefficients, regression c o e f f i c i e n t s , and d i f f e r e n c e s between means. The  r e s u l t s of  TOEFL p r e d i c t i o n s t u d i e s should be i n t e r p r e t e d a c c o r d i n g t o w e l l - e s t a b l i s h e d standards d e f i n i t i o n s on R  2  l i k e Cohen's c o n v e n t i o n a l  (1988) t o a v o i d s u b j e c t i v e n e s s and  arbitrariness. The  second problem i s about what estimates should  be  used t o judge the r e l a t i v e importance among p r e d i c t o r v a r i a b l e s . Some s t u d i e s concluded  t h a t TOEFL s c o r e s were a  b e t t e r , h i g h e r , b e s t , s t r o n g e s t , or lower p r e d i c t o r by comparing the s c o r e s w i t h o t h e r p r e d i c t o r s i n terms of  36  c o r r e l a t i o n c o e f f i c i e n t s or regression  c o e f f i c i e n t s obtained  (Wilcox, 1975; Chase & S t a l l i n g s ; Ho & Spinks, 1985; AACRAO, 1971). These s t u d i e s  judged t h e p r e d i c t o r s '  relative  importance on t h e b a s i s o f s i g n i f i c a n c e t e s t r e s u l t s on improper e s t i m a t e s such as r o r R . C o n v e n t i o n a l l y , change 2  in R  2  o r squared s e m i p a r t i a l  c o r r e l a t i o n i s recommended t o  e s t i m a t e t h e r e l a t i v e importance among p r e d i c t o r (Pedhazur, 1982; Tabachnick & F i d e l l ,  variables  1989).  Summary The  l i t e r a t u r e review i n t h i s chapter h e l p s t o b u i l d  both a c o n c e p t u a l and a m e t h o d o l o g i c a l bases f o r e s t i m a t i n g the p r e d i c t i v e v a l i d i t y o f TOEFL scores on G P A . Theoretically, s o c i a l , educational,  i t was r e v e a l e d  t h a t numerous f a c t o r s i n  l i n g u i s t i c , psychological,  p h y s i o l o g i c a l domains i n f l u e n c e  and  academic achievement. There  i s no one s i n g l e f a c t o r which can f u l l y determine academic success o r f a i l u r e . The unique c o n t r i b u t i o n  o f any s i n g l e  f a c t o r t o academic achievement should be examined i n comparison w i t h o t h e r r e l e v a n t investigate i t within  f a c t o r s . Therefore, t o  TOEFL's p r e d i c t i v e v a l i d i t y , one should c o n s i d e r  a comprehensive context which i n c l u d e s  both  language f a c t o r s and non-language f a c t o r s . For language f a c t o r s , one should f u r t h e r c o n s i d e r d i f f e r e n t language proficiency  (e.g., CALP, BICS), language s k i l l s  (e.g.,  l i s t e n i n g , writing), or l i n g u i s t i c r e g i s t e r s i n subject areas (e.g., mathematics,  science).  37  From a methodological  p e r s p e c t i v e , t h e r e are f i v e major  a s p e c t s which have s i g n i f i c a n t e f f e c t s on the e s t i m a t i o n o f TOEFL's p r e d i c t i v e v a l i d i t y : a n a l y t i c a l models, s u b j e c t variables,  predictors,  They deserve  c r i t e r i a , and r e s u l t  special attention  interpretation.  i n the r e s e a r c h d e s i g n i n  o r d e r t o ensure s a t i s f a c t o r y r e s e a r c h v a l i d i t y .  38  Chapter Three Method T h i s c h a p t e r o u t l i n e s the method of the p r e s e n t study, i n c l u d i n g the program s e t t i n g , p a r t i c i p a n t s , the p r e d i c t o r v a r i a b l e s , t h e c r i t e r i o n v a r i a b l e , the a n a l y t i c a l model, operational  d e f i n i t i o n s o f p r e d i c t i v e v a l i d i t y , and r e s e a r c h  hypotheses. The program  setting  The UBC/Ritsumeikan  Academic  1991 based upon an agreement  Exchange Program began i n  f o r the e s t a b l i s h m e n t o f an  i n t e r n a t i o n a l academic exchange between the U n i v e r s i t y o f B r i t i s h Columbia and Ritsumeikan U n i v e r s i t y .  I t i s the  l a r g e s t exchange program o f t h i s type i n North America. The program operates on an eight-month b a s i s . Each y e a r about 100 undergraduate students from Ritsumeikan U n i v e r s i t y study a t UBC  from September t o A p r i l as a p a r t of t h e i r f o u r - y e a r  u n i v e r s i t y e d u c a t i o n . A f t e r t h a t , they go back t o c o n t i n u e t h e i r studies  i n Japan.  Ritsumeikan U n i v e r s i t y was o r i g i n a l l y founded i n 1869 by Japanese P r i n c e S a i o n j i Kinmochi and i s one of t h e p r i v a t e u n i v e r s i t i e s i n Japan. I t p r e s e n t l y  comprises seven  C o l l e g e s and seven Graduate Schools i n Law,  Economics,  Business A d m i n i s t r a t i o n , Relations,  S o c i a l Sciences,  International  L e t t e r s , S c i e n c e and E n g i n e e r i n g . The  total  enrollment o f s t u d e n t s i n the 1992-93 academic y e a r was about 25,000, of which undergraduate s t u d e n t s were over 23,000 (Ritsumeikan U n i v e r s i t y ,  1993). The r a t i o o f s u c c e s s  3 9  i n a p p l i c a t i o n f o r admission i n t o the U n i v e r s i t y i s about 1:20. A p p l i c a n t s t o the UBC/Ritsumeikan Academic  Exchange  Program are r e q u i r e d t o submit t h e i r academic r e c o r d s , TOEFL o f f i c i a l s c o r e r e p o r t s , as w e l l as w r i t i n g samples i n E n g l i s h f o r e v a l u a t i o n . To h e l p a p p l i c a n t s p r e p a r e f o r w r i t i n g TOEFL, Ritsumeikan U n i v e r s i t y p r o v i d e s TOEFL p r e p a r a t i o n workshops. Based upon both academic a p t i t u d e and E n g l i s h p r o f i c i e n c y , Ritsumeikan U n i v e r s i t y s e l e c t s about 100 q u a l i f i e r s i n t o the program from the p o o l of a p p l i c a n t s i n second- and t h i r d - y e a r courses. The program p r o v i d e s a c o n t e n t - o r i e n t e d c u r r i c u l u m w i t h an emphasis on c r o s s - c u l t u r a l communication. The  instructors  are from t h e Department o f Language E d u c a t i o n a t UBC. E n g l i s h i s used as the medium o f i n s t r u c t i o n . At the b e g i n n i n g o f the program, the students are grouped i n t o  five  c l a s s e s . Each c l a s s i n c l u d e d about 20 students w i t h one t e a c h i n g a s s i s t a n t . They are r e q u i r e d t o complete s i x t h r e e c r e d i t courses i n one academic year, t h r e e t h r e e - c r e d i t courses f o r each term. A l l o f them take courses o f f e r e d by the Department of Language E d u c a t i o n i n the f i r s t term. The courses o f f e r e d i n 1993-94 i n c l u d e d : Communication  Intercultural  i n Second Language Education,  Communication  S k i l l s i n E d u c a t i o n a l S e t t i n g s , Academic D i s c o u r s e i n Second Language E d u c a t i o n , and Second Language E d u c a t i o n Practicum. In the second term, those whose TOEFL t o t a l s c o r e s meet the UBC minimum requirement o f 550 may  attend regular  UBC  40  c l a s s e s i n t h e F a c u l t y o f A r t s and o t h e r f a c u l t i e s f o r which they have p r e - r e q u i s i t e s . The c r e d i t s t h e students earn a t 4  UBC a r e t r a n s f e r a b l e t o t h e i r home u n i v e r s i t y . A l l t h e program students l i v e i n t h e UBC/Ritsumeikan House on t h e campus o f UBC. P a i r s share an apartment w i t h two  Canadian  roommates. I n a d d i t i o n t o d a i l y  life  e x p e r i e n c e , f i e l d s t u d i e s , a buddy program, and o t h e r programs a r e arranged t o enhance t h e s t u d e n t s c u l t u r a l understanding o f Canadian  cross-  1  s o c i e t y . The s t u d e n t s a r e  a l s o i n v o l v e d i n s o c i a l a c t i v i t i e s on and o f f campus, such as a seminar  s e r i e s by t h e UBC P a c i f i c Rim Club and  v o l u n t e e r work a t p r e s c h o o l s . Participants The t a r g e t p o p u l a t i o n o f t h e study was t h e UBC/Ritsumeikan Academic Exchange Program s t u d e n t s . The sample was a t o t a l o f 97 students who e n r o l l e d i n t h e 19931994  program. Among them, 46 students were male and 52  female. The range i n age was from 20 t o 2 3 y e a r s o l d , except one s e n i o r student aged over 60. They were a l l second undergraduate  year  students a t Ritsumeikan U n i v e r s i t y . N i n e t y  f i v e s t u d e n t s majored i n t h e humanities and s o c i a l s c i e n c e s such as Law, Business A d m i n i s t r a t i o n , I n t e r n a t i o n a l R e l a t i o n , Economics, and E n g l i s h L i t e r a t u r e , and o n l y two i n E n g i n e e r i n g . Japanese i s t h e i r f i r s t language  and most o f  A c c o r d i n g t o W i l l i a m McMichael (W. McMichael. p e r s o n a l communication, A p r i l , 1995), c u r r e n t academic c o o r d i n a t o r o f the program, t h e 1994-95 program has a d j u s t e d i t s c u r r i c u l u m s t r u c t u r e and t h e 1995-96 program w i l l have l a r g e r changes.  41  them had not y e t experienced  s t u d y i n g and/or s t a y i n g i n  North America b e f o r e t h e program. Most were a t t h e i n t e r m e d i a t e l e v e l i n E n g l i s h p r o f i c i e n c y . T h e i r TOEFL t o t a l s c o r e mean was 515.96 w i t h a standard d e v i a t i o n o f 26.03. I t was e v i d e n t t h a t t h e sample was q u i t e homogeneous i n terms o f age, n a t i v e language, country o f o r i g i n ,  cultural  background, major f i e l d s , and E n g l i s h p r o f i c i e n c y . The p r e d i c t o r v a r i a b l e s Seven p r e d i c t o r v a r i a b l e s were used i n t h e study based upon s u i t a b i l i t y t o t h e r e s e a r c h q u e s t i o n and a v a i l a b i l i t y i n t h e UBC/Ritsumeikan Academic Exchange program. These p r e d i c t o r v a r i a b l e s were: TOEFL t o t a l s c o r e s , TOEFL s e c t i o n I s c o r e s , TOEFL s e c t i o n I I s c o r e s , TOEFL s e c t i o n I I I s c o r e s , o r a l i n t e r v i e w s c o r e s , w r i t i n g sample s c o r e s , and gender. TOEFL s c o r e s , i n c l u d i n g t o t a l s c o r e s and t h r e e subscores,  served as p r e d i c t o r v a r i a b l e s i n t h e study.  These  s c o r e s were o b t a i n e d from t h e d i f f e r e n t TOEFL a d m i n i s t r a t i o n s a t Ritsumeikan U n i v e r s i t y through t h e I n s t i t u t i o n T e s t i n g Program (ITP)  from January through May  of 1993 when t h e students a p p l i e d f o r admissions  into the  program. Two t h i n g s should be noted.  F i r s t , t h e h i g h e s t TOEFL  s c o r e f o r each student was used i n t h e study. Most  students  i n t h e program wrote t h e TOEFL r e p e a t e d l y i n d i f f e r e n t a d m i n i s t r a t i o n s . There a r e t h r e e f r e q u e n t l y seen a l t e r n a t i v e s f o r use o f TOEFL s c o r e s f o r admissions: t h e h i g h e s t TOEFL score, t h e l a t e s t TOEFL score, or t h e average  42  TOEFL s c o r e . Both Ritsumeikan U n i v e r s i t y and  UBC  c o n s i s t e n t l y used a p p l i c a n t s ' h i g h e s t TOEFL s c o r e s t o e v a l u a t e E n g l i s h language p r o f i c i e n c y f o r program  admission.  Second, TOEFL s c o r e s were obtained through the I n s t i t u t i o n T e s t i n g Program r a t h e r than the r e g u l a r F r i d a y and Saturday  T e s t i n g Programs.  5  ETS  s t a t e s t h a t TOEFL s c o r e s  under the ITP are not a c c e p t a b l e f o r o f f i c i a l  admission  purposes. However, the study had t o use the ITP TOEFL s c o r e s because they were the o n l y a l t e r n a t i v e Ritsumeikan U n i v e r s i t y administered  f o r the program a p p l i c a n t s . UBC  Ritsumeikan U n i v e r s i t y agreed program admissions Manual (ETS, discussions 1994  & 1995)  and  t o use ITP TOEFL s c o r e s f o r  purposes. A c c o r d i n g t o the TOEFL T e s t  1992), the ITP Manual (ETS, (Kantor, R. N.,  1994f),  p e r s o n a l e-mail  and  communications,  between the author of the c u r r e n t t h e s i s and  Dr. Kantor, D i r e c t o r of TOEFL Program O f f i c e , the ITP TOEFL s c o r e s are s t i l l  c o n s i d e r e d s u b s t a n t i a l l y v a l i d and  are  comparable t o s c o r e s earned under the r e g u l a r programs. Two  o t h e r E n g l i s h p r o f i c i e n c y measurements were  a v a i l a b l e i n the program and used as p r e d i c t o r v a r i a b l e s . They were the September o r a l speaking  s c o r e s and  the  September w r i t i n g sample s c o r e s . The purpose of those  two  There are two d i f f e r e n t kinds of TOEFL t e s t i n g programs a c c o r d i n g t o the TOEFL T e s t and Scores Manual (ETS, 1992). The o f f i c i a l TOEFL t e s t i n g programs, i n c l u d i n g F r i d a y and Saturday t e s t i n g programs, are a d m i n i s t r a t e d i n t e r n a t i o n a l l y i n the TOEFL t e s t i n g c e n t e r s . The I n s t i t u t i o n T e s t i n g Program, whose items were p r e v i o u s l y used i n the o f f i c i a l t e s t i n g programs, i s a d m i n i s t r a t e d at l o c a l i n s t i t u t e s around the world. 5  43  measurements was writing s k i l l  t o e v a l u a t e E n g l i s h speaking s k i l l  r e s p e c t i v e l y before i n s t r u c t i o n started, while  TOEFL does not p r o v i d e d i r e c t i n f o r m a t i o n speaking s k i l l s . The  v a l i d i t y and  measurements have not been The  on w r i t i n g  reliability  p l u s marks. See C). The  11-point s c a l e  took  received  p r o f i c i e n c y interview The  and  pre-interview  3 & Appendix  (See  by  Berwick &  p. 2 & Appendix B). Each student  was  assessment of the w r i t i n g samples f o l l o w e d  In a d d i t i o n t o the Gender was  and  five  training in oral  6-point s c a l e  of the TOEFL Test of W r i t t e n E n g l i s h  GPA.  the  t o w r i t e an essay on d e s i g n a t e d t o p i c s w i t h i n  minutes. The  foregoing  90 that  6  language-based  predictors,  i n the  study.  variable  c r i t e r i o n v a r i a b l e i n the  I t was  (TWE).  used as a non-language p r e d i c t o r  criterion The  p.  September w r i t i n g sample s c o r e s were g i v e n  McMichael, 1993,  The  r a t e d by  r a t i n g procedure.  t r a i n e d r a t e r s based on a 1-6  required  20  (including extra  Berwick & McMichael, 1993,  interviewers  two  reported.  minutes. Each student's o r a l performance was on a 0-5  and  of these  September o r a l p r o f i c i e n c y i n t e r v i e w  interviewers  and  study was  the f i r s t - t e r m  c a l c u l a t e d on the b a s i s of a percentage g r a d i n g  c r e d i t w e i g h t i n g system  7  which UBC  adopted i n 1991.  At  Both s c o r e s of the September o r a l p r o f i c i e n c y i n t e r v i e w and s c o r e s of the September w r i t i n g sample w i l l be l a b e l e d as speaking s c o r e s and w r i t i n g s c o r e s r e s p e c t i v e l y i n the following text. The c o n v e r t i b i l i t y among d i f f e r e n t g r a d i n g and c r e d i t w e i g h t i n g systems i n North America i s beyond the scope of 6  44  UBC,  course weight  i s expressed i n c r e d i t s . In g e n e r a l  one  c r e d i t r e p r e s e n t s one hour of i n s t r u c t i o n or two t o t h r e e hours o f l a b o r a t o r y work per week throughout  one  term.  Courses are normally graded on a percentage b a s i s w i t h a c o r r e s p o n d i n g l e t t e r grade a s s i g n e d (UBC, The f i r s t term GPA grades  1993).  i n c l u d e d the average  percentage  i n t h r e e c o u r s e s . They were: EDUC395A, Second  Language E d u c a t i o n Practicum; EDUC490A, R e g i o n a l S t u d i e s In Second Language E d u c a t i o n ; and ENED379, C r o s s c u l t u r a l S t u d i e s i n Second Language E d u c a t i o n . Each was  a three  c r e d i t course. I n s t r u c t o r s i n the Department of Language E d u c a t i o n taught the courses and assessed academic achievement. The course grades were g i v e n based upon a s e t of s p e c i f i c c r i t e r i a o u t l i n e d i n the v a r i o u s course a t the b e g i n n i n g of the term Berwick  & McMichael,  (Berwick & McMichael,  1993). Table 3.1  syllabi 1992;  shows t h a t the s e t o f  c r i t e r i a mainly p l a c e d weights on w r i t t e n t a s k s t o e v a l u a t e s t u d e n t s ' academic achievement. A n a l y t i c model T h i s study used m u l t i p l e r e g r e s s i o n a n a l y s i s as the a n a l y t i c model. The study focused on the e s t i m a t e of the p r e d i c t i v e v a l i d i t y of TOEFL s c o r e s on GPA.  Hence the  r e g r e s s i o n model i s a p p r o p r i a t e and i n t i m a t e l y r e l a t e d t o the primary g o a l of the study. Furthermore, of the r e s e a r c h problem  the complexity  under study r e q u i r e d a  the p r e s e n t study. For d e t a i l e d d i s c u s s i o n on t h i s i s s u e see Cohen & Cohen (1983) and Pedhazur (1982).  45  T a b l e 3.1 Grade c r i t e r i a on d i f f e r e n t a s p e c t s i n the t h r e e courses EDUC395A  (%)  EDUC490A  F i e l d work j o u r n a l  20  20  Oral presentation  20  15  Term paper  20  (%)  ENED379  (%)  10 20  Lab work Final  Examinations  Assignments  30  Progress e v a l u a t i o n s Bibliography Literature  25  15  30  35  30 10  review  10  Participation TOTAL  10 100  100  100  powerful a n a l y t i c t o o l . As a h i g h l y g e n e r a l and v e r y f l e x i b l e d a t a - a n a l y t i c system (Cohen & Cohen, 1983), t h e r e g r e s s i o n model, p a r t i c u l a r l y t h e m u l t i p l e r e g r e s s i o n model, can be a p p l i e d t o i n v e s t i g a t e v a r i o u s f a c t o r s  related  t o t h e p r e d i c t i v e power o f t h e TOEFL s c o r e . The data o f t h e study was processed w i t h SPSS f o r Windows  (Release 6.0).  Operational d e f i n i t i o n s of predictive v a l i d i t y . The p r e s e n t study u t i l i z e d change i n squared R (AR ) as 2  the e s t i m a t o r t o assess p r e d i c t i v e v a l i d i t y . A c c o r d i n g t o Cohen's c o n v e n t i o n a l d e f i n i t i o n s .02,  (Cohen, 1988, pp. 412-414),  .13, and .26 a r e r e s p e c t i v e l y d e f i n e d as s m a l l , medium,  46  and l a r g e e f f e c t s i z e o f R . 2  Based on Cohen's d e f i n i t i o n s ,  the p r e s e n t study d e f i n e d f o u r o p e r a t i o n a l l e v e l s o f predictive validity  Table  (see Table 3.2).  3.2  Four l e v e l s o f P r e d i c t i v e V a l i d i t y  Level  R  Predictive v a l i d i t y  2  1.  .000 -  .019  Negligible  2.  .020 -  .129  Small  3.  .130 -  .259  Medium  4.  .260 -  1.00  Large  Research Hypotheses The p r e s e n t study advanced the f o l l o w i n g r e s e a r c h hypotheses f o r t e s t i n g : 1. TOEFL t o t a l s c o r e s have p r e d i c t i v e v a l i d i t y on term's GPA  f o r the UBC/Ritsumeikan Exchange  first  Program  students. 2. TOEFL s e c t i o n a l s c o r e s have p r e d i c t i v e v a l i d i t y on f i r s t term's GPA  f o r the UBC/Ritsumeikan Exchange  Program  students. 3. W r i t i n g s c o r e s have p r e d i c t i v e v a l i d i t y on term's GPA students.  f o r the UBC/Ritsumeikan Exchange  Program  first  47  4.  Speaking  s c o r e s have p r e d i c t i v e v a l i d i t y on  first  term's GPA f o r the UBC/Ritsumeikan Exchange Program students. 5. Gender has p r e d i c t i v e v a l i d i t y on f i r s t term's GPA for  t h e UBC/Ritsumeikan Exchange Program s t u d e n t s .  Summary T h i s chapter d e l i n e a t e d the r e s e a r c h d e s i g n o f the p r e s e n t study. P a r t i c i p a n t s were 9 7 Japanese exchange s t u d e n t s . The study employed a m u l t i p l e l i n e a r r e g r e s s i o n model t o analyze the r e l a t i o n s h i p s o f TOEFL s c o r e s and o t h e r p r e d i c t o r v a r i a b l e s t o f i r s t term's GPA.  Four o p e r a t i o n a l  l e v e l s o f p r e d i c t i v e v a l i d i t y were d e f i n e d f o r r e s u l t i n t e r p r e t a t i o n . The study t e s t e d f i v e r e s e a r c h  hypotheses.  48  Chapter  Four  Results T h i s chapter summarizes treatment  o f the m i s s i n g data,  s t e p s taken t o check f o r v i o l a t i o n of assumptions, and an a n a l y s i s o f t h e d e s c r i p t i v e data. The chapter p r e s e n t s t h e main f i n d i n g s o f a m u l t i p l e r e g r e s s i o n a n a l y s i s . Treatment of t h e m i s s i n g data An examination  o f t h e data f i l e used i n the p r e s e n t  study showed t h a t t h e r e were t h r e e cases w i t h m i s s i n g v a l u e s and one case which had a s u s p i c i o u s v a l u e on t h e TOEFL section II score. S i n c e only t h r e e m i s s i n g - v a l u e cases were found  from a  sample o f 97, t h e r e were v e r y few chances t h a t a s y s t e m a t i c p a t t e r n e x i s t e d among t h e m i s s i n g - v a l u e c a s e s . In o t h e r words, t h e r e were reasons t o b e l i e v e t h a t the m i s s i n g v a l u e s for  t h e v a r i a b l e s o c c u r r e d randomly. T h e r e f o r e , t h e l i s t w i s e  m i s s i n g - v a l u e treatment treatment  was employed i n t h e study. T h i s  keeps a l l v a r i a b l e s but e l i m i n a t e s t h e m i s s i n g -  v a l u e c a s e s . I t i s a l s o the d e f a u l t f o r the m i s s i n g - v a l u e treatment  i n t h e SPSS f o r Window program. Three cases, two  w i t h m i s s i n g v a l u e s i n speaking s c o r e s and one i n GPA, were e l i m i n a t e d from t h e data  file.  For case 40, t h e TOEFL t o t a l score was 57 0, w i t h t h r e e s e c t i o n a l s c o r e s 50, 68, and 53 r e s p e c t i v e l y . The TOEFL s e c t i o n I I score was s u s p i c i o u s . Note t h a t 68 i s t h e maximum s c o r e i n S e c t i o n I I . I t was almost  i m p o s s i b l e t o reach i t  w h i l e t h e other s e c t i o n a l s c o r e s were around 50. I t was a l s o  49 found t h a t i n a TOEFL t e s t a d m i n i s t r a t e d about two months e a r l i e r than t h e c u r r e n t l y d i s c u s s e d t e s t t h e same person s c o r e d o n l y 513. I t was u n l i k e l y t h a t t h i s student would g a i n about 60 p o i n t s w i t h i n two months. Thus t h e s e c t i o n I I s c o r e o f 68 might be a data e n t r y e r r o r . S i n c e a l l t h e o r i g i n a l r e p o r t s o f TOEFL s c o r e s were a t Ritsumeikan U n i v e r s i t y i n Japan, i t was i m p o s s i b l e t o check t h i s p a r t i c u l a r TOEFL s c o r e . Case 42 was, t h e r e f o r e ,  excluded  from t h e data. Descriptive s t a t i s t i c a l analysis Means and standard d e v i a t i o n s o f a l l t h e v a r i a b l e s a r e shown i n Table 4.1 below. The mean TOEFL t o t a l s c o r e i n t h e study was 515.96 and s e c t i o n a l s c o r e s were 49.74, 53.58, and 51.48 r e s p e c t i v e l y . Standard  d e v i a t i o n (SD) o f t h e TOEFL  t o t a l s c o r e was 2 6.03.  Table 4.1 Means and standard d e v i a t i o n s o f a l l t h e v a r i a b l e s  GPA N M SD  93  GENDER  SPEAK  WRITE  TOEFL  SEC1  SEC2  SEC3  93  93  93  93  93  93  93  71.97  1.55  1.46  2.71  515.96  49.74  53.58  51.48  7.54  .50  .83  .83  26.03  4 .16  3.34  2.91  ETS r e p o r t e d t h a t based on t h e t o t a l o f 1,3 38,682 examinees t e s t e d between J u l y 1991 t o June 1993, t h e mean  50  TOEFL t o t a l score was  519.00 and  TOEFL t o t a l s c o r e s and  SD was  68.00. The  mean  the mean s e c t i o n a l s c o r e s were  490.00, 49.00, 50.00, and  48.00 r e s p e c t i v e l y f o r the  of the t e s t t a k e r s whose n a t i v e  group  language i s Japanese,(ETS,  1993) . Means of t h r e e groups (the t o t a l group, the group o f Japanese examinees, and similar  the  (519.00, 490.00, and  1993-94 program students) were 515.96), but  SDs  under study were almost t h r e e times s m a l l e r the t o t a l group. The the t o t a l group and the sample was  considerable  homogeneous i n terms of TOEFL  r a t h e r than t o a p p l i c a n t s  from the t r u e T a b l e 4.2 among the  i n SD  that  scores.  r e s t r i c t e d to the minimum  randomly  selected  population. below shows a Pearson C o r r e l a t i o n m a t r i x  variables.  When the c o r r e l a t i o n m a t r i x of v a r i a b l e s it  between  the sample under study i n d i c a t e s  a p p l i c a n t s whose TOEFL s c o r e s met  TOEFL score,  sample  than those of  difference  O b v i o u s l y , t h i s i s because the sample was successful  of the  i s obtained,  i s necessary t o perform an omnibus t e s t t o make sure  t h e r e i s an o v e r a l l s i g n i f i c a n t i n t e r r e l a t i o n e x i s t i n g among each p a i r of c o r r e l a t i o n s i n the matrix 1983,  p. 85  & pp.  (Cohen & Cohen,  315-316). I f t h e r e i s no o v e r a l l  s i g n i f i c a n t r e l a t i o n s h i p among the c o r r e l a t i o n , then  the  o v e r a l l r e l a t i o n s h i p i n the m a t r i x r e s u l t s from random sampling e r r o r r a t h e r than from the meaningful  association  51 T a b l e 4.2 Pearson c o r r e l a t i o n matrix o f t h e v a r i a b l e s  GPA  GPA  GENDER  1.000  GENDER  SPEAK  WRITE  SEC1  SEC2  SEC3  TOEFL  .457  .139  .342  . 284  .332  .193  . 365  . 000  . 184  . 001  . 006  . 001  .063  .000  1. 000  . 003  .283  .252  . 067  -.065  . 979  . 006  . 015  .520  .537  .185  .358  . 355  . 238  . 165  . 352  .000  .000  . 022  . 114  . 001  . 183  .297  . 068  .248  . 079  . 004  . 517  . 016  . 313  . 347  .794  . 002  . 001  . 000  .368  .731  .000  . 000  1. 000  SPEAK  WRITE  1.000  SEC1  1.000  SEC2  1. 0 0 0  SEC3  1.000  . 139  .713 . 000  TOEFL  1. 0 0 0  between each p a i r o f v a r i a b l e s . T h i s  i s almost t h e same as  performing an omnibus F - t e s t b e f o r e p o s t hoc t - t e s t s f o r means o f each group i n ANOVA. In t h e p r e s e n t study, a B a r t l e t t Chi-square t e s t was performed t o t e s t the o v e r a l l n u l l h y p o t h e s i s t h a t a l l p o s s i b l e sample c o r r e l a t i o n s among the s e t o f v a r i a b l e s i n the m a t r i x were zero. The r e s u l t r e j e c t s t h e n u l l (p_<.001). T h i s  hypothesis  indicates that there i s a s i g n i f i c a n t  52  i n t e r r e l a t i o n among the e n t i r e s e t o f Pearson c o r r e l a t i o n coefficients. Checking f o r v i o l a t i o n o f assumptions Each a n a l y t i c a l model, and t h e r e g r e s s i o n  such as the c o r r e l a t i o n model  model, has been developed based on  c e r t a i n e s s e n t i a l assumptions. I n t e l l i g e n t use o f a n a l y t i c a l models must meet the assumptions u n d e r l y i n g the models. Violations  o f assumptions l e a d t o e s t i m a t e b i a s e s .  Therefore, checking f o r v i o l a t i o n of underlying  assumptions  i s c o n s i d e r e d t o be an i n d i s p e n s a b l e component i n h e r e n t i n a regression  analysis.  The f o l l o w i n g  sections  w i l l discuss  (a)  two g e n e r a l assumptions u n d e r l y i n g any a n a l y t i c model, i . e . , the  assumption of s p e c i f i c a t i o n e r r o r s  measurement e r r o r s ; a regression influential  and the assumption o f  (b) s i x s p e c i f i c assumptions u n d e r l y i n g  a n a l y t i c model (Berry, 1993);  (c) o u t l i e r s and  points.  The assumption o f s p e c i f i c a t i o n e r r o r . T h i s assumption requires  t h a t an a n a l y t i c model should f l a w l e s s l y  reflect  i t s u n d e r l y i n g r a t i o n a l e r e g a r d i n g the e f f e c t of independent variables  on dependent v a r i a b l e s .  specification errors: i n t o the regression  There are t h r e e t y p e s o f  (a) o m i s s i o n of r e l e v a n t  model;  (b) i n c o r r e c t  s p e c i f i c a t i o n of  the manner i n which independent v a r i a b l e s dependent v a r i a b l e s variables  and  a f f e c t the  (c) i n c l u s i o n of i r r e l e v a n t  (Pedhazur & Schmelkin, 1991, pp.  Specification errors most s e r i o u s  threat  variables  389-390).  are the most damaging as they pose t h e to v a l i d interpretation  of  regression  53  r e s u l t s . However, i t i s d i f f i c u l t t o t e l l whether a l l r e l e v a n t v a r i a b l e s have been i n c l u d e d i n t h e model, i f a l l i r r e l e v a n t v a r i a b l e s have been excluded, o r whether t h e model has been c o r r e c t l y s p e c i f i e d i n t h e context s c i e n c e r e s e a r c h . The p r a c t i c a l way t o a v o i d  of s o c i a l  specification  e r r o r s i s t o use a well-grounded theory t o b u i l d an a n a l y t i c model. As Berry  (1993) has p o i n t e d out, people should  judge  r e g r e s s i o n models by whether these models conform t o t h e i r t h e o r i e s , and thus whether t h e models can be used t o answer t h e i r research questions  (P. 8 ) .  To reduce s p e c i f i c a t i o n e r r o r s i n t h e p r e s e n t  study,  the f o l l o w i n g e f f o r t s were made w i t h i n data and time constraints. 1. V a r i a b l e s were s e l e c t e d f o r a r e g r e s s i o n a n a l y s i s based upon knowledge about language and non-language f a c t o r s t h a t i n f l u e n c e academic achievement (see chapter present  two). The  study used language-based p r e d i c t o r s and a l s o  i n t r o d u c e d an e x p l o r a t o r y non-language v a r i a b l e , gender, into the regression. 2. The study focused  on a c c u r a t e l y e s t i m a t i n g t h e  unique c o n t r i b u t i o n o f TOEFL scores on GPA, r a t h e r than on measuring t h e e f f e c t s o f a l l t h e v a r i a b l e s i n t h e r e g r e s s i o n model. T h i s i s because t h e primary r e s e a r c h  interest i s to  know how w e l l TOEFL s c o r e s , as a s i n g l e p r e d i c t o r , can p r e d i c t GPA, not how much v a r i a n c e  i n GPA can be e x p l a i n e d .  Assumption o f measurement e r r o r s . T h i s  assumption  assumes t h a t a l l v a r i a b l e s under study a r e measured without  54  e r r o r . In r e a l i t y , t e s t s c o r e s unavoidably measurement e r r o r . Berry  include  (1993) has p r o v i d e d an e x t e n s i v e  d i s c u s s i o n o f t h r e e types o f measurement e r r o r s : random measurement e r r o r s , non-random measurement e r r o r s , and measurement e r r o r s i n v o l v i n g t h e use o f proxy v a r i a b l e s (pp. 49-60) . The p r e s e n t study d e a l t w i t h the i s s u e o f measurement e r r o r i n two ways: 1. Information about measurement o f i n d i c a t o r s , GPA, TOEFL s c o r e s , speaking s c o r e s , and w r i t i n g s c o r e s , was gathered. without  I t i s almost  i m p o s s i b l e t o perform  a measurement  e r r o r i n s o c i a l s c i e n c e r e s e a r c h . Information  a l l measures used i n t h e study were gathered  about  i n order t o  i d e n t i f y p o s s i b l e measurement e r r o r s . In the p r e s e n t  study,  both GPA and TOEFL s c o r e s a r e among the most f r e q u e n t l y  used  i n d i c a t o r s i n e d u c a t i o n a l p r a c t i c e , although t h e q u a l i t y o f these measures have been long debated. Both speaking  scores  and w r i t i n g s c o r e s a r e l o c a l l y used w i t h i n the UBC/Ritsumeikan Program. Thus t h e r e i s s u f f i c i e n t i n f o r m a t i o n a v a i l a b l e about TOEFL and GPA, but not much about speaking s c o r e s and w r i t i n g s c o r e s . 2. The f i n d i n g s were i n t e r p r e t e d w i t h s p e c i a l c a r e . The regression  model does not p r o v i d e s u f f i c i e n t power t o handle  measurement e r r o r s l i k e LISREL does. T h e r e f o r e , t h e p r e s e n t study c l e a r l y d i s t i n g u i s h e s t h e d i f f e r e n c e between TOEFL and language p r o f i c i e n c y , and between GPA and academic achievement. With these d i s t i n c t i o n s i n mind, r e s u l t s o f t h e  55  r e g r e s s i o n a n a l y s i s were i n t e r p r e t e d c a r e f u l l y so as avoid  to  overgeneralization. The  assumption of l i n e a r i t y . The  r e l a t i o n s h i p s between p r e d i c t o r s and  nature of criteria,  the linearity  or  n o n - l i n e a r i t y , r e q u i r e s a proper model f o r r e g r e s s i o n a n a l y s i s . As shown i n F i g u r e 4.1, d i s t r i b u t e d and  the r e s i d u a l s are randomly  t h e r e are no s y s t e m a t i c  between the p r e d i c t e d v a l u e s j u s t i f i e s the use  and  the r e s i d u a l s . T h i s  of the l i n e a r r e g r e s s i o n model.  Dependent  Variable:  -  1  GPA  0  1  Regression Standardized  F i g u r e 4.1.  patterns e x i s t i n g  2  Predicted Value  S c a t t e r p l o t of the d i s t r i b u t i o n of  the  residuals. The  assumption t h a t mean of the r e s i d u a l i s zero.  assumption means t h a t the v a r i a n c e constant 4.1  This  of the r e s i d u a l s i s  f o r a l l l e v e l s of the independent v a r i a b l e s .  Figure  a l s o shows t h a t the spread of the r e s i d u a l s does not  i n c r e a s e or decrease w i t h the magnitude of the  predicted  56  values  on the X a x i s . T h i s i n d i c a t e s t h a t the above  assumption was  met.  The assumption t h a t r e s i d u a l s are independent.  This  assumption r e q u i r e s t h a t r e s i d u a l s are independent o f one another. V i o l a t i o n of t h i s assumption, o f t e n r e f e r r e d t o as a u t o c o r r e l a t i o n , a f f e c t s the v a l i d i t y o f t e s t s o f s i g n i f i c a n c e . From F i g u r e 4.1, we a l s o can see t h a t the r e s i d u a l s a r e randomly  s c a t t e r e d above and below the  zero  h o r i z o n t a l band. T h i s t e l l s us t h a t a u t o c o r r e l a t i o n does not occur and the above assumption i s met. The assumption o f normal d i s t r i b u t i o n o f r e s i d u a l s . T h i s assumption r e q u i r e s t h a t r e s i d u a l s should d i s t r i b u t e n o r m a l l y . In the histogram o f F i g u r e 4.2,  the d i s t r i b u t i o n  of the r e s i d u a l s appears approximately normal.  Dependent V a r i a b l e :  GPA  Regression Standardized  F i g u r e 4.2.  Residual  D i s t r i b u t i o n of r e s i d u a l s .  57  The  assumption of the absence of  perfect  m u l t i c o l l i n e a r i t y . T h i s assumption assumes t h a t t h e r e i s no strong  i n t e r c o r r e l a t i o n s among independent v a r i a b l e s .  existence  of c e r t a i n c o r r e l a t i o n among the  The  independent  v a r i a b l e s i n d i c a t e s h i g h m u l t i c o l l i n e a r i t y . The  tolerance  of  an independent v a r i a b l e i s a commonly used measure of m u l t i c o l l i n e a r i t y . In the present study, the t o l e r a n c e the p r e d i c t o r v a r i a b l e s ranged from .63  to  .89.  This  of implies  t h a t the above assumption i s s a t i s f i e d . The  assumption t h a t r e s i d u a l s are not  each o f the  independent v a r i a b l e s . In the p r e s e n t study, a  c o r r e l a t i o n a n a l y s i s was The  c o r r e l a t e d with  performed t o check t h i s assumption.  r e s u l t s showed t h a t a l l the c o r r e l a t i o n c o e f f i c i e n t s  between the  independent v a r i a b l e s and  the  r e s i d u a l s were  except t h a t between gender and  the r e s i d u a l s  T h e r e f o r e , t h i s assumption was  also  D i a g n o s i s of o u t l i e r s and frequently  points  standardized  Two  r e s i d u a l and  o u t l i e r s and  centered  influential  shown i n Appendix I I , a l l the  r e s i d u a l s are below 3 u n i t s from zero,  no o u t l i e r s are .241  satisfied.  used measures, s t a n d a r d i z e d  r e s p e c t i v e l y . As  (r=.371).  i n f l u e n t i a l points.  l e v e r a g e were s e l e c t e d t o d e t e c t  .00  found. However, case 73 has  and  thus  a l e v e r a g e of  which i s t w i c e as l a r g e as the upper l i m i t of normal  leverage values.  I t turned out t h a t t h i s case had  a very  low  The c o n s i d e r a b l e r e l a t i o n s h i p between gender and the r e s i d u a l s c l e a r l y i n d i c a t e s again t h a t gender i s a composite v a r i a b l e which i n t e r r e l a t e s w i t h o t h e r v a r i a b l e s , known and unknown, or c u r r e n t l y a v a i l a b l e and u n a v a i l a b l e f o r research.  58  TOEFL t o t a l score, higher  487, but i t s GPA, 77, was f i v e  points  than t h e GPA mean. Case 73, t h e r e f o r e , was i d e n t i f i e d  as an i n f l u e n t i a l p o i n t and e l i m i n a t e d b e f o r e  performing t h e  multiple regression analysis. Hierarchical regression The  present  analysis  study employs t h e m u l t i p l e  linear  r e g r e s s i o n a n a l y s i s w i t h a h i e r a r c h i c a l procedure i n s t e a d o f a stepwise procedure t h a t i s used most commonly. T h i s d e c i s i o n i s made based on t h e comparison among t h r e e  options  i n t h e procedure o f t h e m u l t i p l e r e g r e s s i o n a n a l y s i s . The  primary purpose o f t h e present  study was t o  e s t i m a t e t h e p r e d i c t i v e v a l i d i t y o f TOEFL s c o r e s . That i s , the study aimed a t e s t i m a t i n g  the unique c o n t r i b u t i o n o f  TOEFL s c o r e s t o GPA, r a t h e r than t h e o v e r a l l c o n t r i b u t i o n o f a l l p r e d i c t o r s t o GPA o r t h e b e s t l i n e a r combination o f p r e d i c t o r s t o p r e d i c t GPA, by p a r t i a l l i n g out t h e r e s t o f the p r e d i c t o r s under study. To accomplish t h i s , t h e key i s s u e was t o determine t h e order o r sequence o f e n t e r i n g t h e p r e d i c t o r s because d i f f e r e n t e n t r y orders  yield different  e s t i m a t e s o f the unique c o n t r i b u t i o n o f a p r e d i c t o r . Generally  speaking, t h e r e a r e t h r e e a l t e r n a t i v e  procedures f o r t h e m u l t i p l e r e g r e s s i o n a n a l y s i s : simultaneous, stepwise, and h i e r a r c h i c a l . In the simultaneous a n a l y s i s , every p r e d i c t o r i s entered r e g r e s s i o n a n a l y s i s simultaneously  into the  and i s p a r t i a l l e d out  from every other p r e d i c t o r i n d i s c r i m i n a t e l y . T h i s procedure can p r o v i d e  a r e g r e s s i o n equation and squared R f o r a l l  59  p r e d i c t o r s i n t h e equation, but i t does not e s t i m a t e t h e unique c o n t r i b u t i o n s o f each v a r i a b l e t o t h e t o t a l v a r i a n c e i n t h e equation. Stepwise a n a l y s i s can estimate t h e unique c o n t r i b u t i o n of p r e d i c t o r s by o b t a i n i n g p a r t i a l c o r r e l a t i o n o r i n c r e m e n t a l v a r i a n c e . However, t h i s procedure  solely  relies  on s t a t i s t i c a l c r i t e r i a t o determine t h e sequence o f e n t e r i n g p r e d i c t o r s . When t h e competing p r e d i c t o r s s u b s t a n t i a l l y connect w i t h each other, t h e p a r t i a l c o r r e l a t i o n o r incremental v a r i a n c e might v a r y  significantly  a c c o r d i n g t o t h e sequence i n which p r e d i c t o r s a r e e n t e r e d . Thus, t h e procedure  o f stepwise a n a l y s i s might c r e a t e  d i f f i c u l t i e s i n e s t i m a t i n g , i n t e r p r e t i n g , comparing, and r e p l i c a t i n g the regression r e s u l t s . The h i e r a r c h i c a l procedure  enters p r e d i c t o r s i n a p r e -  s p e c i f i e d sequence t o estimate t h e unique c o n t r i b u t i o n o f each p r e d i c t o r t o t h e t o t a l v a r i a n c e i n t h e r e g r e s s i o n e q u a t i o n . The c h o i c e o f a p a r t i c u l a r sequence o f p r e d i c t o r s i s made i n advance by t h e purpose and l o g i c o f t h e r e s e a r c h , i n c o n t r a s t t o t h e stepwise r e g r e s s i o n . The h i e r a r c h i c a l procedure  l e a d s t o t e s t s o f t h e hypotheses t h a t d e f i n e t h e  o r d e r and improve our understanding study  o f t h e phenomena under  (Cohen, 1983, pp. 120-125). As Tabachnick  (1989) d i s c u s s e d , simultaneous,  stepwise,  and F i d e l l  and h i e r a r c h i c a l  r e g r e s s i o n can be b e s t used f o r model-estimating, b u i l d i n g , and m o d e l - t e s t i n g r e s p e c t i v e l y  model-  ( P . 150). Thus, i n  60 the p r e s e n t study, h i e r a r c h i c a l procedure t e s t t h e hypotheses o f t h e p r e s e n t The  was s e l e c t e d t o  study.  sequence o f e n t e r i n g t h e p r e d i c t o r s i n t h e p r e s e n t  study was TOEFL t o t a l s c o r e s , w r i t i n g s c o r e s ,  speaking  s c o r e s , and gender. T h i s sequence was mainly based on t h e r e s e a r c h p r i o r i t y o f t h e study because no c a u s a l r e l a t i o n s h i p among t h e p r e d i c t o r s was found.  S i n c e TOEFL  s c o r e s r e f l e c t e d t h e major g o a l o f t h e r e s e a r c h and were t h e primary  focus o f t h e study, they were entered i n t o t h e  equation f i r s t . W r i t i n g s c o r e s and speaking  scores followed  because they were viewed as having l e s s e r r e l e v a n c e t o t h e r e s e a r c h than TOEFL s c o r e s . Gender was entered l a s t because i t was used as an e x p l a n a t o r y v a r i a b l e t o exemplify nonlanguage p r e d i c t o r s ' p r e d i c t i v e v a l i d i t y which was not much documented i n r e s e a r c h l i t e r a t u r e . W i t h i n the TOEFL s e c t i o n a l s c o r e s , t h e sequence f o r e n t e r i n g was from s e c t i o n a l I I , I I I , and I . T h i s was based on a  descending  o r d e r i n terms o f t h r e e s c o r e s ' p r e d i c t i v e v a l i d i t y r e p o r t e d i n t h e p r e v i o u s r e s e a r c h (Abdzi, 1967; Aleamoni, 1974; Harvey, 1979; H e i l & Johnson, 1988; Z i r p o l i , In t h e f o l l o w i n g s e c t i o n , the f i r s t  1988).  hierarchical  r e g r e s s i o n a n a l y s i s was performed t o t e s t the primary h y p o t h e s i s r e g a r d i n g t h e p r e d i c t i v e v a l i d i t y o f TOEFL t o t a l s c o r e s . The second h i e r a r c h i c a l r e g r e s s i o n a n a l y s i s then mainly  served t o examine the unique c o n t r i b u t i o n o f each  TOEFL s e c t i o n a l s c o r e s t o GPA. Note t h a t change i n squared R  61 (AR ) 2  was used i n SPSS as t h e e s t i m a t o r f o r unique  c o n t r i b u t i o n o f each p r e d i c t o r t o GPA i n a h i e r a r c h i c a l regression  analysis.  9  H i e r a r c h i c a l a n a l y s i s w i t h TOEFL t o t a l s c o r e s .  Table  4.3 shows t h a t t h e r e s u l t s o f t h e h i e r a r c h i c a l a n a l y s i s w i t h TOEFL t o t a l  scores.  T a b l e 4.3 Summary t a b l e o f t h e h i e r a r c h i c a l a n a l y s i s w i t h TOEFL t o t a l scores  Step  R  AdjR  1  .377  .  2  .448  3 4  132  2  F  P  AR  000  .  2  142  AF  14 . 8 9 5  .  14 . 8 9 5  .183  11 . 1 8 4  .000  .059  6. 555  .457  .182  7 .731  .000  .008  .584  .311  11 . 2 5 1  .000  .132  AP  .  000  Variable  In:  TOEFL  .012  In:  WRITE  • 859  .357  In:  SPEAK  14. 470  .000  In:  GENDER  In s t e p 1, TOEFL t o t a l s c o r e s entered i n t o t h e regression  9  equation i n order t o determine t h e e x t e n t t o  2 •  •  In t h e p r e s e n t sturdy A R i s i n t e r p r e t e d as t h e amount o f v a r i a n c e added t o R by each p r e d i c t o r a t t h e p o i n t t h a t i t e n t e r s t h e e q u a t i o n i n a h i e r a r c h i c a l procedure. F o r t h e d i s t i n g u i s h e d d i f f e r e n c e s i n the meaning o f unique c o n t r i b u t i o n o f a p r e d i c t o r t o R among t h e t h r e e procedures due t o t h e d i f f e r e n c e s i n h a n d l i n g t h e o v e r l a p p i n g among c o r r e l a t e d p r e d i c t o r s , see Tabachnick and F i d e l l , 1989, pp. 141-142 & pp. 150-154. 2  62  which the TOEFL o v e r a l l score p r e d i c t e d GPA. t h a t 14.20% of the v a r i a n c e i n GPA TOEFL t o t a l s c o r e . The  was  R e s u l t s showed  accounted f o r by  the  r e s t of about 86% of the v a r i a n c e  remained as r e s i d u a l or unexplained  e r r o r which should  be m i s i n t e r p r e t e d as measurement e r r o r . T h i s mainly  not  implies  t h a t the amount of v a r i a n c e had not y e t been e x p l a i n e d . In step 2, w r i t i n g s c o r e s entered i n t o the r e g r e s s i o n e q u a t i o n . The  r e s u l t showed t h a t i t accounted f o r 5.90%  the v a r i a n c e . By adding w r i t i n g s c o r e s , the squared equation  increased to  R i n the  .183.  In step 3, the a d d i t i o n of speaking r e g r e s s i o n equation only i n c r e a s e d 0.70% accounted f o r . T h i s may  of  s c o r e s t o the of v a r i a n c e  i n d i c a t e t h a t speaking p r o f i c i e n c y  of the students d i d not c o n t r i b u t e t o t h e i r academic achievement  significantly.  Step 4 was  used t o examine the e f f e c t of the a d d i t i o n  of an e x p l a n a t o r y non-language v a r i a b l e t o the r e g r e s s i o n model. By a d d i t i o n of gender, the squared and the squared  R change was  .132.  R increased to  .31  T h i s showed t h a t gender  d i f f e r e n c e p l a c e d one of the l a r g e s t weight on  GPA.  To f u r t h e r analyze the e f f e c t of gender d i f f e r e n c e , an ANOVA on TOEFL t o t a l s c o r e s and gender was R e s u l t s showed t h a t t h e r e was TOEFL s c o r e s due  performed.  no s i g n i f i c a n t d i f f e r e n c e of  t o gender, F (1,90) = 1.476, p > .05.  i m p l i e d t h a t the v a r i a n c e of GPA  was  not due  t o gender  d i f f e r e n c e i n TOEFL t o t a l s c o r e s . In other words, the d i f f e r e n c e i n academic achievement appeared not t o be  This  63  affected  s i g n i f i c a n t l y by t h e d i f f e r e n t l e v e l s o f E n g l i s h  proficiency,  but rather,  by t h e d i f f e r e n c e s  i n non-language  f a c t o r s among male and female s t u d e n t s . Hierarchical  a n a l y s i s w i t h TOEFL s e c t i o n a l  scores.  T a b l e 4 . 4 shows t h a t t h e r e s u l t s o f a h i e r a r c h i c a l w i t h TOEFL s e c t i o n a l  analysis  scores.  Table 4 . 4 Summary t a b l e o f t h e h i e r a r c h i c a l a n a l y s i s sectional  tep  R  w i t h TOEFL  scores  AdjR  2  P  AR  . 001  F  Variable  AF  AP  .112  11 . 2 9 7  . 001  In:  SEC2  .489  In:  SEC3  2  .334  .102  11 . 2 9 7  2  .341  .096  5 .857  .004  .005  .482  3  .400  . 131  5 .592  .002  .044  4.590  . 035  In:  SEC1  4  .455  .170  5 . 674  .000  .047  5.130  .026  In:  WRITE  5  .466  .171  4 .762  .007  .010  1.093  .299  In:  SPEAK  6  .589  .300  7 .513  .000  .130  16.870  . 000  In:  GENDER  1  •  The second h i e r a r c h i c a l a n a l y s i s subscores as t h e p r e d i c t o r s  rather  used t h r e e TOEFL  than a composite TOEFL  s c o r e . R e s u l t s showed t h a t , w h i l e h i g h t o l e r a n c e s f o r t h r e e sectional  s c o r e s r e v e a l e d low i n t e r r e l a t i o n s among t h e  subscores, TOEFL s e c t i o n  I I had t h e h i g h e s t squared R  64  2  2  change(AR =.112), compared w i t h s e c t i o n I ( AR =.044), s e c t i o n I I I (AR =.005). In o t h e r words, s e c t i o n I I s c o r e s among them had t h e most importance impact  on the v a r i a n c e i n  GPA. The unique c o n t r i b u t i o n o f w r i t i n g s c o r e s t o t h e equation was .047, speaking s c o r e s was  .010, gender was  .130.  Compared w i t h w r i t i n g s c o r e s .059, speaking  .008,  and gender .132 as shown i n t h e f i r s t  scores  hierarchical  a n a l y s i s , t h e r e s u l t s i n d i c a t e d t h a t t h e p a t t e r n of t h e relative  importance among t h e t h r e e p r e d i c t o r s d i d not  change, w h i l e TOEFL t o t a l s c o r e s were p a r t i t i o n e d i n t o t h r e e s e c t i o n a l scores. Summary The r e s u l t s o f t h e p r e s e n t study show t h a t the p r e d i c t i v e v a l i d i t y o f TOEFL t o t a l s c o r e s was  .142 (p_<.001).  For TOEFL s e c t i o n a l s c o r e s , s e c t i o n I I s c o r e s were t h e most important  o f t h r e e s e c t i o n a l s c o r e s . Among a l l p r e d i c t o r s i n  the study, gender had the h i g h e s t p r e d i c t i v e v a l i d i t y on GPA when TOEFL s e c t i o n a l s c o r e s were used. Gender, TOEFL t o t a l s c o r e s , w r i t i n g s c o r e s , speaking s c o r e s accounted  f o r 31.00%  (p_<.001) o f t h e v a r i a n c e i n GPA. Gender, TOEFL s e c t i o n a l s c o r e s , w r i t i n g s c o r e s , speaking s c o r e s accounted of t h e v a r i a n c e i n GPA (p_<.001).  f o r 3 0.04%  65  Chapter F i v e Discussion T h i s chapter d i s c u s s e s research  findings pertinent  hypotheses, i n t e r p r e t s the  f i n d i n g s , and  draws c o n c l u s i o n s  implications  of the  the of these  study.  P r e d i c t i v e v a l i d i t y of TOEFL t o t a l s c o r e s on The  to  GPA  p r e s e n t study examined the p r e d i c t i v e v a l i d i t y of a  p r e d i c t o r v a r i a b l e i n two c o n t r i b u t i o n t o GPA  ways: (a) e v a l u a t i o n  on the b a s i s of the  l e v e l s of p r e d i c t i v e v a l i d i t y  four  of i t s unique operational  (see Chapter three) and  (b)  assessment of i t s r e l a t i v e importance i n comparison w i t h o t h e r p r e d i c t o r s under study. Results  of the p r e s e n t study show t h a t A R of TOEFL 2  t o t a l s c o r e s i s .142  (p_<.001). T h i s r e s u l t i s comparable  w i t h r e s u l t s of a m e t a - a n a l y s i s of 27 TOEFL p r e d i c t i o n studies  i n which the mean c o r r e l a t i o n c o e f f i c i e n t s of TOEFL  t o t a l s c o r e s and (Yan,  f i r s t y e a r ' s GPA  i s .300,  1994). A c c o r d i n g t o the o p e r a t i o n a l  predictive v a l i d i t y , therefore,  2  is  l e v e l s of  f i r s t term's  r e s u l t s a l s o r e v e a l t h a t TOEFL t o t a l s c o r e s  as the  l a r g e s t compared w i t h the  language-based p r e d i c t o r s , w r i t i n g and  other  speaking  under  two scores.  Thus, TOEFL t o t a l s c o r e s are an important p r e d i c t o r of i n the p r e s e n t study.  GPA.  are  ranked as the second l a r g e s t among a l l the p r e d i c t o r s study and  9%  TOEFL t o t a l s c o r e s have a  medium l e v e l of the p r e d i c t i v e v a l i d i t y on The  i.e., R  GPA  66  Based on these two f i n d i n g s r e p o r t e d above,  i t can be  concluded t h a t h y p o t h e s i s I o f the study i s supported. That i s , TOEFL t o t a l s c o r e s have a medium l e v e l o f the p r e d i c t i v e v a l i d i t y on f i r s t term's GPA  f o r the group o f s t u d e n t s under  study. It  i s not s u r p r i s i n g t h a t TOEFL t o t a l s c o r e s o n l y  account f o r 14.20% o f t h e v a r i a n c e i n GPA.  As shown i n many  s t u d i e s , E n g l i s h language p r o f i c i e n c y i s j u s t one of many f a c t o r s a f f e c t i n g academic achievement. I t appears t h a t  no  s i n g l e f a c t o r alone can completely o r l a r g e l y determine academic achievement. As the unique c o n t r i b u t i o n p r e d i c t o r t o GPA,  therefore,  of a s i n g l e  14.20% c l e a r l y i n d i c a t e s  that  TOEFL t o t a l s c o r e s alone do e x p l a i n a s i g n i f i c a n t amount o f v a r i a n c e i n GPA. itself,  In o t h e r words,  language p r o f i c i e n c y by  among many o t h e r f a c t o r s , does have an important  e f f e c t on academic achievement. We can f u r t h e r a n a l y z e how TOEFL's p r e d i c t i v e on GPA  validity  i s a f f e c t e d by each p a i r o f TOEFL s c o r e s and GPA f o r  each student under study. As shown i n F i g u r e 5.1, we can d i v i d e each GPA-TOEFL p a i r i n t o f o u r d i v i s i o n s by u s i n g the mean GPA  and the mean  TOEFL t o t a l s c o r e s : upper l e f t , upper r i g h t , lower l e f t ,  and  lower r i g h t . Both upper r i g h t and lower l e f t d i v i s i o n s share one commonality: A h i g h TOEFL score corresponds t o a h i g h GPA,  o r a low TOEFL score w i t h a low GPA.  However, both t h e  upper l e f t and lower r i g h t d i v i s i o n s show t h a t a h i g h TOEFL  67  s c o r e goes w i t h a low GPA, o r a low TOEFL w i t h a h i g h GPA. Note t h a t t h e r e a r e 24 cases i n t h e upper l e f t  division,  whereas o n l y 15 i n t h e lower r i g h t . Among these cases,  there  are a t l e a s t 6 cases w i t h TOEFL s c o r e s below 4 80 but t h e i r GPAs a r e above t h e mean GPA, whereas t h e r e i s only one case w i t h a TOEFL s c o r e above 540 and a GPA below 6 0 . T h i s i n d i c a t e s t h a t i n t h e p r e s e n t study about one q u a r t e r o f t h e students who had low l e v e l s o f language p r o f i c i e n c y managed t o achieve academic success. The number o f t h i s sub-group i s h i g h e r than t h a t o f students who have t h e h i g h l e v e l o f language p r o f i c i e n c y but a r e unable t o reach t h e h i g h  level  of academic achievement. In o t h e r words, a good TOEFL s c o r e does not n e c e s s a r i l y guarantee a good GPA, but a low TOEFL s c o r e i s o f t e n a s s o c i a t e d w i t h a good GPA. I t i s these t h a t might c o n s i d e r a b l y decrease t h e magnitude o f t h e  cases  68  p r e d i c t i v e v a l i d i t y of TOEFL s c o r e s . f o r those whose n a t i v e are  involved  They prove again  language i s not E n g l i s h , many f a c t o r s  i n t h e i r academic l e a r n i n g at u n i v e r s i t i e s and  language p r o f i c i e n c y does not always f u n c t i o n as a key  predominantly  element.  P r e d i c t i v e v a l i d i t y of TOEFL s e c t i o n a l s c o r e s on Results  GPA  of the study show t h a t the combination of  t h r e e s e c t i o n a l s c o r e s accounts f o r 16.10% of the i n GPA.  that  This  i s c l o s e t o what TOEFL t o t a l s c o r e s  However, the unique c o n t r i b u t i o n and t h r e e s e c t i o n a l s c o r e s t o GPA  variance do.  r e l a t i v e importance of  are remarkably d i f f e r e n t . I t 2  i s shown t h a t the changes i n R of s e c t i o n I, I I , and are  .044  (p_>.05), .112  (P<.001), and  .005  i n d i c a t e they have s m a l l , medium, and  under study and scores.  This  research  (p_>.05) which  n e g l i g i b l e l e v e l s of  p r e d i c t i v e v a l i d i t y respectively. Section second h i g h e s t  I I s c o r e s have the  p r e d i c t i v e v a l i d i t y among the the h i g h e s t  III  six  predictors  among t h r e e TOEFL s e c t i o n a l  f i n d i n g i s comparable w i t h those i n p r e v i o u s  (Johnson, 1988;  Zirpoli,  1988;  L i g h t , Xu,  1989). Thus, based on the uneven c o n t r i b u t i o n s s e c t i o n a l s c o r e s t o GPA  Morris,  of t h r e e  as w e l l as t h e i r d i f f e r e n t  importance, the c o n c l u s i o n s  f o r h y p o t h e s i s I I are:  I I have a medium l e v e l of p r e d i c t i v e v a l i d i t y , s c o r e s have a small  &  Section  section I  l e v e l of p r e d i c t i v e v a l i d i t y ,  and  s e c t i o n I I I s c o r e s have a n e g l i g i b l e l e v e l of p r e d i c t i v e validity.  69  There i s an i n t e r e s t i n g q u e s t i o n s e c t i o n a l scores be  of TOEFL, why  t o ask:  Among  do s e c t i o n I I scores  three tend t o  so dominant i n p r e d i c t i n g GPA? In the TOEFL t e s t , s e c t i o n I, L i s t e n i n g Comprehension,  measures the a b i l i t y II,  Structure  t o understanding o r a l E n g l i s h ;  and W r i t t e n  s e l e c t e d s t r u c t u r a l and w r i t t e n E n g l i s h ; And  Expression,  (ETS,  1992,  tests recognition  of  grammatical knowledge i n standard  s e c t i o n I I I , Vocabulary and  Comprehension, t e s t s the a b i l i t y English  Section  pp.  6-7).  The  t o understand three  Reading  written  s e c t i o n s measure  10  l i s t e n i n g s k i l l s , w r i t i n g knowledge, r e s p e c t i v e l y . Thus, the  and  reading  f i n d i n g s presented here might  i n d i c a t e t h a t a good GPA  may  demand more w r i t t e n  s k i l l s than spoken E n g l i s h s k i l l s  (i.e.,  a good GPA  may  r e q u i r e more p r o d u c t i v e  r e c e p t i v e s k i l l s of w r i t t e n E n g l i s h  (i.e.,  the course grades r e q u i r e w r i t t e n p r o d u c t i v e  assignments, and  academic t a s k s final  scores  English skills  section II  v s . s e c t i o n I I I s c o r e s ) . As seen i n T a b l e 3.1,  f u l f i l l various  English  section II  v s . s e c t i o n I s c o r e s ) . Furthermore, f o r w r i t t e n skills,  skills  than scores  about 85%  skills  of  to  such as term paper, course  examinations. When students*  academic  achievement i s assessed mainly based on performance i n written expression, i s greater To  the weight of s e c t i o n I I scores  than the o t h e r two •"  sectional  on  GPA  scores.  ETS has not e x p l a i n e d what i s e x a c t l y meant by s t r u c t u r e and w r i t t e n e x p r e s s i o n . S i n c e s e c t i o n I I uses both sentence c o r r e c t i o n and sentence completion t o t e s t b a s i c knowledge about w r i t t e n E n g l i s h , the present study simply l a b e l s s e c t i o n I I as w r i t i n g knowledge i n s t e a d of w r i t i n g s k i l l s .  70  P r e d i c t i v e v a l i d i t y of w r i t i n g s c o r e s on  GPA  R e s u l t s of the study r e v e a l t h a t the change of R w r i t i n g s c o r e s i s .059  of  2  (p_<.05) when TOEFL t o t a l s c o r e s are  used. T h i s means t h a t w r i t i n g s c o r e s have a s m a l l l e v e l o f p r e d i c t i v e v a l i d i t y on GPA.  I t i s a l s o shown t h a t  r e l a t i v e importance of w r i t i n g s c o r e s are ranked  the t h i r d among  f o u r p r e d i c t o r s , behind gender and TOEFL t o t a l s c o r e s ,  and  b e f o r e speaking s c o r e s . These f i n d i n g s t o some extent support h y p o t h e s i s I I I i n the study and  indicate  that  w r i t i n g s c o r e s have a s m a l l l e v e l of p r e d i c t i v e v a l i d i t y  on  GPA. I t i s i n t e r e s t i n g t o note t h a t p r e d i c t i v e v a l i d i t y w r i t i n g s c o r e s i s s u b s t a n t i a l l y lower than t h a t of II  scores  (.047  vs.  .112  f o r AR ) 2  do w r i t i n g s c o r e s have so  c o n t r i b u t i o n t o GPA  section  when TOEFL s e c t i o n a l  s c o r e s are used. Both d e a l w i t h measurement of E n g l i s h , but why  of  written  little  compared w i t h i t s c o u n t e r p a r t ?  There might be t e n t a t i v e e x p l a n a t i o n s t o t h i s q u e s t i o n . For i n s t a n c e , as a l o c a l l y used t e s t i n g instrument, w r i t i n g sample assessment might not possess  the  sufficient  r e l i a b i l i t y and v a l i d i t y i n measuring E n g l i s h w r i t i n g as i t should. T h i s may  r e s u l t i n under-estimation  p r e d i c t i v e v a l i d i t y on GPA.  Also,  skills  of i t s  S e c t i o n I I s c o r e s measure  w r i t i n g knowledge, w h i l e w r i t i n g s c o r e s d i r e c t l y  assess  w r i t i n g s k i l l s . For t h i s group of Japanese students whose E n g l i s h i s a t the i n t e r m e d i a t e l e v e l , they might need more  71 b a s i c w r i t i n g knowledge o f w r i t t e n E n g l i s h  i n order t o  f u l f i l l t h e i r academic l e a r n i n g t a s k s s u c c e s s f u l l y . P r e d i c t i v e v a l i d i t y of speaking s c o r e s on The  GPA  r e s u l t s o f the study i n d i c a t e t h a t speaking s c o r e s  have a n e g l i g i b l e l e v e l of p r e d i c t i v e v a l i d i t y on  GPA  (AR =.O08, p_>.05 o r AR =.010, p>.05, depending on TOEFL 2  2  t o t a l s c o r e s or s e c t i o n a l s c o r e s are used) and  is  c o n s i s t e n t l y ranked as the l e a s t important among a l l the predictors regarding  i n p r e d i c t i n g GPA.  Therefore, hypothesis  the p r e d i c t i v e v a l i d i t y of speaking s c o r e s i s  r e j e c t e d . Due reliability  t o i n s u f f i c i e n c y of i n f o r m a t i o n  and  about  v a l i d i t y of the o r a l i n t e r v i e w  speaking s c o r e s have a medium l e v e l of p r e d i c t i v e must be  Writing  l e f t for future  scores,  s e c t i o n I, I I , and skills:  the  used i n t h i s  academic exchange program, the q u e s t i o n c o n c e r n i n g  on GPA  IV  why validity  analysis.  speaking scores,  and  TOEFL s c o r e s i n  I I I c o u l d be seen t o a s s e s s f o u r  l i s t e n i n g , speaking, r e a d i n g , and  language  writing. It i s  i n t e r e s t i n g t o l o o k a t weights of i n d i v i d u a l s k i l l s  in  p r e d i c t i n g GPA  GPA.  as w e l l as t h e i r o v e r a l l e f f e c t s on  F i r s t , the r e s u l t s of the study suggest t h a t skills  written  i n E n g l i s h are more important than o r a l s k i l l s  p r e d i c t i n g GPA,  as we  have seen t h a t AR  2  of s e c t i o n  s c o r e s i s l a r g e r than t h a t of s e c t i o n I and  AR  2  s c o r e s i s l a r g e r than t h a t of speaking s c o r e s . r e s u l t s tend t o i n d i c a t e t h a t p r o d u c t i v e s k i l l s  in  II  of w r i t i n g Also,  the  in written  72  E n g l i s h are more important than r e c e p t i v e s e c t i o n I I s c o r e s and power on GPA  skills  w r i t i n g s c o r e s have more p r e d i c t i v e  than s e c t i o n I I I s c o r e s .  However, we  t o i n t e r p r e t the r e l a t i v e importance on GPA comprehension and  since  of  are  unable  written  a u r a l comprehension, although A R  2  of  s e c t i o n I s c o r e s exceed t h a t of s e c t i o n I I I . A l l i n a l l , i t seems premature t o draw a c o n c l u s i o n  on the b a s i s  of  f i n d i n g s of the p r e s e n t study about the r e l a t i o n s h i p o f those f o u r language s k i l l s and  academic achievement.  Second, r e s u l t s show t h a t the cumulative R of 2  language-based p r e d i c t o r s , speaking s c o r e s , and  the t h r e e s e c t i o n a l scores,  c o u l d be  2  scores,  (p_<.001). Thus, i t  i n f e r r e d t h a t the o v e r a l l p r e d i c t i v e v a l i d i t y  language f a c t o r s on GPA R  i s .217  writing  five  might have an upper l i m i t .  of language f a c t o r s i n a r e g r e s s i o n  not exceed .25.  of  Probably  model would p r o b a b l y  In o t h e r words, among many o t h e r v a r i a b l e s ,  language f a c t o r s alone might o p t i m i z e t h e i r c o n t r i b u t i o n about one-quarter of academic achievement assessed by P r e d i c t i v e v a l i d i t y of gender on The  GPA.  GPA  r e s u l t s show t h a t gender's A R  when TOEFL t o t a l s c o r e s are used, and TOEFL s e c t i o n a l s c o r e s are used. The gender has  at  2  i s .13 2,  .130  (p_<.001)  (p_<.001) when the  findings indicate  a medium l e v e l of p r e d i c t i v e v a l i d i t y on  Gender i s c o n s i s t e n t l y ranked as one  that  GPA.  of the most powerful  p r e d i c t o r s under study. T h e r e f o r e , i t can be concluded  that  73  gender i s a good p r e d i c t o r i n p r e d i c t i n g GPA V i s strongly  hypothesis  supported.  Many s t u d i e s have a l r e a d y d i f f e r e n c e s do is s t i l l  and  proved t h a t gender  i n f l u e n c e academic achievement. However, i t  s u r p r i s i n g t h a t gender had  c o n t r i b u t i o n t o GPA  such a remarkable  i n the p r e s e n t study. Japanese female  students as a group performed s i g n i f i c a n t l y b e t t e r than male students i n the course grades. T h i s u n a n t i c i p a t e d r a i s e s a question: greatly?  Why  do gender d i f f e r e n c e s  In o t h e r words, why  do  finding  a f f e c t GPA  so  female students a c a d e m i c a l l y  e x c e l over t h e i r male c o u n t e r p a r t s ? As reviewed i n chapter two,  research  on  gender  d i f f e r e n c e s r e v e a l s t h a t gender i s a composite influenced  by and  educational,  impacting on v a r i o u s  l i n g u i s t i c , psychological,  domains. G e n e r a l l y  factor  factors in social, and  physiological  speaking, female students perform  i n language a r t s and male students perform b e t t e r science.  To  f i n d the p o s s i b l e cause f o r the  d i f f e r e n c e i n GPA, s c o r e s was  i n TOEFL  r e s u l t s showed t h a t t h e r e were no  s i g n i f i c a n t gender d i f f e r e n c e s t h r e e courses from which GPA language e d u c a t i o n and  in  gender  an F - t e s t on gender d i f f e r e n c e  conducted. The  better  i n these scores,  although a l l  were o b t a i n e d were about  required  good language p r o f i c i e n c y .  T h i s r e s u l t c l e a r l y i n d i c a t e s t h a t gender d i f f e r e n c e s are not caused by  in  GPA  language f a c t o r s but by other non-language  f a c t o r s . Probably non-language f a c t o r s such as m o t i v a t i o n , time spent on l e a r n i n g , academic  learning  aptitude,  74  l e a r n i n g s t y l e , p r e v i o u s knowledge,  and c u l t u r a l  a d a p t a b i l i t y , might i n d i r e c t l y p l a c e e f f e c t s on GPA through gender d i f f e r e n c e s . Due t o l a c k o f data t o analyze, what k i n d s o f non-language f a c t o r s and how they c o n t r i b u t e f o r t h i s group o f students remain open f o r f u t u r e  t o GPA  research.  Implications The f i n d i n g s i n the p r e s e n t study may implications  have p r a c t i c a l  f o r t h e UBC/Ritsumeikan Academic  Exchange  Program. 1. The main f i n d i n g s i n the study c l e a r l y i n d i c a t e TOEFL t o t a l s c o r e s are a good p r e d i c t o r of f i r s t term's  that GPA  f o r the UBC/Ritsumeikan Program s t u d e n t s . T h e r e f o r e , the program should c o n t i n u e t o use TOEFL t o measure E n g l i s h language p r o f i c i e n c y f o r program admissions. S i n c e TOEFL s e c t i o n I I s c o r e s have the h i g h e s t three s e c t i o n a l scores,  p r e d i c t i v e v a l i d i t y among  they deserve p a r t i c u l a r a t t e n t i o n  f o r admission s e l e c t i o n . 2.  The f i n d i n g s on gender d i f f e r e n c e s  i n GPA  strongly  suggest t h a t non-language f a c t o r s p l a y an important r o l e i n 11  academic achievement.  .  .  Thus, i t i s a d v i s a b l e  t h a t the  T h i s f i n d i n g might l e a d m i s t a k e n l y t o another i m p l i c a t i o n f o r program admissions: i n c l u d i n g more female students i n t o the program and e x c l u d i n g more male ones from the program. In f a c t , t h i s p o l i c y , g i v e n i t was taken, would be not o n l y p o l i t i c a l l y i n c o r r e c t but a l s o l o g i c a l l y o v e r s i m p l i f i e d . As d i s c u s s e d i n the p r e v i o u s c h a p t e r s , t h e t r u e reasons f o r gender d i f f e r e n c e i n academic achievement are not due t o sex d i f f e r e n c e , but r a t h e r , a combination o f p h y s i c a l , c o g n i t i v e , emotional, s o c i a l f a c t o r s embedded i n gender d i f f e r e n c e . T h e r e f o r e , f o r an i n t e l l i g e n t educator, he o r she should always f i n d s p e c i f i c f a c t o r s behind gender d i f f e r e n c e i n academic achievement i n order t o h e l p  75  program should gather as much i n f o r m a t i o n , p a r t i c u l a r l y  non-  language data, as p o s s i b l e i n o r d e r t o s e l e c t the most p r o m i s i n g a p p l i c a n t s . These types of i n f o r m a t i o n i n c l u d e p r e v i o u s GPA  a t Ritsumeikan  U n i v e r s i t y , l e t t e r s of  recommendation, s c o r e s i n academic a p t i t u d e t e s t , p e r s o n a l statements  and  of i n t e r e s t s . More f a c t o r s such  as  c u l t u r a l knowledge, LI l e v e l , m o t i v a t i o n , i n t e l l i g e n c e ,  and  p e r s o n a l i t y , should be taken i n t o c o n s i d e r a t i o n i n making admission d e c i s i o n s . 3. The  f i n d i n g s r e v e a l t h a t TOEFL s c o r e s alone do  not  a b s o l u t e l y ensure academic success. Thus, i t i s recommended t h a t the c u r r e n t l y used minimum TOEFL s c o r e of 550  not  used as a requirement  UBC  c o u r s e s . Rather,  for registration i n regular  an a p p r o p r i a t e c r i t i c a l  range of TOEFL  s c o r e s should be e s t a b l i s h e d f o r program admission management. In p a r t i c u l a r , f o r those who  be  and  have low TOEFL  s c o r e s but c l e a r l y show academic p o t e n t i a l , the d e c i s i o n makers i n the program should have a s p e c i a l p o l i c y f o r them so as t o s a t i s f y t h e i r l e a r n i n g needs and  academic  capabilities. The p r e s e n t study may The  have t h e o r e t i c a l i m p l i c a t i o n s .  i s s u e of whether or not TOEFL s c o r e s can p r e d i c t GPA  been debated f o r over 30 y e a r s . The study examined t h o r o u g h l y the u n d e r l y i n g r a t i o n a l e f o r TOEFL p r e d i c t i o n s t u d i e s and proposed a comprehensive framework f o r the  s t u d e n t s , no matter male or female, academic p o t e n t i a l s .  t o achieve  their  has  76 a n a l y s i s o f f a c t o r s a f f e c t i n g academic achievement.  For i t s  r e s e a r c h d e s i g n , the p r e s e n t study c a r e f u l l y c o n s i d e r e d t e c h n i c a l treatments i n p r e d i c t o r c o l l e c t i o n ,  criterion  s e l e c t i o n , a n a l y t i c a l models, r e g r e s s i o n procedure,  and  v a l i d i t y l e v e l s i n o r d e r t o ensure the c o r r e c t e s t i m a t i o n o f TOEFL scores* p r e d i c t i v e v a l i d i t y . For these reasons, i t may be thought t h a t the p r e s e n t study might have taken a f u r t h e r s t e p i n s o l v i n g the 30-year's TOEFL-GPA p u z z l e i n terms o f i t s comprehensive  r a t i o n a l e and i t s improved  methodology.  L i m i t a t i o n s of the study 1. The p r e s e n t i n v e s t i g a t i o n d i d not i n c l u d e more r e l e v a n t non-language p r e d i c t o r s i n t o the m u l t i p l e r e g r e s s i o n a n a l y s i s due t o t h e i r c u r r e n t u n a v a i l a b i l i t y . T h i s might cause p o s s i b l e s p e c i f i c a t i o n e r r o r s i n the m u l t i p l e r e g r e s s i o n model used. 2. GPA  used i n the study was  from t h r e e "Bridge  Courses" designed s p e c i f i c a l l y f o r the program. Compared t o r e g u l a r UBC  courses, these courses may  have d i f f e r e n t  f e a t u r e s such as course grade standards, i n s t r u c t o r t e a c h i n g a s s i s t a n t ' s a l l o c a t i o n , and communicative environments.  The uniqueness of t h i s type of GPA  and language  might make  u n c e r t a i n the v a l i d i t y and g e n e r a l i z a b i l i t y of the study. 3. The study d i d not estimate the e f f e c t of r e s t r i c t i o n of range i n TOEFL s c o r e s on the r e s u l t s of the m u l t i p l e r e g r e s s i o n s a n a l y s i s . Research l i t e r a t u r e has i n d i c a t e d t h a t r e s t r i c t i o n of range i n admissions w i l l r e s u l t i n u n d e r e s t i m a t i o n of the p r e d i c t i v e v a l i d i t y , but we  still  77  need e m p i r i c a l  evidence t o know t o what extent and under  what circumstances t h i s u n d e r e s t i m a t i o n may occur. Directions  f o r future  research  1. The p r e s e n t study can be expanded i n t o a  time-series  r e s e a r c h p r o j e c t . On t h e b a s i s o f the data a v a i l a b l e f o r f o u r y e a r s (1991-1995), we can examine t h e change p a t t e r n o f TOEFL's p r e d i c t i v e v a l i d i t y on f i r s t term's GPA over y e a r s . It  i s a l s o f e a s i b l e t o compare t h e r e l a t i o n s h i p o f TOEFL's  p r e d i c t i v e v a l i d i t y t o d i f f e r e n t k i n d s o f GPA, such as second term's GPA and f i r s t y e a r ' s GPA. 2. Other a n a l y t i c a l models and s t a t i s t i c a l can be used i n t h e study. For i n s t a n c e , Analysis  techniques  Multivariate  o f V a r i a n c e can be used t o analyze d i f f e r e n t GPA  subscores; H i e r a r c h i c a l L i n e a r Model can be adopted t o examine t h e s p e c i f i c e f f e c t s o f d i f f e r e n t u n i t s such as i n d i v i d u a l , group, course, and i n s t r u c t o r , on TOEFL's p r e d i c t i v e v a l i d i t y ; Linear  Structural Relations  employed t o d i s t i n g u i s h d i r e c t and i n d i r e c t among v a r i a b l e s  can be  relationships  and assess t h e extent o f measurement e r r o r  t h a t may appear. 3. A s e r i e s o f p r e d i c t i o n s t u d i e s  can be developed t o  compare t h e p r e d i c t i v e v a l i d i t y o f TOEFL with those o f o t h e r language t e s t s such as Michigan T e s t o f E n g l i s h Proficiency English  language  (MTELP) and C e r t i f i c a t e o f P r o f i c i e n c y i n  (CPE), and those o f w i t h a p t i t u d e  t e s t s such as  Graduate Record Examinations (GRE) and S c h o l a s t i c Test  (SAT).  Aptitude  78  4. I t should be f u r t h e r examined how p r o f i c i e n c y , i n p a r t i c u l a r , speaking,  language  listening,  reading,  and w r i t i n g , are r e l a t e d t o academic achievement. 5. Case s t u d i e s can be conducted s p e c i a l i s s u e s i n depth.  t o e x p l o r e some  For i n s t a n c e , why  are some students  w i t h good TOEFL s c o r e s unable t o achieve academic Why  success?  do some o t h e r students e v e n t u a l l y overcome t h e i r  language problems and meet t h e i r academic c h a l l e n g e s ? What d i f f e r e n c e s e x i s t between female and male students i n m o t i v a t i o n , c u l t u r a l a d a p t a b i l i t y , IQ, p r e v i o u s GPA,  and  o t h e r domains. 6. D e c i s i o n theory  (see Cronbach & G l a s e r , 1965)  should  be i n t r o d u c e d i n order t o use TOEFL s c o r e s p r o p e r l y f o r admissions  decision-making  and program management.  Conclusion The p r e s e n t study employed TOEFL s c o r e s as w e l l as o t h e r p r e d i c t o r s t o p r e d i c t f i r s t term's GPA  with a m u l t i p l e  r e g r e s s i o n h i e r a r c h i c a l a n a l y t i c approach. For the UBC/Ritsumeikan Academic Exchange Program s t u d e n t s ,  the  f o l l o w i n g c o n c l u s i o n s can be drawn from the f i n d i n g s of the present  study:  1. TOEFL t o t a l s c o r e s alone have a medium l e v e l of p r e d i c t i v e v a l i d i t y on f i r s t term's  GPA.  2. TOEFL S e c t i o n scores I I , s e c t i o n I s c o r e s , s e c t i o n I I I s c o r e s have the p r e d i c t i v e v a l i d i t y on term's GPA  a t medium, s m a l l , and n e g l i g i b l e  respectively.  levels  and first  3.  W r i t i n g s c o r e s alone have a s m a l l l e v e l o f  predictive validity 4.  Speaking  on f i r s t term's  GPA.  s c o r e s alone have a n e g l i g i b l e  predictive validity  on f i r s t term's  level  GPA.  5. Gender alone has a medium l e v e l o f p r e d i c t i v e validity  on f i r s t term's  GPA.  80  Bibliography Adamson, H. D. (1990). ESL students' s k i l l s i n content courses.  use o f academic  E n g l i s h f o r S p e c i f i c Purpose, 9.  67-87. Alderman, D. L. (1982). Language P r o f i c i e n c y as a Moderator V a r i a b l e i n T e s t i n g Academic A p t i t u d e . Educational  Journal of  Psychology 74. 580-87.  American Psychology A s s o c i a t i o n .  (1994). P u b l i c a t i o n  Manual o f American Psychology A s s o c i a t i o n  (4th e d . ) .  Washington, DC: American Psychology A s s o c i a t i o n . American Psychology A s s o c i a t i o n . educational  (1985). Standards f o r  and p s y c h o l o g i c a l t e s t s . Washington, DC:  American Psychology A s s o c i a t i o n . Ames, R. & Ames, C. (1984) Research on m o t i v a t i o n i n education  (volume 1 ) . Orlando, F l o r i d a : Academic Press, I n c .  Arena, L. (ed.) (1990). Language p r o f i c i e n c y : d e f i n i n g , teaching,  & t e s t i n g . New York: Plenum  Press.  Ayers, J . B. & Peters R. M. (1977). P r e d i c t i v e v a l i d i t y of t h e T e s t o f E n g l i s h as a F o r e i g n Language f o r A s i a n graduate students i n e n g i n e e r i n g , Educational  chemistry, o r mathematics.  & P s y c h o l o g i c a l Measurement, 37, 461-463.  Ayers, J . B. & Quattlebaum, R. F. (1992). TOEFL Performance and Success i n a Masters Program i n E n g i n e e r i n g . Educational  and P s y c h o l o g i c a l Measurement, 52, 973-75.  The s t y l e o f t h e present t h e s i s f o l l o w e d c o n s i s t e n t l y the f o u r e d i t i o n o f P u b l i c a t i o n Manual o f t h e American P s y c h o l o g i c a l A s s o c i a t i o n (APA, 1994) throughout t h e t h e s i s .  81  Bachman, L. F. to  (1991). What does language t e s t i n g have  o f f e r ? TESOL Q u a r t e r l y . 25. 671-704. Bachman, L. F.  (1990). Fundamental c o n s i d e r a t i o n s i n  language t e s t i n g . Hong Kong: Oxford U n i v e r s i t y P r e s s . Bachman, L. F., Davidson, F., & Foulkes J . (1990). A comparison  o f the a b i l i t i e s  measured by the Cambridge and  e d u c a t i o n a l t e s t i n g s e r v i c e EFL t e s t b a t t e r i e s . Issues i n A p p l i e d L i n g u i s t i c s . 1. 30-54. Berry, W. Beverly H i l l s ,  (1993). Understanding r e g r e s s i o n CA:  assumptions.  Sage P u b l i c a t i o n s , Inc.  Berry, W & Feldman, S. p r a c t i c e . Beverly H i l l s ,  CA:  (1985). M u l t i p l e r e g r e s s i o n i n Sage P u b l i c a t i o n s ,  Berwick, R. & McMichael, W.  Inc.  (1992). 1991-92 Ritsumeikan  e v a l u a t i o n r e p o r t . Unpublished manuscript, U n i v e r s i t y o f British  Colombia, Vancouver,  British  Berwick, R. & McMichael, W.  Columbia,  Canada.  (1993). 1992-93 Ritsumeikan  e v a l u a t i o n r e p o r t . Unpublished manuscript, U n i v e r s i t y o f British  Columbia, Vancouver,  British  Columbia,  Canada.  Best, J . & Kahn, J . (1989). Research i n e d u c a t i o n (6th ed.).  Englewood C l i f f s , NJ:  Prentice-Hall.  Black, J . (1991). Performance and o v e r a l l academic  achievement.  i n E n g l i s h s k i l l s courses TESL Canada J o u r n a l ,  9,  42-56. Bloom, B. e t a l . (1964). Taxonomy o f e d u c a t i o n a l o b j e c t i v e s : Handbook I I A f f e c t i v e domain. New McKey Company, Inc.  York: David  82  Bloom, B. e t a l . (1956). Taxonomy of e d u c a t i o n a l o b j e c t i v e s : Handbook I C o g n i t i v e domain. New  York:  David  McKey Company, Inc. Bosher, S. & Rowekamp, J . (1992). Language p r o f i c i e n c y and academic s u c c e s s : the refugee/immigrant  i n higher  e d u c a t i o n . E a s t Lansing, MI: N a t i o n a l Center f o r Research  on  Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No. ED353914) Bruner, J . (1966). Toward a t h e o r y of i n s t r u c t i o n . Cambridge, MA:  Harvard U n i v e r s i t y P r e s s .  Buteyn, R. J . (1989). Gender and academic  achievement  i n e d u c a t i o n . E a s t Lansing, MI: N a t i o n a l Center f o r Research on Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No. ED313103) C a r l s o n , S. B.  (1985). R e l a t i o n s h i p of admission t e s t  s c o r e s t o w r i t i n g performance  of n a t i v e and  nonnative  speakers of E n g l i s h . E a s t Lansing, MI: N a t i o n a l Center f o r Research on Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No.  ED268135)  Ching, R. J . & Moore, C. A. What we  (1993). ESL  assessment:  l e a r n when we open Pandora s box. M e t r o p o l i t a n  Universities,  1  3. 35-46.  Choy, S. C ,  Davenport,  B. M.  (1986). The TOEFL:  incomplete t e s t of E n g l i s h p r o f i c i e n c y . C o l l e g e Teaching. 34. 108-110. C h r i s t o p h e r , V.  (1993). D i r e c t and i n d i r e c t  t e s t s c o r e s as measures of language p r o f i c i e n c y  placement and  83  p r e d i c t o r s o f academic success f o r ESL s t u d e n t s . Unpublished master's t h e s i s , U n i v e r s i t y o f B r i t i s h Columbia,  Vancouver,  B r i t i s h Columbia, Canada. Cocking, R & M e s s i e r , J . (1988). L i n g u i s t i c and c u l t u r a l influences  on l e a r n i n g mathematics. H i l l s d a l e , NJ:  Lawrence Erlbaum A s s o c i a t e s ,  Publishers.  Cohen, J . (1988). S t a t i s t i c a l power a n a l y s i s behavioral  sciences  (2nd e d . ) . H i l l s d a l e , NJ:  Cohen, J & Cohen, P. regression/correlation (2nd  research  LEA.  Multiple  a n a l y s i s f o r the b e h a v i o r a l  e d . ) . H i l l s d a l e , NJ: C o l l i e r . V. P.  (1983). A p p l i e d  f o r the  sciences  LEA.  (1989). How  long? A s y n t h e s i s  of  on academic achievement i n a second language. TESOL  Quarterly,  23, 509-531.  C r a n d a l l , J . & Dale, T.  (1987). ESL through c o n t e n t -  area i n s t r u c t i o n : mathematics, Eaglewood  C l i f f s , NJ:  Crowhurst, M. curriculum.  science,  s o c i a l studies.  Prentice-Hall.  (1994). Language and l e a r n i n g a c r o s s t h e  Scarborough, O n t a r i o :  A l l y n & Bacon Canada.  Cummins, J . (1992). Language p r o f i c i e n c y ,  bilingualism,  and academic achievement. In Richard-Clmato, P. A. & Snow, M. A.  (eds.) The m u l t i c u l t u r a l classroom (pp. 16-69).  London: Longman. Cummins, J . (1991). Language development  and  academic  l e a r n i n g . In Malave, L & Duguette, G.  (Eds.).  Culture  Clevedon, England:  and c o g n i t i o n ,  (pp. 161-189).  M u l t i l i n g u a l Matters Ltd.  Language,  84  Cummins, J . (1984). B i l i n g u a l i s m and s p e c i a l  education:  i s s u e s i n assessment and pedagogy. Clevedon, England: M u l t i l i n g u a l Matters L t d . DeMauro, G. (1992). Examination o f t h e R e l a t i o n s h i p s among TSE, TWE,  and TOEFL Scores. Language T e s t i n g . 9. 149-  161. E d u c a t i o n a l T e s t i n g S e r v i c e . (1994a). TOEFL t e s t and s c o r e manual supplement. P r i n c e t o n , NJ: E d u c a t i o n a l T e s t i n g Service. E d u c a t i o n a l T e s t i n g S e r v i c e . (1994b). TOEFL Update. P r i n c e t o n , NJ: E d u c a t i o n a l T e s t i n g S e r v i c e . E d u c a t i o n a l T e s t i n g S e r v i c e . (1994c). 1994-95 B u l l e t i n of i n f o r m a t i o n f o r TOEFL. TWE.  and TSE. P r i n c e t o n , NJ:  Educational Testing Service. E d u c a t i o n a l T e s t i n g S e r v i c e . (1994d). The Researcher. P r i n c e t o n , NJ: E d u c a t i o n a l T e s t i n g S e r v i c e . E d u c a t i o n a l T e s t i n g S e r v i c e . (1994e). C h e c k l i s t f o r u s i n g TOEFL s c o r e s . P r i n c e t o n , NJ: E d u c a t i o n a l  Testing  Service. E d u c a t i o n a l T e s t i n g S e r v i c e . (1994f). TOEFL 1994-1995: T e s t o f E n g l i s h as a f o r e i g n language I n s t i t u t i o n a l t e s t i n g program oversea e d i t i o n . P r i n c e t o n , NJ: E d u c a t i o n a l  Testing  Service. E d u c a t i o n a l T e s t i n g S e r v i c e . (1992a). G u i d e l i n e s f o r the use o f TOEFL s c o r e s . P r i n c e t o n NJ: E d u c a t i o n a l Service.  Testing  85  Educational  Testing Service.  (1992b). TOEFL t e s t and  score manual (1992-93). P r i n c e t o n ,  NJ: E d u c a t i o n a l  Testing  Service. Educational  Testing Service.  (1991). G u i d e l i n e s f o r  TOEFL i n s t i t u t i o n a l v a l i d i t y s t u d i e s . P r i n c e t o n , Educational  Testing  Educational standardized Educational  Service.  Testing Service.  (1989). The uses o f  t e s t s i n American education. Testing  NJ:  Princeton,  NJ:  Service.  F i s c h e r , K. W. & Lazerson, A. (1984). Human development. New York: W. H. Freeman & Company. F l e t c h e r , J . & S t e r n , R. adaptation:  (1986). Language s k i l l s and  A study o f f o r e i g n students i n a Canadian  u n i v e r s i t y . Curriculum  Inquiry.  19, 293-308.  Gradamn, H. L. & Hanania, E. (1991). Language l e a r n i n g background f a c t o r s and ESL p r o f i c i e n c y . Modern Language J o u r n a l . 75, 39-51. Graham, J . G.  (1987). E n g l i s h language p r o f i c i e n c y and  the p r e d i c t i o n o f academic success.  TESOL Q u a r t e r l y .  21.  505-521. Gue, L. R. & Holdaway, E. A. (1973). E n g l i s h p r o f i c i e n c y t e s t s as p r e d i c t o r s o f success i n graduate s t u d i e s i n education.  Language Learning,  23, 89-103.  Hackett, G. e t a l . (1992). Gender, e t h n i c i t y , and s o c i a l c o g n i t i v e f a c t o r s p r e d i c t i n g the academic achievement of students i n e n g i n e e r i n g .  East Lansing,  MI:  National  86  Center f o r Research on Teacher Learning. Reproduction S e r v i c e No. Hale, G. A.,  (ERIC Document  EJ454109)  e t a l . (1983). E f f e c t s of t e s t d i s c l o s u r e  on performance on the T e s t of E n g l i s h as a F o r e i g n Language L e a r n i n g . Hale, G.A. Bibliography,  449-464.  e t a l . (1984). A Comprehensive TOEFL 1963-82. Modern Language J o u r n a l , 6 8 ,  Hale, G. A. content:  33.  Language.  (1988). Student major f i e l d and  45-51.  text  i n t e r a c t i v e e f f e c t s on r e a d i n g comprehension i n the  T e s t of E n g l i s h as a F o r e i g n Language. Language T e s t i n g ,  5,  49-61. Hale, G. A.,  S t a n f i e l d , C.,  & Duran, R.  (1984).  Summaries of S t u d i e s I n v o l v i n g the T e s t of E n g l i s h as a F o r e i g n Language 1963-1982. P r i n c e t o n , NJ:  Educational  Testing Service. Ho,  D. Y. F. & Spinks,  J . A.  (1985). M u l t i v a r i a t e  p r e d i c t i o n of academic performance by Hong Kong U n i v e r s i t y students.  Contemporary E d u c a t i o n a l  Hosley, D. Meredith, K .  Psychology, 19,  (1979). I n t e r - and  c o r r e l a t e s of the TOEFL. TESOL Q u a r t e r l y . Hu,  S. P.  13,  performance of i n t e r n a t i o n a l graduate students.  Hughes, A.  intra-test  209-217.  (1991). E n g l i s h p r o f i c i e n c y and  A b s t r a c t I n t e r n a t i o n a l , 52,  249-259.  academic Dissertation  1626-A.  (1989). T e s t i n g f o r language t e a c h e r s .  York: Cambridge U n i v e r s i t y Press.  New  87  Husen T. & P o s t l e t h w a i t e , N. (Eds.)(1992). The i n t e r n a t i o n a l e n c y c l o p e d i a o f e d u c a t i o n a l ( V o l s . 1-12). New York:  Pergamon. K i s h o r , N. (1994). EPSE596 course notes. Unpublished  manuscript, U n i v e r s i t y o f B r i t i s h Columbia,  Vancouver,  B r i t i s h Columbia, Canada. Kwang, K. & Dizney, H. F. (1970). P r e d i c t i v e  validity  of t h e TOEFL f o r Chinese graduate students a t an American u n i v e r s i t y . E d u c a t i o n a l and P s y c h o l o g i c a l Measurement. 30. 475-477. L i g h t , R. L., & Wan, T. (1991). S o v i e t s t u d e n t s a t U. S. C o l l e g e s : s o c i a l p e r c e p t i o n s , language p r o f i c i e n c y , and academic  achievement.  TESOL Q u a r t e r l y . 25. 179-185.  L i g h t , R. L., Xu,  M., & Mossop, J . (1987). E n g l i s h  P r o f i c i e n c y and Academic Performance  of I n t e r n a t i o n a l  Students. TESOL Q u a r t e r l y . 21. 251-61. Johns A. M. (1981). Necessary E n g l i s h : a f a c u l t y survey. TESOL Q u a r t e r l y . 15. 51-57. Maccoby. E. & J a c k l i n C. (1974). The psychology o f sex d i f f e r e n c e s . S t a n f o r d , CA: S t a n f o r d U n i v e r s i t y P r e s s . Mestre, J . P. (1981). P r e d i c t i n g academic  achievement  among b i l i n g u a l H i s p a n i c c o l l e g e t e c h n i c a l s t u d e n t s , E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 41, 1266-1264. Mohan, B. (1986). Language and c o n t e n t . Reading, Mass: Addison-Wesley.  gg  Morgen, B. S.  (1990). A comparative study of the use o f  s t a n d a r d i z e d E n g l i s h language p r o f i c i e n c y t e s t s by U.  S.  graduate s c h o o l s . C o l l e g e and U n i v e r s i t y . 65. 295-307. M u l l e i n , K. A. 1978). D i r e c t e v a l u a t i o n of second language p r o f i c i e n c y : the e f f e c t o f r a t e r and s c a l e i n o r a l i n t e r v i e w s . Language L e a r n i n g . 28. 301-308. N o r u s i s , M. S t a t i s t i c s Guide  (1994). SPSS-Window: i n t r o d u c t o r y ( f o r SPSS-window r e l e a s e 6.0). Chicago, I L :  SPSS Inc. N o r u s i s , M. Guide  (1988). SPSS-X I n t r o d u c t o r y S t a t i s t i c s  ( f o r SPSS-X r e l e a s e 3). Chicago, I L : SPSS Inc. N o r u s i s , M.  (1985). SPSS-X Advanced S t a t i s t i c s  Guide.  Chicago, I L : SPSS Inc. Nunan, D. l e a r n i n g . New  (1992). Research methods i n language York: Cambridge U n i v e r s i t y P r e s s .  Oltman, P. K.,  S t r i e k e r , L. J.(1988). How  native  language and l e v e l of E n g l i s h p r o f i c i e n c y a f f e c t the s t r u c t u r e o f the T e s t o f E n g l i s h as a F o r e i g n Language (TOEFL). E a s t Lansing, MI: N a t i o n a l Center f o r Research Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e  on  No.  ED296592) O s t l e r , S. advanced  (1980). A survey o f academic  needs f o r  ESL. TESOL Q u a r t e r l y . XIV. 489-501.  Othuon, L.  (1993). A study o f the p r e d i c t i v e  validity  of the Kenya c e r t i f i c a t e o f primary e d u c a t i o n examination: A p p l i c a t i o n o f H i e r a r c h i c a l L i n e a r Models.  Unpublished  89  master's t h e s i s , U n i v e r s i t y o f B r i t i s h Columbia,  Vancouver,  B r i t i s h Columbia, Canada. Pack, A. C. A (1972). Comparison between TOEFL and M i c h i g a n T e s t s c o r e s and student success i n (1) freshman and 2) completing a c o l l e g e program. TESL Reporter, 5, Parkerson, J . A., Lomax, R. G.,  Schiller,  1-7  & 9.  D. & Walberg,  H. J . (1984). E x p l o r i n g c a u s a l models o f e d u c a t i o n a l achievement. J o u r n a l o f E d u c a t i o n a l Psychology, 76, 63 8-64 6. Patkowski, M. S.  (1991). B a s i c s k i l l s t e s t s and  academic s u c c e s s o f ESL c o l l e g e s t u d e n t s . TESOL Q u a r t e r l y , 25. 735-738. Pedhazur, E. (1982). M u l t i p l e r e g r e s s i o n i n b e h a v i o r a l r e s e a r c h . New  York: CBS C o l l e g e P u b l i s h i n g .  Pedhazur, E & Schmelkin. (1991). Measurement, d e s i g n , and a n a l y s i s : I n t e g r a t e d approach. H i l l s d a l e ,  NJ:  LEA,  Publishers. P e i r c e , B. N.  (1992). D e m y s t i f y i n g the TOEFL Reading  T e s t . TESOL Q u a r t e r l y , 26, 665-691. P e r k i n s , K.  (1988). Measuring ESL Readers' A b i l i t y t o  Apply Reasoning i n Reading: A V a l i d i t y Reading Comprehension  Study of the TOEFL  S u b t e s t . J o u r n a l o f Research i n  Reading. 11. 36-49. P e r k i n s , K. & P a r i s h , C.  (1988). What's wrong w i t h  r e a d i n g comprehension t e s t s ? E a s t L a n s i n g , MI:  National  Center f o r Research on Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No. ED297305)  90  Perry, W.  S.  (1989). The r e l a t i o n s h i p o f the T e s t o f  E n g l i s h as a F o r e i g n language  (TOEFL) and o t h e r c r i t i c a l  v a r i a b l e s t o the academic performance  of i n t e r n a t i o n a l  graduate s t u d e n t s . D i s s e r t a t i o n A b s t r a c t I n t e r n a t i o n a l .  50.  422A. Powers, D. E.  (1985). A Survey o f academic demands  r e l a t e d t o l i s t e n i n g s k i l l s . T e s t o f E n g l i s h as a F o r e i g n Language Research Reports Number 20. E a s t Lansing, MI: N a t i o n a l Center f o r Research on Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No. Raimes, A.  ED3 04011)  (1990). The TOEFL t e s t o f w r i t t e n  English:  cause f o r concern. TESOL Q u a r t e r l y . 24. 227-243. Reed, D. J . (1992). The r e l a t i o n s h i p between c r i t e r i o n based l e v e l s o f o r a l p r o f i c i e n c y and norm-referenced s c o r e s of  g e n e r a l p r o f i c i e n c y i n E n g l i s h as a second  language.  System. 20. 329-345. Ritsumeikan U n i v e r s i t y .  (1992). Ritsumeikan U n i v e r s i t y  (1992-1993). Kyoto, Japan: Ritsumeikan U n i v e r s i t y . R i v e r a , C.  (Ed.)  academic achievement.  (1984). Language p r o f i c i e n c y  and  Clevedon, England: M u l t i l i n g u a l  Matters Ltd. Rosenthal, R. & Jacobson, L.  (1968). Pygmalion  classroom; t e a c h e r e x p e c t a t i o n and p u p i l ' s development. New Santos, T.  York: R i n e h a r t and  i n the  intellectual  Winston.  (1988). P r o f e s s o r s ' r e a c t i o n s t o the  academic w r i t i n g o f nonnative-speaking s t u d e n t s . TESOL Q u a r t e r l y . 22. 69-90.  91  Sharon, A. T. (1972). E n g l i s h p r o f i c i e n c y ,  verbal  a p t i t u d e , and f o r e i g n student success i n American  graduate  s c h o o l s . E d u c a t i o n a l & P s y c h o l o g i c a l Measurement, 32, 425431. Snow, C. E., Barnes W. S., Chandler J . , Goodman I . F., & Hemphill L. (1991). U n f u l f i l l e d e x p e c t a t i o n s : Home and s o c i a l i n f l u e n c e s on l i t e r a c y . Cambridge, MA: Harvard U n i v e r s i t y Press. Spolsky, B. (1990). The P r e h i s t o r y o f TOEFL. Language T e s t i n g , 7, 98-118. Sposky, B. (1979). Some major t e s t s . V i r g i n i a : The Center f o r A p p l i e d  Arlington,  Linguistics.  Sposky, B. (1978). Approaches t o Language T e s t i n g . A r l i n g t o n , V i r g i n i a : The Center f o r A p p l i e d  Linguistics.  S p r i n t h a l l , N. & S p r i n t h a l l , R. (1994). E d u c a t i o n a l psychology: a developmental approach  (6th e d . ) . New York:  McGraw-Hill. Swinton,  S. S. & Powers, D. E. (1980). F a c t o r a n a l y s i s  of t h e T e s t o f E n g l i s h as a F o r e i g n Language f o r s e v e r a l language groups. TOEFL Research Reports, 6. E a s t L a n s i n g , MI: N a t i o n a l Center f o r Research on Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No. ED218921) Tabachnick, B. G. & F i d e l l , L. S. (1989). U s i n g multivariate s t a t i s t i c s Inc.  (2nd e d . ) . H a r p e r C o l l i n s P u b l i s h e r s ,  92  Traynor, R.  (1985). The TOEFL: An a p p r a i s a l .  East  Lansing, MI: N a t i o n a l Center f o r Research on Teacher Learning.  (ERIC Document Reproduction S e r v i c e No.  U n i v e r s i t y o f B r i t i s h Columbia.  (1993). The U n i v e r s i t y  of B r i t i s h Columbia Calendar 1993-94. Vancouver, U n i v e r s i t y of B r i t i s h Ushasree, S.  EJ315491)  Canada:  Columbia.  (1990). Academic  adjustment and  scholastic  achievement among the s o c i a l l y disadvantaged. T i r u p a t i , I n d i a : S r i Venkateswara  University.  Verhoeven, L. & Jong, J . (eds.) (1992). The  construct  of language p r o f i c i e n c y : a p p l i c a t i o n s o f p s y c h o l o g i c a l models t o language assessment. P h i l a d e l p h i a : John  Benjamins  Publisher. V i g o t s k y , L. (1986). Thought and language T r a n s . ) . Cambridge,  MA:  MIT  ( K o z u l i n A.,  Press.  Vinke, A. A. & Jochems, W. M. G.  (1993). E n g l i s h  p r o f i c i e n c y and academic success i n i n t e r n a t i o n a l postgraduate e d u c a t i o n . Higher E d u c a t i o n , 26, 275-285. Walberg, H. J . & H a e r t e l , G. D.  (eds.) (1990). The  i n t e r n a t i o n a l encyclopedia of educational evaluation.  New  York: Pergamon P r e s s . Wan, Academic  T., Chapman. D. W.,  & Biggs, D. A.  (1992).  achievement s t r e s s o f i n t e r n a t i o n a l s t u d e n t s  a t t e n d i n g U. S. u n i v e r s i t i e s . Research i n Higher E d u c a t i o n , 33. 607-623.  9 3  Willingham, W. M.  (1990). P r e d i c t i n g c o l l e g e grades: An  a n a l y s i s o f i n s t i t u t i o n a l t r e n d s over two  decades.  P r i n c e t o n , NJ: E d u c a t i o n a l T e s t i n g S e r v i c e . Willingham, W. M. w i t h Young, J . W.,  & M o r r i s , M.  M.  (1985). Success i n c o l l e g e : the r o l e o f p e r s o n a l q u a l i t i e s and academic  a b i l i t y . New  Wilson, K. M.  York: C o l l e g e Entrance Board.  (1982). A comparative a n a l y s i s o f TOEFL  examinee c h a r a c t e r i s t i c s . 1977-1979. E a s t Lansing, MI: N a t i o n a l Center f o r Research on Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No. ED237512) Wilson, K. M.  (1986). The r e l a t i o n s h i p of GRE  General  T e s t s c o r e s t o f i r s t - y e a r grades f o r f o r e i g n graduate s t u d e n t s : Report o f a C o o p e r a t i v e Study. E a s t Lansing, MI: N a t i o n a l Center f o r Research on Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No. ED281862) Wilson, K. M.  (1987). P a t t e r n s o f t e s t t a k i n g and s c o r e  change f o r examinees who  repeat the T e s t of E n g l i s h as a  F o r e i g n Language. E a s t Lansing, MI: N a t i o n a l Center f o r Research on Teacher L e a r n i n g . (ERIC Document Reproduction S e r v i c e No.  ED283831)  Wimberley, D. W.,  McCloud, D. & F l i n n , W.  (1992).  P r e d i c t i n g Success o f Indonesian Graduate Students i n the U n i t e d S t a t e s . Comparative Wolf, F. for  E d u c a t i o n Review, 36. 487-508.  (1986). M e t a - a n a l y s i s : Q u a n t i t a t i v e methods  research synthesis. Beverly H i l l s ,  P u b l i c a t i o n s , Inc.  CA:  Sage  94  Yan,  Z.  (1994). Range r e s t r i c t i o n  selection: a preliminary  and i n d i c a t o r  review o f s i x p r e d i c t i o n s t u d i e s  r e l a t i o n s h i p between TOEFL s c o r e s and GRE. manuscript, U n i v e r s i t y o f B r i t i s h Columbia, B r i t i s h Columbia, Yan,  Z.  Unpublished Vancouver,  Canada.  (1994). A m e t a - a n a l y s i s on s t u d i e s o f TOEFL  s c o r e s ' p r e d i c t i v e v a l i d i t y on f i r s t y e a r ' s GPA Unpublished manuscript, U n i v e r s i t y o f B r i t i s h Vancouver,  on  B r i t i s h Columbia,  Young, J . W.  (1964-1994). Columbia,  Canada.  (1993). Grade adjustment methods. Review  of E d u c a t i o n Research. 63. 151-165. Yule, G.,  Hoffman, P.  (1990). P r e d i c t i n g Success f o r  I n t e r n a t i o n a l Teaching A s s i s t a n t i n a U. S. U n i v e r s i t y . TESOL Q u a r t e r l y . Zeidner, M.  24. 227-43. (1987). A comparison o f e t h n i c ,  age b i a s i n the p r e d i c t i v e v a l i d i t y o f E n g l i s h aptitude  sex,  and  language  t e s t s : some I s r a e l i d a t a . Language T e s t i n g .  4.  55-  71. Z i r p o l i , T. J . , H a l l l a h a n ,  D. P.,  & Kneedler, R.  D.  (1988). The Indonesian p r o j e c t : c o r r e l a t e s o f student performance  i n a s p e c i a l education teachers t r a i n i n g  program. I n t e r n a t i o n a l J o u r n a l  of S p e c i a l E d u c a t i o n . 3.  73-  79. Zhang S.,  Dunkel. P. & Zubovic, Y.  (1992). R e g r e s s i o n  models i n an ESL c o n t e x t : i s s u e s i n c o n s t r u c t i o n i n t e r p r e t a t i o n . TESOL Q u a r t e r l y .  26. 191-196.  and  95  Appendix I  file  educ395a ened379 ened206a 76.00 60 .00 50 .00 56.00 49.67 43 .00 65.00 52 .00 50.00 41.00 58 .33 58.00 57 .00 60.00 58.33 66.00 42 .00 67.00 59 .33 45.00 58.00 75.00 70.00 59.33 31.00 77 .00 59 .67 57 .00 50.00 72 .00 61.33 44.00 70 .00 70 .00 55.00 70.00 60 .00 61. 67 60.00 62 .00 74 . 00 52 .00 62.00 53.0& 62 .33 72 .00 66.00 47.00 62 .33 74.00 58.00 60.00 62 .67 70.00 70.00 43 .00 62.67 75.00 66.00 51.00 63 . 00 72 .00 57 .00 60 .00 63 .33 73 .00 53 .00 62.00 63 .33 75.00 50.00 67 .00 75.00 64 .00 57 .00 60.00 77 . 00 64 . 67 63 .00 63 .00 69 .00 65 . 00 65.00 58.00 72 .00 65 . 00 58.00 62.00 75.00 65.00 65.00 64.00 67 .00 65 .33 63 .00 66.00 73 .00 67 .33 60.00 68.00 74 .00 67 .33 61.00 68.00 74 .00 67.67 52.00 68.00 83 .00 67 . 67 65.00 71.00 68 .00 68 .00 70.00 60.00 74.00 68 . 00 58.00 70.00 76 .00 68 . 00 70.00 60.00 76.00 68.67 61.00 71.00 75.00 69 .00 66.00 68.00 74.00 69.33 74.00 60.00 76 .00 70 . 00 60.00 68.00 82 .00 70 .00 83.00 74.00 54 .00 70.33 60.00 72.00 79 .00 70.33 68.00 63.00 80 .00 70.33 61.00 76.00 75.00 70.67 58.00 74.00 82 .00 71.33 60.00 70.00 84.00 71.33 78.00 72.00 65.00 71.67 64.00 74.00 78 .00 72 .00 70.00 72.00 75.00 72 .33 68.00 70.00 79 .00 72 .33 74.00 74.00 70 .00 72 .67 70.00 72.00 76 .00 72 . 67 71.00 74.00 74.00 73 .00 65.00 70.00 84 .00 73 .00 75.00 74.00 74 .00 .74.33 67.00 77 .00 79 .00 74.33 73.00 71.00 79 .00 74.33 76.00 66.00 81.00 74 .33 67.00 73.00 83 .00 74.33 81.00 70.00 73 .00 74 . 67 77.00 70.00 74 .67 77 .00  ,qp  10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57  The data a  toefl 483.00 483.00 477.00 543.00 497.00 497.00 523.00 523.00 530.00 483.00 510.00 533.00 467.00 513.00 503.00 500.00 517.00 480.00 493.00 530.00 497.00 517.00 483.00 513.00 523.00 490.00 517.00 507.00 500.00 520.00 517.00 490.00 527.00 477.00 530.00 517.00 540.00 487.00 537.00 570.00 550.00 490.00 483.00 513.00 493.00 480.00 550.00 500.00 520.00 520.00 540.00 563.00 470.00 513.00 550.00 497.00 493.00  seel  49.00 45.00 47.00 54.00 49.00  96  qpa 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 . 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97  —  74.67 74.67 74.67 74.67 75.00 75.33 75.33 75.33 75.67 15.61  75.67 75.67 76.00 76.00 76.33 76.67 76.67 77 .00 77.00 77.67 77.67 78.67 79.00 79.00 79.33 19.61  80.00 80.00 80.00 82.33 82 .33 82 .33 82 .67 82 .67 82.67 82.67 83.00 83.33 83.33 87.00  educ3 9 5a ened379 ened206a 70.00 77.00 77 .00 70.00 76.00 78.00 72.00 70.00 82.00 73 .00 67.00 84.00 70.00 78.00 77.00 72.00 75.00 79.00 71.00 73.00 82.00 68.00 74.00 84.00 72 .00 78.00 77.00 77 .00 71.00 79.00 64.00 77 .00 86.00 67.00 71.00 89.00 73 .00 76.00 79.00 74.00 72.00 82 .00 72.00 81.00 76.00 72 .00 83.00 75.00 72 .00 80.00 78.00 78.00 77.00 76.00 72 .00 77 .00 82 .00 74.00 85.00 74.00 74.00 78.00 81.00 71.00 86.00 79.00 77 .00 77 .00 83 .00 70.00 77.00 90.00 78.00 78.00 82.00 73.00 78.00 88.00 78.00 81.00 81. 00 76.00 83.00 81.00 77.00 78.00 85.00 78.00 94.00 75.00 81.00 80.00 86.00 77 . 00 83.00 87 .00 80.00 85.00 83.00 80.00 83.00 85.00 80 . 00 80.00 88.00 78.00 81.00 89.00 87.00 80.00 82 .00 80.00 84.00 86.00 77 .00 83.00 90.00 84.00 87.00 90.00  toef 1 557.00 543.00 473.00 553.00 527.00 470.00 477.00 507.00 540.00 553.00 503.00 503.00 510.00 550.00 500.00 497.00 540.00 487.00 507.00 540.00 503.00 543.00 520.00 540.00 533.00 520.00 543.00 513.00 533.00 480.00 530.00 580.00 573.00 573.00 513.00 497.00 550.00 530.00 560.00 543.00  seel 54.00 49 .00 46.00 55.00 51.00 45.00 44.00 47 .00 56.00 55.00 52.00 48.00 48.00 56.00 41.00 47.00 56.00 39.00 49.00 51.00 50.00 52.00 48.00 50.00 52.00 47 .00 50.00 52 .00 51.00 48.00 50.00 61.00 61.00 56.00 52.00 47.00 59.00 54.00 53.00 56.00  97  1 2 3 4 5  6  7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 •35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 51  sec2 sec3 48.00 48.00 50 .00 50 .00 48 .00 48 .00 54.00 55.00 47 .00 53 .00 51.00 • 51.00 53 .00 52 .00 54.00 53 .00 57 .00 48.00 48 .00 54 .00 52 .00 48.00 58 .00 55.00 50 .00 47 .00 52.00 53 .00 52 .00 49 .00 52 .00 48.00 48.00 54 .00 49 .00 49 .00 52 .00 47 .00 54.00 54.00 53 .00 49 .00 55.00 53 .00 51. 00 48 .00 55.00 54 .00 58 .00 " 51.00 51.00 51.00 49 .00 53 .00. 52 .00 53 .00 51.00 49 .00 54.00 54.00 50.00 53.00 55.00 50.00 61.00 52.00 47 .00 47 .00 52 .00 54.00 53 .00 53 .00 55,00 53 .00 50.00 49.00 56 .00 55.00 68 .00 53 .00 56 .00 51.00 52 .00 48.00 46.00 52 .00 54 .00 51.00 • 54.00 48.00 51.00 48.00 61.00 50.00 53 .00 51.00 58.00 53 .00 56.00 50 .00 54 .00 56.00 58.00 60.00 52 .00 45.00 54.00 49.00 54 .00 59 .00 51.00 50 .OC 51. 0C 50.00  write 2.00 1.00 2.00 3.00 1.00 3.00 1.00 2.00 3 .00 3.00 2.00 4.00 2 .00 3.00 2 .00 3 .00 3.00 2.00 3.00  2.00 2.00 2.00 3 .00 2.00 3 .00  2.00 1.00 3.00 2.00 2.00 4.00 4.00 2.00  2.00 3.00 2.00 4.00 2.00 3.00 3.00 3.00 2.00 2.00 3.00 2.00 2.00 4.00 3.00 4.00  5.00 2.00 2.00 2.00 3.00  3.0C 3.0C 3.0C)  qender 1.00 1.00 1.00 1.00 1.00 2.00 1.00  speak .90 .90 .90 1.90 1.00 1.90 1.90 .90 1.90 1.90 1.90 .90 .90 2.00 1.90 2.00 2.00 1.00 2.00 1.00 1.90 .00 1.00 1.00 .90 1.00 .90 .00 .00 2.00 1.00 .90 .90 2.00 1.00 1.90 .90 1.90 1.00 2.00 2.00 1.00 1.90 .90 1.90 3.00 1.90 1.90 2.00 1.90  .  1.00 1.90 1.00  l.OO 1.0C  toeflpre 483.00  483.00 477.00 543.00  497.00 497.00  507.00 523.00 1.00 530.00 1.00 483.00 1.00 510.00 1.00 533.00 2.00 467.00 1.00 513.00 2.00 503 .00 1.00 500.00 1.00 517.00 . 1.00 480.00 2.00 460.00 1.00 1.00 530.00 497.00 2.00 1.00 517.00 483.00 2.00 513.00 1.00 523.00 1.00 490.00 1.00 517.00 2.00 507.00 1.00 500.00 2.00 520.00 1.00 517.00 1.00 490.00 1.00 487.00 1.00 477.00 1.00 530.00 2.00 517.00 1.00 540.00 2.00 487.00 1.00 523.00 1.00 513.00 2.00 550.00 2.00 490.00 2.00 483.00 1.00 483.00 2.00 493.00 2.00 480.00 1.00 547.00 2.00 487.00 1.00 520.00 1.00 520.00 2.00 520.00 1.00 547.00 1.00 470.00 2.00 493.00 2.00 550.00 1.00 497.00 2 .00 2.00 1 493.00  98  58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97  sec2 57 . 00 60.00 52.00 55.00 54.00 48.00 52.00 55.00 53.00 56.00 51.00 52 . 00 53 .00 56.00 53 .00 52.00 58.00 53.00 52 . 00 61.00 49.00 55.00 56.00 57 .00 56.00 56.00 59.00 52 .00 58.00 49.00 56.00 57.00 58.00 58.00 50.00 52.00 54.00 55.00 61.00 53 .00  sec3 56.00 54.00 44.00 56.00 53.00 48.00 47.00 50.00 53.00 55.00 48.00 51.00 52.00 53 .00 56.00 50.00 48.00 54.00 51.00 50.00 52 .00 56.00 52.00 55.00 52 .00 53 .00 54.00 50.00 51.00 47 .00 53.00 56.00 53 .00 58.00 52.00 50.00 52 .00 50.00 54.00 54.00  write 2.00 2.00 4.00 3 .00 2.00 4.00 3 .00 3 .00 3 .00 2 .00 2.00 2.00 3.00 4.00 2.00 3.00 3 .00 4.00 2.00 2 .00 3 .00 4.00 3 .00 4.00 3 .00 2 .00 2 .00 3 .00 3.00 3 .00 2.00 3 .00 4.00 3.00 3.00 2.00 3.00 4.00 4.00 3.00  speak 1.90 1.90 2.00 1.90 1.90 1.00 1.90 .00 2.00 1.00 1.90 2.00 .00 1.90 1.00 1.00 .90 2.90 1.00 1.90 1.90 2.00 .90 1.90 2.00 1.00 1.00 1.00 1.90 .90 .90 2.00 2 .00 1.00 2.00 .00 2 .00 2 .00 2.00 3 .00  qender 1.00 1.00 2 .00 2.00 2.00 2.00 1.00 2.00 2.00 2 .00 1.00 2.00 2.00 2.00 2.00 2.00 2.00 1.00 2.00 2.00 2.00 2 .00 2.00 1.00 2.00 2 .00 1.00 2.00 1.00 2.00 1.00 2.00 2 .00 2.00 2.00 2.00 2 .00 2.00 2.00 2.00  toefIpre 557.00 517.00 473.00 553.00 527.00 470.00 477.00 507.00 540.00 553.00 503.00 503.00 510.00 550.00 500.00 497.00 520.00 487.00 507.00 540.00 503.00 543.00 520.00 520.00 493.00 513.00 543.00 513.00 533.00 480.00 530.00 577.00 570.00 573.00 513.00 497.00 550.00 530.00 550.00 543.00  Appendix I I  The l i s t of standardized r e s i d u a l s values  qpa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57  78.67 62.33 75.33 75.67 67.67 74.67 82.67 52.00 7 6.67 75.00 74.33 65.00 82.67 65.00 83.33 79.67 71.33 62.67 59.33 61.33 72.67 49.67 72.33 65.33 80.00 83 .33 76.00 76.67 80.00 70.33 69.00 67.67 79.00 77.67 58.33 75.67 74.33 76.33 68.00 82.67 74.33 75.67 87.00 74.67 71.67 74.67 82.67 62.00 73.00 67.33 74.33 79.00 69.33 63.33 59.33 82.33 64.00  zre_tot ! .01185 1 -.26058 .37326 -.16919 -.18236 .76297 1.65650 -2.02214 -.05763 .14203 .32776 -.43412 .26183 -1.09487 .91557 .93115 -.98196 -.62034 -2 .20797 -1.43484 .73991 1 -2.21252 I 1.19734 | -.36852 | 1.47875 .53302 .18139 .49493 ' .81801 -1.27192 .02855 -.67309 .97624 .39796 -2.07501 1.43117 -.03998 .65906 -.70106 .47460 .72534 .55774 1.62708 .22967 1.01556 .58444 1.28205 -.81926 .28247 -.43488 .74850 .56638 .71885 -1.09931 -1.15162 .37496 -.57943  lev tot 1 .03357 .04696 .08908 .02028 .03353 .04023 .06361 .03501 .03317 .03923 .06275 .02645 .06595 .02780 .03011 .02610 .02740 .03393 .02870 .02255 .02744 .05536 .05094 .02076 .02394 .04774 .07434 .02110 .01580 .03182 .03199 .07477 .06004 .04508 .03101 .02962 .01664 .02644 .06366 .08217 .05533 .05063 .07591 .02394 .02852 .05824 .01993 .02711 .05445 .03378 .03733 .01995 .03501 .07792 .07445 .07149 .03394  and l e v e r a g e  z r e sec .00005 -.27775 .45587 -.02946 -.16574 .53343 1.63646 -1.87327 . 00203 .09495 .39459 -.55406 .41617 -1.16161 .98745 .74468 -.86176 -.56548 -2.22204 -1.36927 .68385 -2 .22396 1.15341 -.52355 1.40383 .38734 .18329 .47693 .92246 -1.20212 -.27853 -.52036 .92182 .15958 -1.98387 1.56725 -.04228 .38595 -.60169 .47333 .61600 .48508 1.71437 .23312 1.17619 .57255 1.39658 -.67128 .08020 -.54908 .76494 .44869 .94238 -1.11695 -1.11406 .51442 -.47375  lev sec .05956 .05135 .09910 .03603 .04249 .08277 .06392 .05101 . 11375 .04372 .11503 .04046 .09953 .03486 .04238 .05455 .05509 .04415 . 03879 .09307 .03040 .05557 .05441 .04633 .04503 .06668 .07680 .02220 .02695 .03535 .12271 .10099 .06622 .13103 .03685 .06390 .02403 .16597 .07461 .09221 .08467 .05638 .09016 .02561 .07769 .05847 .04128 .07029 .08797 .05677 .04961 .03077 .07997 .08167 .07686 .08633 .06678  100  58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93  opa 63 .00 64.67 72 .33 83 .00 71.33 65.00 68.67 72.67 68.00 72.00 68.00 70.33 67 .33 77.67 80.00 77 .00 75.33 77 .00 70.00 62.67 73.00 59.67 76.00 79.33 58.33 74.67 74.67 75.67 70.00 82.33 82.33 70.33 74.67 . 63.33 62.33 75.33  zre t o t -.78872 -.64837 .11155 .86294 .03825 -1.21379 -.05833 -.98388 -.07843 -.40896 -.46571 .74946 .23680 .61466 1.56824 1.37816 .11421 .67523 -.93777 -1.92721 -.84607 -1.39631 -.50334 .49989 -1.01632 .27193 .17866 -.12219 .31844 1.60356 2 .09875 -.10089 -.49733 -.94856 -2.44774 1.45488  lev t o t .02744 .03318 .03066 .02740 .06359 .07931 .08838 .07231 .08129 .01664 .05530 .02684 .02513 .02305 .04706 .12488 .07353 .02460 .01693 .01889 .08662 .02875 .03817 .01754 .04976 .09043 .02110 .05862 .02230 .03890 .03482 .02631 .02887 .02250 .02963 .05424  zre sec -.68818 -.64558 -.01559 1.05386 -.03909 -1.18508 -.20536 -1.11125 -.08717 -.47273 -.24876 .82353 .21706 .71146 1.42202 1.11990 .04687 .67218 -.85039 -1.90177 -.86892 -1.37712 -.41273 .44542 -.85125 .24740 .21727 -.10667 .33727 •1.70789 2.04925 -.14970 -.45606 -.68273 -2.65863 1.39603  lev_sec .04791 .03547 .05153 .06134 .07055 .08110 .10605 .12319 .08378 .01968 .10158 .03234 .02765 .05036 .06560 .24115 .08050 .02519 .03661 .03147 .08959 .02915 .04435 .02009 .08982 .11580 .02329 .05977 .02368 .05079 .03743 .03334 .04251 .09999 .08243 .06382  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0078086/manifest

Comment

Related Items