{"Affiliation":[{"label":"Affiliation","value":"Education, Faculty of","attrs":{"lang":"en","ns":"http:\/\/vivoweb.org\/ontology\/core#departmentOrSchool","classmap":"vivo:EducationalProcess","property":"vivo:departmentOrSchool"},"iri":"http:\/\/vivoweb.org\/ontology\/core#departmentOrSchool","explain":"VIVO-ISF Ontology V1.6 Property; The department or school name within institution; Not intended to be an institution name."},{"label":"Affiliation","value":"Language and Literacy Education (LLED), Department of","attrs":{"lang":"en","ns":"http:\/\/vivoweb.org\/ontology\/core#departmentOrSchool","classmap":"vivo:EducationalProcess","property":"vivo:departmentOrSchool"},"iri":"http:\/\/vivoweb.org\/ontology\/core#departmentOrSchool","explain":"VIVO-ISF Ontology V1.6 Property; The department or school name within institution; Not intended to be an institution name."}],"AggregatedSourceRepository":[{"label":"Aggregated Source Repository","value":"DSpace","attrs":{"lang":"en","ns":"http:\/\/www.europeana.eu\/schemas\/edm\/dataProvider","classmap":"ore:Aggregation","property":"edm:dataProvider"},"iri":"http:\/\/www.europeana.eu\/schemas\/edm\/dataProvider","explain":"A Europeana Data Model Property; The name or identifier of the organization who contributes data indirectly to an aggregation service (e.g. Europeana)"}],"Campus":[{"label":"Campus","value":"UBCV","attrs":{"lang":"en","ns":"https:\/\/open.library.ubc.ca\/terms#degreeCampus","classmap":"oc:ThesisDescription","property":"oc:degreeCampus"},"iri":"https:\/\/open.library.ubc.ca\/terms#degreeCampus","explain":"UBC Open Collections Metadata Components; Local Field; Identifies the name of the campus from which the graduate completed their degree."}],"Creator":[{"label":"Creator","value":"Christopher, Virginia Louise","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/creator","classmap":"dpla:SourceResource","property":"dcterms:creator"},"iri":"http:\/\/purl.org\/dc\/terms\/creator","explain":"A Dublin Core Terms Property; An entity primarily responsible for making the resource.; Examples of a Contributor include a person, an organization, or a service."}],"DateAvailable":[{"label":"Date Available","value":"2009-02-20T23:42:12Z","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/issued","classmap":"edm:WebResource","property":"dcterms:issued"},"iri":"http:\/\/purl.org\/dc\/terms\/issued","explain":"A Dublin Core Terms Property; Date of formal issuance (e.g., publication) of the resource."}],"DateIssued":[{"label":"Date Issued","value":"1993","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/issued","classmap":"oc:SourceResource","property":"dcterms:issued"},"iri":"http:\/\/purl.org\/dc\/terms\/issued","explain":"A Dublin Core Terms Property; Date of formal issuance (e.g., publication) of the resource."}],"Degree":[{"label":"Degree (Theses)","value":"Master of Arts - MA","attrs":{"lang":"en","ns":"http:\/\/vivoweb.org\/ontology\/core#relatedDegree","classmap":"vivo:ThesisDegree","property":"vivo:relatedDegree"},"iri":"http:\/\/vivoweb.org\/ontology\/core#relatedDegree","explain":"VIVO-ISF Ontology V1.6 Property; The thesis degree; Extended Property specified by UBC, as per https:\/\/wiki.duraspace.org\/display\/VIVO\/Ontology+Editor%27s+Guide"}],"DegreeGrantor":[{"label":"Degree Grantor","value":"University of British Columbia","attrs":{"lang":"en","ns":"https:\/\/open.library.ubc.ca\/terms#degreeGrantor","classmap":"oc:ThesisDescription","property":"oc:degreeGrantor"},"iri":"https:\/\/open.library.ubc.ca\/terms#degreeGrantor","explain":"UBC Open Collections Metadata Components; Local Field; Indicates the institution where thesis was granted."}],"Description":[{"label":"Description","value":"Much valuable research in the area of English as a second language has focused on\r\ntesting, and though many studies have investigated issues related to test reliability, validity\r\nand the predictive ability of indirect test scores, they do not assess the comparative ability of\r\nindirect and direct test scores to predict academic success. Work of this kind would inform\r\npractice in the area of testing and placement. This study investigates the practical problem of\r\nmaking appropriate placement decisions for students whose test results show wide enough\r\ndiscrepancies to indicate placement in different academic programs, or at different levels\r\nwithin programs. The question of whether scores derived from indirect measures (language\r\nproficiency tests) or direct measures (writing samples) are better indicators of academic\r\nlanguage proficiency is addressed.\r\nThe study also explores the usefulness of grade point average (GPA) as a measure of\r\nacademic success, and proposes the use of average accumulated credit per semester (AACPS)\r\nas an additional measure. Several researchers have questioned the use of GPA as the sole\r\nmeasure of academic success for ESL students, and this study adds to existing research.\r\nThe ability of two types of placement test scores to predict academic success for ESL\r\nstudents in Secondary and University programs is evaluated. Test scores from a) an indirect\r\nmeasure of language proficiency, the Michigan Test of English Language Proficiency, and b)\r\na direct measure, a holistically-scored writing test are assessed as predictors of four measures\r\nof academic success: a) GPA for all courses, b) GPA for courses requiring a higher level of\r\nlanguage proficiency (English or Humanities), c) GPA for courses requiring a lower level of\r\nlanguage proficiency (Math or Sciences), and d) AACPS.\r\nResults of correlation analysis indicated that for University students, academic success\r\nas measured by GPA showed low correlations to both indirect or direct test scores, but direct\r\ntest scores and combined (indirect and direct) test scores correlated moderately well to\r\nacademic success as measured by AACPS. For Secondary students, academic success as\r\nmeasured by GPA for courses requiring a higher level of language proficiency (English or\r\nHumanities) showed moderate correlations to direct test scores. Correlations between test\r\nscores and any other measure of academic success including AACPS were low for Secondary\r\nstudents.","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/description","classmap":"dpla:SourceResource","property":"dcterms:description"},"iri":"http:\/\/purl.org\/dc\/terms\/description","explain":"A Dublin Core Terms Property; An account of the resource.; Description may include but is not limited to: an abstract, a table of contents, a graphical representation, or a free-text account of the resource."}],"DigitalResourceOriginalRecord":[{"label":"Digital Resource Original Record","value":"https:\/\/circle.library.ubc.ca\/rest\/handle\/2429\/4886?expand=metadata","attrs":{"lang":"en","ns":"http:\/\/www.europeana.eu\/schemas\/edm\/aggregatedCHO","classmap":"ore:Aggregation","property":"edm:aggregatedCHO"},"iri":"http:\/\/www.europeana.eu\/schemas\/edm\/aggregatedCHO","explain":"A Europeana Data Model Property; The identifier of the source object, e.g. the Mona Lisa itself. This could be a full linked open date URI or an internal identifier"}],"Extent":[{"label":"Extent","value":"3747331 bytes","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/extent","classmap":"dpla:SourceResource","property":"dcterms:extent"},"iri":"http:\/\/purl.org\/dc\/terms\/extent","explain":"A Dublin Core Terms Property; The size or duration of the resource."}],"FileFormat":[{"label":"File Format","value":"application\/pdf","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/elements\/1.1\/format","classmap":"edm:WebResource","property":"dc:format"},"iri":"http:\/\/purl.org\/dc\/elements\/1.1\/format","explain":"A Dublin Core Elements Property; The file format, physical medium, or dimensions of the resource.; Examples of dimensions include size and duration. Recommended best practice is to use a controlled vocabulary such as the list of Internet Media Types [MIME]."}],"FullText":[{"label":"Full Text","value":"DIRECT AND INDIRECT PLACEMENT TEST SCORES AS MEASURES OF LANGUAGE PROFICIENCY AND PREDICTORS OF ACADEMIC SUCCESS FOR ESL STUDENTS by VIRGINIA LOUISE CHRISTOPHER B.F.A., University of British Columbia, 1976 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENT FOR THE DEGREE OF MASTER OF ARTS in THE FACULTY OF GRADUATE STUDIES (Department of Language Education) We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA December, 1993 <3)VIRGINIA LOUISE CHRISTOPHER, 1993 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department of v W ^ ^ ^ > A - ^ 0 ^ C ^ ^ ' The University of British Columbia Vancouver, Canada Date C j^ieA \/^vlk^^ \\^a^^i3 DE-6 (2\/88) 11 ABSTRACT Much valuable research in the area of English as a second language has focused on testing, and though many studies have investigated issues related to test reliability, validity and the predictive ability of indirect test scores, they do not assess the comparative ability of indirect and direct test scores to predict academic success. Work of this kind would inform practice in the area of testing and placement. This study investigates the practical problem of making appropriate placement decisions for students whose test results show wide enough discrepancies to indicate placement in different academic programs, or at different levels within programs. The question of whether scores derived from indirect measures (language proficiency tests) or direct measures (writing samples) are better indicators of academic language proficiency is addressed. The study also explores the usefulness of grade point average (GPA) as a measure of academic success, and proposes the use of average accumulated credit per semester (AACPS) as an additional measure. Several researchers have questioned the use of GPA as the sole measure of academic success for ESL students, and this study adds to existing research. The ability of two types of placement test scores to predict academic success for ESL students in Secondary and University programs is evaluated. Test scores from a) an indirect measure of language proficiency, the Michigan Test of English Language Proficiency, and b) a direct measure, a holistically-scored writing test are assessed as predictors of four measures of academic success: a) GPA for all courses, b) GPA for courses requiring a higher level of language proficiency (English or Humanities), c) GPA for courses requiring a lower level of language proficiency (Math or Sciences), and d) AACPS. Ill Results of correlation analysis indicated that for University students, academic success as measured by GPA showed low correlations to both indirect or direct test scores, but direct test scores and combined (indirect and direct) test scores correlated moderately well to academic success as measured by AACPS. For Secondary students, academic success as measured by GPA for courses requiring a higher level of language proficiency (English or Humanities) showed moderate correlations to direct test scores. Correlations between test scores and any other measure of academic success including AACPS were low for Secondary students. IV TABLE OF CONTENTS Page ABSTRACT ii TABLE OF CONTENTS iv LIST OF TABLES vii LIST OF FIGURES viii ACKNOWLEDGEMENTS ix CHAPTER I. INTRODUCTION 1 Background of the Study 1 Development of the Problem 3 Purpose of the Study 5 Definition of Terms 6 Summary and Preview 9 CHAPTER II. REVIEW OF THE LITERATURE 11 Introduction 11 Direct and Indirect Language Tests 11 Alternatives for Assessing Writing 16 Predicting Academic Success 21 Academic Criteria for Successful Writing 21 GPA as a Measure of Academic Success 24 Placement Test Scores as Predictors of Success 27 Summary 29 CHAPTER III. METHODOLOGY 32 Design 32 The Setting 33 The Subjects 36 Data Collection Procedures 36 Measures 37 Indirect 37 Direct 38 TABLE OF CONTENTS (cont'd) Page Grade Point Average 40 Academic Credit 43 Hypotheses 43 Data Analysis 45 Correlational Analysis 46 Multiple Regression Analysis 47 Summary 47 CHAPTER IV. RESULTS AND DISCUSSION 48 Introduction 48 Hypotheses 1 and 2 49 Secondary Students 51 University Students 52 Hypotheses 3 and 4 54 Secondary Students 57 University Students 58 Hypotheses 5 and 6 59 Secondary Students 61 University Students 62 Discussion 63 Hypotheses 1, 2, 3 and 4 63 University Students 63 Secondary Students 64 Hypotheses 5 and 6 66 University Students 66 Secondary Students 66 Summary 68 CHAPTER V. SUMMARY, CONCLUSIONS AND IMPLICATIONS OF THE RESEARCH 69 Introduction 69 Summary of Results 69 Conclusions and Implications 71 Secondary Students 71 University Students 72 Limitations of the Study 74 Validity of the In-House Holistic Grading System 74 VI TABLE OF CONTENTS (cont'd) Page Influence of the Setting on the Academic Performance of Secondary Students 75 The Length of the Study 75 Extension of the Research 76 Summary 77 REFERENCES 79 APPENDIX A: Test of Written English (TWE) Scoring Guide 83 APPENDIX B: Letter of Permission 85 APPENDIX C: English Placement Information 86 APPENDIX D: Conversion Table: GPA to Percentages 87 APPENDIX E: Conversion Table: Direct Test Scores to Percentages 88 APPENDIX F: Data for In-house Direct Test 89 APPENDIX G: Data for TOEFL Test of Written English 90 APPENDIX H: Data for Secondary Students 91 APPENDIX I: Data for University Students 93 vu LIST OF TABLES Table Page 1 Data Analysis: Tests and Variables 46 2 Indirect Test Scores, Direct Test Scores and GPA for Secondary and University Students 48 3 Intercorrelations Between Test Scores and GPA 50 4 Predicting Overall GPA: Multiple Regression Analysis Summary for Indirect, Direct and Combined Test Scores 51 5 Intercorrelations Between Test Scores and GPA for Type I and Type II Courses 55 6 Predicting GPA for Course Type I for Secondary and University Students 56 7 Predicting GPA for Course Type II for Secondary and University Students 57 8 Intercorrelations Between Test Scores and Average Accumulated Credit per Semester (AACPS) 60 9 Predicting Average Accumulated Credit per Semester (AACPS): Multiple Regression Analysis for Indirect, Direct and Combined Test Scores 61 Vlll LIST OF nCURES Figure Page 1 Academic Programs and Placement Test Score Requirements 35 2 Letter Grades, Percentages, and GPA Points for Secondary and University Programs 42 IX ACKNOWLEDGEMENTS There are several people who were instrumental in helping me complete this study. I would like to thank Columbia College for allowing me to conduct the research at the College, and for encouraging me in many ways. I also wish to extend my thanks to the members of my thesis committee: Dr. Bernard Mohan, Dr. Lee Gunderson, Dr. Steven Carey and, in particular, Dr. Richard Berwick, who provided invaluable advice and guidance through the many revisions the work required. Finally, and most importantly, I extend my most heartfelt gratitude to my family and friends. I am especially thankful to my two daughters, Vanessa and Alexandra, who accepted responsibility beyond their years, and whose unfailing patience provided me with motivation and encouragement. CHAPTER I. INTRODUCTION BACKGROUND OF THE STUDY Interest in this study grew out of the practical experience of testing ESL students entering college and secondary programs, and placing them in appropriate classes on the basis of test results. This study explores the problem of making correct placement decisions when the results of two tests suggest different placement levels. Extensive research on testing has addressed the important issues of test validity and reliability, and the uses to which test results are put (Brown, 1989; Carroll, 1965; Cervenka, 1978; Cumming, 1989; Johns, 1981; Graham, 1987; Hanania and Shikhani, 1986;Jenks, 1987; Raimes, 1990). Still lacking, though, is research addressing the problem of making placement decisions on the basis of results of two tests whose scores do not concur. ESL proficiency tests take many forms. Placement tests most widely-used for college-level students in ESL classes are objective, multiple-choice tests with vocabulary, grammar, and reading comprehension sections; some include a listening component. Most academic programs administer a writing test, and some, a listening\/speaking test. Test results are used to place students in programs and in levels within programs, and are sometimes used in the prediction of academic success. Researchers and practitioners alike often question the ability of these kinds of tests to accurately measure the language ability of ESL students and assist placement decisions. Results from two different tests may indicate the same level of language proficiency, but there are many cases in which they do not. In these cases, testing and placement personnel are faced with making the difficult 2 decision as to which test results most accurately measure language proficiency, and should, therefore, be the basis upon which placements are made. Besides being used for placement purposes, language test results are often looked to as predictors of academic success. Some test manuals clearly state that proficiency tests should not be used for this purpose, since there are many other variables which affect academic success (University of Michigan English Language Institute, 1977; Educational Testing Service, 1990). Prediction studies, though, have been conducted by the test designers themselves, a useful practice that is likely to continue. Research into how academic success is measured and the predictive ability of test scores is also of relevance to this study, and has been examined by Black (1991), Graham (1987) and others. Prediction studies tend to use grade point average (GPA) as the sole measure of academic success. It is the view of some researchers that the use of GPA as the only measure of academic success is questionable for those ESL students whose level of language proficiency does not meet the entry requirements of post-secondary institutions, most commonly, a TOEFL score of 550 (Light et al., 1987). One of the purposes of this study is to explore the usefulness of GPA as a predictor of academic success for ESL students with various levels of language proficiency, i.e., with TOEFL scores both above and below 550, GPA does not take course load into consideration and thus excludes an important element in second-language learning, namely, the time it takes to acquire the language skills required for academic study. The difference between the language skills required for basic communication and those required for academic study has been explored by Cummins (1979a) and others. The time necessary to achieve a high level of proficiency in these two areas differs greatly. This fact has an 3 impact on the lives of ESL students enrolled in secondary and tertiary academic studies. Expectations placed on ESL students as to the length of time deemed necessary to complete a program of academic studies are often unrealistically short. Such expectations are created with first-language students in mind, a group for which language proficiency is assumed. ESL students are clearly at a disadvantage here, yet are measured by the same criteria as first-language students. DEVELOPMENT OF THE PROBLEM The focus of this study is the prediction of academic success based on the results of two types of tests. For most students, placement test scores derived from two different tests are typically interpreted as measuring a similar general level of language proficiency in that they indicate the same placement level. For example, the scores of both an indirect test such as the TOEFL, and a direct (writing) test may indicate that a student's language proficiency is adequate for study at a first-year university level; the student is placed accordingly. In these cases, placing students in appropriate programs or levels within programs poses few problems for assessors. In a substantial number of cases, though, there is enough discrepancy between the two measures' scores to indicate placement at different levels, or in different programs. For example, an indirect test score such as a TOEFL or MTELP score may be well above the minimum requirement for entry into a university program, but a writing test score for the same student may indicate placement in a university preparatory or ESL program. Assessors are presented with the problem of deciding which measure is to be given more weight. In most cases, the student is given the benefit of the doubt, and 4 is placed according to the higher of the two scores. Instructors often question this practice. There is controversy as to whether students placed in this way have problems maintaining the expected level of performance the curriculum demands, and whether the students would not have been more appropriately placed in the lower level as indicated by the lower test score, particularly when the writing test score is the lower of the two. Research does not indicate which score is a better predictor of academic success, that derived from an indirect test or that derived from a direct (writing) test when there are discrepancies between these scores. While most instructors know that no single score should be used as a basis for placement, there is a special problem when discrepant scores are involved. Most instructors also know that ESL students can become extremely proficient at test-taking through practice, and that, indeed, many students take courses designed to maximize chances of achieving high scores. Language proficiency test scores, then, may be more indicative of students' test-taking skills than of their actual language proficiency. In addition, in a direct test, students are required to demonstrate their practical knowledge of a language and are assessed on the basis of that performance, but are not required to demonstrate a skill on an indirect test. The difference between competence and performance raises the question of whether a test assessing skill in using a language should be considered a more valid measure of language proficiency than one assessing knowledge about a language. A problem related to competence vs. performance pertains to the measurement of academic success, and whether measures designed for first-language learners are appropriate for second-language students. Throughout the literature, grade point average is used in most prediction studies as the only dependent variable. A few researchers question the validity 5 of GPA as a measure of academic success, as it does not reflect the number of courses taken (Black, 1991; Heil and Aleamoni, 1974). This study extends such research in including average accumulated credit per semester (AACPS) as a measure of academic success. In conjunction with GPA, this seems to be an appropriate approach to measuring the academic success of ESL students, particularly until students are better able to compete with their native English-speaking peers more fairly. The length of time required for achieving such a level of proficiency would vary according to individual rates of acquisition of academic language proficiency. Research investigating the phenomenon of variable success rates by course type may inform practice, particularly in the area of counselling and course planning. This study explores this area by conducting prediction studies on two types of courses separated on the basis of a) whether previous experience in the student's native language would be expected, and b) the level of language proficiency required for successful completion of the course. In particular, the study examines the ability of indirect and direct test scores to predict academic success in each of the two course categories. PURPOSE OF THE STUDY The purpose of the study is to investigate whether indirect test scores or direct (writing) test scores should be the deciding factor for placement purposes for students who have discrepancies in these scores. In addition, the research carried out in the study provides insight into the question of whether writing test scores give a clearer indication of academic language proficiency than do indirect test results in terms of predicting academic success. 6 In order to accomplish this purpose, several hypotheses, as presented in Chapter III, were tested; results are presented and discussed in Chapters IV and V. DEFINITION OF TERMS To ensure understanding of the terminology as it is used in the following chapters of the study, the following definitions are included: 1. Direct tests\/Writing tests: (These terms will be used interchangeably.) These are tests which require students to perform an activity which demonstrates directly their proficiency in the use of English. In this study the direct measure is a writing test. Essay tests used as placement measures are usually graded in a holistic manner, using a criterion-referenced scoring guide. An example is the Educational Testing Service's Test of Written English (TWE). 2. Indirect tests\/Objective tests: (These terms will be used interchangeably.) These are tests which do not require students to perform an activity demonstrating their proficiency in English. Instead, they test recognition of grammatically correct, standard English. These tests usually consist of separate sections focusing on specific language sub-skills, contain only multiple-choice items, and are graded objectively. Examples are the Test of English as a Foreign Language (TOEFL), and the Michigan Test of English Language Proficiency (MTELP). 3. Holistic Scoring: A composition evaluation process whereby raters grade texts as a whole in an impressionistic manner; overall writing proficiency is assessed according to a scoring guide by at least two raters. Sample papers displaying the range of possible scores 7 are used as reference points continually throughout the process. 4. Analytical Scoring: A composition evaluation process whereby various features of the text are separated out and graded individually according to specific criteria. One or more raters are used in this method of evaluation. 5. Diagnostic test: A test which isolates specific strengths and weaknesses of an individual in some particular field of knowledge (Lien, 1967), 6. Proficiency test: A test which measures overall ability in English, independent of a particular instructional program. Proficiency tests are often used to assess readiness to work at a particular level of instruction, 7. Placement test: A test whose results determine readiness to work at a particular level of instruction. Placement tests should test skills that will be used in the program of study for which students are being tested. Proficiency tests are often used as placement tests. 8. Test reliability: The extent to which a test is dependable, stable and consistent, when given to different people and\/or administered on different occasions (Page and Thomas, 1977). 9. Inter-rater reliability: The tendency of a test to produce similar assessments by more than one rater. 10. Intra-rater reliability: The tendency of test scores to be consistent for the same rater on different administrations of the same instrument. 11. Test validity: The extent to which a given test is an appropriate measure of what it was intended to measure (Page & Thomas, 1977), 12. Construct validity: The extent to which test performance can be interpreted in terms 8 of certain psychological constructs (Lien, 1967). 13. Content validity: The extent to which the content of a test is judged to be representative of a larger domain of content (McMillan & Schumacher, 1989). 14. Concurrent validity: The extent to which test scores can be correlated with scores from an existing instrument given at about the same time (McMillan & Schumacher, 1989). 15. Predictive validity: The extent to which scores from a test are correlated with future behavior (McMillan & Schumacher, 1989). 16. Academic Success: The achievement (or non-achievement) of standards established by academic institutions. This achievement is usually measured in terms of grade point average (GPA). 17. Average accumulated credit per semester: The average number of course credits accumulated in a specified number of semesters at an academic institution. 18. Competence: Internalized rules about a language that are organized into a system (Ellis, 1985). Includes the ability to recognize grammatically correct, standard English, usually tested by indirect means (answering multiple-choice questions about written English). 19. Performance: The actual use of a system of internalized rules about a language (Ellis, 1985). Includes the ability to demonstrate knowledge of grammatically correct, standard English by performing a skill or an activity e.g., writing an essay. 20. Course Type I: In this study, a category of courses for which previous experience in a student's home country would not be expected, e.g., English Literature, Humanities. 21. Course Type II: In this study, a category of courses for which previous experience 9 in a student's home country would be expected, e.g.. Mathematics, Sciences. 22. Cultural Literacy: Common knowledge or collective memory that allows for communication within a culture. The knowledge assumed in public discourse. (Hirsch, Khett & Trefil, 1988). SUMMARY AND PREVIEW This chapter has described the problem investigated in the study and the practical experience from which interest in the problem arose. Several related issues in testing and placement have been introduced. The research accomplished in this study will add to current knowledge and should be directly applicable to the process of testing college and secondary ESL students for placement purposes. Several studies in the area of language proficiency testing and the measurement of academic success which are relevant to this study have been introduced and the terms used in the study have been defined. A brief description of the contents of the remaining four chapters follows: Chapter II presents a review of related research in a) direct and indirect testing of language proficiency, b) methods of assessing writing and c) measurement and prediction of academic success. Chapter III presents six hypotheses to be tested, describes the methodology of the study, the design, the subjects and the setting. The data collection procedures are included, as well as the five measures used to assess the subjects' level of language proficiency and degree of academic success. Chapter III also describes the two methods of data analysis used. In Chapter IV, the results of the hypothesis-testing are reported. A discussion of these results is presented for each hypothesis in turn. Chapter V draws conclusions from 10 the results of the hypotheses-testing and presents implications for practice in the area of testing and placement. Suggestions for extension of the research are made and limitations of the study are presented. 11 CHAPTER n. REVIEW OF THE LITERATURE INTRODUCTION This chapter presents research in the teaching of English as a second language that has a bearing on the present study. The relevant literature has been divided into three sections. First is research into testing, in particular studies on both indirect and direct language tests and the differences in the way they measure language proficiency are reviewed. The second section includes research into methods of assessing ESL writing, focusing specifically on holistic grading and its validity and reliability. Finally, literature concerning the measurement and prediction of academic success is discussed in three sub-sections and related to the specific issue of the study, i.e., the predictive ability of direct and indirect placement tests. DIRECT AND INDIRECT LANGUAGE TESTS Research on the testing and measuring of ESL students' language ability has raised questions as to the validity and usefulness of the various types of tests used for placing students in academic programs. Depending on the information administrators and teachers want to obtain, four main types of tests are used: proficiency tests, achievement tests, placement tests, and diagnostic tests. This study is concerned with the placement of students, and since both placement and proficiency tests are used for this purpose, these are the two types of tests which will be discussed. The Test of English as a Foreign Language (TOEFL), a widely used, indirect. 12 objectively-scored measure of language proficiency, has been the subject of numerous studies (Raimes, 1990; Light, Xu, and Mossop, 1987; Graham, 1987; Jacobs, Zinkgraf, Wormuth, Hartfiel, and Hughey, 1981; Hanania and Shikhani, 1986; Brown, 1989, and others). Besides testing listening comprehension, grammar, reading comprehension and vocabulary, the TOEFL purports to measure writing ability indirectiy, that is, it measures the abilities which underlie the skill of writing by testing recognition of standard, formal English (Educational Testing Service, 1990). Its recent supplement, the Test of Written English (TWE), is instead, a direct measure of writing ability in that it rates students' actual production. Students' essays are rated against a criterion-referenced scoring guide (Appendix A). Sometimes placement decisions must be made based upon conflicting data from two test sources, one assessing the ability to recognize standard, formal English, and one assessing the ability to produce such English. Raimes (1990) provides background on both TOEFL and TWE tests, relates them to native-speaker tests and questions the need for both when each claims to test writing proficiency. She makes recommendations regarding the uses of these tests, mentioning the importance of the training of readers, an issue also discussed by Norton-Pierce (1992) and Jacobs etal., (1981). Norton-Pierce's (1991) review of the TWE describes the test as a complement to the TOEFL and mentions the high inter-rater reliability achieved. This reliability is attributed to the criterion-referenced scoring guide, which Norton-Pierce views as a major strength, since it gives raters clear descriptions of ability at each level, focuses on meaning and gives readers an opportunity to reward students for what they do well. Holistic assessment is 13 contrasted with analytic scoring which, in its concern with discrete points of language usage, often does not attend to the communicative aspects of writing. But the scoring guide's neglect of the role of the reader\/rater as partner in the communicative act of writing is noted as a shortcoming of the TWE and it is compared to the British Council's English Language Testing Service (ELT) scoring guide, which does include the reader as participant (Norton-Pierce, 1981). Norton-Pierce concludes that the ultimate reliability of the TWE lies in the strength of the reader-training program. Jacobs (1981) is also concerned with the unreliability of essay-readers' evaluations, and notes that differences in experience and academic background of essay readers result in low inter-rater reliability. She quotes Harris (1977) who found that teachers, though ranking content and organization as primary aspects to consider in assessing composition, actually rated essays according to mechanics and sentence-level errors. Conversely, Cooper (in Cooper and Odell, 1987), maintains that high reliability can be achieved by adhering to a holistic scoring guide. Brown and Bailey (1984), tested the reliability of a composition scoring grid developed on the assumption that a precise and informative diagnosis of ESL writing could be made by focusing raters' attention on specific criteria. These researchers found that after subjecting outliers to a third reading, the level of inter-rater reliability fell within acceptable limits. The system of using of a third reader for significantly differing scores is also practised by the Educational Testing Service (ETS) in rating the TWE. In addition to concern with the reliability of scoring grids, researchers are also addressing the issue of the limitations of language proficiency tests and the use to which results are put. Graham (1987) discusses the difference between measuring students' 14 recognition of standard English and their demonstrated use of it, and questions the widespread expectation that placement\/proficiency tests act as predictors of academic success. Studies done by Farhady (1983) and Hanania and Shikhani (1986) show high correlations between direct tests and indirect ones, that is, those demanding production or use of the language. Others question the use of TOEFL or other indirect tests as placement tools and indicate that measures assessing communicative competence would be more appropriate, as these ask students to demonstrate the skill they will later be graded on in their academic careers (Graham, 1987). Perkins (1983) also compares direct and indirect methods of testing writing ability and suggests that when scores on indirect tests and direct (writing) tests concur, indirect tests are as valid as essays when used for placement purposes. The Perkins study, though, does not include cases in which indirect and direct test scores do not concur. These cases are central to this study. Both Brown (1989) and Farhady (1982) deal with most tests' neglect of learner differences and both recommend that tests be designed to suit the population to be tested. Brown finds discrepancies between test results and actual progress and proposes that tests be redesigned to reflect curricula. Farhady suggests that rather than relying on one comprehensive test (such as TOEFL) as is the general practice, administrators should turn to discipline-oriented measures. By this method of testing, communicative competence could be taken into account for disciplines requiring a high level of linguistic ability, e.g., the Humanities. A problem here is that students are usually required to take courses demanding varying levels of language proficiency, from mathematics to literature. The difficulty in 15 creating a single test with the content validity required for the range of disciplines that exist across the wide variety of programs available at most post-secondary institutions makes this kind of testing impractical; students would likely have to write several tests. Though these and other researchers believe that in-house placement and proficiency tests are appropriate, as they are designed specifically for the population, curricula, and standards of a particular school, many institutions rely on TOEFL scores for reasons relating to cost and efficiency. The TOEFL, with its comprehensive assessment and huge scale of administration in approximately 170 countries and areas throughout the world (Davies and West, 1989), remains an appealing placement measure for most academic institutions. Also relevant to the study is research which questions just what language ability encompasses and argues that writing proficiency is quite separate from other language skills. Farhady (1982) suggests that language is not a unidimensional phenomenon. In order for tests to make adequate assessments, they must serve multiple purposes and attend to the variability in students' language ability. Gumming (1989) proposes that writing expertise is a specially developed intelligence and its cognitive characteristics can be applied across languages. He views language proficiency as dependant upon a) the development of the language faculty (a cognitive faculty separate from intelligence) and b) its operation within the parameters of a code, that is, the ability of the language faculty to engage in communication using previously set rules which organize sounds and symbols into oral or written language. He argues that writing performance depends on the contribution of both these factors. Gumming asserts that in academic settings where students are assessed through writing, the distinctions between these two abilities should be made, and they should be 16 developed and measured separately. A test measuring writing ability, therefore, should be used to place students in programs where their future grades will be based upon their writing proficiency, as indirect test scores do not measure writing as a skill. In relation to this study, the literature on ESL proficiency\/placement testing seems to suggest that indirect test scores should not be considered the most accurate indication of language proficiency, particularly when there is a discrepancy between them and other scores which have been shown to be valid measures of language proficiency (Graham, 1987; Brown, 1989; Farhady, 1982). There is a general concern at the college level regarding language proficiency tests and their appropriateness as placement and predictive instruments, particularly in cases where there are discrepancies in scores wide enough to indicate different placement levels for the same student. The research does not address the question of whether a direct score is a better predictor of academic success than an indirect score when the scores show wide discrepancies, an issue this study will address. ALTERNATIVES FOR ASSESSING WRITING Many researchers believe holistic grading of writing is valid, reliable, and a more accurate measure than indirect evaluation. One of the purposes of this study is to test whether holistic grading of a writing sample can provide a more accurate assessment for placement purposes than indirect testing. Holistic grading of the direct test is performed by experienced instructors involved in the placement process at the institution where the study was conducted. The topic of writing assessment at the classroom level has been studied extensively. 17 Many researchers agree that instructors' response to error in writing varies widely and is often not effective as productive feedback for ESL learners. Hendrickson (1978), in an historical treatment of studies of learner errors, reviews attitudes toward error correction and suggests that a separation of error type into the categories of global and local would be useful in determining which are more serious and which should be corrected first. He denotes as global those errors which interfere with communication and as local those which do not. The latter group consists of sentence-level, mechanical errors. Hendrickson argues that an order of error correction should be established, Zamel (1985) agrees with this view. In addition, Zamel asserts that most instructors' responses are inconsistent, imprecise, and concerned with surface-level errors too early in the composition process. She cites Krashen (1982) and supports his contention that production may be inhibited by the monitoring of output which is still in the developing stages. Zamel suggests that communicative effectiveness would be better achieved by allowing students time to apply instructors' responses and incorporate them into the text. On the measurement of communicative effectiveness, Janopoulos (1989) studies the extent to which holistic raters' comprehension of ESL students' texts affects scores. His position is that judgements are made on the degree to which messages sent by the writer correspond to those received by the reader. He uses recall protocols finding that holistic raters do attend to meaning and can recall more content of the higher-quality texts (i.e., those that received higher scores) than they can of the lower-scoring texts. Thus, holistic assessment of writing appears to allow raters to attend to meaning, while analytic scoring methods do not. As such, it is a more appropriate measure for academic writing, as 18 meaning will be the main criterion of future assessments of writing produced for academic courses. In his study of analytic scoring methods, Perkins (1980) examines r-unit length, number of words per r-unit, syntactic complexity, and a complexity index in order to establish which measures discriminate among various levels of writing proficiency. He found that only measures which take absence of error into account (i.e., error-free f-units) discriminated among the holistically-rated compositions. Thus, Perkins questions the validity of indirect measures in evaluating the type of writing required for academic purposes. Instructors must look for properties other than syntactic complexity and mechanically-correct discourse. Perkins also studied the ability of a standard, indirect writing test, the Test of Standard Written English (TSWE), to discriminate among writers at various levels of proficiency. Two important factors must be considered here: first, the TSWE is a native-speaker test and is not designed to evaluate second-language learners. The extremely low scores of Perkins' group attest to this. Second, the TSWE tests recognition of correct English and not the ability to produce it. Perkins points out this second important feature as a fundamental difference between direct and indirect writing tests, a crucial point for this study. One of the hypotheses of this study is that a score derived from a direct test is a) a valid measure of language proficiency for evaluation and placement purposes and b) a better predictor of academic success for courses in which students are not likely to have had previous experience in their home countries. Robb, Ross, and Shortreed's (1986) study contrasting four methods of feedback indicates that practice over time showed gradual increases of mean scores, regardless of 19 feedback method. They are in agreement with Zamel in finding most instructor comments too obscure to be of real help to the students, and that only a small proportion of instructor feedback could be assimilated and transformed into meaningful input at any one stage. Ling (1986) supports Zamel and Robb et al., asserting that assessors should respond to process, encouraging revision for content and meaning, and leave editing for form and mechanics for the final stage of composition. Though the research by Robb et al. (1986), Zamel (1985), and Ling (1986) is mainly concerned with the usefulness of various scoring methods for helping writers improve, it is also frequently the case that essays are also rated and assigned grades by a variety of scoring methods. These researchers seem to agree that holistic grading is the most appropriate method of assessing writing, whether for responding to writing for the purpose of helping writers improve or for rating writing for course grades. In his study comparing the thinking behaviours of experienced and inexperienced raters. Gumming (1990) finds that experienced raters draw on a more comprehensive and developed base of evaluation criteria, and their assessments involve a huge number of interrelated decisions. He compares this multi-faceted process of evaluation with Homburg's more linear scheme (Homburg, 1984) which depicts raters using a system of categories ranging from gross to fine. Following Homburg's model, raters begin with larger categories based on specific features and move to finer ones as the evaluation process proceeds. Though Gumming describes a more multi-dimensional representation of the assessment process, both models describe holistic grading as a measure in which many criteria are employed in an integrative manner to arrive at a score, reflecting the essential property of 20 a piece of writing as an act of communication. The literature shows that concern for content over mechanics demands a style of evaluation that is holistic in nature rather than analytic and discrete. In the holistic rating process, the composition is assessed as a unit, with the rater focusing on the integration of various elements, the writer's control of the discourse, and on how well the essay meets certain established criteria. These criteria constitute the scoring scheme, and must be carefully devised and applied to ensure reliability and validity. Homburg (1984), Gumming (1990), Janapoulos (1989), Jacobs et al., (1981), and Cooper (1977) stress the importance of a well-developed scoring guide and a refined system of reader-training. Problems of inter- and intra-rater reliability can be addressed by ensuring such requirements are met (Cooper, 1977; Jacobs et al. (1981)). Cooper (1977), a strong advocate of holistic evaluation of writing, states that holistic evaluation is \"the most valid and direct means of rank-ordering students\" (p.3). He describes various types of evaluations, discusses analytic scales, and provides procedures for developing the list of features or criteria for such scales. The scoring scales of the TWE and the ESL Composition Profile, (Jacobs et al., 1981) are examples of well-developed holistic scales. Cooper, among others previously mentioned, calls for a stringent reader-training system, appropriate sample papers representative of each grading level (range-finders), and scoring sessions at which raters constantly check the reliability of their ratings against range-finders as well as against other raters. With such a system in place, he claims that scoring reliabilities in the high eighties and low nineties can be achieved for holistically scored measures of writing ability. 21 The findings of the studies cited above indicate that holistic rating of writing is a valid and reliable assessment method for both diagnostic and placement purposes. PREDICTING ACADEMIC SUCCESS Academic Criteria for Successful Writing Faculty opinion and response to error in writing has been widely studied and results have informed both curricula and testing. Johns (1981), in a study of 200 professors' opinions of which skills were most essential for the academic success of second-language students, finds that the receptive skills of reading and listening were rated highest. These findings applied to nine of eleven faculties at both the undergraduate and graduate levels. Writing was rated highest in only the English and Education faculties. Johns questions the emphasis placed on creative writing in ESL curricula and the rating of students' success in this type of discourse being used as a measure of readiness for academic study. The suggestion is made that perhaps TOEFL is, in fact, the best indicator of academic success, as it measures the skills actually required, i.e., the receptive skills. Horowitz (1986) concurs with Johns, suggesting that in order to create realistic writing tasks for ESL students, more emphasis should be placed on the instruction of data re-organization, and less on writing encouraging invention and creation. He states that students would be better-prepared for future academic tasks by this shift. Both Horowitz and Johns agree that for most academic purposes, students' main use of writing will be to respond to the ideas of others, rather than to explore and express personal ideas. Other studies of faculty response to error have found that factors such as professors' 22 age, academic discipline, and whether the instructor is a native or non-native speaker, all affect tolerance of error. Vann, Meyer, and Lorenz (1984) focus on sentence-level errors, and found that response to error varies with the age and academic discipline of the rater, and that there is a hierarchy of error seriousness which raters apply. These researchers are in agreement with Santos (1988), Hendrickson (1978), Zamel (1985), and others in supporting the belief that errors interfering with communication (global) are considered most serious, while surface-level (local) errors not affecting comprehension are most acceptable. Of the sentence-level errors, lexical mistakes are ranked highest in severity, as are those made least often by native speakers. The finding that non-native speaking instructors judge errors more severely than do native speakers is also borne out by Sheorey (1986). The academic success of ESL students is directly affected by the hierarchy of error seriousness that professors have developed. The subjects of the Santos study rated two essays on separate scales focusing on a) content and b) language. Content was judged more severely than language; professors made statistically different distinctions between the two. Raters judged language and content independently, except in the case of lexical errors. The problem of interference with communication caused by such errors was reflected in this judgement. Brown (1991) investigates the differences between writing scores of native and non-native students at the end of first-year writing courses. He found that there were no statistically significant differences between the scores the students received from ESL and English instructors who rated both sets of essays. A feature analysis revealed, though, that instructors from the two backgrounds rated various features differently. Brown's findings support the possibility of a hierarchy of error-seriousness which instructors use to evaluate 23 composition. Despite differences in this hierarchy, however, there seems to exist a commonly-held standard against which essays are measured. These results suggest the possibility that similar scores may be arrived at either by analytic scoring methods, as in Brown's study, or by holistic scoring methods. Further research comparing results of these two methods would be informative. In addition to scoring method, instructional approach has a considerable influence on academic success. Process writing is an instructional approach in which the central focus is on revision; it is the composing process rather than the final product to which the instructor responds (Shih, 1986). This approach is supported by Santos' finding, while Shih (1986) argues for a content-based approach. Shih cites studies of faculty opinion of types of writing tasks required of undergraduate and graduate students, and states that a content-based approach develops the thinking, researching and writing skills needed for academic success. Johns and Horowitz are in agreement here, in that both question the wisdom of emphasizing personal, creative writing when most faculties demand other kinds of skills, (e.g., paraphrasing, summarizing). In a study of student opinion of the academic skills required of advanced ESL students. Ostler (1980) finds a distinction between skills demanded of undergraduate and graduate students. For undergraduates, the findings support Horowitz and Johns in that requirements at this level are more discipline-specific and stress the receptive skills of reading and listening. For graduate students, reading, writing, and oral skills are judged as essential. Researchers seem to agree that writing requirements for undergraduates seem to take the form of reorganizing and commenting on the views of others, which may indicate 24 that ESL placement writing tests should assess students' capabilities with this discourse style. Expression of personal opinion and ability to comment on an abstract idea, then, should not be the type of task demanded in placement tests, as is the current practice. Appropriateness of topic type and the resultant discourse style could be the subject of further investigation. For the purposes of this study, however, it is the value of the writing sample over the indirect test score that is the issue. If topics were changed to elicit discourse styles actually required of undergraduates, as suggested by Johns, Horowitz and others, students would still be required to demonstrate skill in writing. These authors also argue that current research indicates that students' proficiency at the skill of writing should be rated, rather than their ability to recognize standard written English (as is required on indirect tests such as TOEFL and the MTELP). The findings of the studies cited above show how academic success is measured at the specific task level and reveal which factors influence professors' judgements. These decisions reflect students' grades on particular assignments and in particular courses, all of which contribute to grade point average. GPA as a Measure of Academic Success Several researchers have questioned the appropriateness of using GPA as the sole measure of academic success, particularly for ESL students. Gue and Holdaway (1973) concede that the degree of learning in university settings is most easily measured by GPA, but that GPA is only one standard; its limitations as a valid criterion of learning must be kept in mind. Sharon (1972) points out the problem of the variation in grading standards from school to school and suggests that prediction schemes based on GPA should adjust for such differences. The subjective nature of grading practices, exam anxiety, low construct validity 25 and reliability of tests (particularly instructor-made tests), cultural differences regarding ideas about just which behaviours will result in a good grade (answering questions in class, participating in group projects, for example), and the fact that proficiency tests often neglect assessment of sociolinguistic knowledge suggest some of the limitations of using GPA as a valid indication that a specific amount of learning has taken place. Light et al. (1987) indicate that GPA may not be the most important criterion of academic success and suggest that measures such as credit hours earned in addition to professors' and students' evaluations of students' success should be examined. The relationship between language proficiency and ability to earn graduate credits was also studied by Light et al. (1987), who notes that the number of credits earned in the first semester of study correlated significantly to TOEFL score for students who scored under 600. The system of measuring academic success on the basis of GPA is founded on standards and expectations created for first-language students and is culture-based. Constant adjustments and alterations to the traditional assessment system are made at the classroom level by instructors faced with the problem of assessing the degree of learning achieved by ESL students. Contrary to their first-language peers and competitors, ESL students usually have not had the opportunity to internalize the cultural understanding required to make the expectations of the system fair ones, and so are at a relative disadvantage. While intelligence, motivation for success, aptitude for academic study, previous academic record and other factors are comparable to their first-language peers, English language proficiency and cultural literacy are obstacles to academic success for many second-language learners. Difficulties in attempting to meet the standards of the assessment system are reflected very 26 clearly in the failure of many ESL students to meet the time expectations for degree completion set out for first-language students. The pace of learning is often beyond the capabilities of ESL students and adjustments are made in the form of reductions in course load. Cummins (1979) addresses the time element in second language learning, and in the acquisition of academic skills necessary for success. He differentiates between Cognitive\/ Academic Language Proficiency (CALP) and Basic Interpersonal Communication Skills (BICS), and the amount of time required to acquire each: five to seven years for CALP, and two years for BICS. Applied to the study presented here, students with high proficiency scores will have a greater chance for academic success, as they will have likely spent more time in their study of English than lower scorers, and will have proven academic capability through credits earned in their home countries. Students with lower placement scores on language proficiency tests, while having academic backgrounds similar to the higher scorers, hence their acceptance by the college, will probably have spent a fewer number of years in formal English study. These students will likely have achieved a level of proficiency sufficient to be included in Cummins' BICS category, but may still lack certain skills essential for academic success. The element of time is important here, in that this lower group, if given the opportunity to expand their academic language proficiency, would have just as much potential for success as the higher group. The realization of the necessity of a sufficient amount of time to achieve a GPA that truly reflects their capabilities (CALP), is reflected in some students' choices of reducing course loads to a manageable level, thereby extending the time prescriptions deemed appropriate for completion of a particular course 27 of study. Graham cites Heil and Aleamoni (1974), who argue that the number of courses taken is not reflected in GPA, and thus its use as a measure of academic achievement is problematic. Black (1991) suggests that GPA is an indicator of success for students who passed courses on their first attempt, while it is not an accurate measure of true achievement for students requiring one or more attempts to receive passing grades. Factors taken into consideration in the measurement of academic success, then, should include not only GPA and the completion of a particular program, but also the amount of time taken to complete program requirements. This study investigates the aspect of time in measuring academic success by assessing the ability of test scores to predict average accumulated credit per semester. The research to date addressing this issue is scant (but see Light etal., 1987). Placement Test Scores as Predictors of Academic Success The use of test scores as predictors of success as measured by GPA is an important issue in this study, as it is hypothesized that writing scores are better predictors than indirect scores. Problems with using proficiency scores as predictors are discussed by Graham (1987). First-semester GPA and high-school grades or rank in class have been found to be indicators of eventual success. Math and other subject-matter scores, and the Graduate Record Examination - Quantitative (GRE-Q) results have also been found to correlate with academic success. Graham cites numerous studies drawing negative, positive, and mixed conclusions to research correlating scores on measures of English language proficiency and academic success, and states that no clear-cut recommendations for admission can be made. There is also evidence that a high TOEFL score is not an efficient predictor of 28 academic success (Educational Testing Service, 1990; Light et al., 1987). A high TOEFL (e.g., between 550 and 600) is frequently required by institutions, even though they admit ESL students on the basis of other academic criteria. For students with low TOEFL scores, more accurate predictions can be made as to their academic success, as language proficiency becomes a major factor among those contributing to academic success; this group though, are not usually admitted to academic institutions (Graham, 1987). Sharon (1972), in a study of the ability of TOEFL to enhance the predictive ability of the GRE Verbal test, supports Graham and others, finding that academic success as measured by GPA is better predicted by either the GRE Verbal or GRE Quantitative tests for students with lower language proficiency than by the TOEFL. Moreover, GRE scores cannot be predicted by higher TOEFL scores. Finally, Graham indicates that there is a level below which low proficiency will negatively affect academic success, and each institution should establish this level for its particular programs. She is supported by others (Brown 1989; Farhady 1982) in her assertion that English proficiency is only one of many factors which influence academic success, and that several of these factors are not measured by language proficiency tests. Light et al. (1987) suggest that if students with TOEFL scores below 550 are admitted to post-secondary institutions, it is usually on the basis of promising academic ability as demonstrated by other measures. These researchers found that students in this group were not less successful than those whose scores were above 550. Over all, they concurred with the ETS researchers in their finding that TOEFL score alone was not an effective predictor of academic success as measured by GPA. As this section shows, research on the use of placement test scores as predictors of 29 academic success appears to focus on indirect tests and presents mixed conclusions. There have been fewer studies assessing the predictive ability of direct test scores. This study extends the existing research by using both of these types of test scores. It also incorporates a second criterion of academic success, average accumulated credit per semester (AACPS), in addition to GPA. As noted previously, several researchers have indicated that such additional measures would help provide more accurate assessments of the value of test scores as predictors of academic success. SUMMARY Much valuable research has been accomplished in second language testing, particularly in terms of establishing the validity and reliability of particular tests as placement measures, the use of various methods of rating of writing, and the prediction of academic success. There are four main areas in which this study addresses gaps in the current research. 1) Many studies have assessed the ability of indirect test scores to predict academic success as measured by GPA. There is far less research into the predictive ability of direct test scores. This study assesses the ability of both indirect and direct test scores to predict academic success. In addition, the research does not address the problem of placement for students whose indirect and direct test scores show wide enough discrepancies to indicate different placement levels. In assessing the predictive ability of both indirect and direct test scores, this study addresses the question of whether one test score should hold more weight when placement decisions are made for such students. 30 2) Opinion as to which skills are most important for post-secondary academic success is split. Some researchers view writing as the skill in which students need to be most proficient, while others judge the receptive skills to be more important. This divergence of opinion poses the question of whether indirect test scores (which test the receptive skills) are better predictors of academic success than direct test scores (which test the productive skills). The present study explores this question. 3) Most research on the prediction of academic success measures the ability of test scores to predict overall GPA, i.e., GPA is calculated on all courses combined. Studies assessing the ability of test scores to predict academic success in different types of courses are less common, or are not as frequently published. (Two unpublished studies correlate TOEFL scores to individual courses in Arts and English: see Berwick, 1992, 1993; also Graham (1987) regarding students in arts and sciences). This study adds to the scant amount of published research in this area in that it separates courses into two types and assesses the predictive ability of test scores to GPA for the two categories separately. 4) In studies dealing with the prediction of academic success grade point average has traditionally been used as the single dependent variable, despite the fact that some researchers suggest that other measures should be considered (Graham, 1987; Heil and Aleamoni, 1974, Black, 1991). Based on assumptions of second language learning which deal with a) acquisition order (Dulay, Burt, Krashen, 1982) and b) the importance of the element of time in the development of language skills for specific purposes (Cummins, 1979a), this study includes the time factor as a criterion in the evaluation of students' academic success. It uses course load in the form of average accumulated credit per 31 semester as an additional measure of academic success, thus expanding on existing research in which GPA has been the sole measure. This study, though not examining the effects of other factors, it provide a basic investigation of the contribution of direct and indirect test scores to the prediction of academic success. The study will add to the current research in the four areas cited above and perhaps will inform practice. It is important to practitioners working in the area of testing and placement that as much information as possible be available on which to base placement decisions, particularly for students with discrepant placement test scores. This study focuses on the problem of making such decisions, examines the relationship between two types of tests, and evaluates the quality of one test over the other as a predictor of academic success. The next chapter outlines the research methods and lists hypotheses generated to test assumptions based on both practice and research. 32 CHAPTER m . METHODOLOGY DESIGN This study assesses the academic performance of two categories of students, University and Secondary, on four measures: a) direct test scores, b) indirect test scores, c) grade point average (GPA) and d) average accumulated credit per semester (AACPS). The ability of test scores from an indirect measure (the Michigan Test of English Language Proficiency) and a direct measure (a holistically-scored essay) to predict GPA and AACPS are assessed. Correlations between the two types of test scores (direct and indirect) and the two outcomes (GPA and AACPS) are also measured. A description of the setting detailing program levels, courses, and test score requirements for entry into various levels (Figure 1) follows this section. Subsequent sections describe the subjects, the data collection procedures and the measures. Six hypotheses were generated to test assumptions and these, along with a rationale for their formation, are listed. A description of the methods of data analysis used (correlation and multiple regression) and a short summary follow in the final sections of this chapter. A rationale for the design of the study is based on both research and practice. Much research to date seems to indicate that tests should assess students' ability to perform activities which reflect the types of tasks demanded of them in future academic courses. Scores from these tests, then, should be better predictors of academic success than scores from tests which do not test such abilities directly. Practice has indicated that students with lower writing proficiency have a more 33 difficult time achieving a satisfactory level of performance in many of their academic courses. Anecdotal data was gathered through interview and observation at the institution in the study during the year prior to the beginning of the study. This data showed that both students and instructors were of the opinion that low writing proficiency was often the cause of failure in, repetition of, or withdrawal from courses. Types of courses in which students had most trouble were those requiring the highest level of language proficiency, and with which students had little or no previous experience in their home countries (e.g., English Literature). For courses requiring a lower level of language proficiency, and in which the content was familiar from previous experience in the students' home countries (e.g.. Math), writing expertise seemed to have a lesser effect on grades. THE SETTING The setting is a Vancouver-area private college of approximately 500 - 700 students. The following programs are offered: 1) Full-time ESL Program: Non-credit second-language training for both academic and non-academic purposes; five levels from beginners to advanced are available. 2) Secondary Program: Grades 11 and 12 are offered. The program is approved by the British Columbia Provincial Government Ministry of Education; successful completion of government exams is required for graduation. 3) University Preparation Program: A combination of non-credit ESL courses in English and university credit courses. Access to credit courses is limited to courses with the lowest academic language proficiency requirements, e.g.. Math and Computer Science. 34 4) University Transfer Program: First- and second-year university transfer courses. The tuition fees, number of contact hours, and course offerings are comparable to many similar institutions in North America, and students transfer between this college, private and public colleges, universities, and secondary schools. The majority of students at the college are in Canada on student authorizations. The proportion of Canadian citizens averages approximately 10 to 20 percent and at least half of these are recent immigrants to Canada. New students to the college are given a placement test consisting of a) a standard, indirect ESL proficiency test, the Michigan Test of English Language Proficiency (MTELP) Form B and b) a direct measure in the form of a one-hour, holistically scored writing test on an assigned topic. The results of these two media are used to place students in appropriate programs and levels within programs. Students may be placed into any program level, but may not skip a level once placed. Students proceed from the full-time ESL program to either the University Preparation or Secondary Programs, and from the Secondary Program Level 3 to the University Program Level 4. The academic programs, levels, and the test score requirements for both the direct and indirect measures are included in Figure 1. 35 University Program Level 1. 2. 3. 4. Program Levels Full-time ESL University Transfer Program Semester I University Transfer Program Semester II University Program Courses Beginners to Advanced ESL Eng. 098 - ESL & Eng. 090 - Grammar 2 Credit Courses (Type II) (Math, Computer Science, Science) Eng. 099 - ESL & 2 Credit Courses (Type I or II) (except English 101) English 101 & Type I or II Credit Courses MTELP Score 40 or below 4 0 - 5 4 5 5 - 6 0 70 and above Writing Test Score 4 3 2 1 Secondary Program Level 1. 2. 3. Program Levels Full-time ESL Part-time ESL\/ Secondary Program Secondary Program Courses Beginners to Advanced ESL ESL 11\/Eng. 088 - ESL & Eng. 090 - Grammar & 2 Credit Courses (Type II) English 11 or 12 & Type I or II Credit Courses MTELP Score 40 or below 4 0 - 5 4 55 and above Writing Test Score 3 2 1 Type I Courses: English Humanities Type II Courses: Math Computer Science Sciences Figure 1. Academic programs and placement test score requirements. 36 THE SUBJECTS The subjects of the study are high-school- and college-aged students (17 - 23 years old). Over 90% of the subjects are from Asian cultures, with the majority speaking Cantonese and Mandarin as native languages. All subjects were admitted to the college by the same admission criteria, wrote placement tests as previously described, and were placed in levels within programs based on test results. The subjects are divided into two groups based on their program of study, Secondary or University. Numbers of subjects are \u00ab = 53 for the Secondary group and n = 55 for the University group. The six hypotheses are tested on the two groups separately for reasons related to differences in the Secondary and University programs, i.e., GPA and AACPS are calculated according to different scales for the two programs. These differences are outlined in the section in this chapter describing the measures used in the study. DATA COLLECTION PROCEDURES Data on the subjects was collected from transcripts stored in student files and from composite files for each semester. A letter of permission to collect and use the data in the study was received from the principal of the institution (Appendix B). Dates of entry and exit, and progress through academic programs in the form of GPA and credits earned were used as data to test the hypotheses. 37 MEASURES Indirect The indirect measure for assessing students' English language proficiency is the Michigan Test of English Language Proficiency, Form B. This three-part test was designed to assess ESL students' readiness to pursue university-level study and is a retired component of the Michigan English Language Assessment Battery (MELAB). The MELAB is a battery which includes an aural comprehension component or oral interview, and a writing test in addition to the three components that now comprise the MTELP. In the shorter version (MTELP), 100 multiple-choice questions including 40 grammar, 40 vocabulary, and 20 reading comprehension items are to be completed in 75 minutes. Results of validity and reliability assessments conducted on this measure are reported in the test manual (University of Michigan English Language Institute, 1977). Further validity testing of the MTELP has been carried out by Carroll (1965), Cervenka (1978), Dizney (1965), Bauldauf and Dawson (1980), and others. Dizney (1965), in a study testing the concurrent validity of the TOEFL, the MTELP, and two other measures, found a high correlation between the MTELP and the TOEFL. Bauldauf and Dawson (1980) found that as a measure of general academic attainment, the MTELP has significant predictive validity. Scoring of the test administered in the study is by stencil overlay; raw scores are not converted. Cut-off scores for placement are similar to those recommended by the MTELP. Placements are made on the basis of these scores and those of a holistically-scored composition (see Figure 1). 38 Direct The direct measure of language proficiency used in the study is a one-hour composition written on an assigned topic, which varies from term to term. Examples of topics include: Education, Pollution, Cheating and Travel. Topics are chosen to examine students' ability to approach a broad subject, define and limit discussion of it, and present a point of view. Since explicit requirements are not specified in the test instructions, a degree of independence is required of the students. Maturity of ideas and fluency (measured according to the standards of the holistic rating scale) are valued over ability to deal with a specific discourse style. Many writing tests used for placement purposes are designed to elicit a response in one rhetorical style. Such tests reveal which students can and cannot handle a particular academic writing task; it is not difficult to determine two levels of language proficiency through use of this type of test, e.g., pass\/fail. For the programs in this study, though, finer discriminations than pass\/fail need to be made as there is a wide variation in English proficiency among the students applying for entry to the institution. Three levels of language proficiency must be determined by test scores for the Secondary Program, and four levels for the University\/University Transfer Programs. After experimenting with several test types, the college found that the current writing test is able to meet its requirements for placement of students with such a wide range of language proficiency. Compositions are scored holistically by two experienced English instructors; essays on which raters differ by more than one placement level are read by a third rater. The process is overseen by a placement co-ordinator. A description of criteria used in grading the essays and the scoring 39 technique are included in Appendix C. Holistic grading of placement essays has been the practice at the institution in the study for at least ten years and is considered a reliable language proficiency measure there and elsewhere (refer to Chapter II, Review of the Literature). A test assessing the inter-rater reliability of the direct measure used in the study was conducted using four raters scoring 21 essays. A 66.6% consensus was achieved after comparing all four raters' scores and eliminating a subject's score if there was more than a one-point discrepancy. In all, 5 students' essays were eliminated in this way. That is, in 66.6% of the cases, none of the four raters graded an essay more than one point lower or higher than did any other rater of that same essay. The usual practice in the scoring of placement essays at the institution in the study is to use two raters and allow a one-point discrepancy in scores before a third rater is employed. This assessment method is common to other holistic grading schemes, e.g., the TOEFL Test of Written English (TWE). A test assessing the inter-rater reliability of four TWE raters was conducted using four TWE-trained raters scoring the same 21 essays. A 47.6% consensus was achieved following the same procedure indicated above. In all, 11 students' essays were eliminated. The consensus achieved by the TWE raters (47.6%) was notably lower than that achieved by the in-house raters (66.6%). These percentages indicate that the reliability of the in-house direct test is greater than that of the TWE. In order to test the concurrent validity of the in-house direct measure of writing, the same 21 essays were graded by four trained TWE raters using the TWE Scoring Guide (Appendix A) and four experienced raters from the institution in the study using the English 40 Placement Information writing test criteria (Appendix C). A Spearman rank-order correlation between the scores assigned by the two groups of raters achieved a low correlation of -0.015. Although this low correlation suggests no relationship between the two writing tests as direct measures of writing ability, they would seem to have much in common. However, as the concluding chapter will point out, the reliabilities of the two measures are quite different (TWE consensus 47.6%; in-house writing test consensus 66.6%). This difference may be explained by the fact that the two measures employ different rating criteria and rater-training procedures for assessing writing proficiency. Grade Point Average Grade Point Average (GPA) is used as a measure of academic achievement in the study. Its use in first-language contexts is widespread. It was developed with standards and expectations relevant to this (first-language) population, and is applied in the same manner to ESL students in academic institutions throughout North America. Grades achieved in courses are converted according to an ordinal scale and reported on transcripts. The most common use of GPA is as a criterion for acceptance into academic institutions and to programs within institutions. Judgements as to the degree of academic success achieved by students are made upon the basis of this measure. Although this study uses GPA as one measure of academic success, it does so with reservation. GPA is used as the criterion variable in multiple regression analyses in which the contribution that certain variables make to academic success are assessed and compared. Reservations regarding the use of GPA as a criterion of academic success for ESL students are based on both research and practice; they relate to the fact that the element of time, such an important factor for ESL students. 41 is not reflected in GPA. (This topic is discussed in Chapter II, Literature Review). A second problem of using GPA as a measure of academic success for ESL students is that GPA is generally based on all courses in a program of studies. For ESL students, though, a lower success rate is usually found for courses for which previous experience in the native language would be not be expected, that is, English Literature\/Humanities (Course Type I). These courses when taken in English also demand a relatively high level of language proficiency. For courses in which previous experience in the native language would be expected, i.e., Mathematics\/Sciences (Course Type II), a higher success rate is commonly found. Also, these courses do not demand as high a level of academic language proficiency as Course Type I. In Course Type II, the language used as the vehicle of instruction is more separable from the course content than it is for Course Type I. In the most linguistically demanding subjects in Course Type I (English courses), the language is not only the vehicle of instruction, but the content itself. As well. Course Type I subjects demand a greater degree of cultural literacy than do those of Type II. ESL students who have lived in the new culture for less than a year have an exceptional handicap when compared to their first-language peers. In this study, courses have been divided into the two types as described above and GPA has been calculated for the two types separately. The predictive ability of a) indirect test scores and b) direct test scores was also examined separately to assess the degree to which course type affects GPA. Both Secondary and University students are included in the study, but calculation of GPA for these groups is based on different scales. Hence, data from the two programs were analyzed separately in order to ensure that results were accurate and generalizable to like 42 populations. GPA scales used for the institution's Secondary and University programs are shown in Figure 2. Secondary Scale Mark A B c+ c p F AUD DEF TS* R W SP UP TS* Description Points 86-100 4.0 73-85 3.0 67-72 2.5 60-66 2.0 50-59 1.0 BELOW 50 0.0 AUDIT 0.0 DEFERRED 0.0 TRANSFER STANDING N\/A COURSE REPEATED N\/A WITHDRAWAL 0.0 SATISFAC PROG 0.0 UNSATISFAC PROG 0.0 GRANTED ON THE BASIS OF A PASSING MARK IN AN EQUIVALENT COURSE ErrHER ON AN INTERNATIONALLY RECOGNIZED EXAMINATION OR A CANADIAN OR U.S. HIGH SCHOOL TRANSCRIPT Post Secondary Scale Mark A+ A A-B-l-B B-C + c c-D F DEF R W INC Description 95-100 89-94 86-88 82-85 77-81 73-76 67-72 60-66 56-59 50-55 BELOW 50 DEFERRED COURSE REPEATED WITHDRAWAL INCOMPLETE Points 4.3 4.0 3.7 3.3 3.0 2.7 2.3 2.0 1.7 1.0 0.0 0.0 N\/A 0.0 0.0 Figure 2. Letter grades, percentages, and GPA points for Secondary and University Programs. The equivalent values for the course letter grades, the range of percentages for each letter grade, and the number of grade points awarded are shown in Figure 2. Grade point average (GPA) is the average of these points for all courses taken. It is important to note that the GPA scale for the Secondary Program ranges from 0.0 to 4.0 in increments of .5, while the scale for the University Program ranges from 0.0 to 4.3 in increments of .3, .4, or .7. Also, both the letter and percentage scales for the University Program are based on smaller increments of value than are those of the Secondary Program. Due to these differences, the data for the two programs was analysed separately. 43 Academic Credit The time factor in second-language learning (Cummins, 1979) is considered in this study through use of average accumulated credit per semester (AACPS) as a measure of academic success. Accumulated credit is, like GPA, calculated on the basis of course grades, and indicates the number of courses successfully completed per semester and academic year. ESL students often reduce course loads, taking longer than the prescribed time to complete a program of study. These reductions are not reflected adequately in GPA; reducing course loads often serves to ensure a higher GPA than would otherwise have been achieved, as students are able to spend more time on each course. Hence, average accumulated credit per semester measures academic success in a different way than does GPA, as it takes the element of time into account. AACPS is used in the study as the criterion variable in multiple regression analysis in which the contributions of direct and indirect measures of language proficiency are assessed and compared. For each Secondary course successfully completed, students receive one credit. For each full-term University course successfully completed, students receive three credits; for each half-term course, 1.5 credits. Since, as in the case of GPA above, Secondary and University course credits are based on different scales, data from the two categories of students has been analyzed separately in the study to enhance generalizability. HYPOTHESES The hypotheses tested in the study are based on assumptions derived from involvement in the process of administering large numbers of placement tests over a period 44 of several years, placing students in classes upon the basis of such tests, and observing their progress. This extensive experience has led to questions regarding the connection between ESL writing ability and academic success, and the predictive value of direct and indirect placement measures. The sense that direct measures, i.e., writing tests, are more accurate than indirect measures in predicting academic success, and hence more useful in making placement decisions, seems particularly true in two situations: a) for cases in which writing test scores and indirect test scores indicate different placement levels and b) for predicting students' success in courses where background in the native language would not be expected, i.e., English\/Humanities. In addition to the directions indicated by experience in testing and placement, a substantial amount of research in the field suggests that proficiency tests should require students to perform the activity which will be the basis of assessment in future courses. (Johns, 1981; Horowitz, 1986; Shih, 1986; Ostler, 1980; Graham, 1987; Brown, 1989; Farhady, 1982; Cumming, 1989). Thus, writing test scores should be the best predictors of academic success for ESL college students. On the basis of both research and practice, the following hypotheses have been formed. The hypotheses refer to two groups of students. Secondary and University. HI Direct test scores are better predictors of academic success as measured by GPA than are indirect test scores. H2 Both direct and indirect measures together better predict academic success as measured by GPA than either measure separately. H3 Direct test scores are better predictors of GPA than indirect test scores in 45 ycourses where background in the native language would not be expected, i.e., English or Humanities (Course Type I). H4 Direct test scores are not better predictors of GPA than indirect test scores in courses where background in the native language would be expected, i.e.. Math or Sciences (Course Type II). H5 Direct test scores are better predictors of academic success as measured by average accumulated credit per semester (AACPS) than are indirect test scores. H6 Direct and indirect measures together better predict academic success as measured by average accumulated credit per semester (AACPS) than either measure separately. DATA ANALYSIS The six kinds of data collected on the subjects in the study are a) indirect test scores, b) direct test scores, c) GPA for all courses combined, d) GPA for Course Type I, e) GPA for Course Type II and f) AACPS. The two types of analysis, correlation analysis and multiple regression, are applied to the data. Table 1 shows the data analysis procedures, the independent (or comparison) variables and the dependent (or comparison) variables for each hypothesis. Following Table 1, the statistical procedures used in the study are described. Table 1 Data Analysis: Tests and Variables 46 Hypothesis Test Independent\/ Comparison Variables Dependent\/ Comparison Variables Multiple regression analysis Direct\/Indirect test scores Direct\/Indirect\/ Combined test scores Direct\/Indirect test scores GPA GPA for Course Type I GPA for Course Type II AACPS Direct\/Indirect\/ Combined test scores AACPS Correlational Analysis Correlational analysis is used in order to examine the relationship between test scores and the six variables a) indirect test scores, b) direct test scores, c) GPA for all courses, d) GPA for Course Type I, e) GPA for Course Type II, and f) AACPS. The statistic used is the Pearson product-moment coefficient. Results are shown in correlation matrices in Tables 3, 5 and 8. For this study, correlation coefficients of .35 and below 47 are to be considered low, those between .35 and .75 moderate, and those over .75 are to be considered high (McMillan & Schumacher, 1989). Multiple Regression Analysis The testing of the hypotheses requires assessing the relative contribution of three predictor variables, a) direct, b) indirect and c) combined test scores to four criterion variables, a) overall GPA, b) GPA for Course Type I, c) GPA for Course Type II and d) AACPS. The criterion of significance for these analyses is set ztp < .05. Results of these analyses are shown in Tables 4, 6, 7 and 9. SUMMARY This chapter has described the design of the study, the setting, subjects, data collection procedures, and the measures. It has presented six hypotheses to be tested, and described the methods of data analysis used, correlational and multiple regression analysis. Chapter IV will present the results of the testing of the hypotheses and discuss findings for the two groups of students. Secondary and University, separately. 48 CHAPTER IV. RESULTS AND DISCUSSION INTRODUCTION In this chapter, the subjects are further defined by their test scores and GPA (Table 2), the six hypotheses to be tested are re-stated with results for each presented in brief form in the next three sections. The final section of this chapter presents a more extensive discussion of the results. Data pertaining to Hypotheses 1 and 2 are displayed in Tables 3 and 4. Tables 5, 6 and 7 present results for Hypotheses 3 and 4, and Tables 8 and 9 show results for Hypotheses 5 and 6. Table 2 Indirect Test Scores Direct Test Scores and GPA for Secondary and University Students Secondary (n = 53) University (\u00ab = 55) Indirect Test Scores Mean SD 55.32 17.09 58.00 14.96 Direct Test Scores Mean SD eiM 30.10 60.33 26.25 GPA Mean SD 58.90 17.37 62.64 12.21 Table 2 shows the means and standard deviations of the indirect test scores, direct 49 test scores and GPA scores for the two groups. GPA is calculated on a scale of 0.0 to 4.0 for Secondary students and a scale of 0.0 to 4.3 for University students (see Figure 2). In addition the in-house direct tests are assessed on a scale of 0 to 3 for Secondary students and a scale of 0 to 4 for University students (see Appendix C). The indirect test score is calculated as a percentage of 100. To make comparison among the three measures easier, direct test scores and GPA have been converted to percentages (see Appendices D and E for conversion scales). HYPOTHESES 1 AND 2 For Hypothesis 1 and 2 a Pearson product-moment correlation is used to show the relationship between the independent variables a) direct test scores and b) indirect test scores, and the dependent variable GPA. Results are shown in Table 3. Multiple regression is used to assess the contribution of the three predictor variables a) direct test scores, b) indirect test scores and c) combined test scores to the criterion variable GPA. Results are shown in Table 4. For both hypotheses, the two groups of students. Secondary and University, are assessed separately. The hypotheses state: HI Direct test scores are better predictors of academic success as measured by GPA than are indirect test scores. H2 Both direct and indirect measures together better predict academic success as measured by GPA than either measure separately. 50 Table 3 Intercorrelations Between Test Scores and GPA GPA Indirect Direct Test Test Scores Scores Secondary Students (n = 53) GPA - 0.211 0.287 Indirect Test Scores -- 0.625 Direct Test Scores University Students (n = 55) GPA -- -0.181 -0.027 Indirect Test Scores - 0.533 Direct Test Scores 51 Table 4 Predicting Overall GPA: Multiple Regression Analysis Summary for Indirect, Direct and Combined Test Scores Predictors \/S F R R^ R^ (adj) Secondary Students (n = 53) Indirect 0.21 2.38 .212 .045 .026 Test Scores Direct 0.16 4.56* .286 .082 .064 Test Scores Combined 2.29 .289 .084 .047 Test Scores University Students (n = 55) Indirect -0.158 2.15 .197 .039 .021 Test Scores Direct -0.076 1.49 .164 .027 .009 Test Scores Combined 1.19 .209 .044 .007 Test Scores * p = < .05. Secondary Students As indicated in Table 3, correlations between both direct or indirect test scores and GPA are low: r = .287 for direct test scores, and r = .211 for indirect test scores. 52 Results of the multiple regression analysis (Table 4) show that direct test scores accounted for 6.4% of the variance in GPA, while indirect test scores accounted for 2.6%. Though the contribution of direct test scores was higher than that of indirect test scores, neither score contributed a significant amount to GPA. These results do not support Hypothesis 1, as they indicate that neither the direct nor indirect test scores predicted GPA for Secondary students. The results of the test of Hypothesis 2 show that direct and indirect measures combined predicted academic success as measured by overall GPA better than indirect test scores alone, but not better than direct test scores alone. As Table 4 indicates, combined test scores accounted for 4.7% of the variance in GPA (F = 2.29, df = 52, p = n.s.) and did not contribute significantly to the prediction of GPA. This contribution is greater than the 2.6% accounted for by indirect test scores and the 6.4% accounted for by direct test scores. Combined test scores proved not to be reliable predictors of GPA for Secondary students, as so little of the variance was accounted for. University Students Results in Table 3 indicate that both direct and indirect test scores showed no relationship between either indirect or direct test scores and GPA for University students (r = -0.027 and r = -0.181 respectively). The findings of the multiple regression analysis reported in Table 4 show that while the contribution of indirect and direct test scores was similar (0.9% and 2.1% respectively), neither test score accounted for a significant amount of the variance in GPA, and thus did not predict GPA. Thus, as for 53 Secondary students. Hypothesis 1 was not supported for University students. The results of the multiple regression analysis conducted for Hypothesis 2 (Table 4) indicate that the contribution of combined test scores was not significant (R^  adj = 0.7%), Thus, Hypothesis 2 was not supported; combined test scores were not reliable predictors of GPA. It must be noted here that for further comparison, each of the two categories of students, Secondary and University, were further divided into two sub-groups based on discrepancies in their placement test scores for the purpose of comparing differences in GPA between students with low writing test scores, and students with higher writing test scores. The groups were characterized as follows: Group A: Students whose indirect test scores indicated a placement of at least one level higher than their direct test scores. Placement was made according to indirect test scores. Group B: Students whose direct test scores indicated a placement of at least one level higher than their indirect test scores. Placement was made according to direct test scores. Though it was found that Secondary students in Group A (the lower writing test scorers) had significantly lower GPA than those in Group B (the higher writing test scorers), this division between groups could not be sustained for purposes of hypothesis-testing for the balance of the study due to the unusually small number of subjects in Group A Secondary (n = 6) and a wide standard deviation for the direct test scores. It remains a point of interest, though, that the lower scorers on writing test among 54 University students (Group A) did not achieve a lower GPA than the higher writing test scorers (Group B). HYPOTHESES 3 AND 4 For Hypotheses 3 and 4, courses are separated into two categories, Type I and Type II as previously described. Correlation matrices and a multiple regression analysis reflecting these categories were produced for the Secondary and University students. The dependent variable for Hypothesis 3 is GPA for Course Type I; for Hypothesis 4 it is GPA for Course Type II. In both cases, the two independent variables are indirect test scores and direct test scores. The hypotheses state: H3 Direct test scores are better predictors of GPA than indirect test scores in courses where background in the native language would not be expected, i.e., in English or Humanities (Course Type I). H4 Direct test scores are not significantly better predictors of GPA than indirect test scores in courses where background in the native language would be expected, i.e.. Math or Science (Course Type II). Table 5 shows the intercorrelations of GPA and test scores for Course Type I and Course Type II for Secondary and University students. Table 6 indicates results of multiple regression analysis for Course Type I for Secondary and University students, and Table 8 shows results for Course Type II for both groups of students. 55 Table 5 Intercorrelations Between Test Scores and GPAfor Course Type I and Course Type II Type I GPA Type II GPA Indirect Test Direct Test Scores Scores Secondary Students (n = 53) Type I GPA -- 0.428 0.481 0.506 Type II GPA -- -0.087 0.003 Indirect Test - 0.625 Scores Direct Test Scores University Students (n = 55) Type I GPA -- 0.440 Type II GPA Indirect Test Scores Direct Test Scores 0.204 -0.384 -.\u2014 0.073 -0.163 0.553 56 Table 6 Predicting GPA for Course Type I for Secondary and University Students Predictors j8 F R R^ R^ (adj) Secondary Students (n = 53) Indirect 0.52 15.33* .480 .231 .216 Test Scores Direct 0.31 17.52* .506 .256 .241 Test Scores Combined 10.73* .548 .300 .272 Test Scores University Students (n = 55) Indirect 0.13 2.31 .204 .042 .024 Test Scores Direct 0.03 0.29 .071 .005 .000 Test Scores Combined 1.18 .210 .044 .007 Test Scores *p <.05. 57 Table 7 Predicting GPA for Course Type 11 for Secondary and University Students Predictors j8 F R R^  R^  (adj) Secondary Students (n = 53) Indirect Test -0.12 0.39 .089 .008 .000 Scores Direct Test 0.00 0.00 .000 .000 .000 Scores Combined 0.33 .114 .013 .000 Test Scores University Students (n = 55) Indirect -0.52 1.44 .383 .147 .131 Test Scores Direct 9.14* .163 .027 .080 Test Scores -0.13 Combined 4.57* .386 .149 .117 Test Scores *p = < .05. Secondary Students As stated in Hypothesis 3, direct test scores correlated more highly with GPA for Course Type I (English\/Humanities) than indirect test scores, but not significantly higher 58 (r = 0.506; r = 0.481 respectively; see Table 5). Results of the multiple regression analysis shown in Table 6 indicate that direct test scores accounted for 24.1% of the variance in GPA, while indirect test scores accounted for 21.6%. Both direct and indirect test scores appear to predict GPA to a moderate degree for Secondary students. For Hypothesis 4, both direct and indirect test scores showed low correlations with GPA for Course Type II (r = 0.003 and r = -0.087 respectively; see Table 5). In addition, the results of multiple regression analysis shown in Table 7 indicate that neither direct nor indirect test scores contributed significantly to GPA for Course Type II (0.0% in both cases). Thus, test scores do not appear to be predictors of GPA for Course Type II for Secondary students. University Students Results of correlation analysis shown in Table 5 indicate that for University students, neither direct nor indirect test scores correlated significantly to GPA for Course Type I (.073 and .204 respectively; see also Table 7). For Course Type II, both test scores showed negative correlations with GPA, with indirect test scores more highly correlated (r = -0.384 and r = -0.163 respectively). Results of multiple regression analysis (Table 6) indicate that neither direct nor indirect test scores accounted for a significant amount of the variance in GPA for Course Type I (0.0% and 2.4% respectively). Thus, Hypothesis 3 was not supported; neither direct nor indirect test scores predicted GPA for Course Type I for University students. For Course Type II, indirect test scores accounted for more of the variance in GPA than did direct test scores (13.1% and 0.8% respectively; see Table 7). Thus hypothesis 4 was not supported. It is 59 noteworthy that indirect test scores were much better predictors than direct test scores for Course Type II GPA, but since they did not account for a significant amount of the variation in Course Type II GPA, they did not prove to be reliable predictors for University students. HYPOTHESES 5 AND 6 For Hypothesis 5 and 6, the relationship between the dependent variable, average accumulated credit per semester (AACPS) and two independent variables, a) indirect test scores and b) direct test scores is assessed. This relationship is measured by a separate Pearson product-moment correlation for the two groups. Secondary and University. For these hypotheses, multiple regression analysis is used to assess the contribution of three predictor variables, a) indirect test scores, b) direct test scores and c) combined test scores to the criterion variable, AACPS. Hypotheses 5 and 6 state: H5 Direct test scores are better predictors of academic success as measured by average accumulated credit per semester (AACPS) than are indirect test scores. H6 Both direct and indirect measures together better predict academic success as measured by average accumulated credit per semester (AACPS) than either measure separately. Results of the correlation analysis for these two hypotheses are shown in the correlation matrices in Table 8. Table 9 shows the results of the multiple regression analysis for the two hypotheses. 60 Table 8 Intercorrelations Between Test Scores and Average Accumulated Credit per Semester (AACPS) AACPS Indirect Direct Test Scores Test Scores Secondary Students (n = 53) AACPS -- -0.072 0.079 Indirect Test Scores -- 0.625 Direct Test Scores University Students (n = 55) AACPS -- 0.348 0.444 Indirect Test Scores -- 0.533 Direct Test Scores 61 Table 9 Predicting Average Accumulated Credit per Semester (AACPS): Multiple Regression Analysis for Indirect, Direct and Combined Test Scores Predictors \/? F R R^  R^  (adj) Secondary Students (n = 53) Indirect -0.002 0.27 0.071 .050 .000 Test Scores Direct 0.001 0.32 0.079 .060 .000 Test Scores Combined 0.78 0.173 .030 .000 Test Scores University Students (n = 55) Indirect 0.066 7.19* 0.345 .119 .103 Test Scores Direct 0.048 12.99* 0.444 .197 .182 Test Scores Combined 7.00* 0.460 .212 .184 Test Scores p = <.05. Secondary Students The results in Table 8 indicate that for Secondary students, both direct and 62 indirect test scores showed low correlations with AACPS (r = 0.079 and r = -0.072 respectively). As shown in Table 9, the three predictors, direct test scores, indirect test scores, and combined test scores did not contribute to AACPS (R^  adj = 0.0% in all cases). Test scores are unable to predict academic success as measured by AACPS for Secondary students, thus, Hypotheses 5 and 6 were not supported. University Students For University students, results in Table 8 indicate that both direct and indirect test scores showed moderate correlations with AACPS, with direct test scores correlating more highly than indirect test scores (r = .444 and r = .348 respectively). As hypothesized in H5, direct test scores were found to be better predictors of average accumulated credit per semester than were indirect test scores. Table 9 indicates that direct test scores explained 18.2% of the variance (F = 12.99, df = 54, p < .001) with indirect test scores explaining 10.3% (F = 7.19, df = 54, p < .01). The data also supports Hypothesis 6, as combined test scores accounted for the greatest amount of the variance in AACPS at 18.4% (F = 7.09, df = 54, p < .01). In the case of both direct and combined test scores, the moderate correlation coefficients and the high F-values of the multiple regression analysis indicate that for University students, both direct and combined test scores may be useful, though limited, predictors of academic success. 63 DISCUSSION Hypotheses 1. 2. 3 and 4 University Students. In the case of University students, overall GPA, GPA for Course Type I or GPA for Course Type II were not predicted by either indirect or direct test scores. (This finding was also supported by the comparison of GPA scores between lower writing test scorers (Group A) and higher writing test scorers (Group B), in which higher scorers did not achieve higher GPA.) Hypothesis 3, stating that direct test scores would predict GPA for Course Type I (English\/ Humanities), was unsupported. Thus, factors other than language proficiency may have contributed to GPA for these students. There may be several reasons why GPA was not predicted at all for the University students, but was moderately predicted for Course Type I for Secondary students; some are included here. First, more stringent academic expectations are placed on University students than on Secondary students. For University level courses, the volume of content increases dramatically over that of Secondary courses, and only some of the cognitive skills required to process this input are measured by language proficiency tests. There are higher expectations for University students to produce original ideas, and this ability is better measured by direct tests than indirect tests. In addition, research has shown that when grading, instructors put varying degrees of emphasis on the evidence of these ideas as opposed to the linguistic fluency demonstrated in explaining them (Santos, 1988, Sheory, 1986, Vann et al., 1984, and others). Perhaps instructors rate language fluency 64 above evidence of original ideas for Secondary students, but expect more originality from University students. This variation in instructors' expectations could work in two directions, which may further complicate prediction studies. If language proficiency is not valued as highly as a) originality of ideas and b) ability to relate content to other disciplines, then highly proficient writers could fare no better than those of lower proficiency, as long as the lower group's writing skills were adequate to show evidence of the more valued attributes i.e., a) and b) above. Thus, language proficiency appears to be just one factor upon which academic success depends. Also, in the category of Course Type I, English 101 is the only required course for University students; choices are offered within the Humanities, and aptitude for and interest in such elective courses would affect success. Since these attributes are not measured by language proficiency tests, other types of studies would have to be designed to test the contribution of such variables to GPA. Finally, as is evidenced in the tests of Hypotheses 5 and 6, perhaps there are other, more valid measures of academic success for University students. The use of GPA as the sole measure does not take the important element of time into account. The tests for Hypotheses 5 and 6 use AACPS as a criterion variable, and here, test results for University students were quite different from results of the tests for which GPA was the criterion variable. Secondary Students. For Hypotheses 1 and 2, multiple regression analyses revealed that writing scores accounted for only 6.4% of the variance in overall GPA for Secondary students, with the 65 other two independent variables, indirect and combined test scores, accounting for even less (see Table 4). It appears that overall GPA cannot be reliably predicted by placement test scores for the students in the study. These findings indicate that there must be many other factors that contribute to overall GPA, such as motivation, personal attributes, and presence or absence of external pressures, among others. (It is interesting to note that for Secondary students in the smaller comparison groups, lower writing test scorers (Group A) achieved lower overall GPA than did higher writing test scorers (Group B).) In light of the results for Hypotheses 1 and 2, the findings for Hypothesis 3, which indicate that better writers achieve higher GPA for Course Type I (English\/Humanities), are of interest. For Hypotheses 3 and 4, courses were separated into Type I (English\/ Humanities) and Type II (Math\/Sciences), as previously described. Correlations and predictive ability of test scores and GPA for each course type were measured separately. Here, GPA for Course Type I (English\/Humanities) correlated moderately to both direct and indirect test scores (see Table 5), while for Course Type II (Math\/Science) though, both test scores showed low correlations with GPA. In addition results of multiple regression analysis shown in Tables 6 and 7 indicate that test scores were moderate predictors of GPA for Course Type I, but did not predict GPA for Course Type II. These findings indicate that for Secondary students, the use of GPA for Course Type II decreases the ability of the test scores to predict overall GPA. That is, direct and indirect test scores can be relied upon somewhat to predict GPA for courses in which a higher level of language proficiency is required (English\/Humanities), but when GPA is included for courses in which language proficiency is less important (Math\/ Science), 66 predictive ability of the test is weakened. Hypotheses 5 and 6 University Students. For University students, results of tests for Hypotheses 1, 2, 3, and 4 show that academic success as measured by GPA was not predicted by any of the independent variables. But tests for Hypotheses 5 and 6 showed that academic success as measured by (AACPS) was predicted moderately well by two of the independent variables. Both direct and combined test scores predicted academic success by this measure moderately well: R^  adj = 18.2% and 18.4% respectively (see Table 9). The fact that GPA was not predicted for University students but average credit per semester was, raises the question of just what constitutes academic success, and what the criteria for its measurement should be. As discussed by Black (1991), Graham (1987), Light et al. (1987), and others, the element of time is not taken into consideration in the conventional practice of using GPA as the criterion by which academic success is measured. The strategy in which students use withdrawal from and repetition of courses to increase GPA may help to explain these results. Thus, despite the fact that test scores did not predict success by the criterion of GPA, they did predict moderately well for AACPS. Secondary Students. For Secondary students, none of the independent variables predicted academic success as measured by average accumulated credit per semester. This finding differed from the results for University students, where direct and combined test scores predicted success as measured by this criterion variable moderately well. Some possible reasons 67 for the different results for Secondary and University students are included here. Students must have the opportunity to make adjustments to course load in order for average accumulated credit per semester to be an effective measure of academic success. Secondary students, though, are not allowed to withdraw from or repeat courses in the same way University students are, and thus may carry courses in which they are not faring well through to the end of the semester. Secondary programs are less flexible than are University programs; there are 13 required courses and four academic electives in the provincial Senior Secondary Program, while English 101 is the only required course for University students. Also, Secondary students' perceptions of their progress in courses may not be as accurate as that of the more mature University students and they may stay in courses past the withdrawal deadline for this reason. For Secondary students, time is of the essence, as they have a much longer academic road to travel to graduation from university than do students already studying in the University program. Thus, course withdrawal may not be as readily accomplished by Secondary students, even if the option to do so were not so restricted. The students in the study are from educational systems in which programs of study are relatively rigid and are followed as prescribed. This previous academic experience, along with their younger age, may hinder Secondary students' ability to realistically assess their academic paths and the options open to them. Also, Secondary students may be more reluctant to make changes to their programs, as most institutions notify the parents of students under 19 years of age of any program changes including repetition of and withdrawal from courses. Thus, students may be deterred from making 68 changes for fear of parents' disapproval. The opportunity to use the strategy of adjusting course load, then, is not as available to Secondary students as it is to University students due to the design of the particular programs. This fact has an effect on the usefulness of average accumulated credit per semester as a measure of academic success for Secondary students. SUMMARY This chapter presented results for the six hypotheses in tabular and discussion form, then extended the discussion in subsequent sections. Though it was hypothesized that direct test scores would be better indicators of academic success as measured by GPA, this hypothesis was supported only for Secondary students in the case of Course Type I (English\/Humanities). It was not supported for University students for any course type. The results also indicate that direct test scores are better predictors of AACPS than GPA for University students, but that AACPS is not a useful measure at all for Secondary students. The following and final chapter will draw conclusions from the results and examine implications of the research. It will also suggest directions for further research and discuss the limitations of the study. 69 CHAPTER V: SUMMARY, CONCLUSIONS AND IMPLICATIONS OF THE RESEARCH INTRODUCTION The main focus of the study was the question of whether writing test scores give a more accurate indication of academic success than do indirect test scores. The study investigated whether indirect or direct (writing) test scores should be the deciding factor for placement purposes for students who have discrepancies in these scores. Based on the results described in Chapter IV, some conclusions have been drawn that answer these questions to a degree for the students in the study. Results are summarized below for each hypothesis, and conclusions and implications of the study follow. SUMMARY OF RESULTS HI As hypothesized for Secondary students, direct test scores were better predictors of academic success as measured by GPA than were indirect test scores, but only marginally so. Neither test score was found to be a reliable predictor (R^ adj = 6.4% and 2.6% respectively). For University students, neither direct nor indirect test scores predicted GPA (R^adj = 0.9% and 2.1%). H2 As hypothesized for Secondary students, direct test scores predicted academic success as measured by GPA better than either indirect test scores or combined test scores, but only marginally better (R^  adj = 6.4%, 2.6% and 4.7% respectively). Neither test score was found to be a reliable predictor. For University students, combined test scores did not 70 predict GPA. (R^ adj = 0.7%). H3 As hypothesized for Secondary students, direct test scores were better predictors than indirect test scores of GPA for courses where background in the native language would not be expected, i.e., English\/ Humanities (Course Type I) (R^  adj = 24.1% and 21.6% respectively). For the University students tested, neither direct nor indirect test scores predicted GPA for this course type (R^ adj =0.0% and 2.4% respectively). H4 As hypothesized for Secondary students, direct test scores were not significantly better predictors of GPA than indirect test scores where background in the native language would be expected, i.e., Math\/Sciences (Course Type II) (R^  adj = 0.0% for both kinds of test scores). For University students, indirect test scores predicted GPA for Course Type II better than direct test scores ( R^  adj = 13.1% and 8.0% respectively) . H5 For Secondary students, neither direct nor indirect test scores predicted academic success as measured by AACPS ( R^  adj = 0.0% for both kinds of test scores). But, as hypothesized for University students, direct test scores were better predictors of AACPS than indirect test scores (R^  adj = 18.2% and 10.3% respectively). H6 For Secondary students, both direct and indirect test scores together were not better predictors of academic success as measured by AACPS than either measure separately( R^  adj = 0.0%). For University students, direct test scores and both direct and indirect scores combined predicted this measure of academic success equally well (R^  adj = 18.2% and 18.4% respectively). 71 CONCLUSIONS AND IMPLICATIONS As can be seen above, not all of the hypotheses were supported for the students in the study. Some conclusions that can be drawn from the results, and implications for practice are elaborated below. Since results differed in some instances for Secondary and University students, the two groups will be discussed separately. Secondary Students Writing scores as predictors of academic success are only reliable to a moderate degree when predicting success for courses in which previous experience in the native language would not be expected, such as English and Humanities. They are not reliable predictors for courses in which students have accumulated considerable experience, such as Math and Sciences, or when Course Types I and II are combined. Since courses of the first type also require a higher level of language proficiency than do those of the second type, it may be reasonable to conclude that writing proficiency does indicate how well students will do in more linguistically demanding courses. A direction for practice here may be that students with lower writing scores could be counselled into courses with less demanding language proficiency requirements for as long as possible, while providing writing tutorial classes. An important implication that may be drawn from these results for testing and placement personnel is that Secondary students should be placed according to their writing test scores when these scores indicate placement at a level lower than their indirect test scores if Type I courses are to be included in the first semesters of their programs. Finally, it may be possible to conclude from results of the testing of Hypotheses 5 and 6 that the use of AACPS as a measure of academic success for Secondary students is not 72 advisable, as the opportunity to adjust course load to a significant degree is not available to students in this program. University Students Neither writing nor indirect test scores predict academic success as measured by GPA, which indicates that there are other factors that contribute to academic success for University students. The scores used in the study are derived from language proficiency tests and it is not surprising that these scores are not predictors of success for courses in which language proficiency is less important than other types of skills, such as those required for mathematics (e.g., logic, problem-solving). Despite the fact that a certain level of language ability is required in Type II courses (Math\/Sciences), there seem to be other skills that are more important for success. Hence, scores from tests such as the GRE Quantitative may be better predictors for Type II courses than those used in this study (see Graham, 1987). Also, practitioners know that many ESL students use textbooks in their native languages as study tools for courses such as mathematics. This and the fact that students have had previous experience with the content of Type II courses in their home countries substantially decreases their dependence on a high level of proficiency in English for success in these types of courses. Writing and combined test scores do predict academic success moderately well when AACPS instead of GPA is used as the measure of this criterion. These findings indicate that time is indeed an important aspect of academic success; both writing and combined test scores may indicate the length of time it will take students to complete a program of studies (see Cummins (1979a) re: Cognitive\/Academic Language Proficiency). Also evidenced in 73 these results, is that the strategy of dropping and\/or repeating courses is effective. Exposure not only to the subject matter, but to the culture of the class by way of teacher expectations, types of assignments given, lecture style, reading requirements, and level of competition expected from peers may indeed contribute to higher grades when the second attempt at the course is made. But there may be a point beyond which the strategy of retaking a course is not effective, as is shown by the fact that in some cases grades achieved in the second attempt at the course are no higher than those achieved in the first attempt. For these students, perhaps the positive effect of the experience derived from this strategy is not great enough to offset deficiencies in language ability. In support of this possibility, some researchers have suggested that despite the presence of other factors which contribute to academic success, e.g., intelligence, motivation and aptitude, students with particularly low language proficiency are not likely to succeed academically until their level of language proficiency increases (Graham, 1987; Sharon, 1972). Writing and\/or combined test scores could be used as indicators of students' success in English and Humanities courses, as well as their success in using the strategy of dropping and\/or repeating courses. The most widely used criterion of academic success is GPA, a measure developed for first-language students for whom language proficiency is not nearly as much of an obstacle for academic success as it is for ESL students. GPA was not predicted for University students by the test scores in this study, but AACPS was. This may indicate that other measures of academic success need to be developed, as advocated by several researchers (Black, 1991; Graham, 1987; Heil and Aleamoni, 1974; Light, et al., 1987; Gue and Holdaway, 1973). When GPA is used as a measure of academic success, a student's rate 74 of language acquisition may be the factor that is being measured, rather than the degree to which the course content has been learned. LIMITATIONS OF THE STUDY Three limitations of this study that must be mentioned are a) the validity of the in-house holistic grading system, b) the influence of the setting on the academic performance of Secondary students and c) the length of the study. Validity of the In-House Holistic Grading System The validity of the in-house holistic grading system is an important consideration, since direct test scores are derived from this system. In an attempt to establish the concurrent validity of the direct measure used in the study, the in-house writing test was compared to the TOEFL Test of Written English (TWE). A Spearman rank-order correlation between the average scores that four TWE raters and four in-house raters assigned to the same 21 essays achieved a low correlation of -0.015. Though these two measures would appear to be similar in that they are both used a placement tools, and both assess writing by holistic means, there are two important differences which account for their low correlation. First, tests of inter-rater reliability indicated that the in-house writing test had greater reliability than the TWE writing test. In rating 21 essays and allowing for a one-point discrepancy among the four raters in each group, the raters of the in-house measure achieved a 66,6% consensus, while the consensus achieved by the TWE raters was lower at 47.6%. This difference in reliability suggests that the in-house measure is being correlated to a less reliable measure (for the raters used in this study). This discrepancy may be 75 explained by a) there are differences in the rating criteria for the two tests (see Appendices A and C) and b) there are differences in the reader-training systems and rating environments which create quite different rater expectations of the writing samples. An opportunity for the two groups of raters rated each other's tests would be helpful in establishing similar rating criteria, rater expectations of the writing, and thus, more highly correlated scores. Influence of the Setting on the Academic Performance of Secondary students A second limitation of the study is that the research took place in a college, and though Secondary students are enrolled, the college setting is quite different than that of a typical high-school which most ESL students of this age attend. The hours of instruction in the Secondary Program are increased by a minimum of 10% over the number specified by government regulations. There is an average of 21 students per class, and the support services offered in proportion to the small population of the college (400 to 700 students) is high. Also, the college is an independent institution and receives no government funding. This may have a bearing on motivation, parental expectations, and other aspects important to Secondary students' academic success. The college setting, then, may allow other influences to affect success that would not be present for students in public Secondary schools, and hence, may affect generalizability of results obtained in this study. In addition, over 90% of the students in the study were Asian; this may not reflect the demographics of other institutions to which the findings of this study could be extended. The Length of the Study The study incorporated a sampling from records collected over time and qualities inherent in this type of design could have affected results since the correlation between 76 variables usually decreases as the time span of the study increases (McMillan and Schumacher, 1989). Students accumulated grades and credits over a minimum of three semesters (one year), and this period of time provided opportunity for other factors to influence the GPA and AACPS. It also may be useful to extend the study to a full-scale longitudinal study, following several cohorts through their stay at an academic institution. EXTENSION OF THE RESEARCH There are several ways in which the research conducted here could be extended. The ability of placement test scores to predict average accumulated credit per semester could be assessed for Course Type I (English\/Humanities) and Course Type II (Math\/Sciences) separately. Since this measure of academic success was predicted moderately well by direct and combined test scores for University students, one would expect that these scores would predict success in Course Type I to a greater extent, since courses with lower language proficiency demands would be excluded. A more extensive study of factors which are related to academic success would be informative. Among the variables examined in such a study could be students' native language, number of years of previous English study, type of high-school attended in home country, sex, age, education of parents, intelligence, values, loneliness, homesickness, adaptation to Canadian culture, use of English outside of school hours, and degree of social support, and others. A combination of data-collection procedures, both qualitative and quantitative would be appropriate here, since there appears to be such a wide array of factors that contribute to academic success for ESL students. This study explored some of the more 77 quantifiable variables, i.e., test scores, but since so many other factors are likely to influence test results, test data should not necessarily be considered more meaningful than other data. Continued study on academic success and its measurement will help serve ESL students' needs; more appropriate measures for this population need to be found. SUMMARY The study has examined placement test scores as predictors of academic success for ESL students. Some conclusions have been drawn, and suggestions for application to practice have been made. There are three areas in which the study has made a contribution: First, because studies which compare Secondary and University students with regard to the effect of writing proficiency on academic success are not common, the research accomplished here will add to current knowledge in this area. Second, interest in the study arose from the recurring problem of making accurate placement decisions for students with discrepant indirect and direct test scores. The findings of the study may indicate a direction for those involved in this process. Finally, the measurement of academic success was an important issue in the study. GPA is not specifically designed to measure language proficiency, but in using it as a measure of academic success for ESL students, it is often inadvertently put to that purpose. New assessment tools for ESL students must be created, at least for the interim period in which language proficiency is being brought up to the level at which second-language students can fairly compete with native speakers. The measure of the number of credits earned per semester takes a vital aspect of second-language learning into 78 consideration \u2014 time. Perhaps a better indicator of academic success than GPA alone would be a combination of GPA, AACPS and other factors yet to be assessed. 79 REFERENCES Bauldauf, R. B., Jr., & Dawson, R. L. T. (1980). The predictive validity of the Michigan test of English language proficiency for teacher trainees in Papua New Guinea. Educational and Pschological Measurement, 40, 1201-1205. Black, J. (1991). Performance in English skills courses and overall academic achievement. TESL Canada Journal, 9, 42-55. Berwick, R., (1992). Ritsumeikan Evaluation Study. Unpublished manuscript. University of British Columbia, Vancouver. Berwick, R., (1993). Ritsumeikan Evaluation Study. Unpublished manuscript. University of British Columbia, Vancouver. Brown, J. D. (1989). Improving ESL placement tests using two perspectives. TESOL Quarterly, 23, 65-83. Brown, J. D. (1991). Do English and ESL faculties rate writing samples differently? TESOL Quarterly, 25, 587-603. Brown, J. D., & Bailey, K. M. (1984). A categorical instrument for scoring second language writing skills. Language Learning, 34, 21-42. Carroll, J. B. (1965). Review of the Michigan test of English language proficiency. In O. K. Burrows (Ed.), The Sixth Mental Measurements Yearbook (review # 360). Highland Park, New Jersey: Gryphon. Cervenka, E. J. (1965). Review of the Michigan test of English language proficiency. In O. K. Burrows (Ed.), The Eighth Mental Measurements Yearbook (review # 106). Highland Park, New Jersey: Gryphon. Cooper, C. L. (1977). Holistic evaluation of writing. In C. R. Cooper & L. Odell (Eds.), Evaluating Writing: Describing, Measuring, Judging (pp. 2-31). Illinois: National Council of Teachers of English. Cumming, A. (1989). Writing expertise and second language proficiency. Language Learning, 39, 81-141. Cumming, A. (1990). Expertise in evaluating second language composition. Language Testing, 1, 31-51. 80 Cummins, J. (1979a). Cognitive\/academic language proficiency, linguistic interdependence, the optimum age question and some other matters. Working Papers on Bilingualism, 19, 179-205. Davies, S., & West, R. (1989). English Language Examinations. Essex, England: Longman. Educational Testing Service. (1990). TOEFL Test Score Manual Princeton, New Jersey: Author. Ellis, R. (1985). Understanding Second Language Acquisition. Oxford, England: Oxford University Press. Farhady, H. (1982). Measures of language proficiency from the learners' perspective. TESOL Quarterly. 16, 43-59. Graham, J. G. (1987). English language proficiency and the prediction of academic success. TESOL Quarterly, 21. 505-521. Hanania, E., & Shikhani, M. (1986). Interrelationships among three tests of language proficiency: Standardized ESL, cloze, and writing. TESOL Quarterly, 20, 97-109. Harris, W. H. (1977). Teacher response to student writing: A study of the response patterns of high-school English teachers to determine the basis for teacher judgement of student writing. Research in the Teaching of English, 11, 175-185. Hendrickson, J. M. (1978). Error correction in foreign language teaching: Recent theory, research and practice. Modem Language Journal, 62, 386- 398. Hirsch, E. D., Jr., Khett, J. P., & Trefil, J. (1988). The Dictionary of Cultural Literacy: What Every American Needs to Know. Boston: Houghton Mifflin. Homburg, T. J. (1984). Holistic evaluation of ESL compositions: Can it be evaluated objectively? TESOL Quarterly, 18, 87-105. Horowitz, D. M. (1986). What professors actually require: Academic tasks for the ESL classroom. TESOL Quarterly 20. 445-462. Hwang, K., & Dizney, H. F. (1970) Predictive validity of the test of English as a foreign language for Chinese graduate students at an American university. Educational and Psychological Measurement, 30, 475-477. 81 Jacobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F., & Hughey, J. B. (1981). Testing ESL Composition: A Practical Approach. Rowley, Massachusetts: Newbury House. Janopoulos, M. (1989). Reader comprehension and holistic assessment of second language writing proficiency. Written Communication, 6, 218-235. Jenks, F. (1987). Michigan test of English language proficiency. In J. C. Alderson, K. J. Krahnke, &. C. W. Stansfield (Eds.), Reviews of English Language Proficiency Tests (pp. 58-60). Washington, D.C.: TESOL. Johns, A. M. (1981). Necessary English: A faculty survey. TESOL Quarterly, 15, 51-57. Krashen, S. D. (1982). Principles and Practice in Second Language Acquisition. New York: Pergamon Press. Lien, A. J. (1967). Measurement and Evaluation of Learning. Dubuque, Iowa: Wm. C. Brown. Light, R. L., Xu, M., & Mossop, J. (1987). English proficiency and academic performance of international students. TESOL Quarterly, 21, 251-261. Ling, S. (1986). Responding to product in the composing process. TESL Canada, 2, 65-75. Low, G. (1982). The direct testing of academic writing in a second language. System, 10, 247-257. McMillan, J. H., & Schumacher, S. (1989). Research in Education: A Conceptual Introduction. lUinios: Scott, Foresman. Norton-Pierce, B. (1991). TOEFL test of written English (TWE) scoring guide. Educational Testing Service, (Review). TESOL Quarterly, 25, 159-163. Ostler, S. E. (1980). A survey of academic needs for advanced ESL. TESOL Quarterly, 14. 489-502. Page, G. T., & Thomas, J. B. (1977). International Dictionary of Education. London: Kogan Page. Perkins, K. (1980). Using objective methods of attained writing proficiency to discriminate among holistic evaluations. TESOL Quarterly, 14, 61-67. 82 Perkins, K. (1983). On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly, 17, 651-669. Raimes, A. (1990). The TOEFL test of written English: causes for concern. TESOL Quarterly, 24, 427-442. Robb, T., Ross, S., & Shortreed, I. (1986). Salience of feedback on error and its effects on EFL writing quality. TESOL Quarterly, 20, 83-93. Santos, T. (1988). Professors' reactions to the academic writing of nonnative-speaking students. TESOL Quarterly, 22, 69-90. Sheory, R. (1986). Error perceptions of native-speaking and non-native-speaking teachers of ESL. ELT Journal, 40, 306-312. Shih, M. (1986). Content-based approaches to teaching academic writing. TESOL Quarterly, 20, 617-647. University of Michigan English Language Institute, Division of Testing and Certification. (1977). Michigan Test of English Language Proficiency Test Manual. Ann Arbor, Michigan: Author. Vann, R. J., Meyer, D. E., & Lorenz, F. O. (1984). Error gravity: A study of faculty opinion of ESL errors. TESOL Quarterly, 18, 427-440. Zamel, V. (1985). Responding to student writing. TESOL Quarterly, 19, 79-101. 83 APPENDICES (A -1) Appendix A Test of Written English (TWE) Scoring Guide * Readers will assign scores based on the following scoring guide. Though examinees are asked to write on a specific topic, parts of the topic may be treated by implication. Readers should focus on what the examinee does well. Scores 6 Clearly demonstrates competence in writing on both the rhetorical and syntactic levels, though it may have occasional errors. A paper in this category is well organized and well developed effectively addresses the writing task uses appropriate details to support a thesis or illustrate ideas shows unity, coherence, and progression displays consistent facility in the use of language demonstrates syntactic variety and appropriate word choice 5 Demonstrates competence in writing on both the rhetorical and syntactic levels, though it will have occasional errors. A paper in this category is generally well organized and well developed, though it may have fewer details than does a 6 paper may address some parts of the task more effectively than others shows unity, coherence, and progression demonstrates some syntactic variety and range of vocabulary displays facility in language, though it may have more errors than does a 6 paper 4 Demonstrates minimal competence in writing on both the rhetorical and syntactic levels. A paper in this category is adequately organized 84 addresses the writing topic adequately but may slight parts of the task uses some details to support a thesis or illustrate ideas demonstrates adequate but undistinguished or inconsistent facility with syntax and usage may contain some serious errors that occasionally obscure meaning. 3 Demonstrates some developing competence in writing, but it remains flawed on either tlie rhetorical or syntactic level, or both. A paper in this category may reveal one or more of the following weaknesses: inadequate organization or development failure to support or illustrate generalizations with appropriate or sufficient detail an accumulation of errors in sentence structure and\/or usage a noticeably inappropriate choice of words or word forms 2 Suggests incompetence in writing. A paper in this category is seriously flawed by one or more of the following weaknesses: failure to organize or develop little or no detail, or irrelevant specifics serious and frequent errors in usage or sentence structure serious problems with focus 1 Demonstrates incompetence in writing. A paper in this category will contain serious and persistent writing errors, may be illogical or incoherent, or may reveal the writer's inability to comprehend the question. A paper that is severely underdeveloped also falls into this category. Papers that reject the assignment or fail to address the question in any way must be given to the Table Leader. Papers that exhibit absolutely no response at all must be given to the Table Leader. * Educational Testing Service. (1990). 85 Appendix B Letter of Permission 6037 Marlborough Avenue COLUMBIA COLLEGE WW ^<^^'^HZ.. estabiished 1936 X ^ ^ ^ Telephone: (604) 430-6422 Telex: 04-3S2848 VCR FAX: (604)439-0348 February 21, 1991 Ms. Virginia Christopher c\/o Columbia College Dear Virginia: The College hereby gives its permission to conduct reseeirch for your Master's Degree as described in your letter. I take you at your word that any dissections of students will be carried out in full accordance with College policy and in a meuiner least dismiptive to other students and the College as a whole. Good luck in your study 1 Yours truly, COLUIffilA noiASE '-^ lichaei P. Weiss Principal 86 Appendix C English Placement Information Rating Essays 1. Preliminaries: a) Record first impressions - based on opening paragraph b) Quick read through the rest of the essay to confirm or contradict your first impression 2. Criteria (Micro features): a) Grammar (accuracy, complexity) b) Fluency (flow, voice\/tone, diction) c) Maturity (level of ideas, handling of ideas, response to the topic) d) Positive features (a lot to say, use of idiom) 3. Conclusion: a) Potential to succeed in English ? b) Score ? Secondary Program Scale: 1 = Senior Secondary (English 11\/12) 2 = Academic Prep (English 088\/090) 3 = Full-time ESL (retest) University Program Scale: la = English 110\/120 recommended 1 = English 101 2 = English 099 3 = English 098\/090 4 = Full-time ESL (retest) 87 Appendix D Conversion Table: GPA to Percentages Secondary GPA Points 4.0 3.0 2.5 2.0 1.0 0.0 Scale Percentage 100.0 75.0 62.5 50.0 25.0 00.0 University GPA Points 4.3 4.0 3.7 3.3 3.0 2.7 2.3 2.0 1.7 1.0 0.0 Scale Percentage 100.00 93.02 86.04 76.74 69.76 62.79 53.48 46.51 39.53 23.25 00.00 GPA was calculated for each subject, then converted to a percentage based on the table above. Appendix E 88 Conversion Table: Direct Test Scores to Percentages Secondary Direct Test Score 1.0 1.5 2.0 2.5 3.0 Scale Percentage 100.0 67.0 50.0 33.0 00.0 University Direct Test Score 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Scale Percentage 100.0 83.0 67.0 50.0 33.0 16.0 00.0 Essays were rated by two or more raters, scores were averaged and converted to percentages according the scale above. 89 APPENDKF Data for In-house Direct Test ROU 1 2 3 U 5 6 7 8 9 10 11 12 13 U 15 16 17 18 19 20 21 \\ test i 3.0 1.0 1.0 1.0 1.0 1.0 2.0 3.0 3.0 2.0 3.0 1.0 1.0 2.0 2.0 2.0 3.0 2.0 3:0 1.5 1.0 a test2 1 1 2 2 1 2 3 3 3 1 3 3 2 3 3 2 2 2 3 1 1 3 test3 1.5 1.0 1.0 1.0 1.0 1.0 3.0 2.0 2.0 1.5 3.0 1.0 1.0 2.5 2.0 2.0 2.0 2.0 2.0 1.0 1.5 4-tst4 1 3 1 2 1 3 3 3 4 2 4 3 2 2 3 3 3 2 4 4 1 s aver 1.625 1.500 1.250 1.500 1.000 1.750 2.750 2.750 3.000 1.625 3.250 2.000 1.500 2.375 2.500 2.250 2.500 2.000 3.000 1.875 1.125 G> ccX 40.625 37.500 31.250 37.500 25.000 43.750 68.750 68.750 75.000 40.625 81.250 50.000 37.500 59.375 62.500 56.250 62.500 50.000 75.000 46.875 28.125 1 race 7.5 5.0 3.0 5.0 1.0 9.0 17.5 17.5 19.5 7.5 21.0 11.5 5.0 14.0 15.5 13.0 15.5 11.5 19.5 10.0 2.0 Column 1 Scores for rater #1 Column 2 Scores for rater #2 Column 3 Scores for rater #3 Column 4 Scores for rater #4 Column 5 Average of four scores Column 6 Average converted to percentages Column 7 Ranked averages APPENDIX G Data for TOEFL Test of Written English 90 I a 3 tf $r ROU 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 twel 4 5 3 5 3 4 4 4 5 5 4 5 5 3 4 5 5 5 3 4 3 twe2 4 4 4 5 3 3 4 3 3 4 4 4 4 4 4 3 3 3 3 3 3 twe3 4 4 3 4 3 3 4 2 4 5 4 3 3 3 2 3 3 3 3 3 2 twe4 3 4 3 3 2 3 3 2 4 4 3 4 4 2 4 3 3 3 2 2 2 tueav 3.75 4.25 3.25 4.25 2.75 3.25 3.75 2.75 4.00 4.50 Z.7S 4.00 4.00 3.00 3.50 3.50 3.50 3.50 2.75 3.00 2.50 tueX 62.5000 70.8333 54.1667 70.8333 45.8333 54.1667 62.5000 45.8333 66.6667 75.0000 62.5000 66.6667 66.6667 50.0000 58.3333 58.3333 58.3333 58.3333 45.8333 50.0000 41.6667 ratwe 14.0 19.5 7.5 19.5 3.0 7.5 14.0 3.0 17.0 21.0 14.0 17.0 17.0 5.5 10.5 10.5 10.5 10.5 3.0 5.5 1.0 Column 1 Scores for rater #1 Column 2 Scores for rater #2 Column 3 Scores for rater #3 Column 4 Scores for rater #4 Column 5 Average of four scores Column 6 Average converted to percentages Column 7 Ranked averages 91 APPENDIX H Data for Secottdaiy Students (for 27 of the S3 subjects) \\ a 3 H- ^ fo ROW stsecobj stsecwrt stsecgpa stsectyl stsecty2 stsecrsm Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 1 2 3 4 5 6 7 8 9 10 11 12 13 U 15 16 17 18 19 20 21 22 23 24 25 26 27 40 54 45 41 65 42 48 39 37 50 39 39 37 36 47 37 35 39 50 40 41 47 38 34 45 46 50 0 50 0 0 33 0 83 67 50 67 50 50 33 33 100 50 67 50 83 50 86 67 67 89 33 33 33 55.75 14.25 60.25 58,75 15.00 41.75 55.25 75.00 52.75 64.50 53.00 48.00 81.75 45.25 62.50 38.75 46.25 32.25 65.00 39.75 77.25 29.25 47.00 80.00 75.00 42.00 48.25 52.00 10.50 55-37 52.00 8.25 12.50 53.12 56.25 25.00 61.37 50.00 50.37 65.25 46.25 61.37 37.50 31.25 33.25 65.26 42.12 60.37 35.37 41.62 68.75 65.25 25.00 25.00 60.00 25.00 66.50 75.00 12.50 65.00 57-00 91.50 75.00 71.75 54.25 41.50 100.00 45.00 68.75 39.25 62.50 . 25-00 59-50 22-25 95-25 0-00 51.75 92.75 100.00 58.25 79.25 Objective test score Writing test score GPA converted to percentages GPA for Course Type I converted to percentages GPA for Course Type II converted to percentages AACPS 2,37 1.50 3.17 3.00 1-50 1-75 1-75 3-00 1.50 1.67 3.00 3.00 3.30 3.50 3.75 3.30 2.67 2.60 3.00 3.30 3.50 3.30 3.00 3-25 3,25 3.00 3.75 92 Data for Secondary Students (for 26 of the 53 subjects) 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 50 53 53 51 47 47 46 58 68 64 63 61 59 69 89 84 83 82 77 75 75 75 70 92 91 89 33 33 33 33 83 50 33 83 67 67 83 83 83 67 100 100 83 100 100 83 100 100 100 100 100 100 37.50 66.50 65.50 85.50 43.75 88.25 48.00 51.50 77.00 79.25 86.25 61.25 88.50 66.75 50.50 63.50 71.50 76.75 68.25 60.00 53.00 68.00 71.75 53.25 64.50 70.75 25,00 60.00 55.00 62.50 40.00 81.25 41.65 53.12 73.37 78.75 55.62 43.75 65.62 66.62 71.87 70.00 73-00 81.25 68.75 58,25 52,37 68.75 70.75 68.75 67.50 75.00 41.75 73.00 75.00 100.00 62.50 94.25 58.25 66.50 77.50 75.00 95.00 75.00 100.00 75.00 25.00 50.00 71.50 67.75 83.25 50.00 72.75 75.00 40.00 65.00 65.00 25.00 3.75 3.25 3.25 2.66 3.25 2.33 3.25 3.00 3.50 3.75 4.00 3.25 3.00 2.50 3.00 2.80 3.50 2.50 2.75 2.00 3.67 2.00 3.83 3.00 2.30 2.38 Column 1 Objective test score Column 2 Writing test score Column 3 GPA converted to percentages Column 4 GPA for Course Type I converted to percentages Column 5 GPA for Course Type II converted to percentages Column 6 AACPS APPENDIX I Data for University Students (for 28 of the 55 subjects) I a 3 M- S\" G ROW stunobj stunwrt stungpa stuntyl stunty2 stuncrsm 93 Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 1 2 3 4 5 6 7 8 9 10 11 12 13 \u00ab 15 16 17 18 19 20 21 22 23 24 25 26 27 28 51 50 42 45 44 43 41 53 54 53 52 64 59 57 56 55 54 67 66 66 55 95 87 86 80 79 70 52 50 33 50 33 33 33 16 33 33 33 33 67 67 50 67 67 67 67 67 50 67 100 100 100 100 100 100 67 Objective test score Writing test score 61.8600 50.4600 66.0400 77.4400 80.4600 63.7200 59.0600 40.4600 61.6200 72.7900 41.8600 44.8800 69.7600 66.9700 70.9300 71.3900 73.0200 80.2300 68.1300 66.0400 88.3700 51.8600 51.8600 64.6500 63.7200 56.0400 57.9000 63.4800 GPA converted to percentages GPA for Course GPA for Course AACPS 51.2700 52.5500 64.6500 63.6000 73.2500 65.1100 54.3000 66.2700 65.8100 65.6900 40.9300 43.3700 59.8800 58.1300 59.8800 63.4800 58.9500 61.0400 54.4100 61.6200 71.5100 50.8100 49.1800 43.9500 '51.1600 41.2700 54.3000 58.3700 65.8100 46.5100 66.7400 87.6700 82.0900 63.2500 62.0900 62.7900 95.8100 86.0400 43-0200 23.2500 86.0400 69.0600 81.8600 84.8800 89.5300 84.8800 88.3700 69.7600 96.5100 43.0200 54.6500 77.9000 67.4400 63.0200 65.1100 93.0200 Type I converted to percentages Type I I converted to percentages 10.00 13.30 6.00 3.00 6.00 9.00 12.00 6.75 12.00 9.00 9.00 12.00 9.75 9.75 10.00 11.00 9.75 5.33 10.00 6.60 10.00 14.00 16.50 10.00 12.00 8.00 11.00 13.50 94 Data for University Students (for 27 of the 55 subjects) 3. 3 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 48 45 46 49 48 39 51 36 64 39 45 33 53 53 32 82 79 65 77 75 76 72 46 71 70 71 49 83 100 81 81 100 50 56 33 100 50 81 67 67 67 0 67 67 33 67 67 67 67 0 67 50 67 0 61.8605 83.9535 56.0465 59.5349 44.8837 59.7674 56.9767 47.2093 66.2791 73.0233 68.6046 71.1628 62.5581 48.8372 79.5349 82.3256 65.5814 31.1628 78.8372 54.1860 72.0930 64.1860 66.2791 42.0930 62.3256 50.6977 50.0000 53.7209 60.4651 51.7442 52.3256 40.4651 52.9070 53.4884 45.8140 57.9070 71.2791 60.1163 67.4419 56.9767 49.6512 53.4800 68.1395 56.7442 30.2326 46.5100 48.2558 60.9302 70.0000 71.2791 65.0000 63.6046 48.8372 44.4186 66.2791 92.7907 58.1395 66.7442 38.3721 65.1163 50.0000 43.0233 65.1163 67.4419 66.2791 75.3488 66.2791 38.3721 82.3256 86.9767 77.4419 39.5349 83.4884 50.2326 81.3953 39.5349 69.7674 0.0000 46.5116 23.2558 54.4186 10.00 15.00 12.00 11.00 13.50 10.00 12.00 10.00 8.00 9.00 9.00 9.00 8.00 12-00 8.00 8.25 13.50 10.50 15.00 15.00 15.00 9.00 6.00 15.00 6.00 12.00 6.00 Column 1 Objective test score Column 2 Writing test score Column 3 GPA converted to percentages Column 4 GPA for Course Type I converted to percentages Column 5 GPA for Course Type II converted to percentages Column 6 AACPS ","attrs":{"lang":"en","ns":"http:\/\/www.w3.org\/2009\/08\/skos-reference\/skos.html#note","classmap":"oc:AnnotationContainer"},"iri":"http:\/\/www.w3.org\/2009\/08\/skos-reference\/skos.html#note","explain":"Simple Knowledge Organisation System; Notes are used to provide information relating to SKOS concepts. There is no restriction on the nature of this information, e.g., it could be plain text, hypertext, or an image; it could be a definition, information about the scope of a concept, editorial information, or any other type of information."}],"Genre":[{"label":"Genre","value":"Thesis\/Dissertation","attrs":{"lang":"en","ns":"http:\/\/www.europeana.eu\/schemas\/edm\/hasType","classmap":"dpla:SourceResource","property":"edm:hasType"},"iri":"http:\/\/www.europeana.eu\/schemas\/edm\/hasType","explain":"A Europeana Data Model Property; This property relates a resource with the concepts it belongs to in a suitable type system such as MIME or any thesaurus that captures categories of objects in a given field. It does NOT capture aboutness"}],"GraduationDate":[{"label":"Graduation Date","value":"1994-05","attrs":{"lang":"en","ns":"http:\/\/vivoweb.org\/ontology\/core#dateIssued","classmap":"vivo:DateTimeValue","property":"vivo:dateIssued"},"iri":"http:\/\/vivoweb.org\/ontology\/core#dateIssued","explain":"VIVO-ISF Ontology V1.6 Property; Date Optional Time Value, DateTime+Timezone Preferred "}],"IsShownAt":[{"label":"DOI","value":"10.14288\/1.0078128","attrs":{"lang":"en","ns":"http:\/\/www.europeana.eu\/schemas\/edm\/isShownAt","classmap":"edm:WebResource","property":"edm:isShownAt"},"iri":"http:\/\/www.europeana.eu\/schemas\/edm\/isShownAt","explain":"A Europeana Data Model Property; An unambiguous URL reference to the digital object on the provider\u2019s website in its full information context."}],"Language":[{"label":"Language","value":"eng","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/language","classmap":"dpla:SourceResource","property":"dcterms:language"},"iri":"http:\/\/purl.org\/dc\/terms\/language","explain":"A Dublin Core Terms Property; A language of the resource.; Recommended best practice is to use a controlled vocabulary such as RFC 4646 [RFC4646]."}],"Program":[{"label":"Program (Theses)","value":"Language and Literacy Education","attrs":{"lang":"en","ns":"https:\/\/open.library.ubc.ca\/terms#degreeDiscipline","classmap":"oc:ThesisDescription","property":"oc:degreeDiscipline"},"iri":"https:\/\/open.library.ubc.ca\/terms#degreeDiscipline","explain":"UBC Open Collections Metadata Components; Local Field; Indicates the program for which the degree was granted."}],"Provider":[{"label":"Provider","value":"Vancouver : University of British Columbia Library","attrs":{"lang":"en","ns":"http:\/\/www.europeana.eu\/schemas\/edm\/provider","classmap":"ore:Aggregation","property":"edm:provider"},"iri":"http:\/\/www.europeana.eu\/schemas\/edm\/provider","explain":"A Europeana Data Model Property; The name or identifier of the organization who delivers data directly to an aggregation service (e.g. Europeana)"}],"Publisher":[{"label":"Publisher","value":"University of British Columbia","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/publisher","classmap":"dpla:SourceResource","property":"dcterms:publisher"},"iri":"http:\/\/purl.org\/dc\/terms\/publisher","explain":"A Dublin Core Terms Property; An entity responsible for making the resource available.; Examples of a Publisher include a person, an organization, or a service."}],"Rights":[{"label":"Rights","value":"For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https:\/\/open.library.ubc.ca\/terms_of_use.","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/rights","classmap":"edm:WebResource","property":"dcterms:rights"},"iri":"http:\/\/purl.org\/dc\/terms\/rights","explain":"A Dublin Core Terms Property; Information about rights held in and over the resource.; Typically, rights information includes a statement about various property rights associated with the resource, including intellectual property rights."}],"ScholarlyLevel":[{"label":"Scholarly Level","value":"Graduate","attrs":{"lang":"en","ns":"https:\/\/open.library.ubc.ca\/terms#scholarLevel","classmap":"oc:PublicationDescription","property":"oc:scholarLevel"},"iri":"https:\/\/open.library.ubc.ca\/terms#scholarLevel","explain":"UBC Open Collections Metadata Components; Local Field; Identifies the scholarly level of the author(s)\/creator(s)."}],"Title":[{"label":"Title ","value":"Direct and indirect placement test scores as measures of language proficiency and predictors of academic success for ESL students","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/title","classmap":"dpla:SourceResource","property":"dcterms:title"},"iri":"http:\/\/purl.org\/dc\/terms\/title","explain":"A Dublin Core Terms Property; The name given to the resource."}],"Type":[{"label":"Type","value":"Text","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/type","classmap":"dpla:SourceResource","property":"dcterms:type"},"iri":"http:\/\/purl.org\/dc\/terms\/type","explain":"A Dublin Core Terms Property; The nature or genre of the resource.; Recommended best practice is to use a controlled vocabulary such as the DCMI Type Vocabulary [DCMITYPE]. To describe the file format, physical medium, or dimensions of the resource, use the Format element."}],"URI":[{"label":"URI","value":"http:\/\/hdl.handle.net\/2429\/4886","attrs":{"lang":"en","ns":"https:\/\/open.library.ubc.ca\/terms#identifierURI","classmap":"oc:PublicationDescription","property":"oc:identifierURI"},"iri":"https:\/\/open.library.ubc.ca\/terms#identifierURI","explain":"UBC Open Collections Metadata Components; Local Field; Indicates the handle for item record."}],"SortDate":[{"label":"Sort Date","value":"1993-12-31 AD","attrs":{"lang":"en","ns":"http:\/\/purl.org\/dc\/terms\/date","classmap":"oc:InternalResource","property":"dcterms:date"},"iri":"http:\/\/purl.org\/dc\/terms\/date","explain":"A Dublin Core Elements Property; A point or period of time associated with an event in the lifecycle of the resource.; Date may be used to express temporal information at any level of granularity. Recommended best practice is to use an encoding scheme, such as the W3CDTF profile of ISO 8601 [W3CDTF].; A point or period of time associated with an event in the lifecycle of the resource.; Date may be used to express temporal information at any level of granularity. Recommended best practice is to use an encoding scheme, such as the W3CDTF profile of ISO 8601 [W3CDTF]."}]}