Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

’Difficulties’ of integrative evaluation practices : instances of language and context as/in contested… Low, Marylin Grace 1999

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-ubc_1999-463761.pdf [ 12.75MB ]
JSON: 831-1.0078154.json
JSON-LD: 831-1.0078154-ld.json
RDF/XML (Pretty): 831-1.0078154-rdf.xml
RDF/JSON: 831-1.0078154-rdf.json
Turtle: 831-1.0078154-turtle.txt
N-Triples: 831-1.0078154-rdf-ntriples.txt
Original Record: 831-1.0078154-source.json
Full Text

Full Text

'DIFFICULTIES' OF INTEGRATIVE EVALUATION PRACTICES: INSTANCES OF LANGUAGE AND CONTENT AS/IN CONTESTED SPACE(S) by Marylin Grace Low B.Ed., University of British Columbia, 1974 Diploma in Education, University of British Columbia, 1987 M.A., University of British Columbia, 1989 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE STUDIES Language Education We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA © Marylin Grace Low , 1999 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department The University of British Columbia Vancouver, Canada DE-6 (2/88) Abstract Although language is a medium of learning, most educational institutions typically teach and therefore evaluate language separately from content. In second language contexts, recent attention has been given to language/content integration through content-based language instruction. Yet, questions of integrative evaluation (evaluating language and content as one) remain uncertain and difficult. This inquiry explores difficulties invoked when teachers engage in practices of integrative evaluation of English language learners' writing at an international college for Japanese nationals in Canada. Are these difficulties technical problems? Technical rationality has been critiqued by a number of thinkers. Those interested in action research practices, contrast technical rationality with what they call reflective rationality and argue for contextualizing, rather than simplifying, difficult situations. Some with hermeneutic interests argue for an attunement to, rather than concealment of, difficulties of life in the classroom. Others interested in writing instruction, are critical of conventional approaches to writing pedagogy as reductionistic and deterministic. There are a number of instances of difficulty in teachers' integrative evaluation practices. Prior to agreeing on a prompt, many teachers explore texts as interpretive, social literacy but, in their uncertainty of how to mark such a text, they return to a question for which there is a 'correct' and 'controlled' response. Once the prompt and evaluative criteria are established, discordant orientations to evaluation, literacy, and language/content integration complicate teachers' uncertainty. For example, teachers sometimes acknowledge functional views of language/content integration, yet they are vague and uncertain about how to mark in an integrated way. When teachers read texts prior to judgment, they comment that the texts are difficult to interpret and then impose their own 'straightforward' readings on the texts to reduce and simplify the difficulties. These instances raise serious concerns in practices of evaluation, literacy and language/content integration, especially when technical forms of evaluation are paradoxically aligned with social and integrated texts. A turn to hermeneutics troubles a technical hold and invites further inquiry into tensioned moments of integrative evaluation as difficult, living practices. ii TABLE OF CONTENTS Abstract ii Table of Contents iii List of Tables vi List of Figures vii Acknowledgements viii Dedication ix Chapter One Beginning in the Middle of Things: Integrative Evaluation as/in "Pedagon" 1 Locating 'Difficulty' 3 A Legacy of the Modern Imaginary 5 An Instance of Integrative Evaluation 7 Evaluation as a Curricular Practice 12 Questions of the Inquiry 13 Mapping Troubled Terrains 15 Chapter Two Contested Spaces as/in "Stuck Places" 17 Positionalities and Meta-texts 18 Contested Spaces of Living Texts 20 Orientations to Student Evaluation 24 Conceptions of Literacy for Texts in School 27 Dominant Views of Language 32 Restoring Meta-texts to Aporetic Sites 35 Chapter Three Tracing Orientations: A (Re)View of the Literature 36 On Reading Pedagogical Author(itie)s of Language and Content 37 Common Ground of Language and Content 39 A Community of Difference 44 Fissured Texts Embedded in Integrated Practices 48 Preconditions of Decision-making 49 A Legacy of Assessment and Evaluation 49 Language and Content Classroom Assessment 57 Presumptions of Writing and Assessment Within ESP/EAP 62 Disruptions Within Language and Content - Its Language and Its Legacy 71 iii Chapter F o u r Discourses o f Di f f icu l ty : Methods o f Inquiry 74 Contextual Framings 75 The Site 75 The Participants 79 The Act iv i ty of Integrative Evaluation as a Socia l Practice 80 Qualitative Inquiry and Ontological Assumptions 83 Forms o f Discourse i n A c t i v i t y 86 Discourse as a Soc ia l Semiotic Text 89 Under taking this Inquiry 92 Col lec t ing the Data 93 Dea l ing wi th Discourse(s) as Data 94 Legi t imat ing this Inquiry 97 Researcher Positionalities 97 Its Trustworthiness 99 Its Obl igat ion 101 Chapter F i v e Troubled Terrain: Interpreting Instances o f Integrative Evaluat ion 104 Another Instance 105 M a p p i n g Troubled Terrains 107 Site One: Creating Prompts for Language/Content Texts 111 Site T w o : (Re)turning to Cri teria 117 Site Three: Diff icul t ies o f Judging Language/Content Texts 125 (In)commensurate Discourses o f Language/Content Relat ions 135 A (Re)Turn to the Questions 141 Chapter S i x G u i l t y Judgments and R i s k y Practices: A discussion 144 " U n d e r l i n i n g the d i f f icu l ty" o f a " w h o l l y human endeavour" 145 H o w I N o w Understand Integrative Evaluat ion to B e 147 Decentering the Technical 150 F r o m Certainty to Gu i l ty Judgments 154 F r o m Ethics to Obl igat ion as/in R i s k y Practices 157 Effects of Positionalities 161 O n Integrative Evaluation Practices 161 O n Teaching and Inquiry Practices 163 • M i d d l e W a y : " l i v i n g wi th them w e l l " 168 E n d i n g Right in the M i d d l e o f Things 171 iv Postscript 173 Bib l iography '" 179 Append ix 1 Educational Mandate 196 Append ix 2 Academic Standards Committee Plan 200 A p p e n d i x 3 Handout for Firs t Debr ief ing Session 203 Append ix 4 Handout for Second Debrief ing Session 208 A p p e n d i x 5 Funct ional Analys i s o f Sample E L L Composi t ions 215 v L I S T O F T A B L E S Table 1 A Synopsis o f Prevai l ing Evaluat ion Orientations 26 Table 2 A Summary o f T w o Conceptions o f Li teracy 28 Table 3 Ac t iv i ty as a Semiotic of Theory/practice 85 Table 4 M a i n Types o f Discourse Data 86 Table 5 Discourse Ana lys i s : Quali tat ive Methods and S F L Approaches 91 Table 6 Teachers ' Independent Judgments on Sample E L L text #1 137 Table 7 Teachers ' Independent Judgments on Sample E L L text #2 137 Table 8 Sample E L L text #1 and Discourse Layout o f Cause and Effect 139 Table 9 Sample E L L text #2 and Discourse Layout of Cause and Effect 140 Table 10 Highl ights of Interpreted Integrative Evaluat ion Practices 147 vi LIST OF FIGURES Figure 1 The Activity of Integrative Evaluation 82 Figure 2 Partial Contextualization of Integrative Evaluation 82 Figure 3 The Complexity of Activities Related to Integrative Evaluation 83 Figure 4 Activity: Motility Between Sociocultural and Linguistic Knowledge 92 Figure 5 (Re)Marking ELL Writing in Post-Secondary Studies 109 Figure 6 Visual Mapping - a Specific Case of Structuring a Question 116 Figure 7 Visual Mapping - a Specific Case of Deciding on Evaluative Criteria 124 Figure 8 Visual Mapping - a Specific Case of Marking with Indelible Ink 135 vii A C K N O W L E D G E M E N T S M a n y have supported and sustained me i n significant ways throughout the re/searching and wri t ing o f this project. A n d , although I w i l l i ng ly admit that there are many persons whose contributions to this particular work may have been unduly overlooked, I honour the fo l lowing in confidence. I am deeply indebted to those who constituted m y supervisory committee: to m y advisor and research supervisor, Dr . Bernard M o h a n , for his on-going thoughtful supervision and guidance and for his insight into notions o f discourse; to D r . Margaret Ea r ly , for her wil l ingness to inquire with me into spaces o f tension and compassion; to D r . T e d A o k i whose gentle w i s d o m ceaselessly teaches me about the metonymy of language(ing) and its effects. S imi la r ly , the teachers who welcomed me into their practices o f integrative evaluation and wi th w h o m I engaged in many conversations were fundamental to m y commitment to the complet ion o f this project. Debts are also owed to m y friends, Ka ren W i l s o n and X i a o p i n g L i a n g , whose careful readings of this text contributed to it i n important ways. I also acknowledge and thank the members o f a special study-group and others I have studied wi th , who, over the years, engaged in important conversations wi th me about m y work. M y warmest gratitude and appreciation go to Margaret Froese, M y a M c K a y , Pat Palul is , and Aristede Gazetas wi th w h o m I have shared (too) many moments o f m y experiences wi th doubt, uncertainty, learning and inspiration. Thei r gifts o f support and encouragement enriched m y tensioned spaces o f being a teacher, writer and friend. I a m most thankful to my family: to m y parents, whose quiet support has always been heard; to m y husband D o u g , and our sons Dana, R o b i n , and C o d y without whose love, care and sustenance I wou ld not have journeyed this far. I gratefully acknowledge the support of this study by The Soc ia l Sciences and Humani t ies Research C o u n c i l o f Canada in the form of a Doctora l Fe l lowsh ip . vi i i DEDICATION to those who find themselves in aporetic sites of difficulty and uncertainty and stay at the site of struggle . . . Chapter One Beginning in the Middle of Things: Integrative Evaluation as/in "Pedagon" The notion o f hermeneutics with wh ich I w i sh to begin: hermeneutics as an attempt to stick wi th the original difficulty o f life, and not to betray it wi th metaphysics. John Caputo, Radical Hermeneutics Pedagon . . . attempts to articulate the cu l tu ra l space i n w h i c h issues sur rounding pedagogy are contested, enacted and inhabi ted . I n v o k i n g "cul ture" here gives notice o f m y conv ic t ion that the typ ica l parameters o f conversation regarding educational issues, particularly i n the Nor th A m e r i c a n context, are far too narrow, largely because o f the increasing specialisation o f professional languages consequent to the spread o f indus t r i a l i sm . . . . A s adults, we inevi tably suffer the cul tural diseases o f our t ime, but then we reproduce them i n our children to the degree we have not healed ourselves. D a v i d Smi th , Pedagon I f life dwells i n an original difficulty, an original ambiguity that cannot be mastered but only l ived with we l l , the pursuit of such mastery can only lead to immobi l i t y or exhaust ion-i t does not lead to understanding human l i fe-as- l ived i n a deep way. L i f e as something to be mastered seems to deny what we already know about being alive . . . . It returns inquiry in education to the original , serious, and difficult interpretive play in which we l ive our lives together. D a v i d Jardine, Reflections on Education, Hermeneutics and Ambiguity Those who involve themselves in holistic focal practices understand that one's evo lv ing sense o f identity and one's daily practices must always be, i n some way, interpreted in relation to one another . . . . [They are] l ived experiences that permit an openness to the complexi ty o f the relations among things and people. Terrance Carson & Dennis Sumara, Action Research as a Living Practice 1 Al though language is a medium of learning, most educational institutions typical ly teach and therefore evaluate language separately from content. In second language contexts, recent attention has been given to language/content integration through content-based instruction (Stoller, 1999). Yet , questions of integrative evaluation (evaluating language and content as one) remain uncertain and difficult (Norton & Starfield, 1997). A r e these difficulties technical problems? Technical rationality has been crit iqued by a number o f thinkers. Lyota rd (1997) cogently criticises a l ingering Western legacy, a technical rationality, and its effects on society. It has also been crit iqued by Tay lo r (1996) as a "malaise o f modernity"; a rationality i n w h i c h individuals and groups w o r k to gain control , mastery, unity and simplification of their l ives in what is otherwise understood to be a complex and difficult wor ld . Usher and Edwards (1994) explore the implicat ions o f postmodern thinking that challenge prevail ing structures and hierarchies o f educational practices they c l a i m are imposed by a technical rationality. A s part o f the discussion, Altr ichter , Posch, and Somekh (1993), interested in action research practices, contrast technical rationality wi th what they cal l reflective rationality. They argue for: 1) contextualizing, rather than s impl i fying, difficult situations; 2) "dynamic networks o f relationships to assist [] i n taking responsible action i n the face o f complexi ty and uncertainty" (p. 202); 3) the need for "symmetry rather than a hierarchy o f power relations" (p. 203). A s w e l l , Jardine (1998) and Smith (1999), both wi th hermeneutic interests, argue for an attunement to, rather than concealment of, the difficulties i n pedagogic life. The debate continues wi th in second language education, where Z a m e l (1997) and Spack (1997), interested in wri t ing instruction, are cri t ical o f conventional approaches to wr i t ing pedagogy as reductionistic and deterministic. Re- thinking instances o f integrative evaluation practices, i n light o f current critiques o f technical rationality, raises serious concerns o f how difficult practices o f evaluation, literacy, and language/content integration can be understood. Placed right in the middle o f difficulties, this text, as an inquiry into teachers' integrative evaluation practices o f compositions written by Japanese nationals studying i n Canada, turns to hermeneutics and opens a discussion o f interpreting difficulties differently. 2 Loca t ing ' D i f f i c u l t y ' N a m i n g integrative evaluation is an invitation to the reader to consider Smi th ' s (1994, 1999) "cultural space i n wh ich issues surrounding pedagogy are contested, enacted and inhabited" (p. i i ) . F o r those o f us who inhabit the practices o f evaluating E n g l i s h language learners' ( E L L ) ' inscriptions, few wou ld question the agony o f having to leave that presumed necessary but possibly i l lusory mark on the page. It is a pedagogical practice i n w h i c h significant decisions are enacted at seemingly declarative and definitive endpoints. 2 A s significant as the 'mark ' may be, it is the complexif icat ion o f more difficult and spirited presuppositions that lead to the decision o f the mark, the struggle o f coming- to-know 'the' mark and its ( im)possibi l i ty - its "or ig ina l d i f f icul ty" (Caputo, 1987, p. 1) - that is central to this dissertation. 3 Chapter one locates those presuppositions under lying the process o f evaluative decisions as difficulties i n l iv ing practices. In entering such a space influenced by complex historical and current conditions, I recognise that conversations have already begun and w i l l continue long after situated decisions are made. A n d so, i n entering this location, I begin i n the middle o f things. A s an inquiry into the what (as) and where (in) o f integrative evaluation, I have scripted i n the title o f the dissertation and chapter one 'as / in ' . It is a stance that helped me come-to-know integrative evaluation beyond definition. Th ink ing in terms o f as/in invited me to d w e l l in the solidus and consider pedagon as a way o f wri t ing what integrative evaluation becomes through its location in contested sites. Integrative evaluation as 1 I use E L L (English language learner) intentionally to resist use of the common label ESL (English as a Second Language) - a label that has the potential to connotate second as inferior, substandard, not first, supplementary, subsidiary, subordinate. I also believe that the quantification of second languages and cultures (as separate from first), detracts from the richness of language and cultural complexities in which all language learners dwell. 2 In seeking enacted ways, I refer to a notion of enactivism "founded on a manner of thinking that seeks out 'middle ways' amid disparate perspectives. But such middle ways should not be thought of as compromises. Rather, they represent attempts to sidestep seemingly irresolvable tensions by drawing • attention to and offering alternatives for the assumptions that underlie varied opinions" (Davis, 1996, p. xxv). 3 1 was introduced to the term 'complexification' in Davis's (1996) book, Teaching mathematics and support its use in a "general movement across a range of academic disciplines - away from attempts to impose linear and causal models onto phenomena and toward embracing the difficulty and ambiguity of existence." As Davis explains, "it is a recent addition to the English language . . . . Its creation represents a deliberate attempt to affect the way we stand in the world. It is by making up new words that we interrupt the commonsense notions that frame our actions. We enable ourselves to act differently" (1996, p. xvii, original italics). It is with that purpose in mind that several "recent additions to the English language" are interwoven throughout this text. 3 pedagon conjoins "the formal practices and professions o f teaching" w i t h the "struggle or contestation over something o f great importance" (Smith, 1999, p. i ) ; integrative evaluation in pedagon locates itself midst "stuck places", spaces-in-between mult iple texts where traces o f positionalities continually col l ide, egress and generate new life (Lather, 1998); it is a place "that cannot be mastered but only l i ved wi th w e l l " (Jardine, 1992, p. 117). In seeking to locate and name traces o f the struggle, questions o f integrative evaluation became questions o f relations between multiple texts. 4 A s a reader o f Carson and Sumara 's (1997) book, A c t i o n research as a l i v i n g practice. I am reminded that "one's dai ly practices must always be, i n some way, interpreted i n relation to one another" (p. xv) . T h e textured sites, established through interwoven meanings and experiences i n discursive practices o f integrative evaluation, are understood here i n terms of their social dialogue, a v i ew o f meaning I have encountered i n the readings o f Bakh t in (1935/1981) and his use o f the terms heteroglossia, chronotope and carnival, o f Kr is teva (1980) and the pr inciple o f intertextuality, o f B i l l i g , Condor , Edwards, Gane, Midd le ton and Radley (1988) and their notions o f ideological dilemmas and dilemmatic ideologies, and o f Harre, Ha l l i day , Bernstein, Vygo t sky , Foucault , and Bourdieu , each o f whose works are scripted differently but, I believe, are based on similar principles. It is not my intent to review their work here, only to c l a i m that such social theories o f discourse have been legitimated by many scholars from diverse disciplines. L e m k e (1995), a scholar o f poli t ical social semiotics and interested i n the textual poli t ics o f education, refers to intertextuality and comments on how the meaning o f each particular utterance or stretch o f discourse is seen as arising in the relations between texts and social viewpoints and not in relations among l inguis t ic forms as such or among speakers as individuals , (p. 23) T a k i n g this stance, discourse is not an autonomous product o f the ind iv idua l but "functions as part o f a social dialogue (whether the other participants i n this dialogue are considered to be actually present or are only impl ied)" (ibid.). Intertextuality, argued as an 4 The (s) of discourse(s) is used as a way of acknowledging the ever present possibility, and moreover probability, of multiple and incomplete discourses operating within one assumed discourse. If discourses are socio-culturally grounded, as I believe they are, differing socio-cultural activities construct differing discourses. As daily participants in multiple social-cultural activities, we have multiple discourses with which to construct our world. The (s) reminds me that we cannot assume only one discourse is constructing meaning at one time. I believe that multiple discourses are at work establishing networks that have the potential to connect, disconnect, and reconnect at any time. 4 inherent aspect o f discourse, is v iewed as social i n or ig in and always dia logical i n purpose. Intertextual spaces locate sites o f multiple texts, places where one text is read against the background o f other texts expl ici t ly or impl ic i t ly present i n the first, o f texts made i n various times and places that are both historical and current. The location o f educative l iv ing practices claims a generative space (Jardine, 1992). A s a generative space, an inquiry into texts o f integrative evaluation attempts to bring forth the pre-conditions, those taken-for-granted assumptions i n wh ich we, as practitioners o f integrative evaluation, l ive . The task o f the inquiry then is to "re-collect the contours and textures o f life we are already l i v i n g " without "render[ing] such a life our object" (Jardine, p. 116). Resist ing an inquiry into life as factual and objective, this project places itself in the midst o f life and "its or iginal dif f icul ty" , keeping "a watchful eye for the ruptures and the breaks and the irregularities" (Caputo, 1987, p. 1). Jardine (1992), heeding Caputo ' s message, argues, "technical-scientific discourse offers itself up as a remedy to the difficult ies o f l i f e " and then reminds us, rather than s i m p l y be ing a remedy to l i f e ' s d i f f i cu l t i e s , [technical-scientific discourse] has rather come to recast the nature o f l i fe ' s difficulties into precisely the sort o f thing for w h i c h a t echnica l so lu t ion is appropriate; that is , l i f e ' s d i f f icul t ies are technica l problems requi r ing a " t echn ica l f i x . " (p. 117) Resist ing the ca l l o f scientific discourse to cast integrative evaluation as a technical task requiring technical tools, difficulties within integrative evaluative practices can be acknowledged as ambiguous and uncertain. Difficult ies emerge when technical words cannot disinherit or terminate experiences o f a social wor ld . Instead, they are re-instated to tensioned life where traces o f technology and other discourses are contested wi th in integrative evaluation. A Legacy o f the M o d e r n Imaginary Trac ing back to at least the seventeenth century, the predominate epistemological stance o f the "modern" or " technica l" epoch i n education can be l i nked to Descartes. Dav i s (1996) recalls that Descartes made 5 t w o breaks f r o m earl ier perspect ives o f k n o w l e d g e and modes o f i n q u i r y -perspectives w h i c h he rejected as inconsistent and unrel iable mixtures o f fact and fancy . . . . Descartes denounced tradit ion, hearsay, m y s t i c i s m and re l i g ion as he ca l led for the pre-eminence of the "natural l ight" o f (mathematical) reason, (p. 3) Descartes argued for methods that derived irrefutable knowledge c la ims or the ' truth' o f existence (the external universe), laying foundational paths to what has come to be k n o w n as modern science. H i s use o f geometry to deduce the nature o f the wor ld and his distrust o f the senses as a mode o f inquiry led h i m to a pr iv i leging o f rational thought, o f epistemology over ontological concerns, c la iming it a model o f reason that was independent o f the mercurial and untrustworthy senses. H i s we l l k n o w n cogito - / think - became an a x i o m o f indubitable truth, a basis on wh ich he c la imed he could authenticate or deny the 'truth' o f a l l other knowledge. B u i l d i n g his model o f rationality on geometric reason and the cogito - / think -, Descartes established a form o f dualistic thought now infused i n modernist stances o f how many understand the wor ld . Perhaps two of his most noted dichotomous contributions now contested i n many discipl ines are "se l f and other", and, " m i n d and body" . The modern ideal ' s e l f was c la imed to be an autonomous, independent, centred subject, the reference point for meaning. T o situate ' s e l f as the centred subject that was part of, but distinct and independent i n , an objective wor ld , "Descartes also distinguished the background o f the ' no t - se l f w h i c h was col lected under such names as 'other' and ' w o r l d ' . " (Dav i s , 1996, p. 5). 'Other ' and ' w o r l d ' became objects o f truth that cou ld become k n o w n to man through rational thought. In advancing the mind-body duality, Descartes pr ivi leged the m i n d and cla imed that (rational) thinking was the basis for a l l truth. The Cartesian binary was established i n persuasive scientific terms, separating mind from body, as i f they were capable o f existing independently. W i t h the Cartesian orientation came a preoccupation for methods o f inquiry that w o u l d contribute to structuring a more accurate representation o f the wor ld (reality); methods that, in a desire to become more accurate, became more technical in their mechanistic abili ty to reduce, control and master objects (the 'other') o f inquiry . These methods have continued to be a part o f a dominate technical imaginary, perhaps important 6 necessary i l lus ions 5 for the concerns o f the seventeenth century, but ones that may no longer address the complexifications of relations acknowledged in current critiques o f technical rationality. Undoubtedly, Descartes has made significant phi losophical contributions to how many make sense o f the wor ld . H e has given structure, albeit in reductionist terms, to an otherwise difficult, complex and uncertain wor ld . W e are reminded regularly by headlines i n the media o f serious predicaments we face, from the familiar to the global . Have we come to realise the impossibi l i ty o f an autonomous self in an objective wor ld , a necessary i l lus ion that no longer offers refuge from complexifications? It is not m y purpose to reject the Cartesian legacy, only to question its dominance. In asking how the Cartesian orientation and the assumptions it imposes implicate how one lives i n this wor ld gives insight to the consequences o f such a stance, especially for ideations o f both knowledge and education. In later chapters, I further explore the imposit ion o f modernist philosophies wi th in the complexi ty and difficulty o f integrative evaluation. A n Instance o f Integrative Evaluat ion Imagine for a moment peering into a post-secondary c lassroom for Japanese nationals presumably wri t ing i n Engl i sh , for them an other language. 6 The casual observer w o u l d sense the tense determination o f students taking a final exam. K n o w i n g their responses to questions w i l l be evaluated, they struggle to understand expectations o f the question and to be heard. 7 It is a common landscape o f schooling, the mark ing o f written texts, yet i n this seemingly regular and unblurred process, ruptures, breaks and irregularities surface i n the discourses o f teachers judging that wri t ing. 5 Gazettas (1997), borrowing the term necessary illusions from Bazin, constitutes it as a "co-presence" of realism and illusion (image) that "record and represent actual events [to] produce an ambiguous commentary about that world out there, forcing the viewer to realise that illusions and artworks are part of our reality" (p. 217). In turn, I borrow necessary illusions from Gazettas and re-constitute it as imaginary, those positionalities that become identified by participants as "how things are is how they ought to be" (Nichols, 1981, p. 1). 6 An other is written as two words to play on the effects of language in a process of objectifying and nominalising Others that un/intentionally labels, defines and confines the subject. 7 Reference here is to the multiple voices of self (learner), as an attempt to understand which voices get heard and why others are silent or silenced. The struggle is partially to determine which voice(s) the teacher will hear and how they will be valued and to understand how the silent, the unsaid of the said, can be heard. 7 Consider a l i v i n g practice of two teachers marking: After investigating aspects of climate change in class, and as part of the evaluation process, English language learners in a content class were given ten different test questions to respond to within a period of sixty minutes. It was a term-end exam. The following text is one student's response to question number eight: 8. You have been learning about climatic change. Write a well organised paragraph about one of the following topics. A. Explain one cause of climatic change. OR B. Explain how climatic change causes one human or planetary health problem. Global warming is one cause of climatic change. Global warming is to rise temperature unnaturally in an earth. Carbon dioxide in the air increase, temperature is goes up. Therefore, we should think the way of using Carbon dioxide. Carbon dioxide mainly comes from cars and industries. If we leave producing carbon dioxide and rising temperature, we could get ocean water evaporation, and the permanent ice melting. These cause big climatic change. Example are the dead of fish which can't be used to speedy of temperature rising in water and huge storm in which many species could get damages. Many circle of natural chain reaction could be destroyed. Now, we come to the period when we have to confront global warming. It can't stop without our doing anything to it. I am sure we all hope peaceful life and future, so let's think about it and do anything to get safety future. (Written by an E L L , 10/04/95) Teachers enacting integrative evaluation: Tl: I don't think we should give a separate mark for language and for content; I look for each but the mark I give is a whole mark that represents the course, yet in my mind I have separated the content from the language-it may not be right but it is easier to mark that way. T2:1 always mark language and content separately, equal marks for both, I know that sometimes I have trouble deciding which one to mark down when the ideas aren't clear. Still, I think we should give separate marks. Tl: Look at this response on climatic change. How would you mark this? It seems to have the ideas I was looking for but it is so difficult to read. I think 1 know what the student is trying to say. Oh, I don't know how to mark this-I don't know what is language specifically and what is content-I can't tell whether the student's language is interfering with his ideas or his ideas are interfering with his writing. This is not an isolated instance o f integrative evaluation nor is it an example o f the concept o f "integrative evaluation" that is analogous to another instance o f integrative evaluation, as that wou ld imply something pre-given, already defined. Instead, interpretive 8 work attempts to make what has been said about integrative evaluation i n the past readable again by p rovok ing "something already familiar" (Jardine, 1998, p. 40); the instance keeps integrative evaluation alive in interpretations that bu i ld on the story o f integrative evaluation, thus transforming how it can now be understood and making interpretations useful. Presuming to evaluate what students know from what they say is a regular routine o f the language and content classroom. M y experiences wi th the formal school ing o f E n g l i s h language learners have been bounded by what Dav i s (1996) calls the ocular: " o f seeing and observing, o f clarity and i l luminat ion, o f distinct boundaries and so l id objects" (p. xx i ) . Based on m y experiences, this form o f teaching and learning seems to have become disembodied, univocal and predominantly prescriptive. It is a tradition wherein a dominant v i ew seems to be one o f language as an unambiguous and static phenomenon, and where judg ing written tasks can be unproblematic. Yet , throughout the process o f judging E n g l i s h language learners' inscriptions seem to be discourses o f uncertainty, ambiguity and diff iculty that question these assumptions and their imposit ions on teachers' practices. Ev idenced i n the sample, discourse(s) were contested, pol ivocal , unshared, fissured texts o f language and content relations. One o f the ways I attempted to locate the multiple dimensions o f integrative evaluation was to journalise. In going beyond descriptive notetaking and thick description, I wrote f rom the decentered folds o f m y own and others' practices. 8 In so doing so, re-writ ings o f themes, meta-texts and "stuck places" emerged and seemed to have potentially useful interpretations. 9 One journal entry, mapped on page 10, locates two themes wi th in the activity o f integrative evaluation: meaning gaps and language and content relations. 8 In his publication, Teachers Narrating/Narratives Teaching: Pac Rim Experiences, Aoki (1992) invites readers to think of "narrating as an interplay of storying and theming" (p. 29) so the writer can live in their "writing (re-writing) to write a fuller story and to write more deepened thoughts on the themes selected ... theming is a form of experiential inquiry (some call it existential inquiry) that calls for reflective thoughtfulness. For such an inquiry, it may be helpful to be guided by questions that stir us into lived meanings of experiences." (p. 32). For me to journalise in this way was insightful. Within each of the main themes that emerged in this intertextual approach to journal writing, there were traces of possible conflicting sub-themes. 9 Lather (1998) speaks of the "praxis of stuck places" (p. 1) as aporetic moments of generativity wherein "we break with all technical thought, all method and think in wholly other ways". 9 J o u r n a l i s i n g L i v i n g P r a c t i c e s of Integrative E v a l u a t i o n (re-written as meta-texts and structuring aporetic sites as questions) meta-text: literacy concern: are meaning gaps faulty? (written May 24, 1995) Today I talked with a teacher as she marked test questions from a term-end exam. One of the texts she was marking was written by a Japanese national studying in Canada. His response to one question follows: One of climatic change in the world is global warming. Global warming is linked changing diseese 's pattern. My experience with diseese may be part of this and that concerns me for the future of world. It is difficult me to explain but 1 know this is big problem. (O.S.) She commented, "the student starts off well linking global warming and disease patterns, for me it suggests that he knows more than he's written ... in fact, he hasn't said much at all. I like the phrase 'diseese patterns' - he's remembered some of the words. But there are gaps in his answer." It's been my experience that teachers don't attend to the function of language and meaning making. It's as if language can unproblematically take meaning to its destination. Yet, teachers seem to get stuck making sense of the text when the meaning isn't clear. I had been in the classroom when the students wrote this exam and had spoken to as many as I could afterward, asking them how they felt responding to the questions. The student who wrote the above text explained, "I can't express what I know. I am frustrated with English. I search deep inside for my ideas and then I can't tell them because the words are lost." (re-written as theming and locating "stuck places") theming: meaning gaps In re-reading the first writing, I asked, "What was it like for the teacher to experience meaning gaps?" She was aware of the incompleteness of his writing. Did she see this as faulty? Her comment places import on the content or message. Where were the gaps located for the teacher? in content? in language? What is being evaluated? Can we ever know fully what the writer means? The student acknowledges gaps. Are the gaps understood in the same way by writer? by reader? His is a translation lost, a transformation begun, an incompleteness always. It is a place where writers of English, for them an additional language, dwell midst slippery signifiers. Meaning gaps are a site of struggle for both the writer and the reader. theming: language or content? The teacher seemed to focus on the message of the text. Only in search of something else to mark did she attend to elements of language use. Even then, it is content vocabulary that she acknowledges. It seemed as if language and content were autonomous. I remember thinking, how do you separate language from content? How did content become so important? As I think about language and content relations, I've come to know the import of language and some of the ways it works to constitute meaning. In re-reading my writing and theming, I asked, "within what meta-texts was the teacher's story located that made it possible for her to construct her comments in the way she did? She seems to be living within a meta-text that views the reading of meaning gaps as problematic, perhaps even faulty. For her, it seems the student was unable to 'express' or 'represent' his ideas to give a clear message - to tell what he knows. These traces of positions or stances lead to a meta-text of literacy that seems to structure a way of understanding writing where words are secured directly to a meaning, a place where the meaning of a text is autonomous, where meaning gaps are considered faulty. meta-text- language and content relations: how do you separate language from content? Within what meta-text was this theme written? Linguistic concerns of how language and content relate seem to predominate. The teacher's strong emphasis on content would suggest a view of language that carries meaning, a place where meaning is primary and language plays a secondary role. Interpreted in this way, a view of language could claim to be understood void of context, separate from meaning. Taking this stance the teacher would attempt to address content concerns separate from language concerns. 10 Evidence wi th in the sample discourse data on page eight seems to suggest that judg ing E L L composit ions is a necessary but often discomforting, difficult and complex task. The literature speaks to the complexity o f E L L academic wri t ing wi th a c o m m o n and frequently used phrase, language and content, often qualified by the term integrated.10 The academic c lassroom is presumed to be a location o f purposeful and meaningful learning about the wor ld where terms such as content, subject matter, and knowledge are used interchangeably to describe the focus o f learning. In such a context, one can hardly ignore the role o f language i n meaning making. W h e n an academic classroom includes E L L s , teachers are challenged to understand the effects o f a doubl ing o f languages and cultures wi th in acts o f communicat ion. Contained by the learners' resources o f the new language and o f the o ld , confined by the translatory strategies they use, and opened by interacting worldviews, they write their wor(l)ds and, in that wri t ing and the reading o f that wri t ing, traces o f uncertainty and frustration abound." Teachers' w e l l intentioned goals o f fair and consistent evaluation practices as situated judgments are disrupted by the subjectivity and uncertainty o f how to interpret the writer 's intentions wi th in his or her presumed l imi ted conventions o f language and then make sense o f and judge relations o f language and content that are constituted in the wri t ing. 1 2 W i t h i n the community o f relations involved in integrative evaluation, I attempted to interpret the ways i n wh ich evaluative acts both shape and are shaped by teachers, ind iv idual ly and col lect ively . 1 3 C l a iming that wri t ing constitutes meaning (content) through language i n a unified or integrated way contrasts wi th conventional approaches that separate the evaluation o f language and the evaluation o f content. G i v e n that the reader attends to text form (language) and function (content), the question arises o f how to integrate language and content relations. It appears that the separate evaluation o f language and the evaluation 1 0 For example see seminal works of Mohan (1986) and Brinton, Snow & Wesche (1989) and the use of the terms integrated, and language and content. 11 wor(l)ds constitutes the idea that "language and reality are dynamically [and artifactually] intertwined" (Freire, 1983, p. 5). 1 2 The term 'situated judgments' is used to remind the reader that judgments are situated in the interstices of historical, current and subjective conditions, calling into question the possibility of being a neutral and objective evaluation. 1 3 While I recognise the importance of the many intricately interwoven learner relations in integrative evaluation practices and raise questions of this in Chapter six, the focus of this inquiry is on the effects of positional relations of teachers. 11 o f content is not sufficient and, for me, raises questions regarding the assumptions underlying roles o f specialisation that seem to be inherent i n , for example, notions o f ' language' teacher and 'content' teacher. Rather than attempting to ' f i x ' a problem called integrative evaluation, I sought to situate its relations in ever-unfolding instances of individual and collaborative curricular actions by teachers, replete wi th emerging uncertainties and inconsistencies. Th is is supported by Tay lo r ' s (1996) ca l l to question a dominant Western Cartesian be l ie f that one can be i n control o f one's life ( l iv ing practices). 1 4 In part then, this project takes on a br ico l lag ic function - embracing tensions, difference and "stuck places", and opening up to generative possibil i t ies in the "already famil iar" - that calls for a reconceptualising o f the difficulty o f integrative evaluation. Evaluat ion as a Curricular practice Selecting what is appropriate and viable for students to learn and then be judged, are important curricular and evaluation decisions. In language and content curricular practices, topics and the language demands o f those topics are considered. Emphasis is g iven to the curricular 'needs' envisioned i n the new language and culture but complicated by the real wor ld experiences o f day-to-day schooling. It is a context wherein teaching and learning are complex, dynamic and ever-evolving activities. In some circles wi th in the f ield o f cu r r i cu lum studies, scholars such as Grumet (1988) take this stance o f curricular situations c l a iming , "cur r i cu lum is a mov ing fo rm" (p. 172). Yet , the powerful Cartesian influence o f both knowledge and education (including evaluative practices) continues to reduce this movement to a plan o f curr iculum-as-s tow (Jardine, 1998). 1 5 A v iew advanced by Ty le r (1949) saw curriculum-as-plan wi th goals and 1 4 See Taylor's (1996) The malaise of modernity in which he speaks of three malaises of modernity: individualism, instrumental reason and loss of freedom. He argues that individuals, in becoming entrapped in their own assumptions of self-control and self-determination, become inactive in the social discourse(s) of everyday life. Similarly, I believe, if evaluators become intently focussed on a presumed controlled and objective judgments, evaluation becomes an act of mastery that cannot be mastered. So positioned, evaluators often veil themselves from the social and subjective relations that are inherent and everpresent aspects of their task. 1 5 I borrow Jardine's (1998) use of kx\ow\zdgz-as-stasis as a way of calling into question "desires we may have, as educational theorists and practitioners, to get the curriculum 'right,' 'straightened out' once and for all, for such desires requires a basically disintegrative, analytic act aimed at rendering education a closed question, aimed at rendering human life lifelessly 'objective' under the glare of knowledge-as-.rfa.s(.s" (p. 73). 12 measurable objectives and learning outcomes that purposefully prepared the learner for adult l ife. L i f e - l i v i n g practices - was separate from the 'modern ' pedagogical staple o f schools. Cur r i cu lum, and therefore evaluation, was interpreted as a series o f static, w e l l -intentioned, calculated steps to assist the learner to become successful i n adult life. It was predicated on the assumption that it is possible for the contents o f a cu r r i cu lum to have a transcendent validity - one which , for a l l intents and purposes, is independent o f the era, the culture, the classroom, the teacher, and the learners. Such an assumption arises from the modern notions that the w o r l d is pregiven and objectively knowable and that established knowledge o f both the phys ica l and the social wor ld is essentially value free - conceptions that support cur r icu lum developers' goals to identify knowledge objectives that reflect that wor ld and to organise those objectives i n ways that are suited to the linear and tiered structure o f the schooling systems. (Davis , 1996, p. 86) Not iceab ly absent from these discussions has been the role o f the learner. Hence , there has recently been much debate about learner-centred curricular approaches versus knowledge-centred perspectives; debates that continue to be based on the premise that what learners need to k n o w i n adult life can be identified and taught (knowledge as object), learned and evaluated (Barr & Tagg, 1995). The Cartesian way infused i n conventional educational practices provides educators wi th a technical, prescriptive v iew to curriculum. In taking a linear path - f rom goals and objectives to testable learning outcomes - the learner's way is presumed to be clear and out o f danger. Ye t , evidenced i n the dialogue offered on page eight o f this chapter, the notion o f a l inear cu r r i cu lum seems to be displaced and "stuck places" emerge as unpredictable re-routings and re-turns. 1 6 F o r me, they are signs o f unplanned and non-technical curricular life that are a part o f schooling. Disrupt ions that M i l l e r (1997) claims "constitute the l i v e d practice o f our research" (p. 199) leave m y work open to unexpected discourses o f l i v i n g practices o f integrative evaluation. Rather than seeking a definitive, conclusive certainty o f a deductive argument, I explore instances that bu i ld on, and make useful, understandings o f integrative evaluation. Questions o f the Inquiry A l though it is a routine, unquestioned expectation that learners be judged on written responses to assignments and tests, the deliberation o f a grade or mark seemed to be a 1 61 would argue that learners as well experience "stuck places", especially in their engagement of tasks to be evaluated. 13 tensioned and troubled practice. Teachers, well-intentioned to engage in fair and consistent grading practices, contended with uncertainties and ambiguities, as evidenced on page eight, that seemed to disrupt their judgments. I began to structure those disruptions as questions from the stance o f integrative evaluation as a l i v ing practice; questions that are not i n search o f answers but invite interpretation in the unfolding o f complex relations and contrary themes i n the experiences o f evaluators. I structured ever evolving, interrelated questions central to this site o f struggle: 1) H o w do teachers, i n their practices, construct the activity o f integrative evaluation? m particular, do current critiques of a technical rationality contribute to useful interpretations of integrative evaluation practices? 2) H o w do teachers relate practices o f evaluation, literacy, and language and content integration? These questions offered promise o f useful interpretations o f instances that become readable again i n new ways. Teachers unsuspectingly inform themselves and each other wi th their insights and thoughtfulness as part o f a discursive process that has the potential to bu i ld , reconstruct, or alter the current context (Roman, 1992) and open up to an ever-evolving transformation o f integrative evaluation. M a p p i n g Troubled Terrains A textual form maps this text, structured by particular points o f interest that have emerged as troubled historical and current sites of integrative evaluation. The text is written wi th in a c o m m o n and expected mapping o f dissertations whi le knowing that unanticipated disjunctures may occur at interstices where readers are asked to become, in Ru th V i n z ' s (1997) words, "more nomadic than usual, searching for a trace o f meaning by m o v i n g through a sometimes ambiguous textual geography" (p. 245). Disrupt ions, located at different points o f the terrain for each reader, lay hidden, unmarked, i n the effects o f language that are both intentional and unintentional. They occur unexpectedly i n the reading o f the text, as they have done i n its wri t ing; the intention o f m y wri t ing is to offer pathways o f thoughtful interpretations and useful understandings that educe the reader to compose their o w n meanings. This inquiry, its manifestation i n this text, and its defence continue to be a dynamic, ever-changing and on-going landscape o f constituting and 14 reconstituting meaning for the participants, the writer and the readers. Chapter one invites the reader to consider difficulties o f integrative evaluation as l i v i n g practices. The pathways taken i n Chapter two are etched onto a troubled terrain o f language and content pedagogy and identify several key meta-texts that contribute to broader historical and current conditions. It traces imbricated positionalities rooted i n dilemmatic discourse(s) o f integrative evaluation, and relations of evaluation, literacy, and language and content integration. Chapter three takes the reader into the communit ies o f presumed authorities to listen to "where" they dwel l . I ask, what are the espoused c o m m o n grounds and grounds o f difference i n E L L wri t ing assessment wi th in language and content pedagogy and its history? Chapter four details a particular approach to mapping instances o f integrative evaluation through discourse(s). It legitimates methods that centralise the role o f discourse(s) i n inquiry practices and lays out the undertaking o f the study, its contextual frame and responses to questions of trustworthiness and obl igat ion. 1 7 Chapter five maps pathways o f decision-making taken by participants engaged i n integrative evaluation. The mapping, i n part, locates pedagon, those discursive sites o f tensions for teachers i n the pre-conditions o f decis ion-making. G i v e n significant attention are the troubled discourse(s) o f undecidabil i ty as discursive pre-conditions o f decis ion-making. Chapter S i x discusses the findings f rom chapter five through discussions o f val idi ty, ethicality and positionality. Dominance o f the technical is questioned, the possibili ty o f de-centering the Cartesian condi t ion is explored and an attunement to voices of the learner is considered. Reconceptual is ing E L L wri t ing as a "who l ly human endeavour" ( H a m p - L y o n s , 1991, p. 2), suggesting midd le ways o f judging E L L essays are offered, not for purposes o f mastery and objective clarity but as ways "to l ive wi th [the difficulties] w e l l . " The postscript is i n place o f a conclusion. I briefly re-write the disruptions that guided the 'academic ' wr i t ing o f this dissertation and ponder how it might be judged. A s an inquiry, I entered into the middle-of-things. I learned of language and its effects, o f aporetic sites and their difficulties, and o f m y o w n shifting positionalities. F ina l ly , it is a journey already begun, always incomplete and, in part, never ending. , 7 I heed Davis's (1996) call to resist impossible claims of "scientistic ideals of neutrality, objectivity, and generalizability" and instead engage in goals of "viable understandings rather than verifiable facts, relevant interpretations rather than generalizable conclusions" (p. 26), hence I argue for this project's trustworthiness. 15 I attempt to enter into the middle o f the "pedagon" o f instances o f integrative evaluation. Co-speci fy ing relations among the researcher, the subjects, the topics emerging from the folds o f the project, the sites of struggle that get structured as questions, and the historical and current conditions of the inquiry become paths which weave intricately as tangled networks o f words where contrary themes coll ide, the previously unknown becomes present, and st i l l - to-be-known is not-yet-to-be-found i n the on-going, day to day tasks o f teachers judging wri t ing. It is the multiple contrary themes and the sti l l-to-be-known that are l i v e d out as generative spaces o f difference, as "or ig inal diff icult ies" that are o f interest to this inquiry and unfold i n the remaining chapters as an intertextual struggle o f the possible l imits and promise o f the difficulties of integrative evaluation practices as/in contested spaces. 16 Chapter two Contested Spaces as/in "Stuck Places": Evaluation, Literacy, and Language/Content Integration A praxis o f stuck places . . . [refers to] f inding a way out o f situations f rom w h i c h there is no way out . . . where we are out o f our depth and forced to be resourceful, elusive, w i l y in finding a path that does not exist. Patti Lather, Validity, ethics and positionality in qualitative research: Wrestling with the angels Ideology is not reproduced as a c losed system for ta lk ing about the w o r l d . Instead it is reproduced as an incomplete set o f contrary themes, w h i c h continually give rise to discussion, argumentation and di lemmas. M i c h a e l B i l l i g et al . , Ideological Dilemmas Language , t yp i ca l ly , is immersed i n the ongoing l i fe o f a society, as the pract ical consciousness o f that society. Th i s consciousness is inevi tab ly a partial and false consciousness. Robert Hodge and Gunther Kress , Language as Ideology 17 Positionalities and Meta-texts Soc ia l practices as ' l i v i n g ' texts appear to embed diverse pos i t ional i t ies 1 8 ; those necessary i l lusions we have come-to-know that persuade us to assume that "the way things are is the way they ought to be" (Nichols , 1981, p. 1). T o understand l i v i n g texts i n this way involves an awareness of how everyday lives are implicated and accentuated by meta-texts w i th in w h i c h positionalities disrupt as ambiguous and tensioned "stuck places" . 1 9 The purpose o f this chapter then is to explore understandings o f contested spaces and how they can be understood as l iv ing practices of integrative evaluation. First, complex concepts o f posit ionali ty and o f meta-texts are explored. Then, subsequent parts o f this chapter explore various confl ict ing and ambiguous positionalities wi th in meta-texts. In a turn from assumptions of a f ixed, commensurate discourse o f standpoint, I v i ew positionality as a performative interplay of dynamic, contested discursive relations that unsuspectingly frame how we understand the wor ld (Lather, 1998). It is a stance-in-flux that rejects the possibi l i ty o f consistently held systematised solutions or discursive univocal i ty , and acknowledges the partiality of ideological assumptions and their di lemmatic conditions engaged i n a movement o f deliberation and negotiation ( B i l l i g et al . , 1988). Posit ionali ty is influenced by notions o f ideology that "l ink[] the textual and the po l i t i c a l " (Lemke , 1995, p. 2) through relations o f meaning and power in discourse (text). Posit ionali t ies are those c o m m o n taken-for-granted meanings that have become c o m m o n sense and guide everyday life. Constructed i n social dialogue, these meanings have for the most part been inherited f rom others. A s social , they are meanings that are subjected to pol i t ica l acts and as a result some meanings have been privi leged over others. Writ ten as c o m m o n sense, they become unquestioned ways o f understanding o f the wor ld . L e m k e (1995), a Nor th Amer i can educational scholar, summarises 'modern ' c o m m o n sense as a need for certainty and common ground. Rough ly from the time o f Descartes' Meditations, European intellectual traditions have sought to escape from the radical skepticism that ended the earlier age o f 1 8 1 use the term positionality as a main focus of this dissertation; however, other terms denoting similar ideas are orientation, imaginary and ideology. Although I use Billig's work with ideology to further the argument, I minimise the use of the term ideology in my own work because of modernist assumptions associated with 'logos'. 1 9 1 use Lather's term "stuck places" as contested spaces, those aporias wherein teachers struggle within counter-themes, and incomplete and dilemmatic ideological structures to evaluate E L L texts. 18 rel igious faith. The principal strategy for this escape has been to f ind some basis o f certainty other than faith in religious revelation: some c o m m o n or uncommon sense way o f proceeding toward understanding, some safe place f rom w h i c h to begin. Th is strategy produced a new faith i n logic and logica l inquiry, first in philosophy, then i n mathematics, and finally in science. E a c h c la imed to have discovered a trustworthy method o f proceeding, a safe ini t ia l set o f assumptions. These methods cla imed to be universal: va l id by logical necessity, in a l l times and al l places, for a l l people and al l purposes. They became our modern c o m m o n sense, (p. 2) Ques t ion ing the "our" i n "our modern c o m m o n sense", L e m k e asks, "Whose strategy for life produced this c o m m o n sense?" (p. 3). Interested i n issues o f power , meaning and rul ing ideologies, he pursues how the modem common sense we have come to k n o w is a product o f history, constructed by an exclusionary group who were inattentive to the needs of others outside the upper class European intellectual communi ty o f that t ime. W i t h s imi lar interests, Hodge and Kress (1993) c l a im, "The grammar o f a language is its theory of real i ty" (p. 6). O f interest i n this project is not how pr iv i leged meanings structured i n particular ways support the power o f one social group over another, or why one ideology dominates another, instead m y interest is in locating and interpreting the ideological functioning o f those dominant and resistant themes wi thin the discourses o f integrative evaluative practices. Concerned wi th " log ica l necessity" as perhaps being a necessary i l lus ion , I resist the use o f ideo(logy) and use positionality to assist i n interpreting practices o f integrative evaluation. I have scripted the term meta-text to refer to broad concepts wi th in w h i c h numerous fissured positionalities dwe l l . 2 0 In particular, I focus on three meta-texts expl ic i t ly invo lved i n integrative evaluation: 1) curr iculum perspectives and their impl ied approaches to learner evaluation, 2) literacy orientations, and 3) linguistic perspectives on language and content relations. Meta-texts embody orientations that construct a commonsense o f particular aspects o f the daily activities i n which educators engage. F o r example, one o f the meta-texts of integrative evaluation, literacy, involves an understanding o f what it means to read written content. Is reading an autonomous act? Reader response theorists w o u l d argue 2 0 1 argue that multiple positionalities dwell within any meta-text; positionalities that are incomplete, have dominant and resistant counter-themes and give rise to dilemmatic pre-conditions of decision-making (Billig et al., 1988). Given these inherent conditions, tensions can emerge in within and between meta-texts. 19 not. 2 1 Evidence o f a meta-text of literacy at work in teacher's practices i n the current project is found i n the positional traces of their discourses, particularly evident when they engage i n the pre-conditions o f deciding a mark for a particular E L L text. Wha t assumptions underlie meanings made from the E L L texts they read and then judge? W i t h i n the discourse(s), commonsense assumptions o f literacy are endlessly being re-written as contrary themes are interpreted and re-negotiated in the act o f judging that text. Meta-texts, as frames o f reference for tracing commonsense assumptions of l i v ing practices, are not v iewed as offering a number o f shared, whole and complete systems wi th in wh ich individuals choose to act. Instead, meta-texts seem to be regularly c la imed by assumptions o f positionalities that are fragmented, contradictory and partial ( B i l l i g et al . , 1988); meta-texts become fissured texts o f a "false consciousness" (Hodge and Kress , 1993, p. 6) o f integrative evaluation practices. Contested Spaces of L i v i n g Texts It is argued i n Chapter Four that activities comprise interdependent relationships, creating a k i n d o f ecological embodiment of l iv ing practices. Considered i n this way, the activity o f integrative evaluation is viewed as containing sub-activities and is itself a sub-activity o f some larger context. Here, I want to draw attention to the sub-activities or aspects o f l i v i n g practices that seem to locate particular points o f (in)decision, contestation, and ambiguity and then consider, in general, implicated positionalities o f di lemmatic discourse(s) as evidenced i n inter-related ambiguous instances o f integrative evaluation. A number o f sub-activities invo lv ing the pre-conditions of decis ion-making for teachers wi th in the practice o f integrative evaluation may include: 1) how to write the prompt and what information to include; 2) what criterion to use i n the judging o f texts and how to express the criteria; 3) how to read the text; 4) how to apply the evaluative criteria; and, 5) how to (re)mark the text being judged. A lbe i t routine, the decisions inherent i n each sub-activity are difficult ones, especially when the curricular and instructional context is meant to address learners functioning i n another language. A s has already been stated, one of the main purposes o f this project is to explore the 2 1 Reader response theorists are concerned with how readers make meaning from their experience with a text. They claim that "the text cannot [original emphasis] be understood or analysed as an isolated entity." (Beach, 1993, p. 1) 20 activity o f judging E L L written content and its felt tensions. In explor ing difficulties, particularly those emerging in teachers' discourse(s), I consider the work o f B i l l i g et al (1988) and B i l l i g (1996). Al though they take a rhetorical approach to arguing and thinking, m y interest is in their attention to the dilemmatic preconditions o f decision making . They write, The existence o f contrary maxims, or opposing pieces o f folk w i sdom, illustrates that c o m m o n sense possesses a dilemmatic nature. These contrary themes are the preconditions for those dilemmas in wh ich people are faced wi th difficult decisions. It is n o t . . . how people cope wi th these situations and how decisions are made. Our concern is more wi th the dilemmatic preconditions, i n other words wi th those contrary themes wh ich under normal circumstances are reflected i n people 's thoughts. ( B i l l i g et a l , pp. 2-3) W h a t they allude to is that individuals (in this case teachers) are "not to be seen as being fully preprogrammed by neatly systematised plans o f action, wh ich are await ing the appropriate triggering stimulus and which obviate the need for a l l deliberation" ( ibid. , p. 3). Instead, they c l a i m that deliberation, o f what they ca l l "contrary themes", is a necessary occurrence in everyday life. Contrary themes, I wou ld argue, that are not necessarily reduced to "either/or" but include mult iple historical and current traces located i n the contested spaces o f discourse(s) enacted in activity. Rewrit ten as/in integrative evaluation, their theory o f dilemmatic ideologies suggest that even the 'best' systematised plans o f action w i l l not erase a l l o f the di lemmas inherent i n routine acts o f integrative evaluation. They argue that practices are not pre-determined, but are negotiated among relationships enacted in practice. In this project, relations are not only between ' s e l f and 'other', but textual relationships o f evaluation, literacy and language and content integration - intertextual relations where one text is read against the background o f other texts everpresent i n the first. Here, the difficulties as ways o f k n o w i n g and doing integrative evaluation bare themselves to the histories from w h i c h the current moment emerges. Anticipated in disparate discourses are multiple contested themes that offer endless debate and disruption. It is a space that is occupied, not wi th systematised solutions or positions o f univocali ty that are held consistently by people, but wi th di lemmas that are contested and negotiated as conflict ing positionalities shape and are shaped by the interdependent sub-activities within l iv ing practices. F e w w o u l d deny the agon teachers face i n turning the graphite (marks o f a pencil) 21 on a graded text to ink. Yet , it is the preconditions o f that decision, Caputo 's (1993) "undecidabil i t ies", that are o f particular interest to this project. It may be helpful at this point to return to an instance o f difficulty found in the conversation between two teachers that was offered earlier on page eight in chapter one. B o t h teachers comment on whether to give whole or separate marks for language and content. N o other possibilit ies are evidenced. One wants to give a whole mark but acknowledges that s/he separates language/content whi le marking because it is easier. The other wants to give separate marks even though s/he knowledges it is problematic. They both question language and content relations i n asking, does language effect meaning or meaning effect language? If language and content relations are holistic (i.e., integrated), separating them could be v iewed as an act o f ' f i x i n g ' a ' technical ' problem; taking a complexif icat ion and d iv id ing it into arbitrary 'static' , s imple pieces, cou ld be perceived as leading to artificial and possibly unfair judgments. Ye t , there is a felt ease at presuming to separate language and content. Without explor ing further the norms, values and social expectations, and positionalities that make up the fragments o f shared social knowledge lurking beneath the surface (as this w i l l be done i n Chapter five), this instance illustrates how a particular d i lemma (to give a whole or separate mark) presupposes more general meta-textual dilemmatic aspects of practice (integrativist vs separatist v iew o f language and content relations; objectivity vs. subjectivity o f judgments; ease vs. perceived fairness). In part, the dilemmas are presumed to be resolved through a negotiated settlement among contested themes, while others seem to be temporarily resolved through a restructuring o f presumed objective texts to reduce the subjective messiness impl ied in taking an integrative stance. In the d i l emma outlined above, the phrase 'language and content' seems to be a shared c o m m o n ground, yet it also appears to be fragmented, i l lusory and unfixed i n its meaning wi th in and among the teachers i n conversation. B i l l i g et al . (1988) argues that "i t is not merely that the bits and pieces of social knowledge are themselves social ly shared, but that what are shared are confl ict ing bits and pieces" (p. 15). Wha t teachers k n o w about integrative evaluation and what they do to enact their values underlying notions o f fairness and consistency can be a space o f difference wi th in and among individuals; potentially confl ict ing aspects o f a presumed shared, common ground that effect di lemmatic 22 discourse(s). Di lemmat ic aspects, according to B i l l i g et a l , are expressed both expl ic i t ly and impl ic i t ly in routine conversations. They write that explici t acknowledgment o f di lemmas is usually expressed i n the presentation o f both sides o f the debate as reasoned choices. 2 2 The meaning argued for choosing one over another can be generally fo l lowed from the overt discourse structures. Constituting impl ic i t meanings from dilemmatic texts necessitates greater interpretive acts. Implici t meanings, B i l l i g et al . suggest, are the traces o f negative meaning, counter-themes, that may be contained wi thin the same semantic structuring as explici t di lemmas but are hidden from view. They argue that impl ic i t meanings may be unknown to the speaker as w e l l as to the listener. B i l l i g et a l . address the complexi ty o f meaning as it is structured i n discourses as dilemmatic ideology. They begin wi th the assumption o f contrary themes and a im to understand discourse meanings as a k i n d o f dialectic; a k i n d o f discourse that structures explici t , dominant meanings as w e l l as counter-themes o f negation that give rise to deliberation. It w o u l d appear that the teachers' conversation on page eight o f chapter one is an example of just such a dialectic. Dialect ics are presumed to occur because o f the contrary themes evidenced i n discourse(s). Those bits and pieces, the fragments o f shared c o m m o n and contested ground, o f l i v i n g di lemmatic positionalities - those meta-texts and orientations that seem to be incomplete, impl ic i t ly or expl ici t ly conflicting in everyday discourse(s). In considering the work o f teachers engaged i n integrative evaluation and the meta-texts that influence their work, orientations towards evaluation, conceptions of literacy and views o f language continued to emerge. A l though listed as i f in an order o f privilege, I want to acknowledge that each o f these areas have their o w n varied order of import wi th in and between individuals . I also want to acknowledge that each area informs and is informed by the other. F o r example, a strong technological orientation towards evaluation may influence an ind iv idua l ' s v i ew o f language and/or literacy. M y purpose is to locate traces o f positionality o f at least these three dimensions o f integrative evaluation and, i n so doing, make the 2 2 Billig's work seems to be framed by dualistic thinking. However, I read it as at least two sides of a debate and am alert to the possibility of more than two alternatives in teachers' discourses. Interestingly, as I read and then wrote of various orientations within each of the three meta-texts explored in this work, two predominant orientations consistently emerged and were often discussed in binary, opposing terms in the literature. 23 contrary bits and pieces more visible through interpretive work. In the fo l lowing sections, a synopsis o f three meta-texts of integrative evaluation wi th possible positionalities wi th in each are offered. Orientations to Student Evaluat ion Through multifarious educational experiences as teachers and as students, educators have come to engage i n student evaluative practices using their o w n favoured approaches and strategies. A s familiar resources o f evaluative practices, these approaches may unknowing ly shape and be shaped by evaluator's acts i n particular ways. It is a process whereby unquestioned commonsense has the potential to dictate the form of evaluation and how it is understood. W i t h i n pr ivi leged models and techniques, traces o f at least one predominant positionality i n education, framed as a technical orientation (Tyler , 1949), seems to linger. It is a convincing perspective o f scientific rationality that attempts to reduce education, and therefore evaluation, to precise technical acts. A l though the dominance o f one mode l can itself be troubling, of more concern is that, the dominance [of a single mode of evaluation] may lead evaluators to forget that the fo rm of" evaluat ion should be appropriate to the phenomenon to be evaluated and that the eva lua t ion approach shou ld be respons ive to the interests to be served by the evaluation." ( A o k i , 1991, p. 98) It has been my experience that many of the current evaluation practices o f E L L academic wri t ing s t i l l remain loya l to the use of an objective form of evaluation that purports to 'measure' the subjectivities o f wri t ing. One o f the purposes o f this inqui ry is to probe the effects o f such relations between form and phenomenon. Current discussions on curr icu lum offer a number o f perspectives as ways o f understanding educational orientations towards evaluation. 2 3 Summarised i n Table 1, the orientations described lean heavily on the work o f A o k i (1991) and his attempt to expose wor ldv iews that influence the ways program evaluation is understood and gets done. I borrow three o f the curricular orientations he offers i n his discussion o f program evaluation: ends-means, praxical and emic. I then introduce radical hermeneutics as a possible fourth orientation that is gaining prominence i n hermeneutics (Jardine, 1992; Smi th , 1999). It is 2 3 For various readings of current discussions on curriculum orientations and implied learner evaluative practices, see Eisner and Vallance (1974), Pinar (1975), Miller (1983), Miller and Seller (1990), McNeil (1990), Pinar and Reynolds (1992), and Eisner (1994). 24 not m y intention to usurp the values of one orientation for another but to suggest that even wi th alternative ways of envisioning evaluative practices, the rootprints o f the most dominant orientation, ends-means, is unforgiving. A current headline i n a loca l newspaper reads, " R i c h m o n d okays traditional school . " 2 4 A desire to return midst contrary contemporary trends i n education; the roots o f a ' traditional school ' - a return to the basics that presumes to prepare students for adult life - as an ends-means orientation are everpresent. It is a space where the past is rewritten in the present, evolving anew each time the story o f ' traditional school ing ' is told. If traces o f a traditional orientation continue to have a stronghold against alternate current curricular recommended practices, I w o u l d argue that it also continues to have a stronghold in evaluative practices, as it seems that evaluation is often one o f the last accomplishments o f curricular change. A s a prevai l ing wor ldv iew, the 'tradition' referred to is an orientation to technology that began to emerge wi th the rise o f science i n the 17th century. In response came a shift i n the use o f language and the felt need to be precise and univocal . 2 5 Ini t ial ly, the age o f technology was centred in business practices o f precision, efficiency and productivity. Th is orientation eventually found its way into curr iculum practices. Ty le r (1949) wi th his seminal work, Bas i c Principles of Cur r i cu lum and Instruction, was an influential proponent o f a technological orientation i n education. A s a credible legacy from a mainstream social theory that expounded tenets of instrumental reason, a technological orientation o f education for many became the orientation. Knowledge as neutral, finite facts cou ld be attained and evaluated through technical, ends-means acts. It is an evaluative stance immersed in positivists ' traditions of objectivity and concerned wi th questions that ask, what is the most effective and efficient means to evaluate student achievement o f curricular and instructional ends (objectives)? It is an ethos to wh ich many evaluators as educators sti l l knowingly , and unknowingly , remain loyal . 2 4 The headline is from The Vancouver Sun. (Wednesday, June 17, 1998, p. B3) and is reflective of the current educational climate in various parts of British Columbia and across the country. In arguing for traditional schools, images of multiple and conflicting themes are reflected in a veil of back to the basics, yet, in Richmond anyway, the historical and current moment suggests a more deep-rooted interest - that of addressing the assumed problematics of ESLness. 2 5 Smith (1999) in Pedagon writes of the valued ambiguous and discursive quality of language before the rise of science in the seventeenth century. A new orientation to language that demanded precision evolved in response to the age of technology. 25 Table 1 A Synopsis o f Prevai l ing Evaluat ion Orientations Adapted F r o m A o k i (1991) Orientations to evaluation Unde r lv ing assumptions ends-means (technical) focus on instruction (influenced by the work Tyler , 1949) • planned teaching goals and objectives to achieve specific ends that can then be judged • tacit interests i n teachers' control and manipulation o f teaching, learning and evaluat ion 2 6 • need to determine to what level o f certainty instructional objectives have been met • engage i n criterion-referenced activities for purposes o f predictability • measure levels o f attainment o f determinate, commodif icat ion o f knowledge and ski l ls to mark end-points o f curricular study praxical focus on learning (influenced by the work of Freire, 1970) • effect active social change • students and teachers generate complex information from evaluative practices that informs the cur r icu lum • evaluation practices are acts o f cri t ical reflection and transformative action that lead to emancipation and social reconstruction o f under ly ing assumptions emic (insider view) focus on relationships of teaching and learning (influenced by anthropological insights) • evaluation aims to determine the quality o f students' experiences • interest i n students' subjective knowledge i n situ • attempts to access the interrelations o f values, beliefs and actions o f specific activities • emic views attempt to access learners' ways o f knowing • students' evaluate themselves e.g., self and portfolio assessment • evaluation as an act o f thick description radical hermeneutics focus on "keeping the difficulty of life alive" (influenced by the work o f Caputo,1987) • gain insights into students' moments o f diff icul ty (as difference) i n their l i v i n g practices (e.g., "stuck places" when wri t ing content) • resist a metaphysical desire for presence (e.g., written content embeds the said and the unsaid) • interpretation as a performative act (i.e., meaning of a text is created i n recursive interpretations and intertextual negotiations) • evaluation includes negotiated acts o f valuing creative paths l a id whi le w o r k i n g through "stuck places" (i.e., va luing discourse structures written i n cultural hybridity, where language effects meaning that is constituted i n difference) Aoki (1991) writes ends-means orientations as hidden agendas of control and manipulation. 26 The other three orientations, though similar in their interests o f explor ing social and l i v i n g experiences, are rooted i n traces o f positionalities that differ significantly. W h i l e the praxical orientation can be traced to neo-Marxian cri t ical social theory and the need to disclose hidden relations o f power and meaning as experienced through dialectal engagement o f cri t ical reflection and practical action, and to ask whose interests are being served, emic interests target a thick description of the subjective wor ld o f social relations between participants i n situ engaged in particular activities being evaluated. E m i c evaluators are interested i n the views o f participants as they relate to the self-determined quality and relevance o f the experience. The possibility o f a radical hermeneutic stance in evaluative practices is taken up by those interested i n the dynamic and complex, l i v i n g relationships i n schools wherein difficulties as "stuck places" dwel l . In introducing radical hermeneutics, Capu to (1987) offers, hermeneutics always has to do wi th keeping the dif f icul ty o f l i fe a l ive and w i t h keeping its distance from the easy assurances o f metaphysics, (p. 3) It is a v i ew o f the learner in flux that questions the pr ivi leging o f a metaphysical presence and seeks that w h i c h has been silenced by a v iew o f life as objective and factual. Conceptions o f Li teracy for Texts in Schoo l A l though I recognise that conceptions o f literacy are not restricted to traditional notions o f reading and wri t ing, for purposes o f i l luminat ing possible frameworks o f literacy for texts used i n schooling, this discussion w i l l be l imi ted to reading and wri t ing typical school texts. One such literary event, reading student wri t ing, embeds underlying assumptions about what a text is, how it should be written and how it should be read. M a n y of these assumptions have long standing traditions and commonsense appeal, and therefore remain unquestioned. In the western tradition o f formal education, one such prevai l ing ethos described by Street (1984) is an autonomous model o f literacy where text is v iewed as object. Recent literacy debates have called into question such traditional views and offer alternative models to literacy. Fo r example, H i l l and Parry (1994) introduce a pragmatic model o f literacy based on their interest i n texts used i n reading tests and ca l l on us, as others have, to consider text as social communicat ion. S imi la r ly , Hasan & W i l l i a m s (1996), 27 respected for their work i n literacy, c l a i m that "any 'literary event' i n social l ife is necessarily one w h i c h implicate readers, writers and texts understood as language i n use in social contexts" (p. x i ) . Table 2 outlines possible traces o f positionalities o f two dominant conceptions o f literacy for texts used in schooling. I do not assume that teachers locate themselves i n one concept or another and that a l l o f the characteristics listed wi th in one must be embraced by those that give allegiance to it. Table 2 A Summary o f T w o Conceptions o f Li teracy Adapted f rom H i l l & Parry (1994) Conceptions o f school literacv Unde r lv ing assumptions autonomous focus on writer • text as an objective product represents unambiguous and autonomous meaning, it can be separated from its producer • reading/writing as technical acts, l inguistic and autonomous ski l ls are transferable across texts • writer/reader is autonomous to the text social focus on reader • text is social e.g., reader response theories • reading/writing are active mediations o f socially constituted meaning • the act o f producing text continuously hails the voices of other text makers •writing/reading involves social reciprocity Literary events are complex act. Reading student wri t ing is made even more complex by a l ingering desire o f most Eng l i sh writers to communicate directly wi th their readers. A s an autonomous text, the intended meaning o f the writer is presumed to be the meaning o f the text and one the reader must somehow come to know. H o w has wr i t ing and reading come to be understood i n this way? Smi th (1999), leaning on The F l igh t F r o m A m b i g u i t y by Lev ine , offers, "an orientation to technology inspired by the rise o f science creates a demand for precision and univocity i n language" (p. 170). It is a movement that seems to continue to be an influential and persuasive perspective o f the West . Evidence o f wri t ing as a hunt for certainty can be traced to Descartes and his concern for the possibi l i ty o f meaning as indisputable and certain. Husser l i n Eagleton 28 (1983), l ike Descartes, c la imed to establ ish certainty, then, we must first o f a l l ignore , or "put i n brackets" , anything w h i c h is beyond our immediate experience; we must reduce the external w o r l d to the contents o f our consciousness alone, (p. 55) It was the beginning of what is k n o w n today as phenomenology, the science o f pure 'phenomena' as realities o f consciousness. What can be known , and therefore what we can be certain of, was the universal essences and unchanging aspects o f the phenomena under study. It was a move away from abstraction to work wi th the concreteness o f experience, discovering what was essential and invariable. Perception came from k n o w i n g the very essence o f things. H o w was this essence or meaning understood to be communicated? Eagleton (1983) comments that to Husserl , M e a n i n g is something w h i c h pre-dates language; language is no more than a secondary ac t iv i ty w h i c h g ives names to meanings [we] s o m e h o w already possess, (p. 60) This stance privileges meaning, g iv ing it pre-linguistic status and minimises the role o f language i n communicat ion. Eagleton (1983), in explaining Husser l ' s posi t ion, argues, M e a n i n g is an ' intentional object ' . It is a k i n d o f ' i d e a l ' object, i n the sense that it c o u l d be expressed i n a number o f different ways but s t i l l remains the same meaning ... the meaning o f a literary word is f ixed once and for a l l : it is identical w i th whatever 'mental object' the author had i n mind , or ' intended' at the time o f wri t ing, (p. 66) Others, such as Saussure and Wittgenstein, i n questioning his notions o f meaning as object, made revolutionary claims o f language as the producer o f meaning. Ye t , for Saussure some o f the ideas Husser l and Descartes put forth remained close to his work . Saussure's work i n semiotics evidences the import he attributed to the signified (meaning) over the signifier (language) and the significance o f the direct and stable relationship between the two, while acknowledging the arbitrariness o f the sign. The legacy o f what it is to write in Eng l i sh have led many o f us to believe that words carry meaning i n a direct, representational relationship that is absolute and unchanging. Eagleton (1983) summarises A m e r i c a n hermeneuticist Hi r sch ' s perspective on wri t ing and meaning as, a meaning o f a w o r d is ident ical w i th what the author meant by it at the t ime o f writing.. . .There may be a number o f different va l id interpretations, but a l l o f them must move wi th in the 'system of typical expectations and probabil i t ies ' w h i c h the author's meaning permits.(p. 67) 29 In this stance, the author remains the authority of meaning o f his/her text. It appears to be a locat ion where the product can be separated from the producer and stand autonomously as a representation o f ' intended' meaning. If wr i t ing is understood, and therefore 'produced ' i n this way, reading, s imilar ly understood, should be unproblematic. It w o u l d seem then, that the 'hunt for certainty', the knowing of what one has come to consciously know, can be contained i n words. F r o m Descartes dictum: I think therefore I am, the writer cou ld argue, what I say is what I mean. Objectifying meaning and reducing the role o f language to a conduit seems to remove the possibil i ty o f valuing a 'meaning gap' i n wri t ing; it is a stance that works to disinherit ambiguous and uncertain relations i n texts. Hermeneutics, the art o f interpretation, as engaged i n by Heidegger and his German successor, Gadamer, was concerned wi th fundamental questions o f text interpretation: interrogating the relationship between textual meaning (as interpreted by a reader) and the author's intention, and inquir ing into the (im)possibili ty o f 'objective ' , autonomous text (Gadamer, 1977). A l though this work held, and continues to hold, promise, it d id not seem to signif icantly weaken the hold o f Husser l ' s 'truths' and the dominant ideas o f text as a representation o f ' intentional ' meaning; assumptions about text that currently many i n educational circles continue to remain loya l . It seemed that the hunt for certainty, a 'c lose ' reading o f texts, had been allegedly reduced to words that can say what they mean. H i l l and Parry (1994) and Hasan and W i l l i a m s (1996) advocate for a social mode l o f literacy that involve interpretive acts. Ashcroft (1996) notes that, as a form o f dialogic accomplishment, the writer (text) and the reader negotiate meaning wi thin socio-cultural contexts that shape and are shaped by the literary event. In this v iew, literacy becomes a performative, social act o f reciprocity i n which the meaning " o f [reader] responses varies considerably according to differences i n specific social , historical or cultural contexts" (Beach, 1993, p. 7). Attention is given to the il locutionary and perlocutionary forces o f discourse, conveyed by the situation o f the writing/reading event. W r i t i n g ' s perlocutionary force is not its expressive potential o f the grammar and mechanics o f E n g l i s h usage but is the effect o f language actualised in the social act o f wri t ing and reading. The writing/reading event is the place where the language system, and its users converge; where wor(l)ds intersect. M e a n i n g becomes a dialogic accomplishment wi th in a social and 30 inherited context o f dynamic interchange that brings together mult iple wor(l)ds of the writer and reader; where writing/reading generatively alters and (re)constructs meaning. Rooted i n notions o f discourse as a social process, this understanding o f literacy questions how 'true' meaning is possible and instead deals wi th social ly and experiential ly-based realities that are multiple, elusive and susceptible to change. C a l l i n g into question the restrictions rules place on wri t ing, this v iew o f literacy asks to be freed o f those constraints; to be free to write knowing that message w i l l be understood differently and only partially each time it is read. Differently and partially in that the cross-il locutionary (cultural) forces between writer and reader (the situation) create a gap that is installed by their variances o f language and context. Mean ing is constituted from the networking o f differences, w i th the gap resisting complete understanding and representation. Writers wi th in this orientation seem to c l a i m that symbols and signs as words do not directly l ink to the experiences they write; instead, experiences are disrupted and suspended in the slipperiness o f the relationship between the signifier and the signified and between signifier and signifier. M e a n i n g seems not to be created by encoding a pre-existing thought; instead, k n o w i n g (as writer and as reader) is a function o f the gaps and differences that traverse and resist the text. Understanding that words do not directly represent our experiences recognises the potential for multiple and partial interpretations o f meaning in the enactment o f (re)writing i n difference. W r i t i n g i n difference is freed from the immutable laws that confine and reduce k n o w i n g to one truth; wri t ing in difference acknowledges the ambiguity and alterity o f social ly constructed ways o f knowing . Here meaning is not owned; neither the writer nor the reader are held responsible for the meaning o f the text. Instead, meaning is dependent on what is brought to the text by the writer and the reader. In the intricacy, hybridi ty and constant shifting o f language-in-use, there is no one standard code; language, and the meaning it constitutes, arises out o f a struggle to write what is k n o w n and to k n o w what is written. Ashcroft (1996) reminds us, W o r d s are never s i m p l y referential . . . but have a number o f meanings , depending on how they are used [the s i t u a t i o n ] . . . the almost l imit less p ro l ix i ty o f words brought to the site of meaning . . . demonstrates the total dependence o f that meaning upon its 'situated-ness'. (p. 300) 31 Reading and wri t ing seem to connote meaning by the function o f their situation. Ashcrof t (1996) c la ims , "the process o f reading itself is a continual process o f contextualisation and adjustment directly l inked to the constitutive relations wi th in the discursive event" (p. 301). It is a social model o f literacy, where language seems to be inextricably bound to its social reality, functioning to effect meaning in a dialogic accomplishment. Dominant V i e w s o f Language The commonsense o f needing to know how language works seems obvious. In second language classrooms, learning Eng l i sh is pr imari ly thought o f i n l inguist ic terms, differing from a literary perspective emphasised in first language learning. Therefore, l inguist ic assumptions o f language and content relations are briefly reviewed as possible positions teachers take i n judgments of texts. A formal v iew, pr imari ly based on the influential writings o f Chomsky , and a functional v iew, centrally located i n the work o f Ha l l i day , fo rm the basis o f two linguistic orientations to language. M y purpose is to focus on these two dominant positions as they have had significant impact on first, second and foreign language pedagogical practices. There are at least two theoretical stances i n linguistics that provide different assumptions about the general nature o f language and the goals o f l inguist ics: " formal (or structural) and functional" (Schiffrin, 1994, p. 20). A s Schiffr in reports, several writers have contrasted these stances. Kress (1985) makes the formal/functional contrast using the terms "social" and "more traditional l inguistic approach" approach: Perhaps it w i l l help to characterise the latter [the more tradit ional l inguis t ic approach] i n a few sentences. W i t h i n the d isc ip l ine o f l inguis t ics there is a strong and s t i l l dominant strand w h i c h regards the study o f phonology and syntax and their theoretical treatment as 'real' l inguist ics. Th i s strand asserts the autonomy o f l inguis t ics , i n terms o f its theories, methodologies and subject matter. The approach is characterised i n journals such as Language, Journa l o f L i n g u i s t i c s , L i n g u i s t i c Inquiry . In the late 1950s, 1960s and 1970s, this approach was epitomised by the work o f N o a m C h o m s k y and o f the theory o f Trans fo rmat iona l G r a m m a r . A n o t h e r strand has a l w a y s emphasised the socia l dimensions o f language . . . The theoretical w o r k o f M i c h a e l H a l l i d a y is i n that tradit ion. In the mid-1960s , . . . D e l l H y m e s in t roduced the concept o f ' communica t ive competence ' i n reac t ion to C h o m s k y ' s narrow and asocial defini t ion o f l inguis t ic competence. H y m e s ' w o r k has been most important both as a corrective and an alternative theory 32 of language as a social phenomenon, (p. 98) H y m e s (1974:79) makes the fo l lowing contrast between formal (structural) and functional: Structural: structure o f language (code) as grammar. Use merely implements, perhaps limits, may correlate with, what is analysed as code; analysis o f code prior to analysis o f use. Funct ional : structure o f speech (act, event) as ways o f speaking. Ana lys i s o f use prior to analysis o f code; organisation o f use discloses additional features and relations; shows code and use i n integral (dialectical) relation, (p. 79) L e e c h (1983) contrasts formalist and functionalist approaches to l inguist ics i n the fo l lowing way: (a) Formalists (e.g. Chomsky) tend to regard language pr imar i ly as a mental phenomenon. Functionalists (e.g. Hal l iday) tend to regard it pr imar i ly as a societal phenomenon. (b) Formalists tend to explain linguistic universals as der iving f rom a c o m m o n genetic linguistic inheritance o f the human species. Functionalists tend to explain them as deriving from the universality o f the uses to w h i c h language is put i n human society. (c) Formalists are incl ined to explain children's acquisit ion o f language i n terms o f a bui l t - in human capacity to learn language. Functionalists are inc l ined to explain it in terms o f the development o f the child's communicat ive needs and abilities in society. (d) A b o v e a l l , formalists study language as an autonomous system, whereas functionalists study it i n relation to its social function, (p. 46) Schiffr in (1994) further characterises formalist views: Formalis t views . . . argue that although language may very w e l l have social and cognitive functions, these functions do not impinge upon the internal organisation o f language. Newmeyer (1983) captures these qualities i n two defining characteristics: autonomy and modularity. First, autonomy (p.2): the grammar of a language is characterised by a formal autonomous system.That is, the phonology, syntax, and those aspects o f meaning determined by syntactic configuration form a structural system whose primit ive terms are not artifacts o f a system that encompasses both human language and other human facilities or abilities. (Emphasis i n original) . The formal autonomy of the grammar, however, does not prevent intersection wi th other modules: surface features o f phonology, syntax, and semantics can result f rom the interaction o f the "formal grammar" module wi th other equally autonomous modules, each governed by its o w n set o f principles. Such modules might include perceptual psychology, physiology, acoustics, conversational principles, and general principles o f learning and concept formation, (p. 22) Ha l l i day and M a r t i n (1993), using Funct ional Linguis t ics ( S F L ) as an example o f 33 functional l inguist ics, characterise functional views as being " oriented to the description o f language as a resource for meaning . . . [ ; ] concerned wi th texts[;] . . . focuse[d] on solidary relations between texts and social contexts^ a n d ] . . . concerned wi th language as a system for construing meaning" (p. 22-3). S F L is a v i ew that accounts for how language, as a resource, is used to constitute meaning. Language as a systemic functional grammar considers "the system that lies behind the text" (Hal l iday , 1985, p. x i i i ) to explain how language functions 'natural ly ' . In S F L , c o m m o n l y k n o w n as functional grammar, the conceptual framework is one o f rhetoric rather than logic . It is based in systemic theory that views language as a semiotic system, as a theory o f meaning as choice at the level o f discourse. Therefore, the grammar becomes a grammar o f choices, not o f rules. Al though the term grammar is used i n both perspectives, here it means syntax and vocabulary and is technically termed lexicogramrnar (yet frequently shortened to 'grammar ' ) . The focus of functional grammar is on text rather than sentences: a site where meaning is negotiated and the relations between texts and social contexts are central (Hal l iday & Mar t in , 1993). Text v iewed in this way is a concern wi th lexicogrammatical choices i n "mutually predictive" relationships wi th text semantics and the contextual considerations they realise. S F L looks at language from the outside inwards, interpreting language by reference to its place in the social process and its potential for meaning. Grammar , f rom this perspective, is explained in functional terms - i n terms o f its use. The resources and choices o f grammar played out wi th in relations among (con)texts, construct meaning. It is an inclusive, transdisciplinary model " i n wh ich language, l ife, the universe and everything can be viewed i n communicative (i.e., semiotic) terms." (ibid. , p. 23). Funct ional grammar relations give primacy and power to language; in the act o f speaking or wri t ing, meaning is socially constituted. The two views have different emphases. This can perhaps best be demonstrated by the kinds o f questions they may ask. Fo r example, formal linguists may ask about the place o f a grammar rule wi th in the formal system of the language, whereas functional linguists frequently ask about the communicative effect of the message given particular contextual considerations. The potential for difficulty emerges in the partial and open systems constructed wi th in and between each orientation o f formal and functional l inguistics. 34 Restoring Meta-Texts to Aporet ic Sites In a chapter that purposefully draws attention to what appears to be a mul t i fo ld messiness o f a current integrative evaluation landscape, it was not m y intention to provide a possible directory o f meta-texts and orientations that have led to this complexi ty or that w i l l lead to a future moment in integrative evaluation, nor is it to offer a more effective and efficient approach that may be vo id o f dilemmas and deliberations, i f that were possible. Rather, as a starting point, each meta-text offers fragments o f at least two dominant, often confl ic t ing orientations that provide a backdrop o f possible positionalit ies-in-movement that may contribute to aporetic sites in integrative evaluative practices. A t moments o f or iginal dif f icul ty , o f being in "stuck places", teachers work their way through and out o f di lemmas. Some may take w e l l worn paths and find themselves 'stuck' again at the same site at a future moment; others may choose to lay a 'new' path as they work their way through. Apor ias are thought o f as having generative possibilities (Jardine, 1992; Lather, 1998); therefore, it is wi th an on-going interest (inter esse - being i n the middle o f things) i n l i v i n g practices that I listen to the discourses o f teachers i n particular, as they work their way through and out o f aporetic sites. Lis tening for various meta-textual and posit ional relations that mediate and complicate teachers' journeys, I believe w i l l provide useful interpretations of how integrative evaluation can now be understood. Evocations o f familiar paths and invocations o f generative possibilities emerge i n the fo l lowing chapters that offer a more detailed landscape on integrative evaluation through the discourses o f presumed authorities (Chapter three) and o f situated teachers' practices (Chapter five). 35 Chapter Three Tracing Orientations: A (Re)View of the Literature Texts are constantly recycled , appearing i n an endless succession o f texts-about-texts, readings o f readings o f readings o f readings. In order to understand this process we need to be able to see it i n reverse, and read texts as w r i t i n g s o f wr i t i ngs o f wr i t i ngs o f wr i t i ngs . . . d e v e l o p i n g an archaeology o f each text that l inks , however uneasily, wi th the histories o f its future . . . . The text (any g iven text) ceases to be a self-evident uni ty, but appears as a relatively accidental site that marks where a series o f discursive processes have br ief ly co l l ided . Producers (authors, speakers) l i kewise lose the semblance o f unity, and become channels through w h i c h various authors and agencies speak and act: the fissured authors o f fissured texts. Robert Hodge and Gunther Kress , Language as ideology A w ay o f teaching is never innocent. E v e r y pedagogy is imbr i ca t ed i n ideo logy . . . [constituted in] language to define the self, other subjects, the material w o r l d and the relation o f a l l o f these to each other. Ideology is thus inscribed in language practices, entering a l l features o f our experience. B e r l i n , Rhetoric and ideology in the writing class Lingu i s t i c content is inseparable from linguist ic expression. B u t i n research and i n c lassroom practice, this relationship is frequently ignored. In subject matter learning we overlook the role o f language as a med ium o f learning. In language learning we overlook the fact that content is being communicated . . . . W h a t is needed is an integrative approach w h i c h relates language learning and content learn ing , considers language as a m e d i u m o f l ea rn ing and acknowledges the role o f context in communicat ion. Bernard M o h a n , Language and content W e define content-based instruction as the integration o f part icular content wi th language-teaching aims . . . . Ult imately, the goal is to enable students to transfer [language] sk i l l s to other academic courses g iven i n their second language. Thus , both i n its overa l l purpose and i n its implementa t ion , content-based instruction aims at eliminating the artificial separation between language ins t ruc t ion and subject matter classes w h i c h exists i n most educational settings. D o n n a Br in ton , A n n Snow, & Marjor ie Wesche, Content-based second language instruction 36 F o r purposes o f the (re) v iew offered in this chapter - m y v iew o f those presumed texts o f authority on 'language and content' -1 play wi th brackets i n the word , author(itie)s, to remind myse l f as a reader that authorial texts are texts o f presumed informed comment rather than 'truths'. Therefore I begin wi th cautioning the reader to be mindfu l o f how texts are written and read - a theme re-visited throughout this thesis. Through a (re)view o f the literature on language and content, I first establish a common ground among authorities i n the f ie ld and then listen for traces o f their orientations as a way o f explor ing language and content as a communi ty o f difference - o f shifting positionalities and w o r d meanings that reflect the influence o f historical and current conditions on integrative evaluation practices. A s a communi ty o f difference, I begin wi th a brief discussion o f a legacy o f learner assessment and evaluation practices, then explore language and content curricular l inks to its assessment, and, f inal ly, consider the presumptions o f wri t ing, pedagogic interventions and its assessment wi th in E S P / E A P communities, a close relation o f language and content, as a way o f educing interpretations o f what and where o f integrative evaluative practices as they are written in authorial texts in the field. O n Reading Pedagogical Author(itie)s o f Language and Content The first two quotes commenting on author(itie)s' subjectivities i n written texts begin an explorat ion in this chapter o f how positionalities are "inscribed i n language practices" o f authored texts. A s authored texts, they are "constantly recycled . . . texts-about-texts" (Hodge & Kress , 1993, p. 181) and work to dispel the not ion o f text as a unified, autonomous voice that claims one single, complete orientation. Instead, the fragmented or "fissured" text "appears as a relatively accidental site that marks where a series o f discursive processes have briefly co l l ided" (ibid). A s fragmented, authored texts, they are written and read wi th in particular contexts o f situation as an engagement i n social meaning and are, therefore, as B i l l i g et al . reminded us earlier, spaces o f explici t and impl ic i t contrary themes and ambiguities. W h e n constructed in particular ways, they may also convey through their powerful authority "representation as reflection o f a separate real i ty" (Cosgrove & D o m o s h , 1993, p. 27). It is not m y intent to engage i n a discussion o f a crisis 37 of representation here, only to become attuned to fragmented orientations wi th in authorial texts i n a general way and l ink the language and legacy o f their historical roots to current pedagogical spaces where pedagogic acts of language and content dwe l l . The last two quotes at the beginning of this chapter are from well-respected scholars i n education who have made significant contributions to the area of language and content (also k n o w n as C B I or content-based instruction). 2 7 They both ca l l forth a need to recast an artificial separation o f language and content they assert are prevalent i n curricular practices, particularly i n Nor th Amer i ca . They use the base word integrate to c l a i m such a space - use o f a c o m m o n language that, for many, assumes shared meaning. Hence, o f interest i n this chapter is the language and legacy that has contributed both to an articulated c o m m o n ground o f content-based instruction and to disrupted, textured traces o f fragmented orientations that have the potential to invoke dialogues o f difference regarding language and content relations. I lean heavily on the authority o f various members o f C B I communit ies i n attempting to locate some of the discordant orientations evoked in their meta-texts o f language and content. M a n y notions held about meaning and its relationship to language undergird interpretive practices l ived in content-based classrooms, especially where language (in this case Engl i sh) is both a subject o f study and a med ium of learning. Engendered f rom linguist ic theories and pedagogical texts o f author(itie)s, language and content communit ies are expanding as they gain increasing attention wi th in the fields o f immers ion and foreign language teaching. Comprised o f those who c l a im to value meaningful, contextualised learning, these communities place content at the ini t ial stage o f curricular planning and are sensitive to the language needs of learners engaged in the learning o f that content. 2 8 Ye t , wi th in this shared landscape o f complex intentions, partially constructed and di lemmatic 2 7 The work of Brinton, Snow & Wesche, (1989), Crandall (1987), Mohan (1986), Short, (1993, 1994), Snow & Brinton , editors (1997) are viewed as the main authoritative texts on language and content pedagogy. It is an area in the field of English as a second language that seems to have taken on its own signification and is viewed separate from English for academic purposes (EAP) and EAP's broader framework, English for specific purposes (ESP). 2 8 See Snow & Brinton's (1997) introduction in The content-based classroom for further discussion. 38 orientations o f evaluative practices seem to dwel l . 2 9 The purpose o f this chapter is to offer a (re)view of presumed authorial language and content texts where authors discuss their interpretations o f complex issues emerging f rom wi th in planned and unplanned second (immersion) language content-based curricular practices. I guide the reader to a number o f l inked language and content communit ies and consider assumptions embedded i n "integrated" practices - o f both cu r r i cu lum and evaluation, o f conceptions o f wri t ing content and its instruction, and o f language and content relations. 3 0 1 first present a (re)view of the literature on language and content that alerts the reader to an authored common ground. C o m m o n Ground of Language and Content Eskey (1997), i n a discussion o f syllabus design, outlines a br ief genealogy o f content-based instruction based on Stern's insightful commentary o f " two major and largely unreconci led versions o f . . . ' communicat ive ' language teaching ( C L T ) " (p. 132). In the 1970s and 1980s a shift from a dominance o f structural or grammatical designs o f language courses to a communicative syllabus was a movement simultaneously taking place both i n Europe (especially Bri tain) and i n Nor th A m e r i c a (Canale & Swain , 1980; M u n b y , 1978; van E k & Alexander , 1977). B o t h communities were concerned wi th the functionality o f language and the need to take advantage of the learners' communicat ive environment. Eskey informatively explores two divergent approaches to C L T that Stern had identif ied i n the early 1980's: a Br i t i sh-European version based i n "new" l inguis t ics (notions, functions and speech acts) wh ich he cal led the L-approach, and a Nor th A m e r i c a n version based i n psychology and pedagogy (interest i n the learner and the learning process), w h i c h he cal led the P-approach. Eskey (1997) contends, [c]ontent-based instruction is clearly a descendant o f the P-approach, in the sense that it rejects the commonsense notion that the content o f a language course should be language, (p. 133) 2 9 Work in ideology and ESL was established by the work of critical theorists and pedagogues Auerbach (1989), Cummins (1989), Peirce (1989), Pennycook (1989), and by others acknowledging "education is political and ideology is unavoidable" (Benesch, 1993, p. 715). I have found particularly useful more current work in E A P by Benesch (1993), Johns (1991), Raimes (1991a), Santos (1992), and Zamel (1992, 1993, 1997). 3 0 My purpose is to attune the reader to the term 'integration' as it is a commonly used phrase in the language and content literature. In listening to authorial texts, I wonder, how is this term used? 39 H e locates rootprints o f content-based instruction i n the Nor th A m e r i c a n version o f C L T , one he c la ims is not educed from "any k i n d o f l inguist ic analysis". W h i l e I w o u l d agree that C B I in Nor th A m e r i c a may we l l have developed wi thin a P-approach, and that notions o f what content should be frequently neglect language as a topic o f study, over the years questions o f language and content relations seem to remain haunting ones. The language across the curriculum movement in Br i t a in i n the 1970's, archived i n such documents as The B u l l o c k Report. 1975. wh ich focused on the role o f wr i t ing in subject-areas, was a shift towards the integration o f language ski l ls into content courses. The B u l l o c k Report is v iewed as the "first overt expression o f a g rowing movement away f rom the rhetorical, product-oriented wri t ing class - divorced from other subject-matter classes - toward an approach that views wri t ing as an integral part o f any course wi th in the cu r r i cu lum" (Crandal l , 1987, p. 1). It was an attempt to acknowledge the responsibil i ty o f teachers i n a l l subject areas to ensure w r i t i n g sk i l l s are appl ied to authentic tasks such as lab reports, explanat ions o f p r i n c i p l e s and theorems, d i scuss ions o f h i s t o r i c a l causes and effects, or comparisons o f religious or cultural institutions, (p. 2) The f ie ld o f reading has moved in a similar direction in arguing for reading ski l ls to be taught i n content courses and for texts o f 'academic ' substance be used i n the teaching o f reading ski l l s , journeying beyond the use of texts o f literature that have predominated traditional first-language reading classrooms. W h i l e practices i n learning to read and write shifted to acknowledge the significance o f reading and writing to learn, a s imi lar trend i n language instruction has taken place. Learning a language has become not only a study o f language but also a medium in wh ich to learn subject-matter. V a r y i n g curricular approaches to language and content consider the teaching and learning o f both language and content concurrently. 3 1 The notion o f integrating language and content had been a topic o f discussion long before it gained attention i n the f ie ld o f E n g l i s h as a Second Language ( E S L ) . Its history has been recorded by others and therefore w i l l only be highlighted here to establish a common ground. 3 2 A s w e l l , it is not 3 1 For a thorough discussion on prominent 'language and content' curricular models see Brinton, Wesche, & Snow, 1989. 3 2 See seminal works by Brinton, Wesche & Snow (1989), Crandall (1987), Eskey (1997), and Mohan (1986) that offer historical accounts of language and content. 40 m y intent to provide a rationale for the use of language and content approaches i n programs for speakers o f other languages, only to acknowledge that its use is w e l l documented i n Engl ish-speaking countries wi th high immigrant populations. 3 3 Its use is w e l l documented but little has been written that speaks to shared understandings o f integrated language and content practices, a notion explored later i n this chapter. A s w e l l , Eskey points out that few have written i n detail how a content-based syllabus for E L L ' s w o u l d differ f rom one for mainstream classes. Eskey (1997) suggests that "some o f the best w o r k addressing this particular problem is that o f the work o f M o h a n and his colleagues (e.g., Ea r ly , 1990a, 1990b; M o h a n , 1986)" (p. 133). It w o u l d seem then a shared c o m m o n ground w o u l d be found i n principles rather than practices o f language and content pedagogy. Some v iew language and content as an umbrella term for many approaches that base their teachings on principles o f meaningful learning. 3 4 Labels such as theme-based, sheltered, adjunct, and content-based language instruction are used to describe pedagogic sites for C B I . 3 5 A s has already been stated, others suggest that language and content is one o f several defined approaches to communicative language teaching. 3 6 One principle o f a c o m m o n framework is the valuing o f meaningful learning; meaningful learning that is assumed to take place when topics introduced i n class are purposeful and interesting to learners and are l inked to learners' prior knowledge. The practices o f who decides what is meaningful to w h o m is not raised. S imply , many proponents o f language and content argue 3 3 For a thoughtful discussion in support of CBI see Grabe and Stoller, 1997. 3 4 Brown (1994) in his text, Teaching By Principles, offers twelve broad principles of second language learning of which one is meaningful learning. He posits, "meaningful learning 'subsumes' new information into existing structures and memory systems, and the resulting associative links create stronger retention." (p. 18) He argues that language and content retention improve when learners are engaged in subject matter that is of interest to them. 3 5 These categories are promoted and described in detail in the authorial CBI text by Brinton, Snow, & Wesche (1989). Benesch (1992) critiques the accommodationist ideology imbricated in an adjunct model as an approach that "reinforce[s] the status of large lecture classes and the traditional pedagogy of information delivery and retrieval" (p. 1) and perpetuates the secondary status of ESL classes as a "tutoring service" (p. 8). Benesch argues for "an alternative approach to paired courses, one in which the ESL and content faculty co-develop the curriculum" (p. 1) that assumes a critical stance of educational practices. 3 6 Nunan (1991b), in outlining a number of current approaches to the field of language teaching, suggests Communicative Language Teaching (CLT) is an overarching approach characterised by teaching and learning that is interactive, uses authentic texts, attends to learning how to learn and learning how to mean, links classroom activities to the 'real' world, and brings learners personal experiences into the classroom. He discusses six closely aligned terms: learner-centered, cooperative learning, task-based, content-based (language and content) and interactive learning. These terms are also discussed in Brown (1994) as current approaches to second language teaching. 41 that E L L s increase their language learning when engaged in learning that is relevant and meaningful. C o m m o n principle, perhaps uncommon practices. Evident i n authorial texts is that language and content, as a current pedagogic approach, is a pervasive v iew concerned wi th Eng l i sh not only as the subject o f study but also as an active med ium through which subject matter is learned. M o h a n (1986), an educational linguist on the forefront o f language and content concerns, comments, T h e importance o f subject matter and content as a context for language l ea rn ing is now general ly acknowledged i n second language research. S i m i l a r l y , i n first language education there has been m u c h d i scuss ion o f 'language across the cur r icu lum' since the publicat ion o f the B u l l o c k Report " A Language for L i f e " ( B u l l o c k Commi t t ee , 1975). A s a result , it has become w i d e l y accepted that the teaching o f language should be integrated w i t h a l l aspects o f the cu r r i cu lum. " L e a r n i n g , it is n o w clear , i n v o l v e s language not merely as a passive m e d i u m for r ece iv ing concepts . Thus learn ing is not merely through language but w i t h language" ( M a r y l a n d , 1977, ix ) . (p.iii) H o w relations o f language and content cou ld be understood are offered by M o h a n (1986, 1990) i n what he calls a knowledge framework that considers how particular structures o f knowledge are constituted i n language. H i s approach to language and content, rooted i n functional l inguistics, v iews Eng l i sh as a subject o f study and as a m e d i u m o f learning through a l inguist ic lens. Other authorial texts (e.g., Br in ton , Snow, & Wesche , 1989; Cranda l l , 1987) also consider the possibil i ty o f Eng l i sh as subject and m e d i u m but through a pedagogical lens. C o m m o n purpose, differing orientations. Sheppard (1997) adds to the discussion wi th his comment that, C o n t e n t - E S L is a melange o f strategies and methodo log ies , mater ia ls and activities, pol ic ies and practices that share a c o m m o n purpose: the preparation o f E S L students for the E n g l i s h m e d i u m content c lass room through language and content integration . . . . it falls between instruction in the language and instruction through the language, (p. 30) A s a space o f instruction in and through Eng l i sh , the c o m m o n ground o f language and content seems to be a complex ground o f difference where "content-language instruction has many definitions, and classroom practices vary as widely as communit ies , schools and personnel d o " (Sheppard, 1997, p. 22). The c o m m o n ground seems to be an imaginary ideal . In describing content -ESL programs in public schools i n the Uni t ed States, Sheppard c la ims, "the ideal is for an E S L and content teacher to take on part o f the other's job : the 42 E S L teacher systematically reinforces the students' understanding o f content, and the content teacher bui lds the students' proficiency i n E n g l i s h " (ibid.) . Texts describing cur r icu lum development, instructional approaches and classroom assessment suggest the ideal remains an important goal for content-ESL programs. Yet , they also conf i rm a wide range o f perspectives and l i v ing practices o f those involved in language and content practices that may or may not successfully accomplish the ideal . Perhaps the uncertainty o f the space o f instruction in and through Engl i sh , including partial and confl ic t ing orientations o f language and content relations create a complexi ty o f issues that give rise to C B I as a di lemmatic pedagogy. C o m m o n ground to language and content pedagogues seems to be the expectation that language development and content learning be addressed in some way; it is a valuing o f the principle that language cannot be learned in isolation from content and content learning is not v o i d o f language. W h i l e most recognise the need to systematically organise language and content for pedagogical purposes, diverse epistemological and methodological roots seem to disturb the possibi l i ty o f shared common ground o f practices. Language and content, in psycholinguist ic terms, are understood as language items that are presumed to be separate f rom content items. In contrast, functional linguistic terms presume to relate language and content i n a more holistic, integrated fashion. A s contrary themes wi th in fissured texts, they have the potential to co-exist in tension. It is a site o f epistemological bits and pieces o f language and content relations wherein fragmented traces f rom various l inguist ic and philosophic theories, for the most part, lie i n silent conflict. Author(it ie)s seem to be in agreement that whenever language is used, meaning is communicated. Yet , what is impl i c i t ly contested is language and content relations. In sum, a framework for language and content seems to be shaped by the fo l lowing partially shared c o m m o n ground: 1) that Eng l i sh is the subject o f study as w e l l as the med ium through w h i c h content becomes known; 2) that meaningful learning contributes to successful language learning; 3) that languages are learned 'best' through use; 4) that language and meaning are related; and 5) that language development and content learning must both be addressed. W h i l e common themes may exist i n how 'language and content' is understood, so too, mult iple, fragmented and contrary themes become 'norms ' o f how to 43 accompl ish language and content pedagogy. Some argue for mapping separate language and content trails throughout the curr iculum, whi le others seek ways to map them on similar paths. Here in lies a di lemmatic issue o f language and content: the norms, as part o f the framework that momentarily holds still embedded fissured orientations, have the potential to be inconsistent, unstable and uncertain in l iv ing practices o f language and content pedagogy. W h a t is evident i n the literature is that "content-based curr icula is gaining prominence i n a wide range o f contexts" and "many authors refer to successful program outcomes as evidence o f its benefits" (Grabe & Stoller, 1997, p. 5). Increasing interest by educators i n the f ie ld has contributed to the legitimation o f a language and content pedagogy for simultaneous language learning and content learning wi thin academic contexts for E n g l i s h language learners. G i v e n this, orientations that both mediate and complicate authorial texts on pedagogical interventions and learner assessment for content - E S L programs are worthy of (re)view. I explore traces of evaluative C B I orientations and, in particular, those related to evaluating written content. Entailed in the pre-conditions o f integrative evaluation, often the ' f ina l ' act o f a curricular unit, are l inks to planned curricular goals, learning outcomes and instructional approaches to written language and content tasks. The remainder o f this chapter is given to listening for orientations o f authorial texts wi th in these areas. A Communi ty of Difference A l t h o u g h there seems to be an established common, shared ground wi th in the communi ty o f language and content authorities, I wou ld argue, it is also a communi ty o f difference. 3 7 In using the term difference, I am speaking to the hybr id spaces o f language and content pedagogy wherein contrary themes, fissured texts and ambiguous fragments dwe l l . In asking, what are the assumptions o f author(itie)s of language and content texts that a l low them to write what they do, partial and possibly conflict ing traces o f orientations may emerge and provide useful understandings o f language and content as a communi ty o f difference. 3 7 1 use the term difference and distinguish it from diversity - a term often used interchangeably with difference. Diversity implies 'things' as a space where separate parts make up a whole wherein parts as pieces of a complex puzzle are "unlike in nature or qualities; varied" (The Concise Oxford Dictionary, 1984, p. 281). Whereas difference, as 'no-things' in its "dissimilarity, non-identity" (ibid., p. 267), contributes to a hybrid, possibly aporetic space that cannot be separated into parts; a space wherein incomplete assumptions and partial orientations mediate and complicate a community of presumed common ground. 44 A theme o f language (as words, as text) and how it effects practices is offered throughout this chapter. Fo r example, I begin by listening to ideational traces o f how the term integration is understood and used i n pedagogic discussions o f language and content relations. Then, I attempt to track textured orientations in three other areas: 1) a legacy o f assessment that may have contributed to current conditions o f judging language and content written texts; 2) curricular and instructional practices that may have contributed to particular assessment and evaluative practices of E L L written content; and 3)pedagogic second language wri t ing conventions and advocated assessment practices that may have contributed to expectations o f the wri t ing task and how it should be judged. I begin wi th an e tymologica l search for meanings o f integration. Fissured texts embedded in integrated practices M a j o r works addressing C B I seem to al ign wi th what Hodge and Kress (1993) ca l l fissured texts; that is, fragmented bits and pieces o f discursive processes engaged i n a language and content dialogic function coll ide at the textual site, especially i n the use o f the term integration. M y purpose is not to question the val idi ty o f the substance o f these major works nor to summarise them. Instead, I consider orientations to the term integration as it is used in discussions o f how to teach and evaluate language and content wr i t ing that in form the question, where are these texts of author(itie)s located on notions o f ' integrat ion '? The Concise Oxfo rd Dict ionary (1987) offers two definitions o f the verb to integrate. The first states: "made up o f parts; whole , complete" and the second offers, "complete by addit ion o f parts; combine (parts) into a who le" (p. 521). The verb to integrate, as the dictionary states, is derived from the Latin-based verb integrare, "to make w h o l e " (ibid.) . A l though helpful i n highlighting the importance o f completeness, it is replete i n specifying whether the 'parts' remain separated wi th in the whole (as a part-whole relationship w o u l d suggest) or whether the parts are re-configured differently i n process o f integration. F o r example, i n considering language and content relations, does integration act to make language and content inseparable or is segregation maintained wi th in an ecological 'wholeness ' o f meaning-making? These take differing orientations i n l inguist ic and pedagogic discussions on relations between language and meaning. 45 It has been suggested earlier in this chapter that the broader language and content communi ty shares a common view o f language as a med ium for learning and o f the need for language teaching and learning to take place within a l l areas o f curr iculum. Integration seems to be a term frequently used to suggest a way to accomplish this shared v iew. F o r example, Short (1993) in a detailed and thoughtful description o f teacher strategies for integrating language into the social studies curr iculum, comments, when language educators integrate language and content objectives, it is often referred to as content-based ESL or content-based language instruction S o m e teachers . . . refer to this integration as thematic ins t ruct ion . . . S o m e teachers prefer to integrate only one subject area wi th the language instruction . . . Regular and content teachers also integrate language and content instruction. T h i s approach may be termed sheltered instruction i f the students i n the class are a l l E n g l i s h language learners, or language-sensitive content instruction i f the class is heterogeneous . . . . The ma in focus o f these classes is content comprehens ion; however, the teachers are often trained i n E S L techniques to make their instruction more accessible to students learning Eng l i sh , (pp. 582-3) H e r purpose is clearly pedagogical i n identifying three curricular models (similar to the three models offered by Br in ton , Snow, & Wesche, 1989) that c l a i m to integrate language and content. A s part o f a research project, Short (1993) and her colleagues dec ided to define the academic language o f socia l studies broadly to inc lude semantic and syntactic features (such as vocabulary items, sentence structure, transition markers, and cohesive ties) and language functions and tasks that are part o f social studies classroom routines, (p. 595) L ingu i s t i c terms such as semantic and syntactic features are broad and encompassing and suggest a more formal v iew o f language, while attention to language functions begins to address language i n use, an interest o f functional l inguistics. Is integration then a br inging together o f the parts - language objectives and content objectives that continue to be identifiable i n their relation wi th content or do they work to constitute meaning as an ecological whole wherein when a part is altered, the meaning (the whole) is changed? A review of the practical aspects o f the sample unit provided by Short reveals three areas for objectives written as, "language skil ls (e.g., students w i l l listen for the main idea), content ski l ls (e.g., students w i l l identify the principles [my italics] o f the Declarat ion o f Independence), and thinking/study skil ls (e.g., students w i l l classify [my italics] subtopics for an outl ine)" ( ibid. , p. 588). A l though she does not c l a i m a particular l inguist ic stance, 46 the objectives for language and for content wi th in the sample unit remain separated. 3 8 Y e t , i f students are expected to engage in these activities, how does language implicate such structures o f knowledge as principles and classification? Is this not the language o f the content objective? Br in ton , Snow, and Wesche (1989) impl i c i t ly suggest a possible separation o f language and content relations. In a discussion on language and content assessment, they posit, a further area o f overlap . . . lies in the fact that the kinds o f materials and activities used i n content instruction may also be appropriate for evaluating learners' language ski l ls . In other words, the texts and tasks w h i c h appear on language tests i n these courses may closely resemble those on tests o f content mastery; indeed, the same ones may sometimes be used for both purposes, w i th differences only i n scoring criteria, (p. 183) It seems that both texts o f authority are addressing Eng l i sh as subject and m e d i u m at a pedagogical level and the sample units, wi th language objectives contained therein, are based on imp l i c i t assumptions o f separation regarding language and content relations. If language items are separated from content items, then how is context being attended to? Wha t is the language of content objectives? A n d , conversely, what is the content o f language objectives? Does this suggest that they are work ing wi th in a formal v i ew o f language where context is not central to language and meaning? Cranda l l (1987) in an edited work, comments, "The concept o f integrating language instruction wi th subject matter instruction is not new to language educators .... [as a] combined focus - on the subject matter and the E n g l i s h that is used to communicate i t" (p. 1). A g a i n , there is no specific discussion o f l inguistic aspects o f language and content relations i n her introduction. However , in a review of the articles i n the book that have been chosen to represent examples o f language and content integration, authors use terms o f formal grammar such as vocabulary, syntax, semantics, and discourse features in discussions o f language to be developed in the content area. Li t t le , i f any, mention is given to language as a resource for meaning. C o u l d it be that these examples were written by educators that took a formal grammar position i n language and content relations? In 3 8 See Short's (1993) discussion on assessing the integration of language and content. She contends, "the difficulty with assessment centers on isolating the language features from the content objectives so one does not adversely influence the other" (p. 627). This seems to reflect a view of separation. 47 reading authorial texts that seem to integrate language and content f rom a pedagogical posit ion (not a l inguistic one), is it possible for the reader to then engage i n ideations o f language and content practices and not be aware of, or attend to, taken-for-granted assumptions o f particular l inguistic language and content relations? M o h a n (1986) also speaks to integration when he comments, L inguis t i c content is inseparable from linguistic expression. Bu t in research and i n c lassroom practice, this relationship is frequently ignored. In subject matter learning we overlook the role o f language as a med ium of learning. In language learning we overlook the fact that content is being communicated . . . . What is needed is an integrative approach which relates language learning and content learning, considers language as a medium of learning and acknowledges the role o f context i n communicat ion, (p. 1) H e speaks to language and content integration as a place where meaning is made and argues that what remains problematic in the literature is how this integration should get done. Other authorial texts do not seem to make such claims. W h i l e they provide approaches to integration at the level o f curricular design and methods and do not expl ic i t ly l ink their work to l inguist ic models, M o h a n presents an organising framework for language and content integration that purposefully provides practical and theoretical directions for educational po l i cy that is l inked to l inguistic theory. Fo r example, M o h a n , in outl ining discourses about action situations wi th in a notion o f activity, refers to Leech and Svar tv ik ' s (1975) semantic grammar to systematically include grammar, semantic notions and speech acts; it is an understanding o f lexico-grammar that considers what items o f language are used i n relation to the meanings they express; it is a v iew o f functional grammar i n wh ich context and use play a central role. It seems that most authorial voices articulate integration in terms o f pedagogy, wh ich w o u l d a l ign wi th Stern's P-approach, whi le M o h a n considers integration from a l inguist ic perspective, Stern's L-approach. W i t h i n the Nor th Amer i can language and content communi ty , not only is integration conceived i n differing terms (pedagogic and linguistic), but also differing explici t and impl ic i t fragmented orientations are being played out wi th in l inguist ics as formal and/or functional. Discourses o f integration disclose unshared reference points, and language and content relations are often not made explici t therefore contributing to language and content as a dilemmatic, or troubled, pedagogy. 48 Preconditions of Decision-Making Grades written i n insoluble black ink, a common final step i n the evaluative process, have become symbols o f complex judgments that presume to represent predetermined 'standards' o f students' work. O f interest is research that speaks to the pre-condit ions o f judgments leading to the final decision o f the grade or mark. Therefore, the fo l lowing is offered as a reflection on historical and current moments that contribute to specific discussions o f integrative evaluation. A Legacy o f Assessment and Evaluat ion Assessment and evaluation are terms that often go unquestioned i n day to day practices o f teaching and learning. They are expected educational practices we participate i n as teachers and learners. B u t what assumptions are being made? In a recent publicat ion titled Assessment and E S L (Law & Eckes , 1995), the authors discuss, but not define, the terms effective assessment and alternative assessment. Evaluation is not a key term i n this book - the term is not listed in the index nor in the detailed table o f contents. In this reference, assessment is impl ic i t ly described as a process but remains undefined; the authors make assumptions that the readers already understand the fundamental concept o f assessment, and, that either evaluation is insignificant or so integral to assessment and E S L that it is a shared and obvious known, two very different stances. Bachman (1990) i n a w e l l respected text, Fundamental Considerations in Language Testing, suggests that the terms 'measurement ' , 'assessment', and 'appraisal ' are used as synonyms i n the language testing literature and includes the term testing as a k ind o f 'measurement instrument' . A l t h o u g h test shares its e tymological roots with text (texture, textual), its current alignment wi th 'measurement' and 'instrument' suggest that a modernist tradition o f objectivity has been pr ivi leged, showing little interest in the textured lives o f learners. Bachman leans on Weiss (1972) to define evaluation as "the systematic gathering o f information for the purpose o f mak ing decis ions" (Bachman, 1990, p. 22). A brief turning o f the pages reveals inconsistent and somewhat contradictory understandings o f these taken-for-granted terms i n current references i n the field. H o w have these words come to mean what they do? A r e they the fissured texts o f w h i c h Hodge and Kress (1993) speak? A re-turn to the L a t i n and Greek origins o f these words begins to undo their legacy. 49 A c c o r d i n g to the Oxfo rd Concise Dict ionary o f Eng l i sh E tymology , assessment [assess+ment] comes from the La t in stem assidere meaning to sit beside. In M e d i e v a l L a t i n it became assedere, shifting the meaning to determining a levy, to gather information i n order to settle an amount. In modernists times, wi th the import o f scientific rationalism, educational assessment, purposeful in its preciseness, seems to have become the process o f coming to an agreement ( levying or settling) o f an presumed objective measure (amount). Revis i t ing the or iginal meaning, to sit by, one's musings are drawn into the possibi l i ty o f an imaginary o f assessment as a participatory endeavour enacted through subjective positionings between teachers and learners. Evaluat ion [e+valu+ation] from the La t in valere means to be strong, to be o f worth, to estimate the worth. It is a concept critiqued by Dav i s (1996), The business o f evaluat ion tends to assume that there is an external, objective standard against w h i c h . . . can be va l id ly and rel iably measured. In its quest for unbiased certainty, evaluation has taken on a certain mechanical character, framed by checkl i s t s (that are used to identify what someone else has ident i f ied as relevant), explici t criteria (that define not just what we look for, but what we see), and most obviously , some manner o f quantification, (p. 245) It is a va lu ing o f something that presumes the possibi l i ty o f being represented by its quantification in a mark. Davis continues, A hint o f what evaluation ' is really about' might be gleaned f rom a review o f the terms associated w i t h evaluat ive practices, such as ' m a r k i n g ' , ' s c o r i n g ' , and ' g r ad ing ' . W h i l e their metaphoric or igins have been largely forgotten i n the modernist quest for objectivity, we wou ld do w e l l to recall t h a t . . . one's marking, scor ing , and grading o f a learner i nvo lves a certain v io l ence as one leaves ' impress ions ' on that person's body. E v a l u a t i o n i nvo lves a m a r k i n g for l i fe , (ibid.) The terms assessment and evaluation have blended in meaning, becoming teaching acts invo lved i n making sense o f learner, but from diverse ends o f the continuum: assessment as a process integral to the teaching and learning activities o f the classroom, and evaluation as a judgment o f the subjective worth of a product. They are ambiguous and somewhat contradictory terms that are often used interchangeably in the current literature. F o r purposes here, assessment is used to refer to a process(es) o f gathering information to evaluate (by mark, grade and evaluative feedback) a product that presumes to represent the learner's achievement on school tasks. Products have the potential to take alternative forms, but ultimately, I wou ld argue, what gets judged is v iewed as a product o f student 50 work - a sampling o f what the student can achieve at that point i n t ime. 3 9 T h i n k i n g o f these terms i n this way, as mainstream and second language author(itie)s seem to do, establishes an incomplete discourse communi ty o f assessment and evaluation w h i c h shapes and is shaped by interacting complexit ies o f authorial texts and needs o f efficiencies i n teachers' practices. Discourse(s) becomes specialised, w o r k i n g to br ing clarity and closure to the uncertainty o f the evaluation task guided by textured orientations that have the potential to be fissured and dilemmatic; i n marking wri t ing, pre-condi t ion assumptions o f how to achieve fairness and consistency, how language works , and how meaning gets made are complex, may go unquestioned, or may be altered (re-configured) i n the well-intentioned quest to be precise and defendable i n the grading practices o f schools. Quantif ication in numbers (as marks) and letters (as grades) seems to reduce textured complexifications to images o f objectivity and clarity that " involves a mark ing for l i fe" . Educators must ask themselves what has been left un-marked - or perhaps re-marked - in a reductionist move to a numerical judgment, and ask how that wh ich was cast-away or altered also informs judgment. A n d , does the form o f evaluation (technical) a l ign wi th the phenomenon (a social and integrated text) it purports to serve? G i v e n assessment and evaluation are regular and expected practices o f teachers, one must be wary o f the traps o f discourse(s) constituting assessment and evaluation, and be mindfu l o f the e tymological and epistemological legacy that has the potential to lead them astray. Legacies also implicate current conditions in how assessment is classified. The second language literature constitutes assessment o f E L L learners broadly to include proficiency, placement, achievement, and diagnostic testing as categories for judging language competencies. E a c h wi th specific purposes, they can be located according to the cri terion they purport to measure. Proficiency tests a im to "tap global competence i n a language" ( B r o w n , 1994, pp. 257-8) and traditionally have used indirect testing to access receptive ski l ls (listening and reading), "often wi th val idi ty weaknesses: they may confuse oral proficiency wi th literacy skil ls , or they may confuse knowledge about a language wi th abil i ty to use a language." Nunan (1991a) contends that proficiency remains an abstract 3'JI acknowledge that the broader question of evaluation itself, its purpose and the presumptions that underlie it deserve a critical review. However, I follow the path already laid and accept, for purposes of this inquiry, the assumed need to quantify student learning. 51 construct and its testing procedures lack attention to contextual factors that influence learners' performance. 4" C o u l d Nunan be explor ing traces o f an orientation that resist a formal l inguist ic stance? Those charged wi th gate-keeping responsibilities re ly ing on interpretations o f scores from wel l -known proficiency tests such as T O E F L (Test o f E n g l i s h as a foreign language), Cambridge, and the M T E L P ( M i c h i g a n Test o f E n g l i s h language proficiency) may w i s h to heed the warnings o f B r o w n and Nunan and rethink the presumed possibi l i ty o f 'measuring' language v o i d o f context (content) and the effects o f " m a r k i n g for l i f e . " 4 1 Placement and diagnostic tests often focus on particular aspects o f a language such as verb tenses and article usage at the sentence level or identifying discrete sounds i n l is tening. 4 2 These forms o f assessment often tap both productive (speaking, wri t ing) and receptive (listening, reading) skil ls using direct and indirect tests. Frequently, but not always, they include sample material from the curr iculum or course the learner is to be placed i n or is having difficulty with. Sometimes proficiency and diagnostic tests are used for placement purposes. In presuming they are somewhat s imilar and therefore interchangeable, traces o f orientations found wi th in descriptions o f purpose i n these various forms o f assessment seem to view language at the sentence level , v o i d o f contextual (content) factors. Canale & Swain ' s (1979, 1980) theoretical framework o f communicat ive competence has contributed to much o f the current research on language testing. C l a i m i n g an integration o f grammatical, sociolinguistic, and strategic competencies at the level o f discourse, their framework names specific categories o f language that should be integrated. Ye t , it neglects to articulate how such integration could be accomplished. Instead, lists o f what they c l a i m to be essential elements of communicative competence are offered i n isolat ion as an approach to discourse assessment. Bachman (1991), influenced by the 4 0 For further discussion, see Nunan's (1991a) argument on the inappropriateness of 'proficiency' as a main consideration in E L L evaluation. 4 1 As well, others (Benesch, 1991; Raimes, 1990) critique large scale testing and the pedagogical decisions resulting, presumingly based on the needs of individual learners for placement, assessment, credit-granting purposes. They argue against a testing orientation that may deny equal access to a mainstream college education, suggesting such testing is exclusionary, and call for more inclusionary, democratic practices. 4 2 Offered in Brown (1994), diagnostic and placement tests are presented as mechanistic views of language at the sentence and sound level that seems to assume these are necessary and central elements to practices of language learning. 52 ideations o f Canale & Swain ' s framework, developed a model o f language competency that expands their framework, but does not alter the underlying assumptions. S imi l a r to those o f Canale and S w a i n , his assertions recognise a "dynamic interaction" (p. 4) amongst the elements but then advocate a separatist v iew applied in assessment practices. Bachman describes a movement i n the field o f language testing from ski l ls and components to a broader v i ew o f language. H e suggests there is now the recognition that communicative language use (original emphasis) involves a dynamic interaction between the situation, the language user, and the simple transfer o f information, (ibid.) Attending to the multiple, interrelated aspects involved in language use, this shift gives new direction to language testing i n the need to develop tests that not only address socio-cultural and functional aspects o f communicat ion but, as Bachman suggests, are 'authentic' , i n that they require test takers to interact wi th and process both the expl ic i t l inguistic information and the impl ic i t i l locutionary or functional meaning o f the test material, (ibid.) It seems that traces o f language as form linger whi le those o f language as function add, alter, and perhaps fragment theoretical stances i n language testing. M c N a m a r a (1996) confirms this shifting movement when he explains, language assessment is in a period of rapid change . . . where learners have to demonstrate practical command of skil ls acquired, [performance-based assessment] is rapidly replacing more traditional test formats, (p. 1) P romis ing i n its desire to acknowledge contextual factors in the functioning o f language, a change i n orientation from a dominant formal v iew, evidence o f a fissured text emerges i n texts o f author(itie)s as fields o f applied linguistics and psychometrics play out their various and different, possibly conflict ing, orientations. Bachman (1990), in a summary o f "persistent problems and future directions" i n language testing concludes, W e language testers thus cannot a l low ourselves the delusion that current views o f language use and language learning can be ignored because this s impl i f ies the p rob lems o f measurement. N o r can we afford the l u x u r y o f p l a c i n g a l l our confidence i n the correctness o f our applied linguistic theories, i n the hope that the measurement issues w i l l thus evaporate, (p. 352) The complexificat ions o f language testing dwel l i n competing meta-texts and orientations. In attempts to quantify language for measurement v i a psychometric routes, language seems 53 to have become a mechanistic tool. In a desire to acknowledge the importance of context, measurement questions of validity and reliable are raised. Contrary themes have created "persistent problems" and can be traced to ideological dilemmas. That the field calls itself language testing, that the language of language testing can be traced to ideas of preciseness and control, embeds assumptions of the possibility of a test that can measure an 'objective' language.43 It is a tattered trace from a dominant regime that has infiltrated not only the work of language testers but has found a way in to classroom assessment. One articulated purpose of classroom assessment is to assess the students' progress or growth as they participate in curricular activities generated from the classroom. Hence, the term classroom assessment is used to describe the process of evaluating students' achievement of tasks (tests and assignments - both formal and informal) as they pertain to teaching and learning activities in the classroom. Angelo and Cross (1993), respected for their work in 'mainstream' classroom assessment, describe classroom assessment as an approach designed to help teachers find out what students are learning in the classroom and how well they are learning it. This approach is learner-centered, teacher-directed, mutually beneficial, formative, context-specific, ongoing and firmly rooted in good practice, (p. 4) Further, they comment that much of classroom assessment is a subconscious and implicit process .... Teachers depend heavily on their impressions of student learning and make important judgments based on them.... Consequently, the most effective times to assess and provide feedback are before the chapter tests or the midterm and final examinations, (p. 7) Angelo and Cross's intent of classroom assessment is to improve the quality of student learning and provide greater accountability of the teaching and learning process in the classroom, a view also espoused in authorial texts on assessment for second language learners (see Genesee & Upshur, 1996; Law & Eckes, 1995; O'Malley & Valdez Pierce, 1996). In this linear model, both teachers and students become actively involved in the assessment process, a kind of "micro-level, grass-roots assessment movement" (Angelo & 4 3 Although the field of language testing acknowledges a dynamic interaction or integration of discrete elements of language and has established frameworks for assessing communicative competence and language performance, the discrete elements remain discrete rather than integrated as new trends in language testing would suggest. How to assess integration seems to remain ambiguous and unreliable for many. Slater (1998) discusses this in detail in Evaluating causal discourse in academic writing and suggests a functional approach such as offered by Halliday and Martin (1993) that offers the how of integration. 54 Cross, 1993, p. 8) that hints of a technical imaginary wherein "midterm and final examinations" would measure progressive and cumulative knowledge determined by the teacher. Issues with traditional forms of assessment such as paper and pencil tests has led to a re-forming of assessment practices that work toward authenticity and claim alterity.44 Educational reform is currently dominated by discussions of assessment reform.45 That reform, named as alternative and authentic assessment practices, has infiltrated many classrooms including those with English language learners, and, although not a focus of this (re)view, is important in that the re-forming of assessment seems to address only some of the concerns of traditional assessment practices.46 What seems to remain an unexplored question within authentic and alternative assessment practices is the specific evaluation of language and content relations. I lean on Resnick and Resnick's (1991) description of various types of assessment being proposed for mainstream classroom assessments: performance and portfolio. The history of performance assessment as it relates to use in second language contexts is well documented.47 It is a form of assessment that is based on integrated, holistic tasks closely resembling, if not actualising, real tasks. For example, a student performing an oral presentation to a 'real' audience would be evaluated on their ability to synthesise a complex integration of skills and competencies such as fluent and accurate use of the language, effective public speaking and knowledge of the topic - skills and competencies that had been developed during classroom activities, a framework complicit to a 'learning outcomes', means-ends model of curricular instruction. Portfolio assessment differs from performance in that students are involved in gathering a collection of their work completed over a period of time and involved in a self-monitoring of their progress through reflection and self-4 4 The terms alternative and authentic are often used interchangeably in the literature, although alternative has frequently been the broader term that encompasses authentic assessment practices. In this dissertation, I follow a classification offered by Resnick & Resnick (1992) that, I believe, uses the terms interchangeably. 4 5 In Assessing integrated language and content instruction , Short (1993) offers a brief but thoughtful overview of assessment reform as it pertains to broader educational perspectives, and the reform's influence on new trends of the assessment of language minority students. 4 6 See O'Malley & Valdez Pierce (1996) for a thorough discussion of alternative, practical approaches for classroom teachers as of a move to authentic assessment practices for English language learners. 4 7 See McNamara (1996) for an overview of the historical development of performance assessment practices that led to their inclusion in the field of second language testing. 55 assessment of their work.48 This form of assessment has been adapted from traditions in professions of art and design and is a shift from a means-end orientation to more of an emic view of evaluation. These terms and their definitions as outlined above have been operationalized in various and diverse ways in the second language assessment literature. What I find troublesome is that in an attempt to shift from traditional, mechanistic ways (hence new trends are named 'alternatives'), the process has been altered but the evaluation of a product that lies at the root of a graded task is left unattended by authorial texts when describing these alternative positions. For most, whether they use performance, or portfolio assessment practices (or in some combination), when called upon to evaluate the process or the product, there seems to be a presumption that determining a mark or a grade is unproblematic. It is not my intention to enter into a discussion of alternative practices in detail, only to outline common, current approaches advocated by authorial texts and to acknowledge that the act of marking and grading 'language and content' within alternatives approaches to assessment, orientations are presumed and unquestioned. Curricular approaches to assessment as outlined above are helpful reminders of the desire to shift from traditional forms of assessment that seem to isolate and reinforce skills detached from 'real' life, and move to forms of authentic assessment that attend to the development of complex, integrated abilities learners use to thoughtfully address issues in their own lives. However, within the 'how to' descriptions of these various forms of authentic assessment, little attention is given to the actual act of evaluation. It seems to be a promised land, full of presumptions that teachers will know what to do when they arrive. Assessment seems to remain privileged over evaluation in most theoretical and practical discussions of judging ELLs' achievement in academic classrooms. In (re)viewing authorial texts, I have come to be wary of the promised land and suggest instead that it may be a dilemmatic terrain. Given my interest in the pre-conditions of evaluating second language written content, either in traditional or more authentic forms of assessment, I first turn to the small body of literature that specifically addresses language and content classroom assessment. 4 8 For a thoughtful discussion of writing portfolios in first language classrooms see Murphy & Smith (1992). For a review of second language directions in portfolio assessment see O'Malley & Valdez Pierce (1996). 56 Language and Content Class room Assessment The literature addressing specific concerns o f judging language and content tend to emphasise various processes and strategies of collecting evidence o f learner development i n language and content. It has been m y experience that teachers (and I include myself) read the ideas offered in authorial texts with interest, yet when implemented i n the classroom, st i l l struggle wi th evaluation - justifying a mark or grade for a particular task, be it, for example, a portfolio col lect ion or individual pieces of work wi th in the portfolio. G i v e n that the necessity to mark or grade learners' work is a dominate orientation in the broader society, I want to give space to a discussion of how this is dealt wi th and where authorial texts may be posit ioned. One o f the first authorial texts on language and content to give a chapter to this topic was by M o h a n (1986). In the introduction to the chapter, he offers, It should be apparent that each type o f test should only test what it c l a ims to. It should not include, intentionally or otherwise, areas outside its purview. Language tests should test language and content tests should test content. B u t what seems apparent is not so easy to accompl i sh i n practices: language is in ter twined w i t h content. There is obvious ly a language factor i n content tests, because it is not poss ib le to understand content questions wi thout an unders tanding o f the language they are written in . (pp. 122-3) H e then asks, "is there a content factor i n language tests?" and draws attention to the possibi l i ty o f cultural knowledge and its contribution to test bias. It is the beginning o f a discussion that considers differences in semantic (language) and factual (content) inference factors i n test items and implications of bias for learners. H i s posi t ion, founded on an L -approach, c la ims semantic and actual (cultural) inferences as a necessary condi t ion for addressing language/content relations. A few years later Br inton, Snow, and Wesche (1989), i n another seminal text, Content-based second language instruction, also included a chapter on content-based evaluation. The introduction begins with preface to curricular concerns i n the statements, Evaluat ion is an integral part o f the teaching process, and whi le it has some precourse and end-of-course functions . . . its main role i n instructional programs is to measure achievement - to see whether students have mastered course material, and to identify areas where they need additional help. W h e n it is seen to support the objectives o f the course, to be relevant to students, and to bo, fair, it has a positive impact i n an instructional program. T o be effective, the ongoing evaluation o f student performance should include clear communicat ion 57 with students about course goals, the instructor's expectations, and the criteria to be used i n judging student performance, (my emphasis, p. 181) Not ions o f evaluation evolve from a strong P-approach to content-based instruction. N o t only do curricular concerns seem to dictate the conditions for evaluation, its orientation to measurement, reliability, mastery, and linearity, and its attention to course goals, objectives, instructor expectations, and criteria can be traced to aspects o f the dominant means-end perspective. Wha t is suggested i n their chapter on evaluation is that evaluation practices, seemingly framed wi thin technical rationality, can be determined by considering what " i n its broadest terms - language or content ?" and " h o w " o f content-based second language instruction. In a detailed and thorough explanation o f the "what" and " h o w " , the authors offer multiple examples o f test materials, each specifying evaluation objectives o f functional ski l ls , text/discourse type and task. The P-approach to evaluation was evident as evaluation practices were pr imari ly addressed i n curricular terms. Under a sub-heading i n the chapter, "Eva lua t i ng content knowledge" , the "what" seemed to be an assumed "subject matter" and its mastery, whereas under another subheading, "Evaluat ing language knowledge and s k i l l s " , the "what" was defined as "knowledge o f elements o f the l inguis t ic code ... knowledge o f discourse . . . interactive communicat ion skil ls ... academic language use ski l l s ... related study s k i l l s " ( ibid. , p. 186). The sub-headings presume the poss ibi l i ty o f a separation o f language and content and the items listed as the "what" o f language seem to conf i rm that separation. Traces o f orientations seem to suggest a linear v i ew o f cur r icu lum that is embedded i n a model of linearity (Tyler, 1949), and its possibi l i ty o f mastery, i n w h i c h objectives are a means to guide teachers to specific ends for the learner. C o u l d it be that a need for clearly articulated objectives could impose a particular way o f thinking about and evaluating language and content relations? S imi la r ly , Short (1993), in Assessing integrated language and content instruction, takes a P-approach i n ca l l ing for "organising assessment objectives" (p. 627). Traces o f a technological imaginary are evident i n the establishment o f objectives and by c la iming to be assessing instruction. Thoughtfully detailed, she advocates alternative assessment practices to address concerns raised i n content-based instruction and, i n provid ing an assessment matr ix o f the "what" and " h o w " , suggests, 58 [o]verall , assessment should be v i ewed hol i s t ica l ly but i n an integrated language and content course, where students are asked to demonstrate knowledge and abil i ty i n several areas, it is important to separate language issues f r o m subject-area concepts, ( ibid. , p. 635) E x p l i c i t l y here and i n the details that fo l low i n the text, she confirms a separatist's (formal) stance i n addressing language and content relations. In an article i n a current edited text, The content-based classroom. Turner (1997) writes guidelines for teachers i n the development o f content-based language tests. In the mode l she offers stage one requires "c lar i fy ing language instruction goals" and stage two "c lar i fy ing content instruction goals"; it is a v iew that expl ic i t ly separates language and content. The mode l reflects "the iterative nature o f the test-writing process" that w i l l " a l l ow, i f necessary, a return to Stage 1 for clarification o f the instructional purposes o f a program." ( ibid. , pp. 188-9). A g a i n , evaluation, and in this case testing, is placed wi th in a P-approach, focused on instructional goals and purposes that suggest traces o f a means-end orientation to teaching. A n objectified, means-end orientation seems to impose a need to objectify language and content that may impose its separation. Wha t seems to be cal led for is the clarity and precision o f language and content, further traces o f a powerful technical imaginary at w o r k i n ensuring the possibil i ty o f closure (definability) to what is language and what is content. In a discussion of scoring procedure descriptions o f essays items, she suggests that language and content relations be scored "using a holist ic approach ... grammar (5 points); vocabulary (5 points); mechanics (5 points); and content (5 points)" (ibid. , p. 192), providing further evidence to support a formal, objectified v i ew o f language. I wonder i f teachers using this approach to test design w i l l s t i l l struggle wi th the "what" o f an essay - what is language and what is content. In another article on language and content assessment i n the same edited text, W e i g l e and Jensen (1997) speak to the "interaction of language and content" and c l a i m to "expand on the ideas presented i n " (pp. 201-2) Br in ton , Snow, and Wesche (1989) and Turner (1997), impl ic i t ly stating that their foundation is one o f a technical imaginary that embeds a formal v iew o f language. It is a work thoughtful i n forwarding new trends i n testing wi th in a means-end orientation. Assessment and evaluation are complex areas, especially when learners are studying 59 i n another language. Author ia l texts are pioneers i n their o w n right, provid ing scholarship and leadership wi th in the language and content landscape. Predominant in those that took a P-approach was a technical imaginary that seems to dictate a need for an objective stance on language and content relations. The one authorial text that took an L-approach seems to raise concerns related to a functional v iew and the integratedness o f language and content. Meta-texts o f fissured pedagogic and linguistic orientations are evident i n the literature. However , notions o f literacy and how 'reading and wr i t ing ' are understood do not seem to be directly addressed. Is text presumed to be autonomous and objective? F o r those taking a P-approach, where traces of a technical imaginary dwel l , text seem to be conceptualised i n this way. Ye t , taking an L-approach, M o h a n (1986) seems to be one authorial text that raises cultural questions o f how text is understood, a trace that resists the autonomy of texts, and suggests a different orientation to literacy. H o w authorial voices advocate the judgment o f language and content texts cannot be understood without consideration o f pedagogic conventions that attempt to simultaneously address the development o f language and content. W h i l e many explore language and content integration through snapshots o f classroom activity, I turn to those texts that offer detailed syllabus designs and descriptive examples o f how integration is presumed to get done. 4 9 Several advocate the use o f M o h a n ' s (1986) Knowledge F ramework ( K F ) and its associated knowledge structures to develop language and content relations. 5 0 Ea r ly (1996), well-respected i n her work in language and content pedagogy, suggests, "according to M o h a n , each o f these knowledge structures has unique or distinct l inguist ic features w h i c h set it apart structurally from the others. In addition, each of these distinct structures can be graphical ly displayed i n 'key v i sua l s ' " (p. 25). She argues that graphing content lowers l inguist ic demands and al lows learners to explore content whi le developing associated language. Tang (1997) also explores in detail the K F for purposes o f "systematical ly 4 9 More recently Kamhi-Stein (1997) advocates the co-development (ESL teacher and faculty) of multi-step assignments "to insure instruction in the academic language skills required" (p. 52). The detailed multi-step model seems to emphasis organisational writing skills (a P-approach) not language (an L-approach). 5 0 A number of examples of detailed curricular approaches and unit plans that use Mohan's (1986) knowledge framework to plan integrated language and content activities have been published. Common to all of these curricular innovations is addressing 'language and content' relations at the level of discourse, based on a functional linguistic model of language. See Dunbar (1992); Early (1990a, 1990b, 1991, 1996); Early & Tang (1991); Fairhall, (1991); Tang (1991, 1994, 1997). 60 integrating language and content" (p. 69). She advocates the use o f graphics and offers many examples o f how graphics become central to this pedagogical form o f developing language/content relations. For example, Tang explains that as part o f a social studies unit i n h igh school , "the teacher had taught the knowledge structure o f cause-effect and exposed the students to cause-effect graphics. She had also repeatedly pointed out the l inguist ic devices and g iven the students practice in constructing text passages f rom graphics" (p. 76). Others have engaged i n graduate work that employed the knowledge framework as an organised tool for integrating language and content. 5 1 It is a curricular approach rooted i n contextualised lexico-grammatical constructions wherein a dominating L-approach meta-text wi th a functional orientation to language and content relations is fundamental. Other strategies for curricular approaches to language and content development are pr imar i ly founded on the early work o f Crandal l (1987) and Br in ton , Snow, & Wesche (1990). 5 2 The i r work offers innovations developed wi thin a curricular framework (Stern's P-approach) and, as has already been stated, seems to map a means-end approach to cur r icu lum development. In so doing, language and content relations, according to those texts o f authority, seem best addressed separately. Here, language is pr imar i ly v iewed f rom a formal perspective, separate from content, wherein, as Short (1993) reminds us, "the objectives o f an integrated language and content course can be d iv ided into the fo l lowing categories: problem solving, content-area skil ls , concept comprehension, language use, communica t ion ski l ls , individual behaviour, group behaviour, and attitude" (p. 635). Advoca ted is a linear process that works toward goals and separate objectives o f language and o f content established i n the planning stages o f curricular development. It appears that although classroom implementation o f C B I is multifarious, there are two main approaches that are evident i n the literature: Stern's P-approach and L-approach. Those engaged i n a P-approach seem to be guided by particular orientations to curricular practice and strategies advocating a technical imaginary that imposes itself on how language/content relations and literacy are understood. L-approaches to language and 5 1 See graduate work based on Mohan's Knowledge Framework and /or knowledge structures that has been completed at The University of British Columbia: Dempsey ,1994; Grant, 1995; Helmer, 1995; Low, 1989; Sampson, 1998; Slater, 1998. 5 2 See the work of Oxford (1993), Saville-Troika (1991), Short (1991, 1993, 1994), Snow, Met, & Genesee, (1989) for a P-approach to curricular practices of CBI. 61 content, rooted i n functional linguistics, embed traces o f means-end curricular practices but resist a technical imposi t ion on linguistic understandings o f language and content relations. F o r the most part, assumptions o f wri t ing seem to remain static and f ixed. H y b r i d orientations wi th in this community of difference relate to how curr iculum and evaluation are understood and how relations o f language and content are implicated. W h a t seems to be presumed undilemmatic is the act of wri t ing and the possibil i ty o f univocal i ty. I turn now to an influential movement, Eng l i sh for Specif ic Purposes (ESP) to further explore orientations to wri t ing and assessment i n the academic classroom. Presumptions o f W r i t i n g and Assessment W i t h i n E S P / E A P M u c h o f what has been written regarding C B I classroom assessment as been influenced by authorial texts on wri t ing assessment wi th in Eng l i sh for Specif ic Purposes and more specifically, Eng l i sh for Academic Purposes ( E A P ) . Johns (1997), a we l l - known author(ity) i n the E S P movement suggests, E S P practitioners are, for the most part, researchers - comple t ing text and genre analysis , needs assessments, and other studies before des igning their cur r icu la . O n the other hand, C B I practi t ioners seem to focus a lmost e x c l u s i v e l y on pedagogy, d i scuss ing student affect, ins t ruc t ional strategies, and c l a s s r o o m models, (p. 366) Situated at San D i e g o State Univers i ty , she further comments, "I teach i n an E S P program at m y o w n university, and I find the pedagogical contributions o f C B I valuable to m y pract ice" (ibid.) , and then claims in a footnote on the same page, "I ca l l it ' E S P ' because that's m y background; however, Br in ton , Snow, and Wesche (1989) w o u l d identify the adjunct (l inked) model as C B I . " Maintenance boundaries arbitrarily set i n the labels E S P and E A P are temporarily reworked by Johns. Whether E S P or E A P , her readings o f C B I support Stern's designation o f the P-approach and much o f its modern heritage i n N o r t h A m e r i c a n contexts. Concerned wi th 'wr i t ing ' as something to teach and then assess separately f rom the content it constitutes, those interested in Eng l i sh for Specific Puiposes and, wi th in that domain , Eng l i sh for Academic Purposes, contribute significantly to the current language/content condition. G i v e n the interest o f this inquiry is wri t ing wi th in academic contexts, the selected texts for (re)view are from authors who c l a i m a background i n E A P . 62 Z a m e l (1992), an authorial voice on E A P writ ing, offers that wi th the advancement o f process wr i t ing i n first language contexts, "[i]t has become commonplace to characterise the act o f wri t ing as a meaning making, purposeful, evolving, recursive, dialogic, tentative, f lu id , exploratory process" (p. 463) . 5 3 Th is statement suggests a re-posi t ioning i n its fissured break f rom the pr ivi leging o f a technical rationality i n the E A P classroom. It is the process wr i t ing movement that has questioned the validity o f traditional models o f o f wri t ing and its evaluation and turned the f ield towards alternative, authentic assessment practices as discussed earlier in this chapter. 5 4 Ye t , I wonder i f this is l i ved i n the content c lassroom - a place wherein content and how it is constituted becomes the focus o f judgment. I suspect, from reports o f collaborative work between language teachers and content teachers, that technical rationality sti l l remains strong. 5 5 H a m p - L y o n s and K r o l l (1996), both well-respected second language specialists, write o f current issues i n E S L wri t ing assessment. 5 6 In the introduction o f their article, they speak to variables that influence the complexifications i n designing "appropriate and excellent measures o f non-native writers' Eng l i sh language wri t ing competencies . . . to assure opt imal ly fair testing practices" (p. 52). W h i l e it is w e l l intended i n its purpose, I 5 3 Other views comment on shifts in ideology and ESL writing. Johns (1991) offers a current description of three orientations to ESL composition theory: process approaches - cognitivism, expressivism; interactive approaches; and social constructionist approaches. She posits that writing teachers can benefit from exploring underlying assumptions of praxis and argues that current, incomplete ESL composition theories "must include, at the very least, the four elements mentioned by Berlin (1988) [writer, the reader, reality and truth, and sources of language], in addition to other features necessitated by the nature of second language learning and use (e.g. contrastive rhetoric)" while acknowledging "no single, comprehensive theory of ESL composition can be developed on which all agree" (p. 33) 5 4 Hamp-Lyons (1994) argues for the interweaving of assessment and instruction in a variety of forms and practices. 5 5 For discussions on collaborative teaching and assessment practices in the content classroom see Benesch, 1992; Dempsey, 1994; Fradd & Hudelson, 1995; Gee, 1997; Helmer, 1995; Mohan & Low, 1995; Short, 1994; Swain & Miccoli, 1994; Tang, 1994. It seems from many of these discussions that collaborating language teachers and content teachers maintain their 'identity' even in the evaluative process - the content teacher marks the content, the language teacher marks the language - a separatist perspective within a traditional framework that imposes the importance of presumed measurable and objective words. 5 6 Hamp-Lyons, editor of Assessing second language writing in academic contexts, is known for her extensive inquiry into L2 academic writing assessment and Kroll, editor of Second language writing: Research insights for the classroom, is known for her work in L2 writing pedagogy. I (re)view this joint current text as a respectful, authored reflection on the current state of affairs in E A P writing and assessment frameworks and therefore seek its orientations extensively. Its bibliographic reference contains the authorial texts that I had listed in the original search I prepared for this section of the chapter. It is not my purpose to agree or disagree with the ideas presented in this text but to trace the orientations presumed in the authorial voices of these writers. I include other references as appropriate for this discussion of writing assessment frameworks. 63 question its suggestions that the complexifications o f competency can be measured and that "appropriate and excellent measures" can "assure fair practices." Traces o f a dominant technical imaginary permeate the text that seem to become a quantifiable imposi t ion on the complexi ty o f judging wri t ing. In speaking to the entanglements involved in E A P wri t ing assessment, H a m p - L y o n s and K r o l l cr i t ical ly review four different approaches to wri t ing assessment currently i n use, rais ing questions o f the 'unsaid ' i n wri t ing assessment frameworks. The i r concerns o f the implicat ions o f topic and time constraints, the role o f revision, the integration o f ski l ls wherein source texts are used to generate written response, and, the role o f talk (oral/aural ski l ls) are si lenced i n assessment tools that purport to measure ' o n l y ' wr i t ing . Traces o f literacy as social communication, those active mediations of socially constituted meaning, are raised i n concerns o f the multiple texts and interpretations invo lved i n wri t ing impl ied above. Thei r ca l l for test developers to attend to the subjective, the social , suggests a shift f rom text as autonomous and objective - a technical v iew - to a stance that possibly implicates a radical hermeneutics of contextual, textual and intertextual considerations. Yet , this shift remains wi th in a paradigm where the ultimate goal o f developing "appropriate and excellent measures o f non-native writers ' Eng l i sh language wri t ing competencies ... to assure opt imal ly fair testing practices" (1996, p. 52) is a possibi l i ty . Contrary themes col l ide , but ultimately it seems that the need for measures insists on reducing subjectivity to a 'marked ' object. H a m p - L y o n s and K r o l l "search[] for an idealised model for academic wr i t ing assessment" (p. 55) and continue to raise concerns o f a social nature. F o r example, i n seeking useful understandings o f what wri t ing ski l ls are needed in the w o r l d o f academia, they ask, " H o w are ski l l s social ly constructed?", " A r e they the results o f expressive and cognit ive processes that are relatively de-centered from specific academic tasks or expectations?", and " D o they require a social (re)construction o f the wri ter ' s o w n cognit ive and expressive processes to conform to genre or to faculty expectations or convent ions?" These questions seem to l ive the tensions played out between a l inger ing technical imaginary and an emergent social, pragmatic one. Further issues o f contextualization render a single theory o f E A P composi t ion and 64 an ideal assessment framework near impossible (Hamp-Lyons & K r o l l , 1996; Johns, 1990, 1997; Santos, 1992). Divergent views on second language (L2) wr i t ing instruction are shaped by broader v iews o f curr iculum, pedagogy, education and phi losophical musings o f language. 5 7 Assessment, integral to the wri t ing curr iculum but often designated as endpoints, is shaped by that wh ich comes before i n the planning and implementation o f E A P wri t ing curricular practices in the classroom. Diverse assessment frameworks then, are often reflections o f diverse curricular planning rather than a unified curricular plan and framework for E A P wri t ing instruction and assessment. Yet , silent i n this literature are the voices o f test-takers, a signpost that privileges the assumed expertise o f the teacher - traces of an instructional, technical imaginary o f education. Texts on wri t ing across disciplines contributes to emerging theoretical and pragmatic fragments that give import to "situated responsiveness" (Hamp-Lyons & K r o l l , 1996, p. 57), fluctuating reader-writer-topic-language interactivities imbricated by epistemological assumptions imbued in wr i t ing content. 5 8 A l t h o u g h a decade o f theoretical and pragmatic discussions o f E L L process wri t ing (see for example, Raimes, 1987; Santos, 1992; Spack, 1988; Zame l , 1993) legitimate its practice, Santos (1992) argues that the traditional model o f product-oriented, text-centered E L L composi t ion practices continue to dominate current E A P classrooms. In c la iming the pr iv i leging o f a scientific model for second language research that filters into c lassroom practices, she argues Science . . . has been virtually untouched by leftist theory, and it is partly for this reason that linguistics, applied linguistics, and T E S O L , o f w h i c h E S L wri t ing is a branch, have . . . m o d e l l e d ] themselves on the sciences i n their research methodology, a methodology which has as its foundation an idealised adherence to neutrality and objectivity, (p. 8) A c c o r d i n g to Santos, technical rationality remains rooted i n current pragmatic-oriented E A P 5 7 See Blanton, 1995 for a discussion on how the multiple and diverse ideations on teaching L2 writing influence curricular planning and classroom practices. 5 8 See Atkinson & Ramanathan, 1995, Leki & Carson, 1994, and Swales & Feak, 1994 for their authorial texts on the importance of contextualization and the particular "situated responsiveness" at their respective sites of inquiry. Each text explores aspects of the distinctiveness of expectations and requirements of writing in various disciplines at differing levels of study, suggesting that, although distinct variations exist across disciplines, there is a linearity and formulaic application that is both 'expected' and 'required' for each discipline. Early on, Spack (1984) and Zamel (1983) argue against a reductionist view of writing and claim writing to be individual and recursive. In later discussions Spack (1988, 1997) and Zamel (1993, 1997) continue to disrupt linear and formulaic notions of E A P composition. 65 composi t ion pedagogy. 5 9 However , Benesch (1993) argues that even pragmatics is imbricated by what she calls "accommodationist ideology, an endorsement o f traditional academic teaching and o f current power relations i n academia and i n society" (p. 711). Over the past decade, E A P wri t ing research has continued to describe and quantify important elements o f wr i t ing assessment. H o r o w i t z (1986c) explored types o f professors' assignments, wh i l e Santos (1988) and Johns (1991) inquired about professors' responses to E L L wr i t ing . Other aspects o f wri t ing assessment such as influences o f the wr i t ing prompt (Hamp-Lyons & K r o l l , 1996; K r o l l & R e i d , 1994), topic types (Reid , 1990), and topic development ( M c K a y , 1989) are part o f the L 2 composi t ion literature. K r o l l and R e i d (1994) discuss the role of the prompt and a number o f variables that affect the learner's response such as context, content, l inguistics, rhetorics and evaluation. Thei r suggested prompt design guidelines are intended to avoid a poor ly developed p r o m p t . . . that . . . can result i n unfinished, unfocused, or rambl ing essays, many o f wh ich may betray the writer 's misinterpretation o f the quest ion. They also may prove diff icul t to score according to the cr i ter ia test developers had i n mind . (p. 248) A r e they i m p l y i n g that well-developed prompts contribute to finished, focused and precise wri t ing i n another language and culture? Strong traces o f a technical imaginary are evoked i n assuming the possibi l i ty o f these characteristics o f wri t ing, doubled i n languages and cultures, and i n assuming that the more well-developed the prompt - more precise and direct - the more accurate assessment and evaluative practices w i l l be. A l though much has been written on prompts and their importance for assessment, few consider the assumptions underlying their well-intentioned contributions to E A P wri t ing assessment and their effects on a marking-for- l i fe . 6 0 A number o f authorial texts on E A P wri t ing recognise the inherent difficulties i n wr i t ing across at least two cultures. L e k i (1992) comments on the possibi l i ty o f Nor th A m e r i c a n teachers not having access to learners' experiences and assumptions they explore 5 9 Benesch (1993) responds to Santos' claims by suggesting "L2 composition, like all teaching and research, is ideological whether or not we are conscious of the political implications" and that "there is already a substantial body of ESL literature . . . that foregrounds ideology by studying the sociopolitical context, including L2 composition" (p. 706). She implies, but does not state, that sociopolitical practices are breaking with the technical. 6 0 For a fuller discussion on the role of E L L prompts see Hamp-Lyons & Kroll , 1996; Hamp-Lyons & Mathias, 1994; Polio & Glew, 1996. 66 i n their wri t ing. Wha t a teacher may deem an inappropriate response to a topic socio-culturally may have been very appropriate to the learner's first culture experiences. R e i d (1994) writes o f the "myths o f appropriation: the exclusion o f the social context i n wr i t ing , both i n the c lassroom and i n academic discourse communi t ies" (p. 273) and raises her concerns o f "not differentiate [ing] intervention from appropriat ion" (ibid.) i n instructional practices. A l though her concern is l inked to the possible disempowerment o f the learner through acts o f appropriation, her explicitness o f referring to social contexts are traces o f an orientation to a complex, socio-cultural position on literacy. W r i t i n g as cultural (worldview) difference(s) has become a commonplace grammatical nominal izat ion o f culture as " thing". 6 ' Knowledge as wor ldv iew, as k n o w i n g the wor ld , as knowledge o f the wor ld (e.g., subject matter, content), is a master signifier i n a technical imaginary of education wherein there is a presumed possibil i ty o f a cumulative, universal mastery by a l l those who come to know. 6 2 Ye t , not a l l function wi th in a system o f knowledge as mastery and universal. Knowledge , how it is systemised and its purpose i n society, differs among various groups (Bal lard, 1986). Cul tura l difference is most obvious for this discussion on E A P wri t ing. W h e n the writer and the reader do not share the same cultural background assumptions, tensions emerge. In studies by B a l l a r d and C lanchy (1991), Basham and K w a c h k a (1991), and H a m p - L y o n s (1991), culturally disjunctive attitudes to knowledge were identified as a basic difficulty i n the evaluation o f E A P students' work. Basham and K w a c h k a (1991) convincingly argue that some cultures (e.g., those cultures that do not have a written language) use Eng l i sh in innovative ways to construct and perform their native identity and that these ways should not be presumed to be incorrect or less appropriate because they do not conform to the cultural expectations o f the teacher. Th i s brings into question: whose cultural assumptions should standards be based on when readers judge student writ ing? The writers? The readers? Bo th? H o w do we come to k n o w " g o o d " wri t ing? F o r many, the messiness o f cultural difference disrupts 6 1 Those interested in exploring this notion further should read D. Aoki's (1996) musings on The thing of culture. A nominalization of culture becomes an act of categorising - generalising - and counting -objectifying - that which is subjective. 6 2 For a fuller discussion on the (im)possibility of cumulative, universal mastery see Usher & Edwards' (1994) chapter on 'Knowing oneself: subjectivity and mastery'. 67 and interferes. 6 3 The intertextual tensions remain entrapped by the border crossings o f culture as " th ing" . The term interlanguage introduced and developed by Selinker (1972) is a s imilar s imulacrum. P lac ing separated cultural ' things' at opposite ends o f a cont inuum, the technical " th ing" and a l l its boundary maintenance, is at once a desire and an obstacle, wherein m o v i n g learners from writ ing i n one culture to wri t ing i n another requires movement through a space o f interlanguage where the two ' things' mutual ly disrupt and interfere w i th the act o f wri t ing in the target language. The presumed work o f the E A P teacher then (and, I suggest here it is assumed o f al l E S L teachers) is to engage in a linear action o f m o v i n g the learner qu ick ly through this ' temporary' phase to a targeted place cal led ' E n g l i s h ' (Brown , 1994). The imposed structures o f a dominant technical imaginary once again seems to force educators to act i n particular ways. The implicat ions for evaluation seem obvious - judging a objective, autonomous text can be more easily justif ied. 'Er rors ' become labelled as first language interference and are then worked on to be erased or concealed in order to produce an assumed univocal E n g l i s h text. Could ' in ter language ' be one o f l i fe 's or iginal difficulties, the possible space for a radical hermeneutic pedagogy, or has the technical imaginary maintained its dominance and focused L 2 wri t ing pedagogy on the technical imaginary and its confinement o f becoming writers 'just l i ke us '? L e k i and Carson (1994), concerned wi th learners' perceptions o f wr i t ing instruction, found that "at a time when many wri t ing teachers are expanding the content o f E A P wri t ing courses to include crit ical thinking as w e l l as a focus on the heuristic functions o f wr i t ing" , learners "persist i n t rying to turn us back into experts on language, hence their requests for help in increasing their vocabulary and their insistence on having a l l sentence-level errors pointed out to them" (p. 91). 6 4 A s teachers begin their o w n resistance to a mechanical model o f academic wri t ing and shift into the complexi ty o f textured social relations o f cri t ical praxis, learners' felt needs remained on a foundation o f a 6 3 For selective readings on concerns of cultural interference in evaluation see Ballard & Clanchy (1991), Basham & Kwachka (1991), Hamp-Lyons (1991), Hinkle (1995), and Severino (1993). 6 4 Two references from Leki & Carson (1994) - Benesch (1993) and Raimes (1991) - offer a thoughtful discussion on shifting paradigms in L2 writing, paradigms that, in the Leki & Carson study, learners seem to be resisting. 68 f ixed possibil i ty o f technical, in-structured, autonomous givens o f what it means to do academic wri t ing. Th is is complexif ied by expectations o f undergraduate students "to move beyond 'knowledge- te l l ing ' forms o f wri t ing to 'knowledge-transforming' (Bereiter & Scardamalia, 1987), wh ich is believed to be the type o f wri t ing that promotes learning". 6 5 This suggests a shift i n language and content relations from one o f isolat ion and separation to one that hints at a mutual and multiple entanglement that cannot be ignored i f wr i t ing to learn is a focus o f the L 2 wri t ing curr iculum. S imi la r ly , Z a m e l (1992), interested i n wr i t ing practices, resists "a transmission model o f reading, focused on the retrieval o f information f rom a text", and cla ims, wr i t ing , because o f its heuristic, generative, and recursive nature, a l lows students to wri te their way into reading and to d i scover that reading shares m u c h i n c o m m o n wi th wri t ing, that reading, too, is an act o f composing, (p. 463) In heeding Zame l ' s ca l l and others, literacy becomes a dialogic space between mult iple literacies, not an autonomous, objective act. 6 6 In another text, Z a m e l (1993) challenges the traditional notion o f academic discourse as reductionist and formulaic and suggests, Rather than serving the academy, accommodating it, and being appropriated by it, we ought to work wi th others to engage in an enterprise that is far more dynamic , complex, collaborative and intellectually engaging, an enterprise whereby we and our students contribute to, complicate, and transform the academy, (p. 38) It is a stance that questions a technical imaginary and shifts into a radical hermeneutic orientation that values the learners' experiences o f wri t ing meaning across cultures as "this is the way a l l cultures, including academic cultures, come to be, continually re-created by those who enter and the languages they br ing wi th them" (ibid.). She later develops her posit ion into the beginnings o f a transcultural model (Zamel , 1997) that continues to critique mechanistic and reductionist approaches to teaching L 2 wri t ing "that resist engaging students in the messiness and struggle o f authentic work that begins, values and bui lds on . . . " (p. 343) learners o w n effects wi th language. In l i n k i n g this 6 5 See Cumming's (1989) discussion on developing writing expertise through knowledge transformation as a type of writing that fosters learning. 6 6 Hi l l & Parry (1994) and Parry (1996), hold similar positions to Zamel, promoting a dialogic, pragmatic, multi-literate perspective of reading, citing influential factors such as cultural backgrounds, individual variation and the enacting change that happens to both in the process of reading(writing) to learn in another language. The New London Group, comprised of such noted authorial voices as Cazden, Cope and Kalantzis, Fairclough, Gee, Kress, A . Luke, C. Luke, Michaels, and Nakata (as cited in Pennycook, 1996b), argue for multiliteracies and the polysemantics of words wherein "access and power have been recognised as more diverse and complex than before" (p. 170). 69 model to ways o f judging language, she reminds us "that we, l ike our students, br ing our backgrounds, subjectivities, and frameworks o f understanding to our interpretations; and that what we make o f our students and their experiences may very w e l l be an artifact o f these influences" (p. 350). In traces o f a radical hermeneutic stance and pragmatic orientation to literacy, Z a m e l challenges her readers to break from traditional ways o f understanding L 2 wr i t ing and be open to possibil i t ies o f the "messiness and struggle o f authentic work . " H o w do teachers respond to the "messiness and struggle" o f E L L writ ten content? G u i d e d by haunting questions o f consistency and fairness, mult iple studies have sought useful understandings o f various aspects o f judging E L L written content. 6 7 F o r example, much has been written about teacher commentary, exploring feedback and learner revisions i n terms o f dichotomous form/function relations (see authorial texts such as Cohen & Cavalcant i , 1990; Fathman & Wha l l ey , 1990; Ferris, 1995b; L e k i , 1991; Porte, 1997). M o r e recently Ferris, Pezone, Tade and T in t i (1997) attempted to explore the messiness by analysing the discourse of teachers' feedback and its relation to t ime, space, cur r icu lum and learners. Promis ing i n its effort to enter a possible radical hermeneutic space, the methodological approach was scientific in design, a performative contradiction o f orientations as the researchers attempted to make sense o f the 'mess ' . E x p l o r i n g "stereotyped expectations based on students' ethnolinguistic identities", R u b i n and Wi l l i ams-James (1997, p. 139) "caution us to beware o f the apparently persistent tendency to base our grades and evaluations on surface [linguistic] error" (p. 150), a f inding that suggests the dominance o f a technical imaginary ' s imposi t ion to isolate language f rom content. Johns (1991) explored "issues o f academic literacy i n university communi t ies" (p. 167) by ca l l ing into question notions o f academic literacy and its attainment by learners. Interviews with two discipline-specific faculty raised a number o f their concerns relating to the learners abili ty to "satisfy the experts i n an academic discourse communi ty" (p. 176) wh ich suggests a social constructivist orientation. Y e t , near the end o f the discussion, Johns offers a contrary theme i n that "analytic scales 6 7 Discussions of teacher's conceptions of and responses to E L L writing content in academic contexts I found useful, other than ones mentioned in this section of the chapter, are Bachman, Lynch & Mason, 1995; Cumming, 1992; Norton & Starfield, 1997; Vaughan, 1991; and, Shi & Cumming, 1995. 70 seem to be more appropriate for most academic wri t ing than the presently employed holist ic type" (p. 177), imp ly ing a reductionist need to objectify as criteria that w h i c h had been c la imed subjective. Evaluative criteria, it seems, has received E A P attention i n the last decade in its possibil i ty o f gaining consistency i n grading practices. However , even wi th expl ic i t ly stated criteria, teachers vary i n how the criteria are interpreted and implemented (Br indley , 1991; B r o w n , 1995). S imi la r ly , work by C u m m i n g (1990) and Vaughan (1991) suggest teachers approach the common reading and judging task differently. It seems, even though the dominance of a pragmatic, instrumental attempt to clarify and analyse elements o f assessment and evaluative practices, the hybridity o f difference cannot be completely concealed and creates a site of contestation and or iginal difficulty for teachers and for researchers. Perhaps attempts to reduce the "messiness and struggle" through assumed clarity and univocal i ty (sameness) is another example of the technical imaginary at work. It appears as a paradox that seems to render this work 'unsolvable ' as teachers w o r k toward objectivity yet dwel l midst constructions o f day to day evaluative practices that (in)consistently enact particular values and multiple, fragmented perceptions and conceptions at any given moment. Wr i t i ng academic content seems to have la id a predominantly pragmatic path o f orientation that imposes reductionist and formulaic strategies for teachers and for learners. These seem to have become the pr iv i leged (w)rites o f the pass(age) that mark-for-life. Disruptions W i t h i n Language and Content - Its Language and Its Legacy Trac ing orientations of authorial texts on integrative evaluation and related fields presents both anticipated and unanticipated dilemmatic sites. Fragments o f mult iple and incomplete orientations within and between meta-texts were evoked in multiple texts o f authority. Stern's recognition o f divergent pathways (his P and L approaches) i n language and content development was useful in becoming attuned to fragmented pedagogical orientations i n P-approaches that seem to conf i rm dominant technical impositions in evaluative practices, especially evident i n calls for separate language and 71 content evaluative criteria, and in the prevailing, traditional v iew o f text as autonomous and objective. A s we l l , i n tracing the linguistic orientation o f one Nor th A m e r i c a n L-approach, language and content relations were explored as functional relations that seemed to dominate pedagogical interventions developed from this framework, and r esist a technical imposi t ion that presumes to separate and reduce language to a context-less state. O n the language and legacy o f integration, assessment and evaluation, mul t iple perspectives were traced within and between meta-texts. Positionalities o f pedagogy and therefore evaluation seem to continue to impose a technical imaginary through the language o f measurement that dominates other competing meta-texts o f l inguistics and literacy. Traces o f social and functional stances i n these terms become hidden or concealed i n the hunt for certainty in evaluative practices. Assumpt ions o f positionality that have informed L 2 composi t ion research and pedagogy contribute directly to language and content practices. Author ia l texts on L 2 composi t ion research c la imed the predominance o f L l composi t ion research (Johns, 1990; Santos, 1992) and literacy education research ( M c K a y , 1993) as major influences o f the current L 2 condit ion. Traces o f a social v iew of literacy were evident, and even sometimes advocated, in their acknowledgement o f the social practices and issues o f power that construct and implicate a learner's multiple literacies. Entangled wi th in social v iews o f literacy was acknowledgement o f the 'messiness and struggle' o f learners' authentic work (Zamel , 1993). Traces o f a radical hermeneutic orientation to pedagogical interventions and social v iews o f literacy attempted to break wi th the technical. However , i n discussions o f L 2 composi t ion assessment, its pragmatic approach and the need to measure and number learner's work, teachers in the research reported on, although acknowledging the social implicat ions o f judging E L L written content, seemed to re-turn to accommodate a technical, rationality. Author ia l voices influence teachers' practices. Vaughan (1991) notes that research on teacher assessment practices has focused on product-oriented decis ion-making. Neglected has been inquiry into pre-conditions o f decis ion-making judgments and intertextual relations between prompt, criteria, written text and the teachers and learners 72 who create/read/write/use them. The next chapter explores approaches that v iew discourse as central to the interpretation o f instances o f teachers' practices. 73 Chapter Four Discourses of Difficulty: Methods of Inquiry Enac t ing research as a l i ved practice is dai ly work for me. I constantly have to w o r k against m y academica l ly i nduced tendencies to roman t i c i se , general ise , or t echnolog ize the purposes and forms o f . . . research. Fortunately, the c lass room teachers wi th w h o m I research usual ly disrupt any o f these tendencies. These disruptions, i n fact, are what constitute the l i v e d practice o f our research. For , no two days i n the the c lassroom are the same and no one theory holds together the disruptions i n the w o r k i n w h i c h c l a s s r o o m teachers and I are engaged. I must pay attention to these d is rupt ions i n the f i e ld , then, for they da i ly reconf igure not o n l y the cur r icu lum theories that frame my work but also the ways in w h i c h I conduct research wi th teachers. Janet M i l l e r , Disruptions in the field: An academic's lived practice with classroom teachers. T o a certain extent, the aporia of theory and practice has revolved around the ' and ' . There is obv ious ly an ambigui ty i n the term theory and practice. It can mean that theory and practice are l inked. B u t then the question becomes what the nature o f that connec t ion is . O r it might mean that the ' a n d ' signifies difference, that is different domains o f experience, w h i c h remain by def ini t ion, separate and n o n r e l a t i o n a l . . . . Theo r i z ing is a fo rm o f practice when it is oriented to questions o f purpose and c o m m o n concerns. Pract ice involves the mediation o f traditions, and the reflexive responsibil i ty to b r ing that to language and communication. Hans Smits , Living within the space of practice: Action research inspired by hermeneutics [T]he relat ion between a situation and a text has been transposed into the relat ion between a theory text and its pract ical texts. T h i s means that the relation between situation and text can be studied by a discourse comparison between a theory text and a practical text. Bernard M o h a n , The structure of situation and the analysis of text 14 This chapter begins by framing the inquiry contextually through descriptions o f the site and the participants. It is a site where language and content pedagogy is mandated and teachers regularly engage i n the activity of integrative evaluation. A central premise to the methods o f inquiry then is the notion o f activity and its discourse(s). The notion o f activity is explored asw social practices through paradigmatic features o f qualitative research, associated ontological assumptions and related forms o f discourse(s). Discuss ions o f discourse(s) are l inked to Ha l l i day ' s (1978) c l a i m o f language as social semiotic and his w o r k (and others) i n systemic functional l inguist ics ( S F L ) . U s i n g M o h a n ' s (1987) structure o f activity, aspects of S F L and forms o f discourse are brought together as a way o f explor ing the discourse(s) o f l i v i n g practices. F o l l o w i n g that, offered briefly i n a chapter section entitled 'undertaking this inquiry ' are discussions regarding col lect ing the data and its analysis, inc luding examples o f this approach to sample discourse(s) data. The chapter concludes wi th a section on legitimating this inquiry through discussions o f m y cla ims o f researcher posit ionality, and how concerns o f trustworthiness and obl igat ion are addressed. I heed M i l l e r ' s (1997) notion o f 'disrupted' discourse(s), discourses w h i c h are constituted wi th in social practices, also cal led activities. It is an inquiry that resists the possibi l i ty o f the researcher as an objective, neutral observer and questions the 'real worldness ' o f experimental research designs. Instead, research i tself is v i ewed as a social activity, replete wi th disruptions o f l iv ing practices. Contextual Framings The Site The project was undertaken at a Canadian college wi th specifically designed programs and curr icula for Japanese nationals pursuing post-secondary studies in Canada. The college was established in 1987 with, a miss ion to educate global citizens - individuals who understand their o w n and other cultures f rom a g loba l v i ewpo in t and who are c o m m i t t e d to d e v e l o p i n g the knowledge , p rob l em-so lv ing strategies and compass ion needed to participate fully i n a global society. ( A n educational review, 1992, p. 1) Hence, it is a college that offers academic study for Eng l i sh language learners. One documented cur r icu lum principle states that "The academic c u r r i c u l u m . . . weaves together 75 language learning, academic content, diverse cultural experiences and preparation for careers wi th an international focus" (ibid. , p. 3). The programs are designed such that these threads are woven into each course offered. Recognis ing that the learners are learning Eng l i sh as they learn subject matter, the intention o f the curricular approach taken was to "integrate language and content in ways that are meaningful and intellectually chal lenging" ( ibid . , p. 17) and attend to the needs o f post-secondary Japanese learners. 6 8 W i t h established cur r icu lum principles and a curricular focus o f language and content integration (see the col lege 's educational mandate i n Append ix 1), the college sought teachers wi th s imilar interests and qualifications. Often, teachers, individual ly and i n cur r icu lum teams, engaged i n simultaneous curr iculum development, implementation and learner assessment o f integrated language and content learning. 6 9 Questions o f integrative evaluation were considered wi th in the socio/cultural context w h i c h informs and is informed by the teachers who work at the site o f this inquiry. The culture o f the institution, evident in both explicit and impl ic i t ways, has the potential to influence teachers' praxis significantly. Structures o f time and space, o f collaborat ion and autonomy, and o f shared and unshared curricular views seem to influence how integrative evaluation gets done. In briefly describing a school culture at this particular site, m y purpose is to acknowledge some of the multiple influences on daily practices and to establish the import o f the phrase language and content to those at the col lege engaged i n pedagogical practices. 6 8 Institutional documents state explicitly that the academic programs at the urban campus of this Canadian, post-secondary college are build on a framework that addresses the learning of language and content simultaneously. No standard ESL classes were offered, all were content based with a Liberal Arts focus. The Knowledge Framework (Mohan, 1986) was a foundational tool in curricular design in the initial stages of curriculum development, using topic analysis and task (activity) as starting points to determine related language needs. Educational documents clearly describe the role of the teacher at this institution as a language and content teacher but nowhere do they articulate what it means in living practice. 6 9 An expectation of teachers is to work in curriculum teams with other teachers teaching the same course. A position of Curriculum Head, is held by a teacher knowledgeable in curriculum and instruction in a specific content area. The Heads offer support to faculty wrestling with the challenge of being a language and content teacher and issues related to academic standards. Meeting times for curriculum groups are scheduled during the working day and teams are expected to reach consensus on assignments and assessment practices common to all students taking the same course. The. thrust of this thesis comes from the tensions experienced at many of the curriculum team meeting discussions on assessment and academic standards. 76 The institutional culture may influence the discourse(s) o f integrative evaluation. 7 0 Three aspects o f institutional culture considered in this discussion are core pedagogy, structure, and norms. P rof i l ing this curricular landscape, they offer insight into what it means to teach at this institution. Faculty understandings o f core pedagogy at the time o f the inquiry were framed by at least three factors: 1) program goals clarif ied expectations o f faculty to expand learners' knowledge base and develop language ski l l s ; 2 ) an educational goal stated the necessity to integrate language and content; and 3 ) faculty meetings used for professional development opportunities, e.g., various sessions had focussed on integrating language and content learning strategies, thinking ski l ls , testing, target tasks, and developing a global perspective. T i m e was given to regularly scheduled meeting blocks in w h i c h a l l faculty were available for meetings; and to leadership for establishing and clar i fying criteria for evaluation and supporting curr iculum teams i n scope and sequence activities. It is an institution that has been structured wi th a v is ion o f collaboration. C u r r i c u l u m teams develop and fo l low a c o m m o n course outline and use common evaluation tools that contribute to a sense o f shared purpose, yet, most of the faculty teach and evaluate autonomously. It appears to be a supportive environment wherein ideas and materials are openly shared and four meetings a term are given to discussing cross-disciplinary issues such as the overlap o f content, and concerns and accomplishments o f the language learner. Teachers i n this project engaged i n discourse(s) o f integrative evaluation wi th in an institution commit ted to offering a program of substance i n Eng l i sh to learners for w h o m Eng l i sh is an additional language. Their knowledge, beliefs, and values o f evaluation continually shaped and reshaped their l iv ing practices. A s w e l l , different cultural histories and environments brought to those spaces by the experiences o f teachers engaged i n the activity o f integrative evaluation contributed to the complexity. Col lege documents contain broad educational and specific program goals that have 7 0 Werner (1991) describes school culture in terms of mutually reinforcing core pedagogy (e.g., common curricular and instructional goals) structures (e.g., timetable, meetings) and norms (e.g., internal and societal expectations and their power to coerce certain kinds of behaviour). He reminds us that much of school culture is unwritten. Describing the school culture is an attempt to describe why teachers teach what they do in the way they do and what happens when an idea comes into a culture and doesn't fit. My purpose here is to consider how the socio/cultural expectations of the institution shape integrative evaluation. At this site, it is written into the core pedagogy, given time, and teachers talk about evaluating language and content in their everyday practices. Yet, multiple tensions surround their practices, seemingly fed by the ambiguity of what it is and how to do it. 77 been a focus o f numerous professional development activities. Structures o f time, place and leadership encourage collaboration and shared understanding, especially i n the area o f assessment and academic standards. The documents and educational culture provide a mandate for teachers to teach and therefore evaluate language and content. O f interest is how the teachers make sense o f what it means to teach/evaluate language and content wi th in the borders o f this institutional culture. Students at this college study in a two or a four year l iberal arts program. The content wr i t ing used for evaluation purposes in this project was written by students studying i n the second year o f a two year program or years two, three or four o f the four year program. Those that were studying in the two year International Studies program took as core, c o m m o n courses Culture and Current Issues and Experient ia l Studies and chose a major course f rom translating and interpreting, environmental studies, business, or teaching Japanese to non-native speakers. Those that were studying in the four year International Relations program took as core, common courses G l o b a l Studies and Exper ient ia l Studies i n each year two, three and four, and chose a major in business, multicultural and ethnic studies, or translating and interpreting. Typ ica l l y in courses for E n g l i s h for A c a d e m i c Purposes ( E A P ) , students study academic Eng l i sh prior to entry into academic programs. B y contrast, the programs at this college are designed such that E n g l i s h is both the m e d i u m o f learning as w e l l as the subject o f study; that is, the students learn subject matter and the E n g l i s h language concurrently. In particular, the academic writ ing o f subject matter has been given significant pedagogical attention in a systematic way throughout the curr iculum in years two, three, and four across subject areas in the four year program as evidenced i n their curricular documentation. Professional development activities related to teaching academic wri t ing have contributed to the teachers' classroom practice. In turn, how they have come to understand the pedagogical interventions o f teaching the wri t ing o f subject matter has most l ike ly influenced their evaluation practices, particularly how they relate to the teaching and then judging o f the integration o f language/content. Felt tensions among teachers, wi th in and across courses, led to a concerted effort to establish and honour academic standards at the college. F r o m this ini t ia l work in academic standards across courses and years, a two 78 year plan was developed to address concerns o f fairness and consistencies i n evaluating student essays (see A p p e n d i x 2). A l t h o u g h many teachers choose to work independently, the college has encouraged a collaborative environment i n the development o f curr iculum and academic standards. B l o c k s o f time are scheduled into the timetable to ensure cur r icu lum teams (groups o f teachers teaching different sections of the same course) can meet to discuss issues that arise from their teaching and evaluative practices. Opportunities to re/search teachers mark ing independently, and marking collaboratively wi th their curr iculum team members, seemed to offer a r i ch source o f data collection. The Participants The subjects that volunteered for this study are teachers who regularly engage in the mark ing o f texts written by the students they teach. The educational backgrounds o f teachers vary f rom having a Masters degree i n a particular subject area without any formal training in teaching Eng l i sh language learners ( E L L ) to those who are not content specialists but have significant training and experience in teaching Eng l i sh language learners. Some have taught Eng l i sh overseas while others have taught i n the public school system in Br i t i sh C o l u m b i a . A l l teachers at the college teach language and content - the Eng l i sh language and subject matter - i n any one course. This context may be relatively unique i n its twofold purpose o f addressing language and content concerns i n the post-secondary classroom. F o r example, i n a course cal led G l o b a l Studies 200, teachers are expected to extend and develop the students' knowledge and language that w i l l facilitate the learners' abili ty to write causes and effects o f a particular issue such as deforestation. A s part o f teaching and learning, a l l teachers are expected to do integrative evaluation; that is, judge the language and content abili ty and knowledge of the students they teach. A s i n any research project, relationships and silent negotiations between and among al l subjects invo lved in the project (and I include myself) are inherent and must be acknowledged. I am not a stranger to the group o f teachers who volunteered for this study. I have worked at this post-secondary institution for nine years as a teacher and i n mult iple administrative roles, including responsibility for teacher evaluation. I a m conscious o f the 79 pol i t ica l nature o f m y past positions and how those relationships may have influenced the participants' responses. A l though I currently work part-time at this institution i n a coordinating role unrelated to the volunteer participants o f this project, a l l members o f the faculty are aware o f the leadership role I have had i n shaping the curr icu lum and articulating the educational and curricular assumptions that are the foundation o f the institution. They are also aware o f my pursuit of a higher degree in the very field i n w h i c h they work. I wonder, do they see this research activity as potentially educational i n terms o f their professional development? A s a researcher, I am aware that m y histories influence the participants and their conversations wi th me. S imi la r ly , I wonder how the teachers' histories influence me and our interchange. D o they think o f me as an "expert", an "authority", a theorist o f integrative evaluation practices? D o I value their expertise, their teacher knowledge, and/or their practice? D o I believe theory should influence practice? That practice should influence theory? These concerns are addressed further i n the discussion o f the project's trustworthiness later i n this chapter. A t this college, the fo l lowing seem to be assumptions on wh ich evaluating the content wr i t ing o f Eng l i sh language learners are based: 1) that teachers are asked to judge what students k n o w (content) from what they say (language); 2) that students can express what they know; 3) that teachers believe they are work ing towards fairness and consistency in their judgments; and 4) that evaluative criteria are based on the pedagogical interventions o f wri t ing. A s a researcher, how can I understand these assumptions; how are they constituted? C a n they be accessed and, i f so, how? Entangled in the opening up o f understanding integrative evaluation is the relationship between teacher and student, teacher and researcher and student and researcher wi th in the social situation under study. The Act iv i ty of Integrative Evaluation as a Socia l Practice The fo l lowing E L L text was written i n class i n response to a question on causes and effects o f global warming. Now, the world is getting warm. This is called Global warming. How does it affect to'the people? Global warming caused by not natural energy. We burned them after ozon rayer is broken. Also, CO2 and warm air stay on the air. In addition, sun rayer conies from directory to the earth. 80 Global warming will likely worsen air pollution, such as greenhouse gases etc. Also, it will alter rainfall pettern, we worn't be able to get freshwater. In addition, many serious diseases cause and food supilly. . Global warming is very serious one of climatic changes, first we use natural energy, such as sun, river and wind energy instead of not natural energy. If we won't do that, it will favour certain pests and diseases, (om 03/17/94) The fo l lowing discourse was generated by teachers independently comment ing on the same text before making a judgment. Headings used for discourse analysis: Reflecting-on-action: teachers discussing essay marking in general (ACCOUNT) Reflecting-in-action: teachers discussing specific essays ( C O M M E N T A R Y ) Action: teachers assigning a mark or grade (ACTS) Reflecting-in-action: teachers independently discussing specific essays (COMMENTARY) Tl: serious grammar errors have made the answer very unclear don't you think? T 2 : it gives me information...but you're right, the language is a little ungrammatical and the spelling! T 3 : if it's a pass, it's based on the introductory sentences and what I think the student is trying to say because I'm not sure of what he is saying. I'm guessing T 4 : they 're not clear statements about cause and effect. I would have to guess a lot too and that's not something I want to do. T 5 : it doesn 't matter how they say it, I just want the idea. I think the language is getting in the way of the content and I'm not sure how to mark it. These discourses are an instance o f practice teachers engaged i n judging the text o f an E n g l i s h language learner; as a research focus it is an example o f the social practice or activity o f integrative evaluation. A s a common, routine practice o f classroom assessment, it is a pedagogical activity i n wh ich teachers presume to judge what students k n o w from what students say. I use the term 'act ivi ty ' to locate the texts that involve a k ind o f social practice i n an identifiable social context wi th particular kinds o f actors (Spradley, 1980). The social situation central to this project is an activity of teaching. Figure 1 is an attempt to establish the activity o f integrative evaluation as a socio-textual practice. The physical setting is the 81 classroom, a k n o w n place to particular teachers engaged i n the activity o f evaluating written texts o f E L L s . Figure 1 The Act iv i ty o f Integrative Evaluat ion teachers' texts . actors: evaluating V E L L texts place: the classroom as a topographical text A s w e l l as identifying components that comprise an activity, it is important to consider the multiple sub-activities in a social practice. Spradley views the web o f connections as part o f an extensive system of activities; activities that are contained by larger social contexts and that contain smaller elements of activities. They are situated one wi th in another (see Figure 2). The social practice o f the current project becomes a network o f sub-activities i n wh ich the same actors participate to accomplish integrative evaluation. The wider context o f school is evident i n the Col lege ' s educational mandate and i n the A c a d e m i c Standards Commit tee 's two year plan (see A p p e n d i x 1 and 2). F igure 3 is an attempt to illustrate this complexity. Distinctions o f particular activities fade; i n the process o f completing one activity, at least one other activity is implicated. Figure 2 Partial Contextualization o f Integrative Evaluat ion St i l l wider context: Educational system W i d e r context: The school Act iv i ty : Integrative evaluation Sub-activity: Determining evaluative criteria 82 Figure 3: The Complexi ty o f Activi t ies Related to Integrative Evaluat ion M • writing/reading the prompt • the question • the conditions • the evaluative criteria j M • writing/reading (interpreting) the response to the prompt Cul tura l psychologist Co le (1996) posits that activity is grounded i n everyday life events -i n this project, the everyday professional life events o f teachers. C o l e offers that social situations, wherein actors are viewed as active agents i n the co-construction o f jo int activities, are surrounded by contexts o f historical and current significance; those constructed material artifacts and social practices that continually reshape social situations. Qualitative Inquiry and Ontological Assumptions Quali tat ive inquiry explores cultural meaning and "emic" l inguist ic concepts based on assumptions that cultural or social actions are influenced by an ind iv idua l ' s knowledge o f loca l norms (Harre, 1993). In contrast wi th quantitative approaches that privi lege numerical data, qualitative approaches use discourse as data i n their interpretations o f social realities. In the v iew o f some analysts of certain types o f qualitative research, ontological conceptions o f the wor ld consider the individual ' s social realities to be constituted midst dynamic interchanges within his/her social wor ld . 7 1 It is a v iew that values the subjective experiences o f individuals and uses methods to illuminate how individuals create, participate i n and make sense o f the social situations that make up their wor ld . It gives import to the role o f cultural interpretations o f behaviour. In a recent publication on new methods in psychology, Smi th , Harre, and Langenhove (1995) emphasise the central role o f language and discourse i n the construction o f an individual ' s social realities. Harre, a philosopher, i n earlier works (1985, 1993, 1994) has consistently maintained that discourse has a significant role i n 7 1 Ontological assumptions, concerned with the very nature of being in the world, contribute to particular views and interpretations of social reality. 83 teaching/learning to write content judg ing the response social situations; that discursive interactions are inherent in a l l social situations and perform to structure and effect the social realities individuals have come to know. Harre (1993) posits, "socia l behaviour is the structured product o f the joint actions o f intelligent and knowledgeable agents acting to further some end or other. It is not the effect o f causes" (p. 107). Harre and Gil let t (1994) c l a im that activity is a planned and intentional social practice and that individuals ' intentionality can be understood in terms o f discourse (conversations), the socially produced linguistic exchanges wi th in activities. They argue that activity and its ordered, intentional acts are 'actions of discourses' that construct and are constructed by situated, sociocultural knowledge. Discourses, as actions o f social situations, contain sociocultural knowledge of those engaged i n l inguistic exchanges. Spradley (1980) and Harre (1993) both v iew activity as a semiotic o f theory/practice, knowing/doing , cultural knowledge/cultural practice. Spradley (1980) considers cultural knowledge, "the acquired knowledge that people use to interpret experience and generate social behaviour" (p. 5) and cultural action, as fundamental to human experience. H e speaks o f activities as "streams o f behaviour . . . [with] recognisable patterns o f ac t iv i ty" (p. 41) that are culturally interpreted. Harre (1993) compares resources and acts, a theory/practice relationship he claims exists i n al l activities, to l inguistic concepts o f competence and performance. 7 2 H e defines competence as the background knowledge one knows - rules and conventions o f language appropriate to a social situation - that influences, and is influenced by, the performance o f speech acts. Together, resources and actions, competence and performance, inform the social act. In Harre 's terms, analysing these as different types o f discourse gives insight into the socio-cultural act: the actions (practice) o f an activity and the resources (theory) that inform them. B o t h address the socio-cultural knowledge one uses to accomplish the activity. Table 3 aligns Spradley 's and Harre ' s views wi th notions o f theory and practice, and activity. H o w is activity interpreted by its participants? What resources o f cultural knowledge in fo rm and are informed by its practices? In contrast to a process-product 7 2 The relationship of theory and practice, as Smits reminds us on page 70 of this text, is an ambiguous one. For purposes of gaining useful understandings of the discourse datum, I explore the work of social psychologists and functional linguists and how the notion of activity can be considered a semiotic of theory/practice. 84 paradigm o f research on teaching (Brophy & G o o d , 1986), the ontological assumptions that underpin the v iew o f activity presented here focus on cultural interpretations o f behaviour and v i ew discourse as central to the construction of the theory/practice o f an ind iv idua l ' s social realities. Table 3 Ac t iv i t y as a Semiotic o f Theory/Practice Spradley (1980) Harre (1993) activity cultural knowledge resources (knowledge) cultural action acts stream o f behaviour stream o f behaviour Recently the field o f L 2 teaching research has taken particular interest i n investigating the assumption that teachers' practices are inextricably l inked wi th their pedagogical knowledge and beliefs. 7 3 Their work explores teachers' interpretations o f c lassroom activities and reflections on decisions in the classroom. Current emphasis on reflective practice supports this reflection on action. Richards & Lockhar t (1994) remind us, It is necessary to look objectively at teaching and reflect c r i t i ca l ly on what one discovers . . . . It can help achieve a better understanding o f one 's o w n assumptions about teaching as w e l l as one 's o w n teaching practices, it can lead to a richer conceptualisation o f teaching and a better understanding o f teaching and learn ing processes; and it can serve as a basis for self-eva lua t i on and is therefore an impor tant componen t o f p ro fe s s iona l development, (p. 2) Schon ' s (1983) account o f reflective practice includes the notion o f reflection-in-action, a type o f action where the teacher negotiates theory/practice in the practical context o f their c lassroom. In Altr ichter , Posch, and Somekh (1993), Schon 's work on different relationships between professional knowledge and professional action is explored and three types o f action are identified: 1) reflection-on-action as discourse that "distances i tself f rom the f low o f activit ies" (p. 206); 2) reflection-in-action as dealing wi th "new and complex situations . . . or disturbances and problems [that] disrupt the smooth f low o f routinised ac t ion" (p. 205); and 3) tacit knowing- in-act ion as routine "actions or mind-sets w h i c h have been buil t up through frequent repetition" (p. 204). O f significance to this project is 7 3 See the work of Borg, 1998; Johnson, 1992a, 1992b; Woods, 1996. 85 Schon ' s acknowledgement o f reflection-in-action. It is a way o f attending to the discursive preconditions o f decision making involved in integrative evaluation. Forms of Discourse in Activity Harre (1993) and Spradley (1980) both use activity as a unit o f study and offer approaches that investigate the theory/practice of activities; that is, through their methodologies they attempt "to make explici t a l l that is tacit i n a form o f l i f e " (Harre, 1993, p. 107), an important goal i n several forms o f qualitative research. Spradley (1979) discusses methods for an ethnographic interview as an investigation o f the theory o f activity; the interview, removed from the activity, is a k ind of discourse o f reflection. In another text, Spradley (1980) describes approaches to participant observation as a way o f investigating the practice o f an activity; the observation, located midst the activity, records discourses o f action wi th in the activity. W h i l e Spradley is concerned wi th detailed descriptions o f the activity including language, Harre emphasises discourse(s) as the central point and argues that discourses contain evidence o f one's resources (theory) and acts (practice). Fundamental to Harre 's (1993) work is his dist inction Between two kinds o f discourse. There is that wi th wh ich social acts are accomplished and there is that wi th wh ich we comment upon and theorise about those social acts. Accounts are discourses o f the second k i n d . . . . some o f the norms o f social action are made explici t i n accounts, though for a l l sorts o f reasons. In first order discourse, the norms o f action are impl ic i t , (pp .116-7) H e suggests that (discourse) acts are a form of discourse that i m p l y the norms o f actions (practices) o f an activity, whereas discourse accounts o f knowledge as theory and discourse commentaries i n action make evident norms o f social acts. The meanings o f social acts are made more expl ic i t through differing types o f discourses (Table 4). Table 4 M a i n Types o f Discourse data Harre et. al (1985), Harre (1993) accounts o f knowledge reflecting ( ' i n ' or ' o n ' action) commentary on acts (discourse) acts action 86 In F igure 4, Harre 's ma in types o f discourse data are l inked to discussions o f theory and practice, and reflection and action. Exp l i c i t discourse evidence o f theory can be aligned wi th Harre 's accounts o f knowledge while discursive evidence o f practice seems to be found i n both commentary in action and (discursive) acts themselves. However , it seems that i n discussions o f reflection and action, discursive evidence o f reflecting ' i n ' or ' o n ' action can be aligned wi th Harre 's accounts of knowledge and commentaries on acts, whi le action seems pr imar i ly concerned wi th acts o f decision-making. Reflecting-on-action and reflecting-in-action as discourse evidence o f both accounts and commentary, have the potential to make explici t the norms impl ied in the social acts. O f particular interest to this inquiry are the discourses located wi th in the pre-conditions o f acts o f decision-making (e.g., teachers accounts and commentaries prior to deciding on evaluative criteria or a mark or a grade on an essay) as these discourses c l a i m a contested, discursive space o f dilemmas, contrary themes and undecidabilities ( B i l l i g et al . , 1988). A n a l y s i n g , for example, teachers' accounts and commentaries has the potential to make the norms o f marking or grading acts were explici t . Significant i n the data analysis then are the relations o f accounts and commentaries, as they provide pr imary evidence o f the cultural meaning o f the (discourse) act. A s w e l l , Harre et. al (1985) discuss the practical and expressive aspects o f action as " A c t i o n occur[ing] on two levels at once. No t only has one to do the right thing, but one has to appear as the right k i n d o f person" (pp. 88-9). It is a l i n k i n g o f the idea o f acts - a concern for do ing the right thing (e.g., marking 'correct ly ' ) - and the idea o f identity - a concern for being the right k ind o f person (e.g., v iewed as a teacher who is fair and consistent i n marking); it is a posit ioning o f subjects i n discourse(s). O f particular import is Harre and Gi l l e t t ' s (1994) c la im, The sense o f agentic posit ion, the sense that one is the agent o f one's actions and responsible to others for them, is something that we acquire through learning the language and the cul tural conventions for the assignment o f responsibil i ty, (p. I l l ) Aspects o f the practical and the expressive are a l ink to the socio-cultural situatedness spoken o f earlier i n this chapter and o f neo-Vygotskyan approaches that invite images o f the ind iv idua l i n dramatistic, performative notions o f action. H o w teachers make sense o f 87 doing the'r ight ' thing and being the 'r ight ' k i n d o f person as it relates to integrative evaluation is o f particular interest in this project; it raises questions o f norms and conventions, and o f the multiple, and possibly conflicting, interpretations o f the normative and conventional expectations operating wi th in and among individuals . Harre (1993) attends to the means o f actions, the "unfolding o f everyday life and to the performances o f the people who l ive i t . . . an ironic stance, a viewpoint from which life goes forward, becomes v i s ib le" (p. 191). Researching through what Harre calls a dramaturgical model , activity becomes a scene wi th al l its necessary and sustaining i l lusions; it is a s imulacrum o f participants performing their parts in daily routines. V i e w i n g scenes f rom this stance not only implicates the researcher i n his/her part o f the larger performance o f research itself but it has the potential to make visible the diverse scripts, improvisations, interpretations and, as B i l l i g et al . (1988) remind us, the inherent dilemmas enacted wi th in the webs o f connections o f activities. In sum, it is argued that discourse data, wi th a l l its di lemmatic aspects, is central to the methods o f inquiry o f much qualitative research and helps to make the context o f an activity explici t . A l though Harre claims two types o f discourse, I, i n support o f M o h a n ' s (1999) work, make distinctions between three main types: 1) a discourse o f practice that reflects discourse acts o f activities (e.g., discourse collected when teachers engage i n evaluating essays); 2) a discourse o f commentary that serves to interpret meaningful acts o f practice (e.g., discussing possible decisions wi th self, wi th members o f a collaborative mark ing group or wi th a researcher whi le engaged in the marking process); and 3) a discourse o f accounts that provides information o f norms, rules, and conventions for acting 'correct ly ' and offers evidence for cultural knowledge o f an activity (e.g., conversations between teachers, or between a teacher and the researcher that elicits talk about marking wr i t ing i n general). It is essential to relate commentaries and accounts to discourse acts. Whereas conventional interview research methods emphasises accounts (e.g., teachers' responses to questions prior to or after action) and behaviourist research methods emphasise acts (e.g., teachers' behaviours (acts) related to student acts o f achievement), this approach claims that evidence from al l three types o f discourse data are necessary to interpret the meaning o f activity more explici t ly; that is, commentaries and accounts are 88 viewed as primary evidence o f the impl ied cultural meaning o f an act. It is a notion that considers discourse as a social semiotic text. Discourse as a Social Semiotic Text Language arises i n the life o f the individual through an ongoing exchange o f meanings wi th significant others. (Hal l iday, 1978, p. 1) It has been argued that the process of learning language and creating meaning is a social process invo lv ing experience and the interaction wi th oneself and wi th others. The l inguist H a l l i d a y (1978) posits, "the construal o f reality is inseparable f rom the construal o f the semantic system i n wh ich the reality is encoded. In this sense, language is a shared meaning potential, at once both a part o f experience and an intersubjective interpretation o f experience" (p. 1). S F L theory is based on the premise that language is a social semiotic and Ha l l i day argues that "socia l reality (or a 'culture ') is i tself an edifice o f meanings - a semiotic construct" (1978, p. 2). In S F L , semantics is interpreted wi th in a sociocultural context, a context wh ich is itself understood semiotically - the social ly constructed relationships o f meaning between the signifier and the signified. Discourse, considered as "a text - an instance o f language i n use" (Hal l iday & M a r t i n , 1993, p. 24) is seen as a resource for social purposes. Th is stance makes possible the recognit ion o f discourse communit ies and their specialised uses o f language. It is a shift f rom a focus on the indiv idual to v iewing community as a system of social situations or activities o f w h i c h language is central. It is a social theory o f discourse that begins wi th the discourse and seeks to describe its features by its social functions; it is a v iew o f discourse as social semiotic text. H a l l i d a y ' s contribution to l inguistics is significant. B u i l d i n g on Saussure's idea that language is a product o f the social process, Ha l l iday describes how learning takes place through language and how language is used to construe meaning. Language, for S F L linguists, is a meaning-making system. For Hal l iday and Hasan (1985), language is text, not sentences, and always interpreted in its relationship to social structure. They define language "as a resource for meaning, centrally invo lved i n the processes by w h i c h human beings negotiate, construct and change the nature o f social experience" (p. v i ) . Language exchanged i n social contexts, i n wh ich there is always some form o f social structure, derives 89 its meaning from these social activities. It is a theoretical framework that focuses on discourse(s) i n social contexts. One important S F L c l a im is that meaning is situated i n contexts o f situation and can be specified through register variables o f field, tenor, and mode (Hal l iday, 1994). F i e l d refers to the socially recognised activity taking place at the time and includes an activity focus (the k i n d o f social activity - e.g., marking essays) and an object focus (subject matter -e.g., criteria for marking). Tenor refers to the social relationships wh ich hold between the various participants in the interaction and that are specifiable (e.g, tenor i n integrative evaluation activities could reflect the power or status relations between colleagues i n collaborative contexts or between a teacher and the researcher). M o d e refers to the role o f language in the interaction and how it is being used (e.g., teachers' comments can be oral or written, action or reflection). I f we are to apply S F L to Harre 's distinctions, it is important to dist inguish the discourse features o f various discourses emerging from social practices discussed earlier and briefly summarised here: 1) the discourse acts o f the activity that reflect the practice o f the activity, 2) the discourse acts o f the activity that reflect the comments and descriptions o f the practice o f an activity, and 3) the accounts or discourse o f theory that provide evidence o f the cultural knowledge of an activity. W i t h i n the action/reflection dimension o f mode, as discourse "moves from action to reflection there is a progressive distancing f rom the actual event and the experience becomes increasingly vicar ious" (Gerot & W i g n e l l , 1994, p. 11). M a r t i n (1992), an S F L scholar, describes differences between the three types o f discourse data along this dimension i n terms o f discourse features. H e describes the language o f action (e.g., a teacher marking collaboratively comments, "let's mark this one together, I can't figure it out") as verbal actions and speech acts where attention is g iven to the text as interaction rather than as content. H e explains that the language o f commentary denotes specific people and actions, often, but not necessarily, i n the past tense i n reflective comments (e.g, a teacher reflecting on the mark she gave comments, " / think I was too generous with this one"). A n d , he suggests the language o f accounts is about generic things and actions/events and is usually i n the timeless present tense (e.g., a teacher conversing about marking stated, " / always write down my criteria before I 90 mark."). M a r t i n ' s work is an impl ic i t acknowledgement o f the possibi l i ty o f a theory/practice relationship in discourse acts and, therefore, potentially l inks 'mode ' o f discourse wi th qualitative research methods. Table 5 summarises this argument, suggesting possible connections between the qualitative discourse methods o f Harre and S F L w o r k on mode o f discourse by Mar t i n . In addition, Figure 6 illustrates that reflecting-on-action and reflection-in-action have the potential to be constituted by a l l three discourses. F ina l ly , it has been argued that reflecting-in-action, wherein evidence o f al l three forms o f discourses can be found, is a possible site o f contested discourses in the pre-conditions o f decis ion-making , a site o f central interest to this project. It combines methods o f inquiry that address situated and distinct forms o f discourses wherein cultural meanings o f social acts, replete with potential dilemmas, are made more explicit. Table 5 Discourse Ana lys i s : Qualitative Research Methods and S F L Approaches Harre (1993) M a r t i n (1992) (mode) reflecting-on-action e.g., research interviews: teachers' making grading decisions explici t reflecting-in-action e.g., collaborative marking: prior to determining mark account generic (theoretical) account account commentary specific (commentary) commentary commentary (discourse) acts language i n action (participation/ interaction) acts (a judged essay) acts Further delineations o f discourse and sociocultural knowledge are central to the work o f Schieffel in and Ochs (1986). They suggest that knowledge o f language and o f culture, interrelated and inseparable, are learned through the process o f socialisation. They posit, In mak ing sense out o f what people are saying and i n speaking i n a sensible fashion themselves, chi ldren relate l inguistic forms to social situations. Part 91 of their acquired knowledge of a l inguist ic form is the set o f social relations that it forms i n social situations, just as part o f their acquired knowledge o f a social situation includes the linguistic forms that define or characterise it. (p. 2) The work o f Schieffel in and Ochs conceptualises a framework o f language socialisation based on the role o f language in the process o f becoming competent members o f social groups. It is a sociocultural v iew of activity, a site where "activity mediates l inguistic and sociocultural knowledge and that knowledge and activity impact each other" (Ochs, 1988, p. 15). The role o f activity i n language socialisation is outlined i n Figure 4. Figure 4 Ac t iv i t y : M o t i l i t y Between Sociocultural and Linguis t ic K n o w l e d g e sociocultural knowledge M—• activity M • l inguistic knowledge Ochs offers activity as the product o f a sociocultural, discursive practice that has a cumulative, goal oriented set o f actions involving human investment. She claims that activity is the location o f sociocultural and linguistic learning - a place where language crafts, and is crafted by , sociocultural knowledge. In sum, for purposes o f delineating methods o f inquiry used i n this project on integrative evaluation, I argue that activity is a social act or practice wherein at least three forms o f discourse and general related linguistic features can be identified in the analysis o f discourse data. T w o forms o f discourse, accounts and commentaries contribute to interpretations o f the third form of discourse - impl ied cultural meanings o f social acts. Considerat ion is given to a l l three forms o f discourse and their potential to be di lemmatic as a way o f attempting to make more explicit the felt difficulty of l i v ing practices o f integrative evaluation. Under taking This Inquiry Integrative evaluation is explored wi th in triangulated sources o f the discourses 92 gathered from live(d) experiences o f being there.74 Outl ined i n detail be low, the sources o f data include: tape recordings of teachers collaboratively and independently conversing wi th me whi le marking; recorded debriefing sessions; collected participants' oral and written responses to the reporting at the debriefing sessions; recorded conversations wi th students about their marked wri t ing; and, personal journals kept as a researcher and as a teacher marking. The doubl ing o f roles - as a researcher and an actor i n a social situation, - is not regarded as distinct and separate, but as perspectives that have the potential to support, elaborate and i l luminate useful understandings of integrative evaluation and therefore are woven thoughtfully and intertextually throughout the study. Collecting the Data The fo l lowing procedures for data collect ion framed this project: 1. Participants were recruited by a memo circulated to a l l faculty. The request was for me as the researcher to attend and interact in regular collaborative mark ing sessions by cur r icu lum teams and to sit wi th individual faculty as they marked their students' written texts. I contacted the fourteen that responded to my memo, asked them to sign a subject consent fo rm and then established a mutually agreed upon schedule o f sessions to attend. 2 . Three kinds o f sessions were scheduled for me to attend and tape record: four different cur r icu lum teams collaboratively marking a wri t ing assignment completed by al l the students taking the course; ten teachers independently marking assignments written by learners they taught; and, fourteen teachers, who may have participated i n the above sessions, independently marking anonymous assignments written by students at the college. A l t h o u g h I d id not lead the sessions, I asked semi-structured and open-ended questions as teachers engaged i n their marking activities. A t the end o f the sessions, and as part o f the conversation, I asked them to reflect on the activity of integrative evaluation. 3. S i x debriefing sessions were held: two wi th each group o f participants i n the three kinds o f sessions described above. The purpose o f the debriefing sessions were to share draft analysis o f the data collected. Teachers were invited to respond to the analysis oral ly or i n wr i t ing . These sessions were recorded and considered part o f the data col lect ion. 7 4 Britzman (1995) reminds us, "There is a belief and expectation that the ethnographer is capable of producing truth from the experience of being there and that the reader is receptive to the truth of the text" (p. 229). By "being there", we often assume we are capturing the true story of the activity. I sought not one true story but multiple stories of integrative evaluation. 93 D u r i n g the first session, teachers responded to m y interpretations o f the discourse data (see A p p e n d i x 3). A n y anonymous written participants' responses to the debriefing sessions were collected. The participant's feedback led to revisions in the ini t ia l analysis. The purpose o f the second debriefing session was for each group to respond to those revisions that reconstructed their and my collective v iew o f the discourse data from the first debriefing session. A redrafting o f that analysis was circulated to participants invi t ing further feedback (see A p p e n d i x 4). 4. A request for students interested i n participating i n the project was posted. A smal l group o f seven responded and I met wi th them. W i t h the assistance o f an interpreter, I explained the study and asked them to sign an English/Japanese consent fo rm i f they wanted to volunteer. I arranged to meet with each student for half an hour at a later time when they were wri t ing one o f their course assignments. I also asked them to br ing a marked written assignment they had done. A t the meeting, I asked semi-structured questions as they were wri t ing, asked them to reflect on what it meant for them to write subject matter i n E n g l i s h , and asked how it should be judged. I also asked them to respond to the mark they received on the second assignment. Our conversations were tape recorded. Dealing with Discourse(s) as Data Twenty-seven ninety-minute cassette tapes hold the discourse data collected over a period o f three months through the above procedures. E a c h tape was reviewed wi th in one week o f taping and evidence o f accounts, commentaries and acts of integrative evaluation were transcribed as found in the multiple activities o f integrative evaluation. Discourse evidence f rom independent markers and from collaborative marking sessions were used i n the analysis o f the data. The tapes from the debriefing sessions were reviewed and transcribed. The researcher's journal was examined for evidence o f accounts o f integrative evaluation (the focus o f the research and used as evidence i n the discussion o f the findings) and as a reflecting-on-action o f the research project itself (a process o f ref lexivi ty I used to in form m y research at a l l stages and is therefore infused throughout the wr i t ing o f this text). Once collected and transcribed, the discourse data o f integrative evaluation was analysed for Harre ' s notions o f accounts, commentaries, and acts as is illustrated i n the fo l lowing examples. In a l l examples, the texts being evaluated were written by E L L s . 94 I. Collaborative Integrative Evaluation - sample data Context: A group o f three teachers who are experienced markers collaboratively judge 84 responses to the same essay question on mul t icul tura l ism i n the same room at the same time i n an attempt to be fair to students and gain consistency i n grading practices. The researcher is present. Reflecting-in-action: teachers on the process of their marking (COMMENTARY) T 2 : So is it five and five - five for the structure and five for the content? T 3 : I think that's easier to mark with. T 2 : So the opening sentence has to have part of the question in it? R : How did you teach the notion of 'topic sentence'? What do the students understand by this term? Tl: I don't know. I think some of my students are going to fail - they may use a lead or something to catch the reader's interest. That's how we talked about introductions. T 2 : Alright, we didn't ask for a topic sentence so we can't mark for one. T 3 : This sure is difficult to agree on a mark. T 2 : And more time-consuming but the students benefit ultimately. T l : Ya, I agree. Let's stick with it. Action: teachers judging essays collaboratively (ACTS are underlined) T 1 : Look at this one. It didn't answer the question and it has no conclusion. I'd give it a four. Which is the four pile? T 2 : Let me see it. Hmmm...the student has put a lot of thought into it though. What's wrong with this last sentence? It links back to the topic and there are hardly any grammar errors. I think it deserves a six or seven. T 3 : Let me have a look at it. We need to come to a closer agreement. It's actually quite a well written text - just a little off topic. Reflecting-on-action: teachers talking about the collaborative experience after the marking session. (ACCOUNTS) R : As you think back over the last hour and a half, what are your comments about collaborative marking and about fair and consistent evaluation of students' essays, especially in the marking of language and content? Tl: It was good but much more difficult than I thought it would be. T 3 : Ya, I would agree. I feel better about the marks but amazed at the number of times we couldn 't decide. I realised how different we each think about each piece of writing. T 2 : / noticed we had most trouble with what was language and what was content. It seemed so clear with the criteria for language and the criteria for content we laid out but it was not so easy when we read the essays. T1:I think I learned from the experience of hearing your (the other teachers) comments. That was helpful. In general, I think the students benefited from our collective judgments. But, to tell the truth, I still don't know how to mark the essay when the ideas seem to be there but the language is messy or when the language seems fairly readable but the ideas aren't substantial. 95 II. Debrief ing Session on Integrative Evaluat ion - sample data Context: teachers that participated in the independent marking sessions attended a debriefing session where I offered m y interpretation o f the discourse data and asked them to give me their interpretation. Reflecting-on-action about integrative evaluation (ACCOUNTS). T 5 : I think that its hard to separate language and content but it is certainly easier to mark them separately. That's a problem for me. R : When you separate them, how do you think of language? I try to think of the whole text, that helps me to see a closer, more connected relationship. Tl. I see what you 're saying but for me I think of language as grammar, the verb tenses, proper use of nouns, you know, the rules, the bits that distinguish a native writer from a non-native writer. I can count the errors, I can list them, I can show the students their weaknesses in writing. I never really understood what is meant by 'discourse'. III. Researcher 's Journalised Accounts o f Integrative Evaluat ion Context: during the marking sessions, teachers asked about integrative evaluation and about specific texts they were marking. A t the end o f each session, I made a journal entry that reflected on the content o f our discussions as w e l l as on the process o f researching. Reflecting-on-action:one of my journal entries (ACCOUNTS) R : I tell the others that I consider the text first to get a sense o f the purpose o f the writer. Is s/he trying to convince, persuade, argue, define...Starting wi th the whole text takes my eye away from the grammar errors evident i n the wri t ing. I don ' t want to be influenced by grammar errors but I k n o w as soon as I see one, as I d id i n a paragraph the teacher was marking this afternoon, I a m distracted by it, I want to correct it, i t 's almost an automatic response. W h y is the eye more control l ing than the ear? I V . E L L s on what it means to write and judged Context: I sat wi th individual students as they worked on a wr i t ing assignment. W e talked about what it meant to write and be judged on their written content as an E L L . action: a student engaged in writing content (ACT) S I : How do you spell grobal? I always make mistake (commentary on act) . . R : ah, global g-l-o-b-a-l S 1 : Thanks. Umm . . . I want to say these things need each other but I forget word. Do you know? R : You are writing about global systems? Hmm. How about interdependent. Do you know that word? reflecting-in-action: the student engaged in writing content (COMMENTARY) S 1 : 1 make so many mistake here. Look how much I cross out. R : Why do you think you 're having difficulty? S 1 : Vocabulary. My vocabulary is weak. I need more words. I can't remember grammar points too. I want to write correctly. Then ideas stop. Should I think of ideas or rules when I write this. I don't know. It is so hard. 96 reflecting-on-action: a conversation about writing and judging ideas (ACCOUNT) SI: Its hard to write ideas and think about spelling. R : What if you just try to write your ideas? S 1 : Too difficult. I know some things in Japanese I can't tell in English. The teacher marks my language. I know its poor. But, I know this topic. I can't tell her what I know. She doesn 't know what I know. My content mark is low. I don't like this kind assignment. U n d o i n g the discourse data i n this way, l iv ing practices of integrative evaluation and its felt tensions can be located and made sense of. It al lows questions such as, how do teachers do integrative evaluation, why do dilemmas occur where they do, do teachers practice what they espouse and do they espouse what they practice, to be probed. Student voices add a dimension to the project that have the potential to elaborate on the question, can teachers judge what students know from what they say? Legi t imat ing This Inquiry Cul tu ra l geographers Cosgrove and D o m o s h (1993), concerned about issues o f author and authority i n wri t ing, remind us that questions o f representation and communica t ion are dependent upon pr ior questions o f ontology (what constitutes reality), epistemology (how we come to k n o w that rea l i ty) and science (the f o r m a l cons t ruc t ion o f such knowledge) , (p. 25) A s a researcher/writer, these too are m y concerns. Heeding these words, and as a way o f legit imating this inquiry, in what fol lows I discuss questions o f researcher posit ionali ty and the project 's trustworthiness and obligations to this k ind o f research practice. Researcher Positionalities75 A s the researcher o f this project, I question traditional ontological understandings o f discourse as being representative o f an independent reality, and instead v iew communica t ion (discourse) as dilemmatic ( B i l l i g et al . , 1988). Rather than the preciseness and clarity o f reality being reflected in language, I a m open to the possibil i ty o f language as opaque, wherein meanings shift wi th in the act o f communicat ion. A n d , I am open to the possibil i ty that multiple socially constructed ways of coming-to-know integrative evaluation may be constituted i n discourse(s) containing ambiguous and confl ic t ing fragments o f 7 5 In chapter six I discuss the effects of researcher positionalities in inquiry practices in more detail. Here, my intention is to specify what I understand to be my positionalities in this particular research project. 97 posit ionali ty that become l i v i n g 'disruptions' (Mi l l e r , 1997). These notions frame the project and assist i n understanding the discursive contestations, negotiations and nexus I anticipate i n exploring the activity o f integrative evaluation. Positionalities that inform and guide this inquiry o f integrative evaluation, then, are based on assumptions o f 1) qualitative research and certain related ontological assumptions; 2) discourse as a fo rm o f social action; and 3) cultural and historical realism. The stance taken here rejects the idea that the purpose o f research is to search for the ' truth' , natural and universal laws that exist 'out there'. Tradi t ional research has often assumed that data collected w i l l capture the 'true story' of the activity being researched. M y intent is not to discover the 'essence' or nature o f the indiv idual or o f the collaborative group doing integrative evaluation, instead I am interested in the multiple positionalities enacted, positions o f integrative evaluation as instances o f l i v ing practice. I do not seek a single, explanatory reality that may exist for a l l teachers. Instead, I a m interested i n the felt tensions o f those positions, the "socia l preconditions . . . those aspects o f socia l ly shared beliefs w h i c h give rise to the dilemmatic thinking of indiv iduals" ( B i l l i g et a l . , 1988, p. 8). I believe integrative evaluation practices have the potential to be situated wi th in connected historical and cultural frames (e.g., notions o f school culture as outl ined by Werner , 1991) from wh ich emerge multiple perspectives, some o f w h i c h are contradictory, fragmented, and confl ict ing. Teachers (and I include myself) br ing to this process understandings o f language, o f content, o f marking, as have been learned i n social ly constructed contexts throughout various life experiences. Discern ing the influences o f history and culture that have shaped our discourses, that is, how we understand language, content, second language writ ing, w i l l influence integrative evaluation and its collaboration. It is an educational legacy that has influenced ways o f thinking about, and acting upon, the practices o f judg ing E L L essays. W a y s o f thinking, as Haraway (1988), a feminist researcher, argues, are "situated and embodied knowledge . . . [those] partial, locatable, cr i t ical knowledges [that have the potential of] sustaining the possibil i ty o f webs o f connections" (p. 182). Shared meaning, one example o f Haraway ' s web o f connections, is an element o f integrative evaluation. Collaborat ion is the negotiating and connecting o f 98 partial, fragmented social knowledges among group members and wi th in oneself. Haraway ' s ideas give privilege to "contestation, deconstruction, passionate construction, webbed connections, and hope for transformation o f systems o f knowledge and ways o f seeing" (ibid.) . Th i s enables discourses to be heard as historical and cultural real ism; as situated knowledges within activities. In discerning a discourse that wou ld guide m y re/searching, I developed and then engaged i n the l i v ing practice of the fo l lowing research claims: • that subjectivity and partiality are inherent in a l l social science research activities (C l i f fo rd & Marcus , 1986); • that historical contexts are integral aspects o f the condition for useful understandings o f the re/searched activity; • that voices o f participants are heard as they make sense o f their experiences wi th in the re/searched activity; • that there is no 'one true story' . Supported by the work o f Peirce (1995), this research is collaborative and cr i t ical , reflexive and interpretive, and works to engage al l participants in the constituting o f useful understandings o f integrative evaluation. Participants inform each other w i th their insights and thoughtfulness as part of a dialogic process that builds, reconstructs, or alters the current context. Throughout this project I have refrained from pr iv i leging scientific assumptions o f objectivity and neutrality that presume to lead to the validity and reliabili ty o f this research. I acknowledge that m y collect ion and analysis o f the discourse(s) are social situations and therefore contain traces o f my o w n subjective discourse(s). T a k i n g this stance, and guided by the above cla ims, I entered this work cautiously and thoughtfully and now argue i n defence o f its trustworthiness. Its Trustworthiness In questioning this project's validity, the term validity was reconsidered. I asked, are the discourse(s) o f teachers marking reliable? D o they represent truths o f knowledge where there is a direct and transparent l ink between word and thought that is easily recognised and validated? Lather (1995) calls into question the traditional notion o f val idi ty 99 as absolute and categorical and attempts to "reframe validity as mult iple, partial, endlessly deferred" (p. 38). If val idi ty is an impossibi l i ty as Lather posits, then questions must be asked about the validity o f researching integrative evaluation, and o f qualitative research itself. L i n c o l n and G u b a (1985) state that traditional notions o f val idi ty and rel iabil i ty for qualitative research are inappropriate. Instead, the researcher should advocate for the value and logic o f their research project in its credibili ty, transferability, dependability and confirmabil i ty. Th is project's credibil i ty lies in the use o f multiple points o f v iew and reciprocity. O n the importance o f multiple points o f v iew, Goldman-Segal l (1995) comments, W e recognise the internal "strength" (yalidus i n La t in ) o f a report ing, not on ly by its rhetorical abi l i ty to persuade, its compe l l i ng author/ity, and its exc lus ive use o f canon and genre, but by its abi l i ty to bend, to be resil ient, and to be reconfigured into new groupings. Layers on layers. Research also gains strength by br inging together both the discordant and the harmonious. It gains strength by providing a forum for variance and diversity, (p. 3) It is a way to address the problematics o f re/searching "others". I acknowledge that the data as text is a lways 'read' and therefore does not speak for itself, and, as a ' reading ' , interpretive traces of m y point of view are always present. It was therefore important to give back to the participants how the data was v iewed and to check the validus o f the analysis. The debriefing sessions, a form of reciprocity, were offered as a platform to engage i n what Go ldman-Sega l l calls "mul t i - logu ing" where participants (researcher and subjects) i n this project were invi ted to reflect on the anonymous discourses collected (their o w n and others) and engage in mult iple layers o f commentary. Significant aspects o f those mult iple perspectives were then negotiated and became a co-constructed report o f the findings. The inqui ry then became "credible to the constructors o f the mult iple realit ies" ( L i n c o l n & Guba , 1985, p. 296). Transferability, however, is more problematic because it involves two kinds o f decision making - the researcher making decisions on the wri t ing o f mult iple realities that have been mult i - logged, and readers deciding whether or not to transfer the useful understandings to a new context. Is the new context s imilar enough to the one reported that there can be transference? B y articulating the theoretical framework and g iv ing thoughtful description to the context and re/search activities of this project, readers w i l l more readily 100 determine its transferability. Th is project is not disguised as reproducible. M y focus was to record the active discourse constructions o f the complexities o f ' do ing ' integrative evaluation and adapt the strategies I used to capture the discourse as it occurred. This cannot be replicated. Instead, I have noted jarr ing moments, those moments o f unexpected acts, commentaries or accounts o f integrative evaluation, and I have documented any adaptations of the research-as-planned as they took place throughout the project. In any qualitative study invo lv ing human subjects, the research is taking place in an always changing social wor ld and makes dependability o f the project problematic. Resis t ing the possibili ty o f objectivity of the researcher, I used activities o f reciprocity and encouraged a dialogue o f reflexivity as two strategies to address the concerns o f confirmabil i ty. Participants conversed with each other and wi th me whi le evaluating students' texts and whi le reviewing m y reportings o f the data analysis. They reconstructed wi th me, through the debriefing sessions, to locate useful understandings o f our situated knowledges o f integrative evaluation. Recognis ing that our subjectivities continually shaped this project, these strategies were significant i n their work o f confirmabil i ty. Its Obligation Obligat ion , that is, m y responsibilities as a social researcher to those invo lved i n this project, is integral to its legitimation. The main form for addressing obligat ion o f this project, as is c o m m o n wi th other social research projects, is informed consent. 7 6 Informed consent, whi le an act o f well-intentioned 'ethics-in-the-name-of-the-good', must deal wi th "that tenuous and delicate situation" o f decis ion-making "addressed by the name 'undecidabi l i t ies '" (Caputo, 1993, p. 3) that veils a presumed decisive clarity invo lved i n signing the form. K n o w i n g the 'vulnerabi l i ty ' o f obligation, I step carefully through a process o f provid ing information o f the study to prospective participants, rev iewing it wi th them, answering their questions to the best o f m y knowledge, whi le being mindful that their participation was intended to be voluntary. I attempted to establish a context i n w h i c h 7 6 Traditionally, questions of ethicality are addressed in researcher practices. Instead, I support the thoughtful arguments of Caputo (1993) and his writing "against ethics", outlined in more detail in chapter six. Hence I turn to a discussion of my obligations, rather than presumed possibilities of ethicality, as researcher to the participants. 101 the participants had the choice to take part, and the self-determination to withdraw themselves from the project at any point without effect. However we l l intentioned i n what was written i n the informed consent and the discussions that surrounded it, when I began the research, the many obligatory concerns, inherent i n any qualitative research, emerged and required on-going thoughtful (re)consideration. M y attention to the rights and values o f the participants whi le seeking not the ' truth' but useful understandings o f the inquiry was a delicate process. In thinking through what I was seeking, I anticipated that I may be less than satisfied wi th the fairness and consistency of the evaluative practices o f teachers I wou ld encounter. R o m a n (1992), i n discussing the feminist post-modern practices, raises the question o f means-end research. 7 7 It reaffirmed for me that the purpose o f this research was not to have the teachers learn to do integrative evaluation the way I 'wanted it done' . I thoughtfully considered m y framework o f understanding integrative evaluation and the values I br ing to marking students' essays and was mindfu l not to insert them into this inquiry i n any significant way. I was careful not to dominate wi th what may have been possibly perceived as a voice of authority when I interacted wi th the participants as they marked. I was attentive i n offering suggestions whi le honouring their contributions as va l id and worthy i n our conversations. K n o w i n g also that obligatory concerns often arise from the methods used, I thoughtfully considered aspects o f obligation at each stage o f the research. There was always the potential o f naming, o f 'othering' , the participants. I actively contested this i n m y journal wr i t ing . Embedded in the naming or label l ing process is also a possibi l i ty for judging the participants, especially when such close attention is paid to their discourses. F ine (1994) questions what she calls the colonis ing discourse o f othering and writes about the relationship between subjects and researcher as work ing the hyphen, "that is , unpacking notions o f scientific neutrality, universal truths, and researcher dispassion" and invites us to "bra id cr i t ica l and contextual struggle back into our texts" (p. 71). T o work the hyphen, I self-consciously questioned the influence of m y historical relationship wi th the participants and how that might be encoded in their discourse. A s w e l l , I asked, how is what I k n o w of them as 'teachers' (their histories o f learning about E n g l i s h and subject matter) inf luencing 7 7 Means and end talk is described by Roman (1992) as having the potential to redirect the project, both consciously and unconsciously, to a desired outcome. 102 the questions I ask and the comments I make i n the dialogic process o f inqui ry? I wondered, w i l l teachers practice what they tell me as researcher? W i l l there be a dominant discourse? I f so, whose w i l l it be? These were some o f the ' jarr ing' moments, those moments o f m y unanticipated accord and discord i n what I observed and i n the conversations i n w h i c h I participated, as the research process unfolded. 7 8 A s researcher and teacher, I explored m y teacher knowledge and practices reflexively by journal is ing jarr ing moments o f disruption. This was one way to account for m y subjectivities and interested knowledge and used to check whose and what knowledge is legitimated and how new knowledge was constituted. In the act o f re/searching, I acknowledged the relationships and the subjectivities wi th in the context o f this inquiry that I was re/searching to understand. Understanding integrative evaluation through discursive instances o f teachers' practices, each time bui ld ing on past 'readings' and transforming how it can now be understood to be, requires co-specifying relations between the subjects and the researcher, as w e l l as the contextual factors such as the participants, history o f teaching, and the school culture. These were important histories that shape who we are, how we think, what we say and do. A s a researcher I tried to examine cri t ically my relationships wi th other, to k n o w I was not a distanced and neutral observer, but an integral other being researched i n the process o f this inqui ry . A number o f aporetic sites emerged in teachers' practices. The fo l lowing chapter explores those aporetic sites through the discourses o f teachers' doing integrative evaluation. A s interpretive work, I wanted to remain open to generative spaces that educe useful understandings o f difficult relations o f evaluation, literacy, and language/content integration. Jarring moments are described as those times when the unexpected is heard. 103 Chapter Five Troubled Terrain: Interpreting Instances of Integrative Evaluation The same thinkers and theorists ( including us all) move freely f rom one side o f an opposi t ion to the other, as practical constraints or the requirements o f argument demand. The same teachers who espouse the virtues o f c h i l d -centred education, o f learning by discovery and o f the real isat ion o f innate potential also know i n advance what w i l l be discovered, prepare their lessons accord ing to set books and syllabuses, and prepare their students for the knowledge that is to come. M i c h a e l B i l l i g et al . , Ideological Dilemmas B u t there was nothing easy, l inear, predictable or necessari ly successful about any o f our d i s rup t ions o f e a c h o the r ' s a s s u m p t i o n s and interpretations. The teachers' and m y o w n disruptions o f one another 's f i x e d not ions o f what was " g o i n g o n " i n their c lass rooms . . . b e l i e d romant ic ised ideals. The disruptions o f m y data interpretations, o f any one "researcher" and/or "teacher" role and o f one another's perspectives. . .also interrupted any easy or smooth l i v e d relat ionship . . . . D is rup t ions , then, were a da i ly reminder that the relationships and methodologies o f teaching and o f . . . research are ambiguous, contingent, and shift ing. A n d research, w i t h i n and because o f disruptions across power posi t ions , interests, and goals, is a l i v ing and constantly changing practice. Janet M i l l e r , Disruptions in the field: An academic's lived practice with classroom teachers The extent to wh ich it is possible to separate language and content, ideas and expression, remains everywhere an unresolved problem. B o n n i e Nor ton & Sue Starfield, Covert language assessment in academic writing 104 Chapter five explores interpretative work emerging from discourse(s) o f teachers' integrative evaluation practices. First, the aporetic sites of discursive instances are mapped as a way o f understanding integrative evaluation. Then, three "stuck places" are probed: 1) creating and interpreting prompts; 2) establishing and using evaluative criteria; and 3) difficulties pr ior to deciding a mark or grade. In the third "stuck place", I also interweave learner voices wi th teachers' discourse(s) to explore the effects o f positionalities on practices. The approach I use considers the discourse(s) i n terms o f reflecting-on-action (accounts), reflecting-in-action (accounts, commentary and acts) and action (acts as decisions). In this way, traces o f meta-texts, positionalities and their relations can be fo l lowed through teachers' discursive practices o f integrative evaluation. In one o f the last sections o f this chapter, I explore teachers' discourse(s) o f language/content relations specific to two E L L compositions. I then offer a way to consider language/content integration i n E L L composi t ion that builds on the promise o f a functional orientation to l inguist ics. Another Instance Three teachers came together for purposes o f grading essays on 'mul t icu l tura l i sm i n Canada ' written by Eng l i sh language learners i n their classrooms. They reviewed the criteria (stated as structure, organisation, and main points related to the topic) and began the shared reading and judging o f one paper. The fo l lowing is an instance o f the discussion that ensued. Tl: This paper stays with the topic but there are so many errors in grammar. I'd give it a low mark. T 2 : It does have errors but it is descriptive of multiculturalism and does state the main issues. I think its worth more. T 3 : But, look what he's doing, he's offering a very thoughtful response to the question. Tl: But we can't ignore the grammar. Different readings o f the text led to 'disruptions' o f different value commitments . Difference was played out in the tensions o f teachers' practices. Meta-texts o f evaluation, literacy, and language/content intertwined, locating sites o f agon wi thin and among those discourse(s). It was a messiness, a non-linear "stuck place", o f judg ing wr i t ing that the three teachers worked to 'clean up' and f ind their way through. In general, judging E L L compositions seems to be a space o f agon. Ye t , at a 105 workshop presentation I and my colleagues gave on the difficulty o f integrative evaluation, one participant stated, I don't see what the problem is. You just have to be clear on your marking criteria by identifying the language you have taught and expect the students to use and know the content you want in their response and then mark for it. It's not difficult. You can be consistent and objective, you just have to be clear and get it right. (a delegate's response, International T E S O L Convent ion, F lo r ida , 03/1997) The agon o f her espoused theory was noticeably absent. W a s it a tensionless space or was tension concealed wi th in evaluation practices? If disruptions are consistently evident i n discourses o f teachers marking , can so many not be "getting it r ight"? I argue that it is not quite so simple as the conference participant suggests. The complexit ies o f intertextual relations played out in evaluative practices are significant. Interpretations invo lved i n the decision o f a mark are made even more complex wi th judging wri t ing that is doubled i n languages and cultures. A g o n and "stuck places" seem to be integral aspects o f the pre-conditions o f evaluative decisions. This chapter is an attempt to analyse, as an e tymological "undo ing" or " loosening" , teachers' discursive integrative evaluation practices that d w e l l i n "stuck places", agons, and felt messy texts. Integrative evaluation is an activity constituted in discourses o f time, place, social relations, and positionalities. It is within this whole activity that multiple voices, or discourses, emerge and crucial evaluative decisions get made. W h i l e recognising that time, place, and social relations contribute significantly to the complexi ty and uncertainty o f evaluating E L L inscriptions, of particular interest i n this inquiry are relations o f meta-texts o f evaluation, o f literacy and of language/content and the various orientations constructed wi th in those meta-texts. M y purpose is to interpret the difficulties using traces o f positionalities that frequently col l ide and disrupt i n ambiguous, di lemmatic tensions i n teachers' practices. " A g o n " , "stuck places" and messy texts emerged through the folds o f discourse(s). One teacher, "stopped i n her tracks" (Lather, 1998, p. 1), commented, 'This student didn't read the instructions carefully. I was expecting a very specific answer to this question. We studied it in class. He should know what I wanted. I think he's offering another view here but I'm not quite sure what he's saying. How am I supposed to mark this? 106 S/he struggled to judge a learner's work .wi th in a framework o f a planned cur r i cu lum o f what "we studied in class." Questions o f presumed objectivity o f evaluative practices and o f expectations o f knowledge as commodity arise. Ye t , socio-cultural imbrications o f constructing meaning seem to disrupt the possibil i ty o f objectivity. Expectations, influenced by how we understand what it means to learn and by expectations o f what it means to demonstrate that learning, are constituted in wri t ing what we know. "He should know what I wanted" assumes a stance o f teacher as authority, as "one who knows . " D i d the learner not have something o f value to say i n his o w n words? W a s there value only i n the teacher's words? Evaluative practices seem to lay-in-wait for opportunities to entrap students in intentions of fairness and consistency; instructions and criteria so clearly l a id out that there is only one possible answer. Or , is there? Diff icul t ies , caught up i n al l o f this and more, are central to this exploration of the discourse(s) o f integrative evaluation practices. M a p p i n g Troubled Terrains The activity o f evaluating written tasks is a regular practice o f classroom teachers i n academic contexts, yet we know little o f how this activity gets structured nor o f specific "stuck places", those sites o f aporias where complex intertextual relations become entangled and "stop[s] us i n our tracks." Participants i n this inquiry , teachers at a Canadian international post-secondary college, engaged in practices o f integrative evaluation w h i c h I then mapped from conversations and 'being there'. The mapping i n Figure 5, was reconstituted through two sessions o f reciprocity. Unremarkable i n its framing, the process involves remarkably complex pre-conditions o f decision-making, as teachers w o r k wi th in (in)visible and fissured positionalities that effect their work. The process o f first establishing a prompt and then deciding on criteria before judging and offering evaluative feedback appears to be routine, commonsense aspects o f an often unquestioned process that, when restored to " l i fe ' s or iginal diff icul ty", disrupts a s imple linear notion o f the evaluative process and the work o f criteria. Evaluative practices instead seem to include various and partial acts o f (re)reading texts, comparing responses, reviewing a l l papers 107 before commit t ing to individual marks and giv ing evaluative feedback. Teachers in this inquiry constructed their practices in multiple ways; practices wi th in and among teachers were realised in recursive movements and unpredictable differences. M a r k i n g became recursive for teachers as they circ led back returning to that wh ich came before. F o r example, discourse(s) o f teachers included such comments as, Reflection-in-action:discussing the process while marking (COMMENTARY, ACTS) •I can't give a mark yet. I need to go back and check the criteria. •I am writing comments about their organisation problems. Now I have to be clear on why I gave it a six. I need to read through it again to be clear on the six. Another teacher enacts recursive movements when she says, Reflecting-in-action:discussing the process while marking (COMMENTARY) •I can't decide whether it's a sixteen or seventeen. I need to reread the things we listed to watch for. Okay, that helps. No, wait. Where's the assignment instructions? How did we tell the student what was important? How close is my list to the instructions? Hmm. Pretty close. Good. Now how did the student take care of these things? The same teacher's movements, when marking another student's essay, were linear. W i t h the prompt i n front o f him/her as a guide, s/he proceeded to judge the text i n indel ible ink after one reading; there was no turning back. S/he explained, Reflecting-in-action:discussing the process while marking (COMMENTARY) •I haven't got time to reread and I know this student. I have one hour set aside to mark half the class. Issues o f t ime and familiarity, although not explored in this inquiry, are acknowledged as influential i n how evaluative processes get done. 108 C ' a & 03 EC c c > ir. <= c c „ J : 'X. '~ •- so i : 'E. — .o T5 U 'J > 2 O ! J c c C / 2 O o 1 1 c c y 2 ~ 109 In the discursive folds o f pre-decision-making, uncertainty, ambiguity and ambivalence contribute to the difficulties o f deciding a mark. W h i l e reflecting-in-action, several teachers i n different contexts independently commented, Reflecting-in-action:discussing the process while marking (COMMENTARY) •I don't like marking writing. I want to change this criteria now that I am reading what they wrote. The criteria seemed clear when we started but now as I read this answer I'm not sure how to use the criteria and be fair to this student. 'This is why I don't like to give a mark. I never know exactly what they 're saying so how can I know what they know? •Sometimes I get so tired of fighting with the writing, I'm just giving my best guess at a mark. In order to address what mediates and complicates positionalities among evaluators and texts, I turned to what seem to be aporias or "stuck places" as a locat ion o f d i lemmat ic discourses i n teachers' practices o f integrative evaluation. Once located, I traced confl ict ing and fragmented positions to make sense o f multifarious practices teachers engage i n to w o r k their way through "stuck places." A t crucia l points o f dec is ion-making, I listened for dominant discourses that seem to have imposed a particular way out. I worked wi th three sets o f data: teachers' discourses from authentic mark ing sessions, learners' discourses from authentic wr i t ing sessions, and teachers' discourses f rom non-authentic mark ing sessions. "Non-authent ic" sessions were deemed those i n w h i c h teachers marked content written by learners they d id not teach. The conditions for the non-authentic discourses were determined by the inquiry process; teachers had no knowledge o f previous teaching and learning activities o f the writers they judged. "Authen t i c" sessions were those i n wh ich teachers marked the written content o f learners they taught. Trac ing disruptions pr imari ly focused on teachers' discourses i n authentic mark ing sessions. Discourses from authentic sessions w i l l be considered first. In particular, three sites o f tensioned "stuck places" seem evident i n the discourse from ind iv idua l and collaborative authentic mark ing sessions: creating prompts, (re)turning to criteria, and the felt messiness o f inscr ib ing a mark o f indel ible ink. The second set o f data i nvo lved "non-authentic" sessions i n w h i c h teachers' conversations were specific to fourteen E L L composit ions, two o f w h i c h are highlighted in this chapter. 110 Site One: Creating Prompts for Language/Content Texts The part o f the evaluative process that is o f interest i n this inquiry begins wi th teachers creating a wri t ing prompt to wh ich students w o u l d respond. General ly , the prompt structures the task and includes a stimulus (a question or statement), the conditions under w h i c h learners write, and indicates the expectations o f the teacher as aspects o f the response that w i l l be valued ( K r o l l & Re id , 1994). Dur ing discussions o f this part o f the evaluative process, several teachers in varying contexts independently offered, Reflecting-on-action: how a prompt should be structured (ACCOUNT) •First thing I do is think about what we've studied in class. Then I think about the kind of work I expect from them [learners]. Then I'm ready to write out the assignment. I always give them the instructions and tell them how to organise their answer before I let them work on it. •The question needs to be something familiar to them [learners] - especially the vocabulary. •They [learners] always ask me what the assignment is worth so now I have to think about it ahead of time and include that information in the instructions. The pre-conditions o f a ' f ina l ' version o f the prompt involves numerous sub-decisions; decisions o f how questions, instructions, and values get constructed. The fo l lowing instances were collected when four teachers collaborated on essay questions to w h i c h a l l students taking a second year global education course w o u l d respond. Reflecting-in-action: commenting on the prompt while structuring it (COMMENTARY) Tl: There should be choice. T 2 : Instructions should be clear stating the timeframe. T 3 : What about how the essays will be marked. I think that's important. We need to be really clear. T 2 : Ya, we can list the language items and points from the content that we're looking for. T 4 : Even things like double spacing and where to put their name and number. T l : Most of this is fairly routine and straightforward. Shouldn 't take us long. A s they set about their collective task, commentary on the prompt seemed unproblematic for this group; agreement on generalities o f what should be included was "routine and straightforward." In separating "language items and points from the content", two lists were formed. A s teachers continued their discussion and began to write and then comment on the prompt, their discourse fragments suggested uncertainty, ambiguity and, for one, ambivalence. In the structuring o f one o f the essay questions, di lemmatic tensions emerged: 111 Reflecting-in-action:commenting on the prompt while structuring it (COMMENTARY) Tl: Here's the instructions for last year's final exam questions - "Answer all three questions in paragraph form. All questions will be marked for language and content. " They wrote two essays, really they were paragraphs, in 90 minutes. Question one we can probably use as is because we all did the United Nations. R: What was the question? T l : Describe the structures and purposes of the six main organs of the United Nations. T 2 : So if they're going to describe, they're just going to tell us back what the purposes are and what each main organ's responsibility is. We 're not asking them to do anything with it? Like write for new meaning, you know, discover something new? T l : /Vo, not for that one. Does it matter? T 2 : What about giving reasons for why, make them think more not just memorise the answer... encourage them to be creative, find new meaning in their own words. Shouldn't this be rewritten? T 3 : Ya, I was thinking we should ask for more about reasons for the connections between things but I wasn 't clear on the answer myself so I didn't think we could ask the students. T 4 : What exactly do we want from them? We worked on defining the organs. It would be fair to ask for something about defining the organs. T 2 : My point is, do we just want something back or are we asking them to come up with something new - like a new way of looking at the connection? Make their writing work for them in different ways. Teachers seem to be uncertain wi th what is expected i n the response - a return o f the ideas discussed i n class or a turn to new experiences o f connections and relations. One teacher seemed to be concerned wi th his/her o w n ambiguity i n responding to a question on relations and was resistant to restructuring the question to invite unknown responses. Another seemed ambivalent i n the concern for the k ind o f response they were attempting to elicit . They continued the struggle as they reworked the prompt question: Action: structuring the prompt (ACTS ) T 2 : It should be "How did the six organs of the United Nations try to meet the four purposes and seven principles?" T l : So, you want to add that to it. T 3 : Well, I don't know. We also talked about the two goals of peace and security, and dignity. Maybe that's all we need to write down. T4:1 think we ask them to link the organs to the two main goals. T l : OK, how about define one organ and link it to a goal? Would there be the possibility of more than one answer? More than one way to link an organ to a goal? T 3 : Good point. Will we accept all answers as long as they give reasons? T 2 : Well, its easier to explain to the students if we can say these three reasons must be in the answer. Then, maybe we can give them some freedom. We'll just mark for the three reasons and a little life in the paper. T4:1 like that. (they all nod in agreement) 112 Decis ions led to further indecision and then a change o f decision i n the collaborative process. Teachers came wi th questions they had decided w o u l d be appropriate but were w i l l i n g to reconsider when consensus could not be reached about the subject matter. W h a t is evident throughout the conversation is that content was privileged. Expect ing three reasons as the 'markable ' substantive aspect o f the essay and ignor ing what other substance the students might have constituted i n wri t ing was offered as a viable way out o f this "stuck place." There was no mention o f language expectations i n the first thirty minutes o f teacher interaction. W h e n asked for further comments on the k i n d o f response expected, the discussion continued to focus on substantive aspects o f the response and how to el ic i t them. Reflecting-in-action:commenting on the prompt while structuring it (COMMENTARY) R: What kind of a response are you expecting? T 4 : Well, they should talk about what each main organ did in relation to peace and security. They'd have to describe the organ first. T 3 : Can we ask how the organ is set up? T 4 : Well how it is set up is not really related to the goal of security. T 2 : 1 think it is but it would be hard to explain. I don't know if they could handle the language. T: 1 Why don't we stay with the original question. It's clear, we know what the answer is. T 3 : I'd rather change it to be more challenging but I don't know how to word it to get exactly what we're looking for. T 2 : How about, "How do these organs meet or work towards these goals?" T l : One or both of these goals. I think it would be too difficult for my students to take the economic and social council and link it to peace. That's complicated. How would they write that? T 2 : More importantly, how would we mark it? T 3 : Don't we need to find out what has been meaningful learning to them? T 4 : What if we worded it in such a way that they would only have to outline a connection of one of the organs to one of the goals - peace and security maybe? T 3 : 1 still think it has to link to something meaningful to them. Were not giving them any space to do that. Structuring the prompt to elicit texts that were 'markable ' , texts that students could 'handle ' , seem to be important and impose constraints i n this negotiation. N o w fifty minutes into the prompt planning, the teachers had not mediated question one, instead they seemingly made it more complicated. W o r k i n g their way through this "stuck place" became a recursive track for this group o f teachers. E v e n for the resistant ones, those interested i n a k i n d o f performative wri t ing wherein there is ' l i f e ' and meaningful personal connections constituted i n the students own words, positionalities complicated consensus. 113 The way out for the group was a return to the original question: Action: structuring the prompt (ACTS) . T 3 : Ya, I agree. T 4 : It can be done, "How is the security council's responsibilities and functions related to the general goal of peace and security? T 2 : This is getting so complicated. How will they know exactly what we want? How will I know what specifically to look for, there's so much they could say. T 3 : Or not say if they don't understand the question. T 1 : Ya, what is the right answer to that question because I'm not sure? Couldn 't we just reconsider the original question? T 2 : 1 think so. I feel I know what to expect with that one (the others agree). In an attempt to rework the first question, discussions dealing wi th the substantive aspect o f the question lasted 75 minutes. Thei r accounts o f structuring prompts suggested this w o r k was "routine and straightforward." The commentary on wri t ing a particular prompt suggests that this work is much more complex. Negotiat ing amongst teachers, midst di lemmas o f uncertain, ambiguous, and ambivalent expectations o f students, raises questions o f markable learning: how it might be elicited, what it might look l ike , and how one might go about judg ing it. It was a space where teachers were momentari ly "stopped i n their tracks" and explored other possibilities, only to determine the way out o f this particular situation was to return to what they ' knew ' ; what was for them, a predictable response o f commodi ty . Individual teachers creating prompts struggled i n s imilar tensions. Media t ions and complicat ions arose wi th in internal conversations that erupted in commentaries such as, Reflecting-in-action:commenting on the prompt while structuring it (COMMENTARY) •I'm not happy with this question. Too much handholding. Yet, without spelling it out for them, how do I tell them what I am expecting in the essay? •If I write the question this way, they may get confused... but I don't know how else to say it. •I like the question on international debt and know what kind of response I should expect but I don't know how to word the evaluation part on this assignment. I'm tired of dividing language and content, ten for each. Although you know, they never question me on that. I want to move them . .. or maybe me .. . beyond subject-verb agreement, I'm just not sure how. •I want them to move ahead with their thinking on the causes of social problems . . . come up with some new thoughts not just tell me what I said in class. It's more than a synthesis of ideas. And, what do I say to them about evaluation that they will understand? I have to be concrete and yet I can't anticipate how they might respond. I guess stay with the basics of organisation and grammatical accuracy. 114 •I have to keep reminding myself to keep the questions simple, with vocabulary they know. They are ESL. Hmmm, but, at the same time I want to push them to think about these political issues. Yet, it seems the harder I push, the more problems they have with language. You can't win, the harder the content questions, the more errors in their language; simpler the content, the language errors become more obvious. Disrupt ions i n the structuring o f questions emanated i n incommensurate discourses fixated on concerns o f degrees o f substantive complexity, performativity, what it means to be " E S L " , language expectations, and how to inform the students o f what w i l l be valued i n a response that cou ld not be predicted. D u r i n g this conversation on prompts, meta-texts o f both evaluation and surface aspects o f literacy seem to be prominent whi le comments on language are silent. Pr ior i ty is given to specific content and its readability (hence predictability) in relation to how 'markable ' responses w o u l d be. In meta-texts o f evaluation, tensioned commentaries on knowledge (knowing, learning) as transmission or transformation seem to col l ide , fragment and re-emerge i n meta-texts of literacy wherein a text's readability seems to be influenced by both possibil i t ies and predictabilities o f transmissiveness and transformation. F o r some, it seems the more transformative the anticipated response is, the more unreadable and therefore 'unmarkable ' it becomes. Fo r others, it appears that transmissive knowledge is disturbing i n its lack o f thoughtfulness and indulgence i n mirror ing back what is already known . L i t t l e is said o f language/content relations. Perhaps i n the silence there is an assumption o f emptiness i n that words as empty vessels wait patiently for the thoughts they carry. W h a t is evident i n the meta-texts o f language is that for most, when engaged i n pre-determining evaluative purposes, language is separated from content. O n l y one resisted the ca l l momentari ly to separate the two and then called for an alternative v iew in considering language/content relations. Bu t in so doing, s/he acknowledges what is for her /him a pathless path, an impassable passage. The way out becomes a return to what is markable learning. A v isual analysis o f the pre-conditions o f decis ion-making was helpful i n tracking di lemmatic discourses. 7 9 C l a i m i n g that structuring a prompt is often a site o f contestation l ived out i n pre-conditions o f discursive incommensurabili ty, Figure 6 offers a visual 7 9 This is adapted from a work in progress by Mohan and his graduate students at The University of British Columbia. 115 analysis of alternatives, possible outcomes and the decision one group of teachers made as they collaboratively determined the prompt. It is a tracking of the pre-conditions and of the decision when one teacher asked, "Shouldn't the original question be rewritten?" Figure 6 Visual Analysis of Collaborative Decision-Making in a Specific Case of Structuring a Question for a Prompt. Shouldn't the original question be rewritten? alternatives: - original question - informing function possible outcomes & evaluation: ^ - elicit informing (-) predictable responses (+) markable learning (+) choice/decision: other - reasoning function - write to discover new meaning elicit reasoning (+) - unpredictable responses (-) - complex marking (-) ' return to original question, response is predictable and markable (valued) (valued) As teachers worked their way through and out of unsuspecting aporias and messy texts, orientations within meta-texts and meta-texts themselves seem to struggle for control. The question, Shouldn't the original question be rewritten?, was explicitly asked by one teacher in her/his search for an alternative question that would elicit reasoning and creativity in the anticipated response. As a working collaborative group, teachers weigh the possible outcomes before making a decision. Discourse evidence at crucial points of decision-making suggests traces of positionalities in conflict where one can be understood as dominating another. In this case, predictability and markability seem to be valued more than eliciting creativity in the response. Concerns with the 'unknown' response and how to deal 116 with its evaluation was discomforting. Living within what appears to be a predominant meta-text of evaluation when engaged in the structuring of prompts, traces of varying positionalities seem to embody prevailing fragments of a technical imaginary wherein predictability (transmissiveness, commodification and mastery of knowledge) and markability (where assumptions of a rational consciousness imply, T can mark what I know') become persuasively enacted. Resisted initially by some members of the collaborative group, their dialogic pursuit of a function in the prompt that would enact creativity in the response eventually returned to the comfort of 'objective' predictability. Fragments of disparate positionalities had both mediated and complicated intertextual relations of the collaborative group. Predictability and markability convincingly, perhaps even unknowingly, seemed to powerfully force them to return to the original question, one which legitimates learning as giving back that which one has been given, and to structure the prompt in a way that gives power to transmissive knowledge and its markability. Site Two: (Re)Turning to Criteria Site two locates the tensions experienced by teachers as they articulate what is valued and how that might be applied in the marking of ELL written content. It explores the pre-conditions in which teachers work towards a crucial decision point of fixating the paper at a mark or grade. Teachers revisit or create what are presumed to be concrete standards (criteria) on which to base their marking. Others grade without use of explicit, overt criteria. One teacher suggested, Reflecting-on-action: for the mark (ACCOUNT) •I think of my criteria first but I don't always use it. At times I just know the paper is a 'B'. With practice it gets easier to judge them [ELL written content] quickly without always going back and checking. Somehow you just know. An overt, routine terrain maps the process of marking ELL written content, yet in the midst of difficult practices covert acts and unsaid stories seem to perform in resistance to the linear stability presumed to be offered with criteria and grading. Concerned with issues of fairness and consistency, the following conversation took place when three teachers came together to mark common exam questions. Reflecting-in-action: reviewing the criteria just prior to applying it (COMMENTARY) Tl: I'm giving you copies of the criteria we agreed on earlier. Any questions? 117 T 2 : Looks good to me. Five for language, five for content. T 3 : Ya, five points under each heading. We are so organised, this should be easy. (others nod i n agreement) T 1 : We've got 54 papers to mark. Just read through them and put them in one of the ten piles. Then we'll go through the piles, (five minutes pass) T l : Hmmm. Fm having difficulty here. What's the difference between the language of description under language criteria and describe Canada's multicultural situation under the content criteria. Do we give two marks for doing one thing? I don't know what's the difference. E a c h teacher work ing from the same marking scheme presumed to easily divide language and content, w i th "five points under each heading." Setting specific criteria under separate headings o f language and o f content seemed to be for them commonsense and objectively manageable. A theoretical plan was in place. Yet , only minutes into the marking session, uncertainty emerged in interpreting the criteria. In absence o f the present papers, the established criteria had been accepted as a valued imaginary, yet i n its application at least one teacher was i n the midst o f a "stuck place". W a s "I don't know what's the difference" a moment o f impossible language/content separation, a disruption for the teacher? The criteria for language and content had been separated (as in a formal v iew) prior to marking, but in practice the criteria were difficult to separate (as in a functional v iew) . Fragments o f two models o f language enacted i n disruptions o f what was valued and how integrative evaluation was now understood to be. In another case, four teachers agreed on the fo l lowing marking scheme for an assignment that asked students to write one multicultural issue i n Canada. They c la imed it reflected the current focus of the teaching/learning experiences i n their classrooms: The teachers worked wi th in a system they established for themselves. The criteria targeted content vocabulary, content structured as perspectives, and two rules o f Eng l i sh usage focussed at sentence level grammar. Does the list imp ly a separation o f language and content, the first two as criteria of content and the second two criteria o f language or are these criteria o f ' language' both formal and functional? In their account o f the criteria, Criteria for marking essay (25 marks): 1. Uses multicultural vocabulary 2. Expresses at least two different perspectives in their own words 3. Applies the rules of subject-verb agreement 4. Applies the rules of article usage 10 5 5 5 118 teachers assumed shared meaning as they briefly discussed and agreed on the contents o f the list and its application. Yet , i n practice, their commitment to the established criteria sometimes wavered. In one commentary, a teacher explained, Reflecting-in-action: discussions of applying criteria to specific papers (COMMENTARY) •7 think I know what she means here, but she's used the term diversity incorrectly. She does give two perspectives but I have to fill in the gaps with the second one before I could mark it correct. . . several grammar errors with articles. She uses three multicultural words. A student's words appropriated by the words o f the teacher. The form o f at least one perspective was altered, re-formed to what was expected; i n this case, wri t ing "in their own words " (as stated i n the criteria) d id not signify that wh ich the teacher knew and expected i n the response. The response was altered i n order to "mark it correct". Another teacher from the group commented, Reflecting-in-action: discussions of applying criteria to specific papers (COMMENTARY) •The language is fairly easy to mark. You can see the mistakes. It's the content that's hard to access, the language gets in the way. The language is, in some ways I guess, the meaning and I don't know what to do with that. How they use the language to write the perspectives has a whole lot to do with what meaning gets made. I tried to divide up the ten marks into language and content but I couldn 't do it. Diff icul t ies are evident as the teacher makes sense o f language/content relations. The last two criteria on the list suggest formal aspects o f language evaluated at the sentence level . The first two imply a functional v iew wherein content vocabulary and the language that structures perspectives contribute significantly to inscribed meanings. W i t h i n one set o f criteria, it appears that two models of language may be operating. The formal v iew, as i n criteria #3 and #4, is presumably objective and concrete enough not to be contested i n this specific case. However , the criteria on perspectives, operating at the level o f discourse, is contested; i n attempting to separate language and content, the teacher was 'stopped i n his/her tracks' because s/he "couldn't do it." A r e language and content relations independent? interdependent? Is meaning altered when we "fill in the missing words'"! D o disruptions wi th in the mark ing scheme effect judgment? A n account by one teacher offered, 119 Reflecting-in-action: discussions of applying criteria to specific papers (ACCOUNT, COMMENTARY) •I know language and content are integrated but I separate them for marking purposes. See on this one, I give each part a mark then I add them up. It's easier that way for me and for the students to understand. His/her marking scheme aligns a set o f criteria for language and another set o f criteria for content. S imi la r ly , another teacher commented, Reflecting-in-action: discussions of applying criteria to specific papers (ACCOUNT, COMMENTARY) •Each of these arguments has at least two main details, so I give one mark for each . . . 7 usually leave about one mark for the overall organisation, one for the spelling of key words. The words ' language' and 'content' write a wholeness or unity through the additive conjunction o f ' and ' . Ye t , for many there is a sense of ambivalence o f meanings, a co-existence o f opposite poles. W i t h i n the phrase itself, a duality or polarity is easily established that seems to position the teacher i n a model o f language and content that assumes a fragmented structure; a model i n wh ich the 'and ' behaves disjunctively - a disjunctive conjunction. The practice o f fragmenting language/content criteria suggests a v i ew o f separateness, one set o f criteria unrelated to the other. Questions surface i n enacted presumptions o f language and content. D o the sum o f the parts equal, or are they different from, the judgment o f the whole? Is language and content a commensurate or incommensurate discourse, subdivided and fragmented, or integrated and whole? Integrating language and content is complex and, as a meta-text, is ambiguous, contested, and seemingly little understood by many teachers. Instead, a separation occurs when "I give each part a mark then I add them up " because "It's easier that way for me and for the students to understand." M u l t i p l e models o f language and content mediate and complicate the criteria that get established and applied within practices of integrative evaluation. A s I met wi th individual teachers marking E L L inscriptions, I asked them to expla in what criteria, i f any, they used to judge E L L texts and how they applied them. Some offered that the use o f key words studied i n class were expected i n student responses. W h e n asked whether this was a language or a content criterion or language and content criterion, some categorised it as language, others labelled words as content. One accounted for it as a language and content cri terion because "the words made the meaning". 1 2 0 Another, i n a conversation wi th the researcher (T, R ) while marking essays, insisted, Reflecting-on-action: on evaluative criteria (ACCOUNT) T : Use of specialised vocabulary tells me the student knows the content well. I don't pay too much attention to the language, you know, the grammar mistakes unless they really distract me, the message is in important words that express content information and students need to know and use them. R : Are these words criteria for marking language and content? T:Yes, I think so. That's what language and content means to me, using words loaded with meaning in their essay. Reflecting-in-action:on the criteria while engaged in it (COMMENTARY) T: You see in this essay the student used important words like "international debt", 'World Bank', 'crisis', 'north and south divide'. This gives substance to the essay. T: So, for you, this would be an example of a good paper? R : Well, yes but. . . no. Look at the grammar errors. You really have to be blind not to see them. Mostly articles and subject-verb agreement. Oh, they'll learn it as they write more. It takes a long time for them. I didn 't specify those grammar points for marks anyway so I'll just ignore them. I think about language in a general, overall way. In other words, I ask myself how hard was it for me to understand what they are saying. How many times did I get distracted by their language? Reflecting-in-action:on the criteria while engaged in it (COMMENTARY) T:What would I give this essay? I don't know. It's not so easy. I guess I should have been more specific for them and for me in what I was looking for. I just wanted the ideas, I thought the language wouldn 't matter. R:What is it about the language in this essay that does matter to you? T:Well, they are using the vocabulary words I was looking for - I've ticked seven of them in the introductory paragraph and that's really good - but they don't go together so well and that's distracting. I'm not sure what he knows. Some of the meaning is lost to errors. This is a tough one. My criteria don't work in the way I thought they would. This teacher expl ic i t ly states that words, as language, are valued when words are "loaded with meaning", suggesting words gain import when they "express content." Y e t , s/he does not value the specific structuring o f those words because "they don't go together well." D o words f i l l w i th meaning, as a formal model o f language w o u l d suggest? D o words constitute meaning, as a functional model o f language w o u l d c la im? Evidence o f traces o f a formal model o f language locates the teacher in a space where language becomes a vehicle for content, a carrier o f its message, and content words become valued criteria. The teacher's account o f the valued criteria heavily weight words "loaded with meaning.''' Li t t l e value is g iven to those other words that l ink 'content words ' and structure meaning. Y e t , this teacher indicates a discomfort in the way the "words loaded with meaning " are 121 structured that is "distracting", suggesting a trace o f a functional v i ew. There is expl ic i t cornmitment to language that is loaded wi th meaning. Language and content v iewed as content vocabulary is a paradox o f sorts. It is juxtaposed by two confl ic t ing ideas: 1) that language carries meaning, and 2) that language constructs meaning. Uncertainty reigns i n accounting for what language/content relations are and what that means. Evidence i n his/her commentary suggests a re-thinking o f the criteria, confirmed i n the discourses o f his/her active engagement in coming to terms wi th a mark based on an account o f criteria that s/he now realises "don't work" i n practice. M o s t teachers acknowledge the practical ease o f locating grammatical errors. Th is teacher, as others c l a i m to have done, reads the students as E L L , and offers refuge i n ignor ing these errors "unless they really distract me." S/he attempts to disregard sentence level grammatical errors i n written content, attributing them to the students' E L L n e s s . Wha t criteria then were valued and applied? D o the errors that remain, those errors o f distraction, become points o f faulty language and content integration? Points o f distraction l ive out i n significant and haunting uncertainty. W h i c h grammar errors distract? W h i c h errors are unobtrusive? A blurred threshold for grammatical errors o f greater and lesser import seems to have been established for this teacher. Is the threshold a commonly shared threshold for teachers marking E L L written content? D o al l teachers understand grammar i n this way? The stance taken seems to be a formal one o f grammar at the sentence level . Disruptions are easily located but ambiguously dealt wi th . It appears that the teacher's consideration for what grammatical meaning the writer is constructing may be at the level o f discourse, a functional perspective of language o f wh ich the teacher seems to be unaware. Uneasiness arises f rom the uncertainty o f which grammatical errors matter and w h i c h do not. F o r this teacher, language and content were only momentarily inseparable. Contested spaces of establishing criteria for the judgment o f language/content texts seem to emerge as commensurate and disparate fissured discourses. W h i l e considering what evaluative criteria wou ld be used to judge the response to a question written by an E L L she teaches, the teacher conversed wi th the researcher (T, R). A n instance f rom the conversation is offered. 122 Reflecting-in-action: on applying specific evaluative criteria (COMMENTARY) T:They have to explain the arguments or opposing viewpoints on Foreign Aid as presented by the two authors we studied in class. R:Are you looking for anything specifically? T:In each of these arguments, Vve talked to them, they know they have to have an introductory sentence that has something like, "Matsumura argues that. . .", or to use that formula, and they have to have a concluding sentence which says something like "therefore or because of this." I've talked about "Therefore, Matsumura argues that Japan should..." so I'm looking for that. Actually I'm looking for about four marks, one for the introduction and conclusion and then each of these arguments has at least two main details that they include. Reflecting-in-action: on the criteria while engaged in it (COMMENTARY) T:Before I started marking I took the tests and ran through them all clarifying in my mind what I expect and what I'm seeing in their answers. I need to go back and do that now with the ones I've marked. R.So you alter your expectations as you look over the papers each time? T:Sometimes. Ya, I think this mark is too high for the accuracy of the language. Action: applying the criteria (ACTS) T: Look at this one, the student has got the intro and conclusion but doesn't state the details of the argument, only makes general statements. I would give this two out of four. . . Hmmm, I think I'll put the mark in pencil and come back to it after I've read all of the responses to this question. In this instance, the teacher's account o f the criteria to be used seems certain, yet as she engages i n the marking process there is an altering o f the expectations to al ign more closely wi th the responses received. The teacher adds another, previously unmentioned criterion, the accuracy o f language use. A s s/he comes to decide on a mark, the use o f penci l signifies uncertainty and temporality, and i n so doing becomes an act o f (in)decision. A t this point in the mark ing o f E L L s ' written work, the disruptions o f practices are evident. The presumed valued criteria for this task, and written into the prompt, d id not work for this teacher and were therefore altered to account for another factor, the accuracy o f language, ini t ia l ly not inc luded as a criterion. A s a "stuck place", adding a criteria that was unacknowledged i n the prompt helped this teacher work through and find a way out. W i t h i n a meta-text of curriculum/evaluation, positionalities that support linearity are evident. The teacher relies on criteria that reflect what the learners have studied, what they have come-to-know through transmissive activities in the classroom. In reading the wri t ing, s/he seems to be look ing for a reflected image of what has been studied, a text that works to express ' the' meaning she's "looking for." Is it presumed that the reader does not 123 interpret the text in any other way than the writer intended, that there can be only one meaning o f the text and that one can know the intentions o f the writer? The teacher ini t ia l ly lists the criteria and, i n so doing, speaks o f language at the level o f discourse. Discourse structures o f 'argument' work to construct meaning and are valued cohesive devices students are expected to use in their responses. Bu t then, the agon o f accuracy o f language hauntingly enters and s/he is "stop[ped] in his/her tracks." The significance o f accuracy lingers and performs throughout the mapping o f decis ion-making. U s i n g the above discourse sample as a case of decision-making i n v o l v i n g criteria, Figure 7 offers a visual mapping o f discourse(s) o f the pre-conditions o f a particular decis ion regarding criteria, Should "accuracy of the language " be valued? A s this teacher impl ic i t ly ponders the question, s/he considers the alternative choice o f a criterion o f "accuracy o f the language" and weighs the possible outcomes wherein a decis ion is made. Trac ing dominant orientations at crucial points of decision-making, there is evidence that the added criterion, accuracy o f the language, overrides pre-established criteria. W i t h i n a meta-text o f language, a technical imaginary is impl ied wherein the mastery (accuracy) o f a language and its structure is possible, and discourse fragments o f a functional and a formal mode l o f language both mediate and complicate intertextual relations. A s a site o f troubled positionalities, the dominance of accuracy (mastery) compelled her/him to alter the pre-conditions and judge the E L L text in a way that privileges accuracy. Figure 7 V i s u a l M a p p i n g o f an Individual Case o f Dec id ing Evaluat ive Cri ter ia Shou ld "the accuracy o f language' ' be valued? alternatives: • 4 out o f 4 has introduction, conclusion, two main details • other add accuracy possible outcomes & evaluation • won ' t reflect accuracy o f language (-) • reflects accuracy (+) decision: mark altered to reflect accuracy o f language (valued) 124 Site Three: Difficult ies o f Judging Language/Content Texts W i t h i n each site, difficulties play out at important points o f decision o f the ' f i na l ' version o f the prompt, and the ' f inal ' version o f what w i l l be valued and how that w i l l be enacted and shared wi th the learner. I place single quotation marks around the w o r d ' f i na l ' to draw the reader's attention to motile interpretations. A l though the prompt, for example, once decided seems ' f i n a l ' , as it is one presumed shared intention that is g iven to the students, discourse evidence at site three - the felt messiness o f judging language and content texts - suggests that mult iple interpretations o f the ' f ina l ' prompt exist among and wi th in teachers, among and wi thin learners, and between teachers and learners. Therefore, I argue that the ' f i na l ' version is not a ' f ina l ' version at a l l ; understanding the act o f reading a prompt involves inquiry into the relations among forms, writers, readers and contexts o f reading. S imi la r ly , what is valued as a ' f ina l ' list o f criteria, seemingly fixated on a criterion referenced imaginary is presumed to be shared, unambiguous, and stable. Y e t , discourse data suggest evidence to the contrary and point to instability wi th in the pre-conditions o f crucia l decis ion-making points o f (re)marking the text. A d d i n g further complexi ty to an already complex site, I include discourse data from the learners that respond to the prompt i n wr i t ing , the texts to be judged. It is a doubled site o f agon, wherein both teachers and learners struggle to make sense o f the intentionalities o f each other. In reading the prompt, the students ask, what was the intention o f the teacher? In reading the response, the teacher asks, what was the intention of the writer? Reciprocal intentions often become sites o f aporias; d i lemmatic discourses disrupt and teachers once again bear witness to the agon o f integrative evaluation. The activities wi th in the pre-conditions o f the crucial decision o f mark ing and re-marking the essay - fixating it on a scale - varies among teachers. Some carefully review the prompt and place the criteria before them. Others begin by reading the essays to experience how learners have responded. M a n y reread the essays several times, others read a l l the essays once before returning to each one for a focussed evaluative decision. H o w ever they work their way through, at some point teachers must place a mark or grade on each E L L text. Pr ior to this crucial act, there seems to be a discomfort, a felt messiness tensioned by incommensurate discourses o f integrative evaluation. W i t h the prompt set and criteria 125 considered, certain assumptions prevail, Reflecting-on-action:on how evaluation should happen (ACCOUNT) •Just know what you asked them to do, know what you 're looking for and the rest should fall into place, you know, get it straightened out in your mind or on paper first and, if you've done your homework, it should work. If one attends to the criteria in determining the mark, then they w i l l "get things right." A v iew o f consumable knowledge sets the prompt and criteria. The teachers' relation to the prompt and criteria are presumed to be unambiguous and certain. Is this also assumed for students? Different students, once having read the prompt for their assignment, independently commented, Reflecting-on-action:on how learners approach the writing task (ACCOUNT) •The first thing I try do, know what teacher want me to say. Most I guess. I study note from my class so I know what. •Sometimes, I don't like question. It just repeat what study in class. I know more so I try to say but sometimes it worse. I can't say what I want in words I know. •I want to know how much worth. It tell me how hard I work. This one worth 25 mark so I should work very hard. I need know exactly what teacher want. I made mistake before in understanding question. I got low mark. For them, reciprocal intentions l ive in tensioned difference. L i f e as the student knows it and what the teacher "wants" mediates and complicates the written response. In acting on the importance o f determining intentions, students remain uncertain. They study their notes and make sense o f the prompt through the content that has been offered i n class rather than what they k n o w . It is a momentary "stuck place" and they find their way out through a w e l l -worn path o f pr iv i leging consumable knowledge. It seems judgments reflect how w e l l students read intentions rather than what they know of life. It seems to be an act o f interpreting life as corporeal, o f knowledge-as-s tem (Jardine, 1992), and assumptions o f mastery and autonomy; teachers intentions to "get it straightened out first" presume linear and objective evaluative practices and seem to ignore acts of reciprocal intentionality i n students' responses wherein the possibili ty of generativity, movement, and difference can be invoked. Just prior to the crucial moment o f putting indelible ink to paper, an account by one 126 teacher appears clear and decisive within the domain o f grade giving, Reflecting-in-action: on grading a specific paper (COMMENTARY) •I'm giving this paper one grade for both language and content. If I had to give a number for both, I couldn't do it, it's too difficult to explain to the student. But a grade I can give, I know the difference between an "A " and a "B " paper. W h e n conversing about receiving a mark or a grade, many students expressed indifference. Wha t seemed to be important to them was to understand why the mark or grade was g iven i f it was less than anticipated. One learner commented, Reflecting-on-action: on the meaning of marks(ACCOUNT) •I don't care number or grade, I just want high mark. I don't understand either, mostly I trust teacher. Except when it's low, then I want to know what I did wrong. How to improve my language to tell her more what I know. M a n y students seek high marks. A low number or grade fixates them on a scale f rom w h i c h they want to move. Movement comes i n knowing "what I did wrong." A s the teacher moves the pen into position to mark-for-life, s/he begins to question relations between language and content and how s/he might justify the grade. Reflecting-in-action: on grading a specific paper (COMMENTARY) •It's a "B" paper. They don't know enough about the topic. They've used some of the vocabulary but it doesn 't go anywhere. What I am going to say to the student. Why isn 't it an "A " ? Is the student's language a problem or is it his ideas that make the language awkward? I don't know. I don't like this. Just saying it's a "B" isn't enough. I want to be able to explain why. After spending an hour on his/her paper, another student shared, Reflecting-on-action:on the meaning of marks (ACCOUNT) •I want to tell her what I know. I am frustrated with English. I search deep inside for my ideas and then I can't tell them because the words are lost. The words we learned in class are not enough. This student speaks o f language and content relations i n a different way. It seems that meaning is lost i n the l imited resources s/he has in Eng l i sh . Fo r her/him, language is pr ivi leged i n construing ideas, impl ic i t ly suggesting that traces o f a functional v iew linger. Ye t , for many teachers, the meaning o f language and content has become more blurred. After a decade or more o f this phrase being used i n the literature, and its obvious connection to a specific context where both language and content have a role in teaching and learning, teachers struggle wi th the term's complexi ty . In the messiness, what surfaces is tensioned 127 ambiguity o f the meta-text of language and content relations. Al ready framed by meta-texts o f evaluation and literacy that attempt to objectify the end, "my purpose is to determine a mark", fo l lowing a linear progression from teaching to learning to evaluating, "If we covered this in class and expected them to learn it, then we should be testing for it", and reading the text as a re-telling of consumed knowledge , " / look for what we've studied in class". The ' f ina l ' agon seems to be caught i n di lemmas o f how 'language and content' criteria are understood and applied. Consumable knowledge and language mastery dominate the discourse o f one teacher i n the comment, Reflecting-in-action: commenting on writing while marking (COMMENTARY) •Why after teaching these structures on the cause and effect of environmental issues and using them over and over in class, they still can't do it on their own appropriately. It's as if they are blocking it, determined to write in their own Japanese way. I can't give them decent marks if they don't follow the model and use the structures . . . this just isn 't acceptable writing A student confers a presumed need to write l ike 'people l ike us ' , Reflecting-on-action: on what it means to write in another language (ACCOUNT) •I try to write like my teacher. I want her correct everything. But my Japanese is big problem. I can't get rid of it. In teaching structures and genres o f specific academic discourse communit ies , is this an attempt to protect students from entering the messiness o f their doubl ing wor(l)ds o f language and culture? Is its purpose to offer a path o f least resistance to success i n future E n g l i s h endeavours? M a n y students request this assistance, presuming that 'people l ike us ' write it right. I wonder, can E L L inscriptions be reduced to specific, distinct, and predeterminant structures o f definable discipline-specific discourse communit ies? Ac t s o f evaluation, framed by meta-texts o f dilemmatic ideologies between and wi th in , seem reduced to judging the effective teaching and learning o f legitimated tricks o f academic wri t ing so that learners can write l ike 'people l ike us ' . A t another marking session, the fo l lowing dialogue between a teacher and the researcher took place as s/he was i n the ' f ina l ' stages o f marking essay exam questions: Reflecting-on-action: on 'final' judgments(ACCOUNT) T.I read through all the essay questions to get a sense of the range of answers. That helps me think about the spread of marks I will give. 128 R:How important are the original criteria at this point? T:They have a role but sometimes as a teacher I misjudge what I expected from the students, or I see they read the question differently than I thought so for me the criteria changes and I alter what I will give marks for. Reflecting-in-action:on the 'final' judgment of the text(COMMENTARY) T:For example, with these essays I'm just about ready to give them marks. I've read them all, seen the range and now have to make some very difficult decisions. Our system here in terms of evaluation is based on numbers. I have to turn in a number for the student for this course to the registrar at the end of term. R: Can't you give a letter grade for the essay ? T'.Ifl could just give a letter grade it would be easier but the students know the final mark for the course is out of 100. So even if I give a grade, the students want to know what that means in numbers. R:Makes it very difficult. T:And risky . . . the essay is worth twenty marks. Language and content are both important so 10 marks go to content and 10 to language. It's probably not fair. It's a senior course and content should be weighted more . R.Do the students know the 10 and 10? T:Ya, I explained it to them and it's written in the instructions. Action: marking the text with indelible ink (ACTS) T:The problem is with essays like these two. One has written some good ideas I think, but the language is funny, awkward, something's not quite right but the message is there. The other has language that is much more controlled and readable but I don't think they really said very much. If I followed the criteria, I think I would have to give them both the same mark. But, it doesn 't feel right. The first one is better, more informed. I'm giving it a higher mark. F o r this teacher, the site was a dilemmatic one. Practices l ived i n persistent instabilities wi th in pre-conditions o f decision-making, instabilities that seemed to have emerged from fragments o f conf l ic t ing models o f language, the possibil i ty o f unshared interpretations o f the prompt, and the imposed drive to quantify written content. M i d s t his/her agon, language and content relations seem uncontested in the paper wherein "language [] is much more controlled and readable" but empty i n meaning, and contested i n the paper wherein "the language is funny, awkward... but the message is there ". In the cruc ia l point o f decis ion-making , the contested space, the space where the teacher is somehow aware that the student is constructing meaning, wi th in what s/he seems to recognise as faulty wri t ing, is valued more. Students struggle wi th s imilar concerns. One student shared, Reflecting-on-action: on writing and marks (ACCOUNT) •My English is good so I can give my teacher what she wants. I know these thing. I can get high mark I know. But topic big and I know more than she wants. This is big problem for me. I want to say in my own words what I 129 know but probably I make too many ah, how do you say, tech, technical errors. I wish she would mark me for how I tell my knowledge, not how many little errors I make. Teachers and students independently struggle to determine what to write and how to read the wri t ing, both l i v ing within fragmented orientations o f language/content relations. Here, the teacher seems to move beyond the limitations o f v iewing language as grammatical correctness (or formal) and a vehicle for meaning, and instead places value on the faultiness o f a learner's lexicogrammatical (or functional) expression of meaning. S imi la r ly , another teacher wrote this commentary on an E L L paper, Reflecting-in-action: on the meaning of a mark(COMMENTARY) •Although you still have problems with grammar, you have clearly expressed ideas. Remember, when we talk about possible outcomes we need to use 'may' instead of 'will'. F o r this teacher, grammatical correctness seems to be the language side o f the language and content equation, whi le "clearly expressed ideas" and uses o f "may" and "will" speak to a functional use o f language and seem to reflect comments on content. A language and content perspective considers how meaning is made i n language. C o u l d it be that this phrase language and content is understood as a co-joining o f confl ic t ing models as i n ' formal (language) and functional (content)'? O r do these two models confl ict and confuse teachers i n trying to understand the phenomena o f language, and content is understood as something else? W h e n each teacher was asked what the term language and content meant to them, a variety o f responses were given, among and within , individual teachers. In one account o f the d i lemma, a teacher offered, Reflecting-on-action: on the meaning of 'language and content' relations (ACCOUNT) •/ think it means that we need to consider both the language demands and content knowledge in the curriculum we teach. If we teach language and content, we should test for language and content. This is clear to me but, when I have to mark a paper, I still get confused with how the language affects the content and how I should deal with it. In another account, a teacher argued, Reflecting-on-action: on the meaning of 'language and content' relations (ACCOUNT) •Language has to do with how we express an idea and content has to do 130 with what we say. At a distance, that seems obvious and very comfortable to me. But when I have to judge their writing, I have trouble trying to figure out whether it is their language that is affecting what they are saying, or whether it is that they really don't know the content. Up close, I don't understand what I'm looking at. I go back and forth between trying to figure out the message from the language and just wanting to mark for grammar mistakes. It's a messy area. Teachers struggle wi th students' expressions o f meaning and it often becomes a "stuck place." D r a w n to a formal v iew for the comfort it seems to offer, teachers presume to separate language and content i n a move away from the messy area rather than work ing their way through. Students independently shared their experiences in wri t ing content, Reflecting-on-action: on the meaning of ELL writing (ACCOUNT) •I write only what I know I can say correctly. •What I write usually only tells you little what I know. •I need more vocabularies. My ideas are simple without more words. •/ chance writing in own words. But, I know teacher's words are more clearly. The students, once again knowingly l imited by their language resources, write less i n their second language o f what they know i n their first and are haunted by the security o f wr i t ing s imply and correctly and constituting more meaning than they k n o w i n their o w n 'faulty ' words. Con t ro l o f meaning, imp ly ing the possibili ty o f an autonomous text, seems to be valued. Students' discursive discomfort became more explici t i n contexts where they realised the possibil i t ies for knowing more than they could say and saying more than they could k n o w are enacted in their wri t ing. In practice, teachers seem to dwel l in the undecidability o f the judgment. The use o f penci l prevails as the values are erased and then rewritten. The tensions surface i n acts o f erasure and re-marking. Reflecting-in-action: on the felt messiness of deciding (COMMENTARY, ACTS) •/ know I teach language and content but look at this, he's really trying to get the message across. I know he knows the topic, its just that his language skills aren 't very good. He's thinking in Japanese so that's problematic. He'll eventually lose it and think in English. Just tells me he's not there yet. I'll give him 7 out often. No, I have to give him credit for what he's trying to say. This is so hard. I'm changing it to an 8. This teacher separated language and content at the point o f decision making and made a 131 value commitment to privilege content but ignored what the learner was doing wi th language "to get the message across." Tensioned uncertainty emerged i n the attempt to separate meaning constructed i n language even when the learner had a 'p roblem' o f "thinking in Japanese." Is the teacher possibly attempting the impossible? C a n the first language be separated from an other? C a n language be separated from content? Another teacher suggests a struggle i n the valuing o f language and of content from a separatist's v i ew i n the commentary, "It has the ideas I was looking for but it is so difficult to read" and another comments, "the ideas aren't clear." These comments suggest a va lu ing o f ideas over language; "if I know what the student is trying to say I'll give them the marks." Language, often placed i n a secondary position as a mere tool or instrument o f communicat ion, is reflected i n the commentary, Reflecting-in-action: on the felt messiness of deciding (COMMENTARY, ACTS) •There's something in this sentence I don't like. Other places it's too wordy. I bet he didn 't think about his topic before he wrote this. His Japanese is coming through here, must have tried to translated some of this but it doesn't quite work. It's interesting though, a different twist. This is good content. This is very good. So it's just a few sentences. Oh, let me look back at the others. It could be a four or five out of five . .. I'll give him five. I like his ideas. This seems appropriate i n the affairs o f the larger society; the discipl ines o f "content" (e.g., the Sciences) are given more import, more power, than disciplines o f language (e.g., the Arts) . Is it evidence o f a larger structure in society imposed on what was thought to be a "straightforward" activity o f integrative evaluation? It seems that the students use o f his/her lexicogrammatical abilities was ignored. Discourses o f teachers offered evidence o f the continued struggle wi th fragmented orientations to language/content relations. Reference to first language and culture is conveyed as faulty, as interference, as a problem wi th translation. In conversing wi th a student as s/he worked on her/his essay, s/he commented, Reflecting-in-action: on the felt messiness of writing (COMMENTARY) •I am translation student but its more than translate, more than right words and correct grammar. I see how language work affect my way to see world and cultures. It make me change. Sometimes when I write I not Japanese, not Canadian .. .1 'm both - it strange and frustrate, difficult to explain . . . I write my ideas in both languages although English is what you see on paper. I am to be mix and I write my ideas that way now. I think gives me new life. 132 This student struggled wi th a new but uncertain emergent meaning o f translation, coming to realise that translation is not just a matter o f attending to word choice and grammar. In translation, words and worlds seem to unfold i n a hybridity o f doubl ing that is incomplete, neither East nor West , neither Japanese nor Canadian. Wha t s/he had come to k n o w as ' s e l f shifted to an ambiguous, tensioned terrain where ' s e l f and 'other' interplayed their subjectivities i n uncertain ways. Fo r her/him, it seemed to be a generative space that gives her "new life." Teachers v iewing the ' m i x ' as a negative space, work dil igently to remove the interference, the faultiness, the presumed translations that d id not work. I wonder, what w o u l d judgments o f language/content relations be i f the ' m i x ' was v iewed as a generative space? Frequently, i n discussions, the teachers were drawn to the message o f the text i n determining the mark given. H o w does this seemingly natural occurrence in form our understanding o f language/content relations? Is language a tool o f communica t ion or is language communicat ion? D o teachers pay attention to how meaning is made or to something separate cal led content that presumes to pre-exist without language? C a n there be one without the other? W h e n a teacher says, "It doesn't matter how they say it. I just want the idea", is this g iv ing priority to content? W h e n content is pr ivi leged over language, is it a posit ion o f language as a messenger, marginal ized because o f its lack o f meaning potential? W h e n language and content are integrated, is it a posit ion o f language and content as inseparable wherein i n the act o f ' languaging' , content is constituted? A r e these positionalit ies binaries, only one or the other? These musings open new areas o f inquiry for educators essential to understanding the simple phrase, language and content. D i l emmat ic discourses continue to contribute to persistent instabilities o f integrative evaluation practices. In a case o f crucial decision-making regarding an essay, the fo l lowing dialogue ensued: Reflecting-in-action: on the 'final' judgment of the text (COMMENTARY) T:This one I'm still trying to decide. You see she didn't answer the question. She wrote an interesting paper though. Some parts made me think. She didn't just repeat what was learned in class. She has some unique, maybe even original, ways with her own words that made me rethink the topic a bit. But it wasn 't the answer I was looking for. R:Was it a wrong answer or a different answer? T : Hmm. Good question. I think mostly different. There are gaps in 133 meaning. I mean I couldn't understand everything she said. Maybe it's a language problem, she couldn 't express what she wanted to say clearly enough. I'm not sure. She seems to know the topic though. Action: marking the text with indelible ink ( A C T S ) T : I have to decide whether to give this an "A " or not. If I use the criteria that I marked the others with, she might not even pass because she didn 't answer the question. One criteria is that they explain the cause of the debt crisis. There's no causal statements in here. Yet, if this was the answer to the question it would be an "A ". Maybe because she's written differently, I like it and I don't. Look here. She shows me she knows about international debt. It's thoughtful. It's actually a more interesting answer than most of the others. She's written it well to get across her ideas. Few grammatical errors too. But, how can I give her an "A " when she didn't answer the question? It's not fair to the others. I need to treat them the same. If I give her something lower though, I know I'll feel guilty . . . I know its worth an "A ". The visual i n Figure 8 maps the decision-making process. W o r k i n g his/her way through, s/he l i ved tensioned and dilemmatic pre-conditions. A strong commitment to the criteria and to demonstrate consumable knowledge, that is presumed to be recoverable wi th in a static text, conflicts wi th an appreciation of meaning that this student has written "with her own words " something that is unrecoverable in its performativity. H o w does one mark the unexpected? In this case, the crucial decision became a guilty judgment. A crucia l decision point, Should I give this paper an "A"? is di lemmatic. I f an " A " is g iven , presumed fairness and consistency l inked to use o f criteria w o u l d be lost. The criteria reflect expectations of predictability and consumable knowledge. A l though the teacher personally valued the student's "thoughtfulness" and "unique . . . original ways with her own words", even wi th "gaps in meaning", discourse evidence suggests that being perceived as fair and consistent to a l l the students weighs heavily wi th this teacher. S/he holds strongly to the linear notion that what has been taught must be evaluated and what criteria is established must be applied consistently. In order to apply the criteria, the teacher presumed a predictable response. The learner responded differently - an aporia i n the making. W i t h i n a meta-text of literacy, at least two models conflicted: the criteria imp l i ed a static text o f expected causal statements yet the teacher's response to the wri t ing made h i m "re-think the topic" and struggle wi th the "gaps of meaning." A s a site o f mult iple positionalities, criteria as acts o f fairness and consistency dominated the decision 134 and the resulting guil ty judgment. Figure 8 V i s u a l M a p p i n g o f a Specific Case o f M a r k i n g wi th Indelible Ink Shou ld I give this paper an " A " ? alternatives: • other - lower grade possible outcomes & evaluation:'* criteria not applied (-) not fair and consistent (-) • criteria applied (+) fair & consistent (+) guil ty judgment (-) decision: 'other' because criteria was applied for alleged fairness and consistency (valued) (In)commensurate Discourses o f Language/Content Relations Evidence o f persistent instabilities seems to surface when teachers attempt to make sense o f language/content relations in E L L texts prior to ' f ina l ' judgments in non-authentic mark ing sessions. A n opportunity to explore this specifically was provided when teachers were asked to mark at least five o f fourteen samples o f E L L written content provided for them. Fourteen teachers agreed to spend from thirty to sixty minutes marking the texts, texts written by E L L s unknown to them. I have arbitrarily selected two samples o f E L L written content and consider the teachers comments as they specifically relate to language and content relations i n the texts. I then offer an analysis o f knowledge structures identified wi th in those texts, as one way o f considering language and content integration. 8" w Al though multiple factors influence the mark, teachers were asked to specifically focus their judgment and comments on the language and content o f the text. Tables 6 and 7 8 0 Knowledge structure analysis (KSA) is an approach to the discourse analysis of Mohan's (1986) knowledge structures (KS) currently a work in progress by Mohan and his former and current graduate students. For each knowledge structure of classification, description, sequence, principles, choice and evaluation, this work graphically maps the macro-structure of the KS and provides a discourse layout of the most common ways the knowledge structure is constituted. Although still in its infancy, it holds promise for understanding 'language and content' relations. 135 summarise teachers' independent marks and comments on two sample E L L texts. M y intent is not to inquire into w h i c h judgments were 'correct ' but to attend to the differences i n teachers marks as effects o f positionalities wi th in and between meta-texts o f language/content relations. F o l l o w i n g the teachers' comments, Tables 8 and 9 analyse the macro-structure o f each text using knowledge structure analysis to locate integrated macro-structures o f language and content relations elicited by the prompt. In this case, learners responded to a prompt designed to elicit causal explanations o f cl imatic change (see A p p e n d i x 5 for the specific prompt and further examples o f knowledge structure analysis o f E L L texts i n response to the prompt). Evidence i n the discourse consistently suggests teachers struggle to make sense o f language/content relations. In offering a discourse layout o f language structures i n wh ich cause and effect is constituted, I offer another possibi l i ty o f language/ content relations for consideration. H o w marks or comments may alter using this approach is unknown. However , it seems to have the potential to reduce, but not eliminate, what appears to be some o f the tensions within a meta-text o f language/content relations. The written E L L responses i n Tables 6 and 7 are based on the fo l lowing prompt: You have been learning about climatic change. Write a well organised paragraph about one of the following topics. A. Explain one cause of climatic change. OR B. Explain how climatic change causes one human or planetary health problem. K n o w l e d g e structure analysis o f each students' text indicates that most students have some linguistic resources to express linear cause and effect between events i n at least two ways: wi th in the clause and between clauses. They have used conjunction and lexis to realise causal relations. Some students seemed to have difficulty expressing their ideas as i n sentence (8) in sample text 1. However , only one teacher commented on the students' expression o f causes or results o f the topic, "This is a good expression of the effect of climatic change, they've used a conjunction." Instead, comments such as, "This is confusing. What is the writer trying to say?" or "the meaning is clear" frequently preoccupied the teacher i n trying to make sense o f written meaning. 136 Table 7 Teachers' Independent Marks and Comments on Sample ELL Text #1 (l)Global warming is one cause of climatic change. (2) Global warming is to rise temperature unnaturally in an earth.. (3) Carbon dioxide in the air increase, temperature is goes up. (4) Therefore, we should think the way of using Carbon dioxide. (5) Carbon dioxide mainly comes from cars and industries. (6) If we leave producing carbon dioxide and rising temperature, we could get ocean water evaporation, and the permanent ice melting. (7) These cause big climatic change. (8) Example are the dead of. fish which can't be used to speedy of temperature rising in water and huge storm in which many species could get damages. (9) Many circle of natural chain reaction could be destroyed. (10)Now, we come to the period when we have to confront global warming. (11) It can't stop without our doing anything to it. (12) I am sure we all hope peaceful life and future, so let's think about it and do anything to get safety future. (Yumiko's response to an exam question, 11/23/95) Evaluation (10) Commentary on the 'language and content' of sample ELL text #1 9 'the meaning is clear • this is a good expression of the effect of climatic change, they've used a conduction • lots of information but not always clearly expressed 'look how they've structured their ideas and sequencing of ideas and rhetorical structures. I can see cause and effect. 7 •it's important that I fill in the gaps so 1 can understand what the student is saying •I'll give five marks for content and five for langauge...this one gets more for content, the language is a mess 5 •no I think he meant the temperature will rise 4 •This is confusing. What is she trying to say? • I gave it four out of ten. The writer is off topic. She or he did not answer the question. The language is very readable though with minimal errors, •they are not making clear statements about cause and effect. I would have to guess alot and that's not something I want to do 1 Table 8 Teachers' Independent Marks and comments Sample ESL Text #2 (l)Today, there are climatic changes. (2) These are caused by a greenhouse gases, grobal warming and El Nino etc. I explaine it. (3)Greenhouse gases, global warming and El Nino may trigger weird weather so people's life style change. (4) For example, grobal warming make rainfall change.(5) A temparture up so plants and trees can't grow. (6) Also, people have some health problems, too. (7) Climatic change is bad for our life because these may be destroy many thing and change to our life styles. _ (Shino 's response to an exam question, 11/23/95) Evaluation (10) Commentary on the 'language and content' of sample ELL text #2 8 •/ wouldn't mark down too much for all the errors in verb use but the writer has to include the main points and I think they do. •an 'e' on the end of explain and grobal instead of global doesn't oncern me 6 •this is a good beginning, they give lots of information about the results of global warming...look grobal not global •misuse of articles 1 don't pay attention to, I can understand the sentence - it's fairly straight forward 5 •If it's a spelling or grammar error like 'grobal' I circle it only once even though the mistake is repeated •the person has missed the question •they've tried to put it into an essay style not a paragraph 4 •grammatical errors like these bother me if I can't understand what the writer is saying, if it's interrupting the meaning in any way •the language seems to be readable and there's not glaring grammar mistakes •I don't know which question they are trying to answer •its not clearand even if it was there is not enough information here Teachers constantly interpreted what they thought the students meant, crossing out words and adding their own to the student's text. "It's important that I fill in the gaps so I can understand what the student is saying" or 'Wo, / think he meant the temperature will rise." Teachers do not seem to attend to how students use their l ex ico-grammatical knowledge of Engl i sh to make meaning. Instead, attention is sometimes given to structural organisation, sometimes to clarity o f meaning (at a general level), and frequently to grammatical errors at the sentence level. There appears to be a recognised lexicon of content and a recognised lex icon o f language i n the criteria used by the teachers. Content lexis from the discourse(s) data comprise o f such words as ideas, points, details, meaning, topic, understand, and message. Language lexis include terms such as sentence structure, sentence fragment, paragraph, essay, grammar, and spelling. L e x i c o n used to describe the criteria o f language and content evaluation often seem to be distinct and dealt wi th language and content separately rather than i n an integrated way, as i n the fol lowing comment, "I'll give five marks for content and five marks for language." It may be that teachers assume the criteria o f mark ing schemes as a duality o f opposed, or unrelated, elements, "I wouldn't mark down too much for all the errors in verb use but the writer has to include the main points" rather than marking schemes that are relational and a im at the realisation o f meanings. Clear ly , there is little evidence o f a sustained functional perspective in the ' talk ' about evaluation criteria. M o s t express a perspective o f difference, an unrelatedness o f language and content that permeates their use o f criteria. Sometimes evaluators use formal criteria "If it's a spelling or grammar error I circle it only once even though the mistake is repeated", and sometimes they reject them " an 'e' on the end of explain and grobal instead of global doesn 't concern me." Sometimes they ponder the functional relation between content and expression: "misuse of articles I don't pay attention to, I can understand the sentence ", "grammatical errors bother me if I can't understand what the writer is saying, if it's interrupting the meaning in any way." W h i l e some teachers show insight into functional problems, they use different sets o f criteria to evaluate the same text. Some look for meaning, whi le others' judgment is influenced by the viola t ion o f language rules. 138 Table 8 Sample E L L Text 1 and a Discourse Layout o f its Cause and Effect Structures. ( l ) G l o b a l warming is one cause o f cl imatic change. (2) G l o b a l warming is to rise temperature unnaturally in an earth. (3) Carbon dioxide in the air increase, temperature is goes up. (4) Therefore, we should think the way o f using Carbon dioxide . (5) Ca rbon dioxide main ly comes from cars and industries. (6) If we leave producing carbon dioxide and rising temperature, we could get ocean water evaporation, and the permanent ice melting. (7) These cause b ig cl imatic change. (8) Example are the dead o f fish wh ich can't be used to speedy o f temperature r is ing i n water and huge storm i n w h i c h many species could get damages. (9) M a n y circle o f natural chain reaction cou ld be destroyed. (10)Now, we come to the period when we have to confront global warming. (11) It can' t stop without our doing anything to it. (12) I am sure we al l hope peaceful life and future, so let 's think about it and do anything to get safety future. (Mako's response to an exam question, 11/23/95) 1. Between clauses causes/conditions/reasons l ink effects/results we leave producing carbon dioxide and r is ing temperature Carbon dioxide i n the air increase 2. W i t h i n clauses i) agent huge storm to speedy o f temperature r is ing i n water i i) cause/condition/reason these global warming [from global warming] i f could get [when] process: general could get damages can' t be used l ink cause is one cause could be destroyed ocean water evaporation and the permanent ice melting temperature is goes up medium many species dead fish effect/result climatic change climatic change M a n y circle o f natural chain reaction 139 Table 9 Sample E L L Text 2 and a Discourse Layout o f its Cause and Effect Structures. ( l )Today , there are cl imatic changes. (2) These are caused by a greenhouse gases, global warming and E l N i n o etc. (3)Greenhouse gases, global warming and E l N i n o may trigger we i rd weather so people's life style change. (4) Fo r example, global warming make a rainfall change.(5) A temperature up so plants and trees can't grow. (6) A l s o , people have some health problems, too. (7) C l ima t i c change is bad for our life because these may be destroy many thing and change to our life styles. (Shino's response to an exam question, 11/23/95) 1. Between clauses causes/conditions/reasons greenhouse gases, global warming and E l N i n o may trigger weird weather l ink effects/results . so people's l ife style change a temperature up C l ima t i c change is bad for our life so plants and trees can' t grow also people have some health problems too because these may be destroy many thing and change to our life styles 2. W i t h i n clauses i) agent process medium ii) cause/condition/reason these (climatic change) a greenhouse gases, global warming and E l N i n o greenhouse gases, global warming and E l N i n o global warming l ink may be are caused by effect/result destroy many thing change to our life styles these (climatic change) may trigger weird weather make a rainfall change 140 D o they think o f integration of language and content at the local level or global level (of words/ideas or at the level o f genre, for example)? D o their views agree or conflict? Axe expert members o f this community converging in their judgments? In Swales ' (1990) course on thesis wri t ing for E L L s , learners were "mov ing towards membership o f a chosen discourse communi ty v i a effective use o f established genres wi th in that communi ty" (pp. 77-82). W h i l e they discuss the strengths and weaknesses o f the wri t ing at text-level, as w e l l as at sentence and word level , there is no evidence that teachers use the concept o f genre as an integrating element i n their evaluation. "This is a good beginning, they give lots of information about the results of global warming, . . . look grobal not global." Impl ic i t i n one teachers's discourse was a functional perspective when s/he commented, " / look for how they've structured their ideas and then the sequencing of their ideas and the rhetorical structures:" However , this same teacher talked i n a discourse o f fragmentation and unrelatedness when s/he explained her/his reasons for g iv ing a certain mark, "I gave it four out of ten. The writer is off topic. S/he did not answer the question. The language is very readable though with minimal errors." The text is considered f rom two evaluative perspectives, not a single integrated one. There is no relation made between "causes / results" and language use, between communicat ive purpose and textual form, between content and expression. A (Re)Turn to the Questions M y purpose i n this chapter was two-fold: to explore the difficulties i n practices as they emerge in particular instances o f integrative evaluation for a group o f teachers, and, through those difficulties, consider relations between meta-texts o f evaluation, literacy, and language and content integration within l iv ing practices o f integrative evaluation. It is an attempt to undo teacher's discourses, in particular those that di lemmatical ly dwe l l i n 'stuck places ' , agons, and felt messy texts o f integrative evaluation practices. Three sets o f data were considered: teachers' discourses from authentic marking sessions, learners' discourses f rom authentic wri t ing sessions, and teachers' discourses f rom non-authentic mark ing sessions. Probing o f troubled positionalities pr imar i ly focused on teachers' discourses i n authentic mark ing sessions. In particular, three sites o f "stuck places" 141 became evident in the discourse data: creating prompts, (re)turning to criteria, and felt messiness o f inscr ibing marks of indelible ink. Students' voices, often presumed to be present only in the words on the page, were heard in pol ivocal ic relations wi th teachers' discourses wi th in the felt messiness o f integrative evaluation. Throughout activities o f integrative evaluation, disruptions and "stuck places" were consistently evident. Interrelations o f discursive acts, commentaries, and accounts o f the process became sites o f contestation. Teachers' accounts o f integrative evaluation frequently conflicted with their commentaries and acts; establishing criteria seemed "straightforward" unti l a felt messiness emerged when applying it to written content. Traces o f positionalities suggested a privi leging of a technical imaginary wi th in w h i c h knowledge was made supposedly objective, criteria were structured as countable and static, and judgment was a search for univocality and commodif icat ion. Reading texts prior to judgment was often acts o f reading presumed autonomous texts, seeking one meaning - the intended response - one known-in-advance by the teacher. It was w e l l intentioned 'support ' to the E n g l i s h language learners taught. Teachers often appropriated students' words to al ign wi th the structures and knowledge studied in class. Teachers seem to seek specific anticipated responses and other responses became faulty. Students tried to determine what that response might be. Some learners, i n t rying to write l ike "people l ike us", questioned the interference o f their first language and how it led to faulty writ ing. Others, in a similar struggle, recognised that l i v i n g i n the two-folds o f languages and cultures could be a generative space, that one cannot and should not erase the texts (their Japanese language and culture) that have come before. Relations between multiple texts o f teacher, learner, writer, reader, were i n constant movement; reciprocal intentions often col l ided in tensioned, fissured positionalities. The importance o f content prevailed i n discussions o f prompt, criteria, and expected responses. Yet , once into the felt messiness or undecidability of integrative evaluation, discussions o f language and content relations dominated; disruptions o f positionalities i nvo lv ing fragments o f formal and functional theories o f language mediated and complicated l i v i n g practices. "Stuck places" emerged f rom teachers' reflections-in-action wherein evidence o f commentary, accounts, and acts and their relations became a space o f 142 undecidabil i ty enacted through guilty judgments and r isky practices. In most "stuck places", teachers seem to choose dominant wel l -worn paths wi th in a technical imaginary. A t the point o f crucial decision-making, it seemed that teachers heeded the force o f the technical rationality i n their reliance on and re-turn to discursive traces o f mastery i n accuracy, o f curricular linearity, o f the possibility o f knowledge as commodi ty , o f a disintegration o f language and content, o f markable learning and hope for a un ivoca l text. F e w took the risk o f laying down a new passage whi le work ing through uncertain and ambiguous spaces o f l i v ing practices of integrative evaluation. Some o f the difficulties o f integrative evaluation emerged from the mis/alignment of a form of evaluation and the phenomena (a social and integrated text) whose interests it purported to serve. Teachers intentions to "get it straightened out first" presumes linear, objective evaluative practices and seems to conceal other positionalities the two-folds o f integrative evaluation. Fo r most, "the extent to w h i c h it is possible to separate language and content, ideas and expression, remains . . . an unresolved prob lem" (Norton & Starfield, (1997, p. 291). Current critiques o f technical rationality offer useful interpretations o f instances o f integrative evaluation practices. In the insistence of a technical form o f evaluation for texts that are explored as interpretive and social literacy, teachers experience guil ty judgments and r isky practices. What is needed is a form o f evaluation that aligns wi th ambiguous and difficult texts o f language learners. Chapter six is interpretive work that attempts to de-center the technical and begins an exploration o f integrative evaluation practices i n a dialectic o f radical hermeneutic and functional linguistic stances. 143 Chapter Six Guilty judgments and risky practices: a discussion Apprec ia t ing that tenuous and delicate situation o f judgment w h i c h is addressed by the name "undec idab i l i ty" . . . does not detract from the urgency o f decision; it s imply underlines the difficulty. John Caputo, Against ethics In recent years, increasingly curr iculum scholars have opened themselves to the realm of language, linguistics, discourse and narratives to understand their o w n field. W i t h i n this curricular turn, language is understood not so m u c h as a d i sembodied tool o f communica t ion caught up i n an instrumental v i e w o f language, but more so language understood i n an embodied way — a way that al lows us to say, "we are the language we speak ." Ted A o k i , The call of teaching E a c h aspect - task, writer, scoring procedure, reader - interacts wi th the others, creating a complex network o f effects w h i c h to date has e luded our efforts to cont ro l . W e w i l l not, I bel ieve, solve the problems o f wr i t ing assessment unt i l we look at this total picture rather than each facet i n isolat ion . . . . W e must use a context-embedded approach, capi ta l is ing on what has so far been seen as the central p rob lem o f wr i t i ng assessment - the fact that i n a l l its stages it is a who l ly human endeavour. L i z H a m p - L y o n s , Second language writing assessment The major influences in E S L - its scientific orientation i n research, its pragmatism, and the conservatizing effect o f E F L - work against a movement toward either left-wing ideology or radical pedagogy. Terry Santos, Ideology in composition: Ll and ESL 144 " U n d e r l i n i n g the D i f f i c u l t y " o f a " W h o l l y H u m a n Endeavour" Through the beginnings o f a published pilot study ( M o h a n & L o w , 1995), this project has emerged as an inquiry similar to what Dav i s (1996) describes as an interpretative framework o f enactivism that seeks out middle ways amid disparate perspectives. B u t such m i d d l e ways shou ld not be thought o f as compromise s . Ra ther , they represent attempts to sidestep s e e m i n g l y i r resolvable tensions by d rawing attention to and offer ing alternatives for the assumptions that underlie varied opinions, (p. xxv) It is an inquiry that attempts to make sense of, f ind 'middle ways ' through, the diff iculty o f disparate discourses that are evident in the instances o f discursive pre-conditions o f teachers judging E L L inscriptions in academic environs. I structured the struggle as questions o f inquiry which , over the time of the study, have been re-written through an exposure to an obvious process o f continued reading - recursively and anew - as I attempted to make sense o f instances o f integrative evaluation practices. I found myse l f engaged i n a process o f re-structuring as a way to address my increasing awareness of the importance o f complexi ty and difficulty that I was coming-to-know. The questions are restated here: 1) H o w do teachers, i n their practices, construct the activity o f integrative evaluation? In particular, do current critiques of a technical rationality contribute to useful interpretations o f integrative evaluation practices? 2) H o w do teachers relate practices o f evaluation, literacy, and language and content integration? Before highlighting the interpretive work guided by these questions, I turn to introductory comments made i n Chapter One to briefly review current critiques o f technical rationality. The remaining pages focus on the effects o f teacher positions through concerns o f val idi ty and ethics in integrative evaluation practices as a way to "underline the d i f f i cu l ty" o f a " w h o l l y human endeavour." Concerns wi th technical rationality imbricate many levels o f life. In the broader society, L y o t a r d (1997) invites his readers to resist the 'thingness' and linearity o f 145 M o d e r n i t y ' s metaphysical assumptions o f presence, Tay lo r (1991) questions Western beliefs that one can be i n control o f one's l ife, and Caputo (1987) "attempt[s] to stick wi th the or iginal difficulty o f life, and not to betray it wi th metaphysics" (p. 1). In education, Altr ichter , Posch, and Somekh (1993) question technical rationality and ca l l for a reflective rationality. Jardine (1992, 1998) and Smith (1999), both wi th hermeneutic interests, argue for an attunement to, rather than a concealment of, difficulties o f life i n the classroom and M i l l e r (1997) admits to a general tendency to "technologize the purposes and forms o f research" and therefore attends to the experiences that disrupt those tendencies, arguing that the disruptions "da i ly reconfigure not only the curr iculum theories that frame [her] work but also the ways i n w h i c h [she] conduces] research wi th teachers" (p. 199). In second language wr i t ing pedagogy, Z a m e l (1997) and Spack (1997) both c l a i m traditions o f E L L wri t ing pedagogy are reductionistic and deterministic, H a m p - L y o n s & K r o l l (1996) ca l l for a context-embedded approach to wri t ing assessment in resistance to more traditional, context-reduced approaches, while Santos (1992) is cri t ical o f what she calls a major scientific orientation to the teaching o f Eng l i sh as a second/foreign language. Telescoping and inter-related imbrications suggest a strong influence o f broader societal concerns that have filtered down to influence classroom practices, especially i n second language instruction. E a c h instance offered in Chapter five belongs to integrative evaluation practices and "is fecund: [they] keep the story [of integrative evaluation] going... those shared and contested understandings in which we l ive are called to account by this instance, made to 'speak,' change, accommodate, and so to speak, ' learn ' through this encounter" (Jardine, 1998, p. 42, original emphasis). Particular instances draw our attention to integrative evaluation practices so that we do not render it something "that no longer concerns us", but instead remain attuned to the contested sites that give it l i fe . It is an obligation I believe we have to students, especially in instances o f evaluation. Table 10 reflects how each instance has "add[ed] itself to what [integrative evaluation] can now be understood to be" ( ibid. , p. 40). 146 How I Now Understand Integrative Evaluation To Be Table 10 Highlights Interpreted Integrative Evaluat ion Practices 1) H o w do teachers, in their practices, construct the activity o f integrative evaluation? h particular, do current critiques of a technical rationality contribute to useful interpretations o f instances of integrative evaluation practices? • Teachers engaged in difficult practices of mult iple "stuck places" o f uncertainty, conflict and ambivalence when creating the prompt, establishing and applying criteria, and in the pre-conditions o f decision-making. • Teachers experienced hermeneutic acts o f literacy and functional relations o f language and content integration in the pre-conditions of decis ion-making. • Teachers used a technical form of evaluation in order to satisfy technical requirements (submission of a mark) established by the institution. • Teachers f i l led in the meaning gaps of ambiguous texts to ensure univocal i ty (one reading, 'the' reading) of the text when deciding a mark. • A s the researcher, current critiques o f technical rationality helped to question the 'p rob lem' o f difficulty and to become attuned to its imposit ions; i n particular, radical hermeneutics invited a re-positioning o f difficulty to an ambiguous and generative space. 2) H o w do teachers relate practices of evaluation, literacy, and language and content integration? • Pr ior to agreeing to a prompt, teachers explored literacy as ambiguous and social interpretations but, in their uncertainty o f how to mark such a text, they returned to a question for w h i c h there was a 'correct ' and 'control led ' response. • W h e n applying evaluative criteria, teachers were confused by discordant relations o f evaluation, literacy, and language and content integration. They sometimes acknowledged functional views in integrating language and content, yet when they did , they were vague and uncertain about how to mark in an integrated way. • Teachers struggled wi th meaning o f E L L texts at the level of discourse but then focused on language accuracy at the sentence level when deciding a mark. • Teachers said that E L L texts were difficult to interpret but they then imposed their o w n 'straightforward' readings on the texts to reduce and s impl i fy the difficulties. • Teachers acted on dominant technical positions i n acts o f decis ion-making and attempted to al ign a technical form of evaluation with E L L texts by imposing instrumental acts o f simplification and distortion on literacy, and language and content integration. 147 Teachers engaged in difficult practices of mult iple "stuck places" o f uncertainty, conflict and ambivalence framed within a linear, staged context o f creating the prompt, establishing and applying the evaluative criteria, and reading the text prior to decision-making . In this process teachers experienced hermeneutic acts o f literacy in their troubled interpretations o f student texts and explored functional relations in their commentaries on how language constructs meaning. A presumed linear order was disrupted by an increasing number o f undecidabilities as teachers moved closer to determining a mark. Re-turns to the criteria interspersed wi th re-turns to the prompt and re-turns to the text became unpredictable actions in the name of fair and consistent judgments. The (dis)order was often a signpost o f re-marking - a technical form o f evaluation used i n order to satisfy technical requirements (that is, a submission of a mark for each student) established by the institution. Th is became a point o f difficulty i n which teachers found themselves wanting-to-be-sure on the form, f i l l ing in the meaning gaps o f ambiguous texts to ensure one reading, 'the' reading, midst the uncertainty o f their judgments. Current critiques o f technical rationality helped to question the felt need for certainty, for wanting-to-be-sure, for v i ewing difficulty as a problem needing to be fixed and to become attuned to its imposit ions on literacy, and language and content integration. In particular, difficulties were interpreted as generative when re-positioned by radical hermeneutics. In several discursive instances, at least two conflicting models o f language led to an uncertainty o f how to mark criterion-specific language features o f the text. A s w e l l , there was strong evidence o f a content focus in discussions on prompt design, evaluative criteria and overal l purpose of judgment, e.g., "to get the ideas across", and o f a language focus on accuracy at the sentence level. Yet , even with these foci o f presumed shared meaning, mult iple interpretations o f previously agreed upon language and content prompts and/or criteria became uncertain, ambiguous intentions. M i d s t these struggles, teachers explored differing epistemological stances, w i l l i n g to turn a new and somewhat ambiguous theoretical posit ion into practice. Bu t once into the ambiguity and indecision o f uncharted terrain, they chose to return to the familiar and, in so doing, became complic i t in perpetuating a system they knew "needs to be changed, but if we can't agree on a way to do it let's go back to the original question." 148 Pr ior to agreeing to a prompt, teachers explored literacy as ambiguous and social interpretations but, i n their uncertainty o f how to mark such a text, they returned to a question for w h i c h there was a 'correct ' and control led ' response they cou ld justify. W h e n applying evaluative criteria, teachers were confused by discordant relations o f evaluation, literacy, and language and content integration. Teachers applied both formal and functional views o f language in their deliberations. They sometimes acknowledged functional views in integrating language and content, yet when they did, they were vague and uncertain about how to mark i n an integrated way. O f the two theoretical stances, formal l inguistics seemed to have more o f a stabilising presence, whereas a functional framework seemed to be incomplete and unstable, being somewhat unknown to many. A s a way out o f the disparate conditions, a familiar path was taken wherein a formal position, one o f a seemingly less ambiguous system for teachers, dominated decisions o f integrative evaluation. Teachers struggled wi th meaning of E L L texts at the level o f discourse but then focused on language accuracy at the sentence level when deciding a mark. Teachers said that E L L texts were difficult to interpret and then imposed their o w n 'straightforward' readings on the texts to reduce and simplify the difficulties. Assumptions o f reading a text that was fixed and stable and represented what the student knew of language and o f content predominated, whi le an interpretive position and "having to guess what s/he means" was enacted but not valued. Teachers acted on dominant technical positions in acts o f decis ion-making. They seem to conceal their experiences of hermeneutic acts, and functional relations, and attempted to align a technical form of evaluation wi th E L L texts by imposing acts o f simplif icat ion and distortion on literacy, and language and content integration. Technica l rationality operated as a powerful apparatus that seemed to contain and control ways i n wh ich relations o f literacy, and language and content integration got done. The complexifications of integrative evaluation became a difficult act of trying to clean up ambiguous and contrary themes both wi th in and between at least the three meta-texts discussed in this study. The explicitness o f the prompt and the proliferation o f evaluative criteria were attempts to "get it right" by turning a l i v ing practice into technical details that worked to conceal the difficulty. The way out for many then was to f ind "a 149 technical f i x " to the difficulty of a social practice called integrative evaluation. These instances raise serious concerns i n practices o f evaluation, literacy and language/content integration, especially when technical forms o f evaluation are paradoxical ly aligned wi th social and integrated texts. In light o f current critiques o f technical rationality and the interpretations i n Table 10 o f what integrative evaluation can now be understood to be, I argue for a loosening o f the powerful ho ld o f the technical as a 'regime o f truth' on classroom evaluative practices. In so doing, I discuss a decentering o f the technical and possible implications for integrative evaluation re-positioned i n radical hermeneutics and functional linguistics. Then I re-visit the questions o f inquiry in l ight o f teachers' broader concerns o f fairness and consistency through issues o f rel iabi l i ty , val idi ty, and ethics, and i n a discussion of the effects of positionalities on evaluative practices. Decentering the Technical A pre-condition o f decision-making was a space o f indecision; questions-in-flux seemed to enact constant instability and resist the temptation o f 'being sure', o f a mastery that assumes it cou ld bring clarity to decisions. Fo r many teachers pre-conditions emerged as a disabl ing space of impossible difficulty, a "stuck place" for w h i c h the only way out was to travel w e l l worn paths - those instrumental actions o f concealment and containment -to cover up and control the cracks, fragments, breaks and ruptures i n the routine cal led mark ing 'language and content' texts. Concealment, as teachers found out, brought only momentary relief. W h e n faced with another text to mark, the cracks re-turned only to, once again, be concealed, contained, controlled - a series of discomforting and l imi t ing repeat performances that had come to be routine in evaluative practices. I wondered, cou ld "undecidabi l i ty" become an enabling space o f difficulty, as the promise o f enact ivism suggests, wherein work ing the cracks, fragments, breaks, and ruptures enacts a middle way through acts o f denial to conceal, contain, and control? Its promise seemed to l ie in decentering - a breaking with but not casting away - the technical. In what follows, I struggle with undecidabilities o f integrative evaluation and consider issues o f reliabili ty, validity, and ethics that seem to be raised pr imari ly i n language testing research and i n "stuck places" evidenced in this inquiry. I conclude wi th a c a l l for a 150 valuing o f the difficulty o f integrative evaluation and enter a conversation on how, as educators, we might learn to " l ive wi th it w e l l . " M a n y have taken the bo ld step o f arguing against the technical to remind us o f its powerful imposi t ion o f a way o f thinking that has forced many o f us in education to act i n particular ways. Postmodern writers Usher and Edwards (1994) argue that modernist (technical) discourses o f education posit such claims as: 1) knowledge is a master signifier wi th an a im of cumulative and progressive mastery; 2) the teacher, as the-one-who-knows, transmits knowledge unilaterally to the learner; 3) individuals intentionally impart meaning to empty words - language is transparent and can represent fully and completely that w h i c h it signifies; and 4) knowledge is a closed system that is dominating because it forces learners into the system and totalising because it is pre-given and strives to envelop al l there is to k n o w . In taking such a posit ion, I ponder Althusser 's musing, . H o w many [teachers] (the majority) do not even beg in to suspect the " w o r k " the system (wh ich is b igger than they are and crushes them) forces them to do, or worse , put a l l their heart and ingenuity into performing it w i t h the most advanced a w a r e n e s s . . . . So little do they suspect it that their o w n d e v o t i o n con t r ibu tes to the m a i n t e n a n c e and nourishment o f this ideological representation o f the Schoo l . (Althusser cited in C h o w , 1995, p. 108) Dominan t positions (e.g., "the system"), competing wi th other moti le and mult iple traces that emerged i n this study, seem to be powerful determinants o f how teachers accompl ish their work. In Chapter five, the need to measure, to quantify, to objectify as a number wi th in evaluative practices imposed a structure on teachers that inevitably led to a separation o f language and content relations midst their discomfort o f such an act. One teacher's comments seemed to communicate a recurring theme found i n many o f the instances o f discourse, Reflecting-on-action: teacher comments on integrative evaluation practices (ACCOUNTS) •/ know I should somehow consider the language and meaning connection in a more holistic way hut its so difficult and, although it is somewhat arbitrary, I find separating language and content for specific marking purposes the best way to come to a mark that for me is fair and consistent. 151 But, even as I comment on the language, I'm thinking about the meaning and when I comment on the meaning, I'm thinking about how the language was used to make meaning - meaning that I might not even be understanding in the way it was intended. So, at some level even when I separate language and content, I'm not separating them. Paradoxes perform as incomplete and often ambiguous systems as i n the separation/integration of language and content relations. Another performative paradox occurs i n the teacher's comment on interpreting meaning and her/his struggle to k n o w where meaning resides. Th is teacher's use of ' intended meaning ' suggests an assumption of an autonomous text while juxtaposed wi thin knowing her/his reading o f the text may not be as it was ' intended' . A s Caputo (1993, p. 3) c la ims, it is a difficult "tenuous and delicate" space o f "undecidabi l i ty ." What is certain is the imposed necessity o f a numerical value that w i l l mark the text. The structure o f the technical imaginary seems unquestioned and, i n so doing, many teachers continue to assume that their role as evaluators can be objective and distant, that the text can be considered outside o f its context, that language can stand on its own, and that content, in its wanting-to-be-sure, seeks language structures that w i l l carry its certainty. W h i l e the force o f the technical works to enact these assumptions, pockets o f resistance surface i n knowing that unacknowledged relational processes are involved in judging a written text. Evaluat ion, as a marking-for-life, is for most a well-intentioned site where social entanglements o f power, interpretation, and positionality invoke difficulty and "undecidabi l i ty" , and disrupt the felt need for linear practices and presumptions o f autonomous texts. O f interest in this study is integrative evaluation in the context o f l i v i n g , cultural practices. Such an approach acknowledges social relations and their inherent ambiguities and difficulties; it is a re-positioning that questions the assumptions under lying a technical imaginary and the possibility o f taking an ethical stance. I wonder, how can one dwel l i n a technical imaginary i n the name-of-the-good and enact fair and consistent evaluative practices, while reducing voice and situatedness to presumed unitary, definitive, and closed texts, and c la iming the innocence of a language that is bound i n presumed mechanistic work and 'does as it is to ld '? M y intent is not to rescue the readers o f this text f rom the demise o f the technical imaginary, nor is it to offer an 'other', new, and improved posit ion and take up its cause. Instead, I am attempting to show how and the extent to 152 which what currently matters i n integrative evaluative practices is revived - brought back to life - and strengthened by a re-positioning o f 'd i f f icul ty ' . Evaluat ion, as well-intentioned acts, remains at the centre of schooling. W h i l e approaches to evaluative practices have changed over time in an attempt to maintain its good name, the struggle to 'get it right ' continues. Presumed securings to foundations o f presence, objectivity, mastery, and certainty attempt to hold ' technical ' evaluation i n place. Y e t , entangled i n the foundations are "undecidabili t ies" that loosen the ho ld and invoke the tell ing o f a difficult story - a story of integrative evaluation that admits to subjugated knowledges, situatedness, and social relations. It is a painful jouissance o f evaluative practices that claims an impossibil i ty in the name-of-the-good. Evaluat ion, it seems, can no longer maintain accurate reflections o f the worthiness o f the ' real ' ( in this case, the ' rea l ' student). Diff icul t ies wi th its imbued foundational slippages, shifting ontologies and epistemologies, ambiguous cultural practices, and (un)certain constituting effects o f language contribute to an "undecidabil i ty" that has weakened, perhaps even ruptured, a technical hold. V i e w e d in this way, evaluation could be thought o f as an aporetic evaluation, no longer technically sure of itself and therefore becomes an attunement to 'middle ways ' out. The historical condit ion o f scientific reason contributed to a longing for order i n what was thought to be a dis-ordered wor ld and technical, instrumental th inking has no doubt contributed to that ordering. Carr ied over to pedagogical sites, numerical imposi t ioning was an obvious move to an assumed need for control o f evaluative practices. Yet , as evidenced in the discourses of teachers engaged in integrative evaluation, it is not a site o f clarity but one o f contestation. What i f a re-positioning decentered the technical to make possible thinking o f integrative evaluation as aporetic? Such a re-positioning acknowledges both the l imits and promise of a space wherein entanglements o f cultural practices, social relations, and constituting effects of language play out i n performative paradoxes o f "undecidabili t ies". C o u l d it be a re-positioning wh ich imagines integrative evaluation as a movement of linearity and complexity o f evaluation, o f fixedness and openness o f literacy, and o f separateness and integrativeness o f language and content relations? 153 affirms: Research is neither w i n d o w nor mirror . Str ipped o f s imple reference, embedded i n a prol i fera t ion o f differences and failures o f Enlightenment categories, caught between the no longer and the not yet, we cannot not k n o w the n o n -innocence o f language and the weight o f culture i n our portrayals of the wor ld , (pp. 3-4) Evaluat ion, l ike research, dwells in subjectivities and cultural practices. I wonder, as an evaluator, what conditions make it possible to assume a distant and neutral stance? H o w can we c l a i m exact numerical measurements to mark the worth o f one's text? These are questions asked o f difficult stories - difficult stories and their meaning for evaluation i n terms o f val idi ty , as a concern o f ethics, and, o f ethics itself. I now tell o f one difficult story in the name o f validity. F r o m Certainty to Gu i l ty Judgments After a discomforting experience wi th 'undecidabil i ty ' , a teacher placed a mark on an essay and commented, Reflecting-in-action: on legitimating a mark (COMMENTARY) •This is one of the most difficult jobs we do as teachers. I want to treat all the essays fairly and be consistent, but it feels like an impossible task.... Look at this one. I know I'm reading into what has been written here. I am his teacher, I know what was covered in class. Hmmm. I know this mark is important to him. It will impact him now and in the future and it reflects on me as his teacher. That makes me nervous. Some would say it reflects on how well I have taught and how well he has learned. But I don't think it gives those kinds of answers. That mark represents my judgment - what I know of what this student knows based on what and how he has written -at least it's supposed to. But, how can a number represent my marking when it is so subjective - even with this criteria in front of me - and the essay - like all essays in my experience - just isn't totally clear? I know I had to mark it but I'm feeling really uncomfortable - even guilty - in justifying that mark. A discomfort ing experience of g iv ing a mark seems to be l inked, first o f a l l , to underlying epistemological assumptions that the teacher can ' know ' the language and content o f the essay as a representation of the 'presence' o f the learner's knowledge o f content and o f language; assumptions seemingly perpetuated by a regime o f 'truth' that informs the 154 presumed ethical use and interpretation o f the evaluation results. 8 1 Ye t , in the discourse data o f the above example is evidence o f a hesitancy to ' k n o w ' such 'truths' as 'pure presence' , recognis ing that "the essay - like all essays . . . - . . . just isn't totally clear." In questioning a powerful tradition, the teacher validates his/her evaluation o f a mark through the diff iculty o f guil ty judgments. Part o f the difficulty that leads to guilty judgments may relate to concerns o f reliabil i ty (e.g., teachers' repeated desire to be consistent), validity, and ethics (e.g., both val idi ty and ethics are l inked to teachers' need to be fair). I briefly comment on rel iabil i ty before explor ing concerns o f validity and ethics from the language testing literature i n relation to classroom integrative evaluation practices. Assumptions of reliabili ty seem to be embedded i n notions o f replication and repetition in suggesting f ixed, static contexts wherein judgments can be repeated in consistent ways. A troubled technical control, attempting to silence the contextual embodiment o f integrative evaluation, is further troubled through the persistant instabilities o f on-going and ever-changing social relations; it is part o f the difficulty o f rendering a motile site reproducible. I now turn to a discussion o f val idi ty and its assumptions, informed by historical and current conditions, to suggest that the pre-conditions o f decis ion-making contribute to a val idi ty that can never be sure o f itself. A s a way of addressing concerns o f val idi ty that emerged i n the discourse data, I re-visit the first question o f inquiry to consider how the discourse evidence contributes to interpretations o f val idi ty as 'd i f f icul ty ' . V a l i d i t y , as referred to here, is concerned wi th "the conditions o f the legit imation o f knowledge" (Lather, 1994, p. 36); that is, attention is given to the historical and/or current conditions that a l low bounded meanings to be constructed i n particular ways. F o r example, already discussed are the conditions constructed i n the discourse above regarding assumptions o f ' truth' and 'representation' made midst the difficulty o f legit imating the teacher's knowledge through her judgment o f the learner's knowledge o f the content and language o f the essay. Debates i n the field of language testing over the last decade or so, concerned wi th validity and test influence, have moved away from a narrow focus on washback, a consequential aspect of validity that is 'measurement-driven instruction' , to explorations o f 8 1 One of the uses of evaluation results is an ethical concern termed 'washback' and refers to "a set of beliefs about the relationship between testing and teaching and learning" (Hamp-Lyons, 1997, p. 295). 155 broader notions o f impact (Hamp-Lyons , 1997). M a n y use validity criteria categories to legitimate testing instruments and the knowledge they purport to measure: "face validity (student perceptions o f the test), content val idi ty (asking whether tutors thought that test content represented programme content), construct val idi ty (through a correlational study), [and] concurrent validity (with self-assessment, and the assessment o f tutors)" (Fulcher, 1997, p. 115). Current w o r k by M e s s i c k (1994, 1995, 1996) defines a broad notion o f construct val id i ty . 8 2 A l l seem to attempt, albeit i n different ways, the validation o f the testing/evaluation instrument as one that captures ' real ' student competence/performance. It is a stance that continues to be entangled i n the crisis o f representation and its assumption o f an obtainable 'pure presence'. Ye t , teachers also acknowledge a gap in ever-coming-to-know completely what is constituted i n the words written, cal l ing into question the possibil i ty o f 'pure presence' o f knowledge assumed to be represented in the text. In the title of one current article i n language testing, validity is considered an ethical issue (Hamp-Lyons , 1997). Some categories o f validity criteria, such as face validi ty, blur wi th ethical concerns o f test influence and learner affect. It is a shifting ground wherein current discussions o f val idi ty i n research practices question "the continued dominance o f homogeneous criteria as formulaic, readily available codes that reinscribe a realist ontology and serve a regulatory, po l ic ing function" usurping them "to a space o f relational practices i n situated contexts o f inqui ry" (Lather, 1998, p. 4). Exp lo r ing notions o f val idi ty as it relates to research, Lather 's (1986) concept of catalytic val idi ty is one that "represents the degree to w h i c h the research process reorients, focuses, and energises participants toward k n o w i n g reality in order to transform i t" (p. 72). It is an example o f a shifting orientation o f val idi ty - a middle way; it is "a situated, constitutive validity . . . based on multiple voices about what knowledge should matter." (Lather, 1998, p. 4). This shift can be tracked in language testing i n an obscuring of distinctions between validity and ethics (Hamp-Lyons , 1997; L y n c h , 1997). Re-written for evaluative practices, validity becomes an ethical concern wherein legitimation o f what-knowledge-should-matter is constituted by multiple, relational 8 2 Messick describes construct validity as including "evidence and rationales for evaluating the intended and unintended consequences of score interpretation and use in both the short- and long-term...[and] unfairness in test use, and positive or negative washback effects of teaching and learning" (Messick cited in Hamp-Lyons, 1997, p. 298). 156 voices o f the 'testmaker' and 'testtaker'. Evident as traces throughout the discourse data, such possibilit ies emerged when teachers recognised that their method of evaluating was a situated response contextualised by conditions, for example, wi th in which the prompt and criteria were established. In acknowledging the cultural and relational dynamics o f the whole activity, teachers began to question the ethics and responsibility o f assuming the posi t ion 'of expert as a normative condi t ion for evaluative practices and instead began to understand it as a " w h o l l y human endeavour." F r o m Ethics to Obl igat ion as/in R i s k y Practices E th ica l concerns in language assessment, framed by questions o f impact as effects, are o f current import such that a whole issue has been devoted to this topic i n a w e l l respected journal on language testing (see the journal o f Language Testing, 1997, V o l . 14). W h i l e many studies inquire about ethics from perspectives o f teachers and test-makers, the current ca l l is for further research on learners, "v iews and their accounts o f the effects on their l ives o f test preparation, test-taking and the scores they have received on tests" for both traditional and alternative forms o f assessment practices (Hamp-Lyons , 1997, p. 299). In light o f this ca l l , the ethicality o f tests has been questioned (Davies, 1997; Elder , 1997; L y n c h , 1997; Shohamy, 1997). Def in ing ethicality " i n terms o f issues such as harm, consent, confidentiali ty o f data and fairness", L y n c h (1997) bo ld ly asks, can "any t e s t . . . be defended as ethical , or mora l " (p. 315)? H e argues for "alternative forms o f assessment" as "our best hope o f improv ing the moral i ty o f procedures and the decisions that result f rom them" (p. 324) and suggests that their legit imation w i l l require "different procedures from those currently employed for traditional tests...and may involve goals that are difficult to capture wi th precise instruments o f measurement" (ibid.). It is a shift i n the language testing literature that resists the "quest for accurate measurement - and con t ro l" and argues "for [more] educationally and moral ly defensible po l ic ies" (ibid.) . It is , as L y n c h claims, a continued search for ethical assessment - an ethics-in-the-name-of-the-good - that may involve r isky practices i n its defiance o f accurate measurements and control. I turn briefly to the work of Caputo (1993) and his argument against ethics as a way of th inking about ethics as r isky practices. Tak ing a deconstructionist stance, his denial o f 157 ethics and its good name is "an operation aimed at appreciating that tenuous and delicate situation o f judgment w h i c h is addressed by the name 'undecidabi l i ty" ' (p . 3). H e c la ims that to speak against ethics-in-the-name-of-the-good and its deconstruction is to o w n up to the l ack o f safety by w h i c h j u d g i n g is everywhere beset . . . . [It] is the loss o f the assurance, the l ack o f the safe passage, that ethics has a lways p romised . E t h i c s makes safe. It throws a net o f safety under the judgments we are forced to make, the dai ly , hourly decisions that make up the texture o f our l i ve s . E t h i c s l ays the foundations for pr inciples that force people to be good; it c l a r i f i e s concepts , secures j udgmen t s , p r o v i d e s f i r m guardrai ls a long the s l ippery slopes o f fac t ica l l i f e . It provides pr inciples and cri teria and adjudicates hard cases. Eth ics