UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Preventing early reading failure : an examination of the implementation of a phonological awareness program… Smith, Veronica 2004

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2004-931781.pdf [ 4.63MB ]
Metadata
JSON: 831-1.0099785.json
JSON-LD: 831-1.0099785-ld.json
RDF/XML (Pretty): 831-1.0099785-rdf.xml
RDF/JSON: 831-1.0099785-rdf.json
Turtle: 831-1.0099785-turtle.txt
N-Triples: 831-1.0099785-rdf-ntriples.txt
Original Record: 831-1.0099785-source.json
Full Text
831-1.0099785-fulltext.txt
Citation
831-1.0099785.ris

Full Text

PREVENTING E A R L Y R E A D I N G F A I L U R E : A N E X A M I N A T I O N OF T H E I M P L E M E N T A T I O N OF A P H O N O L O G I C A L A W A R E N E S S P R O G R A M FOR YOUNG CHILDREN by V E R O N I C A SMITH  B . A . , Western Washington University, 1982 M . A., Western Washington University, 1985  A THESIS SUBMITTED IN P A R T I A L F U L F I L L M E N T OF T H E REQUIREMENTS FOR T H E D E G R E E OF THE DOCTOR OF PHILOSOPHY in T H E F A C U L T Y OF G R A D U A T E STUDIES (Department of Educational Counselling Psychology, and Special Education)  We accept this thesis as conforming to the required standard  T H E UNIVERSITY OF BRITISH C O L U M B I A September 2004 © Veronica Smith, 2004  Abstract This study examined the effectiveness of an early reading program, Firm Foundations, implemented in 13 kindergarten classrooms as part of a school district wide literacy initiative in 10 schools in a northern community in western Canada. Phonological awareness, spelling, and reading skills of each child (N = 151) in the participating classrooms were measured prior to and after implementation of the program. The level of implementation of the program, the school and child factors that influenced implementation, and the relation between implementation and child outcomes were examined. A l l teachers completed background and belief questionnaires, and principals in the participating schools were interviewed to determine their level of support for the program iri their respective schools. Multilevel modeling techniques that accounted for the nested structure of the data showed that children who were taught more components of the Firm Foundations program were at less risk for reading failure on many of the outcome measures. Factors that were associated with higher levels of implementation of the program were principal support, teacher background and belief in the program, and a smaller number of children identified as at risk for reading failure in each classroom. The results support the use of early literacy programming, which includes phonological awareness and letter-sound mastery, for children at risk for reading failure. These findings are discussed in regard to the importance of conducting theory-based program evaluation of routine practice in classrooms and the importance of understanding the processes underlying successful program dissemination of classroom-based interventions.  iii  Table of Contents Abstract  ii  Table of Contents  Hi  List of Figures  vi  Acknowledgment  vii  Introduction  1  Key Components of Early Literacy Interventions  3  The Case for Early Identification and Prevention in Early Literacy  11  An Examination of Implementation of Early Literacy Programs in Routine Practice  17  Firm Foundations: An Early Literacy Program  22  Research Base and Assumptions_  23  Design and Implementation of the Firm Foundations Program  25  Teacher Training and Support  27  .  Statement of the Problem and Overview of the Study  27  Method  i  Participants...  ;  28  Measures of Implementation Quality  29  Teacher Backgrounds and Beliefs Principal Support  28  29 .  .  30  iv Observations of Quality and Dosage of Literacy Activity in Classrooms  31  Measures of Student Literacy Progress  33  Phonological Awareness  33  ;  Letter Naming  ;  Word Reading  *  - •.'  Spelling  34 34 34  Results  35  Classroom Activities and Settings  35  Relationships Among Ecological Variables  38  Child Outcomes  41  :  Statistical Model for Linking Child Outcomes and Implementation of Firm Foundations  ;  ,  42  Level of Implementation  46  Literacy Components  49  Instructional Methods of the Firm Foundations Program  52  Discussion References  54 ;  ;  64  Appendix I Teacher Questionnaire  81  Appendix II Principal Support Measure  86  Appendix III Observation Protocol  89  Appendix IV Student Literacy Measures  95  V  List of Tables Table 1. Firm Foundations Literacy Components  101  Table 2. Demographic Information for Participating Students  102  Table 3. Percentage of Whole Class, Partial.Class, and Total Class Time Spent on Literacy Activities by Classroom  103  Table 4. Percentage of Time Spent on Firm Foundations Literacy Components by Classroom Table 5. Correlations.....  104 .  105  Table 6. Means and Standard Deviations at Pre and Post Test and ts for Groups of Students for Each Literacy Measure  106  vi  List of Figures Figure 1. Model of Implementation  Acknowledgment  This work would not have been possible without the support of the students, teachers, and administrators who kindly agreed to participate. I would like to acknowledge my supervisor, Dr. Linda Siegel for making the funding for this research study possible through a grant from the Canadian Language and Literacy Research Network and for her encouragement and guidance throughout the research and writing process. The untiring guidance and help of my research committee, Dr. Bruno Zumbo and Dr. Kim Schonert-Reichl, are highly appreciated. Their constructive comments and invaluable suggestions have significantly improved the contents of this dissertation. Mr. Stan Auerbach, instructor and research sage, is highly acknowledged for his contribution to the design of the observational protocol, guidance throughout the process, and his reading of the final manuscript. Dr. Pat Mirenda is acknowledged for inspiring me to embark on a doctoral program of study, placing scholarly challenges before me throughout my academic program, and for her encouragement and support along the way. I wish also to extend my gratitude to some special people who are always there for me in spite of everything. To my beloved son, Thomas, for knowing when I needed a hug; to Roy, for being supportive in many ways; to my mother and father, and the rest of my family for their commitment toward my desire to finish my doctoral degree. Finally, to Angela Jaramillo, whose angle-like support has been there around every corner. September 7, 2004  Introduction During the past 30 years, there has been a dramatic rise in the research on the prevention of reading failure in young children. There is now a corpus of research findings that indicates a consistent relation between reading and phonological awareness. Phonological awareness is defined by Snow, Burns, and Griffin (1998) as the general ability to attend to the sounds of language - for example, that "boy" and "toy" begin with different sounds. Deficits in attending to and manipulating the sounds of language are seen as a probable cause of reading failure in children (e.g., Foorman, Francis, Fletcher, Schatschneidr, & Mehta, 1998; Siegel, 2003; Torgesen et al., 1999; Vellutino et al., 1996) and intervention programs, based on the positive influence of the inclusion of phonological awareness activities, have demonstrated that systematic, explicit instruction in phonemic awareness and phonetic decoding skills produce stronger reading growth in children with phonological weaknesses than do those that do not teach these skills explicitly (Brown & Felton, 1990; Felton, 1993; Foorman et al., 1998; Torgesen et al., 1999; Tunmer & Nesdale, 1985). Nevertheless, despite the growing consensus regarding the best instructional focus or the combination of instructional foci that reduce or prevent reading failure in young children, it is still unclear how much of that instruction, delivered under what conditions, will lead to adequate development of reading in children with phonological weakness (Torgesen, 2000). With the increased understanding of the influence of phonological awareness on reading failure has been a parallel recognition of the critical importance of early preventive efforts in enhancing developmental outcomes for children (Offord, Kraemer,  2 Kazdin, Jensen, & Harrington, 1998). School districts are eager to implement universal programs that will enhance reading outcomes for all children, especially those at risk for reading failure. Additionally, educators are encouraged, or required, to implement programs that are "evidence-based" or have strong empirical support. Wilson, Lipsey, and Derzon (2003) observed that many of the programs that have strong empirical support are demonstration programs that are set up by researchers and conducted largely for research purposes (i.e., to determine program efficacy under controlled conditions). Demonstration programs are typically implemented and evaluated by a researcher mainly for research or demonstration purposes. Such programs would not generally be implemented without the interest of the researcher and are typically delivered by the research staff or providers (e.g., teachers) closely supervised by the researchers. According to Wilson and colleagues, more rare in the evaluation literature are programs implemented on a routine, ongoing basis in schools. In contrast to demonstration programs, routine practice programs are those that already exist in the school on an ongoing basis, are conducted by school staff, and are evaluated either by school-based or outside researchers. At present, there is a very limited knowledge base on the implementation of routine practice programs, factors that influence the quality of implementation, and the relation between the quality of implementation and the outcomes obtained for children in school settings. Such research is essential to understanding the local conditions and training systems necessary for broad and effective program dissemination (Durlak, 1998; Zins, Elias, Greenberg, & Weissberg, 2000). Determining the effectiveness of early reading intervention preventive efforts in classrooms has evolved to the point where understanding the implementation process and  3 the factors that support it is essential to the field's continued growth. The purpose of this study was to examine the implementation of a kindergarten literacy program that includes the recommended practices of systematic, explicit instruction in phonemic awareness and phonological decoding skills and was conducted in routine practice in a school setting. This project was conducted as part of a school district initiative intended to prevent reading failure early in a child's school career and to enhance literacy outcomes for all learners. The following review of the literature will outline the components of early literacy interventions that have demonstrated improved outcomes for at risk readers and briefly provide a rationale for engaging in prevention rather than remedial literacy programs for all beginning readers. The importance of conducting a theory-based evaluation of preventive interventions will be argued and a theoretical model for an implementation evaluation will be discussed. This will be followed with a brief description of the components of the literacy intervention that is the subject of this study.  Key Components of Early Literacy Interventions Becoming literate is a developmental process that involves a connection of listening, speaking, reading, and writing. Implied but only recently understood in the connections one has to make between the spoken and written aspects of language is an explicit understanding of the metalinguistic features of the language. The need for metalinguistic awareness, or "an awareness about language" has come to our attention because of controlled studies that have demonstrated the effects of phonological awareness on early reading and spelling abilities (e.g., Ball & Blachman, 1988, 1991; Bradley & Bryant, 1985; Cunningham, 1990; Lundberg, Frost, & Peterson, 1988; Torgesen, Morgan, & Davis, 1992). Before discussing the key components of early  4 literacy interventions, it is important to clarify what is meant by phonemic awareness, phonological awareness, and phonological decoding. These are terms that are often confused or ill-defined in the reading literature are important to understanding the key components of early literacy interventions. The most carefully studied part of the early reading process has been phoneme awareness. Over 30 years ago, it was demonstrated that phonemes, the smallest units of spoken language, were difficult for some individuals to distinguish between because phonemes fuse or blend together within the spoken syllable (Liberman, Cooper, Shankweiler, & Studdert-Kennedy, 1967). The difficulty in perception of phonemes is complicated by the number of phonemes in any given language. For example, English is made up of approximately 41 to 44 phonemes, depending on the dialect. Phonemic awareness is the ability to focus on and manipulate phonemes in spoken words. It includes skills such as phoneme isolation (e.g., "Tell me the first sound in ball." Ibl), phoneme identity (e.g., "Tell me the sound that is the same in ball, berry, and bad." Ibl), phoneme categorization (e.g., "Which word does not belong? Ball, bad, or ran."), phoneme blending (e.g., "What word is /h/e/l/p/?" (help)), phoneme segmentation (e.g., "How many sounds in the word 'spell'?" (four)), and phoneme deletion (e.g., "What is spoon with the /s/?" (poon) ). Phonological awareness is a broader metalinguistic term, that refers to various types of awareness, phonemic awareness being one of them, but also awareness of other spoken units such as syllables, onsets, and rimes. Phonological awareness activities include rhyming (e.g., "Tell me a word that rhymes with bad."), segmenting a combined word (e.g., "What is cowboy without the cow?"), segmenting a polysyllabic word (e.g., "What is sunny without the sun?"), or segmenting the one-  5 syllable words into their onsets (i.e., the single or multiple consonants that precede vowels as in j_-ump, str-ing) or rimes (i.e., the vowel and following consonants than follow the onset as in, i-ump, str-ing). Phonological decoding, which is also referred to as letter-sound mastery or lettersound knowledge, refers to the ability to make letter-sound correspondences (e.g., that the word tap says /t/-/ae/-/p/). Understanding and instruction of phonological decoding has changed over time. The phonological decoding of the 1960s did not help children understand the systematic links among phonemes and graphemes (Foorman & Torgesen, 2001). In the 1980s, practice in teaching phonics began to highlight the significance patterns and similarities of sound and grapheme representations. A n example of this phonics teaching in kindergarten might be helping children see that the Iml in mat and the Iml in moon were the same sound, and that this sound Iml is represented with the same letter Iml. Instruction in phonics has become more systematic and explicit. This explicitness is aimed at increasing a child's representations of the spoken and written linguistic codes, directly linking phoneme and grapheme relationships. Phonological awareness, and more specifically phoneme awareness, has figured prominently in developmental theories of how children learn to read words (Morris, Bloodgood, Lomax, & Perney, 2003; Siegel, 2003). In addition to teaching phoneme identities in words, (i.e., how sounds are represented by words), letter-sound awareness or phonological decoding is also seen as an important component of learning to read words (National Reading Panel, 2000; Foorman & Torgesen, 2001). Comprehensive descriptions of how word knowledge develops in the beginning reader have emerged (Ehri, 1998; Siegel, 2003). Phonemic awareness progressively  unfolds in reading acquisition and these skills help children learn to read graphemes or words in a variety of ways (Ehri, 1991). At first, beginning readers can only attend to the beginning sound in a spoken word; later to the initial and final sounds; and, finally, to each sound in the word. Increases in phoneme awareness lead to letter-sound processing, which, in turn, allows more and more words to become part of sight word memory. Beginning readers must know how to blend phonemes to decode words and beginning readers must be able to segment words into the phonemes that match up to graphemes so that they can make connections between graphemes and phonemes and store them in memory (Ehri, 1992, 1999). Additionally, phonemic segmentation skills are essential for constructing probable spellings as well as remembering correct spellings of words (Griffiths, 1991). Ehri (1994) contends that all of the processes involved in learning to read and write words require phonemic awareness and phonemic decoding. Improving the ability to focus on and manipulate phonemes in words and to discern larger units of sounds in words can be enhanced (Catts, 1991) and researchers have shown that teaching aspects of phonological awareness significantly improves beginning readers' success in learning to read [National Reading Panel (NRP), 2000]. As early as 1984, Bradley and Bryant (1984), in a study of kindergarten students, demonstrated that specific language experiences of phonological awareness could be offered to young children to positively affect reading acquisition. Bradley and Bryant hypothesized that phonological awareness is a necessary skill in learning to read and, subsequently, many studies have been undertaken to clarify the components of this hypothesized causal connection.  7 The determination of which aspects of phonological awareness and phonic decoding, (i.e., letter-sound knowledge), are most important in learning to read has been the subject of many research studies and has influenced the development of many different types of early reading programs. Many early reading programs have included skill development in rhyming, segmenting and blending, letter-sound mastery, and concepts of print. Each of these skills will be considered in turn in regard to the evidence that supports their unique contributions to the prevention of reading difficulties. Rhyming. There is a persistent belief and observable practice among early reading teachers that instruction in rhyming is related to the prevention of reading difficulties (Yeh, 2003). Early research by Bradley and Bryant (1985) established that many children who have difficulty learning to read also have difficulty with rhyming. In a longitudinal study of reading development, Bradley and Bryant (1985) found that rhyming had a consistent and significant relationship to progress in reading and spelling. However, there is little evidence that training of rhyming contributes significantly to the prevention of later difficulties. In a study by Lundberg (1988) and a subsequent replication by Schneider (1997) the influence of rhyming ability on word and syllable awareness was modest compared to the effects of other skills development, such as segmenting and blending in Danish and German. In a recent study in English, Yeh (2003) tested the individual contributions of different aspects of phonological awareness to reading acquisition. He proposed that rhyming alone was not as effective as explicit instruction of phonemic segmenting and blending in learning to read because it is not easy to teach children to blends and segment phonemes into words, rhyming might serve as an approach for teaching children the structure of words, specifically the onsets (i.e.,  8 beginning sounds in words, such as "b" in bone or "st" as in stop) and rimes (e.g„ "ing" as in string; or " i l l " as in skill). Yeh (2003) proposed that further investigations may discern the effect of teaching young children to segment and blend onsets and rimes, in comparison with the effect of teaching rhyming alone. Without clear direction from the literature, rhyming remains an important component of most early reading programs; however, in recent years, the emphasis in early reading instruction has shifted to the importance of the contributions of segmenting and blending in preventing reading difficulties. Segmenting and Blending. Several studies (e.g., Davidson & Jenkins, 1994; Reitsma & Wesseling, 1998) have compared the contributions of segmentation and blending instruction to reading acquisition. Davidson and Jenkins (1994) studied four groups of young children; one group was taught segmenting only, one group blending only, one group both segmenting and blending, and the final group had neither type of instruction. In this study it was demonstrated that each group performed the skill that they were taught but performed poorly on the skill that they were not taught. On measures of reading and spelling, the segmentation and combinations group performed best, suggesting that segmenting was the superior skill to teach. The importance of blending skill development in young readers was demonstrated by Reitsma and Wesseling (1998) in a study examining the use of a computer assisted training program. They found that only students who achieved superior blending skills from training, as opposed to better segmenting skill, demonstrated better reading skills a year after the treatment ended. Despite these somewhat contradictory findings, the general conclusion in the reading literature is that both segmenting and blending are important in learning to read; and  9 consequently both skills are emphasized in many early reading programs (Foorman, 2001). Letter-Sound Mastery. Another important component of early literacy interventions is the introduction of letters when teaching phonemic awareness. Bradley and Bryant (1983, 1985) conducted some early experiments that provided evidence for the added benefits of including letter-sound instruction with phonemic awareness instruction when teaching reading. In their two year intervention study, children 5 to 7 years old identified with low phonemic awareness skills were randomly assigned to one of three groups of individualized instruction: only phonemic awareness activities; phonemic awareness plus letter-sound instruction; or, in the control group, instruction to cluster words by conceptual categories. The results at the end of the instruction showed that those in the phonemic awareness plus letter-sound instruction group performed better on standardized reading and spelling tasks than either the control group or the phonemic awareness only group. This study offered strong evidence for the combined use of phoneme awareness and letter-sound awareness activities; however, because the intervention was individualized, it was difficult to ascertain the applicability of this type of instruction for groups of children. Subsequently, in series of experimental studies, Blachman and her colleagues (Ball & Blachman, 1991; Blachman, Ball, Black, & Tangel, 1994; Tangel & Blachman, 1992) were able to demonstrate that phoneme awareness instruction, combined with instruction connecting the phonemic segment to alphabet letters, significantly improved the early reading and spelling skills for groups of kindergarten children. In contrast, children who received instruction in letter names or in letter sounds alone did not  10 significantly improve in segmentation skills, early reading skills, or spelling skills, even when compared to comparison groups of children who received neither instruction. These studies confirmed that either individually or in groups, children benefit from instruction that combined both phonemic awareness and letter connections. Concepts of Print. Although given much less attention than the contributions of phonemic awareness and letter-sound connections in learning to read, a child's developing concept ofprint has also been theorized to contribute to the development of reading. Marie Clay (1972) was one of the first reading theorists to discuss the importance of developing an awareness of word units in text, or concepts of print, for beginning readers. From her longitudinal observations of beginning readers, Clay (1991) noted that, in order to read simple texts, the child must "break up his produced speech into word units; locate the visual patterns; move in the correct direction; and co-ordinate the timing of his pointing and looking with his uttering" (p. 162). According to Clay's observations, the act of finger pointing (i.e., matching spoken words to written words) was important in making these letter-sound connections. Morris (1983; 1993) also viewed accurate finger-pointing to spoken text as an important step in learning to read. Morris (1983) found strong positive correlation between beginning readers' ability to fingerpoint a short memorized verse and their ability to segment spoken words into phonemes. In a subsequent investigation, Morris and his colleagues (1993) found that after assessing kindergarteners four times during the school year, early phoneme awareness (i.e., beginning consonant awareness) preceded the ability to finger-point, which in turn preceded a more mature form of phoneme awareness (segmentation), which in turn preceded word recognition ability. A structural equational model of this hypothesized  11  developmental sequence was subsequently applied to longitudinal data (Morris et al., 2003) confirming that relationships exist between concepts of print, specifically the concept of words in text, and developing reading skill. This evidence has provided support for including instruction of concepts of print in early reading programs. In summary, the metalinguistic skills that enhance a child's beginning reading skill have been described. The research indicates that there is a relationship between a child's phonological awareness and success in learning to read, specifically on the development of word recognition. This relationship has influenced the components of many early reading programs. Rhyming remains a key component of many reading programs. The teaching of phonemic awareness, specifically, segmenting and blending of individual phonemes in words, has been shown to be positively associated with success in learning to read. Additionally, when phonemic awareness instruction is combined with :  letter-sound awareness instruction, the reading outcomes for children are more positive. Although given less attention by empirical studies, concepts of print, specifically the concept of word in text, appears to contribute to the developmental sequence of learning to read and remains an important component of many early reading programs. At this point, intervention studies that have incorporated the use of these key components for preventing reading failure in young children will be examined.  The Case for Early Identification and Prevention in Early Literacy During the primary years, developing competency in reading and writing are the expected outcomes for all children. In fact, 75-85% of children learn to read by participating in classrooms where they are engaged in activities that include listening comprehension (i.e., oral language vocabulary development) shared and independent  12 reading, and writing (Simmons, Kame'enui, & Good, 1998). Although competency in reading and writing are needed for success in school, failure to learn to read has a more detrimental long-term negative effect on overall cognitive development (Cunningham & Stanovich, 1998). Research has demonstrated that children who have not learned to read by the end of grade one are at considerable risk of being disabled readers for the remainder of their school years and through adulthood (Frances et al., 1996; Juel, 1988). Additionally, research indicates that children who get off to a poor start in reading rarely catch up (Torgesen, 1997). One major solution to the problem of school failure in general, and reading failure in particular, is early identification and prevention. Studies have clearly demonstrated that children who are at risk for reading failure can be detected early. For example, in a study conducted by Juel (1988), it was found that word recognition skills at the end of the first grade were strongly related to reading proficiency by grade 4. Further, Torgesen (1997) demonstrated that these patterns could be detected as early as kindergarten. Other research has demonstrated that those at risk for reading failure often have lifelong consequences i f not provided with intervention. For example, in a longitudinal study of reading development from kindergarten to grade 12, Shaywitz, Fletcher, and Lyon (1998) found that groups of children identified as at risk for reading failure never caught up with the non-reading disabled group despite the fact that schools identified and provided special education services. The authors concluded that, regardless of how they were defined, reading disabilities are often chronic, lifelong incapacities that lead to problems in a variety of social and vocational areas in adolescence and adulthood.  13 Combined, these studies make clear two critical points: children who are at risk for reading failure can be identified early and, more significantly, children do not typically "catch up" on their own once identified. Both Torgesen (1997) and Juel (1988) concluded that unless addressed with well-designed instruction, struggling readers stay that way. Following is a review of the literature that has examined specific attempts to improve reading skills, first through remedial efforts, then through preventive efforts. Remediation models are those that provide intensive intervention after a child has been identified as reading disabled. In this model, an assessment of reading skills is undertaken and goals are specified to address deficits. In a review of the learning disabilities literature, Lyon and his colleagues (2001) suggested that, although remedial efforts have proven beneficial to many, there are two main reasons why these efforts are not preferred when compared to preventative efforts. First, the standard instruction provided through remediation is frequently too little, too general, and too unsystematic (Lyon et al., 2001). Vaughn studied children with reading disability who were taught for an entire year in public elementary school special education resource rooms (Vaughn, Moody, & Shuman, 1998) and found that the children's instruction was characterized primarily by whole group reading instruction to large groups of children who also varied in grade level ( 3 - 5 grades); and little individualized or differentiated instruction occurred. Although, in a follow-up study two years later, Moody, Vaughn, Hughes, & Fischer (2000) showed that even though more of the teachers were utilizing materials that supported differentiated instruction, neither study found evidence that children made significant gains in reading. Several earlier studies by Vaughn (as cited in Vaughn,  14 Moody, & Shuman, 1998) also failed to find evidence supporting significant gains in reading skills through specialized reading instruction programs. The second reason why remediation models may be less preferred to prevention models is that the instruction may be applied too late (Lyon et al., 2001). For example, Lyon and his colleagues found that children were typically identified with reading disability by grade 3. Many of these children were already far behind and less motivated to learn to read. Juel (1988) found that children who are poor readers by grade 3 read less and this effects overall outcomes because time spent on reading is highly correlated with achievement in learning to read (Allington, 1984; Stanovich & Cunningham, 1998). Thus, because some remedial efforts are applied so late that children have lost their interest or motivation to read, researchers and practitioners are drawn to preventive efforts for children identified as at risk for reading disability to avoid the negative cycle of events that is frequently found in schools when children fall behind their peers in reading early in their school years. Preventive interventions are those that occur before the initial onset of the condition, delay, or disorder (National Institute of Mental Health, 1996). In order for the preventive model to be effective, a specification of the intervention goals is also necessary. Unlike the remedial model where goals are typically designed around deficits, effective preventions require an understanding of the protective factors that contribute to positive outcomes for children identified as at risk (Greenberg & Weissberg, 2001). In the case of early reading interventions for children in kindergarten, clear protective factors have been identified in the previous review, with notable attention given to explicit instruction in phonemic awareness and phonological decoding.  15  Positive results for children that have not yet learned to read have been reported on a variety of prevention demonstration programs where protective factors are identified (e.g., Blachman, Ball, Black, & Tangel, 1994; Bos, Mather, Friedman, Narr, & Babur, 1999; Lesaux & Siegel, 2003; Lundberg, Frost, & Peterson, 1988; O'Connor, 1999). For example, in a strictly phonemic awareness program, Lundberg, Frost, and Peterson (1988) examined the effectiveness of 8 months of instruction in phonemic awareness in a control group design. At the end of the second grade the phonemic awareness group performed better than the control group on both the reading and spelling tasks. Other researchers have replicated these finding (e.g., Kjeldsen, Niemi, & Olofsson, 2003; Kozminsky & Kozminsky, 1995; Schneider et al., 1997). As noted previously in the review of the essential components of reading instruction, Blachman and her colleagues (Ball & Blachman, 1991; Blachman, Ball, Black, & Tangel, 1994; Tangel & Blachman, 1992) were able to demonstrate that phoneme awareness instruction, combined with instruction connecting phonemic segments to letters, significantly improved the early reading and spelling skills of groups of kindergarten children. In a longitudinal study of the effects of a district-wide balanced literacy program that included explicit instruction in phonological awareness and phonological decoding in kindergarten, Lesaux and Siegel (2003) demonstrated that all children, including those with English as a second language, benefited from the early preventive instruction. Foorman and Torgesen (2001), in a review of the critical elements of instruction to promote reading success in all children, concluded that evidence from these programs demonstrates that "dramatic reductions in thejncidence of reading failure [occurs] when explicit instruction of these components are provided by the classroom teacher," (p. 203).  16 On the basis of this converging scientific evidence, the development and implementation of early identification and prevention programs for children at risk for reading failure is sufficient to justify benefits (Lyon, et al., 2001; NRP, 2000). Students at risk for reading failure can be identified early and benefit from preventive interventions. In contrast, remedial programs that have been the cornerstone of special education programs for decades appear to have mixed results. Starting early with systematic, explicit instruction appears to alter the course of outcomes for many children at risk; however, much of the evidence supporting the positive effects of prevention programs comes from demonstration programs, or those developed and conducted by researchers. Whether this research base translates into feasible classroom practice is much less clear (Gersten & Brengelman, 1996). What is missing from the literature is a clear understanding of the processes underlying the successful dissemination of these programs in routine practice by teachers in classrooms. This understanding is warranted by literature that indicates that some teachers do not have the knowledge or skills to teach reading, and that many teachers have not been traditionally required to teach reading as early as kindergarten. Phonemic awareness or phonological awareness are both terms that are often misunderstood (Ehri & Nunes, 2002) and studies have demonstrated that teachers who are experienced and literate have an insufficient grasp of spoken and written language structure, especially aspects such as phonemes or phonological awareness, to teach it explicitly (McCutchen, 2003; Moats, 1995). Additionally, surveys of teachers have revealed that understanding how we think about sounds as they relate to reading is not a consistently taught feature of teacher education programs (Lyon, Vaasen, & Toomey, 1989), and that few teachers in kindergarten identify with the role of teaching  17 phonemic awareness as part of their role in introducing reading to young children (Ukrainetz & Fresquez, 2003). Additionally, across North America, teachers are expected to address the needs of an increasingly diverse group of children. Within one school district, school age populations may vary considerably in terms of their family experiences, cultural experiences, and socioeconomic status (Hertzman, 2004). A n understanding of routine practice conditions is necessary to determine how preventive reading interventions that have demonstrated success in preventing reading failure can be "scaled up" in conditions that vary from school to school and classroom to classroom. In the next section, a model of implementation evaluation of routine practice will be considered.  An Examination of Implementation of Early Literacy Programs in Routine Practice Despite successful results for reading outcomes from demonstration projects conducted primarily by university researchers, there continues to be a gap between what we know from research about "what works" and what is actually implemented by teachers in practice (National Joint Committee on Learning Disabilities, 1999; Stanovich & Stanovich, 1997). Further, recent large-scale observational studies in early elementary classrooms indicate that, in regular primary classroom settings, there is huge variability in the instructional techniques, objectives, and overall quality of the instruction (Pianta et al., 2002). Somehow, successful implementation of prevention programs in school settings must accommodate the variability and the many factors that influence classroom instruction. Examination of this variability and the factors that support or detract from successful implementation is useful for both theoretical and practical reasons. In this section, what is meant by implementation will be defined, procedures for understanding  18 the variability in implementation will be discussed, and a model for evaluating environmental factors that support or detract from implementation will be introduced. Consideration of all of these dimensions will allow the evaluator to test program theory and to provide practical feedback to enhance routine practice of empirically supported instruction. At the outset, it is important to define implementation. Two general features of implementation described by Dane and Schneider (1998) will be adopted in this discussion: implementation promotion and implementation integrity. Implementation promotion refers to all of the supports that are available to support the implementation of the program. These have been outlined by Greenberg et al. (2002) to include: preplanning (capacity, awareness, commitment), quality of materials (design and format of materials), technical support (training delivery and content), and implementer readiness (perceptions, skills, knowledge). Implementation integrity has been referred to as implementation quality (Dane & Schneider, 1998; Gresham, 1989; Gresham, Gansle, Noell, Cohen, & Rosenblum, 1993), fidelity (Moncher & Prinz, 1991) and adherence (Dane & Schneider, 1998). Despite the variation in terms, the integrity of an implementation has been defined as the degree to which an intervention is conducted as it was originally intended (Durlak, 1995; Yeaton & Sechrest, 1981). The science regarding how routine practice programs are implemented in natural conditions is poorly developed (Greenberg, et al., 2002). Recently, a model of implementation was developed and described in a paper commissioned by the Center for Mental Health Services (CMHS) of the U.S. Department of Health and Human Services (Greenberg, et al., 2002). The rationale for creating this model was to provide a summary  19 of the critical factors identified in the literature as having the potential to affect implementation quality and to organize the factors in a coherent way. The model was informed by work conducted in the field of evaluation research focuses on program theory and theory-driven evaluation (e.g., Weiss, 1995), particularly that of Chen (1990; 1998), and was intended for use by a broad audience including practitioners, researchers, and other stakeholders who are invested in deliver high-quality prevention programs (Graczyk, Domitrovich, & Zins, 2003). To date, this model has not been applied in an evaluation of an early literacy reading program designed to prevent reading failure. A brief summary of the aspects of this model that were used in this implementation evaluation follows.  .  According to Chen (1990, 1998), there are two major components of a comprehensive program theory. The first is the causative or causal theory, also known as the program's theory of change, which identifies how the program produces its intended outcomes. Current theories of the development of reading skills in English stress that the ability to process the phonological, (i.e., sound aspects) of words is the most significant underlying process in learning to read (Siegel, 2003). Thus, the theory of change for many early literacy programs is that children who develop phonological awareness skills will develop word decoding skills, which form the basis of obtaining meaning from print. The second component of a comprehensive program theory is the prescriptive theory, which provides guidelines for delivering the intervention and describes the context that is necessary for the successful implementation of the intervention (Greenberg et al., 2001). For evaluation purposes, it is useful to know not only what the program is expected to achieve but also how it expects to achieve it (Weiss, 1998). Identifying the characteristics  20 of an intervention and the environmental conditions that influence program implementation will allow one to test both the causative and prescriptive theories that mediate or moderate program outcomes. Characteristics of an Intervention. Characteristics of a program can influence intervention quality. The first and probably most important group of characteristics are the key components or "active ingredients" of an intervention (Graczyk, Domitrovich, & Zins, 2003). These include the content (e.g., rhyming, segmenting and blending, lettersound mastery, concepts of print), activities, and method of delivery (e.g., direct instruction, hands-on activities). Current estimates indicate that school-based prevention activities utilize approximately 54% of methods and 71% of the content considered representative of best practices (Gottfedson et al., 2000). Additional program characteristics that have the potential to influence implementation quality i f they are not delivered as originally intended are timing and dosage. Timing is the pace at which the program should be delivered (e.g., every day for 10 weeks or all year on a daily basis), and dosage is the prescribed level of exposure to the intervention and refers to "how much" of the intervention participants should be provided. When programs are conducted in natural settings and implemented by multiple teachers, program characteristics can vary widely and will not always be carried out in exactly the same manner and with the same degree of integrity (Weiss, 1998). Environmental Conditions that Influence Program Implementation. Classroombased programs are nested within ecological systems (Graczyk et a., 2003). Different factors within these systems can either strengthen or weaken program implementation in any given setting. Key ecologies for programs conducted in classrooms include those  created by the individual children, the classroom, the school, the school district, and the community (see Figure 1). These ecologies may vary considerably from setting to setting, influencing the program implementation. For example, at the teacher level, characteristics and behaviours such as beliefs or education could influence the program delivery; at the classroom level, classroom composition or the proportion of children at risk could influence the level of implementation; and, at the school level, administrator support could either enhance or detract from the program. Understanding the conditions that affect the program delivery and effectiveness within different ecologies, is important to provide high external validity to both the theory upon which the program was derived and the delivery of prevention intervention itself (Kam, Greenberg, & Walls, 2003). In a report on implementation practices, Greenberg, Domitrovich, Graczyk, and Zins (2000) urged researchers to take action to advance our understanding of the program dissemination process. In a meta-analytic review of studies published up to 1992, Durlak (1997) found that researchers in less than 5% of over 1,200 published prevention studies provide data on program implementation. A recent meta-analytic review of prevention program evaluation studies published from 1992 - 2003 (Wolf, Durlak, & Bryant, 2004) indicated that implementation evaluation has taken on increasing importance as 46% of researchers now report on quality and quantity of implementation. A search of schoolbased research early literacy interventions revealed that few reports exist regarding the relationship between program implementation and child outcomes in programs intended to prevent reading failure in routine practice. The question of how to integrate phonemic awareness and phonological decoding skill instruction within existing or established  22 language arts programs and subsequently measuring the effectiveness of implementing these programs in routine practice is not well researched (Foorman & Torgesen, 2001). In summary, a model of implementation is necessary to ensure that all aspects of an intervention are adequately considered when evaluating a program's effectiveness in routine practice. It is important to identify the essential elements that are hypothesized to be the mechanisms of change and operationalize these elements on a number of dimensions of strength and quality. Additionally, in school settings where routine practices in the implementation of programs are examined, there are a number of potential ecological influences to program delivery. These elements must also be considered as part of an implementation evaluation. This research is necessary to support the research to practice links to ensure that outcomes for children in routine practice conditions will be.as positive as those found in demonstration programs.  Firm Foundations: An Early Literacy Program The purpose of this research was to investigate the effectiveness of the early reading program, Firm Foundations, implemented in kindergarten as part of a school district wide literacy initiative. The school district that participated in this study serves a large geographical region with a population base of 60,000 in northwestern British Columbia, Canada, where resource based industry creates a fluctuating economy that has been in decline in the past five years. The downturn in both the economy and the population has been reflected in school closures and teacher layoffs in the past two years. Despite these circumstances, the district had commenced a project to improve student literacy with a particular focus on early intervention, specifically for primary grade students. This district initiative was in the second year of operation at the time of this  23 evaluation. Literacy objectives included improving the pre-reading skills of 5-year-olds in schools with "high risk" populations and, generally, to improve the pre-reading skills of all kindergarten students. The early reading programs that the district selected to support their initiative was Firm Foundations. What follows is a description of the research base, intervention components, and teacher training that supports the implementation of Firm Foundations. Research Base and Assumptions Firm Foundations is a universal classroom prevention program that was designed by kindergarten teachers to target and teach early literacy skills early in order to decrease the incidence of children at risk for reading failure. The motivation for developing a literacy skills program specifically for kindergarten grew from the authors' mission to incorporate what is,know from "best practices" regarding literacy development in early readers while still honouring the play-based structure valued in most kindergarten classrooms. Kindergarten teachers have always understood the importance of play in the learning of young children. The kindergarten day is short, typically 150 minutes, and it is a struggle to keep a program child-centred, play-based, and language-enriched while also providing children with early literacy skills important for reading success. One of the Firm Foundation authors wrote about this struggle in the following note in the preface of the program, Like many seasoned veterans of the kinder classroom, I have taught long enough to see once again the pendulum swing toward a more skill-oriented environment and philosophy. Consequently, I have been plagued with a dilemma: could I  24 maintain a language rich environment that offered opportunities for the "whole child" to develop, and still address the teaching of specific skills as suggested by the indisputable educational research? (Ann Echols, Firm Foundations, p. 16) The "indisputable research" referred to is the evidence supporting phonological awareness, letter-sound awareness skills, and concepts of print and their connection to preventing reading failure reviewed in a previous section. Additionally, and more of a concern for kindergarten teachers who have been comfortable with a play-based kindergarten program, is the contention that these skill should be introduced in a manner that is explicit and systematic (NRP, 2000; Snow et al., 1998). Explicit instruction and practice to build phonemic awareness and phonemic decoding skills are particularly important for children who are risk for reading failure. Research by Foorman, Francis, Fletcher, Schatschneider, and Mehta (1998) demonstrated that well-balanced and skilled classroom instruction can dramatically reduce the incidence of reading failure in firstand second-grade classrooms without special interventions for most children. Additionally, both Foorman et al. (1998) and Juel and Minden-Cupp (2000) found that explicit instruction were beneficial for children at risk for reading failure. The challenge of creating a program that was explicit and still included the playbased components that are valued in kindergarten, was met by the kindergarten teachers who, in partnership with special education staff, developed the Firm Foundations program. To accommodate the "explicit" and "play-based" tension, the authors identified two instructional conditions essential to implementing literacy skill components of the Firm Foundations program: 1) direct explicit instruction; and 2) "hands-on" activity time  25 where children experiment and experience the instructional components independently or with coaching from the teacher. The authors aligned these two instructional conditions with the instructional components to support a child-centered, play-based, and language enriched kindergarten environment that also supports literacy skill instruction. Design and Implementation of the Firm Foundations Program The literacy skills taught in the program include rhyming, segmenting and blending, concepts of print, and sound symbol mastery. These phonological awareness and letter-sound awareness activities are hypothesized to be the "active ingredients" in helping many types of children learn to read (Ehri & Nunes, 2002). The causal theory, therefore, is that an understanding and ability with these skills enhances children's learning of the alphabetic system for use in their reading and writing. Activities are detailed in Table 1. Firm Foundations Literacy Components. Rhyming. Rhyming is organized in the Firm Foundations program in a hierarchy of easiest to harder rhyming activities. The initial activities involve play with songs and poems, while the latter activities encourage additional skills such as recall and generation of novel rhymes. Segmenting and blending. The authors of Firm Foundations included both blending and segmenting activities in a combination of phonological awareness activities (i.e., manipulation of larger sound units such as compound words and syllables) and phonemic awareness (i.e., awareness of the individual sounds of the word). These activities are ordered hierarchically, starting with segmenting and moving to blending activities.  26 Letter-sound mastery. The letter-sound mastery activities in the Firm Foundations program introduce sound and letter activities initially as separate entities until the end of the Fall semester. Connections between sounds and their corresponding letters are not introduced until the Spring and carry on until the end of the year until they are incorporated into beginning writing using letter sound mastery to generate meaningful written communications. Concepts of print. First described by New Zealand educator Marie Clay (1979), concepts about print refers to what emergent readers need to understand about how printed language works and how print represents language. Firms Foundations has incorporated many of the metalinguisitic skills targeted by Clay (1979) that enhance understanding of print. These include enhancing a child's understanding that print carries a message; books are organized, with a cover, title, and author; reading in English flows in a particular and consistent direction, left to right and top to bottom; printed language consists of letters, words, and sentences; and, recognition or matching of upper- and lower-case letters. Concepts about print activities in Firm Foundations use shared reading, enlarged charts and poems, or other kinds of engaging texts. Toward the end of the program, the authors encourage the use of emergent writing activities. The Firm Foundations program prescriptive theory outlines a suggested timeline for teaching these skills based literacy activities. The literacy skills need to be implemented over the course of a 10-month school year. During the Fall semester, it is suggested that teachers engage in teaching and practice in rhyming, segmenting and blending of compound words, and letter sound activities. During the Winter semester, teachers engage in teaching and practice in segmenting and blending syllables, letter-  sound mastery, and concepts of print. A l l previously introduced skills are reviewed during this time. During the Spring semester, teachers teach and provide practice for segmenting and blending phonemes and continue with letter-sound mastery activities. Review of all previously introduced skills continues. Teacher Training and Support Teacher training to use the Firm Foundations program is conducted by the program authors, who are all practicing kindergarten or support teachers. Teachers are provided with a Firm Foundations manual that clearly outlines the suggested teaching timeline for the literacy skills, detailed circle time (i.e., direct instruction) and centre time (i.e., hands on time) activities for each of the skills, and accompanying games and activities that the teachers must construct prior to use. Training may include a half-day or full day session formal session that is followed with 2 -3 informal meetings spread out throughout the school year at a central location for all trained kindergarten teachers to discuss literacy issues. Attendance to the follow-up sessions was not required by the participants in the study that follows; however, the district engaged a literacy mentor who was available to support teachers on a number of literacy issues, one of which was the Firm Foundations program.  Statement of the Problem and Overview of the Study As noted previously, an understanding of the implementation of early preventive literacy programs is important for practical and theoretical reasons. These reasons frame the purposes of this examination. The first purpose was to both describe the factors that influenced the quality of implementation of the early reading intervention in kindergarten  28 and to describe the differences in implementation of the program on a number of dimensions including adherence to the intervention components and amount of exposure to the intervention components children in classrooms received. It was hypothesized that an understanding of these implementation factors will serve to enhance the participating school district's understanding of their literacy initiative and to determine if local conditions and training systems were adequate to support the desired level of implementation. Knowledge regarding the contextual factors that influenced the quality of implementation of this program may also contribute to existing research on the study of the process of implementation. The second purpose of this examination was to relate the quality of implementation to child outcomes. Such an examination allows for the provision of external validity to early literacy interventions in routine practice and further contributes to an understanding of effective programming for children at risk for reading failure. The study was guided by four questions: (1) In what way did the delivery of early literacy instruction vary across classrooms? (2) Were ecological factors at the level of the child, teacher, or school administration related to the delivery of literacy activities? (3) Did the children, both those at risk for reading failure and those not at risk for reading failure, improve in literacy skill acquisition over the school year? (4) Did the amount or type of literacy activity delivered influence child outcomes?  Method  Participants The participants in the present study came from a larger longitudinal study examining literacy outcomes for children in the elementary years in a northern  29 community in British Columbia, Canada. The present implementation study focused on 13 kindergarten classrooms across 10 schools and comprised of 151 students. The majority of the students were Caucasian and spoke English as a first language. There were 20 students who were indigenous (i.e., Aboriginal) Canadians. Children were classified as at risk for reading failure if their performance on the WRAT3 Reading subtest was at or below the 25th percentile and not at risk for reading failure if they scored at or above the 26th percentile (Lesaux & Siegel, 2003). Table 2 outlines the demographic characteristics of the total student sample. A l l teachers in the participating classrooms completed background and belief questionnaires and all principals in the participating schools were interviewed to determine their level of support for the early literacy program in their respective schools.  Measures of Implementation Quality Teacher Backgrounds and Beliefs Teachers independently completed a background and belief questionnaire (complete questionnaire in Appendix I) at the beginning of the study immediately after consenting to participate. The following dimensions were assessed: teacher background (including education and years of experience); understanding and amount of inservice of the Firm Foundations program (Beesely et al., 2001); ratings on perceived capability of implementing the program (including adequate time, difficulty, and support); and beliefs about the benefit of the program. Teachers were also asked about their effectiveness in identifying children at risk for reading failure, the adequacy of additional support within their schools for children identified as at risk, and use of parents to support their own children.  30 Principal Support The procedure outlined by Kam, Greenberg, and Walls (2003) assessed two measures of principal support on a scale from 1 (Not very supportive) to 3 (Very supportive): (1) Quality of principal support for early literacy, and (2) Quality of principal support of teachers conducting the early literacy program. The first rating measured the extent to which principals showed support in general for the early literacy intervention based on information gathered in an interview with each principal. A high rating indicated that a principal saw the early literacy program as central to the mandate of the school, supported staff effectively, spoke positively about the intervention, and was familiar with the early literacy concepts. The second measure rated the extent to which the participating teachers felt supported by the school or principal in conducting the early literacy program in their school. A low score indicated that the teacher did not feel supported or assisted in conducting the early literacy program. A high score indicated that the teacher believed that the principal was a true collaborator and was seen as essential component in helping the early literacy model to be maximally successful in the building. School principals were interviewed in the spring of the school year. The interviews were transcribed and the responses rated against the criteria describing principal support for the early literacy intervention. Teachers were interviewed by one of the trained observers, and the interviews were transcribed and independently rated by this interviewer based on the criteria describing principal support of teachers. Appendix II outlines the Principal Support criteria in detail. As these two measures were highly positively correlated (r = .92), they were combined and treated as an ordinal variable with  31 three levels that corresponded to the levels of principal support (high, medium, low) (Kam et al., 2003).  Observations of Quality and Dosage of Literacy Activity in Classrooms Observations of literacy activities in classrooms were made over a 5-month period between February-June during the school year, using the teacher as the unit of analysis. The observation protocol was developed to measure the unique dimensions of this early literacy program but was informed by time-based methods for recording behaviour originally developed in single subject research (Richards, Taylor, Ramasamy, & Richards, 1999). Several aspects of classroom environment and literacy program quantity and type were observed on a 2-minute momentary interval-rating scheme. At the beginning of each interval, observers noted the presence or absence of literacy activity and the teacher's instructional focus. These instructional foci included literacy activities that were associated with the Firm Foundations program (i.e., rhyming, segmenting and blending, letter sound mastery activity, concepts of print) and other literacy activities that were not associated with the components of the program but frequently occur in kindergarten classrooms (i.e., storytelling, copying, or sight word reading). Other instructional aspects were observed and recorded but are not reported here. The observations occurred for the entire time that the class was in session (average time: 150 minutes). Each classroom was observed at the teachers' convenience 4-6 times, at intervals of approximately 2-3 weeks. Four undergraduate psychology students were trained through videotape and repeated visits to kindergarten classrooms to conduct the observations. Prior to engaging in the collection of observational data for this study, the observers were required to  32 complete an observation test where their responses to 120 discrete observations of videotaped classroom activity were compared to a standard of observation set by the author. The observers obtained 70 - 85% percent agreement on this test. To calculate the interrater reliability, observers were paired and independently rated 85 intervals during a classroom session. Each observer focused on the teacher and classroom activity during the interval and coded presence of a literacy activity (whole class, partial, or no literacy) and teacher instructional focus (e.g., rhyming, segmenting and blending, letter-sound mastery, concepts of print, general literacy, or no literacy activity). Out of 85 intervals most ratings were in agreement (presence or absence of literacy activity kappa coefficient = .96 and teacher instructional focus kappa coefficient = .94). Observer drift was measured at a mid point during the study using an abbreviated test of the videotaped classroom activities. The observers achieved an average of 92% percent agreement on 42 discreet observations that comprised the observer drift test, indicating that their observational accuracy had improved as they gained experience in the classrooms. Classrooms were observed for the entire time the class was in session (average number of minutes: 150) on 4 - 6 occasions over a 16-week period. The number of observations was determined based on recommendations from the research literature (e.g., Allington, 1984; Baker & Zigmond, 1990) and on the availability of resources. The observers sat away from the classroom activity and were encouraged to be as unobtrusive as possible while still being able to hear. Each observer had a clipboard with the observational protocol and observed both a target child and teacher activity (only teacher observation data are reported here) at 2-minute intervals using a prerecorded tape that  33 beeped to alert the observer to begin to record. The entire protocol and coding criteria appears in the Appendix III.  Measures of Student Literacy Progress Several student-reading measures were used to assess students' progress in phonological awareness, sound/symbol association, word reading, and spelling. These measures are described below (see Appendix IV for details). Phonological Awareness Aspects on phonological processing were examined using several subtests from the Phonological Abilities Test (PAT; Muter, Hulme, & Snowling, 1997). The Rhyme Detection subtest was used to assess the ability to detect rhymes. In this task, the children were presented with four pictures and asked to identify which of three words depicted rhymed with the target word. The Syllable Identification and Phoneme Identification subtests were used to assess the students' ability to identify missing segments of words presented. In the Syllable Identification task, the examiner presented a picture (e.g., a table) to the child, said the first part of the word (i.e., "ta"), and asked the child to finish the word (i.e., "ble"). In the Phoneme Identification task, the examiner presented a picture (e.g., a horse), said the first part of the word (i.e., "hor"), and asked the child to finish the word (i.e., "se"). The Phoneme Deletion subtests were used to assess students' ability to segment words by identifying the initial or final phoneme in words presented. For this task, the examiner presented the child with a picture of the word and then asked him or her to delete a phoneme (initial or final) from the word. When the child was asked to delete initial phonemes from the words, the examiner would say, for example, "Seat  34 without Isl says  ," and when the child was asked to delete final phonemes from the  words, the examiner would say, for example, "Seat without ft/ says  ." Following  standardized procedures, examiners presented the tasks and discontinued if the students were unable to produce a correct response after 5 items. See Appendix IV for a copy of these measures. Letter Naming To determine the level of letter knowledge, each child was asked to name lowercase letters from a page of 26 letters presented in a random order (Lesaux & Siegel, 2003). Word Reading  <  A subtest of the Wide Range Achievement Test - Three (WRAT3; Wilkinson, 1993) was used to assess early reading progress. The WRAT3 Reading subtest (blue form) was used. Here, each child was asked to name capital letters and to read some simple words.  Spelling To examine the children's spelling ability in kindergarten, they were asked to print their names and five simple words (e.g., mom, no, cat, dad, I) (Lesaux & Siegel. 2003). A l l literacy measures were individually administered to kindergarten students to determine their level of reading or pre-reading skill prior to the classroom literacy observations. This testing was conducted by a school staff member, (e.g., the learning assistance teacher), or the child's classroom teacher. The results were scored by  35 university research assistants. Children were classified as at risk for reading failure i f their performance on the WRAT3 Reading subtest was at or below the 25th percentile and not at risk for reading failure if they scored at or above the 26th percentile (Lesaux & Siegel, 2003). Classroom teachers received these child level classifications. After all of the classroom literacy observations were completed, a second administration of the literacy measures was conducted. School personnel completed these assessments with some assistance from the trained classroom observers.  Results The results are presented in four areas: (a) description of time-sampled literacy activities and teacher delivery of these activities by classroom; (b) relations between the ecological variables (i.e., proportion of at risk Children in classrooms, teachers background and beliefs, and principal support) and to the level of implementation of Firm Foundations; (c) child outcomes from pre to post test; and (d) the extent to which the literacy activities were related to student outcomes.  Classroom Activities and Settings Each class was observed for an average of 75 intervals (150 minutes) per day 4 6 times over the observation period. Literacy observation results are based on the timesampled ratings that occurred during each 2-minute interval during the observation session in each of the 13 classrooms. Each observation session was summarized to derive a percentage of the total amount of time spent on literacy in each of the observation categories: total amount of literacy (whole group or partial group) and Firm Foundations literacy activities (i.e., rhyming, segmenting and blending, letter-sound mastery, and  36 concepts of print). A complete list of categories and codes is summarized in Appendix III. Whole group and partial group literacy activities were summed to create a category of "Total Time Spent on Literacy" that represented the total percentage of class time children had the opportunity to engage in literacy activities per observation. Observations were aggregated to derive single scores of percentage of the total time spent on each of whole class, partial class, and total time spent on literacy for each classroom (see Table 3) Similarly, calculations were computed for individual components of literacy activities that were part of the Firm Foundations program (see Table 4). Analysis of the implementation observational data reveals considerable variability among the sample of 13 classrooms in the extent to which the teachers engaged in literacy activities and the amount of time that they devoted to each of those literacy activities. Table 3 displays the percentage of time teachers in classrooms spent on literacy activities in descending order of total amount. The mean total percentage of time spent on literacy activities by classroom ranged from 29% to 66%. Given that classrooms were in session for 150-164 minutes per day, this equal approximately to 43 - 99 minutes per day. Whole class literacy activities, in which all children were engaged in a teacher directed literacy activity, ranged from 14% to 52% of the intervals observed, or between 2 1 - 7 5 minutes per day, depending on the classroom. Children had additional opportunities to engage in a literacy activity (partial class literacy) typically during a centre or tabletop activity during 11% to 26% of the intervals observed across the classrooms. The percentage of intervals spent on literacy varied tremendously depending on the classroom. Some classrooms, such as number 10, had standard deviations of  37 almost half the mean, indicating that the time spent on literacy varied considerably from observation to observation. Table 4 presents the descriptive results of the observed percentage of time spent on the Firm Foundations literacy components by classroom. This table displays the results by classroom in the same descending order introduced in Table 3. What is apparent from this ordering is that those teachers, who spent more time on literacy activities in general also spent more time on Firm Foundations components although this does not represent a one to one correspondence. The amount of time spent on.literacy components varied within classrooms, as indicated by the standard deviations that sometimes exceed the mean, as well as across classrooms, where some aspects of the literacy components were stressed more than others. For example, some teachers spent up to 6% of their time on Rhyming activities while other teachers demonstrated no observed instances of Rhyming. Time spent on Letter-sound Mastery represented the greatest amount of observed intervals followed by Concepts of Print. Each teacher spent at least a small percentage of time on Segmenting and Blending activities; however, some teachers spent considerably more time on this literacy activity than others. Recall from the introduction that the authors of Firm Foundations suggest that Segmenting and Blending, Letter-sound Mastery, and Concepts of Print should all be taught and practiced during the spring of the school year, the time period that this observation occurred. The total percentage of time spent on Firm Foundations program components as a group correlated positively with total percentage of time spent on literacy activities in classrooms in general (r = .797, p < .01), indicating that it was in those classrooms where a high amount of general literacy occurred that Firm Foundations was used. Occurrence  38 of individual components of the Firm Foundations program were not as directly associated with classrooms that spent a lot time on literacy activities. Concepts of Print and Rhyming were positively associated with total percentage of time spent on literacy by classroom (r = .579,p = .038 and r = .655, p = .015, respectively) but neither of the other two components, Segmenting and Blending and Letter-sound Mastery, were significantly correlated with total time spent on literacy by classroom (r = .434, p = .139, and r = .485,p = .093, respectively). This finding indicates that only Segmenting and Blending and Letter-Sound Mastery were unique to the Firm Foundations program.  Relationships Among Ecological Variables Table 5 presents the correlations, means, and standard deviations of the classroom factors, teacher background and belief, principal support, and classroom observations. Attention to the means and standard deviations reveal, based on information obtained on the pretest, that 27.2% of the students were identified as at risk for reading disability at the beginning of the study. By classroom, there was a considerable range in the number of students identified as at risk (Range 0 - 62.5%). The data from the teacher questionnaires revealed that, on average, teachers in the study had a mean of 14 years of teaching experience and most of them held the minimum criteria for teacher certification in the region. On average, the teachers had been using the Firm Foundations program for one year prior to the evaluation and they reported a "high understanding" and belief in the Firm Foundations program (M= 4.15, SD = .80 a n d M = 4.38, SD = .76, respectively), despite that fact that the mean number of hours of inservice was only 2.23 hours per teacher. Teachers reported that, on average, they used the program at least 3 times per week and reported that they found it "somewhat difficult" to incorporate but  39 felt they had "somewhat adequate time" to do so. They reported that they perceived themselves as very effective at identifying children at risk for reading failure and were able to seek out or provide "adequate amounts" of support for students identified as at risk. Very few of the teachers reported using or engaging parents in providing support for those children identified as at risk. ( M = 1.00; "very infrequently"). Principal support ratings in the 13 schools ranged from 1.00 (not supportive at all) to 3.00 (very supportive) with the mean rating 2.23 (supportive). This indicates that most principals saw the early literacy program as central to the mandate of their school, supported staff effectively, and spoke positively about the intervention, but were not familiar with the early literacy concepts of the program. Zero-order correlations of ecological factors associated with the delivery of the literacy program are also presented in Table 4. Correlations between the percentage of children at risk in each classroom and teacher.expressed effectiveness at identifying children at risk were significantly negatively correlated (r = -.69, p = .01), indicating that in classrooms where there were more children identified as at risk, teachers felt less certain about their ability to identify them or, alternatively, in those classrooms where there were less children at risk, teachers felt more confident about their ability to identify them. Engagement of parents in classrooms was also negatively correlated with percentage of children at risk in each classroom (r = -.54, p = .05) indicating that teachers in classrooms with higher percentages of children at risk reported that they engaged the parents less than in other classrooms. Percentage of time spent on the Firm Foundations components in each classroom was also significantly negatively correlated with total percentage of children identified as at risk by classroom (r = -.84,/? = .01) indicating that  40 where there were more children identified as at risk for reading failure in classrooms, there were fewer literacy activities occurring. Correlations between teacher reported frequency of use of Firm Foundations were significant with teacher reported understanding of Firm Foundations, number of inservice hours in Firm Foundations, and adequate time to incorporate Firm Foundations into daily lessons (p < .05 orp < .01). Number of years practicing Firm Foundations was correlated with with frequency of use of the program and the number of inservice hours each teacher reported receiving (r = .64, p < .05 and r = .69, p < .01, respectively). Level of belief in Firm Foundations concepts and teaching methods was negatively correlated with level of difficulty incorporating Firm Foundations (r = -.75, p < .01), meaning that those who had high belief in the program felt that it was easy to incorporate whereas those who felt it was difficult had low belief in the program. A correspondingly significant positive correlation was found between belief in the program and adequate amounts of time to incorporate it into the kindergarten program (r - .60, p < .05). These results indicate that teacher belief and efficacy are significant factors in the acceptance and use of this program. Correlations of teacher reported frequency of additional support in the school for children identified as "at risk" was positively correlated with frequency of use of Firm Foundations, number of years practicing Firm Foundations, and adequate time to incorporate Firm Foundations in the general curriculum (p < .05 orp < .01). There was a negative correlation between use of parents and level of belief in the program. Recall that teachers reported infrequent use of parents to provide support for children at risk. This could indicate that those who had high belief in the program had low use of parents and,  41 alternatively, those that low belief in the program had higher use of parents. The former explanation is probably more accurate as there was generally a high belief in the program expressed and an overall low use of parents to support children at risk. Level of principal support correlated with teacher reported understanding of Firm Foundations and number of years practicing Firm Foundations (p < .05). Teachers in schools with higher levels of principal support reported higher understanding and more years practicing Firm Foundations. Additionally, principal support and teacher reported effectiveness of identifying children at risk for reading failure were positively correlated (p < .05). Of greatest significance for this evaluation, level of principal support positively correlated with observed percentage of time spent on Firm Foundations activities in classrooms (r = .62, p < .05). Teacher reported effectiveness of identifying children at risk for reading failure positively correlated with both observed amount of time spent on literacy activity and observed percent of time spent of time spent on Firm Foundation activities (p < .01) implying that those teachers who spent more time on literacy activities felt more able to identify children at risk than those who did not spend as much time on literacy activities. Taken together, these results indicate that teacher efficacy, classroom composition, and principal support have significant relationships with Firm Foundations program use.  Child Outcomes One of the central questions in this evaluation study was the extent to which the literacy activities had an influence on children's phonological awareness, letter identification, word decoding, and spelling skills. Attrition from pre to post-test was 23 students (13%), due to student moves or absences that prevented the collection of the  42 post-test. Preliminary analyses (t-tests) were conducted to determine whether students with complete data differed on any study variables from the attrition students. No significant differences were found on pretest scores for those children who remained in the study and those children who did not participate in the post-test. Table 6 presents means and standard deviations, at pretest and post-test, for at risk and not at risk students, along with ts for each of the literacy measures. Only those students who had completed both pre and post-test on each of the measures are reported. The sample size for each measure is recorded in parentheses beside each measure. It should be noted that slightly more two thirds of the children completed the phoneme deletion measure at post test. On all measures, distributions for the change score values were normal, with skewness values in the range from 0.00 to 0.28. The table shows that there was an increase over time on all literacy measures for all children, including those who were and were not identified as at risk for reading failure. A l l changes were significant with the exception of the WRAT3-Reading (Wilkenson, 1993) for those children not at risk for reading failure. However, it should be noted that the mean pretest score for these students was well within the normal range for their age level. Statistical Model for Linking Child Outcomes and Implementation of Firm Foundations For the child outcome measures and observation of implementation data, a twolevel classroom effects model (i.e., also known as a hierarchical linear models and random coefficients model) to account for the two levels of data - student change scores (Level 1) within classrooms (Level 2) was used. The intention was to examine the behaviour of Level-1 outcomes (change scores on each of the literacy measures) as a function of Level-2 (classroom) predictors (e.g., levels of implementation of literacy  43 activities) to account for potential differences in student literacy change on outcome measures that resulted from the nesting of students within classes taught by different teachers (Singer, 1998). The rationale for using the two-level classroom effects procedure was that if classrooms exerted a strong effect on difference scores (i.e., high intraclass correlations), individual-level analysis such as is possible in a general linear model might overestimate the statistical significance of the intervention effect, whereas the classroom effects model would take the clustering into account to yield a higher level of statistical confidence. Additionally, it should be noted that the one group pre-post test design of this study has the internal threat of maturation (Shadish, Cook, & Campbell, 2002) and level of implementation of the literacy components allows low implementation classrooms to act as an internal comparison to the high implementation classrooms. This creates an internal comparison to classroom as a group rather than an external comparison group that has been argued by some (Aiken et al., 1998;'Heinsman & Shadish, 1996; Shadish & Ragsdale, 1998) as potentially yielding more accurate results. The logic for this argument is that since all of the children in this district were experiencing the literacy program, an internal comparison group may have more in common with the other children in the district (except level of implementation) than a comparison group drawn from a patently different school district. Given the constraints of the original design of this study, the internal comparison group of implementation level acts as a useful comparison to measure the effectiveness of the literacy program. A multilevel model was conducted for each child outcome change score using SPSS 11.5 software package with SPSS code provided by U C L A Computing Services (2003). This code was originally introduced as S A S " P R O C - M I X E D " by Singer (1998).  44 This analysis began by fitting the multiple unconditional means models, examining variation in each of the computed change scores for each literacy measure to determine i f the data structure was hierarchical for any of the measures. The output generated from this model was evaluated to determine the appropriate second step. The unconditional means model expresses the student-level outcome Yjj using a pair of linked models: one at the student level (Level 1) and another at the classroom level (Level 2). At level 1, the student's outcome was expressed as the sum of the intercept for the student's classroom (PQJ) and a random error (r^) associated with the i student in the j  t h  classroom:  Y , = ( V r,, 1  (la)  At level 2 (the classroom), we express the classroom level intercepts as the sum of the overall mean of the literacy change score (yoo) and a series of random deviations from that mean (UOJ): Poj = Yoo + u j + ry 0  (lb)  Substituting (lb) into (la) yields the multilevel unconditional means model specified below: Yy = yoo [literacy change scores] + UOJ [classroom] + r,j  (lc)  The unconditional means model (lc) for each of the literacy change scores was significantly different from zero (p values ranged from .01 to <.0001). The intraclass correlations for the unconditional means model, which represents the amount of clustering of change scores for students within classrooms, ranged from .03 - .24. Although the values below .10 are considered small (Henderson, 1999) and might be dismissed as nonsignificant, the multilevel models were felt to be an improvement over  t h  45 traditional methods, such as regression and A N O V A , as this research was concerned with estimating random effects (i.e., specifically of classroom) and a decision was made to proceed with the conditional means models to test Level 2 predictors. According to MacKinnon & Lockwood (2003), even when intraclass correlations appear small, they can seriously increase Type 1 errors. B y proceeding with the conditional means models, we are able to examine the unique estimates for each classroom more efficiently than by conducting separate regression equations for each context (Henderson, 1997). The conditional means model following Singer (1998) was developed to test Level 2 predictors. The Level 2 predictors tested were the effect of risk status (coded as 0 and 1) and level of implementation. Following the recommendation of Kam, Greenberg & Walls (2003) in a study of level of implementation and student outcomes, level of implementation was categorized as low and high, with low representing those classrooms that were below the 25 percentile for percentage of time spent on the literacy th  components of the Firm Foundations program and high representing those classrooms that were above the 25 percentile. The low implementation schools spent an average of th  16 minutes per day on the program components and the high implementation schools spent an average of 42 minutes per day on the program components. Analysis included an n of 13 classrooms. The models tested are expressed below:  Yy = YOO [Literacy Change Score] + yoi [Level of Implementation]-!- UQJ [Classroom] + ry (2a)  46 = yon [Literacy Change Score] + yoi [Literacy Level of Implementation] + yoi [Risk Status] + yoi [Level of Implementation x Risk Status] + UQJ [Classroom] + ry (2b) Level of Implementation  .  r  .  -  The first model (2a), the Level 2 conditional means model, was run for each of the literacy change scores using Level of Implementation as the only fixed effect and was significant for the child literacy change scores of Spelling, F(\, 12) 8.39, p = .014. The slope coefficient for Level of Implementation was statistically significant, t (12) = 2.90, p = :014; the coefficient was 1.17, with a standard error of .40. These findings indicate that those students with high implementation had significantly different outcomes than those who received a low level of implementation. Following the methods outlined by Singer (1988) for interpreting the coefficient estimate, the coefficient estimate of 1.17 for Level of Implementation indicates that students who experienced high implementation differ by 1.17 points on the Spelling from those who received low implementation. Cohen's d was 1.69, considered a large effect size, as computed from the T ratio and degrees of freedom (27;/V^(CDRG, 1998). The second model (2b), where both Level of Implementation and Risk Status were used as fixed effects, was significant for the following student literacy change scores: WRAT3 (Wilkinson, 1993), Woodcock Johnson Letter Identification (Woodcock, 1988), Phonological Awareness Test (PAT) - Syllable Identification subtest fMuter, Hulme, & Snowling, 1997), and Phonological Awareness Test (PAT) - Phoneme deletion subtest (Muter, Hulme, & Snowling, 1997). Each of these effects will be discussed separately.  47 There was a significant interaction effect of Level of Implementation x Risk Status for WRAT3 - Reading subtest (Wilkinson, 1993), F ( l , 101) 11.32,p = .001. The slope coefficient for the Level of Implementation x Risk Status interaction was significant, t (101) = 3.36,/? = .001; the coefficient was 18.54, with a standard error of 5.51. Subsequent analysis to understand the simple main effects of this interaction was conducted by running each Level 2 predictor alone as a fixed effect after selecting one group of the other fixed effect (e.g., Level of Implementation after selecting only those children at risk for reading failure; then, Level of Implementation after selecting only those children not at risk for reading failure). These analyses revealed a significant effect for Risk Status when both the high implementation students were selected, F (1, 100) 78.31,/? < .0001 and low levels of implementation were selected, F (1, 45) 9.24, p = .004. This finding indicates that students at risk in both levels of implementation had significantly different scores than those who were not at risk; however, i f one examines the results of the Level of Implementation as a fixed effect selecting only those students at risk, approaching significance, F ( l , 5) 4.29,/? = .09, was found, whereas in the model selecting students not at risk, the fixed effect to test the effects of Level of Implementation was not significant, F (1, 12) .937, p = .35. Thus, high and low implementation of the intervention appears to have had an effect for students at risk; however, although tenuous, an effect approaching significance was found for all students in the high implementation classrooms. The effect of Level of Implementation x Risk Status was significant for WJLetter Identification (Wilkinson, 1993), F(l, 121) 11.05,/? = .001. The slope coefficient for Level of Implementation x Risk Status was statistically significant, t (121) = 3.33,/? =  48 .001; the coefficient was 21.15, with a standard error of 6.36. Subsequent analysis to understand the simple main effects of this interaction revealed a significant effect for Risk Status when only high implementation classroom students were selected, F (1, 80) 13.89,/? < .0001, whereas a non-significant effect was found when only students in low implementation classrooms were selected, F ( l , 4 5 ) 2.53,/? = .133. These findings indicate that students at risk for reading failure in classrooms with high implementation had significantly different outcomes than those who received a low level of implementation. Following the methods outlined by Singer (1988) for interpreting the coefficient estimate, the coefficient estimate of 21.15 for Level of Implementation x Risk Status, indicates that students in classrooms at risk for reading failure who experienced high implementation differ by 21.15 percentile points on the WJ- Letter Identification from those at Risk Students who receive low implementation. Cohen's d was .60, as computed from the T ratio and degrees of freedom (27 Ndf) (CDRG, 1998). The effect of Level of Implementation x Risk Status was significant for PAT Syllable Identification subtest,(Muter, Hulme, & Snowling,. 1997), F ( l , 145) 5.57,/? = .02. The slope coefficient for Level of Implementation x Risk Status was statistically significant, t (145) = 2.36,/? = .02; the coefficient for was 2.16, with a standard error of .92. Subsequent analysis to understand the simple main effects of this interaction revealed a significant effect for Risk Status when only high implementation classroom students were selected, F (1, 99) 10.71,/? = .001, whereas a non-significant effect was found when only students in low implementation classrooms were selected, F (1, 45) .24,/? = .629. These findings again indicate that students at risk for reading failure students in classrooms with high implementation had significantly different outcomes than those  49 who received a low level of implementation. Following the methods outlined by Singer (1988) for interpreting the coefficient estimate, students in classrooms at risk for reading failure who experienced high implementation differ by 2.16 points on the P A T Syllable Identification from those at Risk Students who receive low implementation. Cohen's d was .39, as computed from the T ratio and degrees of freedom (2T Ndf) (CDRG, 1998). Literacy Components Subsequent models examining the Level 2 predictors of each component of the Firm Foundations program, (a) rhyming, (b) segmenting and blending, (c) letter-sound mastery, and (d) concepts of print, were treated as fixed effects with Risk Status for each of the literacy change scores within the conditional means model. Additionally, general literacy activities (i.e., those not part of the Firm Foundations program, such as general book reading), was treated as a fixed effect with Risk Status for each of the literacy change scores within the conditional means model. There were significant effects found for only two of the Firm Foundations components, Segmenting and Blending and LetterSound Mastery. Each of these effects will be discussed below. The effect was significant for Risk Status alone and Level of Segmenting and Blending Implementation x Risk Status for the PAT-Syllable  Identification, F (1, 136) =  3.69,p = .05 and F (1, 140) = 6.78,/? = .01, respectively. The slope coefficient for Risk Status was statistically significant, t (136) = -1.92,p = .05; the coefficient was -3.26, with a standard error of 1.69. This implies that those students who were at risk had a negative slope pattern on the PAT - Syllable Identification. The slope coefficient for Level of Segmenting and Blending x Risk Status was statistically significant, t (140) = 2.60, p = .02); the coefficient was 2.49, with a standard error of .96. Subsequent analysis  50 of this model to understand the simple main effects of the interaction revealed a significant effect for Risk Status when only high segmenting and blending implementation classroom students were selected, F(l, 98) 10.32,p = .002; whereas a non-significant effect was found when only students in low implementation classrooms were selected, F(l, 4.7) 1.73,/?.= .19. These findings again indicate that students at risk for reading failure who were in classrooms with high segmenting and blending implementation had significantly different outcomes than those who received a low level of implementation. Following the methods outlined by Singer (1988) for interpreting the coefficient estimate, of the Level of Implementation of Segmenting and Blending x Risk Status interaction, when at risk students receive a high level of implementation of segmenting and blending they differ 2.49 points from those at risk students who receive a low level of implementation. Cohen's d was .44, as computed from the T ratio and degrees of freedom (27/ Ndf) (CDRG, 1998). A l l other effects for Segmenting and Blending Level of Implementation were non significant.  1  Given the small sample size of this study using a multi-level model, it is worth noting that a similar close to significant effect of segmenting and blending level of implementation x risk status was found with the PAT- Phoneme Identification change score, F (1, 116) = 3.12,/? = .08. The slope coefficient for Level of Implementation Segmenting and Blending x Risk Status was, t (116) = 1.76, p = .08; the coefficient was 2.23, with a standard error of 1.26. Subsequent analysis of this model to understand the simple main effects revealed a significant effect for Risk Status when only high segmenting and blending implementation classroom students were selected, F (1, 95) 3.94, p = .050; whereas a non-significant effect was found when only students in low implementation classrooms were selected, F (1, 32) .81, p = .38. These findings again indicate that students at risk for reading failure that were in classrooms with high segmenting and blending implementation had different outcomes on PAT- Phoneme Identification than those who received a low level of implementation. Following the methods outlined by Singer (1988) for interpreting the coefficient estimate, of the Level of Implementation of Segmenting and Blending x Risk Status interaction, when at risk students receive a high level of implementation of segmenting and blending they differ 2.23 points on PAT-Phoneme Identification from those at risk students who receive a 1  51 The effect was significant for Level of Letter-Sound Mastery Implementation x Risk Status for the WJ- Letter Identification  (Woodcock, 1983), F (1, 123) = 6.25, p =  .014. The slope coefficient for Level of Letter-Sound Mastery x Risk Status was statistically significant, t (123) = 2.50p = .014; the coefficient was 17.15, with a standard error of 6.86. Subsequent analysis of this model to understand the main effects revealed a significant effect for Risk Status when only high letter-sound mastery implementation classroom students were selected, F ( l , 32) 8.42,p = .007; whereas a non-significant effect was found when only students in low implementation classrooms were selected, F (1, 87) .001,/) = .971. These findings indicate that students at risk for reading failure who were in classrooms with high letter-sound mastery implementation had significantly different outcomes on the WJ - Letter Identification  than those who received a low level  of implementation. Following the methods outlined by Singer (1988) for interpreting the coefficient estimate, of the Level of Implementation of Letter-Sound Mastery x Risk Status interaction, when at risk students receive a high level of implementation of lettersound mastery differ 17.15 percentile points on the WJ-Letter  Identification  subtest  from those at risk students who receive a low level of implementation. Cohen's d was .45, as computed from the T ratio and degrees of freedom (2T Ndf) (CDRG, 1998). The effect was significant for Level of Letter-Sound Mastery Implementation x Risk Status for the PAT - Syllable Identification  subtest (Muter, Hulme, & Snowling,  1997), F(l, 145) = 4.01, p = .047. The slope coefficient for Level of Letter-Sound Mastery x Risk Status was statistically significant, t (145) = 2.00 p = .047; the coefficient  low level of implementation. Cohen's d was .32, as computed from the T ratio and degrees of freedom (2T lyfdf) (CDRG, 1998).  52 was 1.79, with a standard error of .89. Subsequent analysis of this model to understand the main effects revealed a significant effect for Risk Status when only high letter-sound mastery implementation classroom students were selected, F ( l , 52) 6.95,p = .01; whereas a non-significant effect was found when only students in low implementation classrooms were selected, F (1, 92) .14, p = .71. These findings indicate that students at risk for reading failure who were in classrooms with high letter-sound mastery implementation had significantly different outcomes on the PAT - Syllable Identification than those who received a low level of implementation. Following the methods outlined by Singer (1988) for interpreting the coefficient estimate, of the Level of Implementation of Letter-Sound Mastery x Risk Status interaction, when at risk students receive a high level of implementation of letter-sound mastery differ 1.79 percentile points on the PATSyllable Identification subtest from those at risk students who receive a low level of implementation. Cohen's d was .33, as computed from the T ratio and degrees of freedom (2TNdfi (CDRG, 1998).  Instructional Methods of the Firm Foundations Program Finally, to test the effects of the instructional methods of direct instruction and "hands-on activity coaching" models examining the Level 2 predictors of Direct Instruction and Hands-on Activity were treated as fixed effects with Risk Status for each of the literacy change scores within the conditional means model.  Yy = yoo [Literacy Change Score] + yoi [Instructional Method] + yoi [Risk Status] + yoi [Instructional Method x Risk Status] + UQJ [Classroom] + ry (2b)  53  Although it would have been more theoretically interesting to include Level of Implementation as a Level 2 predictor within this model, there was insufficient statistical power, given the small sample size, to include an additional predictive variable into the model. Thus, using only those change scores where significant effects of the intervention were found, the fixed effects of Risk Status and Instructional Method were included in the models. There were significant effects found for both of the Firm Foundations instructional methods, Direct Instruction and Hands-on Activities. Each of these effects will be discussed below. The effect was significant for Direct Instruction x Risk Status for the WRAT3 Reading subtest (Wilkinson, 1993), F(l, 128) = 5.12,p = .02. The slope coefficient was statistically significant, t (128) = 2.30,/? = .02; the coefficient was .64, with a standard error of .28. This implies that those students who were at risk who received more direct instruction opportunities had a positive slope patterns on the WRAT3 - Reading and had significantly different outcomes than those who received less direct instruction. Following the methods outlined by Singer (1988) for interpreting the coefficient estimate, when at risk students receive more direct instruction they differ .64 points from those at risk students who receive less direct instruction. Cohen's d was .41, as computed from the T ratio and degrees of freedom (ITNdJ) (CDRG, 1998). This was the only significant effect for direct instruction as a predictor for change scores. The effect was significant for Hands-on Activities for all students for the Spelling change score, F (1, 14) = 4.28,/? = .05. The slope coefficient was statistically significant, t (14) = -2.07,p = .05; the coefficient was -.04, with a standard error of .02. This implies  54 that in those classrooms where more time was spent on Hands-on Activities those students had a negative slope pattern on Spelling. Cohen's d was 1.13, as computed from the T ratio and degrees of freedom (2TNdf) (CDRG, 1998). This was the only significant effect for Hands-on Activities as a predictor of change scores.  •  •••  ,  Discussion  The purpose of this research was to investigate the effectiveness of the early reading program, Firm Foundations, implemented in kindergarten as part of a school district wide literacy initiative. This project was conducted in a school district where 27% of the kindergarten students were identified as at risk for reading failure in the early part of their first year of formal schooling. This number of children at risk for reading failure was reduced to 12% by the end of the school year. The results provide evidence of the effectiveness early literacy programs that include both phoneme awareness and lettersound mastery components in routine practice conditions for children at risk of reading failure. Large significant effects of the intervention were found for all children on the Spelling outcome measure (Cohen's d = 1.39) only in those classrooms where the amount of the program delivered was high. There were significant effects for children at risk for reading failure who received high implementation of the program on most of the literacy measures. These effects ranged from d = 0.32 - 0.66, Cohen's (1988) definition of a medium effect. These effect sizes are comparable to those reported by the National Reading Panel (2001) synthesis of 52 group design treatment comparison studies in which they found the overall effect size for teaching phonemic segmenting and blending skills, 0.53 for reading outcomes, and .59 for spelling outcomes. Specific components of the Firm Foundations program, namely segmenting and blending and letter-sound  55 mastery, were associated with greater gains on the literacy outcome measures suggesting that these two core activities account for the effectiveness of this program for children at risk for reading failure. The results are consistent with the quasi-experimental studies by Lundberg et al. (1988) and Schneider et al. (1997), as well as the meta-analysis by the National Reading Panel (2000), where instruction in segmenting and blending activities were found to improve children's phonemic awareness skills. The results are also consistent with research suggesting that the effectiveness of phonemic awareness activities is enhanced when combined with letter-sound instruction (Bus & Ijzendoorn, 1999; Byrne & Fielding-Barnsley, 1989, 1993; Juel et al., 1986; McGuinness et a., 1995; Schneider et al., 2000; Tunmer & Nesdale, 1985). The importance of the present study is that it extends the work of experimental demonstrations of the effectiveness instruction in phonemic awareness and letter-sound mastery in preventing reading failure in children to routine practice conditions. Whereas experimental demonstrations are conducted under well-controlled conditions and have high internal validity, they often have low external validity, and it is difficult to determine how effective they will be .under typical conditions. The strength of present study is that it provides an example of the utilization of phonemic awareness and letter-sound mastery instruction under "normal" or routine practice school conditions. The Firm Foundations reading intervention is build upon the theoretical premise that young children learn from a balance of child directed, "hands-on," experiences and explicit teacher directed instruction. This implementation evaluation of Firm Foundations revealed that this group of teachers, in the second year of their implementation of the program, relied on direct instruction, typically in circle time activities, to deliver the  56 instructional components of the program. The correlation between high implementation of the program and direct instruction was significant (r = .76); whereas, high implementation of the program and "hands-on" activities were negatively correlated (r = -.56). Additionally, greater direct instruction was associated with more growth in reading skill for at risk students, and more time spent on "hands-on" coaching was negatively associated with growth in spelling ability for all of the students. One could interpret from these results that the direct instruction activities were more positively associated with student success on literacy outcomes, especially for the children at risk for reading failure. Juel & Minden-Cupp (2003), in a study of four grade one classrooms, found some differential effects for literacy component foci and methods used to instruct students who were at risk and to instruct students that were not at risk. Students who were at risk who were in classrooms with more direct instruction that focused on phonemic awareness and letter-sound knowledge performed better on literacy outcomes after a year. However, Juel & Minden-Cupp found some preliminary evidence that, for those children who were not at risk who had experienced more child-directed general literature activities in classrooms, more growth in reading outcomes occurred. The results of the present study, if interpreted cautiously, confirm Juel & Minden-Cupp's finding regarding the benefits of direct instruction for at risk students but do not confirm the findings regarding reading instruction for those not at risk. In fact, in the present study students who were not at risk appeared to benefit from explicit, skills-based literacy instruction, specifically in the area of spelling. However, it needs to be noted that this implementation of Firm Foundations possibly represents an unbalanced implementation of the program that favored the direct instructional methods. The teachers in this study were in the early stages of utilizing this  57 program and many were teaching these instructional components for the first time. It is the opinion of this researcher that given a more balanced implementation of the literacy components, utilizing both the direct instruction and "hands on" activity instructional methods equally, somewhat different findings about both the effectiveness and the differential effects of the program for at risk and not at risk learners in improving literacy skills may have been found. Additionally, the implications that one might be able to draw from a balanced presentation of both instructional methods to students with differing literacy skill levels might have emerged. Further observation of the effects of the "handson" activities as an instructional method with kindergarten children and the differential effects this method has on the literacy development of both at risk and non at risk students is warranted to more completely understand the impact of causative and prescriptive theory of this program. This study serves as a reminder of the critical importance of understanding the many ecological factors that influence the degree to which curricular innovations are implemented as intended. Observations of the delivery of the early literacy program revealed a wide degree of variability in implementation in the classrooms sampled. The questionnaire and interview data indicated that a range of factors were associated with the level of implementation of the reading program in each classroom. Principal support, teacher belief and understanding of the program, and proportion of children at risk for reading failure were all associated with higher levels of implementation of the program. Each of these factors will be discussed briefly.  • '•  First, principal support for the literacy program was associated with teacher understanding and years of use of the program and, most significantly, with higher levels  58 of observed implementation of the literacy components by teachers in classrooms. This means that in those buildings were the principals understood the instructional components of the literacy program and supported the program positively, teachers were observed to spend more time on the program components and engage in more literacy activities in their classrooms, generally. The crucial role of the principal to the success of school based preventive efforts is evident in the literature on leadership in educational research (Waters, Marzano, & McNulty, 2003). In a meta-analysis of research on the effect of principal leadership on student achievement, Waters, et al. found that significant, positive correlation exists between effective school leadership and student achievement. Key leadership responsibilities that were significantly correlated with higher student achievement included not only knowing what to do, but when, how, and why to do it. In the present study, in those schools where the principal saw the literacy program as central to the schools' mandate, supported staff effectively, spoke positively about the program, and, most importantly was familiar with the instructional components of the literacy program, higher implementation of the program occurred which resulted in more positive outcomes for children generally and, specifically, for those at risk for reading failure. These findings are consistent with other reports (Gottfredson & Gottfredson, 2002; Kam, Greenberg, & Walls, 2003) that have found that the degree of principal support influences the quality of implementation of prevention programs. However, questions remain regarding the ways in which the principals' understanding and support of the literacy program influenced the teacher implementation. It is hypothesized that positive relationships at the school level are needed to build a sense of professional  59 community necessary to create the conditions where common values are held and are consistently applied instructionally. This hypothesis is supported by results in the present study where higher principal support was strongly related to teachers who had used the program longer and had stronger belief in the program. According to Graczyk, Domtrovich, & Zins (2003), who have written extensively about improving prevention promotion efforts in schools, "Principals, teachers, and staff need a strong foundation of goodwill, respect, and collaboration in order to meet the challenges of implementing a new program, especially if the intervention includes multiple integrated components. They need to share common goals, communicate openly, exchange ideas, and actively problem-solve with one another. A l l school personnel should be aware of a new program and understand any implications i f may have for their role in the school. A positive school environment allows staff to take risks, support one another, learn from mistakes, and. grow professionally" (p. 312). This strong foundation of collaboration is exemplified by what Waters, Marzano, & McNulty (2003) refer to as the essence of balanced leadership — knowing not only which school changes are most likely to improve student achievement, but also understanding staff and community members' dispositions to change and tailoring leadership practices accordingly. The second factor that influenced the implementation of the program was teacher belief and understanding of the instructional components and instructional methods of the program. The teachers that participated in this study and in the district wide literacy  60 initiative were positively disposed to the theoretical underpinnings of the intervention employed by the district and yet, despite their enthusiasm, their ability to implement the skills based literacy components of the program appear to have been influenced by other factors. When sharing these results with the teachers, they recognized that more careful attention to particular components of the program, especially aspects related to phonemic awareness, preparing the materials to facilitate the "hands-on" components, and engaging parents in the program, would have improved their level of implementation. Several studies have indicated that teachers are not adequately prepared to teach reading and spelling to young children (e.g., Bos, Mather, Friedman-Narr, & Babur, 1999; McCutchen & Berninger, 1999). Teachers who have been practicing professionally for several years typically rely on inservice education to learn how to master the instructional techniques of phonemic awareness (Mather, Bos, & Babur, 2001). By monitoring the amount of delivery of each component of the Firm Foundations program, it was clear that teachers in this study provided less of one of the most essential instructional components of the program, segmenting and blending. When provided with feedback about this trend, the teachers remarked that they welcomed specific feedback regarding implementation because, for some, phonemic-awareness and letter-sound mastery were new instructional foci in kindergarten classrooms. In this study, the teachers revealed that many of them had only received an average of three hours of inservice in the program. It is likely that additional inservice, preparation time to create the "hands-on" activities, and on-line support would have enhanced both the delivery of each of the instructional components and methods.  61 Finally, the classroom composition, or proportion of children at risk, was related to the level of implementation of the literacy program. The most disconcerting finding of the study was that, in those classrooms where there were proportionately more children at risk for reading failure, lower levels of implementation of the instructional components of the program were observed. These lower levels occurred despite high teacher belief in the program and comparable inservice across the teachers in the study. This finding suggests that ongoing instructional support and mentoring would benefit those teachers who express commitment to implementing explicit literacy instruction that includes phonological and letter-sound awareness activities. More explicit and supportive instruction demands a higher level of training and skill for teachers than is usually provided in either teacher education or continuing education programs at present (Moats, 1994). Additionally, supports for teachers to provide more intensive instruction or smaller class sizes need to be in place in those buildings where there is a higher incidence of children at risk for reading failure. The requirement for more intensive instruction for at-risk children must involve a reallocation of resources to make more teacher time available for preventive instruction and, in many cases, will probably require entirely new resources to adequately meet the instructional needs of all children who are at risk for reading failure. Strong administrative leadership and school district commitment for curricular reform must support teacher efforts to engage in preventive interventions for at risk readers. Positive principal support in the way of instructional leadership indicated that teachers benefit and are influenced by forces beyond their classroom doors and that systems of support are required to enhance the dissemination and sustainability of  62 curricular innovations if we are to promote positive student outcomes, especially for those identified as at risk. Limitations Like many studies, this study must be interpreted with caution due to the unique sample of classrooms that served as the unit of analysis. Oddly, both a strength and a limitation of this study was the use of the multi-level models used to analyze the implementation and child literacy outcome data. On the one hand, the two-level classroom effects model accounted for the potential differences in student literacy change on outcome measures that resulted from the nesting of students within classes taught by different teachers (Singer, 1998). The rationale for using the two-level classroom effects procedure was that i f classrooms exerted a strong effect on difference scores (i.e., high intraclass correlations), individual-level analysis such as is possible in a general linear model might overestimate the statistical significance of the intervention effect, whereas the classroom effects model would take the clustering into account to yield a higher level of statistical confidence. On the other hand, although multi-level models provide more accurate estimates of particular samples, these estimates are less generalizable to other samples that do not closely match the studied sample (Singer, 1998). Given the reality of the very large differences between communities in their ethnic/cultural composition, quality of local education, and access to timely inservice, these results need to be interpreted cautiously as Firm Foundations may have different influences in different communities, and implementation may need to be supported accordingly. Another limitation of this study was the lack of a literacy outcome measure that addressed the children's growth over the year in Concepts of Print. The initial design of  63 this study focused on measures that addressed the best known predictors of reading development, phonological awareness. In order to more adequately understand the contribution the instructional component Concepts of Print made to the children's reading growth, a measure to document the children's growth in this area at the pre and post test would have been beneficial. Future Directions This study raises many questions about the factors that influence the successful implementation of curricular innovations. The need continues for study of the multiple facets of the implementation process. Incorporating a theory-based model of evaluating the implementation in routine practice programs will ensure that characteristics of the program, including dosage and quality of delivery, and ecological conditions, such as the conditions in the classroom and the quality of institutional leadership and support, will be adequately examined. Questions remain regarding the nature of the particular supports that teachers in these settings require to enhance their instruction of reading to prevent reading failure and the nature of the particular activities that are likely to have the most benefit for children with differing abilities in these contexts.  64  References Aiken, L. S., West, s. G., Schwalm, D. E., Carroll, J. L., & Hsiung, S. (1998). Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation. Evaluation Review, 22, 207 - 244. Allington, R. (1984). Content coverage and contextual reading in reading groups. Journal of Reading Behavior, 16, 85-96. Baker, J., & Zigmond, N . (1990). Are regular classes equipped to accommodate students with learning disabilities? Exceptional Children, 56, 515-526. Ball, E., & Blachman, B. (1988). Phoneme segmentation training: Effect on reading readiness. Annals of Dyslexia, 38, 208-225. Ball, E., & Blachman, B. (1991). Does phoneme awareness training in kindergarten make a difference in early word recognition and developmental spelling? Reading Research Quarterly, 26, 49-66. Blachman, B. (-1991). Phonological awareness: Implications for prereading and early reading instruction. In Phonological Processes in Literacy: A Tribute to Isabelle Y. Liberman, edited by S. Brady and D. Shankweiler. Hillsdale, NJ: Lawrence Erlbaum Associates. Beesley, J., Echols, A., Lacroix, R., Purvis, L., Calder, H., Ottley, P., Tennant, M . , Molinsky, C. (2001). Firm Foundations: Early literacy teaching and learning. (Available from Program Services, North Vancouver School District, 810 21 Street, North Vancouver, B C , V7P 2C1).  st  Blachman, B., Ball, E., Black, R., & Tangel, D. (1994). Kindergarten teachers develop phoneme awareness in low-income, inner-city classrooms: Does it make a difference? Reading and Writing: An Interdisciplinary Journal, 6, 1-18. Bos, C. S., Mather, N . , Friedman Narr, R., & Babur, N . (1999). Interactive, collaborative professional development in early literacy instruction: Supporting the balancing act. Learning Disabilities Research & Practice, 14, 215-226. Bradley, L., & Bryant, P. (1983). Categorizing sounds and learning to read: A causal connection. Nature, 30, 419-421. Bradley, L., & Bryant, P.(1985). Rhyme and reason in reading and spelling. Ann Arbor, MI: University of Michigan Press. Bryant, P. E., Bradley, L., McLean, M . , & Crossland, I. (1989). Nursery rhymes, phonological skills and reading. Journal of Child Language, 16, 407-428. Brown, A. L., & Felton, R. H . (1990). Effects of instruction on beginning reading skills in children at risk for reading disability. Reading and Writing: An Interdisciplinary Journal, 2,223-241. Bus, A.G. & Van IJzendoorn, M . H . (1999). Phonological awareness and early reading: A meta-analysis of experimental training studies. Journal of Educational Psychology, 91, 403-414. Byrne, B., & Fielding-Barnsley, R. (1993). Evaluation of a Program to Teach Phonemic Awareness to Young Children: A One-Year Follow-Up. Journal of Educational Psychology, 85, 104-111.  • ,  Catts, H. (1991). Facilitating phonological awareness: Role of speech-language pathologists. Language, Speech, and Hearing Services in Schools, 22, 196-203.  66 Chen, H. T. (1990). Theory-driven evaluations. Newbury Park, C A : Sage. Chen, H. T. (1996). A comprehensive typology for program evaluation. Evaluation Practice, 17, 121-130. Chen, H. (1998). Theory-driven evaluations. Advances in Educational Productivity, 7, 15-34. Clay, M . (2000). Concepts About Print: What-Have Children Learned About the Way We Print Language? Portsmouth, N H : Heinemann. Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2 ed). nd  Hillside, NJ: Erlbaum. Conduct Disorders Prevention Research Group (1999). Initial impact of the FAST T R A C K prevention trial for conduct problems: II Classroom effects. Journal of Consulting and Clinical Psychology, 67, 241 - 251. Cook, T. D., Anson, A . R., & Walchi, S. (1993). From causal description to causal explanation: Improving three already good evaluations of adolescent health programs. In S. G. Millstein, A . C . Petersen, & E. O. Nightingale (Eds.) Promoting the health of adolescents: New directions for the twenty-first century (pp. 339-374). Oxford University Press: New York. Cunningham, A. (1990). Explicit vs. implicit instruction in phonological awareness. Journal of Experimental Child Psychology, 50, 429-444. Cunningham, P. M . , & Cunningham, J. W. (2002). Making words: Enhancing the invented spelling-decoding connection. The Reading Teacher, 46, 106-115. Cunningham, A . E., & Stanovich, K. E. (1998). What reading does for the mind. American Educator, 22, 8-15.  67 Dane, A. V . & Schneider, B. H . (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control. Clinical Psychology Review, 18, 23-^5. Davidson, M . , & Jenkins, J. (1994). Effects of phonemic processes on word reading and spelling. Journal of Educational Research, 87, 148-157. Domitrovich, C. E., & Greenberg, M . T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-age children. Journal of Educational and Psychological Consultation, 11, 193-221. Durlak, J. A . (1995). School-based prevention programs for children and adolsecents. Thousand Oaks, C A : Sage. Durlak, J. A . (1998). Why program implementation is important. Journal of Prevention and Intervention in the Community, 17, 5-18. Durlak, J. A., & Wells, A . M . (1997). Primary prevention mental health programs for children and adolescents: A meta-analytic review. American Journal of Community Psychology, 25, 115-152. Ehri, L. C. (1984). How orthography alters spoken language competencies in children learning to read and spell. In J. Downing & R. Valtin (Eds.) Language awareness and learning to read (pp. 119-147). New York: Sringer Verlag. Ehri, L. C. (1991). Development of the ability to read words. In R. Barr, M . L . Kamil, P. Mosenthaal, & P.D. Pearson (Eds.), Handbook of reading research (Vol. 2, pp. 383-417). New York: Longman.  68 Ehri, L. C. (1992). Reconceptualizing the development of sight word reading and its relationship to recoding. In P. Gough, L. C. Ehri, & R. Treiman (Eds.), Reading acquisition (pp. 107-143). Hillsdale, NJ: Erlbaum. Ehri, L. C..(1994). Development of the ability to read words: Update. In R. B . Ruddell, M . R. Ruddell, & H . Singer (Eds.), Theoretical models and processes of reading (4 ed., pp 323-358). Newark, DE: International Reading Association. th  Ehri, L. C. (1999). Phases of development in learning to read words. In J. Oakhill & Beard (Eds.), Reading development and the teaching of reading (pp. 79-108). Oxford, U K : Blackwell. Ehri, L. C., & Nunes, S. R. (2002). The role of phonemic awareness in learning to read. In A . E. Farstrup & S. J. Samuels (Eds.), What research has to say about reading instruction (pp. 110-139). Newark, DE: International Reading Association. Ehri, L. C , Nunes, S. R., Willows, D. M . , Schuster, B. V . , Yaghoub-Zadeh, Z., & Shanahan, T. (2001). Phonemic awareness instruction helps children learn to read: Evidence from the National Reading Panels's meta-analysis. Reading Research Quarterly, 36.250-287.  *  •  ' -  "  Felton, R. (1993). Effects of instruction on the decoding skills of children with phonological processing problems. Journal of Learning Disabilities, 26, 583-589. Foorman, B. R., Francis, D. J., Fletcher, J. M . , Schatschneidr, C., Mehta, P. (1998). The role of instruction in learning to read: Preventing reading failure in at-risk children. Journal of Educational Psychology, 90, 37-55.  69 Foorman, B. R., & Torgesen, J. (2001). Critical elements of classroom and small group instruction promote reading success in all children. Learning disabilities research and practice, 16, 203 - 212. Frances, D. J., Shaywitz, S. E., Stuebing, K. K., Shaywitz, B . A., & Fletcher, J. M . (1996). Developmental lag versus deficit model moels of reading disabilities: A longitudinal individual growth curve analysis. Journal of Educational Psychology, 88, 3-17. Fuchs, D., Fuchs, L. S., Thompson, A., A l Otaiba, S., Yen, L., Yang, N . , Braun, M . , & O'Connor, R. (2002). Exploring the importance of reading programs for kindergartners with disabilities in mainstream classrooms. Exceptional Children, 68,295-311. Gersten, R., & Brengelman, S. U . (1996). The quest to translate research into classroom practice: The emerging knowledge base. Remedial and Special Education, 24, 76 -84. Gillham, J. E., Shatte, A . J., & Reivich, K. (2001). Needed for Prevention Research: Long-Term Follow Up and the Evaluation of Mediators, Moderators, and Lay Providers. Prevention and Treatment, 4, Article 1, Posted March 30, 2001 (Available on the world wide web: http://iournals.apa.Org/prevention/volume4/pre0040001a.html#cl3 Graczyk, P., Domitrovich, C , & Zins, J. E. (2003). Facilitating the implementation of evidence-based prevention and mental health promotion efforts in schools. In M . D. Weist, S. W. Evans & N . A . Lever (Eds.) Handbook of School Mental Health (pp. 301 - 318). Kluwer: New York, N Y .  Gottfredson, G. D., Gottfredson, D. C., Czeh, E. R., Cantor, D., Crosse, S.B., & Hantaan, I. (2000). National study of delinquency prevention in schools. Ellicott City, M D : Gottfredson Associates Inc. Gottfredson, D. C , & Gottfredson, G. D. (2002). Quality of school based prevention programs: Result from a national survey. Journal of Research in Crime and Delinquency, 39, 3 - 3 5 . Greenberg, M . T., Domitrovich, C , & Bumbarger, B. (2001). The prevention of mental disorders in school-aged children: Current state of the field. Prevention and Treatment, 4, Article 1, Posted March 30, 2001 (Available on the world wide web: http://iournals.apa.Org/prevention/volume4/pre0040001a.html#cl3 Greenberg, M . T., Domitrovich, C , Graczyk, P., & Zins, J. (in press). The study of implementation in school based prevention interventions: Theory, research, and practice. (Submitted to the Center for Mental Health Services (CMHS) and Substance Abuse and Mental Health Administration ( S A M H S A ) at the U.S. Department of Health and Human Services) Greenberg, M . T., Kusche, C. A., Cook, E. T., & Quamma, J. P. (1995) Promoting emotional competence in school-aged children: The effects of the PATHS curriculum. Development and Psychopathology, 7, 117-136. Greenberg, M . T., & Weissberg, R. P. (2001). Commentary on "Priorities for Prevention Research at N I M H . " Prevention & Treatment., 4, Article 25. Available on the World Wide Web: http://www.journals.apa.org/preventioiT/volume4/pre0040025c.html.  71 Gresham, F. M . (1989). Assessment of tereatment integrity in school consultation and prereferral intervention. School Psychology Review, 18, 37 -50. Gresham, F. M . (1997). Treatment integrity in single-subject research. In R. D. Franklin, D. B. Allison, B . S. Gorman (Eds.) Design and analysis of single case research (pp. 93 - 117). Lawrence Erlbaum: Mahwah, New Jersey. Gresham, F. M . & Gansle, K . A . (1993). Treatment integrity of school-based behavioural intervention studies: 1980-1990. School Psychology Review, 22, 254-273. Gresham, F. M . , Gansel, K . A., Noell, G. H., Cohen, S. & Rosenblum, S. (1993). Treatment integrity of school based behavioural intervention studies: 1980 1990. School Psychology Review, 22, 254 - 272. Griffiths, P. (1991). Phonemic awareness helps first graders invent spellings and third graders remember correct spellings. Journal of Reading Behavior, 23, 215-233. Hatcher, P., Hulme, C , & Ellis, A. (1994). Ameliorating early reading failure by integrating the teaching of reading and phonological skills: The phonological linkage hypothesis. Child Development, 65, 41-57. Heinsman, K . M . , & Shadish, W. R. (1996). Assignment methods in experimentation: When do nonrandomized experiments approximate the answers from randomized experiments? Psychological Methods, 1, 1 5 4 - 169. Henderson, (1999). Longitudinal growth curve modeling with SAS PROC M I X E D . Benchmarks Online. Retrieved online (January, 2004) at http://www.unt.edu/benchmarks/archives/2002/mav99/rss.htm.  72 International Reading Association. (1998). Phonemic awareness and the teaching of reading: A position statement of the International Reading Association. Newark, DE: Author. Juel, C. (1988). Learning to read and write: A longitudinal study of 54 children from first through fourth grades. Journal of Educational Psychology, 80, 437-477. Juel, C , Griffith, P.L., & Gough, P,B. (1986). Acquisition of literacy: Alongitudinal study of children in first and second grade. Journal of Educational Psychology, 78(4), 243-255. Kam, C , Greenberg, M . T., & Walls, (2003). Examining the role of implementation quality in school-based prevention using the PATHS curriculum. Prevention Science, 4, 5 5 - 6 3 . Kjeldsen, A.-C., Niemi, P., & Olofsson, A . (2003). Training phonological awareness in kindergarten level children: Consistency is more important than quantity. Learning and Instruction, 13, 349 - 365. Kozminsky L., & Kozminsky, E. (1995). The effects of early phonological awareness training on reading success. Learning and instruction, 5, 187 - 201. ;  Lesaux, N. K., & Siegel, L. S. (2003). The development of reading in children who speak English as a second language. Developmental Psychology, 39, 1005 -1019. Liberman, I., Shankweiler,D., Fischer, F., Carter, B. (1974). Explicit syllable and phoneme segmentation in the young child. Journal of Experimental Child Psychology, 75,201-212. Lipsey, M . W., & Cordray, D. S. (2000). Evaluation methods for social intervention. Annual Review of Psychology, 51, 345-375.  Lyon, G.R., Fletcher, J.M., Shaywitz, S.E., Shaywitz, B.A., Torgesen, J.K., Wood, F.B., Schulte, A., & Olson, R. (2001). Rethinking learning disabilities. In C.E. Finn, A.J. Rotherman, and C.R. Hokanson (Eds.), Rethinking special education for a new century (pp. 259-287). Washington, DC: Thomas B. Fordham Foundation and the Progressive Policy Institute. Lyon, G.R., Vaasen, M . , &Toomey, F. (1989). Teachers' perceptions of their undergraduate and graduate preparation. Teacher Education and Special Education, 12, 164-69. Lochman, J. E. (2001).Issues in Prevention With School-Aged Children: Ongoing Intervention Refinement, Developmental Theory, Prediction and Moderation, and Implementation and Dissemination. Prevention and Treatment, 4, Article 1, Posted March 30, 2001 (Available on the world wide web: http://j ournals.apa.orR/prevention/volume4/pre0040001 a.html#c 13 Lundberg, I., Frost, J., & Peterson, O. (1988). Effects of an extensive program for stimulating phonological awareness in preschool children. Reading Research Quarterly, 23, 263-284. MacKinnon, D. P.,& Lockwood, C. M . (2003). Advances in statistical methods for substance abuse prevention research. Prevention Science, 4, 155-171. McGuiness, D., McGuiness, C , & Donohue, J. (1995). Phonologicial training and the alphabetic principal: Evidence for reciprocal causality. Reading Research Quarterly, 30, 830 - 852. Moats, L.C. (1995). The missing foundation in teacher education. American Educator, 19,(2), 9-19.  Moats, L. C. (1994). The missing fundation in teacher education: Knowledge of the structure of spoken and written language. Annals of Dyslexia, 44, 88-102. Moncher, F. J., & Prinz, R. J. ( 1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247 - 266. Moody, S.W., Vaughn, S.R., Hughes, M . Y . , and Fisher, M . (2000). Reading instruction in the resource room: Set.up for failure. Exceptional Children, 16, 305-316. Muter, T., Hulme, C , & Snowling, M . (1997). Phonological Abiliteis Test (PAT). The Psychological Corporation: London. National Institute of Mental Health, (1998). Priotities for Prevention Research (NIH Publication No. 98-4321). Bethesda, M D : Author National Joint Committee on Learning Disabilities (1999). Learning disabilities: Preservice Preparation of general and special education teachers. Rockville, M D : American Speech and Hearing Association. National Reading Panel, (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (National Institute of Health Pub. No. 00-4754). Washington, DC: National Institute of Child Health and Human Development). Pianta, R. C , Payne, C , Cox, M . J., & Bradley, R. (2002). The relation of kindergarten classroom environment to teacher, family, and school characteristics. The Elementary School Journal, 102, 225 - 238. O'Connor, R. (1999). Teachers learning Ladders to Literacy. Learning Disabilities Research & Practice, 14, 203-214.  75 O'Connor, R. (2000). Increasing the intensity of intervention in kindergarten and first grade. Learning Disabilities Research & Practice, 15, 43-54. Offord, D., Kraemer, C , Kazdin, A. Jensen, P. & Harrington, R. (1998). Lowering the burden of suffering from child psychiatric disorder: Trade-offs among clinical, targeted, and universal interventions. Journal of the American Academy of Child and Adolescent Psychiatry, 37(7), 686-694. Rein, M . (1981). Comprehensive program evaluation. In R. A., Levine, M . A . Solomon, G.-M. Hellster, & H. Wollman (Eds.) Evaluation research and practice: Comparative and international perspectives (p. 132 - 148). Beverly Hills, C A : Sage. Reitsma, P., & Wesseling, R. (1998). Effects of computer-assisted training of blending skills in kindergartners. Scientific Studies in Reading, 2, 301-320. Schneider, W., Kuspert, P., Roth, E., & Vise, M . (1997). Short and long-term effects of training phonological awareness in kindergarten: Evidence from two German studies. Journal of Experimental Child Psychology, 66, 311 - 340. Schneider, W., Roth, E., & Ennemoser, M . (2000). Training phonological skills and letter knowledge in children at risk for dyslexia: A comparison of three kindergarten intervention programs. Journal of Educational Psychology, 92, 284 - 295. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasiexperimental designs for generalized causal inference. Shadish, W. R., & Ragsdal, K . (1996). Random versus nonrandom assignment in psychotherapy experiments: Do you get the same answers? Journal of Consulting and Clinical Psychology, 64, 1290 - 1305.  76 Share, D., Jorm, A., MacLean, R., & Matthews, R. (1984). Sources of individual differences in reading acquisition. Journal of Educational Psychology, 76, 13091324. Siegel, L . S. (2003). Basic cognitive processes and reading disabilities. In H . Lee Swanson & Karen R. Harris (Eds.) Handbook of Learning Disabilities (pp. 158 — 179). New York, N Y : Guilford Press. Siegel, L . S., & Ryan, E. B. (1989). The development of working memory in normally achieving and subtypes of learning disabled children. Child Development, 60, 973-980. Singer, J. (1998). Using SAS PROC M I X E D to fit multilevel models, hierarchical models and individual growth models". Journal of Educational and Behavioral Statistics, 24, 323 - 355. Simmons, D., Kame'enui, E. J., & Good, R. H . (1998). Accelerating children's competence in early reading and literacy: Project ACCEL - S (Federal OSEP Grant H324M980127). University of Oregon, Eugene. Snow, C.'E., Burns, M!S., Griffin, P. (1998). Preventing reading difficulties in young children. Washington, D C : National Academy Press. Stahl, S., & Murray, B . (1994). Defining phonological awareness and its relationship to reading. Journal of Educational Psychology, 86, 221-234. Cunningham, A . E., & Stanovich, K . E. (1997). Early reading acquisition and its relation to reading experience and ability 10 years later. Developmental Psychology, 33, 934-945  77 Stanovich, P. J., & Stanovich, K . E. (1997). Research into practice in special education. Journal  of Learning Disabilities,  30, 477-481.  Tangel, D., & Blachman, B.(1992). Effect of phoneme awareness instruction on kindergarten children's invented spelling. Journal of Reading Behavior, 24, 233261.  Torgesen, J.K. (1997). The prevention and remediation of reading disabilities: Evaluating what we know from research. Journal ofAcademic Language Therapy, 1, 11-47. Torgesen, J. K . (2000). Individual differences in response to early interventions in reading: The lingering problem of treatment resisters. Learning  Disabilities  Research and Practice, 15, 55-64.  Torgesen, J. K . (2002). The prevention of reading difficulties. Journal of School Psychology, 40, 7-26.  Torgesen, J. K., Wagner, R. K., Rashotte, C. A., Rosse, E., Lindamood, P., Lindamood, P., Conway, T., & Garvin, C. (1999). Preventing reading failure in young children with phonological processing disabilities: Group and individual responses to instruction. Journal of Educational Psychology, 91, 579-593. Torgesen, J., Morgan, S., & Davis, C. (1992). Effects of two types of phonological awareness training on word learning in kindergarten children. Journal of Educational Psychology, 84, 364-370.  Tunmer, W., & Nesdale, A . (1982). The effects of digraphs and pseudowords on phonemic segmentation in young children. Applied Psycholinguistics,  3, 299-311.  Tunmer W., & Nesdale, A . (1985). Phomenic segmentation skills and beginning reading. Journal and Educational Psychology, 77, 417 - 427.  U C L A Computing Services (2003). SPSS Syntax Example Using SAS Proc Mixed to Fit Multilevel Models, Hierarchical Models, and Individual Growth Models by Judith Singer. Retrieved online at http://www.ats.ucla.edu/stat/spss/paperexamples/singer/default.htm Ukrainetz, T. A., & Fresquez, E. F. (2003). What Isn't Language?" A qualitative study of the role of the speech pathologist. Language, Speech, and Hearing Services in Schools, 34;284-298.  ' . l  VanKleeck, A. (1995). Emphasizing from and meaning separately in prereading and early reading instruction. Topics in Language Disorders, 7, 27-49. Vaughn, S., Moody, S., & Shumm, J. (1998). Broken promises: Reading instruction in the resource room. Exceptional Children, 62, 211 -225. Vellutino, R. R., Scanlon, D. M . , Sipay, E. R., Small, S. G., Pratt, A . , Chen, R., & Denckla, M . B. (1996). Cognitive profiles of difficult to remediate and readily remediated and experiential deficits as basic causes of specific reading disability. Journal of Educational Psychology, 88,601-638. Weiss, C. H. (1998). Evaluation (2 Ed). Saddle Hill, NJ: Prentice Hall. nd  Weissberg, R. P., & Greenberg, M . T. (1998). School and community competenceenhancement and prevention programs. In I. E. Sigel & K . A . Renninger (Eds.) Handbook of child psychology: V o l 4. Child psychology in practice (5 ed., pp th  877-954). New York: John Wiley & Sons. Wilkinson, G. S. (1993). The Wide Range Achievement Test - 3. Wilmington, DE: Jastak Associates.  Willms, D., J. (Ed.). (1999). Quality and inequality in children's literacy: The effects of families, schools, and communities. In D. Keating and C. Hertzman (Eds.) Developmental Health and the Wealth of Nations (pp. 513 - 529). New York: Guildford Press. Willows, D. M . & Ryan, E . B . (1986). The development of grammatical sensitivity and its relation to early reading achievement. Reading Research Quarterly, 21, 253266. Wilson, S. J. & Lipsey, M . W., & Derzon, J. H. (2003). The Effects of School-Based Intervention Programs on Aggressive Behavior: A Meta-Analysis. Journal of Consulting and Clinical Psychology, 71,136-149. Wolf, J. L., Durlak, J. A., & Bryant, F. (2004).Current outcome research on preventive mental health programs for children and adolescents: A meta-analysis spanning 1992 - 2003. Poster presented at the Society for Prevention Research Annual Convention, Quebec City. Woodcock, R. W. (1987). Woodcock Reading Mastery Tests - Revised. Circle Pines, M N : American Guidance Services. Waters, J. T., Marzano, R. J., McNulty, B. A. (2003). Balanced leadership; What 30 years of research tells us about the effect of leadership on student achievement. Aurora, CO: Mid-continent Research for Education and Learning. Yeaton W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156 - 167.  80 Yeh, S., S. (2003). A n evaluation of two approaches for teaching phonemic awareness in Head Start. Early Childhood Research Quarterly, 18, 513 - 529. Zins, J. E., Elias, M . J., Greenberg, M.T., & Weissberg, R.P. (2000). Promoting social and emotional competence in children. In G. G. Bear & K . M . Minke (Eds.), Preventing school problems, promoting school success: Strategies and programs that work. Bethesda, M D : National Association of School Psychologists.  81  Appendix I Teacher Questionnaire UBC Early Literacy Study Kindergarten Teacher Background and Belief Questionnaire  : Name:  ' "Y- .2003. /  :  •  •  School:  Date:  Section One: Please tell us a little bit about yourself. 1.  Are you a male or a female?  Male  Female  To which ethnic or cultural group(s) do/did you or your ancestors belong? (Check all that apply to you) African/Caribbean Asian (Chinese, Japanese, Vietnamese, Korean, etc.) Arab/West Asian (Armenian, Egyptian, Persian, or Iranian, Lebanese, Moroccan) European (Italian, French, German, Austrian, English etc.) _____ First Nations (Native, Indian, Aboriginal) South Asian (Indo-Canadian, East Indian, Pakistani, etc.) Other (please describe) 2.  3.  What is your educational level? Some Undergraduate Coursework Bachelor Degree Post Baccalaureate Diploma  Master's Degree Doctorate Other 4.  What was your area of focus in your degree program? General Elementary Secondary Special Education/Learning disabilities Reading English as a Second Language Other, please describe  5.  How many years have you been teaching in total?  6. . How many years have you taught Kindergarten? 7.  How long have you taught in the Prince George District?  8.  Currently, do you teach one or two classes of Kindergarten children?' one  two  82  Section Two: Please answer the following questions about Firm Foundations. 1.  Have you ever heard of the Firm Foundations program?  yes 2.  Which of these best describes your current understanding of the concepts in Firm Foundations!  1 I am not familiar with the concepts 3.  2 I have a bit of understanding  4 I have an adequate grasp  5 I fully understand  I have not attended a Firm Foundations inservice 1 hour Firm Foundations inservice 3 hour Firm Foundations inservice 5 hour Firm Foundations inservice More than 5 hour Firm Foundations inservice  Are you using the Firm Foundations program in your class?  0 Never 5.  3 I somewhat understand  Which Firm Foundations in-service that you have attended:  . 4.  no ( i f n o , s k i p t o t h e n e x t s e c t i o n )  1 Once/twice  2 Monthly  3 Weekly  4 Daily  How many years have you practiced Firm Foundations concepts in your class? '_ 0 - just starting this year 1 year - started last year .. 2 years plus ++  6.  How difficult is it to incorporate the Firm Foundations concepts into your Kindergarten curriculum? 1 Not difficult  7.  2  3 4 Somewhat difficult  Very difficult  Do you have adequate time to incorporate the Firm Foundations concepts into your Kindergarten curriculum? 1 2 3 Not enough time Somewhat adequate time  8.  5  4  5 Enough time  How strongly do you believe that the concepts in the Firm Foundations program are beneficial in enhancing reading skills in young children? 1 Not beneficial  2  3 4 Somewhat beneficial  5 Very beneficial  83  Section Three: We would like to know about the emphasis that you placed on certain literacy activities in your classroom this school year. Please indicate the time period in the school year that you may have focused on each activity and circle a Literacy Activity  Time Period this Skill is Emphasized  Rhyming 1.  Frequency (Please check one)  During direct instruction  Fall  Winter  Spring  A l l year  During center time activities  Fall  Winter  Spring  A l l year  Fall  Winter  Spring  A l l year  Never Once/twice • Monthly Weekly Daily 0 1 2 3 4  Fall  Winter  Spring  A l l year  Never Once/twice Monthly Weekly Daily 0 1 2 3 4  Fall  Winter  Spring  A l l year  Never Once/twice Monthly Weekly Daily 0 1 2 3 4  Fall  Winter  Spring  A l l year  Never Once/twice Monthly Weekly Daily 0 1 2 3 4  Fall  Winter  Spring  A l l year  Never Once/twice Monthly Weekly Daily 0 1 2 3 4  Fall  Winter  Spring  A l l year  Never Once/twice Monthly Weekly Daily 0 1 2 3 4  During direct instruction  Fall  Winter  Spring  A l l year  Never Once/twice Monthly Weekly Daily 0 1 2 3 4  During center time activities Letter Sounds 1. During direct instruction  Fall  Winter  Spring  A l l year  Never Once/twice Monthly Weekly Daily 0 1 2 3 4  Fall  Winter  Spring  A l l year  Fall  Winter  Spring  A l l year  2.  Segmenting and Blending: Compound Words 1. During direct instruction 2.  During center time activities  Segmenting and Blending: Syllables 1. During direct instruction 2.  During center time activities  Segmenting and Blending: Phonemes 1. During direct' instruction 2.  During center time activities . Letter Naming ' 1.  2.  2.  During center time activities  Never Once/twice Monthly Weekly Daily 0 1 2 3 4 Never Once/twice Monthly Weekly Daily 0 1 2 3 4  Never Once/twice Monthly Weekly Daily 0 1 2 3 4 Never Once/twice Monthly Weekly Daily 0 1 2 3 4  84  Literacy Activity  T i m e P e r i o d the S k i l l is  Frequency  Emphasized  ( p l e a s e c h e c k one)  Concepts about Print * 1.  During direct instruction  2. During center time activities  Fall  Winter  Spring  All year  Fall  Winter  Spring  All year  Fall  Winter  Spring  All year  Fall  Winter  Spring  All year  Never Once/twice Monthly Weekly Daily 0 1 2 3 4 Never Once/twice Monthly Weekly Daily 0 1 2 3 4  Emergent Writing** 1. During direct instruction 2. During center time activities * /""„..„„„»„ „ i  Never Once/twice Monthly Weekly Daily 0 1 2 3 4 Never Once/twice Monthly Weekly Daily 0 1 2 3 4  * ii..:-< _  represents words, directionality of print, matching spoken word with printed word, identifyingfirstand last letter in a word, demonstration of upper and lower case, explanations that punctuation has meaning * * E m e r g e n t W r i t i n g activities may include: any activity that encourages a student to generate his/her own "writing" including scribbles, marks that the child perceives as writing, letter strings, semi-phonetic strings o f letters, phonetic representations o f words. D o e s n o t include copying letters o r words that is not student directed.  S e c t i o n F o u r : P l e a s e t e l l us a little a b o u t y o u r p r a c t i s e s this y e a r f o r t h e c h i l d r e n i n y o u r c l a s s r o o m i d e n t i f i e d as "at r i s k . "  1. How effective were your practise in identifying "children at risk?" • 1 not effective  2-  . 3 .4 somewhat effective :  5 fully effective  2. How much additional support are you able to provide for your students identified as "at risk?" • . Not applicable (i.e., no students identified) No support •  Less than once a week  •  Once a week  Q 2-3 times week •  Daily  Q Other (please describe):  85  3. Which of these best describes your use of Educational Assistant/Learning Assistance to provide additional support for your "at risk" students?  • • • • • •  No personnel support Less than once a week Once a week 2-3 times week Daily Other (please describe):  4. Which of these best describes your use of parents to provide additional support for your "at risk" students?  • • • • • •  No parent support Less than once a week Once a week 2-3 times week Daily Other (please describe):  5. Which describes your practise of engaging parents of "at risk" students in home activities (e.g., home reading, literacy games)?  • • • • • •  No home activities Home activities less than once a week Once a week 2-3 times week Daily Other (please describe):  Please provide any additional comments that might be important for us to understand the literacy context of your classroom or any information that might be important for us to know as part this evaluation.  Thank you for completing this questionnaire. Your responses will help us better understand how literacy activities contribute to reading development. Siegel Literacy Study U B C / U N B C Research Team  86  Appendix II Principal Support Measure Rating Scale of Principal Support for Early Literacy 1. Quality of Principal Support 1 = Not supportive at all As evidenced by not speaking positively or knowingly about Early Literacy with interviewer. 2= Not very supportive . . Occasional support for Early Literacy with interviewer, but does not see success of Early Literacy program and interventions for "at-risk" readers, especially those in Kindergarten, as central to the school's mission or mandate. 3= Somewhat supportive Principal is supportive of teacher's efforts, speaks positively about Early Literacy with interviewer, shows evidence of problem solving obstacles to implementation and is somewhat familiar with Early Literacy materials and has observed at least one lesson or activity. 4 = Very supportive Is a "cheerleader" for the Early Literacy program, supports teacher effectively to implement the support for "at risk" readers and may be providing some of the support him/herself. Sees the Early Literacy program as central to the school mission despite obstacles such as level of teacher training, resistance, or adequate staffing to provide a quality service. Is familiar with the early literacy concepts and materials and has observed or participated in at least one lesson.  2. Quality of Support for the Participating Early Literacy Teacher 1 = Not supportive at all Does not assist the teacher, attend inservice, or become actively involved in any aspect of the early literacy program. Does not appear to understand the issues related to the "at risk" reading and does not work collaboratively with teacher to make needed changes. 2= Not very supportive Occasionally will listen to teacher regarding "at risk" readers and may appear concerned at staff meetings, but does little to assist the teacher in any practical way thus usually does not work collaboratively with the teacher to make needed changes. Occasionally sees principal as a team member with teachers working  87 together to improve reading outcomes in the building, but this is not the usual mode of operation. 3 = Somewhat supportive Usually listens to the teacher and helps to make some of the needed changes in regard to reading by being supportive in providing materials and staffing to support the program. Often speaks the "truth" about issues in the building regarding the "at risk" population and demonstrates a willingness to support the program and improve reading outcomes for children, however, sometimes has many priorities and early intervention is not always on the top of the list. Usually sees themselves and the teachers as a team working together to improve reading outcomes in the building. 4= Very supportive This type of principal develops a true collaboration with the teacher and is seen as an essential component and true partner in helping the early literacy model to be maximally successful in the building. Probe Questions for Principal Support Interview General procedures Book an appointment to speak with the principal to discuss the arrangements that the school made to incorporate the early literacy program as his/her school. Questions 1. Are you or is someone else in your building the best person to discuss the arrangements that your school made to incorporate the early literacy program at your school? 2. Could you give me some initial information about the early literacy project at your school? 3 . How do you see your role in the early literacy project? 4. Are you familiar with the literacy activities that are part of the early intervention? Have you ever observed or participated in a lesson? 5. When students "at risk" for reading failure are identified at your school what sort of services is your school able to provide? Are you able to provide these (any) support to "at risk" readers in Kindergarten? 6. How successful do you feel the early literacy project is at your school?  88 Probe Question for Principal Support of Participating Teacher General procedures Book an appointment to speak with the teacher to discuss the support for early literacy in her school. Questions 1. Is the principal at your school supportive of early literacy program? In what ways? 2. Do you feel that your administrator understands the issues related to "at risk" reading? 3. Has the principal observed any lessons or participated in any way in your literacy program? 4. Do you feel that the principal/school support has hindered/helped your early literacy program in any way? Please explain.  89  Appendix III Observation Protocol Kindergarten Environment Tool School: Teacher : # of students present during observation (e.g., 12/15): Time/Date:  Classroom Description: Upon entering the classroom sketch the layout.  Other information: What is displayed in the classroom? Look for calendar, lists or cards of students' names, cloze activities, question of the day, depictions of alphabets on the wall or on tables, visual representations of like words (e.g., by initial sound, compound words, syllables), emergent writing student work etc.  90 Instructions to Observers: The pages that follow will allow you to capture the activities of the students that you have been assigned to observe in the classroom. Use the abbreviations at the bottom of the page where applicable. Student Code:  Group Size:  Student Engagement on task  Whole Class Literacy? (yes/no/partial)  Teacher Role  Teacher Instructional focus  (task engaged and appropriate) Teacher Role & Activity:  Materials:  Instructional Focus:  Location:  Student Code:  Group Size:  off task (active disruption, passive, not engaged in chosen/assigned task)  Student Engagement on task  Observations/Comments:  Whole Class Literacy? (yes/no/partial)  Teacher Role  Teacher Instructional focus  (task engaged and appropriate) Teacher Role & Activity:  Materials:  Instructional Focus:  Location:  Teacher Role & Activity DI = direct instruction MG = guides/model group MS = guides student observed MO = guides other student Mgmt = classroom management Indep = No interaction, teacher alone Assess = assessment  Materials LA = literacy activities C = calendar CAM = circle area material CTA = centre time activities 0 = other (please describe) M = mixed (please note)  off task (active disruption, passive, not engaged in chosen/assigned task)  Observations/Comments:  Instructional Focus R = rhyming S&B: C = segmenting and blending: compound words S&B:S = segmenting and blending: syllables S&B:P = segmenting and blending: phonemes L N = letter naming LS = letter sound activities CP = concepts of print EW = emergent writing A = assessment H-S RA = home school reading activities GenLit = reading a story to students C = copying words 0 = other  Location Floor Centres Table top Other (describe) Group Size WG = whole group Team = student team (3 -4) Pair = 2 students Indiv = individually Mixed = (please describe) Engagement Rate level of student engagement High/ Medium / Low  91  Siegel Literacy Study Coding Categories A: Instructional Focus 1. R = Rhyming: Includes any of the following activities: • Play with nonsense rhymes • Recognizing rhymes (e.g., Do these words rhyme?) • Identifying rhymes with visual (picture) or auditory cues. • Identifying rhymes (e.g., odd one out) • Recalling rhyming words (i.e., from previously heard stories, poems, songs, and chants) • Generating rhyming words 2. L N = Letter Naming: Includes any of the following • Recitation of the alphabet • Visually discriminate between the letters • Touch letters while reciting the alphabet • Identify specific letters from a group of letters when the name is provided • Identify the letters by name, OR a word that begins with particular letter 3 . LS = letter sound activities: Includes any of the following: • Discriminate between the letter sounds • Identify specific letters from a group of letters when the sound is provided • Identify the letters by sound, OR, a group of words that all begin with a particular sound • Match letter sound with letter • Begin letter-sound mastery for emergent writing 4. CP = concepts of print: Includes any of the following: • Identifying the parts of a book (e.g., front, back, title, author) • Indicating knowledge that print (not the picture) tells the story • Showing where to start reading a stOry, a page, or a sentence • Demonstration of directionality in print (i.e., left to right, top to bottom, return sweep). • Indication that letters and clusters of letters are words • Matching spoken word with printed word • Identifying first and last letter in a word • Demonstration of understanding of upper and lower case letters • Demonstration that different punctuation marks have different meanings 5. S&B: C = Segmenting and Blending: Compound Words: Includes any of the following activities: • Identifying two words in a compound word  92 • •  Deleting a word from a compound word Blending two words into a compound word  3. S&B:S • • •  = Segmenting and Blending: Syllables Break words into syllables Delete a syllable from a word Blend syllables into whole words  4. S&B:P activities • • •  = Segmenting and Blending: Phonemes: Includes any of the following Breaking words into phonemes, beginning with 2- phoneme words Delete a phoneme from a word Blend phonemes into words  5. E W = Emergent Writing: Includes any of the following • Scribbles, or marks, or pictures generally thought of by kids as 'writing' • Letter strings, random letters • Semi-phonetic strings of letters • Phonetic representations of words *** The above represent the Phonological Awareness activities ordered in a somewhat hierarchical manner related to level of complexity for the beginning reader.  Other Literacy Activity Codes 6. C = copying words: Includes any of the following: • letter formation • copying own name from a printed model • copying words from a worksheet 7. SW = sight words • "reading" signs (e.g., exit, boy's washroom, stop signs) • Recognizing commonly used words such as'and','is','the' 8. H-S R A = home school reading activities • Books or activities sent home at the end of the day to one, some, or all of the children 9. GenLit • • •  = General Literacy: Includes any of the following reading a story to students allowing students to "read" to themselves or look at books independently any other literacy not includes in the codes above  8 . 0 = other •  Not a literacy activity  B: Group Size 1. W G = whole group 2. Team = student team (3 -4) 3. Pair = 2 students 4. Indiv = individually 5. Mixed = (please describe e.g. could be W G & I, or W G & P or W G & Team) C: Materials 1. C = calendar 2. C A M = circle area material 3 . 0 = other (please describe) D: Location 1. Floor/Carpet 2. Centres 3. Table top 4 .Other (describe)  E : Teacher Role/Activity 1. • • • •  D l = direct instruction: Involves usually situations when: The teacher is guiding the instruction Often, this occurs when she has the entire class' attention May includes a combination of questioning, information, showing Typically involves attending to group or rapidly from one student to the next in a group  2. M G = guides/model group • Typically involves singing, or chanting along with the W G 3. M S = guides student observed • Guiding or showing individual observed • May also include some one-to-one guidance, instruction 4. M O = guides other student • Guiding or showing any student other than the one observed • May also include some one-to-one guidance, instruction 5. Mgmt = classroom management • Typically involves managing behaviour or providing instructions to line up, behave some expected manner as a group  94  6. • • •  Indep = No interaction, teacher alone Teacher may be working alone at her desk Talking with another adult in the room Observing the children but not interacting with them  7. • • •  Assess = assessment Assessing individual child skill level Typically teacher and student are alone during assessment May occur while students are presenting for the group  Appendix IV Student Literacy Measures  Rhyme Detection Instructions  Examiner: "Here is a picture of a cat. Down here are three more pictures..." (the examiner points to and names each of the 3 choice pictures). Now which of these three - fish, sun or hat rhymes with cat?" Provide the correct answer (hat) if necessary and explain that hat rhymes with cat because they end with the same sound (at). Continue as above with the other 2 demonstration items, giving explanations when necessary. The instructions fro the 10 items are the same as for the demonstration items. Do not give feedback on the test items. If the child fails the demonstration items and the first 5 test items, you may discontinue the test.  Demonstration Items Response Items  Stimulus Word  1. cat 2. ball 3. spoon  fish wall cup  sun bell moon  hat bag ship  Test Items Response Items  Stimulus Word 1. boat 2. key 3. chair 4. house 5. head 6. bell 7. sock 8. train 9. egg 10. car  foot cow car mouse hand bottle clown rain bag star  bike tree table horse bed dress clock tractor spoon bike  coat door bear window eye shell shoe spoon leg cake  V Letter Identification  Instructions Examiner: I am going to show you letters one at a time. Tell me the name of each letter. j  9 I  z  s a e u d w t  f n o c m x V  h r b  q y  p Score  /26  Syllable and Phoneme Identification Instructions for Syllable Identification (Word completion)  Examiner: "Here is a picture of a rabbit. I'm going to say the first part of the word. Can you finish it off for me? Here is a ra..." (The child should respond 'bit.' If the child fails to give the correct answer, say "IF I say ra, you finish the word by saying bit. Let's try it again with rabbit. Ra...." Supply the bit again if necessary.) Repeat as above for the second example, bottle. A full explanation and feedback are given for the two demonstration items. Present the test items 1 to 8 with the instructions, "This is a table. Ta...." Do not give feedback for the test items. If the child fails the demonstration items and the first four test items, the task may be discontinued. Demonstration Items Ra-bbit Bo-ttle Test Items 1. 2. 3. 4. 5. 6. 7. 8.  Ta-ble Pic-ture Cabb-age Mon-ey O-range Sand-wich Mon-ster Lem-on  score:  /8  Instructions for Phoneme Identification  Examiner: "Now we are going to do something that is a bit more difficult. Here is a picture of a watch. Til say the first part - you finish it off. Here is a watch. Wa..." Provide corrective feedback if necessary. Repeat for the demonstration item. cat. Proceed with items 1-8 using the instructions "This is a horse. Hor..." Do not provide feedback for test items. If the child fails the demonstration items and the first four test items, the task may be discontinued. Demonstration Items: _ _ Wa-tch Ca-t Test Items: 1. 2. 3. 4. 5. 6. 7. 8.  Hor-se Fi-sh Kni-fe Shi-p Bo-ne Car-d Ga-te Do-g score'-  /8  Phoneme Deletion  Instructions for Initial Phoneme Deletion: Examiner: "Here is a picture of a bus. I f I say the word /bus/ without the /b/, we'll be left with/us/. Bus without/b/says us. Let's t r y some more. Give all 4 demonstration items, and explain fully, as f o r "bus." Administer items 1 to 8 with the instruction, "Meat without /m/ says...." Do not give feedback f o r the test items. I f the child fails the demonstration items and the f i r s t 4 test items, you may discontinue the task. Demonstration Items bus  sad  pie  cow  T e s t Items 1.  seat  2.  bear  3.  hat  4.  sit  5.  jam  6.  tin  7.  cake  8.  cup  score  /8  Instructions for Final Phoneme Deletion Examiner: "Now this time, instead of taking o f f the f i r s t sound of words, let's t r y and take o f f the last sound. This will make things that are not real words. Here's a picture of a foot. Can you hear the last sound in foot? T h e last sound in f o o t is / t / . Now can you say foot without/t/?  Foot without hi  is f oo."  Give all 4 demonstration items, and explain fully as f o r foot. Administer items 1 to 8 with the instruction, "Meat without hi  says...." Do not give  feedback f o r the test items. I f the child fails the demonstration items and the f i r s t 4 test items, you may discontinue the task. Demonstration Items foot  bag  bell  spoon  T e s t Items 1.  seat  2.  sad  3.  hat  4.  bus  5.  jam  6.  tin  7.  cake  8.  cup  score T o t a l score  /8 /16  iQO  Simple Spelling  Name:  1.  101 Table 1. Firm Foundations Literacy Components Rhyming • Play with nonsense rhymes • Recognize rhyme (Do these words rhyme?) • Identify rhymes (visual then auditory cues) • Identify rhymes (odd one out) • Recall rhyming words (from previously heard stories, poems, songs and chants) • Generate rhyming words Segmenting and Blending* • Hear individual words in a sentence (s) • Identify two words in a compound word (s) • Blend two words into a compound word (b) • Delete a word from a compound word (s) • Break words into syllables (orally) (s) • Blend syllables into whole words (b) • Delete a syllable from a word (s) • Identify words spoken in onset and rime (s) • Break words into phonemes (s) • Blend phonemes into whole words (b) • Delete a phoneme from a word (s) (* Segmenting (s) and blending (b) activities) Letter-Sound Mastery • Discriminate between sounds • Recite the alphabet • Visually discriminate between letters • Touch letters while reciting the alphabet • Discriminate between letter sounds • Identify specific letters from a group of letters (when the letter sound or name is provided) • Identify letters by name, sound, or word • Match letter sound with letter • Begin using letter sound mastery for emergent writing Concepts of Print • Know parts of a book (front, back, title) • Know that print (not the picture) tells a story • Know where to start reading a story, a page, a sentence • Understand directionality in reading (left to right, top to bottom, return sweep) • Know that there are letters and clusters of letters called words • Match spoken word with printed word • Identify first and last letters in words • Know that there are upper and lower case letter • Know that different punctuation marks have different meanings (period, question mark, quotation mark)  Table 2. Demographic Information for Participating Students  Mean Age (months) Gender Male Female Race Aboriginal Caucasian Other First Language English Punjabi Spanish Vietnamese Reading Status Children at Risk for Reading Failure  Sample N = 151 M= 66.95 (SD = 3.60) 68 83 20 128 3 143 4 2 2 41  103  Table 3. Percentage of Whole Class, Partial Class, and Total Class Time Spent on Literacy Activities by Classroom Whole Class Literacy Activity Classroom  M(SD)  Partial Class Literacy Activity M(SD)  Total Percentage of Time Spent on Literacy M(SD)  11  49.43 (10.07)  17.02 (7.17)  66.45 (8.62)  2  46.28 (6.24)  18.87(14.91)  65.15 (10.57)  7  52.63 (12.83)  11.95 (10.99)  64.58 (11.91)  3  50.23 (6.68)  12.88 (3.25)  63.11 (4.97)  1  41.48 (8.27)  21.27(13.66)  62.75 (10.96)  10  42.47 (22.37)  19.63 (16.29)  62.10(19.33)  13  41.90 (7.73)  18.76 (8.97)  60.66 (8.35)  5  31.48(10.88)  22.07 (9.61)  53.55 (10.25)  4  24.17 (9.34)  26.05 (13.42)  50.22(11.38)  8  20.33 (6.21)  19.00 (9.70)  39.33 (7.96)  6  16.38(4.61)  21.85(14.75)  38.23 (9.68)  12  14.45 (8.50)  21.97(12.29)  36.42 (10.40)  9  17.00 (6.82)  12.22 (10.56)  29.22 (8.69)  Grand Total  34.53 (14.25)  18.32 (4.46)  52.85 (13.49)  Note: Teacher 1 and 2 worked in the same school; Teacher 10 and 11 are the same person. Teacher 10 taught in the a.m. and 11 taught in the p.m.  104 Table 4. Percentage of Time Spent on Firm Foundations Literacy Components by Classroom Rhyming  Classroom  M(SD)  Segmenting and Blending M(SD)  Letter Sound Mastery M(SD)  Concepts of Print M(SD)  Total Firm Foundations Components M(SD)  11  5.88 (5.26)  2.02 (3.12)  20.81 (11.51)  6.82 (6.67)  35.53 (6.64)  2  3.13 (2.77)  2.83 (2.92)  4.88 (4.49)  10.66 (9.38)  21.50 (4.89)  7  1.37 (2.68)  9.88 (8.43)  21.55 (18.73)  2.57 (2.20)  35.37 (8.01)  3  6.90 (12.00)  8.65 (5.65)  7.53 (7.73)  14.28 (14.57)  37.36 (9.99)  10  6.03 (5.32)  1.13 (2.16)  15.13 (4.76)  6.38 (5.56)  28.67 (4.45)  13  3.92 (3.95)  4.58 (4.69)  6.08 (6.57)  5.68 (3.43)  17.19(4.66)  5  5.12(3.43)  5.65 (2.14)  14.20 (8.94)  4.97 (5.51)  29.94 (5.01)  4  0(0)  3.97 (6.17)  16.41 (8.15)  1.63 (2.25)  22.01 (4.14)  1  2.05 (2.75)  .91 (1.66)  4.55 (3.61)  7.57 (8.54)  15.08(4.14)  8  .55 (1.10)  2.32 (1.68)  3.08 (1.62)  7.43 (3.47)  13.38 (1.97)  6  0(0)  .35 (.70)  1.80 (3.60)  3.43 (6.85)  5.58 (4.18)  12  1.17(2.86)  2.88 (2.89)  6.98 (3.14)  2.32 (3.37)  13.35 (3.07)  9  .98 <2.19)  .40 (.98)  4.36 (4.17)  1.20(1.20)  6.94(2.14)  Total Average Components  2.85 (2.46)  3.50 (3.32)  9.79 (6.88)  5.76 (3.76)  21.90(4.10)  Table 5. Correlations, Means, and Standard Deviations for Classroom Risk, Teacher Background and Beliefs, Principal Support, and Classroom Observations Variable  |  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  Classroom Factor 1. Percentage of Children in Class Identified as "at Risk" Teacher Background and Beliefs 2. Education 3. Years of Experience  21.06  ,2.23  .44  —  14.85  6.79  4.15  .80  .37  -.30  .45+.  .68*  .10  .53+  —  6. Frequency of Use of FF  .07  .23  -..12  .62*  .69**  -.04  .49+  .20  .38  -.32  .49+  -.10  -.02  -.47+  .28  -.53+  16. Total Percent Direct Instruction Activities 17. Total Percent "Hands-on" activities 16. Total Percent FF Activities  23.52  .32  -.08  Classroom Observations 15. Total Percent Literacy  SD  .12 .14  4. Understanding of Firm Foundations (FF) 5. FF Inservice Hours  7. Number of Years Practicing FF 8. Level of Difficulty Incorporating FF 9. Adequate Time to Incorporate FF 10. Level of Belief in FF Concepts 11. Effectiveness of Identifying Children at Risk 12. Frequency of Additional Support for Children at Risk 13. Use of Parents to Provide Support for Children at Risk Principal Support 14. Level of Principal Support  M  .2.23  1.48  _  3.23  1.09  .67*  .64*  1.92  .86  .18  .24  -.13  .25  2.00  1.15  -.22  .28  .04  .55*  .39  3.76  .93  -.21  .03  -.23  -.01  -.20  4.38  .76  4.30  .75  3.00  1.00  1.00  1.00  -.55* .60*  -.69**  .02  -.17  .47  -.07  .21  .43  .75** .29  -.07  7.19  .12  .21  .34  .76**  .58*  -.54*  .19  .16  -.21  .00  .15  .10  -.12  .-52+  -.09  .64*  .42  .33  .62*  .34  .17  -.24  .64*  .00  .13  -.84**  -.05  -.40  .29  -.27  .04  .22  .44+  .16  -.22  .76**  -.03  .10  -.83**  .07  -.36  .33  -.25  -.01  .21  .40  .10  -.12  .70**  -.13  .11  -.38  -.03  -.17  .02  .16  -.02  .03  .17  -.28  .06  -.63*  .20  -.15  .44+  -.18  -.18  .20  .52+  -.04  -.34  .70**  +p<.10. *p<.05. **p<.0\.  I:  .35  -.08  -.29  .72**  .11  .11  .14  -.27  -.65*  .11  .08  2.23  —  .47+  52.85  13.49  .46  .46  34.53  14.25  .32  .41  -.04  18.32  4.46  -.18  -.01  .62*  19.58  9.49  .80*  Table 6. Means and Standard Deviations at Pre and Post Test and Ts for Groups of Students for Each Literacy Measure Measure  Students not at Risk (n=110) Pretest  Posttest  M(SD)  M(SD)  7.91 (2.78)  8.87(1.86)  5.28(1.81)  1 t-test (2-tailed)  S t u d e n t sa tR i s k (n = 41) Pretest  Posttest  t-test  M(SD)  M(SD)  (2-tailed)  4.05**  7.03 (2.97)  7.89 (2.70)  6.44(1.55)  5.47**  3.58 (2.73)  3.41 (3.07)  5.05 (3.09)  5.68**  3.31 (4.75)  8.34(5.14)  19.45 (5.62)  All Students (« = 151) Pretest  Posttest  t-test  M(SD)  M(SD)  (2-tailed)  2.59*  7.67(2.84)  8.61 (2.15)  4.80**  5.75 (1.97)  4.35**  4.83 (2.22)  6.25(1.69)  6.90**  1.23 (2.13)  3.77 (3.06)  5.04**  2.82 (2.99)  4.07 (3.13)  7.47**  8.36**  1.10(2.26)  5.24 (4.85)  4.27**  2.71 (4.32)  7.49 (5.22)  9.38**  23.76 (2.95)  9.36**  7.68 (5.63)  16.90 (6.56)  10.83*  16.25(7.67)  21.65 (5.38)  12.74**  63.74(17.50)  66.35(20.08)  1.89+  22.23(13.21)  31.00(21.79)  2.47**  52.47(24.74)  56.76(25.84)  3.04**  60.49(18.38)  59.46(17.76)  .81  9.39 (7.66)  31.59(20.78)  7.75**  46.62(27.95)  51.18(22.77)  3.59**  3.34(1.89)  4.74 (1.62)  8.99**  1.17(1.31)  2.63 (1.31)  8.54**  2.75(1.92)  4.16(1.80)  11.56**  Phonological Awareness Rhyme (10) (n=130) Syllable Identification (8) (n = 150) Phoneme Identification(8) (n = 145) Phoneme Deletion (16) (n = 106) Symbol Awareness Letter (n= 1 WJ per (n = Word  Identification (26) 51) centile 129) Reading  W R A T 3 -R p e r c e n t i l e (n= 151) Spelling (maximum 6) (n = 130)  + = p < .07, * = p < .05, * * = p < .001  107 Figure 1. Model of Implementation Evaluation  CHILD LEVEL  CLASSROOM LEVEL  T  SCHOOL LEVEL  Program as Implemented  DISTRICT LEVEL  Modified from Greenberg, Domitrovich, Graczyk, & Zins, 2002  COMMUNITY LEVEL  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0099785/manifest

Comment

Related Items