Education, Faculty of
Educational and Counselling Psychology, and Special Education (ECPS), Department of
DSpace
UBCV
Turri, Mary Gwendolyn
Doctor of Philosophy - PhD
University of British Columbia
Across Canada, thousands of children are struggling readers. In British Columbia alone, between 18 and 22% of students are below provincial reading expectations (BC Ministry of Education, 2016). As a critical component to reading success, yet often neglected in classroom instruction, reading fluency is a viable intervention target to help remediate these challenges. Due to concerns regarding intrinsic motivation, the use of rewards in an educational context is contentious. However, for students who are not already intrinsically motivated to read, providing extrinsic reinforcers such as a tangible reward may serve as a way to effectively engage them in reading activities and encourage practice. The current evidence base for reading fluency interventions shows mixed results for whether adding tangible rewards produces any incremental gains in reading fluency. This study examined the effects of tangible rewards within the context of a standard protocol reading fluency intervention on reading and motivation outcomes. Using a multiple-baseline multiple-sequence across participants design, participants completed an intensive 8-week standard protocol reading fluency intervention with and without rewards, in a counter-balanced sequence across participants. Results indicated that across reading measures, there was no differentiation between the rewards and no rewards conditions, suggesting that rewards did not appear to either help or hinder performance on reading measures. Similarly, rewards did not appear to affect participants’ self-reported motivation to read. Across conditions, the intervention was effective in improving direct fluency gains on instructional passages for five out of six participants across both multiple baselines. For between-session fluency gains, the intervention was effective for one participant. The intervention was not effective in improving generalized gains in reading fluency. Results are discussed in terms of implications for educators, limitations of the study, and need for future research.
THE USE OF REWARDS IN READING FLUENCY INTERVENTIONS: CONSIDERATIONS FOR STUDENT ACHIEVEMENT AND MOTIVATION by Mary Gwendolyn Turri B.A. (Hons), The University of Calgary, 2010 M.A., The University of British Columbia, 2014 A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIRMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in The Faculty of Graduate and Postdoctoral Studies (School Psychology) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) September 2019 © Mary Gwendolyn Turri, 2019 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: The Use of Rewards in Reading Fluency Interventions: Considerations for Student Achievement and Motivation submitted by Mary Gwendolyn Turri in partial fulfillment of the requirements for the degree of Doctor of Philosophy in School Psychology Examining Committee: Dr. Sterett Mercer Supervisor Dr. Pat Mirenda Supervisory Committee Member Dr. Laurie Ford Supervisory Committee Member Dr. Janet Jamieson University Examiner Dr. Lee Gunderson University Examiner iii Abstract Across Canada, thousands of children are struggling readers. In British Columbia alone, between 18 and 22% of students are below provincial reading expectations (BC Ministry of Education, 2016). As a critical component to reading success, yet often neglected in classroom instruction, reading fluency is a viable intervention target to help remediate these challenges. Due to concerns regarding intrinsic motivation, the use of rewards in an educational context is contentious. However, for students who are not already intrinsically motivated to read, providing extrinsic reinforcers such as a tangible reward may serve as a way to effectively engage them in reading activities and encourage practice. The current evidence base for reading fluency interventions shows mixed results for whether adding tangible rewards produces any incremental gains in reading fluency. This study examined the effects of tangible rewards within the context of a standard protocol reading fluency intervention on reading and motivation outcomes. Using a multiple-baseline multiple-sequence across participants design, participants completed an intensive 8-week standard protocol reading fluency intervention with and without rewards, in a counter-balanced sequence across participants. Results indicated that across reading measures, there was no differentiation between the rewards and no rewards conditions, suggesting that rewards did not appear to either help or hinder performance on reading measures. Similarly, rewards did not appear to affect participants’ self-reported motivation to read. Across conditions, the intervention was effective in improving direct fluency gains on instructional passages for five out of six participants across both multiple baselines. For between-session fluency gains, the intervention was effective for one participant. The intervention was not effective in improving generalized gains in reading fluency. Results are discussed in terms of implications for educators, limitations of the study, and need for future research. iv Lay Summary Reading fluency, the ability to read words and other reading material with adequate speed and ease, is an important skill to master in order to successfully read and understand material at school, home, and in the community. Alongside skill-based strategies (e.g., correcting errors, listening to an adult read, repeated practice), providing struggling readers with small incentives, such as a reward, may help engage them in reading practice and ultimately support their reading skill growth. The purpose of this dissertation was to examine whether rewards are a useful component to include in a reading fluency intervention program. The results showed that students had similar performance whether or not rewards were used. Overall, the intervention improved reading after practicing compared to the students’ performance before they started the intervention. In general, the intervention did not show the same improvement for students’ reading fluency on stories they had not yet practiced. v Preface This dissertation is the original work of Mary Gwendolyn Turri, under the supervision of Dr. Sterett Mercer. Drs. Laurie Ford and Pat Mirenda served as members of the supervisory committee and provided input on the study and this manuscript. An early version of the discussion of motivation presented in the Introduction was published in Foundations of Education (Cloth, Turri, & Archambault, 2016), with feedback offered by the co-authors of the manuscript as per typical publication procedures. Two graduate students in School Psychology served as implementation research assistants alongside Ms. Turri. A third School Psychology graduate student conducted independent inter-observer agreement analyses. This study was approved by the University of British Columbia Behavioural Research Ethics Board (BREB; H16-03037-A002). vi Table of Contents Abstract ......................................................................................................................................... iii Lay Summary ............................................................................................................................... iv Preface ............................................................................................................................................ v Table of Contents ......................................................................................................................... vi List of Tables .............................................................................................................................. viii List of Figures ................................................................................................................................ x Acknowledgments ....................................................................................................................... xii Dedication ................................................................................................................................... xiii Introduction ................................................................................................................................... 1 Rewards ..................................................................................................................................... 2 Intrinsic Motivation .................................................................................................................. 3 Rewards and Intrinsic Motivation .......................................................................................... 4 Rewards and motivation in general education ........................................................................ 7 Rewards and reading ............................................................................................................... 9 Rewards in Reading Fluency Interventions ......................................................................... 10 Methodological limitations and considerations for rewards ................................................. 12 Methodological considerations for reading fluency ............................................................. 13 Skill and Performance-based Strategies ............................................................................... 14 Standard-protocol intervention ............................................................................................. 16 The Current Study ...................................................................................................................... 18 Research Questions ................................................................................................................. 18 Hypotheses ............................................................................................................................... 18 Method ......................................................................................................................................... 19 Participants and Setting ......................................................................................................... 19 Materials .................................................................................................................................. 19 Instructional passages ........................................................................................................... 19 Measures .................................................................................................................................. 20 Reading ................................................................................................................................. 20 Reading fluency direct gains .......................................................................................................................... 20 Reading fluency between-session gains. ........................................................................................................ 21 Generalized fluency gains .............................................................................................................................. 21 Reading comprehension ................................................................................................................................. 21 Motivation. ............................................................................................................................ 22 Social validity ....................................................................................................................... 22 Procedures ............................................................................................................................... 23 Leveling ................................................................................................................................ 23 CBMReading monitoring level ...................................................................................................................... 23 HELPS intervention level .............................................................................................................................. 23 Baseline ................................................................................................................................. 24 Intervention phases ............................................................................................................... 24 Intervention. .......................................................................................................................... 25 vii Visual analogue scale and cold read .............................................................................................................. 26 Word drill of passage vocabulary .................................................................................................................. 26 Repeated readings sequence .......................................................................................................................... 27 Graphing and performance feedback ............................................................................................................. 28 Rewards procedure ......................................................................................................................................... 29 Interventionist training .......................................................................................................... 30 Treatment fidelity .................................................................................................................. 30 Inter-scorer agreement .......................................................................................................... 31 Research Design ...................................................................................................................... 31 Data Analysis ........................................................................................................................... 33 Visual analysis ...................................................................................................................... 33 Descriptives and inferential statistics .................................................................................... 34 Effect size .............................................................................................................................. 34 Exploratory statistical analysis ............................................................................................. 35 Results .......................................................................................................................................... 36 Direct Gains on Instructional Passages................................................................................. 37 Between-Session Gains on the First Read of Instructional Passages ................................. 47 Within-session Gains on Instructional Passages .................................................................. 57 Generalized Gains ................................................................................................................... 58 Exploratory Statistical Analysis ............................................................................................ 68 Comprehension ....................................................................................................................... 75 Self-reported Motivation ........................................................................................................ 80 Social Validity ......................................................................................................................... 84 Discussion .................................................................................................................................... 86 Main Findings ......................................................................................................................... 86 Effects of rewards on reading outcomes ............................................................................... 86 Effects of rewards on motivation .......................................................................................... 86 General effectiveness of reading fluency intervention ......................................................... 86 Reading comprehension ........................................................................................................ 87 Response to Rewards .............................................................................................................. 87 Direct vs. Generalized Fluency Gains ................................................................................... 91 Measurement Challenges when Assessing Generalized Reading Fluency Gains .............. 92 Need for Group Designs to Disentangle Small Differences in Effects ................................ 93 Limitations ............................................................................................................................... 96 Implications and Future Directions ...................................................................................... 97 Conclusion ............................................................................................................................. 100 References .................................................................................................................................. 102 Appendix A: Social Validity Scale ........................................................................................... 115 Appendix B: Reward Items ...................................................................................................... 116 Appendix C: Additional Tables ............................................................................................... 117 Appendix D: Additional Figures ............................................................................................. 119 viii List of Tables Table 1 Descriptive statistics for direct gains on instructional passages across conditions and participants for both sequences ..................................................................................................... 40 Table 2 Tau values and weighted mean effect sizes for direct gains across participants and conditions for both sequences ....................................................................................................... 41 Table 3 Descriptive statistics for between-session gains on the first read of instructional passages across participants and conditions for both sequences .................................................. 49 Table 4 Effect size (Tau) values and weighted mean effect sizes for between-session gains on the first read of instructional passages across participants and conditions for both sequences .... 50 Table 5 Paired samples t-test for first to fourth reads on instructional passages in intervention phase across participants and conditions for both sequences ....................................................... 58 Table 6 Descriptive statistics for generalized gains on CBMReading passages across participants and conditions for both sequences ................................................................................................ 59 Table 7 Effect size (Tau) values and weighted mean effects for generalized gains on CBMReading passages across participants and conditions for both sequences ........................... 61 Table 8 Meta-regression for direct gains on instructional passages across conditions and sequences ...................................................................................................................................... 70 Table 9 Meta-regression for direct gains on instructional passages comparing condition B (rewards) and C (no rewards) by sequence ................................................................................... 70 Table 10 Meta-regression for between-session gains for first reads on instructional passages ... 72 Table 11 Meta-regression for between-session gains on first read of instructional passages comparing condition B (rewards) and C (no rewards) by sequence ............................................. 72 Table 12 Meta-regression for generalized gains on CBMReading passages ............................... 74 ix Table 13 Meta-regression for generalized gains on CBMReading passages comparing condition B (rewards) and C (no rewards) by sequence ............................................................................... 74 Table 14 Mean number correct and standard deviations for comprehension questions by sequence and condition ................................................................................................................. 76 Table 15 Self-reported motivation to read across participants and conditions for both sequences....................................................................................................................................................... 81 Table 16 Raw scores and mean scores for social validity ratings across participants and sequences ...................................................................................................................................... 85 Table 17 Percent of non-overlapping data for direct gains on instructional passages across phases .......................................................................................................................................... 117 Table 18 Percent of non-overlapping data for between-session gains on instructional passages across phases ............................................................................................................................... 117 Table 19 Percent of non-overlapping data for generalization passages across phases .............. 118 x List of Figures Figure 1. Direct gains on instructional passages, Sequence A ..................................................... 39 Figure 2. Direct gains on instructional passages, Sequence B ...................................................... 46 Figure 3. Between session gains on first read of instructional passage, Sequence A ................... 52 Figure 4. Between session gains on first read of instructional passage, Sequence B ................... 55 Figure 5. Generalized gains on CBMReading passages, Sequence A .......................................... 64 Figure 6. Generalized gains on CBMReading passages, Sequence B .......................................... 65 Figure 7. Weighted mean effect sizes (Tau) for direct gains on instructional passages by condition and sequence. ................................................................................................................ 70 Figure 8. Weighted mean effect sizes (Tau) for direct gains on instructional passages comparing condition B (rewards) and C (no rewards) by sequence.. ............................................................. 71 Figure 9. Weighted mean effect sizes (Tau) for between-session gains on first read of instructional passages. ................................................................................................................... 72 Figure 10. Weighted mean effect sizes (Tau) for between-session gains on first read of instructional passages comparing condition B (rewards) and C (no rewards) by sequence. ........ 73 Figure 11. Weighted mean effect sizes (Tau) for generalized gains on CBMReading passages for each condition compared to baseline. ........................................................................................... 74 Figure 12. Weighted mean effect sizes (Tau) for generalized gains on CBMReading passages comparing the B (Rewards) and C (No Rewards) conditions ....................................................... 75 Figure 13. Comprehension questions correct, Sequence A .......................................................... 78 Figure 14. Comprehension questions correct, Sequence B ........................................................... 79 Figure 15. Self-reported motivation to read, Sequence A ............................................................ 82 Figure 16. Self-reported motivation to read, Sequence B ............................................................. 83 xi Figure 17. Comparison of direct gains on instructional passages between reads 1 and 4, Sequence A .................................................................................................................................................. 119 Figure 18. Comparison of direct gains on instructional passages between reads 1 and 4, Sequence B .................................................................................................................................................. 120 xii Acknowledgments I would like to thank my supervisor, Dr. Sterett Mercer, for his support not only on this dissertation, but throughout my graduate training. Dr. Mercer always had confidence in me, which I borrowed from during times when I struggled to find it in myself. I also thank my committee members, Drs. Pat Mirenda and Laurie Ford, for their longstanding support. Dr. Mirenda has gone out of her way to support me in pursuing Positive Behaviour Support training, which taught me all about the intricacies of reinforcement and informed my knowledge for this project. I thank her for teaching me that everyone can learn (and deserves to learn) when we find the right way to teach them. Dr. Ford has supported me as part of her extended lab family, been my cheerleader, and always stepped in to help me when I needed it- clear eyes, full hearts! Thank you to my research assistants, Nathalie Sagar and Rachel King, for their hard work and dedication as interventionists. I am also so grateful to Michèle Cheng, for her work in conducting the implementation fidelity and inter-scorer agreement analyses. I am so appreciative of the students who served as the participants in this study, their teachers, and the school’s special education resource teacher, who worked tirelessly to help me bring this study to fruition. To my friends and colleagues who have been on this journey with me, I could not have done this without you. In special recognition of my Masters cohort, the pseudonyms for the participants in this study are named after them. To my “adopted” PhD cohort, we truly are in this together, and your friendship, solidarity, and humour have been invaluable to me. Above all else, thank you to my family, whose unwavering belief in me has been an endless source of encouragement. Because of them, I am so fortunate and grateful that I have never had to look far for support whenever I needed it. I am especially grateful to my parents, who instilled in me the values of hard work, lifelong learning, enthusiasm, kindness, and looking for the good. xiii Dedication To my grandmothers: Phyllis Gwendolyn Scott (1920 – 2017) and Nita Livia Turri (1927 – 2017). Two feisty women who would have been so delighted to call me Dr. Turri. 1 Introduction Across Canada, thousands of children struggle to meet grade-level expectations for reading. Recent data show that between 18 and 22% of British Columbian students are below provincial expectations for reading skill (BC Ministry of Education, 2016). Poor reading achievement puts students at-risk for a host of negative outcomes, including high school dropout and lower postsecondary education attainment (Statistics Canada, 2006; 2009). Without timely intervention, the skill gap between successful and struggling readers may widen; over time, successful readers become more proficient, whereas struggling readers’ skills tend to develop more slowly, or plateau (Stanovich, 1986). Therefore, understanding interventions to support the development of reading skill is an important area of research. As with learning any new behaviour, the fluency stage of reading is essential for overall reading success (Chard, Vaughn, & Tyler, 2002; Joseph, Alber-Morgan, & Neef, 2016; Martens & Witt, 2004). Reading fluency is the ability to read text with speed and accuracy; a fluent reader has moved past the stage of effortful decoding and is able to read sight words and connected text with automaticity (Chard et al., 2002). Achieving fluency has shown to be a strong predictor of overall reading achievement (Fuchs, Fuchs, Hosp, & Jenkins, 2001). From a cognitive perspective, dysfluent readers expend their mental energy decoding words, leaving little capacity to comprehend the material; conversely, fluent readers decode with a degree of automaticity, allowing them to devote more attention to higher-order tasks, such as comprehension (LaBerge & Samuels, 1974). Indeed, interventions aimed at improving reading fluency, such as repeated reading of connected text, have also been shown to produce improvements in reading comprehension (e.g., Stevens, Walker, & Vaughn, 2017; Therrien, 2004). Despite its importance to overall reading success, fluency is often not addressed in 2 classroom instruction. Martens and Witt (2004) suggest that repeated practice of a skill, the basis for building fluency, is a task often left up to the student to complete on their own time, thus it is often not directly addressed in the classroom. Interventions for reading fluency typically involve the stand-alone use or combination of evidence-based instructional strategies generally conceptualized as either skill- or performance-based (Chafouleas, Martens, Dobson, Weinstein, & Gardner, 2004). Skill-based strategies are instructional approaches such as repeated practice of reading material, error correction, and modeling. Performance-based strategies are meant to enhance motivation to read, and may include goal-setting, performance feedback, and contingent rewards. The former two, goal-setting and performance feedback, could be construed as the less controversial of the performance-based strategies. For example, both are emphasized as components of developing self-regulated learning (Butler & Winne, 1995). In contrast, with a deeply rooted and contentious history, the use of contingent rewards as a motivational strategy is more controversial. Rewards For the purpose of the current study, rewards are defined as any tangible object or activity offered for “good” or desired performance. Examples include sticker charts, token or point systems, candy, toys, or finite privileges such as extra computer time. Here, it is important to distinguish between reward and reinforcement. Simply stated, a reward is a thing, whereas reinforcement is an effect (Maag, 2001). A stimulus is reinforcing (i.e., a reinforcer) if it increases the target behaviour over time (B. F. Skinner, 1953). For example, if a teacher provides a student with a sticker every time they raise their hand to answer a question, and the frequency of the student’s hand raising increases, the sticker (i.e., the reward) had a reinforcing effect on hand raising. In contrast, if the hand raising behaviour does not increase, the stickers did not 3 produce a reinforcement effect. Therefore, a reward may or may not function as a reinforcer, although it is typically the hope or assumption that it will. Similarly, it is important to distinguish praise and rewards. As with rewards, verbal (e.g., “nice hand raising!”) or nonverbal (e.g., thumbs-up, high five) praise may also have a reinforcing effect on the target behaviour. Although both praise and tangible rewards can have a reinforcing effect, they are distinguished for the purposes of the current study. In contrast to tangible rewards, verbal and nonverbal praise are considered social reinforcers (B. F. Skinner, 1953). Practically, this can be considered reinforcement that is typically available as a part of the natural social environment; for instance, it is not uncommon to offer a teammate a high-five after scoring a touchdown. In contrast, it is less “natural” to offer that teammate a candy bar. The somewhat unnatural and potentially arbitrary nature of tangible rewards warrants separate consideration from social reinforcement. Intrinsic Motivation The practice of using rewards in schools is a highly contentious topic that typically produces polarizing opinions. There is a rich history of debate regarding how rewards affect learners (e.g., Akin-Little, Eckert, Lovett, & Little, 2004; Cameron, Banko, & Pierce, 2001; Deci, Koestner, & Ryan, 1999; Kohn, 1993,1999; Lepper, Greene, & Nisbett, 1973). The primary argument against using rewards is the potential to undermine the learner’s intrinsic motivation. Intrinsic motivation is generally defined as engagement in tasks and activities for inherent enjoyment or interest, separate from external consequences (Ryan & Deci, 2000). The concept of intrinsic motivation dates back to early animal behaviour studies; classic work by Harlow, Harlow, and Meyer (1950) documented that rhesus monkeys completed mechanical puzzles in the absence of external reinforcement, suggesting the presence of inherent enjoyment 4 in the task (Lepper & Henderlong, 2000; Lepper, Sethi, Dialdin, & Drake, 1997; Sansone & Harackiewicz, 2000). Deci (1975) observed the seemingly innate quality of intrinsic motivation in humans, noting the natural curiosity and propensity for learning inherent in young children. As discussed by Lepper and Henderlong (2000), the suggestion that an organism would engage in an activity for something other than purely extrinsic reinforcement was a departure from radical behaviourism, which dominated the study of motivation at that time. As inquiry and theories developed, it became apparent that intrinsic motivation was demonstrated in humans, but only under certain circumstances (Ryan & Deci, 2000). Thus, research turned to understanding under what conditions intrinsic motivation was enhanced versus undermined; it was through this work that the concept of extrinsic rewards undermining intrinsic motivation developed (Sansone & Harackiewicz, 2000). Rewards and Intrinsic Motivation In 1973, Lepper and colleagues developed the over-justification hypothesis, which posits that intrinsic motivation and task engagement will decrease following removal of the rewards for tasks that individuals already enjoy doing. In the initial examination of this hypothesis, Lepper et al. found that preschoolers who received expected rewards for a drawing activity spent significantly less time engaging in drawing activities during a later free play period than those who received either unexpected or no rewards. More recent work by Akin-Little and Little (2004) examined the hypothesis using the effects of a token economy on rule compliance using an ABA design across three participants with high baseline levels of compliance. Contrary to the hypothesis, no participants showed a significant decrease in rule compliance following removal of the token system. 5 Perhaps the most well-known analyses of the interaction between rewards and intrinsic motivation are the series of meta-analyses by reward proponents (e.g., Cameron & Pierce, 1994; Cameron et al., 2001), and those in opposition to rewards, (e.g., Deci et al., 1999). Deci and colleagues based their analyses on Cognitive Evaluation Theory (CET; Deci & Ryan, 1980), which posits that rewards undermine intrinsic motivation to the extent that they are related to one’s perceived competence and self-determination. Rewards that are informational (i.e., presents an indication of competence) would presumably increase intrinsic motivation by virtue of their competence and self-determination enhancement effects. Conversely, rewards perceived as controlling would reflect negatively on one’s perceived competence and sense of autonomy and volition, thus undermining intrinsic motivation (Deci et al., 2001). Recognizing that casting a wide net on rewards would wash out the heterogeneous effects of individual contingencies, Deci and colleagues further delineated rewards into specific categories. Based on the broad finding that unexpected rewards (participant had no prior knowledge of availability) did not have an undermining effect, but expected rewards did, the latter category was broken down into separate contingencies: task non-contingent, engagement-contingent, completion-contingent, and performance-contingent. Task–non–contingent rewards are offered regardless of participation, whereas engagement–contingent rewards specify participation but with no expectation for quality or completion. Completion–contingent rewards require completion of the task regardless of quality, and performance–contingent rewards are those that are offered for performing the task to a certain criterion. Presented hierarchically, performance–contingent rewards are presumably the most controlling given that the criterion is externally imposed (Deci et al., 2001). Whereas neutral effects were found for task–non-contingent rewards, undermining effects were reported for 6 engagement and completion–contingent rewards. For performance–contingent rewards, undermining effects were detected for studies that utilized a free-choice condition to measure intrinsic motivation, whereas no effects were detected when motivation was measured with self-report. In their competing meta-analysis, Cameron et al. (2001) also detected neutral effects for unexpected rewards and expected task–non–contingent rewards, as well as undermining effects for engagement–contingent rewards. However, rather than categorizing rewards as either completion or performance–contingent, the studies were categorized by specific contingency (e.g., completing a certain number of tasks, exceeding a criterion score). This categorization led to a finer-grained analysis of the conditions and demonstrated the complexity of the interaction between rewards and motivation, in which either neutral, undermining, or enhancing effects on intrinsic motivation were detected. In general, no detrimental effect was detected when rewards were explicitly linked to a performance criterion. Conversely, rewards that were linked to failure or offered in the absence of performance criteria tended to produce undermining effects. More recently, Cerasoli, Nicklin, and Ford (2014) identified the narrow focus on undermining effects as a significant shortcoming of previous meta-analytic work on rewards and intrinsic motivation. They argued that previous work has demonstrated a failure to resolve three paradoxical premises “(a) incentives boost performance, (b) intrinsic motivation boosts performance, and (c) incentives reduce intrinsic motivation” (p. 5), and noted that the narrow focus on the undermining effect of rewards ignores the inextricably interrelated nature of extrinsic and intrinsic factors on performance. In their meta-analysis across educational, occupational, and physical domains, they instead focused on how incentives and intrinsic motivation interact to affect performance, with hypothesized moderation by the salience of the 7 reward relative to the task (direct versus indirect), and task performance type (quality versus quantity). Their findings supported the interactional relationship, demonstrating that intrinsic motivation had an overarching influence on performance, but extrinsic factors held more importance when incentives were directly linked to performance (e.g., receiving extra course credit for completing a certain assignment), and when the performance criteria emphasized quantity (e.g., reading a certain number of books). Cerasoli et al.’s (2014) work lends itself well to the complex interaction of motivation factors and performance in educational settings. Yet, the topic of using rewards in schools is largely conceptualized within the paradigm of extrinsic and intrinsic factors as competing entities. Indeed, the use of rewards in schools continues to be grounded in rhetoric regarding the use of rewards as agents of persuasion, bribery, coercion, and control. In his book, which has reached success in popular culture, Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A’s, Praise, and Other Bribes, Kohn (1993/1999), denounces external reinforcers as bribes intended to induce compliance, in turn thwarting student self-assurance, independence, and self-determination. Deci et al. (2001) argue that “tangible rewards are frequently used to persuade people to do things they would not otherwise do, that is, to control their behaviour” (p. 9). These conceptualizations of rewards are not only misleading, but fail to account for the interplay of both internal and external factors in engaging students and promoting successful and sustainable learning outcomes. Indeed, separating intrinsic and extrinsic motivation may be a trivial distinction with little utility in an educational setting. Rewards and motivation in general education. It makes intuitive sense to consider both intrinsic and extrinsic factors related to student performance in educational contexts. For example, Cerasoli et al. (2014) noted that in order to parse out undermining effects of extrinsic 8 factors, most research tasks were designed to be enjoyable to begin with. Yet, when considering school-based performance, particularly for struggling learners, the assumption that tasks are enjoyable to begin with is unlikely to be the case. Moreover, if research tasks are designed to be enjoyable to begin with, this translates to unclear measurement of baseline levels of performance, which in turn ignores within-subject change in behaviour over time and washes out individual reinforcement effects (Akin-Little et al., 2004). To advocate for pure intrinsic motivation assumes that it is better for students to complete tasks out of their own volition rather than due to external contingencies, such as receiving a reward, getting a good grade, or avoiding punishment. However, this idealistic argument would be difficult, if not infeasible, to pursue in education. Brophy (2004) argued that given externally–driven curricula and outcomes, expecting students to be intrinsically motivated across all aspects of their education is, though a valuable outcome to work towards, decidedly unreasonable. Moreover, expecting widespread intrinsic motivation is overly simplistic; there are numerous factors that may influence a student’s motivation towards academic tasks. Hidi and Harackiewicz (2000) suggest that for unmotivated students, the use of rewards in conjunction with strategies that foster intrinsic motivation may be necessary to promote engagement. They note, “a combination of intrinsic rewards inherent in interesting activities and external rewards, particularly those that provide performance feedback, may be required to maintain individuals’ engagement across complex and often difficult – perhaps painful – periods of learning” (p. 159). Hidi and Harackiewicz’s recommendations are consistent with those provided by Akin-Little et al. (2004) in their synthesis of studies regarding rewards and motivation. First, they note that rewards should be applied with specific reference to performance, and not simply for participation. Second, rewards should not be offered only on a single occasion, rather, rewards 9 should be presented systematically as a means of helping to shape desired behaviour, and then appropriately faded as behaviour change occurs. Additionally, it is important that the reward selected functions as a reinforcer. Rewards and reading. The above reward guidelines fit well into the logic for including rewards in reading interventions. Following these guidelines, rewards should be offered to students for reaching a predetermined criterion though goal-setting (e.g., how many words a student will read in a one-minute reading of a passage). Given that the reader can be an active participant in goal-setting, this may limit the perception of “control” from external factors. Moreover, when goal-setting is coupled with informational feedback (e.g., showing or telling the student how many words they read), it may work to foster competence by signaling to the student that they achieved their goal by working hard through practicing (Deci et al., 2001). For struggling readers, it may be unrealistic to assume that students with low reading skill will choose to engage in reading practice out of their own volition; the matching law dictates that given the choice between two competing behaviours, humans will engage in the behaviour that provides the highest rate of reinforcement (Herrnstein, 1961). Given the task difficulty and response effort required, dysfluent readers may require external motivation in order to engage in reading activities in favour of other competing behaviours (i.e., off-task behaviours or non-responding; C.H. Skinner, 1998). Students who are intrinsically motivated to read are more likely to choose to read, read more, and seek to strengthen their reading (Wigfield & Guthrie, 1997; Wigfield, Guthrie, Tonks, & Perencevich, 2004). Using tangible rewards may be an effective way to encourage reading practice for students who are not already intrinsically motivated to engage in the effortful practice of reading (Martens & Witt, 2004). It follows that with more reading practice, the learner’s skills will increase, allowing access to the natural 10 reinforcement inherent in reading, that is, learning new information on topics of interest. Once this pattern develops, rewards can be faded in favour of natural reinforcement. Rewards in Reading Fluency Interventions The majority of studies that have examined the use of rewards in reading fluency interventions have done so in the context of brief experimental analysis (BEA). BEA is an inductive approach to individualizing interventions based on student response to common treatments (Daly et al., 1999). In BEA, two or more procedures, selected based on preliminary hypotheses about the student’s core deficits, are briefly piloted in various combinations until the most efficacious procedure is determined, which is then selected for on-going intervention (Daly et al., 1999). As can be inferred from the name, the brevity of the assessment is a hallmark of the approach, meaning a maximum of one to three trials are administered for each condition (Wilbur & Cushman, 2006). More recently, BEA studies have started to include an extended analysis in an effort to confirm the findings (e.g., Anderson, Daly, & Young, 2013; Wilbur & Cushman, 2006). BEA is built on an idiographic approach; therefore, it is unsurprising that the results across the existing literature have consistently shown differential effects of rewards on oral reading fluency performance (e.g., Anderson et al., 2013; Daly et al., 1998; Daly, Bonfiglio, Mattson, Persampieri, & Forman-Yates, 2005; Daly & Martens, 2002; Daly, Persampieri, McCurdy, & Gortmaker, 2005; Eckert, Ardoin, Daisey, & Scarola, 2000; Eckert, Ardoin, Daly, & Martens, 2002; Wilbur & Cushman, 2006). Given that each study used different rewards procedures and contingencies, as well as different combinations of performance and skill-based components, it is difficult to make definitive judgments about the effectiveness of rewards, particularly in relation to other components. Yet, these types of judgments are not appropriate for BEA designs, which, by function, are not conducive to, nor are they meant for, a robust analysis 11 of individual components for general effectiveness across students. The short exposure to each intervention component in a BEA is not intended to provide a reliable observation of effects for a single component, such as rewards. Yet, although BEA research does not provide definitive conclusions about the general effectiveness of rewards, the procedures may be amenable to those who are concerned about reward use; in BEA, rewards, like any intervention component, are only used if needed to produce optimal responding. Outside of specific BEA designs, a number of studies have examined the effects of rewards alone and in combination with instructional strategies or performance feedback, or both, on reading fluency, using various within-subject designs. Similar to the BEA findings, differential effects of rewards have been found across studies. Noell et al. (1998) found that the combination of rewards and a skill-based treatment package was more effective than reward alone in achieving mastery on instructional grade-level passages for two of three participants in a multiple-baseline design. For one participant, the skill-based package was more effective without added rewards. Bonfiglio, Daly, Martens, Lin, and Corsaut (2004) found that contingent reward and feedback, a skill-based treatment package, and a combination of both components were equally effective for one subject using a multiple probe design. Strong-Hilsmier, Wehby, & Falk (2016) found that combining contingent rewards and performance feedback with a skill-based intervention was more effective than skill-based alone for three out of four participants using a multiple-baseline design. Clearly, the potential differential effectiveness of rewards in improving reading fluency performance speaks to the complex relationship between reinforcement and instructional components, and most importantly, the learner. Chafouleas et al. (2004) posited that learner-centered factors, such as baseline skill level, play an important role in the interaction of 12 reinforcement contingencies and instructional components. Using an alternating treatments design, the study investigated the additive effect of performance strategies, performance feedback (PF) and contingent reward (CR), alone, and combined with a skill-based strategy, repeated reading (RR), across three participants. All treatments resulted in improvement over baseline; however, for the participants with the highest baseline skill level (highest reading rate and lowest error level), RR alone was most effective overall, followed by RR+PF. For the participant with the lowest baseline skill level, reinforcement contingencies, though not necessarily rewards, were necessary; RR+PF was most effective, followed by RR+PF+CR. At the student level, these results suggest that student skill level may play a role in the necessity for rewards, and in particular, supports the argument that rewards may not necessarily be needed for behaviours that are already at a high rate of responding (i.e., higher baseline skill level). Methodological limitations and considerations for rewards. Looking across these studies illuminates several limitations in terms of how rewards are operationalized and measured, as well as broader methodological considerations with important implications for how improvement in reading fluency is measured and understood in the literature. With respect to rewards, the potential for a diluted reinforcement effect based on reinforcement value and unclear or weak contingencies cannot be ruled out. Noted as a limitation by Chafouleas et al. (2004), but also present in many of the reviewed studies, the identified reading goals, reading only one more word than their previous reading, put an upper limit on the number of words to exceed, which may have diluted the reinforcement effect. As evidenced by the meta-analytic research on rewards and intrinsic motivation (Cameron et al., 2001; Cerasoli et al., 2014; Deci et al., 2001), when considering the relation between motivation and rewards, the contingencies are particularly important. For example, from a cognitive evaluation theory perspective, a modest 13 one-word increase goal may not be sufficient to foster feelings of competence and may instead signal control and low self-determination. Additionally, in some studies (e.g., Daly et al., 2005), participants were pre-selected as being responsive to contingent rewards and were excluded from the study if their reading fluency did not improve from rewards only as part of the pre-screening procedures. Additionally, few studies employed an a priori preference assessment to determine rewards that would have a higher probability of functioning as reinforcers. Only Strong-Hilsmier et al. (2016) conducted an informal preference assessment by having students rank order a list of possible rewards, while Noell et al. (1998) reported an understanding that the selected rewards were identified as potentially effective reinforcers based on their use in a previous intervention. If the chosen reward does not function as a reinforcer, then rewards cannot be expected to function as an effective consequence strategy. For example, if a student perceives a bouncy ball as negative or neutral in value, it is unlikely that receiving the bouncy ball as a reward would have any positive effect on the student’s reading fluency. This may erroneously lead to the conclusion that rewards are not effective. Although an a priori preference assessment does not guarantee the selected rewards will function as reinforcers, it would increase the probability. Methodological considerations for reading fluency. Aside from the methodological limitations of reward operationalization and measurement, a review of this literature also reveals broader considerations for the measurement of oral reading fluency in intervention studies. For example, the results of Noell et al. (1998), Bonfiglio et al. (2004), and Chafouleas et al. (2004) must be interpreted in light of unaccounted-for practice effects due to repeated exposure to a limited number of the instructional and/or generalization passages across conditions. In addition, there is significant variability in the literature in terms of what measures of reading fluency are 14 used as the dependent variable, which has not yet been disentangled in recent meta-analytic work (Lee & Yoon, 2017). For example, operationalizing improvement in reading fluency using the last read of an instructional passage at the end of a repeated reading sequence (e.g., Chafouleas et al., 2004; Noell et al., 1998) is likely to show a greater effect than if performance is measured at the beginning of each session before practice occurs (e.g., Begeny et al., 2015). Both are valuable metrics, but the first read of a new passage may be particularly important in terms of ecological validity, with respect to measuring progress on unpracticed material (Lo, Cooke, & Peirce-Starling, 2011). Similarly, there is variability in the literature with respect to the timing of generalization probes; specifically, whether probes are given after an instructional session (i.e., after learners have practiced reading), or separately. For example, Daly et al. (2005) administered generalization probes at the end of an instructional session, showing more robust evidence of generalization than the modest findings from Noell et al. (1998), who administered the probes in a separate session. Skill and Performance-based Strategies When taking into account both the BEA and additional literatures, it appears that rewards are generally more effective when combined with skill-based strategies. This makes theoretical sense, particularly within a behaviour analytic paradigm applied to academic skill acquisition (Joseph, Alber-Morgan, & Neef, 2016). Eckert and colleagues (2002) differentiate skill-based techniques as antecedent strategies for reading fluency behaviour, with performance-based techniques such as contingent rewards serving as consequences for fluent reading. Skill-based strategies create the acquisition of new skills that are then reinforced with consequence strategies (Daly et al., 2005). Chafouleas and colleagues (2004) suggest that reinforcement (i.e., 15 consequence) strategies are meant to work in conjunction with skill-based (i.e., antecedent) strategies to bring fluent reading “under the stimulus control of printed text” (p. 70). The combination of both strategies is important; although rewards may also serve to strengthen already mastered skills (Noell et al., 1998), no amount of reinforcement can teach a child words they do not already know (Haring, Lovitt, Eaton, & Hansen, 1978, as cited in Daly et al., 2005). In addition, rewards may serve as a means to maximize the reinforcement available for engaging in reading practice, reducing the probability that the student will engage in competing off-task behaviours (C. F. Skinner, 1998; Herrnstein, 1961). When considering the broadest impact for learners, combining both antecedent and consequence strategies may maximize the intervention effectiveness for the greatest number of learners. This logic contrasts with the tenets of individualized intervention approaches, such as BEA, which suggests that interventions for assessment be selected based on hypotheses about the students’ core deficits. For example, many of the studies differentiated between skill and performance strategies based on the premise that reading deficits are skill-based (“can’t do”), performance-based (“won’t do”), or potentially both (Burns & Wagner, 2008; Daly et al., 1997). In everyday practice, the interaction of skill and performance deficits is far more complex (Bonfiglio et al., 2004; Chafouleas et al., 2004). This complexity likely contributes to the mixed results across the literature. In sum, the literature suggests that individual differences can be expected in terms of responsiveness to rewards. Some children may need rewards to engage in reading intervention and reinforce newly learned skills, or simply to engage them in the learning process so that they can attend long enough for learning to occur. For others, praise and performance feedback coupled with skill building may be sufficient. What is important is that rewards are not uniformly rejected based on this differential effectiveness; across all reviewed 16 studies, there were no cases where rewards were detrimental to reading skill development. This is important information to consider when applying this literature to school-based interventions, particularly within standard protocol intervention approaches aimed at providing the most effective support while balancing efficiency for the greatest number of students. Standard-protocol intervention. Although the literature suggests an understanding that individual differences may be expected with regards to rewards, there is a lack of research regarding the effectiveness of rewards in the context of standard protocol intervention. Whereas procedures like BEA are based on an idiographic approach, a standard protocol employs the use of (ideally) an evidence-based treatment package implemented consistently to all students with similar deficits. Using an evidence-based standard treatment protocol is a hallmark of the Data-Based Individualization approach (DBI; National Centre on Intensive Intervention, 2018). DBI is a framework for supporting students’ academic (or behavioural) needs though targeted intervention that is systematically monitored and adapted based on the student’s response (Danielson & Rosenquist, 2014; Lemons, Kearns, & Davidson, 2014). In a DBI framework, a standard treatment protocol is administered as the front-line intervention for students who are not adequately responsive to classroom instruction (Lindström, Gesel, & Lemons, 2019). If the student fails to progress with the standard protocol, adaptations to the protocol are made systematically from least to most individualized until the student shows adequate response to intervention (Lindström et al., 2019). Although there are obvious benefits to approaches such as BEA and others that focus on individualized assessment and intervention, drawbacks such as the potential for measurement error affecting intervention decisions (Burns et al., 2017; Mercer, Harpole, Mitchell, McLemore, & Hardy, 2012; Solomon, Dawes, Duhon, & Poncy, 2018), and variability of procedures for 17 selecting assessment methods and interventions for comparison (Burns & Wagner, 2008), are important to consider. Compared to individualized approaches, a major advantage of standard protocols is the efficiency and control gained when both student progress and treatment fidelity can be monitored across students with the same procedures and on the same criteria (Fuchs, Mock, Morgan, & Young, 2003). When students are served well on carefully selected standard protocol approaches (Christ, 2008), this theoretically opens up resources to support students who require more intense, individualized interventions, implemented by professionals with specialized training, such as learning support teachers or school psychologists. In the context of a standard protocol, the concern is not necessarily about individual differences, but about the general effectiveness of a component across students. If rewards are generally effective for all students, or highly effective for some students, then barring detrimental effects for other students, a rewards procedure should be included in a standard protocol. There are several standard protocol interventions for reading fluency. Popular programs include Read Naturally (Hasbrouck, Ihnot, & Rogers, 1999), Great Leaps (Mercer & Campbell, 1998), and Helping Early Literacy with Practice Strategies (HELPS; Begeny, 2009). Each integrate a number of strategies known to increase reading fluency, and Great Leaps and HELPS include a rewards procedure. In particular, the HELPS intervention, which was shown to outperform Great Leaps on standardized measures of reading fluency (Begeny et al., 2010), has been shown to improve reading fluency across a number of studies (e.g., Begeny, 2011; Begeny, Mitchell, Whitehouse, Harris, & Stage, 2011, Begeny et al., 2015; Mitchell & Begeny, 2014). However, to date, no studies have examined the use of rewards specifically in the context of HELPS or any other standard protocol intervention. 18 The Current Study The purpose of this study is to address the question of whether rewards are a useful component to include as part of a standard protocol reading fluency intervention. Second, this study aims to address the gaps and expand the current research base by using robust methodology to isolate the additive effect of rewards on reading fluency gains for both instructional and generalization passages, while accounting for sequence effects. Additional extensions from previous literature include an informal a priori preference assessment, assessment of self-reported motivation to read, varied assessment of generalized gains on unpracticed passages, and assessment of reading comprehension. Research Questions 1. Does the use of contingent tangible rewards in reading fluency interventions produce differential gains in reading fluency and comprehension compared to interventions with no tangible rewards? 2. Is there an observable difference between self-reported motivation to read during sessions when contingent rewards are used versus sessions when rewards are not used? 3. Do participants rate the standard protocol as an acceptable intervention? Hypotheses Accounting for previous research, individual responses to the use of rewards are anticipated such that some participants may shower higher growth rates in the rewards intervention than others. However, no detrimental effects of rewards on the reading outcomes are expected. Higher ratings of motivation to read are anticipated during the intervention sessions with rewards. Moderately high acceptability ratings are expected for the intervention. 19 Method Participants and Setting Six students attending an Independent general education elementary school located in Vancouver, British Columbia, were recruited to participate in the study. Included were two students each from Grades 2, 3, and 4, who were identified by their classroom teacher and the school’s special education resource teacher as struggling readers. The included participants were English-speaking males (n = 4) and females (n = 2). All participants are identified using pseudonyms in the current study. None of the students were formally identified in any special education category. Two of the students (Ty and Phineas), were receiving phonics-based reading tutoring once per week for the duration of the school year from an external tutor who conducted sessions at the school. All of the participants received academic support from the special education resource teacher approximately twice per week for the duration of the school year. As part of this support, three of the students (Phineas, Marita, & Chris) sometimes practiced oral reading fluency; however, this was one of several areas of support they were receiving (i.e., in addition to working on written output, reading comprehension, or other academic areas). The type and frequency of these supports were consistent throughout the duration of the current study. Parental informed consent and student assent were obtained prior to the commencement of the study. Materials Instructional passages. Passages from the Helping Early Literacy with Practice Strategies passage set (HELPS; Begeny, Mann, Cunningham, & Tsuen, 2009) were used as instructional passages. The HELPS passage set includes 100 instructional passages that gradually increase in difficulty from approximately the Grade 1 to 4 level (Begeny, 2009). The passages 20 are arranged such that the initial third (i.e., passages 1 - 33) are generally suitable for the ability levels for Grades 1 - 2, the next third for Grades 2 - 3, and the final third for Grades 3 - 4 (Begeny, 2009). Measures Several measures were used to assess each of the dependent variables for reading, motivation, and social validity. Reading. Three measures were employed to assess direct and between-session gains on instructional passages, generalized fluency gains, and reading comprehension. With respect to the reading fluency measures, it is important to recognize that oral reading fluency is traditionally conceptualized as including automaticity of word reading, accuracy, and prosody (e.g., Pikulski & Chard, 2005). The emphasis of the intervention components in the current study is on automaticity, which is primarily reflected in repeated reading of connected text (Lee & Yoon, 2017). However, accuracy is also included through the error correction procedures, with prosody generally included through the interventionist modeling good reading (the listening passage preview procedure), though not directly targeted. Reading fluency direct gains. The primary dependent measure, direct gains on oral reading fluency, was measured by words read correctly per minute (WCPM) on the final (fourth) one-minute timed read of the instructional passages (i.e., HELPS passages), assessed at every session. WCPM is calculated by totaling the number of words the student reads in the passage within a minute, minus the number of errors. Errors were defined as miscues (incorrectly read or substituted words), skipped words (words the participant failed to read within 3 seconds), or words read out of order. If the student hesitated on a word for 3 seconds, the interventionist provided the word and marked it as an error. 21 Reading fluency between-session gains. A secondary dependent reading fluency measure was WCPM from the initial (“cold”) one-minute timed read of instructional passages (i.e., HELPS passages), assessed at every session. This is referred to as “between session gains” as it measures the change over time on initial reads of each new instructional passage. Generalized fluency gains. A secondary dependent reading fluency measure was WCPM on a one-minute timed read of a novel generalization passage from the Curriculum Based Measurement for Reading (CBMReading) probe set (FastBridge Learning, 2015). This measure was collected at every 1st and 3rd session of the week. The probe set is divided into 3 levels: Level A (Grade 1); Level B (Grades 2 - 3): Level C (Grades 4 - 6). Technical adequacy for CBMReading is strong; median internal consistency estimates across grade levels range from .90 to .92, and median split-half reliability coefficients are >.95 across grade levels. Median test-retest reliability coefficients range from .82 to .93 from Fall to Spring administrations. Median inter-rater reliability coefficients range from .97 to .98, across all assessed grade levels. Finally, median alternate-form reliability coefficients range from .74 to .94 across grade levels. Standard error of measurement for single probes (M = 6.57) ranged from 3.03 to 10.41 WCPM (FastBridge Learning, 2015). Reading comprehension. A brief measure of reading comprehension, FastBridge CBMComp (FastBridge Learning, 2015), was used to assess gains in reading comprehension over time and was administered alongside every second generalization passage. After reading the passage in full, the interventionist asked the participant 10 questions about the story, scoring the response as either correct or incorrect (i.e., 1 or 0). The total score out of 10 was used as the corresponding data point. In its full form, CBMComp also contains a retell procedure. In order to streamline the assessment process and administration time, only the comprehension questions 22 were administered for the current study. Technical adequacy data for CBMComp is unavailable in the most recently available technical report (FastBridge Learning, 2015). Motivation. Participants were asked to rate their motivation to read on a 10 cm visual analogue scale. They did so by being asked to respond to the prompt “I am looking forward to reading today”, accompanied by a 10 cm line anchored at the low end by, “Not looking forward to reading at all”, and at the high end by, “Really looking forward to reading”. Participants were asked to mark their response on the line, and the distance of their mark from the low end of the line was used as the data point (e.g., 6.8 cm). Because one may (or may not) be motivated to read for a variety of reasons, that may or may not be considered intrinsic motivations, the students were asked a follow-up question, “why did you choose that rating today?” to provide some context for their rating. The use of a visual analogue scale was chosen to maximize response variability beyond what could be provided by the typical Likert scale, particularly given that it is a single-item scale. Comparable reliability has been demonstrated in the use of visual analogue versus Likert scales in questionnaires for children (van Laerhoven, van der Zaag-Loonen, & Derkx, 2007). Social validity. To assess the social validity of the intervention, an adapted version of the Children’s Intervention Rating Profile (CIRP; Witt & Elliott, 1985) was administered at the end of the study. Modeled after Begeny and Martens’s (2006) adaptation of the same scale, the items on the original CIRP were adapted to better suit the construct of acceptability for a reading intervention. For example, “I like the method used for this child’s behaviour problem”, was changed to, “I liked this reading program”. The questions on the scale targeted the participants’ subjective enjoyment of the program, perceived usefulness of the program, and the potential benefit to other children (see Appendix A). For the current study, the number of items was 23 shortened from 7 to 4 for simplicity, as some items were not as amenable to adaptation to a reading intervention (e.g., “The method used to deal with the behaviour problem may cause problems with this child’s friends”). The participants answered each of the four items on a five-point Likert scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). Procedures The research procedures were comprised of leveling, baseline, and intervention phases, each described in detail below. Leveling. Procedures for determining appropriate CBMReading passage levels and start points for the HELPS instructional passages are described below. CBMReading monitoring level. Participants underwent an individual assessment of their oral reading fluency skills to determine a reasonable level for progress monitoring for generalized gains. Participants completed three one-minute timed reads on grade-level FastBridge CBMReading passages (Level A, Grade 1; Level B, Grades 2 and 3; Level C, Grades 4 - 6). The median score for the three passages was used as the decision-making score, which was then compared to the FastBridge cut points for passage level (FastBridge Learning, 2015). Based on their scores, participants were assigned to a progress monitoring level. All participants scored within the general range for their grade level, and were assigned to Level B (for the Grade 2 and 3 students) or Level C (for the Grade 4 students). HELPS intervention level. Participants underwent a leveling assessment with similar procedures set by the HELPS program (Begeny, 2009), in order to determine a reasonable starting point for the intervention passages, based on the participant’s skill level. Participants were individually administered two one-minute timed reads of passages beginning at the lowest level in the passage set, and their average score was compared to cut points for their grade level. 24 The process continued, administering more difficult passages until their average score was within an appropriate difficulty level, as set by the HELPS criteria (Begeny, 2009), which then served as a reasonable start point in the passage set. For the current study, the placement assessments for each participant placed them generally within a single passage range (e.g., 1 to 33), with the exception of Sara, who started on passages at the upper end of the Grade 1 - 2 level, and moved into the Grade 2 - 3 passages. Specifically, three participants began at passage 1 (Marita, Chris, Phineas), two began at passage 5 (Ty and Lee), and one began at passage 25 (Sara). Baseline. Assessment sessions were conducted three times per week for the duration of the baseline phase. At the beginning of the session, participants were asked to respond to the motivation visual analogue scale, followed by an oral response to “why did you choose that rating today?”. Next, the participants engaged in a one–minute timed read of a generalization passage (every first and third session of the week only). Reading comprehension for the corresponding generalization passage was administered at every third session of the week. Next, the participants completed a one-minute timed read of an instructional passage. Their resulting WCPM scores were used as a baseline data point. No intervention components were implemented during the baseline phase. Intervention phases. The order and schedule of administration for the motivation visual analogue scale, generalization passages, and instructional passages were the same as the baseline phase. The motivation visual analogue scale was presented at the beginning of each session, followed by a generalization passage probe to assess generalized gains at every first and third session of the week, with a corresponding reading comprehension probe at every third session of the week. Between-session gains on instructional passages were assessed by the initial cold read 25 of the instructional passage, and direct gains on the instructional passages were assessed by the final read, collected at the end of each intervention session. Intervention. The standard protocol reading fluency intervention package used was an adapted version of the HELPS program (Begeny, 2009). HELPS is an individually administered reading fluency intervention that integrates several evidence-based strategies for improving reading fluency: repeated readings, modeling, systematic error correction, goal-setting, performance feedback, and motivational components including a star chart and occasional tangible rewards (available contingent on star chart progress). For the purposes of the present study, four main modifications to the standard HELPS protocol were employed; first, the reward procedure included the opportunity to receive a tangible reward in every session that the participant’s reading goal was met, during the phases that rewards were used. This is different from the standard HELPS procedure, in which stars on the star chart, but not tangible rewards, are given for meeting the set reading goals. In standard HELPS, tangible rewards are only given when the student reaches the end of a row on the star chart. Second, participants who were not in the rewards condition did not receive the star chart or tangible rewards. Third, participants read from a novel intervention passage each session, as opposed to the HELPS procedures, which use a performance goal as the criterion for moving to a new passage. Fourth, teaching of high frequency vocabulary words using word drill was incorporated as an additional component, replacing a second round of error correction. Additional information and rationale for the modifications are addressed in the following sections. Each participant received the adapted HELPS intervention, applied three times per week, for approximately 15 minutes per session. Previous research using the HELPS intervention suggests that an intervention frequency of three times per week produces greater effects than 26 once per week, with sufficient effects achieved at an average of one and a half sessions per week (Begeny, 2011). The upper limit was selected to promote maximum benefit, and to provide a buffer for any potentially missed sessions due to student absences. The intervention sequence was as follows: 1. Presentation of motivation visual analogue scale. 2. Cold read of HELPS intervention passage. 3. Word drill of passage vocabulary. 4. Repeated readings sequence. a. 2nd timed read. b. Phrase-drill error correction. c. 3rd timed read. d. Listening passage preview. e. 4th timed read. 5. Graphing and performance feedback. 6. Rewards procedure (Rewards condition only). Visual analogue scale and cold read. Procedures for both the presentation of the visual analogue scale and the initial cold read of the instructional passage were the same as those described in the baseline phase. Word drill of passage vocabulary. Following the visual analogue scale and cold read of the instructional passage, each participant engaged in word drill of passage vocabulary using incremental rehearsal. This technique was selected based on the rationale that fluency would increase when word decoding was automatized; that is, if participants could automatically recognize the word as opposed to using decoding strategies, then this would increase their 27 reading accuracy and rate. Positive effects for this procedure on both reading fluency and comprehension have been demonstrated in previous research (e.g., Burns, Dean, & Foley, 2004). In the incremental rehearsal technique, students are presented with words on flashcards, 90% known, and 10% unknown. For the current study, 10 unknown words were taught per passage, systematically folded into a deck of 9 known words. The participant was presented with an unknown word on an index card, and then a known word, repeated until all 9 known words have been added in, at which point a known word was replaced with a new unknown word. The process was as follows: (a) Unknown 1; (b) Unknown 1, Known 1; (c) Unknown 1, Known 1, Known 2 … (j) Unknown 1, Known 1, Known 2, Known 3 … Known 9; (k) Unknown 1, Unknown 2, Known 2; and so on until all words unknown words have been added in. In accordance with procedures outlined by Burns et al., and Lo, Cooke, and Starling (2011), the lead researcher reviewed the instructional passages and identified 10 words expected to be more challenging for the participants (e.g., irregular or low frequency words, proper nouns such as first names or locations). These served as the “unknown” words and were interspersed with 9 words from the Dolch high frequency sight word list at the participant’s instructional grade level, which served as the “known” words. Repeated readings sequence. Following the word drills, the participant engaged in a second one-minute timed read of the instructional passage, followed by phrase-drill error correction. In this procedure, the interventionist selected approximately three to five errors that the participant made during the timed read and had them practice the word embedded within a phrase of the passage. After error correction, the participant engaged in a third timed read. Next, the interventionist introduced a modeling procedure using listening passage preview, whereby the interventionist read the passage aloud, while the participant followed along with his or her 28 finger. Periodically, the interventionist stopped, and the participant provided the word. Following this was the fourth and final timed read. Graphing and performance feedback. The participant’s progress was graphed for the first and fourth timed reads, accompanied by performance feedback. The interventionist graphed the data together with the participant, plotting the WCPM score for the first and fourth timed readings, and providing positive, specific praise and performance feedback (e.g., “You read 50 words the first time you read the passage, but after practicing a few times, you read 85 words! Excellent work!”). Goals were set for WCPM on the repeated readings based on the participants’ initial timed read. Performance goals are an important component to successful repeated reading interventions (Therrien, 2004); in previous studies using rewards and performance feedback, the repeated reading goals were modest. For example, in Chafouleas et al. (2004), participants were told they simply had to beat their initial score by 1 word. In Eckert et al. (2002) a modest goal was proposed at 5% above the initial timed read score. For the present study, a more ambitious goal of 20% above the initial timed read (for example, if the initial read was 85 WCM, the goal would be 102 WCM on the third read). This goal was selected based on the perceived importance of the participant understanding the contingency between working towards and meeting their goal, and receiving a reward. Chafouleas et al. noted that their mixed effects could have been influenced by their low reward criterion. For these reasons, a more ambitious goal was deemed necessary. For the first two participants entering the intervention phase, Lee and Ty, the reading goal was not met within any of the first two or three sessions, respectively. As such, the goal was reduced to a fixed 15 word goal above the initial timed read. Given that a reduced goal would not 29 have changed the outcome for Lee and Ty (i.e., based on their WCPM data, they still would not have met the goal for the already administered sessions), the procedures were adjusted to a +15 word goal for all participants upon their entry into the intervention phase, which is still comparatively much more ambitious than the goals set in the previous studies mentioned above. Following this adjustment, all participants met their reading goals 97% of the time across conditions (1 missed goal for Sara in Rewards condition, 1 additional for Lee in No Rewards condition, and 3 additional for Ty, with only 1 occurring in Rewards condition). Rewards procedure (Rewards condition only). The standard motivational components for the HELPS program include a star chart and performance–contingent tangible rewards based on their star chart progress. Participants are provided with a star chart and have the opportunity to receive two stars per session; they receive one star for effort, and one for meeting their WCPM goal. When the participant reaches the end of the row, they select a tangible reward from an array. In addition, there is a mystery motivator component where students can draw for “bonus stars”, indicated by shaded squares randomly placed on the chart, in order to get to the end of the row faster. For the purposes of the present study, the rewards components in HELPS were modified to include an opportunity for a tangible reward in every session that the participant’s WCPM goal was met. This is consistent with procedures used in previous research and allows for a more direct observation of the effect of rewards on WCPM scores for the purposes of research. In addition, the mystery motivator component was eliminated in an attempt to increase the salience of rewards being directly tied to meeting a performance goal. For the present study, the participant was told they would receive a reward if they met their reading goal during the session (during the rewards phase). At the end of the session, if they met their goal, they selected a 30 tangible reward from an array. Similar to the HELPS procedures, the participants also had the opportunity to select a bonus reward when they reached the end of a row on the star chart. To select the rewards that were included in the array, participants completed an individual informal preference assessment prior to baseline. Each participant was asked to provide a list of items they would like to work for. The lead researcher then compiled the lists and selected at least two items from each participant’s list to include in the array. The highest frequency items endorsed were unique school supplies (e.g., character or sports team branded pencils and notebooks, scented erasers; 100% of participants), Pokemon cards (67% of participants), stress balls (67% of participants), and scented markers (50% of participants). For a complete list of requested reward items, see Appendix B. Interventionist training. The lead researcher/interventionist was a Doctoral student in School Psychology with substantial experience conducting reading fluency interventions for both clinical and research purposes, including previous experience with the HELPS program. Two additional interventionists were graduate students in School Psychology, all with previous experience in reading fluency assessment and intervention. Training in the specific procedures for the present study was conducted in a workshop format facilitated by the lead researcher, including review of training manuals, didactic training, modeling, video scoring practice, and rehearsal with performance feedback. The interventionists were required to meet 100% fidelity on the intervention components prior to commencement of the study. The interventionists were also observed by the lead researcher for the first three weeks of the study to ensure fidelity. Treatment fidelity. During the intervention, all interventionists followed a step-by-step intervention protocol and recorded any missed steps on a corresponding tracking sheet. As a secondary measure of treatment fidelity, the sessions were audio recorded and 30% were 31 reviewed across phases for each participant. The average percentage of steps completed was 99.5% (range 94% to 100%). Inter-scorer agreement. To ensure accurate scoring of the dependent variable, the baseline and intervention sessions were audio recorded. An independent secondary rater reviewed 30% of sessions across phases for each participant. Inter-scorer agreement was calculated by the following formula: lower WCPM score / higher WCPM score x 100. The resulting mean agreements were 98.8 % for instructional passages (range 88% to 100%), 96.3% for generalization passages (range 83% to 100%), and 95.5% for comprehension probes (range 83% to 100%), meeting the criteria for acceptable agreement (Kratochwill et al., 2013). Research Design A multiple-baseline multiple-sequence across participants design (MBL-MS; Noell & Gresham, 2001) was used. In this design, two independent MBL designs, each with three participants, are interpreted both separately and together to allow the comparison of all possible sequences of the treatments. Doing so allows for the examination of any sequence effects. The study was intended to follow a concurrent design, but due to student absences, the resulting design is non-concurrent. In particular, due to multiple absences, both Lee and Phineas entered the first intervention phase a week delayed. Additionally, as noted on the graphs, two students had extended absences during the intervention phases; Ty was absent for two weeks at the phase change between intervention conditions, and Sara was absent for one week just before the last day of data collection. Importantly, all participants started and completed the study within the same 3-month time frame. For simplicity, the data in the graphs are presented concurrently. The intervention schedule was three sessions per week; if a student was absent on a scheduled day, efforts were made to do the session on a different day the same week. If this was 32 not possible, the session was made up the immediately following week. There were three phases for each MBL: (a) baseline, (b) intervention with rewards (“Rewards” condition), and (c) intervention with no rewards (“No Rewards” condition). The sequence of treatment during intervention was A-B-C (Sequence A) and A-C-B (Sequence B), where Baseline = A, Rewards = B, and No Rewards = C. Participants were randomly assigned to a baseline phase and treatment sequence, with the constraint that both sequences were assigned in equal numbers (Noell & Gresham, 2001). Baseline and treatment phase lengths were determined a priori; each baseline included a minimum of three sessions, at which point the first treatment phase was introduced for the first participant in Sequence A (Rewards), and Sequence B (No Rewards). The phase change for the second and third participants in each sequence occurred after one or two additional weeks (six or nine sessions), respectively. Temporally, this equates to one-, two-, and three-week long baseline phases, respectively. A priori baseline phase lengths were selected for both theoretical and practical reasons. First, given that the participants selected were struggling readers, it was expected that little to no discernable skill growth would be demonstrated in the absence of intervention. In addition, the primary interest of the study was to compare the intervention (B vs. C) phases across sequences. Though essential for documenting control, comparison of A vs. B/C phases holds comparatively less utility for understanding whether the use of rewards produces incremental effects to the intervention without rewards. Finally, from a practical standpoint, baseline and intervention phase lengths were constrained by the number of available reading probes and passages, and baseline lengths could not be extended without extending the intervention phases, which was not possible given the available time before the end of the school year. 33 Intervention phase lengths were also determined a priori. Participants received a total of eight weeks of intervention; four weeks for each phase. This equated to a total of 24 data points for the instructional passages for the entire duration, 12 each for each phase. For the generalization and comprehension probes, this equates to 16 and 8 data points for the intervention duration, respectively (eight and four per phase). Data Analysis In terms of interpretation, visual analysis was used as the primary method to assess effects, with supportive information from effect size estimates. Within-phase means, standard deviations and slope estimates are provided as descriptive information only. In order to confirm a basic effect, the finding must first be derived from the visual analysis, and be supported by a statistically significant effect size estimate. Further details on each analysis method are described below. Visual analysis. Visual analysis was used to analyze the data, in accordance with the WWC Standards for Single Case Research, for evidence of a functional relation, that is, at least three demonstrations of the treatment effect (i.e., a “basic effect”), at three different points in time (Kratochwill et al., 2013). First, the data were analyzed for within-phase consistency in level (mean score on outcome within-phase), trend (slope of outcome score over time within-phase), and variability (distribution of outcome scores about the mean within-phase). Next, the data were analyzed across phases to assess the immediacy of effect from phase to phase, the consistency of the effect, and overlap of data across phases. In MBL designs, vertical analysis provides the assessment of replication across subjects. Additionally, vertical analysis is important in order to locate any anomalies in the data that may indicate a threat to internal validity related to history. In a MBL-MS design, Sequences A and B are interpreted separately, 34 as two distinct MBLs. However, visual analysis and statistical analysis (described below) are compared descriptively across the sequences, in order to draw inferences about potential sequence effects (Noell & Gresham, 2001). Descriptives and inferential statistics. To supplement the visual analysis, descriptive statistical analyses were used to compare trends within and across phases, including: mean WCPM by student across phases and measures, and ordinary least squares (OLS) regression slope estimates to provide a measure of overall growth rates in WCPM per week within phases. In addition, paired sample t-tests were conducted to analyze the mean differences between the first and fourth reads on the instructional passages within the intervention sessions. Effect size. The primary effect size measure used for the current study was a variant of Kendall’s (1962) rank-order correlation, Tau, that concurrently accounts for non-overlapping data and improving intervention (phase b or c) trend: τA vs B+trend B (Parker, Vannest, Davis, & Sauber, 2011). The baseline data were first analyzed for statistically significant positive trend, which, if found, was corrected prior to the Tau analyses (Tarlow, 2016). When baseline data were corrected, the Tau variant was then: τA vs B+trend B - trend A (the cases for which this variant was used are identified in the Results section). For each reading fluency measure, individual effect size estimates were also combined to analyze the weighted mean effects within each intervention phase relative to baseline across participants in each sequence (e.g., for Sequence A: Baseline to Rewards; Baseline to No Rewards; and reversed for Sequence B). Weighted mean effect sizes were also calculated for each intervention phase to assess differentiation between the reward conditions (i.e., rewards condition compared to no rewards condition), across participants in each sequence. Percent of non-overlapping data (PND) is provided as an additional effect size measure (Tarlow & Penland, 2016). 35 Exploratory statistical analysis. The effect size estimates (Tau values) were analyzed within a meta-regression for each reading fluency dependent variable across conditions and sequences to test for potential differences in condition (Rewards or No Rewards), order (rewards first or second), and interactions of condition by order relative to baseline. A second meta-regression was conducted for the comparison of the intervention phases (i.e., rewards versus no rewards). Importantly, these analyses are deemed exploratory only, given the extremely limited sample size. 36 Results The results for each dependent variable are discussed below, by measure, sequence, and participant. For clarity, “condition” refers to the version of the intervention in the phase (i.e., with rewards or without rewards); Condition B is the intervention plus rewards, and Condition C is the intervention without rewards. The intervention phases are referred to as phase 1 (first intervention phase), or phase 2 (second intervention phase), across sequences. Baseline refers to the pre-intervention phase. The results will be discussed by measure. The first section addresses direct gains on instructional passages (i.e., the fourth read of the passage during intervention). The second section addresses between-session gains on instructional passages (i.e., the first read of each new instructional passage). The third section will address generalized gains on a novel reading passage. Following the reading fluency measures, reading comprehension and self-reported motivation are discussed. For each measure, a number of different comparisons are addressed. First, in order to determine the effectiveness of the intervention overall, regardless of whether or not rewards are used, visual analysis was conducted across both intervention conditions. Importantly, the primary analysis of this comparison concerns analysis of the first intervention condition relative to baseline; this is a strength of the MBL-MS design, which allows for a direct comparison of both intervention conditions to baseline given that Sequences A and B begin with opposite intervention conditions, allowing for detection of potential sequence effects. Although, under optimal conditions, designs would include a return to baseline following the first intervention phase, this is not possible for reading fluency data, which is acquired gradually across time, and is therefore not amenable to reversals. Moreover, the primary reason for this comparison is to compare across both intervention phases relative to baseline in order to 37 determine the overall effectiveness of the intervention, regardless of rewards, which is not the primary research question for this study, which concerned differentiation between the two intervention conditions. Therefore, decisions for the differential effectiveness of rewards were not made based on the comparisons to baseline, but by both intervention conditions compared to each other, as described below. In terms of effect size analysis, the values of Tau for the second intervention phase relative to baseline are provided simply for adjunct analysis, and were required for use in the exploratory statistical analyses (discussed in the following sections), in which an analysis of all possible sequences of conditions is needed in order to evaluate possible sequence effects. Next, the Rewards and No Rewards conditions are compared to each other in each sequence, which does not include a comparison to baseline. This comparison represents the test of differential effectiveness of rewards (i.e., was reading performance better in one condition relative to the other condition) by directly comparing the two conditions (Sequence A: Rewards to No Rewards; Sequence B: No Rewards to Rewards). Direct Gains on Instructional Passages When comparing both intervention phases to baseline, direct gains are based on the WCPM on the final (4th) read of the instructional passages during the intervention phases compared to the baseline probes. When comparing differences between the intervention phases, direct gains are based on the WCPM of the final (4th) reads in intervention phase 1 compared to intervention phase 2. The graphical representation for the data is presented in Figures 1 (Sequence A), and 2 (Sequence B). Means, standard deviations, and slope estimates for both sequences are presented in Table 1. Effect sizes and weighted mean effect sizes for both 38 sequences are presented in Table 2. Percent of non-overlapping data (PND) is presented for both sequences in Appendix C. Sequence A. Participants in Sequence A began the intervention in the rewards condition, and changed to the no rewards condition in the second intervention phase. Lee. As seen in Figure 1, Lee presented with variable baseline that ended on a descending trend. He demonstrated an immediate response to intervention into the rewards condition following baseline, with an observed change in level and variable improving trend, also supported via statistical analysis of effect size (τ Baseline to Rewards = 0.688, p < .001). In the no rewards condition, Lee’s performance continued to exceeded baseline levels, but his trend initially decreased, then stabilized, with an increasing trend towards the end of the phase. Given that his performance exceeded baseline levels across both intervention phases, which converges with effect size analysis, a basic effect for the intervention compared to baseline was established. In terms of differentiation between the rewards and no rewards conditions, visual analysis indicated an overall flatter (albeit with an improving trend emerging at the end of the phase), but still variable, data path in the no rewards condition compared to the rewards condition. With respect to slope analysis, Lee’s rate of change was 7.25 WCPM in the rewards condition, compared to only .97 WCPM per week in the no rewards condition. However, the effect size for the comparison between the conditions was negligible and non-significant (τ Rewards to No Rewards = 0.008, ns). Given that the visual analysis and effect size did not converge, a basic effect for differentiation between the rewards conditions was not established. 39 Figure 1. Direct gains on instructional passages, Sequence A; *a one-week absence occurred after this data point. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 40 Table 1 Descriptive statistics for direct gains on instructional passages across conditions and participants for both sequences Sequence A Participant Mean WCPM Baseline OLS Slope Baseline Condition Mean WCPM Rewards Condition OLS Slope Rewards Condition Mean WCPM No Rewards Condition OLS Slope No Rewards Condition Lee 94 (10.25) -6.36 142.83 (13.62) 7.25 147.33 (10.90) 0.97 Sara 113.75 (10.32) 9.57 148.92 (16.99) -5.60 131.08 (15.23) -6.37 Marita 34.45 (10.06) -3.38 72.58 (12.90) 1.57 73.5 (9.50) 1.61 Sequence B Participant Mean WCPM Baseline OLS Slope Baseline Mean WCPM No Rewards Condition OLS Slope No Rewards Condition Mean WCPM Rewards Condition OLS Slope Rewards Condition Ty 92.8 (9.20) -12.28 107.58 (21.93) 9.77 140.33 (15.77) 6.50 Chris 48.88 (8.00) 4.54 86.75 (9.79) -1.27 80.92 (10.03) 4.52 Phineas 56.82 (5.34) -0.60 90.58 (8.10) 1.02 99.83 (8.44) -2.68 Note. Standard deviation in parentheses; OLS Slope estimates scaled at change in WCPM per week. 41 Table 2 Tau values and weighted mean effect sizes for direct gains across participants and conditions for both sequences Sequence A Participant Tau Baseline to Rewards Tau Baseline to No Rewards2 Tau Rewards to No Rewards Lee .688 (.062)*** .504 (.088)* .008 (0.083) Sara .034 (.100) 1 -.559 (.069)*** 1 .309 (0.075)* Marita .575 (.058)*** .615 (0.054)*** -.021 (0.083) Weighted Mean Effect Size .537 (0.039)*** .231 (0.038)*** .113 (0.047)* Sequence B Participant Tau Baseline to No Rewards Tau Baseline to Rewards2 Tau No Rewards to Rewards Ty .336 (0.104) .667 (0.065)*** .575 (0.056)*** Chris .493 (0.076)** .695 (0.052) *** -.034 (0.083) Phineas .628 (0.053)*** .554 (0.060)*** .334 (0.074) Weighted Mean Effect Size .548 (.040)*** .644 (.034)*** .372 (.039)*** Note. Standard error in parentheses; Significance codes: *** p < 0.001; ** p < 0.01; * p < 0.05 1Tau variation: τA vs B+trend B - trend A correcting for statistically significant baseline trend 2 Value calculated for exploratory statistical analyses, and not used for primary inferences Sara. As seen in Figure 1, Sara presented with an improving baseline that was statistically significant based on the pre-analysis Tau calculation for baseline trend. At the point of intervention to the rewards condition, Sara presented with an immediate change in level with a variable slight improving trend, and a steep decline at the end of the phase. However, given her improving baseline, her effect size analysis revealed a negligible effect that was nonsignificant between the rewards condition and baseline, after correcting for baseline trend (τ Baseline to Rewards = 0.034, ns). Similarly, although her performance trend in the second intervention phase when rewards were removed did not follow the steep decline of the previous phase, the overall trend 42 was negative. Overall, although an initial response to intervention in the rewards phase was observed visually by a change in level, the corresponding effect size was nonsignificant. Alongside the deteriorating trend in the no rewards condition, Sara’s overall pattern of data does not support a basic effect for the intervention compared to baseline. In terms of differentiation between the rewards and no rewards conditions, visual analysis indicated an overall decreased level when she moved into the no rewards condition, which corresponded to observed decreases in her average WCPM for each condition (mean WCPM for rewards condition = 148.92, SD = 16.99; mean WCPM in no rewards condition = 131.08, SD = 15.23), albeit with some overlap with the previous phase. The corresponding effect size indicated a small positive effect (τ Rewards to No Rewards = 0.309, p = .05), suggesting that Sara appeared to perform better in the rewards condition relative to what would be expected based on her performance in the no rewards condition. Taken together, although Sara’s performance in the no rewards condition decreased in level relative to the rewards condition, her performance had already begun to deteriorate at the end of the rewards condition. This complicates casual inferences regarding the change in conditions; as such, a basic effect was not established for differentiation between reward conditions. Marita. As seen in Figure 1, Marita presented with a variable baseline that ended on a deteriorating trend. She had immediate response to intervention upon entry to the intervention phase, observed by a highly variable but sustained change in level across both intervention conditions compared to baseline, supported by a positive effect size for the initial entry to the rewards phase (τ Baseline to Rewards = 0.575, p < .001). A small improving trend began to establish near the end of the second intervention phase (no rewards), but generally there were no other discernable trends in her data path. Given that her performance in the intervention conditions 43 was above baseline and supported by a positive effect size, a basic effect for the intervention compared to baseline was established. In terms of differentiation between the rewards conditions, Marita’s performance in both conditions was variable, with little discernable trend until the end of the second phase, and complete overlap with each other, supported by a negligible and non-significant effect size (τ Rewards to No Rewards = -0.021, ns); thus, a basic effect for differentiation between the rewards conditions was not established. Vertical analysis. In summary, the results of Sequence A reflect two basic effects (Lee and Marita) for the intervention overall (with or without rewards) on WCPM on instructional passages (direct gains) relative to baseline. Given that there were only two basic effects in favour of the intervention, a functional relation was not established for Sequence A. With respect to differentiation between the reward conditions, no basic effects were established, indicating no observed differences between WCPM performance when rewards were used versus when they were not. Therefore, no functional relation for differentiation between reward conditions was established for Sequence A. Sequence B. Participants in Sequence B began the intervention in the no rewards condition, and changed to the rewards condition in the second intervention phase. Ty. As seen in Figure 2, Ty presented with a generally stable baseline trend following an initially higher data point. He had a variable response to intervention entering the first intervention phase, with complete overlap with baseline until a variable but improving trend established near the middle of the phase. His corresponding effect size was small and non-significant (τ Baseline to No Rewards = 0.336, ns). Despite an initial decrease in level, Ty’s improving trend continued into the second intervention phase, with a variable pattern emerging in the 44 middle of the phase; however, his performance remained well above baseline throughout the phase. Taken together, although Ty’s performance in the first intervention phase had high overlap with baseline and a non-significant effect size, his trend began to improve near the end of the phase, and generally continued into the second intervention phase, which was supported by a significant effect size; thus, a delayed basic effect for the intervention overall compared to baseline was established. With respect to differentiation between the reward conditions, although his between-condition effect size was significant (τ No Rewards to Rewards = 0.575, p < .001), visual analysis indicated a continuation of trend from the no rewards condition to the rewards condition (despite an initial decrease in level in the second phase); thus, a basic effect for the differentiation between rewards conditions was not supported. Chris. As seen in Figure 2, Chris presented with a slight improving baseline trend. He had an immediate change in level at the point of intervention to the no rewards condition, with a variable, slight negative trend emerging near the end of the phase. His corresponding effect size relative to baseline was small to moderate and significant (τ Baseline to No Rewards = 0.493, p = 0.004). Though initially variable at the phase change to the rewards condition, his performance stabilized with an improving trend, and remained above baseline levels with the exception of one data point. Given his improved and sustained level across both conditions compared to baseline, which converges with effect size analysis, a basic effect for the intervention across conditions compared to baseline was established. With respect to differentiation between the rewards conditions, visual analysis indicated complete overlap of data between conditions, supported by a negligible and nonsignificant effect 45 size (τ No Rewards to Rewards = -0.034, ns); thus, a basic effect for differentiation between rewards conditions was not established. Phineas. As seen in Figure 2, Phineas presented with a relatively stable baseline following initial variability. He had an immediate response to intervention in the no rewards condition, with an initially variable but overall positive change in level and a slight positive trend. His corresponding effect size was moderate and positive compared to baseline (τ Baseline to No Rewards = 0.628, p = < .001). His performance in the rewards condition remained above baseline and had no discernable trend. Given his positive and sustained change in level across both intervention phases, alongside converging effect size analysis, a basic effect for the intervention compared to baseline was established. With respect to differentiation between rewards conditions, visual analysis indicated little discernable change in level between the intervention phases, alongside some overlapping data. Correspondingly, the effect size analysis indicated a small effect that was non-significant for his performance in the rewards condition compared to the no rewards condition (τ No Rewards to Rewards = .334, ns); as such, a basic effect for differentiation between the rewards conditions was not established. 46 Figure 2. Direct gains on instructional passages, Sequence B; * a two-week absence occurred after this data point. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 47 Vertical analysis. In summary, the results of Sequence B reflect basic effects for all three participants (including Ty’s delayed basic effect) for the intervention overall (with or without rewards) improving WCPM on instructional passages (direct gains) relative to baseline. Given that three participants presented with a basic effect, a functional relation was established for Sequence B. With respect to differentiation between the reward conditions, no basic effects were established, indicating no observed differences between WCPM performance when rewards were used versus when they were not. Therefore, no functional relation for differentiation between reward conditions was established for Sequence B. Summary for direct gains on instructional passages. In summary, for direct gains on the instructional passages, two basic effects were observed in Sequence A and three in Sequence B. Therefore, there was an overall functional relation for the intervention in Sequence B, regardless of whether rewards were used, and no functional relation in Sequence A. When considered across both MBLs, the intervention yielded direct gains on WCPM for instructional passages for five out of six participants. When considering differential effects between reward conditions, no basic effects were observed in either sequence; thus, no functional relations for differentiation between reward conditions were established. Between-Session Gains on the First Read of Instructional Passages When comparing both intervention phases to baseline, between-session gains are based on the WCPM on the first read of the instructional passages, compared to the baseline probes. When comparing differences between the intervention phases, between-session gains are based on the WCPM on the first read of the instructional passages in intervention phase 1 compared to 48 intervention phase 2. The graphical representation for the data is presented in Figures 3 (Sequence A), and 4 (Sequence B). Means, standard deviations, and slope estimates (change in WCPM per week) for both sequences are presented in Table 3. Effect sizes and weighted mean effect sizes are presented in Table 4. Percent of non-overlapping data (PND) is presented in Appendix C. Sequence A. Participants in Sequence A began the intervention in the rewards condition, and changed to the no rewards condition in the second intervention phase. Lee. As seen in Figure 3, Lee presented with a variable baseline ending on a deteriorating trend. Upon entry into the intervention phase, he had an immediate response to intervention indicated by a positive change in level. His performance was variable with little discernable trend, and a steep but brief decline mid-phase. The corresponding effect size was small and not significant (τ Baseline to Rewards = 0.295, ns). In the no rewards condition, his performance remained above baseline levels overall, with a variable but overall improving trend. Taken together, although visual analysis indicated Lee’s performance in the first intervention phase was above baseline levels aside from a brief decline mid-phase, this was not supported by the effect size analysis. However, the change in level maintained into the second intervention phase, thus rendering a delayed basic effect for the intervention overall when compared to baseline. When comparing his performance between the rewards and no rewards conditions, there was a visual differentiation with respect to trend; specifically, Lee’s trend in the no rewards condition showed a clearer, albeit still variable, trend compared to the indistinguishable trend in the rewards condition. Corresponding, the effect size for between the conditions was moderate and negative (τ Rewards to No Rewards = -0.574, p < .001), after accounting for trend in the no rewards phase, indicating Lee’s performance in the rewards phase was less than expected relative to his 49 performance in the no rewards phase. However, there was little differentiation in level between the conditions, with complete overlap; thus, no basic effect for differentiation between the rewards conditions was established. Table 3 Descriptive statistics for between-session gains on the first read of instructional passages across participants and conditions for both sequences Sequence A Participant Mean WCPM Baseline OLS Slope Baseline Condition Mean WCPM Rewards Condition OLS Slope Rewards Condition Mean WCPM No Rewards Condition OLS Slope No Rewards Condition Lee 94.00 (10.25) -6.35 111.33 (22.10) 2.80 115.67 (12.35) 5.85 Sara 113.75 (10.32) 9.57 117.58 (20.64) -11.21 94.75 (21.44) -7.16 Marita 34.45 (10.06) -3.38 38.08 (13.09) 1.52 41.50 (10.16) 3.05 Sequence B Participant Mean WCPM Baseline OLS Slope Baseline Mean WCPM No Rewards Condition OLS Slope No Rewards Condition Mean WCPM Rewards Condition OLS Slope Rewards Condition Ty 92.80 (9.20) -12.28 90.67 (20.87) -6.11 93.92 (17.19) -1.31 Chris 48.88 (8.00) 4.54 50.58 (9.57) -4.83 54.25 (9.58) 3.88 Phineas 56.82 (5.34) -0.60 51.17 (11.69) 1.03 64.25 (9.72) -6.16 Note. Standard deviation in parentheses; OLS Slope estimates scaled at change in WCPM per week. 50 Table 4 Effect size (Tau) values and weighted mean effect sizes for between-session gains on the first read of instructional passages across participants and conditions for both sequences Sequence A Participant Tau Baseline to Rewards Tau Baseline to No Rewards2 Tau Rewards to No Rewards Lee .295 (.107) .662 (.066)*** -.574 (.056)1*** Sara -.536 (.072)** 1 -.707 (.050)*** 1 .184 (.081) Marita .150 (.085) .267 (.081) -.056 (.083) Weighted Mean Effect Size -.140 (.049)** -.115 (.036)** -.264 (.040)*** Sequence B Participant Tau Baseline to No Rewards Tau Baseline to Rewards2 Tau No Rewards to Rewards Ty -.054 (.117) -.078 (.117) .042 (.083) Chris -.086 (.099) .369 (.086)* .227 (.079) Phineas -.145 (.085) 0.154 (.085) .188 (.080) Weighted Mean Effect Size -.105 (.057) .188 (.054)*** .156 (.047)*** Note. Standard error in parentheses; Significance codes: *** p < 0.001; ** p < 0.01; * p < 0.05 1Tau variation: τA vs B+trend B - trend A correcting for statistically significant baseline trend 2 Value calculated for exploratory statistical analyses, and not used for primary inferences Sara. As seen in Figure 3, Sara presented with an improving baseline that was statistically significant based on pre-analysis Tau calculation for baseline trend. Her performance changed to a variable but overall negative trend at the point of intervention, with a continued highly variable decline across intervention phases. This is supported by a moderate negative effect size for the initial intervention phase compared to baseline (τ Baseline to Rewards = -0.536, p = .002). Given her deteriorating trend across both phases, alongside a significant negative effect size, Sara presented with a negative basic effect for the intervention when compared to baseline. 51 With respect to differentiation between the conditions, given that her performance showed a deteriorating trend across conditions, and the corresponding effect size was negligible and non-significant (τ Rewards to No Rewards = 0.184, ns), a basic effect for differentiation between the reward conditions was not established. Marita. As seen in Figure 3, Marita presented with a variable baseline that ended on a deteriorating trend. At the point of intervention, Marita presented with highly variable performance with no discernable trend, and levels similar to her baseline responding across both intervention phases. This is supported by a non-significant effect size for the initial intervention condition relative to baseline (τ Baseline to Rewards = 0.150, ns). Therefore, her data do not support a basic effect for the intervention overall when compared to baseline. With respect to differentiation between the reward conditions, Marita’s performance across the conditions was similarly variable with complete overlap, alongside a negligible non-significant effect size (τ Rewards to No Rewards = -0.056, ns), thus not rendering a basic effect for differentiation between conditions. 52 Figure 3. Between session gains on first read of instructional passage, Sequence A; * a one week absence occurred after this data point. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 53 Vertical analysis. In summary, the results of Sequence A indicate one basic effect (Lee) for the intervention overall (with or without rewards) improving between-session WCPM gains on instructional passages relative to baseline. Importantly, Lee’s effect was delayed, emerging in the second intervention phase. Sara presented with a negative basic effect, and Marita did not show a basic effect; therefore, there was no functional relation for the intervention on between-session gains was established for Sequence A. With respect to differentiation between the reward conditions, no basic effects were established, indicating no observed differences between WCPM performance when rewards were used versus when they were not. Therefore, no functional relation for differentiation between reward conditions was established for Sequence A. Sequence B. Participants in Sequence B began the intervention in the no rewards condition, and changed to the rewards condition in the second intervention phase. Ty. As seen in Figure 4, Ty presented with a relatively stable baseline following an initially higher data point. He had a highly variable response to intervention upon entry to the no rewards condition, which continued into the rewards condition. The corresponding effect size for the initial intervention condition relative to baseline was negligible and nonsignificant (τ Baseline to No Rewards = -0.054, ns). Given the near-complete overlap with baseline, highly variable data, and a non-significant effect size, a basic effect for the intervention overall compared to baseline was not established. With respect to differentiation between the reward conditions, given the complete overlap between the intervention phases alongside a negligible non-significant effect size (τ No Rewards to Rewards = 0.042, ns), a basic effect for differentiation between reward conditions was not established. 54 Chris. As seen in Figure 4, Chris presented with a slight improving baseline trend observed visually, that was not statistically significant based on the pre-analysis Tau calculation for baseline trend. His performance level initially continued at the point of intervention, and then deteriorated to a variable and declining trend throughout the first phase (τ Baseline to No Rewards = -0.086, ns). At the phase change to the rewards condition, his performance had high overlap with the previous phase, but showed increased stability and a slight positive trend emerging mid-phase and continuing until the end. However, given that the data continued to show high overlap with baseline, a basic effect was not supported. With respect to differentiation between the reward conditions, given the high overlap between the intervention phases alongside a small non-significant effect size (τ No Rewards to Rewards = .227, ns), a basic effect for differentiation between reward conditions was not established. 55 Figure 4. Between session gains on first read of instructional passage, Sequence B; * a two-week absence occurred after this data point. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 56 Phineas. As seen in Figure 4, Phineas presented with a generally stable baseline trend following initial variability. At the point of intervention to the no rewards condition, he showed variable performance with no discernable improvement from baseline (τ Baseline to No Rewards = -0.145, ns). At the phase change to the rewards condition, there was an initial change in level, which then deteriorated back to near baseline levels. Given the high overlap with baseline in both phases, a basic effect for the intervention compared to baseline was not established. With respect to differentiation between the reward conditions, given the high overlap between the intervention phases alongside a small non-significant effect size (τ No Rewards to Rewards = .188, ns), a basic effect for differentiation between conditions was not established. Vertical analysis. In summary, the results of Sequence B indicated no basic effects for the intervention overall (with or without rewards) improving between-session WCPM gains on instructional passages relative to baseline. Therefore, there was no functional relation established for Sequence B. With respect to differentiation between the reward conditions, no basic effects were established, indicating no observed differences between WCPM performance when rewards were used versus when they were not. Therefore, no functional relation for differentiation between reward conditions was established for Sequence A. Summary for between-session gains on first read of instructional passages. In summary, for between-session gains on the instructional passages, an overall functional relation for the intervention, regardless of whether rewards were used, was not established. For Sequence A, one delayed basic effect (Lee) emerged. For Sequence B, no basic effects were established. When considering the differential effects of the rewards versus no rewards conditions, no basic effects emerged in either sequence. 57 Within-session Gains on Instructional Passages The differences between the first and fourth read of the instructional passages were calculated to analyze the gains in WCPM occurring within the intervention sessions. As seen in Appendix D (Figures 17 and 18), there was clear separation for the majority of participants within each intervention session, indicating improvement in WCPM on the same instructional passage within a session. Correspondingly, the results of the paired-sample t-tests were all statistically significant (see Table 5). For Ty, the pattern of his data in the no rewards condition (phase 1) was visually quite different than the rewards condition (phase 2). Specifically, there was little initial separation between the first and fourth reads in phase 1, with two data points for the fourth read falling below the first read. Clearer separation emerged near the end of the phase, with a markedly different and clear separation pattern occurring in the rewards condition (phase 2). 58 Table 5 Paired samples t-test for first to fourth reads on instructional passages in intervention phase across participants and conditions for both sequences Sequence A Participant 1st to 4th Read Rewards Phase t-test 1st to 4th Read No Rewards Phase t-test Lee t(11) = 5.61*** t(11) = 7.91*** Sara t(11) = 6.33*** t(11) = 9.54*** Marita t(11) = 9.02*** t(11) = 7.19*** Sequence B Participant 1st to 4th Read No Rewards Phase t-test 1st to 4th Read Rewards Phase t-test Ty t(11) = 2.44* t(11) = 7.38*** Chris t(11) = 11.75*** t(11) = 12.61*** Phineas t(11) = 12.20*** t(11) = 15.04*** Note. Degrees of freedom in parentheses; Significance codes: *** p < 0.001; * p < 0.05 Generalized Gains When comparing both intervention phases to baseline, generalized gains are based on the WCPM on a single read of a novel passage from the CBMReading passage set compared to the corresponding baseline probes. When comparing the differential effectiveness of the reward conditions, the generalized gains are based on the WCPM of the generalization probe in intervention phase 1 compared to intervention phase 2. The graphical representation for the data is presented in Figures 5 (Sequence A), and 6 (Sequence B). Means, standard deviations, and slope estimates (change in WCPM per week) for both sequences are presented in Table 6. Effect sizes and weighted mean effect sizes are presented in Table 7. Percent of non-overlapping data (PND) is presented in Appendix C. 59 Table 6 Descriptive statistics for generalized gains on CBMReading passages across participants and conditions for both sequences Sequence A Participant Mean WCPM Baseline OLS Slope Baseline Condition Mean WCPM Rewards Condition OLS Slope Rewards Condition Mean WCPM No Rewards Condition OLS Slope No Rewards Condition Lee 94.25 (7.68) 11.80 104.5 (13.10) 10.10 111.88 (9.48) 4.79 Sara 115.33 (8.80) 7.31 110.75 (13.90) -5.86 94.88 (10.38) -5.12 Marita 28.00 (3.66) -0.29 35.50 (10.73) -2.86 32.50 (9.52) -2.49 Sequence B Participant Mean WCPM Baseline OLS Slope Baseline Mean WCPM No Rewards Condition OLS Slope No Rewards Condition Mean WCPM Rewards Condition OLS Slope Rewards Condition Ty 79.25 (7.63) 9.00 87.75 (22.15) -8.26 87.38 (12.58) 2.31 Chris 40.83 (7.63) 0.74 50.75 (6.54) 0.38 44.75 (6.09) 3.19 Phineas 44.88 (8.90) 6.02 54.13 (6.24) 1.93 60.38 (16.64) -3.02 Note: Standard deviation in parentheses; OLS Slope estimates scaled at change in WCPM per week Sequence A. Participants in Sequence A began the intervention in the rewards condition, and changed to the no rewards condition in the second intervention phase. Lee. As seen in Figure 5, Lee presented with a clear improving baseline that was not statistically significant based on the pre-analysis Tau calculation for baseline trend. At the point of intervention, the improving trend from baseline continued despite an initial small decrease in level, and maintained throughout the rewards condition. At the point of intervention into the no 60 rewards condition, there was an initial decline in level, which stabilized to a generally positive but flatter trend compared to the baseline and rewards phase trends. Across phases, Lee showed improvement in average WCPM from baseline (M = 95.25, SD = 7.68) to the final phase (M = 111.88, SD = 9.48). However, although the effect sizes for the initial intervention condition was moderate, positive, and statistically significant compared to baseline (τ Baseline to Rewards = 0.657, p = 0.01), his initial improving baseline trend complicates the Tau analysis, and prohibits causal inferences about the intervention’s effectiveness. Therefore, a basic effect for the intervention overall compared to baseline was not established. With respect to differentiation between the reward conditions, although Lee’s trend in the no rewards condition was flatter in comparison to the rewards condition, there was complete overlap between the conditions, alongside a negligible nonsignificant effect size (τ Rewards to No Rewards = 0.039, ns); thus, a basic effect for differentiation between reward conditions was not established. 61 Table 7 Effect size (Tau) values and weighted mean effects for generalized gains on CBMReading passages across participants and conditions for both sequences Sequence A Participant Tau Baseline to Rewards Tau Baseline to No Rewards2 Tau Rewards to No Rewards Lee .657 (.095)** .623 (.102)** .039 (.125) Sara -.195 (.137) -.657 (.081)** .268 (.116) Marita .270 (.116) .322 (.112) .105 (.124) Weighted Mean Effect Size .347 (.065)*** -.043 (0.055) .144 (.070)* Sequence B Participant Tau Baseline to No Rewards Tau Baseline to Rewards2 Tau No Rewards to Rewards Ty -.095 (.165) .244 (.157) -.058 (.125) Chris .430 (.116)* .401 (.120) -.212 (.119) Phineas -.590 (.081)**1 -.133 (.123)1 .105 (.124) Weighted Mean Effect Size -.232 (.062)*** .164 (.075)* -.059 (.071) Note. Standard error in parentheses; Significance codes: ***p < 0.001; **p < 0.01; *p < 0.05 1Tau variation: τA vs B+trend B - trend A correcting for statistically significant baseline trend 2Value calculated for exploratory statistical analyses, and not used for primary inferences Sara. As seen in Figure 5, Sara presented with an improving baseline trend, that was not statistically significant based on the pre-analysis Tau calculation for baseline trend. Her performance deteriorated to a highly variable but overall deteriorating trend at the point of intervention, rendering a negligible non-significant effect size (τ Baseline to Rewards = -0.195, ns). Her trend stabilized somewhat at the phase change to the no rewards condition, but her performance level was completely below baseline. Given that her performance deteriorated from baseline across phases, there was a negative basic effect established for the intervention. 62 With respect to differentiation between the reward conditions, although Sara’s trend in the no rewards condition was flatter in comparison to the rewards condition, there was near complete overlap between the conditions, alongside a small non-significant effect size (τ Rewards to No Rewards = .268, ns); thus, a basic effect for differentiation between reward conditions was not established. Marita. As seen in Figure 5, although Marita presented with a generally stable baseline that ended on a small improving trend. She demonstrated slight improvement at the point of intervention, indicated by a small change in level, but her performance was variable, and the level was not maintained. The corresponding effect size was small and nonsignificant (τ Baseline to Rewards = 0.270, ns). Marita’s performance in the no rewards condition remained variable and with moderate overlap with the previous phase. Given that her initial change in level was not maintained, alongside a small nonsignificant effect size, an overall basic effect for the intervention compared to baseline was not established. With respect to differentiation between the reward conditions, there was complete overlap between the conditions, alongside a negligible nonsignificant effect size (τ Rewards to No Rewards = .105, ns); thus, a basic effect for differentiation between reward conditions was not established. Vertical analysis. In summary, the results of Sequence A indicated no basic effects for the intervention overall (with or without rewards) improving WCPM on generalization passages relative to baseline. Sara presented with a negative basic effect. Therefore, no functional relation for the intervention across conditions compared to baseline was established for Sequence A. With respect to differentiation between the reward conditions, no basic effects were established, indicating no observed differences between WCPM performance when rewards were 63 used versus when they were not. Therefore, no functional relation for the differentiation between reward conditions was established for Sequence A. 64 Figure 5. Generalized gains on CBMReading passages, Sequence A; * a one-week absence occurred after this data point. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 65 Figure 6. Generalized gains on CBMReading passages, Sequence B. * a two-week absence occurred after this data point. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 66 Sequence B. Participants in Sequence B began the intervention in the no rewards condition, and changed to the rewards condition in the second intervention phase. Ty. As seen in Figure 6, Ty presented with an improving baseline trend that was not statistically significant based on the pre-analysis Tau calculation for baseline trend. His performance during the first intervention phase showed an increased level compared to baseline in the middle of the phase, but was highly variable overall, with three data points below baseline and no discernable trend, supported by a negligible nonsignificant effect size (τ Baseline to No Rewards = -0.095, ns). His performance in the rewards condition remained variable, with an improving trend emerging mid-phase that was not maintained, alongside high overlap with the previous phase. Given his highly variable data, alongside an increased level in the no rewards condition that was not maintained, and a nonsignificant effect size, a basic effect for the intervention overall compared to baseline was not supported. With respect to differentiation between the reward conditions, the data were highly variable in both conditions, alongside complete overlap between the conditions with a negligible nonsignificant effect size (τ No Rewards to Rewards = -0.058, ns); thus, a basic effect for differentiation between reward conditions was not established. Chris. As seen in Figure 6, overall, Chris presented with a slight improving baseline that was not statistically significant based on the pre-analysis Tau calculation for baseline trend. At the point of intervention, a slight change in level was observed, with some overlap with baseline. The effect size was small, positive, and significant (τ Baseline to No Rewards = 0.430, p = .050). At the phase change to the rewards condition, Chris’s performance was initially variable, then stabilized to a small positive trend, but with a level similar to baseline performance. Although there was a slight change in level observed in the initial intervention phase, there was also some overlap with 67 baseline, and the effect ultimately was not maintained, with levels in the second intervention phase returning to baseline levels. As such, a basic effect for the intervention overall compared to baseline was not established. With respect to differentiation between the reward conditions, there was complete overlap between the conditions with a small nonsignificant effect size (τ No Rewards to Rewards = -.212, ns); thus, a basic effect for differentiation between reward conditions was not established. Phineas. As seen in Figure 6, Phineas presented with an improving, somewhat variable baseline trend that was statistically significant based on the pre-analysis Tau calculation for baseline trend. His improving trend continued through both intervention phases, albeit slightly flatter, with a highly variable pattern emerging at the end of the second phase (rewards condition). After correcting for positive baseline trend, his effect size for the first intervention phase relative to baseline was moderate and negative (τ Baseline to No Rewards = -0.590, p = .003), suggesting lower performance than would be expected if baseline trend had continued (even after correcting for baseline trend). His performance in the rewards condition generally followed the trend of the previous phase, with high variability emerging at the end of the phase. Given his improving baseline trend, which generally continued throughout intervention, a basic effect for the intervention overall compared to baseline was not established. With respect to differentiation between the reward conditions, the improving trend in the no rewards condition continued into the rewards condition, albeit with a variable pattern emerging at the end of the phase, and the corresponding effect size was negligible and nonsignificant (τ No Rewards to Rewards = 0.105, ns); thus, a basic effect for differentiation between reward conditions was not established. 68 Vertical analysis. In summary, the results of Sequence B indicated no basic effects for the intervention overall (with or without rewards) improving WCPM gains on generalization passages relative to baseline. Therefore, no functional relation for the intervention across conditions relative to baseline was established for Sequence B. With respect to differentiation between the reward conditions, no basic effects were established, indicating no observed differences between WCPM performance when rewards were used versus when they were not. Therefore, no functional relation for the differentiation between reward conditions was established for Sequence B. Summary for generalized gains on CBMReading passages. In summary, for generalized gains on CBMReading passages, an overall functional relation for the intervention (regardless of whether rewards were used), was not established, as there were no basic effects in either sequence. Similarly, there were no basic effects for differential effectiveness between reward conditions in either sequence; thus, there was no functional relation in either sequence. Exploratory Statistical Analysis The effect size estimates (Tau values) were analyzed within a meta-regression for each reading fluency measure across participants and sequences to test for potential differences. The first group of analyses tested if gains relative to baseline differed by condition (Rewards or No Rewards), order (Sequence A: rewards first, or Sequence B: rewards second), or interactions of condition by order. The second group of analyses tested if the differences between the intervention conditions (rewards vs. no rewards) varied by order (rewards first or second). Importantly, given the extremely limited sample size (n = 3 in each sequence; total N = 6), as well as the resulting effects largely being driven by individual participants with the most 69 robust outcomes, the results must be interpreted with significant caution, and are best interpreted as exploratory analyses that may provide useful questions for future group design work. Exploratory meta-regression for direct gains relative to baseline. As seen in Table 8, the results of the meta-regression indicate statistically significant main effects for condition and order, as well as a significant interaction effect. As seen visually in Figure 7, participants overall appeared to perform about equally well relative to baseline in the first intervention phase across sequences, regardless of whether rewards or no rewards was presented first. However, in the second intervention phase, participants performed worst relative to baseline when rewards were taken away. This potentially suggests the order of when rewards are given may affect performance to the extent that there was a decreased (but still overall positive) performance compared to baseline when rewards were removed; however, this effect was largely driven by one participant (Sara). Correspondingly, there appeared to be a small increase in performance compared to baseline across participants who received rewards second. Exploratory meta-regression for differentiation between reward conditions. The meta-regression tested for potential differences between the rewards conditions depending on order (i.e., rewards first or second). As seen in Table 9, and visually in Figure 8, there was a main effect for order, such that the effects of rewards appeared highest when rewards were given second (i.e., when rewards followed no rewards); however, the effect was quite small. 70 Table 8 Meta-regression for direct gains on instructional passages across conditions and sequences b (SE) p-value Intercept 0.231 (0.038) < 0.001 Condition 0.307 (0.055) < 0.001 Order 0.317 (0.055) < 0.001 Condition x Order -0.2105 (0.076) 0.005 Figure 7. Weighted mean effect sizes (Tau) for direct gains on instructional passages by condition and sequence. Sequence A: Rewards to No Rewards; Sequence B: No Rewards to Rewards. Table 9 Meta-regression for direct gains on instructional passages comparing condition B (rewards) and C (no rewards) by sequence b (SE) p-value Intercept 0.372 (0.039) < 0.001 Moderator (Order) -0.259 (0.061) < 0.001 71 Figure 8. Weighted mean effect sizes (Tau) for direct gains on instructional passages comparing condition B (rewards) and C (no rewards) by sequence. The effect size represents the change from no rewards to rewards. Exploratory meta-regression for between-session gains relative to baseline. As seen in Table 10, the results of the meta-regression did not indicate significant main effects for order or condition; however, a significant interaction effect emerged. Specifically, as seen visually in Figure 9, participants appeared to perform best relative to baseline when rewards followed no rewards; however, the effect was quite small. Exploratory meta-regression for differentiation between reward conditions. The meta-regression tested for potential differences between the rewards conditions depending on order (i.e., rewards first or second). As seen in Table 11, and visually in Figure 10, there was a main effect for order, such that the effects of rewards were highest when rewards were given second (i.e., when rewards followed no rewards); however, the effect was very small. Sequence A Sequence B-1.0-0.50.00.51.0Weighted Mean Effect Size 72 Table 10 Meta-regression for between-session gains for first reads on instructional passages b (SE) p-value Intercept -0.115 (0.036) 0.001 Condition -0.025 (0.060) 0.677 Order 0.010 (0.067) 0.877 Condition x Order 0.318 (0.099) 0.001 Figure 9. Weighted mean effect sizes (Tau) for between-session gains on first read of instructional passages. Sequence A: Rewards to No Rewards; Sequence B: No Rewards to Rewards. Table 11 Meta-regression for between-session gains on first read of instructional passages comparing condition B (rewards) and C (no rewards) by sequence b (SE) p-value Intercept 0.156 (0.047) < 0.001 Moderator (Order) -0.420 (0.062) < 0.001 73 Figure 10. Weighted mean effect sizes (Tau) for between-session gains on first read of instructional passages comparing condition B (rewards) and C (no rewards) by sequence. The effect size represents the change from no rewards to rewards. Exploratory meta-regression for generalized gains relative to baseline. As seen in Table 12, the results of the meta-regression indicated main effects for condition and order, and no interaction effect. Specifically, as visually seen in Figure 11, participants appeared to perform best when receiving rewards, and most poorly in the second intervention phase, regardless of whether or not they were receiving rewards; however, the effects were very small. Exploratory meta-regression for differentiation between reward conditions. The meta-regression tested for potential differences between the rewards conditions depending on order (i.e., rewards first or second). As seen in Table 13, and visually in Figure 12, there was a main effect for order, such that the effects of rewards were highest when rewards were given first; however, the effect was extremely small. Sequence A Sequence B-1.0-0.50.00.51.0Weighted Mean Effect Size 74 Table 12 Meta-regression for generalized gains on CBMReading passages b (SE) p-value Intercept -0.043 (0.055) 0.442 Condition 0.390 (0.085) <0.001 Order -0.190 (0.083) 0.022 Condition x Order 0.007 (0.129) 0.958 Figure 11. Weighted mean effect sizes (Tau) for generalized gains on CBMReading passages for each condition compared to baseline. Sequence A: Rewards to No Rewards; Sequence B: No Rewards to Rewards. Table 13 Meta-regression for generalized gains on CBMReading passages comparing condition B (rewards) and C (no rewards) by sequence ß (Standard Error) p-value Intercept -0.059 (0.071) 0.406 Moderator (Order) 0.202 (0.099) 0.042 75 Figure 12. Weighted mean effect sizes (Tau) for generalized gains on CBMReading passages comparing the B (Rewards) and C (No Rewards) conditions. The effect size represents the change from no rewards to rewards. Comprehension The comprehension variable is represented by the number (out of 10) of correct answers on the comprehension questions administered on every second CBMReading passage. The graphical representation for the data is presented in Figures 13 (Sequence A), and 14 (Sequence B). Means and standard deviations for both sequences is presented in Table 14. Sequence A Sequence B-1.0-0.50.00.51.0Weighted Mean Effect Size 76 Table 14 Mean number correct and standard deviations for comprehension questions by sequence and condition Sequence A Participant Mean Baseline Mean Rewards Condition Mean No Rewards Condition Lee 5.00 (0.00) 7.00 (0.82) 6.50 (3.12) Sara 4.00 (0.00) 6.00 (0.82) 4.80 (1.26) Marita 6.00 (0.00) 6.00 (0.82) 5.30 (0.58) Sequence B Participant Mean Baseline Mean No Rewards Condition Mean Rewards Condition Ty 8.00 (0.00) 6.00 (2.00) 6.00 (2.16) Chris 6.00 (1.41) 6.80 (0.50) 6.00 (0.82) Phineas 5.70 (0.58) 5.30 (1.26) 5.00 (0.82) Note. Standard deviation in parentheses; Possible range of scores is 0 to 10 Sequence A. As seen in Figure 13, comprehension performance was variable across participants at the point of intervention following stable baseline levels. Both Lee and Sara presented with improvements in level in both conditions relative to baseline performance. Despite an outlying sub-baseline level data point in the second phase for each of these participants, mean levels of responding remained above baseline (see Table 14). However, given the extremely limited baseline lengths (1 to 3), any interpretation of effect is compromised. As such, Lee and Sara’s performance represents two basic effects (with limitations due to baseline length) for the intervention on comprehension, with no differentiation across conditions (i.e., rewards or no rewards). Marita’s performance was variable at the point of intervention and across phases, with no observed improvement over baseline in either condition. 77 Sequence B. As seen in Figure 14, no improvement over baseline was observed for any of the participants; therefore, no basic effects were established and thus there was no functional relation. Ty’s performance was below his single baseline point across phases. Following an improving baseline, Chris’s performance generally stabilized in phase 1, and deteriorated in phase 2. Phineas’s performance generally reflected a continuation of baseline performance levels. 78 Figure 13. Comprehension questions correct, Sequence A. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 79 Figure 14. Comprehension questions correct, Sequence B. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 80 Self-reported Motivation The motivation variable is represented by the measured distance along a 0 to 10-centimeter line, presented in centimeters (e.g., 6.8 cm). The graphical representation for the data is presented in Figures 15 (Sequence A), and 16 (Sequence B). Means and standard deviations are reported in Table 15. As seen in Figure 15, the participants in Sequence A demonstrated small but discernable patterns in their ratings across phases, whereas the participants in Sequence B generally showed little differentiation across phases. Specifically, in Sequence A, Lee presented with a small increase in level in the no rewards condition compared to the rewards condition, but with negligible improvement over baseline. Following a variable baseline, Sara presented with continued variability at the point of intervention, with a deteriorating trend emerging mid-phase, continuing into the no rewards condition. Across all participants, Sara’s motivation data produced the only pattern that showed similarities with her pattern of performance on the other dependent variables (i.e., deteriorating performance over time). Finally, Marita presented with a deteriorating baseline which continued into the rewards phase, and then reversed to an improving trend in the no rewards phase. In Sequence B, Ty indicted maximum ratings across nearly all presentations, with no differentiation across conditions. Following a variable baseline, Chris presented with some evidence of improvement over baseline, but with significant variability and several near-zero data points. Phineas presented with generally high responding in all phases, alongside some outlying low responses in each phase. Overall, aside from Sara, the self-reported motivation data did not appear to correspond to any of the reading outcome data. Additionally, only Sara specifically mentioned rewards in her reasons for her motivation ratings, reporting “[I’m not motivated] because I don’t get prizes 81 anymore”, for the first three sessions in the no rewards phases. Across her other ratings and across the other participants, the reasons for their ratings generally fell into two categories: 1) generic statements regarding reading (e.g., “I like reading”, “reading is fun”, “reading helps you learn”), and 2) generic comments not related to reading or the intervention (e.g., “I’m tired”, “it is cold outside”, “it is almost my birthday”). Table 15 Self-reported motivation to read across participants and conditions for both sequences Sequence A Participant Mean Baseline Mean Rewards Condition Mean No Rewards Condition Lee 6.10 (1.21) 5.35 (0.80) 6.54 (1.18) Sara 7.73 (0.71) 8.05 (0.69) 6.96 (0.43) Marita 7.50 (1.03) 4.53 (1.51) 4.43 (2.35) Sequence B Participant Mean Baseline Mean No Rewards Condition Mean Rewards Condition Ty 9.70 (0.52) 9.71 (0.68) 10.0 (0.00) Chris 1.48 (1.54) 4.43 (4.71) 5.03 (3.70) Phineas 7.76 (3.22) 7.9 (2.58) 8.67 (2.63) Note: Standard deviation in parentheses 82 Figure 15. Self-reported motivation to read, Sequence A. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 83 Figure 16. Self-reported motivation to read, Sequence B. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 84 Social Validity Social validity ratings were collected at the end of the study. As seen in Table 16, the mean social validity rating across participants and sequences was 4.68 (SD = 0.78; range = 3 – 5). The mean ratings for participants who ended in the no rewards condition was 4.08 (SD = 0.80), generally similar to the participants ending in the rewards condition (M = 4.68, SD = 0.65). Total raw scores are presented in Table 16. Consistent with her other measures, Sara’s social validity ratings were the lowest of all (14 out of 20), although still generally acceptable. Overall, across participants, the social validity ratings indicated the participants liked the intervention and believed it to be beneficial, given that the large majority of the participant’s ratings for the individual items fell above the mid-point of the rating scale (3 on a scale of 1 to 5), and none fell below. 85 Table 16 Raw scores and mean scores for social validity ratings across participants and sequences Sequence A Participant Total Raw Score Lee 17 Sara 14 Marita 18 Mean total scale score 16.3 (2.08) Mean rating across items 4.08 (0.80) Sequence B Participant Total Raw Score Ty 19 Chris 17 Phineas 20 Mean total scale score 18.7 (1.53) Mean rating across items 4.68 (0.65) Mean total scale score across sequences 17.5 (2.07) Mean rating across items across sequences 4.38 (0.78) Note. Standard deviation in parentheses; Possible range of total scale score is 0 to 20; Items were rated on a scale from 1 to 5. 86 Discussion This study aimed to address the question of whether rewards are a useful component to include as part of a standard protocol reading fluency intervention. To this end, this study used robust single-case design methodology to isolate the incremental effect of rewards on reading fluency gains for both instructional and generalization passages, while accounting for sequence effects over a sustained intervention period for each condition. Additional extensions from previous literature included an informal a priori preference assessment, measurement of self-reported motivation to read, and assessment of reading comprehension. Main Findings Effects of rewards on reading outcomes. In terms of the differential effectiveness of rewards versus no rewards on reading outcomes, results for individual participants did not reveal any basic effects across all fluency measures, or the comprehension measure; thus, no functional relations were established for the differential effectiveness of rewards on any reading measure. Effects of rewards on motivation. With respect to self-reported motivation, the measurement tool used in the current study produced variable and/or generally undifferentiated responding, indicating it did not validly capture the construct of intrinsic motivation. Therefore, the resulting data do not support inferences for whether intrinsic motivation was reduced or affected in any way by rewards. General effectiveness of reading fluency intervention. For direct fluency gains on instructional passages (i.e., baseline to the final read of instructional passages during the intervention phases), two basic effects for Lee and Marita emerged in Sequence A (rewards first). A functional relation was established for Sequence B (no rewards first); specifically, 3 of 3 participants demonstrated basic effects, indicated by a clear improvement over baseline across 87 the intervention phases, albeit with a delayed effect present for Ty. Therefore, across both sequences, the intervention produced direct gains in reading fluency relative to baseline for all but one participant (Sara). In terms of between-session fluency gains on instructional passages (i.e., baseline to the first read of a new instructional passage during the intervention phases), one basic effect for the intervention, regardless of rewards, was established (Lee; Sequence A, delayed effect). A negative basic effect was indicated for Sara (Sequence A). Given that there was only 1 basic effect, no functional relations were established for between-session fluency gains relative to baseline. In terms of generalized fluency gains to novel passages, no basic effects were observed. Additionally, interpretation of the results was complicated by improving baseline trends for 2 participants in each sequence, a negative basic effect for Sara, a variable but generally linear continuation of baseline trend for Phineas, and high overlap with baseline levels for the other participants. Reading comprehension. For reading comprehension, two basic effects emerged for the intervention relative to baseline (Sara and Lee; Sequence A); however, the baseline phase lengths were extremely limited (range of 1 to 3 data points), which complicates the ability to draw conclusions. Response to Rewards The results of the current study suggest that in this sample, there was no differentiation between the rewards and no rewards conditions for any reading measure. Of the previous reading fluency studies that have found differentiated responding to rewards, (e.g., Anderson, Daly, & Young, 2013; Bonfiglio, Daly, Martens, Lin, & Corsaut, 2004; Daly et al., 1998; Daly et al., 88 2005; Noell et al., 1998; Noell, Freeland, Witt, & Gansle, 2001; Wilbur & Cushman, 2006), only some included conditions similar to the current study, in which a skill-based treatment package was combined with performance feedback alone, and with performance feedback plus contingent rewards (e.g., Chafouleas, et al., 2004; Eckert, et al., 2000; Eckert, et al., 2002), which allows for observation of the additive effect of rewards when combined with a full treatment package. Of these, results indicated largely undifferentiated responding with respect to combining skill-based strategies with rewards versus performance feedback for most participants. That is, most participants appeared to perform equally well when either performance feedback or rewards were added to skill-based strategies. For example, using brief experimental analysis (BEA), Eckert et al. (2000) found that rewards produced incremental gains over a skill-based plus performance feedback condition for only one participant out of four. Similarly, in another BEA, Eckert et al. (2002) found three of six participants performed similarly with either performance feedback or contingent rewards accompanying the skill-based component, with skill-based plus reward, skill-based plus performance feedback, or skill-based only emerging as most beneficial for the other three participants, respectively. As noted by Eckert et al. (2002), one potential reason for low differentiation in responding may be common features shared between performance feedback and contingent reward. That is, performance feedback may serve as a reinforcer in and of itself, without the need for a tangible reward. Likewise, a contingent reward component is embedded in performance feedback by virtue of being contingent on a goal. It may be that very few students require an additional tangible reinforcer to produce optimal responding; in the current study, no students required it. 89 Alternatively, it may be that the rewards in the current study simply did not function as reinforcers. With respect to the dimensions of reinforcement, this suggests a number of possible implications, including insufficient reinforcement quality (i.e., the participants did not adequately value the rewards in the array), and/or rate (i.e., reinforcement was not frequent enough), as well as considerations for response effort, concerning the relationship between how much effort the participant needs to put forth in order to access the reinforcer, as well as the value of the reinforcer in relation to the response effort needed. Importantly, the reading performance data do suggest that the use of rewards did not hinder performance, which would have been indicated if participants in Sequence B showed decreased performance when rewards were added. Additionally, although the overall effect of rewards versus no rewards was undifferentiated, the participant’s individual responses yielded unique patterns that are worth noting. Neither Marita nor Lee indicated any disappointment or protest when rewards were removed, although they did appear to enjoy having them initially. Sara was upset when rewards were removed, as noted in her verbal response to the first three self-reported motivation probes following the removal of rewards, and she continued to mention her disappointment throughout the remainder of the intervention. Correspondingly, Ty and Phineas had found out that other students were receiving rewards and consistently asked when it was their turn to earn rewards. These two participants in particular took great care and enjoyment in selecting their rewards and occasionally even mentioned the reward the other had received in session, as they were in the same classroom. This suggests that the rewards appeared to be an enjoyable and fun aspect of the intervention for them, which may have made the intervention sessions more enjoyable; however, even if true, this did not translate to their reading performance. 90 Although there was no evidence of differential effectiveness of rewards for Ty on any of the reading measures, he had an interesting pattern of responding in the rewards condition for within-session fluency gains. He presented with a highly variable pattern between the first and fourth timed reads in the no rewards condition compared to a clear pattern of separation in the rewards condition. Although this pattern does not provide evidence for differential effectiveness of rewards, it may suggest greater motivation to reach the reading goal in order to obtain the reward; however, the initial separation pattern began to emerge near the end of the no rewards condition, which complicates interpretation of any overarching effects of the rewards condition. In terms of individualized responding to the overall intervention, the results of the current study highlighted some unique effects for particular participants. For example, Sara’s lack of basic effect for direct gains, and negative pattern of responding (i.e., deteriorating from baseline) for between-session and generalized gains was not expected, given that the standard HELPS intervention has been shown to improve reading fluency across a number of studies (e.g., Begeny, 2011; Begeny et al., 2010; Begeny, Mitchell, Whitehouse, Harris, & Stage, 2011). Her performance for direct fluency gains on instructional passages showed an initial positive response to intervention in the rewards condition compared to baseline, which was ultimately not maintained. One potential reason for her pattern of responding could be her skill level. In analysis of her performance levels relative to the other participants, she had the highest baseline skill-level. This could be similar to the results from Chafouleas et al. (2004), who found that the participant with the highest baseline skill level responded best to a skill-based only treatment and had the poorest response to reward contingencies. Similarly, it could be that reinforcing Sara’s already high reading skill level may have reduced her responding over time (i.e., an initial reinforcing effect became punishing). However, Sara’s performance did not improve when 91 rewards were removed, which may negate this possibility. It would have been interesting to see whether her pattern would be the same if she had started with the no rewards condition. Alternatively, given that her self-reported motivation ratings also decreased over time concomitantly with her reading performance, it could be that the rewards were not punishing, but rather that the reinforcement effect weakened with time. This potentially suggests she may have benefited from an alternative reinforcement schedule or a different array of rewards. Direct vs. Generalized Fluency Gains Overall, weaker effects were found on between-session and generalized fluency gains compared to direct gains in the current study. Yet, the fact that this study included varied measurement of generalized fluency gains is a strength. Previous meta-analytic work in the oral reading fluency literature (e.g., Lee & Yoon, 2017; Therrien, 2004) has found that compared to reporting direct gains on practiced passages, generalization to unpracticed passages is reported less often, and when the effects are reported, they are generally of lesser magnitude. However, generalization of reading skills to new and varied reading material is ultimately the goal of reading intervention, and demands careful consideration and more representation in future work (Daly, Martens, Barnett, Witt, & Olson, 2007; Lo et al., 2011). The limited evidence for generalization in the reading fluency literature is further complicated by the varied ways in which generalization is measured. For example, when generalization is measured may play a role, such as whether generalization passages are administered at the beginning of the session prior to intervention components, or at the end after the student has built behavioural momentum (e.g., Daly et al., 2005; Noell et al., 1998). The type of generalization passages used is also important, such as passages with many similar words to the practiced passage (i.e., a “high-content overlap” passage), versus novel passages in which 92 few or no words from the practiced passage are intentionally included (Daly, Martens, Kilmer, & Massie, 1996). The former strategy is typically employed under the assumption that repeated reading of the practiced passages creates stimulus control of reading behaviour to the specific words in the passage, which is hoped to generalize when many of the same words appear in a different context (i.e., programming common stimuli by using the same words in a different passage; Ardoin, Binder, Zawoyski, & Foster, 2018). Conversely, the latter strategy, in which a brand new passage is used, is more conducive to true generalization in which responding is transferred to untrained or more distal stimuli, that is, to contexts outside of the immediate instructional environment (Daly, Lentz, & Boyer, 1996; Daly et al., 2007). Ultimately, this type of generalization may be more ecologically valid in the sense that the goal of reading is to help students read a variety of texts across varied contexts (Lo et al., 2011). Finally, generalization programming procedures differ across studies, such as using multiple exemplar training (i.e., practicing many different passages), repeated reading of the same passage multiple times, and/or manipulation of the intervention performance criteria to align with the level of responding expected in the natural environment (e.g., Ardoin et al., 2018; Gibson, Cartledge, Keyes, & Yawn, 2014; Lo et al., 2011; Martens et al., 2019). Ongoing work is needed to continue to identify the optimal and most meaningful ways to measure generalization of reading fluency. Measurement Challenges When Assessing Generalized Reading Fluency Gains The generally weaker effects overall on between-session and generalized fluency gains compared to direct gains were not necessarily surprising given the small expected growth rate for generalized reading fluency over time (Ardoin & Christ, 2009). For example, Begeny et al. (2015) found that participants increased an average of .31 WCPM per passage, or approximately 93 1 WCPM weekly on a three session per week schedule. Begeny et al.’s procedures were similar to the present study, which administered the standard HELPS program three times per week using passages similar to the HELPS passage set, which also slightly increased in difficulty over time. This is particularly relevant to single-case design studies, as growth that small is difficult to detect using visual analysis, particularly given a limited intervention duration. Detection of this rate of growth is further compromised by the high amount of measurement error associated with reading fluency probes, which can range from 4 to 15 WCPM (Christ & Silberglit, 2007). Measurement schedules are particularly important to consider for achieving reliable estimates of effects; Christ, Zopluoglu, Monaghen, & Van Norman (2013) found that a minimum of 8 to 10 weeks with an appropriately dense schedule (e.g., approximately 5 probes per week for 8 weeks and less for 10 weeks), was needed to achieve reliable and valid slope estimates. The schedule and density of data collection in the present study may not have been robust enough to reliably observe generalized fluency growth. Need for Group Designs to Disentangle Small Differences in Effects In terms of the exploratory statistical analyses conducted in the current study, the results yielded some potentially interesting patterns; however, as previously noted, given the extremely limited sample size (n = 3 in each sequence; total N = 6), as well as the resulting effects largely being driven by individual participants with the most robust outcomes, the results must be interpreted with significant caution, and are best interpreted as preliminary analyses that may provide useful questions for future group design work. In terms of direct gains, the meta-regression analyses across participants indicated similarly positive performance in phase 1 compared to baseline, regardless of whether rewards or no rewards was introduced first. However, the evidence suggested potentially more 94 differentiation in performance in phase 2. Specifically, participants showed the best performance relative to baseline in the second intervention phase when the rewards condition followed the no rewards condition. Correspondingly, participants in the opposite sequence, when no rewards followed rewards, appeared to show the worst performance relative to baseline. For between-session gains, there was a significant interaction effect, whereby participants in the second intervention phase seemed to perform the strongest relative to baseline when rewards followed no rewards. For generalized gains, there was a main effect for condition, whereby all participants appeared to perform best in the rewards condition regardless of what order they received them. There was also a main effect for order, whereby participants appeared to perform best relative to baseline when they were in the first intervention phase. When considering the differential effectiveness of the reward conditions compared to each other, the results followed a similar direction for direct gains and between-session gains. Specifically, rewards had the strongest effect in phase 2, that is, when rewards followed no rewards; however, in both cases the effect was very small. For generalized gains, the pattern was opposite such that the effect of rewards was strongest in phase 1 when they were given first; however, this effect was again very small. Taken together, the results from the exploratory statistical analyses suggest some interesting potential patterns that would be beneficial to address in future work using group research design with larger samples, as such designs are better able to detect small differences between intervention conditions. Specifically, there appeared to be some evidence of comparatively weaker differentiation between the rewards and no rewards conditions in phase 1, with greater differentiation emerging in phase 2 for both sequences. This pattern may indicate the importance of considering the timing of when rewards are (or are not) used. Specifically, 95 taking rewards away may have the greatest negative effect, potentially indicating the importance of proper fading procedures. In terms of additive benefits of rewards, there appeared to be overall mild differential effectiveness compared to no rewards. A larger group design would be better able to tease apart this effect and determine whether some students would benefit from rewards more than others, and if rewards may provide a boost in performance. In addition, a larger group design would be better suited for analyzing differences such as the potential moderating effect of high baseline skill level, as was potentially suggested for Sara in the current study. Strengths This study was designed with several strengths. First, the multiple-baseline multiple-sequence design allowed for an analysis of the additive effects of rewards on reading performance across time, whereas the majority of previous work has focused on immediate effects of rewards within-session compared to other treatments (e.g., alternating-treatment designs, brief experimental analysis). Analyzing the participants’ data paths across a sustained period allowed for examination of potential reinforcement effects across time (e.g., Sara’s declining performance throughout the intervention), leading to potentially important considerations for reinforcement contingency management for academic interventions. Second, the use of an informal preference assessment was an important step towards achieving adequate reinforcer quality of the selected rewards for the participants. Third, the attempt to build-in self-reported motivation to read in this study was a departure from previous work, and an important consideration particularly given the controversial nature of rewards and the concern regarding undermined intrinsic motivation. Fourth, this study employed two types of reading fluency generalization measurement from two different passage sets, allowing for a more robust analysis 96 of the generalization of reading skill outside of the immediate instructional environment. This led to important considerations for how to most meaningfully measure generalized gains in reading fluency, as well as highlighting measurement challenges for generalization across the reading fluency literature. Limitations The results of the current study must be interpreted in the context of several limitations. First, the high variability in the data across measures may have masked or distorted the detection of effects. Although, as previously noted, CBM reading data are known for high variability, it would have been beneficial to further mitigate these effects by using averaged performance on more than one probe for each data point, and using a longer duration and/or denser schedule for progress monitoring (Christ et al., 2013). Another limitation concerns the additional reading supports received by the participants outside of the intervention for the current study. Although this support was provided consistently throughout the duration of the study, and the baseline phases for the instructional passages document little to no change in WCPM prior to intervention (with the exception of Sara), there is still the potential for multiple treatment effects. The improving baselines on the generalization passages for two participants in each sequence were not consistent with the baselines for the instructional passages (except Sara); however, the improving trend did not appear to continue into the intervention phases, as there were high levels of overlap with the baseline phase on the generalization passages across most participants. Similarly, although determining baselines and intervention phase lengths a priori was necessary within the confines of the school calendar (i.e., close to the end of the school year), it hindered visual analyses for some participants (e.g., entering a new phase on an improving or unclear trend). 97 With respect to reinforcement, this study was designed to isolate the effects of rewards specifically within the context of a standard protocol intervention. However, the other motivational components included in the intervention, such as goal-setting and performance feedback with verbal praise, may have also functioned as reinforcers. This may be one explanation for the lack of robust differentiation between conditions on the current study; in other words, rewards may not have served incremental reinforcement benefits beyond the other motivational strategies. Future work might consider a similar design with a third condition (skill-based only) in order to isolate the other motivational strategies. Alternatively, it may be that the reinforcement contingencies were not appropriate or robust enough to produce adequate responding, and a denser schedule (e.g., a reinforcer after each timed read) and/or a higher perceived quality of reinforcers was needed. For example, although an informal preference assessment was used in order to gauge the appropriate selection of the rewards to include in the array, a formal individualized assessment was not conducted. Doing so may have further increased the likelihood of the selected rewards functioning as reinforcers. Additionally, the participants met their reading goal for nearly every session, which may have diluted the reinforcement effect over time. Implications and Future Directions The results of the current study indicate that the standard protocol reading fluency intervention was effective in increasing WCPM performance for direct gains on instructional passages, regardless of whether or not rewards were used, for five out of six participants. Although no differentiation was observed between the rewards and no rewards conditions, the rewards procedure is low effort, and appeared enjoyable for all. Most importantly, the use of rewards did not appear to hinder reading fluency performance. 98 With respect to implications for educators, more work is needed to better understand how school-based personnel view the use of rewards in the context of academic intervention. For example, although there is a rich literature base regarding the effects of rewards on intrinsic motivation, few studies have sought the perspectives of teachers on this matter. Given the contention that rewards may undermine motivation (e.g., Deci et al., 1999; Deci et al., 2001; Kohn, 1993, 1999) it is worthwhile to explore educators’ perspectives on the treatment acceptability of rewards and whether they share concerns about intrinsic motivation. Understanding teachers’ perceptions about rewards and whether reward use is an acceptable treatment is important given that they are primary implementors. If teachers find rewards to be an unacceptable treatment, understanding the reasons for their opinions will inform effective consultation for school-based practitioners. In one known study, Hoffmann, Huff, Patterson, and Nietfeld (2009) surveyed 86 elementary school teachers and found that all teachers reported using tangible rewards to some extent in their classroom, but were more likely to use rewards for behaviour management than academic achievement. Additionally, 33% of the sample indicated some conditions to reward use, such as only giving rewards for certain kinds of students, or limited tangible rewards in comparison to praise. Although not addressed in the study, the tendency to use rewards less frequently for academic achievement than for behaviour management may be reflective of the concern for undermined intrinsic motivation. For instance, teachers may be less likely to reward academic behaviour based on the philosophical notion that students should engage in learning for intrinsic reasons. The results from the current study appear to be in line with supporting educators’ individual decision about whether or not to include rewards, given the lack of differentiation 99 between conditions, and that rewards did not hinder performance. However, given that many standard protocols, such as the one used in the current study, are considered evidence-based practices as a package, it may not be prudent to remove a component, such as rewards, without careful consideration. A more robust component analysis would be needed to determine acceptable modifications to a standard approach. Additionally, further work building on the preliminary statistical analysis results found in the current study could better identify potential patterns associated with adding rewards when students are not making adequate gains, and/or whether taking rewards away causes a deterioration of performance. For example, it may be important to provide systematic considerations for fading procedures if rewards are being used, so as not to negatively affect performance. At the same time, given that the rewards and reinforcement contingencies used in the current study may have simply been ineffective, it would be beneficial for future work to explore the potential benefits of rewards using alternative contingencies. For example, more robust preference assessments and increased rates of reinforcement may assist in producing higher rates of responding for reading fluency behaviour. Additionally, given that the reading goal for the participants was stable across the intervention, this may have put an upper limit on the response effort they needed to put forth in order to access the reward. Future work may consider designs such as a changing criterion design, which would apply increased goal criteria through the progression of the study, thereby gradually increasing the response effort required over time. Additionally, it may be that certain kinds of students may particularly benefit from rewards, such as students with both learning difficulties and emotional and behaviour disorders (e.g., Strong-Hilsmier et al., 2016). For individual students in an educational setting, using a data-based individualization approach could help disentangle whether rewards (or another 100 component) are useful to include for a boost in performance if a student is non-responsive to a standard protocol that does not already include them. As such, it may be useful to keep an open mind about including rewards for certain students if it appears they may benefit, could use a boost in performance, or simply to make sessions more enjoyable. With respect to measuring motivation, the lack of findings from the self-report measure of motivation on the current study does not necessarily indicate participants were or were not motivated to read, particularly given the generic and often unrelated qualitative responses to the probes. Rather, the measurement tool did not appear to accurately measure motivation as intended. The difficulty of measuring motivation repeatedly over time (as is required in single-case design) was also found in a similar study, where ceiling effects were observed on a Likert-type scale intended to measure intrinsic motivation to read (Cheng, Mercer, & Saqui, under review). Moreover, having participants self-evaluate the construct of motivation may not have been developmentally appropriate for them, given that it is an abstract concept for younger learners. Future work is needed to find reliable and valid ways to measure intrinsic motivation repeatedly over time in a developmentally appropriate way. Conclusion Overall, the results of this study indicated that there was no differentiation in reading performance when rewards were used versus when they were not. The standard protocol fluency intervention, regardless of whether rewards were used, was effective in increasing direct fluency gains for five out of six participants across two multiple-baseline designs. Only one participant showed between-session fluency gains. No participants showed generalized gains to novel reading material. Two participants (both in the same sequence) showed gains in comprehension; however, the very short baseline phase lengths complicate any causal inferences. Taken together, 101 the results are in line with teachers using their personal preference for whether to use rewards or not, given that there was no evidence that rewards hindered (or necessarily helped) performance. However, based on the exploratory statistical analyses presented, further work is needed to disentangle potential patterns related to a possible deteriorating effect when rewards are removed, and the potential moderating relationship with learner-centered characteristics, such as high initial baseline skill level. 102 References Akin-Little, K. A., Eckert, T. L., Lovett, B. J., & Little, S. G. (2004). Extrinsic reinforcement in the classroom: Bribery or best practice. School Psychology Review, 33(3), 344-662. Akin-Little, K. A., & Little, S. G. (2004). Re-examining the overjustification effect. Journal of Behavioral Education, 13(3), 179-192. doi: 10.1023/B:JOBE.0000037628.81867.69 Anderson, M. N., Daly, E. J., & Young, N. D. (2013). Examination of a one-trial brief experimental analysis to identify reading fluency interventions. Psychology in the Schools, 50(4), 403-414. doi: 10.1002/pits.21682 Ardoin, S. P., Binder, K. S., Zawoyski, A. M., & Foster, T. E. (2018). Examining the maintenance and generalization effects of repeated practice: A comparison of three interventions. Journal of School Psychology, 68, 1-18. doi:10.1016/j.jsp.2017.12.002 Ardoin, S. P., & Christ, T. J. (2009). Curriculum-based measurement of oral reading: Standard errors associated with progress monitoring outcomes from DIBELS, AIMSweb, and an experimental passage set. School Psychology Review, 38(2), 266-283. BC Ministry of Education. (2016). Achievement Reports: Foundational Skills Assessment. Retrieved March 9th, 2019 from: http://www.bced.gov.bc.ca/reports/pdfs/fsa/prov.pdf Begeny, J. C. (2009). Helping Early Literacy with Practice Strategies (HELPS): A one-on-one program designed to improve students' reading fluency. Raleigh, NC: The HELPS Education Fund. Retrieved from http://www.helpsprogram.org Begeny, J. C. (2011). Effects of the helping early literacy with practice strategies (HELPS) reading fluency program when implemented at different frequencies. School Psychology Review, 40(1), 149-157. 103 Begeny, J. C., Laugle, K. M., Krouse, H. E., Lynn, A. E., Tayrose, M. P., & Stage, S. A. (2010). A Control-Group Comparison of Two Reading Fluency Programs: The Helping Early Literacy with Practice Strategies (HELPS) Program and the Great Leaps K-2 Reading Program. School Psychology Review, 39(1), 137-155. Begeny, J. C., Mann, C. M., Cunningham, M. D., & Tsuen, H. Y. (2009). The Helping Early Literacy with Practice Strategies (HELPS) curriculum: Instructional passages developed for use with the HELPS program. Raleigh, NC: The HELPS Education Fund. Retrieved from http://www.helpsprogram.org Begeny, J. C., & Martens, B. K. (2006). Assisting low-performing readers with a group-based reading fluency intervention. School Psychology Review, 35(1), 91-107. Begeny, J. C., Mitchell, R. C., Whitehouse, M. H., Harris Samuels, F., & Stage, S. A. (2011). Effects of the HELPS reading fluency program when implemented by classroom teachers with low‐performing second‐grade students. Learning Disabilities Research & Practice, 26(3), 122-133. doi: 10.1111/j.1540-5826.2011.00332.x Begeny, J. C., Whitehouse, M. H., Methe, S. A., Codding, R. S., Stage, S. A., & Nuepert, S. (2015). Do intervention-embedded assessment procedures successfully measure student growth in reading? Psychology in the Schools, 52(6), 578-593. doi:10.1002/pits.21843 Bonfiglio, C. M., Daly, E. J., Martens, B. K., Lin, L. R., & Corsaut, S. (2004). An experimental analysis of reading interventions: Generalization across instructional strategies, time, and passages. Journal of Applied Behavior Analysis, 37(1), 111-114. doi: 10.1901/jaba.2004.37-111 Brophy, J. E. (2004). Motivating students to learn (2nd ed.). Mahwah, NJ: Erlbaum. 104 Burns, M. K., Dean, V. J., & Foley, S. (2004). Preteaching unknown key words with incremental rehearsal to improve reading fluency and comprehension with children identified as reading disabled. Journal of School Psychology, 42(4), 303-314. doi: 10.1016/j.jsp.2004.04.003 Burns, M. K., Taylor, C. N., Warmbold-Brann, K. L., Preast, J. L., Hosp, J. L., & Ford, J. W. (2017). Empirical synthesis of the effect of standard error of measurement on decisions made within brief experimental analyses of reading fluency. Psychology in the Schools, 54(6), 640-654. doi:10.1002/pits.22022 Burns, M. K., & Wagner, D. (2008). Determining an effective intervention within a brief experimental analysis for reading: A meta-analytic review. School Psychology Review, 37(1), 126-136. Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245-281. Cameron, J., Banko, K. M., & Pierce, W. D. (2001). Pervasive negative effects of rewards on intrinsic motivation: The myth continues. Behavior Analyst, 24, 1-44. doi: 10.1007/BF03392017 Cameron, J., & Pierce, W. D. (1994). Reinforcement, reward, and intrinsic motivation: A meta-analysis. Review of Educational Research, 64(3), 363-423. doi: 10.3102/00346543064003363 Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014). Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis. Psychological Bulletin, 140(4), 980-1008. doi:10.1037/a0035661 105 Chafouleas, S. M., Martens, B. K., Dobson, R. L., Weinstein, K. S., & Gardner, K. B. (2004). Fluent reading as the improvement of stimulus control: Additive effects of performance-based interventions to repeated reading on students' reading and error rates. Journal of Behavioral Education, 13(2), 67-81. doi: 10.1023/B:JOBE.0000023656.45233.6f Chard, D. J., Vaughn, S., & Tyler, B.J. (2002). A synthesis of research on effective interventions for building reading fluency with elementary students with learning disabilities. Journal of Learning Disabilities, 35(5), 386-406. doi: 10.1177/00222194020350050101 Cheng, M.P., Mercer, S.H., & Saqui, S. (2019). Effectiveness of a reading fluency intervention incorporating student choice. Manuscript under review. Christ, T. J. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology (pp. 159–176). Bethesda, MD: National Association of School Psychologists. Christ, T. J., & Silberglitt, B. (2007). Estimates of the standard error of measurement for curriculum-based measures of oral reading fluency. School Psychology Review, 36(1), 130-146. Christ, T. J., Zopluoglu, C., Monaghen, B. D., & Norman, E. R. (2013). Curriculum-Based Measurement of Oral Reading: Multi-study evaluation of schedule, duration, and dataset quality on progress monitoring outcomes. Journal of School Psychology, 51(1), 19-57. doi:10.1016/j.jsp.2012.11.001 Daly, E.J., Bonfiglio, C., Mattson, T., Persampieri, M., Foreman-Yates, K., & Mccomas, J. (2005). Refining the experimental analysis of academic skills deficits: Part I. An investigation of variables that affect generalized oral reading performance. Journal of Applied Behavior Analysis, 38(4), 485-497. doi: 10.1901/jaba.2005.113-04 106 Daly, E. J., Lentz, F. E., & Boyer, J. (1996). The Instructional Hierarchy: A conceptual model for understanding the effective components of reading interventions. School Psychology Quarterly, 11(4), 369-386. doi:10.1037/h0088941 Daly, E. J, Martens, B. K., Barnett, D., Witt, J. C., & Olson, S. C. (2007). Varying intervention delivery in response to intervention: Confronting and resolving challenges with measurement, instruction, and intensity. School Psychology Review, 36(4), 562-581. Daly, E. J., Martens, B. K., Dool, E. J., & Hintze, J. M. (1998). Using brief functional analysis to select interventions for oral reading. Journal of Behavioral Education, 8(2), 203-218. doi: 10.1023/A:1022835607985 Daly, E. J, Martens, B. K., Hamler, K., R., Dool, E. J., Eckert, T. L. (1999). A brief experimental analysis for identifying instructional components needed to improve oral reading fluency. Journal of Applied Behavior Analysis, 32(1), 83-94. doi: 10.1901/jaba.1999.32-83 Daly, E. J., Martens, B. K., Kilmer, A., & Massie, D. R. (1996). The effects of instructional match and content overlap on generalized reading performance. Journal of Applied Behavior Analysis, 29(4), 507-518. doi:10.1901/jaba.1996.29-507 Daly, E. J., Persampieri, M., McCurdy, M., & Gortmaker, V. (2005). Generating reading interventions through experimental analysis of academic skills: Demonstration and empirical evaluation. School Psychology Review, 34(3), 395-414. Danielson, L., & Rosenquist, C. (2014). Introduction to the TEC special issue on data-based individualization. Teaching Exceptional Children, 46(4), 6-12. doi:10.1177/0040059914522965 Deci, E.L. (1975). Intrinsic motivation. New York, NY: Plenum. 107 Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6), 627-668. doi: 10.1037/0033-2909.125.6.627 Deci, E. L., Koestner, R., & Ryan, R. M. (2001). Extrinsic rewards and intrinsic motivation in education: Reconsidered once again. Review of Educational Research, 71, 1-27. doi: 10.3102/00346543071001001 Deci, E. L., & Ryan, R. M. (1980). The empirical exploration of intrinsic motivational processes. In L. Berkowitz (Ed.), Advances in experimental social psychology. (pp. 33-80). New York, NY: Academic Press. doi:10.1016/s0065-2601(08)60130-6 Eckert, T. L., Ardoin, S. P., Daisey, D. M., & Scarola, M. D. (2000). Empirically evaluating the effectiveness of reading interventions: The use of brief experimental analysis and single case designs. Psychology in the Schools, 37(5), 463-473. doi:10.1002/1520-6807 Eckert, T. L., Ardoin, S. P., Daly, E. J., & Martens, B. K. (2002). Improving oral reading fluency: A brief experimental analysis of combining an antecedent intervention with consequences. Journal of Applied Behavior Analysis, 35(3), 271-281. doi: 10.1901/jaba.2002.35-271 Fast Bridge Learning (2015). Formative assessment system for teachers: Technical manual (v2.0). Fast Bridge Learning (fastbridge.org): Minneapolis. Fuchs, L. S., Fuchs, D., Hosp, M. K., & Jenkins, J. R. (2001). Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. Scientific Studies of Reading, 5(3), 239-256. doi: 10.1207/S1532799XSSR0503_3 108 Fuchs, D., Mock, D., Morgan, P. L., & Young, C. L. (2003). Responsiveness-to-intervention: Definitions, evidence, and implications for the learning disabilities construct. Learning Disabilities Research & Practice, 18(3), 157-171. doi: 10.1111/1540-5826.00072 Gibson, L., Cartledge, G., Keyes, S. E., & Yawn, C. D. (2014). The effects of a supplementary computerized fluency intervention on the generalization of the oral reading fluency and comprehension of first-grade students. Education and Treatment of Children, 37(1), 25-51. doi:10.1353/etc.2014.0003 Harlow, H. F., Harlow, M. K., & Meyer, D. R. (1950). Learning motivated by a manipulation drive. Journal of Experimental Psychology, 40(2), 228-234. doi: 10.1037/h0056906 Haring, N. G., Lovitt, T. C., Eaton, M. D., & Hansen, C. L. (1978). The fourth R: Research in the classroom. Columbus, OH: Charles E. Hasbrouck, J. E., Ihnot, C., & Rogers, G. H. (1999). “Read Naturally”: A strategy to increase oral reading fluency. Literacy Research and Instruction, 39(1), 27-37. Herrnstein, R. J. (1961). Relative and absolute strength of response as a function of frequency of reinforcement. Journal of the Experimental Analysis of Behavior, 4(3), 267-272. doi:10.1901/jeab.1961.4-267 Hidi, S., & Harackiewicz, J. M. (2000). Motivating the academically unmotivated: A critical issue for the 21st century. Review of Educational Research, 70(2), 151-179. doi:10.2307/1170660 Hoffmann, K. F., Huff, J. D., Patterson, A. S., & Nietfeld, J. L. (2009). Elementary teachers use and perception of rewards in the classroom. Teaching and Teacher Education, 25(6), 843-849. doi:10.1016/j.tate.2008.12.004 109 Joseph, L. M., Alber-Morgan, S., & Neef, N. (2015). Applying behavior analytic procedures to effectively teach literacy skills in the classroom. Psychology in the Schools, 53(1), 73-88. doi:10.1002/pits.21883 Kendall, M. G. (1962). Rank correlation methods (3rd ed.). New York, NY: Hafner. Kohn, A. (1993/1999). Punished by rewards: The trouble with gold stars, incentive plans, A’s, praise, and other bribes. Boston, MA: Houghton Mifflin. Kratochwill, T. R., Hitchcock, J. H., Horner, R. H., Levin, J. R., Odom, S. L., Rindskoff, D. M., & Shadish, W. R. (2013). Single-case intervention research design standards. Remedial and Special Education, 34(1), 26–38. doi:10.1177/0741932512452794. LaBerge, D., & Samuels, S. J. (1974). Toward a theory of automatic information processing in reading. Cognitive Psychology, 6(2), 293-323. doi: 10.1016/0010-0285(74)90015-2 Lee, J., & Yoon, S. Y. (2017). The effects of repeated reading on reading fluency for students with reading disabilities: A meta-analysis. Journal of Learning Disabilities, 50(2), 213-224. doi: 10.1177/0022219415605194 Lemons, C. J., Kearns, D. M., & Davidson, K. A. (2014). Data-based individualization in reading. Teaching Exceptional Children, 46(4), 20-29. doi:10.1177/0040059914522978 Lepper, M. R., Greene, D., & Nisbett, R. E. (1973). Undermining children's intrinsic interest with extrinsic reward: A test of the "overjustification" hypothesis. Journal of Personality and Social Psychology, 28, 129-137. doi: 10.1037/h0035519 Lepper, M.R., & Henderlong, J.H. (2000). Turning “play” into “work” and “work” into “play”: 25 years of research on intrinsic versus extrinsic motivation. In C. Sansone & J. M. Harackiewicz (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 257-307). San Diego, CA: Academic Press. 110 Lepper, M. R., Sethi, S., Dialdin, D., & Drake, M. (1997). Intrinsic and extrinsic motivation: A developmental perspective. In S. S. Luthar, J. A. Burack, D. Cicchetti, & J. R. Weisz (Eds.), Developmental psychopathology: Perspectives on adjustment, risk, and disorder (pp. 23-50). New York, NY: Cambridge University Press. Lindström, E. R., Gesel, S. A., & Lemons, C. J. (2019). Data-based individualization in reading: Tips for successful implementation. Intervention in School and Clinic, 00(0), 1-7. doi:10.1177/1053451219837634 Lo, Y. & Cooke, N. L. & Starling, A. L. P. (2011). Using a repeated reading program to improve generalization of oral reading fluency. Education and Treatment of Children, 34(1), 115-140. doi: 10.1353/etc.2011.0007 Maag, J. W. (2001). Rewarded by punishment: Reflections on the disuse of positive reinforcement in schools. Exceptional Children, 67(2), 173-186. doi: 10.1177/001440290106700203 Martens, B. K., Young, N. D., Mullane, M. P., Baxter, E. L., Sallade, S. J., Kellen, D., . . . Underberg, J. (2019). Effects of word overlap on generalized gains from a repeated readings intervention. Journal of School Psychology, 74, 1-9. doi:10.1016/j.jsp.2019.05.002 Martens, B. K., & Witt, J. C. (2004). Competence, persistence, and success: The positive psychology of behavioral skill instruction. Psychology in the Schools, 41(1), 19-30. doi: 10.1002/pits.10135 Mercer, C., & Campbell, K. 1998. Great Leaps Reading Program. Micanopy, FL: K-2 Diarmuid, Inc. 111 Mercer, S. H., Harpole, L. L., Mitchell, R. R., Mclemore, C., & Hardy, C. (2012). The impact of probe variability on brief experimental analysis of reading skills. School Psychology Quarterly, 27(4), 223-235. doi:10.1037/spq0000001 Mitchell, C., & Begeny, J. C. (2014). Improving student reading through parents’ implementation of a structured reading program. School Psychology Review, 43(1), 41–58. Morgan, P. L., & Sideridis, G. D. (2006). Contrasting the effectiveness of fluency interventions for students with or at risk for learning disabilities: A multilevel random coefficient modeling meta-analysis. Learning Disabilities Research & Practice, 21(4), 191-210. doi: 10.1111/j.1540-5826.2006.00218.x National Center on Intensive Intervention. (2018). Data-based individualization. Retrieved June, 2019, from https://intensiveintervention.org. Noell, G. H., Freeland, J. T., Witt, J. C., & Gansle, K. A. (2001). Using brief assessments to identify effective interventions for individual students. Journal of School Psychology, 39(4), 335-355. doi:10.1016/s0022-4405(01)00072-3 Noell, G. H., Gansle, K. A., Witt, J. C., Whitmarsh, E. L., Freeland, J. T., Lafleur, L. H., . . . Northup, J. (1998). Effects of contingent reward and instruction on oral reading performance at differing levels of passage difficulty. Journal of Applied Behavior Analysis, 31(4), 659-663. doi:10.1901/jaba.1998.31-659 Noell, G. H., & Gresham, F. M. (2001). A multiple-sequence variant of the multiple-baseline design: A strategy for analysis of sequence effects and treatment comparison. School Psychology Quarterly, 16(2), 207-221. doi:10.1521/scpq.16.2.207.18699 112 Parker, R. I., Vannest, K. J., Davis, J. L., & Sauber, S. B. (2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42(2), 284-299. doi:10.1016/j.beth.2010.08.006 Pikulski, J. J., & Chard, D. J. (2005). Fluency: Bridge between decoding and reading comprehension. The Reading Teacher, 58(6), 510-519. Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25(1), 54-67. doi: 10.1006/ceps.1999.1020 Ryan, R. M., & Deci, E. L. (2002). Overview of self-determination theory: An organismic dialectical perspective. In E. L. Deci & R. M. Ryan (Eds.), Handbook of self-determination research (pp. 3-33). Rochester, NY: University of Rochester. Sansone, C., & Harackiewicz, J. M. (2000). Looking beyond rewards: The problem and promise of intrinsic motivation. In C. Sansone & J. M. Harackiewicz (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 1-9). San Diego, CA: Academic Press. Skinner, B. F. (1953). Science and human behavior. New York, NY: Macmillan. Skinner, C. H. (1998). Preventing academic skills deficits. In T. S. Watson & F. M. Gresham, (Eds.), Handbook of Child Behavior Therapy (pp. 61–82). New York, NY: Plenum. Solomon, B. G., Dawes, J. M., Duhon, G. J., & Poncy, B. C. (2018). False discovery rates when engaging in skill versus performance deficit analysis for academic instructional planning. Assessment for Effective Intervention, 1-10. doi:10.1177/1534508418808365 113 Stanovich, K. E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21(4), 360-407. doi: 10.2307/747612 Statistics Canada. (2006). Eduational outcomes at age 19 associated with reading ability at age 15. Retrieved August 9th, 2016 from http://www.statcan.gc.ca/pub/81-595-m/81-595-m2006043-eng.pdf. Statistics Canada. (2009). Eduational outcomes at age 21associated with reading ability at age 15. Retrieved May 10th, 2019 from https://www150.statcan.gc.ca/n1/pub/81-004-x/2009002/article/10896-eng.htm Stevens, E. A., Walker, M. A., & Vaughn, S. (2017). The effects of reading fluency interventions on the reading fluency and reading comprehension performance of elementary students with learning disabilities: A synthesis of the research from 2001 to 2014. Journal of Learning Disabilities, 50(5), 576-590. doi: 10.1177/0022219416638028 Strong Hilsmier, A., Wehby, J. H., & Falk, K. B. (2016). Reading fluency interventions for middle school students with academic and behavioral disabilities. Reading Improvement, 53(2), 53–64. Tarlow, K. R. (2016). Baseline corrected tau calculator. Retrieved from http://www.ktarlow.com/stats/tau Tarlow, K. R., & Penland, A. (2016). Percentage of nonoverlapping data (PND) calculator. Retrieved from http://www.ktarlow.com/stats/pnd Therrien, W. J. (2004). Fluency and comprehension gains as a result of repeated reading: A meta-analysis. Remedial and Special Education, 25(4), 252-261. doi: 10.1177/07419325040250040801 114 van Laerhoven, H., van der Zaag-Loonen, H. J., & Derkx, B. H. (2004). A comparison of Likert scale and visual analogue scales as response options in children’s questionnaires. Acta Paediatrica, 93(6), 830–835. doi: 10.1080/08035250410026572 Wigfield, A., & Guthrie, J. T. (1997). Relations of children's motivation for reading to the amount and breadth of their reading. Journal of Educational Psychology, 89(3), 420-432. doi: 10.1037//0022-0663.89.3.420 Wigfield, A., Guthrie, J. T., Tonks, S., & Perencevich, K. C. (2004). Children's motivation for reading: Domain specificity and instructional influences. The Journal of Educational Research, 97(6), 299-310. doi: 10.3200/joer.97.6.299-310 Wilber, A., & Cushman, T. P. (2006). Selecting effective academic interventions: An example using brief experimental analysis for oral reading. Psychology in the Schools, 43(1), 79-84. doi: 10.1002/pits.20131 115 Appendix A: Social Validity Scale 1. I liked this reading program. 2. There are better ways to help me with my reading than this reading program. 3. I think this reading program has helped me with my reading. 4. I think this reading program could help other children with their reading. Adapted from: Children’s Intervention Rating Profile (Witt & Elliott, 1985) and Begeny and Martens (2006) 116 Appendix B: Reward Items Cool pencil grips Erasers (e.g., animal shaped; x4) Hair ties Harry Potter stuff Hello Kitty stuff Hockey stuff Lego Minecraft stuff Notebooks Pens and pencils (x2) Pokemon cards (x4) Shopkins stuff Slime (x2) Smelly pencil crayons/markers (x3) Stress balls (x4) Suction-cup toys 117 Appendix C: Additional Tables Table 17 Percent of non-overlapping data for direct gains on instructional passages across phases Participant PND Baseline to Rewards PND Baseline to No Rewards PND Rewards to No Rewards Lee 100% 100% 8% Sara 83% 75% 0% Marita 91.7% 100% 0% Sequence B Participant PND Baseline to No Rewards PND Baseline to Rewards PND No Rewards to Rewards Ty 50% 100% 25% Chris 100% 91.7% 0% Phineas 100% 100% 25% Table 18 Percent of non-overlapping data for between-session gains on instructional passages across phases Sequence A Participant PND Baseline to Rewards PND Baseline to No Rewards PND Rewards to No Rewards Lee 75% 75% 0% Sara 42% 8% 0% Marita 25% 17% 0% Sequence B Participant PND Baseline to No Rewards PND Baseline to Rewards PND No Rewards to Rewards Ty 17% 25% 0% Chris 17% 25% 17% Phineas 8% 42% 25% 118 Table 19 Percent of non-overlapping data for generalization passages across phases Sequence A Participant PND Baseline to Rewards PND Baseline to No Rewards PND Rewards to No Rewards Lee 38% 88% 0% Sara 13% 0% 0% Marita 75% 63% 0% Sequence B Participant PND Baseline to No Rewards PND Baseline to Rewards PND No Rewards to Rewards Ty 63% 25% 0% Chris 13% 63% 0% Phineas 13% 50% 50% 119 Appendix D: Additional Figures Figure 17. Comparison of direct gains on instructional passages between reads 1 and 4 (within-session gains), Sequence A; * a one-week absence occurred after this data point. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details. 120 Figure 18. Comparison of direct gains on instructional passages between reads 1 and 4 (within-session gains), Sequence B; * a two-week absence occurred after this data point. The data are presented concurrently for simplicity, but student absences rendered disruptions to the data collection schedule. Please see the Research Design section for details.
Thesis/Dissertation
eng
School Psychology
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
The use of rewards in reading fluency interventions: considerations for student achievement and motivation
Text