Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

A multiple measurement outcome evaluation of a behaviorally oriented inservice training program Lee, Douglas Spencer 1983

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
831-UBC_1983_A8 L44.pdf [ 5.95MB ]
Metadata
JSON: 831-1.0095832.json
JSON-LD: 831-1.0095832-ld.json
RDF/XML (Pretty): 831-1.0095832-rdf.xml
RDF/JSON: 831-1.0095832-rdf.json
Turtle: 831-1.0095832-turtle.txt
N-Triples: 831-1.0095832-rdf-ntriples.txt
Original Record: 831-1.0095832-source.json
Full Text
831-1.0095832-fulltext.txt
Citation
831-1.0095832.ris

Full Text

A MULTIPLE MEASUREMENT OUTCOME EVALUATION OF A HEHAVIORALLY ORIENTED IN SERVICE TRAINING PROGRAM by DOUGLAS SPENCER LEE B.A., THE UNIVERSITY OF BRITISH COLUMBIA, 1974 M.Ed., THE UNIVERSITY OF BRITISH COLUMBIA, 1978 A THESIS SUBMITTED FOR PARTIAL FULFILLMENT OF THE REQUIREMENT FOR THE DEGREE OF MASTER OF ARTS in THE FACULTY OF GRADUATE STUDIES (Department of Psychology) We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA Feb. 1983 © DOUGLAS SPENCER LEE, 1983 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make i t freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the Head of my department or by his or her representative. It is understood that copying or publication of this thesis'for financial gain shall not be allowed without my written permission. The University of British Columbia 2075 Wesbrook Place Vancouver, B.C. Canada V6T 1W5 signed Department of DE-6 ABSTRACT This study investigated the immediate and long term impact of a portable, behaviorally-oriented, inservice t r a i n i n g program. The t r a i n i n g program consisted of a s i t e v i s i t (pre-test), 5 day workshop (mid-test), 5 follow-up v i s i t s , and a termination v i s i t (post-test). The termination v i s i t occurred 6-8 weeks following the l a s t follow-up. Each t r a i n i n g program spanned approximately 8-9 months. The t r a i n i n g package consisted of lectures, films or video-tapes, demonstrations, role-play, behavioral rehearsal, and homework. Major areas covered during the t r a i n i n g were teaching techniques ( s p e c i f i c a l l y Discrete T r i a l Format teaching), program wr i t i n g , language t r a i n i n g , curriculum development, reinforcement p r i n c i p l e s , maintenance and generalization, behavior management and token economy. The t r a i n i n g team was composed of a coordinator, psychologist, teacher and community nurse. Thirty-two d i r e c t service s t a f f for severely handicapped c l i e n t s served as trainees. These trainees were from t r a i n i n g programs conducted i n four communities i n the Province of B r i t i s h Columbia from 1979-1981. Data on three dependent measures plus consumer s a t i s f a c t i o n measures were gathered using a pre-mid-post i n s t i t u t i o n a l cycle design. The three dependent measures were behavioral terminology, Discrete T r i a l Format teaching and program wr i t i n g . Consumer s a t i s f a c t i o n data were collected at the completion of the workshop and at the termination v i s i t . For the terminology and Discrete T r i a l Format teaching measures s i g n i f i c a n t increases were found at mid-test and post-test i n the trainees' scores and i n the proportion of trainees exceeding pre-established c r i t e r i a for these measures. No pre-test program wri t i n g data were available. There was a s i g n i f i c a n t decrease i n the program w r i t i n g scores from mid to post-test. Consumer s a t i s f a c t i o n data was uniformly p o s i t i v e at both measurement periods. The significance of these r e s u l t s are discussed i n r e l a t i o n to previous research done i n t h i s area. Inferences are made about how factors such as trainee employment contracts, teacher schedules, and reinforcement schedules may have influenced the r e s u l t s . Problems i n conducting t h i s type of research under f i e l d conditions are i d e n t i f i e d and pot e n t i a l areas for further research are suggested. Advisor iv TABLE OF CONTENTS PAGE ABSTRACT i i LIST OF TABLES v i LIST OF FIGURES v i i ACKNOWLEDGEMENTS v i i i INTRODUCTION . 1 LITERATURE REVIEW 10 Lecture vs. Roleplay 15 Lecture vs. Lecture and Modeling 15 Modeling vs. Modeling and Lecture 18 Programmatic Packages 20 Summary of Review 28 HYPOTHESES 31 METHOD 33 Selection of Subjects 33 Trainees 33 Target c l i e n t s of trainees 33 Settings 36 Description of Training Team 36 Training Program Description 4-1 41 Site v i s i t Workshop 41 Follow-up v i s i t s 42 Design 44 V PAGE Measures 47 Knowledge of terminology 47 Program writing 47 DTF teaching s k i l l 4-8 Consumer satisfaction: workshop feedback 4S Consumer satisfaction: follow-up feedback Reliability 49 RESULTS 52 Knowledge of Behavioral Terminology 54 Discrete Trial Format Teaching 54 Program Writing 6 l Tests on Proportion of Trainees Meeting Criteria 6l Proportion of trainees exceeding terminology test criterion... 64 Proportion of trainees exceeding DTF criterion 64 Proportion of trainees exceeding program writing criterion.... 64 Consumer Satisfaction 68 DISCUSSION 71 REFERENCE NOTES , 78 REFERENCES 79 APPENDICES 87 APPENDIX A 8£ APPENDIX B 1 0 8 APPENDIX C n 4 APPENDIX D 1 : L9 APPENDIX E 128 APPENDIX F 134 APPENDIX G 137 APPENDIX H ! 4 5 v i LIST OF TABLES TABLE PAGE 1. Training Programs: Experimental Designs, Populations, and Treatment Outcomes 11 2. Training Group Composition by Occupational Category of Trainee 34 3« Age, Sex and Diagnosis of Target Clients 35 2). Schedule of Workshop Training Components 38 5. Total Time Allotment of Training 43 6. Contingency Tables for Computing Phi Coefficient for the R e l i a b i l i t y of the Three Dependent Measures 50 7- Size of Included and Excluded Groups and Group Mean Scores for Workshop Participants on Dependent Measures 53 8. Mean Percentage Correct on Terminology Measure 55 9. ANOVA on Terminology Data 56 10. Mean Percentage Correct on DTF Measure 57 11. ANOVA on DTF Data 58 12. Mean Percentage Correct on Program Writing Measure 62 13. ANOVA on Program Writing Data 63 14. Contingency Tables of Trainees Meeting 80$ Pass C r i t e r i o n for Terminology Measure 65 15. Contingency Tables of Trainees Meeting 80% Pass C r i t e r i o n for DTF Measure 66 16. Contingency Table of Trainees Meeting 70$ Pass C r i t e r i o n on Program Writing Measure 7^ 17. Mean Consumer Sat i s f a c t i o n Ratings for the Workshop and Follow-up V i s i t s 69 18. ANOVA's on Workshop and Follow-up Consumer Sat i s f a c t i o n Data ... 70 v i i LIST OF FIGURES FIGURE PAGE 1. The Institutional Cycle Design Used to Study the Impact of the Provincial Team's Training Program 45 2. Interaction of Group Mean Scores on DTF Measure 59 ACKNOWLEDGEMENTS The hard work and generosity of many people made t h i s project possible. I am deeply grateful to my advisor, Ken Craig, and thesis committee members, Bob McMahon and Susan Butt for t h e i r guidance and feedback which helped d i r e c t me to t h i s f i n a l product. My sincerest thanks go to Shari Bender, Larry Bierbach, Dot Ewen, Jane Kelly , Karl P e r r i n , Barbara Thomas-Bruzzese, and Debbie Wilson, members of the P r o v i n c i a l Inservice Resource Teams without whose dedicated work t h i s research would not have been possible. F i n a l l y , my love to Pat and the g i r l s , Carol, Dianne and Monica for putting up with me when I was either staring blankly at walls or running into them. 1 INTRODUCTION The history of theoretical concepts and systematic practices concerning care and training for the mentally retarded dates back to the early nineteenth century. This history, which has been well documented by Crisen (1975), Kanner (1967), and Rosen, Clark and Kivitz (1976) can be divided into four relatively distinct phases. 1) 1800-1850 The pioneering work was begun by Itard in France, Sequin in France and the United States, Guggenbuhl in Switzerland and Howe in the United States. Due to the humane and enlightened care given the mentally retarded during this period i t became labelled the moral treatment era. Itard and Guggenbuhl both i n i t i a l l y concieved of mental deficiencies as curable. They saw mental deficiencies as stemming largely from the person's lack of exposure to needed levels of sensory stimulation. Following this concept they devised sensory stimulation programs and sensory training programs within general education systems. The development of programs of this type led Itard and Guggenbuhl to develop special teaching techniques, in order to give their pupils the needed sensory training. Although Itard and Guggenbuhl became disheartened by their inability to "cure" their pupils the demonstrable progress of their pupils greatly impressed the medical and scientific communities of their time. Sequin and Howe continued the pioneering work of developing educational strategies for the mentally retarded. Although Sequin endorsed the concept that mental retardation could be cured, Howe proposed a more moderate approach focused on the amelioration of the handicap. 2) 1850-1900 The early isolated programs of these four pioneers prompted a 2 positive attitude towards the teaching of the mentally retarded. The treatment f a c i l i t i e s started by these pioneers were small and oriented towards each i n d i v i d u a l student. This i n i t i a l concept of small and i n d i v i d u a l l y oriented resources was short l i v e d , as many of the f a c i l i t i e s quickly expanded and became overcrowded. The rapid growth of these small centers into large i n s t i t u t i o n s during t h i s time can be exemplified by t h e i r development i n the United States. P r i o r to Howe's work i n the mid 1800*s there were no f a c i l i t i e s i n the United States for the mentally retarded. Howe started his residence i n 1848. Over the next 40 years 15 f a c i l i t i e s were opened. By 1898 twenty-four public i n s t i t u t i o n s were maintained by 19 states and one by the C i t y of New York. By 1917 a l l but four states were making some i n s t i t u t i o n a l provision. In t o t a l , approximately 37,200 patients were being cared f o r . 3) 1900-1950 The generally positive attitude about the nature of mental retardation, which was cha r a c t e r i s t i c of the early 1800's, had shifted by the turn of the century. This s h i f t culminated i n a movement based on Galton's concept of eugenics, that was antagonistic to the w e l l being of the mentally retarded. The most harmful interpretation of Galton's concept saw the unchecked f e r t i l i t y of the " u n f i t " (e.g., the mentally retarded) as a threat to the continuation of the positive inherited characteristics of the human race. This new movement was very active through to the 1930's and promoted a transformation of the o r i g i n a l positive concepts of retardation and the type of care they should recieve. The new concept of retardation inappropriately linked the mentally retarded to a wide range of s o c i a l maladies such as p r o s t i t u t i o n , robbery and vagrancy. I t was also suggested that mentally retarded parents were the major and growing source of mentally retarded 3 offspring. Following t h i s s h i f t i n thought, i n s t i t u t i o n s increasingly became custodial care f a c i l i t i e s more for the protection of society than the care and education of t h e i r residents. Although t h i s movement l o s t i t s force i n the 1940's, as new studies i n genetics provided evidence contra-dictory to the eugenics concept, the attitudes and services which developed during the early nineteen hundreds persisted into the late 1950's. O 1950 - present A number of forces started to surface during the 1950's which challenged the established b e l i e f s regarding the care and treatment of the mentally retarded. The development of i n d i v i d u a l i z e d i n t e l l i g e n c e t e s t i n g , which started i n the early 1900's, had yielded test data which led to viewing retardation as a gradient and not as a uniform syndrome. In conjunction with t h i s change, specialized syndromes such as autism (Kanner, 1943) were is o l a t e d . DSM I I I (1980) now c l e a r l y distinguishes among four l e v e l s of retardation and among retardation, autism, and other pervasive or s p e c i f i c developmental disorders. From the 1930's onward accumulating evidence suggested that IQ's were not fi x e d but variable. Researchers also started to suggest that besides genetic factors, there were other factors such as poor n u t r i t i o n , inadequate health care, and premature b i r t h which might be s i g n i f i c a n t as e t i o l o g i c a l factors i n determining a person's l e v e l of i n t e l l i g e n c e . Among with the changing s c i e n t i f i c perspectives to mental retardation changes i n attitudes towards the mentally retarded were occurring i n society as w e l l . Coinciding with the growing c i v i l rights movements, a major force towards better treatment for the mentally handicapped (e.g., the mentally retarded, a u t i s t i c and developmentally disabled) was the development of parents' associations. For example, the National Association for Retarded Citizens i n the United States began as a small h group in 1950 but had approximately 50,000 members by 1959. The appointment of the President's Panel on Mental Retardation and their report in 1962 signified a major shift away from the concepts supporting institutional and custodial care for the mentally retarded. The report provided an outline of a national plan for services to the mentally retarded with a specific reference for the need to improve education programs and increase their availability. Thus, by the early 1960's a number of movements were well defined which, from various positions, were attacking the predominately negative attitudes and practices towards the handicapped. These attacks centered on the lack of training and remediation given to the retarded and the almost exclusive use of custodial care facilities as long term residences for this population. A l l these developments led to a resurgence of a positive attitude towards, and more humane treatment for, the mentally handicapped. Two particular developments, originating in the early 1960's, have special relevance for the education and training of the mentally handicapped. The first development was the early demonstrations that operant techniques could be used to teach the mentally retarded new behaviors. The second development was the proposal of the normalization principle (Wolfensberger, 1972). The normalization principle states that the mentally handicapped should be offered patterns and conditions of everyday l i f e which are as close as possible to the identified norms and patterns of the mainstream of society. . Thus the normalization philosophy i s the antithesis of a custodial care philosophy. The combination of the normalization philosophy with an operant technology, which provides teaching techniques for the mentally handicapped, nas formed an educational movement for training the mentally handicapped to lead more independent and satisfying lives. 5 A major component of society's present commitment to the mentally handicapped deals with the rights of these individuals to an education. In the United States ,court and legislative rulings, including the Pennsylvania Right to Education decision (l97l)» the Wyatt vs. Stickney decision in Alabama (1971), and Public Law 94-142 (1975), which legislated the right of a l l handicapped children to an adequate education, exemplify the present commitment (Martin, 198l) . Although a different legal system prevails in Canada, a commitment to education for the mentally handicapped can be seen in the actions of several provincial education departments which have enacted legislation requiring the provision of educational opportunities for the mentally handicapped (Shepherd, 1982). Fulfilling our responsibility to provide adequate education for the small proportion of the population which exhibits severely and profoundly retarded behavior (Grossman, 1977) or autistic behavior (Wing, Yeates, Brierley & Gould, 1976) is a major task. Kaufman (1981), dealing with the issue of whether a l l children were educable, listed eight conditions which make instruction of the severely mentally handicapped particularly difficult. His l i s t follows: 1. A l l function at the extreme lower level of cognitive attainment and adaptive behavior. 2. Most have not acquired basic self-care skills. 3« Most are not known to many of the normal residents of their communities. 4 . Most would never attend a public school unless the law specifically required their enrolment. 5. Most would have survived only a short time a few decades ago but have a lif e expectancy of decades due to modern medical intervention. 6. Most were, until recently, considered "untrainable" or "custodial" cases. 6 7. Most are non-ambulatory. 8. Most show extremely l i t t l e promise of becoming creative, productive citizens even with the most heroic efforts of today's most skilled behavior therapists. The autistic population has perceptual and cognitive impairments not found in the severely and profoundly retarded (Hermelin & O'Connor, 1970). The major behavioral indices of autism as listed in DSM III (1980) are: pervasive lack of responsiveness to other people, gross deficits in language development, and bizarre responses to various aspects of the environment. According to DSM III a differential diagnosis is possible between autism and retardation, although there is often substantial overlap between the two disabilities with 70% of autistic individuals having IQ's less than 70, and 40% with IQ's less than 50. DSM III allows for a joint diagnosis of both autism and mental retardation. Attempts to deal with the above conditions, with a goal of providing adequate education for the severely and profoundly retarded and autistic populations, have led to the development of specialized teaching techniques (Koegel, Russo & Rincover, 1977; Liberty, Haring & Martin, 1981; Lindsley, 1964). The development of operant teaching techniques for the severely mentally handicapped began in the late 1950's and early 1960's. Literature reviews on teaching techniques used with the severely mentally handicapped indicate that behaviorally based teaching techniques predominate within the field (Berkson & Landesman-Dwyer, 1977; Birnbrauer, 1976; Koegel, Egel & Dunlap, 1980; Lovaas & Koegel, 1972). In a review from 1977-1981 of four major journals specializing in mental retardation and autism (American Journal of  Mental Deficiency; Journal of Autism and Developmental Disorders; Journal  of Mental Deficiency Research; Mental Retardation) i t was found that 86% 7 of the articles (n=2<2) specifically dealing with teaching techniques for the severely mentally handicapped populations dealt with behaviorally based teaching techniques. Treatments for autistic individuals have been formulated from psychodynamic, developmental, and physiologic arousal theories, but these have failed to gather empirical support (Lovaas, 1979; Lovaas & Koegel, 1972; Kazdin, 1975; Rimland, 1964). The reviews referenced above indicated that, over the last 20 years, behaviorally based teaching techniques have been used for teaching a wide variety of skills to the mentally handicapped. This breadth of applicability continues to the present. Work in such areas as communication skills (Cuvo & Pdva, 1980; Hung, 1980; Lovaas, 1977), self-help skills (Nelson, Gergenti & Hollander, 1980; Nutter & Reid, 1978; Schleien, Ash, Kierman & Wehman, 1981), vocational skills (Martin, Pallotta-Cunick, Johnstone & Goyos, 1980; Mithaug & Mar, 1980), socialization skills (Strain, Kerr & Ragland, 1979), independent recreational skills (Hill, Wehman & Horst, 1982), and gross motor skills (Gruber, Resser & Reid, 1979) indicate the variety of areas where behaviorally based teaching techniques have recently been used with the severely mentally handicapped. Although behaviorally based techniques have been the treatment of choice to date i t does not mean these techniques are without problems. Generalization and maintenance of teaching gains is a major concern (Rincover & Koegel, 1975; Koegel & Rincover, 1917). The large number of teaching trials, often in the thousands, needed to teach complex behaviors such as language (Goetz, Schuler & Sailor, 1979) may limit the practicality of teaching complex behaviors with the present technology. Finally, the technology has often not proven effective for teaching the client to emit a behavior with the proper latency, intensity or duration to allow the behavior to be easily 8 integrated by others into an ongoing social interaction (Liberty et a l . , 1981). The amount of research evidence supporting the efficacy of behaviorally based teaching techniques i s very encouraging. Yet, i f the severely mentally handicapped are to receive maximum benefit from exposure to these techniques the direct service staff working with these populations must be ski l l e d in their application of behavioral techniques. However, empirical evaluations of training programs designed to teach behaviorally based teaching techniques to direct service staff working with the severely mentally handicapped have been scarce (Donnellan, 1980; Gardner, 1972; Gladstone & Sherman, 1975; Panyon & Patterson, 1974). The need for inservice training i s particularly important at this time as pre-service training appears to be inadequate for preparing staff to work with these populations (Fredericks, Anderson & Baldwin, 1979; Haring, 1980). In the concluding remarks to their study attempting to define competency indicators of teachers of the moderately to profoundly retarded, Fredericks et. a l . , (1979) stated: "There i s nothing i n our experience, nor in the literature, to suggest that students who leave teacher-training institutions have the knowledge and experience to synthesize curricula, philosophy, and management techniques into a smoothly functioning, effi c i e n t , and successful educational system (p.93)." The purpose of the present study was to measure the outcome of a highly portable inservice training program. This training program was conducted by a multidisciplinary team consisting of a psychologist, teacher, and community nurse. This team was trained by members of the Santa Barbara Autism Dissemination Project (Donnellan, LaVigna, Schuler & Woodward, Note l) to conduct training programs using the Santa Barbara 9 model of iriservice training for staff working directly with autistic and other severely mentally handicapped people. The Santa Barbara model used a 5 day workshop and a series of follow-up visits to ensure the maintenance and generalization of the techniques learned during the workshop to the trainees 1 natural working environment. 10 LITERATURE REVIEW The intent of this review i s to summarize research on the evaluation of training programs for direct service personnel who work with severely mentally handicapped populations. This review was conducted through the Psychological Abstracts from 1966-1981 and a direct search of journals specializing i n the severely mentally handicapped and behavior modification. Several of the journals i n these areas have only been published for brief periods of time. The journals specifically reviewed were: American  Association for the Education of the Severely Profoundly Handicapped Review, 1979-1980; American Journal of Mental Deficiency, 1970-1982; Journal of  Abnormal Child Psychology, 1974-1982; Journal of Applied Behavior Analysis, 1970-1982; Mental Retardation, 1970-1982. A total of nine articles was found which empirically investigated the effectiveness of programs for training staff to teach severely mentally handicapped, clients. A l l nine programs focused on training staff to use behaviorally- based teaching techniques. The eleven investigations reported i n these nine papers (Table l) were categorized as attempts to research the effects of either isolated training components or the effects of treatment packages. As the majority of the studies focused on uniquely constructed and complex training methods as independent variables i t was d i f f i c u l t to develop a systematic review. For example, Fredericks et a l . (1979), Gardner (1972), Gladstone and Sherman (1975)f Koegel, Glahn and Nieminen (1978), Panyon and Patterson (1974), Schinke and Wong (1977), and Weissman-Frisch, Crowell and Inman (1980) used lectures i n their training packages. However, only Gardner (1972) and Panyon and Patterson (1974) focused specifically on the lecture component or indicated the duration of the lecture component. Furthermore,.the other TABLE 1 Training Programs: Experimental Designs, Populations, Training and Results Authors Trainees Target Clients Training Techniques Research Design Dependent Variables Foil Up Results Gardner attendant ( 1 9 7 2 ) 2 groups of 1 0 2 0 retarded pop. Panyon & attendant Patterson (1974) attendant 1 5 Gladstone high sch. 7 & Sherman stud. ( 1 9 7 5 ) retarded children profound retarded children profound retarded children 1 4 1 . role-play 6 hrs 2 . lectures 8 hrs 2 group counter-balanced pre/post compar. 1 . instruction multiple 2 . VTR Feedbk. baseline 3. VTR model 1 . lecture only pre/post 2 . lec/live group model compare 3. lec/VTR mod. lectures, VTR model, behav. rehearsal & correct feed-back, praise multiple baseline 1 . role-play play test 2 . knowledge test No beh. obs. of reinf. tech. & child % corr. beh. obs. of teach-ing s k i l l No No beh. obs. of teach-ing s k i l l Yes role-play train, incr. role-play proficiency: lectures incr. know-ledge: order of pres. no d i f f . linear incr. . of reinf. s k i l l & child % corr. across methods both lec/ model equal & both sig. better than lec. only: no child data incr. s k i l l in use of contingent reinf. & instruction giving: pos. changes in child behs. results gen. to other child. TABLE 1 CONT. Authors Trainees N Target Clients Training Techniques Research Design Dependent Foil Results Variables up knowledge No trained home test, att. (n=6) had checklist, sig. improve job satis- on att. check faction and pos. beh. scale, beh. change of obs. of staff and staff and residents residents compared to control homes (n=6).: both groups deer, in job satis, but trained home staff sig. less. beh. obs. No consistent of teach increase in skill & teach, skill child's and child's percent percent corr.: correct teachers skill gener. across chil-dren & behs. taught. beh.obs. No rate of of teach teaching interact. interact. & percent incr. but time not only maint. teaching for spec, beh. Schinke & Wong ( 1 9 7 7 ) Koegel, Russo & Rincover ( 1 9 7 7 ) Fabry & Reid ( 1 9 7 8 ) group-home staff of 1 2 homes 3 0 retarded adults teachers 1 1 autistic children 1 2 foster-grandpar-ents 5 profound retarded adolesc. lectures, case examples, role play, homework 1 2 hrs total 2 group compar. manual, VTR model, beh. rehearsal 2 5 hrs. or less total multi-response baseline instruction, live model, prompts, praise multiple baseline reversal TABLE 1 CONT. Authors Koegel, Glahn & Nieminen (1978) Trainees mothers therapist Fredericks Anderson & Baldwin (1979) teachers Target N Training Research Dependent F o i l Results Clients Techniques Design Variables up autistic 7 1. brief multi- beh. obs. No increase in children modeling response of teach. teach, s k i l l demos. baseline s k i l l & and child's 2, general child's percent corr. training, percent consistent lectures correct only after VTR model general training.: mothers gen. across behs. and children. autistic 6 1. VTR model multi- beh. obs. No increase in children of antece- response of teach. teach s k i l l dent teach baseline s k i l l & dependent on behaviors order of child's type of model 2. VTR model present. percent viewed.: of conse- varied correct. child's perc. quence correct incr. teacher only after behavior the s k i l l incr. in a l l areas. severely lectures, live pre/post beh. obs. of Yes 5 teachers handicap. VTR modeling, compari. vari. teach. met 7/7 c r i t -students beh. rehearsal s k i l l s , rating eria. , 2 tea-with feedback of program chers met 6/7.: 4«5 day train. development, teachers maint. program SPR curr- s k i l l at 1 yr. iculum. follow -H ip .: pos. student gain found. TABLE 1 CONT. Authors Trainees Target Clients Training Techniques Research Design Dependent Variables F o i l up Results Weissman-Frisch, Crowell, & Inman (1980) direct ser- 50 vice, ad-ministrator & others 5-7 S's per group severe/ profound adults didactic, live & VTR modeling, beh. reh. with feedbk., role play, homework. 4.5 day train program pre/post compar. beh. obs., Yes consistent rate scale., pos. increase termin. test, across a l l goal attain- measures ment index, from pre to attitude post.: scale re- 10-12 week clients, foil-up by consumer tele, interv. feedback, 86-91$ of S's telephone indicated interview. use of tech. from train program.: 60-64$ of S's met their own goals. 15 t r a i n i n g programs with a lecture component had unique constellations of other components making up t h e i r f u l l t r a i n i n g program. Although the l i t e r a t u r e review has been divided into four sections, the complexity of the t r a i n i n g programs necessitated some overlap between categories. The four sections of t h i s review examine dif f e r e n t components of i n s t r u c t i o n a l methodology: lecture vs. roleplay; lecture vs. lecture and modeling; modeling vs. modeling and lectures; and programmatic packages. Lecture vs. Roleplay Gardner (1972) compared the use of d i r e c t versus i n d i r e c t t r a i n i n g methods for increasing i n s t i t u t i o n a l attendants' proficiency i n the use of behavioral teaching techniques (giving i n s t r u c t i o n s , prompting, shaping and data co l l e c t i o n ) and knowledge of behavioral p r i n c i p l e s . The t r a i n i n g methods were s i x 1-hour roleplay sessions and eight 1-hour lectures. I t was hypothesized that the d i r e c t methods (roleplay for increasing proficiency and lectures for increasing knowledge) would be more effective than the i n d i r e c t methods (roleplay f o r increasing knowledge and lectures for increasing p r o f i c i e n c y ) . The two groups of attendants (n=10) received the t r a i n i n g methods i n counterbalanced order. Regardless of the order of presentation roleplay was s i g n i f i c a n t l y better than lectures for increasing proficiency and lectures were s i g n i f i c a n t l y better than roleplay at increasing knowledge. Lecture vs. Lecture and Modeling Panyon and Patterson (1974) conducted two experiments which investigated the effects of modeling ( l i v e or videotaped) compared to videotaped playback of the trainees' own behavior, and instructions or lectures. As discussed by Bandura (1977), modeling i s a very powerful method for teaching new behaviors, as i t allows a person to develop symbolic representations of the modeled behavior which can be used later as cognitive guides during the reproduction of the modeled behavior. In the first experiment a • multiple baseline across three trainees (institutional attendants) was used. Tv'0 trainees received a training series of instructions in behavioral techniques (giving instructions and reinforcement), video-taped playback of the trainees own teaching behavior, and a final phase of live modeling. A third trainee received an instruction only phase followed by the live model training phase. The dependent measures were the trainees mean percentage correct for use of praise or verbal comment, primary reinforcement, and physical contact; and the percentage of correct or incorrect child responses. Due to the training phases not being counterbalanced for order of presentation i t was not possible to evaluate the single training components. However, since trainee 3 did not receive the videotaped playback phase and s t i l l showed as large an increase in mean percentage correct as the other two trainees, the data suggested that the videotaped playback only phase was not required. The childrens' mean percent correct also linearly increased across phases. In their second experiment Panyon and Patterson (1974) compared three training methods: 4 hours of lectures only, lectures plus film model, and lectures plus live model. A between groups design with three groups (n=5) of attendants as trainees was used. The dependent measures were the same as in their first study for the trainees* behavior. The results, analyzed using gain scores, indicated that both the lecture plus live model and lecture plus film model were equally successful and both were significantly better than the lecture only group at raising the trainees percent correct use of reinforcement. No data on the target childrens' behavior were given. The results from this study indicated that a lecture or instruction plus model training format was effective for increasing the correct use of reinforcement. Fabry and Reid (1978) used an i n i t i a l instruction only phase in their training program of foster grandparents. However, in this phase, the instructions were about a c t i v i t i e s the foster grandparents could use to teach fine and gross motor s k i l l s to their mentally and physically handicapped clients. Following this i n i t i a l stage, a training package involving l i v e modeling, prompts, and praise was introduced to increase the foster grandparents proficiency i n using behavioral teaching techniques (giving of instructions, prompts, and reinforcement). The mean percentage of intervals that grandparents engaged in teaching behaviors per session was the dependent measure. A multiple baseline design was used with the training for the grandparents introduced sequentially across three motor areas (head and neck movement, reaching, and manipulating objects). An increase i n the percentage of intervals the grandparents engaged i n teaching behaviors occurred only following the introduction of the training program across each of the three motor areas. Assessments of the residents' s k i l l in fine and gross motor movements did not improve u n t i l the implementation of the training package for the foster grandparents had taken place. These studies (Gardner, 1972; Fabry & Reid, 1978; Panyon & Patterson, 1974) indicated that, when used in isolation, lectures or instructions, either about specific behavioral techniques or about ac t i v i t i e s to use for teaching severely mentally handicapped clients, do not result i n trainees increasing their teaching proficiency. Only when training techniques were introduced that allowed the trainees to roleplay or observe a model doing the teaching was proficiency increased. Positive changes in the target client's behavior 18 also did not increase until modeling of the teaching techniques was done for the trainees. However, lectures appeared to be more effective at increasing knowledge of behavioral principles than a roleplay training format. Modeling vs. Modeling and Lecture Koegel et al. (1978) conducted two experiments on the effects of modeling alone and modeling with lectures. The first experiment contrasted the effectiveness of a series of brief modeling demonstrations of a behavioral technique to a package of three 30-min. lectures describing the teaching technique in general, plus two 37-min. videotaped models, correctly and incorrectly using the components of the teaching technique. The behavioral teaching technique contained five components: (instruction or cue), prompts, shaping, consequences and discrete trials. Four parents served as trainees and the target clients were six autistic children. This experiment had two purposes; to assess the effectiveness of both formats for training the parents to teach a variety of target behaviors, and to assess i f the parents could generalize their behavior across children. Two dependent measures were taken. The first measure was based on the parents use of the five components of the teaching technique. The teaching sessions were broken into 30-sec. intervals and the parents' score for each component was computed for each 30-sec. interval, summed across a l l intervals and then averaged across a l l five components, to give an overall index of the parent's behavior per session. The second dependent measure involved computing the percentage of correct responses the children made to the parent's teaching. Using a multiresponse baseline design, each of the parents received from one to four brief demonstrations before receiving the general training package. The results indicated that the parents maintained the proper use 19 of the technique only following the general training package. To obtain a high proficiency level with the teaching technique prior to receiving the general training package the parents needed brief demonstrations of the technique each time they attempted to teach a new target behavior. As a l l parents had at least one session of brief demonstration prior to the general training package i t is not clear whether a general training only phase would have produced the same maintainance effects as a combined brief modeling then general training package. During the brief demonstration phase the childrens' percentage of correct responding increased only following a demonstration for the parent of the correct procedure, focusing on the specific target behavior. Following the general training package the childrens' percentage of correct responding was maintained at a consistently high level, concurrent with the parents maintained high level of proficiency with the teaching technique. The second experiment by Koegel et al. (1978) attempted to evaluate the effects of the two modeling videotapes used in their first study. One tape focused on the antecedent components of the teaching technique while the other focused on the consequent components. At the viewing of each tape the three subjects (teacher, foster mother, and undergraduate) were given a manual parapharasing the script of the appropriate tape. The dependent measure was the same as in the previous experiment except the use of shaping was not measured and data were' kept on the separate components of the teaching technique. Two subjects viewed the antecedent then consequence tapes, while the third subject viewed the tapes in the reverse order. Improvements in the correct application of the. components were dependent upon viewing the appropriate tape, regardless of presentation order. This result indicated that training focusing specifically on the components to be taught, is necessary before proficiency in using those components is demonstrated. 20 Data on the percentage correct of unprompted child responses revealed that following the trainees' exposure to one modeling tape the children's percentage correct increased over baseline but the children's percentage correct was generally highest after the trainees were exposed to both films. The results from Koegel et al. (1978) extend the information regarding the effects of modeling on teaching proficiency. Although brief focused modeling, lecture plus modeling packages, or video modeling of a generalized technique plus written scripts a l l increased teaching proficiency, only modeling of the technique under generalized conditions resulted in the trainees demonstrating maintained high levels of proficiency with the technique across different target behaviors and children. Brief focused modeling did not appear to provide the trainees with a learning condition which facilitated their development of a maintained proficiency with the teaching technique. The lecture and video modeling and video modeling with written script were equally effective at increasing trainee's proficiency, resulting in increased correct responding by children. Programmatic Packages The remaining studies a l l used multiple dependent measures and complex training packages as independent variables. Schinke and Wong (1977) used a training package of lectures, case examples, roleplays, weekly readings from the training book "Families" (Patterson, 1975)> plus resident behavior change homework assignments to train group home staff in six homes for educable and trainable mentally retarded adults. Group home staff in six other homes provided a waiting l i s t control group. The homes were randomly assigned to either the training or control conditions. The training took place in eight 1.5-hr. weekly sessions held at the group homes. The dependent measures were a knowledge test of behavioral concepts and techniques, an attitude checklist of staffs' attitudes towards the residents, a job satisfaction rating scale, and naturalistic observations of 29 staff and resident behaviors. These 29 observation categories were collapsed for analysis into three general categories of positive, neutral, and negative behaviors. As expected from such a training package there were significant pre-post gain score differences for the trained staff compared to the untrained staff on the knowledge test and on observations of the staffs' proficiency in adjusting their consequent behavior (positive, neutral, and negative) in relation to the residents' antecedent positive, neutral, and negative behaviors. Changes in overall rates of positive and negative behaviors in the trained homes were found. The trained staff showed significant increases in positive behaviors and significant decreases in negative behaviors compared to the untrained staff. Increased positive and decreased negative behavior rates of the residents were found concommitantly with the behavior changes of the trained staff. Significant positive increases in the trained staffs' attitude towards the residents were also found. An interesting finding was that job satisfaction ratings decreased pre to post for both groups, with the control-group's decrease being significantly larger than the experimental group. Although the data on the knowledge test and naturalistic observations would appear consistent with the studies reviewed above, the use of multiple (37) univariate t-tests instead of multivariate statistics hampered a clear interpretation of the results. Gladstone and Sherman (1975) trained seven volunteer high school students to work with profoundly retarded children. This study investigated the effects of a training package consisting of a 25-min. videotaped model, behavior rehearsal, and corrective feedback every 5 minutes of rehearsal on the trainees' use of behavior modification techniques to teach profoundly retarded children. Following the i n i t i a l training, which focused on one child, the trainees* ability to generalize their skills was assessed by having them teach a different behavior to a different child. A multiple baseline design across trainees was used. The dependent measures used by Gladstone and Sherman (1975) were observations of the trainees' ability to give correct instructions, give reinforcers, not reward incorrect responses, use physical prompts, and ignore inappropriate and disruptive behavior. The childrens' percentage of correct responses was also used as a dependent measure. Of the five dependent measures focusing on the trainees* behaviors only the use of instructions and contingent reinforcement showed consistent and substantial changes following training. The mean number of instructions given per session decreased by approximately 65$ following training. The mean number of contingent reinforcers given per session increased by approximately 90$ following training. These results were consistent across both target clients for each trainee, indicating that the trainees were able to generalize their behavior across children and behaviors. The percentage of instructions followed by the children was variable. Four of the children in the i n i t i a l training phase reached the 80$ pass criterion while the other three children showed no improvement. However, during the generalization phase a l l children showed positive increases with six of the seven children reaching the 80$ pass criterion. An interesting point was that success in generalizing the teaching behavior to a second child was not dependent on the trainees' success with 23 teaching the first child. The three trainees who were unsuccessful in teaching their first child were able to teach their second chi.ld to criterion. However, the results on the ability of the trainees to generalize their behavior may be confounded, as only two target behaviors were used and they were not counterbalanced for order of presentation. Koegel et al . (1977) investigated the effects of a programmatic package which focused on teaching behaviors similar to those studied by Gladstone and Sherman (1975)• The experiment was designed to assess whether teachers could be taught to use behavioral teaching techniques correctly and whether correct use of the techniques had an effect on the target child's behavior. The training package consisted of a training manual, videotaped modeling, and behavior rehearsal with feedback every five minutes, with more elaborate feedback every 30 minutes. The training time was variable but a l l trainees completed the training within 25 hours. Eleven teachers were used as trainees and 12 autistic children acted as target clients. The dependent measures focusing on teacher proficiency and child's percentage correct responding were the same as in the Koegel et al . (1978) study reviewed above. A modified multiresponse baseline design was used across trainees with the target children and behaviors changing for each session. The results indicated that the training package was effective in raising the teacher's level of proficiency to a consistently high level (over 80% correct). Measurement of the childrens' behavior revealed a positive effect of higher and more consistent correct responding following the teachers'.completing the training package. The results found by Koegel et al. (1977) indicated that the hypothesis that teachers use correct behavioral procedures and that the children can not learn was incorrect. It appeared that, without training, teachers do not use behavioral teaching techniques properly and that under this con-dition the autistic children do not learn. However, when trained in the proper implementation of a behavioral teaching technique, teachers demonstrated a high level of proficiency with a resultant positive effect on the autistic child's performance. The results from Gladstone and Sherman (1975) and Koegel et al. (1977, 1978) suggested two tentative conclusions regarding effective training. First, presenting the components of a teaching technique separately (Gladstone & Sherman, 1975) does not seem to result in as high a trainee proficiency as does presenting the components as a single technique (Koegel et al., 1977, 1978). Second, maintenance and generalization of trainee proficiency occurred only following a general training package. Fredericks et al. (1979) conducted three experiments in an attempt to identify a set of skills, knowledge, and/or behaviors that could be trained to teachers and could be shown subsequently to affect student performance. The first experiment attempted to reveal behavioral indices which discriminated teachers of mentally handicapped children showing high learning gains from teachers of children showing low learning gains. The indices discriminating the high and low gain groups were: length of instruction day, percent of programs task-analyzed, training for aides, daily number of volunteers in class, seven items concerned with delivery of consequences and two items which combined cue and consequence delivery. Following this first experiment, Fredericks et al. (1979) conducted an experiment in which a training program was developed to teach the identified behaviors to seven teachers from the low gain group. Using three classrooms at the. Teaching Research Infant and Child Center in 25 Monmouth Oregon, the teachers, either individually or in groups of two, participated in a 5 day training program consisting of modeling, behavioral rehearsal, video and direct feedback, and seminars. An implementation manual outlining the areas covered during training was given to each trainee following the training program to facilitate the generalization of the techniques learned at the Center to their own classrooms. Training follow-up visits occurred approximately 4-5 months and 7-8 months following training. Eight performance objectives, which reflected the 13 behavioral indices found in the firyt experiment, were used to assess the teachers' proficiency at the completion of the 5 day training period. •V. The results on the eight performance objectives revealed that five of the seven teachers passed a l l criteria (pass criterion for the objectives ranged from 80%-90% correct) and the remaining two teachers passed seven of the eight criteria. Data on four areas thought to be the most important from the 13 original behavioral indices were collected pre and post training and at a 12 month probe. The four areas assessed were delivery of consequences, percentage of time volunteers spend giving instruction, percentage of time instructions were delivered one to one, and percentage of programs task analyzed. Data were collected in these four areas from a l l seven teachers at the three measurement points (except for one teacher whose administrator would not allow the use of volunteers in the classroom). Twenty-one of the 27 pre-post comparisons and 21 of the 27 pre-probe comparisons showed increases, with an overall analysis indicating that the pre-post and pre-probe gain scores for the four measures were a l l significant. Of the post-probe comparisons, 7 increased, 10 remained constant (all these post and probe scores were 100%) and 10 decreased. The mean decrease 26 was 21.8$, with the median decrease being 15$. Differences between the o v e r a l l pre-post and pre-probe mean gain scores did not vary by more than three percent. I t i s i n s t r u c t i v e to note that the consistency i n o v e r a l l mean change scores plus the post-probe data would indicate that there was considerable v a r i a t i o n i n the trainees' post-to-probe scores with the impact of those trainees who increased t h e i r scores being balanced by those who decreased t h e i r scores. Therefore no r e l i a b l e predictions could be made using i n d i v i d u a l teacher's post scores to predict t h e i r probe scores. To test the effects of the t r a i n i n g program on the students' behavior, the students (n=14l) taught by trained teachers (n=li+) were compared to an equal number of students of untrained teachers (n=14). Using the State of Oregon's mandated curriculum guide, average pre-post gain scores were computed for t h i s comparison. The students i n the classrooms of trained teachers gained s i g n i f i c a n t l y more than students of untrained teachers. Weissman-Frisch et a l . (1980) conducted a multiple perspective evaluation of an inservice t r a i n i n g program, for vocational trainers of severely and profoundly retarded adults. The purpose of t h i s study was to demonstrate the use of a multiple perspective evaluation approach to inservice t r a i n i n g programs. This i s a di f f e r e n t purpose from the studies reviewed above, which focused on evaluating the effectiveness of a t r a i n i n g method or package. This study examined an evaluation methodology. The key feature of a multiple perspective evaluation i s that data are to be gathered from as wide a range of people (e.g., t r a i n e r s , trainees, support s t a f f or administrators) as possible, who are involved with the t r a i n i n g program. In order to tap as many perspectives as possible i t was suggested that a much broader range of measurement devices is necessary than the traditional methods promoted in the literature. Fifty trainees were given the 4-g- day training program in groups of five to seven trainees. The three major content areas of the training program were task analysis, training methods, and production supervision techniques. The program package consisted of lectures, live and videotaped modeling, behavior rehearsal with feedback, and vicarious observations. The program was used at the Specialized Training Program at the University of Oregon. The Specialized Training Program contained a model work activity center which was used for several phases of the training program. Dependent measures were developed for seven evaluation areas. Performance instruments were used to measure the trainees' s k i l l in task analysis, use of training methods, and production supervision techniques. The Training Evaluation Kit was used for the trainees to provide consumer ratings of the training. The inservice instructors used the Training Decision Analysis Scale to weigh and rate attainment of inservice objectives. The Goal Attainment Index allowed trainees to weigh and rate specific objectives for each training component. A telephone interview was used as a follow-up to gather self-report data on the trainees performance in the areas of task analysis, training methods, and production supervision. A survey regarding the trainees' attitude towards the mentally handicapped was given before and after training. Finally, a pre-post multiple choice test was used to assess knowledge of terminology and techniques. 28 Results from the multiple choice pre-test and self-report data stating a need f o r t r a i n i n g indicated that from 50$ to 92$ of a l l trainees met the c r i t e r i a for needing t r a i n i n g i n at least one of the three major content areas of the workshop. Post t r a i n i n g results showed that 80$ or more of the trainees passed both the multiple choice post-test and a l l performance c r i t e r i a . Results regarding attitude change, consumer rating of the inservice program, and attainment of goals were a l l uniformily p o s i t i v e . The resu l t s from the attitude survey were of pa r t i c u l a r i n t e r e s t as the data revealed d i f f e r e n t i a l changes i n attitude depending upon the l e v e l of mental handicap of the c l i e n t being rated. Greater po s i t i v e attitude change was noted for trainees rat i n g t h e i r attitude towards the severely handicapped than the mildly handicapped. From the telephone interview follow-up, 10-12 weeks post t r a i n i n g , i t was found that of the approximately 50$ of trainees who wrote goals, between 60$ and 64$ of the goals r e l a t i n g to the three content areas had been met. Furthermore, the trainees indicated that they had trained 126 other s t a f f i n at least one of the three content areas of the tr a i n i n g program. The purpose of the Weissman-Frisch et a l . (1980) study was to demonstrate a multiple perspective evaluation approach to an inservice t r a i n i n g program. The range of measures used indicates the authors did f u l f i l l t h e i r stated purpose of studying the breadth of impact an inservice t r a i n i n g program may have. Summary of Review Several comments can be made with regards to a l l the studies reviewed. I n s t i t u t i o n a l attendants, group home s t a f f , parents, high school students, 29 foster grandparents, vocational trainers, and teachers are a very-diverse group of trainees. Yet a l l the experiments reviewed were able to demonstrate that trainees showed increased proficiency in using behaviorally-oriented teaching techniques. The trainees in these studies received their training from a variety of trainers. The types of trainers used were: behavior modification technicians (Gardner, 1972), attendants with more than one year of experience teaching the profoundly retarded (Panyon & Patterson, 1974)» social work graduate students trained specifically for the study (Schinke & Wong, 1977), a university student (Koegel et al., 1977), a graduate student (Koegel et al., 1978), and a B.A. level psychologist (Fabry & Reid, 1978)• It would appear then that the behaviorally-oriented teaching techniques used to teach severely mentally handicapped individuals can be taught by trainers with varying back-grounds to a wide array of trainees who work directly with these client populations. The total duration of training given and the duration of training segments were factors which were highly variable across the studies reviewed. The total duration of training time varied from 1 hour (Koegel et al., 1978) to 5 days (Fredericks et al., 1979). The duration of training segments varied from 30-niinute (Fabry & Reid, 1978; Koegel et al., 1978), to f u l l working days (Fredericks et al., 1979; Weissman-Frisch et al., 1980). As would be expected there was a relationship between the duration of training and the number of content areas being taught. Fredericks et al. (1979) and Weissman-Frisch et al. (1980) had the longest duration of training and also covered the largest number of content areas. The other studies focused mostly on training specific behavioral teaching skills and took less than 20 hours of training time, except the training by Koegel et al. (1977) which took 25-hours or less. It has been demonstrated that training direct care staff to use behavioral teaching techniques has led to beneficial changes in the target client's behavior (Fabry & Reid, 1978; Fredericks et al., 1979; Gladstone & Sherman, 1975; Koegel et al., 1977, 1978; Panyon & Patterson, 1974; Schinke & Wong, 1977). The results from these studies, except for Fredericks et al. (1979), are limited to the time immediately following training. Fredericks et al. (1979) were able to demonstrate one year following training, that students of trained teachers made significantly greater increases in the number of behaviors learned than students of untrained teachers. Also, Fredericks et al. (1979) and Schinke and Wong (1977) were the only studies which indicated that the target clients' behavior of their trainees made significant changes within naturally occurring environments. Thus, while there is strong evidence that training staff in the proper utilization of behavioral methods produces immediate effects on target client behavior there is l i t t l e evidence that these effects are enduring over time and under natural environmental conditions. 31 HYPOTHESES The purpose of the present investigation was to examine the effects of a training program on three areas of trainee behavior. These effects were tested under field conditions with the training offered on a cyclical basis to trainees in various communities throughout the Province of British Columbia. The three areas of trainee behavior that were measured were program writing, teaching technique, and knowledge of terminology. The program writing measure was an elaboration of the task analysis measurement used by Fredericks et a l . (1979)» while the teaching technique and measurement strategy were refinements of those introduced by Koegel et al. (1977, 1978)* The technique introduced by Koegel et al. (1977, 1978) has become known as the Discrete Trial Format (DTF) of teaching (Donnellan, Gossage, LaVigna, Schuler & Traphage, 1977; Koegel et al., 1982. The terminology test was similar to the tests developed by Gardner (1972), Schinke and Wong (1977), and Weissman-Frisch et a l . (1980) as a measure of the trainees' understanding of terminology related to behavior modification and the severely mentally handicapped. At the completion of the workshop and at the final follow-up visit, consumer satisfaction measures were taken. Kazdin (1977) and Wolf (1978) have made strong arguments for the inclusion of consumer satisfaction measures as indices of the impact of clinical interventions. As suggested by Kazdin (1977), positive consumer satisfaction may be associated with maintenance of behavior change technicians given to clients. Following the example of Fredericks et a l . (1979) and Weissman-Frisch et a l . (1980), who attempted to provide follow-up data on the effects of their training programs, the dependent variables of the present study were measured three times over an 8-9 month period. The hypotheses for this study were a) the trainees' scores on a l l three measures would be higher at the completion of the workshop program (mid-test) and at the termination visit (post-test) than at the site visit (pre-test); and b) the proportion of trainees on each measure who scored above pre-determined pass criterion set for each measure would be higher at the mid-test and post-test than at pre-test. METHOD Selection of Subjects  Trainees All professional and paraprofessional personnel from four training programs conducted in British Columbia, who volunteered and completed the training program and were direct service providers for severely mentally handicapped clients, consituted the subject pool. Direct service providers were people who met the minimum requirement of working directly with clients on at least a once weekly basis. From a pool of 37 direct service providers, a l l those who completed a l l measurements on at least one dependent measure were included in this study. Thirty-two trainees (30 female and 2 male) met this criterion. Table 2 indicates the compo-sition of the experimental training groups by occupational category. The mean years of education for a l l participants was l/j.2 years with a range of 11-19 years. Target clients of trainees During the training program the trainees developed their skills while focusing on a single client before attempting to generalize their skills to a l l clients in their work setting. The target clients were chosen by the trainees. The target clients' learning capabilities (e.g. slow learning rate, lack of ability to generalize behaviors, poor prerequisite learning or work s k i l l , or problems developing communication) were usually presenting great difficulty to the trainee. An alternative target client was chosen i f , in the view of the team members, the proposed target client had such severely disruptive behaviors that immediate concern with managing their disruptive behaviors overshadowed the focus of teaching new behaviors. Table 3 lists the age, sex and diagnosis of 34 Table 2 Training Group Composition by Occupational Category of Trainee Training Group Occupation n Group 1 Teacher 3 Vernon Teacher Aide 4 Nurse 1 Psychologist 1 Group 2 Teacher 1 Nanaimo Vocational Trainer 2 Group 3 Teacher 3 Williams Lake Teacher Aide 2 Childcare Worker 1 Group 4 Teacher 7 Kamloops Teacher Aide 7_ 35 Table 3 Age, Sex and Diagnosis of Target Clients Training Group 1 2 3 4 Number of Target Clients 6 3 Age Mean 11 6.7 Range 7-19 4-19 Sex Male 5 3 Female 1 0 Primary Diagnosis Autism 3 1 Retardation Moderate 0 0 Severe/Profound 1 0 Developmental-Delay " 0 1 Undefined Diagnosis 2 1 14.7 10-23 2 4 11.8 9-16 3 4 1 2 0 1 a A l l specific diagnoses were from the clients' case records. Records for clients with undefined diagnosis either gave non-specific information about diagnosis or were not available to the training team. Individual diagnostic data for group four's target clients were unavailable. The intake criteria for the school that a l l clients attended were: IQ= 25-39, clients with IQ= 40-54 were accepted only i f present functioning was seriously effected by social/emotional difficulties. the target c l i e n t s . Settings Training programs were conducted i n four communities i n B r i t i s h Columbia from June 1979 through to March 1981. The t r a i n i n g locations, with the t o t a l number of participants and t o t a l number of participants who met the c r i t e r i a of trainee l i s t e d i n parentheses, were: Vernon ( l l , 9 ) i Nanaimo (7,3), Williams Lake (9,6), and Kamloops (l6,14). In Vernon, Williams Lake, and. Kamloops the workshop portion of the t r a i n i n g program took place at the school where the largest number of target c l i e n t s attended, while i n Nanaimo a large meeting room adjacent to the school board o f f i c e s was used. A l l follow-up v i s i t s were conducted at the work s i t e of the trainee. Chi-square analyses using the Yates correction f o r continuity (Ferguson, 1971) were computed to measure the d i s t r i b u t i o n of occupational categories and educational levels of the participants across the four t r a i n i n g programs. Occupational categories were teacher, teacher-aide, and non-school personnel. Education was divided into three l e v e l s , 11-13 years, 14-15 years, and 16 years or more of schooling. The r e s u l t s f o r 2 occupational category were, x ^ = 6.09, p_ = n.s. For education l e v e l 2 the re s u l t s were x ^ = 3.56, p_ = n.s. These results indicate that although the t r a i n i n g groups were comprised of trainees from a variety of occupations and educational l e v e l s , the group compositions did not s i g n i f i c a n t l y d i f f e r from each other on these two dimensions. Description of t r a i n i n g team The purpose of the P r o v i n c i a l Inservice Resource Team was described i n a government p o l i c y d i r e c t i v e : "The purpose of the P r o v i n c i a l Inservice Resource Team i s to t r a i n parents, teachers, and others to meet the needs of 37 children with severe behavioral disturbances, particularly those described as 'autistic' (including non-compliant, non-communicating children with severe developmental delays who show "autistic" behaviors). This service will support present residential or day treatment and classroom services by offering training to parents and staff responsible for autistic children in communities which do not, at present, have appropriate local services" (Noble, Note 2). The Provincial Team consisted of a community nurse, teacher, psychologist and half-time coordinator (the coordinator was also responsible for the Woodlands Inservice Resource Team). The community nurse and psychologist were original members of the team who received their training from the Santa Barbara Autism Dissemination Project (Donnellan et al., Note l ) , which conducted a training program for both the Provincial and Woodlands teams in 1978. The contacts between the team and the users of the service could be broken into six components. These were: l) request for service, 2) prereferral contact, 3) site visit, 4) 5 day workshop, 5) follow-up visits, 6) final follow-up visit or termination vis i t . The team's involvement with the users, from request for service to the termination visit, took approximately 8 to 9 months. The training program consisted of the site visit, workshop, follow-ups and termination visit, with the core of the training program being the workshop. Five follow-up visits were conducted, lasting from 2 to 4 days depending on the size of the training group. These follow-ups occurred i n i t i a l l y at 2 to 3 week intervals, and faded to 4 or 5 week' intervals between the last two visits. The termination visit occurred 6 to 8 weeks following the last follow-up visi t . 38 Table 4 Schedule of Workshop Training Components Schedule of Inservice Components Method of Training Day one A. Introductions to -goals of the workshop autism DTF B. Practice with DTF teaching prerequisite skills and teaching using criteria C. Summary of DTF D. Readings Day two A. Review of readings and introduction to base-lining B. Hands-on training with target clients C. Program Writing D. Hands-on teaching using a written program E. Introduction to programmed curriculum F. Maintenance and general-ization G. Readings and writing program Didactic Didactic/Film Didactic, VTR, Role-play, Demonstration Demonstration/Role-play Film Homework Didactic/Discussion Behavior rehearsal and Feedback Didactic, Behavior rehearsal and Feedback Behavior rehearsal and Feedback Film Didactic Homework Table 4 Cont. Day three A. Review and elaboration of program writing B. Hands-on DTF teaching using a written program C. Language and Communication D. Hands-on teaching focusing on language training E. Write language program Day four A. Principles of Behavior Management B. Hands-on teaching using language programs for homework C. Techniques of Behavior Management D. Task analysis E. Overview of last two days F. Write a task analysis Day five A. Curriculum B. Small group planning for the first follow-up visit C. Review of homework D. Token economy Didactic, Discussion and Feedback from Homework Behavior rehearsal and Feedback Didactic Behavior rehearsal and Feedback Homework Didactic Behavior rehearsal and Feedback Didactic/Role-play Didactic/Exercise Film Homework Didactic Discussion Feedback Didactic Table 4 Cont. E. Review of trainees teaching using DTF F. Terminology mid-test G. Consumer satisfaction ratings H. Conclusion Model VTR segments of trainees teaching taken during workshop Knowledge of terminology test Workshop feedback form Discussion 41 Training Program Description The present section contains an overview of the structure and components of the training program. A manual describing the conducting of a training program is contained in Appendix A. Site Visit The site visit was a preparatory visit which occurred 4 to 6 weeks prior to the workshop and lasted for 2 to 3 days. This visit had three major components: collection of pre-test data on terminology, program writing and DTF teaching measures; distribution of the training manuals (Donnellan et al., 1977); program text (LaVigna & Trophagen, 1978), readings, and finalization of necessary administrative details. All three pre-test measures were introduced to the trainees as assessment methods the team used to tailor the content of the workshop to the strengths and weaknesses of the training group, as well as being used for a research project. Each trainee was individually videotaped while teaching the target child he or she would focus on during the workshop. The instructions for the teaching were that the teaching should be a 1-1 structured session of teaching the client a behavior he or she had not yet mastered. The trainees were asked to give the team a copy of the latest program they had written. The terminology test was given to each trainee with the instruction that the test was to be completed within 15 minutes. Workshop Table 4 presents the training components for each day and the methods of training used for each component. An example of the detailed workshop training schedule used for the training programs is contained in Appendix B. The order of introducing the major content areas was DTF teaching, program writing, maintenance and generalization, language/communication, behavior principles and management, curriculum and token economy. Al l didactic presentations, other than the introduction of goals for the workshop, were accompanied by articles or handouts. Trainees were instructed to bring a large binder to the workshop so a l l the articles and handouts could be collated to build a personal reference compendium. During hands-on training the trainees were videotaped on a rotating basis such that a l l trainees were videotaped at least twice during Days 2,3, and 4« These tapes were reviewed and scored each night and verbal feedback was given each day regarding the progress the trainees were making in using DTF teaching. The programs and task analyses written for homework were collected at the beginning of each day. The programs were scored and written feedback was given on the program. Oral feedback to the group as a whole was given at the end of the morning session. Each training day was 7 hours in duration. Excluding time for lunch and breaks, 5k hours were available for training each day, or 27.5 hours for the whole week. Table 5 shows the total time allocated for the various training methods, for a total weekly training duration of 22 hours. The 5k hours not totaled were used for concluding each day, giving out homework, planning for the first follow-up, completing the terminology test, and completing consumer satisfaction ratings. Follow-^up Visits The major goal of the follow-up visits was to help the trainees integrate the skills learned during the workshop into their normal work environment. Once this was accomplished a second goal was to help the trainee generalize their newly developed skills to other clients. Five areas of interest formed the basis for most follow-up visits. Table 5 Total Time Allotment of Training Methods During the Workshop Training Methods Hours Didactic/Discussion 10.5 Films/VTR 3.0 Role-play 1.5 Demonstrations 1.5 Behavior rehearsal 4*5 with feedback General feedback 1.0 Total =22.0 These areas were: observing the trainees teach, discussing any new written programs, reviewing data c o l l e c t i o n , discussing curriculum, and reviewing and monitoring behavior management programs. A l l follow-up v i s i t s and the termination v i s i t were conducted at the trainees' work setting. To provide l i n k s between follow-up v i s i t s goals were set f o r each trainee to work on between v i s i t s . Previous goals were reviewed each v i s i t and new goals set f o r the next meeting. The periods between follow-ups lengthened from an i n i t i a l 2 week period between the workshop and f i r s t follow-up to 5 or 6 weeks between the l a s t two follow-ups. The termination v i s i t , when post-test measures and follow-up consumer s a t i s f a c t i o n ratings were taken, was 6 to 8 weeks a f t e r the f i n a l follow-up. Design Cook and Campbell (1979) used the term i n s t i t u t i o n a l cycle design to describe the design used for t h i s study. This research design has been recommended by several authors (Campbell & McCormack, 1957; Campbell & Stanley, 1966; Cook & Campbell, 1979; M c K i l l i p s , 1979; Powers & Alderman, 1979) f o r use when there i s a cycle to the delivery of treatment or t r a i n i n g . Figure 1 outlines the i n s t i t u t i o n a l cycle design as applied to the present study. Campbell and Stanley (1966) have suggested that when the focus of f i e l d research i s on the recurrent application of a complex independent variable, such as a training.program, the i n s t i t u t i o n a l cycle design (termed the recurrent i n s t i t u t i o n a l cycle design by Campbell and Stanley) probably offers the best quasi-experimental design developed thus f a r . Arguments were also presented by Campbell and McCormack (1957) and Campbell and Stanley (1966) that i f the program being offered i s one Figure 1 The Institutional Cycle Design used to Study the Impact of the Provincial Team's Training Program 1979 1980 1981 J J A S O N D J F M A M J J A S O N D J F M A M J u a u a u n n n n n e e e Group 1 p ma p r i o e d s t Group 2 p m p r i 0 e d s t Group .3 p m p r i o e d s t Group 4 p m p r i o e d s t aAs the mid-test occurred during the workshop the term mid indicates the mid measurement and the occurrance of the workshop. 46 which a l l potential subjects wish to have access to, then the institutional cycle design may be better than a true experimental design. The rationale for this possible superiority was that i t may not be feasible to find a proper control group for a true experimental design which would allow for generalizations to be made to the natural recruitment and training conditions. The institutional cycle design is essentially a combination of the longitudinal and cross-sectional approaches employed in developmental research. As indicated by Campbell and Stanley (1966) and Cook and Campbell (1979) the longitudinal and cross-sectional approaches complement each other in controlling for threats to the internal and external validity of the experiment. For example, a longitudinal design does not control for history but a cross-sectional approach does. However, a longitudinal design does control for attrition while a cross-sectional approach does not. The most problematic threats to internal validity with this design are testing, maturation and selection-maturation interaction. The institutional cycle design allows for several types of comparisons. Each group can be looked at separately, as a pre-mid-post case study of the training impact. Thus the data from each group can be analyzed to see i f they conform to the stated hypotheses. The groups could be compared within a measurement period to assess the similarity of the groups at a common phase of the design. This comparison could, for instance, indicate differential effects of the training among the groups at the mid-test measurement period. Finally, groups which are measured at temporally similar times but are in different training phases can be compared, to test for the effects of history. The cross-sectional comparison of the mid-test for group 1 with the pre-test of group 2 where the mid-test scores of group 1 are different than the pre-test scores A7 of group 1 provides differences which could not be explained by the effects of history. Measures Three performance measures plus two consumer satisfaction measures were collected. The three performance measures, knowledge of terminology, program writing and DTF teaching s k i l l were obtained at the site visit, at the completion of the workshop,and at the termination visit. The consumer satisfaction measures were collected at the completion of the workshop and at the final follow-up. Knowledge of terminology The knowledge of terminology test was a 15 item combined multiple-choice and fill-in-the-blank test. It sampled items concerning behavior modification and issues related to training the severely mentally handicapped. It was used to assess trainees' knowledge of terminology used during the training program. An answer key ensured consistent scoring of the test. Appendix C contains a copy of the terminology test and scoring key. Program writing The aim of writing a program was to have a written document which indicated a l l the necessary information so that another person could read the program and duplicate the teaching environment and activities for the target client. The task analysis measurement used by Fredericks et al. (1979) and Weissman-Frisch et al. (i960) measured those parts of a program related to identifying a l l the subcomponents of a target behavior. A program contained not only a task, analysis or series of teaching steps but also indicated the cue or instruction, conditions for using prompts, pass and f a i l criteria, specification of teaching materials, schedule of reinforcement, how the target behavior was to be maintained or generalized, and auxiliary details necessary to ensure potential duplication of the training procedure. A checklist of Z+0 items was used to score the written programs. A scoring guide which defined a l l items and gave examples of various items was used for a l l scoring. As several programs were collected from each trainee during the workshop, the highest score obtained was used as the score for the workshop. Appendix D contains a copy of the program writing checklist and the scoring guide. DTF teaching s k i l l The DTF teaching measure was developed from the measure used by Koegel et a l . (1977, 1978). The present version of the measurement device has six components: preparation, instructional stimulus, prompt, response, consequence, and intertrial interval. As each trainee was videotaped at least twice during the workshop the highest score obtained was used as the score for the workshop. Appendix E contains a copy of the DTF scoring sheet and scoring guide. Consumer Satisfaction: workshop feedback During the afternoon of the last day of the workshop trainees were asked to complete a feedback sheet containing items related to the organization and preparation of the workshop, content of the workshop and training methods used. Each item was rated on a 5 point scale. As each community had unique requirements some changes in content areas were made for each training program. An example of a workshop feedback sheet is contained in Appendix F. Consumer Satisfaction: follow-up feedback At the termination visit, trainees were given a two part feedback form. The first part asked questions regarding the trainees' use of the techniques learned during the training program. The second part asked 4 9 questions about how effective the team was in helping the trainees generalize and integrate the training materials into their work setting and whether their use of the techniques had helped any clients. A l l items which were rated used a 5 point scale. Appendix G contains both parts of the follow-up feedback form. Reliability Reliability checks were conducted to assess the consistency of interrater scores for the program writing and DTF teaching measures and to assess compliance in using the terminology answer key. The reliability coder for the program writing and DTF teaching measure was a member of the Woodlands Team. As the members from the Woodlands Team had never had contact with the Provincial Team's trainees this ensured blind ratings of these two measures. The terminology test reliability was determined by members of the Provincial team after a l l identifying information was concealed. To ensure that reliability checks were conducted for each test period,- the scores were grouped into the three measurement periods, and scores were then randomly selected from within time periods for the reliability check. The percentage of reliability checks averaged across the three periods were 39% for the terminology measure, 31% for the DTF teaching measure and 24.5% for the program writing measure. The lowest percentage of scores checked within any single time period was 21%. (Percentage of scores checked and percentage agreement per test period can be found in Appendix H). The phi coefficient was used to calculate the reliability of the measures. Table 6 contains 2 X 2 tables indicating the frequencies of agreements and disagreements between the original coder and the reliability coder. The reliability i s based on the agreements of the coders on an item level, rather than for total scores. In general this Table 6 Contingency Tables for Computing Phi Coefficient for the R e l i a b i l i t y of the Three Dependent Measures DTF Original Coder Hit Miss Hit 1180 92 1272 R e l i a b i l i t y Coder Miss 33 92 125 1213 184 1397 Program Writing Original Coder Hit Miss Hit 284 17 301 Rel i a b i l i t y Coder Miss 29 54 83 313 71 384 Terminology Original Coder Hit Miss Hit 510 9 519 R e l i a b i l i t y Coder Miss 13 195 208 523 204 727 51 level of analysis provides a lower-bound estimate of reliability compared to reliability measured by total scores (Hartmann, 1977)* In addition to being a stronger statistical measure than percentage agreement the phi coefficient also allows for a test of the significance of the coefficient using chi-square (Glass & Stanley, 1970). The phi-coefficients were, for DTF teaching, ^  = O.56, with a x = 438.38, p_ < .001; 2 for program writing, ^ = .63 , with a x = 152.39, p. < .001; and for 2 terminology test, £ = .93 , with a x = 623.12, p_ < .001. Thus a l l three measures were substantially and significantly reliable. 1 52 RESULTS Initial analyses for homogeneity of variances were conducted using the F-max test (Winer, 1971). Significant heterogeneity of variances were found between test periods, on a l l three measures. An arc-sine transformation (Winer, 1971) reduced heterogeneity to non-significant levels. However, comparison of the results from analyses using the original and transformed data revealed only minor non-systematic differences between the results. Analyses from both types of data were in perfect agreement regarding significant versus non-significant results. The reason for the similarity of the results was that the sample size across test periods was equal, which allows for sizeable heterogeneity of variance without seriously biasing the results. As no major differences existed between the original and transformed data analysis i t was decided to report a l l results using the original data. To test for differences between those participants included in the analysis as trainees and those participants excluded from the analysis, the two groups were separated and the group mean scores at the mid-test were analyzed using t-tests. Table 7 indicates the number of participants included and excluded, as well as the group mean score for each measure. The comparison between the included and excluded group on the terminology measure indicated a non-significant difference, t (35) = .2837, p_ = n.s. For the DTF measure, the result was also non-significant, t (30) = .4069, p_ = n.s. For the program writing measure, the result was t (35) = 1.67, p_ = n.s., also non-significant. These results indicate that the scores of the included versus excluded participants were not significantly different at the completion of the workshop. Table 7 Size of Included and Excluded Groups and Group Mean Scores for Workshop Participants on Dependent Measures Terminology n x Included 29 83.0 Participants Excluded 8 81.6 DTF n x Included 19 93.5 Participants Excluded 13 92.5 Program Writing n x Included 15 89.87 Participants Excluded 22 83.23 5k Knowledge of Behavioral Terminology A 3 (test period) X 4 (training group) mixed effects ANOVA with repeated measures was used to test for differences in the terminology scores. Table 8 indicates the mean percentage correct for the terminology test and Table 9 summarizes the analysis of variance. A significant test period effect was found, F (2,50) = 50.94, p_ < «001. The groups did not differ, nor was there an interaction between groups and test period. Post comparisons of the significant main effect were done using the Neuman-Keuls test for multiple comparisons on repeated measures (Winer, 1971). The harmonic mean of 'the cell sizes was used for these comparisons as the cell sizes were unequal. The significant comparisons were between pre-test and mid-test, Q (2,50) = 24.66, p_ < ,01 and between pre-test and post-test, Cj (3,50) = 27.14, p_ < .01 . These results support the hypothesis that the trainees would score significantly higher at mid and post-test compared to their scores at pre-test on the terminology test. Discrete Trial Format Teaching A 3 (test period) X 4 (group) mixed effects ANOVA with repeated measures was used to test for differences in DTF teaching' scores. Table 10 shows the mean percentage correct for DTF teaching measure and Table 11 presents the analysis of variance. Significant main effects were, Training Group, F (3,15) = 5-54, £ < .0092; Test Period, F (2,30) =• 35.78, p_ < .0001, and Training Group X Test Period interaction, F (6,30) = 4*47, p_ < .0024.. Figure 2 demonstrates the interaction effect. The interaction effect was broken down into simple main effects. The analysis of the simple main effect for training groups within each 55 Table 8 Mean Percentage Correct on Terminology Measure Test Period Group Pre Mid Post n 1 8 52.75 82.00 79.86 2 3 51.67 73.67 73.67 3 5 55-40 76.40 87.60 4 13 58.92 88.31 79.77 Table 9 Results of Analysis of Variance on Terminology Data Source df ms F Between Training Group 3 242.417 0.99 Error 25 244.441 Within Test Period 2 4707.842 50.94* Period x Group 6 130.244 1.41 Error 50 92.419 *p_ < .0001 Table 10 Mean Percentage Correct on DTF Measure Group Test Period Pre Mid Post n 1 7 36.00 92.00 93.86 2 2 43.50 94.00 99-50 3 3 83.67 94.00 100.00 4 7 72.26 94.57 91.57 Table 11 Results of Analysis of Variance on DTF Data Source df MS F Between Training Group Error 3 15 925.605 167.139 5.54 Within Test Period 2 6226.204 35.78' Period x Group 6 777.213 4.47 Error 30 174.010 * p_ < .01 ** £ < .005 *** 2 < •0001 59 Figure 2 Interaction of Group Mean Scores on DTF Measure Mean Percent Correct Pre Mid Post Test Period test period indicated a significant group effect within the pre-test period, F (3,15) = 4*67, B < '025, but not during subsequent test periods. The analysis of the simple main effects for individual groups' test period revealed several significant effects. Test period was significant for group 1, F (2,48) = 43.29, £ < .001; group 2, F (2,8) = 10.98, £ < .01, and for group 4, F (2,48) = 13.26, p_ < .001, but not for group 3 . Multiple comparisons were done on these significant simple main effects. To test the training group differences at the pre-test time, the harmonic mean for the four groups was used in the Neuman-Keuls test, as the groups had unequal cell sizes. The significant comparisons were between group 1 and group 3, 0* (4,15) = 7.00, p_ < .005; group 1 and group 4, Q (3,15) = 5.33, £ < .005; group 2 and group 3 , £ (3,15) = 5.90, £ < .005, and between group 2 and group 4, Q (2,15) = 4.23, £ < .01 . In other words, group 1 and group 2 were not significantly different from each other, group 3 and group 4 were also not significantly different from each other, but groups 1 and 2 both had pre-test scores significantly lower than groups 3 and 4 . The simple main effects for test period were further analyzed by multiple comparisons. The significant comparisons for group 1 were between pre-test and mid-test, Q (2,30) = 22.49, £ < .005, and pre-test and post-test, Q (3,30) = 23.23, £ < .005. For group 2, the significant differences were between, pre-test and mid-test, ^  (2,30) = 10.84, £ < .005, and mid-test and post-test, Q (3,30) = 12.01, £ < .005. Significant differences for group 4 were between pre-test and post-test, Q (3,30) = 4.78, £ < .01 , and pre-test and post-test, Q (2,30) = 4*14, £ < .01 . This pattern of results supports the hypothesis that the mid and post-test scores would be significantly higher than the pre-test scores, except for group 3 . As can be seen from Table 8, the pre-test mean for group 3 was already above the 80% criteria set as indicative of competency on this measure. This high pre-test score limited the possible amount of change towards supporting the hypothesis, even though there was an increase in mean scores at both mid and post-test periods. Program writing A 2 (test period) X 3 (group) mixed effects ANOVA with repeated measures was used to test for differences in program writing scores. The analysis was reduced in size for two reasons. First, as will be elaborated in the discussion section, only three subjects completed the pre-test measure which left insufficient pre-test data to make inclusion of data for this time period meaningful. Second, only one trainee in group 2 completed both the mid and post-test measures which precluded inclusion of group 2 in this analysis. Table 12 indicates the mean percentage correct for the program writing measure and Table 13 presents the analysis of variance. A significant test period effect was found, F (l , 12) = 30.07, p < .0001. This indicated a significant drop in performance on this measure. Tests on proportions of trainees meeting criteria To test the hypotheses regarding the proportion of trainees exceeding predetermined criteria, the inferential technique for testing differences between proportions for dependent samples suggested by Glass and Stanley (1970) was used. The statistic used in this test is a Z score. In order for the statistic to have an approximately normal distribution with mean = 0 and standard deviation = 1, the sum of the cells being directly tested must be equal to or greater than ten. To f u l f i l l this requirement, the tests were performed by collapsing the groups within each test period and then testing differences amongst measurement periods. Table 12 Mean Percentage Correct on Program Writing Measure Test Period n Mid Post 1 4 87.25 78.00 ° r 0 U p S 3 3 96.00 61.33 4 8 88.88 53.62 Table 13 Results of Analysis of Variance on Program Writing Data Source df MS F Between Training Group 2 376.579 1.37 Error 12 274.059 Within Test Period 1 4424.020 30.07 Period x Group 2 489.871 3.33 Error 12 147.122 * p_ < .0001 64 Proportion of trainees exceeding terminology test c r i t e r i o n Table 14 presents a series of contingency tables that indicate the frequency of trainees passing or f a i l i n g the 80$ c r i t e r i o n f o r t h i s t e s t , contingent upon when the test was administered. The test on proportions described above has a special formula which allows the test to be calculated d i r e c t l y from the contingency tables. The results of t h i s test on proportions revealed s i g n i f i c a n t differences between pre-test and mid-t e s t , Z = 4«12, £ < .001, and between pre-test and post-test, Z = 4*24, £ < .001. Support was thus found for the hypotheses that the t r a i n i n g program would increase the proportion of trainees exceeding the c r i t e r i o n of success from pre-test to mid-test and post-test. Proportion of trainees exceeding DTF c r i t e r i o n Table 15 i s a series of contingency tables that indicate the frequency of trainees passing or f a i l i n g the 80$ c r i t e r i o n f o r t h i s t e s t , contingent upon when the test was administered. The results of the test on proportions revealed s i g n i f i c a n t differences between pre-test and mid-t e s t , Z = 3*46, £ < .001, and between pre-test and post-test, Z = 3.46, £ < .001. These r e s u l t s support the hypothesis that the proportion of trainees exceeding the c r i t e r i o n would be greater at mid-test and post-test than at pre-test. Proportion of trainees exceeding program w r i t i n g c r i t e r i o n Table 16 shoxvs the contingency table for trainees at mid-test and post-test who passed or f a i l e d the 70$ c r i t e r i o n set f o r t h i s measure. The proposed hypothesis regarding t h i s measure could not be tested due to the lack of pre-test data. However, the maintenance of the results obtained at the completion of the workshop can be examined. The r e s u l t s revealed a s i g n i f i c a n t difference between the mid-test and post-test, Z = 2.83, £ < .005. This result indicates a s i g n i f i c a n t decrease i n the proportion of trainees Table 14 Contingency Tables of Trainees Meeting 80% Pass C r i t e r i o n for Terminology Measure Mid-test F a i l Pass Pass 0 a 2 Pre-test F a i l 10 17 Post-test F a i l Pass Pass 0 2 Pre-test F a i l 9 18 Post-test F a i l Pass Pass 4 15 Mid-test F a i l 5 5 dumber of trainees passing pre-test and f a i l i n g mid-test Table 15 Contingency Tables of Trainees Meeting 80% Pass C r i t e r i o n f o r DTF Measure Mid-test F a i l Pass Pass 0 a 6 Pre-test F a i l l 1 2 Post-test F a i l Pass Pass 0 6 Pre-test F a i l 1 12 Post-test F a i l Pass Pass 1 17 Mid-test F a i l 0 1 dumber of trainees passing pre-test and f a i l i n g mid-test Table 16 Contingency Table of Trainees Meeting 70% Pass Criterion on Program Writing Measure Post Fail Pass Pass 8a 6 Mid Fail 1 0 Number of trainees passing Mid and failing Post 68 exceeding the c r i t e r i o n at post-test compared to mid-test. Consumer Sat i s f a c t i o n Participants used a 5 point rating scale to indicate t h e i r s a t i s f a c t i o n with the workshop and follow-up v i s i t s . A l l ratings were done anonymously to l i m i t the possible coersive bias inherent i n t h i s type of r a t i n g . Table 17 shows the mean consumer s a t i s f a c t i o n ratings of the workshop and follow-up v i s i t s f o r the four t r a i n i n g programs. The range of mean ratings for the workshop was 4.2 to 4 .5 , and the range fo r the follow-up v i s i t s was 3*7 to 4«7« Two one-way ANOVA's were computed to test f o r differences among the groups. Table 18 presents the analysis of variance f o r both sets of consumer s a t i s f a c t i o n ratings. For the workshop ratings, the resu l t was non-significant, F (3,34) = 1.23, £ = n.s. The re s u l t s for the follow-up ratings were also non-significant, F (3*24) = 1.89, £ = n.s. These consumer ratings results indicate a consistent, high degree of sa t i s f a c t i o n with the t r a i n i n g program, across a l l groups at both rati n g periods. 69 Table 17 Mean Consumer Satisfaction Ratings for the Workshop and Follow-up Visits Workshop Follow-up 1 4.4 4.4 2 4.5 4.7 Group 3 4.5 3.8 4 4.2 3.7 Table 18 Results of Analysis of Variances on Workshop and Follow-up Consumer Satisfaction Data ANOVA of Consumer Satisfaction Ratings for the Workshop Source df MS F Training Groups 3 0.108 1.23 Error 34 0.133 * non-significant ANOVA of Consumer Satisfaction Ratings for the Follow--ups Source df MS F Training Groups 3 0.747 *-1.89 Error 24 0.345 non-significant DISCUSSION The results of this study indicated that the Provincial Team's training program made a significant positive impact on the trainees' s k i l l in using DTF teaching and their knowledge of behavioral termiriolog by the completion of the workshop. Further, the results suggested that the trainees' knowledge of behavioral terminology and use of DTF teaching were maintained at post-test. As no pre-test measurement was done for the program writing measure, i t is not known what impact the training had in this area. A l l three groups included in the limited analysis on program writing were above the 70% pass criterion set for this measure by the completion of the workshop. However there was a significant decrease in performance on this measure at post-test. The consumer satisfaction ratings of the workshop and follow-up visits were uniformly positive across a l l groups. As indicated in Table 7, not a l l participants who attended the workshop were included as trainees for data analysis. Attrition and staff turnover were the two major reasons for the dropout of direct service staff who did attend the workshop. George and Baumeister (l98l) found a very high (73% per year) turnover rate for managers of community based resources for the mentally handicapped. Data were not kept on attrition rates in communities where the training programs were conducted. The turnover rate appeared to be high, although lower than the rate reported by George and Baumeister (1981). For each measure taken separately the l-ule of inclusion guarded against a bias over test periods due to selective attrition at any one test period (Campbell & Stanley, 1966). The importance of teaching appropriate terminology i s suggested by the widespread use of programmed tests (Becker, 1971; LaVigna & Traphagen, 1978; Patterson, 1975), summary quizzes (Watson, 1973), and use of terminology and short answer questions as units of other behavior modification training programs (Gardner, 1972; Hall, 1978; Schinke & Wong, 1979; Watson, 1972; Weissman-Frisch et a l . , 1980). As suggested by Watson (1972), the rationale behind encouraging trainees to learn appropriate terminology i s that the conceptional learning w i l l allow the trainees to more f u l l y understand the techniques they use. Increases i n the trainees' terminology scores were probably attributable to the combined effects of the team modeling appropriate use of terminology, and the corrective feedback the trainees received concerning their own use of terminology. The differences i n mean scores on the terminology test from pre-test to mid-test and post-test were significant. An inspection of Table 8 shows the group means at mid-test and post-test fluctuated around 80$, the criterion for success. Inspecting the data from Table 14 the proportions of trainees achieving the 80$ criterion on the test indicates that the majority of trainees passed the 80$ criterion at mid-test, regardless of their pre-test scores, and maintained their scores above this criterion from mid to post-test. A possible explanation for these results i s suggested i n a study reported by Watson (1972) i n which the influence of a reinforcement contingency upon quiz performance was investigated. Progress through the training units was either contingent or non-contingent upon success-f u l completion of the quizzes. Within the non-contingent phases test performance was highly variable, while under the contingent condition the performance was very consistent and above criterion. As the Provincial Team's training program was voluntary, i t was similar to the non-contingent condition. It may be that the variations in the trainees' terminology scores were due to a lack of contingent reinforcement which could override the effects of individual motivational differences. The most impressive results of this study were those obtained by the measurement of the trainees' DTF teaching s k i l l . These data were in substantial agreement with those presented by Koegel et al. (1977? 1978). The results from the present study were obtained with approximately 7 hours of instruction vithin the workshop component. Koegel et al. (1977) used approximately 25 hours of instructions. Thus the results from the present study indicated that an acceptable level of sk i l l in DTF teaching can be obtained with much less training than previously indicated. The post-test data from the present study revealed a maintenance level of over 90% correct in the use of DTF teaching for the majority of trainees. This extends the previous data reported by Koegel et a l . (1977, 1978) on the impact of training direct service providers. These two prior studies utilized a post-test immediately following training. The results of the present study suggest that the maintenance of this s k i l l was very good up to the termination visit, 6 to 7 months after i n i t i a l training. Two possible factors which would support the maintenance of the s k i l l are suggested. First, as indicated in the Koegel et al. (1977, 1978) studies, the effect of the proper use of this teaching technique is to dramatically increase the child's rate of correct responding. • Second, from clinical observations made during this study, there is often a reduction in the student's rate of inappropriate behaviors concomitant with the use of DTF teaching. The immediately reinforcing value of these Ik two factors for the trainee would support a high maintenance level. The results from the program writing mid-test indicated a high level of trainee proficiency in this s k i l l , although the post-test scores revealed a significant decrease in proficiency. As indicated in the results, only three trainees completed the pre-test measures. The assumption underlying the gathering of the program writing data was that the trainees would be writing programs which could then be scored with the program writing measure. This assumption vas based on the piloting of the program writing measure prior to this study. Why the assumption was justified with the pilot sample but not the experimental sample can be explained as a function of a history bias. Many of the pilot trainees had been exposed to or trained in program writing prior to testing. For instance, several pilot trainees had received training from Laurel House, a behaviorally oriented treatment center which uses program writing as an essential tool for structuring their training of the mentally handicapped. Very few of the experimental trainees had any such history. Thus, a program writing measure, which could be used generally to gather pre-test data, would need to assume that most trainees would not be actively writing programs. An assumption which could be made about this pre-test situation is that trainees' pre-test scores on a program writing measure were low. A recently reformulated program writing measure now assumes that trainees will not be writing programs. The trainees are now given the structured task of writing a program during the site visit. Pre-test data using this new measure have been collected on two training programs by the Woodlands Team. The mean pre-test score was 18.5% with a range of 6$ - 58fo. As the definitions of trainees included for evaluation were the same for the Woodlands and Provincial Teams, this data offers some post hoc support for 75 the assumption that the trainees for the present study would also have had low pre-test scores. Several factors appeared as possible agents contributing to the clinically and statistically significant reduction in trainees' scores following training. As program writing was a new and difficult s k i l l for most trainees, continuous reinforcement would be necessary to encourage its continuation. There appeared to be few immediate rewards for program writing. The reinforcement of the s k i l l would have to be managed and programmed until the natural nonprogrammed reinforcers for this s k i l l could be established. However, in the normal work setting of the trainees, there seemed to be almost no reinforcement and sometimes punishment^  for this s k i l l . For instance, the trainees working cn a contractual basis in several instances had contracts which precluded payment for any activity not related to working directly with the client. For teachers, at the beginning of each school year, commitments were made to classroom and school routines which made the changes to an individualized program basis an onerous task. Testing was the major threat to the internal validity of this study. The present study used a repeated measures design, with measures having reactive properties. Multiple exposures to the measurement devices could have had a biasing effect on the data. It is not possible to rule out this threat from the terminology and DTF measure as the generally consistent rapid increase in mean scores at mid-test which were maintained at high levels at post-test could potentially be explained as a result of a testing bias. The significant decrease from mid-test to post-test on the program writing measure could also be explained as a result of a testing bias. Threats to the external validity of this study come from a possible selection x training program interaction and the reactive effects of testing. Two types of selection bias are apparent in this study which limit the generalizability of the results. All trainees were volunteers for this training program and only data for trainees who completed a l l three test periods on a dependent measure were included in this study. With these two biases, generalizations to non-volunteer trainees and trainees who do not complete the entire training program would seem inappropriate. The reactive nature of the measures used in this study made i t uncertain whether the data could be generalized to training conditions where no measurement was taken or unobtrusive measurements were used. The present study extends the previous literature on training programs for direct service workers of severely mentally handicapped clients. This study, with i t s cyclical delivery of a training program, indicated that an effective training program, based on methods suggested by previous research, can be delivered to a wide variety of trainees in different settings. The major area of weakness was the maintenance of the trainee's program writing skills. The portability of the training program, where three trainers could train 30-40 trainees a year in their home communities, would suggest that this was a very cost-effective method for delivering a training program. The cost-effectiveness of such a training program would be particularly evident in an area with a widely dispersed population, such as is the case in the Province of British Columbia. The use of DTF teaching as the major teaching technique focused on during the training program extends the research previcusly done on the technique (Koegel et al., 1977, 1978). As mentioned earlier, i n i t i a l high levels of proficiency were obtained in the present study using substantially less training time than Koegel et al. (1977). Also, i t was demonstrated that the high level of proficiency was maintained for 77 up to 6 or 7 months following the workshop. The previous studies by Koegel et al. (1977, 1978) had not studied the long term maintenance of this s k i l l . Three areas can be identified for further research. Although prior research (Fredericks et al., 1979; Gladstone & Sherman, 1975; Koegel et al., 1977, 1978; Panyon & Patterson, 1974; Schinke & Wong, 1977) has shown that training direct service staff has a beneficial impact on the target clients, this has not been demonstrated for the present training program. Until such time as the Provincial Team's training program can be demonstrated to lead to beneficial changes for the target clients, its valve for the target clients is uncertain. A more appropriate measure of program writing s k i l l which is capable of reflecting the present level of this s k i l l in the trainee population is required. Following the development of such an instrument, research on methods for maintaining this s k i l l would be of interest. Finally, the training program of the Provincial Team was a complex package. The use of a dismantling treatment strategy (Kazdin, 1980) to investigate the possibility of reducing the complexity of the package would have both theoretical and practical benefits. 78 REFERENCE NOTES 1 Donnellan, A.M., LaVigna, G.W., Schuler, A.L., & Woodward, P.A. Widespread training and dissemination of behavioral techniques  for classroom use. Unpublished, manuscript, 1979. (Available from A.M. Donnellan, University of Wisconsin, Madison). 2 Noble, J. Provincial Inservice Resource Team (Operational Directive No. 20 - 1978/79). Victoria, British Columbia: Ministry of Human Resources, September 1978. 79 REFERENCES American Psychiatric Association. Diagnostic and statistical manual  of mental disorders (Third edition). Washington, D.C: American Psychiatric Association. Bandura, A. Social learning theory. Englewood Cliffs, N.J.: Prentice-Hall, 1977. Becker, W.C. Parents are teachers. Champaign, 111.: Research Press, 1971. Berkson, G., & Landesman-Dwyer, S. Behavioral research on severe and profound mental retardation (1955-1974). American Journal of Mental Deficiency. 1977, 81, 428-454. SLrnbrauer, J.S. Mental retardation. In H. Leitenber, (Ed.), Handbook of Behavior Therapy. Englewood Cliffs, N.J.: Prentice-Hall, 1976. Campbell, D.T., & McCormack, T.H. Military experience and attitudes towards authority. American Journal of Sociology, 1957, 62, 482-490. Campbell, D.T., & Stanley, J.C. Experimental and quasi-experimental designs for research. Chicago: Rand McNally, 1966. Cook, T*D., & Campbell, D.T. Quasi-experimentation: design and analyses issues for field settings. Chicago: Rand McNally, 1979. Crisen, M.S. Mental retardation: past, present and future. American Psychologist. 1975, 3J3, 800-808. Cuvo, A.J., & Riva, M.T. Generalization and transfer between comprehension and production: a comparison of retarded and non-retarded persons. Journal of Applied Behavior Analysis, 1980, 13_, 315-331. 80 Donnellan, A.M. Educational perspectives of autism: implications for curriculum development and personnell development. In B. Wilcox and A. Thompson (Eds.), Critical Issues in Educating Autistic Children  and Youth. Washington, D.C.: U.S. Department of Education, Office of Special Education, 1980. Donnellan, A.M., Gossage, L.D., LaVigna, G.W., Schuler, A.L., & Traphagen, J.D. Teaching makes a difference: A guide for  developing successful classes for autistic and other severely  handicapped children: teachers manual. Sacremento, Calif.: State Department of Education, 1977. Fabry, P.L., & Reid, D.H. Teaching foster grandparents to train severely handicapped persons. Journal of Applied Behavior Analysis, 1978, 11, 111-123. Ferguson, G.A. Statistical analysis in psychology and education. New York: McGraw-Hill, 1971. Fredericks, H.D., Anderson, R., & Baldwin, V. Identifying competency indicators of teachers of the severely handicapped. American  Association for the Education of the Severely and Profoundly  Handicapped Review, 1979, 4_> 81-95. Gardner, J.M. Teaching behavior modification to nonprofessionals. Journal of Applied Behavior Analysis, 1972, j>, 517-521. George, M.J., & Baumeister, A.A. Employee withdrawal and job satisfaction in community residential facilities for mentally retarded persons. American Journal of Mental Deficiency, 1981, 85_, 639-647* Gladstone, B.W., & Sherman, J.A. Developing generalized behavior-modification skills in high-school students working with retarded children. Journal of Applied Behavior Analysis, 1975, 8, 169-180. Glass, G.V., & Stanley, J.C. Statistical methods in education and  psychology. Englewood Cliffs, N.J.: Prentice-Hall, 1970. Goetz, L., Schuler, A., & Sailor, W. Teaching functional speech to the severely handicapped: Current issues. Journal of Autism and  Developmental Disorders. 1979, 9, 325-343. Grossman, H.J. Manual on Terminology and Classification in Mental  Retardation. Washington, D.C: American Association on Mental Deficiency, 1977. Gruber, B., Resser, R., & Reid, D.H. Providing a less restrictive environment for profoundly retarded by teaching independent walking skills. Journal of Applied Behavior Analysis, 1979, 12, 285-297. Hall, R.V. The responsive teaching-parenting model. Lawrence, Kansas: H & H Enterprises, 1978. Haring, N.G. Review and analysis of professional preparation for the severely handicapped. In B. Wilcox & R. York (Eds.), Quality  education for the severely handicapped. U.S. Department of Education, Office of Special Education, November 2980. Hartmann, D.P. Considerations in the choice of interobserver reliability estimates. Journal of Applied Behavior Analysis, 1977, 10, 103-116. Hermelin, B., & O'Connor, N. Psychological experiments with autistic  children. Oxford, England: Pergamon Press, 1970. H i l l , J.W., Wehman, P., & Horst, G. Toward generalization of appropriat leisure and social behavior in severely handicapped youth: Pinball machine use. The Journal of the Association for the .Severely  Handicapped, 1982, 6, 38-44. Hung, D.W. Training and generalization 'of yes and no mands in two autistic children. Journal of Autism and Developmental  Disorders. 1980, 10, 139-152. Kanner, L. Autistic disturbances of affective contact. Nervous Child, 1943, 2, 217-240. Kanner, L. A history of the care and study of the mentally retarded. Springfield, 111.: Charles C. Thomas, 1967. Kaufman, J.M. Editor's introductions. Analysis and Intervention in Developmental Disabilities, 1981, 1* 1~3« Kazdin, A.E. Behavier modification in applied settings. Homewood, 111. Dorsey Press, 1975* Kazdin, A.E. Assessing the clinical or applied significance of behavior change through social validation. Behavior Modification, 1977, 1, 427-452. Kazdin, A.E. Research design in clinical psychology. New York: Harper & Row, 1980. Koegel, R.L., Egel, A.L., & Dunlap, G. Learning characteristics of autistic children. In W. Sailor, B. Wilcox & L. Brown (Eds.), Methods of instruction for severely handicapped students. Baltimore, Maryland: Paul H. Brookes, 1980. Koegel, R.L., Glahn, T.J., & Neiminen, G.S. Generalization of parent-training results. Journal of Applied Behavior Analysis, 1978, 11, 95-110. Koegel, A.L., & Rincover, A. Research on the difference between generalization and maintenance i n extra-therapy responding. Journal of Applied Behavior Analysis, 1977, 10, 1-12. Koegel, R.L., Russo, D.C, & Rincover, A. Assessing and t r a i n i n g teachers i n the generalized use of behavior modification with a u t i s t i c children. Journal o f Applied Behavior Analysis. 1977, 10, 197-205. LaVigna, G.W., & Traphagen, J . The discrete t r i a l format; A programmed text. Sacramento, Ca.: State Department of Education, 1978. Liberty, K.A., Haring, N.G., & Martin, M.M. Teaching new s k i l l s to the severely handicapped. The Journal of the Association f o r the Severely Handicapped, 1981, 6, 5-13. Lindsley, O.R. Direct measurement and protheses of retarded behavior. Journal of Education, 1964, UJ_, 62-81. Lovaas, 0 .1 . The a u t i s t i c c h i l d : Language development through behavior modification. New York: Irvington Publishers, 1977. Lovaas, 0 .1. Contrasting i l l n e s s and behavioral models for the treatment of a u t i s t i c children: An h i s t o r i c a l perspective. Journal of Autism and Developmental Disorders, 1979, 9, 315-323. Lovaas, 0.1., & Koegel, R.L. Behavior therapy with a u t i s t i c children. In C E . Thorensen (Ed.), Behavior therapy i n education (part l ) . Chicago: University of Chicago Press, 1972. Martin, G., Pallotta-Cunick, A., Johnstone, G., & Goyos, A.C. A supervisory strategy to improve work performance for lower functioning retarded c l i e n t s i n a sheltered workshop. Journal of Applied Behavior Analysis, 1980, 13_, 183-190. Martin, R. A l l handicapped children are educable. Analysis and Intervention in Developmental Disabilities, 1981, 1, 5-11. McKillip, J. Impact evaluation of service programs: Three flexible designs. Evaluation Quarterly, 1979, 2 , 97-104. Mithaug, D.E., & Mar, D.K. The relation between choosing and working prevocational tasks in two severely retarded young adults. Journal of Applied Behavior Analysis. 1980, 13_, 177-182. Nelson, D.L., Gergenti, E., & Hollander, A.C. Extra prompts versus no extra prompts in self care training of autistic children and adolescents. Journal of Autism and Developmental Disorders, 1980, 10, 311-322. Nutter, D., & Reid, D.H. Teaching retarded women a clothing selection s k i l l using community norms. Journal of Applied Behavior Analysis, 1978, 11, 475, 487. Panyon, M., & Patterson, E. Teaching attendants the applied aspects of behavior modification. Mental Retardation, 1974, 12, 30-32. Patterson, G.R. Families: Applications of social learning to family l i f e . Champaign, 111.: Research Press, 1975. Powers, D.E., & Alderman, D.L. Practical techniques for implementing true experimental designs. Evaluation Quarterly, 1979, 3_, 89-96. Rimland, B. Infantile Autism. New York: Appleton-Century Crafts, 1964. Rincover, A., & Koegel, R.L. Settii g generality and stimulus control in autistic children. Journal of Applied Behavior Analysis, 1975, 8, 235-246. Rosen, M., Clark, G.R., & K i v i t z , M.S. The history of mental retardation  Collected papers (2 v o l s . ) , Baltimore, Maryland: University Park Press, 1976. Schinke, S.P., & Wong, S.E. Evaluation of s t a f f t r a i n i n g i n group homes for retarded persons. American Journal of Mental Deficiency, 1977, 82, 130-136. Schlein, S.J., Ash, T., Kierman, J . , & Wehman, P. Developing independent cooking s k i l l s i n a profoundly retarded woman. The  Journal of the Association f o r the Severely Handicapped, 1981, 6, 23-29. Shepherd, M. Disabled and the law. Vancouver, B.C.: People's Law School, 1982. Strain, P.S., Kerr, M.M., & Ragland, E.U. Effects of peer mediated s o c i a l i n i t i a t i o n s and prompting/reinforcement procedures on the s o c i a l behavior of a u t i s t i c children. Journal of Autism and  Developmental D i s a b i l i t i e s , 1979, 9_, 41-54* Watson, L.S. J r . How to use behavior modification with mentally  retarded and a u t i s t i c children: Programs f o r administrators, teachers, parents and nurses. Tuscaloosa, Alabama: Behavior Modification Technology, 1972. Watson, L.S. J r . Child behavior modification: A manual for teachers, nurses and parents. New York: Pergamon Press, 1973* Weissman-Frisch, N., Crowell, F.A., & Inman, D.P. In-servicing vocational t r a i n e r s : A multiple perspective evaluation approach. The Journal of the Association for the Severely Handicapped, 1980, 158-173. 86 Winer, B.J. Statistical principles in experimental design. New York: McGraw-Hill, 1971 Wing, L., Yeates, S.R., Brierley, L.M., & Gould, J. The prevalence of early childhood autism: comparison of administrative and epidemiological studies. Psychological Medicine, 1976, 6, 89-100. Wolf, M.M. Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of  Applied Behavior Analysis. 1978, 11, 203-214. Wolfensberger, W. Normalization: The principle of normalization in  human services. Toronto: National Institute on Mental Retardation, 1972. 87 APPENDICES 88 APPENDIX A 89 Manual for PIRT Training Program Douglas Lee 1980 90 Site Visit The purpose of the site visit is to finalize the organizational and administrative requirements in preparation for the workshop. The team member acting as contact person is responsible for meeting with the intake and review committee to make sure that a l l the necessary arrangements are done for the workshop, (e.g., a room is booked, babysitters and childcare workers have been found, and substitute replacements for trainees are available). As the specific requirements of each community varies widely, i t is usually necessary to draw up an unique l i s t of requirements which must be fulfilled for the workshop. A l l team members are responsible for a certain portion of the trainees and target clients during the site vi s i t . For each trainee, 8 items must be covered: 1. Behavior terminology test administered. 2. DTF teaching video (10 consecutive t r i a l minimum completed) 3. Copy of the trainees most recent written program collected. 4. Video release is signed. 5. Academic record sheet completed. 6. Manual given (money collected i f not a MHR employee). 7« Program text given (money collected i f not a MHR employee). 8. Trainee information sheet completed. All the items except the video taping are.completed during an interview that is either one to one or in a small group (e.g., 3 trainees from the same work setting may be seen together). As the requirement for the site visit will have been specified in some detail during the pre-referral visits, i t is rarely necessary to give more than the briefest overview rationale for requesting the information at the site vi s i t . However, as many trainees have not been videotaped before or seem anxious about being videotaped, as much time as required is spent in making sure the trainees are as relaxed as possible. One way of dealing with the trainee's anxiety is to reassure her or him that the purpose of the videotaping is to help the team gather information which will be used to better tailor the content of the workshop to the trainee's own needs, and to turn on the camera and then walk away from the camera. Also, as the videotaping i s being done with the trainee working with a target client, i t often appears that the trainee's perception is that the video is more on the target eld ent than on herself. The issue of the trainee's perception of the team's focus is an area in which the team often times must spend considerable energy. It has become very obvious that the common perception of consultants coming into a community is that th^ir focus will be almost1 exclusively on the target client. The consultant will come in, "assess" the client, write out a diagnostic/prescriptive report, and then leave. The team's perspective is that they will deal with the client as much as possible only through the trainee. It is the trainee who is the direct target of the team with the target client being the indirect target. Helping the trainees* make this shift is very important in order that their expectations be supportive of the events to take place during the workshop. In previous training programs when much less effort was directed towards ensuring the trainees' perceived themselves as the direct target of the team, some trainees would express dissatisfaction with the team for spending so much time with them, and so l i t t l e time working with target clients. The other major responsibility of the team is to observe the target client. Prior to the site visit, as much written assessment material as possible has been gathered on the target client. These reports are helpful in providing general information. The observations by the team member are aimed at gathering present functional information on how the target client deals with his present environment. Some general areas of interest are: 1) how does the client interact with the teachers/trainers a) during instruction b) during non-instruction 2) how does the client interact with other people 3) how does the client interact with the physical environment a) task materials b) non-task materials Due to the idiosyncratic nature of this particular target client populations'" learning history, learning strategies, and perceptual abilities, just what areas will be focussed on is impossible to detail a priori. During or following this observation period, i t may arise that clarification of what is being observed can only be done by actual interaction with the target client. Often this is done to assess how the client will react to different teaching strategies. As the trainees usually do not have a variety of different teaching strategies in their repetoi-re the team member must do the assessment directly with the target client. Occaions have arisen in the past where the more important question is "What will happen to the trainee/target client interaction i f i t is the trainee who actually implements the new procedure?" An example 93 of where this may be the question is when the interaction is such that the trainee is getting the exact opposite response they want and this appears to be due to the positive and negative contingencies being misapplied to the target client 1s behavior. Since the two people in this interaction may have been involved in this S - R - C system for an extended period of time, i t is possible that the interaction is perceived as a formalized structure by the client. The effects of a new person (team member) altering this structure may be totally different than having the trainee alter the structure of the interactions. In order to accomplish the above mentioned items, the team members usually book approximately hours per trainee. This is broken into introduction (outlining purpose of visit and rapport building), observation and videotaping, and meeting to complete various forms, and the terminology test. As the team is usually at a site for 2-3 days, i t is possible that more observation, videotaping or discussion with the trainees will be scheduled during the vis i t . Workshop Due to the variability between training sites on such factors as the stated needs by the trainee, functional level of target clients, language capability of target client, size of training group, and the role the trainees have within the service delivery context for the target clients, the specific content of a workshop is quite variable. However, there is a structure to the workshop which is identifiable over and above this contextual variability. It is important to remember that the structure and the content to be presented will be somewhat idealized in order to present as clear a model of the workshop format as possible. The description will be broken down into the five days of the workshop. 94 D a y 1 9:00 - 10:15 Introductions: As the trainees may be from several community facilities and the team members may not have met a l l trainees during the site visit, quite formal introductions are necessary in order to start developing a sense of group cohesion and as an i n i t i a l rapport step. Goals: From the trainee's expressed needs, the observational data from the site visit, and review of the assessment materials on the clients the team will draw up a series of goal statements which attempt to pull together the information from these sources. Since the workshop is not the end of the training program, but rather the intensive i n i t i a l learning phase of a protracted training involvement between the team and the trainees i t is also necessary to state along with the goals of the workshop, how the workshop fits into the entire training program. An essential feature to point out is that the trainees are not expected to be able to immediately and independently utilize everything that will be taught during the: workshop. It has happened with some frequency that trainees come to view the workshop as the end of the team's involvement which leads to an overwhelming sense that they cannot possibly learn " a l l this" so fast. This sense of being overwhelmed can greatly interfere with the person's acquisition of the material. Often i t is necessary to reiterate for individual trainees during the workshop that the team will be back several times over the next few months. Autism Overview: Although most of the trainees will be working with clients who have autistic or autistic-like behaviors, i t is quite common that the trainees will not have had a descriptive 95 overview of the syndrome. A film called "Bobby" and/or a verbal overview presentation on autism and the behavioral therapy techniques used with autistic people are given. Even i f several trainees in a workshop have an adequate knowledge of this area, the overview is s t i l l done to try and provide some uniformity to the trainee's level of information and also to indicate the team's behavioral perspective. 10:15 - 10:30 COFFEE 10:30 - 12:00 Introduction to DTF teaching: There are several components to this introduction. 1) "Denise" VTR - a training tape made by PIRT and WIRT which gives a rationale for this type of teaching, defines the components of the technique and gives examples of its use. 2) The team will demonstrate the use of DTF by 2 members role-playing a teaching situation. The behavior which is used in this role-play will be one which has been mentioned by the trainees as a behavior they have trouble teaching. This demo i s followed by a discussion. 3) A further demonstration will be done. The trainees will be broken down into small groups with a team member and the trainees will take turns role-playing teaching the behavior from the previous demonstration. This will be followed by a discussion. 2+) As the majority of trainees will be experiencing difficulties with their clients because they have not found a method for training the client in the prerequisite behavior for working on a task (e.g. attention to task, sitting s t i l l for several minutes), a small lecture will be given on what prerequisite behaviors are 96 and how they relate to the teaching task. As the prior demonstrations or roleplay is often centered on teaching eye-contact, sitting in a seat or keeping your hands s t i l l , the lecture is often an elaboration of the prior exercises. 12:00 - 1:15 LUNCH 1:15 - 2:30 Following-up from the mini-lecture on prerequisite behaviors, the team will demonstrate by role-play, the teaching of eye-contact or sitting s t i l l . This is followed by the trainees breaking into small groups and role-playing the teaching of one of these behaviors. 2:30 - 2:45 COFFEE 2:45 - 3:15 Up to this point the focus has been upon the face-to-face teaching interaction. At this point, the team will role-play a teaching session, introducing the concept of pass and f a i l criteria. After a brief introduction to the concept, the team members will do a role-play in which the "student" will pass from one step to the next and will f a i l one step, necessitating a return to a previous step and a review. 3:15 - 3:45 A film, "Teaching Makes a Difference #1" is shown to summarize much of the day's work. 3:45 - 4:00 Homework assignments: This will involve a review of chapters in the Teaching Makes a Difference manual in preparation for Day 2, plus 1 or 2 other relevant readings. Day 2 9:00 - 9:30 The day opens with a discussion period to clarify questions from the previous day. If a definition l i s t was given out, this is reviewed at this time. 97 9:30 - 9:45 A mini-lecture which defines what a baseline is and a rationale for using a baseline is given. 9:45 - 10:15 The trainees break into small groups with a team member. The target clients are introduced into the workshop environment at this point to start the direct practice sessions between the clients and trainees. This practice session will focus upon the collection of baseline data usually on an attending/ prerequisite behavior. Each trainee will do 10 trials with the client. This is the first session in which the trainee and clients will be videotaped during the workshop. All other direct practice sessions will also be videotaped. The videotaping will be rotated through the groups. Not a l l trainees will be videotaped each session. The baseline data collected on the target clients will be used as the baseline data for the workshop and i t is stressed that i f the trainees want to work on this target behavior after completion of the workshop, a new baseline will have to be done. 10:15 - 10:30 COFFEE 10:30 - 11:00 A lecture is given introducing program writing. Trainees are given an example program and the program writing guide. Using an enlarged version of the example program, each component of the program is discussed. The format of the program writing sheet is very similar to that presented in the "Teaching Makes a Difference" manual. Particular stress is laid on the "how to" of writing a good program. 11:00 - 12:00 The trainees break into small groups and start to write a program. The intent of this session is not to complete an entire program, but to have enough of the program outlined so that i t can be implemented with the clients after lunch. 9 8 12:00 - 1:15 LUNCH 1:15 - 2:00 Direct practice session with clients using program outlines written before lunch. This is a particularly useful point for highlighting confusions and ambiguities in the program that i s being used as the trainees often have used unclear wording or ill-defined terms in their program. However, once other people are attempting to utilize their programs the need for explicit statements becomes evident. Throughout the directed practice sessions occasions arise which enable you to compare the trainee's expectations of the client's ability to perform a certain task (by what they write in their programs) and the client 1s actual ability level. The benefit of having explicit expectations in a program (e.g., how much prompting is required, pass/fail criteria) for comparison to the client's actual behavior can be exemplified. This could take the form of suggesting to the trainee that the program (especially a first draft) is your best guess of how the client will learn this task. How well the client does (as noted from your data) is feedback as to your best guess. By altering components of your program until you get the data you want, two results will occur: the client will learn the behavior and you will end up with a written record of an effective teaching strategy. 2:00 - 2:30 Film "Teaching Makes a Difference #2". This film focuses on the issues of how to build a programmed curriculum for each individual into a special classroom environment. 2:30 - 2:45 COFFEE 2:45 - 3:45 A lecture/discussion is done focussing on maintenance and generalization issues, and, rewards and reinforcement schedules. All these issues are dealt with together in order to provide a guide 99 of how the trainees can move their clients from the schedule of reinforcement and types of rewards used to teach a behavior into schedules and rewards which will maximize the maintenance and generalization of the new behavior. 3:45 - 4:00 Homework assigned: review chapters in manual for tomorrow, complete definitions l i s t ; either complete the program started today, or write one for a new behavior for a different client. Day 3 9:00 - 9:15 Hand in programs written as homework. Review definition l i s t with discussion. 9:15 - 10:00 The program components of prompt, probes and branching were outlined in the program writing lecture on Day 2. They are now gone into in more detail. Specifically various different prompts are outlined (manual, proximity, and within stimulus) and more detailed rules for probing and branching are given. 10:00 - 10:15 During the previous hour, two team members would have reviewed the programs, chosen the best ones for the clients present today, scored them, and written comments. These programs are presented to the group. 10:15 - 10:30 COFFEE 10:30 - 11:45 The trainees break into small groups, each group with a program. The trainees first role-play the program to become familiar with i t and then have a directed practice session with the client, attempting to follow the written program. 11:45 - 12:00 During the preceding practice session, one of the team members has finished marking a l l the programs. The programs with 100 written feedback are given back to the trainees. During this period the marker reviews the programs in general for the whole group. Questions on specific questions will be dealt with through-out the day. 12:00 - 1:15 LUNCH 1:15 - 2:00 Lecture on language/communications training. Due to the highly variable language and cornmunication abilities of the clients, this lecture will often change drastically from workshop to workshop. However, a major intent of this lecture is to focus on the need for developing a communication method which each client can use. As many clients will often exhibit almost no structured communicative ability, time i s also spent on how to start teaching language to someone with l i t t l e or no skills in this area. 2:00 - 2:30 Either a role-play demonstration or a video-tape presentation will be done to exemplify teaching strategy issues related to communication training. 2:30 - 2:45 COFFEE 2:45 - 3:45 A direct practice session with the clients focussing on teaching communication behavior. This practice session is used for gathering ideas to write a language program for homework. 3:45 - 4:00 Homework; review chapters in manual. Write a language program. Complete definition l i s t . Day 4 9:00 - 9:15 Hand in homework, review definitions, and discussion. 9:15 - 10:00 The principles of behavior management are outlined in a° lecture format. As there is a session later in the day focussing on techniques of behavior management, this present session is more 101 on defining operant principles than techniques. 10:00 - 10:15 Discussion of scored programs to be used for directed practice. 10:15 - 10:30 COFFEE 10:30 - 11:45 Role-playing programs and then practicing the implementation of the program with the clients. 11:45 - 12:00 Programs done for homework handed back with general comments to the trainees. 12:00 - 1:15 LUNCH 1:15 - 2:15 Techniques of Behavior Management lecture. Source material can be found in LaVigna & Donnellan (1976). As a focal point of the talk, an example behavior management program is given to the trainees. An oversized version of the example is used by the team member for reference purposes. An explicit bias of this session i s on non-averadve (particularly avoiding intrusive methods) methods of behavior management. As a behavioral observation exercise, videotaped segments are used, where the trainees are asked to count the frequency of a certain behavior during several 2 minute segments. Each person's data is reported which allows for a discussion of observer reliability and to highlight that behavior naturally occurs at a variable rate. 2:15 - 2:30 COFFEE 2:30 - 3:00 An introduction to task analysis is given with its relationship to program writing. A task analysis exercise is done where the trainees break into dyads and with their backs turned to each other, each trainee, in turn, must describe step by step how the other trainee is to do a task. 3:00 - 3:45 Film "Teaching Makes a Difference #3". This film overviews many of the areas covered in the previous two days. 3:45 - 4:00 Homework. Review chapters in Manual. Do a task analysis or write another program. Complete definition l i s t . Day 5 '9:00 - 9:15 Hand in homework, quiz and discussion on definition l i s t and readings. 9:15 - 10:15 Curriculum lecture. Over the past four days, the components which are required to make a curriculum for an individual client have been taught. These components include such content areas as prerequisite behaviors, language skills, behavior management and self-help skills. How to write programs, collect data, set criteria, use shaping, DTF, and schedules of reinforcement are some of the techniques focused on during the workshop. The intent of this lecture is to provide the trainees a schema for pulling a l l this information together to make an overall plan for each client. Much of the material covered is from the literature on program writing. Since putting an individual curriculum together is a very complex task, this session i s very much a conceptual introduction into curriculum. Many of the pragmatic issues of curriculum construction are dealt with during the follow-up visits. 10:15 - 10:30 COFFEE 10:30 - 11:45 Small group planning is done at this time in preparation for the first follow-up. The intent of this planning is to set out what each trainee is going to attempt between the workshop and the first follow-up. Specific requests for information are noted so the required materials can be ready for the follow-up vis i t . As i t is obvious that the trainees could not hope to use a l l the information and skills learned during the workshop immediately, an attempt is made to have the trainees focus on one client, one behavior across 2 or 3 clients, or on using one teaching technique. The integration of the other information from the workshop will be done in a step wise fashion at a pace that is not too disruptive to the trainee's work setting.. 11:45 - 12:00 Homework feedback. General feedback is given regarding the task analysis or programs done as homework. 12:00 - 1:15 LUNCH 1:15 - 1:45 Lecture on token economy. After defining what a token economy is, the time is spent on how to build one. This goes from the i n i t i a l pairing of the token to a primary reinforcer to f u l l token economics with several reinforcers costing differential amounts. 1:45 - 2:30 Each evening the videotapes of that day's direct practice sessions were scored. From the previous 3 days of videotapes, general comments are pulled together to give to the group as a whole. Individual feedback is also given. If i t is possible to find select segments of videotape that highlights the comments made, they are shown after requesting the trainee's permission. 2:30 - 2:45 COFFEE 2:45 - 3:00 The terminology mid-test is given. 3:00 - 3:45 The written trainee feedback is done at this time, followed by an open feedback and comments period with the whole group. 104 3:45 - 4:00 A wind-up period used for closing off the workshop and for final preparations for the follow-up v i s i t . Follow-up Visits The structure of the follow-up visit is more variable than that of the workshop. A main factor for this variability is that the focus during the follow-ups is towards helping each individual trainee generalize the skills and information of the workshop to their normal work setting. As noted in the small group planning session on Day 5 of the workshop, each trainee has specific goals set for themselves. At an abstracted level, the flow of the follow-up could be characterized as moving from the specific to the general. For example, the goals set for the trainee for the first follow-up are specific to one client, one type of behavior or one teaching technique. The intent is to move from this starting point to other clients, types of behavior, or teaching techniques. The rate of movement however, will be highly variable dur to the Environment x Client x Trainee interaction. Recognizing the diversity of the follow-up goals, there are several common themes followed during the large majority of follow-up visits. These are: l) Direct observation of the trainee working with a client. As the trainee is usually requesting a team member to observe a particular teaching session because something isn't working, the team member may role-play or model with the client what i t is that could be done differently. This observation will also include checking on whether there is a written program for this behavior and reviewing the data that has been collected since the previous visit. 105 2) Program Writing: has the trainee written any programs since the previous visit? If so, give feedback. If not, what has stopped the trainee from doing so. Do the trainees want any input on altering an existing program. The programs written by trainees during follow-up are often sketchier than those done in the workshop. This usually gives ample opportunity for discussing many relevant issues. Common components left out are generalization, maintenance, pass-fail criteria and branching or alternate steps i f the program does not work. These issues often can lead into discussions of long term goals (curriculum) and issues pertaining to how the trainee perceives the client's learning style. 3) Data: although the collection of data is part of the teaching and program writing techniques taught during the workshop, special emphasis is placed on data during the follow-up visits. There are two reasons for this. First, data collection is very foreign for most trainees and without support they could not collect any data. Secondly, data is seen by the team members as a cornerstone and feedback mechanism for so much of what they wish to accomplish with the trainee. 4) Curriculum: The issues surrounding curriculum often develop over the entire series of follow-up visits. If curriculum is to be a major issue for the trainees, i t usually becomes important during the third or fourth v i s i t . Such package curriculums as the Behavioral Characteristics Progression (BCP) and Student Progress Report (SPR) are often brought along on a follow-up visit as an aid in helping the trainee develop a structured approach to building an individualized curriculum. 106 5) Behavior Management: A troublesome issue for many trainees is how to deal with the behavioral excesses or deficits which are often exhibited by this client population. It i s a particularly thorny issue in that, in the long run, the best method for dealing with these behaviors is to put the client in an enriched productive training environment suited to his specific needs. In the short term, however, the trainee has an almost impossible time implementing the teaching strategies which could lead to a positive learning environment because of the behavioral excesses or deficits. It is very important that the work and energy which is taken on dealing with a behavioral excess or deficit not overshadow the work needed to build a positive teaching environment. In carrying out a follow-up visit, i t i s important to reinforce the many good things the trainees are doing. Without the support of the team members the trainees can quickly become overwhelmed by the task of integrating a l l the materials presented by the team into everyday practice. Also, because so many of the techniques taught to the trainees are made up of several simple components, the trainees can easily lose track of how much they have learned since each component is so easy to learn. It often takes the perspective of an outsider to help the trainees appreciate how much they have learned. Because there will only be five follow-up visits, the team must start early on reinforcing independence by the trainee. This independence is particularly related to the trainee initiating requests rather than waiting for a suggestion from the team. Once the termination visit is completed, any further contact between the team and the trainees will only occur i f the trainee initiates the request. 107 Termination Visit Besides being the conclusion of the training program, this visit i s the time for gathering the post-test data. A copy of the most recent program written by the trainee is collected, the terminology test is readministered and a videotape segment i s done of each trainee's teaching (a minimum of 10 trials). As well, the trainees are asked to complete a follow-up feedback form. Since the follow-up visits have been scheduled at progressively longer periods with the termination visit being approximately 8 weeks after the last follow-up, there will have already been a naturally occurring distancing between the trainees and team members. The conclusion of the training program is then not a sudden finish, but rather a cumulative end point. Every effort is made at this point to indicate that the team's role is now changing to a consultive role and that further contact will occur only at the request of the trainee. 108 APPENDIX B 109 Provincial In-Service Resource Team NANAIMO WORKSHOP November 26th - 30th 1979 MONDAY, NOVEMBER 26th 1979  9:00 - 9:30 Introduction and Schedule 9:30 - 9:45 "Bobby" Film 9:45 - 10:15 "Why D.T.F." Autism Therapy and Teaching Techniques 10:15 - 10:30 Coffee 10:30 - 10:45 "Denise" VTR and "Group Teaching" VTR utilizing DTF components 10:45 - 11:00 D.T.F. Demonstration 11:00 - 11:10 Discussion of Demonstration 11:10 - 11:20 D.T.F. Demonstration 11:20 - 11:50 Mini Lecture Teaching Pre-requisite Skills 11:50 - 12:00 Discussion 12:00 - 1:15 Lunch 1:15 - 1:45 Demonstration of Teaching Attending Skills 1:45 - 2:30 D.T.F. Role Play eye contact Programme 2:30 - 2:45 Coffee 2:45 - 3:15 D.T.F. Role Play with Criteria 3:15 - 3:45 T.M.D. #1 - film 3:45 - 4:00 Homework Assignment: Review Chapters 2 & 4 & 6 in manual Read Kozloff Handout Complete Definition List 110 NANAIMO WORKSHOP TUESDAY. NOVEMBER 27th 1979  9:00 - 9:30 Quiz and Discussion 9:30 - 9:45 Mini Lectures - Baselining & Assessment 9:45 - 10:15 D.T.F. Hands on - Baselining Eye Contact 10:15 - 10:30 Coffee 10:30 - 11:00 Programme Writing Lecture 11:00 - 12:00 Programme Writing 12:00 - 1:15 Lunch 1:15 - 2:00 D.T.F. - Hands on with Programmes 2:00 - 2:30 T.M.D. #2 2:30 - 2:45 Coffee 2:45 - 3:30 Generalization, Rewards & Reinforcement Schedules, and Maintenance 3:30 - 3:45 Discussion 3:45 - 4:00 Homework: Review Chapt. 5» 12 in manual Read Reward Handout Complete Definitions List Finish Programme started, earlier including Generalization NANAIMO WORKSHOP WEDNESDAY, NOVEMBER 28, 1979 Hand in Homework 9:00 - 9:15 Quiz and Discussion 9:15 - 10:00 Prompts, Probes, Branching 10:00 - 10:15 Discussion of Programmes 10:15 - 10:30 Coffee 10:30 - 10:45 D.T.F. - Role Playing Programmes 10:45 - 11:45 D.T.F. - Hands on with Programmes 11:45 - 12:00 Homework Feedback 12:00 - , 1:15 Lunch 1:15 - 2:00 Teaching Language Lecture 2:00 - 2:15 D.T.F. - Demonstration of Language Programme, V.T.R. 2:15 - 2:30 D.T.F. - Role Playing with Language Programme 2:30 - 2:45 Coffee 2:45 - 3:45 D.T.F. - Hands on re: Language, Assessment, Teaching and Programme Writing 3:45 - 4:00 Homework: Review Chapter 3 Write Language Programme Complete Definition List NANAIMO WORKSHOP THURSDAY, NOVEMBER 29th 1979  9:00 - 9:15 Hand in Homework Quiz and Discussion 9:15 - 10:00 Principles of Behavior Management 10:00 - 10:15 Discussion of 3 Programmes 10:15 - 10:30 Coffee 10:30 - 10:45 D.T.F. - Role Playing with Programmes 10:45 - 11:45 D.T.F. - Hands on with Programmes 11:45 - 12:00 Homework Feedback 12:00 - 1:15 Lunch 1:15 - 2:15 Techniques of Behavior Management 2:15 - 2:30 Coffee 2:30 - 3:00 Task Analysis 3:00 - v3:45 T.M.D. #3 3:45 - 4:00 Homework: Review Chapter 8, 9, & 10 - manual Do a Task Analysis Complete Definitions List NANAIMO WORKSHOP FRIDAY. NOVEMBER 30th 1979 9:00 - 9:15 Hand i n Homework Quiz 9:15 - 10:15 Token Economy 10:15 - 10:30 Coffee 10:30 - 11:45 Small Group Planning 11:45 - 12:00 Homework Feedback 12:00 - 1:15 Lunch 1:15 - '1:45 Trainee's requests for additional information 1:45 - 2:30 Video Feedback 2:30 - 2:45 Coffee 2:45 - 3:00 Mid-test 3:00 - 3:45 Feedback to P.I.R.T. 3:45 - 4:00 Wind up, Preparation for follow-up 114 APPENDIX C 115 Name Date PROVINCIAL INSERVICE RESOURCE TEAM (PRE) and (POST) TEST ON TERMINOLOGY For questions with blanks, please place your answer in the blank. For a l l other questions please circle your choices. This test will take approximately 15 minutes. 1. The teaching procedure where you reinforce closer approximations to the goal behavior is called: -a. chaining b. shaping c. prompting 2. If a teacher or parent does not have complete control over a l l the reinforcement which maintains an undesirable behavior he/she cannot use _____________________ as a procedure for eliminating that behavior. 3. Which of the following is most important in early language teaching: -a. teaching a child to label objects and activities in his environment. b. teaching a child to distinguish vowel and consonant sounds. c. teaching a child to express her/his needs and desires. 4. The ___________________ indicates to the student that i t is time for a particular response to occur. 5. _______________________ is needed in a training program so that we can accurately and reliably assess whether the program is helping the child to learn. 116 It is important to remember that a good assumption to take when teaching an autistic like child i s that, she/he will learn what you teach them: -a. only b. eventually c. never d. always A child's language, which is unintelligible to anyone except someone who knows the child very well, (e.g. the sound "d" to mean down) can be: -a. taught to other language-delayed children in the class b. shaped into more intelligible forms of communication c. extinguished by ignoring i t d. rewarded differentially each time i t occurs If you decide upon a set of for each step of a program, you will more easily be able to decide when to move on to the next step, or when to stop a program. If a behavior (such as self-stimulation) occurs very frequently during the day, is the most appropriate way to collect data. a. event recording b. time sampling recording c. duration recording d. anecdotal recording A prompt is used to a correct response. 117 11. For many programs, particularly self-help skills chaining is most effective. a. backward b. successive c. simple d. contingent 12. Three of the most important pre-learning behaviors which the child must have are: - (circle three alternatives) a. can talk or sign with the instructor b. can work at some task c. has eye contact d. does not have any self stimulation e. has good fine motor skills f. i s quiet for a minimum of 30 seconds g. cooperates with simple commands 13. The two most important criteria that should be met when using prompts are that they must be ___________ and they must be able to be 14. The consequence should be delivered following the child' s response. 15. It is essential to get the child's before giving the instruction. P.I.R.T. SCORING KEY FOR BEHAVIORAL TERMINOLOGY TEST B Extinction (no other) C S ; instructional stimulus; instruction; cue Record keeping; data A B Criteria B Insure; assure; eli c i t A B; C; G Effective: faded; eliminated; extinguished; phased out Immediately; promptly; directly Attention; eye contact 119 APPENDIX D 120 EVALUATION FORM FOR ASSESSING NAME OF PROGRAMME WRITER DTF PROGRAMME WRITING COMPETENCY  1. TOTAL SCORES 2. 3. 4. 5. FACE SHEET Target Behavior a. Stated i n observable behavioral terms (duration, topography, intensity). _ b. Statement of behavior's relation to S^  or instructional stimulus. Prerequisite Skills a. Lists a l l c r i t i c a l , already learned s k i l l s , needed to learn new behavior. Setting/Time a. A l l settings listed as concretely as required for clarity. b. Times of programme use listed. c. Any specified settings or times when programme not used are stated. Materials a. List includes a l l required materials. b. Quantities of a l l materials given. c. Important specifications of materials li s t e d . Baseline a. Number of t r i a l s for baseline. b. Pass crite r i a given. c. d. F a i l c r i t e r i a given. Results given when possible. I f not possible count as N/A. (ie. during workshop). Probe Procedure a. A l l probe steps listed, or the system of probing described. Parts a,b,c, b. Pass crite r i a given. are optional i f listed i n c. F a i l c r i t e r i a given. program steps + = correct/complete - = incorrect/incomplete N/A = not applicable or not taught DATE OF EVALUATION 1. 2. 3. 4. 5. ! ! i PROGRAMME SHEET 121 NAME OF PROGRAMME WRITER 1. 3. 4. Follow-^up/Generalization a. Procedure of monitoring m aintenance of behavior. b. Maintenance pass crite r i a given. c. Maintenance f a i l c r i t e r i a given. d. Branch or recycle step given for f a i l c r i t e r i a . (Optional, i f not stated count as N/A). e. Outline steps for generalizing programme. Instructional Stimulus a. Simple - appropriate language level for the child. I f i s situational, count as N/A. b. Consistent with goal behavior. Prompt a. Prompts described in behavioral terms. • b. Prompts faded in consistent manner I f not for child's functioning level. required for c. Specification of timing of prompt program count (i.e. during forward chaining) a,b,c, as N/A. Response a. Stated in behavioral, measurable terms. b. Time limit for response completion given. c. If time criterion changes that i t i s done consistently. d. Topography or intensity of behavior changes consistently towards target behavior. Consequence/Schedule a. Positive reinforcement specified for correct behavior. b. Punishment i s specified for incorrect behavior. c. Schedule of reinforcement for correct behavior given. d. Schedule of punishment for incorrect behavior given, (optional) Criterion a. Pass criterion for each step given. b. F a i l criterion for each step given. c. Branching or recycling step/system indicated for> f a i l c r iteria, (optional). I d. Pass criterion for probes given. Parts d and e are optional i f listed e. F a i l criterion for probes given, on face page. I 122 NAME OF PROGRAMME WRITER PROGRAMME SHEET 1. 2. 3. 4. 5. Data a. Indicate general method. _ b. Indicate specific notation. t Others i a. Probes are specifically notated in program steps (optional i f listed on face page) | be sure to add the scores from a l l pages. TOTAL ITEMS SCORED MINUS TOTAL ITEMS INCORRECT EQUALS TOTAL ITEMS CORRECT 123 EVALUATION OF PROGRAMME WRITING COMPETENCY FOR D.T.F. PROGRAMME  SCORING KEY DEFINITIONS Face Sheet Target Behavior -a. Stated in observable behavioral terms, with reference to duration, topography, and intensity, i f applicable. The statement of duration should include "time to completion" upon the instruction, i.e. Upon the instruction "get dressed", John will be dressed within 5 minutes. b. Within the description of the target behavior, there is a description of the instructional stimulus or discriminative stimulus for the behav-ior, (see above example) Prerequisite Skills -a. Lists a l l the critical prerequisite skills the child needs in order to complete the task. These skills are skills the child has already mastered and should not be included in the programme as behaviors to learn. Setting/Time -a. All settings for the programme listed as concretely as required for clarity. b. Times of programme implimentation stated i.e. 9:00 - 9:30 a.m. c. Any specified settings or times when the programme is not to be used are listed i.e. dressing programme not used during swim period. controlling voice level programme not used during recess. 124 Scoring Key Definitions - DTF Page 2 Materials -a. List includes a l l required materials. b. Quantities of a l l materials given. c. Any important specification of materials is listed - i.e.: Montessori stacking blocks, not "blocks" ski jacket with large zipper, not "jacket" Baseline Procedure -a. Number of trials for baseline. b. Pass criteria given. c. Fail criteria given. d. Results given, where possible. If not possible (i.e. workshop programme) count as N/A. If programme is in use but no data given count as error. Probe Procedure - (optional) If listed in programme steps count as N/A. Mandatory i f probe not listed in pro-gramme steps. a. A l l probe steps are listed, or the system of probing is described, i.e. after completion of any step after step 2 probe for mastery step. b. Pass criteria given. c. Fail criteria given. Follow-up/Generalization -a. Follow-up procedure described i.e. How to monitor maintenance of behavior after the last step of the programme is completed. b. Pass criteria given. (Pass criteria means termination of follow-up and programme). c. Fail criteria given. d. Fail criteria must be coupled with a branch or recycle step to achieve 125 Scoring Key Definitions - DTF Page 3 mastery level criteria. (Optional). If not stated, count as N/A. e. The steps for generalizing the programme is outlined, i.e.: - changes in reinforcement schedule - new related tasks to learn - other behaviors to be chained with this behavior. Programme Sheet NOTE: If programme has situational S (i.e. presentation of materials is Sd) then count "a" as N/A. Instructional Stimulus -a. Simple - Written at appropriate language level for the child. b. Consistent with goal behavior - i f goal behavior is to get dressed upon the command "get dressed", that the instruction is "get dressed", not "put on your socks" for one step and "put on your shirt" for another step. These other verbalizations should be listed as prompts. Prompt -NOTE: If prompts not required then "a,b,c" are N/A. a. Behavioral description of prompt type i.e. "hand on hand" rather than "manual prompt". b. Prompts are faded in a consistent manner throughout the programme, appropriate to functioning level of the child. c. Specification of the timing of the prompt, i.e. upon completion of steps 1 to 4 prompt for step 5» 126 Scoring Key Definitions - DTF Page k Response -Stated in behavioral, measurable terms. The time limit for the response completion is stated. If the time criteria is to change, that i t is done consistently, i.e.: 1 minute for step 1 - putting on socks 3 minutes for step 3 - putting socks, underwear and pants 5 minutes for mastery - getting completely dressed As the topography for intensity (not duration as that is evaluated in C) of the behavior changes toward the target behavior, that i t is done consistently throughout the programme steps. Consequences/Schedule -Positive Reinforcement specified for correct behavior. Punishment is specified for incorrect behavior. Schedule of reinforcement for correct behavior given. Schedule of punishment for incorrect behavior given. Criterion -Pass criteria for each step given. Fail criteria for each step given. Branching or recycling step indicated for f a i l criteria, or system described i.e. i f f a i l criteria met for steps lb - le recycle to last step. (Optional). If not given count as N/A. Pass criteria for probes given. Fail criteria for probes given. Scoring Key Definitions - DTF Page 5 Data Collection -a. Indicate general method i.e. t r i a l x t r i a l time sample b. Indicate specific notations used. Others -a. Probes are specifically notated in programme steps. (Optional i f listed on face page). + = correct/complete - = incorrect/not complete N/A = not applicable for programme/or not taught DL: jaor 16.7.80 128 APPENDIX E DTF SCORING SHEET INSTRUCTOR'S NAME CHILD'S NAME TAPE # START END PROGRAM INFORMATION DATE OF VIDEO I. II. III. IV. V. VI. TRIALS 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 PREPARATION: A. Fails to get attention B. Improper presentation of R.A.(s) INSTRUCTIONAL STIMULUS: A. Fails to assure attention B. I.S. is not distinct C. I.S. is inconsistent/incomplete PROMPT: A. Prompt'is inconsistent with prog. B. Prompt is improperly timed RESPONSE: ( A. R. is inconsistent with prog. def. B. R. is inconsistent with prog, timi CONSEQUENCE: A. Not easily discernable B. Not immediate C. Inconsistent/ambiguous INTERTRIAL INTERVAL: A. Fails to remove materials B. Fai."i s to record data C. Lack of distinct off-set COMMENTS: DL. 6.8.80 Total possible - Total incorrect = Total correct Percent score 130 EVALUATION FORM KEY DISCRETE TRIAL FORMAT TEACHING SECTION I: PREPARATION A. Preparatory attention: 1. Child is sitting s t i l l 2. Child has hands on table 3. Child is looking at teacher or general work area. 4» Child is quiet (N.B. Problems of judging whether or not the child is attending or not are enormous with some.children: often the correctness of the child's response or the assessment of the individual instructor of the readiness of the child are the only valid criterion the evaluator can use to decide i f the child is attending..) B. Proper presentation of response alternatives: 1. The instructor gives no external cues when presenting response alternatives. 2. The instructor does not obstruct the child's view of the response alternatives. 3. The instructor does not use the response alternatives to gain the child's attention; e.g. does not wave the R.A. (s) in the child's visual field in order to gain or gage the child's attention. SECTION II: INSTRUCTIONAL STIMULUS A. Child attends (see section I-A) B. Instructional stimulus is distinct: 1. Isolated; e.g. there is a pause between everything preceding, specifically the presentation of.the response alternatives or any attempts on the part of the instructor to regain the child's attention. 2. No extra verbalizing, coaching on the part of the instructor, e.g., Do not say the child's name after the I.S. i§ given. 131 C. Instructional stimulus is consistent and complete: 1. The instructional stimulus is always presented exactly as i t is outlined in the program, (it is alright to randomize the instruction i f the program is one of generalization). Note: In the event that the S is non verbal, such as the presentation of materials to be collated being the S to work, the instructional stimulus section is counted as non applicable i.e. N/A. This would also be the case for a program written for free operant eye contact. SECTION III: THE PROMPT (optional) Rule: The prompt is defined as any teacher behavior related to the task or the student which occurs between the Instructional Stimulus being given and the student's first movement to respond. Thus correction procedures are counted in the consequence section. EXCEPTION: If the program is a forward chaining procedure the prompt will occur after the beginning of the student's response. In this case the teacher's behavior would be counted as a prompt. For this to be the case however, you must have verification from the teacher that they are doing a forward chaining procedure. A. The prompt is consistent: 1. Prompt is always presented and faded as outlined in program. 2. The instructor uses no extraneous cues and prompts. B. The prompt is properly timed: 1. The prompt comes between the Instructional Stimulus and the response. For the prompt to be properly timed i t must also come before the end of the time period allotted for the response. To distinguish between late prompts, those that come after the response and a correction, i f the behavior .was used as a correct prompt in the past or is written in the program as a prompt, then mark i t as incorrect in the prompt box. If i t is any other behavior i t is marked in the consequence box. SECTION IV: THE RESPONSE A. The response is made and consequenced in complete accordance with the program definition. (Remember this is meant to evaluate the trainer's use of the behavior definitions for correct and incorrect responses not whether the child responded correctly or not). B. The response is timed within the boundaries outlined in the program definition, e.g. If the response definition states that the response is to occur within 10 sec. of the instruction that the trainer uses this as a criteria for judging the correctness of the response. 132 Note: Thus i f the teacher punished or extinguished a correct response then an error i s marked in the response box. If they reward an incorrect response, again an error i s marked in the response box. Of course, i f either of these situations occurs, i t will also mean the teache'r will be marked wrong in the consequence section for giving an ambiguous consequence. As can be seen then, a l l errors in the instructor's use of the response definition will lead to an error also being marked in the consequence section. This is justified, since the situation leads to very bad teaching behavior. SECTION V: THE CONSEQUENCE A. The consequence is easily discernible: 1. It must be heard or seen clearly on tape. B. The consequence must be immediate: 1. Unless specified the consequence should be given between 1-2 seconds upon completion of the response, or sooner. C. The consequence must be consistent and unambiguous: 1. It must follow exactly the definition outlined in the program. 2. It must be clearly related to the response. 3. The instructor should face the child when delivering a positive consequence, or verbal reprimand. If the trainer is ignoring the child, i t must be obvious on tape. RULE: A l l correction behavior of the instructor is counted in the consequence section. Correction behavior is defined as any instructor behavior towards the materials or the student which follows the beginning of the student's response, and is aimed at changing the child's immediate response. The exception to this definition is during a forward chaining procedure (see Section III Note and Exception). Unless designed as part of the teaching program (written in the program) a l l correction behavior of the instructor is marked as incorrect (item C - consequence is not consistent with program). If there is an acknowledged, structured, correction procedure, in the program and i t is done correctly then i t is evaluated as being ocrrect in the consequence section. SECTION VI: THE INTERTRIAL INTERVAL A. The instructor removed materials immediately following the delivery of the consequence. B. The instructor must record data after each t r i a l . 133 G. The t r i a l must have a distinct offset consisting of a 2-5 second pause before initiating a new t r i a l . However, i f the program calls for massed sequential trials as in collating stacks of papers, where the S is the materials being present, (see Section II-D) then no distinct offset will occur. This will be true for any program where independent work skills are being taught. (N.B.: Reinforcement of on-task behavior during the intertrial interval i s optional and will not be evaluated.) D. The events of the intertrial interval should follow in this sequence: 1. Remove materials 2. Records data 3. Reinforcement of on-task behavior (optional) 4. Pauses to give t r i a l a distince offset. DL:jaor 6.8.80 APPENDIX F PROVINCIAL PI-SERVICE RESOURCE TEAM WORKSHOP TRAINEE FEEDBACK SHEET We would like to have some measure of our effectiveness. Please rate your responses to the following statements with numbers 1 to 5. 5=— Highest, excellent, strongly agree. 4 = High, good, mildly agree. 3 = Average, O.K. 2 = Low, not so good, mildly disagree. 1 = Lowest, really bad, strongly disagree. N.S. = Not sure. TRAINEE FEEDBACK SHEET 1. The preparation for the workshop was good. 2. The organization of the workshop was good. 3. The content of the workshop met my expectations. 4. The first day's introduction helped me understand what P.I.R.T. was, and what to expect. 5 . I liked and learned from the following presentations: a. why D..T.F.? b. Observation Interpretation. c. Behavioral Principles. d. D.T.F. Components. e. Pre-requisite Skills. f. Baselining and Assessment. g. Programme writing lecture. h. D.T.F. vs Free Operant. i . Prompts, Probes, Branches, Maintenance, Generalization and Criteria. j . Communication Issues and Language Teaching. k. Techniques of Behavior Management. 1. Task Analysis. m. Curriculum. n. Token Economy. 6. I liked and learned from the films. 7. I liked and learned from the following components of "hands on" training: a. Demonstrations (watching others). b. Role playing (practicing with adults). c. Hands on (practicing with a child). d. Video-tape feedback. e. Small group planning. 8. I thought the homework and feedback was relevant to my needs. 9. I had many opportunities to ask questions and talk about the workshop topics. 10. Please comment on at least three of the above statements. For example, why you liked or didn't like a presentation. (Please use back of page i f space is not sufficient.) 137 APPENDIX G 138 Inservice Resource Teams Name: FOLLOW-UP FEEDBACK #1 As this is our final follow-up, we would like to gather some data regarding our effectiveness at helping you apply the information and techniques discussed during the workshop and follow-ups to your day-to-day work situation. Please note that any questions asking for a quantity such as how often you have done a specific task should be fi l l e d in with a specific number. For several of the questions, i t may be helpful to consult your records. Please make sure that you have signed your name to a l l the sheets in this form. In advance, we would like to thank you for giving the time and energy required to complete this form. Your effort is appreciated. W.I.R.T./P.I.R.T. 139 Inservice Resource Team FOLLOW-UP FEEDBACK #1 1. Are you using DTF Programme Plans similar to those taught at the workshop? Yes No ______ la. If yes, how many programmes are you using? , lb. If no, why not? 2. Are you using data collection techniques learned during the workshop? Yes No 2a. If yes, how many behaviors are you keeping data on? 2b. If no, why not? . , 3. Are you using a Discrete Trial Format for any of your teaching? Yes No 3a. If yes, for how many tasks are you using i t for? 3b. If no, why not? 4. Have you written a DTF Programme Plan such as those done during the workshop within the last 6 months? Yes No 4a. If yes, how many have you written? 4b. If no, why not? 140 Follow-up Feedback #1 Page 2 5. Do you baseline a c h i l d before teaching a new behavior? Yes No 5a. Do you baseline a c h i l d before you start, a behavior management programme? Yes No 6. Are you using an assessment technique that requires you to take objective data of the behaviors being assessed? Yes No 7. I f you were to do the tasks l i s t e d below, how confident would you be that you could now do them independently? rH -P -p cd C •p -P de an de co •H -P •rl •H • <+H -P CO G o o CD • O G o si o O 1 2 3 4 -p c cu xi •ri S cu Teaching using DTF at an 80$ 2 5 correct c r i t e r i o n . b. Writing a DTF Programme Plan 1 2 3 U 5 at an 80$ correct c r i t e r i o n . c. Keep data and graph i t . 1 2 3 4 5 d. Use a token economy. 1 2 3 4 5 e. Bui l d an Individual Student 1 2 3 4 5 Curriculum. f. Use the Behavior Management 1 2 3 4 5 Techniques taught i n the workshop. g. Teach someone else to use DTF 1 2 3 4 5 h. Teach someone else to use behavior management techniques. Follow-up Feedback #1 Page 3 Think back to before the workshop* Mow do question 7 again. This time rate how confident you were before the workshop at doing these tasks independently. -P a CD -J H -H i—I -P +J <+H rt G P G G ( D C CD O -p T3 3 "d o rt -rl -P -H -P C cn fed C !H O O CD • O CD G o X! O O !> a. Teaching using DTF at an 80% 1 2 3 4 correct criterion. b. Writing a DTF Programme Plan 1 2 3 4 at an 80% correct criterion. c. Keep data and graph i t . 1 d. Use a token economy. 1 e. Build an Individual Student 1 Curriculum. f. Use the Behavior Management 1 2 3 4 5 Techniques taught in the work-shop. g. Teach someone else to use DTF. 1 2 3 4 5 h. Teach someone else to use 1 2 3 4 5 behavior management tech-niques. 2 3 4 5 2 3 4 5 2 3 4 5 12,2 Inservice Resource Teams Name: FOLLOW-UP FEEDBACK #2 As this is our final follow-up, we would like to gather some data regarding our effectiveness at helping you apply the information and techniques discussed during the workshop and follow-ups to your day-to-day work situation. For those questions asking for a rating of 1-5, please circle the appropriate number. Please made sure that you have signed a l l the sheets in this form. A l l comments will be appreciated. Thank you for your cc—operation. W.I.R.T./P.I.R.T 143 Follow-up Feedback #2 Page 1 1. How useful do you feel the content of the workshop has been for you over the last six months. No help Very at a l l helpful 1 2 3 4. 5 2. How much more did the information you recieved during the follow-up visits add to that of the workshop? No help Very at a l l helpful 1 2 3 4 5 3. How effective were we in helping you integrate our input into practical use in your teaching situations? Not effective Very at a l l effective 1 2 3 4 5 4. How much would you attribute any progress of the children to input which we have given to you? Hindered Progress Facilitated progress the same progress 1 2 3 4 5 5. Were there any children you think our input was helpful for? Yes No Please explain. 6. Were there any children you felt our input was a hindrance to? Yes No Please explain. 144 Follow-up Feedback #2 Page 2 How useful to you would i t be to have P.I.R.T./W.I.R.T. come back on a consulting basis every 6-8 months. No use at a l l Very useful 1 2 3 4 5 Do you have any further general comments about our follow-up visits? Do you have any further comments about the team in general? Ik 5 APPENDIX H 1Z+6 Reliability-Percentage of scores checked and percentage agreement per test period. Terminology Measure Percent of scores checked Mean percent agreement Range Pre 35 93.9 89-100 Mid 35 99.1 94-100 Post 48 97.2 94-100 Percent of scores checked Mean percent agreement Range Pre 33 90.2 DTF Measure Mid 26 92.1 80-100 92-100 Post 36 97.00 83-100 Program Writing Measure Percent of scores checked Mean percent agreement Mid 21 86 Post 28 89 Range 79-95 77-95 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            data-media="{[{embed.selectedMedia}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0095832/manifest

Comment

Related Items