Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Student ratings of teaching effectiveness in university personnel decisions Brigden, Susan Rae 1989

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-UBC_1989_A8 B74.pdf [ 8.14MB ]
Metadata
JSON: 831-1.0097732.json
JSON-LD: 831-1.0097732-ld.json
RDF/XML (Pretty): 831-1.0097732-rdf.xml
RDF/JSON: 831-1.0097732-rdf.json
Turtle: 831-1.0097732-turtle.txt
N-Triples: 831-1.0097732-rdf-ntriples.txt
Original Record: 831-1.0097732-source.json
Full Text
831-1.0097732-fulltext.txt
Citation
831-1.0097732.ris

Full Text

STUDENT RATINGS OF TEACHING EFFECTIVENESS IN UNIVERSITY PERSONNEL DECISIONS by SUSAN RAE BRIGDEN B.Sc. , U n i v e r s i t y of B r i t i s h Columbia, 1973 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS i n THE FACULTY OF GRADUATE STUDIES THE DEPARTMENT OF EDUCATIONAL PSYCHOLOGY AND SPECIAL EDUCATION Faculty of Education We accept t h i s thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA October 1989 © Susan Rae Brigden, 1989 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. E d u c a t i o n a l P s y c h o l o g y and Department of S p e c i a l E d u c a t i o n The University of British Columbia Vancouver, Canada Date /g-r^T /S*^ J<?2? DE-6 (2/88) ABSTRACT Student ratings of teaching effectiveness are often considered by u n i v e r s i t y personnel decision-makers when a candidate's teaching effectiveness i s being evaluated. In turn, teaching effectiveness i s often one of several factors considered by these decision-makers when evaluating o v e r a l l f a c u l t y performance f o r the purpose of making personnel decisions with respect to reappointment, promotion, and tenure. Although student ratings has proven to'be a popular f i e l d of research, a review of the l i t e r a t u r e indicated that l i t t l e empirical research has been conducted into student ratings and t h e i r place i n u n i v e r s i t y personnel decisions. The purpose of the study was to explore the ro l e of student ratings i n u n i v e r s i t y personnel decisions. Two versions of a questionnaire were developed and d i s t r i b u t e d to 135 members of the Faculty of Education at the Un i v e r s i t y of B r i t i s h Columbia. One version was sent to 76 Faculty members c l a s s i f i e d , Decision-makers/Administrators (DMAs); a second version was sent to 59 other Faculty members c l a s s i f i e d as Instructors. Forty-seven DMAs and 52 Instructors responded f o r an o v e r a l l response rate of 73%. The r e s u l t s indicate that research i s of primary importance i n a candidate's o v e r a l l performance evaluation and that the r e l a t i v e importance of classroom teaching i n o v e r a l l performance evaluation appears to depend upon the type of appointment a candidate i s being considered f o r . For appointments with tenure, the importance of classroom teaching appears to decrease as the rank of the appointment increases. i i The most important source of information considered in teaching effectiveness evaluation appears to be formal peer reviews, followed by student ratings and, then, the opinions of outside experts; the relative importance of each of these information sources does not appear to vary according to the type of appointment a candidate is being considered for. Limitations of the study are discussed and questions designed to guide future research are presented. i i i TABLE OF CONTENTS Abstract i i Table of Contents i v L i s t of Tables.... v i i L i s t of Figures i x Acknowledgement x CHAPTER I. INTRODUCTION 1 Context of the Study 2 The U n i v e r s i t y Academic Senate Decision of 1978 2 Student Ratings i n the Faculty of Education 3 History 3 Po l i c y 6 Purpose f o r Obtaining Student Ratings 6 Student Rating Forms 7 C o l l e c t i o n Procedures 8 Data Analyses 9 Reporting the Results of Student Ratings 10 Background of the Study 10 Purpose of the Study 11 D e f i n i t i o n of Terms 12 CHAPTER II. RELATED LITERATURE 15 Student Ratings: A Popular F i e l d of Research 15 Student Ratings i n Personnel Decisions: Their Evolution 17 Teaching Effectiveness: A Factor i n Personnel Decisions 22 Research Questions 25 CHAPTER I I I . METHODOLOGY 27 Research Design 27 Description of the Sample 27 Instrumentation 28 The DMA Survey Instrument 31 The Instructor Survey Instrument 33 Data Management 34 Data Analysis 35 i v CHAPTER IV. RESULTS 38 Rate of Response 38 Description of the Respondents 41 Departmental A f f i l i a t i o n 41 Ranks of Respondents 45 Experience with Personnel Decisions 46 Length of Service of Instructors 50 Respondents' Perceptions of Personnel Decisions 51 Assistant Professors Without Tenure 53 Factors Considered i n Overall Performance Evaluation 53 Sources of Information i n Teaching Effectiveness Evaluation 57 Associate Professors Without Tenure 60 Factors Considered i n Overall Performance Evaluation 60 Sources of Information i n Teaching Effectiveness Evaluation 63 Professors Without Tenure 67 Factors Considered i n Overall Performance Evaluation 67 Sources of Information i n Teaching Effectiveness Evaluation 70 Appointments With Tenure 73 Factors Considered i n Overa l l Performance Evaluation 74 Sources of Information i n Teaching Effectiveness Evaluation 78 Comparison of the Relative Importance of Factors 81 Comparison of the Relative Importance of the Sources of Information 83 Respondents' Perceptions of the Purposes of Student Ratings 86 Respondents' S a t i s f a c t i o n With Student Ratings 89 Respondents' Ratings of Weight Statements 90 Respondents' Comments . . 91 CHAPTER V. DISCUSSION AND CONCLUSIONS 95 Summary of the Findings 95 Directions f o r Future Study 99 Limitations of the Study 102 REFERENCES 105 APPENDIX A I l l APPENDIX B 124 APPENDIX C 139 APPENDIX D 150 APPENDIX E 153 APPENDIX F 156 V APPENDIX G 158 V i LIST OF TABLES Table 1: Comparison of Respondents with Non-Respondents by Group 40 Table 2: Frequency of Total Respondents by Group and Department/School 42 Table 3: Population and Sample Size of DMAs by Department 43 Table 4: Population and Sample Size of Instructors by Department.... 44 Table 5: Frequency of Respondents by Group and Rank 45 Table 6: Frequency of DMAs by Committee or Administrative Capacity i n the Survey Sample. 47 Table 7: Frequency of DMAs by Personnel Committee i n the Survey Population 47 Table 8: Years Experience of DMAs on Current Personnel Committee.... 48 Table 9: Frequency of DMAs Who Have P a r t i c i p a t e d i n Other Personnel Dec i s ion-making Capacities 49 Table 10: Frequency of Instructors Who Had Previously P a r t i c i p a t e d i n Personnel Dec i s ion-making Capacities 49 Table 11: Number of Respondents Who Had Pa r t i c i p a t e d i n Personnel Decisions i n Other Capacities by Group 50 Table 12: Length of Service of Instructors i n the Faculty of Education 51 Table 13: T-test Results f o r Factors Considered i n Appointments to Assistant Professor Without Tenure 55 Table 14: Mean Response Ratings f o r Sources of Information Considered i n Appointments to Assistant Professor Without Tenure 58 Table 15: T-test Results f or Factors Considered i n Appointments to Associate Professor Without Tenure 61 Table 16: T-test Results f o r Sources of Information Considered i n Appointments to Associate Professor Without Tenure 65 Table 17: Mean Response Ratings f o r Factors Considered i n Appointments to Professor Without Tenure 68 v i i Table 18: T-test Results f o r Sources of Information Considered i n Appointments to Professor Without Tenure 71 Table 19: Ranks at Which DMAs Responses Concerning Appointments With Tenure are Most Valid/Applicable 74 Table 20: T-test Results f o r Factors Considered i n Appointments With Tenure 76 Table 21: T-test Results f o r Sources of Information Considered f o r Appointments With Tenure 79 Table 22: Ranking of Factors Considered i n Evaluating Overall Performance 82 Table 23: Spearman's Rank Co r r e l a t i o n C o e f f i c i e n t s f o r DMAs' and Instructors' Ranks of the Relative Importance of Factors... 83 Table 24: Ranking of Sources of Information Considered i n Evaluating Teaching Effectiveness 84 Table 25: Spearman's Rank C o r r e l a t i o n C o e f f i c i e n t s f o r DMAs' and Instructors' Ranks of the Relative Importance of Sources of Information 86 Table 26: Mean Response Ratings for Purposes of Student Ratings of Teaching Performance 87 Table 27: T-test Results f o r Value of Student Ratings 90 Table 28: Mean Responses to Weighting Questions 91 Table 29: Frequency of Respondents' Comments by Category 92 v i i i LIST OF FIGURES Figure 1: Population and sample size by group 39 Figure 2: D i s t r i b u t i o n of respondents by department 41 Figure 3: Factors considered i n appointments to Assistant Professor Without Tenure ranked by DMAs' mean response 56 Figure 4: Sources of information considered i n Appointments to Assistant Professor Without Tenure ranked by DMAs' mean response 59 Figure 5: Factors considered i n appointments to Associate Professor Without Tenure ranked by DMAs' mean response 62 Figure 6: Sources of information considered i n appointments to Associate Professor Without Tenure ranked by DMAs' mean response 66 Figure 7: Factors considered i n appointments to Professor Without Tenure ranked by DMAs' mean response 69 Figure 8: Sources of Information Considered i n appointments to Professor Without Tenure ranked by DMAs' mean response 72 Figure 9: Factors considered i n Appointments With Tenure ranked by DMAs' mean response 77 Figure 10: Sources of information considered i n Appointments With Tenure ranked by DMAs' mean response 80 Figure 11: Purposes of student ratings ranked by DMAs' mean response 88 i x ACKNOWLEDGEMENT I would l i k e to express my appreciation to my thesis committee Dr. Robert Conry, Dr. David Bateson, and Dr. Walter Boldt f o r the d i r e c t i o n and encouragement they have given to me these past few years. Many thanks go to the members of the Faculty of Education of the Uni v e r s i t y of B r i t i s h Columbia who p a r t i c i p a t e d i n the study and without whom I could not have conducted t h i s research. I must also express my gratitude to my husband, Stuart, to my sons, Jordan and Matthew, and to a l l my fr i e n d s , e s p e c i a l l y Diane MacLean, f o r the support they have given me and the s a c r i f i c e s they have made i n order that I might complete my research. X CHAPTER I. INTRODUCTION Near the end of the course, the i n s t r u c t o r asks f o r a volunteer to administer and c o l l e c t the Student Evaluation of Teaching Effectiveness ratings forms. The i n s t r u c t o r leaves the classroom and the volunteer reads out a set of in s t r u c t i o n s to the other students, and then everyone takes a few minutes to complete the questionnaire; the students are asked to rate a v a r i e t y of i n s t r u c t o r behaviours as well as to rate the course as a whole. The completed forms are then returned to the student volunteer, who seals them i n an envelope and returns i t to the appropriate place f o r analysis. This scene i s played out across campuses throughout North America every year. Seldin (1984) reports that "almost 70% of [ l i b e r a l arts] colleges used student r a t i n g forms i n 1983" (p. 54). Several other researchers (Aleamoni, 1987a; Murray, 1979; Rippey, 1975) have also concluded that student ratings are i n widespread use at post-secondary i n s t i t u t i o n s . Yet many people who have completed these r a t i n g forms have l i t t l e or no idea of the reasons f o r student evaluations of teaching effectiveness and how the r e s u l t s of t h e i r student ratings are u t i l i z e d . This study i s an exploratory study designed to describe the importance of student ratings i n the personnel decision-making process as i t occurs within the Faculty of Education at the Univ e r s i t y of B r i t i s h Columbia. 1 CONTEXT OF THE STUDY The University Academic Senate Decision of 1978 In March of 1978, the U n i v e r s i t y of B r i t i s h Columbia (U.B.C) Senate adopted the following recommendations with respect to student evaluations of teaching: (a) That annual systematic, objective and cumulative evaluations should be obtained f o r a l l f a c u l t y members and i n s t r u c t o r s , and f o r a l l undergraduate courses f o r which t h i s i s p r a c t i c a b l e . (b) That such evaluation include teaching evaluation, and that teaching evaluation be considered i n reappointment, promotion and tenure decisions. (c) That the timing and form of such evaluation be decided by each f a c u l t y . (d) That the evaluation instruments be developed and administered by the f a c u l t i e s and/or departments concerned. (University of B r i t i s h Columbia Senate Records, 1978, p. 6999) Therefore, since 1978, i t has been the o f f i c i a l i n t e n t i o n of the U.B.C Senate that teaching evaluations by students, also known as student ratings of teaching effectiveness or teaching performance, be considered i n personnel decisions. 2 Student Ratings i n the Faculty of Education In the Faculty of Education the Teacher Evaluation Service (TES) oversees the c o l l e c t i o n of student ratings of teaching effectiveness. The coordinator of TES i s Dr. Robert Conry. The information pertaining to the hist o r y , p o l i c y and procedures of the TES, presented i n the following sections, was obtained as a r e s u l t of an interview with Dr. Conry conducted i n the summer of 1989 ; a l l quotations are taken from t h i s interview. History Student ratings of teaching effectiveness were c o l l e c t e d i n the Faculty of Education even before the 1978 Senate decision; some ins t r u c t o r s i n the Faculty of Education were c o l l e c t i n g t h e i r own ratings on a l i m i t e d basis. I n i t i a l l y , these i n s t r u c t o r s simply obtained "student reactions to t h e i r teaching on t h e i r own because they believed i t was a good idea, and then small groups of those people began to share t h e i r procedures f o r doing that. Eventually there came to be i n t e r e s t at the departmental l e v e l i n a few departments and then at the f a c u l t y l e v e l : but there was no mandated routine". The f i r s t student ratings were c o l l e c t e d f o r use i n professional development rather than f o r helping with personnel decisions. They were gathered because people wanted the information "... i n order to know how they were doing, and to use that knowledge i n whatever way they wished to change and possibly improve t h e i r teaching". 3 In the early seventies, an ad hoc committee constructed a student ratings form which could be used by i n s t r u c t o r s on a s t r i c t l y voluntary basis. A modest o f f i c e f o r coordinating the c o l l e c t i o n of student ratings existed at that time. By the mid-seventies the administration saw a need to c o l l e c t that would a i d i t i n the making of decisions concerning retention, promotion and tenure. This need had come about due to the f a c t that a large number of people were coming up f o r tenure each year, and that " i t was d i f f i c u l t to exercise any kind of common standards when i t came to t h e i r teaching". A committee c a l l e d the Standing Committee for the Evaluation of Teaching (SCET) was formed c i r c a 1974. This committee acted "not so much as an overseeing committee as a helping committee" and was responsible f o r developing an instrument which had students complete questions on d i f f e r e n t facets of an i n s t r u c t o r ' s teaching. This instrument had "15 or 20 items that students f i l l e d out about d i f f e r e n t facets of teaching. It also had a number of items that were c a l l e d 'university obligations'; they didn't have so much to do with teaching effectiveness as with whether the i n s t r u c t o r came to class on time, or cancelled classes without reason. The concern was whether people were l i v i n g up to t h e i r contractual o b l i g a t i o n s . " In 1976, a new committee was struck to design the questionnaire which, except for a few minor modifications, i s s t i l l being used today. P r i o r to the Senate d e c i s i o n of 1978, the use of t h i s questionnaire was, by and large, voluntary; however, f a c u l t y "were required to use i t i f they were being considered f o r promotion or tenure". In January of 1978, Dr. Robert Conry became the sole coordinator of the SCET o f f i c e . In March 1978, the Senate decided that, where pract i c a b l e , student ratings should be obtained f o r a l l f a c u l t y members 4 and i n s t r u c t o r s i n a l l undergraduate courses. As a r e s u l t , the student ratings which had already been c o l l e c t e d i n the Faculty of Education on a voluntary basis now became required f o r most courses. The Dean's O f f i c e of the Faculty of Education proposed that the Senate mandate be modified by including graduate courses and by s e t t i n g the minimum number of students i n a clas s to 5, before student ratings were required. This modification was needed because many Faculty members had a large proportion of graduate l e v e l courses i n t h e i r teaching load and because the average class size of these graduate courses tended to be small. The Faculty voted on t h i s p o l i c y which "... was approved by a vote from the f a c u l t y at large, so i t was a Faculty-wide de c i s i o n " . The spring of 1979 was the f i r s t time across-the-board student ratings of teaching effectiveness were c o l l e c t e d i n the Faculty of Education. The administration of student ratings forms and the analyses of the data were overseen by what was then known as the SCET o f f i c e . Eventually SCET ceased to function as a committee and, i n 1986, -the name of the SCET o f f i c e was changed to the Teacher Evaluation Service (TES), however, i t s role i n the c o l l e c t i o n of student ratings d i d not change. It should be noted that, u n t i l 1988, Physical Education and Recreation (PHED) had i t s own f a c u l t y and was not part of the Faculty of Education. At that time, PHED administered i t s own student ratings forms; i t i s only since PHED has become a school within the Faculty of Education, that the evaluation of PHED in s t r u c t o r s has been under the auspices of TES. 5 P o l i c y The p o l i c y of the Faculty of Education pertaining to student ratings r e f l e c t s the p o l i c y mandated by the Senate i n 1978. The Faculty p o l i c y has been constant over the 10 years since i t was f i r s t adopted by the members of the Faculty i n 1979: student ratings are obtained i n a l l courses with 5 or more students every time the course i s taught. However, at the time the p o l i c y was adopted, summer session courses were exempted and so, f o r both spring and summer session courses i t has been "a l i t t l e b i t ambiguous as to what the i n s t r u c t o r i s obligated to do". Currently, though, student ratings are c o l l e c t e d i n most spring and summer session courses. There are, however, some courses, such as doctoral seminars, where the student r a t i n g form i s not generally used because the questions are i r r e l e v a n t and cannot be f a i r l y answered. The procedures used to c o l l e c t student ratings have been reviewed twice over the years. However, there have not been any changes made to the Faculty p o l i c y pertaining to student ratings since i t was f i r s t adopted. Purpose for Obtaining Student Ratings When i n d i v i d u a l i n s t r u c t o r s f i r s t started to c o l l e c t student ratings, t h e i r main purpose was to provide feedback f o r professional development. Throughout the mid-seventies the purpose of student ratings began to s h i f t as the information they provided was used more often i n personnel decisions. With the adoption, by the Senate, of the "recommendation 6 That such evaluation include teaching evaluation, and that teaching evaluation be considered i n reappointment, promotion and tenure decisions. (University of B r i t i s h Columbia Senate Records, 1978, p. 6999) the major purpose of student ratings became that of providing information to administrators to use i n personnel decisions. Student Rating Forms The 'common' form used to c o l l e c t student ratings was f i r s t developed f o r the 1976-77 school year. Over the years, there has been l i t t l e change i n the items i t contains; some of the items have been reworded to eliminate ambiguities and there has been some change i n the nature of the demographic data that i s c o l l e c t e d . This 'common' form i s used i n approximately 85% of the courses taught i n the Faculty. A t o t a l of f i v e d i f f e r e n t forms (Appendix A) are now used to c o l l e c t student ratings of teaching effectiveness. The most commonly used form has thirty-one items on i t . One global item asks the student to give an ov e r a l l r a t i n g of the i n s t r u c t o r . The remaining t h i r t y items are d i s t r i b u t e d over four d i f f e r e n t facets of teaching: 1) course planning and organization; 2) i n s t r u c t i o n a l technique and s t y l e , 3) use of teaching material and evaluation; and 4) student learning outcomes. At one time, only one form was used throughout the Faculty, however, i n the early 1980's, the Department of Visual and Performing Arts i n Education (VPAE) "decided that the phrasing of the questions was not quite r i g h t f or the nature of t h e i r courses" and so a d i f f e r e n t . v e r s i o n was constructed containing the same number of items and pertaining to the same facets, but written s p e c i f i c a l l y with VPAE i n mind. 7 More recently, three new forms have been constructed f o r use i n the new Teacher Education Program (TEP) courses; they were introduced i n the F a l l of 1987. The items on the forms correspond to three d i f f e r e n t aspects of the TEP; the 'course-wide' form pertains to such things as the design and organization of the course; the ' i n s t r u c t o r - s p e c i f i c ' form pertains to the "effectiveness and s t y l i s t i c c h a r a c t e r i s t i c s of the i n s t r u c t o r " ; and the 'laboratory' form pertains to the labs the students have. Instructors also have the a l t e r n a t i v e of constructing and using t h e i r own evaluation instrument. However, i n order to do so, the instrument must be approved by the Head of t h e i r department. Dr. Conry knows of only one professor that has constructed an alternate instrument that i s used i n place of the commonly used forms. C o l l e c t i o n Procedures At the beginning of each term the department secretaries send a l i s t of a l l the courses and the names of the i n s t r u c t o r s to the TES o f f i c e . An i n d i v i d u a l record of the courses taught i s made f o r each i n s t r u c t o r . The TES o f f i c e sends a copy of t h i s record to each i n s t r u c t o r f or v e r i f i c a t i o n . The i n s t r u c t o r i s also asked to indicate the number of students registered i n each course and then to return the record to the TES o f f i c e . During the l a s t quarter of the course, the TES o f f i c e sends a package to each i n s t r u c t o r with the appropriate number of student r a t i n g forms. According to Dr. Conry, student ratings are c o l l e c t e d during the l a s t quarter of every course for two main reasons: 1) "some of the 8 questions are not pertinent u n t i l near the end of the course"; and 2) "the l i t e r a t u r e shows that data c o l l e c t e d early i n a course y i e l d s higher ratings than i f c o l l e c t e d l a t e r i n the course". On the day that the student ratings are to be c o l l e c t e d , the i n s t r u c t o r i s to appoint a student to administer the questionnaire. A f t e r the i n s t r u c t o r leaves the room, the appointed student i s then to read out the i n s t r u c t i o n s to the rest of the c l a s s , allow the class time to complete the forms, package up the completed forms, and then return them to the TES o f f i c e . Data Analyses The data from the completed questionnaires i s entered onto a disk f i l e on the mainframe computer. The f i r s t analyses are run f i r s t to check for data entry errors. Once i t has been determined that the data are entered c o r r e c t l y , they are analyzed. The r e s u l t s of the s t a t i s t i c a l analyses are then put into the summary report. Each year, a c l u s t e r analysis i s done to generate the norm p r o f i l e s reported i n the graphical summary (Appendix A). They are then drawn on the i n t e r p r e t i v e graph to provide a normative framework to i n s t r u c t o r s f o r in t e r p r e t i n g t h e i r i n d i v i d u a l r e s u l t s . In addition, f a c t o r analyses and, on occasion, regression analyses are conducted mainly as checks on the system. To accommodate the differences between the common form and the TEP forms a separate average f o r the TEP i s computed. This average i s drawn onto the graphical summary of the summary reports sent to TEP i n s t r u c t o r s . 9 Reporting the Results of Student Ratings During the summer, each i n s t r u c t o r receives a summary report. Each report contains the following: 1) a graphical summary showing the norm p r o f i l e s and the r e s u l t s of the i n s t r u c t o r ' s student ratings; 2) a copy of the instrument; 3) a set of guidelines f o r i n t e r p r e t i n g the r e s u l t s ; 4) a page of breakdowns f o r each course that includes the mean and standard deviation f or each item along with the number of people who completed i t ; and 5) some background information about the students who completed the forms. Copies of the graphical summary f o r each i n s t r u c t o r are sent to the Department Head and the Dean. If a more d e t a i l e d summary i s required i t i s requested from the TES o f f i c e where a copy of every i n s t r u c t o r ' s evaluation i s kept f o r every year. BACKGROUND OF THE STUDY Student ratings are but one information source used by administrators when they are determining teaching effectiveness f or the purpose of f a c u l t y evaluation and personnel decisions. Seldin (1984) found that the l i b e r a l arts colleges he surveyed considered a number of d i f f e r e n t f a c t o r s , including classroom teaching, when evaluating the o v e r a l l performance evaluation of a candidate (p. 37). In turn, these colleges also u t i l i z e d a v a r i e t y of "sources [of information] i n evaluating f a c u l t y performance i n the classroom" (p. 51). The U n i v e r s i t y of B r i t i s h Columbia Faculty Handbook (1985) indicates that a v a r i e t y of sources of information may be considered during the 10 process of evaluating teaching at the Univ e r s i t y of B r i t i s h Columbia (U.B.C.). Section 4.02 of the Conditions of Appointment states: Evaluation of teaching s h a l l be based upon the effectiveness rather than the popularity of the in s t r u c t o r , as indicated by his command over subject matter, f a m i l i a r i t y with recent developments i n the f i e l d , preparedness, presentation, a c c e s s i b i l i t y to students and influence on the i n t e l l e c t u a l and s c h o l a r l y development of students. The methods of teaching evaluation may vary; they may include student opinion, assessment by colleagues of performance i n u n i v e r s i t y lectures, outside references concerning teaching at other i n s t i t u t i o n s , course materials and examinations, the c a l i b r e of supervised essays and theses, and other relevant considerations. When the opinions of students or of colleagues are sought, i t s h a l l be done through formal procedures. (p. C-6). One 'formal' way to obtain student opinions of teaching effectiveness i s through the use of. student ratings of teaching effectiveness. The academic units at U.B.C. have the option of u t i l i z i n g the r e s u l t s of student ratings f o r the purposes of making personnel decisions. It i s cle a r that the Faculty of Education i s one academic unit that exercises that option. What i s not clear , however, i s the importance of the r e s u l t s of student ratings i n personnel decisions. PURPOSE OF THE STUDY While there may be many factors involved and many sources of information considered during the process of personnel decision-making the main focus of t h i s study w i l l be on student ratings of teaching effectiveness. At the present time, the Faculty of Education at U.B.C. obtains student ratings of i t s i n s t r u c t o r s . In general, ratings may be used f o r the purposes of formative or summative evaluation of f a c u l t y members. 11 The purpose of t h i s study i s to explore the ro l e of student ratings i n the Faculty of Education when they are used f o r summative evaluation of teaching effectiveness. This study w i l l explore the role of student ratings i n personnel decisions; the importance assigned them as a source of information about teaching effectiveness, and, i n turn, the importance of teaching effectiveness i n o v e r a l l performance evaluation. The personnel decisions r e f e r r e d to i n t h i s study are those that pertain to reappointment, promotion and tenure, but not to merit pay or career progress increments. DEFINITION OF TERMS In t h i s study the following terminology w i l l be used: Personnel decisions r e f e r s to decisions made with regard to the retention, promotion, or the tenure of a f a c u l t y member. Teaching effectiveness (or performance) refe r s to the classroom teaching competence displayed by a f a c u l t y member. Overall ( f a c u l t y ) performance r e f e r s to a f a c u l t y member's performance i n a l l areas of his/her u n i v e r s i t y l i f e ; f a c t o r s considered i n the evaluation of a candidate's o v e r a l l f a c u l t y performance may include, among others, teaching effectiveness, research, and service. 12 Student ratings of teaching effectiveness (or performance) are systematic processes used to "gather information concerning student opinions, reactions, and observations about the course and i n s t r u c t o r " (Arreola, 1987b, p. 40). The r e s u l t s of student ratings are often considered as part of the o v e r a l l f a c u l t y evaluation process. Student ratings program refe r s to the systematic processes and practices u t i l i z e d to obtain student ratings of teaching performance. Teacher Evaluation Service (TES). formerly known as the Standing Committee fo r the Evaluation of Teaching (SCET), i s the o f f i c e which oversees the c o l l e c t i o n of student ratings i n the Faculty of Education Decision-makers/administrators (DMAs) are people i n the Faculty of Education who p a r t i c i p a t e i n personnel decision-making processes. Instructors. f o r purposes of t h i s study, ref e r s to those people i n the Faculty of Education having the rank of Associate Professor, Assistant Professor, or Professor who teach courses but do not p a r t i c i p a t e i n personnel decisions. The term ' i n s t r u c t o r ' (not c a p i t a l i z e d ) i s used when r e f e r r i n g to teachers i n general. Appointment With Tenure (Without Term or Tenured ) i s an appointment that cannot o r d i n a r i l y be terminated before normal retirement age. 13 Appointment Without Tenure (With Term or Untenured) f o r , purposes of t h i s study, ref e r s to a term appointment which implies that the appointee w i l l be reviewed and considered f o r further appointment before the end of the s p e c i f i e d term. Departmental Standing Personnel Committee (DSPC) i s a departmental committee con s i s t i n g of f a c u l t y who make recommendations pertaining to personnel decisions concerning members of t h e i r department. Faculty Standing Personnel Committee (FSPC) consists of representatives of the departments within the Faculty; i t s function i s to review and make recommendations pertaining to Faculty-wide personnel decisions. Senior Appointments Committee (SAC) i s a committee c o n s i s t i n g of Faculty representatives whose function i s to deal with university-wide personnel decisions. 14 CHAPTER II. RELATED LITERATURE Each year students i n the Faculty of Education at the Un i v e r s i t y of B r i t i s h Columbia are required to evaluate t h e i r professors by completing student ratings of teaching effectiveness forms. Anyone who has completed these questionnaires has, no doubt, wondered just what the purpose of student ratings i s and where student ratings f i t into the o v e r a l l picture of f a c u l t y evaluation as i t occurs at t h e i r u n i v e r s i t y . STUDENT RATINGS: A POPULAR FIELD OF RESEARCH Student ratings of teaching effectiveness has proven to be a very i n t e r e s t i n g f i e l d of research as evidenced by the large number of a r t i c l e s published. Ten years ago Murray (1979) observed that "most of the research on teaching evaluation i n higher education deals with student ratings, with well over 1,000 a r t i c l e s and books published on t h i s t o p i c " (p. 1). Interest i n student ratings has not waned since Murray made his observation. Marsh (1984) stated that "the study of students' evaluations has been one of the most frequently emphasized areas i n American educational research" (p. 708). The l i t e r a t u r e published on student ratings has focussed on a v a r i e t y of issues including: t h e i r r e l i a b i l i t y (Costin, Greenough, Menges, 1971; Marsh, 1984; Marsh, F l e i n e r , & Thomas, 1975); t h e i r v a l i d i t y (Cohen, 1981; Frey, Leonard, & Beatty, 1975; Howard, Conway, & Maxwell, 1985; McCallum, 1984; Marsh, 1982, 1984); class s i z e and student ratings (Crittenden, Norf, & L e B a i l l y , 1975; Feldman, 1984; Marsh, Overa l l , & 15 Kesler, 1979); teacher c h a r a c t e r i s t i c s and student ratings (Erdle, Murray, & Rushton, 1985; Feldman, 1986; Marsh, 1983; Meier & Feldhusen, 1979; Na f t u l i n , Ware, & Donnelly, 1973; Perry, Abrarai, & Leventhal, 1979; Ware & Williams, 1975; Williams & Ware, 1976, 1977); grading practices and student ratings (Abrami, Dickens, Perry, & Leventhal, 1980; Ducette & Kenney, 1982; Kennedy, 1975); student ratings and student achievement (Braskamp, Caulley, & Costin, 1979; Centra, 1977b; Mendelson, Canaday, & Hardin, 1978); f a c u l t y concerns with student ratings (Aleamoni, 1987a); and student ratings and personnel decisions (Centra, 1979; L i n , McKeachie, & Tucker, 1984; Murray, 1979; Seldin, 1984). The main purposes of student ratings, indicated i n the l i t e r a t u r e , are to provide information to a) administrators to a i d i n decisions concerning promotion/tenure and s a l a r i e s (summative evaluation) (Murray, 1979: Marsh, 1984; Stevens, 1987); b) in s t r u c t o r s f o r i n s t r u c t i o n a l improvement (formative evaluation) (Murray, 1979: Marsh, 1984; Stevens, 1987); and c) students to use during the process of course and i n s t r u c t o r s e l e c t i o n (Murray, 1979: Marsh, 1984; Stevens, 1987). In addi t i o n to the above, other purposes suggested i n the l i t e r a t u r e include: i for the " a l l o c a t i o n of teaching or f a c u l t y resources" ( G i l , 1987, p.57); f o r the "development of awareness, s e n s i t i v i t y , and appreciation of teaching" ( G i l , 1987, p.57); to "contribute to the understanding of the operation of the department or u n i v e r s i t y as a whole" (Stake, 1987, p. 1); and f o r research on teaching (Murray, 1979: G i l , 1987; Marsh, 1984). 16 Yet, even though the l i t e r a t u r e indicates that i t i s one of the main purposes f o r c o l l e c t i n g student ratings, Marsh (1984) observed that: researchers often indicate that students' evaluations are used as one basis for personnel decisions, but there i s a dearth of research on the p o l i c y practices that a c t u a l l y are employed i n the use of student ratings. (p. 748) A review of the l i t e r a t u r e confirms t h i s ; although a number of a r t i c l e s (Aleamoni, 1987a, 1987b; Arreola, 1987a; Centra, 1977a, 1979, 1987; Gillmore, 1984) address the issue of student ratings i n teaching evaluation, very l i t t l e empirical research has focussed on the importance of students ratings i n personnel decisions. Research has been conducted, however, on the following: an assessment of f a c u l t y evaluation practices i n post-secondary i n s t i t u t i o n s i n the southern United States (Moomaw, 1977); how the type of "information provided i n the evaluation of a promotion candidate's teaching a b i l i t y would a f f e c t the promotion dec i s i o n " (Salthouse, McKeachie, & Lin , 1978, p. 178); "the e f f e c t on promotion decisions of two d i f f e r e n t methods of presenting data from student evaluations" (Lin, McKeachie, & Tucker, 1984, p. 583); and the sources of information considered i n evaluating teaching effectiveness (Seldin, 1984). There i s a need f o r further research into student ratings and t h e i r place i n personnel decisions. STUDENT RATINGS IN PERSONNEL DECISIONS: THEIR EVOLUTION U.B.C. i s hardly the f i r s t u n i v e r s i t y to implement student ratings as a form of i n s t r u c t o r evaluation. Centra (1987) stated that "a few people, such as Remmers (1949) at Purdue and Guthrie (1954) at the Uni v e r s i t y of Washington, researched student ratings i n the t h i r t i e s and 17 f o r t i e s " and that "at that time, the ratings were pretty much on a voluntary basis" (p. 48). Rippey (1975) observed that "student r a t i n g forms are employed i n one form or another i n well over h a l f of the colleges and u n i v e r s i t i e s " (p. 951). Murray stated (1979) that "recent surveys indicated that approximately 68% of u n i v e r s i t i e s i n North America have some sort of student evaluation programme, and the percentage of u n i v e r s i t i e s using student ratings i s s t e a d i l y increasing" (p. 1). Concurring with Murray, Centra (1987) noted that " i n recent years, student ratings have become required at most i n s t i t u t i o n s " (p. 48). Yet what i s t h e i r history? How d i d student ratings come to be used as a method of teacher evaluation? According to Arreola (1987b) "a number of major student-rating systems had t h e i r beginnings as student government e f f o r t s " (p. 39) i n order to provide information to students f o r the purposes of course s e l e c t i o n or, as he put i t , "to provide students with a guide as to which fa c u l t y ' s courses to take and which to avoid" (p. 39). He suggested that student governments have not continued to play a major role i n i n s t r u c t o r evaluation programs. This i s due to a lack of p a r t i c i p a t i o n by i n s t r u c t o r s i n evaluation systems which, of necessity, could only rate those i n s t r u c t o r s who volunteered, and to the f a c t that "after a student-r a t i n g system had been established and had operated f o r a while, the student government would ei t h e r lose i n t e r e s t , run out of money, or both" (Arreola, 1987b, p. 40). Once student governments were either unable or u n w i l l i n g to continue administering student ratings, i t f e l l upon the u n i v e r s i t y administration to take over, thereby e s t a b l i s h i n g the "university's s o - c a l l e d student 18 evaluation programs" (Arreola, 1987b, p. 40). In some cases, therefore, student ratings of i n s t r u c t o r s came about as a r e s u l t of the actions of student governments. As control of student ratings s h i f t e d from student governments to u n i v e r s i t y administrative units, i t appears that the purposes of student ratings also s h i f t e d . Whereas, t h e i r i n i t i a l purpose was to provide information to students to a i d i n course s e l e c t i o n , student ratings began to provide formative information to i n s t r u c t o r s f o r professional development. More recently, the formative use of student ratings has often been superseded by the summative use "where they play an important role i n personnel decisions" (Centra, 1987, p. 48). A review of the U.B.C. Senate Records f o r 1974 indicate that the Ad Hoc Committee on Teaching Evaluation had found that student ratings were used by a number of Fa c u l t i e s and Schools, at U.B.C, as a method of evaluating teaching ( F i l e 6-3, p.6265). Four years l a t e r , i n 1978, the following i s recorded i n the Senate minutes: An inventory summary contained i n the appendix of the report of the Senate committee [Summer 1977] indicated the widespread use of teaching evaluation procedures at U.B.C. Nearly a l l f a c u l t i e s and departments used student questionnaires of some kind. Teaching and/or course evaluations, completed by students, were mandatory on an annual basis i n 12 out of 20 departments i n the Faculty of Arts, i n 5 out of 9 departments i n the Faculty of Science, and i n 5 out of the other 9 f a c u l t i e s . ... The committee believed that most f a c u l t y members agreed that regular evaluation was useful and desirable, and that student opinion could be obtained by questionnaires. ... Members of the committee were of the opinion that systematic, objective and cumulative student evaluations should be obtained f o r a l l teaching and a l l courses. (University of B r i t i s h Columbia Senate Records, 1978, F i l e 7-4, p. 6998) In March 1978, the U.B.C. Senate decided that student ratings were to be obtained f o r as many f a c u l t y members and in s t r u c t o r s as was practicable and that they were to be considered i n personnel decisions. 19 The minutes of May 18, 1983 meeting of the U.B.C. Senate record the following: Last November [November 1982], Student Senate Caucus conducted an informal survey of evaluation procedures. ... The survey showed that Senate guidelines were, generally, adhered to and that most F a c u l t i e s , Schools and Departments considered teaching evaluations to be a major consideration i n decisions on tenure, promotion and salary increment". (University of B r i t i s h Columbia Senate Records, 1983, F i l e 8-4, p.8029) The r e s u l t s of t h i s 'informal' survey were not included i n the Senate Records so the procedures used to conduct the survey cannot be examined. The minutes indicate that teaching evaluations were "a major consideration" i n personnel decisions yet there i s a need to know the r e l a t i v e importance of student ratings when compared to other sources of information used i n personnel decisions. Another issue that needs to be addressed i s whether or not the r e l a t i v e importance of student ratings varies depending upon the type of appointment a candidate i s being considered f o r . Therefore, i n order to get a better picture of the role of student ratings i n personnel decisions, the importance of student ratings i n d i f f e r e n t types of appointment decisions should be investigated. Student ratings are not the only method used to evaluate i n s t r u c t o r s . Ondrack & O l i v e r (1986) have i d e n t i f i e d several other methods of appraising i n s t r u c t o r s i n educational settings i n use across North America; supervisory appraisal, peer appraisal, s e l f - a p p r a i s a l , and appraisal by outsiders. However, Aleamoni (1987a) stated that "most of the i n s t r u c t i o n a l evaluation systems currently i n existence consist of only one component - namely students evaluations" (p. 25). Seldin's (1984) research showed that student ratings play a v i t a l r ole i n 20 i n s t r u c t i o n a l evaluation systems. He found that 67.5% of the 616 l i b e r a l arts colleges surveyed throughout the USA i n 1983 always used "systematic student r a t i n g s " to evaluate teaching performance (p. 45). However, his survey also indicated that while student ratings are a very important source of information, "the evaluation of departmental chair and dean" (Seldin, p. 44) are s t i l l the most frequently used sources of information considered i n evaluating teaching i n l i b e r a l arts colleges. The actual weight given to student ratings i n personnel decisions i s d i f f i c u l t to as c e r t a i n from Seldin's research because he rank ordered the survey r e s u l t s . Geis (1984) suggested that caution should be used when "equating ranks with weights" (p. 105). Ratings of teaching performance by students continue to play an important role i n teaching evaluation because they are generally considered to be r e l i a b l e and v a l i d sources of information which are r e l a t i v e l y free from the bias of extraneous variables, (Abrami, Perry, & Leventhal, 1982; Aleamoni, 1981; Cohen, 1981; Marsh, 1984). Murray (1979) argued that student ratings of teaching are s u f f i c i e n t l y r e l i a b l e and v a l i d to be used as one of several sources of information i n administrative decisions r e l a t i n g to f a c u l t y salary, tenure, and promotion, (p. 31) In addition, a number of researchers (Aleamoni, 1981; Aubrecht, 1984; Scriven, 1981) consider that, as consumers, students deserve to have input, and, that as "the people clo s e s t to the a c t i v i t y being assessed [they] are generally i n the best p o s i t i o n to supply relevant information" (Aubrecht, 1984, p. 89). Student ratings are the preferred method for c o l l e c t i n g t h i s information. 21 TEACHING EFFECTIVENESS: A FACTOR IN PERSONNEL DECISIONS Teaching i s only one of three components upon which f a c u l t y evaluation i s t y p i c a l l y based; Gillmore (1984) observed: Employment decisions regarding f a c u l t y within i n s t i t u t i o n s of higher learning are almost u n i v e r s a l l y based on the competence and d i l i g e n c e with which f a c u l t y perform i n three r e l a t e d r o l e s : research, teaching, and service to the i n s t i t u t i o n and to the wider community. The r e l a t i v e weights placed on these three r o l e s depends upon the i n s t i t u t i o n , and upon the d i s c i p l i n e within the i n s t i t u t i o n . Nonetheless, i n s t i t u t i o n s of higher learning almost uniformly attach high importance to the teaching r o l e , although often times that high value i s more i n the i d e a l than i n r e a l i t y . (p. 557) Seldin (1984) found that the deans of the l i b e r a l a r t s colleges he surveyed indicated that classroom teaching was "the single most important consideration i n evaluating f a c u l t y performance f o r purposes of tenure, promotion i n rank, and retention" (p. 73), yet he suggests that, i n r e a l i t y , f a c t o rs such as research and p u b l i c a t i o n a c t u a l l y carry a greater weight as f a r as personnel decisions are concerned. In h i s comments on Seldin's r e s u l t s , and the role that evaluation of teaching plays i n o v e r a l l f a c u l t y evaluation, Chickering (1984) wrote: i n our heart of hearts, those of us who have been close to the evaluation process know that, come promotion and tenure time i n most i n s t i t u t i o n s , f i r s t consideration i s given to research and p u b l i c a t i o n . If those c r i t e r i a are s a t i s f i e d , and i f there i s no s i g n i f i c a n t difference among candidates, then teaching effectiveness comes into play. F i n a l l y , a nod may be given to service contributions. (p. 92) The l i t e r a t u r e suggests, therefore, that even though.it i s often indicated that teaching effectiveness c a r r i e s equal weight with research when i t comes to i t s influence on personnel decisions, i n a c t u a l i t y i t s impact may not be as great and should be investigated further. 22 Seldin's survey was d i s t r i b u t e d to the deans of four year l i b e r a l a rts colleges. How would the r e s u l t s be d i f f e r e n t i f those responsible for f a c u l t y evaluation and personnel decisions at u n i v e r s i t i e s were surveyed? Perhaps a d i f f e r e n t emphasis would be put upon the factors considered i n o v e r a l l f a c u l t y evaluation, e s p e c i a l l y since teaching takes precedence over research at the college l e v e l and the reverse tends to the case at a un i v e r s i t y ? For example, Geis (1984) suggested that i t may be more important f o r u n i v e r s i t y f a c u l t y to display "public evidence of scholarship" (p. 105) than i t i s f o r college f a c u l t y . It would not be appropriate to assume that the factors considered important i n personnel decisions by the deans of l i b e r a l arts colleges are the same as those that are important to administrators of u n i v e r s i t i e s ; there i s a need to investigate what factors are considered important by u n i v e r s i t y administrators when a candidate's o v e r a l l performance i s being evaluated. In 1984, Lin , McKeachie and Tucker used a questionnaire to assess judges' "opinions about the r e l a t i v e weight that teaching a b i l i t y and research p r o d u c t i v i t y should be given at the Univ e r s i t y of Michigan and the r e l a t i v e weight of teaching a b i l i t y and research p r o d u c t i v i t y i n the judge's own decisions on promotion" (p. 587). The response format of the items allowed the respondent to indicate the weight given i n each s i t u a t i o n along a scale which ranged from 20% to 75%. The mean response was calculated and reported f o r each item. Their r e s u l t s indicated "that the majority considered research p r o d u c t i v i t y to have greater weight than teaching a b i l i t y i n both the u n i v e r s i t y c r i t e r i a f o r promotion and i n t h e i r own judgments" (p. 588). They found that l i t t l e weight was given to service i n promotion decisions. 23 It would appear that equal weight i s not accorded to the factors of teaching, research/scholarship, and service at U.B.C, for according to the U.B.C. Faculty Handbook (1985): Candidates f o r appointment, reappointment, appointment without term, or promotion. ... are judged p r i n c i p a l l y on performance i n both teaching and i n sc h o l a r l y a c t i v i t y . Service to the academic profession, to the University, and to the community w i l l be taken into account, but, while service to the Unive r s i t y and the community i s important, i t cannot compensate f o r d e f i c i e n c i e s i n teaching and s c h o l a r l y a c t i v i t y . (p. C-5). This passage indicates that teaching and research/scholarship play important roles i n personnel decisions at U.B.C. However, the r e l a t i v e importance given to these factors are not apparent from the Handbook and needs to be described. ' In summary, student ratings of teaching effectiveness are rou t i n e l y administered by u n i v e r s i t i e s and colleges. They help provide information to administrators f o r the evaluation of teaching effectiveness, to f a c u l t y f o r self-improvement, and to students f o r course s e l e c t i o n . Although student ratings provide information to administrators about teaching effectiveness, they are only one of several sources of information which may be considered during the process of teaching evaluation. In turn, teaching i s only one fa c t o r of f a c u l t y performance that i s considered when evaluating o v e r a l l f a c u l t y performance f o r the purpose of making personnel decisions. The factors a c t u a l l y considered by administrators of u n i v e r s i t i e s during the process of personnel decision-making need to be investigated further. At the same time, the contribution student ratings make to these decisions needs to be delineated. 2 4 RESEARCH QUESTIONS Survey research indicates that student ratings of teaching effectiveness play an important role i n the evaluation of teaching effectiveness of f a c u l t y members at some colleges and u n i v e r s i t i e s . In turn, teaching effectiveness i s one of several factors considered by college and u n i v e r s i t y administrators when evaluating o v e r a l l f a c u l t y performance f o r the purpose of making personnel decisions with respect to reappointment, promotion, and tenure. Yet a review of the l i t e r a t u r e indicates that the ro l e student ratings play i n u n i v e r s i t y personnel decisions needs to be investigated further. At U.B.C, the people responsible f o r making personnel decisions have the option of considering student ratings when evaluating a f a c u l t y member's teaching effectiveness. However, the role student ratings play i n personnel decisions at U.B.C. has not been c l e a r l y described. The purpose of t h i s i n v e s t i g a t i o n i s to explore the ro l e student ratings of teaching performance play i n personnel decisions as they occur i n one f a c u l t y at U.B.C. - the Faculty of Education. The research questions of t h i s study are: 1. a) What factors are considered important i n a candidate's o v e r a l l performance evaluation f o r the purposes of reappointment, promotion, and tenure by the people who p a r t i c i p a t e i n personnel decisions i n the Faculty of Education? 25 b) What i s the r e l a t i v e importance of teaching effectiveness among factors u t i l i z e d i n a candidate's o v e r a l l performance evaluation? a) What sources of information are considered important i n teaching effectiveness evaluation by those who p a r t i c i p a t e i n personnel decisions i n the Faculty of Education? b) What i s the r e l a t i v e importance of student ratings of teaching effectiveness among sources of information considered i n a candidate's teaching effectiveness evaluation? c) Does the importance of student ratings i n teaching effectiveness evaluation vary according to the type of appointment a candidate i s being considered for? How do Instructors' perceptions of the role of student ratings i n personnel decisions compare with the perceptions of those who p a r t i c i p a t e i n personnel decisions? What do DMAs and Instructors perceive to be the purposes of student ratings of teaching effectiveness? 26 0 CHAPTER III. METHODOLOGY RESEARCH DESIGN Survey research methodology was used to c o l l e c t the data; s e l f -administered survey questionnaires were d i s t r i b u t e d v i a campus mail to the subjects. A questionnaire was chosen as the preferred method f o r c o l l e c t i n g data as i t made i t possible to survey a l a r g e r number of subjects than would have been possible i f the researcher had conducted interviews. In addition, i t was f e l t that the use of anonymous questionnaires might increase the l i k e l i h o o d of response from those subjects who considered the research topic to be s e n s i t i v e . In order to c o l l e c t the data needed to describe the r o l e of student ratings of teaching effectiveness i n u n i v e r s i t y personnel decisions, the questionnaire was d i s t r i b u t e d to members of the Faculty of Education having the rank of Assistant, Associate, or F u l l Professor. DESCRIPTION OF THE SAMPLE The survey population f o r the study consisted of 135 members of the Faculty of Education; 76 subjects who w i l l be r e f e r r e d to as Decision-makers/Administrators (DMAs) and 59 subjects who w i l l be r e f e r r e d to as Instructors. Those f a c u l t y who were known to be on leave or extended vacation during the data c o l l e c t i o n period were not included i n the sample frame. 27 The DMAs were those Faculty members, i d e n t i f i e d by the Heads of Department and other Senior Administrators, who p a r t i c i p a t e d i n the personnel decision-making process within the Faculty. They were people who belonged to a Department Standing Personnel Committee (DSPC), the Faculty Standing Personnel Committee(FSPC), or the Senior Appointments Committee (SAC). The Instructors were those remaining f a c u l t y members who were not i d e n t i f i e d as belonging to one of the above mentioned committees, but who had attained the rank of Assistant, Associate, or F u l l Professor; those people within the Faculty who were c l a s s i f i e d as Sessional Lecturers, V i s i t i n g Lecturers, Teachers, V i s i t i n g Professors, or Adjunct Professors were not included i n the sample. INSTRUMENTATION It was necessary to develop a questionnaire f o r the study which would s p e c i f i c a l l y address the research questions, as no e x i s t i n g questionnaires were deemed sui t a b l e . Seldin's survey (Appendix, 1984) served as a model f o r the sections of the questionnaire pertaining to the factors considered i n o v e r a l l performance evaluation and the sources of information considered i n teacher effectiveness evaluation. Some items are s i m i l a r to those used by L i n et. a l . i n t h e i r 1984 research. The remaining items were developed as a r e s u l t of a review of the l i t e r a t u r e and through discussions with people experienced i n both personnel decisions within a u n i v e r s i t y s e t t i n g and i n survey research design. 28 The questionnaires were anonymous. An i d e n t i f i c a t i o n number written on each questionnaire was removed upon return of the completed questionnaire. The i d e n t i f i c a t i o n number was included f o r the purposes of record-keeping so as to avoid sending out ad d i t i o n a l questionnaires to those respondents who had already completed and returned the f i r s t one. Respondents were asked to r e f r a i n from putting t h e i r name anywhere on the questionnaire. Two versions of the questionnaire were developed; DMAs were sent the questionnaire A Survey of the Role of Student Ratings of Teaching  Performance (Appendix B) while Instructors were sent the questionnaire Instructor Perceptions of the Role of Student Ratings of Teaching  Performance (Appendix C). In large part the two questionnaires were i d e n t i c a l , however the Instructors' survey was shorter as i t required the respondent to only complete items concerning personnel decisions made about appointments to h i s or her own rank while the DMAs were required to complete items pertaining to personnel decisions concerning several d i f f e r e n t types of appointments. There were some differences i n the background information sections of the two versions. In addition, there were two questions pertaining to Instructors' perceptions of student ratings which were not included i n the DMAs' version of the questionnaire. The DMAs' questionnaire was 14 pages i n length and the one for Instructors was 10 pages long. A d i f f e r e n t covering l e t t e r was written by the researcher f o r each version of the questionnaire (Appendix D). The purpose of the research and questionnaire was introduced i n each covering l e t t e r . 29 One version of the questionnaire was sent to each subject, along with the appropriate covering l e t t e r i n early June of 1989. An addressed envelope was included and the completed questionnaires were returned v i a campus mail to the invest i g a t o r c/o the Teacher Evaluation O f f i c e . The Teacher Evaluation O f f i c e was chosen as the return address as there are sometimes d i f f i c u l t i e s sending mail to graduate students when the campus mail service i s used. The returned envelopes were put aside f o r the researcher to pick up. Two weeks a f t e r the questionnaires were f i r s t d i s t r i b u t e d a follow-up l e t t e r , along with an extra copy of the questionnaire and the i n i t i a l covering l e t t e r , was sent out to each subject from whom a completed questionnaire had not yet been received. Two versions of the follow-up l e t t e r (Appendix E) were written i n order to correspond to the two versions of the questionnaire. A f t e r another two weeks, a f i n a l l e t t e r reminding the subjects of questionnaire and the importance of t h e i r p a r t i c i p a t i o n i n the research project, was sent out (Appendix F). The questionnaire was constructed to meet the following c r i t e r i a : (1) to provide data f o r answering the research questions, (2) to flow i n a l o g i c a l sequence, and (3) to provide a format that would be appealing and easy f o r subjects to respond to. Each version of the r e s u l t i n g questionnaire had three sections. Descriptions of each version are following. 30 The DMA Survey Instrument The questionnaire d i s t r i b u t e d to the DMAs, A Survey of the Role of  Student Ratines of Teaching Performance (Appendix B), consisted of three sections each of which are described below.. Section I: This section consisted of a seri e s of s i x background questions. The respondents were asked to indicate t h e i r department, rank, the capacity i n which they p a r t i c i p a t e d i n personnel decisions at the time the research was conducted, the length of time they had p a r t i c i p a t e d i n t h i s capacity, whether or not they had ever p a r t i c i p a t e d i n personnel decisions i n another capacity, and, i f so, what that capacity was. Section I I : In t h i s section respondents were asked to indicate how s/he, t y p i c a l l y , considered (1) a number of d i f f e r e n t f a c tors when a candidate i s to be reviewed f o r promotion, reappointment or tenure, and (2) a v a r i e t y of sources of information that might be included i n the evaluation of a f a c u l t y member's teaching performance when s/he i s being reviewed f o r promotion, retention or tenure. The importance of each f a c t o r or source of information was indicated on a four-point Likert-type. This section of the questionnaire consisted of four subsections. Each subsection was on a d i f f e r e n t coloured page and dealt with only one p a r t i c u l a r type of personnel decision. The f i r s t subsection pertained to decisions concerning promotion to the rank of Assistant Professor Without Tenure, the second subsection pertained to decisions concerning promotion to the rank of Associate Professor Without Tenure , the t h i r d subsection 31 pertained to decisions concerning promotion to the rank of Professor Without Tenure, and the fourth section pertained to decisions concerning Appointments With Tenure. It was hoped that by l i m i t i n g each subsection to one s p e c i f i c type of personnel d e c i s i o n i t would make i t easier f o r the subject to complete these questions. It was decided that a l l Appointments With Tenure would be dealt with i n one subsection because i t was f e l t that the increased length of the questionnaire which would have been necessary to deal with each d i f f e r e n t rank might reduce the response rate. Instead, the respondent was asked to indicate the rank or ranks f o r which his/her responses to the questions pertaining to Appointments With Tenure were most applicable. Section I I I : The four subsections were as follows: (1) items pertaining to the perceived purposes f o r c o l l e c t i n g student ratings as indicated on a f i v e - p o i n t Likert-type scale; (2) items pertaining to the perceived values of student ratings as indicated on a f i v e - p o i n t Likert-type scale; (3) items pertaining to the perceptions of the respondents with regard to the weight given to student ratings and o v e r a l l performance evaluation as indicated on an eleven-interval scale; and (4) any other comments the respondent might have with respect to the ways i n which student ratings are used i n personnel decisions within the Faculty. 32 The Instructor Survey Instrument The questionnaire d i s t r i b u t e d to the Instructors, Instructor  Perceptions of Student Ratings of Teaching Performance (Appendix C), consisted of the three sections described below.. Section I: This section consisted of a series of f i v e background questions. The respondents were asked to indicate t h e i r department, rank, the length of service as a member of the Faculty of Education, whether or not they had ever p a r t i c i p a t e d i n personnel decisions, and, i f so, i n what capacity they had p a r t i c i p a t e d . Section I I : In t h i s section respondents were asked to indicate how s/he f e l t the people who make personnel decisions t y p i c a l l y consider (1) a number of factors when a candidate i s to be reviewed f o r promotion, reappointment or tenure, and (2) a v a r i e t y of sources of information that might be included i n the evaluation of a f a c u l t y member's teaching performance when s/he i s being reviewed f o r promotion, retention or tenure. Responses were given on a four-point Likert-type scale. This section of the questionnaire consisted of two subsections. Each subsection was on a d i f f e r e n t coloured page. The f i r s t subsection pertained to decisions concerning Appointments Without Tenure at the rank of the respondent while the second subsection pertained to decisions concerning Appointments With Tenure at the rank of the respondent. 33 Section I I I : The four subsections were as follows: (1) items pertaining to the perceived purposes f o r c o l l e c t i n g student ratings as indicated on a f i v e - p o i n t Likert-type scale; (2) items pertaining to the perceived values of student ratings as indicated on a f i v e - p o i n t Likert-type scale; (3) items pertaining to the perceptions of the respondents with regard to the weight given to student ratings and o v e r a l l performance evaluation as indicated on an eleven-interval scale; and (4) any other comments the respondent might have with respect to the ways i n which student ratings are used i n personnel decisions within the Faculty. I n i t i a l versions of the questionnaires were reviewed f o r content v a l i d i t y by people who were not only experts i n the f i e l d of student ratings and survey design but who also had f a m i l i a r i t y with the personnel decision-making process within the Faculty of Education. In addition, several people who had been employed as i n s t r u c t o r s i n the Faculty, reviewed the i n i t i a l versions of the questionnaires f o r c l a r i t y of i n s t r u c t i o n s and ease of use. The f i n a l version was reviewed f o r errors and omissions by one of the above-mentioned experts before the questionnaires were printed. The f i n a l versions of the questionnaire was not p i l o t e d due to time constraints. In addition, i t was f e l t that the sample frame would be severely reduced by u t i l i z i n g subjects f o r a p i l o t study. DATA MANAGEMENT When the completed surveys were received the ID number was removed, a l l information on the surveys was coded with a number, copied onto 34 f o r t r a n sheets, and then entered into data f i l e s . A l l data were reviewed and v e r i f i e d f o r accuracy. Instructors' responses were entered under the type of personnel d e c i s i o n pertaining to t h e i r rank. The written comments were grouped by s i m i l a r responses and then categorized. DATA ANALYSIS The SYSTAT (Wilkinson, 1988) s t a t i s t i c a l analysis program for personal computers was employed for data a n a l y s i s . Frequencies of a l l v a l i d responses were computed f o r the items included i n the background information section of the questionnaires, f o r the factors and the sources of information considered i n each of the four possible types of personnel decisions addressed i n the questionnaire, f o r the purposes of student ratings, f o r the value of student ratings and f o r the weights given to student ratings. Mean response ratings of the DMAs and of the Instructors were calculated f o r the factors and the sources of information considered i n each type of personnel d e c i s i o n addressed i n the questionnaire. Cases with missing data were deleted before the c a l c u l a t i o n s . Myers (1979) suggests that Hotelling's T 2 s t a t i s t i c can be used when i t i s of " i n t e r e s t to ask i f the two population centroids are the same" when there are two groups and scores obtained on several d i f f e r e n t scales f o r each subject (p. 470). Hence, Hotelling's T 2 s t a t i s t i c was used to test f o r a s i g n i f i c a n t multivariate difference between the two groups (DMAs and Instructors) and the way they rated the factors f o r each type of personnel decisions addressed i n the questionnaire. The alpha l e v e l was 35 set at .05. SYSTAT dropped a l l cases f o r which there was any missing data. A post hoc analysis using univariate t - t e s t s was used to i d e n t i f y sources of variance f o r any s i g n i f i c a n t differences indicated by the multivariate analysis of variance. The alpha l e v e l was again set at .05. Timm (1975) suggests that s e t t i n g the alpha l e v e l of the univariate tests at the same l e v e l as the o v e r a l l multivariate t e s t "leads to a smaller p r o b a b i l i t y of accepting a f a l s e n u l l hypothesis and hence to a smaller type II error " (p. 166). As t h i s i s an exploratory study, the decreased p r o b a b i l i t y of a type II error (the p r o b a b i l i t y of accepting a n u l l hypothesis that should be rejected) was chosen while accepting the s l i g h t increase i n the p r o b a b i l i t y of a making a type I error (the p r o b a b i l i t y of r e j e c t i n g a n u l l hypothesis when i t should have been an acceptable a l t e r n a t i v e ) when conducting the univ a r i a t e . Hotelling's T 2 s t a t i s t i c was also used to tes t f o r a s i g n i f i c a n t multivariate difference between the two groups and the way they rated the sources of information f o r each type of personnel decision. A post hoc analysis using univariate t - t e s t s was used to i d e n t i f y sources of variance for any s i g n i f i c a n t differences indicated by the multivariate analysis of variance. To estimate the r e l a t i v e importance of the fact o r s , to those who evaluate a candidate's o v e r a l l performance, the factors were rank ordered using the DMAs' mean response ratings. The sources of information were also rank ordered using the DMAs' mean response ratings to estimate t h e i r r e l a t i v e importance i n a candidate's teaching effectiveness evaluation. To i l l u s t r a t e the probable perceptions Instructors hold about the r e l a t i v e importance of the factors considered i n personnel decisions, they 36 were ranked according to the Instructors' mean response ratings. The ranking process was repeated f o r the Instructors' response ratings f o r the sources of information. Kirk (1978), states that the Spearman rank c o r r e l a t i o n c o e f f i c i e n t (Spearman's rho) " i s used to describe the degree of agreement between paired data that are i n the form of ranks"; hence, Spearman's rank c o r r e l a t i o n c o e f f i c i e n t was calculated to determine the degree of agreement between the DMAs and Instructors with respect to the r e l a t i v e importance of the fa c t o r s and the sources of information considered i n personnel decisions. Mean response ratings were calculated f o r the purpose, value, and weight statements, the statements were rank ordered based on the DMAs' mean response ratings, and Hotelling's T 2 analyses were calculated. Cases with missing data were dropped from the c a l c u l a t i o n s . 37 CHAPTER IV. RESULTS This chapter presents the r e s u l t s of the survey. The rate of response i s reported, followed by a d e s c r i p t i o n of the respondents and, the respondents' perceptions of a number of issues r e l a t i n g to student ratings and personnel decisions. RATE OF RESPONSE In June 1989, 135 questionnaires were d i s t r i b u t e d to members of the Faculty of Education; 76 questionnaires were sent to f a c u l t y members who pa r t i c i p a t e i n personnel decision-making processes, and 59 were sent to f a c u l t y members who do not p a r t i c i p a t e i n personnel decision-making processes, hereafter r e f e r r e d to as Instructors. Faculty members known to be on leave or away on vacation during the data c o l l e c t i o n period were not included i n the survey population. A t o t a l of 99 questionnaires were returned f o r an o v e r a l l response rate of 73%. This response rate includes the returns from the i n i t i a l and follow-up mailings. The response rate of 73% was judged to be s a t i s f a c t o r y considering the t y p i c a l response rate to mailed questionnaires and that the research topic may have been a sen s i t i v e subject f o r some people. The respondents were 52 DMAs and 47 Instructors. Of the research population, 68% of the DMAs responded while 80% of the Instructors responded (Figure 1). It would seem, therefore, that the Instructors are better represented i n t h i s sample because of the higher response rate, and that the sample of respondents might d i f f e r from the research population 38 i n that i t consists of a higher proportion of Instructors. Ferguson (1981) suggests that a "Chi-square may be used to test the representativeness of a sample where c e r t a i n population values are known" (p. 206). A Chi-square value was calculated; the obtained r e s u l t (Chi-square = 3.11, df =1) was not s i g n i f i c a n t , so on t h i s basis, i t was concluded that the proportion of respondents i n the sample does not d i f f e r s i g n i f i c a n t l y from that i n the population. Population = 135 DMAs = 76 Instructors = 59 DMAs = 52 Instructors = 47 (68%) (80%) Sample = 99 (73%) Figure 1: Population and sample size by group. It can be seen from Table 1 that of the non-respondents 67% were DMAs while 33% were Instructors. There may be a number of reasons why, o v e r a l l , a la r g e r proportion of Instructors responded than d i d DMAs. F i r s t l y , as the questionnaire explored the role of student ratings i n personnel decisions, the questions were more pertinent and, thereby, possibly more threatening to those people i d e n t i f i e d as DMAs than they were to the Instructors. For those DMAs who might have found the subject somewhat threatening, the f a c t that the background section of the 39 questionnaire asked the respondent to indicate his or her department and rank may have made i t d i f f i c u l t f o r them to complete. Secondly, as DMAs are f a m i l i a r with the personnel decision-making process, some DMAs may have found the format of the questionnaire incompatible with t h e i r b e l i e f s about how personnel decisions are made, t h i s problem was pointed out by one respondent who wrote "I can't respond to the items on the coloured pages. It i s the i n t e r a c t i o n between these items, not t h e i r 'single' e f f e c t s , that are c r i t i c a l f o r me." On the other hand, many of those people c l a s s i f i e d as Instructors may have had r e l a t i v e l y l i t t l e d i f f i c u l t y with the survey format as the majority of them had not had any experience making personnel decisions and, therefore, may not have had much d i f f i c u l t y r e c o n c i l i n g the format with t h e i r own personal perceptions of the personnel-decisions making process as i t occurs within the Faculty. Table 1: Comparison of Respondents with Non-Respondents  bv Group Respondents Non-Respondents Group n Percent n Percent DMAs 52 52.53 24 66.67 Instructors 47 47.47 12 33.33 Total 99 100.00 36 100.00 4 0 DESCRIPTION OF THE RESPONDENTS Departmental A f f i l i a t i o n Figure 2 shows the t o t a l number of respondents f o r each department by group, while Table 2 reports the frequency of the t o t a l respondents f or each department by group. The number of respondents per department ranges from a low of 6 for Soc i a l and Educational Studies (SEDS) to a high of 21 fo r Physical Education and Recreation (PHED). 25 • Instructors 20 DMAs 15 O a 3 10 5 0 Department Figure 2: Distribution of respondents by department. 41 Table 2: Frequency of Total Respondents by Group and Department/School DMAs Instructors Department/School n Percent n Percent Administration, Adult and Higher Education (AAHE) 7 7.07 • 5 5.05 Counselling Psychology (CNPS) 8 8.08 1 1.01 Educational Psychology and Special Education (EPSE) 6 6.06 11 11.11 Language Education (LANE) 10 10.10 2 2.02 Mathematics and Science Education (MASC) 6 6.06 8 8.08 Soc i a l and Educational Studies (SEDS) 6 6.06 . . N/A N/A Visual and Performing Arts i n Education (VPAE) 3 3.03 5 5.05 Physical Education and Recreation (PHED) 6 6.06 15 15.15 Total 52 52.53 47 47.47 Table 3 reports the population and sample sizes of DMAs by department. The percent response of DMAs by department ranges from a low of 32% for SEDS to a high of 100% for Administration, Adult and Higher Education (AAHE), Counselling Psychology (CNPS), and Mathematics and Science Education (MASC). The response rates f o r DMAs from other departments are 63% f o r Language Education (LANE), 67% f o r Educational Psychology and Special Education (EPSE), and 75% f o r Visual and Performing Arts i n 42 Education (VPAE). Table 4 shows the population and sample sizes of Instructors by department. The response rate of Instructors ranges from a low of 33% i n CNPS to a high of 100% i n VPAE. It can be seen from Tables 2 to 4 and Figure 2 that a l l people i n the research population of the SEDS department, regardless of rank, were i d e n t i f i e d as DMAs and none were i d e n t i f i e d as Instructors, t h i s i s due to the f a c t that a l l f a c u l t y members i n SEDS p a r t i c i p a t e i n personnel decisions at some l e v e l . A s i m i l a r s i t u a t i o n occurs i n the LANE department where 16 out of 19 members of the research population of t h i s department were i d e n t i f i e d as being members of a personnel committee. Table 3: Population and Sample Size of DMAs by Department Department/ DMAs  School Population Sample % Response AAHE 7 7 100.00 CNPS 8 8 100.00 EPSE 9 6 66.67 LANE ,16 10 62.50 MASC 6 6 100.00 SEDS 19 6 31.58 VPAE 4 3 75.00 PHED 7 6 85.71 Total 76 52 68.42 43 Table 4: Population and Sample Size of Instructors bv Department: Department/ Instructors School Population Sample % Response AAHE 6 5 83.33 CNPS 3 1 33.33 EPSE 14 11 78.57 LANE 3 2 66.67 MASC 11 8 72.73 SEDS N/A N/A N/A VPAE 5 5 100.00 PHED 17 15 88.24 Total 59 47 79.66 Table 2 also shows that the proportion of DMAs from SEDS who responded i s low (32%) when compared to DMAs of other departments. This low response rate may, i n part, be due to the fa c t that the version of the questionnaire d i s t r i b u t e d to DMAs was long and pertained to personnel decisions f o r appointments to a number of d i f f e r e n t ranks some of which may not have been f a m i l i a r to the non-respondents i n SEDS, e s p e c i a l l y those who may have only p a r t i c i p a t e d i n personnel decisions pertaining to the ranks of Assistant and/or Associate Professor. There i s some evidence to support t h i s suggestion i n that a number of respondents from SEDS indicated that they had only been able to complete a portion of the questionnaire as they had not p a r t i c i p a t e d i n any personnel decisions pertaining to any ranks other than t h e i r own. 4 4 Ranks of Respondents Respondents were asked to indicate t h e i r p r o f e s s o r i a l rank. Table 5 reports the percentage of respondents by group and rank. The majority of the DMAs who responded hold the ranks of Associate Professor With Tenure a (40.3%) and Professor With Tenure (40.4%). This i s to be expected as i t i s , o r d i n a r i l y , the most senior professors who are members of personnel committees. Table 5: Frequency of Respondents by Group and Rank Rank DMAs % Instructors n % Total No rank indicated by respondent Assistant Professor Without Tenure 0 0.0 0 0.0 2.1 10 21.3 1.0 10 10.1 Associate Professor Without Tenure 1.9 4.3 3.0 Professor Without Tenure 5.8 2.1 4.0 Assistant Professor With Tenure 6 11.5 11 23.4 17 17.2 Associate Professor With Tenure 21 40.4 17 36.2 38 38.4 Professor With Tenure 21 40.4 5 10.7 26 26.3 Total 52 100.0 47 100.0 99 100.0 45 The d i s t r i b u t i o n of ranks among the Instructors d i f f e r s from that of the DMAs i n that a greater proportion of them hold the more junior p r o f e s s o r i a l ranks. One Instructor declined to indicate a rank. Experience with Personnel Decisions The DMAs were asked to indicate the capacity i n which they currently p a r t i c i p a t e i n personnel decisions. Table 6 reports the frequencies of t h e i r responses. The majority (78.85%) of respondents were members of Departmental Standing Personnel Committees (DSPC), which i s consistent with the f a c t that the majority of the DMAs (86.84%) to whom questionnaires were d i s t r i b u t e d belonged to a DSPC. Table 7 shows the d i s t r i b u t i o n of committees to which the DMAs of the survey population belonged. These figures were a r r i v e d at as a r e s u l t of information obtained from administrators throughout the Faculty. It can be seen from Table 6 that some DMAs p a r t i c i p a t e i n personnel decisions i n more than one capacity; these c a p a c i t i e s were not known to the researcher p r i o r to c o l l e c t i o n of the data; therefore, they were not included i n the groupings indicated i n Table 7. 46 Table 6: Frequency of DMAs by Committee or Administrative Capacity in the Survey Sample Committee Number Percent Departmental Standing Personnel Committee (DSPC) 41 78.85 Faculty Standing Personnel Committee (FSPC) 3 5.77 Head of Department 5 9.62 Dean/Associate Dean/Assistant Dean 1 1.92 Senior Appointments Committee 1 1.92 Multiple c a p a c i t i e s 1 1.92 Total 52 100.00 Table 7: Frequency of DMAs by Personnel Committee i n the Survey Population Committee Number Percent Departmental Standing Personnel Committee (DSPC) 66 86.84 Faculty Standing Personnel Committee (FSPC) 8 10.53 Senior Appointments Committee (SAC) 2 2.63 Total 76 100.00 47 The majority of DMAs indicated considerable experience making personnel decisions; nearly 60% had p a r t i c i p a t e d i n personnel decisions i n t h e i r current capacity f o r s i x years or more (Table 8). More that 80% had pa r t i c i p a t e d i n personnel decisions f o r more than 3 years suggesting that the majority of respondents should have substantial knowledge of how personnel decisions are made by the committee or committees to which they belong. Table 8: Years Experience of DMAs on Current Personnel Committee Years i n Current Capacity Number Percent l e s s than 2 years 10 19.23 3 to 5 years 11 21.15 6 to 10 years 18 34.62 more than 10 years 13 25.00 Total 52 100.00 DMAs were asked to indicate i f they had ever p a r t i c i p a t e d i n personnel decisions i n another capacity other than the one they currently held. Forty-eight percent of the DMAs indicated that they had previously p a r t i c i p a t e d i n personnel decisions i n other capacities (Table 9). Table 11 summarizes the other capacities held by DMAs. 48 Table 9: Frequency of DMAs Who Have Participated i n Other  Personnel Decision-making Capacities Item Number Percent Yes 25 48.08 No 27 51.92 Total 52 100.00 Instructors were asked to indicate i f they had ever p a r t i c i p a t e d i n any personnel decisions. Forty-nine percent of the Instructors indicated that they had, at one time, p a r t i c i p a t e d i n personnel decisions (Table 10). Of those Instructors who had p a r t i c i p a t e d i n personnel decisions, 78.26% had done so as members of a Department Standing Personnel Committee (Table 11). This experience would undoubtedly add to t h e i r f a m i l i a r i t y with the personnel decision-making process. Table 10: Frequency of Instructors Who Had Previously Participated  i n Personnel Decision-making Capacities Item Number Percent Yes 23 48.94 No 24 51.06 Total 47 100.00 49 Table 11: Number of Respondents Who Had Participated i n Personnel Decisions i n Other Capacities by Group DMAs Instructors Capacity n Percent n Percent DSPC 5 20.00 18 78.26 FSPC 5 20.00 1 4.35 SAC 2 8.00 0 0.00 Head of Department 4 16.00 0 0.00 Dean/Associate Dean/ Assistant Dean 4 16.00 1 4.35 Multiple c a p a c i t i e s 5 20.00 3 13.04 Total 25 100.00 23 100.00 Length of Service of Instructors The Instructors were asked to indicate how long they had been members of the Faculty of Education; Table 12 summaries t h e i r responses. A large majority of them (74%) indicated that they had been members of the Faculty of Education f o r 6 or more years. Nearly ninety percent (87%) of the i n s t r u c t o r s have been f a c u l t y members f o r 3 or more years suggesting that most Instructors should have some f a m i l i a r i t y with the personnel decision-making process. SO Table 12: Length of Service of Instructors i n the Faculty of Education Length of Service i n Years l e s s than 2 years 3 to 5 years 6 to 10 years more than 10 years Total Number Percent 6 6 6 29 12.77 12.77 12.77 61.70 47 100.00 RESPONDENTS' PERCEPTIONS OF PERSONNEL DECISIONS A l l respondents were asked to indicate the importance of a number of factors and sources of information considered by DMAs during the personnel decision-making process. Respondents who were c l a s s i f i e d as DMAs were asked to give the response that best indicated how important they t y p i c a l l y consider the factor or source of information to be when they make personnel decisions. Respondents c l a s s i f i e d as Instructors were asked to indicate how important they f e l t the fac t o r or information source i s to the people who make personnel decisions. DMAs were asked to provide responses f o r personnel decisions concerning appointments to Assistant Professor Without Tenure, Associate Professor Without Tenure, Professor Without Tenure and Appointments With Tenure; Instructors were asked to provide feedback f o r decisions concerning Appointments With and Without Tenure to t h e i r rank only. 51 Appendix G reports the respondents' ratings of the importance of each f a c t o r or source of information that could be considered f o r each type of personnel d e c i s i o n addressed i n the questionnaire. The frequencies of a l l v a l i d responses f o r both DMAs and Instructors are reported i n the tables. The response a l t e r n a t i v e s , a f t e r recoding, are: 1 (Not At A l l Important), 2 (Not Very Important), 3 (Important), and 4 (Very Important). Mean responses f o r each f a c t o r and source of information by respondent group were ca l c u l a t e d using only those cases f o r which there were no missing data. In the following sections, Tables 13 to 18, 20, and 21 report the mean response ratings f o r both DMAs and Instructors. Hotelling's T 2 s t a t i s t i c s were calculated to see i f there were any s i g n i f i c a n t differences between DMAs and Instructors with respect to the way they rated the importance of the various factors and sources of information f o r each type of personnel decision. Univariate t - t e s t s were conducted whenever any s i g n i f i c a n t differences were indicated by the multivariate analysis of variance. Hotelling's T 2 s t a t i s t i c and the re s u l t s of the univariate t - t e s t s , where applicable, are included i n Tables 13 - 18, 20 and 21 below. The DMAs' mean responses f o r the factors were rank ordered from la r g e s t to smallest; the la r g e r the mean response the greater the rated importance of the f a c t o r i n a candidate's o v e r a l l teaching performance. Figures 3, 5, 7, and 9 report the r e s u l t s of the ranking of each f a c t o r . S i m i l a r l y , the DMAs mean responses f o r the sources of information were rank ordered from l a r g e s t to smallest; the larger the mean response the greater the rated importance of the source of information i n a 52 candidate's teaching effectiveness evaluation. The r e s u l t s of t h i s process are reported i n Figures 4, 6, 8, and 10. For comparative purposes, Figures 3 to 10 also show the mean response ratings of the Instructors. The midpoint of the response scale (2.5) i s indicated on the graphs with a broken l i n e . Assistant Professors Without Tenure Factors Considered i n Overall Performance Evaluation Appendix G-l presents the respondents' ratings of the importance of factors that could be considered i n the o v e r a l l performance evaluation of a candidate f o r Assistant Professor Without Tenure. A l l v a l i d responses were included i n the computation of the frequencies. Respondents were given the opportunity to indicate any ad d i t i o n a l factors they considered to be important. Three DMAs each indicated one other f a c t o r to be important to them: 'scholarly p o t e n t i a l ' ; 'whether or not the candidate i s l i k e d ' ; and a 'candidate's f i e l d of scholarship' (not shown i n t a b l e ) . These ad d i t i o n a l factors were not included i n the ov e r a l l frequencies or i n any further c a l c u l a t i o n s as they were indicated by so few of the DMAs. Table 13 reports the mean response r a t i n g f o r each f a c t o r by group. The Hotelling's T 2 analysis showed a s i g n i f i c a n t multivariate difference between the two groups (Hotelling's T 2 =2.40; p = 0.02; df = 12, 49). The r e s u l t s of the univariate t - t e s t s are shown i n Table 13 . S i g n i f i c a n t differences were found between the mean responses of DMAs and the mean responses of Instructors f o r the factors 'classroom teaching', 53 'publications', 'public service', 'consultation', and 'student program advising'. Figure 3 shows that, when i t comes to o v e r a l l performance evaluation of a candidate f o r Assistant Professor Without Tenure, the fa c t o r with the highest rated importance, based on the DMAs' mean response r a t i n g , i s 'research a c t i v i t y ' , followed by 'classroom teaching', 'publications' , and then 'supervision of graduate study'. The lower ratings of the remaining factors indicate that they have r e l a t i v e l y l e s s importance i n o v e r a l l performance evaluation to DMAs. It can also be seen from Figure 3 that, based on the Instructors' mean response ratings, 'publications' has the highest rated importance, c l o s e l y followed by 'research a c t i v i t y ' . The factors 'classroom teaching' and 'supervision' are rated somewhat l e s s important. The remaining factors were rated by Instructors as having r e l a t i v e l y l e s s importance i n personnel decisions. 54 Table 13: T-test Results for Factors Considered i n Appointments to  Assistant Professor Without Tenure Mean A~ Mean B b t p Factors n = 45 n = 17 Classroom teaching 3. .36 2. .77 2. .78 0. .01 * Research activity- 3, .58 3, .65 -0. .43 0. .67 Publications 3. .27 3. .82 -3. .29 0. ,00 * Public service 2, .09 1 .71 2, .17 0. .03 * Consultation (Governments/business) 1. .89 1. .53 2. .05 0. .04 * A c t i v i t y i n professional s o c i e t i e s 2. .51 2. .24 1. .36 0. .18 Supervision of graduate study 2. .78 2. .59 0. .82 0, .42 Student program advising 2. .18 1. .77 2. .07 0. .04 * University, f a c u l t y or department committee work 2. .11 2, .00 0. .57 0. .57 Length of service at rank 1. .96 2 .12 -0. .64 0. .53 Public recognition 2. .31 2. .12 0. .94 0. .35 Personality c h a r a c t e r i s t i c s 2. .42 2. .29 0. .46 0. ,65 Hotelling's T 2 =2.40 df = 12, 49 p - 0.02 aMean A = Mean response DMAs bMean B = Mean response Instructors * s i g n i f i c a n t at alpha = .05 Note. Response Range: 1 (Not at a l l Important) to 4 (Very Important) 55 Research Teaching Publication Supervision Societies O Personality o ^ Recognition Prg Advising Committees Service Yrs at Rank Consultation - i 1 1 r i 1—:—i r _l I I l_ i 1 1 r O Instructors • DMAs I I I . L_ 2 3 Mean Response Figure 3: Factors considered i n appointments to Assistant Professor Without Tenure ranked by DMAs' mean response. 56 Sources of Information i n Teaching Effectiveness Evaluation Appendix G-2 presents respondent ratings of the importance of various sources of information that could be considered i n the teaching effectiveness evaluation of a candidate f o r Assistant Professor Without Tenure. A l l v a l i d responses were included i n the computation of the frequencies. Respondents were given the opportunity to indicate any a d d i t i o n a l information sources they considered to be important. Two DMAs each indicated one other information source to be important to them; 'personal observation of teaching p r a c t i c e s ' and 'PE student evaluations' (not shown i n t a b l e ) . These a d d i t i o n a l information sources were not included i n the o v e r a l l frequencies or i n any further c a l c u l a t i o n s as they were indicated by so few of the DMAs. Table 14 reports the mean response r a t i n g f o r each information source by group. The Hotelling's T 2 analysis showed a nonsignificant r e s u l t (Hotelling's T 2 = 1.93; p = 0.06; df = 14, 34). Figure 4 shows that, when candidates f o r Assistant Professor Without Tenure are being evaluated f o r teaching effectiveness, the information source with the highest rated importance, based on the DMAs mean response ratings, i s 'peer reviews', followed by 'formal student ratings', and then, 'opinion of outside experts'. The lower ratings of the remaining sources of information appear to indicate that they have r e l a t i v e l y l e s s importance i n teaching effectiveness evaluation to DMAs. 57 Table 14: Mean Response Ratines for Sources of Information Considered i n Appointments to Assistant Professor Without Tenure Mean A a Mean B b Source of Information n = • 38 n = • 11 Formal Student Ratings 3. 24 3. 09 Formal Peer Reviews 3. 29 2. ,91 Informal student opinion 2. .18 2. 00 Informal opinions of colleagues 2. 32 2. ,64 Evaluation by Head of Department 2. ,55 3. ,27 Evaluation by Dean 2. .11 2, .73 Opinion of outside experts 3. ,00 2. .64 Self-evaluation or report 2. .18 2, .18 Course s y l l a b i and examinations 2. .58 2, .36 Student examination performance 1. .90 1. .73 Grade d i s t r i b u t i o n s 1. .82 1. .91 Enrollment i n e l e c t i v e classes 1. .76 1. .82 Long-term follow-up of students 1. .82 1. .36 Research/Publication 2. .55 3, .18 Hotelling's T 2 = 1.93 p = 0.06 df = 14, 34 *Mean A = Mean response DMAs bMean B = Mean response Instructors Note. Response Range: 1 (Not at a l l Important) to 4 (Very Important) 58 o O CI t-H O o o (73 Peer Reviews Stud Ratings Expert Opin Syllabi/Exam Dpt Head Eva Research/Pub Peer Opin Self Eval Student Opin Dean Eval Student Exam Grade Dist Alumni Opin Elective Enr n 1 1 r "i 1—:—i r I I Instructors • DMAs J i i i i i i i i i i i i i i_ 2 3 Mean Response Figure 4: Sources of information considered i n appointments to Assistant Professor Without Tenure ranked by DMAs' mean response. 59 It can be seen from Figure 4 that, based on the Instructors' mean response ratings, 'evaluation by head of department' has the highest rated importance, followed by ' research/publication', 'formal student ratings', and then 'formal peer reviews'. The remaining sources of information were rated by Instructors as having r e l a t i v e l y l e s s importance i n personnel decisions. Associate Professors Without Tenure Factors Considered i n Overall Performance Evaluation Appendix G-3 presents respondent ratings of the importance of factors that could be considered i n the o v e r a l l performance evaluation of a candidate f o r Associate Professor Without Tenure. A l l v a l i d responses were included i n the computation of the frequencies. Respondents were given the opportunity to indicate any ad d i t i o n a l factors they considered to be important. One of the DMAs indicated 'depth of understanding' to be another important f a c t o r (not shown i n t a b l e ) . One of the Instructors indicated 'development - praxis' as another important f a c t o r (not shown i n t a b l e ) . These a d d i t i o n a l factors were not included i n the o v e r a l l frequencies or i n any further c a l c u l a t i o n s as they were indicated by so few respondents. The Hotelling's T 2 analysis showed a s i g n i f i c a n t multivariate difference between the two groups (Hotelling's T 2 = 2.59; p = 0.01; df = 12, 44). Table 15 shows the r e s u l t s of the univariate t - t e s t s . S i g n i f i c a n t differences were found between the group means f o r the factors 60 Table 15: T-test Results for Factors Considered i n Appointments to  Associate Professor Without Tenure Mean A~ Mean B b t p Factors n = 42 n = 15 Classroom teaching 3. .33 3. .20 0. .68 0. .50 Research a c t i v i t y 3. .76 3. .60 0. .96 0. .34 Publications 3. .71 3. .47 0. ,36 0. .18 Public service 2. .19 2, .13 0. .23 0. .82 Consultation (Gove rnment s/bus ine s s) 2. .95 3. .07 -0. .54 0. ,59 A c t i v i t y i n professional s o c i e t i e s 2. .60 2. .33 1. .33 0. .19 Supervision of graduate study 3, .26 2, .73 2. .71 0. .01 * Student program advising 2. .31 1. .80 2. .40 0. .02 * University, f a c u l t y or department committee work 2. .60 .1. .93 3. .14 0. .00 * Length of service at rank 2. .17 2. .00 0. .65 0. ,52 Public recognition 2 .55 2 .80 -0, .95 0. .35 Personality c h a r a c t e r i s t i c s 2 .12 2. .53 -1. .34 0. .19 Hotelling's T 2 = 2.59 df = 12, 44 p = 0.01 "Mean A = Mean response DMAs bMean B = Mean response Instructors * s i g n i f i c a n t at alpha = .05 Note. Response Range: 1 (Not at a l l Important) to 4 (Very Important) 61 Research Publication Teaching Supervision Consultation O Societies O r ^ Committees Recognition Prg Advising Service Yrs at Rank Personality 2 3 4 "i 1 1 1 1—:—i 1 1 1 1 1 1— _l I L. Q Instructors • DMAs - i i i i_ 2 3 Mean Response Figure 5: Factors considered i n appointments to Associate Professor Without Tenure ranked by DMAs' mean response. 62 'supervision of graduate study', 'student program advising', and 'committee work'. Figure 5 shows that, when i t comes to o v e r a l l performance evaluation of a candidate f o r Associate Professor Without Tenure, the f a c t o r with the highest rated importance, based on the DMAs' mean response r a t i n g , i s 'research a c t i v i t y ' , followed by 'publications', 'classroom teaching', 'supervision of graduate study', and then 'consultation'. The lower ratings of the remaining factors appear to indicate that they have r e l a t i v e l y l e s s importance i n o v e r a l l performance evaluation to DMAs. It can be seen from Figure 5 that, based on the Instructors' mean response ratings, 'research a c t i v i t y ' has the highest rated importance, followed by , 'publications', 'classroom teaching', 'consultation', 'public recognition', and then 'supervision of graduate study'. The remaining factors were rated as having r e l a t i v e l y l e s s importance by the Instructors i n personnel decisions. Sources of Information i n Teaching Effectiveness Evaluation Appendix G-4 presents respondent ratings of the importance of sources of information that could be considered i n the teaching effectiveness evaluation of a candidate f o r Associate Professor Without Tenure. A l l v a l i d responses were included i n the computation of the frequencies. Respondents were given the opportunity to indicate any ad d i t i o n a l information sources they considered to be important. Two of the DMAs each indicated one other information source to be of importance to them: 'personal observation of teaching p r a c t i c e s ' and 'PE student evaluations' 63 (not shown i n t a b l e ) . These a d d i t i o n a l information sources were not included i n the o v e r a l l frequencies or i n any further c a l c u l a t i o n s as they were indicated by so few of the DMAs. The Hotelling's T 2 analysis showed a s i g n i f i c a n t multivariate difference between the two groups (Hotelling's T 2 = 2.19; p = 0.03; df = 14, 35). Table 16 shows the r e s u l t s of the univariate t - t e s t s . S i g n i f i c a n t differences were found between the group means f o r the source of information 'course s y l l a b i and examinations'. Figure 6 shows that, when candidates for Associate Professor Without Tenure are being evaluated f o r teaching effectiveness, the information source with the highest rated importance, based on the DMAs mean response ratings, i s 'peer reviews', followed c l o s e l y by 'formal student ratings'. The information sources 'opinion of outside experts', 'research/publication', 'course s y l l a b i and examinations', and 'evaluation by head of department' have a lower rated importance. The lower ratings of the remaining sources of information indicate that they have r e l a t i v e l y l e s s importance i n teaching effectiveness evaluation to DMAs. It can be seen from Figure 6 that, based on the Instructors' mean response ratings, 'formal student r a t i n g s ' has the highest rated importance, followed by 'peer reviews', 'research/publication' and 'evaluation by head of department', and then 'opinion of outside experts', The remaining sources of information were rated by Instructors as having r e l a t i v e l y l e s s importance i n personnel decisions. 64 Table 16: T-test Results for Sources of Information Considered in  Appointments to Associate Professor Without Tenure Mean A a Mean B b t p Source of Information n = 38 n = 12 Formal Student Ratings 3.32 3.25 0. .28 0. .78 Formal Peer Reviews 3.45 3.17 .1. .35 0. .19 Informal student opinion 2.11 2.00 0. .42 0. .68 Informal opinions of colleague 2.32 2.00 1. .29 0, .20 Evaluation by Head of Department 2.63 2.92 -1. .18 0. .25 Evaluation by Dean 2.24 2.42 -0, .56 0, .58 Opinion of outside experts 2.95 2.75 0. .57 0. .57 Self-evaluation or report 2.24 2.17 0. .26 0. .79 Course s y l l a b i and examinations 2.74 2.08 2. .57 0. .01 * Student examination performance 2.13 2.08 . 0. 19 0, .85 Grade d i s t r i b u t i o n s 1.87 1.92 -0, .23 0, .82 Enrollment i n e l e c t i v e classes 1.87 2.00 -0. .54 0. ,59 Long-term follow-up of students 1.84 2.08 -0. .87 0. .39 Research/Publication 2.74 2.92 -0, .48 0. .64 Hotelling's T 2 = 2.19 df - 14, 35 p = 0.03 aMean A = Mean response DMAs bMean B = Mean response Instructors * s i g n i f i c a n t at alpha = .05 Note. Response Range: 1 (Not at a l l Important) to 4 (Very Important) 65 1 o O r—I o o o u o (73 Peer Reviews Stud Ratings Expert Opin Research/Pub Syllabi/Exam Dpt Head Eva Peer Opin Dean Eval Self Eval Student Exam Student Opin Elective Enr Grade Dist -Alumni Opin 1 HT _i i u O Instructors • DMAs 2 3 Mean Response Figure 6: Sources of information considered i n appointments to Associate Professor Without Tenure ranked by DMAs' mean response. 66 Professors Without Tenure Factors Considered i n Overall Performance Evaluation Appendix G-5 presents respondent ratings of the importance of factors that could be considered i n the o v e r a l l performance evaluation of a candidate f o r Professor Without Tenure. A l l v a l i d responses were included i n the computation of the frequencies. Respondents were given the opportunity to indicate any ad d i t i o n a l factors they considered to be important. One of the DMAs indicated that the 'quality of debate' was another important f a c t o r (not shown i n t a b l e ) . This a d d i t i o n a l f a c t o r was not included i n the o v e r a l l frequencies or i n any further c a l c u l a t i o n s as i t was only indicated by one DMA. Table 17 reports the mean response ratings f o r each f a c t o r by group. The Hotelling's T 2 analysis was not s i g n i f i c a n t (Hotelling's T 2 =1.43; p - 0.22; df = 12, 26). Figure 7 shows that, when i t comes to o v e r a l l performance evaluation of a candidate f o r Professor Without Tenure, the factor with the highestrated importance, based on the DMAs' mean response rating, i s 'research a c t i v i t y ' , followed by 'publications', 'supervision of graduate study', and then 'classroom teaching'. The lower ratings of the remaining factors appear to indicate that they have r e l a t i v e l y l e s s importance i n ov e r a l l performance evaluation to DMAs. It can also be seen from Figure 7 that, based on the Instructors' mean response ratings, 'research a c t i v i t y ' has the highest rated importance, followed by 'classroom teaching', 'publications', and then 67 Table 17: Mean Response Ratings for Factors Considered i n Appointments to Professor Without Tenure Mean A a Mean B b Factor n = 33 n = 6 Classroom teaching 3. .46 3. .33 Research a c t i v i t y 3, .91 3, .83 Publications 3, .82 3. ,17 Public service 2 .49 2. .00 Consultation (Gove rnment s/bus ine s s) 2. .15 2. .33 A c t i v i t y i n professional s o c i e t i e s 2. .70 2. .17 Supervision of graduate study 3 .64 3. .00 Student program advising 2. .58 2. .50 University, f a c u l t y or department committee work 2. .61 2. .50 Length of service at rank 2. .15 2. ,00 Public recognition 2. .94 2. .67 Personality c h a r a c t e r i s t i c s 2. .21 2. ,33 Hotelling's T 2 = 1.43 p = 0.22 df = 12, 26 "Mean A = Mean response DMAs "Mean B = Mean response Instructors Note. Response Range: 1 (Not at a l l Important) to 4 (Very Important) 68 1 2 — i — i — i — i — p Research Publication Supervision Teaching Recognition O Societies o & Committees Prg Advising Service Personality Consultation Yrs at Rank ~i 1—:—i r ~\ 1 r J i i i 1 i i_ • Instructors • DMAs _ J i i_ 2 3 Mean Response Figure 7: Factors considered i n appointments to Professor Without Tenure ranked by DMAs' mean response. 69 'supervision of graduate study'. The remaining factors were rated by Instructors as having r e l a t i v e l y l e s s importance i n personnel decisions. Sources of Information i n Teaching Effectiveness Evaluation Appendix G-6 presents respondent ratings of the importance of various sources of information that could be considered i n the teaching effectiveness evaluation of a candidate f o r Professor Without Tenure. A l l v a l i d responses were included i n the computation of the frequencies. Respondents were given the opportunity to indicate any add i t i o n a l information sources they considered to be important. One DMA indicated 'personal observations of teaching p r a c t i c e s ' to be another important source of information. This a d d i t i o n a l information source was not included i n the o v e r a l l frequencies or i n any further c a l c u l a t i o n s as i t was only indicated by one DMA. The Hotelling's T 2 analysis showed a s i g n i f i c a n t multivariate difference between the two groups (Hotelling's T 2 = 3.17; p = 0.01; df = 14, 23). Table 18 shows the univariate t analyses. A s i g n i f i c a n t difference was found between the mean responses of DMAs and Instructors f o r 'enrollment i n e l e c t i v e classes'. Figure 8 shows that, when candidates f o r Professor Without Tenure are being evaluated f o r teaching effectiveness, the information source with the highest rated importance, based on the DMAs mean response ratings, i s 'peer reviews', followed by 'formal student ratings', 'opinion of outside experts', and then 'research/publication'. The lower ratings 70 Table 18: T-test Results for Sources of Information Considered in Appointments to Professor Without Tenure Mean A a Mean B b t p Source of Information n = 32 n = 6 Formal Student Ratings 3. .34 3. .67 -0. .96 0. .34 Formal Peer Reviews 3. .50 3. .50 0. .00 1. .00 Informal student opinion 2. .13 2. .17 -0. .12 0. .91 Informal opinions of colleagues 2. .34 2. .33 0. .03 0. .98 Evaluation by Head of Department 2. .50 3. .33 -2, .09 0. .05 Evaluation by Dean 2. .19 2. .83 -1, .35 0. .19 Opinion of outside experts 3. .13 2. .83 0, .62 0. .54 Self-evaluation or report 2. .19 2. .67 -1. .31 0. .20 Course s y l l a b i and examinations 2. .63 2. .83 -0. .60 0. .56 Student examination performance 1. .94 3. .00 no variance Grade d i s t r i b u t i o n s 1. .91 2 .17 -0. .84 0. .41 Enrollment i n e l e c t i v e classes 1. .94 3. .17 -3. .48 0. .00 * Long-term follow-up of students 2. .03 2, .83 -1. .92 0. .06 Research/Publication 2. .81 3. .17 -0. .64 0. .53 Hotelling's T 2 = 3.17 df = 14, 23 p = 0.01 "Mean A = Mean response DMAs bMean B = Mean response Instructors * s i g n i f i c a n t at alpha = .05 Note. Response Range: 1 (Not at a l l Important) to 4 (Very Important) 71 Peer Reviews Stud Ratings Expert Opin Research/Pub Syllabi/Exam Dpt Head Eva Peer Opin Dean Eval Self Eval Student Opin Alumni Opin Elective Enr Student Exam Grade Dist "i 1 r _! I L. ~i 1 1 1 1 r 3 D Instructors • DMAs _i i i i_ 2 3 Mean Response Figure 8: Sources of information considered i n appointments to Professor Without Tenure ranked by DMAs' mean response. 72 of the remaining sources of information appear to indicate that they have r e l a t i v e l y l e s s importance i n teaching effectiveness evaluation to DMAs. It can be seen from Figure 8 that, based on the Instructors' mean response ratings, 'formal student ra t i n g s ' has the highest rated importance, followed by 'formal peer reviews', 'evaluation by head of department', 'research/publication', and then 'enrollment i n e l e c t i v e classes'. The remaining sources of information were rated by i n s t r u c t o r s as having r e l a t i v e l y l e s s importance i n personnel decisions. Appointments With Tenure The DMAs were asked to indicate the rank or ranks f o r which t h e i r responses concerning Appointments With Tenure were most v a l i d or applicable; Table 19 summarizes t h e i r responses. It can be seen from Table 19 that a number of d i f f e r e n t responses were given with varying frequencies; therefore, i t was decided to aggregate a l l responses into one category, Appointments With Tenure, rather than into a l l of the categories indicated i n Table 19. 73 Table 19: Ranks at Which DMAs Responses Concerning Appointments With  Tenure are Most Valid/Applicable Rank Indicated Number Percent No rank indicate 17 32.69 Assistant Professor 9 17.31 Associate Professor 9 17.31 F u l l Professor 1 1.92 Assistant and Associate Professor 5 9.62 Associate and F u l l Professor 4 . 7.69 A l l ranks 7 13.46 Total 52 100.00 Factors Considered i n Overall Performance Evaluation Appendix G-7 presents respondent ratings of the importance of factors that could be considered i n the o v e r a l l performance evaluation of a candidate f o r a p o s i t i o n With Tenure. A l l v a l i d responses were included i n the computation of the frequencies. Respondents were given the opportunity to indicate any add i t i o n a l factors they considered to be important. One of the DMAs indicated ' r a t i o n a l powers' to be another important f a c t o r (not shown i n ta b l e ) . Three of the Instructors each indicated another a d d i t i o n a l f a c t o r . The addi t i o n a l f a c t o r s were ' p o l i t i c s ' , 'grant a c q u i s i t i o n ' , and 'creative endeavour'. These a d d i t i o n a l factors were not included i n the o v e r a l l 74 frequencies or i n any further c a l c u l a t i o n s as they were indicated by so few respondents. The Hotelling's T 2 analysis showed a s i g n i f i c a n t multivariate difference between the two groups (Hotelling's T 2 = 2.05; p = 0.03; df = 12, 72). Table 20 shows the univariate t analyses. S i g n i f i c a n t differences were found between the mean responses of DMAs and the mean responses of Instructors f o r the factors 'classroom teaching', 'supervision of graduate study', 'student program advising' and 'committee work'. Figure 9 shows that, when i t comes to o v e r a l l performance evaluation of a candidate f o r an Appointment With Tenure, the fa c t o r with the highest rated importance, based on the DMAs' mean response r a t i n g , i s 'research a c t i v i t y ' , followed by 'publications','classroom teaching', and then 'supervision of graduate study'. The lower ratings of the remaining factors appear to indicate that they have r e l a t i v e l y l e s s importance i n ov e r a l l performance evaluation to DMAs. It can also be seen from Figure 9 that, based on the Instructors' mean response ratings, 'publications' has the highest rated importance, followed by 'research a c t i v i t y ' , 'supervision of graduate study', and then 'classroom teaching'. The remaining factors were rated as having r e l a t i v e l y l e s s importance i n personnel decisions by Instructors. 75 Table 20: T-test Results for Factors Considered i n Appointments  With Tenure Mean A a Mean Bta t p Factor n = 44 n = 41 Classroom teaching 3. .52 2. .85 3. .89 0. ,00 * Research a c t i v i t y 3. .84 3. ,78 0. .71 0. ,48 Publications 3. .77 3. .81 0. .36 0. .72 Public service 2, .32 2. .12 1, .11 0, .27 Consultation (Gove rnment s/bus ine s s) 2. .09 2. .02 0. .40 0. .69 A c t i v i t y i n professional s o c i e t i e s 2. .64 2. .46 1. .09 0. .28 Supervision of graduate study 3. .23 2. .93 1. .89 0. .06 * Student program advising 2. .30 1. .90 2. .32 0. .02 * University, f a c u l t y or department committee work 2. .59 2. .15 2. .89 0. .01 * Length of service at rank 2. .25 2. .10 0. .81 0. .42 Public recognition 2 .52 2, .44 0, .42 0. .68 Personality c h a r a c t e r i s t i c s 2, .21 2. .32 -0. .55 0. ,59 Hotelling's T 2 = 2.05 df = 12, 72 p = 0.03 aMean A = Mean response DMAs toMean B = Mean response Instructors * s i g n i f i c a n t at alpha = .05 Note. Response Range: 1 (Not at a l l Important) to 4 (Very Important) 76 Research Publication Teaching Supervision Societies 0 Committees O ^ Recognition Service Prg Advising Yrs at Rank Personality Consultation 3 4 I i i i i • Instructors • DMAs _i i i u 2 3 Mean Response Figure 9: Factors considered i n Appointments With Tenure ranked by DMAs' mean response. 77 Sources of Information i n Teaching Effectiveness Evaluation Appendix G-8 presents respondent ratings of the importance of various sources of information that could be considered i n the teaching effectiveness evaluation of a candidate f o r a p o s i t i o n With Tenure. A l l v a l i d responses were included i n the computation of the frequencies. Respondents were given the opportunity to indicate any ad d i t i o n a l information sources they considered to be important. Three of the DMAs each indicated another information source of importance to them. The addi t i o n a l information sources were 'knowledge and reasoning,' 'quality of journal', and 'PE student evaluations'. These a d d i t i o n a l information sources were not included i n the o v e r a l l frequencies or i n any further c a l c u l a t i o n s as they were indicated by so few of the DMAs. The Hotelling's T 2 analysis showed a s i g n i f i c a n t multivariate difference between the two groups (Hotelling's T 2 = 2.61; p = 0.01; df = 14, 60). Table 21 shows the univariate t analyses. S i g n i f i c a n t differences between the mean responses of DMAs and the mean responses of Instructors f o r 'formal peer reviews' and 'course s y l l a b i ' and Examinations. Figure 10 shows that, when candidates f o r Appointments With Tenure are being evaluated f o r teaching effectiveness, the information sour.ce with the highest rated importance, based on the DMAs mean response ratings, i s 'peer reviews', followed by 'formal student ratings', 'opinion of outside experts', and then 'research/publication'. The lower ratings of the remaining sources of information appear to indicate that they have r e l a t i v e l y l e s s importance i n teaching effectiveness evaluation to DMAs. 78 Table 21: T-test Results for Sources of Information Considered  for Appointments With Tenure Mean A3- Mean B b t p Source of Information n = 39 n = 36 Formal Student Ratings 3. .41 3. ,14 1. ,48 0. ,14 Formal Peer Reviews 3, .54 3. .11 2, .59 0. .01 * Informal student opinion 2. .10 1. .94 0. ,89 0. ,38 Informal opinions of colleagues 2. .41 2. .31 0. .56 0; .57 Evaluation by Head of Department 2. .67 3. .14 -2, .35 0, .02 * Evaluation by Dean 2, .23 2. .69 -1, .93 0, .06 Opinion of outside experts 3. .05 2. .67 1. .73 0. .09 Self-evaluation or report 2 .28 1, .94 1. .73 0, .09 Course s y l l a b i and examinations 2, .69 2, .17 2. .77 0. .01 * Student examination performance 1, .97 1. .89 0. .51 0. .61 Grade d i s t r i b u t i o n s 1 .95 1, .89 0, .37 0, .71 Enrollment i n e l e c t i v e classes 1. .92 1. .97 -0, .27 0. .79 Long-term follow-up of students 1. .97 1. .69 1. .35 0. .18 Research/Publication 2 .90 .3, .28 -1, .51 0. .14 Hotelling's T 2 = 2.61 df = 14, 60 p = 0.01 "Mean A = Mean response DMAs bMean B = Mean response Instructors * s i g n i f i c a n t at alpha = .05 Note. Response Range: 1 (Not at a l l Important) to 4 (Very Important) 79 o •t-4 -t-> a o <4-l O o o (72 Peer Reviews Stud Ratings Expert Opin Research/P ub Syllabi/Exam Dpt Head Eva Peer Opin Self Eval Dean Eval Student Opin Student Exam Alumni Opin Grade Diat Elective Enr T 1 r 1 1—;—i r _i i i i_ n 1 1 r • Instructors • DMAs - i i i i_ 2 3 Mean Response Figure 10: Sources of information considered i n Appointments With Tenure ranked by DMAs' mean response. 80 It can be seen from Figure 10 that, based on the Instructors' mean response ratings, 'research/publications' has the highest rated importance, followed, with the same rat i n g , by 'formal student ratings' and 'evaluation by department head, and then 'peer reviews'. The remaining sources of information were rated as having r e l a t i v e l y l e s s importance i n personnel decisions by i n s t r u c t o r s . Comparison of the Relative Importance of Factors Table 22 summarizes the r e l a t i v e importance, based on mean response, of the factors considered i n o v e r a l l performance evaluation f o r a l l types of appointments addressed i n the questionnaire. The r e s u l t s f o r both the DMAs and the Instructors are reported i n the table (DMA r e s u l t s are i n bold type). For DMAs, the most important fa c t o r , regardless of appointment, i s research a c t i v i t y . Publications i s second f o r a l l appointments except Assistant Professor Without Tenure; i t i s ranked t h i r d f or Assistant Professor. Classroom teaching i s ranked second i n importance f o r Assistant Professor Without Tenure, t h i r d f o r Associate Professor Without and Appointments With Tenure, and fourth f o r Professor Without Tenure. Supervision of graduate study has a r e l a t i v e l y high importance when i t comes to o v e r a l l performance evaluation as i s ranked t h i r d f o r Professor Without Tenure and fourth f o r the other types of appointments. Table 22 shows that Instructors' perceptions of the r e l a t i v e importance of the factors varies somewhat from that of the DMAs. Research a c t i v i t y i s most important f o r appointments to Associate or F u l l Professor 81 Table 22: Ranking of Factors Considered i n Evaluating Overall  Performance Type of Appointment  Without Tenure With Factor Assistant Associate F u l l Tenure Classroom teaching 2 3 4 3 3 3 2 4 Research a c t i v i t y 1 1 1 1 2 1 1 2 Publications 3 2 2 2 1 2 3 1 Public service 10 10 9 8 11 9 11.5 9 Consultation 12 5 11.5 12 (governments/business) 12 4 8.5 11 A c t i v i t y i n 5 6.5 6 5 professional s o c i e t i e s 6 8 10 5 Supervision of 4 4 3 4 graduate study 4 6 4 3 Student program 8 9 8 9 advising 10 12 6.5 12 Committee work 9 6.5 7 6 9 11 6.5 8 Length of service 11 11 11.5 10 at rank 7.5 10 11.5 10 Public recognition 7 8 5 7 7.5 5 5 6 Personality 6 12 10 11 Ch a r a c t e r i s t i c s 5 7 8.5 7 DMAs (bold) n = 45 42 33 44 Instructors (plain) n - 17 15 6 41 82 Without Tenure and second for Associate Professor Without Tenure and Appointments With Tenure. For Instructors, publications are f i r s t i n importance f o r Assistant Professor Without Tenure and Appointments with Tenure, second f o r appointment to Associate Professor Without Tenure , and t h i r d f o r Professor Without Tenure. Supervision of graduate study i s ranked t h i r d f o r Appointments With Tenure, fourth f o r appointments to Assistant or Associate Professor Without Tenure, and s i x t h f o r F u l l Professor Without Tenure. Table 23 reports the Spearman's rank c o r r e l a t i o n c o e f f i c i e n t s f o r the factors reported by rank i n Table 22. The p r o b a b i l i t y that each c o r r e l a t i o n c o e f f i c i e n t i s s i g n i f i c a n t l y d i f f e r e n t from zero i s also reported. The c o r r e l a t i o n c o e f f i c i e n t s are a l l s i g n i f i c a n t l y d i f f e r e n t from zero; the rank order f o r each type of appointment shows a moderate degree of agreement. Table 23: Spearman's Rank Correlation Coefficients for DMAs' and Instructors' Ranks of the Relative Importance of Factors Type of Appointment  Without Tenure With Assistant Associate F u l l Tenure 0.911 0.746 0.852 0.874 < .01 < .02 < .01 < .01 Comparison of the Relative Importance of the Sources of Information Table 24 summarizes the r e l a t i v e importance, based on mean response, of the sources of information considered i n teaching effectiveness 83 Spearman's Rho p (N = 12) Table 24: Ranking of Sources of Information Considered i n Evaluating Teaching Effectiveness Type of Appointment Without Tenure Information Source Assistant Associate F u l l With Tenure Formal Student Ratings Formal Peer Reviews 2 3 1 4 2 1 1 2 2 1 1 2 2 2.5 1 4 Informal student opinion Informal opinions of colleagues Evaluation by Head of Department Evaluation by Dean 8.5 10 7 5.5 5.5 1 10 5 11 12 7 12 6 3.5 8.5 6 10 12 7 12 6 3 8.5 8.5 10 10.5 7 7 6 2.5 9 5 Opinion of outside experts Self-evaluation or report Course s y l l a b i and examinations 3 6.5 8.5 9 4 8 3 5 8.5 7 4.5 9 3 8.5 8.5 11 5 8.5 3 6 8 10.5 Student examination performance Grade d i s t r i b u t i o n s 11 13 12.5 11 10 9 12.4 14 12.5 6 14 13.5 11.5 12.5 13 12.5 Enrollment i n e l e c t i v e classes 14 12 12.5 12 12.5 4.5 14 9 Long-terra follow-up of students 12.5 14 14 9 11 8.5 11.5 14 Research/Publication 5.5 2 4.5 3.5 4 4.5 4 1 DMAs (bold) n = 38 Instructors (plain) n = 11 38 12 32 6 39 36 84 evaluation f o r a l l types of appointments addressed i n the questionnaire; the r e s u l t s f o r both DMAs and Instructors are reported (DMA r e s u l t s are i n bold-face type). The r e l a t i v e importance, to DMAs, of three of the sources of information i s very consistent regardless of the type of appointment: the information source, formal peer reviews, i s ranked most important f o r a l l types; formal student ratings i s ranked second; and opinion of outside experts i s t h i r d . Research/publication i s ranked fourth f o r a l l appointments except Assistant Professor Without Tenure where i s ranked s i x t h . The r e l a t i v e importance of the sources of information f o r Instructors d i f f e r s from that of the DMAs. It can be seen from Table 24 that the r e l a t i v e importance of the factors varies according to the type of appointment. Formal peer review i s ranked second f or appointments to Associate and F u l l Professor Without Tenure and fourth f o r the remaining two types of appointments. Formal student ratings i s ranked f i r s t f o r Associate and F u l l Professor Without Tenure, second f o r Appointments With Tenure, and fourth f o r Assistant Professor Without Tenure. Opinion of outside experts, c o n s i s t e n t l y rated t h i r d by DMAs, i s rated f i f t h i n importance f o r Associate Professor Without Tenure, s i x t h f o r Assistant Professor Without Tenure and Appointments With Tenure, and seventh f o r F u l l Professor Without Tenure. Table 25 reports the Spearman's rank c o r r e l a t i o n c o e f f i c i e n t s f o r the sources of information ranked i n Table 24. The p r o b a b i l i t y that each c o r r e l a t i o n c o e f f i c i e n t i s s i g n i f i c a n t l y d i f f e r e n t from zero i s also reported. The c o r r e l a t i o n c o e f f i c i e n t f o r F u l l Professor Without Tenure i s not s i g n i f i c a n t l y d i f f e r e n t from zero (rho = 0.531, p > .05) i n d i c a t i n g 85 that the ranks reported i n Table 24, for t h i s type of appointment, do not show a s i g n i f i c a n t degree of agreement between groups; that i s , the rank order of the various sources of information appears to d i f f e r f o r DMAs and Instructors. Each of the other types of appointments show a moderate degree of agreement. Table 25: Spearman's Rank Correlation Coefficients for DMAs' and Instructors' Ranks of the Relative Importance of Sources of  Information Type of Appointment  Without Tenure With Assistant Associate F u l l Tenure Spearman's Rho 0.76 0.79 0.53 0.77 p (N = 14) < .01 < .01 > .05 < .01 RESPONDENTS' PERCEPTIONS OF THE PURPOSES OF STUDENT RATINGS Respondents were asked to indicate the extent to which they agreed with a number of purpose statements. Appendix G-9 reports the frequency of responses f o r each response category. A l l v a l i d responses were included i n the computation of the frequencies. The Hotelling's T 2 analysis was nonsignificant (Hotelling's T 2 = 0.95; p =0.47; df = 7, 84). Table 26 reports the mean response r a t i n g f o r each group. The DMAs mean responses of were used to rank order the purpose statements. Figure 11 shows the order of r e l a t i v e importance of the purpose statements based on DMAs' mean response. 86 Table 26: Mean Response Ratings for Purposes of Student Ratings of  Teaching Performance Mean A a Mean B b n=49 n=43 To provide information to in s t r u c t o r s f o r professional development To provide information to administrators to a i d i n personnel decisions 3.49 3.77 4.22 4.02 To provide students with information to use during the process of course and i n s t r u c t o r s e l e c t i o n 2.31 2.67 To provide information to a i d i n the a l l o c a t i o n of teaching or f a c u l t y resources 2.55 2.58 To a i d i n the awareness, s e n s i t i v i t y , and appreciation of teaching 3.02 2.95 To contribute to the understanding of the operation of the department or u n i v e r s i t y as a whole 2.45 2.44 To provide information f o r research on teaching 2.37 2.47 Hotelling's T 2 = 0.95 df = 7, 84 p = 0.47 *Mean A = Mean response DMAs bMean B = Mean response Instructors Note. Response Range: 1 (Strongly Disagree) to 5 (Strongly Agree) 87 c a 0) CO o Personnel Decisions Professional Development Awareness of Teaching Resource Allocation Course Selection Understand Operations Research on Teaching 1 2 - i — i — i — i — | — 4 5 - i 1 1 1 1 r • Instructors • DMAs _i i i_ _i i i i_ 2 3 4 Mean Response Figure 11: Purposes of student ratings ranked by DMAs' mean response. 88 In addition to showing a rank order of the importance of the statements, Figure 11 shows the mean response of both the DMAs and the Instructors. The dotted l i n e on the graph indicates the mid-point (3.0) of the response scale; the response a l t e r n a t i v e s ranged from 1 (Strongly Agree) to 5 (Strongly Disagree). It can be seen the purpose statement with the highest DMAs' mean response r a t i n g i s 'to provide information to administrators to a i d i n personnel decisions', followed by 'to provide information to ins t r u c t o r s f o r professional development'. The DMAs' mean response ratings f o r the remaining purpose statements a l l f e l l below the mid-point of the scale i n d i c a t i n g that, f o r the most part, the respondents do not f e e l that they r e f l e c t the purposes of student ratings. RESPONDENTS' SATISFACTION WITH STUDENT RATINGS Table 27 summarizes the respondents' ratings on a f i v e - p o i n t L i k e r t scale f o r two items; the f i r s t pertains to the degree with which student ratings r e f l e c t teaching performance and the second pertains to the value of student ratings i n personnel decisions. Hotelling's T 2 (Hotelling's T 2 = 3.75; p = 0.03; df = 2, 89) test indicated a s i g n i f i c a n t difference between groups. Post hoc univariate t - t e s t s showed a s i g n i f i c a n t d i f f e r e n c e between group means f o r the item asking them to indicate how s a t i s f i e d they are that the student ratings process accurately r e f l e c t s teaching performance. 89 Table 27: T-test Results for Value of Student Ratings Mean A" Mean B b Factor n = 50 n = 41 Refle c t teaching performance 2.67 3.24 -2.37 0.02 * Value to personnel decisions 3.12 3.15 -0.12 0.91 Hotelling's T 2 = 3.75 df = 2, 89 p = 0.03 "Mean A = Mean response DMAs bMean B = Mean response Instructors * s i g n i f i c a n t at alpha = .05 Note. Response Range: 1 (Strongly Disagree) to 5 (Strongly Agree) RESPONDENTS' RATINGS OF WEIGHT STATEMENTS Respondents were asked to respond to four questions having to do with the weight given to student ratings and to teaching effectiveness decisions. Their mean responses are reported i n Table 28. The Hotelling's T 2 (Hotelling's T 2 = 0.88; p = 0.48; df = 4, 82) analysis was not s i g n i f i c a n t i n d i c a t i n g that DMAs and Instructors responded i n the same manner to these questions. Table 28 shows that both the DMAs and the Instructors f e e l that the weight that should be and that a c t u a l l y i s given to student ratings i n teaching effectiveness evaluation i s between 41 to 50%. Both groups of respondents indicated that the weight that should be given to teaching effectiveness i n o v e r a l l performance evaluation l i e s between 41 and 50%, but that the weight a c t u a l l y given to teaching l i e s somewhere between 21 and 30%. 90 Table 28: Mean Responses to Weighting Questions Mean A" SD Mean B b SD n = 47 n = 40 Weight that should be given to Student Ratings 6. .11 2. .36 6. .80 2. .41 Actual weight given to Student Ratings 6. .83 2. .64 6. .15 2, .71 Weight that should be given to Teaching Performance 6. .15 2. .27 6. .43 2, .12 Actual weight given to Teaching Performance 4. .85 2. .09 4. .45 2. .22 Hotelling's T 2 = 0.88 p = 0.48 df = 4, 82 *Mean A - Mean response DMAs; bMean B = Mean response Instructors Note. Response Categories: 1 = 0 % ; 2 = 1 - 10%; 3 = 11 - 20%; 4 = 21 -30%; 5 = 31 - 40%; 6 = 41 - 50%; 7 = 51 - 60%; 8 = 61 - 70%; 9 = 71 - 80%; 10 = 81 - 90%; 11 = 91 - 100% RESPONDENTS' COMMENTS Respondents were provided with the opportunity to write down any add i t i o n a l comments they had regarding the ways i n which student ratings are used f o r purposes of personnel decisions i n the Faculty of Education. A t o t a l of 38 respondents wrote comments; 25 DMAs and 13 Instructors. Several respondents chose to make comments about more than one issue r e s u l t i n g i n a t o t a l of 54 comments. The comments f e l l into f i v e categories: 1) the use made of student ratings by administrators; 2) the 91 importance of teaching i n o v e r a l l performance evaluation; 3) the l i m i t a t i o n s of student ratings; 4) the need f or formative evaluations f o r professional development; and 5) comments concerning the research questionnaire. Table 29 reports the frequency of comments i n each category. Table 29: Frequency of Respondents' Comments by Category Comment DMAs Instructors n" Percent n" Percent Use made of student ratings by administrators 6 17.14 3 15.79 Importance of teaching i n personnel decisions 7 20.00 5 26.32 Limitations of student ratings 16 45.72 6 31.57 Need f o r formative evaluations of teaching 3 8.57 5 26.32 Comments about research questionnaire 3 8.57 0 00.00 Total 35 100.00 19 100.00 "This value includes multiple responses provided by respondents The majority of the comments had to do with the l i m i t a t i o n s of student ratings of teaching effectiveness as they are c o l l e c t e d i n the Faculty of Education. Several DMAs suggested that the student ratings form i s not applicable to a l l teaching s i t u a t i o n s i n the Faculty. For example, one DMA wrote, "SCET presumes one kind of teaching, a sort of standard version that i s not applicable to many course s i t u a t i o n s and teaching s t y l e s . " Other DMAs expressed concern with the v a l i d i t y of the 92 instruments and the e f f e c t of teacher personality on student ratings. The Instructors comments r e f l e c t the same concerns as those of the DMAs. The next most popular category of comment, expressed by both DMAs and Instructors, concerned the importance of teaching effectiveness i n o v e r a l l performance evaluation. Typical of the view of many of the DMAs who wrote comments i s t h i s : It has been my view that i n pers. [ s i c ] decisions i n our department we use the SCET ratings to ensure there i s a modicum of teaching a b i l i t y (acceptable) but that's i t . We don't r e a l l y care about the teaching nearly as much as we do about publications (the number of them) and a b i l i t y to a t t r a c t research grants. My view i s teaching ratings only account f o r 10% of the weight i n these decisions - I'd l i k e to see that increased. A s i m i l a r view was expressed by an Instructor who wrote: I believe that teaching performance should be an important c r i t e r i o n i n the promotion of f a c u l t y . However, i n my experience, teaching performance, as judged by students and peers, seems to be of l i t t l e r e a l importance. Publications and grants are a l l that matter. A number of respondents expressed concern over the use made of student ratings by administrators. One of the DMAs wrote, "In our dept [si c ] student ratings play a key role and I would l i k e to see t h i s continue."; however, the majority of the comments were l e s s p o s i t i v e . For example, one DMA wrote, "The problem i s not r e a l l y with the student ratings process as such. It i s i n the way our administrators use t h i s information!"; another one wrote, " I t has been my experience that the weighting often varies depending upon the i n d i v i d u a l being considered." The f i n a l category of comments expressed the need f o r some sort of formative evaluation f o r the purposes of professional development. For example, one DMA wrote "Student evaluations should also help teachers improve t h e i r performance. The SCET instrument i s l i t t l e value here." An 93 Instructor wrote "It seems to me that the current system does not (nor can i t ) contribute to the professional development of instructors". 9 4 CHAPTER V. DISCUSSION AND CONCLUSIONS SUMMARY OF THE FINDINGS The r e s u l t s are summarized here f o r each of the research questions. Question 1 a) What factors are considered important i n a candidate's o v e r a l l performance evaluation f o r the purposes of reappointment, promotion, and tenure by the people who p a r t i c i p a t e i n personnel decisions i n the Faculty of Education? The most important factor i n o v e r a l l performance evaluation f or purposes of reappointment, promotion, and tenure appears to be research a c t i v i t y ; t h i s f a c t o r i s considered to be most important regardless of the type of appointment a candidate i s being considered f o r . Other factors which appear to be considered to be of importance are publications, classroom teaching and supervision of graduate study. The r e l a t i v e importance of these factors seems to vary according to the type of appointment a candidate i s being considered f o r . Question 1 b) What i s the r e l a t i v e importance of teaching effectiveness among factors u t i l i z e d i n a candidate's o v e r a l l performance evaluation? The r e l a t i v e importance of teaching effectiveness evaluation i n o v e r a l l performance evaluation appears to depend upon the type of 95 appointment a candidate i s being considered f o r . For appointments without tenure, the importance of classroom teaching decreases as the rank of the appointment increases. In other words, teaching effectiveness appears to play a l e s s e r r o l e when a candidate i n being considered f o r Professor Without Tenure than when s/he i s being considered f o r Assistant Professor Without Tenure. In cases where the candidate i s being considered f o r a tenured p o s i t i o n , the r e s u l t s indicate that research a c t i v i t y and publications are both considered to be more important than classroom teaching. When the decision-makers were asked to indicate the weight they f e l t i s a c t u a l l y given to teaching i n o v e r a l l performance evaluation, t h e i r mean response ratings indicated that i t i s assigned a weight of between 21 to 30%. Question 2 a) What sources of information are considered important i n teaching effectiveness evaluation by those who p a r t i c i p a t e i n personnel decisions i n the Faculty of Education? The three sources of information that appear to be most important to the people who p a r t i c i p a t e i n personnel decisions, i n order of r e l a t i v e importance, are: 1) formal peer reviews; 2) formal student ratings; and 3) opinion of outside experts. These three sources of information hold the same r e l a t i v e importance i n personnel decisions regardless of the type of appointment a candidate i s being considered f o r . Based on the mean response ratings of the DMAs, the other sources of information presented i n the questionnaire c l e a r l y appear to be l e s s important. 96 Question 2 b) What i s the r e l a t i v e importance of student ratings of teaching effectiveness among sources of information considered i n a candidate's teaching effectiveness evaluation? Based on the mean response ratings of those who p a r t i c i p a t e i n personnel decisions, formal peer reviews appear to be the most important source of information considered when a candidate i s being evaluated f o r teaching effectiveness. Student ratings appear to be the second most important source of information. The actual weight given to student ratings i s d i f f i c u l t to a s c e r t a i n from t h i s study. The DMAs indicated that the weight that i s a c t u a l l y given to student ratings i n teaching effectiveness evaluation l i e s somewhere between 51 to 60%. However, i f t h i s i s the case then student ratings would have to have a higher r e l a t i v e importance when compared to other sources of information. Given the consistency with which student ratings were indicated as the second most important source of information across type of appointment decisions, i t must be concluded that the method used to as c e r t a i n the weight accorded student ratings were not appropriate. Question 2 c ) Does the importance of student ratings i n teaching effectiveness evaluation vary according to the type of appointment a candidate i s being considered for? The r e l a t i v e importance of student ratings i n teaching effectiveness evaluation does not appear to vary according to the type of appointment a candidate i s being considered f o r ; student ratings were rated second i n 97 importance, behind formal peer reviews, as a source of information i n teaching effectiveness evaluation by the people who p a r t i c i p a t e i n personnel decisions i n the Faculty of Education f o r a l l four types of personnel decisions addressed i n the questionnaire. Question 3. How do Instructors' perceptions of the role of student ratings i n personnel decisions compare with the perceptions of those who p a r t i c i p a t e i n personnel decisions? Instructors were asked to indicate how they f e l t the people who pa r t i c i p a t e i n personnel decisions t y p i c a l l y considered a number of factors and a number of sources of information i n personnel decisions. The responses of the Instructors showed a f a i r degree of agreement with the DMAs. However, the Instructors' responses appear to indicate that t h e i r perceptions of what personnel decision-makers think are important and what the decision-makers indicate to be important to them d i f f e r i n several ways. These differences are e s p e c i a l l y apparent f o r those factors or sources of information deemed r e l a t i v e l y l e s s important o v e r a l l . However, an important exception i s the fac t o r "classroom teaching" f o r which the group mean response ratings were s i g n i f i c a n t l y d i f f e r e n t f o r appointments to Assistant Professor Without Tenure and Appointments With Tenure. In both cases, the DMAs had a higher mean response r a t i n g and a higher r e l a t i v e importance f o r these factors than d i d the Instructors. For these two types of appointments, DMAs f e l t classroom teaching was a more important f a c t o r than d i d the Instructors. 98 Of the sources of information considered by DMAs to be most important, a s i g n i f i c a n t difference was found f o r the information source formal peer review. Instructors rated peer reviews f o r Appointments With Tenure lower i n importance than d i d the DMAs. Question 4. What do DMAs and Instructors perceive to be the purposes of student ratings of teaching effectiveness? The main purpose, indicated by respondents, for c o l l e c t i n g student ratings i s to provide information to administrators to a i d i n personnel decisions. A second but l e s s important purpose i s to provide information to i n s t r u c t o r s for professional development. A number of respondents indicated i n t h e i r comments that they f e l t that the present student ratings process i s l i m i t e d as a form of formative evaluation, and that there was a need f o r a system which would contribute to the professional development of the i n s t r u c t o r s . DIRECTIONS FOR FUTURE STUDY The intent of t h i s study was to explore the role of student ratings of teaching effectiveness i n personnel decisions. As a r e s u l t of the findings several questions have been generated to guide future research. 99 1. How i s the information provided by student ratings of teaching effectiveness used i n personnel decisions? This study has explored the r e l a t i v e importance of student ratings i n teaching effectiveness evaluation. However, the r e l a t i v e order of importance does not indicate how they are u t i l i z e d as an information source during the decision-making process. Future research should investigate t h i s issue. 2. What elements of the student ra t i n g s ' summary report are u t i l i z e d by the decision-makers during the personnel decision-making process? The student ra t i n g s ' summary report provided to the decision-makers consists of several d i f f e r e n t elements. Future research should investigate and i d e n t i f y which of theses elements are u t i l i z e d by the decisions-makers during the decisions-making process. 3. Does the way i n which student ratings are used vary according to the circumstances? There are a number of circumstances which could a f f e c t the way i n which student ratings are used i n personnel decisions. Some of these circumstances are; the type of appointment a candidate i s being considered f o r , the department the candidate i s a member of and, the l e v e l of the committee making the recommendation. Future research should investigate and determine i f the circumstances surrounding the de c i s i o n a f f e c t they way i n which student ratings are used. 100 4. What i s the importance of student ratings i n university-wide personnel decisions? In any i n s t i t u t i o n where the dec i s i o n processes are m u l t i - t i e r e d i t would be of value to determine i f the c r i t e r i a used i n the preliminary committees are the same . as those used i n the upper-most l e v e l s of the personnel decision-making process. As the number of people who p a r t i c i p a t e i n university-wide personnel decisions i s few, i t would be p r a c t i c a l to conduct personal interviews with these people i n order to determine the importance of student ratings i n personnel decisions as they occur on a university-wide committee. 5. How do members of the Faculty of. Education use the r e s u l t s of student ratings f o r professional development? One of the recognized purposes, i n the l i t e r a t u r e at large, of student ratings i s f o r professional development to improve teaching. Future research should investigate how the members of the Faculty of Education use the summary reports f o r professional development. 6. How informed are the members of the Faculty of Education with regard to the mandate of the Teacher Evaluation Service? Future research should investigate and determine the degree to which Faculty are f a m i l i a r with the p o l i c i e s , purposes, and procedures of the Teacher Evaluation Service. Some respondents indicated d i s s a t i s f a c t i o n with the student ratings process. This d i s s a t i s f a c t i o n may be due to a lack of f a m i l i a r i t y with the p o l i c i e s , purposes, and procedures of the Teacher Evaluation Service. The Administration could use the r e s u l t s of 101 such research to determine what type of ad d i t i o n a l information should be provided to the Faculty with regard to the mandate of the TES. LIMITATIONS OF THE STUDY There are several methodological issues, including the choice of method of data c o l l e c t i o n , possible response bias, lack of a p r o b a b i l i t y sample, g e n e r a l i z a b i l i t y of the r e s u l t s , and the conclusions which can be drawn from the r e s u l t s which, require c a r e f u l consideration. The f i r s t l i m i t a t i o n of the study l i e s i n the choice of a mailed survey questionnaire to c o l l e c t the data. There are a number of problems inherent i n the use of a questionnaire. Berdie and Anderson (1974) suggest that "owing to the nature of questionnaires, the ways to check the r e l i a b i l i t y and the v a l i d i t y of questionnaire items are l i m i t e d " (p. 20). One way to address the problems of r e l i a b i l i t y and v a l i d i t y i s to conduct a p i l o t study. However, i n the case of t h i s study, where the intent was to do a population-wide survey of a small circumscribed population, i t was concluded that conducting a p i l o t study f o r purposes of instrument refinement could severely reduce the target population. Therefore, expert opinion was sought f o r feedback about the q u a l i t y of the questionnaire. Another problem associated with the use of a survey questionnaire i s the problem of response bias. Although the o v e r a l l response rate of 73% i s considered to be s a t i s f a c t o r y f o r a mailed questionnaire, there i s a p o s s i b i l i t y of response bias. When the rate of response of each group i s examined, i t can be seen that 68% of the DMAs responded compared with 80% of the Instructors. Although the Chi-square value used to tes t the 102 representativeness of the sample was not s i g n i f i c a n t , the f a c t that a smaller proportion of DMAs responded must be taken into consideration when examining the r e s u l t s of the survey, e s p e c i a l l y when the response rate of DMAs by department i s considered. Examination of the rate of response of the DMAs by department shows that Social and Educational Studies department had only a 32% response rate, i n d i c a t i n g that members of t h i s department are not f a i r l y represented i n the sample. This low response rate on the part of SEDS may be due, i n part, to the f a c t that a l l members of the department were c l a s s i f i e d as DMAs. Some of the people with lower s e n i o r i t y may have ref r a i n e d from completing the questionnaire because they were not f a m i l i a r with a l l aspect of personnel decision-making. The data were c o l l e c t e d during the month of June; a time when many f a c u l t y members do not keep regular o f f i c e hours. It i s possible that the o v e r a l l response rate would have been higher had the questionnaires been d i s t r i b u t e d at another time of the year. The present study was l i m i t e d to the Faculty of Education at the Unive r s i t y of B r i t i s h Columbia and, as such, the r e s u l t s of t h i s study are l i m i t e d i n t h e i r g e n e r a l i z a b i l i t y . The r e s u l t s reported p e r t a i n only to the personnel decision-making processes as they occur within the Faculty of Education as indicated by members of the Departmental and Faculty Standing Personnel Committees. The personnel decision-making process as i t occurs on a university-wide basis has not been investigated. This study has i d e n t i f i e d the r e l a t i v e importance of student ratings among sources of information u t i l i z e d i n personnel decisions but the actual weight given to student ratings has not been ascertained. 103 In conclusion, the r e s u l t s of t h i s exploratory study indicate that student ratings do play a ro l e i n personnel decision-making processes i n the Faculty of Education. They are rated second i n r e l a t i v e importance behind formal peer reviews as a source of information about teaching effectiveness. In order to get a cl e a r e r picture of the importance of student ratings i n personnel decisions i t i s necessary to understand the r e l a t i v e importance of teaching effectiveness i n o v e r a l l personnel evaluation. While teaching effectiveness i s an important f a c t o r i n personnel decisions i n the Faculty of Education, research a c t i v i t y appears to be of primary importance f o r a l l types of appointments. The r e l a t i v e importance of teaching effectiveness appears to vary according to the type of appointment being considered. I t ' s importance tends to decrease as a candidate moves up i n rank. Hence, the r e l a t i v e importance of student ratings of teaching effectiveness i n o v e r a l l performance evaluation also appears to decrease as a candidate moves up i n rank. 104 REFERENCES Abrami, P. C , Dickens, W. J., Perry, R. P., & Leventhal, L. (1980). Do teacher standards f o r assigning grades a f f e c t student evaluations of instruction? Journal of Educational Psychology. 72. 107-118. Abrami, P. C., Perry, R. P., & Leventhal, L. (1982). The r e l a t i o n s h i p between student personality c h a r a c t e r i s t i c s , teacher ratings, and student achievement. Journal of Educational Psychology. 74 (1), 111-125. Aleamoni, L. M. (1981). Student ratings of i n s t r u c t i o n . In J. Millman (Ed.), Handbook of teacher evaluation, (pp. 110-145). Beverley H i l l s , C a l i f . : Sage Publications. Aleamoni, L. M. (1987a). Typical f a c u l t y concerns about student evaluation of teaching. In L. Aleamoni (Ed.), , Techniques for  evaluating and improving i n s t r u c t i o n , (pp. 25-32). San Francisco: Jossey-Bass Inc. Aleamoni, L. M. (1987b). Some p r a c t i c a l approaches f o r f a c u l t y and administrators. In L. Aleamoni (Ed.), Techniques for evaluating and  improving i n s t r u c t i o n , (pp 75-781. San Francisco: Jossey-Bass Inc. Arreola, R. A. (1987a). A f a c u l t y evaluation model f o r community and junior colleges. In L. Aleamoni (Ed.), Techniques f o r evaluating  and improving instruction.(pp. 65-74) t San. Francisco: Jossey-Bass Inc. Arreola, R. A. (1987b). The r o l e of student government i n f a c u l t y evaluation. In L. Aleamoni (Ed.), Techniques for evaluating and  improving instruction.(pp. 39-46). San Francisco: Jossey-Bass Inc. Aubrecht, J . D. (1984). Better f a c u l t y evaluation systems. In P. Seldin (Author), Changing practices i n f a c u l t y evaluation.(pp. 85-91). San Francisco, C a l i f . : Jossey-Bass. Berdie, D. R. , & Anderson, J. F. (1974) Questionnaire: Design and use. Metuchen, N. J.: The Scarecrow Press, Inc. 105 Braskamp, L. A., Caulley, D., & Costin, F. (1979). Student ratings and i n s t r u c t o r s e l f - r a t i n g s and t h e i r r e l a t i o n s h i p to student achievement. American Education Research Journal. 16/3), 295-306 Centra, J. A. (1979). Determining f a c u l t y e ffectiveness: assessing  teaching, research, service f o r personnel decisions and improvement. San Francisco: Jossey-Bass Centra, J. A. (1987). Formative and summative evaluation: Parody or paradox? In L. Aleamoni (Ed.), Techniques f o r evaluating and  improving instruction.(pp. 47-56). San Francisco: Jossey-Bass Inc. Centra, J. A. (Ed.). (1977). Renewing and evaluating teaching: New  d i r e c t i o n s f o r higher education. San Francisco, C a l i f . : Jossey-Bass , Inc. Centra, J. A. (1977). Student ratings of i n s t r u c t i o n and t h e i r r e l a t i o n s h i p to student learning. American Educational Research  Journal. 14(1), 17-24. Chickering, A. W. (1984). Faculty evaluation: Problems and solutions. In P. Seldin (Author), Changing practices i n f a c u l t y evaluation, (pp. 91-96). San Francisco, C a l i f . : Jossey-Bass. Cohen, P. A. (1981). Student ratings of i n s t r u c t i o n and student achievement: A meta-analysis of multisection v a l i d i t y studies. Review of Educational Research, 51(3), 281-309. Costin, F., Greenough, W. T., Menges, R. J. (1971). Student ratings of college teaching: r e l i a b i l i t y , v a l i d i t y and usefulness. Review of  Educational Research. 41(5), 511-535. Crittenden, K. S., Norr, J. L., & L e B a i l l y , R. K. (1975). Size of u n i v e r s i t y classes and student evaluations of teaching. Journal of  Higher Education. 46, 461-470. DuCette, J., & Kenney, J. (1982). Do grading standards a f f e c t student evaluations of teaching? Some new evidence on an old question. Journal of Educational Psychology. 74(3), 308-314. 106 Erdle, S., Murray, H. G., & Rushton, J. P. (1985). Personality, classroom behavior, and student ratings of college teaching effectiveness: A path a n a l y s i s . Journal of Educational Psychology. 77(4), 394-407. Feldman, K. A. (1984). Class si z e and college students' evaluations of teachers and courses: A closer look. Research i n Higher Education. 21(1), 45-91 Feldman, K. A. (1986). The perceived i n s t r u c t i o n a l effectiveness of college teachers as r e l a t e d to t h e i r personality and a t t i t u d i n a l c h a r a c t e r i s t i c s : A review and synthesis. Research i n Higher  Education. 24(2), 139-213. Ferguson, G. A. (1981). S t a t i s t i c a l analysis i n psychology and  education. New York: McGraw-Hill, Inc. Frey, P. W., Leonard, D. W., & Beatty, W. W. (1975). Student ratings of i n s t r u c t i o n : V a l i d a t i o n research. American Educational Research  Journal. 12(4), 435-447. Geis, G. L. (1984). The context of evaluation. In P. Seldin (Author), Changing practices i n f a c u l t y evaluation, (pp. 101-107). San Francisco, C a l i f . : Jossey-Bass. G i l , D. H. (1987). I n s t r u c t i o n a l evaluation as a feedback process. In L. Aleamoni (Ed.), Techniques f o r evaluating and improving  instruction.(pp. 57-64). San Francisco, C a l i f : Jossey-Bass Inc. Gillmore, G. M. (1984). Student ratings as a fa c t o r i n f a c u l t y employment decisions and periodic review. Journal of College and  University Law. 10(4), 557-576. Howard, G. S., Conway, C. G. , & Maxwell, S. E. (1985). Construct v a l i d i t y of measures of college teaching effectiveness. Journal of  Educational Psychology. 77(2), 187-196. Kennedy, W. R. (1975). Grades expected and grades received: Their r e l a t i o n s h i p to students' evaluations of f a c u l t y performance. Journal of Educational Psychology. 67.(1), 109-115. Kirk, R. E. (1978). Introductory s t a t i s t i c s . Monterey, C a l i f . : Brooks/Cole Publishing Co. 107 L i n , Y. G., McKeachie, W. J., & Tucker, D. G. (1984). The use of student ratings i n promotion decisions. Journal of Higher Education. 55(5) . 583-589 McCallum L. W. (1984). A meta-analysis of course evaluation data and i t s use i n the tenure decision. Research i n Higher Education. 21(2), 150-158. Marsh, H. W. (1983). Multidimensional ratings of teaching effectiveness by students from d i f f e r e n t academic settings and t h e i r r e l a t i o n to student/course/instructor c h a r a c t e r i s t i c s . Journal of Educational  Psychology. 75(1), 150-166. Marsh, H. W. (1984). Students' evaluations of u n i v e r s i t y teaching: Dimensionality, r e l i a b i l i t y , v a l i d i t y , p o t e n t i a l biases, and u t i l i t y . Journal of Educational Psychology. 76. 707-754. Marsh, H. W. (1982). V a l i d i t y of students' evaluations of college teaching: A multitrait-multimethod analysis. Journal of Educational Psychology. 74(2), 264-269. Marsh, H. W., F l e i n e r , H., & Thomas, C. S. (1975). V a l i d i t y and usefulness of student evaluations of i n s t r u c t i o n a l q u a l i t y . Journal  of Educational Psychology. 67(6), 833-839. Marsh , H. W., Over a l l , J. U., & Kesler, S. P. (1979). Class s i z e , students' evaluations, and i n s t r u c t i o n a l effectiveness. American  Educational Research Journal. 16(1), 57-69. Meier R. S. & Feldhusen, J. F. (1979). Another look at Dr. Fox: E f f e c t of stated purpose f or evaluation, lecture expressiveness, and density of lecture content on student ratings. Journal of  Educational Psychology. 71(3), 339-345. Moomaw, W. E. (1977). Practices and problems i n evaluating i n s t r u c t i o n . In J. Centra (Ed.), Renewing and evaluating teaching: New  di r e c t i o n s f o r higher education. San Francisco, C a l i f . : Jossey-Bass, Inc. Murray, H. G. (1979). Student evaluation of u n i v e r s i t y teaching: Uses  and abuses (Report No. 5). Uni v e r s i t y of B r i t i s h Columbia, Centre fo r the Improvement of Teaching and Evaluation. 108 Myers, J. L. (1979) Fundamentals of experimental design. Boston, Mass.: A l l y n and Bacon, Inc. N a f t u l i n , D. H., Ware, J. E., J r . , & Donnelly, F. A. (1973). The Doctor Fox l e c t u r e : A paradigm of educational seduction. Journal of  Medical Education. 48, 630-635. Ondrack, D. A. & O l i v e r , C. (1986). A review and analysis of performance  appraisal processes: Volumes 1 to 3. Toronto: M i n i s t r y of Education. Perry, R. P., Abrami, P. C , & Leventhal, L. (1979). Educational seduction: The e f f e c t of i n s t r u c t o r effectiveness and lecture content on student ratings and achievement. Journal of Educational  Psychology. 71(1), 107-116. Rippey, R. M. (1975). Student evaluations of professors: Are they of value? Journal of Medical Education. 50, 951-958. Scriven, M. (1981). Summative teacher evaluation. In'-J. Millman (Ed.), Handbook of teacher evaluation, (pp. 110-145). Beverley H i l l s , C a l i f . : Sage Publications. Salthouse, T. A., McKeachie, W. J., & L i n . Y. G. (1978). An experimental i n v e s t i g a t i o n of factors a f f e c t i n g u n i v e r s i t y promotion decisions. Journal of Higher Education. 49(2), 177-183. Seldin, P. (1984). Changing practices i n f a c u l t y evaluation. San Francisco: Jossey-Bass. Stake, R. (1987). The evaluation of teaching on campus. Unpublished manuscript. Stevens, J. J. (1987). Using student ratings to improve i n s t r u c t i o n . In L. Aleamoni (Ed.), Techniques f o r evaluating and improving  i n s t r u c t i o n . (pp. 83-92). San Francisco: Jossey-Bass Inc. Timm, N. H. (1975). Multivariate Analysis: With applications i n  Education and Psychology. Monterey, C a l i f . : Brooks/Cole Publishing Co. 109 U n i v e r s i t y of B r i t i s h Columbia Faculty Handbook. (1985). Faculty Association, U n i v e r s i t y of B r i t i s h Columbia. Univ e r s i t y of B r i t i s h Columbia Senate Records. (1974). Uni v e r s i t y of B r i t i s h Columbia, The Library, Special C o l l e c t i o n s D i v i s i o n , Box 6, F i l e 6-3. Univ e r s i t y of B r i t i s h Columbia Senate Records. (1978). Uni v e r s i t y of B r i t i s h Columbia, The Library, Special C o l l e c t i o n s D i v i s i o n , Box 7, F i l e 7-4. Uni v e r s i t y of B r i t i s h Columbia Senate Records. (1983). Uni v e r s i t y of B r i t i s h Columbia, The Library, Special C o l l e c t i o n s D i v i s i o n , Box 8, F i l e 8-4. Ware, J. E., J r . , & Williams, R. G. (1975). The Dr. Fox e f f e c t : A study of l e c t u r e r effectiveness and ratings of i n s t r u c t i o n . Journal of  Medical Education. 50, 149-156. Wilkinson, Leland. (1988) SYSTAT: The system f o r s t a t i s t i c s [Computer program]. Evanston, I I : SYSTAT, Inc. Williams, R. G, & Ware, J. E., J r . (1977). An extended v i s i t with Dr. Fox: V a l i d i t y of student s a t i s f a c t i o n with i n s t r u c t i o n ratings a f t e r repeated exposures to a l e c t u r e r . American Educational  Research Journal. 14(4), 449-457. Williams, R. G, & Ware, J. E., J r . (1976). V a l i d i t y of student ratings of i n s t r u c t i o n under d i f f e r e n t incentive conditions: A further study of the Dr. Fox e f f e c t . Journal of Educational Psychology. 68(1), 48-56. 110 APPENDIX A Instructions f o r administering the student evaluation of teaching questionnaires. Faculty of Education Annual teaching evaluation - common form. Faculty of Education Annual Teaching Evaluation - VPAE form. Faculty of Education Annual Teaching Evaluation - TEP i n s t r u c t o r form. Faculty of Education Annual Teaching Evaluation - TEP course-wide form. Faculty of Education Annual Teaching Evaluation - TEP laboratory form. Faculty of Education summary report Annual Teaching Evaluation 111 INSTRUCTIONS FOR ADMINISTERING THE STUDENT EVALUATION OF TEACHING QUESTIONNAIRES TO THE INSTRUCTOR: Enclosed are copies of the student questionnaire to be used as part of the annual evaluation of your classroom teaching. As you are aware, this instrument is one of a set of procedures used by our Faculty to evaluate the teaching of instructional staff. This questionnaire is to be completed IN THE ABSENCE OF THE INSTRUCTOR. Allow fifteen minutes at the end of a class period for the students to complete i t . Please appoint a student to administer the questionnaire. TO THE STUDENT ADMINISTRATOR: You are asked to ensure that ALL questionnaires are collected and returned to: SCET OFFICE, SCARFE ANNEX ROOM 24 or FACULTY MAIL ROOM, c/o SCET. RETURN ALL unused questionnaires; use the envelope provided. DO NOT RETURN THESE  MATERIALS TO THE INSTRUCTOR. If you experience any di f f i c u l t y , please contact the SCET Office, or Dr. R. Conry, In administering the forms, please: ~ — - -1. Distribute the questionnaires. 2. Write the instructor's name, the course number, and the section number on the chalkboard. 3. Read the instructions given below, to the class. 4. When the class has finished, collect the questionnaires, including unused ones,- seal them in the return manila envelope, and return them as indicated above. . READ THIS ALOUD AFTER QUESTIONNAIRES ARE DISTRIBUTED: INSTRUCTIONS 1. The questionnaire you have just received is used by the Faculty of Education to assess student opinions on the quality of faculty teaching. DO NOT write your name anywhere on this document; i t i s to be anonymous. 2. The accuracy of your responses to these items w i l l determine how precisely the instructor's teaching can be judged. Please take this task seriously and consider each item carefully. Make the most accurate judgments you can. You are asked for an independent assessment, so please DO NOT discuss the items with other students. 3. The questionnaire i t s e l f has i t s own instructions. Please read them care-ful l y before attempting to answer the items. In particular, make sure you recognize that the low end of the scale (1) means you disagree with an item, while the top end of the scale (7) means you agree strongly with the item. NOTE also that a "zero" response is available. Use this response  i f you believe the item does not apply to this particular class. 4. The "student background information" on the front page is optional, but please be aware this this information is very helpful in determining the pattern of an instructor's evaluation. 5. Use either pen or pencil, but please be sure that each item is marked clearly. SCET/06/86 112 FACULTY OF EDUCATION ANNUAL TEACHING EVALUATION Instructor's Name. Course # Section # S T U D E N T B A C K G R O U N D I N F O R M A T I O N D a t e o f r e s p o n d i n g t o t h i s q u e s t i o n n a i r e : . Y o u r a g e : . m o n t h y e a r . Y o u r s e x : y e a r s . I n y o u r c a s e t h i s c o u r s e i s : r e q u i r e d I I e l e c t e d . W h i c h o f t h e f o l l o w i n g i s most d e s c r i p t i v e o f y o u r p r o g r a m o f s t u d i e s ? ( c h e c k one o p t i o n ) L~D A r e g u l a r B . E d , p r o g r a m . O A g r a d u a t e t r a n s f e r ( S t h y e a r ) p r o g r a m . L~Z] 0 A q u a l i f y i n g p r o g r a m . CZ] W h a t year a r e y o u c u r r e n t l y r e g i s t e r e d i n ? ( c h e c k one o p t i o n ) d m n « h • a n d • S t h ( B . E d . ) • 3 r d • S t h ( t r a n s f e r ) O t h e r • • • • M a l e • F e m a l e A M a s t e r s o r D o c t o r a l p r o g r a m . D i p l o m a M a s t e r s / D o c t o r a l O t h e r In t h e F a c u l t y o f E d u c a t i o n , s t u d e n t s a r e a s k e d t o e v a l u a t e t h e q u a l i t y o f i n s t r u c t i o n i n e v e r y c o u r s e . I n r e s p o n d i n g t o t h e i t e m s g i v e n h e r e , p l e a s e k e e p i n m i n d t h a t y o u a r e a s k e d t o r a t e t h e I n s t r u c t o r a n d t h o s e a s p e c t s o f t h e c o u r s e o v e r w h i c h h e o r s h e c l e a r l y h a s c o n t r o l . P l e a s e d o n o t e v a l u a t e o t h e r r e l a t e d f a c t o r s , s u c h a s d e p a r t m e n t o r f a c u l t y p o l i c i e s . T h i s i s t o b e a n e v a l u a t i o n o f t h e i n -d i v i d u a l i n s t r u c t o r ' s t e a c h i n g a n d r e l a t e d a c t i v i t i e s a n d o u t c o m e s . Consider ing all facets of this Instructor's teaching, I would rate It as: ( C H E C K ONE O F T H E B O X E S S H O W N B E L O W ) 7 • E x c e l l e n t S • G o o d 3 • • V e r y g o o d 4 • J u s t a d e q u a t e • L e s s t h a n a d e q u a t e . . 1 U P o o r I n c o m p e t e n t F o r t h e r e m a i n i n g i t e m s , a s e v e n - p o i n t r a t i n g s c a l e i s p r o v i d e d f o r y o u t o r e c o r d y o u r r e s p o n s e s . P l e a s e i n t e r p r e t i t t o h a v e t h e f o l l o w i n g m e a n i n g : 1 2 3 4 5 6 7 D I S A G R E E D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y A l t h o u g h t h i s s c a l e c a n b e a p p l i e d i n a g e n e r a l w a y t o t h e f o l l o w i n g 3 0 I t e m s , it m a y n o t b e g r a m m a t i c a l l y p r e c i s e i n a f e w c a s e s . P l e a s e k e e p i n m i n d t h a t a r a t i n g o f ' 7 ' i s A L W A Y S i n t e n d e d t o r e f l e c t a n e x t r e m e l y F A V O U R A B L E r e a c t i o n t o I n s t r u c t i o n In y o u r c o u r s e , a n d a r a t i n g o f ' 1 ' i s A L W A Y S I n t e n d e d t o r e f l e c t a n e x t r e m e l y U N F A V O U R A B L E r e a c t i o n . If y o u w i s h t o u s e a s c a l e p o i n t w h i c h d o e s n o t h a v e a l a b e l a b o v e — e . g . , a ' 6 ' b e c a u s e y o u f e e l t h a t y o u f i t s o m e w h e r e b e t w e e n ' a g r e e s o m e w h a t ' a n d ' a g r e e v e r y s t r o n g l y , ' p l e a s e f e e l f r e e t o d o s o . If y o u f e e l t h a t a n y i t e m i s t o t a l l y I r r e l e v a n t t o t h e q u a l i t y o f I n s t r u c t i o n In y o u r c o u r s e , c i r c l e t h e ' 0 ' ( z e r o ) o p t i o n p r o v i d e d . T h i s m a y o c c u r , f o r e x a m p l e , i n t h e c a s e o f a n I t e m w h i c h a s k s a b o u t e x a m s i n a c o u r s e w h i c h h a s n o e x a m s . P L E A S E N O T E t h a t t h e ' z e r o ' o p t i o n i s n o t t h e s a m e a s a n e u t r a l r e a c t i o n t o i n s t r u c t i o n , w h i c h w o u l d b e i n d i c a t e d b y t h e ' 4 ' r e s p o n s e . P L E A S E T U R N T H E P A G E A N D C O N T I N U E Y O U R R A T I N G 113 R E M E M B E R , C I R C L E "0" ONLY IF Y O U THINK T H E ITEM IS IRRELEVANT T O THIS C O U R S E . O T H E R W I S E U S E THIS S C A L E : 1 2 3 4 5 6 7 D I S A G R E E D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y 1. T h e a m o u n t o f w o r k r e q u i r e d b y t h e i n s t r u c t o r i n t h i s c o u r s e i s r e a s o n a b l e . 0 1 2 3 4 5 6 7 2 . T h e i n s t r u c t o r ' s c l a s s p r e s e n t a t i o n s a r e w e l l o r g a n i z e d . 0 1 2 3 4 5 6 7 3 . A s s i g n m e n t s r e q u i r e d b y t h e i n s t r u c t o r a r e u s e f u l l e a r n i n g e x p e r i e n c e s . 0 1 2 3 4 5 6 7 4 . T h e i n s t r u c t o r i s t e a c h i n g m e t o l e a r n m o r e o n m y o w n i n t h i s s u b j e c t . 0 1 2 3 4 5 6 7 5 T h e i n s t r u c t o r u s e s s c h e d u l e d c l a s s t i m e e f f i c i e n t l y . 0 1 2 3 4 5 6 7 J L — W n s n s t u d e n t s a r e c o n f u s e d , t r i e i n s t r u c t o r a t t e m p t s t o c l a r i f y . 0 1 2 3 4 5 6 7 7. L e c t u r e s a n d o t h e r p r e s e n t a t i o n s a r e c o n s i s t e n t w i t h t h e s u b j e c t m a t t e r i n t h e c o u r s e o u t l i n e . 0 1 2. 3 4 5 6 7 8 T h e i n s t r u c t o r h a s h e l p e d m e t o u n d e r s t a n d r e l a t i o n s h i p s a m o n g i m p o r t a n t t o p i c s a n d i d e a s . 0 1 2 3 4 5 6 7 9. T h e l e v e l o f i n t e l l e c t u a l a c t i v i t y r e q u i r e d b y t h e i n s t r u c t o r i s a p p r o p r i a t e f o r t h i s k i n d o f c o u r s e . 0 2 3 4 5 6 7 1 0 . T h e i n s t r u c t o r c o m m u n i c a t e s c l e a r l y i n c l a s s . 0 2 3 4 5 6 7 1 1 . E x a m s a n d q u i z z e s h a v e a l l o w e d m e t o d e m o n s t r a t e a d e q u a t e l y w h a t I h a v e l e a r n e d . 0 1 2 3 4 5 6 7 12. T h e t e a c h i n g i n t h i s c o u r s e h a s m o t i v a t e d m e t o s t u d y t o p i c s o n m y o w n i n i t i a t i v e . 0 1 2 3 4 5 6 7 1 3 . T h e i n s t r u c t o r p r o v i d e s a d e q u a t e c l a s s t i m e t o r q u e s t i o n s . 0 2 3 4 5 6 7 14. T h e i n s t r u c t o r p r e s e n t s c o n c e p t s i n a m a n n e r t h a t a i d s s t u d e n t l e a r n i n g . 0 2 3 4 5 6 7 15. E v a l u a t i o n p r o c e d u r e s ( e x a m s , g r a d i n g , e t c . ) a r e f a i r . 0 2 3 4 5 6 7 1 6 . A s a r e s u l t o f t a k i n g t h i s c o u r s e . 1 h a v e b e c o m e m o r e i n t e r e s t e d i n t h e s u b j e c t . 0 2 3 4 5 6 7 17. T h e i n s t r u c t o r e m p h a s i z e s o u t c o m e s w h i c h m o s t s t u d e n t s c a n a c h i e v e i n t h e t i m e a l l o c a t e d . 0 2 3 4 5 6 7 18. T h e i n s t r u c t o r i s a v a i l a b l e t o h e l p s t u d e n t s o u t s i d e o f c l a s s . 0 2 3 4 5 6 7 19. T h e t e x t (if s e l e c t e d b y t h e i n s t r u c t o r ) a n d o t h e r r e a d i n g m a t e r i a l s h a v e b e e n i m p o r t a n t t o m y u n d e r s t a n d i n g o f t h e c o u r s e . 0 2 3 4 5 6 7 2 0 . I n s t r u c t i o n i n t h i s c o u r s e h a s h e l p e d m e t o d o w o r k t h a t i s t h e h i g h e s t q u a l i t y 1 c a n a c h i e v e . 0 2 3 4 5 6 7 2 1 . C o u r s e d i f f i c u l t y e s t a b l i s h e d b y t h e i n s t r u c t o r s e e m s a p p r o p r i a t e f o r m o s t s t u d e n t s i n t h e c l a s s . 0 2 3 4 5 6 7 2 2 . T h e i n s t r u c t o r ' s r e s p o n s e s t o q u e s t i o n s a r e c l e a r . 0 2 3 4 5 6 7 2 3 . A s s i g n m e n t s c o n t r i b u t e t o t h e a c h i e v e m e n t o f c o u r s e o b j e c t i v e s . 0 2 3 4 5 6 7 2 4 . M y k n o w l e d g e / s k i l l s h a v e i n c r e a s e d a s a r e s u l t o f t a k i n g t h i s c o u r s e . 0 2 3 4 5 6 7 2 5 . T h e i n s t r u c t o r h a s m a d e c o u r s e o b j e c t i v e s a n d r e q u i r e m e n t s c l e a r . 0 2 3 4 5 6 7 2 6 . T h e i n s t r u c t o r s e e m s t o t a k e a c c o u n t o f s t u d e n t s ' a b i l i t i e s , n e e d s , a n d i n t e r e s t s . 0 2 3 4 5 6 7 2 7 . E v a l u a t i o n p r o c e d u r e s p r o v i d e u s e f u l f e e d b a c k f o r s t u d e n t i m p r o v e m e n t . 0 2 3 4 5 6 7 2 8 . 1 w o u l d r e c o m m e n d t h i s i n s t r u c t o r ' s c o u r s e t o o t h e r s t u d e n t s w i s h i n g t o a c q u i r e k n o w l e d g e / s k i l l s i n t h i s a r e a . 0 2 3 4 5 6 7 2 9 . T h e i n s t r u c t o r s t a t e d c l e a r l y t h e b a s i s f o r e v a l u a t i n g s t u d e n t s . 0 2 3 4 5 6 7 3 0 . T h e b a s i s f o r e v a l u a t i o n , a s s t a t e d , w a s f o l l o w e d . 0 2 3 4 5 6 7 114 VAPAE Form FACULTY OF EDUCATION ANNUAL TEACHING EVALUATION Course # Instructor's Name Section # S T U D E N T B A C K G R O U N D I N F O R M A T I O N D a t e o f r e s p o n d i n g t o t h i s q u e s t i o n n a i r e : m o n t h y e a r . Y o u r s e x : Y o u r a g e : y e a r s . I n y o u r e a s e t h i s c o u r s e I s : r e q u i r e d e l e c t e d . W h i c h o f t h e f o l l o w i n g I s m o s t d e s c r i p t i v e o f y o u r p r o g r a m o f s t u d i e s ? ( c h e c k o n e o p t i o n ) O A r e g u l a r B . E d , p r o g r a m . O A g r a d u a t e t r a n s f e r ( 5 t h y e a r ) p r o g r a m . CZ) CD A q u a l i f y i n g p r o g r a m . L~H W h a t y e a r a r e y o u c u r r e n t l y r e g i s t e r e d I n ? ( c h e c k o n e o p t i o n ) D 1 s t CD 4 t h • 2 n d • S t h ( B . E d . ) • 3 r d • S t h ( t r a n s f e r ) O t h e r • • • • M a l e • F e m a l e A M a s t e r s o r D o c t o r a l p r o g r a m . D i p l o m a M a s t e r s / D o c t o r a l O t h e r I n t h e F a c u l t y o f E d u c a t i o n , s t u d e n t s a r e a s k e d t o e v a l u a t e t h e q u a l i t y o f i n s t r u c t i o n i n e v e r y c o u r s e . I n r e s p o n d i n g t o t h e I t e m s g i v e n h e r e , p l e a s e k e e p i n m i n d t h a t y o u a r e a s k e d t o r a t e t h o I n s t r u c t o r a n d t h o s e a s p e c t s o f t h e c o u r s e o v e r w h i c h h e o r s h e c l e a r l y h a s c o n t r o l . P l e a s e d o n o t e v a l u a t e o t h e r r e l a t e d ( a c t o r s , s u c h a s d e p a r t m e n t o r f a c u l t y p o l i c i e s . T h i s i s t o b e a n e v a l u a t i o n o f t h e In-d i v i d u a l I n s t r u c t o r ' s t e a c h i n g a n d r e l a t e d a c t i v i t i e s a n d o u t c o m e s . Consider ing all facets of this Instructor's teaching, I would rate It as: ( C H E C K ONE O F T H E B O X E S S H O W N B E L O W ) E x c e l l e n t G o o d 3 • • V e r y g o o d 4 • J u s t a d e q u a t e • L e s s t h a n a d e q u a t e _ _ _ 1 L J I n c o m p e t e n t P o o r F o r t h e r e m a i n i n g I t e m s , a s e v e n - p o i n t r a t i n g s c a l e i s p r o v i d e d f o r y o u t o r e c o r d y o u r r e s p o n s e s . P l e a s e I n t e r p r e t It t o h a v e t h e f o l l o w i n g m e a n i n g : 1 2 3 4 S 6 7 D I S A G R E E D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y A l t h o u g h t h i s s c a l e c a n b e a p p l i e d In a g e n e r a l w a y l o t h e f o l l o w i n g 3 0 i t e m s , it m a y n o t b e g r a m m a t i c a l l y p r e c i s e I n a f e w c a s e s . P l e a s e k e e p In m i n d t h a t a r a t i n g o f ' 7 ' Is A L W A Y S I n t e n d e d t o r e f l e c t a n e x t r e m e l y F A V O U R A B L E r e a c t i o n t o I n s t r u c t i o n In y o u r c o u r s e , a n d a r a t i n g o f V I s A L W A Y S I n t e n d e d t o r e f l e c t a n e x t r e m e l y U N F A V O U R A B L E r e a c t i o n . If y o u w i s h t o u s e a s c a l e p o i n t w h i c h d o e s n o t h a v e a l a b e l a b o v e — e . g . , a ' 6 ' b e c a u s e y o u f e e l t h a t y o u f i t s o m e w h e r e b e t w e e n ' a g r e e s o m e w h a t ' a n d ' a g r e e v e r y s t r o n g l y , ' p l e a s e f e e l f r e e t o d o s o . It y o u f e e l t h a t a n y i t e m Is t o t a l l y I r r e l e v a n t t o t h e q u a l i t y o f I n s t r u c t i o n i n y o u r c o u r s e , c i r c l e t h e '0' ( z e r o ) o p t i o n p r o v i d e d . T h i s m a y o c c u r , f o r e x a m p l e , In t h e c a s e o f a n I t e m w h i c h a s k s a b o u t e x a m s In a c o u r s e w h i c h h a s n o e x a m s . P L E A S E N O T E t h a t t h e ' z e r o ' o p t i o n Is n o t t h e s a m e a s a n e u t r a l r e a c t i o n t o I n s t r u c t i o n , w h i c h w o u l d b e i n d i c a t e d b y t h e ' 4 ' r e s p o n s e . P L E A S E T U R N T H E P A G E A N D C O N T I N U E Y O U R R A T I N G 115 R E M E M B E R , C IRCLE "0" ONLY IF YOU THINK T H E ITEM IS IRRELEVANT TO THIS C O U R S E . OTHERWISE USE THIS S C A L E : 1 2 3 4 5 6 7 D I S A G R E E D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y 1. The amount of work r e q u i r e d by the I n s t r u c t o r In t h i s course i s reasonable. 0 1 : 3 u > & 7 2. The I n s t r u c t o r ' s c l a s s p r e s e n t a t i o n s are w e l l o r ganized. 0 i ; 3 4 : 6 7 3. Assignments r e q u i r e d by the I n s t r u c t o r are u s e f u l l e a r n i n g e x p e r i e n c e s . o i ; 3 4 > 6 7 4. The I n s t r u c t o r i s t e a c h i n g me to l e a r n more on my own i n t h i s s u b j e c t . 0 1 i 3 4 > 6 7 5. The I n s t r u c t o r uses scheduled c l a s s time e f f i c i e n t l y . o i ; 3 4 > 6 7 6. When students are confused, the I n s t r u c t o r attempts to c l a r i f y . o i ; 3 4 i 6 7 7. Class a c t i v i t i e s and other p r e s e n t a t i o n s are c o n s i s t e n t w i t h the s u b j e c t matter i n the course o u t l i n e . o i : 3 4 > 6 7 8. The I n s t r u c t o r has helped me to understand r e l a t i o n s h i p s among important t o p i c s and id e a s . 0 1 1 3 4 . > 6 7 9. The l e v e l of i n t e l l e c t u a l a c t i v i t y r e q u i r e d by the i n s t r u c t o r i 6 a p p r o p r i a t e f o r t h i s k i n d of course. o \ ' . ! 3 4 5 6 7 10. The i n s t r u c t o r communicates and demonstrates c l e a r l y i n c l a s s . 0 1 . 3 4 > 6 7 11. Assignments have allowed me to demonstrate adequately what I have l e a r n e d . 0 1 . 3 4 > 6 7 12. The teaching i n t h i s course has motivated me to e x p l o r e t o p i c s on my own i n i t i a t i v e . 0 1 ! 3 4 5 6 7 13. The i n s t r u c t o r p r o v i d e s adequate c l a s s time f o r questions and d i s c u s s i o n s . 0 1 I 3 4 > 6 7 14. The I n s t r u c t o r presents concepts In a manner that a i d s student l e a r n i n g . 0 1 . 1 3 4 . > 6 7 15. E v a l u a t i o n procedures ( g r a d i n g , p r o j e c t , p r e s e n t a t i o n s of work, a u d i t i o n s , e t c . ) are f a i r . 0 1 1 3 4 5 6 7 16. As a r e s u l t of t a k i n g t h i s course, 1 have become more i n t e r e s t e d I n the s u b j e c t . 0 l 1 3 4 . 5 6 7 17. The I n s t r u c t o r emphasizes outcomes which most students can achieve i n the time a l l o c a t e d . 0 1 , ! 3 4 1 6 7 18. The I n s t r u c t o r i s a v a i l a b l e to help students o u t s i d e of c l a s s . o i : 3 4 S 6 7 19. T e a c h i n g / l e a r n i n g and o t h e r reading m a t e r i a l s have been important to my understanding of the course. 0 1 ! 3 4 S 6 7 20. I n s t r u c t i o n i n t h i s course has helped me to do work that I s the h i g h e s t q u a l i t y I can achieve. 0 1 1 3 4 . > 6 7 21. Course d i f f i c u l t y e s t a b l i s h e d by the i n s t r u c t o r seems a p p r o p r i a t e f o r most students i n the c l a s s . o i ; 3 4 < 6 7 22. The I n s t r u c t o r ' s responses to q u e s t i o n s are c l e a r . o i : 3 4 : 6 7 23. Assignments c o n t r i b u t e to the achievement of course o b j e c t i v e s . o l ; 1 3 4 ! 6 7 26. My k n o w l e d g e / s k i l l s have Increased as a r e s u l t of Caking t h i s course. 0 1 ! 3 4 > 6 7 25. The I n s t r u c t o r has made course o b j e c t i v e s and requirements c l e a r . 0 1 ! 3 4 ! 6 7 26. The I n s t r u c t o r seems to take account of st u d e n t s ' a b i l i t i e s , needs, and I n t e r e s t s . 0 1 1 3 4 . , 6 7 27. E v a l u a t i o n procedures p r o v i d e u s e f u l feedback f o r student growth. 0 1 ! 3 4 > 6 7 28. 1 would recommend t h i s I n s t r u c t o r ' s course to oth e r students w i s h i n g to acquire k n o w l e d g e / s k i l l s i n t h i s a r e a . 0 1 I 3 4 ! 6 7 29. The I n s t r u c t o r a t a t e d c l e a r l y the b a s i s f o r e v a l u a t i n g studenta. 0 1 ! 3 4 S 6 7 30. The s t a t e d b a s i s f o r e v a l u a t i o n has t o date been f o l l o w e d . 0 1 ! 3 4 . 6 7 116 Instructor's Name mm FACULTY OF EDUCATION ANNUAL TEACHING EVALUATION Course # Laboratory Section // _ m o n t h _ S T U D E N T B A C K G R O U N D I N F O R M A T I O N D a t e o f r e s p o n d i n g t o t h i s q u e s t i o n n a i r e : Y o u r a g e : y e a r s . W h i c h o l t h e f o l l o w i n g i s m o a t d e s c r i p t i v e o f y o u r p r o g r a m o f s t u d i e s ? ( c h e c k i A r e g u l a r B . E d , p r o g r a m . l_J A g r a d u a t e t r a n s f e r program. . y e a r . Y o u r s e x :  • M a l e • F e m a l e i o p t i o n ) CD A M a s t e r s o r D o c t o r a l p r o g r a m . • A 3rd year^B.Ed.^ • A q u a l i f y i n g p r o g r a m . W h a t y e a r a r e y o u c u r r e n t l y r e g i s t e r e d i n ? ( c h e c k o n e o p t i o n ) • O t h e r • 3rd • 4th • 5th (B.Ed.) 0 5th (transfer) 1 I Diploma I I Masters/Doctoral • Other I n t h e F a c u l t y o f E d u c a t i o n , s t u d e n t s a r e a s k e d t o e v a l u a t e t h e q u a l i t y o f I n s t r u c t i o n i n e v e r y c o u r s e . I n r e s p o n d i n g t o t h e I t e m s g i v e n h e r e , p l e a s e k e e p i n m i n d t h a t y o u a r e a s k e d t o r a t e I h o i n s t r u c t o r a n d t h o s e a s p e c t s o l t h e c o u r s e o v e r w h i c h h e o r s h e c l e a r l y h a s c o n t r o l . P l e a s e d o n o t e v a l u a t e o t h e r r e l a t e d l a c t o r s , s u c h a s d e p a r t m e n t o r f a c u l t y p o l i c i e s . T h i s i s l o b e a n e v a l u a t i o n o f t h e i n -d i v i d u a l I n s t r u c t o r ' s t e a c h i n g a n d r e l a t e d a c t i v i t i e s a n d o u t c o m e s . A . Consider ing all facets of this Instructor ' s teaching, I would rat* It as: ( C H E C K ONE O F T H E B O X E S S H O W N B E L O W ) r • E x c e l l e n t 5 • G o o d 3 • •  V e r y g o o d 4 •  J u s t a d e q u a t e • L e s s t h a n a d e q u a t e . . 1 Lj P o o r I n c o m p e t e n t F o r t h e r e m a i n i n g I t e m s , a s e v e n - p o i n t r a t i n g s c a l e Is p r o v i d e d l o r y o u t o r e c o r d y o u r r e s p o n s e s . P l e a s e I n t e r p r e t It t o h a v e t h e f o l l o w i n g m e a n i n g : 1 2 3 4 5 6 7 D I S A G R E E D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y A l t h o u g h t h i s s c a l e c a n b e a p p l i e d In a g e n e r a l w a y l o t h e f o l l o w i n g 1 2 I t e m s , It m a y n o t b e g r a m m a t i c a l l y p r e c i s e In a l e w c a s e s . P l e a s e k e e p I n m i n d t h a t a r a t i n g o f ' 7 ' i s A L W A Y S I n t e n d e d t o r e f l e c t a n e x t r e m e l y F A V O U R A B L E r e a c t i o n t o I n s t r u c t i o n In y o u r c o u r s e , a n d a r a t i n g o f ' V I s A L W A Y S I n t e n d e d t o r e f l e c t a n e x t r e m e l y U N F A V O U R A B L E r e a c t i o n . If y o u w i s h t o u s e a s c a l e p o i n t w h i c h d o e s n o t h a v e a l a b e l a b o v e — e . g . , a ' 6 ' b e c a u s e y o u f e e l t h a t y o u f i t s o m e w h e r e b e t w e e n ' a g r e e s o m e w h a t ' a n d ' a g r e e v e r y s t r o n g l y , ' p l e a s e f e e l f r e e t o d o s o . If y o u f e e l t h a t a n y i t e m Is t o t a l l y I r r e l e v a n t t o t h e q u a l i t y o f I n s t r u c t i o n In y o u r c o u r s e , c i r c l e t h e ' 0 ' ( z e r o ) o p t i o n p r o v i d e d . T h i s m a y o c c u r , l o r e x a m p l e . In t h e c a s e o f a n I t e m w h i c h a s k s a b o u t e x a m s In a c o u r s e w h i c h h a s n o e x a m s . P L E A S E N O T E t h a t t h e ' z e r o ' o p t i o n I s n o t t h e s a m e a s a n e u t r a l r e a c t i o n t o I n s t r u c t i o n , w h i c h w o u l d b e i n d i c a t e d b y t h e ' 4 ' r e s p o n s e . INSTRUCTOR-SPECIFIC FORM 6787 P L E A S E T U R N T H E P A G E A N D C O N T I N U E Y O U R R A T I N G 117 R E M E M B E R , C IRCLE "0" ONLY IF YOU THINK T H E ITEM IS IRRELEVANT T O THIS C O U R S E . OTHERWISE U S E THIS S C A L E : 1 2 3 4 S 6 7 D I S A G R E E D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y 1. The I n s t r u c t o r ' s c l a s s presentations are w e l l organized. 0 1 2 3 4 5 6 7 2. The I n s t r u c t o r uses scheduled c l a s s t i n e e f f i c i e n t l y . 0 1 2 3 4 5 6 7 3. The I n s t r u c t o r has helped me to understand r e l a t i o n s h i p s among important t o p i c s and ideas. 0 2 3 4 5 6 7 4. When students are confused, the i n s t r u c t o r attempts to c l a r i f y . 0 J 2 3 4 5 6 7 5. Lectures are c o n s i s t e n t with the subject matter i n the course o u t l i n e . 0 2 3 4 5 6 7 6. This i n s t r u c t o r ' s teaching has motivated me to study t o p i c s on my own i n i t i a t i v e . 0 2 3 4 5 6 7 7. The i n s t r u c t o r communicates c l e a r l y i n c l a s s . 0 2 3 4 5 6 7 8. The i n s t r u c t o r presents concepts i n a manner that a i d s student l e a r n i n g . . 0 2 3 4 5 6 7 9. I n s t r u c t i o n i n t h i s course has helped me to do work that i s the highest q u a l i t y I can achieve. 0 2 3 4 5 6 7 10. The i n s t r u c t o r i s a v a i l a b l e to help students outside of c l a s s . 0 2 3 4 5 6 7 11. The i n s t r u c t o r ' s responses to questions are c l e a r . 0 L 2 3 4 5 6 7 12. I would recommend t h i s i n s t r u c t o r to other students wishing to acquire knowledge/skills i n t h i s area. 0 2 3 4 5 6 7 118 FACULTY OF EDUCATION ANNUAL TEACHING EVALUATION Course # Inst rue tor(3)_ S T U D E N T B A C K G R O U N D I N F O R M A T I O N O M a l e CH F e m a l e D a t a o f r a a n a n r l l n n I n l h l « m i a a l l a n n a i r a - m n n l h v a a r . Y o u r a n : Y o u r a g e : y e a r s . W h i c h o f t h e f o l l o w i n g I s m o a t d e s c r i p t i v e o f y o u r p r o g r a m o f s t u d i e s ? ( c h e c k o n e o p t i o n ) 1 1 A r e g u l a r B . E d , p r o g r a m . 1 1 A g r a d u a t e t r a n s f e r program. • A M a s t e r s o r D o c t o r a l p r o g r a m . Q A 3rd year B.Ed. Q A q u a l i f y i n g p r o g r a m . • O t h e r program. W h a t y e a r a r e y o u c u r r e n t l y r e g i s t e r e d I n ? ( c h e c k o n e o p t i o n ) Q 3rd • 5th (transfer) • Other Q 4 th • Diploma O 5th (B.Ed.) • Masters/Doctoral I n t h e F a c u l t y o f E d u c a t i o n , s t u d e n t s a r e a s k e d t o e v a l u a t e t h e q u a l i t y o f I n s t r u c t i o n In e v e r y c o u r s e . I n r e s p o n d i n g t o t h e i t e m s g i v e n h e r e , p l e a s e k e e p i n m i n d t h a t y o u a r e a s k e d t o r a l e t h e I n s t r u c t o r a n d t h o s e a s p e c t s o f t h e c o u r s e o v e r w h i c h h e o r s h e c l e a r l y h a s c o n t r o l . P l e a s e d o n e t e v a l u a t e o t h e r ( e l a t e d f a c t o r s , s u c h a s d e p a r t m e n t o r f a c u l t y p o l i c i e s . T h i s i s t o b e a n e v a l u a t i o n o l t h e i n -d i v i d u a l I n s t r u c t o r ' s t e a c h i n g a n d r e l a t e d a c t i v i t i e s a n d o u t c o m e s . Consider ing all facets o l this COURSE , I would rats It as : ( C H E C K ONE OF T H E B O X E S S H O W N B E L O W ) 7 O E x c e l l e n t 5 CD • V e r y g o o d 4 • G o o d J u s t a d e q u a t e 3 CD L e s s t h a n a d e q u a t e . . _ _ 1 L _ | I n c o m p e t e n t 2 CD P o o r F o r t h e r e m a i n i n g I t e m s , a s e v e n - p o i n t r a t i n g s c a l e Is p r o v i d e d f o r y o u t o r e c o r d y o u r r e s p o n s e s . P l e a s e I n t e r p r e t It t o h a v e t h e . f o l l o w i n g m e a n i n g : 1 2 3 4 s 6 7 D I S A G R E E D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y A l t h o u g h t h i s s c a l e c a n b e a p p l i e d In a g e n e r a l w a y t o t h e f o l l o w i n g 17items, It m a y n o t b e g r a m m a t i c a l l y p r e c i s e In a f e w c a s e s . P l e a s e k e e p In m i n d t h a t a r a t i n g o f '7' i s A L W A Y S I n t e n d e d t o r e f l e c t a n e x t r e m e l y F A V O U R A B L E r e a c t i o n t o I n s t r u c t i o n In y o u r c o u r s e , a n d a r a t i n g o f ' V i s A L W A Y S I n t e n d e d t o r e f l e c t a n e x t r e m e l y U N F A V O U R A B L E r e a c t i o n . If y o u w i s h t o u s e a s c a l e p o i n t w h i c h d o e s n o t h a v e a l a b e l a b o v e —e . g . , a ' 6 ' b e c a u s e y o u f e e l t h a t y o u f i t s o m e w h e r e b e t w e e n ' a g r e e s o m e w h a t ' a n d ' a g r e e v e r y s t r o n g l y , ' p l e a s e f e e l f r e e t o d o s o . If y o u f e e l t h a t a n y I t e m i s t o t a l l y I r r e l e v a n t t o t h e q u a l i t y o f I n s t r u c t i o n In y o u r c o u r s e , c i r c l e t h e ' 0 ' ( z e r o ) o p t i o n p r o v i d e d . T h i s m a y o c c u r , f o r e x a m p l e , i n t h e c a s e o f a n I t e m w h i c h a s k s a b o u t e x a m s In a c o u r s e w h i c h h a s n o e x a m s . P L E A S E N O T E t h a t t h e ' z e r o ' o p t i o n I s n o t t h e s a m e a s a n e u t r a l r e a c t i o n t o I n s t r u c t i o n , w h i c h w o u l d be i n d i c a t e d b y t h e ' 4 ' r e s p o n s e . RC;kpg Nov. 5/87 COURSE-WIDE FORM P L E A S E T U R N T H E P A G E A N D C O N T I N U E Y O U R R A T I N G 119 R E M E M B E R , C IRCLE "0" ONLY IF YOU THINK T H E ITEM IS IRRELEVANT T O THIS C O U R S E . OTHERWISE USE THIS S C A L E : 1 2 3 4 5 6 7 D I S A G R E E D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y The amount of work required i n t h i s course Is reasonable. 0 1 2 3 4 5 6 7 Assignments required i n t h i s course are useful learning experiences. 0 This course i s encouraging me to learn more on my own i n t h i s subject. 0 The l e v e l of i n t e l l e c t u a l a c t i v i t y required i s appropriate to th i s kind of course. Exams and assignments have allowed me to demonstrate adequately what 1 have learned. Evaluation procedures (exams, grading, etc.) are f a i r . 2 3 4 5 6 2 3 4 5 6 9. 10. 11. 12. This course emphasizes outcomes which most students can achieve in the time a l l o c a t e d . The text(s) and other reading materials have been important to my understanding of the course. Assignments contribute to the achievement of course objectives. 2 3 4 5 6 2 3 4 5 6 As a r e s u l t of taking th i s course, I have become more interested i n the subject. Course d i f f i c u l t y seems appropriate f o r most students i n the c l a s s . Evaluation procedures provide u s e f u l feedback for student improvement. 2 3 4 5 6 2 3 4 5 6 13. 14. 15. 16. 17. My knowledge/skills have increased as a r e s u l t of taking t h i s course. Objectives and requirements for t h i s course are c l e a r . The basis f o r evaluating students was c l e a r l y stated. 2 3 4 5 6 2 3 4 5 6 2 3 4 5 6 The basis for evaluation, as stated, was follows. A l l things considered, I think I w i l l be a better teacher for having taken t h i s course. 120 Instructor's N a m e . FACULTY OF EDUCATION ANNUAL TEACHING EVALUATION Course // Laboratory Section // S T U D E N T B A C K G R O U N D I N F O R M A T I O N D a t e o f r e s p o n d i n g t o t h i s q u e s t i o n n a i r e : . Y o u r a g e : y e a r s . . m o n t h . . y e a r . Y o u r s e x :  • M a l e • F e m a l e W h i c h o f t h e f o l l o w i n g l a m o a t d e s c r i p t i v e o f y o u r p r o g r a m o f s t u d i e s ? ( c h e c k o n e o p t i o n ) A M a s t e r s o r D o c t o r a l p r o g r a m . • A r e g u l a r B . E d , p r o g r a m . A g r a d u a t e t r a n s f e r program. O O t h e r • A 3rd y e a g . E d . ^ g A q u a l i f y i n g p r o g r a m , W h a t y e a r a r e y o u c u r r e n t l y r e g i s t e r e d I n ? ( c h e c k o n e o p t i o n ) • 3rd • 4th • 5th (B.Ed.) I I 5th (transfer) I I Diploma I I Masters/Doctoral • Other I n t h e F a c u l t y o f E d u c a t i o n , s t u d e n t s a r e a s k e d t o e v a l u a t e t h e q u a l i t y o f I n s t r u c t i o n i n e v e r y c o u r s e . I n r e s p o n d i n g t o t h e I t e m s g i v e n h e r e , p l e a s e k e e p i n m i n d t h a t y o u a r e a s k e d t o r a t e t h e I n s t r u c t o r a n d t h o s e a s p e c t s o f t h e c o u r s e o v e r w h i c h h e o r s h e c l e a r l y h a s c o n t r o l . P l e a s e d o n o t e v a l u a t e o t h e r r e l a t e d f a c t o r s , s u c h a s d e p a r t m e n t o r f a c u l t y p o l i c i e s . T h i s i s t o b e a n e v a l u a t i o n o f t h e i n -d i v i d u a l I n s t r u c t o r ' s t e a c h i n g a n d r e l a t e d a c t i v i t i e s a n d o u t c o m e s . Consider ing all tacets of this Instructor's teaching. I would rate It aa: ( C H E C K ONE O F T H E B O X E S S H O W N B E L O W ) 7 • E x c e l l e n t 5 • G o o d 3 • • V e r y g o o d • J u s t a d e q u a t e • L e s s t h a n a d e q u a t e 1 U I n c o m p e t e n t P o o r F o r t h e r e m a i n i n g I t e m s , a s e v e n - p o i n t r a t i n g s c a l e i s p r o v i d e d f o r y o u t o r e c o r d y o u r r e s p o n s e s . P l e a s e I n t e r p r e t It t o h a v e t h e f o l l o w i n g m e a n i n g : 1 2 3 4 5 6 7 D I S A G R E E D I S A G R E E A G R E E 0 A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y A l t h o u g h t h i s s c a l e c a n b e a p p l i e d In a g e n e r a l w a y t o t h e f o l l o w i n g 27 i t e m s , it m a y n o t b e g r a m m a t i c a l l y p r e c i s e i n a f e w c a s e s . P l e a s e k e e p In m i n d t h a t a r a t i n g o f ' 7 ' Is A L W A Y S I n t e n d e d t o r e f l e c t a n e x t r e m e l y F A V O U R A B L E r e a c t i o n t o i n s t r u c t i o n i n y o u r c o u r s e , a n d a r a t i n g o f ' V Is A L W A Y S I n t e n d e d t o r e r l e c t a n e x t r e m e l y U N F A V O U R A B L E r e a c t i o n . If y o u w i s h t o u s e a s c a l e p o i n t w h i c h d o e s n o t h a v e a l a b e l a b o v e — e . g . , a ' 6 ' b e c a u s e y o u f e e l t h a t y o u f i t s o m e w h e r e b e t w e e n ' a g r e e s o m e w h a t ' a n d ' a g r e e v e r y s t r o n g l y , ' p l e a s e f e e l f r e e t o d o s o . If y o u f e e l t h a t a n y I t e m Is t o t a l l y I r r e l e v a n t t o t h e q u a l i t y o f I n s t r u c t i o n In y o u r c o u r s e , c i r c l e t h e " 0 * ( z e r o ) o p t i o n p r o v i d e d . T h i s m a y o c c u r , f o r e x a m p l e . In t h e c a s e o f a n I t e m w h i c h a s k s a b o u t e x a m s i n a c o u r s e w h i c h h a s n o e x a m s . P L E A S E N O T E t h a t t h e ' z e r o ' o p t i o n I s n o t t h e s a m e a s a n e u t r a l r e a c t i o n t o I n s t r u c t i o n , w h i c h w o u l d b e i n d i c a t e d b y t h e ' 4 ' r e s p o n s e . LABORATORY FORM P L E A S E T U R N T H E P A G E A N D C O N T I N U E Y O U R R A T I N G 121 R E M E M B E R , CIRCLE "0" ONLY IF Y O U THINK T H E ITEM IS IRRELEVANT TO THIS C O U R S E . OTHERWISE USE THIS S C A L E : 1 2 3 4 5 6 7 D I S A G R E E : D I S A G R E E A G R E E A G R E E V E R Y S O M E W H A T S O M E - V E R Y S T R O N G L Y W H A T S T R O N G L Y i . The amount of work required In thla laboratory Is reasonable. 0 1 2 3 4 5 6 7 2. This Instructor's laboratories are well organized. 0 1 2 3 4 5 6 7 3. Laboratory evaluation procedures provide useful feedback for student improvement. 0 1 2 3 4 5 6 7 4. The laboratory Instructor l a teaching me to learn more on my own In thla subject. 0 1 2 3 4 5 6 7 5. The lsboratory Instructor uses scheduled time e f f i c i e n t l y . 0 1 2 3 4 5 6 7 6. When students are confused* the laboratory Instructor attempts to c l a r i f y . 0 1 2 3 4 5 6 7 7. The lsboratory Instructor seated c l e a r l y the basis for evaluating students. 0 1 2 3 4 5 6 7 8. The laboratory Instructor hss helped me to understand relationships among Important topics and ideas. 0 1 2 3 4 5 6 7 9. The l e v e l of I n t e l l e c t u a l a c t i v i t y required i n this laboratory i s appropriate. 0 1 2 3 4 5 6 7 10. The laboratory instructor communicates c l e a r l y . 0 1 2. 3 4 5 6 7 11. The baals for evaluation, aa stated, was followed. 0 1 2 3 4 5 6 7 12. The teaching In this lsboratory haa motivated me to atudy toplca on my own I n i t i a t i v e . 0 1 2 3 4 5 6 7 13. The laboratory instructor provides sdequate claaa time f o r questions. 0 1 2 3 4 5 6 7 14. The laboratory Inatructor explains concepts i n a manner that aids student learning. 0 1 2 3 4 5 6 7 15. Inatructlon In thla laboratory haa helped ma to do work that Is the highest quality I can achieve. 0 1 2 3 4 5 6 7 16. The laboratory emphasizes outcomes which most students can achieve in the time allocated. 0 1 2 ! 4 5 6 7 17. The laboratory inatructor i s available to help students outside of claaa. 0 1 2 ) 4 5 6 7 18. My knowledge/skills have lncreaaed as a result of taking this laboratory. 0 1 2 ) 4 5 6 7 19. The laboratory Instructor haa made course objectives and requirements c l e a r . 0 1 2 ) 4 5 6 7 20. The laboratory instructor's responses to questions are clear. 0 1 2 3 4 5 6 7 21. I would recommend th i s laboratory Instructor to other students wishing to acquire knowledge/skills In this srea. 0 1 2 ) 4 5 6 7 22. The laboratory instructor takes account of students' s b l l i t i e s , needs, and Interests. 0 1 2 3 4 5 6 7 lONL* STUDENTS IN SECONDARY TEACHER EDUCATION PROCRAMS SHOULD RATE THE NEXT FIVE ITEMS 23. Assignments required by the laboratory instructor sre useful learning experiences. 0 1 2 3 4 5 6 7 24. Laboratory sessions are consistent with the subject mstter i n the course outline. 0 1 2 3 4 5 6 7 25. Laboratory evaluation procedurea have allowed me to demonatrate adequately what I have learned. 0 1 2 3 4 5 6 7 26. Evaluation procedures for laboratory aaaignments are f a i r . 0 1 2 3 4 5 6 7 27. Laboratory assignments contribute to the achievement of course objectives. 0 1 2 3 4 5 6 7 122 SUMMARY REPORT ANNUAL TEACHING EVALUATION FACULTY OF EDUCATION G R O U P S I Z E S : G r o u p A i s 1 9 % o f t h o s e w h o w e r e e v a l u a t e d G r o u p B i s 4 9 % o f t h o s e w h o w e r e e v a l u a t e d G r o u p C i s 1 5 % o f t h o s e w h o w e r e e v a l u a t e d G r o u p 0 i s 1 4 % o f t h o s e w h o w e r e e v a l u a t e d G r o u p E i s 3 % o f t h o s e w h o w e r e e v a l u a t e d F A C E T D E R N I T I O N S : F a c e t o n e i s : C o u r s e p l a n n i n g & o r g a n i z a t i o n F a c e t t w o i s : I n s t r u c t i o n a l t e c h n i q u e & s t y l e F a c e t t h r e e i s : U s e o f t e a c h i n g m a t e r i a l & e v a l u a t i o n F a c e t four i s : S t u d e n t l e a r n i n g o u t c o m e s O v e r a l l i s : A s i n g l e i t e m ( s e e f r o n t p a g e o f E v a l u a t i o n F o r m ) B A S I S F O R T H I S P R O F I L E I S : s t u d e n t s i n s e c . s t u d e n t s i n s e c . s t u d e n t s i n s e c . s t u d e n t s i n s e c . s t u d e n t s i n s e c . s t u d e n t s i n a l l : - S E C T I O N B R E A K D O W N S A N D I T E M R E S U L T S A P P E A R O N T H E F O L L O W I N G P A G E S " 123 APPENDIX B A Survey of the Role of Student Ratings of Teaching Performance 124 A SURVEY OF" THE ROUE OF" STUDENT RATINGS OF" TEACH I IMG PERFORMANCE Quest. # The purpose of this questionnaire is to determine the role Student Ratings of Teaching Performance (Annual Teaching Evaluations) play in personnel decisions in the Faculty of Education. The personnel decisions referred to in this questionnaire are only those uhich pertain to reappointment, promotion and tenure but not merit pay or career progress increments. I. BACKGROUND INFORMATION The follouing information is for s t a t i s t i c a l and record keeping purposes only. Your responses are completely c o n f i d e n t i a l and all r e s u l t s Hill be aggregated across respondents. Please check the appropriate response or supply the appropriate information. 1. With which department or school are you most closely associated? _a. Department of Administrative, Higher and Adult Education _b. Department of Counselling Psychology _c. Department of Educational Psychology and Special Education _d. Department of Language Education _e. Department of Mathematics and Science Education _f. Department of Social and Educational Studies _g. Department of Visual and Performing Arts _h. School of Physical Education and Recreation 2. Which of the following best describes your rank within this faculty? a. Assistant Professor With Term (Without Tenure) _b. Associate Professor With Term (Without Tenure) c. Professor With Term (Without Tenure) d. Assistant Professor Without Term (With Tenure) e. Associate Professor Without Term (With Tenure) f. Professor Without Term (With Tenure) g. Instructor (With or Without Tenure) 3. Which of the following best describes the capacity in which you currently participate in personnel decision-making processes within this faculty? a. As a member of the Departmental Standing Personnel Committee b. As a member of the Faculty Standing Personnel Committee c. As Head of Department d. As Dean/Associate Dean/Assistant Dean e. Other (please specify) ROLE OF STUDENT RATINGS (OVER) PAGE 1 125 4. How long have you participated in personnel decisions in the capacity identified in Question #3? a. less than 2 years b. 3 to 5 years c. 6 to 10 years d. more than 10 years 5. Have you ever participated in personnel decisions in any other capacity other than one identified in Question #3? YES NO 6. If YES to tt5i Please identify the capacities in which you have participated in the personnel decision-making processes within this faculty. (Check all that apply to you). a. Member of a Departmental Standing Personnel Committee b. Member of the Faculty Standing Personnel Committee c. Member of the Senior Appointments Committee d. Head of Department e. Dean/Associate Dean/Assistant Dean f. Other II. THE ROLE OF STUDENT RATINGS OF TEACHING PERFORMANCE IN PERSONNEL DECISIONS The purpose of the follouing section of the survey is to determine the role of Student Ratings of Teaching Performance (Annual Teaching Evaluations) in personnel decisions in the Faculty of Education at U.B.C. Part II of the survey consists of four subsections. Each subsection is a d i f f e r e n t colour and deals uith only one p a r t i c u l a r type of personnel decision because it is recognized that the c r i t e r i a considered may vary according to the type of personnel decision being made. It is hoped that by limiting each subsection to one s p e c i f i c type of personnel decision it Hill make it easier for you to complete the survey. you are asked to complete the rest of this questionnaire by responding to each item on your oun behalf and not as a representative of any committee of uhich you are a member. ROLE OF STUDENT RATINGS (CONTINUED) PAGE 2 126 DECISIONS CONCERNING ASSISTANT PROFESSORS WITH TERM (WITHOUT TENURE) The items an these gold pages pertain to the appointment at or promotion to the rank of Assistant Professor Hith Term (Uithout Tenure). As you complete these pages, please give the responses that best indicate hou you, typically, consider these factors uhen making your decisions. 1. How important i s each of the following factors when a candidate i s to be appointed at or promoted to the rank of Assistant Professor With Term (Without Tenure)? Please indicate how important each of these factors i s to you by placing a c i r c l e around one response in each row. VERY IMPORTANT a. Classroom teaching b. Research activity c. Publications d. Public service e. Consultation (governments/business). f. Activity in professional societies g. Supervision of graduate study. h. Student program advising.... i . University, faculty or department committee work... j. Length of service at rank... k. Public recognition 1. Personality Characteristics. m. Other .. (please speci fy) IMPORTANT 2 2 2 2 2 2 2 2 2 2 2 NOT VERY IMPORTANT 3 3 3 3 3 3 3 3 3 3 3 NOT AT ALL IMPORTANT 4 4 4 4 4 4 4 4 4 4 4 ROLE OF STUDENT RATINGS (OVER) PAGE 3 127 What sources of information are p r inc ipa l l y considered by you in the evaluation of a faculty member's teaching performance when a candidate i s to be appointed at or promoted to the rank of Assistant Professor With Term (Without Tenure)? Please indicate the importance of each type of information source by placing a c i r c l e around one response in each row. VERY IMPORTANT a. Formal Annual Teaching Evaluations by students (SCET) b. Formal Peer Reviews c. Informal Student Opinion d. Informal Opinions of Colleagues e. Evaluation by Head of Department f. Evaluation by Dean g. Opinion of Outside Experts... h. Self-evaluation or Report.... i . Course Syllabi and Examinations j . Student Examination Per formanee k. Grade Distributions 1. Enrollment in Elective Classes m. Long-term Follow-up of Students (Alumni Opinions) n. Research/Publication o. Other ... (please speci fy) IMPORTANT 2 2 2 2 2 2 NOT VERY IMPORTANT 3 3 3 NOT AT ALL IMPORTANT 4 4 4 ROLE OF STUDENT RATINGS (CONTINUED) PAGE 4 128 DECISIONS CONCERNING ASSOCIATE PROFESSORS WITH TERM (WITHOUT TENURE) The items on these green pages pertain to the appointment at or promotion to the rank of Associate Professor Hith Term (Uithout Tenure). As you complete these pages, please give the response that best indicates hon you, typically, consider sthese factors uhen making your decisions. How important i s each of the following factors when a candidate i s to be appointed at or promoted to the rank of Associate Professor With Term (Without Tenure)? Please indicate how important each of these factors i s to you by placing a c i r c l e around one response in each row. VERY IMPORTANT a. Classroom teaching b. Research activity c. Publications... d. Public service e. Consultation (governments/business) f. Activity in professional societies g. Supervision of graduate study h. Student program advising..... i . University, faculty or department committee work.... j. Length of service at rank.... k. Public recognition 1. Personality Characteristics.. m. Other (please specify) IMPORTANT 2 2 2 2 2 2 2 2 2 2 2 NOT VERY IMPORTANT 3 3 3 3 NOT AT ALL IMPORTANT 4 4 4 4 4 4 4 4 4 4 4 ROLE OF STUDENT RATINGS (OVER) PAGE 5 129 2. What sources of information i n p r inc ipa l l y considered by you in the evaluation of a faculty member's teaching performance when a candidate i s to be appointed at or promoted to the rank of Associate Professor With Term (Without Tenure)? Please indicate the importance of each type of information source by placing a c i r c l e around one response in each row. VERY IMPORTANT IMPORTANT NOT VERY IMPORTANT NOT AT ALL IMPORTANT a. Formal Annual Teaching Evaluations by students (SCET) b. Formal Peer Reviews c. Informal Student Opinion d. Informal Opinions of Colleagues e. Evaluation by Head of Department f. Evaluation by Dean g. Opinion of Outside Experts... h. Self-evaluation or Report.... i . Course Syllabi and Examinations j. Student Examination Performance k. Grade Distributions 1. Enrollment in Elective Classes m. Long-term Follow-up of Students (Alumni Opinions) n. Research/Publication o. Other (please specify) 2 2 2 2 3 3 3 3 ROLE OF STUDENT RATINGS (CONTINUED) - PAGE 6 130 DECISIONS CONCERNING PROFESSORS WITH TERM (WITHOUT TENURE) The items on these yellou pages pertain to the appointment at or promotion to the rank of Professor Kith Term (Uithout Tenure). As you complete these pages, please give the response that best indicates hou you, typically, consider these factors Hhen making your decisions. How important i s each of the following factors when a candidate i s to be appointed at or promoted to the rank of Professor With Term (Without Tenure)? Please indicate how important each of these factors i s to you by placing a c i r c l e around one response in each row. VERY IMPORTANT a. Classroom teaching b. Research activity c. Publications d. Public service e. Consultation (governments/business) f. Activity in professional societies g. Supervision of graduate study h. Student program advising.... i . University, faculty or department committee work... j. Length of service at rank... k. Public recognition. 1. Personality Characteristics. m. Other .. (please specify) IMPORTANT 2 2 2 2 2 2 2 2 2 2 2 2 2 NOT VERY IMPORTANT 3 3 3 3 3 3 3 3 3 3 3 NOT AT ALL IMPORTANT 4 4 4 4 ROLE OF STUDENT RATINGS (OVER) PAGE 7 131 2. What sour C M of information are pr inc ipa l l y considered by you in the evaluation of a faculty member's teaching performance when a candidate i s to be appointed at or promoted to the rank of Professor With Term (Without Tenure)? Please indicate the importance of each type of information source by placing a c i r c l e around one response in each row. VERY IMPORTANT IMPORTANT NOT VERY IMPORTANT NOT AT ALL IMPORTANT a. Formal Annual Teaching Evaluations by students (SCET) b. Formal Peer Reviews c. Informal Student Opinion..... d. Informal Opinions of Colleagues e. Evaluation by Head of Department f. Evaluation by Dean g. Opinion of Outside Experts... h. Self-evaluation or Report.... i . Course Syllabi and Examinations j. Student Examination Performance k. Grade Distributions 1. Enrollment in Elective Classes m. Long-term Follow-up of Students (Alumni Opinions) n. Research/Publication o. Other _ • • • (please speci fy) 3 3 3 ROLE OF STUDENT RATINGS (CONTINUED) PAGE 8 132 DECISIONS CONCERNING APPOINTMENTS WITHOUT TERM (WITH TENURE) The Hens on these pink pages pertain to Appointments Uithout Term (kith Tenure). As you complete these pages, please give the response that best indicates hon you, typically, consider these factors uhen making your decisions. How important i s each of the following factors when a candidate i s to be appointed without term (With Tenure)? Please indicate how isportant each of these factors i s to you by placing a c i r c l e around one response in each row. VERY IMPORTANT a. Classroom teaching. b. Research activity c. Publications................ d. Public service e. Consultation (governments/business) f. Activity in professional societies g. Supervision of graduate study h. Student program advising.... i . University, faculty or department committee work... j. Length of service at rank... k. Public recognition 1. Personality Characteristics. m. Other ; .. (please speci fy) IMPORTANT 2 2 2 2 2 2 2 2 2 2 2 NOT VERY IMPORTANT 3 3 3 3 3 3 3 3 3 3 3 NOT AT ALL IMPORTANT 4 4 4 4 ROLE OF STUDENT RATINGS (OVER) PAGE 9 133 b. c. d. f. 9-h. i . k. 1. What sources of Information are pr inc ipa l l y considered by you in the evaluation of a faculty member's teaching performance when a candidate i s to be appointed without term (with Tenure)? Please indicate the importance of each type of information source by placing a c i r c l e around one response in each row. VERY IMPORTANT Formal Annual Teaching Evaluations by students (SCET) Formal Peer Reviews...... Informal Student Opinion. Informal Opinions of Colleagues. Evaluation by Head of Department Evaluation by Dean .., Opinion of Outside Experts., Self-evaluation or Report... Course Syllabi and Examinations Student Examination Per formance Grade Distributions. Enrollment in Elective Classes Long-term Follow-up of Students (Alumni Opinions) Research/Publication. Other IMPORTANT 2 2 2 2 2 (please specify) 2 2 2 NOT VERY IMPORTANT NOT AT ALL IMPORTANT At which rank(s) do you feel your responses with regard to the questions about Appointment Without Term (With Tenure) are most applicable/valid? _a. Assistant Professor c. Professor b. Associate Professor ROLE OF STUDENT RATINGS (CONTINUED) PAGE 10 134 III. STUDENT RATINGS OF TEACHING PERFORMANCE The follouing questions pertain to the Student Ratings of Teaching Performance (SCET) obtained in the Faculty of Education each year. 1. Check the alternative which best indicates how you feel. a. Student Ratings are collected to provide information to instructors for professional development. STRONGLY AGREE NEUTRAL DISAGREE STRONGLY AGREE DISAGREE • • • • • b. Student Ratings are collected to provide information to administrators to aid in personnel decisions. STRONGLY AGREE NEUTRAL DISAGREE STRONGLY AGREE DISAGREE • • . • • • c . Student Ratings are collected to provide students with information to use during the process of course and instructor selection. STRONGLY AGREE NEUTRAL DISAGREE STRONGLY AGREE DISAGREE • • • • • d. Student Ratings are collected to provide information to aid in the allocation of teaching or faculty resources. STRONGLY AGREE NEUTRAL DISAGREE STRONGLY AGREE DISAGREE • • • • • e. Student Ratings are collected to aid in the awareness, sensitivity, and appreciation of teaching. STRONGLY AGREE NEUTRAL DISAGREE STRONGLY AGREE DISAGREE • • • • • ROLE OF STUDENT RATINGS (OVER) PAGE 11 135 f. Student Ratings are collected to contribute to the understanding of the operation of the department or university as a whole. STRONGLY AGREE • AGREE • NEUTRAL DISAGREE STRONGLY DISAGREE • • • Student Ratings are collected to provide information for research on teaching. STRONGLY AGREE • AGREE NEUTRAL DISAGREE • • • STRONGLY DISAGREE • 2. Check the alternative which best indicates how you feel. Generally, how satisfied are you that the results of the Student Ratings process accurately reflects the teaching performance of faculty members? VERY SATISFIED SATISFIED NEUTRAL DISSATISFIED VERY DISSATISFIED • • • • • How valuable do you feel the information provided by Student Ratings is to the people who make personnel decisions? EXTREMELY VALUABLE VALUABLE NEUTRAL • • • NOT VERY VALUABLE • NOT AT ALL VALUABLE • ROLE OF STUDENT RATINGS (CONTINUED) PAGE 12 136 3. Please read each statement and then indicate your response by placing a check along the scale in the interval of your choice. a. How much weight do you feel should be given to Student Ratings of Teaching Performance by the people who make personnel decisions when they are evaluating a faculty member's teaching performance? 01 1- 11- 21- 31- 41- 51- 61- 71- 81- 91-101 201 30X 401 SOX 601 70X BOX 90X 100X b. How much weight do you feel is actual ly given to Student Ratings of Teaching Performance by the people who make personnel decisions when they are evaluating a faculty member's teaching performance? OX 1- 11- 21- 31- 41- 51- 61- 71- 81- 91-10X 20X 30X 40X 50X 60Z 70X 80X 901 100X c. How much weight do you feel should be given to teaching performance when a faculty member's overal l performance is being evaluated for the purpose of making personnel decisions? OX 1- 11- 21- 31- 41- 51- 61- 71- Bl- 91-10X 20X 30X 40X 50X 60X 70X BOX 90X 100X d. How much weight do you feel is actual ly given to teaching performance by the people who make personnel decisions when they are evaluating a faculty member's overal l performance? OX 1- 11- 21- 31- 41- 51- 61- 71- 81- 91-10X 20X 30X 40X 50X 601 70X 801 90X 100X ROLE OF STUDENT RATINGS 137 (OVER) PAGE 13 4. Please use the space below to write any additional comments you may have regarding the ways in which Student Ratings of Instructor Performance are used In personnel decision-making processes in th is faculty. ROLE OF STUDENT RATINGS PAGE 14 138 APPENDIX C Instructor Perceptions of the Role of Student Ratings of Teaching Performance 139 INSTRUCTOR PERCEPTIONS OF" THE ROLE OF STUDENT RATINGS OF" TEACHING PERFORMANCE Quest. # The purpose of this questionnaire is to determine the perceptions instructors hold about the role of Student Ratings of Teaching Performance (Annual Teaching Evaluations) in personnel decisions in the Faculty of Education. The personnel decisions referred to in this questionnaire are those which pertain to reappointment, promotion and tenure but not to merit pay or career progress increments. I. BACKGROUND INFORMATION The following information is for statistical and record keeping purposes only. Your responses are completely confidential and all results will be aggregated across respondents. Please checj( the appropriate response or supply the appropriate information. ' 1. With which department or school are you most c l o s e l y a s s o c i a t e d ? a. Department of A d m i n i s t r a t i v e , Higher and Adult Education b. Department of C o u n s e l l i n g Psychology c. Department of Educational Psychology and S p e c i a l Education d. Department of Language Education e. Department of Mathematics and Science Education f. Department of S o c i a l and Educational Studies g. Department of V i s u a l and Performing A r t s h. School of P h y s i c a l Education and Recreation Which of the f o l l o w i n g best d e s c r i b e s your rank w i t h i n t h i s f a c u l t y ? a. A s s i s t a n t Professor With Term (Without Tenure) b. A s s o c i a t e Professor With Term (Without Tenure) c. Professor With Term (Without Tenure) d. A s s i s t a n t Professor Without Term (With Tenure) e. A s s o c i a t e Professor Without Term (With Tenure) f. Professor Without Term (With Tenure) g. I n s t r u c t o r (With or Without Tenure) INSTRUCTOR PERCEPTIONS (OVER) PAGE 1 140 How long have you been a member of the Faculty of Education at U.B.C? a. less than 2 years b. 3 to 5 years c. 6 to 10 years d. more than 10 years 4. Have you ever participated in personnel decisions for the purposes of reappointment, promotion or tenure? (e.g., as a.member of a Standing Personnel Committee) YES NO 5. If YES to #4: Please identify the capacities in which you have participated in the personnel decision-making processes within this faculty. (Check all that apply to you). a. Member of a Departmental Standing Personnel Committee b. Member of the Faculty Standing Personnel Committee c. Member of the Senior Appointments Committee d. Head of Department e. Dean/Associate Dean/Assistant Dean f. 0ther._ (please specify) II. THE ROLE OF STUDENT RATINGS OF TEACHING PERFORMANCE IN PERSONNEL DECISIONS The purpose of the follouing section of the survey is to determine instructor perceptions of the role of Student Ratings of Teaching Performance (Annual Teaching Evaluations) in personnel decisions in the Faculty of Education. Part II of the survey consists of tuo subsections. The first subsection deals with Appointments Uitb Term while second deals uith Appointments Uithout Term because it is recognized that the c r i t e r i a considered may vary according to the type of personnel decision being made. It is hoped that by limiting each section to one s p e c i f i c type of personnel decision it uill make it easier for you to complete the survey. INSTRUCTOR PERCEPTIONS (CONTINUED) PASE 2 141 DECISIONS CONCERNING APPOINTMENTS WITH TERM (WITHOUT TENURE) The items an these pink pages pertain to Appointments Uith Term (Uithout Tenure). As you complete these pages, please give the responses that best indicate hou you feel these factors are typically considered by people uho make personnel decisions regarding your rank in the Faculty of Education. How important is each of the following factors when a candidate is to be appointed at or promoted to your rank With Term (Without Tenure?) Please indicate how important you feel each of these factors is to the people who make personnel decisions by placing a circle around one response in each row. a. Classroom teaching.... b. Research activity c. Publications d. Public service e. Consultation (governments/business) f. Activity in professional societies g. Supervision of graduate study h. Student program advising.... i . University, faculty or department committee work... j. Length of service at rank... k. Public recognition 1. Personality Characteristics. m. Other (please specify) VERY IMPORTANT IMPORTANT NOT VERY IMPORTANT 3 3 3 3 NOT AT ALL IMPORTANT 4 4 4 4 INSTRUCTOR PERCEPTIONS (OVER) PAGE 3 142 2. Whit t o u r C M of information are p r i n c i p a l l y considered i n the evaluation of a faculty member's teaching performance when a candidate i s to be appointed at or promoted to your rank With Term (Without Tenure)? Please indicate how important you feel each of these sources of information i s to the people who make personnel decisions by placing a c i r c l e around one response i n each row. VERY IMPORTANT a. Formal Annual Teaching Evaluations by students (SCET) b. Formal Peer Reviews c. Informal Student Opinion d. Informal Opinions of Col leagues e. Evaluation by Head of Department f. Evaluation by Dean g. Opinion of Outside Experts... h. Self-evaluation or Report.... i . Course Syllabi and Examinations j. Student Examination Per formance k. Grade Distributions 1. Enrollment in Elective Classes m. Long-term Follow-up of Students (Alumni Opinions) n. Research/Publication o. Other (please specify) IMPORTANT 2 2 2 2 2 NOT VERY IMPORTANT NOT AT ALL IMPORTANT INSTRUCTOR PERCEPTIONS 143 (CONTINUED) PAGE 4 DECISIONS CONCERNING APPOINTMENTS WITHOUT TERM (WITH TENURE) The items on these green pages pertain to Appointments Uithout Term (With Tenure). As you complete these pages, please give the response that best indicates hou you feel these factors are typically considered by people uho make personnel decisions regarding your rank in the Faculty of Education. 1. How important i s each of the following factors when a candidate i s to be appointed at or promoted to your rank Without Term (With Tenure)? Please indicate how important you feel each of these factors i s to the people who make personnel decisions by placing a c i r c l e around one response in each row. VERY NOT VERY NOT AT ALL IMPORTANT IMPORTANT IMPORTANT IMPORTANT a. Classroom teaching b. Research activity c. Publications d. Public service e. Consultation (governments/business) f. Activity in professional societies g. Supervision of graduate study h. Student program advising i . University, faculty or department committee work.... j. Length of service at rank.... k. Public recognition 1. Personality Characteristics.. m. Other • _ _ • • • (please specify) 2 2 2 2 INSTRUCTOR PERCEPTIONS (OVER) PAGE 5 144 2. What t o u r C M of information art p r inc ipa l l y considered in the evaluation of a faculty member's teaching performance when a candidate i s to be appointed at or promoted to your rank Without Term (With Tenure)? Please indicate how important you feel each of these sources of information i s to the people who make personnel decisions by placing a c i r c l e around one response in each row. VERY IMPORTANT IMPORTANT NOT VERY IMPORTANT NOT AT ALL IMPORTANT a. Formal Annual Teaching Evaluations by students (SCET) b. Formal Peer Reviews.... c. Informal Student Opinion d. Informal Opinions of Col leagues e. Evaluation by Head of Department f. Evaluation by Dean g. Opinion of Outside Experts... h. Self-evaluation or Report.... i . Course Syllabi and Examinations j. Student Examination Per formance k. Grade Distributions 1. Enrollment in Elective Classes m. Long-term Follow-up of Students (Alumni Opinions) n. Research/Publication o. Other (please speci fy) 2 2 INSTRUCTOR PERCEPTIONS (CONTINUED) PAGE 6 145 III. STUDENT RATINGS OF TEACHING PERFORMANCE The following questions pertain to the Student Ratings of Teaching Performance (SCET) obtained in the Faculty of Education each year. 1. Check the alternative which best indicates how you feel. Student Ratings are collected to provide information to instructors for professional development. STRONGLY AGREE • AGREE • NEUTRAL • DISAGREE • STRONGLY DISAGREE • Student Ratings are collected to provide information to administrators to aid in personnel decisions. STRONGLY AGREE • AGREE • NEUTRAL DISAGREE STRONGLY DISAGREE • • • Student Ratings are collected to provide students with information to use during the process of course and instructor selection. STRONGLY AGREE • AGREE • NEUTRAL DISAGREE STRONGLY DISAGREE • • • Student Ratings are collected to provide information to aid in the allocation of teaching or faculty resources. STRONGLY AGREE • AGREE • NEUTRAL DISAGREE STRONGLY DISAGREE • • • Student Ratings are collected to aid in the awareness, sensitivity, and appreciation of teaching. STRONGLY AGREE • AGREE • NEUTRAL • DISAGREE • STRONGLY DISAGREE • INSTRUCTOR PERCEPTIONS 146 (OVER) PAGE 7 f. Student Ratings are collected to contribute to the understanding of the operation of the department or university as a whole. STRONGLY AGREE NEUTRAL DISAGREE STRONGLY AGREE DISAGREE g. Student Ratings are collected to provide information for research on teaching. STRONGLY AGREE NEUTRAL DISAGREE STRONGLY AGREE DISAGREE • • • • • 2. Check the alternative which best indicates how you f e e l . a. Generally, how satisfied are you that the results of the Student Ratings process accurately reflects your own teaching performance? VERY SATISFIED NEUTRAL DISSATISFIED VERY SATISFIED DISSATISFIED b. Generally, how satisfied were you with your most recent Student Ratings of Teaching Performance? VERY SATISFIED NEUTRAL DISSATISFIED VERY SATISFIED DISSATISFIED • • • • • c. How useful were your most recent student ratings as feedback concerning your strengths and weaknesses? EXTREMELY USEFUL NEUTRAL NOT VERY NOT AT ALL USEFUL USEFUL USEFUL • • • • • d. How valuable do you feel the information provided by Student Ratings is to the people who make personnel decisions? EXTREMELY VALUABLE NEUTRAL NOT VERY NOT AT ALL VALUABLE VALUABLE VALUABLE • • • • • INSTRUCTOR PERCEPTIONS (CONTINUED) PAGE 8 147 3. Please read each statement and then indicate your response by placing a check along the scale in the interval of your choice. a. How much weight do you feel should be given to Student Ratings of Teaching Performance by the people who make personnel decisions when they are evaluating a faculty member's teaching performance? OX 1- 11- 21- 31- 41- SI- 61- 71- 81- 91-101 201 301 401 SOI 601 70! 801 901 1001 b. How much weight do you feel is actually given to Student Ratings of Teaching Performance by the people who make personnel decisions when they are evaluating a faculty member's teaching performance? 01 1- 11- 21- 31- 41- 51- 61- 71- 81- 91-10X 201 30! 401 50X 601 70X 80! 90! 100! c. How much weight do you feel should be given to teaching performance when a faculty member's overal l performance is being evaluated for the purpose of making personnel decisions? OX 1- -11- 21- 31- 41- 51- 61- 71- 81- 91-10X 20! 30X 40X SOX 60X 70X SOX 90X 100X d. How much weight do you feel is actual ly given to teaching performance by the people who make personnel decisions when they are evaluating a faculty member's overall performance? OX 1- 11- 21- 31- 41- 51- 61- 71- 81- 91-10X 20X 30X 40X SOX 60X 70X BOX 90X 100X INSTRUCTOR PERCEPTIONS (OVER) PAGE 9 148 Please use the space below to write any additional comments you may have regarding the ways in which Student Ratings of Instructor Performance are used in personnel decision-making processes in th is faculty. INSTRUCTOR PERCEPTIONS PAGE 10 149 APPENDIX D Cover letter from the researcher to the DMAs. Cover letter from the researcher to the Instructors. 150 APPENDIX E F i r s t follow-up l e t t e r to the DMAs. F i r s t follow-up l e t t e r to the Instructors. 153 APPENDIX F Final follow-up letter. 156 APPENDIX G Table G-l: Respondent Ratings (In Percent) f o r Factors Considered f o r Appointments to Assistant Professor Without Tenure Table G-2: Respondent Ratings (In Percent) of Sources of Information f o r Appointments to Assistant Professor Without Tenure Table G-3: Respondent Ratings (In Percent) of Factors Considered for Appointments to Associate Professor Without Tenure Table G-4: Respondent Ratings (In Percent) of Sources of Information for Appointments to Associate Professors Without Tenure Table G-5: Respondent Ratings (In Percent) of Factors Considered f o r Appointments to Professor Without Tenure Table G-6: Respondent Ratings (In Percent) of Sources of Information for Appointments to Professor Without Tenure Table G-7: Respondent Ratings (In Percent) of Factors Considered f o r Appointments to Professor With Tenure Table G-8: Respondent Ratings (In Percent) of Sources of Information f o r Appointments With Tenure Table G-9: Respondent Ratings (In Percent) of Purpose Statements 158 Table G-l: Respondent Ratings (In Percent) for Factors Considered for Appointments to Assistant Professor Without Tenure Response Category**  Factor 1 2 3 4 n Classroom teaching 46. 00 44. .00 10. .00 0. .00 50 26. ,32 36, .84 31, .58 5, .26 19 Research a c t i v i t y 64. 00 32. .00 4. .00 0. .00 50 63. .16 36, .84 0, .00 0 .00 19 Publications 42. 00 48. .00 10. .00 0. .00 50 73. ,68 26, .32 0, .00 0, .00 19 Public service 0. .00 26. .00 60. .00 14. .00 50 0. ,00 11, .11 55, .56 33, .33 18 Consultation 0. .00 14. .00 60. .00 26 .00 50 (governments/business) 0. ,00 0, .00 55 .56 44 .44 18 A c t i v i t y i n 8. .00 40. .00 46. .00 6. .00 50 professional s o c i e t i e s 0. .00 38, .89 44 .44 16 .67 18 Supervision of 18. .00 50. .00 22. .00 10. .00 50 graduate study 11. ,11 50, .00 33 .33 5, .56 18 Student program 4. .00 30. .00 46 .00 20. .00 50 advising 0. .00 10 .53 63 .16 26, .32 19 Committee work 0. .00 30. .00 50. .00 20. .00 50 0. .00 15, .79 69 .42 15, .79 19 Length of service 4. .08 22. .45 36 .73 36. .73 49 at rank 10. .53 10 .53 57 .89 21 .05 19 Public recognition 4. .26 31. .91 48 .94 14. .89 47 0. ,00 26, .32 57 .89 15. .79 19 Personality 14. .89 34. .04 27 .66 23. .40 47 Ch a r a c t e r i s t i c s 5. .26 36, .84 36 .84 21. .05 19 "1 (Very Important) - 4 (Not At A l l Important) DMAs in bold type/Instructors i n p l a i n type 159 Table G-2: Respondent: Ratings (In Percent) of Sources of Information for  Appointments to Assistant Professor Without Tenure Response Category'*  Source of Information 1 2 3 4 n Formal Student 40 .00 46. 00 8 .00 6. .00 50 Ratings 27 .78 55. 56 11 .11 5. .56 18 Formal Peer 40 .00 52. 00 6 .00 2. .00 50 Reviews 21 .05 63. 16 10 .53 5 .26 19 Informal student 4 .00 26. 00 52 .00 18. .00 50 opinion 0 .00 26. 32 57 .89 15 .79 19 Informal opinions 4. .00 44. 00 36 .00 16. .00 50 of colleagues 5 .26 47. 37 42 .11 5 .26 19 Evaluation by Head 6. .38 55. 32 29 .79 8. .51 47 of Department 35 .29 47. .06 17 .65 0 .00 17 Evaluation by Dean 6 .52 30. 43 32 .61 30. .43 46 20 .00 33. 33 26 .67 20, .00 15 Opinion of outside 32 .00 38. 00 22 .00 8. .00 50 experts 23 .53 35. 29 29 .41 11, .76 17 Self-evaluation or 2, .08 31. 25 47 .92 18. .75 48 report 5 .56 27. 78 33 .33 33, .33 18 Course s y l l a b i and 8 .16 57. 14 18 .37 16. .33 49 examinations 0 .00 44. 44 44 .44 11, .11 18 Student examination 0. .00 12. 77 63 .83 23. .40 47 performance 0 .00 5. 56 61 .11 33, .33 18 Grade d i s t r i b u t i o n s 0, .00 8. 33 62 .50 29. .17 48 0 .00 11. 11 61 .11 27, .78 18 Enrollment i n 0 .00 8. 51 57 .45 34. .04 47 e l e c t i v e classes 0. .00 6. 25 62 .50 31. .25 16 Long-term follow-up 4 .44 8. 89 46 .67 40. .00 45 of students 0, .00 0. 00 41 .18 58. .82 17 Research/Publication 25 .00 37. 50 10 .42 27. .08 48 42 .11 42. 11 5 .26 10, .53 19 "1 (Very Important) - 4 (Not At A l l Important) DMAs i n bold type/Instructors i n p l a i n type 160 Table G-3: Respondent Ratings (In Percent) of Factors Considered for  Appointments to Associate Professor Without Tenure Response Category"  Factor 1 2 3 4 n Classroom teaching 36. 96 58. 70 4. 35 0. .00 46 46. .67 26. .67 26. ,67 0. .00 15 Research a c t i v i t y 80. 43 17. 39 2. 17 0. .00 46 73. .33 13. ,33 13. ,33 0. .00 15 Publications 76. 09 21. 74 2. 17 0. .00 46 66. .67 13. ,33 20. 00 0. .00 15 Public service ' 2. .17 32. 61 47. 83 17. 39 46 13. .33 20. ,00 33. 33 33. ,33 15 Consultation 0. 00 28. 26 47. 83 23. 91 46 (governments/business) 0. .00 20. ,00 53. 33 26. ,67 15 A c t i v i t y i n 2. 17 56. 52 34. 78 6. 52 46 professional s o c i e t i e s 0. .00 53. .33 26. 67 20. ,00 15 Supervision of 32. 61 54. 35 10. 87 2. 17 46 graduate study 6. ,67 66. 67 20. 00 6. ,67 15 Student program 2. 17 36. 96 45. 65 15. 22 46 advising 0. .00 13. 33 53. 33 33. .33 15 Committee work 4. 35 54. 35 34. 78 6. 52 46 0. .00 26. 67 40. 00 33. .33 15 Length of service 6. 67 22. 22 46. 67 24. 44 45 at rank 0. ,00 33. 33 33. 33 33. ,33 15 Public recognition 8. 89 51. 11 20. 00 20. .00 45 20. ,00 46. ,67 26. 67 6. ,67 15 Personality 11. .90 23. 81 28. 57 35. 71 42 Ch a r a c t e r i s t i c s 13. ,33 46. 67 20. 00 20. ,00 15 *1 (Very Important) - 4 (Not At A l l Important) DMAs i n bold type/Instructors i n p l a i n type 161 Table G-4: Respondent Ratings (In Percent) of Sources of Information for Appointments to Associate Professors Without Tenure Response Category"  Source of Information 1 2 3 4 n Formal Student Ratings 41.30 40.00 45.65 40.00 8.70 20.00 4.35 0.00-46 15 Formal Peer Reviews 47.83 28.57 47.83 42.86 4.35 28.57 0.00 0.00 46 14 Informal student opinion 4.55 0.00 25.00 26.67 52.27 26.67 18.18 46 . 67 44 15 Informal opinions of colleagues 4.55 0.00 38.64 26.67 47.73 40.00 9.09 33.33 44 15 Evaluation by Head of Department 8.89 20.00 53.33 60.00 31.11 13.33 6.67 6.67 45 15 Evaluation by Dean 9.52 21.43 35.71 28.57 23.81 35.71 30.95 14.29 42 14 Opinion of outside experts 33.33 38.46 37.78 15.38 17.78 38.46 11.11 7.69 45 13 Self-evaluation or report 4.55 0.00 34.09 42.86 40.91 28.57 20.45 28.57 44 14 Course s y l l a b i and examinations 11.11 0.00 57.78 28.57 22.22 57.14 8.89 14.29 45 14 Student examination performance 4.44 0.00 24.44 28.57 51.11 42.86 20.00 28.57 45 14 Grade d i s t r i b u t i o n s 0.00 0.00 17.78 7.14 51.11 71.43 31.11 21.42 45 14 Enrollment i n el e c t i v e classes 2.38 0.00 11.90 28.57 54.76 28.57 30.95 42.86 42 14 Long-term follow-up of students 4.65 7.14 9.30 14.29 44.19 50.00 41.86 28.57 43 14 Research/Publication 38.64 35.71 25.00 42.86 13.64 14.29 22.73 7.14 44 14 *1 (Very Important) - 4 (Not At A l l Important). DMAs i n bold type/Instructors i n p l a i n type 162 Table G-5: Respondent Ratings (In Percent) of Factors Considered for  Appointments to Professor Without Tenure Response Category"  Factor 1 2 3 4 n Classroom teaching 47. .37 50. .00 2 .63 0. .00 38 50 .00 33, .33 16 .67 0, .00 6 Research a c t i v i t y 91. .89 8. .11 0 .00 0. .00 37 83 .33 16, .67 0 .00 0, .00 6 Publications 83. .78 16. .22 0 .00 0. .00 37 66 .67 0, .00 16 .67 16, .67 6 Public service 5. .41 43. .24 48. .65 2. .70 37 0, .00 16, .67 66, .67 16, .67 6 Consultation 2. .70 32. .43 48 .65 16. .22 37 (governments/business) 0, .00 50, .00 33 .33 16, .67 6 A c t i v i t y i n 8. .11 54. .05 35. .14 2. .70 37 professional s o c i e t i e s 0. .00 33, .33 50 .00 16. .67 6 Supervision of 64. .86 29. .73 2. .70 2. 70 . 37 graduate study 50 .00 16, .67 16 .67 16. .67 6 Student program 22. .22 30. .56 30. .56 16. .67 36 advising 16. .67 33, .33 33 .33 16. .67 6 Committee work 10. .81 45. .95 32 .43 10. .81 37 16 .67 33, .33 33 .33 16, .67 6 Length of service 5 .41 32. .43 32 .43 29. .73 37 at rank 0. .00 33, .33 33 .33 33, .33 6 Public recognition 27. .78 50. .00 8 .33 13. .89 36 33 .33 16 .67 33 .33 16 .67 6 Personality 5 .88 41. .18 17. .65 35. .29 34 Ch a r a c t e r i s t i c s 16. .67 33, .33 16 .67 33. .33 6 "1 (Very Important) - 4 (Not At A l l Important) DMAs in bold type/Instructors i n p l a i n type 163 Table G-6: Respondent Ratings (In Percent:) of Sources of Information f o r  Appointments to Professor Without Tenure Response Category"  Source of Information 1 2 3 4 n Formal Student 50. .00 36. .84 7. .89 5. .26 38 Ratings 66. .67 33. .33 0. .00 0. ,00 6 Formal Peer 51. .35 43. .24 5. .41 0. .00 37 Reviews 50. .00 50, .00 0. .00 0. ,00 6 Informal student 5. 56 25. .00 50. .00 19. .44 36 opinion 0. .00 33, .33 66. .67 0. .00 6 Informal opinions 5. .56 41. .67 38. .89 13. .89 36 of colleagues 0. .00 33, .33 66. ,67 0. ,00 6 Evaluation by Head 16. .67 36. .11 33. .33 13. .89 36 of Department 33. 33 66, .67 0. ,00 0. ,00 6 Evaluation by Dean 14. 71 29. .41 14. .71 41. .18 34 0. .00 83, .33 16. ,67 0. ,00 6 Opinion of outside 47. .22 25. .00 16. .67 11. .11 36 experts 16. .67 66, .67 0. ,00 16. .67 6 Self-evaluation or 2. 86 31. .43 40. .00 25. .71 35 report 16. .67 33, .33 50. .00 0. .00 6 Course s y l l a b i and 8. .33 58. .33 19. .44 13. .89 36 examinations 0. .00 83, .33 16. .67 0. .00 6 Student examination 0. .00 22. .86 45. .71 31. .43 35 performance 0. .00 100 .00 0. .00 6 Grade d i s t r i b u t i o n s 0. .00 20. .00 54. .71 34. .29 35 0. .00 16 .67 83. .33 0. .00 6 Enrollment i n 5. .71 14. .29 45. .71 34. .29 35 e l e c t i v e classes 16. .67 83, .33 0. .00 0. ,00 6 Long-term follow-up 8. .33 16. .67 36. .11 38. .89 36 of students 16. .67 50, .00 33. .33 0. .00 6 Research/Publication 44. .44 16. .67 13. 89 25. .00 36 50. .00 16, .67 33. .33 0. .00 6 "1 (Very Important) - 4 (Not At A l l Important) DMAs i n bold type/Instructors i n p l a i n type 164 Table G-7: Respondent Ratings (In Percent) of Factors Considered for  Appointments to Professor With Tenure Response Category"  Factor 1 2 3 4 n Classroom teaching 61. 22 32. 65 6. .12 0. .00 49 32. .56 27. .91 34, .88 4. .65 43 Research a c t i v i t y 81. 63 18. 37 0. .00 0. .00 49 76. ,74 23. ,26 0, .00 0, .00 43 Publications 77. 08 22. 92 0. .00 0. .00 48 79. ,07 20. 93 0, .00 0. .00 43 Public service 4. 08 34. 69 48. .98 12. .24 49 4. ,65 34. 88 32, .56 27. .91 . 43 Consultation 0. 00 30. 61 42. .86 26. .83 49 (governments/business) 0. ,00 33. ,33 38, .10 28. .57 42 A c t i v i t y i n 4. 08 55. 10 36. .73 4. .08 49 professional s o c i e t i e s 7. ,14 52. ,38 21, .43 19. .05 42 Supervision of 29. 17 60. 42 6. .25 4. 17 48 graduate study 20. ,93 58. ,14 13, .95 6. .98 43 Student program 2. 13 40. 43 38. .30 19. 15 47 advising 0. ,00 28. 57 35, .71 35. ,71 42 Committee work 0. 00 62. 50 29. .17 8. ,33 48 2. ,33 34. ,88 39, .53 23. ,26 43 Length of service 6. .00 28. 00 40. .00 26. .00 50 at rank 6. ,98 18. ,60 51, .16 23. ,26 43 Public recognition 6. .25 54. 17 20. .83 18. .75 48 14. ,63 36. ,59 26, .83 21. ,95 41 Personality 8. .70 32. 61 32. .61 26. .09 46 Ch a r a c t e r i s t i c s 9. ,30 34. 88 32. .56 23. ,26 43 "1 (Very Important) - 4 (Not At A l l Important) DMAs in bold type/Instructors i n p l a i n type 165 Table G-8: Respondent Ratings (In Percent) of Sources of Information for Appointments With Tenure Response Category"  Source of Information 1 2 3 4 n Formal Student Ratings Formal Peer Reviews 52.08 37.21 50.00 32.56 35.42 46.51 45.83 46.51 8.33 9.30 4.17 13.95 4.17 6.98 0.00 3.98 48 43 48 43 Informal student opinion Informal opinions of colleagues Evaluation by Head of Department Evaluation by Dean 4.26 2.33 4.17 6.98 15.56 40.48 11.36 25.00 23.40 18.60 45.83 27.91 44.44 38.10 25.00 30.00 55.32 48.84 41.67 48.84 33.33 14.29 31.82 30.00 17.02 30.23 8.33 16.28 6.67 7.14 31.82 15.00 47 43 48 43 45 42 44 40 Opinion of outside experts Self-evaluation or Report Course s y l l a b i and examinations 31.91 21.43 2.17 2.38 17.02 2.44 36.17 35.71 41.30 26.19 46.81 29.27 25.53 28.57 39.13 33.33 25.53 51.22 6.38 14.29 17.39 38.10 10.64 17.07 47 42 46 42 47 41 Student examination performance Grade d i s t r i b u t i o n s 0.00 0.00 0.00 0.00 23.91 26.19 21.74 19.05 52.17 40.48 52.17 52.38 23.91 33.33 26.09 28.57 46 42 46 42 Enrollment i n el e c t i v e classes 4.55 0.00 13.64 27.50 50.00 42.50 31.82 30.00 44 40 Long-term follow-up of students 6.67 7.14 15.56 14.29 37.78 30.95 40.00 47.62 45 42 Research/Publication 42.22 58.54 24.44 21.95 "1 (Very Important) - 4 (Not At A l l Important) DMAs i n bold type/Instructors i n p l a i n type 13.33 14.63 20.00 4.88 45 41 166 Table G-9: Respondent Ratings (In Percent) of Purpose Statements Response Category" Purpose 1 2 3 4 5 n Purpose Ata 23.53 39.22 9.80 13.73 13.73 51 31.91 38.20 2.13 14.89 12.77 47 Purpose B = 39.22 45.10 9.80 5.88 0.00 51 32.61 45.65 15.22 4.35 2.17 46 Purpose C d 1.96 9.80 29.41 33.33 25.49 51 2.22 24.44 26.67 31.11 15.56 45 Purpose D" 2.00 20.00 22.00 44.00 12.00 50 6.52 15.22 28.26 32.61 17.39 46 Purpose E* 10.20 30.61 24.49 20.41 14.29 49 4.44 44.44 8.89 24.44 17.78 45 Purpose F E 1.96 13.73 25.49 43.14 15.69 51 4.55 9.09 27.27 45.45 13.64 44 Purpose G h 1.96 9.80 25.49 50.98 11.76 51 0.00 18.18 29.55 34.09 18.18 44 DMAs i n bold type/Instructors i n p l a i n type "1 (Strongly Disagree) - 5 (Strongly Agree) taTo provide information to instr u c t o r s f o r professional development. cTo provide information to administrators to a i d i n personnel decisions. dTo provide students with information to use during the process of course and i n s t r u c t o r s e l e c t i o n . "To provide information to a i d i n the a l l o c a t i o n of teaching or f a c u l t y resources. *To a i d i n the awareness, s e n s i t i v i t y , and appreciation of teaching. eTo contribute to the understanding of the operation of the department or u n i v e r s i t y as a whole. HTo provide information f o r research on teaching. 167 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0097732/manifest

Comment

Related Items