Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Cognitive strategies in judgment : the effect of purpose, cue dimensionality, and cognitive complexity.. Kishor, Nand 1987

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata


UBC_1987_A1 K58.pdf [ 6.95MB ]
JSON: 1.0054638.json
JSON-LD: 1.0054638+ld.json
RDF/XML (Pretty): 1.0054638.xml
RDF/JSON: 1.0054638+rdf.json
Turtle: 1.0054638+rdf-turtle.txt
N-Triples: 1.0054638+rdf-ntriples.txt
Original Record: 1.0054638 +original-record.json
Full Text

Full Text

COGNITIVE STRATEGIES IN J U D G M E N T : T H E E F F E C T OF PURPOSE, C U E DIMENSIONALITY, A N D COGNITIVE COMPLEXITY ON STUDENT E V A L U A T I O N O F INSTRUCTORS by NAND KISHOR B.Ed., M.A. The University of the South Pacific A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR T H E DEGREE OF DOCTOR OF PHILOSHOPHY  in THE FACULTY OF GRADUATE STUDIES EDUCATIONAL PSYCHOLOGY & SPECIAL EDUCATION  We accept this thesis as conforming to the required standard  THE UNIVERSITY OF BRITISH COLUMBIA September 22,  1987  © NAND KISHOR,  1987  In  presenting  degree freely  at  this  the  available  copying  of  department publication  of  in  partial  fulfilment  University  of  British  Columbia,  for  this or  thesis  reference  thesis by  this  for  his  and  scholarly  or  thesis  study.  her  for  I  of I  further  purposes  gain  that  agree be  It  is  shall  requirements  agree  may  representatives.  financial  the  not  that  the  Library  permission  granted  by  understood be  allowed  permission.  Department of  Educational  T h e U n i v e r s i t y o f British 1956 Main Mall Vancouver, Canada V6T 1Y3 D  a  t  e  DE-6(3/81)  September  Psychology & Special  Columbia  22nd.,  1987.  for  Education  an  advanced  shall for  the that  without  head  make  it  extensive of  my  copying  or  my  written  ABSTRACT This investigation focused on describing cognition in performance judgment of teaching  in  higher  dimensionality  The  observed  on  was  information. purpose  education.  Information  and  measurement  integration  cognitive  experiment:  were  summative  and  appraisal  importance  strategies  were  Exploratory  schema  complexity on halo in performance  of  subjective  complexity.  of good instructor  Seventy subjects  influence  purpose and  examined analysis  profiles, and  and  utilization in  relation  focused  on the  cue  effect  on  of to the  of cognitive  ratings.  assigned randomly to two purpose formative judgment.  conditions in the  Two questionnaires,  two  rating  tasks, and a Role Construct Repertory grid were adminstered  for data collection.  The data were analyzed through regression modeling at the  individual level and  via analysis of variance procedures at the group level.  The  results  indicate  that  the  impact  of  cue  dimensions  is  strong  on  subjective importance and utilization of information but varies with the purpose of appraisal.  Raters  information feedback  valued  and  utilized  trait  more  than  behavior  in evaluation required for personnel decisions. Where evaluation  on the  quality of teaching  and  expressed  raters utilized behavior information more than information judgment  information  utilization is  a  function  suggests  that  of purpose  cue  need  for  improvement,  trait information. This pattern of  saliency  and  the  was  of  information  in  dimensionality, and  performance  that  appraisal  purpose has an effect on raters' cognition through schematic processing.  The  results  also  show  that  the  use  of  varied  strategies  in  mentally  integrating dimensions of information is affected Although used  subjects  mainly  noncompensatory  used  compensatory  strategies  cognitive complexity also affects  as  well.  by raters' cognitive complexity.  strategies,  the  Exploratory  complex individuals  analysis  shows  that  halo in rating judgments. The findings seem to  support the validity of student rating of instructors, and the utility of cognitive complexity construct in understanding performance judgment.  It is suggested that the influence of schematic processing and cue saliency be  addressed  in further  theorizing and  research  on performance  judgment.  As  well, the inclusion of purpose of judgment and developmental constructs, such as cognitive  complexity, is  recommended  for  processes.  iii  theorizing and  research  on judgment  TABLE OF CONTENTS Abstract  ii  List of Tables  vi  List of Figures  vii  Acknowledgements  viii  I. T H E A. B. C. D. E.  N A T U R E OF T H E STUDY Introduction The Problem The Purpose of the Study The Context Significance of the Study  1 1 4 8 10 11  II. R E V I E W O F T H E L I T E R A T U R E A. Influences on Judgment Processes 1. Cognitive Limitations 2. Mental Models 3. Judgment Task Environment B. Cognitive Views of Performance Judgment 1. Cognitive Distortion in Performance Judgment 2. Performance Appraisal as Person Perception 3. Performance Appraisal as Prototype Matching 4. Performance Appraisal as Social Perception C. Research on Raters' Cognition 1. Cognitive Effect of Appraisal Purpose 2. Effect of Cue Dimensionality 3. Influence of Cognitive Complexity D. Evaluation of Teaching E . Summary  13 13 13 15 17 19 20 21 22 24 26 26 32 33 38 41  III. H Y P O T H E S E S , Q U E S T I O N S , A N D M E T H O D A. Rationale for Hypotheses and Method '. B. Hypotheses and Exploratory Questions 1. Importance of Information 2. Utilization of Information 3. Information Importance and Utilization Consistency 4. Information Integration 5. Exploratory Questions C. Methodology 1. Subjects 2. Instruments a. Importance of Information Measure b. Rating Judgment Task A c. Cognitive Complexity Measure d. Good Instructor Schema Measure e. Rating Judgment Task B  44 44 48 48 49 49 49 50 50 50 51 51 52 55 57 59  iv  3. Experimental Design and Variables 4. Data Collection  60 62  IV. A N A L Y S I S A N D R E S U L T S A. Test of the Hypotheses 1. Importance of Information 2. Utilization of Information 3. Subjective Importance and Utilization Consistency 4. Information Integration B. Exploratory Analysis 1. Measuring A Good Instructor Schema 2. Cognitive Complexity and Halo  64 64 65 69 73 75 82 82 84  V. DISCUSSION A. Importance and Utilization of Information 1. Effect of Purpose 2. Effect of Cue Dimensionality 3. Interactive Effect of Purpose and Cues 4. Subjective Importance and Utilization Consistency B. Information Integration 1. Effect of Purpose 2. Effect of Cognitive Complexity C. Findings From Exploratory Analysis 1. Measurement of Schema 2. Cognitive Complexity and Halo D. Summary of the Findings and Conclusions E . Strengths and Limitations of the Study F. Implications G. Directions for Further Research  86 86 86 88 90 93 94 96 97 98 98 100 102 105 107 113  VI. R E F E R E N C E S  117  VII. A P P E N D I X A. Glossary B. Important Information Measure 1. For Summative Condition 2. For Formative Condition C. Performance Rating Task A 1. For Summative Condition 2. For Formative Condition 3. Coding and Rotation of Profiles D. Cognitive Complexity Measure E. Good Instructor Schema Measure F. Performance Rating Task B  v  132 132 134 134 135 136 136 138 140 141 142 147  LIST OF TABLES Table  1: The  Effect  of Purpose  and  Cue Dimensionality on Importance  Table 2: Mean Importance of Performance Table  Ratings 66  Related Information  68  3: Mean, Median and Range of Variance Explained by Regression Models 70  Table 4: The Effect of Purpose and Cue Dimensionality on Information  Utilization 71  Table 5: Mean Regression Weights for Formative and Summative Judgment Table  6: Variance Variates  Extracted  Table 7: Mean and Standard  from  Original  Sets  of  Variables  Deviation of Variance Explained  Table 8: Mean Diagnostic Ratios in Good Instructor Schema Profile  vi  by  .... 73  Canonical 74 75 83  LIST OF FIGURES Fig. 1: Mean Importance Rating of Cues  67  Fig. 2: Mean Weight of Information Utilized  72  Fig. 3: Plot of Cell Means for Subjects 3 and 8 (Formative)  77  Fig. 4: Plot of Cell Means for Subjects  78  13, 27, 29 and 30 (Formative)  Fig. 5: Plot for Cell Means of Subjects 2, 5, 6 and 15 (Summative)  79  Fig. 6: Plot for Cell Means of Subjects  80  19, 31, 32 and 33 (Summative)  vii  ACKNOWLEDGEMENTS I thank  my program  for his continued support,  advisor and research encouragement,  supervisor, Dr. Marshall  and guidance throughout  Arlin,  my tenure  at  U B C . Thanks also to members of my dissertation committee, Dr. Ronald Jarman and Dr. Peter  Grimmett, for  their  helpful comments  in the  completion of this  dissertation.  As well, thanks to Professors Walter Boldt, Seong-Soo Lee, Anne Triesman, Nancy and  Suzuki,  Ian  Patricia Arlin,  Housego  for  Daniel Kahneman, Robert Conry, Helga  providing diverse  and  rich  intellectual  was  affiliated  Jacobson,  stimulation  that  contributed to my development.  During  the  course  of my  study  I  to  The  Center  for  the  Study of Teacher Education (CSTE) at U B C . This dissertation is a result of the cooperation  between  the  Department  of  Educational  to  CSTE  Psychology  &  Special  Education and C S T E . An  acknowledgement  Educational Canadian  Research  is  Services  International  offered &  for  Computing (UBC) for  Development Research  Center  a  Doctoral Research  for  a  Fellowship,  Assistantship,  Doctoral Fellowship,  and The University of the South Pacific (Fiji) for study leave.  No dissertation would be completed without the care and concern of many friends. mention.  To I  them, am  I  express  greatly  my  indebted  sincere to  Dr.  gratitude. Daniel  Two  Birch  persons  (UBC) and  need  special  Dr. Diana  Kendall (Canberra) for the inspiration and support they provided me to study in this part of the world.  On a more personal note, I am grateful to my parents for instilling in me the  belief best  summarized  in the  words  "awake,  arise,  and  stop  not  till  the  goal is reached." M y achievement is dedicated to them. To Lila, L i n , and Kevin I owe too much for their patience, love and care. viii  I.  A.  T H E NATURE  O F T H E STUDY  INTRODUCTION  Human judgment, from  cues  or  phenomenon choice,  the  information  of every  and the  mental to  human  act  make  of weighing and  an  experience.  selection of the  inference  about  combining information some  criterion,  is  a  It includes evaluation, decision making,  response  after  perceiving the  stimulus. Human  judgment is an inescapable aspect of thinking. As a result, it is being studied by researchers from various disciplines. Within the psychological literature, two lines of  inquiry  on  how  approach  has  been  judgment;  another  we  make  judgments  and  decisions  are  identifiable.  One  to apply prescriptive models of choice for predicting human approach has been to describe human judgment  as  constrained  by cognitive mechanisms.  Prescriptive models, though successful in describing simpler automatic processes,  fail  to describe judgments  mental  that require thoughtful deliberations (Pitz &  Sachs, 1984). Prescriptive models of human judgment are derived from probability theory and from Expected Utility theory (von Neumann & Morgenstern, 1947). A hybrid of these theories is the Bayesian decision theory (Edwards, 1968; Raifa & Schlaifer,  1961), widely applied in the  study  of predictive judgment. Prescriptive  models provide a set of rules for combining beliefs and preferences judgment,  but  as  descriptions  of human judgment  setbacks (Einhorn & Hogarth, 1981; Schoemaker, axioms  are  violated in human  induction  Tversky & Kahneman, 1983).  1  in making a  prescriptive models have  had  1982). A number of prescriptive  (Kahneman,  Slovic,  & Tversky,  1982;  THE N A T U R E OF T H E STUDY / 2 Another approach has been to describe human judgment as affected by the cognitive  functioning  inquiry have  of  the  human  mind. Theoretical insights  in  this  line of  been gained from the recognition that human behavior depends on  the nature of the environment, the nature of the organism, and the means the organism  has  developed  for  coping  with  the  environment  Hogarth, 1981; Piaget, 1936/1970; Simon & Newell,  (Brunswik,  1952;  1971). Within this approach,  the main thrust has been to uncover how information processing mechanisms of the mind constrain judgments.  A number of errors  and inconsistencies in human  judgment have been identified (for reviews see Einhorn & Hogarth, 1981; Slovic, Fischhoff, attributed  &  Lichtenstein,  to  cognitive  1977).  These  limitations,  errors  and  judgmental  inconsistencies heuristics,  have  and  been  schematic  processingt (Hogarth, 1980; Kahneman, et al., 1982, Taylor & Crocker, 1981).  The certain  present study  factors  affect  was  an  attempt  to  enrich  our  understanding  cognitive processing of information in evaluative  According to a conceptualization by Hogarth (1980), judgment a  system  composed of three elements.  of how judgment.  takes place within  First, is the person; second, is the  task  environment within which the person makes judgments; and third, are the actions that result the  from judgment  environment.  complexity  or  developmental  This  the aspect  which  study  disposition of  information and purpose  the  may  focused for  the  on  the  processing  person;  cue  affect  first  two  both the  mulitdimensional  dimensionality  person  elements.  or  the  and  Cognitive  data,  for judgment were two aspects of the task  The task environment was performance  tTerms peculiar to Appendix A .  subsequently  was  a  nature  of  environment.  judgment.  domain of this  study  are  included in the  glossary in  THE NATURE OF THE STUDY / 3 Performance management. training  judgment  It  and  is  central  development,  evaluate the performance  plays to  an  important  part  decisions  related  important  promotion  and  career  in  human  to  staff  planning.  resources recruitment,  The  pressure  to  of individuals usually develops from productivity concerns  in an industrial setting and accountability concerns in professional occupations like teaching.  Not surprisingly, since Thorndike's  supervisors' ratings  of their subordinates,  The literature concerning performance attempting  to  improve  ratings  study  of the  considerable effort  eliminating errors and biases in performance  studies  (1920)  has  halo effect  been devoted to  evaluation.  evaluation is replete with psychometric  and  rating  scales  (Borman  1975; Dickinson & Zellinger, 1980). A large number of performance and instruments  have been devised. A fair number  of studies  &  Dunnette,  rating  have  and the halo effect  scales  also focused  on rater training in order to find ways to eliminate common rating errors as leniency-stringency, central tendency,  in  such  (Bernardin & Pence,  1980).  Despite  much  the progress DeCotiis  and effort,  there is considerable  made in the field of performance  & Petit,  comprehensive approaches  interest  1978;  review  in  article,  resolving  appreciably successful,  Kane  & Lawler, Landy  problems  and in  and any further  dissatisfaction  appraisal research  (Borman, 1978;  1979; Landy & Farr, Farr  (1980)  performance research  concluded  appraisal  1980). that  have  on improving the  and rater training would probably be a futile endeavor.  with  In  a  previous not  rating  been format  They stated, "It is time  to stop looking at the symptoms of bias in rating and begin examining potential causes"  (p.  101).  They  suggested  that further  research  should  examine  raters'  THE N A T U R E OF T H E STUDY / 4 cognition.  Similar  suggestions  have  been  made  by  others  as  well  (DeNisi,  Cafferty, & Meglino, 1984; Feldman, 1981; Ilgen & Feldman, 1983).  Researchers in the past have not paid much attention to a rater's cognition in  performance judgment.  Expressing this concern, Wexley  compared traditional performance  appraisal research  and Klimoski  to a black box approach to  describing human behavior, where a worker's performance represents the the  rating  represents  the  response,  and  errors  weaknesses in the instruments, or lack of rater processing in performance judgment  seems  (1984)  and  biases  are  stimuli, seen  as  training. Understanding cognitive  critical  to our  understanding  of the  problems in performance appraisal in particular, and human judgment in general.  B. THE PROBLEM Most of the available research on judgment processes deals with probabilistic inference in gambles, business and medical decision making, and risk perception. Evaluative  judgment  context for research major  review papers  such  as  performance  on judgment processes (Einhorn  & Hogarth,  appraisal in the  has  not  1981; Pitz  & Sachs,  & Lichtenstein, 1977). However, there is a need  processes  in  appraisal,  because  a  popular  past, as can be seen  Fischhoff,  performance  been  inconsistencies  1984;  from Slovic,  to study judgment and  biases  are  a  pervasive problem in this area (Landy & Farr, 1980).  Previous research on judgment strategies stimulus  features  in a judgment  task.  has concentrated on a number of  These include cue inter-relationships, set  size effects, number and format of cues, cue response compatibility, extremity of  THE NATURE OF T H E STUDY / 5 information, cue redundancy, and primacy and recency effects on judgment (for a review that  see  Slovic  other  & Lichtenstein,  features  information  of  presentation  dimensionality  the  stimuli  (Crocker,  (Wallsten,  1971). More recently, researchers  1980)  such  as  information  1981; Tversky also  affect  load  have  (Payne,  found 1976),  & Kahneman, 1981), and cue  cognitive processes  that  produce  a  effect  of  judgmental response.  Nevertheless,  relatively  little  attention  has  been  paid  to  the  purpose of judgment in the study of mental strategies in judgment. It has been stressed  that  judgment  is  Hogarth,  1978;  Hogarth,  purposes),  how judgments  primarily 1980). are  exercised  Because  formed  may  to  facilitate  an  action  be  influenced  action  serves by  (Einhorn  &  a  purpose  (or  the  purpose  for  which a judgment is required. It is hard to think of a situation where evaluative judgment does not serve a purpose. Whether purpose may  formal  determine  the type of information necessary for a judgment, and thereby have an effect on the utilization of cues needs to be examined.  A affects has  common result the  been  use  in research  of information  considered  mainly  on cue dimensionality is that cue saliency  in judgments. in  terms  of  Cue saliency in previous research number,  frequency,  and  perceptual  characteristics of the stimuli. The emerging general theory is open as to what determines cue saliency (Wallsten & Barton, 1982). In certain judgment situations, cue dimensionality may be reflected in information content. For example, cues in performance  judgment  Hence, research  provide information  is needed  to  determine  concerning traits whether  information content and the purpose of judgment.  and  role behaviors.  cue saliency is a function  of  THE NATURE OF T H E STUDY / 6 Furthermore, in their review paper Pitz and Sachs (1984) have noted that developmental  constructs  have  been  largely  ignored  judgment processes. These authors drew attention  in  the  Cognitive complexity, a developmental construct, processing  Tripodi,  multidimensional  1966).  Therefore,  data  (Bieri,  whether  human level  cognitive complexity.  relates to a person's disposition  Atkins,  cognitive  of  to the role of a person's  of moral development, but another factor may be a person's  in  study  Briar,  complexity  Leaman, affects  Miller,  &  use  of  the  information integration strategies needs examination.  Researchers be adequately &  Tversky,  of human judgment  1982). For  a  meaningful  and  purpose  afforded  researchers have  (Zedeck  & Cascio,  rating judgments. behaviors  (Wexley  Further,  already 1982)  & Klimoski,  have  studying performance judgment. aspects "treats"  in  processing  (mentally  a  and  formal rules" (Kahneman issues  raised  to study  the  above,  role of appraisal  cognitive complexity (Schneier,  1984),  it  judgment  is based  provided  an  proposed  on traits  environment  information  processing  Landy and Farr (1980) suggested  the  1977) on and role  where  the  by information content could be studied.  rating judgment  integrates)  cannot  are largely unknown (Landy & Farr,  attempted  as performance  researchers  reasoning  an ideal task environment. Besides, the causes of  effect of cue dimensionality represented  Several  that "human  investigation of the  and biases in performance judgment  1980),  noted  described in terms of content-independent  performance judgment errors  have  were  available  the  manner  information  and  approaches  to  that important  in which the  a  rater  purpose  of  appraisal. Cooper (1981) proposed that cognitive distortion, introduced by a rater's beliefs and implicit theories  in the processing stages, was the  source of halo in  THE N A T U R E OF T H E STUDY / 7 performance  ratings.  cognitive structures Meglino  (1984)  Ilgen  and  and the  Feldman  prototype  suggested  the  (1983)  drew  attention  matching processes.  importance  of  a  rater's  memory,  processing of performance  these  on  a  common  emphasis  is  the  effect  of  a  rater's  DeNisi, Cafferty,  complexity, and cognitive style on the views,  to  and  cognitive  information. In  purpose  on  a  rater's  cognition, and on the effect of a rater's cognitive complexity on rating properties.  Nevertheless, raters'  cognition  Hassett,  1984;  the is  available research  not  Murphy,  only  limited,  Balzer,  findings but  Kellam,  also  on the  effect  unclear  (Mclntyre,  & Armstrong,  of purpose Smith,  1984; Williams,  cognitive  purpose.  Likewise,  performance Schneier,  the  ratings, 1977;  researchers ratings,  processing  the  although  performance  findings  are  Lahey  in  in  the  contradictory &  past  on  Saal,  have  judgment of  (Bernardin,  1981;  focused  Sauser mainly  cognitive complexity may  the construct represents a person's  effect  affect  is  cognitive  Cardy, &  on  affected  clarify  appraisal  complexity  &  Pond,  by  Carlyle,  1981).  psychometric  &  DeNisi,  Blencoe, & Cafferty, 1985; Zedeck & Cascio, 1982). These studies fail to how  on  on  1982;  Moreover,  properties  of  information integration because  disposition in processing multidimensional data  (Bieri, et al., 1966).  Futhermore, neither the cognitively oriented theoretical models (Cooper, 1981; DeNisi  et  al.,  1984;  Ilgen & Feldman,  1983; Landy & Farr,  1980), nor  empirical studies on raters' cognition (Mclntyre, Smith, & Hassett,  the  1984; Murphy,  Balzer, Kellam, & Armstrong, 1984; Williams, DeNisi, Blencoe, & Cafferty, 1985; Zedeck  & Cascio,  performance  1982)  judgment.  have  addressed  However,  cue  the  influence  dimensionality  of cue  has  been  dimensionality in found  to  affect  THE NATURE OF T H E STUDY / 8 judgment 1984).  strategies  Important  in other cue  areas (Slovic & Liechtenstein, 1971; Pitz  dimensions in performance  judgment  are  & Sachs,  traits  and  role  behaviors because  "what a person is" and "what a person does" make up the  appraisal  (cf.  content  demonstrate  Wexley  &  Klimoski,  1984).  There  is  little  research  to  empirically the conditions under which trait and behavior information  become salient. Therefore, whether trait and behavior cues bear  an influence on  the utilization of performance information needs investigation.  In mental  essence, processes  purpose  we that  determines  thereby, affects  do not lead  the  have to  type  a  clear  understanding  a judgment of  information  of influences on  in performance necessary  for  the  appraisal. Whether a  judgment,  the utilization of cues, lacks evidence. Further, we lack  and  theories  that may explain what determines cue saliency in performance judgment, and we lack  evidence on whether  cognitive complexity affects  the' way  raters mentally  combine performance information.  C. T H E P U R P O S E  OF T H E STUDY  This investigation tested  hypotheses  pertaining to the influence of appraisal  purpose, cue dimensionality, and cognitive complexity on how rating judgments formed. Specifically,  the  effects  of these  variables were  examined on  are  subjective  valuation, utilization, and integration of information - the processes that lead to a rating response (Anderson, 1981).  The importance  effects and  of purpose utilization  and cue dimensionality were observed on subjective  of performance  related  information. Rating  judgments  THE were  required  performance  for  the  purposes  of  - formative judgment,  (a)  NATURE  OF T H E STUDY  providing feedback  on  the  / 9  quality of  and (b) indicating the suitability for promotion  - summative judgment. The ratings - were an expression of judgment only, and did not  include  any  justification,  guidance,  and  recommendations  for  improvement;  because the main interest in this study was on how rating judgments are formed and  not  on  how  evaluations  dimensionality was represented purpose of  are  to  be  communicated  for  development.  Cue  by trait and role information. The predictions that  and cue dimensionality will influence subjective importance and utilization  information,  and  that  cue  utilization  will  be  consistent  with  subjective  importance of cues, were examined.  The effects of  of purpose  cue integration  strategies.  broad types: compensatory use  when  cues  are  noncompensatory mulitiplicatively  and cognitive complexity were The information integration  and noncompensatory.  combined  strategy across  is  additively or in  dimensions  use  when  by  averaging are  1970,  use  strategies were of two  A compensatory  cues  (Einhorn,  observed on the  across  combined  1971;  strategy  is in  dimensions; interactively  Hogarth,  1980).  predictions that appraisal purpose, and cognitive complexity will influence the  a or  The use  of information integration strategies were examined.  In addition, this study served two exploratory purposes. One was to explore the  measurability  characteristics  of  of good instructor a  good  instructor.  schema The  cognitive complexity on halo in performance  - the  other  mental  was  to  representation explore  the  of the  effect  of  ratings, using correlational techniques  to assess halo (Pulakos, Schmitt, & Ostroff, 1986).  THE  D. THE  N A T U R E OF  THE  S T U D Y / 10  CONTEXT  The  primary  goal of this investigation was  to describe cognitive phenomena  underlying human judgment in performance appraisal. The imposes its own describing and  constraints on understanding  design  how  performance judgment, this study that the nature, amount, and  The human  use  goal  of  certain variables affect cognitive processes  in  was  and  goal of a study often  procedure. To  achieve  conducted as a laboratory experiment, so  presentation of information could be controlled.  of a laboratory procedure, however, relies on  cognitive  processing  varies  little  when  further assumption is that people have no functioning in a simulated  the  task. The  task  the assumption that  demands  are  similar.  A  special reason to distort their mental  fundamental premise of this study  related studies in the literature) is that how  a person processes  (and  of  information in a  contrived setting can provide important heuristic clues as to the mental strategies underlying  performance judgment. These  clues  then may  indicate directions for  research within the ecological reality of the phenomena of interest.  Performance judgment takes place in many settings. A this study  was  levels  the  of  appraisal of teaching. Evaluation of teaching takes educational  system.  The  present  appraisal of teaching in higher education. A performance been  done  reasoned  appraisal which on  that  suitable setting for  student a  deal  evaluation  similar  interpretation of the results.  sample  with of  study  was  anchored  majority of the previous  purpose  and  judgment  at all in  task  Accordingly, would  the  studies of  cognitive complexity  university teaching.  and  place  have  it  was  facilitate  the  T H E N A T U R E O F T H E S T U D Y / 11 Furthermore,  students  in  higher  education  are  increasingly  required  to  provide evaluations on their instructors. Interest in student evaluation of teaching at  colleges and universities is growing (Dunkin  At  the  & Barnes,  university level, evaluation of teaching is required  feedback  to  controversy  the  instructor  surrounds  the  and  for  reliability  tenure/promotion and  validity  of  1986; Marsh, for  the  purposes  decisions, student  1984).  but  of  much  evaluations  of  teaching (Centra, 1979; Cohen, 1981).  E. SIGNIFICANCE  OF T H E STUDY  The benefits of the present investigation are mainly at the theoretical level. Studying how task processes  in  features  judgment  performance judgments  and  of  individual  teaching,  characteristics  increases  our  influence the cognitive understanding  are formed. If we know how judgments  of  how  are produced, we  may succeed in reducing the fallibility in human judgment in general, and in the evaluation of teaching in particular.  Although know  much  all evaluative judgment about  the  effect  of  may  purpose  serve on  specific purposes,  how  performance  appraisal  addition, the  purpose  on performance  judgment  from  a  not  judgments  mentally processed. The results here may clarify our understanding of  we do  of the  are effect  cognitive perspective.  opportunity to infer cognitive processing of performance  In  information  experimentally, may provide clues to the conditions leading to bias and inaccuracy in  performance  judgment.  A n accumulation  of such  knowledge  may  provide  a  knowledge base from which may evolve a formal model of the rater as theoretical basis for measurement  procedures (Feldman, 1981; Krantz et al., 1971).  T H E N A T U R E O F T H E S T U D Y / 12 Several strategies  studies  are  on  human  affected  by  judgment  cue  processes  dimensionality.  In  have most  found of  that  these  judgment  studies,  dimensions of the stimuli are arrayed from most to least salient, and the of cue saliency on judgment in  this  study  reflected  traits  the effect  is observed (Wallsten, 1980). A s cue dimensionality and  role behaviors, the  findings may  suggest  the  importance of conceptualizing cue dimensionality in terms of semantic features of information.  And judgment  finally, is  the  neglect of developmental constructs  unfortunate  (Pitz  &  Sachs,  1984).  on research  Investigating  how  in human cognitive  complexity, a developmental construct, affects the use of cue integration strategies may further  add to our knowledge of processes  in judgment. The problems people  have in using multiple strategies for integrating cues into a judgment,  may well  be a result of their developmental maturity. Developmental constructs may add a new dimension to theorizing on human judgment.  II. REVIEW OF T H E L I T E R A T U R E  This  chapter  provides  a  review  literature  related  to  appraisal.  This  accomplished  of  influences  cognitive on  is  cognitive  on judgment  perspectives  raters'  cognition  discussion  of  processes  the  pertinent  in  human  in five parts.  processes.  The  concerning  presented  in  evaluation  of  part  judgment  and empirical  and  performance  The first part (A) is  second  on performance judgment. is  theoretical  part  (B)  is  A n analysis  three  teaching.  (C).  The  The  a  a  discussion  discussion  of research  fourth  chapter  part  of  bearing  (D) is  a  with  a  concludes  summary in part five (E).  A. INFLUENCES ON J U D G M E N T  Researchers Generally,  the  have  studied  literature  PROCESSES  many  indicates  factors that  which may  judgment  affect  and  human judgment.  decision  processes  affected by (1) the cognitive limitations of the mind; (2) mental models, or  cognitive  structures;  (3)  features  of the task  environment.  are  schemata  These influences  the formation of judgment are discussed in the following three  on  sub-sections.  1. Cognitive Limitations  The  limitations  of  our  mental  apparatus  affect  amounts of information available to our sensory a  number  available limited  of  limitations.  (Kahneman, (Miller,  1956),  Attention  1973). and  The  the  is  storage  processing  13  a  we  modalities.  scarce capacity of  how  the  information  with  large  The human mind has  resource of  cope  is  and  never  working mainly  totally  memory in  a  is  serial  REVIEW OF T H E LITERATURE / manner  (Newell  immediate (Miller,  Simon,  1972).  Although the  control of a person, it allows  1956)  manner.  &  or five  Serial  manipulation of only  "chunks" (Simon, 1974)  processing  produces  Shiffrin,  in  working  memory  1971)  because  the  is  recency  and  under  about seven  the  "bits"  primacy  bias  in  extracting  may be based. Furthermore, unless  rehearsed,  processing  is  of unrelated information in a serial  meaning from information on which judgments information  working memory  14  it  duration  is  is  rapidly  brief,  lost  usually  (Atkinson less  &  than  ten  seconds (Murdock, 1961).  Limitations on attention anticipatory see  (Neisser  & Becklon,  tends to determine  have  a  strong  expectations, (Webster, aspects  what  tendency  than  1964).  to For  consonant  and memory storage make  seek  memory, mean  attentional  sequential  that the  result,  personal  view  resource,  processing,  that  researchers  brief  that  a person  been  is  generally  consistent  their  expectations  have more  to  people  with  their  cognition given  expects  found that  disconfirm  social  selective and  found weight  that than  1980).  limited  storage  execution  mind usually can process  what  It has  may  of  are  nonconsonant information (Nisbett & Ross,  Limited  a  information  information  instance,  with  As  a person does see.  to  seek  1975).  perception  times,  capacity and  of  the  selective  working perception  only a small amount of information at  one time. These limitations require an individual to develop and utilize simplifying cognitive  processing operations  judgment task.  in order to deal with large amounts  of data in a  REVIEW OF T H E LITERATURE  / 15  2. Mental Models  Cognitive processing of large amounts of data is facilitated by categorization of  information (Bruner, Goodnow, & Austin,  objects in  and events has shown that information is commonly stored and processed  relation to  mental  models,  Smith  & Medin,  1981).  It  judge  an  event,  or  object,  stimulus set Mervis,  1975)  is  therefore  procedure  or  schemata  natural by  the  of one's schema  (Mervis & Rosch,  and  economical  degree  or schema  knowledge  1975),  which  (Rumelhart, 1977,  are  structures,  scripts  (Abelson,  abstract  knowledge  psychologists 1976),  have  prototypes  one  frames  to an earlier and more general term, schema  (Bartlett,  1977,  cognitive  &  may  observed  1980;  Rosch & 1986)  in our mind.  terms  (Cantor  1981;  the  structures used  nuclear scenes (Tomkins, 1979), and reference  Rumelhart,  to  that  or mental models (Holland, Holyoak, Nisbett, & Thagard,  models  describe  (Minsky,  prototypes,  is representative  Mental To  1956). Research on categorization of  like  frames  Mischel,  1977),  (Leyton, 1986), in addition 1932; Piaget,  1936/1970;  1980). Taylor and Crocker (1981) defined a schema as a  structure  that  consists  in  part  the  representation  of  some  defined stimulus domain. The schema contains general knowledge about that  domain, including the  attributes,  as  well  as  specification  specific  of the relationships  examples  or instances  among  of the  its  stimulus  domain, (p. 91)  In assumed A  a judgment to be  number  of  task,  mental  utilized via the such  heuristics  popularized by Kahneman  models,  effortless, commonly  and Tversky  schema  or  knowledge  structures  are  readily available simplifying heuristics. used  (1972,  in judgmental  tasks  have  been  1973). These include judgment by  REVIEW OF T H E LITERATURE representativeness, adjusting.  judgment  Kahneman  economical  and  systematic  errors  speculated  the  and  availability,  Tversky  useful, in  by  suggest  that  they  lead  nonetheless, human  mediating  judgment.  role  of  the  an  act  and  between a sample an  actor,  or  judgment although in  Cooper  (1981)  involves  anchoring  and  these  heuristics  are  circumstances  and  Feldman  heuristic  in  generally,  between  an  (1981)  degree of  and a  outcome  to  performance  an assessment of the  and a population, an instance  more  16  by  certain  representativeness  judgment. Judging by representativeness correspondence  and  /  and  category, a  mental  model of some sort (Kahneman & Tversky, 1984).  Mental judgment.  models  Schema  facilitate give  structure  and^ retrieval,  fill  planning  and  anticipating  Crocker,  1981).  prototype  matching  to  inferences  faulty  schema.  in  and  Nago,  &  these  when  &  the  Sprafka,  as  experience, and  future  provide  (Bruner,  functions  of  (1981)  but  1978),  the  for  basis  1971;  poor  (Kuhn,  evidence  policy 1970)  processing  an explanatory  of  among  decisions as  concept,  1981;  schematic  is  in  encoding solving,  Taylor  processing could also  incongruent faulty  bias  problem  Hastie,  schema,  prejudice  introduce  information  configuration  cite  also  determine  but schematic  stimulus  1980),  belief in discredited theory  schema is widely used  processing,  judgment,  Crocker  Stasser,  to  data,  the  facilitates  and  (Elstein,. Shulman, Spitzer,  missing  Given  Taylor  cognitive  with  medical  & or lead  one's  diagnosis  jury  members  (Davis,  (Janis  & Mann,  1977),  induced by  schemata.  Fiedler (1982) points  Although out  that  attempts to proivide verification by measuring schema are rare.  Similarly, simplifying heuristics cause systematic human  judgment  and  decision  making  (Kahneman,  errors and inconsistencies Slovic,  &  Tversky,  in  1982).  REVIEW  O F T H E L I T E R A T U R E / 17  Judgment of prototypicality is made through the use the  representativeness  representativeness against  of  which the there  is  insensitive  to  prior  illusion  of  a  evidence  validity  stimulus  configuration is that  information  of chance,  (Tversky  particular  stimulus  However,  misconception  heuristic  the  &  is  Kahneman,  judged  by  of  it  is  the  (Kahneman & Tversky,  in  regression  nature),  A n error  judgment may therefore result from a person becoming insensitive data  due  to  his/her  reliance  on  simplifying  heuristics,  a  of  the  schema  makes  phenomena,  1973).  a  & Crocker,  heuristic  causal  1972,  1974);  invoking  compared (Taylor  representativeness  (unless  misconception  of judgmental heuristics like  1981). people  leads and or  to  to  an  bias  in  to variations in  prototype  matching  or  schematic processing.  3. Judgment T a s k E n v i r o n m e n t  Investigators strategy research  use has  gambles,  have  inter-item  number  contextual  factors  that  affect  in a judgmental task other than performance judgment. Most of the focused  business,  cue-response  identified  on and  stimulus  features.  medical  judgment  compatibility,  consistency,  1971). The results  and  have  set  size,  primacy  Earlier  investigated  extremity  and  research  recency  generally demonstrated  of  mostly  cue  to  inter-relationships,  information,  effects  relating  redundancy,  (Slovic  & Lichtenstein,  that presentation  format of cues  relates to variations in judgment strategies.  More recent research has examined other features influence  judgment  information  or  processes.  information  Researchers load  affects  have  of information which may  found  information  that search  the and  amount  of  integration  REVIEW strategies  in  judgment  tasks.  For  complexity of the judgment task number  of cue  dimensions.  as  total  decreased  the  (1982)  The results  amount  found  increased demands  Payne  by varying the  of  between cognitive load and the Fischhoff  example,  OF T H E LITERATURE / (1976)  manipulated  number of alternatives  showed  that  information  the  search  increased,  in  judgment  indicating  on memory, and Shaklee  strategies  and Mims  as  the  and the  for information  complexity of information. Similarly,  changes  18  a  trade-off  Shaklee and  information  (1982) found a  load  tendency  to use simpler but less accurate strategies as memory demands increased.  The given  manner in which  to  subjects  Kahneman  to  as  negatively  instructions  the  task  manipulated  framing on subjects' that  perform  (1981)  information  a judgment  and  inferences  to  the  task  is  affect  judgment  information  positively  presented  is  framed,  one  as  and  of  the  the  well.  presentation.  instructions  Tversky  They  found  in probabilistic judgment.  subjects  and  a  and  presented  strong  effect  of  Crocker (1981) suggested  important  factors  influencing  people's judgment of covariation in data.  In saliency and  the  of cues  integration  proceeds made  from  as  Although been  majority of past research on information processing in judgment, the has  strategies. the  most  additional cue  little  been  to  have  For example, salient  has  are  been  define  a  the  dimension  dimensions  saliency effort  found to  an  and  profound impact  1980).  However,  in  a  of  the  stimulus,  considered  (Tversky  important  explanatory  manipulate  study  utilization  anchoring and adjustment  it  &  (Nisbett  Consequently, there is no theory addressing how cue saliency (Wallsten,  on cue  by  Wallsten  and  and  adjustment  Kahneman, variable, &  heuristic  1974).  there  Ross,  is  has  1980).  may be determined Barton  (1982),  cue  REVIEW OF T H E LITERATURE / 19 saliency was manipulated by varying perceptual characteristics and the probability of  occurrence. They found that despite the  were  also  manipulations generally  responsive intended  had the  to to  perceptual affect  predicted effect,  prominence of probabilities, subjects  features  dimensional whereby  of  the  salience  stimuli.  and  Perceptual  processing  subjects traded off  order  perceptual and  probabilistic features.  In  the past,  the effect of cues or stimulus dimensionality in research on  judgment processes seems to have been mainly researched in terms of perceptual characteristics assumed  to  Lichtenstein, as  well.  extent,  and induce  probability of differential  occurrence  salience  cues,  (Wallsten  the  factors  & Barton,  which were  1980;  Slovic &  1971). However, stimulus information may depend on other factors  In perception research, one time,  of  frequency,  and  intensity  view  is  that information is  (Kubovy,  1981).  in meaning,  Another view  is  that  information is in structure (Garner, 1974); the potential origins of structure are experience,  constraints,  statistics,  analysis,  and  geometry  (Cutting,  1987).  Perspectives in social perception and judgment suggest that information is in the concreteness  of  the  stimuli  (Nisbett,  Borgida,  Crandall,  & Reed,  1976)  and  personal implicit theories (Nisbett & Ross, 1980).  B. COGNITIVE VIEWS O F P E R F O R M A N C E  The  current emphasis  JUDGMENT  in performance appraisal research is  on the  entire  performance judgment process of which a rater's cognition is an important aspect. Attention is  drawn to  rating response,  a rater's  with a view  cognitive  functioning in the  formation of the  to examining the potential causes of biases and  REVIEW errors  in  performance  judgment  have  cognitive  psychology.  viewed  in  been  terms  prototype  judgment.  Several  derived from theories Generally,  of  person  matching.  it  and  These  is  theoretical  that  perception,  perspectives  models  and research  suggested  social  OF T H E LITERATURE /  are  of  performance  in cognitive  performance  cognitive  discussed  and  social  appraisal  categorization,  in  the  20  following  be and four  sub-sections.  1. Cognitive Distortion in Performance Judgment  One  of the  was  outlined  by  step  process.  The  step was and  process  oriented  Borman  (1978).  He  first  evaluating  step  the  the third step was  rating on a had  earliest  was  behaviors mentally  performance  discussion  to  viewed  to  dimension.  performance  observing  work-related  in terms  of the  that  in  appraisal behaviors.  effectiveness  Although the processing  third  arriving  at  to  evaluation  as  a  The  they  three second  represent,  arrive at a  single  step in Borman's model  of performance information, he did  processes in terms of a rater's cognitive suggesting  performance  weighting the evaluations  direct implications for cognitive  not explicate  approaches  a  functioning. He limited his  single  rating,  raters  somehow  combine the information from multiple dimensions.  Cooper cognitive  (1981)  distortion  elaborated may  be  Borman's introduced  (1978) in  model,  and  performance  speculated  judgment.  how  Cognitive  distortion refers to the phenomenon of observations being distorted in such a way that  raters  both  due  to  failure  addition  the of  lose to  information  and  add information. The loss  retrieve may  observed  result  from  information a  rater's  of  information  stored implicit  in  may  memory,  theories  of  occur  whereas illusory  REVIEW OF T H E LITERATURE / covariation of the of  judgment  cognitive  rating dimensions.  (Tversky  distortions.  lacking,  many  of  &  the  He suggested that the reliance on heuristics  Kahneman,  However,  he  21  1974)  appear  to  stated  that  because  could  only  be  processes  be  a  factor  systematic  hypothesized.  causing  research Cooper  was  (1981)  interpreted the halo error as cognitive distortion, drawing upon implicit personality theory literature (Schneider, Chapman,  1969),  Addressing  halo,  until  is  there  1973), research on covariation judgment  (Chapman &  and theorizing on people's construction of reality Cooper  (1981)  felt,  "Prospects  a better appreciation of the  cognitive distortions" (p.  for  eliminating  (Kelly, 1955).  it  remain  slim  halo-reduction barriers represented  by  235).  2. Performance Appraisal as Person Perception  and  Upon  reviewing  Farr  (1980)  a  large  body  conceptualized  a  performance  appraisal  process.  examined  a specific  phenomenon  implicit  as  personality  of  theory,  literature  on  coherent  representation  They  proposed  performance  that  rating, of  performance  the  Landy entire  rating  of person perception, from the perspective  and Wherry's psychometric  theory  be of  of rating (Wherry  & Bartlett, 1982).  The of  major components  appraisal  cognitive  and  strategies  the  of Landy  rating  process.  and Farr's The  of the. rater and the  (1980) model  rating  process  administrative  is  are  the  context  comprised  of  the  appraisal system  of  the  organization. In the main, the authors suggest that cognitive characteristics of the rater,  the  being  rated  purpose have  of  rating,  significant  and  organizational  bearing  on  variables  performance  such  judgment.  as  the  position  In  their  model,  R E V I E W O F T H E L I T E R A T U R E / 22 special  emphasis  information. about  the  The way  is  placed  authors in  on  raters'  expressed  which  the  potential  concern,  raters  record information if we hope to increase  Landy  and Farr  (1980)  cognitive  noted  processing  "We  observe,  encode,  that  cognitive  They proposed that  the  "treats"  mentally  information,  rater  could be  a  or  potential  learn store,  characteristics  systematically.  a  performance much  source  of  integrates variance  more  retrieve,  the validity of ratings" (p.  not been investigated way  must  of  of  and  100).  raters  have  cognitive  complexity  several  dimensions  in rating judgments.  and of They  also stressed the central importance of the purpose of performance judgment.  3. Performance A p p r a i s a l as Prototype M a t c h i n g  An Feldman is  alternative  based  on  theories  (Rosch,  and  performance  on  1978), person perception  was  proposed  by  and Feldman (1983). Their model  cognitive  categorization  (Cantor & Mischel,  and  1977,  prototype  1979), and  and controlled processing of information (Schneider &  to  Ilgen  of matching  and  Feldman  an employee  with  (1983), the  consciously  monitored.  prototype,  matching process  When  an  assignment  to  employee the  performance  attributes  categories in a rater's mind. The assignment  stored  appraisal  1977).  According outcome  of  literature  the theory of automatic  Shiffrin,  model  (1981), and later elaborated by Ilgen  matching on  process  judgment  of prototypes  is  an  representing  to a category is either automatic or  demonstrates  category  is  via  behavior automatic  similar  to  the  processing;  the  is accomplished automatically with little mental effort.  When an  REVIEW employee's is  O F T H E L I T E R A T U R E / 23  behavior is unlike the prototype in some respect,  involved  because  special  cognitive  effort  controlled processing  and  attention  are  required in  the  the  recall of information about  prototype matching process.  Hence, employee result  Feldman et  as  traits  recall being  are  of  p.  the 140).  on  categorization  a  direct  in  rater's  the  halo,  based  on  performance  greater  a  distinct  category.  with  In  category.  will  be  under-evaluation  the  and  behaviors  trait Thus,  halo  are  rating, "the  the more  effect" (Feldman,  over-evaluation,  Feldman to  human  cognition,  appraisal  may  Feldman  be  affected  (1981) by  suggested  selective  that  attention,  and memory distortions via the use of simplifying heuristics of & Kahneman,  cognitive  (Kelly,  evidence  laboratory  covary  because  which arise from the nature of the categorization processes  research  (Tversky  complexity  and  into  encoded. Haloed ratings  "neither overt prejudice nor motivational biases are necessary  rater expectancies, judgment  information,  an  130).  Based  that  that  classified  person  Discussing  produce such results, (p.  when  together  stimulus  (1981) contended,  itself  prototypical  similar  recalled  prototypical 1981,  suggested  may be influenced by how the information was  from  treated  al.  1955) was  1974).  structure may  also  available,  investigations  and  Additionally, personal  influence Feldman  of  ratee  related psychometric and field research.  the  Feldman et  construct  system  categorization  (1981)  concluded  categorization  al.  suggested  or  cognitive  of ratees. by  processes,  As no  recommending together  with  REVIEW  4. Performance A p p r a i s a l  Another cognitive by  view  Perception  of performance  appraisal process  has been  DeNisi, Cafferty, and Meglino (1984). Their model is based  and  SrulPs (1981) model of category  performance These  appraisal  operations  memory et  as Social  O F T H E L I T E R A T U R E / 24  al.  and  the  include  the  consider,  as  accessibility.  product  of  information  integration  a  "performance  al's  of  cognitive  acquisition,  of information to appraisal  is  mainly on Wyer  DeNisi et  set  social  model portrays  organization,  exercise  in  operations.  retrieval  form a judgmental  an  presented  from  rating. DeNisi  social  perception  and  cognition embedded in an organizational context requiring both formal and implicit judgment" (p.  362).  DeNisi et al. (1984) claimed that the purpose for appraisal and the role of the  rater  model. frame  as  an  active  The purpose of  reference  interpretation.  They  good  guides  worker,  information  seeker  are  two  for appraisal may predispose or  a  schema  suggested what  that  which  schema,  information  is  distinct  features  of  their  the rater to select an internal  guides  the  information  mentally  sought  and  stored  how  search  and  prototype  that  of  a  information  is  interpreted. Acquisition of job-relevant information is considered the primary input, which  may  pressures  also be  affected  by  factors  other  than  one's schema,  such  as  time  operating on the rater and the nature of the rating instrument.  DeNisi information evaluation accessibility  et  al. (1984) emphasized  search and is  and  different considered  the  integration rating  need  strategies,  instruments.  critical.  for an examination  A  given  The  rater's  role  cognitive  different of  a  of a  rater's  purposes  raters's  complexity  for  memory and  field  R E V I E W O F T H E L I T E R A T U R E / 25 dependence-independence influences  essence, the cognitive views of performance judgment discussed  largely  and  social  but  Ilgen  their  and  does  reach  of  forming  that  and  is,  1977)  common on  Feldman,  1983;  literature  these  1981). stimulus  weighing  is  are  acquisition A  the  Acquisition of features  and  presumably deliberate,  similar views,  more  tend  explicit  undoubtedly  necessary,  "operates"  on  information gathered,  as  may  happens  information,  and involves  be  in  that  raters'  in the  cognition  Landy  & Farr,  models  (Copper,  1981;  of  is  controlled  performance  DeNisi,  1980). In these views,  of performance judgment  how rating judgments  process  and raters' cognitive  et it is  al.  1984;  actually (Schneider 1972).  Ilgen  suggested that  are formed. Although how raters process  to  highly  appraisal is  complexity  the  perceptual  and deeper processing of information (Craik & Lockhart,  thread  in  emphasize  information  alone,  integrating  is  perception  to  person  weighs and integrates  (Anderson, by  (1984) views  above are  person  reflect  These  process.  with  models  al.  research.  judgment  dealing  theoretical  and DeNisi et  empirical  the  distorted  emphasis  purpose  result,  mentally  the judgment  A  a  and  Although information  Comparatively,  Shiffrin,  theories  needed  complete  a judgment  illusions.  As  acquisition.  not  automatic  on  related  Feldman (1983),  statement  information,  &  on  cognition.  information it  style are also considered important  in processing information related to performance.  In based  dimension of cognitive  bear  the & the  influences  information is  of central importance, these models have not addressed the importance of cues on which judgments that  cue  may be based. However, researchers  dimensionality  is  an  important  judgment strategies (cf. Part A.3 in this  variable review).  in other areas  affecting  have  information  shown  use  and  R E V I E W O F T H E L I T E R A T U R E / 26  C. R E S E A R C H ON RATERS' COGNITION  Theorizing on cognitive processing of performance information has indicated a number of important areas The  importance  repeatedly  of  stressed.  for investigation  appraisal purpose  and  in the raters'  The empirical literature  in  performance appraisal process. cognitive  these  complexity  areas  is  has  reviewed  in  been this  section.  1. Cognitive Effect of Appraisal Purpose  A  number  of  researchers  have  suggested  judgment in the performance appraisal process al.,  1984;  Landy  by  Landy  & Farr,  and Farr variable.  administrative  decisions  of  research  These were  studies  studies.  Ilgen  appraisal  & Feldman, 1983;  purpose reported  significantly However,  performance evaluation may have 1984;  that  importance  is  1978;  for evaluation that  & Farr,  for  DeNisi et  than  postulated  operates as  ratings ratings that  a  required  for  done  for  the  purpose  for  the  effects beyond rater motivation  Landy  purpose  1977). Earlier studies reviewed  less lenient it  of  (DeCotiis & Petit,  Zedeck & Kafry,  (1980) indicated  motivational  purpose  1980;  the  (DeNisi, et al.,  1980). The purpose for which an  is conducted may cue raters to search for and utilize  certain types of  information, and thus serve a cognitive function.  One of the first studies revealing a cognitive rating judgments examined  the  information  was  effect  conducted by Zedeck and Cascio (1982).  of appraisal purpose  utilization  effect of appraisal purpose on  policy.  They  These  researchers  on rater accuracy, discriminability, and  used  policy  capturing  methodology  and  R E V I E W O F T H E L I T E R A T U R E / 27 operationalized  information  utilization  weights) of the various dimensions 33  supermarket  decisions  about  checkers. merit pay  retention of employees. ANOVA  for  testing  in  Ratings  on  increases,  a  hypothesis  was  weighted  "skill in human relations" most  and  significant.  and retention The  need  on  purpose  terms  appraisal purpose  skill."  seven  the  of  bagging  the  weighting  of information presented  function  development  of  point  for  (regression  in ratee  scale  vignettes of  were  required  development/training,  for  and for  Each rater's standard deviation of ratings was used in an the  found  terms  rater  The  purposes  researchers  discriminability training.  group  Only  the  evaluating  for  heavily,  relied  equally  interpreted  between  this  of organizational and consumer perspectives  and the on  ratees  effect  as  a  of appraisal  merit  pay  raises  groups evaluating for  "organizational ability and  information because  utilization policy  of the  in  appropriateness  of the dimensions weighted for different appraisal purposes.  Although Zedeck and Cascio (1982) presented some evidence effect  of  appraisal  purpose  information utilization - they the  use  on  rating  judgments,  did not examine  of information integration  strategies.  the  their  study  on the is  appraisal purpose, a criticism that  DiNisi,  Blencoe,  &  Cafferty,  1985).  has An  limited  to  effect of appraisal purpose on  Furthermore, they  did not  with a theoretical framework for interpreting information utilization as of  cognitive  also been alternative  proceed  a function  echoed by others (Williams, explanation  for  information  utilization in their study may be offered in terms of the processing of personality information  and job behaviors for different  is based on schematic  processing,  purposes.  This  a theoretical perspective  alternative  explanation  from which predictions  could be made about the usage of information across different jobs, when ratings are done for different purposes.  R E V I E W O F T H E L I T E R A T U R E / 28 A  weak effect of appraisal purpose on psychometric qualities of ratings  reported rated  by  Mclntyre,  videotaped  effectiveness  lectures  were  hiring decisions.  Smith,  purpose  Hassett  acted  by  (1984).  drama  A  sample  students.  of  undergraduates  Ratings  on  teaching  required for three purposes: research, course improvement, and  Subjects were instructed about the purpose of their rating before  presenting the videotapes. the  and  was  of  rating  A post-experimental was  appropriately  questionnaire was perceived  by  the  used to check if different  Because  of unequal cell sizes and heterogeneity  of variances  for every  variable  investigated,  their  of the  at a conservatively  the  researchers  adjusted  presented  (reduced degree  results  of freedom)  groups. dependent  main effects  alpha level  of .10,  and  concluded a weak effect of appraisal purpose on leniency and accuracy of ratings.  Mclntyre et al. (1984) used the final ratings (the product) as the variable  to  cognition.  draw This  information  inferences  about  prevented  processing  the  in raters'  the  effect  researchers cognition,  of  appraisal  from  which  discussing  may  have  different purposes of appraisal. Although 15 of their subjects the  analysis,  the  authors  did not  report if  the  excluded  whether  been  on  raters'  aspect  of  influenced  were excluded were  by  from  equally  Comparing their own  with those of Zedeck and Cascio (1982), Mclntyre et al. (1984) questioned the  variable was  directly address results,  any  subjects  distributed across the three purpose conditions in the study. results  purpose  dependent  Mclntyre  a  purely cognitive  one,  although  their  study  did not  information utilization and integration. However, in discussing the et  al. argued that  mainly an emotional effect. purpose of research (their  the  perceived purpose  of rating may  have  One of the groups in their study was rating for the study),  function of performance evaluation.  but rating for research  is  hardly ever  a real  R E V I E W O F T H E L I T E R A T U R E / 29 A  study that has more directly examined the" effect of appraisal purpose on  raters'  cognitive  Cafferty  processes  (1985) who examined  utilization.  In the  on  covariation  in  for  principles  the  three  different  had  a  Yet,  types  the  different required  on  researchers  information".  In  eight  information  request  on  arcsine transformation)  a  that  raters  experiment,  resulting showed and  were  they  a function  of  purposes  followed  by  consensus,  in information requested, to  serve a cognitive  perhaps and  not  the  use  of  Blencoe,  that  Kelley's  and  different  consensus to  raters'  consensus, on  judgments  information. covariation search  Subjects  and  for were  covariation  preferences  did not show a significant  (after  main effect of  information was preferred the  pattern  the researchers concluded that appraisal purpose  appears  function of that  and consistency  subjects  appraisal purpose  appraisal purpose.  MANOVA  (1971)  whether  "sensitive  appraisal purpose. For all three purposes, distinctiveness most,  in  investigated  consistency, A  light  investigated  ratings  micro computer.  across  in  distinctiveness,  distinctiveness,  presented  They  differently,  consistency,  second  DeNisi,  consistency, distinctiveness, and  ratees,  theory.  information  suggested  their  Williams,  presented  hypothetical  types of information as to  they  Analysis on mean  effect  by  how purpose influenced information acquisition and  attribution  of  purposes.  limited  conducted  first experiment,  consensus information  used  was  affecting  the  type  information, a  Cascio's (1982) finding discussed  information. Discussing  of information searched  conclusion  which  contradicts  for, but Zedeck  above.  Williams et al. (1985) did not study the interaction between rating purposes and  the type of information presented  flaw in their study was  that  information  in  search,  but  they  to the subjects.  investigated  cognition,  Moreover, a possible logical  information utilization followed by  information  search  most  likely  precedes  REVIEW  OF T H E LITERATURE /  30  information utilization. Therefore, their conclusion that raters may search for but not  use  from  different  the  illogical  investigated. preferences be the great  types  information  order  in  in the  same.  second experiment.  leap  effects  preferences  with  contribution of the  for  conclusion Preference  their  first  ratings,  three  different  search on  the  experiment,  leaves  and  a gap  did  of information may not  that not  types  of  information  is  a  is perhaps  they  step  correlate  rating judgments.  forward  in  a  did not seek information  in our understanding of the  types of information to  were  of information  and addressing information use fact  suffers  utilization  analysis  comparison to other studies on the topic, their theoretical  different  purposes,  and  and use  on their part. Moreover, the  in the  rating  information  that  Analyzing preferences  inferential  in  which  They seem to make  interaction  in  of  relative  Nevertheless,  stance in presenting  studying  the  effects  of  purpose on raters' cognition.  The and and  effect  of appraisal purpose  Armstrong  (1984).  decisions.  delivered  by  rating,  evaluation  "well-defined"  affect  required  rate  were each  indicate  affect  observing  for  research  in the  of observing  lecturer's  each  the  lecturer. of  frequency  in observing  teacher  behavior  and  the  and  four videotaped  lectures  purpose  performance with  The  of  accuracy  the  the  on  of  their  standard  which  twelve  investigators  performance  However, they found that appraisal purpose did influence accuracy  making  frequency  accuracy  behavior  for  informed of the  the  teacher  by Murphy, Balzer, Kellam,  subjects evaluated  observed  did not  accuracy  (b)  in  investigated  Subjects (a)  and  were  were  student  students.  forms,  behaviors  the  five  required to  appraisal purpose it  Ratings  Forty  graduate  and were  teacher  did  accuracy  evaluating teaching performance was  personnel  that  on  found  ratings;  nor  critical behaviors.  relationship between  in evaluating  teaching  REVIEW performance. purpose  OF T H E LITERATURE /  This latter finding led the experimenters  affects  the  way  raters  process  to speculate  information, without  31  that appraisal  necessarily  affecting  the general level of ratings.  In  the  features find  of the  a  clear  purpose al.  four  studies  study  and  reviewed  procedures  substantial  of research" (their  1984;  Murphy,  et  al.,  above,  are  pattern  noteworthy.  effect  studies)  the  of as  The two  appraisal one  1984). Because  of  purpose  of the  results  studies that had  certain did not  "rating  rating purposes  rating for the  and  for  the  (Mclntyre, et  purpose  of research  is  hardly ever a true function of performance appraisal, it may not have invoked a particular  schema  implicit theories of  appraisal  raters  to  reasonably also  to  provide a basis  for judgment.  Prototypes  and  may exist in terms of performance effectiveness but not in terms  for  engage strong  required  or prototype  research in  functions.  prototype  cognitive  appraisals  sampling of a sufficiently  matching,  effect  on  Therefore,  only  large  of  and  appraisal  two  or  it  would for  the  four  ratees,  a reasonable  been  difficult  researchers  purpose.  number of occasions.  about the limitations of not aggregating  have  to  find  Moreover, these which  did  for  not  a  studies allow  a  Epstein (1980) has  warned  number of occasions  in the  study of human behavior.  With  the  exception  of one  experiment  by  Williams et  al.  (1985),  and  the  study by Zedeck and Cascio (1982), the rest of the studies have mainly analyzed the on how  final  ratings  (the  product) and drawn inferences  about the  effect of purpose  raters' cognition. The effect of purpose on the processes in cognition, that information is mentally  attention.  Only  one  of  the  weighted, studies  utilized, and integrated, (Williams,  et  al.,  has  1985)  received has  is  little  presented  R E V I E W O F T H E L I T E R A T U R E / 32 information using a theoretical rationale, although performance evaluation may not be a case of attribution of causality. And finally, none of the studies provided a wide  enough  Researchers  scale  for  an  unrestricted  of information integration  points for functional measurement  expression  suggest the  of  use  subjective  judgments.  of a scale with about  20  of subjective judgment (Anderson, 1982).  2. Effect of Cue Dimensionality  In the  past  saliency  research  on human judgment  of  has  cues,  strategies (Nisbett  been  & Ross,  1980;  found  processes,  to  Wallsten,  have  an  in performance judgment  Feldman,  1983; Landy & Farr,  Researchers effect  of  cue  rating  judgments  Balzer,  1986;  investigated (1985),  the  addressing  Mclntyre, use  provided that  1981;  DeNisi,  appear  to  on  information  use  and  Kehoe,  1984;  consistency  cue  the  integration  importance of  in theoretical perspectives  cognition  &  on  mainly  et  al.,  1984;  also  overlooked  on  Ilgen  &  1980). •  1984;  Zedeck  Murphy, &  of information of different  aspects of cue dimensions.  effect  rater  dimensionality (Cardy  (Cooper,  dimensionality,  1980). In contrast,  cue dimensionality appears to have been oversighted cognition  cue  have  integration et  Cascio,  al,  1984;  1982).  types is  that  in  One  the  processing Murphy study  & that  of Williams et  al.  and consensus information can be perceived  as  However, their discussion is limited to suggesting  that  raters were sensitive to covariation information.  In of  performance judgment,  performance  information  trait and behavior (Wexley  &  are  Klimoski,  the 1984).  two  main  Besides,  dimensions trait  and  REVIEW behavior  are  naturally occurring dimensions  (Cantor & Mischel,  1979;  Nisbett & Ross,  is conceived of as an exercise Feldman,  1983;  reviewed  here  judgments.  Landy sought  information in  person  perception  1980). Although performance judgment  in person perception (DeNisi, et al., 1984; Ilgen &  & Farr, the  of  O F T H E L I T E R A T U R E / 33  1980),  effect  Furthermore, none  of  of  none  trait  the  of  and  studies  the  studies  behavior sought  appraisal purpose and cue dimensionality on the  on  rater  information  the  subjective  cognition on  interactive  rating  effects  of  importance and actual  use of information in the formation of rating judgments.  3. Influence of Cognitive Complexity  Considerable which  influences  states that the  construct,  make  persons  people's  has  dealt  perceptions  with and  it has  generally  dimensions  when  only are  very  gross  believed  been  they  given  postulated  perceive  discriminations  to  employ  cognitive  evaluations  although various writers have  employ few to  literature  as  of  Vannoy  events.  somewhat that  different  some persons  and evaluate among  many  complexity  a  variable (1965)  meanings are  prone  to to  stimuli, or are inclined  dimensions  dimensions,  for  and/or  meaning; to  other  mak<;  fine  discriminations among the dimensions they utilize.  Cognitive complexity is a construct that emerged from Kelly's (1955) theory of  personal  constructs,  person-objects (Vannoy,  1965,  differentiated cognitively  in one p.  system  and is  commonly defined  as  a  special environment in a complex 385).  A  cognitively  of dimensions  complex  for processing  simple person (Bieri, Atkins,  Briar,  "disposition  view  or differentiated  person the  to  has  a  manner"  relatively  behavior of others  Leaman,  Miller,  the  & Tripodi,  more  than a 1966).  REVIEW O F T H E LITERATURE / Bieri  et  al state  dimensions  that  cognitive  complexity  attributed to stumili (i.e.  is  the  ability to  differentiation)  and the  34  discriminate between  ability to discriminate  within each dimension (i.e. articulation).  The  construct  of  cognitive  variable  in  studies  moderater performance the  (Kennedy,  performance  cognitive Farr,  1980)  organize  and  et  cognitive  integrate  their  have  al.,  by  effect of cognitive  behavior  1984;  Ilgen  &  examined  (Mitchell,  emphasized  may  describe  thoughts,  and  reflect  as  1970),  a  team  1976). Theorists of  the  role  Feldman,  complexity  complexity  been  making (Menasco,  also  dimensions they use to describe what they  The  previously  leadership  process  (DeNisi,  Raters'  of  has  1971), and decision  appraisal  complexity  complexity  the  a  1983;  way  the  of  rater's  Landy  in which  relative  &  they  number  of  perceive.  on performance appraisal was  Schneier (1977). Schneier defined cognitive  complexity as  first found  "the degree to which  a person possesses the ability to perceive behavior in a multidimensional manner" (p. grid.  541), In  lenient, than  and his  measured  study,  and used did  the  suggested  that  complexity increase  a wider  cognitively to  variable  cognitively  the  and the  complex  range  cognitive  the  raters  Role  raters.  there  demands  is  Construct Repertory (REP)  demonstrated  on behaviorally  simple  degree  using  On  less halo,  anchored rating scales  these  findings,  compatibility between  of the  were less  appraisal process,  Schneier  a  rater's there  (BARS) (1977) cognitive  will be an  in the psychometric quality of the resultant ratings.  Following cognitive  the  Schneier's  complexity  of  (1977) findings, raters  may  a number of reviewers  relate  to  effective  suggested  performance  that  appraisal  REVIEW OF T H E LITERATURE / (DeCotiis  & Petit,  1980) . In the Feldman pointed  1978;  cognitive  (1981), out  process  Landy  the  and  relationship  of  As  models  a  a  Cardy,  1981) ,  surprisingly, Schneier's  but  these  Carlyle,  DeNisi et cognitive  several  complexity 1982;  al.  & Zedeck,  &  have  Saal, not  have in  also  making  empirically tested  rating  1981;  been  (Part B),  (1984)  have  performance  Lahey  Kafry,  complexity  researchers  to  findings  Jacobs,  earlier in this review  and  rater's  result,  (Bernardin,  1979;  discussed  (1981),  of  cognitive &  & Borman,  Farr  importance  performance judgments. the  Dunnette  35  effectiveness  Sauser  confirmed  & in  Pond, any  of  investigations.  Lahey  and  Saal  using  three  scales.  Cognitive complexity  grid,  factor  different  (1981)  analysis  investigated  cognitive  of  the  complexity  measures  of undergraduate  the  R E P grid  cognitive  compatibility  and  students was  data,  and  ratings were obtained on a seven point B A R S ,  a  four  hypothesis  different  rating  assessed with a R E P  sorting  task.  Performance  three point mixed standard rating  scales, seven point graphic rating scales, and simple "alternate" three point rating scales.  No differences  function  of  in leniency,  cognitive  complexity,  halo, or range or from  the  restriction emerged  interaction  of  either  cognitive  as  a  complexity  with scale format.  As  Lahey  and  Saal  (1981)  used  multimethod  assessments  of  cognitive  complexity and rating properties, they considered their study a comprehensive of  the  study  cognitive may  have  their analyses, seven  point  compatibility reduced  they  scales  the  hypothesis. chances  of  Nevertheless, obtaining  used transformed ratings were  transformed  to  the  some  procedures  expected  results.  in  test their  In  all  as data points. The ratings on the  three  points  in  order  to  equate  the  R E V I E W O F T H E L I T E R A T U R E / 36 metric  for  the  leniency effect, scales  to  repeated  measures  ANOVA.  For  example,  in  the  analysis  of  a composite rating was obtained first by transforming seven point  three  transformations  and  then  would  have  by  calculating  reduced the  the  mean  across  variability in the  diminished the effect of cognitive complexity. Furthermore,  dimensions.  data, they  The  which perhaps  assessed halo by  calculating the standard deviation of ratings across the rating dimensions for each rater-ratee combination, which is  an inadequate  measure  of halo according to  a  recent Monte Carlo study by Pulakos, Schmitt, and Ostroff (1986).  Bernardin,  Cardy,  and  Carlyle  (1982)  re-examined  the  role  of  cognitive  complexity as a predictor of appraisal effectiveness in a series of experiments. In their  first  experiment,  instructors  on  performance three grid,  two  28  undergraduates  separate  dimensions.  scales.  The  same  point graphic rating scale. and halo was  scale  scale  to  format.  procedure  examine There  and the  dimensions  rating errors as was  results  were  as  in the  of  BARS  their  represented was  psychology  consisting  of  five  on ae second  measured by  a REP  In  officers from  the  first  two  effect  well as their own, Bernardin  of  second  third  et  complexity  complexity. experiment,  "The  and  analysis  in which  the  accuracy of ratings given to  experiment,  a pool of 65  experiments.  transformed to a three  function of cognitive  examined on the  on 11 dimensions using two rating scales. as  were  similar in the  instructors.  patrol  a  significant  no  two  same  a  The data on B A R S were  complexity was  evaluated two  was  Cognitive complexity  effect of cognitive hypothetical  scale  three  indexed by the standard deviation of each rater's ratings on  five dimensions for each ratee. point  One  rated  (selection  31  police  sergeants  criteria is not stated)  The analysis and the results were  Based on the results  of previous  al. concluded, "the plethora of null  the  studies findings  REVIEW OF T H E LITERATURE certainly casts doubt on at least the  generalizability of the cognitive  / 37  compatibility  theory, if not also its validity."  The from  null findings in the  some  of  transformation  the  procedures  of ratings  of  the  they  cognitive  the  a  their  three  different  officers  (random  on  performance  judgment  One of the  weaknesses was  al.  scale  The  might  have  were being compared,  selection  complexity  Bieri  point  that the raters' evaluations  suggests  with  experiments.  sergeant appraised only two  above  measure  possibly  cannot  be  assumed  criteria is not stated).  weaknesses.  R E P grid  et  to  in  discussion  methodological of  point  al. (1982) resulted  data, and thereby, perhaps diminished the effect of  means  evaluated  because the selection  The  ten  adopted  complexity. Furthermore, the fact that each  65 patrol officers  although  they  from a  reduced the variability in the cognitive  study by Bernardin et  because  the  same  (1966)  in  all  of  that  the  elements and a  study  research, these elements (e.g.  of  previous  studies constructs  research suffered  father)  the  and constructs  of  a  number  of  lack of face  grid  In  (e.g.  influence  above  the  clinical judgment.  the  from  reviewed on  on  used as  validity  the  REP  introduced  performance shy-outgoing)  by  judgment may lack  face validity. Another common limitation in the studies considered above is in the assessment of halo. The standard deviation as an index of halo is now known to be problematic (Pulakos, et probably  resulted  cognitive  complexity.  might  have  in  masked  a  al.,  loss  1986). Additionally, the transformation of ratings  of  Because interaction  variability,  transformation effects  as  and  thus,  diminished  of  data  reduces  (Winer,  1971).  well  concern with psychometric properties of ratings, has  the  effect  of  nonadditivity,  it  And finally,  perhaps prevented the  the  study  REVIEW of  the  O F T H E L I T E R A T U R E / 38  information integration strategies which complex  and simple raters use in  performance judgment.  However, some support relating to the come  from  other  "conceptual study  level",  areas. a  concept  supervisor-supervisee  supervisors identified and  Researchers  as  similar  in to  education, the  interactions.  conceptually  cognitive compatibility hypothesis  notion  for of  example,  cognitive  Thies-Sprinthall  differentiated  have  has used  complexity,  (1980)  reported  were more flexible,  to that  responsive,  recognized a wider range of teaching behaviors than the supervisors identified  at a lower level of conceptual development. "abstract"  supervisors,  development,  engaged  supervisees,  asked  that  is,  Likewise, Grimmett (1984) found that  supervisors  at  a  higher  level  of  conceptual  in conjoint appraisal of teaching-learning episodes  more  open  ended  questions,  and  elicited  with  the  ideas  from  the  in  study  on  supervisees more than the "concrete" supervisors.  D.  EVALUATION  Thorndike evaluation  of  O F  TEACHING  (1920)  initially  teaching.  Even  characterized today,  the  appraisal  halo of  error  teaching  a  performance  (like  appraisal of performance in other occupational settings) is jeopardized by problems in  rating.  The  performance of  great  at elementary  concern  of  reliability  and secondary  and  validity  of  teaching  to be a matter  1981;  Hawley,  Medley,  Holley,  evaluation  of  teaching  at  endorsed in recent years  (Centra,  1979),  this level as well (Cohen, 1980,  ratings  levels continues  &  student  of  school  (Evertson  Although widely  question  1981; Marsh,  colleges  and  1982;  universities  has  1982). been  similar problems are present  1984;  Marsh & Overall,  1980).  at  R E V I E W O F T H E L I T E R A T U R E / 39 Current  theorizing  and  research  on  teacher  evaluation  dominated by a concern with instrument development 1985),  and  evaluation  with  the  (McGreal,  problems 1983;  of  determining  Millman,  1981;  appears  still to  be  (Peterson, Micceri, & Smith, criteria  Stiggins  and  &  techniques  Bridgeford,  for  1985).  Relatively little attention is given to the rater's ability to draw inferences  and to  the cognitive processes that may underlie performance judgments. Not surprisingly, it has been psychology  pointed out that the to  research  on  widely realized (Shavelson,  Investigation the  potential contribution of theories  teaching  and  teacher  education  from  has  not  cognitive yet  been  1985).  of bias in student ratings of instructors has  focused mainly on  "Dr. Fox effect" or "educational seduction". Researchers of the Dr. Fox effect  suggested content  that  expressive  taught  in  or  enthusiastic  arousing  reactions  behavior  toward  was  as  instructors,  important and  as  the  questioned  the  validity of student ratings (Naftulin, Ware, & Donnelly, 1973).  One  of  evaluation effect  the  Dr.  Fox  studies  (Meier & Feldhusen,  on  any  of  rating  expressiveness and lecture  content.  rating factors  "an  expressive  but  lecturer  evaluation of him" (p. observed Leventhal,  not  in almost  on  can 343).  all of the  and Perry,  the  1979). The stated  the  five  investigated  measures;  student generate  Similar effect Dr. Fox  (1982), the  did  halo  the  purpose  purpose  The authors  effect  which  with  effect on all concluded  influences  In  a  meta-analysis  by  that  others'  of expressiveness or enthusiasm  studies.  of  had no  interact  had a significant  achievement. a  of  purpose of evaluation  nor  Expressiveness  effect  was  Abrami,  proportion of rating variance accounted for by  expressiveness had a weighted mean of .293  across twelve  studies.  REVIEW The as  O F T H E L I T E R A T U R E / 40  early Dr. Fox studies have been criticized for lack of external as well  internal  validity  (Frey,  1978).  In  their  meta-analysis,  Abrami  et  al.  (1982)  found that although there were methodological flaws in several of the studies and disagreement  on  the  issue of validity of student  had  a large impact on ratings, and lecture  on  student  achievement.  Even  if  ratings, expressiveness typically  content  the  Dr.  typically had a large impact Fox  studies  methodologically sound, these studies address bias as affective not  consider  the  validity  of  student  ratings  from  a  are  considered  phenomena, and do cognitive  information  processing perspective.  The  cognitive  processing of information in evaluation of teaching may vary  depending  on  normative  conception  ideology the  one's  conception of  and requires  exemplar"  (p.  of  teacher  teaching. effectiveness  a judgment  28).  It  traits  and  methods  can be  assumed  because  important  desirable  &  resourcefulness,  McKeachie,  Shulman from  between  Shulman's prototypes,  1975;  personal  characteristics  evaluation of instructors researchers warmth,  derived  from  exemplars,  to  (1986),  one's  the  "the  theory  conception  point of  or and  view  that  or mental models of  and good teaching. The good teacher schema may be in terms of  behaviors  are  is  of correspondence  those who judge teaching likely have a good teacher  According  of  have  good  traits  and  teaching  1983),  of  (Medley,  found instructor enthusiasm,  and leadership are important traits  Marsh,  use  and  planning,  effective 1979).  sociability,  (Cohen, 1981;  presentation  1975,  Marsh,  1983,  1984).  Kulik  clarity, grading,  communication, and research activity are important behaviors (Cohen, 1981; Leonard, & Beatty,  In  Frey,  R E V I E W O F T H E L I T E R A T U R E / 41  E. SUMMARY This  review started  with  a discussion on how the  the mind and certain features of the task that  schemata  and simplifying heuristics. However, schematic processing may  bias  inconsistencies  operate  as  cope  theorized,  with  the  affect human judgment. The  indicates  and  to  cognitive limitations of  limitations of attention  in our judgment.  and  although  Schemata  schema  is  and  are  widely  memory,  assumed used  concept, little effort has been devoted to its mesurement.  is presented,  also  affect  be  determined  by  perceptual  as  how we form judgments.  characteristics,  use  introduce  an  and  explantory  Certain features of the  found that cue saliency has a strong effect on judgment may  we  to exist  task such as the amount of information, cue saliency, and the manner information  literature  in which  Researchers  strategies.  structure  have  Cue saliency  and  content  of  information, and personal implicit theories.  The cognitive perspectives These  views  process  indicate  information.  a  on performance  number  Two  of  of the  variables variables  influences on how raters form rating judgments  judgment that  were  may  reviewd as  influence  suggested  as  how  having  well. raters  important  are cognitive complexity and  the  purpose for appraisal. However, research findings concerning the effect of purpose on raters' cognition are limited and equivocal. Likewise, the findings on the  effect  of cognitive complexity on the halo effect, are contradictory.  Studies examined, clarify  but  how  which no the  sought clear  the  results  formation  of  effect emerged. rating  of  purpose The  on  findings  judgments  are  raters' are  cognition  mixed, and  affected  by  were fail  to  purpose.  REVIEW OF T H E LITERATURE /  42  Researchers have mainly analyzed the ratings and not the processes that lead to a rating response, that investigated  which include information utilization and integration. One study information utilization lacked a theoretical basis for interpreting  the  results;  another  study  that  addressed  information utilization did not  examine  the  interaction between purpose and information utilization. Moreover, the  purpose  conditions used in some of the studies were unrealistic, and the number of ratees evaluated by the subjects quite small for the study of raters' judgment strategies.  Studies judgment,  that  were  pessimistic,  investigated  also  the  examined.  particularly  in  effects  The  regard  of cognitive  conclusions to  the  psychometric qualities of ratings, but the of  assesing  halo  by  inapropriate  studies have  methodological  flaws,  properties  ratings.  studies  of  The  most  effect  of  pessimistic  means. they  of  complexity  focused  reviewed  here  these  cognitive  conclusions  Futhermore,  have  of  on performance  not  are  complexity  on  may be a  only  exclusively  did not  studies  some on  consider  of  result these  psychometric the  effect  of  cognitive complexity on the use of varied information combination strategies.  Researchers dimensionality strategies.  has  The  what determines of  cues  on  of a  human strong  literature  effect  reviewed  cue saliency.  raters'  judgment  cognition.  on  processes information  indicates  an  is  the  addressed in the prevalent cognitive perspectives  Consequently, there  found  utilization  absence  The studies reviewed Nor  have  here  importance  of  a  that  and clear  integration theory  did not seek the of  cue  cue  on  effect  dimensionality  on performance appraisal.  are a number of matters  that need to be resolved. We  need to study the effect of appraisal purpose on cognitive  processes that lead to  a  rating  judgment.  Specifically,  whether  REVIEW  O F T H E L I T E R A T U R E / 43  purpose  affects  utilization,  and integration  needs to be  examined,  on  cognition  be  Further,  raters'  purpose  and  cue  could  clarified.  dimensionality  conjointly  so we  influence  information  that  the  need  effect  valuation, of purpose  to  determine  whether  judgment  processes,  so  that  what affects cue saliency in performance judgment could be identified. Research is also  needed  information  to  study  integration  variables  that  cognitive  complexity  correlational judgment,  affect  means.  procedures  the  influence  strategies,  so  of  cognitive  that  performance judgment. and  halo  needs  And finally, to  measure  given  to  we  learn  Additionally, be  the  schema  complexity  explored importance  at  the  more the  the  about  use  of  individual  relationship between  with of  on  halo  assessed  schemata  individual  level  in  by  human  need  to  be  explored as well.  The chapter.  hypotheses  and methodology  of  the  study  are  presented  in  the  next  III. HYPOTHESES, QUESTIONS, AND METHOD This  chapter  outlines  methodology.  In the  methodology.  The  second  first  the  hypotheses,  part (A) is  hypotheses  part (B). The design  and  the  exploratory rationale  exploratory  questions,  for the  questions  and  research  hypotheses  and the  are  presented  and data collection procedures are  in  the  described in the  third part (C).  A. RATIONALE FOR HYPOTHESES AND METHOD It  has  been  theorized  that  appraisal  purpose  cognition (DeNisi, et al., 1984; Landy & Farr, the jargon of cognitive  psychology, the  facilitates  discuss by for  this  an  of  on  the  perceived  purpose  generally  the  (Bruner,  1957)  The categories. template three  input of  good  (Collins  appraisal worker  1974). Many  theorists  spreading activation. Activation of a concept  &  Loftus,  may  schema,  1975;  activate and  Ortony,  prototypes,  thereby,  1978). mental  provide  accessible Thus,  the  models,  or  cognitive readiness  for the interpretation and utilization of performance information.  different conceptions Hastie's  of the term schema could be classified into broad  (1981) classifications  schemata.  types:  priming  of a category or schemata, and  priming, makes that concept and related knowledge in memory more processing  raters'  1982). In  purpose for appraisal becomes a  processing of the input (Loftus & Loftus,  phenomena in terms  effect  1980; Zedeck & Cascio,  stimulus. A priming stimulus activates knowledge hence,  has  person  Taylor  and  schema,  include central tendency,  Crocker event  (1981)  schema,  44  suggest and  role  that  procedural, and  schemata  schema.  are  of  Relevant  to  HYPOTHESES, performance judgment seem to be the schema  include  behavior  in  prototypes  1981). Person schema &  Mischel,  (Wexley  like  1977). Appraisal content 1984).  traits and effective methods cues  in  of people  professor  the  or  teaching,  in terms  fireman  of people  also comprises  Good  A N D M E T H O D / 45  of role and person schemata.  models  consist of prototypes  & Klimoski,  behavior  notions  or mental  particular occupations  QUESTIONS,  Role  of the  roles or  (Taylor &  Crocker,  in terms  of traits (Cantor  trait and behavior information  particularly, depends  upon  desirable  or role behaviors (Medley, 1979). Therefore, trait and  evaluation  of  college  instructors  may  interact  in  the  formation of rating judgments for different purposes.  The  formation  information  through  conceptualization Anderson  have  based  of  rating  on  from  one's  is  judgment  processes  Valuation  Valuation  influence  a the  information  weighting.  schema  is  (1981).  pertinent  of  the  mental  valuation,  and  information  involves stimuli  in  subjective,  mental  of  involves  the  or  process  working  and  models  integration  therefore,  schemata.  influence  subjective  importance  integration stimuli  on  refers  valuation  and utilization  to  according to  the  thought some  information into a response,  of  processes  rule  to  of  theory  and  directly  In  through  Thus,  direct  in  to  the  judgment,  the  information,  which  may  a  information.  response.  affect  the  The concept  differentially  The  - may  rules  of  weighted  of combining  when modeled mathematically, reflect the information  performance  observation  their  susceptible  integration strategies; the weightings of cues in the rules reflect information.  determining  schemata  the  by and  person  combine  This  proposed  and  that  of  salient  performance  performance  produce  integration.  extracting  memory,  of a good worker - a combination of role an  of  transformaion  of  a  judgment, person  in  a  rater  action,  may retrieve  the utilization of obtain  information  information  from  HYPOTHESES, memory, or use  observations  subjectively weighted  However, importance  recorded by others,  but the  is  some  information in  controversy  as  to  they  know,  information obtained is  their judgment,  whether  people  can  reflect  their  subjective  suggested that people  tell more  which  weighting policy. Nisbett and Wilson (1977) have than  A N D M E T H O D / 46  and transformed into a rating response.  there  of  QUESTIONS,  which casts doubt on people's  may  report  ability to report, retrospectively,  the importance of information in their own judgment. Further,  in social judgment  research, Brehmer (1976) found that although individuals generally weighted fairly accurately, they failed to apply this knowledge have  been  Hunter,  reported by  as  well  (Slovic  consistently.  & Lichtenstein,  cues  Similar findings 1971;  Schmitt &  1977). On the other hand, Ericson and Simon (1978), and Surber (1985)  have provided evidence out  others  the  whether  raters  to the contrary. Therefore, it would be interesting to find  in performance judgment  show  consistency  in expressing  the  subjective importance of cues and the actual utilization of similar cues.  Furthermore, cue integration strategies to developmental construct  that  seems  to  be  refers  to  a  differentiated relatively than  a  affect  person's  person's  use  cognitive  less  of  complexity.  complex  the  1965).  system  A  of  person  cognitively complex person may employ than a cognitively simple person.  varied  to view  (Vannoy,  differentiated  relatively  the  disposition  manner  more  rarely been  aspects of an individual (Pitz & Sachs,  might a  have  studied  1984). A developmental  information  integration  Theoretically, cognitive task  environment  cognitively  dimensions (Bieri,  et  in relation  in  complex  for al.,  a  complexity complex  person  processing 1966).  strategies  has  or a  information, Therefore,  a  more strategies of integrating information  HYPOTHESES, Researchers  have  integration models basic  (Anderson,  categories,  between  compensatory hand, &  refer  Marcus,  strategy,  different  1981,  compensatory  Compensatory models trade-offs  studied  dimensions  models  is  not  kinds  Einhorn,  and  entail the  QUESTIONS, A N D M E T H O D / information  1970).  These  noncompensatory  linear additive of  of  information.  interactive.  models  models  such  Einhorn,  as  1970,  The  conjunctive  or  Hogarth,  1980).  elimination-by-aspects  of  two  1980).  information in  models use In  on  the  other  of cues (Billings  a  noncompensatory  (Tversky,  amount of information utilized per alternative is variable because one dimension may not be compensated  into  (Hogarth,  integration  Noncompensatory  1971;  fall  and  and averaging strategies involving  to integration strategies that involve interactive 1983;  utilization  47  1972),  the  a low score on  by a high score on another, causing the  elimination of certain information and alternatives.  The  strategies  of • information  utilization  and  integration  assessed by regression models or policy capturing analysis 1982;  Cadwell  Kleinmuntz, 1982). they  1979;  Jenkins, Norman,  However, models represent  protocols the  &  same  1960).  1970,  Einhorn  not et  al.  models.  They  underlying processes  but at  different  the  judgment tasks  be  traditionally  Borko & Cadwell,  Einhorn,  Slovic & Lichtenstein, 1971;  regression  that linear regression Moreover,  1986;  Einhorn,  of judgment may  (Hoffman,  and linear  1986;  (e.g.  are  Kleinmuntz,  Zedeck & Cascio,  isomorphic with the  (1979)  compared  process-tracing  of  generality,  strongly  of  the  linear  indicates  that  judgment is captured (Goldberg, 1968).  regression  models  in  a  wide  capture  and argued  models do capture the interactive and contingent  success  processes  concluded that both models levels  &  processes. variety  of  some fundamental charactersitic of human  HYPOTHESES,  QUESTIONS, A N D M E T H O D /  48  B. H Y P O T H E S E S A N D E X P L O R A T O R Y QUESTIONS  In  light  chapters, purpose  of  this and  the  discussion  study  attempted  (b)  information,  and (c) the  hypotheses questions  to  cue dimensionality  information,  information  above  the  on  consistency  stated  are presented  the  discussion  hypotheses  subjective  between  in  concerning  importance  the (a)  previous the  and  strategies  in  sub-sections  in the fifth  in  of  utilization  of  subjective importance and  performance  one  to  four  judgment.  below,  and  utilization of  The the  use  l^A:  Appraisal  exploratory  sub-section.  purpose  will  affect  subjective  importance  ratings of trait and behavior information in performance judgment.  Hypothesis 1.B:  Cue  dimensionality  will  affect  subjective  importance  ratings of trait and behavior information in performance judgment.  Hypothesis  l.C:  conjointly affect  Appraisal the subjective  purpose  and  cue  dimensionality  will  importance ratings of trait and behavior  information in performance judgment.  of  research  1. Importance of Information  Hypothesis  two  effects  effect of purpose and cognitive complexity on the  integration are  test  and  H Y P O T H E S E S , Q U E S T I O N S , A N D M E T H O D / 49 2. Utilization  of Information  Hypothesis 2~A: Appraisal purpose  will  affect utilization of trait and  behavior information in performance judgment.  Hypothesis 2.B: Cue dimensionality will affect utilization of trait and behavior information in performance judgment.  Hypothesis 2.C: conjointly  affect  Appraisal  purpose  the utilization  and  of trait  cue  dimensionality  and behavior  will  information in  performance judgment.  3. Information Importance and Utilization Hypothesis 3:  Cue utilization  in  Consistency performance  judgment  will  be  consistent with subjective importance of cue dimensions.  4. Information Integration Hypothesis 4: In comparison judgment  to formative judgment,  in summative  raters will combine cue dimensions using a noncompensatory  strategy i n addition to a compensatory  strategy.  Hypothesis 5: In comparison to cognitively simple raters, cognitively complex raters will  combine cue dimensions using a noncompensatory  strategy i n addition to a compensatory  strategy.  HYPOTHESES,  QUESTIONS,  AND METHOD  / 50  5. Exploratory Questions  The interest.  current study These  also  questions  sought  arise  from  reviewed in chapter two. First, explanatory lacking is  concept in several  (Fiedler, 1982).  inappropriate  et  the  for  two  questions  following  two  points  studies,  al.,  use  attempts  the  literature  as  an important  to quantitatively measure it are  of standard deviations  1986).  of exploratory  in  although schema has been used  Second, the  (Pulakos,  answers  Hence,  the  in assessing halo  questions  of  exploratory  interest were as follows: 1.  Can a person's good instructor schema profile be measured quantitatively?  2.  What is the effect of cognitive complexity on halo, when halo is measured by  C.  correlational techniques?  METHODOLOGY  1. Subjects  Seventy University  students  of British  enrolled  in  the  Faculty  of  Columbia voluntarily served  sample of seventy was  considered sufficiently  Education  as  subjects  programs in this  at  The  study. The  large in order to detect medium to  large effects of the independent variables. A consideration in choosing the sample was  the  familiarity with the judgment task.  Students in the  education programs  at this university are required to evaluate their instructors, and were assumed to comprehend the performance judgment task as intended. They were to  have  sufficient  instructors.  Six  knowledge  subjects  in the  of  teaching  sample  were  to  be  able  to  make  graduate students,  23  also  assumed  judgments were  on  in the  HYPOTHESES,  Q U E S T I O N S , A N D M E T H O D / 51  fourth year, and 42 were in the third year of their teacher There were  11 males and 69  education programs.  females.  2. Instruments  Five  measures  performance  rating  questionnaires cues  or  other  was  The  a  administered and  a  self-report of  other  Task  B  was  questionnaire  measures  single  in the are  described  study:  of the  importance  to  was  measure  a  A contained  study  below  the  of  separate  grid.  of  sentences.  sections,  two  of  the  attached  to  a  university  good  instructor  hypothetical ratee in  One  subjects  subjects'  are reported in the  in  questionnaires,  performance  27  described  two  Repertory  related  vignette  present  this  Construct  measure  information  a  in  Role  Performance rating Task  these instruments these  tasks,  dimensions  instructor. schema.  were  profiles;  the  The reliabilites of  next and  chapter. A l l of included in  the  Appendix.  a. Importance of Information Measure  Dimensionality information because (Wexley traits  &  of  was  reflected  appraisal content  Klimoski,  and behaviors  cues  1984).  The  in  trait  mainly comprises  subjective  and  items  of  trait and role information  importance  related to performance was  behavior  individuals  measured  using a  attached  to  questionnaire  that listed ten items of information. Five of these items concerned traits and five concerned  teaching  alternatively.  behaviors.  The  trait  and  behavior  items  were  listed  HYPOTHESES, The  selection  of  their importance as Frey,  Leonard,  McKeachie, that  most  the  ten  Marsh,  important  resourcefulness,  included  in  the  instrument  was  based  identified in the instructor evaluation literature (Cohen,  & Beatty,  1975;  cues  Q U E S T I O N S , A N D M E T H O D / 52  and  1975;  Hildesbrand, Wilson,  1983,  1984).  These  & Dienst,  researchers  instructor  traits  are  leadership;  most  important  have  enthusiam,  1971;  1981;  Kulik  commonly  sociability,  behaviors  on  &  found  warmth,  are  planning,  presentation clarity, grading, communication, and reserach activity. The Importance of Information Measure is included in Appendix B.  Subjects rated the importance of each item of information on separate point interval scales. least important, The  mean  measures  4  ratings  Three points  = on  important, trait  on the and  scales were anchored as  7  =  and behavior  most items  important type were  used  in  seven  follows:  1  =  of information. the  analyses  as  of subjective importance of trait and behavior dimensions.  b. Rating Judgment Task A  A methodological limitation in a majority of previous studies was a  small  number  of  ratees,  usually  performance judgment Task A consisted dimensions  two of 27  to  eight.  hypothetical  allows  control over  study. Moreover, the use situation details  the ratee  present profiles  study, on four  at three levels. Hypothetical ratee profiles limited the amount of detail  that could be included, but were judged to be suitable that  In  the use of  students  the  amount  of information which was  of hypothetical profiles  normally evaluate  (cf. Cadwell & Jenkins, 1985).  because it is a procedure  on  general  were justified impressions  essential  in  this  because in a real  and not  on  specific  HYPOTHESES,  Part  Keeping  in  A . l ) , it  was  would  not  mind  the  limitations  reasoned  result  in  an  that  in  QUESTIONS,  human  mental  A N D M E T H O D / 53 capacities  including four dimensions  information  overload,  thus,  in the  allowing  (Chapter ratee  2,  profiles  thoughtful  rating  judgments. Additionally, a greater number of dimensions would have resulted in a huge  number  of  possible  profiles,  have  occured at a smaller number of times  could have adversly affected  Each  and  the  cues  in  a  fractional replicate  at each of the  chosen  levels,  profile presented  information on the  The traits were enthusiasm  two  most  salient  traits  and resourcefulness;  clarity and grading-marking. These  traits and behaviors  because  their  power  popularity  and  explanatory  (factor  The average, profiles four  four  information  average, (3 ). 4  other  and  concentration,  dimensions  or  cues,  average)  make  81  below  To expect the measures and  1975; Marsh,  have  motivation.  been  Therefore,  81  unrealistic to  at  maintain  1975;  three  these  Hildesbrand,  levels  combinations  profiles in  of  chosen  1983).  different  participants to judge  would  each  were  been developed and  construct validated through research (Frey, Leonard, & Beatty, 1971; Kulik & McKeachie,  Information  loadings)  dimensions in the existing evaluation instruments which have  Wilson, & Dienst,  and the  and the behaviors were  presentation of  which  the impact of the cues on raters evaluations.  two most salient behaviors from the ten included in the Importance of Measure.  would  of  (above ratee  and to respond to  terms  subjects'  of  their  time,  concentration  for  thoughtful judgments, they were required to rate only 27 of all possible profiles.  The fractional  27  ratee  replicate  profiles  presented  procedure  (Winer,  to  the 1971).  subjects  were  Choosing  a  obtained by using a one-third  fractional  HYPOTHESES, replicate  ensured  average,  or below  each  that  average  performance  levels.  each  dimension the  dimension  same  was  was  Q U E S T I O N S , A N D M E T H O D / 54 expressed  as  number of times.  expressed  nine  Hence, each dimension or cue had the  either  In the  times  at  above  average,  27 ratee  each  of  profiles,  the  three  same chance of being used in the  formation of the rating judgments.  The  27  replicates  were  obtained  from  the  fractional  factorial  tables  developed by Conner and Zelen (1959). These tables have been prepared so that: (a)  no  main  effects  are  aliased  with  other  main  effects,  or  aliased  factor interactions; (b) as few  two factor interactions as possible  other  (c)  two  factor  interactions;  with higher order interactions 2).  In  the  one-third  two  factor  interactions  are termed measurable  replication  chosen  for  the  with  two  are aliased with  which are  only  aliased  (Connor & Zelen, 1959,  present  study,  six  p.  first-order  interactions (AB, A C , A D , B C , B D , CD) were measurable.  The  arrangement  of  the  dimensions  within  a  profile  was  on  a  Latin  squares pattern. This was done to ensure an even distribution of any recency or primacy  effects  across  all dimensions.  The set  of 27  ratee  profiles  into booklets following Latin squares rotation as well. This was there  is  some  evidence  that  the  serial  position  of  ratees  was  collated  necessary  because  has  an  performance ratings, although no general pattern has emerged  to date  Farr,  rater  1980).  "observer  The  drift"  as  rotation well,  if  of  ratee  there  profiles  was  any.  balanced  out  Observer  drift  effect  (Landy &  fatigue  refers  on  to  and  shifting  criteria at different points in the rating task, which can cause carry over effects, leniency  or  stringency  in  ratings.  The  factorial  arrangement pattern are included in Appendix C.  combination  and  Latin  squares  HYPOTHESES, A  methodological  limitation  in  scales that do not allow sufficient of  subjective  a  numerical rating judgment  information  integration  (Anderson,  1982).  suitability quality used  judgments.  for  two  the  use  of  response  restrict the  expression  provided for  ratee profile, because researchers use  response  of  a  scale  or judgment  condition) condition).  appraisal purposes,  reading the  with  was  about  studying  20  required in  of feedback  As  set  the  same  anchors  points  terms  and in terms  neutral  purpose  of appraisal, which was  subjects rated each of the profiles  rating judgment presented taken  in one  by marking a point on the on a separate  to  code  each  arrangement of its dimensions The  was  on  of profiles  (very  poor,  of the was  average,  were used to mark three points on the scale.  After  was  the  (formativet  different  studies  an eighteen point scale was  on each  for promotion (summative  outstanding)  was  In this study,  rating  of performance  stimulus,  previous  variability, and thereby,  suggest  The  Q U E S T I O N S , A N D M E T H O D / 55  instructions  and  two  intended  as  session. They indicated their  scale for each  profile. Each  page in order to inhibit comparative profile,  so  that  the  factorial  profiles,  one  for  each  profile  ratings.  Care  combination  and  could be traced for the purposes of data  typical  a priming  analyses.  appraisal purpose,  are  included in Appendix C.  c. Cognitive Complexity Measure  Cognitive Repertory  complexity  (REP) grid  was  measured  introduced  by  using  Kelly  a  (1955)  version and  of  the  revised  Role Construct by  Bieri  et  al.  t The use of the term formative should not imply specific recommendations for improvement (as it may connote in teacher evaluation literature), because for the purposes of this stud} , the ratings required were an expression of judgment only. 7  HYPOTHESES,  Q U E S T I O N S , A N D M E T H O D / 56  (1966). The R E P grid measure has been found to be valid and reliable across a number of samples. For example, test-retest reliabilities of .68,  .86  and .82  have  been reported in other studies (Schneier, 1979).  However, unlike previous studies, the same elements (roles) and constructs as in  Bieri  face  et  al.  validity  guidelines  in  a  provided  male teacher) face  (1966)  validity  were study  by  not  used  on  performance  Easterby-Smith  and constructs (e.g. of  the  measure.  in the  present  study  judgment.  (1980,  1982),  Instead,  more  decisive-indecisive) were  The constructs  important characteristics of a teacher  (cf.  chosen  because  Hawley, 1982;  lack  following  the  relevant  chosen  were  these  (e.g.  to increase  thosee  Medley,  roles  which  1982,  reflect  Millman,  1981). The modification of roles and constructs is not likely to have affected reliability of the instrument because of  the  measure,  reliability  of  a  the  such modification provides an alternate  procedure which has  R E P grid  measure  been  of  previously used  cognitive  complexity  to  the  form  establish  (Schneier,  the  the  1979).  However, the reliability of the R E P measure in the present study is reported in the next chapter.  The and six  the  10 by  10 (roles by constructs) grid listed roles horizontally on the top  constructs  point bipolar  vertically on the  scale.  The subjects  right.  The constructs  decided the  degree  were  calibrated on  a  to which each construct  applied to each person inserted for the roles. In this measure, the use of many degrees of constructs in describing each role person as a  few  degees, indicates complexity (Bieri,  provide a rating in each cell of the  et  grid.  of different judgments made, and because  al.,  opposed to using one or  1966).  Subjects  were  asked to  The grid was  scored for the number  the scoring was  reversed (as is usually  HYPOTHESES, done)  the  smaller  number  of  different  Q U E S T I O N S , A N D M E T H O D / 57  judgments  made  represented  the  most  complexity. The R E P grid used in this study is included in Appendix D.  d. Good Instructor Schema Measure  This measure good an  instructor individual  McCauley,  Stitt,  estimation,  was  was  used for the puposes  schema. level  A  technique  has  been  and Segal adapted  for  of exploring the measurement of a  quantitatively  offered  by  measuring  McCauley  and  stereotypes  Stitt  (1978,)  at and  (1980). This procedure, based on Bayesian probability for  the  measurement  of  a  subject's  of  people,  good  instructor  schema.  In  drawing  generalizations  p(characteristic B/group A) = B) the it  divided by  is  the  measure of B.  of  the  When  Using diagnostic  attribute of the  of  the  quantitative  rule  states:  which  1.0,  the  =  p(B) x  "diagnostic  the  L R is  ratio" (DR), because  occurrence  occurrence  L R , where  of  of A  A  revises  describes  the  nothing  that is, the occurrence of A has no diagnostic value.  McCauley  or less than  diagnostic  stereotype.  by  DR is  a series of studies,  departure  first  degree  the  ratio of greater  Bayes  p(characteristic B) times the p(group A/characteristic  p(group A). In other words, p(B/A)  about the probability of B,  the  classes  likelihood ratio p(A/B)/p(A). L R is called the  probability  The  about  These  measure  other existing group measures.  of  ratio  et  1.0 from  al.  (1978,  indicates 1.0  1980)  concluded that  a stereotype  indicates  the  a  quantitatively.  strength  of  an  authors claim that their Bayesian technique  is  stereotypes  to  A definite  or  schema,  advantage  and  it  is  superior  of this procedure is that the  HYPOTHESES, diagnostic  ratio allows  schema  measurement  QUESTIONS,  AND METHOD /  58  at an individual level in quantitative  terms.  Several  other  studies  have  since  measured  stereotypes  estimating procedure. Rasinski, Crocker, and Hastie normative criterion based on subjects' perceiver's use of subjective  stereotypes present  was  study.  presented  adapted  for  Probability  unrelated  attributes  estimated  four  of  by  measuring  a  university  probabilities.  The  on population predictions.  et  al.  subjects'  were  Copley, and Johnson  stereotypes quantitatively in a study  McCauley  estimates  probability  (1985) constructed a Bayesian  probabilities. McCauley, Durham,  analyzing the impact of personal experience  procedure  a  own stereotypes in a study analyzing social  (1985) used probability estimates to measure  The  via  (1978,  good  obtained  instructor.  following  is  instructor  on  For  an  1980)  eight each  example  for  measuring  schema  related  and  attribute, of  the  in  the two  subjects  probability  statements for the behavior "presents the subject matter with clarity". 1.  p(behavior):- What percentage  of instructors present  the  subject  matter  with  clarity? 2.  p(behavior/group):- What percentage  of g o o d  instructors  present  the  subject  matter with clarity? 3.  p(group/behavior):-  What  percentage  of  all  instructors  who  present  the  subject matter with clarity a r e good instructors? 4.  p(group):- What percentage  of all instructors are good instructors?  Each  of  presented  of on  the a  four separate  parts  page.  the  probability  Subjects  were  estimation asked  to  questionnaire provide  their  was best  HYPOTHESES, estimates,  as  they  had  no  way  of  knowing  probabilities were required to encourage The  second  each  two  attribute,  others.  The  percentage which  diagnostic  figures the  ratios  were  the  exact  A N D M E T H O D / 59  answers.  The  first  subjects to engage in Bayesian  were  reflected  QUESTIONS,  used  saliency  to  of  computed  p(group). The good instructor schema measure  compute an  by  reasoning.  a diagnostic  attribute  in  dividing  two  ratio for  relation  to  the  p (group/behavior)  by  is included in Appendix E .  e. Rating Judgment Task B  Performance rating Task B was cognitive  complexity  methodological measure deviation  halo  in  weakness in the  of halo of  on  (Pulakos,  dimensional  et  presented ratings.  in order to  It  has  majority of previous al.,  ratings  1986). Previous across  explore  been studies  the  suggested is  the  but  Pulakos,  that  a  inappropriate  studies relied on the  ratees,  effect of  et  standard  al.  (1986)  recommended the use of correlation between dimensional ratings.  In this study, halo was way.  Rating Task  B consisted  assessed by correlation techniques  but in a modified  of a description of an instructor in a number of  sentences. Although the rating scale included eight dimensions,  information related  to  other  performance  were  supplied  dependability; grading-marking,  was for and  supplied four  on  four  dimensions:  deliberately  enthusiasm,  dimnensions preparation,  withheld  communication,  only.  In  presentation,  for  the  other  and  scholarship.  words,  data  sociability,  and  four This  dimensions: increased  the  possibility of halo in the ratings. Rating Task B is included in Appendix F .  The  subjects  were  neither  informed  of  the  information  withheld,  nor  HYPOTHESES,  Q U E S T I O N S , A N D M E T H O D / 60  instructed to provide a rating on all dimensions. They rated each dimension on a seven point scale.  From  the  ten decomposed  ratings, two  ratings were  computed  for each subject: the average rating on the dimensions with missing information rating in  "x,"  the  and the  vignette  cognitively  -  average  rating  complex  rating on the  "y."  and simple  The  two  dimensions  ratings  with information included  (x,y)  rater groups, were  for  each  subject  in  the  used to compute a correlation  reflecting the degree of halo in the ratings of complex and simple raters.  3. Experimental Design and Variables  In  testing  the  hypotheses,  These  were  appraisal  variables. cue  dimensionality  simple).  The  utilization  the  present  purpose  study  (formative  had  and  three  independent  summative  conditions),  (trait and behavior), and cognitive complexity (complex and  dependent (regression  variables weights)  were of  subjective  trait  and  importance behavior  ratings,  information,  the and  compensatory and noncompensatory information integration strategies.  was  In  the  exploratory  the  independent  analysis  concerning halo in ratings, cognitive  variable. For schema  measurement,  complexity  the dependent  measures  were diagnostic ratios in the good instructor schema profiles.  Equal conditions complex  numbers by  of  and simple  data  were  random distribution of rater groups were  percent of the subjects,  The  participants  analysis  the  assigned  to  questionnaire  the  booklets.  created by eliminating the  ranked by their cognitive complexity  was  two  performed on  two  levels.  rating The  purpose  cognitively  middle  twenty  scores.  Individual level  analysis  HYPOTHESES, was  performed  between-groups  For  extract  examination  ANOVA  and  procedure, known as study.  This  previous  choice  related  Cascio,  1982;  information  measures  to  represent  dependent  variables  61 for  analysis of interest.  the  factorial  to  QUESTIONS, A N D M E T H O D /  of  information  regression  (Slovic  utilization, &  there  Lichtenstein,  are 1971).  policy capturing or len's modeling, was  was  based  research Zedeck  on  &  utilization  on the  performance  Kafry,  and  wide  Cadwell & Jenkins, 1986; Norman,  is  of  policy  appraisal  1977),  integration  use  and of  in  two The  regression  used in the  present  capturing analysis  (Cadwell, various  interest  paradigms,  1985;  Zedeck  in &  other  areas  where  &  Cadwell,  1982;  (Borko  1986).  In policy capturing analysis, a multiple linear regression model is utilized as a descriptive et  al.,  1979;  regression Y,  =  vector  b ^ ! of  coded 2,  +  the  1,  was  b X 2  rating  respectively. were  Shavelson,  analysis  represented  A,  tool, capturing the  four  and 0  cues  for  or  b X 3  +  3  from or  above  bflXj,. rating  judgment  information average,  on the  precoded  policy 1971;  structural multiple  coefficients  (Einhorn,  average,  The  and below  A, cue  1971;  1977).  follows:  X,  to  dimensions  average  the X„ were  performance,  27 ratee profiles in rating Task  of the  reflected  as  linear  represented  and  performance  indicated how much influence  1970,  Zedeck & Kafry,  Task  dimensions.  values  in a subject's rating, and thus, utilization  A  In this equation, Y ,  A subject's rating responses, for the  regressed  Lichtenstein,  1986).  performed for each subject. The equation was  responses  unstandardized regression had  Webb, & Burstein,  +  2  various aspects of vicarious functioning (Einhorn,  the  dimensions.  each dimension  subject's information  Einhorn,  et  al.,  The  1979;  weighting Slovic  &  HYPOTHESES, In  the present investigation,  hypothesis  testing  (Zedeck  clusters  (raters  ANOVA  procedures.  testing  in  &  hypotheses  policy capturing was  Cascio,  1982).  and  formative  summative The  regression  concerning  Q U E S T I O N S , A N D M E T H O D / 62  weights  information  The  models.  The  regression  integration  amount  model,  of  reflected  noncompensatory strategy by  the  nonlinear  Einhorn,  1970,  strategies  variance the  and  was  component  1971,  of  R s 2  utilization  were  a  of  apriori  were  were  and  data for groups  compared  used  as  integration  or  using  data  for  (Anderson,  1977)  identified  explained  use  policies  conditions)  1977; Zedeck & Cascio, 1982; Zedeck & Kafry,  Information  used to generate  by  by  the  comparing  linear  compensatory  regression  component  strategy.  The  in  the  use  of  a  inferred from the amount of variance accounted for in  regression  Weldon & Gargano,  modeling  (Billings  &  Marcus,  1983;  1985).  4. D a t a C o l l e c t i o n  After subjects  approval from  were  administered 11,  13,  sought the  and  21.  the  university  instruments Every  to  the  person  was  order with one exception.  were  primed  other  half  make  were  committee  for voluntary participation in the  same  to  ethics  subjects  The exception  primed  to  make  as  rating  study.  research, The  individually and also  administered  rating judgments  on  was  the  same  experimenter in groups of  instruments  that one half of the  feedback judgments  (formative for  student  in  subjects  condition);  promotion  the  the  decisions  (summative condition).  After  a  brief  introduction  about  the  general  nature  of  the  project,  each  HYPOTHESES, subject this  received  order:  (1)  a  questionnaire  a  letter  of  booklet.  consent;  This  (2)  a  QUESTIONS, A N D M E T H O D /  63  booklet  in  page  contained  outlining  the  the  following  content  of  the  questionnaire booklet and the purpose of rating; (3) the Importance of Information Measure; (4) instructions B;  (7)  Good Instructor Schema Measure; (5)  including the purpose of rating as  the  presented,  the  REP  grid.  All  subjects  task,  A with  the  priming stimulus; (6) Rating Task  completed  the  instruments,  in about 30 to 40 minutes. As the subjects  the judgment  Rating Task  in  the  order  had some familiarity with  no difficulties  were  encountered  in the  of  and  the  are  administration of  the  instruments.  The chapter.  analyses  the  data  results  presented  in  the  next  IV. ANALYSIS AND RESULTS This chapter presents the results in two parts. The tests of the are presented in the first part  (A). The second part  hypotheses  (B) includes the results of  the exploratory analyses.  Most  of  the  analysis  (Dixon,  1983)  statistics  analyses  performed were  was  done  software.  using  The  assumptions  is  ANOVA,  77  by  expressed  dividing  2  (Nie,  by  "eta  squared".  1983)  underlying  examined throughout. Where  analysis of variance has been employed, the variables  SPSS:X  the  the  classical  SS ff t e  e c  by  SS^tal  BMDP  statistical  approach to  strength of association between Because  in  the  case  tends to be upwardly biased, a conservative estimate the  and  (Hays,  1981;  Keppel,  1973;  of  a  the  nested  was obtained Vaughn  &  Corballis, 1969).  A. TEST OF THE HYPOTHESES The research hypotheses they  are  hypotheses  presented were  The hypotheses  here  as  were presented in the previous chapter (Part 3.B); well  for  directional, they were  ease  of  reference.  cast into null  Although  the  form for statistical  are translated into statistical terms (effects)  research testing.  corresponding to the  statistical models of analysis. The criterion for rejection of the hypotheses  was a  two-tail  of  test  hypotheses  at  the  required  conventional different  alpha  level  approaches,  concerning the variables involved are included.  64  of  .05.  As  supporting  the  evaluation  statistical  the  information  ANALYSIS  A N D R E S U L T S / 65  1. Importance of Information  Hypothesis  l.A  importance  ratings  Hypothesis  l.B  stated  that  Appraisal  Purpose  of  triat  and  behavior  information  stated  that  Cue  Dimensionality  will  will  affect  subjective  in  performance  judgment.  affect  subjective  importance  ratings of trait and behavior information in performance judgment. Hypothesis l . C stated  that  subjective  Appraisal  importance  Purpose ratings  and  of  Cue  trait  Dimensionality  and  behavior  will  conjointly  information  in  affect  performance  judgment.  The  subjective  information was  importance  measured by the  reliabilities  (Cronbach  conditions,  respectively.  summative  appraisal purpose,  point scale. for  alpha)  of  The average  repeated measure  Purpose were  with  as  met  the  trait  0.81  0.73 with  indicated the  in a  formative either  a  dimensions  dependent  and the  and  of  measures  Cue  ' analysis  Dimensionality  as  between-subjects factor.  (Winer,  1971).  The  can be seen in Table  Appraisal  or  a  item on a seven average  importance  variables which were treated  of  variance  (SPSS:X  the  within-subjects  The assumptions  results  are  presented  ANOVAR)  factor  Purpose  was  not  1,  the  in  between-subjects effect,  significant,  and  underlying the Table  relationship between the variables is graphically displayed in Figure  As  of  summative  formative  importance of each  importance for trait items, two  behavior  of Information Measure which had  and  primed  and  as  a  of Cue Dimensionality.  repeated  performed  to  Importance  Subjects,  behavior items formed the  A  given  F(l,68) = 0.238.  that  1,  was  Appraisal analyses and  the  1.  is,  Therefore,  the the  effect null  ANALYSIS A N D RESULTS hypothesis was  l . A was  significant,  between  not rejected. The within-subjects  F(l,68) = 39.692,  Appraisal  Purpose  Consequently,  null  the  strength  of  was  0.118. These results  a  significant  subjective  and  hypotheses association  interaction  importance  p<.05, Cue  l.B  and  between  and  so  effect  ratings  of  of  effect of Cue Dimensionality was  Dimensionality,  Cue Dimensionality  Purpose  trait  the  and  interaction  F(l, 68) = 13.683,  l . C were rejected.  indicate a significant  / 66  As and  effect p<.05.  estimated  by 77 ,  importance  ratings  2  effect of Cue Dimensionality, and and  behavior  Cue  Dimensionality  information  in  on  performance  judgment.  Table 1  The Effect of Purpose and Cue Dimensionality on Importance Ratings  Source  A-Appraisal Purpose S-Within  DF  Mean Sqs.  F Ratio  TJ  1  0.257  0.238  0.002  68  1.081  2  B-Cue Dimensionality  1  13.578  39.692*  0.118  AB  1  4.681  13.683*  0.041  68  0.342  BS-Within  * p<.05.  the  ANALYSIS A N D RESULTS  / 67  Importance rating 6.0 5.9 5.8 5.7 -  I Formative  Summative Appraisal Purpose  Fig.  Further contributed between ratings test) three and  analysis  to  the  Appraisal for each  in Table behavior one  dimensions  trait are  1: Mean Importance Rating of Cues  was  group  difference  Purpose  dimension  2  undertaken  indicate  dimensions dimension lower  for  and are  that  to  on  Cue  identify  Cue  two  Table  2.  and  The The  behaviors  t  the  which  interaction  average  importance  values  (independent  appraisal groups differed significantly  (planning-preparation, (enthusiasm). the  and  Dimensionality  Dimensionality.  reported in the  traits  summative  The  grading-marking, means  group,  on  suggesting  the that  on  communication) three raters  behavior making  ANALYSIS judgments as  did  for promotion did not consider these behavior dimensions the  enthusiasm more  A N D R E S U L T S / 68  raters was  important  evaluating  for  considered - most for  the  feedback.  important by  summative  group.  summative judgment,  but the  difference  Except  activity,  the  for  research  In  mean  the  trait  both groups,  Leadership was  between the ratings  show  groups that  as important  category, it  was  more was the  although  significantly  important for not  significant.  formative  group  considered behavior dimensions more important than trait dimensions, although the difference between the groups was not statistically  significant on all dimensions.  Table 2  M e a n R a t e d Importance of P e r f o r m a n c e Related Information  Formative  Cue  Dimension  Summative  M  SD  M  SD  t  Planning-preparation  6.11  1.2  5.62  0.7  1.79*  Lecture presentation  6.20  1.0  6.09  0.7  0.56  Grading-marking  5.35  1.3  4.71  1.5  1.91*  Communication  6.40  0.9  5.94  1.3  1.76*  Research activity  3.85  1.7  4.14  1.7  -0.70  Enthusiasm  5.20  1.1  5.86  1.3  -2.32*  Sociability  3.89  1.4  4.31  1.5  -1.28  Resourcefulness  4.94  1.2  5.37  1.2  -1.49  Leadership  4.77  1.3  5.03  1.4  -0.77  Warmth  4.17  1.4  4.66  1.3  -1.52  Behavior  Trait  * p<.05. Note: N = 35 in all groups. Scale was  7 points.  ANALYSIS AND RESULTS /  69  2. Utilization of Information  Hypothesis  2.A stated that  and  behavior  Cue  Dimensionality  performance  information  in  will  judgment.  Appraisal  performance  affect  Purpose judgment.  utilization  Hypothesis  l.C  will affect utilization of trait  of  trait  stated  Hypothesis and  that  2.B  behavior  Appraisal  stated  that  information  Purpose  and  in Cue  Dimensionality will conjointly affect utilization of trait and behavior information in performance judgment.  The the  relative  ratee  profiles  procedure for  described  was in  of  trait  measured chapter  3  and  by  behavior  dimensions  regression  (Part  C.3).  modeling  A  of  or  regression  information  policy  model  in  capturing  was  computed  each person to obtain the subject's information utilization policy. The four cue  dimensions 27  utilization  in the  profiles  alpha)  for  were  in  profiles  rating  Task  summative  uncorrelated,  were regressed  and  the  A.  The  reliabilities  formative  regression  on the  raters,  weights  vector were  of ratings  0.74  respectively. were  treated  and  except  linear  for  subject  variance explained  regression 3  in  models  the  by the  estimated  summative  were  group.  main effects in formative  the  (Cronbach  the  coded  vectors  as  an  index  of  significant  Statistics  0.84  to  As  relative utilization of the different types of information (Pedhazur,  The  given  on  the  1982).  for  every  subject  the  proportion  and summative judgment  of are  presented  in Table 3. As shown in Table 3, it appears that slightly more rating  judgment  variance  formative  group appear to be more linearly consistent in their rating judgments,  R =.81, 2  may  be  explained  in formative  than the raters in the summative  judgment.  group, i ? = . 7 6 . 2  Also,  raters  in  the  ANALYSIS A N D RESULTS  / 70  Table 3  M e a n , M e d i a n , a n d Range of V a r i a n c e E x p l a i n e d b y R e g r e s s i o n Models  Statistic  R  Formative  2  R  2  Summative  Mean  .81  .76  Median  .84  .77  Highest  .94  .92  Lowest  .60  .62*  * Would be .21 if one outlier (Subject 3) is included.  The  average  behavior weights  items was  hypotheses for  regression weight  for  each  significant)  subject  for trait items  (subject  formed the  two  3  was  behavior  and  trait  items  were  treated  dimensions. A repeated measures A N O V A  average  included because  dependent  2.A, 2.B, and 2.C (Zedeck & Kafry,  and the  weight for one  variables in testing  of  the  the  null  1977). The two average  weights  as  of  repeated  measures  was performed with Cue Dimensionality  as the within-subject factor and Appraisal Purpose as the between-subjects The  cue  factor.  assumptions underlying the analysis were examined, and as the heterogeinity  assumption was  met,  Borko & Cadwell,  the  1982)  summation of policy captruring The results  data was  are summarized in Table  possible  (cf.  4 and illustrated  in Figure 2.  The  between-subjects  statistically rejected.  significant,  The  effect,  that is, the effect of Appraisal Purpose was not  F(l,68) = 1.802.  withhin-subjects  effect  Therefore, null of  Cue  hypothesis  Dimensionality  2.A  was  was  not  significant,  ANALYSIS AND RESULTS F(l,68) = 55.481, Purpose  and  hypotheses association  p<.05, Cue  2.B  and  and  so  was  Dimensionality, 2.C  were  the  interaction  F(l, 68) = 23.25,  rejected.  As  between  p<.05.  estimated  by  the  /  Appraisal  Consequently, TJ ,  the  2  71  null  strength  between Cue Dimensionality and utilization of information was  of  0.296.  Table 4  The Effect of Purpose and Cue Dimensionality on Information Utilization  Source  DF  Mean Sqs.  1  0.323  68  0.179  B-Cue Dimensionality  1  17.151  55.481*  0.296  AB  1  7.187  23.250*  0.124  A-Appraisal Purpose S-Within  BS-Within  *  68  F Ratio  1.802  rj  2  0.006  0.309  p<.05.  These significant utilization  results interactive  indicate effect  a of  of trait and behavior  seen in Figure  strong  influence  Appraisal  of  Purpose  Cue and  Dimensionality, Cue  Dimensionality  information in performance judgment.  3 that unlike formative  evaluation,  and  It  a on  can be  trait and behavior information  contributed almost equally in the formation of summative  judgments.  A N A L Y S I S A N D R E S U L T S / 72 Regression weight  I Formative  Summative Appraisal Purpose  Fig.  In  further  determined  for  2: Mean Weight of Cues Utilized  analysis, both  the  summative  data were not heterogenious  utilization and  of  each  formative  information  groups.  As  the  when tested for in the A N O V A ,  dimension  policy capturing  group average main  effects of the information dimensions, as reflected by the regression weights, tested for significance by comparing the means  was  with zero via a t  were  test (Norman,  1986). The average regression weight per group is presented in Table 5.  As well were  as  shown for  in Table  5,  all main effects  formative judgment,  effectively  utilized. As  were  which indicates  reflected  in  the  significant for summative  that  mean  all  as  information dimensions  weights,  presentation clarity  ANALYSIS was  AND RESULTS  /  73  the most heavily weighted dimension for both appraisal purposes, followed by  grading-marking for formative judgment Least  weighted  dimensions  were  and enthusiasm  resourcefulness  in  for summative formative  judgment.  judgment  and  grading-marking in summative judgment.  Table 5  Mean Regression Weights for Formative and Summative Judgment  Formative  Appraisal  Summative  Appraisal  Main Effects  Mean  t  Mean  t  bl-enthusiasm  1.43  15.23*  1.90  15.41*  b2-present,  3.17  21.11*  2.47  12.67*  b3-resourcefulness  1.04  10.64*  1.29  15.60*  b4-grading-marking  1.61  13.68*  1.21  11.23*  clarity  * p<.05.  3. Subjective Importance and Utilization Consistency  Hypothesis  3  consistent  with  hypothesis  can be  above.  stated  that  subjective  for  utilization  importance  gleaned  Nevertheless,  cue  from a  the  more  of  cue  pattern direct  in  performance dimensions.  of results  test  of  the  judgment Support  for hypotheses hypothesis  a  will for  be this  1 and  2  substantial  correlation between the mean importance ratings and the mean regression weights for  trait and behavior information was predicted.  ANALYSIS A  canonical correlation analysis  consistency reflected  by  subjective mean  between  regression  within  of  set  The  (ratings)  given  to  trait  relationship  used to test the  importance  weights.  importance  weight  linearity  subjective  was  of  between  multicollinearity were  two  ratings sets  trait  and  and  variables, to  of  and  utilization  normality  of  satisfactory  for  analysis  two  sets  of  test  for  eigenvalues.  p<.05; the  showed  variables.  second  a  Two The  was  significant canonical  first 0.30,  in  Table  6,  that  the  canonical  correlation  2  null  canonical  hypothesis variates  in the  the  and  as  mean  (2)  the The and  examination  of  1983).  was  0.55,  with  the  accounted  cues  distribution,  significant  3 was  74  of no  subject.  relationship  were  X (l) = 6.14, p<.05  correlation removed. As a result, seen  and substantial correlations  (1)  each  their  scatter plots and distributional statistics (Tabachnik & Fidell,  The  were  of  information,  information  be  null hypothesis  variables  behavior  behavior  found  AND RESULTS /  by  the  Bartlett's  X (4) = 29.99, 2  first  rejected, for  between  canonical  and it can be  large  amounts  variance in the original variables. Table 6  Variance Extracted from Original Sets of Variables by Canonical Variates  Original Sets of Variables  Variate  Cann.  Corr.  Importance Rating  Regression Weight  1  0.55  27.3%  59.1%  2  0.30  72.7%  40.9%  of  A N A L Y S I S A N D R E S U L T S / 75 4. Information Integration  Hypotheses  4  noncompensatory  and  5  information  addressed  the  integration  analysis of these hypotheses  use  strategies  in  compensatory  rating  and  judgments.  The  involved estimating linear and nonlinear mathematical  models of information use.  In a mathematical  strategy  of information produces  or configural use  of  model, use  of a  noncompensatory  significant interactions  among  cue dimensions (Billings & Marcus, 1983). Given the fractional factorial design of the rating task in this study, the six first-order interaction effects B C , B D , CD) were  Two model  measureable.  models  (ME) -  analysis.  The  were  the  estimated  linear  other  interaction effects  (AB, A C , A D ,  was  for  regression a  each  subject.  model  regression  One  developed  was  in  the  model including both  (MEI). For each subject,  the  policy  the  subtracting R - M E 2  main  effects  capturing  main and  the  from R - M E I  left  2  the proportion of variance explained by all two-way interaction effects,  R -INTER 2  (since the error variance is common to both models).  The proportion of variance explained by R ^ - I N T E R ranged from .01 to .13 and  .01  to  and  standard  interaction  .24  for  deviations  terms  interactive use  formative  are  of cues  of  and the  presented  summative amounts  in Table  of 7. It  ratings,  respectively. The  variance was  would result in a greater  accounted  assumed  amount  that  for the  means by  the  greater  of explained variance  (Billings & Marcus, 1983).  In  order  to  determine  whether  a  subject  addition to using a linear additive strategy,  integrated  cues  interactively in  hierarchical regression was performed,  ANALYSIS A N D RESULTS f  76  Table 7  Mean and Standard Deviation of Variance Explained  Formative  Group  Summative  Group  R s  M  SD  M  SD  R -MEI  .86  .07  .83  .09  R -ME  .81  .08  .76  .12  R -INTER  .05  .03  .07  .05  2  2  2  2  where  the main effects and interaction effects were predictor variables entered in  two  blocks.  block (linear  component),  and  the interaction terms were entered in the second block (nonlinear  component).  The  decision  due  second  The main effects were entered  block  information  rule  was  was  significant,  dimensions  integration strategy  Hypothesis summative  was  4  4,  the  if the the  increment subject  was  multiplicatively,  stated  that  condition  strategy  in  amount  used in an A N O V A  in  will  addition  is,  of variance  a  due  using  to  be a  formative cue  compensatory to  to  to  the  combining  the  noncompensatory  strategy.  combine  to  explained  considered  that  comparison  first  in variance  in addition to a compensatory  judgment  noncompensatory hypothesis  that  in the  the  judgment,  dimensions strategy.  interaction  raters using  To test  terms  and the null hypothesis  was not rejected.  a null  (R -INTER) 2  (Anderson, 1977). The effect of Appraisal Purpose  not significant F(l,68) = 2.34,  in  was  ANALYSIS At subjects  the  individual  in the  level,  eight  subjects  in  formative group (17.1%) were  both compensatory  the  summative  identified as  and noncompensatory strategies.  and in Figures 5 and 6 for the  severe  non-parallelism depicts the use  (Anderson,  1982). The graphs portray  four cues, subjects 3  (Fig.  3,  left  8  subjects  6 subjects in the formative in the  summative  combined the  average cue level  in addition to  pair  A  and  -  above  B,  below  3: Plot of Cell Means for subjects  Cues: A B *— > < — * C D  group. A  of an interactive, noncompensatory that  six  combining cues using  interactive  and  interactively. Similar pattern is noticeable for the rest of the  Fig.  and  use  strategy of  the  also used two pairs of cues interactively. For example, subject  panel)  below  (22.9%)  The configural impact of cue  dimensions is illustrated in Figures 3 and 4 for the group,  A N D R E S U L T S / 77  enthusiasm presentation clarity resourcefulness grading-marking  the  pair  C  and D ,  subjects.  average cue level  3 and 8 (Formative)  above  ANALYSIS  12.0 11.5 11.0 10.5 10.0 9.5 9.0 8.5 8.0 7.5 7.0 6.5 6.0 5.5 5.0 4.5 4.0 below  ^ 'gCues: F  —  . __  4  Subject 13  Subject 27  Subject 29  Subject 30  average cue level  above  Piot of Cell Means for Subjects 13 -'  :  o u p  A B C D  -  A N D R E S U L T S / 78  enthusiasm presentation clarity resourcefulness grading-marking  e c t s  below  average cue level  above  97 9Q A on ™ ^7. 29, and 30 (Formative)  ANALYSIS  AND RESULTS  12.0 11.5 11.0 10.5 10.0 9.5 9.0 8.5 8.0 7.5 7.0 6.5 6.0 5.5 5.0 4.5 4.0  Subject 2  Subject 5  12.0 11.5 11.0 10.5 10.0 9.5 9.0 8.5 8.0 7.5 7.0 6.5 6.0 5.5 5.0 4.5 4.0  Subject 6  Subject 15  below  Fig.  average cue level  above  below  average cue level  above  5: Plot of Cell Means for Subjects 2, 5, 6, and 15 (Summative)  Cues: A B C D  -  enthusiasm presentation clarity resourcefulness grading-marking  / 79  ANALYSIS  A N D R E S U L T S / 80  12.0 11.5 11.0 10.5 10.0 9.5 9.0 8.5 8.0 7,5 7.0 6.5 6.0 5.5 5.0 4.5 4.0  Subject 19  Subject 31  12.0 11.5 11.0 10.5 10.0 9.5 9.0 8.5 8.0 7.5 7.0 6.5 6.0 5.5 5.0 4.5 4.0  Subject 32  Subject 33  below  Fig. Cues:  #  average cue level  above  below  6: Plot of Cell Means for Subjects 19, 31, 32, A B C D  -  enthusiasm presentation clarity resourcefulness grading-marking  average cue level  above  and 33 (Summative)  ANALYSIS  A N D R E S U L T S / 81  Hypothesis 5 stated that in comparison to cognitively simple raters, complex raters will combine cue  dimensions  to a compensatory strategy.  using a noncompensatory  Cognitive complexity was  strategy  in addition  measured by the R E P grid  which was  scored for the total number of different pairs of responses across  constructs.  The  0.94,  using  smallest  reliability (Cronbach  the  total  score  the median was  In the  ratings  middle 20  Cognitive  percent  complexity  were  significantly  (M=91.32,  constructs.  the  two  most  groups  (N=14) scores  116 to different  SZ)=9.8  of  the  As  complexity.  and  ranged 188, on  subjects  from  M= 146.89,  (100  was  The smallest  items)  reversed, score  was  simple and cognitively  to  excluded  102,  with 28 subjects cognitive  scoring  were  66  grid  was the 67,  188.  (cognitively  of the  whole  the  and the largest score was  create  scores ranged from  ten  reflected  109,  order to  on  alpha)  the  and  from  complex),  the  cognitive  analysis. simplicity  in each group. The two groups  complexity  scores,  SD= 19.4  for  complex  due  to  t(54)= 13.52, p<.05, and  simple  v  groups,  respectively).  To that  is,  test  R -INTER  ANOVA  2  R -INTER  hypothesis was  (Anderson,  ^(1,54) = 7.45, 2  null  1977).  p<.05.  indicate  used  the  variance  as  the  dependent  The  Hence,  that  5,  effect  the  cognitively  null  of  nonlinear  variable for each  Cognitive  hypothesis  complex  the  raters  Complexity  was  rejected.  (M=6.2,  component,  subject in an was  significant  The  means  SD = 3.3)  of  combined  dimensions interactively more than the cognitively simple raters (M = 4.0, SD = 2.5), in addition to using a compensatory  strategy.  ANALYSIS AND RESULTS / 82 B. E X P L O R A T O R Y ANALYSIS  There were two questions of exploratory interest in the present study. One related to the measurement of good instructor schema, and the other concerned the effect of cognitive complexity on halo in ratings. The analysis and results pertaining to these questions are presented below in two separate sub-sections.  1. Measuring A Good Instructor Schema  The  impact  of  a  person's  schema  has  been  highlighted  in  several  information processing conceptions of performance evaluation (DeNisi, et al., 1984; Feldman,  1981).  attention  (Fiedler,  However, 1982).  measurement  of  Therefore, it was  schema  has  of interest  not to  received  explore  instructor schema profile could be measured. A measure of stereotype  much  if a good developed  by McCauley et al. (1978, 1980) was adapted to measure quantitatively the good instructor schema profile held by the raters.  It was expected that the attributes irrelevant to the good instructor schema would  not  hold a  diagnostic  value.  According  to  the  Bayesian  procedure by  McCauley et al. (1978, 1980), a diagnostic ratio (DR) indicates the strength of an attribute in the schema to the extent the ratio differs from diagnostic  ratios  were  (p(group/behavior)/p(group))  computed  from  probability estimates  the for  each  1.0. Therefore,  probability  estimates  of  attributes  the  ten  included in the questionnaire. The mean DRs for the entire sample were tested for difference from 1.0 by t tests, and the results are reported in Table 8. An examination of the pattern in the DRs indicates that because  the DRs for the  A N A L Y S I S A N D R E S U L T S / 83 irrelevant lower this  attributes  are  not  significantly  in comparison to  the  relevant  study  seems  a  valid  different  attributes,  procedure  for  from  the  measuring  1.0  and are  Bayesian a  good  also much  technique instructor  used  in  schema  profile.  Table 8  Mean Diagnostic Ratios in Good Instructor Schema Profile  Dimension  M  SD  t  Social workt  1.31  1.3  1.92  Travelling!  1.42  1.8  1.91  Enthusiasm  2.45  4.2  2.90*  Presentation  2.67  4.4  3.15*  Outgoing  2.09  3.6  2.53*  Resourcefulness  2.57  5.0  2.66*  Preparation  2.81  5.1  2.97*  Grading  2.53  4.5  2.83*  Leadership  2.18  3.6  2.74*  Communication  2.67  5.3  2.65*  * p<.05 t  irrelevant attributes  The into the  identification effect  of a good instructor schema  of appraisal purpose on attribute  permitted further exploration  strength  in the  schema profile.  ANALYSIS AND RESULTS / Trait  attributes  summative  were  expected  to  be  condition. In formative  be more salient  more  salient  than  condition, behavior  than trait attributes.  In statistical  behavior  attributes  terms,  treated  mean as  repeated The  was  the  of  the  of  behaviors  of  a  Profile  mean  within-subject  Appraisal  Appraisal  Schema  a significant  1  levels  ANOVA.  effect  i< (l,68) = 4.65, and  two  measures  main  effect  DR for trait items and the  Purpose  Purpose was  was  significant,  p<.05. The significant in  the  good  were expected  to  DR for behavior  was  Schema  the  items were  Profile,  significant,  p<.05.  Purpose and the  Schema  Profile  was  a  factor.  F(l,68) = 1.30. Also,  The there  Schema Profile,  interaction shows that the saliency  instructor  in  between-subjects  F(l,68) = 5.57,  interaction between Appraisal  in  expected.  factor,  not  attributes  an interaction between  purpose, and trait and behavior schema scores (DRs) was  The  84  of traits  influenced  by  the  purpose for judgment.  2. Cognitive Complexity and H a l o  Exploratory ratings  of  decomposed of  .71  analysis  complex  and  ratings  on ten  was  also done  simple  raters.  different  to  examine  the  In  rating  Task  dimensions.  (Cronbach alpha). The vignette evaluated,  only  five  of the  was  missing.  was  the  dimensions;  Two  average  average of  ratings  supplied, and the other was  made  contained  the  were computed on  average  B,  Rating Task  information pertaining to ratings  degree of halo  dimensions  the  provided  B had a reliability  information relating to  the  other  for each for  subjects  in  five  subject.  which  dimensions One rating  information  of ratings for the dimensions  information was withheld, resulting in a pair of ratings (x,y) for each  was  on which subject.  A N A L Y S I S AND The  ratings  subjects in the halo. The 0.77,  (x,y)  of the  subjects in the  simple group were correlated  corrrelations were 0.48,  p<.05  significant  for  in  the  both  correlations, was  cognitively  groups, the  weaker  correlations for the two by  28  z  Fisher's  show that  simple halo  r  and  raters.  Although  effect, indicated  the  by  the  for cognitively  (Guilford  &  28  degree of  correlations strength  complex  Frutcher,  complex individuals rated  the  cognitively complex raters  groups differed significantly, z = 1.76, of  group  separately to obtain the  p<.05, for the  (lower correlation)  transformation  cognitively  complex  R E S U L T S / 85  and  were of  the  raters.  The  p<.05, when 1978). These  tested results  with less halo compared to  the  cognitively simple raters.  In summary, the data did not either for subjective was effect  importance ratings or for cue  a significant effect of Cue of  Purpose  utilization  of  strategies but  support a main effect of Appraisal  and  Cue  information. cognitive  utilization. Nevertheless, there  Dimensionality, and  Dimentionality Purpose  did  on not  both  quantified, the  detected, and  the  also a significant interactive subjective  influence  complexity did. In exploratory  schema was  effect of appraisal  cognitively complex group rated  Purpose,  importance  information  and  integration  analysis, a good instructor  purpose on  schema profiles  with less halo than the  group. These findings are discussed in the following chapter.  was  simple  V. DISCUSSION  This final chapter provides a review of the in  relation to theories  findings  and issues in research considered  interpretation of the findings from confirmatory analysis (A) A  and second (B) parts. Exploratory results summary of the  The  strengths  sixth for  findings  and the  conclusions  (G) parts, respectively,  in earlier chapters. The is presented  are discussed  are  the  in the  first  in the third part (C).  are presented  and limitations of the study are discussed  (F) and seventh  and their interpretation  in part four  (D).  in part five (E). In the implications and directions  further research.  A. IMPORTANCE A N D UTILIZATION OF INFORMATION  1. Effect of Purpose  It  was  subjective did  speculated  importance  that  purpose  of performance judgment  and actual utilization of performance  will influence  the  information. The data  not support a main effect of appraisal purpose, either on importance ratings,  or on cue utilization.  Lack in  of support for a cognitive  some of the  previous  but certain differences  studies  effect of appraisal has  (Murphy,  between these  et  studies  al.  and the  These researchers  analyzed rating responses (the  variables  present  in  the  information (the  study  processes). Because  were  1984;  subjective  also been  Mclntyre, present  et  al.,  1984),  and  dependent  weighting  how information is mentally weighted  86  case  are worth noting.  product), whereas the importance  the  of  reflects  DISCUSSION / 87 cognitive  strategies,  direct in the al.,  the  present  investigation  study  of rater  than in others  cognition  (Murphy,  may  et  be  al.,  considered  1984;  more  Mclntyre,  et  1984). However, the main effect of appraisal purpose on rater cognition  was  not observed in the current study either.  Previous  findings  were inconsistent 1985,  Zedeck  on  the  (Murphy et  &  Cascio,  effect  of  al., 1984;  1982).  appraisal  purpose  Mclntyre et  In  an  al.,  on  raters'  1984;  investigation  of  Williams,  raters'  utilization policies, Zedeck and Cascio (1982) found a significant  cognition et al.,  information  effect of purpose,  but Williams, et al. (1985) suggested the opposite. This mixed findings were from studies that used different methodologies. capturing current  methodology  study  agreement present different  as  with  study  was  well. that  does  used  However,  to  not  methodologies,  lead  that  not underlying the inconsistent  is  investigate  the  of Zedeck and to  As did Zedeck and Cascio (1982), policy  findings  in the  Cascio's  a  (1982).  complete  policy  information present  purpose  interpreted  and cues was  in  study  not  are  the  resolution,  capturing versus  it  does  analysis  of  suggest  in the  that  ratings,  are  findings of previous research.  the interaction effects are significant be  in  Therefore, even though  The interaction effect of purpose and cue dimensions  cannot  utilization  isolation  significant,  in an analysis (Kirk,  the  1982).  effect  As  was  significant.  When  of variance, the main effects the  of purpose  predicted is  interaction  considered  of  further in  the discussion on the interactive effect of purpose and cue dimensionality.  DISCUSSION /  88  2. Effect of Cue Dimensionality  It  was  importance the  main  speculated  of  cue  dimensionality  importance  dimensionality.  of  information  uniform, but the effect was ratings.  cue  and actual utilization of performance effect  subjective  that  As estimated  by  The  and  utilization mentally rating  lies  in  TJ , 2  level  (Craik  & Lockhart,  judgments  involves  reporting involves  the  subjective  and  stronger  effect  speculation  then  11.8%  at  of the  which  1972).  importance  that  if  cue  cue  subjective  of  supported  dimensionality  similar  on  the  information  was  variance in importance ratings associated  with cue  information  It  is  conceivable  processing of  and  into  dimensionality  in  biases  likely to be more pronounced when one  the a  have  that  The  been  processed  making  concentration  formation  of  rating response.  judgments given  Therefore,  utilization  allows  performance judgment,  actually forms the judgment,  the than  incoming information is  information  enter  in information  actually  mental  information.  transformed  schema-induced  may  and  dimensions.  stronger effect of cue dimensionality  deeper  mentally  of  of  utilization  prototype matching processes, whereby  meaning  the  stronger in information utilization than in importance  for the  the  influence  information. The data  effect  29.6% of the variance in cue utilization was  One explanation  will  the the  they  are  than  when  the  effect  performance judgment.  After  one decides what kinds of information would be valuable.  As discussed of all,  cue it  strong  in chapter two,  dimensionality is  the  influence  on  content of  cue  there  information  of  are  few  processing  studies that in  address  information  on  which judgments  are  dimensionality  in  this  it  study  bears  based, out.  and  the  Besides,  the  DISCUSSION / 89 effect  of  cue  dimensionality  on  information  utilization  is  previous literature on human judgment research (Nisbett  a  common  & Ross,  finding in  1980; Wallsten  & Barton,  1982). A s cue dimensions in this study were traits and behaviors, one  aspect  cue  of  salience  appears  to  be  the  nature  of  the  information  concretely the information can be mentally represented (Nisbett,  An  examination of the  summative  and  presentation was  formative  clarity  in  grading-marking  dimensional use conditions,  arriving for  at  their  formative  gave  most  weight  rating judgments.  appraisal,  but  In  student teachers,  that in both  or  the  enthusiasm  priority second  for  appraisal. In fact, grading-marking was least utilized in summative raters were  how  et al. 1976).  of information showed  raters  -  to  place  summative  appraisal. The  and therefore, the clarity of presentation may have  been important to them.  The either  finding  purpose  studies in the Fox  effect  that  enthusiasm  of evaluation  is  was  not  the  surprising, when  Dr. Fox tradition (Naftulin, et have  suggested  that  al.,  expressive  most we  weighted  consider  dimension  the  conclusions  for of  1973). Researchers of the Dr. behavior  portraying  instructor  enthusiasm is an important source of variation in student ratings (Abrami, et al., 1982).  On  such  conclusions,  the  validity  doubted, even though many of the 1978). In the of  presentation  present  tapes  of  acted  lectures,  was  ratings  of  instructors  important but not to the  in making this  mind that in Dr. Fox studies enthusiasm video  students'  is  Dr Fox studies lacked internal validity (Frey,  study, enthusiasm  clarity. However,  of  whereas  was in  comparison we  exclusion  should keep  in  manipulated behaviorally by using  the  present  study  enthusiasm  presented as a concept and students attached their subjective meaning to it.  was  DISCUSSION / 90 Another finding in this study runs against on the  validity of students'  "grading-satisfaction" and  marks  1980).  they  The  ratings of instructors for summative  hypothesis  receive  finding  the  the  instructors  present  (Cohen,  investigation  1981; is  Marsh  to  dimensionality  had  a  strong  effect  on  &  the  grading-making was considered of least importance in summative  Cue  evaluation. The  is that students base their ratings on the  from  in  a popular belief that casts doubt  grades Overall,  contrary  evaluation.  subjective  importance  utilization of information, but the predicted interactive effect of cue  and  dimensionality  and purpose was significant as well. Therefore, the effect of cue dimensionality is further discussed in the next section.  3. Interactive Effect of P u r p o s e and Cues  As cue  predicted,  the  dimensionality  on  performance 11.8%  of  utilization,  the as  12.4%  of  supported  both  information.  The  main  importance  estimated  by  T? . 2  cue  the  nature  of  information  cue  ratings  and  of  actual  29.6%  of  interactive  purpose  of  accounted  for  variance effect  interactive  in  performance  in  cue  of purpose  variance in importance  Although the  and  utilization  ratings  effect  of this effect that leads to greater  processing  human resoning has multiple causes.  influence  dimensionality  and  4.1% of the  utilization.  relatively weaker, it is the significance into  of  Comparatively, the  accounted for in  interactive  importance  effect  in  variance  the  subjective  variance  and cue dimensionality and  data  judgment,  was  insight because  DISCUSSION / 91 It  was  importance  found  to  that  in  the  formative  behavior information; on the  judgment  condition  analysis,  similar results  was  raters  gave  more were  other  importance obtained  judgment  to  on  condition  gave  hand, raters in the  trait  information.  information utilization,  In  more  summative an  but  identical the  effect  stronger. The stronger interactive effect (like the stronger main effect of cue  dimensionality) processing involve  on  (Craik  deeper  information & Lockhart,  processing  utilization  may  be  a  result  1972). Actually making the  and mental  concentration  than  of  the  depth  rating judgments reporting the  of  may  subjective  importance of information.  Various theoretical points of the  of view can be drawn upon for the interpretation  interactive effect of appraisal purpose and cue dimensionality. It has  suggested that reference  or  appraisal purpose orients  schema,  which  the  guides the  been  rater to select an internal frame of  interpretation  of performance  information  (DeNisi et al., 1984). This conception implies that purpose operates as a priming stimulus  (Loftus  mentally  stored  and  Ilgen  prototype  and  &  Crocker, Klimoski,  prototypes  1974), or  is  (Bruner, 1971;  &  based  Mischel,  1977)  profiles and  content,  frame  that  reference  the  basic  behaviors  1981).  Ilgen  result  mechanism  Nisbett  traits  embedded  comprises  the  set  of  (1981) from  a  and utilization of information.  cognitive  1977;  comprise  is  rating judgments  interpretation  too,  of  (Taylor & Crocker,  Cantor & Mischel,  Schema  1984).  the  suggested  considered  1981).  1981). Appraisal  and  schemata  Feldman (1983)  processing  Crocker,  (Cantor  Loftus,  matching or schema  Schematic judgment  &  & Ross,  embedded in  traits  role  in  in 1980;  person  human Taylor schema  schema  (Taylor &  and behaviors  (Wexley &  DISCUSSION / Therefore,  it  seems  that  person and role schemata, the  information  behavior  related  purpose  activates  which provides to  information may  the  performance.  be that  more  than  information. Hence,  importance affected to  be  and  by  other  utilization  matches  of  that  appraisal purpose.  through  researchers  schematic  (Landy  and role  schemata,  it can be  The cognitive  1980;  and  in  et  of  processing  to  trait  and  respectively.  schema  profile  becomes  speculated  that  subjective  rating judgments  effect  not  Murphy  in terms  given  "initialized"  information  processing,  & Farr,  the  profile  importance  induced by person  information  schema  basis for evaluating or  The  Performance salient  a  92  is  indirectly  of appraisal purpose  as  direct  al.,  1984;  as  envisaged  Mclntyre  appears by  et  some  al., " 1984;  Williams, et al., 1985).  The effect of purpose on information utilization in the Cascio did  (1982)  not  offer  suggesting  to  study  is  interpreted in terms theoretical  subjects  of schema  interpretation,  utilized  and  information  utilization  restricted  from  as  well.  They  their  discussion  and  consumer  manager  The utilization of "bagging skill" and "skill in human relations" may  resulted  relate  be  any  that  perspectives. have  may  study by Zedeck and  role  from  schematic  and  person  possible,  then  pervasiveness  Certain  processing  schemata, taken  because  these dimensions  respectively.  together,  the  If  this  of information  interpretation  findings  would  of  their  support  the  of schema driven judgment in performance appraisal as well.  findings  in  the  current  study  (discussed  in  subsequent  sections)  support the above interpretation of the interactive effect of cue dimensionality and purpose. If it can be or implicit theories,  assumed  then the  that subjective  importance reflected  finding of consistency  mental  between subjective  models  importance  DISCUSSION / 93 and  cue  utilization  endorses  the  result  showing  that  purpose  supports  exploratory appraisal  schematic  the  good  processing  instructor  priming  function  explanation.  schema of  Likewise,  profiles  purpose  and  processing interpretation of the interactive effect of cue dimensions  the  varied the  with  schematic  and purpose.  4. Subjective Importance and Utilization Consistency  The data supported the importance analysis  ratings  revealed  information  and  that  they  utilization  reported as  weighting  of  significant  variates  depicting  significant variables  the  were  variates  Tabachnik & Fidell,  The weights,  close that  information  1977).  the  (Nisbett The  to  there  is  Unless had  a  difficult  revealed  subjective  two  Canonical  raters  correlation used  the  rating task.  An  importance and objective by  the  fact  importance  1983;  set  two  and' the  obtaining  in each  (Marascuilo & Levin,  that  ratings  relationship,  variables  subjective  actually  performing the  substantial  were only  between  two  of original  Pedhazur,  1982;  1983).  concluded that people  Levine,  utilization  been  actually  own judgment  information.  rating judgments,  dimensions  extracted.  relationship  is,  similar  relationship between subjective  when  would have  concerning consistency  important prior  information  cue  of  in making the  indication of a substantive  weights  hypothesis  between  consistency utilized,  is  subjective  importance  ratings  and  between information reported as interesting  because  some  objective  important and  researchers  have  are unable to report the importance of information in their & Wilson, substantive  utilization of information in this  1977;  Slovic & Lichtenstein,  relationship study  between  does not  support  1971;  importance such  a  Schmitt & ratings  and  conclusion. The-  DISCUSSION / finding  in  substituted around  this  study  self  reports  .60  Murphy,  between  1983;  Cook  is  similar  in  regression  the  actual  to  reported  equations  and  & Stewart,  that  in  and  found  predicted judgments  1975;  other  studies,  median  (Blazer,  Hoepfl & Huber,  1970).  94  which  correlations  Rohrbaugh & Some  of these  studies have been reviewed by Surber (1986). Surber's subjects were also able to self-report  the  which made  importance  him question  of  information  the  in judgment  validity of  extreme  of  children's  conclusions  achievement,  regarding people's  inability to report the importance of factors in their judgment.  B. INFORMATION INTEGRATION  Psychologists see  have  long studied judgment and decision processes (for  Einhorn & Hogarth, 1981; Pitz & Sachs,  Most  of  these  studies  multiattribute  judgment  analyses.  this  In  extract task  tradition,  1984;  mathematical  and the  infer  Slovic & Lichtenstein, 1971).  models  information  present  study  reviews  of  judgment  integration  investigated  made  from  how  in  a  regression  dimensions  of  information were mentally combined in performance judgment.  The  use  examined. high  One  and low  judgment chooses  of  was  information  The  use  hold  Hogarth,  1981;  the  compensatory  other  multiple  (Hogarth,  strategies  broad categories  of  information integration  strategy  levels of information between  task. to  two  in  a  was cut-off  1980). wide  the  The  existing  Pitz & Sachs,  1984;  a  dimensions  noncompensatory  strategies  variety  where  to  of judgment  tasks  may  in order to strategy  combine  literature  person  strategies  multiple  suggests (Dawes,  a  1979;  the  person  dimensions  that  Slovic & Lichtenstein, 1971).  trade-off  simplify  where  was  of  compensatory Einhorn  &  D I S C U S S I O N / 95 Mathematical for  individuals to  models, infer  specifically  multiple regression  their information integration  equation fits the compensatory strategy, effects  model could reflect  Einhorn,  1970,  whereas  a noncompensatory  1971; Einhorn & Hogarth,  variance explained by the  two  models  models,  strategies.  were  A  developed  linear additive  a nonlinear main and interaction  strategy  (Billings & Marcus,  1983;  1981). A comparison of the amount of  indicates  which model better  describes  the  strategies being used (Einhorn, et al., 1979).  Overall,  only  20  percent  of  the  subjects  in  this  study  combined  cues  interactively, that is, used some form of a nonlinear, noncompensatory strategy in addition  to  using  compensatory  a  compensatory  strategies  or  strategy.  linear  This  (Anderson, 1981;  &  decision  1971).  Also,  processing  behavior.  Linear  individual. The mean R raters,  respectively.  (1976)  suggestion  apply  this  These that  knowledge  consistency  values  2  addresses  were  consistently.  the  the  linear consistency indicated  and .83  weigh  However,  cues  by  the  hypotheses  were  tested  does not fairly  The results are discussed in the next two  the  a  large  1974;  Slovic  use  of  linear,  1967).  in information R  2  for  each  use  sub-sections.  of  support Brehmer's  accurately  linear consistency  concerning  in  for formative and summative  result of the number of cues and the correlation between the  Two  hold  & Corrigan,  recommend  high levels of consistency  individuals may  surprising finding for  (Edwards & Tversky,  was  .86  a  models  Dawes  theorists  compensatory strategies for optimal judgments  The regression modeling also  not  additive/averaging  number of judgment situations Lichtenstein,  is  is  but  fail  likely to be  to a  cues.  integration  strategies.  D I S C U S S I O N / 96  1. Effect o f P u r p o s e  The  data  did  not  support  integration  strategies  summative  condition and six  the  hypothesis  that  the  will vary with purpose of judgment.  of both compensatory  in the  formative  and noncompensatory  condition were  strategies.  the  ratee  vary  results  may  profiles  are varied. There is evidence  with  (Crowder, Zedeck  the  if the  & Hogarth,  (1977)  average,  concerning  different  presented  1981; nine  situations present  in  which  of  graduate  For  example,  models  integration corfigural  (Birnbaum & Stegner,  Norman & Louviere,  were  1974;  Stumpf needed  identified  as  as  information dimensions  in  strategy.  format  Payne,  of  1976;  dimensions,  the  judgment  Wright,  only  four  1974).  were  1976).  task When  effectively  school  applicants,  significant  or  cues  is  not  nonlinear,  1981;  Einhorn,  London  account  (1981)  for  the  always  1970,  1971;  1971).  the  noncompensatory  Stumpf & London,  and to  of  differences  case. cue  There  Linear  multiple  are  integration  Janis & Mann,  is  1977;  1981; Wallsten & Budescu, 1981).  found way  that student  configural and  or  manager  argued that the  regression  models  are  nonlinear subjects  wide application  of the linear model may in part be a result of an artifact of regression (Simon,  users  that people's judgment strategies  evaluated job applicants. Moreover, it has been  itself  in the  in their study concerning appraisal of nurses. In another  selection  linear  subjects  as well  existed between the two, four, and six cue conditions (Einhorn  Further,  information  For summative  number of  demand and presentation  Einhorn  Kafry  utilized on the study  cognitive  1976;  and  be  of  Eight  formative purposes, subjects generally used a compensatory  The  use  very  analysis  robust  with  DISCUSSION / 97 respect to departures from linearity (Dawes,  Nevertheless,  1979; Pedhazur, 1982).  an interesting observation was  that the  both compensatory and noncompensatory strategies, scores from  the  rest  of the  subjects.  14  subjects  differed on cognitive  The subjects  who  used  both  who  used  complexity  compensatory  and noncompensatory strategies were markedly more complex, and fell in the top quartile  (67  to  97)  sample  was  67  to  cognitive  of the 188  complexity  cognitive  (lower  on the  complexity  scores indicate  use  scores.  The range  for the  greater  complexity).  The effect of  of information combination strategies is  entire  discussed  next.  2. Effect of Cognitive Complexity  The hypothesis information  that cognitively complex raters would use  integration  supported.  A n analysis  terms  the  in  complex was  and  simple  raters  developmental  rater  greater seem  noncompensatory  Cognitive  of  regression  significantly  complex  strategy  to  in  the  addition  amount  model  revealed  groups. for  a  a  variance a  The mean  complex  have  of  to  raters,  tendency  compensatory  strategy,  accounted  by  significant  which  for  difference  variance  to  a noncompensatory  due  to  use  both  interaction  between  interaction  suggests  was  that  the  terms  cognitively  compensatory  and  strategies.  complexity constructs  is on  cue  a  developmental integration  research on human judgment processes  has  construct. been  (Pitz & Sachs,  The  generally 1984).  effect neglected  of in  Although  cognitive  complexity has been a variable of interest in performance judgment, the  emphasis  DISCUSSION / in  the  past has been mainly on psychometric properties of ratings. Few  if any, have investigated and its  the use  studies,  of information integration strategies by complex  simple raters. Yet, the finding in the present study is intriguing because congruence  Vannoy,  with cognitive  98  complexity  theory  (Bieri et  al.,  1966;  Kelly,  of  1955;  1965).  C. FINDINGS FROM E X P L O R A T O R Y ANALYSIS  1. Measurement of Schema  Numerous  authors  have  highlighted  the  role  of  schema  processing of information in performance appraisal (DeNisi, et Feldman,  1983).  schema  may  For  example,  determine  DeNisi  information  et  search  al.  suggested  and  and  al.  that  1984; a  interpretation  schematic  good  in  Ilgen & worker  performance  judgment.  Although widely accepted and to  as a theoretical construct, measurement  its utilization has received little attention dace  making  have  taken for granted that  appraisals.  associated  This  deficiency  in  with measuring schema.  raters  of schema  (Fiedler, 1982). Most of the studies apply their good worker schema in  research  Consequently,  is  perhaps the  due  present  to  the  study  difficulty  attempted  to  explore the extent to which schema could be measured quantitatively.  McCauley et individually stereotype  held holds  al. (1978, stereotypes. a  1980) In  diagnostic  outlined a procedure for the  this value  measure, if  its  a  characteristic  score  (diagnostic  measurement of related ratio)  to  the  differs  DISCUSSION / 99 significantly from one. good  instructor  stereotypes, Crocker,  This procedure was  schema  have  1981;  held  related  Wyer  by  and  adapted to measure  subjects  in  unrelated  & Srull,  this  study.  dimensions  1981), it was  quantitatively the  Since  (Hastie,  expected  that  schema, 1981;  confirmed  the  (community)  work behavior were  schemas as  indicated by the  expectation. of lesser  Professional  relevance  pattern of diagnostic  to  &  1980).  travelling  subjects'  ratios  Taylor  that is, would not  be considered strong attributes of the schema (McCauley, et al. 1978;  results  do  dimensions irrelevant  to a good instructor schema would not hold diagnostic values,  The  as  and  social  good instructor  in the  schema  profiles.  This affirmed the validity of the procedure used to measure schema. It should be mentioned,  however,  were  not  an  were  the  ones  that  exhaustive  the  relevant  list,  commonly  attributes  although  found  in  the  of  a  dimensions  performance  good  instructor  included  rating  in  scales.  the  The  schema measure  irrelevant  attributes were selected on an ad hoc basis.  As  the  schema  in  data  quantitative  examine  if  purposes.  The  terms  different  of trait  profiles  further  schema  profiles  of  and behavior  of  for  terms,  composition  raters. The saliency schema  provided support  raters  relative to trait dimensions formative  condition.  This  discussion  regarding the  were  the  was  and  notable  influence  it was  the  schema inference  appraisal purpose  good instructor undertaken  different  expected  to  for  condition; the  in the  supports of  so  a  was  by  to behavior dimensions  summative  finding  was  of  analysis  activated  profiles  dimensions,  in  measurement  exploratory  schema  of traits relative  the  be  to  appraisal  different  in  the  two  was  notable in the  saliency  profiles  of behaviors  of raters in the  drawn being  groups of  earlier  in  this  mediated  by  the  DISCUSSION / schemata  activated.  reasoning  will  information schema  However,  have  to  await  utilization. It  profiles  and  was  firm a  empirical  search  not  information  for  possible  confirmation  a  direct link  to  utilization  establish  in  the  for  this  between  the  direct  present  100  line  schema link  study  of and  between  because  all  dimensions in the schema measure were not included in the information.  2. Cognitive Complexity a n d Halo  Since  Schneier's  complexity  on  (1977)  performance  exploratory  appraisal  research,  has  been  the  emphasized  (Cooper, 1981; DeNisi, et al. 1984; Dunnette & Borman, 1983;  Landy  investigated  & Farr,  1980). Following Schneier's  the relationship between cognitive  ratings  (Bernardin,  Sauser  & Pond,  et  al.,  1981).  1982;  These  Cardy  studies  impact by  of  cognitive  many  authors  1979; Ilgen & Feldman,  findings,  numerous  researchers  complexity and halo in performance  & Carlyle, generally  1982;  Lahey  failed to  & Saal,  confirm that  1981;  cognitive  complexity affects the amount of halo in ratings.  The findings show that the skeptical conclusions the  importance  premature. rated  In  cogntive  comparison  to  with less halo. This is  but quite  contrary to  Carlyle,  1982;  present  study  construct 1984;  of  with  Lahey tend  to  respect  the  complexity the  cognitively  of others  1981;  support the to  Dunnette & Borman,  performance simple  a finding consistent  findings  & Saal,  in  appraisal  of earlier researchers  raters,  the  with Schneier's  (Bernardin,  Sauser  & Pond,  predictive  power  effectiveness  evaluation,  et  may  complex  of  the  1982;  cognitive 1981;  be  raters  (1977) results, Cardy &  1981). The results  (Cooper,  1979; Ilgen & Feldman,  al.,  about  of the  complexity  DeNisi,  1983; Landy & Farr,  et  al.  1980).  DISCUSSION / In many respects, not  find  studies, a  a  the present study was  significant  the present  measure  effect  of cognitive  study also  of cognitive  101  similar to those studies which did  complexity  on  halo.  As  did the  other  used the Role Construct Repertory (REP) grid  complexity  (Bieri, et al., 1966). Moreover, the  as  descriptive  statistics for cognitive complexity scores compare well with the norms provided by Schneier (1979), and the descriptive data reported in other studiest. 96  college  195,  subjects  with  the  range  of  (lower  scores  that  Sauser  median  66  the  in  to  at  188,  REP  Similarly,  the  had  that  the  (e.g.  an  had a  subjects  103  on  in  range  this  cognitive  improvement  male  measure  study  70  at  However,  roles  gave  (1981) the  median  complexity).  grid  critical-uncrtical)  96.  with  indicate  and Pond's  of  study  complexity in this  teacher)  and  face  validity  greater  For example, 66  to  had  a  scores  study  was  constructs for  (e.g.  use  in  a  performance appraisal task.  A major refinement in this study that may have brought about the result was results,  the measure  halo effect was  The  inappropriateness  al.,  (1986),  used by  the  who  questionable  Pulakos  correlations. correlation either  et  al.,  of halo effect.  In previous studies that reported negative  indexed by the  standard deviation of dimensional ratings.  of  noted  this  supplied  or  index  that  the  of halo  majority  standard deviation halo  Additionally, between  positive  effect  as  ratings  t Not all studies report scores of their samples  of  as  the  dimensions  thus,  the  creating  descriptive  been  published an index  present  described in chapter for  withheld,  in  has  on a  discussed  by  studies  they  of halo.  study 3,  halo  which  was  necessary  situation  statistics  was  for  where  the  Pulakos,  et  scrutinized,  As recommended measured measured  using by  information halo  cognitive  was  the was  highly  complexity  D I S C U S S I O N / 102 probable. Subjects on  were not forced into rating every dimension, but they did so  the basis of what was known about the ratee. However, the finding here  should be interpreted cautiously because, for exploratory purposes, only one ratee vignette (Rating Task B) was used in assessing the halo effect.  D. SUMMARY OF THE FINDINGS AND This appraisal  investigation began purpose,  with  CONCLUSIONS  the primary  cue dimensionality,  objective  and cognitive  of examining how  complexity  affect the  subjective importance, utilization, and integration of information in judgment. The task enviornment was performance judgment  of teaching i n higher education. The  effects of purpose and cue dimensionality were observed on subjective importance and utilization of trait and role information. The use of cue integration strategies was  examined  i n relation  to purpose  and cognitive  complexity.  Exploratory  analysis focused on the measurement of good instructor schema profiles, and on the effect of cognitive complexity  on halo  in performance  ratings. The findings  and the conclusions that can be drawn from these findings, are presented below.  1.  There  was  no  appreciable  effect  of appraisal  purpose  on subjective  importance and utilization of trait and behavior information in. performance judgment  of teaching.  Nor did purpose  bear  an  influence  on how  performance information was mentally combined. These findings suggest that appraisal  purpose  does  not have  a  direct  impact  on  raters'  mental  processing of information in performance judgment. Its effect, however, may be mediated by other factors because purpose did interact with the cues.  DISCUSSION / Cue  dimensionality  utilization  of  had a  strong impact on both subjective  information; the  effect  than in importance ratings. It is data on which judgments seems  an  important  performance content.  However,  the  the  stronger  importance  in information utilization  cues or information that  provide  Cue  affecting saliency  impact  the  utilization  may  be  of cues may  a  of  function  vary with  of  purpose  because  Appraisal  influenced  cue  dimensionality  conjointly  in  information  observed.  and  the  information  interaction between cue dimensionality and purpose was  purpose  and  are based. As a result, the nature of information  factor  judgment.  was  103  an  subjective  importance and utilization of trait and behavior information. On the  average,  raters valued (subjective importance) and utilized trait information more than behavior as  information in judgments  personnel  feedback  decisions.  on  the  saliency  appraisal  of  quality  of  information  purpose  has  a  For formative judgments, teaching,  more than trait information. This that  required for  an  is  effect  raters  summative where  the  utilized  purpose  such  rating provided  behavior  information  pattern of information utilization suggests a  function on  of  raters'  purpose  cognition  as  but  well, but  and  that  through  the  purposes,  but  schema it activates.  Information  dimensions  presentation  clarity, an aspect of behavior  attention other  in both summative  dimension  judgment.  were weighted  of  behavior  Enthusiasm,  a  differently  for different  information, was  and formative judgments. information  trait  was  dimension,  was  least  given the  Grading-marking,  weighted  important  in  but  most the  summative not  to  the  DISCUSSION / exclusion  of  other  student  evaluation  affected  by  dimensions of  rewards  of  information.  instructors in  terms  is  rational,  of  grades,  These and  or  104  Findings suggest  may  haloed  not by  that  be  necessarily  an  instructor's  enthusiasm.  There was  consistency  between what raters subjectively  considered important  information and their utilization of similar information in making the rating judgments.  This finding suggests that people's judgments  their^ subjective  values,  and that people  do have  the  are consistent  with  ability to report what  factors they may consider in making judgments.  Compared  to  the  cognitively  simple  raters,  complex  varied strategies in mentally combining dimensions performance. complex  Although the  individuals  indicates  that  subjects  used  cognitive  raters  made  use  of information related to  mainly used compensatory strategies,  noncompensatory complexity,  the  of  strategies  disposition  as  to  stimuli in a differentiated manner, a development  well.  This  the  finding  view multidimensional  construct, affects  the  use  of strategies in mentally integrating performance information.  A  lower  degree  complex subjects.  of  halo  effect  was  observed  Given their disposition to view  a differentiated manner, the cognitively prone  to  halo  psychometric  in  error.  Hence,  characteristics  of  cognitive  the  of  cognitively  multidimensional stimuli in  complex individuals seem to be less complexity  performance ratings,  indexed by correlational techniques.  ratings  may  also  especially  affect  when  halo  the is  DISCUSSION / 8.  The  validity  endorsed. to  be  of  a  schema  As expected,  nondiagnostic.  measure  of  items not related  This  finding  a to  indicates  good the  schema  that  quantitatively measuring stereotypes (McCauley,  instructor  a  profile  was  profile turned out  Bayesian  et al.,  105  procedure for  1978,  1980)  has  the  potential to be developed as a measure of schema.  Like  all  limitations.  research,  the  The conclusions  of the strengths  present  investigation  drawn above  has  should therefore  some  strengths  be entertained  and  in light  and limitations of the study discussed in the next part.  E. STRENGTHS AND LIMITATIONS OF THE STUDY In order to test hypotheses of theoretical exercise  control  over  the  experimental procedure was  information  significance,  provided  to  the  it  was  subjects.  chosen so that some of the extraneous  necessary  to  Therefore,  an  variables  (e.g.  the amount of information) could either be controlled. Consequently, a strength of the  study  was  compromise  its  its  external  can  have  more information about the different  given  the  occasions.  theoretical significance, study  were  to  be  results.  In  a  normal  appraisal  instructor, usually  Nevertheless,  as  a  means  limits  in  would have  vivo, only been  a  possible,  a  the  setting  obtained from for  crude  because the  raters  many  investigating  examination  study  may  generalizability  the  a simulated task offered certain advantages.  conducted  information processing  internal validity of  validity. The controlled setting  that  and  be  internal validity. However,  may  sources  questions If the  of  the  of  present raters'  nature and amount  of information is usually difficult to control in a normal situation.  DISCUSSION / The serious  concern  one,  integration  external  validity  because  findings  obtained  are  conditions, Levin et  for  not  predictive al. have  juror judgments, controlled factors  only  evidence  occupational setting  evaluated  choice, is  and  affect our daily lives" (p.  Besides,  meaningfully  reviewed  require  unsatisfying external 1983).  for  in  research  either  It  should be  teaching.  and  to,  need  studies  but  also,  (Levin, Louviere,  basic  dimensionality, However,  if the  on  one  dismiss  may  progressive  the  philosophy (cf.  a  information  under  certain  & Schepanski,  1983).  validity of laboratory studies on  and  decisions.  hiring ideal  to  place  determine  to  They  study  concluded, "The  how  judgment  can  pursue  two  the  and  different  both  that  goals  within  compromise  (Anderson,  1981,  the  p.  91).  raised in relation to the that  estimating  was  cognitive  findings  be  of external  pursue  goal"  intention  and  not  of  relevant  decisions  the  focus  that  to  clarify  complexity  here may  results and  Simon, 1968).  as search  on  the  vary from  lacking in for  effect  goal  of  As these goals may  one  study  results,  will  usually  rendering  Moreover,  study  values  the  the  the  them  issue  of  purpose of research (Mook,  of this  population  goals:  on  was  on information  student  evaluation  Description and understanding, rather than prediction, was  The  differences  to  reiterated  not  laboratory  goal of understanding behavior.  procedure  validity should be  processing,  goal.  "attempts  compromises  study  191).  psychological  incompatible,  present  related  the  integrated  predicting behavior, and the be  then  the  in  of real life behavior  laboratory  are  of  106  of  appraisal  formation  primary  purpose,  cue  rating  judgments.  an actual performance  evaluation,  external  conditions  of  the  of  validity,  that  would  or  adopt  account  a for  more the  DISCUSSION / The  confidence  in the  results  instruments  used to collect  investigation  were respectable  It study and  should be  the  pointed  analysis  Therefore, the results  on the  of the  and ranged from moderate (.7)  out that  followed  depends  data. The reliabilities  were sampled systematically the  of a study  a  fixed  the  cues and the  reliability of the  instruments  rather  purpose  here may be due to the  than  a  in  this  in  this  to high (.94).  conditions  in order to include those of most effects  107  significance,  random effects  model.  specification of the variables, and  generalization to other purposes and cues would require due caution.  F. I M P L I C A T I O N S  The relate  findings  in  this  study  to the theoretical points  research  and  policy  on  have  of view  several  implications.  that informed this  performance judgment,  and  These  implications  investigation,  relate  to  certain  relate  to  issues  in  performance appraisal and human judgment in general.  The  results  indicate  interactively  influence  information.  This  theoretical Farr, raters'  finding  models  1980).  the  of  is  appraisal  subjective has  not  these as  implications  models  direct  purpose  importance for  performance judgment  Although  cognition  that  as  seem  for  may  processing  operate  only  performance  as  by  (DeNisi,  et  dimensionality  utilization of  the the  in terms  stimulus  cue  the  performance  1983;  effect  of  authors.  purpose  The  particular  1984).  oriented Landy  of schematic  activating al.,  of  cognitively  & Feldman,  credible,  suggested  a priming  information  some  (Ilgen  purpose on a rater's cognition could be viewed Purpose  and  and  effect  & on of  processing. schemata  Furthermore, in  DISCUSSION / future  theorizing,  the  effect  of  purpose  could  be  discussed  in  relation  to  108 cue  saliency.  Cue saliency in past research on human judgment has been manipulated by varying  perceptual  current  study,  features,  trait  valuation (subjective  and  frequency, behavior  and  cues  the  had  order  an  of  effect  information. on  both  as  importance) and utilization. Therefore, theoretical  important  influence  on  information  use  the  information developments  addressing cue saliency should consider information structure in terms of dimensions  In  in judgment.  semantic  Information  content may determine how concretely the cues could be represented mentally.  The  concept  of  schema  utilization  as  a  heuristic  retrieving information from memory is well established psychological literature. Schema  and the  (DeNisi  assumed  to  et  al,  exist  1984;  Cooper,  and operate  first to provide some evidence on  schema  measurement  profiles  could  specific  schemata.  guide  the  be  and  that that  speculated  utilization  of  performance  biases  present  have  1983).  study  is  performance  processing.  tentative evidence  appraisal schema,  information.  judgment  and errors in performance  as a product of schematic  & Feldman,  The  to confirm their theories If  and social  purpose  activated  are  perhaps  the  There  is  to  activate  evidence  schema  appraisal may  schema  appraisal purpose,  than to explore is  that  others  driven, be  analysis  that  appears by  as  Schema  for this assumption. From the exploratory  is  1982).  in the cognitive  and  in performance appraisal models  Ilgen  theorized.  It  Fischhoff,  systematic  1981;  and utilization, we  quantified,  seek information mostly &  as  organizing  like concepts of stereotypes and implicit  personality theory have become key constructs well  for  better  people (Shaklee  causes  of  understood  DISCUSSION / The  cognitive  effect  schematic  processing  judgment.  The  information necessary make on  results  to to  as  make ensure  accurate  here  has  show  evaluations that  of  appraisal  speculated,  raters  appraisals.  accessibility  of  purpose,  even  implications  that  raters  for  different  have  access  for  may  if  accuracy in actually  purposes. to  the  information.  As  utilized for  different  purposes  there  should  be  require it  different will  appropriate information  are  may  be  many  performance evaluation documented by Bernardin and Beatty information  through  performance  Therefore,  Accuracy in performance judgment  relevant  mediated  109  to  dependent  purposes  (1984), the  clarified for the  be  of  types of  formulation  of prescriptive principles, and for the design of appropriate rating instruments.  As  a  study  of  valuation (subjective theory  (Anderson,  raters'  cognition,  at  their  investigation  1981)  provided this  perspective.  on how  rating responses.  they  As a  Systematic  done  in the  result,  past.  Lopes (1982), for example,  rating judgments  has  are  formed,  specifically,  which  in the  From such knowledge  past  has  mainly focused  errors (Bernardin & Pence,  If systematic  biases  1980;  can be  emerged  rating judgments  final ratings shown  or products,  that judgments  are produced. Thus,  identifying  and integration strategies, may accumulate knowledge rating judgments.  information  differences  analyzing how  be improved if one can identify how the judgments how  on  valued and utilized information in  are formed seems more informative than analyzing the as  focused  importance), utilization, and integration. Information integration  between individuals and groups, arriving  this  information  can  studying utilization  for prescriptions to improve  might flow implications for rater training, on  how  raters  could avoid  psychometric  Mclntyre, et al., 1984).  identified  in judgment  strategies,  specifically  in  DISCUSSION / information  valuation,  eliminated (Fischhoff, and  implicit  and integration,  1982). Investigating  theories  understanding Moreover,  utilization,  of the  on  of performance rating instruments teaching  behaviors  (Abrami,  evaluation  biases or cognitive  it may contribute toward the  question  reflect  might be  reduced,  the impact of mental models,  performance  cognitive  they  will  as  implicit theories  Leventhal, & Dickens,  a  affecting  to whether  if  not  schemata,  provide  distortions  110  better  judgments.  factor  structures  or dimensions  of actual  1981;  Larson,  1979;  Whitely  & Doyle, 1976).  The  results  complexity  theory,  compensatory analysis  of  of  Thus,  was  the  stressed DeNisi  effect  because  of  importance  al.  1984;  of the  use  cognitive  oriented  of a  complex  strategies.  The  complexity  on  complexity  1980).  rating task  that  cognitive  performance  appraisal research seems premature  judgment,  results  of  performance  complexity  of  raters  in this  may  not  the  this  of  both  exploratory  appraisal provide 1977), for halo complex to  raters.  be  conclusion  rightly  is  1981;  tenuous  Although further  the conclusion of some a  useful  (Bernardin, et  Lahey & Saal, 1981; Sauser & Pond,  cognitive  appraisal (Cooper,  study, be  of use  seems  in assessing halo.  researchers  If cognitive  made  However,  to reinforce the results  1982;  validity  raters  of performance  research is needed  & Carlyle,  the  simple than in cognitively  models  & Farr,  single  support  compatibility proposition (Schneier,  cognitive  Landy  to  that  in cognitively of  process  tend  finding  cognitive  stronger  in the et  the  also  noncompensatory  some support for the in ratings  study  given  and the  this  al.,  variable 1982;  in  Cardy  1981).  complexity of the rater is an important variable in. performance  it may be  used in rater selection.  One could also attempt  to  enhance  DISCUSSION / raters' cognitive  complexity through rater training. Although growth in a person's  cognitive complexity may not be easy to achieve, is possible (Sprinthall & Thies-Sprinthall,  Moreover, integration aspects,  Pitz  of  and  research  such  111  as  moral  Sachs  there is some evidence  that it  1983).  (1984)  pointed  out  on judgment  and  decision  development  (Rest,  1979),  that  there  processes which  has  with  may  been  little  developmental  affect  ability to treat multidimensional stimuli. The findings in the present  a  person's  study imply  that the difficulty people have in using different strategies may well be a result of their cognitive complexity, a developmental construct (Bieri, et al., 1966;  Kelly,  1955).  The  patterns  controversial Arguments (Centra,  of  issues have  1979).  information  concerning  been The  made chief  use  the  for  found  validity  and  concern  of  against  among  in  this  student the  study  evaluation  validity of  instructors  address  is  that  certain  of  instructors.  student  evaluations  students  may  be  overly biased by how the instructors grade and mark students work, that is, the "grading-satisfaction current  study,  hypothesis"  (Cohen,  grading-marking was  formative  or  in  summative  important  dimension  not  evaluation.  in summative  1981;  In  Marsh  given fact,  evaluation.  the  & Overall, highest  1980).  priority,  grading-marking was  Nor was  enthusiasm,  a  In  the  either the  in  least  factor in  "educational seduction," utilized to the exclusion of presentation clarity. Hence, the dismay  among instructors  about  the  validity of student  evaluations  seems to be  overstated.  The  finding  of  differential  use  of  trait  and  behavior  information  in  DISCUSSION / performance judgment raises  some concern regarding evaluation of teaching at the  school level. Educators advocate and  formative  achieving  function  the  at  that teacher  the  functions  same  relation  to  the  summative  and  perspective  as  might  in  of  formative well.  provide  important  purpose  feedback deciding  on the  summative  cognitive  cause  demands  are  (Payne,  be difficult  reasoned and unbiased  The consistency  once  but  which may  may  at  behaviors,  heuristics  difficult  appraisal, then  may  means  evaluation  greater  organizational  purpose  judgment  acknowledging  1983; Millman,  purpose  Neither  evaluation ought to serve  time,  from  (Darling-Hammond, Wise & Pease, in  112  load.  1976,  to  a  well  perspectives  Achieving  are  Thus,  accomplish without  appraisal for  dilemma  traits  Researchers  1982).  an  served  consider  erroneous judgment  in  1981). If information is utilized  poses  rating.  difficulties  behavior  conducting  be  the  summative  from  because equally both  a or  a  both  conitive  supervisor even  functions  in  more one  have  found that  simplifying  used  more often  when  the  twin  increasing  function the  task  of  teacher  chances  of less  evaluations.  between subjective  importance of information and utilization  of similar information is a finding that addresses an issue in human judgment in general. ability  to  1977; have  Some researchers report the  Slovic  have  importance  & Lichtenstein,  questioned  reached pessimistic of information  1971;  Schmitt  conclusions  in judgment  & Levine,  the validity of such conclusions  1985). The findings in this study raise doubts  1977).  concerning  peoples'  (Nisbett  & Wilson,  Other  researchers  (Ericson & Simon, 1980; Surber, about the validity of the  extreme  conclusions  concerning people's inability to report the importance of information in  their  judgment.  own  trained  Another  implication  of  this  for performing performance evaluation, the  finding  is  that  if  people  are  effect of the training is likely  DISCUSSION / to  transfer  subjective  positively,  values,  provided  schemata,  that  such  training  the  perception, judgment  results  confirmed  an  impact  on  raters'  or implicit theories.  The present study transferred theories person  has  113  and  certain  decision  and literature from the psychology of making  hypothesized  to  performance  relationships.  Thus,  appraisal, and  the  theories  and  research on person perception, and judgment  and decision  making may be  useful  in  judgment  called for. The  results  other  here  contexts  also  reinforce  particularly unravel other  as  the  where  idea  areas  of  the  where policy  causes rating  evaluative  that  information utilization  some  specifically,  well,  of  an and  attempt integration,  problems  judgments  capturing  to  in  are  research, and can be used for hypothesis  understand is  a  may  raters'  potentially  performance required.  methodology  is  be  a  rich  route  to  or  even  in  judgment,  Further,  cognition,  "lens  valuable  modeling",  tool  in  testing.  The findings and the implications which have been drawn from the suggest additional inquiry. Some  areas  such  findings,  of additional research are discussed  in the  next section.  G. DIRECTIONS FOR FURTHER RESEARCH As  discussed  several  purposes.  effects  model.  dimensionality Similar  by Bernardin and Beatty The present  Therefore, and  theoretic  study  how  influence  perspectives  used  other  utilization as  used  (1984), performance judgment  only  two  appraisal of in  of the  purposes  in a  fixed  purposes  interact  with  cue  information this  serves  study  is  yet  may  to be  be  determined.  drawn  upon  to  DISCUSSION / develop testable propositions. The present the  perspective  of  role  and  person  study  addressed cue dimensionality from  schemata,  and  the  predicted direction. Other theoretical bases for presenting should be explored as be  developed  consistency,  As content,  attribution  and distinctiveness  judgment  have  would  been  performance judgment  results  were  in  the  performance information  a further policy capturing study could  to  study  the  utilization  of  consensus,  information (Kelley, 1971).  are  (Einhorn  dimensions  studies  theory  strategies  and demand  information Few  from  well. For example,  114  quite  sensitive  & Hogarth, 1981), yield  done  on  different the  to  whether  results  effect  of  (Anderson, 1977). Ideally,  changes a  remains  varying  the design  in  task  format,  greater  number of  to  determined.  be  numbers  of  cues  will have  to be  in  such  that the amount of information is not confounded with the number of information dimensions. 1982),  As  further  and cognitive pressures  may  explicate  the  interaction  load, because performance judgment  order effects  neutralized profiles.  research  processes  is  (Payne,  between often  1976,  1980,  appraisal purpose  conducted  under  time  primacy  and  (DeNisi, et al., 1984).  The recency  information load affects judgment  by  in  which  on  judgment.  rotating  Information  information.  information In  the  squares)  the  presented  first  may  determine  of behavior cues, and vice versa.  presented  current  (Latin  We may therefore  concerning cue saliency  is  usually  study,  the  has order  information dimensions also  have  effects  if trait cues dilute the  Such research will have  and for the development  effect in  on  the  was ratee  subsequent  diagnostic  value  implications for theory  of rating instruments.  DISCUSSION / The tested that  as  cognitive a  despite  of  appraisal  between-subjects the  and summative errors  effect  factor  recommendation teacher  in judgment  may  creep  in this  of  evaluation  purpose  many  on  study.  information  inadvertantly.  A  seem  to  performing both  at once may be cognitively  in  utilization  The findings  educators,  further  115 was  imply  formative  so demanding that study  would  be  to  address this issue by using appraisal purpose as a within-subject factor.  The schematic  speculation  may  the  effect  of  appraisal purpose  is  mediated  through  processing needs empirical verification. It is suggested that schema  in the gaps" in the we  that  ask  if  information we  schema  and  exclusive  but which operate  compare  the  cognitive  receive  halo  are  (Taylor & Crocker, the  simultaneously.  aspects  of  halo  same  Hence,  with  1981). Therefore,  phenomenon,  a further  schema  "fill  or  study  utilization.  mutually  would be  Such  to  research  would be contingent upon developing a measure of schema at an individual level.  Increasing  rating  accuracy  is  a  prime  goal  in  performance  appraisal. We  may expect greater accuracy in the ratings if information is carefully scrutinized, compared, eliminated and weighted.  Hence, a further study could examine  rating  the  accuracy  is  a  function  of  combined by  the  rater. We may  compensatory  and noncompensatory  manner  test the  in  which  hypothesis  strategies,  make  the  that more  cues  raters  are who  accurate  whether mentally use  both  ratings  than  those who rely merely on compensatory strategies.  This  is  perhaps  one  of the  strategies in relation to cognitive replicate  the  findings  in  different  first  studies  complexity. judgment  to  explore  A n obvious situations  information  integration  next step would be  with  varying  tasks.  to  Such  DISCUSSION / research  would  complexity,  not  but  only  also  establish  reveal  the  how  well-founded  extent  to  which  is  the  theory  developmental  of  116  cognitive  constructs  may  relate to processes in human judgment.  Finally,  as  were positive,  the  the  results  from  the  Bayesian measure  exploratory  of schema  analysis  used  in  in this  this  investigation  study  needs to be  refined through further research. The effect of cognitive complexity on halo should also be studied further using correlational techniques  In  conclusion,  it  is  appropriate  to  note  that  to index halo.  if  performance  particular and human judgment in general is  to be improved, we  more  judgments.  about  how  investigation  people  speak  dimensionality,  and  to  the  the  arrive  at  need  for  construct  their  incorporating  of  cognitive  The  purpose  judgment have  findings  in  to learn in  this  for  judgment,  cue  in  research  and  complexity  theorizing on judgment processes. A n understanding of what information is utilized and  when,  judgments,  and may  what provide  factors a  affect  knowledge  the base  integration from  which  the fallibility in our judgments could be determined.  of  that  information  conditions  for  into  decreasing  VI. Abelson,  R.  P.  making.  (1976).  In  J.  S.  REFERENCES  Script  processing  in  Carroll  & J . W.  Payne,  attitude  formation  (Eds.),  behavior (pp. 33-45). Hillsdale, N J : Erlbaum.  Abrami, P. C , Dickens, W. J . , Perry, R. P., & Leventhal, teacher standards for assigning grades affect student instruction? Journal of Educational Psychology, 72, 107-118. Abrami,  P.  C,  Leventhal,  L., &  Perry,  R.  P.  (1982).  and  decision  Cognition and social  L . (1980). evaluations  Educational  Do of  seduction.  Review of Educational Research, 52, 446-464. Anderson, B. numbers  L . (1977). Differences in teachers judgment of verbal and numerical cues. Organizational Performance, 19, 68-88.  Anderson, N . H . (1981). Academic Press. Anderson, N . H . (1982). Academic Press.  policies  for  varying  Behavior and Human  Foundations of information integration theory. New York: Methods of information integration theory. New  Atkinson, R. C , & Shiffrin, R. M . (1971). The memory. Scientific American, 224, 82-90.  control  processes  of  York:  short-term  Balzer, W. K . , Rohrbaugh, & Murphy, K . R. (1983). Reliability of actual and predicted judgments across time. Organizational Behavior and Human  Performance, 32, 109-123.  Fischhoff, B. (1982). Debiasing (pp. 422-444). In D. Kahneman, P. Slovic, & A . Tversky, (Eds.), Judgement under uncertainty: Heuristics and biases. New York: Cambridge University Press. Bartlett,  F . C. (1932).  Bernardin,  H.  J.,  &  Remembering. Cambridge, M A : Cambridge University Press. Beatty,  R.  W.  (1984).  human behavior at work. Boston: Kent.  Performance appraisal: Assessing  Bernardin, H . J . , & Pence, E . C . (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65, 60-66. Bernardin, H . J . , Cardy, R. L . , & Carlyle, J . J . (1982). Cognitive and appraisal effectiveness: Back to the drawing board. Journal  Psychology, 67, 151-160.  Bieri,  complexity  of Applied  J . , Atkins, A . L . , Briar, S., Leaman, R. I., Miller, H . , & Tripodi, T. (1966). Clinical and social judgment: The discrimination of behavioral information. New York: Wiley. 117  REFERENCES  /  118  Billings, R. S., & Marcus, S. A . (1983). Measures of compensatory and noncompensatory models of decision behavior: Process tracing versus policy capturing. Organizational Behavior and Human Performance, 31, 331-352. Birnbaum, M . H . , & Stegner, S. E . (1981). Measuring the importance of cues in judgment for individuals: Subjective theories of IQ as a function of hereditary and environment. Journal of Experimental Soical Psychology, 17, 159-182. Borko, H . , & Cadwell, J . (1982). Individual differences in teachers' decision strategies: A n investigation of classroom organization and management decisions. Journal of Educational Psychology, 74, 598-610. Borman, W. C , & Dunnette, M . D. (1975). Behavior based versus trait-oriented performance ratings: A n empirical study. Journal of Applied Psychology, 60, 561-565. Borman, W. C. (1978). performance ratings. Brehmer,  B.  (1976).  Exploring  upper  limits  of  reliability  and  validity  in  Journal of Applied Psychology, 63, 135-144.  Social  judgment  theory  and  the  analysis  of  interpersonal  conflict. Psychological Bulletin, 83, 985-1003. Bruner, J . S. (1957). On perceptual readiness. Bruner, J . S.  (1971).  Psychological Review, 64, 123-152..  Beyond the information given: Studies in the psychology of  thinking. New York: W. W. Norton & Co. Bruner, J . S., Goodnow, J . , & Austin, York: Wiley. Brunswik, E . (1952). Press.  G . A . (1956).  A study of thinking. New  Conceptual framework of psychology. University of Chicago  Cadwell, J . , & Jenkins, J . (1985). Effects of the semantic similarity of items on student ratings of instructors. Journal of Educational Psychology, 77, 383-393. Cadwell, J . , & Jenkins, J . (1986). Teachers' judgments about their students: The effect of cognitive simplification strategies on the rating process. American  Educational Research Journal, 23, 460-475.  Cantor, N . , & Mischel, W. (1977). Traits as prototypes: Effects on recognition memory. Journal of Personality and Social Psychology, 35, 38-49. Cantor, N . , & Mischel, W. (1979). Prototypes in person perception. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 3-52). New York: Academic Press.  In  L.  12, pp.  REFERENCES /  119  Cardy, R. L . , & Kehoe, J . F . (1984). Rater selective attention ability and appraisal effectiveness: The effect of cognitive style and the accuracy of differentiation among ratees. Journal of Applied Psychology, 69, 589-594. Centra, J . A . (1979).  Determining faculty effectiveness. San Francisco: Jossy-Bass.  Chapman, L . J . , & Chapman, J . P. (1969). Illusory correlation as the use of valid psychodiagnostic signs. Journal of Abnormal 271-280.  Psychology, 74,  Cohen, P. A . (1980). Effectiveness of student rating feedback college instruction: A meta-analysis. Research in Higher 321-341.  Education, 13,  Cohen,  student  achievement:  Review  of Educational  P. A . (1981). Student  ratings  meta-analysis of multi-section Research, 51, 281-309.  of instruction  validity  and  studies.  an obstacle to  for  Collins, A . M . , & Loftus, E . F . (1975). A spreading activation semantic processing. Psychological Review, 82, 240-247.  improving  theory  A  of  W. S., & Zelen, M . (1959). Fractional factorial experiment designs for factors at three levels (Applied mathematics series, No. 54). Washinton D C :  Connor,  US Cook,  Department of Commerce.  R. L . , & Stewart, T. R. (1975). A comparison obtaining subjective desciptions of judgmental policy. and Human Performance, 13, 31-45.  Cooper, W. H . (1981). Ubiquitos halo. Craik,  of  seven  methods  of  Organizational Behavior  Psychological Bulletin, 90, 218-244.  F . I. M . , & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671-684.  Crocker,  J.  (1981).  Judgement  of  covariation  by  social  perceivers.  Bulletin, 90, 272-292.  Psychological  Crowder, R. G. (1976).  Principles of learning and memory. Hillsdale, N J : Erlbaum.  Cutting, J . E . (1987). 38, 61-90.  Perception  and  information.  Annual Review of Psychology,  Darling-Hammond, L . , Wise, A . E . , & Pease, S. R. (1983). Teacher evaluation in the organizational context: A review of the literature. Review of Educational  Research, 53, 285-328.  Davis, J . H . , Spitzer, C. E . , Nagao, D . H . , & Stasser, Brandstatter & J . H . Davis, & H . Schuler, (Eds.), Decisions (pp. 33-52). Beverly Hills, C A : Sage.  G.  (1978).  In  H.  Dynamics of Group  R E F E R E N C E S / 120 Dawes, R. M . (1979). The robust beauty of improper linear making. American Psychologist, 34, 571-582. Dawes,  R.,  &  Corrigan,  Psychological Bulletin, DeCotiis,  T . , & Petit,  B. (1974). 81, 95-106.  A . (1978).  Linear  models  The performance  models in decision  in  decision  appraisal  process:  making.  A  model  and some testable propositions. Academy of Management Review, 3, 369-373. DeNisi, A . S., Cafferty, T. P., & Meglino, B . M . (1984). A cognitive view of the performance appraisal process: A model and research propositions. Organizational Behavior and Human Performance, 33, 360-396. Dickinson, T . L . , & Zellinger, P. M . (1980). A comparison anchored rating and mixed standard scale formats.  Psychology, 65, 147-154. Dixon, W. J . (Ed.), (1983). BMD University of California Press.  biomedical  of the  behaviorally  Journal  of Applied  computer programs. Berkley:  Dunkin, M . J . , Barnes, J . (1986). Research on teaching in higher education. In M. C. Wittrock, (Ed.), Handbook of research on teaching (3rd. ed., pp.  754-777). New York: Macmillan. Dunnette,  M . D . & Borman, W. C . (1979).  Personnel  selection and  classification  systems. Annual Review of Psychology, 30, 477-525.Easterby-Smith, M . (1980). How to use repertory grids in H R D . Human  Development, 4(2), 3-32.  Resource  M . (1982). The design, analysis and interpretation of repertory grids: CSML working paper, (pp. 49). Lancaster, U K : C S M L working paper,  Easterby-Smith,  University of Lancaster. Edwards, W . (1968). Conservatism in human Kleinmuntz, (Ed.), Formal representation of York: Wiley. Edwards, W . , & Tversky, A . (1967).  information  processing.  In  B.  human judgment (pp. 17-52). New  Decision making. Baltimore: Penguin Books.  Einhorn, H . J . (1970). The use of nonlinear, noncompensatory making. Psychological Bulletin, 73, 221-230.  models in decision  Einhorn, H . J . (1971). The use of nonlinear, noncompensatory models as a function of the task and amount of information. Organizational Behavior and Human Performance, 7, 86-106. Einhorn, H . J . , & Hogarth, R. M . (1978).  Confidence in judgment:  the illusion of validity. Psychological Review, 85, 395-416.  Persistence of  REFERENCES /  121  Einhorn, H . J . , & Hogarth, R. M . (1981). Behavioral decision theory: Process of judgment and thought. Annual Review of Psychology, 32, 53-88. Einhorn, H . J . , Kleinmuntz, D . N . , & Kleinmuntz, B. (1979). Linear regression and process-tracing models of judgment. Psychological Review, 86, 465-485. Elstein,  A.  S.,  Shulman,  L.  S.,  &  Sprafka,  S.  A.  (1978). Medical problem Harvard University  solving: An analysis of clinical reasoning. Cambridge: Press.  Epstein, S. (1980). The stability of behavior: Implications research. American Psychologist, 35, 790-806. Ericson,  K . A . , & Simon, H . A .  (1980).  Verbal  reports  as  for  psychological  data.  Psychological  Review, 87, 215-251. Evertson, C. M . , & Holley, F . M . (1981). Classroom observation. In J . Millman (Ed.), Handbook of teacher evaluation. Beverly Hills, C A : Sage. Feldman, J . M . (1981). Beyond performance appraisal. Journal Feldman, K . teachers  A. as  attribution  theory:  Cognitive  processes  in  of Applied Psychology, 2, 127-148.  (1983). related  Research in Higher  The seniority and instructional experience of college to the evaluations they receive from their students. Education, 18, 93-124.  Fiedler, K . (1982). Causal schema: Review and popular construct. Journal of Personality 1001-1013.  criticisim  of  research  on  and Social Psychology,  a  42,  Frey, P. W . (1978). A two dimensional-analysis of students ratings of instruction. Research in Higher Education, 9, 60-91. Frey,  P. W . , Leonard, D. W . , & Beatty, W. W. (1975). Student ratings instruction: Validation research. American Educational Research Journal, 327-336.  of  12,  Garner, W. R. (1974). The processing of information and structure. Hillsdale, N J : Erlbaum. Goldberg, L . R. (1968). Simple clinical judgment. American  models  or  simple  processes?  Some  research on  Psychologist, 23, 483-496.  Grimmett, P. P. (1984). The supervision conference: A n investigation of supervisory effectiveness through analysis of participants' conceptual functioning. In P. P. Grimmett, (Ed.), Research in teacher education: Current problems and future prospects in Canada (pp. 131-166). Vancouver, B.C: University of British Columbia.  R E F E R E N C E S / 122 Guilford,  J . P., & Frutcher, B. (1978).  psychology. Tokyo: McGraw-Hill.  Fundamental statistics in education and  Hamilton, D . L . (1979). A cognitive-attributional analysis of stereotyping. In L . Berkowitz (Ed.), Advances in experimental social psychology (Vol. 12, pp. 53-84). New York: Academic Press. K . R. (1980). The integration of research in judgment and Boulder, Colorado: Center for Research in Judgment University of Colorado..  Hammond,  theory'.  decision Policy,  Hastie, R. (1981). Schematic principles in human memory. In E . I. Higgins, C. P. Herman, & M . P. Zanna (Eds.), Soical cognition: The Ontario symposium  (Vol. 1, pp. 39-88). Hillsdale, N J : Erlbaum.  Hawley, R. C . (1982). Assessing Research Associates. Hays, W. L . (1981).  teacher performance. Amherst, M A : Educational  Statistics (3rd. ed.). New York: Holt, Rinehart & Winston.  Hildebrand, M . , Wilson, R. C , & Dienst, E . R. (1971). Evaluating university teaching. Berkeley: Center for Reasearch and Development in Higher Education, University of California. Hoepfl,  R. T . , & Huber,  G. P. (1970).  A study  of self-explicated  utility  models.  Behavioral Science, 15, 408-414. Hoffman,  P.  J.  (1960).  The  paramorphic  Psychological Bulletin, 57, 116-131.  Hogarth, R. M . (1980). York: Wiley.  representation  of  clinical  judgment.  Judgement and choice: The psychology of decision. New  Hogarth, R. M . (1981). Beyond discrete biases: Functional and dysfunctional aspects of judgmental heuristics. Psychological Bulletin, 90, 197-217 Holland,  J.  H . , Holyoak,  K . J . , Nisbett,  R.  E., &  Thagard,  P.  R.  (1986).  Induction: Processes of inference learning and discovery. Cambridge, M A : M I T Press. Ilgen,  D . R., & Feldman, J . M . (1983). Performance appraisal: A process In B. M . Staw, & L . Cummings (Eds.), Research in organizational  (Vol. 5, pp. 141-197). Greenwich, C T : J A I Press.  Jacobs, R., Kafry, D . , & Zedeck, S. (1980). Expectations rating scales. Personnel Psychology, 33, 595-640. Janis, I. & Mann, L . (1977). Kahneman, D. (1973).  focus.  behavior  of behaviorally anchored  Decision making. New York: Free Press.  Attention and effort. New York Prentice Hall.  REFERENCES /  123  Kahneman, D . , Slovic, P., & Tversky, A . (1982). (Eds.). Judgement under uncertainty: Heuristics and biases. New York: Cambridge University Press. Kahneman,  D.,  &  Tversky,  A.  (1972).  Subjective  probability:  A  judgment  of  representativeness. Cognitive Psychology, 3, 430-454. Kahneman,  D.,  &  Tversky,  A.  (1973).  On  the  psychology  of  prediction.  Psychological Review, 80, 237-251. Kahneman,  D.,  &  Tversky,  A.  (1984).  Choices,  values,  and  frames.  American  Psychologist, 39, 341-350. Kane,  J . S., & Lawler, E . E . (1979). Performance appraisal effectiveness: Its assessment and determinants. In B. Staws (Ed.), Research in organizational behavior (Vol. 1, pp. 425-478). Greenwich, C T : J A I Press.  Kelley, H . H . (1971). Learning Press. Kelly,  G. A . (1955). Norton Press.  Attribution of social interaction. Morristown, N J : General The psychology of personal constructs (Vol. 1). New York:  Kennedy, J . L . (1971). The system approach. A preliminary exploratory the relation between composition and financial performance in games. Journal of Applied Psychology, 55, 46-49. Keppel, G . (1973). Design N J : Prentice Hall. Kirk,  and analysis: A researcher's handbook. Englewood Cliffs,  R. E . (1982). Experimental design: ed.). Belmont, C A : Brooks/Cole.  Krantz,  D. H . , Luce,  study of business  R. D., Suppes,  Procedure for the behavioral sciences (2nd  P.,  & Tversky, A . (1971).  measurement (Vol. 1). New York: Academic Press.  Foundations of  Kubovy, M . (1981). Concurrent-pitch segregation and the theory of indispensible attributes. In M . Kubovy, & J . R. Pomerantz, (Eds.), Perceptual Organization  (pp. 55-98). Hillsdale, N J : Erlbaum. Kuhn,  T. S. (1970). The structure University of Chicago Press.  of scientific revolutions (2nd ed.). Chicago:  Kulik,  J . A , & McKeachie, W. J . (1975). The evaluation of teachers in higher education. In F . Kerlinger, (Ed.), Review of Educational Research (Vol. 3,  pp. 210-240). Itasca, IL: Peacock. Lahey, M . A . , & Saal, F . E . (1981). Evidence incompatible with a cognitive compatibility theory of rating behavior. Journal of Applied Psychology, 66, 706-715.  REFERENCES / Landy, F . J . , & Farr, 87, 72-107.  J . L . (1980).  Performance  rating.  Psychological Bulletin,  Larson, J . R. (1979). The limited utility of factor analytic study of implicit theories in student ratings of teacher  techniques behavior.  Educational Research Journal, 16, 201-211. Levin,  124  I. P., External  Louviere, J . J . , Schepanski, A . A . , & Norman, K . validity tests of laboratory studies of information Organizational Behavior and Human Performance, 31, 173-193.  for  the  American  L . (1983). integration.  Loftus, G . R., & Loftus, E . F . (1974). The influence of one memory retrieval on a subsequent memory retrieval. Memory and Cognition, 3, 467-471. Lopes, L . L . (1982). Procedural debiasing. Technical Report Wisconsin Human Information Processing Program. Leyton, M . (1986). Principles of the human cognition system.  information  structure  No. 15. Madison, WI:  common  to  six  levels  of  Information Science, 38, 1-120.  L . A . , & Levin, J . R. (1984). Multivariate statistics in the social sciences: A Researcher's guide. Monterey, C A : Brooks/Cole.  Marascuilo,  Marsh, H . W. (1983). Multidimensional ratings of teaching effectiveness by students from different academic settings and their relation to student/course/ instructor characteristics. Journal of Educational Psychology, 75, 150-166.  Marsh, H. W. (1984). Student's evaluation of university teaching: Dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology, 76, 707-754..  Marsh, H . W., & Overall, J . U . (1980). Validity of students' evaluations of teaching effectiveness: Cognitive and affective criteria. Journal of Educational  Psychology, 72, 468-475. McCauley, C , Durham, M . , Copley, J . B., & Johnson, J . P. (1985). Patients' perceptions of treatment for kidney failure: The impact of personal experience on population predictions. Journal of Experimental Social  Psychology, 21, 138-148.  McCauley, C , & Stitt, C. L . (1978). A n individual and quantitative measure stereotypes. Journal of Personality and Social Psychology, 36, 929-940. McCauley,  C , Stitt,  C. L . , & Segal,  M . (1980).  Stereotyping:  From  of  prejudice to  prediction. Psychological Bulletin, 87, 195-208. McGreal, T. C. (1983). Successful teacher Association of Curriculum Development.  evaluation. Alexandria,  Virginia:  REFERENCES  /  125  Mclntyre, R. M . , Smith, D. E . , & Hassett, C. E . (1984). Accuracy of performance ratings as affected by rater training and perceived purpose of rating. Journal of Applied Psychology, 69, 147-156. Medley, D. (1979). The effectiveness of teachers. In P. Paterson & H . Walberg, (Eds.), Research on teaching: Concepts, findings and implications. Berkely, CA: McCutchan. Medley, D. M . (1982). Teacher competency testing and teacher education. Charlottesville, VI: Association of Teacher Educators and the Bureau of Educational Research, University of Virginia. Menasco, M . B. (1976). Experienced conflict in decision making as a function of level of cognitive complexity. Psychological Reports, 39, 923-933. Mervis,  C. B.,  &  Rosch,  E.  (1981).  Categorization  of  natural  objects. Annual  Review of Psychology, 32, 89-115. Meier, R. S., & Feldhusen, J . F . (1979). Another look at D r . Fox: Effect of stated purpose for evaluation, lecturer expressivness and density of lecture content on student ratings. Journal of Educational Psychology, 71, 339-345. Miller, G . A . (1956). Magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97. Millman, J . (1981).  Handbook of teacher evaluation. London: Sage.  Minsky, M . (1975). A framework for representing knowledge. In P. H . Winston (Ed.), The psychology of computer vision. New York: McGraw Hill. Mitchell,  T.  R.  (1970).  Leader  complexity  and  leadership  Personality and Social Psychology, 16, 166-174. Mook,  D. J . (1983). 379-387.  In  defense  of  external  validity.  style.  Journal of  American Psychologist, 38,  Murdock, B. B. (1961). The retention of individual items.  Journal of Experimental  Psychology, 62, 618-625. Murphy, K . R., & Balzer, W. K . (1986). Systematic distortions in memory in memory-based behavior ratings and performance evaluations: Consequences for rating accuracy. Journal of Applied Psychology, 71, 39-44. Murphy, K . R., Balzer, W. K . , Kellam, K . L . , & Armstrong, J . G . (1984). Effects of the purpose of rating on accuracy in observing teacher behavior and evaluating teaching performance. Journal of Educational Psychology, 76, 45-54.  REFERENCES Naftulin, D . H . , Ware, J . E . J r . , & Donnelly, lecture: A paradigm of educational seduction. 48, 630-635. Neisser, U., specified  & Becklon, R. events. Cognitive  (1975).  Selective  F. A.  (1973).  / 126  The Dr. Fox  Journal of Medical Education, looking:  Attending  to  visually  Psychology, 7, 480-494.  U . (1976). Cognition and reality: Principles and implications of cognitive psychology. San Francisco: Freeman.  Neisser,  Newell, A . , & Simon, H . A . (1972). N J : Prentice Hall. Nie,  N . H . , (1983), (Ed.). York: McGraw-Hill.  Human Problem Solving. Englewood Cliff's,  SPSS: Statistical package for the social sciences. New  Nisbett, R., Borgida, E . , Crandall, R., & Reed, H . (1976). Popular induction: Information is not necessarily informative. In J . W. Payne, (Eds.), Cognition and social behavior (pp. 113-133). Hillsdale, N J : Erlbaum. Nisbett,  R., & Ross,  L . (1980).  Human inference: Strategies and shortcomings of  social judgment. Englewood Cliffs, N J : Prentice Hall. Nisbett, R. E . , & Wilson, T . D. (1977). The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology, 35, 250-256. Norman,  K . L . (1986).  Importance  of factors  in the review  of grant  proposals.  Journal of Applied Psychology, 71, 156-162. Norman, K . L . , & Louviere, J . J . (1974). transportation: Two modeling approaches. 753-758. Ortony,  A.  (1978).  Remembering,  Integration  of  attributes  in bus  Journal of Applied Psychology, 59,  understanding,  and  representation.  Cognitive  Science, 2, 53-69. Payne, J . W. (1976). Task complexity and contingent making: A n information search and protocol analysis. and Human Performance, 16, 366-387. Payne,  J.  W. (1980).  Information  processing  theory:  processing  in  decision  Organizational Behavior  Some  concepts applied  to  decision making. In T . S. Wallsten (Ed.), Cognitive processes in choice and  decision behavior (pp. 95-115). Hillsdale, N J : Erlbaum. Payne, J . W. 382-401.  (1982).  Contingent  decision  behavior.  Psychological Bulletin,  Pedhazur, E . J . (1983). Multiple regression in behavioral research: prediction (2nd ed.). New York: Holt, Rinehart & Winston.  92,  Explanation and  REFERENCES /  127  Peterson, D . , Micceri, T . , & Smith, B. O. (1985). Measurement of teacher performance: A study in instrument development. Teaching and Teacher Education, 1, 63-67. Piaget, J . (1936/1970). La naissance de Vintelligence chez I'enfant. intelligence in children, 1970]. Paris: Delachau et Niestle. Pitz,  G. G . , & Sachs, N . J . (1984). Judgment and decision: application. Annual Review of Psychology, 35, 139-163.  [The growth  Theory  of  and  Pulakos, E . D . , Schmitt, N . , & Ostroff, C. (1986). A warning about the use of a standard deviation across dimensions within ratees to measure halo. Journal of Applied Psychology, 71, 29-32. Raifa, H . & Schlaifer, R. (1961). Business School.  Applied statistical decision theory. Mass: Harvard  Rasinski, K . A . , Crocker, J . , & Hastie, R. (1985). Another look at sex stereotypes and social judgments: A n analysis of social perceiver's use of subjective probabilities. Journal of Personality and Social Psychology, 49, 317-326. Rest,  J . R. (1979). Development of Minnesota Press.  in judging moral issues. Minneapolis: University  Rosch, E . (1978). Principles of categorization.  In E . Rosch, & B. B. Lloyd (Eds.),  Cognition and categorization (pp. 27-48). Hillsdale, N J : Erlbaum. Rosch, E . , & Mervis, C. structure of categories.  (1975).  Family  resemblances:  Studies  in  internal  Cognitive Psychology, 7, 753-605.  Rumelhart, D. E . (1977). Understanding and summarizing brief stories. In D. LaBerge & S. J . Samuels (Eds.), Basic processes in reading: Perception and  comprehension (pp. 265-303). Hillsdale, N J : Erlbaum.  Rumelhart, D. (1980). Schemata: The building blocks of cognition. In R. Spiro, B. Bruce, & W. Brewer, (Eds.), Theoretical issues in reading comprehension. Hillsdale, N J : Erlbaum. Rumelhart, D. E . , & Ortony, A . (1977). The representation memory. In R. C. Anderson, R. J . Spiro, & W. E .  Schooling and the acquisition of knowledge (pp.  of knowledge in Montague (Eds.), 99-135). Hillsdale, N J :  Erlbaum. Sauser, W. I., & Pond, S. B. (1981). Effects of rater training and participation on cognitive complexity: An exploration of Schneier's cognitive reinterpretation. Personnel Psychology, 34, 563-577.  Scripts, plans, goals and understanding: An inquiry into human knowledge structures. Hillsdale, N J : Erlbaum.  Schank, R., & Abelson, R. P. (1977).  REFERENCES / Schmitt, N . , & Levine, R. L . (1977). Statistical and subjective problems and proposals. Organizational Behavior and Human 15-30. Schneider,  D.  J.  (1973).  Implicit  personality  theory:  A  weights:  R. M . (1977). Controlled I. Detection, search, and  Review, 84, 1-66.  review.  Psychology, 62, 541-548. E . (1979). and norm  Measuring tables for  Psychological  and automatic human attention. Psychological  Schneier, C. E . (1977). Operational utility and psychometric behavioral expectation scales: A cognitive reinterpretation.  Schneier, C. validity,  Some  Performance, 20,  Bulletin, 79, 294-309. Schneider, W., & Shiffrin, information processing:  128  characteristics  of  Journal of Applied  cognitive complexity: Developing reliability, a personality instrument. Educational and  Psychological Measurement, 39, 599-612. Schoemaker, P. J . H . (1982). evidence and limitations. Shaklee,  H . , & Fischhoff,  The expected utility model: Its  variants,  purposes,  Journal of Economic Literature, 20, 529-563.  B.  (1982).  Strategies  of information  search  in  causal  analysis. Memory and Cognition, 10, 520-530. Shaklee, H . , & Mims, M . (1982). Sources of error in judging event covariations: Effects of memory demands. Journal of Experimental Psychology: Learning,  Memory, and Cognition, 8, 208-224.  Shavelson, R. (1985, April). Schemata and teaching routines: A historic perspective. Paper presented in the symposium on "Teacher Thinking: Relationships among Schemata, Routines, and Teaching Effectiveness", Annual meeting of the American Educational Research Association, Chicago. Shavelson, R. J . , Webb, N . M . , & Burstein, L . (1986). Measurement of teaching. In M . C. Wittrock, (Ed.), Handbook of Teaching (3rd. ed., pp. 50-91). New York: Macmillan. Shulman, L . S. (1986). Paradigms and teaching: A contemporary perspective.  research on teaching (3rd ed., pp.  research programs in the study of In M . C. Wittrock (Ed.), Handbook of 3-37). New York: Macmillan.  Simon, H . A . (1968). On judging the plausibility of theories. In J . F . Staal & B. V a n Rootselaar, (Eds.), Logic, methodology and philosophy of science. The  International Congress for Logic, Methodology and Philosophy of Science, III. Amsterdam: New  Holland.  Simon, H . A . (1974). How big is a chunk?  Science, 183, 482-488.  REFERENCES  / 129  Simon, H . A . (1976). Discussion: Cognition and social behavior. In J . S. Carroll & J . W. Payne (Eds.), Cognition and social behavior (pp. 253-267). Hillsdale, N J : Erlbaum. Simon, H . , & Newell, A . (1971). Human problem solving: theory in 1970. American Psychologist, 26, 145-159.  The state  of the  Slovic, P., & Lichtenstein, S. (1971). Comparison of Bayesian and regression approaches in the study of human processing in judgment. Organizational Behavior and Human Performance, 6, 649-744. Slovic,  P., Fischhoff,  B . , & Lichtenstein,  S.  (1977).  Behavioral decision  theory.  Annual Review of Psychology, 28, 1-39. Categories and concepts. Cambridge, M A :  Smith, E . E . , & Medin, D. L . (1981). Harvard Univeristy Press.  Sprinthall, N . A . , & Thies-Sprinthall, L . (1983). The need for theoretical development in educating teachers: A cognitive development perspective. In K . R. Howey & W. E . Gardner, (Eds.), The education of teachers: A look  ahead. New York: Longman.  Stiggins, R. J . , & Bridgeford, N . J . (1985). Performance assessment for teacher development. Educational Evaluations and Policy Analyses, 7, 85-97. Stumpf, S. A . , & London, M . (1981). Capturing rater policies in evaluating candidates for promotion. Academy of Management Journal, 24, 752-766. Surber, C. F . (1985). Measuring the importance of information in judgment: Individual differences in weighting ability and effort. Organizational Behavior and Human Decision Processes, 35, 156-178. Tabachnik, B. G . , & Fidell, L . S. (1983). Harper & Row.  Using multivariate statistics. New York:  Taylor, S. E . , & Crocker, J . (1981). Schematic bases of social information processing. In E . Higgins, C. Herman, & M . Zanna (Eds.), Social cognition: The Ontario symposium (Vol. 1, pp. 89-134). Hillsdale, N J : Erlbaum. Thies-Sprinthall,  L . (1980).  Supervision:  A n educative  or  mis-educative  process.  Journal of Teacher Education, 21(4), 17-20. Thorndike,  E . L . (1920).  A  constant  Applied Psychology, 4, 25-29.  error  in psychological  ratings.  Journal of  Tomkins, A . (1979). Script theory: Differential magnification of affects. In H . E . Howes, & R. A . Dienstbier (Eds.), Nebraska symposium on motivation (Vol. 26). Lincoln, Neb: University of Nebraska Press.  REFERENCES / Tversky,  A.  (1972).  Elimination by  aspects:  A  theory  of  choice.  130  Psychological  Review, 79, 281-299. Tversky, A . , & Kahneman, D. (1973). Availability: A heuristic frequency and probability. Cognitive Psychology, 5, 207-232. Tversky, A . , & Kahneman, D. (1974). Judgement and biases. Science, 185, 1124-1131.  under  Tversky, A . , & Kahneman, D. (1981). The framing psychology of choice. Science, 211, 453-458. Vannoy, J . S. personality 385-396.  (1965). Generality construct. Journal  of  cognititive  for  judging  uncertainity:  of  decisions  Heuristic  and  complexity-simplicity  as  of Personality and Social Psychology,  the  a  2,  Vaughn, G . , & Corballis, M . C. (1969). Beyond tests of significance: Estimating strengths of effects in selected A N O V A designs. Psychological Bulletin, 72, 204-213. von  Neumann,  J.  &  Morgenstern,  O.  (1947).  behavior (2nd. ed.). N J : Princeton University.  Theory of games and economic  Wallsten, T. S. (1980). Processes and models to describe choice and inference behavior. In T. S. Wallsten, (Ed.), Cognitive processes in choice and decision behavior (pp. 216-237). N J : Lawrence Erlbaum & Associates. Wallsten, T. S., information  & Barton, C. for decisions.  (1982).  Processing  probabilistic  multidimensional  Journal of Experimental Psychology: Learning, Memory, and Cognition, 8, 363-384.  Wallsten, T. S., & Bedescu, D. V . (1981). Additivity and nonadditivity in judging M M P I profiles. Journal of Experimental Psychology: Human Perception and Performance, 7, 1096-1109. Webster, W. C. (1964). Decision making in the employment Industrial Relations Center, McGill University.  interview. Montreal:  Weldon, E . , & Gargano, G. M . (1985). Cognitive effort in additive task groups: The effect of shared reponsibility on the quality of multiattribute judgments. Organizational Behavior and Human Decision Processes, 36, 348-361. Wexley,  K.  N.,  &  Klimoski,  R.  (1984).  Performance  appraisal:  An  update.  Research in Personnel and Human Resources Management, 2, 35-79. Whitley,  S.  E., &  Doyle,  K.  O.  (1976).  Implicit  theories  Ameircan Educational Research Journal, 13, 241-253.  in  student  ratings.  REFERENCES  / 131  Williams, K . J . , DeNisi, A . S., Blencoe, A . C , & Cafferty, T . P. (1985). The role of appraisal purpose: Effects of purpose on information acquisition and utilization. Organizational Behavior and Human Decision Processes, 35, 314-339. Winer, B. J . (1971). Statistical York: McGraw Hill.  principles in experimental design (2nd ed.). New  Wright, P. (1974). The harrased decision  maker: Time pressures, distractions and  the use of evidence. Journal of Applied Psychology, 59, 555-561. Wyer,  R. S., & Srull, T. K . (1981). Category accessibility: Some theoretical and impirical issues concerning the processing of social stimulus information. In E . Higgins, C. Herman, & M . Zanna (Eds.), Social cognition: The Ontario  symposium (Vol. 1, pp. 161-197). Hillsdale, N J : Erlbaum.  Zedeck, S., & Cascio, W. F . (1982). Performance appraisal decisions as a function of rater training and purpose of the appraisal. Journal of Applied  Psychology, 67, 752-758.  Zedeck, S., & Kafry, D. (1977). Capturing rater policies for processing evaluation data. Organizational Behavior and Human Performance, 18, 269-294.  VII.  A.  APPENDIX  GLOSSARY  appraisal  purpose  serve.  - the  There  are  function which a performance judgment  two  common  functions  university instructor: summative in  making  provides study  personnel  feedback  an  such  instructor  formative judgment  was  as  on  tendency  cognitive  error  -  multidimensional  his/her  a  person's  manner,  as  evaluation  Formative  quality of the  of  to  to  of  a  used  judg,emt  teaching.  need  In  this  improve, and  to the instructor for improvement.  - rating toward the middle of the  complexity  intended  Summative judgment is  promotions.  an expression  excluded guidance and recommendations central  performance  and formative.  decisions  to  of  is  disposition  measured  by  a  to  scale. view  modified  behavior  version  of  in the  a  Role  Construct Repertory grid (Bieri, et al., 1966). compensatory  strategy  dimensions  of  - a category  information  of mental  by  strategies of integrating  trading-off  between  dimensions  different  using  an  additive or averaging rule. The amount of variance explained by the linear component strategy  in  the  regression  (Billings  model  indicated  &  Marcus,  1983;  the  nature  of  an  terms  of  a  the  Einhorn,  use  of  1970,  a  compensatory  1971;  Weldon  &  Gargano, 1985). cue  dimensionality  -  performance comprised behavior  profile  an  item  dimension  in of  information  comprised  item trait  of or  information behavior.  A  concerning a personality  an  item  of  information  presented trait  in  a  dimension  characteristic;  concerning  a  a  role  (teaching) behavior. decomposed  rating  - a  numerical rating  on  one  of  the  separate  seven  point  interval scales included in rating Task B. good  instructor attributes  schema in  the  - the  abstract mental distribution of behavior and trait  prototype  of  a  good  university  instructor.  The  good  instructor schema was measured by a Bayesian procedure, adapted from the stereotype measure or  the  outlined by McCauley et al. (1978,  characteristics  encoded  in the  by the diagnostic ratios in the schema 132  good  instructor  measure.  1980). The attributes schema  were  indexed  APPENDIX / halo  effect  -  an  effect  of  the  information that causes the study,  halo  separate information  was  indexed  into  a  similarity in decomposed  by  the  correlation between  judgment,  indexed  - the weight assigned  the  rating  capturing analysis  by  ratings. In the decomposed  error  judgments.  The  regression  of  present  ratings  of  combining using  nonlinear  compensatory  or  a  policy  portrayed a subject's information utilization policy.  (Slovic  strategy different  -  component  a  category  dimensions  mulitiplicative  noncompensatory  of information when the  - rating on the side of leniency  noncompensatory  a  to dimensions  & Lichtenstein, 1971; Zedeck & Cascio, 1982;  and  dimensions  strategy.  utilization  formulating  leniency  between  - the mental combination of items of information in the final  noncompensatory information  similarity  rater groups. integration  mind  perceived  133  of  rules.  in  the  strategy  mental  amount  regression  (Billings  1977).  favourableness.  information  The  in  Zedeck & Kafry,  or  of  weights  strategies  by of  & Marcus,  determining variance  model  of  indicated  1983;  interactively cut-off  explained the  use  Einhorn,  levels by  the  of  1970,  a  1971;  Weldon & Gargano, 1985). performance  profile  - a profile description of a hypothetical  comprising items of performance  related  university instructor  information in terms  of traits  and  behaviors. rating  judgment  - an overall numerical rating given  a hypothetical  university  instructor on an  to  a profile description of  18 point interval scale, indicating  suitability for promotion or need for improvement. stringency  error  - rating on the side of severity  schematic  processing  or  unfavourableness.  - schema based interpretation and utilization of information  (Taylor & Crocker,  1981)  A P P E N D I X / 134  B. IMPORTANT INFORMATION M E A S U R E  1. For Summative Condition  Different  types  information  evaluation  of  dimensions  listed  information  on some dimensions than  in  a  of  university below.  can be  instructor.  In making  obtained  in  The information  your  to  make  may reflect  rating judgment,  others.  making promotion decisions, indicate  order  Because your  for each dimension  different  you may like ratings  an  more  will be used  how important  will  it be for you to receive information of a particular type. Circle a number on the scale  provided  important,  to  the  right  of each  dimension.  On these  scales  1  =  least  4 = important , and 7 = most important.  Planning, preparation  1  2  3  4  5  6  7  Enthusiasm  1  2  3  4  5  6  7  Lecture  1  2  3  4  5  6  7  Sociability  1  2  3  4  5  6  7  Resourcefulness  1  2  3  4  5  6  7  Grading, marking  1  2  3  4  5  6  7  Leadership  1  2  3  4  5  6  7  Communication  1  2  3  4  5  6  7  Warmth  1  2  3  4  5  6  7  1  2  3  4  5  6  7  Research  presentation  activity  APPENDIX /  135  2. For Formative Condition  Different  types  information  in  listed  information  on some dimensions than others. Because the  express  teaching,  a  indicate  information  of a  need  In  for  your  improvement  for each dimension particular type.  right of each dimension. and 7 =  making  The  obtained  dimensions  below.  instructor.  be  of  to  university  can  evaluation  is  a  of  -  information  order may  rating judgment,  to  make  reflect  you  may  purpose  an  different like  more  of your rating  provide feedback, on the quality of  how important will it be for you to receive  Circle  On these scales  a  number 1 =  on  the  scale  least important,  provided 4  =  to  the  important ,  most important.  Planning, preparation  1  2  3  4  5  6  7  Enthusiasm  1  2  3  4  5  6  7  Lecture  1  2  3  4  5  6  7  Sociability  1  2  3  4  5  6  7  Resourcefulness  1  2  3  4  5  6  7  Grading, marking  1  2  3  4  5  6  7  Leadership  1  2  3  4  5  6  7  Communication  1  2  3  4  5  6  7  Warmth  1  2  3  " 4  5  6  7  1  2  3  4  5  6  7  Research  presentation  activity  A P P E N D I X / 136 C. P E R F O R M A N C E RATING TASK A  1. For Summative Condition  There are 27 profilest of instructors presented here, one per page. You are asked to rate each one of these profiles on the scale at the bottom of the page. Try  not to compare  one profile with  another  - it is important  each profile on its own merit. Do the ratings on your  subjective  that you rate criteria,  and  use the same criteria for all the profiles.  Each profile is comprised of observations teaching. The observations  are  recorded at  made on 4 dimensions related to  three levels: below average,  average,  and above average. It is suggested that you complete the rating of these profiles in one session. You may take as long as you wish.  It is important that you keep in mind the function your rating will serve. Remember that your rating is required to make promotion  decisions  on  the  instructors whose profiles are presented here. Because promotions are crucial decisions affecting the institution as well as the individual, evaluative ratings become imperative. In considering  these instructors for promotion, the heads of  the  departments and the deans will use your ratings in making their decisions. Promotion to a higher rank means granting pay increases and perhaps tenure. Therefore, you are asked to evaluate these instructors very thoughtfully. Please turn over the page and begin the Rating Task A .  t Only one is included here.  A P P E N D I X / 137  Instructor P13  Observation Recorded as "XX" Information Dimension  Below Average  enthusiasm  Above Average  Average  XX  presentation  clarity  XX  resourcefulness  XX  grading and marking  XX  How suitable i s this instructor for PROMOTION to a higher rank? C i r c l e a point on the scale below :  1 2  3  4  5  1 1 1 I very poor  6  7  1 1  8  9  10 11 12 13 14 15 16 17 18  I  average  I  1 I 1 I- I  I I outstanding  please turn over to the next page.  APPENDIX / 138 2. F o r Formative Condition  There are 27 profiles! of instructors presented here, one per page. You are asked to rate each one of these profiles on the scale at the bottom of the page. Try  not to compare one profile with another - it is important that you rate  each profile on its own merit. Do the ratings on your subjective criteria,  and  use the same criteria for all the profiles.  Each profile is comprised of observations made on 4 dimensions related to teaching. The observations are recorded at three levels: below average,  average,  and above average. It is suggested that you complete the rating of these profiles in one session. You may take as long as you wish.  It is important that you keep in mind the function your rating Remember  that  improvement  or  the  main  provide  purpose  of  feedback.  your  rating  Evaluative  is  ratings  to  express a  provide  the  will serve. need for instructors  information on their effectiveness. The ratings will not be seen by the heads of the departments or any one else, and will not affect pay or tenure of the instructors. However, the general evaluative feedback you will provide may lead the instructors to improve their performance for the benefit of other students. As instructors need evaluative  feedback to self-improve,  you  are asked  to  evaluate these instructors  carefully.  Please turn over the page and begin the R a t i n g Task A.  t Only one is included here.  APPENDIX /  Instructor  F24  Observation Information Dimension grading  Below Average  .  and marking  Recorded as  Above Average  Average  XX  enthusiasm  ..  presentation  "XX"  clarity  ..  XX  XX  resourcefulness  ..  XX  E v a l u a t e t h i s i n s t r u c t o r ' s performance. In order to provide h i m / h e r some F E E D B A C K , c i r c l e a p o i n t o n t h e s c a l e b e l o w :  1  very  2  3  4  5  6  7  8  I  I  1  I  I  1  I  poor  9  10  11  I I  12  13  I  14  turn over  I  16  17  I I  18  outstanding  average  please  1  15  to  the  next  page.  139  APPENDIX /  140  3. Coding and Rotation of Profiles  C  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 • 19 20 21 22 23 24 25 26 27  0  D  I  N  ROTATION  G  A  B  c  D  0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2  0 2 1 0 1 1 0 2 1 1 0 2 1 0 2 1 0 2 2 1 0 2 1 0 2 1 0  0 1 2 2 0 1 1 2 0 0 1 2 2 0 1 1 2 0 0 1 2 2 0 1 1 2 0  0 1 2 2 0 1 1 2 0 0 1 2 2 0 1 1 2 0 0 2 2 2 0 1 1 2 0  A B C D A B C D A B C D A B C D A B C D A B C D A B C  B C D A B C D A B C D A B C D A B C D A B C D A B C D  Source for coding: Connor & Zelen (1959). Values: 0 - below average, 1 - average, 2 - above average.  C D A B C D A B C D A B C D A B C D A B C D A B C D A  D A B C D A B C D A B C D A B C D A B C D A B C D A B  APPENDIX  D. COGNITIVE There  are  COMPLEXITY 1 0 different  / i4i  MEASURE  persons  to  rate  on  1 0 different  dimensions.  The  persons are listed on top and the dimensions are listed on the right of the grid below.  In rating  the persons, focus  to mind in each or  indecisive)  that  case.  For each  you feel  on a particular  person, choose  is best for the  use the scale corresponding the category (e.g.  individual  the  rating  individual  you may bring  category  you have  (e.g decisive  in mind.  1, 2, 3 for decisive or 4,  indecisive) to provide a rating for that person in the appropriate cell  5,  Then 6 for  in the grid.  When 3'ou finish, all cells in the grid will be filled up with a rating.  1 1  I indecisive  decisive 1  i  rf 1 introvert  extrovert 4  considerate 4 practical I  1-  r  progressive i  '  1  uncritical t  •  1  open-minded I  1  1  good-humored I  ,  1  systematic  I——r-  ,  ,  -f r-r impractical  1  .  independent I  I  inconsiderate  1  dependent r-  conservative f-+critical —  1  .  close-minded I  1  t  ill-humored  I  1  r  unsystematic 1  APPENDIX  /  142  E. GOOD INSTRUCTOR SCHEMA MEASURE Directions:  Attached question  asks  four sets of questions concerning university instructors. Each  are for  your  best  estimate  of  percentages in whole numbers between  You  are  not  expected  However,  you  are  requested  estimate.  Make  sure  you  to to  a  proportion.  State  your  estimates  as  1 and 99.  know  the  exact  percentages  complete  every  question  understand the  difference  for  based  between  the  questions.  on  the  your  four  best  different  sets of questions. Example: 1.  What percentage of students do their homework regularly?  2.  What percentage of G O O D students do their homework regularly?  3.  What  percentage  GOOD  students?  4.  A L L students  who  do  you  notice  second question is students  the  difference?  about G O O D  who really are G O O D  homework  but  their  homework  %  regularly are  %  What percent of A L L students are G O O D  Did  doing  of  %  is  asking  First  students?  question is  %  about A L L students.  The  students  only. The third question is about A L L  students.  The Final question is not specific about  for  what  percentage  of  students  are  GOOD  questions.  Please  students generally.  You  are  requested  not  to  come  back  to  the  completed  turn over the page and begin with the first set of 10 questions.  APPENDIX / First set of 10  143  queations  1.  What percentage  of instructors show enthusiasm?  2.  What percentage  of instructors present the material with clarity?  3.  What percentage  of instructors have outgoing personalities?  4.  What percentage  of instructors are resourceful?  5.  What percentage  of instructors plan and prepare thoroughly?  6.  What  percentage  of  community service?  instructors  actively  % % %  %  participate  in  % social  work  or  %  7.  What percentage  of instructors grade papers very well?  8.  What percentage  of instructors show leadership?  9.  What percentage  of instructors do a lot of professional travelling?  10.  What percentage  of instructors are effective communicators?  Please do not revise your estimates. Go onto the next page.  % % % %  APPENDIX Second set of 10  /  144  questions  1.  What percentage  2.  What  of G O O D instructors show enthusiasm?  percentage  of  GOOD  instructors  present  the  %  material  with  clarity?  % 3.  What percentage  of G O O D instructors have outgoing personalities?  4.  What percentage  of G O O D instructors are resourceful?  5.  What  percentage  of  GOOD  instructors  plan  %  %  and  prepare  thoroughly?  % 6.  What percentage  of G O O D  community service?  instructors actively  participate in social work or  %  7.  What percentage  of G O O D instructors grade papers very well?  8.  What percentage  of G O O D instructors show leadership?  9.  What percentage  of  GOOD  instructors  do  a  lot  of  % %  professional  travelling?  % 10.  What  percentage  of  GOOD  instructors  are  effective  %  Please do not revise your estimates. Go onto the next page.  communicators?  APPENDIX /  145  Third set of 10 questions 1.  What  percentage  instructors? 2.  What percentage  of  A L L instructors  What percentage  of A L L instructors  What  percentage  instructors? 5.  What percentage  of  A L L instructors  What percentage  who  present  who  have  of  A L L instructors  who  8.  What percentage  of A L L instructors who  9.  of  ARE 10.  A L L instructors  good instructors?  outgoing  personalities A R E  are  resourceful  ARE  good  actively  participate  in social work  %  who  show  leadership  ARE  good  % of A L L instructors who do a lot of professional  good instructors?  What percentage  clarity  of A L L instructors who grade papers very well A R E good  What  What percentage  material with  plan and prepare thoroughly A R E  of A L L instructors who  %  instructors?  the  %  instructors? percentage  good  %  or community service A R E good instructors? 7.  ARE  %  good instructors? 6.  enthusiasm  %  good instructors? 4.  show  %  A R E good instructors? 3.  who  travelling  %  of A L L instructors  who  are  effective  %  Please do not revise your estimates. Go onto the next page.  communicators A R E  APPENDIX /  146  Fourth set - just O N E question  1.  There  What percentage of A L L instructors are G O O D instructors?  are  estimates.  no  more  percentage  questions.  Please  DO  NOT  %  REVISE  your  APPENDIX /  147  F. P E R F O R M A N C E RATING T A S K B  Read  the  vignette  of  an  instructor  presented  below.  description, rate the instructor on the dimensions following the  After  reading  the  vignette.  Dr. T comes to class on time and is always very well prepared. Dr. T tries to present the subject matter clearly, but the students are often left confused. As a result, many students do not turn up for Dr. T's classes regularly. However, they all enjoy Dr. T's company and speeches at functions, parties, and other gatherings. Dr. T can accept criticism from students and also from colleagues. When asked, Dr. T takes up responsibilities on committees, and often does very well. Dr. T drives a Mustang, plays tennis, loves music, and seems to be a happy person most of the time.  Circle a number on the scales provided to the right of the these scales 1 =  poor, 4 =  average, and 7 =  dimensions. On  outstanding.  Planning, preparation  1  2  3  4  5  6  7  Enthusiasm  1  2  3  4  5  6  7  Lecture presentation  1  2  3  4  5  6  7  Sociability  1  2  3  4  5  6  7  Resourcefulness  1  C  Si  3  4  5  6  7  Grading, marking  1  2  3  4  5  6  7  Leadership  1  2  3  4  5  6  7  Communication  1  2  3  4  5  6  7  Dependability  1  2  3  4  5  6  7  Research activity  1  2  3  4  5  6  7  


Citation Scheme:


Usage Statistics

Country Views Downloads
United States 10 1
China 8 45
Germany 3 3
United Kingdom 1 0
City Views Downloads
Guangzhou 4 0
Mountain View 4 1
Ashburn 4 0
Shenzhen 3 45
Unknown 2 3
Bayreuth 1 0
Redmond 1 0
London 1 0
Houston 1 0
Beijing 1 0

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}
Download Stats



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items