THE DEVELOPMENT AND EVALUATION OF (A NEW PERFORMANCE APPRAISAL AND TRAINING EVALUATION INSTRUMENT: THE BEHAVIOUR DESCRIPTION INDEX BY HERMANN F. SCHWIND Mechanical Engineer, E n g i n e e r i n g C o l l e g e Mannheim (Germany) 196 0 I n d u s t r i a l Engineer, REFA I n s t i t u t e , Worms (Germany) 1965 B.B.A., U n i v e r s i t y o f Washington, 1970 M.B.A., U n i v e r s i t y of Washington, 19 7 2 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS OF THE DEGREE OF DOCTOR OF PHILOSOPHY i n THE FACULTY OF GRADUATE -STUDIES The F a c u l t y of Commerce and Business A d m i n i s t r a t i o n We accept t h i s t h e s i s as conforming to the r e q u i r e d standard THE UNIVERSITY OF BRITISH COLUMBIA September, 1978 (3) Hermann F. Schwind, 19 7 8 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the Head of my Department or by his representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. HERMANN F. SCHWIND ASSISTANT PROFESSOR . r Industrial Relations Management Department of The University of British Columbia 2075 Wesbrook Place Vancouver, Canada V6T 1WS Sept. 29, 1978 i i ABSTRACT T h i s s t u d y had as a major o b j e c t i v e t h e development and v a l i d a t i o n of a new b e h a v i o u r - o r i e n t e d a p p r a i s a l i n s t r u m e n t w i t h i n c r e a s e d i n f o r m a t i o n con-t e n t and improved r a t i n g c h a r a c t e r i s t i c s as compared t o B e h a v i o r a l l y Anchored R a t i n g S c a l e s . The s t u d y was co n d u c t e d i n a s e q u e n t i a l manner, b e g i n n i n g w i t h a r e v i e w of the l i t e r a t u r e on the problems o f performance a p p r a i s a l and t r a i n i n g e v a l u a t i o n and the n a t u r e and c h a r a c t e r i s t i c s of d i f f e r e n t performance e v a l u -a t i o n methods, f o l l o w e d by s e v e r a l d e v e l o p m e n t a l phases r e l a t i n g t o t h e d e v e l -opment o f c r i t i c a l i n c i d e n t s and the c r e a t i o n o f the i n s t r u m e n t s . The r e s e a r c h c o n c l u d e d w i t h f i e l d s t u d i e s f o r the purpose o f e s t a b l i s h i n g t he c o n s t r u c t v a l i d i t y of t h e i n s t r u m e n t s and d e t e r m i n i n g t h e i r r a t i n g c h a r a c t e r i s t i c s when a p p l i e d i n an o r g a n i z a t i o n a l e n v i r o n m e n t . The l i t e r a t u r e r e v i e w s r e v e a l e d two major i s s u e s i n the a r e a s o f p e r -formance a p p r a i s a l and t r a i n i n g e v a l u a t i o n : c r i t e r i o n and methodology. The methodology problem i s open t o much c o n t r o v e r s y , but on the c r i t e r i o n i s s u e the tendency i s t o f a v o u r m u l t i p l e and b e h a v i o u r - o r i e n t e d c r i t e r i a . A f t e r d e t e r m i n i n g a f o c a l j o b f o r whi c h an a p p r a i s a l i n s t r u m e n t had t o be d e v e l o p e d , workshops w i t h f i v e s u p p o r t i n g o r g a n i z a t i o n s were o r g a n i z e d . Each workshop was a t t e n d e d by f i v e j o b incumbents, f i v e s u p e r i o r s and f i v e s u b o r d i n a t e s . The purpose o f t h e s e meetings was t o d e v e l o p c r i t i c a l i n c i -d e n t s , o r samples o f e f f e c t i v e o r i n e f f e c t i v e j o b b e h a v i o u r as o b s e r v e d by p e e r s , s u p e r i o r s and s u b o r d i n a t e s , thus i n c l u d i n g e v e r y a s p e c t o f the j o b . The c o l l e c t e d items were e d i t e d t o conform t o p r o p e r E n g l i s h and t o a v o i d r e d u n d a n c i e s . The items were then l i s t e d i n random o r d e r and s u b m i t t e d t o j u d g e s ( e x p e r t j o b incumbents) who made d e c i s i o n s on the v a l i d i t y of the items and the j o b d i m e n s i o n o r c a t e g o r y t o wh i c h each item b e l o n g e d . Only items on wh i c h 80% o f the j u d g e s agreed were r e t a i n e d . i i i S i n c e the r e m a i n i n g item pool was s t i l l t o o l a r g e t o be s u b m i t t e d t o an i n t e r - o r g a n i z a t i o n a l body o f j u d g e s , a panel o f e x p e r t s made up o f t r a i n -i n g managers o f the p a r t i c i p a t i n g o r g a n i z a t i o n s s e l e c t e d 159 items a c c o r d i n g to agreed-upon c r i t e r i a ( l o w e s t s t a n d a r d d e v i a t i o n and 100% agreement among j u d g e s ) t o e n s u r e t h a t o n l y the b e s t items were chosen. A l i s t o f t h e s e items was s u b m i t t e d t o 200 j u d g e s o f each p a r t i c i p a t i n g o r g a n i z a t i o n . The ju d g e s were asked t o d e c i d e whether each item was a v a l i d sample o f a j o b incumbent's j o b b e h a v i o u r , and t o r a t e i t on a 1 t o 7 s c a l e as t o t h e degree o f e f f e c t i v e n e s s o f t h e j o b b e h a v i o u r i t d e s c r i b e d . items were r e t a i n e d when 80% o f a l l j u d g e s and 60% o f the j u d g e s o f i n d i v i d u a l o r g a n i z a t i o n s a g reed on t h e i r v a l i d i t y and t h e i r s t a n d a r d d e v i a t i o n d i d not exceed 1.5. One hundred and twenty items (or 75%) were r e t a i n e d . Two type s o f i n s t r u m e n t s were d e v e l o p e d from the item p o o l , a seven^ p o i n t b e h a v i o u r a 1 1 y anchored r a t i n g s c a l e (BARS), and a t w e n t y - i t e m b e h a v i o u r d e s c r i p t i o n index w i t h Y e s / N o / U n c e r t a i n r e s p o n s e s (BDI). A t h i r d i n s t r u m e n t , a s e v e n - p o i n t g r a p h i c r a t i n g s c a l e (GRS) was based on the o f f i c i a l p e r f o r -mance a p p r a i s a l forms o f t h e p a r t i c i p a t i n g o r g a n i z a t i o n s . To a s s e s s the v a l i d i t y o f the r e s p e c t i v e i n s t r u m e n t s , two f i e l d s t u d i e s were u n d e r t a k e n . Two groups o f s u p e r i o r s (_M^ = 3 1 , = ^2) r a t e d t h e i r s u b o r d i n a t e s u s i n g a l l t h r e e i n s t r u m e n t s . The c o n s t r u c t v a l i d i t y o f the i n -s t r u m e n t s was a s s e s s e d t h r o u g h the Campbel 1-Fiske [1959] m u 1 1 i t r a I t - m u l 1 1 -method m a t r i x , an a n a l y s i s o f v a r i a n c e , and c o r r e l a t i o n s w i t h ( r e l a t i v e l y ) independent performance c r i t e r i a . A l l i n s t r u m e n t s showed s i g n i f i c a n t con-v e r g e n t v a l i d i t y but o n l y BARS and BDI demonstrated s i g n i f i c a n t d i s c r i m i n a n t v a l i d i t y . C o r r e l a t i o n s w i t h the independent c r i t e r i a o f performance were h i g h e s t f o r the BDI. A second g o a l o f the f i e l d s t u d i e s was t o compare the t h r e e i n s t r u m e n t s on p s y c h o m e t r i c c h a r a c t e r i s t i c s such as h a l o , l e n i e n c y , c e n t r a l t e n d e n c y , i V r e l i a b i l i t y , i n f o r m a t i o n c o n t e n t , a n d r a t e r p r e f e r e n c e . The r e s u l t s i n d i -c a t e t h a t i n m o s t c o m p a r i s i o n s t h e BDI d e m o n s t r a t e d s u p e r i o r i t y . H o w e v e r , i t h a s n o t b e e n v a l i d a t e d a s y e t f o r u s e i n i n d u s t r y . TABLES OF CONTENTS Page Tables of Contents iv L i s t of Tables vi L i s t of Figures v i i i L i s t of Appendices ix Acknowledgements x Chapter I: Introduction 1 Chapter I I : Performance Appraisal and Training Evaluation: The C r i t e r i o n Issue 10 Purpose of a Performance Appraisal 10 Formal Performance Appraisal Systems 11 Selec t i o n of C r i t e r i a 14 Chapter I I I : A L i t e r a t u r e Review on Behaviorally Anchored Rating Scales 34 Behaviorally Anchored Rating Scales 34 Psychometric Properties 35 Optimal Development Procedures and Formats 59 Ef f e c t s of Various Factors on Rating C h a r a c t e r i s t i c s 65 Optimal Number of Job Dimensions 69 Modified BARS Instruments 71 Discussion 72 A Summary of L i t e r a t u r e Reviews on Training Evaluation 78 Need f or Improvement of BARS 89 Proposal for a New Performance Appraisal and Training Evaluation Instrument 98 Hypotheses to be Investigated 112 V I TABLES OF CONTENTS (continued) Page Chapter IV: Method 118 Sponsors of the Project and Research S i t e 118 Target Group 121 Development of BARS and BDI 124 Development of a Third Instrument, a Graphic Rating Scale 134 Ratee and Rater Sample 135 Analyses 138 Psychometric C h a r a c t e r i s t i c s 152 Chapter V: Results 157 Discussion 187 Chapter VI: Summary 193 Overview 193 Possible Problems With and Limitations for the BDI 199 Recommendations f or Future Research 202 References 208 Appendices (See L i s t of Appendices) LIST OF TABLES Table No T i t l e Page II.1 Purposes of Performance Appraisals 12 111.1 Comparative Studies and Non-Comparative Studies With Data (See Appendix A) 111.2 Comparative Studies on Leniency E f f e c t 42 111.3 Comparative Studies on Dimension I n t e r c o r r e l a t i o n 47 111.4 Comparative Studies on Int e r r a t e r R e l i a b i l i t y 52 111.5 Job Dimensions of Department Managers 57 111.6 Behaviorally Based Rating Scales 58 111.7 Problems With BARS 97 IV.1 Assets and Branches of the Five Canadian Chartered Banks Which Supported t h i s Study 155 V . l Means and Standard Deviations for Each Performance Dimension on GRS, BARS, and BDI Responses of Group 1 157 V.2 Means and Standard Deviations for Each Performance Dimension on GRS, BARS, and BDI Responses of Group 2 158 V.3 Normalized Skewness C o e f f i c i e n t for Each Perfor-mance Dimension on GRS, BARS, and BDI Responses of Group 1 159 V.4 Normalized Skewness C o e f f i c i e n t f o r Each Perfor-mance Dimension on GRS, BARS, and BDI Responses of Group 2 "" 160 V.5 Co r r e l a t i o n Between Dimension Scores of Each of the Three Instruments f o r Group 1 162 V.6 Correlation Between Dimension Scores of Each of the Three Instruments for Group 2 163 V. 7 Score D i s t r i b u t i o n f o r Five Dimensions of Each GRS, BARS, and BDI Instrument for Group 1 165 V.8 Score D i s t r i b u t i o n for Five Dimensions of Each GRS, BARS, and BDI Instrument f o r Group 2 166 V.9 Information Content f o r BARS and BDI as Measured by Percentage of Act u a l l y U t i l i z e d C r i t i c a l I n c i -dents From Total Available Incident Pool 168 V I I i LIST OF TABLES (continued) Table:.No • Title Page V.10 Test-Retest Re l i a b i l i t i e s for GRS, BARS, and BDI 169 /o.'.V.ll Multitrait (Performance Dimensions), Multimethod (GRS vs BARS vs BDI) Matrix for Bank Accountants, 171 Group 1 V.12 Multitrait (Performance Dimensions), Multimethod (GRS vs BARS vs BDI) Matrix for Bank Accountants, Group 2 172 V.13 Analysis of Variance of Correlations from Table V . l l 176 V.14 Analysis of Variance of Correlations from Table V.12 177 V.15 Correlation Between Dimension Scores of GRS, BARS, BDI, and the Index of Actual Promotion 180 V.16 Correlation Between Dimension Scores of GRS, BARS, BDI, and the Indexes of Overall Performance and Promotabilty 181 V.17 Percentage Distribution of Rankings (1st, 2nd, and 3rd) of GRS, BARS, and BDI by Expert Judges, and Kendall's Coefficient of Concordance (W) for a l l Ratings 184 LIST OF FIGURES Figure No T i t l e I I I I I Model of Managerial Effectiveness LIST OF APPENDICES x Content Page Appendix A Table I I I . l , Comparative Studies and Non-Comparative Studies With Data 219 Appendix B Studies Which Used BARS for Purposes Other Than Performance Appraisals 229 Appendix C Descriptive Publications Using BARS 231 Appendix D The Behavior Description Index (BDI) 233 Appendix E Job Description Index (JDI) 239 Appendix F ' Samples of Job Descriptions of Assistant c Branch Managers of Canadian Chartered Banks 244 Appendix G Cover L e t t e r 260 Appendix H Item V a l i d a t i o n and Rating L i s t 263 Appendix I Behaviorally Anchored Rating Scale (BARS) 274 Appendix K Graphic Rating Scale (GRS) 280 Appendix L Rating Instructions 282 Appendix M A l t e r n a t i v e Rating Format for the BDI 285 Appendix N Performance C r i t e r i a 291 Appendix 0 Rater Preference Questionnaire 296 x i: -ACKNOWLEDGEMENTS The author i s indebted to Professor Larry F. Moore for his invaluable assistance, advice, and support over the period of thi s study. The Ph.D. committee was very supportive and h e l p f u l . Special thanks should go to Professors Merle Ace, Peter Frost, Vance M i t c h e l l , and Ron Taylor. Gratitude i s also expressed to the Instit u t e of Canadian Bankers, represented by i t s Deputy Director for Research and Development, Peter H. Wigram, and to the Training and Personnel O f f i c e r s of the Canadian Imperial Bank of Commerce, the Bank of Montreal, the Bank of Nova Scotia, The Royal Bank of Canada, and the Toronto Dominion Bank, for th e i r generous support of t h i s study. My thanks also go to Miss Lorraine Ripley and Mrs. Marilyn Logan for typing the proposal and dr a f t s . The Killam Predoctoral Fellowship and the f i n a n c i a l support of the In s t i t u t e of I n d u s t r i a l Relations helped to make thi s study possible. 1 CHAPTER I INTRODUCTION The need for improvement i n the quality and nature of the instruments and methods used i n measuring managerial performance and effectiveness i n the area of t r a i n i n g evaluation has long been recognized and discussed [Campbell, Dunnette, Lawler, and Weick, 1970; Cummings and Schwab, 1973; Drucker, 1954; Levinson, 1976; McGregor, 1957]. The c r i t i q u e on the q u a l i t y and methodology of measurement for the l a t t e r purpose i s e s p e c i a l l y strong [Campbell, 1971; McGehee and Thayer, 1961, p. 256]. The most outspoken c r i t i c on the q u a l i t y and nature of measurements of managerial performance probably was Douglas McGregor [1957]. He soundly condemned the conventional approach with i t s emphasis on the evaluation of personality t r a i t s and "negative appraisals" [p. 94]. In his opinion most raters are i l l equipped for making such judgements, and, because of that, f e e l uneasy when requested to do so. In his words: The respect we hold for the inherent value of the i n d i v i d u a l leaves us distressed when we must take r e s p o n s i b i l i t y for judging the personal worth of a fellow man. Yet the conventional approach to performance appraisal forces us, not only to make such judgments and to see them acted upon, but also to communicate them to those we have judged. Small wonder we r e s i s t . I P - 9 0 ] 2 As a solution to the problem McGregor suggested to s h i f t the emphasis from appraisal to analysis, from a subor-dinate's passive role as a receiver of orders to his active role as a goal setter and a p a r t i c i p a n t i n the decision making process of the organization. F i n a l l y , and perhaps most important i n connection with t h i s study, he said: "...the accent i s on performance, on actions r e l a t i v e to goals", and because of that, "There i s less tendency for the personality of the subordinate to become an issue." [p. 92] With other words, McGregor suggested that superiors should focus on observable behaviour instead of on personality t r a i t s which cannot be observed but have to be infer r e d . Peter. Drucker i s less known for h i s research i n the f i e l d of management, but i s famous for his a b i l i t y to describe and analyse management problems. When he talked about the measurement of managerial performance he put i t squarely: Appraisals must be based on performance. Appraisal i s judgment; and judgment always requires a d e f i n i t e standard. To judge means to apply a set of values; and value judgments without clear, sharp and public standards are i r r a t i o n a l and a r b i t r a r y . They corrupt a l i k e the judge and the judged. No matter how " s c i e n t i f i c " , no matter even how many insights i t produces, an appraisal that focuses on "potential", on "personality", on "promise", — on anything that i s not proven and provable performance — i s an abuse. [1954, p. 150] 3 Here again we have the emphasis on performance and a de-emphasis of "personality" and "potential." Peter Drucker then goes on to develop the idea of "Management by Objectives", where superior and subordinate discuss organizational and personal goals and i d e a l l y come to an agreement on mutually acceptable goals. In Peter Drucker's terms "Management by Objectives" makes any contributions by an employee "tangible", i . e . easier to assess. S t i l l , the question remains: How do we measure these "tangible" contributions? Campbell et a l . [1970] are quite e x p l i c i t on the requirements of an e f f e c t i v e performance appraisal instrument, s p e c i a l l y for managerial positions: What i s needed i s some way of systematically recording observations across the whole domain of desired managerial job behaviours and tabulating how often they accomplish agreed-upon objectives i n the most c r i t i c a l areas of managerial behaviour. [p. I l l ] Besides that they suggest: "In addition, we s h a l l want observations wherever possible from more than a single vantage point." [p. 115] Not only the superior should get involved i n the evaluation of performance, but peers and even subordinates should p a r t i c i p a t e i n order to get d i f f e r e n t views and to cover every angle of the job. According to the above-offered recipe, i t should not be too d i f f i c u l t to develop an appropriate instrument for performance appraisal purposes. Campbell et a l . , even propose an instrument they suggest would come close to 4 the requirements mentioned: a combination of Flanagan's [1954] C r i t i c a l Incident Method with Smith and Kendall's retr a n s l a t i o n approach [Smith and Kendall, 196 3], also known as Behaviorally Anchored Rating Scale (BARS), or as Behavior Expectation Scale (BES). Flanagan defined the c r i t i c a l requirements of a job as those behaviours which are c r u c i a l i n making a difference between doing the job e f f e c t i v e l y and doing i t i n e f f e c t i v e l y . C r i t i c a l incidents, as the term implies, are simply reports by q u a l i f i e d observers of job related a c t i v i t i e s that were e s p e c i a l l y e f f e c t i v e or ineffective, i n accomplishing parts of t h e i r jobs. Such job behaviour samples are actual behaviour accounts, c o l l e c t e d and recorded by superiors, job incumbents and subordinates. The c h a r a c t e r i s t i c s attributed to the BARS instrument by Flanagan, Campbell, et a l . , and Smith and Kendall made i t a very promising tool for performance appraisal purposes. The question, then, i s whether the BARS instrument l i v e d up to these expectations. Similar problems as i n the f i e l d of performance appraisal have been encountered i n the related area of tr a i n i n g evaluation. I t i s related since both—performance appraisal and t r a i n i n g evaluation—assess employees' performances, only for d i f f e r e n t purposes, as w i l l be shown. (See also Campbell et a l . , 1970, and McGehee and Thayer, 1961) . In the Annual Review of Psychology, Campbell [1971] wrote a scathing review of the l i t e r a t u r e on t h i s topic. 5 He pointed out that l i t t l e progress had been made i n the area of experimental design since Campbell and Stanley. [1963] published t h e i r booklet on experimental and quasi-experimental design. In general, for the evaluation of t r a i n i n g , the methodology issue appears to be largely unresolved. Campbell and Dunnette [1968] and MacKinney [1957] strongly advocate the s c i e n t i f i c approach, i . e . the u t i l i z a t i o n of before and after measures and experimental and control groups. Andrews [1966] and Argyris [1968] , on the other hand, argue i n favour of somewhat relaxed standards, depending on the purpose of the measurement. Each side uses strong arguments to defend i t s views. For reasons explained elsewhere [Schwind, 19 75a & b], t h i s writer supports Andrew's and Argyris' view. On the c r i t e r i o n issue we f i n d a si m i l a r controversy. The recommendations made by d i f f e r e n t authors span the range from the use of r e l a t i v e l y unsophisticated reaction measure-ments [Andrews, 1966], to knowledge increase [Mack, 1974], attitude change [Kohn and Parker, 1969; A b a t i e l l o , 1967], behaviour change [Campbell et a l . , 1970], organizational results [Kirkpatrick, 1959a & b, 1960a & b], to r e l a t i v e l y complex cost/benefit analyses [Odiorne, 1964], Many authors argue against the use of a single c r i t e r i o n i n t r a i n i n g evaluation [Campbell, 1971; Campbell 6 et a l . , 1970; Hamblin, 1974; Hesseling, 1966]. As Campbell et a l . , put i t : "The e f f e c t s of t r a i n i n g must be evaluated against a number of c r i t e r i a which r e f l e c t more than one aspect of the organization's goals" [p. 282]. However, the combination of d i f f e r e n t c r i t e r i a seems to have i t s problems too. According to Kirkpatrick [1959a & b, 1960a & b] the most desirable c r i t e r i a for assessing t r a i n i n g programme effectiveness would be organizational r e s u l t s , but. d i f f i c u l t i e s i n determining cause and e f f e c t relationships between tr a i n i n g programme outcomes and d i f f e r e n t organizational outcomes render t h i s appraisal impractical. Instead, there seems to be a strong trend i n the f i e l d of t r a i n i n g evaluation to focus on behavioural change as the most relevant c r i t e r i o n [Campbell, 1971; Campbell et a l . , 1970; Ferguson, 1968; Goldstein, 1974]. This does not imply that the recommendation to use multiple c r i t e r i a i s neglected. Some of the above-mentioned authors suggest to use a combi-nation of c r i t e r i a to cover every aspect of t r a i n i n g , or, i n other words, to use a systems approach to t r a i n i n g evaluation [Campbell et a l . , 1970, p. 285; Goldstein, 1974, pp. 17-18]. But the emphasis seems to be on behaviour oriented c r i t e r i a , and the reason i s given by Campbell ," [1976]: C r i t e r i o n measures should be a r e f l e c t i o n of what the i n d i v i d u a l a c t u a l l y does; that i s , they should represent an assessment of accomplishments that are d i r e c t l y under the individual's control. [p. 35] 7 And, elsewhere [1971] he said: What i s needed are c a r e f u l l y developed obser-vational and measurement techniques for sc a l i n g actual behavior changes. ...the method of scaled expectations [Smith and Kendall, 1963] . i s suggested as a d i f f i c u l t but very useful strategy. [p. 577, emphasis added]' Here again, as i n the discussion on performance appraisals, we encounter the recommendation to use behaviourally oriented scales, or, even more s p e c i f i c , BARS, as an appropriate tool to measure outcomes of t r a i n i n g and development programmes. Of course, i t would be comforting to know whether behaviourally oriented scales r e a l l y are the answer to the problems we encounter i n the areas of performance and tra i n i n g evaluation. This study intends, f i r s t l y , to take a closer look at the d i f f e r e n t views and opinions on the desirable c h a r a c t e r i s t i c s of a performance appraisal and tr a i n i n g evaluation instrument; secondly, to look at the resu l t s of relevant research studies to narrow down the scope of this study to the development of an instrument with improved c h a r a c t e r i s t i c s as compared to already available instruments, e.g. BARS; and, t h i r d l y , to test the new instrument i n an organizational environment. More s p e c i f i c a l l y , Chapter II w i l l review the l i t e r a t u r e relevant to c r i t e r i a development, i . e . the discussion on the issue of single vs. multiple, and t r a i t vs. behaviour oriented c r i t e r i a . I t w i l l try to answer the 8 question which combination may be best suited for either performance appraisal or t r a i n i n g evaluation purposes. It w i l l also look at the p o s s i b i l i t y that d i f f e r e n t combinations may be preferable for d i f f e r e n t purposes. Chapter III w i l l review the l i t e r a t u r e on behaviour oriented r a t i n g scales, and w i l l focus on t h e i r r a t i n g c h a r a c t e r i s t i c s as compared to other scales. I t w i l l also look at d i f f e r e n t procedures for the development of the scales and the influence of various factors on the application of the scales. I t w i l l include a summary of the l a t e s t l i t e r a t u r e reviews on t r a i n i n g evaluation, with focus on problems i n methodology and instrumentation. In addition, the needs for improvement of behaviour oriented scales w i l l be discussed. F i n a l l y , the objectives of the study w i l l be outlined, with emphasis on the development of a new approach to performance evaluation and the assessment of t r a i n i n g and development programmes. Chapter IV i s concerned with the methodology of the study. I t also outlines the sponsors, the target group, d i f f e r e n t research s i t e s , and the development of the d i f f e r e n t instruments u t i l i z e d i n the study for comparison purposes. A description of the suggested analyses concludes t h i s chapter. 9 The re s u l t s of the study are reported and discussed i n Chapter V. Special emphasis i s given to the analysis of the rating c h a r a c t e r i s t i c s of the d i f f e r e n t instruments, such as halo, leniency, central tendency, r e l i a b i l i t y , and convergent and discriminant v a l i d i t y . Some of the important findings are compared to and contrasted with* other relevant studies. Chapter VI presents the summary and conclusions of the study. I t discusses the li m i t a t i o n s of the study and points to the po t e n t i a l a p p l i cation of the new instrument i n d i f f e r e n t areas of personnel management, and discusses the possible implications for each. The susggestions for relevant future research conclude t h i s chapter. 19 CHAPTER II PERFORMANCE APPRAISAL AND TRAINING EVALUATION: THE CRITERION ISSUE Purposes of a Performance Appraisal There are at lea s t three p r i n c i p a l reasons for the use of a formal performance appraisal programme [Glueck, 1974, Chapter 9]: 1. To provide feedback to employees. Properly designed and implemented, a performance appraisal system provides the occasion for reviewing a person's work-related behaviour. Es p e c i a l l y for self-improvement, i t i s necessary that each employee knows where he stands with his superior. Feedback implies a discussion of past d e f i c i e n c i e s which may lead to improved organizational performance while the discussion of strengths reinforces desirable job behaviour. 2. To provide better data for promotion decisions. This i s perhaps the most important use of a performance evaluation. I t i s i n the common i n t e r e s t of both management and employees to promote employees into positions where they can most e f f e c t i v e l y u t i l i z e t h e i r a b i l i t i e s . There i s , however, a d i s t i n c t i o n to be made between a performance evaluation for the purpose of feedback and for the purpose of promotion. We deal with past performance for the purpose of the f i r s t , and with p o t e n t i a l performance for the purpose of the second. 3. To provide data for wage and salary administration. Ideally, management seeks to t i e the administration of organizational rewards to performance. In the absence of d i r e c t measures of performance (piece sales, etc.) performance appraisal i s a useful source of information on an employee's worth to the organization. Formal Performance Appraisal Systems If we accept the above statements on the purpose of performance appraisals as univ e r s a l l y relevant for personnel p o l i c y implementation i n organizations, one could reasonably expect widespread u t i l i z a t i o n of w e l l -developed formal appraisal,systems. The f i r s t assumption seems to be correct, the second less well-founded. According to a survey of the Bureau of National A f f a i r s , 93 percent of the firms surveyed have appraisal programmes [Colby and Wallace, 1975]. Yet, only 10 percent of the personnel managers of those firms f e l t that t h e i r appraisal programmes were e f f e c t i v e . In an e a r l i e r study, Spriegal [1962] surveyed 567 firms and found that less than 46 percent of the responding firms had appraisal programmes. An equal percentage of the respondents reported that they once had, but discontinued, appraisal programmes, at le a s t for executive-level jobs. Within those firms reporting i t s use, performance evaluation was most frequently used as an aid to performance counselling. Other applications mentioned were promotion, wages, t r a i n i n g needs, improving morale, and discovery of supervisory p o t e n t i a l . Surveys t y p i c a l l y indicate more use of formal performance appraisal procedures f o r nonexempt white-c o l l a r employees, supervisors, and middle-level managers than for bl u e - c o l l a r workers. For example, a National Industrial Conference Board study found 63 percent of manufacturing firms surveyed had performance appraisal programmes for white-collar employees, while only 4 3 percent of firms i n the same sample had performance appraisal plans for b l u e - c o l l a r workers. The same study l i s t s the purpose of performance appraisal as follows [NICB, 1964, p. 17]: Table I I . l PURPOSES NO. OF COMPANIES % Wage or salary determination Promotion Training and development To help supervisors know t h e i r employees To l e t workers know t h e i r progress Transfer Follow-up interviews Discharge Layoff Personnel research 114 122 102 101 102 92 57 77 44 48 69 73 61 61 61 59 34 46 27 29 From these studies, i t can be concluded that organizations use performance appraisal primarily for two reasons: to provide management with a t o o l for c o n t r o l l i n g and d i r e c t i n g i t s employees' a c t i v i t i e s (counselling and training) and to provide a r e l a t i v e l y objective ( i . e . standardized) basis for tying the administration of organizational rewards to performance, [Cummings and Schwab, 1973, p. 55]. The preceding discussion indicates the usefulness of a performance appraisal system for the implementation of personnel p o l i c i e s . To .paraphrase Beach [1969]: the organizational choice i s not whether to appraise employee performance, but how. This leads to an important decision to be made by the organization; namely: which aspects of an employee's performance are to be evaluated. Does, for example, appearance play a major role i n his performance? Are t r a i t s enough? Gr should job related behaviour be included? This points to the next topic to be discussed: the development of relevant c r i t e r i a . Selection of C r i t e r i a An important part of the performance appraisal process i s the development of v a l i d and relevant performance standards or c r i t e r i a [Megginson, 1977, p. 198]. Blum and Naylor [1968] define a c r i t e r i o n simply as "a measure of the goodness of a worker" [p. 174 Horst [1936] i s more s p e c i f i c when he defines i t as "The measure of success or f a i l u r e i n an a c t i v i t y i s what i s te c h n i c a l l y known as a c r i t e r i o n " [p. 20]. Bechtold [1947] puts more emphasis on "success" by defining a c r i t e r i o n as "...a means of describing the performance of individuals on a success continuum", [p. 357]. Brogden and Taylor [1950] r e f l e c t a si m i l a r view when they define a c r i t e r i o n as a measure of , "...the contribution of the i n d i v i d u a l to the o v e r a l l e f f i c i e n c y of the organization" [p. 139]. Perhaps one of the most thorough discussions of the concept of a c r i t e r i o n i s offered by Nagle! [1953]. He perceives a c r i t e r i o n as a measure of the degree of success by an i n d i v i d u a l on a given a c t i v i t y . I t follows, then, that a c r i t e r i o n i s an operational d e f i n i t i o n of the extent to which an employee exhibits desirable job behaviours. This d e f i n i t i o n seems to be similar to what Campbell, et a l . [197 3] c a l l e d performance. Nagle [195 3] provides a useful framework for the c r i t e r i o n development process, basing i t on three important considerations: 1. the relevancy of the c r i t e r i o n ; 2. the r e l i a b i l i t y of the c r i t e r i o n ; 3. the combination of c r i t e r i o n measures into a composite score. [p. 2731 He points out that the emphasis on these three problems should not be interpreted as to mean that these are the only problems of c r i t e r i o n construction and selection [p. 273]. Nagle treats each of the three problems mentioned above as a separate aspect of c r i t e r i o n development. Cummings and Schwab [1973], however, consider the problem of relevancy (construct v a l i d i t y ) as the primary concern of the c r i t e r i o n development process [p. 80]. The following discussion i s based on t h i s orientation and, i n addition, comments on the topic of c r i t e r i o n content ( i . e . personal t r a i t s as opposed to job behaviours). Relevancy (Construct V a l i d i t y ) Relevancy i s the term used to describe the r e s u l t of associating a c r i t e r i o n measure (an operational de f i n i t i o n ) with an unmeasurable ultimate c r i t e r i o n [Nagle, 1953; Thorndike, 1949]. More recently, the concept has been refined and relabeled "construct v a l i d i t y " [Cronbach and Meehl, 1955; Fiske, 1971].. Despite e f f o r t s to quantify the concept [Campbell and Fiske, 1959], the process of establishing construct v a l i d i t y remains e s s e n t i a l l y judgmental [Cummings and Schwab, 1973, p. 73]. Although a researcher may gather considerable converging quantitative evidence for the existence of construct v a l i d i t y , there are no s t a t i s t i c a l tests, per se, that would allow precise p r o b a b i l i t y statements about i t s existence. Before any claim of construct v a l i d i t y for a c r i t e r i o n measure can be allowed, the c r i t e r i o n must f i r s t be judged as c r u c i a l to the individual's performance on the job or other a c t i v i t y of i n t e r e s t [Fiske, 1971; Guion, 1965]. The c r u c i a l aspect of a construct can be i d e n t i f i e d by assessing the extent to which a c r i t e r i o n measure i s free of c r i t e r i o n contamination or deficiency [Cummings and Schwab, 1973, p. 72] . C r i t e r i o n contamination i s that variance i n the actual c r i t e r i o n which i s unrelated to the ultimate c r i t e r i o n . In contrast, c r i t e r i o n deficiency i s the degree to which the c r i t e r i o n i s lacking relevant aspects necessary to the ultimate c r i t e r i o n . C r i t e r i o n deficiency and c r i t e r i o n contamination are undesirable c h a r a c t e r i s t i c s i n a c r i t e r i o n . Construct v a l i d a t i o n i s the process of establishing the extent to which a measure of the construct i s free of both. R e l i a b i l i t y As mentioned before, i t i s very d i f f i c u l t to make any precise p r o b a b i l i s t i c statements about c r i t e r i o n relevancy, because i t i s not possible to compare the c r i t e r i o n with a more representative measure of a construct [Nagle, 1953]. (It would make l i t t l e sense to develop a less representative measure i f a better one were available.) There i s no such problem when i t comes to the assessment of the r e l i a b i l i t y of c r i t e r i o n scores [Guilford, 1954, p. 350]. I t may have been t h i s f a c t which led some investigators to overemphasize the importance of c r i t e r i o n r e l i a b i l i t y [Nagle, 1953]. While overemphasis should be avoided, r e l i a b i l i t y i s an important aspect of the c r i t e r i o n development process [A.P.A. Standards, 1974, E 4.4, p. 35]. For the purpose of discussion, the d i f f e r e n t sources of variance i n performance appraisal scores can be c l a s s i f i e d into two general types: variable and constant error [Cummings and Schwab, 1973]. Variable Errors. Variable or random errors occur when raters are inconsistent across ratings [Guion, 1965, Chapter 4]. There are two possible reasons f o r these inconsistencies. One has to do with the degree of agreement among two or more raters about an appraisee's performance on a s p e c i f i c measure. The other deals with the degree of agreement between two judgements, using the same instrument at two points i n time. These two sources of possible errors are estimated through i n t e r r a t e r r e l i a b i l i t y and t e s t - r e t e s t r e l i a b i l i t y . Interrater r e l i a b i l i t y r e f l e c t s the agreement between raters across one or more ratees or dimensions to be rated [Cummings and Schwab", . 1973] . Commonly, such an agreement i s represented by a measure of association such as a c o r r e l a t i o n c o e f f i c i e n t among the ratings assigned to a ratee or dimension by two 19 or more raters. The degree of agreement among raters may be influenced by several factors, such as job pos i t i o n r e l a t i v e to ratee, e.g. superior, peer, or subordinate [Borman, 1974]; chain of command, e.g. d i r e c t superior or second l e v e l superior [Campbell, Dunnette, Arvey, and Hellervik, 1973]; or sex [Norton, Gustafson, and Foster, 1977] . Other factors pertain to the evaluation i n s t r u -ment. If items are too vague or scales too ambiguous, low i n t e r r a t e r agreement may be the r e s u l t . [Cummings and Schwab, 1973] Interrater r e l i a b i l i t y i s a r e l a t i v e l y desirable measurement characteristicvbecause i t o f f e r s a good i n d i -cation of the v a l i d i t y of an appraisal. If there i s low agreement among the ratings by d i f f e r e n t r a t e r s , then at le a s t one of the ratings must be incorrect [Cummings and Schwab, 1973]. Unfortunately, based on the r e l i a b i l i t y c o e f f i c i e n t , i t i s not possible to determine which of.the raters i s wrong. However, i f the ratings are plotted on a scatter diagram i t may be possible to i d e n t i f y raters who deviate s i g n i f i c a n t l y from other raters. Test-retest r e l i a b i l i t y , on the other hand, i s generally a less desirable index of r e l i a b i l i t y i n a performance appraisal process;.;than i s i n t e r r a t e r r e l i a b i l i t y for at l e a s t two reasons [Cummings and Schwab, 1973; Guilford, 1954]. F i r s t , an indiv i d u a l ' s behaviour varies from one time to another; people do not always do the same thing i n the same way each and every time. Such variance i n behaviour does not necessarily r e f l e c t low,interrater r e l i a b i l i t y ; rather i t r e f l e c t s the dynamic nature of human performance [ G h i s e l l i , 1956; Inn, Hulin, and Tucker, 1972] . Second, t e s t - r e t e s t r e l i a b i l i t y i s less useful than i n t e r r a t e r r e l i a b i l i t y , because i t allows the rater's memory to influence the consistency of the scores he assigns to a given ratee, regardless of actual changes i n the person's job performance [A.P.A. Standards, 1977, p. 48; Dunnette, 1966, Chapter 2]. Obviously, to the extent that the consistency of the performance scores assigned to an i n d i v i d u a l are a function of the rater's memory, rather than the individual's performance, r e l i a b l e , but inaccurate, performance scores w i l l be the r e s u l t [Guion, 1965, Chapter 2]. Given the shortcomings of t e s t - r e t e s t r e l i a b i l i t y for purposes of performance appraisal, i n t e r r a t e r r e l i a b i l i t y i s preferred, where obtainable. I f , however, an assessment of i n t e r r a t e r r e l i a b i l i t y i s not possible, then s u f f i c i e n t time should be allowed to pass between the time ratings are made to d u l l the rater's memory for purposes of te s t - r e t e s t analysis. The problem, of course, i s i n a r r i v i n g at a d e f i n i t i o n of s u f f i c i e n t time that does not negate the u t i l i t y of performance feedback and the organization's need to appraise performance i n an ongoing manner. Constant Errors. Constant errors demonstrate one s a l i e n t c h a r a c t e r i s t i c : an unreal s i m i l a r i t y between scores [Cummings and Schwab, 1973]. I t may become apparent i n a rating assignment to an i n d i v i d u a l ratee ( i n t r a - i n d i v i d u a l e r r o r ) , or i n ratings given to a number of ratees ( i n t e r i n d i v i d u a l errors).. Because raters vary i n t h e i r a b i l i t y to judge performance objectively, performance ratings are*subject to the well-known halo, leniency, and central tendency errors. According to Guion [1965, Chapter 4], halo i s perhaps the most common error i n performance appraisal. I t refers to the tendency of an evaluator to rate a person i n a similar manner across q u a l i t i e s because of a general, o v e r a l l impression derived from a single quality or t r a i t of the person, whether favourable or unfavourable. Halo i s caused by the evaluator's i n a b i l i t y to dis t i n g u i s h between the d i f f e r e n t c h a r a c t e r i s t i c s of ratees. A second type of constant error i s leniency. I t describes a rater's tendency to be either too lenient or too s t r i c t i n his ratings, r e s u l t i n g i n markedly skewed d i s t r i b u t i o n s of ratings across in d i v i d u a l s . Leniency occurs because some raters are "nice guys" and tend to rate everyone favourably, while others are "hard-nosed" and tend to rate everyone unfavourably. While halo i s caused by the i n a b i l i t y or unwillingness of the rater to d i s t i n g u i s h within i n d i v i d u a l s , leniency i s caused by the i n a b i l i t y or unwillingness to di s t i n g u i s h between i n d i v i d u a l s . Central tendency i s the t h i r d type of constant error, characterized by a r e s t r i c t e d v a r i a b i l i t y around the midpoint of the scale. I t . i s caused by the unwillingness of the rater to use extreme ratings, e.g. very poor, outstanding. Dimensions of C r i t e r i a There are three possible approaches to develop performance measures: 1. to choose a single or global measure on the basis of i t s rela t i o n s h i p with other measures; 2. to combine c r i t e r i o n measures into a single composite score; or 3. to develop multidimensional measures [Inn, Hulin, and Tucker, 1972; James, 1973] . The Single C r i t e r i o n . There are several reasons why a single c r i t e r i o n may be d e f i c i e n t as a useful device for measuring performance. Commonly, performance constructs are multidimensional [Brumback and Vincent, 1970; G h i s e l l i , 1960; Peres, 1962; Roach and Wherry, 1970; Ronan, 196 3; Seashore, Indik, and Georgopoulos, 1960]. I f so, then a single c r i t e r i o n w i l l not adequately r e f l e c t performance and, therefore, the assessment w i l l be d e f i c i e n t [Nagle, 1953], Also, as Schmidt and Kaplan [19 71] point out, d i f f e r e n t patterns of behaviour within and across performance dimensions can r e s u l t i n equally successful performance i n a global sense. I t would be obviously d i f f i c u l t to cover these differences with a single c r i t e r i o n , making i t impossible to determine i n d i v i d u a l differences i n job success. This could be a serious problem i f an individual's performance within a p a r t i c u l a r job dimension would be s i g n i f i c a n t l y below standard, because th i s f a c t would be concealed by an o v e r a l l judgement [ T i f f i n and McCormick, 1965, p. 51]. Dunnette [1963] points out that a single c r i t e r i o n very l i k e l y i s both d e f i c i e n t and/or contaminated. I t i s d e f i c i e n t i f i t excludes important aspects of a job; i t i s contaminated i f i t contains i r r e l e v a n t aspects of a job. These c h a r a c t e r i s t i c s can be viewed as a v i o l a t i o n of Nunnally's domain sampling model as i t relates to the performance construct [Nunnally, 1967, p. 175]. The Composite C r i t e r i o n . The arguments i n favour of a composite c r i t e r i o n seem to focus on the need for an "overall measure of success" of an i n d i v i d u a l [Schmidt and Kaplan, 1971, p. 419]. T i f f i n and McCormick [1965,.p. 52] point to c e r t a i n circumstances where such a measure would be preferable, e.g. for personnel administration purposes (such as ra i s e s , promotion, e t c . ) , where some ordering of personnel along a single continuum i s desirable. However, advocates of a composite measure must face the problem of how to create t h i s "global e n t i t y from a group of i n d i v i d u a l measures" [Blum and Naylor, 1968, p. 184]. A number of alternative procedures have been proposed by Nagle [1953]. Multiple C r i t e r i a . The reason that the concept of a workable composite c r i t e r i o n has a great deal of i n t u i t i v e appeal i s perhaps because i t seems to be l o g i c a l and r e l a t i v e l y simple. However, there appears to be an increasing disenchantment taking place with i t s use i n modern selection research [Blum and Naylor, 1968]. Quite a few recognized researchers i n the f i e l d of personnel se l e c t i o n advocate the use of. multiple c r i t e r i a for several reasons. G h i s e l l i [1956] argues strongly against the use of composite c r i t e r i a . As he puts i t : If the proposition i s accepted that c r i t e r i a are multidimensional with the dimensions being independent, or at le a s t r e l a t i v e l y so, then the s i t u a t i o n i s not an easy one. There i s no way to combine the independent scores of an i n d i v i d u a l into a single value that w i l l describe him uniquely. Rather, i t w i l l be necessary to locate t h i s p osition i n the multidimensional c r i t e r i o n space. This can be accomplished i n either of two ways: each c r i t e r i o n can be predicted separately and the individual's position i n the space be estimated, or the space can be divided into parts and that portion of the space i n which the i n d i v i d u a l i s most l i k e l y to f a l l could be estimated by the discriminate function. [p. 2] Dunnette [196 3] i s rather firm i n his statements against composite measures and i n favour of multi-dimensional c r i t e r i a . He states: The point of a l l th i s i s to suggest that much selection and v a l i d a t i o n research has gone astray because of an overzealous worshipping of the c r i t e r i o n with an accompanying w i l l -o'-the-wisp searching for a best single measure of job success. The r e s u l t has been an over-s i m p l i f i c a t i o n of the complexities involved i n test v a l i d a t i o n and the prediction of employee success. Investigators have been reluctant to consider the many facets of success and the concomitant in v e s t i g a t i o n of the prediction of many success measures and instead p e r s i s t i n an u n f r u i t f u l e f f o r t to predict the c r i t e r i o n . Thus, I say: junk the c r i t e r i o n ! Let us cease searching for single or composite measures of job success and proceed to undertake research which accepts the world of success dimensionality as i t r e a l l y e x i s t s . [p. 252] A d i f f e r e n t aspect of the composite vs. multiple c r i t e r i a problem i s discussed by Schmidt and Kaplan [1971]. They see the major controversy a r i s i n g from the d i f f e r e n t views on the nature of performance c r i t e r i a . Advocates of composite measures stress the economical c h a r a c t e r i s t i c of a performance c r i t e r i o n [Gaylord and Brogden, 1964; Nagle, 1953; Thorndike, 1968; Toops, 1944]. On the other hand, supporters of multiple c r i t e r i a seem to perceive the meaning of a c r i t e r i o n as being primarily psychological or behavioural i n nature [ C a t t e l l , 1957; Dunnette, 1963; G h i s e l l i , 1956; Guion, 1965, p. 114; Haire, I960]. I t appears that the controversy i s actually unwarranted because each approach has i t s value, and perhaps both are complementary rather than competitive. To quote Schmidt and Kaplan [1971]: The t y p i c a l p r a c t i c i n g i n d u s t r i a l psychologist probably seeks both economic and psychological ends i n the v a l i d a t i o n process. From the point of view of the c r i t e r i o n end of the prediction equation, the implication of t h i s fact i s that he should, i d e a l l y , weight c r i t e r i o n elements, regardless of t h e i r i n t e r c o r r e l a t i o n s , into a composite representing an economic construct i n order to achieve his p r a c t i c a l goals, and, at the same time, he should analyze the relationships between predictors and separate c r i t e r i o n elements i n order to achieve h i s psychological goals. [p. 432] C r i t e r i o n Content Issue Besides the composite vs. multiple c r i t e r i a issue i n personnel research there seems to be another problem area where experts express d i f f e r e n t views, and that i s the content issue. Kavanagh [1971] describes two schools of thought i n t h i s f i e l d , one advocating the use of t r a i t s , and the other advocating behaviour as the focus of the performance c r i t e r i o n development process. Kavanagh himself advocates the t r a i t approach on the grounds that i t s use i s so widespread, i . e . , p r a c t i t i o n e r s seem to prefer i t . In contrast, Dunnette [1966] suggests that performance evaluations should focus on p o t e n t i a l l y observable behaviours rather than on the p o s s i b l y i r r e l e v a n t personality t r a i t s of appraisees. One has to look at empirical studies i n order to be able to make a decision on which approach should be used. T r a i t C r i t e r i a . A look into the l i t e r a t u r e on performance appraisals reveals l i t t l e empirical support for the use of t r a i t s as an e f f e c t i v e measurement of performance. Graham and Calendo [1969] investigated the relationship between personality measures and supervisory ratings and personal q u a l i t i e s or mannerism of employees. Out of 40 correlations between t r a i t and performance measures only two were s i g n i f i c a n t (p < .05). On the other hand, out of 64 correlations between t r a i t measures and supervisory ratings of. personal c h a r a c t e r i s t i c s 30 were s i g n i f i c a n t (p < .05). In a si m i l a r study, Taylor et a l . [195 8] compared t r a i t and behavioural scales with supervisory ratings of o v e r a l l performance. They found that the behavioural scales demonstrated s i g n i f i c a n t l y higher i n t e r r a t e r r e l i a b i l i t y than did the t r a i t scales (p < .01). Only when the t r a i t scales descriptions were stated i n terms of on-the-job work rel a t i o n s did t h e i r r e l i a b i l i t y c o e f f i c i e n t approach that of the behaviour scales. These re s u l t s are c l e a r l y not i n favour of the t r a i t oriented scales as suitable measures of job performance. In addition, the lack of s i g n i f i c a n t correlations between t r a i t scales and job performance measures raises the question of construct v a l i d i t y of t r a i t ratings for job performance. Behavioural C r i t e r i a . During the l a s t two decades behaviour oriented measures of job performance became more popular, beginning with Flanagan's [1954] C r i t i c a l Incident technique, expanded and modified by Smith and Kendall [1963] and Campbell et a l . [1970]. According to Dunnette [1966], the behavioural approach to performance appraisal reduces much of the ambiguity of t r a i t ratings by reducing the performance construct to the job i t s e l f (see also Harari and Zedeck, 1972). This desirable c h a r a c t e r i s t i c of behavioural scales i s empirically supported by F o g l i , Hulin, and Blood's [1971] and Smith and Kendall's [1973] generally high i n t e r r a t e r agreement (r > .97) on the scale value assigned to behavioural descriptions and by the conceptual unidimensionality of behavioural incidents within scales [Campbell et a l . , 1970]. Blood [19 74] points to several p o s i t i v e "spin-off e f f e c t s from the use of behaviourally oriented scales. (1) They extend the domain of evaluated performance. As Borman [1972] has demonstrated, when performance evaluation scales are developed at d i f f e r e n t l e y e l s i n an organization (superior, peers, subordinates) i t i s possible to get d i f f e r e n t behavioural dimensions from each group. Using as many organizational perspectives as possible (e.g. even cl i e n t s ) would considerably broaden the job behaviour domain. (2) They a s s i s t i n the development of t r a i n i n g programmes (see also Campbell et a l . , 1973). Since s k i l l s required on the job are described i n s p e c i f i c behavioural terms — as compared to the name of a s k i l l domain — shortcomings are much easier i d e n t i f i e d and s p e c i f i c t r a i n i n g programmes can be developed. In addition, since behaviour description can be stated i n negative ways trainers are able to specify unwanted behaviours trainees should learn to avoid. (3) They enable assessment of agreement on organizational p o l i c y . This i s possible, because for scale development behavioural items are rated according to the l e v e l of performance they describe. Items with a large variance are usually discarded. However, these items may be the most important ones for organizational development, because they may pinpoint areas of organizational p o l i c y . The mean item ratings can be c o l l e c t e d from members of d i f f e r e n t organizational l e v e l s , e..g. foremen and workers, and compared as to the i r agreement. If both groups disagree on the d e s i r a b i l i t y of s p e c i f i c behaviours, then obviously there i s a communication problem which should be corrected. In a recent review of the l i t e r a t u r e on the content issue i n performance appraisal, Kavanagh [1971] questioned the superiority of behaviour oriented c r i t e r i a over t r a i t s as suggested by Brumback and Vincent [1970] on the ground that the research evidence was equivocal. In a reply Brumback [1972] defended his views and offered additional reasons why job oriented c r i t e r i a are preferable. In a rejoinder Kavanagh [197 3] pointed out that the reasons mentioned by Brumback were not supported by empirical research and were often equally v a l i d for t r a i t c r i t e r i a . I t must be emphasized here that both Kavanagh and Brumback made no reference to the before mentioned Graham and Calendo [1969] study (discussed under " T r a i t C r i t e r i a " , p. 27 )/ which would have shed more l i g h t on the issues Kavanagh addressed i n his review, although the study had been published e a r l i e r i n the same journal i n which Kavanagh's and Brumback's a r t i c l e s appeared. 1 Graham and Calendo's research.is p a r t i c u l a r l y i n t e r e s t i n g i n th i s connection because of t h e i r stated objective which was to compare "...the degree to which personality measures obtained from workers are associated with supervisory ratings of job performance and the personal q u a l i t i e s or mannerisms of the worker" [p. 484]. As mentioned, the behaviour oriented c r i t e r i a came out ahead. xThe probable reason why Kavanagh did not mention t h i s study was that his review was submitted p r i o r to the publication of the Graham/Calendo a r t i c l e . Summary A c r i t e r i o n was defined as a measure of the extent to which an employee exhibits desirable job behaviours. C r i t e r i o n contamination and c r i t e r i o n deficiency was discussed i n rel a t i o n s h i p to the construct v a l i d i t y of a performance measurement. I t was shown that the r e l i a b i l i t y problem i n the c r i t e r i o n development process i s caused by two general types of error: variable and constant error. The possible dimensions of cri.teria were discussed, and i t was pointed out that the choice between composite and multiple c r i t e r i a cannot be made on the basis of superiority of either approach, but has to be made according to the intended use. And, f i n a l l y , the problem of t r a i t vs. behaviour oriented measurement was reviewed, and evidence was presented favouring the use of behavioural ratings. There are several' conclusions which can be drawn from th i s discussion. Firstly, i t w i l l be very d i f f i c u l t to do a sound construct v a l i d a t i o n of a performance appraisal and t r a i n i n g evaluation instrument because of the lack of proper independent c r i t e r i a . Perhaps only a: token v a l i d a t i o n may be possible. Secondly, i t seems safe to suggest that a single c r i t e r i o n i s inappropriate for both performance appraisal and tra i n i n g evaluation purposes. However, i t would be desirable to design instruments for both purposes i n such a way as to allow the use of either a composite c r i t e r i o n or multiple c r i t e r i a or both at the same time, depending on the objectives of the measurement. Thirdly, the structure and rating procedure of such instruments should be designed to reduce variable and constant errors made by raters. And fourthly, since t r a i t oriented scales appear to be less v a l i d for performance measurements, i t . follows that performance appraisal and tra i n i n g evaluation instruments should u t i l i z e behaviour oriented c r i t e r i a . The present study t r i e s to incorporate the suggestions and recommendations made above. CHAPTER III A LITERATURE REVIEW ON BEHAVIORALLY ANCHORED RATING SCALES Only 3 years have passed since Schwab, Heneman and DeCotiis [19 75] reviewed the l i t e r a t u r e on Behaviorally Anchored Rating Scales (BARS). During the period from 1970 to 1974 they found nine studies which were useful for evaluation. Since then 24 additional investigations into the c h a r a c t e r i s t i c s of BARS have been conducted. Six other studies during 1970 and e a r l i e r were not included i n the Schwab et a l . review but are part of t h i s survey, which covers a t o t a l of 40 studies. Of these, 21 compare BARS with other scales, and 19 o f f e r quantitative data on the c h a r a c t e r i s t i c s of BARS. These studies and t h e i r most important findings are reported i n Table I I I - l . Because of the volume of the l i s t i n g i t i s shown i n an appendix (see Appendix A). Separately, 21 other studies are l i s t e d which use BARS for purposes other than performance appraisals (Appendix B). As far as these publications contain data relevant to this discussion they are mentioned i n Table I I I - l (see Appendix A). Approximately 20 papers and publications on BARS are available which are not included i n t h i s survey because they are not comparative, descriptive i n nature, and/or do not contain any data relevant to the survey (see Appendix C). The Schwab et a l . review concentrated on three psychometric issues: (1) the s u s c e p t i b i l i t y of BARS to leniency e f f e c t s , (2) whether or not BARS get at independent performance dimension (halo), and (3) the r e l i a b i l i t y of BARS. The present review looks.at the same issues, but extends i t to other topics: (4) optimal development procedures and formats, (5) e f f e c t s of various factors on rating c h a r a c t e r i s t i c s , (6) optimal number of dimensions, (7) modified BARS instruments. Behaviorally Anchored Rating Scales The C r i t i c a l Incident Method In his search for a useful tool to measure work performance, Flanagan [1954] developed a technique which he c a l l e d the C r i t i c a l Incident Method. He defined the c r i t i c a l requirements of a job as those behaviours which are c r u c i a l i n making a .difference between doing the job e f f e c t i v e l y and doing i t i n e f f e c t i v e l y . C r i t i c a l incidents, as the term implies, are simply reports by q u a l i f i e d observers of the things people did that were e s p e c i a l l y e f f e c t i v e or i n e f f e c t i v e i n accomplishing parts of t h e i r jobs. Such incidents are actual behavioural accounts recorded as stories or anecdotes and obtained from managers, job incumbents, and others close to the job being studied. In i t s simplest form the C r i t i c a l Incident Method consists of l i s t i n g s of c r i t i c a l , incidents which are then compared to exhibited ,. behaviour of employees to be rated. The underlying objective of t h i s procedure i s to obtain a job-specif scale of behaviour effectiveness. The o r i g i n a l of rating scales with "concrete examples" [Taylor, 196,8] can be traced back to Thorndike [1910] who used i t to judge the q u a l i t y of handwriting. Example-anchored scales were also u t i l i z e d to determine c h a r a c t e r i s t i c s of drawings [Albee and Hamlin, 1949], for t r a i t ratings [Guilford 1954, p. 269], and rating of mental health [Luborsky, 1962] . Development of Behaviorally Anchored Rating Scales The f i r s t step i n developing BARS i s to ask a sample of job incumbents and/or t h e i r immediate supervisors and — w h e r e applicable — subordinates to write short descriptions of an incumbent:1 s job behaviour they have eith e r observed or heard about. Each behavioural..description i s characterized by the s p e c i f i c a t i o n of a single job s i t u a t i o n and the behaviour i n response to t h a t s i t u a t i o n . To be c r i t i c a l , an i n c i d e n t must occur i n a job s i t u a t i o n where the purpose or i n t e n t o f the behaviour i s f a i r l y c l e a r and where i t s consequences are d e f i n i t e enough to leave l i t t l e doubt concerning i t s e f f e c t on job performance [Flanagan, 1954]. Although the l a t t e r d e f i n i t i o n suggests d e s c r i p t i o n o f extreme behaviour, a c r i t i c a l i n c i d e n t need not r e f e r e x c l u s i v e l y to the extremes o f performance. A f t e r they are gathered, the p o o l of b e h a v i o u r a l i n c i d e n t s are u s u a l l y e d i t e d to conform to an expected behaviour format, t h a t i s , each d e s c r i p t i o n of an incumbent's job behaviour i s p r e f a c e d w i t h the phrase "Could be expected...." The i n t e n t of the phrase i s to allow the r a t e r to g e n e r a l i z e . f r o m what he has seen the r a t e e do i n a s i t u a t i o n to what he would expect the r a t e e to do i n a p a r t i c u l a r s i t u a t i o n , r e g a r d l e s s o f the o p p o r t u n i t y to a c t u a l l y observe the r a t e e . A f t e r the e d i t i n g and r e p h r a s i n g , the i n c i d e n t s are c a t e g o r i z e d . U s u a l l y the r e s e a r c h e r reads, s o r t s , and then l a b e l s groups of i n c i d e n t s i n terms of s i m i l a r i t y , or the r e s e a r c h e r f i r s t q u a l i t a t i v e l y i d e n t i f i e s a s e t of dimensions and then s o r t s statements to t h e i r s i m i l a r i t y i n meaning to the a p r i o r i d e f i n e d dimensions. To a v o i d c r i t e r i o n contamination through personal biases, an incident r e a l l o c a t i o n or "retranslation" procedure i s used [Smith and Kendall, 1963]. In the re t r a n s l a t i o n process experts (job-knowledgeable employees) are provided with a l i s t of job dimension d e f i n i t i o n s and asked to assign the incidents to the behavioural dimensions they f e e l they describe. C r i t e r i a of retention are included i n the procedure for determining the extent to which a p a r t i c u l a r incident i s part of a dimension. Those incidents meeting the c r i t e r i a . a r e retained for subsequent use. Following the re t r a n s l a t i o n procedure the incidents are rated on a Likert-type scale (usually from 1 to 7, or 1 to 9) as to the degree of effectiveness they characterize i n each job dimension. The mean rating for an incident determines i t s scale value, while the Standard Deviation (SD) of the mean rating i s viewed as an index of ambiguity. The procedure requires the ambiguous incidents ( i . e . , incidents with an SD i n excess of some minimum value) be excluded from the scales. The retained incidents are then ordered within the performance dimensions they anchor i n terms of t h e i r mean scale values. The usual arrangement of anchors i s a v e r t i c a l graphic scale, consisting of a v e r t i c a l l i n e , marked i n equal-appearing i n t e r v a l s , with incidents arranged along i t s length according to th e i r mean value. Each scale i s headed by a dimension d e f i n i t i o n and usually omits the scale value for each incident of behaviour. In using the scale, the rater i s instructed f i r s t to read the dimension d e f i n i t i o n for a scale. Then, he i s asked to read each incident s t a r t i n g at the bottom and reading toward the top u n t i l he reads an incident that exceeds the ratee's " t y p i c a l " best job behaviour. He then returns to the highest " t y p i c a l incident" and checks i t as i n d i c a t i v e of the ratee's job performance within that dimension of the job. The value of the incident, checked by the rater determines the performance score on that dimension. It i s possible to score the scales i n at lea s t two ways. F i r s t , i f a summation i s desired, scores can be summed across dimensions for a ratee. Second, i f a performance p r o f i l e i s desired, the score on each scale can be reported. Ty p i c a l l y , the l a t t e r format i s used. Differences Between BARS and Other Scales The most commonly used scale for performance appraisal purposes i s the Graphic Rating Scale (GRS) [Glueck, 1974, p. 292, see sample i n Appendix K ] . The evaluator i s presented with a chart or graph and asked to rate employees on each c h a r a c t e r i s t i c l i s t e d . The number of c h a r a c t e r i s t i c s rated may vary from a few to several dozen. The ratings can be i n a series of boxes or they can be on a continuous scale from 1 to 5 or 1 to 7. In the l a t t e r case, the evaluator places a check above descriptive anchors ranging from unsatisfactory to outstanding. Ty p i c a l l y , these ratings are then assigned points. For example, "outstanding" may be assigned a score of 5 and unsatisfactory a score of 1 or 0. A somewhat less popular scale i s the Summated Rating Scale (SRS) [Campbell et a l . , 1973]. The development of SRS i s s i m i l a r to the development of BARS, as the job to be rated i s divided into dimensions. These dimensions are then broken down into t h e i r major elements (behaviour descriptions and/or functions). Individuals are then rated on whether they exhi b i t each behaviour never, ra r e l y , often, or always. An individual's rating on a dimension i s the average item response for that dimension. T h e r e a r e o t h e r s c a l e s a v a i l a b l e f o r p e r f o r m a n c e a p p r a i s a l p u r p o s e s , b u t t h e y a r e t o o d i f f e r e n t i n n a t u r e f r o m BARS and f o r t h a t r e a s o n a r e n o t u s e d i n c o m p a r i s o n s t u d i e s . P s y c h o m e t r i c P r o p e r t i e s L e n i e n c y E f f e c t O u t o f t w e n t y - o n e c o m p a r a t i v e s t u d i e s r e v i e w e d , f i f t e e n a d d r e s s e d t h e p r o b l e m o f l e n i e n c y e f f e c t s o n BARS a n d o t h e r s c a l e s . L e n i e n c y i s d e f i n e d a s a s h i f t i n mean r a t i n g s f r o m t h e m i d p o i n t o f t h e s c a l e i n t h e f a v o u r a b l e d i r e c t i o n [ S h a r o n a n d B a r t l e t t , 1969]. The r e s u l t s a r e shown i n T a b l e I I I - 2 . E i g h t s t u d i e s i n d i c a t e t h a t BARS a r e s i g n i f i c a n t l y o r d e m o n s t r a b l y s u p e r i o r , s e v e n d i d e i t h e r n o t f i n d a s i g n i f i c a n t d i f f e r e n c e o r f o u n d o t h e r s c a l e s t o be s u p e r i o r . Schwab e t a l . [19 75] c a u t i o n a g a i n s t t h e c o m p a r i s o n o f BARS w i t h o n l y one a l t e r n a t i v e m e t h o d b e c a u s e o f t h e d i f f i c u l t y i n d e t e r m i n i n g t h e t r u e a v e r a g e r a t i n g . They e i t h e r recommend t h e u s e o f more t h a n two s e t s o f i n s t r u m e n t s o r t o a s c e r t a i n a g r e a t e r a p r i o r i k n o w l e d g e o f t h e g r o u p b e i n g e v a l u a t e d ; One way t o s o l v e t h e l a t t e r p r o b l e m was shown b y D e C o t i i s [197 4 ] , who w r o t e f i v e v i g n e t t e s d e s c r i b i n g b e l o w -a v e r a g e , a v e r a g e , a n d a b o v e - a v e r a g e p e r f o r m e r s , m a k i n g TABLE 2. COMPARATIVE STUDIES ON LENIENCY EFFECT LEN D A D-C IENCY EFFECT r i t h p r c.r.i 1 es S t u d y BARS 1 Sea 1 e L e n i e n c y j P o i n t s Len i ency Type Sea 1 e Poi n t s BARS S u p e r i o r ? j Yes/No | — 1 Remarks B a r r e t t e t . a l . ( 1 9 5 8 ) _ h i . i 1 ess 15 GRS 15 i Yes L e n i e n c y measured w i t h F i s h e r ' s " g " (Skewness) T a y l o r ( 1 9 6 8 ) lower ( X 2 = . 0 0 1 ) 7 GRS 4 Yes Campbel1 ( 1 9 7 3 ) l e s s 9 • SRS Yes d a t a not c o m p a r a b l e be-cause o f d i f f e r e n c e i n number o f s c a l e p o i n t s Borman 6 V a l I o n ( 1 9 7 4 ) 1 . 4 3 9 1 . 1 4 GRS 3 Ho Burnuska & Hoilman 0 9 7 * 0 s i i g h t l y 1 ess 9 • GRS f ,-No d i f f e r e n c e n o t s i g n i f i c a n t Borman & Dunnette / 1 om) 1 . 7 7 9 1.98 2 . 0 4 • NAS TF 9 9 Yes BARS v s . Non-anchored s c a l e (NAS) v s . T r a i t Format (TF) -( 1 if.}.). Keaveny & McGann ( 1 9 7 5 ) . 1-71 9 1 .92 GRS 9 Yes B e r n a r d i n , A l v a r e s & Cranny 0 9 7 6 ) 1 . 0 2 7 1 . 1 2 . 5 8 •• SRS 1 SRS 2 7 7 No TABLE 2 COMPARATIVE"STUD IES ON LENIENCY EFFECT ( c o n t i n u e d ) 1 LENIENCY EFFECT BARS Other S e a l e s Study Len i ency Sea l e Po i n t s Len i ency Type Sea 1 e P o i n t s BARS S u p e r i o r ? Yes/No Remarks B e r n a r d i n, L a S h e l I s , Smi t h & A l v a r e s ( 1 9 7 6 ) . 0 8 7 . 8 3 SRS 7 Yes Friedman & C o r n e l i us ( 1 9 7 6 ) no d i f f e r e n c e 7 no d i f f e r e n c e GRS 7 No da t a no r e p o r t e d Mi 1 l a r d e t . a l . ( 1 9 7 6 ) B: 1.99 T: . 2 9 9 ,37 GRS 9 Yes B = B e s t P e r f o r m a n c e T •» T y p i c a l P e r f o r m a n c e Zudeck, K a f r y & Jacobs ( 1 9 7 6 ) . 9 8 7 1 l a : 1 . 3 2 1 l b : 1 . 12 I l i a : 1 . 3 2 11 l b : 1 . 0 3 ... 7 No Other S c a l e s were GRS w i t h h d i f f e r e n t s c o r i n g f o r m a t s . D i f f e r e n c e be-tween BARS and 1 M b n o t s i qn i f i c a n t . B e r n a r d ! n (1977) .67 7 .Gk . 5 8 SRS] S R S 2 7 7 No d i f f e r e n c e not s i g n i f i -c a n t DeCot i i s ( 1 9 7 7 ) 1 ess 7 GRS 9 k t o 8 No H a r i d a s , F r o s t £ Barnowe ( 1 9 7 7 ) 1 ess 7 GRS 5 Yes BARS l e s s skewed 4^ i t possible to tes t the accuracy of BARS without contamination caused by personal knowledge of ratees by raters. Only 5 out of 15 studies mentioned used three d i f f e r e n t rating instruments. Borman and Dunnette [19 75] compared BARS with a non-anchored scale and a t r a i t format and found the BARS instrument to be superior. Bernardin [1977] and Bernardin, Alvares and Cranny [1976] used, besides BARS, two types of Summated Rating Scales (SRS). The r e s u l t of the f i r s t study indicated no s i g n i f i c a n t difference, while the second study showed one of the SRS to be superior to BARS. Zedeck, Kafry, and Jacobs [1976] used bars and two types of Graphic Rating Scales (GRS) with d i f f e r e n t scoring formats. Again, no s i g n i f i c a n t difference was found. However, the BARS instrument was at le a s t as good as the best GRS instrument. In the f i f t h study, DeCotiis [1977] compared BARS with a Numerically Anchored Rating Scale (NARS), and a GRS based on t r a i t s . None of the scales showed any superiority with respect to leniency. Judging only from the re s u l t s of the f i v e studies discussed above, the outcome i s equivocal. None of the d i f f e r e n t scale types scored consistently better. However, the attention of the reader must be drawn to a study conducted by Bernardin, LaShells, Smith, and Alvares [19 76] which i s of special relevance here. By employing an optimal approach i n BARS development, Bernardin et a l . were able to minimize the leniency e f f e c t s i g n i f i c a n t l y (.08 vs. .89 of a SRS as measured by deviation from midpoint). Although the g e n e r a l i z a b i l i t y of t h e i r findings i s questionable, i t seems j u s t i f i e d to recommend a ca r e f u l look into the procedures used during the developmental phase of BARS. (The study i s discussed i n more d e t a i l under "Optimal Development Procedures", p. 59 ) . Dimension Independence (HALO) Dimension independence i s an important c h a r a c t e r i s t i c of r a t i n g scales, since i t would be meaningless to develop performance measurements for d i f f e r e n t job dimensions i f they were highly i n t e r -correlated. Dimension independence was one of the major advantages hypothesized for. BARS, mainly because of the use of the r e t r a n s l a t i o n procedure which was s p e c i f i c a l l y developed to eliminate behaviour descriptions that are not c l e a r l y categorizable [Smith and Kendall, 1963]. Most investigators reviewed have measured halo ef f e c t s by c o r r e l a t i n g and averaging the rating scores of ratees across a l l performance dimensions. A d i f f e r e n t measure was applied by Bernardin [1977] and Bernardin, Alvares, and Cranny 11976], who defined halo as the extent to which raters place a ratee at the same l e v e l on d i f f e r e n t dimensions [Bernardin, 1977, p. 5]. In this case, a lower mean score indicates a stronger halo e f f e c t . As Schwab et a l . [1975] point out, i t i s u n r e a l i s t i c to expect t o t a l independence between dimensions. Therefore, the question has to be: How much better — i f any — are BARS? Sixteen of the twenty-one comparative studies tested for halo e f f e c t s (see Table III-3). In eight studies BARS exhibited s i g n i f i c a n t l y less halo than other scales. Out of the remaining eight studies six showed equivocal r e s u l t s .[Bernardin, 1977; Borman and Vallon, 1974; Burnaska and Hollmann, 1974; DeCotiis, 1977; Friedman and Cornelius, 1976; and Haridas, Frost, and Barnowe, 1977], while i n two studies other scales were found to be superior [Bernardin, Alvares, and Cranny, 1976; and Zedeck, Kafry, and Jacobs, 1976]. The findings do not demonstrate a clear s u p e r i o r i t y of BARS, although i n the majority of studies the instrument fared quite well i n comparison with other scales, being at lea s t equal to the best of the other scales or s i g n i f i c a n t l y better. Perhaps the cautious TABLE 3 COMPARATIVE STUDIES ON DIMENSION INTERCORRELATION BARS OTHER SCALES Study Halo E f f e c t Number of Dimensions Halo Ef f e c t Types Number of Dimensions BARS Superior Yes/No Remarks Barrett et. a l . (1953) .51 .56 8 .56 - .63 .50 - .65 GRS.^ GRS2 8 Yes Taylor (1968) lower ( X 2= .001) 6 GRS 6 Yes Campion (1972) .01 - .27 2 .10 - .62 .67 - .87 EP 2 Yes *BARS vs. 2 Employment Tests Campbell et. a l . (1973) .28 - .63 9 . 35. - . 76 SRS Yes Eorm.an & Vallon (1974) .73 5 .73 GRS No Burnaska' & Hollmann (1974) "somewhat le s s " * 5 GRS No *AN0VA design Borman £ Durmette (1975) • less (p < .02) 14 GRS Yes Keaveny & McGann (19 75) .21 - .58 13 .33 - .66 GRS 13 Yes TABLE 3 COMPARATIVE STUDIES ON DIMENSION INTERCORRELA'iION (continued) BARS OTHER £ CALES Study Kalo E f f e c t Number of Dimensions Halo E f f e c t Types Number of Dimensions BARS Superior Yes/No Remarks Bernardin, Alvares, £ Cranny (1976) .36* • {mean) 9 * .45 .39* (mean) SRS-, SRS 2 9 No * Halo defined as extent to which raters place a ratee at the same l e v e l on d i f f e r e n t dimensions;higher score indicates less halo, (no s i g n i f i c a n t difference) Friedman 5 Cornelius (1976) no difference* 10 no difference* GRS 10 No data not reported M i l l a r d , Luthans, & Ottemann (19 76) high single factor loading per dimension* 6 mixed loadings, unclear factor structure. GRS 6 Yes *Halo was tested through factor analysis; high, single load-ings per factor was perceived as evidence of l i t t l e halo. Zedeck, Kafry, & Jacobs (1976) I: .00 - .68 rnedian: .35 9 I l a : .07 - .72* median: .39 I l i a : -.23 - '.53 median: .23 I I l b : -.17 - .75 median: .43 SRS GRS 9 No *BARS (I) vs. SRS ( I l a + l i b ) vs. GRS ( I l i a + I l l b ) (a and b d i f f e r i n scoring method) 3err.ardin (1977) 1.13* 7 .96* 1.09* (mean) SRS! SRS2 7 7 No * see d e f i n i t i o n of halo i n Bernardin et. a l . (1976) DeCotiis (1977) r = .73 man 6 r , = .91 m d n .85 GRS GRS 11 10 No CO TABLE 3 COMPARATIVE STUDIES ON DIMENSION INTERCORRELATION (continued) BARS OTHER SCALES Study' Halo Effect Number of Dimensions Halo Ef f e c t Types Number ot Dimensions EARS superior Yes/No Remarks Haridas, Frost & Barnowe (1977) .19 (mean) 8 .31 (mean) GRS 8 Yes Keaveny & McGann (1977) .36 13 .47 .47 GRS. GRS^ 13 13 Yes • conclusion i s j u s t i f i e d that, with respect to halo, BARS appear to be moderately better than other scales. There i s enough empirical evidence to suggest that i t i s possible to s i g n i f i c a n t l y reduce halo error through appropriate t r a i n i n g of raters (see discussion under "Effects of Various Factors on Rating Character-i s t i c s " , p. 65 ) . This i s not only true for BARS but also for other scales [Latham, Wexley, and P u r s e l l , 1975] . Kafry, Jacobs, and Zedeek [1977] add some int e r e s t i n g thoughts to the issue of dimension independence. They see as one possible explanation a "true" r e l a t i o n s h i p among performance dimensions where "conceptually pure dimensions (may) represent d i f f e r e n t elements of a more general factor" [p. 4]. They recognize that t h i s would be empirically d i f f i c u l t to prove since "true" values, i n a performance evaluation sense, may be known only i n controlled laboratory settings. A second explanation, namely that a rater's capacity to d i f f e r e n t i a t e between d i f f e r e n t dimensions may be limited, i s discussed more thoroughly under "Optimal Number of Job Dimensions" Interrater R e l i a b i l i t y This review w i l l not discuss the issue of scale r e l i a b i l i t y , since i t i s dealt with i n several studies (e.g., Smith and Kendall, 1963; Landy and Guion, 1970; F o g l i , Hulin, and Blood, 1971]. I t i s s u f f i c i e n t to say that very high scale r e l i a b i l i t i e s were found, usually i n the .95 to .99 range. Schwab et a l . [1975] reported only three studies which compared BARS and other scales with regard to in t e r r a t e r r e l i a b i l i t y . They suggest that BARS o f f e r only moderate r e l i a b i l i t y [p. 557]. The present review i s based on thirteen comparative studies which address the same question. As Table III-4 demonstrates, seven studies yielded superior r e s u l t s for the BARS instrument, while six found no differences or concluded that other scales were superior. I t i s d i f f i c u l t to arrive at a conclusive judgement based on these r e s u l t s . Additional information can be gained by looking at the non-comparative studies which report r e l i a b i l i t y scores. The eleven studies of t h i s nature (they are l i s t e d as part of Table I I I - l i n Appendix A) show i n t e r r a t e r r e l i a b i l i t y scores ranging from .07 to 1.00 (mean = .47). This compares with a range of .25 to .89 (mean = .59) of the twelve comparative studies. (The TABLE 4 COMPARATIVE STUDIES ON INTERRATER RELIABILITY IMTERRA1 CER RELIABILITY Stud-/ BARS Other Scales Types bAKS superior.." Yes/No Remarks Barrett et. a l . (1953) .67 .51 .53 .50 GRS GRS Yes Two types of BARS vs. two types of GRS Maas (1965) .58 .47 .47 • .69 . 35 . 34 .34 . 34 GRS GRS GRS GRS Yes 4 groups Peters & McCormick (1966) .27 - .84 mean: .62 .06 - .87 mean: .57 GRS Yes Taylor (1968) .64 - .89 mean: .43 .10 - .63 mean: .43 GRS Yes Campbell et. a l . (1973) • .31 - .55 mean: .43 .42 - .58 mean: .53 ' SRS No Borman & Vallon (1974) .61 .56 GRS Yes Burnaska & Kollmann (1974) .78 .81 .63 GRS GRS No Borman & Dunnette (1975) higher NA: p< .05 TF: p< .02 NA TF Yes NA = Non-anchored scale (GRS) • TF = T r a i t format (GRS) TABLE 4 COMPARATIVE STUDIES ON INTERRATER RELIABILITY (.continued) INTERRATER RELIABILITY Study BARS Other Scales Types BARS Superior? Yes/No Remarks Grower (1975) .25 - .36 .00 - .08 .07 - .11 GRS SRS Yes Bernardin, Alvares £ Cranny (1976) 1.46 1.32 1.37 SRS SRS No Intra-dimensional l e v e l differences were used to estimate i n t e r r a t e r r e l i a b i l i t y (higher value indicates lower r e l i a b i l i t y ) Bernardin, LaShells, Sniith & Alvares (1976) .76 .75 SRS No Bernardin (1977) no difference SRS No ANOVA, F (2,42) = 2.78, n.s. DeCotiis (1977) r = . 6 4 man r . - . 8 2 mdn _ 8 i GRS GRS No range for a l l GRS and SRS i s .11 to .87, mean = .45). Although t h i s additional information has l i t t l e ; s c i e n t i f i c value, i t at lea s t indicates that BARS developed so far o f f e r only moderate r e l i a b i l i t y , thus supporting Schwab et al . ' s conclusion. There are several possible reasons for the r e l a t i v e l y low i n t e r r a t e r r e l i a b i l i t y of ra t i n g scales One reason given by some researchers i s the use of raters from d i f f e r e n t organizational levels [Borman, 1974; Campbell et a l . , 1973; DeCotiis, 1977; Schneier and Beatty, 1977; Zedeck and Baker, 1972; Zedeck, Imparato, Krausz, and Oleno, 1974]. Zedeck et a l . o f f e r three plausible explanations: 1. Different supervisory groups d i f f e r i n t h e i r opportunity to observe and evaluate subordinates. 2. D i f f e r e n t supervisory groups value or perceive d i f f e r e n t behaviours as important i n terms of meeting job requirements. 3. Different supervisory groups value the same behaviours d i f f e r e n t l y . Zedeck et a l . asked two groups (supervisors and subordinates) to p a r t i c i p a t e independently i n the development of BARS. Although both groups i d e n t i f i e d s imilar behaviour dimensions, the behavioural incidents defining the dimensions were valued d i f f e r e n t l y , thus giving support for the t h i r d proposition. In a s i m i l a r study, Borman [1974] found that two groups (instructors and secretaries) i d e n t i f i e d d i f f e r e n t performance dimensions for the s e c r e t a r i a l job. The r e s u l t i n g i n t e r r a t e r r e l i a b i l i t y was lower when one group used the BARS instrument developed by the other groups-An additional possible cause for low i n t e r r a t e r r e l i a b i l i t y , multidimensional!ty, i s suggested by Schwind [1976; 1977]. When c r i t i c a l incidents are generated, the intention i s to sample to a s i g n i f i c a n t degree the behaviour domain of a job. The second step i s then to categorize the behaviour samples into meaningful job dimensions [Smith and Kendall, 1963] . The problem i s that there are often so many job dimensions that i t i s impractical to develop scales for each one. (See Table I I I - l , Appendix A, for an in d i c a t i o n of the number of possible dimensions.) Job dimensions are usually collapsed into a manageable number, e.g. 5 to 10 (see Kafry, Jacobs, and Zedeck, 1977, for a relevant discussion). Schwind points out that many, i f not most, BARS have used job dimensions with behaviour samples which are multidimensional. The rater using such scales may be put into a p o s i t i o n of having to make very d i f f i c u l t choices, opening the rating procedure to possible biases, l i k e leniency, halo, and central tendency. To i l l u s t r a t e the problem, an example i s taken from Campbell, Dunnette, Arvey and Hellervik [1973]. Campbell et a l . i d e n t i f i e d , nine job dimensions of department managers (see Table 111-5);. A BARS for one of the dimensions was included as an example (see -.. Table III-6). A rough analysis of the behaviour items seem to point to f i v e r e l a t i v e l y independent types of behaviour. There, Items 1 and 2 (1 and 2 are also the scale points) seem to describe "making promises", 3 and 5 "dir e c t i n g " , 4 and 6 "showing consideration", 7 and 9 "t r a i n i n g " , and 8 "delegation of r e s p o n s i b i l i t y . I t i s conceivable that a l l of these behaviours could be displayed at the same time and be judged "most t y p i c a l . " If thi s i s true, then i t i s also conceivable that d i f f e r e n t raters (in this case the store manager and assistant store manager) make d i f f e r e n t choices; i . e . , select d i f f e r e n t behaviours to evaluate a ratee. I t may be that the store manager emphasizes d i f f e r e n t behaviours than the assistant manager or that both have d i f f e r e n t attitudes toward the department manager, causing one to perceive only posi t i v e behaviours while the other concentrates on negative behaviours. Table III - 5--Job Dimensions of Department Manager 1. Supervising sales personnel 2. Handling customer complaints and making adjustments 3. Meeting day-to-day deadlines 4. Merchandise ordering 5. Developing and planning s p e c i a l promotions 6. Assessing sales trends and acting to maintain merchandising p o s i t i o n 7. Using company systems and following through on administrative operations 8. Communicating relevant information to associates and to higher management 9. Diagnosing and a l l e v i a t i n g s p e c i a l department problems. from Campbell, J.P., Dunnette, M.D., Arvey, R.D., and H e l l e r v i k , L.V., "The Development and Evaluation of Behaviorally Based Rating Scales." Journal of Applied Psychology, V o l . 57, No. 1 (1973) 15-22. 58 Table III - 6 Behaviourally Based Rating Scales Could be expected to give h is sales personnel confidence and a strong 1 sense of r e s p o n s i b i l i t y by dele-gating many important jobs to them. Could be expected to exhibit courtesy and respect toward h is sales personnel. Could be expected to be rather c r i t i c a l of store standards i n front of h i s own people, thereby r i s k i n g t h e i r developing poor a t t i t u d e s . Could be expected to go back on a promise to an i n d i v i d u a l whom he had told could transfer back into previous department i f she/he didn't l i k e the new one. — Could be expected to conduct a f u l l day's sales c l i n i c with two new sales personnel and thereby develop them into top sales people i n the department. - 8 Could be expected never to f a i l to conduct t r a i n i n g meetings with h is people weekly at a scheduled hour and to convey to them exactly what he expects. Could be expected to remind sales personnel to wait on customers instead of conversing with each other. Could be expected to t e l l an i n d i v i d u a l to come i n anyway even though she/he c a l l e d i n to say she/he was i l l . Could be expected to make promises to an i n d i v i d u a l about her/his salary being based on department sales even when he knew such a pr a c t i c e was against company p o l i c y . F i g . 1 Scaled expectations r a t i n g scale for the effectiveness with which the department manager supervises h i s sales personnel. from Campbell, J.P., Dunnette, M.D., Arvey, R.D., and H e l l e r v i k , L.V., "The Development and Evaluation of Behaviorally Based Rating Scales." Journal of Applied Psychology, V o l . 57, No. 1 (1973) 15-22. While the low to moderate i n t e r r a t e r r e l i a b i l i t y i n this study (.31 - .55) may have been caused by using rater groups from d i f f e r e n t organizational l e v e l s , i t can be expected that even raters from the same l e v e l w i l l choose d i f f e r e n t behaviours i f independent or overlapping behaviour samples are u t i l i z e d i n BARS. This raises the question whether i t i s generally possible to develop purely unidimensional BARS, given the complexity of most jobs.. I f not, then i t i s unli k e l y that this instrument w i l l ever demonstrate desirable le v e l s of i n t e r r a t e r r e l i a b i l i t y . Optimal Development Procedures and Formats In the discussion of the res u l t s of t h e i r review Schwab et a l . conclude that "...there i s l i t t l e reason to believe that BARS are superior to alternative evaluation instruments." [p. 557] One reason, they suggest, may have to do with the development procedures for BARS, which may enhance or reduce t h e i r q u a l i t y . They point s p e c i f i c a l l y to three aspects: 1. only a small number of incidents are actually used on the completed scale from the t o t a l i n i t i a l pool, thus r e s u l t i n g i n perhaps i n s i g n i f i c a n t representation of a behaviour domain; 2. the re t r a n s l a t i o n agreement c r i t e r i o n for retaining incidents (usually some spe c i f i e d percentage) may not be stringent enough, r e s u l t i n g i n ambiguity; 3. the standard deviation c r i t e r i o n for item ratings may be set so that a substantial amount of disagreement about the scale value of an item i s permitted, again causing ambiguity. Bernardin, LaShells, Smith and Alvares [19 76] attempted to develop optimal procedures and formats for BARS. The following variations were tested: a) r e t r a n s l a t i o n c r i t e r i o n 5 0% to 60% vs. 80% and over; b) categorization and item rating by same group vs. categorization and item r a t i n g by d i f f e r e n t groups; c) use of the completed instrument by participants i n scale development vs. use by non-participants; d) continuous vs. non-continuous scale ( i . e . scales without and with scale points); e) two scoring methods; f) scales with statements intended to c l a r i f y dimensions vs. scales without such statements. The scales r e s u l t i n g from these d i f f e r e n t approaches were tested for d i s c r i m i n a b i l i t y , i n t e r r a t e r r e l i a b i l i t y , r a t i n g v a r i a b i l i t y , and leniency error. Based on t h e i r findings, Bernardin et a l . recommend the following procedures and formats: 1. Have one group of participants place c r i t i c a l incidents into dimension categories. Then, a f t e r e x t r i c a t i n g those that do not meet at lea s t a 60% placement c r i t e r i o n , organize incidents into t h e i r placed categories and have another group rate them on the i r d e s i r a b i l i t y . This procedure, while requiring more time and personnel, appears to foster a better comparison of the r e l a t i v e merits of each incident i n the context of the other incidents within a dimension. Such a procedure can ultimately r e s u l t i n more stable items within each scale. 2. Have the rater population develop .dimension c l a r i f i c a t i o n statements to be placed at anchor points on each scale. Again, while requiring more time and expense, such a format appears . to i n h i b i t leniency error and increase rating d i s c r i m i n a b i l i t y across raters. 3. Follow the Smith and Kendall (1963) recommendations for ra t i n g i n which the rater places observed c r i t i c a l incidents d i r e c t l y relevant to the ratee at the positions on the scale where he feels they belong. A summary rating i s then compiled by averaging the scale values of the new items. Such a procedure appears to i n h i b i t leniency error, increase r a t i n g v a r i a b i l i t y across dimensions and within ratees and increase d i s c r i m i n a b i l i t y across raters. [p. 79] However, i t should be emphasized that the optimal approach to the development of BARS does not mean that a scale.superior to other appraisal instruments w i l l r e s u l t . The studies by Bernardin [1977] and Bernardin, Alvares, and Cranny [1976] j u s t i f y t h i s caveat. In t h i s connection, because of some methodological flaws, Bernardin et a l . question the findings of the Campbell et a l . [1973] study which concluded that the BARS instrument used was superior to a summated scale. The main c r i t i c i s m was that the BARS instrument, because of i t s more rigorous development procedures, may have had an unfair advantage i n the comparison. To t e s t t h e i r assumption, Bernardin et a l . developed BARS and SRS under comparable, stringent conditions and found that the summated scale showed less leniency error and greater i n t e r r a t e r r e l i a b i l i t y than the BARS format. They concluded: The implications from t h i s study are c l e a r . The methods of constructing scales are as important as the finished product. Any form of a rigorously developed scale, be i t (BARS), summated, or any other format, may be psychometrically superior to scales that have not been rigorously developed. Thus, i n comparing rating formats, the method of scale development should be f i r s t examined. I f comparable e f f o r t has not been exercised to insure equally rigorous scales, the implications of the re s u l t s are hopelessly confounded. The study by Bernardin [1977] showed sim i l a r r e s u l t s . However, he points out that a BARS instrument, under cert a i n conditions, may have advantages, e.g. i t could provide s p e c i f i c behavioural feedback to ratees. As he puts i t : When the emphasis of appraisal changes from promotional and wage setting c r i t e r i a to means of performance improvements and behavioural change, such a n c i l l a r y "spin-offs" of the (BARS) approach loom large. [p. 8] Campion, Greener, and Wernli [1973] investigated whether d i r e c t observation of c r i t i c a l behaviours or the r e c a l l method i n c o l l e c t i n g behavioural samples had any influence on scale c h a r a c t e r i s t i c s . They concluded that both approaches seemed to be of equal value for generating behavioural samples, both i n terms of t h e i r r e l i a b i l i t y and t h e i r g e n e r a l i z a b i l i t y across rater groups. A d i f f e r e n t arrangement of c r i t i c a l incidents on the f i n a l scale was tested by Haridas, Frost, and Barnowe [1977]. The incidents were placed i n random order on the scale, without mentioning the degree of effectiveness they described. In the opinion of Haridas et a l . , the absence of numerical anchors beside the behavioural samples i s l i k e l y to reduce central tendency, leniency, and halo i n two ways: a) Since the raters do not have any numerical scales aiding them i n ordering the various behavioural items, they are l i k e l y to assess the implications of each behaviour c a r e f u l l y before choosing any items as the most appropriate behaviour of the ratee, and b) ignorance of the mean 'effectiveness' score of various behavioral items on the part of the raters i s l i k e l y to make th e i r choice of items across performance dimensions v i r t u a l l y independent. [p. 6] Haridas et a l . compared this BARS instrument with a graphic rating scale and found BARS to be superior i n leniency, v a r i a b i l i t y , and halo. Zedeck, Kafry, and Jacobs [1976] compared three formats (BARS, SRS, and GRS), the l a t t e r two being scored a d d i t i o n a l l y i n two d i f f e r e n t ways. The r e s u l t s did not j u s t i f y the inference of any superiority of one format or scoring method over another; only that response formats and scoring systems that have the same underlying frame of reference may lead to d i f f e r e n t interpretations with respect to performance evaluation. Zedeck et a l . recommend that, rather than wasting more time and e f f o r t i n studying the e f f e c t s of format and scoring procedures and response biases, i t should be assumed that some raters and formats are more prone to such biases and variations than others, and that the stress should be on t r a i n i n g of evaluators to be as perceptive and objective as possible. E f f e c t s of Various Factors on Ratings P a r t i c i p a t i o n i n Scale Development Smith and Kendall, i n the i r a r t i c l e [1963] on the development of a BARS instrument, predicted that p a r t i c i p a t i o n i n the development of the scale by those who eventually use i t would improve the v a l i d i t y of ratings. This proposition was tested f i r s t by Keaveny and McGann [1975], but t h e i r i n v e s t i g a t i o n f a i l e d to fin d any advantages i n the ratings of those raters who participated compared to those who did not. The same question was addressed by Bernardin, LaShells, Smith and Alvares [1976] and Friedman and Cornelius [1976] . While Bernardin et al'. found no s i g n i f i c a n t differences i n d i s c r i m i n a b i l i t y , i n t e r r a t e r r e l i a b i l i t y , r ating v a r i a b i l i t y , and leniency, Friedman and Cornelius concluded that rater p a r t i c i p a t i o n led to greater convergent v a l i d i t y , less halo, and lower leve l s of variance attributable to ra t i n g errors for both the BARS and GRS instruments they compared. P a r t i c i p a t i o n did not lead to higher levels of discriminant v a l i d i t y . 66 The l a t e s t study investigating t h i s question was undertaken by Kafry, Jacobs, and Zedeck [19 77]. No differences were found between the two groups (participants and non-participants) with regard to mean values, standard deviations and i n t e r c o r r e l a t i o n s of scale evaluations; however, as i n the Friedman/ Cornelius study, a lack of discriminant v a l i d i t y was evident. The results seem to indicate that p a r t i c i p a t i o n i n scale development does not necessarily lead to improved rating c h a r a c t e r i s t i c s . Training The e f f e c t of rater t r a i n i n g on various character-i s t i c s of ratings with BARS has been explored i n three studies. Bernardin and Walter [1977] compared rating c h a r a c t e r i s t i c s of raters who were exposed to (a) 1 hour or more tr a i n i n g on ra t i n g errors, the use of BARS, and were required to keep a diary to record c r i t i c a l incidents; (b) 1 hour t r a i n i n g and exposure to BARS, (a): 1 hour tra i n i n g , no exposure, and (d) no t r a i n i n g . The f i r s t group performed s i g n i f i c a n t l y better than the other groups with respect to leniency, halo, and i n t e r r a t e r r e l i a b i l i t y , while the second group performed better than the l a s t two i n the same categories. Group four did l east well. Borman [1975] had managers rate 6 vignettes, describing the performance of hypothetical f i r s t - l i n e supervisors. A 5 minute t r a i n i n g session on halo error s i g n i f i c a n t l y decreased the halo e f f e c t but, unexpectedly, lowered i n t e r r a t e r r e l i a b i l i t y . No e f f e c t on leniency and v a l i d i t y was found. However, the question of g e n e r a l i z a b i l i t y of these findings may be raised. Borman points out that the res u l t s may not hold up i n practice, i f the ratees were actual persons f a m i l i a r to the raters. Norton, Gustafson, and Foster [1977] used a simi l a r approach to t e s t -- among other variables — the e f f e c t of t r a i n i n g . Two cases ~ with a male and female version for each —• were used to describe one more and one less competent managerial performance. Male and female managers used BARS to evaluate each case before and afte r being exposed to lectures on leniency, halo, and central tendency. Raters were s i g n i f i c a n t l y less lenient i n t h e i r ratings a f t e r t r a i n i n g , as was the score variance; i . e . raters were more a l i k e . However, tr a i n i n g had no e f f e c t on halo. The evidence strongly suggests that appropriate t r a i n i n g of raters has a s i g n i f i c a n t e f f e c t on the rating c h a r a c t e r i s t i c s of BARS, as f i r s t suggested by Borman and Dunnette [1975]. As they put i t : Raters are seldom s k i l l e d i n making systematic work-related behavior observations. They need to become adept at observing and recording relevant job behaviors so that they may be better equipped with the information necessary for. making accurate evaluations of employee performance. [p. 5 65] Other Factors Cascio and Valenzi [1977] studied the e f f e c t s of two levels of r a t e r s ' and.ratees' job experience and education on BARS ratings. Although s t a t i s 1 1 c a l l y s i g n i f i c a n t differences were found, the differences were so weak and overlapping that the investigators concluded that the variables had no p r a c t i c a l e f f e c t s on the ratings. Finley, Osburn, Dubin and Jeanneret [1977] tested the impact of d i f f e r e n t scale formats on BARS ratings by contrasting disguised versus obvious scale formats, and s p e c i f i c anchors versus more general descriptions of behaviour. They found no consistent differences i n leniency or central tendency among the scales, but the o v e r a l l r e s u l t s suggest that the obvious scale format may be somewhat superior to the disguised format. The influence of a rater's and ratee's sex (and t r a i n i n g , see above) on BARS ratings were investigated by Norton, Gustafson, and Foster [1977]. The r e s u l t s showed that female raters were generally less lenient than male raters, and even more so a f t e r t r a i n i n g . One of the scales ( i n i t i a t i v e ) appeared to be sexually biased. Female ratees (described i n a case study which had male and female versions) were rated higher on th i s scale. Norton et a l . suggest that when confronted with the same l e v e l of i n i t i a t i v e the raters may have rated the female subject higher than the male on t h i s scale because i n our culture i n i t i a t i v e t r a d i t i o n a l l y has been seen as a more masculine c h a r a c t e r i s t i c . The other scales were unaffected by the sex of the ratee. Optimal Number of Job Dimensions Kafry, Jacobs, and Zedeck [1977] r a i s e the i n t r i g u i n g question whether there i s an optimal number of scale dimensions for BARS or ratings scales i n general. As Kafry et a l . suggest, one of the reasons why BARS demonstrate a r e l a t i v e l y high degree of halo may be that a rater's a b i l i t y to judge according to d i f f e r e n t dimensions i s limi t e d , perhaps to a number from 5 to 10. Another reason could be that a "true" relationship exists between performance dimensions. Previous studies on BARS generated between 3 [Dickinson and Tice, 1973] and 24 dimensions [Zedeck and Kafry, 1975]. However, when factor analysis was used to determine the number of n o n - t r i v i a l factors, a number between four and seven seemed to be dominant [Campbell et a l . , 1973; Finley et a l . , 1977; Kafry, Jacobs and Zedeck, 1977; Keaveny and McGann, 1975; Palef and Steward, 1971]. This would be i n l i n e with M i l l e r ' s [1956] discussion of the "magical number seven", where he suggests that the human capacity for processing information clues may be limited to a number around seven, "plus or minus two". As Kafry et a l . point out, discovering the optimal number of dimensions may reduce the amount of time and e f f o r t required for the proper use of BARS, either through combination of several scales or through elimination of weak scales. This approach may have a s i g n i f i c a n t impact on the application of BARS, because the high investment of man-hours i n the development phase i s one of the major disadvantages of this type of scale [Borman and Dunnette, 19 75]. Modified BARS Instruments Borman [1976] created a modified version of the BARS instrument, the Behavior Summary Scales (BSS). Based on the c o l l e c t i o n of very s p e c i f i c behaviour incidents, broader, more general behaviour descriptions were developed. Borman argues that these general statements of performance would add perspective to the depiction of performance on BARS and, i f properly developed, may even be preferable to s p e c i f i c behavioural examples. He tested the new scales and found acceptable scale c h a r a c t e r i s t i c s . Unfortunately, no d i r e c t comparison with BARS was made. Palef and Steward [1971] combined the BARS approach with a scoring method developed by Bass [1956]. The l a t t e r method s p e c i f i c a l l y attempts to reduce leniency error. In addition to going through the BARS development procedures ( c o l l e c t i n g , categorizing, r e t r a n s l a t i n g , item rating) participants had to rate the degree of d e s i r a b i l i t y of each behaviour item. Based on thi s measurement special scoring weights were developed. The r e s u l t i n g scoring c h a r a c t e r i s t i c s showed a low degree of halo error and high d i s c r i m i n a b i l i t y between high and low performance groups. No comparisons with a second scale were made. Discussion Is the BARS instrument superior to other scales? Out of twenty-one comparative studies, twelve were po s i t i v e , s i x equivocal, and three were negative. Thus, a d e f i n i t e answer to the above question cannot yet be given but, on the other hand, i t can be pointed out that "on balance" BARS fared quite well. Is thi s "modest superi o r i t y " [Borman and Dunnette, 1975], enough to j u s t i f y the considerable costs to develop behavioural scales and are there other q u a l i t i e s which are i n favour of the scales? Before we try to answer these questions, some other items should be mentioned f i r s t . As Bernardin, Alvares and Cranny [1976] j u s t l y point out, most comparisons between BARS and other scales were less than f a i r . When comparable rigorous development procedures were employed, BARS and Summated Rating Scales (SRS) were either equal [Bernardin, 1976] or SRS were superior [Bernardin, Alvares and Cranny, 1976]. If we accept, these findings as generalizable, i t follows that for psychometric c h a r a c t e r i s t i c s there i s no reason to prefer one type of scale to the other. The decision of which one to choose could be based on a v a i l a b i l i t y or convenience, or other factors. Even spe c i a l t r a i n i n g for raters does not seem to lead to any d i f f e r e n t i a l advantage for eit h e r scale, since i t has been shown that ratings based on d i f f e r e n t instruments are p o s i t i v e l y influenced by exposing raters to instructions on psychometric problems of performance rating [Bernardin and Walter, 19 77.; Borman, 1975; Latham et a l . , 1975; and Norton et a l . , 1977] . Then, are there other q u a l i t i e s which are i n favour of BARS? When we look at the comments i n the conclusions of most researchers who used BARS and/or compared them to other scales, the answer has to be p o s i t i v e . Several researchers mention the " i n t u i t i v e appeal" of BARS [Bernardin, LaShells, Smith and Alvares, 1976; Bernardin and Walter, 1977; Zedeck, Jacobs, and Kafry, 1976] which, i n most cases, i s not a s u f f i c i e n t reason to adopt a cert a i n type of scale. Borman [1976] expresses his pos i t i v e attitudes toward BARS i n stronger terms; he suggests that "Behavior scaling methodology...has provided a unique and seductively elegant strategy for developing job performance rating scales" [p. 2]. Other investigators are more s p e c i f i c i n t h e i r evaluation. Bernardin and Walter [1977] suggest that with BARS ratees receive s p e c i f i c examples of t h e i r perceived good and bad behaviours. In t h e i r opinion, t h i s c h a r a c t e r i s t i c i s very important, since the major purpose of most appraisal systems i s to ultimately improve performance. I f t h i s i s the case, then s p e c i f i c behavioural feedback on one's performance would be preferable to mere numbers. Quite e x p l i c i t i n t h e i r opinion are Landy, Farr, Saal and Freytag [1976] : Behaviorally anchored scales possess some properties not possessed by the more commonly used graphic rating scales. The most important of these properties i s the potential for counseling and feedback. ...Furthermore, i t appears as i f behaviorally anchored scales can be developed i n one setting and e f f e c t i v e l y used i n other settings. [p. 757] They also suggest that one of the major objections to the construction of BARS, the excessive time and expense involved, can be eliminated by careful use of resources. After finding only "modest sup e r i o r i t y " for BARS as compared to t r a i t - o r i e n t e d scales, Borman and Dunnette [1975] concluded: One would be better o f f , perhaps, to argue the usefulness of the behavior scaling methodology... on the wealth of information about job requirements and job performance yielded by the technique.... [p. 565] 75 This i s i n l i n e with Campbell et a l . ' s [1973] recommendation that: ...psychologists should be trying to measure and predict major dimensions of performance rather than effectiveness, since a measure of effectiveness i s one or more steps removed from what the i n d i v i d u a l actually does. [p. 71, emphasis added] Si m i l a r l y , Schwab et a l . [1975] hypothesize: ...the major advantage of BARS may stem from the dimensions generated rather than from any p a r t i c u l a r superiority provided by behavioral versus numerical anchors. [p. 559] Keaveny and McGann [1975] point to a s p e c i f i c area where BARS could be p o t e n t i a l l y h e l p f u l to personnel managers: In practice, performance evaluations are frequently contaminated by three types of rater error: leniency, halo, and imprecision. The cost associated with f a i l u r e to remove these errors has increased. There i s an increasing trend among white-collar employees to dispute discharges. Employers expose themselves to charges of discrimination and c i v i l s u i ts because of f a u l t y performance evaluations ("Rough going", 1975). This suggests that i n circumstances where key decisions are based largely or e n t i r e l y on the r e s u l t s of ratings, for example, wage and salary or promotion and discharge decisions, the extra e f f o r t involved i n developing behavioral examples to anchor the rating scales may be j u s t i f i e d . [p. 702] The same investigators [1977] used BARS to have students assess teaching effectiveness of college professors and compared the scale to two types of graphic scales. BARS showed the lea s t contamination with respect to school year, major, current GPA, current grade, and expected f i n a l grade. I f these findings w i l l hold up i n r e p l i c a t i o n studies, a t . l e a s t i n the area of teaching evaluation, BARS may be superior to other scales. In a more recent study, Keaveny and McGann [197 8] evaluated BARS and two types of graphic scales as to the i r e f f e c t on role c l a r i t y for university i n s t r u c t o r s . The dimensions tested were: understanding of the performance dimension rated; understanding where the ratee stands on each performance dimension; and understanding behavioural"changes which would improve ratings on each performance dimension. BARS were found to be superior to the other rating scale; formats on each dimension of evaluation. Blood [1974] perceives several b e n e f i c i a l c h a r a c t e r i s t i c s of BARS. According to him, behavioural expectation procedures can be used to broaden the domain of performance evaluation by including d i f f e r e n t perspectives of the job, e.g. from superiors, peers, subordinates, and c l i e n t s . BARS.may aid tr a i n i n g managers to determine t r a i n i n g requirements by pointing to s p e c i f i c behavioural shortcomings of an employee. Thirdly, BARS may enable managers to assess t h e i r agreement on organizational p o l i c i e s by analyzing c r i t i c a l incidents with a large rating variance. And l a s t l y , BARS may determine the accuracy of i n t r a -organizational communication.of p o l i c i e s by looking at item ratings from d i f f e r e n t organizational l e v e l s (see also M i l l a r d , Luthans, and Ottemann, 1976). As demonstrated, i n the opinion of many researchers BARS seem to have some very desirable properties which other normally used scales do not have. Even i f only a few of these c h a r a c t e r i s t i c s w i l l hold up i n future research, i n many cases i t may be worthwhile for researchers and personnel pr a c t i t i o n e r s to invest time and e f f o r t i n the development of BARS. A Summary of Literature Reviews on Training Evaluation When skimming the l i t e r a t u r e on i n d u s t r i a l t r a i n i n g and managerial development, and counting the number of a r t i c l e s , one has the impression that the subject of t r a i n i n g evaluation i s uppermost on the mind of t r a i n i n g directors and researchers i n t h i s f i e l d . During the period between 1955 and 1974,some 70 a r t i c l e s were written, dealing exclusively with the problems of evluating t r a i n i n g programmes; i n addition, more than 20 books have been published which deal either with evaluation research only or devote a f u l l chapter to the subject. However, i f one analyzes the publications and looks for concrete results i n the form of controlled research studies and theory developments, the disappointment i s great. Most of the publications urge trainers to measure the outcome of t h e i r t r a i n i n g programmes i n order to obtain feedback regarding the effectiveness of t h e i r e f f o r t s , but l i t t l e or no help i s provided on how to proceed or what instruments to use. (See also Campbell, 1971, pp. 365-66 and 594; Campbell et a l . , 1970, p. 471.) Esp e c i a l l y serious seems to be the lack of proper evaluation instruments. If any evaluation i s done at a l l then i t i s usually of the "How did you l i k e i t ? " v a r i e t y . In order to be useful a t r a i n i n g evaluation has to take into account the objectives of the programme. The objectives have to be operationalized and an appropriate instrument has to be selected or, i f necessary, to be developed. We w i l l b r i e f l y look into the underlying theory of measuring t r a i n i n g effectiveness, summarize several l i t e r a t u r e reviews on the same topic, and draw some conclusions from the findings. Underlying Theory When i s a management t r a i n i n g and development programme eff e c t i v e ? Is i t e f f e c t i v e when the participants " l i k e d " the programme? Is i t e f f e c t i v e when the participants evidently have gained i n knowledge, changed their attitudes, or improved t h e i r job related behaviour? Or, i s i t only effective: when the programme pays o f f i n the form of higher p r o f i t s ? The answer i s simple: A programme i s e f f e c t i v e when i t has achieved i t s goals. The important point i n t h i s d e f i n i t i o n i s the implication that the c r i t e r i o n for evaluating the effectiveness of a programme i s already b u i l t into the plan.for the programme. Robert F. Mager [1962] i n h i s booklet Preparing Instructional Objectives, makes a strong plea for a clear statement of course objectives: When c l e a r l y defined goals are lacking, i t i s impossible to evaluate a course or program e f f i c i e n t l y , and there i s no sound basis for selecting appropriate materials, content, or i n s t r u c t i o n a l methods. After a l l , the machinist does not select a tool u n t i l he knows what operation he intends to perform. [p. 3] And he c i t e s another author, Paul Whitmore: "The statement of objectives of a t r a i n i n g program must denote measurable attributes observable i n the graduate of the program, or otherwise i t i s impossible to determine whether or not the program i s meeting the objectives." [Ibid.]- 2 In order to state meaningful c r i t e r i a for an appropriate t r a i n i n g programme we have to know what i d e n t i f i e s an e f f e c t i v e manager. Guion [1965, p. 466] o f f e r s a description of what an e f f e c t i v e manager has to do: The success of an executive l i e s l a r g e l y i n meeting major organization goals through the coordinated e f f o r t s of h i s organization; i n part, at l e a s t , these e f f o r t s depend upon the kind of influence the executive has upon those whose work, his own behavior, ^touches.. . The executive's own behavior contributes to the achievement of organizational goals only by i t s influence on the perceptions, attitudes, and motives of other people i n the organization and on t h e i r subsequent behavior. -Unfortunately, Mager does not give a reference for the quote. Campbell, et a l . [1970, p. 105] l i k e Guion, stress the point that a manager's behaviour i s the basis for his effectiveness: We define e f f e c t i v e managerial job behavior as any set of managerial actions believed to be optimal for i d e n t i f y i n g , assimilating, and u t i l i z i n g both i n t e r n a l and external resources toward sustaining, over the long term, the functioning of the organizational unit for which a manager has some degree of r e s p o n s i b i l i t y . As an i l l u s t r a t i o n of the kind of factors and th e i r i n t e r a c t i o n which determine managerial behaviour, Campbell, et a l . o f f e r a model [p. 11] which includes such concepts as 1. i n d i v i d u a l c h a r a c t e r i s t i c s 2. organizational environment 3. organizational r e s u l t s . (Figure 1) Training i s part of the "organizational environment" i n t h i s model. I t could be viewed as an opportunity offered by the organization to improve a manager's a b i l i t i e s and perhaps even his motivation. I t follows that, i n order to make less e f f e c t i v e managers more effective,, a successful t r a i n i n g programme has to change a manager's job behaviour. And i n order to assess any impact of a t r a i n i n g programme on such a behaviour an instrument i s needed which measures actual behaviour change. FIGURE I I I . l ORGANIZATIONAL ENVIRONMENT V I n d i v i d u a l c h a r a c t e r i s t i c s I n t e l l i g e n c e Aptitudes Knowledge Temperament Preferences Expectations f Job behavior f ( A b i l i t y , motivation opportunity) feedback Organizational r e s u l t s P r o f i t • maximization Organizational e f f i c i e n c y High p r o d u c t i v i t y feedback PERSON PROCESS PRODUCT SOURCE: Campbell, J.P., Dunnette, M.D., Lawler, E.E., and Weick, K.E., Managerial Behavior, Performance and E f f e c t i v e n e s s , New York: McGraw-Hill Book Co., 19 70, p. 11. oo N3 The following l i t e r a t u r e review i s a summary of several reviews of publications i n the f i e l d of t r a i n i n g research and s p e c i f i c a l l y t r a i n i n g evaluation. Summary Review John P. Campbell expressed his f r u s t r a t i o n with the l i t e r a t u r e on Personnel Training and Development i n a chapter i n the 1971 Annual Review of Psychology by saying: "...the t r a i n i n g and development l i t e r a t u r e i s voluminous, non-empirical, nontheoretical, poorly written and d u l l . " [p. 565] He attacked s p e c i f i c a l l y the t r a i n i n g evaluation methodology which, i n his opinion, c r i e s for innovation. He suggested to forget about the either/or approach to tr a i n i n g evaluation which would imply a dichotomous outcome, i . e . , either the program has value or i t doesn't. According to Campbell, i t would be much more important to be concerned with ways to measure behavioural outcomes and t h e i r i n t e r a c t i o n with other subsystems i n the organization, and the d i f f e r e n t i a l e f f e c t s of competing t r a i n i n g strategies. Campbell's negative impression i s i n l i n e with the findings by Catalanello and Kirkpatrick [1968] who surveyed 110 firms to investigate the evaluation, methods used to measure the effectiveness of training, and development programmes. The majority of the companies r e l i e d on reaction measurements ("How did you l i k e the program?") while only a few t r i e d to measure behaviour or organizational r e s u l t s . Catalanello and Kirkpatrick concluded that t r a i n i n g research i s s t i l l i n i t s "infancy". A thorough review of the l i t e r a t u r e was done by Campbell, et a l . [1970] i n the f i e l d of management development. They analysed 84 studies with respect to methodology, instrumentation and outcomes and found that more than half of the studies used no, or only some, controls and that two-thirds used " i n t e r n a l " c r i t e r i a , l i k e reaction, knowledge or attitudes. Campbell et al... strongly recommended the use of "external" c r i t e r i a , such as behaviour change, to measure the outcome of any t r a i n i n g programme. Using this recommendation, Schwind [1975] reviewed the l i t e r a t u r e on Training Evaluation and found that out of 71 a r t i c l e s dealing with t h i s topic, only two described studies which were methodologically sound (e.g., random sampling, pre- and post-test, control group) and u t i l i z e d instruments measuring actual behaviour change. One of the studies was done by Schwartz, S t i l w e l l , and Scanlan [1968], using the Leader Behavior Description Questionnaire and the C r i t i c a l Incident Method to evaluate the short and long range ef f e c t s of a management development programme. The study methodology was an i n s t i t u t i o n a l cycle design. (One group receives treatment while a second group serves as a control group. This cycle i s repeated with reversed roles.) The c r i t i c a l incidents were gathered through an open-ended interview procedure which asked the participants to describe incidents which represented examples of successful and unsuccessful.experiences. Then the incidents were analysed according to the question: "Were the incidents handled better by the trained than by the untrained group?" Unfortunately, Schwartz et a l . off e r no information on the v a l i d i t y of t h e i r instrument, nor do they report any other psychometric c h a r a c t e r i s t i c s . Their use of the LBD Questionnaire i s at least of questionable value, since this instrument i s based on very general leader behaviour descriptions, and i s not at a l l job s p e c i f i c , one of the major requirements for assessing managerial effectiveness [Campbell et a l . , 1970, p. 106]. The second study, Wallace and Weitzel [1972], used for the f i r s t time the C r i t i c a l Incident Method and- -the Retranslation Method together for the purpose of a t r a i n i n g evaluation i n co-operation with a large u t i l i t y firm i n the U.S. In order to generate a pool of c r i t i c a l incidents they interviewed experienced l i n e managers. Ninety-eight incidents were divided into 13 categories c i t e d by the managers. Two randomly selected judges then t r i e d to "retranslate" the items, i . e . , sort the items into the categories (behaviour dimensions) that they most c l o s e l y represented. In addition the items were rank-ordered from the most c r i t i c a l i n terms of success to the most c r i t i c a l i n terms of f a i l u r e . Fifty-seven of the ninety-eight items survived and were declared v a l i d . Wallace and Weitzel did not report any r e l i a b i l i t y data. They d i d , however, demonstrate content v a l i d i t y by having experts (job incumbents) validate each item, and construct v a l i d i t y by applying the instrument i n a f i e l d study [Wallace and Weitzel, 1970], On the other hand, t h e i r procedure to generate these items i s unusual. Instead of asking for actually observed behaviour events they had job incumbents define goals or objectives of t h e i r job and then describe actions to achieve these goals. There i s a question as to how c l o s e l y these "imagined" c r i t i c a l behaviours match exhibited behaviour. - F i n a l l y , Wallace and Weitzel u t i l i z e d only, job incumbents and did not ask superiors and/or subordinates for t h e i r aspects of the job incumbent's behaviour, as suggested by Campbell et a l . [1970, p. 115].. 87 Nevertheless, t h i s study i s important for several reasons. I t demonstrated that measurement and experimental rigor are feasible i n evaluating management t r a i n i n g programmes. I t also showed that the u t i l i t y ' s t r a i n i n g programme was not very successful. One might speculate what the r e s u l t would have been had the investigators asked how the participants " l i k e d " the programme. The conclusions that can be drawn from the above-mentioned reviews are that the measurement of t r a i n i n g effectiveness, as i t i s presently practised i n organizations has two major shortcomings: 1. lack of adequate methodology 2. lack of appropriate measuring instruments. The f i r s t problem can be remedied only by either having s c i e n t i s t s conduct the evaluation research or by having t r a i n i n g personnel acquire the necessary expertise and then do a sound evaluation of t h e i r own t r a i n i n g e f f o r t s . The second problem rests mainly with s o c i a l s c i e n t i s t s since i t requires f a m i l i a r i t y with the problems of c r i t e r i o n development and v a l i d a t i o n . Because of t h i s , one may reason that i t i s the s c i e n t i s t s ' r e s p o n s i b i l i t y to provide p r a c t i t i o n e r s with the appropriate tools for tra i n i n g programme evaluation i n the same way they develop psychological tests for h i r i n g purposes. So f a r , as discussed, two attempts were made i n the area of t r a i n i n g evaluation to u t i l i z e scales which assess behaviour change. However, those investigations did not report on the rating c h a r a c t e r i s t i c s of the instruments used, nor were there any comparisons with other scales to f i n d out whether there are perhaps more appropriate methods to assess outcomes of t r a i n i n g and development programmes. The only conclusion which can be drawn i s that behaviourally oriented scales seem to be quite promising for the purpose of t r a i n i n g evaluation. The following chapter w i l l discuss the need for improved instruments i n the areas of performance appraisal and programme assessment with special emphasis on behaviour oriented scales, and, secondly, i t w i l l outline more s p e c i f i c a l l y the objectives of t h i s study. 89 Need for Improvement of BARS The discussion so far seems to indicate that with regard to certa i n psychometric problems, such as halo, leniency, central tendency, and r e l i a b i l i t y , the BARS instrument i s not the ultimate solution. Only when the focus i s on s p e c i f i c feedback on shortcomings with regard to job performance, t r a i n i n g d e f i c i e n c i e s and tra i n i n g r e s u l t s , do BARS — according to.some researchers -- seem to have certa i n advantages. One of the major disadvantages of BARS i s the amount of time and e f f o r t required to develop the instrument. The problem i s exacerbated by the fact that the instrument had to be developed for each job i n an organization. Recently Goodale and Burke [1975] demonstrated that i t may be possible to create BARS for d i f f e r e n t jobs i n the same organization. I t would enhance the a p p l i c a b i l i t y and economic u t i l i t y of BARS even more i f an instrument could be developed which could be used for similar jobs i n d i f f e r e n t organizations. Therefore, one of the questions which w i l l be asked i n this study i s : i s i t possible to develop a single BARS instrument to assess si m i l a r jobs i n d i f f e r e n t organizations? As mentioned above, another area i n which the BARS instrument needs improvement i s i t s r a t i n g c h a r a c t e r i s t i c s , i . e . the eff e c t s of halo, leniency, central tendency, and i n t e r r a t e r r e l i a b i l i t y . As has been shown i n the previous chapters of thi s paper, several d i f f e r e n t approaches have been t r i e d for improvement of BARS, such as t r a i n i n g , p a r t i c i p a t i o n i n scale development, various development procedures and formats, but the main findings suggest that what worked for BARS worked " also for other scales. I t appears that i n order to improve the rating c h a r a c t e r i s t i c s of BARS, either the scale format of BARS or the rating procedures or both require modification. How thi s may be accomplished w i l l be discussed l a t e r i n thi s chapter. I t i s also claimed that the often expressed view that BARS provide very s p e c i f i c feedback to ratees [Campbell et a l . , 1970; Zedeck et a l . , 1974., among others]' i s not warranted, or at l e a s t that the value of t h i s feedback i s lower than may be expected. What s p e c i f i c information does a ratee receive when he i s told that his performance i n a certa i n job dimension i s "very i n e f f e c t i v e " because according to the judgment of h is superior his "expected" behaviour had been equated with a s p e c i f i c behaviour description anchored on the lower end of a BARS scale? Even i f the ratee knows the s p e c i f i c behavioural anchor i t may not be an i d e n t i f i a b l e job behaviour to him, because these samples "...represent not actual observed behaviours but inferences and predictions from observations" [Smith and Kendall, 1963, p. 150]. Borman expressed a similar view when he wrote: Unfortunately, these comparison and f i t t i n g steps [as required by BARS] are d i f f i c u l t for most raters because often they have a hard time discerning any behavioral s i m i l a r i t y between a ratee's performance and the highly s p e c i f i c behavioral examples, used to anchor the scale. [Borman, 19 76, p. 3] It i s suggested — and w i l l be shown — that the presently used BARS format severely l i m i t s the feedback value for performance appraisal and tr a i n i n g need determination purposes. S i m i l a r l y , i t i s suggested that the information value of a performance rating based on BARS for promotional purposes may be.quite li m i t e d since the simple " t y p i c a l " behaviour item used as a c r i t e r i o n with BARS may have l i t t l e relevance to the pot e n t i a l (future) performance which should be assessed to determine promotability as recommended by Glueck [1974, p. 288]. Of relevance here are the findings of studies done by H. D. Meyer [1963, 1965a, 1965b] regarding.factors r e l a t i n g to promotion. He showed that managers who exhibited ce r t a i n behaviours were more l i k e l y to be promoted than managers who did not demonstrate these behaviours (e.g., delegation of r e s p o n s i b i l i t i e s , f l e x i b i l i t y with d e t a i l s , showing less anxiety, being less manipulative of people, etc.) [H. D. Meyer, 1965b, pp. 6-7; Chruden and Sherman, 1976, p. 346]. Given the fact that BARS u t i l i z e only between 7 and 9 behaviour samples per dimension out of a much larger item pool, the chance that a scale contains many job behaviours relevant to future performance i s small indeed. I f a rater wants to assess a subordinate's promotability, he s t i l l has. to think of the many ef f e c t i v e and i n e f f e c t i v e behaviour events not included i n the scale but exhibited by job incumbents and po t e n t i a l l y relevant to job behaviour required i n an advanced p o s i t i o n . I t may be possible to develop a BARS instrument s p e c i f i c a l l y for promotional purposes, u t i l i z i n g anchors consisting of behaviour items relevant to promotability. But this has not been t r i e d yet. There remains the vexing question of how to avoid the waste of valuable information with the use of BARS, or, to put i t d i f f e r e n t l y , how to increase the information content of the presently used BARS formats. A t y p i c a l example of t h i s problem i s the Goodale/Burke [1975] study mentioned before. Out of 360 incidents generated, less than 70 were retained; or less than 19%. As Schwab et a l . [1975] point out: . . . i f one assumes that the o r i g i n a l pool of incidents ... represents behaviors that an evaluator may see and assess i n an applied setting, instruments defined and anchored by r e l a t i v e l y few examples could create at l e a s t two problems. F i r s t , the evaluator may have d i f f i c u l t y assigning observed behaviors to s p e c i f i c dimensions. Second, the evaluator may have d i f f i c u l t y deciding the scale value of effectiveness of the observed behavior against the examples provided. Both of these problems would obviously be p o t e n t i a l sources of error variance. [p. 558] But how i s i t possible to increase the information content of BARS? Zedeck, Jacobs, and Kafry [1976] made an attempt to salvage at l e a s t part of the discarded behaviour samples by developing p a r a l l e l forms of BARS for the same target group. However, the main purpose of t h e i r experiment was to improve the opportunities to measure the psychometric c h a r a c t e r i s t i c s of BARS, e.g. r e l i a b i l i t y , agreement among raters on anchor points, etc., but not to increase the information content. The u t i l i z a t i o n of longer scales (e.g. 1 to 15) instead of the usual 1 to 7,. or 1 to 9 scale points may be a possible solution, but t h i s method may run into the problem that raters are possibly l i m i t e d i n t h e i r c a p a b i l i t y to d i s t i n g u i s h between more than 7 or 9 information cues.l (See M i l l e r , 1956. Of relevance here i s also the study by Lissetz and Green [1975] on the optimal number of scale points with regard to r e l i a b i l i t y . ) Given the pr e v a i l i n g BARS format with 7 to 9 anchor points, there seem to be limited p o s s i b i l i t i e s for improving the instrument with respect to information content. This raises the question whether the scale format can be changed i n such a way as to accommodate more behaviour incidents without v i o l a t i n g M i l l e r ' s recommendation to l i m i t measuring scales to 5,. 7 or 9 items. Conceivably, t h i s could change the basic structure of the BARS instrument, which i s based on the assumption that the use of behavioural anchors reduces the ambiguity c h a r a c t e r i s t i c of t r a d i t i o n a l (e.g. GRS) rating scales by giving the rater s p e c i f i c behavioural cues from which generalizations about ratee behaviour can be made [Dunnette, 1966;. Smith and Kendall, 1963], DeCotiis [1977], who compared BARS with other instruments by c o r r e l a t i n g r a t i n g scores with an independent c r i t e r i o n , cautions against t h i s view: The most tenuous aspect of this assumption i s that raters can accurately generalize from a single example of behavior (a scale anchor) to an array of job behaviours (a performance l e v e l ) . The res u l t s of the present study suggest that t h i s assumption i s unwarranted even when raters are provided with substantial opportunities to observe the behavior they are to evaluate. [p. 264] and he continues: i t may be that raters perceive job performance as a configuration or gest a l t of behavior and not as an event.... If so, one would expect a rater to experience d i f f i c u l t y i n generalizing from a s p e c i f i c behavioral anchor to t y p i c a l ratee behavior. [p. 265] I t should be pointed out that DeCotiis acknowledges the laboratory nature of hi s study i n which raters rated vignettes instead of r e a l people, thus l i m i t i n g the g e n e r a l i z a b i l i t y of the r e s u l t s . Nevertheless, the implications are in t e r e s t i n g and give reason to question the above-mentioned assumptions made by Dunnette [1966] and Smith and Kendall [1963]. If ratees have d i f f i c u l t i e s i n generalizing from a single example of job behaviour to an: array of job behaviours, then why not discard t h i s questionable inference and replace i t with a d i r e c t evaluation of demonstrated e f f e c t i v e or i n e f f e c t i v e job behaviours, i . e . ask raters whether they actually observed the ratee exhibit desirable or undesirable job behaviours? Raters could be aided i n th e i r rating procedure by providing them with l i s t s of e f f e c t i v e and i n e f f e c t i v e behaviour items. These l i s t s could be developed i n a similar rigorous way as the BARS instrument, i . e . c o l l e c t i o n of items, v a l i d a t i o n , and re t r a n s l a t i o n , but without the problem of l i m i t i n g the number of scale items to 7 or 9. By not having the behaviour items rated as to the i r degree of effectiveness as i t i s presently part of the development procedures for BARS, a certain.amount of information i s l o s t , i . e . degree of effectiveness of each item. However, i t i s believed that more information w i l l be gained by increasing the number of scale items than by requiring a rater to equate a given pre-rated behaviour sample (scale anchor) with the "most t y p i c a l " behaviour of a ratee. 3 Given the above described shortcomings of the BARS instrument on the one hand, and the sought after, more desirable c h a r a c t e r i s t i c s on the other, there seems to be a l o g i c a l conclusion: develop an instrument whose scale format permits the u t i l i z a t i o n of more job behaviour samples, and whose rating procedures avoid or reduce the pr o b a b i l i t y of rating errors. How t h i s may be accomplished w i l l be outlined i n the res t of the chapter which w i l l discuss the c h a r a c t e r i s t i c s of a new performance appraisal and tr a i n i n g evaluation instrument, incorporating some of the changes suggested above. To summarize the shortcomings of BARS and emphasize the need for improvements, Table II.I-7 l i s t s the problems encountered with BARS, points to possible causes, and outlines possible solutions. 3 I t i s also believed that any use of scales with a "good-bad" continuum such as GRS, SRS, and BARS w i l l r e s u l t i n higher degrees of halo, leniency, and central tendency. However, t h i s w i l l not be part of the present investigation. Table III-7 97 Shortcomings of BARS Possible Causes Suggested. Changes for Improvement High Halo Low dimension independence Single choice among multi dimens i o n a l behaviour items Global (gestalt) view of performance by raters Interaction of above factors More s p e c i f i c c r i t e r i a through higher s e l e c t i o n standards. Remove choice r e s t r i c t i o n . More s p e c i f i c performance c r i t e r i a and rater t r a i n i n g . Combination of above solutions High Leniency Large number of anchors Attitudes of raters (leniency) Single choice among multidimensional behaviour items Interaction of above factors "Yes-No" and "?" anchor. Remove "good-bad" connotation by using "Yes-No" anchors. Remove choice r e s t r i c t i o n . Combination of above solutions High Central Tendency Attitudes of raters and extreme anchors on "good-bad" continuum Avoid extreme scale anchors. Low Interrater R e l i a b i l i t y Single choice among multidimensional behaviour items Use of raters from d i f f e r e n t organizational l e v e l s Remove choice r e s t r i c t i o n s , increase number of items. Use of raters from same organizational l e v e l (to assess i n t e r r a t e r r e l i a b i l i t y ) Low Information Content for Performance Appraisal and Training Need Determination Purposes 7 to 9 anchor format Increase number of behaviour items Low Information Value for Feedback and Counselling Purposes One " t y p i c a l " behaviour item per job dimension Use of a l a r g e r number of items to cover the t o t a l job behaviour domain. Low Information Value for Promotion Purposes One " t y p i c a l " behaviour item per job dimension Use of a larger number of items relevant to future performance High Development Costs Time and e f f o r t required for scale development Restricted to s i m i l a r jobs i n same organizations Simplify procedures ( i f p o s s i b l e ) . Develop scales applicable to s i m i l a r job i n d i f f e r e n t organizations (economy of s c a l e ) . 98 Proposal for a New Performance Appraisal and Training Evaluation Instrument The proposed new instrument (Appendix D), c a l l e d the Behaviour Description Index (BDI), employs a format sim i l a r to Smith, Kendall, and Hulin's [1969] Job Description Index (JDI) k (for example see Appendix E). Instead of the currently common u t i l i z a t i o n of only 7 to 9 behaviour descriptions per job dimension i n a BARS instrument, a larger sample of the t o t a l behaviour domain of a job dimension i s used i n the same way objectives are used i n the JDI scales. I f , for example, the t o t a l number of retained c r i t i c a l incidents were 5 0 per job dimension, i t i s t h e o r e t i c a l l y possible to use a l l 50 i n one scale. However, since a job often has more than 10 dimensions, the evaluation of 500 items would be very cumbersome and fatiguing. For p r a c t i c a l purposes, then, i t i s proposed to l i m i t the number of items to 20 per dimension. This number i s a r b i t r a r y and can be changed i f future research should suggest otherwise. Examples of e f f e c t i v e and in e f f e c t i v e behaviour are l i s t e d i n random order. "*The JDI i s a measure of f i v e areas of Job S a t i s f a c t i o n : Work I t s e l f , Supervision, Peers, Pay, and Promotion. A Comparison Between BDI and BARS Rating Procedure for the BDI Raters who use the BDI.' respond to the question: "Does th i s employee exhi b i t the following l i s t e d behaviours consistently?" In the present study a l i s t of 2 0 behaviour samples per dimension i s used (see Appendix D), but i t may be remembered that t h i s number i s a r b i t r a r y . Similar to the rating procedure of the Job Description Index, the BDI provides raters with three response choices: Yes, No, and Uncertain, to be written as "Y", "N", and "?" i n check-off squares which appear after every item. 5 These responses may be l a t e r transformed (preferably by a d i f f e r e n t person or a computer/ for reasons explained below) into point scores according to the following rules: If a rater responds to a posi t i v e (effective) behaviour sample p o s i t i v e l y (Yes), i . e . e f f e c t i v e behaviour i s exhibited 3 points If a rater responds to a negative ( i n e f f e c t i v e behaviour sample negatively (No), i . e . i n e f f e c t i v e behaviour i s not exhibited .... 3 points If a rater responds vice versa, i . e . negatively to a p o s i t i v e behaviour or p o s i t i v e l y to a negative behaviour 0 points If a rater responds with "?" (uncertain) .. 1 point 5Tb quote Smith et a l . [1963]: "This format i s used to minimize response sets which are more l i k e l y to aris e i f response alternatives are printed i n a fixed order on the page." [p. 69] In the l a s t case i t i s assumed that when a rater i s uncertain whether a ratee exhibited a cer t a i n behaviour or not, i t indicates a somewhat lower p r o b a b i l i t y that the ratee w i l l e x h i b i t the.desirable behaviour (or vice versa). This i s i n l i n e with the empirical findings by Smith et a l . [1969], who concluded that an "uncertain" response was more in d i c a t i v e of a negative response than a p o s i t i v e one [p. 79]. As mentioned, the rater responds only with yes, no, or uncertain, and usually w i l l not convert these responses into numerical values himself. I t i s suggested that this i s done by another person or a computer i n order to avoid any hint to the rater about the point value of the evaluation. I t i s assumed that these procedures w i l l reduce leniency, because the rater does not know how high or low.the ratee scored per dimension. This should also reduce halo error, because when a rater i s uncertain about the t o t a l score on a dimension he i s probably less able to transfer h i s evaluation to other dimensions. Thirdly, i t should reduce or eliminate central tendency, because the extreme anchors of a continuous scale (e.g. extremely e f f e c t i v e , very ineffective;, or outstanding, very poor) are avoided. An additional factor which may help to reduce lenience with the BDI instrument i s the avoidance of a neutral scale point (such as the fourth anchor of a seven point BARS instrument) which i s believed to contribute to a rater's tendency toward more favourable ratings i n performance ap p r a i s a l s . 6 The p o s s i b i l i t y for converting Yes/No/? responses to numerical scores should prove to be quite h e l p f u l to a personnel and/or t r a i n i n g manager. For example, i t would be possible to determine i n advance what scores would be acceptable or unacceptable as a r e s u l t of a performance appraisal or an assessment of tr a i n i n g needs, or, to put i t d i f f e r e n t l y , standards or norm scores could be established. Since the maximal number of score points per dimension i s 60 (20 items x 3 points) 0 - 3 0 may mean urgent t r a i n i n g required (or, i f measured afte r t r a i n i n g ; t r a i n i n g i n e f f e c t i v e ) 31 - 40 may mean tr a i n i n g recommended 41 - 50 may mean refresher course may be h e l p f u l 51 - 60 may mean no tr a i n i n g required. Another p o s s i b i l i t y would be to determine the degree of effectiveness by ca l c u l a t i n g the percentage of exhibited desirable (and not exhibited undesirable) job behaviours based on the t o t a l number of l i s t e d 6 T h i s assumption i s based on a personal experience by the investigator. Appraisers (managers) mentioned that to them "average" performance was unacceptable, i . e . had a negative connotation. behaviour samples, either per dimension or, as a global measure, for the job domain as a whole. These measures, of course, could serve only as rough guidelines. I t i s believed that the main advantage of the BDI as an appraisal instrument l i e s i n i t s enumeration of every e f f e c t i v e and very i n e f f e c t i v e behaviour, thus allowing personnel and t r a i n i n g managers a precise and detailed assessment of an employee's strengths and weaknesses, i n contrast to BARS where a score represents a single behaviour sample which has to be generalized to an employee's actually observed behaviour, or, perhaps worse, range of behaviour. Rating Procedures for BARS Unfortunately, there are no uniform procedures for the use of BARS. Smith and Kendall [1963] recommend that a rater f i r s t think of and record examples of a ratee's behaviour relevant to the BARS instrument i n question. Then he/she compares the l e v e l of performance represented by each observed example with the performance leve l s r e f l e c t e d by scaled behavioural anchors. The point on the scale where a " f i t " occurs; i . e . the scale point at which the l e v e l of performance associated with an observed example seems about the same as the l e v e l of performance represented by a behavioural example, determines the performance rating to be assigned. This procedure i s followed for each observed behaviour example, and the ratings are averaged to that scale. (See also Bernardin, 1977; Bernardin et a l . , 1976; Borman, 1975; Schneier, 1977.) A s l i g h t l y d i f f e r e n t approach i s suggested by Dunnette [1966], A rater i s instructed to think c a r e f u l l y of the ratee's t y p i c a l performance related to a behaviour scale and then to compare the l e v e l of that performance with the l e v e l of performance represented by the behavioural anchors. The rater makes a rating by judging where the observed performance " f i t s i n " with respect to the scaled behavioural example. A t h i r d approach i s used by Campbell et a l . [1973]. The rate r i s asked to use the scaled expectations procedure to rate both the " t y p i c a l " and "best" performance of each subject. I t i s assumed that t h i s combination w i l l reduce leniency error i n the t y p i c a l r a t i n g . Some other methods have been t r i e d [Landy et a l . , 1976; Norton et a l . , 1977], but the three examples described above sh6.uldii.be s u f f i c i e n t to show that there are d i f f e r e n t approaches to the use of BARS, and that some of the methods require considerable involvement on the side of the rater, e s p e c i a l l y the approach recommended by Smith and Kendall [19 63]. However, i t seems to be the most promising with regard to rater biases [Bernardin et a l . , 19 76]. Characteristics of BDI and BARS In contrast to the rather complex structure and rating procedures of the BARS instrument [Borman, 1975; Borman and Dunnette, 1975; Schneier, 1977(a) & (b); Schneier and Beatty, 1976; Schwab et a l . , 1975; Zedeck et a l . , 1976] i t i s believed that the BDI of f e r s a r e l a t i v e l y simple approach to performance r a t i n g . I t does not involve "awkward judgments" [Borman, 1975; p. 4], nor does i t burden the rater requiring "many s p e c i f i c judgments and fine discrimination i n the perception of complex and numerous job behaviors" [Schneier, 1977, p. 542, emphases added]. Although the BDI requires more judgements than a BARS instrument, these judgements are of the nature of simple Yes, No, and "?" responses and should c e r t a i n l y not require "cognitive complex raters" as recommended by Schneier [1977] for the use of BARS. As a matter of fa c t , the JDI, from which t h i s r a ting procedure was adopted, ...does not require that the respondent be able to make abstractions or understand long, vague sentences with several q u a l i f i c a t i o n s , but only that he understand the general meaning of single words or short phrases. [Smith et a l . , 1969, p.. 70] I t i s expected that, because the BDI combines the rigorous development process of BARS with the less complex rating procedure of the JDI, ratings based on the BDI instrument w i l l e x h i b i t less error variance than ratings based on the BARS format. The BDI does not use the expectations phraseology recommended by Smith and Kendall [1963] for the BARS format since i t seems to complicate the rating process: Use of t h i s expectation phraseology requires the rater to i n f e r job performance, to use an i m p l i c i t b e l i e f concerning the probable in t e r c o r r e l a t i o n s of sets of ratee behavior as the basis for assigning a score on a given dimension. [Dickinson and Tice, 1973, p. 422, emphases added] Instead, the descriptions of behaviour samples are u t i l i z e d i n the o r i g i n a l language as received from job incumbents during the development process. This process of d i r e c t l y matching observed and scaled behaviour should be easier for raters and should lead to more, accurate ratings than the i n f e r e n t i a l process discussed previously. I t i s also believed that a BDI format permits a more accurate assessment of an employee's l e v e l of performance than a BARS format. Ratings which r e f l e c t behaviour actually drawn from the desired behaviour domain of the job are assumed to be more s o l i d indicators of e f f e c t i v e performance than expectation ratings which are taken to r e p r e s e n t some g l o b a l d e s c r i p t i o n o f a c t u a l behaviour. The f i r s t measurement c o n s i s t s o f counting and checking o f f observed behaviour on a l i s t . w h i c h r e p r e s e n t s a s i g n i f i c a n t p a r t — i f not a l l — of a job behaviour domain, w h i l e the second measurement i s based on a mental a s s o c i a t i o n between a cue (from s c a l e anchors r e p r e s e n t i n g a s m a l l p a r t o f a behaviour domain) and a f r e e r e c a l l of observed behaviour. The f i r s t measure should be e a s i e r to r e p l i c a t e , r e s u l t i n g i n h i g h e r r e l i a b i l i t y o f a BDI instrument. The d i f f e r e n t s t r u c t u r e o f the two instruments i m p l i e s a l s o t h a t BARS and BDI scores have d i f f e r e n t meanings. The f o l l o w i n g example should i l l u s t r a t e t h i s p o i n t . I t i s assumed t h a t a r a t e r checks o f f item #3 on a seven or nine p o i n t BARS to i n d i c a t e t h a t i n h i s o p i n i o n the r a t e e e x h i b i t e d job behaviour w i t h the same l e v e l of performance as rep r e s e n t e d by the marked item. I t i s assumed, too, t h a t the same item i s a l s o p a r t o f a r e l e v a n t BDI s c a l e . The 3-point score on the BARS format a c t u a l l y means t h a t the r a t e e — i n the o p i n i o n o f the r a t e r — can be expected to e x h i b i t g e n e r a l l y moderately i n e f f e c t i v e job behaviour. The same r a t e r would respond to the e q u i v a l e n t item on the BDI format (without the e x p e c t a t i o n terminology) w i t h "Yes" (an i n e f f e c t i v e job behaviour was e x h i b i t e d ) . But there i t means t h a t the r a t e e a c t u a l l y d i d e x h i b i t a very s p e c i f i c i n e f f e c t i v e job behaviour consistently, with no inferences about any general l e v e l of job performance. In addition, the BDI response by i t s e l f has l i t t l e s i g nificance with respect to the o v e r a l l score which i s made up of 20 i n d i v i d u a l responses. The u t i l i z a t i o n of a large number of c r i t i c a l incidents with the BDI would overcome one of the major shortcomings of BARS, namely the waste of information. It i s conceivable that i n many instances 20 or 30 c r i t i c a l incidents w i l l encompass the t o t a l behaviour domain of a job or a job dimension. If i t i s not the case, then at least the scale items of a BDI instrument represent a much larger sample than the items on a BARS format. For example, i f the t o t a l domain consists of 50 items, then 30 items represent 60% of the domain as compared to 14% with a sample of 7. If t h i s i s true — and i t seems to be obvious — then most BARS instruments are wanting. As Campbell et a l . [1970] put i t : Many measures of job effectiveness—most p a r t i c u l a r l y many at the managerial l e v e l — are seriously d e f i c i e n t i n that they include only a few rather than a l l or many of the behavioral elements making up a job. [p. 107, double emphases added] More items per scale should also have a p o s i t i v e influence on i n t e r r a t e r and scale r e l i a b i l i t y [Cronbach, I960, p. 130; Kerlinger, 1973, p. 454]. This i s e s p e c i a l l y true i f we accept Borman's [1976] observation that a rater who uses BARS may experience d i f f i c u l t i e s i n a r r i v i n g at a judgement because of the comparison and f i t t i n g steps involved. This means, however, that a higher interpretative burden i s placed upon the rater which i n turn may influence r e l i a b i l i t y (and v a l i d i t y ) [Kerlinger, 1973, p. 538].. BDI scales, on the other hand, do not require any "comparison and f i t t i n g steps." They simply consist of a random sample of the job behaviour domain, with a provision for a check-off response to indicate whether a certain behaviour was exhibited or not, thus simplifying the rating process. The larger number of job behaviour samples of the BDI format should also improve the information value of a performance appraisal for feedback and counselling purposes. Probably one of the most important aspects of a performance appraisal process i s receiving feedback to the question asked by a subordinate: "How can I improve on my job?" The BDI scale should be p a r t i c u l a r l y useful i n providing an answer to this question because i t enables the superior to point out s p e c i f i c a l l y those behaviours the employee has to correct In order to show improved performance on the same job. I t i s a l s o the b e h a v i o u r a l s p e c i f i c i t y o f the BDI s c a l e t h a t bears d i r e c t l y upon the i d e n t i f i c a t i o n of job t r a i n i n g needs, another major use of the r e s u l t s o f the a p p r a i s a l p r o c e s s . I t i s l i k e l y t h a t some of the behaviours r e q u i r e d to demonstrate improved performance on a g i v e n job w i l l not be p a r t of the employee's behaviour r e p e r t o i r e . He f i r s t must be made aware of h i s shortcomings, and then be f a m i l i a r i z e d w i t h the behaviours r e q u i r e d to s u c c e s s f u l l y do the job [Buehler, 196 9 ] . BARS instruments appear to provide l i t t l e a s s i s t a n c e i n such s i t u a t i o n s because of t h e i r l i m i t e d sampling of the job behaviour domain. In c o n t r a s t , the BDI, because of i t s l a r g e r i n f o r m a t i o n content, can be ah a i d to the t r a i n i n g process where i d e n f i c i a t i o n o f s p e c i f i c t r a i n i n g or l e a r n i n g o b j e c t i v e s i s d e s i r e d . In a d d i t i o n , the i n c r e a s e d i n f o r m a t i o n content of the BDI should p r o v i d e s u p e r i o r s w i t h more ac c u r a t e feedback f o r promotion d e c i s i o n s . As has been shown e a r l i e r [Meyer, 1969; see d i s c u s s i o n on p. 9 1 ] , c e r t a i n types o f behaviour e x h i b i t e d by subordinates seemed to be r e l a t e d to t h e i r subsequent promotion. I f t h i s i s t r u e , then the BDI format should h e l p s u p e r i o r s to t focus t h e i r o b s e r v a t i o n s more on behaviours which are r e l e v a n t f o r promotion, and, secondly, i t should a i d s u p e r i o r s and t r a i n e r s i n r e i n f o r c i n g such behaviours. Furthermore, i t would aid i n the creation of t r a i n i n g programmes which aim at the development of managerial personnel and executive trainees by providing well defined t r a i n i n g objectives, namely, desirable supervisory behaviour. There i s ^ a t ..least.'..on'ermorerreason why the BDI may be superior to BARS. I t i s the lack of the good/bad connotation ofa positive/negative continuous scale, l i k e the e f f e c t i v e , i n e f f e c t i v e dichotomy of the BARS instrument. The Yes/No response of the BDI should help to keep the "evaluation tone" [Cummings and Schwab, 1973, p. 114] out of the appraisal process, and cast the superior more into the role of a performance observer and reporter rather than a judge. As a r e s u l t , subordinates may be less defensive and perceive the appraisal feedback as less threatening to t h e i r self*-* esteem. (This perceived threat and defensive reaction has been well documented i n a study by Kay, Meyer and French, 1965.) i i i A l l t h e s e c h a r a c t e r i s t i c s c ombined, i . e . 1. a r i g o r o u s d e v e l o p m e n t p r o c e s s ; 2. l e s s c omplex and more a c c u r a t e r a t i n g p r o c e d u r e s ; 3. b e t t e r s a m p l i n g o f t h e j o b b e h a v i o u r domain; 4. a v o i d a n c e o f good/bad a n c h o r s ; 5. t h e u s e o f Yes/No r e s p o n s e s and s e p a r a t e c o n v e r s i o n t o n u m e r i c a l s c o r e s ; 6. t h e p o s s i b i l i t y t o d e v e l o p norm s c o r e s ; 7. a v o i d a n c e o f e x p e c t a t i o n p h r a s e o l o g y ; and 8. r e c o r d i n g o f a c t u a l l y o b s e r v e d b e h a v i o u r s i n s t e a d o f i n d i c a t i n g an expectation or b e l i e f that an observed variety of behaviours i s equivalent to a degree of e f f e c t i v e performance as represented by a single behaviour anchor; seem to favour the BDI instrument when compared to the BARS format with regard to ease of handling, information value, and rating c h a r a c t e r i s t i c s . Hypotheses to be Investigated I t i s the major objective of t h i s study to investigate the psychometric properties of the BDI, a behaviour oriented scale with the advantages claimed for BARS, but without or with less of the disadvantages discussed. Because of the many factors involved i n the development and v a l i d a t i o n of a new instrument, a large number of hypotheses could be tested. However, i n order to keep the study i n manageable boundaries, i t w i l l be limited to some basic investigations into the nature and c h a r a c t e r i s t i c of the new instrument as compared to currently used approaches to performance appraisal and t r a i n i n g evaluation. The following hypotheses w i l l be tested: H^ The BDI scale w i l l be superior to BARS with respect to such r a t i n g c h a r a c t e r i s t i c s as: ) h a l o e f f e c t I t i s b e l i e v e d t h a t s i n c e r a t e r s u s i n g t h e BDI do n o t know t h e p o i n t s c o r e o f a d i m e n s i o n r a t i n g , t h e y a r e l e s s a b l e t o t r a n s f e r t h e r a t i n g s f r o m one d i m e n s i o n t o o t h e r d i m e n s i o n s ; t h e BDI s h o u l d a l s o be more s p e c i f i c w i t h i t s s c a l e s b e c a u s e o f b e t t e r s a m p l i n g o f t h e j o b b e h a v i o u r d o m a i n , t h u s c a u s i n g h i g h e r d i m e n s i o n i n d e p e n d e n c e . ) l e n i e n c y e f f e c t The BDI a v o i d s s c a l e s w i t h e v a l u a t i v e c h a r a c t e r i s t i c s , s u c h a s e f f e c t i v e / i n e f f e c t i v e , h i g h / l o w , e t c . , a l l w i t h g o o d / b a d c o n n o t a t i o n w h i c h p u t t h e r a t e r i n t o a p o s i t i o n o f a j u d g e . The Yes/No/? r e s p o n s e s o f t h e BDI> on t h e o t h e r h a n d , a l l o w s a r a t e r t o p l a y more t h e r o l e o f a n e u t r a l o b s e r v e r . The BDI a v o i d s a l s o t h e n e u t r a l p o i n t o f a BARS i n s t r u m e n t w h i c h seems t o be an a d d i t i o n a l s o u r c e o f l e n i e n c y . ) c e n t r a l t e n d e n c y e f f e c t C e n t r a l t e n d e n c y i s d e f i n e d a s t h e u n w i l l i n g n e s s o f r a t e r s t o u s e e x t r e m e r a t i n g s , s u c h as " e x c e l l e n t " , " v e r y p o o r " . The BDI d o e s n o t u s e s u c h a n c h o r s . d) r e l i a b i l i t y The BDI u t i l i z e s a larger number of tes t items which should increase r e l i a b i l i t y . a) The BDI w i l l demonstrate s i g n i f i c a n t convergent v a l i d i t y with BARS. It i s assumed that both instruments measure the same underlying construct, namely performance as perceived by superiors. b) The BDI w i l l demonstrate superior discriminant v a l i d i t y compared to BARS. It i s believed that the job dimensions rated on a BDI format are more c l e a r l y defined than the dimensions of a BARS format, because of the better sampling of the job behaviour domain with the BDI. The BDI w i l l be rated as superior to BARS on the following dimensions: a) s u i t a b i l i t y for feedback and counselling purposes. Because of the larger number of behaviour samples, the BDI allows raters to point to s p e c i f i c incidents when they assess the strengths and weaknesses of a ratee. 115 b) s u i t a b i l i t y for promotion purposes. I t i s believed that the BDI contains more job behaviour items relevant to the promotability of an employee than BARS. c) s u i t a b i l i t y for wage and salary evaluation purposes. I t i s assumed that the BDI allows a more detailed evaluation of an employee's accomplishments through better sampling of the job behaviour domain, and the p o s s i b i l i t y to develop norm scores. d) s u i t a b i l i t y for tr a i n i n g need determination. Through the higher s p e c i f i c i t y and improved sampling of the job behaviour domain of a BDI, a rater i s able to better pinpoint s p e c i f i c shortcomings of a ratee than with the use of BARS. In addition, i t allows to determine more accurately which t r a i n i n g programme i s suited best to correct the detected shortcomings. e) s u i t a b i l i t y for developing s p e c i f i c t r a i n i n g programmes and formulating t r a i n i n g objectives. The use of the BDI as a survey to determine the t r a i n i n g needs i n a department or an organization would allow t r a i n i n g managers and instructors to develop t r a i n i n g programmes which could be better suited to organizational and i n d i v i d u a l t r a i n i n g needs than surveys conducted with BARS. f) ease of use. It i s believed that r a t i n g procedures for the BDI are less complex than the procedures for BARS. Behaviourally oriented scales can be used for sim i l a r jobs i n d i f f e r e n t organizations. I t has been shown that BARS need not be job s p e c i f i c [Goodale and Burke, 1975]. However, thus far BARS has been developed for each organization separately. An attempt w i l l be made to show that behaviour oriented scales can be developed for d i f f e r e n t organizations, provided that the jobs for which they are developed contain s i m i l a r job dimensions. The BDI w i l l show an additional advantage over BARS: I t w i l l demonstrate less waste of information (i . e . w i l l have higher information content) than BARS. The BDI uses the same development c r i t e r i a as the BARS instrument, but does not require the scrapping of items which do not f i t a certa i n scale point, or the elimination of one or more items i f several items have the same scale value ( i . e . i f three items are rated "2", only one can be used). CHAPTER IV METHOD Sponsors of the Project and Research Site The main sponsor of the research project was the Institute of Canadian Bankers (ICB). I t i s part of the Canadian Bankers' Association, an i n t e r e s t group formed by the ten chartered banks i n Canada. 1 The Inst i t u t e i s responsible for a l l educational a c t i v i t i e s of the Association and organizes nationwide t r a i n i n g programmes for employees of the Canadian banking industry. Other sponsors were f i v e of the largest chartered banks: The Royal Bank, Bank of Montreal, Canadian Imperial Bank of Commerce, Toronto Dominion Bank, and the Bank of Nova Scotia. These banks agreed to support the study by permitting the use of the i r B r i t i s h Columbia branches for the i n i t i a l data c o l l e c t i o n , and the i r national organizations for the v a l i d a t i o n of the instruments. *Bank of Montreal, The Bank of Nova Scotia, Toronto Dominion Bank, The P r o v i n c i a l Bank of Canada, Canadian Imperial Bank of Commerce, The Royal Bank of Canada, Banque Canadienne Nationale, The Mercantile Bank of Canada, Bank of B r i t i s h Columbia, Unity Bank of Canada. The support and co-operation of the ICB and the fi v e banks provided an i d e a l environment for the study. 2 The banks allowed the testing of the instruments as performance appraisal tools while, at a l a t e r date, they may be tested with the t r a i n i n g programmes offered by the Ins t i t u t e as to t h e i r a p p l i c a b i l i t y as t r a i n i n g evaluation devices. Since i t has been proposed to develop behaviour scales for d i f f e r e n t organizations, the question of the actual differences between Canadian chartered banks has to be addressed. Although there are no studies available which compare i n t e r n a l structure and p o l i c i e s , some information can be gathered from publications on the Canadian banking system and i t s i n s t i t u t i o n s . For example, The Fi n a n c i a l Post Survey of Industries [1976] shows that the f i v e banks which participated i n the study vary s i g n i f i c a n t l y i n t h e i r assets and number of branches (see Table IV.1). G a i l b r a i t h [1970] observed that: while the structure of the [Canadian] chartered banks i s that of bank branching system, i t i s not a homogeneous system" [p. 12]. He points out 2The i n i t i a l generous support given by the banks was reduced d r a s t i c a l l y during the v a l i d a t i o n stage; i . e . assessment of bank managers with the new instruments, and involvement of subordinates i n the appraisal process. The probable reason was a strong unionization drive among bank employees [Silverman, 1977], For t h i s reason, some of the o r i g i n a l l y planned v a l i d a t i o n analyses, e.g. i n t e r r a t e r r e l i a b i l i t y , factor analysis, had to be dropped or changed. that some banks are i n t e r p r o v i n e i a l , or national, while others concentrate on one or a few provinces, and some can even be c a l l e d i n t e r n a t i o n a l because of t h e i r s i g n i f i c a n t interests outside Canada. Some clues about differences i n banking p o l i c y between Canadian banks are provided by Dougherty [19 75] who compared d i f f e r e n t approaches to investment decisions, d i f f e r e n t emphases on investment areas, and differences i n c l i e n t e l e . (See also G a i l b r a i t h , 1963; Neufeld, 1964 & 1972.) The above information, scarce as i t i s , seems to indicate that there are some s i g n i f i c a n t internal, (structure and policy) differences between the Canadian chartered banks. I t supports the investigator's own impression which he gained when studying the personnel management area of the banks,, s p e c i a l l y performance appraisals and t r a i n i n g p o l i c i e s . From the customer's point of view, banks look very much a l i k e . However, the differences grow larger the closer a look one takes. This w i l l be confirmed i n the following chapter i n the discussion on s i m i l a r i t i e s and differences i n job r e s p o n s i b i l i t i e s of the target group which.; involved i n t h i s study. Target Groups The investigator arranged a meeting with the l o c a l t r a i n i n g managers of f i v e p a r t i c i p a t i n g banks i n order to explain the purpose of the study, the procedures, and to answer any questions the managers may have had. The second objective was to determine the target group for which the evaluation instruments should be developed. A l l participants of the meeting were of the opinion that the target group should occupy a position which was very t r a i n i n g intensive, i n order to make the development of the instruments worthwhile. The investigator made the suggestion that a job be selected which would enable superiors, peers, and subordinates to p a r t i c i p a t e i n the study. The reason for t h i s was that the v a l i d i t y of the instrument would be higher i f every aspect of the job -- i . e . , as perceived by supervisors, peers, and subordinates — could be taken into account. This pretty much determined the f i n a l choice of the target group as far as the t r a i n i n g managers were concerned: bank accountants. To non-bankers the job t i t l e "bank accountant" i s somewhat misleading. The Webster d e f i n i t i o n of an accountant i s a person whose job i s "the f i g u r i n g and recording of f i n a n c i a l accounts". In a banking organization, the accountant i s the Assistant Branch Manager or, i n some banks, the "Administration Manager". He reports d i r e c t l y to the Branch Manager and i s responsible for the day-to-day routine work i n a branch: supervision of clerks and t e l l e r s , intra-,and. i n t e r -bank correspondence, d a i l y , weekly, monthly and quarterly balance checks and audits, personnel administration and t r a i n i n g , customer contacts, and marketing on a l o c a l basis. Because of the wide variety of r e s p o n s i b i l i t i e s an accountant has to undergo considerable t r a i n i n g before he assumes his job and he i s expected to "keep up" constantly. He p a r t i c i p a t e s i n general i n t e r n a l management courses (Supervision, Decision Making, Communication, e t c . ) , and s p e c i f i c job-related courses (Accounting, Marketing, Customer S o l i c i t a t i o n ) . In addition, many accountants p a r t i c i p a t e i n the ICB's Fellow Program, a complementary programme to the tr a i n i n g i n d i v i d u a l banks provide. Examples of ICB courses are: Business Administration, Economics, Fina n c i a l Accounting, and Business Strategy. I t appears from the above given information that the selection of bank accountants (or assistant branch managers) as a target group for performance appraisal and t r a i n i n g evaluation purposes was an appropriate one: a group i n a managerial position which i s t r a i n i n g intensive and observable from three angles, namely superiors, peers, and subordinates. The generic t i t l e "bank accountant" could be interpreted to describe an i d e n t i c a l job i n d i f f e r e n t banks. This i s c e r t a i n l y not the case. Actually, t h i s t i t l e i s o f f i c i a l l y used only i n one bank; other t i t l e s are: administration o f f i c e r , administrative o f f i c e r , and assistant branch manager. The r e s p o n s i b i l i t i e s of these managers may vary considerably from bank to bank and even.from branch to branch i n one bank, depending on the size of the branch, geographical location, and type of customers (see Appendix F for samples of job descr i p t i o n s ) . For example, one bank has nine d i f f e r e n t l e v e l s of assistant branch manager. However, the differences i n job r e s p o n s i b i l i t i e s are more i n emphases than content, e.g. the assistant branch manager of a large branch w i l l be less concerned with t r a i n i n g and more with administrative and personnel r e s p o n s i b i l i t i e s than the assistant manager of a small branch, who w i l l divide his time more equally over the d i f f e r e n t areas. Other differences, of course, are i n policies,, rules, procedures, and norms. I t i s the contention of the t r a i n i n g managers that the d i f f e r e n t jobs have enough s i m i l a r i t y i n content that common job dimensions can be i d e n t i f i e d (see discussion on job dimensions i n the following chapter, p. 124 ). 3 Development of BARS and BDI 124 D e f i n i t i o n of Job Dimensions The t r a i n i n g managers of the fi v e p a r t i c i p a t i n g banks were asked to describe the important job dimensions of an accountant; i . e . , the major areas the accountant i s concerned with. Several t r a i n i n g managers suggested that the job description of an accountant be consulted. The investigator r e s i s t e d the idea on the ground that the t r a i n i n g managers should think i n terms of "what i s " instead of "what ought to be". To make the task easier and at the same time to explain the C r i t i c a l Incident Method, the managers were asked to generate c r i t i c a l incidents related to the job of an accountant. I t must be pointed out that a l l t r a i n i n g managers had occupied the pos i t i o n of an accountant for at l e a s t two years and had been out of the job for not longer than 2 1/2 years, thus they were f a m i l i a r with the job c h a r a c t e r i s t i c s . After one hour about 6 0 incidents had been created. A lengthy discussion involving a l l participants resulted i n the categorization of the incidents into 7 job dimensions which, after careful analysis, were reduced—unanimously— to the following 5 ( l i s t e d i n alphabetical order): Administration Customer Relations Marketing Personnel Training For some time i t was debated whether to c o l l a p s e T r a i n i n g i n t o P e rsonnel, but i t was agreed t h a t T r a i n i n g was a s u f f i c i e n t l y s t r o n g dimension to warrant a separate category. Generation o f B e h a v i o r a l I n c i d e n t s A 4-hour workshop was o r g a n i z e d w i t h each bank. P a r t i c i p a n t s were 5 accountants, 5 branch managers, and 5 su b o r d i n a t e s . The s e l e c t i o n was done by the t r a i n i n g managers who had been i n s t r u c t e d to i n v i t e p e r s o nnel from s m a l l , medium, and l a r g e r branches and to a s c e r t a i n t h a t the accountants were experienced employees, not j u s t r e c e n t l y promoted. No r e c o g n i z a b l e b i a s was i n t r o d u c e d by t h i s procedure. The f i r s t t h i r t y minutes o f the workshop were used to i n f o r m the p a r t i c i p a n t s o f the purpose of the workshop, the study, and i t s meaning to t h e i r o r g a n i z a t i o n . Great emphasis was gi v e n the l a s t p o i n t i n order to motivate the p a r t i c i p a n t s to co-operate w i l l i n g l y . I t was a l s o s t r e s s e d t h a t the workshop would be a g r e a t l e a r n i n g experience f o r everyone s i n c e never b e f o r e i n t h e i r c a r e e r d i d they have t h i s i n t e n s i v e exposure to the d e s c r i p t i o n s o f e f f e c t i v e and i n e f f e c t i v e job behaviour. The C r i t i c a l Incident technique was explained i n d e t a i l and samples of behaviour descriptions were given. From the ensuing discussion i t became apparent that many of the participants did not distinguish between attitudes and behaviour. The problem was e a s i l y solved when the differences were made clear by giving examples, and appropriate behaviour samples were generated by the p a r t i c i p a n t s . The following 3 1/2 hours were, devoted to writing down c r i t i c a l behaviour incidents. To aid i n t h i s endeavour the 5 job dimensions defined by the t r a i n i n g managers were written on the blackboard. The participants were asked whether they could think of possible other dimensions. A few were suggested, but i t could be demonstrated—to the s a t i s f a c t i o n of the group—that the new dimension f i t into one already defined. During the session the investigator c i r c u l a t e d among the participants to answer any questions and check randomly the q u a l i t y of the incidents produced. Frequently, good and poor examples were discussed with the whole group. At the end of the session participants were asked to record any incidents they might encounter during the following two weeks and to report these incidents to t h e i r t r a i n i n g managers. After the m e e t i n g t h e i n v e s t i g a t o r met s e v e r a l o f t h e p a r t i c i p a n t s o n o t h e r o c c a s i o n s , a nd i t was i n v a r i a b l y a f f i r m e d t h a t t h e w o r k s h o p h a d b e e n p e r c e i v e d a s an e x c e l l e n t l e a r n i n g e x p e r i e n c e . The b r a n c h m a n a g e r s a n d a c c o u n t a n t s e s p e c i a e x p r e s s e d i n t e r e s t i n r e c e i v i n g a c o p y o f t h e f i n i s h e d i n s t r u m e n t . I n e a c h w o r k s h o p more t h a n a t h o u s a n d b e h a v i o u r d e s c r i p t i o n s w e r e g e n e r a t e d . A f t e r r e m o v i n g r e d u n d a n t a nd i n a p p r o p r i a t e i n c i d e n t s b e t w e e n 400 t o 600 i t e m s p e r b a n k r e m a i n e d . T h e s e w e r e e d i t e d t o c o n f o r m t o t h e " e x p e c t e d b e h a v i o u r " f o r m a t s u g g e s t e d b y S m i t h and K e n d a l l [1963]. I n c i d e n t V a l i d a t i o n a n d R e t r a n s l a t i o n b y I n d i v i d u a l B a n k s A f t e r t h e i n c i d e n t s h a d b e e n e d i t e d t h e y w e r e t y p e d o n l e g a l - s i z e p a p e r a nd s e n t t o f i v e j u d g e s t o be v a l i d a t e d . T h i s was done f o r e a c h b a n k s e p a r a t e l y . The j u d g e s w e r e a c c o u n t a n t s who w e r e s e l e c t e d b y t h e t r a i n i n g m a n a g e r s f o r t h e i r j o b e x p e r i e n c e a n d g o o d p e r f o r m a n c e . They w e r e a s k e d t o c h e c k o f f e v e r y i t e m t h e y c o n s i d e r e d t o be a " t y p i c a l " b e h a v i o u r d e s c r i p t i o n f o r t h e j o b o f a n a c c o u n t a n t . I t e m s were r e t a i n e d i f 4 o u t o f t h e 5 j u d g e s (80%) a g r e e d o n t h e v a l i d i t y . S e c o n d l y , t h e y w e r e a s k e d t o c a t e g o r i z e t h e i t e m s i n t o t h e f i v e p r e - i d e n t i f i e d j o b d i m e n s i o n s b y w r i t i n g t h e f i r s t l e t t e r o f t h e r e s p e c t i v e d i m e n s i o n s i n f r o n t o f t h e i t e m . Items were r e t a i n e d i f 4 o u t o f 5 j u d g e s (80%) a g r e e d on t h e d i m e n s i o n s . Between 200 and 300 i t e m s p e r bank s u r v i v e d t h e s e two s t e p s . I n c i d e n t R a t i n g b y I n d i v i d u a l Banks Items w h i c h s u r v i v e d t h e v a l i d a t i o n and r e -t r a n s l a t i o n p r o c e s s were l i s t e d a g a i n and d i s t r i b u t e d t o 20 a c c o u n t a n t s o f e a c h bank, a l l o f them n o t p r e v i o u s l y i n v o l v e d i n t h e p r o j e c t . A c o v e r i n g l e t t e r (see A p p e n d i x G) e x p l a i n e d t h e p u r p o s e o f t h e p r o j e c t and p r o c e d u r e s t o be f o l l o w e d . The p a r t i c i p a n t s were a s k e d t o r a t e e a c h i t e m on a 1 t o 7 s c a l e a c c o r d i n g t o t h e d e g r e e o f e f f e c t i v e n e s s o f j o b b e h a v i o u r t h e y p e r c e i v e d e a c h i t e m d e s c r i b e d . 3 I t was s u g g e s t e d , t h a t r a t e r s s t a r t w i t h i t e m s on one j o b d i m e n s i o n ( e . g . , A d m i n i s t r a t i o n ) and r a t e them t o g e t h e r , b e c a u s e t h i s a p p r o a c h w o u l d q u i c k l y e s t a b l i s h a frame o f r e f e r e n c e . f o r t h e i t e m s . A f t e r t h e r a t i n g s were c o m p l e t e d p a r t i c i p a n t s were a s k e d t o r a t e t h e f i v e j o b d i m e n s i o n s a c c o r d i n g t o t h e d e g r e e o f i m p o r t a n c e f o r t h e j o b o f an a c c o u n t a n t . 3 T h i s r a t i n g was done f o r t h e p u r p o s e o f d e v e l o p i n g i n s t r u m e n t s f o r e a c h bank, o r i n - h o u s e measurements. A l t h o u g h i t was n o t r e q u i r e d f o r t h e d e v e l o p m e n t o f t h e a c r o s s o r g a n i z a t i o n i n s t r u m e n t s , i t p r o v i d e d a u s e f u l c r i t e r i o n f o r a r e d u c t i o n o f t h e number o f i t e m s i n t h e o v e r a l l i t e m p o o l , as d e s c r i b e d on page 130 Incident Selection C r i t e r i a for Individual Bank Item Pool Items were retained i f : 1. 4 out of 5 experts (80%) agreed on the v a l i d i t y of the item; 2. 4 out of 5 experts (80%) agreed on the categorization of the item; 3. the standard deviation of the item ratings did not exceed 1.0. (This standard was possible because of the large number of items available.) Incident- Validation".and Rating by a l l Banks Since the behaviour descriptions for the instruments had to be v a l i d for a l l banks, the f i n a l s e l e c t i o n had to be made from the t o t a l item pool of the banks, consisting of approximately 1,000 items. The national t r a i n i n g directors of the banks were reluctant to submit such a voluminous l i s t to a large group of experts i n t h e i r organization because of the time and e f f o r t involved, but they were w i l l i n g to allow a smaller number of items to be submitted. Because of this l i m i t a t i o n , an i n i t i a l s e l e ction had to be made. At a meeting with the t r a i n i n g managers of the 5 l o c a l banks, the following procedure was Worked out: 1. pick 5 0 items per dimension with the lowest standard deviation from the t o t a l pool; 2. discard redundant items and pick replacements with the next higher standard deviation; 3. re t a i n items i f a l l judges agree to the selection. Between 30 and 35 items per dimension (or a t o t a l of 159) survived this process. The items were l i s t e d i n random order and mailed by the national t r a i n i n g centre of each bank to a group of 200 judges consisting of 50 branch managers, 100 accountants, and 5 0 c l e r i c a l employees. They were asked to judge each item as to i t s v a l i d i t y as a t y p i c a l job behaviour of an accountant and, secondly, to rate i t on a 1 to 7 scale as to the degree of effectiveness i t described i n t h e i r opinion (see Appendix H). Out of 1,000 mailed rating l i s t s , 511 usable were returned (51% response r a t e ) . Incident Selection C r i t e r i a for BARS and BDI The responses were subject to a Simple Data Analysis (BMD 01D) 4, r e s u l t i n g i n means, standard deviations, number, of responses, maxima, minima, and ranges. This analysis was done for each bank ^Biomedical Computer Programs, University of C a l i f o r n i a , Berkeley. separately, and for a l l data combined. This allowed for a convenient (visual) comparison of the degree of agreement among the judges of the i n d i v i d u a l banks and among a l l judges. Items were retained i f they f u l f i l l e d the following requirements: 1. 80% of a l l judges, and 2. 60% of the judges of the i n d i v i d u a l banks agreed on the v a l i d i t y of the i tem; 3. the standard deviation of the item did not exceed 1.5. Out of 159 items, 120 (75%) could be retained. Composition of the BARS Instrument The c r i t i c a l incidents for the f i n a l BARS instrument were selected according to a) ratings closest to a scale point (1 to 7), and b) lowest possible standard deviation. As i t turned out, none of the items was rated close to the neutral (neither e f f e c t i v e nor in e f f e c t i v e ) scale anchor, #4. This problem i s discussed by Landy and Guion [1970], who suggest that p o l a r i z a t i o n i s a necessary consequence of eliminating items which do not have meaning to i n d i v i d u a l s . The items were then arranged according to t h e i r ratings and the f i v e dimensions they represented (see Appendix I ) . At the top.of each scale the dimension i s i d e n t i f i e d and a description of the job r e s p o n s i b i l i t i e s (for t h i s dimension only) i s given, followed by the rating procedure: Read every example of the job incumbent's behavior and then put a check-mark (/) by the sample that best represents how you would expect the incumbent you are rat i n g to t y p i c a l l y perform i n thi s aspect of his job. Composition of the BDI For the purpose of thi s study i t was a r b i t r a r i l y decided to u t i l i z e 20 job behaviour samples per job dimension of the BDI, or 100 i n t o t a l . 5 Examples of e f f e c t i v e and i n e f f e c t i v e behaviour were selected according to the lowest standard deviation to reduce item ambiguity as far as possible. (The st r i c t n e s s of t h i s c r i t e r i o n w i l l depend on the number of items available and the number of incidents to be u t i l i z e d i n the scale.) The main reason the standard deviation c r i t e r i o n was applied was that the data was conveniently available since i t was required for the item se l e c t i o n of the BARS format. I t i s believed that instead of the standard deviation c r i t e r i o n , a random sampling 5Future research must decide whether 20 items or a number above or below are appropriate. I t i s assumed that many factors w i l l influence this decision; e.g. nature and complexity of the job, number of job dimension, purpose of evaluation, to name a few. o f i t e m s f r o m t h e t o t a l i t e m p o o l p e r d i m e n s i o n may be s u f f i c i e n t , t h u s somewhat r e d u c i n g t h e t i m e a nd e f f o r t r e q u i r e d t o d e v e l o p t h e BDI a s c o m p a r e d t o d e v e l o p t h e BARS f o r m a t . The s i g n i f i c a n c e o f t h e s t a n d a r d d e v i a t i o n c r i t e r i o n f o r t h e BDI f o r m a t i s n o t known and h a s t o be i n v e s t i g a t e d s e p a r a t e l y . S i m i l a r t o t h e BARS f o r m , t h e BDI s c a l e b e g i n s w i t h t h e j o b . d i m e n s i o n a s a h e a d i n g , f o l l o w e d b y t h e d e s c r i p t i o n o f t h e r e s p o n s i b i l i t i e s o f t h e j o b i n c u m b e n t i n t h i s s p e c i f i c a r e a ; e . g . ( f o r A d m i n i s t r a t i o n ) q u a r t e r l y a n a l y s e s , m o n t h l y r e p o r t s , e t c . The r a t i n g p r o c e d u r e i s n e x t a nd r e a d s : P l e a s e THINK CAREFULLY b e f o r e y o u a n s w e r - t h i s q u e s t i o n : Does t h i s j o b i n c u m b e n t e x h i b i t t h e f o l l o w i n g d e s c r i b e d b e h a v i o r c o n s i s t e n t l y ? (Does i t d e s c r i b e h i s t y p i c a l b e h a v i o r ? ) Use "Y" f o r Y e s , "N" f o r No, and " ? " i f y o u a r e n o t c e r t a i n . [See A p p e n d i x D j 134 Development of a Third Instrument, a Graphic Rating Scale I t was f e l t that i t would be useful and informative to compare the s p e c i a l l y developed BARS and BDI instruments to the performance appraisal format presently used by the banks. This i s also i n l i n e with the recommendation given by Schwab et a l . [19 75] with regard to measuring leniency and scale independence. They suggest that i t i s ri s k y to make inferences about r e l a t i v e leniency e f f e c t s using only two measures, because i t may be impossible to determine what the true average ratings should have been. Si m i l a r l y , i f i t i s assumed that performance on various dimensions i s i n t e r r e l a t e d , then comparisons based on only two measures may be inadequate. The performance appraisal instrument used by the i n d i v i d u a l banks i s a Graphic Rating Scale (GRS), consisting mainly of t r a i t ratings (e.g. i n i t i a t i v e , leadership, e t c . ) , and global ratings (e.g. o v e r a l l performance i n customer r e l a t i o n s , administration, t r a i n i n g , etc.) on a 7-point scale, ranging from outstanding to unacceptable. Although the forms were d i f f e r e n t from bank to bank, the contents were sim i l a r enough to allow the use of a seven-point Likert-type scale for a global evaluation for each of the dimensions of the job of a bank manager (see Appendix K). I t should be emphasized, however, since the GRS has never been validated and c e r t a i n l y was not developed i n a sim i l a r rigorous way as the BARS and BDI instruments, that any conclusions based on re s u l t s of comparisons between the three instruments have to be made with t h i s difference i n mind. Ratee and Rater Sample Two groups of assistant branch managers of two d i f f e r e n t banks were rated by the i r superiors. Raters for Group 1 consisted of 31 branch managers from branches of one bank i n three Maritime provinces, Nova Scotia, Prince Edward Island, and Newfoundland. Raters for Group 2 were made up of 42 branch managers of a second bank, who attended two-week t r a i n i n g seminars at the bank's t r a i n i n g center i n Toronto. Members of.the second group represented a l l provinces of Canada. The f i r s t group of raters was v i s i t e d i n th e i r branches by t r a i n i n g instructors of the bank and briefed thoroughly on the nature of the research project and the c h a r a c t e r i s t i c s of the d i f f e r e n t instruments to be used. In addition, they received written instructions with regard to possible r a t i n g 136 errors, such as halo, leniency, and central tendency (see Appendix L ) . 6 The investigator was also available to answer remaining questions v i a telephone. Since for t h i s group the semi-annual performance evaluation had taken place 10 to 15 weeks before the research project, i t was f e l t that i t was necessary to repeat the appraisal together with the BARS and BDI evaluation to ensure that the raters had the same type of performance i n mind for a l l three instruments. The nature of the second group and i t s treatment must be explained i n more d e t a i l . The second bank had permitted to u t i l i z e i t s national t r a i n i n g center i n Toronto for data c o l l e c t i o n on an ongoing basis to allow the investigator to accumulate a large number of responses for — eventually — more complex s t a t i s t i c a l , analyses, e.g. factor a n a l y s i s . 7 The investigator was i n v i t e d to attend the f i r s t seminar i n t h i s series, and given the opportunity to address the trainees and the trainers i n order to explain the research project i n d e t a i l . He also gave a 30 minute lecture on possible rating errors and how 6The instructions were general i n nature, i . e . did not refer to a s p e c i f i c instrument. 7The process w i l l be slow since the type of raters used i n t h i s v a l i d a t i o n study — branch managers — meet only once a month i n groups of 6 to 15 for two weeks. to avoid them. Another 30 minutes were devoted to open discussion and to explain the rating procedures for each instrument. Both trainees and trainers participated i n the i n s t r u c t i o n sessions. For the l a t t e r i t was important, since they had to give similar presentations to the following groups. The p a r t i c i p a t i n g branch managers then rated t h e i r respective assistant branch managers, using the three d i f f e r e n t instruments. Four d i f f e r e n t groups resulted i n 42 evaluations. Twenty-two of the seminar participants were also available f o r a t e s t - r e t e s t analysis. A l l three instruments were applied during the f i r s t day of the seminars and a second time at the end of the second week. Analyses The scale analysis focuses on two aspects of the three instruments: v a l i d i t y and psychometric charac-t e r i s t i c s . The discussion on v a l i d i t y w i l l include face, content, and construct v a l i d i t y , while the d i s -cussion on scale c h a r a c t e r i s t i c s w i l l focus on halo, leniency, central tendency, r e l i a b i l i t y , and informa-tion content. Face v a l i d i t y i s the extent to which an instrument "looks l i k e " i t measures what i t i s intended to measure [Nunnally, 1967, p. 99]. I t i s the simplest kind of instrument v a l i d a t i o n and i s purely judgemental, From this point of view the face v a l i d i t y of BARS and BDI are very high for the following reasons: 1. Job performance i s defined as behaviour that has been evaluated (measured) i n terms of i t s contribution to the goals of the organization [Campbell et a l . , 1973, p. 17]; 2. Training i s usually concerned with change (improvement) of job related behaviour [Otto and Glaser, 1970, p.3]; i n other words, job performance and t r a i n i n g are both concerned with job behaviour. Since the BARS and the BDI are based on c r i t i c a l incidents (samples of job behaviour), behaviour i s measured with behaviour. The GRS used i n t h i s study does not fare that well i n thi s comparison, because i t i s based on the description of areas of r e s p o n s i b i l i t i e s (e.g., Administration) and a global evaluation of the l e v e l of performance on a 7-point scale. Content v a l i d i t y i s demonstrated by showing how well the content of a test samples the si t u a t i o n about which conclusions are to be drawn [Roscoe, 1969, p. 107]. Similar to face v a l i d i t y , i t implies the use of some expertise, i . e . , judgments by experts. As Kerlinger [1973] points out, for content v a l i d i t y the items of a te s t must be studied, each item being weighed for i t s presumed representativeness of the universe. This means that each item must be judged for i t s relevancy to the property being measured [p. 459]. The content v a l i d i t y of the BARS and the BDI has been established by using experts for the item v a l i d a t i o n . They answered the question: "Is thi s a t y p i c a l behaviour for the job of an accountant?" Not only does t h i s approach assure a strong relevancy of each item, i t should r e s u l t also i n a good i n d i c a t i o n of content v a l i d i t y , namely the degree of agreement on the v a l i d i t y of each item among the judges [Nunnally, 1967, pp. 79-83], As may be remembered, items were retained i f 80% of the judges agreed on i t s representativeness of the job behaviour domain. No such in d i c a t i o n i s available for the GRS instrument. r Construct v a l i d ! t y i s defined as the degree to which the variance i n a given set of measures i s due to variance i n the underlying construct [Guion, 1965, p. 128]. He cautions, however, that construct v a l i d a t i o n i s not a one-shot a f f a i r , but has to be i n f e r r e d from the weight of research evidence gathered i n many independent studies. Cronbach and Meehl [1955] express a similar opinion by saying that a construct i s not defined by an i s o l a t e d event but by a "nomological network" of associations and situations i n which i t acts. The process of i n f e r r i n g construct v a l i d i t y i s described by Cronbach [1970]: Construct v a l i d i t y i s evaluated by investigating what psychological q u a l i t i e s a t e s t measures; i . e . , by determining the degree to which certa i n explanatory concepts or constructs account for performance on the t e s t . [p. 444] As already mentioned i n the introduction, the process of establishing c o n s t r u c t ' v a l i d i t y remains e s s e n t i a l l y judgemental. This fact makes i t d i f f i c u l t to express construct v a l i d i t y i n p r o b a b i l i s t i c terms. However, there are techniques available which r e s u l t . i n data interpretable as q u a n t i f i c a t i o n of construct v a l i d i t y . Factor analysis i s one popular measure [Cronbach and.Meehl, 1955; Eysenk, 1950; Guilford, 1948], a second method i s Campbell and F i s k e 1 s [1959] multitrait-multimethod matrix. It was intended to use factor analysis as one measure to e s t a b l i s h construct v a l i d i t y , but because of an i n s u f f i c i e n t number of participants i n the v a l i d a t i o n stage, this approach had to be abandoned. Since the BDI u t i l i z e s 10 0 variables ( c r i t i c a l incidents) a sound factor analytic approach would have required at l e a s t 400 to 500 respondents [Cattel, 1952; Gorsuch, 1974, p. 269], a number the supporters of t h i s study were reluctant to commit. In the process of construct v a l i d a t i o n i t i s very important to demonstrate that tests have convergent and discriminant v a l i d i t y [Kerlinger, 1973, p. 426; Campbell and Fiske, 1959]. Convergent v a l i d i t y r e f l e c t agreement among raters i n assessing dimensions of job behaviour. This i s demonstrated by correlations between the same dimensions, using d i f f e r e n t instruments, being s i g n i f i c a n t l y d i f f e r e n t from zero. Discriminant v a l i d i t y r e f l e c t s the independence of the performance dimensions. Campbell and Fiske [1959] suggest that scores on a dimension should correlate more highly with an independent e f f o r t to measure the same dimension than with measures designed to assess d i f f e r e n t dimensions which employ the same rate r . Campbell and Fiske's MT-MM matrix seems to be an appropriate method to show evidence of construct v a l i d i t y since one of the objectives of thi s study i s to demonstrate that the two instruments developed for t h i s investigation measure the same underlying construct, assumed to be the performance of bank managers as perceived by thei r superiors. (No superiority i n v a l i d i t y i s suggested.) Kavanagh, MacKinney, and Wolins [1971] perceive the MT-MM matrix as: " . . . c e r t a i n l y one of the best procedures available for operationally assessing construct v a l i d i t y . . . " [p. 35], and Zedeck and Baker [1972] used i t because " . . . i t provides evidence of construct v a l i d i t y " [p. 58]. One of the requirements for the application of the MT-MM matrix i s method independence. I t can be assumed with r e l a t i v e certainty that the GRS instrument i s independent from the BARS and BDI format, since the two instrument types have l i t t l e i n common. I t i s more d i f f i c u l t to demonstrate scale independence between BARS and BDI since, as one may argue, both u t i l i z e the same pool of behavioural incidents. However, i t w i l l be shown that scale independence can be assumed. Although the BARS and the BDI instuments use items from a common item pool, the BDI format usually contains only 3 to 4 of the BARS items per dimension (see Appendices D and I ). Perhaps more important, the BARS format u t i l i z e s only one item for measuring purposes; i . e . , the scale value of one item represents the rating score. This item may or may not be contained i n a BDI scale,,or, i n other words, a BARS/BDI c o r r e l a t i o n may or may not r e f l e c t i n t e r -dependence. But even i f the same item; (with d i f f e r e n t meanings, as has been shown) i s used i n both scales, the interdependence should be n e g l i g i b l e . For example, i f one item i s removed from a BDI t o t a l score (per dimension) the maximal impact i t may have i s 3 points (out of 60). To test for interdependence, a random sample of 10 ratees w i l l be selected and t h e i r BARS and BDI scores correlated, once with si m i l a r items i n both scales, and the second time with the BARS item removed from the BDI scale. Before any claims of construct v a l i d i t y for a c r i t e r i o n measure can be allowed, the c r i t e r i o n must f i r s t be judged as c r u c i a l to the indiv i d u a l ' s performance i n the job or other a c t i v i t y of i n t e r e s t [Fiske, 1.9 71; Guion, 1965]. The problem i s to determine c r i t e r i a which are actually crucial,, that i s , determine success or f a i l u r e on the job, or, as Cummings and Schwab [1973] put i t more pointedly, to ". . . [establish] the elements of the job-success construct." [p. 74] They suggest that the construct v a l i d i t y problem i s extremely d i f f i c u l t to solve operationally, because the construct consists of a mental d e f i n i t i o n which i s d i f f i c u l t - t o r elate to the measure of the construct. What, then, are the elements of the job success construct for a bank manager? Obviously, the measure of job success for a bank manager i s based on the c r i t e r i a used by the banks i n t h e i r performance appraisal forms, i . e . the judgment of a manager i s based on the degree to which he f u l f i l l s the requirements of his p o s i t i o n . For.example, a manager should be judged successful i f he demonstrates at l e a s t acceptable performance i n such areas as I n i t i a t i v e , Capacity, Attitude, Judgment, Organization, Staff Training, Oral Expression, etc., to name a few. But here the problem of operationalization appears. How do we measure " i n i t i a t i v e " or "capacity"? An even more pertinent question is: Are the factors l i s t e d above useful c r i t e r i a for a bank manager's performance? The discussion on the c r i t e r i o n issue i n Chapter 11" ended with the statement: " . . . the lack of s i g n i f i c a n t c o r r e l a t i o n between t r a i t scale and job performance raises the question of construct v a l i d i t y of t r a i t ratings for job performance." [p. 28 ] This should rai s e some doubt as to the v a l i d i t y of the banks' performance rating instruments. But who i s the ultimate judge on the matter of relevancy of c r i t e r i a ? Some of the banks have used t h e i r GRS instruments for ten and twenty years with only minor changes. This should indicate that, at l e a s t for bankers, the instruments have a ce r t a i n degree of v a l i d i t y , or, to put i t d i f f e r e n t l y , Canadian bankers chose to define l e v e l of performance as the degree to which the c r i t e r i a described i n t h e i r performance appraisal forms are achieved. Behavioural s c i e n t i s t s are prone to ask for v a l i d a t i o n against "objective measures" before they are w i l l i n g to concede a s i g n i f i c a n t degree of v a l i d i t y to any instrument. But what are "objective measures"? For t h i s study, one may suggest as objective performance c r i t e r i a such items as number and/or frequency of promotion, merit pay increases, or any measure of output. But how objective are these c r i t e r i a ? How strongly does a promotion or pay increase depend upon an employee's performance on the job? What degree of influence have factors l i k e personal l i k i n g between superior and subordinates, s e n i o r i t y , education, kinship, a b i l i t y to appear to be successful, connections, or hundreds of other factors which are outside the control of a manager, such as sudden organizational demands (vacancies), economic conditions, competition, etc., etc.? Even output i s not under the sole control of an employee, but may depend on group norms, work environment, work flow, (subjective!) q u a l i t y standards, tra i n i n g , etc. [Guion, 1965, p. 90; Thorndike and Hagen, 1955, p. 545].. Campbell [197.6] put i t i n proper perspective when he said: "The conventional wisdom says o b j e c t i v i t y i s good, but perhaps a more accurate wisdom says an objective c r i t e r i o n i s a subjective c r i t e r i o n once removed" [p. 39]. He adds that, since subjective judgements can be found i n any so-called objective measures of effectiveness, i t makes i t very risk y to use available objective measures as c r i t e r i a i n research studies. Nevertheless, this investigator i s w i l l i n g to accept the r i s k . I t was mentioned before ("Development of a Third Instrument", p. 134 ) that i t would probably be i l l u s t r a t i v e and informative to compare the BARS and BDI instruments with the performance appraisal system used by the banks, and i t was cautioned that any comparisons should be done with the s p e c i f i c shortcomings of the banks' GRS ( i . e . , global r a t i n g , less rigorous development, no validation) i n mind. To demonstrate how well the three instruments (GRS, BARS, BDI) tap the construct they are supposed to measure (job performance as perceived by bank managers) and to obtain an estimate of construct v a l i d i t y , Campbell and Fiske's [1959] m u l t i t r a i t -multimethod matrix (MT-MM Matrix) w i l l be used. I t means that — f o r the sake of comparison — the banks' d e f i n i t i o n of .managerial performance, as described i n the GRS, w i l l be employed as the c r i t e r i o n the BARS and BDI instruments w i l l be assessed by. In addition, i n the opinion of the investigator, other perhaps more objective c r i t e r i a w i l l be u t i l i z e d . One of the two banks which co-operated i n the v a l i d a t i o n phase made data on two indicators of performance ava i l a b l e : a promotability rating and an o v e r a l l performance r a t i n g . The second bank provided data on actual promotions of p a r t i c i p a t i n g bank accountants. Of the two indicators of performance of the f i r s t bank, the promotability index seems to be the more objective one since i t has to be accompanied by a detailed written explanation given by the rater, and i t has to be discussed with the rater's d i r e c t supervisor. The promotion information provided by the second bank appeals to represent also a somewhat more objective measure than the items used i n the GRS. Some Opinions on the C r i t e r i o n Issue and the Construct Validation Problem As Cummings and Schwab [1973] indicated, i t i s extremely d i f f i c u l t to operational!ze construct v a l i d i t y because of the c r i t e r i o n problem. Who i s to decide what constitutes a proper c r i t e r i o n ? In a discussion on organizational effectiveness, Campbell [1976] makes a statement which seems to be equally relevant to managerial performance: " . . . i n the end, organizational effectiveness i s what the relevant parties decide what i t should be. There i s no higher authority (in a secular society) to which we can appeal." Applied to the present study, i t suggests that the performance of bank managers i s what the same managers (or t h e i r superiors) decided what i t should be. Campbell adds: "This does not preclude the behavioral s c i e n t i s t ' s imposing his own value system as to what constitutes effectiveness; but i t should be recognized for what i t i s " [p. 43]. An i n t e r e s t i n g view on the issue of appropriate approaches to test v a l i d a t i o n i s expressed by Lawshe [1975]. According to him, the answer to the question whether to use construct v a l i d a t i o n (or content validation) r e a l l y depends on the degree of abstractness of the r a t i n g decision. As he sees i t , the behaviours constituting job performance domains cover the range from the d i r e c t observable, through the reportable, to the highly abstract behaviour. He suggests that content v a l i d a t i o n would be appropriate at the concrete end of the continuum (observable behaviour), but that construct v a l i d a t i o n should be applied at the abstract end of the continuum (with behaviours such as deductive.reasoning). The implication could be that content v a l i d a t i o n (through a panel of judges) should be used to validate the BDI because i t uses observable behaviour items while the GRS may require construct v a l i d a t i o n since i t uses more abstract items ( t r a i t s , job dimensions). The BARS format may be somewhere between the two because i t requires inferences from s p e c i f i c anchors to a range of observable behaviour. To quote Lawshe: . . . when a high l e v e l of abstraction i s involved and when the magnitude of the " i n f e r e n t i a l leap" becomes s i g n i f i c a n t , job incumbents and supervisors normally do not have the ins i g h t to make the required judgments. When these conditions obtain, we t r a n s i t i o n from content v a l i d i t y to a construct v a l i d i t y approach." [p. 566] Ebel. [1977; see also 1961] i s quite outspoken on h i s views about t e s t v a l i d a t i o n and the need for content or construct v a l i d a t i o n . ' He says: "When the test behaviors sample representatively a population of c r i t e r i o n behaviors, or when the test behaviors constitute the only kind of c r i t e r i o n behaviors that are available for observation, the question of v a l i d i t y need never a r i s e . " [p. 58] He suggests that the accuracy of the measurement of what i s intended to be measured can be expressed perhaps more appropriately by a c o e f f i c i e n t of g e n e r a l i z a b i l i t y ( r e l i a b i l i t y ) . He adds: Some would say that content v a l i d i t y i s i n f e r i o r to, or less s c i e n t i f i c a l l y ... respectable than c r i t e r i o n related v a l i d i t y . This view i s mistaken i n my opinion. Content v a l i d i t y i s the only basic foundation for any kind of v a l i d i t y . If the t e s t does not have i t , the c r i t e r i o n measure used to v a l i d i t y the t e s t must have i t . And one should never apologize for having to exercise judgment i n v a l i d a t i n g a test . Data never substitute for good judgment. [p. 59] Ebel's view seems to be somewhat extreme, but i s c e r t a i n l y thought provoking. In his opinion, behavioral s c i e n t i s t s d i s t o r t r e a l i t y because they tend to oversimplify. They are at t h e i r best, as he sees i t , when the topics studied involve only few variables and l i n e a r r e l a t i o n s h i p s . He concludes: "The v a l i d a t i o n of constructs, and of measures of quantitative constructs, may.be a matter of appropriate concern i n t h e o r e t i c a l psychology. I t should be of no r e a l concern, at the present stage of i t s development, to those of us engaged i n achievement or job t e s t i n g . " [p. 61] However, Tenopyr [1977] asserts that much content oriented t e s t development must take constructs into account. She suggests to combine content and c r i t e r i o n -related methods and recommends: "...to do a thorough job of content-oriented te s t .construction and follow i t with a token c r i t e r i o n - r e l a t e d study" [p. 52]. This advice seems to be p a r t i c u l a r l y relevant to t h i s study since the c r i t e r i a used here are c e r t a i n l y not purely objective, but the best available under the circumstances. Since the investigator i s convinced that the BARS and BDI instruments developed i n t h i s study are based, on -- to use Tenopyr' s words -- a "thorough job of content-oriented test construction", i t should be s u f f i c i e n t to follow i t up with a "token c r i t e r i o n - r e l a t e d study" to show that the BARS and BDI are v a l i d performance appraisal tools. 152 Psychometric Characteristics Halo The halo e f f e c t w i l l be assessed by averaging the c o r r e l a t i o n between ratings on a l l dimensions [Barrett, et a l . , 1958]. The rationale for th i s approach i s that raters who discriminate very l i t t l e between job dimensions tend to have high correlations between dimension scores. Conversely, raters who regard each dimension separately and are not reluctant to assign r e l a t i v e l y disparate ratings to indivi d u a l s on different. dimensions, w i l l have a comparatively low i n t e r c o r r e l a t i o n between dimension scores. Leniency Leniency w i l l be measured by assessing the negative skewness of the rating scores i n every dimension [Barrett, et a l . , 1958]. Negative skewness characterizes a concentration of scores toward the high end of the score scale with the curve trailing....6ff to the l e f t [Roscoe, 1969, p. 29]. Strictness (negative leniency) would be indicated by a concentration of scores toward,the lower end of the score scale with the curve t r a i l i n g o f f to the r i g h t . The degree of skewness w i l l be assessed by using a formula given by Blalock [19 72]. 153 Central Tendency Indicative of a central tendency e f f e c t i s the lack of extreme p o s i t i v e and negative ratings. I t w i l l be assessed, by comparing the degree of implication of the high and low ranges of the d i f f e r e n t rating scales. Information Content No attempts have been made i n previous research studies to quantify the information content of behaviourally oriented scales. I t would be useful, but very d i f f i c u l t , to determine a percentage measure of actually, used behaviour samples from the t o t a l job behaviour domain. A good i n d i c a t i o n of the degree of u t i l i z a t i o n of the job domain i s the percentage of actually used items from the item pool which survived the v a l i d a t i o n and re t r a n s l a t i o n process. R e l i a b i l i t y The t e s t - r e t e s t r e l i a b i l i t y of the d i f f e r e n t measures may be established by repeated evaluation of the same ratees by the same raters over a period of time. In addition, i t was planned to assess i n t e r -rater r e l i a b i l i t y of the d i f f e r e n t instruments by comparing the ratings of two groups of raters, superiors and subordinates of the target group. At the l a s t moment, for i n t e r n a l reasons, the supporting organizations declined to allow subordinates to evaluate t h e i r superiors. Rater Preferences Rater preferences w i l l be assessed by asking them to rank the three instruments according to the di f f e r e n t purposes they may be used for; i . e . feedback and counselling, promotion, wage and salary adminis-t r a t i o n , t r a i n i n g need determination, t r a i n i n g program development and objective formulation, and ease of usage. The Kendall c o e f f i c i e n t of concordance (W) w i l l be used to measure the agreement among the judges as to the ranking of the instruments. While t h i s does not allow to determine s t a t i s t i c a l significance of the superiority of any instruments, i t i s suggested that the best estimate of the true or absolute ranking i s provided, when W i s s i g n i f i c a n t , by the order of the various sums of ranks [Kendall, 1948, p. 87, as quoted i n Siegel, 1956, p. 238]. Table IV.1 Assets and Branches of the fi v e Canadian Chartered Banks which supported t h i s survey Assets i n Branches i n B i l l i o n $ Canada The Royal Bank of Canada 25 1,4 30 Canadian Imperial Bank 22 1,744 of Commerce Bank of Montreal 18 1,287 Bank of Nova Scotia 16 900 Toronto Dominion Bank 13 87 0 Source: The Fi n a n c i a l Post Survey of Industries, 1976 156 CHAPTER V RESULTS Leniency E f f e c t The means and standard deviations of the ratings of Group I and Group II are shown i n Table V . l and Table V.2 respectively. Since the BDI u t i l i z e s a 60-point scale and the GRS and BARS a 7-point scale, the standardized skewness c o e f f i c i e n t for each performance dimension scores of the three instruments are shown i n Table V.3 and Table V.4. An examination of the three skewness c o e f f i c i e n t s for each dimension i n both tables reveals that the values for the BDI format are generally closer to zero than i s the case of the other formats, thus supporting Hypothesis l b . The GRS format y i e l d s the highest degree of negative skewness for both groups. TABLE V . l MEANS AND STANDARD DEVIATIONS FOR EACH PERFORMANCE DIMENSION ON GRS, BARS, AND BDI RESPONSES OF GROUP 1 Performance GRS BARS BDI Dimensions Mean S.D. Mean S.D. Mean S.D. A 5.1 1.01 5.5 1.02 48.2 5.80 C 4.9 .8 4.8 .73 46.3 6.01 M 4.0 .74 4.5 1.11 33.7 10.73 P 4.8 .87 5.2 1.12 47.3 6.48 T 4.7 .77 5.4 .85 46.8 6.96 N=31 Note: A = Administration C = Customer Relations M = Marketing P = Personnel T = Training GRS was scored on a ,7-point scale BARS was scored on a 7-point scale BDI was scored on a 60-point scale TABLE V.2 MEANS AND STANDARD DEVIATIONS FOR EACH PERFORMANCE DIMENSION ON GRS, BARS, AND BDI RESPONSES OF GROUP 2 Performance GRS BARS BDI Dimensions Mean S.D. Mean S.D. Mean S.D. A 5.2 .95 4.9 1.10 47.9 12.93 C 4.6 .82 5.1 .82 47.7 9.29 M 3.2 .70 4.2 1.06 36.6 11.94 P 4.9 .95 5.1 1.18 50.6 8.95 T 5.1 .80 4.8 .85 45.9 9.05 N=4 2 Note: A = Administration C = Customer Relations M = Marketing P = Personnel T = Training GRS was scored on a 7-point scale BARS was scored on a 7-point scale BDI was scored on a 60-point scale TABLE V.3 NORMALIZED SKEWNESS COEFFICIENT FOR EACH PERFORMANCE DIMENSION ON GRS, BARS, AND BDI RESPONSES OF GROUP 1 GRS BARS BDI A -2.68 - .52 - .18 C - .72 1.5 2 -1.02 M -1.80 - .74 -1.53 P - .92 - .85 .16 T -2.08 .34 .94 Mean -1.6 4 - .79 - .76 N=31 Note: The mean skewness c o e f f i c i e n t s were calculated by using the absolute scores. The skewness c o e f f i c i e n t s were calculated according to the formula = 3 (X - Md) [Blalock, 1972] k a ' X = Mean Md = Median a = Standard deviation TABLE V.4 NORMALIZED SKEWNESS COEFFICIENT FOR EACH PERFORMANCE DIMENSION ON GRS, BARS, AND BDI RESPONSES OF GROUP 2 GRS BARS BDI A -1.58 -1.51 - .68 C -1.02 - .20 - .92 M -1.75 -1.22 .82 P - .62 .10 .35 T -1.92 -1.82 - .76 Mean -1.38 - .97 - .70 N=42 Note: The mean skewness c o e f f i c i e n t s were calculated by using the absolute scores. The skewness c o e f f i c i e n t s were calculated according to the formula = 3 (X - Md) [Blalock, 1972] X = Mean Md = Median a = Standard deviation Dimension Independence (Halo) The correlations between performance dimensions i n the two studies are reported i n Table V.5 and Table V.6. The BDI format seems to suffer less from a halo e f f e c t than the BARS and GRS instruments according to the sign test, for both groups. The resu l t s i n Table V.5 indicate that 9 out of 10 correlations are s i g n i f i c a n t for the GRS format, as compared to 6/10 for the BARS and 3/10 for the BDI instruments. For the second group the res u l t s are 10/10 for GRS, 9/10 for BARS, and 2/10 for BDI. The median correlations for each instrument are: GRS = 45, BARS = 34, and BDI = 20 for Group I, and GRS = 75, BARS = 53, and BDI = 12 for Group I I . (Hypothesis l a supported.) TABLE V.5 CORRELATIONS BETWEEN DIMENSION SCORES OF EACH OF THE THREE INSTRUMENTS FOR GROUP 1 GRS BARS BDI A/C 5 9 * * 44* 19 A/M 43* 23 -10 A/P 74** 66** 45* A/T 63** 50** 51** C/M 46** 17 16 C/P 48** 45* 21 C/T 43* 41* 38* M/P 29 27 07 M/T 68** 12 09 P/T 47** 56** 29 Median 45 34 20 N=31 A = Administration *p < .05 C = Customer Relations **p < .01 M = Marketing P = Personnel T = Training TABLE V.6 CORRELATIONS BETWEEN DIMENSION SCORES OF EACH OF THE THREE INSTRUMENTS FOR GROUP 2 GRS BARS BDI A/C 87** 42** 17 A/M • 62** 64** 09 A/P 88** 07 42** A/T 78** 90** 09 C/M 62** 53** 22 C/P 83** 46** -15 C/T 75** 36* -03 M/P 68** 5 9 * * 33* M/T 62** 54** -07 P/T 76** 76** 16 Median 75 53 12 N=42 A = Administration *p < .05 C = Customer Relations **p < .01 M = Marketing P = Personnel T = Training 164 C e n t r a l T e n d e n c y I n o r d e r t o m e a s u r e t h e c e n t r a l t e n d e n c y e f f e c t o f t h e t h r e e i n s t r u m e n t s t h e s c o r e d i s t r i b u t i o n f o r t h e f i v e d i m e n s i o n s o f e a c h i n s t r u m e n t a n d e a c h g r o u p a r e shown i n T a b l e V.7 a n d T a b l e V.8. I t i s i n t e r e s t i n g t o s e e t h a t t h e GRS s c o r e s f o r b o t h g r o u p s c o n t a i n no s c o r e s f o r t h e two l o w e s t s c a l e p o i n t s ( " u n a c c e p t a b l e " a n d " m o d e r a t e " ) a nd f o r t h e h i g h e s t s c a l e p o i n t ( " e x c e l l e n t " ) . F o r t h e BARS a n d t h e BDI i n s t r u m e n t s o f b o t h g r o u p s o n l y t h e l o w e s t s c a l e p o i n t o r s c o r e r a n g e i s n o t u t i l i z e d . A s c o m p a r e d t o t h e BARS and B D I , t h e GRS e x h i b i t s a much s t r o n g e r c e n t r a l t e n d e n c y e f f e c t . As d i s c u s s e d i n t h e c h a p t e r o n t h e d e v e l o p m e n t o f t h e BARS i n s t r u m e n t , t h e r e a r e no s c a l e a n c h o r s f o r t h e m i d p o i n t o f t h e d i m e n s i o n s c a l e s . The d a t a do n o t i n d i c a t e a s u p e r i o r i t y o f t h e BDI o v e r t h e BARS f o r m a t . A s a r e s u l t , H y p o t h e s i s l c h a s t o be r e j e c t e d . TABLE V.7 SCORE DISTRIBUTION FOR FIVE DIMENSIONS OF EACH GRS, BARS, AND BDI INSTRUMENT FOR GROUP 1 Scores GRS BARS Scores BDI 1 0 0 .0-1.0 0 2 0 11 11-20 4 3 15 23 21-30 34 4 48 0 31-40 47 5 57 62 41-50 41 6 35 39 51-60 29 7 0 20 166 TABLE V.8 SCORE DISTRIBUTION FOR FIVE DIMENSIONS OF EACH GRS, BARS, AND BDI INSTRUMENT FOR GROUP 2 Scores GRS BARS Scores BDI 1 0 0 0-10 0 2 0 22 11-20 19 3 23 34 21-30 44 4 78 0 31-40 61 5 70 82 41-50 55 6 39 47 51-60 31 7 0 25 N=42 Information Content The degree of u t i l i z a t i o n of the available information i s measured by c a l c u l a t i n g the percentage of actually used c r i t i c a l incidents from the t o t a l available c r i t i c a l incidents which survived the l a s t v a l i d a t i o n process. The res u l t s are shown i n Table V.9. The information content for the GRS could not be determined since i t does not use job behaviour samples. The data show that the BDI u t i l i z e d more than three times as many c r i t i c a l incidents as the BARS, or 83 percent as compared to 25 percent for the l a t t e r . R e l i a b i l i t y •The t e s t - r e t e s t r e l i a b i l i t y i s assessed by co r r e l a t i n g each rater's scores from two d i f f e r e n t measurements of the same job dimensions using the same instruments, the time difference between the two measurements being two weeks. Table V.10 indicates the range of the score correlations for each instrument and the median score correlations for a l l raters on each instrument. The GRS format shows scale r e l i a b i l i t i e s ranging from .72 to .98, with a median r = .86. The BARS format comes close with a t e s t -retest r e l i a b i l i t y of .85(md), and ranges from .41 to 1.00, while the BDI shows a r e l i a b i l i t y score of .95 (md), with a range from .68 to .99. (Hypothesis Id i s supported.) 168 TABLE V.9 INFORMATION CONTENT FOR BARS AND BDI AS MEASURED BY PERCENTAGE.OF ACTUALLY UTILIZED CRITICAL INCIDENTS FROM TOTAL AVAILABLE INCIDENT POOL Total Available U t i l i z e d by U t i l i z e d by Pool BARS BDI # # % # % 120 30 25 100 83.3 TABLE V.10 TEST-RETEST RELIABILITIES FOR GRS, BARS, AND BDI GRS BARS BDI Range of r: .72 - .98 .41 - 1.00 .68 - .99 Median .86 .85 .95 N=22 Convergent and Discriminant V a l i d i t y The next mode of analysis i s the m u l t i t r a i t -multimethod approach suggested by Campbell and Fiske [1959]. The present study resulted i n two 15 x 15 m u l t i t r a i t (5 performance dimensions), multimethod (GRS vs. BARS vs. BDI) matrices, shown i n Table V.11 (Group 1) and Table V.12 (Group 2). Campbell and Fiske define convergent v a l i d i t y as the observation of s i g n i f i c a n t correlations when two d i f f e r e n t methods are used to measure the same variable. A l l instruments seem to demonstrate adequate convergent v a l i d i t y as indicated by the number of s i g n i f i c a n t correlations i n the v a l i d i t y diagonals i n both tables. The only exception i s the Training dimension i n the GRS/BARS and BARS/BDI co r r e l a t i o n i n Group 1 and the GRS/BDI co r r e l a t i o n i n Group 2. Discriminant v a l i d i t y i s indicated i n two ways. F i r s t , the entries i n the v a l i d i t y diagonal can be compared with t h e i r corresponding row and column entries i n the heterotrait-heteromethod triangles (indicated by broken l i n e s ) . This yi e l d s 8 comparisons for each performance factor i n which the diagonal value should be higher than the row and column values, i f discriminant v a l i d i t y i s present. In Table V . l l , the diagonal entry i s higher for 30 out of 40 such • TABLE V . l l MULTITRAIT (PERFORMANCE DIMENSIONS), MULTIMETHOD (GRS vs. BARS vs. BDI) MATRIX FOR BANK ACCOUNTANTS BARS GRS BARS BDI Note; A C M P T A C M P T 02 ( 4 3 ^ ^ (40)N^9 55 50 3 7 08 4 1 - 0 3 3 7 N ( 5 8 ) \ 4 4 3 8 | 4 2 2 5 \ ( 6 0 K J _ 8 | 3 5 4 3 0 5 _ 2 6 \ (15) ( 5 3 ) \ l 3 -11 3 5 3.1 | 3 6 \ ( 3 8 ^ 11 ft ft 17 26 (62) Nj l 30 j 47 11 -0 7 \ ( 5 7 ) Q6J 63 18 -02 35\ (38) M 44 N 23 1 7 \ 66 45 2 7 s 50 41 .12 ft* "VT (72) S3?. 21 K I2Z v ( 3 6 ) \ 0 7 56 36 28 61 20 38 11 If 12 38 30 00 -42X^(35) sQ4 07 1 5 \ (24) BDI C M V -10 16 45 21 51 38 P T 29 N=31; (. ) = Validity Diagonal; Heterodimensional-Heteromethod Triangle; and P ^ " - ^ =». Heterodimensional-Monomethod Triangle; *p < .05, **p < .01 MULTITRAIT (PERFORMANCE GRS A C M GRS BARS BDI Note: TABLE V.12 DIMENSIONS), MULTIMETHOD (GRS v s . BARS v s . BDI) MATRIX FOR BANK ACCOUNTANTS BARS . T A C M P T A C M P T A C 87^ M 62 62 P 88 83 T 78 75 ^76) x46^ 35 53 X 1 \ **N |58\(43) N^ 9 53 37 ^48) N31 * * \ 35 17\(40) ^1 70 36 37 60\-(84) 61 60 79 70 58 . ( 3 ^ - 4 7 54 W U / - . 1 . ^ - r 64 19 I 23\^47) \52 49 -02 | | 12 -17\(58 ) \32 -19 | i \ A A V | 21 -07 51 \{50) x ^ 5 | I 18 22 -35 45\(20) M BDI C M P (56)'VU -64 52 |.08N^46)>-25 40 22 33 -09s ,(58) \52 40 33 - 4 4 - 5 7 X A A V (55)\ 20 22 -24 -:n 4>\ (38) 17N 09 22 s 42 -15 -33 09 -03 -07 16" N =/ ( 2. ( ) = Validity Diagonal; L ^ " ^ = Heterodimensional-Heteromethod Triangle, and P " " - ^ = Heterodimensional-Monomethod Triangle; *p < .05, **p < .01 comparisons (or 75 percent) for the GRS/BARS co r r e l a t i o n s , as compared to 38 for the GRS/BDI (95 percent) and 36 for the BARS/BDI correlations (90 percent). The "Training" scale accounts for 9 out of the 16 discrepancies. In Table V.12, the diagonal entry i s higher for 28 out of 40 comparisons (or 70 percent) for the GRS/BARS corr e l a t i o n s , as compared to 32 for the GRS/BDI (80 percent) and 34 for the BARS/BDI correlations (85 percent). A second index of discriminant v a l i d i t y involves comparing the v a l i d i t y diagonal entries (same t r a i t s but d i f f e r e n t methods) to the corresponding row and column entries i n the heterotrait-monomethod triangles (indicated i n s o l i d l i n e s ) . This implies that the correlations should be higher when d i f f e r e n t methods are used to measure the same dimension than when di f f e r e n t dimensions are measured by the same method. In Table V . l l , for the GRS only 5 out of 20 comparisons yielded higher entries i n the v a l i d i t y diagonal, as compared to 14 for the BARS and 15 for the BDI. These differences (15 exceptions for the GRS, 6 for the BARS, and 5 for the BDI) demonstrate the d i f f e r e n t i a l discriminant power of the three instruments. A c h i -square t e s t of independence indicates that the BARS and BDI formats have s i g n i f i c a n t l y fewer exceptions than the GRS at p < .05 (x 2 =4.6 and 5.0, df = 1). The differences between BARS and BDI were not s i g n i f i c a n t . For Group 2 (Table V.12), only one entry i n the v a l i d i t y diagonal was higher than a corresponding entry i n the monomethod triangles for the BRS, or 1 out of 20, as compared to 9/20 for the BARS and 20/20 for the BDI. The differences were s i g n i f i c a n t at the p < .05 l e v e l for BARS over GRS (x 2 = 4.26, df = 1), at the p < .001 l e v e l for BDI over GRS (x = 25.87, df = 1), and at the p < .01 l e v e l for BDI over BARS (X 2 = 7.58, df = 1). The re s u l t s of th i s analysis seem to show evidence i n support of Hypotheses 2a and 2b, namely, that the BDI w i l l demonstrate, s i g n i f i c a n t convergent v a l i d i t y with and superior discriminant v a l i d i t y to BARS. A further question concerns the pattern of relationships among t r a i t s when measured by the three rating methods. The ranks of the correlations i n the three monomethod triangles were compared. In the f i r s t group, the Spearman rank c o r r e l a t i o n between GRS and BARS i s .138, between GRS and BDI .50, and between BARS and BDI .76. While a l l correlations are s t a t i s t i c a l l y s i g n i f i c a n t , the proportion of variance that i s common i s .14, .25, and .58, suggesting that the structure of relationships among performance dimensions i s d i f f e r e n t when measured by the GRS and BARS, GRS and BDI, and BARS and BDI. The corresponding results for the second study lead to similar conclusions (rGRS/BARS = -.39; rGRS/BDI = .32, and rBARS/BDI = .25). In addition to the v i s u a l comparisons of the correlations i n the multitrait-multimethod matrices and the q u a n t i f i c a t i o n of the differences between the score correlations of the d i f f e r e n t instruments through chi square and sign test, an analysis of variance approach was used as suggested by Kavanagh, MacKinney and Wolins [1971], based on an e a r l i e r discussion by Boruch, Larkin, Wolins, and MacKinney [1970] . Tables V.13 and V.14 summarize the res u l t s of the analysis of variance, using the correlations from Tables V . l l and V.12, respectively. Since i t i s of in t e r e s t to compare the amount of variance due to each source, variance components were also computed. According to Kavanagh et a l . , these components allow inferences to be made about meaningful e f f e c t s , p a r t i c u l a r l y r e l a t i v e to unexplained variance (error), while c o n t r o l l i n g the sample si z e . As shown i n the above-mentioned tables, the r e s u l t s of the significance tests on the main e f f e c t and interactions indicate that TABLE V.13 ANALYSIS' OF VARIANCE OF CORRELATIONS FROM TABLE V . l l Source df MS Variance F Component Indexes Managers 30 3.1 26.72* .20 63 Managers x T r a i t s 120 1.705 14.69* 53 .81 Managers x Methods 6 0 2.325 20.04* ,44 78 Error 240 .116 12 * p < .01 TABLE V.14 ANALYSIS OF VARIANCE OF CORRELATIONS FROM TABLE V.12 Variance Source df MS F Component Indexes Managers 41 3.84 64.0* .25 .81 Managers x T r a i t s 164 1.61 26.8* .51 .89 Managers x Methods 82 2.31 38.5* .45 .88 Error 328 .06 .06 * p < .001 each source of variance i s s i g n i f i c a n t for both groups (p < .01 and p < .001). Adequate d i f f e r e n t i a t i o n among ratees across a l l methods and t r a i t s i s demonstrated by the s i g n i f i c a n t managers e f f e c t . In m u l t i t r a i t -multimethod terms, the scales possess convergent v a l i d i t y . Although the Manager-Method i n t e r a c t i o n i s s i g n i f i c a n t , the larger variance component o f the Manager-Trait i n t e r a c t i o n indicates that the ordering of ratees was due more to t r a i t discrimination than to method bias, or, i n other words, the scales show evidence of discriminant v a l i d i t y . However, the s i g n i f i c a n t method bias confounds the f i r s t r e s u l t (Manager or ratee e f f e c t ) . I t implies that the type, of instrument used determines, at l e a s t i n part, the ordering of ratees. These findings, together with the implications of the Spearman Rank Order analysis, support the assumption that the three instruments used i n t h i s study are somewhat d i f f e r e n t i n nature which, of course, raises the question of construct v a l i d i t y . For t h i s reason, i t i s j u s t i f i e d to ask for additional evidence that the instruments tap the same underlying construct. Tables V.15 and V.16 show the correlations between the dimension scores of the three instruments and an index of actual promotion (Group 1) and two indexes of o v e r a l l performance and promotability (Group 2) . .The res u l t s i n Table V.15 seem to indicate -given that promotion i s an accurate r e f l e c t i o n of performance — that the Administration and Personnel dimensions are more important for the pos i t i o n of bank manager than Customer Relations, Training, and Marketing at l e a s t as long as the GRS i s used. Ratings based on BARS show moderate correlations (three s i g n i f i c a n t ) with actual promotion across a l l dimensions while the BDI ratings have s i g n i f i c a n t correlations for a l l but one of the dimensions. The findings seem to demonstrate that BDI ratings are better predictors of promotion than ratings based on other instruments, thus supporting Hypothesis 3b. The re s u l t s i n Table V.16 show that the Index of Overall Performance 1 correlates s i g n i f i c a n t l y with the Administration dimension of the GRS and with the Personnel dimension of the same instrument. I t seems to confirm the findings i n the previous study that performance ratings based on the GRS are biased i n favour of or put. heavier emphasis on administrative and"personnel .handling s k i l l s . As i n the previous 1See Appendix N for a more detailed explanation. TABLE V.15 CORRELATIONS BETWEEN DIMENSION SCORES OF GRS, BARS, BDI, AND THE INDEX OF ACTUAL PROMOTION (AP)t GRS AP BARS AP BDI AP A 52** 39* 64** C 27 22 31 M 07 38* 60** P 41* 25 42* T 16 38* 59** Median 21 38 59. N=31 *p < .05 **p < .01 tSee Appendix _ N for a more detailed description. 181 TABLE V.16 CORRELATIONS BETWEEN DIMENSION SCORES OF GRS, BARS, BDI, AND INDEXES OF OVERALL PERFORMANCE (OP) AND PROMOTABILITY (PR) OP PR OP PR OP PR A 66** 40** 35* 43** 62** 48** C 04 22 21 13 62** 46**' M -12 -31* 33* 17 48** 33* P 38*" 31* 46*'* 50**" 63** 49** T 17 03 56** 56*'v 60** 63** Median 17 31 35 43 62 48 N=42 *p < .05 **p < .01 182 study, BARS ratings show moderate but s i g n i f i c a n t correlations with o v e r a l l performance, while BDI ratings show r e l a t i v e l y high correlations ( a l l s i g n i f i c a n t ) with the OP index. Again, on the average, the BDI instrument seems to tap the performance dimensions of a bank manager better than the other instruments. The f i n a l comparison with the Promotability Index shows generally moderate correl a t i o n s , with only the BDI ratings being a l l s i g n i f i c a n t . The results seem to j u s t i f y the conclusion that the three instruments measure — at l e a s t p a r t i a l l y — the same underlying construct, but with d i f f e r e n t emphases on d i f f e r e n t aspects of i t . The BDI format appears to have the most balanced scales. The f i n a l mode of analysis i s the ranking of the d i f f e r e n t instruments according to preferences of raters and the c a l c u l a t i o n of Kendall's C o e f f i c i e n t of Concordance (see Table V.17). Raters were the same group of branch managers who p a r t i c i p a t e d i n the f i r s t study plus the l o c a l d i s t r i c t personel manager and four d i s t r i c t t r a i n i n g o f f i c e r s (N = 36). The following task was put to them: Please rank each instrument according to your preference for i t s use as a: 1. feedback and counselling t o o l ; 2. tool to determine wage and salary increase; 3. tool to determine promotion; 4. tool to determine t r a i n i n g needs of employees; 5. tool to develop s p e c i f i c t r a i n i n g programmes and to formulate precise t r a i n i n g objectives; and the l a s t request was worded somewhat d i f f e r e n t l y : 6. Please rank each instrument according to the ease with which you arrive at a r a t i n g decision when using i t . Three check-off squares were provided aft e r each option, with the sequence of the instrument names randomized to avoid response sets (see Appendix o for a sample). 184 TABLE V.17 PERCENTAGE DISTRIBUTION OF RANKINGS ( 1 s t , 2nd, 3 r d ) OF GRS, BARS, AND BDI BY EXPERT JUDGES, AND KENDALL'S COEFFICIENT OF CONCORDANCE (W) FOR A L L RATINGS Q u e s t i o n # P e r c e n t a g e D i s t r i b u t i o n o f R a t i n g s W 1 s t GRS 2nd 3 r d 1 s t BARS 2nd 3 r d , 1 s t BDI 2nd , 3 r d 1 11 89 0 0 0 100 89 11 0 .90* 2 42 58 0 0 0 100 58 4 2 0 .76* 3 67 33 0 0 0 100 33 67 0 .78* 4 17 72 11 0 11 89 83 17 0 .74* 5 9 69 22 0 22 78 91 9 0 .73* 6 28 72' 0 0 0 100 . 7 2 28 0 .80* N=36 J u d g e s , c o n s i s t i n g o f 31 b r a n c h m a n a g e r s , 1 d i s t r i c t p e r s o n n e l m a n a g e r , a nd 4 d i s t r i c t t r a i n i n g o f f i c e r s . *p < .001 SUMS OF RANKS t GRS BARS BDI 1 62 0 154 2 142 12 62 3 12 204 0 t a s s u g g e s t e d b y K e n d a l l t o e s t i m a t e t h e a b s o l u t e r a n k i n g [ K e n d a l l , 1 9 4 8 , p. 87] The re s u l t s show that the raters were i n general agreement as to their rankings. The c o e f f i c i e n t of . concordance i s s i g n i f i c a n t at the .001 l e v e l for every ranking. For a l l purposes but promotion was the BDI instrument preferred. The various sums of ranks — as an estimate of the absolute ranking -- were BDI = 154 i n the f i r s t rank as compared to GRS = 62 and BARS = 0; second rank GRS = 142, BDI = 62, and BARS = 12; and t h i r d rank: . BARS = 204, GRS =. 12, and BDI = 0. The results support Hypotheses 3a, b, c, d, e and f. The reader should be reminded that, because of the limited comparability of the GRS with the s p e c i a l l y developed BARS and BDI instrument, the GRS format was not included i n the hypotheses formulated for the objectives of t h i s study.. Hypothesis 3b stated that raters w i l l prefer the BDI over BARS because i t i s better suited for promotion purposes. This proposition was confirmed, but the GRS was preferred to the BDI, a somewhat unexpected result.. The most probable explanation — supported by remarks made by raters — i s that with regard to promotion decisions managers were more comfortable using the GRS they were much more fa m i l i a r with. Some apparently f e l t that promotability should include more than just the exhibition of appropriate behaviour; i . e . the demonstration of " i n i t i a t i v e , leadership, judgment, mental alertness, etc." The remark by the investigator that those t r a i t s have to be expressed i n behaviours, l e f t them unimpressed. I t i s int e r e s t i n g to note that the BARS format was never ranked higher than second, and then always second to the BDI. I t should also be emphasized that the raters were not exposed to any information on the evaluation of the instruments i n question, e.g. l i t e r a t u r e review or verbal i n s t r u c t i o n . When introducing the instruments the investigator described only the nature of each instrument and the appropriate rating procedure as explained on top of every rating sheet. A l a s t investigation was done to answer the question, what influence the removal of an item used on both BARS and BDI instruments would have on the score correlations between the two instruments. Ten rating sheets were selected where a BDI scale contained the same item which was checked o f f on the corresponding BARS format. F i r s t the c o r r e l a t i o n between the two format scores was calculated. Then the scores of the BDI items i n question were subtracted from the BDI dimension scores and the cor r e l a t i o n between the BARS scores and the new BDI scores calculated again. The new co r r e l a t i o n value was not s i g n i f i c a n t l y d i f f e r e n t from the former value (.626 vs. .622), thus supporting the assumption that the removal of a common item from a BDI scale would have a ne g l i g i b l e impact on BARS and BDI i n t e r c o r r e l a t i o n s . Discussion This study attempted to show, f i r s t , that i t i s possible to develop behaviourally oriented scales to assess managerial performance with the same or improved advantages and fewer or reduced disadvantages exhibited by BARS, with no s a c r i f i c e i n rigour: of development; secondly, that behaviourally oriented scales can be developed for similar jobs i n d i f f e r e n t organizations. The results seem to support most of the propositions formulated e a r l i e r . Hi — The BDI w i l l be superior to BARS with respect to such rating c h a r a c t e r i s t i c s as: a) halo b) leniency c) central tendency d) r e l i a b i l i t y The BDI showed supe r i o r i t y over BARS with regard to a l l rating c h a r a c t e r i s t i c s but central tendency, which seemed to be similar for both instruments, i . e . raters did not u t i l i z e the lower extreme ranges of both scales. However, both instruments did s i g n i f i c a n t l y better than the banks' GRS format whose two lowest scale points and the highest scale point were never used. I t i s suggested that the n o n - u t i l i z a t i o n of the lowest scale points of a l l three instruments i s a r e s u l t of prescreening of ratees and not a scale c h a r a c t e r i s t i c . The p r o b a b i l i t y o f f i n d i n g a p e r s o n i n a m a n a g e r i a l p o s i t i o n e x h i b i t i n g " u n a c c e p t a b l e " o r " e x t r e m e l y i n e f f e c t i v e " p e r f o r m a n c e , o r n o t e x h i b i t i n g any e f f e c t i v e b e h a v i o u r b u t a l l i n e f f e c t i v e b e h a v i o u r s , s h o u l d be v e r y l o w i n o r g a n i z a t i o n s w i t h r e g u l a r p e r f o r m a n c e r e v i e w s a n d s t r i n g e n t p r o m o t i o n c r i t e r i a . H2 a) — The BDI w i l l d e m o n s t r a t e s i g n i f i c a n t c o n v e r g e n t v a l i d i t y w i t h BARS. The v i s u a l i n s p e c t i o n o f t h e m u l t i t r a i t -m u l t i m e t h o d m a t r i x s u p p o r t s t h i s h y p o t h e s i s , b a s e d on t h e s i g n i f i c a n t c o r r e l a t i o n i n t h e v a l i d i t y d i a g o n a l . S u p p o r t f o r t h i s p r o p o s i t i o n c a n a l s o be f o u n d i n t h e m u l t i v a r i a t e a n a l y s i s o f t h e m a t r i x , b u t s i n c e t h i s r e s u l t i s c o n f o u n d e d b y s i g n i f i c a n t and e v e n l a r g e r m e t h od b i a s i t i s d i f f i c u l t t o make a s t r o n g c a s e f o r c o n v e r g e n t v a l i d i t y b a s e d o n t h e s e a n a l y s e s . T h e r e f o r e , H2a c a n n o t be a c c e p t e d w i t h c e r t a i n t y . However t h e c o r r e l a t i o n s o f BARS and BDI s c o r e s ( p l u s GRS) w i t h o t h e r c r i t e r i a , s u c h as o v e r a l l p e r f o r m a n c e , p r o m o t a b i l i t y , a n d a c t u a l p r o m o t i o n , seem t o i n d i c a t e t h a t a l l . i n s t r u m e n t s m e a s u r e t o a c e r t a i n d e g r e e t h e same u n d e r l y i n g c o n s t r u c t , w i t h t h e BDI s h o w i n g t h e l a r g e s t common v a r i a n c e w i t h t h e s e c r i t e r i a , a l t h o u g h n o t b e y o n d 43%. S i n c e o t h e r s t u d i e s r e p o r t c o r r e l a t i o n s o f p e r f o r m a n c e m e a s u r e m e n t s w i t h " o b j e c t i v e " c r i t e r i a 2 i n t h e u p p e r r a n g e o f .50 t o .70 [ B a r r e t t e t a l . , 1 9 5 8 ; D i c k i n s o n and T i c e , 1 9 7 3 ] , t h e c o n c l u s i o n seems t o be j u s t i f i e d t h a t t h e BDI shows e v i d e n c e o f c o n s t r u c t v a l i d i t y . H2 b) — The BDI w i l l d e m o n s t r a t e s u p e r i o r d i s c r i m i n a n t v a l i d i t y t o BARS. T h i s h y p o t h e s i s i s c l e a r l y s u p p o r t e d b y t h e r e s u l t s o f t h e m u l t i t r a i t - m u l t i m e t h o d a n a l y s i s . (The a n a l y s i s o f v a r i a n c e i n d i c a t e s a l s o a s i g n i f i c a n t d i s c r i m i n a n t e f f e c t , b u t i t doe s n o t a l l o w t o c o n c l u d e s u p e r i o r i t y o f one i n s t r u m e n t o v e r t h e o t h e r . ) H3 — R a t e r s w i l l p r e f e r t h e BDI o v e r BARS b e c a u s e : a) i t i s b e t t e r s u i t e d f o r f e e d b a c k and c o u n s e l l i n g p u r p o s e s ; b) i t i s b e t t e r s u i t e d f o r p r o m o t i o n p u r p o s e s ; c) i t i s b e t t e r . s u i t e d f o r wage and s a l a r y e v a l u a t i o n p u r p o s e s ; d) i t i s b e t t e r s u i t e d f o r t r a i n i n g need d e t e r m i n a t i o n ; e) i t i s b e t t e r s u i t e d t o d e v e l o p s p e c i f i c t r a i n i n g programmes and f o r m u l a t e t r a i n i n g o b j e c t i v e s ; f ) i t i s e a s i e r t o u s e . D e s p i t e t h e f a c t t h a t t h e r a t e r s were v e r y f a m i l i a r w i t h t h e GRS — s i n c e t h e y had u s e d i t f o r y e a r s — a new i n s t r u m e n t , t h e BDI, was p r e f e r r e d f o r a l l p u r p o s e s b u t p r o m o t i o n . F o r t h e l a t t e r p u r p o s e 2 o u t p u t , o v e r a l l p e r f o r m a n c e , a b s e n t e e i s m , t e n u r e . they f e l t more comfortable with t h e i r GRS. Some raters remarked that, just reading through the BDI made them aware of many important items they r a r e l y thought of when completing performance appraisals. The four tr a i n i n g instructors who participated i n the evaluation were unanimous i n the i r agreement that the BDI was the most useful of the three instruments for tr a i n i n g purposes. The BARS instrument was not appealing to most raters. I t appeared that they had d i f f i c u l t i e s associating diverse observed behaviour with few s p e c i f i c behaviour descriptions representing d i f f e r e n t levels of e f f e c t i v e performance. The reason a majority of raters preferred the GRS format over the BDI for promotion purposes may l i e i n the p o s s i b i l i t y that raters include more than just observed behaviour into t h e i r promotion decisions. I t i s assumed that raters use a gest a l t approach when a r r i v i n g at a conclusion on the promotability of an employee. To make such a judgement, a rater has to imagine the ratee (his subordinate) i n roles he — the rater — was never able to observe, roles which very l i k e l y require behaviours the subordinate i s rar e l y required to exhibit i n his current po s i t i o n , or only to a limited degree. I f t h i s i s true — i t would have to be investigated i n a separate study — then the BDI i s indeed an inadequate instrument for promotion evaluation, since i t r e f l e c t s only currently required behaviour, not pote n t i a l behaviour. However, i t should not be too d i f f i c u l t to develop a BDI based on behaviours t which are desirable for promotion purposes, or, a BDI developed for a-higher position could be u t i l i z e d to assess an employee's pote n t i a l for t h i s s p e c i f i c promotion. The GRS format may have an advantage i n such a case since — i n i t s o r i g i n a l form — i t l i s t s many personality t r a i t s , giving a rater the opportunity to assess an.employee's pote n t i a l on a broader spectrum of c r i t e r i a than provided by the BDI. Perhaps a combination, of GRS/BDI may be optimal (for promotion purposes). Actually, the BARS format should have an advantage i n t h i s matter because of i t s expectation phraseology. However, i t probably suffers from the same problem the BDI presumably exhibits: use of currently desirable behaviours, the problem being i n t e n s i f i e d by the much smaller number of behaviour samples. H4 — Behaviourally oriented scales can be used for sim i l a r jobs i n d i f f e r e n t organizations. This hypothesis i s supported by the 80% (minimum) agreement among the judges of a l l banks and 60% (minimum)' agreement among judges of i n d i v i d u a l banks on the question whether each c r i t i c a l incident l i s t e d represented a.valid job behaviour of an assistant branch manager. Additional support can be seen i n the s i g n i f i c a n t correlations of the two behavioural scales with other performance c r i t e r i a , such as actual promotion, o v e r a l l performance and promotability,. assumed to be an i n d i c a t i o n of construct v a l i d i t y . I t was also predicted that the BDI would cause less waste of information ( i . e . , would demonstrate higher information content) than BARS. This prediction i s c l e a r l y supported, not only by showing that the BDI instrument u t i l i z e d more than three times the number of behaviour items than the BARS format, but also through the preference expressed by most raters for the BDI, a preference based mainly on the richness and precision of the information content of the new instrument, as explained by raters. CHAPTER V SUMMARY Overview Four years ago, when t h i s study began, the books and a r t i c l e s dealing with BARS instruments were quite optimistic with regard to the i r assumed superiority over other measures, the optimism being supported by i n i t i a l l y promising r e s u l t s . With growing popularity, more detailed and better designed studies probed deeper into the c h a r a c t e r i s t i c s of BARS and compared t h i s instrument with so far more commonly used ones, such as GRS and Summated Rating Scales (SRS). As i t turned out, when other scale formats were exposed to the same rigorous development procedures usually required for BARS, re s u l t s were often comparable or even better than for BARS. I f the r e l a t i v e l y simpler GRS or SRS instruments could perform equally, there were no reasons to invest into the s i g n i f i c a n t time and e f f o r t s needed for BARS. However, most researchers agreed that despite some problems BARS had one c h a r a c t e r i s t i c the other approaches were lacking: more and better information for raters and ratees. If the rating c h a r a c t e r i s t i c s and a p p l i c a b i l i t y of the BARS format could be improved, t h i s type of performance measure unquestionably would be favoured by s c i e n t i s t s and prac t i t i o n e r s a l i k e . The a p p l i c a b i l i t y of BARS could be extended by showing that the instrument did not have to be job s p e c i f i c . However, the improvement of the rating c h a r a c t e r i s t i c s of BARS proved to be more d i f f i c u l t , the results being equivocal or only moderately p o s i t i v e . The results of the studies so far indicated that attempts to improve BARS beyond what had been done seemed to be f u t i l e . I t was then that the idea of a new type of behaviourally oriented rating scale was born, based on the u t i l i z a t i o n of a larger sample of the job behaviour domain than that used by BARS, and a d i f f e r e n t rating procedure, less prone to r a t i n g errors. Conclusions From the results of the study s e v e r a l — c a u t i o u s — conclusions can be drawn. F i r s t , behaviourally oriented scales need not be organization s p e c i f i c . This i s an important finding, s i m i l a r to the one by Goodale and Burke [1975] who demonstrated that BARS need not be job s p e c i f i c . The extension of the use of behaviour-oriented scales to other, but s i m i l a r , jobs i n the same organization and now even to s i m i l a r jobs i n d i f f e r e n t organizations i n the same industry should make the s i g n i f i c a n t l y higher investment i n time and e f f o r t for the development of behaviour-oriented scales more worthwhile. I t allows organizations to pool t h e i r resources to develop such instruments, or to delegate such a job to an i n s t i t u t i o n to which a l l related organizations belong as members; e.g. the Institute of Canadian Bankers, the Trust Companies Association of Canada, the Canadian Manufacturers Association, and others. As a second r e s u l t of the study, i t has been shown that i t i s possible to improve the psychometric c h a r a c t e r i s t i c s of behaviour-oriented scales i n several ways. One of the major shortcomings of the BARS format was the severely l i m i t e d u t i l i z a t i o n of the t o t a l pool of information available on the job behaviour domain. The switch from a Likert-type scale—which l i m i t e d the number of behaviour samples used per scale to the number of anchor p o i n t s — t o an anchorless format which allows t h e o r e t i c a l l y the use of the t o t a l job behaviour domain, seems to be a breakthrough i n the search for a performance appraisal instrument with a large and s p e c i f i c amount of feedback information and a superior rating c h a r a c t e r i s t i c than BARS. And, t h i r d l y , because of the favourable rating c h a r a c t e r i s t i c s , the increased u t i l i z a t i o n of job information, and the very s p e c i f i c feedback to raters, evaluators, and ratees, i t i s possible that the BDI can be used as a basis or at le a s t a s i g n i f i c a n t part of a new Personnel Management System, u t i l i z i n g s i m i l a r types of information i n d i f f e r e n t areas. BDI-type of information can be used f o r : 1. Job Descriptions, to define the scope of a c t i v i t i e s and major r e s p o n s i b i l i t i e s connected with a job; 2. Job Analyses, to determine the behavioural aspects of a job; 3. Job S p e c i f i c a t i o n , to describe required s k i l l s to perform a s p e c i f i c job adequately; 4. Selection Aid, to check whether a " s k i l l e d " applicant actually possesses the s k i l l s for a s p e c i f i c job; 5. Determining Training Needs, to assess i n a detailed way an employee's strengths and weaknesses, allowing the development of very s p e c i f i c t r a i n i n g and development programmes which i n turn w i l l increase the pr o b a b i l i t y of programme success; 6. Assessment of Trainer's Effectiveness, to measure the s p e c i f i c strengths and weaknesses of instructors by focusing on t h e i r e f f e c t i v e and i n e f f e c t i v e teaching behaviour [Schwind, 1977a]; 197 7. Development of very s p e c i f i c t r a i n i n g and development programmes and the formulation of precise t r a i n i n g objectives; 8. Measuring Training and Development Effectiveness, to assess i n d e t a i l the outcomes of a l l types of programmes with the aim of changing a participant's behaviour; 9. Performance Appraisals, to determine an employee's strengths and weaknesses (shortcomings) i n the job behaviour domain [Schwind, 1976, 1978]. The appraisal can be used for d i f f e r e n t purposes, such as: a) counselling, e.g. where to improve b) feedback, e.g. giving information on standing i n the organization c) wage and salary administration, e.g. rewards could be offered to those whose job behaviour f u l f i l l s c e r t a i n basic standards; 10. Assessment of Agreement on Organizational Policy, to measure the accuracy of communication of organizational p o l i c i e s ; e.g. d e s i r a b i l i t y of s p e c i f i c behaviours, s p e c i f i c a l l y at d i f f e r e n t organizational l e v e l s ; and 11. Cross-Cultural Training, to prepare managerial and other personnel for overseas assignments [Schwind, 1977b] I t appears that the new behaviour-oriented evaluation scale, the Behavior Description Index, may be: a needed addition to, or perhaps a replacement for, some of the instruments or processes used up u n t i l now i n the area of Personnel Management. Only future research can t e l l how strong i t s p o s i t i o n w i l l be. Possible Problems with and Limitations for the BDI The f i r s t r e s u l t s of these two studies on the psychometric c h a r a c t e r i s t i c s of the BDI seem to be encouraging. But they are just t h i s : f i r s t r e s u l t s . J u s t i f i a b l y , the APA Publication Manual [1974] cautions against "...single studies with [a] new scale on a s p e c i f i c population" [p. 22]. One of the problems which should be mentioned i s c e r t a i n l y that of g e n e r a l i z a b i l i t y . The BDI i n the present studies was developed for a s p e c i f i c job (assistant branch manager);in a s p e c i f i c organizational setting (banking industry). I t s t i l l has to be tested for more and less complex jobs than that of an assistant branch manager i n a bank and i n d i f f e r e n t industries. Another important l i m i t a t i o n for the use of the BDI i s the s i g n i f i c a n t amount of time and e f f o r t required for the development of the new instrument. This undesirable c h a r a c t e r i s t i c probably rules out the BDI as a performance appraisal instrument for jobs with only a few incumbents, and small organizations with l i m i t e d resources. The same reasons — time and e f f o r t required — may also preclude the u t i l i z a t i o n of the BDI for organizations i n an unstable environment, requiring frequent organizational changes... If the r e s p o n s i b i l i t i e s , and with that the types of desirable and undesirable behaviours of a job change, then the job s p e c i f i c BDI would have to be redeveloped. In addition, the new instrument seems to be limited to positions whose job behaviours are re a d i l y observable. The question remains to be answered whether the BDI. i s applicable to positions l i k e bookkeeper, auditor, comptroller, research s c i e n t i s t , or top executives, to name a few of the more d i f f i c u l t to control jobs. I t i s also possible that performance evaluators are using a gest a l t approach for t h e i r performance ratings to arrive at a composite c r i t e r i o n , perhaps combining such factors as observed behaviour, i n f e r r e d personality t r a i t s , s o c i a l class origins,< etc. I f that were true, the BDI would be d e f i c i e n t i n so far as i t concentrates on observable behaviour. For a gesta l t approach perhaps an instrument with a combination of c r i t i c a l job behaviours and personality t r a i t s may be able to explain a higher percentage of the variance between these c r i t e r i a and the performance construct. 201 And f i n a l l y , the BDI may not be very s e n s i t i v e to s p e c i f i c s i t u a t i o n a l f a c t o r s . I t i s c o n c e i v a b l e t h a t under c e r t a i n circumstances employees are not able to e x h i b i t d e s i r a b l e behaviours or are f o r c e d to e x h i b i t u n d e s i r a b l e behaviours (e.g. group p r e s s u r e , f a u l t y communication, m i s p e r c e p t i o n , e t c . ) . The BDI i n i t s p r e s e n t form does not p r o v i d e the o p p o r t u n i t y f o r r a t e r s (or ratees) to c o r r e c t f o r such i n f l u e n c e s . As mentioned, the f i r s t r e s u l t s are encouraging. But they should be viewed w i t h the p o s s i b l e problems and l i m i t a t i o n s of the new instrument i n mind. T h i s was a p r e l i m i n a r y i n v e s t i g a t i o n . I t i s understood t h a t the BDI has not been v a l i d a t e d as y e t f o r use i n i n d u s t r y . R e c o m m e n d a t i o n s f o r F u t u r e R e s e a r c h The p r e s e n t s t u d y i s o f an e x p l o r a t o r y n a t u r e . A new method h a s b e e n c o n t r i v e d , d e v e l o p e d , a n d t e s t e d . B u t , g i v e n human n a t u r e a n d o r g a n i z a t i o n a l c o n s t r a i n t s , i t w o u l d be p r e s u m p t u o u s t o assume t h a t t h e new i n s t r u m e n t w o u l d e v e n be c l o s e t o b e i n g p e r f e c t . I n t h e f o l l o w i n g p a r a g r a p h s some s u g g e s t i o n s w i l l b e o f f e r e d on how t o s i m p l i f y a n d i m p r o v e t h e d e v e l o p m e n t p r o c e d u r e s f o r t h e BDI f o r m a t a n d w h a t v a r i a t i o n s i n t h e f o r m a t , w e i g h t i n g , a n d r a t i n g t e c h n i q u e s may be t e s t e d t o i m p r o v e t h e r a t i n g c h a r a c t e r i s t i c s o f t h e i n s t r u m e n t . F u r t h e r m o r e , s u g g e s t i o n s a r e made on w h a t t y p e o f f a c t o r s may be i n v e s t i g a t e d t o d e t e r m i n e t h e i r i n f l u e n c e on s u c h r a t i n g e r r o r s a s h a l o , l e n i e n c y , a n d c e n t r a l t e n d e n c y . D e v e l o p m e n t P r o c e d u r e s D u r i n g t h e d e v e l o p m e n t p h a s e o f t h i s s t u d y no d i s t i n c t i o n s w e r e made b e t w e e n j o b b e h a v i o u r s a m p l e s f o r t h e BARS a n d t h e BDI i n s t r u m e n t s . A c t u a l l y t h e i t e m s w e r e t a k e n f r o m t h e same i t e m p o o l . The s e l e c t i o n o f i t e m s f o r t h e BARS f o r m a t was done a c c o r d i n g t o t h e f o l l o w i n g c r i t e r i a : 1. 80% o f a M j u d g e s ( f r o m d i f f e r e n t b a n k s ) , a n d 2. 60% o f t h e j u d g e s o f t h e i n d i v i d u a l b a n k s h ad t o a g r e e on t h e v a l i d i t y o f t h e i t e m s , a n d 203 3. the standard d e v i a t i o n of the r a t i n g s c o u l d not exceed 1.5. Then those items, were chosen whose mean r a t i n g s were c l o s e to an anchor p o i n t on the 1 to 7 s c a l e . The items f o r the BDI were chosen a c c o r d i n g to the same c r i t e r i a except t h a t c l o s e n e s s to an anchor p o i n t was i r r e l e v a n t . Instead, i t was decided to use the standard d e v i a t i o n c r i t e r i o n as the f i n a l s creen; i . e . items w i t h lower SDs were p i c k e d f i r s t . I t so happened t h a t the SD of 1.5 y i e l d e d enough items to c o n s t r u c t the BDI. Otherwise the SD c o u l d have been r a i s e d s l i g h t l y , perhaps to 1.6. However, i t must be remembered t h a t the SD c r i t e r i o n was a v a i l a b l e o n l y because the BARS instrument was developed together w i t h the BDI. For f u t u r e develop-ments o f BDI s c a l e s — independently from BARS format — the r a t i n g o f items a c c o r d i n g to t h e i r degree o f e f f e c t i v e n e s s would be unnecessary and c o u l d be r e p l a c e d perhaps by a r a t i n g a c c o r d i n g to the importance of a job behaviour sample f o r doing a job e f f e c t i v e l y (e.g. "very important", "somewhat important", "not important", or " c r i t i c a l " , " l e s s c r i t i c a l " , "not c r i t i c a l " ) . Of course, i t i s p o s s i b l e to s e l e c t the itmes f o r the BDI instrument on a random b a s i s . T h i s would be a s i m p l e r and more economic method. The disadvantage of t h i s approach would be t h a t perhaps some very ..crucial items are l e f t out. For t h i s reason i t i s suggested t h a t a r a t i n g a c c o r d i n g to importance i s more d e s i r a b l e . Response S t y l e s The BDI i n the p r e s e n t study used Yes, No, ? responses for. r a t e r s . Some p a r t i c i p a n t s i n the v a l i d a t i o n phase o b j e c t e d to the "black-white" approach and recommended, an "always-often-sometimes-never" response.' A study t e s t i n g t h i s method i s c u r r e n t l y underway. 3 However, i t i s expected t h a t the use of a numerical s c a l e w i l l l e a d to the same problems which the BDI was designed to reduce or overcome: h a l o , l e n i e n c y , and c e n t r a l tendency, so c h a r a c t e r i s t i c o f g r a p h i c and summated r a t i n g s c a l e s . D i f f e r e n t Weights Instead of u n i f o r m l y r a t i n g each behaviour item wi t h 3 - 1 - 0 p o i n t s , d i f f e r i n g weights can be a p p l i e d . The b a s i s o f such an approach c o u l d be the degree o f importance of an item, e.g. a "very important" behaviour sample c o u l d r e c e i v e a weight o f 5 i n s t e a d of 3. D i f f e r e n t weights c o u l d a l s o be used f o r d i f f e r e n t dimensions. I t i s c o n c e i v a b l e t h a t f o r an a s s i s t a n t bank manager the dimension " A d m i n i s t r a t i o n " i s somewhat 3See Appendix M f o r sample o f a d i f f e r e n t response s t y l e . more important than "Training". The res u l t s of dimension "A" could then be mu l t i p l i e d by a certa i n factor. Number of Job Behaviour Samples In this study i t was a r b i t r a r i l y decided to u t i l i z e 20 job behaviour samples per dimension. Interviews with raters revealed that a l l would have gone along with 30 items. At lea s t from the interviews i t appeared that — with 5 dimensions — the number 4 0 may be a c r i t i c a l point where a fatigue e f f e c t might take place. Most would have rejected an instrument with 50 items per dimension.. I t i s hypothesized that the optimal number of items per dimension i s a function of the number of job dimensions. (The more dimensions a job has, the lower i s the optimal number per dimension A second possible consideration may be the use of the instrument. For ra t i n g purposes 20 or 30 items may be optimal. However, as described i n the discussion there are many possible applications for the BDI, e.g. Job S p e c i f i c a t i o n , Job Analysis, Job Design, among others. It i s conceivable that for such purposes, the t o t a l available item pool may be u t i l i z e d . Effects of Various Factors on BDI Ratings Several investigators have studied the impact of such variables as P a r t i c i p a t i o n i n Scale Development, Training of Raters, Raters' and Ratees' Job Experience, Education, and Sex, and Job Levels on the rating c h a r a c t e r i s t i c s of the BARS format (see discussion under A l l these and other variables (e.g. Personality Characteristics of Raters, as suggested by Zedeck, Kafry and Jacobs, 1976), could be investigated i n connection with the BDI. Since t h i s instrument appears to be less complex than the BARS format, the above-mentioned factors, e s p e c i a l l y education, job experience, and tr a i n i n g can be expected to have a s i g n i f i c a n t e f f e c t on it's rating ^ c h a r a c t e r i s t i c s . Construct V a l i d i t y Although the construct v a l i d i t y of the BDI seems to have been established, i t was also emphasized that construct v a l i d a t i o n i s not a one-shot approach, and that more studies and confirmations are needed, es p e c i a l l y confirmation as a r e s u l t of using d i f f e r e n t methods. Ideally, the dimensions of the BDI should be confirmed by a factor analytic approach. If a factor analysis of rat i n g scores would r e s u l t i n 5 factors with behaviour items loading high on th e i r relevant factors and low on others, the outcome would support the opinion of experts who agreed on the 5 job dimensions of an assistant branch manager of a bank. A study to t e s t the BDI through factor analysis i s i n progress. Summary This study answered a few questions about behaviourally.oriented scales i n general and a new instrument, the BDI, i n p a r t i c u l a r . But many new questions were developed and have to be answered before the new technique can be recommended beyond the application for which i t was developed, namely to measure the effectiveness of t r a i n i n g programmes of the Institute of Canadian Bankers. There i s . l i t t l e doubt that .the use of behaviour-based rating scales has advantages, but the scales have to be s u f f i c i e n t l y superior i n rating and feedback c h a r a c t e r i s t i c s to j u s t i f y the amount of time and e f f o r t required for the i r development. The res u l t s pertaining to the BDI so far have been very encouraging. Hopefully, the resu l t s w i l l hold up under d i f f e r e n t circumstances and analyses, and with d i f f e r e n t investigators. Only then can i t . be concluded that the BDI i s a v a l i d and useful instrument;. „ References A b a t i e l l o , A.A. "An objective evaluation of attitude change i n t r a i n i n g " Training and Development Journal, Nov. 1967, 23-34 Albee, G.W., & Hamlin, R. "An i n v e s t i g a t i o n of the r e l i a b i l i t y and v a l i d i t y of judgments i n f e r r e d from drawings", Journal of C l i n i c a l Psychology, 1949, 5_, 389-392 American Psychological Association, D i v i s i o n of Industrial-Organizational Psychology. P r i n c i p l e s f o r the V a l i d a t i o n and Use of Personnel Se l e c t i o n Procedures, 1974, Section B American Psychological Association. Standards f o r Educational and Psychological Tests. Washington: APA Inc., 1977 Andrews, K.R. The Effectiveness of University Management Development Programs. Boston: Harvard U n i v e r s i t y , 1966 Argyris, C. "Issues i n evaluating laboratory education", I n d u s t r i a l Relations, Oct. 1968, 8_, 28-40 Barrett, R.S., Taylor, E.K., Parker, J.W., & Martins, W.L. "Rating scale content: 1. Scale information and supervisory r a t i n g s " , Personnel Psychology, 1958, 11, 333-346 Bass, B.M. "Reducing leniency i n merit r a t i n g " , Personnel Psychology, 1956, 9_, 359-369 Beach, D.S. Personnel. New York: MacMillan, 1970 Beatty, R.W., Schneier, G.E., & Beatty, J.R. "An empirical i n v e s t i g a t i o n of perceptions of ratee behavior frequency and ratee behavior change using behavioral expectation scales (BES)", Personnel Psychology, • Winter 1977, 30,. 647-658. Bechtold, H.P. "Problems of e s t a b l i s h i n g c r i t e r i o n measures", i n D.B. S t u i t (Ed.), Personnel research and test development i n the Bureau of Naval Personnel. Princeton: Princeton University Press, 1947, 357-380 Bernardin, H.J. "Behavioral expectation scales vs. summated scales: A f a i r e r comparison", Journal of Applied Psychology, 1977, 62, 422-427 Bernardin, H.J., Alvares, K.M. , & Cranny, C.J. "A comparison of behaviorally anchored r a t i n g scales to summated scales", Journal of Applied Psychology, 1976, 61, 564-570 Bernardin, H.J., LaShells, M.B., Smith, P.C., & Alvares, K.M. "Behavioral expectation scales: E f f e c t s of developmental procedures and formats", Journal of Applied Psychology, 1976, 61, 75-79 209 Bernardin, H.J., & Walter, C.S. "Ef f e c t s of rater t r a i n i n g and d i a r y -keeping on psychometric error i n rat i n g s " , Journal of Applied Psychology, 1977, 62, 64-69 Blalock, H.M. J r . S o c i a l S t a t i s t i c s (2nd ed.) New York: McGraw-Hill Book Co., 1972 Blood, M.R. "Spin-offs from behavioral expectation scale procedures", Journal of Applied Psychology, 1974, 59, 513-515 Blum, M., & Naylor, J.C. I n d u s t r i a l Psychology. New York: Harper & Row, 1968 Borman, W.C. "The ratings of i n d i v i d u a l s i n organizations: An al t e r n a t i v e approach", Organizational Behavior and Human Performance, 1974, 12, 105-124 Borman, W.C. " E f f e c t s of i n s t r u c t i o n s to avoid halo error on r e l i a b i l i t y and v a l i d i t y of performance evaluation r a t i n g " , Journal of Applied Psychology, 1975, 60_v 556-560 Borman, W.C. "Behavior summary scales f o r rating, job performance: The scales' development and empirical demonstration of t h e i r v a l i d i t y " , unpublished paper. Minneapolis: Personnel Decisions, Inc., 1976 Borman, W.C, & Dunnette, M.D. "Behavior-based vs. t r a i t - o r i e n t e d performance ratings: An empirical study", Journal of Applied Psychology, 1975, 60, 561-566 Borman, W.C., & Vallon, W.R. , "A view of what can happen when behavioral expectation scales are developed in. one s e t t i n g and used i n another", Journal of Applied Psychology, 1974, 59_, 197-201 Boruch, R.F.., Larkin, J.D., Wolins, L. , & MaeKinney, A.C. "Alternative methods of an a l y s i s : Multitrait-multimethod data",, Educational and Psychological Measurement, 1970, 30, 833-853 Brogden, H.E., & Taylor, E.K. "The d o l l a r c r i t e r i o n — a p p l y i n g the cost accounting concept to c r i t e r i o n construction", Personnel Psychology, 1950, 3_> 133-154 Brown, E.M. "Influence on t r a i n i n g , method and r e l a t i o n s h i p on the halo e f f e c t " , Journal of Applied Psychology, 1968, 52_, 195-199 Brumback, G.B. "A reply to Kavanagh's 'The Content Issue i n Performance Appraisal: A Review" 1, Personnel Psychology, 1972, 25,'567-572 Brumback, G.B., & Vincent, J.W. "Factor analysis of work performed data for a sample of administrative, p r o f e s s i o n a l , and s c i e n t i f i c p o s i t i o n s " , Personnel Psychology, 1970, 23_, 101-107 Bruning, J.L., & Kin t z , B.L. Computational Handbook of S t a t i s t i c s (2nd ed.) Glenview, 111.: Scott, Foresman & Co., 1977 Buehler, R.E. "Job re l a t e d behavior r a t i n g scale", Training and Development Journal, 1969, 23_, 3: 14-21 Burnaska, R.F., & Hollmann, T.D. "An empirical comparison of the r e l a t i v e e f f e c t s of rater response biases on three r a t i n g scale formats", Journal of Applied Psychology, 1974, 59_, 307-312 Campbell, D.T., & Fiske, D.W. "Convergent and discriminant v a l i d i t y by the multitrait-multimethod matrix", Psychological B u l l e t i n , 1959, 56, 81-95 Campbell, D.T., & Stanley, J.C. Experimental and Quasi Experimental Designs for Research. New York: Rand McNally, 1963 Campbell, J.P. "Personnel t r a i n i n g and development", i n Annual Review of Psychology, Vol. 22. Palo A l t o : Annual Reviews Inc., 1971 Campbell, J.P., & Dunnette, M.D. "Effectiveness of T-group experiences i n managerial t r a i n i n g and development", Psychological B u l l e t i n , 1968, 70, 73-104 Campbell, J.P., Dunnette, M.D., Arvey, R.D., & H e l l e r v i k , L.W. "The development and evaluation of behaviorally based r a t i n g scales", Journal of Applied Psychology, 1973, 58, 15-22 Campbell, J.P., Dunnette, M.D., Lawler, E.E., & Weick, K.E. Managerial Behavior, Performance, and Effectiveness . New York: McGraw-Hill Book Co., 1970 Campion, J . , Greener, J . , & Wernli, S. "Work observation vs. r e c a l l i n developing behavioral examples for r a t i n g s c a l e s " , Journal of Applied Psychology, 1973, 58, 286-288 Cascio, W.F., & Valenzi, E. "Behaviorally anchored r a t i n g s c a l e s : E f f e c t s of education and.job experience of raters and ratees", Journal of Applied Psychology, 1977, 62, 278-282 Catalanello, R.F.., & K i r k p a t r i c k , D.L. "Evaluating t r a i n i n g programs", Training and Development Journal, May 1968, 2-9 C a t t e l l , R.B. Factor Analysis. New York: Harper & Brothers, 1952 Chruden, H.J., & Sherman, A.W. Personnel Management (5th ed.) C i n c i n n a t i : Sough-Western Publ., Co., 1976 Colby, J.D., & Wallace, R.L. "Performance a p p r a i s a l : Help or hindrance to employee p r o d u c t i v i t y " , The Personnel Administrator, Oct. 1975, 20, p. 38 f f . Cronbach, L.J. Essentials of Psychological Testing (2nd ed.) New York: Harper & Row, Publ., 1960 Cronbach, L.J. "Test v a l i d a t i o n " , i n R.L. Thorndike (Ed.) Educational Measurement (2nd ed.) Washington, D.C: American Council on Education, 1971, 443-507 211 Cronbach, L.J., & Meehl, P.E. "Construct v a l i d i t y i n psychological t e s t s " , Psychological B u l l e t i n , 1955, 52_, 281-302 Cummings, L.L., & Schwab, D.P. Performance i n Organizations. Glenview, 111.: Scott, Foresman and Co., 1973 DeCotiis, T.A. "The development and evaluation.of behaviorally anchored r a t i n g scales f o r the job of p o l i c e patrolman", unpublished Ph.D. d i s s e r t a t i o n , University of Wisconsin, 1974 Dickinson, T.L., & Tiee, T.E. "A multitrait-multimethod analysis of scales developed by r e t r a n s l a t i o n " , Organizational Behavior and Human Performance, 1973, 9_, 421-438 Dougherty, K. " R i v a l banks won't copy Royal's two-tier system", ' The Mail Star. Halifax, May 17, 1975, p. 35 Drucker, P.F. The Practice of Management. New York: Harper & Brothers Publishers, 1954 Dunnette, M.D. "A note on. the c r i t e r i o n " , Journal of Applied Psychology, 1963, 47, 251-254 Dunnette, M.D. Personnel Selection and Placement. Belmont, C a l .: Wadsworth, 1966 Dunnette, M.D., & Campbell, J.P. F i n a l Report: Development of the Penny Career Index. Minneapolis: Personnel Decisions, 1969 Ebel, R.L. "Must a l l tests be v a l i d ? " American Psychologist, 1961, 16, 640-647 Ebel, R.L. "Comments on some problems of employment t e s t i n g " , Personnel Psychology, 1977, 30, 55-63 Eysenck, H.J. " C r i t e r i o n a n a l y s i s — a n a p p l i c a t i o n of the hypothetico-deductive method to factor a n a l y s i s " , Psychological Review, 1950, 57, 38-53 Ferguson, W.C. "Quantitative evaluation of t r a i n i n g using student reaction", Training and Development Journal, 1968, 22, 11: 36-43 The F i n a n c i a l Post Survey of Industries. Toronto: Maclean-Hunter, 1976 Finley, D.M., Osburn, H.G., Dubin, J.A., & Jeanneret, P.R. "Behaviorally based r a t i n g s c a l e s : E f f e c t s of s p e c i f i c anchors and disguised scale continua", Personnel Psychology, Winter 1977, 30, 659-670 Fiske, D.W. Measuring the Concepts of Personality. Chicago: University of Chicago Press, 1971 Flanagan, J.C. " C r i t i c a l requirements: A new approach to employee evaluation", Personnel Psychology, 1949, 4_, 419-425 212 Flanagan, J.C. "The c r i t i c a l incident technique", Psychological B u l l e t i n , 1954, 51, 327-358 Flanagan, J . C , & Burns, R.K. "The employee performance record: A new appraisal and development t o o l " , Harvard Business Review, 1955, 33, 95-102 F o g l i , L., Hulin, C.L., & Blood, M.R. "Development of f i r s t - l e v e l behavioral c r i t e r i a " , Journal of Applied Psychology, 1971, 56, 3-8 French, W. The Personnel Management Process (3rd ed.) Boston: Houghton M i f f l i n Co., 1974 Friedman, B.A., & Cornelius, E.T. " E f f e c t s of rater p a r t i c i p a t i o n i n scale construction on the psychometric c h a r a c t e r i s t i c s of two r a t i n g scale formats", Journal of Applied Psychology, 1976, 6_1, 210-216 Galbraith, J.A. The Economics of Banking Operations: A Canadian Study. Montreal: M c G i l l University Press, 1963 Galbraith, J.A. Canadian Banking. Toronto: The Ryerson Press, 1970 G h i s e l l i , E.E. "Dimensional problems of c r i t e r i a " , Journal of Applied Psychology, 1956, 40, 1-4 G h i s e l l i , E.E. " I n d i v i d u a l i t y as a factor i n the success of management personnel", Personnel Psychology, 1960, 13_, 1-10 Goldstein, I . I . Training: Program Development and Evaluation. Monterey, Cal.: Brook/Cole Publ., Co., 1974 Goodale, J.G.., & Burke, R.J. "Behaviorally based r a t i n g scales need not be job s p e c i f i c " , Journal of Applied Psychology, 1975, 60, 389-391 Gorsuch, R.L. Factor Analysis. P h i l a d e l p h i a : W.B. Saunders Co., 1974 Graham, W.K., & Calendo, J.T. "Personality correlates of supervisory rat i n g s " , Personnel Psychology, 1969, 22, 483-487 Gu i l f o r d , J.P. "Factor analysis i n a test-development program", Psychological Review, 1948, 55_, 79-94 Gu i l f o r d , J.P. Psychometric Methods (2nd ed.) New York: McGraw-Hill, 1954 Guion, R.M. Personnel Testing. New York: McGraw-Hill, 1965 Hamblin, A.G. Evaluation and Control of Training. London: McGraw-Hill, 1974 Harari, 0., & Zedeck, S. "Development of behaviorally anchored scales for the evaluation of fa c u l t y teaching", Journal of Applied Psychology, 1973, 5_8, 261-265 213 Haridas, T.P., Frost, P.J., & Barnowe, J.T. "Performance Dimensions of teaching behavioral sciences i n a Canadian u n i v e r s i t y " , Proceedings of the Administrative Science Association of Canada, 1977 Hardyck, CD., & Petrinovich, L.F. S t a t i s t i c s for the Behavioral Sciences. P h i l a d e l p h i a : W.B. Saunders Co., 1969 Henry, H.W. "Performance evaluation of s c i e n t i s t s ' current methods", Management of Personnel Quarterly, 1962, 1, p. 13 Hesseling, P. Strategy of evaluation research i n the f i e l d of supervisory and management t r a i n i n g . Assen, The Netherlands: Van Gorcum, 1966 Horst, P. "Obtaining a composite measure from a number of d i f f e r e n t measures", Psychoihetrica, 1936, 1, 53-60 Inn, A., Hulin, C.L., & Tucker, L. "Three sources of c r i t e r i o n variance: S t a t i c dymensionality, dynamic dimensionality, and i n d i v i d u a l dimensionality", Organizational Behavior and Human Performance, 1972, 8, 58-73 James, L.R. " C r i t e r i o n models and construct v a l i d i t y f o r c r i t e r i a " , Psychological B u l l e t i n , 1973, 80, 75-83 Kafry, D., Jacobs, R., & Zedeck, S. • " E f f e c t s of rater p a r t i c i p a t i o n on d i s c r i m i n a b i l i t y of behavioral expectation scales", unpublished paper. Berkeley: U n i v e r s i t y of C a l i f o r n i a , 1977 Kavanagh, M.J. "The content issue i n performance a p p r a i s a l : A review", Personnel Psychology, 1971, 24_, 653-668 Kavanagh, M.J., MacKinney, A.C., & Wolins, L. "Issues i n managerial performance: Multitrait-multimethod analyses of r a t i n g s " , Psychological B u l l e t i n , 1971, 75, 34-49 Kay, E.H., Meyer, H., & French, R.P. " E f f e c t s of threat i n a performance appraisal interview", Journal of Applied Psychology, 1965, 49, 311-317 Keaveny, T.J., & McGann, A.F. "A comparison of behavioral expectation scales and graphic r a t i n g scales", Journal of Applied Psychology, 1975, 60, 695-703 Keaveny, T.J., & McGann, A.F. "Reducing contamination i n students' evaluation of college professors", unpublished manuscript, University of Wyoming, 1977 Keaveny, T.J., & McGann, A.F. "Performance appraisal format: Role c l a r i t y and evaluation c r i t e r i a " , unpublished manuscript, University of Wyoming, 1978 214 Kerlinger, F.N. Foundation of Behavioral Research. New York: Holt, Rinehart and Winston, 1973 Kir k p a t r i c k , D.L. "Techniques for evaluating t r a i n i n g programs, I." Journal of the American Society of Training Directors, 1959, 13, 3-9 (a) Kirk p a t r i c k , D.L. "Techniques for evaluating t r a i n i n g programs, I I . " Journal of the American Society of Training Directors, 1959, 13, 21-26 (b) Ki r k p a t r i c k , D.L. "Techniques for evaluating t r a i n i n g programs, I I I . " Journal of the American Society of Training Directors, 1960, 14, 13-18 (a) Ki r k p a t r i c k , D.L. "Techniques for evaluating training,programs, IV." Journal of the American Soceity of Training D i r e c t o r s , 1960, 14, .28-32 (b) Kohn ., V., & Parker, T.C. "Some guidelines for evaluating management development seminars", Training and Development Journal, July 1969, 18-23 Landy, F.J., Farr, J.L., Saal, F.E., & Freytag, W.R. "Behaviorally anchored scales for r a t i n g the performance of p o l i c e o f f i c e r s " , Journal of Applied Psychology, 1976, 6_1, 750-758 Landy, F.L., & Guion, R.M. "Development of scales for the measurement of work motivation", Organizational Behavior and Human Performance, 1970, 5_, 93-103 Latham, G.P., Wexley, K.N., & P u r s e l l , E.J. "Training managers to minimize r a t i n g errors i n the observation of behavior", Journal of Applied Psychology, 1975, 6£, 550-555 Lawshe, C.H. "A quantitative approach to content v a l i d i t y " , Personnel Psychology, 1975, 28, 563-575 Levinson, H. "Appraisal of What Performance?", Harvard Business Review, 1976, 54, 30-48 L i s s i t z , R.W., & Green, S.B. " E f f e c t of the number of scale points on r e l i a b i l i t y : A Monte Carlo approach", Journal of Applied Psychology, 1975, 60, 10-13 Luborsky, L. " C l i n i c i a n s ' judgment of mental health: A proposed sc a l e " , Archives of General Psychiatry, 1962, _7, 407-417 Mack, H. "Evaluating the organizational development e f f o r t " , Training and Development Journal, March 1974, 42-47 Mager, R.F. Preparing I n s t r u c t i o n a l Objectives. Palo A l t o , C a l . : Fearon Publ. Inc., 1962 215 McGehee, W., & Thayer, P.W. Training i n Business and Industry. New York: John Wiley & Sons, Inc., 1961 McGregor, D. "An uneasy look at performance a p p r a i s a l " , Harvard Business Review, May-June 1957, 89-94 Megginson, L.C. Personnel and Human Resources Administration (3rd ed.) Homewood, 111.: Richard D. Irwin, Inc., 1977 Meyer, H.D. "A four year study of management promotions i n the Jewel Tea Co., as rela t e d to three measures of achievement motivation", Ditto Report, Jewel Tea Co., 1963, as c i t e d i n Campbell, Dunnette, Lawler & Weick, Managerial Behavior, Performance and Effe c t i v e n e s s . New York: McGraw-Hill Book Co., 1970, 190-191 Meyer, H.D. "Promotion p r o b a b i l i t y c o n t r o l i n using promotion as a c r i t e r i o n for assessment test v a l i d a t i o n , I.", Mimeographed Report, Jewel Tea Co., 1965(a), as c i t e d i n Campbell, Dunnette, Lawler, & Weick, Managerial Behavior, Performance and Effectiveness . New York: McGraw-Hill Book Co., 1970, 190-191 Meyer, H.D. "Promotion p r o b a b i l i t y control, i n using promotion as a c r i t e r i o n for assessment test v a l i d a t i o n , I I . " , Mimeographed Report, Jewel Tea Co., 1965(b), as c i t e d i n Campbell, Dunnette, Lawler, & Weick, Managerial Behavior, Performance and Effe c t i v e n e s s . New York: McGraw-Hill Book Co., 1970, 190-191 M i l l a r d , G.W., Luthans, F., & Ottemann, R.L. "A new breakthrough for performance appraisal",. Business Horizon, August 1976, 66-73 M i l l e r , G.A. "The magical number seven, plus or minus two: Some l i m i t s to our capacity for processing information", Psychological Review, 1956, 63, 81-97 Nagle, B.F. " C r i t e r i o n development", Personnel Psychology, 1953, b_, 271-289 National I n d u s t r i a l Conference Board. "Personnel practices i n factory and o f f i c e : Manufacturing", Studies i n Personnel P o l i c y , No. 194, 1964 Neufeld, E.P. "Money and Banking i n Canada", The Carlton L i b r a r y No. 17. McClelland and Steward Ltd., 1964 Neufeld, E.P. The F i n a n c i a l System i n Canada. Toronto: Macmillan of Canada, Ltd., 1972 Norton, S.D., Gustafson, D.P., & Foster, C.E. "Assessment for management p o t e n t i a l : Scale design and development, t r a i n i n g e f f e c t s and rater/ratee sex e f f e c t s " , Academy of Management Journal, 1977, 20, 117-131 Nunnally, J.C. Psychometric Theory. New York: McGraw-Hill, 1967 216 Odiorne, G.S. "The need for an economic evaluation of a t r a i n i n g program", Training and Development Journal, 1964, 18, 3: 3-12 Olivero, J. "The v a l i d a t i o n of r a t i n g scales through the p r i n c i p l e of multiple operationism", unpublished Master's Thesis, Ohio State University, 1961 Otto, CP., & Glaser, R.O. . The Management of Training. Reading, Mass.: Addison-Wesley Publ., Co., 1970 Palef, S., & Steward, C P . "Development of a r a t i n g scale", Studies i n Personnel Psychology, 1971, 3_, 1-20 Roach, D.E., & Wherry, R.J. Sr. "Performance dimensions of m u l t i -l i n e insurance agents", Personnel Psychology, 1970, 23, 235-250 Ronan, W.W. "A factor analysis of eleven job performance measures", Personnel Psychology, 1963, . 16_, 255-267 Ronan, W.W., & Prien, E.P. Perspectives on the Measurement of Human Performance. New York: Appleton-Century-Crofts, 1971 Roscoe, J.T. Fundamental Research S t a t i s t i c s f o r the Behavioral Sciences. New York: Holt, Rinehart & Winston, Inc., 1969 "Rough going: More o f f i c e workers b a t t l e being f i r e d by suing t h e i r bosses", Wall Street Journal, June 18, 1975, 1-16 Runkel, P.J., & McGrath, J.E. Research on Human Behavior. New York: Holt, Rinehart and Winston, Inc., 1972 Schmidt, M.J. Understanding and Using S t a t i s t i c s . Lexington, Mass.: D.C Heath & Co. , 1975 Schneier, C.E. "Operational u t i l i t y and psychometric c h a r a c t e r i s t i c s of behavioral expectation scales: A cognitive r e i n t e r p r e t a t i o n " , Journal of Applied Psychology, 1977, 62, 541-548 (a) Schneier, C.E. "The influence of rater's cognitive c h a r a c t e r i s t i c s on the r e l i a b i l i t y and v a l i d i t y of r a t i n g s c a l e s " , Proceedings of the Academy of Management Meeting, Orlando, F l o r i d a , 1977 (b) Schneier, C.E., & Beatty, R.W. "Toward a cognitive theory of the performance appraisal process (PA): A f i e l d i n v e s t i g a t i o n of the e f f e c t s of matching rater cognitive structure with cognitive requirements of PA. formats", paper presented at the Academy of Management Meeting i n Kansas C i t y , 1976 Schwab, D.P., Heneman, H.C, & DeCotiis, T.A. "Behaviorally anchored r a t i n g scales: A review of the l i t e r a t u r e " , Personnel Psychology, 1975, 28, 549-562 217 Schwarz, F.C., S t i l w e l l , W.P., & Scanlan, B.K. " E f f e c t s of management development on managerial behavior and subordinate perception", Training and Development Journal, 1968, A p r i l , pp. 38-50;. May, pp. 24-34 Schwind, H.F. "Thoughts on t r a i n i n g evaluation", Canadian Training Methods, 1975, 8_, 14-15 (a) Schwind, H.F. "Issues and problems i n training, evaluation, I: The methodology", Canadian Training Methods, 1975, J5, 22-25 (b) Schwind, H.F. "Issues i n t r a i n i n g evaluation, I I : The c r i t e r i o n " , Canadian Training Methods, 1975, 8_, 14-15 (c) Schwind, H.F. "Training evaluation: A review of the l i t e r a t u r e " , unpublished Working Paper, University of B r i t i s h Columbia, 1975 (d) Schwind, H.F. "New ways to evaluate performance of sales personnel", Proceedings of the A t l a n t i c Schools of Business Conference, St. John's, Nf l d . , 1976 Schwind, H.F. "New ways to evaluate teaching and t r a i n i n g e f f e c t i v e n e s s " , paper presented at the Adult Education Research Association meeting i n Minneapolis, A p r i l 1977 (a); a v a i l a b l e as ERIC Document ED 141507 from ERIC Clearinghouse on Career Education, The Ohio State U n i v e r s i t y , Columbus, Ohio, 43210 Schwind, H.F. " C r o s s - c u l t u r a l t r a i n i n g : A behavioral approach", paper presented at the European International Business Association Annual Meeting i n Uppsala, Sweden, 1977 (b) Schwind, H.F. "An a l t e r n a t i v e to Behaviorally Anchored Rating Scales: The Behavior Description Index", Proceedings of the Academy of Management Meeting, San Francisco, 1978 Sharon, A.T., & B a r t l e t t , C.J. " E f f e c t s of i n s t r u c t i o n a l conditions i n producing leniency on two types of r a t i n g scales", Personnel Psychology, 1969,..22, 251-263 Sie g e l , S. Nonparametric S t a t i s t i c s for the Behavioral Sciences. New York: McGraw-Hill Book Co., 1956 Silverman, P. "CLC a l l set for n a t i o n a l bank unionization d r i v e " , The F i n a n c i a l Post, Toronto, Dec. 3, 1977, p. 3 Smith, P.C., & Kendall, L.M. "Retranslation of expectations: An approach to the construction of unambiguous anchors for r a t i n g scales", Journal of Applied Psychology, 1963, 4_7, 149-155 Smith, P.C., Kendall, L.M., & Hulin, C.L. The Measurement of S a t i s f a c t i o n i n Work & Retirement. Chicago: Rand, McNally & Co., 1969 218 Spriegel, W.R. "Company practices i n appraisal of managerial performance", Personnel, 1962, 39, 77-82 Taylor, E.K., Barrett, R.S., Parker, J.W., & Martens, L. "Rating scale content, I I . E f f e c t of r a t i n g on i n d i v i d u a l s cales", Personnel Psychology, 1958, .11, 519-533 Taylor, J. "Rating scales as measures, of c l i n i c a l judgment: A method for increasing scale r e l i a b i l i t y and s e n s i t i v i t y " , Educational and Psychological Measurement, 1968, 2S_, 747-766 Tenopyr, M.L. "Content-construct confusion", Personnel Psychology, 1977, 30, 47-54 T i f f i n , J . , & McCormick, E.J. I n d u s t r i a l Psychology (5th ed.). Englewood C l i f f s , N.J.: P r e n t i c e - H a l l , 1965 Thorndike, E.L. "Handwriting", Teachers College Record, 1910, p. 11 Thorndike, R.L. Personnel Selection. New York: John Wiley, 1949 Varney, G.H. "Performance a p p r a i s a l — i n s i d e and out", The Personnel Administrator, Nov-Dec 1972, 17, p. 17 Wallace, M.J., & Weitzel, W. "Development and a p p l i c a t i o n of behavioral t r a i n i n g c r i t e r i a " , Experimental P u b l i c a t i o n System, American Psychological Association, Issue No. 6, June 1970, Ms. No. 220A Wallace, M.J., & Weitzel, W. "The experimental evaluation of supervisory t r a i n i n g " , Experimental P u b l i c a t i o n System, American Psychological Association, Issue No. 6, June 1970, Ms. No. 221A Wallace, M.J., & Weitzel, W. "Evaluation of a t r a i n i n g program", Academy of Management Proceedings, August 1972, 320-323 Whisler, T.L., & Harper, S.F. Performance Appraisal. New York: Holt, Rinehart & Winston, 1962 Zedeck, S., & Baker, T. "Nursing performance by behavioral expectation sc a l e s : A m u l t i t r a i t - m u l t i r a t e r a n a l y s i s " , Organizational Behavior and Human Performance, 1972, ]_, 457-466 Zedeck, S., Imperato, N., Krausz,. M. , & Oleno, T. "Development of behaviorally anchored r a t i n g scales as a function of organizational l e v e l " , Journal of Applied Psychology, 1974, 59j, 249-252 Zedeck, S., Jacobs, R., & Kafry, D. "Behavioral expectation: Development of p a r a l l e l forms and analysis of scale assumptions", Journal of Applied Psychology, 1976, 61_, 112-115 Zedeck, S., & Kafry, -D. "Evaluation of the developers of behavioral expectation scales", unpublished paper. Berkeley: U n i v e r s i t y of C a l i f o r n i a , 1975 Zedeck, S., Kafry, D., & Jacobs, R. "Format and scoring v a r i a t i o n s i n behavioral expectation evaluation", Organizational Behavior and Human Performance, 1976, 1_7, 171-184 Appendix TABLE 1 COMPARATIVE STUDIES AND NON-COMPARATIVE STUDIES WITH DATA Study Ratees Raters Number of dimensions Number of scale points I n terrater R e l i a b i l i t y BARS Other Dimension Intercorrelation (Halo) BARS Other Leniency (Deviation from mean) BARS Other BARS superior? • Yes/No Remarks . Barrett et. a l . (1958) C l e r i c a l Workers (n=?) F i r s t Level Supervisor Second Level Supervisor 8 15 .67 . .53 .51 .51 .51 .56-.63 .56 .50-.65 less Yes 2 BARS vs. 2 GRS Maas (1965) Students (n=360) (n=500) (n=188) (n=172) Student Inter-viewers (n=12) 6 11 .58 .35 -.47 .34 .47 .34 .69 .34 Yes BARS vs. GRS 4 d i f f e r e n t groups Peters & McCormick (1966) Cases Students 5 7 .27-.84 .06-.87 Yes EARS vs. GRS Taylor (1968) Cases Psycholo-g i s t s (n=6) 6 7 .64-.89 .10-.63 X2=.001 X2=.001 Yes BARS vs. GRS Landy £ Guion (1970) Chemical engineers (n=19) Other engineers (n=141) 2 peers per ratee 7 5 .51-.82 .11-.70 no comparison TABLE 1 COMPARATIVE STUDIES AND NON-COMPARATIVE STUDIES WITH DATA (continued) Study Ratees Raters Number of dimensions Number of scale points I n terrater R e l i a b i l i t y BARS Other Dimension Intercorrelation (Halo) ; BARS Other Leniency (Deviation from mean) BARS Other BARS Superior? Yes/No Remarks Taylor et. a l . (1970) Cases Community workers (n=15) 4 6 .77-.82 .33-.58 no comparison Hakel (1971) Student nurses Nursing • administra-tors and superiors (n=9) 5 9 -.20-.72 median: .34 strong no comparison Palef £ Stewart (1971) Off i c e clerks (n=71) Super-visors 10 9 .08-.72 no comparison Campion (1972) Mechanics (n=32) Super-visors 2 .01-.27 .10-.62 Yes EARS vs. Employment Tests Zedeck & Baker (1972) Nurses (n=98) Head-nurses (H) (n=9) Super-visors .(S) (n=5) 5 9 ,24-.52 H: .38-.62 S: .49-.82 no comparison K> TABLE 1 COMPARATIVE STUDIES AND NON-COMPARATIVE STUDIES WITH DATA (continued) Study Ratees Raters Number of dimensions Number of scale points I n terrater R e l i a b i l i t y BARS Other Dimension Intercorrelation (Halo) BARS Other Leniency (Deviation from mean) 3ARS Other BARS Superior? Yes/No Remarks Campbell et. a l . (1973) Depart-ment managers (n=573) Store managers (S) and Assistant store managers (A) 9 9 .31-.55 .42-.58 S: S: .28-.63 .35-.76 5: .98 .57 A: .84 .49 Yes BARS (7pt) vs. SRS (4pt) "A" scores not reported Dickinson s Tice (1973) F i r e -fighters (n=149) Super-visor using c h e c k l i s t (S) Peer using checklist(PC) Peer using nomination (PN) 3 9 S vs.PC .11-.24 S vs.PN .14-.25 . PC vs.PN .07-.10 S: .28-.46 PC: . 34-.43 PN: .24-.54 no comparison Williams S S e i l e r (1973) Engineers (n=202) Se l f (SF) and Super-visors (S) 7 5 .33 .24 .60 .48 SF: .08-.58 S: .42-.79 SF: .37-.70 S: .61-.86 Yes EARS vs. GRS Arvey & Hoyle (1974) Programmers & Analysts (n=200) Super-visors (n=103) 11 i .40-.71 .42-.72 comparison between t y p i c a l BARS and Guttman type BARS TABLE 1 COMPARATIVE STUDIES AND NON-COMPARATIVE STUDIES WITH DATA (continued) Study Ratees Raters Number of dimensions Number of scale points I n terrater R e l i a b i l i t y BARS Other Dimension Intercorrelation (Halo) BARS Other Leniency (Deviation from mean) BARS Other BARS Superior? Yes/No Remarks Borman (1974) Secretaries Peers (P) and Academic Instructors . (Al) 4 3 7 P: .17-.30 .10-.20 A l : .24-.54 ..21-.78 P: .19-.73 .18T.62 A l : .44-.48 ^.18-.05 no comparison Borman & ValIon (1974) Nurses (n=7-29 per rater pair) Head Nurse (H) & Assistant Head Nurse (A) 5 9 .61* .56* .73* .73* 1.43* 1.14* No BARS vs. GRS * . • corroined ratings (H+A) Burnaska & Hoilmanh (1974) Instructors (n=3) Students 5 9 .78 a: .81 b: .63 high high s l i g h t l y less No BARS vs. 2 GRS (a+b) DeCotiis (1974) Vignettes Police O f f i c e r s 6 9 .38-1.00 no comparison Borman (1975) Vignettes (n=6) Managers (n=90) 6 7 Before: .81-.89 After: .58-.82 .67 .91 no comparison TABLE COMPARATIVE STUDIES AND NON-COMPARATIVE STUDIES WITH DATA (continued) Study Ratees Raters Number of dimensions Number of scale points I n terrater R e l i a b i l i t y BARS Other ' Dimension Intercorrelation (Halo) -. BARS Other Leniency (Deviation from mean) BARS Other BARS Superior? Yes/No Remarks Borrnan S Dunnette (1975) Navy Junior O f f i c e r s (n=126) Navy O f f i c e r s (n=88) 14 9 higher a: p< .05 b: p< .02 less a: p< .02 - 1.77 a: 1.98 b: 2.04 Yes BARS vs. 2 GRS (a+b) Grcner (1975) College Recruiters Students (n=72) 8 9 .25-.36 a:.00-.08 b:.0 7 - . l i No BARS vs. GRS (a) and SRS (b) Keaveny & McGann (19-7 5) Instructors Students (n=55) 13 9 .21-.58 .33-.66 1.71 1.92 No BARS vs. GRS Bernardin, Alvares & Cranny (1976) Instructors (n=27) Students 9 7 1.46 • a: 1.32 (lower) b: 1.37 .36 a: .45 b: .39 1.02 a: 1.12 b: .58 No BARS vs. 2 GRS (a+b) Bernardin, LaShells, Smith S Alvares C976) Instructors (n=81) Students (n=l,428) 6 7 .76 .75 .08 .89 Yes BARS vs. SRS 3orrr.an (1976) • Navy Recruiters (n=24) Se l f (S) s Peer (P) S Supervisor(Sp) 9 10 S: .28-P: .40 Sp: .70 S: 2.5 P: 2.0 Sp:2.7 no comparison TABLE 1 COMPARATIVE STUDIES AND NON-COMPARATIVE STUDIES WITH DATA (continued) Study Ratees Raters Number of dimensions Number of scale points I n terrater R e l i a b i l i t y BARS Other Dimension In t e f c o r r e l a t i o n (Halo) BARS Other Leniency (Deviation from mean) BARS Other BARS Superior? . Yes/No Remarks Friedman £ Cornelius (1976) A i r Force Reserve Of f i c e r s (n=25) Instructors 10 7 no difference no difference No BARS vs. GRS Landy et. a l . (1976)-Police O f f i c e r s Peers (?) s Superiors (S) 9 9 P:.47-.73 S:.57-.69 P:.58-.68 S:.54-.67 P:1.46-1.83 S:l.31-1.53 no comparison M i l l a r d et. a l . (1976) Interviewers (n=117) Supervisors 6 9 .70-.84 less Best: 1.99 Typical: .29 .37 Yes •BARS vs. GRS Best=3est Per-formance Typical=Typical Performance Zedeck, Jacobs & Ka f ry (1976) Instructors Students (n=44) (n=51) 9 7 A: .31-1.98 B: .59-1.82 no comparison Zedeck, Kafry & Jacobs (1976) Instructor (n=l) Students (n=141) 9 7 I:.00-.68 Ila:.07-.72 (.35) (.39) Ilia:-.23-.53 (.23) IIIb:-.17-.75 (-.43) I:.98 I l a : 1.32 l i b : 1.12 I l i a : 1.32 I l l b : 1.03 No BARS vs. SRS & GRS (5 scoring methods: I, Ila+b, Illa+b) score i n paren-thesis=mean score Ln :ABLE 1 COMPARATIVE STUDIES AND NON-COMPARATIVE STUDIES WITH DATA . (continued) Study Ratees Rate rs Number of dimensions Number of scale points I n terrater R e l i a b i l i t y BARS Other Dimension -Intercorrelation (Halo) BARS. Other Leniency (Deviation from mean) BARS Other BARS Superior? Yes/No Remarks Bernardin (1977) Instructors (n=15) Students (n=154) 7 7 no differences 1.13 a: .96 (SD) b: 1.09 (SD) .67 .64 .58 NO BARS vs. 2 SRS • (a+b) Halo measure based on standard devia-t i o n (SD); higher value indicates less halo. Bernardin & Walter (197 7) Instructors (n=13) Students (n=156) 7 7 I. 59 1.27 . 1.15 1.06 no comparison 4 groups with d i f f e r e n t exposure to t r a i n i n g (high to low) Cascio & Valenzi (19.77) Police O f f i c e r s (n=299) Sergeants (n=71) 8 9 .84-.91 a) 1.46 1.96 b) 1.83 1. 33 c) 1.40 1.90 no comparison . a) rater experience low and high b) rater education/ low and high c) ratee experi-ence, low and high DeCotiis (1977) Vignettes (n=5) Supervisors (n=28) 6 7 rmdn .64 .82 .81 : rmdn .73 .91 same No BARS vs 2 GRS 1 COMP/sHATIVE STUDIES KID ECS-COJIPARlvriVE STUDIES WITH DATA (continued) £ t u d y R a t o o s R a t e r s •ta U-i C-0 0 •H 0 c •0 !D 3 -H - ' CT 2& 0 0 3 o 3 io Intorratar R e l i a b i l i t y BARS Other Dimension Intorcorrelation, (Halo) BARS Other Leniency (Deviation froia mean) BARS Other BARS Superior? Yes/Ho Remarks F i n l e y , O s b o r n , D u b i n , J e a n n e r e ( 1 9 7 7 ) D e p a r t m e n t S t o r e M g r s ( n = 1 7 7 ) F i r s t L e v e l S u p e r v i s o r s S e c o n d L e v e l S u p e r v i s o r s 11 7 .35-.56 .35 3 Different BARS Formats were compared H a r i d a s , F r o s t & B a r n o v e ( 1 9 7 7 ) I n s t r u c t o r s • S t u d e n t s 8 7 .19 . .31 less Yes BARS vs. GRS TABLE 1 COMPARATIVE STUDIES AND NON-COMPARATIVE STUDIES WITH DATA (continued) Study Ratees Raters Number of dimensions Number of scale points I n terrater R e l i a b i l i t y BARS Other Dimension Inter c o r r e l a t i o n (Halo) BARS other Leniency (Deviation from mean) BARS Other BARS Superior? Yes/No Remarks Kafry et. a l . (1977) Nurses (n=153) Public Health Supervisors (n=20) 24 .7 P: .50 N: .43 >:.40-2.2 1:1.1-1.7 no comparison Participants i n . BARS development (P) vs. non-participants (N) Keaveny & McGann, (1977) Instructors ' Students 13 9 .36 a: .42 b: .47 Yes BARS vs. 2 GRS (a+b) BARS had leas t contamination Motcwidlo £ 3crr?.an (1377) Army Platoons (n=47) Of f i c e r s (n=32) 8 9 .34-. 70 .40-.76 .47-1.70 no comparison Norton et. a l . (19 77) Cases (n=2) Managers (n=3,261) 6 9 .12-.46 A 1 : .35 . A 2: .07 B i : .34 B 2: .14 no comparison Ai=Case A before t r a i n i n g A2=after t r a i n i n g Bi= Case B before trai n i n g B2= a f t e r t r a i n i n g 229 Appendix B 230 USE OF BARS FOR PURPOSES OTHER THAN PERFORMANCE APPRAISAL Study Purpose Maas (1965) Landy & Guion (1970) Wallace & Weitzel (1970) F o g l i , Hulin, & Blood (1971) Hakel (1971) Campion (1972) Fineman & Payne (1973) Williams & S e i l e r (1973) Arvey & Hoyle (1974) Arvey & Neel (1974) Van Fleet (1974) Ivancevich (1976) Zedeck, Jacobs, & Kafry (1976) Motowidlo & Borman (1977) Interviewing Measuring Work Motivation Training Evaluation Determining Job C r i t e r i a Improvements of Interviewing Personnel Selection Management of Objectives Relationships Between E f f o r t & Per-formance Guttman Scale Based on BARS Expectancy Predictions Determining Elements of Leadership Expectancy Predictions Development of P a r a l l e l Forms of BARS Measuring Morale Zedeck & Kafry (1977) Strategy Assessment 231 Appendix C D e s c r i p t i v e P u b l i c a t i o n s Using BARS Study Thorndike (1910) Luborsky (1962) Smith & K e n d a l l (1963) Buehler (1969) Wallace & W e i t z e l (1970» Wallace & W e i t z e l '(1970 b) Wallace & W e i t z e l (1972) Campion, Greener & Wernli (1973) H a r a r i & Zedeck (1973) Blood (1974) Goodale & Burke (1975) Schneier & Beatty (1976) Beatty, Schneier & Beatty (1977) F i n l e y , Osborn, Dubin & Jeanneret (1977) K a f r y , Jecobs & Zedeck (1977) Keaveny & McGann (1977) Schneier (1977; a) Schneier (1977 b) Keaveny & McGann (1978) 233 Appendix D ) 23h Job Part: Administration Description of R e s p o n s i b i l i t i e s : quarterly analyses, monthly reports, routine work, bryj^ch security, i n t e r n a l correspondence, correspondence with other banks, cooperation with manager, supplies, housekeeping, repairs Rating Procedure: Please, THINK CAREFULLY before you answer the question: Does this job incumbent e x h i b i t the i n following described behavior a l l the time? (Does i t describe his t y p i c a l behavior?) Use "Y" for Yes, "N" for No, and "?" i f you are not c e r t a i n . Delegates work to subordinates e f f e c t i v e l y by taking care that i t i s evenly d i s t r i b u t e d according to a b i l i t y and knowledge of each employee. Has to borrow frequently required s t a t i o n a r i e s from other branches r e s u l t i n g in delays and disruption of work flow. Is unable to organize his own work load, i s constantly "catching up" thus s e t t i n g a poor example for others. Is very conscious of s e c u r i t y , and e f f e c t i v e l y controls procedures thereby r e s u l t i n g in a l l personnel observing rules and regulations. Ensures that housekeeping of banking are i s neat and desks are not messy, thus making the Branch look presentable and a pleasant place to do business. Drops work on a subordinate's desk to be completed without reviewing i t with that p a r t i c u l a r i n d i v i d u a l , causing confusion and backlog. Has a high standard of l e t t e r w r i t i n g , good typing, and ensures that l e t t e r s are not signed by junior personnel. Keeps meticulous records and encourages s t a f f to do the same, thereby reducing errors and rework. Keeps close control on cash holdings thus minimizing r i s k s and maximizing p r o f i t s . Offers suggestions to superiors i n e f f o r t to improve effectivensss of new or e x i s t i n g p o l i c i e s . Is unable to develop a good diary system, causing confusion and delays with work. Does not care that correspondence i s answered d a i l y . Knows a new procedure well before explaining i t to the s t a f f . Keeps a check on amount of overtime and finds out reason. Analyzes the duties to be performed i n order that matters of high p r i o r i t y are attended to f i r s t . Keeps down administrative costs through e f f i c i e n c y of operation, i . e . , s t a f f are p r o f i t a b l y employed, over ordering of stationary i s kept to a minimum, li g h t s are not l e f t on a l l night, s t a f f do not work unnecessary overtime. Knows his manuals well and i s able to quickly f i n d procedures to follow. Keeps management informed of a l l aspects of branch operation. Takes proper safeguards to combat frauds, etc., such as holding funds u n t i l cheques c l e a r , etc. Plans s t a f f vacation so as to provide adequate customer service. 2 3 5 Job Part: Customer Relations Description of R e s p o n s i b i l i t i e s : complaints, correspondence, s p e c i a l service (counseling in complex matters), service maintenance, sp e c i a l e f f o r t s ; Rating Procedure: Please, THINK CAREFULLY before you answer the question: Does t h i s job incumbent e x h i b i t the i n following described behavior a l l the time? (Does i t describe his t y p i c a l behavior?) Use "Y" for Yes, "N" for No, and "?" i f you are not c e r t a i n . Sees that customer correspondence i s answered the same day i t a r r i v e s . Mixes with customers wherever possible i n order to e s t a b l i s h a good personal r e l a t i o n s h i p . Ensures that a l l employees are courteous toward customers. Gives customer problems top p r i o r i t y and takes steps to see that the problems do not ar i s e again. Loses h is temper at a customer i n public. When answering customer's telephone queries i s short and abrasive giving the customer a f e e l i n g of not being cared about. When a customer appears, acknowledges t h e i r presence and l e t s them know that he w i l l be with them i n a minute i f unavoidably involved. Joins in community a c t i v i t i e s such as Junior Chamber of Commerce and presents a good bank image. If a customer i s not sure of an entry to his account this accountant w i l l take time to explain the nature of the entry ( i . e . , why he was charged for being overdrawn, etc.) I f a customer i s ignorant of a bank's f a c i l i t i e s ( i . e . , accounts, d r a f t s , etc.) this accountant w i l l find out the customer's needs and then make suggestions. Knows many customers by name. Accepts phone c a l l s when s e r v i c i n g a customer. Explains bank p o l i c i e s to customers, e.g., why t h e i r cheque i s not cashed. Apologizes to customer when s t a f f makes an error. j Tries to s e l l the customer the services most p r o f i t a b l e to the bank, rather than to the customer's best i n t e r e s t . Forgets the names of important customers. Argues long and loud with a customer as to why a cheque should not be cashed. Makes appointments at the convenience of the customer i f possible. T e l l s complaining customer to take h is business elsewhere. Ensures that a l l s t a f f members are aware of the standards of dress and neatness to avoid possible negative customer reactions. 236 Job Part: Marketing Description of R e s p o n s i b i l i t i e s : advertising, display material, l o c a l campaigns, open house, marketing of bank services Rating Procedure: Please, THINK CAREFULLY before you answer the question:- Does this job incumbent e x h i b i t the in following described behavior a l l the time? (Does i t describe his t y p i c a l behavior?) Use "Y" for Yes, "N" for No, and "?" i f you are not certain. Plans each campaign ahead and sets the goals for his branch. Ensures public i s aware of new and b e n e f i c i a l timely services through in-branch advertising. Is conscious of a l l opportunities for new business and makes management aware of any new customer with p o t e n t i a l for more business. Is unable to answer customer i n q u i r i e s about bank services because of lack of product knowledge, causing d i s s a t i s f a c t i o n and a bad bank image. Does not use materials provided to advertize the Bank's service. Sets up a s p e c i a l sales area, e a s i l y accessible to the.customer, to inform about a new b~ank service and to market the service. U t i l i z e s marketing posters to the best advantage i n the branch and regularly moves or replaces them. Does not take advantage of contact with non-customers to s o l i c i t new accounts. Has new advertising material for each item, makes s t a f f aware of i t . Does nothing ot market bank services. " I f the customer wants i t he w i l l ask for i t " . When a new campaign i s announced t h i s accountant w i l l c a l l a s t a f f meeting, asking for suggestions how the branch can p a r t i c i p a t e e f f e c t i v e l y . Makes s t a f f aware of f u l l range of bank services. Ensures that counter o f f i c e r s are c r o s s - s e l l i n g the bank's services. Follows through on outlined promotion campaign. Tries to improve on areas where the branch i s weak, e.g., i f there are a number of unrented safety deposit boxes, he w i l l set up a campaign to s e l l them. Has a sound knowledge of products and services, enabling him to explain a l l advantages and benefits to a customer. Ensures that d i r t y or torn advertising posters are replaced promptly. F a i l s to inform s t a f f about new campaigns. Is not aggressive i n s e l l i n g new bank services. Offers incentives to s t a f f to promote bank services during campaigns. 237 Job Part : Personnel Description of R e s p o n s i b i l i t i e s : salary administration, performance app r a i s a l , h i r i n g and f i r i n g , d i s c i p l i n a r y actions, motivation, encouragement, moral support, personal problems, vacation and holiday scheduling Rating Procedure: Please, THINK CAREFULLY before you answer the question: Does this job incumbent e x h i b i t the i n following described behavior a l l the time? (Does i t describe his t y p i c a l behavior?) Use "Y" for Yes, "N" for No, and "?" i f you are not certain. Shows great confidence i n subordinates and openly displays t h i s with the r e s u l t that they develop to meet his expectations. Is openly c r i t i c a l of subordinates i n front of both other s t a f f members and customers. Confers with members of the s t a f f regarding decisions which w i l l a f f e c t them. Ensures that a l l s t a f f members are aware of the duties they have to perform. Encourages more e f f i c i e n t work by t r u s t i n g an employee;-gives more r e s p o n s i b i l i t y to an i n d i v i d u a l as the q u a l i t y of work improves. Does not support decisions made by a subordinate (makes exceptions to the r u l e s ) . Praises p u b l i c a l l y for tasks completed .well and constructively c r i t i c i z e s i n private those i n d i v i d u a l s who have produced les s than adequate r e s u l t s . Gives everybody the same evaluation on the Performance Appraisal Reports even i f they are not doing a good job. Creates the impression that an "appointment" i s necessary for a subordinate who wants to discuss a personal problem. Continually discusses performance of others behind t h e i r backs creating a f e e l i n g of mistrust among the s t a f f . Explains benefits package and presents to the new entrant brochures to read a f t e r the plans are discussed. Relies on in-branch promotion as much as possible and makes s t a f f f e e l that getting ahead i s not an unreal objectives. T e l l s employees to mind t h e i r own business when they come out with new ideas. Sets r e a l i s t i c goals for every employee. Keeps s t a f f involved i n changes through regular meetings, encourages feedback- p o s i t i v e or negative - or p o l i c y changes. Frequently changes his mind and issues contradictory i n s t r u c t i o n s , having an adverse e f f e c t on morale of s t a f f and on work flow. Does not accept r e s p o n s i b i l i t y for errors but passes blame on to subordinates. Makes promises to i n d i v i d u a l s regarding promotion and salary which cannot be followed through. Creates a better working environment by maintaining close contacts with a l l s t a f f members ( i . e . , holds frequent discussions both casual and job-oriented.) Discusses private personal problems of subordinates with others. 238' Job Part: Training Descriptive of R e s p o n s i b i l i t i e s : any job rela t e d t r a i n i n g and development a c t i v i t y , explanations, support in problem solving, advise, informing, up-dating, course scheduling, t r a i n i n g follow-up Rating Procedure: Please, THINK CAREFULLY before you answer the question: Does this job incumbent e x h i b i t the i n following described behavior a l l the time? (Does i t describe his t y p i c a l behavior?) Use "Y" for Yes, "N" for No, and "?" i f you are not certain. Does not teach employee a l l aspects of job, expects an employee to learn i t by himself, e.g., says: "Don't know, look i t up i n the book." Ensures that new methods are put in t o p r a c t i c e through proper t r a i n i n g . Uses trainees as "Joe-boys", thus wasting valuable t r a i n i n g time. Ensures that i f a trainee i s trained by another s t a f f member that the l a t t e r i s competent enough to do the t r a i n i n g . Obtains t r a i n i n g aids from the Training D i v i s i o n to a s s i s t with in-branch tr a i n i n g . Has patience with slow learners, even i f some questions are asked several times. Explains why things are done, not only how. Encourages employees to d i r e c t work-related questions to him i f they have any doubts. Conducts follow-up and evaluation of any t r a i n i n g program. w i l l take the time to f a m i l i a r i z e himself with the t r a i n i n g material provided before using i t . Ensures that every t r a i n i n g l i t e r a t u r e applying to employees' job i s a v a i l a b l e . Organizes job rotation with the e f f e c t that each clerk know at least two jobs - i d e a l for vacation periods and absences. Is inconsistent i n explaining concepts and processes, thus leading to confusion. Encourages s t a f f to p a r t i c i p a t e i n courses the t r a i n i n g center has to o f f e r . Provides on-the-job t r a i n i n g to subordinates when necessary. Makes sure that h i s or her subordinates have the proper t r a i n i n g f o r the job they are doing i n order to achieve high e f f i c i e n c y . Takes time to explain new or amended procedures. Establishes an adequate t r a i n i n g l i b r a r y . Makes a habit of st e a d i l y reviewing the progress of subordinates, making them aware of strengths and weaknesses with suggestions for improvements where needed. Moves trainee on to next phase of job before assessing performance on previous post. Gives close attention to inexperienced or new s t a f f members when they are having t h e i r i n i t i a l contacts with branch customers. 239 Appendix E Sample of the JDI Think of the kind of supervision that you get on your job. How well does each of the following words describe your supervisor? In the blank beside each word below, put: Y i f i t describes the supervision you get on your job N i f i t does NOT describe i t ? i f you cannot decide SUPERVISION ON PRESENT JOB asks my advice hard to please impolite praises good work t a c t f u l i n f l u e n t i a l up-to-date doesn't supervise enough quick tempered t e l l s me where I stand annoying stubborn knows job w e l l bad i n t e l l i g e n t leaves me on my own around when needed lazy from Smith, P.C., Kendall, L.M., & Hulin, C.L. The Measurement of S a t i s f a c t i o n i n Work and Retirement. Chicago: Rand, McNally & Co., 1969 241 Appendix F 242 BRANCH ORGANIZATION MANUAL JOB DESCRIPTION A S S I S T A N T MANAGER / ADMIN ISTRAT ION MANAGER lVe.le.tz Om) BASIC FUNCTION Re s p o n s i b l e f o r p l a n n i n g , o r g a n i z i n g and s u p e r v i s i n g the g e n e r a l o f f i c e o p e r a t i o n and f o r l e n d i n g a d m i n i s t r a t i v e support t o the Manager. MAJOR DUTIES . S u p e r v i s e and ensure adherence to p o l i c i e s and p r o -cedures r e l a t i v e to the a d m i n i s t r a t i v e f u n c t i o n i n the branch. T h i s r e q u i r e s t h a t he: - Communicate i n d i v i d u a l j o b d e s c r i p t i o n c o n t e n t , e n s u r i n g mutual u n d e r s t a n d i n g , and proper d e l e -g a t i o n of work. . - E s t a b l i s h e f f e c t i v e a u d i t and c o n t r o l procedures, t o ensure t a s k s a re p r o p e r l y performed. T h i s w i l l i n c l u d e r e g u l a r p a r t i c i p a t i o n i n adjustments and a u d i t s , i n c l u d i n g t e l l e r s ' cash counts and the d a i l y a u d i t of the Cash Book and G e n e r a l Ledger, and c u s t o d i a n d u t i e s i f r e q u i r e d . - Communicate and e n f o r c e r e g u l a t i o n s r e l a t i v e t o . cash custody and s a f e g u a r d i n g . • - S u p e r v i s e the t i m e l y and a c c u r a t e s u b m i s s i o n o f a l l branch r e p o r t s . - Monitor the p r o f i t a b i l i t y of s e r v i c e s a v a i l e d o f and ensure the a c c u r a t e c a l c u l a t i o n and a c c o u n t i n g o f revenues, and j u s t i f i c a t i o n of c o s t s . - Keep informed of and implement new o r g a n i z a t i o n a l c oncepts, methods, and s c h e d u l i n g t e c h n i q u e s which improve the o p e r a t i n g e f f i c i e n c y o f the branch. - Ensure proper maintenance of premises and equipment. . C o - o r d i n a t e and s u p e r v i s e the e f f e c t i v e development, ' t r a i n i n g , a p p r a i s a l and f a i r a p p l i c a t i o n o f compensation p o l i c i e s to a l l branch p e r s o n n e l . T h i s r e q u i r e s t h a t he - Assess the c u r r e n t and lon g e r term p e r s o n n e l r e -quirements of the branch and develop an a p p r o p r i a t e a c t i o n p l a n . 243 - App ly and a d m i n i s t e r the B a n k ' s p o l i c i e s and p r o -grammes r e l a t i v e to per fo rmance a p p r a i s a l and compensa t ion f o r a l l p e r s o n n e l under h i s s u p e r v i s i o n . - A d m i n i s t e r the B a n k ' s " o n - t h e - j o b " t r a i n i n g p r o -grammes f o r development and more e f f e c t i v e u t i l i z a t i o n of. b r a n c h p e r s o n n e l . - I d e n t i f y h i g h p o t e n t i a l p e r s o n n e l and a r r a n g e f o r a c c e l e r a t e d development th rough f o r m a l t r a i n i n g programmes w i t h i n the Bank and e l s e w h e r e . . Support the Manager i n i m p r o v i n g the B r a n c h ' s share o f p r o f i t a b l e l o a n , d e p o s i t and s e r v i c e b u s i n e s s . T h i s r e q u i r e s t h a t he: . - I d e n t i f y c u s t o m e r s ' n e e d s , e f f e c t i v e l y s e l l the B a n k ' s s e r v i c e s and t r a i n p e r s o n n e l i n t h i s r e g a r d . - I n i t i a t e and a s s i s t i n imp lement ing b u s i n e s s p r o -mot ion c a m p a i g n s . . . - Promote e f f i c i e n t and c o u r t e o u s s e r v i c e . . Recommend a n n u a l and l o n g e r term o b j e c t i v e s f o r growth and p r o f i t o f the b r a n c h and a s s i s t the Manager i n d e -v e l o p i n g p l a n s f o r a c h i e v i n g t h e s e o b j e c t i v e s . D e v e l o p and c o n t r o l r e a l i s t i c budget and p r o f i t p l a n s and m a i n -t a i n a c c u r a t e r e c o r d s f rom which . t rends can...be. e s t a b - . l i s h e d . . N o t w i t h s t a n d i n g the major and p r i o r i t y f u n c t i o n s o u t l i n e d h e r e t o f o r e , the A s s i s t a n t Manager may be ^ < r e q u i r e d to a s s i s t the Manager i n the d a y - t o - d a y o :z l e n d i n g o p e r a t i o n s and must keep a b r e a s t o f l e n d i n g p o l i c i e s i n o r d e r t o assume r e s p o n s i b i l i t y f o r the ^ l e n d i n g f u n c t i o n d u r i n g the Manager ' s a b s e n c e . AUTHORITY' A u t h o r i t y l i m i t s commensurate w i t h a s s i g n e d r e s p o n s i b i l i t y w i l l be d e t a i l e d by way of memorandum and may be r e v i s e d f rom t ime t o t i m e . KEY RELATIONSHIPS ' . . . R e p o r t s t o : M a n a g e r . ' S u p e r v i s e s : L i s t the t i t l e s of p o s i t i o n s s u p e r -v i s e d as shown on the o r g a n i z a t i o n c h a r t . R e l a t i o n s h i p s , L i s t the t i t l e s o f p o s i t i o n s w i t h w i t h : whom f r e q u e n t a s s o c i a t i o n i s m a i n -t a i n e d . ( R e f e r to s e c t i o n on j o b d e s c r i p t i o n s ) . CLASS IFlCATIOH NOTE OBJECTIVE ACCOUNTING LEVEL 2 Accountant - U s u a l l y found a t L e v e l 2 branches w i t h s t a f f comple-ments r a n g i n g from 6 - 1 0 where the managers d i r e c t the d a y - t o -day o p e r a t i o n s f u n c t i o n t hrough the a c c o u n t a n t s and p e r s o n a l l y check the cash book and g e n e r a l l e d g e r . A l s o found a t new urban branches where the A c c o u n t a n t i s f r e q u e n t -l y r e q u i r e d to s u p e r v i s e the branch d u r i n g the Manager's absence f o r s h o r t p e r i o d s i n the i n i t i a l b u s i n e s s development s t a g e . To lend s t r o n g s u p p o r t to the Manager o f a s m a l l branch. A work-ing l e v e l p o s i t i o n t h a t a l s o s e r v e s to c o n t i n u e the t r a i n i n g and development o f o f f i c e r s i n the t e c h n i q u e s o f branch bank admin-i s t r a t i o n . . WORK PERFORMED SUPERVISI ON RECEIVED P r o v i d e s l e a d e r s h i p and d i r e c t i o n to o t h e r c o u n t e r o f f i c e r s ; p r o-motes and s e l l s the f u l l range o f bank s e r v i c e s ; a c t s as a pr i me r e f e r r a l p o i n t f o r the more i n v o l v e d c o u n t e r t r a n s a c t i o n s . A s s i s t s w i t h the D a i l y Comparison o f E n t r i e s ; a u d i t s a v a r i e t y of bookkeeping e n t r i e s and r e c o r d s . M a i n t a i n s b ranch s c h e d u l e s o f r e g u l a r and s p e c i a l r e c o r d b a l a n c e s , r e t u r n s and r e p o r t s ; v e r i f i e s the b a l a n c e s h e e t s of branch r e c o r d s ; p r e p a r e s and/or checks u s u a l l y on a r o t a t i o n b a s i s a l l branch r e t u r n s and r e p o r t s ; c o - o r d i n a t e s the c o m p i l a t i o n of i n f o r m a t i o n and makes the q u a r t e r l y p r o f i t c l o s i n g e n t r i e s . M a i n t a i n s i n . j o i n t c u s t o d y the n e g o t i a b l e c o l l a t e r a l and s a f e -k e e p i n g s e c u r i t i e s , or i s a Regu1ar c u s t o d i a n under the S e c u r i t i e s -U n i t Record System. M a i n t a i n s c u s t o d y o f the n o n - n e g o t i a b l e s e c u r i t i e s . A s s i s t s the Manager by s u p e r v i s i n g the p r o c e s s i n g o f the d a y - t o -day branch w o r k f l o w , the s c h e d u l i n g o f work and the t r a i n i n g o f the s t a f f . P r e p a r e s s t r a i g h t f o r w a r d o f f i c i a l c o r r e s p o n d e n c e f o r r e v i e w and s i g n a t u r e o f s u p e r i o r ; p r e p a r e s g e n e r a l c o r r e s p o n d e n c e p e r t a i n -ing t o the work f u n c t i o n ; s i g n s g e n e r a l c o r r e s p o n d e n c e o f a s t r a i g h t f o r w a r d n a t u r e . M a i n t a i n s the n e g o t i a b l e , n o n - n e g o t i a b l e c o l l a t e r a l and s a f e k e e p -ing s e c u r i t i e s r e c o r d s ; ensures forms a r e p r o p e r l y c o m p l e t e d , r e f e r s d o c u m e n t a t i o n to s u p e r i o r who r e t a i n s the r e s p o n s i b i l i t y f o r i t s v a l i d i t y and c o r r e c t n e s s ; f o l l o w s up to o b t a i n m i s s i n g o r c o r r e c t e d forms as r e q u i r e d . May be d e s i g n a t e d the o f f i c e r r e s p o n s i b l e f o r m a i n t a i n i n g a h i g h s t a n d a r d o f t e l l e r s e r v i c e , e t c . (The s t a n d a r d phrase i s t o be rec o r d e d . ) May s u b s t i t u t e f o r the Manager o r A c c o u n t a n t as r e q u i r e d . P erforms o t h e r d u t i e s as r e q u i r e d w h i c h may i n c l u d e t a k i n g term-pi an a p p l i c a t i o n s . A f t e r a s h o r t b r e a k - i n p e r i o d performs d u t i e s w i t h o u t a p p r e c i a b l e spur from s u p e r i o r . S u p e r i o r f o l l o w s p r o g r e s s through f r e q u e n t d i s c u s s i o n s and a r e g u l a r r e v i e w o f the work performed. R e f e r s r o u t i n e s i t u a t i o n s o f a c o n t e n t i o u s n a t u r e . SUPERVISION GIVEN KNOWLEDGE AND EXPERIENCE P r o v i d e s work s u p e r v i s i o n . A s s i g n s t a s k s , p r o v i d e s i n s t r u c t i o n s and checks work p r o d u c t , may p r e p a r e performance a p p r a i s a l r e p o r t s on s u b o r d i n a t e s f o r t r a i n i n g p urposes. R e q u i r e s a good knowledge o f bank s e r v i c e s and r e l a t e d a c c o u n t i n g and r e p o r t i n g p r o c e d u r e s . E x p e r i e n c e g a i n e d on A c c o u n t i n g L e v e l 1 and i n some cases L e v e l 2 p o s i t i o n s i s u s u a l l y r e q u i r e d . R e v i s e d November 1968 245 C L A S S ) F l C A T I O N NOTE WORK PERFORMED SUPERVISION RECEIVED SUPERVISION GIVEN KNOWLEDGE AND EXPERIENCE R e v i s e d March I969 Management - P . G . 1 - 3 - 5 B.A.0.- The r e s p o n s i b i l i t y f o r s u p e r v i s i n g t he day-to-day o p e r a t i o n s f u n c t i o n i n c l u d i n g the D a i l y Comparison o f E n t r i e s i s d e l e g a t e d to t h i s l e v e l . The Manager c l o s e l y f o l l o w s the o p e r a t i o n s f u n c t i o n t h r o u g h f r e q u e n t d i s c u s s i o n s and c o n d u c t s t he p e r s o n n e l f u n c t i o n w i t h a s s i s t a n c e from t h i s o f f i c e r . P o s i t i o n s a r e u s u a l l y found a t L e v e l 3 and a c t i ve Lev e l 2 u n i t s w i t h s t a f f complements r a n g i n g from 9 - 16-Ensures a h i g h s t a n d a r d o f customer s e r v i c e by d e v e l o p i n g i n the s t a f f an awareness f o r prompt e f f i c i e n t a t t e n t i o n t o c u s -tomers' needs; promotes and s e l l s the f u l l range o f bank s e r -v i c e s ; a s s i s t s s e t t i n g up s p e c i a l campaigns t o i n c r e a s e branch b u s i n e s s through the prom o t i o n o f s e r v i c e s ; e n s u r e s the e f f e c -t i v e use of. a d v e r t i s i n g m a t e r i a l . U s u a l l y i s d e s i g n a t e d the o f f i c e r r e s p o n s i b l e f o r m a i n t a i n i n g a h i g h s t a n d a r d of t e l l e r s e r v i c e , e t c . Respons i b l e f o r the D a i l y Comparison o f E n t r i e s ; checks the cash book and g e n e r a l l e d g e r , e t c . S u p e r v i s e s the o p e r a t i o n s f u n c t i o n ; e n s u r e s the day-to-day work f l o w i s p r o c e s s e d i n ac c o r d a n c e w i t h bank r e g u l a t i o n s and s t a n d a r d s , e n s u r e s the c o r r e c t n e s s o f branch r e c o r d s and p r o -per c o m p l e t i o n of r e t u r n s and r e p o r t s . C o n s u l t s w i t h s u p e r i o r c o n c e r n i n g the p l a n n i n g and s c h e d u l i n g o f work, the t r a i n i n g program, the i n s t a l l a t i o n o f new systems and a c c o r d i n g l y f o l l o w s an agreed upon c o u r s e o f a c t i o n . In branches a s s i g n e d an "Ac c o u n t a n t T r a i n e e " , d i r e c t s the t r a i n e e through the i n - b r a n c h t r a i n i n g program e n s u r i n g p r o p e r i n s t r u c t i o n and gu i d a n c e . As r e q u i r e d , p r e p a r e s and s i g n s s p e c i a l a p p r a i s a l r e p o r t s d i r e c t e d t o the D i s t r i c t T r a i n i n g O f f i c e r . M a i n t a i n s i n j o i n t c u s t o d y the n e g o t i a b l e c o l l a t e r a l and s a f e -k e e p i n g s e c u r i t i e s , o r i s a R e g u l a r c u s t o d i a n under the S e c u r i t i e s - U n i t Record System. R e s p o n s i b l e f o r the v a l i d i t y and c o r r e c t n e s s o f n e g o t i a b l e N c o l l a t e r a l s a f e k e e p i n g and d e p o s i t account a u t h o r i z a t i o n forms; .ensures m i s s i n g or c o r r e c t e d forms a r e o b t a i n e d as r e q u i r e d . P r e p a r e s o f f i c i a l and g e n e r a l c o r r e s p o n d e n c e p e r t a i n i n g t o the o p e r a t i o n s f u n c t i o n ; s i g n s g e n e r a l c o r r e s p o n d e n c e c o n c e r n i n g m a t t e r s o f a n o n - c o n t e n t i o u s n a t u r e . Ensures good branch housekeeping h a b i t s a r e m a i n t a i n e d and t h a t p r o p e r s e c u r i t y p r e c a u t i o n s a r e f o l l o w e d . May a s s i s t w i t h the loans f u n c t i o n as a mi nor p a r t o f the j o b . S u b s t i t u t e s f o r the Manager o r A s s i s t a n t Manager - A d m i n i s t r a -t i on as requ i r e d . Works under the s u p e r v i s i o n o f the Manager. S u p e r i o r keeps i n -formed of the day-to-day o p e r a t i o n s f u n c t i o n through f r e q u e n t d i s c u s s i o n s d u r i n g which problem a r e a s a r e d e f i n e d and c o u r s e s o f a c t i o n e s t a b l i s h e d . S u p e r v i s e s c l o s e l y t he work o f a l l s t a f f . A s s i g n s d u t i e s , g i v e s i n s t r u c t i o n s and checks work p r o d u c t . P r e p a r e s performance, a p p r a i s a l r e p o r t s on s t a f f f o r r e v i e w and d i s c u s s i o n w i t h i n d i - v i d u a l s by s u p e r i o r . ( A l t h o u g h a s s i s t a n t a c c o u n t a n t s do not per-form a p e r s o n n e l f u n c t i o n as such, they do t r a i n and i n s t r u c t o t h e r s and c o r r e c t m i s t a k e s . They a r e r e q u i r e d t o r e p o r t t h e i r v i e w s on the o f f i c e s i t u a t i o n s t o the A s s i s t a n t . Manager -A d m i n i s t r a t i o n . ) R e q u i r e s a good knowledge o f bank s e r v i c e s and a good u n d e r s t a n d -i n g o f o f f i c e p r o c e d u r e s and systems. E x p e r i e n c e i s u s u a l l y g a i n e d on a c c o u n t i n g L e v e l 2 and i n some cases L e v e l 3 p o s i t i o n s . 246 CLASSIFICATION NOTE WORK PERFORMED SUPERVISION RECEIVED SUPERVISION GIVEN KNOWLEDGE AND EXPERIENCE R e v i s e d March 1 9 6 9 Management - P.G. 7 ~ 9 - 11 B.A.O. - The b a s i c r e s p o n s i b i l i t i e s a r e t o see t h a t the o f f i c e o p e r a t e s a c c o r d i n g t o bank s t a n d a r d s and to s u p e r v i s e and d e v e l o p the s t a f f . The incumbent keeps the Manager f u l l y i nformed o f the s e f u n c t i o n s through f r e q u e n t d i s c u s s i o n s . Recommendations • c o n c e r n i n g the o p e r a t i o n s and p e r s o n n e l f u n c t i o n s a r e p r e s e n t e d i n f i n a l form f o r manager's a p p r o v a l p r i o r t o i m p l e m e n t a t i o n . These p o s i t i o n s a r e u s u a l l y found a t L e v e l h o r a t v e r y a c t i v e L e v e l 3 branches w i t h s t a f f complements i n the v i c i n i t y o f . • Promotes and s e l l s bank s e r v i c e s t hrough s u p p o r t s t a f f ; m a i n t a i n s a h i g h s t a n d a r d o f customer s e r v i c e by d e v e l o p i n g i n the s t a f f an awareness f o r prompt and e f f i c i e n t a t t e n t i o n t o cust o m e r s ' needs; e n s u r e s t h a t p o s i t i o n s a r e a d e q u a t e l y s t a f f e d d u r i n g b u s i n e s s h o u r s , o r g a n i z e s s p e c i a l campaigns to i n c r e a s e branch b u s i n e s s t h r o u g h the p r o m o t i o n o f s e r v i c e s ; e n s u r e s e f f e c t i v e use o f a d v e r t i s i n g m a t e r i a l . Handles minor c o m p l a i n t s . R e s p o n s i b l e f o r the D a i l y Comparison o f E n t r i e s ; e n s u r e s the branch o p e r a t e s a c c o r d i n g t o bank r e g u l a t i o n s and s t a n d a r d s ; persona 1 1 y checks the cash book and g e n e r a l l e d g e r ; e n s u r e s p r o p e r charges a r e c o l l e c t e d f o r a l l s e r v i c e s r e n d e r e d ; e n s u r e s the c o r r e c t n e s s o f branch r e c o r d s and the a c c u r a t e r e p o r t i n g o f i n f o r m a t i o n c a l l e d f o r on r e t u r n s and r e p o r t s ; p r e p a r e s o f f i c e c o r r e s p o n d e n c e i n f i na 1 form f o r s u p e r i o r ' s s i g n a t u r e ; p r e p a r e s and/or r e v i e w s and s i g n s g e n e r a l c o r r e s p o n d e n c e . Performs the p e r s o n n e l f u n c t i o n ; c o n d u c t s a t r a i n i n g program t o de v e l o p a s t a f f knowledgeable i n the performance o f t h e i r d u t i e s and i n the pr o m o t i o n o f bank s e r v i c e s ; recommends t o s u p e r i o r f o r a p p r o v a l i n t r a - o f f i c e t r a n s f e r n e c e s s a r y to ensure c o n t i n u e d h i g h q u a l i t y s e r v i c e and t o c o n t r i b u t e t o i n d i v i d u a l d e v e l o p -ment and growth; p r e p a r e s performance a p p r a i s a l r e p o r t s i n f i n a l f orm f o r r e v i e w and s i g n a t u r e o f s u p e r i o r ; d i s c u s s e s approved r e p o r t s w i t h o f f i c e r s c o n c e r n e d , ( i n some i n s t a n c e s i n v o l v i n g s e n i o r o f f i c e r s , the manager may conduct the performance a p p r a i -s a l i n t e r v i e w s ) ; c o u n s e l s s t a f f i n the performance o f t h e i r d u t i e s and con d u c t s i n i t i a l c o r r e c t i o n a l i n t e r v i e w s . In branches a s s i g n e d as "A c c o u n t a n t T r a i n e e " , d i r e c t s the t r a i n e e t h r o u g h the i n - b r a n c h t r a i n i n g program e n s u r i n g p r o p e r i n s t r u c -t i o n and g u i d a n c e . As r e q u i r e d , p r e p a r e s and s i g n s s p e c i a l a p p r a i s a l r e p o r t s d i r e c t e d t o the D i s t r i c t T r a i n i n g O f f i c e r . P l a n s and s c h e d u l e s the work o f the branch o r department t o maximize the u t i l i z a t i o n o f p e r s o n n e l and equipment; r e v i e w s j o b s t r u c t u r e s , work f l o w , systems and p r o c e d u r e s , recommends changes t o s u p e r i o r f o r a p p r o v a l ; implements approved changes and c l o s e l y f o l l o w s t h e i r e f f e c t i v e n e s s . Ensures good b r a n c h housekeeping h a b i t s a r e m a i n t a i n e d and t h a t p r o p e r s e c u r i t y p r e c a u t i o n s a r e f o l l o w e d . May a s s i s t w i t h loans f u n c t i o n as a mi nor pa r t o f the j o b . S u b s t i t u t e s f o r the Branch Manager o r A s s i s t a n t Manager - Admin-i s t r a t i o n i n h i s absence. Performs o t h e r d u t i e s as r e q u i r e d . Works under the g e n e r a l s u p e r v i s i o n of- the Branch Manager o f A s s i s t a n t Manager - A d m i n i s t r a t i o n . Keeps s u p e r i o r informed o f the o p e r a t i o n s f u n c t i o n through f r e q u e n t d i s c u s s i o n s . A l l s i t u a t i o n s which r e s u l t i n changes i n the o p e r a t i o n s f u n c t i o n and p e r s o n n e l a r e d i s c u s s e d w i t h the manager and approved p r i o r t o i m p l e m e n t a t i o n . C l o s e l y s u p e r v i s e s t h e s t a f f and a l l phases o f t h e branch r o u t i n e work. Handles minor p e r s o n n e l problems. To be e f f e c t i v e the incumbent must be s u p p o r t e d by one o r two key peop l e w i t h whom he works c l o s e l y . A s s i s t a n t a c c o u n t a n t s work t h r o u g h the A s s i s t a n t Manager - A d m i n i s t r a t i o n when d e a l i n g w i t h department and. s e c t i o n heads. R e q u i r e s a good knowledge o f bank s e r v i c e s , p r o c e d u r e s and systems and a good u n d e r s t a n d i n g o f p e r s o n n e l p o l i c i e s and p r a c t i c e s . A mature approach and the a b i l i t y t o o r g a n i z e , t r a i n and d e v e l o p s t a f f a r e p r e r e q u i s i t e s . P r e v i o u s e x p e r -i e n c e i n the more d i f f i c u l t A c c o u n t i n g L e v e l 3 p o s i t i o n s would p r o v i d e the n e c e s s a r y knowledge and background. 247 Proposed T i t l e : Proposed CIass i f i cat ion This position is found in a l l Level 5 branches and any other branch were the present Accountant's position is c l a s s i f i e d Level 5-Assistant Manager - Administration Assistant Management 2 Object i ve: Respons i b i1i t i es: The objective of this position is to provide the Branch Manager with a senior o f f i c e supervisor capable of the ove r a l l organization, planning and day-to-day operation of a large branch. The incumbent's major occupation is to work through and with others, build t h e i r s k i l l s , develop an e f f e c t i v e and e f f i c i e n t branch organization staffed by competent, well trained people capable of handling bank business, promoting bank services and ensuring that the branch projects a f r i e n d l y , business-l i k e i mage. It cannot be overly emphasized that success in t h i s position w i l l be measured by the incumbent's e f f e c t i v e -ness in reaching these objectives. Customer Service - Responsible for a high standard of develops in his s t a f f an awareness to customers' •up ensures customer service; for prompt and e f f i c i e n t attention needs; through delegates and close follow-that positions are adequately staffed during business hours; handles majority of customers' complaints. Organization and Planning - Supported by a smal1 group of key people, organizes, plans and schedules the work of the branch to maximize the u t i l i z a t i o n of personnel and equipment; reviews and improves job structures, work flow, systems and procedures; i n s t a l l s new methods and c l o s e l y follows t h e i r effectiveness. Training, Development and Supervision of Staff - Per-forms the personnel function; plans and implements a tr a i n i n g program to develop a s t a f f knowledgeable in the performance of th e i r duties and in the promotion of bank services; arranges i n t r a - o f f i c e transfers necessary to ensure continued high q u a l i t y service and to contribute to individual development and growth; rec r u i t s s t a f f ; conducts performance appraisal and correctional interviews; prepares and signs reports for Manager's review; grants or recommends salary i n - creases in accordance with established procedures. Prof i tabi1i ty - Responsible for minor p r o f i t s , reviews and analyzes branch services to ensure that p r o f i t -able charges are levied, reports results and i f warranted recommends to the Manager increases in rates; 248 Respons i b i 1 i t i es f o l l o w s p r o g r e s s o f minor p r o f i t s and i s r e s p o n s i b l e ( C o n 1 t ) t h rough d e l e g a t e s f o r the c o l l e c t i o n o f c h a r g e s f o r a l l s e r v i c e s r e n d e r e d ; i n i t i a t e s , d i r e c t s and p a r t i -c i p a t e s i n s p e c i a l campaigns to i n c r e a s e branch f i g u r e s t h r o u g h the p r o m o t i o n o f s e r v i c e s ; p l a n s e f f e c t i v e use o f a d v e r t i s i n g m a t e r i a l . A c c o u n t i n g - R e s p o n s i b l e t h r o u g h d e l e g a g e s f o r t h e c o r r e c t n e s s o f branch r e c o r d s and the a c c u r a t e r e p o r t -ing o f i n f o r m a t i o n c a l l e d f o r on r e t u r n s and r e p o r t s , the D a i l y Comparison o f E n t r i e s and f o r s e e i n g the branch o p e r a t e s a c c o r d i n g t o bank r e g u l a t i o n s and s t a n d a r d s ; examines and s i g n s r e t u r n s and r e p o r t s ; p r e p a r e s and s i g n s o f f i c i a l c o r r e s p o n d e n c e f o r a p p r o v a l o f the Manager, s i g n s c o r r e s p o n d e n c e o f a g e n e r a l n a t u r e . Premises and Equipment - R e s p o n s i b l e f o r maintenance o f p r e m i s e s and equipment, g e n e r a l branch housekeep-ing and the adherence t o branch s e c u r i t y p r e c a u t i o n s ; e n s u r e s j a n i t o r s e r v i c e s , r e p a i r s o r r e d e c o r a t i o n s a r e performed as r e q u i r e d ; approves o r . recommends e x p e n d i t u r e s i n a c c o r d a n c e w i t h e s t a b l i s h e d i n s t r u c -t i o n s . P e r forms o t h e r d u t i e s as r e q u i r e d . S u p e r v i s i o n : R e p o r t s t o the Manager and keeps him informed t h r o u g h r e g u l a r d i s c u s s i o n s and r e v i e w s . Major changes i n o r g a n i z a t i o n o r p r o c e d u r e s a r e d i s c u s s e d p r i o r t o i n s ta 1 1 a t i on. Knowledge and R e q u i r e s a good knowledge o f bank p o l i c i e s , p r o c e d u r e s , E x p e r i e n c e : systems and s e r v i c e s . Must be mature and p o s s e s s the a b i l i t i e s t o o r g a n i z e , s u p e r v i s e , t r a i n and d e v e l o p s t a f f . Must be a b l e t o work t h r o u g h o t h e r s . 249 Note: T h i s p o s i t i o n i s found i n l a r g e f u l l y d e p a r t m e n t a l i z e d branches where t h e p r e s e n t A c c o u n t a n t ' s p o s i t i o n i s c l a s s i f i e d L e v e l 6. Proposed T i t l e : A s s i s t a n t Manager - A d m i n i s t r a t i o n Proposed C l a s s i f i c a - A s s i s t a n t Management 3 t i o n : O b j e c t i v e : The o b j e c t i v e s o f t h i s p o s i t i o n i s t o p r o v i d e the Branch Manager w i t h a s e n i o r o f f i c e s u p e r v i s o r c a p a b l e o f the o v e r a l l o r g a n i z a t i o n , p l a n n i n g and day-to-day o p e r a t i o n o f a l a r g e b r a n c h . The incumbent's major o c c u p a t i o n i s t o work t h r o u g h and w i t h o t h e r s , b u i l d t h e i r s k i l l s , d e v e l o p an e f f e c t i v e and e f f i c i e n t branch o r g a n i z a t i o n s t a f f e d by competent, w e l l t r a i n e d p e o p l e c a p a b l e o f h a n d l i n g bank b u s i n e s s , p r o m o t i n g bank s e r v i c e s and e n s u r i n g t h a t the branch p r o j e c t s a f r i e n d l y , b u s i n e s s -1 i ke image. I t cannot be o v e r l y - e m p h a s i z e d t h a t s u c c e s s i n t h i s p o s i t i o n w i l l be measured by the incumbent's e f f e c t i v e -ness i n r e a c h i n g t h e s e o b j e c t i v e s . Respons i b i 1 i t i e s : Customer S e r v i c e - R e s p o n s i b l e f o r a h i g h s t a n d a r d o f customer s e r v i c e ; t h r o u g h a s s i s t a n t s and department heads d e v e l o p s i n h i s s t a f f an awareness f o r prompt and e f f i c i e n t a t t e n t i o n t o c u s t o m e r s ' needs; r e v i e w s and approves s c h e d u l e s t o e n s u r e t h a t p o s i t i o n s a r e adequate-l y s t a f f e d d u r i n g b u s i n e s s h o u r s . Respons i b l e f o r the i n v e s t i g a t i o n of c u s t o m e r s ' c o m p l a i n t s . O r g a n i z a t i o n and P l a n n i n g - R e s p o n s i b l e f o r the o v e r a l l o r g a n i z a t i o n of the b r a n c h ; c o - o r d i n a t e s and d i r e c t s the work t o maximize the u t i l i z a t i o n o f p e r s o n n e l and equipment; r e v i e w s and p l a n s w i t h department heads and a s s i s t a n t s o f f i c e o r g a n i z a t i o n , j o b s t r u c t u r e s , work methods, systems and p r o c e d u r e s ; r e s p o n s i b l e t h r o u g h s u b o r d i n a t e s f o r i m p l e m e n t a t i o n o f new methods. T r a i n i n g , Development and S u p e r v i s i o n o f S t a f f - Admi n- i s t e r s the p e r s o n n e l f u n c t i o n ; p l a n s and d i r e c t s a t r a i n i n g program to d e v e l o p a s t a f f k n o w l e d g e a b l e i n the performance o f t h e i r d u t i e s and i n the p r o m o t i o n o f bank s e r v i c e s ; a s s i s t e d by department heads a r r a n g e s i n t r a - o f f i c e t r a n s f e r s n e c e s s a r y t o e n s u r e c o n t i n u e d h i g h q u a l i t y s e r v i c e and t o c o n t r i b u t e t o i n d i v i d u a l ] development and growth; r e c r u i t s s t a f f ; c o n d u c t s p e r -formance a p p r a i s a l and c o r r e c t i o n a l i n t e r v i e w s ; a p proves and s i g n s r e p o r t s and recommendations f o r s a l a r y i n -c r e a s e s w i t h i n p o l i c y g u i d e l i n e s ; p r e p a r e s and s i g n s r e p o r t s on department heads and a s s i s t a n t s , recommends s a l a r y i n c r e a s e s and p r o m o t i o n s f o r a p p r o v a l o f t h e Manager. P r o f i t a b i 1 i t y - In i t i a t e s and r e v i e w s a n a l y s i s o f minor p r o f i t s ; i n v e s t i g a t e s u n s a t i s f a c t o r y t r e n d s ; approves 250 Respons i b i 1 i t i e s : ( C o n 1 t d ) o r recommends i n c r e a s e d c h a r g e s where r a t e s l e v i e d a r e i n c o n s i s t e n t w i t h s e r v i c e s ; c o - o r d i n a t e s s p e c i a l p r o -m o t i o n a l campaigns and d i r e c t s the e f f e c t i v e use o f a d v e r t i s i n g m a t e r i a l . A ccount i ng- - R e s p o n s i b l e t h r o u g h d e l e g a t e s f o r the c o r r e c t n e s s o f branch r e p o r t s and the a c c u r a t e r e p o r t -ing o f i n f o r m a t i o n c a l l e d f o r on r e t u r n s and r e p o r t s , the D a i l y Comparison of E n t r i e s and f o r s e e i n g the branch o p e r a t e s a c c o r d i n g t o bank r e g u l a t i o n s and s t a n d a r d s ; examines and s i g n s r e t u r n s and r e p o r t s ; p r e p a r e s and s i g n s o f f i c i a l c o r r e s p o n d e n c e ; s i g n s c o r r e s p o n d e n c e o f a g e n e r a l n a t u r e . P r e m i s e s and Equipment - R e s p o n s i b l e f o r maintenance o f p r e m i s e s and equipment and the adherence t o s e c u r i t y p r e c a u t i o n s ; e n s u r e s j a n i t o r s e r v i c e s , r e p a i r s o r r e d e c o r a t i o n s a r e performed as r e q u i r e d ; a p proves o r recommends e x p e n d i t u r e s i n a c c o r d a n c e w i t h e s t a b l i s h e d i n s t r u c t i o n s . P e r forms o t h e r d u t i e s as r e q u i r e d . Superv i s i on: R e p o r t s to and works under the d i r e c t i o n o f the Manager r e f e r r i n g o n l y unusual s i t u a t i o n s . Keeps the Manager informed t h r o u g h r e g u l a r i n f o r m a l d i s c u s s i o n s . Knowledge and Exper i ence: Must have a good g e n e r a l knowledge o f bank p o l i c i e s , p r o c e d u r e s , systems and s e r v i c e s ; must be s k i 11ed i n s u p e r v i s i n g and d e v e l o p i n g p e o p l e and i n o r g a n i z i n g work i n a complex s e t t i n g ; must be p e r c e p t i v e and p o s s e s s good l e a d e r s h i p q u a l i t i e s . C a p a b i l i t i e s and q u a l i f i c a t i o n s s h o u l d have been p r e v i o u s l y d e m o n s t r a t e d . 251 T h i s o f f i c e management p o s i t i o n i s found i n l a r g e , complex and f u l l y d e p a r t m e n t a l i z e d branches ( u s u a l l y a c e n t r a l branch) where the p r e s e n t A c c o u n t a n t ' s p o s i -t i o n i s c l a s s i f i e d L e v e l 1. A s s i s t a n t Manager - A d m i n i s t r a t i o n The o b j e c t i v e o f t h i s p o s i t i o n i s t o p r o v i d e the; Branch Manager w i t h a s e n i o r o f f i c e s u p e r v i s o r c a p a b l e o f the o v e r a l l o r g a n i z a t i o n , p l a n n i n g and day-to-day o p e r a t i o n o f a l a r g e b r a n c h . The incumbent's major o c c u p a t i o n i s t o work through and w i t h o t h e r s , b u i l d t h e i r s k i l l s , d e v e l o p an e f f e c t i v e and e f f i c i e n t b r anch o r g a n i z a t i o n s t a f f e d by competent, w e l l t r a i n e d p e o p l e c a p a b l e of h a n d l i n g bank b u s i n e s s , p r o moting bank s e r v i c e s and e n s u r i n g t h a t the branch p r o j e c t s a f r i e n d l y , b u s i n e s s -l i k e i mage. I t cannot be o v e r l y emphasized t h a t s u c c e s s i n t h i s p o s i -t i o n w i l l be measured by the incumbent's e f f e c t i v e n e s s i n r e a c h i n g t h e s e o b j e c t . i v e s . Customer S e r v i c e - R e s p o n s i b l e f o r the o v e r a l l e f f e c t i v e -ness o f customer s e r v i c e t o e n s u r e a h i g h s t a n d a r d i s m a i n t a i n e d ; a r r a n g e s w i t h department heads f o r group d i s c u s s i o n s d i r e c t e d toward d e v e l o p i n g i n the s t a f f an awareness of the need f o r prompt and e f f i c i e n t customer s e r v i c e ; a s s i s t s department heads w i t h e x p l a n a t i o n and i n s t a l l a t i o n o f new s e r v i c e s and fo1 1ows development through r e p o r t s , d i s c u s s i o n s and o b s e r v a t i o n s ; r e s p o n -s i b l e f o r the i n v e s t i g a t i o n o f customer c o m p l a i n t s . O r g a n i z a t i o n and P l a n n i n g - R e s p o n s i b l e f o r the d e v e l o p - ment and r e f i n e m e n t o f p o s i t i o n s w i t h i n the branch t o e n s u r e t h a t where p o s s i b l e , adequate p r o m o t i o n a l oppor-t u n i t i e s e x i s t and t h a t the c o n t i n u e d e f f i c i e n t o p e r a -t i o n o f the b ranch i s a s s u r e d ; i n c o l l a b o r a t i o n w i t h a s s i s t a n t s and department heads p l a n s and s c h e d u l e s the work o f the branch f o r maximum e f f e c t i v e u t i 1 i z a t i o n o f p e r s o n n e l and equipment; r e v i e w s the o r g a n i z a t i o n and s t r u c t u r e o f each department by e x a m i n i n g a n a l y s i s sub- mi t t e d by department heads; approves recommended changes i n e x i s t i n g methods to promote an e f f i c i e n t o p e r a t i o n t hrough s t a n d a r d i z a t i o n , improvement, s i m p l i -f i c a t i o n , d i s c o n t i n u a n c e o r new systems; r e v i e w s the i n s t a l l a t i o n o f new p r o c e d u r e s . T r a i n i n g , Development and S u p e r v i s i o n o f S t a f f - Respon-s i b l e f o r the a d m i n i s t r a t i o n o f p e r s o n n e l f u n c t i o n ; c l o s e l y s u p e r v i ses the r e c r u i t m e n t , t r a i n i n g , p o s i t i o n e v a l u a t i o n and s a l a r y a d m i n i s t r a t i o n programs; r e v i e w s proposed p e r s o n n e l r e q u i r e m e n t s ; r e v i e w s performance a p p r a i s a l r e p o r t s and recommendations p r e p a r e d by d e partment-heads, a u t h o r i z e s and/or recommends s a l a r y a d j u s t m e n t s w i t h i n p o l i c y g u i d e l i n e s ; c o n d u c t s p e r f o r -mance and c o r r e c t i o n a l i n t e r v i e w s o f a more d i f f i c u l t n a t u r e ; p r e p a r e s performance a p p r a i s a l r e p o r t s and Note: Proposed T i t l e : Proposed C l a s s i f i c a t i o n : Respons i b i 1 i t i es: 252 recommendations on immediate a s s i s t a n t s and department heads f o r a p p r o v a l o f the Manager; t h r o u g h d i s c u s s i o n s w i t h a s s i s t a n t s and department heads, approves and/or a r r a n g e s p r o m o t i o n s n e c e s s a r y to e n s u r e c o n t i n u e d h i g h q u a l i t y s e r v i c e and t o c o n t r i b u t e t o i n d i v i d u a l d e v e l -opment and growth. P r o f i t a b i l i t y - Reviews department heads' a n a l y s i s o f minor p r o f i t s ; i n i t i a t e s i n v e s t i g a t i o n o f u n s a t i s f a c t o r y t r e n d s ; approves o r recommends i n c r e a s e d c h a r g e s where r a t e s l e v i e d a r e i n e o n s i s t e n t w i t h s e r v i c e s ; c o - o r d i n -a t e s s p e c i a l p r o m o t i o n a l campaigns and d i r e c t s the e f f e c t i v e use o f a d v e r t i s i n g m a t e r i a 1 . Account i ng - Revi ews the branch a u d i t and c u s t o d y p r o c e d u r e s and p r a c t i c e s t o e n s u r e c o n f o r m i t y w i t h bank r e q u i r e m e n t s ; approves and s i g n s r e p o r t s and r e -t u r n s ; p r e p a r e s o r approves and s i g n s o f f i c i a l and g e n e r a l c o r r e s p o n d e n c e . Premises and Equipment - R e s p o n s i b l e f o r maintenance o f p r e m i s e s and equipment and the b ranch s e c u r i t y p r e -c a u t i o n s . Superv i s i o n : R e p o r t s to and works under the d i r e c t i o n o f the Manager r e f e r r i n g o n l y unusual s i t u a t i o n s o f a p o l i c y n a t u r e . Keeps management informed on g e n e r a l c o n d i -t i o n s t h r o u g h r e g u l a r i n f o r m a l d i s c u s s i o n s . Knowledge and Must have a good g e n e r a l knowledge o f bank p o l i c y , E x p e r i e n c e : p r o c e d u r e s , s e r v i c e s and branch p r a c t i c e s ; must be s k i l l e d i n s u p e r v i s i n g and d e v e l o p i n g p e o p l e , co-o r d i n a t i n g and o r g a n i z i n g work i n a. h i g h l y complex s e t t i n g ; must be p e r c e p t i v e and p o s s e s s good l e a d e r -s h i p qua 1 i t i e s ; must have proven ab i 1 i t y . R e s p o n s i b i 1 i t i e s : ( C o n 1 t d ) 253 Note: Proposed T i t l e : Proposed C l a s s i f i c a t i o n : O b j e c t i v e : Respons i b i 1 i t i e s : T h i s i s a s e n i o r o f f i c e management p o s i t i o n i n a com-p l e x c e n t r a l branch h a v i n g many s p e c i a l i z e d d e p a r t -ments and a s t a f f o f a p p r o x i m a t e l y 400 p o s s e s s i n g a wide range o f s k i l l s . A s s i s t a n t Manager - A d m i n i s t r a t i o n A s s i s t a n t Management 5 The o b j e c t i v e o f t h i s p o s i t i o n i s t o p r o v i d e t h e Branch Manager w i t h a s e n i o r o f f i c e s u p e r v i s o r c a p a b l e o f the o v e r a l l o r g a n i z a t i o n , p l a n n i n g and day-to-day o p e r a -t i o n o f a l a r g e b r a n c h . The incumbent's major occ u p a -t i o n i s t o work t h r o u g h and w i t h o t h e r s , b u i l d t h e i r s k i l l s , d e v e l o p an e f f e c t i v e and e f f i c i e n t branch o r -g a n i z a t i o n s t a f f e d by competent, w e l l t r a i n e d p e o p l e c a p a b l e o f h a n d l i n g bank b u s i n e s s , p r o m o t i n g bank s e r -v i c e s and e n s u r i n g t h a t the b ranch p r o j e c t s a f r i e n d l y , b u s i n e s s l i k e image. I t cannot be o v e r l y emphasized t h a t s u c c e s s i n t h i s p o s i t i o n w i l l be measured by the incumbent's e f f e c t i v e -ness i n r e a c h i n g t h e s e o b j e c t i v e s . Customer S e r v i c e - R e s p o n s i b l e f o r the overa 1 1 e f f e c t i v e -ness o f customer s e r v i c e t o e n s u r e a h i g h s t a n d a r d i s m a i n t a i n e d ; t h r o u g h d i s c u s s i o n s w i t h department heads e v a 1 u a t e s t h e i r p l a n s f o r . d e v e l o p i n g i n the s t a f f an awareness o f the need f o r prompt and e f f i c i e n t customer s e r v i c e ; d e l e g a t e s r e s p o n s i b i l i t y f o r the i n s t a l l a t i o n and e x p l a n a t i o n o f new s e r v i c e s , e v a 1 u a t e s r e s u l t s t h r o u g h r e p o r t s , d i s c u s s i o n s and o b s e r v a t i o n s ; r e s p o n -s i b l e f o r the i n v e s t i g a t i o n o f c u s tomers' c o m p l a i n t s . O r g a n i z a t i o n and P l a n n i n g - R e s p o n s i b l e f o r . t h e develop-ment and r e f i n e m e n t o f p o s i t i o n s w i t h i n t h e branch t o e n s u r e t h a t where p o s s i b l e , adequate p r o m o t i o n a l o p p o r t u n i t i e s e x i s t and t h a t t h e . c o n t i n u e d e f f i c i e n t o p e r a t i o n o f the branch i s a s s u r e d ; r e v i e w s work sched-u l e s t o e n s u r e maximum e f f e c t i v e u t i l i z a t i o n o f p e r -s o n n e l and equipment; approves department manager's p r o p o s a 1 s f o r the r e - o r g a n i z a t i o n and development o f systems and p r o c e d u r e s w i t h i n t h e i r a r e a ; f o l l o w s t h r o u g h f r e q u e n t d i s c u s s i o n s the i n s t a l l a t i o n o f new and r e v i s e d systems and p r o c e d u r e s . T r a i n i n g , Development and S u p e r v i s i o n o f S t a f f - Co- o r d i n a t e s the p e r s o n n e l f u n c t i o n t h r o u g h the P e r s o n n e l O f f i c e r ; c l o s e l y s u p e r v i s e s the a d m i n i s t r a t i o n o f the r e c r u i t m e n t , t r a i n i n g , P o s i t i o n E v a l u a t i o n and S a l a r y A d m i n i s t r a t i o n programs; r e v i e w s proposed p e r -s o n n e l r e q u i r e m e n t s , p e r s o n a l l y i n t e r v i e w s a p p l i c a n t s f o r more s e n i o r p o s i t i o n s b e f o r e h i r i n g , r e v i e w s p e r f o r m a n c e a p p r a i s a l r e p o r t s and recommendations p r e p a r e d by department heads, a u t h o r i z e s and/or recommends s a l a r y a d j u s t m e n t s w i t h i n p o l i c y g u i d e l i n e s ; 254 Respons i b i 1 i t i es: ( C o n 1 t ) S u p e r v i s i o n : Knowledge and Exper i ence: c o n d u c t s performance and c o r r e c t i o n a l i n t e r v i e w s of a more d i f f i c u l t n a t u r e ; p r e p a r e s performance a p p r a i -s a l r e p o r t s and recommendations on immediate a s s i s -t a n t s and department heads f o r a p p r o v a l o f the Manager; t h r o u g h ] , d i s c u s s i o n s w i t h a s s i s t a n t s and department heads, approves and/or a r r a n g e p r o m o t i o n s n e c e s s a r y to e n s u r e c o n t i n e d h i g h q u a l i t y s e r v i c e and t o c o n t r i b u t e t o i n d i v i d u a l development and growth. P r o f i t a b i 1 I t y - Examines department head's p l a n s and p r o p o s a 1 s f o r the development o f p r o f i t s i n t h e i r a r e a s ; approves i n c r e a s e d changes where r a t e s l e v i e d a r e i n c o n s i s t e n t w i t h s e r v i c e s ; r e v i ews '."ther r e s u l t s o f b u s i n e s s development campaigns. R e p o r t s t o and works under the d i r e c t i o n o f the Manager r e f e r r i n g o n l y unusual s i t u a t i o n s of a p o l i c y n a t u r e . Keeps management i n f o r m e d on g e n e r a l c o n d i t i o n s t h r o u g h r e g u l a r i n f o r m a l br i e f i ngs. Must be an e x p e r i e n c e d o f f i c e manager w i t h a broad knowledge o f p e r s o n n e l a d m i n i s t r a t i o n - r e c r u i t i n g , s e l e c t i o n , p r o m o t i o n , p o s i t i o n e v a l u a t i o n , s a l a r y a d m i n i s t r a t i o n , m e r i t r a t i n g and a p p r a i s a l , s y s tems, p r o c e d u r e s and o f f i c e o r g a n i z a t i o n ; must be a b l e t o work under d i r e c t i o n w i t h i n g e n e r a l p o l i c i e s and gu i d e 1 i nes; must p o s s e s s 1 e a d e r s h i p qua 1 i t i e s and be s k i l l e d i n s u p e r v i s i n g and d e v e l o p i n g p e o p l e . 255 BANK POSITION DESCRIPTION POSITION TITLE ADMINISTRATION OFFICER POSITION NUMBER INCUMENT(S) surname & i n i t i a 1 s..on 1 y BRANCH DEPT.(H.O. o r D i v . o n l y ) DIVISION DATE POSITION CHARACTERISTICS - (NATURE, SCOPE AND ORGANIZATIONAL RELATIONSHIPS) T h i s p o s i t i o n a s s i s t s the Manager by assuming' r e s p o n s i b i 1 i t y f o r the e f f i c i e n c y o f branch day-to-day a c c o u n t i n g and o p e r a t i o n a l a c t i -v i t i e s . S u p e r v i s e s branch p e r s o n n e l p e r f o r m i n g a c c o u n t i n g d e t a i l and d i r e c t -l y p e rforms the. more advanced a s p e c t s t h e r e o f . O r g a n i z e s c l e r i c a l work p r o c e d u r e s , s u p e r v i s e s c o u n t e r s e r v i c e t o the p u b l i c and a c t s as the p r i n c i p a l c o u n t e r o f f i c e r i n the b r a n c h , c a l l i n g f o r i n i t i a t i v e and s a l e s m a n s h i p i n p r o m o t i n g v a r i o u s Bank s e r v i c e s t o the p u b l i c . A p p r a i s e s branch manning p o s i t i o n and communicates r e q u i r e m e n t s and recommendations t o the Manager. C l o s e l y s u p e r v i s e s o n - t h e - j o b t r a i n -ing o f c l e r i c a l employees and m a i n t a i n s a l l p e r s o n n e l r e c o r d s and d o c u m e n t a t i o n . P r e p a r e s a l l branch s t a t i s t i c a l r e p o r t s and r e t u r n s t o Manager and D i v i s i on. Is a n s w e r a b l e t o the Manager f o r the e f f i c i e n c y o f branch o p e r a t i o n s . STATISTICAL DATA - ( P e r t i n e n t Data Only) - Branch complement -ACCOUNTABILITY - ( i . e . Answerab1e f o r the f o l l o w i n g end r e s u l t s ) PROVIDE e f f e c t i v e s u p p o r t T t o the Manager's p r o m o t i o n a l and s a l e s e f f o r t s by o r g a n i z i n g and s u p e r v i s i n g branch a c c o u n t i n g o p e r a t i o n s and c o u n t e r s e r v i c e t o e n s u r e a c o n t i n u i n g h i g h l e v e l o f customer s a t i s f a c t i o n . CONTRIBUTE t o branch o p e r a t i n g e f f i c i e n c y by o r g a n i z i n g c l e r i c a l p r o c e d u r e s and s u p e r v i s i n g o n - t h e - j o b t r a i n i n g and j o b r o t a t i o n amongst branch c l e r i c a l p o s i t i o n s . PREPARE r e q u i r e d r e p o r t s and r e t u r n s f o r use by Manager and D i v i s i o n . SERVE the p u b l i c a t the c o u n t e r ; PROMOTE and EXPLAIN the advantages o f the B a n k i s range o f s e r v i c e s . . 256 BANK POSITION DESCRIPTION POSITION T I T L E A S S I S T A N T MANAGER- PO S I T I O N NUMBER 0157-1104 BRANCH ADMINISTRATION I NCUMBENT(S) ( s u r n a m e & i n i t i a l s o n l y ) DEPT.(H.O. o r D i v . O n l y ) D I V I S I O N DATE POSITION CHARACTERISTICS - ( N a t u r e , S c o p e a n d O r g a n i z a t i o n a l R e l a t i o n s h i p s ) T h i s p o s i t i o n p r o v i d e s k e y s u p p o r t t o t h e M a n a g e r by a s s u m i n g r e s p o n -s i b i l i t y f o r a l l a s p e c t s o f b r a n c h o p e r a t i o n a l a d m i n i s t r a t i o n - p e r s o n n e l a c c o u n t i n g , r e p o r t i n g a n d o r g a n i z i n g f o r o p e r a t i o n a l e f f i c i e n c y - t h e r e b y f r e e -i n g t h e M a n a g e r f o r t h e v i t a l a r e a s o f b u s i n e s s d e v e l o p m e n t , c r e d i t , p r o f i t p l a n n i n g a n d o v e r a l l r e v i e w o f b r a n c h p r o g r e s s . D i r e c t l y o r by d e l e g a t i o n t o s u b o r d i n a t e s , p o s i t i o n d i s c h a r g e s a d m i n i s t r a -t i v e a c c o u n t a b i l i t i e s i n t h r e e f u n c t i o n a l a r e a s : P e r s o n n e l - E n s u r e s t h a t b r a n c h c o m p l e m e n t p o s i t i o n s a r e f i l l e d by q u a l i f i e d i n c u m b e n t s , i n v o l v i n g s e l e c t i o n , p l a c e m e n t a n d p r o m o t i o n o f b r a n c h e m p l o y e e s up t o a n d i n c l u d i n g t h e 7 5 ~ p o i n t l e v e l . C o m m u n i c a t e s c l o s e l y w i t h D i v i s i o n on m a n n i n g r e q u i r e m e n t s a n d n e g o t i a t e s d i r e c t l y w i t h c l e r i c a l j o b a p p l i -c a n t s s u b j e c t t o s p e c i f i c p o l i c y g u i d e l i n e s a n d f i n a l c o n f i r m a t i o n o f D i v i s i o n a n d s u p e r v i s e s m a i n t e n a n c e o f p e r s o n n e l r e c o r d s a n d d o c u m e n t a t i o n . O v e r s e e s b r a n c h o n - t h e - j o b t r a i n i n g p r o g r a m s , a p p r a i s e s e m p l o y e e s ' p e r f o r m -mance a n d p l a n s i n t e r n a l t r a n s f e r s f o r t h e b e s t b a l a n c e b e t w e e n t r a i n i n g n e e d s a n d o p e r a t i n g r e q u i r e m e n t s . C o u n s e l s a n d i n s t r u c t s b r a n c h s t a f f t o -w a r d i m p r o v e s s a l e s a n d s e r v i c e e f f e c t i v e n e s s . O p e r a t i o n s - R e v i e w s e f f i c i e n c y o f b r a n c h s y s t e m s , p r o c e d u r e s , a n d s t a n d a r d s o f c u s t o m e r s e r v i c e ; i n i t i a t e s c h a n g e s i n w o r k f l o w a n d , a s n e c e s s a r y , c o n -t r i b u t e s r e c o m m e n d a t i o n s t o D i v i s i o n a l O p e r a t i o n s D e p a r t m e n t f o r c h a n g e s i n t h e number and c o n t e n t o f c l e r i c a l p o s i t i o n s r e q u i r e d t o meet Bank s t a n d a r d s f o r r o u t i n e a n d p r o d u c t i v i t y . A c c o u n t i n g a n d R e p o r t i n g - O v e r s e a s , a l l b r a n c h a c c o u n t i n g a n d i n t e r n a l a u d i t p r o c e d u r e s a n d g e n e r a t e s t h e i n f o r m a t i o n n e c e s s a r y f o r management c o n t r o l by s u p e r v i s i n g t h e p r o d u c t i o n o f a l l b r a n c h r e p o r t s a n d r e t u r n s t o M a n a g e r a n d D i v i s i o n Is a n s w e r a b l e t o t h e M a n a g e r f o r t h e e f f e c t i v e a d m i n i s t r a t i o n a n d ' . c o n t r o l o f b r a n c h o p e r a t i o n s S T A T I S T I C A L DATA - ( P e r t i n e n t D a t a O n l y ) B r a n c h C o m p l e m e n t -ACCOUNTABILITY - ( B a s i c r e q u i r e m e n t s o f t h e p o s i t i o n ) ORGANIZE AND OVERSEE t h e p e r s o n n e l , a c c o u n t i n g , r e p o r t i n g a n d o p e r a t i o n s f u n c t i o n s o f t h e b r a n c h . MAINTAIN b r a n c h m a n n i n g r e q u i r e m e n t s ; SELECT a n d PLACE c a n d i d a t e s f o r p o s i -t i o n s up t o a n d i n c l u d i n g t h e 7 5 - p o i n t 1 e v e l . ORGANIZE AND OVERSEE b r a n c h t r a i n i n g p r o g r a m s ; PLAN a n d IMPLEMENT b r a n c h e m p l o y e e s ' j o b p r o g r e s s i o n t o y i e l d b e s t t r a i n i n g e x p o s u r e c o n s i s t e n t w i t h o p e r a t i n g r e q u i r e m e n t s . OVERSEE b r a n c h s y s t e m s , p r o c e d u r e s a n d c u s t o m e r - s e r v i c e ; I N I T I A T E p r o c e d u r -a l c h a n g e s t o m a i n t a i n o p e r a t i o n a l e f f i c i e n c y . S U P ERVISE p r o d u c t i o n o f r e p o r t s a n d r e t u r n s t o M a n a g e r andj.-Di v i s i o n . 257 BANK POSITION DESCRIPTION POSITION TITLE ASSISTANT MANAGER - POSITION NUMBER 0181-1104 BRANCH ADMINISTRATION I NCUMBENT (S) _^ (Name S i n i t i a l s o n l y ) DEPT.(H.O. o r D i v . o n l y ) DIVISION DATE POSITION CHARACTERISTICS - ( N a t u r e , Scope and O r g a n i z a t i o n a l R e l a t i o n s h i p s ) T h i s key p o s i t i o n p r o v i d e s key s u p p o r t t o the m a r k e t i n g e f f o r t s o f the Manager o f a l a r g e branch by assuming r e p o n s i b i 1 i t y f o r a l l a s p e c t s o f branch o p e r a t i o n a l a d m i n i s t r a t i o n and customer s e r v i c e - p e r s o n n e l , a c c o u n t i n g , r e -p o r t i n g and o r g a n i z i n g f o r o p e r a t i o n a l e f f i c i e n c y - t h e r e b y f r e e i n g t he Manager f o r the v i t a l a r e a s o f b u s i n e s s development, c r e d i t , p r o f i t p l a n n i n g and o v e r a l l r e v i e w o f branch p r o g r e s s . D i r e c t o r y o r by d e l e g a t i o n t o s u b o r d i n a t e s p o s i t i o n d i s c h a r g e s a d m i n i s -t r a t i v e a c c o u n t a b i l i t i e s i n t h r e e f u n c t i o n a l a r e a s . P e r s o n n e l - Ensures t h a t b ranch complement p o s i t i o n a r e f i l l e d by q u a l i f i e d incumbents, i n v o l v i n g s e l e c t i o n , placement and pr o m o t i o n o f branch employees up t o and i n c l u d i n g t he 75 p o i n t l e v e l . Communicates c l o s e l y w i t h D i v i s i o n on manning r e q u i r e m e n t s and n e g o t i a t e s d i r e c t l y w i t h c l e r i c a l j o b a p p l i c a n t s s u b j e c t to s p e c i f i c p o l i c y g u i d e l i n e s and f i n a l c o n f i r m a t i o n o f D i v i s i o n s . Oversees branch o n - t h e - j o b t r a i n i n g programs, a p p r a i s e s employees'; p e r f o r -mance and p l a n s i n t e r n a l t r a n s f e r s f o r b e s t b a l a n c e between o p e r a t i n g r e -qu i r e m e n t s and t r a i n i n g e x p o s u r e . M o t i v a t e s s u b o r d i n a t e s toward improved s a l e s and s e r v i c e e f f e c t i v e n e s s t h r o u g h c o n s t r u c t i v e i n t e r v i e w s and i n s t r u c -t i o n s . O p e r a t i o n s - Reviews e f f i c i e n c y o f branch o p e r a t i n g s y s t e m s , programming o f work f l o w and s t a n d a r d s o f customer s e r v i c e s as n e c e s s a r y , c o n t r i b u t e s recommendations t o D i v i s i o n O p e r a t i o n s Department f o r changes i n the number, and c o n t e n t o f c l e r i c a l p o s i t i o n s r e q u i r e d t o meet Bank s t a n d a r d s f o r r o u t i n e and p r o d u c t i v i t y . A c c o u n t i n g and R e p o r t i n g - Oversees branch a c c o u n t i n g p r o c e d u r e s and r e v i e w s e f f e c t i v e n e s s o f i n t e r n a l a u d i t c o n t r o l s . Generates the i n f o r m a t i o n nece-s s a r y f o r management c o n t r o l by o v e r s e e i n g t he p r o d u c t i o n o f a l l b ranch r e p o r t s and r e t u r n s t o Manager and D i v i s i o n . Is a n s w e r a b l e t o the Manager f o r t h e e f f e c t i v e a d m i n i s t r a t i o n and con-t r o l o f branch o p e r a t i o n s and f o r t he l e v e l o f customer s e r v i c e p r o v i d e d . STATISTICAL DATA - ( P e r t i n e n t Data Only) ACCOUNTABILITY - ( i . e . A n swerable f o r t he f o l l o w i n g end r e s u l t s ) ORGANIZE and OVERSEE the p e r s o n n e l , a c c o u n t i n g , r e p o r t i n g and o p e r a t i o n s f u n c t i o n s o f the branch t o e n s u r e e f f e c t i v e customer s e r v i c e as w e l l as o p e r a t i n g e f f i c i e n c y . MAINTAIN branch manning r e q u i r e m e n t s ; SELECT and PLACE c a n d i d a t e s f o r p o s i -t i o n s up t o and i n c l u d i n g t he 7 5 _ p o i n t l e v e l . ORGANIZE and OVERSEE branch t r a i n i n g programs; PLAN and IMPLEMENT branch employees' j o b p r o g r e s s i o n t o y i e l d b e s t t r a i n i n g e x p o s u r e c o n s i s t e n t w i t h o p e r a t i n g r e q u i r e m e n t s REVIEW branch s y s t e m s , p r o c e d u r e s and customer s e r v i c e : INITIATE p r o -c e d u r a l changes t o m a i n t a i n o p e r a t i o n a l e f f i c i e n c y . SUPERVISE p r o d u c t i o n o f r e p o r t s and r e t u r n s t o Manager and D i v i s i o n . 258 BANK POSITION DESCRIPTION POSITION TITLE ASS ISTANT MANAGER- POSITION NUMBER 0223-1104 ADMINISTRATION INCUMBENT(S) (Surname 6 . i n i t i a l s o n l y ) BRANCH DEPT.(H.O. o r D i v . Only) DIVISION DATE POSITION CHARACTERISTICS - ( N a t u r e , Scope and O r g a n i z a t i o n a l R e l a t i o n s h i p s ) T h i s p o s i t i o n p r o v i d e s key s u p p o r t t o the Manager o f a major branch by assuming d i r e c t i o n o f a l l g e n e r a l a d m i n i s t r a t i v e and s e r v i c e f u n c t i o n s i n o r d e r t o f r e e the s e n i o r p o s i t i o n s f o r c o n c e n t r a t i o n on v i t a l a c t i v i t i e s i n b u s i n e s s development, c r e d i t , p r o f i t p l a n n i n g and o v e r a l l r e v i e w o f branch p r o g r e s s . A c c o m p l i s h e s a c c o u n t a b i l i t i e s i n manning, a c c o u n t i n g , r e p o r t i n g and o p e r a t i o n s m a i n l y t h r o u g h department heads; r e s o l v e s o p e r a t i o n a l problems r e f e r r e d by t h e s e s u b o r d i n a t e s , r e l a y i n g o n l y p o l i c y q u e s t i o n s and s i m i l a r major m a t t e r s t o the Manager. P e r s o n n e l - Ensures t h a t a l l branch p o s i t i o n s a r e a d e q u a t e l y manned, i n -v o l v i n g s e l e c t i o n , placement and p r o m o t i o n o f branch employees up t o and i n c l u d i n g the 7 5 - p o i n t l e v e l . Communicates c l o s e l y w i t h D i v i s i o n on mann-i n g r e q u i r e m e n t s and n e g o t i a t e s d i r e c t l y w i t h c l e r i c a l j o b a p p l i c a n t s , sub-j e c t t o p o l i c y g u i d e l i n e s and f i n a l c o n f i r m a t i o n o f D i v i s i o n . P l a n s p e r - . sonn e l p r o g r e s s i o n w i t h i n the b r a n c h , a p p r a i s e s performance and p r o v i d e s l e a d e r s h i p f o r o n - t h e - j o b t r a i n i n g programs aimed a t y i e l d i n g maximum j o b p r o f i c i e n c y and s a l e s e f f e c t i v e n e s s i n the branch s t a f f . O p e r a t i o n s - Reviews e f f i c i e n c y o f branch o p e r a t i n g s y s t e m s , s t a n d a r d s o f customer s e r v i c e , and the number a n d . c o n t e n t o f c l e r i c a l p o s i t i o n s r e -q u i r e d t o h a n d l e branch w o r k l o a d s . Develops independent s o l u t i o n s t o o p e r -a t i o n a l problems t h r o u g h b a s i c s t a t i s t i c a l a n a l y s i s and s u b m i t s a p p r o p r i a t e recommendations t o D i v i s i o n O f f i c e . C o u n s e l s department heads on funda-mental problems o f o r g a n i z a t i o n and programming o f work f l o w w i t h i n t h e i r a r e a s o f o p e r a t i o n s . A c c o u n t i n g and R e p o r t i n g - Reviews e f f e c t i v e n e s s o f departments' a c c o u n t i n g p r o c e d u r e s and i n t e r n a l a u d i t c o n t r o l s . Oversees and c o o r d i n -a t e s the p r o d u c t i o n o f a l l r e p o r t s and r e t u r n s n e c e s s a r y f o r the i n f o r m a -t i o n o f branch and D i v i s i o n management. Is a n s w e r a b l e t o the Manager f o r the e f f e c t i v e a d m i n i s t r a t i o n and con-t r o l o f branch o p e r a t i o n s . STATISTICAL DATA - ( P e r t i n e n t d a t a o n l y ) Branch Complement -ACCOUNTABILITY - ( B a s i c Requirements o f the P o s i t i o n ) ORGANIZE, CO-ORDINATE and OVERSEE the p e r s o n n e l , a c c o u n t i n g , r e p o r t i n g and o p e r a t i o n s f u n c t i o n s o f the b r a n c h . MAINTAIN branch manning r e q u i r e m e n t s ; SELECT and PLACE c a n d i d a t e s f o r p o s i -t i o n s up t o and i n c l u d i n g the 7 5 - p o i n t l e v e l . REVIEW i n t e r n a l t r a i n i n g and development programs; PLAN and IMPLEMENT branch employees' j o b p r o g r e s s i o n t o y i e l d b e s t t r a i n i n g e x p o s u r e c o n s i s t e n t w i t h o p e r a t i n g r e q u i r e m e n t s . REVIEW branch systems, p r o c e d u r e s and customer s e r v i c e ; INITIATE p r o c e d u r a l changes t o m a i n t a i n o p e r a t i o n a l e f f i c i e n c y . OVERSEE p r o d u c t i o n o f r e p o r t s and r e t u r n s t o Manager and D i v i s i o n . 259 t R E V 4 t a i P O S I T I O N D E S C R I P T I O N BANK P O S I T I O N T I T I F A K H H I S T a A T I O H C . - I - I C E R p n s m r i u M U M P E R 0 1 ^ 9 - 2 2 0 4 . . I N C U M B E N T I S L ( t>UHNAME AND I N I T I A L S O N L V I HWANCH . : : — — P L P T . IH O OR DIV. O N L Y I _ D I V I S I O N . : : - D A T E . . _ P O S I T I O N C H A R A C T E R I S T I C S - I N A T U P E . S C O P E A N D O R G A N I Z A T I O N A L R E L A T I O N S H I P S ! This position i s accountable to organise and supervise branch day-to-day operations i n order to provide the operational efficiency and counter sales and service effectiveness necessary to support the manager's business development efforts and contribute to the achievement of branch's p r o f i t a b i l i t y objectives. Directly or by delegation to selected subordinates, position discharges accountabilities i n three functional areas: Personnel - To keep branch c l e r i c a l positions manned by qualified individuals, position supervises on-the-job training of branch personnel and arranges, with Division or by direct negotiation vdth job applicants, to f i l l c l e r i c a l vacancies in the branch complement. Offers to job applicants are governed by well defined Divisional policy guidelines and their f i n a l confirmation. Position i s authorized to implement internal transfers up to and including the ?5-point level and i s continually concerned vdth reconciling training needs vdth operating r^.-iuirements. Appraises subordinates' performance and gives then counsel and motivation for personal improvement i n job proficiency and sales effectiveness. • . Operations - Oversees branch systems, procedures and customer service; remains alert for operational problems and applies practical insight into the programming and revision of cle r i c a l work flow. As necessary, calls for Operations assistance i n reviewing the number and content of c l e r i c a l positions required to process branch -workload. Accounting and Reporting: - Oversees branch accounting and internal audit procedures and generates the information necessary for management control by preparing, or supervising preparation of, a l l branch reports and returns to Manager and Division. Is answerable to the Manager for the efficiency of branch operations. S T A T I S T I C A L D A T A - I P E H T I N C N T D A T A O N L V I A C C O U N T A B I L I T Y - " C - A N S W E R A B L E F O R T H E F O L L O W I N G E N D R E S U L T S ! O K L S A I U Z E A I - J D SUPERVISE the personnel, accounting, reporting and operations functions of the branch. ' KAII1TAIH branch manning requirements; SELECT and PLACE candidates for positions up to and including the 75 point l e v e l . SUPERVISE training of branch c l e r i c a l personnel; INITIATE internal transfers for maximum training and operating effectiveness. OVERSEE branch systems, procedures, counter sales effectiveness and customer service; INITIATE' procedural changes necessary to maintain operational efficiency and achieve objectives established by management for these areas. SUPERVISE production of reports and returns to Manager and Division. •260 Appendix G 261 THE UNIVERSITY OF BRITISH COLUMBIA 2075 WESBROOK MALL VANCOUVER, B.C., CANADA V6T 1W5 FACULTY OF COMMERCE AND BUSINESS ADMINISTRATION Hermann F. Schwind P l e a s e read c a r e f u l l y To a l l p a r t i c i p a n t s i n the development o f a T r a i n i n g E v a l u a t i o n Instrument In the process . j6 f d e v e l o p i n g the TEI s c a l e we have completed the f o l l o w i n g s t e p s : l 1. d e v e l o p i n g c r i t i c a l i n c i d e n t s f o r the j o b o f a bank a c c o u n t a n t 2 . v a l i d a t i n g t he c r i t i c a l i n c i d e n t s item by item 3. c a t e g o r i z i n g each i t e m i n t o one o f the f i v e agreed upon j o b c a t e g o r i e s o f an a c c o u n t a n t : A d m i n i s t r a t i o n , Customer R e l a t i o n s , M a r k e t i n g , P e r s o n n e l , T r a i n i n g ( a b b r . : A,C,M,P.T) We - the t r a i n i n g manager and the r e s e a r c h e r - would l i k e t o thank a l l o f you who p a r t i c i p a t e d d u r i n g t he p r e v i o u s phases arid t h o s e who a r e asked t o p a r t i c i p a t e i n the f o u r t h s t e p , t he r a t i n g o f the r e m a i n i n g items, i . e . t h o s e w h i c h s u r v i v e d t he second and t h i r d s t e p . A seven p o i n t s c a l e i s used, r a n g i n g from " t o t a l l y u n a c c e p t a b l e b e h a v i o u r " ( o r g r o s s l y i n e f f e c t i v e ) t o " e x t r e m e l y d e s i r a b l e b e h a v i o u r " (or h i g h l y e f f e c t i v e ) . t o t a 1 l y undes i r a b l e behav i o u r j n e i t h e r d e s i r a b l e n o r u n d e s i r a b l e b e h a v i o u r J L _ extreme 1y des i r a b l e behav i o u r 1 o r g r o s s 1y i n e f f e e t i v e behav i o u r J -3 n e i t h e r e f f e c t i v e nor i n e f f e c t i v e b e h a v i o u r _L 0 1 h i g h l y e f f e c t i v e b e h a v i o u r A c t u a l l y , you may choose between two seven p o i n t s c a l e s , the o t h e r u s i n g n e g a t i v e numbers f o r the u n d e s i r a b l e ( i n e f f e c t i v e ) b e h a v i o u r . But t h i s i s t h e o n l y d i f f e r e n c e . Choose anyone you f e e l c o m f o r t a b l e w i t h , but p l e a s e be c o n s i s t e n t , i . e . use o n l y one s c a l e f o r the r a t i n g p r o c e s s . The b e s t way t o r a t e the items i s t o choose one o f the f i v e c a t e g o r i e s d e f i n e d as the main d i m e n s i o n s o f an a c c o u n t a n t ' s j o b : A d m i n i s t r a t i o n , Customer..Relat i o n s , M a r k e t i n g , P e r s o n n e l , and J _ r a i n i n g , and r a t e item by ite m i n t h i s c a t e g o r y . T h i s way you w i l l q u i c k l y get a f e e l i n g where 262 i n d i v i d u a l items s h o u l d f i t on the above shown s c a l e s . A l l items on the a t t a c h e d l i s t a r e marked a c c o r d i n g t o the c a t e g o r i e s they b e l o n g t o as j u d g e d by 5 a c c o u n t a n t s . So p l e a s e choose a c a t e g o r y and r a t e i t em by i t e m , a n s w e r i n g the q u e s t i o n : "What degree o f e f f e c t i v e n e s s do you f e e l the item d e s c r i b e s ? " , o r : "How d e s i r a b l e i s the d e s c r i b e d b e h a v i o u r ? " . W r i t e the a t t r i b u t e d p o i n t v a l u e i n f r o n t o r below the item. A l l items w i l l be r e t a i n e d on w h i c h judges (you) a g r e e w i t h a s t a n d a r d d e v i a t i o n o f l e s s than 1 .5. The s t a n d a r d d e v i a t i o n i n d i c a t e s agreement o r d i s a g r e e m e n t among j u d g e s . A f t e r you f i n i s h e d one c a t e g o r y t a k e t h e second and r e p e a t the r a t i n g p r o -c e s s u n t i l a l l items a r e r a t e d . A f t e r the r a t i n g use the a t t a c h e d s h e e t and rank the f i v e c a t e g o r i e s i n terms o f t h e i r i mportance f o r the j o b of an a c c o u n t a n t (1 = most impor-t a n t , 5 = l e a s t i m p o r t a n t ) . P l e a s e r e t u r n the l i s t o f items and the rank-i n g s h e et t o your t r a i n i n g manager. Thank you f o r your c o o p e r a t i o n . T h i s e x e r c i s e a l o n e s h o u l d be o f i n t e r e s t t o you because never b e f o r e d i d you see a more i l l u s t r a t i v e l i s t o f e f f e c t i v e and i n e f f e c t i v e b e h a v i o u r o f an a c c o u n t a n t . Every p a r t i c i p a n t w i l l r e c e i v e a copy o f the f i n a l T r a i n i n g E v a l u a t i o n S c a l e . S i n c e r e l y , Hermann F. Schwind Research A s s i s t a n t Ph.D. cand. e n c l : 1 i tern l i s t 1 r a t i n g s h eet PS. Should you have any q u e s t i o n s c o n c e r n i n g the r a t i n g (or a n y t h i n g e l s e ) p l e a s e c a l l me a t , o r - s h o u l d I not be i n my o f f i c e - l e a v e a message a t I ' l l c a l l back. 263 Appendix H 264 ADMINISTRATION 1 2 Grossly I n e f f e c t i v e 6 7 Highly E f f e c t i v e Delegates work, t o subordinates e f f e c t i v e l y by taking care that I t i s evenly d i s t r i b u t e d according to a b i l i t y and knowledge of each employee. Has to borrow frequently required s t a t i o n a r i e s from other branches r e s u l t i n g i n delays and d i s r u p t i o n of work flow. Is unable to organize h i s own work load, i s constantly "catching up",f thus s e t t i n g a poor example f o r others • I s very conscious of s e c u r i t y , and e f f e c t i v e l y c o n t r o l s procedures, thereby r e s u l t i n g i n a l l personnel observing r u l e s and r e g u l a t i o n s . Ensures that housekeeping of banking area i s neat and desks are not messy, thus making the Branch look presentable and a pleasant place to do business. Drops work on a subordinate's desk to be completed without reviewing i t with that p a r t i c u l a r i n d i v i d u a l , causing confusion and backlog. Does not always observe deadlines for reports therefore causing hasty jobs, o f t e n i n c o r r e c t . • Has a high standard of l e t t e r writing,-good typing, and ensures that l e t t e r s are not signed by j u n i o r personnel. Keeps meticulous records and encourages s t a f f to do the same, thereby reducing e r r o r s and rework. Keeps close c o n t r o l on cash holdings thus minimizing r i s k s and maximizing p r o f i t s . Does not have a c l o s e r e l a t i o n s h i p with h i s immediate superior and as a r e s u l t communication i s poor and management i s frequently unaware of problems u n t i l they are c r i t i c a l . O f f e r s suggestions to superiors i n e f f o r t to improve e f f e c t i v e n e s s of new or e x i s t i n g p o l i c i e s . _ Ensures that a l l cross-checks f o r d i f f e r e n c e s are c a r r i e d out as soon as p o s s i b l e and by a q u a l i f i e d person. Is unable to develop a good d i a r y system, causing confusion and delays with work. Does not care that correspondence i s answered d a i l y . Knows a new procedure w e l l before explaining i t to the s t a f f . Checks vorV. b e f o r e i t "org out of the n f f i c . Has the Current Account and ^ e r f p n a l Cfcenuirj; ledgers balanced d a i l y . Keeps a check on amount of overtime and f i n d s out reason. Sees that charges are c o l l e c t e d properly. VALID? RATING 1 - 7 DO NOT WRITE IN THIS COLUMN (1) (2) (3) (4) (5). (?)__ C 8 ) _ < « ) _ (10)_ ( U ) _ (12) _ (13) _ (M)_ (15) (16)_ (")_ (19) _ (20) _ (21) ^ !<22>_ |(23)_ I<2*>_ (25) 265 Administration 2 -Analyzes the duties co be performed i n order that natters of high p r i o r i t y are attended to f i r s t . Keeps down admi n i s t r a t i v e costs through e f f i c i e n c y of operation, i . e . , s t a f f are p r o f i t a b l y employed, over ordering of stati o n a r y i s kept to a minimum, l i g h t s are not l e f t on a l l night, s t a f f do not work unnecessary overtime. Knows h i s manuals w e l l and i s able to quickly f i n d procedures to follow. j Observes l i m i t s of h i s own authority and r e f e r s a l l appropriate j decisions to manager. j. Delegates as much routine as possible, making core e f f i c i e n t use J of h i s t i n e by concentrating on more important problems. !. Keeps management informed of a l l aspects of branch operation. Sees to i t that balancing of ledgers e t c . , i s performed i n accordance with the bank's regulations. Checks general ledger e n t r i e s each morning, thus having an o v e r a l l c o n t r o l of the branch's accounting operations. Takes proper safeguards to combat frauds^ e t c . , such as holding funds u n t i l cheques c l e a r , e t c . Readily accepts the r e s p o n s i b i l i t y for decision-making i n ret u r n i n g cheques K.S.F., etc. Plans s t a f f v a c a t i o n so as to provide adequate customer s e r v i c e . Ensures that equipment i s maintained i n good working order. Checks tir.e sheets weekly and ensures accurate r e p o r t i n g . Keeps the s t a f f informed of new p o l i c i e s by ensuring d a i l y c i r c u l a r s are read by a l l . VALID RATING 1 - 7 DO NOT WRITE IN THIS COLUMN* (26) (27) _ (28) _ (29) _ <30)_ ( 3 D . (32) _ (33) ^ (34) _ (35) _ (36) __ (37) _ (38) _ (39) 266 CUSTOMER RELATIONS 1 2 Grossly I n e f f e c t i v e 6 7 Highly E f f e c t i v e Sees that customer correspondence i s answered the sane day i t a r r i v e s . Mixes with customers wherever possible i n order to e s t a b l i s h a good personal r e l a t i o n s h i p . Ensures that a l l employees are courteous toward customers. Is unable to say "no" to a customer r e q u i r i n g s p e c i a l treatment, thereby c r e a t i n g added burden on s t a f f members. Gives customer problems top p r i o r i t y and takes steps to see that the problems do not a r i s e again. Loses h i s temper at a customer i n p u b l i c . I f c l i e n t openly h o s t i l e does not get into an argument with him but o f f e r s to introduce to higher up. F a i l s to respond to a complaint of an important c l i e n t , r e s u l t i n g i n a l o s s of that p a r t i c u l a r account. When answering customer's telephone queries i s short and abrasive g i v i n g the customer a f e e l i n g of not being cared about. Treats srcall deposit customers as "small people". Leaves l e t t e r s from customers often unattended f o r long periods of t i n e or hands them to more j u n i o r s t a f f members who do not know the answer. When a customer appears, acknowledges t h e i r presence and l e t s them know that he w i l l be with them i n a minute i f unavoidably involved. Has customer complaints re f e r r e d to and handled by a senior o f f i c e r . I f a custoner i s complaining loudly that the s e r v i c e i s t e r r i b l e , the l i n e s are l o n g , t h i s accountant w i l l take the customer's deposit over the counter or stand behind the t e l l e r s and write up passbooks f o r then. J o i n s i n community a c t i v i t i e s such as Junior Chamber of Commerce and presents a good bank image. I f a customer i s not sure of an entry to h i s account t h i s accountant w i l l take time to explain the nature o f the entry ( i . e . , why he was charged for being overdrawn, etc.) I f a custoaer i s ignorant of a-bank's f a c i l i t i e s ( i . e . , accounts, d r a f t s , etc.) t h i s accountant w i l l f i n d out the customer's needs and then rake suggestions. Knows many customers by name. Accepts phone c a l l s when s e r v i c i n g a customer. Allows long l i n e - u p s of customers to form without being aware of then u n t i l i t i s brought to h i s a t t e n t i o n . V A L I D ? • R A T I N G 1 - 7 DO NOT WRlTi: IK THIS COLUMN (1) <2> _ _ Of (4) (5) (6) _ (7) _ (8) _ (9) . (10) _ j (12)_ (11) _ (15)_ U6)_ (17) _ (18) _ ! (19). I ! (20) (21) _ (22) _ (23) _ (24) _ (25) 267 Customer Relations . -» VALID? RATING 1 - 7 i Do NOT WRITE ; IN THIS ! COLUMN j > Explains bank p o l i c i e s to customers, e.g., why t h e i r cheque i s not cashed. i J (26) j Apologizes to customer-when s t a f f makes an e r r o r . (27) j Answers customer i n q u i r i e s i n terms which are r e a d i l y understoo d. (23) j T r i e s to s e l l the customer the services most p r o f i t a b l e to the bank, rather than to the customer's best i n t e r e s t . i l (29) 1 E f f e c t i v e l y c o n t r o l s good customer service by supervising t e l l e r s d a i l y d u t i e s to f i t around counter duties as they a r i s e . ' i i t (30) i Forgets the names of important customers. ( 3 1 ) _ ; Is not rude or t a l k s back to customer when customer i s rude. (32) ; Argues long and loud with a customer as to why a cheque should not be cashed. 1 (33) j Hakes appointments at the convenience of the custor.er i f po s s i b l e . i i (34) ' T e l l s complaining customer to take h is business elsewhere. (35) Always r e f e r s customer i n q u i r i e s to subordinates rather than dealing with the problem himself. (36) Ensures that a l l s t a f f members are aware of the standards of dress and neatness to avoid possible negative customer rea c t i o n s . (37) i I. i ! I 268 MARKETING1 1 2 Grossly i n e f f e c t i v e 6 7 Highly E f f e c t i v e Plans each campaign ahead and sets the goals for h i s branch. Ensures p u b l i c i s aware of new and b e n e f i c i a l timely services through in-branch a d v e r t i s i n g . Is conscious o f a l l opportunities f o r hew business and makes management aware of any new customer with p o t e n t i a l f o r more-business. Is unable to answer customer i n q u i r i e s about bank serv i c e s because of l a c k of product knowledge, causing d i s s a t i s f a c t i o n and a bad bank image. Does not use m a t e r i a l s provided t o advertize the Bank's s e r v i c e s . Sets up a s p e c i a l s a l e s area, e a s i l y a c c e s s i b l e to the customer, to inform about a new bank servi c e and to market the s e r v i c e . Reacts negatively to any campaign's, causing s i m i l a r reactions by subordinates. U t i l i z e s marketing posters to the best advantage i n the branch and r e g u l a r l y moves or replaces them. Does not reward subordinates f o r extra e f f o r t s i n promoting bank s e r v i c e s , thus causing i n d i v i d u a l s concerned to lose enthusiasm. Doe3 not take advantage of contact with non-customers to s o l i c i t new accounts. Has new a d v e r t i s i n g material f o r each item, makes s t a f f aware of i t . Sends out a d v e r t i s i n g l e a f l e t s with c r e d i t memos. Does nothing to market bank s e r v i c e s . " I f the customer wants i t he w i l l ask f o r i t " . Looks f o r new ways and ideas from within the Bank or from out-si d e to promote ( s e l l ) bank s e r v i c e s . When a new campaign i s announced t h i s accountant w i l l c a l l a s t a f f meeting, asking for suggestions how the branch can p a r t i c i p a t e e f f e c t i v e l y . Makes s t a f f aware of f u l l range of bank s e r v i c e s . Ensures that counter o f f i c e r s are c r o s s - s e l l i n g the bank's s e r v i c e s . Advertizes s a f e t y deposit boxes when there are none i n the branch, Follows through on out l i n e d promotion campaign. T r i e s to improve on areas where the branch i s weak, e.g., i f there are a number of unrcnted safety deposit boxes, he w i l l s e t up a campaign to s e l l them. VALID? RATING 1 - 7 DO NOT WRITE 111 THIS COLUMN. (1) (2)" Of (4) (6). <7>_ (8)_ (9) _ (10) . (11) . (12) _ (13) (14)_ (15) (16) . (17) . (18) _ (19) (20) (25) 2 6 9 Marketing Tends to s e l l a l l s e r v i c e s of the Bank at once to each customer, causing confusion. Has a sound knowledge of products and s e r v i c e s , enabling him to explain a l l advantages and b e n e f i t s to a customer. Guards against d i s p l a y i n g out-of-date m a t e r i a l . Ensures that d i r t y o r torn a d v e r t i s i n g posters are replaced promptly. F a i l s to inform s t a f f about new campaigns. Is not aggressive i n s e l l i n g new bank s e r v i c e s . Offers i n c e n t i v e s t o s t a f f to promote bank s e r v i c e s during campaigns. VALID? RATING 1 - 7 DO NOT WRITE IN THIS COLUMN (26) _ (27) _ 1 (28)_ j<29)_ ! <30)_ (31)_ (32) 270 PERSONNEL 1 2 C r o s s l y I n e f f e c t i v e 6 7 Highly E f f e c t i v e Shows great confidence i n subordinates and openly d i s p l a y s t h i s with the r e s u l t that they develop to meet his expectations. Is openly c r i t i c a l of subordinates i n front of both other s t a f f Berbers and customers. Confers with members of the s t a f f regarding d e c i s i o n s which w i l l a f f e c t them. Does not acknowledge and encourage e f f o r t s of subordinates thereby l o s i n g a m o t i v a t i o n a l t o o l . Ensures that a l l s t a f f members are aware of the duties they have t o perform. Does not look ahead i n the way of r e l i e f or part-time s t a f f , therefore w i l l be short of people during vacation time. Encourages core e f f i c i e n t work by t r u s t i n g an employee; gives more r e s p o n s i b i l i t y to an i n d i v i d u a l as the q u a l i t y o f work Improves. Does not support d e c i s i o n s made by a subordinate (makes exceptions to the r u l e s ) . P r a i s e s p u b l i c a l l y f o r tasks completed w e l l and c o n s t r u c t i v e l y c r i t i c i z e s i n p r i v a t e those i n d i v i d u a l s who have produced l e s s than adequate r e s u l t s . Gives everybody the same evaluation on. the Performance A p p r a i s a l Reports even i f they are not doing a good job. Ensures that personnel records are kep-t r i g h t up-to-date, and that reports are w r i t t e n on time, and that s a l a r y reviews are not overlooked. Creates the impression that an "appointment" i s necessary f o r a subordinate who wants to discuss a personal problem. C o n t i n u a l l y discusses performance o f others behind t h e i r backs c r e a t i n g a f e e l i n g o f mistrust among the s t a f f . Uses h i s performance a p p r a i s a l records to thoroughly discuss the s t a f f members' performance over the period covered and l e t s them know what i s expected of them i n the future. Explains benefits''package and presents to the new e n t r a n t brochures to read a f t e r the plans are discussed. R e l i e s on in-branch promotion as much as possible aad rakes s t a f f f e c i that g e t t i n g ahead i s not an unreal o b j e c t i v e . Bases h i s h i r i n g d e c i s i o n s on c r e d e n t i a l s and background and not on general appearance. T e l l s employees to mind t h e i r own business when they cosie out with new ideas. Introduces new s t a f f members to a l l the s t a f f . Sets r e a l i s t i c goals f o r every employee. 271 Personnel 2 -Is very f a i r . If he sees someone i s not suit e d to one job at the branch he w i l l consider him/her f o r other jobs and puts him/her up f o r tr a n s f e r . Keeps s t a f f involved i n changes through regular nectings, encourages feedback - p o s i t i v e or negative - or p o l i c y changes Frequently changes h i s mind and Issues contradictory I n s t r u c t i o n s , having an adverse e f f e c t on morale of s t a f f and oa work flow. Does not accept r e s p o n s i b i l i t y f o r e r r o r s but passes blame on t o subordinates. Makes promises to i n d i v i d u a l s regarding promotion and s a l a r y which cannot be followed through. Creates a b e t t e r working environment by maintaining close contacts with a l l s t a f f members ( i . e . , holds frequent discussions both casual and job^oriented.) Ensures that a l l s t a f f members are made aware of t h e i r r e s p ective s a l a r y ranges and when the next Increase w i l l be considered. Is always looking f o r people to move up i n the branch. Does not inform h i s subordinates that t h e i r progress has b« u n s a t i s f a c t o r y before giving them no t i c e . Makes performance a p p r a i s a l s a " s u r p r i s e " to subordinates; Discusses p r i v a t e personal problems of subordinates with others. L i s t e n s to subordinate's point of view when a mistake has been made and then gives constructive c r i t i c i s m . VALID? RATING 1 - 7 DO KOT WRITE IN THIS COLUMN (26) _ (27) _ (28) _ (29) _ (30) _ (31) _ (32) _ (33) _ (34) _ (35) _ (36) _ (37) 272 j TRAININO DO NOT WRITE IN THIS COLUMN VALTD? RATING 1 - 7 (I) (2) 1 2 3 4 5 6 7 Grossly Highly I n e f f e c t i v e E f f e c t i v e (3) (4) (5) Does not teach employee a l l aspects of job, expects an employee to l e a r n i t by himself, e.g., says: "Don't know, look i t up i n the book." (6) Ensures that new methods are put into p r a c t i c e through proper t r a i n i n g . (7) Does not care whether s t a f f i s c r o s s - t r a i n e d . (8) Uses trainees as "Joe-boys", thus wasting valuable t r a i n i n g t i n e . (9) Ensures that i f a t r a i n e e Is trained by another s t a f f member that the l a t t e r i s competent enough to do the t r a i n i n g . (10) T r i e s to l e a r n as much about each job as p o s s i b l e because i f someone t r a i n i n g a new person doesn't have answers to c e r t a i n questions, he would. (11) Obtains t r a i n i n g a i d s from the T r a i n i n g D i v i s i o n to a s s i s t with In^-branch t r a i n i n g . (12) Has patience with slow learners, even i f some questions are asked s e v e r a l t i n e s . Explains why_ things are done, not only how. (13) Encourages employees to d i r e c t work-related questions to h i n i f they have any doubts. (14) Recogniees when employees are having d i f f i c u l t y i n performing c e r t a i n tasks and sees to i t that the problems are solved through appropriate timing. (15) Says: " I explain i t only once, you better l i s t e n ! " (16) Ensures that t r a i n i n g programs are applied to i n d i v i d u a l s In accordance with the l a t t e r ' s a b i l i t y and progress rather than on the b a s i s of expediency d i c t a t e d by branch requiremients. (17) Considers sending an employee to t r a i n i n g programs a waste of time. (18) Has the a b i l i t y to convey p r e c i s e , accurate and a r t i c u l a t e job I n s t r u c t i o n s . (19) Conducts follow*up and evaluation of any t r a i n i n g program. (20) W i l l take the t i n e to f a m i l i a r i z e himself with the t r a i n i n g m a t e r i a l provided before using i t . (21) W i l l t e l l a t r a i n e e exactly what i s expected of him i n a s p e c i f i c t r a i n i n g program. (22) Ensures that every t r a i n i n g l i t e r a t u r e applying to employees' jobs i s a v a i l a b l e . • (23) E f f e c t i v e l y uses t r a i n i n g a i d s , e.g., blackboard, overhead p r o j e c t o r s , f l i p c h a r t s , video-tape. (24) Organizes job r o t a t i o n with the e f f e c t that each c l e r k knows at l e a s t two jobs - i d e a l for vacation periods and absences. (25) 273 T r a i n i n g 2 VALID? RATING 1 - 7 DO NOT WRITE IN THIS COLUMN Is inconsistent i n e x p l a i n i n g concepts and processes, thus l e a d i n g to confusion. (26) Encourages s t a f f to p a r t i c i p a t e i n courses the t r a i n i n e center has to o f f e r . (27) Regularly checks back on the work of t r a i n e e s . (28) Provides on-the-job t r a i n i n g to subordinates when necessary. (29) Makes sure that h i s or her subordinates have the proper t r a i n i n g f o r the job they are doing i n order to achieve nigh e f f i c i e n c y . (30) Takes time to e x p l a i n new or amended procedures. (31) W i l l prepare a s u i t a b l e work area f o r the person to be t r a i n e d . (32) Schedules employees f o r t r a i n i n g courses. (33) Plans the tasks before a trainee a r r i v e s at the branch. (34) E s t a b l i s h e s an adequate t r a i n i n g l i b r a r y . (35) Makes a habit of s t e a d i l y reviewing the progress of subordinates, making them aware of strengths and weaknesses with suggestions f o r improvements where needed. (36) Obtains feedback from t r a i n e r as to progress of t r a i n e e . (37) Moves t r a i n e e on to next phase of Job before assessing performance on previous post. (38) Gives c l o s e a t t e n t i o n to inexperienced or new s t a f f members when they are having t h e i r i n i t i a l contacts with branch customers. (39) 27,4 Appendix I 275 Job Part: Administration Description of R e s p o n s i b i l i t i e s : quarterly analyses, monthly reports, routine work, branch security, i n t e r n a l correspon-dence with other banks, cooperation with manager, supplies, housekeeping, repairs Rating Procedure: Read every example of the job incumbent's behavior and then put a check-mark (•/) by the example that best represents how you could expect the incumbent you are rating to t y p i c a l l y perform i n t h i s aspect of his job. This Accountant (Administration O f f i c e r ) : 7. can be expected to be conscious of security, and e f f e c t i v e l y control procedures, thereby r e s u l t i n g i n a l l personnel observing rules and regulations. 6. can be expected to keep down administrative costs through e f f i c i e n c y of operation, i . e . , s t a f f are p r o f i t a b l y employed, over ordering of stationary i s kept to a minimum, l i g h t s are not l e f t on a l l night, s t a f f do not work unnecessary overtime. 5. can be expected to have the Current Account and Personal Chequing ledgers balanced d a i l y . 3. can be expected not to observe deadlines for re-ports therefore causing hasty jobs, often incorrect. 2. can be expected to be unable to organize his own work load, constantly try i n g to "catch up", thus setting a poor example for others. 1. can be expected to be unable to develop a good diary system, causing confusion and delays with work. 276 Job Part: Customer Relations Description of R e s p o n s i b i l i t i e s : complaints, correspondence, spe c i a l service (counselling i n complex matters), service maintenance, special e f f o r t s Rating Procedure: Read every example of the job incumbent's behavior and then put a check-mark ( ) by the example that best represents how you could expect the incum-bent you are rating to t y p i c a l l y perform i n t h i s aspect of his job. This Accountant (Administration O f f i c e r ) : 7. can be expected to ensure that a l l employees are .courteous toward customers. 6. can be expected to give customer problems top p r i o r i t y and to take steps to see that the problems do not arise again. 5. can be expected to see that customer correspondence i s answered the same day i t arrive s . 3. can be expected to accept phone c a l l s when se r v i c -ing a customer. 2. can be expected to leave l e t t e r s from customers often unattended for long periods of time or handing them to more junior s t a f f members who do not know the answer. 1. can be expected to lose his temper at a customer i n public. 277 Job Part: Marketing Description of R e s p o n s i b i l i t i e s : advertising, display material l o c a l campaigns, open house, marketing of bank services Rating Procedure: Read every example of the job incumbent's behaviour and then put a check-mark ( ) by the example that best represents how you could expect the incumbent you are rating to t y p i c a l l y perform i n t h i s aspect of his job. This Accountant (Administration O f f i c e r ) : 7. can be expected to have sound knowledge of products and services, enabling him to explain a l l advan-tages and benefits to a customer. 6. can be expected that when a new campaign i s announced he w i l l c a l l a s t a f f meeting, asking for suggestions how the branch can p a r t i c i p a t e e f f e c t i v e -l y . 5. can be expected to ensure public i s aware of new and b e n e f i c i a l timely services through in-branch advertising. 3. can be expected not to reward subordinates for e f f o r t s i n promoting bank services, thus causing individuals concerned to lose enthusiasm. 2. can be expected to do nothing to market bank services. "If the customer wants i t he w i l l ask for i t " . 1. can be expected to be unable to answer customer i n q u i r i e s about bank services because of lack of product knowledge, causing d i s s a t i s f a c t i o n and a bad bank image. 278 Job Part: Personnel Description of R e s p o n s i b i l i t i e s : salary administration, per-formance appraisal, h i r i n g and f i r i n g , d i s c i p l i n a r y actions, motivation, encouragement, moral support, per-sonal problems, vacation and holiday scheduling Rating and Procedure: Read every example of the job incumbent's behavior and then put a check-mark (/) by the example that best represents how you could expect the incum-bent you are rating to t y p i c a l l y perform i n thi s aspect of his job. This Accountant (Administration O f f i c e r ) : 7. can be expected to praise p u b l i c a l l y for tasks completed well and constructively c r i t i c i z e s i n private those individuals who have produced less than adequate r e s u l t s . 6. can be expected to show great confidence i n sub-ordinates and openly displays t h i s with the r e s u l t that they develop to meet his expectations. 5. can be expected to ensure that personnel records are kept r i g h t up-to-date, and that reports are written on time, and that salary reviews are not overlooked. 3. can be expected to make performance appraisals a "surprise" to subordinates. 2. can be expected not to support decisions made by a subordinate (makes exceptions to the r u l e s ) . 1. can be expected not to accept r e s p o n s i b i l i t y f or errors but passes blame on to subordinates. J o b P a r t : T r a i n i n g D e s c r i p t i o n o f R e s p o n s i b i l i t i e s : any j o b r e l a t e d t r a i n i n g and d e v e l o p m e n t a c t i v i t y , e x p l a n a t i o n s , s u p p o r t i n p r o -b l e m s o l v i n g , a d v i s e , i n f o r m i n g , u p - d a t i n g , c o u r s e s c h e d u l -i n g , t r a i n i n g f o l l o w - u p R a t i n g P r o c e d u r e : Read e v e r y example o f t h e j o b i n c u m b e n t ' s b e h a v i o r and t h e n p u t a check-mark (•/)• by t h e example t h a t b e s t r e p r e s e n t s how y o u c o u l d e x p e c t t h e i n c u m b e n t y o u a r e r a t i n g t o t y p i c a l l y p e r f o r m i n t h i s a s p e c t o f , h i s j o b . T h i s A c c o u n t a n t ( A d m i n i s t r a t i o n O f f i c e r ) : 7. c a n be e x p e c t e d t o make a h a b i t o f s t e a d i l y r e -v i e w i n g h i s p r o g r e s s o f s u b o r d i n a t e s , making them aware o f s t r e n g t h s and w e a k n e s s e s w i t h s u g g e s t i o n s f o r i m provements where n e e d e d . 6. c a n be e x p e c t e d t o c o n d u c t f o l l o w - u p and e v a l u a -t i o n o f any t r a i n i n g p r o g r a m . 5.. c a n be e x p e c t e d t o e f f e c t i v e l y e u s e t r a i n i n g a i d s , 5 e.g., b l a c k b o a r d , o v e r h e a d p r o j e c t o r s / f l i p c h a r t s , v i d e o - t a p e . 3. c a n be e x p e c t e d t o move t r a i n e e on t h e n e x t p h a s e o f j o b b e f o r e a s s e s s i n g p e r f o r m a n c e on p r e v i o u s p o s t . 2. c a n be e x p e c t e d t o be i n c o n s i s t e n t i n e x p l a i n i n g c o n c e p t s and p r o c e s s e s , t h u s l e a d i n g t o c o n f u s i o n . 1. c a n be e x p e c t e d t o s a y : " I e x p l a i n i t o n l y o n c e , y o u b e t t e r l i s t e n " . 280 Appendix K 281 Global Rating Scale A team of t r a i n i n g managers of 5 Canadian banks agreed that.the job of a bank accountant (a d m i n i s t r a t i o n o f f i c e r ) can be d i v i d e d i n t o 5 areas: A d m i n i s t r a t i o n , Customer Rel a t i o n s , Marketing, Personnel, and T r a i n i n g . You are asked to give an o v e r a l l r a t i n g of the job performance of your accountant (ad m i n i s t r a t i o n o f f i c e r ) i n the 5 areas mentioned, using a 7 p o i n t s c a l e . His performance i s : / unaccept- mediocre s l i g h t l y average s l i g h t l y s u p e r i o r outstanding able below above average average Job Area Please c i r c l e the appropriate number Administration Customer Relations 1 1 2 t 3 1 4 1 5 • 6 t 7 1 1 1 2 i 3 1 4 1 5 i 6 1 7 i Marketing 1 1 2 1 3 1 4 1 5 1 6 1 7 » Personnel T r a i n i n g 1 I 2 . 1 3 t 4 i 5 t 6 i 7 t 1 , i 2 1 3 i 4 1 5 i 6 1 7 282 Appendix L Rating Errors 283 Be aware of the 3 main problems with performance r a t i n g : Halo, Leniency, Central Tendency. Halo i s the tendency of a rater to rate an i n d i v i d u a l s i m i l a r l y on d i f f e r e n t c h a r a c t e r i s t i c s , e.g. an employee i s rated high on industriousness, therefore he i s rated high on a l l other t r a i t s . Of course, t h i s i s not necessarily true. A worker can be very industrious, yet the qua l i t y of h i s work may be very low. In order to reduce the halo e f f e c t a rater has to look at every c h a r a c t e r i s t i c or job dimension i n d i v i d u a l l y . Example: the ratings (on a 7 point scale) 6,6,6,6,7,6 very l i k e l y contain a large halo e f f e c t , while the ratings 5,6,4,7,3,5 seem to contain less halo. Leniency i s the i n c l i n a t i o n of a rater to rate people consistently above average, i . e . to be a "nice guy". This i s , of course, a d i s t o r t i o n , since i f we take the average performance of a group, some people must be below and some above average. To avoid or reduce the leniency e f f e c t raters have to define "average" performance or have to i d e n t i f y a member of.the work group who works at the average performance l e v e l of the group, and then compare the other members of the group with him. F i n a l l y , the central tendency e f f e c t takes place when a rater i s reluctant to use extreme ratings, e.g. excellent or very poor, even i f warranted. The r e s u l t i s a "peaking" of ratings i n the middle range. It can be avoided by consciously i d e n t i f y i n g the outstanding and very poor performers and applying the appropriate ratings. These rater errors are enhanced by the vagueness of many t r a i t descriptions or d e f i n i t i o n s and the fact that they are not observable. The use of job related and observable behavior as performance c r i t e r i a should reduce these errors. A rater's awareness of these common mistakes w i l l c e r t a i n l y aid i n t h i s task. 285 Appendix M 286 Job Part: . Administration Description of R e s p o n s i b i l i t i e s : quarterly analyses, monthly reports, routine work, branch security, i n t e r n a l correspondence, correspondence with other banks, cooperation with manager, supplies, housekeeping, repairs Rating Procedure: Please, THINK CAREFULLY before you answer the question: How offen does this job incumbent e x h i b i t the following behavior? C i r c l e the appropriate anchor. e.g. Always Delegates work to subordinates e f f e c t i v e l y by taking care that i t i s e v e n l y + Some-times d i s t r i b u t e d according to a b i l i t y and knowledge of each employee Has to borrow frequently required s t a t i o n a r i e s from other branches r e s u l t i n g in delays and d i s r u p t i o n of work flow. Is unable to organize h i s own work load, i s constantly "catching up" thus s e t t i n g a poor example for others. Is very conscious of s e c u r i t y , and e f f e c t i v e l y controls procedures thereby r e s u l t i n g i n a l l personnel observing rules and regulations. Ensures that housekeeping of banking are i s neat and desks are not messy, thus making the Branch look presentable and a pleasant place to do business Drops work on a subordinate's desk to be completed without reviewing i t with that p a r t i c u l a r i n d i v i d u a l , causing confusion and backlog. Has a high standard of l e t t e r w r i t i n g , good typing, and ensures that l e t t e r s are not signed by j u n i o r personnel. 1 Keeps meticulous records and encourages s t a f f to do the same, thereby reducing errors and rework. Keeps close c o n t r o l on cash holdings thus minimizing r i s k s and maximizing p r o f i t s . O ffers suggestions to superiors i n e f f o r t to improve e f f e c t i v e n s s s of new or e x i s t i n g p o l i c i e s . Is unable to develop a good diary system, causing confusion and delays with work. Does not care that correspondence i s answered d a i l y . Knows a new procedure w e l l before explaining i t to the s t a f f . Keeps a check on amount of overtime and finds out reason. Analyzes the duties to be performed i n order that matters of high p r i o r i t y are attended to f i r s t . Keeps down administrative costs through e f f i c i e n c y of operation, i . e . , s t a f f are p r o f i t a b l y employed, over ordering of stationary i s kept to a minimum, l i g h t s are not l e f t on a l l night, s t a f f do not work unnecessary overtime. Knows h i s manuals w e l l and i s able t o q u i c k l y f i n d procedures t o f o l l o w . Keeps management informed of a l l aspects of branch operation. Takes proper safeguards to combat frauds, etc., such as holding funds u n t i l cheques c l e a r , etc. Plans s t a f f vacation so as to provide adequate customer service. Often 287 Job Part: Customer Relations Description of R e s p o n s i b i l i t i e s : complaints, correspondence, s p e c i a l service (counseling in complex matters), service maintenance, s p e c i a l efforts,-Rating Procedure: Please, THINK CAREFULLY before you answer the question: How often does this job incumbent e x h i b i t the following behaviors? C i r c l e the appropriate anchor, e.g. i © 1 1 Always Sees that customer correspondence i s answered the same day i t a r r i v e s . I Mixes with customers wherever possible i n order to e s t a b l i s h a good personal r e l a t i o n s h i p . , '. Ensures that a l l employees are courteous toward customers. i Gives customer problems top p r i o r i t y and takes steps to see that the problems do not a r i s e again. I Loses h i s temper at a customer i n p u b l i c . I When answering customer's telephone queries i s short and abrasive g i v i n g the customer a f e e l i n g of not being cared about. 1 When a customer appears, acknowledges t h e i r presence and l e t s them know that he w i l l be with them i n a minute i f unavoidably involved. ' Joins i n community a c t i v i t i e s such as Junior Chamber of Commerce and presents a good bank image. 1 If a customer i s not sure of an entry to h i s account t h i s accountant w i l l take time to explain the nature of the entry ( i . e . , why he was I charged for being overdrawn, etc.) I f a customer i s ignorant of a.bank's f a c i l i t i e s ( i . e . , accounts, d r a f t s , etc.) t h i s accountant w i l l f i n d out the customer's needs and then malce ' suggestions. Knows many customers by name. I — Accepts phone c a l l s when s e r v i c i n g a customer. I Explains bank p o l i c i e s to customers, e.g., why t h e i r cheque i s not cashed. Apologizes to customer when s t a f f makes an err o r . ( Tries to s e l l the customer the services most p r o f i t a b l e to the bank, rather than to the customer's best i n t e r e s t . I — Forgets the names of important customers. ^ ! Argues long and loud with a customer as to why a cheque should not be cashed. i — Makes appointments at the convenience of the customer i f p o s s i b l e . , T e l l s complaining customer to take h i s business elsewhere. ( Ensures that a l l s t a f f members are aware of the standards of dress and neatness to avoid possible negative customer reactions. ' Often Some-t i m e ^ ) 288, advertising, display material, l o c a l campaigns, open house, Job Part:- Marketing Description of R e s p o n s i b i l i t i e s : marketing of bank services Rating Procedure: Please, THINK CAREFULLY before you answer the question: How often does this job incumbent e x h i b i t the following behaviors? C i r c l e the appropriate anchor, e.g. , , , © Always Plans each campaign ahead and sets the goals f o r h i s branch. Ensures p u b l i c i s aware of new and b e n e f i c i a l timely services through in-branch a d v e r t i s i n g . Is conscious of a l l opportunities for new business and makes management aware of any new customer with p o t e n t i a l for more business. Is unable to answer customer i n q u i r i e s about bank services because of lack of product knowledge, causing d i s s a t i s f a c t i o n and a bad bank image. Does not use materials provided to advertize the Bank's s e r v i c e . Sets up a s p e c i a l sales area, e a s i l y a c cessible to the customer, to inform about a new bank service and to market the service. U t i l i z e s marketing posters to the best advantage i n the branch and r e g u l a r l y moves or replaces them. Does not take advantage of contact with non-customers to s o l i c i t new accounts. lias new a d v e r t i s i n g material for each item, makes s t a f f aware of i t . Does nothing ot market bank servic e s . " I f the customer wants i t he w i l l ask for i t " . When a new campaign i s announced t h i s accountant w i l l c a l l a s t a f f meeting, asking f o r suggestions how the branch can p a r t i c i p a t e e f f e c t i v e l y . Makes s t a f f aware of f u l l range of bank s e r v i c e s . Ensures that counter o f f i c e r s are c r o s s - s e l l i n g the bank's servic e s . Follows through on o u t l i n e d promotion campaign. T r i e s to improve on areas where the branch i s weak, e.g., i f there are a number of unrented safety deposit boxes, he w i l l set up a campaign to s e l l them. Has a sound knowledge of products and s e r v i c e s , enabling him to explain i a l l advantages and b e n e f i t s to a customer. Ensures that d i r t y or torn a d v e r t i s i n g posters are replaced promptly. F a i l s to inform s t a f f about new campaigns. Is not aggressive i n s e l l i n g new bank s e r v i c e s . Offers incentives to s t a f f to promote bank services during campaigns. Often ( Some-times 289 Job Part: . Personnel Description of R e s p o n s i b i l i t e s : salary administration, performance app r a i s a l , h i r i n g and f i r i n g , d i s c i p l i n a r y actions, motivation, encouragement, moral support, personal problems, vacation and holiday scheduling Rating Procedure: Please, THINK CAREFULLY before you answer the question: How often does this job incumbent exhibit the following behaviors? C i r c l e the appropriate anchor, ' • t ' - f ^ 1 1 1 Always Shows great confidence i n subordinates and openly displays t h i s with the r e s u l t that they develop to meet h i s expectations. Is openly c r i t i c a l of subordinates i n front of both other s t a f f members and customers. < 1 Confers with members of the s t a f f regarding decisions which w i l l a f f e c t them. ' Ensures that a l l s t a f f members are aware of the duties they have to perform. 1 Encourages more e f f i c i e n t work by t r u s t i n g an employee; gives more r e s p o n s i b i l i t y to an i n d i v i d u a l as the q u a l i t y of work improves. 1 Does not support decisions made by a subordinate (makes exceptions to the r u l e s ) . 1 Praises p u b l i c a l l y f o r tasks completed w e l l and con s t r u c t i v e l y c r i t i c i z e s i n p r i v a t e those i n d i v i d u a l s who have produced l e s s than adequate r e s u l t s . ' Gives everybody the same evaluation on the Performance Appraisal Reports even i f they are not doing a good job. 1 Creates the impression that an "appointment" i s necessary f o r a subordinate who wants to discuss a personal problem. ' Continually discusses performance of others behind t h e i r backs creating a f e e l i n g of mistrust among the s t a f f . I — Explains b e n e f i t s package and presents to the new entrant brochures to read a f t e r the plans are discussed. I — Relies on in-branch promotion as much as possible and makes s t a f f f e e l that getting ahead i s not an unreal objectives. I — T e l l s employees to mind t h e i r own business when they come out with new ideas. I — Sets r e a l i s t i c goals f o r every employee. i Keeps s t a f f involved i n changes through regular meetings, encourages feedback- p o s i t i v e or negative - or p o l i c y changes. I — Frequently changes h i s mind and issues contradictory i n s t r u c t i o n s , having an adverse e f f e c t on morale of s t a f f and on work flow. I — Does not accept r e s p o n s i b i l i t y for errors but passes blame on to subordinates. Makes promises to i n d i v i d u a l s regarding promotion and s a l a r y which cannot be followed through. I — Creates a better working environment by maintaining close contacts with a l l s t a f f members ( i . e . ; holds frequent discussions both casual and I — job-oriented.) Discusses p r i v a t e personal problems of subordinates with others. , ; -4-Often Some-times Never 290 Job Part: Training Descriptive of R e s p o n s i b i l i t i e s : any job rela t e d t r a i n i n g and development a c t i v i t y , explanations, support i n problem s o l v i n g , advise, informing, up-dating, course scheduling, t r a i n i n g follow-up Rating Procedure: Please, THINK CAREFULLY before you answer the question: How often does this job incumbent e x h i b i t the following behaviors? C i r c l e the appropriate anchor, e.g. , Q j , Always Does not teach employee a l l aspects of job, expects an employee to learn i t by himself, e.g., says: "Don't know, look i t up i n the book." Ensures that new methods are put i n t o p r a c t i c e through proper t r a i n i n g . Uses trainees as "Joe-boys", thus wasting valuable t r a i n i n g time. Ensures that i f a trainee i s tra i n e d by another s t a f f member that the l a t t e r i s competent enough to do the t r a i n i n g . Obtains t r a i n i n g aids from the T r a i n i n g D i v i s i o n to a s s i s t with in-branch t r a i n i n g . Has patience with slow learners, even i f some questions are asked s e v e r a l times. Explains why things are done, not only how. Encourages employees t o d i r e c t work-related questions to him i f they have any doubts. Conducts follow-up and evaluation of any t r a i n i n g program. W i l l take the time to f a m i l i a r i z e himself with the t r a i n i n g material provided before using i t . Ensures that every t r a i n i n g l i t e r a t u r e applying to employees' job i s a v a i l a b l e . Organizes job r o t a t i o n with the e f f e c t that each c l e r k know at l e a s t two jobs - i d e a l f o r vacation periods and absences. Is i n c o n s i s t e n t i n ex p l a i n i n g concepts and processes, thus leading t o confusion. Encourages s t a f f to p a r t i c i p a t e i n courses the t r a i n i n g center has to o f f e r . Provides on-the-job t r a i n i n g to subordinates when necessary. Makes sure that h i s or her subordinates have the proper t r a i n i n g f o r the job they are doing i n order t o achieve high e f f i c i e n c y . Takes time to explain new or amended procedures. Establishes an adequate t r a i n i n g l i b r a r y . Makes a habit of s t e a d i l y reviewing the progress of subordinates, making them aware of strengths and weaknesses with suggestions f o r improvements where needed. Moves trainee on to next phase of job before assessing performance on previous post. Gives close a t t e n t i o n to inexperienced or new s t a f f members when they are having t h e i r i n i t i a l contacts with branch customers. Some-times 3ften Never ••••291 Appendix N INDEX OF OVERALL PERFORMANCE (OP) The o v e r a l l performance evaluation of a ratee consists of a global rating on a five - p o i n t scale with the following anchors: (1) not acceptable; (2) improve-ment required; (3) good; (4) very good; (5) excellent. The o v e r a l l performance ra t i n g i s done a f t e r the ratee has been assessed on 17 items: acceptance of r e s p o n s i b i l i t y , accuracy, attitude, capacity, customer r e l a t i o n s , i n i t i a t i v e , judgement, knowledge of duties, mental alertness, o r a l expression, organization, personal neatness, s t a f f r e l a t i o n s , s t a f f supervision, s t a f f t r a i n i n g , success i n s o l i c i t i n g business, written expression. INDEX OF PROMOTABILITY (PR) The p r o m o t a b i l i t y o f a r a t e e i s assessed on a f i v e - p o i n t s c a l e , w i t h the f o l l o w i n g s c a l e anchors: (5) As soon as there i s a s u i t a b l e opening; (4) Withi: a year o r two; (3) L i k e l y to q u a l i f y f o r promotion, but time i s u n c e r t a i n ; (2) Only l a t e r a l t r a n s f e r recommended; (1) Not l i k e l y to q u a l i f y f o r f u r t h e r promotion. Together w i t h the above-described a s s e s s -ment, the r a t e r has to submit a d e t a i l e d w r i t t e n e x p l a n a t i o n o f the recommendation which has to be d i s c u s s e d w i t h the r a t e r ' s d i r e c t s u p e r v i s o r . Because o f t h i s p r o v i s i o n , i t i s assumed t h a t the PR i s a r e l a t i v e l y o b j e c t i v e measure of a r a t e e ' s performance. INDEX OF ACTUAL PROMOTION (AP) The bank i n question has nine d i f f e r e n t l e v e l s of assistant branch managers, depending on s i z e , geographical location (national, p r o v i n c i a l , and c i t y d i s t r i c t ) , and type of c l i e n t e l e . Assistant branch managers are promoted into t h e i r p o s i t i o n t y p i c a l l y from such positions as personnel o f f i c e r , o f f i c e supervisor, account manager, and senior clerk, according to a b i l i t y and performance. The l e v e l into which a candidate i s promoted depends on his/her degree of a b i l i t y and l e v e l of job performance. Although most candidates are promoted one l e v e l at a time, quite a few jump two or three l e v e l s , i n exceptional cases even four. The AP index, as applied i n t h i s study, has been developed i n co-operation with the personnel manager of the respective bank, the t r a i n i n g o f f i c e r , and f i v e branch managers. I t indicates whether an assistant branch manager (the ratee) had been promoted the l a s t time either one l e v e l (AP = 1)., two lev e l s (AP = 2), three levels (AP = 3 ) , or four (AP = 4). I t should be noted that the AP i s not a precise indicator of an employee's performance as indicated by the number of le v e l s of a promotion. I t may occur that a p o s i t i o n becomes vacant for which there are no e l i g i b l e candidates, i . e . candidates available may have performance ratings which would make them e l i g i b l e only for a lower l e v e l p o s i t i o n . In such a case the position may be f i l l e d with a candidate who normally would not q u a l i f y . This may work the other way too, i . e . a position may be vacant for which candidates are overqualified. I t happens frequently that i n such cases candidates accept the lower l e v e l promotion, rather than wait for a suitable p o s i t i o n . However, such cases are exceptions. ; 296 Appendix 0 You are asked to rank the Graphic Rating Scale (GRS), the Behaviorally Anchored Rating Scale (BARS), and the Behavior Description Index (BDI) according to your preference for each instrument when applying i t for d i f f e r e n t purposes, e.g. counselling, promotion, etc. (1 = f i r s t choice, 2 = second, 3 = third) 1. Please rank each instrument according to your preference for i t s use as a feedback and counselling t o o l . BARS BDI GRS • • ..• BDI BARS GRS Please rank each instrument accordxng to your i 1 | 1 i i preference for i t s use as a t o o l to determine j | | | | ( wage and salary increases. GRS BDI BARS Please rank each instrument according to your i 1 i i i j preference for i t s use as a tool,.to determine | | I I |_ I promotion. BDI BARS GRS Please rank each instrument according to your i j i 1 i 1 preference for i t s use as a t o o l to determine | | I | | | t r a i n i n g needs of employees. 5. Please rank each instrument according to your preference f or i t s use as a t o o l to develop s p e c i f i c t r a i n i n g programs and to formulate precise t r a i n i n g objectives. GRS BARS BDI. • • BARS GRS BDI Please rank each instrument according to the i 1 i 1 i i ease with which you a r r i v e at a r a t i n g I I I |. | | decision when using i t . PUBLICATIONS "Thoughts on T r a i n i ng E v a l u a t i o n " , Canadian T r a i n i ng Methods, V o l . 7, #1, June 1975, pp. 14-15. "Issues in T r a i n i ng E va l u a t i o n , I: The Methodology", Canadian T r a i n i ng Methods, V o l . 7, #3, August 1975, pp. 22-25. "Issues T r a i n i ng E va l u a t i o n , I I : The C r i t e r i o n " , Canadian T r a i n i ng Methods, V o l . 7, #4, October 1975, pp. 14-15. "Personnel Problems i n I n t e rna t i ona l Companies and J o i n t Ventures in Japan: A Comparis ion Between U.S. and European Managements", (w i th R ichard B. Peterson) Journa l o f I n t e rna t i ona l Business S t ud i e s , Spr ing 1977, 8. 1. "Count ing Ac t i ons Louder Than Words: Behav iora l Sca les of Job Performance" i n L o i s - e l l i n Datta ( e d . ) : Improving Eva l ua t i on s ; Sage P u b l i c a t i o n s , I n c . , Bever ly H i l l s , C a l i f o r n i a , i n p ress . "Personnel Problems i n I n t e r na t i ona l Companies and J o i n t Ventures in Japan" , (with R ichard B. Peterson) Proceedings o f the Academy of Management Meeting i n New Or leans , 1975. " A t t i t u d i n a l D i f f e rences Between Anglophone and Franco-phone Canadian Managers Toward C o l l e c t i v e Rep resen ta t i on " , (with Vance M i t c h e l l ) Proceedings o f the Canadian A s s o c i a t i o n of A dm i n i s t r a t i v e Sc iences Meeting in Quebec C i t y , 1976. " G e n e r a l i z a b i l i t y of the JDI Sca les to Canadian Respondents", (wi th Vance M i t c h e l l ) Proceedings of the Canadian A s s o c i a t i o n f o r Adm i n i s t r a t i v e Sc iences Meeting i n F r e d e r i c t o n , June 1977. " A t t i t u de s of American and European Expa t r i a t e Managers Toward. Japanese", (with R ichard B. Peterson) Proceedings of the Academy of Management Meeting i n Or lando, F l o r i d a , August 1977. "Some Evidence Aga ins t the Convergence Hypothes i s " , (with R ichard B. Peterson) Proceedings o f the Academy o f I n t e r na t i o na l Bus iness Meeting i n Or lando, F l o r i d a , August 1977.