Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Monitoring and maintaining competence of health professionals Cutshall, Patricia 1978

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-UBC_1978_A8 C98.pdf [ 11.15MB ]
Metadata
JSON: 831-1.0055951.json
JSON-LD: 831-1.0055951-ld.json
RDF/XML (Pretty): 831-1.0055951-rdf.xml
RDF/JSON: 831-1.0055951-rdf.json
Turtle: 831-1.0055951-turtle.txt
N-Triples: 831-1.0055951-rdf-ntriples.txt
Original Record: 831-1.0055951-source.json
Full Text
831-1.0055951-fulltext.txt
Citation
831-1.0055951.ris

Full Text

MONITORING AND MAINTAINING COMPETENCE OF HEALTH PROFESSIONALS by PATRICIA CUTSHALL B . S c . N . , Syracuse U n i v e r s i t y , 1961 A t h e s i s submitted i n p a r t i a l f u l f i l l m e n t of the requi rements fo r the degree of MASTER OF ARTS i n THE FACULTY OF GRADUATE STUDIES Department of Adu l t Educat ion We accept t h i s as conforming to the r e q u i r e d standard THE UNIVERSITY OF BRITISH COLUMBIA June 1978 © P a t r i c i a C u t s h a l l , 1978 In p r e s e n t i n g t h i s t h e s i s i n p a r t i a l f u l f i l m e n t of the requirements f o r an advanced degree a t the U n i v e r s i t y of B r i t i s h Co lumbia , I agree tha t the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r re fe rence and s t u d y . I f u r t h e r agree that p e r m i s s i o n f o r e x t e n s i v e copying of t h i s t h e s i s f o r s c h o l a r l y purposes may be granted by the Head of my Department or by h i s r e p r e s e n t a t i v e s . I t i s understood that copying or p u b l i c a t i o n of t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be a l lowed wi thout my w r i t t e n p e r m i s s i o n . P a t r i c i a C u t s h a l l Department of Adul t Educat ion The U n i v e r s i t y of B r i t i s h Columbia 2075 Wesbrook P l a c e Vancouver, Canada V6T 1W5 Date 30 June 1978 i i A b s t r a c t H e a l t h sc iences l i t e r a t u r e from 1970 - 1977 which p e r t a i n s to m o n i t o r i n g and m a i n t a i n i n g competence of h e a l t h p r o f e s s i o n a l s i s a b s t -r a c t e d and r e p o r t e d . Recogn iz ing that competence i s d e f i n e d by the c r i t e r i a and standards used to e v a l u a t e i t s p resence , t h i s t h e s i s uses Donabedian's e v a l u a t i o n model to o rgan ize the bu lk of the l i t e r a t u r e rev iewed . Whi le the model a p p l i e s to competence of the h e a l t h care system, t h i s paper l i m i t s d i s c u s s i o n to competence of h e a l t h p r o f e s -s i o n a l s . I t c o n s i d e r s l i t e r a t u r e concerned w i t h d e f i n i t i o n , e v a l u a t i o n , and r e s t o r a t i o n / m a i n t a i n c e a c t i o n r e l a t e d to knowledge and s k i l l s p o s s e s -sed by p r a c t i o n e r s ( s t r u c t u r e ) , c l i n i c a l performance ( p r o c e s s ) , and r e s u l t s of care (outcome). A d d i t i o n a l l y i t p rov ides i n f o r m a t i o n about the h i s t o r i c a l development of and cu r ren t t rends i n c r e d e n t i a l i n g mechan-isms intended to c e r t i f y competence, exp lo res i n some d e t a i l the i s s u e of mandatory c o n t i n u i n g e d u c a t i o n , and d i s c u s s e s contemporary s o c i a l and p o l i t i c a l phenomena which i n f l u e n c e and are i n f l u e n c e d by competence m o n i t o r i n g a c t i v i t i e s . The c o n c l u d i n g chapter o f f e r s o b s e r v a t i o n s on the c u r r e n t s t a t e - o f - t h e a r t w i t h respect to e v a l u a t i n g and m a i n t a i n i n g com-petence and makes recommendations f o r f u r t h e r r e s e a r c h , development and implementat ion of competence m o n i t o r i n g a c t i v i t i e s c o n s i s t e n t w i t h the North American s o c i a l c o n t e x t . F i n a l l y , s e v e r a l i m p l i c a t i o n s f o r c o n t i n -u ing e d u c a t o r s ' r o l e s and r e s p o n s i b i l i t i e s v i s a v i s m o n i t o r i n g and m a i n -t a i n i n g h e a l t h p r o f e s s i o n a l s ' competence are d i s c u s s e d . i i i Table of Contents Page I I n t r o d u c t i o n 1 Purpose 2 Procedure 2 D e f i n i t i o n of terms 3 O r g a n i z a t i o n of the rev iew 4 "Jl A s s e s s i n g P o t e n t i a l f o r C l i n i c a l Performance: S t r u c t u r e 7 Se l f -Assessment Procedures 7 Randomly A v a i l a b l e Test Instruments 8 Se l f -Assessment Procedures f o r Se lec t Groups 9 Formal T e s t i n g of P r a c t i t i o n e r s 12 Tests f o r F u l l Range of P r a c t i c e 13 T e s t i n g i n S p e c i a l t y Areas 16 S p e c i a l Purpose Assessment Procedures 17 N o n - t r a d i t i o n a l T e s t i n g Procedures 17 I n d i v i d u a l i z e d P h y s i c i a n P r o f i l e .18 Tests of C l i n i c a l D e c i s i o n Making 18 Tests of I n t e r a c t i v e S k i l l 21 Summary and Comment 22 H I Assess ing C l i n i c a l Performance: Process 26 Systemat ic Assessment of T o t a l Performance 26 Genera l D i s c u s s i o n s of Systematic Performance E v a l u a t i o n . . . 27 i v Page Performance E v a l u a t i o n Us ing I m p l i c i t Standards 28 Performance E v a l u a t i o n With E x p l i c i t but Imprecise Standards 33 Performance E v a l u a t i o n Using E x p l i c i t Standards 37 Systemat ic Assessment of S i t u a t i o n - S p e c i f i c Performance 40 S i t u a t i o n - S p e c i f i c Process E v a l u a t i o n : I n d i v i d u a l P r a c t i c e . 42 S i t u a t i o n - S p e c i f i c Process E v a l u a t i o n : Agency S e t t i n g s 45 S i t u a t i o n - S p e c i f i c Process E v a l u a t i o n : Reg iona l Systems . . . . 50 Unplanned E v a l u a t i o n of Performance 59 The P lace of Work 60 The P r o f e s s i o n 61 The L i c e n s i n g A u t h o r i t y 63 Summary and Comment 68 IV Assess ing Resu l t s of C l i n i c a l Performance: Outcome 72 Genera l Measures of H e a l t h Status 72 S i t u a t i o n - S p e c i f i c Outcomes E v a l u a t i o n 74 Summary and Comment 80 V Assess ing Combinations of S t r u c t u r e , Process and Outcome 82 Concurrent E v a l u a t i o n of S t ruc tu re -Process -Outcome Combinations 82 Concurrent E v a l u a t i o n of S t r u c t u r e , P r o c e s s , Outcome 83 S t r u c t u r e - P r o c e s s Combinations 85 Agency Based Process-Outcome E v a l u a t i o n 88 V Page R e g i o n a l l y Based Process-Outcome E v a l u a t i o n 94 Outcome Assessment w i t h S e l e c t i v e E v a l u a t i o n of Process 97 Random E v a l u a t i o n of Process and Outcome of Care 102 Summary and Comment 105 VI S e l e c t i o n of E v a l u a t i o n and A c t i o n S t r a t e g i e s 108 Genera l C o n s i d e r a t i o n s i n Choosing E v a l u a t i v e S t r a t e g i e s . . . . 108 The E v a l u a t i o n - A c t i o n Process as A p p l i e d i n H e a l t h Care . . . 109 P o s s i b l e Frames of Reference f o r E v a l u a t i o n I l l The Ro le of Values i n E v a l u a t i o n 113 Dec id ing What W i l l Be Evaluated 114 The S t ruc tu re -Process -Outcome Debate 114 I m p l i c i t Versus E x p l i c i t Standards 118 D e r i v a t i o n of C r i t e r i a and Standards 121 Data Management 124 P a t i e n t Care Records as a Data Source 124 Systems F a c i l i t a t i n g R e t r i e v a l of Data 127 E v a l u a t i o n Based I n t e r v e n t i o n s 129 Summary and Comment 137 VI I The North American S o c i a l Context 141 The S o c i a l Ecology of M a i n t a i n i n g Cpmpetence 142 Needs and R ights of P r o f e s s i o n a l s 142 Needs and R ights of Consumers 145 v i Page C o l l e c t i v e A c t i o n 147 The R i s i n g Costs of H e a l t h Care 149 M a l p r a c t i c e 151 Hea l th Manpower 153 Regu la to ry Mechanisms and Trends 154 H i s t o r i c a l Development of Regulatory Mechanisms 155 L i c e n s u r e and C e r t i f i c a t i o n Today 159 R a t i o n a l i z a t i o n of L i c e n s i n g Procedures 161 Broader Base of L i c e n s i n g C o n t r o l 163 M u l t i p l e Assessment Procedures 164 R e - c r e d e n t i a l i n g 165 Mandatory Cont inu ing Educat ion 168 H i s t o r i c a l Background 168 Pros and Cons of U n i v e r s a l Mandatory Cont inu ing Educat ion . 170 S e l e c t i v e Mandatory Cont inu ing Educat ion 175 A c c r e d i t i o n of Cont inu ing Educat ion Programs 177 Summary and Comment 180 V I I I Observat ions and Recommendations 182 L i t e r a t u r e C i t e d 191 v i i L i s t of Tables Page I Types of D i s c i p l i n a r y A c t i o n Taken w i t h American P h y s i c i a n s 1963 - 1967 65 I I Causes f o r D i s c i p l i n a r y A c t i o n Taken w i t h American P h y s i c i a n s 1963 - 1967 65 I I I Advantages and Disadvantages of Measurement of S t r u c t u r e 115 IV Advantages and Disadvantages of Measurement of Process 115 V Advantages and Disadvantages of Measurement of Outcome 116 VI Ev idence of Des i red Change i n Reported E v a l u a t i o n - A c t i o n P r o j e c t s 131 VI I Arguments i n Favor of Mandatory Cont inu ing Educat ion 171 V I I I Arguments i n Oppos i t ion to Mandatory Cont inu ing Educat ion 172 L i s t of F i g u r e s 1 I n t e r a c t i o n Process A n a l y s i s (Robert F. Bales) 23 2 Change Agent Behaviors Which C o n t r i b u t e to E v a l u a t i o n - A c t i o n Programs 188 - 1 -Chapter 1 INTRODUCTION C o s t , a v a i l a b i l i t y , and q u a l i t y of h e a l t h care are c u r r e n t l y among the most c o n t e n t i o u s and prob lemat ic i s s u e s of the s e v e n t i e s i n Nor th American s o c i e t y . " S o l u t i o n s " such as consumer c o n t r o l of the d e -l i v e r y of those s e r v i c e s and c o n t i n u i n g educat ion i n h e a l t h care f o r both the consumers and p r o v i d e r s are s u b j e c t s of c o n t i n u i n g and v igo rous d e -b a t e . Both problems and s o l u t i o n s have i n c r e a s i n g l y been t r a n s l a t e d i n t o ve ry r e a l e v e n t s . Most n o t a b l e of these are s e v e r a l recent l e g i s l a t i v e a c t i v i t i e s o f f a r reach ing s i g n i f i c a n c e : sweeping changes i n the r e g u l a t i o n of h e a l t h p r o f e s s i o n a l s as has occur red w i t h the P r o f e s s i o n s Code i n Quebec (400) and w i t h the H e a l t h D i s c i p l i n e s Act i n Ontar io (229) ; the e s t a -b l ishment of P r o f e s s i o n a l Standards Review Organ i za t ions i n the Un i ted S t a t e s as an amendment to the S o c i a l S e c u r i t y Act ( 5 1 5 , 5 1 6 , 5 1 8 ) ; the enactment i n s e v e r a l s t a t e s of "mandatory c o n t i n u i n g educat ion" l a w s ; and , p a r a d o x i c a l l y , a slow down i n l e g i s l a t i o n e s t a b l i s h i n g new c a t e -g o r i e s of l i c e n s e d h e a l t h p e r s o n n e l , t h i s l a s t recommended by the U .S . Department of H e a l t h , Educat ion and Welfare ( 5 1 3 ) . W i t h i n the p r i v a t e s e c t o r , popular l i t e r a t u r e r e f l e c t s a growing c o n v i c t i o n tha t r e s p o n s i -b i l i t y f o r h e a l t h and h e a l t h care cannot be l e f t s o l e l y to h e a l t h p r o f e s -s i o n a l s ( 8 2 , 2 5 0 ) . Reac t ions of h e a l t h care p r o v i d e r s to a l l t h i s a c t i v i t y are v i g o r o u s . The h e a l t h , s c i e n c e s l i t e r a t u r e of the s e v e n t i e s i s no tab le f o r - 2 -the volume of a t t e n t i o n g i ven problems connected w i t h ensur ing that h e a l t h care i s acceptab le i n terms of c o s t , a v a i l a b i l i t y and q u a l i t y . With respect to q u a l i t y , the competence of h e a l t h care p r a c t i t i o n e r s i s one of the more s i g n i f i c a n t determinants and i s the sub jec t of t h i s paper . Purpose The purpose of t h i s paper i s to i d e n t i f y , through a l i t e r a t u r e r e v i e w , recent developments i n the h e a l t h care f i e l d which are r e l a t e d to s t r a t e g i e s f o r a s s e s s i n g and m a i n t a i n i n g competence of h e a l t h p r o f e s -s i o n a l s . Procedure L i t e r a t u r e from the h e a l t h f i e l d was surveyed w i t h s p e c i a l a t t e n t i o n to i tems from m e d i c i n e , d e n t i s t r y , n u r s i n g and pharmacy. The scope of the search was l i m i t e d to l i t e r a t u r e of American and Canadian o r i g i n pub l i shed from 1970 to June 1977 and to concern w i t h competence of the p r a c t i c i n g p r o f e s s i o n a l (as opposed to the p r o f e s s i o n a l a t entrance to p r a c t i c e ) . A Medl ine search and rev iew of p e r i o d i c a l s f o r the 1970 -73 pe r iod on the same sub jec t by Neylan (362) was u t i l i z e d as a base and was extended to i n c l u d e r e s e a r c h a b s t r a c t s and government documents. P e r i o d i c a l l i t e r a t u r e , r e s e a r c h a b s t r a c t s and government documents from the 1974 - 77 p e r i o d were rev iewed . Ch ief sources f o r data r e t r i e v a l - 3 -were D i s s e r t a t i o n A b s t r a c t s , ERIC, Medl ine and PSRO In fo rmat ion C l e a r i n g House, operated by C a p i t a l Systems Group, I n c . under a c o n t r a c t w i t h the Un i ted S t a t e s Department of H e a l t h , Educat ion and W e l f a r e . Th is survey should be cons idered a r e p r e s e n t a t i v e sample of the l i t e r a t u r e a v a i l a b l e , not an exhaust i ve s e a r c h . D e f i n i t i o n of Terms A c c r e d i t a t i o n - the process by which an agency or o r g a n i z a t i o n eva lua tes and recogn i zes a program of study or an i n s t i t u a t i o n as meeting c e r t a i n predetermined q u a l i f i c a t i o n s or s t a n d -a r d s . A u d i t - a sys temat ic approach to e v a l u a t i o n of care which r e s u l t s i n documentation by h e a l t h p r o f e s s i o n a l s of how f a r t h e i r ca re conforms to t h e i r own standards of adequacy or ex -c e l l e n c e . C e r t i f i c a t i o n - the process by which a non-governmental agency or a s s o c -i a t i o n g rants r e c o g n i t i o n to an i n d i v i d u a l who has met c e r t a i n pre -determined q u a l i f i c a t i o n s s p e c i f i e d by tha t agency or a s s o c i a t i o n . Competence - the q u a l i t y of being f u n c t i o n a l l y adequate i n performing the tasks and assuming the r o l e s of a s p e c i f i e d p o s i t i o n w i t h the r e q u i s i t e knowledge, a b i l i t y , c a p a b i l i t y , s k i l l , judgement, a t t i t u d e s and v a l u e s ( 1 7 7 ) . Cont inu ing Educat ion - l e a r n i n g e x p e r i e n c e s , formal or i n f o r m a l , designed to en la rge and/or update the knowledge and s k i l l s of p r a c -t i c i n g h e a l t h p r o f e s s i o n a l s . C r e d e n t i a l i n g - the r e c o g n i t i o n of p r o f e s s i o n a l or t e c h n i c a l competence. The c r e d e n t i a l i n g process may i n c l u d e r e g i s t r a t i o n , c e r t -i f i c a t i o n , l i c e n s u r e , p r o f e s s i o n a l a s s o c i a t i o n member-s h i p , or the award of a degree or o ther academic r e c o g -n i t i o n i n the f i e l d . C r i t e r i o n - a v a r i a b l e expressed i n such a way t h a t i t s presence or absence as evidence of adequacy can be a s c e r t a i n e d , eg d e s c r i b e s p e n i c i l l i n , b lood pressure i s 150/90. When used as a s i n g l e measure, a c r i t e r i o n may represent a s t a n d a r d . - 4 -Licensure - process by which an agency of government, or i t s dele-gate, grants permission to persons meeting predetermined q u a l i f i c a t i o n s to engage i n a given occupation and/or to use a p a r t i c u l a r t i t l e ; or, grants permission to i n s t i -tutions to perform s p e c i f i e d functions within t h e i r j u r i s d i c t i o n . Peer Review - an a l l - i n c l u s i v e term for health care review e f f o r t s involving the evaluation by p r a c t i s i n g health profession-a l s of the q u a l i t y and e f f i c i e n c y of services performed by other health professionals of the same d i s c i p l i n e . Registration - the process by which an agency or organization maintains a public l i s t of i n d i v i d u a l s who meet pre-determined c r i t e r i a s p e c i f i e d by the agency which maintains the r e g i s t e r . Where the r i g h t to r e g i s t e r and be reg i s t e r e d i s protected by law, r e g i s t r a t i o n may be considered l i c e n s u r e . Standard - a c r i t e r i o n , or degree of measurement of a set of c r i -t e r i a , which may be considered adequate for a stated purpose such as an award, d i s c i p l i n e , further education. A standard may be expressed as a single c r i t e r i o n or may be expressed as a point on a percentage or other scale of measurement with respect to the same c r i t e r i o n i n a sam-ple of cases or with respect to a set of related c r i t e r i a i n one or a sample of cases. Variable - an a t t r i b u t e or type of behavior, eg physician's pre-sc r i b i n g pattern or patient's blood pressure, which i s accepted as an ind i c a t o r of adequacy and which i s expres-sed or used i n such a way that a measure of the v a r i a b l e which w i l l be accepted as evidence of adequacy i s or must be a d d i t i o n a l l y s p e c i f i e d . Organization of the Review This paper reports relevant l i t e r a t u r e by addressing four major issues related to evaluating and maintaining competence of health profes-s i o n a l s : d e f i n i t i o n , evaluation, action for restoration/maintenance, and s o c i a l context. The f i r s t of these, d e f i n i t i o n , i s fundamental to the other three. T h e o r e t i c a l l y competence connotes adequacy i n a r o l e . R e a l i s t i c a l l y however, the variables and standards used to judge i t s pre-- 5 -sence or absence become i t s o p e r a t i o n a l d e f i n i t i o n ; i . e . competence i s d e f i n e d through the a c t of e v a l u a t i o n . There are many ways that components of competence can be c a t e g o r i z e d fo r e v a l u a t i o n . Seve ra l models fo r e v a l u a t i o n i n h e a l t h care and educat ion have been proposed ( 1 1 3 , 1 2 0 , 1 2 6 , 1 4 4 , 1 4 6 , 2 8 1 , 4 2 5 , 4 7 3 ) . Because of i t s s i m p l i c i t y , r e l e v a n c e , and c o n s i d e r a b l e i n f l u e n c e on h e a l t h c a r e , the model proposed by Donabedian (144,146) i s used as a frame of re fe rence i n subsequent c h a p t e r s . Donabedian suggests tha t a l l elements i n the h e a l t h care system can be c l a s s i f i e d i n one of three major c a t e g o r i e s : s t r u c t u r e ( r e -s o u r c e s ) , process ( a c t i v i t i e s ) , or outcome ( r e s u l t s ) . Each can a l s o be i s o l a t e d , d e f i n e d , and measured a g a i n s t a s t a n d a r d . S t r u c t u r e i n c l u d e s such v a r i a b l e s as f u n d s , a b i l i t i e s of p e r s o n n e l , space; process i n v o l v e s the behavior of p r a c t i t i o n e r s , p a t i e n t s , and a d m i n i s t r a t i v e systems; o u t -come focuses on whether s e r v i c e s were used , the p a t i e n t got b e t t e r , or the community b e n e f i t t e d . When the scope of concern i s narrowed from the h e a l t h care system to the competence of p r a c t i t i o n e r s , the three areas c o n s t r i c t to knowledge and s k i l l s ( s t r u c t u r e ) , c l i n i c a l performance ( p r o -c e s s ) , and a l t e r e d h e a l t h s t a t e of the p a t i e n t (outcome). The l i t e r a t u r e reviewed i s o rgan ized i n e i g h t c h a p t e r s : Chapter 1 - I n t r o d u c t i o n - e s t a b l i s h e s purpose of the r e v i e w , d e f i n e s terms, d e s c r i b e s the procedure used and the o r g a n i z a t i o n of the paper . Chapter 2 - A s s e s s i n g P o t e n t i a l f o r C l i n i c a l Performance: S t r u c t u r e - c o n s i d e r s developments r e l a t e d to judgements about and a c t i o n on the knowledge, s k i l l s or a t t i t u d e s p r a c t i t i o n e r s b r i n g to the j o b . - 6 -Chapter 3 - Assessing C l i n i c a l Performance: Process - surveys developments i n the d e f i n i t i o n and evaluation of per-formance during the giving of care and i n a c t i v i t i e s under-taken to correct sub-standard performance or maintain and improve s a t i s f a c t o r y performance. Chapter 4 - Assessing Results of C l i n i c a l Performance: Outcome - Considers e f f o r t s being made to judge and improve competence of health professionals by focussing on the r e s u l t s of care. Chapter 5 - Assessing Combinations of Structure, Process and Outcome - describes combined approaches to evaluation and follow up action which have developed l a r g e l y i n response to the d i s -advantages inherent i n focussing on structure, process, or outcome i n i s o l a t i o n . Chapter 6 - Selection of Evaluation And Action Strategies - i d e n t i f i e s a number of areas where the l i t e r a t u r e i s of suf-f i c i e n t volume and v a r i e t y to permit tentative generaliza-tions h e l p f u l to decision making i n planning for evaluation of competence and follow-up a c t i o n . Chapter 7 - The North American Social Context - discusses l i t e r a t u r e concerned with psychological considera-tions related to evaluation and with s o c i a l and p o l i t i c a l developments relevant to evaluating and maintaining compe-tence of health professionals. Chapter 8 - Observations and Recommendations - Offers some general observations and recommendations to con-tinuing educators. The v a r i e t y and quantity of relevant l i t e r a t u r e has resulted i n a complex structure for most of the chapters. It w i l l be u s e f u l , before reading each chapter, to study the chapter's t o p i c a l outline i n the table of contents. In this way the organizational plan for each chapter, i t s contents, and t h e i r r e l a t i o n s h i p w i l l be more apparent. - 7 -Chapter I I ASSESSING POTENTIAL FOR CLINICAL PERFORMANCE: STRUCTURE Th is chapter w i l l examine recent developments i n a s s e s s i n g the knowledge, s k i l l s and a t t i t u d e s which p r a c t i t i o n e r s b r i n g to a c l i n i c a l s i t u a t i o n . Of concern are e v a l u a t i o n approaches which attempt to d e f i n e and t e s t d e s i r e d p r o f e s s i o n a l b e h a v i o r . The assumption which u n d e r l i e s a " t e s t i n g " approach to assessment of competence i s tha t those p r o f e s s i o n -a l s who can perform i n a d e s i r e d manner under t e s t c o n d i t i o n s are l i k e l y to do so i n p r a c t i c e . These approaches w i l l be d i s c u s s e d w i t h i n three broad c a t e g o r i e s : s e l f - a s s e s s m e n t procedures , formal t e s t i n g of p r a c t i -t i o n e r s , and n o n - t r a d i t i o n a l t e s t i n g procedures . Se l f -Assessment Procedures Use of s e l f t e s t i n g as a v e h i c l e f o r e s t i m a t i n g present compe-tence and f o r m o t i v a t i n g and g u i d i n g cont inued development of competence has probably r e c e i v e d most a t t e n t i o n from the med ica l p r o f e s s i o n . Caplan (86) enumerates s e v e r a l advantages of s e l f - a s s e s s m e n t : the p h y s i c i a n has evidence of h i s knowledge d e f i c i t s and t h e r e f o r e i s mot ivated to l e a r n ; r e l a t e d l e a r n i n g i s s e l f - p a c e d , p r i v a t e and n o n - t h r e a t e n i n g . Paper and p e n c i l t e s t i n g can r e a d i l y prov ide i n f o r m a t i o n about r e c a l l of f a c t u a l i n f o r m a t i o n and c l i n i c a l problem s o l v i n g , thus the fundamental bases of p r o f e s s i o n a l p r a c t i c e are t e s t e d . F i n a l l y , the cho ice of a c o r r e c t i v e -8-l e a r n i n g mode i s open and can accommodate to l e a r n e r p re ferences and a v a i l a b l e r e s o u r c e s . Randomly A v a i l a b l e Test Instruments Probably the l e a s t i n d i v i d u a l i z e d assessment t o o l s a v a i l a b l e to p r a c t i t i o n e r s are " t e s t y o u r s e l f " f e a t u r e s which appear i n p r o f e s s i o n a l j o u r n a l s . The " S e l f - A s s e s s m e n t M i n i Program" which appears i n the I n t e r -n a t i o n a l J o u r n a l of Dermatology (375) i s a good example of a r e g u l a r l y appear ing j o u r n a l s e r i e s which b r i e f l y presents a c l i n i c a l problem, asks r e l e v a n t q u e s t i o n s , d i s c u s s e s answers, and l i s t s r e f e r e n c e s f o r f u r t h e r s tudy . "Nurs ing D e c i s i o n s : Exper iences i n C l i n i c a l Problem S o l v i n g " (366) p rov ides an example of t h i s approach found i n the n u r s i n g l i t e r a -t u r e . "Thyro id D i s o r d e r s " (213) i s a v a r i a t i o n on t h i s approach; t h i s i s a s e l f - s t u d y program, complete w i t h o b j e c t i v e s , which begins w i t h p r o v i -s i o n of c l i n i c a l i n f o r m a t i o n and then prov ides a t e s t i n g and s c o r i n g s e r v i c e . P a r t i c i p a n t s who complete the t e s t s u c c e s s f u l l y are awarded c o n t i n u i n g educat ion c r e d i t under the American Nurses A s s o c i a t i o n ' s c o n -t i n u i n g educat ion a c c r e d i t a t i o n mechanism. In a d d i t i o n to m a t e r i a l a v a i l a b l e to p r a c t i t i o n e r s i n j o u r n a l s , a t l e a s t one p u b l i s h i n g house (464) has pub l i shed an e n t i r e s e l f - a s s e s s -ment s e r i e s i n which each volume focusses on a p a r t i c u l a r c l i n i c a l s p e c i -a l t y a r e a . Whi le the m a j o r i t y of these are f o r p h y s i c i a n s or n u r s e s , the s e r i e s i n c l u d e s one volume on c l i n i c a l pharmacy and a few on the v a r i o u s paramedica l t e c h n o l o g i e s . In a d d i t i o n to t e s t quest ions and answers , the reader i s g iven a r a t i o n a l e f o r the answers and d i r e c t i o n f o r f u r t h e r r e a d i n g . The p u b l i s h e r g i v e s no i n f o r m a t i o n on the process used f o r i n i t i a l development of the m a t e r i a l , nor on whether there have been -9-attempts to e s t a b l i s h r e l i a b i l i t y or v a l i d i t y . Performance norms, i f they e x i s t , are not g i v e n , and standards f o r adequate performance are not suggested . A l l of these a i d s to l e a r n i n g assume that the p r a c t i t i o n e r w i l l s e l e c t those r e l e v a n t to h i s p r a c t i c e , that he has access to suggested support m a t e r i a l s i f r e q u i r e d , and that he i s ab le to l e a r n e f f e c t i v e l y to c o r r e c t knowledge d e f i c i t s . There i s no evidence that t h i s does or does not o c c u r . Depending on chance to get the r i g h t s e l f - a s s e s s m e n t t o o l s to the r i g h t p r a c t i t i o n e r a t the r i g h t t ime i s l i k e l y a h i g h l y i n e f f i c i e n t a p p l i c a t i o n of t h i s approach. S e l f Assessment Procedures f o r S e l e c t Groups The development of s e l f - a s s e s s m e n t t o o l s u s e f u l to s p e c i f i c a l l y d e f i n e d p r a c t i c e groups presumably i n c r e a s e s the l i k e l i h o o d o f re levance to p r a c t i c e and of t i m e l y a v a i l a b i l i t y . The American A s s o c i a t i o n of R e s p i r a t o r y T h e r a p i s t s (380) and the American C o l l e g e of D e n t i s t s (342) have both developed s e l f - a s s e s s m e n t procedures s p e c i f i c a l l y f o r t h e i r members. The d e n t a l s e r i e s i s four t e s t s of 150 q u e s t i o n s each wh ich covers a l l content areas of d e n t i s t r y . Use of the t e s t s , i n c l u d i n g c o n -f i d e n t i a l r e p o r t of r e s u l t s , c o s t s d e n t i s t s $50.00 each . In a move to s t i m u l a t e i n t e r e s t i n the s e l f - a s s e s s m e n t t e s t , the c o l l e g e developed a f i f t y i tem inst rument which cou ld be a d m i n i s t r a t e d w i t h i n an hour . This i s c u r r e n t l y be ing g i ven i n d e n t a l s o c i e t y meetings where p a r t i c i -pants score t h e i r own p a p e r s . Apparent l y response i s f a v o r a b l e d e s p i t e the f a c t that no d i r e c t i o n f o r f u r t h e r study i s g i v e n . The major use of s e l f - a s s e s s m e n t procedures has been by med ica l s p e c i a l t y s o c i e t i e s . Rubin (445) r e p o r t s tha t 14% of a l l p r a c t i c i n g - 1 0 -o p t h a l m o l o g i s t s p a r t i c i p a t e d i n the f i r s t s e l f - a s s e s s m e n t examinat ion sponsored by the American Academy of Opthalmology and Oto laryngo logy i n 1971. Th is t e s t covered the s p e c i a l t y ' s e i g h t b a s i c knowledge areas and took four hours to complete . P a r t i c i p a n t s r e c e i v e d c o n f i d e n t i a l r e s u l t s , i n f o r m a t i o n on norms by t ime e lapsed s i n c e g r a d u a t i o n from medica l s c h o o l , and study r e f e r e n c e s f o r each q u e s t i o n . A l though by more recent standards the approach used was somewhat c rude , R u b i n ' s account of the Academy's exper ience w i t h p l a n n i n g , p u b l i c i t y , t e s t a d m i n i s t r a t i o n , and user r e a c t i o n i s i n s t r u c t i v e . He r e p o r t s that many p a r t i c i p a n t s f a i l e d to understand the s t r e n g t h s , l i m i t a t i o n s , and a p p r o p r i a t e use of exam r e s u l t s . One q u e s t i o n the r e p o r t r a i s e s i s , " i s i t more h e l p f u l to s u r -vey a l l knowledge sub -a reas e q u a l l y (as t h i s t e s t d id ) or to weight the exam to r e f l e c t t y p i c a l case l o a d ? " One might c o n c l u d e , as the Academy d i d , t h a t a survey of the un ive rse of knowledge a p p l i c a b l e to that p r a c -t i c e area i s most h e l p f u l , but t h i s avo ids the q u e s t i o n , " i s there any po in t i n sharpening knowledge on something you are not l i k e l y to use?" Th is d e c i s i o n has been l e f t to the p r a c t i t i o n e r . The American Academy of Neurology (324) has designed i t s " e d u c a t i o n exam" p r i m a r i l y f o r r e s i d e n t s and newly graduated n e u r o l o g i s t s to a s s i s t them i n i d e n t i f y i n g weak areas needing f u r t h e r study before t a k i n g Board examinat ions . E v a l u a t i v e data from the 1972 and 1974 exams show the t e s t s to be r e l i a b l e , d i s c r i m i n a t i n g , and v a l i d . Scores are r e l e a s e d to r e s i d e n t s , t h e i r t r a i n i n g d i r e c t o r s , and to p r a c t i t i o n e r s . There i s no f o l l o w - u p and no guidance fo r f u r t h e r s t u d y . Far more s t r u c t u r e d as an e d u c a t i o n a l t o o l i s SESAP, the S u r g i c a l Educat ion and Se l f -Assessment Program o f f e r e d by the American C o l l e g e of Surgeons ( 4 6 7 ) . Th is i s d e s c r i b e d as a s e l f - c o n t a i n e d , comprehensive a i d to c o n t i n u i n g educat ion c o n t a i n i n g over seven hundred assessment i t e m s , b l a c k and white and c o l o r i l l u s t r a t i o n s , and an a u t h o r -i t a t i v e c r i t i q u e (essay answer) of each i tem w i t h r e f e r e n c e s and r e p r i n t s e r v i c e . The American Co l lege of P h y s i c i a n ' s M e d i c a l Knowledge S e l f -Assessment Program (MKSAP) i s designed not o n l y fo r a s p e c i f i c g roup , but a l s o f o r a s p e c i f i c purpose ( 7 8 , 4 3 6 , 5 3 9 ) . O r i g i n a l l y a s e l f - a s s e s s m e n t program much l i k e those d e s c r i b e d above, the t h i r d and f o u r t h v e r s i o n s of t h i s program were designed to a s s i s t p h y s i c i a n s to prepare f o r the A m e r i -can Board of I n t e r n a l Medic ine r e c e r t i f i c a t i o n exam. Enro l lment f o r MKSAP I I I was t h i r t y thousand, and the r e c e r t i f i c a t i o n exam was passed by 95% of those e n r o l l e d who took the exam. MKSAP IV i n c l u d e d a s y l l a b u s i n the form of a c o n c i s e study guide of c u r r e n t i n f o r m a t i o n (what exper ts b e l i e v e d to have been important developments of the l a s t f i v e y e a r s ) ; t e s t m a t e r i a l s f o r e leven sub jec t areas w i t h f a c t u a l and c l i n i c a l problem s o l v i n g quest ions i n each ; e v a l u a t i v e data ( p a r t i c i p a n t s can s e l f - s c o r e or have t h i s done by computer) ; and comments, re fe rence books , and i l l u s -t r a t i o n s f o r r e v i e w i n g one 's answers. E v a l u a t i o n of MKSAP has been l i m i t e d to q u a n t i f y i n g i t s p o p u l a r i t y and p a r t i c i p a n t s ' success on r e c e r t i f i c a t i o n exams; no e f f e c t on p r a c t i c e has been demonstrated nor i s t h i s p lanned . I t may be concluded t h a t , w i t h the except ion of m e d i c i n e , the h e a l t h p r o f e s s i o n s have not as yet shown great i n t e r e s t i n the s e l f -assessment approach to m a i n t a i n i n g competence. In summarizing the extent to which s e l f - a s s e s s m e n t i s used i n medic ine however, Lever idge (296) s t a t e s tha t over twenty med ica l s o c i e t i e s now o f f e r programs to members; he p r o j e c t s tha t t h i s p r a c t i c e w i l l grow. Whi le i n d i v i d u a l programs -12-vary, most i n c l u d e t e s t i n g of knowledge required f o r s p e c i a l t y area per-formance, d i r e c t i o n f o r f u r t h e r study, and p r o v i s i o n of the necessary m a t e r i a l s f o r study. Those programs which score t e s t s f o r members appear to go to great lengths to protect p a r t i c i p a n t s ' anonymity and most use i n f o r m a t i o n on aggregate t e s t s r e s u l t s to provide d i r e c t i o n i n planning f o r other types of co n t i n u i n g education. There seems to have been l i t t l e e f f o r t to a s c e r t a i n whether self-assessment a c t i v i t y has a p o s i t i v e e f f e c t on p r a c t i c e behaviors or on h e a l t h outcomes. Rather, providers and users have tended to be con-tent to assume that r e l e v a n t knowledge increases as a r e s u l t of p a r t i c i -p a t i o n and that t h i s , i n i t s e l f , i s s u f f i c i e n t j u s t i f i c a t i o n . Butt (78), i n d i s c u s s i n g e v a l u a t i o n of self-assessment programs, s t a t e s "some things that count can't be measured and some things that can be measured don't count." Formal Tes t i n g of P r a c t i t i o n e r s In c o n t r a s t to self-assessment, formal t e s t i n g procedures are l e s s l i k e l y to focus on motivating and a i d i n g improvement of e s s e n t i a l knowledge and more l i k e l y to be used f o r a summative e v a l u a t i o n of a p r a c t i t i o n e r ' s knowledge, s k i l l s , or a t t i t u d e s . These assessment proce-dures are c h a r a c t e r i z e d by c o n t r o l l e d a d m i n i s t r a t i o n , and o f t e n concern f o r s e c u r i t y , to ensure that the evaluee cannot cheat. Information on t h i s group of assessment procedures w i l l be considered i n three categor-i e s : t e s t s which survey the f u l l range of p r o f e s s i o n a l p r a c t i c e w i t h i n the d i s c i p l i n e , t e s t s which focus on a s p e c i a l t y area of p r a c t i c e , and a small group of other formal s p e c i a l purpose assessment procedures. - 1 3 -Test f o r F u l l Range of P r a c t i c e Broad spectrum t e s t i n g of p r a c t i c i n g p r o f e s s i o n a l s i s a lmost n o n - e x i s t e n t , perhaps r e f l e c t i n g the r e a l i t y tha t once past sc reen ing f o r i n i t i a l l i c e n s u r e , the t y p i c a l p r a c t i t i o n e r ' s scope of p r a c t i c e narrows immedia te l y . Indeed, the f i r s t of the three assessment procedures i n t h i s category mentioned i n the l i t e r a t u r e i s o f t e n used p r i o r to l i c e n -s u r e . The examinat ion prov ided by the p r e s t i g i o u s N a t i o n a l Board of M e d i c a l Examiners (NBME) i s intended to eva luate a p h y s i c i a n ' s read iness f o r independent p r a c t i c e . While p a r t i c i p a t i o n i s v o l u n t a r y , the s t a t u s rewards are h i g h and many s t a t e s w i l l accept success i n the " n a t i o n a l boards" i n l i e u of a l i c e n s u r e examinat ion ( 1 0 5 ) . Formed i n 1916 w i t h the o b j e c t i v e of o f f e r i n g h i g h q u a l i t y exams i n the p r a c t i c e of medic ine fo r e x t e r n a l agenc ies ' d i s c r e t i o n a r y u s e , the NBME exam evolved as a three par t assessment procedure : I, b a s i c s c i e n c e s ; I I , c l i n i c a l sub -j e c t s ; I I I , l a b and bedside p r a c t i c a l exam (128) . P a r t s I and I I have undergone c o n s i d e r a b l e ref inement as a r e s u l t of r e s e a r c h and development i n c o g n i t i v e t e s t i n g and today are l a r g e l y c o n s t r u c t e d i n w r i t t e n o b j e c t -i v e i tem fo rmat . In 1968, because of the d i f f i c u l t i e s i n c o n t r o l l i n g c l i n i c a l s i t u a t i o n s and a c h i e v i n g r e l i a b i l i t y i n s c o r i n g , p a r t I I I underwent a major r e v i s i o n . The NBME chose the c r i t i c a l i n c i d e n t technique as a way to d e f i n e what w i l l be t e s t e d . Over three thousand i n c i d e n t s were c o l -l e c t e d from over s i x hundred docto rs who superv i sed i n t e r n s . The i n c i -dents f e l l i n t o n ine major a r e a s : h i s t o r y , p h y s i c a l , t e s t s and p r o c e -d u r e s , d i a g n o s t i c acumen, t rea tment , judgement and s k i l l i n g i v i n g c a r e , c o n t i n u i n g c a r e , p h y s i c i a n - p a t i e n t r e l a t i o n s , and r e s p o n s i b i l i t i e s of a -14-physician. From these, three separate methods were devised for use i n part I I I : (a) motion pictures and multiple choice items to test observa-t i o n and diagnostic s k i l l , (b) patient management problems to test c l i n i -c a l problem solving, (c) presentation of combinations of patient data, e.g. lab test r e s u l t s , x-rays, recordings, tracings to test c l i n i c a l information. While without question t h i s examination assesses a broad range of physician behavior, i t should be remembered that a t t i t u d e s and in t e r - a c t i v e s k i l l s are tested only i n d i r e c t l y and motor s k i l l s not at a l l i n the NBME format. The New York State Regents External Degree Program i n Nursing (B.S.N.) moves beyond the constraints of the NBME exam and assesses per-formance i n the c l i n i c a l s e t t ing (509). S t i l l under development, t h i s program w i l l make av a i l a b l e to pr a c t i c i n g nurses a baccalaureate degree i n nursing acquired through successful challenge of a sequential battery of cognitive tests of general subjects, relevant c l i n i c a l subjects, and three performance exams: c l i n i c a l performance, which involves a p p l i c a -t i o n of nursing process i n caring for several c h i l d and adult patients i n the h o s p i t a l s e t t i n g ; health assessment performance, which tests a b i l i t y to obtain, analyze, and summarize e s s e n t i a l c l i n i c a l information; and professional performance, which tests a b i l i t y to use a v a r i e t y of r e -sources and strategies i n a s s i s t i n g c l i e n t s and their f a m i l i e s i n a v a r i -ety of setting s . The c l i n i c a l performance examination i s the only one of the three performance exams developed and tested to date. It i s also used as part of the assesssment procedure for the New York State External Degree Program i n Nursing (Associate Degree) which i s preparatory to lic e n s u r e . - 1 5 -Because c o n t r o l l e d use of the c l i n i c a l s i t u a t i o n as t e s t mater -i a l i s un ique , the procedure i s worth examining i n some d e t a i l . The exam c o n s i s t s of one out of two p o s s i b l e c l i n i c a l l a b o r a t o r y t e s t s ( e . g . dem-o n s t r a t i o n of the c o r r e c t p r e p a r a t i o n of m e d i c a t i o n s ) , fo l lowed by exam-i n a t i o n i n two out of a p o s s i b l e three a d u l t p a t i e n t care s i t u a t i o n s and i n one out of a p o s s i b l e two c h i l d care s i t u a t i o n s . In each of the p a t i e n t care s i t u a t i o n s , the evaluee must comply w i t h p r e - e s t a b l i s h e d standards fo r p lann ing and e v a l u a t i n g care and must demonstrate a l l of the s p e c i f i c a l l y d e f i n e d " c r i t i c a l e lements" of nurse b e h a v i o r . There must be no jeopardy of p a t i e n t p h y s i c a l or emot ional s a f e t y and no v i o l a -t i o n of a s e p s i s . There are twenty p o s s i b l e areas of care which can be tes ted ( 2 9 5 ) . Lenburg (294) d e s c r i b e s the g e n e r a l procedures which were used to develop the examinat ion and d i s c u s s e s as w e l l the procedures used f o r o r i e n t i n g h o s p i t a l s t a f f and t r a i n i n g e v a l u a t o r s . R e l i a b i l i t y i s s a t i s f a c t o r y . By J u l y 1976, t h i r t y - e i g h t a p p l i c a n t s had completed the exam and 92% of these had passed State Boards ( the n a t i o n a l average i s 81% f o r a s s o c i a t e degree graduates and 88% f o r b a c c a l u r e a t e degree g r a d -u a t e s ) . L i b b y (303) has developed a broad assessment approach f o r e v a l -u a t i o n of pharmacy t e c h n i c i a n performance; t h i s has been used on ly f o r r e s e a r c h . He developed a combinat ion inst rument that i n c l u d e d a t o o l f o r a t t i t u d e t e s t i n g , a f i f t y i tem c o g n i t i v e t e s t , and f i v e s k i l l a p p r a i s a l i tems to determine p r o f i c i e n c y a t t h i r t e e n de f ined t e c h n i c i a n performance requ i rements . Of most i n t e r e s t was h i s f i n d i n g that both t e c h n i c i a n s and the pharmacists w i t h whom they worked overest imated the t e c h n i c i a n s ' a b i l i t y to perform a t an acceptab le s t a n d a r d . The work has not been r e p -l i c a t e d . - 1 6 -T e s t i n g i n S p e c i a l t y Areas Current l i t e r a t u r e r e v e a l s l i t t l e of what i s probably the e x i s t i n g l e v e l of a c t i v i t y i n t e s t i n g competence i n s p e c i a l t y a r e a s . No doubt t h i s i s because s p e c i a l t y c e r t i f i c a t i o n i s not new to the h e a l t h scene and t h e r e f o r e l i k e l y to a t t r a c t l i t t l e a t t e n t i o n . Benson (46) d e s c r i b e s the American Board of I n t e r n a l Medic ine three l e v e l c e r t i f i c a t i o n p r o c e s s . Th is c o n s i s t s of three years of r e s i -dency t r a i n i n g , ongoing e v a l u a t i o n of c l i n i c a l competence d u r i n g t r a i n -i n g , and c o g n i t i v e t e s t i n g . Whi le the Board i s e x e r t i n g some e f f o r t to improve t r a i n i n g e v a l u a t i o n procedures , the mainstay of e v a l u a t i o n i s the c e r t i f i c a t i o n exam. Performance i n t h i s c o r r e l a t e s h i g h l y w i t h c l i n i c a l e v a l u a t i o n r a t i n g s i n those s u b - s p e c i a l t i e s which have used s t r u c t u r e d r a t i n g systems dur ing the r e s i d e n c y p e r i o d . The Board has developed a m a t r i x of th ree d imens ions : tasks of the p h y s i c i a n , e . g . take a h i s t o r y ; r e q u i s i t e a b i l i t i e s ( c o g n i t i v e , a f f e c t i v e , psychomotor) ; and c l i n i c a l problems, e . g . d i s e a s e s and organ systems i n v o l v e d . Th is has prov ided the framework a g a i n s t which the exam i s c o n s t r u c t e d . R e c e r t i f i c a t i o n w i l l be by w r i t t e n exam a l s o ; the Board has cons idered and r e j e c t e d peer r e v i e w , c h a r t a u d i t , and o r a l and computer exams as not p r e s e n t l y meeting the c r i t e r i a of v a l i d i t y , r e l i a b i l i t y , and f e a s i b i l i t y . The c e r t i f i c a t i o n exam of the Canadian Co l lege of Fami ly P h y s i c i a n s i s somewhat bo lder i n i t s approach and broader i n scope ( 3 2 0 ) . I t uses m u l t i p l e cho ice and p a t i e n t management problems as w e l l as f i l m s t r i p s , v i d e o t a p e s , o r a l s , r o l e p l a y i n g and s imulated o f f i c e v i s i t s us ing p r o f e s s i o n a l a c t o r s . No i n f o r m a t i o n i s g i ven on r e l i a b i l i t y or v a l i d i t y t e s t i n g . - 1 7 -C e r t i f i c a t i o n i s not so widespread among the other p r o f e s s i o n s . Both d e n t i s t r y and n u r s i n g m a i n t a i n s p e c i a l t y c e r t i f i c a t i o n programs which are d e s c r i b e d i n the l i t e r a t u r e i n terms of t h e i r p u r -poses , p rocedures , and component p a r t s 2 3 , 2 4 , 3 0 , 9 1 , 1 4 8 , 5 4 6 ) . D e t a i l s about t e s t c o n s t r u c t i o n and e v a l u a t i o n are not prov ided however. S p e c i a l Purpose Assessment Procedures The l i t e r a t u r e p rov ides i n f o r m a t i o n about on ly three a d d i t i o n a l assessment p rocedures ; two of these were developed fo r h i g h l y s p e c i f i c purposes . S i n g l e t o n (475) d e s c r i b e s assessment procedures developed to c o n f i r m s u c c e s s f u l complet ion of a nurse p r a c t i t i o n e r e d u c a t i o n a l p r o -gram. Johnson (264) admin is te red a n u t r i t i o n a l knowledge t e s t to p h y s i c -i ans to i d e n t i f y the need f o r med ica l school c u r r i c u l u m changes, c o n t i n u -i n g e d u c a t i o n , and c o n s u l t a t i o n from n u t r i t i o n i s t s . The t h i r d p rocedure , the use of a s k i l l s i n v e n t o r y , i s d e s c r i b e d i n genera l terms by Cantor ( 8 4 ) . These employer -produced c h e c k l i s t s are most o f t e n s e l f - a d m i n i s t e r -ed by the employee and are used as a b a s i s f o r c l i n i c a l placement and i n s e r v i c e p l a n n i n g . N o n - T r a d i t i o n a l T e s t i n g Procedures A f i n a l group of t e s t i n g procedures , p r i m a r i l y i n developmental s t a g e s , w i l l be d e s c r i b e d i n t h i s s e c t i o n . As might be expected , these are l a r g e l y n o n - t r a d i t i o n a l e v a l u a t i o n approaches and are not e x t e n s i v e l y used . Inc luded are the d e s c r i p t i o n of a h i g h l y i n d i v i d u a l i z e d t e s t i n g system, a group of t e s t s of c l i n i c a l d e c i s i o n making, and a f i n a l group -18-of assessment procedures which look a t i n t e r a c t i v e s k i l l s . I n d i v i d u a l i z e d P h y s i c i a n P r o f i l e The U n i v e r s i t y of Wiscons in I n d i v i d u a l P h y s i c i a n P r o f i l e (395, 476) i s an exper imenta l program which at tempts to prov ide a s e l f - a s s e s s -ment inst rument which i s r e l e v a n t to the p r a c t i c e of the i n d i v i d u a l p h y s i c i a n . Diagnoses from the s u b s c r i b e r p h y s i c i a n ' s p r a c t i c e are coded a c c o r d i n g to the I n t e r n a t i o n a l C l a s s i f i c a t i o n of D i s e a s e s , Adapted (ICDA) and fed i n t o a computer which c o n t a i n s a t e s t bank of some four thousand items coded f o r d i a g n o s t i c category and degree of d i f f i c u l t y . The comp-u te r c a t e g o r i z e s the diagnoses and p r i n t s out a m u l t i p l e cho ice exam which r e f l e c t s p r o p o r t i o n a t e l y the c a t e g o r i e s of d iagnoses present i n the p r a c t i c e . The g r e a t e r the f requency of a p a r t i c u l a r c l i n i c a l problem i n the p h y s i c i a n ' s p r a c t i c e , the more complex the r e l a t e d quest ions become. Once the exam i s completed and s c o r e d , a computer a n a l y s i s of the p h y s i c i a n ' s areas of s t r e n g t h and weakness i s sent to him along w i t h a set of recommendations drawn from a computer ized Educat ion Resource Index, which i n c l u d e s e v e r y t h i n g from conferences to home study m a t e r i -a l s . E a r l i e r i n the project^ persona l e d u c a t i o n a l c o n s u l t a t i o n was used to i n t e r p r e t t e s t r e s u l t s and a i d i n p lann ing f o r f u r t h e r e d u c a t i o n , how-ever t h i s has a p p a r e n t l y been abandoned. The major d isappointment of the program i s i t s u n d e r u t i l i z a t i o n and cos t i n time and money. E v a l u a t i o n has shown that users f i n d the program of va lue i n a s c e r t a i n i n g e d u c a t i o n a l needs. Tests of C l i n i c a l D e c i s i o n Making The most w e l l known of t h i s group i s the p a t i e n t management - 1 9 -problem (PMP), wherein a c l i n i c a l problem i s d e s c r i b e d i n some d e t a i l and the evaluee i s asked to make s e q u e n t i a l d e c i s i o n s about i t s i n v e s t i g a t i o n and management. Whi le the N a t i o n a l Board of Med ica l Examiners has used t h i s assessment approach as a t e s t i n g method s ince the l a t e s i x t i e s , the PMP i s s t i l l e v o l v i n g . Barro (41) has d e s c r i b e d C h r i s t i n e McGui re ' s c u r r e n t work (322) on p a t i e n t s i m u l a t i o n s a t the U n i v e r s i t y of I l l i n o i s as d i f f e r e n t from that used by the NBME, i n tha t l e s s beg inn ing i n f o r m a t i o n i s g iven and a branching p r o g r e s s i o n through the problem g ives the evaluee i n f o r m a t i o n on ly i n response to h i s d i a g n o s t i c ques t ions before r e q u i r i n g him to make f u r t h e r d i a g n o s t i c f o r t h e r a p e u t i c d e c i s i o n s . Th is i n t e r a c t i o n w i t h the problem i s presumably a more accurate s i m u l a t i o n of r e a l i t y - - - indeed the t e s t taker may even " k i l l " h i s p a t i e n t thus ending h i s t e s t problem premature ly . In McGui re ' s s i m u l a t i o n s , a l l p o s s i b l e responses are c l a s s -i f i e d and weighted by exper ts to r e f l e c t the r e l a t i v e va lue to the p a t i -ent of the t h e r a p i s t ' s d e c i s i o n . An E f f i c i e n c y Score (per cent of h e l p -f u l responses) and a P r o f i c i e n c y Score (per cent of a c t i o n s agree ing w i t h e x p e r t s ' d e c i s i o n s i n s e l e c t i n g h e l p f u l and a v o i d i n g harmful a c t i o n s ) are combined to form a Competence Score . R e l i a b i l i t y of t h i s approach i s h i g h . V a l i d i t y has been checked by c h a r t a u d i t of r e a l p a t i e n t s ; a l though fewer c h a r t i tems were recorded than were s e l e c t e d on comparable PMP's those that were s e l e c t e d i n PMP's were found i n c h a r t s . Whether performance on PMP's c o r r e l a t e s w i t h more d i r e c t p e r -formance measures i s unknown, a l though Pa lva and Korhonen ( 3 7 8 ) , i n a study i n v o l v i n g performance on a PMP r e l a t e d to d rug - induced a g r a n u l o -c y t o s i s , found t h a t outcomes on the s i m u l a t i o n c o r r e l a t e d w i t h a c t u a l outcomes i n a l o c a l med ica l s e r v i c e . S p e c i f i c a l l y , 10% of the student - 2 0 -t r e a t e d p a t i e n t s i n the s i m u l a t i o n would have d i e d from improper t r e a t -ment; exper ience w i t h the l o c a l med ica l s e r i c e showed an i d e n t i c a l m o r t a -l i t y r a t e over a four year p e r i o d . Th is r a t h e r unique study was c h a r a c -t e r i z e d by n a t u r a l l y o c c u r r i n g c o n t r o l s that would not o r d i n a r i l y have been a v a i l a b l e to v a l i d a t e the PMP. The l o c a l med ica l s e r v i c e was s t a f f -ed almost e x c l u s i v e l y by graduates of the t r a i n i n g program, and the d i sease e n t i t y i s s u f f i c i e n t l y r a r e that graduates cou ld not have lea rned by exper ience i n the i n t e r v e n i n g p e r i o d . Moreover, a g r a n u l o c y t o s i s m o r t -a l i t y i s d i r e c t l y a t t r i b u t a b l e to p h y s i c i a n d e c i s i o n making and a c t i o n , and not s u b j e c t to o ther v a r i a b l e s . The authors t h e r e f o r e concluded t h a t f o r t h i s c l i n i c a l problem, performance on the PMP was p r e d i c t i v e of c l i n -i c a l performance. The MERIT P r o j e c t (Model f o r E v a l u a t i o n and R e c e r t i f i c a t i o n Through I n d i v i d u a l i z e d Tes t ing ) was developed w i t h f i n a n c i a l a s s i s t a n c e from the American Board of I n t e r n a l Medic ine and has r e c e i v e d c o n s i d e r -ab le a t t e n t i o n ( 1 8 0 , 3 8 6 ) . The sub jec t p h y s i c i a n develops a p e r s o n a l i z e d p r a c t i c e p r o f i l e by us ing a d i a r y technique f o r one hundred consecut i ve p a t i e n t s to c o l l e c t data on c h i e f c o m p l a i n t , exam and l a b r e s u l t s , t r e a t -ment, long term management and outcomes. These sample cases are b o i l e d down to three r e p r e s e n t a t i v e p a t i e n t care s i m u l a t i o n s which the p h y s i c i a n " t r e a t s " through a computer t e r m i n a l ; the i n t e r a c t i o n i s conducted i n n a t u r a l language. Performance i s scored on f i v e d imens ions : thorough -ness of e x a m i n a t i o n , d i a g n o s t i c a c c u r a c y , e f f i c i e n c y , appropr ia teness of management, and danger and d i scomfor t to p a t i e n t . Feedback i s immediate. V a l i d a t i o n of the MERIT procedure has been accompl ished us ing an a u d i t of o f f i c e records f o r c r i t i c a l concepts . Those concepts found i n a c t u a l p a t i e n t management were a l s o a p p l i e d to the p h y s i c i a n ' s management - 2 1 -of the computer c a s e s ; c o r r e l a t i o n s were a c c e p t a b l e . Development cos t i s an obvious problem w i t h the MERIT approach. Each s i m u l a t i o n takes s i x hours author t i m e , twenty hours f o r t r i a l and r e v i s i o n , and f i f t y hours fo r programming. Al though MERIT was developed as a p o s s i b l e r e c e r t i f i c a -t i o n procedure , the h i g h c o s t s a s s o c i a t e d w i t h t e s t development and s e c -u r i t y measures make i t more u s e f u l as an e d u c a t i o n a l t o o l . A d d i t i o n a l l y , the use of a p e r s o n a l i z e d r a t h e r than a standard t e s t f o r summative e v a l -u a t i o n cou ld be a c o n t e n t i o u s i s s u e . There seems to have been l i t t l e e x p l o r a t i o n of i t s p o t e n t i a l as a d i a g n o s t i c t o o l f o r p lann ing the p r a c -t i t i o n e r ' s cont inued e d u c a t i o n . Another d e c i s i o n making t e s t , R i m o l d i ' s t e s t of d i a g n o s t i c s k i l l s , i s a l s o d e s c r i b e d by Barro ( 4 1 ) . Th is s imple inst rument p resents the evaluee w i t h a w r i t t e n p a t i e n t s i t u a t i o n and a set of s m a l l c a r d s , each of which shows on one s i d e a h i s t o r y , p h y s i c a l or l a b q u e s t i o n which he may choose to "ask" by t u r n i n g the card over to d i s p l a y the i n f o r m a -t i o n sought. D i a g n o s t i c s k i l l i s scored accord ing to the number of quest ions a s k e d , t h e i r u s e f u l n e s s , and sequence. No doubt the l a t e n t image and computer ized a p p l i c a t i o n of PMP's are seen as e a s i e r to manage than packs of c a r d s , thus account ing f o r the f a c t tha t t h i s system appears to be l i t t l e used . N e v e r t h e l e s s , i t i s a p p e a l i n g i n the s i m p l i -c i t y of m a t e r i a l s r e q u i r e d . Tests of I n t e r a c t i v e S k i l l In her rev iew of p h y s i c i a n assessment p rocedures , Barro (41) has i d e n t i f i e d a few l i t t l e - u s e d but i n n o v a t i v e approaches. She notes t h a t E l s t e i n i n 1972 used a c t o r s to p rov ide a h i s t o r y ( p h y s i c a l a s s e s s -ment data i s s imply prov ided i n w r i t i n g ) ; the p h y s i c i a n i s assessed by -22-observers f o r h i s i n t e r v i e w i n g and c o u n s e l l i n g s k i l l s us ing a c r i t e r i o n check l i s t . Th is a l lows some c o n t r o l of the v a r i a b l e s w i t h which the p h y s i c i a n must d e a l i n the a c t u a l c l i n i c a l s i t u a t i o n . The amount, type and order of i n f o r m a t i o n gathered i s sub jec t to s c r u t i n y . R e l i a b i l i t y and v a l i d i t y of t h i s approach have not been e s t a b l i s h e d . Barro a l s o summarizes s e v e r a l attempts a t e v a l u a t i n g p a t i e n t -p r a c t i t i o n e r communicat ion: Gozz i (1969) , Korsch (1968) , Davis (1971) , and Addler and Enelow (1966) a l l based t h e i r assessment attempts on B a l e ' s I n t e r a c t i o n Process A n a l y s i s . Gozzie and Korsch counted " b l o c k s " and " f a c i l i t a t o r s " i n an episode of communication (and demonstrated a reasonable degree of r e l i a b i l i t y and of v a l i d i t y ) to produce a per fo rm-ance index . Davis developed a more e l a b o r a t e coding system fo r twelve c a t e g o r i e s of communication b e h a v i o r . Th is a d a p t a t i o n of B a l e ' s system was reproduced i n B a r r o ' s rev iew and t y p i f i e s attempts a t communication measurement under t e s t c o n d i t i o n s (see F igure 1 ) . Summary and Comment T h i s chapter has reviewed approaches to e v a l u a t i n g the know-l e d g e , s k i l l s and a t t i t u d e s which p r a c t i t i o n e r s b r i n g to the p a t i e n t care s i t u a t i o n . The three broad t o p i c areas covered were s e l f - a s s e s s m e n t p r o -cedures , formal t e s t i n g of p r a c t i t i o n e r s , and n o n - t r a d i t i o n a l t e s t i n g procedures . S e l f - t e s t i n g dev ices a v a i l a b l e i n p r o f e s s i o n a l l i t e r a t u r e are popular as c o n t i n u i n g educat ion measures. They assume a p r a c t i t i o n e r w i l l s e l e c t t e s t m a t e r i a l r e l e v a n t to h i s a r e a , w i l l have access to r e --23-Social-Emotional Area A{ Positive Task Area Neutral Social-Emotional Area D Negative 1 Shows solidarity, raises other's status, gives help, reward 2 Shows tension release, jokes, laughs, shows satisfaction 3 Agrees, shows passive acceptance, un-derstands, concurs, complies 4 Gives suggestion, direction, implying autonomy (or other 5 Gives opinion, evaluation, analysis, expresses feeling, wish 6 Gives orientation, information, repeats, clarifies, confirms 7 Asks for orientation, information, repetition, confirmation 8 Asks for opinion, evaluation, analysis, expression of feeling 9 Asks for suggestion, direction, pos-sible ways of action 10 Disagrees, shows passive rejection, formality, withholds help 11 Shows tension, asks for help, withdraws out of field 12 Shows antagonism, deflates other's status, defends or asserts self KEY a b e d a Problems of Communication b Problems of Evaluation c Problems of Control d Problems of Decision e Problems of Tension Reduction f Problems of Reintegration A Positive Reactions B Attempted Answers C Questions D Negative Reactions Interaction Process Analysis. After Robert F. Bales, Interaction Proc-ess Analysis: A Method for the Study of Small Groups (Reading, Mass.: Addison-Wesley, 1950). Used by permission of Robert F. Bales. Figure 1 - 2 4 -l a t e d l e a r n i n g m a t e r i a l s , and w i l l be ab le to l e a r n from them. There i s not evidence tha t t h i s happens. In an e f f o r t to overcome these p o t e n t i a l l i a b i l i t i e s , over twenty (296) of the s p e c i a l t y med ica l s o c i e t i e s i n the Uni ted S ta tes have developed s e l f - a s s e s s m e n t procedures fo r t h e i r mem-b e r s ' u s e . Whi le these vary i n t h e i r approach and degree of s o p h i s t i c a -t i o n , a t y p i c a l program w i l l t e s t the p h y s i c i a n , prov ide feedback about areas of s t r e n g t h and weakness, d i r e c t him i n f u r t h e r s tudy , and prov ide r e l e v a n t l e a r n i n g m a t e r i a l s . Aggregate t e s t r e s u l t s are used to p l a n c o n t i n u i n g educat ion programs f o r groups . There seems to have been l i t t l e e f f o r t to a s c e r t a i n whether s e l f - a s s e s s m e n t a c t i v i t y has a p o s i -t i v e e f f e c t on p r a c t i c e behav ior or on h e a l t h outcomes. R a t h e r , p r o v i d -ers and users have tended to be content to assume t h a t g i ven i n f o r m a t i o n about knowledge d e f i c i t s ? the p r a c t i t i o n e r w i l l be mot ivated to l e a r n , and that t h i s i n i t s e l f i s s u f f i c i e n t j u s t i f i c a t i o n . Formal t e s t i n g procedures f o r p r a c t i c i n g p r o f e s s i o n a l s are l i m i t e d to summative e v a l u a t i o n procedures used p r i m a r i l y f o r c r e d e n t i a l -i n g at an advanced l e v e l . There are few attempts to eva luate a broad range of p r a c t i t i o n e r b e h a v i o r s , r e f l e c t i n g the r e a l i t y t h a t the scope of p r a c t i c e g e n e r a l l y narrows to a s p e c i a l t y a rea w i t h i n the d i s c i p l i n e soon a f t e r l i c e n s u r e o c c u r s . A l though a c t i v e s p e c i a l t y c e r t i f i c a t i o n programs e x i s t , the d e t a i l s of t e s t development and c o n s t r u c t i o n f o r these p r o -grams are i n f r e q u e n t l y r e p o r t e d i n the l i t e r a t u r e . O c c a s i o n a l l y fo rmal t e s t i n g procedures are developed f o r s p e c i a l purposes such as r e s e a r c h . I t does not appear that fo rmal t e s t i n g procedures are used to a s s i s t p r a c t i t i o n e r s w i t h p r o f e s s i o n a l development. The l i t e r a t u r e a l s o r e f l e c t s a c o n t i n u i n g and a c t i v e i n t e r e s t i n i n n o v a t i v e approaches to t e s t i n g h e a l t h p r o f e s s i o n a l s . Most of t h i s -25-developmental work appears to be t a k i n g p lace w i t h i n the med ica l p r o f e s -s i o n . I t may be concluded that to the extent that assessment of the s t r u c t u r e of h e a l t h p r o f e s s i o n a l s ' competence e x i s t s , the p r o f e s s i o n s are i n c l i n e d to be thorough i n t h e i r t e s t i n g of lower l e v e l s of c o g n i t i v e f u n c t i o n i n g , are beg inn ing to f i n d e f f e c t i v e ways to assess a b i l i t y i n c l i n i c a l d e c i s i o n making (which i s cons idered by many to be the most c r i t i c a l a rea of p r a c t i c e ) and are j u s t beg inn ing to exp lore t e s t i n g psychomotor s k i l l s or a t t i t u d i n a l l y r e l a t e d p r o f e s s i o n a l b e h a v i o r s . P e r -haps t h i s l a s t i s more unders tandab le , as consensus on v a l u e s i s d i f f i -c u l t to ach ieve i n s o c i e t y a t l a r g e . I t i s to be hoped that the New York State c l i n i c a l n u r s i n g e v a l u a t i o n exper ience w i l l prove tha t i t i s p o s s i -b l e to r e l i a b l y measure and make judgement about the adequacy of a wider range of p r o f e s s i o n a l behav io rs which the p r a c t i t i o n e r b r i n g s to the job than has been commonly assumed to be p o s s i b l e i n the p a s t . Accept ing tha t r e l i a b l e e v a l u a t i o n of c a p a b i l i t y f o r c e r t a i n aspec ts of p r o f e s s i o n a l p r a c t i c e i s p o s s i b l e , and that t e s t s f o r e v a l u a -t i o n of a f u l l e r range of c l i n i c a l behav io rs can be deve loped , i t i s important to recognize that even the most d e s i r a b l e of these assessment techniques are but proxy measures f o r r e a l - w o r l d c l i n i c a l performance. For t h i s reason e v a l u a t i o n of the s t r u c t u r e of competence may always be more u s e f u l i n fo rmat ive than i n summative e v a l u a t i o n . -26-Chapter I I I ASSESSING CLINICAL PERFORMANCE: PROCESS Cur rent concern w i t h the i s s u e of competence d e r i v e s from a d e s i r e to assure that p r a c t i t i o n e r s w i l l do the best t h i n g p o s s i b l e f o r t h e i r c l i e n t s . The most d i r e c t assessment of competence w i l l judge the p r a c t i t i o n e r ' s b e h a v i o r , i . e . what he does , as he c a r r i e s out h i s p r o f e s -s i o n a l f u n c t i o n s . Th is chapter examines recent l i t e r a t u r e concerned w i t h a s s e s s i n g the knowledge, s k i l l s , or a t t i t u d e s which p r a c t i t i o n e r s e x h i b i t i n c l i n i c a l p r a c t i c e . There are three major s e c t i o n s : sys temat ic assessment of t o t a l performance, sys temat ic assessment of s i t u a t i o n -s p e c i f i c performance, and unplanned e v a l u a t i o n of performance. Sys temat ic Assessment of T o t a l Performance Whi le i t i s an overstatement to say that the f u l l range of any p r a c t i t i o n e r ' s performance i s ever a s s e s s e d , t h i s s e c t i o n w i l l rev iew r e p o r t s of assessment procedures which eva luate a broad spectrum of p r o -f e s s i o n a l p r a c t i c e b e h a v i o r s . Many of these schemes are open ended; i . e any behavior which can be observed or measured w i l l be c o n s i d e r e d . Others i d e n t i f y the areas of behav ior to be a s s e s s e d , but these areas c o l l e c t i v e l y i n c l u d e a wide range of p o s s i b l e p r o f e s s i o n a l b e h a v i o r s . Th is l i t e r a t u r e i s c a t e g o r i z e d and d i scussed under four head ings : g e n -e r a l d i s c u s s i o n of sys temat ic performance e v a l u a t i o n ; performance e v a l u a -- 2 7 -t i o n us ing i m p l i c i t s tandards ; performance e v a l u a t i o n w i t h e x p l i c i t , but imprec ise s t a n d a r d s ; performance e v a l u a t i o n us ing e x p l i c i t s t a n d a r d s . General D i s c u s s i o n s of Systemat ic Performance E v a l u a t i o n There are few g e n e r a l statements or g u i d e l i n e s fo r performance e v a l u a t i o n which prov ide s u b s t a n t i a l i n f o r m a t i o n to e v a l u a t o r s . Perhaps the most comprehensive overview of performance a p p r a i s a l found i n t h i s l i t e r a t u r e search i s prov ided by Haar and H icks ( 2 1 0 ) . These authors d e s c r i b e the advantages and d isadvantages of the most common types of assessment approaches: essay techn ique , f i e l d rev iew ( s e v e r a l people rev iew performance and a r r i v e a t consensus ) , fo rced cho ice ( the r a t e r must choose d e s c r i p t i v e statements of behav ior most and l e a s t l i k e the e v a l u e e ) , c o l l e c t i o n s of c r i t i c a l i n c i d e n t s , rank ing of evaluees i n r e l a t i o n to each o t h e r , g raph ic and numer ica l r a t i n g s c a l e s , check l i s t s , q u a n t i t a t i v e work s t a n d a r d s , and assessment a g a i n s t p r e - e s t a b l i s h e d work o b j e c t i v e s (management by o b j e c t i v e s ) . The twenty - four r e f e r e n c e s , most l y from management l i t e r a t u r e , p rov ide f u r t h e r i n f o r m a t i o n f o r the r e a d e r . M a r r i n e r (328) d i s c u s s e s the pros and cons of many of these same t e c h n i q u e s ; she a l s o d e s c r i b e s such common r a t e r e r r o r s as that of h a l o , l o g i c a l a s s o c i a t i o n , p r o x i m i t y , c e n t r a l tendency, and l e n i e n c y , and suggests techniques to overcome t h e s e . D e t a i l e d d i s c u s s i o n of how to eva luate c l i n i c a l performance us ing the c r i t i c a l i n c i d e n t technique i s presented by We ins te in ( 5 2 7 ) ; check l i s t , by McPherson (323) ; and r a t i n g s c a l e s , by Dohner ( 1 4 0 ) . Severa l w r i t e r s ( 9 , 2 1 0 , 2 7 4 , 3 2 8 , 4 8 7 ) o u t l i n e d e s i r a b l e a t t r i -butes f o r any performance e v a l u a t i o n system. C o l l e c t i v e l y , they conclude tha t assessment techniques must be s e l e c t e d or designed fo r the purpose - 2 8 -f o r which they w i l l be used and must be congruent w i t h the agency 's ph i losophy and g o a l s ; s tandards must be c l e a r and unambiguous; there must be c a r e f u l documentat ion; e v a l u a t o r s should be t r a i n e d i n the use of the i n s t r u m e n t , i n a v o i d i n g common r a t e r e r r o r s , and i n problem s o l v i n g and i n t e r v i e w i n g ; management must support and enforce e v a l u a t i o n p o l i c i e s ; assessment should be ongoing to provide fo r immediate and meaningfu l feedback; and the pr imary focus should be the e v a l u e e ' s cont inued growth. A l b r e c h t (9) emphasizes the importance of evaluee p a r t i c i p a t i o n i n the e v a l u a t i o n process i f behavior change i s to o c c u r . Stevens (487) g i ves c o n s i d e r a b l e a t t e n t i o n to summative as w e l l as fo rmat i ve e v a l u a t i o n , and l i s t s common reasons f o r f a i l u r e of the e v a l u a t i o n process when i t must r e s u l t i n d i s c i p l i n e : poor documentat ion , i n a p p r o p r i a t e l y severe d i s c i p l i n e , use of standards not r e l a t e d to the e v a l u e e ' s r e s p o n s i b i l i t i e s , and f a i l u r e to use due p r o c e s s . Performance E v a l u a t i o n Using I m p l i c i t Standards The most uns t ruc tu red approach to performance e v a l u a t i o n i s to l e a v e e n t i r e l y u n s p e c i f i e d e i t h e r the parameters of behavior to be e v a l u -ated or the standards of a c c e p t a b i l i t y . A s t r a t e g y which p rov ides s l i g h t l y more s t r u c t u r e i s to d e s c r i b e the parameters or v a r i a b l e s to be a s s e s s e d , but to l eave the standards u n s t a t e d . L i t e r a t u r e concerned w i t h both of these approaches i s d i scussed below. The Massachusetts Department of H e a l t h (330) r e p o r t s e v a l u a t i n g competence of employee p h y s i c i a n s by the s imple expedient of commiss ion-ing an e x t r a - m u r a l s i t e rev iew team of th ree respected p h y s i c i a n s to assess adequacy of med ica l s e r v i c e s i n the s e v e r a l h o s p i t a l s owned by the - 2 9 -s t a t e . The rev iew team i s a p p a r e n t l y a t l i b e r t y to dec ide which areas of performance to eva luate and to dec ide what l e v e l of behav ior i s a c c e p t -a b l e . The team i s r e p o r t e d l y a b l e to i d e n t i f y s u p e r i o r , adequate, mar -g i n a l l y adequate and incompetent p h y s i c i a n s . A l l o w i n g the performance v a r i a b l e s and standards to remain unstated before rev iew seems to be used f r e q u e n t l y i n p s y c h i a t r i c s e r -v i c e s . Two r e p o r t s (297,360) from mental h e a l t h c e n t e r s d e s c r i b e us ing a m u l t i - d i s c i p l i n a r y case conference format i n which the t h e r a p i s t ' s man-agement of a case i s eva luated and recommendations are made. Both r e p o r t s i n d i c a t e tha t t h i s approach r e t a i n s the f l e x i b i l i t y necessary to accommodate d i f f e r e n t t h e r a p e u t i c s t y l e s , i s h i g h l y e d u c a t i o n a l , i s u s e -f u l i n i d e n t i f y i n g problem a r e a s , and ach ieves sav ings by reduc ing unnec-e s s a r y or prolonged use of s e r v i c e s . Because no measurement a g a i n s t s t a t e d standards take p lace over time however, i t i s d i f f i c u l t to support these c la imed advantages w i t h e v i d e n c e . One of these authors (360) has i n d i c a t e d tha t where i t i s necessary to s h i f t focus from educat ion and c o n s u l t a t i o n to judgement of adequacy, i . e . when performance f a l l s below the i m p l i e d min imal s t a n d a r d , there emerge a number of problems which are d i f f i c u l t to r e s o l v e . K e r s t e i n and Weissman (276) were ab le to use a s i m i l a r approach when they reviewed the c h a r t s of t h i r t y - s i x n o n - h o s p i -t a l i z e d s u i c i d e a t t e m p t e r s . They found roughly o n e - h a l f of these cases had been adequate ly t r e a t e d and i d e n t i f i e d d e f i c i e n c i e s both i n r e s i d e n c y t r a i n i n g and p s y c h i a t r i c s e r v i c e s d e l i v e r y . The p r a c t i c e of s p e c i f y i n g the v a r i a b l e s to be eva luated and l e a v i n g standards unstated i s a l s o commonly seen . Hadamard et a l . (211) r e p o r t a n a l y z i n g c o n s u l t a t i o n requests from h o s p i t a l employed p h y s i c i a n s i n order to assess the adequacy of t h e i r performance. In t h i s i n s t a n c e , - 3 0 -the parameters to be cons idered were s p e c i f i e d us ing a form developed elsewhere to eva luate candidates fo r board a c c r e d i t a t i o n i n i n t e r n a l m e d i c i n e ; they addressed such broad v a r i a b l e s as h i s t o r y t a k i n g , comp-l e t e n e s s of p h y s i c a l exam, e t c . Standards of a c c e p t a b i l i t y were l e f t u n s p e c i f i e d . P h y s i c i a n s took seven minutes each to analyze two hundred s i x t y c o n s u l t a t i o n notes and judge performance. The i r judgements c o r r e -l a t e d w i t h s u b j e c t i v e r a t i n g s g iven the h o s p i t a l d o c t o r s by t h e i r super -i o r s , thus l e a d i n g the study group to conclude tha t the approach has m e r i t . Another study team (461) analyzed t r a n s c r i p t i o n s (from d i c t a -t i o n ) of h o s p i t a l employed p h y s i c i a n s ' i n i t i a l assessments and treatment p l a n s . Again broad areas of behav ior to be cons idered were s p e c i f i e d , but the standard of adequacy l e f t to the e v a l u a t o r . D e f i c i e n c i e s were i d e n t i f i e d by category of p h y s i c i a n ( e . g . c l e r k s were h i g h i n omiss ions from h i s t o r y , j u n i o r r e s i d e n t s handled pneumonias p o o r l y , e t c . ) Th is approach was s u f f i c i e n t l y s e n s i t i v e that s e n i o r med ica l s t a f f i n v a r i a b l y scored b e t t e r than t h e i r j u n i o r s . R e l i a b i l i t y t e s t s were not done. The l i t e r a t u r e i n d i c a t e s tha t the task of i d e n t i f y i n g p e r f o r m -ance v a r i a b l e s f o r e v a l u a t i o n has been approached i n a v a r i e t y of ways. K i m b a l l (274) suggests tha t the job d e s c r i p t i o n be analyzed to d e f i n e and d e s c r i b e broad c a t e g o r i e s of nurse b e h a v i o r , e . g . l e a d e r s h i p . She then goes on to suggest that performance can then be evaluated by c o l l e c t i n g anecdota l d e s c r i p t i o n s of b e h a v i o r , c l a s s i f y i n g these by c a t e g o r y , and e s t i m a t i n g the degree of a c c e p t a b i l i t y of each area of b e h a v i o r . The U n i v e r s i t y of Miami (305) approached t h i s task by d e f i n i n g broad b e h a v i -o r a l c a t e g o r i e s from a c u r r i c u l u m framework and r a t i n g med ica l s tudents a g a i n s t a four po in t s c a l e . T e s t - r e t e s t r e l i a b i l i t i e s are repor ted as - 3 1 -good, c o r r e l a t i o n s between s e l f and peer r a t i n g s were h i g h , as were c o r -r e l a t i o n s of r a t i n g s w i t h g rades . I t i s suggested that t h i s approach cou ld be used f o r r a t i n g performance of h o s p i t a l med ica l s t a f f . The q u e s t i o n of whether r a t i n g s would c o r r e l a t e w i t h other measures of p e r -formance i s d i s c u s s e d . Other approaches to d e f i n i n g c a t e g o r i e s of behav ior are a l s o suggested i n the l i t e r a t u r e . H a r r i s (219) developed an o b s e r v a t i o n guide fo r nurse p r a c t i t i o n e r s by s t a r t i n g w i t h a l i t e r a t u r e r e v i e w ; then o r g a n -i z i n g the m a t e r i a l us ing a conceptua l framework "dimensions of c a r e , " e . g . assessment ; and f i n a l l y having these v a l i d a t e d by a n a t i o n a l l y recogn ized expert g roup . In another s tudy , Ward (524) i d e n t i f i e d f i f t e e n areas of h e a l t h needs and problems, s p e c i f i e d r e l a t e d competenc ies , then used nurse p r a c t i t i o n e r s i n f i f t y s t a t e s to v a l i d a t e the i n f o r m a t i o n . Return data were a l s o c o r r e l a t e d w i th p r a c t i c e , p o p u l a t i o n , and economic s e t t i n g s . The American Nurses A s s o c i a t i o n ' s Standards of Nurs ing P r a c t i c e ( 2 0 ) , which s p e c i f y the dimensions which must be cons idered when s e l e c t -i n g measurable c r i t e r i a f o r assessment of performance, u t i l i z e the steps of n u r s i n g process as a conceptua l framework. These were de r i ved p r i m a r -i l y from expert p r a c t i t i o n e r o p i n i o n . The standard statements e x i s t i n s e t s : a gener i c set which i s a p p l i c a b l e to a l l n u r s i n g care s i t u a t i o n s , and a number of s p e c i a l t y sets which are a p p l i c a b l e to such areas as m a t e r n i t y n u r s i n g , mental h e a l t h n u r s i n g , e t c . They are intended to and have been used as a po in t of re fe rence fo r e v a l u a t i o n systems. Lamberton et a l . (287) repor t us ing the ANA standards i n deve -l o p i n g peer rev iew s k i l l s i n a masters l e v e l f a m i l y nurse c l i n i c i a n program. This program prov ides s t r u c t u r e d o p p o r t u n i t i e s f o r nurses f i r s t - 3 2 -to develop pe rsona l comfort through i n c i d e n t a l contac t w i t h b e h a v i o r a l o b j e c t i v e s and through i n f o r m a l g i v i n g and r e c e i v i n g of feedback i n r e -l a t i o n to t h e s e , then to enter a more formal phase of case p r e s e n t a t i o n and group rev iew i n the presence of f a c u l t y , and f i n a l l y to p r a c t i c e one -- t o - o n e unsuperv ised peer rev iew us ing p a t i e n t r e c o r d s . As w i t h those i n s t a n c e s d e s c r i b e d above, the elements of performance to be cons idered are s p e c i f i e d , but standards of acceptab le performance are l e f t i m p l i e d . Th is a r t i c l e would be of i n t e r e s t to any p r o f e s s i o n a l group concerned w i t h deve lop ing a t t i t u d e s and s k i l l s i n e v a l u a t i o n . Gudmunsen (206) a l s o used the ANA Standards to o r g a n -i z e a q u e s t i o n n a i r e to r a t e nurses ' c l i n i c a l performance. The r a t h e r h i g h c o r r e l a t i o n between performance scores on t h i s t o o l and State Board ( l i c e n s i n g exam) scores of study p a r t i c i p a n t s i n d i c a t e s tha t t h i s t o o l may have some v a l i d i t y and perhaps m e r i t s f u r t h e r development and t e s t -i n g , s i n c e i t i s more e a s i l y admin is te red than d i r e c t c l i n i c a l o b s e r v a -t i o n or records r e v i e w . The q u e s t i o n n a i r e ( s e l f and s u p e r v i s o r o p i n i o n ) technique has a l s o been used and r e p l i c a t e d w i t h some degree of success i n n u r s i n g by Dyer ( 1 5 0 , 1 5 2 ) . These l a s t two e v a l u a t i o n approaches seem more s u i t a b l e f o r r e s e a r c h than f o r ongoing e v a l u a t i o n of c l i n i c a l comp-e tence . In d e v i s i n g a way to assess p h y s i c i a n performance i n p e d i a t r i c c a r d i o l o g y , Johnson e t a l . (263) used yet another approach to i d e n t i f y i n g areas of c l i n i c a l performance. Two eminent c a r d i o l o g i s t s , f o r n i n e t y m i n u t e s , d i c t a t e d a l l of the knowledge, s k i l l s , and a t t i t u d e s they e x -pected to see i n a competent p r a c t i t i o n e r . These were l a t e r e d i t e d f o r impor tance , c a t e g o r i z e d , and v a l i d a t e d by med ica l center s t a f f a g a i n s t s i m i l a r statements by the r e l e v a n t s p e c i a l t y med ica l s o c i e t y . The - 3 3 -authors acknowledge s e v e r a l l i m i t a t i o n s i n the t o o l , but found i t u s e f u l i n p r o v i d i n g d i r e c t i o n f o r c u r r i c u l u m p lann ing and c o u n s e l l i n g of med ica l s t u d e n t s , i n t e r n s and c a r d i o l o g y r e s i d e n t s . The t o o l seemed to be acceptab l y s e n s i t i v e . F i n a l l y , Donabedian (144) used h i s t o r i c a l and c o n -temporary views of the p h y s i c i a n r o l e to produce an exhaust i ve l i s t of d e s i r e d p h y s i c i a n behav iors which might be p r o p e r l y e v a l u a t e d . Performance E v a l u a t i o n With E x p l i c i t But Imprecise Standards There appears to have been a t l e a s t some e f f o r t not on ly to develop f a i r l y complete d e s c r i p t i o n s of b e h a v i o r a l elements to be e v a l u a -t e d , but a l s o to develop measurement systems p e r m i t t i n g a statement of standards of a c c e p t a b i l i t y wh ich , w h i l e not e n t i r e l y p r e c i s e , are more e x p l i c i t than i m p l i e d . The Tate Instrument (353) was developed i n the m i d - s i x t i e s and i s s t i l l w ide l y used and a p p a r e n t l y i n f l u e n t i a l as an e v a l u a t i o n a p -p roach . Tate c o l l e c t e d c r i t i c a l i n c i d e n t s of s t a f f nurse behavior which had been recorded by two hundred f o r t y head nurses and c l a s s i f i e d them i n t o f i v e major c a t e g o r i e s : knowledge and judgement, c o n s c i e n t i o u s n e s s , s k i l l i n human r e l a t i o n s , o r g a n i z a t i o n a l a b i l i t y , and o b s e r v a t i o n a l a b i l -i t y . The r a t e r observes and records s i g n i f i c a n t nurse behav io rs ( c r i t i -c a l i n c i d e n t s ) and , us ing these as an i n f o r m a t i o n b a s e , es t imates the e v a l u e e 1 s l e v e l of performance i n each category a g a i n s t a v i s u a l s c a l e which ranges from very unaccepable behav ior to ou ts tand ing b e h a v i o r . The r a t e r i s a s s i s t e d i n t h i s by being prov ided w i t h "benchmark" d e s c r i p t i o n s of nurse b e h a v i o r , arranged a t g iven p o i n t s a long the s c a l e , which are thought to be r e p r e s e n t a t i v e of the behavior at tha t l e v e l . For i n -s t a n c e , the " l e s s than a c c e p t a b l e " benchmark on the human r e l a t i o n s s c a l e -34-might be " t h i s nurse cou ld be expected to ignore p a t i e n t ' s comments regard ing f e a r of what he w i l l say under a n a e s t h e s i a " . Benchmark d e -s c r i p t i o n s are chosen on ly when many nurses agree on t h e i r p lace on the s c a l e . The c r i t i c a l i n c i d e n t s recorded by the eva lua to r are used f o r fo rmat i ve e v a l u a t i o n and then d i s c a r d e d . The completed r a t i n g s c a l e i s r e t a i n e d as par t of the summative e v a l u a t i o n r e c o r d . On t e s t s , r e l i a b i -l i t y v a r i e d from r=29 ( o b s e r v a t i o n a l a b i l i t y ) to r=70 (human r e l a t i o n s ) . The c a t e g o r i e s and b e h a v i o r a l d e s c r i p t i o n s used fo r the s c a l e s are now d a t e d , but the approach i s a p p e a l i n g i n i t s s i m p l i c i t y . The work a p -p a r e n t l y has not been tes ted a g a i n s t other i n s t r u m e n t s . E lenbas and Jacoby (157) used a s i m i l a r approach to develop a b e h a v i o r a l l y based r a t i n g s c a l e fo r pharmacy r e s i d e n t s . The f i n a l v e r s i o n of the inst rument measured twenty-two c h a r a c t e r i s t i c s w i t h i n four broad c a t e g o r i e s (drug i n f o r m a t i o n , p a t i e n t work-up and m o n i t o r i n g , com-mun ica t ion and p r o f e s s i o n a l i n t e r a c t i o n s , s e r v i c e r e s p o n s i b i l i t i e s ) , w i t h s i x t y - e i g h t benchmark b e h a v i o r a l d e s c r i p t i o n being p r o v i d e d . The authors i n d i c a t e that the t o o l i s a p p l i c a b l e to almost any c l i n i c a l pharmacy s i t u a t i o n : there i s no i n f o r m a t i o n as to whether or not the t o o l has been t e s t e d f o r v a l i d i t y or r e l i a b i l i t y . Other measurement s c a l e s have been developed accord ing to f requency of performance, complex i t y of performance, or degree of super -v i s i o n r e q u i r e d . Multnomah County Oregon p u b l i c h e a l t h nurses (261) i d e n t i f i e d ten f a c e t s of i n t e r p e r s o n a l communication, then asked peers to es t imate anonymously accord ing to the frequency, e . g . 20% of the t i m e , w i t h which these behav io rs were demonstrated. R e s u l t s were u s e f u l fo r c o u n s e l l i n g , a l though a c c e p t a b i l i t y of t h i s system was not t o t a l l y p o s i t i v e . R e s u l t s were a l s o used f o r i n p u t , a long w i t h s u p e r v i s o r r a t i n g of p r a c t i c e and s e l f - a s s e s s m e n t , f o r d e c i s i o n s about p romot ion . - 3 5 -B i d w e l l and Froebe (52) desc r ibed l e v e l s of nurse performance w i t h i n s e v e r a l b e h a v i o r a l c a t e g o r i e s on both c l i n i c a l and a d m i n i s t r a t i v e t r a c t s us ing Bloom's taxonomy as a way to organize tasks w i t h i n a c a t e -gory accord ing to i n c r e a s i n g l e v e l s of c o m p l e x i t y . Lower p o s i t i o n s w i t h i n the n u r s i n g s e r v i c e s demanded performance of l e s s complex tasks worth one or two p o i n t s , wh i le h igher p o s i t i o n s demanded more complex a n a l y s i s and s y n t h e s i s behav iors and c a r r i e d h igher p o i n t v a l u e s . A l -though t h i s t o o l r a t h e r d i s t o r t s the meaning and i n t e n t of the taxonomy, n e v e r t h e l e s s a p r o g r e s s i o n i n the complex i t y of s k i l l s i s apparent i n the task l i s t s which are reproduced i n the a r t i c l e . The s c o r i n g system a l l o w s f o r a yes -no response f o r s imple tasks and f o r a g rea te r range of response f o r the more complex t a s k s . I f the r a t e r chooses not to award f u l l po in t va lue ass igned to a g iven t a s k , t h i s d e c i s i o n must be j u s t i -f i e d . The U n i v e r s i t y of Mich igan Med ica l Center (49) used l i t e r a t u r e and job d e s c r i p t i o n s to i d e n t i f y and d e f i n e "major nurse performance f a c t o r s " , and then ass igned to each category three to f i v e l e v e l s of p e r -formance accord ing to the degree of s u p e r v i s i o n r e q u i r e d . Each l e v e l was ass igned po in t v a l u e s . The more independent ly a nurse can perform a g iven f u n c t i o n , the g r e a t e r number of p o i n t s are awarded. To be r a t e d a t a g iven l e v e l , the behav ior desc r ibed i n that l e v e l must be r e g u l a r and r e c u r r i n g i n the n u r s e s ' s p r a c t i c e . The t o t a l p o i n t s earned determines the s i z e of the m e r i t pay i n c r e a s e , and there i s a m i n i m a l l y a c c e p t a b l e l e v e l below which c o u n s e l l i n g , d i s c i p l i n e or t e r m i n a t i o n o c c u r s . Th is i s an i n t e r e s t i n g a p p l i c a t i o n of the m e r i t p r i n c i p l e as a b a s i s f o r d e c i -s i o n s about s a l a r y i n c r e a s e . A f i n a l approach which makes measurement standards more e x p l i -c i t used the nurse r a t e r ' s knowledge of the range of performance p o s s i -- 3 6 -b l e . S l a t e r (220,223) developed an e i g h t y - f o u r i tem s c a l e arranged i n s i x s u b - s e c t i o n s : p s y c h o s o c i a l i n d i v i d u a l , p s y c h o s o c i a l group, p h y s i c a l , g e n e r a l , communicat ion, and p r o f e s s i o n a l e d u c a t i o n . The f i v e p o i n t s c a l e aga ins t which the e v a l u e e ' s a c t i o n i s compared fo r measurement i s the care known by the r a t e r to be g iven by f i r s t l e v e l s t a f f n u r s e s . The r a t e r i s asked to t h i n k of the worst and the best nurses i n t h i s category which she has known. Using these as r e f e r e n c e p o i n t s , she i s then i n -s t r u c t e d to enter the names of nurses she has known a t a l l f i v e p o i n t s on the s c a l e . The inst rument has c o n s i d e r a b l e f l e x i b i l i t y i n that the s c a l e can be expanded to n ine or e leven p o i n t s ; can be used i n any c l i n i c a l s e t t i n g ( there are a number of cue behav io rs which both prov ide concrete examples of behav iors which f a l l w i t h i n each i tem and which can be v a r i e d to s u i t the c l i n i c a l s e t t i n g ) ; and can be used f o r r e t r o s p e c t i v e or c o n -cu r ren t r a t i n g s . In the l a t t e r case a two and o n e - h a l f hour o b s e r v a t i o n per iod i s n e c e s s a r y . The s c a l e has been shown to be s e n s i b l e and s u r -p r i s i n g l y r e l i a b l e . I t i s apparent tha t r e l i a b l i l i t y would s u f f e r w i t h a r a t e r whose past c l i n i c a l exper ience i s l i m i t e d or a t y p i c a l . The QUALPACS Sca le (522) uses the same approach to measurement and employs many of the S l a t e r Sca le i t e m s . QUALPACS, however, i s d e -s igned to measure n u r s i n g care r e c e i v e d by a g iven p a t i e n t or set of p a t i e n t s and thus i s s u i t a b l e i n most i n s t a n c e s on ly f o r judgements about the competence of a s e r v i c e group such as the n u r s i n g u n i t s t a f f . Measurement of i t s s i x t y - e i g h t i tems must be made by a nurse us ing d i r e c t unobt rus ive o b s e r v a t i o n of p a t i e n t s r e c e i v i n g care over a two hour p e r i o d and by i n d i r e c t means such as c h a r t rev iew and d i s c u s s i o n w i t h p e r s o n -n e l . The t o t a l time to rev iew care r e c e i v e d by one p a t i e n t i s th ree h o u r s , but the r a t e r can observe two p a t i e n t s s i m u l t a n e o u s l y . QUALPACS - 3 7 -ach ieves an acceptab le l e v e l of r e l i a b i l i t y and has been tes ted to a l i m i t e d extent fo r v a l i d i t y . An i n t e r e s t i n g independent study by S m o l i n s k i (481) showed that when twenty -e igh t n u r s i n g a c t s were de r i ved from the QUALPACS Scale and submitted to seventy n u r s e - p a t i e n t p a i r s i n two h o s p i t a l s , there were not s i g n i f i c a n t d i f f e r e n c e s i n care as p e r -ce ived by p a t i e n t s compared to judgements of nurses who used t h i s i n s t r u -ment. Both QUALPACS and the S l a t e r Sca le have proven u s e f u l i n i d e n t i f y -ing e d u c a t i o n a l needs , measuring change a f t e r e d u c a t i o n a l or a d m i n i s t r a -t i v e i n t e r v e n t i o n , and i n i d e n t i f y i n g management changes necessary i n the d e l i v e r y of s e r v i c e ( 1 6 2 , 2 2 0 ) . Performance E v a l u a t i o n Us ing E x p l i c i t Standards While e v a l u a t i v e approaches d e s c r i b e d i n t h i s s e c t i o n s t i l l r e l y h e a v i l y on a r a t e r judgement, they are cons t ruc ted so that the behavior to be cons idered and the d e c i s i o n about l e v e l or standard of performance can be made i n a f a i r l y p r e c i s e and r e a d i l y q u a n t i f i a b l e f a s h i o n . The s i m p l i s t of these inst ruments are check l i s t s of what are thought to be the c r i t i c a l p r a c t i t i o n e r b e h a v i o r s . The eva lua to r s imply notes presence or absence of each behav ior , and the l e v e l of performance i s s t a t e d as a composite or percentage of performance i tems e x h i b i t e d . P h y s i c a l t h e r a p i s t s (238) developed such a system fo r use i n peer r e v i e w , and o c c u p a t i o n a l t h e r a p i s t s (216) employed a s i m i l a r a p -p roach . Both groups i d e n t i f i e d weak areas of performance and found t h i s a good base fo r e d u c a t i o n a l p l a n n i n g ; the o c c u p a t i o n a l therapy group repor ted documented improvement i n the q u a l i t y of performance. Rubin (444) d e s c r i b e s one h o s p i t a l ' s exper ience w i t h e v a l u a t i n g - 3 8 -nurse performance us ing e x p l i c i t statements of d e s i r e d behavior f i r s t through the use of c h a r t a u d i t and l a t e r through the use of o ther data c o l l e c t i o n techniques i n c l u d i n g d i r e c t o b s e r v a t i o n , p a t i e n t i n t e r v i e w , and i n c i d e n t r e p o r t r e v i e w s . The index (per cent of requi rements met) serves as an i n d i c a t o r of u n i t achievement; l i k e QUALPACS, t h i s i s p a t i e n t - c e n t e r e d and thus one can eva luate competence of the nurses c o l -l e c t i v e l y who cared fo r that p a t i e n t , but not tha t of an i n d i v i d u a l n u r s e . A l l i n d i c e s of l e s s than 100% are reviewed and t y p i c a l l y l e a d to recommendations f o r both a d m i n i s t r a t i v e change and i n s e r v i c e e d u c a t i o n . Phaneuf 's Nurs ing A u d i t (347 ,390 ,391 ,392 ,393) i s a w i d e l y used c h a r t a u d i t t o o l a p p l i c a b l e to most s e t t i n g s where the p a t i e n t record i s an i n t e g r a l par t of p r o v i d i n g comprehensive and c o n t i n u i n g n u r s i n g care f o r p a t i e n t s . The a u d i t i s a p p l i e d r e t r o s p e c t i v e l y to a ten per cent sample of a l l d ischarged p a t i e n t s or f i f t y c a s e s , whichever i s g r e a t e r , and takes f i f t e e n minutes per c h a r t . The numerous process i tems to be evaluated were developed us ing as a framework seven b a s i c areas of nurse performance o r i g i n a l l y proposed by L e s n i k and Anderson; s c o r i n g o p t i o n s are yes , no , u n c e r t a i n and not a p p l i c a b l e . Each performance area i s weighted by importance ( e . g . " o b s e r v a t i o n " i s twenty per cent of t o t a l s c o r e , " r e p o r t i n g and r e c o r d i n g " i s ten per c e n t , e t c . ) so that t o t a l score r e f l e c t s emphasis on those areas deemed most i m p o r t a n t . A l though i tems are c o n s t r u c t e d to r e f l e c t a broad spectrum of n u r s i n g f u n c t i o n s , data c o l l e c t i o n i s l i m i t e d to i n f o r m a t i o n which can be obta ined from the c h a r t . D e f i c i e n c e s i d e n t i f i e d p e r t a i n to a l l nurses who c o n t r i b u t e d to the c h a r t . The author c a u t i o n s tha t these must be viewed i n the context of the agency 's resources and s e r v i c e p o l i c i e s when c o r r e c t i v e a c t i o n s are being p lanned . - 3 9 -P e l l e y (384) r e p o r t s on the development of a method f o r c o n -cu r ren t rev iew of n u r s i n g care which ,used A b d e l l a h ' s twenty-one problems as an o r g a n i z i n g framework. Geared f o r minimum demand on resources (one — h a l f hour per p a t i e n t ) , the a u d i t rev iews fo r evidence of s p e c i f i c nurse behav io rs i n the care p l a n , record and at the p a t i e n t ' s bedside and can be used on day or evening s h i f t . The a u d i t has been found u s e f u l f o r immediate feedback and i n f l u e n c e on nurse b e h a v i o r . Performance i n d i c e s are computed, and c o n s i s t e n t l y weak areas become top p r i o r i t y fo r i n s e r -v i c e . Ramirez (404) d e s c r i b e s how one h o s p i t a l i n s t i t u t e d a t h i r t e e n i tem a u d i t to assess q u a l i t y of w r i t t e n n u r s i n g care p l a n s . A sample of f i v e p lans per n u r s i n g u n i t every other month was a u d i t e d f o r a q u a l i t a -t i v e index (per cent of a p p l i c a b l e c r i t e r i a met ) . Th is was averaged w i t h a q u a n t i t a t i v e index (per cent of p a t i e n t s on u n i t w i t h p lans ) to y i e l d a u n i t s c o r e . Th is procedure , i n combinat ion w i t h feedback and e d u c a t i o n , has been shown to be e f f e c t i v e i n i n c r e a s i n g the q u a n t i t y and q u a l i t y of n u r s i n g care p l a n s . The H o s p i t a l f o r S i c k C h i l d r e n i n Toronto (526) developed a f a r more s o p h i s t i c a t e d process measuring inst rument fo r n u r s i n g s e r v i c e which s t i l l makes min imal demands on r e s o u r c e s . Th is i n s t i t u t i o n chose to attempt i s o l a t i o n of a few v a r i a b l e s that were h i g h l y i n d i c a t i v e of care i n preference to the usua l route of deve lop ing a comprehensive t o o l where s e v e r a l i tems are f r e q u e n t l y h i g h l y r e l a t e d and prov ide l i t t l e a d d i t i o n a l i n f o r m a t i o n . Development i n v o l v e d us ing expert o p i n i o n to i d e n t i f y c r i t e r i a , and p r a c t i t i o n e r response to v a l i d a t e t h e i r impor tance ; as a l a s t step c r i t e r i a were examined and those found to be redundant , i n v a r i -ant or i r r e l e v a n t were d e l e t e d . A s i x po in t s c a l e was used . When the inst rument (SAVE - S e l e c t e d A t t r i b u t e V a r i a b l e E v a l u a t i o n ) was v a l i d a t e d - 4 0 -a g a i n s t QUALPACS however, i t was found to be o v e r l y generous and not s e n s i t i v e . Items from both QUALPACS and SAVE were sub jected to s e v e r a l c r i t e r i a ( f requency of occurence , s e n s i t i v i t y to range of q u a l i t y and change i n q u a l i t y , a b i l i t y to r e f l e c t a de f ined area of care e t c . ) to y i e l d twenty-two i t e m s . Sub -se ts were then c o n s t r u c t e d , based on c o r -r e l a t i o n s w i t h i n the o v e r a l l i n s t r u m e n t , and a l l s u b - s e t s now show a c o r -r e l a t i o n of .90 w i t h QUALPACS. SAVE r e q u i r e s about o n e - h a l f hour of rev iew time per p a t i e n t and i s repor ted to be p a r t i c u l a r l y e f f e c t i v e when used f o r immediate feedback to and c o u n s e l l i n g of p r a c t i t i o n e r s . Jackson and Smith (254) employed the concept of i n d i c a t o r behav io rs ( s e l e c t e d behav iors thought to p r e d i c t performance f o r a c a t e -gory of behav io r ) to eva luate community pharmacist performance. Compe-tence was d e f i n e d u s i n g f i v e c a t e g o r i e s of pharmacist f u n c t i o n s de r i ved from an American Pharmaceut ica l A s s o c i a t i o n task fo rce r e p o r t on pharma-c i s t s ' r o l e s . Two t r a i n e d "shoppers" and a c l i n i c a l i n v e s t i g a t o r ra ted pharmac is ts ' performance on c r i t e r i a designed to represent the f i v e c a t e -g o r i e s . A l though a composite index of the q u a l i t y of performance cou ld be c a l c u l a t e d , the f a c t that no attempt was made to manipulate the weights of the c r i t e r i a (which v a r i e d c o n s i d e r a b l y i n number f o r the v a r i o u s c a t e g o r i e s ) makes the q u a l i t y index q u e s t i o n a b l e . Never the less t h i s approach might be a h e l p f u l one to i d e n t i f y c o n t i n u i n g educat ion needs of community p h a r m a c i s t s . Systemat ic Assessment of S i t u a t i o n - S p e c i f i c Performance R e c e n t l y , i n c r e a s i n g a t t e n t i o n has been g iven not to whether performance as a whole i s adequate, but r a t h e r to whether performance was - 4 1 -or i s adequate i n r e l a t i o n to a s p e c i f i c c l i n i c a l problem. U s u a l l y the s i t u a t i o n chosen f o r rev iew w i l l be care of p a t i e n t s i n a s p e c i f i c d i a -g n o s t i c category ( e . g . pneumonia), r e c e i v i n g a g i ven procedure ( e . g . laminectomy) , f a l l i n g w i t h i n a g iven age group ( e . g . the h o s p i t a l i z e d t o d d l e r ) , or e x h i b i t i n g a p a r t i c u l a r f u n c t i o n a l problem ( e . g . i m m o b i l -i t y ) . The e v a l u a t i o n study i s u s u a l l y l i m i t e d i n t i m e , ralther than ongo-i n g . T y p i c a l l y a study t o p i c ( e . g . pneumonia) i s chosen, performance c r i t e r i a ( e . g . the p h y s i c i a n w i l l order a f l a t p l a t e of the chest ) are s p e c i f i e d , a l l or a sample of a p p l i c a b l e cases which f a l l w i t h i n the study t o p i c are surveyed f o r compliance w i t h performance c r i t e r i a , and an index (per cent of compl iance w i t h each c r i t e r i o n ) i s computed. Vary ing standards of a c c e p t a b i l i t y w i l l have been s e t , depending on the purpose of the study and the perce ived importance of the c r i t e r i o n . U s u a l l y f o l l o w - u p a c t i o n i s taken to c o r r e c t a l e s s than acceptab le i n d e x , and t h e o r e t i c a l l y a second study i s done to t e s t the e f f e c t i v e n e s s of c o r r e c -t i v e a c t i o n . Th is e v a l u a t i o n - a c t i o n procedure and i t s v a r i a t i o n s are known most f r e q u e n t l y i n the jargon of the h e a l t h sc iences l i t e r a t u r e as a " q u a l i t y assurance program". A l though borrowed as a concept from the f i e l d of i n d u s t r i a l q u a l i t y c o n t r o l , Clement Brown (70,71) i s most o f t e n c r e d i t e d w i t h i t s p o p u l a r i z a t i o n i n h e a l t h . Because of i t s recency and c o n s t a n t l y changing c h a r a c t e r i s t i c s , the l i t e r a t u r e i n t h i s s e c t i o n i s p a r t i c u l a r l y d i f f i c u l t to c a t e g o r i z e i n any meaningful way. One cou ld u s e , as a o r g a n i z i n g concept , a time frame ( p r o s p e c t i v e , c o n c u r r e n t , and r e t r o s p e c t i v e rev iew p rocedures ) ; a source of data frame ( d i r e c t o b s e r v a -t i o n , o p i n i o n based on r e c a l l , c h a r t s and r e c o r d s , reco rd a b s t r a c t s , and payment c l a i m s ) ; purpose (cost c o n t r o l , improvement of q u a l i t y , summative -42-evaluation of quality); type of follow-up action (administrative action, education, counselling, claims rejection, discipline, etc); or the admin-istrative structure which organizes the evaluation of care (taken from simple to complex, these would be individual practice, agencies, and regional or other systems organized to provide health care). None of these organizing frameworks emerges as outstandingly useful or simple. Further, the available literature lacks a common lan-guage and i s sufficiently vague as to defy classification, necessitating, in many cases, an educated guess as to where the particular report f i t s . Since most of these evaluation-action programs are described within the context of a particular administrative structure however, the discussion which follows i s organized in three broad categories: situation-specific process evaluation in the individual practice, situation-specific process evaluation in the agency setting, and situation-specific process evalua-tion in a regional system. Situation-Specific Process Evaluation: Individual Practices Not surprisingly, there is almost no literature available in this category. Until recently, the independent practitioner has been accountable only to an unsophisticated and not very knowledgeable payor and subject to the scrutiny of no one save his patient. Moreover, health practitioners have received l i t t l e encouragement or training in system-atic appraisal of their own practices. Caplan (86) has stated that independent practitioner audit i s both desirable and achievable; the medical profession in particular i s pioneering developments in this area. Lyons and Payne's research study (313) of the degrees of rela-tionships among thirteen diagnostic categories of office care performance of ninety-three primary care physicians i s an early effort at situation-- 4 3 -s p e c i f i c performance e v a l u a t i o n i n the i n d i v i d u a l p r a c t i c e s e t t i n g . Using conformance to predetermined c r i t e r i a as i n d i c a t e d i n p a t i e n t r e c o r d s , mean scores of i n d i v i d u a l p h y s i c i a n performance w i t h i n d i a g n o s -t i c c a t e g o r i e s were computed and c o r r e l a t e d . An o v e r a l l average i n t e r — d i a g n o s t i c c o r r e l a t i o n of p lus .25 was obta ined w i t h c l u s t e r s of c o r r e l -a t i o n s apparent . The i n v e s t i g a t o r s concluded that wh i le there appears to be some homogeneity of performance among d i a g n o s t i c c a t e g o r i e s , t h i s i s not enough to conclude that there are " h i g h and low per formers" or to p rec lude f r u i t f u l c o n t i n u i n g educat ion a c t i v i t y . The Assessment of Performance p r o j e c t (ASPERF) of the American S o c i e t y of I n t e r n a l Medic ine i s an outs tand ing and unique example of e f f o r t on the par t of p r a c t i t i o n e r s c o l l e c t i v e l y to prov ide i n d i v i d u a l p h y s i c i a n s w i t h a t o o l f o r a s s e s s i n g performance us ing s i t u a t i o n - s p e c i f i c performance c r i t e r i a ( 2 1 4 , 2 1 5 ) . The b a s i c format fo r the data c o l l e c t i o n inst rument i s the s e q u e n t i a l s e r i e s of steps used i n c l i n i c a l d e c i s i o n making: h i s t o r y and p h y s i c a l , problem i d e n t i f i c a t i o n , i n v e s t i g a t i v e management, d i a g n o s i s , t reatment f o r s h o r t - t e r m g o a l s , c o n t i n u i n g c a r e , treatment f o r l o n g - t e r m g o a l s , and c o n t r a i n d i c a t e d management.) The p r o j e c t s t a f f has i d e n t i f i e d the p a t i e n t problems and diagnoses most commonly seen by i n t e r n i s t s . For each of these c l i n i c a l s i t u a t i o n s and f o r each step i n the c l i n i c a l d e c i s i o n fo rmat , exper ts have i d e n t i f i e d performance c r i t e r i a c r i t i c a l to that problem. Each c r i t e r i a set i s coded by p a t i e n t problem and by u s i n g the I n t e r n a t i o n a l C l a s s i f i c a t i o n of D i s e a s e s , Adapted (ICDA) or i t s h o s p i t a l c o u n t e r p a r t (HICDA). The ASPERF process i s i n i t i a t e d by the p h y s i c i a n who wishes e v a l u a t i o n of h i s p r a c t i c e ; s t a f f survey h i s records to see i f they are s u f f i c i e n t l y complete to use f o r e v a l u a t i o n and to see i f h i s p r a c t i c e -44-p r o f i l e i n d i c a t e s s u f f i c i e n t c r i t e r i a se ts are a v a i l a b l e to sample that p r a c t i c e . I f these requi rements are met, records are surveyed by ASPERF s t a f f us ing the e s t a b l i s h e d c r i t e r i a s e t s . The p r a c t i t i o n e r i s then g iven feedback about h i s performance, guidance i n p lanning c o r r e c t i v e or e n r i c h i n g c o n t i n u i n g e d u c a t i o n , and an appointment f o r reassessment . ASPERF i s as yet unproved as a p r o f e s s i o n a l development program. Never -t h e l e s s , i t i s promis ing s e l f - h e l p a i d f o r the i n t e r n i s t and a model which other p r o f e s s i o n a l groups cou ld p r o f i t a b l y examine. There have been two s t u d i e s which cou ld c o n t r i b u t e to e v a l u a -t i o n i n the i n d i v i d u a l p r a c t i c e s e t t i n g . One d i s t r i c t of the American P s y c h i a t r i c A s s o c i a t i o n i n C a l i f o r n i a surveyed the p r a c t i c e s of i t s members to e s t a b l i s h t reatment norms l o c a l l y by d i a g n o s i s , age, sex e t c . (290). The i n v e s t i g a t o r s f e e l that reco rds -based e v a l u a t i o n us ing r e -g i o n a l norms a l l o w s f o r c o n f i d e n t i a l i t y to be mainta ined and permits the necessary range of treatment m o d a l i t i e s to be i n c l u d e d . The p r o j e c t seems more d e s t i n e d to prov ide data f o r c l a i m s rev iew than for sound a p p r a i s a l of p r a c t i c e however. A more promis ing p r o j e c t i s that undertaken by a consort ium of e i g h t American med ica l o r g a n i z a t i o n s to s e l e c t and v a l i d a t e performance c r i t e r i a fo r h e a l t h s u p e r v i s i o n of c h i l d r e n i n four age groups and f o r ambulatory management of u r i n a r y t r a c t i n f e c t i o n , b r o n c h i a l asthma, and t o n s i l l o p h a r y n g i t i s . The s e v e r a l d e t a i l e d r e p o r t s of t h i s a m i b i t i o u s p r o j e c t (327,373,499) are worth pursuing because of the l e s s o n s to be learned from the methodology used . A f t e r e s t a b l i s h i n g an i n i t i a l l i s t of performance c r i t e r i a , a n a t i o n a l panel of three hundred e i g h t y - e i g h t expert academicians and p r a c t i t i o n e r s evaluated the i t e m s ' re levance to outcome and a c c e p t a b i l i t y f o r peer and s e l f - a s s e s s m e n t . The same c r i t e r -- 4 5 -i a were then tes ted i n a broad spectrum of p r a c t i c e s e t t i n g s . Some of the important l e s s o n s learned were tha t many of the c r i t e r i a s e l e c t e d cou ld not be supported by c i t i n g r e s e a r c h , that p h y s i c i a n s c o n s i s t e n t l y overest imated the amount of documentation present i n p a t i e n t r e c o r d s , that an e l a b o r a t e v a l i d a t i o n process d i d l i t t l e to improve c r i t e r i a , and that there was a h i g h l e v e l of agreement between academicians and p r a c t i -t i o n e r s . S i t u a t i o n - S p e c i f i c Process E v a l u a t i o n : Agency S e t t i n g s Many genera l d e s c r i p t i o n s or i n t e r p r e t a t i o n s of agency-based " q u a l i t y assurance" are a v a i l a b l e i n the l i t e r a t u r e . Most are s imply e n t h u s i a s t i c r e - e x p l a n a t i o n s of the same p r o c e s s , o c c a s i o n a l l y w i t h some embel l i shment . Dav idson 's i n t e r p r e t a t i o n (118) i s t y p i c a l i n i t s t reatment of the s u b j e c t . He recommends a f i v e step process of p a t i e n t care a p p r a i s a l i n which an agency committee: 1. chooses a study t o p i c . 2 . develops process c r i t e r i a , i . e . d e s i r a b l e a c t i o n s the p r a c t i t i o n e r should taken when faced w i t h the problem. 3 . measures the degree of compliance w i t h the c r i t e r i a i n a sample of c a s e s . 4 . ana lyzes r e s u l t s , then p lans and takes a c t i o n to i n c r e a s e compliance i f t h i s i s not a c c e p t a b l e . 5 . measures degree of compl iance aga in to see i f the d e s i r e d change a c t u a l l y o c c u r r e d . A l a r g e s c a l e e f f o r t to encourage adopt ion of t h i s process i n Washington State was mounted i n the e a r l y sevent ies as a c o n t i n u i n g e d u -c a t i o n p r o j e c t which i n v o l v e d cooperat i ve e f f o r t by s e v e r a l o r g a n i z a t i o n s - 4 6 -i n a m u l t i - f a c e t e d program i n v o l v i n g w r i t t e n and a u d i o v i s u a l m a t e r i a l s , workshops and c o n s u l t a t i o n s (117) . I t was hoped tha t the process would be both e d u c a t i o n a l i t s e l f , e s p e c i a l l y dur ing step 2 , and would i d e n t i f y v a l i d c o n t i n u i n g educat ion needs . Del Bueno (124) emphasizes the i m p o r t -ance of c o n s i d e r i n g i n step 4 the cos t i m p l i c a t i o n s of the v a r i o u s a l t e r -n a t i v e s fo r a c t i o n . Dohner (139) e l a b o r a t e s on step 4 by i n s i s t i n g t h a t any a c t i o n to c o r r e c t d e f i c i t s should at the l e a s t i n c l u d e feedback of r e s u l t s to a l l s t a f f , p r o v i s i o n fo r d i s c u s s i o n among s t a f f , and agreement on s p e c i f i c recommendations f o r a c t i o n . S lee (478) suggests tha t s tep 3 should be a screen ing dev i ce o n l y and that step 4 w i l l be most p r o f i t a b l y accompl ished i f d e v i a t i o n s from the d e s i r e d behaviour are analyzed on a case by case b a s i s . A t ime frame should be used and d e a d l i n e s f o r c o r r e c t i o n of d e f i c i t s imposed. He a l s o prov ides a check l i s t f o r admins t ra to rs to use i n order to d e t e r -mine whether t h e i r q u a l i t y c o n t r o l systems are adequate. In a separate a r t i c l e (479) he p o i n t s out that t h i s process can be a p p l i e d i n four d i f f e r e n t types of rev iew : c l a i m s rev iew of i n d i v i d u a l cases to d e t e r -mine m e d i c a l l y j u s t i f i a b l e s e r v i c e s , concur rent or u t i l i z a t i o n rev iew of i n d i v i d u a l cases to determine j u s t i f i a b l e s e r v i c e s , r e t r o s p e c t i v e c h a r t rev iew of homogenous groups of p a t i e n t s to compare t y p i c a l s e r v i c e g i ven w i t h standards set by s t a f f , and r e t r o s p e c t i v e rev iew of i n d i v i d u a l cases tha t d e v i a t e from e s t a b l i s h e d s tandards . He b e l i e v e s the purpose of these l a s t two are p r i m a r i l y e d u c a t i o n a l , w h i l e the f i r s t two are u s e f u l p r i m a r i l y fo r c o s t c o n t r o l . The s i m i l a r i t i e s i n a l l these approaches stem from the p o s i t i o n of the J o i n t Commission on A c c r e d i t a t i o n of H o s p i t a l s i n promoting q u a l -i t y assurance through a c c r e d i t a t i o n requ i rements . JCAH standards are - 4 7 -d i r e c t e d toward q u a l i t y p r o t e c t i o n through r e t r o s p e c t i v e med ica l a u d i t , c o n t i n u i n g medica l educat ion based on a u d i t , u t i l i z a t i o n rev iew and p r o -v i s i o n f o r the c r e d e n t i a l s committee to have access to a u d i t f i n d i n g s (398) . In a d d i t i o n to genera l d i s c u s s i o n s about q u a l i t y assurance us ing process c r i t e r i a , there are numerous d e s c r i p t i o n s of the method as a p p l i e d i n the s p e c i f i c agency s e t t i n g . Shaw (471) r e p o r t s on two of t h i r t e e n demonstrat ion p r o j e c t s . In one h o s p i t a l dramat ic changes i n med ica l and a l l i e d therapy of s t roke p a t i e n t s have occurred as a r e s u l t of d e f i c i e n c i e s uncovered by med ica l a u d i t . In the o t h e r , l e s s s p e c t a -c u l a r changes i n documentation of t reatment p r a c t i c e s have r e s u l t e d from q u a l i t y assurance procedures . Hol loway e t a l . (243) d i s c u s s us ing a m o d i f i e d nominal g roup/delph i technique to develop h o s p i t a l l e v e l s - o f -care c r i t e r i a fo r use i n u t i l i z a t i o n r e v i e w . Tom et a l . (503) r e p o r t us ing head nurses to screen fo r d e v i a t i o n s from u t i l i z a t i o n rev iew c r i t e r i a and paying p h y s i c i a n s to rev iew t h e s e . They c l a i m that the system has decreased l e n g t h of s t a y , prov ided e d u c a t i o n a l b e n e f i t i n that rev iewers g a i n i n s i g h t i n t o how t h e i r c o l l e a g u e s are doing t h i n g s , and g i ven p h y s i c i a n s an o p p o r t u n i t y to d i s c u s s cases w i t h a rev iewer a t no e x t r a c o s t . Dev i t and I rons ide ( 1 3 2 ) , on the other hand, present two s t u d i e s (mastectomy and cholecystectomy) i n which no improvement i n performance cou ld be n o t e d . They a l s o suggest c a u t i o n i n i n t e r p r e t i n g changes i n performance s i n c e many f a c t o r s other than a u d i t - b a s e d educat ion may a f f e c t r e s u l t s of r e e v a l u a t i o n . E t h r i d g e and Packard (160) d e s c r i b e i n c o n s i d e r a b l e d e t a i l , an a u d i t system which uses standard n u r s i n g care plans fo r one hundred d i a g n o s t i c c a t e g o r i e s . Bas ic care f low sheets (work records ) are the - 4 8 -pr imary source of documentat ion . C l e r k s count the number of t imes a n u r s i n g a c t i o n was performed and c a l c u l a t e an index based on the number of t imes i t should have been performed. If a m i n i m a l l y acceptab le score i s not met, those records not complying w i t h the standards are e x t r a c t e d f o r r e v i e w . A l l data can be summarized by n u r s i n g u n i t , by med ica l c a t e -g o r y , by care p l a n , by process i t e m , e t c . The authors c l a i m use of t h i s system f a c i l i t a t e s a u d i t , s i m p l i f i e s s h i f t r e p o r t s , reduces nurses n o t e s , improves o r i e n t a t i o n fo r new s t a f f , and prov ides a more c o n s i s t e n t approach to n u r s i n g c a f e . Graydon (202) eva luated the care i n f i v e exper imenta l and two c o n t r o l h o s p i t a l s us ing f i v e process c r i t e r i a i n f i v e d i a g n o s t i c c a t e g o r -i e s f o r a year p r i o r to i n s t i t u t i n g feedback, fo r a year of feedback to the exper imenta l h o s p i t a l s , and f o r s i x months t h e r e a f t e r . I t was found that most b e h a v i o r changes i n both groups were t r a n s i t o r y and the few p e r s i s t e n t changes that occurred d i d not vary between the study group and c o n t r o l h o s p i t a l s . H i s c o n c l u s i o n was tha t feedback i n t h i s i n s t a n c e d i d not improve performance. However, defendants of the q u a l i t y assurance procedure w i l l be q u i c k to po in t out tha t step 4 , c a r e f u l a n a l y s i s of d e v i a t i o n and use of a p p r o p r i a t e c o r r e c t i v e a c t i o n , was l a r g e l y ignored and these r e s u l t s were t h e r e f o r e q u i t e p r e d i c t a b l e . There i s support f o r t h i s po in t of v i e w . In d i s c u s s i n g the a p p l i c a t i o n of process a u d i t i n a Veterans A d m i n i s t r a t i o n H o s p i t a l and a r e h a b i l i t a t i o n h o s p i t a l , Ashbaugh and McKean (31) conclude that the vas t m a j o r i t y of d e f i c i e n c i e s are a s s o c i a t e d w i t h performance r a t h e r than knowledge problems, and t h a t departmental d e c i s i o n i s necessary to determine whether feedback a lone or w i th some other form of c o r r e c t i v e a c t i o n w i l l be n e c e s s a r y . Richman and P i n s k e r (421) d e s c r i b e a s imple a u d i t procedure - 4 9 -which depends on manual e x t r a c t i o n of data from records and a m u l t i -d i s c i p l i n a r y d i s c u s s i o n of the i d e n t i f i e d pa t te rns of c a r e . Recommenda-t i o n s a r i s i n g from these conferences have been shown to improve the q u a l i t y of c a r e . Agencies which prov ide ambulatory care have repor ted l e s s a c t i -v i t y than have h o s p i t a l s . Thompkins (498) d e s c r i b e s e f f o r t s to develop a system f o r a u d i t i n g the a c t i v i t i e s of p h y s i c i a n a s s i s t a n t s us ing c l i n i c a l a l g o r i t h m s ( p r o t o c o l s which d i c t a t e a s e r i e s of s e q u e n t i a l a c t i o n s , each step dependent on the outcome of the prev ious one ) . To date these have been developed fo r e leven e a s i l y managed pr imary care problems. Computer a n a l y s i s of the completed p r o t o c o l shows the p h y s i c i a n a s s i s t a n t h i s e r r o r s i n f o l l o w i n g the a l g o r i t h m , and i t has been demonstrated that p e r -formance improves a f t e r feedback. Other c l i n i c s r e p o r t us ing these problem o r i e n t e d p r o t o c o l s fo r the cont inuous mon i to r ing of paramedics ' performance a l s o ( 5 , 2 8 4 ) . The U n i v e r s i t y of Vermont Hea l th Care Cente r , i n a unique v a r i -a t i o n of the b a s i c f i v e step process q u a l i t y assurance system, uses p a t i e n t p a r t i c i p a t i o n i n the a u d i t as a way of widening the data base beyond the t r a d i t i o n a l records rev iew ( 5 7 ) . The Harvard Community Hea l th P l a n , w i t h a membership of four thousand, employs a t o t a l l y computer ized records system ( 4 0 ) . The s i x t y p h y s i c i a n s and f o r t y - f i v e nurse p r a c t i -t i o n e r s use the Computer Stored Records System to both enter and r e t r i e v e a l l d e s i r e d p a t i e n t d a t a . S ince standards of care by d i a g n o s i s and p rob -lem are a l s o s t o r e d , the p r a c t i t i o n e r r e c e i v e s immediate i n f o r m a t i o n i f h i s d e c i s i o n has dev ia ted from,agreed upon a c t i o n . A f t e r i n s t a l l a t i o n of t h i s immediate feedback system, p r a c t i t i o n e r behavior i n management of beta hemoly t ic s t r e p t o c o c c a l i n f e c t i o n s changed i n the d e s i r e d d i r e c t i o n . - 5 0 -t i o n . Randomized t r i a l s w i t h c o n t r o l s are planned as other standards are deve loped. A computer ized records system has a l s o been shown to be u s e -f u l f o r rev iew and improvement of care i n a community mental h e a l t h c l i n i c ( 5 0 1 ) . Diamond et a l . (135) d e s c r i b e a peer rev iew and feedback system i n a d e c e n t r a l i z e d community mental h e a l t h center wh ich , when l i n k e d to c o n t i n u i n g med ica l e d u c a t i o n , i n f l u e n c e d s h i f t s from polypharmacy toward more a p p r o p r i a t e use of s p e c i f i c p s y c h o t r o p i c m e d i c a t i o n s . Russo e t a l . (448) developed a system fo r r a p i d rev iew of ambulatory p a t i e n t records us ing sets of process c r i t e r i a (about f i v e c r i t e r i a per c o n d i t i o n ) f o r t w e n t y - s i x common c o n d i t i o n s . Of three hundred t h i r t y - s e v e n records s e l -ected f o r r e v i e w , 25% were not rev iewab le and , of the remain ing 75%, v i r t u a l l y o n e - h a l f were d e f i c i e n t i n one or more c r i t e r i a . P r a c t i t i o n e r compl iance ( the t a r g e t group was p r i v a t e p h y s i c i a n s , house s t a f f , m e d i c a l s tudents and nurse p r a c t i t i o n e r s ) cou ld not be c o n c l u s i v e l y shown to improve dur ing the f i v e month study p e r i o d . The o n l y c o r r e c t i v e a c t i o n used was p r i v a t e feedback and d i s c u s s i o n . Using t h i s system i t was p o s s i b l e fo r sen io r p e d i a t r i c i a n s and the record l i b r a r i a n to rev iew one c h a r t each m i n u t e . Skipper (447) r e p o r t s tha t i n c r i t e r i a such as cho ice of medic ine and use of s p e c i a l i s t c o n s u l t a n t s , behavior improved when the peer group a u d i t e d med ica l care i n a f a m i l y medic ine c l e r k s h i p . S i t u a t i o n - S p e c i f i c Process E v a l u a t i o n : Reg iona l Systems G e n e r a l l y , r e g i o n a l rev iew systems have t h e i r o r i g i n s i n t h i r d par ty payment o r g a n i z a t i o n s . P r i o r to the sevent ies both Medicare and Blue Cross and Blue S h i e l d P l a n s pioneered c l a i m s r e v i e w . These e a r l y e f f o r t s were aimed p r i m a r i l y a t a v o i d i n g unnecessary s e r v i c e s and not a t q u a l i t y c o n t r o l . - 5 1 -One e x c e p t i o n to t h i s e a r l y p a t t e r n of r e g i o n a l rev iew was a study of p h y s i c i a n behav io r (382) which was i n i t i a t e d a t the request of the Hawai i M e d i c a l A s s o c i a t i o n i n 1968 and completed i n 1971. The study c o n s i s t e d of four p a r t s : an episode of i l l n e s s study i n which p h y s i c i a n panels developed d i a g n o s i s - s p e c i f i c " o p t i m a l " management c r i t e r i a f o r p r e , d u r i n g , and post h o s p i t a l care and weighted these fo r impor tance ; an o f f i c e care study i n which c r i t e r i a were chosen f o r t h e i r importance i n management of ambulatory p a t i e n t s ; a h o s p i t a l o r g a n i z a t i o n q u e s t i o n n a i r e ; and a c o n t i n u i n g educat ion s tudy . The h o s p i t a l and o f f i c e study combined twenty-one d i a g n o s t i c c a t e g o r i e s i n twenty-two Hawai i h o s p i t a l s and i n the o f f i c e p r a c t i c e s of a l l i n v o l v e d p h y s i c i a n s . The weighted c r i t e r i a made p o s s i b l e an e x p r e s s i o n of scores as a p h y s i c i a n performance index (PPI) i n terms of per cent of c r i t e r i a met. Th is data cou ld be repor ted by p h y s i c i a n , by h o s p i t a l , by d i a g n o s t i c c a t e g o r y , and by c r i t e r i o n . Some of the more s i g n i f i c a n t f i n d i n g s were that a p p r o p r i a t e admiss ions were a t a h i g h r a t e , but l e n g t h s of s tay were g e n e r a l l y l e s s a p p r o p r i a t e ; s p e c i a l i s t s c a r i n g f o r p a t i e n t s i n t h e i r area of e x p e r t i s e scored b e t t e r than other p h y s i c i a n s , but t h i s was not r e l a t e d to Board c e r t i f i c a t i o n ; numbers of years of p r a c t i c e d i d not i n f l u e n c e s c o r e s ; and a f t e r two years of e d u c a t i o n a l e f f o r t s i n four h o s p i t a l s u s i n g e i g h t d i a g n o s t i c c a t a g o r i e s , a remeasure of PPI showed d e s i r a b l e , i f somewhat uneven behav ior change i n some but not a l l d i a g n o s t i c c a t e g o r i e s and h o s p i t a l s . The i n v e s t i g a t o r s concluded that a r e g i o n a l approach to p h y s i c i a n p e r f o r -mance rev iew i s u s e f u l because one can take a more g l o b a l approach to h e a l t h problems. I t has been s t a t e d tha t a number of c l a i m s rev iew o r g a n i z a -t i o n s e x i s t e d p r i o r to the s e v e n t i e s . In a d d i t i o n to p r i v a t e and p u b l i c - 5 2 -h e a l t h i n s u r o r s , groups of p h y s i c i a n s formed foundat ions i n an e f f o r t to demonstrate t h a t p h y s i c i a n s cou ld o f f e r q u a l i t y med ica l care a t a r e a s o n -ab le cost wi thout i n t e r v e n t i o n of non -medica l p a r t i e s . Many of these rev iew o r g a n i z a t i o n s r e c e i v e d f i n a n c i a l a s s i s t a n c e fo r development of e v a l u a t i o n systems when the Un i ted S t a t e s government prov ided funds f o r Exper imenta l M e d i c a l Care Review O r g a n i z a t i o n s (EMCRO's) to develop work-i n g models f o r sys temat ic and ongoing rev iew of med ica l care ( 3 5 4 ) . I t was s t i p u l a t e d tha t each p r o j e c t would be based on e x p l i c i t c r i t e r i a and standards and would i n c o r p o r a t e rev iew f i n d i n g s i n l o c a l c o n t i n u i n g e d u -c a t i o n programs. Twelve o r g a n i z a t i o n s were funded, i n c l u d i n g the Hawai i M e d i c a l A s s o c i a t i o n and some of the e x i s t i n g f o u n d a t i o n s ; some p r i v a t e insurance companies were i n c l u d e d as coopera t ing o r g a n i z a t i o n s . One of these f o u n d a t i o n s , the San Joaquin Foundat ion f o r M e d i -c a l Care , r e p o r t s tha t a f t e r m o n i t o r i n g the p r e s c r i b i n g and d i s p e n s i n g p a t t e r n s of p h y s i c i a n s and p h a r m a c i s t s , a 12.1% drug expendi ture sav ings was documented and changes i n p h y s i c i a n behavior were ach ieved ( 4 9 2 ) . The pr imary mode of c o r r e c t i v e a c t i o n taken w i t h d e v i a n t p r a c t i t i o n e r s was persona l l e t t e r s and phone c a l l s , c l a i m s d e n i a l be ing used w i t h o n l y the most r e c a l c i t r a n t . Under EMCRO f u n d i n g , t h i s foundat ion a l s o i n s t i t u t e d a m e d i c a l peer rev iew back-up system f o r c l a i m s adjustment ( 7 4 ) . For t h i r t e e n of f i f t e e n procedures ( s i x types of i n j e c t i o n s , s i x types of p h y s i c i a n v i s i t s , and three l a b t e s t s ) a s t a t i s t i c a l l y s i g n i f i c a n t , though non -u n i f o r m , r e l a t i o n s h i p was found between the per cent of b i l l i n g c l a i m s ad jus ted and subsequent changes i n p r a c t i c e p a t t e r n s as measured by decreases i n the monthly numbers of these s e r v i c e s be ing c l a i m e d . The p h y s i c i a n - s e r v i c e d rev iew procedure accounted f o r 15% of d o l l a r amounts - 5 3 -of c l a i m s adjustments and more than paid fo r i t s e l f , but the bu lk of d o l l a r s ad jus ted (85%) were a d m i n i s t r a t i v e ( c l e r i c a l ) r e v i e w . The author of t h i s r e p o r t concluded that peer rev iew may c o n t r o l q u a l i t y more than c o s t . A l s o , s i n c e r e l a t i o n s h i p s between ad justed c l a i m s and changed p r a c t i c e behav io rs were not p e r f e c t , there c l e a r l y are l i m i t s to which c l a i m s rev iew w i l l e f f e c t p r a c t i c e p a t t e r n s . K a i s e r Permanente, as a comprehensive group h e a l t h care founda -t i o n , i s concerned w i t h both ambulatory and h o s p i t a l c a r e . In K a i s e r ' s rev iew program, t r a i n e d l a y rev iewers screen a l l c l a i m s a g a i n s t e s t a b l i -shed d i a g n o s i s - s p e c i f i c performance c r i t e r i a . Approx imately 85% of a l l c l a i m s go through t h i s screen and the remaining 15% are r e f e r r e d to a p h y s i c i a n rev iewer who may s e t t l e the c l a i m or r e f e r to a committee of p h y s i c i a n s . M e d i c a l rev iewers have access to a computer generated p h y s i c i a n p r o f i l e and p a t i e n t h i s t o r y to a s s i s t i n making d e c i s i o n s . H a r r i n g t o n , i n d i s c u s s i n g t h i s procedure , s t a t e s t h a t more e f f i c i e n c y may be a t t a i n e d i n l e t t i n g smal l c l a i m s go th rough . ( 2 1 8 ) . The New Mexico peer rev iew system, another EMCRO p r o j e c t which covers both i n p a t i e n t and ambulatory c a r e , has been d e s c r i b e d e x t e n s i v e l y ( 6 6 ) . Th is study reached a number of s i g n i f i c a n t c o n c l u s i o n s about the e f f e c t of p h y s i c i a n peer r e v i e w , us ing process c r i t e r i a , on the c o s t and q u a l i t y of h e a l t y c a r e : 1. p h y s i c i a n involvement i n rev iew had a major impact on q u a l i t y of c e r t a i n aspects of care but d i d not reduce c o s t s ; 2 . net sav ings to the Medica id program were brought about by the a d m i n i -s t r a t i v e a c t i v i t i e s of the f i s c a l i n t e r m e d i a r y i n coopera t ion w i t h the peer rev iew o r g a n i z a t i o n ; 3 . peer rev iew of ambulatory care based on computer ized data from c la ims forms i s f e a s i b l e ; 4 . peer rev iew of ambulatory c a r e , i n most c a s e s , w i l l i n c r e a s e the use of s e r v i c e s and - 5 4 -thus i n c r e a s e c o s t s ; 5 . c o o p e r a t i o n of the f i s c a l i n t e r m e d i a r y and peer rev iew o r g a n i z a t i o n suggested that sav ings from the former a c t i v i t y might d e f r a y the h igher expend i tures brought about by peer rev iew e f f o r t s ; 6 . Medica id saved money by r e c l a s s i f y i n g n u r s i n g home p a t i e n t s to lower l e v e l s of c a r e , but t h i s approach should be eva luated to see that humane-ness of care was not s a c r i f i c e d ; 7. l e n g t h of s tay approach to c o n t r o l -l i n g h o s p i t a l c o s t s was i n e f f e c t i v e ; 8 . a l e v e l of care approach might be t r i e d f o r c o n t r o l l i n g h o s p i t a l c o s t s , p r o v i d i n g i t i s found to be s u c -c e s s f u l f o r n u r s i n g homes. References to q u a l i t y improvement i n the New Mexico study are made p r i m a r i l y on the b a s i s of a p p r o p r i a t e use of i n j e c t i o n s , which improved d r a m a t i c a l l y dur ing the study p e r i o d . This was thought to be brought about f i r s t through e d u c a t i o n a l methods a n d , where t h i s f a i l e d , c l a i m s d e n i a l . Changed behav ior towards more a p p r o p r i a t e use of i n j e c t -i ons was a l s o a s s o c i a t e d w i t h group p r a c t i c e , w i t h board c e r t i f i c a t i o n and be ing a p e d i a t r i c i a n , and w i t h be ing a doctor of medic ine ( 6 5 ) . When one examines the c r i t e r i a a c t u a l l y i n use , the c l a i m s regard ing improved q u a l i t y of med ica l care i n genera l seem somewhat o v e r s t a t e d . Th is r e p o r t (66) a l s o reviews a number of o ther prev ious or concurrent cost c o n t r o l programs. The authors conclude that f i g u r e s showing c o s t ga ins are based on tenuous reason ing and inadequate data and that t h e r e f o r e more work must be done be fo re g e n e r a l i z a t i o n s about the e f f e c t s of these c l a i m s rev iew programs can be made. Another EMCRO p r o j e c t was developed by the Utah P r o f e s s i o n a l Review O r g a n i z a t i o n (UPRO). In f a c t , three separate systems ( f o r h o s p i -t a l , neighborhood h e a l t h c e n t e r , and ambulatory care) of rev iew are i n o p e r a t i o n . The ambulatory rev iew program has r e c e i v e d the most a t t e n t i o n - 5 5 -i n the l i t e r a t u r e ( 3 5 5 , 3 5 7 , 3 5 8 , 3 7 3 ) . Process or d e s i r e d p a t i e n t manage-ment c r i t e r i a are developed fo r p rocedures , f o r common diagnoses and f o r drug t h e r a p i e s . S p e c i f i e d c o n t r a i n d i c a t i o n s and p o s s i b l e omiss ions are a l s o s t a t e d . About n i n e t e e n thousand c la ims per week are screened w i t h one thousand c l a i m s needing p h y s i c i a n r e v i e w . Th is i s done w i t h i n the context of a computer generated p a t i e n t h i s t o r y . A summary sheet a l l o w s the rev iewer to make one or more of a number of recommendations. Feed -back to the dev iant p rov ider i s graded from an i n q u i r i n g l e t t e r to i n f o r -mation to warning to d e n i a l of payment. Aggregate data about d e v i a t i o n s from d e s i r e d c r i t e r i a are used by the Utah Academy f o r Cont inu ing M e d i c a l E d u c a t i o n . The p r o j e c t d i r e c t o r s b e l i e v e t h i s approach permits o b j e c t i v e and s u b j e c t i v e judgements about p h y s i c i a n competence. I t a l s o p rov ides i n f o r m a t i o n about abuse of the payment system by p a t i e n t s . A l though c l a i m s can be d e n i e d , the major t h r u s t has been e d u c a t i o n a l , and t h i s has been s u c c e s s f u l . Conc lus ions have not been reached about cos t e f f e c t s . UPRO's h o s p i t a l rev iew program has a l s o a t t r a c t e d some a t t e n t i o n because i t employs nurse rev iewers to screen h o s p i t a l admiss ions a g a i n s t p r e - e s t -a b l i s h e d c r i t e r i a , dec reas ing the r e q u i r e d amount of more expensive p h y s i c i a n time ( 3 7 3 ) . A l l f i n a l d e c i s i o n s which run counter to the a d m i t t i n g p h y s i c i a n ' s wishes are made by a p h y s i c i a n who d i s c u s s e s t h i s w i t h the a d m i t t i n g d o c t o r ; there i s p r o v i s i o n fo r appeal to a committee of p e e r s . Confus ion as to whether c l a i m s rev iew programs us ing process c r i t e r i a c o n t r o l c o s t and/or q u a l i t y i s n o t i c e a b l e i n the l i t e r a t u r e to d a t e . In another a r t i c l e on t h i s s u b j e c t , Sayetta (454) ques t ions on both methodo log i ca l and i n t e r p r e t i v e grounds e a r l i e r f i n d i n g s and c o n c l u -s i o n s about the e f f e c t i v e n e s s of yet another EMCRO p r o j e c t , the C e r t i f i e d - 5 6 -H o s p i t a l Admiss ion Program which was o r i g i n a t e d by the Med ica l Care Foundat ion of SacramentOj C a l i f o r n i a . These r e p o r t s and o thers c h a l l e n g -i n g the f i n d i n g s of the EMCRO and s i m i l a r p r o j e c t s r a i s e s e r i o u s q u e s t -ions as to whether a sound e x p e r i e n t i a l base e x i s t e d fo r the n a t i o n a l PSRO l e g i s l a t i o n which fo l l owed the EMCRO p r o j e c t s . In 1972 Congress amended the S o c i a l S e c u r i t y Act to prov ide f o r the es tab l i shment of P r o f e s s i o n a l Standards Review O r g a n i z a t i o n s (PSRO's ) . Based p r i m a r i l y on e a r l i e r exper ience w i t h EMCRO p r o j e c t s , t h i s law has i n f l u e n c e d profoundly the r a t e and d i r e c t i o n of deve lop ing performance e v a l u a t i o n systems i n a l l o f the p r o f e s s i o n s , but none more markedly than American m e d i c i n e . A number of a u t h o r i t a t i v e (515,516,518) and i n t e r p r e t i v e (45 ,222 ,350 ,490) r e f e r e n c e s e x i s t to e x p l a i n t h i s l a w . B a s i c a l l y the law prov ides f o r the fo rmat ion of r e g i o n a l o r g a n i z a t i o n s of p h y s i c i a n s (PSRO's) which w i l l rev iew care paid through f e d e r a l l y funded Med ica re , M e d i c a i d , or Materna l C h i l d H e a l t h Programs to determine that the h e a l t h s e r v i c e prov ided i n shor t s tay f a c i l i t i e s was m e d i c a l l y n e c e s -s a r y , c o n s i s t e n t w i t h p r o f e s s i o n a l l y recognized c r i t e r i a of c a r e , and cou ld not have been prov ided more e c o n o m i c a l l y . PSRO's may a l s o r e v i e w ambulatory and extended care s e r v i c e s . Each PSRO i s expected to e s t a -b l i s h r e g i o n a l care c r i t e r i a and recommend award or d e n i a l of payment. PSRO's must rev iew h e a l t h s e r v i c e s prov ided by other h e a l t h workers i n c o n s u l t a t i o n w i t h those workers . In a d d i t i o n to the concerns expressed regard ing the e f f e c t i v e -ness of the EMCRO's which served as prototypes fo r the PSRO l e g i s l a t i o n , the law has a l s o been c r i t i c i z e d on other grounds: i t r e q u i r e s a phys ic ian -dominated rev iew p r o c e s s , and t h e r e f o r e may perpetuate the p r a c t i c e of c r i s i s - o r i e n t e d h e a l t h c a r e ; d o c t o r - p a t i e n t c o n f i d e n t i a l t y i s - 5 7 -th reatened ; p h y s i c i a n reviewers may be sub jec t to defamat ion s u i t s ; the rev iew process encourages "cookbook" m e d i c i n e ; and i t s implementat ion r e q u i r e s an a d m i n i s t r a t i v e s t r u c t u r e that may be a b u r e a u c r a t i c n i g h t -mare. Which of these f e a r s are imagined and which are r e a l remain to be seen . S u l l i v a n (490) r e p o r t s c o n s i d e r a b l e r e s i s t a n c e to the law from American d o c t o r s , however i t i s q u i t e c l e a r that g r a d u a l l y p h y s i c i a n s i n most s e c t i o n s of the count ry have p a r t i c i p a t e d i n the fo rmat ion of PSRO's . Many of the former EMCRO's have been des ignated PSRO's f o r t h e i r r e g i o n . F r e d e r i c k (170) d e s c r i b e s i n d e t a i l l a t e r o p e r a t i o n s of the New Mexico o r g a n i z a t i o n and the Sacramento Foundat ion , p r o v i d i n g i n f o r m a t i o n on the r o l e s and f u n c t i o n s of p e r s o n n e l , o p e r a t i n g c o s t s , and e s t i m a t e s of s a v i n g s . New o r g a n i z a t i o n s have a l s o emerged. M i l l e r et a l . (340) d e s c r i b e the development, implementat ion and impact of the Ohio U t i l i z a t i o n and Med ica l Care Assessment Program, which i s a p r o c e s s -o r i e n t e d rev iew system fo r p s y c h i a t r i c s e r v i c e s . Inc luded are p r o s p e c t -i v e , concurrent and r e t r o s p e c t i v e rev iew of q u a l i t y , q u a n t i t y and cost of i n p a t i e n t care of a l l p a t i e n t s by d i a g n o s t i c c a t e g o r y , p a t i e n t , p h y s i c -i a n , h o s p i t a l d i s t r i c t and s t a t e . C r i t e r i a u n d e r l y i n g the computer based system were developed by consensus of p r a c t i c i n g p h y s i c i a n s . E f f e c t s of the program have been measurable : admiss ion r a t e s and lengths of s tay have dec reased , as has the r a t e of d e f i c i e n c i e s i n care i d e n t i f i e d . Q u a l i t y of med ica l records has improved and p a t i e n t s ' r i g h t s are now documented. R e s t u c c i a and Hol loway (413) repor t on a study i n one h o s p i t a l to determine whether the c o r r e c t l e v e l of care had been p r e s c r i b e d by the p h y s i c i a n . When the nurse c o o r d i n a t o r of the s tudy , u s i n g predetermined - 5 8 -c r i t e r i a , judged a p a t i e n t to be i n a p p r o p r i a t e l y l o c a t e d on a p a r t i c u l a r day, she a l s o i d e n t i f i e d the a p p r o p r i a t e l o c a t i o n ( s k i l l e d n u r s i n g f a c i l -i t y , home h e a l t h c a r e , o u t p a t i e n t c a r e , or no care) and the reason the p a t i e n t remained i n the h o s p i t a l . Approx imately 10% of almost two t h o u -sand days were judged to be i n a p p r o p r i a t e , and the most s i g n i f i c a n t b a r r i e r s to a p p r o p r i a t e u t i l i z a t i o n were u n a v a i l a b i l i t y of a p p r o p r i a t e h e a l t h s e r v i c e and the a t t e n d i n g p h y s i c i a n ' s c o n s e r v a t i v e management of the p a t i e n t . C l e a r l y there i s a l i m i t to the impact c o n t i n u i n g educat ion programs f o r p h y s i c i a n s w i l l have on the s i t u a t i o n . In a r a t h e r a t y p i c a l r e g i o n a l l y based rev iew p r o j e c t which f o c -ussed on q u a l i t y of drug u t i l z a t i o n , an i n t e r - d i s c i p l i n a r y team developed process c r i t e r i a fo r ten s i n g l e e n t i t y drugs commonly used i n s k i l l e d n u r s i n g f a c i l i t i e s ( 4 8 8 ) . U t i l i z a t i o n p a t t e r n s i n f i v e f a c i l i t i e s were c h a r t e d , and the g r e a t e s t v a r i a n c e was found w i t h regard to the areas of drug mon i to r ing and i n d i c a t i o n s f o r u s e . Each f a c i l i t y r e c e i v e d i n f o r m a -t i o n about i t s own drug u t i l i z a t i o n p a t t e r n and d e v i a t i o n from the e s t a -b l i s h e d s t a n d a r d s . Desp i te t h i s feedback, a subsequent survey showed no changes i n u t i l i z a t i o n p a t t e r n s . Knoben ( 2 7 8 ) , i n t h o u g h t f u l overv iew of the cu r ren t s t a t u s of drug u t i l i z a t i o n rev iew , p o i n t s out tha t most systems are r e g i o n a l l y based , concerned more w i t h q u a n t i t y than q u a l i t y s t a n d a r d s , and as yet have f a i l e d to become an i n t e g r a l par t of the m e d i -c a l care rev iew p r o c e s s . In g e n e r a l , i t seems that r e g i o n a l l y based rev iew systems are ab le to e f f e c t o n l y moderate change i n p h y s i c i a n b e h a v i o r , even when u s i n g feedback, other e d u c a t i o n a l approaches, and a l a s t r e s o r t d e n i a l of payment. I t i s perhaps f o r t u n a t e that the PSRO l e g i s l a t i o n a l l o w s rev iew procedures to be de legated to agencies to be c a r r i e d out on an in -house - 5 9 -b a s i s where the agency has the c a p a c i t y to do s o . Jack (252) and Nelson (355) both advocate i n - h o u s e rev iew i n preference to e x t e r n a l r e v i e w systems, p o i n t i n g out that re levance to i n d i v i d u a l p r a c t i c e s , o p p o r t u n i t y fo r p a r t i c i p a t i o n i n a l l phases of rev iew a c t i v i t i e s , and g r e a t e r speed i n r e c e i v i n g feedback are f a c t o r s which c o n t r i b u t e to the g rea te r success of in -house systems i n changing p r a c t i t i o n e r s ' b e h a v i o r s . In complet ing t h i s s e c t i o n on sys temat ic assessment of s i t u a -t i o n - s p e c i f i c performance, i t should be noted that as t h i s approach to e v a l u a t i o n has become more w i d e l y used , source documents p r o v i d i n g model se ts of process c r i t e r i a f o r d i f f e r e n t c l i n i c a l problems have been p r o -duced. These e x i s t f o r medic ine ( 1 0 4 , 4 5 7 ) , n u r s i n g ( 8 7 , 5 0 7 ) , and p o d i -a t r y ( 2 1 ) . The o r i g i n of these m a t e r i a l s has been both p r o f e s s i o n a l groups and p r i v a t e p r o j e c t s , and the process used to d e r i v e the c r i t e r i a i s not always made c l e a r . These r e f e r e n c e s would be u s e f u l s t a r t i n g p o i n t s f o r any p r o f e s s i o n a l group w ish ing to develop p a t i e n t care c r i -t e r i a , but are not intended fo r wholesale and u n t h i n k i n g adopt ion and u s e . Unplanned E v a l u a t i o n of Performance The l i t e r a t u r e i n d i c a t e s that random or unplanned judgements as to the adequacy of p r o f e s s i o n a l s ' performance tend to occur p r i m a r i l y w i t h i n three j u r i s d i c t i o n s : the p lace of work ( u s u a l l y a h o s p i t a l ) , the p r o f e s s i o n i t s e l f , and the l i c e n s i n g a u t h o r i t y . The r e g u l a r i t y and c o n -s i s t e n c y w i t h which judgements are a p p l i e d i n these areas i s such tha t most would agree w i th Derbysh i re (128) who s t a t e s w i t h re fe rence to -60-l i c e n s i n g and d i s c i p l i n i n g of p h y s i c i a n s i n the Un i ted S t a t e s , " there i s no system - there are so many v a r i a b l e laws and r e g u l a t i o n s " . The P l a c e of Work The most l i k e l y l o c a t i o n for incompetent behav ior to be n o t i c e d even when sys temat ic assessment has not been undertaken i s a p lace of work where p r a c t i t i o n e r s are f u n c t i o n i n g i n c l o s e contact w i t h c o l -l e a g u e s . Rosenberg (429), i n d i s c u s s i n g the deaths of tw in p h y s i c i a n s i n New York from apparent b a r b i t u a t e a d d i c t i o n , d e s c r i b e s the d i f f i c u l t i e s of h o s p i t a l s i n e f f e c t i v e l y p o l i c i n g inept p h y s i c i a n s . The b r o t h e r s ' s u p e r i o r s had to keep weighing the i n d i v i d u a l ' s r i g h t to c o n f i d e n t i a l i t y and due process a g a i n s t the r i g h t of the p u b l i c to be informed and p r o -t e c t e d . A l though compla in ts were r e g i s t e r e d a t one h o s p i t a l f o r over a year before the men were d i s m i s s e d , and subsequent to that f o r over two years a t a second h o s p i t a l be fore d i s m i s s a l , i t came as a s u r p r i s e to a l l tha t the b r o t h e r s were drug a d d i c t s . A l though h o s p i t a l and other agencies have always been l i a b l e f o r the p r o f e s s i o n a l competence of employees under the d o c t r i n e of r e -spondeat s u p e r i o r , the s c r u t i n y of non-employee s t a f f has t r a d i t i o n a l l y been a f o r m a l i t y and l a r g e l y l i m i t e d to i n s t a n c e s where compla in ts have been made (341). Th is s i t u a t i o n has changed r a p i d l y s i n c e the c o u r t s have enunciated a corporate l i a b i l i t y d o c t r i n e which makes h o s p i t a l s a l s o r e s p o n s i b l e fo r t a k i n g a l l reasonable steps to ensure that p h y s i c i a n s who have s t a f f p r i v i l e g e s perform i n an adequate manner (94,191,242,359,446, 483). As agencies are h e a v i l y dependent on p r o f e s s i o n a l s themselves to determine what behav ior i s and i s not c o r r e c t , r e s p o n s i b i l i t y f o r m o n i -t o r i n g p h y s i c i a n s w i th h o s p i t a l p r i v i l e g e s i s de legated to med ica l s t a f f , - 6 1 -which may a l s o be he ld l i a b l e ( 4 3 0 ) . Ludham (312) adv i ses tha t medica l s t a f f adopt a low key approach to d i s c i p l i n e w i t h f i r s t e f f o r t s d i r e c t e d at c o n t r o l l i n g and c o r r e c t i n g u n d e s i r a b l e behav ior us ing the support of the p h y s i c i a n ' s c o l -l e a g u e s . Every e f f o r t should be made to avo id c o n f r o n t a t i o n and f rank d i s c i p l i n e ; r e f e r r a l to a s t a t e board fo r i n v e s t i g a t i o n should be a l a s t r e s o r t . M i t c h e l l (345) i s perhaps somewhat more h e l p f u l . He o u t l i n e s ten r e a l i s t i c and p o s i t i v e a c t i o n s t h a t any h e a l t h agency should take i n order to i n s t i t u t e and to m a i n t a i n an e f f e c t i v e system f o r d e a l i n g w i t h a l l e g a t i o n s tha t a med ica l s t a f f member i s g i v i n g poor c a r e . He a l s o advocates responding to problems w i t h a graded s e r i e s of a c t i o n s b e g i n -n ing w i t h educat ion - feedback and c u l m i n a t i n g i n r e s t r i c t i o n of p r i v i -l e g e s ; he does not i n c l u d e r e f e r r a l to the l i c e n s i n g body. Other w r i t e r s prov ide s i m i l a r i n f o r m a t i o n about e v a l u a t i o n and c o n t r o l of med ica l s t a f f , i n c l u d i n g c o n s i d e r a t i o n of the r i g h t of s t a f f to due process and of the d i f f i c u l t area of p h y s i c i a n - a d m i n i s t r a t o r r e l a t i o n s h i p s ( 4 4 , 6 9 , 9 4 , 1 9 1 , 3 5 9 , 4 4 6 ) . The P r o f e s s i o n The l i t e r a t u r e i n d i c a t e s t h a t the a b i l i t y and w i l l i n g n e s s of h e a l t h p r o f e s s i o n a l s to p o l i c e themselves and t h e i r peers i s a t best v a r i a b l e . There i s a t l e a s t some evidence that p r o f e s s i o n a l s do not always ac t i n the p u b l i c i n t e r e s t when aware of unacceptable conduct among t h e i r peers ( 1 2 9 ) . Wertheimer and Manasse (532) found that phar -mac is ts who have records of v i o l a t i n g drug r e g u l a t i o n s are d i f f e r e n t i n no measurable way (soc io -demographic v a r i a b l e s , grades e t c . ) from t h e i r peers and were not i n any way s i n g l e d o u t , seen , or t r e a t e d by other - 6 2 -pharmacists as d i f f e r e n t . They concluded that deviance i n the matter of adherence to drug r e g u l a t i o n s i s not o n l y t o l e r a t e d but condoned by the norm group. Jago ( 2 5 7 ) , i n d i s c u s s i n g the t r a d i t i o n a l r o l e that d e n t a l s o c i e t i e s p lay i n peer r e v i e w , s t a t e s that h i s t o r i c a l l y these s o c i e t i e s have acted more as an a r b i t r a t o r f o r fees and that peer rev iew committees might more a p t l y be named "peer j u s t i f i c a t i o n commit tees" . Cohen (99) r e f e r s to the r e l u c t a n c e of doc to rs to c rea te d i f f i c u l t i e s f o r t h e i r p e e r s . On the other hand, the l i t e r a t u r e of the s e v e n t i e s i n d i c a t e s that p r o f e s s i o n a l s o c i e t i e s are beg inn ing to take a more a c t i v e and aggress i ve i n t e r e s t i n e v a l u a t i o n and d i s c i p l i n e of p o t e n t i a l l y incompe-tent p r a c t i t i o n e r s . The C a l i f o r n i a Nurses A s s o c i a t i o n has s ince the e a r l y s e v e n t i e s organized P r o f e s s i o n a l Performance Committees w h i c h , among other t h i n g s , ac t on compla in ts of a p r a c t i t i o n e r ' s performance ( 8 1 ) . H o s p i t a l s and medica l s o c i e t y boards are now e x p l o r i n g ways of working t o g e t h e r , r a t h e r than i n i s o l a t i o n from each o ther ( 3 9 4 ) . American medic ine i s s u f f i c i e n t l y concerned about " d i s a b l e d " p h y s i c i a n s tha t i n 1975 the AMA sponsored a n a t i o n a l conference on the i s s u e ( 1 3 8 ) . One nove l approach d i r e c t e d toward h e l p i n g the a l c o h o l i c , d r u g - a d d i c t e d or m e n t a l l y i l l p h y s i c i a n i s repor ted by the New York Med-i c a l S o c i e t y ( 1 7 7 ) . A cadre of v o l u n t e e r doc to rs respond to compla in ts about p h y s i c i a n s by q u i e t l y i n v e s t i g a t i n g and o f f e r i n g he lp to the p h y s i -c i a n i f he i s i n t r o u b l e . Th is does not prec lude a complainant going to the s t a t e b o a r d , but as more p h y s i c i a n s become • aware of t h i s i n f o r m a l and e a r l y a s s i s t a n c e , they may use t h i s to a i d the p h y s i c i a n i n f o r m a l l y i n s t e a d of w a i t i n g fo r s u f f i c i e n t evidence to accumulate to j u s t i f y fo rmal d i s c i p l i n a r y a c t i o n . S i m i l a r p r o j e c t s have sprung up i n other - 6 3 -s t a t e s . U n f o r t u n a t e l y , because of t h e i r very n a t u r e , no records e x i s t to serve as evidence of these programs' e f f e c t i v e n e s s . Furthermore a p r a c -t i t i o n e r being a ided by one of these programs i s not r e s t r a i n e d i n any way from c o n t i n u i n g to p r a c t i c e . / A d d i t i o n a l ev idence of an awakening i n t e r e s t i n p r o f e s s i o n a l s ' r e s p o n s i b i l i t i e s i n e f f e c t i v e e v a l u a t i o n of peers i s prov ided by a r e s o l -u t i o n passed by the 1976 annual meeting of the American M e d i c a l A s s o c i a -t i o n which urged a l l peer rev iew committees to r e f e r to a p p r o p r i a t e d i s c i p l i n a r y bodies any p h y s i c i a n s who do not meet "accepted p r o f e s s i o n a l s tandards" ( 1 1 ) . F i n a l l y , i t should be noted that i n many Canadian p r o v i n c e s , a p r o f e s s i o n a l o r g a n i z a t i o n i s the de f a c t o l i c e n s i n g body. A c t i v i t i e s of these and other l i c e n s i n g boards are d i s c u s s e d n e x t . The L i c e n s i n g A u t h o r i t y The p u b l i c h a s , through the l i c e n s i n g system, prov ided a t l e a s t one v e h i c l e f o r e v a l u a t i n g a h e a l t h p r o f e s s i o n a l whose behav ior i s q u e s t -i o n a b l e . A l l l i c e n s i n g laws have p r o v i s i o n fo r r e v o c a t i o n of l i c e n s e . Th is c o n t r o l i s u l t i m a t e l y i n the hands of the p u b l i c , though more o f t e n than not a d m i n i s t r a t i o n of the law i s de legated to p r o f e s s i o n a l o r g a n i z a -t i o n s or to a government agency composed l a r g e l y of appointed p r o f e s s i o n -a l s . The r e s p o n s i b i l i t y f o r ensur ing competence of p r o f e s s i o n a l s i s a complex i s s u e , one w i t h which Claude-Armand Sheppard d e a l t s u c c i n c t l y a t a recent conference on r e g u l a t i o n of the p r o f e s s i o n s (472) when he s t a t e d : "not o n l y do they ( those r e s p o n s i b l e ) have to v e r i f y c o n t i n u i n g p r o f e s s -i o n a l p h y s i c a l and mental competence, but they must a s s i s t p r o f e s s i o n a l s i n keeping up to date w i t h developments i n t h e i r f i e l d s , as w e l l as d e a l -64-with the very real d i f f i c u l t i e s of professionals who face obsolescence due to one form or another of incompetence. Indeed, while the protection of the public must be of paramount concern, i t would be unfair and unwise to subject professionals, to insecurity, arbitrary rejection and unemployment." (One might add as a significant consideration the loss of investment made by the public in the training of skilled manpower). Sheppard goes on to say: "furthermore, inadequate knowledge, poor physical condition or mental d i f f i c u l t i e s do not necessarily incapacitate a professional or render him completely unfit. The consequences will depend on the nature of the profession, the character of an individual's practice and a l l the circum-stances of the case". Regulation of the health professional through licensing i s con-cerned with professional competence, mental and physical competence, and adherence to an ethical code. Cohen (99) itemizes some of the many di f f i c u l t i e s licensing boards incur in discharging their regulatory re-sponsibilities: reluctance of practitioners to cause d i f f i c u l t i e s for their peers; lack of expertise of board members with regard to the pro-cess required for acceptable disciplinary action; a tendency of the courts of appeal to reverse board disciplinary decisions (often due to inept handling and violation of "due process" by boards); and licensing laws which lay down grounds for discipline but are inadequate because they are incomplete or so imprecise as to be useless. An extreme example of this i s the medical practice law in Washington, D.C; at present, only conviction of a felony or a crime or moral turpitude are grounds for loss of license in this jurisdiction (504). Golin adds to this l i s t of d i f f i c u l t i e s the claim that state boards have inadequate budgets and are assigned to overworked and inexperienced legal counsel (192). Many of these problems are also cited by Derbyshire (128), who probably i s responsible for the most thorough research done to date on the nature and extent of professional discipline of American physicians. - 6 5 -U n f o r t u n a t e l y no comparable pub l i shed i n v e s t i g a t i o n s i n the other p r o f e s -s ions or i n Canada were found dur ing t h i s s e a r c h . Tables I and I I summar-i z e h i s f i n d i n g s as to types of and grounds f o r d i s c i p l i n e of American p h y s i c i a n s dur ing the four year p e r i o d . Types of D i s c i p l i n a r y A c t i o n Taken w i t h American P h y s i c i a n s 1963 - 1967 P r o b a t i o n 375 Revocat ion 334 Suspension 161 Reprimands 68 TOTAL 938 Table I Causes f o r D i s c i p l i n a r y A c t i o n Taken w i t h American P h y s i c i a n s 1963 - 1967. N a r c o t i c s 440 Menta l Incompetence 94 Fraud and Dece i t 74 C o n v i c t i o n of Felony 72 A b o r t i o n s 71 A l c o h o l i s m 41 P r o f e s s i o n a l Conduct 68 Mora l Turp i tude 26 Gross M a l p r a c t i c e 7 Fraud and A p p l i c a t i o n 6 Gross Immoral i ty 3 Fee S p l i t t i n g 1 Gross Misconduct 1 TOTAL 904 Table I I I t i s d i f f i c u l t to know whether the s i t u a t i o n i s the same ten years l a t e r . Moreover as many l i c e n s i n g laws are unc lear on incompetence or u n p r o f e s s i o n a l conduct (which i s sometimes taken to mean incompe-t e n c e ) , i t i s d i f f i c u l t to know whether i n the above i n s t a n c e s incompe-tence was a c o - e x i s t i n g f a c t o r and , i f s o , , whether i t was cause or e f f e c t . - 6 6 -What i s known i s that v a r i o u s a c t i o n s to p r o t e c t the p u b l i c from d e f i c i e n t p r a c t i t i o n e r s are now being i n i t i a t e d by both the p r o f e s s -i ons and the p u b l i c through a number of l e g i s l a t i v e reforms ( 2 4 1 ) . In 1969 F l o r i d a enacted a " s i c k d o c t o r " s t a t u t e which permits the m e d i c a l board to take a c t i o n on a l i c e n s e f o r c o n d i t i o n s which prove i n j u r i o u s or l e a d to inadequate performance ( 3 7 7 ) . S p e c i f i e d are o rgan ic i l l n e s s , mental or emot ional d i s o r d e r s , drug and a l c o h o l abuse, l o s s of motor s k i l l , and d e t e r i o r a t i o n through a g i n g . The law has served as a model f o r other s t a t e laws enacted i n the s e v e n t i e s . In 1975 t h i s same med ica l p r a c t i c e ac t was amended to a l l o w the board to d i s c i p l i n e fo r immoral or u n p r o f e s s i o n a l conduct , incompetence, neg l igence and w i l f u l misconduct . The law i n c l u d e s a c lause which permits acceptance of d i s c i p l i n e by h o s -p i t a l s , med ica l s t a f f s , PSRO's and p r o f e s s i o n a l s o c i e t i e s as ev idence of u n p r o f e s s i o n a l conduct . A p a r a l l e l law makes i t mandatory fo r h o s p i t a l s to r e p o r t any d i s c i p l i n a r y a c t i o n taken w i t h i n t h i r t y d a y s . The board has a wide range of a c t i o n s at i t s d i s p o s a l i n c l u d i n g med ica l and p s y c h i -a t r i c examinat ion , c o u n s e l l i n g , r e s t r i c t i o n of scope of p r a c t i c e , p r o b a -t i o n , p r e s c r i p t i o n of s u p e r v i s i o n or e d u c a t i o n , and s u s p e n s i o n , r e v o c a -t i o n or d e n i a l of l i c e n s e . V i r g i n i a (203) New Mex ico , and A r i z o n a (508) are a l s o among the s t a t e s which have impai red p h y s i c i a n s t a t u t e s . A r i z o n a ' s l e g i s l a t i o n r e q u i r e s p h y s i c i a n s , medica l s o c i e t i e s and h o s p i t a l s to r e p o r t to the l i c e n s i n g board any i n f o r m a t i o n which appears to show a doctor may be incompetent , u n f i t f o r safe p r a c t i c e or g u i l t y of u n p r o f e s s i o n a l conduct ( 1 2 9 , 5 0 8 ) . I t would seem that l e g i s l a t i o n of t h i s nature prec ludes the format ion i n these j u r i s d i c t i o n s of the i n f o r m a l " h e l p i n g networks" r e f e r r e d to e a r l i e r . A 1975 C a l i f o r n i a s t a t u t e not on ly d i r e c t s -67-h o s p i t a l s to r e p o r t a l l p r i v i l e g e l i m i t a t i o n s and d i s m i s s a l s , but a l s o prov ides immunity to the i n f o r m a n t , and g i v e s the med ica l l i c e n s i n g board expanded i n v e s t i g a t o r y powers, c o n t r o l of i t s own s t a f f and funds , and the power to suspend p r a c t i c e pending a d i s c i p l i n a r y hear ing ( 1 8 9 ) . I l l i n o i s has e s t a b l i s h e d a separate (from l i c e n s i n g board) med ica l d i s c i p l i n a r y board and g i ven i t c o n s i d e r a b l e power as w e l l ( 1 5 3 ) . New l e g i s l a t i o n i n Quebec recogn izes the j u d i c i a l nature of the p r o f e s s i o n a l d i s c i p l i n a r y process and i t has attempted to i n c r e a s e i t s e f f e c t i v e n e s s by p r o v i d i n g that a government appointed a t t o r n e y w i l l a c t as chairman of the d i s c i p l i n a r y committee of each one of the l i c e n s e d p r o f e s s i o n s i n the prov ince ( 9 0 , 4 0 0 ) . Ontar io has used l e g i s l a t i o n to c l a r i f y , r e g u l a r i z e , and s t rengthen the d i s c i p l i n a r y process f o r f i v e major h e a l t h p r o f e s s i o n s ( 2 2 9 ) , and B r i t i s h Columbia has enacted changes i n d i s c i p l i n a r y procedures f o r r e g i s t e r e d nurses ( 4 0 8 ) . I t i s of i n t e r e s t that the m a j o r i t y of l e g i s l a t i v e a c t i v i t y repor ted i s concerned w i t h the d i s c i p l i n e of p h y s i c i a n s . As the c e n t r a l f i g u r e i n the h e a l t h team i t i s p r e d i c t a b l e that pr imary concern w i l l be d i r e c t e d a t t h i s p r o f e s s i o n a l , however i t i s a l s o f a i r l y apparent that i n the Uni ted S t a t e s the present med ica l m a l p r a c t i c e c r i s i s f i g u r e s h e a v i l y i n the f l u r r y of "crackdown" l e g i s l a t i o n ( 1 2 9 , 1 8 9 ) . In Canadian n u r s i n g there appears to be some a c t i v i t y aimed at d e v i s i n g r e g u l a t i o n s to enable the l i c e n s i n g or r e g i s t e r i n g body to more e f f e c t i v e l y and e q u i t a b l y apply the d i s c i p l i n a r y p r o c e s s . The C o l l e g e of Nurses of Ontar io has e s t a b l i s h e d a set of " s tandards" which are r a t h e r g e n e r a l statements of n u r s i n g f u n c t i o n s to be a p p l i e d by the R e g i s t r a -t i o n , Compla in ts , and D i s c i p l i n a r y Committees of the C o l l e g e ( 1 0 2 ) . I t i s assumed that the nurse i s competent u n t i l found incompetent through - 6 8 -d i s c i p l i n e ; s i n c e one of the standards i s "seek he lp and guidance when unable to perform " , a r e g i s t r a n t who does not attempt to perform beyond h i s a b i l i t y would apparent l y not be a t r i s k f o r d i s c i p l i n e . The Order of Nurses of Quebec sets out a d e t a i l e d d e s c r i p t i o n of e t h i c a l behav io rs i n twelve c a t e g o r i e s (370) and a conceptua l p lan f o r e v a l u a t i o n of n u r s i n g s e r v i c e s and performance (371) . The R e g i s t e r e d Nurses ' A s s o c -i a t i o n of B r i t i s h Columbia i s under tak ing to d e s c r i b e through a " S t a t e -ment of Roles and Funct ions of R e g i s t e r e d Nurses" behav io rs s u b j e c t to j u r i s d i c t i o n a l a u t h o r i t y by the A s s o c i a t i o n ; a d d i t i o n a l l y the A s s o c i a t i o n has s p e c i f i c a l l y d e f i n e d incompetence, which i s j u s t i f i c a t i o n f o r d i s c i -p l i n e i n that j u r i s d i c t i o n ( 3 3 ) . There were found no d i s c u s s i o n s of a c t i v i t y connected w i t h p r o f e s s i o n a l d i s c i p l i n e i n the d e n t a l or pharmacy l i t e r a t u r e , a l though Pavone, a d e n t a l educato r , suggests t h a t l i c e n s i n g boards should be g i ven the a u t h o r i t y to order remedia l educat ion f o r d e f i c i e n t d e n t i s t s ( 3 8 1 ) . He then goes on to enumerate a l l of the d i f f i c u l t i e s t h i s a c t i o n would e n t a i l and concludes that t h i s w i l l be an unpopular task which should be shared by a consor t ium of boards , u n i v e r s i t i e s and schools and a s s o c i a -t i o n s . Summary and Comment The p u b l i c and h e a l t h p r o f e s s i o n a l s c o l l e c t i v e l y have d e a l t w i th the d i f f i c u l t i s s u e of e v a l u a t i n g h e a l t h p r a c t i t i o n e r s ' c l i n i c a l behav io rs by deve lop ing a v a r i e t y of techniques and approaches. The l i t e r a t u r e concerned w i t h these approaches has been d i s c u s s e d i n t h i s - 6 9 -chapter w i t h i n three broad s e c t i o n s : sys temat ic assessment of t o t a l p e r -formance, sys temat i c assessment of s i t u a t i o n - s p e c i f i c performance, and unplanned e v a l u a t i o n of performance. D i s c u s s i o n of sys temat i c assessment of t o t a l or a broad range of c l i n i c a l behav ior i s found i n the n u r s i n g l i t e r a t u r e and , to a n o t a b l y l e s s e r e x t e n t , i n the l i t e r a t u r e of medic ine* and pharmacy. I t should be noted that nurses have l a r g e l y been organized and recompensed f o r s e r v i c e as an employer group and the a v a i l a b l e l i t e r a t u r e i s perhaps r e f l e c t i v e of the long e s t a b l i s h e d a b i l i t y and a u t h o r i t y of an organized and knowledgeable payor to demand a c c o u n t a b i l i t y fo r s e r v i c e . Taken as a whole , the l i t e r a t u r e i n d i c a t e s tha t e v a l u a t i o n approaches which l o o k a t a broad range of p r o f e s s i o n a l behav io rs vary g r e a t l y i n c o m p l e x i t y ; s p e c -i f i c i t y of performance c r i t e r i a and s tandards ; and the ex tent to which they have been t e s t e d f o r r e l i a b i l i t y , v a l i d i t y and u t i l i t y fo r fo rmat i ve e v a l u a t i o n . In p a r t i c u l a r t h i s m a t e r i a l r e v e a l s that t r a d i t i o n a l pe r fo rm-ance a p p r a i s a l methods from b u s i n e s s , i n d u s t r y , and educat ion have been adapted to meet the requirements of the c l i n i c a l envi ronment . Most o f t e n a conceptua l framework i s dev ised to o rgan i ze c l i n i c a l performance i n t o major c a t e g o r i e s : b e h a v i o r a l d e s c r i p t o r s are at tached to these ; r a t i n g s c a l e s and c h e c k l i s t s are used as measurement t o o l s ; and anecdota l notes are employed as suppor t i ve m a t e r i a l . The p a u c i t y of i n f o r m a t i o n about the u t i l i t y of these approaches to a c t u a l l y i n f l u e n c e and improve compe-tence may e x p l a i n the more recent emergence of e v a l u a t i o n systems which * Barro (41) rev iews s e v e r a l s t u d i e s which attempted to assess a wide range of p h y s i c i a n performance p r i o r to 1970 ( T a y l o r , 1954; P e t e r s o n , 1956; C l u t e , 1963; Jungfer and L a s t , 1964) . These were p r i m a r i l y r e -search s t u d i e s whose methods never found t h e i r way i n t o common use , probably f o r reasons of expense, l a c k of a u t h o r i t y to i n i t i a t e such s u r -v e i l l a n c e , and l a c k of a c c e p t a b i l i t y to p h y s i c i a n s and/or p a t i e n t s . -70-focus not on g l o b a l c o n c e p t u a l i z a t i o n s of performance, but r a t h e r on h i g h l y s p e c i f i c performance c r i t e r i a . L i t e r a t u r e d e a l i n g w i t h e v a l u a t i o n of s i t u a t i o n - s p e c i f i c c l i n i -c a l performance i s much more recent and i s found p r i m a r i l y i n m e d i c a l and , to a l e s s e r e x t e n t , n u r s i n g l i t e r a t u r e . Borrowed from the f i e l d of i n d u s t r i a l q u a l i t y c o n t r o l , t h i s approach i n v o l v e s i d e n t i f y i n g s p e c i f i c c r i t e r i a to be met f o r s p e c i f i c c l i n i c a l problems, measuring to see that they are met (most o f t e n us ing p a t i e n t r e c o r d s , a b s t r a c t s or payment c l a i m s as a data source) , t a k i n g c o r r e c t i v e a c t i o n i f d e s i r e d behav iors were not p r e s e n t , and remeasuring to a s c e r t a i n success of c o r r e c t i v e a c t i o n . S i t u a t i o n - s p e c i f i c performance e v a l u a t i o n i s not o f t e n used i n i n d i v i d u a l p r a c t i c e s , a l though p h y s i c i a n i n t e r e s t i n t h i s i s g rowing . On the other hand, much a c t i v i t y of t h i s nature i s found i n h e a l t h care a g e n c i e s , and a v a r i e t y of c o r r e c t i v e a c t i o n s are employed: feedback , c o u n s e l l i n g , peer p r e s s u r e , e d u c a t i o n a l s e s s i o n s , a d m i n i s t r a t i v e changes, and removal of p r a c t i c e p r i v i l e g e s . Much a c t i v i t y i s undertaken at the r e g i o n a l l e v e l a l s o , l a r g e l y as a r e s u l t of f e d e r a l l e g i s l a t i o n i n the Uni ted S t a t e s , w i t h the most common c o r r e c t i v e a c t i o n s being persona l feedback , i n d i v i d u a l and group e d u c a t i o n a l e f f o r t s , and d e n i a l of pay -ment. There i s evidence that s i t u a t i o n - s p e c i f i c performance e v a l u a t i o n i s p a r t i c u l a r l y u s e f u l i n f a c i l i t a t i n g and documenting change i n p r o f e s s -i o n a l s ' c l i n i c a l b e h a v i o r s ; tha t c o n t i n u i n g educat ion i s o n l y one of s e v e r a l major change s t r a t e g i e s ; t h a t change s t r a t e g i e s must be used s e l e c t i v e l y to be e f f e c t i v e ; and that w h i l e these rev iew a c t i v i t i e s may i n c r e a s e q u a l i t y of c a r e , t h e i r c o n t r i b u t i o n to cost c o n t r o l i s d e -b a t a b l e . There i s some i n d i c a t i o n a l s o that in -house rev iew a c t i v i t i e s are more l i k e l y to b r i n g about d e s i r e d behavior change than e x t e r n a l or - 7 1 -r e g i o n a l rev iew . Two a d d i t i o n a l observa t ions might be made: f i r s t , t h i s approach measures h i g h l y v i s i b l e t e c h n i c a l s k i l l s and a c t i o n s more l i k e l y to be documented i n records than are h e a l t h teach ing or use of c e r t a i n comfort measures; second, where c r i t e r i a are e s t a b l i s h e d by c l i n i c a l problem, t h i s approach g e n e r a l l y assumes tha t d i a g n o s t i c data have been a c c u r a t e l y c o l l e c t e d . Thus i t i s apparent t h a t a r a t h e r narrow range of c l i n i c a l behav io rs i s being e v a l u a t e d . F i n a l l y , l i t e r a t u r e concerned w i t h random or unplanned e v a l u a -t i o n of p r o f e s s i o n a l s ' c l i n i c a l behav io rs i n response to compla in ts i n d i c a t e s tha t as a r e s u l t of the co rpora te l i a b i l i t y d o c t r i n e , agencies are i n c r e a s i n g l y s y s t e m a t i z i n g e v a l u a t i o n of a l l p r a c t i t i o n e r s . A d d i -t i o n a l l y , the p r o f e s s i o n s are us ing a v a r i e t y of approaches to moni tor and take a p p r o p r i a t e a c t i o n w i t h q u e s t i o n a b l y competent members. F i n a l l y the p u b l i c h a s , through l e g i s l a t i o n and p r o v i s i o n of r e s o u r c e s , acted i n support of a more e f f e c t i v e and e q u i t a b l e a p p l i c a t i o n of the d i s c i p l i n a r y process where p r o f e s s i o n a l incompetence i s an i s s u e . I t can be concluded from h e a l t h s c i e n c e s l i t e r a t u r e of the e a r l y and m i d - s e v e n t i e s tha t there have been c o n s i d e r a b l e i n t e r e s t i n and development of methods to eva luate c l i n i c a l performance of p r a c t i t i o n -e r s . Fur ther work i s r e q u i r e d however, be fore more than t e n t a t i v e g e n e r -a l i z a t i o n s can be made about the u s e f u l n e s s of the v a r i o u s approaches . In a d d i t i o n t h i s work must be viewed w i t h i n the contex t of yet another i s s u e : g i ven that we can develop e f f e c t i v e approaches f o r e v a l u a t i n g the process of g i v i n g h e a l t h c a r e , to what extent should we judge the p r a c t i -t i o n e r by what he does r a t h e r than by the r e s u l t s he achieves? -72-Chapter IV ASSESSING RESULTS OF CLINICAL PERFORMANCE: OUTCOME E s c a l a t i n g h e a l t h care c o s t s have caused the p u b l i c and p r o f e s -s ions to q u e s t i o n the va lue and d e s i r a b l i l t y of many h e a l t h care p r a c -t i c e s , and by e x t e n s i o n of t h i s r e a s o n i n g , to conclude that competence of p r a c t i t i o n e r s should be judged by the r e s u l t s a c h i e v e d , i . e . by whether c l i e n t h e a l t h improved. The assumption u n d e r l y i n g outcomes e v a l u a t i o n i s very s imply t h a t i f d e s i r a b l e r e s u l t s were a c h i e v e d , the care g i v e r s must have been competent. Some of the problems a s s o c i a t e d w i t h t h i s t h i n k i n g , e . g . the f a c t tha t many f a c t o r s not w i t h i n the c o n t r o l of the p r a c t i t i o n -er i n f l u e n c e h e a l t h s t a t e , w i l l be d i s c u s s e d i n Chapter V I . Despi te such problems, developments i n outcomes based e v a l u a t i o n have made an i m p o r t -ant c o n t r i b u t i o n to what i s known about a s s e s s i n g competence of h e a l t h p r o f e s s i o n a l s . The l i t e r a t u r e concerned w i t h a s s e s s i n g r e s u l t s of c l i n i -c a l performance i s o rganized i n t h i s chapter i n two broad s e c t i o n s : g e n e r a l measures of h e a l t h s t a t u s and s i t u a t i o n - s p e c i f i c outcomes e v a l u a -t i o n . Genera l Measures of H e a l t h S ta tus In t h i s s e a r c h , there emerged few accounts of genera l or non -s p e c i f i c h e a l t h s t a t e measures used fo r e v a l u a t i o n of h e a l t h p e r s o n n e l . Most of those i n e x i s t e n c e have been c o n s t r u c t e d f o r r e s e a r c h and d e s i g n -ed to app ly to t a r g e t p o p u l a t i o n s . - 7 3 -Bergner (48) d e s c r i b e s development of the S ickness Impact P r o f i l e (SIP) which measures h e a l t h s t a t u s and i s used as an i n d i c a t o r of q u a l i t y of h e a l t h c a r e . Evidence of the v a l i d i t y of the SIP i s found i n the h i g h c o r r e l a t i o n s of the SIP scores w i t h s e l f - a s s e s s m e n t of h e a l t h s t a t u s , c l i n i c i a n assessment of h e a l t h s t a t u s , and other measures of d y s f u n c t i o n . McDonald and M a r t i n i (319) r e p o r t the use of a condensed v e r s i o n of the SIP as an outcome measure of pr imary c a r e . Zyzansk i et a l . (548) d e s c r i b e the r e v i s i o n of a p r e v i o u s l y developed s e l f - a d m i n i s t e r e d s c a l e of p a t i e n t s a t i s f a c t i o n w i t h pr imary care g i ven by p h y s i c i a n s . The m o d i f i c a t i o n and r a t i o n a l e are p r o v i d e d , the complete s c a l e i s reproduced and s c o r i n g i s e x p l a i n e d so i t can be used without f u r t h e r i n f o r m a t i o n . Using t h i s and another a r t i c l e which i s r e f e r e n c e d , one cou ld c a p i t a l i z e on t h i s work to develop a s i m i l a r s a t i s f a c t i o n index f o r o ther h e a l t h care personnel and other s e t t i n g s . Janke (258) has developed an o p e r a t i o n a l m u l t i - d i m e n s i o n a l model fo r h e a l t h s t a t u s measurement, the Relevant Hea l th S ta tus (RHS). Based on the premise that h e a l t h s t a t u s i s a composite of b i o - m e d i c a l s t a t e , p a t i e n t s a t i s f a c t i o n and s o c i a l f u n c t i o n , the RHS t o o l p rov ides an approach fo r c o l l e c t i n g , o r g a n i z i n g and express ing i n f o r m a t i o n about these v a r i a b l e s fo r aggregates of p a t i e n t s i n r e l a t i o n to the h e a l t h care system o b j e c t i v e s and environment. Janke c l a i m s tha t t h i s approach to o r g a n i z a t i o n of data prov ides a common i n f o r m a t i o n base fo r p l a n n e r s , e d u c a t o r s , a d m i n i s t r a t o r s , p r o v i d e r s and l a y d i r e c t o r s . D i r e c t i o n s f o r a p p l i c a t i o n of the RHS model to s e v e r a l problem a r e a s , i n c l u d i n g educa -t i o n and e v a l u a t i o n of h e a l t h care p a r t i c i p a n t s , are p r o v i d e d . Barro ( 4 1 ) , i n her rev iew of p h y s i c i a n performance measures, devotes a chapter to outcome approaches. She d e s c r i b e s v a r i o u s i n f l u e n --74-t i a l writers' (Donabedian, Shapiro, Elinson, Williamson, Siegal) concept-ualizations of outcome, a l l of which were arrived at non-empirically. She also discusses Sanazaro's and Williamson's 1968 effort to arrive at a classification of patient outcomes through analysis of c r i t i c a l incidents reported by internists; mentions three studies aimed at assessing patient dissatisfaction; overviews three studies which attempt to measure patient compliance; and summarizes attempts at measuring outcomes using diagnosis-specific physical data (Payne), general condition (Williamson), physical function (Katz), and social function (several). Much of the literature reviewed by Barro predates 1970. Situation-Specific Outcomes Evaluation Several writers provide comprehensive descriptions of assess-ment procedures which are based on desired outcomes for a highly specific patient population. In this context, patient outcome i s variously defin-ed as "the health status of the patient at suitable end point or s t a b i l i -zation point following a reasonably well-defined episode of care" (318), "an alteration in the health status of the consumer as an end result of the activities of professionals" (239), "effects or results of care" (544), and "end results to be achieved through activities performed by or under direction of the nurse" (262). Such authorities as McClure (318), Williamson (535), Lang (288), Taylor (495), Zimmer (541,542,543,544) and Hilger (239) agree on the essential steps in situation-specific outcomes assessment: - 7 5 -1 . S e l e c t and d e f i n e a p a t i e n t p o p u l a t i o n w i t h a common problem. H i l g e r suggests tha t numerous frames of r e f e r e n c e may be used : developmental s t a g e s , med ica l d iagnoses , d i a g n o s t i c or t reatment procedures , or f u n c t i o n a l problems. 2 . I d e n t i f y a time f rame, e . g . end of a c r i t i c a l pe r iod or i n p a t i e n t phase, or s p e c i f y s e v e r a l s i g n i f i c a n t stages i n the n a t u r a l h i s t o r y of the c o n d i t i o n . 3 . I d e n t i f y the most s i g n i f i c a n t p a t i e n t problems fo r t h i s c o n d i t i o n , e . g . f l a t u l e n c e , p o s t - c o l o s t o m y . 4. State expected h e a l t h s t a t u s i n r e l a t i o n to t h i s problem fo r each stage of the c o n d i t i o n i n the time frame, e . g . 0 - 5 days , f l a t u l e n c e x8 ; 5 -8 d a y s , x4 ; 8 -10 days , 0 . 5 . Set an acceptab le degree of goa l achievement, e . g . 80% of p a t i e n t p o p u l a t i o n w i l l meet e x p e c t a t i o n s . 6 . And as w i t h process q u a l i t y assurance approaches, measure a c t u a l h e a l t h s t a t e of a sample or the whole p o p u l a t i o n a g a i n s t the c r i -t e r i a and s t a n d a r d , i n v e s t i g a t e probable causes f o r unacceptable d e v i a t i o n s from the s t a n d a r d , take c o r r e c t i v e a c t i o n and remeasure. D i r e c t measures of process or s t r u c t u r e are not taken (535, 5 4 3 ) , a l though both o b v i o u s l y must be cons idered i n a n a l y s i s of unaccept -ab le outcomes. Zimmer (543,544) recommends s p e c i f y i n g , but not measur-i n g , the a c t i v i t i e s and resources most impor tan t , i . e . powerful and c o s t e f f e c t i v e , to achievement of each d e s i r e d outcome. Hea l th p r o f e s s i o n a l s w ish ing to undertake r e s u l t s - o r i e n t e d e v a l u a t i o n of care w i l l f i n d some a s s i s t a n c e i n the form of d i r e c t i o n s or examples i n the l i t e r a t u r e . Zimmer (541,542) g i v e s g u i d e l i n e s f o r assessment of n u r s i n g s e r v i c e which i n c l u d e a d e s c r i p t i o n of the c h a r a c t -e r i s t i c s of outcome c r i t e r i a , i n s t r u c t i o n f o r the f o r m u l a t i o n of se ts of c r i t e r i a , examples of a p p l i c a t i o n of the c r i t e r i a f o r e v a l u a t i o n of d e l i v e r e d c a r e , and a d i s c u s s i o n about peer , i n t e r - d i s c i p l i n a r y , and consumer p a r t i c i p a t i o n i n e s t a b l i s h i n g outcome c r i t e r i a . Zimmer's t r e a t -ment of t h i s sub jec t i s c o n c i s e , comprehensive and based on the e x p e r i -ence of many n u r s e s , however supplementary read ing would be r e q u i r e d . - 7 6 -Another r e l a t i v e l y complete re fe rence f o r the n u r s i n g p r o f e s s i o n i s the American Nurses A s s o c i a t i o n ' s G u i d e l i n e s f o r Review of Nurs ing Care a t  the L o c a l L e v e l which d i s c u s s e s i n some d e t a i l the processes which can be used to d e v e l o p , e d i t and v a l i d a t e outcome c r i t e r i a ( 5 1 4 ) . Both Zimmer's manual (542) and the ANA g u i d e l i n e s (514) p rov ide s e v e r a l sample se ts of outcome c r i t e r i a f o r common n u r s i n g problems. Other authors have p ropos -ed s p e c i f i c n u r s i n g outcome measures f o r e v a l u a t i o n of n e u r o l o g i c a l p a t i e n t s (495) ; of r e s p i r a t o r y s t a t u s a f t e r abdominal , p e l v i c or t h o r a c i c surgery ( 2 6 2 ) ; o f f u n c t i o n a l progress of p a t i e n t s w i t h conges t i ve hear t f a i l u r e ( 2 7 , 2 8 ) , of h e a l t h s t a t e a f t e r hysterectomy ( 3 5 ) , and of progress of ado lescents w i t h anorex ia nervosa ( 1 2 7 ) . P h y s i c i a n s w i l l be e s p e c i a l l y i n t e r e s t e d i n the second of a two volume r e p o r t (34) which presents the r e s u l t s of a study to develop d i s e a s e - s p e c i f i c shor t term outcome measures fo r a s s e s s i n g the q u a l i t y of m ed ica l and/or s u r g i c a l care d e l i v e r e d to p a t i e n t s w i t h one of e i g h t d i f f e r e n t c o n d i t i o n s . For each c o n d i t i o n there i s a t e c h n i c a l chapter which presents epidemiology or p r e - s u r g i c a l c o n d i t i o n ; a v a i l a b l e data on inc idence and p r e v a l e n c e , the use of med ica l s e r v i c e s and a s s o c i a t e d d i s a b i l i t y and m o r t a l i t y ; o p e r a t i o n a l d e f i n i t i o n s of the d i sease or i n d i c a t i o n s fo r the s u r g i c a l procedures i n the p a t i e n t p o p u l a t i o n to be s t u d i e d ; an i d e n t i f i c a t i o n of f a c t o r s not under the c o n t r o l of med ica l care tha t may i n f l u e n c e outcomes from the d i s e a s e or surgery r e g a r d l e s s of the q u a l i t y of care p r o v i d e d ; a method f o r c o n t r o l l i n g the f a c t o r s not a f f e c t e d by the q u a l i t y of med ica l c a r e ; s p e c i f i c outcome standards tha t cou ld be used to assess the q u a l i t y of care g iven to p a t i e n t s w i t h the c o n d i t i o n or s u r g i c a l procedure ; d i s c u s s i o n of an expert p a n e l ' s j u d g e -ment of r e l a t i v e importance and s e n s i t i v i t y of each outcome i n a s s e s s i n g - 7 7 -q u a l i t y of med ica l c a r e ; and es t imates of the s i z e of the sample frame necessary to i d e n t i f y a s p e c i f i c number of p a t i e n t s w i t h a g i ven c o n d i -t i o n . Whi le the book i s most u s e f u l f o r med ica l a u d i t , a l l h e a l t h p r o -f e s s i o n a l s w i l l be i n t e r e s t e d i n the appended i tem pools fo r a n x i e t y and d e p r e s s i o n , a c t i v i t y l i m i t a t i o n , p a t i e n t s a t i s f a c t i o n , soc io -demographic f a c t o r s , b e l i e f i n e f f i c a c y of c a r e , and h e a l t h p e r c e p t i o n . Ryge and Snyder (450) surveyed d e n t a l l i t e r a t u r e and concluded tha t e v a l u a t i o n methods to date were e i t h e r too complex f o r r e a l wor ld a p p l i c a t i o n or l a c k e d s p e c i f i c c r i t e r i a . They developed a r a t i n g system fo r a s s e s s i n g q u a l i t y of d e n t a l r e s t o r a t i o n s : " s a t i s f a c t o r y " , w i t h s u b -c a t e g o r i e s of "meets a l l s tandards" and "observe a t next v i s i t " ; and "not a c c e p t a b l e " , wi th s u b - c a t e g o r i e s of " r e p l a c e fo r p revent ion" and " r e p l a c e s t a t i m " . S p e c i f i c c r i t e r i a and corresponding key words were developed f o r each of the four c a t e g o r i e s w i t h respec t to th ree c h a r a c t e r i s t i c s : su r face and c o l o r , anatomic fo rm, and marg ina l i n t e g r i t y . On t r i a l s , i n t e r and i n t r a - r a t e r r e l i a b i l i t i e s were h i g h . The authors b e l i e v e tha t the combinat ion of r a t i n g s and reason g i ven f o r the r a t i n g s p rov ides c o n -s t r u c t i v e i n f o r m a t i o n f o r i n d i v i d u a l c o r r e c t i v e a c t i o n , f o r e v a l u a t i o n of t r a i n i n g programs, or f o r i d e n t i f y i n g s p e c i f i c needs f o r c o n t i n u i n g educat ion of d e n t a l p e r s o n n e l . The system cou ld be expanded to other types of r e s t o r a t i v e work. Adv ice on d e r i v i n g p r a c t i c e - r e l e v a n t outcome measures i s o f f e r -ed i n s e v e r a l a r t i c l e s . A number of authors ( 2 7 , 3 5 , 2 6 2 , 3 1 8 ) emphasize that p r a c t i t i o n e r s , not academics or r e s e a r c h e r s , should p lay the pr imary r o l e . McClure (318) mentions that a De lph i technique can be used w i t h p h y s i c i a n s to a r r i v e a t consensus about a p p r o p r i a t e outcomes. Johnson ( 2 6 2 ) , r a t h e r than r e l y on nurse judgement as to d e s i r a b l e r e s p i r a t o r y -78-s t a t u s a f t e r su rge ry , had nurses c o l l e c t data on each of the four p a r a -meters to be measured by observ ing a l a r g e sample of o p t i m a l l y r e c o v e r i n g p a t i e n t s . Another important c o n s i d e r a t i o n fo r those under tak ing s i t u a -t i o n - s p e c i f i c outcomes assessment i s the des ign and maintenance of c l i n i -c a l r e c o r d s . The t r a d i t i o n a l process o r i e n t e d c l i n i c a l r e c o r d , which tends to prov ide more i n f o r m a t i o n about what the p rov ide r d i d than about the h e a l t h s t a t e of the c l i e n t , does not r e a d i l y lend i t s e l f to an e f f i c i e n t system of outcome assessment. W i r t s h a f t e r and Messel (536) propose a m o d i f i c a t i o n to i n p a t i e n t records so tha t a "minimum care assurance data s e t " c o l l e c t s and o rgan izes i n f o r m a t i o n by d i a g n o s i s -s p e c i f i c outcome and r e l a t e d p a t i e n t management, thus f o r c i n g the d o c u -mentat ion of e a c h . S t r a t e g i e s fo r ensur ing tha t outcomes based e v a l u a t i o n w i l l have optimum impact have r e c e i v e d c o n s i d e r a b l e a t t e n t i o n i n the l i t e r a -t u r e . McGuire (321) p laces g reat emphasis on r a t i o n a l s e l e c t i o n of the t o p i c which w i l l be s t u d i e d . She advocates an approach i n which c o n d i -t i o n s r e s p o n s i b l e fo r the m a j o r i t y of the p o p u l a t i o n ' s d i s a b i l i t y are graphed i n terms of degree of d i s a b i l i t y a t each age i n three d i f f e r e n t ways: f i r s t , as they would e x i s t wi thout p revent ion or i n t e r v e n t i o n ; second, as they would e x i s t g iven f u l l a p p l i c a t i o n of present knowledge and r e s o u r c e s ; and t h i r d , as they c u r r e n t l y e x i s t i n the ta rget p o p u l a -t i o n . Smal l gaps between the f i r s t and second i n d i c a t e t a r g e t s f o r b i o m e d i c a l r e s e a r c h and/or resource a l l o c a t i o n ; l a r g e gaps between the second and t h i r d i n d i c a t e t a r g e t s f o r o r g a n i z a t i o n a l change and/or educa -t i o n a l i n t e r v e n t i o n . H i r s c h (240) b e l i e v e s tha t a d i a g n o s i s - s p e c i f i c outcomes approach i s more d e f i n i t i v e than i s necessary fo r a c h i e v i n g most d e s i r --79-able changes for least Investment. He proposes an evaluation of hospital care in terms of reasons for admission and objectives of management. In the preliminary study of five hundred consecutive discharges from five hospitals, i t was found that adult patients could be classified in one of twenty-five reasons - for - admission categories. Objectives for hospit-alization can be similarly defined within an assigned time frame. Each admitted patient is monitored against the expectations specified and, where a deviation i s apparent, the record i s reviewed. As this i s a con-current procedure, poor care is immediately corrected and practitioner learning i s highly relevant. McClure (318) recommends selecting for study patient conditions or problems which are frequent in occurrence, disabling, amenable to med-ic a l treatment, and probably not being adequately treated at present. He also emphasizes that several outcome evaluation studies should be admini-stered in an overlapping sequence and that greatest benefits in terms of improved patient care w i l l be achieved i f many groups of problems are studied in preference to continuing to refine the few already studied: "there i s too much sub-standard care to allow the luxury of refinement." Finally, there are a limited number of references which consi-der particularly useful applications of outcome based evaluation. Several (27,35,240) cite the use of sequential outcome measures as a guide to concurrent practice; Aydelotte (35) refers to these specifica-tions as "daily working material and part of the intellectual equipment of each nurse". Additionally such measures are used for patient teaching tools (27), establishing contracts with patients (127), identifying learning needs (27,85) and for specifying essential content for educa-tional programs (85). -80-G o n e l l a e t a l . (194) suggest that outcomes-based measures are good f o r comparat ive study of p o p u l a t i o n groups . H i s r e s e a r c h team e s t a b l i s h e d a s c a l e to measure s e v e r i t y of h e a l t h s t a t e f o r each s i g n i f i -cant stage i n the n a t u r a l h i s t o r y of a c o n d i t i o n , c o l l e c t e d data on p r o p o r t i o n s of p a t i e n t s a t each l e v e l on the s e v e r a l s c a l e s , then made i n f e r e n c e s about the q u a l i t y of care p r i o r to tha t p o i n t . T h i s concept has been a p p l i e d to e ighteen d i f f e r e n t c o n d i t i o n s . In the example r e -viewed ( a p p e n d i c i t i s ) , a s i g n i f i c a n t l y h igher p r o p o r t i o n of p a t i e n t s from "other government insurance schemes" had ruptured appendices a t admiss ion than d i d p a t i e n t s from Medicare or p r i v a t e i n s u r o r s , thus i n d i c a t i n g a problem w i t h p r i o r process f o r the f i r s t p o p u l a t i o n group. Summary and Comment Outcomes based e v a l u a t i o n of h e a l t h p r o f e s s i o n a l s i s p r e d i c a t e d on the s u p p o s i t i o n that i f d e s i r e d r e s u l t s i n terms of improved c l i e n t h e a l t h are achieved then competence can be assumed, and that i f d e s i r e d r e s u l t s are not a c h i e v e d , competence must be q u e s t i o n e d . The l i t e r a t u r e d e a l i n g w i t h outcomes-based assessment can be subd iv ided i n t o genera l measures of h e a l t h s t a t u s and s i t u a t i o n - s p e c i f i c outcomes e v a l u a t i o n . Genera l measures of h e a l t h s t a t u s have not been developed or used to any extent as an a i d i n e v a l u a t i n g competence of h e a l t h p e r s o n n e l . Most of those i n e x i s t e n c e have been used on ly f o r r e s e a r c h and are designed to app ly to ta rge t p o p u l a t i o n s , not to i n d i v i d u a l c l i e n t s . To the ex tent that a p r a c t i t i o n e r , or a group of p r a c t i t i o n e r s , i s s o l e c o n t r i b u t o r to the care of that p o p u l a t i o n , some t e n t a t i v e -81-generalizations might be made about the adequacy of performance. Because of the many factors a f f e c t i n g general health status however, i t may be concluded that these measurement approaches probably have more value for health care services planning than for any e f f e c t i v e contribution to c o n t r o l l i n g competence of health care professionals. To control for the numerous variables a f f e c t i n g the c l i e n t ' s health state, health care providers attempting evaluation of services have inc r e a s i n g l y defined outcomes i n terms of a few highly s i g n i f i c a n t i n d i c a t o r s relevant to the p r a c t i t i o n e r ' s a c t i o n . The l i t e r a t u r e which deals with s i t u a t i o n - s p e c i f i c outcomes evaluation r e f l e c t s preoccupation with the steps of the evaluative process i t s e l f . A d d i t i o n a l l y , i t i s concerned with providing c r i t e r i o n examples and resource information and with maximizing benefits through wise s e l e c t i o n of study topics and innovative use of the evaluation process. Incomplete patient care records have been i d e n t i f i e d as a b a r r i e r to e f f e c t i v e use of systematic s i t u a t i o n - s p e c i f i c outcomes assessment. It i s worth noting that l i t e r a t u r e dealing with an outcomes based approach to evaluation of health professionals i s small i n volume. Its primary l i a b i l i t y ( f a i l u r e to account for other v a r i a b l e s influencing health state) and i t s primary asset (the inherently appealing l o g i c of accepting r e s u l t s as the f i n a l proof of competence/incompetence) have influenced the mushrooming development of evaluation approaches which measure and judge adequacy of resources, professional behavior, and health outcomes i n varying combinations. Most of these e f f o r t s have at l e a s t some implications for assessing p r a c t i t i o n e r s ' competence. -82-Chapter V ASSESSING COMBINATIONS OF STRUCTURE, PROCESS AND OUTCOME I t i s p o s s i b l e to eva luate three broad parameters i n d i c a t i v e of h e a l t h p r o f e s s i o n a l s ' competence: s t r u c t u r e ( s k i l l s and knowledge p o s s e s -sed by the p r a c t i t i o n e r ) , process (behavior e x h i b i t e d i n p r a c t i c e ) , or outcome ( h e a l t h s t a t e of the p a t i e n t ) . In response to some of the advantages and l i m i t a t i o n s i n h e r e n t i n each of these ( t h i s t o p i c w i l l be d i s c u s s e d i n Chapter V I ) , the h e a l t h care i n d u s t r y has i n c r e a s i n g l y t u r n -ed to the use of e v a l u a t i o n s t u d i e s which focus on v a r y i n g combinat ions of these three a r e a s . Th is chapter d i s c u s s e s l i t e r a t u r e concerned w i t h such combined e v a l u a t i o n approaches under the f o l l o w i n g broad head ings : concurrent e v a l u a t i o n of v a r y i n g s t r u c t u r e - p r o c e s s - o u t c o m e combinat ions ; outcome assessment w i t h s e l e c t i v e e v a l u a t i o n of p r o c e s s ; and random e v a l u a t i o n of outcome and process of c a r e . I t should be noted that t h i s rev iew does not i n c l u d e s t u d i e s which attempt to e s t a b l i s h s t a t i s t i c a l c o r r e l a t i o n s between combinat ions of s t r u c t u r e , p r o c e s s , and outcome. These are more p r o p e r l y the concern of r e s e a r c h i n c l i n i c a l or h e a l t h care a d m i n i s t r a t i o n . Concurrent E v a l u a t i o n of S -P -0 Combinat ions . Th is s e c t i o n w i l l c o n s i d e r l i t e r a t u r e d e a l i n g w i t h each of s e v e r a l p o s s i b l e combinat ions of s t r u c t u r e , process and outcome -83-evaluation. Concurrent evaluation of a l l three parameters is considered f i r s t , followed by a discussion of structure-process combinations. The third topic is agency based process-outcome evaluation, and the fourth topic is regionally based process-outcome evaluation. Reports of struc-ture-outcome evaluation activities were not found in the literature. Concurrent Evaluation of Structure, Process and Outcome A comprehensive and lucid discussion of the theoretical advan-tages of concurrent evaluation of structure, process and outcome is offered by Aydelotte (36). Her frame of reference, however, is health care service, not health professionals' competence. Thus her comments on structure apply to resources, equipment etc. as well as to knowledge and s k i l l s held by practitioners. She advocates ongoing evaluation of the pattern of care (staff performance, provision of resources, etc.) against the effects care is supposed to have; such a system would predict prob-lems of patient groups, describe desired health outcomes for these patients, prescribe necessary practitioner activities to achieve such outcomes, and specify the resources (including knowledge and s k i l l s of practitioners) required for such a c t i v i t i e s . The health agency would work to ensure the presence of a l l of these elements in a continuing program of measurement and action. Like other authors, Aydelotte empha-sizes the importance of both defining the patient population to control for variables grossly influencing health outcome and selecting outcome indicators which relate to the problem under study, are influenced by health care, and reflect patient health status at c r i t i c a l points of care including exit from the system. A manual (409) published by the Regis-tered Nurses Association of British Columbia promotes structure-process--84-outcome evaluation i n an agency s e t t i n g by providing d i r e c t i o n s for establishing an evaluation system, a rat i o n a l e for these d i r e c t i o n s , sample c r i t e r i a and forms, and further references on the subject. There were found no accounts i n the l i t e r a t u r e of concurrent evaluation of structure, process and outcome where the focus within the structure or resource area i s that of p r a c t i t i o n e r s ' knowledge and s k i l l s . In f a c t , l i t e r a t u r e dealing with comprehensive evaluation of a l l three areas i s s u r p r i s i n g l y sparse even when the more common s t r u c t u r a l elements of equipment and f a c i l i t i e s are considered. Ramey (403) i n 1973 reported the development of such a t o o l f or nursing service, however there i s l i t t l e a t tention to outcomes. Glasson (187), four years l a t e r , described a more s i t u a t i o n - s p e c i f i c evaluation t o o l for nursing outcome and process and indicated that simultaneously the h o s p i t a l ' s " q u a l i t y control survey" measures for such u n i v e r s a l l y desirable s t r u c t u r a l e l e -ments as clean environment, complete charts, etc. It i s apparent that this i s not an integrated evaluation system i n which the s t r u c t u r a l e l e -ments chosen f o r measurement are deemed s p e c i f i c a l l y necessary f o r nurse performance i n r e l a t i o n to a given patient population. Selvaggi (466) described the experiences of a nursing service which conducted s t r u c t u r e -process evaluation and process-outcome evaluation simultaneously. Again, i t does not appear that the two a c t i v i t i e s were linked i n any e f f e c t i v e way. Schonfeld (456), i n discussing evaluation of dental care systems, proposed a framework which would encompass structure, process and outcome parameters. This a r t i c l e suggests evaluators must address four areas of concern: s p e c i f i c procedure, o r a l cavity, c l i e n t , and group; and four dimensions of resources: t e c h n i c a l , l o g i s t i c a l , organi-z a t i o n a l and f i n a n c i a l . -85-The Medicaid Vendor System (MVS) (434), a computer-based claims review system for the New York City Medicaid Program, i s an imaginative use of combined stucture-process-outcome evaluation on a regional b a s i s . Described as an " e c l e c t i c approach to q u a l i t y c o n t r o l , " i t c l e a r l y i s aimed at l o c a t i n g the greatest numbers of, the most s i g n i f i c a n t to health, and the most correctable deficiences i n medical care for l e a s t e f f o r t . B a s i c a l l y t h i s system i s oriented to process; the computer pr i n t s out exceptions to patterns of p r a c t i c e for the provider group or i t s s p e c i a l t y j and these are followed up by c l o s e r analysis of deviant i n d i v i d u a l provider claim p r o f i l e s . The s t a t i s t i c a l p r o f i l e s f o r p r o v i d -er groups are used also to i d e n t i f y aggregate deviations from acceptable standards. The l a t t e r indicates need for educational programs or for manipulation by changing incentives or the q u a l i f i c a t i o n s of providers able to claim reimbursement. This monitoring process i s supplemented by assessment of structure ( o f f i c e inspections of deviant p r a c t i t i o n e r s ) and outcome (routine follow-up sampling of patient h e a l t h ) . P r i o r i t i e s f or c o r r e c t i v e action are deviations from outcome. Within t h i s group those p r a c t i t i o n e r s with exceptional provider p r o f i l e s are investigated f i r s t . The MVS i s seen by the authors of t h i s a r t i c l e as an optimally pragmatic approach to evaluation. An annual saving of almost $2 m i l l i o n i s claim-ed, but this i s not a net f i g u r e . Structure-Process Combinations The l i t e r a t u r e search yielded few accounts of structure-process evaluation. A study team under contract to the American Heart Associa-t i o n produced process c r i t e r i a and related s t r u c t u r a l requirements f o r each stage of common conditions connected with heart disease, cancer and -86-stroke (171). An occupat ional hea l th group has produced s t ructure and process c r i t e r i a for p lanning , admini s ter ing , and implementing care by nurses i n occupat ional hea l th se t t ings (484). F i n a l l y , an extensive pro ject to develop a method for evaluat ing the s t ructure and process of nursing care i n acute general hosp i t a l s has been underway for some time and i s discussed below i n some d e t a i l . The Methodology for Monitor ing the Quality of Care, developed by the Medicus Corporat ion under a contract with the United States government, i s an instrument which attempts to evaluate a wide range of nursing resources and behaviors (225,226,232,260). It i s s t i l l under development. The pro jec t s t a f f synthesized and expanded on c r i t e r i a from e x i s t i n g process instruments such as QUALPACS, c o r r e c t i n g omissions and e l i m i n a t i n g redundancies and items without d i scr iminatory power. Items were placed i n a c l a s s i f i c a t i o n s t ructure of s i x major categories (four re la ted to nurs ing process , while the others assessed un i t management and support serv ices ) and severa l sub-categories . The e n t i r e instrument was subjected at successive stages to r igorous s t a t i s t i c a l ana lys i s to insure r e l i a b i l i t y , s e n s i t i v i t y , and i n t e r n a l consistency of the construct as a whole. The instrument s t ructure i s capable of expansion to incorporate a d d i t i o n a l items and/or factors unique to a given c l i n i c a l s i t u a t i o n . A l l c r i t e r i a are designated by a p p l i c a b i l i t y to type of p a t i e n t , i . e . s e l f care , p a r t i a l care , complete care , and in tens ive care ( i t was c o n s i -dered that items d i f f e r e d most i n a p p l i c a b i l i t y by i n t e n s i t y of i l l n e s s rather than by age, disease e t c . ) . The items are a l so coded by type of u n i t , i . e . general m e d i c a l - s u r g i c a l , recovery room, newborn nursery ; and by source of informat ion, i . e . char t , pa t ient in terv iew, nurse observa-t i o n e t c . -87-Th e assessment procedure i s i n i t i a t e d by generating an evalua-ti o n work sheet, from the master l i s t of over two hundred twenty c r i -t e r i a , which contains a sample of items relevant to the type of patient and unit to be assessed. This s e l e c t i o n process i s such that any one work sheet covers the en t i r e spectrum of possible items, but se l e c t s randomly from the major categories and sub-categories. This i s thought to make i t impossible f o r the nurses being evaluated to know which per-formance c r i t e r i a w i l l be assessed. With 10% of the unit's census being reviewed at random times, the process y i e l d s a computer calculated per-formance index ( t o t a l score and by major category) each month. The Medicus Quality Monitoring Methodology has been s u c c e s s f u l -l y applied i n nineteen p i l o t h o s p i t a l s of varying sizes and types and has required considerable e f f o r t to achieve an acceptable l e v e l of rater r e l i a b i l i t y . The method has been shown to r e f l e c t r e s u l t s of educational programs and changes i n unit leadership (226) and i s regarded as useful for i n s e r v i c e planning (172). Extensive use of the instrument as a t o o l for a c t u a l l y i d e n t i f y i n g performance problem areas and providing s u f f i -cient d i r e c t i o n for cor r e c t i v e action has not been reported. Thus, although regarded by many as the ultimate i n summative evaluation of group performance at the nursing unit l e v e l , the methodology has yet to prove i t s worth as a t o o l f o r group formative evaluation. The p r i n c i p a l investigators indicate that i t may be possible, through modification of the methodology, to provide performance p r o f i l e s of i n d i v i d u a l p r a c t i -tioners s u i t a b l e to peer review procedures (226). Attempts to v a l i d a t e c e r t a i n items on th i s instrument against outcomes i n patients with chronic heart f a i l u r e and abdominal hysterectomy have been moderately successful (226,230,231). -88-Agency B a s e d P r o c e s s - O u t c o m e E v a l u a t i o n A number o f g r o u p s a n d i n d i v i d u a l s a p p e a r t o h a v e r e c o g n i z e d t h e d e s i r a b i l i t y o f a c o m b i n e d p r o c e s s - o u t c o m e e v a l u a t i v e a p p r o a c h a n d t o h a v e d e v e l o p e d a s t a n d a r d p r o c e d u r e , o f t e n s u p p o r t e d by t e x t s , f o r m s , a n d w o r k s h o p s , t o i m p l e m e n t t h e i r c h o s e n a p p r o a c h i n a n agency s e t t i n g . D a v i d s o n (119) o f f e r s a n e n t i r e t e x t b o o k o f g u i d a n c e f o r n u r s e s a n d o t h e r s r e s p o n s i b l e f o r i n - h o u s e r e v i e w a c t i v i t i e s w h i c h w i l l meet PSRO e x p e c t a t i o n s . B a s i c i n f o r m a t i o n a b o u t t h e c h a r a c t e r i s t i c s and p u r -p o s e o f e a c h t y p e o f r e v i e w ( a d m i s s i o n c e r t i f i c a t i o n , c o n t i n u e d s t a y r e v i e w , r e t r o s p e c t i v e m e d i c a l c a r e e v a l u a t i o n s t u d i e s a n d p r o f i l e a n a l y -s i s ) i s g i v e n . P r e s e n t a t i o n o f a l l s u b j e c t s i s s u p p o r t e d w i t h r e f e r -e n c e s , t a b l e s , c h a r t s , and s a m p l e f o r m s . S i m i l a r r e f e r e n c e s a r e a v a i l -a b l e f o r e s t a b l i s h i n g r e v i e w s y s t e m s i n l o n g t e r m c a r e f a c i l i t i e s ( 2 5 1 , 3 1 1 ) . T h e C o m m i s s i o n on P r o f e s s i o n a l a n d H o s p i t a l A c t i v i t i e s ( C P H A ) , a M i c h i g a n b a s e d d a t a p r o c e s s i n g s e r v i c e f o r h o s p i t a l s , h a s l o n g p r o v i d e d i n f o r m a t i o n i n a f o r m w h i c h c a n b e m o n i t o r e d t o i n d i c a t e a d e q u a c y o f s e r v i c e ( 5 2 0 ) . I n 1972 t h e M e d i c a l A u d i t Study W o r k s h e e t was d i s s e m i n a -t e d t o a s s i s t h o s p i t a l s i n a s s e s s i n g t h e i r s e r v i c e p a t t e r n s a g a i n s t r e g i o n a l norms f o r t h e most common d i a g n o s t i c and t r e a t m e n t c a t e g o r i e s o f p a t i e n t s ; f o r e x a m p l e , n o r m a l a p p e n d i c e s r e m o v e d , a n t i b i o t i c s p r e s c r i b e d i n o u t - p a t i e n t d e p a r t m e n t s , e t c . were e a s i l y n o t e d i f d e v i a n t f r o m r e g i o n a l p r a c t i c e s . I n 1974 t h e C o m m i s s i o n i n c o r p o r a t e d e m e r g i n g e v a l u a -t i o n p r a c t i c e s i n t o a new s y s t e m c a l l e d Q u a l i t y A s s u r a n c e M o n i t o r (QAM). The QAM i s a c o m p u t e r i z e d d a t a r e t r i e v a l s y s t e m w h i c h a l l o w s a p a r t i c i p a -t i n g h o s p i t a l t o r e c e i v e r e t r o s p e c t i v e i n f o r m a t i o n a b o u t p a t t e r n s o f c a r e f o r d i a g n o s t i c a n d t r e a t m e n t g r o u p s a c c o u n t i n g f o r a b o u t 50% o f a l l - 8 9 -admiss ions a g a i n s t e i t h e r r e g i o n a l norms or i t s own s e l f - i m p o s e d s t a n d -a r d s . The i n f o r m a t i o n i s o rgan ized f o r rev iew as t o t a l h o s p i t a l data and a l s o by d i a g n o s i s , by s u r g i c a l procedure and by m e d i c a l s e r v i c e . The Commission prov ides a d v i c e , forms, and d i r e c t i o n on how agencies can e s t a b l i s h t h e i r own process and outcome c r i t e r i a , s y s t e m a t i c a l l y rev iew a c t u a l performance, p l a n to take c o r r e c t i v e a c t i o n , and document the e n t i r e process f o r e x t e r n a l l y imposed or i n t e r n a l a c c o u n t a b i l i t y r e q u i r e -ments. The Commission emphasizes the importance both of deve lop ing on ly those c r i t e r i a where c o r r e c t i v e a c t i o n w i l l be taken ( t h i s avo ids wander-i n g i n t o areas where research i s needed) and of remeasuring a f t e r c o r r e c t i v e a c t i o n ( t h i s avo ids pu rsu ing f r u i t l e s s c o r r e c t i v e a c t i o n ) . M a r t i n (329) s t u d i e d the use of the CPHA system i n one h o s p i t a l f o r a u d i t of care of s i x d i a g n o s t i c groups. H i s repor t ana lyzes cos ts and t e c h n i c a l problems, d i s c u s s e s p o t e n t i a l ex tens ions of the system and concludes that the c h i e f problems l i e not w i t h data m a n i p u l a t i o n and d i s -p l a y , but r a t h e r w i t h data c o l l e c t i o n , v a l i d i t y and i n t e r p r e t a t i o n . Holloway et a l . (244) compared the CPHA system to a manual system f o r c o l l e c t i n g data not conta ined on the face sheets of m e d i c a l r e c o r d s . The r e s u l t s i n d i c a t e d the CPHA system was l e s s c o s t l y i f more than 41% of h o s p i t a l i z e d p a t i e n t s were rev iewed; was as t i m e l y as the manual system; p rov ided fewer data elements than p h y s i c i a n s r e q u e s t e d ; and was l e s s p r o t e c t i v e of human e r r o r . A f t e r e v a l u a t i o n of the r e l a t i v e importance of these r e s u l t s , the manual system was recommended f o r implementat ion . Exper ience w i th t h i s system i s a l s o d e s c r i b e d by Chapin et a l . (93) . Th is two hundred and n i n e t y - s i x bed h o s p i t a l used the rev iew program to d i r e c t c o n t i n u i n g med ica l educat ion a c t i v i t i e s , which u s u a l l y took the form of i n f o r m a l group d i s c u s s i o n of r e s u l t s r a t h e r than s t r u c t u r e d - 9 0 -educat ion s e s s i o n s . Whi le the h o s p i t a l was s u f f i c i e n t l y s a t i s f i e d w i t h the system to expand i t s use to other a r e a s , whether or not change i n the d e s i r e d d i r e c t i o n a c t u a l l y occur red i s not d i s c u s s e d . The C a l i f o r n i a H o s p i t a l and M e d i c a l A s s o c i a t i o n s , supported by the C a l i f o r n i a Nurses A s s o c i a t i o n , have cooperated to develop t h e i r own approach to process-outcome e v a l u a t i o n of care (474) . Whi le designed to meet requirements of e x t e r n a l groups such as the J o i n t Commission of H o s p i t a l A c c r e d i t a t i o n and PSRO, t h i s C a l i f o r n i a rev iew system emphasizes the e d u c a t i o n a l aspects of a u d i t . The u s u a l e v a l u a t i o n - a c t i o n process i s r a t h e r f u l l y d e s c r i b e d i n t h i s a r t i c l e which u s e f u l l y i n c l u d e s such pragmatic comments as "be sure you can get data on your c r i t e r i a , " and "group r a t i f i c a t i o n of c r i t e r i a (as opposed to m a i l i n g or p o s t i n g f o r r a t i f i c a t i o n ) i s v a l u a b l e because i t promotes l e a r n i n g , ach ieves b e t t e r c r i t i q u e , a l l o w s a p u b l i c commitment to be made". The author a l s o c o n s i -ders the d i f f i c u l t s tep of problem a n a l y s i s more f u l l y than most , empha-s i z e s the importance of f i x i n g a c c o u n t a b i l i t y and d e a d l i n e s f o r c o r r e c t -i v e a c t i o n , and reminds readers of the l e g a l importance of keeping a u d i t records separate from m e d i c a l records and f r e e of data which breach c o n f i d e n t i a l i t y . Hanna (217) d e s c r i b e s the a p p l i c a t i o n of the C a l i f o r n i a system i n the n u r s i n g s e r v i c e of a two hundred bed community h o s p i t a l . Aga in the approach to problem a n a l y s i s i s more f u l l y d i s c u s s e d i n t h i s a r t i c l e than i s the u s u a l c a s e . In a d d i t i o n to the s p e c i f i c a l l y developed and s t r u c t u r e d v e r -s i o n s of process-outcome e v a l u a t i o n d e s c r i b e d above, a number of authors share exper iences w i t h t h i s genera l approach . Gold (190) r e p o r t s an uns t ruc tu red approach to peer rev iew by nurse c l i n i c i a n s u s i n g i m p l i c i t - 9 1 -c r i t e r i a and standards and concludes that es tab l i shment of e x p l i c i t p r o c -ess and outcome c r i t e r i a by p r a c t i t i o n e r s i n the s p e c i a l t y a rea i s a c r i t i c a l requirement f o r s u c c e s s f u l peer rev iew a c t i v i t y . Medicus Systems (79) p rov ides d i r e c t i o n to nurses f o r e s t a b l i s h i n g p r o c e s s -outcome c r i t e r i a , conduct ing a p a t i e n t care e v a l u a t i o n s t u d y , and p l a n -n ing c o r r e c t i v e a c t i o n . R e l a t i o n s h i p s w i t h n u r s i n g care p lans and problem o r i e n t e d records are a l s o i l l u s t r a t e d . E th r idge and Packard (160) d e s c r i b e i n c o n s i d e r a b l e d e t a i l the i n t e g r a t i o n of three a d m i n i s t r a t i v e t o o l s of n u r s i n g s e r v i c e ( a u d i t , care p l a n s , p a t i e n t records ) i n t o a system f o r ongoing s i t u a t i o n - s p e c i f i c e v a l u a t i o n of c a r e . In some i n s t a n c e s process and outcome are combined i n one s tudy , and i n other cases the a u d i t may concern i t s e l f w i t h on ly process or outcome. P r i o r i t y i n a n a l y s i s and c o r r e c t i o n i s g i ven to s t u d i e s showing unacceptable outcomes and, w i t h i n t h i s group, those c h a r t s showing unacceptab le process are attended f i r s t ; thus the most c l e a r l y unacceptable s i t u a t i o n s r e c e i v e a t t e n t i o n even when resources are s c a r c e . Shanahan (469) r e p o r t s the use of process and outcome c r i t e r i a to assess the care g i ven to one p a t i e n t group (acute myocard ia l i n f a r c -t i o n ) by s e v e r a l d i f f e r e n t n u r s i n g u n i t s i n the same h o s p i t a l at d i f f e r -ent stages of i l l n e s s . Whi le the i n i t i a l g o a l of t h i s p r o j e c t , which was to determine whether n u r s i n g performance i n one a rea e f f e c t e d p a t i e n t outcomes i n another a r e a , was not r e a l i z e d , the committee concluded that conduct ing a u d i t s which measure n u r s i n g care p rov ided to one group of p a t i e n t s at d i f f e r e n t stages of h o s p i t a l i z a t i o n has great p o t e n t i a l , and that e v a l u a t i o n by u n i t or department a lone p l a c e s unnecessary l i m i t a -t i o n s on e f f o r t s to improve c a r e . P h a r m a c i s t s ' s e r v i c e s were assessed by Keys et a l . (272) i n an i n t e r e s t i n g study i n which independent rev iewers u s i n g i m p l i c i t s tandards - 9 2 -cons idered adequacy of drug communications i n terms of p o t e n t i a l f o r b e n e f i t t i n g p a t i e n t and of a c t u a l impact on p a t i e n t h e a l t h . The method showed a h igh degree of i n t e r - r e v i e w e r r e l i a b i l i t y and moderate c o r r e l a -t i o n s between process and outcome. The h i g h i n t e r - r a t e r r e l i a b i l i t y was l i k e l y ach ieved because rev iewers d i s c u s s e d the i n f o r m a t i o n a b s t r a c t e d from the char t before proceeding to an independent judgment of adequacy. M e d i c a l care e v a l u a t i o n u s i n g combined process-outcome i n a r u r a l f i f t y bed h o s p i t a l i s d e s c r i b e d by F i t z g e r a l d (165) . This a r t i c l e would be of i n t e r e s t to s m a l l opera t ions w i t h l i m i t e d r e s o u r c e s ; the h o s p i t a l ' s system meets the requirements of e x t e r n a l bodies as w e l l as that of i t s board of d i r e c t o r s f o r an a c c o u n t a b i l i t y mechanism. Format f o r data d i s p l a y sheets and summary of a u d i t r e s u l t s a re i l l u s t r a t e d . In a more complex approach to m e d i c a l care e v a l u a t i o n , Rosenberg (432,433) proposes a format f o r c r i t e r i a which overcomes the problems a s s o c i a t e d both w i t h long l i s t s of process c r i t e r i a to cover any e v e n t u a l i t y and w i t h shor t term outcomes-only c r i t e r i a which ignore long term r e s u l t s of c a r e . H i s d e t a i l e d format f o r d i s e a s e - s p e c i f i c c r i t e r i a i n c l u d e s c r i t i -c a l d i a g n o s t i c c o n s i d e r a t i o n s , p o s s i b l e a s s o c i a t e d prob lems, p a t t e r n s of t reatment , expected response to treatment (shor t term outcome, c o m p l i c a -t i o n s , long term outcome), d i s p o s i t i o n management, and c e r t a i n u t i l i z a -t i o n c r i t e r i a ( i n d i c a t i o n s f o r a d m i s s i o n , d i s c h a r g e , and leng th of s t a y ) . The author acknowledges the method's drawbacks: the need f o r s u b s t a n t i a l resources ( e . g . twelve minutes/char t i s used to a b s t r a c t n e c -essary data) and the p o s s i b l e u n a c c e p t a b i l i t y of c l o s e r e g u l a t i o n of p r o c e s s . The l a t t e r might not be a problem i f c r i t e r i a were se t w i t h i n the agency, but e x t e r n a l l y imposed c r i t e r i a of t h i s nature are not l i k e l y to be t o l e r a t e d . - 9 3 -When one cons ide rs that most outcomes are a f f e c t e d by a c t i o n s of many h e a l t h care workers , i t i s s u r p r i s i n g that the l i t e r a t u r e c o n -t a i n s r e l a t i v e l y few re fe rences to m u l t i - d i s c i p l i n a r y process-outcome a u d i t . Davidson (119) r e f e r s to the need f o r m u l t i - d i s c i p l i n a r y i n v o l v e -ment i n a u d i t , but the context f o r t h i s r e f e r e n c e i s j u s t i f i c a t i o n f o r nurse p a r t i c i p a t i o n i n medica l a u d i t . Dunham (147) a l s o supports the concept and r e p o r t s that one h o s p i t a l , u s i n g the C a l i f o r n i a system desc r ibed e a r l i e r , employs m u l t i - d i s c i p l i n a r y rev iew of outcome and p rov ides f o r u n i - d i s c i p l i n a r y process a u d i t e i t h e r c o n c u r r e n t l y or s u b -s e q u e n t l y . In another h o s p i t a l of two hundred and twenty beds , m e d i c a l , n u r s i n g and sometimes o ther s t a f f s e p a r a t e l y e s t a b l i s h e d t h e i r own process and outcome c r i t e r i a (communication about developments i s p r o v i d -ed f o r through c r o s s - r e p r e s e n t a t i o n on commit tees ) , but then a n a l y s i s of a l l r e s u l t s of a l l a u d i t s i s conducted i n a m u l t i - d i s c i p l i n a r y conference which y i e l d s j o i n t s t a f f recommendations (283) . The author notes that t h i s confe rence , which i s v i d e o - t a p e d , serves as an e f f e c t i v e e d u c a t i o n a l r e s o u r c e . U n f o r t u n a t e l y the e f f i c a c y of t h i s approach i n terms of chang-ed s e r v i c e p a t t e r n s i s not d i s c u s s e d . In a ra re comparison of two p a t i e n t care ( j o i n t m e d i c a l -nurs ing ) a u d i t s f o r d i a b e t i c k e t o a c i d o s i s , one i n a t e a c h i n g center and the other i n a s m a l l community h o s p i t a l , rev iewers concluded that m u l t i -d i s c i p l i n a r y a u d i t was e f f e c t i v e and c l i n i c a l input d i d not vary s u b s t a n -t i a l l y w i t h h o s p i t a l d i f f e r e n c e , but m o d i f i c a t i o n s of c r i t e r i a needed to be made a c c o r d i n g to l o c a l norms and i n s t i t u t i o n a l p o l i c y (101) . E q u a l l y important i s the a u t h o r s ' o b s e r v a t i o n that w h i l e a u d i t serves to i d e n t i f y d e f i c i e n c i e s , i t cannot d i c t a t e c o r r e c t i v e responses ; h e a l t h p r o f e s s i o n -a l s must c o r r e c t problems by pu rsu ing methods that r e f l e c t l o c a l needs - 9 4 -and a v a i l a b l e r e s o u r c e s . F i n a l l y , the North Dakota S t a t e H o s p i t a l (185) repor ts m u l t i - d i s c i p l i n a r y a u d i t s of char ts u s i n g a check l i s t of process and outcome v a r i a b l e s . C r i t e r i a and standards are i m p l i c i t ; review and f o l l o w - u p a c t i o n a re handled u s i n g d i s c u s s i o n procedures which the author b e l i e v e s to be an e f f e c t i v e form of c o n t i n u i n g e d u c a t i o n . Th is c o n c l u -s i o n i s not supported by e v i d e n c e . R e g i o n a l l y Based Process-Outcome E v a l u a t i o n Claims review mechanisms account f o r most r e g i o n a l l y based process-outcome e v a l u a t i o n developments to d a t e . The Utah P r o f e s s i o n a l Review O r g a n i z a t i o n (UPRO) has developed a system which p r o v i d e s d i a g n o s i s - s p e c i f i c concurrent review to a s c e r t a i n p r o p r i e t y of h o s p i t a l admiss ion and to a s s i g n a norm-based length of s tay (356) . At the same time t h e r a p e u t i c management o b j e c t i v e s , s t a t e d i n the form of h e a l t h o u t -comes, are e s t a b l i s h e d . Outcomes data are c o l l e c t e d throughout the h o s p i t a l s tay and on d i s c h a r g e ; i d e n t i f i e d d e v i a t i o n s p o i n t to areas f o r a n a l y s i s and c o r r e c t i v e a c t i o n . P r e l i m i n a r y exper ience shows a h a l f - d a y r e d u c t i o n i n l e n g t h of s tay and a s l i g h t decrease i n admiss ion r a t e s at a cost of $9 per admiss ion rev iewed . Where a n a l y s i s of a u d i t r e s u l t s i n d i c a t e s e d u c a t i o n a l needs, the i n f o r m a t i o n i s r e f e r r e d to a s t a t e - w i d e med ica l c o n t i n u i n g educat ion f o u n d a t i o n . The author n o t e s , however, tha t the e f f e c t i v e n e s s of e d u c a t i o n a l i n t e r v e n t i o n has yet to demonstrated. A C a l i f o r n i a based PROSO prototype (204) has developed " c r i t e r -i a mapping" as a process-outcome e v a l u a t i o n technique which min imizes the problem of t r y i n g to apply a g e n e r a l l i s t of d i a g n o s i s - r e l a t e d c r i t e r i a to a s p e c i f i c p a t i e n t . Th is method uses s e q u e n t i a l judgments based on s p e c i f i c c l i n i c a l data about the i n d i v i d u a l p a t i e n t to assess q u a l i t y , - 9 5 -thus l e a v i n g room f o r a l t e r n a t e m e d i c a l d e c i s i o n s . I t i s expected that t h i s method w i l l more a c c u r a t e l y r e f l e c t p h y s i c i a n i n t e n t and tha t p r o c -e s s , when measured by t h i s approach, may c o r r e l a t e b e t t e r w i t h outcome. Denta l c l a i m s rev iew has found combined process-outcome e v a l u a -t i o n a t the r e g i o n a l l e v e l an e s p e c i a l l y u s e f u l approach. DeJong and Dunning (121) repor t that twenty years exper ience i n the Un i ted Kingdom has cu lminated i n a system where in p r a c t i c e p a t t e r n s of d e n t i s t s are monitored through comparison w i t h norms f o r the p r a c t i t i o n e r group. S i g n i f i c a n t d e v i a t i o n s are ana lyzed more c a r e f u l l y , and t r e a t e d p a t i e n t s r e c a l l e d so that completed treatment may be examined. Th is genera l approach has been extended to p r o s p e c t i v e review of proposed d e n t a l therapy i n many American d e n t a l insurance schemes ( 5 0 , 1 7 5 , 4 9 4 ) . These mechanisms are c h a r a c t e r i z e d by : 1. P r i o r a u t h o r i z a t i o n of the p l a n of treatment based on submit ted x - r a y s and examinat ion f i n d i n g s . 2. Rout ine m o n i t o r i n g of q u a l i t y of t reatment (on a l l c l a i m s , a l l c l a i m s over a g i ven d o l l a r f i g u r e , or a sample of a l l c l a i m s ) through exam-i n a t i o n of p o s t - t r e a t m e n t x - r a y s , supplemented by d i r e c t examinat ion of a sample of p a t i e n t s and/or computer a n a l y s i s of p r a c t i c e p r o -f i l e s . 3 . Fo l low -up on a l l c o m p l a i n t s . 4. A graded system of d e c i s i o n making wherein the m a j o r i t y of rev iew d e c i s i o n s are made by d e n t a l a u x i l i a r i e s w i t h p r o v i s i o n f o r appeal f i r s t to s t a f f d e n t i s t s , then to d e n t a l c o n s u l t a n t s , and f i n a l l y to a peer rev iew committee. Th is i s u s u a l l y operated by the r e g i o n a l or s t a t e d e n t a l s o c i e t y . There e x i s t s i n the d e n t a l l i t e r a t u r e a number of p roposa ls and genera l g u i d e l i n e s f o r the es tab l i shment and o p e r a t i o n of these peer rev iew bodies ( 3 9 , 6 1 , 3 7 2 , 4 4 9 ) . There has been an i n c r e a s i n g e f f o r t to s p e c i f y f i r m and l o c a l l y r e l e v a n t treatment c r i t e r i a (449 ,494) , however most rev iew programs have r e l i e d on a l r e a d y developed t o o l s to e v a l u a t e -96-t e c h n i c a l q u a l i t y of care ( 1 7 4 , 4 5 0 ) . Of p a r t i c u l a r i n t e r e s t as a re fe rence i n the e v a l u a t i o n of d e n t a l care i s F r iedman's A Guide f o r E v a l u a t i o n of Denta l Care (173) , which p r e s c r i b e s e v a l u a t i v e s t r a t e g i e s i n c o n s i d e r a b l e d e t a i l . Inc luded are d i s c u s s i o n s of the important f e a t u r e s to be c o n s i d e r d i n e v a l u a t i n g d e n t a l p r a c t i c e , process and outcome c r i t e r i a f o r a s s e s s i n g t e c h n i c a l q u a l i t y of c a r e , forms and a s c o r i n g system f o r d i r e c t and i n d i r e c t measurement, and t a b l e s which c l a s s i f y the e v a l u a t i v e i n f o r m a t i o n one might seek i n a s s e s s i n g o v e r a l l d e n t a l programs. Friedman has a l s o developed a d e n t a l care index (174) which e s t a b l i s h e s process and outcome c r i t e r i a f o r e v a l u a t i o n of adequacy of d e n t a l s e r v i c e s to a t a r g e t popu-l a t i o n ; as such t h i s t o o l assesses the competence of the d e l i v e r y system (one component of which i s competence of a group of d e n t i s t s ) . B a i l l e t et a l . (38) repor t the development and t e s t i n g of a system f o r d e n t a l assessment which c o n s i d e r s h i s t o r y , p h y s i c a l exam, d i a g n o s i s , t reatment p l a n , and treatment g i v e n . They used recogn ized d e n t a l p r a c t i t i o n e r s to e s t a b l i s h normat ive c r i t e r i a . Both d i r e c t and i n d i r e c t data c o l l e c t i o n by t r a i n e d rev iewers showed that h i g h r e l i a b i l -i t y was achieved f o r most i t e m s . Content v a l i d i t y was e s t a b l i s h e d and concurrent v a l i d i t y was s a t i s f a c t o r y as evidenced by low, but s i g n i f i -c a n t , c o r r e l a t i o n s w i t h a s u b j e c t i v e e v a l u a t i o n method. The authors g i ve d e t a i l s on development as w e l l as t r a i n i n g and e v a l u a t i o n t i m e , and c o n -c lude that t h i s method i s reasonable as an approach to l a r g e s c a l e e v a l u -a t i o n of d e n t a l care and to needs i d e n t i f i c a t i o n i n c o n t i n u i n g e d u c a t i o n . Another approach to e v a l u a t i o n of d e n t a l care i s that used by the Uni ted States Ind ian Hea l th Serv i ce ( 2 ) . Assessment of t e c h n i c a l q u a l i t y of s e r v i c e , q u a l i t y of care ( i n t e r p e r s o n a l r e l a t i o n s , t e a c h i n g - 9 7 -e t c . ) and q u a l i t y of s e r v i c e to community i s i n c l u d e d . W i t h i n each area v a r i a b l e s are chosen and c r i t e r i a s t a t e d ( e . g . contour approximates normal anatomy) and standards are s p e c i f i e d ( e . g . 85% of a l l r e s t o r a t i o n s must meet c r i t e r i a ) . The c r i t e r i a tend to be a mix tu re of process and outcome and appear to have been s e l e c t e d on the b a s i s of the ease w i t h which data can be c o l l e c t e d . E v a l u a t i o n and f o l l o w - u p c o u n s e l l i n g a re c a r r i e d out p r i m a r i l y i n a s u p e r v i s o r - s u b o r d i n a t e r e l a t i o n s h i p . The Ind ian Heal th S e r v i c e scheme i s most n o t a b l e f o r i t s very comprehensive approach to e v a l u a t i n g a l l aspects and l e v e l s of the d e l i v e r y of d e n t a l s e r v i c e . Outcome Assesment With S e l e c t i v e E v a l u a t i o n of Process Whi le r e c o g n i z i n g the importance of i n f o r m a t i o n about process i f one i s to c o r r e c t inadequate outcomes of h e a l t h c a r e , there are a growing number of people who b e l i e v e that the most cos t e f f e c t i v e p r o c e -dure i s to then eva lua te f i r s t f o r outcome, and, i f d e f i c i t s are found, to assess the same sample a g a i n s t process c r i t e r i a i n order to l o c a t e the l i k e l y cause. Th is s e c t i o n w i l l d i s c u s s exper iences i n outcome a s s e s s -ment w i t h s e l e c t i v e e v a l u a t i o n of p r o c e s s . The o r i g i n a t o r of t h i s g e n e r a l approach seems to be W i l l i a m s o n (534) , who advocates a system of p r i o r i t i e s f o r four types of e v a l u a t i v e a c t i o n . W i l l i a m s o n b e l i e v e s that g r e a t e s t impact on h e a l t h f o r the popu-l a t i o n r e c e i v i n g care w i l l be achieved by promoting c o r r e c t and e a r l y d i a g n o s i s so as to i n c r e a s e the p r o p o r t i o n s of p r e v e n t a b l e or c o r r e c t a b l e c o n d i t i o n s . Thus the f i r s t p r i o r i t y would be to eva lua te f o r c o r r e c t - 9 8 -d i a g n o s i s , both missed cases (or f a l s e negat i ves ) and cases diagnosed f o r the c o n d i t i o n i n c o r r e c t l y ( f a l s e p o s i t i v e s ) ; t h i s of course means r e -d iagnos ing the p o p u l a t i o n r e c e i v i n g care a f e a t apparent l y not attempted by h e a l t h agencies d e s p i t e W i l l i a m s o n ' s t h e s i s . The second p r i o r i t y would be to e v a l u a t e f o r inadequate outcomes of c a r e ; a g a i n , concern f o r f i n d i n g c o s t - e f f e c t i v e ways to impact h e a l t h s t a t e has prompted W i l l i a m s o n to suggest that outcomes should be assessed u s i n g a s c a l e of f u n c t i o n a l h e a l t h r a t h e r than d i a g n o s i s - s p e c i f i c outcome meas-u r e s . H is suggested s c a l e i s s i x l e v e l s : no impairment ; measurable impairment , asymptomatic ; symptomatic, w o r k i n g ; not w o r k i n g , ambulatory ; b e d r i d d e n ; dead. T h i r d and f o u r t h p r i o r i t i e s would be e v a l u a t i o n of process when degree of accuracy i n d i a g n o s i s or degree of p a t i e n t f u n c -t i o n i n outcome i s unacceptab le . Schroeder (458) d e s c r i b e s one attempt to use W i l l i a m s o n ' s approach i n a h e a l t h mainta inance o r g a n i z a t i o n . Three p a t i e n t p o p u l a -t i o n s were s e l e c t e d f o r s t u d y : d e p r e s s i o n , requirements f o r c o n t r a c e p -t i o n , and h y p e r t e n s i o n . The i n v e s t i g a t i o n r e v e a l e d widespread under -d i a g n o s i s (44 - 74%) and inadequate t h e r a p e u t i c outcomes i n two p a t i e n t groups. There was c o n s i d e r a b l e d i f f i c u l t y w i t h data c o l l e c t i o n . The i n v e s t i g a t o r s chose c o n t r a c e p t i o n to f o l l o w through the a n a l y s i s -c o r r e c t i o n - r e e v a l u a t i o n c y c l e . D e s p i t e s e v e r a l c o r r e c t i v e a c t i v i t i e s i n c l u d i n g e d u c a t i o n , program and personne l changes, no improvement was documented on reassessment . I t was concluded that f u r t h e r ref inement of t h i s approach i s r e q u i r e d before i t s widespread use i s f e a s i b l e f o r improving or c o n t r o l l i n g q u a l i t y . W i l l i a m s o n ' s s t r a t e g y of e v a l u a t i n g outcomes then process has had c o n s i d e r a b l e i n f l u e n c e i n process-outcome e v a l u a t i o n approaches. The - 9 9 -MVS system (434) and one n u r s i n g s e r v i c e e v a l u a t i o n system (160) , both descr ibed e a r l i e r , used t h i s p r i o r i t y system f o r a n a l y s i s of s i m i l t a n -eous ly c o l l e c t e d d a t a . More common i s c o l l e c t i o n of outcomes d a t a , and, when d e v i a t i o n s o c c u r , subsequent c o l l e c t i o n of p rocess d a t a . N i c h o l s (363) d i s c u s s e s t h i s as a p p l i e d to n u r s i n g . She p o i n t s out that the care p l a n w i t h p a t i e n t o r i e n t e d o b j e c t i v e s (outcomes) and nurse o r i e n t e d p lans (process) p rov ides the e v a l u a t i v e framework. Outcomes should be assessed and, i f c r i t e r i a are not s u f f i c i e n t l y met, process c r i t e r i a should be measured to p r o v i d e d i r e c t i o n f o r c o r r e c t i v e a c t i o n . I f the process c r i t e r i a i n t h i s s i t u a t i o n are found to be met, t h i s i s ev idence that the p l a n of care i t s e l f i s inadequate or o ther i n f l u e n c i n g v a r i a b l e s are not c o n t r o l l e d . G o n e l l a (193) b u i l d s on the concept of outcome then process e v a l u a t i o n by propos ing a way of i d e n t i f y i n g the many process v a r i a b l e s which may have c o n t r i b u t e d to inadequate outcomes. H is twelve c e l l framework f o r outcome-process s t u d i e s cons ide rs outcomes i n terms of three dimensions (degree of p r e v e n t i o n , degree of p a t i e n t d y s f u n c t i o n , and degree of f a m i l y and community d y s f u n c t i o n ) and four c a t e g o r i e s of c o n t r i b u t i n g process v a r i a b l e s ( p h y s i c i a n performance, h o s p i t a l p e r f o r m -ance, and i n f l u e n c e of p h y s i c a l and s o c i a l env i ronment ) . The M e d i c a l A u d i t (MA) p o r t i o n of the American H o s p i t a l A s s o c i -a t i o n ' s Q u a l i t y Assurance Program a l s o uses W i l l i a m s o n ' s approach. As d e s c r i b e d by Brown and Sa le (72) MA i s c h a r a c t e r i z e d by an o r g a n i z a t i o n a l s t r u c t u r e which i s c o n t r o l l e d by med ica l s t a f f r e p o r t i n g through the a d m i n i s t r a t i o n to t r u s t e e s . The program e s t a b l i s h e s c r i t e r i a f o r c o n -f i r m a t i o n of d i a g n o s i s , management of therapy , and outcome. Outcomes s c r e e n i n g i s done f i r s t and d e v i a t i o n s are submit ted to process s c r e e n -- 1 0 0 -i n g . A l l d e v i a t i o n s are s t u d i e d and c o r r e c t i v e a c t i o n of an a d m i n i s t r a -t i v e , e d u c a t i o n a l , or d i s c i p l i n a r y nature may be t a k e n . A insworth ( 6 , 8 ) , i n d i s c u s s i n g t h i s program, emphasizes the importance of s e l e c t i n g study t o p i c s of h igh i n c i d e n c e or where outcomes p r e s e n t l y show h i g h cost or m o r t a l i t y . With the p o s s i b l e e x c e p t i o n of PSRO l e g i s l a t i o n , the g r e a t e s t i n f l u e n c i n g f a c t o r i n e v a l u a t i o n of h e a l t h s e r v i c e s appears to be the a c c r e d i t a t i o n requirements of the J o i n t Commission on A c c r e d i t a t i o n of H o s p i t a l s (JCAH). Hartman (221) overviews the h i s t o r i c a l development of e v a l u a t i v e approaches p r i o r to present day requ i rements . P o r t e r f i e l d (397) s t a t e s that present JCAH standards r e q u i r e an a u d i t system that i s f l e x i b l e , e f f i c i e n t , c l i n i c a l l y sound, capable of documentat ion, and a c t i o n o r i e n t e d . The system must be based on measurement of e x p l i c i t c r i t e r i a , and i t i s recommended tha t the a u d i t be a r e t r o s p e c t i v e outcome a u d i t (223) . Rosenberg (431) emphasizes that an i n s t i t u t i o n which compl ies w i t h the JCAH a u d i t requirements w i l l l i k e l y a l s o s a t i s f y requirements f o r PSRO r e v i e w . To a s s i s t agencies to meet these s t a n d -a r d s , the JCAH has developed a model performance e v a l u a t i o n procedure (PEP) . The PEP Pr imer (265) i s a d e t a i l e d handbook which d e s c r i b e s a c t i o n s to be taken to implement an e f f e c t i v e p a t i e n t care a u d i t program. In a d d i t i o n to d i r e c t i o n s and suppor t ing forms f o r an a u d i t p rocedure , such r e l a t e d t o p i c s as data r e t r i e v a l , c i v i l l i a b i l i t y , and records laws are d i s c u s s e d . The PEP methodology i s e s s e n t i a l l y an outcomes based a u d i t w i t h b u i l t - i n p r o v i s i o n f o r s e l e c t i v e e v a l u a t i o n of process i n that p a t i e n t care i s s c r u t i n i z e d when c o m p l i c a t i o n s o c c u r . I f u n j u s t i f i a b l e v a r i a t i o n s are i d e n t i f i e d , a n a l y s i s of cause i s urged and the process aud i t i s used at t h i s p o i n t as needed. - 1 0 1 -D i s c u s s i o n s of the PEP methodology as a p p l i e d i n h o s p i t a l s throughout the country are numerous and y i e l d l i t t l e i n f o r m a t i o n not d i s c u s s e d i n p rev ious c h a p t e r s . Knowlton (279) urges i n i t i a t i n g PEP by choosing n o n - c o n t r o v e r s i a l t o p i c s , where i t i s u n l i k e l y tha t d e f i c i e n c i e s w i l l be i d e n t i f i e d , as a way of ca lming s u s p i c i o u s p h y s i c i a n s . Ashbaugh and McKean (32) applaud PEP as a sound and e f f i c i e n t approach to q u a l i t y c o n t r o l , but p o i n t out i t should not be e x c l u s i v e l y r e l i e d on to i d e n t i f y c o n t i n u i n g educat ion needs. Such educat ion would be remedia l w i thout promoting new advances i n medic ine s u f f i c i e n t l y . T h e i r h o s p i t a l s u p p l e -ments a u d i t based c o n t i n u i n g med ica l e d u c a t i o n w i t h b i - w e e k l y l i t e r a t u r e review meetings and/or guest s p e a k e r s . A n a l y t i c a l d i s c u s s i o n s of c r i t e r -i a se ts developed a c c o r d i n g to the PEP format are the focus of a d d i t i o n a l a r t i c l e s concerned w i t h review of s e r v i c e s to p a t i e n t s w i t h acute and ch ron ic o t i t i s media (505) ; acute myocard ia l i n f a r c t i o n (154) ; t o n s i l l e c -tomy w i t h adenoidectomy ( 4 ) ; and pulmonary t u b e r c u l o s i s ( 3 ) . Whi le the PEP Primer appears to have been written primarily f o r m ed ica l a u d i t , nurses f i n d the procedure readily adaptab le to e v a l u a t i o n of nurs ing s e r v i c e (223 ,465) . D i d d i e (137) supports n u r s i n g ' s use of the d i s e a s e focus i n s e l e c t i n g a p a t i e n t p o p u l a t i o n because c h a r t s are e a s i l y r e t r i e v e d and because t h i s focus l a y s a base f o r f u t u r e m u l t i - d i s c i p l i n -ary c o l l a b o r a t i o n . She a l s o emphasizes that c r i t e r i a should o r i g i n a t e w i t h p r a c t i t i o n e r s but be supported by recent l i t e r a t u r e . Reproduct ion of forms used by her h o s p i t a l i n the v a r i o u s a u d i t procedure s teps makes D i d d i e ' s a r t i c l e a u s e f u l r e f e r e n c e . R i n a l d i and Rubin (422) s t a t e tha t a n a l y s i s of d e v i a t i o n s i d e n t i f i e d by PEP should i n v o l v e the p r a c t i t i o n e r s concerned and can use such r e a d i l y a v a i l a b l e process and s t r u c t u r e i n f o r m a t i o n as saved assignment s h e e t s , d a i l y census r e p o r t s , and process -102-audits; follow-up action should include positive feedback and praise for excellence. The nursing literature also contains analysis of c r i t e r i a sets based on the PEP format for acute myocardial infarction (468) and essential hypertension (470). A pharmacy audit committee has applied the PEP methodology to a drug-specific audit (27 3). The group chose use of Warfarin in pulmonary embolus because of i t s potential for adverse effects, frequency of use and application to future disease-specific audits. They found on analy-sis that deviations were due to inadequate c r i t e r i a , unacceptable profes-sional practice of doctors and nurses, and poor documentation. Pharmacy shares with nursing the problem of retrieval of records focussing on a study topic other than disease or surgical procedure: i t was found impossible to extract charts of a l l patients receiving warfarin. In completing the review, of literature concerned with evaluat-ing combinations of structure, process and outcome, i t should by noted that the medical profession in the United States has produced a number of model c r i t e r i a sets which can be used as references by those planning medical care evaluation studies (10,13,14,15,530). Random Evaluation of Outcome and Process of Care Unplanned evaluation of process and outcome of care w i l l occur when i t is known or suspected that outcomes of care are poorer than they should have been. Under tort law, professionals may be held liable for such damages as injury, unnecessary suffering, or death resulting from professional negligence. For c i v i l l i a b i l i t y to be established, three -103-conditions must exist: damage, negligence, and a causal relationship (472). In this instance, the courts become evaluator and the standard for practitioner behavior is that of a reasonable prudent practitioner with the same experience and training (201,399). While formerly common practice in the community was accepted as the behavior of a reasonably prudent practitioner, this is no longer so. In 1968 a Massachusetts court ruled that a practitioner is required to follow a general standard of care applicable to a l l specialists in his f i e l d (94,242). Tort law exists primarily to provide compensation for wrongs done to a person (95). It is d i f f i c u l t to know to what extent fear of l i t i g a t i o n promotes careful competent practice, and the literature rarely considers this as a mechanism for evaluating or controlling competence of health care workers. Pritchard (399) outlines several problems associa-ted with using tort law to ensure competence. The most important of these are high costs of defensive medicine (the certain result of a situation which combines the absence of competitive market pressures and the presence of increased threat of law suits); the freedom that the incompetent practitioner has to continue in practice until damage has been done and proved; and the necessity for the victim to recognize his injury, know i t was the result of negligence, and depend on the testimony of professionals (whose malpractice insurance premiums w i l l rise with every successful suit) to prove negligence. Coroner laws, which provide for investigation of unexpected and suspicious deaths, are limited compensation at best for this last defect (292). If there are few who expect _ the c i v i l l i a b i l i t y system to control competence, there are a growing number who advocate aggressive monitoring of professionals' competence as the only way to control sky-- 1 0 4 -r o c k e t i n g m a l p r a c t i c e se t t lements and insurance premiums ( 2 2 , 1 8 2 ) . There has been some hope expressed that the PSRO l e g s i s l a t i o n , which moni to rs and i n f l u e n c e s c e r t a i n p r a c t i t i o n e r behav io rs w h i l e s i m u l t a n -eous ly p r o v i d i n g immunity from l i a b i l i t y to p r a c t i t i o n e r s adher ing to PSRO s t a n d a r d s , w i l l e f f e c t i v e l y decrease the s i z e and number of m a l -p r a c t i c e se t t lements (545) . Most agree , however, t h a t w h i l e PSRO s t a n d -ards w i l l supersede or supplement those t r a d i t i o n a l l y a p p l i e d i n m a l p r a c t i c e p roceed ings , other cause and e f f e c t r e l a t i o n s h i p s between the PSRO l e g i s l a t i o n and m a l p r a c t i c e a c t i v i t y are l e s s c l e a r ( 6 4 , 1 1 5 , 2 7 7 , 5 1 7 , 5 4 5 ) . The p o t e n t i a l f o r i n f o r m a t i o n from m a l p r a c t i c e se t t lements to c o n t r i b u t e to more r e g u l a r and sys temat i c e v a l u a t i o n has been recogn i zed by medic ine and d e n t i s t r y . A committee of o r thoped ic s p e c i a l i s t s and lawyers de r i ved a l i s t of avo idab le adverse events by a n a l y z i n g th ree hundred and f i f t y m a l p r a c t i c e cases i n v o l v i n g o r thoped ic surgery that had reached the a p p e l l a t e cour t between 1910 and 1972 (493) . The events were c l a s s i f i e d and then used as a s t a r t i n g p o i n t from which to develop a taxonomy of avo idab le adverse med ica l events . The author p o i n t s out tha t such a l i s t cou ld be developed f o r any group of p r a c t i t i o n e r s and from sources other than m a l p r a c t i c e l i t i g a t i o n ( e . g . d i s c i p l i n a r y h e a r i n g s , h o s p i t a l i n c i d e n t r e p o r t s ) . Once the system i s developed and weights are ass igned to i t e m s , i t cou ld be used as a f u n c t i o n a l t o o l f o r m o n i t o r i n g and a n a l y z i n g performance. Mi lgrom ( 3 3 9 ) , who performed a s i m i l a r p r o c e -dure on a l l d e n t a l m a l p r a c t i c e s u i t s between 1900 and 1974, suggests tha t t h i s approach a l l o w s the i d e n t i f i c a t i o n of p r a c t i t i o n e r behav io rs that are c l e a r l y and c a u s a l l y t i e d to outcome and thus i s a more v a l i d e v a l u a -t i o n approach . The f a c t that i n e f f e c t i v e and w a s t e f u l p r a c t i t i o n e r behavior would go unnot iced i s not r e c o g n i z e d . -105-Summary and Comment Theoretically, one can evaluate professionals' competence by looking at structure, process, or outcome elements, or at combinations of these. The health care literature concerned with combined approaches recognizes the potential advantages of an integrated evaluation system which links structure, process, and outcome studies, but provides limited reports of i t s application. The most notable of these is the Medicaid Vendor System, a regional medical services monitoring scheme (434). Even the MVS, however, looks at structural elements in terms of the delivery system rather than in terms of the knowledge and s k i l l s possessed by professionals. Reports of structure-process evaluation activity are equally limited, although the Methodology for Monitoring Quality of Nursing Care (225,226,232,260) is notable both for i t s comprehensiveness and the care with which the system to date has been developed. Agency based process-outcome evaluation reports are numerous. A number of influential organizations have developed complete process-outcome evaluation systems which they promote through workshops, consult-ation services, and supporting materials replete with special forms and words. Perhaps because process-outcome evaluation is more recently popular and major measurement problems were worked through earlier, this area of literature treats analysis of deficiencies and choice of correct-ive action somewhat more fully than does the literature discussed i n previous chapters. The literature describing experience with agency based process-outcome evaluative procedures is particularly rich in examples of -106-v a r i a t l o n s to the b a s i c approach; these range from s imple to complex systems employing smal l to l a r g e d o l l a r r e s o u r c e s . Reports of the s e v e r a l p r o f e s s i o n s j o i n t l y m o n i t o r i n g t h e i r s e r v i c e s appear f o r the f i r s t time w i t h m e d i c a l - n u r s i n g combinat ions account ing f o r the m a j o r i t y . L i t e r a t u r e concerned w i t h r e g i o n a l l y based process-outcome e v a l u a t i o n y i e l d s i n f o r m a t i o n about one m e d i c a l care e v a l u a t i o n program w i t h a cor responding r e g i o n a l c o n t i n u i n g educat ion d e l i v e r y system which cou ld not be shown to be e f f e c t i v e . Another r e g i o n a l m e d i c a l care rev iew program has developed a measurement system, " c r i t e r i a mapping" , which a l lows a cho ice of process and/or outcome c r i t e r i a depending on d e v e l o p -ments i n the p a t i e n t ' s c o n d i t i o n d u r i n g the course of h i s i l l n e s s . D e s c r i p t i o n s of r e g i o n a l process-outcome e v a l u a t i o n of d e n t a l care are p a r t i c u l a r l y i m p r e s s i v e i n terms o f scope, p r e c i s e measurement p r a c t i c e s and involvment of the many p a r t i e s concerned ( p a t i e n t s , i n s u r o r s , p r o f e s -s i o n a l s ) . Another l i t e r a t u r e a rea cons idered i n t h i s chapter i s outcome assessment w i t h s e l e c t i v e e v a l u a t i o n of p r o c e s s . In t h i s approach process e v a l u a t i o n u s u a l l y occurs a f t e r outcomes have been found dev iant from the e s t a b l i s h e d s t a n d a r d . Th is approach i s w i d e l y used , apparent l y as a r e s u l t of promotion of the PEP e v a l u a t i o n methodology developed by the power fu l J o i n t Commission on A c c r e d i t a t i o n of H o s p i t a l s . W i l l i a m -son 's work (534) on p r i o r i t i e s f o r e v a l u a t i o n i s an even e a r l i e r i n f l u e n -c i n g f a c t o r , however, and s i x years l a t e r t h i s remains a u s e f u l and important i tem f o r study by those concerned w i t h process-outcome e v a l u a -t i o n . Random e v a l u a t i o n of a l l e g e d neg l igence r e s u l t i n g i n adverse outcomes occurs i n connect ion w i t h m a l p r a c t i c e l i t i g a t i o n . The extent to - 1 0 7 -which tor t law influences or controls health p r o f e s s i o n a l s ' competence i s unknown. Because malpractice action involves few p r a c t i t i o n e r s and i s a f t e r the f a c t , i t has l i t t l e d i r e c t impact i n terms of evaluating health p r o f e s s i o n a l s ' p r a c t i c e . The threat of malpractice a c t i o n , however, i s thought to act as a deterrent to incompetent p r a c t i c e . As structure, process, outcome, and combined evaluation approaches have, i n succession, come in t o widespread use, the experience has c l a r i f i e d a number of conceptual issues of a te c h n i c a l and contextual nature which are related to monitoring and/or promoting professional's competence. In some cases research has been part of t h i s process of growth and understanding. The next two chapters w i l l discuss the more prominent of these issues. - 1 0 8 -Chapter VI SELECTION OF EVALUATION AND ACTION STRATEGIES Recent exper ience has c l a r i f i e d a number of conceptua l i s s u e s r e l a t e d to e v a l u a t i o n and a c t i o n s t r a t e g i e s i n h e a l t h c a r e . Reports and d i s c u s s i o n s of e a r l y e f f o r t s are beg inn ing to permit both i d e n t i f i c a t i o n of the s p e c i f i c d e c i s i o n s which must be made and g e n e r a l i z a t i o n s about the i m p l i c a t i o n s that a t l e a s t some of these d e c i s i o n s have . Th is chapter cons ide rs t h i s i n f o r m a t i o n i n the f o l l o w i n g s e c t i o n s : Genera l C o n s i d e r a t i o n s i n Choosing E v a l u a t i v e S t r a t e g i e s , Dec id ing What W i l l Be E v a l u a t e d , Data Management, and E v a l u a t i o n Based I n t e r v e n t i o n s . Genera l C o n s i d e r a t i o n s i n Choosing E v a l u a t i v e S t r a t e g i e s Wr i te rs agree that there are some u n i v e r s a l a t t r i b u t e s that one w i l l attempt to ach ieve i n any approach to e v a l u a t i o n i n h e a l t h c a r e . These i n c l u d e v a l i d i t y (125 ,435) , r e l i a b i l i t y ( 1 2 5 , 4 3 5 ) , s u f f i c i e n t s e n s i t i v i t y to a l l o w a n a l y s i s which w i l l l e a d to a c t i o n ( 1 2 5 , 4 3 5 , 5 0 6 ) , c a p a c i t y f o r comprehensiveness ( 1 2 , 1 2 5 ) , and p r a c t i c a l i t y ( 1 2 , 1 2 5 , 4 3 5 , 5 0 6 ) . Such approaches w i l l a l s o p r o v i d e f o r f u l l documentation ( 1 2 , 1 2 5 ) , f o r r a t i f i c a t i o n of c r i t e r i a and standards by p rov ide rs whose care i s be ing eva luated ( 1 2 , 1 2 5 ) , and f o r s tandards which are s u f f i c i e n t l y f l e x -i b l e to permit i n n o v a t i o n and c l i n i c a l advancement (125) . In a d d i t i o n to - 1 0 9 -be ing m i n d f u l of these g e n e r a l l y d e s i r a b l e c h a r a c t e r i s t i c s , e v a l u a t i o n p lanners concerned w i t h competence of h e a l t h p r o f e s s i o n a l s w i l l wish to have a w e l l developed understanding of the e v a l u a t i o n - a c t i o n process as a p p l i e d i n h e a l t h c a r e , p o s s i b l e frames of re fe rence f o r e v a l u a t i o n , and the r o l e of va lues i n e v a l u a t i o n . The E v a l u a t i o n - A c t i o n Process as A p p l i e d i n Hea l th Care Fundamental ly e v a l u a t i o n i s no more than e s t a b l i s h i n g what w i l l be measured, measuring and r e c o r d i n g r e s u l t s , and making a judgement as to whether these are adequate or a c c e p t a b l e . Sometimes the judgement about adequacy i s b u i l t i n t o p r e - e s t a b l i s h e d s t a n d a r d s . In h e a l t h c a r e , e v a l u a t i o n most o f t e n i s undertaken to p rov ide i n f o r m a t i o n on which to base f u r t h e r a c t i o n . Hence most d i s c u s s i o n s concerned w i t h competence of h e a l t h p r o f e s s i o n a l s beg in w i t h the e v a l u a t i v e process but then extend to f u r t h e r s teps a l l o w i n g fo r a n a l y s i s of e v a l u a t i v e r e s u l t s , a p p r o p r i a t e f o l l o w - u p a c t i o n , and r e - e v a l u a t i o n to determine e f f e c t i v e n e s s of a c t i o n . This e v a l u a t i o n - a c t i o n sequence has been the o r g a n i z i n g i n f l u -ence throughout t h i s paper . Severa l m u l t i - s t e p e v a l u a t i o n - a c t i o n sequences have been d e s c r i b e d i n p rev ious chapters and w i l l not be repeated h e r e . P lanners w ish ing to rev iew the e v a l u a t i o n - a c t i o n process i n g r e a t e r d e t a i l have s e v e r a l r e f e r e n c e s from which to draw. Brown's b i - c y c l e concept (70 ,71) i s remarkable not only because i t i s one of the f i r s t c o n c e p t u a l i z a t i o n s of the process a p p l i e d to competence of p r o f e s s i o n a l s , but a l s o because i t s twelve s teps leave l i t t l e margin f o r o v e r s i g h t . Measur ing the  Q u a l i t y of P a t i e n t Care ( 2 5 5 ) , the Ambulatory Care E v a l u a t i o n Pr imer (335) , the Q u a l i t y Assurance Program f o r M e d i c a l Care i n the H o s p i t a l ( 1 6 ) , Q u a l i t y Assurance i n Long Term Care F a c i l i t i e s ( 3 1 1 ) , the PEP - 1 1 0 -Pr imer ( f o r h o s p i t a l s ) ( 2 6 5 ) , G u i d e l i n e s f o r Review of Nurs ing Care a t  the L o c a l L e v e l (514) , and "Measur ing Q u a l i t y of Nurs ing Care: P a r t s I and I I " (306) a l l are comprehensive r e f e r e n c e s on e v a l u a t i o n i n h e a l t h care that prov ide the reader w i t h a w e l l developed e x p l a n a t i o n of s teps i n the e v a l u a t i o n - a c t i o n sequence. As p lanners i d e n t i f y the d i s c r e e t s teps they w i l l under take , the probable cos ts beg in to assume more concrete fo rm. The l i t e r a t u r e o f f e r s l i t t l e i n f o r m a t i o n on t h i s important m a t t e r . Zimmer (543) e s t i -mates that 1 - 2 % agency manpower e f f o r t should be expended on q u a l i t y assurance a c t i v i t y , and McClure (318) b e l i e v e s t h a t 1 - 2% of a h o s p i -t a l ' s gross income can p r o f i t a b l y be spent on such a c t i v i t y . W i l l i a m s o n (535) es t imates that a f u l l - t i m e med ica l records a n a l y s t , 10% of a f u l l -t ime e q u i v a l e n t p h y s i c i a n p o s i t i o n , and vo lun tee red p h y s i c i a n t ime are r e q u i r e d f o r m ed ica l outcomes e v a l u a t i o n s t u d i e s . S p e c i f i c i n f o r m a t i o n on c o s t s of med ica l and n u r s i n g a u d i t programs and u t i l i z a t i o n rev iew i s prov ided by three h o s p i t a l s w i t h l i m i t e d e x p l a n a t i ons of the methods used to a r r i v e a t these f i g u r e s ( 2 0 7 , 3 7 6 , 5 2 1 ) . In 1976, Medicus Systems est imated c o s t s of i t s Methodology f o r M o n i t o r i n g Q u a l i t y of Nurs ing Care at $1,000 i n i t i a l and $150/month o p e r a t i n g c o s t . Th is assumed t r a i n i n g of observers cou ld be absorbed and on ly minor m o d i f i c a t i o n s to the e x i s t -i n g computer program were r e q u i r e d ( 2 2 6 ) . Exper ience has shown t h i s est imate to be low (197) . A number of e x i s t i n g e v a l u a t i v e s t r a t e g i e s are d i s c u s s e d by Costanzo and V e r t i n s k i (113) i n terms of whether they r e q u i r e l i m i t e d or ex tens i ve resources f o r implementat ion . Costs a s s o c i -ated w i t h the l a r g e s c a l e ( n a t i o n a l ) development of s i t u a t i o n - s p e c i f i c c r i t e r i a have been c a r e f u l l y documented by the consor t ium of p r o f e s s i o n a l o r g a n i z a t i o n s which p ioneered a methodology f o r s p e c i f y i n g c r i t e r i a f o r - I l l -e v a l u a t i o n i n ambulatory c h i l d h e a l t h care ( 3 7 4 ) . I t i s c l e a r tha t w i t h so l i t t l e r u l e - o f - t h u m b i n f o r m a t i o n on c o s t s , the p lanner w i l l wish to p r o j e c t a n t i c i p a t e d c o s t s c a r e f u l l y when s e l e c t i n g e v a l u a t i o n - a c t i o n s t r a t e g i e s . P o s s i b l e Frames of Reference f o r E v a l u a t i o n The frame of re fe rence fo r t h i s paper has been the c o n t i n u i n g competence of h e a l t h p r o f e s s i o n a l i n p r a c t i c e . No attempt has been made to d i s t i n g u i s h e v a l u a t i v e approaches which make judgements about i n d i v i -d u a l p r o f e s s i o n a l s from those which make judgements about groups of p r o f e s s i o n a l s ; the former w i l l be d e s i r a b l e i n some i n s t a n c e s . Given t h i s frame of r e f e r e n c e , the process of care ( c l i n i c a l performance) i s taken to be a d i r e c t measure of competence w i t h s t r u c t u r e (knowledge and s k i l l s ) and outcome ( r e s u l t s of care) remain ing as i n d i r e c t measures. This i n t e r p r e t a t i o n i s supported by Hegyvary and Haussmann (232) and Donabedian ( 1 4 1 ) . Given other frames of r e f e r e n c e , however, s t r u c t u r e or outcome, and i n d i v i d u a l s , groups or whole p o p u l a t i o n s might be more i m p o r t a n t . In p r a c t i c e , e v a l u a t i o n programs may attempt to serve more than one purpose , t h e r e f o r e i t i s important to have a way of c o n c e p t u a l i z i n g many d i f f e r e n t frames of r e f e r e n c e . Donabedian (141,143) has suggested tha t a u s e f u l v iew can be obta ined by r e c o g n i z i n g that one w i l l u s u a l l y w i s h to focus on a par t of at l e a s t three d imens ions : 1 . i n d i v i d u a l p r a c t i t i o n e r - groups of p r o v i d e r s - care systems 2. i n d i v i d u a l p a t i e n t - case load - t a r g e t p o p u l a t i o n 3 . p h y s i c a l f u n c t i o n - p s y c h o l o g i c a l f u n c t i o n - s o c i a l f u n c t i o n -112-Each of these three-part dimensions can be thought of as the dimensions of a cube with the p o s s i b i l i t y of i n t e r s e c t i o n at anyone of a number of points. If one i s concerned with the p h y s i c a l health of the i n d i v i d u a l c l i e n t as affected by an i n d i v i d u a l p r a c t i t i o n e r , one w i l l choose d i f f e r -ent c r i t e r i a than would be the case i f the focus of concern were psycho-s o c i a l health of the caseload of a group of providers. As the frame of reference changes, c r i t e r i a i d e n t i f i e d as desirable may c o n f l i c t . For example, the nurse may e l e c t to give superior care to an i n d i v i d u a l patient at the expense of her caseload; the p h y s i c a l needs of a target population w i l l be served by c a r e f u l reporting of venereal disease con-tacts at the r i s k of the i n d i v i d u a l ' s psychological or s o c i a l f u n c t i o n -ing. In s e l e c t i n g evaluation s t r a t e g i e s , i t i s important to c l a r i f y the frame of reference and purpose; i f there i s more than one frame of r e f e r -ence to be used or purpose to be served, one must i d e n t i f y possible areas of c o n f l i c t and malte an t i c i p a t o r y decisions. The subject "competence of health p r o f e s s i o n a l s " b a s i c a l l y i s concerned with the health worker as he acts to support and increase the physical, psychological and sometimes s o c i a l function of patients f o r whom he i s responsible. This frame of reference can and should be expan-ded to include groups of professionals i f action s t r a t e g i e s can be e f f e c t i v e l y planned for groups. It i s therefore important to e s t a b l i s h exactly what behaviors might be included for evaluation. The l i t e r a t u r e indicates several approaches have been used: 1. d e s c r i p t i o n and c l a s s i f i c a t i o n of behaviors that occur i n p r a c t i c e , through use of experience (144), goal a n a l y s i s , (103) or c r i t i c a l incident technique (166,291,455). N 2. multidimensional conceptual schemes. One (76,106) used the dimen-sions of content (developed through a l i s t of most important health -113-p r o b l e m s ) , a b i l i t i e s , and t a s k s ; a second (42) chose the problem s o l v i n g p r o c e s s , c l i n i c a l d i s c i p l i n e ( p e d i a t r i c s , p s y c h i a t r y , e t c . ) , and context of care ( a c u t e , r e h a b i l i t a t i o n e t c . ) . 3 . h e a l t h needs and problems which g i ve r i s e to r e q u i r e d behav iors ( 5 2 4 ) . 4. l i t e r a t u r e rev iew (219 ,232) . The c h i e f va lue of a conceptua l framework f o r e v a l u a t i o n of performance i s that i t a l l o w s one to i d e n t i f y the un ive rse which can be e v a l u a t e d . Once c r i t e r i a are s e t , such a framework a l s o p rov ides a reminder about areas of performance not chosen f o r e v a l u a t i o n . In v iew of the c u r r e n t p r e - o c c u p a t i o n w i t h e v a l u a t i n g t e c h n i c a l d e c i s i o n s about p h y s i c a l c a r e , t h i s i s an important p o i n t . The Role of Values i n E v a l u a t i o n De Geyndt (120) s t a t e s that " to d e f i n e q u a l i t y i s to l i m i t i t s meaning; ( t h i s ) r e f l e c t s a va lue judgement". A l though Donabedian acknowledged va lues as a major i n f l u e n c e i n s e l e c t i n g e v a l u a t i o n s t r a t e -g i e s b e t t e r than ten years ago ( 1 4 2 ) , i t i s o n l y s i n c e 1975 that the i s s u e has been f u l l y acknowledged i n the l i t e r a t u r e . Lang (288,289) e x p l i c i t l y s t a t e s that va lues c l a r i f i c a t i o n i s a d i s t i n c t step i n the process of e v a l u a t i o n i n n u r s i n g c a r e . Truby (506) a l l u d e s to the importance of va lues when she advocates c o n s i d e r a t i o n of the agency 's ph i losophy when making e v a l u a t i o n p l a n s . She extends t h i s f u r t h e r by say ing that e v a l u a t i o n p lanners should be aware of e x a c t l y where they are w i l l i n g to change and should be guided by p o s s i b l e d e c i s i o n s that w i l l r e s u l t and a sense of data needed. Moore (346) exp lo res four common va lue systems which have an important i n f l u e n c e i n North American h e a l t h care systems: s u r v i v a l , r a t i o n a l i t y , hedonism, and the J u d e o - C h r i s t i a n e t h i c . Somers (482) i d e n t i f i e s expec ta t ions which -114-the public holds f o r the health care system: technical competence; a good doctor-patient r e l a t i o n s h i p ; i n s t i t u t i o n s and procedures which w i l l monitor competence, and c o r r e c t i v e action which w i l l be taken as necessary. Methods for values c l a r i f i c a t i o n and s p e c i f i c a t i o n are beyond the scope of t h i s paper. Nevertheless i t i s apparent that thoughtful writers support the notion that e x p l i c i t statements of values help avoid c o n f l i c t and give d i r e c t i o n to planners as they make decisions about evaluative s t r a t e g i e s . Deciding What W i l l be Evaluated Given an established frame of reference, a way of i d e n t i f y i n g the possible range of c l i n i c a l behaviors, and a cl e a r sense of values, the planner w i l l need to make further decisions about what to evaluate, whether to e s t a b l i s h e x p l i c i t c r i t e r i a and standards or r e l y on the considered judgement of reviewers ( i m p l i c i t standards), and about how to select c r i t e r i a and standards i f these are to be s p e c i f i e d . L i t e r a t u r e concerned with these questions i s reviewed i n t h i s section. The Structure - Process - Outcome Debate As conceptualized by Donabedian, q u a l i t y of care assessment r e l i e s on one or more basic types of information: structure, process or outcome. E a r l i e r chapters have shown that the bulk of evaluative a c t i v -i t y i n the f i r s t h a l f of the century was directed at measuring profess-i o n a l ' s knowledge and s k i l l s (structure); that the f i f t i e s and s i x t i e s -115-saw an increased emphasis on evaluation of performance (process), and that in the seventies, after a brief f l i r t a t i o n with outcomes evaluation, there emerged a decided preference for combined process - outcome studies. In selecting one or more of these basic types of information for evaluation, the advantages and disadvantages associated with each must be considered. These have been discussed extensively in the l i t e r -ature and are summarized and referenced in Tables III, IV, and V. Advantages and Disadvantages of Measurement of Structure Advantages -- generally reliable (335) - easy, relatively inexpensive, to carry out (62) Disadvantages - questionable validity re impact on patient health (62,67,142,335) Table III Advantages and Disadvantages of Measurement of Process Advantages -- when a process criterion is not met, planners have direct information about what requires correction. (54,120,335,452) - corrective action can focus on individuals or on groups of providers, whichever i s efficient and appropriate (233,335) familiar, ie. providers find i t easy to establish c r i t e r i a (62,335) - relatively easy to carryout (54) - permits concurrent review and feedback of results at a time when i t is most meaningful (7,62) - able to consider problem areas of missed and erroneous diagnoses (534) Disadvantages - Questionable causal validity re improved patient health state (7,67,133,145,163,233,435) - tends to perpetuate a sometimes indefensible status quo, with such results as rising health costs etc (145,318,335) - may increase malpractice l i t i g a t i o n rate (7) Table IV - 1 1 6 -Advantages and Disadvantages of Measurement of Outcome Advantages -- h i g h degree of v a l i d i t y ( 3 1 8 ) , prov ided there i s adequate c o n t r o l f o r other i n f l u e n c i n g v a r i a b l e s (226,332) , , - a l l o w s f o r c o n s i d e r a t i o n f o r a l l p o s s i b l e c a u s a t i v e f a c t o r s , thus w i l l l ead to most improvement f o r l e a s t cost (163,318,543) - capable of f o c u s s i n g on a t a r g e t p o p u l a t i o n , not j u s t a case load (335) - s t i m u l a t e s i n n o v a t i o n i n care (335) , - a i d s i n d e c i s i o n s about h e a l t h p o l i c y ( 1 2 0 , 6 7 ) . Disadvantages -- i d e n t i f i e s problem but does not a i d i n d e f i n i n g cause ( 5 4 , 6 2 , 1 4 2 , 1 4 5 , 2 3 3 , 3 3 5 , 4 5 2 ) . - d i f f i c u l t c o n t r o l f o r v a r i a b l e s other than case or to i d e n t i f y p r o v i d e r ( s ) r e s p o n s i b l e ( 5 4 , 2 3 3 , 4 3 5 ) . - l i t t l e a v a i l a b l e i n f o r m a t i o n or agreement on what c o n s t i t u t e s r e a s o n a b l e s outcome measures (62 ,67 ,142 ,145 ,335) , . - d i f f i c u l t to develop r e l i a b l e outcome measures f o r some problems ( 5 4 , 2 3 3 , 3 3 5 , 4 5 2 ) - d i f f i c u l t to c o l l e c t d a t a , e s p e c i a l l y on long term outcomes ( 6 7 , 1 4 2 , 1 4 5 , 3 3 5 , 4 5 2 ) - when a p p l i e d w i t h i n a case load or f a c i l i t y , assumes p a t i e n t assessment and d i a g n o s i s has been c o r r e c t (159),. Table V Most e v a l u a t i o n s t u d i e s i n h e a l t h care share the common assump-t i o n that adequate resources c o n t r i b u t e to adequate d i a g n o s i s and t r e a t -ment which i n t u r n r e s u l t i n a b e t t e r h e a l t h s t a t u s than would otherwise be the c a s e . In s e l e c t i n g s t r u c t u r e and process v a r i a b l e s and c r i t e r i a , the p lanner w i l l attempt to choose those which have a h i g h p o s i t i v e c o r r e l a t i o n w i t h f a v o r a b l e outcomes. One immediate problem, as i n d i c a t e d by the t a b l e s , i s that few such c o r r e l a t i o n s are known or can be p r e d i c t -ed w i t h c e r t a i n t y . E s t a b l i s h i n g r e l a t i o n s h i p s between elements of s t r u c -t u r e , process and outcome i s the bus iness of c l i n i c a l r e s e a r c h ( 6 2 , 2 3 3 , 4 5 2 ) . The demands i n terms of p r e c i s e measurement are g r e a t e r i n such - 1 1 7 -c l i n i c a l t r i a l s than i n the e v a l u a t i o n - a c t i o n programs aimed at f i n d i n g gross d e f i c i t s i n care and t a k i n g c o r r e c t i v e a c t i o n . C l i n i c a l r e s e a r c h i n the t e c h n i c a l aspects of medica l care has a long t r a d i t i o n and a growing body of l i t e r a t u r e . The other p r o f e s s i o n s are beg inn ing to develop such a knowledge base as w e l l . Less w e l l d e v e l -oped are s t u d i e s which attempt to e s t a b l i s h c o r r e l a t i o n s between s t r u c -t u r e , process and outcome f o r n o n - t e c h n i c a l areas or f o r care d e l i v e r y systems. Severa l i tems r e t r i e v e d as r e p o r t s of e v a l u a t i o n a c t i v i t y proved to be r e p o r t s of c l i n i c a l t r i a l s w i t h the major purpose being that of e s t a b l i s h i n g c o r r e l a t i o n s . Most ( 1 4 9 , 2 0 5 , 2 3 1 , 2 4 9 , 3 3 7 , 3 8 2 , 4 1 5 , 4 1 6 , 459,489,490) were s t r u c t u r e - p r o c e s s s t u d i e s whose r e s u l t s o f f e r l i t t l e comfort to those w ish ing to use s t r u c t u r e v a r i a b l e s as proxy measures of the adequacy of c a r e . One study (149) p a r t i c u l a r l y germane to competence e v a l u a t i o n showed no r e l a t i o n s h i p between n u r s e s ' scores on a knowledge-theory t e s t and the observed use of r e l a t e d p r i n c i p l e s i n the performance of n u r s i n g p r a c t i c e . Process-outcome s t u d i e s (186,253,364) and two s t u d i e s i n c o r p o r a t i n g v a r i a b l e s from a l l three areas (226,428) were a l s o i d e n t i f i e d . Brook et a l . ( 6 7 ) , i n a monograph on outcome assessment , summarize f o r t y - s e v e n e v a l u a t i o n s t u d i e s , some of which e s t a b l i s h c o r r e -l a t i o n s as w e l l . I t should be noted that a p r e - r e q u i s i t e of r e s e a r c h to e s t a b l i s h c o r r e l a t i o n s i s the a v a i l a b i l i t y of r e l i a b l e measurement methods. A f u r t h e r d i f f i c u l t y i n e s t a b l i s h i n g process-outcome c o r r e l a -t i o n s i n h e a l t h care i s that once a process becomes c o n v e n t i o n a l p r o f e s s -i o n a l p r a c t i c e i t becomes p r a c t i c a l l y and e t h i c a l l y d i f f i c u l t to submit to c l i n i c a l t r i a l ( 6 2 ) . Given the cu r ren t problems of e v a l u a t i o n s t u d i e s us ing s t r u c -t u r e , process or outcome measures i n i s o l a t i o n , the tendency to take - 1 1 8 -concurrent measures i n at l e a s t process and outcome areas i s unders tand -a b l e . Pre-eminence may be g i ven to one of these a r e a s , depending on the frame of r e f e r e n c e . Because the o b j e c t of e v a l u a t i o n i s an unders tanding of whether d e s i r a b l e r e s u l t s can be encouraged and u n d e s i r a b l e r e s u l t s p r e v e n t e d , i t i s l i k e l y that such combined measures w i l l always occupy the a t t e n t i o n of e v a l u a t o r s seek ing improvements i n the d e l i v e r y of h e a l t h care g e n e r a l l y or the performance of h e a l t h p r o f e s s i o n a l s s p e c i f i -c a l l y (141) . I m p l i c i t Versus E x p l i c i t Standards Having made the d e c i s i o n to eva luate s t r u c t u r e , p r o c e s s , o u t -come or combinat ions of t h e s e , the e v a l u a t o r can s p e c i f y the s tandard by which care w i l l be judged adequate ( e x p l i c i t s tandards ) or he can s imp ly make a judgement based on some i n t e r n a l i z e d s tandard ( i m p l i c i t s t a n d -a rds ) . Krumme (286) s t a t e s c a t e g o r i c a l l y that measurement and e v a l u a -t i o n approaches u s i n g i m p l i c i t s tandards are h i g h l y u n r e l i a b l e . She c l a i m s tha t s c o r i n g systems which r a t e on a comparat ive b a s i s (poor , average, good, ) aga ins t performance of a re fe rence group, or a c c o r d i n g to degree of guidance r e q u i r e d , permit an i m p r e c i s e measure which depends on each e v a l u a t o r ' s i n t r i n s i c s t a n d a r d s . These w i l l vary between rev iewers ( i n t e r - r e v i e w e r r e l i a b i l i t y ) and i n the same rev iewer over t ime ( i n t r a -rev iewer r e l i a b i l i t y ) . R i chardson ' s study (420) of judgements of the q u a l i t y of c l i n i c a l care i n r e t r o s p e c t i v e a n a l y s i s of c h a r t s i n o b s t e -t r i c s , p e d i a t r i c s , and surgery shows that i n t e r - r e v i e w e r r e l i a b i l i t y u s i n g i m p l i c i t c r i t e r i a and standards was unacceptably l ow . D e c i s i o n s were i n f l u e n c e d by c l i n i c a l b i a s , missed i t e m s , geographic d i f f e r e n c e s i n - 1 1 9 -s t a n d a r d s , and r e l u c t a n c e to c r i t i c i z e . He concluded tha t the number of independent judges r e q u i r e d to reach a s t a b l e judgement of care q u a l i t y exceeded the number l o g i s t i c a l l y a v a i l a b l e to meet rev iew demands, and that p r e - e s t a b l i s h e d c r i t e r i a and standards should be u s e d . In a subse -quent study ( 4 1 9 ) , he demonstrated that where e x p l i c i t process c r i t e r i a fo r a g iven c o n d i t i o n become too numerous f o r e f f i c i e n t r e v i e w , i t i s p o s s i b l e to reduce these to manageable numbers. H i s s t r a t e g y i n v o l v e s r a t i n g a sample of c h a r t s u s i n g e x p l i c i t c r i t e r i a ( i n t h i s demonst rat ion there were s i x t y - o n e ) , r a t i n g the same c h a r t s through a s u b j e c t i v e rev iew p r o c e s s , then s u b j e c t i n g t h i s data to three consecut i ve a n a l y t i c a l methods. In t h i s case i t was shown that e v a l u a t i o n of t w e n t y - s i x c r i -t e r i a would produce v i r t u a l l y the same r e s u l t s . Other researchers are not so w i l l i n g to abandon i m p l i c i t s t a n d -ards as u n d e s i r a b l e or i m p r a c t i c a l . Peters (388) showed that judgements u n d i r e c t e d by i n s t r u c t i o n and unmodif ied by c o n s u l t a t i o n are s u f f i c i e n t l y accurate i f a s e l e c t i v e rev iew s t r a t e g y i s u s e d . By s u b m i t t i n g on ly records p r e v i o u s l y r a t e d as u n s a t i s f a c t o r y to subsequent rev iew , and i n c l u d i n g them w i t h p r e v i o u s l y unrated records so tha t rev iewers won't know they have been i d e n t i f i e d as u n s a t i s f a c t o r y , e f f i c i e n t use of rev iewer time i s a c h i e v e d . The p r o p o r t i o n of u n s a t i s f a c t o r y records i n any sample should approximate 50%. P e t e r s showed t h a t , assuming 15% of the o r i g i n a l sample i s u n s a t i s f a c t o r y i f a record i s i d e n t i f i e d as inadequate three t imes i t has a 99% p r o b a b i l i t y of be ing u n s a t i s f a c t o r y . The net r e s u l t i s that some u n s a t i s f a c t o r y , records escape but the worst are i d e n t i f i e d at a low investment of t i m e . McCla in (317) has exp lored the use of conf idence s c a l e s to i n c r e a s e r e l i a b i l i t y where i m p l i c i t s tandards are u s e d . He found the - 1 2 0 -s c a l e of l i t t l e use i n i n c r e a s i n g i n t e r - r e v i e w e r r e l i a b i l i t y and c o n c l u d -ed t h a t t h i s i s best improved through d i s c u s s i o n and i n s t r u c t i o n . I n t r a -rev iewer r e l i a b i l i t y can be achieved by us ing rev iewers who have f u l l conf idence i n t h e i r judgements and who use the conf idence s c a l e as a f u n c t i o n of t h e i r t r u s t i n the data presented f o r r e v i e w . Techniques f o r i d e n t i f y i n g rev iewers who use a conf idence s c a l e i n t h i s manner are d i s c u s s e d . Brooks (63) has shown that even when r e l i a b i l i t y of method i s not at i s s u e , a c c e p t a b i l i t y of care g i ven two hundred n i n e t y - s i x p a t i e n t s w i t h four d i f f e r e n t c o n d i t i o n s v a r i e d from 1.4% to 6 3 . 2 % , depending on whether process or outcome and e x p l i c i t or i m p l i c i t c r i t e r i a and s t a n d -ards were used . Judgements about the process of care us ing e x p l i c i t s tandards y i e l d e d the lowest percentage of acceptab le r e c o r d s . Another s tudy , however, found l i t t l e v a r i a t i o n i n q u a l i t y judgements between e x p l i c i t and i m p l i c i t s tandards rev iew ( 3 6 5 ) . V a r i a t i o n s l i k e l y depend as much on the nature of the e x p l i c i t c r i t e r i a (most of the above s t u d i e s were conducted us ing r a t h e r long l i s t s of e x p l i c i t c r i t e r i a of d o u b t f u l v a l i d i t y ) as on the f a c t tha t c r i t e r i a and standards are e x p l i c i t or i m p l i c i t . Research u s i n g s h o r t e r l i s t s of h i g h l y c r i t i c a l e x p l i c i t c r i t e r i a might w e l l show h igher agreement w i t h an i m p l i c i t s tandards approach to e v a l u a t i o n . The above s t u d i e s suggest tha t e x p l i c i t s tandards are a d v a n t a -geous because they are g e n e r a l l y more r e l i a b l e and can be a p p l i e d by l e s s expensive p e r s o n n e l , but that i m p l i c i t s tandards may be more v a l i d . Th is view i s supported by some authors ( 6 2 , 1 4 4 ) . I t may a l s o e x p l a i n the growing p r a c t i c e of u s i n g e x p l i c i t s tandards as a sc reen ing dev ice o n l y . In t h i s approach a l l i n s t a n c e s of non-compl iance are f u r t h e r eva luated by - 1 2 1 -p r o f e s s i o n a l s u s i n g i m p l i c i t s tandards before any problem a n a l y s i s i s a t tempted. I t may be t e n t a t i v e l y concluded tha t at present the use of e x p l i c i t c r i t e r i a and standards does not o b v i a t e the need f o r e v e n t u a l l y a p p l y i n g i m p l i c i t ones . D e r i v a t i o n of C r i t e r i a and Standards Donabedian (144) po inted out ten years ago tha t c r i t e r i a and standards can be s e l e c t e d from data about e x i s t i n g p r a c t i c e s ( e m p i r i c a l c r i t e r i a ) or from what i s cons idered to be good p r a c t i c e (normative c r i t e r i a ) . Whi le r e c o g n i z i n g that use of e m p i r i c a l c r i t e r i a may p e r p e t u -ate an unfor tunate and u n d e s i r a b l e s t a t u s quo, Donabedian has a l s o caut ioned that such survey data w i l l g i v e us v a l u a b l e i n f o r m a t i o n about what has been a c h i e v a b l e and p r a c t i c a l to d a t e . Sanazaro (451,452) has f u r t h e r developed the e m p i r i c a l and normative c r i t e r i a concept . He p o s t u l a t e s three b a s i c types of c r i t e r i a and s t a n d a r d s : s t a t i s t i c a l ( e m p i r i c a l ) ; normat ive ( d e r i v e d from consen -sus about good p r a c t i c e ) ; and s c i e n t i f i c ( v a l i d a t e d by c l i n i c a l t r i a l ) . Sanazaro sees e m p i r i c a l c r i t e r i a as endors ing the s t a t u s quo and r e j e c t s these as i n a p p r o p r i a t e f o r e v a l u a t i o n of q u a l i t y . He s u b d i v i d e s norma-t i v e c r i t e r i a i n t o g e n e r a l consensus c r i t e r i a which have been i d e n t i f i e d by p r a c t i t i o n e r s as r e p r e s e n t a t i v e of good c a r e , and e s s e n t i a l c r i t e r i a which have been i d e n t i f i e d by exper ienced and recogn ized exper ts as c r i t i c a l to a d e s i r e d outcome f o r a s p e c i f i e d problem. Sanazaro p laces i n the g e n e r a l consensus group the innumerable c r i t e r i a s e t s , o f t e n r e f e r -red to as laundry l i s t s and cookbooks, which l i s t a l l commonly accepted process measures f o r any g i ven c o n d i t i o n . He s t a t e s that u s i n g t h i s type of c r i t e r i a set w i l l not on ly i n c r e a s e c o s t s of care as the c r i t e r i a are - 1 2 2 -adhered t o , but a l s o w i l l l e a d to i n e f f e c t i v e c o r r e c t i v e measures. Other authors po in t to documented d i f f i c u l t i e s i n r e l a t i n g popular but o t h e r -wise u n j u s t i f i e d process c r i t e r i a to d e s i r e d outcomes ( 1 3 3 , 1 6 3 , 4 2 8 ) . Sanazaro favo rs the e s s e n t i a l c r i t e r i a approach to e v a l u a t i o n d e s p i t e the r i s k of non-acceptance by p r a c t i t i o n e r s who do not "own" them. He goes on to emphasize that the t h i r d type of c r i t e r i a , those which have been s c i e n t i f i c a l l y v a l i d a t e d by c l i n i c a l t r i a l , are the c r i t e r i a of c h o i c e when a v a i l a b l e . Whi le Sanazaro 's c o n c e p t u a l i z a t i o n i s h e l p f u l and h i s p r e f e r -ences are t h e o r e t i c a l l y a t t r a c t i v e , there are a d d i t i o n a l f a c t o r s to c o n s i d e r . The f i r s t i s that p r a c t i t i o n e r s and exper ts may not vary g r e a t l y i n what they cons ide r to be impor tan t . Whereas Wagner (519) found that educat ion and exper ience i n f l u e n c e d the s e l e c t i o n of process c r i t e r i a f o r med ica l c a r e , the n a t i o n - w i d e p e d i a t r i c c r i t e r i a demonstra -t i o n p r o j e c t showed that there was a h i g h l e v e l of agreement between p r a c t i t i o n e r and expert groups (327 ,499) . The second i s that even i f exper ts and p r a c t i t i o n e r s do not agree on c r i t e r i a s p e c i f i c a t i o n , a number of authors b e l i e v e p r a c t i t i o n e r s should set c r i t e r i a anyway. Reasons commonly g i ven are that the process i s thought to be e d u c a t i o n a l (125,159,387) or i s thought to i n c r e a s e l i k l i h o o d of p r a c t i t i o n e r support of the rev iew process ( 1 2 5 , 3 5 5 , 4 6 0 ) . Exper ience i n Utah (355) suggests that t h i s l a t t e r reason may be a v a l i d one. Some sources suggest that p r a c t i t i o n e r s modify or r a t i f y p r e - e s t a b l i s h e d c r i t e r i a as a compromise to the dilemma of expert versus p r a c t i t i o n e r o r i g i n a t e d c r i t e r i a ( 1 2 , 1 2 5 , 2 6 5 , 4 7 4 , 5 1 4 ) . Al though i t may be premature to g e n e r a l i z e about the g r e a t e r d e s i r a b i l i t y of expert generated c r i t e r i a , the cho ice of e s s e n t i a l -123-c r i t e r i a i n preference to "laundry l i s t s " seems supportable i n terms of p r a c t i c a l i t y as well as v a l i d i t y (125). (It has already been suggested that judgements using e s s e n t i a l c r i t e r i a are more l i k e l y to show agree-ment with judgements using i m p l i c i t standards). Thompson and Obsorne (499) emphasize that where c r i t e r i a are not supported by c l i n i c a l research, i t i s important to accept only those c r i t e r i a on which there i s a high l e v e l of agreement as to t h e i r c r i t i c a l i t y . Before leaving the discussion of standards, the use of indices should be mentioned. Payne (315) suggests that measurement of perform-ance can be c a r r i e d out by at l e a s t two general approaches: using single d i s t i n c t measures or combining several measures into one number as an index which i s usually expressed as per cent of c r i t e r i a met. The f i r s t choice i s indicated when a few behaviors w i l l define the component under study, when few measures are available or can be assumed to be represent-a t i v e of others, when few a n a l y t i c comparisons are planned, or when the purpose i s to improve performance by feedback about d i s c r e e t behaviors. The second choice i s j u s t i f i e d when performance i s conceptualized as a composite of many imperfectly r e l a t e d items, when s t a t i s t i c a l r e l i a b i l i t y of the construct i s increased with more items, when complex analysis i s planned, when purpose i s summative evaluation only, or when purpose i s formative evaluation but summated indices are used as a f i r s t screen to d i r e c t a ttention to areas of greatest need. His research on the use of item importance weighting i n a r r i v i n g at index scores suggest that weighting can reduce the number of c r i t e r i a needed, and that d i f f e r e n t i a l and equal weighting w i l l produce the same r e s u l t s , but too short l i s t s of c r i t e r i a may decrease r e l i a b i l i t y of r e s u l t s . The most common practice i n health care evaluation to date has been to regard each c r i t e r i o n chosen for measure as i t s own standard,i.e. - 1 2 4 -100% of a l l c r i t e r i a must have been met i f care i s regarded as a c c e p t a -b l e . I t has been suggested that compuls ive p u r s u i t of o c c a s i o n a l s i t u a -t i o n s of non-compl iance i s not c o s t - e f f e c t i v e . I n c r e a s i n g l y authors are encouraging e v a l u a t o r s to set other standards of a c c e p t a b i l i t y , e . g . 80% of the c r i t e r i a met, based on the t h r e s h o l d a t which a c t i o n appears worth t a k i n g (125) . Sanazaro (451) w r i t e s that where c r i t e r i a sets are based on genera l consensus c r i t e r i a the l i m i t s of a c c e p t a b i l i t y are n e c e s s a r i l y low; where they are based on the e s s e n t i a l c r i t e r i a approach ,a 100% l e v e l of compliance must be expected . Data Management D e c i s i o n s about how i n f o r m a t i o n w i l l be c o l l e c t e d , c l a s s i f i e d , s t o r e d , r e t r i e v e d and repor ted are c r u c i a l to e f f e c t i v e e v a l u a t i o n . Whi le a l l p o s s i b l e aspects of data management are not d i s c u s s e d i n the l i t e r a t u r e searched , a t l e a s t two t o p i c s are the o b j e c t of c o n s i d e r a b l e a t t e n t i o n . These are p a t i e n t care records as a data source and systems which f a c i l i t a t e r e t r i e v a l of d a t a . P a t i e n t Care Records as a Data Source Rosenfe ld (435) s t a t e s that methods of data c o l l e c t i o n i n c l u d e d i r e c t o b s e r v a t i o n , rev iew of r e c o r d s , and rev iew of record a b s t r a c t s . He suggests tha t p a t i e n t records and a b s t r a c t s are u s u a l l y chosen because these methods are l e s s c o s t l y than d i r e c t o b s e r v a t i o n and possess g rea te r r e l i a b i l i t y . There are dec ided d isadvantages to records rev iew however. Comprehensiveness i s u s u a l l y s a c r i f i c e d (62 ,286,435) because records are - 1 2 5 -inadequate sources of i n f o r m a t i o n about r e l a t i o n s h i p s , communicat ion, p s y c h o l o g i c a l and s o c i a l aspects of management. Koran (285) b e l i e v e s the most s i g n i f i c a n t f a i l u r e of records rev iew i s that i t cannot judge accuracy of p h y s i c a l f i n d i n g s or t e c h n i c a l q u a l i t y of treatment p r o c e -d u r e s . Brook suggests that even though records t h e o r e t i c a l l y cou ld be u s e f u l i n judging q u a l i t y of d e c i s i o n making and documenting the f a c t of t reatment , they are c u r r e n t l y an inadequate source of data about even these v a r i a b l e s ( 6 2 ) . Th is l a s t d isadvantage has s t i m u l a t e d a number of i n v e s t i g a t i v e s t u d i e s . A major p r o j e c t to e s t a b l i s h n a t i o n a l process c r i t e r i a f o r c h i l d h e a l t h s u p e r v i s i o n and three d i seases (327,374,500) showed that o v e r a l l documentation of the c r i t e r i a was about 50% d e s p i t e the f a c t tha t p r a c t i t i o n e r s had p r e v i o u s l y agreed that these c r i t e r i a cou ld be found i n r e c o r d s . Measurements and l a b data were recorded f r e q u e n t l y ; c o u n s e l i n g i tems i n f r e q u e n t l y . In another study of ambulatory c l i n i c records ( 4 4 8 ) , 25% of the sample cou ld not be reviewed and of the remain ing 75%, v i r t u -a l l y o n e - h a l f proved to be l a c k i n g documentation about one or more of the e s t a b l i s h e d c r i t e r i a . In s t i l l another study which assessed ambulatory care f o r two c h r o n i c , f i v e acute and three p revent i ve c a t e g o r i e s ( 3 0 7 ) , i n v e s t i g a t o r s concluded rev iew of o f f i c e records i s not f e a s i b l e . The above s t u d i e s do not answer the ques t ion "was care g i ven and s imply not r e c o r d e d , or was care not g i ven and t h e r e f o r e not r e c o r d -ed?" At l e a s t three s t u d i e s i n ambulatory care s e t t i n g s attempt to answer t h i s . Long and Rogers (309) used d i r e c t unobt rus ive o b s e r v a t i o n of care of p a t i e n t s w i t h s t r e p t o c o c c a l th roat i n f e c t i o n s . They showed that p r a c t i t i o n e r s adhered to 12 - 100% of the c r i t e r i a they agreed a f te rward were important and a c t u a l l y recorded 0 - 94% of c r i t e r i a . S i x - 1 2 6 -out of four teen c r i t e r i a were recorded 40% (or more) l e s s f r e q u e n t l y than performed. Bentson (47) r e p o r t s that observers of f i f t y - n i n e p a t i e n t -r e s i d e n t encounters i n a f a m i l y med ica l care u n i t at the U n i v e r s i t y of Western Ontar io counted 2.45 problems per encounter a c t u a l l y d e a l t w i t h , w h i l e r e s i d e n t s recorded on ly 1.51 problems d e a l t w i t h . When problems were c l a s s i f i e d as somat ic , emot ional and s o c i a l , the emot ional and s o c i a l problems were l e s s o f t e n r e c o r d e d . Another study (547) showed that when i n f o r m a t i o n i n f i f t y - o n e tape recorded p a t i e n t encounters was compared w i t h the cor responding medica l r e c o r d s , the d i a g n o s i s , c h i e f c o m p l a i n t , scheduled appointments , n o n - drug t rea tments , d i a g n o s t i c s t u d i e s and names of medic ines were w e l l r e c o r d e d . Converse l y , m e d i c a -t i o n dosages, l e v e l s of f u n c t i o n , compl iance , and reasons f o r f o l l o w - u p were poor l y r e c o r d e d . The i n v e s t i g a t o r s b e l i e v e that the p r a c t i t i o n e r s s t u d i e d i n t h i s u n i v e r s i t y c l i n i c were not t y p i c a l . For example, o ther s t u d i e s show d i a g n o s i s recorded 60% of the time as compared w i t h 97% i n t h i s s tudy . An important p o i n t i s that a l l of the above s t u d i e s were done i n ambulatory care s e t t i n g s . There i s some evidence that as o r g a n i z a -t i o n a l s t r u c t u r e of care becomes more f o r m a l i z e d , documentation of care g iven becomes more complete ( 3 8 , 3 0 7 , 5 0 0 ) . Thus the problem of records be ing an i n a c c u r a t e r e f l e c t i o n o f r e a l i t y may i n c r e a s e w i t h independance of p r a c t i c e . A r e l a t e d f i n d i n g which has i m p l i c a t i o n s fo r o v e r a l l c o n -s i d e r a t i o n of competence i s tha t there appears to be a p o s i t i v e r e l a t i o n -sh ip between the q u a l i t y of med ica l records and the q u a l i t y of care ( 3 1 4 , 5 4 7 ) . A l though i n - p a t i e n t records apparent l y have not been s c r u t i n -i z e d f o r t h e i r a b i l i t y to r e f l e c t r e a l i t y , there has been some concern - 1 2 7 -about i n - p a t i e n t records a b s t r a c t s . Donabedian (142) s t a t e s tha t p a t i e n t records are a poor enough r e f l e c t i o n of r e a l i t y and tha t records a b -s t r a c t s are even f u r t h e r removed from the r e a l w o r l d . One h o s p i t a l r e p o r t s d i f f i c u l t y g e t t i n g s u f f i c i e n t data from a b s t r a c t s to meet i t s a u d i t requirements ( 2 4 4 ) . Exper ience i n o ther h o s p i t a l s has shown that t r a i n i n g of records a b s t r a c t o r s and a n a l y s t s i s e s s e n t i a l to prevent unwarranted d i l u t i o n of data ( 2 6 9 , 2 9 3 ) . In one r a t h e r e x t e n s i v e study designed to i n v e s t i g a t e t h i s problem area (352) , a t r a i n e d team r e -a b s t r a c t e d a n a t i o n a l sample of p a t i e n t records of Medicare and Med ica id p a t i e n t s . Comparison w i t h the a b s t r a c t i n g s e r v i c e records showed h i g h agreement w i t h p a t i e n t records f o r data on l e n g t h of s t a y and p a t i e n t c h a r a c t e r i s t i c s but lower l e v e l s of agreement f o r pr imary d iagnoses (65.2%) and procedures (73 .2%) . The f i n d i n g s suggest that broader d i a g n o s t i c coding i s more r e l i a b l e , that i t i s o f t e n d i f f i c u l t to d e t e r -mine pr imary d i a g n o s i s , that i n d i v i d u a l d iagnoses a f f e c t r e l i a b i l i t y , and that the q u a l i t y of the h o s p i t a l ' s a b s t r a c t i n g s e r v i c e i s s i g n i f i c a n t and dependent on t r a i n i n g p rocedures . Systems F a c i l i t a t i n g R e t r i e v a l of Data I f p r o b l e m - s p e c i f i c process and outcome c r i t e r i a are monitored u s i n g p a t i e n t r e c o r d s , the records of p a t i e n t s w i t h s p e c i f i c problems must be r e a d i l y r e t r i e v a b l e and i n f o r m a t i o n r e l e v a n t to each c r i t e r i o n must be a v a i l a b l e and e a s i l y l o c a t e d by c l e r i c a l p e r s o n n e l . The l i t e r a t u r e r e v e a l s some attempts to c o r r e c t the present inadequacy of documentation i n p a t i e n t r e c o r d s . Severa l w r i t e r s have recogn ized the importance of the problem o r i e n t e d record (POR) as a promis ing record format which not on ly demands documentation of i n f o r m a -- 1 2 8 -t i o n r e l e v a n t to e v a l u a t i o n , but a l s o o rgan izes i t i n a form which i s e a s i l y scanned ( 5 4 , 7 0 , 8 0 , 4 5 2 ) . This record system i n c l u d e s the c o l l e c -t i o n of d a t a , the f o r m u l a t i o n of problems based on d a t a , the development of p lans and treatment f o r each problem, and f o l l o w - u p through the use of notes s p e c i f i c to each problem. Without s p e c i f y i n g the POR f o r m a t , other authors d e t a i l minimum data requirements as those data elements which d e s c r i b e the achievement of p a t i e n t - s p e c i f i c , p r o b l e m - s p e c i f i c o b j e c t i v e s (536) . Sanazaro (452) and White (533) d e s c r i b e e a r l y e f f o r t s to e s t a b -l i s h a n a t i o n a l Uniform Minimum B a s i c Data Set w i t h s u b - s e t s f o r ambula -t o r y care and long term c a r e . K e l l e r (269) b e l i e v e s that by s t a n d a r d i z -i n g records formats i t would be p o s s i b l e to monitor q u a l i t y of care not on ly fo r case loads and f a c i l i t y c l i e n t e l e , but a l s o f o r e n t i r e t a r g e t p o p u l a t i o n s . Problem c l a s s i f i c a t i o n systems are an a d d i t i o n a l c h a l l e n g e to those p r o f e s s i o n s w i s h i n g to undertake p r o b l e m - s p e c i f i c e v a l u a t i o n . Without a way to c l a s s i f y problems, i t i s d i f f i c u l t to move beyond e v a l u -a t i o n of the i n d i v i d u a l case to e v a l u a t i o n of a sample of l i k e c a s e s . I t i s e q u a l l y d i f f i c u l t to compare r e s e a r c h f i n d i n g s which e s t a b l i s h process-outcome c o r r e l a t i o n s f o r s p e c i f i c problems. Both of the above are c r u c i a l to e f f e c t i v e and e f f i c i e n t q u a l i t y of care e v a l u a t i o n . The medica l p r o f e s s i o n has done c o n s i d e r a b l e work i n t h i s area over a per iod of years and i s ab le to draw on a v a r i e t y of c l a s s i f i c a -t i o n s i n c l u d i n g the 9 t h g e n e r a t i o n of the I n t e r n a t i o n a l C l a s s i f i c a t i o n of D i s e a s e s , the h o s p i t a l a d a p t a t i o n of ICDA and more recent e f f o r t s which modify med ica l language f o r computer m a n i p u l a t i o n . These i n c l u d e the e a r l y SNDO (Standard Nomenclature of Diseases and Operat ions ) which uses the dimensions of anatomica l topography and e t i o l o g y , SNOP (Systemat ized - 1 2 9 -Nomenclature of Pathology) which uses the dimensions of typography, morphology, e t i o l o g y , and f u n c t i o n , SNOMED (Systemat ized Nomenclature of Medic ine) which expands and r e f i n e s the three f i l e s of topography, morphology and e t i o l o g y and adds procedures , f u n c t i o n s and u t i l i t y . Th is l a s t i n c l u d e s syntax usages , a d j e c t i v e s , q u a n t i f i e r s , s igns and symptoms e t c . ( 4 3 , 1 7 8 , 1 8 3 , 5 3 1 ) . Recent work has a l s o e s t a b l i s h e d the ICHPPC ( I n t e r n a t i o n a l C l a s s i f i c a t i o n of Hea l th Problems i n Pr imary Care) which can be combined w i t h a c l a s s i f i c a t i o n system fo r p a t i e n t s problem from the p a t i e n t ' s p e r c e p t i o n and language ( 5 2 8 , 5 3 3 ) . The n u r s i n g l i t e r a t u r e shows a beg inn ing awareness of the importance of c l a s s i f i c a t i o n of n u r s i n g d i a g n o s i s f o r r e s e a r c h and e v a l u -a t i o n i n n u r s i n g . While the n u r s i n g l i t e r a t u r e i s l a r g e l y s p e c u l a t i v e and t h e o r e t i c a l ( 5 3 , 4 3 9 ) , some beg inn ing attempts a t a c l a s s i f i c a t i o n system are be ing made ( 1 8 3 ) . There were no re fe rences i n the l i t e r a t u r e to h i g h l y systema-t i z e d c l a s s i f i c a t i o n systems f o r d e n t i s t r y or pharmacy. The q u e s t i o n of a m u l t i - u s e system s u i t a b l e to the e v a l u a t i o n and r e s e a r c h needs of a l l h e a l t h p r o f e s s i o n a l s was not d i s c u s s e d e i t h e r ; i t i s probable that SNOMED has such a c a p a c i t y or cou ld be adapted f o r m u l t i d i s c i p l i n a r y use ( 1 8 3 ) . E v a l u a t i o n Based I n t e r v e n t i o n s A p lanner i n t e r e s t e d i n i n s t i t u t i n g an e f f e c t i v e e v a l u a t i o n -a c t i o n sequence w i l l need to move beyond the t e c h n i c a l ques t ions of e v a l u a t i v e s t r a t e g y to those tha t a n t i c i p a t e us ing e v a l u a t i o n i n f o r m a t i o n to achieve d e s i r e d r e s u l t s . While the l i t e r a t u r e r e p o r t i n g s u c c e s s f u l / - 1 3 0 -u n s u c c e s s f u l e v a l u a t i o n - a c t i o n sequences can be looked to as a r i c h source of i d e a s , few p a t t e r n s have emerged from which g e n e r a l i z a t i o n s can be made or d e c i s i o n g u i d e l i n e s f o r m u l a t e d . Table VI summarizes a sample of these e v a l u a t i o n - a c t i o n sequences i n terms of the v a r i a b l e ( s ) chosen as an i n d i c a t o r of compe-tence (Competence Measure) , type of i n t e r v e n t i v e or c o r r e c t i v e a c t i o n used to upgrade c u r r e n t p r a c t i c e ( I n t e r v e n t i o n ) , and whether there was any evidence of change i n the d e s i r e d d i r e c t i o n (Improvement). The contents of the t a b l e permit a l i m i t e d number of o b s e r v a -t i o n s to be made. F i r s t , the i n t e r v e n t i o n s tha t have been s u c c e s s f u l i n promoting d e s i r e d change are n o t a b l e f o r t h e i r v a r i e t y . E d u c a t i o n a l programs; a d m i n i s t r a t i v e a c t i o n i n the form of new p o l i c i e s , procedures and s e r v i c e s ; adopt ion of p r o t o c o l s or w r i t t e n p rocedures ; d e n i a l of insurance c l a i m s ; pe rsona l and i n d i v i d u a l i z e d feedback ; and a d d i t i o n s of new s t a f f and equipment a l l have been s u c c e s s f u l l y used to c o r r e c t what i n i t i a l l y appeared to be a performance problem. C l e a r l y the e v a l u a t i o n -a c t i o n sequence made popular i n h e a l t h care through Brown's b i - c y c l e concept of c o n t i n u i n g educat ion i s a p p l i c a b l e to the d i a g n o s i s of more than e d u c a t i o n a l problems and of more than problems r e l a t e d to competence of h e a l t h p r o f e s s i o n a l s . One h o s p i t a l (308) r e p o r t s tha t of twenty-one s t u d i e s (seventeen c l i n i c a l problems eva luated by process techniques and four by outcome assessment ) , o n l y f i v e y i e l d e d recommendations f o r a c t i o n which would change p h y s i c i a n b e h a v i o r s . The remaining s i x t e e n i n v o l v e d changes i n c r i t e r i a , a d m i n s t r a t i v e a c t i o n , s p e c i a l s t u d i e s , or no a c t i o n at a l l . The t a b l e a l s o i n d i c a t e s tha t i t i s not uncommon to undertake more than one c o r r e c t i v e a c t i o n f o r an i d e n t i f i e d problem. A s i g n i f i c a n t p r o p o r t i o n of these p r o j e c t s e i t h e r were not c a r r i e d through to r e - e v a l u a t i o n or t h i s l a s t step was s imply not Table VI EVIDENCE OF DESIRED CHANGE IN REPORTED EVALUATION - ACTION PROJECTS REFERENCE QUALITY MEASURE INTERVENTION IMPROVEMENT Barnet t (40) ( m u l t i d i s c l p l i n a r y ) t reatment of beta hemoly t i c s t r e p t o c c a l t h r o a t i n f e c t i o n s computer generated immediate feedback f o r dev iant behavior yes Brook (66) (medic ine) a n t i b i o t i c p r e s c r i b i n g p a t t e r n s e d u c a t i o n , feedback, c la ims d e n i a l yes Condon (107) (nurs ing ) head nurse p e r c e p t i o n of improvement i n s t a f f p e r f o r m -ance re 4 s teps n u r s i n g procedure workshops on how to c a r r y out i n s e r v i c e educat ion y e s , but ga ins v a r i a b l e C h r i s t o f f e l (97) (medic ine) death r a t e - m y o c a r d i a l i n f a r c t i o n new equipment purchased unknown (medic ine) d i a g n o s t i c techniques and a n t i b i o t i c c h o i c e , r e s o l -t i o n of pneumonia e d u c a t i o n , new l a b procedure f o r specimen c o l l e c t i o n unknown (medic ine) i n c i d e n c e o f n e o n a t a l pneumothoraces inc reased char t documentation then f u r t h e r i n v e s t i g a t i o n unknown (medic ine) pathology of removed u te rus feedback and d i s c u s s i o n yes . . . c o n t i n u e d (medicine/ nurs ing ) i n s u l i n p r e s c r i b i n g p a t t e r n s . . . d i a b e t i c teach ing p r a c t i c e s . . . . i n c i d e n c e of u r i n a r y t r a c t educat ion p r e p a r a t i o n of d i a b e t i c teaching m a t e r i a l new treatment p r o t o c o l new d i a g n o s t i c p r o t o c o l ye s , and i n c i d e n t a l l y l e n g t h of s tay was decreased (medic ine) r a t e of j u s t i f i a b l e surgery f o r duodenal u l c e r c o u n s e l l i n g , p r o v i s i o n of c o n s u l t -a t i o n to o f f e n d i n g s t a f f members unknown (medicine/ nu rs ing ) d i a g n o s i s and care of p a t i e n t s w i t h s c h i z o p h r e n i a p o l i c y changes, inc reased s t a f f c r e a t i o n of new u n i t , p r o t o c o l f o r s u i c i d e precaut ions unknown D e v i t t (132) (medic ine) degree of pathology found a f t e r mastectomy and cholecystectomy educat ion no Diamond (40) (medic ine) use of psychot roph ic medicat ions educat ion yes Dyer (151) (nurs ing ) p a t i e n t care score der i ved from s e v e r a l ins t ruments head nurse m o d e l l i n g , a s s i s t a n c e w i t h goa l s e t t i n g yes i n t i t i a l l y but most ga ins l o s t at 12 months E s c o v i t z (159) (medic ine) d i a g n o s t i c accuracy d i a g n o s i s s p e c i f i c process c r i t e r i a change i n c o n s u l t i n g program educat ion yes yes but not to a c c e p t a b l e l e v e l . . . con t inued Glasson (187) ( n u r s i n g d i s c h a r g e p l a n n i n g , a l l p a t i e n t s e d u c a t i o n , changes i n records forms yes Graydon (202) (medic ine) p rocess c r i t e r i a f o r 5 common d iagnoses feedback of r e s u l t s from source e x t e r n a l to h o s p i t a l no Haussmann (226) (nurs ing ) composite of s e v e r a l process measures i n s e r v i c e educat ion yes r e o r g a n i z a t i o n of u n i t yes r e t u r n of p r e v i o u s l y s i c k head nurse yes Lewis (299) (medic ine) p e r i n a t a l death r a t e , i n c i d e n c e o f s p e c i f i c s u r g i c a l p r o c e d -u r e s , completeness of p h y s i c a l exams educat ion no , except p h y s i c a l exams were more complete M i l l e r (340) (medic ine) s e v e r a l p r o b l e m - s p e c i f i c c r i t e r i a , p a t i e n t s ' r i g h t s , admiss ion r a t e s , l e n g t h of s t a y type of i n t e r v e n t i o n not repor ted yes M i t c h e l l (344) (medic ine) l e n g t h of s tay f o r u n c o m p l i -ca ted cholecystectomy educat ion yes Mosley (348) (medic ine) a t t i t u d e s c a l e , work per fo rm-ance exper imenta l educat ion yes f o r work, no f o r a t t i t u d e . . . c o n t i n u e d Ramirez (404) (nursing) q u a n t i t y and q u a l i t y of nurs i n g care plans feedback and education yes Rubenstein (442) (medicine) s e r v e r a l process measures i . e . use of IV hepari n , packed c e l l s , e t c . education yes Russo (448) (medicine) d i a g n o s t i c and treatment c r i t e r i a f o r 26 common c o n d i t i o n s p r i v a t e d i s c u s s i o n , feedback yes Sanzaro (451) (medicine) c o m p l i c a t i o n r a t e post cholecystectomy education yes, from 25% to 13% pathology of removed append-i c e s education yes, from 45% to 19% i n a p r o p r i a t e use of a n t i b i o t i c s education yes, 70% to 40% composite performance i n d i c e s using process measures f o r 8 c o n d i t i o n s education yes, but s l i g h t and v a r i a b l e among co n d i t i o n s follow-up of abnormal lab r e s u l t s education no pathology of removed append-i c e s a d m i n i s t r a t i v e requirement to provide relevant d i a g n o s t i c i n f o r m a t i o n yes, d i d not p e r s i s t when requirement removed .. .continued process c r i t e r i a for diagnosis and treatment of acute sore throat treatment protocol yes process c r i t e r i a for several conditions computer generated feedback for deviant behavior yes, no for control group rate of justifiable pelvic surgery personal discussion with offending surgeons, provision of criteria to staff yes Schroeder (458) (medicine) rate of abortion and delivery in population desiring inform-ation about contraception staff discussion, reorganization services, revision of records format, sex counselling services no Shaw (471) (medicine) treatment of stroke victims education, additional (expert) staff yes Skipper (477) (medicine use of specialty consultants, type of intervention not reported yes Talley (492) (medicine/ pharmacy) prescribing and dispensing patterns feedback, claims denial yes Watkins (534) (pharmacy) performance in problem situ-ations, test of knowledge and attitudes programmed learning unit yes for performance and knowledge no for attitude .continued W i l l i a m s o n s e v e r a l process d i a g n o s i s - s p e c i f i c and outcome s t u d i e s educat ion p o s s i b l y educat ion p o s s i b l y educat ion no educat ion no a d m i n i s t r a t i v e a c t i o n p o s s i b l y a d m i n i s t r a t i v e a c t i o n p o s s i b l y Table VI - 1 3 7 -r e p o r t e d . There i s at l e a s t some evidence to suggest the former i s the case ( 1 8 4 , 3 5 1 ) . This hampers both the i n v e s t i g a t o r , as he cannot know whether a c t i o n has been e f f e c t i v e , and the development of sound p r i n -c i p l e s or d e c i s i o n r u l e s which can govern the conduct of e v a l u a t i o n -a c t i o n sequences. F i n a l l y , i t i s apparent from the t a b l e that few i n v e s t i g a t o r s have addressed the q u e s t i o n of whether improvement i s t r a n s i e n t or l a s t i n g . Put another way, there i s as yet l i t t l e concern as to whether s t r a t e g i e s f o r m a i n t a i n i n g changed behavior are needed. I t should be noted that Table VI represents a s m a l l sample of the r e p o r t s of e v a l u a t i o n - a c t i o n sequences a v a i l a b l e and prov ides o n l y the most b a s i c i n f o r m a t i o n . Th is i s a p o t e n t i a l l y f r u i t f u l a rea f o r i n v e s t i g a t i o n and i t i s l i k e l y tha t an i n - d e p t h study of a l a r g e r sample might w e l l i n d i c a t e p a t t e r n s t y p i c a l of s u c c e s s f u l and u n s u c c e s s f u l p r o j e c t s . Quest ions worth a s k i n g a r e : i s success i n f l u e n c e d by p r a c -t i o n e r s ' p e r c e p t i o n of whether there i s a problem? by s e v e r i t y of problem? by e x i s t e n c e of an o n - s i t e a d m i n i s t r a t i v e s t r u c t u r e f o r e v a l u a -t i o n a c t i v i t i e s ? by degree of p r a c t i t i o n e r p a r t i c i p a t i o n i n the e v a l u a -t i o n - a c t i o n p r o j e c t ? While these are on ly a few of many p o s s i b l e ques t ions they are l i s t e d because c u r r e n t l i t e r a t u r e suggests these may be i n f l u e n c i n g f a c t o r s and a l s o because many of the present r e p o r t s c o n -t a i n i n f o r m a t i o n r e l a t i v e to these i s s u e s . Summary And Comment Exper ience w i t h e v a l u a t i o n - a c t i o n approaches to improving care i s s u f f i e n t l y w e l l repor ted i n the l i t e r a t u r e to prov ide p lanners w i t h -138-some g u i d e l i n e s . There i s genera l agreement that an e f f e c t i v e system w i l l be planned w i t h a t t e n t i o n to v a l i d i t y , r e l i a b i l i t y , p r a c t i c a l i t y , comprehensiveness, and p o t e n t i a l f o r g i v i n g d i r e c t i o n to and e v a l u a t i n g a c t i o n . I t w i l l be capable of documentat ion. At the same time i t w i l l a l l o w fo r p a r t i c i p a t i o n by p r a c t i t i o n e r s and w i l l not s t i f l e i n n o v a t i o n . P lanners should proceed from a base which i s c h a r a c t e r i z e d by a c l e a r l y d e f i n e d frame of r e f e r e n c e , a sense of the un ive rse of p r o f e s s i o n a l behav iors which c o l l e c t i v e l y are an express ion of competence, and an unambiguous statement of v a l u e s . The d e c i s i o n as to which category of v a r i a b l e s ( s t r u c t u r e , p r o c e s s , outcome) w i l l be measured should be made w i t h c o n s i d e r a t i o n f o r the decided advantages and d isadvantages a s s o c i a t e d w i t h each . A c h i e f problem area i s the low or n o n - e x i s t a n t c o r r e l a t i o n s between s t r u c t u r e and p r o c e s s , process and outcome, and s t r u c t u r e and outcome measures. Th is i s a p r i o r i t y area of r e s e a r c h f o r a l l the p r o f e s s i o n s . Such research demands a g rea te r degree of s c i e n t i f i c r i g o r than do e v a l u a t i v e s t u d i e s undertaken to improve c a r e ; the two should not be confused . In an e f f o r t to minimize d isadvantages and maximize advantages, i n v e s t i g a -t o r s are i n c r e a s i n g l y t u r n i n g to c o l l e c t i o n of process and outcome data w i t h respect to any g i ven problem. There has a l s o been a marked tendency to use e x p l i c i t c r i t e r i a and standards i n e v a l u a t i o n , r a t h e r than leave these t o t a l l y to the judgement of the r e v i e w e r . Th is approach i s seen as l e s s c o s t l y and more r e l i a b l e , but i t s pr imary u s e f u l n e s s may be as a sc reen ing dev ice i n mass e v a l u a t i o n . As yet the need f o r v a l i d i t y has tended to l e a d e v a l u a t o r s to use i m p l i c i t s tandards i n a f i n a l rev iew of i d e n t i f i e d d e v i a t i o n s . E x p l i c i t c r i t e r i a may be e s t a b l i s h e d us ing a v a r i e t y of approaches i n c l u d i n g the use of s t a t i s t i c s which r e f l e c t e x i s t i n g p r a c --139-t i c e , popular opinion about what represents good p r a c t i c e , expert opinion about practice e s s e n t i a l to desired outcome, or research derived evidence of what constitutes good p r a c t i c e . In the absence of the l a t t e r , essen-t i a l c r i t e r i a are seen as preferred even though they r i s k non-acceptance by the p r a c t i t i o n e r group. The l i t e r a t u r e addresses some but not a l l aspects of data man-agement. Patient records appear to be a r e l a t i v e l y cheap and r e l i a b l e source of data i n i n s t i t u t i o n a l s e t t ings, but are not so u s e f u l i n less structured s i t u a t i o n s . They are obviously l i m i t e d i n the kind of inform-ation they provide. Some attempts have been made to increase a v a i l a b i l -i t y of data by proposing minimum basic data s e t s . The problem oriented record i s seen as a useful data base for evaluation purposes. Retrieval of aggregate data about care of s p e c i f i c patient populations and of information about established structure-process-outcome c o r r e l a t i o n s i s hampered by the lack of problem-based c l a s s i f i c a t i o n systems i n a l l professions except medicine. F i n a l l y , reports of evaluation-action projects which have been undertaken i n the past several years make i t c l e a r that improvements i n care can be effected; that change strategies are of many types, of which continuing education i s only one; that many studies f a i l to remeasure to determine success of actions taken; and that almost no attempts have been made to investigate or ensure the permanence of successful change. One can conclude that a beginning has been made i n that tenta-t i v e guidelines have been developed for the evaluation aspects of evaluation-action p r o j e c t s . Understanding of the problem analysis and action phases as they exist i n health care i s not so well established. As reports become more numerous and complete, i t i s l i k e l y that some -140-h e l p f u l generalizations can be made. Other areas needing research and development are problem-based c l a s s i f i c a t i o n systems relevant to the professional practice of a l l health workers; further testing of the r e l i a b i l i t y of structure, process and outcome measures; work which establishes c o r r e l a t i o n s between these; and organization and provision of information on cost and impact of evaluation-action systems. The need for t h i s l a s t item may be viewed as a pragmatic r e f l e c t i o n of the concerns of a society which i s i n c r e a s i n g l y aggressive i n questioning the competence of the e n t i r e health care d e l i v e r y system. A c t i v i t i e s concerned with evaluation and maintainance of professional competence must be viewed within the larger s o c i a l context; t h i s i s the focus of the next chapter. -141-C h a p t e r V I I THE NORTH AMERICAN SOCIAL CONTEXT M a i n t a i n i n g c o m p e t e n c e o f h e a l t h p r o f e s s i o n a l s i m p l i e s t h a t t h e r e w i l l be b o t h e v a l u a t i o n and a p p r o p r i a t e a c t i o n i n l i g h t o f t h a t e v a l u a t i o n . C o n s e q u e n t l y , t h e b u l k o f t h i s p a p e r has d e a l t w i t h l i t e r a -t u r e r e l a t i n g t o t h e s e two i s s u e s . E v a l u a t i n g a n d m a i n t a i n i n g c o m p e t e n c e a r e n o t s o l e l y m e c h a n i s t i c e n d e a v o r s w h i c h e m p l o y s t e r i l e p r o c e d u r e s h o w -e v e r . These a c t i v i t i e s a r e u n d e r t a k e n by a n d f o r human b e i n g s i n d i v i d u -a l l y a n d c o l l e c t i v e l y . The l i t e r a t u r e s e a r c h e d d e a l s w i t h some o f t h e p s y c h o l o g i c a l c o n s i d e r a t i o n s a n d s o c i o - p o l i t i c a l d e v e l o p m e n t s w h i c h b o t h i n f l u e n c e a n d a r e i n f l u e n c e d by e v a l u a t i n g a n d m a i n t a i n i n g c o m p e t e n c e o f h e a l t h p r o f e s s i o n a l s . T h i s l i t e r a t u r e i s d e s c r i b e d i n C h a p t e r V I I . T h e f i r s t s e c t i o n , The S o c i a l E c o l o g y o f M a i n t a i n i n g Compe-t e n c e , i s a g e n e r a l d i s c u s s i o n o f l i t e r a t u r e i t e m s r e l a t e d t o p s y c h o -s o c i a l , e c o n o m i c , and p o l i t i c a l i s s u e s w h i c h seem t o be most d i r e c t l y r e l a t e d t o e v a l u a t i n g and m a i n t a i n i n g c o m p e t e n c e . The s e c o n d s e c t i o n , R e g u l a t o r y M e c h a n i s m s and T r e n d s , d e a l s w i t h l i t e r a t u r e c o n c e r n e d w i t h s o c i e t y ' s a t t e m p t s t o r e g u l a t e p r o f e s s i o n a l s and c e r t i f y t h e i r c o m p e -t e n c e . The l a s t s e c t i o n i s M a n d a t o r y C o n t i n u i n g E d u c a t i o n . T h i s s p e c i -f i c c r e d e n t i a l i n g t r e n d i s s i n g l e d o u t f o r s p e c i a l d i s c u s s i o n b e c a u s e o f i t s s p e c i a l i n t e r e s t f o r c o n t i n u i n g e d u c a t o r s . -142-The Social Ecology of Maintaining Competence Developments i n monitoring competence which are discussed i n this paper have taken place i n a decade characterized by a growing under-standing about the needs and rights of individuals, the use of the group (collectivism) as a powerful tool to ensure protection of these r i g h t s , and a growing awareness that the expenditure of limited resources w i l l be decided i n the p o l i t i c a l arena. While not wishing to report f u l l y the health care l i t e r a t u r e dealing with these issues, i t i s in s t r u c t i v e to note their influence on practices for monitoring competence of health professionals. Needs and Rights of Professionals Self regulation i s one of the t r a d i t i o n a l hallmarks of a profession. To suggest that peer review i s a new concept would be mis-leading. It i s apparent,however, that evaluative practices i n the four professions considered have recently become more intensive, more s p e c i f i c and i n some instances more d i r e c t . In such a s i t u a t i o n , both economic and personal security are threatened. When a professional i s accused or suspected of inadequate practice, he has a right to due process - the f a i r hearing of his case. The increased awareness of the importance of due process i s reflected i n the l i t e r a t u r e and has been referred to i n e a r l i e r chapters. Personal security i n the sense of good reputation or favourable self-concept i s more d i f f i c u l t to protect however. Concern for economic security and need for the esteem of s e l f and others i n f l u -ences attitudes about evaluation, especially evaluation by a peer group. There seems to be l i t t l e l i t e r a t u r e dealing with professionals' - 1 4 3 -a t t i t u d e s about peer r e v i e w . At Duke U n i v e r s i t y ' s Department of P s y c h i -a t r y , s t a f f members' a t t i t u d e s about peer rev iew were measured w i t h a t h i r t y - s i x i tem q u e s t i o n n a i r e ( 1 9 8 ) . Whi le t w o - t h i r d s of the respondents conceded tha t peer rev iew was needed, most tended to b e l i e v e tha t rev iew panels should c o n s i s t of s e n i o r s t a f f p s y c h i a t r i s t s , thus showing both a preference f o r i n t e r p r e t i n g the concept "peer" r a t h e r b road l y and an i n c l i n a t i o n to accept the a u t h o r i t y of g r e a t e r knowledge and e x p e r i e n c e . Among a l l respondents , hav ing a degree of resentment about peer rev iew was s i g n i f i c a n t l y c o r r e l a t e d w i t h see ing a low need f o r i t . Dekker (123) r e p o r t s that i n a 1976 nat ionwide survey of American p h y s i c i a n s a s k i n g whether i n t e n s i f i e d peer rev iew a c t i v i t i e s would improve p a t i e n t c a r e , 45% s a i d " y e s " , 50% s a i d "no" and 5% were " u n d e c i d e d " . Whi le the nature of the responses i n d i c a t e d l i t t l e o u t r i g h t o p p o s i t i o n , s k e p t i c i s m was widespread. Among the ques t ions r a i s e d : How do you agree on a d e f i n i -t i o n f o r q u a l i t y care? Is non -s tandard care sub -s tandard? Is poor care worse than no care a t a l l ? Desp i te what appears to be f a i r l y n e u t r a l a t t i t u d e s on the p a r t of the m a j o r i t y of the med ica l p r o f e s s i o n , the tendency to keep e v a l u a t i v e a c t i v i t i e s at arms l e n g t h i s n o t i c e a b l e : "We must f a c e s u r g i c a l committees of our p e e r s , but these committees examine the records never does a committee i n s i s t on sc rubb ing i n , t a k i n g notes as we o p e r a t e , or t e s t i n g our knots our o f f i c e s are s a c r o s a n c t , and we w r i t e down what we p lease r a r e l y are our d e s c r i p t i o n s of f i n d i n g s , d i a g n o s i s or t reatment c h a l l e n g e d " ( 2 2 8 ) . D e n t i s t r y has not concerned i t s e l f g r e a t l y w i t h a t t i t u d e s about peer r e v i e w . Jago (257) w r i t e s that h i s t o r i c a l l y d e n t a l s o c i e t i e s have acted p r i m a r i l y as an a r b i t r a t o r fo r fees and tha t peer rev iew committees - 1 4 4 -might more a p t l y have been named "peer j u s t i f i c a t i o n commit tees" . I f Jago i s c o r r e c t , d e n t i s t s ' exper iences w i t h peer rev iew have been n o n -t h r e a t e n i n g ; thus one might expect present a t t i t u d e s about peer rev iew to be r e l a t i v e l y n e u t r a l . D i r e c t and p u r p o s e f u l performance rev iew has a long h i s t o r y w i t h i n the n u r s i n g p r o f e s s i o n . The n u r s e ' s s t a t u s as an employee has meant c o n t i n u i n g s u p e r v i s i o n and e v a l u a t i o n by an a d m i n i s t r a t i v e s u p e r i o r from w i t h i n the p r o f e s s i o n . Recent n u r s i n g l i t e r a t u r e employs a f a r narrower i n t e r p r e t a t i o n of peer rev iew than s imply e v a l u a t i o n by a f e l l o w p r o f e s s i o n a l however. This l i t e r a t u r e i n f e r s t h a t a peer i s of the same e d u c a t i o n , r a n k , and q u a l i f i c a t i o n w i t h i n the p r o f e s s i o n . Pardee (379) c l a r i f i e s the i s s u e of "what i s a p e e r , " by suggest ing tha t the d e f i n i -t i o n of peer w i l l be f l e x i b l e i n r e l a t i o n to the s e t t i n g , purpose and goals of r e v i e w ; i . e . one must know what i s to be eva luated before one can i d e n t i f y the p e e r . For example, a head nurse w i l l have other head nurses as peers when r e v i e w i n g a d m i n i s t r a t i v e performance and w i l l have s t a f f and c l i n i c a l c o o r d i n a t o r as peers when r e v i e w i n g c l i n i c a l s k i l l s . The n u r s i n g l i t e r a t u r e g i ves guidance to nurses w i s h i n g to i n s t i t u t e peer rev iew systems ( 1 9 , 8 1 , 1 9 0 , 3 1 6 , 3 7 9 ) . The adv ice o f f e r e d cou ld apply to any e v a l u a t i v e system, however, and i s not r e l a t e d s p e c i f i c a l l y to rev iew by p e e r s . There are no a t t i t u d e surveys repor ted i n the n u r s i n g l i t e r a -t u r e , but two " p e r s o n a l exper ience" r e p o r t s (190,224) show that nurses f i n d peer rev iew both t h r e a t e n i n g and reward ing . Both s i t u a t i o n s i n v o l v e d f a c e - t o - f a c e i n t e r a c t i o n and used d i r e c t o b s e r v a t i o n and records as a source of d a t a . Both a l s o repor ted the rev iew to be more u s e f u l - 1 4 5 -when c h a r a c t e r i z e d by e x p l i c i t s tandards and a s t r u c t u r e d rev iew p r o c e -dure . D e s p i t e recent r e p o r t s of n e u t r a l i t y or support of i n t e n s i f i e d peer rev iew a c t i v i t i e s by the v a r i o u s p r o f e s s i o n s , a 1974 Wicons in M e d i -c a l J o u r n a l e d i t o r i a l (228) may be a more honest and u s e f u l assessment of p r o f e s s i o n a l a t t i t u d e s about peer r ev iew : "At one end of the p s y c h o l o -g i c a l continuum are the l u s t y e x h i b i t i o n i s t s who l o u d l y d e c l a r e that they are not a f r a i d of peer r e v i e w , that everybody makes m i s t a k e s , and tha t mis takes must be t a l k e d about , are the ve ry s t u f f of teach ing (or s t a f f ) programs, and those who cannot take the heat should get out of the k i t c h e n Another p o s i t i o n would be that of the d e l i c a t e doctor who i s i n a b j e c t t e r r o r of anyone at a l l l o o k i n g a t h i s work. He w i l l go to any l e n g t h , d e v i s e whatever i n t e l l e c t u a l s t ra tegem, to prevent the a b r o g a t i o n of h i s r i g h t to p r i v a c y , h i s God-given r i g h t to p r a c t i c e medic ine as he sees f i t — a l l human beings f a l l somewhere on the s c a l e no matter what the beauty of the l o g i c of a rev iew system, most of us w i l l be r a t t l e d by i t . For these reasons I b e l i e v e peer rev iew pass ions as w e l l as peer rev iew a n x i e t i e s should be taken s e r i o u s l y " . The w r i t e r ' s c o n -c l u s i o n i s a s e n s i b l e and s e n s i t i v e one. There i s c o n s i d e r a b l e evidence to date that a t t e n t i o n has been l a v i s h e d f a r more on rev iew procedures than on rev iew p a s s i o n s . R ights and Needs of Consumers I f p r o f e s s i o n a l s have r i g h t s and needs which must be r e s p e c t e d , so do the consumers whom they s e r v e . The r i g h t to p r i v a c y , a c o n t e n t i o u s i s s u e throughout contemporary Nor th American s o c i e t y , has profound - 1 4 6 -i m p l i c a t i o n s f o r rev iew mechanisms, whether they are based on p a t i e n t records or d i r e c t o b s e r v a t i o n of c a r e . To what extent can a n o n - i n v o l v e d p r o f e s s i o n a l de l ve i n t o records or observe the g i v i n g or r e c e i v i n g of care before he has s e r i o u s l y v i o l a t e d the p a t i e n t ' s r i g h t to p r i v a c y or perhaps even a d v e r s e l y a f f e c t e d tha t care? This q u e s t i o n does not appear to have been s e r i o u s l y exp lored i n e v a l u a t i o n l i t e r a t u r e , a l though i t i s l i k e l y tha t much of the l i t e r a t u r e concerned w i t h p a t i e n t s ' r i g h t s i n med ica l r e s e a r c h would be a p p l i c a b l e . E l a b o r a t e c o n f i d e n t i a l i t y r e g u l a t i o n s have been developed i n connect ion w i t h PSRO rev iew (361 ,427) . Desp i te e a r l i e r statements tha t PSRO data should be made a v a i l a b l e to other government o r g a n i z a t i o n s f o r h e a l t h care p lann ing and d i s c i p l i n a r y purposes , such i n f o r m a t i o n i s c a r e f u l l y guarded i n the name of c o n f i d e n t i a l i t y . Th is arrangement appears to have been designed to p r o t e c t the p r o f e s s i o n a l as much as the consumer, but perhaps r e a l i s t i c a l l y r e f l e c t s the degree of t h r e a t the p r o f e s s i o n s w i l l t o l e r a t e before t h e i r members tend to subvert i n s t e a d of support e v a l u a t i o n systems. I t i s thought tha t the presence and p a r t i c i p a t i o n of consumers i n a l l areas of h e a l t h c a r e , i n c l u d i n g e v a l u a t i o n of p r o f e s s i o n a l s , w i l l ensure that the i n t e r e s t s of the consumer are p r o t e c t e d . Cunningham (116) r e p o r t s that the J o i n t Commission on A c c r e d i t a t i o n of H o s p i t a l s w i l l p rov ide f o r hear ings fo r consumers i n connect ion w i t h h o s p i t a l a c c r e d i t a t i o n s u r v e y s , and s t a t e s that the American p r i n c i p l e of l a y i n t e r f e r e n c e i n p o l i t i c s and educat ion i s spread ing to m e d i c i n e . Consu-mer p a r t i c i p a t i o n i n h e a l t h care i s l a r g e l y a motherhood i s s u e however, - 1 4 7 -and most of the l i t e r a t u r e i s r a t h e r vague on e x a c t l y how consumers can p r o d u c t i v e l y p a r t i c i p a t e . Angevine (29) i s t y p i c a l i n t h i s regard when he says that consumers must have input i n t o standards but does not suggest how t h i s can be e f f e c t e d . M i t c h e l l (343) i s more s p e c i f i c , suggest ing that consumers can c o n t r i b u t e to determin ing the ameni t ies to be prov ided i n the h e a l t h care system, can examine r e s u l t s of e v a l u a t i o n , and can dec ide n o n - t e c h n i c a l i s s u e s of p a t i e n t c a r e . He s t a t e s t h a t there i s evidence that f e a r s of consumers t r y i n g to d i c t a t e t e c h n i c a l d e c i s i o n s are u n -founded. Maloney (325) suggests that the e t h i c a l s tandards of a p r o f e s -s i o n should r e f l e c t the v a l u e s of s o c i e t y at l a r g e , and that p r o f e s s i o n a l behavior standards which have t h e i r o r i g i n i n a p r o f e s s i o n a l code of e t h i c s , e . g . a ban on a d v e r t i s i n g , should be s u b j e c t to rev iew by c o n s u -mers. Jago (257) d i s c u s s e s consumerism i n h e a l t h care and rev iews two "consumer p a r t i c i p a t i o n " models which attempt to q u a n t i f y the degree of consumer i n f l u e n c e i n a system. He a l s o mentions that Trantow has searched and f a i l e d to f i n d any ev idence tha t consumer p a r t i c i p a t i o n has i n c r e a s e d q u a l i t y . There i s a n o t i c e a b l e t rend toward l e g i s l a t i o n i n both Canada and the Un i ted S t a t e s tha t mandates consumer p a r t i c i p a t i o n on l i c e n s i n g boards ; t h i s w i l l be d i s c u s s e d i n the s e c t i o n Regulatory Mechanisms and Trends . C o l l e c t i v e A c t i o n The use of group to ensure that the needs and r i g h t s of the i n d i v i d u a l are met i s not a new concept i n North American s o c i e t y , the - 1 4 8 -labour movement be ing the most n o t a b l e example. I f p a t i e n t s ' r i g h t s or consumer groups have banded together o u t s i d e government to assure or promote h e a l t h p r o f e s s i o n a l s ' competence, t h i s i s not repor ted i n the body of l i t e r a t u r e searched . On the other hand, the s e v e r a l h e a l t h p r o f e s s i o n s have o r g a n i z -ed to ensure that they are s p e c i f i c a l l y and m e a n i n g f u l l y i n c l u d e d i n the a c t i v i t i e s aimed at promoting or e v a l u a t i n g competence. A most n o t a b l e example i s the dominance of the organized p r o f e s s i o n a l groups i n the devlopment of the many standard c r i t e r i a se ts r e f e r r e d to e a r l i e r . B lumste in (56) r e f e r s to the i n f l u e n c e of o rgan ized medic ine i n g a i n i n g a p o s i t i o n of almost u n i l a t e r a l c o n t r o l of PSRO a c t i v i t y . Other authors repor t c o n s i d e r a b l e a c t i v i t y on the p a r t s of d e n t i s t r y (367) and n u r s i n g (25) o r g a n i z a t i o n s to g a i n power i n the PSRO d e c i s i o n making s t r u c t u r e . The n u r s i n g l i t e r a t u r e d e a l s s p e c i f i c a l l y w i t h c o l l e c t i v e b a r g a i n i n g as a t o o l to i n f l u e n c e mat te rs r e l a t e d to promoting and m o n i -t o r i n g competence. Jacox (256) , Ramphel (405) , K e l l o g (270) a l l suggest c o l l e c t i v e b a r g a i n i n g can be used by nurses to i n c r e a s e the i n f l u e n c e of p r a c t i t i o n e r s i n p r a c t i c e - r e l a t e d d e c i s i o n s . Ramphel b e l i e v e s tha t a s t rong s t a f f nurse o r g a n i z a t i o n s t rengthens the b u r e a u c r a t i c head of n u r s i n g s e r v i c e by i n c r e a s i n g e x p e r t i s e and - p r o v i d i n g support i n the face of i n t e r f e r e n c e from the med ica l s t a f f and h o s p i t a l a d m i n i s t r a t i o n . Both Ramphel and Jacox emphasize that to avo id a c o n f l i c t of i n t e r e s t , the a c t i v i t i e s of peer rev iew must be kept separate from those of the b a r g a i n i n g u n i t even i f the l a t t e r brought the former i n t o b e i n g . Ramphel b e l i e v e s the main avenue open to the employed p r a c t i t i o n e r who wishes to assume r e s p o n s i b i l i t y f o r q u a l i t y i s c o l l e c t i v e a c t i o n through the p r o f e s s i o n a l a s s o c i a t i o n . Jacox shares t h i s v iew but adds tha t - 1 4 9 -p o l i t i c a l a c t i v i t y and p a r t i c i p a t i o n i n s e t t i n g c r i t e r i a and standards and i n e v a l u a t i o n programs are powerfu l t o o l s as w e l l . The C a l i f o r n i a Nurses A s s o c i a t i o n has pub l i shed s p e c i f i c g u i d e l i n e s f o r peer rev iew groups which operate i n c o n j u n c t i o n w i t h b a r g a i n i n g u n i t s ( 8 1 ) . Nurses have used job a c t i o n , i n c l u d i n g s t r i k e s , to g a i n power i n d e c i s i o n s about e v a l u a t i o n of care ( 3 1 6 , 4 0 2 , 5 4 0 ) . Maas e t a l . (316) p rov ide a p a r t i c u l a r l y d e t a i l e d account of the development of a n u r s i n g o r g a n i z a t i o n i n one i n s t i t u t i o n brought i n t o be ing v i a a c o n t r a c t u a l agreement which made nurses c o l l e c t i v e l y r e s p o n s i b l e f o r the d e l i v e r y of nurs ing s e r v i c e s , w i t h the n u r s i n g a d m i n i s t r a t i o n r e s p o n s i b l e f o r e x e c u -t i n g p o l i c i e s dec ided by the o r g a n i z a t i o n . As other p r o f e s s i o n a l s i n c r e a s i n g l y assume an employee s t a t u s , the p o s s i b i l i t y of c o l l e c t i v e b a r g a i n i n g as a t o o l to g a i n i n f l u e n c e i n competence m o n i t o r i n g a c t i v i -t i e s may be of g r e a t e r i n t e r e s t to these g roups . The R i s i n g Costs of H e a l t h Care In the s e v e n t i e s concern about h e a l t h care has expanded from concern about the i n d i v i d u a l to concern about the community. As h e a l t h care consumes an ever l a r g e r p r o p o r t i o n of a v a i l a b l e r e s o u r c e s , ques t ions about the e f f i c i e n t d e l i v e r y of t h a t care and i t s va lue to the h e a l t h of the community are asked w i th i n c r e a s i n g f requency . Blum (55) b e l i e v e s that i f our g o a l i s b e t t e r h e a l t h f o r the t o t a l p o p u l a t i o n , maximum b e n e f i t w i l l be d e r i v e d i f we s h i f t our e v a l u a t i v e focus from behav io rs of p r o f e s s i o n a l s to behav io rs of p a t i e n t s , i . e . we should measure h e a l t h s t a t u s , p r a c t i c e s and behav io rs a t en t r y to care and h e a l t h s t a t u s on e x i t . T h i s , of c o u r s e , i s the ph i losophy of Marc La Londe's A New  P e r s p e c t i v e on H e a l t h Care f o r Canadians ( 8 2 ) . M i t c h e l l (343) s t a t e s , - 1 5 0 -"Obv ious ly we must b e g i n to t h i n k about the o v e r a l l h e a l t h system and sub-systems w h i l e not n e g l e c t i n g the n e c e s s i t y f o r t e s t i n g the e x c e l -l e n c e of i n d i v i d u a l a c t s of med ica l c a r e " . Assuming tha t the competence of p r o f e s s i o n a l s w i l l cont inue to be examined, the i s s u e of cos t cannot be i g n o r e d . In d i s c u s s i n g the r e l a t i o n s h i p of q u a l i t y to c o s t , Lewis (298) p o s t u l a t e s tha t there are a t l e a s t three g e n e r a l se ts of c o s t - q u a l i t y f u n c t i o n c u r v e s : 1. l i n e a r and p o s i t i v e ( i . e . the b e t t e r the q u a l i t y , the h igher the c o s t ) ; these seem to be a s s o c i a t e d w i t h the more complex m e d i c a l phenomena, e . g . end stage r e n a l d i s e a s e 2 . a s c a t t e r diagram ( i . e . there i s no r e l a t i o n s h i p between q u a l i t y and c o s t ) ; one would see t h i s w i th such s e l f - l i m i t i n g d i s e a s e s as the common c o l d 3 . q u a d r a t i c i n form ( i . e . 80-90% of e x c e l l e n c e to be ach ieved w i l l be gained w i t h a r e l a t i v e l y s m a l l inves tment , and a cont inued investment w i l l y i e l d on ly marg ina l g a i n s ) ; a f a m i l i a r example would be the i n f o r m a t i o n d e r i v e d from a t h i r t y minute p h y s i c a l as opposed to the exhaust i ve i n - h o u s e check-ups which c o r p o r a t i o n s arrange f o r t h e i r top e x e c u t i v e s . Lewis recommends r e s e a r c h to develop p r o b l e m - s p e c i f i c c o s t e f f e c t i v e n e s s c u r v e s . Such an accomplishment cou ld have profound e f f e c t s on both f o r m u l a t i o n of rev iew c r i t e r i a and the focus of rev iew a c t i v i -t i e s . Rubenste in e t a l , (443) r e p o r t u s i n g process and outcome c r i t e r i a to eva luate emergency room care g i ven women w i t h u r i n a r y t r a c t i n f e c t i o n i n two h o s p i t a l s . In both i n s t a n c e s process and outcome . scores were p o s i t i v e l y c o r r e l a t e d . The data suggest tha t c o r r e l a t i o n i s c u r v i l i n e a r and that there i s a t h r e s h o l d process score below which a poor outcome i s extremely l i k e l y . This study i n d i c a t e s that the g r e a t e s t improvements i n outcome of u r i n a r y t r a c t i n f e c t i o n may r e s u l t from r a i s i n g the q u a l i t y of care from a poor to an adequate l e v e l , r a t h e r than from an adequate to an o p t i m a l l e v e l . - 1 5 1 -The i n f o r m a t i o n prov ided by Lewis and Rubenste in does not support the b e l i e f s of those who c l a i m that PSRO rev iew can e f f e c t i v e l y and s i m u l t a n e o u s l y c o n t r o l cost and improve q u a l i t y . B lumste in (56) contends tha t the PSRO l e g i s l a t i o n , which encourages wholesale adherence to optimum process or outcome c r i t e r i a w i thout regard f o r t h e i r economic impact or impor t , i s a p o t e n t i a l l y e f f e c t i v e and unopposed inst rument f o r i n c r e a s i n g h e a l t h e x p e n d i t u r e s . Whi le perhaps i n c r e a s i n g the t e c h n i c a l q u a l i t y of s e r v i c e s prov ided the i n d i v i d u a l p a t i e n t , the e f f e c t i v e n e s s of the h e a l t h care s e c t o r i n m a i n t a i n i n g or improving the h e a l t h of a d e f i n -ed p o p u l a t i o n may s u f f e r . Severa l other w r i t e r s (51,161,196) have a l s o p r e d i c t e d that the e f f e c t of PSRO l e g i s l a t i o n w i l l be to i n c r e a s e d i r e c t h e a l t h care e x p e n d i t u r e s . I n i t i a l evidence i n d i c a t e s t h a t PSRO-type o r g a n i z a t i o n s a c h i e v e , a t b e s t , m a r g i n a l cost c o n t r o l e f f e c t s and some gains i n q u a l i t y ( 3 8 9 , 4 0 1 , 4 2 4 ) . I t may be concluded that e n t h u s i a s t i c and v i g o r o u s implementa-t i o n of any rev iew program without concern fo r i t s d i r e c t and i n d i r e c t c o s t s and b e n e f i t s cou ld w e l l y i e l d a net l o s s r a t h e r than a net g a i n i n the h e a l t h s t a t e of a g i ven p o p u l a t i o n . M a l p r a c t i c e M a l p r a c t i c e l i t i g a t i o n as a mechanism f o r the o c c a s i o n a l e v a l u -a t i o n of the process and outcome of care has been d i s c u s s e d i n Chapter V. The economic e f f e c t s of m a l p r a c t i c e l i t i g a t i o n on the h e a l t h care system, p a r t i c u l a r l y i n the Un i ted S t a t e s , have i n f l u e n c e d other developments r e l a t e d to e v a l u a t i o n of p r o f e s s i o n a l s . PSRO a c t i v i t y , f o r i n s t a n c e , has been opposed i n the b e l i e f tha t i t w i l l i n c r e a s e m a l p r a c t i c e l i t i g a t i o n by exposing poor c a r e ; i t has a l s o been supported w i t h the p r e d i c t i o n - 1 5 2 -tha t i t w i l l decrease c la ims by p revent ing shoddy p r a c t i c e ( 1 6 9 , 4 3 8 ) . At l e a s t one l i a b i l i t y i n s u r o r has prov ided i n c e n t i v e f o r sys temat i c e v a l u -a t i o n by reduc ing the agency 's co rpora te pr imary coverage m a l p r a c t i c e premium on the assumption that the agency 's rev iew program w i l l not o n l y c o n t r o l u n d e s i r a b l e p r a c t i c e s , but a l s o document proper care ( 2 2 7 ) . L i a b i l i t y - c o n s c i o u s a d m i n i s t r a t o r s are c a r e f u l to keep rev iew records separate from p a t i e n t records ( 3 9 0 ) . Some j u r i s d i c t i o n s support t h i s p r e c a u t i o n by r u l i n g that such records are not d i s c o v e r a b l e as e v i d e n c e . Such i s the concern about m a l p r a c t i c e or co rpora te neg l igence that a new term, " r i s k management program", has appeared i n h e a l t h care l i t e r a t u r e ( 2 1 2 , 2 4 5 ) . A t y p i c a l program i n v o l v e s an a g g r e s s i v e q u a l i t y assessment system supported by a l a r g e budget , a mechanism f o r h a n d l i n g consumer c o m p l a i n t s , management planned campaigns aimed a t promoting good r e l a t i o n s between s t a f f and p a t i e n t s , and t i g h t l y s t r u c t u r e d mechanisms f o r e v a l u a t i n g competence and extending c e r t a i n p r a c t i c e p r i v i l e g e s on ly to s p e c i f i e d p e r s o n n e l . Agency-based l i m i t a t i o n of p r a c t i c e as a s t r a t e g y to avo id l a w -s u i t s has been d i s c u s s e d to some extent i n Chapter I I I . Severa l authors q u e s t i o n whether l i m i t a t i o n of h o s p i t a l p r i v i l e g e s i s j u s t i f i a b l e , as i t puts the h o s p i t a l i n the p o s i t i o n of being a "super l i c e n s i n g board" ( 8 8 , 1 0 0 , 3 9 6 ) . The J o i n t Commission on A c c r e d i t a t i o n of H o s p i t a l s suggests that such procedures need to be a p p l i e d f l e x i b l y , w i t h c o n s i d e r -a t i o n f o r a v a i l a b l e manpower, the needs of the p o p u l a t i o n , and a v a i l a b i l -i t y of a l t e r n a t i v e resources (396) . H o s p i t a l s may a l s o use e x t e r n a l mechanisms, such as s p e c i a l t y c e r t i f i c a t i o n or s p e c i a l l i c e n s e s , as a par t of t h e i r l i t i g a t i o n defense program ( 3 1 0 , 4 1 2 ) . Whether the d e s i r e d e f f e c t i s ever obta ined i s not documented, but E g e l s t o n (155) p r e d i c t s an -153-i n c r e a s i n g use of both i n - h o u s e and e x t e r n a l l y awarded c r e d e n t i a l s , e a r n -ed through a v a r i e t y of e v a l u a t i v e mechanisms, f o r t r u s t e e s , a d m i n i s t r a -t o r s , p h y s i c i a n s , n u r s e s , and a l l i e d h e a l t h p e r s o n n e l . Thus i t appears that e x i s t i n g e v a l u a t i v e dev ices l e a d i n g to a c r e d e n t i a l w i l l cont inue to be used and new ones , p r i m a r i l y agency based , i n c r e a s i n g l y w i l l be s u p e r -imposed on t h i s c r e d e n t i a l i n g non -sys tem. H e a l t h Manpower Rapid t e c h n o l o g i c a l developments coupled w i t h concern f o r r i s -i n g c o s t s has r e s u l t e d i n a p r o l i f e r a t i o n of c a t e g o r i e s of h e a l t h workers (236) . Some of t h e s e , l i k e the p h y s i c i a n a s s i s t a n t , are new workers . Dekker (122) r e p o r t s tha t i n the e leven years s i n c e Duke U n i v e r s i t y open-ed the f i r s t formal t r a i n i n g program f o r p h y s i c i a n a s s i s t a n t s , some f o r t y s t a t e s have enacted l e g i s l a t i o n govern ing t h e i r u s e . Canada has not used t h i s worker . In both c o u n t r i e s n u r s e s ' r o l e s have expanded to i n c l u d e a p h y s i c i a n a s s o c i a t e worker most o f t e n r e f e r r e d to as the nurse p r a c t i -t i o n e r . Bu l lough (75) s t a t e s that n u r s i n g has moved through two major phases i n l i c e n s u r e ( b a s i c r e g i s t r a t i o n and mandatory r e g i s t r a t i o n ) and i s c u r r e n t l y i n a t h i r d ( r o l e e x p a n s i o n ) . In a d d i t i o n , a v a r i e t y of t e c h n i c a l and a u x i l l i a r y workers are be ing t r a i n e d and employed to support newer developments i n s o c i a l and m e d i c a l s e r v i c e s . There a r i s e s the q u e s t i o n , how w i l l these workers be eva luated and otherwise c o n t r o l -led? Some w r i t e r s q u e s t i o n us ing t r a d i t i o n a l l i c e n s i n g systems (176, 325,336) fo r these groups . They po int out tha t new worker c a t e g o r i e s were developed so that h i g h l y pa id and e x t e n s i v e l y educated p r o f e s s i o n a l s would not be doing t e c h n i c a l work. L i censure would r e s t r i c t e n t r y to the - 1 5 4 -o c c u p a t i o n , thus l i m i t i n g a v a i l a b i l i t y of s e r v i c e s and a r t i f i c i a l l y i n f l a t i n g wages. A d d i t i o n a l l y , i t i s b e l i e v e d that l i c e n s i n g promotes r i g i d c a t e g o r i e s of workers , f u r t h e r i n t e r f e r i n g w i t h e f f i c i e n t use of manpower i n a t ime of r a p i d change ( 1 3 4 , 3 3 6 ) . A prominent lawyer i n the American h e a l t h care f i e l d has s u g -gested a l e g i s l a t i v e a l t e r n a t i v e , i n s t i t u t i o n a l l i c e n s u r e ( 2 0 8 , 2 3 5 , 2 3 6 , -4 9 7 ) . Under the Hershey p r o p o s a l , i n s t i t u t i o n s would be r e s p o n s i b l e f o r e s t a b l i s h i n g task l i s t s , job d e s c r i p t i o n s and fo r m o n i t o r i n g the compe-tence of the personne l f i l l i n g such p o s i t i o n s . Th is would be done under the s u r v e i l l a n c e of the s t a t e l i c e n s i n g agency. The p roposa l has sparked s i m i l a r a l t e r n a t i v e p roposa ls ( 1 6 7 ) . Desp i te c o n s i d e r a b l e d i s c u s s i o n , there does not appear to be any s e r i o u s moves toward i n s t i t u t i o n a l l i c e n -s u r e , perhaps because as Guy (208) p o i n t s o u t , such an approach would remove a l l the checks and b a l a n c e s . The c o n t r o l , e v a l u a t i o n , and e f f i c i e n t use of h e a l t h manpower cont inues to be a major i s s u e , compl i ca ted by d i f f i c u l t y of r e a c h i n g n a t i o n a l standards ( 1 2 8 , 1 8 8 ) ; movement of workers , e s p e c i a l l y women, i n and out of the work fo rce (68) ; the presence of h e a l t h p r o f e s s i o n a l s t r a i n e d f o r another h e a l t h care system; and the " f o r e i g n graduate problem" ( 1 2 8 , 2 7 1 , 5 1 1 ) . This i s s u e , and many of those d i s c u s s e d e a r l i e r i n t h i s s e c t i o n , have promoted c o n s i d e r a b l e a c t i v i t y w i t h r e s p e c t to formal c r e d e n t i a l i n g mechanisms fo r h e a l t h care personnel throughout North Amer ica . Regu la to ry Mechanisms and Trends The c r e d e n t i a l i s the cur rency of competence i n North American - 1 5 5 -s o c i e t y . The c r e d e n t i a l i s not competence any more that the d o l l a r b i l l i s g o l d . The u t i l i t y of the c r e d e n t i a l i s g r e a t l y diminshed i f one f a i l s to understand the purpose f o r or c o n d i t i o n s under which i t was g r a n t e d . H i s t o r i c a l development of r e g u l a t o r y mechanisms, l i c e n s u r e and c e r t i f i c a -t i o n today , and c u r r e n t t rands i n c r e d e n t i a l i n g p r a c t i c e s w i l l be d i s c u s -sed i n t h i s s e c t i o n . H i s t o r i c a l Devlopment of Regu la tory Mechanisms The idea tha t s o c i e t y has the r i g h t and r e s p o n s i b i l i t y to c o n t r o l competence of h e a l t h p r o f e s s i o n a l s i s not new. The Code of Hammurabi, c i r c a 1955 B . C . , s p e c i f i e d that those p h y s i c i a n s w i t h records showing a h i g h degree o f m o r t a l i t y amongst t h e i r p a t i e n t s l o s t t h e i r own l i v e s (331) . Derbysh i re ( 1 2 8 ) , p rov ides the f o l l o w i n g account of the d e v e l -opment of r e g u l a t i o n i n medic ine up to the t w e n t i e t h c e n t u r y . The Code of F r e d e r i c k I I , 13th century S i c i l y , l a i d down e d u c a t i o n a l requ i rements , set f e e s , r e g u l a t e d e t h i c a l conduct and other p r a c t i c e b e h a v i o r s , and a l lowed the med ica l f a c u l t y to l i c e n s e p h y s i c i a n s . Surgeons were appren -t i c e d and e v e n t u a l l y examined by masters i n the t rade p r i o r to j o i n i n g a g u i l d . When England i n 1518 granted a c h a r t e r to the Royal C o l l e g e of P h y s i c i a n s and Surgeons, the p r i n c i p l e of s e l f - r e g u l a t i o n of the p r o f e s -s i o n was w e l l e s t a b l i s h e d . L i c e n s u r e by s c h o o l and r e g u l a t i o n by peers i s the system s t i l l predominant i n Europe today . The r e l a t i v e l y u n s t r u c t u r e d s o c i e t y i n the deve lop ing New World c o l o n i e s prov ided few of the t r a d i t i o n a l c o n t r o l s . At f i r s t , t r a i n i n g of p h y s i c i a n s was by a p p r e n t i c e s h i p . In the e a r l y n i n e t e e n t h c e n t u r y , exam-i n a t i o n by the l o c a l med ica l s o c i e t y was r e q u i r e d by law; l a t e r m e d i c a l - 1 5 6 -degrees from e d u c a t i o n a l i n s t i t u t i o n s were accepted i n l i e u of an examin-a t i o n . By the end of the C i v i l War, however, "d ip loma m i l l s " were common and a degree c a r r i e d l i t t l e meaning. Moreover, l o c a l m e d i c a l s o c i e t i e s were o f t e n l i t t l e more than groups of c u l t i s t s . F o r these reasons the concept of d i r e c t examinat ion and l i c e n s u r e by the s t a t e became a r e a l i -t y . I t i s worth n o t i n g tha t both the Uni ted States C o n s t i t u t i o n and the B r i t i s h North America Act (Canada) ass igned mat te rs of h e a l t h to the r e s p o n s i b i l i t y of the s e v e r a l s t a t e s and p r o v i n c e s , r a t h e r than the f e d e r a l governments. The d e c i s i o n s which North American s o c i e t y had made r e g a r d i n g c o n t r o l of p h y s i c i a n s set a p a t t e r n which was to i n f l u e n c e the r e g u l a t i o n of a l l h e a l t h p r o f e s s i o n a l s fo r the next h a l f c e n t u r y . Roemer (426) notes that the contents of l i c e n s i n g laws f o r a l l h e a l t h occupat ions are s i m i l a r because they are model led a f t e r med ica l p r a c t i c e a c t s . G e n e r a l l y they e s t a b l i s h the scope of p r a c t i c e and p r e s c r i b e the c o n d i t i o n s and methods by which l i c e n s u r e can be o b t a i n e d , m a i n t a i n e d , and r e v o k e d . In r e l a t i o n to the l a t t e r , incompetence i s a f r e q u e n t l y mentioned c o n d i t i o n fo r some form of d i s c i p l i n e , the u l t i m a t e s a n c t i o n be ing r e v o c a t i o n of l i c e n s e ( 1 2 8 ) . As the h e a l t h care d e l i v e r y system cont inued to take shape i n the e a r l y t w e n t i e t h c e n t u r y , a d d i t i o n a l attempts to assure competence were i n i t i a t e d . The most s i g n i f i c a n t were a c c r e d i t a t i o n of p r o f e s s i o n a l t r a i n i n g programs, a c c r e d i t a t i o n of h e a l t h care a g e n c i e s , and c e r t i f i c a -t i o n of s p e c i a l t y q u a l i f i c a t i o n s . H i s t o r i c a l l y , a c c r e d i t a t i o n of e d u c a t i o n a l i n s t i t u t i o n s and programs has been a v o l u n t a r y non-governmental process of e v a l u a t i o n which aims to assure that educat ion i s of h i g h q u a l i t y ( 3 6 2 ) . Approval - 1 5 7 -by the s t a t e or p rov ince a l s o e x i s t s i n some occupat ions and g e n e r a l l y i s aimed at ensur ing minimum s t a n d a r d s . Between 1900 and 1930 a c c r e d i t a t i o n was begun i n the f i e l d s of m e d i c i n e , n u r s i n g , d e n t i s t r y , o c c u p a t i o n a l therapy , p h y s i c a l the rapy , p o d i a t r y and osteopathy ; by 1972, a c c r e d i t a -t i o n mechanisms e x i s t e d f o r a t o t a l of f o r t y - t w o occupat ions ( 4 2 6 ) . The number and types of a c c r e d i t i n g s e r v i c e s f o r p r o f e s s i o n a l educat ion programs today are e x t e n s i v e and the s i t u a t i o n i s compl i ca ted by shared and o v e r l a p p i n g j u r i s d i c t i o n s . In the Uni ted S t a t e s , there are even two a c c r e d i t i n g agenc ies to a c c r e d i t the a c c r e d i t o r s . As these s e r v i c e s developed i n the f i r s t h a l f of the c e n t u r y , l i c e n s i n g laws began to i n c l u d e g r a d u a t i o n from an a c c r e d i t e d schoo l as a requirement f o r l i c e n -sure ( 1 0 5 , 1 2 8 , 4 2 6 , 5 1 1 ) . A c c r e d i t a t i o n of h e a l t h care agenc ies a l s o began around s i x t y years ago i n the Uni ted States when the American Co l lege of Surgeons i n i t i a t e d a H o s p i t a l S t a n d a r d i z a t i o n Program ( 8 3 ) . E v e n t u a l l y , c o o p e r a -t i v e agreements among Canadian and American med ica l and h o s p i t a l a s s o c i -a t i o n s l e d to these groups sponsor ing the es tab l i shment of the J o i n t Commission on A c c r e d i t a t i o n of H o s p i t a l s (JCAH) i n 1951. The Canadian C o u n c i l on H o s p i t a l A c c r e d i t a t i o n (CCHA) was e s t a b l i s h e d as a separate e n t i t y i n 1958, and the JCAH then devoted e x c l u s i v e a t t e n t i o n to American i n s t i t u t i o n s . A c c r e d i t a t i o n s e r v i c e s a re g r a d u a l l y be ing broadened to i n c l u d e extended care i n s t i t u t i o n s as w e l l as ambulatory , p e r s o n a l , and i n t e r m e d i a t e care s e t t i n g s . Many of CCHA and JCAH standards r e l a t e to e v a l u a t i o n of personnel and s e r v i c e s (83) . In a d d i t i o n to these v o l u n -t a r y a c c r e d i t a t i o n programs, most h e a l t h care i n s t i t u t i o n s are a f f e c t e d by a v a r i e t y of government l i c e n s i n g arrangements and f i n a n c i n g c o n t r o l s ( 4 2 6 ) . - 1 5 8 -S p e c i a l t y c e r t i f i c a t i o n i s g e n e r a l l y a v o l u n t a r y process and the f u n c t i o n of p r o f e s s i o n a l a s s o c i a t i o n s (426) . U s u a l l y post graduate e d u c a t i o n a l requirements and s u c c e s s f u l comple t ion of an examinat ion are r e q u i r e d . This form of c r e d e n t i a l i s r a r e l y r e g u l a t e d by l a w , but i s o f t e n an important requirement f o r r e c e i v i n g s t a f f p r i v i l e g e s . In a d d i -t i o n an agency w i t h s p e c i a l t y c e r t i f i e d s t a f f may be i n a more f a v o r a b l e p o s i t i o n to r e c e i v e a c c r e d i t a t i o n or garner p r i v a t e or government funds fo r s p e c i a l p r o j e c t s (426) . In summary, as r e c e n t l y as the f i f t i e s , the p u b l i c and the p r o f e s s i o n s i n the Uni ted States and Canada were a p p a r e n t l y s a t i s f i e d w i t h a system f o r a s s u r i n g competent care which r e s t e d upon three broad f o u n d a t i o n s : 1 . l i c e n s u r e of the h e a l t h p r o f e s s i o n a l a f t e r he had graduated from an approved or a c c r e d i t e d e d u c a t i o n a l program; s u c c e s s f u l l y completed an e x a m i n a t i o n , and met v a r i o u s o ther requ i rements . ( In most Canadian prov inces government c o n t r o l i s i n d i r e c t w i t h these f u n c t i o n s be ing de legated to p r o f e s s i o n a l a s s o c i a t i o n s ) ; 2. c e r t i f i c a t i o n , u s u a l l y by the p r o f e s s i o n , of a d d i t i o n a l s p e c i a l i z e d q u a l i f i c a t i o n s f o r those p r o f e s s i o n a l s w i s h i n g to be so r e c o g n i z e d ; and 3 . a combinat ion of v o l u n t a r y s e l f - r e g u l a t i o n and government r e g u l a t i o n of h e a l t h care agencies where a t l e a s t some p r o f e s s i o n a l s p r a c t i c e a t l e a s t some of the t i m e . Of t h e s e , o n l y the f i r s t was u n i v e r s a l , and there i s ev idence to suggest tha t t h i s has not been e f f e c t i v e l y implemented ( 9 9 , 1 2 8 , 3 3 6 , 3 6 9 , 4 2 7 ) . For a d e t a i l e d and comprehensive rev iew of past and present r e g u l a t o r y mechanisms, s e v e r a l summaries p rov ide more e x t e n s i v e i n f o r m a -t i o n ( 6 0 , 1 2 8 , 3 6 2 , 4 2 6 , 5 1 1 , 5 1 2 , 5 1 3 ) . - 1 5 9 -L i c e n s u r e and C e r t i f i c a t i o n Today The l i c e n s e i s the c r e d e n t i a l common to the four p r o f e s s i o n s cons idered i n t h i s paper . W r i t e r s agree tha t i t s purpose i s p r o t e c t i o n of the p u b l i c from incompetent p r a c t i t i o n e r s ( 9 5 , 1 0 0 , 1 3 6 , 1 6 7 , 2 0 0 , 2 7 5 , 4 7 2 ) . D e s p i t e t h i s apparent agreement, Cohen and M i i k e w r i t e that there i s c o n f u s i o n on t h i s po in t and that some b e l i e v e l i c e n s u r e i s f o r a d m i s -s i o n of new members to a p r o f e s s i o n ( 1 0 0 ) . K i n k e l a and K i n k e l a ' s (178) more d e t a i l e d e x p l a n a t i o n of the p r o t e c t i o n of the p u b l i c argument can be summarized as f o l l o w s : 1. The r i g h t to pursue a l a w f u l c a l l i n g , bus iness or p r o f e s s i o n i s a fundamenta l , but not an a b s o l u t e r i g h t ; i . e . i t i s one which i s subord inate to the p u b l i c good. 2 . The s t a t e p r o t e c t s the p u b l i c good through the e x e r c i s e of i t s p o l i c e power; t h i s power extends to a l l matters p e r t a i n i n g to the p u b l i c h e a l t h , w e l f a r e and s a f e t y . 3 . L i censure of h e a l t h care p r o f e s s i o n a l s i s one example of the proper e x e r c i s e of the s t a t e ' s p o l i c e power. 4. The s t a t e l e g i s l a t u r e i s the on ly body empowered to determine what l i c e n s u r e laws s h a l l b e . 5 . A d m i n i s t r a t i o n of the law may be d e l e g a t e d . A l though w r i t t e n about American law , each o f the statements above a p p l i e s under Canadian law to the s e v e r a l p r o v i n c e s . With r e f e r e n c e to the l a s t i t e m , d e l e g a t i o n i n the Un i ted Sates i s most o f t e n to a s t a t e - a p p o i n t e d board or commiss ion ; i n Canada, to a named p r o f e s s i o n a l o r g a n i z a t i o n . In both c o u n t r i e s t h i s r a t i o n a l e i s used to l i c e n s e , or g i v e p e r m i s s i o n t o , des ignated i n d i v i d u a l s to p r a c t i c e and to bar a l l o thers from p r a c t i c e (mandatory l i c e n s u r e ) . Sheppard (472) and F o r n i (167) n o t e , however, that l i c e n s u r e may a l s o s imply confe r e x c l u s i v e r i g h t to use a s p e c i f i e d t i t l e ( v o l u n t a r y l i c e n s u r e ) . L i c e n s e d groups m a i n t a i n a r e g i s t e r of - 1 6 0 -those l i c e n s e d to p r a c t i c e or to use a p r o t e c t e d t i t l e , and they o f t e n use the term " r e g i s t r a t i o n " . This i s a c o n f u s i n g po in t as any o r g a n i z a -t i o n can i s s u e a c e r t i f i c a t e of membership r e g i s t r a t i o n to i t s members, w i t h or wi thout permiss ion of the s t a t e . To e s t a b l i s h tha t a h e a l t h worker i s l i c e n s e d , i . e . c o n t r o l l e d i n some way by the s t a t e , one must know that the law p r o t e c t s h i s r i g h t to p r a c t i c e and/or ho ld t i t l e . I f one c o n s i d e r s the market ing e q u i v a l e n t of these two s t r a t e g i e s (mandatory l i c e n s u r e would be l i k e removing from the s h e l f a l l goods thought to be d e f e c t i v e ; v o l u n t a r y l i c e n s u r e would be s i m i l a r to l e a v i n g a l l goods on the s h e l f , but l a b e l l i n g those thought to be s a f e ) , the economic i m p l i c a -t i o n s become o b v i o u s . C l e a r l y a key p o i n t i n d e c i d i n g tha t mandatory l i c e n s u r e i s i n the p u b l i c i n t e r e s t i s i n p r e d i c t i n g how much damage a p o t e n t i a l l y d e f i c i e n t p r a c t i t i o n e r i s l i k e l y to do and how s u c c e s s f u l l y h i s performance can be monitored to prevent damage. C e r t i f i c a t i o n i s used by the p r o f e s s i o n s u s u a l l y to recogn ize s u p e r i o r performance, o f t e n i n a narrow p r a c t i c e area or s p e c i a l t y (148, 3 0 6 ) . L loyd (306) s t a t e s tha t c e r t i f i c a t i o n i s important i n a s s u r i n g h igh q u a l i t y because i t e s t a b l i s h e s standards above the minimum l i c e n s i n g l e v e l . Dunkley (148) n o t e s , however, the h i s t o r i c a l ev idence that c e r t i -f i e d e x c e l l e n c e i n one g e n e r a t i o n becomes a minimum standard f o r s p e c i a l -t y p r a c t i c e i n the n e x t . C h i l d and Zuidema (95) w r i t e that as p r o f e s -s i o n a l o r g a n i z a t i o n s develop standards i n response to s e l f - r e g u l a t o r y p r i v i l e g e s , these have been s u b s t a n t i v e l y borrowed and w r i t t e n i n t o l a w . An example would be the use of the N a t i o n a l Board of Med ica l Examinat ions t e s t s f o r l i c e n s u r e purposes . I t can be concluded from the fo rego ing t h a t l i c e n s u r e i s a s s o -c i a t e d w i t h s t a t e or p r o v i n c i a l c o n t r o l and the concept of a minimum - 1 6 1 -standard of performance below which the h e a l t h , s a f e t y , or w e l l - b e i n g of the p u b l i c i s j e o p a r d i z e d . C e r t i f i c a t i o n , on the other hand, i s more o f t e n a s s o c i a t e d w i t h standards above minimum l e v e l and w i t h p r i v a t e c o n t r o l , u s u a l l y from w i t h i n the p r o f e s s i o n . Both of these mechanisms, together w i t h r e g i s t r a t i o n p r a c t i c e s of groups not recogn ized by law and w i t h c e r t i f i c a t i o n p r a c t i c e s of e d u c a t i o n a l i n s t i t u t i o n s which award d i p l o m a s , c e r t i f i c a t e s , and degrees , comprise a complex and formal c r e d e n t i a l i n g network f o r h e a l t h p r o f e s s i o n a l s . R a t i o n a l i z a t i o n of L i c e n s i n g Procedures - Trend One L i c e n s i n g p r a c t i c e s are as v a r i e d as the s t a t e s and prov inces r e g u l a t i n g them; Derbysh i re b l u n t l y d e s c r i b e s the s i t u a t i o n as "chaos" (128) . Perhaps the most n o t a b l e c r e d e n t i a l i n g t rend i n the s e v e n t i e s has been c o n s i d e r a b l e s t u d y , d r a f t i n g and passage of l i c e n s i n g laws which attempt to ach ieve a system which i s c o n s i s t e n t , l o g i c a l and e f f e c t i v e . Th is be ing attempted i n a number of ways: 1. The l i c e n s i n g of new c a t e g o r i e s of h e a l t h workers i s be ing d i s c o u r -aged ( 9 0 , 1 5 5 , 3 0 6 , 3 2 5 , 4 1 4 , 5 1 0 , 5 1 1 , 5 1 2 ) . The u s u a l reasons g i v e n are that l i c e n s i n g c o n t r i b u t e s to r i s i n g h e a l t h care c o s t s and has not been shown to be e f f e c t i v e (325) . L i c e n s i n g i s cons idered j u s t i f i -ab le on ly where consumers w i l l be s u b s t a n t i a l l y harmed by l a c k of c o n t r o l and there are no a l t e r n a t i v e methods of p r o t e c t i o n . Seve ra l sources ( 9 0 , 4 1 4 , 4 4 0 , 5 1 0 , 5 1 1 , 5 1 2 ) e s t a b l i s h f a i r l y e x t e n s i v e c r i t e r i a which should be met before an occupat ion i s l i c e n s e d . 2. Where l i c e n s i n g i s deemed n e c e s s a r y , v o l u n t a r y l i c e n s u r e ( p r o t e c t i o n of t i t l e ) i s favored i f t h i s w i l l a f f o r d adequate p r o t e c t i o n . Reserved t i t l e i s cons idered s u f f i c i e n t f o r p r o f e s s i o n a l s who p r a c t i c e i n a s t r u c t u r e d environment sub jec t to ongoing c o n t r o l ( 9 0 ) . - 1 6 2 -3 . Where l i c e n s i n g i s used , there i s an i n c r e a s i n g tendency to favor d e f i n i n g the scope of p r a c t i c e ( 3 3 , 9 0 , 1 9 9 , 2 3 7 , 4 4 1 , 5 1 2 ) . Rozovsky ( 4 4 1 ) , i n d i s c u s s i n g d e f i n i t i o n of n u r s i n g p r a c t i c e , c a u t i o n s tha t attempts to d e f i n e n u r s i n g i n law ignores the d e s i r a b i l i t y of change. He suggests tha t who does what, and when, i s o f t e n a matter of p r o f e s s i o n a l judgement. 4 . N a t i o n a l l i c e n s i n g exams are f a v o r e d , w i t h the s tandard be ing l e f t to the s t a t e or prov ince ( 1 3 6 , 4 1 4 , 5 1 0 , 5 1 1 , 5 1 2 ) . N a t i o n a l standards f o r c e r t i f i c a t i o n are a l s o advocated ( 3 0 6 , 4 1 4 , 5 1 0 , 5 1 1 , 5 1 2 ) . The t rend towards n a t i o n a l s tandards i s supported by the i n c r e a s i n g s o p h i s t i c a -t i o n and w ide -sp read use of equ i va lency and p r o f i c i e n c y exams (156, 2 7 1 , 4 1 4 , 5 1 0 , 5 1 2 ) . 5 . The f e a s i b i l i t y of l i m i t e d l i c e n s u r e ( r e s t r i c t i o n of p r a c t i c e to a s p e c i a l t y area w i t h i n the d i s c i p l i n e ) i s exp lored and u s u a l l y r e j e c t -e d . Whi le t h i s has been suggested as an approach to b a s i c l i c e n s u r e ( 5 8 , 8 8 , 4 9 1 ) , the more f requent suggest ion i s an u n l i m i t e d b a s i c l i c e n s e , w i t h s p e c i a l t y c e r t i f i c a t i o n used to des ignate those a b l e to g ive adequate care i n a s p e c i a l t y p r a c t i c e area ( 2 6 , 3 0 6 , 4 9 7 , 5 3 8 ) . Opin ions as to whether t h i s should be c o n t r o l l e d by law v a r y . 6. Present mechanisms are be ing strengthened by p r o v i d i n g a d m i n i s t r a t i v e agenc ies and boards w i t h adequate s t a f f and budget to e f f e c t i v e l y implement e x i s t i n g l i c e n s i n g l a w s , e s p e c i a l l y the p r o v i s i o n s f o r d i s c i p l i n e ( 9 0 , 4 1 4 , 5 1 0 , 5 1 2 ) . 7. There i s a s t rong tendency to e s t a b l i s h a s i n g l e l i c e n s i n g a u t h o r i t y w i t h i n each j u r i s d i c t i o n which would oversee a l l l i c e n s i n g a c t i v i -t i e s , thus i n c r e a s i n g the l i k l i h o o d of compliance w i t h e s t a b l i s h e d p o l i c y ( 6 0 , 9 0 , 2 3 7 , 4 4 0 , 4 6 2 ) . - 1 6 3 -Broader Base of L i c e n s i n g C o n t r o l - Trend Two K e l l y s t a t e s tha t the background, q u a l i f i c a t i o n s and ph i losophy of l i c e n s i n g board members are of prime importance to the l i c e n s i n g p r o c e s s , as these people eva luate q u a l i f i c a t i o n s of a p p l i c a n t s f o r l i c e n -sure ( 2 7 1 ) . I t i s t h e r e f o r e not s u r p r i s i n g tha t the compos i t ion of l i c e n -s i n g boards has been q u e s t i o n e d . This has r e s u l t e d i n a move to extend the membership base of the r e g u l a t o r y body beyond the p r o f e s s i o n be ing r e g u l a t e d (306) . Most f r e q u e n t l y an attempt i s made to i n c l u d e consumers on the r e g u l a t i n g b o a r d . Al though Derbyshi re (130) s t a t e s that h i s t o r i c a l l y p h y s i c i a n s have opposed l a y people on b o a r d s , t h i s i s a n o t i c e a b l e t rend i n l i c e n s u r e p r a c t i c e s and i s supported by many groups i n c l u d i n g m e d i c a l ones ( 8 9 , 1 0 0 , 1 6 4 , 2 3 7 , 2 3 5 , 4 1 4 , 4 6 2 , 5 1 0 , 5 1 1 ) . Jus t why t h i s i s b e l i e v e d to be important i s not so c l e a r . Advocates of l a y membership suggest t h a t , a t the very l e a s t , t h i s i s a mechanism f o r p u b l i c a c c o u n t a b i l i t y . R . E . A l l e y , Chairman of the Board of the Consumers A s s o c i a t i o n of Canada, b e l i e v e s l a y p a r t i c i p a t i o n w i l l be more e f f e c t i v e i f the l a y r e p r e s e n t a -t i v e must repor t back to a group. "In my e x p e r i e n c e , the i n t e r n a l dynamic of working committees or boards i s such tha t a l l p a r t i c i p a n t s get caught up i n the common r u l e s of l o g i c and schemes of p e r c e p t i o n . This i s not always or even u s u a l c u l p a b l e c o - o p t i o n of the l a y p e r s o n , but r a t h e r a phenomenon more l i k e a c c u l t u r a t i o n to the m i l i e u " (368) . Arguing a g a i n s t l a y membership on r e g u l a t o r y bod ies i s a k i n to argu ing a g a i n s t motherhood, but a t l e a s t one author f l a t l y s t a t e s tha t l a y persons add to the cos t and not the e f f e c t i v e n e s s of l i c e n s i n g boards (267) . Less common than consumer p a r t i c i p a t i o n i n r e g u l a t i o n of h e a l t h p r o f e s s i o n a l s i s the suggest ion that a l l i e d h e a l t h groups be i n v o l v e d . - 1 6 4 -Th is has been favored by some groups i n the Un i ted S t a t e s ( 1 6 4 , 4 1 4 , 4 6 2 , 510) . A d d i t i o n a l l y , the P r o f e s s i o n s Code i n Quebec (90,440) prov ides f o r an i n t e r d i s c i p l i n a r y board and i n t e r p r o f e s s i o n a l a d v i s o r y c o u n c i l ; t h i s board oversees the r e g u l a t o r y a c t i v i t i e s of each p r o f e s s i o n a l group. M u l t i p l e Assessment Procedures - Trend Three Another prominent c r e d e n t i a l i n g t rend which i s i n c r e a s i n g l y d i scussed i n the l i t e r a t u r e i s the use of d i f f e r e n t types of assessment procedures p r i o r to the g r a n t i n g of a c r e d e n t i a l . U n t i l r e c e n t l y the almost u n i v e r s a l p r a c t i c e has been to award a c r e d e n t i a l a f t e r s u c c e s s f u l complet ion of an a c c r e d i t e d e d u c a t i o n a l program and a w r i t t e n or o r a l exam ( 3 0 1 ) . The s e v e n t i e s has seen the emergence of many d i f f e r e n t requirements and assessment techniques used a lone and i n v a r y i n g combina-t i o n s . Most of the more unorthodox assessment a c t i v i t y has been under -taken by the p r o f e s s i o n s i n connec t ion w i th c e r t i f i c a t i o n p rocedures . In p r e s e n t i n g g u i d e l i n e s f o r p h y s i c i a n r e c e r t i f i c a t i o n , Brewer (59) s t a t e s that assessment should be based on a po int system which c o n s i d e r s t r a i n i n g and present p r o f e s s i o n a l s t a n d i n g , p o s t - g r a d u a t e s t u d y , r e - e x a m i n a t i o n of b a s i c knowledge i n the s p e c i a l t y , and p e r i o d i c rev iew of surgery performed or p a t i e n t s t r e a t e d . Reals (406) r e p o r t s that many s p e c i a l t y s o c i e t i e s are making use of a v a r i e t y of t e c h n i q u e s , i n c l u d i n g e x a m i n a t i o n s , r e q u i r e d p a r t i c i p a t i o n i n c o n t i n u i n g e d u c a t i o n , and peer rev iew of p r a c t i c e . The American Co l lege of Radio logy has adopted the p r i n c i p l e of o n - s i t e e v a l u a t i o n of p h y s i c i a n performance, wh i le the American Co l lege of Family P r a c t i c e i s examining the f e a s i b i -l i t y of records rev iew ( 4 8 6 ) . The American Board of I n t e r n a l Medic ine has adopted a three par t c e r t i f i c a t i o n process which c o n s i s t s of t r a i n i n g - 1 6 5 -i n an approved program, on -go ing e v a l u a t i o n of c l i n i c a l competence d u r i n g tha t program a g a i n s t s p e c i f i e d performance c r i t e r i a , and c o g n i t i v e t e s t -i n g ( 4 6 ) . The s p e c i a l t y c e r t i f i c a t i o n program sponsored by the American Nurses A s s o c i a t i o n a l s o depends on a v a r i e t y of assessment p rocedures . Bas ic to the c e r t i f i c a t i o n process i n any of the s p e c i a l t i e s are the requirements tha t nurses must have a t l e a s t two y e a r s ' exper ience and be c u r r e n t l y p r a c t i c i n g i n the s p e c i a l t y , must pass a w r i t t e n e x a m i n a t i o n , and must submit , f o r panel rev iew a g a i n s t p r e - e s t a b l i s h e d s t a n d a r d s , both documented evidence of hav ing g i ven e x c e l l e n t care and r e f e r e n c e s from p r o f e s s i o n a l c o l l e a g u e s ( 2 3 , 2 4 , 9 1 , 1 4 8 ) . In a more recent development, the ANA has committed i t s e l f to a t w o - l e v e l c e r t i f i c a t i o n program which w i l l i n c l u d e c e r t i f y i n g f o r competence (adequacy) i n a s p e c i a l t y as w e l l as f o r e x c e l l e n c e . Al though a masters degree i n n u r s i n g i s p r e - r e q u i s i t e to c e r t i f i c a t i o n a t the l e v e l of e x c e l l e n c e , i t i s not c l e a r e x a c t l y how these two l e v e l s of performance w i l l be e s t a b l i s h e d or eva luated ( 2 6 ) . The Washington State Board of Nurs ing has e s t a b l i s h e d l i c e n s u r e r e q u i r e -ments a p p l i c a b l e to two s p e c i a l l y l i c e n s e d c a t e g o r i e s of Reg is te red Nurse : Advanced and S p e c i a l i z e d ; these requirements i n c l u d e ev idence of p a r t i c i p a t i o n i n c o n t i n u i n g educat ion r e l a t e d to the s p e c i a l t y , a resume of c u r r e n t c l i n i c a l p r a c t i c e , and a minimum of three r e f e r e n c e s document-ing cont inued c l i n i c a l competence (538) . R e - C r e d e n t i a l i n g - Trend Four There i s genera l support f o r the idea tha t any c r e d e n t i a l which c e r t i f i e s a degree o f competence should be s u b j e c t to p e r i o d i c renewal based on evidence of c o n t i n u i n g competence ( 1 , 1 6 4 , 3 0 6 , 3 3 3 , 4 0 7 , 4 1 4 , 4 8 0 , -166-510). The American Board of Medical Specialties, comprised of some twenty-two specialty groups, has adopted a position which supports volun-tary recertification of medical specialists; a l l of the member boards have individually endorsed this. In discussing the ABMS position how-ever, Reals (406) raises a number of potentially problematic issues: older physicians may experience considerable personal d i f f i c u l t y i f they are not grandfathered, certifications would have to be dated and the public and professions informed as to the meaning of such dating, there may be a decrease in available medical manpower as physicians prepare to meet recertification requirements, and, f i n a l l y , there is the as yet unanswered question of whether recertification is worth the costs in the presence of more direct performance monitoring procedures. Recognizing the problems inherent in any large scale recreden-tialing scheme, government groups responsible for licensing are inclined to move rather slowly, leaving exploratory work to be done by the more elite and smaller professional groups. A federal United States committee on credentialing of health manpower has recommended adoption of mechan-isms which assure continued competence and intensive study of the effect-iveness of these mechanisms as an item of high priority (414,510,512). A Michigan Task Force fi n a l report on the same subject recommends that each of the professional licensing boards in the state be given six years to experiment with various approaches to assessing competency and to develop a plan for competency-based relicensure (182). The report has not been acted upon. The literature offers l i t t l e in the way of definitive proposals or answers on the "how to" of competency-based recredentialing. Child and Zuidema (95) advocate a national registry in which data about - 1 6 7 -s u r g i c a l o p e r a t i o n s would be entered a f t e r peer e v a l u a t i o n ; cooperat ion wi th the r e g i s t r y would be a c o n d i t i o n of r e c e r t i f i c a t i o n . In J u l y 1976, the Georgetown U n i v e r s i t y H e a l t h P o l i c y Center i s s u e d a "model med ica l p r a c t i c e a c t " , recommending tha t s t a t e med ica l boards r e q u i r e r e l i c e n s i n g every s i x y e a r s , based on e v a l u a t i o n of the q u a l i t y of care prov ided by the a p p l i c a n t . While vague on e x a c t l y how t h i s might o c c u r , i t i s s u g -gested tha t recommendation of a s p e c i a l t y board cou ld be accepted as evidence of competence (131) . R i chards (417) has o f f e r e d a more developed proposa l on how to e f f e c t competency-based r e l i c e n s u r e . He suggests a f i v e s tep p r o c e s s : 1 . f o r each p r a c t i t i o n e r , develop a p r a c t i c e p r o f i l e and s e l e c t the r e l e v e n t s t a n d a r d s ; 2 . rev iew h i s records a g a i n s t s tandards ; 3 . examine r e l e v a n t knowledge ci£ accept attendance at r e l e v a n t c o n t i n u -i n g educat ion o f f e r i n g s ; 4. t e s t c r i t i c a l s k i l l s ov_ accept documented evidence of these s k i l l s ; 5 . rev iew r e f e r e n c e s , c o m p l a i n t s , and peer statements f o r ev idence of e t h i c a l f i t n e s s . R ichards e s t i m a t e s that a th ree year r e l i c e n s u r e c y c l e would be p o s s i -b l e . He b e l i e v e s tha t i n such a system 95% of p r a c t i t i o n e r s should r e l i c e n s e wi thout d i f f i c u l t y , that 1 - 2 % would l o s e l i c e n s e s , and that the remainder would r e q u i r e remed ia l work. There i s no i n f o r m a t i o n as to how these es t imates are reached . I t does not appear that R ichards has cons idered c o s t s except to say i f we b e l i e v e competency based r e l i c e n s u r e i s important we must be w i l l i n g to f i n a n c e the process and bear the cost of r e s u l t s . - 1 6 8 -Mandatory Cont inu ing Educat ion There e x i s t s an enormous a r r a y of l i t e r a t u r e which focuses on the i d e a that r e q u i r i n g p r o f e s s i o n a l s to p a r t i c i p a t e i n c o n t i n u i n g educa-t i o n w i l l ensure t h e i r cont inued competence. The d i s c u s s i o n which f o l l o w s i s based on a sample of r e l e v e n t i t e m s ; i t i s not an exhaus t i ve r e v i e w . The m a t e r i a l i s o rgan ized under the broad headings of b a c k -ground, pros and cons of mandatory c o n t i n u i n g e d u c a t i o n , mandatory c o n -t i n u i n g educat ion f o r s p e c i a l purposes , and a c c r e d i t i o n of c o n t i n u i n g educat ion educat ion programs. H i s t o r i c a l Background When consumerism and human r i g h t s became c o n t e n t i o u s i s s u e s i n the s i x t i e s , the a p p l i c a t i o n of these concepts to h e a l t h care became i n e v i t a b l e . P r o t e c t i o n of the p u b l i c from shoddy h e a l t h care p r a c t i c e s was a cha l lenge to government and the p r o f e s s i o n s . Howard (248) r e p o r t s that a 1967 Un i ted S ta tes N a t i o n a l Adv i so ry Commission on H e a l t h Manpower noted the need f o r cont inued l e a r n i n g and concluded t h a t making educat ion o p p o r t u n i t i e s a v a i l a b l e to p r o f e s s i o n a l s would not assure t h e i r u t i l i z a -t i o n wi thout i n c e n t i v e s such as r e q u i r i n g examinat ion or educat ion f o r r e l i c e n s u r e . Whi le not n e c e s s a r i l y agree ing w i t h the c o n c l u s i o n , the American Med ica l A s s o c i a t i o n d i d recogn ize the need f o r cont inued l e a r n -i n g and i n 1968 e s t a b l i s h e d the v o l u n t a r y P h y s i c i a n ' s R e c o g n i t i o n Award as a way of encouraging and commending p h y s i c i a n s who c a r r i e d out a pe rsona l program of c o n t i n u i n g m e d i c a l educat ion ( 3 2 6 ) . The pressures i n the s e v e n t i e s to make educat ion o p p o r t u n i t i e s a v a i l a b l e , or even to make t h e i r u t i l i z a t i o n mandatory, have been enumer-- 1 6 9 -ated by a number of a u t h o r s . Among these p ressures are demand by consum-ers fo r the b e t t e r care ( 3 3 8 , 3 4 9 ) ; a d e s i r e on the par t of the p r o f e s s i o n s to impress p o l i t i c i a n s and pressure groups w i t h t h e i r good i n t e n t i o n s to p ro tec t the p u b l i c ( 9 2 , 1 1 2 , 4 1 8 , 4 6 3 ) ; the need to f i n d ways of coping w i t h new developments i n med ica l s c i e n c e and h e a l t h s e r v i c e d e l i v e r y p a t t e r n s ( 1 1 2 , 3 3 8 , 4 6 3 ) ; a shortage of resources to implement any more e f f e c t i v e ( e . g . competency-based) c o n t r o l system (418) ; and a b e l i e f that p a r t i c i p a t i o n i n ongoing p r o f e s s i o n a l educat ion w i l l decrease p r a c t i c e e r r o r s , decrease the i n c i d e n c e of s u c c e s s f u l m a l p r a c t i c e s u i t s , and keep insurance premiums down ( 1 8 2 , 3 3 3 ) . C o r b a l l y (112) adds to t h i s l i s t a need to i n c r e a s e educat ion i n s t i t u t i o n enro l lments i n order to b o l s t e r sagging revenue and a d e s i r e to i n c r e a s e p r o f e s s i o n a l e l i t i s m . I t i s i n s t r u c t i v e to note tha t concern about cos t i s as ev ident as concern about q u a l i t y i n the pressures which have been i d e n t i f i e d . Whether or not these f o r c e s i n d i v i d u a l l y are l e g i t i m a t e reasons f o r r e q u i r i n g p a r t i c i p a t i o n i n e d u c a t i o n , they c o l l e c t i v e l y have r e s u l t e d i n c o n s i d e r a b l e l e g i s l a t i v e a c t i v i t y . I t i s d i f f i c u l t to know the extent of such a c t i v i t y i n the North American j u r i s d i c t i o n s at any one t i m e . O c c a s i o n a l l y a r t i c l e s have summarized the cur rent s t a t e of a f f a i r s p e r t a i n i n g to mandatory c o n t i n u -i n g educat ion i n medic ine ( 2 3 3 , 2 3 4 ) , n u r s i n g ( 4 8 5 , 5 3 7 ) , d e n t i s t r y ( 9 2 ) , and pharmacy (164) . With the except ion of medic ine however, these appear i r r e g u l a r l y and o f f e r an incomplete p i c t u r e . The year end "Report on Med ica l Educat ion i n the Uni ted S t a t e s " i n the J o u r n a l of the American  M e d i c a l A s s o c i a t i o n i s h e l p f u l f o r l e a r n i n g of recent developments i n connect ion w i t h mandatory c o n t i n u i n g med ica l e d u c a t i o n . The above l i t e r -a t u r e r e v e a l s a recent slow-down i n the t rend toward r e q u i r i n g c o n t i n u i n g - 1 7 0 -educat ion as a c o n d i t i o n of r e l i c e n s u r e . L e g i s l a t i o n may be d i s c r e t i o n -a r y , i . e . a l l o w l i c e n s i n g boards to r e q u i r e e d u c a t i o n , or d i r e c t i v e , i . e . demand such c o n t r o l measures ( 3 3 3 , 5 3 7 ) . As j u r i s d i c t i o n s have sought ways of implementing mandatory c o n t i n u i n g e d u c a t i o n , there has been d i s c u s s i o n about a p p r o p r i a t e r o l e s f o r some of the i n v o l v e d groups . Knox (282) s t a t e s tha t p r o f e s s i o n a l a s s o c i a t i o n s should d e f i n e areas of s p e c i a l i z a t i o n and r e l a t e d p r a c t i c e standards so as to g i ve d i r e c t i o n to e d u c a t i o n a l p l a n n i n g . He a l s o s t a t e s that v a r i o u s types of sponsors should cont inue to o f f e r e d u c a t i o n -a l programs, as t h i s w i l l p rov ide the p r a c t i t i o n e r w i t h l e a r n i n g o p p o r t u -n i t i e s r i c h i n d i f f e r e n t p e r s p e c t i v e s and w i l l keep c o n t i n u i n g educat ion f l e x i b l e and respons ive to the needs of the i n d i v i d u a l . In another a r t i c l e , Knox (2 80) emphasizes the r e s p o n s i b i l i t y of p r o f e s s i o n a l a s s o c i -a t i o n s f o r suppor t ing members who wish to engage i n s e l f - d i r e c t e d l i f e -long l e a r n i n g . Weir (5 29) s t a t e s e m p h a t i c a l l y tha t i n s t i t u t i o n s of h igher educat ion do not have a r o l e i n the p o l i c i n g of p r o f e s s i o n a l s , but can make t h e i r best c o n t r i b u t i o n s by i d e n t i f y i n g the pros and cons of mandatory c o n t i n u i n g e d u c a t i o n ; a s s i s t i n g i n the i d e n t i f i c a t i o n of c o n t i n u i n g educat ion needs ; p a r t i c i p a t i n g i n e d u c a t i o n a l p lanning^ b r i n g -ing new developments to the p r a c t i c e a r e n a ; and c o n t r i b u t i n g to e v a l u a -t i o n , r e s e a r c h and development i n c o n t i n u i n g e d u c a t i o n . Pros and Cons of U n i v e r s a l Mandatory Cont inuing Educat ion The bu lk of the l i t e r a t u r e i s concerned w i t h making educat ion mandatory f o r a l l h e a l t h p r o f e s s i o n a l s as a c o n d i t i o n of r e l i c e n s u r e . (A more s e l e c t i v e a p p l i c a t i o n of compulsory e d u c a t i o n e x i s t s i n both theory and p r a c t i c e and i s d i s c u s s e d i n the next s e c t i o n . ) The reasons which - 1 7 1 -have been advanced f o r and a g a i n s t mandatory c o n t i n u i n g educat ion (MCE) are summarized i n t a b l e s V I I and V I I I . I t should be remembered that the contex t f o r these arguments i s the p roposa l of u n i v e r s a l c o n t i n u i n g e d u -c a t i o n as a requirement f o r r e l i c e n s u r e . Arguments i n Favor of MCE 1 . MCE w i l l make educat ion more a v a i l a b l e and a c c e s s i b l e to p r o f e s s i o n a l s ( 3 8 5 ) . 2 . MCE w i l l r e s u l t i n b e t t e r q u a l i t y e d u c a t i o n ; i . e . improved i n s t r u c t i o n a l d e s i g n ( 3 8 5 ) . 3 . MCE w i l l , a t the ve ry l e a s t , ensure that p r a c t i t i o n e r s have access to needed knowledge ( 5 1 1 ) . 4 . MCE must be u s e f u l , o therwise i t s p o p u l a r i t y would not cont inue to i n c r e a s e ( 2 4 6 ) . 5 . MCE w i l l s a t i s f y the p u b l i c demand f o r a c c o u n t a b i l i t y ; i . e . i t w i l l make the p r o f e s s i o n s " l o o k good" (112) . 6 . MCE i s the o n l y p o s i t i v e a c t i o n f o r competence that s o c i e t y can p r e s e n t l y a f f o r d and ach ieve (418) ; i n p a r t i c u l a r i t i s r e l a t i v e l y inexpens i ve i n terms of bookkeeping (130) and c o n t i n u i n g a v a i l a b i l i t y of manpower. 7 . MCE w i l l make p r a c t i t i o n e r s consc ious of the importance of cont inued competence,thus w i l l a c t as a m o t i v a t i n g f a c t o r f o r improved p r a c t i c e ( 5 1 1 ) . 8 . P a r t i c i p a t i o n i n MCE i s p o s i t i v e l y c o r r e l a t e d w i t h e a r l y adopt ion of i n n o v a t i o n s which i s c o r r e l a t e d w i t h a r e p u t a t i o n of competence among peers (2 4 6 ) . 9 . A l though MCE cannot guarantee competence, not hav ing to p a r t i c i p a t e i n e d u c a t i o n a l events guarantees even l e s s (1 11) . 10 . MCE has an e f f e c t on competence which i s analogous to drops i n a pond.j i . e . our measures a r e n ' t d i s c r i m i n a t i n g enough to s u b s t a n t i a t e impact of p a r t i c i p a t i o n i n one o f f e r i n g , but e f f e c t i s cumulat i ve ( 9 8 ) . Table VI I - 1 7 2 -I t can be seen that suppor t ing arguments f o r MCE range from the reason -ab le to the i l l o g i c a l and the i r r e l v e n t . Most of the arguments are based on o p i n i o n r a t h e r than e v i d e n c e . Only the l a s t four t r y to e s t a b l i s h a r e l a t i o n s h i p between MCE and competence, which i s the r a i s o n d ' e t r e f o r l i c e n s u r e . Arguments a g a i n s t MCE which appear i n the l i t e r a t u r e are more numerous than those which support i t . They can be rough ly c l a s s i f i e d i n t o three broad groups of s ta tements : those which c l a i m that MCE has no e f f e c t on competence or i s so ve ry l i m i t e d i n i t s e f f e c t as to be u n j u s t -i f i a b l e ; those which c l a i m MCE cannot be e f f e c t i v e l y implemented, and those which p r e d i c t t h a t MCE w i l l have u n d e s i r a b l e e f f e c t s on h e a l t h c a r e : Arguments i n O p p o s i t i o n to MCE ( those q u e s t i o n i n g the e f f e c t of MCE on competence) 1 . There i s l i t t l e ev idence that c o n t i n u i n g educat ion makes a d i f f e r e n c e to competence under i d e a l l e t a lone mandatory c o n d i t i o n s ( 9 2 , 1 1 0 , 1 6 4 , 2 8 2 , 3 0 2 , 4 8 6 ) and a b s o l u t e l y no ev idence a t a l l that i t w i l l guarantee competence ( 4 6 3 , 5 3 7 ) . 2 . Educat ion i s on ly one v a r i a b l e a f f e c t i n g competence and there are more powerfu l v a r i a b l e s . Cont inu ing educat ion cannot en fo rce e t h i c a l behav ior nor p r o t e c t a g a i n s t inadequate performance not r e l a t e d to knowledge ( 1 1 2 ) . 3 . Whi le cont inued l e a r n i n g i s necessary to competence, tha t l e a r n i n g may occur i n a number of ways of which random educat ion i s on ly one (423); e v a l u a t i o n , feedback and s e l e c t i v e c o r r e c t i v e a c t i o n are more e f f e c t i v e f a c t o r s which i n f l u e n c e competence ( 7 1 , 2 4 7 , 4 2 3 , 4 8 6 , 5 1 1 ) . . cont inued - 1 7 3 -4 . Whi le p a r t i c i p a t i o n i n an educat ion program can be l e g i s l a t e d , l e a r n i n g cannot . I t i s an i n t e r n a l p r o c e s s , the r e s u l t o f which can be measured on ly by t e s t i n g ( 3 7 , 1 1 0 , 2 5 9 , 3 0 2 , 3 8 5 ) . 5 . Teachers have had MCE s i n c e the e a r l y p a r t of the c e n t u r y , and there are many incompetents i n the teach ing p r o f e s s i o n ( 2 4 6 ) . (arguments c l a i m i n g tha t MCE cannot be e f f e c t i v e l y implemented) 6 . I t i s not c l e a r what c o n s t i t u t e s d e s i r a b l e c o n t i n u i n g educat ion ( 1 0 8 , 2 4 8 ) , e i t h e r i n terms of q u a n t i t y (108) or content ( 2 5 9 ) , i . e . re levance to p r a c t i c e . 7 . There are not s u f f i c i e n t e d u c a t i o n a l resources to p rov ide c o n t i n u i n g educat ion f o r a l l p r o f e s s i o n a l s ( 1 1 0 , 2 4 8 , 3 8 5 , 4 2 3 ) . 8 . Even i f demands f o r educat ion cou ld be met i n terms of q u a n t i t y , to p r o v i d e educat ion r e l e v a n t to each p r a c t i t i o n e r ' s needs would be i m p o s s i b l e ( 3 7 , 1 0 8 , 2 8 2 , 4 2 3 ) . 9 . To c r e a t e a system which prevents who lesa le c h e a t i n g would be a b u r e a u c r a t i c nightmare ( 1 6 8 ) . (arguments which p r e d i c t MCE w i l l have u n d e s i r a b l e e f f e c t s ) . 10 . MCE w i l l f o s t e r the growth of poor ("mass educat ion" type or substandard l o c a l ) programs produced to meet heavy demand ( 1 8 2 , 3 3 8 , 3 8 5 , 4 2 3 ) ; t h i s w i l l be a s s o c i a t e d w i t h the r a p i d development of vested i n t e r e s t groups who w i l l p l a c e e x t r a demands on resources ( 2 8 2 , 3 8 5 , 4 6 3 ) . 1 1 . I nnovat i ve and p o t e n t i a l l y e f f e c t i v e n o n - t r a d i t i o n a l programs w i l l be d iscouraged ( 7 1 , 2 8 2 , 3 0 2 , 3 8 5 ) . 12. More r a t i o n a l approaches to maintenance of competence w i l l not be developed because of f a l s e complacency about what MCE can a c h i e v e ; r i t u a l a t tendance , " a l i t t l e l i k e p a i n t i n g the o p e r a t i v e s i t e w i t h w a t e r c o l o r s " (37) w i l l l end f a l s e s t a t u s and c r e d i b i l i t y w h i l e d i v e r t i n g a t t e n t i o n from the r e a l i s s u e ( 7 1 , 9 2 , 2 4 8 , 4 2 3 , 4 6 3 , 4 9 6 ) . 1 3 . P r a c t i t i o n e r s may be fo rced to p a r t i c i p a t e i n i r r e l e v a n t e d u c a t i o n ; r i g i d requirements w i l l d i scourage p r o f e s s i o n a l s from be ing i n d i v i d u a l l y r e s p o n s i b l e f o r t h e i r l e a r n i n g and performance ( 1 1 0 , 1 6 4 , 3 0 2 ) . . . . cont inued - 1 7 4 -14. MCE w i l l i n c r e a s e the parano ia of the m a j o r i t y of p r o f e s s i o n a l s who are competent ( 2 8 3 ) . 1 5 . MCE w i l l f a i l , and i n the p r o c e s s , d i s c r e d i t c o n t i n u i n g educat ion f o r the c o n t r i b u t i o n i t can make ( 2 8 2 , 4 2 3 ) . 16 . MCE w i l l u n j u s t i f i a b l y r e s u l t i n the s i g n i f i c a n t l o s s of med ica l manpower ( 1 8 2 ) . 17 . Educat ion i n s t i t u t i o n s may be h e l d l i a b l e when t h e i r l e a r n e r s do not perform and insurance premiums w i l l r i s e ( 3 8 5 ) . 18 . The bookkeeping generated by MCE ( p o i n t s , p e r s o n n e l , a c c r e d i t i n g systems) w i l l consume too many resources ( 3 7 , 3 8 5 ) . Table V I I I As w i t h the arguments f o r MCE, the above statements a l s o are p r i m a r i l y o p i n i o n and range from the c l e a r l y h y s t e r i c a l to the p l a u s i b l e . However, g i ven what i s known about l e a r n i n g , e d u c a t i o n , f a c t o r s i n f l u e n c i n g compe-t e n c e , and sheer numbers of p r o f e s s i o n a l s i n v o l v e d , the two se ts of arguments taken together favor the c o n c l u s i o n that u n i v e r s a l MCE cannot assure the competence of h e a l t h p r o f e s s i o n a l s and may not even be the best cho ice of methods to promote competence. In the vas t m a j o r i t y of cases , MCE w i l l a f f e c t p r a c t i t i o n e r s a l r e a d y meeting minimum standards and moving toward optimum ones; t h i s i s not c o n s i s t e n t w i t h the purpose of l i c e n s i n g . "Cont inu ing educat ion we should have but making c o n t i n u -i n g educat ion mandatory f o r l i c e n s u r e wi thout assurance of i t s v a l i d i t y f o r cont inued competence may be l i k e f o l l o w i n g A l i c e i n t o Wonderland where t h i n g s are not what they seem" ( 4 6 3 ) . Before abandoning the n o t i o n of MCE as a l t o g e t h e r i n a p p r o p r i a t e , i t i s important to c o n s i d e r tha t MCE has a p p l i c a t i o n s other than that of a u n i v e r s a l requirement f o r l i c e n -s u r e . - 1 7 5 -S e l e c t i v e Mandatory Cont inu ing E d u c a t i o n . I t appears tha t on some o c c a s i o n s , MCE can and i s be ing used to some e f f e c t f o r s p e c i a l s i t u a t i o n s w i t h i n the l i c e n s u r e sys tem. Sheppard (472) and Castonguay (90) both r e f e r to p r o v i s i o n w i t h i n the P r o f e s s i o n s Code of Quebec f o r a p r o f e s s i o n a l i n s p e c t i o n committee, w i t h i n each pro -f e s s i o n a l c o r p o r a t i o n , w i t h power to i n v e s t i g a t e performance and to p r e -s c r i b e such educat ion as i t sees f i t . Seve ra l of the Canadian p rov inces r e q u i r e r e f r e s h e r courses f o r nurses who have been i n a c t i v e , as does Oregon ( 4 8 5 , 5 3 7 ) . New Hampshire and Washington both have s p e c i a l l i c e n s e c a t e g o r i e s f o r nurses w i t h advanced t r a i n i n g , and r e t e n t i o n of these l i c e n s e s i s dependent, among other t h i n g s , on evidence of p a r t i c i p a t i o n i n r e l e v e n t c o n t i n u i n g educat ion (53 7 , 5 3 8 ) . Suther land has suggested that v e t e r i n a r i a n s be l i c e n s e d i n s p e c i a l t y areas o n l y , and be r e q u i r e d to take a t r a n s i t i o n course before moving to a new s p e c i a l t y a rea ( 4 9 1 ) ; Bradshaw has made a s i m i l a r recommendation f o r n u r s i n g (5 8 ) . The use of mandatory c o n t i n u i n g educat ion f o r non-compulsory c r e d e n t i a l s i s w idespread . The AMA's P h y s i c i a n s R e c o g n i t i o n Award (PRA) a l r e a d y r e f e r r e d to i s the prototype fo r almost a l l o thers ( 3 2 6 , 3 3 3 ) . The PRA r e q u i r e s one hundred f i f t y hours of p a r t i c i p a t i o n i n c o n t i n u i n g educat ion over a pe r iod of three y e a r s . Of these h o u r s , s i x t y must be "approved" c o n t i n u i n g e d u c a t i o n , w h i l e the other hours may be from a v a r i e t y of c a t e g o r i e s i n c l u d i n g non-approved c o n t i n u i n g educat ion p r o -grams, m e d i c a l t e a c h i n g , p a p e r s , books , e x h i b i t s , independent study and "other m e r i t o r i o u s l e a r n i n g e x p e r i e n c e s . " While the PRA i t s e l f i s v o l u n -t a r y , o r g a n i z a t i o n s such as county med ica l s o c i e t i e s use the PRA mechan-ism as a requirement f o r cont inued membership (309) ; s t a t e s r e q u i r i n g c o n t i n u i n g educat ion f o r r e l i c e n s u r e accept the PRA as proof of educa-- 1 7 6 -t i o n a l p a r t i c i p a t i o n (3 3 3 ) . The u l t i m a t e r e s u l t i s tha t the PRA i s o f t e n l e s s than v o l u n t a r y . S i m i l a r v o l u n t a r y r e c o g n i t i o n systems are operated by both med ica l (268) and n u r s i n g (18,485) o r g a n i z a t i o n s . Other p r o f e s s i o n a l o r g a n i z a t i o n s make c o n t i n u i n g educat ion a c o n d i t i o n of r e c e r t i f i c a t i o n . The most notab le examples are some of the med ica l s p e c i a l t y groups i n the Un i ted States (4 06) and the ANA n u r s i n g s p e c i a l t y c e r t i f i c a t i o n programs ( 9 1 ) . T e c h n i c a l l y s p e a k i n g , c o n t i n u i n g educat ion i n t h i s s i t u a t i o n i s v o l u n t a r y i n the sense that p r a c t i t i o n e r s do not r e q u i r e these creden-t i a l s f o r p r a c t i c e . The extent to which c e r t i f i c a t i o n s and other awards are r e q u i r e d f o r many p o s i t i o n s , f o r r e c e i p t of r e s e a r c h g r a n t s , f o r i n s t i t u t i o n a l a c c r e d i t i o n e t c . i s c o n s i d e r a b l e however. In t h i s c o n t e x t , s o - c a l l e d v o l u n t a r y c o n t i n u i n g educat ion becomes de f a c t o mandatory con-t i n u i n g e d u c a t i o n . Whether mandatory c o n t i n u i n g educat ion i s j u s t i f i a b l e on a s e l e c t i v e b a s i s i s a debatable p o i n t . One might argue t h a t where d i r e c t performance a p p r a i s a l i s a p r a c t i c a l i m p o s s i b i l i t y , and the e d u c a t i o n i s r e l e v e n t to p r a c t i c e and has b u i l t - i n t e s t i n g p rocedures , MCE i s a j u s t i -f i a b l e a l t e r n a t i v e . On the other hand, once deve loped , t e s t i n g proce-dures cou ld be used wi thout re fe rence to the process by which knowledge and s k i l l s were a c q u i r e d . I t might be argued: MCE f o r r e c e r t i f i c a t i o n i n the absence of u n i v e r s a l MCE f o r r e l i c e n s u r e c o u l d c o n c i e v a b l y r e s u l t i n a s i t u a t i o n d e s c r i b e d by Knox as a h i g h l y favourab le one. He would con-c e n t r a t e a l l a v a i l a b l e educat ion resources on s e l e c t e d t a r g e t g roups , t a k i n g advantage of the d i f f u s i o n process and what i s known about i n f l u e n c e of o p i n i o n l e a d e r s (282) . -177-A c c r e d i t a t i o n of Continuing Education Programs Along with the various systems for recording professionals' p a r t i c i p a t i o n i n continuing education, there have developed p a r a l l e l systems for reviewing and approving continuing education programs. There i s l i t t l e discussion i n the l i t e r a t u r e as to why t h i s has happened. Some reasons given include a desire to protect the professional from inade-quate programs (410,447), a b e l i e f that education must be of high q u a l i t y i f i t i s to have a pos i t i v e e f f e c t on the q u a l i t y of care (17,410), and an expectation that a review process can improve the q u a l i t y of programs through formative evaluation (410,447). Also, t r a d i t i o n cannot be d i s -counted; accreditation/approval of pre-professional programs has a long h i s t o r y . Brown (71) distinguishes between a c c r e d i t a t i o n of i n s t i t u t i o n s providing continuing education, which he refers to as fourth l e v e l evalu-ation, and evaluation of programs themselves, which he c a l l s t h i r d l e v e l evaluation. (Learner outcomes evaluation i s second l e v e l ; patient out-comes, f i r s t l e v e l ) . He believes that c o r r e l a t i o n with impact on care decreases i n moying from l e v e l one to l e v e l four. In the continuing education f i e l d the term "accreditation" has incr e a s i n g l y been applied to fourth l e v e l or i n s t i t u t i o n a l evaluation, while "approval" i s reserved for t h i r d l e v e l or program evaluation. This i s a departure from custom-ary usage. As program review systems attempt to cope with the increasing volume of continuing education programs, they tend to develop i n s t i t u -t i o n a l a c c r e d i t a t i o n procedures (17,326,333). The AMA's extensive continuing education review system, devel-oped to support the Physicians's Recognition Award, has many of the elements of l a t e r systems. At the program evaluation l e v e l , an approved -178-program is " one having sufficient scope and depth of coverage to form an educational unit which i s planned, coordinated, administered, and evaluated in terms of a specific educational objective, such as a defined level of knowledge or a specific performance s k i l l , to be attained by a physician participating in the program" (3 26). At the institutional accreditation level, administrative educational methods and evaluation practices are scrutinized (I 14). At the end of 1976, over eight hundred institutions were AMA accredited; this included medical schools, commun-it y hospitals, specialty societies, c l i n i c s and federal government faci-l i t i e s . Cost to the institution was $500 for i n i t i a l survey and $250 for resurvey (3 33) . The nursing profession in the United States has developed a similar two level continuing education review system wherein the American Nurses Association approves programs directly in some instances, but more often accredits other organizations such as state nurses associations, specialty organizations, federal nursing services and universities. These organizations are then authorized to offer programs said to be ANA approved. With the exception of universities, the accredited organiza-tion may also review and approve offerings of other sponsors (4 10). Institutional accreditation c r i t e r i a are similar to those used in the AMA's PRA; program approval c r i t e r i a are detailed and would require well developed program plans. In Canada, at least three provincial nursing organizations operate continuing education evaluation systems at the program review level (410,411,453). Al l use detailed program planning c r i t e r i a and two of these (410,411) ask reviewers to make a terminal judgement as to whether nursing care w i l l be improved as a result of the program. -179-D e n t i s t r y (300), pharmacy (164) and m e d i c a l technology (77) are other p r o f e s s i o n s which have i n s t i t u t e d c o n t i n u i n g educat ion approva l p rocedures . The program developed by the American S o c i e t y of M e d i c a l Techno log is ts (77) i s no tab le fo r i t s emphasis on h e l p i n g program spon- . ' s o r s : " I t i s not the p o l i c y o f the committee to re fuse a c c r e d i t a t i o n of programs d e f i c i e n t i n any of ( these) a r e a s , but r a t h e r to o f f e r c o n -s t r u c t i v e suggest ions so that sponsors may upgrade the program o f f e r i n g to enable i t to meet (our) s tandards of q u a l i t y and e f f e c t i v e n e s s . " Formative program e v a l u a t i o n i s a l s o h e a v i l y emphasized i n the rev iew system sponsored by the Reg is te red Nurses and Reg is tered P s y c h i a t r i c Nurses' A s s o c i a t i o n s i n B r i t i s h Columbia (4 10). L i k e a l l systems of c r e d i t , c o n t i n u i n g educat ion a p p r o v a l systems r e q u i r e a cur rency or u n i t o f v a l u e . Whi le the l e a r n e r - educa-t i o n a l event con tac t hour i s commonly u s e d , the c o n t i n u i n g educat ion u n i t i s employed a l s o . Developed o u t s i d e the h e a l t h system to meet demands f o r a common language to communicate the v a l u e of a c o n t i n u i n g educat ion e v e n t , the C . E . U . i s d e f i n e d as " t e n hours of p a r t i c i p a t i o n i n an organ-i z e d c o n t i n u i n g educat ion exper ience under r e s p o n s i b l e s p o n s o r s h i p , capable d i r e c t i o n and q u a l i f i e d i n s t r u c t i o n " (,109, 209,266). A f u r t h e r e l a b o r a t i o n of j u s t what t h i s means i s prov ided by Bruno (73) , who s t a t e s that o r g a n i z a t i o n s awarding C . E . U . ' s should r e q u i r e tha t the event be o f f e r e d i n response to i d e n t i f i e d need ; that there be a statement of o b j e c t i v e s , purpose , and r a t i o n a l e ; t h a t content be o rgan ized and pre-sented i n a l o g i c a l sequence; tha t c l i e n t e l e be i n c l u d e d i n p l a n n i n g ; tha t a p p r o p r i a t e e v a l u a t i o n procedures be u s e d , e t c . These c r i t e r i a are qu i te s i m i l a r to those used i n program l e v e l rev iew i n most of the c o n t i n u i n g educat ion approva l systems i n the h e a l t h .care f i e l d . -180-I f there has been any attempt to eva luate the e f f e c t of program rev iew systems on q u a l i t y of care or even i n terms of r e l i a b i l i t y and improved e d u c a t i o n a l programs, t h i s d i d not emerge i n the l i t e r a t u r e searched . In format ion on the c o s t s of such rev iew programs i s not r e a d -i l y a v a i l a b l e e i t h e r . In v iew of f e a r s tha t such rev iew systems may i n h i b i t e f f e c t i v e programming, t h i s f a i l u r e to eva luate i s q u i t e s u r p r i s -i n g . Summary and Comment The c e n t r a l i s s u e i n m a i n t a i n i n g competence of h e a l t h p r o f e s -s i o n a l s i s d e f i n i t i o n and measurement of competence, fo l l owed by a c t i o n c a l c u l a t e d to r e s t o r e and/or m a i n t a i n i t . Surrounding t h i s i s s u e i s the North American s o c i a l c o n t e x t . Developing e v a l u a t i o n systems have p a r t i -c u l a r l y i n f l u e n c e d and been i n f l u e n c e d by p r o f e s s i o n a l s ' human needs f o r success and respec t and t h e i r c i v i l r i g h t s to due p r o c e s s ; consumers' r i g h t s to p r i v a c y and to p a r t i c i p a t i o n i n d e c i s i o n s about pe rsona l h e a l t h care and about p u b l i c h e a l t h p o l i c y ; growing s o p h i s t i c a t i o n on the par t of a l l groups about the a p p l i c a t i o n o f power through c o l l e c t i v e a c t i o n ; the r i s i n g c o s t s of h e a l t h care ; the growing volume of m a l p r a c t i c e l i t i -g a t i o n ; and the p r o l i f e r a t i o n of c a t e g o r i e s of h e a l t h care w o r k e r s . The above f a c t o r s have generated a p r o d i g i o u s amount of a c t i -v i t y i n the v a r i o u s h e a l t h manpower c r e d e n t i a l i n g mechanisms which attempt to make some statement about the p r o f e s s i o n a l ' s competence. The major c r e d e n t i a l i n g methods a p p l i c a b l e to i n d i v i d u a l s are l i c e n s u r e ( t o assure a minimum standard of competence w i t h a v iew to p r o t e c t i n g the p u b l i c from harm) and c e r t i f i c a t i o n ( u s u a l l y used to i n d i c a t e achievement -181-of a l e v e l of excellence within a specialty) . C e r t i f i c a t i o n of program completion by educational a u t h o r i t i e s i s a supplementary c r e d e n t i a l i n g mechanism. The more prominent c r e d e n t i a l i n g trends include action to bring order to a j u r i s d i c t i o n ' s l i c e n s i n g p o l i c i e s and procedures; to extend control of l i c e n s i n g administration beyond the profession concern-ed; to use a v a r i e t y of evaluation techniques p r i o r to awarding or renew-ing a credential;, and to l i m i t the time for which a c r e d e n t i a l i s v a l i d , imposing conditions for renewal which are thought to r e l a t e to compe -tence. Mandatory continuing education (MCE), which i s one of the more common conditions for r e l i c e n s u r e , has been p a r t i c u l a r l y contentious. Most arguments for or against MCE have been based on opinion rather than research or experience, however the weight of argument at t h i s time indicates that the costs of universal mandatory continuing education for relicensure w i l l outweigh i t s effectiveness i n assuring a minimum l e v e l of competence. More research on the impact of continuing education, on the influence of voluntary vs mandatory systems, and on the e f f e c t of approval mechanisms i s needed. In reviewing the s o c i a l context within which health profession-a l s ' competence must be maintained, i t becomes apparent that the tech-n i c a l implementation of evaluation and action programs i s heavily i n f l u -enced by surrounding issues. A developing program must recognize and work with such issues i f i t i s to be maximally e f f e c t i v e . F i n a l l y , i t should be r e i t e r a t e d that the l i t e r a t u r e reviewed i n t h i s chapter represents a very cursory overview of material tangen-t i a l l y r elated to maintaining health professionals' competence. Most of the subjects discussed could well be the subject of extensive l i t e r a t u r e survey alone. They are presented here to provide necessary perspective. - 1 8 2 -Chapter V I I I OBSERVATIONS AND RECOMMENDATIONS T h i s c o n c l u d i n g chapter w i l l not summarize the f i n d i n g s r e p o r t -ed i n the t h e s i s . The reader i s r e f e r r e d to the f i n a l "Comment and Summary" s e c t i o n s of Chapters I I - V I I f o r that i n f o r m a t i o n . Chapter V I I I w i l l f i r s t o f f e r some genera l comments and then d i s c u s s the i n f l u -ence of the North American p o l i t i c a l r e a l i t y , the s t a t e - o f - t h e - a r t of e v a l u a t i o n - a c t i o n s t r a t e g i e s , and the c o n t r i b u t i o n s that c o n t i n u i n g educators can most e f f e c t i v e l y make to the e v a l u a t i o n and mainta inance of h e a l t h p r o f e s s i o n a l s ' competence. F i r s t , a b r i e f comment should be made about the b a s i c assump-t i o n s that p u b l i s h e d l i t e r a t u r e d e s c r i b e s recent developments and c u r r e n t p r a c t i c e s i n the f i e l d , and that the four h e a l t h p r o f e s s i o n s s t u d i e d are the r i c h e s t i n f o r m a t i o n s o u r c e s . With respect to the f i r s t assumpt ion , i t became obvious d u r i n g t h i s study that the l i t e r a t u r e i n t h i s r a p i d l y deve lop ing area has lagged behind cu r ren t developments to a s i g n i f i c a n t e x t e n t . Th is i s r e f l e c t i v e of Garvey 's f i n d i n g s that t ime l a p s e from work i n i t i a t e d to j o u r n a l p u b l i c a t i o n i s t h i r t y to t h i r t y - s e v e n months ( 1 8 1 ) . Those w i s h i n g to know about c u r r e n t work i n the f i e l d of e v a l u a -t i n g and m a i n t a i n i n g competence of h e a l t h p r o f e s s i o n a l s should make e x t e n s i v e use of p e r s o n a l c o n t a c t s by telephone and a t c o n f e r e n c e s . They should a l s o request and be prepared to pay f o r t e c h n i c a l r e p o r t s and p r e - p u b l i c a t i o n d r a f t s o f f i n a l r e p o r t s of work underway. Whi le the second assumption proved to be an a c c e p t a b l e one, i n r e t r o s p e c t , the b u l k -183-of u s e f u l i n f o r m a t i o n was found i n m e d i c a l and n u r s i n g l i t e r a t u r e . Developments i n these two d i s c i p l i n e s should be f o l l o w e d c a r e f u l l y . One f i n a l g e n e r a l o b s e r v a t i o n : the language used i n the l i t e r -a ture i s con fus ing and u n h e l p f u l , perhaps because the f i e l d i s s t i l l i n i t s f o r m a t i v e s t a g e s . To date there i s no r e a l c o n s i s t e n c y or i d e n t i f i -ab le p a t t e r n to the te rmino logy employed by authors even w i t h i n the same p r o f e s s i o n . P a r t i c u l a r l y v u l n e r a b l e to both conceptua l and semantic con fus ion are the f o l l o w i n g se ts of terms: s t r u c t u r e , p r o c e s s , outcome; v a r i a b l e , i n d i c a t o r , c r i t e r i o n , s t a n d a r d , index ; measurement, e v a l u a t i o n ; minimum, optimum ( s t a n d a r d s ) ; l i c e n s u r e , r e g i s t r a t i o n . U n t i l t h i s d i f f i -c u l t y i s r e s o l v e d , i t w i l l remain important to be a l e r t to the concepts being d i s c u s s e d i n any g i ven r e p o r t , whether or not d e f i n i t i o n s have been p r e s e n t e d . S i m i l a r l y , w r i t e r s should be met i cu lous i n both d e f i n i t i o n and use of te rms . The competence of p r o f e s s i o n a l s i s fundamenta l l y a s o c i a l i s s u e . This demands that the goa l be p r o t e c t i o n of the p u b l i c and the means to that goa l be a c c e p t a b l e to g e n e r a l s o c i e t y on p s y c h o l o g i c a l , r s o c i a l , p o l i t i c a l and economic grounds. Recent h i s t o r y i n d i c a t e s tha t an i n c r e a s i n g l y s o p h i s t i c a t e d p u b l i c i s a c c e p t i n g r e s p o n s i b l i t y f o r p e r s o n a l h e a l t h s t a t e and p u b l i c h e a l t h p o l i c y . The g e n e r a l s o c i a l c l i m a t e emphasizes c i v i l r i g h t s of both p a t i e n t s and p r a c i t i o n e r s , w h i l e p o l i t i -c a l and economic fo rces demand cos t c o n t r o l and a c c o u n t a b i l i t y . Jago (257) reminds readers that s o c i e t y now s e r i o u s l y ques t ions the p r o f e s s -i o n a l dominance model i n h e a l t h c a r e . These i n f l u e n c i n g f a c t o r s have converged so that t r a d i t i o n a l s t r u c t u r e s such as t o r t law and c r e d e n t i a l i n g mechanisms have been a p p l i -ed w i t h i n c r e a s e d v i g o r ; i n many i n s t a n c e s they have been r e v i s e d to - 1 8 4 -enhance e f f e c t i v e n e s s . Th is has r e s u l t e d i n s a l u t o r y changes, such as the more e f f i c i e n t use of d i s c i p l i n a r y boards ; i n changes where the e f f e c t i s as yet unknown, such as the a d d i t i o n of consumer members to r e g u l a t o r y b o d i e s ; and i n changes where d e s i r e d impact i s h i g h l y ques -t i o n a b l e and indeed may be c o u n t e r p r o d u c t i v e , as i s the case w i t h u n i v e r -s a l mandatory c o n t i n u i n g educat ion l a w s . Concurrent w i t h t r a d i t i o n a l mechanisms being s t reng thened , more i n n o v a t i v e approaches, p r i m a r i l y p r a c t i c e - b a s e d e v a l u a t i o n - a c t i o n p r o -grams, are be ing deve loped . Whi le these h o l d c o n s i d e r a b l e promise f o r the e f f e c t i v e mainta inance of competence and are t h e r e f o r e of c o n s i d e r -ab le i n t e r e s t to c o n t i n