UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Selecting research and development projects Hartley, Karen Marie 1974

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
831-UBC_1974_A4_6 H37.pdf [ 4.97MB ]
Metadata
JSON: 831-1.0093051.json
JSON-LD: 831-1.0093051-ld.json
RDF/XML (Pretty): 831-1.0093051-rdf.xml
RDF/JSON: 831-1.0093051-rdf.json
Turtle: 831-1.0093051-turtle.txt
N-Triples: 831-1.0093051-rdf-ntriples.txt
Original Record: 831-1.0093051-source.json
Full Text
831-1.0093051-fulltext.txt
Citation
831-1.0093051.ris

Full Text

SELECTING RESEARCH AND DEVELOPMENT PROJECTS by KAREN MARIE HARTLEY B . S c . , M c G i l l Un i v e r s i t y , 1971 A THESIS SUBMITTED IN PARTIAL FULFILMENT THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE . i n the F a c u l t y of COMMERCE AND BUSINESS ADMINISTRATION We a c c e p t t h i s t h e s i s as c o n f o r m i n g to th requ i red s tanda rd THE UNIVERSITY OF BRITISH COLUMBIA A p r i l \31k In presenting th i s thesis i n p a r t i a l fu l f i lment of the requirements for an advanced degree at the University of B r i t i s h Columbia, I agree that the Library s h a l l make i t f ree ly avai lab le for reference and study. I further agree that permission for extensive copying of t h i s thesis for scholar ly purposes may be granted by the Head of my Department or by h i s representatives. I t i s understood that copying or pub l icat ion of t h i s thesis for f i n a n c i a l gain s h a l l not be allowed without my wr i t ten permission. Department of Management Science  Faculty of Commerce and Business AAdministrati on The Universi ty of B r i t i s h Columbia Vancouver 8, Canada Date May 10, 1974 ABSTRACT P r e v i o u s l y proposed a lgor i thms f o r s e l e c t i n g research and development p r o j e c t s are rev iewed. A new method fo r a n a l y s i s of p r o j e c t s e l e c t i o n d e c i s i o n s i s deve loped. This method inc ludes such fea tu res as m u l t i p l e c r i t e r i a , funding ranges , and m u l t i p l e s o l u t i o n s . Methods of handl ing m u l t i p l e c r i t e r i a are d i s c u s s e d . An i n t e r a c t i v e p r o j e c t s e l e c t i o n a l g o r i t h m i s developed and implemented. Three c h a r a c t e r i s t i c s of the a lgo r i thm improve i t s use fu lness and a c c e p t a b i l i t y to d e c i s i o n makers; 1) the a l g o r i t h m i s based on a r e l a t i v e l y r e a l i s t i c model of the d e c i s i o n s i t u a t i o n s , 2) i t r e q u i r e s d i r e c t p a r t -i c i p a t i o n on the part of the d e c i s i o n maker and a l lows some user c o n t r o l of e v a l u a t i o n c r i t e r i a , and 3) i t ' s data requirements are modest. TABLE OF CONTENTS Page ABSTRACT i i LIST OF TABLES . v LIST OF FIGURES v i ACKNOWLEDGEMENT. . . v i i Chapter 1 A SURVEY OF PROJECT SELECTION ALGORITHMS 1 1.1 Scor ing Models 4 1 .2 L i near Model s 6 1.3 N o n - l i n e a r Models 7 1.4 Zero-one Models . . 8 1.5 U t i l i t y Models . . . . . . . . 9 1.6 P r o f i t a b i l i t y Index Models . . . . . . . . . 10 1.7 R isk A n a l y s i s . • 11 1.8 Dec i s i on Trees . 11 1.9 L i m i t a t i o n s of the Models . . . 12 2 RATIONAL DECISION MAKING WITH MULTIPLE CRITERIA . 1 6 2.1 Sequent ia l E l i m i n a t i o n Methods . . . . . . . 19 2.2 S p a t i a l P r o x i m i t y Methods'. 23 2.3 Mathematical Programming Methods 27 i i i Chapter Page 2.4 U t i l i t y Funct ion Methods 29 2 . 4 . 1 D i r e c t l y Assessed Preference Techniques. 29 2 . 4 . 2 I n f e r r e d Pre ference Techniques . . . . 33 3 A NEW PROJECT SELECTION ALGORITHM . 47 3.1 Der i v ing a L i n e a r U t i l i t y Funct ion 47 3.2 D e r i v i n g a P iecewise L inear U t i l i t y F u n c t i o n . 51 3 .3 Formulat ing the O b j e c t i v e Funct ion 59 3.4 Formulat ing the C o n s t r a i n t s . 63 3 .5 The Formulat ion with a L inear U t i l i t y Funct ion 66 3.6 The Formulat ion wi th a P iecewise L inear U t i l i t y Funct ion 67 4 THE UTILITY AND ACCEPTABILITY OF THE ALGORITHM 72 4.1 Features of the A l g o r i t h m 72 4 .2 Data Requirements. . . . . . . 74 4 . 3 The Usefulness of the A lgo r i thm 75 4.4 P o s s i b l e Extens ions and Improvements to the A lgo r i thm 78 BIBLIOGRAPHY 84 APPENDIX A SAMPLE PROBLEM . . . . . . . . . 92 i v LIST OF TABLES Table Page I Methods of Handl ing M u l t i p l e C r i t e r i a . . 20 II Sample Problem Data 94 V LIST OF FIGURES Figure Page 2.1 Obta in ing an I n d i f f e r e n c e Curve by the MacCrimmon-Toda Method 25 2.2 Con junc t i ve U t i l i t y Funct ion 39 2.3 D i s j u n c t i v e U t i l i t y Funct ion . 41 3.1 Flow Chart of the A l g o r i t h m 49 3.2 P iecewise L inear U i l i t y Funct ions . . . . . . . . 55 3.3 I l l u s t r a t i o n s of Inc reas ing and Decreasing Returns 59 3.4 P r o b a b i l i t y of Success Funct ions 61 v i ACKNOWLEDGEMENT The author wishes to acknowledge the a s s i s t a n c e of Dr. I l an V e r t i n s k y , who as t h e s i s s u p e r v i s o r , prov ided much encouragement and many h e l p f u l suggest ions in both the deve lop -ment and w r i t i n g s tages . Dr. Cary Swoveland and Dr. J e f f S idney , as members of the committee, made many use fu l suggest ions as w e l l . Thanks i s a l so due Abe Landsberg who programmed the a l g o r i t h m . v i i Chapter 1 A SURVEY OF PROJECT SELECTION ALGORITHMS The problem of s e l e c t i n g research and development p ro -j e c t s i s an important and d i f f i c u l t one. The f u t u r e compet i -t i v e p o s i t i o n of many companies depends on the e f f e c t i v e n e s s of t h e i r research and development program. The d e c i s i o n s that are made now w i l l determine whether the company w i l l remain i n b u s i n e s s , and i f s o , what business i t w i l l be i n . Research and development programs are e q u a l l y important to governments in t h e i r attempt to c o n t r o l , d i r e c t , and s t i m u l a t e change. S ince many b i l l i o n s of d o l l a r s are spent each year on research and development there i s j u s t i f i a b l e concern t h a t t h i s money should be inves ted w i s e l y . Much of the d i f f i c u l t y in s e l e c t i n g research and deve lop -ment p r o j e c t s a r i s e s from the u n c e r t a i n t i e s surrounding the s e l e c t i o n d e c i s i o n . The outcome of each p r o j e c t (success or f a i l u r e ) ^ may be uncer ta in due to t e c h n o l o g i c a l c o n s i d e r a t i o n s , I t i s o f t e n d i f f i c u l t to c l a s s i f y the outcome of a p r o j e c t as s u c c e s s o r f a i l u r e . However , f o r p l a n n i n g p u r p o s e s a p r o j e c t may be c o n s i d e r e d s u c c e s s f u l i f j t a c h i e v e d the g o a l i t s e t out t o , and u n s u c c e s s f u l o t h e r w i s e . 1 2 and the value of each p r o j e c t , even i f s u c c e s s f u l , i s u n c e r t a i n due to unforeseeable f u t u r e needs and c o n d i t i o n s . S ince more p r o j e c t s are a v a i l a b l e than can be undertaken, due to budget and other resource c o n s t r a i n t s , the o r g a n i z a t i o n must s e l e c t the ones which they cons ider to be of most v a l u e . However, the goals toward which the o r g a n i z a t i o n should be s t r i v i n g are o f ten u n c l e a r , and the c o n t r i b u t i o n of research and development to the achievement of these goals even more vague. This makes the s e l e c t i o n problem d i f f i c u l t . "If b a s i c data and c l e a r - c u t measures of the e f f e c t i v e n e s s of a p p l i e d research were a v a i l a b l e there would appear to be l i t t l e d i f f i c u l t y e s t a b l i s h i n g an o b j e c t i v e b a s i s f o r s e l e c t i n g projects" [ 6 4 ] . Any attempt to e s t a b l i s h a s e l e c t i o n procedure i s based, i m p l i c i t l y or e x p l i c i t l y , on a model of the d e c i s i o n p r o c e s s . I d e a l l y t h i s model would i n c l u d e a l l the s i g n i f i c a n t fea tu res of the problem.- However, s ince any model i s only an a b s t r a c t i o n of the r e a l - w o r l d s i t u a t i o n , i t n e c e s s a r i l y i n c o r -porates many assumptions and s i m p l i f i c a t i o n s . This i s e s p e c i a l l y t rue when d e a l i n g w i th a l a r g e number of u n c e r t a i n t i e s . However, d e s p i t e the approximations inherent in the model and the uncer -t a i n t y of the d a t a , the s e l e c t i o n procedure may s t i l l be v a l u a b l e f o r " . . . d e c i s i o n s have to be made, u s u a l l y on inadequate d a t a , 2 Most p r o j e c t s e l e c t i o n a l g o r i t h m s assume t h e r e i s a l a r g e s e t of a v a i l a b l e p r o j e c t s f rom wh ich to c h o o s e . Whitman and Landau Q82H and Gee C3^H d i s p u t e t h i s a s s u m p t i o n , e s p e c i a l l y in the c h e m i c a l i n d u s t r y , but in g e n e r a l i t seems to be a r e a -s o n a b l e one . 3 and they w i l l be made i n t u i t i v e l y on the d a t a , whatever i t s q u a l i t y . Anything that can be done to q u a n t i f y the bases f o r these d e c i s i o n s and to demonstrate the l o g i c a l consequences of the assumptions i s a step i n the r i g h t d i r e c t i o n " [ 7 ] . Much of the value r e s u l t i n g from the development of a s e l e c t i o n procedure would come from the model-developmerit phase i t s e l f . The development of a model would fo rce the d e c i s i o n maker to s p e c i f y , in d e t a i l , h is goals and any assumptions or i m p l i c i t c o n s t r a i n t s he imposes. Any i r r a t i o n a l i t i e s would then be c l e a r l y i l l u s t r a t e d . An a n a l y s i s of cu r rent p r a c t i c e s would help to e l i m i n a t e i n c o n s i s t e n c i e s between the d e c i s i o n maker's personal o b j e c t i v e s and the o v e r a l l o r g a n i z a t i o n a l o b j e c t i v e s . Current " r u l e s of thumb" cou ld .be tes ted f o r t h e i r r a t i o n a l i t y and e f f e c t i v e n e s s . Long range p lann ing and d i s c u s s i o n of goals would be s t i m u l a t e d . Thus, an a n a l y s i s of the d e c i s i o n problem would r e s u l t in a b e t t e r understanding of the many u n c e r t a i n t i e s i n v o l v e d and t h e i r e f f e c t on the r e s u l t s of the d e c i s i o n p r o c e s s . The s e l e c t i o n procedure that would r e s u l t from the development of the model would h o p e f u l l y f a c i l i t a t e a more c o n s i s t e n t t reatment of d e c i s i o n problems. Many models and s e l e c t i o n procedures have been sug -gested in the l i t e r a t u r e . They may be broadly d i v i d e d i n t o q u a l -i t a t i v e and q u a n t i t a t i v e approaches. The q u a l i t a t i v e methods g e n e r a l l y c o n s i s t of a check l i s t of d e s i r a b l e p r o p e r t i e s . P r o j e c t s are ra ted wi th respect to each c h e c k l i s t c r i t e r i o n as 4 s imply " f a v o r a b l e " or "un favo rab le " or on a numeric s c a l e . To be s e l e c t e d , a p r o j e c t ' s r a t i n g s must f o l l o w an " a c c e p t a b l e " p a t t e r n , or must meet c e r t a i n s p e c i f i e d minimum l e v e l s wi th respect to each c r i t e r i o n . Such methods are imprec ise f o r they depend h e a v i l y upon the a b i l i t y of the d e c i s i o n maker to ra te o b j e c t i v e l y and c o n s i s t e n t l y . Furthermore, they prov ide no i n d i c a t i o n of the r e l a t i v e value of the acceptab le p r o j e c t s and hence f u r n i s h l i t t l e i n f o r m a t i o n or guidance concerning the a p p r o p r i a t e funding l e v e l f o r each p r o j e c t . The s e l e c t i o n p ro -cedure may a l so be d i f f i c u l t to de f ine and app l y . What con -s t i t u t e s an " a c c e p t a b l e " pa t te rn of r a t i n g s ? What should the minimum acceptab le l e v e l of a c r i t e r i o n - be? Is i t reasonable to r e j e c t a p r o j e c t that does not meet the minimum l e v e l in one area but f a r exceeds i t i n another? C l e a r l y these quest ions are i m p o r t a n t . Answering them i s the f i r s t step in the development of a more p r e c i s e , and s o p h i s t i c a t e d d e c i s i o n method. The q u a n t i t a t i v e models that have been suggested in the l i t e r a t u r e may be c l a s s i f i e d i n t o e i g h t general c a t e g o r i e s : s c o r i n g models, l i n e a r , n o n - l i n e a r , z e r o - o n e , and u t i l i t y models , p r o f i t a b i l i t y i n d i c e s , r i s k a n a l y s i s and d e c i s i o n t r e e s . 1.1 SCORING MODELS The s c o r i n g models are the least s o p h i s t i c a t e d of the q u a n t i t a t i v e methods. They use the same type of r a t i n g s as the q u a l i t a t i v e methods to determine a numerical p r o j e c t s c o r e . 5 The Mott ley -Newton method [64] ra tes each p r o j e c t on a three point s c a l e wi th respect to f i v e c r i t e r i a . The r a t i n g s are then m u l t i p l i e d to produce the p r o j e c t s c o r e . Gargu i lo et at. [32] and Hertz and Car lson [42] suggest a method whereby each p r o j e c t i s rated as " f a v o r a b l e , " " u n f a v o r a b l e , " or "no o p i n i o n " wi th respect to the c r i t e r i a which are d i v i d e d in to three c l a s s e s ; economic, t e c h n i c a l , and commercial f a c t o r s . The number of each type of response in each c l a s s i s counted and a score f o r that c l a s s c a l c u l a t e d . The scores f o r the three c l a s s e s are m u l t i p l i e d together to form the o v e r a l l p r o j e c t s c o r e . The s c o r i n g models have many of the defec ts of the q u a l i t a t i v e methods. They depend h e a v i l y on the d e c i s i o n maker 's r a t i n g a b i l i t y , and prov ide only an o r d i n a l ranking of p r o j e c t s . S ince the r a t i n g s are on an a r b i t r a r y s c a l e , i t i s not p o s s i b l e to know how much b e t t e r one p r o j e c t i s than another . I t may not even be p o s s i b l e to know i f the best p r o j e c t i s "good . " (They may a l l be of l i t t l e value to the o r g a n i z a t i o n . ) S c o r i n g models do however have severa l d e s i r a b l e f e a t u r e s . S ince much r e l i a n c e must be p laced on the d e c i s i o n maker, very l i t t l e data on the p r o j e c t i s needed. Thus, the method i s most usefu l fo r d e c i s i o n s concerning pure research p r o j e c t s and p r o j e c t s i n t h e i r e a r l y stages of development where l i t t l e concrete i n f o r m a t i o n i s a v a i l a b l e on t h e i r cos ts and b e n e f i t s . The p r o j e c t scores can a l so be used to help diagnose a p r o j e c t ' s weak p o i n t s . They can i l l u s t r a t e those areas where the p r o j e c t 6 could be improved. Furthermore, s c o r i n g models can be cons t ruc ted to i n c l u d e non-economic. c r i t e r i a that are d i f f i c u l t to q u a n t i f y f o r use in more s o p h i s t i c a t e d models. 1.2 LINEAR MODELS L i n e a r models o f ten use a type of s c o r i n g system as w e l l . The methods suggested by Pound [68] and Dean and Nishry [21] use a weighted sum of p r o j e c t r a t i n g s as the o b j e c t i v e f u n c t i o n . Pound determines the a p p r o p r i a t e weights by i n t e r v i e w -ing the d e c i s i o n maker. Dean and Nishry suggest o b t a i n i n g them from s t a t i s t i c a l analyses of past d e c i s i o n s . Both methods prov ide only an o r d i n a l ranking of p r o j e c t s . Nutt [66] has developed a l i n e a r model f o r s e l e c t i n g m i l i t a r y p r o j e c t s . The " e f f e c t i v e n e s s " of each d i f f e r e n t p r o j e c t at s i x d i s c r e t e fund -ing l e v e l s i s c a l c u l a t e d by c o n s i d e r i n g var ious m i l i t a r y needs and g o a l s . A l i n e a r program i s so lved and the r e s u l t s i n d i c a t e the l e v e l at which each p r o j e c t should be funded, and the man-power that w i l l be r e q u i r e d . Asher [1] suggests maximizing expected p r o f i t i n an L . P . model which a l l o c a t e s a non-homogeneous work f o r c e to p r o j e c t s . The l i n e a r models g e n e r a l l y prov ide more i n f o r m a t i o n about funding l e v e l s and the r e l a t i v e values of the var ious p r o j e c t s . However they requ i re more data to do t h i s ; est imates of p r o f i t , or e f f e c t i v e n e s s , and p r o b a b i l i t i e s of success are r e q u i r e d . 7 •1.3 NON-LINEAR MODELS Many of the n o n - l i n e a r models view the s e l e c t i o n p ro -cedure as a s e q u e n t i a l problem. At the beginning of each p lann ing per iod o ld p r o j e c t s a r e . r e v i e w e d , and new p r o j e c t s are e v a l u a t e d . The research and development program f o r the next per iod i s s e l e c t e d from t h i s c o l l e c t i o n of o l d and new p r o j e c t s . This view of the problem leads n a t u r a l l y to a dynamic programming formu-l a t i o n . Hess [44] suggests a method fo r maximizing the expected d iscounted net p r o f i t . A d iscount f a c t o r must be s p e c i f i e d and est imates are requ i red of the t o t a l expected d iscounted gross p r o f i t a c c r u i n g from each p r o j e c t i f i t i s s u c c e s s f u l i n the n^*1 p e r i o d , as we l l as the p r o b a b i l i t y i t w i l l be s u c c e s s f u l i n t h the n p e r i o d , f o r a l l per iods i n the p lanning h o r i z o n . The : p r o b a b i l i t y of success i s assumed to be an exponent ia l f u n c t i o n of the cu r ren t funding l e v e l and past funding l e v e l s or cu r rent funding l e v e l a l o n e . The r e s u l t of the procedure i s an opt imal funding l e v e l f o r each p r o j e c t in each p lanning p e r i o d . Bobis et a l . [ 7 , 9] have suggested m o d i f i c a t i o n s to Hess 's method. They developed a d i s t r i b u t i o n of the cost of com-p l e t i n g a p r o j e c t (success or f a i l u r e ) by r e q u i r i n g es t imates of the l e a s t , most l i k e l y , and g r e a t e s t expected complet ion c o s t . The p r o b a b i l i t y of success in any year i s then the p r o b a b i l i t y of complet ion at the c u r r e n t expendi ture l e v e l m u l t i p l i e d by the p r o b a b i l i t y of t e c h n i c a l , l e g a l , eng ineer ing and 8 commercial s u c c e s s . The opt imal a l l o c a t i o n of. funds, i s determined from est imates of s a l e s , c o s t s , p r i c e s , time requ i red f o r commer-c i a l i z a t i o n and p r o b a b i l i t y of s u c c e s s . Since the method i s so dependent on the d a t a , they suggest r e p l a c i n g po in t es t imates by d i s t r i b u t i o n s and s i m u l a t i n g to ob ta in a more accurate v a l u e . Souder [73] and Rosen [69] have attempted to s i m p l i f y Hess 's method by a l l o w i n g each p r o j e c t to be funded at c e r t a i n d i s c r e t e va lues o n l y , and by assuming that the p r o b a b i l i t y of success i s a f u n c t i o n of the cur rent funding l e v e l a l o n e . They i n c o r p o r a t e the a d d i t i o n a l c o n s t r a i n t that there i s a "minimum and maximum amount that can be spent on each p r o j e c t over i t s research and development l i f e " [ 6 9 ] , These methods are based on a more r e a l i s t i c view of the d e c i s i o n p rocess . However t h e i r onerous data requirements make them d i f f i c u l t to use except on commercial p r o j e c t s i n an advanced s t a t e of development. The o b j e c t i v e f u n c t i o n of these models i s to maximize p r o f i t . No other p o s s i b l e goals are c o n s i d e r e d . Dean and Hauser [ 2 0 ] , however, have suggested dynamic programming methods f o r use in a m i l i t a r y context which opt imize severa l d i f f e r e n t c r i t e r i a . 1.4 ZERO-ONE MODELS The zero -one models such as those devised by Minkes and Samuels [ 6 1 ] , Freeman [ 3 1 ] , and Dean and Nishry [21] are a l l very s i m i l a r . They a l l propose maximiz ing an index of value 9 sub jec t to budget and resource c o n s t r a i n t s . Minkes and Samuels suggest the p o s s i b i l i t y of maximizing expected present value but impose an a d d i t i o n a l c o n s t r a i n t that the t o t a l r i s k i n v o l v e d in the program (weighted sum of the var iances of p r o j e c t re tu rn ) i s l e s s than a s p e c i f i e d amount. Freeman uses an index of value which i s not n e c e s s a r i l y p r o f i t o r i e n t e d and which must be developed by the o r g a n i z a t i o n concerned. He prov ides f o r three d i s c r e t e l e v e l s of funding to be c o n s i d e r e d . Dean and Nishry develop two models; one which maximizes the present value of f u t u r e p r o f i t s and another using a s c o r i n g - t y p e approach which maximizes some non-economic measure of v a l u e . The zero -one models are very s i m i l a r i n approach to the l i n e a r ones, and r e q u i r e about the same amount of d a t a . The ones which r e s t r i c t the p o s s i b l e funding l e v e l to one value are more a p p r o p r i a t e f o r p r o j e c t s wi th a f a i r l y we l l determined c o s t . 1.5 UTILITY MODELS The u t i l i t y models suggested by Cramer and Smith [16] and Green [40] are an attempt to e x p l i c i t l y handle r i s k c o n s i d e r a -t i o n s . They take in to account the f a c t that i t i s more important to minimize loss than to maximize g a i n . Cramer and Smith attempt to reduce the value of a p r o j e c t to i t s c e r t a i n t y e q u i v a l e n t by e s t i m a t i n g a c o e f f i c i e n t of r i s k avers ion and a c o e f f i c i e n t of 10 d i v e r s i f i c a t i o n from the u t i l i t y f u n c t i o n obta ined from the d e c i s i o n maker. P r o j e c t s are then s e l e c t e d or r e j e c t e d on the b a s i s of t h e i r c e r t a i n t y e q u i v a l e n t . Green a l so obta ins a u t i l i t y f u n c t i o n from the d e c i s i o n maker and uses i t to a id i n the d e c i s i o n . p r o c e s s . 1.6 PROFITABILITY INDEX MODELS The p r o f i t a b i l i t y index models , H i r sch and F i s h e r [ 4 5 ] , Olsen [ 6 7 ] , Bobis and Atk inson [ 8 ] , and Disman [23] are a l l based on the same i d e a ; the r a t i o of some measure of the value of the p r o j e c t to some measure of the cost i s used to i n d i c a t e the p r o j e c t ' s d e s i r a b i l i t y . The d i f f e r e n c e s l i e in the measures of value and cost used. Disman's method i s cons idered usefu l [ 2 3 ] . He suggests c a l c u l a t i n g the maximum expendi ture j u s t i f i e d (MEJ) which i s the present value at some acceptab le ra te of r e t u r n of the income generated by the p r o j e c t . This i s mul -t i p l i e d by the p r o b a b i l i t y of t e c h n i c a l and commercial success and d i v i d e d by the t o t a l est imated research and development c o s t s . Severa l other more s p e c i a l i z e d i n d i c e s have been devised [ 3 6 ] . P r o f i t a b i l i t y index methods are s i m i l a r to s c o r i n g models in that they prov ide a s i n g l e numeric measure of the d e s i r a b i l i t y of each p r o j e c t , and an o r d i n a l ranking of p r o j e c t s . Whereas s c o r i n g methods are most s u i t a b l e f o r pure research p r o j e c t s and those p r o j e c t s in t h e i r e a r l y stages of development 11 which have only imprec ise data a v a i l a b l e , p r o f i t a b i l i t y index models are most s u i t a b l e f o r commercial p r o j e c t s , and p r o j e c t s near c o m p l e t i o n , where accurate est imates can be made of costs and b e n e f i t s . 1.7 RISK ANALYSIS Risk a n a l y s i s i s a s i m u l a t i o n technique f o r d e t e r m i n -ing the p r o b a b i l i t y d i s t r i b u t i o n of r e t u r n on a p r o j e c t [ 4 1 , 59, 8 0 ] . D i s t r i b u t i o n f u n c t i o n s must be s p e c i f i e d f o r each f a c t o r which a f f e c t s r e t u r n . Mal loy [59] suggests using a beta d i s -t r i b u t i o n so that only est imates of the l o w e s t , most l i k e l y , and h ighest p o s s i b l e values are requ i red to d e f i n e the d i s t r i -b u t i o n s . The p r o j e c t ' s development i s s imulated by choosing a value f o r each f a c t o r accord ing to i t s d i s t r i b u t i o n and com-b i n i n g these values in the a p p r o p r i a t e way. Bobis et at. [ 7 , 9] have suggested using a s i m i l a r technique to determine more r e l i a b l e est imates of the f a c t o r s requ i red in t h e i r non-l i n e a r model . 1.8 DECISION TREES Many of the d e c i s i o n t ree models use r i s k a n a l y s i s as a s o l u t i o n techn ique . A d e c i s i o n t ree i s a g r a p h i c a l r e p r e s e n -t a t i o n of the expected stages of p r o j e c t development. Each f u t u r e d e c i s i o n po in t and chance outcome po int i s represented by 12 a node in the t r e e . Hespos and Strassman [43] suggest us ing r i s k a n a l y s i s on each p o s s i b l e path through the t ree ( i f there are not too many), or e l i m i n a t i n g some paths by dominance, and a n a l y z i n g the r e s t in more d e t a i l . Locket t and Freeman [55] and Locket t and Gear [56] suggest sampling at each chance o u t -come p o i n t , and reducing the problem to a d e t e r m i n i s t i c l i n e a r program. This procedure i s repeated many t i m e s , r e s u l t i n g in a set of f e a s i b l e programs which are opt imal fo r one p a r t i c u l a r s t a t e of the w o r l d . The f i n a l program i s s e l e c t e d by examining t h i s set f o r p r o j e c t s which are always s e l e c t e d or never s e l e c t e d and f o r other s i g n i f i c a n c e p a t t e r n s . An i n t e g e r p r o -gramming method of s o l v i n g the d e c i s i o n t ree problem d i r e c t l y (without s i m u l a t i o n ) i s g iven by Gear and Locket t [ 3 3 ] . I t becomes unso lveab le however, when there are many chance outcome p o i n t s . 1.9 LIMITATIONS OF THE MODELS Very few of the models and s e l e c t i o n procedures tha t have been suggested i n the l i t e r a t u r e have a c t u a l l y been im-plemented. Baker and Pound [5] suggest two reasons f o r t h i s : l a c k of t e s t i n g and computat ional e x p e r i e n c e , and lack of r e a l i in many of the models. The most c l e a r l y u n r e a l i s t i c f e a t u r e of many of the models i s the o b j e c t i v e f u n c t i o n . The goal of p r o f i t maximiza -t i o n i s the only one cons idered in most c a s e s . In a study of 13 the u t i l i t y and a c c e p t a b i l i t y of p r o j e c t s e l e c t i o n models , Souder found that t h i s need not n e c e s s a r i l y be the o n l y , nor even the primary g o a l . In h i s s t u d y , "none of the seven a d m i n i -s t r a t o r s in te rv iewed i n d i c a t e d a s t rong p r o c l i v i t y to pursue the maximizat ion of expected v a l u e s . Several a d m i n i s t r a t o r s i n d i c a t e d a d e f i n i t e r e j e c t i o n of such o b j e c t i v e s . A l l the a d m i n i s t r a t o r s viewed severa l non-monetary goals as paramount c o n s i d e r a t i o n s . Some a d m i n i s t r a t o r s i n d i c a t e d that va r ious i n t r i n s i c p r o p e r t i e s of the p o r t f o l i o s themselves could be more important c o n s i d e r a t i o n s than the shor t term p r o f i t a b i l i t y s t a t i s t i c s " [78]'. The s c o r i n g - t y p e models and some of the l i n e a r models cons ider other types of g o a l s . However the method of combining them i n t o an o b j e c t i v e f u n c t i o n i s g e n e r a l l y q u i t e a r b i t r a r y and thus t h e i r r e l a t i v e importance in the model i s not the same as t h e i r r e l a t i v e importance in the eyes of the d e c i s i o n maker. What i s needed t h e n , i s a technique to combine any of the p o s s i b l e goals i n t o an o b j e c t i v e f u n c t i o n in accordance w i th the d e c i s i o n maker's p r i o r i t i e s . "The assumption that there e x i s t s an opt imal s o l u t i o n or a set of opt imal s o l u t i o n s to a problem i n v o l v i n g m u l t i p l e c r i t e r i a i m p l i e s the e x i s t e n c e of some preference o rder ing def ined over the set of f e a s i b l e values of the c r i t e r i a " [ 2 4 ] . The problem i s then reduced to one o f f i n d i n g t h i s "pre ference o r d e r i n g , " a sub jec t which i s d i scussed i n the next chapte r . 14 Another u n r e a l i s t i c f e a t u r e of many of the models concerns the assumption made about funding l e v e l s . The z e r o -one models assume funding i s p o s s i b l e at only one l e v e l . Many of the l i n e a r and n o n - l i n e a r models assume funding i s p o s s i b l e at any l e v e l . A more r e a l i s t i c approach would be to a l low a p r o j e c t to be funded in a range or not at a l l . Freeman [31] cons iders t h i s idea but does not implement i t . He suggests that there i s a " c r i t i c a l cost l e v e l " below which the value and p r o b a b i l i t y of success of a p r o j e c t i s very s m a l l , and a " s a t i a t i o n p o i n t " above which a d d i t i o n a l funding c reates l i t t l e a d d i t i o n a l value and inc reases the p r o b a b i l i t y of success by an i n s i g n i f i c a n t amount. These upper and lower bounds could a l s o be d i c t a t e d by o r g a n i z a t i o n a l p o l i c y . There may be an upper l i m i t on the amount that may be r i s k e d on any one p r o j e c t and a lower l i m i t determined by the l e a s t amount that i s " reasonab le" to i n v e s t in a p r o j e c t . The p r o b a b i l i t y of success of a p r o j e c t i s d i r e c t l y r e l a t e d to i t s funding l e v e l . Most of the more s o p h i s t i c a t e d models r e q u i r e a s u b j e c t i v e est imate of the p r o b a b i l i t y of success of each p r o j e c t at a g iven l e v e l . This est imate i s used to develop the r e l a t i o n s h i p between funding and p r o b a b i l i t y of s u c c e s s . A study by Souder [74] on the v a l i d i t y of s u b j e c t i v e p r o b a b i l i t y of success e s t i m a t e s , suggests tha t they are g e n e r a l l y v a l i d and r e l i a b l e al though they may not always be a c c u r a t e l y communicated due to u l t e r i o r motives on the part of the d e c i s i o n maker and o r g a n i z a t i o n a l p r e s s u r e s . 15 if The p r o j e c t s e l e c t i o n model which w i l l be developed i n the f o l l o w i n g chapters w i l l attempt to d e s c r i b e the s e l e c t i process more r e a l i s t i c a l l y by i n c l u d i n g m u l t i p l e c r i t e r i a , funding ranges , and p r o b a b i 1 i t y of success e s t i m a t e s . Chapter 2 RATIONAL DECISION MAKING WITH ' MULTIPLE CRITERIA Rat iona l d e c i s i o n making i m p l i e s a g o a l - d i r e c t e d cho ice among a l t e r n a t i v e courses of a c t i o n . G o a l - d i r e c t e d cho ice among a l t e r n a t i v e s r e q u i r e s both a knowledge of the correspondence between a c t i o n s and outcomes, and a s u b j e c t i v e preference o r d e r -ing among outcomes. This chapter focuses on the l a t t e r , the development of methods f o r e v a l u a t i n g outcomes in terms of goals and o rder ing them on the bas is of t h e i r s u b j e c t i v e v a l u e . Outcomes can be descr ibed by a set of v a r i a b l e s or a t t r i b u t e s , which r e f l e c t the dimensions through which the ou t -come c o n t r i b u t e s to or d e t r a c t s from the u l t i m a t e goals or o b j e c t i v e s of the d e c i s i o n maker. For example, i n choosing a house, the a l t e r n a t i v e s can be presented in terms of such a t t r i -butes as space , p r i c e , convenience of l o c a t i o n , c o n d i t i o n and f a c i l i t i e s , e t c . , which d e s c r i b e each house complete ly wi th respect to the d e c i s i o n maker 's g o a l s , and thus d e f i n e c r i t e r i a f o r e v a l u a t i o n . These c r i t e r i a may correspond d i r e c t l y to the g o a l s , or they may be s imply i n d i c a t o r s which are r e l a t e d to the g o a l s . For another example cons ider a government agency 16 17 tha t wishes to fund a number of p r o j e c t s . Two of i t s goals might be to inc rease employment and to mainta in Canadian s o v e r e i g n t y . One of the a t t r i b u t e s used in t h i s case f o r p r o j e c t e v a l u a t i o n corresponds d i r e c t l y to the f i r s t , g o a l , i . e . the number of jobs c r e a t e d . The second goal can be measured in severa l ways, fo r example, by (1) the r e s u l t i n g percentage inc rease in Canadian ownership of f i r m s , and (2) the i n c r e a s e in the amount of l o c a l raw m a t e r i a l s processed i n Canada. These l a s t two c r i t e r i a are merely i n d i c a t o r s which are r e l a t e d to the second g o a l . M u l t i p l e c r i t e r i a d e c i s i o n problems have a w e l l -def ined s o l u t i o n i f there e x i s t s a preference o rder ing de-f i n e d over the f e a s i b l e values of the c r i t e r i a which i s complete and t r a n s i t i v e . Completeness means that a l l a l t e r n a t i v e s can be compared and the one wi th the g r e a t e s t r e l a t i v e va lue can be found . This may be a d i f f i c u l t requirement i n p r a c t i c e . The d e c i s i o n maker may be able to choose between two a l t e r n a t i v e s which g ive him $500 or $800, but may f i n d i t more d i f f i c u l t to choose between a l t e r n a t i v e s which g ive him $500 or a t r i p to H a w a i i . In the f i r s t case he. need only compare the l e v e l s of the r e l e v a n t a t t r i b u t e (money). In the second case he must r e l a t e the d i f f e r e n t a t t r i b u t e s (money, t r i p ) to an under l y ing goal ( p o s s i b l y p r e s t i g e ) in order to determine which one has the g r e a t e r r e l a t i v e v a l u e . In the context of the p r o j e c t s e l e c t i o n problem the a l t e r n a t i v e s (programs of p r o j e c t s ) wi11 18 g e n e r a l l y d i f f e r along the l e v e l of the r e l e v a n t a t t r i b u t e s ra ther than by having d i f f e r e n t a t t r i b u t e s a l t o g e t h e r . There -f o r e they w i l l tend to be e a s i e r to compare. T r a n s i t i v i t y means that i f a l t e r n a t i v e B i s p r e f e r r e d to a l t e r n a t i v e A (the r e l a t i v e value of B i s g rea te r than the r e l a t i v e value, of A ) , and a l t e r n a t i v e C i s p r e f e r r e d to a l t e r n a -t i v e B, then a l t e r n a t i v e C i s p r e f e r r e d to a l t e r n a t i v e A. This i s a reasonable assumption wften a l l a l t e r n a t i v e s are r e a d i l y comparable , i . e . completness h o l d s . In many s i t u a t i o n s a 11 that i s requ i red of the d e c i s i o n maker i s that he rank h is a l t e r n a t i v e s , by c o n s i d e r i n g each c r i t e r i o n and the r e l a t i o n s h i p s between them. However, i n the case of a p r o j e c t s e l e c t i o n d e c i s i o n , the number of a l t e r n a t i v e s i s l a r g e . An a l t e r n a t i v e in t h i s case i s a p o r t f o l i o of R&D p r o j e c t s w i th a s p e c i f i e d funding l e v e l f o r each. Each p r o j e c t under c o n s i d e r a t i o n i s not s imply one p o s s i b l e component of a research and development program. I t i s a r e p r e s e n t a t i v e of a set of p o s s i b l e components, d i s t i n g u i s h e d by t h e i r funding l e v e l s . Each funding l e v e l i m p l i e s a d i f f e r e n t p r o b a b i l i t y of success f o r the p r o j e c t , and thus r e s u l t s in a d i f f e r e n t amount of the a t t r i b u t e s of that p r o j e c t . Therefore there i s an i n f i n i t e number of p o s s i b l e research and development programs, and i t i s imposs ib le to so lve the s e l e c t i o n problem by s imply ranking the a l t e r n a t i v e s . What i s requ i red i s an e x p l i c i t p r e f -erences o rder ing that may be used to f i n d an opt imal p o r t f o l i o of p r o j e c t s . 19 There are many methods of determining preference o rder ings desc r ibed i n the l i t e r a t u r e . These may be c l a s s i f i e d in to s e q u e n t i a l e l i m i n a t i o n methods, s p a t i a l p r o x i m i t y methods, mathematical programming methods, and u t i l i t y f u n c t i o n methods ( d i r e c t l y assessed preference techniques and i n f e r r e d preference t e c h n i q u e s ) . ^ Table 1 summarizes the c l a s s i f i c a t i o n and l i s t s the methods which f a l l i n t o each ca tegory . 2 .1 SEQUENTIAL ELIMINATION METHODS Sequent ia l e l i m i n a t i o n methods order the a l t e r n a t i v e s by comparing them e i t h e r to each other or to a s t a n d a r d . I f a l t e r n a t i v e s are compared to each o t h e r , those which prov ide l e s s of a l l c r i t e r i a than another can be e l i m i n a t e d . This dominance technique i s used by Terry [79] as an i n i t i a l f i l t e r i n s e l e c t i n g new product a r e a s . Genera l l y very few a l t e r n a t i v e s can be e l i m i n a t e d t h i s way. Other methods must be used to determine the o rder ing of the remaining a l t e r n a t i v e s . Another method of o rde r ing a l t e r n a t i v e s by comparing them to each other i s l e x i c o g r a p h y . The a l t e r n a t i v e s are ranked on the bas is of t h e i r r a t i n g on the most important c r i t e r i o n . I f any of the a l t e r n a t i v e s prove to be equal i n v a l u e , then the r a t i n g , on the next most important c r i t e r i o n i s c o n s i d e r e d . T h i s c l a s s i f i c a t i o n i s e s s e n t i a l l y the one p roposed by MacCrimmon C 5 8 ] . Table I Methods of Handl ing M u l t i p l e C r i t e r i a CLASSIFICATION METHOD CHARACTERISTICS A. S e q u e n t i a l E l i m i n a t i o n Methods Domi nance L e x i c o g r a p h y E l i m i n a t i o n by Aspects C o n j u n c t i v e ' D i s j u n c t i v e a l t e r n a t i v e s are compared to each o t h e r ; those a l t e r n a t i v e s w i th lower l e v e l s of a l l c r i t e r i a may be e l i m i n a t e d . a l t e r n a t i v e s are ordered on the b a s i s of t h e i r r a t i n g on the most important c r i t e r i o n most d i s c r i m i n a t o r y c r i t e r i o n used to order a l t e r n a t i v e s 0 a l t e r n a t i v e s are compared to a s t a n d a r d ; e l i m i n a t e a l l whose worst c r i t e r i o n l e v e l does not meet the s t a n d a r d ; non-compensatory accept a l t e r n a t i v e s i f any c r i t e r i o n l e v e l exceeds the s t a n d a r d , non-compensatory B. S p a t i a l P r o x i m i t y Methods I n d i f f e r e n c e Map M u l t i - d i m e n s i o n a l S e a l i ng a set of i n d i f f e r e n c e or t r a d e o f f curves are o b t a i n e d f o r each p a i r of c r i t e r i a ; a l t e r n a t i v e s can then be p o s i t i o n e d wi th r e s p e c t to these c u r v e s , and or o r d e r i n g o b t a i n e d a l t e r n a t i v e s are l o c a t e d in a m u l t i - d i m e n s i o n a 1 space so t h a t t h e i r va lue i s i n v e r s e l y p r o p o r t i o n a l to t h e i r d i s t a n c e from the i d e a l p o i n t C. Mathemat ica l Programming Methods Goal Programmi ng I n t e r a c t i v e T r a d e o f f Technique an a l t e r n a t i v e i s generated which min imizes the weighted sum of d e v i a t i o n s from a set of g o a l s ; uses L . P . b e t t e r a l t e r n a t i v e s are i t e r a t i v e l y generated by r e q u i r i n g the user to s p e c i f y t r a d e o f f s between c r i t e r i a at the c u r r e n t p o i n t and using them to f i n d the best d i r e c t i o n to move D. U t i l i t y F u n c t i o n Methods 1 . Di r e c t l y Assessed P r e f e r e n c e Techniques D i r e c t R a t i ng Method Ranking Methods Gamble Methods Ordered M e t r i c Methods d e c i s i o n maker r a t e s c r i t e r i o n l e v e l s on an a r b i t r a r y s c a l e c r i t e r i o n l e v e l s ranked g e n e r a l l y us ing p a i r w i s e compar isons one v a r i a n t , d e c i s i o n maker asked f o r p r o b a b i l i t y p (z ) which makes a gamble made up of two c r i t e r i o n l e v e l s x and y o c c u r r i n g w i th p r o b a b i l i t i e s p(z ) and 1 - p (z ) r e s p e c t i v e l y as v a l u a b l e to him as l e v e l z d i f f e r e n c e s between u t i l i t y ad jacent c r i t e r i o n l e v e l s are r a n k e d , and u t i l i t i e s ass igned a c c o r d i n g l y CONTINUED Table I (Cont inued) CLASSIFICATION METHOD CHARACTERISTICS I n f e r r e d P r e f e r e n c e Techniques S u c c e s s i v e Comparison Methods T r a d e o f f Methods L i n e a r & P i e c e w i s e L i n e a r Forms C o n j u n c t i v e - Form D i s j u n c t i v e Form L o g a r i t h m i c Form E x p o n e n t i a l Form Combinat ion Form i n e q u a l i t i e s between sums of u t i l i t i e s are o b t a i n e d and used to a s s i g n u t i l i t i e s to c r i t e r i o n l e v e l s e . g . s i n g l e t r a d e o f f method d e r i v e s the u t i l i t y f u n c t i o n f o r one c r i t e r i o n from a t r a d e o f f or i n d i f f e r e n c e curve between i t and another c r i t e r i o n and the u t i l i t y f u n c t i o n of the other c r i t e r i o n U = k=l 3 k X k or U = I a?X. + k=l K K n a. u = n x. *• k - 1 k u = n k = l 1 2 3 k " 3 k ^ k " X k ) ' U = I a. log X, k = l n e K K k = l n n k = l m i n X c o e f f i c i e n t s e s t i m a t e d by a p p l y i n g l i n e a r r e -g r e s s i o n to past d e c i -s ions or s i m u l a t e d d e c i s i o n s i t u a t i o n s pre -sented to the d e c i s i o n maker 22 E l i m i n a t i o n by aspects i s s i m i l a r to lex i cography except that i n s t e a d of choosing the most important c r i t e r i o n as the bas is f o r compar ison, the one with the most d i s c r i m i n a t o r y power (g rea tes t range , fewest t i e s , e t c . ) i s chosen. These two methods of handl ing m u l t i p l e c r i t e r i a seem to have been developed because they are convenient ra ther than because they model any conscious or l o g i c a l s t r a t e g y fo r f i n d i n g the best a l t e r n a t i v e . There are two methods of comparing a l t e r n a t i v e s to a s t a n d a r d . One i s to examine each a l t e r n a t i v e f o r i t s "worst" c r i t e r i o n l e v e l , and accept or r e j e c t i t on the bas is of whether or not t h i s worst l e v e l meets the s t a n d a r d . This c o n j u n c t i v e s t r a t e g y r e s u l t s in a set of acceptab le a l t e r n a t i v e s which exceed the standard wi th respect to each c r i t e r i o n . I f a govern -ment agency was concerned wi th keeping i t s fund g r a n t i n g program f r e e of p u b l i c c r i t i c i s m , i t might adopt such a d e c i s i o n p ro -cedure to prevent an o b v i o u s l y poor performance wi th respect to any one c r i t e r i o n . The other method i s a d i s j u n c t i v e s t r a t e g y . In t h i s case the best c r i t e r i o n l e v e l of each a l t e r n a t i v e i s found , and i f i t exceeds the s t a n d a r d , then the a l t e r n a t i v e i s a c c e p t a b l e . This s t r a t e g y r e s u l t s i n a set of a l t e r n a t i v e s which exceed the standard wi th respect to at l e a s t one c r i t e r i o n . A research cent re that wishes to develop an e x c e l l e n t r e p u t a t i o n might opt to use such a s t r a t e g y to r e c r u i t s c i e n t i s t s on the bas is of t h e i r s t ronges t f i e l d of endeavor. 23 In each case the problem of o rde r ing the accepted a l t e r n a t i v e s remains , and must be r e s o l v e d us ing one of the other methods. Because of the r e s t r i c t i v e l o g i c used in these methods, they are a p p l i c a b l e i n a l i m i t e d number of cases o n l y . The r e s u l t i n g acceptab le set of a l t e r n a t i v e s i s very d i f f e r e n t w i th the two methods. Dawes [17] d i s c u s s e s the c h a r a c t e r i s t i c s of groups of people s e l e c t e d f o r va r ious p o s i t i o n s by the two methods. The s e q u e n t i a l e l i m i n a t i o n methods are non-compensa-t o r y . A high l e v e l of achievement w i th respec t to one c r i t e r i o n cannot compensate f o r a d e f i c i e n c y in another a r e a . In many s i t u a t i o n s t h i s assumption i s too r e s t r i c t i v e . However there are cases where these techniques have been e f f e c t i v e . Kleinmuntz [52] uses a sequence of e l i m i n a t i o n methods of t h i s type to model a c l i n i c a l p s y c h o l o g i s t judg ing t e s t p r o f i l e s (MMPI) as being "normal" or " a b n o r m a l . " Smith and Greenlaw [72] use a s i m i l a r technique to model a p s y c h o l o g i s t s e l e c t i n g a p p l i c a n t s f o r j o b s . In both s t u d i e s the s u b j e c t s were asked to d e s c r i b e the techniques they were using as they made a set of d e c i s i o n s . T h e i r d e s c r i p t i o n s and d e c i s i o n s were examined to d e r i v e a set of s e q u e n t i a l r u l e s which could be a p p l i e d to other problems. 2 .2 SPATIAL PROXIMITY METHODS S p a t i a l p r o x i m i t y methods are o rder ing techniques 24 which r e l y more h e a v i l y on a geometric r e p r e s e n t a t i o n . The i n d i f f e r e n c e map technique obta ins i n d i f f e r e n c e curves f o r each p a i r of c r i t e r i a . An i n d i f f e r e n c e curve i s a set of p o i n t s , each- having two c o - o r d i n a t e s , which represent the l e v e l s of the two c r i t e r i a . A l l combin-a t i o n s of the two c r i t e r i a which are seen by the d e c i s i o n maker as having the same r e l a t i v e value are loca ted on the same i n d i f f e r e n c e c u r v e . MacCrimmon and Toda [57] d e s c r i b e an e f f i c i e n t method of c o n s t r u c t i n g i n d i f f e r e n c e c u r v e s . One p o s s i b l e combinat ion of the two c r i t e r i a , say ( a , b ) , i s chosen as a re fe rence p o i n t . S ince i t i s assumed that a higher l e v e l of any c r i t e r i o n i s always p r e f e r r e d to a lower one, a l l po in ts "north e a s t " of the re fe rence point are more v a l u a b l e than the re fe rence p o i n t , and can be thought of as the "accept r e g i o n . " A l l po in ts "south west" of the re fe rence po int are l e s s v a l u a b l e , and can be thought of as the " r e j e c t " r e g i o n . These areas may be blocked o f f and excluded from f u r t h e r c o n s i d e r a t i o n as in F igure 2 . 1 . A c i r c l e i s drawn around ( a , b , ) . The d e c i s i o n maker i s then asked to compare in t u r n , the m i d - p o i n t of each of the areas which f a l l in the open reg ion to the re fe rence p o i n t . I f the m i d - p o i n t i s accepted ( r e j e c t e d ) more po in ts on the arc i n the d i r e c t i o n of the r e j e c t (accept) reg ion are compared to the re ference po int u n t i l one of them F i g u r e 2 . 1 O b t a i n i n g an I n d i f f e r e n c e Curve by the MacCrimmon-Toda Method 26 i s r e j e c t e d (accepted ) . Each time a po in t i s compared to the re fe rence p o i n t , another area may be blocked o f f as belonging to e i t h e r the accept or r e j e c t r e g i o n . Th is process i s cont inued u n t i l the open area i s narrow enough f o r an i n d i f f e r e n c e curve to be drawn through i t . A l l po in ts on the i n d i f f e r e n c e curve would have approx imate ly the same s u b j e c t i v e va lue as the re fe rence po in t ( a , b ) . Other i n d i f f e r e n c e curves f o r the same two c r i t e r i a , cor responding to d i f f e r e n t va lue l e v e l s are der i ved by choosing a d i f f e r e n t re fe rence p o i n t . The same technique i s a p p l i e d to a l l other p a i r s of c r i t e r i a to d e r i v e a complete i n d i f f e r e n c e map. The a l t e r n a t i v e s can then be p o s i t i o n e d i n t h i s m u l t i -d imensional space wi th respect to the i n d i f f e r e n c e curves and a complete o rder ing among them o b t a i n e d . M u l t i - d i m e n s i o n a l s c a l i n g i s another p r o x i m i t y o rde r ing t e c h n i q u e . An i d e a l a l t e r n a t i v e i s assumed to e x i s t . The value of any other a l t e r n a t i v e i s assumed to be i n v e r s e l y p r o p o r t i o n a l to the " d i s t a n c e " of that a l t e r n a t i v e from the i d e a l p o i n t . K lahr [51] d e s c r i b e s a method of l o c a t i n g the r e a l and i d e a l a l t e r n a t i v e s i n a m u l t i - d i m e n s i o n a l space. He f i r s t r e q u i r e s the d e c i s i o n maker to judge the s i m i l a r i t y of a set of a l t e r n a t i v e s . The a l t e r n a t i v e s are then p o s i t i o n e d , c o n s i s t e n t w i th the near -ness assumpt ion . This can always be done in an N- l d imensional space, where N i s the number of a l t e r n a t i v e s . The next step i s to i t e r a t i v e l y reduce the d i m e n s i o n a l i t y of the space u n t i l 27 i t i s l e s s than or equal to the number of c r i t e r i a , s i n c e one would expect to need no more than one dimension f o r each c r i t e r i o n t h a t i s s i g n i f i c a n t i n the d e c i s i o n making p r o c e s s . The dimen-s i o n f i n a l l y chosen i s the one that minimizes s t r e s s (a measure of departure from p e r f e c t f i t ) . K lahr a p p l i e s t h i s technique to the problem of graduate admiss ion d e c i s i o n s . The method however i s c o m p u t a t i o n a l l y d i f f i c u l t and has not been p a r t i c u -l a r l y s u c c e s s f u l . 2 .3 MATHEMATICAL PROGRAMMING METHODS There are two mathematical programming methods: goal programming and an i n t e r a c t i v e t r a d e o f f techn ique . Goal p ro -gramming r e q u i r e s the d e c i s i o n maker to s p e c i f y d e s i r e d or acceptab le l e v e l s of a set of g o a l s . A goal may correspond to one of the c r i t e r i a or may be combinat ions of the c r i t e r i a . The amount of d i s s a t i s f a c t i o n accompanying any a l t e r n a t i v e which does not s a t i s f y the goals i s determined by o b t a i n i n g we ight ing c o e f f i c i e n t s f o r d e v i a t i o n s i n each d i r e c t i o n from each g o a l . The weighted sum of these d e v i a t i o n s i s used as the o b j e c t i v e f u n c t i o n i n a standard LP m i n i m i z a t i o n procedure . The r e s u l t -ing opt imal s o l u t i o n i s the c r i t e r i o n l e v e l s of the best a l t e r -n a t i v e . Lee and C layton [54] apply the technique to the schedu l ing of an academic department. 28 The i n t e r a c t i v e t r a d e o f f technique assumes that a g loba l o b j e c t i v e f u n c t i o n e x i s t s but does not r e q u i r e i t to be d e f i n e d . At any f e a s i b l e a l t e r n a t i v e the d e c i s i o n maker 's t r a d e o f f s among the c r i t e r i a in the neighbourhood of the a l t e r -na t i ve are determined. The d e c i s i o n maker i s asked to choose between a l t e r n a t i v e s of the form ( f l 9 . . . f ) and (fx + A fi, f 2 » . . . , - f . j + A f . j , . . . f r ) where f are the c r i t e r i o n l e v e l s . The A f i and Af . are v a r i e d u n t i l the d e c i s i o n maker i s i n d i f f e r e n t between the two a l t e r n a t i v e s . (This technique i s s i m i l a r to the method used in d e r i v i n g i n d i f f e r e n c e c u r v e s . ) Then the t r a d e o f f between c r i t e r i o n i and c r i t e r i o n 1 , w.j = - A f i / A f i s used in approx imat ion to the g rad ient of the g loba l u t i 1 i t y f u n c t i o n at the cur rent p o i n t . This a p p r o x i -mation i s used in the o b j e c t i v e f u n c t i o n of a mathematical programming a l g o r i t h m which determines the best d i r e c t i o n to move, i . e . the d i r e c t i o n in which b e t t e r po in ts than the cu r ren t one l i e . A set of po ints at va r ious d i s t a n c e s from the cu r ren t po int in that d i r e c t i o n i s then generated and the d e c i s i o n maker i s asked to choose the best one. I f t h i s new a l t e r n a t i v e i s "good enough" the procedure t e r m i n a t e s , otherwise i t i s repeated . G e o f f r i o n , Dyer, and Fe-inberg [25 , 35] have developed t h i s t e c h -nique and a p p l i e d i t to the schedu l ing of an academic department. These techniques c o n s t r u c t the best a l t e r n a t i v e rather - than choosing i t from a set of e x p l i c i t l y p r e - d e f i n e d a l t -e r n a t i v e s . They can only be used in s i t u a t i o n s such as a p r o j e c t 29 s e l e c t i o n problem where the c r i t e r i a take on cont inuous v a l u e s . App ly ing these techniques to the p r o j e c t s e l e c t i o n problem r e s u l t s in the opt imal combinat ion of c r i t e r i o n l e v e l s . 2.4 UTILITY FUNCTION METHODS The l a s t method of d e a l i n g wi th m u l t i p l e c r i t e r i a to be cons idered here i s the use of u t i l i t y f u n c t i o n s . In the l i t e r a t u r e the term " u t i 1 i t y f u n c t i o n " has a s p e c i f i c meaning which depends on the ax iomat ic system being f o l l o w e d . However, here i t w i l l s imply mean a mapping which ass igns to an a l t e r n a -t i v e a rea l number ( u t i l i t y ) which i n d i c a t e s the r e l a t i v e worth of the a l t e r n a t i v e . This mapping has the two p r o p e r t i e s men-t ioned p r e v i o u s l y ; i t i s complete and i t i s t r a n s i t i v e . There are two main c l a s s e s of methods used i n d e r i v i n g u t i l i t y f u n c t i o n s ; d i r e c t l y assessed preferences and i n f e r r e d p r e f e r e n c e s . F ishburn [29] desc r ibes 24 methods of d i r e c t l y a s s e s s i n g p r e f e r e n c e s . The bas ic ideas behind these methods are d iscussed below. 2 . 4 . 1 D i r e c t l y Assessed Preference Techniques A l l of the methods assume that the u t i l i t y of an a l -t e r n a t i v e i s the sum of the u t i l i t i e s of the i n d i v i d u a l c r i t e r i o n 1 e v e 1 s , i . e. 30 U ( a l t e r n a t i v e ) = U i ( x i ) + u 2 ( x 2 ) + . . . +u n ( x n ) where the x^ are the l e v e l s of the c r i t e r i a achieved by . the a l t e r n a t i v e and i s the u t i l i t y f u n c t i o n fo r the i c r i t e r i o n . F ishburn [ 2 8 , 2 9 , 3 0 ] and v. W i n t e r f e l d t and F i s c h e r [83] d i s c u s s the c o n d i t i o n s under which, t h i s a d d i t i v i t y assumption h o l d s . The d i r e c t l y assessed preference methods f i n d u t i l i t y f u n c t i o n s u^  fo r each of the c r i t e r i a . The s i m p l e s t method i s the d i r e c t r a t i n g method. The' d e c i s i o n maker i s asked to ass ign a u t i l i t y to each c r i t e r i o n l e v e l by r a t i n g i t on an a r b i t r a r y s c a l e . The other methods ask the d e c i s i o n maker quest ions on h is p r e f e r e n c e s , then attempt to ass ign u t i l i t i e s c o n s i s t e n t wi th h is responses . Ranking methods ask fo r an o rder ing of the f e a s i b l e c r i t e r i o n l e v e l s . This g e n e r a l l y can be reduced to a s e r i e s of p a i r w i s e compar isons. For example, to f i n d the most p r e f e r r e d c r i t e r i o n l e v e l of n f e a s i b l e ones, the d e c i s i o n maker may choose a c a n d i d a t e , then compare t h i s l e v e l with each other f e a s i b l e l e v e l in t u r n , in order to v e r i f y the hypothesis that i t i s more p r e f e r r e d . U t i l i t i e s are then ass igned to each f e a s i b l e c r i t e r i o n l e v e l in accordance with t h i s r a n k i n g . One way would be to ass ign the most p r e f e r r e d l e v e l a u t i l i t y of n, the next p r e f e r r e d n-1 , e t c . . Gamble methods r e q u i r e the d e c i s i o n maker to choose between gambles made up of var ious c r i t e r i o n l e v e l s which may occur wi th var ious p r o b a b i l i t i e s . One such method uses the l e a s t and most d e s i r a b l e l e v e l of a c r i t e r i o n , say X. and Y. 31 r e s p e c t i v e l y . For a set of other c r i t e r i o n l e v e l s ( z^ ) , the d e c i s i o n maker i s asked to est imate the p r o b a b i l i t y p(z..) f o r which z^ has the same value to him as a gamble between and y. , where y^ occurs wi th p r o b a b i l i t y p(z^.) and x^ occurs wi th p r o b a b i l i t y 1 - p (z . . ) . The u t i l i t y of c r i t e r i o n l e v e l z^ i s then p (z i ) u ( y i ) + [1 - p(z i ) ] u ( x . ) . The u t i l i t y of l e v e l s x . and y. can be a r b i t r a r i l y set ( fo r example to 1 and 100) and the u t i l i t y of z^ determined. Ordered met r i c methods r e q u i r e a ranking of the d i f -ferences between c r i t e r i o n l e v e l s which are adjacent in u t i l i t y . An i n i t i a l ranking of the c r i t e r i o n l e v e l s i s necessary . One example, the d i r e c t ordered met r i c method, r e q u i r e s the d e c i s i o n maker to rank the u t i l i t y d i f f e r e n c e s between four adjacent c r i t e r i o n l e v e l s . Suppose u (a) < u(b) < u(c) < u ( d ) and 0 < [u(d) - u (c ) ] < [u(c) - u(b) ] < [u(b) - u ( a ) ] . Numerical assignments c o n s i s t e n t w i th t h i s met r i c rank ing may be made. One p o s s i b l e way of a s s i g n i n g numbers i s to set the u t i l i t y of the l e a s t d e s i r e d l e v e l to 0 and set u(y) - u(x) to k when t h i s d i f f e r e n c e i s the k ^ one i n the r a n k i n g . For the example above t h i s i m p l i es u(a) = 0 , u(b) - u(a) u(c) - u(b) u(d) - u(c) 3 , u (b) = 3 , 2, u (c) = 5 , 1 , u (d) = 6 . 32 Success ive comparison methods work wi th groups of u t i l i t y ad jacent c r i t e r i o n l e v e l s , , and attempt to ass ign u t i l -i t i e s to them by examining i n e q u a l i t i e s between sums of u t i l i t i e s . For example, suppose again u (a) < u(b)< u(c) < u (d ) . Compare u(d) w i th u(b) + u ( c ) . I f u(d) < u(b) + u ( c ) , compare u(c) wi th u(a) + u (b ) . I f u(d) > u(b) + u(c) compare u ( d ) . w i t h u(a) +u(b) + u ( c ) . Continue in t h i s manner u n t i l a complete set of i n e q u a l i t i e s i s o b t a i n e d . Numerical u t i l i t i e s may now be ass igned to each c r i t e r i o n l e v e l c o n s i s t e n t wi th these i n -e q u a l i t i e s . Many v a r i a n t s of t h i s method are p o s s i b l e . There are severa l usefu l types of t r a d e - o f f methods. The s i n g l e t r a d e - o f f method uses a set of i n d i f f e r e n c e curves between two c r i t e r i a , and the p r e v i o u s l y der i ved u t i l i t y f u n c -t i o n f o r one of the c r i t e r i a i n order to de r i ve the u t i l i t y f u n c t i o n f o r the o t h e r . I f ( t i , S i ) i s on the same i n d i f f e r e n c e curve as ( t 2 , s 2 ) then u ( t i ) + u ( s i ) = u ( t 2 ) + u ( s 2 ) . I f u ( t i ) i s known, a set of such equat ions can be so lved f o r u(s^) by s e t t i n g u(s^) f o r some i , to . an a r b i t r a r y v a l u e . One p o s s i -b i l i t y would be to set u(s - n ) to °> where s m i - n i s the l e a s t p r e -f e r r e d s l e v e l . D i r e c t l y assessed preference techniques are e f f e c t i v e in determin ing u t i l i t y f u n c t i o n s , however the c o n d i t i o n s under which the a d d i t i v i t y assumption holds are f a i r l y r e s t r i c t i v e and may not be s a t i s f i e d in a l l p r o j e c t s e l e c t i o n d e c i s i o n s i t u a t i o n s . 33 2 . 4 . 2 I n f e r r e d Preference Techniques I n f e r r e d preference methods attempt to deduce the d e c i s i o n maker's u t i l i t y f u n c t i o n from choices he makes. These choices may be ac tua l past d e c i s i o n s i n s i m i l a r s i t u a t i o n s or they may be d e c i s i o n s made on s imulated problems presented to him. Some hypothesis must now be made about the form of the u t i l i t y f u n c t i o n . If i t is assumed to be l i n e a r or " q u a s i - 1 i n e a r " then the c o e f f i c i e n t s are est imated using standard l i n e a r r e -g ress ion t e c h n i q u e s . I f the u t i l i t y f u n c t i o n i s thought to be n o n - l i n e a r and i n v o l v e i n t e r a c t i o n s among the a t t r i b u t e s , then a n a l y s i s of var iance i s -used to determine the u t i l i t y f u n c t i on. I n f e r r e d preference techniques have been p a r t i c u l a r l y s u c c e s s f u l in p r e d i c t i n g d e c i s i o n s . Huber, Sahney and Ford [49] have used the technique to model p r o f e s s i o n a l s judg ing the q u a l i t y of h o s p i t a l wards. Huber, Daneshgar and Ford [48] have a p p l i e d the techniques to job s e l e c t i o n d e c i s i o n s . Both s t u d i e s used severa l types of f u n c t i o n a l forms an-d found that the l i n e a r one had the g r e a t e s t p r e d i c t i v e power. Einhorn [27] used a l i n e a r , and two " q u a s i - l i n e a r " f u n c t i o n s to model job s e l e c t i o n d e c i s i o n s and graduate school admission d e c i s i o n s . He found that the c o n j u n c t i v e form was most p r e d i c t i v e i n the 2 • A q u a s i - l i n e a r f u n c t i o n i s any f u n c t i o n w h i c h can be made l i n e a r by a s i m p l e t r a n s o f r m a t i o n . For e x a m p l e , the m u l t i -p l i c a t i v e form U = TT X. i s q u a s i - l i n e a r s i n c e l o g U = Z log X. i • i ' i 1 i s l i n e a r . 1 34 job s e l e c t i o n c a s e , and the l i n e a r f u n c t i o n was most a p p l i c a b l e to the admission d e c i s i o n problem. The c o n j u n c t i v e form models the s t r a t e g y which r e j e c t s a l t e r n a t i v e s unless they exceed some standard wi th respect to a l l c r i t e r i a . I f even one c r i t e r -ion l e v e l i s l e s s than the standard then the a l t e r n a t i v e has a low u t i l i t y ( i . e . i t i s a non-compensatory s t r a t e g y ) . He argues that t h i s i s a reasonable s t r a t e g y i n s i t u a t i o n s where choosing the wrong a l t e r n a t i v e would be very c o s t l y (such as job s e l e c -t i o n ) . In such c a s e s , that type of c o n s e r v a t i v e , non-compensa-to ry s t r a t e g y would minimize the chances of c o s t l y m i s t a k e s . Goldberg [39] used the same f u n c t i o n s plus two more " q u a s i -l i n e a r " f u n c t i o n s to model psychologists judg i ng (MMPI) t e s t p r o f i l e s . He concluded that the l i n e a r f u n c t i o n best p r e d i c t e d d e c i s i o n s . Hoffman and Wiggins [47] s tud ied the same problem using a l i n e a r f u n c t i o n , a q u a d r a t i c f u n c t i o n , and a s ign model (a l i n e a r combinat ion of scores and f u n c t i o n s of s c o r e s ) . There was some evidence that i n t e r a c t i o n s among c r i t e r i a a f f e c t e d d e c i s i o n s ( c o n f i g u r a l i t y ) , but even so the l i n e a r f u n c t i o n p r e -d i c t e d d e c i s i o n s w e l l . Several authors have suggested using i n f e r r e d p r e f e r -ence techniques to develop a u t i l i t y f u n c t i o n which would then be used as a d e c i s i o n making too l to help e l i m i n a t e c o s t l y i n -c o n s i s t e n c i e s . Bowman [10] and Kunreuther [53^| have used the techniques i n severa l managerial d e c i s i o n making s i t u a t i o n s , ranging from inventory p o l i c y and product ion schedu l ing d e c i s i o n s , 35 to equipment replacement p o l i c y d e c i s i o n s . They c l a i m that mistakes due to a manager's i n c o n s i s t e n c y are more c o s t l y than mistakes caused by his misconcept ion of the problem. The i r exper ience has i n d i c a t e d that most managers have a good under-s tanding of the problems that face them, and from s i m i l a r past s i t u a t i o n s are aware of what f a c t o r s and i n d i c a t o r s are most impor tant . In severa l cases they have been able to show that t h i s type of a n a l y s i s of past d e c i s i o n s has led to a d e c i s i o n -making procedure which a t t a i n e d b e t t e r r e s u l t s than d e c i s i o n s made by f o l l o w i n g a p o l i c y developed from a more a n a l y t i c a l study o f - t h e s i t u a t i o n . Yntema and Torgerson [84] have suggested a computer-aided d e c i s i o n system which would reduce the amount of time spent on r o u t i n e d e c i s i o n s as wel l as help e l i m i n a t e i n c o n s i s -t e n c i e s . They use an a n a l y s i s of va r iance technique to de r i ve the d e c i s i o n maker's u t i l i t y f u n c t i o n which i s then used as the d e c i s i o n r u l e by the computer. They found tha t "main e f f e c t s " (the l i n e a r p o r t i o n of the f u n c t i o n ) provided an e x c e l l e n t approx imat ion to the u t i l i t y f u n c t i o n i f u t i l i t y was monoton ica l l y i n c r e a s i n g in a l l c r i t e r i a , i . e . a higher c r i t e r i o n l e v e l i s always p r e f e r r e d to a lower one. In a l l of the s t u d i e s mentioned the decisions p r e d i c t e d with a l i n e a r u t i l i t y f u n c t i o n c o r r e l a t e d h igh l y w i th ac tua l d e c i s i o n s . - This i s not meant to suggest that the human d e c i s i o n process i s n e c e s s a r i l y l i n e a r . However a l i n e a r model has been 36 shown to be an e x c e l l e n t "paramorphic r e p r e s e n t a t i o n " [18] of d e c i s i o n makers in many s i t u a t i o n s . That i s , wh i le the d e c i s i o n process may not be l i n e a r , the general l i n e a r model i s powerful enough to reproduce most d e c i s i o n s wi th very l i t t l e e r r o r . Goldberg [39] has found that . . . for a number of d i f f e r e n t judgment tasks and across a considerable range of judges, the simple l i n e a r model appeared to characterize quite adequately the judg-mental processes involved - in s p i t e of the reports of the judges that they were using cues in a highly configural manner. ... Consequently if one's purpose is to repro-duce the responses of most judges, then a simple l i n e a r model w i l l normally permit the reproduction of 90-100% of t h e i r r e l i a b l e judgment variance, probably in most - if not a l l - c l i n i c a l judgment tasks. Since our purpose i n d e r i v i n g a u t i l i t y f u n c t i o n to help analyze the p r o j e c t s e l e c t i o n problem i s to reproduce d e c i s i o n s , i t would seem a p p r o p r i a t e to use an i n f e r r e d p r e f e r -ence t e c h n i q u e . From the s t u d i e s c i t e d one would expect a l i n e a r f u n c t i o n to p r e d i c t d e c i s i o n s w e l l . However, to a l low more f l e x i b i l i t y to model people and s i t u a t i o n s where i t i s not the most a p p r o p r i a t e form, one might wish to cons ider some of the n o n - l i n e a r f u n c t i o n s as w e l l . The f u n c t i o n s which are inc luded i n the I n t e r a c t i v e U t i l i t y Assessment Procedure (IUAP) developed are the l i n e a r , c o n j u n c t i v e , d i s j u n c t i v e , l o g a r i t h m i c and exponent ia l f u n c t i o n s , and a combinat ion of the c o n j u n c t i v e and d i s j u n c t i v e f u n c t i o n s . 37 The l i n e a r f u n c t i o n i s of the f o l l o w i n g form: n U = I a. X. , k=I K K where U i s the u t i l i t y of an a l t e r n a t i v e , X^ i s the l e v e l of c r i t e r i o n k achieved by the a l t e r n a t i v e , and a^ i s the c o e f f i c e n t found by the 1 inear r e g r e s s i o n procedure . In many cases i t seems to be t rue that wh i le u t i l i t y i s m o n o t o n i c a l l y i n c r e a s i n g wi th respect to each c r i t e r i o n , at some po int the incremental u t i l i t y of a u n i t i nc rease in c r i t e r i o n l e v e l dec reases . In other words the u t i l i t y f u n c t i o n e x h i b i t s decreas ing r e t u r n s . The s i m p l e s t method of mode l l ing the decreas ing re tu rns c h a r a c t e r i s t i c i s wi th a p iecewise l i n e a r concave f u n c t i o n . In that case the u t i l i t y of a u n i t of c r i t e r i o n k, up to the po in t i s a | . When the c r i t e r i o n l e v e l i s g reater than B^, each a d d i t i o n a l 2 1 2 1 u n i t i s worth a^ r a t h e r than a ^ , and a^ < a ^ . Thus . the u t i l i t y f u n c t i o n would be as f o l l o w s : n U = I a ' X, + k=l K K The l i n e a r forms are compensatory i n tha t a high l e v e l of one c r i t e r i o n can make up f o r a low l e v e l of another . The c o n j u n c t i v e u t i l i t y f u n c t i o n used by Goldberg [39] was o r i g i n a l l y proposed by Einhorn [ 2 7 ] : - a m i n X k ' B k 38 k=l where the are c o e f f i c i e n t s obta ined by app ly ing l i n e a r re-g r e s s i o n to the equation log U I a. log X, k=l In one dimension the f u n c t i o n i s a parabola•(see•Figure ' 2 . 2 ( a ) ) . A low l e v e l of any c r i t e r i o n would reduce the u t i l i t y of the a l t -e r n a t i v e r e g a r d l e s s of how high the other c r i t e r i o n l e v e l s . m i g h t The d i s j u n c t i v e f u n c t i o n , a l s o o r i g i n a l l y proposed by E i n h o r n , models the s t r a t e g y which chooses a l t e r n a t i v e s on the b a s i s of t h e i r "best " c r i t e r i o n l e v e l . The general form i s n u = n k=l where d^ i s the maximum p o s s i b l e value of c r i t e r i o n k. The co -e f f i c i e n t s a^ are determined by app ly ing l i n e a r r e g r e s s i o n to log U = -I a. log fd - X ] . k=i K i K K ) (a) F i g u r e 2.2 Con junc t i ve U t i l i t y F u n c t i o n s \ 40 In one dimension the f u n c t i o n i s a hyperbola (see F igure 2.3 ( a ) ) . The c l o s e r a c r i t e r i o n l e v e l i s to i t s maximum value d^ , the l a r g e r the f r a c t i o n V ( d ^ - X ^ ) and the l a r g e r the u t i l i t y ass igned to the a l t e r n a t i v e . When the c r i t e r i o n l e v e l i s high the f u n c t i o n prov ides a great deal of d i s c r i m i n a t o r y power ( i . e . the d e r i v a t i v e i s l a r g e ) . This corresponds to choosing between a l t e r n a t i v e s on the bas is of t h e i r best c r i t e r i o n l e v e l . The d i s j u n c t i v e form i s a l s o non-compensatory. I f an a l t e r n a -t i v e ' s best c r i t e r i o n l e v e l i s only average, then s i m i l a r average l e v e l s of other c r i t e r i a w i l l not make i t more a t t r a c t i v e than an a l t e r n a t i v e which has one very high score and the r e s t very low. The next two models, the l o g a r i t h m i c and the exponen-t i a l were suggested by Goldberg [39] to determine whether a l o g a r i t h m i c t r a n s f o r m a t i o n of the c r i t e r i o n l e v e l s ( l o g a r i t h m i c ) , or the value judgements (exponent ia l ) would prov ide a b e t t e r f i t to the observed u t i l i t y f u n c t i o n . The l o g a r i t h m i c form i s n n a k X k U = T a. log X, , and the exponent ia l form i s U = n e , k=l K K k = 1 n or log U = I a k X k . t i o n of Another n o n - l i n e a r f u n c t i o n to cons ider the c o n j u n c t i v e and d i s j u n c t i v e forms: i s a combina-(a) . F i g u r e 2 .3 D i s j u n c t i v e U t i l i t y Func t ions 42 u = n k=l n x k or log U = I a j l o g X k - log ( d k - X k ) n k=l where d^ i s the maximum p o s s i b l e value of c r i t e r i o n k and the a are c o e f f i c i e n t s found by app ly ing l i n e a r r e g r e s s i o n to the second form. A l t e r n a t i v e s which have a high l e v e l wi th respect to one c r i t e r i a are g iven a high u t i l i t y s i n c e X^ and 1/d^ - X^ would both b e . l a r g e . However, none of the other c r i t e r i a can be too low or the u t i l i t y would be reduced. c o e f f i c i e n t s f o r a l l of the forms are obta ined by app ly ing a l i n e a r r e g r e s s i o n technique to the r e s u l t s of a set of d e c i s i o n s . The d e c i s i o n maker i s f i r s t asked to s p e c i f y the lowest and h ighest p o s s i b l e values of each of the c r i t e r i a he wishes to c o n s i d e r . This i n f o r m a t i o n i s used to randomly generate a l t e r -na t i ves ( s e t s o f c r i t e r i o n l e v e l s ) . The d e c i s i o n maker i s then i n t e r a c t i v e l y asked to ra te the a l t e r n a t i v e s on an a r b i t r a r y s c a l e from 1 -100. One i s the u t i l i t y of an a l t e r n a t i v e which has the lowest p o s s i b l e values of a l l c r i t e r i a , and one hundred i s the u t i l i t y of an a l t e r n a t i v e w i th the h ighest p o s s i b l e values of a l l c r i t e r i a . These r a t i n g s r e f l e c t the d e c i s i o n maker's In the I n t e r a c t i v e U t i l i t y Assessment Procedure the 43 concept ion of the d i f f e r e n c e in value between such sets of c r i t e r i o n l e v e l s . They have no i n t r i n s i c . s i g n i f i c a n c e , and are meaningful only in r e l a t i o n to one another . The d e c i s i o n maker could j u s t as wel l have used a 100-300 s c a l e to ra te the a l t e r n a t i v e s . The a l t e r n a t i v e s are rated one at a time as they are generated . The d e c i s i o n maker i s then presented wi th the e n t i r e set of a l t e r n a t i v e s and may ad jus t h is r a t i n g s . A c o n s i s t e n c y check i s a p p l i e d to the set of r a t i n g s . Any a l t e r n a t i v e which dominates another (has h igher l e v e l s on a l l c r i t e r i a ) , and ye t has a lower r a t i n g , i s pointed o u t , and the d e c i s i o n maker i s asked to r e - e v a l u a t e i t . Once the r a t i n g s are i n t e r n a l l y con -s i s t e n t i . e . s a t i s f y the t r a n s i t i v i t y assumpt ion , and the d e c i s i o n maker i s s a t i s f i e d with them, they are used as the dependent v a r -i a b l e in a l i n e a r r e g r e s s i o n procedure , and the c o e f f i c i e n t s f o r the var ious tynes of u t i l i t y f u n c t i o n s are determined. S l o v i c and L i c h t e n s t e i n [71] d i scuss the problems that can a r i s e when the a l t e r n a t i v e s used i n the d e c i s i o n s i t u a t i o n s posed to de r i ve the u t i l i t y f u n c t i o n have randomly generated c r i t e r i a l e v e l s . I f the ass igned c r i t e r i a l e v e l s are u n r e a l i s t i c or o u t s i d e the range the d e c i s i o n maker i s accustomed to h a n d l -i n g , then he may not be able to a c c u r a t e l y ra te the a l t e r n a t i v e s . However, the technique used in the IUAP program, ass igns random values between the maximum and minimum p o s s i b l e values of each c r i t e r i o n . In that way each value i s r e a l i s t i c and the problem does not a r i s e . 44 Another problem occurs when the d e c i s i o n c r i t e r i a are not independent . In that case randomly generated c r i t e r i a l e v e l s could wel l v i o l a t e expected r e l a t i o n s between c r i t e r i a . For example, i f two of the c r i t e r i a were the amount of energy consumed and the inc rease in the p o l l u t i o n l e v e l , one would expect a high l e v e l . o f one c r i t e r i o n to be a s s o c i a t e d wi th a high l e v e l of the o t h e r . The random generat ion technique used here would not guarantee t h a t . In s i t u a t i o n s where the expected r e l a t i o n s h i p i s not observed , the d e c i s i o n maker e s s e n t i a l l y d i s regards one of the c o n f l i c t i n g c r i t e r i a . In cases where other c r i t e r i a support the hypothesis that one of the c r i t e r i o n l e v e l s i s wrong, t h i s i s a reasonable procedure. However at other t imes i t leads to i n c o n s i s t e n c y and u n r e l a b i T i t y . One method of o b v i a t i n g the problem would be to employ a r e j e c t i o n t e c h n i q u e . The generated a l t e r n a t i v e s could be examined before being presented to the d e c i s i o n maker, and r e j e c t e d i f the c r i t e r i o n l e v e l s v i o l a t e the expected r e l a t i o n s h i p s . Another cause of i n c o n s i s t e n c y i s the a v a i 1 a b i 1 i t y of a la rge number of c r i t e r i a . Einhorn [27] has shown that as the number of c r i t e r i a i n c r e a s e s , c o n s i s t e n c y decreases . One p o s s i b l e e x p l a n a t i o n of t h i s phenomenon i s that when the number of c r i t e r i a to be cons idered exce.eds the d e c i s i o n maker 's i n f o r m a t i o n p rocess ing c a p a c i t y , he chooses only a subset of them on which to base h is judgment. This s u b s e t , however, i s not the same each t i m e . This hypothes is i s strengthened by the f a c t that the number of c r i t e r i a that are s t a t i s t i c a l l y 45 s i g n i f i c a n t in d e c i s i o n s i s - g e n e r a l l y s m a l l e r than the d e c i s i o n maker would e s t i m a t e . He may remember us ing them a l l at one time or another , but the more important ones dominate. S l o v i c and L i c h t e n s t e i n f e e l that the most important c r i t e r i o n g e n e r a l l y accounts fo r 40% of a d e c i s i o n maker 's p r e d i c t a b l e v a r i a n c e , and the three most . important account f o r 80%. The number of c r i t e r i a that may be used wi thout causing s e r i o u s i n c o n s i s t e n c i e s appears to depend a great deal on the problem being cons idered and the r e l a t i o n s h i p between the c r i t e r i a . Whereas only the most important 4 or 5 are g e n e r a l l y s t a t i s t i c a l l y s i g n i f i c a n t , 10 are al lowed in the IUAP program and could probably be con -s idered before the d e c i s i o n maker would be faced wi th an " i n f o r m a t i o n o v e r l o a d " which i s assumed to lead to i n c o n s i s t e n c y . Of the many methods fo r handl ing m u l t i p l e c r i t e r i a c o n s i d e r e d , the i n f e r r e d preference technique fo r d e r i v i n g u t i l i t y f u n c t i o n s used by the IUAP program seems to be the most a p p r o p r i a t e in a n a l y z i n g the p r o j e c t s e l e c t i o n problem. Most of the s e q u e n t i a l e l i m i n a t i o n methods reduce the set of p o s s i b l e best a l t e r n a t i v e s , but the remaining ones must s t i l l be ordered us ing some other method. The r e s t r i c t i v e l o g i c of severa l of the methods and t h e i r non-compensatory nature make them i n a p p r o -p r i a t e in many c a s e s . They are h e u r i s t i c methods of handl ing m u l t i p l e - c r i t e r i a d e c i s i o n s i t u a t i o n s which are not based on a l o g i c a l s t r a t e g y . The s p a t i a l p rox imi t y methods are computa-t i o n a l l y more complex than the i n f e r r e d preference t e c h n i q u e . They are a l so more d i f f i c u l t f o r the d e c i s i o n maker to under -46 stand and use . T h e r e f o r e , from a p r a c t i c a l po in t of v iew, the i n f e r r e d preference technique i s more s u i t a b l e . The mathematical programming methods are a p p l i c a b l e to the p r o j e c t s e l e c t i o n problem s ince i t has cont inuous c r i t e r i o n l e v e l s . They have been s u c c e s s f u l in s o l v i n g severa l r e a l - w o r l d problems and might prove e f f e c t i v e in s o l v i n g the p r o j e c t s e l e c t i o n problem as w e l l . They are more d i f f i c u l t to implement than the i n f e r r e d preference technique however, and have not been used f o r that reason . The d i r e c t l y assessed preference techniques fo r d e r i v i n g u t i l i t y f u n c t i o n s are based on the assumption t h a t the u t i l i t y f u n c t i o n s f o r each c r i t e r i a are a d d i t i v e . The c o n d i t i o n s under which t h i s a d d i t i v i t y assumption holds may not be met in many p r o j e c t s e l e c t i o n d e c i s i o n s i t u a t i o n s and thus these methods are not always a p p r o p r i a t e . In fe r red preference techniques such as IUAP have been shown to be very s u c c e s s f u l in modeling d e c i s i o n makers and p r e d i c t i n g t h e i r p r e f e r e n c e s . Since t h i s modeling c a p a b i l i t y i s the major goal in developing a p r o j e c t s e l e c t i o n a l g o r i t h m , t h i s technique seems to be the a p p r o p r i a t e one to use. Chapter 3 A NEW PROJECT SELECTION ALGORITHM In any r a t i o n a l d e c i s i o n process i t i s necessary to eva luate outcomes in terms of g o a l s . In the l a s t chapter i t was suggested that the most appropr ia te e v a l u a t i o n method f o r the p r o j e c t s e l e c t i o n problem i s to der i ve a measure of the u t i l i t y of each a l t e r n a t i v e as a f u n c t i o n of i t s c r i t e r i o n l e v e l s . The most e f f i c i e n t and p r a c t i c a l method of d e r i v i n g a u t i l i t y f u n c t i o n in t h i s case i s the i n f e r r e d preference techn ique . This technique der i ves a u t i l i t y f u n c t i o n by asking the d e c i s i o n maker to reveal h is preferences through c h o i c e . There are severa l forms t h i s u t i l i t y f u n c t i o n may take but many s t u d i e s suggest tha t a l i n e a r form o f ten p r e d i c t s d e c i s i o n s b e s t . T h e r e f o r e , the l i n e a r fo rm, and a m o d i f i c a t i o n of i t , the p iecewise l i n e a r fo rm, were s e l e c t e d f o r the new d e c i s i o n a l g o r i t h m . 3.1 DERIVING A LINEAR UTILITY FUNCTION The I n t e r a c t i v e U t i l i t y Assessment Procedure (IUAP) f o r i d e n t i f y i n g a u t i l i t y f u n c t i o n i s used in the p r o j e c t 47 48 s e l e c t i o n a lgo r i thm to de r i ve the l i n e a r form. The d e c i s i o n maker . i s f i r s t asked by a c o n v e r s a t i o n a l subrout ine to. d e s c r i b e h is d e c i s i o n problem. How many p r o j e c t s are being considered? How many c r i t e r i a are r e l e v a n t ? What i s the suggested or requested l e v e l of funding f o r each p r o j e c t , and what i s the maximum and minimum funding l e v e l p o s s i b l e ? What l e v e l of each c r i t e r i o n would each p r o j e c t achieve (assuming i t i s s u c c e s s f u l and funded at the requested l e v e l ) ? This i n f o r m a t i o n i s used to determine the maximum and minimum value of each cr i t e r i on , . X m a . x and X m k n . Then, sets of c r i t e r i o n l e v e l s between these bounds are randomly generated . These sets of c r i t e r i o n l e v e l s can be thought of as " p s e u d o - p r o j e c t s . " S ince each p r o j e c t can be funded at an i n f i n i t e number of l e v e l s between i t s upper and lower funding bounds, a combination of p r o j e c t s funded wi th var ious amounts might r e s u l t in a set of c r i t e r i o n l e v e l s wi th the same values as the p s e u d o - p r o j e c t . Each pseudo -p ro jec t i s t h e r e f o r e rep -r e s e n t a t i v e of a p o s s i b l e research and development program ( p o r t f o l i o of p r o j e c t s ) . At t h i s po in t in the a l g o r i t h m , IUAP i s used to i n t e r a c t i v e l y i l l i c i t preference r a t i n g s and d e t e r -mine r e g r e s s i o n c o e f f i c i e n t s f o r the l i n e a r form. The next step in the procedure i s to determine how we l l the l i n e a r model p r e d i c t s the d e c i s i o n maker's r a t i n g s . Three d i f f e r e n t c o e f f i c i e n t s of the c o r r e l a t i o n between the F igure 3.1 i s . a f low chart of the a l g o r i t h m . Describe the P r o j e c t s Double the number of pseudo-projects to be generated at each stage.  Have you already derived a| u t i l i t y function? N Is your u t l l i t y f u n c t i o n l i n e a r ? Generate pseudo-projects, rate them. PWL3 Formulate and solve with the l i n e a r form. STOP. Apply regression to f i n d c o e f f i c i e n t s , c a l c u l a t e c o r r e l a t i o n s . Do you wish to f i n d a b e t t e r f i t t i n g u t i l i t y f u n c t i o n ? . N 13 any c o r r e l a t i o n greater than 0.9? HAva you already attempted to de r i v e the piecewiss 1inear form? Have you found and replaced the worst point max(3,4m/5) times? ±1 Find the worst point. Is t h i s worst point near any other? Is t h i s the f i r s t t i n e the worst point has been found? -y-N Generate a new pseudo-project i n the neighborhood of the worst point and replace the worst point with I t . Figure 3.1 Flowchart of the A lgor i thm Are a l l of tho c r i t e r i o n l e v e l s of tho worst point i n the middle of the range of possible values? Use i t s c r i t e r i o n l e v e l s as the breakpoints. Generate new pseudo-projects, rate them, f i n d -the c o e f f i c i e n t s of the pieoewise l i n e a r form by r e g r e s s i o n , c a l c u l a t e the c o r r e l a t i o n s . -Is the piecewise l i n e a r u t i l i t y f u n c t i o n concave? Hive i; th« points been tested? N Find the next worst point, i . e . the point with the next l a r g e s t d i f f e r e n c e between a c t u a l and predicted scores. Are any of the c o r r e l a t i o n s f o r the• piecewise l i n e a r form b e t t e r than the corresponding ones f o r the l i n e a r form? Is the piecewise l i n e a r f u n c t i o n accurate enough? PWL2 Foraulate and solve with the piecewise l i n e a r f o r a . STOP. Figure 3.1 (cont inued) 51 actua l and p r e d i c t e d r a t i n g s are c a l c u l a t e d by the program; the Pearson or product moment c o r r e l a t i o n , the Spearman or rank c o r r e l a t i o n , and a d i s t a n c e - e r r o r c o r r e l a t i o n . S ince these c o r r e l a t i o n s are c a l c u l a t e d on the same data used to de r i ve the r e g r e s s i o n c o e f f i c i e n t s one would expect them to be h i g h . At t h i s po in t a new set of pseudo -p ro jec ts could be generated and r a t e d . . The c o r r e l a t i o n s between the actua l and p r e d i c t e d r a t i n g s of t h i s set would be a b e t t e r i n d i c a t o r of the goodness of f i t of the l i n e a r model. However, another set of r a t i n g s f o r t h i s purpose seems to p lace an unnecessary burden on the d e c i s i o n maker. A c o r r e l a t i o n l e v e l of 0.9 was a r b i t r a r i l y chosen as a s t a n d a r d . I f any of the c o r r e l a t i o n s are g reater than 0.9 then the l i n e a r model w i l l probably f i t we l l enough to be u s e f u l . However, the d e c i s i o n maker i s g iven the opt ion to cont inue the process in the hope of f i n d i n g a b e t t e r f i t t i n g u t i l i t y f u n c t i o n . I f none of the c o r r e l a t i o n s are g reate r than 0 . 9 , or i f the d e c i s i o n maker e x e r c i s e s his opt ion to c o n t i n u e , then a p i e c e -wise l i n e a r u t i l i t y f u n c t i o n i s der i ved in an attempt to improve the correspondence between choices and p r e d i c t i o n s . 3.2 DERIVING A PIECEWISE LINEAR UTILITY FUNCTION If the d e c i s i o n maker's u t i l i t y f u n c t i o n i s best approximated by a p iecewise l i n e a r f u n c t i o n , we would expect the 52 c o n f i g u r a t i o n of pseudo -p ro jec ts to resemble one of the cases dep ic ted in F igure 3 . 2 . The pseudo -p ro jec t wi th the most poor ly p r e d i c t e d score i s found by computing the abso lu te value of the d i f f e r e n c e between the p r e d i c t e d score and the ac tua l score fo r a l l p s e u d o - p r o j e c t s , and i d e n t i f y i n g the l a r g e s t such d i f f e r e n c e d . To e l i m i n a t e the p o s s i b i l i t y that t h i s d e v i a t i o n i s s imply an e r ro r on the part of the d e c i s i o n maker, he i s asked to ra te another pseudo -p ro jec t in the same neighbour -hood as the worst p o i n t . The c r i t e r i o n l e v e l s of t h i s new po in t are set at randomly generated values between the two adjacent c r i t e r i o n l e v e l s . This new pseudo -p ro jec t r e p l a c e s the most poor ly f i t t e d one and the l i n e a r r e g r e s s i o n procedure i s r e -a p p l i e d us ing the. new set of r a t i n g s as dependent v a r i a b l e s . Once again the pseudo -p ro jec t wi th the most poor ly p r e d i c t e d score i s found. This c y c l e i s repeated u n t i l one of three The adjacent c r i t e r i o n l e v e l s are found by examining each pseudo -p ro jec t to f i n d the g rea tes t c r i t e r i o n l e v e l l e s s than the cor responding one f o r the worst p o i n t , and the s m a l l e s t c r i t e r i o n l e v e l g reater than the cor responding one f o r the worst p o i n t , f o r a l l c r i t e r i a . Another method of generat ing a new pseudo -p ro jec t in the neighbourhood of the worst po in t would be to cons ide r a c i r c l e of rad ius r about the p o i n t . C r i t e r i o n l e v e l k of the new po in t would then be generated at random W P W P W P from the i n t e r v a l (X k - r , X k + r ) , where X k i s the c r i t e r i o n l e v e l , of the worst p o i n t . 53 3 th ings happens. 1.) The l a s t worst po int found i s near one of the prev ious worst p o i n t s . In that case an area has been i d e n t i f i e d where the l i n e a r form does not p r e d i c t w e l l , and one might suspect that a p iecewise l i n e a r f u n c t i o n would be more a p p r o p r i a t e . 2 . ) The c o r r e l a t i o n between the ac tua l and p r e d i c t e d r a t i n g s improves enough that the d e c i s i o n maker decides the l i n e a r form i s adequate. 3 . ) The i t e r a t i o n counter k becomes g reate r than max ( 3 , 4 m / 5 ) , where m i s the number of c r i t e r i a being c o n s i d e r e d . In t h i s c a s e , a f t e r a " reasonab le" number of i t e r a t i o n s , no area has been found where the l i n e a r form c o n s i s t e n t l y p r e d i c t s r a t i n g s p o o r l y . One may conclude t h e r e f o r e , that the d e c i s i o n maker's u n d e r l y -ing u t i l i t y f u n c t i o n i s not p iecewise l i n e a r and that a p i e c e -wise l i n e a r r e p r e s e n t a t i o n of i t would not l i k e l y p r e d i c t r a t i n g s b e t t e r than the l i n e a r form. If case 1.) occurs then the next s t e p , in the a l g o r -ithm i s to f i n d the breakpo ints B k of the p iecewise l i n e a r form m U = £ ' [ a 2 X k +.. (a* - a k ) m i n ( X k , B k ) ] . k ~ 1 k The b reakpo in ts are where the phenomenon of decreas ing re turns a 3 T h e new w o r s t p o i n t i s n e a r t h e l a s t w o r s t p o i n t i f 11 o f i t s c r i t e r i o n l e v e l s f a l l i n t h e i n t e r v a l / v m a x v m i n ) / v m a x Y m i n \ \ X r. " X I. V * \, ~ L- I \ K - k k , x k • U k -k . N K N , w h e r e X. i s t h e k*"^1 k c r i t e r i o n l e v e l o f t h e o l d w o r s t p o i n t a n d N i s t h e t o t a l n u m b e r o f p s e u d o - p r o j e c t s . g e n e r a t e d . 54 o c c u r s , i . e . where the marginal value of a u n i t inc rease in the c r i t e r i o n l e v e l dec reases . For c r i t e r i o n l e v e l s l e s s than the b r e a k p o i n t , the added value of one u n i t of c r i t e r i o n k 1 i s a k . When the c r i t e r i o n l e v e l i s g reater than B^, the added 2 2 1 value of one u n i t i s a^ and a^ < a ( < -One p o s s i b l e method of o b t a i n i n g the breakpoints i s to ask the d e c i s i o n maker f o r them. He may be aware of a d i s c o n t i n u i t y in h is u t i l i t y f u n c t i o n wi th respect to one or more c r i t e r i a , and may be able to i n d i c a t e where i t o c c u r s . In most ins tances however, t h i s w i l l not be the c a s e . The computerized procedure f i n d s the breakpoints by examining the l a s t worst po int found. If a l l of i t s c r i t e r i o n l e v e l s are i n the middle h a l f of the range of p o s s i b l e values of that c r i t e r i o n , 2 5 7 5 i . e . between and X^ in F igure 3 . 2 , then case (a) in F i g -u re . 3 . 2 i s a p p l i c a b l e . The c r i t e r i o n l e v e l s , of the po int are then taken as the breakpoints B^ f o r each c r i t e r i o n . I f a l l of the c r i t e r i o n l e v e l s of t h i s worst po int are not in the middle h a l f of the range of p o s s i b l e values of the c r i t e r i o n , then case (b) or (c) of F igure 3.2 may a p p l y . In t h i s case the c r i t e r i o n l e v e l s of t h i s po int cannot be used as the b r e a k p o i n t s . Some other poor ly f i t t e d pseudo -p ro jec t wi th c r i t e r i o n l e v e l s c l o s e r to the middle of the range must be found. The pseudo-p r o j e c t s are search ing in decreas ing order of d (d i s tance be-tween ac tua l and p r e d i c t e d r a t i n g s ) u n t i l one s a t i s f y i n g the "middle of the range" r u l e i s found ( in which case i t s c r i t e r i o n l e v e l s are used as the b r e a k p o i n t s ) , or u n t i l h a l f of the 55 (c) Figure 3.2 P iecewise L inear U t i l i t y Funct ions p s e u d o - p r o j e c t s have been examined. If the most poor ly p r e d i c t e d h a l f of the p s e u d o - p r o j e c t s does not c o n t a i n a po in t in the middle of the range.,- then the d i s t r i b u t i o n of sample po in ts does not resemble . case (b) or (c) of F igure 3 . 2 , and i t i s u n l i k e l y that a p iecewise 56 l i n e a r form would be a more accurate r e p r e s e n t a t i o n of the d e c i s i o n maker's ac tua l u t i l i t y f u n c t i o n , than the l i n e a r fo rm. Once the breakpoints of the p iecewise l i n e a r form have been determined, the next step i s to f i n d the c o e f f i c i e n t s 1 2 a k and a k , . k = l , . . . m . This i s done by app ly ing r e g r e s s i o n to m *C 1 1 2 2 u = z a k x k + a k x k . where = min (X^, B^) and X k 2 = max (X k - B k , 0 ) . Since twice as many c o e f f i c i e n t s as before are being c a l c u l a t e d , another 4m pseudo -p ro jec ts are generated and i n t e r a c t i v e l y rated before the r e g r e s s i o n technique i s a p p l i e d . The next step in the procedure i s to determine which of the two forms (the l i n e a r or the p iecewise l i n e a r ) f i t s b e s t . The c o r r e l a t i o n c o e f f i c i e n t s between the ac tua l r a t i n g s given the o r i g i n a l set of pseudo -p ro jec ts by the d e c i s i o n maker, and the r a t i n g s p r e d i c t e d by the p iecewise l i n e a r form are c a l c u l a t e d . These c o r r e l a t i o n c o e f f i c i e n t s are compared with the corresponding ones fo r the l i n e a r form. The l i n e a r form has an u n f a i r advantage in t h i s comparison s ince i t s c o r r e l a t i o n c o e f f i c i e n t s are c a l c u l a t e d on the same data as was used to de r i ve i t s r e g r e s s i o n c o e f f i c i e n t s . Here again another set of pseudo -p ro jec ts could be generated and used 57 to t e s t the two models more f a i r l y . But the i m p o s i t i o n of a t h i r d set of r a t i n g s was found to be unacceptable to many d e c i s i o n makers. If any of the c o r r e l a t i o n c o e f f i c i e n t s fo r the p iecewise l i n e a r form are g reater than the corresponding ones f o r the l i n e a r fo rm, d e s p i t e the b ias in favour of the l i n e a r fo rm, then the p iecewise l i n e a r form would seem to f i t b e s t . I f a l l of the c o e f f i c i e n t s f o r the l i n e a r form are g reate r than the ones fo r the p iecewise l i n e a r fo rm, then the l i n e a r form may or may not prov ide a be t te r f i t . However, f o r lack of another method of measuring the f i t of the two forms, the l i n e a r one would be chosen in that case . If the d e c i s i o n maker i s s t i l l not s a t i s f i e d with the f i t of the chosen u t i l i t y f u n c t i o n the whole procedure could be repeated . This time more pseudo -p ro jec ts would be generated at each stage i n - o r d e r to f i n d a b e t t e r f i t t i n g u t i l i t y f u n c t i o n . The quest ion of how many pseudo -p ro jec ts to generate at each stage i s an important one. The more r a t i n g s a v a i l a b l e the b e t t e r the f i t of the u t i l i t y f u n c t i o n . However, i f too many r a t i n g s are r e q u i r e d , the d e c i s i o n maker w i l l become t i r e d and/or annoyed wi th a r e s u l t i n g d e c l i n e in accuracy . There must be an optimum number of pseudo -p ro jec ts to generate which would minimize these two types of problems. C l e a r l y t h i s number should be r e l a t e d in some way to the number of c r i t e r i a used. As the number of c r i t e r i a (and, i n a sense , degrees of freedom in the u t i l i t y f u n c t i o n ) i n c r e a s e , more po in ts w i l l be needed f o r a good f i t . The number of pseudo-p r o j e c t s to generate was set at 4m, when m i s the number of 58 c r i t e r i a . If 5 c r i t e r i a were cons idered r e l e v a n t , which i s more than the number most s t u d i e s found to be s t a t i s t i c a l l y s i g n i f i c a n t in i n f l u e n c i n g r a t i n g s , then 20 pseudo -p ro jec ts would be generated . Twenty was found to be a reasonably la rge number of r a t i n g s with which most d e c i s i o n makers-can cope. The number of c r i t e r i a i s c e r t a i n l y not l i m i t e d to 5 , however. I f the d e c i s i o n maker wishes to use more, then he must ra te a co r respond ing l y l a r g e r number of p s e u d o - p r o j e c t s . When the p iecewise l i n e a r form i s being d e r i v e d , another set of 4m pseudo -p ro jec ts i s generated . The quest ion a r i s e s as to whether the d e c i s i o n maker can rate t h i s new set of pseudo -p ro jec ts so that the r a t i n g s are c o n s i s t e n t wi th the ones f o r the prev ious s e t . Since the type of d e c i s i o n maker t h i s procedure i s intended to model i s one who i s f a m i l i a r enough with h is problem and h is preferences so that h is v iew-po in t would not change as a consequence of the e x e r c i s e , he i s cons idered to be capable of c o n s i s t e n c y . As an a ide to a c h i e v i n g c o n s i s t e n c y the prev ious set of pseudo -p ro jec ts and r a t i n g s are presented to him as re fe rence p o i n t s . 2 A t h i r d quest ion which a r i s e s i s why a^ should be l e s s than a ^ . The r e g r e s s i o n procedure c e r t a i n l y does not 2 guarantee t h i s . I f in f a c t i t turns out that a^ i s g reater than a ^ , then the d e c i s i o n maker's responses do not s a t i s f y the assumption of decreas ing re turns with respect to c r i t e r i o n k (see F igure 3 . 3 ) . Since our s o l u t i o n procedure depends on c o n c a v i t y , and i t would seem u n l i k e l y that a d e c i s i o n maker's 59. u t i l i t y f u n c t i o n should e x h i b i t i n c r e a s i n g re tu rns when d e a l i n g with r e a l problems i f t h i s s i t u a t i o n o c c u r s , then the p i e c e -wise l i n e a r form i s d i sca rded and the l i n e a r form chosen. Another method of d e a l i n g with the problem would be to r e - a p p l y the r e g r e s s i o n procedure to a new form which i s p iecewise l i n e a r wi th respect to only those c r i t e r i a which •sat i s f y the decreas ing re tu rns assumpt ion , i . e . <• ^ 1 1 2 2. U = I a k X k + £ a k X k + a k X. k<D K K keD K K K K where D i s the set of those c r i t e r i a which e x h i b i t decreas ing r e t u r n s . 3 . 3 FORMULATING THE OBJECTIVE FUNCTION Once the a p p r o p r i a t e u t i l i t y f u n c t i o n has been chosen i t can be combined wi th the d e c i s i o n maker's knowledge of the c r i t e r i o n l e v e l s of each of the p r o j e c t s under c o n s i d e r a t i o n , to c o n s t r u c t an o b j e c t i v e f u n c t i o n f o r use in a mathematical programming a l g o r i t h m . Consider f i r s t the l i n e a r form. The t o t a l u t i l i t y of any p o r t f o l i o of p r o j e c t s i s m U = I a k k=l K where X k^ i s the l e v e l of c r i t e r i o n k that p r o j e c t j a c t u a l l y a c h i e v e s . X ^ v a r i e s wi th the amount of funds a l l o c a t e d to 60 p r o j e c t j and cannot be known e x a c t l y u n t i l a f t e r the p r o j e c t i s completed. However, the d e c i s i o n maker has some s u b j e c t i v e est imate of the p r o b a b i l i t y that p r o j e c t j w i l l be s u c c e s s f u l when funded at var ious l e v e l s . The expected value of i s then x|<jPj ( x j ) » where X k j . i s the l e v e l of c r i t e r i o n k that the d e c i s i o n maker s p e c i f i e s in h is d e s c r i p t i o n of the p r o j e c t , and P. ( x . ) i s the p r o b a b i l i t y that p r o j e c t j w i l l succeed when funded at x . . The expected u t i l i t y of any research and 3 development program i s then (j, hi w] • Souder [77 ,78] has s t u d i e d the e f f e c t on r e s u l t i n g d e c i s i o n s of three d i f f e r e n t forms of the P - ( x . ) f u n c t i o n . J J The three forms he chose were-a l i n e a r f u n c t i o n , a p iecewise l i n e a r f u n c t i o n and a n o n - l i n e a r f u n c t i o n ( e i t h e r exponent ia l or S - s h a p e d ) . These f u n c t i o n s were der i ved by f i t t i n g curves "through po in ts s p e c i f i e d by the d e c i s i o n maker (see F igure 3 . 4 ) . He found that the l i n e a r , f u n c t i o n r e s u l t e d in d e c i s i o n s which maximized p r o f i t , but that the p iecewise l i n e a r f u n c t i o n was most o f ten p r e f e r r e d because i t had other d e s i r a b l e p r o p e r t i e s . "The p iecewise model was more f r e q u e n t l y p r e f e r r e d than the other two fo rms , l a r g e l y because of i t s a b i l i t y to s e l e c t compromise p o r t f o l i o s . The 'compromise' p o r t f o l i o s were those m E(U) = I a k=l j r Piecewise L i n e a r F u n c t i o n N o n - l i n e a r F u n c t i o n s E x p o n e n t i a l and S-shaped A l l three forms are d e r i v e d by a s k i n g the d e c i s i o n maker f o r the p r o b a b i l i t y p r o j e c t j w i l l succeed i f funded at the v a r i o u s l e v e l s . F i g u r e J.k P r o b a b i l i t y of Success F u n c t i o n s 62 y i e l d i n g acceptab le a n t i c i p a t e d p r o f i t s w h i l e s t i l l m a i n t a i n -ing minimum fund ing ' on imminent f a i l u r e outcome p r o j e c t s and p r o v i d i n g a balance of i n t r i n s i c p o r t f o l i o a t t r i b u t e s " [ 7 8 ] , The p iecewise l i n e a r f u n c t i o n i s more s u i t a b l e fo r m u l t i p l e c r i t e r i o n s i t u a t i o n s because of t h i s compromise c h a r a c t e r i s t i c . In that phase of the procedure where the d e c i s i o n maker i s asked to d e s c r i b e the p r o j e c t s he i s c o n s i d e r i n g , he must s p e c i f y the requested or suggested l e v e l of funding f . , vJ and the upper and lower funding bounds u. and £. r e s p e c t i v e l y , J J f o r each p r o j e c t j . These bounds may be set by o r g a n i z a t i o n a l p o l i c y , or they may be s imply l i m i t s which seem reasonable to the d e c i s i o n maker. The upper bound might be the po in t where the d e c i s i o n maker f e e l s inc reased funding would not s i g n i f i -c a n t l y i n c r e a s e the p r o b a b i l i t y of s u c c e s s . The lower bound might be such that i f the p r o j e c t were to be funded at an amount below i t , there would be v i r t u a l l y no p o s s i b i l i t y of s u c c e s s . In a d d i t i o n to s p e c i f y i n g these funding l e v e l s , the d e c i s i o n maker must est imate the p r o b a b i l i t y of success of each p r o j e c t when funded at i t s upper and lower bounds. The prob-a b i l i t y of success of p r o j e c t j when funded at l e v e l x . , (x . = 0 , or £ . ^ x . < u . ) - i s then W = V J + ( 3J - o j J yj 63 w h e r e . P - ( x . ) i s the p r o b a b i l i t y p r o j e c t j w i l l succeed when i t i s funded at 3 . i s the p r o b a b i l i t y i t w i l l succeed when 3 3 funded at u . , z . i s a 0 or 1 v a r i a b l e which i n d i c a t e s whether 3 3 or not p r o j e c t j i s funded at a l l ( i . e . x . i s at l e a s t ' A . ) , and y . i s a v a r i a b l e between 0 and 1 which i s the amount of funding above I. a l l o c a t e d to p r o j e c t j . Using t h i s n o t a t i o n , 3 x . = £ . z . + ( u . - £ . ) y . , • J J J J J J and m E(U) = I a k=l. This i s the express ion which w i l l be used as the o b j e c t i v e f u n c t i o n i n the mathematical programming a l g o r i t h m . 3.4 FORMULATING THE CONSTRAINTS The next step i s to determine the c o n s t r a i n t s w i t h i n which the d e c i s i o n maker must work. These are t y p i c a l l y con -s t r a i n t s on resources such as money and manpower. The d e c i s i o n maker must s p e c i f y the amount of each resource i used by each p r o j e c t j when funded at i t s requested funding l e v e l f . , denoted by S.JJ , and the t o t a l amount of each resource a v a i l a b l e , R^  . The resource c o n s t r a i n t s may then be formulated as j = l kj ( AJ ZJ + ( 3 . a j } V 64 n n < B or j = I n < R or I n < R. where i s the amount of resource i r equ i red per d o l l a r i n -vested i n p r o j e c t j and B i s the t o t a l budget. There i s an assumption i m p l i c i t in t h i s f o r m u l a t i o n of the resource c o n s t r a i n t s . The amount of resource i r e q u i r e d by p r o j e c t j i s assumed to vary l i n e a r l y wi th the amount of funding x . . This i s a f a i r l y reasonable assumption f o r most types of resources when x . f a l l s i n the range C £ . , u - ] . This range i s where changes i n funding r e s u l t in s i g n i f i c a n t changes i n the p r o b a b i l i t y of success and thus in the amount of each a t t r i b u t e produced and each resource consumed. Outs ide of t h i s range changes in funding may not be r e f l e c t e d by changes i n resource requ i rements . I t would seem reasonable that the consumption of resources such as energy and manpower, which are e s s e n t i a l l y cont inuous (can be purchased in any amounts) would vary l i n e a r l y wi th funding in t h i s range. However there are some types of resources which are zero -one i n n a t u r e . For example, i f a p iece of e l a b o r a t e equipment i s necessary (such as a computer or a c y c l o t r o n ) , t h r e e - q u a r t e r s of the machine would not be p a r t i c u l a r l y u s e f u l . In cases l i k e t h i s 65 t h e n , the assumption i s not v a l i d . . Most resources cons idered i n p r o j e c t s e l e c t i o n problems however, are of the cont inuous t y p e . Other types of c o n s t r a i n t s are p o s s i b l e as w e l l . There may be r e l a t i o n s h i p s among the p r o j e c t s such that i f one p r o j e c t or group of p r o j e c t s i s funded, then others may not be. For example, a government agency may be concerned tha t no more than N p r o j e c t s from each reg ion of the country are funded. In that case the necessary set of c o n s t r a i n t s are • I • z . < N V I , J £ J £ J where i s a set which indexes a l l p r o j e c t s submitted from region I. Another p o s s i b l e r e l a t i o n s h i p among. p r o j e c t occurs when one p r o j e c t depends on r e s u l t s from another . In that case the dependent p r o j e c t A should not be funded unless the independent p r o j e c t B i s a l so funded. The c o n s t r a i n t < z g w i l l i n s u r e that p r o j e c t A i s not funded (z^ = 0) i f p r o j e c t B i s not funded ( z g = 0 ) . Other types of r e l a t i o n s h i p s may be envisaged and can be r e a d i l y i n c l u d e d in the c o n s t r a i n t s e t . Another usefu l type of c o n s t r a i n t would be to s p e c i f y minimum l e v e l s of some or a l l c r i t e r i a which must be achieved by t h e ' p o r t f o l i o -of. p r o j e c t s . These c o n s t r a i n t s would have the fo l1owi ng form: 66 1. (a.Z. + (g. j = l . J J 3 where M k i s the minimum acceptab le l e v e l of c r i t e r i o n k. A government agency might wish to d e f i n e such a set of min -imum standards in order to ensure approval of a l l t h e i r funding programs. 3 .5 THE FORMULATION WITH A LINEAR UTILITY FUNCTION The complete f o r m u l a t i o n of the p r o j e c t s e l e c t i o n problem wi th a l i n e a r u t i l i t y f u n c t i o n i s then as f o l l o w s : m n max J a k £ k=l K j=l a . z . + ( 3 . - a . ) y . J J J J J kj s . t . I j = l l.z . + (u . - I.) y . J J J J J (1) n S . . £ . z . + (u . - £ . ) y . J J J J J < R- Vi (2) y . z . J3 J VJ O ) C o n s t r a i n t s (3) insure that p r o j e c t j i s funded at l e a s t at i t s lower bound i f i t i s funded at a l l . other c o n s t r a i n t s 67 (4) z . = 0 or 1 , and 0 < y . < 1 J j Vj (5) The p r o j e c t s e l e c t i o n problem may now be so lved using a mixed i n t e g e r branch and bound a l g o r i t h m . 3.6 THE FORMULATION WITH A PIECEWISE LINEAR UTILITY FUNCTION If the p iecewise l i n e a r u t i l i t y f u n c t i o n i s chosen as the one which p r e d i c t s the d e c i s i o n maker's r a t i n g s b e s t , then the o b j e c t i v e f u n c t i o n in t h i s f o r m u l a t i o n must be changed. In that c a s e , the t o t a l u t i l i t y of any p o r t f o l i o of p r o j e c t s i s m U = I k=l n ak _I] X k j + (a* - a ^ ) m i n [ _ ^ X k j , B n , where X k j . i s the l e v e l of c r i t e r i o n k a c t u a l l y achieved by p r o j e c t j . Again t h i s may be t r a n s l a t e d in to expected u t i l i t y using the two p r o b a b i l i t y of success est imates prov ided by the d e c i s i o n maker. E(U) = I k=l i hi J = <ak - » k > " 1 n [ I, x k j <VJ + ( 6 j - a i ] y j ' • B ' 68 In order to convert t h i s o b j e c t i v e f u n c t i o n i n t o a usefu l fo rm, two new sets of v a r i a b l e s are r e q u i r e d ; t^ and s^ t^ i s the t o t a l expected l e v e l of c r i t e r i o n k and s^ i s the amount by which t h i s l e v e l exceeds the breakpoint B^, i f any. Now the problem may be formulated as f o l l o w s : max I a , t k=l k"k s . t . t. = I a j Z j + ( 3 j " a j } y j V k (1) J V a .z . . + (B . - a . ) y . , X . , - B. jk k V k (2) I l . z . + (u . - £ . ) y , . J J J J J J (3) 69 n S "I J I t 4 j= i J Jt.z . + (u . - I.) y , J J J J J J < n. Vi (4) y j < Z j Vj (5) other c o n s t r a i n t s (6) z . = 0 , 1 and 0 < y . < 1 VJ (7) t k , s k > 0 Vk (8) C o n s t r a i n t s ( 3 ) - ( 7 ) are the same as the c o n s t r a i n t s in the prev ious f o r m u l a t i o n . C o n s t r a i n t s (1) set t k to the expected l e v e l of c r i t e r i o n k and c o n s t r a i n t s (2) set s k to the d i f f e r e n c e between the l e v e l of c r i t e r i o n k and the breakpoint B k > I f t h i s d i f f e r e n c e i s n e g a t i v e , then s k = 0 by ( 8 ) . s k w i l l never be ass igned a value g reate r than the excess of c r i t e r i o n l e v e l k over B k because the c o e f f i c i e n t of s k i n the o b j e c t i v e f u n c t i o n 1 2 i s negat ive (by concav i t y a k > a k ) . T h e r e f o r e , o p t i m a l i t y w i l l f o r c e s k to be as smal l a non -negat ive value as p o s s i b l e . 70 This f o r m u l a t i o n of the p r o j e c t s e l e c t i o n problem can a l s o be so lved by a branch and bound mixed i n t e g e r a l g o r i t h m . The r e s u l t of app ly ing the a l g o r i t h m i s the funding l e v e l of each p r o j e c t f o r the f i v e best s o l u t i o n s to the problem. The d e c i s i o n maker i s not s imply presented wi th the opt imal s o l u -t i o n f o r two reasons . One i s that the data he prov ides and the o b j e c t i v e f u n c t i o n that i s der i ved conta in e r r o r s and i n -a c c u r a c i e s . There i s no guarantee then that the " o p t i m a l " s o l u t i o n i s r e a l l y o p t i m a l . I t should have no more s i g n i f i c a n c e than another s o l u t i o n whose value i s very c l o s e . These ideas w i l l be i n v e s t i g a t e d more thoroughly i n the next c h a p t e r . The second reason i s that there may be other c o n s i d e r a t i o n s such as p o l i t i c a l ones which could not be inco rpora ted i n t o the u t i l i t y f u n c t i o n or c o n s t r a i n t s e t . I f the d e c i s i o n maker i s presented wi th a range of s o l u t i o n s wi th near " o p t i m a l " v a l u e s , then he may choose between them on the bas is of such c o n s i d e r a t i o n s wi thout s e r i o u s l y a f f e c t i n g performance. The p r o j e c t s e l e c t i o n problem, when viewed as a m u l t i p l e o b j e c t i v e d e c i s i o n problem, can be so lved by the i n t e r -a c t i v e procedure descr ibed above. The d e c i s i o n maker i s asked to ra te sets of pseudo -p ro jec ts in order to der i ve a measure of the u t i l i t y of a l t e r n a t i v e s as a f u n c t i o n of t h e i r c r i t e r i o n l e v e l s . This u t i l i t y f u n c t i o n i s combined wi th p r o b a b i l i t y of success est imates provided by the d e c i s i o n maker, to produce an express ion f o r the expected u t i l i t y of any research and 71 development program which i s used as the o b j e c t i v e f u n c t i o n in a .mathematical programming a l g o r i t h m . When t h i s o b j e c t i v e f u n c t i o n i s combined wi th c o n s t r a i n t s formulated wi th the data prov ided by the d e c i s i o n maker, the problem can. be so lved wi th a branch and bound mixed in teger a l g o r i t h m . The usefu lness of the r e s u l t i n g s o l u t i o n s and some of the ways the procedure may be extended and improved are d i scussed in the next chapte r . Chapter 4 THE UTILITY AND ACCEPTABILITY OF THE ALGORITHM Most p r o j e c t s e l e c t i o n a lgor i thms proposed to date have rece ived l i t t l e acceptance from those invo l ved i n the f u n d -ing of research and development p r o j e c t s . Severa l authors have attempted to e x p l a i n t h i s phenomenon. Baker and Pound [5] suggest three reasons f o r i t : (1) the u n r e a l i s t i c f e a t u r e s and assumptions i m p l i c i t i n many of the models, (2) the onerous data requirements of most of the a l g o r i t h m s , and (3) the lack of comprehensive t e s t i n g and repor ted computat ional exper ience with rea l problems. 4.1 FEATURES OF THE ALGORITHM The a l g o r i t h m suggested here i s based on a more r e a l i s t i c model of the p r o j e c t s e l e c t i o n d e c i s i o n s i t u a t i o n . In p a r t i c u l a r p r o f i t maximizat ion i s not assumed to be the only o b j e c t i v e . For many o r g a n i z a t i o n s such as government fund g r a n t i n g a g e n c i e s , t h i s o b j e c t i v e i s i r r e l e v a n t . In other cases the o r g a n i z a t i o n i s i n t e r e s t e d i n ' p r o f i t but not to the e x c l u s i o n of other goal 72 73 d imens ions . Often once a reasonable re tu rn on investment i s ensured , other c r i t e r i a are t r i g g e r e d . The a l g o r i t h m suggested here cons iders a l l c r i t e r i a , .in the manner and to the extent they are r e f l e c t e d i n the d e c i s i o n maker's judgements. This i s in c o n s t r a s t to other m u l t i - c r i t e r i a p r o j e c t s e l e c t i o n a l g o r i t h m s , such as s c o r i n g type models , which s p e c i f y p re -determined methods of combining goal d imens ions . Most other p r o j e c t s e l e c t i o n a lgor i thms assume the cost of a p r o j e c t i s f i x e d . The p r o j e c t i s e i t h e r funded at that l e v e l or i t i s not funded at a l l . However, the cost of any p r o j e c t i s only an est imate of the amount requ i red to ensure a reasonable p r o b a b i l i t y of s u c c e s s . L i k e any est imate i t i s sub jec t to e r r o r s . I f the p r o j e c t were to be funded at an amount l e s s than i t s " c o s t , " then i t would be l e s s l i k e l y to succeed , but there would s t i l l be some p r o b a b i l i t y of s u c c e s s . Converse l y , i f the p r o j e c t were funded above i t s " c o s t , " then i t would be more l i k e l y to succeed. This i s t rue only w i t h i n a c e r t a i n range, however, For any p r o j e c t , there e x i s t s a lower funding bound such that i f i t i s a l l o c a t e d any l e s s , there i s v i r t u a l l y no p r o b a b i l i t y of s u c c e s s , and an upper funding bound such that i f i t i s a l l o c a t e d any more, there i s no s i g n i f i -cant inc rease i n the p r o b a b i l i t y i t w i l l succeed. The proposed a l g o r i t h m a l lows the amount of funding to f a l l anywhere between these lower and upper bounds. The r e l a t i o n s h i p between the funding l e v e l and the p r o b a b i l i t y of success of the p r o j e c t i s 74 assumed to be a p iecewise l i n e a r f u n c t i o n as desc r ibed i n . t h e l a s t chapte r . These two f e a t u r e s , m u l t i p l e - c r i t e r i a and funding i n an i n t e r v a l , make t h i s model a more accurate r e p r e s e n t a t i o n of the p r o j e c t s e l e c t i o n d e c i s i o n s i t u a t i o n and improve the usefu lness of the a l g o r i t h m f o r r e a l d e c i s i o n s i t u a t i o n s . 4 .2 DATA REQUIREMENTS The data requ i red by the a l g o r i t h m i s not as ex tens i ve as that requ i red by most other a l g o r i t h m s . The d e c i s i o n maker must be able to d e s c r i b e each p r o j e c t . He must be able to s p e c i f y the l e v e l s of a l l r e l e v a n t c r i t e r i a tha t each p r o j e c t a c h i e v e s , and the amount of each scarce resource each p r o j e c t r e q u i r e s . This data i s the minimum requ i red f o r r a t i o n a l s e l e c -t i o n . Often p r o j e c t d e s c r i p t i o n s or proposals con ta in a l l the data necessary . However, the requ i red p r o b a b i l i t y of success est imates may not be as r e a d i l y a v a i l a b l e and o f ten must be prov ided by the p r o j e c t e v a l u a t i o r . A study by Souder [74] of the v a l i d i t y of s u b j e c t i v e p r o b a b i l i t y of success est imates i n d i c a t e s that most exper ienced p r o j e c t eva luato rs can assess the p r o b a b i l i t y of success of any p r o j e c t s u r p r i s i n g l y w e l l , though in some cases t h i s assessment i s not a c c u r a t e l y communi-cated f o r p o l i t i c a l reasons . 75 4.3 THE USEFULNESS OF THE ALGORITHM While the a lgo r i thm was not t e s t e d in r e a l problem s i t u a t i o n s , 1 i t has some a t t r a c t i v e f e a t u r e s . The a l g o r i t h m i s general enough to be usefu l in any p r o j e c t s e l e c t i o n d e c i s i o n s i t u a t i o n . Any type of c r i t e r i a may be used and enough data would g e n e r a l l y be a v a i l a b l e on even somewhat l e s s - s t r u c t u r e d pure research p r o j e c t s . I t i s probably l e a s t usefu l however, fo r development- type p r o j e c t s whose only goa l , i s monetary. Several of the other a lgor i thms such as the dynamic programming ones, are b e t t e r s u i t e d to such c a s e s . S ince the problem i s a combinat ional one ( i n v o l v e s in teger c o n s t r a i n e d v a r i a b l e s ) , the value of the opt imal s o l -u t i o n i s not a cont inuous f u n c t i o n of the value of the r i g h t -hand-s ide ( resource a v a i l a b i l i t i e s ) . I t i s thus p o s s i b l e that a s l i g h t inc rease in the a v a i l a b l e amount of any resource would r e s u l t in a la rge inc rease in the value of the opt imal s o l u t i o n , and a very d i f f e r e n t funding p a t t e r n . To t e s t the s e n s i t i v i t y of the s o l u t i o n to changes in the r i g h t - h a n d - s i d e , the a l g o r i t h m may be r e - r u n without r e - d e r i v i n g the u t i l i t y f u n c t i o n . Souder has d iscussed the " u t i l i t y and a c c e p t a b i l i t y " of p r o j e c t s e l e c t i o n a l g o r i t h m s . He suggests (and i s echoed by B e a t t i e and Reader [6]) that a se r ious shortcoming of the 1 A sample problem and i t s s o l u t i o n are desc r ibed in t h e A p p e n d i x . 76 a lgor i thms i s that only one s o l u t i o n , the supposedly opt imal s o l u t i o n , i s p r o v i d e d . This de fec t was overcome in the p ro -posed a lgo r i thm by p r o v i d i n g the f i v e best s o l u t i o n s ( i f f i v e f e a s i b l e . o n e s e x i s t ) . Since the data the d e c i s i o n maker p ro -v ides i s not e x a c t , and the u t i l i t y f u n c t i o n der i ved i s only an approximate measure of the s u b j e c t i v e value of any p o r t f o l i o of p r o j e c t s , no s o l u t i o n can be cons idered o p t i m a l . Several s o l u t i o n s may have almost the same v a l u e , and given the accuracy of the d a t a , none of them can be sa id to be the bes t . There are o f ten c o n s i d e r a t i o n s which cannot be inc luded in the u t i l i t y f u n c t i o n nor in the problem's c o n s t r a i n t s . If the d e c i s i o n maker has a set of s o l u t i o n s to choose f rom, he can d i s c r i m i n a t e among them on the bas is of these other c o n s i d e r a t i o n s . In that way the a l g o r i t h m i s usefu l as a too l fo r a n a l y z i n g the d e c i s i o n s i t u a t i o n . . I t s i m p l i f i e s the d e c i s i o n by reducing the number of p o s s i b l e c h o i c e s . However, i t i s not intended to rep lace the d e c i s i o n maker. His p a r t i c u l a r s k i l l s and i n s i g h t s are s t i l l used. He i s in f a c t an i n t e g r a l part of the a l g o r i t h m . Rather than r e p l a c i n g the d e c i s i o n maker, the a l g o r i t h m attempts to f o r m a l i z e some of h is thought p rocesses . The U t i l i t y f u n c t i o n which i s der i ved i s a f o r m a l i z a t i o n of p r e f -erence cho ices made by the d e c i s i o n maker. It can be a p p l i e d to other d e c i s i o n s i t u a t i o n s and would r e s u l t in c o n s i s t e n t responses . The d e c i s i o n maker cannot be c o n s i s t e n t . He i s a f f e c t e d by extraneous f a c t o r s he can n e i t h e r c o n t r o l nor com-77 pensate f o r . Shepard [70] has d iscussed t h i s phenomenon and terms i t " s u b j e c t i v e n o n - o p t i m a l i t y . " It occurs when a d e c i s i o n maker choses an a l t e r n a t i v e he b e l i e v e s to be b e s t , and indeed , i t may be best with respect to h is cur rent s t a t e of mind and the s t i m u l i he i s exposed t o . However, at a l a t e r t i m e , detached from these s t i m u l i he recognizes that the choice was not opt imal wi th respect to his t rue p r e f e r e n c e s . The d o o r - t o -door encyc lopedia salesman for example, and other "p ressure" salesmen c a p i t a l i z e on t h i s f o i b l e of human natu re . O f t e n , a "paramorphic r e p r e s e n t a t i o n " of a d e c i s i o n maker, by a l l e v i a t i n g st rong s i t u a t i o n a l s t i m u l i can out -per fo rm the d e c i s i o n maker by r e f l e c t i n g h is t rue p r e f e r e n c e s . Bowman [10] and Kunreuther [53] have suggested that i n c o n s i s t e n c y i s the major cause of c o s t l y d e c i s i o n e r r o r s . They have used d e c i s i o n models based on past d e c i s i o n s made by the d e c i s i o n maker to improve h is performance, and have had e x c e l l e n t r e s u l t s in a v a r i e t y of a r e a s . Because of the d e c i s i o n maker's d i r e c t involvement in the a l g o r i t h m , and the c o n t r o l of c r i t e r i a and e v a l u a t i o n methods a l l o w e d , i t i s hoped that t h i s a lgo r i thm w i l l be more acceptab le a,s a d e c i s i o n making too l than p r e v i o u s l y suggested ones. It has another use as w e l l , however, the der i ved u t i l i t y f u n c t i o n can be used fo r d i a g n o s t i c purposes , showing the d e c i s i o n maker how he a c t u a l l y makes c h o i c e s , i . e . which c r i t e r i a he cons iders most impor tant . Any d i s c r e p a n c i e s between these c r i t e r i a and the c r i t e r i a he b e l i e v e s he uses can be pointed out . Any 78 i l l o g i c a l f e a t u r e s can thus be e l i m i n a t e d from his u t i l i t y f u n c t i o n . His u t i l i t y f u n c t i o n can be compared wi th those of other d e c i s i o n makers in the o r g a n i z a t i o n . Any d i f f e r e n c e s can be d i s c u s s e d , and p o s s i b l y an o r g a n i z a t i o n a l o b j e c t i v e f u n c t i o n can be agreed upon. A b e t t e r u t i l i t y f u n c t i o n found by such a n a l y s i s can be used to help analyze f u t u r e d e c i s i o n problems more c o n s i s t e n t l y . The o p t i m i z a t i o n s e c t i o n of the a l g o r i t h m can be used a l o n e , i f a u t i l i t y f u n c t i o n was p r e v i o u s l y e s t i m a t e d . 4 .4 POSSIBLE EXTENSIONS AND IMPROVEMENTS TO THE ALGORITHM Several extens ions and improvements to the a l g o r i t h m are p o s s i b l e . The p iecewise l i n e a r f u n c t i o n can be extended to i n c l u d e more than two p ieces in the hope of f i n d i n g a b e t t e r f i t t i n g u t i l i t y f u n c t i o n . I f N p ieces are d e s i r e d N-l poor ly f i t t e d p o i n t s are r e q u i r e d as b r e a k p o i n t s . The c u r r e n t a l g o r i t h m may be e a s i l y mod i f ied to f i n d these break po in ts and the co -e f f i c i e n t s a^ f o r each p iece i . More pseudo -p ro jec t r a t i n g s wou1d be requ i red to f i t a l i n e a c c u r a t e l y i n each of the new p i e c e s . The p i e c e - w i s e l i n e a r f o r m u l a t i o n may be r e a d i l y ex-tended to i n c l u d e N p ieces by i n t r o d u c i n g s ^ , i = 1 , ••• • , N - l . s^ i s the amount by which the t o t a l l e v e l of c r i t e r i o n k, t^ exceeds the i**1 b r e a k p o i n t s . The complete f o r m u l a t i o n i s as f o l l o w s : 79 max m I a k t k -N - 1 I S. . t . t. = I k j - 1 { J J r i I a k j S k ( 3 , - a . ) y , ) J J J Vk (1) 1 > n k j k V k , £ (2) J I .z . + (u . - I.) y . < B (3) I j = l , £ . z . + ( U . - Z .) y J < D j ' JJ) f, •.' R * V £ (4) other c o n s t r a i n t s (6) Zj = 0 , 1 , and 0 < y. < 1 (7) v k , a (8) 80 a t h where B K i s the upper bound on c r i t e r i o n l e v e l k f o r the I p i e c e . I f B £ S t f c < B £ + 1 , then s-J > 0 f o r I = 1 ,• • • , f and o p t i m a l i t y fo rces s£ = 0 f o r a l l £ > f , s ince (a£ - a k + 1 ) < 0 by the concav i t y of the p iecewise l i n e a r form. The value of l e v e l t^ of the k**1 c r i t e r i o n i s then a^ • t^ minus ( a k - a*) f o r each u n i t t^ exceeds B * , minus (a^ - a^) f o r each u n i t t k 2 t h exceeds B K , e t c . The amount of t k that f a l l s i n t o the SL 0 piece i s thus valued at a k . This f o r m u l a t i o n can be r e a d i l y so lved by a branch and bound mixed i n t e g e r a l g o r i t h m as b e f o r e . Another p o s s i b l e ex tens ion would be to approximate some of the f i v e n o n - l i n e a r u t i l i t y f u n c t i o n forms wi th a p iecewise l i n e a r f u n c t i o n . There may be cases where one of these forms would prov ide a b e t t e r f i t to the d e c i s i o n maker's u t i l i t y f u n c -t i o n than the l i n e a r form or a d i r e c t l y der i ved p iecewise l i n e a r m - a . form. The c o n j u n c t i v e fo rm, U = II X k , i s concave i f the k= 1 m are g reate r than 1 , and the l o g a r i t h m i c fo rm, U = T a., log X . , k=l K K i s concave i f the are p o s i t i v e . These forms could be a c c u r a t e l y approximated by a concave p iecewise l i n e a r i z a t i o n . The p i e c e -wise l i n e a r f o r m u l a t i o n g iven above would then be a p p l i c a b l e . These extens ions would make the a l g o r i t h m more f l e x i b l e ; ab le to model more d e c i s i o n makers and d e c i s i o n s i t u a t i o n s . In the great m a j o r i t y of cases however, the l i n e a r form or the p iecewise l i n e a r form wi th only two p ieces w i l l be as accurate a r e p r e s e n t a t i o n of the d e c i s i o n maker's u t i l i t y f u n c t i o n as the q u a l i t y of the data j u s t i f i e s . 81 The d e c i s i o n model developed in the l a s t chapter ; i s a p p l i c a b l e to p r o j e c t s where the b e n e f i t l e v e l s are r e l a t e d to the funding l e v e l through the p r o b a b i l i t y of success term. Such p r o j e c t s e i t h e r succeed , in which case a l l the b e n e f i t s are r e c e i v e d , or f a i l , in which case none of the b e n e f i t s are a c h i e v e d . Many p r o j e c t s however, do not have t h i s s u c c e s s -f a i l u r e s t r u c t u r e . The b e n e f i t l e v e l s in such cases are d i r e c t l y r e l a t e d to f u n d i n g , wi th no p r o b a b i l i t y of success c o n s i d e r a t i o n s necessary . To model such s i t u a t i o n s , P - ( A - ) or could, be i n t e r p r e t e d as the p ropor t ion of the b e n e f i t l e v e l X k ^ which would be achieved i f the p r o j e c t were to be funded at a. ra ther 0 than the requested funding l e v e l f . . S i m i l a r l y P . ( u - ) or 3 j would be the p r o p o r t i o n of X^j achieved i f p r o j e c t j were to be funded at u ^ P - ( x - ) would then be l i n e a r between u . J J J J a n d % j . where x . = i.z. + ( 3 ^ - a. ) y. . Often p r o j e c t s are judged on sets of c r i t e r i a which f a l l i n t o both c a t e g o r i e s . For example, cons ider a p r o j e c t wi th the aim of deve lop ing a new o p t i c a l i n s t r u m e n t . If two of the c r i t e r i a used to eva luate i t are export s a l e s , and jobs c r e a t e d , then the former f a l l s i n to the s u c c e s s - f a i l u r e c a t e g o r y , wh i le the l a t t e r depends d i r e c t l y on funding l e v e l and i s e s s e n t i a l l y independent of 82 the outcome of the p r o j e c t . These s i t u a t i o n s may be modelled by c o n s i d e r i n g P - ( x - ) to be a p r o b a b i l i t y in the one case and J J a p ropor t ion i n the o t h e r . Other types of p r o j e c t s that the a l g o r i t h m may be modi f ied to handle are those which extend across more than one p lanning p e r i o d . For the l i n e a r u t i l i t y f u n c t i o n the f o r m u l a t i o n would be as f o l l o w s : P m n m a x & k ? ! a k [ a i J z i J + ( e i j - a i j ) ] x i k j s . t Z . . z . . + (u•• - a• •) y• •] < B. V i ( i ) 11 I !ui [ * i j Z l J • (M,J - t 1 d ) y , j ] « R u V i . i J • " ' 1 J y 1 j ' < Z 1 j V i . j other c o n s t r a i n t s (4) z. . = 0 or 1 , 0 < y . . < 1 V i , j (5) where the index i = 1 , . . . P r e f e r s to the p lanning p e r i o d , and a l l the v a r i a b l e s are as p r e v i o u s l y d e f i n e d . The p ro jec t s e l e c t i o n d e c i s i o n model presented here 83 attempts to be a more r e a l i s t i c r e p r e s e n t a t i o n of the rea l d e c i s i o n s i t u a t i o n . Because of t h i s g reater r e a l i s m and the p a r t i c i p a t i o n requ i red of the d e c i s i o n maker i t i s hoped that i t w i l l be a more acceptab le and usefu l too l fo r a n a l y z i n g p r o j e c t s e l e c t i o n d e c i s i o n s i t u a t i o n s than p r e v i o u s l y developed a l g o r i t h m s . B I B L I O G R A P H Y Asher , D.T . , "A L inear Programming Model f o r the A l l o c a -t i o n of R&D E f f o r t s , " IRE Transactions on Engineering Management, Dec. 1962. Atk inson , A . C . , A . H . B o b i s , "A Mathemtical Bas is f o r the S e l e c t i o n of Research P r o j e c t s , " IEEE Transactions on Engineering Management, Feb. 1969. Augood, D .R . , "A Review of R&D E v a l u a t i o n Methods," IEEE Transactions on Engineering Management, Nov. 1973. Aumann, R . J . , " S u b j e c t i v e Programming" in Human Judgement and Optimality, M.W. S h e l l e y and G.L . Bryan ( e d s . ) , W i l e y , 1 964. Baker , N .R . , W.H. Pound, "R&D P r o j e c t S e l e c t i o n : Where We S t a n d , " IEEE Transactions on Engineering Management, Dec. 1964. B e a t t i e , C . J . , R.D. Reader, Quantitative Management in Research and Development, Chapman and Ha l l L t d . , 1971. B o b i s , A . H . , A. A t k i n s o n , "A Mathematical Bas is f o r the S e l e c t i o n of Research P r o j e c t s , " IEEE Transactions on Engineering Management, Feb. 1969. : : , "Ana lyz ing P o t e n t i a l Research P r o j e c t s , " Chem. Eng., Feb. 1970. B o b i s , A . H . , T . F . Cooke, G.H. Paden, "A Funds A l l o c a t i o n Methods t Improve the Odds f o r Research S u c c e s s , " Research Management, March 1971. 84 85 10. Bowman, E . H . , "Cons is tency and O p t i m a l i t y i n Manager ial D e c i s i o n M a k i n g , " Management Science, 1963, p. 310. 11. Brandenburg, R . G . , " P r o j e c t S e l e c t i o n in I n d u s t r i a l Research and Development: Problems and D e c i s i o n P r o c e s s e s , " i n Research Program Effectiveness, M.C. Y o v i t s ( e d . ) . 12. C l a y t o n , R. , "A Convergent Approach to R&D Planning and P r o j e c t S e l e c t i o n , " Research Management, Sept . 1971. 13. C e t r o n , M . J . , J . M a r t i n o , L. Roepcke, "The S e l e c t i o n of R&D Program Content -Survey of Q u a n t i t a t i v e Methods," IEEE Transactions on Engineering Management, Mar. 1967. 14. Charnes, A . , A . C . S t e d r y , "A Chance -Const ra ined Model f o r R e a l - t i m e Contro l in Research and Development Management," Management Science, Apr . 1966. 15. Churchman, C.W., R .L . A c k o f f , "An Approximate Measure of V a l u e , " Operations Research, 1954, pp. 172 -187 . 16. Cramer, R . H . , B .E . S m i t h , " D e c i s i o n Models f o r the S e l e c -t i o n of Research P r o j e c t s , " Engineering Economist, V o l . 9 , J a n - F e b . 1964. 17. Dawes, R . M . , " S o c i a l S e l e c t i o n Based on M u l t i d i m e n s i o n a l C r i t e r i a , " Journal of Abnormal and Scoial Psychology, V o l . 68, 1964. 18. , "A Case Study of Graduate Admiss ions : A p p l i c a t i o n of Three P r i n c i p l e s of Human D e c i s i o n M a k i n g , " American Psychologist, V o l . 26, 1971. 19. Dean, B . V . , Evaluating, Selecting and C o n t r o l l i n g Research and Development Projects, American Management A s s o c i a t i o n I n c . , 1968. 20. Dean B . V . , L . E . Hauser, "Advanced M a t e r i a l Systems P l a n n i n g , " IEEE Transactions on Engineering Management, March 1967. 21. Dean, B . V . , M . J . N i s h r y , "Scor ing and P r o f i t a b i l i t y Models f o r Eva lua t ing and S e l e c t i n g E n g i n e e r i n g - P r o j e c t s , " Operations Research, J u l y - A u g . 1965. 86 22. Dean, B . V . , S . S . Sengupta, "Research Budgeting and P r o j e c t S e l e c t i o n , " IRE Transactions on Engineering Management, Dec. 1962. 23. Disman, S . , " S e l e c t i n g R&D P r o j e c t s f o r P r o f i t , " Chemical Eng., Dec. 1962. 24. Dyer, J . S . , " I n t e r a c t i v e Goal Programming," Management Science, Sept . 1972. 25. , "A T ime-shar ing Computer Program f o r the S o l u -t i o n of the M u l t i p l e C r i t e r i a P rob lem, " Management Science, V o l . 19, No. 12 , 1973. 26. E i n h o r n , H . J . , "The Use of N o n - l i n e a r , Non-compensatory Models in D e c i s i o n M a k i n g , " Psychological Bulletin, March 1 970. 27. , "Use of N o n - l i n e a r , Non-compensatory Models as a Funct ion of Task and Amount of I n f o r m a t i o n , " Organi-zational Behaviour and Human Performance, V o l . 6 , 1971. 28. F i s h b u r n , P . C . , "Independence i n U t i 1 i t y Theory wi th Whole Product Sets , " Operations Research, J a n . - F e b . 1 965. 29. : — , "Methods of E s t i m a t i n g A d d i t i v e U t i l i t i e s , " Management Science, V o l . 1 3 , No. 7, 1967. 30. , " A d d i t i v e U t i l i t i e s wi th F i n i t e S e t s : A p p l i c a t i o n s in the Management S c i e n c e s , " Naval Research L o g i s t i c s Quarterly, 1 9 6 5 . , 31. Freeman, R . J . , "A S t o c h a s t i c Model f o r Determining the S i ze and A l l o c a t i o n of the Research Budget , " IRE Transac-tions on Engineering Management, March 1960. 32. G a r g u i l o , G . P . , J . Hannoch, D.B. H e r t z , T. Lang, "Developing Systemat ic Procedures f o r D i r e c t i n g Research Programs," IRE Transactions on Engineering Management, March 1961. 33. Gear , A . E . , A . G . L o c k e t t , "A Dynamic Model of Some M u l t i -Stage Aspects of R&D P o r t f o l i o s , " IEEE Transactions on Engineering Management, Feb. 1973. 87 34. Gee, R . E . , "A. Survey of Current P r o j e c t S e l e c t i o n P r a c t i c e s , " Research Management, Sept . 1971. 35. G e o f f r i o n , A . M . , J . S . Dyer, A. F e i n b u r g , "An I n t e r a c t i v e Approach f o r M u l t i - C r i t e r i o n O p t i m i z a t i o n , wi th an A p p l i c a t i o n to the Operat ion of an Academic Department," Management Science, Dec. 1972. 36. G i l l e s p i e , J . S . , A . E . Gear, "An A n a l y t i c a l Methodology f o r Comparing the S u i t a b i 1 i t y of Management Sc ience M o d e l s , " IEEE Transactions on Engineering Management, Nov. 1973. 37. Go ldberg , L . R . , "S imple Models or Simple Processes? Some Research on C l i n c i a l Judgements," American Psychologist, 1 968. 38. - , " M a n vs . Model of Man: A R a t i o n a l e Plus. Some Evidence fo r a Method of Improving on C l i n i c a l I n f e r -e n c e s , " Psychological B u l l e t i n , V o l . 7 3 , No. 6 , 1970. 39. , "F ive Models of C l i n i c a l Judgement: An E m p i r i c a l Comparison between L i n e a r and N o n - l i n e a r Repre-s e n t a t i o n s of the Human Inference P r o c e s s , " Organizational Behavior and Human Performance, V o l . 6 , 1971. 40. Green, P . G . , "R i sk A t t i t u d e s and Chemical Investment D e c i s i o n s , " Chemical Engineering Progress, J a n . 1963. 4 1 . H e r t z , D . B . , "R isk A n a l y s i s i n C a p i t a l Investment , " E.B.R., V o l . 4 2 , No. 1 , 1 964. 42. H e r t z , D . B . , P. C a r l s o n , " S e c t i o n , E v a l u a t i o n and Contro l of Research and Development P r o j e c t s , " i n Operations Research in Research and Development, B.V. Dean ( e d . ) , New York , W i l e y , 1963. 43 . Hespo.s, R . F . , P . A Strassman , " S t o c h a s t i c D e c i s i o n Trees f o r the A n a l y s i s of Investment D e c i s i o n s , " Management Science, Aug. 1965. 44. Hess, S . W . , "A Dynamic Programming Approach to R&D Budgeting and P r o j e c t S e l e c t i o n , " IRE Transactions on Engineering Management, Dec. 1962. 88 45 . H i r s c h , J . H . , E .F . F i s h e r , "A New Procedure Prov ides a Bas is f o r Determining What P r o j e c t s to S t a r t , S top , C o m m e r c i a l i z e , " Research Management, J a n . 1968. 46. Hoffman, P . J . , "Cue Cons is tency and C o n f i g u r a l i t y in Human Judgement," i n Forman Representation of Human Judgement, B. Kleinmuntz ( e d . ) , W i l e y , 1968. 47 . Hoffman , P . J . , N. Wigg ins , "Three Models of C l i n i c a l Judgement," Journal of Abnormal and Social Psychology, Feb. 1968. 48 . Huber, G . P . , R. Daneshgar, D.L. Fo rd , "An E m p i r i c a l Com-par i son of F ive U t i l i t y Models f o r P r e d i c t i n g Job P r e f e r -e n c e s , " Organizational Behaviour and Human Performance, 1971 , V o l . 6. 49. Huber, G . P . , V .K . Sahney, D.L. Fo rd , "A Study of Sub jec -t i v e E v a l u a t i o n M o d e l s , " Behavioral Science, 1969, V o l . 14. 50. Keeny, R . L . , " U t i l i t y Independence and Preferences f o r Mul t . i - A t t r i b u t e d Consequences," Operations Research, J u l y - A u g u s t , 1971 . 51. K l a h r , D. , " D e c i s i o n Making in a Complex Environment: the Use of S i m i l a r i t y Judgements to P r e d i c t Preferences," Management Science, V o l . 15 , No. 1 1 , 1969. 52. K le inmuntz , B . , "The Process ing of C l i n i c a l In format ion by Man and Machine , " i n Formal Representation of Human Judgement, B. Kleinmuntz ( e d . ) , New York , W i l e y , 1968. 53. Kunreuther , H . , "Extens ions of Bowman's Theory of Manager ial D e c i s i on.Making , " Management Science, A p r i l 1 969. 54. Lee , S . , E.R. C l a y t o n , "A Goal Programming Model f o r Academic Resource A l l o c a t i o n , " Management Science, V o l . 18, No. 8 , 1972. 55. L o c k e t t , A . G . , P. Freeman, " P r o b a b i l i s t i c Networks and R&D P o r t f o l i o S e l e c t i o n , " Oper. Res. Quarterly, Sept . 1 970. 89 56. L o c k e t t , A . G . , A . E . Gear , "Representat ion and A n a l y s i s of M u l t i - S t a g e Problems in R&D," Management Science, Apr . 1 973. 57. MacCrimmon, K . R . , M. Toda, "The Exper imental Determinat ion of I n d i f f e r e n c e Curves , " The Review of Economic Studies, V o l . XXXVI , No. 4 , 1 969. 58. MacCrimmon, K . R . , "An Overview of M u l t i p l e O b j e c t i v e D e c i s i o n M a k i n g , " U n i v e r s i t y of B r i t i s h Columbia Working P a p e r . 1 7 9 , February 1973. 59. M a l l o y , J . B . , "R isk A n a l y s i s of Chemical P l a n t s , " Chem. Eng. Prog., Oct . 1971. 60. Marschak, J . A . , "Models , Rules of Thumb, and Development D e c i s i o n s , " in Operations Research in Research and Develop-ment, B.V. Dean ( e d . ) , John Wiley and Sons, I n c . , 1963. 61 . Minkes , A . L . , J . M . Samuels, " A l l o c a t i o n of R&D Expenditure i n the F i r m , " Journal of Management Studies, pp. 6 2 - 7 2 , 1 966. 62. Moore, J . R . , N.R. Baker , "Computational A n a l y s i s of Scor ing Models f o r R&D P r o j e c t S e l e c t i o n , " Management Science, Dec. 1969. 63. Moskowitz , H . , "An Experimental I n v e s t i g a t i o n of D e c i s i o n Making i n a S imulated Research and Development E n v i r o n -ment," Management Science, Feb. 1973. 64. M o t t l e y , C M . , R.D. Newton, "The S e l e c t i o n of P r o j e c t s f o r I n d u s t r i a l R e s e a r c h , " Operations Research, Nov . -Dec . 1 959 . 65. Muk, R . L . , " P r o j e c t S e l e c t i o n in the Petroleum I n d u s t r y , " Research Management, Sept . 1971. 66. N u t t , A . B . , "An Approach to Research and Development E f f e c t i v e n e s s , " IEEE Transactions on Engineering Management, Sept . 1965. 67. O l s e n , J . J . , "Winds of Change in I n d u s t r i a l Chemical R e s e a r c h , " Chem. Eng. News, May 1961. 90 68. Pound, W.H . , "Research Program S e l e c t i o n : Tes t ing a Model in the F i e l d , " IEEE Transactions on Engineering Management, March 19 64. 69. Rosen, E . M . , W.E. Souder, "A Method f o r A l l o c a t i n g R&D E x p e n d i t u r e s , " IEEE Transactions on Engineering Management, Sept . 1965. 70. Shepard , R.N. , "On S u b j e c t i v e l y Optimum S e l e c t i o n Among M u l t i - a t t r i b u t e A l t e r n a t i v e s , " in Human Judgement and Optimality, M.W. S h e l l e y , and G.L . Bryan ( e d s . ) . 7 1 . S l o v i c , P . , S . L i t c h e n s t e i n , "Comparison of Bayesian and Regression Approaches to the Study of In format ion P r o -cess ing in Judgement," Organizational Behavior.and Human Performance, V o l . 6 , 1971. 72. S m i t h , R. , P . S . Greenlaw, " S i m u l a t i o n of a P s y c h o l o g i c a l D e c i s i o n Process i n Personnel S e l e c t i o n , " Management Science, V o l . 13 , No. 8 , 1967. 73. Souder, W . E . , " S e l e c t i n g and S t a f f i n g R&D P r o j e c t s v i a Operat ions R e s e a r c h , " Chemical Eng. Prog., Nov. 1967. 74. , "The V a l i d i t y of S u b j e c t i v e P r o b a b i l i t y of Success Forecasts by R&D P r o j e c t Managers," IEEE Trans-actions on Engineering Management, Feb. 1969. 75. — , "Comparative A n a l y s i s of R&D Investment Mode ls , " AIIE Transactions on Engineering Management, March 1972. 76. — , "A Scor ing Methodology f o r Assess ing the S u i t a b i l i t y of Management Sc ience Mode ls , " Management Science, June 1972. 77. — : , " A n a l y t i c a l E f f e c t i v e n e s s of Mathematical Models f o r R&D P r o j e c t S e l e c t i o n , " Management Science, A p r i l 1973. 78. : , " U t i l i t y and Perce ived A c c e p t a b i 1 i t y of R&D P r o j e c t S e l e c t i o n M o d e l s , " Management Science, Aug. 1973. 91 79. T e r r y , H. , "Comparative Eva lua t ion of Performance Using M u l t i p l e C r i t e r i a , " Management Science, V o l . 9 , No. 3 , 1963. 80. Thorngr in , J . T . , " P r o b a b i l i t y Technique Improves Investment A n a l y s i s , " Chem.' Eng., Aug. 1967. 8 1 . Weingar tner , H.M. , " C a p i t a l Budgeting of I n t e r r e l a t e d P r o j e c t s : Survey and S n y t h e s i s , " Management Science, Mar. 1966. 82. Whitman, E . S . , E .F . Landau, " P r o j e c t S e l e c t i o n in the Chemical I n d u s t r y , " Research Management, Sept . 1.971. 83. v. W i n t e r f e l d t , D. , G.W. F i s h e r , " M u l t i - A t t r i b u t e U t i 1 i t y Theory: Models and Assessment P r o c e d u r e s , " U n i v e r s i t y of Michigan Techn ica l Report 011313-7 -T . 84. Yntema, D . B . , W.S. Torgerson , "Man-Computer Cooperat ion in Dec is ions Requ i r ing Common Sense , " IRE Transactions on Human Factors in E l e c t r o n i c s , March 1961. APPENDIX - A SAMPLE PROBLEM Consider a government f u n d - g r a n t i n g agency which has a number of p r o j e c t s under c o n s i d e r a t i o n . In order to use the p r o j e c t s e l e c t i o n a l g o r i t h m as a d e c i s i o n too l in a n a l y z i n g the s i t u a t i o n , three types of inputs to the program a r e r e q u i r e d . The f i r s t input requ i red i s i n f o r m a t i o n about goals and c o n s t r a i n t s . The d e c i s i o n maker, in c o n s u l t a t i o n wi th the p o l i c y a n a l y s i s s t a f f of the agency must decide upon the goals which should be achieved by the p a r t i c u l a r program, and the i n d i c a t o r s that best r e f l e c t achievement wi th respect to these g o a l s . Often goals may be cas t as c o n s t r a i n t s . For example, one goal might be the encouragement of economic growth in depressed r e g i o n s . This goal could be represented by a requirement that a minimal p ropor t ion of the funds be spent in such r e g i o n s . Other goals such as dec reas ing unem-ployment, i n c r e a s i n g the GNP, and improving the q u a l i t y of l i f e -could be inc luded d i r e c t l y in the o b j e c t i v e f u n c t i o n . The resources requ i red by the p r o j e c t s must then be appra ised in terms of t o t a l resources a v a i l a b l e . C l e a r l y c a p i t a l , or funds to be a l l o c a t e d , i s cons idered to be a scarce r e s o u r c e . Other examples of scarce resources may be var ious types of s k i l l e d l a b o u r , energy, or s p e c i f i c types of equipment. Re-source a v a i l a b i l i t i e s may impose c o n s t r a i n t s on subsets of p r o j e c t s or on a l l of them. 92 93 The second type of input necessary f o r the program i s i n f o r m a t i o n from the p r o j e c t e v a l u a t o r . The eva lua to r i s r e s p o n s i b l e for. e s t i m a t i n g the p r o b a b i l i t y of success of each p r o j e c t . Consequent ly , he must eva luate the competence and c a p a b i l i t i e s of the a p p l i c a n t s , the mer i t s of the proposal in terms of t e c h n o l o g i c a l requ i rements , and f i n a l l y , the compat-a b i l i t y of the a p p l i c a n t s c a p a b i l i t i e s and the proposed p r o j e c t . On the bas is of these c o n s i d e r a t i o n s , he should be able to i n d i c a t e the lower funding bound or c r i t i c a l po int below which the p r o j e c t would have l i t t l e p r o b a b i l i t y of success and the upper funding bound or s a t i a t i o n po in t above which an i n c r e a s e in funding would not be r e f l e c t e d in a s i g n i f i c a n t inc rease in the p r o b a b i l i t y of success of the p r o j e c t . P r o b a b i l i t y of success est imates given that the p r o j e c t i s funded at the two bounds are r e q u i r e d . The a p p l i c a n t must prov ide the t h i r d phase of input to the program. He must s p e c i f y the l e v e l of funding d e s i r e d fo r the p r o j e c t , and d e s c r i b e the p r o j e c t in terms of resources r e q u i r e d , b e n e f i t s produced, and other cos ts i n c u r r e d . ^ A proposed schedule i s a l so necessary so that the d e c i s i o n maker can d i scount the b e n e f i t s to f a c i l i t a t e comparisons between p r o j e c t s wi th d i f f e r e n t t im ing p a t t e r n s . "Other c o s t s " may i n c l u d e s o c i a l cos ts and e x t e r -n a l i t i e s such as an inc rease in p o l l u t i o n l e v e l . 94 In the example c h o s e n , the government agency has the f o l l o w i n g t h r e e g o a l s ; 1) i n c r e a s i n g employment 2) i m p r o v i n g the b a l a n c e of p a y m e n t s , and 3) i m p r o v i n g the q u a l i t y of l i f e . A c h i e v e -ment w i t h r e s p e c t to the f i r s t g o a l can be measured d i r e c t l y by the number of j o b s c r e a t e d by each p r o j e c t s . Two i n d i c a t o r s may be used to measure a c h i e v e m e n t w i t h r e s p e c t to the second g o a l ; 1) e x p o r t s a l e s , and 2) added v a l u e . Q u a l i t y of 1 i f e a c c o r d i n g to t h i s a g e n c y , c o n c e r n s s t a b i l i z a t i o n r a t h e r n t h a n i m p r o v e m e n t , and c o n s e q u e n t l y the p o l l u t i o n produced by each p r o j e c t i s c o n s i d e r e d to be an i n d i c a t o r r e l a t e d t o the t h i r d g o a l . The agency has f i v e p r o j e c t s under c o n s i d e r a t i o n w i t h the f o l l o w i n g c h a r a c t e r i s t i c s . Project 1 Projects 2 Project 3 Project 4 Project 5 Resource cap i ta l (mi l l i on $) 1 5 15 2 10 Required ski 1 led labour 1 s k i l l e d labour 2 300 100 100 50 150 Benefits jobs created 500 . 100 150 300 200 'Produced export sales ($10,000) 2 3. 5 2 6 value added ' ($10,000) 5 10 30 5 10 po l lu t ion created -1 -1 -10 -1 -13 Funding ( in lower bound 1 2 10 1 5 'mi l l ion $) requested level 1 5 15 2 10 upper bound 2 15 20 ' '4 15 at lower bound .8 .5 .7 •8 .7 at upper bound 1.0 .85 .98 1.0 .95 TABLE II SAMPLE PROBLEM DATA The fo l lowing pages are the output from the in te rac t i ve project se lect ion algorithm applied to th i s problem. 95 WELCOME TO 'MODEM' , A MULTIPLE OBJECTIVE DECISION MAKING PROGRAM. MODEM MAY HELP YOU DECIDE WHICH OF A NUMBER OF PROJECTS TO SUPPORT AND TO WHAT EXTENT SUPPORT SHOULD BE GIVEN. BACKUP: ENTERING 'BACK' WILL CAUSE REVERSION TO A PHEVIOUS QUESTION. TERMINATION: ENTERING ' E N D ' , 'HALT ' OR 'STOP' WILL STOP MODEM IMMEDIATELY. HOW MANY PROJECTS ARE THESE? (MAXIMUM IS 10.) 5 WHAT IS THE NAME OF PROJECT 1? one WHAT IS THE NAME OF PROJECT 2? two WHAT IS THE NAME OF PROJECT 3? three WHAT IS THE NAME OF PROJECT U? four WHAT IS THE NAME OF PROJECT 5? five WHAT IS THE TOTAL NUMBER OF BENEFITS (OBJECTIVES, PAYOFFS, OUTPUTS) FOR ALL OF THESE PROJECTS? (MAXIMUM OF 10.) 4 WHAT IS THE NAME OF BENEFIT 1? jobs WHAT IS THE NAME OF BENEFIT 2? export sales WHAT IS THE NAME OF BENEFIT 3? value added WHAT IS THE NAME OF BENEFIT 4? pollution TIME TO FILL , IN THE PROJECT-BENEFIT MATRIX. WHAT IS THE JOBS PAYOFF FOR ONE? 500 WHAT IS THE EXPORT SALES PAYOFF FOR ONE? 2 WHAT IS THE VALUE ADDED PAYOFF FOR ONE? 5 WHAT IS THE POLLUTION PAYOFF FOR ONE? 1 WHAT IS THE JOBS PAYOFF FOR TWO? ] 00 WHAT IS THE EXPORT SALES PAYOFF FOR TWO? 3 WHAT i s THE VALUE ADDED PAYOFF FOR TWO? 10 WHAT IS THE i POLLUTION PAYOFF FOR TWO? t •5 WHAT IS THE JOBS PAYOFF FOR THREE? |5Q WHAT IS THE EXPORT SALES PAYOFF FOR THREE? 5 WHAT IS THE VALUE ADDED PAYOFF FOR THREE? 30 WHAT IS THE POLLUTION PAYOFF FOR THREE? -•10 96 WHAT IS THE JOBS PAYOFF FOR FOUR? 300 WHAT IS THE EXPORT SALES PAYOFF FOR FOUR? 2 WHAT IS THE VALUE ADDED PAYOFF FOR FOUR? 5 WHAT IS THE POLLUTION PAYOFF FOR FOUR? -1 WHAT IS THE JOBS PAYOFF FOR FIVE? 200 WHAT IS THE EXPORT SALES PAYOFF FOR FIVE? 6 WHAT IS THE VALUE ADDED PAYOFF FOR FIVE? 10 WHAT IS THE POLLUTION PAYOFF FOR FIVE? -13 WHAT IS THE INITIAL REQUESTED LEVEL OF FUNDING FOR ONE? 1 WHAT IS THE MINIMUM ACCEPTABLE SUPPORT FOR ONE? 1 WHAT IS THE PROBABILITY OF SUCCESS OF ONE AT THIS FUNDING? ANSWER SHOULD BE IN INTEGRAL PERCENT — A NUMBER BETWEEN 0 AND 100. 80 WHAT IS THE MAXIMUM ACCEPTABLE SUPPORT FOR ONE? 2 WHAT IS THE PROBABILITY OF SUCCESS OF ONE AT THIS FUNDING? 100 WHAT IS THE INITIAL REQUESTED LEVEL OF FUNDING FOR TWO?5 WHAT IS THE MINIMUM ACCEPTABLE SUPPORT FOR TWO? 2 WHAT IS THE PROBABILITY OF SUCCESS OF TWO AT THIS FUNDING? 50 WHAT IS THE MAXIMUM ACCEPTABLE SUPPORT FOR TWO? 15 WHAT IS THE PROBABILITY OF SUCCESS OF TWO AT THIS FUNDING? 85 WHAT IS THE INITIAL REQUESTED LEVEL OF FUNDING FOR THREE? 15 WHAT IS THE MINIMUM ACCEPTABLE SUPPORT FOR THREE? 10 . WHAT IS THE PROBABILITY OF SUCCESS OF THREE AT THIS FUNDING? 70 WHAT IS THE MAXIMUM ACCEPTABLE SUPPORT FOR THREE? 20 WHAT IS THE PROBABILITY OF SUCCESS OF THREE AT THIS FUNDING? 98 WHAT IS THE INITIAL REQUESTED LEVEL OF FUNDING FOR FOUR? 2 WHAT IS THE MINIMUM ACCEPTABLE SUPPORT FOR FOUR? 1 WHAT IS THE PROBABILITY OF SUCCESS OF FOUR AT THIS FUNDING? 80 WHAT IS THE MAXIMUM ACCEPTABLE SUPPORT FOR FOUB? 4 WHAT IS THE PROBABILITY OF SUCCESS OF FOUR AT THIS FUNDING? 100 97 WHAT IS THE INITIAL REQUESTED LEVEL OF FUNDING FOR FIVE? 10 WHAT IS THE MINIMUM ACCEPTABLE SUPPORT FOR FIVE? 5 WHAT IS THE PROBABILITY OF SUCCESS OF FIVE AT THIS FUNDING? 70 WHAT IS THE MAXIMUM ACCEPTABLE SUPPORT FOR FIVE? 15 WHAT IS THE PROBABILITY OF SUCCESS OF FIVE AT THIS FUNDING? 95 HOW MANY DISTINCT RESOURCES ARE THERE FOR THE 5 PROJECTS (MAXIMUM OF 10.) 3 WHAT IS THE NAME OF RESOURCE 1? capital WHAT IS THE NAME OF RESOURCE 2? labor 1 WHAT IS THE NAME OF RESOURCE 3? labor 2 PLEASE FILL IN THE PROJECT-RESOURCES MATRIX. WHAT AMOUNT OF CAPITAL IS REQUIRED FOR ONE? 1 WHAT AMOUNT OF LABOR 1 IS REQUIRED FOR ONE? 300 WHAT AMOUNT OF LABOR 2 IS REQUIRED FOR ONE? 0 WHAT AMOUNT OF CAPITAL IS REQUIRED FOR TWO? 5 WHAT AMOUNT OF LABOR 1 IS REQUIRED FOR TWO? 100 WHAT AMOUNT OF LABOR 2 IS REQUIRED FOR TWO? 0 WHAT AMOUNT OF CAPITAL IS REQUIRED FOR THREE? 15 WHAT AMOUNT OF LABOR 1 IS REQUIRED FOR THREE? 0 WHAT AMOUNT OF LABOR 2 IS REQUIRED FOR THREE? 100 WHAT AMOUNT OF CAPITAL IS REQUIRED FOR FOUR? 2 WHAT AMOUNT OF LABOR 1 IS REQUIRED FOR FOUR? 0 WHAT AMOUNT OF LABOR 2 IS REQUIRED FOR FOUR? 50 WHAT AMOUNT OF CAPITAL i s REQUIRED FOR FIVE? 100 WHAT AMOUNT OF LABOR 1 IS REQUIRED FOR FIVE? 150 WHAT AMOUNT OF LABOR 2 IS REQUIRED FOR FIVE? 0 WHAT IS THE TOTAL MAXIMUM AVAILABLE FOR CAPITAL? 20 WHAT IS THE TOTAL MAXIMUM AVAILABLE FOR LABOR '1? 450 WHAT IS THE TOTAL MAXIMUM AVAILABLE FOR LABOR 2? 120 WOULD YOU LIKE TO DERIVE A UTILITY FUNCTION? yes THE FOLLOWING ARE PSEUDO-PROJECT PROFILES WITH BENEFITS ORDERED THE WAY YOU PRESENTED THEM. PLEASE SCORE EACH PSEUDO-PROJECT FROM 1 (WORST POSSIBLE) COBRESPONDING TO A BENEFIT PROFILE OF CORRESPONDING TO A BENEFIT PROFILE 40 1 2 - 1 9 TO 100 (BEST POSSIBLE) 1000 9 40 0 BENEFITS: 1 2 3 4 PSEUDO-•PROJECT 1: 716 2 5 - 1 0 SCORE? 39 PSEUDO- PROJECT 2: 507 4 22 -U SCORE? 59 PSEU DO-•PROJECT 3: 873 7 26 - 1 4 SCORE? 65 PSEUDO- PROJECT 4: 146 7 34 - 2 SCORE? 60 PSEUDO-•PROJECT 5: 53 8 13 0 SCORE? 51 PSEUDO-•PROJECT 6: 264 6 30 - 16 SCORE? 35 PSEUDO-•PROJECT 7: 895 3 39 - 3 SCORE? 76 PSEUDO-•PROJECT 8: 808 7 10 - 5 SCORE? 72 PS EU DO-•PROJECT 9: 644 7 19 - 1 5 SCORE? 51 PSEUDO-•PROJECT 10: 309 7 35 - 4 SCORE? 67 PSEUDO--PROJECT 1 1: 851 4 17 -12 SCORE? 57 PSEUDO-•PROJECT 12: 422 4 39 - 16 SCORE? 42 PS EU DO-•PROJECT 13: 988 5 . 35 - 1 5 SCORE? 65 PSEUDO-•PROJECT 14: 140 4 - 9 - 9 SCORE? 30 PSEUDO-PROJECT 15: 948 2 39 - 10 SCORE? 65 PSEUDO-PROJECT 16: 816 8 6 - 8 SCOBE? 66 FOLLOWING IS AN ORDERED DISPLAY OF THE PSEUDO-PROJECTS: PROJ # SCORE BENEFITS 1 76 895 3 39 - 3 2 72 808 7 10 - 5 3 67 • 309 7 35 - 4 4 66 816 8 6 - 8 5 65 873 7 26 - 1 4 6 65 988 5 35 - 1 5 7 65 948 2 39 - 1 0 8 60 146 7 34 - 2 9 59 507 4 22 - 4 10 • 57 851 4 17 - 1 2 11 51 53 8 13 0 12 51 644' 7 19 - 1 5 13 42 422 4 39 - 1 6 14 39 716 2 5 - 1 0 15 35 264 6 30 - 1 6 16 30 140 4 9 - 9 YOU MAY NOW CHANGE SCORES. WHICH PSEUDO-PROJECT'S SCORE DO YOU WISH TO CHANGE? A ZERO MEANS THAT NO MORE CHANGES ARE DESIRED. 4 WHAT IS 4 ' S NEW SCORE? 64 FOLLOWING IS AN ORDERED_DISPLAY OF THE PSEUDO-PROJECTS PROJ # SCORE B E N E f 1T b 1 76 895 3 39 - 3 2 72 808 7 10 - 5 3 67 309 7 35 - 4 4 65 873 7 26 - 1 4 5 65 988 5 35 - 1 5 6 65 948 2 39 - 1 0 7 64 816 8 6 - 8 8 60 146 7 34 - 2 9 59 507 4 22 - 4 10 57 851 4 17 - 1 2 11 51 53 8 13 0 12 51 644 7 19 - 1 5 13 42 4 22 4 39 - 1 6 14 39 716 2 5 - 1 0 15 35. 264 6 30 - 1 6 16 30 140 4 9 - 9 r i C ^ S E r D o S c T " 8 " 5 ™ ^ DO YOU WISH TO CHANGE? A ZERO MEANS THAT NO MORE CHANGES ARE DESIRED. 0+ :NAL BANKING OF PSEUDO-PROJECTS PROJ # SCORE BENEFITS 1 76 895 3 39 -3 2 72 808 7 10 -5 3 67 309 7 35 -4 4 65 873 7 26 -14 5 65 988 5 35 -15 6 65 948 2 39 -10 7 64 816 8 6 -8 8 60 146 7 34 -2 9 59 507 4 22 -4 10 57 851 4 17 -12 1 1 51 53 8 13 0 12 51 644 7 19 -15 13 42 422 4 39 -16 14 39 716 2 5 -10 15 35 264 6 30 -16 16 30 140 4 9 -9 COEFFS: 4.6E-02 4.9E 00 6.5E-01 1.4E 00 PSEUDO-PROJECT ACTUAL PREDICTED 1 76 76.64 2 72 70.38 3 67 65.20 4 65 70.82 5 ' 65 70.78 6 65 64.14 7 64 68.70 8 60 59.95 9 59 51.23 10 57 52.25 11 51 49.78 12 51 54.34 13 42 41.15 14 39 31.42 15 35 37.78 16 30 18.78 SCORING BY MISTAKES*DISTANCE METHOD: SCORE = 4 B-SQUARE = 383.8 CORRELATION BY PEARSON *S METHOD: R = 0.96 SPEARMAN CORRELATIONS: R-SUB-S = 0.94 DISTANCE*ERROR METHOD: TAU = 0.82 LEXICOGRAPHIC ORDERING OF BENEFITS: BENEFIT NUMBER OF ORDER ERRORS 1 6 2 4 3 7 4 7 IS THIS UTILITY FUNCTION ACCURATE ENOUGH? no WORST POINT IS 16 ANOTHER PSEUDO-PROJECT TO SCORE: 136 5 9 - 9 SCORE? 36 FOLLOWING IS AN ORDERED DISPLAY OF THE PSEUDO-PROJECTS PROJ # SCORE BENEFITS 1 76 895 3 39 - 3 2 72 808 7 10 - 5 3 67 309 7 35 - 4 4 65 87 3 7 26 - 1 4 5 65 988 5 35 - 1 5 6 65 948 2 39 - 1 0 7 64 816 8 6 - 8 8 60 146 7 34 - 2 9 59 507 4 22 - 4 10 57 851 4 17 - 1 2 11 51 53 8 13 0 12 51 644 7 19 - 1 5 13 42 422 4 39 - 1 6 14 39 716 2 5 - 1 0 15 36 136 5 9 - 9 16 35 264 6 30 - 1 6 YOU HAY NOW CHANGE SCORES. WHICH P S E U D O - P R O J E C T » S SCORE DO YOU WISH TO CHANGE? A ZERO MEANS THAT NO MORE CHANGES ARE DESIRED. 0 FINAL RANKING OF PSEUDO-PROJECTS: PROJ # SCORE BENEFITS 1 76 895 3 39 - 3 2 72 808 7 10 - 5 3 67 309 7 35 - 4 4 65 873 7 26 - 1 4 5 65 988 5 35 - 1 5 6 65 948 2 39 - 1 0 7 64 816 8 6 - 8 8 60 146 7 34 - 2 9 59 507 4 22 - 4 10 57 851 4 17 - 1 2 11 51 53 8 13 0 12 51 644 7 19 - 1 5 13 42 422 4 39 - 1 6 14 39 716 2 5 - 1 0 15 36 136 5 9 16 35 264 6 30 - 9 - 1 6 COEFFS: 4 . 5 E - 0 2 5 .0E 00 6 . 4 E - 0 1 1 .4E 00 PS ED DO-PROJECT ACTUAL PREDICTED 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 76 72 67 65 65 65 64 60 59 57 51 51 42 39 36 35 76 .22 70 .65 65.44 70 .99 70.64 63 .66 69 . 12 60 .25 51*23 52 .22 5 0 . 3 9 54 .68 4 1 . 1 1 31.34 23 .86 38 .08 SCORING BY MISTA KES*DISTANCE METHOD: SCORE = 7 R-SQUARE = 4 13.5 CORRELATION BY PEARSON'S METHOD: R = 0.95 SPEARMAN CORRELATIONS: R-SUB-S = 0.94 DISTANCE*ERROR METHOD: TAU = 0.82 LEXICOGRAPHIC ORDERING OF BENEFITS: BENEFIT NUMBER OF ORDER ERRORS 1 7 2 5 3 8 4 7 IS THIS UTILITY FUNCTION ACCURATE ENOUGH? no WORST POINT IS 15 THE FOLLOWING ARE PSEUDO-PROJECT PROFILES WITH BENEFITS ORDERED THE WAY YOU PRESENTED THEM. PLEASE SCORE EACH PSEUDO-PROJECT FROM 1 (WORST POSSIBLE) CORRESPONDING TO A BENEFIT PROFILE OF 40 1 2 - 1 9 TO 100 (BEST POSSIBLE) COR RESPON DING TO A BENEFIT PROFILE 1000 9 10 0 BENEFITS: 1 2 3 4 PSEUDO-PROJECT 1: 895 3 39 - 3 SCORE= 76 PSEUDO-PROJECT 2 : 808 7 10 - 5 SCORE= 72 PSEUDO-PROJECT 3: 309 7 35 - 4 SCORE= 67 PSEUDO-PROJECT 4: 873 7 26 - 1 4 SCORE= 65 PSEUDO-PROJECT 5: 988 5 35 - 1 5 SCORE= 65 PSEUDO-PROJECT 6 : 948 2 39 - 10 SCORE= 65 PSEUDO-PROJECT 7: 816 8 6 - 8 SCOBE= 64 PSEUDO-PROJECT 8: 146 7 34 - 2 SCORE= 60 PSEUDO-PROJECT 9: 507 4 22 - 4 SCORE= 59 PSEUDO-PROJECT 10: 851 4 17 - 1 2 SCORE= 57 PSEUDO-PROJECT 11: 53 8 13 0 SCORE= 51 PSEUDO-PROJECT 12: 644 7 19 - 1 5 SCORE= 51 PS EU DO-PROJECT 13: s 422 4 39 - 1 6 SCORE= 42 PSEUDO-PROJECT 14: 7 16 2 5 - 1 0 SCORE= 3 9 PSEUDO-PROJECT 15: 136 5 9 - 9 SCORE= 36 PSEUDO-PROJECT 16: 264 6 30 - 16 SCORE= 35 PSEUDO-PROJECT 1: 860 7 27 - 1 2 SCORE? 70 PSED DO-PROJECT 2: 121 9 13 - 1 1 SCORE? 59 PSEUDO-PROJECT 3 : 272 8 38 - 12 SCORE? 58 PSEUDO-PROJECT 4: 772 5 15 - 9 SCORE? 62 PSEUDO-PROJECT 5 : 624 8 11 - 1 3 SCORE? 54 PSEUDO-PROJECT 6: 653 4 30 - 1 7 SCORE? 41 PSEUDO-PROJECT 7 : 477 3 15 - 1 0 SCORE? 42 PSEUDO-PROJECT 8: 877 2 19 - 3 SCORE? 60 PSEUDO-PROJECT 9 : 635 6 5 - 1 9 SCORE? 21 PSEUDO-PROJECT 10: 618 7 28 -11 SCORE? 67 PSEUDO-PROJECT 11: 48 6 4 - 17 SCORE? 1 PSEUDO-PROJECT 12: 631 5 17 - 1 4 SCORE? 71 PSEUDO-PROJECT 13: 584 3 31 - 7 SCORE? 57 PSEUDO-PROJECT 14: 955 8 34 - 7 SCORE? 90 PSEUDO-PROJECT 15: 531 5 33 - 1 0 SCORE? 62 PSEUDO-PROJECT 16: 760 2 21 - 1 6 SCORE? 31 FOLLOWING IS AN ORDERED DISPLAY OF THE PI PROJ # SCORE BENEFITS 1 90 955 8 34 - 7 2 71 631 5 17 - 1 4 3 70 860 7 27 - 1 2 4 67 618 7 28 - 1 1 5 62 772 5 15 - 9 6 62 531 5 33 - 1 0 7 60 877 2 19 - 3 105 8 59 421 9 13 - 1 1 9 58 272 8 38 - 1 2 10 57 584 3 31 - 7 11 54 624 8 1 1 - 1 3 12 42 477 3 15 - 1 0 13 4 1 653 4 30 - 1 7 14 31 760 2 21 - 1 6 15 21 635 6 5 - 19 16 1 48 6 4 - 1 7 YOU MAY NOW CHANGE SCORES. WHICH PSEUDO-PROJECT'S SCORE DO YOU WISH TO CHANGE? A ZERO MEANS THAT NO MORE CHANGES ARE DESIRED. AN ORDINAL DOMINANCE CONFLICT IN SCORING HAS BE FN FOUND BETWEEN 2 AND 3. THESE TWO ARE INDICATED BY ASTERISKS IN THE FOLLOWING PRESENTATION. PLEASE CHANGE ONE OR BOTH SCORES SO THAT A CONFLICT WILL NOT OCCUR. PROJ # SCORE BENEFITS 1 90 955 8 34 - 7 2 71 631 5 17 -14 3 70 860 7 27 - 1 2 4 67 618 7 28 - 1 1 5 62 772 5 15 - 9 6 62 531 5 33 - 1 0 7 60 877 2 19 - 3 8 59 421 9 13 - 1 1 9 58 272 8 38 - 1 2 10 57 584 3 31 - 7 11 54 624 8 1 1 - 1 3 12 42 477 3 15 - 1 0 13 41 653 4 30 - 1 7 14 31 760 2 21 - 1 6 15 21 635 6 5 - 1 9 16 1 48 6 4 - 1 7 WHICH PSEUDO-PROJECT'S SCORE DO YOU WISH TO CHANGE > A ZERO MEANS THAT NO MORE CHANGES ARE DESIRED. 2 WHAT IS 2 'S NEW SCORE? 4 7 FOLLOWING IS AN ORDERED DISPLAY OF THE PSEUDO-PROJECTS: PROJ # SCORE BENEFITS 1 90 955 8 34 - 7 2 70 860 7 27 - 1 2 3 67 618 7 28 - 1 1 4 62 .772 5 15 - 9 5 62 531 5 33 - 1 0 6 60 877 2 19 - 3 7 59' 421 9 13 - 1 1 8 58 272 8 38 - 1 2 9 57 584 3 31 - 7 10 54 624 8 11 - 1 3 11 4 7 631 5 17 -14 12 42 477 3 15 - 1 0 13 41 653 4 30 - 1 7 14 31 760 2 21 - 1 6 106 15 21 635 6 5 - 1 9 16 1 48 6 4 - 1 7 YOU MAY NOW CHANGE SCORES. WHICH PSEUDO-PROJECT'S SCORE DO YOU WISH TO CHANGE? A ZERO MEANS THAT-NO MORE CHANGES ARE DESIRED. 0 FINAL RANKING OF PSEUDO-PROJECTS: PROJ # SCORE BENEFITS 1 90 955 8 34 - 7 2 70 860 7 27 - 1 2 3 67 618 7 28 - 1 1 4 62 772 5 15 - 9 5 62 531 5 33 - 1 0 6 60 877 2 19 - 3 7 59 421 9 13 - 1 1 8 58 272 8 38 - 1 2 9 57 584 3 31 - 7 10 54 624 8 11 - 1 3 11 47 631 5 17 - 1 4 12 42 477 3 15 - 1 0 13 41 653 4 30 - 1 7 14 31 760 2 21 - 1 6 15 21 635 6 5 - 1 9 16 1 48 6 4 - 1 7 BENEFIT 1, COEFFS: BENEFIT 2, COEFFS: BENEFIT 3 , COEFFS: BENEFIT 4, COEFFS: 0.0519567 4.6440487 0.6867285 - 0 . 8 2 5 6 5 3 1 0 .0300714, 3. 358241 1 , 0 . 5 3 4 6 4 0 3 , BREAKPOINT: 507.0000000 BREAKPOINT: 4.0000000 BREAKPOINT: 22.0000000 2 .1860657, BREAKPOINT: - 4 . 0 0 0 0 0 0 0 -PROJECT ACTUAL PREDICTED 1 76 78 .62 2 72 72 .03 3 67 7 0 . 0 7 4 65 6 4 . 6 9 5 65 64 .06 6 65 63 .27 7 64 66.32 8 60 59.41 9 59 63.33 10 57 52 .75 11 51 55.74 12 51 51 .42 13 42 41 .77 14 39 35 .53 15 36 2 7 . 5 5 16 35 35.46 LINEAR R-SQUARE = 413.5222 ; PIECEWISE LINEAR: = 168.978 CORRELATION BY PEARSON'S METHOD: R = 0 .98 SPEARMAN CORRELATIONS: R-SUB-S = 0 .96 D I S T A N C E * E R R O R M E T H O D : T A U = 0 . 8 8 L E X I C O G R A P H I C O R D E R I N G O F B E N E F I T S : B E N E F I T N U M B E R O F O R D E R E R R O R S SOLUTION 1 VALUE=80.4617: SUPPORT ONE WITH RESOURCES AS FOLLOWS: 1 .366666 OF CAPITAL 409.9995 OF LABOR 1 0 . OF LABOR 2 SUPPORT TWO WITH RESOURCES AS FOLLOWS: 1 .999999 OF CAPITAL 40. OF LABOR 1 0 . OF LABOR 2 SUPPORT THREE WITH RESOURCES AS FOLLOWS 14.25 OF CAPITAL 0 . OF LABOR 1 95 . OF LABOR 2 SUPPORT FOUR WITH RESOURCES AS FOLLOWS: 1 . OF CAPITAL 0 . OF LABOR 1 2 5 . OF LABOR 2 DO NOT SUPPORT F IVE . S O L U T I O N 2 V A L U E = 8 0 . 6 2 6 9 : S U P P O R T O N E W I T H R E S O U R C E S A S F O L L O W S : 1 . 1 1 6 6 6 5 O F C A P I T A L 3 3 4 . 9 9 9 3 O F L A B O R 1 0 . O F L A B O R 2 S U P P O R T T W O W I T H R E S O U R C E S A S F O L L O W S : 1 . 9 9 9 9 9 9 O F C A P I T A L 4 0 . O F L A B O R 1 0 . O F L A B O R 2 2 . 3 4 8 7 7 8 SUPPORT THREE WITH RESOURCES AS FOLLOWS 9.999999 OF CAPITAL 0 . OF LABOR 1 6 6 . 6 6 6 7 OF LABOR 2 SUPPORT FOUR WITH RESOURCES AS FOLLOWS: 1 . 8 8 3 3 3 2 OF CAPITAL 0 . OF LABOR 1 4 7 . 0 8 3 3 OF LABOR 2 SUPPORT FIVE WITH RESOURCES AS FOLLOWS: 5 . OF CAPITAL 7 5 . OF LABOR 1 0 . OF LABOR 2 SOLUTION 3 VALUE=78.061 1: SUPPORT ONE WITH RESOURCES AS FOLLOWS: 1.249999 OF CAPITAL 374.9995 OF LABOR 1 0 . OF LABOR 2 DO NOT SUPPORT TWO. SUPPORT THREE WITH RESOURCES AS FOLLOWS 12. 2045 OF CAPITAL 0 . OF LABOR 1 81.3636 OF LABOR 2 SUPPORT FOUR WITH RESOURCES AS FOLLOWS: 1.545455 OF CAPITAL 0 . OF LABOR 1 38. 6364 OF LABOR 2 SUPPORT FIVE WITH RESOURCES AS FOLLOWS: 5 . OF CAPITAL 75 . OF LABOR 1 0 . OF LABOR 2 SOLUTION 4 VALUE=77.3058: SUPPORT ONE WITH RESOURCES AS FOLLOWS: 1.499999 OF CAPITAL 449.9995 OF LABOR 1 0 . OF LABOR 2 109 DO NOT SUPPORT TWO. SUPPORT THREE WITH RESOURCES AS FOLLOWS: 14.25 OF CAPITAL 0 . OF LABOR 1 95. OF LABOR 2 SUPPORT FOUR WITH RESOURCES AS FOLLOWS: 1. OF CAPITAL 0 . OF LABOR 1 2 5 . OF LABOR 2 DO NOT SUPPORT F IVE . SOLUTION 5 VALUE=71.9458: SUPPORT ONE WITH RESOURCES AS FOLLOWS: 1.116665 OF CAPITAL 334.9993 OF LABOR 1 0 . OF LABOR 2 SUPPORT TWO WITH RESOURCES AS FOLLOWS: 1 .999999 OF CAPITAL 40. OF LABOR 1 0 . OF LABOR 2 DO NOT SUPPORT THREE. SUPPORT FOUR WITH. RESOURCES AS FOLLOWS: 3.999998 OF CAPITAL 0 . OF LABOR 1 99.9999 OF LABOR 2 SUPPORT FIVE WITH RESOURCES AS FOLLOWS: 5 . OF CAPITAL 75. OF LABOR 1 0 . OF LABOR 2 DO YOU WISH TO BEGIN ANOTHER RUN (B) OR END (E)? e MODEM TERMINATING NORMALLY.. . 110 The f i v e s o l u t i o n s provided by the a l g o r i t h m have s i m i l a r s u b j e c t i v e values but suggest very d i f f e r e n t funding p a t t e r n s . At t h i s po int the d e c i s i o n maker would be requ i red to choose between them on the bas is of any n o n - q u a n t i f i a b l e i n f o r m a t i o n or c r i t e r i a he may posess . S e n s i t i v i t y runs with d i f f e r e n t amounts of resources a v a i l a b l e would po int out any important d i s c o n t i n u i t i e s in the value of the o b j e c t i v e and f a c i l i t a t e the f i na1 dec i s i on. 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            data-media="{[{embed.selectedMedia}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0093051/manifest

Comment

Related Items