c , / COMPUTATIONAL METHODS FOR MARKOV DECISION PROBLEMS B . S c , Seoul N a t i o n a l U n i v e r s i t y , 1969 M . S c , Texas Tech U n i v e r s i t y , 197^ A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE-STUDIES (Management Science) We a c c e p t t h i s t h e s i s as c o n f o r m i n g THE UNIVERSITY OF BRITISH COLUMBIA -.MOON CHIRL SHIN by t o t h e r e q u i r e d s t a n d a r d August 1980 © Moon Chir l Shin, 1980 In p r e s e n t i n g t h i s t h e s i s in p a r t i a l f u l f i l m e n t o f the r e q u i r e m e n t s f o r an advanced degree at the U n i v e r s i t y o f B r i t i s h C o l u m b i a , I a g r e e t h a t the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r r e f e r e n c e and s t u d y . I f u r t h e r agree t h a t p e r m i s s i o n f o r e x t e n s i v e c o p y i n g o f t h i s t h e s i s f o r s c h o l a r l y p u r p o s e s may be g r a n t e d by the Head o f my Department or by h i s r e p r e s e n t a t i v e s . I t i s u n d e r s t o o d that c o p y i n g o r p u b l i c a t i o n o f t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be a l l o w e d w i t h o u t my w r i t ten pe rm i ss ion . Department o f ^^nMac//rnJbnxJ' Sui/fl Cf. The U n i v e r s i t y o f B r i t i s h Co lumbia 2075 Wesbrook Place Vancouver, Canada V6T 1W5 Date ABSTRACT In th i s thesis we study computational methods for f i n i t e d i s -counted Markov decision problems and f i n i t e discounted parametric Markov decision problems over an i n f i n i t e horizon. For the former problem our emphasis i s on f ind ing methods to s i gn i f i c an t l y reduce the e f fo r t required to determine an optimal po l i cy . We discuss the implementation of Porteus 1 scalar extrapolat ion methods in the modified pol icy i t e ra t i on algorithm and show that the resul ts using only a f ina l sca lar extrapolat ion w i l l be the same as those obtained by applying scalar extrapolat ion at each i t e ra t i on and then using a f i na l sca lar extrapolat ion. Action e l iminat ion procedures for pol icy i t e ra t i on and modified pol icy i t e ra t i on algorithms are presented. The purpose of these techniques i s to reduce the s ize of the act ion space to be searched in the improvement phase of the algorithm. A method for e l iminat ing non-optimal actions for a l l subsequent i terat ions using upper and lower bounds on the optimal expected tota l discounted return i s presented along with procedures for e l iminat ing actions that cannot be part of the pol icy chosen in the improvement phase of the next i t e r a t i on . A numerical comparison of these procedures on' Howard's automobile replacement problem and on a large randomly generated problem suggests that using modified pol icy i t e ra t i on together with one of the s ingle i te ra t ion e l iminat ion procedures w i l l lead to large savings in the computational time for problems with large state spaces. Modif icat ions of the algorithm to reduce storage space are also discussed. i i For the f i n i t e discounted Markov decision problems in which the reward vector i s parameterized by a sca lar we present an algorithm to determine the optimal pol icy for each value of the parameter within an i n t e r va l . The algorithm is based on using approximations of values to . resolve d i f f i c u l t i e s caused by roundoff er ror . A lso, several action e l imina-t ion procedures are presented for th i s problem. B i - c r i t e r i on Markov decision problems and Markov decision problems with a s ingle constraint are formu-lated as parametric Markov decision problems. A numerical comparison of algorithms with and without act ion e l iminat ion procedures i s carr ied out on a two c r i t e r i on version of Howard's automobile replacement problem. The resul ts suggest ;that the algorithm with one of the act ion e l iminat ion procedures w i l l lead.to e f f i c i e n t so lut ion of th i s problem. TABLE OF CONTENTS Page ABSTRACT i i LIST OF TABLES vi LIST OF FIGURES v i i ACKNOWLEDGMENT v i i i CHAPTER I. INTRODUCTION 1 CHAPTER I I . IMPROVED COMPUTATIONAL METHODS FOR DISCOUNTED MARKOV DECISION PROBLEMS 4 1. Introduction 4 2. Preliminaries 7 3. Scalar Extrapolations. . . . . . . 11 4. Elimination of Non-optimal Actions 23 5. Elimination of Actions for One Iteration . . . 28 6. Action Elimination Algorithms 34 7. Computational Results 46 8. Conclusions 53 CHAPTER I I I . COMPUTATIONAL METHODS FOR PARAMETRIC MARKOV DECISION PROBLEMS 56 1. Introduction 56 2. Preliminaries 58 i v Page 3. An Algorithm Based on Parametric Linear Programming 59 4. An Algorithm for Finding e -Optimal 68 Po l i c i e s . . . . . . . . . . 5. Action El iminat ion 73 6. Action El iminat ion Algorithms 76 7. Appl icat ion of Markov Decision Problems with One Parameter . 84 8. Computational Results 94 9. Conclusions and Extensions 96 REFERENCES 99 APPENDIX 102 v LIST OF TABLES Table Page 2.1 Data for the Automobile Replacement Problem 48 2.2 Comparison of Action El iminat ion Procedures with Pol icy I terat ion - Automobile Replacement Problem 51 2.3 Comparison of Action El iminat ion Procedures with Pol icy I terat ion - Randomly Generated Problem. . . 52 2.4 Computational Time 54 3.1 Comparison of Select ion Rules 97 3.2 Comparison of Action El imination Procedures 97 vi LIST OF FIGURES Figure Page 2.1 Implication of Proposit ion 2.6 30 2.2 Flow Chart of an Action El iminat ion Algorithm 35 v i i ACKNOWLEDGMENT I wish to thank Professor Martin L. Puterman for his help, advice and espec ia l ly for his careful reading of my several drafts of th is thes is ; Sharon Hal ler , for turning hundreds of pages of scrawls and correct ion into th is f i na l manuscript; and my wife Yaeree Oh for encouraging me throughout th is ordeal . In add i t ion, I am grateful to the National Science Engineering Research Council for the f inanc ia l support through Grant A-3609. vi i i 1. CHAPTER I INTRODUCTION In th i s thesis we consider computational methods for f i n i t e state and act ion i n f i n i t e horizon discounted Markov decis ion problems. In Chapter 2 we propose several methods to improve the e f f i c i ency of methods for solving these problems and in Chapter 3 present algorithms for solv ing f i n i t e discounted Markov decision problems parameterized by a s ingle sca lar . The po l icy i te ra t ion method • i s usual ly at t r ibuted to Howard [14]. Later, Puterman and Shin [26] and van Nunen [29,30] developed independently the modified pol icy i te ra t ion to reduce computation for problems with large state spaces. In Chapter 2 we are concerned with increasing the e f f i c i ency of the po l i cy i t e ra t i on and modified pol icy i te ra t ion procedures. Porteus and Totten [23] and Porteus [21] investigated the use of extrapo-l a t i on methods such as the t r i v i a l sca lar extrapolat ion, the t r i v i a l lower bound extrapolat ion, row sum extrapolat ion and delayed sca lar extrapolat ion to reduce the computational e f fo r t in the evaluation of expected tota l d i s -counted return of a Markov and semi-Markov chain. A l l of these extrapola-t ions reduce to sca lar extrapolat ions in the discounted Markov decision problem considered here. We discuss the implementation of these sca lar extrapolat ion methods in the modified pol icy i te ra t ion algorithm and show that the resu l ts using only a f i na l scalar extrapolat ion w i l l be the same as those obtained by applying sca lar extrapolat ion at each i t e ra t i on and then using a f i na l sca lar extrapolat ion. 2. To reduce computations in the improvement phase the idea of using bounds on the optimal return function to ident i fy actions that are not part of an optimal pol icy was introduced by MacQueen [16]. Then Porteus [19,20], Hastings and Mello [10], Hastings [11], Hastings and van Nunen [12] and Hu'bner [15] studied action e l iminat ion procedures for value i t e ra t i on algorithms and var iants. Gr inold [9] studied an action e l iminat ion procedure for po l icy i t e r a t i on . Motivated by the discovery in Puterman and Shin [26] that modified pol icy i t e ra t i on was considerably more e f f i c i e n t than value i t e r a -t ion and in many cases pol icy i t e r a t i on , we introduce bounds and action e l iminat ion procedures for the pol icy i t e ra t i on and modified pol icy i t e ra t i on algorithms.. A method for e l iminat ing non-optimal actions for a l l subsequent i te ra t ions using upper and lower bounds on the optimal expected tota l d i s -counted return i s presented along with procedures for e l iminat ing actions that cannot be part of the pol icy chosen in the improvement phase of the next i t e r a t i on . A numerical comparison of these procedures on Howard's [14] automobile replacement problem and on a large structured randomly generated problem suggests that using modified pol icy i t e ra t i on together with one of the s ingle i t e ra t i on e l iminat ion procedures w i l l lead to large savings in the computational time for problems with large state spaces. Modif icat ions of the algorithms to reduce storage space are also discussed. In Chapter 3 we study f i n i t e discounted Markov decision problems in which the reward vector i s parameterized by a sca lar and present an algorithm to determine the optimal po l icy for each value of the parameter with in an i n t e r va l . The algorithm is based on a modif icat ion of parametric l i near programming in which approximations of values are used to resolve d i f f i c u l t i e s caused by roundoff error . A lso, several act ion e l iminat ion 3. procedures are presented for th i s problem.. B i^cr i ter ion Markov decis ion problemsand Markov decis ion problems with a s ingle constraint are then formulated as parametric Markov decision problems. A numerical comparison of algorithms with and without act ion e l iminat ion procedures i s carr ied out on a two c r i t e r i on version of Howard's [14] automobile replacement problem. The resu l ts suggest that using the algorithm with one of the act ion .". el imination., procedures is an e f f i c i en t method for solv ing th is problem. 4. CHAPTER II IMPROVED COMPUTATIONAL METHODS FOR DISCOUNTED MARKOV DECISION PROBLEMS 1. Introduction. In th i s chapter we consider methods for increasing the e f f i c i ency of the pol icy i t e ra t i on (Howard [14]) and modified pol icy i t e ra t i on (Puterman and Shin [26] and van Nunen [31 g32]) algorithms for solv ing f i n i t e d i s -counted Markov decision problems over an i n f i n i t e horizon. The modified pol icy i t e ra t i on algorithm was suggested by Porteus [19] and Morton [18] and i t s properties were studied by van Nunen [31,32], Puterman and Brumelle [24,25], Puterman and Shin [26], van der Wal [30] and Rothblum [27]. In a ser ies of papers Porteus [21] and Porteus and Totten [23] invest igated the use of extrapolat ion methods to reduce the computational e f fo r t to evaluate the expected tota l discounted return of a Markov or semi-Markov chain. Porteus and Totten [23] sought improved i t e ra t i ve methods such as the t r i v i a l sca lar extrapolat ion and the t r i v i a l lower bound ext ra-polat ion to compute the expected discounted return in a Markov chain. Recently Porteus [21] studied other improved i t e ra t i ve schemes such as row sum extrapolat ion and delayed scalar extrapolat ion for Markov and semi-Markov chains. A relevant survey i s that of Porteus [22]. A l l of the 5. extrapolat ions reduce to sca lar extrapolat ions in the discounted Markov decision problem considered here.- The implementation of these sca lar extra-polation schemes in the modified pol icy i t e ra t i on algorithms i s studied in Section 3 and i t i s shown that the pol icy selected using modified pol icy i t e ra t i on with and without sca lar extrapolat ions i s ident ica l and the value obtained by the algorithm using only a f i na l scalar extrapolat ion w i l l give exactly the same resu l t as that obtained by using scalar extrapolat ions at each i t e ra t i on in the evaluation phase and then applying a f i na l sca lar extrapolat ion. However, when there are unequal row sums, such as in the case of semi-Markov decision problems, scalar extrapolat ions may have an important inf luence on the resu l t s . The idea of using bounds on the optimal return function to ident i fy actions that are not part of an optimal pol icy and reduce computations in the improvement phase was introduced by MacQueen [16]. At each i t e ra t i on of a value i t e ra t i on algorithm such actions are i den t i f i ed and discarded from the act ion set for al1 subsequent i t e ra t i ons . Porteus [19] and [20] strengthened MacQueen's bounds and extended them to more general decision processes. Later, Hastings and Mello [10] and Porteus [20] considered a modif icat ion of MacQueen1s and Porteus 1 [19] procedures that reduces storage requirements. Grinold [9] used bounds s im i la r to those of Porteus [20] to develop an act ion e l iminat ion procedure for pol icy i t e r a t i on . Much of th i s material i s surveyed in an a r t i c l e by White [35]. In the above papers, act ions are el iminated only i f they are nonoptimal. Hastings [11] proposed a procedure to el iminate an act ion for one or more i te ra t ions a f te r which, i t could reenter the decision set. His work was concerned with the average reward c r i t e r i a and was extended 6. by Hubner [15] to discounted problems with arb i t ra ry discount factors . Hastings and van Nunen [12] appl-ied Hubner's resul ts to the case where the discount factor was less than one and through an example showed that th i s procedure el iminated more actions than that of Hastings and Mello [10]. In a l l of these papers, with the exception of that of Grinold [9] , e l iminat ion procedures were developed for value i t e ra t i on algorithms and var iants . Motivated by the discovery in [26] that modified pol icy i t e ra t i on was considerably more e f f i c i e n t than value i t e ra t i on and in many cases pol icy i t e r a t i on , we introduce bounds and e l iminat ion procedures for the modified pol icy i t e ra t i on and pol icy i t e ra t i on algorithms. The bounds are motivated by the Newton method representation for modified pol icy i t e ra t i on in [26]. However, as pointed out by Porteus in a pr ivate communication, they can be derived from resul ts in [20]. In Section 4 a method for e l iminat ing non-optimal actions for a l l subsequent i te ra t ions using upper bounds and lower bounds on the optimal expected tota l discounted return at each i t e ra t i on of modified pol icy i te ra t ion and pol icy i t e ra t i on algorithms i s presented. In Section 5 pro-cedures for e l iminat ing actions that cannot be part of the pol icy chosen in the improvement phase of the next i t e ra t i on are developed. The re l a t i on -ship with the act ion e l iminat ion procedure of Hastings and van Hunen [12] and Hubner [15] i s discussed. Section 6 presents modified pol icy i t e ra t i on algorithms which incorporate the above act ion e l iminat ion procedures. The storage require-ments of these algorithms are discussed and some methods to reduce them are proposed. Numerical resul ts appear in Section 7 where a comparison of these procedures on Howard's [14] automobile replacement problem and on a sparse randomly generated test problem with a large act ion set are presented. 7. 2. Pre!iminaries. We consider a discrete time finite state and action discounted Markov decision problem over an infinite horizon. Let I be the set of finite states labeled by integers i=l,2,«««,N. For each i e l , we have a set of finite actions denoted by k=l , 2 ,• ••• . The policy space is defined by the Cartesian product of each action set, that is, A = x x ••• x K^ . By a policy 6eA, we mean 6(i) e IC. Then a strategy TT is defined by a t 1 2 t sequence {6 e A, t=1,2,••-} = {5-,6-,•••,6 ,•••} and is an element of strategy space denoted by IT. = A x A x A strategy TT is stationary i f Let us consider a sequential decision problem where at each stage t = 0 ,1 ,2 ,««* we observe the state of the system and then chooser action. More precisely, i f the system is in state iel at stage t and an action kelC is selected, the expected one period reward is r(i,k) and the system moves to state jel at stage t+1 with probability P(i,j,k). We assume that I P(i,J,k) = 1 for iel and keK. and Max Max |r(i ,k) | < M < + °°. iel keK. Denote the one period discount rate by X e [ 0 , 1 ) . For 6eA define P^ by P 6(i,j) = P(i,j,6(i)) and r 6 by r 6 ( i ) = r( i , 6 ( i ) ) . Let V denote the Banach space of bounded real valued functions on I with norm ||v|| = Max |v(i)|. iel For each TTEII, let v77 be the expected total discounted reward vector. For each stationary strategy TT = (6,6,«««)» let v be the expected 8. to ta l discounted reward vector. Then the problem we study is that of f inding k ~k an optimal pol icy 6 eA and a v e V such that (1) v* = v 6 = Max v 6 . 6eA k k k That there ex is ts an optimal stat ionary strategy TT = (6 ,6" ,•••)> i . e . , * there ex is ts an optimal pol icy 5 , in the discounted Markov decision problem, i s shown in Blackwell [2,3] and Denardo [6] . For a given ve V and SeA define the l i near operators B, T, B g and T~ mapping V to V by (2) Bv ( i , S ( i ) ) = r ( i , 6 ( i ) ) + A I P ( i , j , S ( i ) ) v( j ) - v ( i ) j e l (3) T v ( i , 6 ( i ) ) E r ( i , 6 ( i ) ) + A _ P ( i , j , 6 ( i ) ) v ( j ) j e l (4) B 6 v ( i ) = Bv ( i , 6 ( i ) ) (5) T .v ( i ) = T v ( i , 6 ( i ) ) and define the pol icy 6 y and the operator H:V -> V by (6) Hv = B. v = Max B.v 6 v 6eA 6 where 6 y i s the pol icy that maximizes the r ight hand side of (4) for each i at v. This maximum is attained at each v e V since A is f i n i t e . 9. •k The problem defined by (1) i s equivalent to f ind ing a v e V and a 6 e A tha t - sa t i s fy (7) B v* = Hv* = 0. 6 This is the usual functional equation of dynamic programming writ ten in the form of Puterman and Brumelle [25]. For c l a r i t y we w i l l use 6 , r , P , B and T instead of 6 , o J n' n' n' n n v o n r , P g , B g and T- . The symbol 1 w i l l denote a column vector of v„ v v n n n ones and for ueV we adopt the notation sp[u] = Max u( i ) - Min u ( i ) . i e l i e l Our analysis w i l l be based on the fol lowing modified pol icy i t e ra t i on algorithm. IV I n i t i a l i z a t i o n Select e>0, to be used as a stopping c r i t e r i on . Set n = 0 and define 6 eA by o J r Q ( i ) = r ( i,6 Q ( i ) ) = Max k e K_ r ( i , k ) , i e l . Set (8) v 0 ^ (Min r ( i ) ) . 1. i e l 10. I I . Evaluation Phase: Calculate v +-j = T ™ + \ n i t e r a t i v e l y by successive approximations using T . (The order of the algorithm, m, can be predetermined or else chosen to be (9) Min{s:sp[T^ + 1v n - T ^ ] < e}.) Increment n by one. I I I . Improvement Phase: Find <5n such that B v = B. v = Maxx .B xv = Hv . n n 6 n n 6eA 6 n n I f sp[B v 1 < e, go to IV. Otherwise return to I I . r L n n - 3 IV. Final Extrapolat ion: Set v = v + B v + B v • 1 where B v = Min B v ( i ) . n n n 1 - A n n n n . T n n^ ' i e l Some comments about th i s algorithm are in order. The se lect ion of v Q in the i n i t i a l i z a t i o n phase i s to ensure that Hvn > 0 for a l l n. As demonstrated by Puterman and Shin [26, Thm. 1] th i s ensures monotone convergence of the sequence of i terates {v n>. In pract ice the order of the algorithm, m, w i l l usual ly be determined by (9) . However th i s c r i t e r i a might be too str ingent and require excessive ca lcu la t ions . I f the order 11 is prespec i f ied, though not necessar i ly f ixed from i t e ra t i on to i t e r a t i on , th i s algorithm is s im i l a r to that presented in [26]. In that set t ing we observed that when m = +°°, the algorithm was pol icy i t e ra t i on and that m = 0 gave value i t e ra t i on . As a consequence of the stopping c r i t e r i a , and the extrapolat ion in IV the sequence {v^Jwi l l terminate with a value function v that i s eA(l-X)"^ opt imal. This i s shown in Proposit ion 2.4. The fol lowing proposit ions summarize some resul ts that w i l l be needed in the sequel. The proof of the f i r s t appears in Puterman and Shin [26] and the proof of the second in Puterman and Brumelle [25]. Proposit ion 2.1. rm+l Suppose v +-j = T_ v... Then n n m m l l 0 ) V l " v n + l0 <«V V n - »„ + l0 ("V % Proposit ion 2.2. (Support Inequal ity) For any u, veV (11) Hu > Hv + (AP - I ) (u-v) . v 3. Scalar Extrapolat ions. To reduce the computational e f fo r t to evaluate the expected tota l discounted return in Markov and semi-Markov chains Porteus and Totten [23] and Porteus [21] developed the extrapolat ion methods ca l l ed t r i v i a l lower 12. bound extrapolat ion, t r i v i a l sca lar extrapo lat ion, row sum extrapolat ion and delayed sca lar extrapolat ion. In the Markov chains a r i s ing from d i s -counted Markov decision problems a l l of the above extrapolat ion methods reduce to sca lar extrapolat ion methods. For value i t e ra t i on Porteus [20] mentioned ". . .When there are equal row sums, such an extrapolat ion would have an en t i re l y predictable inf luence; that i s , the resu l ts without such extrapolat ion would d i f f e r by a constant (pointwise) and the same po l i c i es would be 'opt imal ' at each i t e r a t i on . . . ." and suggested using the extrapo-la t i on at each i t e r a t i on . But in th is sect ion we show that the algorithm in Section 2 w i l l give the same resu l t as that using sca lar extrapolat ions at each i t e ra t i on and then applying the f ina l sca lar extrapolat ions for modified pol icy i t e r a t i on . For semi-Markov chains or unequal row sums Markov chains these resul ts do not hold. extrapolat ion schemes into the modified pol icy i t e ra t i on procedure. Suppose in the evaluation phase of the algorithm in Section 2 we ca lcu late v +-j i t e r a t i v e l y using one of these extrapolat ion schemes with v n as the i n i t i a l value. Then the resu l t ing i t e r a t i ve extrapolat ion method can be repre-sented by: In th is sect ion we discuss the incorporation of these sca lar (12) v, s ,s-l n (13) s v n where o v v n n 13. s X s-1 (14) = J]Zxf V n ^ for the t r i v i a l lower bound extrapolat ion [TLB.E], (15) cSn = v [B~v S _ 1 + B v S _ 1 ] 1 for the t r i v i a l sca lar n C.\\-A) n n n n extrapolat ion and other extrapolat ions [TSE], J U L i £ i n n and B v £ = Max B v ! ( i ) . n n - T n n v ' Note that the quan t i t y ; c n defined in (14) and (15) i s a sca lar times a column vector of ones. Hence these are ca l led sca lar extrapolat ions. The fol lowing lemma gives the value of v s and B v s a f ter the n n n sth i t e ra t i on of these i t e ra t i ve extrapolat ion methods. Lemma 2.1. At i t e ra t i on s of the above i t e r a t i ve extrapolat ion methods (16) v S = T Sv + I X s - ^ c £ n £=1 n <17> V n ' ^ n ' V n ^ ' n Proof. For s = 1 , from (12) and (13) 1 x / • 1 v„ = T v + c n n n n Assuming (16) i s true for s = p, from (3), (.12) and (13) 14. P+l X P , P+l n n n n J-LU . D+l T ( T V + V A ^ V ) + c[ n v n n ^ nJ n Therefore (16) is true for any s, From (10) and (16) s s-1 S-1 [v n + I (AP ) XB v + y A S~\: ] = r + AP v - v + y (AP ) B v - Y (AP ) B v n n n n ^ n' n n Z _ Q K A V n n £=T n s-1 + U-1) J 2=1 Define (AP ) SB v = Min (AP ) s B v ( i ) • n' n n i e i n n n (AP ) SB v = Max (AP ) SB v ( i ) . • n' n n . T v n' n n v ' From Lemma 2.1 we obtain the fol lowing resu l t which exhib i ts the re lat ionsh ip between T s +^v and v s + ^. n n n Lemma 2.2. At i t e ra t i on s of the i t e ra t i ve extrapolat ion method (18) v S + 1 = T S + 1 v + c + . v ' n n n n+1 where V l = T T A J ^ V V n ] f o r T L B E : n + l = 2TTAT ^ r f V n + (AP n ) S B n v n ] T for TSE. Proof. From (16) F i r s t we consider TLBE. From (14) and (17) C n + 1 =TAT B v S 1 (1-A) n 16. (20). = s ( r « » s " ' ! ' For TSE i t fol lows from (15) and (17) that + ( X P n ) S B n V n ? + (A -1) ( j^'V) A f ' A S " £ c £ . £=1 n Subst i tut ing (20) or (21) into (19) We show in the., f o i l owing theorem that for the stopping e r i t i e r i on (9) in the evaluation phase we. w i l l have the same order of the alqorithm, m , at n i t e ra t i on n whether or not we use any of the i t e ra t i ve extrapolat ion methods. (2D) A 'n+1 17. Theorem 2.1. At i t e r a t i on s in the evaluation phase (22) s p ( v S + 1 - v S ) = sp (T S + 1 v - T Sv ) . Proof. From (16) s+1 s / XS+1 T S v = s P ( T n v n " V n } since adding a constant to each of the vectors does not change sp[u]. Lemma 2.2 and Theorem 2.1 imply that using these sca lar extrapolat ion schemes at each i t e ra t i on in the evaluation phase of the algorithm w i l l give the same resu l t as using these sca lar extrapolat ions only at the end of the i t e ra t i on in the evaluation phase. This resu l t appears in Theorem 2.2 below. Suppose we use extrapolat ions only at the end of i t e ra t i ng T n in the evaluation phase. Then we ex i t the evaluation phase of i t e ra t i on n+1 with value v l! 1 + 1 defined by 1 8 . ( 2 3 ) v = T v v ' n+1 n n W vn+l = V l + C n +1 and vo = V where <25> Vi=wUP/vi'forTLBE <26> cn +l " 2TTXT » p n > V n + <*Pn> W 1 f°r TSE and m is the order of algorithm in the evaluation phase at iteration n of the algorithm. For c lar i ty m is used instead of m n. Let v n be the value obtained in the evaluation step without the above extrapolation scheme. Then the following theorem gives the relation-ship between v .j and v n + ^ . Theorem 2 . 2 . At iteration n of the algorithm (27) V l = V l + V l where : 4.1 = - r r V r U p A v 1 for T L B E n+1 ( 1 - X ) v n' n n and ' 4.1 = o / / , \ [ U P A v + ( X P ) m B v ] 1 for T S E . n+1 2 ( 1 - X ) L V n n n v n' n n J Proof. Note v' = v . o o For TLBE, at n = 0 from (18) with s v l = v l + c l Suppose (27) i s true at i t e ra t i on (n-1), i . e . , (28) v = v + c . n n n Then at i t e ra t i on n+1 from (10), (23) and (24) ' = i m + 1 v' + c' n+1 n n n+1 m , (29) = v + I (AP ) B v' + c . n £=0 n n n n + 1 From (28) B v' = B (v + c ) n n n v n n' = r + A P v + A P c - v - r •n n n A r n L n n L n r + A P v - v +AC - c n n n n n n (30) = B v + (A-1) c n n • ' n From (30) cLi = T A T (^ P A V' n+1 (1-A) • _ j r _ n _ n / 0 , \ , m+l (31) = c n + 1 - A c n S u b s t i t u t i n g ( 1 0 ) , ( 2 8 ) , (30) and (31) i n (29) we o b t a i n , m 0 , V n + 1 = V n + C n + I ( X P n ) LB V + ( X - l ) C ] + C ^ - A™ ' C n+1 n n ^ Q • n ' L n n • ' n J n+1 n m m . m . = v n + I ( V B n V V l + C n + ^ W V H X > c n n £=0 n n n n 1 n £=0 n £=0 n , m+l - A C n = v + Y (AP ) l v + c £=0 n n = v n + l + C n+1 • For TSE a l l o f t h e r e s u l t s a r e t h e same as f o r TLBE e x c e p t t h a t V i = 2TAI + ( A P n ) \ v ; ] 1 ^ ^ - [ ( A P ^ V n ^ ^ ^ ^ n ^ ^ P n ^ V n ' 1 + A m ( A - l ) c ] oTTTY KXP )mB V + (AP )mB V ] 1 + A m + 1 C 2(1 -A) n ' n n v j r n i n J r (32) = c + v + A m + 1 c v ' n+1 - n T h e r e f o r e u s i n g (32) i n s t e a d o f (31) the r e s u l t f o l l o w s , 21. The fol lowing theorem shows that at each i t e ra t i on we w i l l se lect the same pol icy and have the same test value, s p [B n v n ] , for the stopping c r i t e r i on in the improvement phase of the algorithms whether we use scalar extrapolat ions or not. Theorem 2.3. At i t e ra t i on n of the algorithm <33> V n = V n + ^ c n (34) {6 n e A|B 6 V ; = Max B ^ } = {6 neA|B v n = Max B 6 v n } n 6eA n (35) sp[B v ' ] - sp[B v ] . \ / n n K L n n J Proof. From (27) V n = V v n + cn> = r 6 + XP 6 (v n + c n ) - ( V n + c n ) = r 6 + . x P 6 v n - v n + ( A - l ) c n = B 6 v n + ( A - l ) c n . (34) and (35) fol low from (33). Corol lary 2.2. In the f i na l extrapolat ion step of the algorithm v = v' + B v 1 + B v' 1 n n n (1-X) n n = v + B v + - P T Y Y -Bnv 1 for TLBE n n n (1-X) n n and » • vn + V n + 2TW ^Vn + V P ' = vn + V n + [ + V n ] 1 f o r T S E ' Proof. For TLBE from (27) v = V + V h + T W B n V n 1 = vn + c n + Bn< vn + Cn> + TT^xT B n ( v n + c n } ? = v + c + B v + (X- l )c + T T T T B V 1 - Xc n n n n n (1-X) n n n = v + B v + / -A \ B v 1 . n n n (1-X) n n For TSE from (27) * = v n + B n v n + 2 (^x7 C V i + W ? = v n + c n + V v n + cn> + ZT^xJ t B n ( v n + V + V V c n ^ ^ 23. .= v + c + B v + (A- l )c + O / T \ \ [B v 1 + (A- l)c n n n n • ' n 2(1-X; L n n • ' n + B v 1 + (A-l)c ] n n n = v + B v + TrrrTY [B^T + B v ] 1 + c + (A-l )c - Ac n n n 2(1-A) L n n n n J n • ' n n = v + B v + ^ r A - r [B~v~ + B v ] 1 . n n n 2(1-A) n n n n J Hence Corol lary 2.2 implies that the value obtained by the algorithm using only the f ina l sca lar extrapolat ions w i l l give exactly the same resu l t as that obtained by using sca lar extrapolat ions at each i t e ra t i on in the evaluation phase and then applying the f i na l sca lar extrapolat ion. 4. El iminat ion of Non-optimal Actions. We w i l l use the fol lowing notation in th is sect ion Bv( i ,k) = r ( i , k ) + A I P ( i , j , k ) v ( j ) - v ( i ) . Throughout th i s sect ion except where noted we w i l l assume that {vn> is generated by the modified pol icy i t e ra t i on algorithm described in Section 2. Proposit ion 2.3. For each n 24. <36> \ + B „ v n ) 1 < v„ + B nv„ + B n v n ) l < v n + 1 + B n v n ) l < v < v + B v + (—• B v )1 - n n n V l - A n n y - n 1-A n nJ where v , n = T m + 1 v . n+1 n n Proof. Applying the support inequal i ty (11) at v gives B n v n > Hv + (AP 6* - I)(v n - v ) Rearranging terms and noting Hv = 0 implies that v * 5 V n + (I-AP J " 1 B n v n < v n + B n v n + ( ^ B n v n ) T < V + ( y ^ B V )1 - n V l - A n n' ic This establ ishes the upper bounds for v . To derive the lower bounds, use the support inequal i ty at v n to obtain H v * > V n + U P n - I ) C v * - v n ] . Rearranging terms we f ind that 25. v* > v + (I-AP ) " ' B v + I (AP ) S B v - n v n/ n n v i r n n m . 0 0 . = v + t (AP ) SB v + Y (AP ) SB v " s=0 n n n s=m+l n n n ,m+l - n+1 M-A n w The l a s t inequal i ty follows from the de f i n i t i on of v +-|. Next observe that from (10) i t follows that ,m+l ~ m ,m+l Vn+i + ( T ^ - B v )1 = v + B v + y ( A P ) B v + ( f ^ - B v )1 n+1 M-A n n' n n n ^ n y n n M-A n n ; > v + B v + (y^r- B v )1 > v + ( ~ - B v )1 - n n n M-A _ n j r - n V l - A n nJ which establ ishes the resu l t . When {vn> is generated by pol icy i t e r a t i on , in which case m = +°°, * * then the f i r s t lower bound for v in (36) becomes v n +^ < v . The extreme lower bound and upper bound for v in (36) were establ ished by MacQueen [16] for {v } generated by value i t e r a t i on . For pol icy i t e r a t i on , Grinold [9] establ ished the second lower bound for v*. Our resul ts for modified pol icy i t e ra t i on are obvious extensions of these ea r l i e r resu l t s ; however, the t ightes t lower bound for v* i s new. As a consequence of the stopping c r i t e r i a and the f ina l-extrapo-l a t i on of the algorithm in Section 2 the sequence {vn> w i l l terminate with -1 a value function v that i s eA(l-A) optimal. This i s shown in the fol lowing proposi t ion. 26. Proposit ion 2.4. A -1 v i s e A( l - A ) optimal, i . e . , v( i ) < v*(i) < v( i ) + e X ( l - A ) " 1 for a l l i e I. Proof. From (36) v ( i ) = tflH^il^B^^v-O) and v*(i) < v „ ( i ) + B v ( i ) + -n\, TV - n v ' n n v ' (T-A) n n v n < 1 ) + BnV 1) + T T ^ V n + TTXT (Vn" " Vn> v( i ) + e A ( l - A ) " 1 . Action e l iminat ion i s based on the fol lowing resu l t of MacQueen [16]. The short proof is included for completeness. Proposit ion 2.5. * I f Bv ( i , k ) < 0 then k i s a non-optimal act ion in state i . Proof. Suppose k* is an optimal act ion in state i . Then Bv*(i,k*) = 0 > Bv ( i , k ) . Therefore k i s non-optimal. Combining these two proposit ions we obtain an act ion el iminat ion procedure for modified pol icy i t e r a t i on . Theorem 2.4. Suppose at i t e ra t i on n that (37) r(1.k) U l P (1 . j.k)y n(J) + (^ B ^ ) < v n + 1 (1) + ( A _ Then action k e IC is non-optimal in state i . Proof. From (36) and (37) r ( i , k ) + x I P ( i , j , k ) v* ( j ) < r ( i , k ) + x £ P( i , j , k ) [ v n ( j ) j e l ~ j e l n + ( T ^ B X ) ! ] V l - X n n J = r ( i , k ) + A I P ( i , j , k ) v ( j ) j e l n M - X n n' , X m + 1 < W 1 * + ' I T Vn) < v Hence Bv ( i , k ) < 0 and by Proposit ion 2.5 i t fol lows that k e K i i s non-optimal. In the case of pol icy i t e ra t i on Corol lary 2.3. Suppose {vn> is generated by pol icy i t e ra t i on and that (38) r ( l , k ) + X J P ( i , j , k ) v n ( j ) + ( ^ BMT) < v n + 1 ( i ) Then k e L i s non-optimal in state i . 28. Grinold [7] replaced the term v .j on the righthand side of (38) by e i ther v n ( i ) + ( ^ B n v n ) or v n ( i ) + B n v n ( i ) + ( ^ B n v n ) in his act ion e l iminat ion algorithms. Since v ^ i s a t ighter lower bound for v than e i ther of these two bounds and i s calculated pr ior to an act ion e l iminat ion step (see Section 6) , i t i s surpr is ing that he did not use the stronger tes t , (38), for act ion e l iminat ion. 5. El iminat ion of Actions for One I te ra t ion . The purpose of th i s sect ion i s to develop procedures to el iminate actions for a s ingle i t e ra t i on of a modified pol icy i t e ra t i on algorithm. We use the fol lowing notat ion. D v = v - v ? > 1 n,J> n+£ n * - 1 D V n , £ = f £ * D v n , £ ^ The fol lowing lemma is of a technical nature and important in our development. Lemma 2.3. At i t e ra t i on n (39) B v , = (XP ) m + 1 B v x ' n n+1 v n' n n where Tm+1 v . i = T v n+1 n n Proof. From the de f in i t i on of B v^ and (10) B n V l = r n + . (^n" 1 ) V l m s=0 B v + (XP ) m + 1 B v - B v = (XP ) m + 1 B v n n v n' n n n n v n' n n The fol lowing proposit ion i s the basis for a s ingle i t e ra t i on ; e l iminat ion procedure. Proposit ion 2.6. Suppose (40) B v n + 1 ( i , k ) < [ ( x P n ) m + 1 B n v n ] ( i ) . Then k E K. cannot a t ta in the maximum in state i in the improvement phase of i t e ra t i on n+1. Proof. From (39) and (40) we obtain B v n + 1 ( i , k ) < [ ( X P n ) m + 1 B n v n ] ( i ) = B n v n + 1 ( i ) < H v n + 1 ( i ) From which we conclude the resu l t . Figure 2.1 shows schematical ly the impl icat ion of Proposit ion 2.6. I f B v n + 1 ( i , k ) < B n v n + 1 ( i ) = [ ( X P n ) m + 1 B n v n ] ( i ) then B v n + 1 ( i , k ) < Figure 2.1. Implication of Proposition 2.6. 31. B n v n + ^( i ) < Hv n +-|( i) as shown in Figure 2.1. Therefore k E K. can be el iminated at i t e ra t i on n+1. Noting that for pol icy i t e r a t i on , m = +°°, we conclude the fo l lowing: Corol lary 2.4. Suppose {v n} i s generated by pol icy i t e ra t i on and B v n + 1 ( i , k ) < 0. Then act ion k E K. cannot be optimal in state i at i t e ra t i on n+1. To make use of Proposit ion 2.4 in act ion e l iminat ion we must obtain bounds for both expressions in (40) that are eas i l y computable from the quant i t ies ava i lab le at the completion of an evaluation step. These bounds are obtained in the course of proving the fol lowing theorem which gives an act ion el iminat ion procedure. Theorem 2.5. Suppose at i t e ra t i on (n+1) that for some I< n (41) r.(i,k) +Xl P ( i , j , k ) v ( j ) + I D7 1 - v n + 1 ( i ) < X(XP ) m B n v n j e l p=£ H ' Then act ion k e i s non-optimal in state i at i t e ra t i on (n+1) Proof. From (40) and (41) B v n + 1 ( i , k ) = r ( i , k ) + X J P ( i , j , k ) v n + 1 ( j ) - v n + 1 ( i ) 32. r ( i , k ) + X J P ( i . j , k ) [ v n + 1 ( j ) - v A ( j ) + v A ( j ) ] - v N + 1 ( 1 ) r ( i , k ) + X I P ( i , j , k ) v „ ( j ) + X I P( i , j ,k )Dv„ n j e l 1 j e l £,n-£+l v / I ) n+1 (42) < r ( i , k ) + A £ P ( i , j , k ) v p ( j ) + X J T J v ^ - v . J j ) j e l 34 p=£ p ' ' n 1 < X(XP )mB v = XP [(XP )mB v • l ] ( i ) v n' n n n • n y n n J V ; < [ ( A P n ) m + 1 B n v n ] ( i ) Therefore by Proposit ion 2.4, k e K. i s non-optimal in state i of i t e ra t i on (n+1). An a l te rnat ive act ion e l iminat ion algorithm can be based on the fol lowing co ro l l a ry . Corol lary 2.5. Suppose at i t e ra t i on (n+1) that for some £ < n /„(j) + XDV„ „,T - V , n ( l ) <- A l A f I O V . £ w / £,n-£+l n+1v ' • n' n n (43) r ( i , k ) + X I P ( i , j , k ) v 0 ( j v0 n _ 0 + 1 - v ^ O ) X (XP j m B v j e l Then action k e IC i s non-optimal in state i at i t e ra t i on (n+1). n Proof. Using Dv , instead of Y Dv n , in (42) gives the resu l t *,n-x,+ i p = £ p,i 33. Single i t e ra t i on act ion e l iminat ion algorithms for pol icy i t e ra t i on can be obtained by replacing the bound on the r ight hand side of (41) or (43) by 0. We discuss the re la t ion of the resul ts in th i s sect ion to those of Hastings and van Nunen [12] and Hubner [15]. Hastings and van Nunen have proposed the fol lowing test for detecting non-optimal actions at i t e ra t ion n + I of value i t e r a t i on ; an act ion k E K. i s non-optimal in state i at i t e ra t i on n + I i f To invest igate the equivalence of th i s test and the test based on (41) we set m=0 and £=n in (41) and £=1 in (44). This i s because with m=0, modified pol icy i t e ra t i on i s value i t e ra t i on and with £=1 Hastings and van Nunen's algorithm el iminates actions at the next i t e ra t i on . We f ind in th is case that the two procedures are equivalent. Hubner1s e l iminat ion procedure [15] i s based on replacing the l a s t term on the l e f t hand side of (44) by n+£-l (44) v . + 1 ( i ) - [ r ( i , k ) + X I P ( i , j , k ) v ( j ) ] - X I sp[v , -v ] > 0 (45) 5, ik S P [ V P + 1 " V where • 6...= Max X(l - I M i n ( P ( i , j , k ) , P ( i , j , k ' ) ) ) 1 K k 'elO j e l and 34. 5 = Max A(l - I M i n ( P ( i , j , k ) , P ( i ' , j , k ' ) ) ) . ( i , k ) , ( i ' , k ' ) j e l He showed that 6.^ <6< X and that a tes t based on subst i tut ing (45) into (44) i s more e f f i c i e n t than using (44) d i r e c t l y . In appl icat ions 6.^ and <5 are usual ly approximated with more eas i l y computable quant i t ies . 6. Action El iminat ion Algorithms In th i s sect ion we apply resul ts from the previous sections to develop computational procedures. Po l icy i t e ra t i on and modified pol icy i t e ra t i on algorithms including act ion e l iminat ion are represented by the flow chart in Figure 2 .2 . For the act ion e l iminat ion step we se lect one of the procedures below. The f i r s t , which i s due to Grinold [9] applies to pol icy i t e ra t i on only, while the remainder have versions for both pol icy i t e ra t i on and modified pol icy i t e r a t i o n . Except in the f i r s t case we state the modified pol icy i t e ra t i on versions of these procedure. I . (Pol icy I terat ion) Suppose that at i t e ra t i on n+1 (46) r(1.k) • A I P(I.J.k) v n ( j ) < » n (1) + ^ B nv n - ^ 1HT . j e l Then act ion k e in state i i s non-optimal and need not be considered in the improvement phase of i te ra t ions n+1, n+2,««« . I I . Suppose that at i t e ra t i on n+1. 35. I n i t i a l i z a t i o n Pol icy Evaluation Action El iminat ion 1 r Pol icy Improvement Over Reduced Action Set NO YES Figure 2.2. Flow Chart of an Act ion El iminat ion Algorithm. 36. (47) r ( i , k ) + X Y P ( i , j , k ) v ( j ) < v , , ( i ) + ^ 8 v ~ B v v ' v ' . L r v ' n ' n+1 ' 1 - X . n n 1-X n n Then a c t i o n k e K. i n s t a t e i i s non-opt imal and need not be considered i n the improvement phase o f i t e r a t i o n s n+1 , n + 2 , " « . In the case o f p o l i c y i t e r a t i o n se t m = +°° i n (46) to ob ta in (48) . r ( i , k ) + A I P ( i , j , k ) v n ( j ) < v n + 1 ( i ) - ^ B M T . j e l The above two procedures de tec t non-opt imal ac t i ons and e l i m i n a t e them from a l l subsequent i t e r a t i o n s . The f o l l o w i n g two procedures e l i m i n a t e ac t i ons a t the subsequent i t e r a t i o n o n l y . Consequent ly, i f a c t i o n k e K.j has been e l i m i n a t e d i n s t a t e i a t i t e r a t i o n n, the t e s t q u a n t i t y r ( i , k ) + X _ P ( i , j , k ) v ( j ) has not been computed and hence i s not a v a i l -ab le f o r a c t i o n e l i m i n a t i o n a t i t e r a t i o n n+1. Because o f t h i s f a c t , we in t roduce the f o l l o w i n g . Def ine E n ( i ) to be the se t o f ac t i ons t h a t have been e l i m i n a t e d i n s t a t e i a t i t e r a t i o n n and se t E ( i ) = 0. Def ine F ( i ) to be the se t o f ac t i ons t h a t have been e l i m i n a t e d f o r i t e r a t i o n s £+1,£+2,»««,n and l a s t eva luated a t i t e r a t i o n I, i . e . , n ( ^ ( i ) ) c where the s u p e r s c r i p t c i n d i c a t e s complement. F £ , n ( i ) = n E ( i ) P=J>+I P The e l i m i n a t i o n procedures are as f o l l o w s : 37. I I I . Suppose that at i t e ra t i on n+1 and for k e K. - E ( i ) l n^ ' (49) r ( i . k ) + A J P ( i , j , k ) v n ( j ) + A D v n J < v n + 1 ( i ) + . . X U P ^ V n ' Then k e E n + 1 ( i ) . Further, i f for I = n-1,n-2,•••,1 and k e F ( i ) the fol lowing A/) n holds: (50) r ( i , k ) + X I P ( i , j , k ) v ( j ) + A I D7 < v n + 1 (1 )+ A(AP )mB v . j e l * p=£ P' 1 n + l —Q Q-H Then k e E n + 1 ( i ) . IV. Replace (50) by (5!) r ( i , k ) • X ^ P ( i . j . k ) + X D » t > n . t + 1 < » n + 1 ( 1 ) + X ( A P n ) \ v n , For pol icy i t e r a t i on , the quantity A(AP n ) m B n v n i s replaced by 0 in (49)-(51) We now discuss the implementation of procedures III or IV in the modified pol icy i t e ra t i on algorithm. We i n i t i a l i z e by se lect ing v 38. according to (8). We next evaluate v-, = Tm+^v . I t may be computationally e f f i c i e n t to perform act ion e l iminat ion before the f i r s t improvement. That is because for act ion e l iminat ion we compute (52) r ( i , k)+A I P(i , j,k) v ( j ) + XD7Q , - V l ( i ) . j e l ' Since v Q i s a constant vector we see that (52) i s equal to r ( i , k ) + (Min r (i).).+ \W - v ( i ) i e l ' which can be evaluated by fewer ca lcu lat ions than the test quantity r ( i , k ) + A £ P ( i , j , k ) v-,(j) in the subsequent improvement step. But i t w i l l not el iminate many actions since (Min r (i))+TJvn , i s usual ly larger than AT P(i , j , k )v , (j)., I - A . e I o u,i • • . •Therefore action e l iminat ion procedures w i l l s tar t from i t e ra t i on 1. n The dif ference between III and IV i s the quant i t ies I Dv -, .. p=£ P' 1 a n d D v £ , n + l - £ • N o t e t h a t ( 5 3 ) D v £ , n + l - £ = ^ax [ v n + 1 ( i ) - v £ ( i ) ] = Max[ I (v (1) - v ( i ) ) ] l c l n=0 r 1 V i e l " i e l p=l n n _ I Max[v ( i ) - v ( i ) ] = I Dv p=£iel H H p=£ P = 39. Hence IV w i l l el iminate more actions than III but require stor ing the vectors v l >'• * * » v n« I n pract ice th is algorithm converges quite quickly and th i s addit ional storage w i l l be f a i r l y i n s i gn i f i c an t . Note also that the bound used on the r ight hand side of (49), (50) and (51) is X ( X P n ) m B n v n = X M i n ^ V V n ^ 1 ^ -i e l I t is evident from the proof of Theorem 2.4 that a better lower bound would be Min ( [ ( A P n ) m + 1 B v ] ( i ) ) . i e l However th i s would require an addit ional ca lcu la t ion af ter evaluating v n + i while A ( A P n ) m B n v n i s readi ly ava i lab le . This i s because (54) (AP )mB v = T m + 1 v - T m v . v ' • n' n n n n n n Both terms on the r ight hand side of (54) are compuated in the evaluation step. We present the deta i led algorithm incorporating act ion e l iminat ion procedure I I , III and IV into the modified pol icy i t e ra t i on procedure. Minor changes w i l l give the corresponding algorithms for pol icy i t e r a t i on . We w i l l i n i t i a l i z e an addit ional array, Flag ( i , k ) , to ind icate whether an act ion has been el iminated. I t i s subscripted as n only for the purpose of s tat ing the algorithm. STEP 1) I n i t i a l i z a t i o n : Select e > 0 to be used as a stopping c r i t e r i on . Set n = 0 f ind 6 e A such that o r ( i ) = r ( i , 6 ( i ) ) = Max r ( i,k), i e I. keK^ Set V M < M i n r 0 ( i ) ) . . ? . Calculate v-, = T m + 1 v„ where m i s determined by l o o J s+1 s m = min{s: sp[T v - T v ] < e} ^L n n n n J -Set n = n+1 and F l ag n ( i , k ) = 0 for a l l i e l and k e K . . Go to step 2 STEP 2) Improvement Phase: For each i e I, k e K.. i f F l ag n ( i , k ) < 0 ca lculate Tv ( i , k ) Otherwise do not ca lcu late T v n ( i , k ) . Find 6 n such that Tv ( i , 6 ( i ))= Max {Tv n ( i , k ) | F l ag n ( i , k ) < 0} n n k e K n n Compute B n v n and B ^ . I f sp(B v ) <.:e go to step 5. Otherwise go to step 3. STEP 3) Evaluation Phase: v ., = T m + 1 v where m is determined by n+1 n n m = Min {s: sp[T* + 1 v n - T* < e} . Go to step 4. STEP 4) El imination Phase: In procedure I I ; m+1 I f F l a g n ( i , k ) < 0 and V n ( i . k ) - v n + 1 ( i ) < TH I^TVn " Tj^If B^T, then F l a g n + 1 ( i , k ) = 1. Otherwise F lag n + 1 ( i , k ) = F l a g n ( i , k ) . Set n = n+1 and go to step 2. In procedure I I I ; S e t T v n + 1 ( i , k ) = T v n ( i , k ) + T5vn > 1 . I f , T v n + 1 ( i , k ) < v n + 1 ( i ) + X(XP n)'" B n v n then F l a g n + 1 ( i , k ) = 1. Otherwise F l a g n + 1 ( i , k ) = 0. Set n = n+1 and go to step 2. In procedure IV; I f F l ag n ( i , k ) < 0 and Tv^ i . k ) + XDT^ < v n + 1 ( i ) + A(XP n ) m B n v n then Flag + 1 ( i , k ) = n. I f F lag n (k ,k) < 0 and Tv n ( i , k ) + XD7 ^ > v n + 1 ( i ) + X(XP n ) m B n v n then F l a g n + 1 ( i , k ) = F l ag n ( i , k ) = 0. 42. I f F l ag n ( i , k ) = £ > 0 and Tv £ ( i , k ) + D V ^ ^ < v n + 1 ( i ) + A(XP n ) m B n v n then F l a g n + 1 ( i , k ) = A. If F l a g n ( i , k ) = £ > 0 and-Tv^i ,k) + * D v M = A + 1 > v n + l ( i ) + X(XP n ) n i B n v n then F l a g n + 1 ( i , k ) = 0. Set n = n+1 and go to step 2. STEP 5) Final Extrapolat ion: Set v = v n + B n v n + J^J B ^ T and stop. Some comments about th is algorithm are in order. For procedure III we do not have to store the ind icator of the las t evaluated i t e r a t i on , £, because the only term added to the l e f t hand side of (50) at each i t e r a -t ion is always Dv , . On the other hand, the las t evaluated i t e r a t i on , £, n, i must be stored for procedure IV because the value of ADv^ n „ £ + i must be stored for procedure IV because the value of ADv^ n ^ in (56) depends on v n +-| and v^. Therefore in procedure IV, i f k e F^ n ( i ) then F l ag n ( i , k ) = £. In the above algorithms with act ion e l iminat ion procedures we must store the N x M array, Tv ( i , k ) , and the N x M array, F l ag ( i , k ) , in addit ion to the storage space required for the algorithm without an act ion el iminat ion procedure. Here N i s the number of states and M i s the number of actions in each state assuming each state has an equal number of act ions. Furthermore in procedure IV we require addit ional space to store the vectors v - | » * " > v n and in procedure III addit ional space to store the vector v . Therefore at least 2N x M addit ional storage space is required for the algorithms above. 2 But compared to the space to store the N x M array f o r the t rans i t i on p robab i l i t i e s , P ( i , j , k ) , th i s addit ional storag^requtrement i s not too s i gn i f i can t . 43. In the cases where even 2N x M addit ional words cannot be put into core storage, we can u t i l i z e another method to decrease the 2N x M spaces to N x M spaces by using P ( i , l , k ) as F l ag ( i , k ) . Changes to be made in the algorithms above are as follows:. ' (a) We i n i t i a l i z e P ( i , j , k ) = A P ( i , j , k ) < 1 for i e l , j e l , ke^ Drop X in the ca lcu la t ion of Tv n ( i , k ) and Bv ( i ,k"). (AP )mB v becomes < Pn>\V (b) In the improvement phase: I f P ( i , l , k ) < 1 ca lcu late T v n ( i , k ) . Otherwise do not ca lcu late T v n ( i , k ) . Find 8^ such that T v n ( i,6 n ( i ) ) = Max { T v j i , k ) |P(i ,1 ,k) < 1} n n keK. n i (c) In the el iminat ion phase: For procedure I I ; I f P ( i , j , k ) < 1 and (47) i s s a t i s f i ed then P ( i , l , k ) = 1 + P ( i , l , k ) . Otherwise do not change P ( i , l , k ) . For procedure I I I ; I f T v n + ] ( i , k ) < v n + 1 ( i ) A (P n ) m B n v n then P ( i , l , k ) = 1. + P ( i , l , k ) . Otherwise, P ( i , l , k ) = P ( i , l , k ) - 1 i f P ( i , l , k ) > 1 and P ( i , l , k ) = P ( i , l , k ) i f P ( i , l , k ) < 1. For procedure IV; I f P ( i , l , k ) < 1 and (49) i s s a t i s f i ed then P ( i , l , k ) = n + P ( i , l , k ) . I f P ( i , l , k ) < 1 and (49) i s not s a t i s f i ed then P ( i , l , k ) = P ( i , l , k ) . I f P ( i , l , k ) > 1 and (51) with l = greatest integer in P ( i , l , k ) i s s a t i s f i ed then P ( i , l , k ) = P ( i , l , k ) . I f P ( i , l , k ) > 1 and (51) with l.= greatest integer in P ( i , l , k ) i s not s a t i s f i ed then P ( i , l , k ) = P ( i , l , k ) - £. 44. For a problem with large state and act ion spaces, the t rans i t i on p robab i l i t i e s may be stored in a f i l e or on a tape and read at each improve-ment phase. In th i s case, i f we use P(i,1,k) as F lag( i ,k ) we have to read P ( i , l , k ) from a f i l e or a tape to check whether an act ion i s e l im i -nated or not. The operation of reading from a f i l e or tape into core storage i s quite i ne f f i c i en t and requires considerable CPU time. This can be avoided by stor ing F l ag ( i , k ) and Tv( i ,k) in the same array for procedures II and I I I . Storing F l ag ( i , k ) and Tv( i ,k) in the same array may not be done for procedure IV since we have to ident i fy the l as t evalu-ated i t e r a t i on , £, of each act ion el iminated. To store FIag( i ,k) and Tv( i ,k) in the same array, we use a sca lar constant v that is greater than ic — Max v ( i ) . This sca lar value v can be calculated eas i l y in the i n i t i a l i z a -Ie l t ion phase as fol lows: r ( i ) = r ( i , 6 ( i ) ) = Max r ( i , k ) , i e I. 0 0 keK. 1 * v = ,y ,x Max r ( i ) > Max v ( i ) . [ l ~ X > i e l 0 i e l Using v above the changes necessary in the algorithm for procedure II and III are as fo l lows: (a) In improvement phase: I f F l ag n ( i , k ) < 0 ca lcu late T y n ( i 5 k ) and set F lag h( . i ,k) = Tv n ( i , k ) - v «0. (b) In e l iminat ion phase: For procedure I I ; I f F l a g n ( i , k ) < 0 and F l ag n ( i , k ) + v < v + - j ( i ) + A m + 1 „ .. X rny V n - IAT V n •then ^ W1^ =1 45. Otherwise F l a g n + 1 ( i , k ) = F l a g n ( i , k ) . For procedure I I I ; I f F l agy l ' , k ) < 0 and A p ( i , k ) = F l ag n ( i , k ) + * + X ° Y l < V l ( i ) + X t X P n ) m B n v n t h e n F 1 a V l ( i ' k ) = V l ( i ) + XXP n ) m B n v n - A n ( i , k ) > 0. I f F l ag n ( i , k ) < 0 and A n ( i , k ) > v n + ] ( i ) + X(XP n ) m B n v n then F l a g n + 1 ( i , k ) = Flag ( i , k ) . I f F l a g y l , k ) > 0 and C n(1,k) = - F l a g n ( i , k ) + v n ( i ) + X(XP R , 1 ) \ ^ + * % f l < v n + 1 ( i ) + M A P n ) m B n v n then F l a g n + 1 ( i , k ) = v n + l ( 1 ) + " C n ( i ' k ) > ° ' I f F l ag n ( i , k ) > 0 and C n ( i , k ) > v n + 1 ( i ) + X(XP n ) m B n v n then F l a g n + 1 ( i , k ) = 0. Some comments about the two storage saving methods are in order. ( i , k ) and ( i , k ) are jus t the terms on the l e f t hand side of (49) and (50) respect ive ly . A ( i , k ) and C n ( i , k ) are used for c l a r i t y but are not required to be stored. For the value i t e ra t i on algorithm an a l ternat ive approach to reduce storage has been developed by Hastings and Mello [10] and Porteus [20]. Their methods el iminate fewer actions than those based on the best ava i lab le bounds because they use less t ight bounds to reduce storage. Using P ( i , l , k ) as Flag ( i , k ) the i r methods require no addit ional storage space. While in the case where the P ( i , j , k ) array is stored in a f i l e or tape and a F lag( i ,k) - type array i s used to indicate whether or not an act ion has been el iminated, a modif icat ion s im i l a r to that described above using F lag( i , k ) and Tv( i ,k) in the same array with t ighter bounds than in Hastings and Mello [10] and Porteus [20] w i l l el iminate more actions without increasing storage. For modified pol icy i t e ra t i on a s im i l a r approach to that in [10] and [20] would el iminate far fewer actions than the methods herein because v , i s much greater than 46. 7. Computational Results To determine the e f f i c iency of the algorithm described in the previous sect ion we solved Howard's [14] automobile replacement problem and a randomly generated problem with discount rates .8333, .86956, .909 and .9532. These discount rates were chosen because they were also used by Grinold [9] in test ing his procedure. We f i r s t describe the automobile replacement problem in d e t a i l . Automobile Replacement Problem. (Howard [14, p. 54].) Consider the problem of automobile replacement over a time interva l of ten years. The state of the system, i , i s described by the age of the car in three-month periods; i running from 1 to 40. Every three months we decide whether to keep our present car (k=l) or to trade i t in for a car of age k-2; k running from 2 to 41. In order to keep the number of states f i n i t e , a car of age 40 remains a car of age 40 forever ( i t i s con-sidered to be essent ia l l y worn out) . The actions ava i lab le in each state are: to keep the present car for another quarter (k=l) or to buy a car of age k-2, k=2,3,•••,41. The problem has 40 states with 41 actions in 40 each state. Hence there are 41 possible stat ionary s t rateg ies . The data supplied are the fo l lowing: C- i s the cost of buying a car of age i : i s the trade- in value of a car of age i ; E. is the expected cost of operating a car of age i. unt i l i t reaches age i+1; and P.. i s the probab i l i ty that a car of age i w i l l survive to be i+l without incurr ing a p roh ib i t i ve l y expensive repa i r . 47. The probab i l i ty defined here is necessary to l im i t the number of states. A car of any age tha thas a hopeless breakdown i s immediately sent to state 40. Natural ly , P 4 Q = 0. Using our ea r l i e r notation we have r ( i , k ) = -E.J for k = 1 r ( i , k ) = T. - C k _ 2 - E k _ 2 for k > 1 P ( i , j , k ) = 4 1 - P. j = i+1 j = 40 f others for k = 1 P ( i , j , k ) = P k-2 J" - i + 1 1 - P kJ2 j = 40 [ for k > 1 0 others 1. The numerical values for these parameters are l i s t e d in Table 2.1 The Randomly Generated Problem. The randomly generated problem had 40 states and 100 actions in each. I t was generated as fo l lows: the data for the expected one period rewards, r ( i , k ) , were generated from a truncated normal d i s t r i bu t i on , the t rans i t i on p robab i l i t i e s , P ( i , j , k ) , were selected so that there were non-zero entr ies at three random locat ions in each state for each action and the non-zero probab i l i t i es were generated from a uniform d i s t r i bu t i on . Table 2.1 Data for the Automobile Replacement Problem Age i n Cost Trade-in Operating Survival Age i n Cost Trade-in Operating Survival Periods Value Expense Probabi l i ty Periods Value Expense Probab i l i ty i C. T. E. P. i c. T. E. p. i ^ 1 •v 0 $2000 $1600 $50 1.000 1 1840 1460 53 0.999 21 $345 $240 $115 0.925 2 1680 1340 56 0.998 . 22 330 225 118 0.919 3 1560 1230 59 0.997 23 315 210 121 0.910 4 1300 1050 62 0.996 24 300 200 125 0.900 5 1220 .. 980 65 0.994 25 290 190 129 0.890 6 1150 910 68 0.991 26 280 180 133 0.880 7 1080 840 71 0.988 27 265 170 137 0.865 8 900 710 75 0.985 28 250 160 141 0.850 9 840 650 78 0.983 29 240 150 145 0.820 10 780 600 81 0.980 30 230 145 150 0.790 11 730 550 84 0.975 31 220 140 155 0.760 12 600 480 87 0.970 32 210 135 160 0.730 13 560 430 90 0.965 33 200 130 167 0.660 14 520 390 93 0.960 34 190 120 175 0.590 15 480 360 96 0.955 35 180 115 182 0.510 16 440 330 100 0.950 36 170 n o 190 0.430 17 420 310 103 0.945 37 160 105 205 0.300 18 400 290 106 0.940 38 150 95 220 0.200 19 380 270 109 0.935 39 140 87 235 0.100 20 360 255 112 0.930 40 130 80 250 0 R.A. Howard, "Dynamic Programming and Markov Processes." M.I.T. Press, Cambridge, Massachusetts, 1960. PAGE 4 9 HAS BEEN OMITTED 50. The computer program for the randomly generated problem is in the Appendix. The randomly generated problems in Puterman and Shin [26] were not used since the optimal po l i c ies for these problems were found in one or two \ i t e ra t i ons . In the problem used here the structure varied considerably from pol icy to pol icy and consequently required more e f fo r t to solve. This was advantageous for invest igat ing the properties of our algorithm. We solved these problems using pol icy i t e ra t i on and modified pol icy i t e ra t i on alone and with each of the action el iminat ion procedures. The evaluation phase was terminated by using (9) with e = 0.1 This value of e was also used for stopping the algorithm. A l l ca lcu lat ions were performed on the Univers i ty of B r i t i s h Columbia AMDAHL 470 computer using the codes in the Appendix. Results reported in Tables 2.2. and 2.3 are the f ract ion of actions el iminated at each i t e r a t i on . These resul ts are based on using pol icy i t e r a t i on . We found that when modified pol icy i t e ra t i on was used resul ts did not d i f f e r in every case. In the cases where d i f fe rent resu l ts were obtained the modified pol icy i t e ra t i on resul ts are included in parentheses. For modified pol icy i t e r a t i on , we did not use procedure I which was c lea r l y dominated by procedure I I . I t i s in terest ing to note that in a l l cases except those i n d i -cated by * in Table 2.2 the number of actions el iminated by procedures III and IV increased at each i t e r a t i on . The reason for the decrease in those cases n marked by * is that T Dv - . i s not a very good upper bound for v ,, - v„. p=£ P>' n+1 I When the better upper bound, Dv^ ^ was employed th i s monotonicity property was pfeserved (see (53)). I t was in terest ing to note that although . Table 2.2 Comparison of Action Elimination Procedures with Policy I terat ion -Automobile Replacement Problem (Numbers are fract ion of actions eliminated at i terat ion n, asterisked cells are described in text . ) .8333 .86956 .909 .9532 I I I I l l IV I I I I l l IV I I I I l l IV I I I I l l IV 1 .0 .0 .4573 .4573 .0 .0 .4707 .4707 .0 .0 .4152 .4152 .0 .0 .3409 .3409 2 .0561 .0707 .5518 .6659 .0 .0 .5037 .6311 .0 .0 .4183 .5530 .0 .0 .3^77 (.3171) .4470 3 .2549 .2683 .7152 .8561 .0982 .1000 (.0994) .5829 .8238 .0 .0 .5366 .7726 .0 .0 .3T28 (.3134) .5311 4 opt. opt. opt. opt. .4585 .4604 (.4616) .8183 .9110 .0823 .0835 .5671 .8695 .0 .0 .6628 (.6622) .7927 5 opt. opt. opt. opt. .1628 .1652 (.1640) .7732 .8921 .0 .0 .4354 .8195 6 .3738 .3744 (.3738) .8512 (.8506) .9372 opt. opt. opt. opt. opt. opt. opt. opt. CXI Table 2.3 Comparison of Action Elimination Procedures with Policy I te ra t i Randomly Generated Problem (Numbers are fractions of actions eliminated at i terations n) n x H .8 333 .86956 .909 .9532 I I I I I I IV I I I I l l IV I I I I l l IV I I I I l l IV 1 .0 .0 .6972 .6972 .0 .0 .6525 .6525 (.6520) .0 .0 .6255 .6255 .0 .0 .5667 .5667 2 .0100 .0445 (.0345) .7620 (.7617) .9163 (.9160) .0 .0010 (.0) .7592 .9150 (.9152) .0 .0 .7445 (.7438) .9122 (.9117) .0 .0 .5800 (.5797) .8530 3 .6005 .6915 (.6797) .8920 (.8917) .9765 .3833 .5427 (.4947) .8872 (.8877) .9765 (.9763) .0510 .1513 (.0963) .8817 (.8820) .9752 (.9750) .0 .0 .8547 .9710 4 .9602 .9663 (.9645) .9540 (.9545) .9877 .9027 .9065 (.9055) .9338 .9858 (.9860) .9197 .9238 (.9217) .9575 (.9580) .9877 (.9880) .6005 .6542 (.6130) .9433 (.9422) .9877 5 .9890 .9890 .9865 .9900 opt. opt. opt. opt. opt. opt. opt. opt. opt. opt. opt. opt. 6 opt. opt. opt. opt. 53. the number of actions el iminated at each i t e ra t i on increased in the re-maining cases, we could not conclude that an act ion el iminated at a par t i cu la r i t e ra t i on would remain el iminated at subsequent i t e ra t i ons . This i s the reason for def in ing the sets F £ n ( i ) for procedures III and IV. For the automobile replacement problem, we also combined pro-cedure I I , with III and IV. In each case we obtained the same resul ts as using procedures III and IV alone. The CPU times to solve problems using various procedures are shown in Table 2.4. Reduction of CPU times using procedures I I I and IV were s i gn i f i c an t in each problem. For instance, in the randomly generated problem with X = .8333, pol icy i t e ra t i on with procedure IV took only 34.8 per cent of the CPU time taken by pol icy i t e ra t i on without any act ion el iminat ion procedure. Modified pol icy i t e ra t i on was slower than pol icy i t e ra t i on in every case. This i s due to the small state spaces of the problems. This issue i s discussed in deta i l in Puterman and Shin [26, p. 1134]. 8. Cone!usions. We were very encouraged by our computational resu l t s . As i n d i -cated in Table 2.2 and 2.3 procedures III and IV were very e f fec t ive in decreasing the s ize of the act ion space to be searched during an improve-ment step. In problems with a large number of act ions, th i s w i l l decrease computational requirements considerably, i . e . , i f there are N states and M actions then without act ion e l iminat ion each improvement step w i l l require 2 MN mul t ip l i ca t ions while i f 100a per cent of the actions have been e l im i -2 nated, only (l-a)MN mul t ip l i ca t ions would be required. Also as indicated Table 2.4 Computational Time CPU Time (Sees.) for the auto- CPU Time (Sees.) for the mobile replacement problem randomly generated problem X \ °c Modified Pol icy Modified Pol icy Pol icy Iterat ion Iterat ion Pol icy I terat ion I terat ion * .683 .812 2.422 2.457 .. I .682 - 1.549 _ .8333 II .652 .789 1.457 1.538 III .432 .564 .861 .919 IV .429 .534 .844 .884 * .856 1.061 2.015 2.076 I .820 - 1.629 _ .86956 II .796 1.025 1.531 1.622 III .501 .717 .852 .920 IV .476 .678 .815 .860 * 1.197 1.492 2.024 2.077 I 1.171 - 1.759 _ .909 II 1..134 1.420 1.666 1.758 III .660 .944 .863 .935 IV .586 .840 .823 .874 * 1.040 1.494 2.013 2.103 I 1.090 - 1.910 -.9532 II 1.063 1.542 1.830 1.926 III .717 1.224 .968 1.045 IV .648 1,099 .872 .927 (* indicates resu l ts without action el iminat ion procedures.) 55. in Table 2.4 we obtained s i gn i f i c an t reduction in computation time by inc luding act ion act ion el iminat ion procedures III and IV. For problems with large state spaces, Puterman and Shin [26] found modified pol icy i t e ra t i on to be considerably more e f fec t ive than pol icy i t e ra t i on and value i t e ra t i on with a l l i t s var iants. We recommend using modified pol icy i t e ra t i on together with procedure IV for solv ing d i s -counted Markovian decis ion problems with large state spaces and pol icy i t e ra t i on together with procedure IV for problems with small state spaces and large act ion spaces. 56. CHAPTER III COMPUTATIONAL METHODS FOR PARAMETRIC MARKOV DECISION PROBLEMS 1. Introduction. In th i s chapter we study f i n i t e discounted Markov decision problems (MDP) in which the reward vector i s parametrized by a scalar and develop algorithms to solve them. The algorithms use dynamic programming methods based on properties of the opt imal i ty equation and resu l ts in the previous chapter on act ion e l iminat ion. Without act ion el iminat ion the method i s s im i la r to the parametric simplex algorithm of l inear programming ( c f . Dantzig [5]) . Because of the sens i t i v i t y of resul ts to roundoff error a modif icat ion based on approximations to the expected to ta l discounted returns is also presented. These algorithms are of interest because b i - c r i t e r i on MDP and MDP subject to a s ingle constra int can be formulated as problems with the reward vector including a s ingle parameter. Henig [13] invest igated a general c lass of dynamic programs with vector c r i t e r i on and presented condit ions which imply that the set of po l i c i es with nondominated tota l returns can be characterized and approxi-mated by the set of a l l nondominated stat ionary po l i c i e s . Viswanathan, Aggarwal and Nair [33] and Henig [13] have suggested solut ion procedures for MDP with vector c r i t e r i on based on solut ion procedures for vector / c r i t e r i on l inear programming methods. Later, White and Kim [34] reformulated the MDP with vector c r i t e r i on as a spec ia l l y structured pa r t i a l l y observed MDP and introduced two procedures for solv ing them based on successive approximations and pol icy i t e ra t i ons . These methods in [34] were developed by Small wood and Sondik [28] and Sondik [29] for pa r t i a l l y observed problems. However, a shortcoming of these methods in [34] i s that they require more computational time and storage than the method proposed here simply because they require more evaluation and improvement steps. In Section 2 the problem and notation are defined. Algorithms based on parametric l inear programming with simplex and block pivot ing are developed using dynamic programming terminology in Section 3. The algorithm in Section 3 is motivational and the actual computational method i s d i s -cussed in Section 4 using approximations of values to resolve the d i f f i -cu l t i e s caused by roundoff er ror . In Section 5 an act ion e l iminat ion procedure for th i s problem i s developed and the deta i led algorithms are presented in Section 6 to reduce computational e f f o r t s . B i - c r i t e r i on MDP and one-constrained MDP are formulated as parametric MDP in Section 7. Numerical resu l ts appear in Section 8 where a comparison of algorithms with and without act ion e l iminat ion procedures on Howard's [14] automobile replacement problem with another c r i t e r i on are presented. 58. 2. Pre! iminar ies. In th is chapter we use the same notation as in Chapter 2 except for the addit ional quant i t ies defined below. Let a denote the sca lar parameter of var iat ion in the one-stage rewards and define constants a and a such that a < a <_ a. When the system i s in state i e I and an act ion k e K. i s se lected, r ( i , k ) and d ( i , k ) are the one-stage rewards. Let r 6 ( i ) and d 6 ( i ) be r ( i,6 ( i ) ) and d( i ,6 ( i ) ) . Define r 6 by r 6 = r 6 + ad 6 . For each S e A l e t v^ and u be the expected to ta l discounted rewards with 6 6 respect to r and d . Then a (1) v 6 = v 6 + a u 6 . a For each a, the problem we study i s that .of f ind ing a 6* e A a * and v e V such that a 6* (2) v* = v a = Max v 6 a a ~ . a 6eA For a given v, u e V and 6 e A, define the l inear operators T ,^ S g , and G g mapping V to V by (3) T$!v = r 6 + AP.v o a • 6 (4) S.u E d 6 + AP„u o 6 (5) (6) Ggu = S gu - u and define the po l icy 6 and the operator H a : V ->• V by a (7) H°v = v = Max B? v a For k = 6 ( i ) e K. we use the fol lowing notation: B?v ( i ) = B av ( i , 6 ( i ) ) = B av ( i , k ) S o r ' c r v ' a ' G 6 u( i ) = Gu( i,6( i ) ) = Gu(i,k) . The problem defined by (2) i s equivalent to f inding v and 6 a a that sa t i s f y (8) B a v* = Hav* = 0 3* a a a For c l a r i t y we w i l l use A 6 , v . u . B n , T n and H n instead <5 * * 6 n a n a n a n of r , 8 , v , u n , B n , T n and H n . a n a a 3. An Algorithm Based on Parametric Linear Programming. The l i near programming formulation of the discounted MDP was f i r s t given by D'Epenoux [7]. Mine and Osaki [17] studied the re l a t i on -ship between po l icy i t e ra t i on and l i nea r programming and showed that the po l icy i t e ra t i on i s a modif icat ion of the simplex algorithm of l i near 60. programming in which the pivot operations for many var iables are performed simultaneously. Here we develop an algorithm based on parametric l i near programming to solve the parametric MDP. In th i s algorithm pivot ing i s done on a block of var iables or states. We use dynamic programming terminology throughout. The fol lowing proposit ion i s of a technical nature and i s needed in the sequel. Proposit ion 3.1. The fol lowing equa l i t ies hold for a l l a e [a ,a] and 6 e A; (9) v 6 = v + (I - A P J " 1 B?v a o o (10) u 6 = u + (I - AP J " 1 G6u Proof. We have v + (I - A P J _ 1 B % = v + (I - A P j ' T [ r 6 + (AP. - I)v] o o 6 a o = (I - A P J " V = v 6 6 a a u + (I - A P J " 1 G^ u = u + (I - A P J _ 1 [ d 6 + {\PS - I)u] 0 0 0 0 = (I - AP J~V = u 6 . 0 61 Proposit ion 3.2, Suppose 6 n i s optimal at a n - Then O D B n v n ( i , k ) < 0, i e l and k e K., and <5 (12) v a " = v n + ( a - a n ) u n Proof. Inequal ity (11) i s an immediate consequence of (7) and the de f i n i t i on of v . From (1) and the de f i n i t i on of v n and u n , 6 n 5 n + 6 n 6 n + , 6 n + , , 6 n v = v +au = v + a u + ( a a )u v n + ( a - a n ) u n As a consequence of (12) the fol lowing lemmas hold. Lemma 3.1. Suppose 6 n i s optimal at a . Then 6 (13) B av n ( i , k ) = B nv ( i , k ) + (o>a )Gu ( i , k ) , i e I and k e K, and \ / a \ * i n n n v (1.4) B? v 6 " = 0 . 6 n a 62. 6 Proof. Equation (13) fol lows from subst i tut ing (12) into B°S/a ( i , k ) and rearranging terms. From (13) we have B " n V n V" = ° S l n C e V n = ( I ~ ^sJ 'V a n d u n = ( I " X P6 ^ n n Therefore (14) fol lows. Lemma 3.2. Suppose 6 (15) B av n ( i , k ) < 0 , i e I and k e K . . Then 6^ i s optimal at a. Proof. The resu l t fol lows immediately from (14). - B " v M , k ) Define A n = { ( i , k ) |Gu n ( i , k ) > C}, R ( i , k ) = G 9. k ) and n ' ' l e t $ denote the nul l set. Then the fol lowing theorem is the basis for the algorithm to solve the parametric MDP. Theorem 3.1 The optimal po l i cy <5n at a n i s optimal for a sa t i s fy ing a n - a - an+l w n e r e a n + ] i s defined by (16) and Vl = a n + a a Min (R ( i , k ) } i f A n f + oo i f A „ -Proof. I f A = $, i . e . , Gu ( i , k ) < 0 for a l l i e I and k e K . then for a l l 6 n a > a n , B a v a n ( i , k ) < 0, i e I and k e K i from (13) and (11). Therefore from (15) 6n i s optimal for a l l a > a n. Otherwise at a sa t i s fy ing a n < a < a n + a B a v a n ( i , k ) < 0 for ( i , k ) t A n from (13) and (11) and Thus (a - a n ) < a < R p ( i , k ) for ( i , k ) e A n . B \ n ( i , k ) = B n v n ( i , k ) + (a-a n) Gu n ( i , k ) < 0 for ( i , k ) e \ Therefore from (15) &n i s optimal for a sa t i s fy ing a n < a < a n + a. At a n + 1 < a define 6^ as fo l lows: 64. (A) where Vl*1* = 6 n ( i > for i t J 6 n+ l ( i ) =-k* f o r i £ l n I n = {i e I |R n ( i , k ) = a / ( i , k ) e A n} K- ='{k e K . |R n ( i , k ) - a / ( i , k ) e A n , i e y and k^ e K. i s determined by Gu ( i , k*) = Max G u ( i , k ) . keK? n * n * n When there ex i s t t i e s in determining k^ e K.. we choose any k^ e IC . This se lect ion of a new optimal pol icy at a is d i f fe rent from np-1 the usual se lect ion of a new optimal pol icy i f an approach s im i la r to parametric l i near programming was used. In that case only one act ion would be changed and the se lect ion rule is Vl ( i ) = 6 n ( i ) f o r 1 ? " 6 n + 1 ( i ) = k for i = i where i and k are selected by Gu ( i*,k*) = Max {Gu n ( i ,k)} . i £ l n k e K n 65. Each of these se lect ion rules w i l l give a new optimal pol icy at a n +-| and a better pol icy than the current po l i cy , 6^ , over the region of a, a > a + - j , But the se lect ion rule (A) may save computation i f there is more than one state in I . Therefore we w i l l use the se lect ion rule (A) in the sequel even though the same resul ts hold with the other se lect ion ru le . Corol lary 3.1 Suppose 6n+-| i s selected using the rule (A) and a n + 1 < a 6 +1 6 Then at a both 6 and 6 ,, are optimal and v n > v n for a l l a > a n+1 n n+1 r a a n+1 Proof. Using (1), (9), (10) and the se lect ion ru le (A), 6n+l 6n+l , , Vl v = v + (a - a ) u an+l an n+1 n - \ + < V T arA + ( I" X PVi r l [ BVl V" + (Vr an)\ + 1 un^ 6n = v an+l Because 6 n i s optimal at a n + 1 from Theorem 3.1, 6 n + 1 i s also optimal at an+l For a > a n +-| from the se lect ion rule (A) 66. Therefore from (9) and (17) v 6 n + 1 = v 6 n + (I - XP, ) - 1 B? v 6 " V l V l 6n 00 c <S„ 5 6 6 = v n + I (xp_ ) S B^ v > v n + B^ v n > v n a s=o V i V i a - a V i a a In the fol lowing coro l la ry we show that at a ± n . v and n+1' n+1 B 6 + 1 V l c a n b e c a l c u l a t e d d i r e c t l y from the values ava i lab le at a . Corol lary 3.2. At a n + - | we have (18) v , - i = v„ + (a , , - a )u , v ' n+1 n v n+1 n' n' Proof. (18) fol lows from the fact that 6 n i s s t i l l optimal at a n + 1 Equation (13) evaluated at a +-| gives (19) . From the above we develop the fol lowing algorithm to solve the problem for a l l a e [ a , a ] . STEP 1) Set a Q = a and n = 0. Find 6 , v , B 0 v Q ( i , k ) and U Q = l 6 (I - AP 6 ) " ' d °. Go to step 2. o STEP 2) Calculate Gu ( i , k ) = d( i ;k ) + A I P ( i , j , k ) u ( j ) - u ( i ) for n • ' uel n n a l l i e I and k e K., Go to step 3. STEP 3) I f Gu n ( i , k ) < 0 for a l l i e I and k e K . , stop. In th i s case 6 n remains optimal for a l l a > a . Otherwise, set a^+^ = a + where a i s defined by (16). Choose 6 +-j using the se lect ion rule (A). Go to step 4. STEP 4) I f a n + 1 > a , stop. Otherwise set 'n+1 = v n + ( a n+l " a n ) u n B n + 1 v n + ] ( i , k ) = B \ ( i , k ) + ( % + 1 - a n )Gu n ( i , k ) Jn+1 n+1 un+l Set n = n+1 and go to step 2. Note that the ca lcu la t ion of (I~AP„ )"'^ in step 4 can be im-Vl plemented by block pivoting'. One might conjecture that for some m a n = < a for a l l n > m, i . e . we do not f ind an optimal pol icy for a, a < a < a. But th is i s not n -the case because even i f a ,, = a , u > u . Hence the f in i teness of the m+l m m+l m pol icy set implies that for some k > m, a, > a , K III However d i f f i c u l t i e s might ar i se because of roundoff. In par-t i c u l a r might not include potent ia l l y optimal actions because Gu n ( i , k ) might have the wrong sign or a n +^ might be less than a n because for some ( i , k ) e A^, B n v n ( i , k ) might be pos i t ive instead of negative. In the f i r s t case, a nonoptimal pol icy might be selected while in the l a te r case the algorithm would cyc le. A method to a l l e v i a te these d i f f i c u l t i e s i s pre-sented in the next sect ion. 4. An Algorithm for Finding ^-Optimal Po l i c i e s . In th i s section we develop an algorithm based on approximations of the values of v n and u^ to resolve the problems caused by roundoff when we implement the algorithm of the previous sect ion. This algorithm is based on redef in ing R n ( i , k ) and A^ to el iminate cyc l ing and ensure se lec-t ion of optimal act ions. 6 n Let v n and u n be approximations of v^ and u where 6 n i s defined such that Hn0 = B^ v , e = e . + a e' for e > 0 and e' > 0, sp(H nv ) < n 6 -.n . n o n . -• o n n -(1-X)e n and sp(Gg u*n) < (l-X)e'. Then we have the fol lowing proposit ion. 69. Proposit ion 3.3. Suppose sp(H nv ) = sp(B§ v ) < (1-A) £ Then 6 i s e -opt imal, i.e., 6 6 (20) v n < v < v n + e . v ' a - n - a n n n K 6 * Proof. The l e f t inequal i ty of (20) is true because v < v - = v = v . an an an From (9) n n n n < ( I -AP 6 n rVv n - ( I - A P ^ " ^ where v - Max B^ v ( i ) and v n = Min B§ v ( i ) , 6 n i e l 6 n n 6 n n i e l 6 n n 6 Therefore v < v n + e and (20) holds. n - a n n 70. Define v n by a (21) v a n = v n + (a * a n) u n . We drop hats to f a c i l i t a t e typing and use the same notation as in Section 3. But a l l v n , u n and 6 n are v n , u'n and 6 . Then the fol lowing lemma i s immediate from (21). Lemma 3.3. For a > a. n Lemma 3.4. 6 6 Suppose for a > a , B?v n< B^ v n for a l l 6 e A. Then Sn v r -• n * o a 6 a n is e -optimal at a where e = e„ + (a a ) e1'. a r a n n' 6 6 Proof. Note H av n = B? v n . a o n a sp(H av n ) = sp(B? v n ) = sp(B" v + (a-a )G X u ) < sp(Bj v n ) + (a-a n) sp(G g u n) < 0-A)e n + (a-a n) (1-X)e' = (1-X)[e n + (a-an.) £'] = (1-X) e a. Therefore 6n i s E^-optimal at a from Proposit ion 3.3. Lemma 3.4 indicates that we can f ind the region where 6^ i s £ a-optimal by ident i fy ing a for which 6 6 u a „ n _ D a w n _ D n w , / X p H v n = B, v = B x v + (a-a )G~ u . a 6 a 6 n v n' 6 n Redefine A n = { ( i , k ) |Gu n ( i , k ) - G u n ( i , 6 n ( i ) ) > 0} and R n ( i , k ) by B V ( i , 6 n ( i ) ) - B% ( i , k ) K n l 1 , M Gu n(1,k) - G u n ( i , 6 n ( i ) ) Theorem 3.2. The ^-opt imal pol icy 6 n at a n i s e -optimal for a sa t i s fy ing a n < a < a n + 1 where a n + 1 i s defined by (23) a n + 1 = a n + a a -Min (R n ( i , k ) } i f A „ t $ ( i , k ) £ A„ n n + oo i f A n = Proof. Using Lemma 3.4 and (22) instead of (11) and (15), and (13) in the proof Theorem 3.1 gives the resu l t . At an+-| < a we w i l l have the same rule for se lect ing 6 n + 1 as the se lect ion rule (A) in Section 3 except using R n ( i , k ) and A p redefined here. Let us ca l l i t the se lect ion rule (B). Also we can have the same se lect ion ru le based on the change of a s ingle act ion as done in Section 3 by using R ( i , k ) and A redefined here. Then each of these se lect ion rules 6 6 , gives an e .-.-optimal po l icy at a by def in ing v n = v , = v n n+1 n+1 a n + 1 n+1 a n + ] . Corol lary 3.3. Suppose 6 n + 1 i s selected using the rule (B) and a -| < a. Then at a n + 1 both 6 n and 6 n + 1 are -opt imal. Proof. From (23) Rn+1 n _ Rn+1 _ Dn+1 un+l V l \ + l " \ + , V - l - " \ V l = V l I .e . , 6 n and 6 n + 1 are ^ - o p t i m a l at a from Lemma 3.4. At a n + 1 , v n + 1 and B " + 1 v n + 1 can be calculated d i r ec t l y from the values ava i lab le at a n in the fol lowing way: <24> vi • »„+ (vi - V u „ <25> B f V l - V n + <Vl - % ' V n The changes to be made in the algorithm of the previous sect ion are as fo l lows: (a) In step 1, f ind v Q with H°v 0 = B ° V q and sp(H 0 V ( ) ) < (1-X) e. (b) In step 3, a i s defined by (24) and choose 6 ^ using the se lec-t ion rule (B). (c) In step 4, f ind un+-j with sp(G g u n + i ) < ( l^)e'-5. Action E l iminat ion. The purpose of th i s sect ion i s to develop, act ion e l iminat ion pro-cedures for the algorithm of the previous sect ion. To el iminate a certa in act ion at step 2 and 3 of the algorithm we need upper bounds on B n v n ( i , k ) , Gu n ( i , k ) and a +-|. A special feature of th i s procedure i s e l iminat ing actions at the subsequent i t e ra t i on only by using bounds for a + - j . We use the fol lowing notat ion. (26) UGu^(i,k) = Gu ( i , k ) + I [XDun , - Dun . ( i ) ] for p > s+1 P b q=s H» q ' or (27) = Gu s ( i , k ) + XDU S j p _ s - D u S f P _ s ( i ) for p > s+1 (28) UBv^(i,k) = B\(i,k) + V ( a p + ] - a p )UGUp _ 1 ( i ,k) for n > £+1 P — $b where Du„ Du^ , , DiT ^ „ and Du „ „ are defined in Section 5 of q,T q,l s,p-s s,p-s Chapter 2. Lemma 3.5. S £ UGUp(i,k) and UBv n ( i ,k) are the upper bounds of GUp(i.k) and B n v n ( i , k ) respect ive ly. 74. Proof. GUp(i,k) < UGu p( i ,k) fol lows from the resul ts in the proof of Theorem 2.5 and Corol lary 2.5. From (25) and Gu ( i , k ) < UGu ( i , k ) B \ ( i , k ) = B n - 1 v n _ 1 ( i , k ) + ( v a n - l ) G V l ( i ' k ) B n " 2 v n _ 2 ^ k , ) ^ t ^ ^ - a ^ ^ G u ^ ^ i . k ) + •( V V l ) G V l ( i ' k ) = B \ ( 1 , k ) + Y ( a p + r a p ) G u p ( i , k ) < B £ v £ ( i , k ) + T(ap+ra p)UGu £- 1(i,k) = UBv £ ( i ,k) P & Assuming a*+-| > a n +-| i s known at i t e ra t i on n, the fol lowing lemma i s the basis for an act ion e l iminat ion procedure. Lemma 3.6. Suppose at i t e ra t i on n (29) B ^ V ? ( i , k ) < B ^ V ? ( i ) • an+l n an+l Then k e i s not -optimal in state i at a n + 1 . Proof. I f Gu n ( i , k ) - Gu n ( i,6 n ( i ) ) < 0 then ( i , k ) £ A n and so keK. i s not e n +-|-optimal at a n + 1 . Otherwise, from (29) and (25) B n v n ( i,k) + (a^ + 1-a n )Gu n ( i,k) < B \ ( i , 6 n ( i ) ) + (Vr% ) G u n ( i * ' Hence ( a * + r a n ) [ G u n ( i , k ) - Gu n ( i ,S n ( i ) ) . ] < [B n v p ( i , 6p ( i ) ) - B n v n ( i , k ) ] . Because Gu n ( i , k ) - Gun ( i , 6p ( i ) ) > 0, R n(i,k) > (a n + 1-« n) > ( a n + l " a n ^ Therefore keK^ i s not e ^-opt imal in state i at a n + - j . To use Lemma 3.6 in act ion e l iminat ion we need bounds for the terms in (26), (27) and (28). Theorem 3.3. Suppose at i t e ra t i on n that for some I < n-l (30) UBv*(i,k) + i^^^XUk) % .vn.(i.) + / ( a ^ r a n ) G ^ u n ( i ) . Then keK^ i s not e n +-j-optimal in state i at a + - j . Proof. From (22), Lemma 3.5 and (30) a' 6 (31) B n + 1 V a | M i , k ) = B n v n ( i,k) + (a^ + 1-a n )Gu n ( i,k) < UBv*(i,k) + ( a ^ - a ^ U G u ^ i . k ) <Byn( i) + K + l - n ) V " ( i ) = B f l v - V l ( i ) The resu l t fol lows from Lemma 3.6. 76. For £ = n we obtain the fol lowing co ro l l a ry . Corol lary 3.4. Suppose at i te ra t ion n B \ ( i , k ) + ( a ; + 1 - a n ) U G u ^ ( i , k ) < 8 ^ ( 1 ) + ( * > + v ^ i ) . Then k e K. i s not -optimal in state i at a +-|. Proof. Using B n v n ( i , k ) instead of UBvjj(i,k) in (31) gives the resu l t . Finding appropriate a^+-j and using UGUp~^(i,k) in (26) or (27) we can develop several act ion e l iminat ion algorithms. 6. Action El imination Algorithms. In th is section we develop spec i f i c computational procedure based 1 £-1/ \ on choosing appropriate values for a n +-| and UGun ( i , k ) in Theorem 3.3 and Corol lary 3.4. These action e l iminat ion procedures w i l l be included at the second-step-of-the-algorithm presented in Section 4. This i s a l i t t l e d i f fe rent from the action el iminat ion procedures in Chapter 2. The reason is that we have to ca lcu late G^ u^ at step 3 even for the algorithm without act ion e l iminat ion. We use E n ( i ) defined in Section 6 of Chapter 2 but redefine F ( i ) to be the set of actions that have been eliminated for i te ra t ions £, £ + l , . . . , n , i . e . , 77. P 36 i _ Act ion E l i m i n a t i o n Procedure ( I ) ; a +-| = a. Suppose a t step 2 o f i t e r a t i o n n in the a l g o r i t h m o f Sect ion 4 and f o r k E K. - ^ ( i ) (32) B n v n ( i , k ) + ( a - y U G u ^ t i . k ) < B n v n ( i , 6 n ( i ) ) + (a-% )Gu n ( i , & n ( i ) ) . Then k e E n ( i ) : Furthermore, i f f o r I = n - l , n - 2 , • • • , 1 and k e F , ( i ) the . f o l l o w i n g ho lds : (33) " UBvjJ(i ,k) + ( a - a ^ U G u ^ d - . k ) < B n v p ( i , 6 n ( i ) ) + (a-a n ) G u n ( i , < $ n ( i ) ) . Then k e En(i)-Let procedure ( I - l ) and (1 -2 ) denote the a c t i o n e l i m i n a t i o n p ro -cedure ( I ) w i t h UGUp(i,k) o f (26) and (27) r e s p e c t i v e l y . Then UBvj j ( i , k ) and U G u ^ ( i , k ) i n (32) and (33) can be c a l c u l a t e d as f o l l o w s : (34) U B v ^ ( i , k ) = U B y ^ d . k ) + ( ^ - a ^ ) U G i £ ] ( i , k ) f o r I < n -2 . o r (35) = B n _ : ! v n _ 1 ( i , k ) + ( v a n - l ) U G u n - l ( i , k ) f o r 1 = n _ 1 ' (36) U G u ^ O . k ) = G u ^ d . k ) + D i T n _ u l - D u n _ 1 } 1 ( i ) . 78. In procedure ( I - l ) ; (37) UGu £ _ 1 ( i , k ) = UGu £~](i,k) + ADU^ 1 - Du"n_1 ^ i ) for £ < n-1. In procedure (1-2); (38) U G u J - ' d.k) - U S u J l j d . k ) + A ( % . 1 > n + 1 . t - B u w > n . t , for £ < n-1. D u n - l . l < 1 > ' F l ag n ( i , k ) indicates whether an act ion k e K.^ i s el iminated or not. In procedure (1-2) Flag ( i , k ) = £ i f k e F n ( i ) . But in procedure ( I - l ) we do not have to store £ for k z F n ( i ) because the term added to £, n UGu £ _ 1 ( i , k ) for UGu £~](i,k) in (37) i s ADu^ ] - D u n l ^ i ) . Redefine A N by A N E { ( i , k ) | Gu n ( i , k ) - G u n ( i , 6 n ( i ) ) > 0 and F l a g n ( i , k ) < 0}. When there is a difference between procedure ( I - l ) and (1-2), procedure (1-2) is shown in parenthesis in the fol lowing algorithm. STEP 1) Set a = a and n •= 0. Find S . v , B°v and u„ with H°v = B° v , ' o — o o 6 o o O O Q O sp(H°v0 < e Q and sp(G^ U q ) < e 1 . I n i t i a l i z e F l ag Q ( i , k ) = -1 for a l l i e I o and k e and ca lcu late Gu Q ( i ,k ) for a l l i e I and k e and go to step 3. STEP 2) Calculate G„ u . 6 n n I f Flag . j( i ,k) < 0 and (32) i s sa t i s f i ed then F l ag n ( i , k ) = 1 [ F l ag n ( i , k ) = n] and do not ca lcu late Gu n ( i , k ) . I f F l ag n -|(i,k) < 0 and (32) i s not s a t i s f i e d then F l a g n ( i , k ) = F l ag n -|(i,k) and ca lcu late Gu n ( i , k ) . I f F l ag n -|(i,k) > 0 and (33) i s sa t i s f i ed then F l ag n ( i , k ) = F lag n -|(i,k) and do not ca lcu late Gu ( i , k ) . I f F l ag n -|(i,k) > 0 and (33) i s not sa t i s f i ed then F l ag n ( i , k ) = -1 and ca lcu late B n v n ( i , k ) and Gu n ( i , k ) . Go to step 3. STEP 3) I f Gu ( i , k ) - Gu ( i , ( i ) ) < 0 for a l l i e l and k e K. where F l a g n ( i , k ) < 0, then 6 n remains a-optimal fo r a l l a sa t i s fy ing a > a n and stop. Otherwise, set as in (23) and choose by the se lect ion rule (B). STEP 4) I f a > a stop. Otherwise set v = v + (a ,,-a )u and ' n+1 - ^ n+1 n v n+1 n' n ca lcu late u n + ] with sp(G g u ^ ) < e ' . Set B n + 1 v n + 1 ( i ,k) = B n v n ( i , k ) + ( an+r an )Gu n ( i ,k ) for k e K.. where F l ag n ( i , k ) < 0. For k e K^ . where F l ag n ( i , k ) > 0, ca lcual te U B v £ + ] ( i , k ) and UGu^ j ( i , k ) using (34) or (35) and (36) or (37) [using (34) or (35) and (36) or (38) depending on I = F l a g n ( i , k ) ] . Set n = n+1 and return to step 2. 80. Some comments about th i s algorithm are in order. UBv n ( i ,k) and UGu^ ( i , k ) can be stored in the same arrays as B v n and Gu n ( i , k ) respect ive ly . UGu ( i , k ) of (27) i s a t ighter upper bound for Gu ( i , k ) r r than UGu ( i , k ) of (26). Hence procedure (1-2) w i l l el iminate more actions P than procedure ( I - l ) , but requires storage of the vectors u 0,u-j,»«»,u , whereas procedure ( I - l ) only requires storage of the vector u . •. Therefore the algorithm above with act ion e l iminat ion procedure ( I - l ) or (1-2) requires at leas t an N x M array, Flag ( i , k ) , i n addit ion to the storage space . required for the algorithm without the act ion e l iminat ion procedure. Here N is the number of states and M i s the number of actions in each state assuming each state has an equal number of act ions. Using P ( i , l , k ) for Flag (i.,k) as in Section 6 of Chapter 2 we can reduce the addit ional storage requirements. Furthermore, for a problem with a large state and action space, the t rans i t i on probab i l i t i es may be stored in a f i l e or on a tape and read whenever needed. Denote a n ( i ) by a n ( i ) = B n v n ( i , n ( i ) ) + (a-a )Gu ( i , 6 ( i ) ) . In th is case a method for stor ing only F l a g n ( i , k ) and a n ( i ) instead of B n v n ( i , k ) and F l a g n ( i , k ) i s based on rearranging terms in (33) to obta in; (39) U B v ^ i . k ) + (S - a^JUGu^jO . k ) + (a-%)UDu~n_i j " DVl,l ( i ) ] < a n ( i ) where UBv^^i . k ) = B n _ 1 v -, ( i , k ) for I = n-1 . The quantity F l ag n -|(i,k) i s redefined as e i ther B n v n ( i , 5 n ( i ) ) -B n v n ( i , k ) or UBv £_-,(i,k) + ( a - a ^ )UGuJ'] ( i ,k) - a n - 1 ( i ) with UBvjJ_,(i,k) = 81. Bn"^v -,(i,k) for £ = n- l . I f an act ionk e K. is not eliminated at i terat ion n-l i ( n - l ) , Flag n -|( i,k) takes on the former value which is nonnegative while i f the action is eliminated i t takes on the la t ter value which is negative. Then (32) becomes for k e K. - E n ( i ) V I n-1 (40) b n ( i , k ) =- Flag n _ 1 ( i ,k) + (S-an) [UGuJj-1 ( i ,k) - Gu p ( i,6 n ( i ) ) ] < 0 and (33) or (39) becomes for k e F£ n _-|( i) (41) c n ( i , k ) = Flag n_ 1 ( i , k ) + a ^ ^ i ) + ( a - a ^ U D ^ ^ - D i T ^ ^ i ) ] - a n ( i ) < 0. In the above procedure we require an array of a n ( i ) instead of B n v n ( i , k ) . Therefore there wi l l be signif icant storage saving i f there is large action space in each state. Let ( I - l 1 ) denote this storage saving procedure. Then the changes to be made in the algorithm with procedure ( I - l ) to incorporate procedure ( I - l 1 ) are as follows: In s tep! ; In i t ia l i ze Flag ( i , k ) = B°v Q ( i , 6 ( i ) ) - B 0 v Q ( i , k ) > 0 for al1 i e l and k e K.. In step 2; I f Flagn -|(i,k) > 0 and (40) is satisfied then 1 Flag n ( i ,k) = b n ( i , k ) < 0 and do not calculate Gu n ( i ,k) . I f Flagn_-| ( i , k ) > 0 and (40) is not satisfied then Flag n ( i ,k) = Flagn -|( i,k) and calculate Gu n ( i ,k) . I f Flagn -|( i,k) > 0 and (41) is satisfied then Flag n ( i ,k) = 82. c ( i , k ) and do not ca lcu late Gu n ( i , k ) . I f Flag n_- ] ( i , k ) > 0 and (41) i s not s a t i s f i ed then ca lcu late B n v n ( i , k ) and Gu n ( i , k ) and set F l ag n ( i , k ) = a n ( i ) - (a-a n )Gu n ( i,S n ( i ) ) - B% n ( i , k ) = B \ ( i , 6 n ( i ) ) - B n v n ( i , k ) > 0. In step 3; A n i s defined by A n ='{(i,k)|Gu ( i , k ) -G u n ( i , 6 n ( i ) ) > 0 and F l a g y l , k ) > 0}. I f Gu n ( i , k ) - Gu n(i ,c^(i)) < 0 for a l l i e I and k e K. where Flag ( i , k ) > 0, then 6 n remains a-optimal for a l l a, a > or and stop. In step 4; (Add the fo l lowing ca lcu la t ions . ) I f Flag ( i , k ) > 0, reset F l ag n ( i , k ) = F l a g n ( i , k ) + ( a p + 1 - a n ) [ G u n ( i , 6 n ( i ) ) - Gu n ( i , k ) ] -B n + 1 v n + 1 ( i , 6 n + 1 ( i ) ) - B n + 1 v n + 1 ( i , k ) >0. This kind of storage saving procedure may not be used for proce-dure (1-2) since there, we have to store £, i . e . , the l a s t evaluated i t e ra t i on for B i i v £ ( i , k ) for each k e F £ n _ - | ( i ) . The fol lowing action e l iminat ion procedures are based on a t ighter upper bound for a +-|, than a. Action El iminat ion Procedure ( I I ) ; = a n + - j . Here a l l the resul ts are exactly the same as in procedure (I) except we use a +-j instead of a. a n +-j sa t i s fy ing ct^-j < an+-| < a can be calculated at the beginning of step 2 of i t e ra t i on n using the fol lowing method. where 83. F i r s t f ind ( i ' . k 1 ) at the end of step 3 of i t e ra t i on n-l so that G V l ( i ' ' k , ) ~ G u n - l ( i ' ' 6 n r l ( i , ) ) Max {Gu n _ 1 ( i , k ) -Gu n _ 1 ( i,6 n_ 1 ( i ) ) } i e T n - l l An _ i " {( i ,k) | G u n l ( i ,k) > Gu n _ 1 ( i,6 n ( i ) ) and F l a g ^ i . k ) < 0}, a 1 = Min ( i ,k)eA n_ 1 {R n_-,( i ,k)} 5 Jn-1 = { i I R n - 1 ( i ' k ) = a ' 1 ( i ' k ) e A n - 1 } and K1 T?n-1 i = { k e K i I R n - l ( i ' k ) = o t ' 1 ( i ' k ) e A n -1 } f o r 1 e J n -1 Note R -j ( i , k ) is defined in Section 4. Then at the beginning of step 2 ca lculate a n + 1 as fo l lows; define a h+1 a n + R n ( i ' , k ' ) i f G u n ( i ' , k ' ) > Gu n ( i,6 n ( i ) ) a otherwise and set a p + 1 = min(a, « n + 1 ) . Then a , > a , because a , = a + Min {R n ( i ,k)> and i f Gu ( i ' . k ' ) > ( i ,k)eA n n Gu n ( i',6 n ( i ' ) ) then ( i ' . k 1 ) e A n 84. Denote by ( I I - l ) and (I1-2) the above procedure with UGu ( i , k ) of (26) and P (27) respect ive ly . Then a l l the algorithms with procedure ( I I - l ) and (11-2) are exact ly the same as the algorithms with procedure ( I - l ) and (1-2) except we replace d by a n + - j . A l l of the above resu l ts concerning storage saving also apply. 7. Appl icat ion of Parametric Markov Decision Problems. The most simple appl icat ion of a MDP with one parameter i s sens i -t i v i t y analysis of the one-stage reward when i t i s in the form, r^ = r + ad . Another appl icat ion i s MDP with two c r i t e r i a . The object ive of the discounted MDP with vector c r i t e r i on i s to determine strategies with non-dominated expected tota l discounted rewards. In th is set t ing v ( i ) is an S-vector of {v-j ( i ) ,v^(i) • • » V g ( i ) } where S i s the number of c r i t e r i a , i . e . , v i s a N x S matrix where N i s the number of s tates. Then the problem is to f ind the set of TT e IT, the nondominated set, such that there does not ex i s t a p e n with the propert ies: v p > v77 and for some i £ I, v p ( i ) j v 7 T ( i ) . Henig [13] has shown that i t i s su f f i c i en t to r e s t r i c t our attent ion to stat ionary strategies in determining the nondominated v77 since any nondominated v77 generated by a nonstationary strategy w i l l be contained in the convex hul l of the set of a l l nondominated v . I t has been shown in Henig [13] and Viswanathan, Aggarwal and Nair [33] that the set of a l l 6 e A which generate nondominated v can be determined by considering the fol lowing sca lar c r i t e r i on problem. 85. <5* (42) v* = v a = Max {v 6 -a} a a 6eA ' Where v «a i s an inner product of v and a, a i s a column vector of S (a-,,a 9 •••£«-), a > 0 for s = 1,2,«««,S and I a = 1. ^ ~ s=l For the MDP with two c r i t e r i a the problem can be writ ten as fo l lows: 6* (43) v* = v a = Max {a n v 6 + a 9 v 6 } a a 6eA 1 1 d 1 where a-j + a^ = 1, a-j > 0, a^ > 0 and a = (a-j^)*". Sett ing a-| = l ^ , (43) becomes * 6 a (44) v* = v 2 = Max {v 6 + a9(vf> - v 6 )} 2 2 6eA where 0 < a^ < 1. r r r A rR Then set t ing a = a^, u = v^ - v^ and v = v-j the problem (44) i s exactly the same as MDP with one parameter. Another interest ing app l i cat ion of an MDP with one parameter i s a problem with one constra int . From now on we extend the range of decisions to include randomized (or mixed) s t ra teg ies . Let II denote the set of a l l randomized s t ra teg ies . Nonrandomized stat ionary strategies w i l l be ca l led pure s t rateg ies . The problem to be considered here is to f ind a n e n such that 86. (45) av" = Max av" TrelT. s . t . au 1 1 > b where a ( i ) i s the probab i l i ty that the system is in state i e l at stage 0 with I a ( i ) - = 1 and a ( i ) >0 for each i e l , b is a sca lar and v77 and u77 Iel " " are the expected to ta l discounted reward vectors corresponding to one stage reward vectors r and d respect ive ly for strategy Trell. Note the con-s t r a i n t au 7 1 < b can be transformed to au 7 7 > b by set t ing u17 = -u* and b = -b. To solve the above problem we introduce the fol lowing problem: * TT (46) av a = Max av77 ' a a irell where v7^ = v77 + au 7 7 and a > 0. Then we have the fol lowing lemma describing the re lat ionsh ip of these two problems above. Lemma 3.7. * * TT Suppose TT^ i s optimal in the problem (46) for some a and au a = b. Then TT^ i s optimal in the problem (45). Proof. Suppose TT were not optimal i n the problem (45), then there would a * * TT ,. IT ex is t a strategy TT' such that av > av a and au77 > b = au a . Then av77 + aau77 > av 0 1 + aau a . This contradicts the fact that TT* i s optimal in the problem (46). Hence TT i s optimal in the problem (45), 87. The optimal strategy TT^ of (45) given a w i l l have an optimal pure strategy independent of a since the problem (46) given a i s an ordinary discounted MDP. Therefore the problem reduces to that of f ind ing 6* such a that * (47) v a = Max v° a 6eA a 6 6 6 where v = v + au and a > 0. Recall A i s the set of nonrandomized a po l i c i e s . The problem (47) can be solved using the algorithm developed in Section 3 with a = 0 over the region of a. But an upper bound for a, i . e . a, i s not given and so the region of a may be unbounded. We w i l l show the boundedness of a in Proposit ion 3.5 and the stopping rules of the algorithm that resu l t in an optimal strategy to the problem (45). We 6 w i l l use the notation of Section 3 for a ,6 , v and u where v = v n + ^ ^ n n n n n a n u n and u n = u n . The fol lowing proposit ion motivates the stopping rules of the algorithm for so lv ing the problem (45). Proposit ion 3.4. 6 n Vl v > v Vi 6 At i t e ra t i on n+1, u = u n + ] > u n = u n and i f a n + 1 > 0, Proof. From Proposit ion 3.1 at a n + 1 and the se lect ion rule (A) u " + ' " V l = " n * » - \ + 1 u n > u „ From Corol lary 3.1 88. V l + V l V l 6 n 6 n - 6 n n+1 a n + 1 % + 1 n+1 • Therefore v n < v n since u > u n and a n + ] > 0. The fo l lowing proposit ion implies that the region of a to be considered i s ac tua l ly bounded. Proposit ion 3.5. There.exists such that for a l l a > a^, 6^ i s optimal in the problem (47) and 6 N i s also optimal for the problem of Max u . 6eA Proof. The fact that un+-j > u n in Proposit ion 3.4 and the f in i teness of pol icy space implies that there ex is ts a^ such that for a l l a > a^, 6 N i s optimal in the problem (47). Suppose 6^ i s not optimal for the problem of maximizing u . Then choose 6' such that u = Max u . There ex is ts i 1 e l such that u ( i 1 ) > 6eA u ( i 1 ) since u >u ' and u - f u . Then there ex is ts a' such that for *~ cS (5 a l l a > a' > a N , a ( u 6 ' ( i ' ) - u N ( i ' ) ) > v N ( i ' ) - v 6 ' ^ ' ) , i . e . , v ^ i ' ) + 6' 6N 6N au ( i 1 ) > v ( i 1 ) + au ( i 1 ) . This contradicts the fact that for a l l a > a^, 6^ i s optimal in the problem (47). Hence 6^ i s optimal for the problem of Max u 6 . SeA 89. Therefore using the algorithm in Section 3 6 can be found for a each a, 0 < a < where w i l l -be determined in the algorithm. The fol lowing theorems indicate how to modify the algorithm of Section 3 to f ind the * optimal strategy, TT , of the problem (45). Theorem 3.4. * Suppose auQ> b. Then TT = (6 o,6 o,«««) i s an optimal pure strategy to the problem (45). Proof. Since a - ot = 0, 6 is optimal in the problem of Max av77 without g TreH the constraint and <5 sa t i s f i e s the constra int , i . e . , au 0 = au > b. o ' ' o -Therefore 6 n i s optimal in the problem (45). Theorem 3.5. i ) Suppose auQ < b and au n = b for n > 1. Then IT = (6 ,6 ,«<•••) i s an optimal pure strategy to the problem (45). i i ) Suppose au Q < b and au n < b < a u n + 1 for n > 0. Then the strategy, TT , of using• (6n,6n,•«•••) with probab i l i ty p and (6 n + 1,6 + 1,• • •) k k with probab i l i t y (1-p ) at stage 0 where p = ( a u n + l " b ^ a u n + V a u n ^ i s optimal in the problem (45). 90. Proof. F i r s t observe in ( i i ) au" = P* a u n + 0 " P* ) a u n + - | = [ (au n + 1 ~b)/ ( a u n + ] - a u n ) ] a u n + [(b - a u n ) / ( a u n + 1 - a u n ) ] a u n + ] .= b. And in ( i i ) TT* i s also optimal in the problem (46) for a = a n +-j since 6 n and are both optimal in the problem (46) at « n + -| . Therefore for ( i ) and ( i i ) au = b and TT i s optimal in the problem (46). Hence by Lemma 3.7 TT i s optimal in the ; problem (45). Theorem 3.6. Suppose auj^ i < b. Then there i s no feas ib le strategy to the problem (45). 6N Proof. Note au^ = au and 6^ i s optimal at a^. Then by Proposit ion 3.5 6 N i s optimal for the problem of Max u , i . e . , ( ' S ^ , S ^ , » » - ) i s an optimal SeA ~ TT 6N strategy for the problem of Max u . Because au < b there is no feas ib le TTEII strategy. Note ( i i ) of Theorem 3.5 leads to choosing an in feas ib le strategy with pro-bab i l i t y p*. To resolve th i s problem we can consider two a l ternat ives — one of using a stat ionary randomized strategy and the other of f ind ing the best pure strategy. Proposit ion 3.6. Suppose au Q < b and au n < b < au n +-| for n > 0. Then there ex ists an optimal stat ionary randomized strategy. 91 Proof. Assuming a n +-j > 0 [otherwise Theorem 3.4 holds at a +-j = 0 and we have an optimal pure strategy] v > v n by Proposit ion 3.4. Let TT* be the optimal strategy defined in ( i i ) of Theorem 3.5. Then there ex is ts p such that v = (I - AP) r where P = pP f i + ( U p ) P 6 and n n+1 ~ n ^ n+1 A I / V ^ ^ 1 r= pr + ( l - p ) r because (I-AP) r i s continuous in p with (NAP) r = (5 6 (5 (5 v n at p = 1 and (I-AP)-" 1? = v n + 1 at p = 0 and v n > v77* > v n + 1 . I . e . , there ex is ts an optimal stat ionary randomized strategy. Theorem 3.7. Suppose au Q < b and au n < b < au n +^ for n > 0. Then (<$n+-] ,6n+-|, • • •) i s an optimal pure strategy among a l l pure strategies feas ib le to the problem (45). Proof. Suppose there were a better pure feas ib le strategy, say (61,6',•«>«), than ( 6 n + 1 , 6 n + 1 , • • • ) , i . e . , ay 6 > av n + 1 and au 5 > b. Sett ing p' = (au -b)/(au -au n ) we have P 'au n + ( l - p ' ) au = b. I f p' > p* then p'av n + ( l - p ' ) a v 6 ' > p'av n + (l-p'")av n + 1 > 6 6 ~" p*av n + (l-p*)av n + \ The l a s t inequal i ty follows from Proposit ion 3.4. Therefore by denoting TT' as a strategy using (6 ,6 •••••) with probab i l i ty n n , 6 p' and (6',6 ' , " 0 with probab i l i ty (1-p 1) at stage 0, av77 = p'av n + ( l - p ' ) a v 6 ' > p*av n + (l-p*)av n + 1 = av77* and au 7 7 ' = p'au n.+ ( l - p ' ) a u 6 ' = b. Thus TT' i s a better strategy than TT* contradict ing Theorem 3.4. 92. g 6 5 I f p' < p* then p'av n + ( l -p ' )av 6 ' > p'av n + (1-p')av n + 1 and p'au n + ( l -p ' )au 6 ' = b = p*aun + • (.l-.p*)au n + 1 > p*au + ( l - p ' ) a u n + 1 6 because u n + ^ > u n from Proposition 3.4.. and p' < p*. Therefore p'av n + ( l -p ' )av 6 ' + V l [ p ' a u n + (1-P')au 6 t 3*> P'av n + (l-p')av n + 1 + 6 - n + 1 [ p ' a u n + ( l - p ' ) a u n + 1 j . I .e . , p'[av + a n + 1 a u n ] + ( l -p ' ) [av 6 + < V l a u ] > P'^ a v n + V l a u n ] + n - P l ) . [ a v n + 1 + a n + 1 a u n + 1 ] . Hence av 5 + a n +- )au < S > av n + 1 + a n + 1 a u n + 1 since p' < 1. by the assumption of au n < b. This contradicts the fact that <5n+1 i s optimal in the problem (47). Therefore ( 5 n + l , 6 n + r * ^ i s a n 0 P t i m a l P u r e strategy among a l l pure strategies feas ib le to the problem (45). The stopping rules to be changed or added in the algorithm of Section 3 are as fo l lows: (a) At the end of step 1; I f au Q > b, stop. Then 6 q i s optimal in the problem (45). (b) At the beginning of step 3; I f Bu ( i , k ) < 0 for a l l i e l and keK.j and au^ < b, stop. There i s no feas ib le strategy to the problem (45). I f Bu n ( i , k ) < 0 for a l l i e l and keK;. and au n > b, stop. Then TT* defined in Theorem 3.5 is optimal. (c) At the beginning of step 4; I f au n > 0 [instead of " i f a n + 1 > a"], stop. Then TT* defined in Theorem 3.5 i s optimal. 93. Furthermore, a in Theorem 3.4, a in ( i ) of Theorem 3.5 and 0 n a n + i in ( i i ) of Theorem 3.5 give the shadow price of the constraint in the problem (45). Another appl icat ion of MDP with one parameter i s the MDP with one ra t io constra int . Solution procedures for MDP with a ra t io c r i t e r i on were studied by Derman [8] for the undiscounted case and by Aggarwal, Chandrasekaran and Nair [1] and Brosh, Sch l i f e r and Schweitzer [4] fo r the discounted case.: Here we consider the fol lowing ra t io constrained MDP; (48) Max av77 - au 1 7 Trell s . t . ^ > R aD77 " where v ( i ) , - 0 0 < v ( i ) < + °° and u ( i ) , 0 < u ( i ) < + <» are the expected tota l discounted returns and costs respect ively in state i i f we take a strategy TT. This problem can be rewrit ten as (49) Max av77 - au 7 7 Trell s . t . av77 - R au7 7 > 0. Then set t ing v77 = v77 - u 7 7, u77 = v77 - Ru77 and b = 0, the problem (49) i s exact ly the same as the problem (45). A l l the resul ts for the problem (45) hold for th is problem. 94. 8. Computational Results. In th is section we discuss the resul ts of applying the algorithms developed in Section 4 to a two-cr i ter ion version of Howard's [14] automobile replacement problem. Included are a comparison of the algorithms under the various se lect ion rules and act ion e l iminat ion procedures. (5 (5 Let r-j and r 2 be the expected one-stage rewards corresponding to v-| and v 2 in (43) and defined as fo l lows: r-j( i ,k) = r ( i , k ) defined for the automobile replacement problem in Section 8 of Chapter 2 and r 2 ( i , k ) = -i f k f 1 400 i f k = 1 where r 2 ( i , k ) i s an addit ive expected one-stage u t i l i t y function for time preference i f we take action keK i in state i e l . Therefore i f we keep the present car of age i (k=l) then the time preference u t i l i t y for one period w i l l be 400//i+1• I f w e trade i t in for a car of age k-2 the one-period time preference u t i l i t y w i l l be 4 0 0 / A - l . Resetting r ( i , k ) = r-j ( i , k) and d( i , k ) = r 2 ( i , k ) - r-j(i,k) the nondominated strategies of the automobile replacement problem with two c r i t e r i a can be generated by solv ing the fol lowing problem; 95. Max (v^ + au 6} 6eA 0 < a < 1, 6 6 where v and u are the expected tota l discounted reward corresponding 6 6 to r and d respect ive ly . In the notation of th i s chapter th i s means we take a = 0 and a = 1. We solved th is problem with X = .96 and X = .97, ( l-X)e = 0,001 and ( l-A)e' = 0.001 using the algorithm with and without act ion e l imina-t ion procedures. A l l ca lcu lat ions were performed on the Univers i ty of B r i t i s h Columbia AMDAHL 470 computer using the codes in the Appendix. For the algorithms without act ion e l iminat ion we used the se lect ion ru le (B) and the other se lect ion rule (denoted from now on as (C)) of changing only a s ingle action as described in Section 4. The se lect ion rule (B) leads to block pivot ing while the other se lect ion rule leads to simplex p ivot ing. Block pivot ing always gives a value of u +-|- greater than or equal to that of simplex p ivot ing, which implies that block pivot ing w i l l generate optimal po l i c ies over the region o f a in fewer i te ra t ions than simplex p ivot ing. The i n i t i a l optimal act ions, i . e . at a = 0, with X = 0.96 are k = 18 (trade in for the car of age 16, i . e . , the 4 year-old car) for i from 1 to 7 and from 28 to 40 and k = 1 (keep the current car) for i from 8 (2 year-old car) to 27 (6 3/4 year-old car ) . The action k = 2 (trade in for a new car) is optimal in a l l states for a > 0.782133444 and X = 0.97 are k = 14 (trade in for thecar of age 12, i . e . , the 3 year-old car) for i from 1 to 3 and from 27 to 40 and k = 1 for i from 4 (1 year-old car) to 26 (6 1/2 year-old car ) . A l l optimal actions for a > 0.781641042 are k = 2 in a l l s tates. In between optimal actions for A = 0.96 and A = 0.97 are presented in the Appendix. The CPU times required a f te r the i n i t i a l i z a t i o n and the number of i te ra t ions (r in the algorithm of Section 4) required are shown in Table 3.1 for the algorithm with the se lect ion rule (B) and (C) and without action e l iminat ion. Clear ly the algorithm using the rule (B) i s much faster than the rule (C). This is because for A = .97, 17 actions changed together at the one par t i cu la r value of a and for A = .96 there were values of a at which 17 actions and 16 actions changed together. Thus for the algorithms with action el iminat ion procedures we used only the se lect ion rule (B). The CPU times required af ter the i n i t i a l i z a t i o n and the average f ract ion of actions el iminated for the algorithm with each act ion e l imina-t ion procedure are shown in Table 3.2. Reduction of CPU times using act ion el iminat ion procedure (11-1) and (11-2) were s i gn i f i c an t . For instance, with A = .96 the algorithm with the act ion el iminat ion procedure (II-2) took only 47.81 per cent of the CPU time taken by the algorithm with the ru le (B) and without act ion el iminat ion and only 28.78 per cent of the CPU time taken by the algorithm only with the rule (C). 9. Conclusions and Extensions. The algorithm developed in Section 4 can be implemented to solve a problem with a large state and action space. As shown by the resul ts in Table 3.1 use of an algorithm with the se lect ion rule (B) i s recommended. Even though we used codes based on pol icy i t e r a t i on , s im i l a r codes based 97. Table 3.1 Comparison of Select ion Rules [CPU Times (Sees.)/Number of Iterat ions Required] Algorithm with the Algorithm with the Select ion Rule (B) Select ion Rule (C) Discount R a t e ^ ^ 0.96 8.623/37 14.326/68 0.97 7.121/31 9.875/47 Table 3.2 Comparison of Action El iminat ion Procedures [CPU Times (Sees.)/Average Fraction of-Actions El iminated] (AN. Discount Rate^. Procedure ( I - D Procedure (1-2) Procedure (U -1 ) Procedure (H-2 ) 0.96 7.158/.2939 6.104/.4553 4.792/.6681 4.123/.7778 0.97 5.829/.3193 4.645/.5065 4.530/.5925 3.880/7127 98. on modified policy i terat ion can be developed and wi l l be more ef f ic ient for problems with very large state and action spaces. Results based on using the action elimination procedure ( I I - l ) and (11-2) are also very encouraging and we recommend incorporating them in an algorithm. An interesting extension would be to develop an algorithm for the problem with a vector of parameters. This would have application to more than two cr i ter ion MDP and MDP with several constraints. The methods developed here would be applicable but modifications to e f f i c ie in t l y search the parameter set would be required. 99. REFERENCES 1. Aggarwal, V., Chandrasekaran, R. and Nair, K.P.K., "Markov Ratio Decision Processes", JOTA 21, 1977, 27-37. 2. B lackwel l , D., "Discrete Dynamic Programming", Ann. Math. S t a t i s t . 33, 1962, 719-726. 3. B lackwel l , D., "Discounted Dynamic Programming1', Ann. Math. S t a t i s t . 36, 1965, 226-235. 4. Brosh, I . , S ch l i f e r , E., Schweitzer, P . J . , "Generalized Markovian Decision Processes", Ze i t s ch r i f t fur Operations Research 21, 1977, 173-186. 5. Dantzig, G.B., Linear Programming and Extensions, Princeton Univ. Press, Princeton, New Jersey, 1963. 6. Denardo, E., "Contraction Mappings in the Theory Underlying Dynamic Programming", SIAM Rev. 9, 1967, 165-177. 7. D'Epenoux, F., "Sur un Problemsde Production et de Stockage dans 1'Aetoire", Revue Frang. Rech. OpeVationelle 14, 1960, 3-16; (translat ion~Th~Man. S c i . 10, 1963, 98-108). 8. Derman, C , "On Sequential Decision and Markov Chains", Man. S c i . 9, 1962, 16-24. 9. Gr ino ld, R., "El iminat ion of Suboptimal Actions in Markov Decision Problems", Oper. Res. 21, 1973, 848-85. 10. Hastings, N.A.J, and Mello, J.M.C., "Tests for Suboptimal Actions in Discounted Markov Programming", Man. S c i . 19, 1973, 1019-1022. 11. Hastings, N.A.J . , "A Test for Nonoptimal Actions in Undiscounted F in i te Markov Decision Chains", Man. S c i . 23, 1976, 87-92. 100. 12. Hastings, N.A.J . , and van Nunen, J .A .E .E . , "The Action El iminat ion Algorithm for Markov Decision Processes", in Markov Decision Theory, Eds. H.C. Tijms and J . Wessels, Mathematical Centre Tract 93, Amsterdam, 1977. 13. Henig, M.I., "Mu l t i c r i t e r i a Dynamic Programming", Ph.D. Thesis, Yale Univ. , New Haven, Conn., 1978. 14. Howard, R.A., Dynamic Programming and Markov Processes, MIT Press, Cambridge, 1960. 15. Hubner, G., "Improved Procedures for El iminat ing Suboptimal Actions in Markov Programming by the Use of Contraction Propert ies", Transactions of the Seventh Prague Conference, Volume A, 1977, 257-263. 16. MacQueen, J . , "A Test for Suboptimal Actions in Markov Decision Problems", Qper, Res. 15, 1967, 559-561. 17. Mine, H. and Osaki, S., Markovian Decision Process, Amer. E lsev ier Pub. Co., Inc., New York, 1970. 18. Morton, T. , "Undiscounted Markov Renewal Programming v ia Modified Successive Approximations", Qper. Res. 19, 1971, 1081-1089. 19. Porteus, E., "Some Bounds for Discounted Sequential Decision Processes", Man. S c i . 18, 1971, 7-11 20. Porteus, E., "Bounds and Transformations for F in i te Markov Decision Chains", Qper. Res. 23, 1975, 761-784. 21. Porteus, E., "Improved I tera t ive Computation of the Expected Discounted Return in a Semi-Markov Chain", Working Paper, Stanford Univers i ty , 1978. 22. Porteus, E., "Overview of I terat ive Methods for Discounted F in i te Markov and Semi-Markov Decision Chains", to appear i n : D. White Ed.: Pro, of I n t l . Conf. on Markov Decision Processes, Manchester, 1979. 23. Porteus, E. and Toten, J . , "Accelerated Computation of the Expected Discounted Return in a Markov Chain", Qper. Res. 26, 1978, 350-358. 101. 24. Puterman, M.L. and Brumelle, S.L. , "The Analyt ic Theory of Pol icy I te ra t ion" , in Dynamic- Programming and Its Appl icat ions, Pro. of I n t l . Conf. on. Dynamic Programming, t a . , FI.L. Puterman, Academic Press, New York, 1978. 25. Puterman, M.L. and Brumelle, S.L. , "On the Convergence of Po l icy I terat ion in Stationary Dynamic Programming", Math, of Qper. Res. 4, 1979, 60-69. 26. Puterman, M.L. and Shin, M.C., "Modified Pol icy I terat ion Algorithms for Discounted Markov Decision Problems", Man. S c i . 24, 1978, 1127-1137. 27. Rothblum, (J., "Iterated Successive Approximation for Sequential Decision Processes", Working Paper, Yale Univ., New Haven, 19 79. 28. Smallwood, R.D. and Sondik, E . J . , "The Optimal Control of Pa r t i a l l y Observable Markov Processes over a F in i te Horizon", Qper. Res. 21, 1973, 1300-1322. 29. Sondik, E . J . , "The Optimal Control of Pa r t i a l l y Observable Markov Processes over the In f i n i t e Horizon: Discounted Costs", Qper. Res. 26, 1978, 282-304. 30. van der Wal, J . , "Discounted Markov Games: Generalized Pol icy I te ra -t ion Method", JOTA 25, 1978, 125-138. 31. van Nunen, J .A .E .E . , "A Set of Successive Approximation Methods for Discounted Markovian Decision Problems", Ze i t s ch r i f t fur Qper. Res. 20, 1976, 203-208. 32. van Nunen, J .A .E .E . , Contracting Markov Decision Processes, Mathe-matical Center Tract 71, Amsterdam, 1976. 33. Viswanathan, B., Aggarwal, V.V. and Nair , K.P.K., "Mult ip le C r i t e r i a Markov Decision Processes", NgrtlvHolland TIMS Studies in Management Science, 6, 1977, 263-272. ' 34. White, C C . and Kim, K.W., "Two Solut ion Procedures for Solving Vector Cr i te r ion Markov Decision Processes", Working Paper, Univ. of V i r g i n i a , Cha r l o t t esv i l l e , V i r g i n i a , 1979. 35. White, D.J . , "El iminat ion of Non-optimal Actions in Markov Decision Processes", in Dynamic Programming and Its Appl icat ions, Pro. of I n t l . Conf. on Dynamic Programming, Ed., M.L. Puterman, Academic Press, New York, 1978. 102. APPENDIX Page A) Code for Pol icy I terat ion without Action El iminat ion . . . . . . . 103 B) Code for Pol icy I terat ion with Procedure (I) . . . . , . 105 C) Code for Po l icy I terat ion with Procedure ( I I ) . . . , 107 D) Code for Pol icy I terat ion with Procedure ( I I I ) 109 E) Code for Pol icy I terat ion with Procedure (IV). . . , 112 F) Code for Modified Pol icy I terat ion without Action E l iminat ion. . . 115 G) Code for Modified Pol icy I terat ion with Procedure (II) 117 H) Code for Modified I terat ion with Procedure ( I I I ) 120 I) Code for Modified Pol icy I terat ion with Procedure (IV) 123 J) Code for Generating Random Data 126 K) Code for Generating Data of Two Cr i te r ion Automobile Replacement Problem . . . 127 L) Code for Parametric MDP with Select ion Rule (B) 128 M) Code for Parametric MDP with Select ion Rule (C) I 3 4 N) Code for Parametric MDP with Procedure ( I - l ) 140 0) Code for Parametric MDP with Procedure (1-2) 147 P) Code for Parametric MDP with Procedure ( I I - l ) . 155 Q) Code for Parametric MDP with Procedure (II-2) 163 R) Computational Result of Two Cr i te r ion Automobile Replacement Problem Using Code L with Discount Rate of .96 171 S) Computational Result of Two Cr i te r ion Automobile Replacement Problem using Code .bwi th Discount . Rate of .97. . . ,% . . . . . . . . . . . . . . 175 C * * * A ) CODE FOR P O L I C Y I T E R A T I O N WITHOUT A C T I O N E L I M I N A T I O N C * * * D I M E N S I O N V ( 4 0 ) , C ( 4 0 , 1 0 0 ) , C P V ( 4 0 , 1 0 0 ) , P ( 4 0 , 1 0 0 , 4 3 J , 1 1 P O L ( 4 0 ) 1 A ( 4 0 , 4 0 ) , T ( 4 0 , 4 0 ) , 8 ( 4 0 , 1 ) , I P E R M ( 1 0 0 ) , K K P O L ( 4 0 ) REWIND8 REWIND9 NDIMA=40 NDIMBX=40 NDIMT=40 NS0L=1 R E A D ( 5 , 1 ) N S T , N A C , D I S C R , D E R R 1 F 0 R M A T ( I 5 , I 5 , 2 F i 0 . 5 ) R E A D ( 8 ) ( ( C ( I » K ) , 1 = 1 , N S T ) , K = 1 , N A C ) DO 2 1 = 1 , N S T R E A D ( 9 ) U P < I , K , J ) , K = 1 , N A C ) , J = 1 , N S T ) 2 CONTINUE C A L L T I M E ( O ) ITER=0 E L M = 0 . TNM=NST*NAC KN = 0 DO 50 1 = 1 , N S T DO 20 K = 1 , N A C I F ( K . E Q . l ) GO TO 10 I F ( OC I . GE . C ( I , K ) ) GO TO 20 10 O C I = C ( I , K ) I I P=K 20 CONTINUE B( I , 1 ) = 0 C I K K P O L f I ) = I I P DO 40 J = 1 , N S T I F U . E Q . J l GO TO 3 0 AI I , J ) = - D I S C R * P ( 1 , 1 I P , J ) GO TO 4 0 30 A ( I , J ) = L . - D I S C R * P ( I , I I P , J ) 4 0 C O N T I N U E 50 C O N T I N U E GO TO 130 6 0 I T E R = I T E R + 1 DO 120 1 = 1 , N S T DO 110 K = 1 , N A C I F ( F L A G ! I , K ) . G E . 1 . 0 ) GO TO 110 KN=KN+1 P V = 0 . DO 100 J = 1 , N S T P V = P V + P ( I , K , J ) * V ( J ) 1 0 0 CONTINUE C P V ( I , K ) = C ( I , K ) * D I S C R * P V I F ( K N . E Q . l ) GO TO 105 I F ( C V l . G E . C P V ( I , K ) ) GO TO 110 1 0 5 C V I = C P V ( I , K ) I I P=K 1 1 0 CONTINUE KN = 0 DO 115 J = 1 , N S T IF I I . E Q . J ) GO TO 1 1 3 A( I , J ) = - D I S C R * P ( I , I I P , J ) GO TO 115 1 1 3 A( I , J ) = 1 . - D I S C R * P ( I , I I P , J ) 104. 115 CONTINUE B( I , 1 ) = C ( I , I I P ) B v i = c v i - v m I I P O L d J = H P IF( I . E Q . H GO TO 116 IF (8VMAX.GE. BVI ) GO TQ 117 BVMAX=BVI GQ TO 120 116 BVMAX=BVI . BVMIN=BVI GO TO 120 117 If(BVMIN.LE.BVI) GO TO 120 BVMIN=BVI 120 CONTINUE DO 125 1=1,NST IF( IIPQL(I).NE.KKPOLU ) ) GO TO 127 125 CONTINUE GO TO 200 127 DO 129 1=1,NST KKPOLl I ) = 11 P O L f I ) 129 CONTINUE C*** SOLVE A SYSTEM OF EQUATIONS A*V=B 3/ CALLING SUBROUTINE FSLE 130 CALL FSLE(NST,NDIMA,A,NSOL,NDIM8X,B,V,IPERM,NDIMT,T, E)ET, JEXP) IF (DET) 135,175,135 135 GO TO 60 175 WRITE(6,180) 130 FORMAT!'SOLUTION FAILED') 200 CALL T IM E I I,1J WRITE 16,210) 210 FORMAT (10X, •STATE*,10X,'POLICY* , 15X, •RETURN',10X,*OPT. POLICY') WRITE (6,250) < I ,NAC,V( I),11 P O L ( I ) , I = 1,NST) 250 FORMAT (1 OX,13,12X,13,10X,E16.7,10X,I 3/) WRITE (6,260) 260 FORMAT(10X,»DISCR»,10X,* DERR'»13 X » • ITERATION*) WRITE (6,300) DISCR,DERR,ITER 300 FORMAT(10X,F 5.3,3X,F8.5,8X,15 ) STOP END € * * * B) CODE FOR P O L I C Y I T E R A T I O N WITH PROCEDURE I I ) D I M E N S I O N V { 4 0 ) , C { 4 0 , 1 0 0 ) , C P V ( 4 0 , 1 0 0 ) , P i 4 0 , 1 0 0 , 4 0 ) , 1 1 P O L ( 4 0 ) » I A l 4 0 , 4 0 ) , T ( 4 0 , 4 0 ) , 8 ( 4 0 , 1 ) , I P E R M l 1 0 0 ) , K K P O L ( 4 0 ) , F L A 3 < 4 0 , 1 0 0 ) , 2 F V J 4 0 ) REWIND8 REWIND9 NDIMA=40 NDIMBX=40 NDIMT=40 N S O L = l R £ A D < 5 , 1 ) N S T , N A C , D I S C R , D E R R 1 F O R M A T ( I 5 , I 5 , 2 F 1 0 . 5 ) R E A D ! 8 3 ( ( C ( I , K ) , I = I , N S T ) , K = 1 , N A C ) DO 2 1 = 1 , N S T R E A D I 9 ) { ( P C I , K , J ) , K = 1 , N A C ) , J = 1 , N S T ) 2 CONTINUE C A L L T I M E ( O ) I T E R = 0 E L M = 0 . TNM=NST*NAC KN=0 DO 50 1 = 1 , N S T DO 20 K = 1 , N A C F L A G ( I , K ) = 0 . I F I K . E Q . 1 ) GO TO 10 I F ( O C I . G E . C U , K ) ) GO TQ 20 10 O C I = C ( I , K ) 11 P=K 20 CONTINUE B I I , 1 ) = 0 C I K K P O L < I ) = I I P DO 40 J = 1 , N S T I F ( I . E Q . J ) GO TO 3 0 A l I , J ) = - D I S C R * P ( I , I I P , J ) GO TO 4 0 30 A l I , J ) = 1 . - D I S C R * P ( I , I I P , J ) 4 0 CONTINUE 5 0 CONTINUE GO TO 1 3 0 6 0 I T E R = I T E R + 1 DO 120 1 = 1 , N S T DO 110 K = 1 , N A C I F ( F L A G U , K ) . G E . 1 . 0 ) GO TO 1 1 0 KN=KN+1 P V = 0 . DO 100 J = 1 , N S T p y = P V + P ( I , K , J ) * V I J ) 1 0 0 CONTINUE C P V i i f K l=CUtKI +DISCR*PV I F ( K N . E Q . l ) GO TO 105 I F ( C V I . G E . C P V ( I , K ) ) GO TO 110 1 0 5 C V I = C P V ( I , K ) 11 P=K 1 1 0 C O N T I N U E KN=0 DO 115 J = 1 , N S T I F ( I . E Q . J ) GO TO 1 1 3 A l I , J ) = ~ D I S C R * P ( I , I I P , J ) G O T O 1 1 5 I U D -1 1 3 A ( I t J ) = i « - D I S C R * P ( I » I I P f J J 1 1 5 C O N T I N U E B 1 1 , n = c 1 1 , 1 1 p ) B V I = C V I - V ( I ) F V I I ) = V ( I ) I I P O L I I ) = I I P I F I I . E Q . U G O T O 1 1 6 I F ( 8 V M A X . G E . B V I ) G O T O 1 1 7 B V M A X = B V I G O T O 1 2 0 1 1 6 B V M A X = B V 1 B V M I N = B V I G O T O 1 2 0 1 1 7 I F I B V M I N . L E . B V I ) G O T O 1 2 0 6 V M I N = B V I 1 2 0 C O N T I N U E D O 1 2 5 1 = 1 , N S T I F ( I I P G L C I ) . N E . K K P O L U ) ) G O T O 1 2 7 1 2 5 C O N T I N U E G O T O 2 0 0 1 2 7 D O 1 2 9 1 = 1 , N S T K K P O L I I ) = I I P O L ( I ) 1 2 9 C O N T I N U E C * * * S O L V E A S Y S T E M O F E Q U A T I O N S A * V = B B Y C A L L I N G S U B R O U T I N E F S L E 1 3 0 C A L L F S L £ ( N S T , N D I M A , A , N S Q L , N D I M B X , B , V , I P E R M , N D I M T , T , 3 E T , J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 I F f I T E R . E Q . O ) G O T O 6 0 A D D 3 = B V M I N / ( l . - O I S C R ) S U B T = ( D I S C R * B V M A X ) / ( l . - D I S C R J D O 1 7 1 1 = 1 , N S T A O V i = F V ( I ) + A D D 3 - S U B T D O 1 7 0 K = 1 , N A C I F ( F L A G ! I , K ) . G E . 1 . 0 ) G O T O 1 7 0 I F ( C P V I I f K J . G E . A D V I ) G O T O 1 7 0 F L A G ! I » K ) = 1 . E L M = E L M + 1 . 1 7 0 C O N T I N U E 1 7 1 C O N T I N U E P R C N T = £ L M * 1 0 0 . / T N M W R I T E < 6 , 1 7 2 ) P R C N T 1 7 2 F O R M A T ( 5 X , « P R C N T ' , 5 X , F 1 0 . 5 ) G O T O 6 0 1 7 5 W R I T E ( 6 , 1 8 0 ) 1 8 0 F O R M A T ! ' S O L U T I O N F A I L E D * ) 2 0 0 C A L L T I M E U . l ) W R I T E ( 6 , 2 1 0 ) 2 1 0 F O R M A T ( 1 0 X , • S T A T E * , 1 0 X , » P O L I C Y • , 1 5 X , ' R E T U R N * , 1 0 X , ' O P T . P O L I C Y ' ) W R I T E ( 6 , 2 5 0 ) ( I » N A C » V ( I ) , I I P O L ( I ) , 1 = 1 , N S T ) 2 5 0 F O R M A T ( 1 0 X , 1 3 , 1 2 X , 1 3 , 1 0 X , E 1 6 . 7 , 1 O X , I 3 / ) W R I T E ( 6 , 2 6 0 ) 2 6 0 F O R M A T ( 1 0 X , * D I S C R * , 1 0 X , ' D E R R ' » 1 0 X , * I T E R A T I O N * ) W R I T E ( 6 , 3 0 0 ) D I S C R , D E R R , I T E R 3 0 0 F O R M A T ( 1 0 X , F 5 . 3 , 8 X , F 8 . 5 , 8 X , 1 5 ) S T O P E N D c*** 1 0 7 -C * * * C) CODE FOR P O L I C Y I T E R A T I O N WITH PROCEDURE ( I I ) C * * * D I M E N S I O N V I 4 0 ) , C ( 4 0 , 1 0 0 ) , C P V ( 4 0 , 1 0 0 ) , P ( 4 0 , 1 0 0 , 4 0 ) , I I P O L ( 4 0 ) , l A i 4 0 , 4 0 ) , T 1 4 0 , 4 0 ) , B ( 4 0 , 1 ) , I P E R M ( 1 0 0 ) , K K P O L ( 4 0 ) , F I A S ( 4 0 , 1 0 0 ) , REWIND8 REWIND9 NDIMA=40 NDIMBX=40 NDIMT=40 NSGL=1 R £ A D ( 5 , 1 ) N S T , N A C , D I S C R , D E R R 1 F O R M A T ( 1 5 , I 5 , 2 F 1 0 . 5 ) RE A D ( 8 ) ( ( C ( I , K ) » I = 1 » N S T ) , K = 1 , N A C ) DO 2 1 = 1 , N S T RE A D ( 9 ) ( ( P ( I , K t J ) , K = l , N A C ) , J = 1 , N S T ) 2 CONTINUE C A L L T I M E ( O ) I T E R = 0 E L M = 0 . TNM=NST*NAC KN=0 DO 50 1 = 1 , N S T DO 20 K = 1 , N A C FL A G ( I » K ) = 0 . I F t K . E Q . l ) GO TO 10 I F ( O C I . G E . C ( I , K ) ) GO TO 20 10 O C I = C ( I , K ) I I P=K 2 0 C O N T I N U E B f I , 1 ) = 0 C I K K P O L I I ) = I I P DO 40 J = l , N S T I F ( I . E Q . J ) GO TO 3 0 A ( I , J ) = - D I S C R * P ( I , I I P , J ) GO TO 4 0 3 0 A ( I , J i = l . - D I S C R * P ( I , I I P , J ) 4 0 C O N T I N U E 50 CONTINUE GO TO 1 3 0 6 0 IT E R=IT E R +1 DO 120 1 = 1 , N S T DO 110 K = 1 , N A C I F { F L A G ( I , K ) . G E . 1 . 0 ) GO TO 1 1 0 KN=KN+1 P V = 0 . DO 100 J = 1 , N S T PV = P V * - P ( I , K » J ) * V ( J ) 1 0 0 CONTINUE C P V I I , K ) = C ( I , K) +-DI S C R * PV I F ( K N . E Q . l ) GO TO 105 I F I C V I . G E . C P V ( I , K ) ) GO TO 110 1 0 5 C V I = C P V ( I , K ) I I P = K 1 1 0 CONTINUE KN=0 DO 115 J = 1 , N S T I F ( I . E Q . J ) GO TO 1 1 3 A ( I , J ) = - D I S C R * P I I , I I P , J ) GO TO 1 1 5 1 1 3 4( I , J ) = 1 . ~ D I S C R * P ( I , I I P , J ) 108. 1 1 5 CONTINUE 8( I , 1 ) = C I I , I I P ) B V I = C V I - V ( I ) I I POL 11 ) = I IP I F < I . E Q . l ) GO TO 116 I F ( B V M A X . G E . B V I > GO TO 117 BVMAX=BVI GO TO 1 2 0 1 1 6 BVMAX=BVI B V M I N = B V I GO TO 1 2 0 1 1 7 I F I B V M I N . L E . B V I ) GO TO 120 B V M I N = B V I 1 2 0 CONTINUE 0 0 125 1 = 1 , N S T I F ( I I P G L I I ) . N E • K K P Q L ( I ) ) GO TO 1 2 7 1 2 5 CONTINUE GO TO 2 0 0 1 2 7 DO 129 1 = 1 , N S T K K P O L C I J = I I P O L ( I ) 129 CONTINUE C * « * SOLVE A SYSTEM OF EQUATIONS A*V=B 8Y C A L L I N G S J B R O U T IME F S L E 1 3 0 C A L L F S L E l N S T , N D I M A , A , N S 0 L , N D I M 3 X , 8 , V , I P E R M , N D l M r , T , D E T , J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 I F I I T E R . E Q.O) GO TO 60 S U B T = ( D I S C R * 8 V M A X ) / ( l . - D I S C R ) DO 171 1 = 1 , N S T A D V I = V ( I 1 - S U B T DO 170 K= 1 , N A C I F I F L A G ( I,K ) . G E . 1.0) GO TO 170 I F { C P V { I , K I . G E . A D V I ) GO TO 1 7 0 F L A G { I,K)=1. EL M=ELM + 1 . 1 7 0 C O N T I N U E 171 CONTINUE P R C N T = E L M * 1 0 0 . / T N M U R I T E ( 6 V 1 7 2 ) PRCNT 1 7 2 F Q R M A T(5X » » P R C N T»,5X,Fi0 . 5 ) GO TQ 60 1 7 5 W R I T E(6,180) 1 8 0 F O R M A T ! ' S O L U T I O N F A I L E D ' ) 2 0 0 C A L L T I M E !1,1) WRITE ( 6 , 2 1 0 ) 2 1 0 FORMAT ( 1 OX,•S T A T E • , 1 OX,•POLICY• ,15X,'R E T U R N ',13<» ' O P T . P O L I C Y ' ) WRITE ( 6 , 2 5 0 ) ( I , N A C , V I I ) , I I P O L ( I ) , 1 = 1 , N S T ) 2 5 0 FORMAT (10X,13,12X,I 3 ,10X,E16.7,1 OX,I 3 / ) WRITE (6,260) 2 6 0 FORMAT(10X,•Di SCR *,10X,•DERR',1 OX, » I T E R A T I O N • ) WRITE ( 6 , 3 0 0 ) D I S C R , D E R R , I T E R 3 0 0 F O R M A T(10X,F5 . 3,8X,F8 . 5,8X , I 5 ) STOP END C * * * 0 3 CODE FOR P O L I C Y I T E R A T I O N WITH PROCEDURE ( I I I ) C * * * D I M E N S I O N V ( 4 0 ) , 0 1 4 0 , 1 0 0 ) , C P V ( 4 0 , 1 0 0 ) , P 1 4 0 , 1 0 0 , 4 0 1 , I I P O L ( 4 0 ) 1 A ( 4 0 , 4 0 ) , T ( 4 0 , 4 0 ) , B ( 4 0 , 1 ) , I P E R M ( 1 0 0 ) , K K P O L ( 4 0 ) , F L A 3 ( 4 0 , i 0 0 ) , 2 F V I 4 0 ) REWIND8 REWIND9 NDIMA=40 N D I M B X = 4 0 NDIMT=40 NSQL=1 R E A D ( 5 , 1 ) N S T , N A C . D I S C R , D E R R 1 F O R M A T C I 5 , I 5 , 2 F 1 0 . 5 ) R E A D ! 8 ) ( ( C ( I , K ) , 1 = 1 , N S T ) , K = 1 , N A C ) DO 2 1 = 1 , N S T R E A D 1 9 ) ( ( P ( I , K , J ) , K = 1 , N A C ) , J = 1 , N S T ) 2 C O N T I N U E C A L L T I M E ( O ) ITER=0 E L M = 0 . TNM=NST*NAC KN=0 DO 5 0 1 = 1 , N S T DO 20 K = 1 , N A C F L A G 1 I , K ) = 0 . I F ( K . E Q . l ) GO TO 10 I F ( O C I . G E . C ( I , K ) ) GO TO 2 0 10 OC I = C ( I , K ) I I P = K 2 0 CONTINUE B ( I , 1 ) = 0 C I K K P O L C I ) = I I P DO 4 0 J = i , N S T IF ( I . E Q . J ) GO TO 3 0 AC I , J ) = - D I S C R * P ( I , I I P * J ) GO TO 4 0 3 0 A ( I , J ) = 1 . - D I S C R * P ( I , I I P , J ) 4 0 C O N T I N U E 5 0 CONTINUE GO TO 130 6 0 I T E R = I T E R + I DO 120 1 = 1 , N S T DO 110 K = 1 , N A C I F i F L A G ( I , K ) . G £ . 1 . 0 ) GO TO 110 KN=KN+1 P V = 0 . DO 100 J = 1 , N S T P V = P V * P ( I , K , J ) * V ( J ) 1 0 0 CONTINUE C P V ( I , K ) = C ( I , K ) + D I S C R * P V I F C K N . E Q . l ) GO TO 1 0 5 I F ( C V I . G E . C P V ( I , K ) ) GO TO 110 1 0 5 C V I = C P V C I , K ) I I P = K 1 1 0 CONTINUE KN=0 DO 115 J = 1 , N S T I F ( I . E Q . J ) GO TO 1 1 3 A ( I , J ) = - D I S C R * P { I , I I P , J ) GO TO 1 1 5 1 1 0 -1 1 3 A( I » J ) = l . - D I S C R * P ( I » 1 1 P » J ) 1 1 5 CONTINUE BC I , 1 ) = C U , I I P ) B V I = C V I - V ( I ) F V ( I ) = V ( I ) I I P O L l I l = I I P I F ( I . E Q . l ) GO TO 1 1 6 I F ( B V M A X . G E . B V I i GO TO 117 BVMAX=BVI GO TO 120 1 1 6 BVMAX=8VI B V M I N = B V I GO TO 120 117 I F I B V M I N . L E . B V I ) GO TO 120 B V M I N = B V I 120 C O N T I N U E 0 0 125 1 = 1 , N S T I F ( 1 1 POL I I ) . N E . K K P O L 1 1 ) ) GO TO 1 2 7 1 2 5 C O N T I N U E GO TO 2 0 0 127 DO 1 2 9 1 = 1 , N S T K K P O L ( I > = I I P O L U ) 1 2 9 CONTINUE C * * * SOLVE A S Y S T E M QF EQUATIONS A * V = B BY C A L L I N G S U B R O U T I N E F S L E 130 C A L L F S L E ( N S T , N D I M A , A , N S O L , N D I M B X , B , V , I P E R M » N D I M T , T , D E T , J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 I F ( I T E R . E Q . O ) GO TO 6 0 0 0 163 1 = 1 , N S T D V I = V ( I ) - F V ( I ) I F ( I . E Q . l ) GO TO 1 6 1 I F { D V M A X . G E . D V I ) GO TO 163 1 6 1 DVMAX=DVI 1 6 3 C O N T I N U E A D D 1 = D I S C R * D V M A X E L M = 0 . DO 171 1 = 1 , N S T ADVI=V( I ) DO 170 K = 1 , N A C C P V U , K ) = C P V ( I , K ) + A D D 1 I F I C P V I I t K J . L T . A O V l ) GO TO 165 F L A G ( I , K ) = 0 . GO TO 170 1 6 5 F L A G ( I , K ) = 1 . ELM=ELM + 1 . 170 CONTINUE 171 CONTINUE P R C N T = E L M * 1 0 0 . / T N M W R I T E ( 6 , 1 7 2 ) PRCNT 172 FORMAT t 5 X , • P R C N T * , 5 X , F 1 0 . 5 I GO TO 6 0 1 7 5 W R I T E ! 6 , 1 8 0 ) 1 8 0 F O R M A T ! ' S O L U T I O N F A I L E D * ) 2 0 0 C A L L T I M E I l t D WRITE ( 6 , 2 1 0 ) 2 1 0 FORMAT ( 1 0 X , ' S T A T E » , 1 0 X , » P O L I C Y • , 1 5 X , * R E T U R N ' , 1 0 X , » O P T . P O L I C Y ' ) WRITE ( 6 , 2 5 0 ) C I , N A C , V ( I ) , I I P O L ( I ) , 1 = 1 , N S T ) 2 5 0 FORMAT ( 1 0 X , 1 3 , 1 2 X , 1 3 , 1 0 X , E 1 6 . 7 , 1 O X , I 3 / ) WRITE ( 6 , 2 6 0 ) 2 6 0 F O R M A T ( 1 0 X , ' D I S C R * , 1 0 X , * D E R R * , 1 0 X , * I T E R A T I O N ' ) WRITE <6,300) DISCR,DERR,ITER 300 FORMAT <10X,F5-3,8X,F8.5,8X,I 5 I STOP END c*** ' 1 1 2 -C * * * E ) C O D E F O R P O L I C Y H E R A T I O N W I T H P R O C E D U R E U V ) C * * * D I M E N S I O N V ( 4 0 ) , C ( 4 0 , 1 0 0 ) , C P V ( 4 0 , 1 0 0 ) , P ( 4 0 , 1 0 0 , 4 0 ) » I I P O L ( 4 0 ) , 1 A ( 4 0 , 4 0 ) , T ( 4 0 , 4 0 ) , B ( 4 0 , 1 ) , I P E R M I 1 0 0 ) , K K P O L ( 4 0 ) , F L A 3 ( 40, 1 0 0 ) , 2 F V ( 4 0 , 1 0 0 ) , D V M X { 1 0 0 ) R E W I N D 8 R E W I N D 9 N D I M A = 4 0 N D I M B X = 4 0 N D I M T = 4 0 N S O L = l R E A D ! 5 , i ) N S T , N A C , D I S C R , D E R R 1 F O R M A T ! 1 5 , 1 5 , 2 F 1 0 . 5 ) R E A D ( 8 ) ( ( C ( I , K ) , 1 = 1 , N S T ) , K = 1 , N A C ) D O 2 1 = 1 , N S T R E A D ! 9 ) ( ( P C I , K , J ) , K = 1 , N A C ) , J = 1 , N S T ) 2 C O N T I N U E C A L L T I M E ( O ) I T E R = 0 E L M = 0 . T N M = N S T * N A C K N = 0 D O 5 0 1 = 1 , N S T D O 2 0 K = 1 , N A C F L A G ( I , K ) = 0 . I F ( K . E Q . l ) G O T O 1 0 I F ( O C I . G E . C ( I , K ) ) G O T O 2 0 1 0 O C I = C ( I , K > 1 1 P = K 2 0 C O N T I N U E B ( I , l ) = G C I K K P O L ( I ) = I I P D O 4 0 J = 1 , N S T I F ( I . E Q . J ) G O T O 3 0 A C I , J ) = - D I S C R * P ( I , I I P , J ) G O T O 4 0 3 0 A ( I , J ) = l . - D I S C R * P ( 1 , 1 I P , J ) 4 0 C O N T I N U E 5 0 C O N T I N U E G O T O 1 3 0 6 0 I T E R = I T E R * 1 D O 1 2 0 1 = 1 , N S T D O 1 1 0 K = 1 , N A C I F ( F L A G ( I t K U G E . l . Q ) G O T O 1 1 0 K N = K N t - 1 P V = 0 . D O 1 0 0 J = 1 , N S T P V = P V + P ( I , K , J ) * V ( J ) 1 0 0 C O N T I N U E C P V ( I , K ) = C ( I , K ) + D I S C R * P V I F ( K N . E Q . l ) G O T O 1 0 5 I F ( C V I . G E . C P V ( I , K ) ) G O T O 1 1 0 1 0 5 C V I = C P V ( I , K ) 1 1 P = K 1 1 0 C O N T I N U E K N = 0 D O 1 1 5 J = l , N S T I F ( I . E Q . J ) G O T O 1 1 3 A ( I , J ) = - D I S C R * P ( I , I I P , J ) GO TO 1 1 5 113 A( I » J ) = 1 . - D I S C R * P ( I » I I P » J ) 1 1 3 -1 1 5 CONTINUE 81 1 , 1 ) = C ( I , I I P ) B V I = C V I - V ( I ) F V ( I , I T E R ) = V ( I ) I I P O L ( I ) = I I P I F l I . E Q . l ) GO TO 116 I F I B V M A X . G E . B V I ) GO TO 117 BVMAX=BVI GO TO 1 2 0 1 1 6 BVMAX=BVI B V M I N = B V I GO TO 1 2 0 1 1 7 I F ( B V M I N . L E . B V I ) GO TO 120 B V M I N = B V I 1 2 0 C O N T I N U E 0 0 125 1 = 1 , N S T I F ( I I P O U I ) . N E . K K P O L C 1 1 ) GO TO 1 2 7 1 2 5 C O N T I N U E GO TO 2 0 0 1 2 7 DO 129 1 = 1 , N S T K K P O L I I ) = 11 POL 11) 1 2 9 CONTINUE C * * * SOLVE A SYSTEM OF EQUATIONS A * V = B BY C A L L I N G S U B R O U T I N E F S L E 1 3 0 C A L L F S L E ( N S T , N D I M A , A , N S O L , N D I M B X , B , V , I P E R M , N D I M T , T , S E T , J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 I F ( I T E R . E Q . O ) GO TO 6 0 DO 163 K = l , I T £ R DO 162 1 = 1 , N S T D V I K = V ( I ) - F V ( I , K ) I F ( I . E Q . l ) GO TO 1 6 1 I F ( D V M A X . G E . D V I K ) GO TO 162 1 6 1 DVMAX=DVIK 1 6 2 C O N T I N U E DVMX(K)=DVMAX 1 6 3 CONTINUE E L M = 0 . DO 171 1 = 1 , N S T A D V I = V ( I ) DO 170 K = 1 , N A C L = F L A G ( I , K ) I F ( L . L E . O ) GO TO 1 6 5 C P V A D = C P V ( I , K ) + D I S C R * D V M X ( L ) I F ( C P V A D . L T . A D V I ) GO TO 167 F L A G l I , K ) =0 . GO TO 170 1 6 5 CP V AD=CPV ( I , K) f-DI S C R * D VMX( I T E R ) I F ( C P V A D . G E . A D V I ) GO TO 170 F L A G ! I , K ) = I T E R 1 6 7 E L M = E L M * 1 . 170 CONTINUE 171 CONTINUE P R C N T = E L M * 1 0 0 . / T N M W R I T E ( 6 , 1 7 2 ) PRCNT 1 7 2 F 0 R M A T ( 5 X , ' P R C N T ' , 5 X , F 1 0 . 5 ) GO TO 6 0 1 7 5 W R I T E ( 6 , 1 8 0 ) 1 8 0 FORMATt * SOLUTION F A I L E D ' ) 2 0 0 C A L L T I M E t 1 , 1 3 WRITE 1 6 , 2 1 0 ) 2 1 0 FORMAT ( 1 O X , • S T A T E » , 1 0 X , • P O L I C Y • , 1 5 X , • R E T U R N * , 1 Q X , • 3 P T . P O L I C Y 1 ) WRITE ( 6 , 2 5 0 ) ( I , N A C , V ( I ) , I I P O L ( I ) , 1 = 1 , N S T ) 2 5 0 FORMAT ( 10X , 13 , 1 2X , 13 , 1 OX , E 1 6 , 7 , 1 OX , I 3 / ) WRITE ( 6 , 2 6 0 ) 2 6 0 F O R M A T ( 1 0 X , • D I S C R " , 1 0 X , » D E R R * , 1 Q X , * I T E R A T I O N * ) WRITE ( 6 , 3 0 0 ) D I S C R , D E R R , I T E R 3 0 0 F O R M A T ( 1 0 X , F 5 . 3 , 8 X , F 8 . 5 » 8 X , 1 5 ) STOP ENO c*** C * * * F ) CODE FOR M O D I F I E D P O L I C Y I T E R A T I O N WITHOUT ACTION E L I M I N A T I O N D I M E N S I O N V ( 4 Q ) , C V I 4 0 ) , I I P O L t 4 0 ) , C P V I 4 0 , 1 0 0 ) , C i 40 , 1 0 0 ) , 8 VI 4 0 ) , l P t 4 0 , 1 0 0 , 4 0 ) * O P 1 4 0 , 4 0 ) , 0 C 1 4 0 ) REWINDS REWIND9 R E A D ( 5 , 1 ) N S T , N A C D I S C R , D E R R 1 F 0 R M A T ( I 5 , I 5 , 2 F 1 0 . 5 ) R E A D ( 8 ) ( ( C ( I , K ) , 1 = 1 , N S T ),K = 1 , N A C ) DO 2 1 = 1 , N S T R E A D ( 9 ) ( ( P { I , K , J ) » K = l » N A C ) » J = 1 , N S T ) 2 CONTINUE C A L L T I M E ( O ) I T E R = 0 E L M = 0 . TNM=NST*NAC KN=0 DO 50 1 = 1 , N S T DO 20 K= 1 , N A C I F ( K . E Q . i ) GO TO 1 0 I F ( O C I . G E . C ( I , K ) ) GO TO 20 10 O C I = C ( I , K ) I I P = K 2 0 C O N T I N U E I F U . E Q . l ) GO TO 3 0 I F I C M N . L E . O C I ) GO TO 4 0 3 0 CMN=OCI 4 0 O C M ) = O C I 11 P O L ( I ) = I I P DO 4 5 J = 1 , N S T OPIItJ)-P( I , I I P , J ) 4 5 CONTINUE 5 0 C O N T I N U E A D = ( D I S C R * C M N ) / ( l . - D I S C R ) DO 5 5 1 = 1 , N S T C V ( I ) = O C ( I ) + A D 5 5 CONTINUE NORD=0 GO TO 130 6 0 IT ER=ITER+1 DO 120 1 = 1 , N S T DO 110 K = 1 , N A C I F ( F L A G U , K ) . G E . 1 . 0 ) GO TO 110 KN=KN+1 P V = 0 . DO 100 J = 1 , N S T P V = P V + P I I , K , J ) * V ( J ) 1 0 0 C O N T I N U E C P V ( I , K ) = C U , K ) + D I S C R * P V I F I K N . E Q . l ) GO TO 105 I F ( C V I . G E . C P V I I , K ) ) GO TO 1 1 0 1 0 5 C V I = C P V i I , K ) 11 P=K 110 CONTINUE KN=0 DO 115 J = 1 , N S T O P U , J ) = P ( I , I I P , J ) 1 1 5 CONTINUE OC ( I ) = C ( I , I I P ) c v m = c v i " ° -I I POL ( I j = I I P BVI=CV I - V ( I ) I F ( I . E Q . l ) GO TO 1 1 6 I F I B V M A X . G E . B V I ) GO TO 117 8VMAX=BVI GO TO 120 116 BVMAX=BVI B V M I N = B V I GO TO 1 2 0 117 I F I B V M I N . L E . B V I ) G O T O 120 B V M I N = B V I 120 CONTINUE E R R = B V M A X - B V M I N I F ( ERR . . L E . DERR) GO TO 2 0 0 NQRD=0 1 3 0 NORD=NORD+1 DO 140 1 = 1 , N S T O P V = 0 . DO 135 J = 1 , N S T G P V = O P V + O P ( I , J ) * C V ( J ) 1 3 5 CONTINUE V I = 0 C ( I ) + D I S C R * O P V V I I ) = V I P B V = V I - C V U ) I F ( I . E Q . l ) GO TO 1 3 9 I F I P B V M X . G E . P B V ) GO TO 136 PBVMX=PBV GO TO 140 1 3 6 I F ( P B V M N . L E . P B V ) G O T O 140 PBVMN=PBV GO TO 1 4 0 1 3 9 PBVMX=PBV PBVMN=PBV 1 4 0 CONTINUE E R R = P B V M X - P 8 V M N I F ( E R R . L E . D E R R ) GO TO 1 5 1 DO 141 1 = 1 , N S T CV.11) =V ( I ) 1 4 1 CONTINUE GO TO 1 3 0 151 WRITE ( 6 , 1 5 5 ) NORD 1 5 5 F O R M A T ! 1 0 X , ' N O R D * , 5 X , I 4 ) GO TO 6 0 2 0 0 C A L L T I N E I l f l l WRITE ( 6 , 2 1 0 ) 2 1 0 FORMAT ( 1 0 X , » S T A T E » , 1 0 X , • P O L I C Y * , 1 5 X , • R E T U R N ' , 1 0 K , • D P T . P O L I C Y 1 ) A D = D I S C R * B V M I N / ( l . - D I S C R ) DO 2 0 5 1 = 1 , N S T V ( I ) = C V ( I ) + A D 2 0 5 CONTINUE WRITE ( 6 , 2 5 0 ) ( I , N A C , V ( I ) , 1 1 P O L ( I ) , I = I , N S T ) 2 5 0 FORMAT ( 1 0 X , 1 3 , 1 2 X , I 3 , 1 0 X , E 1 6 . 7 , 1 O X , 1 3 / ) WRITE ( 6 , 2 6 0 ) 2 6 0 F O R M A T { 1 0 X , * D I S C R ' , 1 0 X , ' D E R R * , 1 0 X , 1 I T E R A T I O N * ) WRITE ( 6 , 3 0 0 ) D I S C R » D E R R , I T E R 3 0 0 F O R M A T ( 1 0 X , F 5 . 3 , 8 X , F 8 . 5 , 8 X , 1 5 ) STOP END C * * * G) CODE FOR M O D I F I E D P O L I C Y I T E R A T I O N WITH PROCEDURE ( I I - l C * * * D I M E N S I O N V { 4 0 ) , C V ( 4 0 ) , I I P 0 H 4 0 ) , C P V ( 4 0 , 1 0 0 ) , C ( 4 0 , 1 0 0 ) , B V ( 4 0 ) 1 P ( 4 0 , 1 0 0 , 4 0 ) , O P ( 4 0 , 4 0 ) , 0 C < 4 0 ) , F L A G ! 4 0 , 1 0 0 ) REWIND8 REWIND9 R E A D ( 5 , 1 ) N S T , N A C , D I S C R , D E R R 1 F O R M A T ! 1 5 , 1 5 , 2 F 1 0 . 5 ) RE A D ( 8 1 ( { C ( I , K ) , 1 = 1 , N S T ) , K = l , N A C ) DO 2 1 = 1 , N S T RE A D ( 9 ) ( ( P ( I , K , J ) , K = 1 , N A C ) , J = 1 , N S T ) 2 CONTINUE C A L L T I M E ( O ) I T E R = 0 E L M = 0 . TNM=NST*NAC KN = 0 DO 50 1 = 1 , N S T DO 20 K = i , N A C F L A G ( I , K ) = 0 . I F ( K . E Q . l ) GO TO 1 0 I F ( O C I . G E . C I I , K ) ) GO TO 20 10 O C I = C ( I , K ) 11 P=K 2 0 CONTINUE I F ( I . E Q . l ) GO TO 3 0 I F ( C M N . L E . O C I ) GO TO 4 0 3 0 CMN=OCI 4 0 O C ( I ) = 0 C I I I P O L I I ) = I I P DO 4 5 J = 1 , N S T 0 P ( I , J ) = P ( I , I I P , J ) 4 5 C O N T I N U E 5 0 C O N T I N U E A D = ( D I S C R * C M N ) / ( l . - D I S C R ) DO 55 1 = 1 , N S T C V ( I ) = O C ( I ) + A D 5 5 C O N T I N U E NORD=0 GO TO 1 3 0 6 0 I T E R = I T E R + l DO 120 1 = 1 , N S T DO 110 K = 1 , N A C I F ( F L A G ( I , K ) . G E . 1 . 0 ) GO TO 110 KN=KN+1 P V = 0 . DO 100 J = 1 , N S T P V = P V + P ( I , K , J ) * V ( J ) 1 0 0 CONTINUE C P V ( I , K ) = C ( I , K ) + D I S C R * P V I F ( K N . E Q . l ) GO TO 105 I F ( C V I . G E . C P V ( I , K ) ) GO TO 110 1 0 5 C V I = C P V ( I , K ) I I P=K 1 1 0 CONTINUE KN=0 DO 115 J = 1 , N S T O P I I , J ) = P ( I , I I P , J ) 115 C O N T I N U E OC ( I ) = C ( I » 1 1 P ) C V ( I ) = C V I n P O L t n = 1 1 P B V I = C V I ~ V ( I ) I F U . E Q . l ) GO TO 116 I F I B V M A X . G E . B V I ) GO TO 117 BVMAX=BVI GO TO 1 2 0 116 BVMAX=BVI B V M I N = 8 V I GO TO 120 117 I F I B V M I N . L E . 8 V I ) GO TO 120 B V M I N = B V I 1 2 0 CONTINUE E R R = B V M A X - B V M I N I F ( E R R . L E . D E R R ) GO TO 2 0 0 NORD=0 1 3 0 N0RD=N0RD+1 DO 140 1 = 1 , N S T O P V = 0 . DO 1 3 5 J = 1 , N S T Q P V = O P V + O P ( I , J ) * C V ( J ) 1 3 5 C O N T I N U E VI=OC( I ) + D I S C R * O P V VI I )=VI P B V = V I - C V ( I ) I F U . E Q . l ) GO TO 1 3 9 I F I P B V M X . G E . P B V ) GO TO 136 PBVMX=PBV GO TO 1 4 0 1 3 6 I F ( P B V M N . L E . P B V ) GO TO 1 4 0 PBVMN=PBV GO TO 1 4 0 139 PBVMX=PBV PBVMN=PBV 140 CONTINUE E R R = P B V M X - P B V M N IF I E R R . L E . D E R R ) GO TO 151 DO 141 1 = 1 , N S T C V U ) = V U ) 141 CONTINUE GO TO 130 1 5 1 WRITE ( 6 , 1 5 5 ) NORD 1 5 5 F O R M A T ( 1 0 X , « N O R D » , 5 X , 1 4 ) I F I I T E R . E Q . O ) GO TO 6 0 S U B T = ( D I S C R * B V M A X - ( D I S C R * * ( N 0 R D + 1 ) ) * B V M I N ) / ( l . - D I S C R ) DO 171 1 = 1 , N S T ADVI=V( D - S U B T DO 170 K = l , N A C I F C F L A G l I , K ) . G E . i . O ) GO TO 170 I F I C P V l I , K ) . G E . A D V I ) GO TO 1 7 0 F L A G ( I » K ) = 1 . ELM=ELM*-1 . 1 7 0 CONTINUE 171 CONTINUE P R C N T = E L M * 1 0 0 . / T N M W R I T E ( 6 , 1 7 2 ) PRCNT 1 7 2 F O R M A T ( 3 0 X , • P R C N T * , 5 X , F 1 0 . 5 ) GO TO 6 0 2 0 0 C A L L T I M E ( 1 , 1 ) WRITE (6,210) l | y ' 210 FORMAT (1OX,•STATE•,10X,•POL ICY•,15X,•RETURN 1,10X,•3PT. POLICY1) AD=DISCR*BVMlN/l1.-DISCR) DO 205 1=1,NST V( I)=CV(I)+AD 205 CONTINUE WRITE (6,250) (I,NAC» V(I),11POL(I),1 = 1,NST) 250 FORMAT I 1 OX,13,12X,13,10X,E16.7,10X,13/) WRITE (6,260) 260 FORMAT(10X,•DISCR *,10X,•DERR *,10X »•ITERATION') WRITE (6,300) DISCR,DERR,ITER 300 FORMAT(10X,F5.3»8X»F8.5»8X»I5) STOP END C * * * H) CODE FOR MODIFIED POLICY ITERATION WITH PROCEDURE ( I I I ) C * * * DIMENSION V < 4 0 ) , C V { 4 0 ) , 1 1 P O L < 4 0 ) , C P V C 4 0 , 1 O O ) , C I 4 0 , 1 0 0 ) , B V ( 4 0 ) , I P I 4 0 , 1 0 0 , 4 0 ) , O P ( 4 0 , 4 0 ) , O C ( 4 0 ) , F L A G ( 4 0 , 1 0 0 ) , F V { 4 0 ) REWIND8 REWIND9 R E A D ( 5 , 1 ) NST,NAC,DISCR*DERR 1 F Q R M A T ( I 5 , I 5 , 2 F 1 0 . 5 ) R E A D ! 8 ) ( ( C ( I t K ) , 1 = 1 , N S T ) , K = l , N A C ) DO 2 1=1,NST R E A D 1 9 ) { i P l I , K , J ) , K = 1 , N A C ) , J = 1 , N S T ) 2 CONTINUE CALL T I M E ( O ) IT£R=0 ELM=0. TNM=NST*NAC KN=0 DO 50 1=1,NST DO 20 K=1,NAC F L A G ( I , K ) = 0 . I F ( K . E Q . l ) GO TO 10 I F I Q C I . G E . C C I , K ) ) GO TO 20 10 O C I = C C I , K ) 11 P=K 20 CONTINUE I F ( I . E Q . l ) GO TO 30 I F C C M N . L E . O C I ) GO TO 4 0 30 CMN=OCI 40 O C ( I ) = O C I 11 P O L ( I ) = 11P DO 45 J = i , N S T O P U , J ) = P ( I , I I P , J ) 45 CONTINUE 50 CONTINUE A D = ( D I S C R * C M N ) / ( 1 . - D I S C R ) DO 55 1=1,NST CV11 ) = 0 C ( I ) + A D 55 CONTINUE NORD=0 GO TO 130 60 ITER=ITER+1 DO 120 1=1,NST DO 110 K=1,NAC I F { F L A G ( I , K ) . G E . 1 . 0 ) GO TO 110 KN=KN+1 P V = 0 . DO 100 J=l , N S T P V = P V + P ( 1 , K , J ) * V ( J ) 100 CONTINUE C P V ( I , K ) = C ( I , K ) + D I S C R * P V I F I K N . E Q . 1 ) GO TO 105 I F I C V I . G E . C P V ( I , K ) ) GO TO 110 105 C V I = C P V ( I , K ) I I P = K 110 CONTINUE KN=0 DO 115 J=1 ,NST G P ( I , J } = P ( I , I I P , J ) 115 CONTINUE 0 C ( I ) = C ( I , I I P ) C V ( I ) = C V I 11 P O L { I ) = 1 1 P B V I = C V I - V ( I ) F V U ) = V ( I ) I F U . E Q . l ) GO TO 1 1 6 I F { B V M A X . G E . B V I ) GO TO 117 BVMAX=BVI GO TO 1 2 0 116 BVMAX=BVI B V M I N = B V I GO TO 120 1 1 7 I F I B V M I N . L E . B V I ) GO TO 120 B V M I N = B V I 1 2 0 CONTINUE E R R = B V M A X - B V M I N I F ( E R R . L E . O E R R ) GO TO 2 0 0 NORD=0 1 3 0 NORD=NORD+I DO 140 1 = 1 , N S T O P V = 0 . DO 135 J = 1 , N S T O P V = O P V + O P ( I , J ) * C V ( J ) 1 3 5 CONTINUE V I = O C ( I ) + O I S C R * O P V VI I ) = V I P B V = V I - C V U ) I F U . E Q . l ) GO TO 1 3 9 I F ( P B V M X . G E . P B V ) GO TO 136 PBVMX=PBV GO TO 140 1 3 6 I F ( P B V M N . L E . P B V ) GO TO 140 PBVMN=PBV GO TO 140 1 3 9 PBVMX=PBV PBVMN=PBV 1 4 0 CONTINUE E R R = P B V M X - P B V M N I F ( E R R . L E . O E R R ) GO TO 151 DO 141 1 = 1 , N S T cvm=v(i) 1 4 1 CONTINUE GO TO 1 3 0 151 WRITE ( 6 , 1 5 5 ) NORD 1 5 5 F 0 R M A T ( 1 0 X , « N O R D « , 5 X , I 4 ) I F ( I T E R . E Q . 0 ) GO TO 6 0 DO 163 1 = 1 , N S T D V I = V ( I ) - F V ( I ) I F U . E Q . l ) GO TO 161 I F ( D V M A X . G E . D V I ) GO TO 163 161 DVMAX=DVI 1 6 3 C O N T I N U E A D D 1 = D I S C R * D V M A X A D D 2 = D I S C R * P B V M N E L M = 0 . DO 171 1 = 1 , N S T ADV I = V ( I ) + A D D 2 DO 170 K = 1 , N A C C P V ( I , K ) = C P V ( I , K ) + A D D 1 I F ( C P V ( I , K ) . L T . A D V I ) GO TO 165 F L A G U , K ) = 0 . GO TO 1 7 0 1 6 5 F L A G ( I , K } = 1 . ELM=ELM+1. 1 7 0 CONTINUE 17.1 CONTINUE P R C N T = E L M * 1 0 0 . / TNM W R I T E ( 6 , 1 7 2 ) PRCNT 1 7 2 F O R M A T ( 3 0 X , • P R C N T » , 5 X , F 1 0 . 5 ) GO TO 6 0 2 0 0 C A L L TI ME 11 * 1) WRITE ( 6 , 2 1 0 ) 2 1 0 FORMAT ( 1 0 X , « S T A T E • , 10X , • P O L I C Y ' , 1 5 X , ' R E T U R N ' , 1 0 X , « O P T . P O L I C Y ' ) A D = D I S C R * B V M I N / ( l . - D I S C R ) DO 2 0 5 1 = 1 , N S T V( I ) = C V ( I ) + A D 2 0 5 CONTINUE WRITE ( 6 , 2 5 0 ) ( I , N A C , V ( I ) , 1 1 P O L ( I ) , 1 = 1 , N S T ) 2 5 0 FORMAT ( 1 0 X , 1 3 , 1 2 X , 1 3 , 1 0 X , E 1 6 . 7 , 1 O X , I 3 / ) WRITE ( 6 , 2 6 0 ) 2 6 0 F O R M A T ( 1 0 X , * D I S C R ' , 1 O X , ' D E R R ' , 1 0 X , ' I T E R A T I O N ' ) WRITE ( 6 , 3 0 0 ) D I S C R , D E R R , I T E R 3 0 0 F O R M A T l 1 0 X , F 5 . 3 , 8 X , F 8 . 5 , 8 X , I 5 ) STOP END C * * * I ) CODE FOR M O D I F I E D P O L I C Y I T E R A T I O N WITH PROCEDURE ( I V ) C * * * D I M E N S I O N V I 4 0 ) , C V ( 4 0 ) , 1 1 P O L ( 4 0 ) , C P V ( 4 0 , 1 0 0 ) , C ( 4 0 , 1 0 0 ) , B V I 4 0 ) , 1 P ( 4 0 , 1 0 0 , 4 0 ) , O P ( 4 0 , 4 0 ) , 0 C ( 4 0 ) , F L A G ( 4 0 , 1 0 0 ) , F V ( 4 0 , 1 0 0 ) , D V M X ( 1 0 0 ) REWIND8 REWIND9 R E A D ( 5 , 1 ) N S T , N A C , D I S C R , D E R R 1 F O R M A T ( 1 5 , 1 5 , 2 F 1 0 . 5 ) RE A D ( 8 ) ( ( C ( I , K ) , I = 1 , N S T ) » K = 1 » NAC) DO 2 1 = 1 , N S T R E A D ! 9 ) i ( P i I , K , J ) , K = U N A C ) , J = i , N S T ) 2 C O N T I N U E C A L L T I M E I O ) I T E R = 0 E L M = 0 . TNM=NST*NAC KN=0 DO 50 1 = 1 , N S T DO 20 K = 1 , N A C F L A G ! I , K ) = 0 . I F ( K . E Q . l ) GO TO 10 I F ( O C I . G E . C ( I , K ) ) GO TO 20 10 O C l = C ( I , K ) I I P = K 20 CONTINUE I F ( I . E Q . l ) GO TO 3 0 I F ( C M N . L E . O C I ) GO TO 4 0 30 CMN=OCI 4 0 O C ( I ) = O C I 11 P O L ( I ) = I I P DO 4 5 J = 1 , N S T O P ( I , J ) = P ( I , I I P , J ) 4 5 C O N T I N U E 5 0 CONTINUE A D = ( D I S C R * C M N ) / ( l . - D I S C R ) DO 55 1 = 1 , N S T C V ( I ) = 0 C ( I ) + A D 5 5 C O N T I N U E NORD=0 GO TO 130 6 0 I T E R = I T E R + 1 DO 120 1 = 1 , N S T DO 110 K = 1 , N A C I F ( F L A G ( I , K ) . G E . l . 0 ) GO TO 110 KN= K N + 1 P V = 0 . DO 100 J = 1 , N S T P V = P V f P ( I , K , J ) * V ( J ) 1 0 0 C O N T I N U E C P V ( I , K ) = C ( I , K ) + D I S C R * P V I F ( K N . E Q . l ) GO TO 105 I F ( C V I . G E . C P V ( I , K ) ) GO TO 110 1 0 5 C V I = C P V I I,K) I I P = K 1 1 0 CONTINUE KN=0 DO 115 J = 1 , N S T O P ( I , J ) = P ( I , I I P , J ) 115 C O N T I N U E o c m = c u , I I P I C V ( I ) = C V I IIPOL( n = n p B V I = C V I - V ( I ) F V ( I , I T E R ) = V ( I ) I F ( I . E Q . l ) GO TO 116 I F ( B V M A X . G E . B V I ) GO TO 117 BVMAX=BVI GO TO 120 1 1 6 BVMAX=BVI B V M I N = B V I GO TO 1 2 0 117 I F { B V M I N . L E . B V I ) GO TO 120 B V M I N = B V I 1 2 0 C O N T I N U E E R R = B V M A X - B V M I N I F I E R R . L E . O E R R ) GO TO 2 0 0 NORD=0 1 3 0 N0R0=N0RD+1 DO 140 1 = 1 , N S T O P V = 0 . DO 135 J = i , N S T O P V = O P V « - O P ( I , J ) * C V ( J ) 1 3 5 C O N T I N U E V I = 0 C ( I ) + D I S C R * O P V VI I)=VI P B V = V I - C V { I ) IF ( I . E Q . l ) GO TO 1 3 9 I F ( P B V M X . G E . P B V ) GO TO 136 PBVMX=PBV GO TO 1 4 0 1 3 6 I F ( P B V M N . L E . P B V ) GO TO 140 PBVMN=PBV GO TO 1 4 0 1 3 9 PBVMX=PBV PBVMN=PBV 1 4 0 CONTINUE ERR=PBVMX—PBVMN IF ( E R R . L E . O E R R ) GO TO 151 DO 141 1 = 1 , N S T C V ( I ) = V ( I ) 1 4 1 CONTINUE GO TO 1 3 0 151 WRITE ( 6 , 1 5 5 ) NORD 1 5 5 F O R M A T ( 1 0 X , • N O R D 1 , 5 X ,I 4 ) I F ( I T E R . E Q . 0 ) GO TO 6 0 0 0 163 K = 1 , I T E R DO 162 1 = 1 , N S T D V I K = V ( I ) - F V ( I , K ) I F U . E Q . l ) GO TO 1 6 1 I F ( D V M A X . G E . D V I K ) GO TO 162 161 DVMAX=DVIK 1 6 2 CONTINUE DVMX(K)=DVMAX 1 6 3 C O N T I N U E A D D 2 = D I S C R * P B V M N E L M = 0 . DO 171 1 = 1 , N S T A D V I = V ( I ) f A D D 2 DO 170 K = i , N A C L = F L A G ( 1 1 K ) 125 I F ( L . L E . O ) GO T O 1 6 5 C P V A D = C P V { I , K ) + D I S C R * O V M X ( L ) I F J C P V A D . L T . A D V I ) GO T O 1 6 7 F L A G ( I » K ) = 0 • GO T O 1 7 0 1 6 5 C P V A D = C P V ( I , K ) + D I S C R * D V M X ( I T E R ) I F ( C P V A D . G E . A D V I ) GO T O 1 7 0 F L A G ( I » K ) = I T E R 1 6 7 E L M = E L M + 1 . 1 7 0 C O N T I N U E 1 7 1 C O N T I N U E P R C N T = E L M * 1 0 0 . / T N M W R I T E ( 6 , 1 7 2 ) P R C N T 1 7 2 F O R M A T ! 3 Q X , • P R C N T 1 , 5 X » F 1 0 . 5 ) G O T O 6 0 2 0 0 C A L L T I M E ( 1 , 1 ) W R I T E ( 6 , 2 1 0 ) 2 1 0 F O R M A T ( 1 0 X , • S T A T E » , 1 0 X , » P O L I C Y » , 1 5 X , 1 R E T U R N • , 1 0 X , • 3 P T . P O L I C i * * ) A D = D I S C R * B V M I N / ( l . - D I S C R ) DO 2 0 5 1 = 1 , N S T V U ) = C V ( I ) + A D 2 0 5 C O N T I N U E W R I T E ( 6 , 2 5 0 ) ( I , N A C , V ( I ) , 1 1 P O L ( I ) , 1 = I , N S T ) 2 5 0 F O R M A T ( 1 O X , 1 3 , 1 2 X , I 3 , 1 0 X , E 1 6 . 7 , 1 O X , I 3 / ) W R I T E ( 6 , 2 6 0 ) 2 6 0 F O R M A T ( 1 0 X , » D I S C R » , 1 0 X , ' D E R R * , 1 0 X , • I T E R A T I O N » ) W R I T E ( 6 , 3 0 0 ) D I S C R , D E R R , I T E R 3 0 0 F O R M A T ( 1 0 X , F 5 . 3 , 8 X , F 8 . 5 , 8 X , I 5 ) S T O P E N D C * * * J ) CODE FOR G E N E R A T I N G RANDOM DATA C * * * DIMENS I ON P ( 4 0 0 , 1 0 0 ) , I P ( 1 0 0 ) , C ( 1 0 0 , REWIND8 REWIND9 RE A D ( 5 , 1 0 ) N S T , N A C , N Z E R O , S , R 10 F 0 R M A T ( 3 1 5 , 2 F 1 0 . 3 ) X = R A N D N I S ) Y=RAND(R) J N Z = I R A N D ( - N S T ) DO 2 0 0 1 = 1 , N S T DO 100 K = 1 , N A C 2 0 X = 0 . - F R A N O N ( S ) I F I X . L T . O . ) X = - X I F I X . G E . 2 . ) GO TO 20 I C I K = 1 0 0 0 „ - X * 1 0 0 . C ( I , K ) = I C I K - I * 1 0 100 CONTINUE 2 0 0 CONTINUE W R I T E 1 8 ) ( ( C ( I , K ) , I = 1 , N S T ) , K = 1 , N A C ) 1 = 1 2 5 0 DO 4 0 0 K = 1 , N A C 0 0 3 0 0 J = 1 , N S T P ( K , J ) = 0 . 3 0 0 CONTINUE ISUM=0 DO 3 2 0 L = 1 , N Z E R 0 I Y = F R A N D ( R ) * 1 0 0 . + 1 0 0 . I S U M = I S U M * I Y I P ( U = I Y 3 2 0 CONTINUE N M Z R = N Z E R 0 - 1 M = l S S U M = 0 . 3 4 0 P K L = 1 Q 0 * I P I M ) / I S U M 3 5 0 J N Z = I R A N D ( N S T ) I F ( P ( K , J N Z ) . G T . O . O i GO TO 3 5 0 P ( K , J N Z ) = P K L * 0 . 0 i SSUM=SSUM*-P ( K , J N Z ) I F ( M . G E . N M Z R ) GO TO 3 6 0 M=M+1 GO TO 3 4 0 3 6 0 J N Z = I R A N D ( N S T ) I F ( P ( K , J N Z J . G T . O . O ) GO TO 3 6 0 P ( K , J N Z ) = 1 . - S S U M 4 0 0 CONTINUE WRITE ( 9 ) ( ( P ( K , J ) , K - 1 , NAC) , J = l , M S T ) I F ( I . G E . N S T ) GO TO 5 0 0 1 = 1*1 GO TO 2 50 5 0 0 W R I T E ( 9 , 5 1 0 ) 5 1 0 F O R M A T ( • S C O N T I N U E WITH D R 4 0 X 4 0 * ) STOP END C * * * K) CODE FOR G E N E R A T I N G DATA OF TWO C R I T E R I O N AUTOMOBILE 1 2 7 , C REPLACEMENT PROBLEM D I M E N S I O N C S t 5 0 ) , T ( 5 0 ) , E ( 5 0 ) » P R ( 5 0 ) , C ( 5 0 * 5 0 ) v P ( 5 3 , 5 0 ) , C T H R ( 5 0 , 5 0 ) REWIND8 REWIND9 DO 2 0 0 1 = 1 , 4 1 R E A D ( 5 , 1 0 0 ) C S ( I ) , T ( I ) , E ( I ) , P R ( I ) 1 0 0 FORMAT ( F 1 0 . 1 , F 1 0 . 1 , F 1 0 . 1 , F 1 0 . 3 ) W R I T E(6, 1 0 0 ) C S ( I ) , T l I ) , E ( I ) , P R ( I ) 2 0 0 CONTINUE DO 5 0 0 1 = 1 , 4 0 M= I + l AI=M DO 4 0 0 K = l , 4 1 I F ( K . E Q . I ) G O TO 2 1 0 L = K - 1 AL =L C l I , K ) = T ( M ) - C S ( L ) - E ( L ) C T H R ( I » K ) = 4 0 0 . / ( A L * * 0 . 5 ) - C ( I , K ) GO TO 2 20 2 1 0 C ( I , K ) = - E ( M ) C T H R ( I , K ) = 4 0 0 . / ( A 1 * * 0 . 5 ) ~ C ( I , K ) 2 2 0 DO 3 0 0 J = l , 4 0 I F ( K . G E . 2 ) GO TO 2 5 0 I F d . G E . 3 9 l GO TO 2 4 5 I F ( J . E Q . M ) GO TO 2 3 0 I F ( J . E Q . 4 0 ) GO TO 240 P ( K , J ) = 0 . GO TO 3 0 0 2 3 0 P ( K , J ) = P R ( M ) GO TO 3 0 0 2 4 0 P ( K , J ) = 1 . - P R ( M ) GO TO 3 0 0 2 4 5 I F ( J . E Q . 4 0 ) GO TO 2 4 6 P ( K , J ) = 0 . GO TO 3 0 0 2 4 6 P ( K , J ) = 1 . 0 GO TO 3 0 0 2 5 0 I F ( K . £ Q . 4 1 ) GO TO 2 8 0 I F U . E Q . L ) GO TO 2 6 0 I F U . E Q . 4 0 I GO TO 2 7 0 P ( K , J ) = 0 . GO TO 3 0 0 2 6 0 P ( K , J ) = P R ( L I GO TO 3 0 0 2 7 0 P ( K , J ) = 1 . - P R ( L ) GO TO 3 0 0 2 8 0 I F U . E Q . 4 0 ) GO TO 2 9 0 P ( K , J ) = 0 . 0 GO TO 3 0 0 2 9 0 P ( K , J ) = 1 . 0 3 0 0 CONTINUE 4 0 0 CONTINUE W R I T E ( 9 ) ( ( P ( K , J ) , K = 1 , 4 1 ) , J = 1 , 4 0 ) 5 0 0 C O N T I N U E W R I T E ( 8 ) { ( C t l t K ) , C T H R ( I , K ) , I = 1 , 4 0 ) , K = 1 , 4 1 ) STOP END C * * * L ) C O D E F O R P A R A M E T R I C MDP W I T H S E L E C T I O N R O L E (3) C * * * R E A L * 8 T A L P H t D A L P H , A L P H A t A , B , T , V , D E T f C O N D , U f R V f O J , B V f B J , P V , P U , 1 0 B V , O B U , B V l K , B U I K , O O B U , B V M X , B V M M , B U M X , 3 U M M , R B V , D 3 J , 3 J M X , P D U M X , 2 D U M A X , E X A L P , S A L P H , A D E P S , S B E P S D I M E N S I O N V I 5 0 ) , R l 5 0 , 5 0 ) , P ( 5 0 , 5 0 , 5 0 ) , 1 1 P O L 1 5 0 ) , D ( 5 0 , 5 3 ) , A l 5 0 , 5 0 ) , 1 T { 5 0 , 5 0 ) , 8 ( 5 0 , 1 ) , I P E R M ( 1 0 0 ) , K K P O L 1 5 0 ) , U ( 5 0 ) , R V ( 5 3 ) , 0 J ( 5 0 ) , 2 B V 1 5 0 , 5 0 ) , B U I 5 0 , 5 0 ) , B U M 5 0 ) R E W I N D 8 R E W I N D 9 E R R = 0 . 0 0 1 E P S L N = 0 . 0 0 0 0 0 1 N D I M A = 5 0 N D I M B X = 5 0 N D I M T = 5 0 N S 0 L = 1 R £ A D ( 5 , 1 ) N S T , N A C , D I S C R , A L P M N , A L P M X 1 F 0 R M A T ( 2 I 5 , 3 F 1 0 . 5 ) A L P H A = A L P M N R E A D ! 8 ) ( ( R ( I , K ) , D { I , K ) , 1 = 1 , N S T ) , K = 1 , N A C ) I T E R = 0 I T R = 0 0 0 2 1 = 1 , N S T R E A D ( 9 ) U P ( I , K , J ) , K = 1 , N A C ) , J = 1 , N S T ) 2 C O N T I N U E DO 5 0 1 = 1 , N S T DO 2 0 K = 1 , N A C R D I K = R ( I , K ) + A L P H A * D ( I » K ) I F ( K . E Q . 1 ) GO T O 1 0 I F ( O R I . G E . R D I K ) G O TO 2 0 1 0 O R I = R D I K I I P = K 2 0 C O N T I N U E B I 1 , 1 ) = O R I K K P O L ( I ) = I I P D O 4 0 J = 1 , N S T I F ( I . E Q . J ) G O T O 3 0 A ( I , J ) = - D I S C R * P ( I , I I P , J 3 GO T O 4 0 3 0 At I , J ) = 1 . - 0 I S C R * P ( I , I I P , J ) 4 0 C O N T I N U E 5 0 C O N T I N U E G O TO 1 3 0 6 0 I T E R = I T E R + 1 DO 1 2 0 1 = 1 , N S T DO 1 1 0 K = 1 , N A C P V = 0 . DO 1 0 0 J = i , N S T P V = P V + P ( I , K , J ) * V ( J ) 1 0 0 C O N T I N U E R P V I K = R t I , K ) + A L P H A * D ( I , K ) + D I S C R * P V I F ( K . E Q . 1 ) G O T O 1 0 5 I F ( R V I . G E . R P V I K ) G O T O 1 1 0 1 0 5 R V I = R P V I K 1 1 P = K 1 1 0 C O N T I N U E DO 1 1 5 J = 1 , N S T I F ( I . E Q . J ) G O T O 1 1 3 A ( I , J ) = - D I S C R * P ( I , I I P , J ) GO TO 1 1 5 1 1 3 A ( I , J ) = 1 . - D I S C R * P ( I , 11 P , J I ]• 1 1 5 CONTINUE 8( 1 , 1 ) = R ( I » 11 P ) * A L P H A * D ( I , 1 1 P ) 11 POL I I ) = I I P 1 2 0 CONTINUE DO 125 1 = 1 , N S T I F t I I P O L U ) . N E . K K P O L ( I ) ) GO TO 1 2 7 1 2 5 CONTINUE GO TO 190 127 DO 129 1 = 1 , N S T K K P O L ( I ) = I I P O L ( I ) 129 CONTINUE C * * * SOLVE THE SYSTEM OF EQUATIONS AV=B BY C A L L I N G SUBROUTINE SLE 130 C A L L S L E ( N S T , N D I M A , A , N S O L , N D I M B X , B , V , I P E R M , N D I M T , T , D E T , J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 GO TO 6 0 1 7 5 W R I T E ( 6 , 1 8 0 ) 1 8 0 F O R M A T ! 5 X , * SOLUTION F A I L E D ' ) GO TO 7 0 0 1 9 0 0 0 2 0 0 1 = 1 , N S T I I P = I I P O L i I ) P V = 0 . DO 192 J = 1 , N S T P V = P V + P ( I , I I P , J ) * V ( J ) 1 9 2 CONTINUE RV i I ) = R ( I , I I P ) + A L P H A * D f I , 1 I P ) * D I S C R * P V R B V = R V I I ) - V I I ) I F ( I . E Q . l ) GO TO 198 I F ( R B V . G T . B V M X ) GO TO 1 9 4 IF ( R B V . L T . B V M N . ) GO TO 196 GO TO 2 0 0 1 9 4 BVMX=RBV GO TO 2 0 0 1 9 6 BVMN=RBV GO TO 2 0 0 198 BVMX=RBV BVMN=RBV 2 0 0 CONTINUE VERR=BVMX-BVMN I F ( I T R . E Q . O . A N D . V E R R . L E . E R R ) GO TO 2 0 6 A D D = D I S C R * B V M N / ( 1 . - D I S C R ) DO 2 0 2 1 = 1 , N S T VI I ) = R V ( I ) + A D D 2 0 2 CONTINUE I F ( V E R R . L E . E R R ) GO TO 2 0 4 ITR=ITR+1 GO TO 1 9 0 2 0 4 W R I T E ( 6 , 2 0 5 ) ITR 2 0 5 F O R M A T ( 5 X , ' N U M B E R OF I T E R A T I O N I N T - O P E R A T I O N F2\ V I S ' , 5 X , I 5 ) GO TO 6 0 2 0 6 W R I T E I 6 , 2 0 5 ) ITR W R I T E ( 6 , 2 0 8 ) A L P H A 2 0 8 F O R M A T ( 5 X , ' A L P H A = ' , F 2 0 . 1 4 ) A D D = B V M N / ( 1 . - D I S C R ) DO 2 0 9 1 = 1 , N S T VI I ) = VI I)+ADD 2 0 9 CONTINUE W R I T E ( 6 , 2 1 0 ) 2 1 0 F O R M A T ( 1 0 X , ' S T A T E ' , 1 0 X , ' P O L I C Y ' , 1 5 X , ' R E T U R N ' , 1 0 X , ' O P T . P O L I C Y 1 ) HR I TE I 6 , 2 2 0 ) ( 11 NAC , VI I ) , .11 POL ( I ) * I = 1 , NS T ) 2 2 0 FORMAT ( 1 0 X , I 3 , 1 2 X , I 3 , 1 0 X , E 1 6 . 7 , 1 O X , 1 3 / ) 1 3 0 W R I T E ( 6 , 2 3 0 ) 2 3 0 F O R M A T ! 1 0 X , V D I S C R ' , 1 0 X , • I T E R A T I O N 1 ) W R I T £ ( 6 , 2 4 0 ) D I S C R , I T E R 2 4 0 F O R M A T ( 1 O X « F 7 . 5 , 6 X , I 5) W R I T E ( 6 , 2 4 2 ) 2 4 2 F O R M A T ( / / / I X , O P T I M A L A C T I O N ! S ) TO BE CHOSEN FQ* THE CURRENT 1 Q P T I M A L P O L I C Y ' / / / ) C * * * F I N D THE I N V E R S E OF M A T R I X A BY C A L L I N G SUBROUTINE INV C A L L I N V ( N S T , N D I M A , A , I P E R M , N D I M T , T , D E T , J E X P , C O N D J I F ( D E T ) 2 4 5 , 6 2 0 , 2 4 5 2 4 5 DO 2 5 0 1 = 1 , N S T T D = 0 . DO 247 J = 1 , N S T 1 1 P = I I P O L ( J ) T D = T D + T ( I , J ) * D ( J , I I P ) 2 4 7 CONTINUE U( I )=TD 2 5 0 CONTINUE I T R = 0 2 6 0 DO 3 0 0 1 = 1 , N S T I I P = I I P O L ( I ) P U = 0 . DO 2 6 5 J = 1 , N S T P U = P U + P ( I , I I P , J ) * U ( J ) 2 6 5 CONTINUE D U ( I ) = D ( I , I I P ) + D I S C R * P U DBU=DU( I ) - U ( I ) IF I I . E Q . l ) GO TO 2 9 0 I F ( D B U . G T . B U M X ) GO TO 2 7 0 I F ( D B U . L T . 8 U M N ) GO TO 2 8 0 GO TO 3 0 0 2 7 0 8UMX=DBU GO TO 3 0 0 2 8 0 BUMN=DBU GO TO 3 0 0 2 9 0 BUMX=DBU BUMN=DBU 3 0 0 CONTINUE UERR=8UMX-BUMN I F ( U E R R . L E . E R R ) GO TO 3 1 0 DO 3 0 5 1 = 1 , N S T U ( I ) = D U ( I ) 3 0 5 CONTINUE I T R = I T R + 1 GO TO 2 6 0 3 1 0 A D D = D I S C R * B U M N / ( l . - D I S C R ) DO 320 1 = 1 , N S T U ( I ) = D U ( I ) + A D D 3 2 0 CONTINUE W R I T E ( 6 , 5 9 0 ) ITR DO 3 2 5 1 = 1 , N S T DO 3 2 3 K = 1 , N A C P V = 0 . DO 3 2 1 J = 1 , N S T P V = P V + P ( I , K , J ) * V ( J ) 3 2 1 C O N T I N U E 8 V ( I , K ) = R ( I , K ) + A L P H A * D ( I , K ) + D I S C R * P V - V ( I ) 3 2 3 CONTINUE 3 2 5 CONTINUE C A L L T I M E ( O ) ITER=1 3 30 0 0 3 4 0 1 = 1 , N S T DO 3 3 5 K = 1 , N A C PU = 0 . DO 3 3 3 J = 1 , N S T P U = P U + P ( I » K , J ) * U ( J ) 3 3 3 C O N T I N U E B U { I , K ) = D ( I , K ) + D I S C R * P U - U U ) 3 3 5 CONTINUE 3 4 0 CONTINUE KN = 0 DO 3 8 0 1 = 1 , N S T I I P = I I P O L I I i O B V = B V U , I I P ) O B U = B U U , I I P J DO 3 7 0 K = 1 , N A C I F ( K . E Q . I I P ) GO TO 3 7 0 B V I K = B V U , K ) B U I K = B U ( I , K J I F I B U I K . L E . O B U ) GO TO 3 7 0 I F ( B V I K . G T . Q B V ) GO TO 3 6 0 KN=KN+1 T A L P H = I - B V I K + O B V ) / ( B U I K - O B U ) I F I K N . E Q . l ) GO TO 3 5 0 ADEPS = DALPH+-EPSLN S B E P S = D A L P H - £ P S L N I F { T A L P H . G E . A D E P S ) GO TO 3 7 0 I F I T A L P H . L E . S 8 E P S ) GO TO 3 5 0 I F ( B U I K . L E . B U I ( I ) ) GO TO 3 7 0 K K P O L ( I ) = K B U I ( I ) = B U I K GO TO 3 7 0 3 5 0 OALPH= T A L P H DO 3 5 7 J = l , N S T I F U . E Q . l ) GO TO 3 5 5 K K P O U J ) = I I P O L l J ) BU I I J ) = 0 . GO TO 3 5 7 3 5 5 K K P O L ( J ) = K B U I { J ) = B U I K 3 5 7 CONTINUE GO TO 3 7 0 3 6 0 W R I T E ! 6 , 3 6 5 ) I , K , B V I K , B U I K , O B V , O B U 3 6 5 F 0 R M A T ( 5 X , • B E T T E R P O L I C Y EX I S T « , 5 X , 2 1 5 , 4 E 1 6 . 7 ) 3 7 0 CONTINUE 3 8 0 C O N T I N U E 3 8 5 I F ( K N . E Q . O ) GO TO 6 0 0 AL PHA=ALPHA+DALPH DO 3 9 3 1 = 1 , N S T I F { K K P O L { I ) . E Q . 1 1 P O L ( I ) ) GO TO 3 9 3 W R I T E ( 6 , 3 9 0 ) A L P H A , I , K K P O L ( I ) 3 9 0 F O R M A T ( 5 X , ' A L P H A = » , F 2 0 . 1 4 , 5 X , • I = , , I 5 , 5 X , , K = « , I 5 ) 3 9 3 CONTINUE I F ! A L P H A . G E . A L P M X ) GO TO 7 0 0 I F { A L P H A . L T . A L P M N ) GO TO 7 0 0 3 9 9 DO 9 5 0 K = 1 , N S T I F I K K P O L ( K ) . E Q . I I P O L I K ) ) GO TO 9 0 0 I F R S T = K K F R S T = K K P G L I K ) DO 4 1 0 J = l , N S T I F ( J . E Q . I F R S T ) GO TO 4 0 0 R V t J ) = - D I S C R * P < I F R S T , K F R S T » J} GO TO 4 1 0 4 0 0 R V ( J ) = 1 . - D I S C R * P I I F R S T , K F R S T » 4 1 0 CONTINUE DO 4 3 0 1 = 1 , N S T P U = 0 . DO 4 2 0 J = 1 , N S T P U = P U + R V ( J ) * T ( J , I ) 4 2 0 CONTINUE DU( I ) = P U 4 3 0 CONTINUE P U = D U ( I F R S T ) I F f P U . E Q . O . ) GO TO 6 2 0 DO 4 5 0 J = 1 , N S T IF i J , E Q . I F R S T ) GO TO 4 4 0 D U ( J ) = - D U ( J ) / P U GO TO 4 5 0 4 4 0 D U I J ) = 1 / P U 4 5 0 CONTINUE DO 4 8 0 1 = 1 , N S T P V = T ( I , I F R S T ) DO 4 7 0 J = l , N S T I F ( J . E Q . I F R S T ) GO TO 4 6 0 T I I , J ) = T ( I , J ) + P V * D U ( J ) GO TO 4 7 0 4 6 0 T I I , J J = P V * D U I J ) 4 7 0 CONTINUE 4 8 0 CONTINUE 9 0 0 I I P O L ( K ) = K K P O L I K ) 9 5 0 CONTINUE DO 4 9 0 1 = 1 , N S T VI I J =Vl I J t - D A L P H * U ( I ) PU = 0 . DO 4 8 5 J = l , N S T I I P = I I P O L ( J ) PU = P U + T ( I , J ) * 8 U ( J » I I P ) 4 8 5 CONTINUE U( I ) = U ( I ) +PU 4 9 0 CONTINUE ITR=0 5 0 0 DO 550 1 = 1 , N S T I I P = I I P O L U ) PU = 0 . DO 510 J = 1 , N S T PU = PU + P ( I , I I P , j ) * U ( J ) 5 1 0 CONTINUE DU,( I ) = D l I , I I P ) < - D I S C R * P U D 8 U = D U ( I ) - U I I ) I F U . E Q . l ) GO TO 5 4 0 I F ( D B U . G T . B U M X ) GO TO 5 2 0 I F ( D B U . L T . B U M N ) GO TO 5 3 0 GO TO 5 5 0 5 2 0 BUMX=DBU GO TO 5 5 0 5 3 0 BUMN=DBU GO TO 5 5 0 5 4 0 BUMX=D8U BUMN=D8U 5 5 0 CONTINUE UERR=8UMX-BUMN 133. IF I U E R R . L E . E R R ) GO TQ 5 7 0 DO 5 6 0 1 = 1 » N S T U(I)=DU•( I ) 5 6 0 CONTINUE I T R = I T R H GO TO 5 0 0 5 7 0 A D D = D I S C R * B U M N / ( 1 . - D I S C R ) DO 5 8 0 1 = 1 , N S T U( f ) = D U ( I ) + A D D 5 8 0 CONTINUE W R I T E ( 6 , 5 9 0 ) ITR 5 9 0 F O R M A T ( 5 X , * NUMBER OF I T E R A T I O N I N S - O P E R A T I G N FOX U I S » , 5 X , I 5 > DO 5 9 9 1 = 1 , N S T DO 5 9 7 K = 1 , N A C 8 V ( I » K ) = B V ( I , K ) + D A L P H * B U ( I , K ) 5 9 7 CONTINUE 5 9 9 CONTINUE I T E R = I T E R * - 1 GO TO 3 3 0 6 0 0 W R I T E ( 6 , 6 1 0 ) 6 1 0 FORMAT L 5 X , ' O P T I M A L FOR ALL ALPHA GREATER THAN THE C J R R E N T V A L U E ' ) GO TO 7 0 0 6 2 0 WRITE ( 6 , 6 3 0 ) 6 3 0 FORMAT ( 5 X , ' I N V E R S I O N F A I L E D ' ) 7 0 0 C A L L T I M E ( 1 , 1 ) STOP END C * * * M) CODE FOR P A R A M E T R I C MDP WITH S E L E C T I O N RULE (Ci 134. C * * * RE A L * 8 T A L P H , D A L P H , A L PHA » A , B , T » V , D E T , C O N D , U » RV » D J , B t f » B J , P V , P U , 1 O B V , G B U , B V I K , B U I K , G O B U , B V M X , B V M N » BUMX,3UMN » R B V , D 3 J , D J M X , P D U M X , 2 D U M A X , E X A L P , S A L P H , A D £ P S , S B E P S D I M E N S I O N V ( 5 0 ) , R I 5 0 , 5 0 ) , P ( 5 0 , 5 3 , 5 0 ) , 11 P O L { 5 0 ) , 0 1 5 0 , 5 0 ) , A 1 5 Q . 5 0 ) , I T ( 5 0 , 5 0 ) , B ! 5 0 , 1 ) , I P E R M ( 1 0 0 ) , K K P O L I 5 0 ) , UM 5 0 ) , R V 1 5 3 ) , 0 J { 5 0 ) , 2 B V ( 5 0 , 5 0 ) , B U ( 5 0 , 5 0 ) , B U I ( 5 0 ) REWIND8 REWIND9 E R R = 0 . 0 0 1 E P S L N = 0 . 0 0 0 0 0 1 NDIMA=50 N D I M B X = 5 0 NDIMT=50 NS0L=1 R E A D ( 5,l) N S T , N A C , D I S C R , A L P M N , A L P M X 1 F 0 R M A T ( 2 I 5 , 3 F 1 0 . 5 ) AL PHA=ALPMN R E A D ! 8 ) ( ( R ( . I , K ) , D ( I , K ) , 1 = 1 , N S T ) , K= 1 , N A C ) I T E R = 0 I T R = 0 DO 2 1 = 1 , N S T RE AD (9 ) ( { P { I , K , J ) , K = 1 , N A C ) , J=l , i M S T ) 2 CONTINUE DO 5 0 1 = 1 , N S T DO 20 K = 1 , N A C R D I K = R { I , K ) > A L P H A * D ( I , K ) I F ( K . E Q . l ) GO TO 10 I F I O R I . G E . R D I K ) GO TO 20 10 O R I = R D I K I I P=K 2 0 CONTINUE B ( I , 1 ) = O R I K K P O L { I ) = I I P DO 4 0 J = 1 , N S T I F I I . E O . J ) GO TO 3 0 A l I , J ) = - D I S C R * P { I , I I P , J ) GO TO 4 0 3 0 A l I , J ) = 1 . - D I S C R * P ( I , 1 I P , J ) 4 0 CONTINUE 5 0 CONTINUE GO TO 1 3 0 6 0 I T E R = I T E R +1 DO 120 1 = 1 , N S T DO 110 K = 1 , N A C P V = 0 . DO 100 J = 1 , N S T P V = P V + P ( I , K , J ) * V ( J ) 1 0 0 CONTINUE R P V I K = R ( I , K ) + A L P H A * D { I , K ) * D I S C R * P V I F ( K . E Q . l ) GO TO 105 I F ( R V I . G E . R P V I K ) GO TO 110 1 0 5 R V I = R P V I K I I P=K 1 1 0 CONTINUE DO 115 J = i , N S T I F U . E Q . J ) GO TO 1 1 3 A U , J ) = - D I S C R * P ( I , I I P , J ) GO TO 1 1 5 1 1 3 A ( I , J ) = 1 . - D I S C R * P ( I , I I P , J ) 1 1 5 CONTINUE 135. B( I , 1 ) = R ( I , 1 1 P ) + A L P H A * D { I , 1 1 P ) 11 P O L ( I 1 = 1 I P 1 2 0 CONTINUE 0 0 125 1 = 1 , N S T I F ( I I P Q L t I ) . N E . K K P O L i I ) ) GO TQ 1 2 7 1 2 5 CONTINUE GO TO 1 9 0 1 2 7 DO 129 1 = 1 , N S T K K P O L ! I ) = 11 POL ( I ) 1 2 9 CONTINUE O * * SOLVE THE SYSTEM OF EQUATIONS AV=3 BY C A L L I N G SUBROUTINE S L E 130 C A L L S L E ( N S T , N D I M A , A , N S 0 L , ! M D I M B X , 3 , V , I P E R M , N O I M T , T , 3 E T , J E X P ) I F I D E T ) 1 3 5 , 1 7 5 , 1 3 5 135 GO TO 6 0 1 7 5 W R I T E ( 6 , 1 8 0 ) 180 F O R M A T ! 5 X , ' S O L U T I O N F A I L E D ' ) GO TO 7 0 0 1 9 0 DO 2 0 0 1 = 1 , N S T I I P = I I P G L ( I ) P V = 0 . DO 192 J = 1 , N S T PV = PV + P ! I , I I P , J ) * V ( J ) 192 CONTINUE R V ( I ) = R ( I , I I P ) + A L P H A * D ( 1 , 1 1 P ) + D I S C R * P V R B V = R V ! I ) - V ( I ) I F I I . E Q . l ' ) GO TO 1 9 8 I F ( R 8 V . G T . B V M X ) GO TO 1 9 4 I F ! R B V . L T . B V M N ) GO TO 196 GO TO 2 0 0 1 9 4 BVMX=RBV GO TO 2 0 0 196 BVMN=RBV GO TO 2 0 0 198 BVMX=R3V BVMN=R8V 2 0 0 CONTINUE VERR=BVMX-BVMN I F I I T R . E Q . O . A N D . V E R R . L E . E R R I GO TO 2 0 6 A D D = D I S C R * B V M N / ! 1 . - 0 I S C R ) DO 202 1 = 1 , N S T V ! I ) = R V ! I ) + A D D 2 0 2 CONTINUE I F ( V E R R . L E . E R R ) GO TO 2 0 4 I T R = I T R + 1 GO TO 1 9 0 2 0 4 W R I T E I 6 , 2 0 5 ) I T R 2 0 5 F O R M A T ( 5 X , • N U M B E R OF I T E R A T I O N I N T - O P E R A TI ON FOX V I S « , 5 X , I 5 ) GO TO 6 0 2 0 6 W R I T E ( 6 , 2 0 5 ) I T R W R I T E I 6 , 2 0 8 ) A L P H A 2 0 8 F 0 R M A T 1 5 X , ' A L P H A = ' , F 2 0 . 1 4 ) A D D = S V M N / t 1 . - D I S C R ) DO 2 0 9 1 = 1 , N S T V I I ) = V ! I ) + A D D 2 0 9 CONTINUE W R I T E ( 6 , 2 1 0 ) 2 1 0 FORMAT! 1 0 X , * S T A T E ' , 1 0 X , ' P O L I C Y ' , 1 5 X , ' R E T U R N * , 1 0 X , ' O P T . P O L I C V " ) W R I T E ( 6 , 2 2 0 ) U , N A C , V ( I ) , 1 1 P O L ( I ) , 1 = 1 , N S T ) 2 20 F 0 R M A T I 1 0 X , I 3 , 1 2 X , I 3 , 1 O X , E 1 6 . 7 , 1 O X , I 3 / ) W R I T E ! 6 , 2 3 0 ) 1 3 6 -2 3 0 F 0 R M A T J 1 0 X , ' D I S C R * , 1 0 X , * I T E R A T I O N * ) W R I T E ( 6 , 2 4 0 ) D I S C R , I T E R 2 4 0 F O R M A T ( 1 0 X , F 7 . 5 , 6 X , 1 5 ) W R I T E ( 6 , 2 4 2 ) 2 4 2 F 0 R M A T ( / / / 1 X , O P T I M A L A C T I O N ! S ) TO BE CHOSEN FOR THE CURRENT 1 0 P T I M A L P O L I C Y * / / / ) C * * * F I N D THE I N V E R S E OF MATRIX A BY C A L L I N G SUBROUTINE INV C A L L I N V ( N S T , N D I M A , A , I P E R M , N D I M T , T , D E T , J E X P , C O N D ) I F ( D E T ) 2 4 5 , 6 2 0 , 2 4 5 2 4 5 DO 2 5 0 1 = 1 , N S T T D = 0 . DO 247 J = 1 , N S T I I P = I I P O L ( J ) T D = T D + T ( I , J ) * D ( J , I I P ) 2 4 7 CONTINUE U ( I ) = T D 2 5 0 CONTINUE I T R = 0 2 6 0 DO 3 0 0 1 = 1 , N S T 1 1 P = I I P O L ( I ) P U = 0 . DO 2 6 5 J = 1 , N S T P U = P U + P 1 1 , 1 I P , J ) * U ( J ) 2 6 5 CONTINUE D U I I ) = D ( I , I I P ) + D I S C R * P U D B U = D U ( I ) - U ( I ) I F ( I . E Q . l ) GO TO 2 9 0 I F ( D B U . G T . B U M X ) GO TO 2 7 0 I F ( D B U . L T . B U M N ) GO TO 2 8 0 GO TO 3 0 0 2 7 0 BUMX=DBU GO TO 3 0 0 2 8 0 BUMN=DBU GO TO 3 0 0 2 9 0 BUMX=DBU BUMN=DBU 3 0 0 CONTINUE UERR=BUMX-BUMN I F ( U E R R . L E . E R R ) GO TO 3 1 0 DO 305 1 = 1 , N S T UI I)=DU( I ) 3 0 5 CONTINUE I T R = I T R + l GO TO 2 6 0 3 1 0 A D D = D I S C R * B U M N / ( 1 . - D I S C R ) DO 3 2 0 1 = 1 , N S T UI I ) = D U ( I ) + ADD 3 2 0 CONTINUE W R I T E ( 6 , 5 9 0 ) ITR DO 3 2 5 1 = 1 , N S T DO 3 2 3 K = 1 , N A C P V = 0 . DO 321 J = 1 , N S T P V = P V + P ! I , K , J ) * V ( J ) 3 2 1 CONTINUE B V ( I , K ) = R ( I , K ) * A L P H A * D ( I , K ) + D I S C R * P V - V I I ) 3 2 3 CONTINUE 3 2 5 CONTINUE C A L L T I M E ( O ) ITER=1 137. 3 3 0 DO 3 4 0 1 = 1 , N S T DO 3 3 5 K = 1 , N A C P U = 0 . DO 333 J = 1 , N S T P U = P U + P ( I , K , J ) * U ( J ) 3 3 3 CONTINUE BUI I . , K ) = D ( I , K ) + D I S C R * P U - U ( I ) 3 3 5 CONTINUE 3 4 0 CONTINUE KN=0 DO 380 1 = 1 , N S T I I P = I I P O L ( I ) O B V = B V ( I , I I P ) O B U = B U I I , I I P ) DO 3 7 0 K = 1 , N A C I F 1 K . E Q . I I P J GO TO 3 7 0 B V I K = B V ( I , K ) B U I K = B U { I , K ) I F ( B U I K . L E . O B U ) GO TO 3 7 0 I F ( B V I K . G T . O B V ) GO TO 3 6 0 KN=KN+1 B U I K = B U I K - O B U TALPH= ( - B V 1 K + Q 8 V ) / BUI K I F ( K N . E Q . l ) GO TO 3 5 0 I F I D A L P H . L T . T A L P H ) GO TO 3 7 0 I F ( D A L P H . G T . T A L P H ) GO TO 3 5 0 I F ( BUI K . L E - . GOBU J GO TO 3 7 0 3 5 0 D A L P H = T A L P H I F R S T = I KFRST=K OOBU=BUIK GO TO 3 7 0 3 6 0 W R I T E ( 6 , 3 6 5 » I , K , B V I K , B U I K , 0 B V , 0 B U 3 6 5 F 0 R M A T 1 5 X , ' B E T T E R P O L I C Y EX I S T • , 5 X , 2 I 5 , 4 E 1 6 . 7 ) 3 7 0 CONTINUE 3 8 0 CONTINUE 3 8 5 I F ( K N . E Q . O ) GO TO 6 0 0 A L P H A = A L P H A * D A L P H 11 P O L ( I F R S T ) = K F R S T W R I T E ( 6 , 3 9 0 ) A L P H A , I F R S T , K F R S T 3 9 0 F 0 R M A T ( 5 X , ' A L P H A = • , F 2 0 . I 4 , 5 X , » I = ' , I 5 , 5 X , ' K = ' , I 5 ) I F ( A L P H A . G E . A L P M X ) GO TO 7 0 0 I F ( A L P H A . L T . A L P M N ) GO TO 7 0 0 DO 4 1 0 J = l , N S T I F ( J . E Q . I F R S T ) GO TO 4 0 0 R V ( J ) = - D I S C R * P ( I F R S T , K F R S T , J ) GO TO 4 1 0 4 0 0 R V ( J ) = 1 . - D I S C R * P ( I F R S T , K F R S T , J ) 4 1 0 CONTINUE DO 4 3 0 1 = 1 , N S T P U = 0 . DO 4 2 0 J = 1 , N S T P U = P U + R V ( J ) * T ( J , I ) 4 2 0 CONTINUE DIM I ) = PU 4 3 0 CONTINUE PU = D U ( I F R S T ) I F J P U . E Q . O . ) GO TO 6 2 0 DO 4 5 0 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 4 0 , D U ( J ) = - D U ( J ) / P U i M -GO TO 4 5 0 4 4 0 D U ( J ) = 1 / P U 4 5 0 CONTINUE DO 4 8 0 1 = 1 , N S T P V = T ( I , I F R S T ) DO 4 7 0 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 6 0 T( I , J ) = T ( I , J ) + P V * D U ( J ) GO TO 4 7 0 4 6 0 T( I , J ) = P V * D U ( J ) 4 7 0 CONTINUE 4 8 0 CONTINUE DO 4 9 0 1 = 1 , N S T V i I )=V ( I ) +-DALPH*U( I ) P U = 0 . DO 4 8 5 J = 1 , N S T I I P = I I P O L l J ) P U = P U + T ( I , J ) * B U ( J , I I P ) 4 8 5 CONTINUE U ( I ) = U ( I ) f P U 4 9 0 CONTINUE I T R = 0 5 0 0 DO 5 5 0 1 = 1 , N S T 1 1 P = 1 1 P O L ( I ) PU = 0 . DO 5 1 0 J = l , N S T P U = P U + P ( I , I I P , J ) * U ( J ) 5 1 0 CONTINUE D U ( I ) = D ( 1 , 1 I P ) + O I S C R * P U D B U = D U ( I ) - U ( I ) I F U . E Q . l ) GO TO 5 4 0 I F ( D B U . G T . B U M X ) GO TO 5 2 0 I F ( D B U . L T . B U M N ) GO TO 5 3 0 GO TO 5 5 0 5 2 0 BUMX=DBU GO TO 5 5 0 5 3 0 BUMN=DBU GO TO 5 5 0 5 4 0 BUMX=DBU BUMN=DBU 5 5 0 CONTINUE UE RR=BUMX-BUMN I F ( U E R R . L E . E R R 3 GO TO 5 7 0 DO 5 6 0 1 = 1 , N S T U ( I ) = O U U ) 5 6 0 C O N T I N U E I T R = I T R + 1 GO TO 5 0 0 5 7 0 A D D = D I S C R * B U M N / ( l . - D I S C R ) DO 5 8 0 1 = 1 , N S T U ( I ) = D U ( I ) + A D D 5 8 0 CONTINUE W R I T E ( 6 , 5 9 0 ) ITR 5 9 0 FORMAT ( 5 X , • NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FQ-l U I S » , 5 X , I 5 ) DO 5 9 9 1 = 1 , N S T DO 597 K = 1 , N A C B V U f K ) = B V ( I , K ) * D A L P H * 3 U ( I , K ) 5 9 7 CONTINUE 5 9 9 CONTINUE I T E R = I T E R + 1 GO TO 3 3 0 6 0 0 W R I T E ( 6 , 6 1 0 ) 6 1 0 FORMAT( 5 X , ' O P T I M A L FOR A L L ALPHA GO TO 7 0 0 6 2 0 WRITE ( 6 , 6 3 0 ) 6 3 0 FORMAT ( 5 X , ' I N V E R S I O N F A I L E D * ) 7 0 0 C A L L T I M E I 1 , 1 ) STOP END 139. GREATER THAN THE CURRENT V A L U E * ) C * * * N) CODE FOR P A R A M E T R I C MDP WITH PROCEDURE ( I - l ) C*** 140. RE A L * 8 T A L P H . D A L P H , A L P H A , A , B , T , V , D E T , C O N D , U , R V , D J , B V , B U , P V , P U , L OB V , OBU , BV I K , 3U I K , OOBU , B VMX , B VM M , B'J MX , BUM M , RBV , D-3 J , 3 J M X , PDUMX , 2 D U M A X , E X A L P , S A L P H , A D E P S , S B E P S D I M E N S I O N V ( 5 0 ) , R ( 5 0 , 5 0 ) , P ( 5 0 , 5 0 , 5 0 ) , I I P Q L { 5 0 } , D ( 5 0 , 5 Q ) , A { 5 0 , 5 0 ) , L T t 5 0 , 5 0 ) , B ( 5 0 , 1 ) , I P E R M ! 1 0 0 ) , K K P O L ( 5 0 ) , U ( 5 0 ) , R V ( 5 3 ) , 0 U ( 5 0 ) , 2 B V ( 5 0 , 5 0 ) , B U ( 5 0 , 5 0 ) , F L A G ( 5 0 , 5 0 ) , 3 U I { 5 0 ) REWIND8 REWIND9 E R R = 0 . 0 0 l ' E P S L N = 0 . 0 0 0 0 0 1 NDIMA=50 N D I M B X = 5 0 NDIMT=50 NS0L=1 RE A D ( 5 , 1 ) N S T » N A C , D I S C R » A L P M N » A L P M X 1 F O R M A T ( 2 1 5 , 3 F 1 0 . 5 ) ALPHA=ALPMN EX ALP=ALPMX T N A C = N S T * N A C E L M = 0 . A P R N T = 0 . RE AD( 8 ) ( ( R ( I , K ) , D ( I , !< ) ,1=1 , N S T ) , K = i , N A C ) I T E R = 0 ITR=0 DO 2 1 = 1 , N S T RE A D ( 9 ) ( ( P ( I , K , J ) , K = 1 , N A C ) » J = 1 , N S T ) 2 CONTINUE DO 5 0 1 = 1 , N S T DO 20 K = l , N A C F L A G ( I , K ) = - 1 R D I K = R ( I , K ) + A L P H A * D ( I , K ) I F ( K . E Q . l ) GO TO 10 I F ( O R I . G E . R D I K ) GO TO 2 0 10 O R I = R D I K 11 P=K 2 0 CONTINUE B ( I , 1 ) = 0 R I K K P O H I ) = I I P DO 40 J = 1 , N S T I F { I . E Q . J ) GO TO 3 0 A ( I , J ) = - D I S C R * P ( I , I I P , J ) GO TO 4 0 3 0 A( I , J ) = l . - D I S C R * P ( I , M P , J ) 4 0 CONTINUE 5 0 CONTINUE GO TO 1 3 0 6 0 I T E R = I T E R * 1 DO 120 1 = 1 , N S T DO 110 K = l , N A C P V = 0 . DO 100 J = i , N S T P V = P V + P I I , K , J ) * V ( J ) 1 0 0 CONTINUE R P V I K = R ( I , K ) + A L P H A * D ( I , K ) +DI S C R * P V I F I K . E Q . 1 ) GO TO 1 0 5 I F I R V I . G E . R P V I K ) GO TO 110 1 0 5 R V I = R P V I K 11 P=K 110 CONTINUE DO 115 J = 1 , N S T I F I I . E Q . J ) GO TO 1 1 3 A! I , J ) = - D . I S C R * P U , I I P , J ) GO TO 115 1 1 3 A ! I , J ) = 1 . - D I S C R * P ! I ,IIP, J ) 115 CONTINUE B ( I , 1 ) = R ( 1 , 1 1 P ) + A L P H A * D i I , 11P ) 11 P O L ! I ) = I I P 1 2 0 CONTINUE DO 125 1 = 1 , N S T I F I I I P O L ( I ) . N E . K K P G L I I ) ) GO TO 1 2 7 1 2 5 C O N T I N U E GO TO 1 9 0 1 2 7 DO 129 1 = 1 , N S T K K P O L I I ) = 11 P O L ( I ) 1 2 9 CONTINUE C * * * S O L V E THE SYSTEM OF EQUATIONS AV=B BY C A L L I N G SUBROUTINE S L E 1 3 0 C A L L S L E ! N S T » N D I M A , A » N S 0 L » N D I M B X » 8 , V , I P E R M , N D I M T , T , 'JET, J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 GO TO 6 0 1 7 5 W R I T E ( 6 , 1 8 0 ) 180 F O R M A T ! 5 X , ' S O L U T I O N F A I L E D ' ) GO TO 7 0 0 1 9 0 DO 2 0 0 1 = 1 , N S T I I P = I I P O L { I ) PV = 0 . DO 192 J = 1 , N S T PV = PV + P i I , I I P , J ) * : V U ) 1 9 2 CONTINUE RV I H = R ( I » I I P ) + A L P H A * D ( I , I I P ) + 0 1 5 C R * P V RB V=RV ( I ) - V I I ) I F U . E Q . l ) GO TO 1 9 8 I F C R B V . G T . B V M X ) GO TO 1 9 4 I F I R B V . L T . B V M N ) GO TO 1 9 6 GO TO 2 0 0 1 9 4 8VMX=RBV GO TO 2 0 0 1 9 6 BVMN=RBV GO TO 2 0 0 198 BVMX=RBV BVMN=RBV 2 0 0 CONTINUE VERR=BVMX-BVMN I F ( I T R . E Q . O . A N D . V E R R . L E . E R R ) GO TO 206 A D D = D I S C R * B V M N / { I . - D I S C R ) DO 2 0 2 1 = 1 , N S T VI I ) = R V ( I ) + A D D 2 0 2 CONTINUE I F ( V E R R . L E . E R R ) GO TO 2 0 4 I T R = I T R + 1 GO TO 1 9 0 2 0 4 . W R I T E ! 6 , 2 0 5 ) I T R 2 0 5 F O R M A T ! 5 X , ' N U M B E R OF I T E R A T I O N I N T - O P E R A T I ON FOX V I S « , 5 X , I 5 ) GO TO 6 0 2 0 6 W R I T E ( 6 , 2 0 5 ) I T R W R I T E ! 6 , 2 0 8 ) A L P H A 2 0 8 F O R M A T ( 5 X , ' A L P H A = ' , F 2 0 . 1 4 ) A D D = B V M N / ( 1 . - D I S C R ) DO 2 0 9 1 = 1 , N S T V ( I ) = V (I)+ A D D 2 0 9 CONTINUE 142. W R I T E I 6 , 2 1 0 ) 2 1 0 F O R M A T { 1 O X , * S T A T E ' , 1 0 X , • P O L I C Y * , 1 5 X , ' R E T U R N • , 1 O X , ' 3 ? T . P O L I C Y ' ) WRITE ( 6 , 2 2 0 ) ( I , N A C , V ( I ) , 11 POL U ) , 1 = 1 , MS T) 2 2 0 F O R M A T ! 1 O X , I 3 , 1 2 X , 1 3 , 1 O X , E 1 6 . 7 , 1 O X , I 3 / ) W R I T E ( 6 , 2 3 0 ) 2 3 0 F O R M A T ( 1 O X , ' D I S C R ' , 1 0 X , • I T E R A T I O N ' ) W R I T E ( 6 , 2 4 0 ) D I S C R , I T E R 2 4 0 F O R M A T ! 1 0 X , F 7 .5 , 6 X , 1 5 ) W R I T E ( 6 , 2 4 2 ) 2 4 2 F O R M A T ( / / / i X , ' # * * OPTIMAL A C T I O N ! S ) TO BE CHOSEN FOX THE CURRENT 1 0 P T I M A L P O L I C Y ' / / / ) C * * * F I N D THE I N V E R S E OF M A T R I X A BY C A L L I N G SUBROUTINE INV C A L L I N V ( N S T , N D I M A , A , I P E R M , N D I M T , T , D E T , J E X P , C O N D ) I F ! D E T ) 2 4 5 , 6 2 0 , 2 4 5 2 4 5 DO 2 5 0 1 = 1 , N S T T D = 0 . DO 247 J = 1 , N S T 1 1 P = I I POL I J ) T D = T D + T ! I , J ) * D ( J , 1 1 P ) 2 4 7 CONTINUE U I I ) = T D 2 5 0 C O N T I N U E ITR=0 2 6 0 DO 3 0 0 1 = 1 , N S T I I P = I I P O L ! I ) P U = 0 . DO 2 6 5 J = 1 , N S T P U = P U + P I I , I I P , J ) * U ( J ) 2 6 5 C O N T I N U E D U ( I ) = D ( I , I I P ) + D I S C R * P U D B U = D U I I ) - U ( I ) I F I I . E Q . l ) GO TO 2 9 0 I F ( D B U . G T . B U M X ) GO TO 2 7 0 I F ( D 8 U . L T . B U M N ) GO TO 2 8 0 GO TO 3 0 0 2 7 0 BUMX=DBU GO TO 3 0 0 2 8 0 BUMN=DBU GO TO 3 0 0 2 9 0 BUMX=DBU BUMN=DBU 3 0 0 CONTINUE ' UERR=BUMX-BUMN I F ( U E R R . L E . E R R ) GO TO 3 1 0 DO 3 0 5 1 = 1 , N S T U ! I ) = D U I I ) 3 0 5 CONTINUE I T R = I T R + 1 GO TO 2 6 0 3 1 0 A D D = O I S C R * B U M N / I 1 . - D I S C R ) DQ 3 2 0 1 = 1 , N S T U( I ) = D U ( I ) + A D D 3 2 0 CONTINUE W R I T E ( 6 , 5 9 0 ) ITR DO 3 2 5 1 = 1 , N S T DO 3 2 3 K = 1 , N A C PV = 0 . DO 3 2 1 J = 1 , N S T P V = P V * P U , K , J ) * V ( J ) 3 2 1 CONTINUE 143. BV ( I , K ) = R ( I , K) + A L P H A * D l I »K ) + - D I S C R * P V - V ( I ) 3 2 3 CONTINUE 3 2 5 CONTINUE C A L L T I M E I O J ITER=1 3 3 0 DO 3 4 0 1 = 1 , N S T B t I , 1 ) = U ( I ) DO 3 3 5 K = 1 , N A C P U = 0 . DO 3 3 3 J = 1 , N S T P U = P U + P ( I , K , J ) * U ( J ) 3 3 3 CONTINUE B U ( I , K I = D I I , K ) t - D I S C R * P U - U ( I i 3 3 5 CONTINUE 3 4 0 CONTINUE KN = 0 DO 3 8 0 1 = 1 , N S T i i p = i i P O L i n O B V = B V ( I , I I P ) O B U = B U ( I , I I P ) DO 370 K = 1 , N A C I F ( K . E Q . I I P ) GO TO 3 7 0 B V I K = B V ( I , K ) B U I K = B U ( I , K ) I F ( B U I K . L E . O B U ) GO TO 3 7 0 I F I B V I K . G T . 0 8 V ) GO TO 3 6 0 KN=KN+1 T A L P H={-BVIK4 - 0 B V ) / ( B U I K - O B U ) I F ( K N . E Q . l ) GO TO 3 5 0 A D E P S = D A L P H + E P S L N SB E P S = D A L P H - E P S L N I F C T A L P H . G E . A D E P S ) GO TO 3 7 0 I F ( T A L P H . L E . S B E P S ) GO TO 3 5 0 I F ( B U I K . L E . B U I ( I ) ) GO TO 3 7 0 K K P O L ( I ) = K B U I ( I ) = B U I K GO TO 3 7 0 3 5 0 D A L P H = T A L P H DO 3 5 7 J = 1 , N S T I F ( J . E Q . I ) GO TO 3 5 5 K K P O L ( J ) = 11 P O L ( J ) B U I ( J ) = 0 . GO TO 3 5 7 3 5 5 K K P O L ( J ) = K B U I ( J ) = B U I K 3 5 7 CONTINUE GO TO 3 7 0 3 6 0 W R I T E ( 6 , 3 6 5 ) I , K , B V I K , B U I K , O B V , O B U 3 6 5 F O R M A T ( 5 X , • B E T T E R P O L I C Y E X I S T * , 5 X , 2 I 5 , 4 E 1 6 . 7 ) 3 7 0 CONTINUE 3 8 0 C O N T I N U E 3 8 5 I F ( K N . E Q . O ) GO TG 6 0 0 ALPHA=ALPHA+DALPH DO 3 9 3 1 = 1 , N S T I F { K K P O L ( I ) . E Q . 1 1 P O L ( I ) ) GO TO 3 9 3 W R I T E ( 6 , 3 9 0 ) A L P H A , I , K K P O L ( I ) 3 9 0 F 0 R M A T ( 5 X , ' A L P H A = * , F 2 0 . 1 4 , 5 X , * I = ' , I 5 , 5 X , , K = « , I 5 ) 3 9 3 CONTINUE P R C N T = ( E L M / T N A C ) * 1 0 0 . W R I T E ( 6 , 3 9 4 ) PRCNT 3 9 4 F O R M A T ( 5 X , ' P R C N T * , 5 X , F 1 0 . 6 ) APRNT=APRNT +PRCNT EL M=0. I F ( A L P H A . G E . A L P M X ) GO TO 7 0 0 I F ( A L P H A . L T . A L P M N ) GO TO 7 0 0 3 9 9 DO 9 5 0 K = 1 , N S T IF I K K P O L I K ) . E Q . 11 P O L ( K ) ) GO TO I F R S T = K K F R S T = K K P O L ( K ) DO 4 1 0 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 0 0 RV ( J ) = - D I S C R * P ( I F R S T , K F R S T , J ) GO TO 4 1 0 4 0 0 R V ( J ) = 1 . - 0 1 S C R * P ( I F R S T , K F R S T , J ) 4 1 0 CONTINUE DO 4 3 0 1 = 1 , N S T PU = 0 . DO 4 2 0 J = 1 , N S T P U = P U + R V ( J ) * T ( J , I ) 4 2 0 CONTINUE D U ( I ) = PU 4 3 0 CONTINUE P U = D U ( I F R S T ) I F I P U . E Q . O . ) GO TO 6 2 0 DO 4 5 0 J = 1 , N S T I F l J . E Q . I F R S T ) GO TO 4 4 0 D U ( J ) = - D U ( J ) / P U GO TO 4 5 0 4 4 0 D U ( J ) = 1 / P U 4 5 0 C O N T I N U E DO 4 8 0 1 = 1 , N S T P V = T ( I , I F R S T ) DO 4 7 0 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 6 0 T( I , J ) = T ( I , J ) + P V * D U ( J ) GO TO 4 7 0 4 6 0 Tf I , J ) = P V * D U ( J ) 4 7 0 CONTINUE 4 8 0 CONTINUE 9 0 0 I I P O L ( K ) = K K P O L ( K ) 9 5 0 CONTINUE DO 4 9 0 1 = 1 , N S T VI I ) = V 1 I ) + D A L P H * U ( I ) P U = 0 . DO 4 8 5 J = l , N S T 1 1 P = I I P O L ( J ) P U = P U * T ( I , J ) * B U ( J , I I P ) 4 8 5 CONTINUE UI I ) = U ( I ) + P U 4 9 0 CONTINUE ITR=0 5 0 0 DO 5 5 0 1 = 1 , N S T I I P = I I P O L ( I ) PU = 0 . DO 510 J = 1 , N S T P U = P U + P ( I , I I P , J ) * U ( J ) 5 1 0 CONTINUE GUI I)=D1 11 I I P . ) + D I S C R * P U D B U = D U ( I ) - U ( I ) IF I I . E Q . l ) GO TO 540 145. I F ( D B U . G T . B U M X ) GO TO 520 I F I D B U . L T . B U M N ) GO TO 5 30 GO TO 5 5 0 5 2 0 BUMX=DBU GO TO 5 5 0 5 3 0 8UMN=DBU GO TO 5 5 0 5 4 0 BUMX=DBU BUMN=DBU 5 5 0 CONTINUE UERR=BUMX-BUMN I F ( U E R R . I E . E R R ) GO TO 5 7 0 DO 5 6 0 1 = 1 , N S T U l I ) = D U ( 1 ) 5 6 0 CONTINUE I T R = I T R U GO TO 5 0 0 5 7 0 A D O = D I S C R * B U M N / ( l . - D I S C R ) DO 5 8 0 1 = 1 , N S T U l I ) = D U ( I ) + ADD D U U ) = U 1 I ) - B 1 I , 1 ) I F U . E Q . l ) GO TO 5 7 5 I F l D U i I ) . L E . D U M X ) GO TO 580 5 7 5 D U M X = D U ( I ) 5 8 0 C O N T I N U E W R I T E 1 6 , 5 9 0 ) I T R 5 9 0 FORMAT 1 5 X , * NUMBER OF I T E R A T I O N I N S - 3 P E R A T I ON F O * U I S « , 5 X , I 5 3 DUMX=DISCR*DUMX DO 5 9 9 1 = 1 , N S T DO 5 9 7 K = 1 , N A C B V ( I , K ) = B V ( I , K ) + D A L P H * B U U , K ) BU( I , K ) = 8 U ( I , K ) + D U M X - D U 1 I ) 5 9 7 CONTINUE 5 9 9 CONTINUE IT ER= I T E R U 8 2 0 KN=0 DO 8 8 0 1 = 1 , N S T B C I , 1 ) = U ( I ) I I P = I I P O L ( I ) PU = 0 . DO 8 3 0 J = 1 , N S T P U = P U + P U , I I P , J ) * U l J ) 8 3 0 CONTINUE B U U , I I P ) = D 1 I , I I P ) * - D I S C R * P U - U ( I ) O B V - B V f I > I I P ) O B U = B U U , I I P ) D 8 U = 0 B V + 1 E X A L P - A L P H A ) * O B U DO 8 7 0 K = i , N A C I F 1 K . E Q . I I P ) GO TO 8 7 0 R B V = B V ( I , K ) * ( E X A L P - A L P H A ) * B U ( I , K ) IF ( F L A G 1 I , K ) . L E . O . ) GO TO 8 4 0 I F ( R B V . L T . D B U ) GO TO 8 5 3 F L A G l I , K ) =- 1 . P V = 0 . P U = 0 . DO 8 3 5 J = l , N S T P V = P V + P 1 I , K , J ) * V I J ) P U = P U + P ( I , K , J ) * U ( J ) 8 3 5 CONTINUE BV ( I » K ) =R ( I , K) +ALPHA*D { I , K) + DI S C R * P V - V ( I ) 1 4 6 . B U I I , K ) = D ( I , K ) + D I S C R * P U - U ( I ) GO TO 8 5 5 8 4 0 I F I R B V . L T . D B U ) GO TO 8 5 0 P U = 0 . DO 845 J = 1 , N S T PU = PU + P U , K , J ) * U ( J ) 8 4 5 CONTINUE B U ( I , K ) = D ( I , K ) + D I S C R * P U - U { I ) GO TO 8 5 5 8 5 0 F L A G U , K ) = 1 . 8 5 3 ELM=ELM+1. GO TO 8 7 0 8 5 5 B V I K = B V U , K ) 8 U I K = B U < I , K ) IF I B U I K . L E . O B U ) GO TO 8 7 0 I F { B V I K . G T . O B V ) GO TO 8 6 0 KN=KN+1 T A L P H = ( - B V I K + O B V ) / { B U I K - O B U ) I F I K N . E Q . I ) GO TO 8 5 7 A D E P S = D A L P H « - E P S L N SB E P S = D A L P H - E P S L N I F I T A L P H . G E . A D E P S ) GO TO 8 7 0 I F ( T A L P H . L E • S B E P S ) GO TO 8 5 7 I F ( B U I K . L E . B U I U ) ) GO TO 8 7 0 K K P O L I I I = K B U I ( I ) = B U I K GO TO 8 7 0 8 5 7 DALPH= T A L P H DO, 8 5 9 J = l , N S T I F I J . E Q . I ) GO TO 8 5 8 KK P O L ( J ) = 11 P O L ( J ) B U I U > = 0 . GO TO 8 5 9 8 5 8 K K P O L ( J ) = K BU I ( J ) = B U I K 8 5 9 C O N T I N U E GO TO 8 7 0 8 6 0 W R I T E I 6 . 8 6 5 ) I , K , B V I K , B U I K , O B V , O B U 8 6 5 F O R M A T ! 5 X , ' B E T T E R P O L I C Y E X I S T * » 5 X , 2 I 5 » 4 E L 6 . 7 ) 8 7 0 CONTINUE 8 8 0 CONTINUE GO TO 3 8 5 6 0 0 W R I T E < 6 , 6 1 0 ) 6 1 0 F Q R M A T ( 5 X , ' O P T I M A L FOR ALL ALPHA GREATER THAN THE C J R R E N T V A L U E ' ) GO TO 7 0 0 6 2 0 WRITE 1 6 , 6 3 0 ) 6 3 0 FORMAT { 5 X , * I N V E R S I O N F A I L E D ' ) 7 0 0 C A L L T I M E I l , ! ) P V = I T E R A P R N T = A P R N T / P V W R I T E ( 6 , 7 5 0 ) APRNT 7 5 0 F O R M A T ( 5 X , * AVERAGE PRCNT = ' , 5 X , F 1 0 . 6 ) STOP END C * * * 0 ) CODE FOR P A R A M E T R I C MDP WITH PROCEDURE ( 1 - 2 ) 147. C * * * R E A L * 8 T A L P H , D A L P H , A L P H A , A , B , T , V , D E T , C O N D , U , R V , D J , 8 V , 3 J , P V , P U , 1 0 B V , O B U , B V I K , B U I K , OOBJ , BVMX , BVM VI, B J M X , 3 U M ^ , R B V , 0 3 J , 3 J M X , P D U M X , 2 D U M A X , E X A L P , S A L P H , A D E P S , S B E P S D I M E N S I O N V ( 5 0 ) , R { 5 0 , 5 0 ) , P ( 5 0 , 5 0 , 5 0 ) , 11 POL ( 5 0 ) ,I3( 5 0 , 5 0 ) , A ( 5 0 , 5 0 ) , U l 5 0 , 5 0 ) , B ( 5 0 , 1 ) , I P E R M ! 1 0 0 ) , KKPOL ( 5 0 ) , ' J I 5 0 ) , R V I 5 3 ) , 3 J { 50 ) , 2 B V ( 5 0 , 5 0 ) , B U I 5 0 , 5 0 3 , F L A G ( 5 0 , 5 0) , D U M X ( 5 0 ) , P D U M X ( 5 3 ) , 3 J I ( 5 0 ) REWINDS REWIND9 E R R = 0 . 0 0 1 E P S L N = 0 . 0 0 0 0 0 1 NDIMA=5 0 NDIMBX=50 N D I M T = 5 0 N S O L = i RE A D ( 5 , 1 ) N S T , N A C , D I S C R , A L P M N , A L P M X 1 F O R M A T ! 2 1 5 , 3 F 1 0 . 5 ) AL PHA=ALPMN EX ALP=ALPMX T N A C = N S T * N A C E L M = 0 . A P R N T = 0 . RE AD ( 8 ) ( ( R ( I » K . ) , D ( I , K I , 1 = 1 , N S T ) , K = 1 , N A C ) I T E R = 0 ITR=0 DO 2 1 = 1 , N S T RE AD ( 9 H ( P ( I , K , J ) , K = I , N A C ) , J = 1 , N S T ) 2 CONTINUE DO 50 1 = 1 , N S T DO 20 K = 1 , N A C F L A G l I , K ) = - 1 R D I K = R ( I , K ) + A L P H A * D ( I , K ) I F ( K . E Q . l ) GO TO 10 I F ( O R I . G E . R D I K ) GO TO 20 10 O R I = R D I K I I P=K 2 0 C O N T I N U E B( 1 , 1 ) = ORI K K P O L I I 1 = 11P DO 40 J = 1 , N S T I F ( I . E Q . J ) GO TO 3 0 A( I , J ) = - D I S C R * P ( 1 , 1 I P , J ) GO TO 4 0 3 0 A ( I , J ) = i . - D I S C R * P ( I , I I P , J ) 4 0 CONTINUE 5 0 CONTINUE GO TO 1 3 0 6 0 I T E R = I T E R U DO 120 1 = 1 , N S T DO 110 K = 1 , N A C P V = 0 . DO 100 J = 1 , N S T P V = P V + P ( I , K , J ) * V ( J ) 1 0 0 C O N T I N U E R P V I K = R ( I , K ) + A L P H A * D ( I , K ) * - D I S C R * P V I F ( K . E Q . l ) GO TO 105 I F ( R V I . G E . R P V I K ) GO TO 110 1 0 5 R V I = R P V I K i r P = K 1 1 0 CONTINUE , z 0 0 115 J = 1 , N S T I F ( I . E Q . J ) GO TO 113 A ( I , J ) = - D I S C R * P ( I , I I P , J ) GO TO 115 1 1 3 A ( I f J ) = l . - D I S C R * P ( I »1 1 P , J ) 1 1 5 CONTINUE B ( 1 , 1 ) = R ( I , I I P ) * A L P H A * D ( I , I I P ) I I P O L ( I ) = I I P 1 2 0 CONTINUE DO 1 2 5 1 = 1 , N S T I F I I I P O L I I ) . N E . K K P O L ( I ) ) GO TO 1 2 7 1 2 5 CONTINUE GO TO 1 9 0 127 DO 129 1 = 1 , N S T K K P O L I I ) = 11 POL I I ) 129 CONTINUE C * * * S O L V E THE SYSTEM OF EQUATIONS AV=8 BY C A L L I N G SUBROUTINE S L E 130 C A L L S L E ( N S T » N D I M A » A , N S 0 L , N D I M B X » B » V » I P E R M , N D I M T , r , 3 E T » J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 GO TO 6 0 175 W R I T E ( 6 , 1 8 0 ) 1 8 0 F O R M A T ! 5 X , ' S O L U T I O N F A I L E D ' ) GO TO 7 0 0 1 9 0 DO 200 1 = 1 , N S T I I P = I I P Q L ( I ) P V = 0 . DO 192 J = 1 , N S T PV = PV + P ( I , I I P , J ) * V ( J ) 1 9 2 CONTINUE R V ( I ) = R ( I , I I P ) + A L P H A * D ( I , I I P ) + D I S C R * P V R B V = R V I I ) - V i I ) I F ( I . E Q . i ) GO TO 1 9 8 I F ( R B V . G T . B V M X ) GO TO 194 I F ( R 8 V . L T . B V M N ) GO TO 1 9 6 GO TO 2 0 0 1 9 4 8VMX=RBV GO TO 2 0 0 1 9 6 BVMN=RBV GO TO 2 0 0 198 BVMX=RBV BVMN=RBV 2 0 0 CONTINUE VERR=BVMX-BVMN I F ( I T R . E Q . O . A N D . V E R R . L E . E R R ) GO TO 2 0 6 A O D = D I S C R * B V M N / ( l . - D I S C R ) DO 2 0 2 1 = 1 , N S T VI I ) = R V U ) +ADD 2 0 2 CONTINUE I F ( V E R R . L E . E R R ) GO TO 2 0 4 I T R = I T R + 1 GO TO 1 9 0 2 0 4 W R I T E ( 6 , 2 0 5 ) ITR 2 0 5 F O R M A T ( 5 X , ' N U M B E R OF I T E R A T I O N I N T - O P E R A T I O N FO^ V I S ' , 5 X , I 5 ) GO TO 6 0 2 0 6 W R I T E ( 6 , 2 0 5 ) ITR W R I T E ( 6 , 2 0 8 ) A L P H A 2 0 8 F O R M A T ( 5 X , ' A L P H A = « , F 2 0 . 1 4 ) A D D = B V M N / ( l . - D I S C R ) 00 209 1 = 1 , N S T V( I ) = V { I J + ADO 2 0 9 CONTINUE 1 4 9 -W R I T E { 6 » 2 1 0 ) 2 1 0 FORMAT { 1 0 . X , 1 S T A T E ' , 1 0 X , ' P O L I C Y ' , 15X» ' R E T U R N ' , 1 0 X , « 3 ? T . P O L I C Y • ) W R I T E 1 6 , 2 2 0 ) ( I , N A C , V ( I ) , I I P O L ( I ) , I = 1 , N S T ) 2 2 0 F O R M A T ! 1 O X , 1 3 , 1 2 X , 1 3 , 1 0 X , E 1 6 . 7 , 1 O X , I 3 / ) W R I T E ( 6 , 2 3 0 ) 2 3 0 FO RM AT U O X , ' D I S C R ' , 1 0 X , ' I T E R A T I O N ' ) W R I T E I 6 , 2 4 0 ) D I S C R , I T E R 2 4 0 F O R M A T ( 1 0 X , F 7 . 5 , 6 X , I 5 ) W R I T E 1 6 . 2 4 2 ) 2 4 2 F O R M A T < / / / I X , » * * * O P T I M A L A C T I O N ! S ) T3 BE CHOSEN FO* THE CURRENT 1 0 P T I M A L P O L I C Y ' / / / ) C * * * F I N D THE I N V E R S E OF M A T R I X A BY C A L L I N G SUBROUTINE INV C A L L I N V I N S T , N D I M A , A , I P E R M , N D I M T , T , D E T , J E X P , C O N D ) I F ( D E T ) 2 4 5 , 6 2 0 , 2 4 5 2 4 5 DO 2 5 0 1 = 1 , N S T T D = 0 . DO 2 4 7 J = 1 , N S T I I P = I I P O L ! J ) T D = T D + T ( I , J ) * D { J , I I P ) 2 4 7 CONTINUE U ! I ) = TD 2 5 0 CONTINUE I T R = 0 2 6 0 DO 300 1 = 1 , N S T I I P = I I P O L ! I ) PU = 0 . DO 2 6 5 J = 1 , N S T PU=PU + P I I , 1 I P , J ) * U ! J ) 2 6 5 CONTINUE D U I I ) = D I I , I I P ) + D I S C R * P U D B U = D U ! I ) - U ! I ) I F I I . E Q . l ) GO TO 2 9 0 I F I D B U . G T . B U M X ) GO TO 2 7 0 I F ( D B U . L T . B U M N ) GO TO 2 8 0 GO TO 3 0 0 2 7 0 BUMX=DBU GO TO 3 0 0 2 8 0 BUMN=DBU GO TO 3 0 0 2 9 0 BUMX=DBU BUMN=DBU 3 0 0 CONTINUE UERR=BUMX-BUMN I F I U E R R . L E . E R R ) GO TO 3 1 0 DO 305 1 = 1 , N S T U ( I ) = D U ( I ) 3 0 5 CONTINUE I T R = I T R + 1 GO TO 2 6 0 3 1 0 A D D = D I S C R * B U M N / ( 1 . - D I S C R ) DO 3 2 0 1 = 1 , N S T U ! I ) = D U ! I ) + A D D 3 2 0 CONTINUE W R I T E ! 6 , 5 9 0 ) ITR DO 3 2 5 1 = 1 , N S T DO 3 2 3 K = 1 , N A C P V = 0 . DO 321 J = 1 , N S T P V = P V + P < I , K , J ) * V ( J ) 3 2 1 CONTINUE B V ( I , K ) = R ( J , K ) + A L P H A * D ( I , K ) + D I S C R * P V - V ( I ) 3 2 3 CONTINUE 3 2 5 CONTINUE C A L L T I M E ( O ) ITER=1 3 3 0 DO 3 4 0 1 = 1 , N S T A ( I , I T E R ) = U ( I ) DO 3 3 5 K = 1 , N A C PU = 0 . DO 3 3 3 J = 1 , N S T P U = P U + P ( I , K , J ) * U ( J ) 3 3 3 CONTINUE B U ( I , K ) = D ( I , K ) * D I S C R * P U - U ( I ) 3 3 5 CONTINUE 3 4 0 CONTINUE KN=0 DO 3 8 0 1 = 1 , N S T I I P = I I P O L ( I ) OB V = B V t I « 1 1 P I G B U = B U I I , I I P ) DO 3 7 0 K = 1 , N A C I M K . E Q . I I P 1 GO TO 3 7 0 B V I K = B V ( I , K ) B U I K = B U ( I , K ) I F ( B U I K . L E . O B U ) GO TO 3 7 0 I F I B V I K . G T . O B V ) GO TO 3 6 0 KN=KN+1 T A L P H = ( - B V I K + G 8 V ) / ( B U I K - O B U ) I F ( K N . E Q . l ) GO TO 3 5 0 A D £ P S = D A L P H + E P S L N S B E P S = D A L P H - E P S L N I F ( T A L P H . G E . A D E P S ) GO TO 3 7 0 I F I T A L P H . L E . S B E P S ) GO TO 3 5 0 I F ( B U I K . L E . B U I ( I ) ) GO TO 3 7 0 K K P O L ( I ) = K B U I U ) = B U I K GO TO 3 7 0 3 5 0 D A L P H = T A L P H DO 3 5 7 J = 1 , N S T I F ( J . E Q . I ) GO TO 3 5 5 K K P O L ( J ) = 1 1 P O L ( J ) BU I ( J ) = 0 . GO TO 3 5 7 3 5 5 K K P O L ( J ) = K B U I ( J ) = B U I K 3 5 7 CONTINUE GO TO 3 7 0 3 6 0 W R I T E ( 6 , 3 6 5 ) I , K , B V I K , B U I K , 0 B V , Q 8 U 3 6 5 F O R M A T ! 5 X , • B E T T E R P O L I C Y E X I S T ' , 5 X , 2 I 5 , 4 E 1 6 . 7 ) 3 7 0 CONTINUE 3 8 0 CONTINUE 3 8 5 I F ( K N . E Q . O ) GO FO 6 0 0 AL PHA=ALPHA + DALPH DO 3 9 3 1 = 1 , N S T I F ( K K P O L ( I ) . E Q . I I P O L ( I ) ) GO TO 3 9 3 W R I T E ( 6 , 3 9 0 ) A L P H A , I , K K P O L ( I ) 3 9 0 FORMAT I 5 X , ' A L P H A = • , F 2 0 . 1 4 , 5 X , • I = « , I 5 , 5 X , ' K = » , I 5 ) 3 9 3 CONTINUE P R C N T = ( E L M / T N A C ) * 1 0 0 . W R I T E I 6 , 3 9 4 ) PRCNT 3 9 4 F O R M A T ( 5 X , • P R C N T • , 5 X , F 1 0 . 6 ) APRNT = APRNT+-PRCNT E L M = 0 . I F ( A L P H A . G E . A L P M X J GO TO 7 0 0 I F < A L P H A . L T . A L P M N ) GO TO 7 0 0 3 9 9 DO 9 5 0 K = 1 , N S T I F I K K P O L ( K ) . E Q . I I P O L ( K ) ) GO TO I F R S T = K K F R S T = K K P O L ( K ) DO 4 1 0 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 0 0 RV1 J ) = - D I S C R * P 1 I F R S T , K F R S T , J ) GO TO 4 1 0 4 0 0 R V ( J ) = 1 . - D I S C R * P { I F R S T , K F R S T , J ) 4 1 0 CONTINUE DO 4 3 0 1 = 1 , N S T PU = 0 . DO 4 2 0 J = 1 , N S T P U = P U + R V ( J ) * T ( J » I ) 4 2 0 CONTINUE D U 1 I ) = P U 4 3 0 CONTINUE P U = D U I I F R S T ) I F I P U . E Q . O . ) GO TO 6 2 0 DO 4 5 0 J = 1 , N S T I F I J . E Q . I F R S T ) GO TO 4 4 0 D U { J ) = - D U ( J ) / P U GO TO 4 5 0 4 4 0 D U ( J ) = 1 / P U 4 5 0 CONTINUE DO 4 8 0 1 = 1 , N S T P V = T { I , I F R S T ) DO 4 7 0 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 6 0 T( I , J ) = T < I , J ) + P V * D U I J ) GO TO 4 7 0 4 6 0 T( I , J ) = P V * D U I J ) 4 7 0 CONTINUE 4 8 0 CONTINUE 9 0 0 I I P O L ( K ) = K K P O L ( K ) 9 5 0 CONTINUE 0 0 4 9 0 1 = 1 , N S T V i I ) = V < I ) + D A L P H * U ( I ) P U = 0 . DO 4 8 5 J = 1 , N S T I I P = I I P O L ( J ) P U = P U + T ( I , J ) * B U ( J , I I P ) 4 8 5 CONTINUE U l I ) = U ( I ) + P U 4 9 0 CONTINUE I T R = 0 5 0 0 DO 5 5 0 1 = 1 , N S T I I P = I I P O L < I ) PU = 0 . DO 510 J = 1 , N S T PU = P U + P ( I » 1 1 P , J ) * U ( J ) 5 1 0 CONTINUE D U ( I ) = D ( I , I I P ) + D I 5 C R * P U DB U=DUI I ) - U { I ) 1 5 I F I I . E Q . l ) GO TO 5 4 0 I F ( D B U . G T . B U M X ) GO TO 5 2 0 I F i D B U . L T . B U M N ) GO TO 5 3 0 GO TO 5 5 0 5 2 0 BUMX=DBU GO TO 5 5 0 5 3 0 BUMN=DBU GO TO 5 5 0 5 4 0 BUMX=DBU BUMN=DBU 5 5 0 CONTINUE UERR=BUMX-BUMN I F ( U E R R . L E . E R R ) GO TO 5 7 0 DO 560 1 = 1 , N S T U i I ) = D U I I ) 5 6 0 CONTINUE I T R = I T R + 1 GO TO 5 0 0 5 7 0 A D D = D I S C R * B U M N / I 1 . - D I S C R ) DO 5 8 0 1 = 1 , N S T U i I ) =DUI I)+ADD 5 8 0 CONTINUE W R I T E ( 6 , 5 9 0 ) I T R 5 9 0 F Q R M A H 5 X , 1 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOX U I S ' , 5 X , I 5 ) DO 5 9 4 K = l , I T E R DO 5 9 2 1 = 1 , N S T D U I K = U I I ) - A ( I , K ) I F I K . E Q . I T E R ) D U ( I ) = D U I K I F ( I . E Q . l ) GO TO 5 9 1 I F I D U M A X . G E . D U I K ) GO TO 5 9 2 5 9 1 DUMAX=DUIK 5 9 2 CONTINUE I F I I T E R . E Q . l ) GO TO 5 9 3 I F I K . E Q . I T E R ) GO TO 5 9 3 P D U M X ( K ) = D U M X ! K ) 5 9 3 D U M X I K ) = D I S C R * D U M A X 5 9 4 CONTINUE DO 5 9 9 1 = 1 , N S T DO 597 K = 1 , N A C BV I I , K ) = B V ( I , K ) + D A L P H * B U ( I , K ) L = FLAG i I , K ) I F I L . L E . O . O R . L . E Q . I T E R ) GO TO 5 9 5 B U M , K ) = B U U , K ) + D U M X ( L ) - P D U M X i L ) - D U U ) GO TO 5 9 7 5 9 5 B U i I , K i = B U ( I , K I + D U M X ( I T E R ) - D U ( I ) 5 9 7 CONTINUE 5 9 9 CONTINUE I T E R = I T E R + 1 8 2 0 KN=0 DO 8 8 0 1 = 1 , N S T A ( I , I T E R ) = U ( I ) 1 1 P = I I P O L i 1 1 P U = 0 . DO 8 3 0 J = 1 , N S T PU = PU + P I I , 1 I P , J ) * U ( J ) 8 3 0 CONTINUE BUI I , I IP 1 = 0 ( I , I I P I * D . I S C R * P U - U ( I » OBV=BVI I , I I P ) 0 8 U = 8 U M , 11 P) D B U = O B V + ( E X A L P - A L P H A ) * Q B U 153 DD 8 7 0 K = 1 , N A C I F I K . E Q . I I P ) GO TO 8 7 0 RBV=BV( I , K ) + ( E X A L P - A L P H A ) * B U ( I , K ) I F ( F L A G l I , K ) . L E . O . > GO TO 8 4 0 I F ( R B V . L T . D B U ) GO TO 8 5 3 F L A G I I , K ) = - 1 . P V = 0 . P U = 0 . DO 8 3 5 J = 1 , N S T PV = PV + P ( I , K , J ) * V U ) PU=PU + P< I , K , J ) # U U ) 8 3 5 CONTINUE B V ( I f K ) = R ( I , K i + A L P H A * D ( I , K ) + D I S C R * P V - V U ) B U ( I , K ) = D < I , K ) + D I S C R * P U ~ U U ) GO TO 8 5 5 8 4 0 I F I R B V . L T . D B U ) GO TO 8 5 0 P U = 0 . DO 845 J = 1 , N S T P U = P U + P ( I t K , J ) * U ( J ) 8 4 5 C O N T I N U E BU I I , K ) = D ( I , K ) + D I S C R * P U - U ! 1 1 GO TO 8 5 5 8 5 0 F L A G ! I , K ) = I T E R - 1 8 5 3 E L M = E L M + 1 . GO TO 8 70 8 5 5 B V I K = B V ( I , K ) B U I K = B U ( I , K ) I F ( B U I K . L E . O B U ) GO TO 8 7 0 I F ( B V I K . G T . 0 8 V ) GO TO 8 6 0 KN=KN+1 T A L P H = ( - B V I K + G B V ) / ( 8 U I K - Q B U ) I F I K N . E Q . l ) GO TO 8 5 7 A D E P S = D A L P H + E P S L N S B E P S = D A L P H - E P S L N I F ( T A L P H . G E . A D E P S ) GO TO 8 7 0 IF I T A L P H . L E . S B E P S ) GO TO 8 5 7 I F { B U I K . L E . B U H I I ) GO TO 8 7 0 KKPOL { I ) = K B U I ( I ) = B U I K GO TO 8 7 0 8 5 7 DALPH=TALPH DO 859 J = 1 , N S T I F U . E Q . l ) GO TO 8 5 8 K K P O L U ) = I I P O L U ) BUI U ) = 0 . GO TO 8 5 9 8 5 8 K K P O L I J ) = K B U I U ) =BUIK 8 5 9 CONTINUE GO TO 8 7 0 8 6 0 W R I T E ( 6 , 8 6 5 ) I , K , B V I K , B U I K , O B V , Q B U 8 6 5 F O R M A T ( 5 X , » B E T T E R P O L I C Y EX I S T ' , 5 X , 2 1 5 , 4 E 1 6 . 7 ) 8 7 0 CONTINUE 8 8 0 CONTINUE GO TO 3 8 5 6 0 0 W R I T E { 6 , 6 1 0 ) 6 1 0 F O R M A T ( 5 X t * OPT IMAL FOR ALL ALPHA GREATER THAN THE I J R R E N T V A L U E ' ) GO TO 7 0 0 6 2 0 WRITE ( 6 , 6 3 0 ) 6 3 0 FORMAT i5X,•INVERSI ON F A I L E D ' ) 7 0 0 C A L L T I ME ( 1 , I ) . I b 4 -P V = I T E R AP RNT=APRNT/PV W R I T E ( 6 , 7 5 0 ) APRNT 7 5 0 F O R M A T { 5 X , • A V E R A G E PRCNT = » , 5 X , F 1 0 . 6 ) STOP END C * * * P ) CODE FOR P A R A M E T R I C MOP WITH PROCEDURE ( I I - 1 J 155 • C * * * R E A L * 8 T A L P H , D A L P H , A l P H A , A , B , T , V , D E T * C O N D , U , R V , D J , B V , 8 J , P V , P U , 1 0 B V , O B U , B V I K , B U I K , O O B U , B V M X , B V M N , B U M X , 3 U M N , R B V , Q 3 J ,3'JMX , PDUMX, 2 D U M A X , E X A L P , S A L P H , A D E P S , S B E P S D I M E N S I O N V ( 5 0 ) , R ( 5 0 , 5 0 ) , P { 5 0 , 5 0 , 5 0 ) , I I P O K 5 0 ) , D ( 5 9 , 5 0 ) , A { 5 0 , 5 3 3 , 1T( 5 0 , 5 0) , B ( 5 0 , 1 ) , I P E R M ! 1 0 0 ) , K K P 0 L < 5 0 ) , U ( 5 0 ) , R V ( 5 3 ) , 3 J ( 5 0 ) , 2 B V I 5 0 , 5 0 ) , B U ( 5 0 , 5 0 ) , F L A G ( 5 0 , 5 0 ) » B U I ( 5 0 ) REWIND8 REWIND9 E R R = 0 . 0 0 1 E P S L N = 0 . 0 0 0 0 0 1 NDIMA=50 N D I M B X = 5 0 NDIMT=50 N S 0 L = 1 RE A D ( 5 , 1 ) N S T , N A C , D I S C R , A L P M N , A L P M X 1 F O R M A T ( 2 1 5 , 3 F 1 0 * 5 ) AL PHA = ALPMN T N A C = N S T * N A C E L M = 0 . A P R N T = 0 . R E A D I 8 ) ( ( R I I , K 3 , D ( I , K ) , 1 = 1 , N S T ) , K = l , N A C ) ITER=0 I T R=0 DO 2 1 = 1 , N S T R E A D ( 9 ) I ( P ( I , K , J ) , K = i , N A C ) , J = 1 , N S T ) 2 CONTINUE DO 5 0 1 = 1 , N S T DO 20 K = 1 , N A C F L A G ( I , K ) = - l RD IK=R ( I , K ) + - A L P H A * D ( I , K ) I F ( K . E Q . l ) GO TO 10 I F ( O R I . G E . R D I K ) GO TO 2 0 10 O R I = R D I K I I P=K 20 CONTINUE B ( 1 , 1 ) = 0 R I K K P O L ( I ) = I I P DO 4 0 J = 1 , N S T I F U . E Q . J ) GO TO 3 0 A( I , J ) = - D I S C R * P U , I I P , J ) GO TO 4 0 3 0 A U , J ) = 1 . - D I S C R * P ( I , I I P , J ) 4 0 CONTINUE 5 0 CONTINUE GO TO 130 6 0 IT ER=ITER + 1 DO 120 1 = 1 , N S T DO 110 K = 1 , N A C PV = 0 . DO 100 J = 1 , N S T P V = P V + P ( I , K , J ) * V ( J ) 1 0 0 CONTINUE R P V I K = R M , K ) * - A L P H A * D ( I , K ) +DI S C R * P V I F l K . E Q . l ) GO TO 1 0 5 I F ( R V I . G E . R P V I K ) GO TO 1 1 0 1 0 5 R V I = R P V I K I I P=K 1 1 0 CONTINUE DO 115 J = 1 , N S T I F ( I . E Q . J ) GO TO 1 1 3 1 5 A ( I , J ) = - D I S C R * P I 1 , I I P , J ) GO TO 1 1 5 113 A ( I , J ) = l . - D I S C R * P I I , I I P , J 1 1 1 5 CONTINUE B! I , 1 ) = R ( I , I I P ) + A L P H A * D I I , I I P ! 11 P O L ! I ) = I I P 1 2 0 CONTINUE DO 1 2 5 1 = 1 , N S T I F ! I I P O L ! I ) . N E . K K P O L I I ) ) GO TO 1 2 7 1 2 5 CONTINUE GO TO 1 9 0 1 2 7 DO 129 1 = 1 , N S T K K P O L ! I J = I I P O L ( I ) 1 2 9 CONTINUE C * * * S O L V E THE SYSTEM OF EQUATIONS AV=B BY C A L L I N G SUBROUTINE S L E 130 C A L L S L E ! N S T » N D I M A , A » N S 0 L , N D I M B X , 8 , V » I P E R v l , N i ) I M T , T , J E T , J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 GO TO 6 0 1 7 5 W R I T E ! 6 , 1 8 0 ) 1 8 0 F 0 R M A T 1 5 X , ' S O L U T I O N F A I L E D ' ) GO TO 7 0 0 1 9 0 DO 2 0 0 1 = 1 , N S T I I P = I I P O L U ) P V = 0 . DO 192 J = 1 , N S T PV = P V + P ! 1 , 1 I P , J ) * V { J ) 1 9 2 CONTINUE R V ( I ) = R ! I , I I P ) + A L P H A * D { I , I I P ) + D I S C R * P V R B V = R V ( I ) - V ( I ) I F ( I . E Q . l ) GO TO 198 I F I R B V . G F . B V M X ) GO TO 1 9 4 I F ( R B V . L T . B V M N ) GO TO 196 GO TO 2 0 0 1 9 4 BVMX=RBV GO TO 2 0 0 1 9 6 BVMN=R8V GO TO 2 0 0 1 9 8 BVMX=RBV BVMN=RBV 2 0 0 CONTINUE VERR=BVMX-BVMN I F I I T R . E Q . O . A N D . V E R R . L E . E R R ) GO TO 2 0 6 A D D = D I S C R * B V M N / ( 1 . - D I S C R ) DO 2 0 2 1 = 1 , N S T VI I ) =RV I I)+ADD 2 0 2 CONTINUE I F I V E R R . L E . E R R ) GO TO 2 0 4 IT R= IT R + I GO TO 1 9 0 2 0 4 W R I T E I 6 , 2 0 5 ) ITR 2 0 5 F O R M A T ! 5 X , » N U M B E R OF I T E R A T I O N I N T - D P E R A T I O N FOX V I S ' , 5 X , I 5 ) GO TO 6 0 2 0 6 W R I T E ! 6 , 2 0 5 ) ITR W R I T E ( 6 , 2 0 8 ) A L P H A 2 0 8 F O R M A T ! 5 X , ' A L P H A = ' , F 2 0 . 1 4 ) A D D = B V M N / U . - D I S C R ) DO 209 1 = 1 , N S T V( I ) = V U ) + A D D 2 0 9 CONTINUE W R I T E ! 6 , 2 1 0 ) • 2 1 0 FORMAT U Q X , ' S T A T E ' , 10X , • POL I CY • , 1 5X , ' R ETU RN ' , 1 OX, * 3? T . P O L I C Y * J W R I T E ( 6 , 2 2 0 ) ( I , N A C , V ( I J , I I P O L ( I ) , 1 = 1 , N S T ) 2 20 F O R M A T ( 1 0 X , 1 3 , 1 2 X , I 3 , 1 O X , E 1 6 . 7 , 1 O X , 1 3 / ) W R I T E ! 6 , 2 3 0 ) 2 3 0 F O R M A T { 1 0 X , ' D I S C R ' , 1 0 X , * I T E R A T I ON * 3 W R I T E ! 6 , 2 4 0 ) D I S C R , I T E R 2 4 0 F O R M A T ! 1 0 X , F 7 . 5 , 6 X , 1 5 ) W R I T E ! 6 , 2 4 2 ) 2 4 2 F O R M A T t / / / I X , • * * * OPTIMA L A C T I O N I S ) T3 8E CHOSEN FOR THE CURRENT 1 0 P T I M A L P O L I C Y ' / / / ) C * * * F I N D THE I N V E R S E OF M A T R I X A BY C A L L I N G SUBROUTINE INV C A L L I N V { N S T » N D I M A , A , I P E R M » N D I M T » T , D E T » J E X P » C Q N D ) I F I D E T ) 2 4 5 , 6 2 0 , 2 4 5 2 4 5 DO 2 5 0 1 = 1 , N S T TD = 0 . DO 2 4 7 J = 1 , N S T I I P = I I P O L I J ) T D = T D * T ! I , J ) * D ! J , I I P ) 2 4 7 CONTINUE U ( I ) = T D 2 5 0 CONTINUE ITR=0 2 6 0 DO 3 0 0 1 = 1 , N S T I I P = I I P 0 L U 1 PU = 0 . DO 2 6 5 J = 1 , N S T PU=PU + P ( I , I I P , J ) * U ( J ) 2 6 5 CONTINUE DUI I ) = D { I , I I P ) * D I S C R * P U DBU=DU( I ) - U ! I ) I F U . E Q . l ) GO TO 2 9 0 I F { D 8 U . G T . B U M X ) GO TO 270 IF I D B U . L T . B U M N ) GO TO 2 8 0 GO TO 3 0 0 2 7 0 BUMX=DBU GO TO 3 0 0 2 8 0 BUMN=DBU GO TO 3 0 0 2 9 0 BUMX=DBU BUMN=DBU 3 0 0 CONTINUE UERR=BUMX-BUMN IF I U E R R . L E . E R R ) GO TO 3 1 0 DO 3 0 5 1 = 1 , N S T U l I ) =DU( I ) 3 0 5 C O N T I N U E I T R = I T R + 1 GO TO 2 6 0 3 1 0 A D D = D I S C R * B U M N / ( l . - D I S C R ) DO 3 2 0 1 = 1 , N S T U! I ) = D U ! I ) + ADD 3 2 0 CONTINUE W R I T E ( 6 , 5 9 0 ) ITR DO 3 2 5 1 = 1 , N S T DO 3 2 3 K = 1 , N A C P V = 0 . DO 3 2 1 J = 1 , N S T PV = P V + P ( I , K , J ) * V ( J ) 3 2 1 CONTINUE B V ( I , K ) = R < I , K ) + A L P H A * D { I , K ) + 0 1 S C R * P V ~ V < I ) 1 5 8 -3 2 3 CONTINUE 3 2 5 CONTINUE C A L L T I M E ( O ) ITER=1 3 3 0 DO 3 4 0 1 = 1 , N S T B ( I , 1 ) = U ( I ) DO 3 3 5 K = 1 , N A C P U = 0 . DO 3 3 3 J = 1 , N S T P U = P U + P ( I » K , J ) * U ( J ) 3 3 3 CONTINUE B U I I , K ) = D ( I , K ) + D I S C R * P U - U ( I ) 3 3 5 CONTINUE 3 4 0 CONTINUE KN=0 DO 3 8 0 1 = 1 , N S T 1 1 P = I I P O L ( I ) O B V = B V ( I , I I P ) OBU=BU< I , I I P ) DO 370 K = 1 , N A C I F ( K . E Q . I I P ) GO TO 3 7 0 B V I K = B V ( I , K ) BU IK=BU( I , K ) I F ( B U I K . L E . O B U ) GO TO 3 7 0 I F ( B V I K . G T . O B V ) GO TO 3 6 0 KN=KN+1 T A L P H = ( - B V I K + O B V ) / ( B U I K - O B U ) I F ( K N . E Q . l ) GO TO 3 5 0 A D £ P S = D A L P H + E P S L N S 8 E P S = D A L P H - E P S L N I F ( T A L P H . G E . A O E P S ) GO TO 3 7 0 I F I T A L P H . L E . S B E P S ) GO TQ 3 5 0 I F ( B U I K . L E . B U I ( I i 3 GO TO 3 7 0 KKPOL 1 1 ) = K BUI I I ) = B U I K GO TO 3 70 3 5 0 D A L P H = T A L P H DO 3 5 7 J = 1 , N S T I F ( J . E Q . I ) GO TO 3 5 5 K K P O L ( J ) = I I P O L ( J ) B U I I J ) = 0 . GO TO 3 5 7 3 5 5 K K P O L ( J ) = K BU I ( J ) = B U I K 3 5 7 CONTINUE GO TO 3 7 0 3 6 0 W R I T E ( 6 , 3 6 5 ) I , K , B V I K , B U I K , O B V , O B U 3 6 5 F 0 R M A T 1 5 X , ' B E T T E R P O L I C Y EX I S T • , 5 X , 2 1 5 , 4 E 1 6 . 7 ) 3 7 0 CONTINUE 3 8 0 C O N T I N U E 3 8 5 I F ( K N . E Q . O ) GO TQ 6 0 0 AL PHA= A L P H A * D A L P H DO 3 9 3 1 = 1 , N S T I F ( K K P O L ( I ) . E Q . 1 1 POL( I ) ) GO TO 3 9 3 W R I T E ( 6 , 3 9 0 ) A L P H A , I , K K P O L ( I ) 3 9 0 F O R M A T ! 5 X , ' A L P H A = • , F 2 0 . 1 4 , 5 X , » I = ' , I 5 , 5 X , ' K = ' , I 5 ) 3 9 3 CONTINUE P R C N T = { E L M / T N A C 1 * 1 0 0 . W R I T E ( 6 , 3 9 4 ) PRCNT 3 9 4 F 0 R M A T 1 5 X , • P R C N T ' , 5 X , F 1 0 . 6 J APRNT=APRNT+PRCNT EL M=0. I F ( A L P H A . G E . A L P M X ) GO TO 7 0 0 I F ( A L P H A . L T . A L P M N ) GO TO 7 0 0 I F ( K N . E Q . l ) GO TO 3 9 8 KN = 0 0 0 3 9 7 1 = 1 , N S T K K P = K K P O L ( I ) O O B U = B U ( I , K K P ) n p = i i P O L m G B V = B V l I , I I P ) O B U = B U I I , I I P ) DO 396 K = 1 , N A C I F t K . E Q . I I P ) GO TO 3 9 6 I F ( F L A G ! I , K ) . G T . O . ) GO TO 3 9 6 8 U I K = B U ( I , K ) I F I B U I K . L E . O O B U ) GO TO 3 9 6 B V I K = B V I I , K ) I F ( 8 V I K . G T . 0 B V ) GO TO 3 9 6 B U I K = B U I K - O B U T A L P H = ( O B V - B V I K ) / B U I K KN=KN*1 I F ( K N . E Q . l ) GO TO 3 9 5 ' I F ISALPH.LT.TALPH) GO TO 3 9 6 I F ( S A L P H . G T . T A L P H ) GO TO 3 9 5 I F I B U I K . L E . P U ) GO TO 3 9 6 3 9 5 S A L P H = T A L P H I S C N D = I KSCND=K P U = B U I K 3 9 6 CONTINUE 3 9 7 CONTINUE GO TO 3 9 9 3 9 8 KN=0 3 9 9 DO 9 5 0 K = 1 , N S T I F I K K P O L I K ) . E Q . I I P O L I K ) ) GO TO I F R S T = K KF RST=KKPOL iK) DO 4 1 0 J = 1 , N S T IF I J . E Q . I F R S T ) GO TO 4 0 0 R V ( J ) = - D I S C R * P I I F R S T , K F R S T , J ) GO TO 4 1 0 4 0 0 R V i J ) = 1 . - D I S C R * P ( I F R S T , K F R S T , J ) 4 1 0 CONTINUE DO 4 3 0 1 = 1 , N S T PU = 0 . DO 4 2 0 J = l , N S T P U = P U + R V I J ) * T ( J , I ) 4 2 0 CONTINUE DU ( I )=PU 4 3 0 CONTINUE P U = D U ( I F R S T ) I F I P U . E Q . O . ) GO TO 6 2 0 DO 4 5 0 J = 1 , N S T I F { J . E Q . I F R S T ) GO TQ 4 4 0 D U ( J ) = - D U J J ) / P U GO TO 4 5 0 4 4 0 0 U ( J J = 1 / P U 4 5 0 CONTINUE DO 4 8 0 1 = 1 , N S T 160. P V = T ( I , I F R S T ) DO 4 70 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 6 0 TI 1 , J ) = T ( I , J ) + P V * D U 1 J ) GO TO 4 7 0 4 6 0 T ( I » J ) = P V * D U ( J ) 4 7 0 CONTINUE 4 8 0 CONTINUE 9 0 0 11 P O L ( K ) = K K P O L ( K ) 9 5 0 CONTINUE DO 4 9 0 1 = 1 , N S T V( I ) = V I I ) + D A L P H * U I I ) P U = 0 . DO 4 8 5 J = 1 , N S T 11P=11 POL <JI PU=PU + T ( I , J ) * B U U » H P ) 4 8 5 CONTINUE U ( I ) = U I I ) + P U 4 9 0 CONTINUE ITR=0 5 0 0 DO 5 5 0 1 = 1 , N S T I I P = I I P O L ( I ) P U = 0 . DO 510 J = 1 , N S T P U = P U + P I I , I I P , J ) * U I J ) 5 1 0 CONTINUE ' D U U ) = D I I , I IP.) + D I S C R * P U D B U = D U ( I ) - U ( I ) IF { I . E Q . l ) GO TO 5 4 0 I F I Q B U . G T . B U M X ) GO TO 5 20 I F I D B U . L T . B U M N ) GO TO 5 3 0 GO TO 5 5 0 5 2 0 BUMX=DBU GO TO 5 5 0 5 3 0 BUMN=DBU GO TO 5 5 0 5 4 0 8UMX=DBU BUMN=DBU 5 5 0 CONTINUE UERR=BUMX-BUMN I F ( U E R R . L E . E R R ) GO TO 5 7 0 DO 5 6 0 1 = 1 , N S T UI I ) = D U ( I ) 5 6 0 CONTINUE I T R = I T R + 1 GO TO 5 0 0 5 7 0 A D D = D I S C R * B U M N / 1 1 . - D I S C R ) DO 5 8 0 1 = 1 , N S T U l I ) = 0 U I I ) + 4 0 0 D U ( I ) = U ( I ) - B I I , 1 ) IF ( I . E Q . l ) GO TQ 5 7 5 I F ( D U ( I ) . L E . D U M X ) GO TO 5 8 0 5 7 5 D U M X = D U ( I ) 5 8 0 CONTINUE W R I T E ( 6 , 5 9 0 ) ITR 5 9 0 F O R M A T ( 5 X , ' N U M B E R OF I T E R A T I O N I N S - O P E R A T I O N FOX U I S « , 5 X , I 5 ) DUMX=DISCR*DUMX DO 5 9 9 1 = 1 , N S T DO 5 9 7 K = 1 , N A C B V U , K ) = B V ( I , K ) + D A L P H * B U ( I , K I 161. B U ( I , K ) = B U ( I » K ) + D U M X - D U I I ) 5 9 7 CONTINUE 5 9 9 CONTINUE I T E R = I T E R + 1 I F ( K N . E Q . O ) GO TO 8 1 0 I I P = I I P O L ( I S C N D ) P U = 0 . PV = 0 . DO 8 0 0 J = 1 , N S T P U = P U + P ( I S C N D , I I P , J ) * U ( J ) P V = P V + P ( I S C N D , K S C N D , J ) * U ( J ) 8 0 0 CONTINUE P U = D { I S C N D , I I P ) + D I S C R * P U - U ( I S C N D ) P V = D ( I S C N D , K S C N D ) * D I S C R * P V - U ( I S C N D ) I F i P U . G E . P V ) GO TO 8 1 0 E X A L P = A L P H A * ( B V ( I S C N D , I IP ) - BV ( I S C N D , K S C N D ) ) / I P V - ? U ) I F ( E X A L P . L T . A L P H A . O R . E X A L P . G T . A L P M X ) GO TO 8 1 0 GO TO 8 2 0 8 1 0 EX ALP=ALPMX 8 2 0 KN=0 DO 8 8 0 1 = 1 , N S T B ( I , 1 ) = U ( I ) I I P = I I P O L ( I ) PU = 0 . DO 8 3 0 J = 1 , N S T PU = PU + P ( I » I IP , J ) * U ( J ) 8 3 0 CONTINUE BUI I t I IP ) =01 I , I I P ) + D I S C R * P U - U I I ) 0 8 V = B V l I , I I P ) O B U = B U ( I , I I P ) D B U = O B V + { E X A L P - A L P H A ) * Q B U DO 8 7 0 K = 1 , N A C I F ( K . E Q . I I P ) GO TO 8 7 0 RBV=BV( I , K ) + ( E X A L P - A L P H A ) * B U ( I , K ) I F ( F L A G ! I , K ) . L E . O . ) GO TO 8 4 0 I F I R B V . L T . D B U ) GO TO 8 5 3 F L A G ! I , K ) = - 1 . P V = 0 . PU = 0 . DO 8 3 5 J = 1 , N S T P V = P V * P ( I , K , J ) * V ( J ) PU = PU + P I I , K , J ) * U I J ) 8 3 5 CONTINUE BV I I , K ) = R ( I , K ) * A L P H A * D ( I , K ) * D I S C R * P V - V ( I ) B U ( I , K ) = D ( I , K ) + D I S C R * P U - U ( I ) GO TO 8 5 5 8 4 0 I F ( R B V . L T . D B U ) GO TO 8 5 0 PU = 0 . DO 845 J = 1 , N S T P U = P U + P ( I , K , J ) * U ( J ) 8 4 5 CONTINUE BU ( I , K ) = 0 ( I , K ) * D I S C R * P U - U ( I ) GO TO 8 5 5 8 50 F L A G ( I , K ) = 1 . 8 5 3 ELM=ELM + 1 . GO TO 8 7 0 8 5 5 8 V I K = B V { I , K ) B U I K = B U ( I , K ) I F I B U I K . L E . Q B U ) GO TO 8 7 0 I F ( B V I K . G T . O B V ) GO TO 8 6 0 1 6 2 -KN=KN+1 T A L P H = ( - B V I K + O B V ) / ( B U I K - Q B U ) I F ( K N . E Q . l ) GO TO 8 5 7 A D E P S = D A L P H + E P S L N S B E P S = D A L P H - E P S L N I F I T A L P H . G E . A D E P S ) GO TO 8 7 0 I F ( T A L P H . L E . S B E P S ) GO TO 8 5 7 I F ( B U I K . L E . B U I ( I ) ) GO TO 8 7 0 KKPOL ( I )=K BUim = B U I K GO TO 8 7 0 8 57 D A L P H = T A L P H DO 8 5 9 J = 1 , N S T I F U . E Q . l ) GO TO 8 5 8 KK POL ( J ) = 11 POL ( J ) B U I U ) = 0 . GO TO 8 5 9 8 5 8 K K P O L ( J ) = K B U I U ) = B U I K 8 5 9 CONTINUE GO TO 8 7 0 8 6 0 W R I T E ( 6 , 8 6 5 ) I , K , B V I K , B U I K , O B V , O B U 8 6 5 F O R M A T ( 5 X , ' B E T F E R P O L I C Y EX I S T ' , 5 X , 2 1 5 , 4 E 1 6 . 7 ) 8 7 0 CONTINUE 8 8 0 CONTINUE GO TO 3 8 5 6 0 0 W R I T E ( 6 , 6 1 0 ) 6 1 0 F 0 R M A T ( 5 X , ' O P T I M A L FOR ALL ALPHA GREATER THAN THE C J R R E N T V A L U E 1 ) GQ TO 7 0 0 6 2 0 WRITE ( 6 , 6 3 0 3 6 3 0 FORMAT ( 5 X , » I N V E R S I O N F A I L E D ' ) 7 0 0 C A L L T I M E ( l , l ) P V = I T E R APRNT = A P R N T / P V W R I T E ( 6 , 7 5 0 ) APRNT 7 5 0 F O R M A T ( 5 X » ' A V E R A G E PRCNT = ' , 5 X , F 1 0 . 6 ) STOP END C * * * Q) CODE FOR P A R A M E T R I C MDP WITH PROCEDURE (11-2) C*** 163. RE A L * 8 T A I P H , D A L P H , A L P H A , A , B , T , V , D E T , C O N D , U , R V , O J , B V , 8 J , P V , P U , 1 G B V , 0 B U , B V I K , B U I K , O O B U , B V M X , B V M M , B U M X , 3 U M M , R B V , 0 3 J , D J M X , P D U M X » 2 D U M A X , E X A L P , S A L P H , A D E P S , S 8 E P S DIMENS ION V ( 5 0 ) . R f 5 0 , 5 0 ) , P 1 5 0 , 5 0 , 5 0 ) , 1 1 P O L I 5 0 ) , D I 5 0 , 503 , A { 5 0 , 503 , I T ( 5 0 , 5 0 ) , B ( 5 0 » 1 ) » I PERM 1 1 0 0 ) , K K P O L ( 5 0 ) , U 1 5 0 ) , R V { 5 3 ) , 3 J I 5 0 ) , 2 B V ( 5 0 , 5 0 ) , B U I 5 0 , 5 0 ) , F L A G ( 5 0 , 5 0 ) , D U M X ( 5 0 ) , P O U M X i 5 3 ) , B U I ( 5 0) REWIND8 REWIND9 E R R = 0 . 0 0 1 E P S L N = 0 . 0 0 0 0 0 1 ND IMA = 50 NDIMBX=50 NDIMT=50 NS0L=1 R E A D ( 5 , 1 ) N S T , N A C , D I S C R , A L P M N . A L P M X 1 F 0 R M A T ( 2 I 5 , 3 F 1 0 . 5 ) AL PHA=ALPMN T N A C = N S T * N A C E L M = 0 . A P R N T = 0 . R E A D ! 8 ) ( ( R ( I , K ) , D ( I , K ) , 1 = 1 , N S T ) , K = 1 , N A C ) I T E R = 0 I T R = 0 DO 2 1 = 1 , N S T RE A D ( 9 ) ( ( P ( I , K , J ) , K = 1 , N A C ) , J = 1 , N S T ) 2 CONTINUE DO 50 1 = 1 , N S T DO 20 K = 1 , N A C FL A G ( I , K ) = - 1 RD IK=R ( I , K ) t-AL P H A * D 1 1 , K ) I F I K . E Q . l ) GO TO 10 I F ( O R I . G E . R D I K ) GO TO 2 0 10 O R I = R D I K I I P=K 2 0 CONTINUE B ( I , 1 ) = 0 R I K K P O L I I ) = 1 1 P DO 4 0 J = 1 , N S T I F { I . E Q . J ) GO TO 3 0 A U , J ) = - O I S C R * P U , I I P , J ) GO TO 4 0 3 0 A l I * J ) = 1 . - D I S C R * P ( I , 1 1 P , J ) 4 0 CONTINUE 50 CONTINUE GO TO 1 3 0 6 0 I T E R = I T E R + 1 DO 120 1 = 1 , N S T DO 110 K = 1 , N A C PV = 0 . DO 100 J = 1 , N S T P V = P V + P ( I , K , J ) * V 1 J ) 1 0 0 CONTINUE R P V I K = R U , K ) # - A L P H A * D 1 I , K ) + D I S C R * P V I F I K . E Q . l ) GO TO 1 0 5 I F ( R V I . G E . R P V I K ) GO TO 110 1 0 5 R V I = R P V I K I I P = K 110 CONTINUE DO 115 J = i , N S T , I F ( I . E Q . J ) GO TO 1 13 1 1 AC I , J ) = - D I S C R * P ( I , I I P , J ) GO TO 1 1 5 113 A ( I , J ) = 1 . - D I S C R * P ( I , I I P , J ) 1 1 5 CONTINUE BI I , 1 ) = R ( I , I I P ) + A L P H A * D ( I , I I P ) I I P O L ( I ) = I I P 1 2 0 C O N T I N U E DO 125 1 = 1 , N S T I F ( I I P O L ( I ) . N £ . K K P G L ( I ) ) GO TO 127 1 2 5 CONTINUE GO TO 1 9 0 127 DO 1 2 9 1 = 1 , N S T K K P O L ( I ) = I I P O L ( U 1 2 9 CONTINUE C * * * S O L V E THE SYSTEM OF EQUATIONS AV=8 BY C A L L I N G S U B R O U T I N E ' SLE 130 C A L L S L E ( N S T , N D I M A , A , N S O L , N D I M B X , B , V , I P E R M , N D I M T , T , 3 E T , J E X P ) I F ( D E T ) 1 3 5 , 1 7 5 , 1 3 5 1 3 5 GO TO 6 0 175 W R I T E ( 6 , 1 8 0 ) 1 8 0 F O R M A T ! 5 X , * S O L U T I O N F A I L E D ' ) GO TO 7 0 0 1 9 0 DO 200 1 = 1 , N S T I I P = I I P O L ( I ) PV = 0 . DO 192 J = 1 , N S T P V = P V + P ( I , I I P , J ) * V ( J ) 1 9 2 CONTINUE R V ( I ) = R ( I , I I P ) + A L P H A * D ( 1 , 1 1 P ) + D I S C R * P V R B V = R V ( I ) - V ( I ) I F ( I . E Q . l ) GO TO 198 I F ( R B V . G T . B V M X ) GO TO 1 9 4 I F ( R B V . L T . B V M N ) GO TO 196 GO TO 2 0 0 1 9 4 BVMX=RBV GO TO 2 0 0 1 9 6 BVMN=RBV GO TO 2 0 0 1 9 8 BVMX=RBV BVMN=RBV 2 0 0 CONTINUE VERR=BVMX-BVMN I F ( I T R . E Q . 0 . A N D . V E R R . L E » £ R R ) GO TO 2 0 6 A D D = D I S C R * B V M N / ( I . - D I S C R ) DO 202 1 = 1 , N S T V C I ) = R V ( I ) + ADD 2 0 2 CONTINUE I F ( V E R R . L E . E R R ) GO TO 2 0 4 I T R = I T R + 1 GO TO 1 9 0 2 0 4 W R I T E ( 6 , 2 0 5 ) I T R 2 0 5 F 0 R M A T ( 5 X , ' N U M B E R OF I T E R A T I O N I N T - O P E R A T I O N Fll V I S * , 5 X , I 5 ) GO TO 6 0 2 0 6 W R I T E ( 6 , 2 0 5 ) I T R W R I T E ( 6 , 2 0 8 ) A L P H A 2 0 8 F O R M A T ( 5 X , ' A L P H A = « , F 2 0 . 1 4 ) A D D = B V M N / l 1 . - D I S C R ) DO 2 0 9 1 = 1 , N S T VC I ) = V( I ) t-ADO 2 0 9 CONTINUE W R I T E ( 6 , 2 1 0 ) ' 1 5 b -2 1 0 F O R M A T ( 1 0 X , • S T A T E ' , 1 Q X , * POL I C Y 1 , 1 5 X , ' R E T U R N ' , 1 Q X , ' O P T . P O L I C Y ' ) WRITE ( 6 , 2 2 0 H I , N A C , V ( 13 ., I I P O L I I » , 1 = 1 , NST) 2 20 F O R M A T ! 1 0 X , I 3 , 1 2 X , 1 3 , 1 0 X , E 1 6 . 7 , 1 0 X , 1 3 / > W R I T E ( 6 , 2 3 0 I 2 3 0 F O R M A T ( 1 0 X , • 0 1 S C R * , 1 0 X , • I T E R A T I O N ' ) W R I T E ! 6 , 2 4 0 ) D I S C R , I T E R 2 4 0 F O R M A T ! 1 0 X , F 7 . 5 , 6 X , I 5 ) W R I T E ( 6 , 2 4 2 ) 2 4 2 FORMAT I / / / I X , • * * * O P T I M A L ACT I O N ! S ) TO BE CHOSEN FOR THE CURRENT 1 0 P T I M A L P O L I C Y ' / / / ) C * * * F I N O THE I N V E R S E OF M A T R I X A BY C A L L I N G SUBROUTINE INV C A L L I N V ! N S T , N O I M A , A , I P E R M , N D I M T , T , D E T , J E X P , C O N O ) I F ( D E T ) 2 4 5 , 6 2 0 , 2 4 5 2 4 5 DO 2 5 0 1 = 1 , N S T TD = 0 . DO 247 J = 1 , N S T 1 1 P = I I P O L ( J ) T D = T D * T { I , J ) * D ( J , I I P ) 2 4 7 CONTINUE U( I ) = TD 2 5 0 CONTINUE I T R = 0 2 6 0 DO 3 0 0 1 = 1 , N S T I I P = I I P O L ( I ) PU = 0 . DO 2 6 5 J = 1 , N S T PU = P U + P ! I , I I P , J ) * U ( J ) 2 6 5 CONTINUE DUI I ) = D ( I , I I P ) + D I S C R * P U DBU=DUl I ) - U I I ) I F U . E Q . l ) GO TO 2 9 0 I F i D B U . G T . B U M X ) GO TO 2 7 0 I F ( D 8 U . L T . B U M N ) GO TO 2 8 0 GO TO 3 0 0 2 7 0 BUMX=OBU GO TO 3 0 0 2 8 0 BUMN=OBU GO TO 3 0 0 2 9 0 BUMX=DBU BUMN=DBU 3 0 0 CONTINUE UERR=BUMX-8UMN I F ( U E R R . L E . E R R ) GO TO 3 1 0 DO 3 0 5 1 = 1 , N S T Ut I ) = D U ( I ) 3 0 5 CONTINUE I T R = I T R + 1 GO TO 2 6 0 3 1 0 A D D = D I S C R * B U M N / ( l . - D I S C R ) DO 320 1 = 1 , N S T U( I ) =DU ( I ) +-ADD 3 2 0 C O N T I N U E W R I T E 1 6 , 5 9 0 ) ITR DO 3 2 5 1 = 1 , N S T DO 3 2 3 K = l , N A C P V = 0 . DO 3 2 1 J = 1 , N S T P V = P V + P ( I , K , J ) * V ( J ) 3 2 1 C O N T I N U E B V ! I , K ) = R U , K ) + A L P H A * D ! I , K ) + D I S C R * P V - V i I ) 166. 3 2 3 C O N T I N U E 3 2 5 C O N T I N U E C A L L T I M E I O ) I T E R = 1 3 3 0 D O 3 4 0 I = l T N S T A{ I , I T E R ) = U ( I ) D O 3 3 5 K = 1 , N A C P U = 0 . D O 3 3 3 J = 1 , N S T P U = P U + P ( I » K , J ) * U l J ) 3 3 3 C O N T I N U E B U ( I » K ) = D ( I , K ) * D I S C R * P U - U ! I ) 3 3 5 C O N T I N U E 3 4 0 C O N T I N U E K N = 0 D O 3 8 0 1 = 1 , N S T I I P = I I P O L ! I ) G B V = B V I I , I I P ) O B U = B U ( I , I I P ) D O 3 7 0 K = 1 , N A C I F ( K . E Q . I I P ) G O T O 3 7 0 B V I K = B V < I , K ) B U I K = B U < I , K ) I F ( B U I K . L E . O B U ) G O T O 3 7 0 I F ( B V I K . G T . O B V ) G O T O 3 6 0 K N = K N + 1 T A L P H = ! - B V I K « - 0 B V ) / { B U I K T Q 8 U ) I F C K N . E Q . l ) G O T O 3 5 0 A D E P S = D A L P H + E P S L N S B E P S = D A L P H - E P S L N I F ! T A L P H . G E . A D E P S ) G O T O 3 7 0 I F I T A L P H . L E . S B E P S ) G O T O 3 5 0 I F I B U I K . L E . B U I I I ) ) G O T O 3 7 0 K K P Q L U ) = K B U I { I ) = B U I K G O T O 3 7 0 3 5 0 D A L P H = T A L P H D O 3 5 7 J = 1 , N S T I F ( J . E Q . I ) G O T O 3 5 5 K K P O L ( J I = 11 P O L ( J ) B U I ( J ) = 0 . G O T O 3 5 7 3 5 5 K K P O L ( J ) = K B U I ( J ) = B U I K 3 5 7 C O N T I N U E G O T O 3 7 0 3 6 0 W R I T E ( 6 , 3 6 5 ) I , K , B V I K , B U I K , 0 B V , 0 B U 3 6 5 F O R M A T { 5 X , • B E T T E R P O L I C Y E X I S T ' , 5 X , 2 1 5 , 4 E 1 6 . 7 ) 3 7 0 C O N T I N U E 3 8 0 C O N T I N U E 3 8 5 I F I K N . E Q . O ) G O T O 6 0 0 A L P H A = A L P H A + D A L P H D O 3 9 3 1 = 1 , N S T I F I K K P O L U ) . E Q . 1 1 P O L I I ) ) G O T O 3 9 3 W R I T E I 6 , 3 9 0 ) A L P H A , I , K K P O L I I ) 3 9 0 F O R M A T ! 5 X , • A L P H A = • , F 2 0 . 1 4 , 5 X , ' I = ' , I 5 , 5 X , ' K = ' , I 5 ) 3 9 3 C O N T I N U E P R C N T = < E L M / T N A C ) * 1 0 0 . W R I T E ( 6 , 3 9 4 ) PRCNT 3 9 4 F O R M A T ( 5 X , ' P R C N T ' , 5 X , F 1 0 . 6 ) 1 6 7 -AP RNT=APRNT + PRCNT E L M = 0 . I F ( A L P H A . G E . A L P M X ) GO TO 7 0 0 I F ( A L P H A . L T . A L P M N ) GO TO 7 0 0 I F ( K N . E Q . l ) GO TO 3 9 8 KN = 0 DO 3 9 7 1 = 1 , N S T K K P = K K P O L ( I ) O O B U = B U U , K K P I I I P = I I P O L ( I ) OB V=BV( I , 11 P ) O B U = B U ( I , I I P ) DO 3 9 6 K = 1 , N A C I F ( K . E Q . I I P ) GO TO 396 I F ( F L A G ( I , K ) . G T . 0 . ) GO TO 3 9 6 B U I K = B U ( I , K ) I F ( B U I K . L E . O O B U ) GO TO 3 9 6 B V I K = B V ( I » K ) I F ( B V I K . G T . O B V ) GO TQ 3 9 6 B U I K = B U I K - O B U T A L P H = ( O B V - B V I K ) / B U I K KN=KN+1 I F C K N . E Q . l ) GO TO 3 9 5 I F I S A L P H . L T . T A L P H ) GO TO 3 9 6 I F ( S A L P H . G T . T A L P H ) GO TO 3 9 5 I F ( B U I K . L E . P U ) GO TO 3 9 6 3 9 5 S A L P H = T A L P H ISCND=I KSCND=K PU=BUIK 3 9 6 CONTINUE 3 9 7 CONTINUE GO TO 3 9 9 3 9 8 KN=0 3 9 9 DO 9 5 0 K = 1 , N S T I F I K K P O L ( K ) . E Q . I I P O L I K ) ) GO TO 9 0 0 I F R S T = K K F R S T = K K P O L ( K ) DO 4 1 0 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 0 0 R V I J ) = - D I S C R * P i I F R S T , K F R S T , J ) GO TO 4 1 0 4 0 0 R V I J ) = 1 . - D I S C R * P ! I F R S T , K F R S T , J l 4 1 0 CONTINUE DO 4 3 0 1 = 1 , N S T P U = 0 . DO 4 2 0 J = 1 , N S T P U = P U + R V ( J ) * T ( J , I ) 4 2 0 CONTINUE D U ( I ) = P U 4 3 0 CONTINUE P U = D U ( I F R S T ) I F I P U . E Q . O . I GO TO 6 2 0 0 0 4 5 0 J = i , N S T I F ( J . E Q . I F R S T ) GO TO 4 4 0 D U ( J ) = - D U ( J ) / P U GO TO 4 5 0 4 4 0 D U < J ) = 1 / P U 4 5 0 CONTINUE 0 0 4 8 0 1 = 1 , N S T 1 6 8 ' PV = T 1 1 , I F R S T ) DO 4 7 0 J = 1 , N S T I F ( J . E Q . I F R S T ) GO TO 4 6 0 T U , J ) = T U , J ) + P V * D U ( J ) GO TO 4 7 0 4 6 0 TI I , J ) = P V * D U { J ) 4 7 0 CONTINUE 4 8 0 CONTINUE 9 0 0 I I P O L ( K ) = K K P O L I K ) 9 5 0 CONTINUE DO 4 9 0 1 = 1 , N S T V U ) = V ( I ) + D A L P H * U ( I ) P U = 0 . DO 4 8 5 J = l , N S T I I P = I I P O L ( J ) P U = P U + T ( I , J ) * B U ( J , I I P ) 4 3 5 CONTINUE U( I.) =U( I )+PU 4 9 0 CONTINUE I T R = 0 5 0 0 DO 5 5 0 1 = 1 , N S T I I P = I I P O L U ) P U = 0 . DO 510 J = 1 , N S T PU=PU*-P U , I I P , J ) * U I J ) 5 1 0 CONTINUE D U < I ) = O I 1 , 1 I P ) + D I S C R * P U D B U = D U ( I ) - U ( I ) I F U . E Q . l ) GO TO 5 4 0 I F ( D 8 U . G T . B U M X ) GO TO 5 2 0 I F ( D B U . L T . B U M N ) GO TO 5 3 0 GO TO 5 5 0 5 2 0 BUMX=DBU GO TO 5 5 0 5 3 0 BUMN=OBU GO TO 5 5 0 5 4 0 BUMX=DBU BUMN=DBU 5 5 0 CONTINUE UERR=BUMX~BUMN I F I U E R R . L E . E R R ) GO TO 5 7 0 DO 5 6 0 1 = 1 , N S T U( I ) = D U I I ) 5 6 0 CONTINUE I T R = I T R + 1 GO TO 5 0 0 5 7 0 A D D = D I S C R * B U M N / ( l . - D I S C R ) DO 5 8 0 1 = 1 , N S T U l I )=DUI I)+ADD 5 8 0 CONTINUE W R I T E 1 6 , 5 9 0 ) ITR 5 9 0 F O R M A T { 5 X , • N U M B ER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S « , 5 X , I 5 ) DO 594 K = l , I T E R DO 592 1 = 1 , N S T D U I K = U ( I ) - A ( I , K ) I F I K . E Q . l TER) D U U ) = D U I K I F U . E Q . l ) GO TO 5 9 1 I F ( D U M A X . G E . D U I K ) GO TO 592 5 9 1 DUMAX=DUIK 5 9 2 CONTINUE l 6 9 > I F { I T E R . E Q . 1 3 GO TO 5 9 3 I F ( K . E Q . I T E R ) GO TO 5 9 3 P D U M X ( K ) = D U M X ( K ) 5 9 3 D U M X ( K ) = D I S C R * D U M A X 5 9 4 CONTINUE DO 5 9 9 1 = 1 , N S T DO 5 9 7 K = 1 , N A C 8 V ( I » K ) = B V ! I , K ) + D A L P H * 8 U ( I , K ) L = F L A G ( I , K ) I F ! L . L E . Q . O R . L . E Q . I T E R ) GO TO 5 9 5 B U ( I , K ) = B U ! I , K ) + D U M X ! L ) - P D U M X I L ) - D U ! I ) GO TO 5 9 7 5 9 5 B U ( I , K ) = B U ! I , K ) * D U M X I I T E R ) - D U ( I ) 5 9 7 CONTINUE 5 9 9 CONTINUE I T E R = I T E R + 1 I F I K N . E Q . O ) GO TO 8 1 0 1 1 P = I I P O L ! I S C N D ) P U = 0 . PV = 0 . DO 8 0 0 J = 1 , N S T PU=PU + P ( I S C N D , I I P , J ) * U I J ) P V = P V * - P ! I S C N D , K S C N O , J ) * U ! J ) 8 0 0 CONTINUE P U = D { I S C N D , 1 1 P 3 * D I S C R * P U - U 1 1 S C N D ) P V = D ( I S C N D , K S C N D J + D I S C R * P V - U < I S C N D ) IF >( P U . G E . P V ) GO TO 310 EX ALP=ALPHA+-{BVI I S C N D , I I P 3 - B V ( I S C N D , K S C N D ) 3 / I P V - ? J 3 I F { E X A L P . L T . A L P H A . O R . E X A L P . G T . A L P I 4 X ) GO TO 810 GO TO 8 2 0 8 1 0 E X A L P = ALPMX 8 2 0 KN=0 DO 8 8 0 1 = 1 , N S T A t I , I T E R ) = U < I ) 11P=11 POL I I ) P U = 0 . DO 8 3 0 J = 1 , N S T P U = P U + P ! I , I I P , J ) * U « J ) 8 3 0 CONTINUE BUI I » 11 P ) = D t I , I I P ) + D I S C R * P U - U t I ) O B V = B V ( I , 11 P) O B U = B U ( I , I I P ) D B U = O B V + ( E X A L P - A L P H A ) * O B U DO 8 7 0 K = 1 , N A C I F ( K . E Q . I I P ) GO TO 8 7 0 RBV=BV( I , K ) * ( E X A L P - A L P H A ) * B U U , K ) I F ( F L A G ( I , K ) . L E . O . ) GO TO 8 4 0 I F ( R B V . L T . D B U ) GO TO 8 5 3 FL A G ! I » K ) = - 1 . P V = 0 . P U = 0 . DO 8 3 5 J = 1 , N S T P V = P V + P ( I , K , J ) * V { J ) P U = P U + P ( I , K , J ) * U ( J ) 8 3 5 CONTINUE BV ( I , K ) = R ( I , K 3 + A L P H A * D ( I , K ) + D I S C R * P V - V ( I 3 B U ( I , K ) = D ( I , K ) * - D I S C R * P U - U ( I ) GO TO 8 5 5 8 4 0 I F ( R B V . L T . D B U ) GO TO 8 5 0 P U = 0 . 1 * DO 8 4 5 J = 1 , N S T P U = P U + P ( I , K , J ) * U ( J ) 8 4 5 CONTINUE B U ( I , K ) = D ( l , K ) + D I S C R * P U - U ( I ) GO TO 8 5 5 8 50 F L A G ( I , K ) = 1 T E R - 1 8 5 3 E L M = E L M + 1 . GO TO 8 7 0 8 5 5 B V I K = B V ( I , K ) B U I K = B U ( I , K ) I F ( B U I K . L E . O B U ) GO TO 8 7 0 I F ( B V I K . G T . O B V ) GO TO 8 6 0 KN=KN+1 T A L P H = ( - 8 V I K * - O B V ) / ( B U I K - 0 B U ) I F I K N . E Q . 1 ) GO TO 8 5 7 A D E P S = D A L P H + E P S L N S B E P S = O A L P H - E P S L N I F ( T A L P H . G E . A D E P S ) GO TO 8 7 0 I F ( T A L P H . L E . S B E P S ) GO TO 8 5 7 I F ( B U I K . L E . B U H I ) ) GO TO 8 7 0 K K P O L I I ) = K B U I U I = B U I K GO TO 8 7 0 8 5 7 DA L P H = T A L P H DO 8 5 9 J = 1 , N S T I F ( J . E Q . I ) GO TO 8 5 8 K K P O L ( J ) = I I P O L ( J ) B U I ( J ) = 0 . GO TO 8 5 9 8 5 8 K K P O L ( J ) = K B U I ( J ) = B U I K 8 5 9 CONTINUE GO TO 8 7 0 8 6 0 W R I T E(6, 8 6 5 ) I , K , B V I K , B U I K , Q B V , O B U 8 6 5 F 0 R M A T ( 5 X , • B E T T E R P O L I C Y EX I S T ' , 5 X , 2 1 5 , 4 E 16 . 7 ) 8 7 0 CONTINUE 8 8 0 CONTINUE GO TO 3 8 5 6 0 0 W R I T E ( 6 , 6 1 0 ) 6 1 0 F D R M A T ( 5 X , f O P T I M A L FOR A L L ALPHA GREATER THAN THE C J H R E N T VALUE * ) GO TO 7 0 0 6 2 0 WRITE ( 6 , 6 3 0 ) 6 3 0 FORMAT ( 5 X , • I N V E R S I O N F A I L E D * ) 7 0 0 C A L L TI M E ( 1 » 1 ) P V = I T E R A P R N T = A P R N T / P V . W R I T E ( 6 , 7 5 0 ) APRNT 7 5 0 F O R M A T ( 5 X , ' A V E R A G E PRCNT = ' , 5 X , F 1 0 . 6 ) STOP END * * * R) COMPUTATIONAL R E S U L T OF TWO C R I T E R I O N AUTOMOBILE REPLACEMENT * * * PROBLEM U S I N G CODE L WITH DISSGUNT RATE OF . 9 6 NUMBER OF I T E R A T I O N ALPHA = 0 . 0 STATE 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 2 0 21 22 2 3 24 2 5 26 IN T - O P E R A ! I ON FOR V I S 3 P O L I C Y RETURN O P T . POL I 41 - 3 . 2 5 5 2 7 6 1 E f 3 4 18 41 - 0 . 2 7 7 2 7 6 1 E + 0 4 18 41 - 0 . 2 3 8 2 7 6 1 E + 0 4 13 41 - 0 . 3 0 6 2 7 6 1 E + 0 4 18 41 - 0 . 3 1 3 2 7 6 1 E + 0 4 18 41 - 0 . 3 2 0 2 7 5 1 E + 0 4 18 41 - 0 . 3 2 7 2 7 6 1 E + 0 4 18 41 - 3 . 3 3 2 7 2 3 7 E + 0 4 1 41 - 0 . 3 3 7 7 9 2 4 E + 0 4 1 41 - 0 . 3 4 2 7 1 2 5 E + 0 4 1 41 - 0 . 3 4 7 4 3 7 9 E * 0 4 1 41 - 0 . 3 5 1 8 7 9 6 E + 0 4 1 41 - 0 . 3 3 6 0 6 2 3 E + 0 4 1 41 - 0 . 3 5 0 0 3 8 8 E + - 0 4 1 41 - 0 . 3 5 3 7 4 0 3 E + 0 4 1 41 - 0 . 3 5 7 2 7 6 1 E + 0 4 1 41 - 0 . 3 7 0 5 2 5 1 E + 0 4 1 41 - 0 . 3 7 3 6 0 2 4 E + 0 4 1 41 - 0 . 3 7 6 5 2 2 2 E + 0 4 1 41 - 0 . 3 7 9 2 9 7 9 E + 0 4 1 41 - 0 . 3 8 1 9 4 2 0 E f 0 4 1 41 - 0 . 3 3 4 4 6 6 3 E + 0 4 1 41 - 0 . 3 3 6 8 6 4 3 E + 0 4 1 41 - 0 . 3 3 9 1 3 4 5 E + 0 4 1 41 - 0 . 3 9 1 3 7 6 4 E + 0 4 1 41 - 0 . 3 9 2 7 7 9 1 E + 0 4 I 27 41 - 0 . 3 9 4 2 Q 1 9 E + 0 4 1 28 41 - 0 . 3 9 5 2 7 5 1 E + 0 4 18 29 41 - 0 . 3 9 6 2 7 6 1 E + 0 4 18 3 0 41 - 0 . 3 9 5 7 7 5 1 E * 0 4 18 31 41 - 0 . 3 9 7 2 7 6 1 E + 0 4 18 3 2 41 - Q . 3 9 7 7 7 6 1 E * 0 4 18 3 3 41 - 0 . 3 9 8 2 7 6 1 E + - 0 4 18 34 41 - 0 . 3 9 9 2 7 6 1 E + 0 4 18 3 5 41 - 0 . 3 9 9 7 7 6 1 E + 0 4 18 3 6 41 - 0 . 4 0 0 2 7 6 1 E + 0 4 18 3 7 41 - 0 . 4 0 0 7 7 6 1 E * - 0 4 18 38 41 - 0 . 4 0 1 7 7 6 1 E + 0 4 18 3 9 41 - 0 . 4 0 2 5 7 6 1 E + 0 4 18 4 0 41 - G . 4 0 3 2 7 6 1 E + 0 4 18 01 SCR I T E R A T I O N 0 . 9 6 0 0 0 6 * * * O P T I M A L A C T I O N ! S ) TO BE CHOSEN F 3 * THE " J R R E N T OPTIMAL P O L I C Y NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR J I S 21 ALPHA = 0 . 0 0 2 2 3 5 2 0 8 0 5 5 3 6 I = 7 K = 1 NUMBER OF I T E R A T I O N I N S - O P E R A T I ON FOR U I S 0 ALPHA = 0 . 0 4 7 8 2 7 1 1 2 6 7 2 2 9 I = 5 K = 1 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 0 5 7 5 3 7 0 9 6 5 2 3 0 6 I = 27 K = 18 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 A L P H A = 0 . 0 6 8 2 4 4 6 7 7 1 3 9 2 9 I = 5 K = I NUMBER OF I T E R A T I O N I N S - Q P E R A T I Q M FOR J I S 0 ALPHA = 0 . 0 7 5 4 4 0 8 7 2 8 5 0 2 2 I = 4 K = 1 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR J IS 0 A L P H A = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 1 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 2 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 3 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 27 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 2 3 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 29 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 3 0 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 31 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 32 K •= 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 3 3 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 34 K = 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I = 35 K •= 14 A L P H A 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I 3 6 K 14 ALPHA 0 . 0 3 6 7 0 2 1 1 9 2 0 3 0 0 I 3 7 ;< 14 ALPHA 0 . 0 8 6 7 0 2 1 1 9 2 Q 3 0 0 I 3 3 K 14 A L P H A 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I 3 9 K 14 ALPHA = 0 . 0 8 6 7 0 2 1 1 9 2 0 3 0 0 I 40 K = 14 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR J IS 0 ALPHA 0 . 2 1 4 5 9 9 4 5 5 1 2 7 2 4 I 25 K 14 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 2 5 0 4 9 3 5 3 8 8 8 9 9 1 I 1 K — 1 NUM3ER OF I T E R A T I O N I N S - Q P E R A T I O N FOR U IS 0 ALPHA = 0 . 2 5 5 6 5 0 3 2 1 5 5 6 3 4 I • 3 K 1 NUMBER OF I T E R A T I O N IN S - O P E R A T I O M FOR U IS 0 ALPHA 0 . 2 6 1 5 5 0 6 3 7 7 0 5 3 7 I = 2 K 1 NUMBER OF I T E R A T I O N "IN S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 3 2 8 9 4 3 0 7 8 2 9 0 5 7 I - 25 K 14 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 ALPHA 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 2 5 K = 2 A L P H A 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 26 K 2 ALPHA = 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 27 K •ZZ 2 A L P H A 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I •• 2 8 K zz 2 ALPHA 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 29 K zz 2 A L P H A 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 30 K zz 2 ALPHA 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I 31 K zz 2 ALPHA = 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 32 K zz 2 A L P H A ' 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 33 K zz 2 A L P H A • = 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I .= 3 4 K zz 2 ALPHA • 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 35 K zz 2 ALPHA : 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 3 6 K =r 2 ALPHA : 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 37 K zz 2 ALPHA = 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 33 K zz 2 A L P H A • 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I - 39 K zz 2 ALPHA : 0 . 4 2 4 6 3 1 0 4 2 3 6 1 5 1 I = 4 0 IC zz 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 2 5 6 1 6 6 4 6 1 3 7 8 4 I = 24 K 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 4 2 8 9 3 4 2 5 3 7 6 8 9 I = 22 K zz 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 4 4 2 8 4 2 5 6 8 0 7 6 8 I = 23 K zz 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 5 4 4 4 6 8 7 7 7 3 1 3 2 I = 21 X zz 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U IS 0 ALPHA •- 0 . 4 6 7 4 5 4 7 8 6 2 0 6 7 4 I = 23 K .-— 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 4 8 0 2 8 3 8 1 2 3 4 6 9 8 I = 18 K zz 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR J IS 0 ALPHA •• 0 . 4 8 1 0 5 3 8 4 2 7 4 6 7 1 I = 19 K zz 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR J IS 0 A L P H A = 0 . 4 8 5 8 2 2 8 4 1 1 0 1 0 4 I = 7 K zz 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I ON FOR J I S 0 ALPHA = 0 . 4 8 7 3 8 7 2 3 1 4 3 2 4 5 I = 11 K zz 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 9 0 3 2 2 0 0 2 2 9 0 2 6 I = 17 K zz 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 A L P H A = 0 . 4 9 4 3 8 1 2 8 0 8 1 4 2 2 I = 12 K zz 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA s 0 . 4 9 5 8 4 2 9 1 4 4 0 3 4 4 I = 15 K zz 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 4 9 7 9 8 6 6 2 1 2 7 9 6 8 I = 13 K zz 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 4 9 9 2 7 5 4 6 4 5 3 5 5 0 I = 16 K zz 2 173. N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 A L P H A = 0 . 5 0 2 1 5 6 9 6 8 7 0 0 2 7 I = 1 4 K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 A L P H A = 0 . 5 0 9 7 7 4 2 7 2 8 5 3 9 2 I = 1 0 K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R u I S 0 A L P H A = 0 . 5 2 4 7 8 6 1 8 7 3 1 6 0 2 I = 3 K 2 N U M B E R O F I T E R A T I O N I N S - Q P E R A T I O N F O R J I S 0 A L P H A = 0 . 5 2 6 0 1 8 0 3 8 9 8 3 3 9 I 9 K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 A L P H A = 0 . 5 5 0 4 3 7 7 3 7 5 0 9 3 6 I 6 K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 A L P H A = 0 . 5 5 6 0 0 0 1 0 9 7 1 5 0 0 I 3 K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 A L P H A = 0 . 5 7 4 9 8 9 8 9 3 1 2 6 8 5 I 5 K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 A L P H A = 0 . 6 0 4 6 6 1 8 3 4 1 3 9 0 4 I K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 A L P H A = 0 . 6 6 8 2 8 5 6 8 1 7 2 7 1 7 I : 2 K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 A L P H A = 0 . 7 8 2 1 3 3 4 4 3 8 9 1 9 6 I = 1 K 2 N U M B E R O F I T E R A T I O N I N S - O P E R A T I O N F O R J I S 0 O P T I M A L F O R A L L A L P H A G R E A T E R T H A M T H E C U R R E N T V A L U E E X E C U T I O N T E R M I N A T E D 1 7 : 4 0 : 1 9 T = 1 1 . 0 9 4 R : = 0 $ 3 . 7 9 $ 3 . 8 7 , $ 4 . 7 8 T $ S I G * * * S ) COMPUTATIONAL RESULT OF TWO C R I T E R I O N AUTOMOBILE REPLACEMENT 175 * * * PROBLEM U S I N G CODE L WITH D I S : O U N T RATE OF . 9 7 NUMBER OF I T E R A T I O N I N T - O P E R A T I O N FOR V IS ALPHA = 0 . 0 STATE P O L I C Y 1 41 RETURN 0 . 3 9 2 4 7 1 7 E + 0 4 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 2 3 24 25 2 6 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 - 0 . 4 0 4 4 7 1 7 E * 0 4 - 0 . 4 1 5 4 7 1 7 E + 0 4 - 0 . 4 3 3 1 6 4 2 E * 0 4 - 0 . 4 3 9 8 0 6 & E + 0 4 - 0 . 4 4 6 2 0 2 2 E + 0 4 - 0 . 4 5 2 2 8 8 2 E + 0 4 - 0 . 4 5 8 0 8 8 3 E + 0 4 - 0 . 4 6 3 5 1 9 7 E + 0 4 - Q . 4 6 8 7 6 5 1 E * 0 4 - 0 . 4 7 3 7 7 8 6 E + 0 4 - 0 . 4 7 8 4 7 1 7 E + - 0 4 - 0 . 4 3 2 8 7 2 7 E + 0 4 - 0 . 4 3 7 0 0 7 3 E + 0 4 - 0 . 4 9 0 8 9 8 8 E + 0 4 - 0 . 4 9 4 5 6 8 7 E + 0 4 - 0 . 4 9 7 9 2 8 1 E * 0 4 - 0 . 5 0 1 0 9 3 6 E * 0 4 - 0 . 5 0 4 0 8 0 0 E * 0 4 - 0 . 5 0 S 9 0 Q 8 E * 0 4 - 0 . 5 0 9 5 6 8 6 E + 0 4 - 0 . 5 1 2 0 9 4 4 E + 0 4 - 0 . 5 1 4 4 7 1 4 E + 0 4 - 0 . 5 1 6 6 6 6 1 E + 0 4 - 0 . 5 1 8 5 6 8 6 E + 0 4 - 0 . 5 2 0 1 7 5 2 E * 0 4 O P T . P O L I C Y 14 14 14 27 41 - 0 . 5 2 1 4 7 1 7 E + 0 4 14 28 41 - 0 . 5 2 2 4 7 1 7 E < - 0 4 14 29 41 - Q . 5 2 3 4 7 1 7 E + 0 4 14 30 41 - 0 . 5 2 3 9 7 1 7 E + 0 4 14 31 41 - 0 . 5 2 4 4 7 1 7 E + 0 4 14 32 41 - 0 . 5 2 4 9 7 1 7 E + 0 4 14 33 41 - 0 . 5 2 5 4 7 1 7 E + 0 4 14 34 41 - 0 . 5 2 5 4 7 1 7 E + 0 4 14 35 41 - 0 . 5 2 6 9 7 1 7 E + 0 4 14 36 41 - 0 . 5 2 7 4 7 1 7 E + 0 4 14 3 7 41 - 0 . 5 2 7 9 7 1 7 E * 0 4 14 3 8 41 - 0 . 5 2 8 9 7 1 7 E + 0 4 14 39 41 - 0 . 5 2 9 7 7 1 7 E + 0 4 14 4 0 41 - 0 . 5 3 0 4 7 1 7 E + 0 4 14 01 SCR I T E R A T I O N 0 . 9 7 0 0 0 9 * * * OPTIMAL ACT I O N ( S ) TO BE CHOSEN F 0 ^ THE CURRENT OPTIMAL P 3 L I C Y NUMBER OF I T E R A T I O N IN . S - O P E R A T I O N FOR U IS 34 ALPHA = 0 . 1 4 0 6 6 8 9 4 6 7 5 5 9 4 I = 26 K = 14 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U IS 0 ALPHA = , 0 . 2 0 2 1 5 3 4 8 2 6 5 7 1 8 I = 3 K = 1 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 2 1 2 7 0 9 0 1 9 1 0 0 1 8 I = 2 K = 1 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 2 1 4 6 1 1 4 3 3 2 3 4 6 9 I = 1 K = 1 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR J IS 0 ALPHA = 0 . 2 7 6 3 2 9 7 4 5 8 3 8 9 6 I = 25 K = 14 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 3 9 2 6 8 3 7 8 2 2 3 7 0 7 I = 24 K = 14 NUMBER OF I T E R A T I O N I N S - Q P E R A T I Q N FOR U IS 0 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 24 K = 2 A L P H A = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 25 K = 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 2 6 K = 2 A L P H A = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 27 K = 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 28 K = 2 A L P H A = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 2 9 K = 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 30 K = 2 A L P H A = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 31 K = 2 A L P H A = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 32 K = 2 A L P H A = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I = 33 K = 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I 34 K 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I 3 5 K - 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I 36 K - 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I 37 K - 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I 38 K 2 ALPHA = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I 39 K = 2 A L P H A = 0 . 3 9 8 8 3 7 1 4 5 2 7 1 9 6 I 40 K •= 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 1 7 3 7 3 7 6 8 0 0 0 9 8 I 22 K - 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U IS 0 A L P H A = 0 . 4 1 9 0 3 8 7 7 3 9 0 0 4 6 I 23 K = 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR J IS 0 ALPHA = 0 . 4 3 0 5 6 7 0 9 4 4 7 7 3 9 I 21 K = 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR J IS 0 ALPHA = 0 . 4 4 5 3 9 8 5 0 5 9 9 7 2 7 I 20 K 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 5 9 8 8 9 1 8 5 0 2 0 6 8 I 18 K 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 6 0 7 8 6 8 5 8 1 0 4 9 1 I < 19 K 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR J I S 0 ALPHA = 0 . 4 6 9 1 7 3 5 7 9 9 9 7 8 8 I ' 7 K = 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 6 9 7 8 7 8 9 3 6 2 2 1 1 I 11 K = 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 4 7 2 3 5 1 0 4 3 1 5 3 2 2 I 17 K — 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 4 7 7 0 8 6 6 0 5 3 9 6 4 2 I 12 K = 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR J IS 0 ALPHA = 0 . 4 7 8 3 9 0 2 5 2 7 2 6 8 4 I 15 K - 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 4 8 0 9 6 0 7 6 4 2 4 8 5 1 I = 13 K = 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 4 8 2 0 8 2 2 1 7 0 6 5 3 9 I • 16 K — 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 A L P H A = 0 . 4 8 5 3 7 8 6 1 1 9 5 1 7 4 I = 14 K 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 4 9 5 3 8 2 6 7 4 5 5 8 1 2 I = 10 K = 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 5 1 2 0 0 5 9 3 5 3 9 4 6 7 I = 8 K 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 5 1 3 1 7 5 4 7 1 3 5 4 1 4 I - 9 K 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U I S 0 ALPHA = 0 . 5 4 2 0 0 8 4 5 8 9 1 8 1 5 I = 6 K = 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 5 4 9 2 5 3 1 8 2 4 1 4 4 3 I = 3 K = 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 5 6 8 4 4 1 8 2 4 6 0 4 4 7 I •• = 5 K 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U I S 0 A L P H A = 0 . 5 9 9 3 3 1 3 8 6 5 7 3 8 1 I 4 K = 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 6 6 6 0 9 3 8 6 3 0 9 0 9 7 I •• 2 K 2 NUMBER OF I T E R A T I O N I N S - O P E R A T I O N FOR U IS 0 ALPHA = 0 . 7 8 1 6 4 1 0 4 1 8 5 9 9 6 I ' 1 K = 2 NUMBER OF I T E R A T I O N IN S - O P E R A T I O N FOR U IS 0 177. O P T I M A L FOR ALL ALPHA GREATER THAN THE CURRENT V A L J E E X E C U T I O N T E R M I N A T E D 1 7 : 4 5 : 1 0 T = 1 0 . 4 1 9 RC=0 $ 3 . 5 3 $ 3 . 5 9 , $ 4 . 5 1 T
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- Computational methods for Markov decision problems
Open Collections
UBC Theses and Dissertations
Featured Collection
UBC Theses and Dissertations
Computational methods for Markov decision problems Shin, Moon Chirl 1980
pdf
Page Metadata
Item Metadata
Title | Computational methods for Markov decision problems |
Creator |
Shin, Moon Chirl |
Date Issued | 1980 |
Description | In this thesis we study computational methods for finite discounted Markov decision problems and finite discounted parametric Markov decision problems over an infinite horizon. For the former problem our emphasis is on finding methods to significantly reduce the effort required to determine an optimal policy. We discuss the implementation of Porteus' scalar extrapolation methods in the modified policy iteration algorithm and show that the results using only a final scalar extrapolation will be the same as those obtained by applying scalar extrapolation at each iteration and then using a final scalar extrapolation. Action elimination procedures for policy iteration and modified policy iteration algorithms are presented. The purpose of these techniques is to reduce the size of the action space to be searched in the improvement phase of the algorithm. A method for eliminating non-optimal actions for all subsequent iterations using upper and lower bounds on the optimal expected total discounted return is presented along with procedures for eliminating actions that cannot be part of the policy chosen in the improvement phase of the next iteration. A numerical comparison of these procedures on Howard's automobile replacement problem and on a large randomly generated problem suggests that using modified policy iteration together with one of the single iteration elimination procedures will lead to large savings in the computational time for problems with large state spaces. Modifications of the algorithm to reduce storage space are also discussed. For the finite discounted Markov decision problems in which the reward vector is parameterized by a scalar we present an algorithm to determine the optimal policy for each value of the parameter within an interval. The algorithm is based on using approximations of values to resolve difficulties caused by roundoff error. Also, several action elimination procedures are presented for this problem. Bi-criterion Markov decision problems and Markov decision problems with a single constraint are formulated as parametric Markov decision problems. A numerical comparison of algorithms with and without action elimination procedures is carried out on a two criterion version of Howard's automobile replacement problem. The results suggest that the algorithm with one of the action elimination procedures will lead to efficient solution of this problem. |
Genre |
Thesis/Dissertation |
Type |
Text |
Language | eng |
Date Available | 2010-03-23 |
Provider | Vancouver : University of British Columbia Library |
Rights | For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use. |
DOI | 10.14288/1.0095141 |
URI | http://hdl.handle.net/2429/22353 |
Degree |
Doctor of Philosophy - PhD |
Program |
Business Administration - Management Science |
Affiliation |
Business, Sauder School of Operations and Logistics (OPLOG), Division of |
Degree Grantor | University of British Columbia |
Campus |
UBCV |
Scholarly Level | Graduate |
AggregatedSourceRepository | DSpace |
Download
- Media
- 831-UBC_1980_A1 S46.pdf [ 7.96MB ]
- Metadata
- JSON: 831-1.0095141.json
- JSON-LD: 831-1.0095141-ld.json
- RDF/XML (Pretty): 831-1.0095141-rdf.xml
- RDF/JSON: 831-1.0095141-rdf.json
- Turtle: 831-1.0095141-turtle.txt
- N-Triples: 831-1.0095141-rdf-ntriples.txt
- Original Record: 831-1.0095141-source.json
- Full Text
- 831-1.0095141-fulltext.txt
- Citation
- 831-1.0095141.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0095141/manifest