PROXIMAL NORMAL ANALYSIS IN DYNAMIC OPTIMIZATION By PHILIP DANIEL LOEWEN B.Sc, The University of Alberta, 1981 M.Sc, The University of British Columbia, 1983 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF T H E REQUIREMENTS FOR T H E DEGREE OF DOCTOR OF PHILOSOPHY in T H E FACULTY OF GRADUATE STUDIES Department of Mathematics We accept this thesis as conforming to the required standard T H E UNIVERSITY OF BRITISH COLUMBIA November 1985 © Philip Daniel Loewen, 1985 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. P h i l i p Loewen Department of Mathematics The University of British Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 Date 5 December 1985 Abstract Proximal normal analysis is a relatively new technique whose power and breadth of applicability are only now being realized. Given an optimization problem, there are many ways to define a "value function" which describes the changes in the problem's minimum value as certain parameters are varied. The epigraph of this function, namely the set of points lying on or above its graph, is a set whose geometry is intimately connected both with necessary conditions for optimality in the original problem and with the problem's sensitivity to perturbations. Proximal normal analysis is the geometrical technique which allows such information to be derived from a study of this fundamental set. In the first chapter we illustrate the technique in the simple model framework of a finite-dimensional mathematical programming problem, and describe its consequences for parameter sensitivity in optimal control. Chapter II presents a detailed proof of the fundamental geometric result, called the "proximal normal formula", in Hilbert space. The proof is distilled from the more general work of Borwein and Strojwas (1985), who were the first to make this basic ingredient of the method available in infinite dimensions. This extension is of considerable practical interest: in Chapter III it makes possible a proximal normal analysis of state constraints in optimal control, which gives rise to a new form of the maximum principle for state constrained problems. Limiting techniques and existence theorems are key ingredients in proximal normal analysis. Chapter IV gives a new existence theorem for open-loop stochastic optimal control problems in which compactness of the control set is not required, but instead a growth condition is imposed on the problem's running cost. In addition to their independent interest, the methods and results of Chapter IV enable us to use proximal normal analysis to investigate parameter sensitivity in stochastic optimal control in Chapter V. A byproduct of this analysis is a new proof of the Stochastic Maximum Principle which is more direct (if slightly more technical) than the proofs current in the literature, and which provides a rigorous interpretation of the multipliers. (ii) T a b l e o f C o n t e n t s Abstract ii List of Figures v Acknowledgment vi Chapter I. An Overview of Proximal Normal Analysis 1 Section 1. On Lagrange Multipliers 2 Section 2. Generalized Gradients 4 Section 3. Lagrange Multipliers Revisited 9 Section 4. Parameter Sensitivity in Optimal Control 14 Chapter II. A Proximal Normal Formula in Hilbert Space 23 Section 1. Elementary Geometry of Hilbert Space 24 Section 2. Best Approximation by Closed Sets 26 Section 3. Proximal Normals and Clarke's Normal Cone 29 Section 4. The Proximal Normal Formula 32 Section 5. Proper Points 41 Section 6. Rockafellar's Theorem 43 Section 7. Appendix: Some Euclidean Geometry 47 Chapter III. State Constraints in Optimal Control 49 Section 1. The Value Function 49 Section 2. Proximal Normals 51 Section 3. Convergence 56 Section 4. Constraint Qualifications and Necessary Conditions 59 Section 5. Comparison to Known Conditions 63 Chapter IV. Existence Theory for a Stochastic Bolza Problem 65 Section 1. The Probability Background 66 Section 2. Convergence in Distribution 69 (iii) Section 3. Martingales and Their Representations 71 Section 4. Problem Formulation 77 Section 5. Existence Theory 83 Section 6. Extensions of Theorem 5.1 92 Section 7. A Compact Control Set 96 Appendix. Goor's Existence Theory 99 Chapter V. Parameter Sensitivity in Stochastic Optimal Control 101 Section 1. Perturbed Dynamics 101 Section 2. The Cost Functional 109 Section 3. Necessary Conditions for an Unconstrained Stochastic Control Problem . . . 112 Section 4. Constraints and the Value Function 117 Section 5. The Stochastic Maximum Principle for Constrained Problems 135 References 141 (iv) List of Figures Figure 0. The "image" of problem (1.1) in R 2 2 Figure 1. The sets E and B 38 Figure 2. Lemma 4.2 38 Figure 3. Lemma 4.3 39 Figure 4. Lemma 4.4 39 Figure 5. Lemma 4.5 40 Figure 6. Lemma 4.6 40 (v) Acknowledgement It gives me great pleasure to thank Frank Clarke for serving as my research supervisor. Professor Clarke's optimism and love for mathematics are infectious; his breadth of knowledge and seemingly infallible intuition nothing less than awe-inspiring. Besides encouraging me implicitly by his example, Frank also spent countless hours discussing mathematics with me, and made enough suggestions to keep me busy for the foreseeable future. Among the many people to whom this project owes its completion and I owe my thanks, he deserves the first position. I must also thank Professor Perkins for teaching me probability and for many helpful suggestions about Chapter IV. Professor Haussmann introduced me to the field of dynamic optimization in my first year of graduate studies, for which I will always be grateful. He also inspired Chapters IV and V during a 1984-85 course on the Stochastic Maximum Principle and took care of many administrative matters relating to my graduation. David Hare also helped me overcome the logistic difficulties inherent in being a student three thousand miles removed from his University. The McGill School of Computer Science generously allowed me to use TeX on its UNIX system to produce the final version of the thesis. The support of NSERC through a 1967 Science Scholarship has also been indispensable. Thank you all. Finally, I thank my wife Kimberley. Her periodic exhortations to take heart and work had great effect, being spoken in love. This work is respectfully dedicated to Kimberley Loewen. (vi) Chapter I. An Overview of Proximal Normal Analysis Proximal normal analysis is a young technique which is quickly establishing itself as an indispensable tool in the study of optimization. This thesis is an attempt to explain and add momentum to this process. The body of the work stands on two legs: exposition and research. First, Chapter I introduces proximal normal analysis in the model framework of finite-dimensional mathematical programming. This simple context clarifies the general form of the method and suggests the sort of results it can be expected to yield in more general settings. Chapter I continues with a review of parameter sensitivity in optimal control, one of the earliest triumphs of proximal normal analysis in dynamic optimization. After gathering the necessary technical equipment in Chapter II, most notably an infinite-dimensional version of the crucial "proximal normal formula", we go on to demonstrate two new applications of proximal normal analysis in the field of dynamic optimization. In the first of these (Chapter III) we show how it allows the derivation of a new form of the maximum principle for deterministic problems with state constraints. In the second (Chapter V) we apply it to stochastic control problems to obtain a new proof of the Stochastic Maximum Principle which affords a precise interpretation of the multipliers. Chapter IV contains an existence theorem for a stochastic optimal control problem in Bolza form which is used in Chapter V, but is also of independent interest. In each case, proximal normal analysis reveals an interplay between value functions, geometrical objects, and necessary conditions which is deep enough to yield impressive new results and wide enough to suggest many other areas for fruitful investigation. - 1 -S e c t i o n 1 . O n L a g r a n g e M u l t i p l i e r s Consider the optimization problem (1.1) min {l(x) : g{x) = 0} , where I and g are smooth real-valued functions defined on a Banach space X. No matter how large the space X may be, this problem has a natural "image" in R2, namely the set (1.2) C = {(g(x),£(x)) : x £ X) In the example illustrated in Figure 0 below, C is the lightly drawn curve. ( V ' ( O ) . - l ) Fig. 0. The "image" of problem (1.1) in R2. The minimization problem above can now be viewed as a two-step process: first find the lowest point on the y-axis lying in C, and then find an element of X realizing this image. If we were to seek the lowest points on other vertical lines, the following value function V:R—+ Ru {+00} would emerge: V{a) = min {v : (a, v) £ C} = min {£(x) : g[x) = a} . - 2 -In Fig. 0, the heavy line traces the graph of V. Note that in spite of the smoothness of g and £, the function V may easily be discontinuous or even take the value + 0 0 . Without more hypotheses, we can only reasonably expect V to be lower semicontinuous. But suppose for the moment that V is finite-valued and differentiable at a = 0, with V(0) = £(x) for some x € X. Then the point (V(0) , — l) is the downward normal to the graph of V at the point (0, V(0)). If V is convex near 0, as illustrated in Fig. 0, it follows that one has (1.3) ((V'(0),-l),(a,V(a)) - (0,V(0)))<0 for all a near 0. In particular, for all x near x, one has g(x) near g(x) = 0, so ((V'(0),-l),(g(x),l(x)) - (g(x),t(x))) < 0 (1.4) ^ t{x) - V'{0)g{x) < t{x) - V{0)g{x). Hence the right side has a local minimum over X at x, which forces its derivative to vanish: (1.5) Dt{x) -V'{0)Dg(x) = 0. (Here Dt is the Frechet derivative of t.) This is the familiar Lagrange multiplier rule for constrained minimization problems, with the "multiplier" being — V'(0). Thus in the usual Lagrange multiplier rule one may interpret the multiplier corresponding to a given solution as an index of the marginal effects of perturbations of the corresponding constraint. This simple introduction demonstrates three things: 1. Differential analysis of the value function can be used to prove multiplier rules. The analysis proceeds by applying known necessary conditions for an unconstrained problem to a "perpendicular inequality" like (1.3). 2. Conversely, known multiplier rules involve constituents which may have an interpretation as the marginal value associated with some appropriate perturbation. 3. Even for problems with smooth data, the value function can behave very badly. The promising tone of articles 1 and 2 is in sharp contrast with the sad facts of life set down in article 3: differential analysis of V may indeed be profitable, but is often impossible. Besides the cowardly option of simply giving up in despair, there are two ways to deal with this conflict. The first is to impose a sufficient number of additional conditions (linearity, convexity, etc.) to guarantee - 3 -that V is smooth . Such condit ions often disa l low the s tudy of m a n y problems of subs tant ia l p r a c t i c a l interest . T h e second poss ib i l i ty is to broaden the scope of the phrase "di f ferent ia l analys i s " i n some way w h i c h allows i ts app l i ca t ion to i l l -behaved funct ions . T h i s has been a t tempted by m a n y authors for m a n y reasons, but the "general ized gradient" discovered by F r a n k H . C l a r k e i n 1973 has proven especial ly useful i n the s tudy of d y n a m i c o p t i m i z a t i o n . W e now review those aspects of Clarke ' s theory most essential to the s tudy of p r o x i m a l n o r m a l analysis . S e c t i o n 2. G e n e r a l i z e d G r a d i e n t s Let X be a B a n a c h space, and let f:X —• E u { + 0 0 } be an extended-real-valued funct ion on X. Suppose / is lower semicontinuous , and that x € X is a point where f(x) is finite. It is quite possible that / may fa i l to be d i f f e ren t i a t e at x: one way to exp la in this s i tua t ion is that there is no single po int Df(x) €E X* w h i c h adequately summarizes the l o c a l behav iour of / near x. Of ten there are several points i n X * w h i c h each give p a r t i a l in format ion about f's l o c a l behav iour . C larke ' s generalized gradient of f at x, denoted df{x), is a weak*-closed convex subset of X* w h i c h contains a l l such points . In this section we discuss the precise def in i t ion and propert ies of the set-valued opera t ion df(). A l l the definit ions and theorems are taken f r o m C l a r k e (1983). L i p s c h i t z F u n c t i o n s . L e t us begin by consider ing the case w h e n / is Lipschitz near x, that is, w h e n there is a constant K and a ne ighbourhood U of x such tha t l/(y)-/(*)l<*l|y-*ll Vx.yetf. In this case, the fo l lowing quant i ty is finite for any t; e X: 0/~ » , : f(x + hv)-f{x) (2.1) f (x;v) = l i m s u p ac—*x hlO T h i s quant i ty is cal led the generalized directional derivative of f at x in direction v. A s a funct ion of v, i t is pos i t ive ly homogeneous and subaddit ive , so we may define (2.2) df(x) = {S e X * : x h hiO a n d convergence in uniform on every compact set of v-values. In Clarke's terminology, f is strictly differentiable at x, with Dtf(x) = $. T h e u t i l i t y o f g e n e r a l i z e d g r a d i e n t s is b a s e d n o t o n l y u p o n t h e e v i d e n t s i m i l a r i t y b e t w e e n t h e i r d e f i n i t i o n s a n d t h o s e o f t h e c l a s s i c a l c o n c e p t s , b u t a lso b e c a u s e t h i s s i m i l a r i t y is p r e s e r v e d i n m a n y o f t h e r u l e s o f c a l c u l u s . F o r e x a m p l e , t h e c l a s s i c a l r u l e s c o n c e r n i n g t h e d e r i v a t i v e o f a s u m o f t w o s m o o t h f u n c t i o n s , o r o f a s c a l a r m u l t i p l e o f a s m o o t h f u n c t i o n , h a v e n a t u r a l c o u n t e r p a r t s i n t e r m s o f g e n e r a l i z e d g r a d i e n t s . M o r e o v e r , d i f f e r e n t i a t i o n r u l e s f o r c e r t a i n n o n c l a s s i c a l m e a n s o f c o m b i n i n g f u n c t i o n s c a n be e a s i l y e x p r e s s e d i n t e r m s o f g e n e r a l i z e d g r a d i e n t s . 2.2 P r o p o s i t i o n . Let fi,..., fk- X —» R be functions Lipschitz near x. (a) For any scalars ci and c^, d ( c i / i + c 2 / 2 ) ( x ) C c i d / i ( x ) + c 2 3 / 2 ( x ) . Equality holds if c 2 = 0 , o r i f / 2 is continuously differentiable near x. (b) 3 ( A / 2 ) ( S ) C A ( 2 ) 3 / 3 ( Z ) + / 2 ( x ) 3 A ( x ) . (c) The minimum function V(x) : = m i n { / i ( x ) : i = 1 , . . . , k} is Lipschitz near x and dV{x) C c o U { 3 / i ( x ) : t€ !(£)}, where I(x) := a r g m i n { / , ( x ) : i = 1 , . . . , k} is the set of indices at which the minimum defining V is attained. (d) If f\ takes on a local minimum value at x, then 0 € dfi[x). - 5 -Part (c) of Prop. 2.2 is especially suggestive in the present context because it can be viewed as a very simple instance of our general problem, namely the differential analysis of a certain minimum-value function V as perturbations x displace the data of the associated minimization problem. The problem in (c) is admittedly elementary—one is simply required to choose the smallest of A; scalars—but the issues noted in Section 1 are already perceptible. For instance, the function V may well fail to be smooth even when each is linear, so classical differential analysis of V is impossible. However, an elegant and illuminating description of the local behaviour of V is available in terms of generalized gradients. At the risk of some oversimplification, the goal of this thesis may be summarized as an attempt to find calculus results analogous to (c) in which the minimum defining V is taken over more general sets than {l, 2,..., fc}. Of course, Clarke's conception of the generalized gradient as a set-valued mapping instead of a point-valued one raises issues with no obvious precursors in classical analysis. For instance, one must consider the closure properties of the mapping df(-). 2.3 Proposition. Let x,- and be sequences in X and X* such that € df(x{) Vi. Suppose that X{ converges to x and that f is a weak*-cluster point of {&}• Then f~€E df(x). In our discussion of Lagrange multipliers in Section 1, the geometrical significance of the classical derivative was prominent, as it led to line (1.3). Generalized gradients also have their geometrical side, which is best understood in terms of the "epigraph" of the function /. The epigraph of f is the set of points (x, r) in X x R lying on or above the graph of /: epi / := {(x, r) 6 X x R : r > /(x)} . Let us first consider, for fixed x, the set (2.3) epi/°(x;) = { ( ¥ ) e X x R : r > / ° ( x » } . Since /°(x;) is subline ar, epi/°(x;) is a closed convex cone with vertex at 0. This cone is the (Clarke) tangent cone to epi/ at (x,/(x)). In the case of a smooth function /:Rn —• R, f°[x;v) = (Df[x),v) is a linear function of v and its epigraph, which we are calling T e p i / ( x , f(x)), is the half-space lying above the hyperplane through (x,/(x)) and supporting the graph of /. Thus - 6 -the Clarke tangent cone corresponds to our usual notion of tangency in the classical case. Moreover the (Clarke) normal cone to e p i / at (x,/(x)), defined by (2.4) JV e p i /(£,/(x)) = { ( f , / 9 ) e X * x R : ((?,/?), (v, r)) < 0 V(v, r) € T e p i /(x,/(*))} is a closed convex cone with vertex at 0 obeying the rule (2.5) ?ea/(x) {c,-l)eNepif(x,f(x)). This corresponds to the well-known classical result that if / is C 1 near x, then the vector (Df(x), — 1) is normal to the graph of / at (x, /(x)). Non-Lipschitz Functions. The Clarke tangent cone can be defined in considerably more generality than the previous paragraph suggests. Indeed, generalized notions of tangency and normality are the foundations of a whole theory of "nonsmooth analysis", a structure whose keystone is the generalized gradient. These foundations are also basic features of this thesis, and we have devoted all of Chapter II to their study. Let us therefore be content with a sketch of their significance. For any closed set C Q X and any point c in C, it is possible to define the tangent cone to C at c, denoted Tc(c), in a way consistent with that cited above when C = epi / for / Lipschitz. (This is done in Definition II.3.1 below.) The normal cone to C at c is then given by (compare line (2.4)) Nc(c) = {c e X* : (<,v) < 0 V v £ Tc(c)}. By analogy with line (2.5), it is now possible to define a set df(x) via (2.6) df(x) = {c&X* : (c,-l)eNepif(x,f(x))}. This set may also be called the generalized gradient of / at x since it agrees with the previous definition when / is Lipschitz near x. However, the definition in line (2.6) makes sense for any function / for which epi / is locally closed near (x,/(x)). In particular, the generalized gradient is thus defined for any lower semicontinuous function /: X —» R U { + 0 0 } and point x where /(x) < + 0 0 . However, in this broad class of functions the generalized gradient is known only to be a weak*-closed set (perhaps not weak*-compact); furthermore, it may be empty. To obtain 3/(x) = 0, it is necessary - 7 -that each vector in JV e pi/(x, /(£)) have zero in its second component. The local information about / near x is then concentrated in the asymptotic generalized gradient of f at x, namely (2.7) d°°f{x) = U : (?,0) € JV.pi/(2,/(»))}. The asymptotic generalized gradient is always a closed convex cone containing 0, which can be intuitively understood as the set of directions in which / is particularly badly-behaved (i.e., nonLipschitz). Together, df[x) and d°°f(x) contain all the local information inherent in Nepif{x,f{x)): (2.8) Nepif(x,f(x)) = {\{?,-1) : A > 0, ? € 8f(x)} U {(f,0) : fGc?°°/(*)}. The Finite-Dimensional Case. When X = R n, the following desirable results become available. 2.4 Proposition. Let /:R n —* R U { + 0 0 } and a point x where f(x) < +00 be given. If epi / is locally closed near (z, f(x)), then one has Nepif(x, f[x)) ^ {0}. In particular, 5/(x)u(a°°/(x)\{0}) ^ 0 . 2.5 Proposition. Let f: R r e —* R U { + 0 0 } and x € R n be given. If f(x) < + 0 0 and epi / is locally closed near (x,f(x)), then the following are equivalent: (a) df(x) is a bounded nonempty set; (b) 3°°/(x) = {0},-(c) f is Lipschitz near x. An invaluable tool in actually computing df(x) for a function / satisfying the conditions of Prop. 2.5 is the "proximal normal formula", which indicates how to calculate JV e pi/(x, /(£))• For any closed set C C R" and point 0 : Vi ± C at Ci —• ?, v< —+ 0 Chapter II proves a version of this formula valid in any Hilbert space, and provides more details concerning the general theory of normal and tangent cones. Before beginning that study, however, let us reconsider the problem treated informally in Section 1. In Section 1 we suggested that differential analysis of a certain value function was the key to both the proof of a Lagrange multiplier rule and the interpretation of the multipliers, and moreover that Clarke's nonsmooth analysis allowed such progress despite the inapplicability of classical calculus. This section justifies that claim, relying to a certain extent on Clarke (1983), Section 6.5, and on Rockafellar (1982). We consider locally Lipschitz functions I: R n —• R, g:Hn —* Ra, and define The minimization problem defining V(a) is called P[ot): we say that x solves P[a) if g(x) + ct = 0 and t{x) = V{a). Let us make the following hypothesis throughout this section: (H) V(0) < +oo, and for some e > 0 the level set K = {x € R n : £{x) < V(0) + e} is compact. Hypothesis (H) guarantees that a satisfactory existence theory can be developed for this family of problems, and is crucial to the convergence arguments of Prop. 3.3 below. 3.1 Lemma, (a) For any a such that V(ct) < V(0) + e, problem P(ct) has a solution lying in int K. (b) The set epiV is locally closed near (0, V(0)). Proof, (a) If V(a) < V(0) + e, then no generality is lost in restricting the set of admissible values of i to a compact subset K' of int K = {x € R n : l[x) < V(0) + e}. Let i< be a minimizing sequence in K'. Then by passing to a subsequence if necessary, one has Xi —* x for some x S K', and also Section 3. Lagrange Multipliers Revisited V:Ra — Ru{+oo} by V(a) := min{£(i) : x € Rn, g{x) + a = 0}. g(xi) + a = 0, Vi. - 9 -In the limit as i —* oo we find that g(x) + a = 0 and £{x) < V(a). Hence x £ K' solves P{a). (b) Let (a,,«j) be a sequence in epiV converging to a point (ct,v) with v < V(0) + e. Then without loss of generality we may assume Vi ) - g(x)\2 + \ |£(x') - £(x)|2 . That is, the locally Lipschitz function of x' on the right side has a global minimum at x' = x. By Prop. 2.2(a)(d), it follows that oed[xt() + (p,g())](x). Dividing both sides by |(/3, — A)| ^ 0 gives the result. //// - 10 -T h e n e x t s t e p is t o c o n s i d e r a l i m i t o f n o r m a l i z e d p e r p e n d i c u l a r s . 3 . 3 P r o p o s i t i o n . If ( f t — A ) = l i m ( f t , — A , ) f o r a sequence of perpendiculars ( f t , — A , ) ± e p i V a t (a,-,tij) —+ (0,V(0)), then P{0) has a solution x for which 0 € S [ A £ ( ) +(ft 0 . Proof. A s s u m e f i r s t t h a t Y = {x}. B y P r o p . 2 .4 , iV " e p i v ( 0 , V ( 0 ) ) ^ { 0 } . H e n c e t h e p r o x i m a l n o r m a l f o r m u l a ( 2 . 1 0 ) a s s e r t s t h a t s o m e s e q u e n c e o f n o r m a l i z e d p e r p e n d i c u l a r s h a s a n o n z e r o l i m i t . T h i s l i m i t m u s t o b e y t h e c o n c l u s i o n s o f P r o p . 3 .3 . I f A = 0 i n t h o s e c o n c l u s i o n s , t h e s t a t e m e n t o f o u r t h e o r e m h o l d s w i t h ( f t - A ) = ( f t 0 ) . I f A > 0 , t h e n o u r s t a t e m e n t h o l d s w i t h ( f t - A ) = ( / 0/A, - l ) . I f t h e se t Y c o n t a i n s m o r e t h a n o n e p o i n t , fix a n y x € Y a n d r e p e a t t h e d e v e l o p m e n t a b o v e w i t h £ r e p l a c e d b y £ ( x ) = £ ( x ) + |x — x | 2 . H y p o t h e s i s ( H ) r e m a i n s v a l i d f o r t h i s n e w p r o b l e m , w h i c h h a s the unique solution x. It follows from the previous paragraph that there exists (/9, —A) £ R a x R such that A €{0,1}, o e a ( A £ ( ) + A | ( ) - x | 2 + (/3,a()))(z), A + |/?| > 0. Since the second term in the sum whose generalized gradient is computed above is smooth, with derivative 0 at x, the desired conclusion follows from Prop. 2.2(a). //// The development leading to Thm. 3.4 shows how proximal normal analysis permits the derivation of a Lagrange multiplier rule. More specifically, it allows necessary conditions for a constrained problem to be derived from known necessary conditions for an unconstrained problem. (See the proof of Prop. 3.2.) This proof of the Lagrange multiplier rule is more elementary than that given by Clarke (1983), Thm. 6.1.1, p. 228, which relies on Ekeland's theorem. But in addition to its debatable aesthetic advantages, Thm. 3.4 has two unquestionable merits. First, if the functions t and g are smooth, the auxiliary problem leading to Prop. 3.2 is also smooth. In fact, the whole proof of the Lagrange multiplier rule then requires no nonsmooth analysis at all, except for the definition of iV epiv (0, V(0)) which suggests the appropriate auxiliary problem. This is in contrast with the proof based on Ekeland's theorem, in which nonsmooth terms enter the auxiliary problems even for smooth data. (Another general technique for proving necessary conditions, called "exact penalization", also introduces nonsmooth auxiliary elements into problems whose data are smooth. Chapter III below shows that by avoiding this, proximal normal analysis allows a significant improvement in the necessary conditions for optimal control problems with smooth state constraints.) The second main advantage of the current proof is that it adds rigor to the traditional interpretation of Lagrange multipliers as marginal values. This is the content of Thm. 3.6 below, which can be viewed as a precise version of the results suggested in Section 1. (Theorem 3.6 is due to Rockafellar (1982); the development in this section follows Clarke (1983), Section 6.5.) 3.5 Definition. Let x € R n and A > 0. A vector 0 G Ra is an index A multiplier corresponding to x if o e d [ A £ ( ) + ( £ , < , ( • ) ) ] ( * ) . - 12 -The set of all such vectors is denoted Mx (x); we also write MX(Y) = [J Mx(x). 3.6 Theorem. Assume (H). Then Y / 0, and one has 57(0) = co[M 1{Y) n 57(0) + M°(y) n 5°°7(0)]. IfM°(Y) = {0}, then 5°°V(0) = {0} and the previous equation becomes dV{0) = co [M 1{Y) D 57(0)]. Proof. To prove the first statement, we will apply Prop. II.6.2 with D = M1(Y) n 57(0) and D°° — M°(Y) n 5°°7(0). It therefore suffices to show that Nepi v (0, V (0)) = co[iV U iV°°], where N = {\{c, -1) : f e M X(F) n 57(0), A > 0} N°° = { (j,0) : f£ Af^yjnS 0 0^^)}. The inclusion Nepiv(0,V(0)) D co[/Y U 7Y°°] is an obvious consequence of the definitions of 57(0) and 5°°7(0) (lines (2.6) and (2.7) above). To see the reverse inclusion, note that by the proximal normal formula (2.10), JVepi v (0,7 (0)) is the closed convex cone generated by certain limits of perpendiculars. If (/9,—A) is such a limit with A = 0, then 0 €E M°(Y) by Prop. 3.3, while 0,0) e NepiV(0,7(0)) by construction implies J9 6 5°°7(0). Hence 0,0) e N°°. And if 0, — A) is such a limit with A > 0, then 0 = 0/X e M1{Y) by Prop. 3.3, while (/?,-1) € NepiV(0,7(0)) by construction implies /9 £ 5°°7(0). Hence A(ft —1) € N. Combining these two possibilities gives JVepiv(0,V(0)) Cco[NuN°°], as required. To prove the second statement we note that if M°(Y) = {0} then the cone D°° = {0} is certainly pointed. The result would then follow from Prop. II.6.5 if we could prove D°° ~D 0+ D. (This notation is introduced in Lemma II.6.3.) It is quite easy to show that M°(Y) 2 0 +M 1(F), while 5°°7(0) 2 0+57(0) for any function 7 is a well-known result of nonsmooth analysis. Combining these two statements justifies the application of Prop. II.6.5. //// - 13 -Corollary 1. If M° (Y) = {0} and M1{Y) is the singleton {£}, then V is strictly differentiable at 0 and D,V{0) = p. Proof. By Thm. 3.6, M°(Y) = {0} implies d°°V(0) = {0}. Consequently V is Lipschitz near 0 by Prop. 2.5. In fact, Thm. 3.6 implies that 3V(0) = {£}, so the result follows from Prop. 2.1. //// Among the interesting byproducts of the first-order information provided by Thm. 3.6 is the following sufficient condition for local surjectivity of the mapping g:H.n —• R". In the smooth case, it reduces to the well-known Surjective Mapping Theorem, which states that if the a x n matrix Dg(x) has rank o, then g is locally surjective near x. Corollary 2. Suppose M°(x) = {0} for some x S B/1. Then there exist positive constants r\ and M such that for all a € r\B, there exists some y e R" obeying g{y) + a = 0 and \y — x\< M |a|. Proof. Define £(y) = \y — x\ and consider the value function V(a) := min {£(y) : g[y) + a = 0} . The unique solution to P(0) is x, and M°(x) = {0} implies d°°V(0) = {0} by Thm. 3.6. According to Prop. 2.5, V is Lipschitz near 0. That is, there exists M > 0 and r) e (0,e/M) such that V{a) < V(0) + M\a\ Va e r)B. Noting that V(0) = 0 and applying Lemma 3.1(a) gives the desired result. //// A result like Corollary 2 is of interest in optimization theory because it says something about the stability of the set of feasible points under perturbations of the data. Clarke (1983), Section 6.6 proves a more general result involving inequality constraints and an abstract constraint as well as equality constraints. - 14 -Section 4. Parameter Sensitivity in Optimal Control The previous section showed how proximal normal analysis allows both the derivation of necessary conditions and a sensitivity analysis for a simple problem. In this section we review some results of Loewen (1983) which show that even when necessary conditions are known in advance, proximal normal analysis can be used to interpret them and perform an independent analysis of parameter sensitivity. The first-order dependence of a problem's minimum value on various parameters is of considerable interest in its own right. It finds theoretical application in dynamic programming and the Hamilton-Jacobi equation, for example, and also has consequences for such practical issues as controllability. There is also its obvious utility in identifying which parameters have the greatest effect on the problem's value. In linear programming, the latter application is sufficient to explain "why the dual vector is cherished by oil company vice-presidents and not just by mathematicians." (Franklin (1980).) The Problem. In this section we study perturbations of the following differential inclusion problem: (P) min{£(z(0),i(T)) : x{t) e F{t,x[t)) a.e. \0,T], (i(0),x[T)) E S} . The objective of problem (P) is to choose an arc (i.e., an absolutely continuous function) x: [0, T] —• R n satisfying the dynamic constraint (4.1) i(()ef(l,i(()) a.e. [0, T] and the endpoint constraint (x{0),x[T)) e S while minimizing £ over all such arcs. Line (4.1) is a differential inclusion phrased in terms of a set-valued mapping, or multifunction, F: [0, T] xR" —* Rn. An arc x obeying (4.1) is called an F-trajectory; if it also obeys (i(0), z(T')) € S, then x is called an admissible F-trajectory. A special case of problem (P) is the Mayer problem arising when one is given a control set U C R m and a function /: [0,T] xR"x R m -* R n and instructed to solve (P) with the modified dynamics (4.2) x(t) = f(t, x{t), u(t)) a.e. [0, T] for some measurable u: [0, T] -» U. - 15 -Filippov's lemma asserts that if we take F{t, x) := f(t,x,U), then any F-trajectory actually obeys (4.2). The converse is clear, so the admissible arcs for the Mayer problem built around (4.2) are the same as those for (P). Under the following hypotheses, necessary conditions for problem (P) are known. Throughout this section, B denotes the open unit ball in R n. (hi) The multifunction F: [0, T] X R n —• R n has nonempty compact convex values. For each fixed x S R n, F(-,x) is measurable on [0, T\. That is, for any closed set C C R n, the following "inverse image" is a Lebesgue measurable set: {t G [0, T] : F(t, x) n C ^ 0} . (h2) There is a function k(t) £ /^[O, T] such that (a) F[t, x) C k(t)B Vi e [0,T], x e R", (b) for each fixed t e [0, T] and x e R n, one has F{t,y)CF{t,x)+k{t)\y-x\B Vy € R n. We define KF = exp ( f Q r k{t) dt}. (h3) The constraint set S C R" x R n is closed, and {x : (x, y) € S} is compact. (h4) The objective function V. R n x R" -» R is Lipschitz of rank on R " x R " . Note that the convexity hypothesis in (hi) means that we are working with the "relaxed problem" in the sense of Warga (1972). The necessary conditions are phrased in terms of the Hamiltonian H: [0, T\ x R n x R r a —• R and the distance {unction ds:R 2 r l —• R defined by H{t,x,p) :=Bup{{p,v) : v e F{t,x)} ds{v) := inf{|v - s\ : a € 5} 4.1 Theorem. Assume (hl)-(h4), and fix any r > (2Kt + 2)(1 + KF lni£». If the arc x() solves (P), then there exist a scalar fj, and an arc p: [0, T] —> R", not both zero, such that (a) M e {0,1}, (b) (-i>[t),x{t))edH{t,x(t),p{t)) a.e. [0, T], (c) (p(0), -p(T)) e /if + r\{fi, E)\dds(x(0),x(T)) for some e € dl[x{0),x{T)). Here dH signiSes the generalized gradient of H in the (x, p) variable only, and E := /xf + (—p(0), p(T)). Proof. This follows easily from Clarke (1983), Thm. 3.5.2, p. 147. //// - 16 -A full discussion of the relationship between Thm. 4.1 and Pontryagin's maximum principle is given by Clarke (1983). Note that since Ns[s) = |J rdds{a) (Clarke (1983), Prop. 2.4.2, p. 51), condition (c) implies r>0 the simpler form (c') (p(0),-p(T)) € n$+Ns{x(0),x{T)) for some f € dt{x{0),x{T)). Conclusion (c') is only slightly weaker than (c) for large values of r, and has the advantage of containing no explicit r-dependence. Hypotheses (hl)-(h4) specifying (P) restrict the application of Thm. 4.1 to global solutions z ( ) . This is not an essential restriction. In fact, for any S > 0, the result remains valid whenever the arc x{) only solves (P) relative to all feasible arcs y() obeying the relationship \\y — xW^ < S. This is equivalent to the requirement that the graph of y be contained in the set Ts(x):={(t,y)G[0,T\xnn : \y - x(t)\ < 6} , called the tube of radius S about x. The key point here is that for the purposes of Thm. 4.1, hypotheses (hl)-(h4) need only hold within Ts{x). This observation is important in Chapter III, where we will use it to reduce the size of the constants Kp and Kt once a solution is known. Perturbations. Let us now consider the effect of a finite-dimensional parameter a in R a on problem (P). We define a family of problems P(a) by P(a) min{£(x(0),x(r),a) : x{t) € F(t, x{t), a) a.e. [0,T], (x(0), x(T), a) e S} . The value function V : R a —• R U { + 0 0 } is then given by V(a) := inf P(ct). Here we intend to explain (omitting most proofs, which are given in Loewen (1983)) how proximal normal analysis leads to a valuable formula for 3V(0). First, however, we must place some mild restrictions on the a-dependence of the data defining P(a). These hypotheses also involve the set Y, which consists of all arcs solving the nominal problem P(0). (HI) The multifunction F: [0, T\ X R n x R a —• R n has nonempty compact convex values. For each fixed {x,a) € R" x R a, F(-,x,a) is measurable on [0,T]. (H2) There is a function k(t) e L 1^, T] such that - 17 -(a) F{t, x, a) C k(t)B V(t, x, a) £ [0,T) x R" x R°, (b) for each fixed t £ [0, T] and (x, a) £ R" x Ra, one has F(t, y, p) C f (t, x, a) + fc(t) |(y, /?) - (x, a)\ B V(y, p) £ R" x R*. (H3) The set 5 C R" x R n X R" is closed and {x : (x, y, a) £ S) is compact. Moreover, for any x() £ y, the multifunction Ns() is closed at (x(0), x(T), 0). (H4) The objective function £: R" X R" X R a-» R is globally Lipschitz of rank Kt. The closure hypothesis of (H3) requires that if (si, vi) is a sequence converging to (s,v) and obeying Vi £ Ns(si) Vi, then one has v £ Ns(s) at least whenever the limit point s has the form (x(0), x(T), 0) for some x £ F. This mild assumption holds automatically if the cone iVs(x(0), x(T),0) is pointed. (A set is pointed if zero cannot be obtained as a positive linear combination of its nonzero elements.) A detailed discussion of other conditions ensuring the validity of (H3) is given by Loewen (1983). 4.2 Lemma, (aj IfV(a) < +oo then problem P(a) has a solution. Just as in Section 3, consideration of a single perpendicular vector leads to a certain auxiliary problem which can be solved by known methods. Since the auxiliary problem depends on a, we must introduce a more general Hamiltonian via 4.3 Proposition. Let (/?, —A) ± epiV at (2,u). Then P(a) has a solution x() to which there corresponds an arc (p, q): [0, T] —» R" X R a and a constant p. such that (b) The function V is lower semicontinuous near 0. Proof. See Loewen (1983). / / / / X(t,x,a,p) :=sup{(p,u) : v e F(t,x,a)}. (a) A>0, /x€{0,l>, ft.+ \\{p,q)\\ > 0, (b) {-p{t),-q{t),x{t)) € dX{t,x{t),a,p{t)) a.e. [0,T], (c) (p(0),-p(T),-q (T)) £ \n$ + Ns(x(0),x(r),S) forsome?£3£(x(0),x(T),a). (d) - 9(0) = nP. - 18 -Proof. Since V(2) < v < +00, Lemma 4.2(a) ensures that P(a) has a solution x. For any a S R a and any F-trajectory x() admissible for P(a), one has V(a) < £(x(0),x{T),a) < £(x(0),x(T),a)+v- £(x(0),x(T),a), so (a,£(x(0),x(T),a) - £(x(0), x[T), 2) + v) lies in epiV. From inequality (2.9), it follows that A£(x(0), x(T), 2 ) - (/?, 2) < A£(x(0), x(T),a)-(£, a) + ± |(a - a, £(x(0), x(T), a) - £(x(0), x(T),2)) |2 Since equality holds when x = x and a = 2, we find that the arc (x(), 2) provides a global solution for an unperturbed differential inclusion problem whose data are 2 k *i, y, y i ) = A£(x, y, y i) - ,0)) : (t>, 0) e x, xx)} = ^(^X.Xi.p). Applying Thm. 4.1, we find that there is an arc (p, q): [0, T] —• R™ X R a and a scalar ft such that (a) (ie{0,l})ri + \\(p,q)\\>0. (b) (-p(t), -g(t),£(t),0) 6 dH(t,x{t),a,p{t),q(t)) a.e. [0,T]. This implies (-p(t),-^),J(())e3)/(t,i(t),S,P(t)) a.e. [0,T]. (c') (p(0), g(0), -p(IT), -q{T)) € M [A(£, 0, r,, m) - (0, 0,0)] + %(x(0), 2, x(T), 3) for some ( e ^ ^ O e ^ o ) , ^ ) , ^ ) . Condition (c') reduces to the pair of conditions (p{0),-p(T),-q{T)) e/xA0, |(ft-A)| = l , (b) (-p(t),-ff(t),±(t)) edM{t,x(t),0,p{t)) a.e. [0,T], (c) (p(0),-p(T),-q(T)) e\c + Ns(x(0),x{T),0) for some f € di(x{0),x{T),0), (d) -9(0) - ft Proof. See Loewen (1983), pp. 34-36. //// The appropriate definition of the multiplier sets is now clear. - 20 -4.5 Definition. Let x solve P(0). An arc (p, g):[0, T] —+ R" X R a is an index A multiplier corresponding to x if it satisfies conditions (b) and (c) of Prop. 4.4. The set of all such arcs is denoted Mx(x), and we define M'(Y)= \jM"(x). x€Y We also define a mapping A from the space of multipliers to R a as follows: A(p,?) = -g(0). The desired result describing the first-order dependence of V on a near 0 is the following analogue of Thm. 3.6. 4.6 Theorem. Assume (Hl)~(H5) and suppose V(0) < +oo. Then Y ^ 0, and one has dV[o) = CO^A[M 1 (F)] ndV(o) + A[M°{Y)] na°°v(o)). If A [ M 0 ( Y ) ] is pointed, then the closure operation is superfluous and one also has a°°v(o) = C O ( A [ M ° ( Y ) ] na°°v(o)). Proof. Similar to Thm. 3.6. //// Although sensitivity analysis was our primary objective, a new multiplier rule for perturbed problems is a byproduct of this investigation. C o r o l l a r y 1. If x solves P(0) then it has an index A multiplier (p, q) for some A > 0, with X+\q(0)\>0. Proof. See Loewen (1983). //// Other desirable results also emerge as corollaries. The fundamental issue of controllability, for example, can be addressed as follows. (Note that since the hypothesis below involves only index 0 multipliers, it is independent of the cost function I.) - 21 -C o r o l l a r y 2. If A[M°{x)] = {0} for some arc x admissible for P(0), then there exists e > 0 such that for every a €E eB, one has an arc y(-) obeying y(t) £ F(t, y{t), a) a.e. [0, T], (y(0), y{T), a) € S. See Loewen (1983), Chapter IV, for a more detailed controllability result which generalizes Clarke (1983), Thm. 3.5.3. Extensions. Loewen (1983) also presents a sensitivity analysis for differential inclusion problems with free terminal time. These results, together with a detailed investigation of their consequences for controllability and the nonlinear time-optimal control problem, may be found in Clarke and Loewen (1984) and Clarke and Loewen (to appear). The following chapters extend the sensitivity analysis of this section in two more significant ways. In Chapter III, infinite-dimensional perturbations are used, and a new form of the maximum principle for state-constrained problems ensues. In Chapter V we revert to finite-dimensional perturbations, but apply them to a stochastic optimal control problem. - 22 -Chapter II. A Proximal Normal Formula in Hilbert Space Clarke's calculus of generalized gradients is distinguished by its mutually complementary geometric and analytic aspects, which extend those of ordinary calculus. In calculus, one considers a smooth function f:X —• R for which the gradient V/(x) is defined analytically in terms of limits. The resulting object has a geometric interpretation: the ray in direction (V/(x),—1) is the outward normal to the epigraph of / at the point (x, f(x)). (The epigraph of / is the set epi/:= {(x, r) 6 X x R : r > f(x)}.) The pedagogical progression from analysis to geometry in the smooth case accurately reflects the sequence of steps used to solve many applied problems: one first computes a gradient, and then uses its geometric properties. In Clarke's calculus, a generalized gradient can be defined for any lower semicontinuous f:X —+ RU {+00}: it is a weak*-closed convex subset of X*. The resulting object has a geometrical side which is perfectly analogous to that for smooth functions—if Nepif(x,f(x)) denotes the (Clarke) normal cone to the closed set epi/ at the point (x, /(x)), then (1) df(x) = {c&X* : U,-l)e/V e p i /(x,/(x))}. The fundamental difference between Clarke's theory and the ordinary calculus becomes clear when one turns to applications. In the problems to be considered in subsequent chapters, the desired results are analytic statements which could be derived from a formula for 3/(x); the appropriate solution is based on doing the geometry first, i.e. computing Nepl f{x, f(x)), and then drawing the analytic conclusions from (1). The geometric approach is valuable because, whereas the analytic definition of df(x) is difficult to apply, an elegant sequential characterization of Neplj(x, f(x)) is available. This result, called a proximal normal formula, is presented in Clarke (1983), Section 2.5. - 23 -Theorem. Let C C R n be a closed set containing a point c. Then Nc{c) = co{ lim Vi : u,- is a bounded sequence of proximal i—*oo normals to C at base points c,- —• c in C}. Until very recently, this formula was known only for finite-dimensional sets C, so the geometrical approach to df(x) outlined above could be applied only to functions whose domain was R n. Even in the finite-dimensional context, however, the proximal normal formula has made significant progress possible in the study of sensitivity, controllability, and time-optimality in optimal control. (See Section 1.4.) This useful formula has recently been extended to reflexive Banach spaces X by Borwein and Strojwas (1985), under the hypotheses that the norm of X is Frechet differentiable away from 0 and Kadec. This chapter shows that any Hilbert space satisfies these requirements and presents a simple proof of the Borwein-Strojwas theorem in this context. Chapter III presents a significant application of the theorem. Section 1 . Elementary Geometry of H i l b e r t Space Throughout this section we consider a real Hilbert space H equipped with the inner product (•, •) and the norm ||||. The open unit ball of H is denoted by B, and its boundary by S. 1.1 Proposition. Let x , y S H. Then \\x\\ = 1 and | | i — y\\ < S imply shows that H is uniformly convex.) Proof. By the parallelogram identity, any x,y € H obey \\x + y\\2 = 2\\xf-r2\\yf-\\x-yf >2||a:||2 + 2 ( | | x - y | | - | | x | | ) 2 - | | a : - y | | 2 = 4 | |x|| 2-4 | |x | | l l z - y l l + H z - y l l 2 >4 | | x|| 2-4 | | x | | | | x - y | | . 2 > 1 — S, as required. //// x + y > 1 - S. (This Hence if ||x|| = 1 and ||x — y|| < S we obtain x + y - 24 -1 . 2 P r o p o s i t i o n . Let x, y be any unit vectors in H. Then for any 6 > 0, (x,y) > 1 — 6 implies | | z — y\\ < \/26. In particular, (x, y) = 1 implies x = y. Proof. B y d e f i n i t i o n o f t h e H i l b e r t space n o r m , a n y x,y € H o b e y b - y | | 2 = ^ l l 2 + l | y | | 2 - 2 ( x , y ) . H e n c e | |x| | = | |y| | = 1 a n d (x,y) > 1 — 6 i m p l y t h e d e s i r e d i n e q u a l i t y : | | x - y | | 2 < 2 - 2 ( 1 - 5 ) = 26. Uf/ 1 .3 P r o p o s i t i o n . The norm topology and the weak topology of H coincide on the set S. (In other words, a Hilbert space norm is a Kadec norm.) Proof. L e t TV b e a s t r o n g l y o p e n s u b s e t o f S. W e m u s t s h o w t h a t N is a l so w e a k l y o p e n . T o d o so , w e w i l l s h o w t h a t a n y p o i n t x £ N h a s a w e a k n e i g h b o u r h o o d c o n t a i n e d i n N. F i x x 6 N. S i n c e N is s t r o n g l y o p e n , t h e r e e x i s t s e > 0 s u c h t h a t ( x + eB) n S C N. N o w c o n s i d e r t h e w e a k - o p e n set U = { y € H : ( x , y ) > 1 — | e 2 } : c e r t a i n l y x € U. M o r e o v e r , i f y e U n S, t h e n | |x - y | | 2 = 1 - 2 ( x , y ) + 1 = 2 ( 1 - ( x , y » < e2. T h u s U n 5 C ( x + eB) n S C TV. / / / / 1 .4 C o r o l l a r y . L e t { x n } be a s e q u e n c e i n i f . 7 / xn — • x ^ 0 a n d | | x „ II — l l x l l , th en xn —> x . x Proof. C o n s i d e r t h e s e q u e n c e yn = i n S. F o r a n y z €E H, w e h a v e lFn|| lF»ll I F I I H e n c e y n - ^ - » y = 77—7: . B y P r o p o s i t i o n 1.3, y n —» y , i .e. l l x l l x = l i m ( flr ) x r S i n c e S—Lr —•• 1 as n —+ 0 0 , i t f o l l o w s t h a t x n —» x . //// - 25 -1.5 Proposition. For any x 6 H \ {0}, the norm of H is Frechet differentiable at x, with derivative x R ' Proof. Observe that i H r = i i x + » r - i i « r - 2 ( x f w > = (\\X + v\\-\\x\\)(\\x + v\\ + \\x\\)-2(x,v). Rearranging terms in this equation yields the identity I1* + " I | - H - ( P T T , " ) + (iwriMi) 2 1 1 x 1 1 z + v + x \\x + v\\ + ||x|| Both terms on the right side evidently tend to 0 as v —+ 0. //// 1.6 Corollary. Let x ^ 0 be given in H. Then the following assertions about a vector v S H are equivalent. (a) ||t)|| = l a n d ( v ) x ) = ||x||. (c) v is the Fr6chet derivative of the norm at x. Proof, (b)^(c) was the content of Proposition 1.5; (b)=>-(a) is obvious; and (a)=>-(b) is an instance of Proposition 1.2 with 6 = 0. //// Section 2. Best Approximation by Closed Sets We continue to study the geometry of the Hilbert space H. If C C H is a nonempty closed subset of H, we may define the distance function from C as follows: dc(s): =inf{||s-e|| : c e C) . (The alternate notation d(x; C) = dc(x) will also be used below.) A point c € C at which the infimum defining dc(x) is attained is called a best approximation to x in C, or else a nearest point to x in C. When H is simply finite-dimensional Euclidean space, the Heine-Borel theorem implies that every x € H admits a best approximation from C. For a general Hilbert space H, however, compactness is a more elusive property and best approximations become harder to find. If the set C is convex, then - 26 -every x G H admits a unique best approximation (Rudin (1973), Thm. 12.3), but if C is not convex then there may be points x with no best approximation. For example, consider the Hilbert space I2 of square-summable real sequences with basis elements ei = (1,0,0,...), e 2 = (0,1,0,...), etc. In this space, the origin has no best approximation from the closed set C = {(l + ^)en • n G N}. It can be shown, however, that "most" of the points in £ 2 do have best approximations from C. The objective of this section is to prove a general theorem of this type, due to Ka-Sing Lau. The following technical lemma about the shape of the unit ball in H will be used in the proof of Lau's theorem. For any unit vector x G H, r G (0,1), and 8 G (0,1), we define Mr(x, 6) = [rx + (1 - r + 6)B] \ B. 2.1 Lemma. Let r G (0, l) be fixed. Then for any e > 0 there exists 8 > 0 so small that for any x G S, one has \\yi - y2\\ < e Vy 1,t/2 G Mr[x, 6). Proof. For any y G Mr(x, S), \\v\\ < \\y - rx\\ + |H| < (1 - r + 6) + r = 1 + 6. Likewise, l<||y|| 2 = ||rx+(y-rx)|| 2 = ||rx||2 + 2 ( r x , y - r x ) + ||y-rx|| 2 < r 2 + 2r (x, y) - 2r 2 + (1 - r + S)2. Rearrangement of terms gives 2r(x, y) > l + r 2 - ( l - r + £ ) 2 ~ ( x , y ) > l - ( i - l ) * - - U 2 . By Prop. 1.2, it follows that for 6 G (0,1), II1 _ v\ where K = \j is independent of x. Hence for any x G S and 8 G (0,1), y 1 ; y 2 G M r(x, 8) imply ||Vi ~ Kill < ||»i - *|| + II* - y2|| < 2Ky/S. Given any e G (0, 2K), one needs only to choose 8 = ( T - ^ T ) to verify the Lemma. //// V 2K ' 2 . 2 R e m a r k . L e t u s d e f i n e a n o n n e g a t i v e r e a l - v a l u e d f u n c t i o n d o n H b y d(x) := hm+ s u p | ^ * _ * , c 2 - c i ^ : < i ( 2 ; C ( j ( i ) ) < 5, C i , c 2 € , w h e r e C « ( x ) = [ x + ( d c ( z ) + n C. O b s e r v e t h a t d(c) = 0 f o r a l l c i n C. F o r i f c € C, t h e n C $ ( c ) = (c + 6.B) n C . H e n c e c i , c 2 £ C « ( c ) i m p l y | | c 2 — c i | | < 26: t h u s d ( c ) = 0. C o n t r a p o s i t i o n s h o w s t h a t i f d(x) > 0 t h e n x £ C , so d c ( a O > 0. 2 . 3 T h e o r e m . L e t C C H be closed and nonempty. Then every point x in the set G = { x € H : d{x) = 0 } h a s a best approximation from C. Proof. F i x a n y x &G. W e w i l l c o n s t r u c t a n e a r e s t p o i n t t o x i n C. C o n s i d e r a n y s e q u e n c e { c „ } f o r w h i c h c „ £ C i ( x ) . T h e n { | | c n | | } is b o u n d e d , so a l o n g s o m e s u b s e q u e n c e ( w e d o n o t r e l a b e l ) , cn c o n v e r g e s w e a k l y t o s o m e p o i n t cx £ H. T h i s is o u r c a n d i d a t e f o r t h e n e a r e s t p o i n t t o x i n C. N o t e t h a t s i nce | | c n — x | | < dc(x) + £ f o r e a c h n , w e h a v e \\cx — x\\ < dc(x). T h e r e f o r e i t su f f i ces t o p r o v e cx £ C. F i x a n y e > 0 . T h e d e f i n i t i o n o f d y i e l d s a c o r r e s p o n d i n g 6o s u c h t h a t t h e i n d i c a t e d s u p r e m u m is less t h a n e w h e n e v e r 0 < 6 < 6Q. C h o o s e N > 1/6Q. T h e n w h e n e v e r m > n > N, w e h a v e / c „ - x \ / c „ - x \ / cn- x e>\T\ n'°n ~ Cm / = \ ii ii'c" ~ x ) + \ ii i i ' I _ c \ l l c n - z | | / \ | | c n - a : | | / \ | | c „ - x | | ( ] £ ^ i i f , e m " s ) > l | e " L e t t i n g m - » o o h e r e , w e find dc{x) > \\cx - x\\ > ^ °" _ ^ ||.cs ~x^ > \\cn - x\\ - e > dc{x) - e. S i n c e e > 0 is a r b i t r a r y , i t f o l l o w s t h a t \\cx — x | | = dc{x) a n d t h a t \\cn — x | | —+ \\cx — x | | . S i n c e ( c n — x ) c o n v e r g e s w e a k l y t o (cx — x ) a lso , w e d e d u c e t h a t cn —* cx ( C o r o l l a r y 1.4) . S i n c e C is c l o s e d , w e h a v e c x £ C as r e q u i r e d . / / / / T h e f o l l o w i n g t h e o r e m j u s t i f i e s t h e s t a t e m e n t t h a t m o s t p o i n t s o f H h a v e b e s t a p p r o x i m a t i o n s f r o m C b y d e m o n s t r a t i n g t h a t i n f a c t , m o s t p o i n t s o f H l ie i n G. - 28 -2.4 Theorem. The set G = {x € H : d(x) = 0} is a dense Gs subset ofH. Proof. Consider the increasing sequence of sets Fn = {x e H : d(x) > 1/n}. It suffices to show that oo each Fn is closed and nowhere dense, since then the G$ set G = P| (H \ Fn) is dense by Baire's n=l theorem. To show that Fn is closed, fix any x0 Fn. Then there exists A > 0 such that 0 < 6 < A implies \ i r — - \ > c 2 ~ ci) < - - A VzeCs(x0)+8B, clt c 2 € C6{x0). \ l l z - a : o | | I n Pick 8 = |A. Then x € x o + 6"J3 implies d c ( x ) < dcfao) + 5: hence Cg(x) C C A ( X O ) . In particular, d(z;Cs(x)) < 8 implies d(z — (x — x o ) ; C « ( x ) ) < 28, and thus d(z — ( x — xo);C&{xo)) < 28 < A. Consequently x € x o + SB implies that for all z such that d(z; Cg(x)) < 8 and all Ci, c 2 £ C«(xo) Q / g - ( x - x Q ) - x 0 \ < 1 ^ \ | | z - ( x - x 0 ) - x o l l ' 2 V n " By definition, 0 implies dc[0) > 0 by Remark 2.2. Finally, we lose no generality in scaling the inner product in such a way as to make dc{0) = 1. By Lemma 2.1, there exists A > 0 such that ||J/2 - J/i|| < — — ~ Vyi,j/ 2 e M r ( x , A), x e 5. Now for S = \A, pick any x 0 S C f i(0): Define n + 1 xi = ., 0 M and x r = r x j . Then Cs[xr) C M r(xi,25), so we have I F O I I \ T T — - n r . C 2 ~ ci ) < | | « 2 - ci|| < — i - r Vc!,c 2 € C«(xr), z S / J . M p - ^ r l l / n + 1 In particular, d ( x r ) < l/n, a contradiction. Thus each Fn is nowhere dense. //// Section 3. Proximal Normals and Clarke's N o r m a l Cone As we saw in the introduction, the normal cone to a closed nonempty subset C of H is a key ingredient in the theory of generalized gradients. However, it is not a basic ingredient: rather, it is defined indirectly in terms of the tangent cone, which we now introduce. - 29 -5.1 Definition. Let C be a nonempty closed subset of H containing some point c. The tangent cone to C at c, denoted Tc{c), is the set of vectors y € H obeying any one of the following three equivalent conditions. / v dc{x + ty) -dc{x) (a) limsup - = 0. at—»c t (b) For every pair of sequences c< —• c in C and t{ \ 0, there is a sequence y,- —* y in H such that Ci + UyiGC Vi. (c) For every s > 0 there exist 6 > 0, A > 0 such that C n [x + t{y + eB)] ^ 0 Vi € C n (c + 55), te(0,A]. The equivalence of definitions (a) and (b) is proven in Clarke (1983), Theorem 2.4.5; the same method establishes (a=>c), while (c=>b) is obvious. Here is a particularly useful characterization of Tc{c), quoted from Treiman (1983), Thm. 2.1. 3.2 Proposition. Let c and C be as in Definition 3.1. Then y 0 2c(c) if and oniy if there exist e > 0 and sequences c» —+ c in C and Aj > 0 such that Cn[* + [0,\i]{y + eB)]=9 Vi. Now the normal cone is defined by polarity with the tangent cone. 3.3 Definition. Let C be a nonempty closed subset of H containing some point c. The normal cone to C at c, denoted Nc[c), is defined by Nc(c) = T£(c) = {veH : (v,y)<0 Vy £ Tc(c)}. Since the geometric features of the generalized gradient are more closely linked to the normal cone than to the tangent cone, a direct characterization of Nc{c) would certainly be a useful companion to Definition 3.3. Such a characterization is the goal of this chapter: it is given in Section 4. The fundamental idea behind it is that there ought to be a relationship between normals and perpendiculars. - 30 -3.4 Definition. A vector v S H is perpendicular to C at a point c S C i f c + v i s a point whose best approximation from C is c. This situation is written as v ± C at c, and each of the vectors tv, t > 0, is called a proximal normal to C at c. The set of all proximal normals to C at c is denoted PNo[e). 3.5 Proposition. T i e following assertions about v € H and c€C are equivalent. (a) v is a proximal normal to C at c. (b) For some t > 0, = dc{c + tv). (c) For some t > 0, (v, c - c) < ^- \\c - c]|2 Vc e C. Moreover, the same t will serve in both (b) and (c). Proof. (a-Ob) restates the definition of a proximal normal. To prove (b<=^ c), observe that the following three statements about c e C are equivalent for any t > 0: \\tv\\2 = \\(c + tv)-n2<\\(c + tv)-cf, t2 ||u||2 < ||c - c||2 + 2 (c - c, tv) + t2 \\v\\2 , < v , c - c ) < l | | c - C | | 2 . //// 3.6 Remarks. 1. Note that 0 is perpendicular to C at every point. Thus PNc[c) is never empty. Moreover, the results of Section 2 indicate that most points of H have a best approximation from C. Each of these points lying outside C gives rise to a nonzero element of PNc{c) at some point c G C. 2. In Clarke (1983), the definition of perpendicular requires that c provide the unique best approximation in C for the vector c + v. Since the ball of H is strictly convex, the proximal normal vectors corresponding to Clarke's more restrictive definition are identical to those discussed here. 3. Another widely used concept of normality to a given closed set involves "Frechet normals." A vector v € H is Frechet normal to C at a point c"£ C if for every e > 0 there is a neighbourhood Ne of c" such that (v,c-c) < e\\c -c|| V c e C n J V , . From Proposition 3.5(c), we see that every proximal normal to C at c" is automatically a Frechet normal to C at c". The converse is false even in R 2, where one may take C = {(x, y) : y < |a;l3^2}, c"= (0,0), and v = (0,1). In this example v is a Frechet normal to C at ?, but not a proximal normal. - 31 -Section 4. T h e Proximal N o r m a l Formula The main result of this chapter is a formula which allows the computation of Clarke's normal cone in terms of the proximal normal vectors discussed in Section 3. It asserts that NQ[C) = Rc{c), where Rc(c) is the closed convex cone Rc (c) = co{w-lim Vi : v< is a bounded sequence of proximal normals to C at corresponding base points c,- —• c}. As one might expect, this is proven by establishing two set inclusions, one of which is relatively straightforward. 4.1 Proposition. Nc{c) 2 Rc{c)-Proof. Let v lie in the set whose closed convex hull is computed to yield Rc(c). By definition, there is a bounded sequence {iij} tending weakly to v and a corresponding sequence of base points c< —• c such that Vi € PNc{ci) for each t. By Proposition 3.5(c), there exists a sequence e,- > 0 such that (vi, x — Ci) < Si \\x — Ci\\2 Vx £ C. Choose a sequence ti decreasing to 0 such that —+ 0. Then for any y £ Tc{c), Definition 3.1(b) asserts that there is a sequence yi in H converging to y such that Ci + tiyi € C Vi. The proximal normal inequality quoted above gives (vi, {a + tiyi) - ^ ) < ei ||(ct- + tiyi) - Ci\\2 Vi , <=> (vi,yi) < Uei ||yi||2 Vi. Since i,-e,- —• 0 and {||yi||} is bounded, we find that lim sup (v,-, y,) < 0. t—•oo Now since v,- converges weakly to v, we have (v,y) = .lim (vi,y) t—*oo < lim sup («,-, y.) + limsup (vit y - yt). t—^ 00 t—* oo The first term on the right side is known to be non-positive; the second is zero because is bounded and yi —• y. Hence (v, y) < 0. Since y 6 Xc(c) w a s arbitrary, we find that {v, y) < 0 Vy £ Tc{c), i.e. t> £ Nc(c). Since Nc(c) is known to be closed and convex, it must contain the closed convex hull of all such points v, namely Rc(c). / I l l - 32 -The assertion that Nc(c) C Rc[c) is considerably more difficult to prove. It is equivalent to the statement that Tc(c) 2 Rc(c)> o r - ^ \ ^ c ( c ) != H\RQ(C). We prove this last fact by considering any unit vector y e H\ Tc(c) and exhibiting a v € Rc(c) such that (v,y) > 0. The construction is based on Proposition 3.2. Since y £ Tc(c), there is an c > 0, a sequence of points Cj —• c in C, and constants A^ > 0 such that Cn[a + {0,Xi](y + eB)] = 9 Vi. For each i , we construct a unit proximal normal vector u< to C at a point cj near c» such that each t)j has a rather large inner product with y. The weak limit of a subsequence of {«,} then provides the desired v. The direction y will clearly play a significant part in this effort. We let yx denote the orthogonal complement of the subspace Ry in H. Then every vector v S H can be written as v = Ax + py for some x 6 yx n S and A, /x G R. In this decomposition, ||Ax -f- py\\2 = A 2 + p?. The next few lemmas deal with the situation for a fixed i . No generality is lost in translating the problem so that Cj = 0. For the e > 0 given above, we consider the cone K= \Jt(y + eB). t>o We also fix a g (0,1) (a specific choice will be made later) and define the set A 2 E = {Ax + ny : ^ + fi2 < 1, xey±r)S}. (In the simplest of all special cases, when H = R 2, E is an ellipse whose major axis lies along the y-axis, and each of the following five lemmas has a simple geometrical interpretation. Figures 1-6 at the end of this section are included to justify this statement and to make the proof easy to follow.) 4.2 Lemma. Let x E yx C\ S. For any A, /x e R one has Xx + nyEK n > |A|. Proof. The point Ax + fxy lies in i f if and only if there is a scalar p > 0 such that ||p(Ax + fiy) - y f 0 and — - — — < e 2. The result A2+/i2 ' A 2 + A I 2 - • ~ I + ( M/A ) 2 follows. //// - 33 -4.S Lemma. Let p > 0. If Xx + fiy S (y + (1 + p)E) \ K for some x € y x n S and constants X, fi € R, then 1 + V l + P(2 + P)(l + a- 2m- 2) , /x < v , v _ S where m = 1 + a *m * (Note that the same estimate for fi holds if we assume t(Xx + p.y) G t(y 4- (l + p)E) \K for any t > O.J Proof. If fi < 0 the conclusion is evident, so assume n > 0. Observe that Xx + p.y S y + (1 + p).E is equivalent to (*) ^ A 2 + ( M - 1 ) 2 < ( 1 + , ) 2 , while Xx + y.y £ K is equivalent to < m \X\. Substituting this into (*) yields (l + a_2m~2) p? - 2(1 - p(2 + p) < 0. The positive root of the quadratic function of fj. on the left here gives the maximum permissible value of fi, which is precisely the estimate written above. //// 4.4 Lemma. Suppose points VQ = XoXo + Hoy and vx = AiXi + /xiy in H are given (xo, x\ € y 1- C\S) such that t>i lies on the boundary of VQ + pE for some p > 0. If (y,vo — vx) > pr\ for some r) > 0, then for the unit vector n in direction — (XiXi — Ao£o) — a 2(^x — P-o)y o n e has - 7 ~ T T 7 2 • Proof. Without loss of generality take vo = 0, p = 1. Then observe that H-AxS! - a 2 M l y f = A 2 + «V? = <*2(l " A) + <*V? = " 2 [ l - (1 - a 2) M 2]. This function of fix takes its maximum value on (—oo, —ij] at fii = —rj. Therefore In ) = a a ( ~ M i ) > a2n K n' V } ||-Ani -aViy|| ~ V ' a 3 [ l - ( l - o 2 h a ] ' The indicated inequality is a rearrangement of this one. //// - 34 -Now the reason for studying E so closely is that it can be considered as the unit ball of a new norm on H. Indeed, if we define a bijective linear mapping L:H—*H by I(Ax + py) = ^ x + py V x € y - L n 5 , A,/i6R, then a new inner product on the vector space H is obtained by defining ((Ai i ! + pxy, A 2 x 2 + p2y)) = (^(AiX! + piy), L(X2x2 + p2y)) = a - {Xl,X2) + P\p2. a* 2 A 2 The new inner product induces the norm |||Ax + py\\\ = — j + p? whose unit ball is E. Clearly the new norm is equivalent to the original one, so (if, ((•, •))) is complete—i.e. is a Hilbert space in its own right. The key fact to be used below is that |||-|||-proximal normals correspond to || ||-proximal normals. 4.5 Lemma. Suppose that X\Xi + piy has a closest point c = (AiXi + piy) + (Aoxo + poy) € C (xo, xi e y1- n S) with respect to the norm |||-|||. Then a proximal normal vector to C at c is ~ _ _ ( A Q X Q + o?poy \ \||A0a:o + <*2Mo!/||/ ' Proof. Without loss of generality, assume Xi = p^ = 0. Let r = |||c|||, and note that |AoXo + a2/ioy|| = |(A O , O J 2 M O)| where || is the Euclidean norm in R 2. Then it suffices to show that c + a 2 r n has ||-||-closest point c in C. For this, we need only prove that c + o.2rn + a2rB C rE, i.e. that for any x e yx n S and any A, p € R with A 2 + p? < 1, one has \c +a2rn +a2r[Xx +py)\\\ < r2 A 0x 0 - a r A Q X Q |(A0,a2Mo)| + a rAx Mo a*rp0 2 + a rp |(A0la=2Aio)| < r2 V |(Ao,a2Ato)|/ \ |(A0,a2/xo)|/ + Mo This would be assured if we could prove that . 2 \ 2 4 T 2 a rpQ 2 |(A0,a2Mo)| < r' Ao f 1 - i n + 2 a 2 r f l - ) |A0A| + a 4 r 2 A 2 + 4 l2 Q r^p 2 Mo - T7\ o—77 + a: r/i \(X0,a2p0)\ This, however, is precisely the statement of Corollary 7.2, in which one must simply replace (x, y) by (A, p) throughout. //// 4.6 Lemma. Let c S C be a point for which there exists A > 0 such that C n [c + (0, A](y + £J3)] = 0. Then for any TJ > 0 there exists a point c € C and a unit vector v € H such that fa) ||c-c|| < V l (b) v is a ||-||-proximal normal to C at c, t.i (4m)- 1 v / T ^ W < U , ! / > - [15 +(4m)- 2] 1/ 2' w A e r e m = — — • Proof. Without loss of generality, we may take c = 0. Let r/ > 0 be given. Choose a = (4m) - 1, where m = . Then for any p € (0, |) and 0 < t < r?/4, we argue as follows. By Theorem 2.4, the open set t(y + pi?) in (iJ, ((•, •))) contains a point p = t(Xx + fiy) which has a best approximation c = A i + fly in C. Note that since 0 S C, (t) |||p-c|||<|||p|||<|||p-ty|||+||N|| t{n- j) > t ( l - p - |) > ^ t/2 1/2 by our choice of p. By Lemma 4.4, with rj = ... —rnr ^ — — ^ 1/4, we obtain |||p-c||| 1 + p a (4m)- 1 (v, y}> ~ [ l 6 _ l + Q2ji/2 [15+(4m)- 2] 1/ 2' where v is the ||||-unit vector in direction — (Ai — tXx) — a2(]u — tp.)y. By Lemma 4.5, this v is a proximal normal to C at c. //// - 36 -We may now complete the proof of the proximal normal formula. 4.7 Theorem. Nc(c) C Rc{c). Proof. We follow the program set forth at the beginning of this section. Pick any unit vector y $ Tc(c). Then by Proposition 3.2, there is some e > 0 and a sequence of points Cj —• c in C and constants Aj \ 0 such that Cn[a + (0,Xi}{y + eB)] = 0 Vi. y'l _ e2 (4m) - 1 Let m = and S = 7 —.—-r— . , > 0. By Lemma 4.6, there exists for each i a point e [15 + (4m)_-iJ1/-i c~i € (ci 4- jB) n C at which there is a unit vector v,- proximal normal to C and obeying (t;,-, y) > 6. Now the bounded sequence {t\} must have a subsequence converging weakly to some v G H. By definition, v € Rc(x). Along this subsequence (which we do not relabel), (u,-,t/) > 6 Vi implies (v>y) > 5 > 0. Hence y £ RQ(C). This completes the proof. //// In the case where H = R n, weak convergence and strong convergence are indistinguishable. In particular, if is a bounded sequence converging (weakly) to v, then either v = 0 or 77—^77 —* 77—77 Ik II IMI (weakly). (This is false in £2-take «j = ei + ej.) Hence Clarke's (1983) Prop. 2.5.7, p. 68, is embedded in the proof given here. 4.8 Corollary. Let C C R n be a closed set containing a point c. Then Nc{c) is the closed convex cone generated by the set \ lim 7 ^ - : : Vi ± C at c< —> c, vt —• 0 > U {0} . [i^ oo \Vi\ J Diagrams. The elementary nature of our proof of Thm. 4.7 becomes apparent when one coinsiders the simple geometric significance of each step in the argument. Figures 1-6 below facilitate this by presenting the situation in R 2. We have chosen y = (0, 1) and x = (1,0) in these diagrams, and labelled the axes A and p to reflect the general case. Figure 1 is introductory, and the enumeration of Figs. 2-6 corresponds to that of Lemmas 4.2-4.6. - 37 -As Fig. 3. Lemma 4.3 gives an explicit upper bound on // for all points in [y + (l + p)E] \ K, the shaded region. Fig. 4. Lemma 4.4 asserts that if v0 — vx has a large y-component, then so does the related vector n. - 39 -Section 5. Proper Points Given a nonempty closed set C in the Hilbert space H, a point c S C is called proper if JVc(c) contains nonzero elements. In the finite-dimensional setting, the proximal normal formula implies that the proper points of C are precisely the boundary points of C. 5.1 Proposition. Let C C R n be a closed set containing a point c. Then ATc(c) = {0} <=>• ceintC. Proof. (<=) If c e intC, then Tc{c) = R n by Def. 3.1, and Nc{c) = {0} follows by polarity. (=») Conversely, if c ^ int C, then there is a sequence Xi converging to c such that each i,- lies outside C and has a closest point c; in C. Let Vi = i,- — c;. Then Cj —» c because | c i — c| < | C J — Xj| + | i j — c| < 2 | i j — c|, and Vi —+ 0 by the same estimate. The proximal normal formula asserts that Nc(c) contains all (weak) limit points of the sequence -p^-r. Since we are in a finite-dimensional setting, some unit vector is such a limit point. Hence Nc{c) ^ {0}. //// The proof of Prop. 5.1 would carry over to arbitrary spaces H if only every sequence of unit vectors had a subsequence converging weakly to a nonzero limit. However, this need not be the case. A detailed account of what can go wrong in the infinite-dimensional case is provided by the following proposition, which shows that a sequence of unit vectors converges weakly to 0 if and only if it is "almost orthogonal". 5.2 Proposition. For a weakly convergent sequence of unit vectors {v,} C H, the following assertions are equivalent. (a) Vi 0. (b) VJif e N, lim sup \(vit vk)\ = 0. »—oo f c = 1 K Proof. (o=*-6) Obvious. -41 -(6=»o) We will prove "not (a) implies not (b)". Thus, let Voo denote the weak limit of w,-, and assume ||voo||2 = e > 0. Then there is a sequence of convex combinations of the form »=i t=i for which Wk —• Woo strongly. Consequently there exists some K € N so big that \\U>K — «oo|| < e/2. Write («,-, tUif) = (Vi,WK - Voo) + (Vi, > (vi} Woo) - \\v>K ~ «oo || > («i,«oo) - e/2. The RHS tends to the limit e - e/2 > 0, so we find liminf (u^iw/f) > e/2 > 0. The left side of this inequality is itself majorized by liminf sup (vi,Vk), «-"°° fe=l K so (b) fails as required. //// Now suppose c is a boundary point of C. The only way to obtain Nc{c) = {0} is if every sequence of proximal normal vectors (and there are a great many such sequences) is "almost orthogonal". This requires that the boundary of C be extraordinarily ill-behaved—so rough, in fact, that one might expect a very mild regularity condition on C (weak closure?) to eliminate the possibility completely. No such global result is known yet, but the proximal normal formula may be the key to solving this problem. In the remainder of this section we outline three additional approaches. One generic result on the propriety of C's boundary is given by Borwein and Strojwas (1985b), Thm. 5.1. Its proof involves Ekeland's theorem; in our setting, the statement reduces to the following. 5.3 Theorem. If C C H is a closed set, then the set of points c for which Nc(c) ^ {0} is dense in the boundary of C. A generic fact like Thm. 5.3 is tantalizing, but until proper points are completely characterized, one often wishes to know whether a specific point is proper for C. Our next proposition shows that if c G C is a point supporting (locally) a cone with nonempty interior, then c is proper. -42 -5.4 Proposition. Let C C H be a closed set containing a cluster point c. If K is a cone with vertex at 0 for which there exists ij > 0 such that C D (c + {K njjfl)) = {c}, then int Kn Tc\c) = 0 . In particular, if int K ^ 0, then Tc(c) # i f and Nc{c) ^ {0}. Proof. Observe that 0 ^ int i f , since this would force i f = H and then c + ( i f D r?B) = c + r\B would meet C in a set strictly larger than {c}. So for any y € int i f , there exists some e € (0,1) such that 0 ^ y + eB C i f . By assumption, Hence y ^ Tc(c) by Prop. 3.2; this completes the proof. //// Another approach to proving that a specific point c S C is proper relies on the following notion, due to Borwein and Strojwas (1984). 5.5 Definition. The closed set C C H is called compactly epi-Lipschitzian at c € C if there exist 6 > 0, e > 0, A > 0, and a compact set K C i f such that C n (c + SB) + teB C C + tK Vte(0,A). Note that any finite-dimensional set is compactly epi-Lipschitzian at all its points. 5.6 Theorem (Borwein-Strojwas (1984)). Let C C H be a closed set which is compactly epi-Lipschitzian at c. Then Nc (c) = {0} <=• c e int C. Section 6. Rockafellar's Theorem Previous sensitivity results based on proximal normals have relied on a geometrical result due to Rockafellar (1982), Prop. 15. His statement and proof of this result are intrinsically finite dimensional. In this section we extend Rockafellar's proposition to arbitrary normed spaces, beginning with a topology-free version of the result. -43 -6.1 Lemma. Let X be a reai vector space containing a nonempty subset D and a cone D°° with vertex at 0. Define cones N, JV°° C X X R via N={X(d,-l) : A > 0 ,de D}, TV00 = {(d°°,0) : <*°° € D ° ° } . Tien (aj {t> : (v,-1) € co [JV U N°°]} = co [D + D°°}, (b) {v : (v,0)eco[iVuiV o o]} = c o D 0 0 . Proof. Note that both (a) and (b) are automatic if D — 0. We therefore assume D =^ 0. (a) Let I = {v : (v, - l ) e co [JV U JV00]} denote the left-hand side of (a). For any de D and d°° e D ° ° , we have 2(d,-l) £ JV and (2, —1) € co [JV U N°°]: thus there exist n € N and /ij > 0 (i = 1,... ,n) with = 1 s u c ^ that fc n ( v , - l ) = ^ M i K ° ° . 0 ) + ^ « M * , - 1 ) t'=i t=fc+i for some df,..., 0, and dk+\,..., dn e D. Now if fc = 0 this shows that v e co [D + {0}] C co [D + D ° ° ] , so we assume fc > 1. Also, the second component of this equation forces fc < n, and n 52 MiAj = 1-i = fc + l Now simply rewrite v as fc n-1 v = ^ 2pi[^pnXndn + df) + 52 + M n A „ 1=1 fc t=fc+i = 52w^ nA„K + 7-d°°)+ Pi*i{di + 0) + UnK i = l , = k+l fc t'=i (dn + 0). This expresses v as a convex combination of n points in D + D ° ° . (The coefficients are nonnegative since J2i=x M» — a n c ^ their sum is fc n—1 fc n fc-MnAn 52 W + 53 ^*'A» + M n A n ~ fc^" 51^ * = H *^A» = t=l i=fc+l i = l i=fc+l Thus indeed L Q co [D + D°°}. (b) The left-hand side of (b) is convex and contains 2?°°, so {v : [v, 0) G co [J\T U N°°]} 2 co D°°. For the reverse inclusion, note that any v in the left-hand side must be the convex combination of finitely many vectors of the form (d?°,0) for d?° G D°°. (Clearly, no nonzero vectors from N may be included in this convex combination.) Hence v G coD°°, as desired. / / / / 6.2 Proposition (Rockafellar). Let X be a normed vector space containing nonempty subsets D and D°°; assume that D°° is a cone with vertex at 0. Define cones N, N°° as in Lemma 6.1. Then one has {v : (v, - 1 ) e co[N U TV00]} = co[D + D°°). Moreover, if co [N U JV°°] is closed, then one obtains {v : {v,-l) GcotJVuiV0 0]} = co\D-rD°°} {v : (v,0) GcofiVu/V 0 0]} = c o £ » ° ° . Proof. The stronger conclusions follow immediately from Lemma 6 . 1 when one writes in co[iV U N°°] for co [N U JV°°]. The first statement is the only one requiring elaboration. According to Lemma 6 . 1 , it suffices to show that {« : (v, - 1 ) G co[JV U TV00]} = {v : (u,-l) G co [JV U N°°}}. Upon defining the convex cone C = co[JV U JV°°] and the closed affine subspace S = X x { — 1 } of X x R, we find that this reduces to showing Snc\C = d[SnC], where cl denotes closure. Since S (~l cl C is a closed set containing S D C, we only need to prove S n c l C C cl[.SnC]. Suppose, therefore, that ( t i o o , — 1 ) G S D c l C . Then there must be some sequence (vit —ri) G C tending to (t; T O , — 1 ) . The norm on X X R forces Vj —• v^, r ; -+ 1 . Consequently the sequence (w./rj, - 1 ) G S D C tends to (««,, - 1 ) , i.e. (v T O , - 1 ) G cl[5 n C] as required. / / / / Sufficient conditions for co [N U N°°} to be closed may be based on the following simple lemma. 6 . 3 L e m m a . Let D C X be a closed set, and let D°° C X be a closed cone with vertex at 0 such that D°° D 0+D = | l i m A f d< : A< — 0 , d> € I > j T h e n JV U N°° is closed. Proof. C l e a r l y JV°° = D°° X { 0 } is c l o s e d . I t t h e r e f o r e su f f i ces t o s h o w t h a t i f a s e q u e n c e Xi(di,— 1) i n JV c o n v e r g e s t o s o m e p o i n t ( u , — A ) , t h e n t h i s l i m i t p o i n t l ies i n JV U N°°. W h e n A > 0 , t h e s e q u e n c e di c o n v e r g e s t o u / A , w h i c h w e m a y d e n o t e b y d € D s i nce t h e set D i s c l o s e d . H e n c e (v, — A ) = X[d, — 1 ) 6 N. A l t e r n a t i v e l y , w h e n A = 0 w e find v € 0+D C D°° b y a s s u m p t i o n , so M ) € J V ~ . / / / / T h e f o l l o w i n g l e m m a , p r o v e n i n R o c k a f e l l a r ( 1 9 8 2 ) , l e a d s t o a u s e f u l s i m p l i f i c a t i o n o f P r o p . 6 .2 w h e n X = TL n. 6 . 4 L e m m a . L e t C C R n be a closed cone with vertex at 0. If C is pointed, then coC is closed and pointed. 6 . 5 P r o p o s i t i o n ( R o c k a f e l l a r ) . T a i e X = R n . Suppose that the sets D and D°° of Prop. 6.2 are closed, and that D°° D 0+D. If D°° is pointed, then co [ j V U N°°] is closed and hence the stronger conclusions of Prop. 6.2 hold. Proof. T h e se t JV U N°° is c e r t a i n l y a c o n e w i t h v e r t e x a t 0. I t is c l o s e d b y L e m m a 6 .3 , so a c c o r d i n g t o L e m m a 6.4 i t su f f i ces t o s h o w t h a t N U JV°° is p o i n t e d . I f t h i s w e r e n o t t h e case, t h e n t h e r e w o u l d b e p o s i t i v e s c a l a r s A ° ° , A | ° , • • • , A ° ° a n d A i , A 2 , • . . , A m s u c h t h a t l m 52Ar(^.o) + E^(^.-1) = (0.°) f o r s o m e p o i n t s dk € D, d£° S D°° \ { 0 } . T h i s c l e a r l y f o r c e s m = 0 , a n d t h u s i m p l i e s t h a t D°° is n o t p o i n t e d . T h i s c o n t r a d i c t s o u r a s s u m p t i o n s . / / / / - 4 6 -The only intrinsically finite-dimensional link in the chain of arguments supporting Prop. 6.5 is Lemma 6.4. It can be false even in the Hilbert space (? of square-summable real sequences. To see this, let ti denote the sequence whose ith entry is 1 while all others are 0, and define C = (J \{ei : k G N} . The set C is evidently a closed pointed cone with vertex 0 in £ 2 . However, coC is not closed. For the point x = ( § i 41 8> • • •) ^ e s o u t s i d e coC, all of whose members are sequences with only finitely many nonzero terms, but inside coC. Section 7 . Appendix: Some Euclidean Geometry The Hilbert-space computations of Section 4 involve a distinguished direction y, and treat all vectors in y1 alike. Thus a good model for these computations can be displayed in the Cartesian plane: we take the y-axis to correspond to the distinguished direction y in the general theory, and visualize all of yx as the x-axis. For a fixed a G (0,1), we now consider the ellipse E={(x,y) : ^ + y2 0, independent of a, such that every admissible arc x for P(ct) obeys H x ^ < M. Now it is easy to find a function a 6 L2 with arbitrarily small L2-norm such that a > M on a set of positive measure. For such an a, problem P(ct) has no admissible arcs and V(a) = +00. So we must be prepared to deal with the possibility, even if V(0) < +00, that every neighbourhood of 0 in L2 contains points a where V(a) = +00. Even though the data of the problem are smooth, the resulting value function may be astonishingly discontinuous. Let Y denote the collection of all arcs x(-) solving P(0). (We will shortly show that V j& 0.) The following hypotheses will be used in this chapter: (Hi) The multifunction F : [0, T] X R n —» R " has nonempty compact convex values. For each fixed x G R n , F(-, x) is measurable. (H2) There is a function k(t) e L2[0, T] such that (a) F{t, x) C k{t)B Vt € [0, T], x € R n , (b) for each fixed t €E [0, T] and i S R " , one has F(t,y) C F(t,x) + k(t) \y - x\B Vy € R " . We define KF = exp (/or k(t) dt). (H3) The set C o constraining the initial point is compact. (H4) The terminal cost £: R n —• R is Lipschitz of rank Kt on R n . (H5) The state constraint function g: [0, T] X R " —• R a is Lebesgue measurable in t and continuously differentiable in x, with iffa:(*, a:) [ < Kg for all (t, x) e [0, T] X R n . Moreover, /0T \g(t, x)\2 dt < +00 Vs 6 R n . Hypotheses (H1)-(H5) allow the existence of solutions to P(oc) to be proven by the direct method. 1 . 1 Theorem, (aj Let a sequence i < of F-trajectories be given, with 2^ (0) 6= C o Vi. Then {x,} has a subsequence converging uniformly to an F-trajectory x such that x(0) £ C o . (b) If there is a sequence cti £ L 2([0,T ] ,R a ) for which the sequence x< in (a) also obeys g(t, x«(t)) + oti(t) < 0 a.e. Vi, and if a< -^-+ a for some a, then along the subsequence in (a), one also obtains g(t, x(t)) + a(t) < 0 a.e. - 50 -(c) IfV(a) < +00 for some a, then P(ct) has a solution. (d) The function V is weakly sequentially lower semicontinuous. (In particular, V is norm-lower semicontinuous.) Proof, (a) T h i s is a C o r o l l a r y to C l a r k e (1983), T h m . 3.1.7, p . 118. (b) A c c o r d i n g to (H5) , the difference sup \g{t, xi{t)) - g[t, x{t))\ < Kg ||x< - x | | T O t tends to 0 a long the subsequence described i n (a). Hence the funct ions g[t,Xi[t)) + a , ( t ) converge weakly i n L2 to the funct ion g(t,x(t)) + a[t). N o w each element of this sequence lies i n — P , where P is the "pos i t ive cone" defined by P= {r{)&L2 : r[t) > 0 a.e.} . T h e cone P is convex and strongly closed, hence w e a k l y closed. Therefore the l i m i t funct ion g{t, x(t)) + a(t) lies i n — P also, as required . (c) L e t Xi be a m i n i m i z i n g sequence of P-tra jector ies . A c c o r d i n g to (a) and (b), i n w h i c h we take ai = a, there is a subsequence along w h i c h Xi tends u n i f o r m l y to an arc x sat is fying a l l the constraints of p r o b l e m P ( a ) . A n d by (H4) , £ ( x ( T ) ) = l i m £ ( x < ( T ) ) = V ( a ) . T h u s x solves P ( a ) . (d) L e t cti a i n L2, w i t h V(c * i ) —• v. We must show v > V(a). If v = +oo this is t r i v i a l , so assume v < +oo. T h e n (c) gives solutions ii to P(o : « ) for w h i c h V{cti) = i{xi(T))~ and g(t, Xi(t)) + cti(t) < 0 a.e. B y parts (a) and (b), we have t> = t{x[T)) for some l i m i t i n g P- t ra j ec tory x obey ing g(t,x(t)) + a ( t ) < 0 a.e. Hence V(a) < v as required . / / / / Section 2. Proximal Normals L e t (/3, — A ) be p r o x i m a l n o r m a l to the (weakly sequentially-) closed set e p i V at (a,v). T h e n V ( 2 ) < v < +oo impl ies P ( 2 ) has a so lut ion x w i t h v > £(x(T)). N o w for any P- t r a j ec to ry x ( ) s t a r t ing i n C o , one has V(-g(-,x())) 0 such that (2.1) ((ft —A), (- 0. This implies that $ = v — l(x[T)) gives a global minimum to the (smooth) right-hand side over the set [0, +oo). Hence the right derivative of this expression must vanish if $ > 0, or at least be nonnegative if s" = 0. We write this as follows. (2.4) A>0, X[v-e(x{T))] =0. If we now take s = s" in (2.2), we obtain (2.5) 0< ( f t r - f ) + M | | r - r | | 2 Vr € P, where P = {r € L2 : r[t) > 0 a.e.} and r[t) = — a(t) — g(t,x(t)). This statement is the definition of —ft 6 PNp(r). By the proximal normal formula, it follows that —p 6 Np{r). Finally, since P is closed and convex, Np is the normal cone in the sense of convex analysis. Therefore (2.5) gives (2.6) (ft r - f) > 0 Vr € P. If we now put r = r and s = s" in (2.1), the result is (2.7) (ft g(t, x)) + A£(x) < (ft g(t, x)) + A£(x) + M \\g(t, x) - g(t, x)f + M |£(x) - £ (x ) | 2 . - 52 -This shows that i solves the optimal control problem of minimizing the right-hand side over all F-trajectories x with x(0) 6 Co-We are about to use the inner product in L2 for the first time. Since it has not been used before, lines (2.4), (2.6), (2.7) remain valid in any Hilbert space of functions G satisfying the mild conditions (i) jf(-,i()) £ G for all F-trajectories x\ (ii) P is a closed convex subset of G. Different choices for G yield different necessary conditions and raise different technical problems in the proofs below. Such difficulties arise mainly in the weak convergence arguments of Section 3 below: the salubrious properties of the weak topology on L2 explain why we have chosen this space for G. Here are our conclusions about proximal normality, phrased in terms of the Hamiltonian H[t,x,p) :=sup{(p,/) : feF{t,x)}. 2 . 1 Theorem. Let (/3, —A) be proximal normal to epiV at (a, v). Define the constant R depending on (fi, —A) by equation (2.9) below. Then for any R > R, problem -P(S) has a solution x to which there corresponds an arc p: [0, T] —* R " obeying the following conditions, (a) A > 0, (fi, r - r) > 0 Vr € P, t») (l((' t)) + W ' " " ' S l ' ) ) ) ^ f f ( . „ ( , ) , p W ) « . [ 0 , T ] , (c) P (o) s «^/i + | p ( o)| 3« 0 a.e.}, and prime denotes transpose. Proof. Conclusion (a) is a transcription of lines (2.4) and (2.6). The other two conclusions follow from line (2.7), as we now show. The objective functional minimized by x is A£(x(T)) + M |£(x(T)) - £ (x (T) ) | 2 + C /?(t)'S(i,x(t)) dt + M T \g[t,x{t)) - g(t,x(t))\2 dt. Jo Jo This functional can easily be transformed into Mayer form by introducing a new state variable y(t) € R obeying y{t) = p(t)'g{t,x{t))-rM\g{t,x{t)) - g{t,x{t))\2 , y(0) = 0. - 5 3 -We deduce that the arc (x(i),y(t)), where y(t) = P(r)'g[r,x(r)) dr, minimizes t(x(T), y(T)) = A£(s(T)) + y{T) + M \i(x(T)) - t{x{T))\2 over all (n + l)-dimensional trajectories for the multifunction F(t,x,y) =F{t,x) X {Pit)'g{t,x) + M\g{t,x) - g{t,x-[t))\7} obeying (i(0), y(0)) e CQ X {0}. This is a problem for which Thm. 1.4.1 provides necessary conditions. These conditions involve the constants K? and Kp, which we now estimate. On a sufficiently small tube about the solution x (see the remarks following Thm. 1.4.1) the Lipschitz rank of the quadratic terms in £ and F can be made arbitrarily small, so any choices of these constants obeying the following inequalities will suffice: Kt> \Kt + l, Kp>exp ^[k{t) + \B(t)\Kg] dt^. In particular, let us choose K?= XKt + 2, K? > exp ^ k(t) dt + KgVf ||/?||2 j = KF exp (KgVf ||/3||2) . Then we obtain the explicit expression (2.9) R=(2XKt + 6) (2 + KFexp(KgVf\\p\\2){lnKF + KgVfm\2]) . Now in the terminology of Thm. 1.4.1, the endpoint constraint set for the problem we are studying is S = C0 X {0} X R n X R. This implies p = 1. The transversality condition (c) of Thm. 1.4.1 implies that for some c € dt(x(T)), one has (P(O), g(o), -p(T), -g(T)) eA(o,o,c,o) + (o,o,o,i) + i?|(i, E)\ [ddCo(m) x Bx {(0,0)}], E = (0,0, A ?, 1) - ( P(0), g(0), -P(T),-q(T)) = (-p(0), -g(0), 0,0). We deduce that P(o)eR |(i, E)\ddCo(x{o))> (2.10) -P(T) = Xc, -q(T) = l. - 54 -(t,x(t),P(t)) T h e H a m i l t o n i a n f o r o u r a u x i l i a r y p r o b l e m is H(t,x,y,p,q) = sup{((p,g),(w,tu)) : [v, w) e F(t, x, y ) | = H{t,x,p)+q[p{t)'g{t,x) + M\g{t,x) - g(t,x{t))\2) . S i n c e H is i n d e p e n d e n t o f y , t h e c o s t a t e q is c o n s t a n t ; w e find q = — 1 . T h u s E = ( — p ( 0 ) , 1 , 0 , 0 ) a n d | ( 1 , E)| = \J2 + | p ( 0 ) | 2 . C o n c l u s i o n (c ) n o w f o l l o w s f r o m ( 2 . 1 0 ) . T h e H a m i l t o n i a n i n c l u s i o n f o r t h e a u x i l i a r y p r o b l e m l e a d s t o ( ~ ! ( t ) ) &d(x'p) [H^x^ -m'9(t,x)-M\g{t,x)-g(t,x(t))\2] = dH(t,x(t),p(t)) - (p(t)'gx(t,x(t)),0). C o n c l u s i o n ( b ) f o l l o w s f r o m t h i s . / / / / T h e s e c o n d s t a t e m e n t o f c o n c l u s i o n (a ) c a n b e r e g a r d e d as a c o m p l e m e n t a r y s l a c k n e s s c o n d i t i o n o n t h e f u n c t i o n ft T h e f o l l o w i n g r e s u l t m a k e s t h i s p r e c i s e . 2.2 T h e o r e m . Let p € L2[0, T] a n d rePbe given. The following are equivalent: (a) ( f t r - f ) > 0 V r e P , (b) 0{t) > 0 a .e . , ft(t)'f(t) = 0 a.e. Proof. (a=>b) L e t b(t) = m i n { f t ( t ) : t = 1 , 2 , . . . , a}. D e f i n e E = {t : b(t) < 0 } a n d _ / r ( t ) i f t £ E, \ f ( t ) + ej i f t € E a n d i = m i n a r g m i n < { f t ( t ) } . T h e n (a ) i m p l i e s 0 < / P{t)'(r{t)-r{t))dt= f b(t)dt<0, Jo JE so i n d e e d m(E) = 0 . N e x t , c o n s i d e r F = {t : ft(t)'r(t) > 0 } . S i n c e r\t) > 0 a.e. b y a s s u m p t i o n a n d ft(t) > 0 a.e. is n o w k n o w n , w e h a v e 0(t)'r[t) > 0 a.e. C o n s i d e r »•(*) = |?(t )/fW+f ( ( )/ 1 o ,T|\FW-C e r t a i n l y r(t) > 0 a.e. , so (a ) i m p l i e s 0< [ p(t)'(r{t)-r{t))dt = -\ f p(t)'r{t) dt < 0. Jo JF - 55 -Thus m(F) = 0 and (b) is established. (b=>a) If (b) holds, then certainly / 0 T fi(t)'?(t) dt = 0, so (fi,r-9) = C fi(t)'r(t)dt. Jo For any r G P, the integrand is nonnegative almost everywhere, whence the integral cannot be negative. //// S e c t i o n 3. C o n v e r g e n c e Suppose that V(0) < +oo, and that (fi, —A) is obtained as the weak limit of a bounded sequence of proximal normals (fii,— A<) to epiV at base points (ai,vi) converging (strongly) to (0, V(0)). Let Xi be the corresponding solutions to P(cti), with multipliers p< as in Thm. 2.1. By Thm. 1.1 we have n —* x uniformly along some subsequence, where x solves P(0). Also, of course, A = lim Aj is nonnegative. Just as in the proof of Thm. 1.1(b), g(t,Xi(i)) converges to i —* oo g(t,x(t)) strongly in L2. In particular, the inner products (fii(), — «»(•) — g(-,xi())) converge to (£(•), ~g{-, *(•)))• Since we also have (fii, r) -» (fi,r) Vr € P, it follows from Thm. 2.2 that (3.1) ( / V ( ) + <7(,*())) >0 VreP. Next, observe that the constants Ri defined in terms of (fii,— Xi) by (2.9) form a bounded sequence. Hence we may use the same number R = sup{i2< : i € N} for each i when applying transversality condition (c) of Thm. 2.1. This condition implies . p i ( 0 ) eRddCo(xi(o)) Vi. V 2 + IP.(O)I2 Since the sequence on the left side is bounded, it has a convergent subsequence, whose limit we denote by p(0)/^2+\p(0)\2. By Prop. 1.2.3, it follows that (3.2) p(0) e Ry/2+\p[0)\2ddCo(*(0)) C NCo(*(0)). Similarly, we have -pi(T) € \idt(xi(T)) with A< -» A, zt- x, and dl(xi(T)) C KtB Vi. Hence along a further subsequence, Pi(T) converges to a point we denote by p(T), for which Prop. 1.2.3 gives (3.3) -p(T) e Xdt(x(T)). - 56 -L e t us s h o w t h a t s u p Hoo < oo« I n t e g r a t i n g t h e first c o m p o n e n t o f t h e H a m i l t o n i a n i n c l u s i o n ( T h m . 2 . 1 ( b ) ) a n d u s i n g t h e L i p s c h i t z c o n d i t i o n o n H c o m p u t e d b y C l a r k e ( 1 9 8 3 ) , P r o p . 3 .2 .4 , w e f i n d Pi(t) - Pi{T) + j ft(a)'fo(a,*.-(«))d«ejT k{s) \Pi{s)\Bds => \pi{t)\< XtKe + TiKg ^ \fii(s)\2 <*sj + £ k(s) \Pi(s)\ ds, T h e first R H S t e r m is u n i f o r m l y b o u n d e d i n t b e c a u s e A; c o n v e r g e s ; t h e s e c o n d , b e c a u s e fii is a b o u n d e d s e q u e n c e i n L 2 . T h u s t h e r e is a c o n s t a n t C f o r w h i c h \Pi{t)\ < C + £ k{s) \Pi{s)\ds. A p p l i c a t i o n o f G r o n w a l l ' s l e m m a g ives |p , ( t ) | < C e x p ^ j f k[s) ds^j < C K F . T h e r i g h t s i d e is i n d e p e n d e n t o f t a n d t as r e q u i r e d . T o see t h a t t h e s e q u e n c e {p,} is e q u i c o n t i n u o u s , n o t e t h a t m u c h as a b o v e , \Pi(t) - p.-W| < J* |A-(r)'fo(r, x , ( r ) ) | dr + J* *(r) |p,(r)| dr < (t-a)iKg J*\Pi{r)\a d r j ' k{r) dr. H e r e w e h a v e w r i t t e n M f o r t h e finite n u m b e r s u p Hp.H^. A g a i n , s i n c e fii c o n v e r g e s w e a k l y i n L2, i i t is a b o u n d e d s e q u e n c e i n L 2 . H e n c e t h e first R H S t e r m is m a j o r i z e d b y K(t — s)i f o r s o m e K i n d e p e n d e n t o f t . T h u s u n i f o r m e q u i c o n t i n u i t y o f t h e f a m i l y {p,} f o l l o w s . S i n c e t h e f a m i l y {pi} is u n i f o r m l y b o u n d e d a n d e q u i c o n t i n u o u s , i t h a s a s u b s e q u e n c e c o n v e r g i n g u n i f o r m l y t o s o m e c o n t i n u o u s f u n c t i o n p o b e y i n g ( 3 . 2 ) a n d ( 3 . 3 ) . C o n s i d e r n e x t t h e f u n c t i o n s u , ( t ) = fii[t)'gx{t,x<(t)) - p , ( t ) . H y p o t h e s i s ( H 2 ) a n d C l a r k e ( 1 9 8 3 ) , P r o p . 3 .2 .4 , p . 1 2 1 , i m p l y t h a t \ui{t)\ l k l l 2 < I I M - ) l l 2 ^ p l M l o o - 57 -so rn i s a b o u n d e d s e q u e n c e i n L2 w h i c h m u s t t h e r e f o r e a d m i t a s u b s e q u e n c e c o n v e r g i n g w e a k l y t o s o m e f u n c t i o n u . N o w t h e b o u n d e d c o n v e r g e n c e t h e o r e m i m p l i e s t h a t gx(t,Xi(t)) c o n v e r g e s s t r o n g l y i n L2[0,T] t o gx{t,x(t)). H e n c e Pi(t)'gx{t,x<(t)) c o n v e r g e s w e a k l y i n L2 t o P{t)'gx(t,x(t)) a n d t h e r e l a t i o n s h i p -Pi(t) = Ui{t) - ft («)'&(«, «,•(*)) i m p l i e s t h a t — p\ c o n v e r g e s w e a k l y i n L2 t o u — fi'gx[t,x). T h i s a l l o w s us t o pass t o t h e l i m i t i n t h e r e l a t i o n s h i p Pi{t)=Pi{0)- [ [M«)-M«Y9x{a,Xi{a))]ds Vt Jo a n d o b t a i n p[t) = p ( 0 ) - f [u{a) - P{s)'gx{S) x{s))] ds V t . H e n c e p(t) is a n a r c . N o w f o r e a c h i, c o n c l u s i o n (c ) o f T h m . 2.1 i m p l i e s t h a t (-gW + AWW*.-«W)] edH{t,Xi(t),Mt)) a.e. T h e p r o o f o f C l a r k e (1983), T h m . 3.1.7, p . 118 s h o w s t h a t t h i s i m p l i e s (3-4) (i ( (? ) + / , ( t ) ' f a ( ' ' x W ) ) 6 ^ ( t , x W l P W ) a.e. T h e f o l l o w i n g r e s u l t s u m m a r i z e s c o n c l u s i o n s (3.l)-(3.4). 3.1 Theorem. A s s u m e (Hl)-(H5). Let ( f t —A) be a w e a i limit of a bounded sequence of proximal normals as described above. Then P{0) has a solution x for which some absolutely continuous function p ( ) obeys (a) A > 0 , P[t) > 0 a.e., P{t)'g(t, x{t)) = 0 a.e. ( b ) (-jW + W f o(^-W)) 6^ ( t > a ( 0 i P^ a.e. f c j p(0) € JVCo(*(<))), - p ( T ) = A a £ ( x ( T ) ) . - 58 -S e c t i o n 4 . C o n s t r a i n t Q u a l i f i c a t i o n s a n d N e c e s s a r y C o n d i t i o n s C o n c l u s i o n s ( a ) - ( c ) o f T h m . 3 . 1 a re t h o s e w e w i s h t o p r o p o s e as a n e w se t o f n e c e s s a r y c o n d i t i o n s f o r p r o b l e m P(0). H o w e v e r , T h m . 3 . 1 does n o t i m m e d i a t e l y j u s t i f y t h i s b e c a u s e i t c o n t a i n s n o n o n t r i v i a l i t y c o n d i t i o n . L a u ' s n e a r e s t p o i n t t h e o r e m (see S e c t i o n I I . 2 ) g u a r a n t e e s t h a t m a n y s e q u e n c e s o f p r o x i m a l n o r m a l u n i t v e c t o r s e x i s t , b u t c a n n o t r u l e o u t t h e p o s s i b i l i t y t h a t a l l o f t h e m c o n v e r g e w e a k l y t o z e r o . I f w e t a k e (fi,—X) = ( 0 , 0 ) i n T h m . 3 . 1 t h e n c o n c l u s i o n s ( a ) - ( c ) h o l d t r i v i a l l y f o r t h e a rc p = 0. So t h e final s t e p i n p r o v i n g n e c e s s a r y c o n d i t i o n s f o r p r o b l e m P(0) is t o d e m o n s t r a t e t h a t some s e q u e n c e o f p r o x i m a l n o r m a l u n i t v e c t o r s h a s a n o n z e r o l i m i t . T h e p r o x i m a l n o r m a l f o r m u l a o f C h a p . I I c o m e s i n h e r e . A l l t h e h a r d w o r k i n t h a t c h a p t e r w a s d e v o t e d t o p r o v i n g t h e i n c l u s i o n (4 .1 ) i V e p i v ( 0 , V ( 0 ) ) C J 2 e p i v ( 0 , V ( 0 ) ) , w h e r e R d e n o t e s t h e c l o s e d c o n v e x c o n e g e n e r a t e d b y w e a k l i m i t s o f p r o x i m a l n o r m a l s . (See S e c t i o n I I . 4 . ) O u r l a b o u r s n o w b e a r f r u i t . 4 . 1 T h e o r e m . If Nepiv ( 0 , V ' (O) ) contains nonzero points, then there is a solution x to problem P{0) for which one can find a s c a l a r A > 0 , a function fi e I? ( [ 0 , T], R ° ) , a n d a n a r c p ( ) s u c h t h a t A + \\P\\2 > 0 a n d conclusions (a)-(c) of Thm. 3.1 hold. B e f o r e c o m p a r i n g t h e s e c o n d i t i o n s t o t h o s e c u r r e n t i n t h e l i t e r a t u r e , l e t us i n v e s t i g a t e t h e c o n d i t i o n NepiV (0,V (0)) ^ { 0 } . C a l m n e s s . P r o b l e m P is c a l l e d calm at a if V(a) < 00 a n d b m i n f ^ - r f ) = m > - o o . Hoc — a | | C a l m n e s s is a w e l l - r e s p e c t e d c o n s t r a i n t q u a l i f i c a t i o n i n o t h e r s e t t i n g s — t h e c a l c u l u s o f v a r i a t i o n s a n d m a t h e m a t i c a l p r o g r a m m i n g , f o r e x a m p l e . See C l a r k e ( 1 9 8 3 ) f o r a d i s c u s s i o n o f t h e s e m a t t e r s . O u r p r e s e n t c o n c e r n is t o s h o w t h a t c a l m n e s s g u a r a n t e e s t h e n o n t r i v i a l i t y o f t h e n e c e s s a r y c o n d i t i o n s i n t r o d u c e d i n T h m . 4 . 1 . - 5 9 -4.2 Proposition. If problem P is calm at 0, then NeT>iv (0, 7(0)) jt {0}. In fact, dV(0) ^ 0. Proof. We will apply Prop. II.5.4, with H = £2([0,T],Ra) x R, C = epi7, and c = (0,7(0)). Now epiV is locally closed near (0,7(0)) and this point is a cluster point of epi7. So it suffices to exhibit a cone K with nonempty interior such that for some rj > 0, (t) epi7 n [(0,7(0)) + (KnVB)] = {(0,7(0))}. Our candidate for K is K=\Jt[{0t-l) + eB\, t>o -y/l - e2 where e > 0 is chosen so small that < m — 1. Evidently, (0, —1) € intiif. Moreover, e - V l - e2 Lemma II.4.2 shows that {ct,v) e K if and only if v < ||a||. Thus for any r/ > 0, any (a, v) e (0, 7(0)) + K n r)B with a ^ 0 obeys {*) 0<||a|| 0 so small that whenever (a, v) e epi 7, one has , ^ n II II ^ t>-7(0) ^ 7(a)-7(0) ^ 1 ( . . ) o<|M|<„ n ^ > - L J f - ] [ _ U > m _ - . Clearly no point (a, v) can obey both (*) and (**), so (t) follows. According to Prop. II.5.4, /W(0,7(0))^{0}. In fact, Prop. II.5.4 says more than this. It affirms that for any S € (0, e) and any sufficiently small value of t, the open convex set t[(0,-l) + 6B] CintKCH is disjoint from the closed convex set Tepiv(0,7(0)). Hence (Rudin (1973), Thm. 111.3.4(a), p. 58) there is a unit vector (ft —A) in H for which sup ((ft -A), r e p i v (0,7(0))) < inf ((ft -A), t[(0, -1) + SB]) . Noting that 0 £ T e pi v (0,7(0)) always holds and evaluating the right side gives 0 < sup ((ft -A), r e p i V (0,7(0))) < t(X - S). - 60 -T h i s c l e a r l y i m p l i e s A > 6 > 0 . B u t s i n c e l e p i v -( 0 , V ( 0 ) ) is a c o n e , t h e m i d d l e e x p r e s s i o n m u s t c o n t i n u e t o s a t i s f y t h e i n d i c a t e d i n e q u a l i t y w h e n m u l t i p l i e d b y a n y p o s i t i v e n u m b e r p. T h u s i t c a n n o t b e p o s i t i v e , a n d t h e e q u a t i o n s u p ( ( / ? , - A ) , r e p i v ( 0 , V ( 0 ) ) > = 0 i m p l i e s [fi, - A ) € NepiV{0, V ( 0 ) ) . I t f o l l o w s t h a t fi/X € 8V{0). //// A l t h o u g h t h e c a l m n e s s c o n d i t i o n is d i f f i c u l t t o v e r i f y w i t h o u t f u r t h e r i n f o r m a t i o n a b o u t t h e s p e c i f i c p r o b l e m u n d e r i n v e s t i g a t i o n , i t is p o s s i b l e t o say c o n f i d e n t l y t h a t " m a n y p r o b l e m s are c a l m . " I n d e e d , t h e f o l l o w i n g r e s u l t i m p l i e s t h a t w h e n e v e r t, F, a n d g a re g i v e n s a t i s f y i n g ( H 1 ) - ( H 5 ) , t h e set o f a f o r w h i c h P is c a l m a t a is dense i n D o m V = {a€L2 : V(a) < +00} . I t is q u o t e d f r o m A u b i n a n d E k e l a n d ( 1 9 8 4 ) , T h m . V I . 6 . 5 , p . 2 8 1 . 4 . 3 P r o p o s i t i o n . Let X be a smooth Banach space and V: X —» R U {+00} a lower semicontinuous function on X. For any e > 0 , t h e r e is a dense subset of D o m V i n which each 3 obeys the following: there is a continuous linear functional fi €E X* and a scalar rj > 0 s u c h t h a t V(o) - V(a) > (fi, a - 3 ) - e | |a - 3 | | V a e 3 + rjB. I n o u r s e t t i n g X = L2 is c e r t a i n l y a s m o o t h s p a c e , a n d a c o n t i n u o u s l i n e a r f u n c t i o n a l fi € X* c a n b e i d e n t i f i e d w i t h a n e l e m e n t o f L2. T h e k e y i n e q u a l i t y o f P r o p . 4 .3 b e c o m e s VJ^W * (* i H i l ) - * - M - v . e * + ,(* \ V(a) V a . C l e a r l y , x is t h e u n i q u e s o l u t i o n o f P ( 0 ) , a n d i n p a r t i c u l a r V ( 0 ) = V ( 0 ) . W e w i s h t o use T h m . 4 . 1 t o find n e c e s s a r y c o n d i t i o n s f o r x. T h e p r e r e q u i s i t e f o r t h i s is t h a t J V e p . ^ ( 0 , V ( 0 ) ) ^ { 0 } . U n f o r t u n a t e l y , t h i s is n o t a n o b v i o u s c o n s e q u e n c e o f Nepiv{0,V[0)) {o}, e v e n t h o u g h V[a) > V(a) V a . ( F o r i n s t a n c e , i t is n o t n e c e s s a r y t h a t 7 ^ . ^ ( 0 , V ( 0 ) ) 2 Nepiv(0,V(0))—the r e a d e r is i n v i t e d t o find c o n t i n u o u s f u n c t i o n s V,V-.TL ~* R s u c h t h a t V[a) > V(a) V a , V(0) = V ( 0 ) , a n d JV p . - ( 0 , V ( 0 ) ) n J V e p i V ( 0 , V ( 0 ) ) = { 0 } . ) So t h i s p r o g r a m w i l l o n l y s u c c e e d i f w e s t r e n g t h e n t h e c o n s t r a i n t q u a l i f i c a t i o n . C a l m n e s s w i l l d o . 4 . 4 T h e o r e m . Suppose P is calm at 0. Then for any x €E Y there is a nonzero [fi, — A ) e L2 X R a n d a n a r c p ( ) s u c h t h a t (a) A > 0 , 0{t) > 0 a.e. , /3(t)'g(t,x(t)) = 0 a.e. ( b ) + a . e . (c) P ( 0 ) e i V C o ( x ( 0 ) ) , -p(T) = Xdl(x(T)). Proof. F i x a n y x e Y, a n d c o n s i d e r p r o b l e m P(ct) d e f i n e d a b o v e . S i n c e V ^ a ) > V(ct) f o r a l l a a n d V ( 0 ) = V " ( 0 ) , t h e c a l m n e s s o f p r o b l e m P a t 0 i m p l i e s t h e c a l m n e s s o f p r o b l e m P a t 0. H e n c e 7 V " e p . - ( 0 , V ( 0 ) ) 7^ { 0 } , a n d T h m . 4 . 1 g ives a n o n z e r o (fi,-A) e L2 x R a n d a n a r c (p(),q{)) s a t i s f y i n g c e r t a i n c o n d i t i o n s . T h e s e c o n d i t i o n s i m p l y t h a t q = 0 , a n d t h a t p ( - ) o b e y s t h e d e s i r e d c o n c l u s i o n s ( a ) - ( c ) . / / / / S e n s i t i v i t y A n a l y s i s . I n a d d i t i o n t o a l l o w i n g t h e d e r i v a t i o n o f n e c e s s a r y c o n d i t i o n s , T h m . 3 .1 l e n d s i t s e l f t o a n a n a l y s i s o f t h e m a r g i n a l e f fec ts o f p e r t u r b a t i o n s t o t h e s t a t e c o n s t r a i n t . - 6 2 -4 . 5 D e f i n i t i o n . L e t x b e a n a d m i s s i b l e F - t r a j e c t o r y a n d A a n o n n e g a t i v e s c a l a r . T h e n t h e p a i r ( a ) - ( c ) o f T h m . 4 . 4 h o l d . T h e c o l l e c t i o n o f a l l s u c h p a i r s is d e n o t e d Mx(x), a n d MX(Y) i s t h e u n i o n o f t h e se ts Mx (x) o v e r x e Y. W e d e f i n e a m a p p i n g A f r o m t h e s p a c e o f m u l t i p l i e r s t o L2 v i a A ( P ) y 9 ) = /9. 4 . 6 T h e o r e m . I f 7 ( 0 ) < + o o a n d (H1)-(H5) hold, then Y ^ 0 a n d N o t e t h a t T h m . 4 .6 h o l d s t r i v i a l l y i f dV(0) = 0, so t h e o n l y w a y t o ge t u s e f u l i n f o r m a t i o n f r o m i t i s t o i n t r o d u c e c o n d i t i o n s e x c l u d i n g t h i s p o s s i b i l i t y . A s P r o p . 4 . 2 s h o w s , t h e c a l m n e s s c o n d i t i o n is s u f f i c i e n t t o d o t h i s . T h e n e c e s s a r y c o n d i t i o n s o f V i n t e r a n d P a p p a s ( 1 9 8 2 ) o r o f C l a r k e ( 1 9 8 3 ) , T h m . 3 .2 .6 a r e as f o l l o w s . F o r s i m p l i c i t y , w e t a k e t h e s t a t e c o n s t r a i n t d i m e n s i o n a = 1 . 5.1 T h e o r e m . Assume (H1)-(H5), a n d assume moreover that g(t, x) is lower semicontinuous in t. If x solves P(0), there exist a scalar A > 0 , a n o n n e g a t i v e . R a d o n m e a s u r e p, and an a r c q such that (a) p is supported on the set S = {t (E [0 , T] : g(t, x(t)) = 0 } . (p,/3) 6 AC([0,T],Rn) x J D 2 ( [ 0 , T ] , R a ) i s a n index A multiplier corresponding t o a; i f c o n c l u s i o n s Proof. S i m i l a r t o t h e first p a r t o f T h m . 1.3.6. //// S e c t i o n 5 . C o m p a r i s o n t o K n o w n C o n d i t i o n s W e c a n o b t a i n t h e s e c o n c l u s i o n s f r o m T h m . 4 . 4 b y d e f i n i n g a n e w a rc q(-) as f o l l o w s : T h e c o n c l u s i o n s o f T h m . 4 .4 t h e n b e c o m e (a ) A > 0 , /9 ( t ) > 0 a.e. , g{t,x(t)) = 0 a.e. - 6 3 -( b ) ( j W ) edH(ttx(t),q(t)+J*gx(8,x(s))0(s)dsy ( c ) g ( 0 ) € NCo(x(0)), -q(T) - fgx{s,x(s))P(s) ds € Xdl{x[T)). Jo T h e a d v a n t a g e s o f T h m . 4 .4 n o w b e c o m e c l e a r : i t s h o w s t h a t t h e m e a s u r e y, a p p e a r i n g i n t h e k n o w n c o n d i t i o n s c a n b e a s s u m e d t o b e a b s o l u t e l y c o n t i n u o u s w i t h r e s p e c t t o L e b e s g u e m e a s u r e , w i t h a s q u a r e - i n t e g r a b l e d e n s i t y p. W e o b t a i n t h i s d e s i r a b l e c o n c l u s i o n b y a n e q u a l l y a t t r a c t i v e m e t h o d o f p r o o f , b a s e d o n t h e g e o m e t r i c a l s t r u c t u r e o f a c e r t a i n e p i g r a p h i n t i m a t e l y r e l a t e d t o t h e s t a t e c o n s t r a i n t i t s e l f . M o r e o v e r , g is a l l o w e d t o b e m e r e l y m e a s u r a b l e i n t. T h e c o s t o f t h e s e a d v a n c e s is a c o n s t r a i n t q u a l i f i c a t i o n ( c a l m n e s s ) w h i c h is n o t a s s u m e d i n t h e s t a n d a r d r e s u l t s c i t e d a b o v e , b u t w h i c h is k n o w n t o h o l d a r b i t r a r i l y n e a r a n y p r o b l e m o f i n t e r e s t . S m o o t h n e s s o f g i n x, a c o n d i t i o n m a n y p r o b l e m s o b e y , is a r e q u i r e m e n t o f o u r t h e o r y b u t is m e r e l y a s p e c i a l case o f t h e r e s u l t s c i t e d a b o v e . - 6 4 -C h a p t e r I V : E x i s t e n c e T h e o r y f o r a S t o c h a s t i c B o l z a P r o b l e m The previous chapters have confirmed the intimate connection between sensitivity to perturbations and necessary conditions in optimization problems. The analysis itself, however, makes frequent and essential use of limiting arguments—both in proving vital existence theorems and in the proximal normal formula itself. For this reason, any attempt to study the sensitivity of a stochastic control problem with proximal normals must begin with an investigation of appropriate limiting techniques. In this chapter we introduce the techniques of convergence in distribution, tightness, and martingale representation by showing them at work in a new existence theorem for a constrained Bolza problem of stochastic optimal control. These methods have been used before to study the existence of optimal stochastic control laws, but never in such generality as that of Thm. 5.1 below. Our work owes its basic approach to Kushner (1975), but it improves on his result by allowing the control set U to be unbounded, by treating objective functionals of Bolza form, and by allowing the incorporation of soft constraints. We also invoke stronger martingale representation theorems, and thereby weaken some of Kushner's technical hypotheses. Our method of proof also has the pedagogical advantage of replacing the use of Skorokhod's theorem with a well-known closure theorem from deterministic optimal control which clarifies the connection between deterministic and stochastic existence theories. Finally, our hypotheses are stated more explicitly than Kushner's, making them easier to verify in practice. This chapter reviews the required probability theory in Sections 1-3 and formulates the stochastic control problem precisely in Section 4. The main existence theorem is proven in Section 5, after which Section 6 adds several significant extensions. Section 7 is devoted to a comparison between the main existence theorem and its counterpart involving a compact control set. - 65 -Section 1. The Probability Background Standard textbooks on the general theory of stochastic processes like Jacod (1979), Dellacherie-Meyer (1975), and Ikeda-Watanabe (1981) present the extensive theoretical basis for our Chapters IV and V in detail. Our aim in this section is simply to collect those aspects of the theory which are critical to the development below. We begin with a summary of commonly used notation. Notation. [0, T] denotes a fixed time interval used throughout this chapter; Vt means Vt £ [0, T]. S denotes a metric space; S is Polish if it is complete and separable. S(5) denotes the Borel c-field of S. (B(S) = a{U C S : U is open.}.) Bm denotes B(R m). C t m denotes C([0, t], R m). CJ™ denotes B(C™), completed with respect to Wiener measure. See text below. Cm,Cm denote Cp, Cy, respectively. sAt denotes min{s,t}. xt(s) denotes x(s A t), when x(-) is a function. 7 X S denotes the product cr-field, when two a-fields 7 and S are given. We write x: (fi, 7) —+ (S, 5) if the mapping x: fl —* 5 is measurable with respect to the cr-fields 7 and S. When S = B[S), we sometimes express this by writing x € 7. Stochastic Processes. A filtered probability space (fi, 7, 7t,P) consists of a set fi equipped with a cr-field 7 on which a probability measure P is defined. The filtration {7t : t € [0, T]} is a family of sub-cr-fields of 7 obeying 7, C 7t whenever 0 < s < t < T. The usual hypotheses regarding such a filtered space are the following three conditions: (i) The measure space (fi, 7, P) is complete. (ii) The filtration 7% is right-continuous, i.e. 7t = 7t+ Vt, where 7t+ = C\,>o (iii) To 2 {A e 7 : P{A) = 0}. - 66 -A n S - v a l u e d stochastic process is a m e a s u r a b l e m a p p i n g x: [ 0 , T] X ft —• S; t h e p r o c e s s x is c a l l e d ^ - a d a p t e d i f x{t, •) is ^ - m e a s u r a b l e f o r e a c h t € [0, T ] . I t is c o n v e n t i o n a l t o s i m p l i f y t h e n o t a t i o n i n v o l v i n g t h e p r o c e s s x b y l e a v i n g t h e w - d e p e n d e n c e i m p l i c i t w h e n e v e r p o s s i b l e . T h u s Xt o r x(t) a re o f t e n u s e d t o d e n o t e e i t h e r t h e m a p p i n g x(t, •) o r o n e o f i t s v a l u e s x(t,cj)—context d i s t i n g u i s h e s t h e t w o p o s s i b i l i t i e s . A g i v e n p r o c e s s xt d e f i n e s a filtration {7tx : t S [0 , T ] } as f o l l o w s : 7tx = f r o m ft i n t o Cd. I n d e e d , a p a r t i c u l a r l y i m p o r t a n t e x a m p l e o f B r o w n i a n m o t i o n is o b t a i n e d w h e n (ft, 7 ) = {Cd, Cd) a n d t h e p r o b a b i l i t y m e a s u r e P is Wiener measure, d e n o t e d b y W. I n t h i s case t h e i d e n t i t y m a p is a B r o w n i a n m o t i o n , a n d (ft, 7 ) = (Cd,Cd) is c a l l e d canonical path space. T h e ( c o m p l e t e d ) filtration g e n e r a t e d b y t h e i d e n t i t y m a p is s i m p l y cd = JWJ. - 6 7 -I t o ' s I n t e g r a l . L e t (Cl,7,7t,P) b e a filtered s p a c e s a t i s f y i n g t h e u s u a l h y p o t h e s e s . S t o c h a s t i c i n t e g r a t i o n i n t h e sense o f K . I t 6 is a n o p e r a t i o n w h i c h e x c h a n g e s o n e s t o c h a s t i c p r o c e s s f o r a n o t h e r . I f t h e i n p u t is a n R n X d - v a l u e d p r o c e s s crt a d a p t e d t o 7t a n d o b e y i n g ( 1 . 1 ) E l T \°A2dr Jo < +00, t h e n f o r a n y ^ - B r o w n i a n m o t i o n w i n R d , t h e o u t p u t is a n R " - v a l u e d p r o c e s s d e n o t e d b y / 0 oy dwr. I n t e g r a l n o t a t i o n f o r t h e n e w p r o c e s s is j u s t i f i e d b y t h e m a n y s i m i l a r i t i e s b e t w e e n t h e o p e r a t i o n a l p r o p e r t i e s o f t h e s t o c h a s t i c i n t e g r a l a n d t h o s e o f c o n v e n t i o n a l m e a s u r e - t h e o r e t i c i n t e g r a l s . H o w e v e r , i t i s n o t m e a n t t o s u g g e s t a n y p a r t i c u l a r c o m p u t a t i o n a l s t r a t e g y . F o r i n s t a n c e , i t is u s u a l l y i n c o r r e c t t o t r y t o c o m p u t e f£ ar[ui) dwr(u>) as a R i e m a n n - S t i e l t j e s i n t e g r a l f o r e a c h fixed u. (See F l e m i n g a n d R i s h e l ( 1 9 7 5 ) , p . 1 1 2 , f o r a s t a n d a r d c o u n t e r e x a m p l e . ) T h i s t a c t i c o n l y w o r k s o n r a t h e r s i m p l e i n t e g r a n d s — f o r e x a m p l e , t h o s e w h i c h are s i m p l e p r e d i c t a b l e p rocesses i n t h e t e c h n i c a l sense. W e w i l l use t h e t w o i n e q u a l i t i e s b e l o w t h r o u g h o u t t h i s c h a p t e r . 1 . 1 P r o p o s i t i o n ( A n i n e q u a l i t y o f B u r k h o l d e r ) . L e t wt be a Brownian motion on R d , a n d let at be an n x d-matrix valued process obeying (l.l). Then for any 6, e > 0 one has J > | J{)(xrdwr > e | <5/e 2 + p|y" | a r | 2 d r ><$J Vte[0,T]. 1 . 2 P r o p o s i t i o n ( B u r k h o l d e r - D a v i s - G u n d y ) . For any exponent p e [0, 00) there is a constant Cp, depending only upon p and n , s u c h t h a t for any process a as i n Proposition 1.1, (BDG) E ^ ' Or dwr < C p E \ 0 : y{A) < z{A') + e, z[A) < y{Ae) + e, 6 S ) . ( H e r e A' d e n o t e s t h e set {t £ [0, T] : d i s t ( t , . A ) < e } . ) I n f a c t d m a k e s Z[0, T] i n t o a c o m p l e t e s e p a r a b l e m e t r i c s p a c e . T h e s e f a c t s a re s p e c i a l cases o f t h e r e s u l t s i n B i l l i n g s l e y ( 1 9 6 8 ) A p p e n d i x I I I ; t h e y a p p e a r m o r e e x p l i c i t l y i n P r o k h o r o v ( 1 9 5 6 ) . I n t e r m s o f t h e i n c r e a s i n g f u n c t i o n s u s e d t o d e f i n e t h e s p a c e Z\Q, T] i n t h e first p l a c e , t h e t o p o l o g y o f w e a k c o n v e r g e n c e c o r r e s p o n d s t o p o i n t w i s e c o n v e r g e n c e a t c o n t i n u i t y p o i n t s ( B i l l i n g s l e y ( 1 9 6 8 ) , S e c t . 3 , p . 1 7 ) . T h e f o l l o w i n g p r o p o s i t i o n s u m m a r i z e s t h e s e o b s e r v a t i o n s a n d i n c o r p o r a t e s t h e H e l l y - B r a y s e l e c t i o n t h e o r e m . l . S P r o p o s i t i o n . Z[0, T] is a complete separable metric space in which a sequence of functions {zk} converges to z if and only if zk(t) —• z{t) for each point t where z is continuous, and for t = T. A closed subset S of Z\0, T] is compact if and only if there is a constant M > 0 s u c h t h a t z[T) < M for all z e S. S e c t i o n 2. C o n v e r g e n c e i n D i s t r i b u t i o n W h e n p r o v i n g e x i s t e n c e t h e o r e m s f o r o p t i m i z a t i o n p r o b l e m s b y t h e " d i r e c t m e t h o d , " o n e seeks t o i s o l a t e a s o l u t i o n as t h e l i m i t o f a w e l l - c h o s e n m i n i m i z i n g s e q u e n c e . P r o x i m a l n o r m a l a n a l y s i s a lso r e l i e s o n l i m i t i n g a r g u m e n t s . T h e s e t w o c o n s i d e r a t i o n s m o t i v a t e a s e a r c h f o r t h e a p p r o p r i a t e n o t i o n o f l i m i t s i n s t o c h a s t i c o p t i m a l c o n t r o l p r o b l e m s . W e a k c o n v e r g e n c e o f p r o b a b i l i t y m e a s u r e s a p p e a r s t o b e t h e c o r r e c t a n s w e r : i n t h i s s e c t i o n w e r e v i e w t h i s m o d e o f c o n v e r g e n c e . - 6 9 -2 . 1 D e f i n i t i o n . L e t a n y m e t r i c s p a c e S b e g i v e n , a n d s u p p o s e t h a t P a n d Pk, k = 1 , 2 , . . . , a r e p r o b a b i l i t y m e a s u r e s o n (S, B(S)). T h e n t h e s e q u e n c e o f m e a s u r e s Pk converges weakly to P, d e n o t e d Pk P, i f a n d o n l y i f o n e o f t h e f o l l o w i n g t h r e e e q u i v a l e n t c o n d i t i o n s is s a t i s f i e d . (a ) / f{s) dPk(s) —+ / f{s) dP[s) f o r a l l b o u n d e d , u n i f o r m l y c o n t i n u o u s f:S —+ R . Js Js ( b ) K m s u p Pk (F) < P{F) f o r a l l c l o s e d sets F C S. k—>-oo ( c ) l i m Pk{A) = P[A) f o r a l l se ts A G 8{S) s u c h t h a t P ( b d y A ) = 0 . fc—*-oo ( T h e e q u i v a l e n c e o f c o n d i t i o n s ( a ) - ( c ) is p r o v e n i n B i l l i n g s l e y ( 1 9 6 8 ) , p . 11 . ) I t is c l e a r t h a t t h e w e a k l i m i t o f a s e q u e n c e o f p r o b a b i l i t y m e a s u r e s is u n i q u e . W e a k c o n v e r g e n c e o f m e a s u r e s is i n t i m a t e l y r e l a t e d t o t h e n o t i o n o f " t i g h t n e s s . " A f a m i l y T I o f p r o b a b i l i t y m e a s u r e s o n (S, B(S)) i s tight i f f o r e v e r y e > 0 t h e r e is a c o m p a c t set K C S f o r w h i c h e v e r y P i n I I o b e y s P(K) > 1 - e. T h e f o l l o w i n g t h e o r e m , p r o v e n i n B i l l i n g s l e y ( 1 9 6 8 ) , S e c t i o n 6, p . 3 5 , s h o w s w h y t i g h t n e s s is so i m p o r t a n t . 2.2 T h e o r e m ( P r o k h o r o v ) . Let S be a Polish space on which a family I I of probability measures is given. Then I I is t i g h t if and only if every sequence chosen from TI h a s a w e a i i y convergent subsequence. W h e n e v e r a r a n d o m e l e m e n t x: ( f t , 7, P) —• (S,B(S)) is g i v e n , a m e a s u r e Px i s i n d u c e d o n ( 5 , B{S)) as f o l l o w s : PX{A) = (Po I - 1 ) (A) = P{w : x{w) S A} V i G B{S). T h i s o b s e r v a t i o n sets u p a c o r r e s p o n d e n c e b e t w e e n r a n d o m e l e m e n t s a n d p r o b a b i l i t y m e a s u r e s w h i c h a l l o w s t h e p r e c e d i n g n o t i o n s t o b e r e w o r d e d as f o l l o w s . S u p p o s e xL: ( f 2 M 7t, PL) —» (S, B(S)), t G / , is a f a m i l y o f r a n d o m e l e m e n t s o f S. T h i s f a m i l y is tight i f f o r e v e r y e > 0 t h e r e is a c o m p a c t s u b s e t K o f 5 1 s u c h t h a t ( P t ° x Z l ) [ K ) > l - e Vie I. I f I = N , t h e n t h e s e q u e n c e xL converges in distribution t o t h e l i m i t x: (Q, 7, P) (S, B(S)) i f t h e i n d u c e d m e a s u r e s PL o i " 1 c o n v e r g e w e a k l y t o P o W e d e n o t e t h i s b y xL-^-*x. P r o k h o r o v ' s t h e o r e m t a k e s t h e f o l l o w i n g f o r m i n t h i s a l t e r n a t e t e r m i n o l o g y . - 7 0 -2 . 3 T h e o r e m ( P r o k h o r o v ) . Let S be a Polish space equipped with a family of random elements E = {xl:(n„Z,Pl)-+(S,B(S))}ieJ. The family S is tight if and only if every sequence Xi chosen from E has a subsequence converging in distribution to some random element x: (Cl, 7,P) —* (S, B(S)). A n i m p o r t a n t c o n s e q u e n c e o f c o n v e r g e n c e i n d i s t r i b u t i o n a r i ses f r o m D e f . 2 . 1 ( b ) . I f Xi~^-*x a n d F i s a c l o s e d s u b s e t o f S, t h e n P{x €F}> l i m s u p P , { x , <= P } . «'—»oo T h i s f a c t w i l l b e u s e d i n S e c t i o n 5. T h e f u n d a m e n t a l r o l e o f t i g h t n e s s m a k e s i t i m p o r t a n t t o b e ab le t o d e t e c t t h i s p r o p e r t y . B i l l i n g s l e y ( 1 9 6 8 ) , T h m . 8 .2 , p . 5 5 , g ives t h e f o l l o w i n g c r i t e r i o n i n t h e space S = Cn. 2 . 4 P r o p o s i t i o n . A sequence s^ : ( f i , , 7i,Pi) —* (Cn,Cn) of random elements of Cn is tight if and only if the following two conditions hold: (i) l i m s u p Pi { | z , ( 0 ) | > R} = 0 , it—>oo (ii) l i m s u p P j < « — 0 + i>N JV — oo ~ s u p \xi(t) — Xi(s) \ > e 0 = 0 V e > 0 . S e c t i o n 3 . M a r t i n g a l e s a n d T h e i r R e p r e s e n t a t i o n s L e t us fix a filtered space (Cl, 7, 7t, P) s a t i s f y i n g t h e u s u a l h y p o t h e s e s t h r o u g h o u t t h i s s e c t i o n . A n ^ - a d a p t e d p r o c e s s m t t a k i n g v a l u e s i n Rn a n d o b e y i n g m o = 0 is a n 7t-martingale i f E | m t | < + o o V t € [0 , T\ a n d i f (3 .1 ) 0 < s < t < T E [ m t | 7,} = m , a.s. C l e a r l y a n y ^ - m a r t i n g a l e is a lso a n ^ " - m a r t i n g a l e . A square-integrable martingale is a m a r t i n g a l e w h i c h o b e y s ( 3 . 2 ) E | m t |2 < + o o V t e [0 , T\\ -71-i f m t h a s c o n t i n u o u s s a m p l e p a t h s , t h e n t h e r e is a u n i q u e c o n t i n u o u s i n c r e a s i n g n o n n e g a t i v e d e f i n i t e n X n m a t r i x - v a l u e d p r o c e s s qt s u c h t h a t go = 0 a n d (mt) ( m t ) ' — qt is a m a t r i x - v a l u e d ^ - m a r t i n g a l e . T h e p r o c e s s qt is c a l l e d t h e quadratic variation of mt, a n d d e n o t e d b y ( m ) t . F o r e a c h i,j = 1 , 2 , . . . , n , l e t (m l,m3)t d e n o t e t h e s c a l a r p r o c e s s d e f i n e d b y t h e ( i , y ) - c o m p o n e n t o f (m)t. T h e n m\m\ — (mxtm3'') is a s c a l a r - v a l u e d ^ - m a r t i n g a l e . T h e f o l l o w i n g p r o p o s i t i o n s h o w s t h a t f o r c o n t i n u o u s p r o c e s s e s , c o n v e r g e n c e i n d i s t r i b u t i o n p r e s e r v e s t h e m a r t i n g a l e p r o p e r t y . I t uses t h e n o t i o n o f " u n i f o r m i n t e g r a b i l i t y , " f o r w h i c h a n e x c e l l e n t r e f e r e n c e is B i l l i n g s l e y ( 1 9 6 8 ) , p p . 3 2 - 3 3 . 3 . 1 P r o p o s i t i o n . Let ( x f c , m f c ) : (nfc, ?k,Pk) -* ( C n + t , C n + t ) be a s e q u e n c e of random elements of Cn+t. Defining ?k to be the Bltration generated by (xk,mk), suppose that mk is an Jk-martingale for each fc. If (xk, m f c ) - ^ - » ( x , m ) for some random element ( x , m ) : ( f i , 7 , P) —» ( C n + t , C n + t ) and if the sequence {\\mk\\} i s uniformly integrable, then the limit process mt is an ?t-martingale. Here 7t is the Bltration generated by ( x , m ) . Proof. F i x a n y J V e N a n d c h o o s e a n y 0 < s < t < T a n d 0 < ti < t2 < • • • < t^ < s. L e t p: R ' J V n + J V £ ' R b e a n a r b i t r a r y b o u n d e d , u n i f o r m l y c o n t i n u o u s f u n c t i o n . T h e n f o r e a c h k, t h e m a r t i n g a l e c h a r a c t e r o f mk i m p l i e s E g (xk{t1),xk{tN),mk{t1),mk(tN)) (mk(t) - mk{s)) = 0. N o w s ince is a u n i f o r m l y i n t e g r a b l e s e q u e n c e , so is t h e s e q u e n c e o f r e a l - v a l u e d r a n d o m v a r i a b l e s w h o s e e x p e c t a t i o n s a re c o m p u t e d a b o v e . M o r e o v e r , t h i s s e q u e n c e c a n b e v i e w e d as t h e i m a g e o f t h e s e q u e n c e (xk,mk) u n d e r a c o n t i n u o u s m a p G : C n + l —• R . H e n c e t h e s e q u e n c e o f i n t e g r a n d s c o n v e r g e s i n d i s t r i b u t i o n t o G(x,m): b y u n i f o r m i n t e g r a b i l i t y i t f o l l o w s t h a t w e c a n l e t —» oo a b o v e t o o b t a i n E g[x{t1),...,x{tN),m(t1),...tm(t1f)){rn[t)-rn(a)) = 0 . S i n c e N, g, a n d ti,..., t # are a r b i t r a r y , t h i s s h o w s t h a t E [ m t — m, | 7,] = 0 , as r e q u i r e d . / / / / - 72 -S u p p o s e t h a t (Cl, 7, 7t,P) c a r r i e s a n ^ j - B r o w n i a n m o t i o n wt w i t h v a l u e s i n Rd. I f a n ^ - a d a p t e d p r o c e s s ot w i t h v a l u e s i n R n X d is g i v e n s a t i s f y i n g ( 1 . 1 ) , w e c a n d e f i n e j*ordwr. This process is a martingale. I n o t h e r w o r d s , 0 < s < t < T E ^ or dwr 7, j = J oT dwT a.s T h e q u a d r a t i c v a r i a t i o n o f t h i s m a r t i n g a l e is t h e n X n m a t r i x - v a l u e d p r o c e s s (3.3) j^f oTdw)j = j oro'Tdr; i n p a r t i c u l a r , i f 2 r* (3.4) E / oT dwr = E / \or\2 dr, \Jo J 0 w h e r e \o\2 — tr(oo'). P r o p o s i t i o n s 1.1 a n d 1.2 a b o v e a re a c t u a l l y s p e c i a l cases o f i n e q u a l i t i e s v a l i d f o r a r b i t r a r y m a r t i n g a l e s . I t is m o r e t h a n a h a p p y c o i n c i d e n c e t h a t t h e p r o c e s s f£ or dwT t u r n s o u t t o b e a m a r t i n g a l e . T h e r e l a t i o n s h i p b e t w e e n m a r t i n g a l e s a n d s t o c h a s t i c i n t e g r a l s is s t r o n g e n o u g h t h a t a s o r t o f c o n v e r s e t o (3.3) is a v a i l a b l e . I n t h e case o = I, i t t a k e s t h e f o l l o w i n g f o r m : 3.2 L e m m a (Doob). If mt is a n 7t-martingale with continuous R d - v a i u e d sample paths for which E |mt|2 < + o o V t > 0 a n d (m)t = tl, then m is an 7t-Brownian motion. The converse is also true. D o o b ' s l e m m a c a n b e g e n e r a l i z e d s u b s t a n t i a l l y . I n f a c t , v i r t u a l l y a n y c o n t i n u o u s m a r t i n g a l e w h o s e q u a d r a t i c v a r i a t i o n is a b s o l u t e l y c o n t i n u o u s w i t h r e s p e c t t o L e b e s g u e m e a s u r e c a n b e r e p r e s e n t e d as a s t o c h a s t i c i n t e g r a l . L e t u s m a k e t h i s p r e c i s e . S u p p o s e a c o n t i n u o u s R n - v a l u e d m a r t i n g a l e m is g i v e n o n o u r fixed space ( f i , 7,7t,P), a n d t h a t ( m ) t = f* oro'r dr f o r s o m e p r e d i c t a b l e n x d m a t r i x v a l u e d p r o c e s s o w i t h d < n. I n a c e r t a i n r i g o r o u s l y d e f i n a b l e sense, i t f o l l o w s t h a t t h e r e is a B r o w n i a n m o t i o n w i n R d w i t h r e s p e c t t o w h i c h m t = /Q or dwr. T h i s is t h e c o n t e n t o f P r o p . 3.3, b e l o w . B e f o r e s t a t i n g t h e r e p r e s e n t a t i o n t h e o r e m p r e c i s e l y , l e t us n o t e t h a t i t m a y b e i m p o s s i b l e t o c o n s t r u c t w o n t h e g i v e n space (Cl, 7,7t,P). F o r e x a m p l e , i f mt = 0 a n d t h e er- f ie lds 7 a n d 7t a re a l l t r i v i a l , t h e n t h e r e is s i m p l y n o r o o m f o r a n ^ j - B r o w n i a n m o t i o n o f a n y d i m e n s i o n o n t h i s s p a c e . T o eliminate this possibility, we augment the given space with a copy of canonical d-dimensional Wiener space (Cd,Cd,Cf, W) as follows: (n, f, P) = (n x cd, 7 x cd, P x w) (3,5) %=n ( * + * x cun). h>0 Now (fi, 7, ft,P) is a filtered space satisfying the usual hypotheses, and any random variable x originally defined on ft can be readily replaced by a random variable x = i o f on fl, where 7r(u>,ct/) = w is the natural projection. Clearly x=z. The fine points of the extension (3.5) are explored in detail by Jacod (1979), Section X.2(b), p. 332. He shows that for the natural embedding i r ~ 1 ( 7 ) of the original cr-field 7 in f, a random variable x defined on (ft, 7, P) is T T - 1 (5")-measurable if and only if x = x o 7r for some random variable x on (ft, 7, P) with x=x. Also, ir~ 1(7t) = n~ 1(7) n ft> and any 7T-1(7)-measurable random variable x = x o n obeys E[2 | %] =E[i | ir _1(7 t)] = E[i | 7t] o 7T P - a.s. Thus all relevant properties of x and are retained in the passage to x, ft, while the enlarged space (ft, f, ft,P) most assuredly has room to contain a d-dimensional Brownian motion. We may now state the promised representation theorem, essentially due to Doob. The version here, which allows d < n and imposes no nonnegativity condition on cr, is taken from Jacod (1979), Thm. (14.45), p. 466. 3.S Proposition. Let mt be a continuous Rn-vaiued 7t-maxtingale with (m)t = /Q oro'T dr for some 7t-predicta.ble n X d matrix valued process cr with d < n. Then there is a d-dimensional 7t-Brownian motion wt on (ft, f, ft, P) such that, with fht = mt ° ?r and 3y = oy o w, one has fht= f or dwT Vt e [0, T\, P - a.s. Jo Moreover, if the rank of the matrix a is identically equal to d, this conclusion remains valid with (ft, 7, 7t, P) = (ft, 7, 7t, P) and TT = identity. That is, no extension of the given space is required. A significant application of Prop. 3.3, first noted by Wong (1971), deals with "quasimartingales." Let x be an ^-adapted process with cadlag sample paths in Rn. For any partition 0 < t\ < t% < - 74 -• • < tk < T o f [0, T], d e f i n e fc ( 3 . 6 ) v a r (x-M tk) = ^2\E[x{ti+1)-x{ti) \ 7tt]\ + \x{T)\, »=1 V a r ( x ) = s u p { E v a r ( x ; t i , . . . , t f c ) : k e N , 0 < i i < • • • < tk < T) . T h e p r o c e s s x is a quasimartingale i f V a r ( x ) < + o o . N o t e t h a t a n y ^ - m a r t i n g a l e m « is a u t o m a t i c a l l y a q u a s i m a r t i n g a l e , s i n c e f o r a n y p a r t i t i o n a n d E | m r | < + o o . ( I n p a r t i c u l a r , a B r o w n i a n m o t i o n is a q u a s i m a r t i n g a l e e v e n t h o u g h a l m o s t a l l i t s s a m p l e f u n c t i o n s h a v e u n b o u n d e d v a r i a t i o n o n e v e r y i n t e r v a l . ) T h e l i n e a r i t y o f c o n d i t i o n a l C o n v e r s e l y , w e m i g h t e x p e c t t h a t a t y p i c a l q u a s i m a r t i n g a l e s h o u l d b e d e c o m p o s a b l e i n t o t h e s u m o f a m a r t i n g a l e a n d s o m e o t h e r p r o c e s s . T h e p r o p e r t i e s o f t h e o t h e r s u m m a n d a re d e s c r i b e d i n t h e f o l l o w i n g t h e o r e m , o r i g i n a l l y p r o v e n b y F i s k , b u t q u o t e d h e r e f r o m J a c o d ( 1 9 7 9 ) , T h m ( 5 . 3 6 ) , p . 174 . 3 . 4 P r o p o s i t i o n . A given process x on ( f ) , 7, Tt,P) is a quasimartingale if and only if where mt is an It-martingale and at is a predictable process whose sample paths have finite variation on [0 , T] P — a.s. This decomposition is unique. J a c o d ( 1 9 7 9 ) , P r o p . ( 9 . 1 4 ) , p . 2 8 5 , a lso s h o w s t h a t i f a g i v e n q u a s i m a r t i n g a l e x o n ( f2 , T, ?t,P) h a p p e n s t o b e p t - a d a p t e d , w h e r e Qt is a filtration s a t i s f y i n g t h e u s u a l h y p o t h e s e s a n d Qt Q 7t V t , t h e n x is a lso a q u a s i m a r t i n g a l e w i t h r e s p e c t t o (ft, 7,Qt,P). O f c o u r s e , t h e c a n o n i c a l d e c o m p o s i t i o n o f x g i v e n i n (3 .7 ) m a y c h a n g e w h e n o n e passes t o t h i s s m a l l e r filtration. E u g e n e W o n g ( 1 9 7 1 ) h a s g i v e n a n e x p l i c i t c o n s t r u c t i o n o f t h e ( ^ - r e p r e s e n t a t i o n o f a q u a s i m a r t i n g a l e w h o s e c a n o n i c a l ^ - d e c o m p o s i t i o n t a k e s t h e f o r m v a r ( m ; t i , . . . , t f c ) = 0 + | m r | e x p e c t a t i o n i m p l i e s t h a t t h e s u m o f a q u a s i m a r t i n g a l e w i t h a m a r t i n g a l e r e m a i n s a q u a s i m a r t i n g a l e . (3 .7 ) x t - x 0 = m t + o t V i e [0 , T], P - a .s. , ( 3 . 8 ) W e q u o t e h i s r e s u l t as T h m . 3.6 b e l o w , a s s u m i n g t h a t / is a n ^ J - a d a p t e d p r o c e s s o b e y i n g (3 .9 ) - 75 -a n d t h a t m « is a c o n t i n u o u s s e c o n d - o r d e r ^ t - m a r t i n g a l e w i t h ( 3 . 1 0 ) E ( m ) r < +00. W o n g ' s o r i g i n a l p r o o f is p h r a s e d s o m e w h a t d i f f e r e n t l y , a n d is v a l i d i n t h e m o r e g e n e r a l case w h e n m is a " l o c a l l y s q u a r e i n t e g r a b l e m a r t i n g a l e . ' ' I t r e l i e s o n t h e f o l l o w i n g l e m m a . 3 . 5 L e m m a ( W o n g ) . Let 7t and Qt be two Bltrations of (CI, 7,P) satisfying the usual hypotheses, and suppose that mt and nt are continuous second-order 7f and §t-martingales. If mt — n* i s a process whose sample paths have bounded variation, then (m)t = ( n ) t V i € [0 , T] a.s. 3 . 6 T h e o r e m ( W o n g ) . L e t x be an R n - v a i u e d ^ - m a r t i n g a i e of the form (3.8) satisfying (3.9) and (3.10). Suppose that there is an 7t-predictable n x d matrix valued process cr with d < n such that (m)t = JQ cTo'r dr. Then for any Sltration Qt obeying the usual hypotheses and 7* — 9t — ?t, there is a d-dimensional §t-Brownian motion wt on (CI, ?,"§t, P) such that Proof. W e f o l l o w W o n g ( 1 9 7 1 ) , T h m . 4 . 2 , p . 6 2 9 . I t is k n o w n t h a t Xt is a q u a s i m a r t i n g a l e w i t h r e s p e c t t o t h e filtration Qt- T h e r e f o r e t h e r e is a V i £ [0, T\, P - a.s. (Here (CI, f, P) is the product space defined in (3.5).) £ t - m a r t i n g a l e nt a n d a ^ - p r e d i c t a b l e p r o c e s s a t o f b o u n d e d v a r i a t i o n s u c h t h a t ( 3 . 1 1 ) Xt = XQ + at + nt V i e [0,T], P- a.s. W o n g ' s p r o o f ( h i s p . 6 3 0 ) s h o w s t h a t ( 3 . 1 2 ) a.s. S u b t r a c t i n g r e p r e s e n t a t i o n ( 3 . 1 1 ) f r o m t h e o r i g i n a l r e p r e s e n t a t i o n ( 3 . 8 ) l e a d s t o mt-nt = at- fr dr V i e [ 0 , T], P - a.s. - 76 -N o w t h e r i g h t s ide h e r e is a p r o c e s s w h o s e s a m p l e p a t h s h a v e b o u n d e d v a r i a t i o n , P — a.s. So b y L e m m a 3 .5 , w e h a v e (n)t = (m)t = j cTa'T dr V t e [0, T], P - a.s. Jo H e n c e P r o p . 3.3 g ives a n R e v a l u e d B r o w n i a n m o t i o n to o n (fl, 7, $ti P) s u c h t h a t ( 3 . 1 3 ) nt=far dwr V t e [0 , T], P - a.s. Jo A s w e h a v e seen a b o v e , ( 3 . 1 2 ) i m p l i e s t h a t o n t h e e x t e n d e d space (fl, 7, P), ( 3 . 1 4 ) at = f E [ £ | §r] dr V t € [ 0 , T ] , P - a.s. Jo C o m b i n i n g ( 3 . 1 1 ) , ( 3 . 1 3 ) , a n d ( 3 . 1 4 ) g i ves t h e d e s i r e d r e s u l t . / / / / B e c a u s e o f t h e s i m p l e s t r u c t u r e o f t h e p r o d u c t space (fl, 7, P) a n d t h e f a c t t h a t a l l p rocesses x o n t h e o r i g i n a l s p a c e (fl, 7,P) r e t a i n t h e i r e s s e n t i a l p r o p e r i t e s i n t h e p a s s a g e t o (fl, f,P), t h e s u p e r s c r i p t t i l d e is o f t e n s u p p r e s s e d i n a p p l i c a t i o n s o f W o n g ' s t h e o r e m . W e w i l l use t h i s c o n v e n t i o n i n t h e s e c t i o n s t o f o l l o w . S e c t i o n 4 . P r o b l e m F o r m u l a t i o n S t o c h a s t i c D y n a m i c s . W e s t u d y a r a n d o m d y n a m i c a l s y s t e m w h o s e s t a t e x e v o l v e s i n R " u n d e r t h e i n f l u e n c e o f a B r o w n i a n m o t i o n w i n Hd (d < n) a n d a c o n t r o l s i g n a l u c h o s e n f r o m a p r e a s s i g n e d c l o s e d set U C R m . T h e m o t i o n t a k e s p l a c e o n t h e ( g i v e n ) finite t i m e i n t e r v a l [0 , T], s t a r t i n g f r o m a fixed i n i t i a l v a l u e XQ. T h u s t h e d y n a m i c s a re d e s c r i b e d b y t h e I t o e q u a t i o n (4 .1 ) xt = x0 + / f(r,x,ur)dr+ / er(r , x)dwT. Jo Jo I t is c o n v e n t i o n a l t o w r i t e ( 4 . 1 ) i n d i f f e r e n t i a l f o r m as f o l l o w s ( 4 . 2 ) dxr = / ( r , x, ur) dr + 0 a n d f) E ( 0 , 1 ) s u c h t h a t k(t, *)| < «! (i + ||i||f) vt e [o, r], i e cn. T h i s a s s u m p t i o n e v i d e n t l y i m p l i e s t h a t f o r s o m e fci, o n e h a s \a{t,x)\ 2. a n d a s s u m e (H1)-(H2). I / E | z 0 | p < K0 a n d E / 0 T | u r | p d r < K, then for any control-state pair (u, x) one has E | | z f < M , f o r a constant M which depends only on T, k\, n , p, KQ, and K. In particular, M is independent of the specific choices of XQ and u obeying the indicated conditions. - 78 -Proof. H y p o t h e s i s ( H 2 ) a n d ( B D G ) s p o n s o r t h e f o l l o w i n g c a l c u l a t i o n . | z t | < | z o | + / \f{r,x,ur)\dr+\ a(r,x)dwr Jo Uo = » | x t |p < KP ^ | x 0 | p + T"'1 J* fcp(l + | | x | | r + | u r | ) p d r + IjT* a(r,s) = • E | | x | | P < Kp (K0 + T ^ k l K , (T + j f E | | x | | P d r + i f ) + i f p C p J\l + "E \\xfr) d r ) => E H ^ C o + C i rE | | x | | P d r Jo f o r s o m e c o n s t a n t s Kp, Co, Cx. B y G r o n w a l l ' s i n e q u a l i t y , M = CoeClT su f f i ces . / / / / T h e O b j e c t i v e F u n c t i o n a l . T h e c o s t o f a g i v e n c o n t r o l - s t a t e p a i r ( u , x ) is m e a s u r e d b y t h e f u n c t i o n a l ( 4 . 3 ) A [ u , x ] : = E £ ( x T ) E x r ) + / L(r,x,ur)dr Jo A n admissible pair ( u , x ) is a c o n t r o l - s t a t e p a i r f o r w h i c h A [ u , x ] < +00. T h e p o i n t w i s e c o s t £ : R 2 n R a n d t h e r u n n i n g c o s t L: [0 , T\ X Cn X U R i n ( 4 . 3 ) m u s t s a t i s f y ( H 4 ) - ( H 5 ) b e l o w . ( H 4 ) L(t, x , u ) is m e a s u r a b l e i n t, c o n t i n u o u s a n d C ^ - m e a s u r a b l e i n x , a n d c o n t i n u o u s i n u ; I is c o n t i n u o u s . ( H 5 ) T h e f u n c t i o n t t a k e s o n n o n n e g a t i v e v a l u e s , a n d t h e r e e x i s t s a > 0 s u c h t h a t L(t, x , u ) > a | u | 2 f o r a l l ( t , x , u ) € [0, T ] x C " 1 x [/ . ( N o t e t h a t i f U is a c o m p a c t se t , t h i s r e q u i r e m e n t c a n b e r e p l a c e d b y t h e a s s u m p t i o n t h a t L > 0.) C e s a r i ' s C o n d i t i o n . E v e n i n t h e d e t e r m i n i s t i c s p e c i a l case o f o u r e x i s t e n c e T h m . 5 . 1 b e l o w , a c e r t a i n u p p e r s e m i c o n t i n u i t y p r o p e r t y m u s t b e a s s u m e d . T h i s h y p o t h e s i s n e e d s n o f o r t i f i c a t i o n i n t h e s t o c h a s t i c case. I t is s t a t e d i n t e r m s o f t h e m u l t i f u n c t i o n Q: [0 , T] X Cn <—* R n x R d e f i n e d b y Q(t,x) = {(f(t,x,u),L(t,x,u) +r) : u e U, r > 0 } . C e s a r i ' s c o n d i t i o n is t h e f o l l o w i n g h y p o t h e s i s . ( H 6 ) F o r a l l (t, x) G \0,T] X Cn, w i t h t h e p o s s i b l e e x c e p t i o n o f a set w h o s e p r o j e c t i o n o n t o t h e t - a x i s h a s L e b e s g u e m e a s u r e z e r o , o n e h a s (Q) Q(t,x)=f]co |J Q(s,y). €>0 | » - t | < « ||y-x||<* N o t e t h a t u n d e r ( H 6 ) , Q(t,x) m u s t b e c l o s e d a n d c o n v e x b e c a u s e i t is t h e i n t e r s e c t i o n o f c l o s e d c o n v e x se ts . I t is easy t o v e r i f y t h a t u n d e r o u r s t a n d i n g a s s u m p t i o n t h a t U is c l o s e d , t h e g r o w t h c o n d i t i o n o f ( H 5 ) a n d o u r c o n t i n u i t y c o n d i t i o n s o n / , L a u t o m a t i c a l l y m a k e Q(t, x) c l o s e d . F l e m i n g a n d R i s h e l ( 1 9 7 5 ) , S e c t . I I I . 4 , p . 68 s h o w t h i s . M o r e o v e r , t h e y p r o v e t h a t i f / a n d L a re c o n t i n u o u s i n t a t s o m e p o i n t t o a n d Q(to,xo) i s k n o w n t o b e c o n v e x f o r s o m e x o , t h e n p r o p e r t y ( Q ) h o l d s a t (to,xo). ( T h e i r L e m m a I I I . 5 . 4 , p . 72 . ) M o r e g e n e r a l c o n d i t i o n s i m p l y i n g ( H 6 ) a re g i v e n b y C e s a r i ( 1 9 8 3 ) . P r o b l e m ( P ) . T h e s t o c h a s t i c c o n t r o l p r o b l e m ( P ) is t o c h o o s e a n a d m i s s i b l e p a i r ( u , x ) s u c h t h a t A [ u , x ] e q u a l s t h e i n f i m u m o f A [ u , x ] o v e r a l l a d m i s s i b l e p a i r s ( u , x ) as d e f i n e d a b o v e . T h i s l a t t e r n u m b e r is d e n o t e d i n f ( P ) : i t is d e f i n e d e v e n i f n o o p t i m a l p a i r ( u , x ) e x i s t s . ( I f t h e r e a re n o a d m i s s i b l e p a i r s , t h e n i n f ( P ) = + o o . ) D i s c u s s i o n o f H y p o t h e s e s . T h e m e a s u r a b i l i t y a n d c o n t i n u i t y c o n d i t i o n s o f ( H i ) a r e s t a n d a r d . I n K u s h n e r ( 1 9 7 5 ) t h e y a re s u p p l e m e n t e d b y a n a s s u m p t i o n , d e n o t e d ( A 3 ) , t h a t d = n a n d t h a t a is t h e p o s i t i v e s q u a r e r o o t o f E = aa'. T h e s e c o n d i t i o n s m a y b o t h b e t r a c e d t o K u s h n e r ' s use o f W o n g ' s t h e o r e m , w h o s e o r i g i n a l f o r m r e q u i r e d t h e s e c o n d i t i o n s . H o w e v e r , t h e v e r s i o n o f W o n g ' s t h e o r e m p r e s e n t e d a b o v e as T h m . 3.6 a v o i d s t h e s e h y p o t h e s e s — a n i m p r o v e m e n t r e s u l t i n g d i r e c t l y f r o m J a c o d ' s c a r e f u l f o r m u l a t i o n o f o u r P r o p . 3 .3 . T h e g r o w t h c o n d i t i o n s o f ( H 2 ) a re u s e d n o t o n l y i n L e m m a 4 . 1 , b u t a lso t o e s t a b l i s h t i g h t n e s s a n d u n i f o r m i n t e g r a b i l i t y i n S e c t i o n 5. S i m i l a r c o n d i t i o n s are r e q u i r e d e v e n i n t h e d e t e r m i n i s t i c case. T h e p r e s e n c e o f t w o c o n d i t i o n s o n a, t h e s e c o n d i m p l i e d b y t h e first, r e f l e c t s a d e s i r e t o m a k e d o w i t h t h e s e c o n d o n e t h r o u g h o u t . O u r d i s t i n c t i o n b e t w e e n K,I a n d ki c l a r i f i e s t h e f a c t t h a t t h e s t r i c t i n e q u a l i t y 0 < 1 is n e e d e d o n l y i n t h e p r o o f o f P r o p . 5 .5 : e v e r y w h e r e else w e use t h e w e a k e r c o n d i t i o n i n v o l v i n g fci. T o b e m o r e s p e c i f i c , t h e p r o o f s o f P r o p . 5.5 a n d L e m m a 4 . 1 s h o w t h a t t h e size o f is l i m i t e d o n l y b y t h e m o d u l u s o f i n t e g r a b i l i t y o f t h e a d m i s s i b l e c o n t r o l s . I f t h e g r o w t h c o n d i t i o n o f ( H 5 ) is r e p l a c e d b y L(t, x , u ) > a | u | 2 p f o r s o m e p > 1 , t h e n a n y < p w i l l se rve i n ( H 2 ) . H y p o t h e s i s ( H 3 ) c a n b e r e l a x e d c o n s i d e r a b l y : see p a r a g r a p h 6 .5 . T h e m e a s u r a b i l i t y a n d c o n t i n u i t y h y p o t h e s e s o f ( H 4 ) a r e s t a n d a r d . - 80 -H y p o t h e s i s ( H 5 ) is s u b s t a n t i a l l y w e a k e r t h a n t h e a s s u m p t i o n t h a t is a c o m p a c t set u s e d b y K u s h n e r ( 1 9 7 5 ) a n d t h r o u g h o u t t h e l i t e r a t u r e . T h i s is t h e m a i n c o n t r i b u t i o n o f t h e c u r r e n t c h a p t e r . T h e r e q u i r e m e n t t h a t t h e c o n t r o l p r o c e s s u b e x f - a d a p t e d i n t h e d e f i n i t i o n o f a c o n t r o l - s t a t e p a i r c a n b e d e f e n d e d o n t h e p r a c t i c a l g r o u n d s t h a t a t a n y t i m e t, t h e c o n t r o l l e r s h o u l d use o n l y t h e i n f o r m a t i o n o b t a i n a b l e f r o m o b s e r v i n g t h e s t a t e p r o c e s s i t s e l f a t t i m e s b e f o r e t. T h i s i n f o r m a t i o n is d e f i n e d b y t h e a - f i e l d 7tx. F o r t h e o r e t i c a l c o m p l e t e n e s s , h o w e v e r , w e s h o u l d m e n t i o n t h e a p p a r e n t l y l a r g e r c lass o f c o n t r o l s u w h i c h a re a l l o w e d t o d e p e n d o n a n y t h i n g i n t h e k n o w n u n i v e r s e w h i c h t a k e s p l a c e b e f o r e t i m e t: t h a t i s , c o n t r o l s w h i c h a re a d a p t e d t o 7t r a t h e r t h a n t o 7*. W e w i l l n o w s h o w t h a t t h i s f a m i l y o f c o n t r o l s o f fe rs n o a d v a n t a g e (as m e a s u r e d b y A ) o v e r t h e m o r e p r a c t i c a l f a m i l y c h o s e n a b o v e . S u p p o s e a filtered space ( f t , 7, 7t,P) a n d a n ^ - B r o w n i a n m o t i o n u>t a re g i v e n s u c h t h a t Ut is a n ^ - a d a p t e d p r o c e s s s a t i s f y i n g ( 4 . 1 ) f o r s o m e ^ - a d a p t e d p r o c e s s xt- A s s u m e E fQ T \ur\2 dr < +oo. T h e n b y W o n g ' s T h e o r e m ( T h m . 3 . 6 ) , t h e r e is a n e x t e n s i o n o f ( f t , 7,7t,P) c a r r y i n g a B r o w n i a n m o t i o n wt s u c h t h a t ( 4 . 4 ) x t = x0 + f E [ / ( r , x , ur) \ 7r x] dr + f a ( r , x ) dwT. Jo Jo ( W e c o n t i n u e t o d e n o t e t h i s e x t e n s i o n b y ( f t , 7, 7t, P).) T h e o b j e c t i v e v a l u e c a n b e w r i t t e n as f o l l o w s , u s i n g c o n d i t i o n i n g a n d F u b i n i ' s t h e o r e m : ( 4 . 5 ) A [ u , x ] = E ^ ( x r , E x r ) + J E[L(t,x,ut) \ 7']dt . N o w t h e p r o c e s s (j(t, w), L(t, w)) = E [ ( / ( t , x , Ut), L(t, x , Ut)) | 7*"^ is ^ - a d a p t e d a n d o b e y s (4 .6 ) (j[t,u),L[t,u)) eQ{t,x(-,u>)) a . e . t e f O . T ] , a.s. E q u a t i o n ( 4 . 6 ) is j u s t i f i e d b y t h e f o l l o w i n g L e m m a . 4 . 2 L e m m a , (a.) Let ( f t , 7, P) be a given probability space carrying a set-valued mapping T: ft —+ R n a n d a n 7-measurable mapping g: ft — • R n with E |^| < + o o . Suppose that T has nonempty closed convex values, and that for some a-Beld Q C 7, the mapping ui —* s u p { p • 7 : 76 F ( w ) } J s - 8 1 -Q-measurable for each p G R n . Suppose further that the set {p G R n : s u p p • T(w) < +00} has nonempty interior almost surely. Then under these conditions, g(w) G r(o>) a.s. = • E[g\ 5 ] ( w ) e r ( w ) a .s . (b) For each fixed t G [0, T ] , T ( w ) = Q(t, x(-, w)) a n d Q = ?tx obey the hypotheses in (a). Proof, ( a ) F o r e a c h p e Q n , p • E[g \ §\ = E [ p • gr | Q] < E [ s u p ( p • T) \ Q] = s u p ( p • T) a.s. F o r fixed w , t h e s u b l i n e a r f u n c t i o n o f p o n t h e R H S is finite o n a c o n v e x se t w i t h n o n e m p t y i n t e r i o r : t h e i n e q u a l i t y t h e r e f o r e h o l d s f o r a l l p G R n a n d (a) f o l l o w s . ( b ) T h e l i n e a r g r o w t h o f / a n d t h e s u p e r q u a d r a t i c g r o w t h o f L i m p l y i n t { p G R n + 1 : s u p p • Q(t,x{-,w)) < +00} = R n X ( - o o , 0 ) a.s. / / / / L e t u s n o w c o n s i d e r t h e m e a s u r e space M = [0, T] x Cl e q u i p p e d w i t h t h e c r - f ie ld M g e n e r a t e d b y a l l se ts F G B X J s u c h t h a t f o r e a c h fixed t, t h e p r o j e c t i o n {u : ( t , w ) G F} l ies i n 7tx a n d f o r e a c h fixed w, t h e p r o j e c t i o n {t : (t, w) G F} l ies i n B. B e n e g (1971) s h o w s t h a t a m a p p i n g o f [0, T] X f l i n t o R n + 1 i s x t - a d a p t e d i f a n d o n l y i f i t is M-measurable. H e n c e i n p a r t i c u l a r \f,L) is ^ - m e a s u r a b l e . B e n e g (1971), L e m m a 5, p . 460, g ives a n i m p l i c i t f u n c t i o n l e m m a a l m o s t p e r f e c t l y s u i t e d t o t h i s s i t u a t i o n . U s i n g h i s n o t a t i o n , w e t a k e ( M , M) as d e f i n e d a b o v e , A = R n + 1 , a n d U = U x [0, +00). A t y p i c a l e l e m e n t o f U w i l l b e d e n o t e d b y ( u , v). L e t us d e f i n e k: M X U —* A a n d y: M —• U b y k{t,w,u,v) = (f(t,x{-,u}),u), L(t,x{-,u),u) + vj y(t,w) = (f{t,w), L{t,w)). T h e n k(t,w,u,v) is . M - m e a s u r a b l e i n (t,co) f o r e a c h fixed (u,v), a n d c o n t i n u o u s i n ( u , v) f o r each fixed (t,u). A l s o y i s M - m e a s u r a b l e a n d o b e y s y(t,w) G k(t,u),U) = Q{t,x(-,w)) b y ( 4 . 6 ) . N o w t h e set U is a d m i t t e d l y n o n - c o m p a c t , b u t i t is c l o s e d a n d a - c o m p a c t , a case i n w h i c h BeneS 's l e m m a is e a s i l y seen t o r e m a i n v a l i d . T h e c o n c l u s i o n o f t h i s l e m m a is t h a t t h e r e is a n M - m e a s u r a b l e m a p p i n g ( u , v): [0, T\ x Cl -r U s u c h t h a t ( / ( * , « ) , Z ( t , W ) ) = (j(t,x[;u),u{t,u)), L{t,x{;U),u(t,w))+v{t,w)y - 82 -N o w [u,v) i s ^ " - a d a p t e d s ince i t is . M - m e a s u r a b l e , a n d ( 4 . 4 ) ( 4 . 5 ) b e c o m e Vt dt > A [ u , x ] . T h i s d e m o n s t r a t e s t h a t a n y a d m i s s i b l e p a i r ( u , x ) f o r w h i c h u is m e r e l y ^ - a d a p t e d c a n b e r e p l a c e d b y a n a d m i s s i b l e p a i r (u, x ) f o r w h i c h u is ^ - a d a p t e d , w i t h o u t i n c r e a s i n g t h e c o s t . I n f a c t , w e c a n say m o r e . BeneS ( 1 9 7 1 ) , p p . 4 5 0 - 4 5 1 , s h o w s t h a t t h e . M - m e a s u r a b i l i t y o f u i m p l i e s t h a t t h e r e is a m e a s u r a b l e m a p v:[0,T] X Cn —» U s u c h t h a t v is C " - a d a p t e d a n d u ( t , w ) = v(t,x(-,w)). T h u s t h e use o f ^ - a d a p t e d c o n t r o l s is e q u i v a l e n t t o t h e use o f f e e d b a c k c o n t r o l s . I n s u m m a r y , t h e r e a re t w o w a y s t o d e f i n e p r o b l e m ( P ) . T h e first uses nonanticipative controls, i .e. [ / - v a l u e d s t o c h a s t i c p rocesses tt d e f i n e d o n a filtered space (Cl,7, 7t,P) w i t h ^ t - B r o w n i a n m o t i o n s u c h t h a t ut is m e r e l y ^ - a d a p t e d a n d so lves (4 .1 ) t o g e t h e r w i t h s o m e ^ - a d a p t e d p r o c e s s Xt. T h e s e c o n d seems m o r e r e s t r i c t i v e : i t uses feedback controls. T h e s e a re C " - a d a p t e d f u n c t i o n s v: [0 , T] x Cn —* U w i t h t h e p r o p e r t y t h a t f o r s o m e space ( H , 7,7t, P) c a r r y i n g a B r o w n i a n m o t i o n Wt, t h e r e is a n 5 t - a d a p t e d p r o c e s s x t o b e y i n g T h e a r g u m e n t s a b o v e s h o w t h a t t h e use o f f e e d b a c k c o n t r o l s is n o t r e s t r i c t i v e . I n d e e d , f o r a n y a d m i s s i b l e n o n a n t i c i p a t i v e c o n t r o l t h e r e is a n a d m i s s i b l e f e e d b a c k c o n t r o l g i v i n g r i s e t o a n o b j e c t i v e v a l u e a t l e a s t as s m a l l . O f c o u r s e f e e d b a c k c o n t r o l s a re n o t m o r e g e n e r a l e i t h e r , s ince a n a d m i s s i b l e f e e d b a c k p a i r (v,x) s o l v i n g (4 .7 ) g ives r i se t o a n o n a n t i c i p a t i v e s o l u t i o n p a i r ( u , x ) f o r (4 .1 ) b y s i m p l y d e f i n i n g u(t,uj) = v(t, x ( - , o ; ) ) . T h u s t h e n u m b e r i n f ( P ) is t h e s a m e i n t h e f e e d b a c k a n d n o n a n t i c i p a t i v e f o r m u l a t i o n s , a n d t h e f e e d b a c k p r o b l e m h a s a s o l u t i o n i f a n d o n l y i f t h e n o n a n t i c i p a t i v e p r o b l e m d o e s . (4 .7 ) S e c t i o n 5 . E x i s t e n c e T h e o r y T h i s w h o l e s e c t i o n is d e v o t e d t o t h e p r o o f o f t h e f o l l o w i n g f a c t . - 83 -5 . 1 Theorem. Assume (H1)-(H6). (a) If problem (P) has an admissible pair then it has a solution. (b) Indeed, let any sequence of admissible pairs { ( u f c , x f c ) } be given, such that the objective values Xk = A [ u f c , x f c ] converge to some real number X. Then there is an admissible pair (u,x) for (P) such that, along a subsequence, xk-^-*x in Cn and A [ u , x] < A. C l e a r l y c o n c l u s i o n (a ) f o l l o w s f r o m c o n c l u s i o n ( b ) . F o r ( H 5 ) e n s u r e s t h a t i n f ( P ) > 0 , w h i l e i n f ( P ) < +00 b e c a u s e a n a d m i s s i b l e p a i r e x i s t s . H e n c e o n e c a n c o n s t r u c t a s e q u e n c e o f a d m i s s i b l e p a i r s as d e s c r i b e d i n ( b ) f o r w h i c h t h e o b j e c t i v e v a l u e s t e n d t o A = i n f ( P ) . T h e n t h e a d m i s s i b l e p a i r ( u , x ) g i v e n b y ( b ) so lves t h e p r o b l e m . H o w e v e r , s t a t e m e n t ( b ) is s o m e w h a t m o r e g e n e r a l t h a n a s i m p l e e x i s t e n c e t h e o r e m . I t c a n b e v i e w e d as a l o w e r - s e m i c o n t i n u i t y c o n c l u s i o n a b o u t A w h i c h h o l d s g l o b a l l y a n d n o t j u s t n e a r o p t i m a l i t y . T h e s i g n i f i c a n c e o f t h i s w i l l b e c o m e c l e a r l a t e r . W e n o w t u r n o u r a t t e n t i o n t o t h e p r o o f o f ( b ) . L e t a n y s e q u e n c e { ( u f c , x f c ) } as d e s c r i b e d i n ( b ) b e g i v e n . E a c h e n t r y ( u f c , x f c ) i n t h i s s e q u e n c e c a r r i e s w i t h i t a p r o b a b i l i t y s p a c e (Clk, 7k,Pk) a n d a B r o w n i a n m o t i o n wk. W e w i l l t r y t o s i m p l i f y t h e n o t a t i o n i n t h e f o l l o w i n g a r g u m e n t s b y s u p p r e s s i n g t h e s u p e r s c r i p t k o n Pk w h e n e v a l u a t i n g t h e p r o b a b i l i t y o f a n e v e n t w h i c h is c l e a r l y t a k e n f r o m 7k, a n d a g r e e i n g t h a t f o r a r e a l - v a l u e d f u n c t i o n g d e f i n e d o n ( f 2 f c , 7k), w e w i l l w r i t e E<7 f o r e x p e c t a t i o n w i t h r e s p e c t t o Pk. N o w t h e c o n v e r g e n t sequence A [ u f c , x f c ] is c e r t a i n l y b o u n d e d . S i n c e A [ u f c , x f c ] > a E \uk\2 dr b y ( H 5 ) , t h e r e is s o m e c o n s t a n t fc3 > 0 s u c h t h a t E f | u * | 2 dr < k3 Vfc. Jo T h e first s t e p i n p r o v i n g ( b ) is t o s t u d y t h e c o n v e r g e n c e p r o p e r t i e s o f t h e s t a t e p rocesses xk. T h i s c o m m e n c e s w i t h P r o p . 5 .2 , u s i n g t h e n o t a t i o n *1,*(*)= ff(r,xk,uk)dr, Jo x2'k(t)= fc(r,xk)dwT, Jo zk(t)= fL(r,xk,uk)dr. Jo - 84 -5 . 2 P r o p o s i t i o n . The sequence of quadruples (xk,x1'k,x2'k,zk), considered as a collection of random vectors in C3n x Z, i s tight. Proof. T o p r o v e t i g h t n e s s , w e m a y c o n s i d e r each c o m p o n e n t s e q u e n c e i n d i v i d u a l l y . W e b e g i n w i t h t h e s e q u e n c e { z 1 , f e } . F o r e a c h w e fi a n d fc € N , o n e h a s s u p l * 1 - * ^ ) - z 1 , f c ( s ) | < s u p f*\f{r)Xk,v.k)\dr 0 s u c h t h a t Pi s u p \x1>k(t)-x1>k{s)\>e\ e f i > w e first fix fc a n d w r i t e o~k(t) f o r t h e m a t r i x cr(t, xk). T h e t - c o m p o n e n t o f t h e n - v e c t o r x2,k is 3=iJ° A c c o r d i n g t o I k e d a - W a t a n a b e ( 1 9 8 1 ) , T h m . I I - 7 . 2 ' , p . 9 1 , t h e r e a re B r o w n i a n m o t i o n s wk, j = 1 , . . . , d ( b a s e d p e r h a p s o n a n e x t e n s i o n o f ( f l f c , 7k, Pk)) s u c h t h a t d e>x* 0 w e h a v e P\ s u p \e'i(x2>k(t)-x'2'k(S))\>e\< P 0 < t - » < « . = 1 V o / v o / > e > . F o r e a c h w , o~k is a b o u n d e d m e a s u r a b l e f u n c t i o n o n [ 0 , T ] so | | f f f e | | is d e f i n e d i n L°°[0, T\. W e p a r t i t i o n t h e R H S i n t o t h e se ts { | | ( 7 * | | 3 > 2 * } a n d { | | a f c f < i ? } , w h e r e u p o n i t is e v i d e n t t h a t t h e L H S is n o t l a r g e r t h a n p {Kir > A+pI «p Ei*?(f)-^(oi >« l " " J I 0 < t ' - « ' < . R « JTj < E | k i 2 / J 2 + X>{ s u p | ^ ( t ' ) - ^ ( s ' ) | > £ / d } . N o w f o r a l l k a t o n c e , L e m m a 4 . 1 g i ves a c o n s t a n t u p p e r b o u n d f o r E | | a * f < 2 * 1 ( l + E | | s f ) . H e n c e f o r a n y g i v e n e > 0 a n d rj > 0, w e first c h o o s e R so t h a t E | | c f c | | 2 /R < »7/2 f o r a l l k. T h e n w e n o t e t h a t t h e s e c o n d t e r m o f t h e R H S a b o v e t e n d s t o 0 as S —* 0 b e c a u s e a l m o s t - s u r e c o n v e r g e n c e i m p l i e s c o n v e r g e n c e i n p r o b a b i l i t y ; h e n c e t h e s e c o n d t e r m is a l so s m a l l e r t h a n r? /2 f o r a l l 6 > 0 s u f f i c i e n t l y s m a l l . A l s o , t h e n u m e r i c a l v a l u e o f t h e s e c o n d t e r m is i n d e p e n d e n t o f k a n d i. I t is t h e r e f o r e p o s s i b l e t o c o m b i n e t h e c o m p o n e n t w i s e e s t i m a t e s f o r t = 1 , 2 , . . . , n i n t o t h e f o l l o w i n g : l i m s u p P < s u p \x2'k(t) - x2'k(s)\> e\ = 0. 5->0+ k {o ivj < ^ E ^ L(r,xk,uk)dry S i n c e t h e s u p r e m u m o v e r A: o f t h e R H S is b o u n d e d , i t f o l l o w s t h a t l i m s u p P \zk £ KN } = 0. JV —oo T h a t is , t h e s e q u e n c e {zk} is t i g h t . / / / / - 86 -L e t u s n o w a p p l y P r o k h o r o v ' s T h e o r e m ( P r o p . 2 . 3 ) . I t s t a t e s t h a t t h e r e is a s u b s e q u e n c e ( w h i c h w e d o n o t r e l a b e l ) a l o n g w h i c h {xk,x1'k,x2'k,zk) c o n v e r g e s i n d i s t r i b u t i o n t o a r a n d o m e l e m e n t [x,x1,x2,z):(Q,7,P)^C3nxZ. W e w i s h t o use t h i s l i m i t i n g q u a d r u p l e t o c o n s t r u c t t h e a d m i s s i b l e p a i r w h o s e e x i s t e n c e is a s s e r t e d b y T h m . 5 . 1 ( b ) . L e t us s t a r t b y c o n s i d e r i n g t h e f o l l o w i n g s u b s e t s o f C 3 n X Z: SN = {(x,!1,!2^) : ( x , x \ x 2 ) e C3n, zeZ, x ( t ) = x0 + x*{t) + x 2 ( t ) , x » = x 2 ( 0 ) = 0, x leACn, / I x 1 ^ ) ! 2 dr< N2, Jo z(T) < N, E a c h set is c l o s e d i n C 3 n X Z. T o j u s t i f y t h i s s t a t e m e n t , l e t ( x f c , x 1 , f c , x 2 ' f c , z f c ) 6 Sfj b e a s e q u e n c e c o n v e r g i n g t o s o m e p o i n t [x,x l>x2,z). I t f o l l o w s i m m e d i a t e l y t h a t x(i) — XQ + x1^) + x 2 ( t ) V t , t h a t x ^ O ) = x 2 ( 0 ) = 0, a n d t h a t z(T) < N. T h e c o n d i t i o n | | i 1 , f c | | 2 < N Vk i m p l i e s t h a t x 1 € ACn a n d || i11| 2 < N. ( T h i s is a s t a n d a r d f a c t i n t h e c a l c u l u s o f v a r i a t i o n s , c e n t r a l t o t h e p r o o f o f T o n e l l i ' s c l a s s i c a l e x i s t e n c e t h e o r e m . ) I t r e m a i n s o n l y t o c h e c k t h a t ( x 1 ( t ) , i ( t ) ) e Q(t, x ( ) ) a.e. T h i s is a c o r o l l a r y o f t h e f o l l o w i n g w e l l - k n o w n c l o s u r e t h e o r e m u s e d i n d e t e r m i n i s t i c c o n t r o l t h e o r y . 5 . 3 Proposition. Assume that the sets Q(t,x()) satisfy hypothesis (H6). Let (xk,zk) e ACn X Z be a sequence of functions obeying ( x f c ( t ) , z f c ( t ) ) e Q ( t , x f c ( - ) ) a.e. o n [0,T], Vfc. If xk —* x uniformly for some x € AC and zk —* z pointwise a.e. for some z e Z, then one has (x{t),z{t)) e Q ( t , x ( ) ) a.e. o n [0,T\. Proof. T h e s t a t e m e n t is v e r y s i m i l a r t o t h a t o f c l o s u r e t h e o r e m 1 5 . 2 . i g i v e n b y C e s a r i ( 1 9 8 3 ) , p . 4 4 4 . T w o d i f f e r e n c e s a r e w o r t h m e n t i o n i n g , h o w e v e r . F i r s t , C e s a r i ' s t h e o r e m i n v o l v e s a m u l t i f u n c t i o n Q d e p e n d i n g o n ( t , x t ) 6 [0,T] x R n i n s t e a d o f o n ( t , x ( - ) ) G [0,T] X Cn. H i s p r o o f r e q u i r e s o n l y m i n o r c h a n g e s t o t r e a t t h e m o r e g e n e r a l case. S e c o n d , C e s a r i uses a s e q u e n c e o f a b s o l u t e l y c o n t i n u o u s f u n c t i o n s f o r z i n s t e a d o f u s i n g e l e m e n t s o f Z. H i s e x a c t c o n c l u s i o n is t h a t i f t h e p o i n t w i s e l i m i t z(t) h a s a r e p r e s e n t a t i o n as t h e s u m o f a n a b s o l u t e l y c o n t i n u o u s f u n c t i o n c ( t ) a n d a s i n g u l a r f u n c t i o n s(t), t h e n ( x , c) € ACn+1 so lves t h e i n d i c a t e d d i f f e r e n t i a l i n c l u s i o n . N o w i n o u r c o n t e x t , t h e l i m i t z S Z c e r t a i n l y h a s s u c h a d e c o m p o s i t i o n , so o u r c o n c l u s i o n s t a t e d a b o v e f o l l o w s f r o m C e s a r i ' s . H o w e v e r , t h e r e q u i r e m e n t t h a t e a c h zk b e a b s o l u t e l y c o n t i n u o u s is i m p o r t a n t t o C e s a r i ' s p r o o f . W e m u s t t h e r e f o r e e x p l a i n w h y h i s a r g u m e n t r e m a i n s v a l i d d e s p i t e t h e p o s s i b i l i t y o f a s t r i c t l y p o s i t i v e r i n t h e s t a t e m e n t b zk(t) dr = zk(b) - zk(a) + r f o r s o m e r > 0. t H e r e t h e s p e c i a l s h a p e o f t h e sets Q i n t e r v e n e s . T h e i d e n t i t y Q{t, x) = Q{t, x) + { 0 } x [ 0 , +00) V ( t , x ) 6 [0, T\ x Cn is p r e c i s e l y t h e o b s e r v a t i o n n e e d e d t o e x t e n d C e s a r i ' s p r o o f t o i n c l u d e t h e c u r r e n t p r o p o s i t i o n . / / / / N o w f o r e a c h f i x e d k, w e i n v e s t i g a t e P { ( x f c , x1,k, x 2 ' f c , zk) 6 S j \ r } . T h e f o l i o w i n g c o n d i t i o n s h o l d w i t h p r o b a b i l i t y o n e : xk{t) = x0 + x1'k{t) + xa'k{t) V t , x 1 ' f c ( 0 ) = x 2 ' f c ( 0 ) = 0, xi,k e A C n ^ ( i 1 . * ^ ) , * * ^ ) ) G <9 ( t ,x f c ( ) ) a.e. O n l y t h e u p p e r b o u n d s o n | | x 1 , f c | | 2 a n d zk(T) r e m a i n t o c o n s i d e r . B y ( H 2 ) , w e h a v e a n d t h e e x p e c t a t i o n o n t h e r i g h t - h a n d s ide is b o u n d e d u n i f o r m l y i n A; b y L e m m a 4 . 1 a n d t h e o b s e r v a t i o n f o l l o w i n g T h e o r e m 5 . 1 . A l s o , P{zk(T) > N} < ±E ^ L(r, xk, «*) dr. H e r e t h e R H S is u n i f o r m l y b o u n d e d i n k b e c a u s e w e s t a r t e d w i t h a c o n v e r g e n t s e q u e n c e o f o b j e c t i v e v a l u e s . W e t h e r e f o r e h a v e l i m i n f P { ( x f c , x 1 ' f c , x 2 , f c , z f c ) G S N } = 1 . N->oo fc l V ' ' B y D e f . 2 . 1 ( b ) (see t e x t f o l l o w i n g T h m . 2 . 3 ) , i t f o l l o w s t h a t t h e l i m i t i n g 4 - t u p l e (x,xx,x2, z) l ies i n 00 t h e set SH a . s . T h i s p r o v e s t h e f o l l o w i n g a s s e r t i o n . N=i - 88 -5 . 4 P r o p o s i t i o n . The limiting quadruple [x,!1 ,x2 ,z) of Prop. 5.2 obeys the following conditions with probability one: x{t) = x0 + x1 {t) + x2 ( i ) V t , x1 ( 0 ) = x2 ( 0 ) = 0 , x1 £ ACn, fT I i 1 ( r ) | 2 dr < o o , z(T) < c o , Jo ( i 1 ( * ) , * ( * ) ) eQ{t,x{-)) a.e. L e t u s n o w use W o n g ' s t h e o r e m t o s h o w t h a t x2 c a n b e r e p r e s e n t e d as a s t o c h a s t i c i n t e g r a l o f t h e r e q u i r e d f o r m . W e first d e f i n e a filtration Jt o n (Q, 7, P) as t h e filtration g e n e r a t e d b y (xt) x2). 5 . 5 P r o p o s i t i o n . There exists an extension of ( 0 , 7, 7t, P) o n which the conclusions of Prop. 5.4 remain valid, and which supports a d-dimensional Bownian motion wt such that xt = x0+ f E [ i J | f r x ] dr + [ cr(r, x) dwr V t e [0 , T], a.s. Jo Jo Proof. F r o m P r o p . 5 .4 w e h a v e xt = xo + f x^dr+ x2 V t e [0 , T], a.s. Jo C o n d i t i o n ( 3 . 9 ) o f W o n g ' s t h e o r e m is e v i d e n t ; i t r e m a i n s t o v e r i f y t h e m a r t i n g a l e p r o p e r t i e s o f x2. W e first s h o w t h a t j | | x 2 ' f c | | 2 j is a u n i f o r m l y i n t e g r a b l e s e q u e n c e . I n d e e d , ( B D G ) g i ves E||z2'fc||2//' < Cp fT E\a(r,xk)\2/0 dr Jo < C p / c : ^ ( l + E | | xf c | | 2 ) dr. T h e R H S is b o u n d e d u n i f o r m l y i n k; u n i f o r m i n t e g r a b i l i t y f o l l o w s b e c a u s e 2 / / 9 > 2. N o w f o r e a c h fc, x2'k is a m a r t i n g a l e w i t h r e s p e c t t o t h e filtration g e n e r a t e d b y (xk,x2'k). So b y P r o p . 3 . 1 , i t f o l l o w s t h a t x2 is a n ^ - m a r t i n g a l e . T h e q u a d r a t i c v a r i a t i o n o f x2,k is qk = / * er(r , xk)cr(r, xk)' dr. B y d e f i n i t i o n , t h i s m e a n s t h a t ( x 2 ' f c ) ( x 2 ' f c ) ' — qk is a m a r t i n g a l e f o r e a c h fc. T h e c a l c u l a t i o n a b o v e s h o w s t h a t t h i s m a r t i n g a l e h a s a u n i f o r m l y i n t e g r a b l e s e q u e n c e o f s u p r e m u m n o r m s . M o r e o v e r , qk is a c o n t i n u o u s i m a g e o f t h e c o n v e r g e n t ( i n d i s t r i b u t i o n ) s e q u e n c e xk, so o n t h e space o f c o n t i n u o u s m a t r i x - v a l u e d f u n c t i o n s w e h a v e (x2'k) (x2>k)' - q k ^ (x2) (x2)' - q, - 8 9 -w h e r e qt = /Q c(r, x)a{r, x)' dr. A c c o r d i n g t o P r o p . 3 . 1 , i t f o l l o w s t h a t ( x 2 ) ( x 2 ) ' — qt is a n ^ - m a r t i n g a l e . B y d e f i n i t i o n , t h e q u a d r a t i c v a r i a t i o n o f x 2 i s qt, i .e. < x 2 ) t = f a(r,x)o(r,x)' dr. Jo T h e p r o c e s s a(r, x ) is c l e a r l y ^ " - p r e d i c t a b l e , so W o n g ' s t h e o r e m ( T h m . 3 .6 ) a p p l i e s . I t c o n c l u d e s t h a t o n a c e r t a i n e x t e n s i o n o f (fl, 7, 3t>P) t h e r e is a d - d i m e n s i o n a l B r o w n i a n m o t i o n wt s u c h t h a t xt=x0-r f E [ x r J | 7*] dr+ f < r ( r , x ) dwr V t £ [0, T ] , a .s . Jo Jo T h e e x p l i c i t c o n s t r u c t i o n o f t h i s e x t e n s i o n i n S e c t i o n 3 m a k e s i t c l e a r t h a t t h e p r o p e r t i e s o f t h e l i m i t i n g q u a d r u p l e d e s c r i b e d i n P r o p . 5.4 r e m a i n i n t a c t . / / / / L e t us n o w e x a m i n e t h e f o l l o w i n g c o n c l u s i o n o f P r o p . 5 .4 : (i l{t,u),i[t,w)yj eQ(t,x(;w)). L e m m a 4 .2 i m p l i e s t h a t i f w e d e f i n e ( / ( t , w ) , i ( t , w ) j = E ^ ( x J , i t ) | , t h e n (7{t,w),L[t,uj)eQ{t,x[-,w)) Vt€[0,r], a .s . I n t h i s n o t a t i o n , t h e a r g u m e n t g i v e n i n t h e t e x t f o l l o w i n g L e m m a 4 .2 e s t a b l i s h e s t h e f o l l o w i n g r e s u l t . 5 . 6 P r o p o s i t i o n . T h e r e i s a n ^ - a d a p t e d process {u,v):[0,T\ X fl —» U x [0,+oo) f o r which f(t,w) = f(t,x,ut) a.e. [0,T], a.s. , L(t,w) = L(t,x, Ut) + vt a.e. [0,r], a.s. T h e f o l l o w i n g P r o p o s i t i o n c o m p l e t e s t h e p r o o f o f T h m . 5 . 1 . 5 . 7 P r o p o s i t i o n . T h e p a i r (tt ,x ) is admissible, and A [ u , x ] < A. Proof. P r o p o s i t i o n s 5 . 4 - 5 . 6 s h o w t h a t u : [0, T] x fl —• U is a n ^ - a d a p t e d p r o c e s s f o r w h i c h dxt = f{t, x, Ut) dt + /0T z(t) dt a.s. , so l i m i n f Ezk(T) > E f L[t)dt = E f L{r,x,ur)dr+ [ vrdr , k—"X> JQ JQ JQ = > E / L(r,x,uT)dr < l i m i n f Ezk{T)-E / vrdr. (*) Jo fc_>0° Jo S i n c e xk(T)-^-*x(T) a n d E | x f c ( T ) | 2 is b o u n d e d u n i f o r m l y i n A; b y L e m m a 4 . 1 , u n i f o r m i n t e g r a b i l i t y i m p l i e s E x f c ( ! T ) — • E x ( T ) . T h e t e c h n i c a l L e m m a 5.8 b e l o w s h o w s t h a t t h i s i m p l i e s £ ( x f c ( r ) , E x f c ( r ) ) - ^ £ ( x ( T ) , E x ( r ) ) . A s e c o n d a p p l i c a t i o n o f F a t o u ' s l e m m a n o w g ives Et{x(T),Ex(T)) < l i m i n f E £ ( x f c ( T ) , E x f c ( T ) ) . ( * * ) fc—^oo C o m b i n i n g ( * ) a n d ( * * ) , w e o b t a i n t h e i n e q u a l i t y p r o c l a i m e d b y T h m . 5 . 1 ( b ) : A [ u , x ] < l i m i n f A [ u \ x f c ] - E / v(r) dr < A. //// fc-*00 JQ 5 . 8 L e m m a . S u p p o s e a sequence xk o f R ^ - v a J u e d random variables converges in distribution to a random variable x, and that moreover E x f c — • E x . Then for any continuous function £: R n x R n — • R , one has £ ( x f c , E x f c ) - ^ - » £ ( x , E x ) . Proof. L e t a n y e > 0 b e g i v e n . S i n c e { x f c } c o n v e r g e s i n d i s t r i b u t i o n , t h e r e is a c o m p a c t set Ko s u c h t h a t P{xk € Ko} > 1 — e Vk. C h o o s e a c o m p a c t set K\ c o n t a i n i n g t h e s e q u e n c e { E x f c } . T h e n o n K = Ko X Ki, t h e f u n c t i o n £ is u n i f o r m l y c o n t i n u o u s , so t h e r e e x i s t s 6 > 0 so s m a l l t h a t | ( x , y ) - ( x ' , y ' ) | < 6 f o r ( x , y ) , ( x ' , y ' ) S K i m p l i e s | £ ( x , y ) - £ ( x ' , y ' ) | < e. T h e n c h o o s e N € N s u c h t h a t k > N f o r c e s | E x f c - ~Ex\< 6. I t f o l l o w s t h a t P { | £ ( x f c , E x f c ) - £ ( x f c , E x ) | >e}< P{xk (£ K0} < e. H e n c e | £ ( x f c , E x f c ) - £ ( x f c , E x ) | - ^ 0 . N o w s ince £ ( - , E x ) is c o n t i n u o u s , £ ( x f c , E x ) - ^ £ ( x , E x ) . B y B i l l i n g s l e y ( 1 9 6 8 ) , T h m . 4 . 1 , p . 2 5 , i t f o l l o w s t h a t £ ( x f c , E x f c ) - ^ £ ( x , E x ) . / / / / - 9 1 -S e c t i o n 6 . E x t e n s i o n s o f T h e o r e m 5 . 1 T h e s t a n d i n g a s s u m p t i o n t h a t t h e r u n n i n g c o s t L a n d t h e t e r m i n a l cos t £ a re n o n n e g a t i v e c a n c l e a r l y b e r e l a x e d . W e n e e d o n l y a s s u m e t h a t L a n d £ a re b o u n d e d b e l o w , a n d t h a t L sa t i s f i es a g r o w t h c o n d i t i o n o f t h e f o r m L(t, x, u ) > a | u | 2 — 7 f o r s o m e a > 0, 7 > 0. T h i s s e c t i o n is d e v o t e d t o a c o l l e c t i o n o f s i m i l a r o b s e r v a t i o n s — m o s t n o t q u i t e so o b v i o u s — r e g a r d i n g t h e h y p o t h e s e s u n d e r w h i c h e x i s t e n c e is a s s u r e d . 6 . 1 T h e D e t e r m i n i s t i c C a s e . T h e case c = 0, i n w h i c h t h e s t o c h a s t i c d y n a m i c s o f S e c t i o n 4 r e d u c e t o d e t e r m i n i s t i c f u n c t i o n a l d i f f e r e n t i a l e q u a t i o n s , is p e r m i t t e d b y t h e h y p o t h e s e s g o v e r n i n g p r o b l e m (P). W h e n ( H I ) is f u r t h e r s p e c i a l i z e d so t h a t / d e p e n d s o n l y u p o n t h e c u r r e n t p o s i t i o n o f t h e s t a t e p r o c e s s x a n d n o t o n i t s p a s t , T h m . 5 . 1 r e d u c e s t o a s t a n d a r d e x i s t e n c e t h e o r e m i n d e t e r m i n i s t i c o p t i m a l c o n t r o l . See, f o r e x a m p l e , F l e m i n g a n d R i s h e l ( 1 9 7 5 ) , T h m . I I I . 4 . 1 , p . 6 8 . H o w e v e r , s e v e r a l issues r a i s e d b y t h i s c o m p a r i s o n d e s e r v e c o m m e n t . F i r s t , o u r r e s u l t r e q u i r e s n o L i p s c h i t z c o n d i t i o n o n / , w h e r e a s F l e m i n g a n d R i s h e l m a k e t h i s a s s u m p t i o n i n t h e i r c o n d i t i o n ( 2 . 4 ) , p . 6 2 . T h e L i p s c h i t z c o n d i t i o n is u s e d t o e n s u r e t h e e x i s t e n c e a n d u n i q u e n e s s o f s o l u t i o n s t o t h e g o v e r n i n g e q u a t i o n f o r e v e r y p o s s i b l e c h o i c e o f « : o u r t h e o r y a v o i d s t h i s b y i g n o r i n g u n i q u e n e s s c o m p l e t e l y a n d m a k i n g t h e e x i s t e n c e o f a c o r r e s p o n d i n g s o l u t i o n o n e o f t h e p r e r e q u i s i t e s f o r t h e a d m i s s i b i l i t y o f u . W h e n e x i s t e n c e a n d u n i q u e n e s s a re r e q u i r e d i n t h e s t o c h a s t i c c o n t e x t f o r o t h e r r e a s o n s (as t h e y w i l l b e i n C h a p . V ) , t h e I t o c o n d i t i o n s a r e u s e d . T h e s e a re m o r e s t r i n g e n t t h a n F l e m i n g a n d R i s h e l ' s h y p o t h e s i s (2 .4 ) o n p. 6 2 . S e c o n d , t h e d e t e r m i n i s t i c t h e o r y c a n b e d e v e l o p e d u n d e r t h e g r o w t h c o n d i t i o n L(t, x,u) > a | u | p — 7 f o r s o m e p > 1 , w h e r e a s w e r e q u i r e p > 2 . H e r e t h e s t o c h a s t i c t h e o r y c a n n o t b e s t r e n g t h e n e d . T h e e x p o n e n t 2 is e s s e n t i a l b e c a u s e i t is t h e r e c i p r o c a l o f B r o w n i a n m o t i o n ' s i n d e x o f H o l d e r c o n t i n u i t y . T h i s a c c o u n t s f o r t h e s p e c i a l r o l e o f 2 i n P r o p o s i t i o n s 1.1 a n d 1.2, w h i c h are c r u c i a l t o o u r p r o o f . - 9 2 -F i n a l l y , n o t e t h a t s e t t i n g a = 0 r e a l l y d o e s r e n d e r t h e p r o b l e m c o m p l e t e l y d e t e r m i n i s t i c , s i nce w e h a v e s h o w n t h a t t o a n y a d m i s s i b l e c o n t r o l u ( t , w) t h e r e c o r r e s p o n d s a f e e d b a c k c o n t r o l v(t,x) f o r w h i c h t h e d y n a m i c s b e c o m e x ( i ) = f(t,x, v{t,x)), x(0) = xo. N o e x t e r n a l r a n d o m i z a t i o n is p r e s e n t h e r e . H o w e v e r , T h m . 5 . 1 m u s t n o t b e i n t e r p r e t e d as i m p l y i n g t h e e x i s t e n c e o f a n o p t i m a l f e e d b a c k c o n t r o l l a w f o r d e t e r m i n i s t i c p r o b l e m s . T h i s is b e c a u s e t h e p h r a s e " f e e d b a c k c o n t r o l " u n f o r t u n a t e l y h a s t w o d i f f e r e n t m e a n i n g s . I n t h e s t o c h a s t i c t h e o r y , i t r e f e r s o n l y t o t h e f u n c t i o n a l d e p e n d e n c e o f t h e c o n t r o l l a w o n t h e s t a t e p r o c e s s — s o m e t h i n g w h i c h is a l w a y s t r i v i a l l y p r e s e n t i n t h e d e t e r m i n i s t i c case. I n t h e d e t e r m i n i s t i c t h e o r y , " o p t i m a l f e e d b a c k c o n t r o l " d e s i g n a t e s a s i n g l e f u n c t i o n v(t, x) w h i c h so lves a l l t h e v e r s i o n s o f p r o b l e m (P) r e g a r d l e s s o f t h e i n i t i a l p o i n t i n ( t , x ) - s p a c e . W e h a v e fixed t h e i n i t i a l p o i n t ( 0 , x o ) t h r o u g h o u t t h e a r g u m e n t s a b o v e , so t h e o p t i m a l c o n t r o l g e n e r a t e d b y T h m . 5 . 1 is a f e e d b a c k c o n t r o l o n l y i n t h e sense o f f u n c t i o n a l d e p e n d e n c e , a n d n o t n e c e s s a r i l y i n t h e sense u s e d b y d y n a m i c p r o g r a m m e r s . 6 . 2 S t o c h a s t i c C a l c u l u s o f V a r i a t i o n s . A v e r y i m p o r t a n t s p e c i a l case o f p r o b l e m (P) a r i ses w h e n / ( £ , x , u ) = u , U = R n . T h e n t h e d y n a m i c s a re s i m p l y dxt = u t dt + o~(t, x) dwt, x ( 0 ) = x o , u e R n , a n d p r o b l e m ( P ) b e c o m e s a stochastic calculus of variations p r o b l e m . See F l e m i n g ( 1 9 8 3 ) . I f w e a s s u m e t h a t L is c o n t i n u o u s i n t, t h e n h y p o t h e s i s ( H 6 ) h o l d s w h e n e v e r L(t, x , •) is a c o n v e x f u n c t i o n f o r e a c h p a i r (t, x ) . 6 . 3 I n c o r p o r a t i n g C o n s t r a i n t s . S u p p o s e t h e c r i t e r i a f o r a d m i s s i b i l i t y i n p r o b l e m (P) a r e t i g h t e n e d b y a d d i n g a s y s t e m o f s o f t c o n s t r a i n t s s u c h as rT E E £ , ( x r , E x r ) + / L , ( r , x , u r ) d r Jo £ y ( x T , E x r ) + / L,(r,x,ur)dr Jo a | u | 2 . T h e g r o w t h o f a n y s i n g l e c o m p o n e n t o f L su f f i ces t o i m p l y t h a t E \v.k\2 dr i s b o u n d e d u n i f o r m l y i n k f o r a n y s e q u e n c e o f a d m i s s i b l e a r c s w i t h c o n v e r g i n g v e c t o r o b j e c t i v e v a l u e s , a n d t h i s l e a d s t o t h e b o u n d e d n e s s o f E | | x f c | | 2 b y L e m m a 4 . 1 . T h e r e m a i n d e r o f t h e p r o o f c a r r i e s t h r o u g h as b e f o r e , e x c e p t t h a t w e n o w c o n s i d e r t h e (I + l ) - f o l d p r o d u c t Z I+1 as t h e m e t r i c space i n w h i c h z(t) = J* L(r, x, ur) dr t a k e s i t s v a l u e s . H y p o t h e s i s ( H 6 ) a lso i n v o l v e s t h e v e c t o r f u n c t i o n L i n a n a t u r a l w a y . ( W e w i l l d i s c u s s t h i s f u r t h e r b e l o w . ) T o t r e a t t h e e q u a l i t y c o n s t r a i n t s , w e t h i n k o f Lj, j = 1, 2 J , as a d d i t i o n a l c o m p o n e n t s o f t h e f u n c t i o n / a n d c o n s e q u e n t l y r e q u i r e t h a t Lj o b e y t h e l i n e a r g r o w t h e s t i m a t e s a n d L i p s c h i t z c o n d i t i o n s o f ( H l ) - ( H 2 ) . S i n c e t h e p r o o f o f T h m . 5 .1 s h o w s E x 1 ' f c ( T ) — • E x x ( T ) a l o n g a s e q u e n c e w i t h c o n v e r g e n t o b j e c t i v e v a l u e s , i t f o l l o w s t h a t t h e s e h y p o t h e s e s w i l l p r e s e r v e t h e v a l u e o f e a c h E L j ( r , x k , u k ) dr i n t h e l i m i t as Jfc-» o o . T o g u a r a n t e e t h a t t h e v a l u e s E£y(x f c(r), E x f c ( T ) ) a re a lso p r e s e r v e d i n t h e l i m i t , i t su f f i ces t o m a k e s u r e e a c h o f t h e s e s e q u e n c e s is u n i f o r m l y i n t e g r a b l e . W e t h e r e f o r e a l l o w £y t o b e a n y c o n t i n u o u s f u n c t i o n o b e y i n g (6 .2 ) | / y ( a : , « ) | < Jby(l + |x j« + | c | r ) V(x, e) e R 2 n f o r s o m e c o n s t a n t s k}- > 0 , q € ( 0 , 2 ) , a n d r > 0. T h e n t h e a r g u m e n t r e p l a c i n g t h e d e r i v a t i o n o f ( * * ) i n t h e p r o o f o f P r o p . 5.7 w o u l d r u n as f o l l o w s . ( W e s u p p r e s s t h e s u b s c r i p t j.) O b s e r v e t h a t £ ( X J . , E X £ ) - ^ - + £ ( X T , E X T ) . M o r e o v e r , E \i(xkT,Ex* ) \ 2 , q < £ ( l + E | x £ | 2 + | E x £ | 2 r / * ) . T h e R H S is u n i f o r m l y b o u n d e d i n A: b y L e m m a 4 . 1 a n d t h e c o n v e r g e n c e o f E x £ , so u n i f o r m i n t e g r a b i l i t y i m p l i e s equality i n E £ ( x r , E x r ) = Urn E £ ( x £ , E x £ ) . - 9 4 -H a v i n g b r i e f l y d e s c r i b e d t h e h y p o t h e s e s o n £,• a n d c o r r e s p o n d i n g t o ( H l ) - ( H 5 ) , l e t us e x p l i c i t l y m e n t i o n t h a t f o r t h e c o n s t r a i n e d p r o b l e m t h e m u l t i f u n c t i o n Q{t, x ( ) ) : [0,T] X Cn <-»• R n + J X R 1 + i b e c o m e s Q(t,x{)) = {(F(t,x,v,),L(tix,u) + r) : F= {f,LuL2l. ..,Lj), L = ( L o , L-i, L-2,..., L-i), ueU, re R 1 + / o b e y s r< > 0 V i } . T h e w o r d i n g o f h y p o t h e s i s ( H 6 ) r e m a i n s t h e s a m e . 6 . 4 M o r e G e n e r a l P o i n t C o s t s . T h e p o i n t cos t f u n c t i o n a l £ i n p r o b l e m (P) e n t e r e d t h e p r o o f o f T h m . 5 . 1 o n l y i n t h e l a s t p a r t o f P r o p . 5 .7 . ( A l s o £ > 0 w a s i m p l i c i t l y u s e d e a r l i e r . ) A l o o k a t t h a t a r g u m e n t s h o w s t h a t £ c o u l d e a s i l y b e a l l o w e d t o d e p e n d o n a n y n u m b e r o f p o i n t s i n a d d i t i o n t o T. I n p a r t i c u l a r , \i § < Ti < T2 < • • < TN = T is a n y finite p a r t i t i o n o f [0 , T], T h m . 5 . 1 r e m a i n s v a l i d w h e n £: R 2 W n —» [0, oo ) is a c o n t i n u o u s f u n c t i o n o f t h e f o r m £ = £ ( x ( T 1 ) , E x ( T 1 ) , x ( r 2 ) , E x ( T 2 ) , . . . , x ( r / , ) , E x ( r / , ) ) . C o n s i d e r a t i o n o f s u c h g e n e r a l p o i n t - d e p e n d e n c e is p r o m p t e d b y K u s h n e r ( 1 9 7 2 ) . T h e r e m a r k s o f p a r a g r a p h 6.3 s h o w h o w s u c h g e n e r a l p o i n t c o s t s c a n a lso b e u s e d i n a n y c o n s t r a i n t s a d d e d t o t h e p r o b l e m . 6 . 5 R a n d o m I n i t i a l V a l u e . H y p o t h e s i s ( H 3 ) r e g a r d i n g t h e i n i t i a l p o i n t XQ c a n b e r e l a x e d c o n s i d e r a b l y . F o r i n s t a n c e , T h m . 5 .1 r e m a i n s v a l i d u n d e r t h e a s s u m p t i o n t h a t X o i s a r a n d o m v a r i a b l e w i t h a g i v e n d i s t r i b u t i o n o n R n . U n d e r t h e a d d i t i o n a l h y p o t h e s i s t h a t E | x o | 2 < + o o , L e m m a 4 . 1 r e m a i n s v a l i d f o r p = 2 , a n d t h e t i g h t n e s s o f a n y m i n i m i z i n g s e q u e n c e is e s t a b l i s h e d j u s t as b e f o r e . T h e p r o o f o f T h m . 5 . 1 t h e n p r o c e e d s . T h e o n l y c h a n g e is t h a t t o d e d u c e P r o p . 5 .4 , o n e m u s t a d d a c o m p o n e n t a c c o u n t i n g f o r xo t o t h e sets S j v , w h i c h n o w b e c o m e s u b s e t s o f R r a X C 3 n X Z c o n s i s t i n g o f a l l p o i n t s (XQ, X, X 1 , X 2 , Z) s a t i s f y i n g t h e s a m e d e f i n i n g c o n d i t i o n s as b e f o r e . I n d e e d , t h e m e t h o d o f t h e p r e v i o u s p a r a g r a p h w i l l s h o w t h a t t h e r a n d o m v a r i a b l e xo c a n b e c o n s i d e r e d as a n a d d i t i o n a l c h o i c e v a r i a b l e , p r o v i d e d s o m e c o n d i t i o n is i m p o s e d t o e n s u r e t h a t a l o n g a n y m i n i m i z i n g s e q u e n c e , E | X Q | 2 is u n i f o r m l y b o u n d e d . ( T h i s n o t o n l y p r e s e r v e s t h e c o n c l u s i o n o f L e m m a 4 . 1 , b u t a lso i m p l i e s t h a t x * is a t i g h t s e q u e n c e o f r a n d o m e l e m e n t s o f R n . ) D e m a n d i n g t h a t - 95 -t h e r a n d o m i n i t i a l v a l u e t a k e i t s v a l u e s i n s o m e p r e d e t e r m i n e d c o m p a c t se t A C R n w o u l d c e r t a i n l y a c c o m p l i s h t h i s , as w o u l d s e v e r a l w e a k e r c o n d i t i o n s . I n d e e d , b y e l i m i n a t i n g w - d e p e n d e n c e f r o m t h e s e a r g u m e n t s , w e c a n see t h a t e x i s t e n c e is a lso a s s u r e d i f XQ i s a s s u m e d t o b e a deterministic c h o i c e v a r i a b l e i n t h e c o m p a c t set A. C o n s t r a i n t s o f t h e f o r m d i s c u s s e d i n p a r a g r a p h 6.3 c a n b e i n c l u d e d i n a n y p r o b l e m w h e r e XQ is f r e e a n d r a n d o m , a n d t h e g e n e r a l p o i n t c o s t s I c a n t h e n b e a l l o w e d t o d e p e n d u p o n x o i E x o b y t h e e x p l a n a t i o n o f p a r a g r a p h 6 . 4 . T h e s p e c i a l case o f T h m . 5 . 1 a r i s i n g w h e n t h e c o n t r o l se t U i s c o m p a c t is r a t h e r w i d e l y a p p l i c a b l e , b u t i t d i f f e r s f r o m s t a n d a r d r e s u l t s i n s e v e r a l w a y s t h a t d e s e r v e a c l o s e r l o o k . I n t h i s s e c t i o n w e c o m p a r e t h i s s p e c i a l case t o t h e r e s u l t s o b t a i n e d w h e n e x p l i c i t a p p e a l s t o t h e c o m p a c t n e s s o f U a re a l l o w e d t h r o u g h o u t t h e p r o o f . K u s h n e r ' s w o r k ( 1 9 7 5 ) is r e p r e s e n t a t i v e o f t h e l a t t e r . W h e n r e s t r i c t e d t o t h e d e t e r m i n i s t i c r e g i m e , o u r d i s c u s s i o n m a y b e v i e w e d as a c o m p a r i s o n o f T h e o r e m s I I I . 2 . 1 , p . 6 3 , a n d I I I . 4 . 1 , p . 6 8 , o f F l e m i n g a n d R i s h e l ( 1 9 7 5 ) . T o m a k e f o r a m e a n i n g f u l d i s c u s s i o n , l e t us s k e t c h t h e u s u a l t h e o r y f o r t h e case w h e n U is c o m p a c t . I t b e g i n s w i t h h y p o t h e s e s a l l o w i n g r a n d o m i n i t i a l c o n d i t i o n s (see p a r a g r a p h 6 . 5 ) . ( h i ) S a m e as ( H i ) . ( h 2 ) S a m e as ( H 2 ) . ( h 3 ) T h e r e a re a n e x p o n e n t q > 2 a n d a c o n s t a n t k2 > 0 s u c h t h a t f o r a l l a d m i s s i b l e i n i t i a l v a l u e s x o i o n e h a s E \ xo\q < k2. ( N o t e : i f q > 2 t h e n w e c a n t a k e 0 = 1 , /Ci = ki i n ( h 2 ) . ) ( h 4 ) S a m e as ( H 4 ) . ( h 5 ) F o r s o m e c o n s t a n t s q 6 [0, q) a n d r > 0 , o n e h a s a c o n s t a n t fc3 > 0 s u c h t h a t S e c t i o n 7 . A C o m p a c t C o n t r o l S e t \L{t,x,u)\ < fc3(l+ | | * | | ? ) \t(v,e)\ o lly-*ll<« |»-t|<« - 9 6 -f o r a l l (t,x) e [0 , T ] X Cn, w i t h t h e p o s s i b l e e x c e p t i o n o f a set w h o s e p r o j e c t i o n o n t o t h e t - a x i s h a s L e b e s g u e m e a s u r e z e r o . 7 . 1 Theorem. Suppose that U is compact, and that (hl)-(h6) hold. (a) If problem (P) has an admissible pair then it has a solution. (b) Indeed, let any sequence of admissible pairs {(uk,xk)} for (P) be given, such that the objective values A [ u f c , xk] converge to some real number A. Then there is an admissible pair (u,x) for (P) such that, along a subsequence, xk-^-*x in Cn and A[u,x] = A. Proof. T h i s is e s s e n t i a l l y K u s h n e r ' s ( 1 9 7 5 ) T h e o r e m 3 . 1 , p . 3 5 0 . W e t h e r e f o r e s i m p l y s k e t c h t h e p r o o f , e m p h a s i z i n g i t s d i f f e r e n c e s f r o m t h e p r o o f o f T h e o r e m 5 . 1 . T h e s e q u e n c e o f 5 - t u p l e s (xk, xk, x1,k, x2'k, zk) is s t i l l t i g h t , b u t i n a d i f f e r e n t m e t r i c s p a c e . T h e g r o w t h c o n d i t i o n s ( h 5 ) i m p l y t h a t {zk} is tight in C. I n d e e d , w e c a l c u l a t e s u p \zk(t)-zk(s)\ < s u p / J f c 3 ( l + | | a f c | | * ) d r . 0 0 i n d e p e n d e n t o f k s u c h t h a t E s u p \zk(t) -zk{s)\ < RS 0e\< RS/e. k \0 ) , L(t,w)j = E [ ( i * , i t ) | ^ a j , t h e m e a s u r a b l e s e l e c t i o n t h e o r e m o f BeneS g ives a n a d a p t e d c o n t r o l ut s u c h t h a t ( / ( * ) W ) , Z ( t , a , ) ) = ( / ( * , H e n c e ( u , i ) is a c o n t r o l - s t a t e p a i r s o l v i n g ( 4 . 1 ) . M o r e o v e r , L is exactly L, n o t s o m e t h i n g l a r g e r as i n P r o p . 5 .7 . ( T h i s is c r i t i c a l . ) S i n c e zk{T)-^z{T) a n d {zk(T)} i s u n i f o r m l y i n t e g r a b l e b y ( h 5 ) , w e h a v e E z f c ( T ) — E z ( T ) . A l s o , E £ ( z f c r . , E xf cr ) — • E £ ( z 5 - , E if c - ) b y ( h 5 ) a n d t h e u n i f o r m i n t e g r a b i l i t y a r g u m e n t f o l l o w i n g e q u a t i o n (6 .2 ) i n p a r a g r a p h 6 .3 . So i n d e e d A f u * , ! * ] — • A [ u , i ] = A. / / / / N o w l e t u s c o m p a r e T h e o r e m s 5 . 1 a n d 7 . 1 . T h e p r a c t i t i o n e r w h o o n l y n e e d s a n e x i s t e n c e r e s u l t w i l l u n d o u b t e d l y p r e f e r T h e o r e m 5 . 1 . F o r h y p o t h e s e s ( h l ) - ( h 4 ) a r e i d e n t i c a l t o ( H l ) - ( H 4 ) i n t h e case o f c o m p a c t U, w h e r e a s ( H 5 ) n o w r e q u i r e s o n l y t h a t £ a n d L b e b o u n d e d b e l o w — a c o n s i d e r a b l y w e a k e r c o n d i t i o n t h a n t h e g r o w t h c o n d i t i o n s o f ( h 5 ) . A l s o , ( h 6 ) i m p l i e s t h a t Qo[t, x) i s c o m p a c t a n d c o n v e x f o r e a c h (t, x), w h e n c e Q(t,x) = Qo(t,x) + { ( 0 , r ) : r > 0 } is c l o s e d a n d c o n v e x : i n t h e v e r y c o m m o n s e t t i n g w h e n / a n d L a re c o n t i n u o u s i n t, t h i s is a s i t u a t i o n i n w h i c h ( H 6 ) is k n o w n t o h o l d . So i n c o m p a r i n g T h e o r e m s 5 . 1 ( a ) a n d 7 . 1 ( a ) , w e see t h a t T h m . 5 . 1 ( a ) o b t a i n s t h e s a m e c o n c l u s i o n as T h m . 7 . 1 ( a ) u n d e r w e a k e r a s s u m p t i o n s . H o w e v e r , t h e m e t h o d s o f T h m . 5 .1 d o n o t c o m p l e t e l y ec l i pse t h o s e o f T h m . 7 . 1 b e c a u s e o f t h e d i f f e r e n c e s m a n i f e s t e d i n t h e i r ( b ) p a r t s . T h e s e s h o w t h a t t h e w e a k e r h y p o t h e s e s o f T h m . 5 . 1 ( b ) l e a d o n l y t o a " l o w e r - s e m i c o n t i n u i t y " r e s u l t , o n A , w h e r e a s T h m . 7 . 1 ( b ) c o n c l u d e s t h a t A is " c o n t i n u o u s ' ' i n s o m e sense. O f c o u r s e t h e s e d i f f e r e n t s t a t e m e n t s lose t h e i r d i s t i n c t i v e n e s s w h e n a p p l i e d t o a m i n i m i z i n g s e q u e n c e i n a n a t t e m p t t o p r o v e e x i s t e n c e , b u t t h e r e a re s i t u a t i o n s i n w h i c h t h e c o n t i n u i t y c o n c l u s i o n s o f T h m . 7 . 1 h a v e o t h e r uses. W e w i l l d i s c u s s s u c h a case i n C h a p t e r V . - 98 -A p p e n d i x . G o o r ' s E x i s t e n c e T h e o r y . R o b e r t M . G o o r ( 1 9 7 6 , 1979) h a s r e c e n t l y p r o p o s e d c e r t a i n n e w e x i s t e n c e t h e o r e m s f o r s t o c h a s t i c o p t i m a l c o n t r o l p r o b l e m s w h i c h a p p e a r t o g e n e r a l i z e t h e w o r k o f K u s h n e r ( 1 9 7 5 ) . T h i s a p p e n d i x d e m o n s t r a t e s t h a t a k e y l e m m a u s e d i n G o o r ' s " p r o o f o f t h e s e r e s u l t s is f a l s e , a n d h e n c e t h a t h i s a s s e r t i o n s a b o u t e x i s t e n c e m u s t b e r e g a r d e d as c o n j e c t u r e s — n o t as t h e o r e m s . G o o r ' s s t a t e m e n t s a b o u t e x i s t e n c e a re d i s t i n g u i s h e d b y t h e i r s t r o n g f o r m u l a t i o n . I n o t h e r w o r d s , t h e p r o b a b i l i t y s p a c e ( Q , 7, P) a n d B r o w n i a n m o t i o n Wt a re fixed t h r o u g h o u t h i s a r g u m e n t s , r a t h e r t h a n b e i n g c o n s i d e r e d as a d d i t i o n a l c h o i c e v a r i a b l e s . T h e f o u n d a t i o n f o r t h e s t r o n g a p p r o a c h is L e m m a 1.7, p . 9 0 9 o f G o o r ( 1 9 7 6 ) . I t i n v o l v e s a s e p a r a b l e B a n a c h space Y a n d a c o m p l e t e s e p a r a b l e m e t r i c s p a c e X. B o t h X a n d Y a r e e q u i p p e d w i t h t h e i r B o r e l a - f i e l d s , a n d t h e i d e n t i t y m a p o n X is d e n o t e d b y i ' x - A n o n a t o m i c p r o b a b i l i t y m e a s u r e p is g i v e n o n X. G o o r ' s L e m m a . Let yk'-X —* Y, k = 1 , 2 , . . . , be a s e q u e n c e of measurable maps such that fx ||yfc(a;)|| dp(x) < +00 for each k and such that the sequence of probability measures Pk = M ^ X i S / f c ) - 1 » d e f i n e d on the Borel subsets of X xY, converges w e a i i y to some probability measure P. Then there exists a measurable map y: X —• Y such that P = p{ix, S / ) - 1 > i e > ( * x , yfc) converges in distribution to ( t . x , y ) . C o u n t e r e x a m p l e . L e t X = [ 0 , l ] a n d w r i t e p f o r L e b e s g u e m e a s u r e . T a k i n g Y = R , d e f i n e t h e s e q u e n c e o f s i m p l e m e a s u r a b l e f u n c t i o n s y ^ : [ 0 , l ] —• R v i a y fc( t ) = tk, w h e r e t = 0.£1*2*3 • • • is t h e b i n a r y e x p a n s i o n o f t. ( T o e n s u r e t h a t yk is w e l l - d e f i n e d , w e i n s i s t t h a t t h e b i n a r y e x p a n s i o n o f e v e r y t G [O, l ) c o n t a i n i n f i n i t e l y m a n y z e r o s , a n d set y j c ( l ) = 1 V/c.) I t is a s i m p l e m a t t e r t o v e r i f y t h a t t h e p r o b a b i l i t y m e a s u r e s Pk d e f i n e d o n t h e B o r e l sets o f [ 0 , l ] x R b y Pk(S) =p{te [ 0 , 1 ] : (t,yk{t)) e S } c o n v e r g e w e a k l y t o t h e p r o d u c t m e a s u r e P = p X (|5 {o} + 2 ^ { ! } ) - G o o r ' s s t a t e m e n t c l a i m s t h a t t h e r e e x i s t s a m e a s u r a b l e y : [ 0 , l] —» R s u c h t h a t P = / x ( i ' x , y ) - 1 . T h i s is a b s u r d . I n d e e d , i f s u c h a m a p p i n g y d i d e x i s t , t h e n c l e a r l y y(t) = 0 o n a set Z C [o, l] o f m e a s u r e p{Z) = | . I n t e r m s o f Z, w e m a y c o n s t r u c t t h e m e a s u r a b l e s u b s e t S = Z X { 1 } o f [ 0 , l ] x R f o r w h i c h p [t : ( t , y(t)) £ S) = 0 , w h e r e a s P(S) = ^p(Z) > 0. T h i s is a c o n t r a d i c t i o n . - 9 9 -G o o r ' s P r o o f . G o o r ' s p r o o f o f t h e l e m m a q u o t e d a b o v e is m a r r e d b y a s u b t l e m i s u s e o f m u l t i - i n d e x n o t a t i o n . T h e b a s i c i d e a is t o c o n s t r u c t m a p p i n g s f r o m t h e i n t e r v a l [ 0 , l ] i n t o X xY w h i c h g i v e r i se t o t h e l a w s Pk a n d P. S k o r o k h o d ( 1 9 6 5 ) , p . 1 0 , p r o v i d e s a m o d e l f o r t h i s c o n s t r u c t i o n : a s e q u e n c e o f n e s t e d p a r t i t i o n s o f t h e r a n g e space is u s e d t o d e f i n e a c o r r e s p o n d i n g s e q u e n c e o f n e s t e d p a r t i t i o n s o f t h e i n t e r v a l [ 0 , l ] . S k o r o k h o d ' s c o n s t r u c t i o n e n s u r e s t h a t t h e c o r r e s p o n d e n c e b e t w e e n t h e p a r t i t i o n s o f X x Y a n d o f [ 0 , l ] p r e s e r v e s t h e r e l a t i o n s h i p o f set i n c l u s i o n ; G o o r ' s c o n s t r u c t i o n does n o t . So p a r a g r a p h 5 o f h i s p r o o f d o e s n o t r e a l l y " f o l l o w S k o r o k h o d ' s m e t h o d . " T h e m a p p i n g s d e f i n e d i n t h a t p a r a g r a p h a re not " d e f i n e d i n a n a l o g y t o t h e c o n s t r u c t i o n o f S k o r o k h o d , " so c o n v e r g e n c e m a y w e l l f a i l . ( I n t h e case o f t h e c o u n t e r e x a m p l e a b o v e , f o r i n s t a n c e , t h e m a p p i n g s z™ d o n o t c o n v e r g e a t a l l . ) O f c o u r s e , t h e c o n c l u s i o n s o f G o o r ' s p a r a g r a p h 5 c a n b e r e s c u e d b y t a k i n g m o r e c a r e w i t h t h e c o r r e s p o n d e n c e b e t w e e n t h e p a r t i t i o n s o f X X Y a n d [ 0 , l ] . H o w e v e r , t h e i n c o r r e c t c o r r e s p o n d e n c e is e s s e n t i a l t o t h e d e v e l o p m e n t o f p a r a g r a p h 6. I f p a r a g r a p h 5 is t o b e s a l v a g e d t h e n t h e a s s e r t i o n t h a t h = h-k, k = 0 , 1 , 2 , . . . i n p a r a g r a p h 6 m u s t b e d i s c a r d e d . ( T h i s a s s e r t i o n is m a n i f e s t l y fa lse i n t h e c o u n t e r e x a m p l e d i s c u s s e d a b o v e . ) B u t t h i s s t a t e m e n t is t h e k e y t o t h e w h o l e p r o o f . A c k n o w l e d g e m e n t . T h e a u t h o r t h a n k s U l r i c h H a u s s m a n n a n d E d P e r k i n s f o r i n s i s t i n g o n a s e c o n d l o o k a t G o o r ' s L e m m a . P r o f e s s o r P e r k i n s s u g g e s t e d t h e c o u n t e r e x a m p l e a b o v e . - 100 -C h a p t e r V . P a r a m e t e r S e n s i t i v i t y i n S t o c h a s t i c O p t i m a l C o n t r o l T h i s c h a p t e r is d e v o t e d t o a s t u d y o f d e t e r m i n i s t i c p e r t u r b a t i o n s o f t h e c o n s t r a i n e d s t o c h a s t i c c o n t r o l p r o b l e m i n t r o d u c e d i n C h a p t e r I V . O u r a p p r o a c h is b a s e d o n a p r o x i m a l - n o r m a l a n a l y s i s o f t h e p r o b l e m ' s v a l u e f u n c t i o n , a n d h e n c e m a k e s e x p l i c i t use o f t h e e x i s t e n c e t h e o r y o f C h a p t e r I V , t h e l i m i t i n g t e c h n i q u e s a n d r e p r e s e n t a t i o n t h e o r e m s o f S e c t i o n s I V . 2 - 3 , a n d t h e u n c o n s t r a i n e d S t o c h a s t i c M a x i m u m P r i n c i p l e . T h e l a t t e r , w h i c h n e c e s s i t a t e s a s m o o t h f o r m u l a t i o n , is p r e s e n t e d i n t h e f i r s t t h r e e s e c t i o n s b e l o w . S e c t i o n 1 i n v e s t i g a t e s h o w s l i g h t p e r t u r b a t i o n s o f t h e s y s t e m ' s i n i t i a l v a l u e a n d c o n t r o l l a w a f f e c t i t s e v o l u t i o n ; i n S e c t i o n 2 , t h e c o n s e q u e n c e s o f t h i s v a r i a t i o n f o r t h e cos t f u n c t i o n a l a re c o n s i d e r e d . T h e s e p r e l i m i n a r y r e s u l t s a l l o w t h e d e r i v a t i o n o f t h e u n c o n s t r a i n e d S t o c h a s t i c M a x i m u m P r i n c i p l e i n S e c t i o n 3. I n t h a t s e c t i o n w e d i s c u s s t h e c o n c l u s i o n s a v a i l a b l e i n t h e n o n a n t i c i p a t i v e f o r m u l a t i o n , a n d d e s c r i b e a g e n e r a l t y p e o f f e e d b a c k f o r m u l a t i o n w h i c h g ives m o r e s a t i s f y i n g r e s u l t s . P r o x i m a l n o r m a l a n a l y s i s is t h e s u b j e c t o f S e c t i o n 4 , w h i c h is t h e h e a r t o f t h i s c h a p t e r . T h e r e , a f a m i l y o f s t o c h a s t i c c o n t r o l p r o b l e m s i n d e x e d b y a finite-dimensional p a r a m e t e r is u s e d t o d e f i n e a " v a l u e f u n c t i o n " w h o s e g e n e r a l i z e d g r a d i e n t is c a p t u r e d i n T h m . 4 . 8 . S e c t i o n 5 e x p l o r e s s o m e c o n s e q u e n c e s o f t h i s c h a r a c t e r i z a t i o n , w h i c h i n c l u d e a n e w p r o o f o f t h e S t o c h a s t i c M a x i m u m P r i n c i p l e f o r c o n s t r a i n e d p r o b l e m s . H y p o t h e s e s . I n t h i s c h a p t e r w e s t u d y s t o c h a s t i c s y s t e m s o f a m o r e s p e c i a l i z e d f o r m t h a n t h o s e o f C h a p . I V . W e s t i l l a s s u m e t h a t U C R m is a g i v e n c l o s e d se t , b u t n o w t h e s y s t e m ' s e v o l u t i o n o n t h e fixed i n t e r v a l [ 0 , T] is d e t e r m i n e d b y t h e Markovian I t o e q u a t i o n S e c t i o n 1 . P e r t u r b e d D y n a m i c s (1 .1 ) - 1 0 1 -M o r e o v e r , a t l e a s t f o r t h e f i r s t t w o s e c t i o n s , w e a s s u m e t h a t t h e filtered s p a c e ( f t , 7, 7t, P) i s fixed i n a d v a n c e , a n d t h a t i t c a r r i e s a d - d i m e n s i o n a l ^ - B r o w n i a n m o t i o n wt w h i c h c a n n o t b e c h a n g e d . T h e c o e f f i c i e n t s / : [0,T\ x R " x R m — • R n a n d c:\0,T\ x R n - » R n X d m u s t s a t i s f y ( H 1 ) - ( H 2 ) b e l o w , w h i c h a re s t r o n g e r c o n d i t i o n s t h a n t h e i r c o u n t e r p a r t s i n C h a p . I V . ( H I ) f(t, x, •) is a c o n t i n u o u s f u n c t i o n o f u £ U, u n i f o r m l y i n (t, x); a l so , f o r e a c h (t, u) € [0 , T ] x U, b o t h f(t, -,u) a n d a(t, •) a re c o n t i n u o u s l y d i f f e r e n t i a b l e f u n c t i o n s o f x £ R " . ( H 2 ) T h e r e is a c o n s t a n t kx > 0 s u c h t h a t f o r a l l ( t , x, u) £ [ 0 , T j x R " x U, o n e h a s \f(t,x,u)\ + \ 2 , w e fix o u r a t t e n t i o n o n a s p e c i f i c p a r a m e t e r v a l u e a a n d m a k e t h e f o l l o w i n g h y p o t h e s i s . ( H 3 ) T h e r e a re J 0 - m e a s u r a b l e r a n d o m v e c t o r s X0 € R n a n d A € R n x a s u c h t h a t E | J f 0 T < + ° ° a n d |J4(O>)| < fc2 f o r a l l w, a n d XQ(Q) = XQ + A(a — a). N o t e t h a t w h e n Xo = 0 , a = n, a n d A = I, ( H 3 ) a l l o w s t h e case o f a v a r i a b l e d e t e r m i n i s t i c i n i t i a l c o n d i t i o n xo{ct) = a — a; a n d t h a t w h e n 5 = 0, a = 1 , a n d A is a b o u n d e d r a n d o m v a r i a b l e i n R r e , ( H 3 ) a l l o w s t h e s c h e m e xo[ct) = Xo + ctA r e m e n i s c e n t o f t h e c a l c u l u s o f v a r i a t i o n s . V a r i a t i o n s . S u p p o s e a n y ^ - a d a p t e d c o n t r o l p r o c e s s u is g i v e n , a n d l e t x d e n o t e t h e c o r r e s p o n d i n g s o l u t i o n o f ( 1 . 1 ) . W e w i s h t o i n v e s t i g a t e t h e d i f f e r e n c e b e t w e e n x a n d t h e s o l u t i o n x o b t a i n e d w h e n a p a i r [a, u) n e a r (a, u) is u s e d . T h e f o l l o w i n g L e m m a w i l l h e l p . - 102 -1 . 1 L e m m a . Let )\ dt < +00 a.s. Then there is a null set M C [0, T] such that for each t £ M one has d '* 1 >- > • * — / —• • — >" 1 — i a . s . at f r o o / . See K u s h n e r ( 1 9 7 2 ) , L e m m a 1 , p . 5 5 6 . / / / / T o a p p l y L e m m a 1 . 1 , first t a k e 0 b y ( ut i f t e [0 , s - s ] u t i f t S (s — e, s] u t i f te[s,T\. T h e p e r t u r b e d p a r a m e t e r v a l u e s w i l l b e a e : = a + e a . L e t x e d e n o t e t h e s o l u t i o n o f (1 .1 ) c o r r e s p o n d i n g t o ( a £ , u e ) . I n t h i s s e c t i o n w e w i l l i n v e s t i g a t e t h e e v o l u t i o n o f : = x\ — x t . T o i m p r o v e t h e r e a d a b i l i t y o f t h e r e s u l t s , w e w i l l use t h e f o l l o w i n g n o t a t i o n : A / £ ( t , x ) = f{t,x, ul)-f{t,x,ut), f(t) = / ( t , x ( t ) , u ( t ) ) , fx(t) = fx(t,x(t)Mt)), c{t) = o-{t, x ( t ) ) , $x(t) = o+ T h e k e y r e s u l t is P r o p o s i t i o n 1.6: i t s t a t e s t h a t t h e d i s c r e p a n c y & , w h i c h is d e f i n e d b y £ = ( x 0 ( a£ ) - x 0 ( S ) ) + f\f{r,xl,ur) - f(r,xr,ur) + A / £ ( r , x < ) ) d r (1 -3 ) J ° t + / ( « r ( r , x £ ) - a ( r , x r ) ) d i y r , Jo is v e r y w e l l a p p r o x i m a t e d b y t h e v a r i a t i o n - o f - p a r a m e t e r s s o l u t i o n o f t h e i n h o m o g e n e o u s l i n e a r i z e d e q u a t i o n yl = eAa + [\fx(r)y< + A / £ ( r , x r ) ) dr + f ax(r)y* dwr. Jo Jo - 103 -1.2 Lemma. For each p e [0 , q], there is a c o n s t a n t M > 0 f o r which E\\V\\p 0 . Proof. O b s e r v e first t h a t b y J e n s e n ' s i n e q u a l i t y w e h a v e E HT< ( E | | e T ) P ' V p e [ 0 , g ] . H e n c e t h e g e n e r a l r e s u l t f o l l o w s f r o m t h a t f o r t h e case p = q, w h i c h w e n o w t r e a t . N o t e t h a t A / e ( r , x ) = 0 u n l e s s r e (s — e, s\. H e n c e t h e r e is a c o n s t a n t K\ > 0 f o r w h i c h J e n s e n ' s i n e q u a l i t y g i ves 0 , E \\et < K2E(|x0(a£) - x 0 ( 3 ) | * + / " | / ( r , C P ) - f(r,xr,ur)f d r (1 .4 ) t t + e ' - 1 | A / ' ( r , x £ ) | « d r + £ | a ( r , * « ) - a ( r , xV)f dr) . N o w | x o ( a e ) — x o ( 2 ) | < k2e | a | b y ( H 3 ) , a n d \f(r,xcr,ur) - / ( r , x r , u r ) | < ^ | £ r | , ^ ( r . x ^ - c r ^ x ^ l ^ f c x l ^ l ( b y ( H 2 ) ) . M o r e o v e r , t h e l i n e a r g r o w t h c o n d i t i o n o f ( H 2 ) i m p l i e s t h a t |Ar ( r , x £ ) | < |/(r,x«,^)l + | / ( r , S r ) | < 2 * 1 ( l + |x«|), so t h e r e a re c o n s t a n t s .K^ , 7^4 s u c h t h a t E | A / £ ( r , x £)r < ^ ( l + E l x ' l 5 ) < i f * -( T h e s e c o n d i n e q u a l i t y f o l l o w s f r o m L e m m a I V . 4 . 1 . ) U s i n g t h e s e f a c t s i n (1 .4 ) i m p l i e s t h a t , f o r s o m e c o n s t a n t Ks, E||r||F 0 for which one has E sup |A/£(t,a;t£) - Af t(t,xt)\q < Me9 Ve > 0 . te[o,r] Proof. The Lipschitz condition implied by (H2) gives \Ar(t,xt) - A/£(t,zt)| < \f{t,x\,u\) - f(t,xuul)\+ \f(t,xi*) - f(t,xu^t)\ < 2k,\Q Thus the result follows directly from Lemma 1.2. //// Now that we know is a process relatively small in magnitude, we turn with confidence to the linearization of (1.1) about (3, x, u): Note that since a is an n x d matrix, we must interpret d axydxv = akydwk, k = l where ak is the k-th column of a and wk is the fc-th component of w. Note also that equation (1.5) satisfies the ItS conditions, so has a pathwise unique solution t/£ for every e > 0 . Let us compare y\ Proof. Let z\ = £| — y\. Then by the mean-value theorem there are a constant 6 l and processes ip\, V>t, all with values in [0, l], for which (1.5) with ft. 1.4 Lemma. For any p €E [2 , q), there is a function 6(e) ~ 0 such that Ve > 0 . where H€ = e(Dx0(a + ed'a) - A)a, - 105 -Now for any p S [2, q), (1.6) E |iiP| p < e" |a|PE \Dx0(a + e6 ca) - A\ p =: e ph(e). The function h(e) ~ 0 because DXQ is continuous for each co and uniformly bounded by (H3). Next, Corollary 1.3 gives a constant M such that E | | / T < E ^ _jAr(r,x*r) - A f ( r , J r ) | drj < Ee?' 1 [' | A / £ ( r,x^ ) - A / £ ( r,x r)| p dr (1.7) •/«-« < e p _ i / Mepdr J i—t < M e 2 p . To control J/, we use (BDG): E||J£||P < C P E / " T \ { c T x { r , x r + rrC)-^{r,xr))C\P dr Jo < CPE [WW* fT \ax(r,xr + - a,(r)|" dr ) (1.8) \ Jo } < C'p (E | | *T)P / ' E k,(r, £ r + rX) ~ Zx(r)r/{*-p) dr^ < e pj(e). In the last step, the second factor is bounded by a multiple of e p by Lemma 1.2, and the function j(e) is then defined as the appropriate constant multiple of the third factor. Note that j(e) ~ 0 because crx(r, •) is a bounded continuous function and tends to 0 in probability for each r. In just the same way, there is a function k(e) ~ 0 for which ._ / .T -/(-- \ \ ' ' - p ' / '* ( i g ) E\\KT 0, E||*lf < * ( £ E | | Z T dr + sp6(e)y Gronwall's inequality now implies that E ||z*||p < Me p8(e) for some M > 0, as required. / / / / - 106 -T o c o m p l e t e o u r s t u d y o f t h e e v o l u t i o n o f £ | , w e use t h e v a r i a t i o n o f p a r a m e t e r s f o r m u l a t o w r i t e t h e a p p r o x i m a t e s o l u t i o n o f t h e i n h o m o g e n e o u s e q u a t i o n ( 1 . 5 ) i n t e r m s o f i t s h o m o g e n e o u s c o u n t e r p a r t ( 1 . 1 0 ) d s}$(t,s)[f(s,x„u,) - f(s,x„u,))). 1 .5 P r o p o s i t i o n . For a n y p £ [2, q) there is a function 8(e) ~ 0 for which E | £ - e ( * ( t , 0 ) A a + I{t > •}*(«, s) [f(s, x„ u.) - / ( « , x , , u . ) ] ) | P < e"6(e) holds for all t £ [0 , s — e] U [ s , T ] a n d for all e > 0. F o r t £ (s — e, s) there is a constant M > 0 for which the inequality remains valid if the right-hand side is replaced by epM. Proof. I n v i e w o f L e m m a 1.4, i t su f f i ces t o p r o v e t h i s e s t i m a t e w i t h £ | r e p l a c e d b y y{. A n d b y d e f i n i t i o n o f t h e d i f f e r e n c e y\ - e($(t,0)Aa+ I{t > s}$(t, s) [f(s, x„ v.,) - f(s,x„u,)]) is z e r o f o r a l l t € [0 , s — e ] , a.s. W e t h e r e f o r e a d v a n c e t o t h e case t £ (s — e, s), w h e r e w e h a v e ( 1 . 1 1 ) y\ = e$(s-e,0)Aa+ f fx(r)yrdr+[ ax(r)yr dwr + f A / £ ( r , x r ) d r . T h e first t e r m o b e y s ( 1 . 1 2 a ) E | e $ ( s - e,0)Aa - s$(s,0)Aa\p < A V E | $ ( s - e , 0 ) - $ ( s , 0 ) | p = : e ^ o (e ) -- 107 -S i n c e $ is j o i n t l y c o n t i n u o u s a n d h a s m o m e n t s o f a l l o r d e r s , u n i f o r m i n t e g r a b i l i t y e n s u r e s t h a t ^ o ( e ) ~ 0- T h e first i n t e g r a l o b e y s (1 -126) I T IWr dr" 0 , E \e~x \ A / £ ( r , x r ) a Y - A / £ ( 5 , x , ) KK^c-1!' E | A / £ ( r , x r ) | ' d r + E | A r ( S , x . ) | ' ) , a n d t h e r i g h t s ide a d m i t s a c o n s t a n t u p p e r b o u n d u n i f o r m l y i n e b y L e m m a I V . 4 . 1 a n d t h e l i n e a r g r o w t h c o n d i t i o n i n ( H 2 ) . - 108 -Now for all t > s, Afe(t,xt) = 0, so (1.5) implies that j/| = $(i,s)t/*. For any fixed p in [2,g), we choose a q €E (p, q). Then the uniform boundedness of the moments of $ gives a constant K > 0 for which E \yl - e^[t,s)Ar(s,x.)\P < E|$(t,s)| p \y\ - eAr(s,x.)\p < (E|$(t > S)r^- p ) ) ( , ~ P ) / ' (E |y ' -eAr { s ,x , )\P) and Brownian motion u>t of Section 1 throughout this section. The cost of a given initial condition a and ^-adapted control process u : [0, T] X ft —• U is measured by the functional A[a,tt] := E Here, as in Section 1, it denotes the pathwise unique strong solution of the Ito equation (l.l) under Hypotheses (Hl)-(H3). The constant vector /9 € R ° is included for later theoretical use, even though many applied problems are covered by the case /9 = 0. The pointwise cost £:R n —» R and the running cost L: [0, T] x R™ x U —» R must satisfy (H4) and (H5) below. (H4) L(t, x, •) is a continuous function of u € U, uniformly in (t, x); also, for each fixed (t, u) G [0, T] X U, both £ and L(t, •, u) are continuously differentiable. (H5) There is an exponent q € [l,g — 1) and a constant > 0 such that for all (t,x, u) e [ 0 , T]xR"x U} one has \L(t,x,u)\< M l + 1*1* + |« |«) , \L*(t,x, u)| < /c3(l + I x l ' " 1 + M ' ) , \t(x)\ < * 3 ( i + w ) , i4(x)i < M I + i ^ r 1 ) . The Ito conditions and (H5) imply that A[a, u] is well-defined and finite for any ^-adapted stochastic process u : [0, TJ x ft —• U obeying (2.1) E f \urf dr < +oo. Jo - 109 -(ft a) + l(xT) + f L(t,xt,ut)dt Jo V a r i a t i o n s . S u p p o s e n o w t h a t t h e c o n t r o l s t r a t e g i e s Ut a n d u» f e a t u r e d i n S e c t i o n 1 a c t u a l l y o b e y c o n d i t i o n ( 2 . 1 ) . T h e n so d o e s u £ f o r e v e r y e > 0 , so A [ o : £ , t t e ] is w e l l - d e f i n e d f o r a l l e > 0 . M o r e o v e r , ( H 4 ) a n d ( H 5 ) v e r i f y t h e h y p o t h e s e s o f L e m m a 1.1 f o r t h e f u n c t i o n s s } $ ( r , s) A / £ ( s , x,)] dr + / £ + J £ , Jo - 110 -w h e r e V = C' Zx{r)[& - e(9(r,0)Aa + I{r > s } $ ( r , a) A / £ ( s , x , ) ) ] dr, Jo J<=[ (Lx(r,xr + BrVr,uT) - 2 , ( r ) ) £ dr. Jo N o w f o r a c o n s t a n t p < q s u f f i c i e n t l y n e a r q t h e c o n j u g a t e e x p o n e n t o b e y s < q, s o P r o p . 1.5 p - 1 g ives a c o n s t a n t M > 0 a n d a f u n c t i o n So ~ 0 s u c h t h a t E | / £ | < £ (E | £ * ( r ) | ^ ) ' (E\C - e(*{r,0)Aa + I{r > s } * ( r , . ) A / « ( * , x.)) \") ' dr < M e £ ^ E | } ! x ( r ) | ^ ) " dr + e60(e) £ \lx(r)\ ^ ) ' dr =:eSi(e). S i n c e E Lx(r) < K ( l + E | x r | ' + E\u rf) is i n t e g r a b l e , so is i t s - — - p o w e r : h e n c e 6 i ( e ) ~ 0 . S i m i l a r l y , L e m m a 1.2 i m p l i e s t h a t s o m e K > 0 o b e y s V\Je\ J* = f ( A L £ ( r , x £ ) - A L £ ( r , x r ) ) dr. J > — t N o w t h e e x i s t e n c e o f i(e) ~ 0 s u c h t h a t E|/£| < er ' (e) f o l l o w s f r o m a s s u m p t i o n (2 .2 ) j u s t as l i n e ( 1 . 1 4 ) f o l l o w e d f r o m ( 1 . 2 ) . A n d J £ = I (L(r,x\,ur) - L(r,xr,uT)) dr + f (L(r,xr,ur) - L(r,xer,ur)) dr. Js-t Jt—t T h e s e t w o i n t e g r a l s c a n b e t r e a t e d s i m i l a r l y , so w e d i s c u s s o n l y t h e first o n e . B y t h e M e a n V a l u e t h e o r e m t h e r e is a p r o c e s s 6T w i t h v a l u e s i n [ 0 , 1 ] s u c h t h a t t h e e x p e c t a t i o n o f t h e first i n t e g r a l is 1 1 1 m a j o r i z e d b y E f \Lx(r,xr + 6rer,ur)£\ dr < f ( E | L x ( r , 2 r + * r £ , u r ) | ^ ) V (E||r||P)' dr < K e f ( E | L x ( r , £ r + 0 r £ , u P ) | ^ ) V d r = : e j ( « ) . H e r e j ( c ) ~ 0 b e c a u s e t h e g r o w t h c o n d i t i o n s o f ( H 5 ) a n d ( 2 . 1 ) i m p l y t h a t t h e q u a n t i t y i n p a r e n t h e s e s is i n t e g r a b l e . F i n a l l y , w e c o n s i d e r t h e p o i n t c o s t s . T h e s e o b e y ( 2 . 6 ) E(/(*5.) - £(xT)) =E[eis(xT)(9[T,0)Aa + 9[T, S ) A / e ( s , x , ) ) + V + Je] w h e r e F = lx{xT) [ f f - e ( * ( T , 0 ) i 4 o + * ( ! , a ) A / £ ( s , 2 , ) ) ] , J £ = ( ^ ( 2 r + ^ f ) - 4 ( * r ) ) e f f o r s o m e t9£ €E [ 0 , 1 ] . H o l d e r ' s i n e q u a l i t y , u n i f o r m i n t e g r a b i l i t y , a n d L e m m a 1.2 g i v e a f u n c t i o n j(e) ~ 0 f o r w h i c h E | Jc\ < ej(e) m u c h as i n t h e a r g u m e n t s a b o v e . L i k e w i s e , E | J £ | < ei(e) f o r s o m e i(e) ~ 0 f o l l o w s f r o m P r o p . 1.5. C o m b i n i n g ( 2 . 4 ) , ( 2 . 5 ) , a n d (2 .6 ) g ives t h e d e s i r e d r e s u l t . / / / / Section S. Necessary Conditions for an Unconstrained Stochastic Control Problem T h e d a t a o f t h e p r e v i o u s s e c t i o n s a l l o w t h e f o r m u l a t i o n o f s e v e r a l s t o c h a s t i c c o n t r o l p r o b l e m s . T h e m o s t r e a d i l y c o m p r e h e n s i b l e o f t h e s e is the strong problem with nonanticipative controls: g i v e n (Q, 7, 7t,P) a n d wt as i n S e c t i o n s 1 - 2 , find a S R a a n d a n ^ - a d a p t e d p r o c e s s Ut t a k i n g v a l u e s i n U a n d s a t i s f y i n g (2 .1 ) s u c h t h a t A [ a , u ] e q u a l s t h e i n f i m u m o f A [ a , u ] o v e r a l l p o s s i b l e c h o i c e s o f ( o , u ) . N e c e s s a r y c o n d i t i o n s f o r o p t i m a l i t y i n t h i s p r o b l e m c a n b e s t a t e d i n t e r m s o f t h e pre-Hamiltonian H: [0 , T ] x R n X R " X U — R d e f i n e d b y H{t, x, p, u ) = p'f(t, x, u) - L(t, x, u). - 112 -(Prime denotes transpose.) Indeed, upon defining (3.1) P i = - M S r ) * ( r , a ) - [ 2 « ( r ) * ( r , « ) dr, the key equation of Prop. 2.1 becomes (3.2) A c — A = eE - P o - A a + H( s,x„p„u,) - H(s,x„p„u,) + eS(e). If (a, u) solves (P), then the left side must be nonnegative. Dividing equation (3.2) by e > 0 and letting e —+ 0 + then gives (3.3) 0 < E (P, a) - % A c t + H{s,x„p„u,) - H(s,x„p„u,) This conclusion is valid for all s £ AV(u, u) and all a € R ° . We obtain the following version of the stochastic maximum principle. 3.1 Proposition. Suppose (3, u) solves the strong problem with nonanticipative controls. Then the process pt defined by (3.1) has the following properties. For any 5t-adapted control u: [0, T] x Q —• U obeying the integrability condition (2.1), there is a null set M(u, u) C [0,T] such that (3.4) E i ' p 0 = 0, (3.5) EH{s,xa,pt,u,) < EH{s,x„p„u,) V s £ A / ( u , u ) . Proof. Line (3.4) holds because (3.3) is valid for all a S R a ; (3.4) and (3.3) together imply (3.5). / / / / The conclusions of Prop. 3.1 unfortunately involve the comparison control Ut in the null set M(u,u) of line (3.5). The results of Kushner (1972) and Haussmann (1985) both offer global versions of this condition. Kushner does this by introducing an explicit assumption regarding the approximability of admissible controls (his Assumption 2.3, p. 552). Haussmann, on the other hand, constructs a rather large family of comparison controls on which a global version of (3.5) holds without further hypotheses. We follow his method here. Suppose that an ^-adapted process with sample paths in CL is given. We will show that line (3.5) holds for a single null set M(u,xp) C [0, T\, provided that the comparison control ut is of - 113 -" ^ - f e e d b a c k f o r m " — t h a t i s , p r o v i d e d ut is ^ - a d a p t e d a n d o b e y s t h e i n t e g r a b i l i t y c o n d i t i o n (2.1). A t e a c h i n s t a n t t € [0, T], t h e r a n d o m v a r i a b l e tit d e f i n e d b y s u c h a c o n t r o l l ies i n t h e set uty>) = v(/L,??,PiU) (3.6) = { u ( t f ( t A - ) ) : v€ L9{Cn,Cn,Poi,- l;V)} . C o n s i d e r n o w t h e f o l l o w i n g set o f " s i m p l e ^ - f e e d b a c k c o n t r o l s . " L e t A b e t h e a l g e b r a o f s u b s e t s o f C l g e n e r a t e d b y a l l sets o f t h e f o r m {(•)) = I »=1 I{(t A -,w)) : v 6 V} . N o t e t h a t s i n c e e v e r y v £ V is a b o u n d e d f u n c t i o n , e v e r y c o n t r o l i n V is b o u n d e d , h e n c e a d m i s s i b l e . W e m a y d e f i n e t h e n u l l se t A / ( u , rp) as t h e u n i o n o f M(u, u) o v e r a l l u e "V. O u t s i d e o f t h i s n u l l s e t , b o t h (1.2) a n d (2.2) h o l d s i m u l t a n e o u s l y f o r a l l u 6 V. C o n s e q u e n t l y t h e s a m e is t r u e o f l i n e (3.5), w h i c h w e m a y n o w e x t e n d b y t a k i n g l i m i t s . 3 . 2 T h e o r e m ( S t o c h a s t i c M a x i m u m P r i n c i p l e ) . Suppose ( a , u ) solves the strong problem with nona.nticipa.tive controls. Suppose further that a continuous Tt-adapted process ipt with values in R e i s given. Then there is a null set M(u,rp) C [0, T] such that for any ipt-adapted comparison control tit obeying (2.1), one has (3.8) EA'p0 = P, (3.9) EH{s,x,,p„u,) <-EH{s,a„p„u,) Vs ^ M(u, rp). Proof. T h e first c o n c l u s i o n f o l l o w s i m m e d i a t e l y f r o m (3.4); w e n e e d o n l y t o s h o w h o w t h e s e c o n d o n e f o l l o w s f r o m (3.5). F o r t h i s , w e use t h e n u l l set M(u,rf>) d e f i n e d a b o v e , o u t s i d e o f w h i c h (3.5) h o l d s s i m u l t a n e o u s l y f o r a l l u € f . - 1 1 4 -T h e k e y t o t h e p r o o f is t h a t f o r e a c h t, t h e c o u n t a b l e se t Vt = {u(t,) : U 6 V } is a d e n s e s u b s e t o f Ut(ip) i n t h e t o p o l o g y o f L * ( f ) , 7?, P; U). T h i s is t h e c o n t e n t o f H a l m o s ( 1 9 5 0 ) , e x . 4 2 ( 1 ) , p . 1 7 7 . (See a lso H a l m o s , T h m . 4 0 . b , p . 1 6 8 ; a f e w m o r e d e t a i l s a p p e a r i n H a u s s m a n n ( 1 9 8 5 ) , S e c t i o n 5.) So f o r a n y fixed s £ M[u, rp) a n d r a n d o m v a r i a b l e u 6 [/,, t h e r e is a s e q u e n c e uk —* u a.s. a n d i n LP a l o n g w h i c h (3 .5 ) h o l d s f o r e a c h fc. W e c o m p l e t e t h e p r o o f b y s h o w i n g t h a t ( 3 . 1 0 ) EH (S, x„p„uk)^EH{s,x„p„u). I n v i e w o f t h e d e f i n i t i o n o f H, w e c o n s i d e r t h e s u m m a n d s p' f a n d L s e p a r a t e l y . H y p o t h e s i s ( H 5 ) i m p l i e s t h a t t h e s e q u e n c e E \L(s, X,, uk)f^q i s u n i f o r m l y b o u n d e d , s o EL(s, x,,uk) —*• EL(s, x,, u ) b y u n i f o r m i n t e g r a b i l i t y . A s f o r p ' / , first choose a n y r € {q,q) a n d c o n s i d e r t h e i n t e g r a l e x p r e s s i o n (3 .1 ) f o r p't. U s i n g t h e f a c t t h a t $ ( t , r ) h a s m o m e n t s o f a l l o r d e r s , e a c h o f w h i c h is b o u n d e d u n i f o r m l y i n fc, r e p e a t e d a p p l i c a t i o n o f H o l d e r ' s i n e q u a l i t y g ives a finite fc-independent u p p e r b o u n d f o r E\p,\ r/q. I n p a r t i c u l a r , c h o o s e r s u f f i c i e n t l y n e a r t o q t h a t - > 1 : t h e n t h e r e is a n e x p o n e n t ( i n t h e i n t e r v a l 9 9 ( • — • — , - ) w h i c h w i l l n e c e s s a r i l y o b e y — - — < q. T h e r e f o r e s o m e p o w e r 0 > 1 o b e y s 0$ < r/q a n d \q - 1 qj $ - 1 8—-— < q, so w e d e d u c e Ei7./(.,*.,«fc)r < ( E i p ; r ) 1 / f (Ei/(.,2.,« f c)r ( f- i ,) l f" 1 , / f ) is p r e c i s e l y t h e set o f a l l ^ - m e a s u r a b l e a n d [ / - v a l u e d r a n d o m v a r i a b l e s w i t h finite q-th m o m e n t s . I t f o l l o w s t h a t u\ G Ut(ip) f o r a l l t, a n d t h a t t h e m a x i m u m c o n d i t i o n ( 3 . 5 ) is t r u l y g l o b a l . - 115 -Q u a d r a t i c P e n a l i z a t i o n . T h e n e c e s s a r y c o n d i t i o n s o f T h m . 3 .2 d o n o t c h a n g e i f t h e o b j e c t i v e f u n c t i o n a l i n c o r p o r a t e s a q u a d r a t i c p e n a l i z a t i o n t e r m o f a c e r t a i n f o r m . I n d e e d , l e t £ a n d L b e f u n c t i o n s o b e y i n g ( H 4 ) - ( H 5 ) a n d d e f i n i n g t h e f u n c t i o n a l A [ c t , u] : = E (p, a) + £ ( i r ) + J Z(r , x r,u r ) a Y S u p p o s e a p a i r ( 3 , u) i s g i v e n , a n d t h a t t h e u n c o n s t r a i n e d s t o c h a s t i c c o n t r o l p r o b l e m o f t h i s s e c t i o n is c o n s i d e r e d w i t h t h e m o d i f i e d c o s t f u n c t i o n a l ( 3 . 1 1 ) A p[a, u] = A [ c t , u] + p A[a, u] — A [ 3 , u] + p \ct — 3 | 2 f o r s o m e fixed p > 0 . N o w i f ( 3 , u) m i n i m i z e s A p a n d w e d e f i n e p e r t u r b e d p a i r s (a£,u£) as a b o v e , t h e n P r o p . 2 . 1 a p p l i e s t o b o t h A a n d A t o g i v e f u n c t i o n s 6(e), 6(e) ~ 0 f o r w h i c h A p[a £, ue] - A p [ 3 , u] = e E {(0, a) - p'0Aa + AH(s, v.,)] + e6(e) (3-12) I r/~ V ~, ~ 1 ~ I2 ' - r e 2 p E \^P, a) - p0Aa + AH(s, u,)J + 5 ( e ) . H e r e H a n d p a re d e f i n e d as a b o v e , H a n d p are t h e i r o b v i o u s a n a l o g u e s , a n d AH(s,u) := H(s,x„p„u,) - H(a,x„p„u). J u s t as (3 .3 ) f o l l o w s f r o m ( 3 . 2 ) , i t a lso f o l l o w s f r o m ( 3 . 1 2 ) , a n d t h e p r o o f o f T h m . 3.2 t h e n p r o c e e d s w i t h o u t c h a n g e . T h i s f a c t w i l l be u s e f u l l a t e r . 3.3 T h e o r e m . The statement of Theorem 3.2 remains true if w e consider any objective functional of the form (3.11) in which A obeys (H4)-(H5). 3.4 T h e F u n d a m e n t a l M a t r i x . I n t h e c o n v e r g e n c e a n a l y s i s t o f o l l o w i n S e c t i o n 4 , t h e s t r u c t u r e o f t h e a d j o i n t p r o c e s s p t m u s t be c l e a r l y u n d e r s t o o d . M o s t o f t h e p r o p e r t i e s o f pt f o l l o w f r o m t h o s e o f t h e f u n d a m e n t a l m a t r i x $(t,r) g i v e n b y ( 1 . 1 0 ) , w h i c h w e s u m m a r i z e h e r e . L e t us i n t r o d u c e $ t ( = $ ( * > 0 ) ) a n d as t h e p a t h w i s e u n i q u e c o n t i n u o u s n X n m a t r i x p rocesses s o l v i n g ( 3 . 1 3 a ) d$t = fx(t)®tdt + dx(t)$tdu>u * o = A ( 3 . 1 3 6 ) d9t = - * t [fx(t) + f f « ( t ) f f , ( * ) ] dt - Vtdx{t)dwt, •*<> = /. - 116 -H e r e w e h a v e u s e d t h e f o l l o w i n g n o t a t i o n : d Sx(t)$t dt«t = 53a*(t)*t dwk, k=l it=i d Vtax (*) dwt = Y^Vt2x (*) dw k. k=l W i t h t h e s e d e f i n i t i o n s , e q u a t i o n s ( 3 . 1 3 ) i m p l y t h a t d(^t^t) = 0 , s o t h a t * t $ t = I V t a.s. C o n s e q u e n t l y = V t a.s. ; s i nce t h e s o l u t i o n $ o f ( 1 . 1 0 ) is p a t h w i s e u n i q u e , i t m u s t o b e y ( 3 . 1 4 ) = $ t * r = S t ^ T 1 V t , r a .s . B o t h $ t a n d = so lve l i n e a r S D E ' s w i t h b o u n d e d c o e f f i c i e n t s . H e n c e f o r e v e r y p > 2 , t h e p r o o f o f L e m m a I V . 4 . 1 y i e l d s a c o n s t a n t Cp s u c h t h a t ( 3 . 1 5 ) E ( s u p | $ t |p + s u p |*rXr J < 0 s u c h t h a t f o r a l l (t, x, u ) e [0 , T ] x R n x U, \f{t,x,u)\ + W{t,x)\ < ki[l+\x\), fx{t,x,u) + \o-x(t, x) \ < fci. ( h 3 ) T h e r e a re g i v e n r a n d o m v e c t o r s ~X0 € R n a n d A € R n X a s u c h t h a t E \X0\q < +00 a n d |-A| ^ fc2- F o r a n y u £ U w i t h c o r r e s p o n d i n g p r o b a b i l i t y space ((1,7, Tt,P), t h e i n i t i a l d i s t r i b u t i o n o f xt o b e y s x0(ot)=X0 + Act. - 119 -( h 4 ) L e t L = ( £ _ , Z , 0 ) 6 R ; x R . A s s u m e t h a t f o r e a c h (t, u) € [0, T ] X U, b o t h L(t, •, u) a n d £(•) a re d i f f e r e n t i a t e ; t h a t L, Lx, I, tx a re j o i n t l y c o n t i n u o u s i n a l l t h e i r a r g u m e n t s ; a n d t h a t t h e c o n t i n u i t y o f L[t,x, •) a n d Lx(t,x, •) is u n i f o r m i n (t, x). ( h 5 ) T h e r e is a n e x p o n e n t q G [ l , g — 1) a n d a c o n s t a n t A;3 > 0 s u c h t h a t f o r a l l (t,x, u) £ [0, T ] X R n x U, o n e h a s L(t,x,u)\ < fc3(l+ | x | ? ) , |2»(*,x,u)| < fcsJl+N*-1), | £ ( x ) | < fc3(l + lac]*), < fc3(l + M ' "1 ) -( h 6 ) F o r e a c h ( t , x ) , w i t h t h e p o s s i b l e e x c e p t i o n o f a set w h o s e p r o j e c t i o n o n t o t h e t - a x i s h a s L e b e s g u e m e a s u r e z e r o , t h e f o l l o w i n g ( c o m p a c t ) set is c o n v e x : | ^ / ( t , x , « ) , L{t,x,u), fx(t,x,u), I x (t ,x,u)j : u e f / j . H y p o t h e s i s ( h 6 ) is m o r e r e s t r i c t i v e t h a n t h e c o r r e s p o n d i n g c o n v e x i t y c o n d i t i o n ( h 6 ) i n C h a p . I V . T e c h n i c a l l y , t h i s e x t r a c o n v e x i t y a s s u m p t i o n o n fx a n d Lx is r e q u i r e d f o r c o n v e r g e n c e o f t h e a d j o i n t p rocesses w h e n w e t a k e l i m i t s as p a r t o f t h e p r o x i m a l n o r m a l a n a l y s i s t o f o l l o w . P r a c t i c a l l y , h o w e v e r , ( h 6 ) d o e s n o t s i g n i f i c a n t l y w e a k e n t h e t h e o r y t o b e d e v e l o p e d b e l o w b e c a u s e i t is a u t o m a t i c a l l y s a t i s f i e d b y a n y p r o b l e m w h i c h is " s u f f i c i e n t l y r e l a x e d . " See C l a r k e ( 1 9 8 3 ) , S e c t i o n 5 .5 , a n d W a r g a ( 1 9 7 2 ) . 4.1 Lemma. The value function is lower semicontinuous near 0. Proof. C h o o s e a n y ( a , A) n e a r 0 , a n d l e t { ( a f c , A f c ) } b e a n y s e q u e n c e w i t h l i m i t ( a , A ) . W i t h o u t loss o f g e n e r a l i t y , w e m a y pass t o a s u b s e q u e n c e a n d a s s u m e t h a t V(ak,Xk) - + v = l i m i n f V ( a f c , A f c ) . W e m u s t s h o w V(a, A) < v. T h i s is e v i d e n t i f v = +oo, so a s s u m e v < +oo. I n t h i s case, T h m . I V . 7 . 1 ( a ) a p p l i e s a t e a c h fc t o g i v e a c o n t r o l uk € U a n d a v e c t o r pk € w i t h n o n n e g a t i v e c o m p o n e n t s s u c h t h a t (-pk - A * , V(ak, A f c ) , - A + ) = A [ a f c , u f c ] . S i n c e pk is b o u n d e d b y ( h 5 ) , t h i s s e q u e n c e h a s a c o n v e r g e n t s u b s e q u e n c e , a l o n g w h i c h T h m . I V . 7 . 1 ( b ) g ives a c o n t r o l uEU e x p r e s s i n g t h e l i m i t i n t h e f o r m (-P- X-,v,-X+) = A[Q,U]. T h i s s h o w s t h a t V(a, A) < v, as r e q u i r e d . / / / / - 120 -P e r p e n d i c u l a r s . S u p p o s e t h a t s o m e v e c t o r (P, 0 i n R 7 ( t h i s m e a n s p h a s n o n e g a t i v e c o m p o n e n t v a l u e s ) a n d t > 0 , t h e c o n t r o l u g ives r i se t o t h e f o l l o w i n g p o i n t : ^ a , ( - A _ [ a , u ] -p,-A+[a,u]), A 0 [ o ! , u ] + t ) . T h i s p o i n t l ies i n e p i V b y i n s p e c t i o n . T h u s P r o p . 11.3.5 i m p l i e s U3, - ^ 1 . . 2 ^ 2 ( a ' - A - ~ ^ > - A + > A o + t ) - ( S . A - , A + , v ) . I f w e c h o o s e a = 3 a n d u = u i n ( 4 . 1 ) , t h e c o n s t r a i n t s o f p r o b l e m P ( 3 , A) a re s a t i s f i e d a n d s u b s t a n t i a l c a n c e l l a t i o n o c c u r s . W e o b t a i n (4 .2 ) 0 < 0 a n d t > 0. S i n c e e q u a l i t y h o l d s i n (4 .2 ) w h e n (t,p) = ( « - V ( 3 , A ) , - A _ - A _ ) > 0 , t h e R H S o f ( 4 . 2 ) m u s t h a v e n o n n e g a t i v e r i g h t d e r i v a t i v e s i n (t, p) a t t h i s l o c a l m i n i m u m p o i n t . T h e r e f o r e 0 a n d 0 . I n f a c t i f a n y c o m p o n e n t o f t h e m i n i m i z i n g (t, p) is s t r i c t l y p o s i t i v e t h e n t h e d e r i v a t i v e m u s t a c t u a l l y v a n i s h t h e r e . T h u s w e ge t t h e c o m p l e m e n t a r y s lackness c o n d i t i o n s (4 .3 ) y > 0 > 0 , V o ( V ( 3 , A) —v)=0, ( 4 . 4 ) P->0, ( p - , A - + A _ ^ = 0. L i n e ( 4 . 3 ) s i m p l y r e s t a t e s t h e g e o m e t r i c a l l y o b v i o u s f a c t s t h a t a p e r p e n d i c u l a r t o a n e p i g r a p h c a n n o t b e d i r e c t e d u p w a r d , a n d t h a t i f i t is b a s e d o n a v e r t i c a l s ide o f t h e e p i g r a p h t h e n i t c a n n o t b e d i r e c t e d d o w n w a r d e i t h e r . L i n e (4 .4 ) is t h e p r e c u r s o r o f t h e u s u a l c o m p l e m e n t a r y s l a c k n e s s c o n d i t i o n o n t h e m u l t i p l i e r s o f t h e c o n s t r a i n e d S t o c h a s t i c M a x i m u m P r i n c i p l e . - 1 2 1 -L e t u s n o w ux t = v — V(a, X) a n d p = — (A_ + A _ ) i n ( 4 . 1 ) , a n d o b s e r v e t h a t t h e i n e q u a l i t y b e c o m e s (4.5) - (/3|3) + 'A[a, u] + i | (a - 2 , A [a, u] - A) |2 . T h i s i n e q u a l i t y is v a l i d f o r a l l a £ R a , u € U, a n d e q u a l i t y h o l d s w h e n (ct,v) = ( 2 , u ) . H e n c e t h i s c h o i c e r e p r e s e n t s t h e s o l u t i o n t o a n u n c o n s t r a i n e d s t o c h a s t i c c o n t r o l p r o b l e m . T h i s p r o b l e m h a s a w e a k f o r m u l a t i o n , s i n c e e v e r y e l e m e n t o f U c a r r i e s i t s o w n p r o b a b i l i t y s t r u c t u r e . B u t i f w e r e g a r d t h e p r o b a b i l i s t i c f r a m e w o r k s p e c i f i e d b y u as f i x e d , t h e n u so lves t h e s t r o n g p r o b l e m w i t h n o n a n t i c i p a t i v e c o n t r o l s i n t h a t s e t t i n g . T h e r e f o r e T h m . 3.3 p r o v i d e s n e c e s s a r y c o n d i t i o n s . R e c a l l f r o m ( 3 . 6 ) t h e n o t a t i o n Ut[i>) = | u : f i —• U : u is 7% — m e a s u r a b l e j ; w e w i l l c h o o s e t/> l a t e r . 4 . 2 P r o p o s i t i o n . L e t (fi, -'Poll Then there is a solution u € U to P ( 2 , A) for which

) C [0, T] and a probability structure (Cl,7,7t,P), f t , s u c h t h a t t h e R n - v a l u e d process (4.6) p't = -p' obeys M*r)$r + y Lx(r)$Tdr •r 1 (4.7) E A ' p 0 = -fi, (4.8) EH{t,xt,puut,£)>VH{t,xupt,u,). Here the pre-Hamiltonian H is defined by (4.9) H{t, x, p, u , o) is o b t a i n e d v i a (4.10) ( f t & p . - f t , ) = l i m . ^ ' ^ ' " i i l = ^ 0k> 0 . L e t t h e p r o b a b i l i t y s t r u c t u r e a s s o c i a t e d w i t h uk b e l a b e l l e d ( f i f c , Jk, 7k, Pk) a n d wk. N o w s ince A [ o : f c , u f c ] is a b o u n d e d s e q u e n c e i n R / + 1 + J ( t h i s f o l l o w s f r o m L e m m a I V . 4 . 1 a n d t h e g r o w t h c o n d i t i o n s ( h 5 ) ) , w e m a y a s s u m e t h a t — A * — pk —* —p b y p a s s i n g t o a s u i t a b l e s u b s e q u e n c e . A l s o , V ( a f c , A f c ) < vk w h i l e vk —• V(0,0): b y L e m m a 4 . 1 , i t f o l l o w s t h a t V ( a f c , A f c ) — * V ( 0 , 0 ) . I n s h o r t , w e m a y a s s u m e t h a t A [ a f c , u f c ] — • (—p, V ( 0 , 0 ) , 0 ) b y p a s s i n g t o a s u b s e q u e n c e . N o w as T h m . I V . 7 . 1 s h o w s , t h e r e is a f u r t h e r s u b s e q u e n c e ( w h i c h w e d o n o t r e l a b e l ) a l o n g w h i c h xk c o n v e r g e s i n d i s t r i b u t i o n t o a p r o c e s s x w h i c h i n t u r n c a n b e r e a l i z e d b y a c o n t r o l s o l v i n g P ( 0 , 0 ) . F o r each k, t h e c o n c l u s i o n s o f P r o p . 4 .2 h o l d f o r (Bk, tp1^, — (pk) a n d s o m e p r o c e s s pk. B u t b e f o r e a n y a s s e r t i o n s c o n c e r n i n g t h e c o n v e r g e n c e o f pk c a n b e a d v a n c e d , w e m u s t r e c a l l t h a t t h e m e t h o d s o f C h a p . I V u s e d t o o b t a i n xk-^-*x s t u d i o u s l y a v o i d e d a n y a s s e r t i o n t h a t t h e c o n t r o l s uk c o n v e r g e d i n a n y w a y a t a l l . I n v i e w o f t h i s , an e l e m e n t a r y a p p r o a c h t o c o n v e r g e n c e i n , say, (4 .6 ) is o u t o f t h e q u e s t i o n . W e m u s t r e t u r n t o t h e m e t h o d s o f C h a p . I V t o s h o w t h a t t h e c o n t r o l r e a l i z i n g x c a n a c t u a l l y b e c h o s e n t o f a c i l i t a t e c o n v e r g e n c e o f t h e m u l t i p l i e r s . E v e n t h i s is less s t r a i g h t f o r w a r d t h a n i t m i g h t a p p e a r . T h e d i f f i c u l t y c e n t r e s o n t h e i ssue o f ^ - a d a p t e d n e s s , w h i c h is c r i t i c a l t o t h e use o f B e n e g ' s m e a s u r a b l e s e l e c t i o n t h e o r e m as i n C h a p . I V . T o r e s o l v e i t , w e m u s t m a k e e x p l i c i t use t h e d e f i n i t i o n o f pt i n t e r m s o f t h e f u n d a m e n t a l m a t r i x E a c h $ f c is ^ - a d a p t e d , a n d t h e e s t i m a t e s o f p a r a g r a p h 3.4 w i l l a l l o w us t o s h o w t h a t t h e s e m a t r i x p rocesses c o n v e r g e i n t h e a p p r o p r i a t e sense a n d t h e n d e d u c e t h e c o n v e r g e n c e o f pk f r o m d e f i n i t i o n ( 4 . 6 ) . T h i s p r o g r a m m e f o l l o w s t h e c o n c e p t u a l l i nes o f P r o p s . I V . 5 . 2 - I V . 5 . 7 , b u t is t e c h n i c a l l y m o r e d i f f i c u l t b e c a u s e so m a n y m o r e p rocesses a re r e q u i r e d t o c o n v e r g e s i m u l t a n e o u s l y . - 123 -W e w i l l use t h e f o l l o w i n g n o t a t i o n , b a s e d o n t h e c o n s t i t u e n t s o f t h e d y n a m i c s , t h e o b j e c t i v e f u n c t i o n a l s , t h e a d j o i n t p r o c e s s , a n d t h e p r e - H a m i l t o n i a n . *1,fc(«) = f f(r,xk,uk)dr, Jo *2'k(t)= f' cr(r,xk)dwk, Jo zk(t)= f L(r,xk,uk)dr, Jo P1'* = M*r)*r. p2>k(t)= f Lx(r,xk,ukr)$kdr, Jo h l>k(t)= f f(r,xkriuk)dr, Jo h2'k(t)= ffSWy'f^x^^dr, Jo « 1 ' * ( t ) = f / * ( » - , x f c , u f c ) $ f c d r , Jo * > 2 f c ( t ) = f *x(r,xk)*kdwr, Jo «tf = ( * ? ) _ 1 . 4 . 3 P r o p o s i t i o n . T h e following sequence of 15-tuples is tight as a s e q u e n c e o f r a n d o m vectors in the space R n x C 3 n x CI+1+J x C 3 n x C x C x C 4 < n X n ) x C d : ( * * ( * * ) , x f c ( ) , x 1 ^ ) , x 2 ' f c ( ) , * * ( • ) , p f c ( ) , p 1 - * , p 2 - f c ( ) , fc1-^-), h2>k(), *>fc, * 2 - \ * f c , «,*(•)). Hence it has a subsequence converging in dbtribution to a 15-tuple ( x 0 ( 0 ) , * ( • ) , x ^ ) , x2 ( ) , * ( • ) , P ( ) , p \ P 2 ( ) , h l(), h2(), * . \ $2, «,(•)). Proof. F o r t u n a t e l y , t i g h t n e s s c a n b e p r o v e n c o m p o n e n t b y c o m p o n e n t . R e c a l l a l so t h a t a n y s e q u e n c e w h i c h c o n v e r g e s i n d i s t r i b u t i o n is a u t o m a t i c a l l y t i g h t b y P r o k h o r o v ' s t h e o r e m . T h u s xk(ctk) is t i g h t b e c a u s e i t c o n v e r g e s i n d i s t r i b u t i o n t o t h e r a n d o m v e c t o r XQ o f ( h 3 ) , a n d u>k(-) is t i g h t b e c a u s e t h i s s e q u e n c e o f B r o w n i a n m o t i o n p rocesses is a c t u a l l y c o n s t a n t i n d i s t r i b u t i o n . ( M o r e o v e r t h e l i m i t i n g p r o c e s s tu ( - ) m u s t t h e r e f o r e b e a B r o w n i a n m o t i o n , b u t we w i l l e x p l o r e t h i s l a t e r . ) P r o p . I V . 5 . 2 p r o v e s t h a t (xk(-),x1'k(-),x2,k(-)) a re t i g h t i n C 3 n . I n f a c t , t h e p r o o f g i v e n t h e r e t h a t x 1 ' * w a s t i g h t a p p l i e s e q u a l l y w e l l t o p2'k, h1,k, h2'k, a n d $ 1 , f c . ( I t e v e n s i m p l i f i e s a l i t t l e b i t i n t h e p r e s e n t s e t t i n g . ) L i k e w i s e , t h e t r e a t m e n t o f x 2 ' f c g i v e n i n P r o p . I V . 5 . 2 a p p l i e s a l so t o $ 2 ' f c . T h e - 124 -r e l a t i o n $ f c = / + $ 1 , f e . + $ 2 ' f c t h e n i m p l i e s t h a t $ f c is t i g h t a l so . T h e t i g h t n e s s o f z f c ( ) i s p r o v e n i n T h m . I V . 7 . 1 . T o p r o v e t h a t p 1 ' * is t i g h t i n RN, i t su f f i ces t o s h o w t h a t E j p 1 ' * ) is u n i f o r m l y b o u n d e d . A n d f o r t h i s , o n e s i m p l y uses ( h 5 ) t o w r i t e EIP1-*! < E|4(4)I | | * * | | < EJk3 (1 + ||*T_1) ||*FC|. T h e R H S is u n i f o r m l y b o u n d e d i n A; b y H o l d e r ' s i n e q u a l i t y , s i nce | | $ * | | i s u n i f o r m l y b o u n d e d i n a n y V, p > 1 , a n d || is u n i f o r m l y b o u n d e d i n a n y Lr, r € [ l , ? ) . T h e p r o o f o f t i g h t n e s s f o r tyk = ( $ f c ) 1 i s l e f t t o t h e r e a d e r , w i t h t h e f o l l o w i n g h i n t : t h e f u n c t i o n tyk sa t i s f i es t h e l i n e a r S D E ( 3 . 1 3 b ) w i t h b o u n d e d c o e f f i c i e n t s . W r i t i n g t h i s e q u a t i o n i n i n t e g r a l f o r m a n d p r o v i n g t h e t i g h t n e s s o f e a c h t e r m o n t h e r e s u l t i n g R H S s e p a r a t e l y , j u s t as w e h a v e d o n e r e p e a t e d l y a b o v e , w i l l s h o w t h a t tyfc is t i g h t a l so . F i n a l l y , t h e t i g h t n e s s o f p 1 ' * , p2'k, a n d Wk, t o g e t h e r w i t h t h e r e l a t i o n pk(t) = - (?k)' [ p 1 ' " + p2'k(T) - f2 x Cd w h o s e 1 6 - t u p l e s o b e y t h e f o l l o w i n g r e l a t i o n s h i p s . F i x r £ (<7,?). - 125 -x(t) = x0 + x1 (t) + x2(t) V t x ^ O ) = x2 ( 0 ) = 0 , p(t) = -(r)| 2 drl) er(t,x(t),p2(t),*t,^r1) a.e. [o,r]. N e x t w e m u s t p r o d u c e a n a p p r o p r i a t e c o n t r o l u ( t , w ) . T h i s r e q u i r e s t h a t t h e p r o b a b i l i t y space ( f2 , 7, P) o n w h i c h t h e l i m i t i n g 1 5 - t u p l e o f P r o p . 4 .3 is d e f i n e d b e e q u i p p e d w i t h a filtration. A s u f f i c i e n t l y l a r g e filtration m a y b e d e f i n e d i n t e r m s o f t h e c o n t i n u o u s p r o c e s s V>t = {X0, A,xt>xl,x2,zt)pt,p2,hl,h2,t,$l,2,vjt) . W e t a k e f o r 7t t h e filtration g e n e r a t e d b y rpt. ( T h e c o n s t a n t p rocesses i n t h e first t w o c o m p o n e n t s o f rp e n s u r e t h a t X o a n d A a re ^ - a d a p t e d . ) E v i d e n t l y ipt is t h e l i m i t i n d i s t r i b u t i o n ( o n t h e space o f c o n t i n u o u s f u n c t i o n s ) o f t h e c o n t i n u o u s p rocesses = ( x 0 ^ i ^ x ^ x ^ ^ x ^ ^ ^ ^ p ^ p ^ ^ h ^ ^ h ^ ^ $ ^ $ ^ ^ $ 2 • ^ ^ ) . W e w i l l use t h e s e p rocesses w h e n a p p l y i n g P r o p . 4 .2 f o r e a c h fc. - 127 -4 . 5 P r o p o s i t i o n . T h e limiting process w(-) in Prop. 4.3 i s a n ft-Brownian motion in R d . Moreover, there is a n ^ - a d a p t e d process u : [Q,T] X ft —+ U s u c h t h a t x t = x 0 ( 0 ) + / f{r,xr,ur)dr + < j ( r , x r ) d u > r V t , • 0 T pt = -(p" IX{XT)®T + J^ Lx(r,xr,ur)$Tdr $ t - 1 V t , $t = I+ [ fx(r,xr,ur)$rdr+ [ cx(r, xr)$r dwr V t , J o J o z t = / L ( r , x r , u r ) d r V t , Jo hi = f 9- lf{r,xr,Ur)dr V t , Jo h 2 = f p2^f(r,xT,vir)dr V t . Jo Proof. B e i n g t h e l i m i t i n d i s t r i b u t i o n o f B r o w n i a n m o t i o n p r o c e s s e s , «/(•) is c e r t a i n l y a B r o w n i a n m o t i o n . I t is a d a p t e d t o 7t b y c o n s t r u c t i o n . W e s h o w t h a t i t is a c t u a l l y a n 5 t - B r o w n i a n m o t i o n i n t h e c o u r s e o f j u s t i f y i n g t h e s i x i n t e g r a l r e p r e s e n t a t i o n s l i s t e d a b o v e . F i r s t l e t us t r e a t t h e s t o c h a s t i c i n t e g r a l s . S i n c e e a c h t r i p l e {wk, x 2 , f c , $ 2 > f c ) i s a c o n t i n u o u s s e c o n d - o r d e r ^ ( ^ f c ) - m a r t i n g a l e o n (Qk,fk,Pk) a n d s ince i(>k-^-*ip, P r o p . I V . 3 . 1 i m p l i e s t h a t t h e l i m i t p r o c e s s (w,x,$2) is a c o n t i n u o u s s e c o n d - o r d e r ^ - m a r t i n g a l e . T h e q u a d r a t i c v a r i a t i o n o f t h e l i m i t i n g m a r t i n g a l e is a l i t t l e a w k w a r d t o w r i t e d o w n , b e c a u s e $ d e n o t e s a n n X n m a t r i x . L e t us 2 t e m p o r a r i l y t h i n k o f x a n d $ f c - ^ - t - $ , t h e c o n t i n u i t y p r o p e r t i e s o f c, E , a n d o f i n t e g r a l f u n c t i o n a l s i m p l y t h a t - 128 -t h e p rocesses ( v f c ) t c o n v e r g e i n d i s t r i b u t i o n t o t h e p r o c e s s , f o r w h i c h w e use t h e s u g g e s t i v e n o t a t i o n (v)t, g i v e n b y <«>t = f Jo a ocr' c r E ' E E C T ' E E ' dr. T h u s ( « * ) ( « * ) ' - {vk)-^vv' - (v); s i nce t h e l e f t s ide is a u n i f o r m l y i n t e g r a b l e s e q u e n c e o f ? J (V> f e ) - m a r t i n g a l e s , t h e r i g h t s ide is a n ^ - m a r t i n g a l e b y P r o p . I V . 3 . 1 . B y d e f i n i t i o n , i t f o l l o w s t h a t t h e p r o c e s s w e h a v e d e n o t e d b y (v)t r e a l l y is t h e q u a d r a t i c v a r i a t i o n o f « f , a n d t h a t i t is r e a l i z e d h e r e as t h e i n t e g r a l o f t h e f o l l o w i n g m a t r i x t i m e s i t s t r a n s p o s e : J a{r,xr) E(r,x r,* r). T h i s (d + n + n 2 ) X d m a t r i x h a s ( r o w ) r a n k i d e n t i c a l l y e q u a l t o d , so b y P r o p . I V . 3 . 3 , t h e space ( f l , 7, Tt,P) i t s e l f c a r r i e s a n ^ - B r o w n i a n m o t i o n wt w i t h v a l u e s i n R d s u c h t h a t " w{t) ' I x 2 ( t ) = «(*) = / a{r,xr) dwr .* 20. Jo E ( r , x r , $ r ) T h e first d c o m p o n e n t s o f t h i s e q u a t i o n i m p l y t h a t w% = Wt f o r a l l t, e x c e p t o n a n e g l i g i b l e se t o f n 2 w - v a l u e s , so t h e s e p rocesses are i n d i s t i n g u i s h a b l e . T h e n , r e v e r t i n g f r o m v e c t o r s i n R t o m a t r i c e s i n R n X n , t h e s e c o n d t w o b l o c k s o f t h i s i d e n t i t y y i e l d t h e d e s i r e d s t o c h a s t i c i n t e g r a l r e p r e s e n t a t i o n s : x 2 ( t ) = / a(r, xr)dwr, Jo $ 2 ( t ) = / ox(r,xr)$rdwr. Jo T h e s e c o n d s t e p is t o p r o d u c e a s u i t a b l e c o n t r o l u ( t , w ) . A s i n C h a p . I V , t h i s c a n b e d o n e b y a p p e a l i n g t o t h e s e l e c t i o n l e m m a g i v e n b y BeneS ( 1 9 7 1 ) . W e c o n s i d e r t h e m e a s u r e space M = [0 , T] X f l t o g e t h e r w i t h t h e cr - f ie ld M w i t h r e s p e c t t o w h i c h . M - m e a s u r a b i l i t y is e q u i v a l e n t t o V> t - a d a p t e d n e s s . (See t e x t f o l l o w i n g L e m m a I V . 4 . 2 . ) W e t a k e R = R n x R ' + 1 + J x R n x R x R x R n X n , a n d d e f i n e k: M X U —* R a n d y : M —• U b y k{t,u,u) = ( / ( t , x ( i , w ) , u ) , L(t,x(t,u),u),Lx(t,x(t,u),u)$(t,u), *(«. w)_1/(«, * ( * , w ) , u ) , p 2 ( i , a / ) $ ( t , w)- lf(t, x{t,w), u ) , /,(t, x(t,oj), «)*(«, «)), y(t,u) = (x1{t,u)),z{t,w),f(t,w)/h^t,uj)/h2{t,w),$1{t,u>)y T h e n k(t,w,u) is . M - m e a s u r a b l e i n ( t ,a>) f o r e a c h fixed u e 17, a n d c o n t i n u o u s i n u f o r e a c h f i x e d (t,oj). A l s o y is . M - m e a s u r a b l e ( b e c a u s e , f o r e x a m p l e , xt = l i m — — is x t - a d a p t e d ) a n d - 129 o b e y s y[t,w) £ k{t,w,U) = T(t, xt,p 2,$t,$t T h e c o n c l u s i o n o f B e n e g ( 1 9 7 1 ) , L e m m a 5, p . 4 6 0 is t h a t t h e r e is a n .M-measurable m a p p i n g u : [0, T] X fl —» U s u c h t h a t y(t,w) e q u a l s ( / ( t , x ( t , w ) , u ( i , w ) ) , L{t,x(t,u),u(t,u)), Lx{t,x{t,u),u{t,u})){t,u)), ^ ( t . w J - V ^ x ^ ^ ^ ^ w J J . p ^ t . w J ^ ^ w J - V ^ i U . w J . u ^ a ; ) ) , fx{t,x[t,w),u{t,w))*(t,uj). R e c a l l i n g t h e d e f i n i t i o n o f y ( t , w ) a n d t h e r e p r e s e n t a t i o n s o f x2 a n d $ 2 a l r e a d y g i v e n , t h e i n t e g r a l r e p r e s e n t a t i o n s o f t h i s p r o p o s i t i o n n o w f o l l o w f r o m P r o p . 4 . 4 . / / / / L e t u s s u m m a r i z e o u r findings a n d c o m p l e t e t h e s t u d y o f c o n v e r g e n c e . 4 . 6 Theorem. L e t (/9, EH{t,xt,pt,v,0), ( 4 . 1 7 ) E i ' p 0 = -0. Here the pre-Hamiltonian H is deSned by p' f — tp'L as in (4.9), and the process pt is given explicitly by ( 4 . 1 8 ) p't = - $ 4(x T)*r + y Lx{r,xT,uT)
0, together with the complementary slackness condition ( 4 . 1 9 ) £ _ > 0 , < £ _ , A _ | 0 , u ] ) = 0 . Proof. W e h a v e a l r e a d y c o n s t u c t e d t h e l i m i t p r o c e s s x a n d i t s c o r r e s p o n d i n g c o n t r o l u , a n d v e r i f i e d t h a t p t a n d $ t h a v e t h e c o r r e c t r e p r e s e n t a t i o n s . T o see w h y u so lves P ( 0 , 0 ) , r e c a l l t h a t A [ af c , u f c ] c o n v e r g e s t o (—p, V ( 0 , 0 ) , 0 ) b y a s s u m p t i o n f o r s o m e p > 0 i n R / , a n d t h a t b o t h zk(T)-^-*z(T) - 130 -a n d xk(T)-^x(T). S i n c e A [ a f c , u f c ] = E [ £ ( x j . ) + z f c ( T ) ] , a s t a n d a r d u n i f o r m i n t e g r a b i l i t y a r g u m e n t i m p l i e s t h a t E [ £ ( x T ) + z(T)] = A [ 0 , u ] = ( - p , V ( 0 , 0 ) , 0 ) . T h i s s h o w s t h a t u sa t i s f i es a l l t h e c o n s t r a i n t s o f P ( 0 , 0 ) a n d a t t a i n s t h e i n f i m u m , as c l a i m e d . S i n c e ( £ > _ , £ > 0 ) i s a l i m i t o f v e c t o r s w i t h n o n n e g a t i v e c o m p o n e n t s , i t s c o m p o n e n t s m u s t a lso b e n o n n e g a t i v e . T o p r o v e t h e c o m p l e m e n t a r y s l a c k n e s s c o n d i t i o n ( 4 . 1 9 ) , n o t e t h a t P r o p . 4 .2 g ives ( p * _ 1 A . [ of c , u f c ] + A * . ) = 0 VJfc. W e k n o w t h a t —+ , a n d c o n s i d e r a m e a s u r a b l e C ' - a d a p t e d m a p p i n g v. [0 , T] X C l —• U w h i c h is c o n t i n u o u s i n i t s s e c o n d a r g u m e n t f o r e v e r y v a l u e o f t h e first. T h e g r o w t h c o n d i t i o n s o f ( h 2 ) a n d ( h 5 ) a l l o w o n e t o use t h e d o m i n a t e d c o n v e r g e n c e t h e o r e m t o s h o w t h a t t h e m a p p i n g o f Cn x Cn x C l x T L I + 1 + J i n t o R d e f i n e d b y (x,p,ip, 0, o n e h a s 0 < - T [ E i y ( r , x rf c , p * , u * , ^ ) - E F ( r , x f c , p f c , t ; ( r , ^ ) , ^ f c ) ] dr. E J a — t T h e r e f o r e t h e a r g u m e n t s a b o v e a l l o w us t o t a k e t h e l i m i t as k — • c o t o ge t 0 < - / [BH{r,xr,pr,ur,lp)-EH{r,xr,pr,v{r,rP),p)} dr. N o w s ince t h e a d p a t e d m a p s v(r, xp) w i t h c o n t i n u o u s ^ - d e p e n d e n c e a re d e n s e i n t h e m e a s u r a b l e a d a p t e d m a p s « ( r , rp) i n t h e sense o f a l m o s t - s u r e c o n v e r g e n c e w i t h r e s p e c t t o dt x dP^,, t h i s l a s t r e l a t i o n s h i p m u s t a c t u a l l y h o l d f o r a l l m e a s u r a b l e a n d V ' t -adapted p rocesses v. T h e set o f a l l s u c h v's m a y b e c a l l e d t h e set o f a l l V ' - feedback c o n t r o l s , o f w h i c h w e h a v e d e s c r i b e d a c o u n t a b l e s u b s e t V(V') i n t h e t e x t p r e c e d i n g T h m . 3 .2 . B y L e m m a 1 . 1 , t h e r e is a n u l l set M(u, yp) s u c h t h a t f o r a n y v S V(V') a n d s £ M(u,ip), o n e h a s 1 f l i m - / \H[r,xr,pr,ur,(p) - H (r, xr, pr, vr, ). *-*o+ e Jt_s B y u n i f o r m i n t e g r a b i l i t y , w e find t h a t E # ( s , x „ p „ u „ £ ) > -EH{s,x„p„v„$) Vs £ M{u,rp), V u g V(V-). T h i s is p r e c i s e l y t h e s i t u a t i o n i n w h i c h t h e p r o o f o f T h m . 3.2 s h o w s t h a t t h e d e n s i t y o f V, (ip) i n U,(ip) i m p l i e s t h e g l o b a l c o n c l u s i o n ( 4 . 1 6 ) . / / / / - 132 -T h e c o n v e r g e n c e a n a l y s i s o f t h i s s e c t i o n is t h e f o u n d a t i o n o f t h e a p p l i e d r e s u l t s t o f o l l o w . B e f o r e t u r n i n g t o t h e s e , h o w e v e r , l e t u s p r e s e n t a s o m e w h a t s t r o n g e r H a m i l t o n i a n i n e q u a l i t y . C o n s i d e r t h e a d a p t e d a d j o i n t p r o c e s s p< = E [ p t | 7t\. W i t h r e s p e c t t o t h i s p r o c e s s , ( 4 . 1 6 ) a n d ( 4 . 1 7 ) m a y b e r e p l a c e d b y ( 4 . 2 1 ) H(t,xt,Puuuip)> H(t,xupuv,). ( 4 . 2 2 ) EA'po = -P E q u a t i o n ( 4 . 2 2 ) f o l l o w s f r o m ( 4 . 1 7 ) b y c o n d i t i o n i n g ; t o see w h y ( 4 . 2 1 ) h o l d s , n o t e t h a t s i nce H is l i n e a r i n p, a n d b o t h / a n d L a re ^ J - a d a p t e d , t h e i n e q u a l i t y i n ( 4 . 2 1 ) is e q u i v a l e n t t o •E[H{t,xt,pt,v.t,£) | 7t] >E[H{t,xt,pt,v,£) \ 7t}. T h u s i f ( 4 . 2 1 ) is f a l s e , t h e r e m u s t b e a t i m e t ^ M(u), a r a n d o m v a r i a b l e u e Ut, a n d a set F € Tt w i t h P{F) > 0 f o r w h i c h t h e r e v e r s e i n e q u a l i t y p r e v a i l s . B u t i n t h i s case t h e n e w r a n d o m v a r i a b l e v{u) = ut{to)I{u £ F} + u ( w ) J { w 6 F} is a n e l e m e n t o f Ut{i>) f o r w h i c h c o n d i t i o n i n g o n 7t g i ves EH{t, xt,pt,ut,p) < EH{t,xt,pt,v,lp). T h u s t h e f a l s i t y o f ( 4 . 2 1 ) c o n t r a d i c t s t h e p r o v e n s t a t e m e n t ( 4 . 1 6 ) . C o n c l u s i o n ( 4 . 2 1 ) c a n n o t b e fa l se . 4 . 7 D e f i n i t i o n ( M u l t i p l i e r S e t s ) . L e t a filtered p r o b a b i l i t y space ( f2 , 7, 7t,P) s a t i s f y i n g t h e u s u a l h y p o t h e s e s b e g i v e n . S u p p o s e i t s u p p o r t s a n ? t - B r o w n i a n m o t i o n u>t w i t h v a l u e s i n TLd a n d a n ^ t - a d a p t e d p r o c e s s u: [ 0 , T ] x Q —+ U w h i c h so lves P ( 0 , 0 ) . T h e n a p a i r ( R n is c a l l e d a n index H(t,xt,pt,v,ip) a.s. V t £ AV(u) V u € Ut, p't = -vs 'E ^ ( * r ) * r * t - 1 + jT Lx{r, xr, ur)^r^ dr *« = /+/ fx(r,xr,ur)$Tdr+ ax ( r , xr) $r dwr, Jo Jo - 133 -0, 0, ( v ? _ , A _ [ 0 , u ] > = 0 . H e r e Ut d e n o t e s t h e se t o f a l l ^ - m e a s u r a b l e r a n d o m v a r i a b l e s v. Cl —+ U. T h e se t o f a l l s u c h p a i r s ( 0 , S (= A[M*{Y)] n dV(0)} A r ° ° = { ( ? ) 0 ) : ? G A[M°{Y)] n 3 ° ° V ( 0 ) } . F o r a n y u n i t v e c t o r {P,,p) l ies i n MLFI0(Y). I n d e e d , i f 0 t h e n t h e p o s i t i v e h o m o g e n e i t y o f t h e c o n d i t i o n s d e f i n i n g a m u l t i p l i e r i m p l i e s t h a t {fi/ o ) l ies i n M1(Y). B u t T h m . 4 .6 a lso s h o w s t h a t A ^ £ > / £ > o > p / £ > o ) = {Pl'Po, £>T/£?<>) is a v e c t o r f o r w h i c h (jl/fio, fi^/ T , — (po) £ N. O n t h e o t h e r h a n d , i f T , 0 ) G Nepiv(0, V ( 0 ) ) b y t h e p r o x i m a l n o r m a l f o r m u l a . B y d e f i n i t i o n o f d°°V(0), i t f o l l o w s t h a t (0, d°°V(0) = { 0 } => dV{0) ?9=> M1(Y) ^ 0. T h i s ser ies o f i m p l i c a t i o n s c a n b e s u m m a r i z e d as f o l l o w s : (5 .1 ) M1(Y)U[M°(Y)\{0}}^9. L i n e ( 5 . 1 ) is a c o n c i s e s t a t e m e n t o f t h e S t o c h a s t i c M a x i m u m P r i n c i p l e . 5 . 1 T h e o r e m ( S t o c h a s t i c M a x i m u m P r i n c i p l e ) . Assume (hl)-(h6). If problem P ( 0 , 0 ) h a s a feasible control process then it h a s a s o l u t i o n . Moreover, at least one of the optimal controls in the set Y has a multiplier pair as d e f i n e d i n 4.7 for which either is f o r m u l a t e d o n l y i n t e r m s o f t h e s t o c h a s t i c d y n a m i c s a n d t h e s o f t c o n s t r a i n t s : t h e o b j e c t i v e f u n c t i o n a l is i r r e l e v a n t w h e n H(t,xt,pt, v, 0, ->0, ( ^ _ , A _ [ 0 , u ] ) = 0 . H e r e t h e p r e - H a m i l t o n i a n H(t, x, p, u , , p , g ) = (-Ep'0A-q'0, , p , g) = (0,