Science, Irving K. Barber Faculty of (Okanagan)
Computer Science, Mathematics, Physics and Statistics, Department of (Okanagan)
DSpace
UBCO
Wang, Ziyuan
Doctor of Philosophy - PhD
University of British Columbia
In this thesis, we investigate regularities in variational analysis and explore their algorithmic applications. The term "regularities" refers to favorable properties of functions and operators that lead to pleasant behavior of algorithms designed for convex problems in the absence of convexity, including the Kurdyka-Åojasiewicz property and variational convexity. We extend the forward-reflected-backward splitting methodâa convex optimization algorithmâand its variations to nonconvex setting, both with and without a traditional assumption on Lipschitz smoothness. Our analysis hinges on analyzing merit functions associated with nonconvex extensions of the forward-reflected-backward splitting method that decrease along iterates generated by these algorithms. The Kurdyka-Åojasiewicz property guarantees these extensions converge globally to a stationary point of the objective function and more. We investigate systematically the level proximal subdifferential by characterizing its existence, single-valuedness, connection to Lipschitz smoothness, and more. These results reveal the level proximal subdifferential provides deep insights into variational analysis and optimization, which, among other things, lead to novel characterizations of variational convexity and of Lipschitz-type properties of proximal operators in the absence of convexity. We also investigate calculus rules and algorithmic applications of variational convexity. Aided by these results, new insights regarding the local behavior of several celebrated algorithms in nonconvex setting are obtained. Finally, we introduce the left and right Bregman level proximal subdifferentials and investigate them systematically. New correspondences among properties of the Bregman proximal operators, the underlying functions, and the associated subdifferential operators are also established. Asymmetry and duality gap occur in this process as consequences of Bregman distance.
Regularities in Variational Analysisand Their Applications in NonconvexOptimizationbyZiyuan WangB.S., Chongqing Jiaotong Univeristy, 2018M.Sc., The University of British Columbia, 2020A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinIrving K. Barber Faculty of Science(Mathematics)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)May 2025© Ziyuan Wang, 2025The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Regularities in Variational Analysis and Their Applicationsin Nonconvex Optimizationsubmitted by Ziyuan Wang in partial fulfilment of the requirements of thedegree of Doctor of PhilosophyDr. Xianfu Wang, Irving K. Barber Faculty of ScienceSupervisorDr. Heinz Bauschke, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. John Braun, Irving K. Barber Faculty of ScienceSupervisory Committee MemberDr. Jahangir Hossain, School of EngineeringUniversity ExaminerDr. James Burke, Univeristy of WashingtonExternal ExamineriiAbstractIn this thesis, we investigate regularities in variational analysis and exploretheir algorithmic applications. The term âregularitiesâ refers to favorableproperties of functions and operators that lead to pleasant behavior of algo-rithms designed for convex problems in the absence of convexity, includingthe Kurdyka- Lojasiewicz property and variational convexity.We extend the forward-reflected-backward splitting methodâa convexoptimization algorithmâand its variations to nonconvex setting, both withand without a traditional assumption on Lipschitz smoothness. Our analysishinges on analyzing merit functions associated with nonconvex extensions ofthe forward-reflected-backward splitting method that decrease along iteratesgenerated by these algorithms. The Kurdyka- Lojasiewicz property guaran-tees these extensions converge globally to a stationary point of the objectivefunction, as well as provable convergence rates of these algorithms.We investigate systematically the level proximal subdifferential by char-acterizing its existence, single-valuedness, connection to Lipschitz smooth-ness, and more. These results reveal the level proximal subdifferential pro-vides deep insights into variational analysis and optimization, which, amongother things, lead to novel characterizations of variational convexity and ofLipschitz-type properties of proximal operators in the absence of convexity.We also investigate calculus rules and algorithmic applications of variationalconvexity. Aided by these results, new insights regarding the local behaviorof several celebrated algorithms in nonconvex setting are obtained. In par-ticular, variational sufficiency, a special case of variational convexity, guar-antees that proximal gradient method converges to local minimizers ratherthan just stationary points.Finally, we introduce the left and right Bregman level proximal subdif-ferentials and investigate them systematically. New correspondences amongproperties of the Bregman proximal operators, the underlying functions, andthe associated subdifferential operators are also established. Asymmetry andduality gap occur in this process as consequences of Bregman distance.iiiLay SummaryNonconvexity is pervasive in the modern era of big data, yet it remainschallenging to see when and why classic algorithms in convex optimizationwould work in the nonconvex setting, where the well-established theory ofconvex functions and monotone operators collapses.In this thesis, we investigate regularities in variational analysis, whichgeneralize pleasant structures of convex functions and monotone operators,and explore their applications in nonconvex optimization.ivPrefaceThe thesis is based on the following papers [59, 92, 94, 95, 96] and a work inprogress [97]. They are restated below for convenience:1. X. Wang and Z. Wang, Malitsky-Tam forward-reflected-backward split-ting method for nonconvex minimization problems, Computational Op-timization and Applications, 82 (2022), pp. 441â463. This paper is thebasis for Chapter 3.2. X. Wang and Z. Wang, A Bregman inertial forward-reflected-backwardmethod for nonconvex minimization, Journal of Global Optimization,89 (2024), pp. 327â354. This paper is the basis for Chapter 4.3. Z. Wang, A. Themelis, H. Ou, and X. Wang, A mirror inertial forward-reflected-backward splitting: convergence analysis beyond convexity andLipschitz smoothness, Journal of Optimization Theory and Applica-tions, 203 (2024), pp. 1127â1159. This paper is the basis for Chap-ter 5.4. X. Wang and Z. Wang, Every proximal mapping is a resolvent of levelproximal subdifferential, Optimization Letters, 18 (2024), pp. 1237â1252. This paper is the basis for Chapter 65. H. Luo, X. Wang, Z. Wang, and X. Yang, Level proximal subdif-ferential, variational convexity, and pointwise Lipschitz smoothness,arXiv preprint, arXiv:2406.00648, (2024). This preprint is the basisfor Chapters 7 and 8.6. Z. Wang, A. Themelis, and X. Wang, The Bregman level proximalsubdifferential and new characterizations on Bregman proximal oper-ators, work in progress, (2024). This work in progress is the basis forChapter 9.All authors contributed equally to [59,92,94,95].Generative AI was used to assist detecting grammar errors of this dis-sertation.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xGlossary of Notation . . . . . . . . . . . . . . . . . . . . . . . . . xiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 The goals of this thesis . . . . . . . . . . . . . . . . . . . . . . 21.3 The contributions of this thesis . . . . . . . . . . . . . . . . . 3Chapter 2: Preliminaries . . . . . . . . . . . . . . . . . . . . . . 62.1 Functions and operators . . . . . . . . . . . . . . . . . . . . . 62.2 The subdifferential family . . . . . . . . . . . . . . . . . . . . 92.3 Convexity and duality . . . . . . . . . . . . . . . . . . . . . . 132.4 Parametric minimization . . . . . . . . . . . . . . . . . . . . . 182.5 Monotonicity and nonexpansiveness . . . . . . . . . . . . . . 232.6 Various notions of smoothness . . . . . . . . . . . . . . . . . . 242.7 Prox-regular functions . . . . . . . . . . . . . . . . . . . . . . 282.8 The Kurdyka- Lojasiewicz property . . . . . . . . . . . . . . . 29viTABLE OF CONTENTS2.9 The Bregman distance . . . . . . . . . . . . . . . . . . . . . . 34Chapter 3: Forward-reflected-backward splitting for noncon-vex minimization problems . . . . . . . . . . . . . . 383.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.2 A forward-reflected-backward splitting method . . . . . . . . 393.3 Basic properties . . . . . . . . . . . . . . . . . . . . . . . . . . 403.4 Global convergence to stationary points . . . . . . . . . . . . 453.5 Convergence rates . . . . . . . . . . . . . . . . . . . . . . . . 483.6 An application to nonconvex feasibility problems . . . . . . . 533.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57Chapter 4: A Bregman inertial forward-reflected-backwardmethod for nonconvex minimization . . . . . . . . 594.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.2 The BiFRB method and its abstract convergence . . . . . . . 604.3 Stepsize strategies . . . . . . . . . . . . . . . . . . . . . . . . 734.4 Bregman proximal subproblem formulae . . . . . . . . . . . . 784.5 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . 824.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84Chapter 5: A mirror inertial forward-reflected-backward split-ting for minimization problems beyond convexityand Lipschitz smoothness . . . . . . . . . . . . . . . 865.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 865.2 The algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 895.3 Relative smoothness and weak convexity . . . . . . . . . . . . 905.4 Algorithmic analysis toolbox . . . . . . . . . . . . . . . . . . 975.5 Convergence analysis . . . . . . . . . . . . . . . . . . . . . . . 1105.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118Chapter 6: Every proximal operator is the resolvent of a levelproximal subdifferential . . . . . . . . . . . . . . . . 1196.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1196.2 Level proximal subdifferential . . . . . . . . . . . . . . . . . . 1216.3 Fundamental properties . . . . . . . . . . . . . . . . . . . . . 1236.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1276.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134Chapter 7: Finer properties of level proximal subdifferential 135viiTABLE OF CONTENTS7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1357.2 Level proximal subdifferentiability . . . . . . . . . . . . . . . 1357.3 Single-valued level proximal subdifferential . . . . . . . . . . . 1407.4 Optimality conditions . . . . . . . . . . . . . . . . . . . . . . 1437.5 Pointwise Lipschitz smoothness . . . . . . . . . . . . . . . . . 1447.6 Integration on level proximal subdifferential . . . . . . . . . . 1477.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153Chapter 8: Variational convexity, relative monotonicity, andlocal nonexpansiveness . . . . . . . . . . . . . . . . 1548.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1548.2 Variational convexity and variational strong convexity . . . . 1568.3 Relative monotonicity . . . . . . . . . . . . . . . . . . . . . . 1638.4 Characterizing variational convexity . . . . . . . . . . . . . . 1648.5 Characterizing variational strong convexity . . . . . . . . . . 1708.6 Calculus rules of variational convexity . . . . . . . . . . . . . 1718.7 Algorithmic applications . . . . . . . . . . . . . . . . . . . . . 1758.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182Chapter 9: The Bregman level proximal subdifferential . . . . 1839.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1839.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . 1869.3 Bregman proximal map and Moreau envelope . . . . . . . . . 1929.4 The Bregman level-proximal subdifferential . . . . . . . . . . 1999.5 Pointwise characterizations . . . . . . . . . . . . . . . . . . . 2069.6 Global characterizations . . . . . . . . . . . . . . . . . . . . . 2129.7 A perspective from generalized conjugacy . . . . . . . . . . . 2269.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229Chapter 10:Conclusion and future work . . . . . . . . . . . . . 23110.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23110.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233viiiList of TablesTable 3.1 Comparing FRB to DR, iTseng, and DRh. . . . . . . . 57Table 4.1 Comparing BiFRB and iFRB to FRB, DR and iTseng. 84ixList of FiguresFigure 6.1 An illustration of Example 6.2. . . . . . . . . . . . . . 120Figure 6.2 Resolvents in Example 6.12. . . . . . . . . . . . . . . 128Figure 6.3 Resolvents in Example 6.13. . . . . . . . . . . . . . . 129Figure 6.4 Resolvents in Example 6.14. . . . . . . . . . . . . . . 130Figure 7.1 An illustration of Example 7.23. . . . . . . . . . . . . 153Figure 8.1 A correspondence between convex and variationallyconvex settings. . . . . . . . . . . . . . . . . . . . . . 169Figure 9.1 A correspondence between various notions in Euclideanand Bregman settings in Theorem 9.48. . . . . . . . . 185Figure 9.2 A correspondence among various notions in Euclideanand Bregman settings established in Section 9.6.3. . . 185Figure 9.3 An illustration of Example 9.38. . . . . . . . . . . . . 209Figure 9.4 An illustration of Example 9.55. . . . . . . . . . . . . 222xGlossary of Notationcl f The closure of a function f 8conv f The closed convex hull of a function f 15conv f The convex hull of a function f 15eλf The Moreau envelope of a function f with pa-rameter Î»20ââenvÏÎ»f The left Bregman Moreau envelope of a func-tion f with parameter λ192Pλf The proximal operator of a function f withparameter Î»21âââproxÏÎ»f The left Bregman proximal operator of a func-tion f with parameter Î»182fâ The Fenchel conjugate of a function f 16fââ The Fenchel biconjugate of a function f 16dom f The effective domain of a function f 7epi f The epigraph of a function f 8j The function 12 â¥Â·â¥2 7domT The domain of an operator T 8gphT The graph of an operator T 8Tâ1 The inverse of an operator T 8[r]â [r]â = max(0,âr) for a real number r â R 95bdryC The boundary of a set C 7B The unit open ball {x â Rn |â¥xâ¥ < 1} 7C The closure of a set C 7dist(Â·, C) The distance function of a set C 9intC The interior of a set C 7δC The indicator function of a set C 9riC The relative interior of a set C 13xiGlossary of NotationN Natural numbers {0, 1, . . .} 7R Real numbers (ââ,â) 7R The extended-real line [ââ,â] 7âFf The Fenchel subdifferential of a function f 10âÌf The Fre´chet subdifferential of a function f 10âf The gradient a function f 11âf The limiting subdifferential of a function f 10âpf The proximal subdifferential of a function f 10ââf The singular subdifferential of a function f 11xiiAcknowledgementsFirst and foremost, I cannot thank enough my supervisor Dr. Xianfu Wangfor his broad knowledge and unwavering support. You are always availableto help whenever needed (even when I interrupt you), while granting me thefreedom to explore by myself. The independence you endowed in me turnsout to be an invaluable asset for my career. And above all, you always giveme the second chance to start again when I fail, for which I will forever bein debt.I would like to thank Dr. Heinz Bauschke for his constant encourage-ments and support, from which I have benefited tremendously. I deeplythank Dr. John Braun for agreeing to be my committee member. I wouldalso like to thank Dr. Jahangir Hossain for being my university examiner,and Dr. James Burke for being my external reviewer. I am grateful to Dr.Warren Hare, the graduate program coordinator, for his always timely as-sistance throughout my graduate study and job hunting.I thank Dr. Andreas Themelis for the remarkable collaborating experi-ence and the laughters we shared through either online calls or my visit toFukuoka. I also wish to thank Dr. Kaoru Yamamoto, for her warm welcomeand assistance during my stay in Japan.I wish to thank all my friends and colleagues at UBC Okanagan for theircomradery, which makes this journey quite joyful. A special thank mustgo to Dr. Jiaqi Li, who, among other things, constantly reminds me thatmathematics is not the only matter in the world.xiiiDedicationTo my parentsxivChapter 1Introduction1.1 BackgroundTo obtain clear images of cancer tumors, how does one retrieve the lostinformation in X-ray crystallography? This task can be modeled as a mini-mization problem [43], which often takes the following formminxâRnf(x), (1.1)where f : Rn â R is an objective to be minimized. Besides medical imaging,optimization problems are ubiquitous in modern applications, ranging frommachine learning, signal processing, computer vision, just to name a few.When the objective f is convex, which means that f is a bowl-shapedfunction, first-order convex optimization algorithmsâalgorithms utilizingfirst-order (generalized) derivativesâhave been investigated extensively, withfruitful applications and well-established theory hinging on convex analysisand monotone operator theory. These algorithms are also called splittingalgorithms, in the sense that they take advantages of structures of the prob-lem (1.1) by splitting the objective f into components with nicer properties.We refer interested readers to monographs [11, 28, 76, 82] and survey pa-pers [34,71] for detailed expositions on this line of work.However, these classic convex methods fall short in the contemporarysettingânonconvexity has become pervasive in the modern era of big data.More precisely, owing to the failure of convex analysis and monotone opera-tor theory beyond convexity, the following questions are not well-understood:Q1 When and how fast does an algorithm converge?Q2 If an algorithm converges, is the limit a stationary point, or a localminimizer, or a global minimizer of the problem?As an undesirable consequence, practitioners may resort to applying blindlyalgorithms designed for convex problems to solve nonconvex ones in theirreal-world applications, not knowing when and why these algorithms wouldwork.11.2. THE GOALS OF THIS THESISTo resolve these challenges, recent advancements in the optimizationcommunity revolve around identifying favorable properties of nonconvexfunctions using mathematical tools in variational analysis [81], and therebymay guarantee pleasant behavior of algorithms in the absence of convexity.The term âregularitiesâ appeared in the title of this thesis refers to thesefavorable properties, including the Kurdyka- Lojasiewicz property and vari-ational convexity, which will be introduced in details in the section below.1.2 The goals of this thesisBefore stating our gaols, we provide more context by introducing regularitiesto be considered in the sequel.Introduced by Moreau half a century ago [63,64], the proximal operatorPλf : Rn â Rn of a function f : Rn â R with parameter λ > 0 is definedby(âx â Rn) Pλf(x) = argminwâRn{f(w) + 12λ â¥w â xâ¥2},which lies at the very heart of first-order algorithms for both convex andnonconvex problems; see [11, 71, 77]. Indeed, these algorithms typically it-erate a proximal operator or its variation, and thus converge towards to asolution. When f is convex, pleasant features of Pλf are well-understood,underpinning the convergence of various convex optimization algorithms;see [11]. However, the picture is unclear in nonconvex setting. It is thusparamount to characterize the behavior of Pλf in the absence of convexity.The Kurdyka- Lojasiewicz (KL) property (see Definition 2.78) is a cel-ebrated regularizing property in nonconvex optimization, which originatesfrom the work by Polyak [75] on numerical optimization in 1963, and theworks by Lojasiewicz [56] in 1963 and by Kurdyka [48] in 1998 on algebraicgeometry. Roughly speaking, first-order algorithms may converge globallyto a stationary point of a nonconvex objective function in the presence ofthe KL property. Therefore this property has been utilized extensively innonconvex optimization to guarantee pleasant behavior of traditional al-gorithms in the absence of convexity; here we mention only the seminalpapers [2, 3, 19,20,21].Discovered by Rockafellar [78] in 2019, variational convexity is a new reg-ularity condition in variational analysis with emerging interest. In a seriesof articles, Rockafellar [78, 80] explored algorithmic applications of varia-tional convexity. A particularly appealing feature is that local optimalitycan be guaranteed under variational convexity. For instance, proximal point21.3. THE CONTRIBUTIONS OF THIS THESISmethod may converge to a local minimizer for a nonconvex function in thepresence of variational convexity; see [78, Theorem 4]; see also [80, Section2], [47, Section 6] and [89] for more details on this line of work.The goals of this thesis are to extend the forward-reflected-backward split-ting method in convex optimization to nonconvex setting using the KL prop-erty, to introduce and to investigate systematically new subdifferentials that,among other things, facilitate novel characterizations of proximal operatorsand their extensions in the absence of convexity, to characterize variationalconvexity and to explore its algorithmic applications.1.3 The contributions of this thesisIn this section we summarize the main contributions constituting this thesis,in the same order as they appear in the sequel.In Chapter 2 we summarize key concepts and classic results in convexanalysis, monotone operator theory, and variational analysis.In Chapter 3, we extend the forward-reflected-backward (FRB) splittingmethod for inclusion problems of monotone operators to structured noncon-vex minimization problems. By assuming the generalized concave Kurdyka- Lojasiewicz (KL) property of a quadratic regularization of the objective,we show that the FRB method converges globally to a stationary point ofthe objective and enjoys the finite length property. Numerical experimentssuggest that FRB is competitive compared to the Douglas-Rachford methodand the Bot¸-Csetnek inertial Tsengâs method. This chapter is based on [92].In Chapter 4, we propose and investigate the convergence of a Bregmaninertial forward-reflected-backward (BiFRB) method for nonconvex com-posite problems. Assuming the generalized concave Kurdyka- Lojasiewiczproperty, we obtain sequential convergence of BiFRB, as well as conver-gence rates on both the function value and actual sequence. One distin-guishing feature in our analysis is that we utilize a careful treatment ofmerit function parameters, circumventing the usual restrictive assumptionon the inertial parameters and stepsizes. We also present formulae for theBregman subproblem, supplementing not only BiFRB but also the work ofBot¸-Csetnek-La´szlo´ and Bot¸-Csetnek. This chapter is based on [94].In Chapter 5, we investigate a Bregman and inertial extension of theforward-reflected-backward algorithm applied to structured nonconvex min-imization problems under relative smoothness. To this end, the proposedalgorithm hinges on two key features: taking inertial steps in the dual space,and allowing for possibly negative inertial values. The interpretation of rel-31.3. THE CONTRIBUTIONS OF THIS THESISative smoothness as a two-sided weak convexity condition proves beneficialin providing tighter stepsize ranges. Our analysis begins with studying anenvelope function associated to the algorithm that takes inertial terms intoaccount through a novel product space formulation. Such construction sub-stantially differs from similar objects in the literature and could offer newinsights for extensions of splitting algorithms. Global convergence and ratesare obtained by appealing to the Kurdyka- Lojasiewicz property. This chap-ter is based on [96].In Chapter 6, we investigate Rockafellarâs level proximal subdifferentialâa new subdifferential whose resolvent always coincides with a proximal op-erator, even without hypoconvexity. We present various fundamental prop-erties and provide concrete examples of the level proximal subdifferential,and explore its relation with classic subdifferentials in variational analysis.This chapter is based on [95].In Chapter 7, we study the existence, single-valuedness, optimality condi-tions, and integration of level proximal subdifferential. We also characterizea pointwise Lipschitz smoothness, of which the level proximal subdifferentialserves as a natural characterization. These results reveal that, as a powerfultool, level proximal subdifferential provides deep insights into variationalanalysis and optimization. Results in this chapter are selected from [59].In Chapter 8, we characterize the variational and variational strong con-vexity of a function by nonexpansiveness-type properties of the associatedproximal operator and by the relative monotonicity of the level proximal sub-differential. Our results generalize the celebrated correspondence betweenconvex functions, monotone subdifferentials, and firmly nonexpansive prox-imal operators and reveal that these classic notions in convex optimizationreemerge locally on variational convex problems. Aided by these charac-terizations, new insights regarding the local behavior of proximal gradientmethod and the KrasnoselâskiËÄ±-Mann iteration are obtained. In particular,variational sufficiency guarantees that these methods converge to local min-imizers rather than just critical points. Operations that preserve variationalconvexity are also investigated. Results in this chapter are selected from [59].Finally in Chapter 9, we introduce the left and right Bregman level prox-imal subdifferentials and investigate them systematically. Amazingly, everyBregman proximal operator is the resolvent of a Bregman level proximalsubdifferential under a standard range assumption, even without convexity.Aided by this pleasant feature, we establish new correspondences among use-ful properties of the Bregman proximal operator, the underlying function,and the (left) Bregman level proximal subdifferential. Unlike the Euclideansetting, asymmetry and duality gap emerge as natural consequences of the41.3. THE CONTRIBUTIONS OF THIS THESISBregman distance. Along the way, we relax several redundant assumptionsin results by Kan and Song and by Wang and Bauschke. We also char-acterize the existence and single-valuedness of the Bregman level proximalsubdifferential, investigate coincidence results, and make an interesting con-nection to relative smoothness. Abundant examples are provided to justifythe necessity of our assumptions. This chapter is based [97].5Chapter 2PreliminariesThroughout,Rn is the standard n-dimensional Euclidean space,equipped with inner product â¨Â·, Â·â© and the induced norm â¥xâ¥ = ââ¨x, xâ©for every x â Rn. The real line is denoted by R = (ââ,â) and the ex-tended real line is R = [ââ,â]. The set of positive numbers is R+ ={r â R |r â¥ 0} and the set of strictly positive numbers is R++ = {r â R |r > 0}.The set of natural numbers is N = {0, 1, 2, . . .}. For simplicity, we write everysingleton set {a} = a when there is no ambiguity. Throughout, we setj = 12 â¥Â·â¥2 .The unit open ball in Rn is denoted by B = {x â Rn |â¥xâ¥ < 1} . For a func-tion f and a number η > 0, we write[f < η] = {x â Rn |f(x) < η} , [f â¤ η] = {x â Rn |f(x) â¤ η} .The interior, closure, and boundary of a set C â Rn are respectively denotedas intC, C, and bdryC = C intC.In the remainder of this chapter, we recall elements from convex analysis,monotone operator theory, and variational analysis. Readers are referredto [11,76,81] for detailed expositions on these subjects.2.1 Functions and operatorsIn this section we introduce notations and concepts related to extended-real-valued functions f : Rn â R and set-valued operators T : Rn â Rn.Definition 2.1. Let f : Rn â R be extended real-valued.(i) The (effective) domain of f is dom f = {x â Rn |f(x) <â}.(ii) The epigraph of f is epi f = {(x, r) â Rn Ã R |f(x) â¤ r}.62.1. FUNCTIONS AND OPERATORS(iii) The infimum of f over a set C â Rn isinfCf = inf {f(x) |x â C } .In particular, the infimum of f is infRn f . If the infimum is attained,we write infC f = minC f .(iv) The minimum of f over a set C â Rn isminCf = min {f(x) |x â C } .and the set of minimizers of f over a set C â Rn isargminCf ={{x â C |f(x) = infC f } , if infC f <â;â , if infC f =â.Definition 2.2. Let C â Rn be a subset.(i) The indicator function of C is δC : Rn â R defined as δC(x) = 0 ifx â C and â otherwise.(ii) The distance function of C is dist(Â·, C) : Rn â R defined by(âx â Rn) dist(x,C) = infyâCâ¥xâ yâ¥ .When C = â , dist(x,C) =â for all x â Rn.Definition 2.3. Let f : Rn â R. We say that f is(i) proper, if f(x) > ââ for every x â Rn and there is at least one x â Rnsuch that f(x) <â.(ii) lower semicontinuous (lsc) at x¯ if f(x¯) = lim infxâx¯ f(x). f is lowersemicontinuous if it is lsc at every x â Rn.(iii) differentiable at x¯, if x¯ â int dom f Ì¸= â and there exists a vectorv¯ â Rn such thatlimxâx¯f(x)âf(x¯)ââ¨v¯,xâx¯â©â¥xâx¯â¥ = 0.This vector v¯ is called the gradient of f at x¯, denoted by v¯ = âf(x¯).Definition 2.4. Let f : Rn â R. The closure of a function f : Rn â R iscl f : Rn â R defined by cl f(x) = lim infxâ²âx f(x) for all x â Rn.72.1. FUNCTIONS AND OPERATORSNext we turn to set-valued operators T : Rn â Rn, that is, an operatorwhose outputs are (possibly empty) sets.Definition 2.5. Let T : Rn â Rn be a set-valued operator.(i) The domain of T isdomT = {x â Rn |T (x) Ì¸= â } .(ii) The graph of T isgphT = {(x, u) |u â T (x)} .(iii) The inverse of T is Tâ1 : Rn â Rn defined by(ây â Rn) Tâ1(y) = {x â Rn |y â T (x)} .Definition 2.6. Let (Ck)kâN be a sequence of subsets Ck â Rn.(i) The outer limit of (Ck)kâN isLim supkââCk = {x â Rn |(âÎµ > 0)(âK â N)(âk â¥ K) x â Ck + εB} .(ii) The inner limit of (Ck)kâN isLim infkââCk = {x â Rn |(âÎµ > 0)(âK â N)(âk > K) x â Ck + εB} .(iii) We say that Ck â C if C = Lim supkââCk = Lim infkââCk.Definition 2.7. Let T : Rn â Rn be a set-valued mapping.(i) The outer limit of T at x¯ isLim supxâx¯T (x) =âxkâx¯Lim supkââT (xk)= {v¯ â Rn |(âxk â x¯)(âvk â T (xk))vk â v¯} .We say that T : Rn â Rn is outer semicontinuous at x¯, ifLim supxâx¯T (x) = T (x¯).82.2. THE SUBDIFFERENTIAL FAMILY(ii) The inner limit of T at x¯ isLim infxâx¯ T (x) =âxkâx¯Lim infkââT (xk)= {v¯ â Rn |(âxk â x¯)(âvk â T (xk))vk â v¯} .We say that T : Rn â Rn is inner semicontinuous at x¯, ifLim infxâx¯ T (x) â T (x¯).Definition 2.8. Let T : Rn â Rn be a set-valued mapping. We say that Tis locally bounded around x¯, if there exists δ > 0 such thatâxâx¯+Î´BT (x) is bounded.2.2 The subdifferential familyIn this section, we recall subdifferentials in variational analysis; see, e.g., [81].Definition 2.9. Let C â Rn be a subset set and let f : Rn â R be afunction. Let (xk)kâN be a sequence and let x¯. We say that (xk)kâN(i) C-attentively convergences to x¯, denoted by xkCââ x¯, if xk â x¯ withxk â C for all k â N.(ii) f -attentively convergences to x¯, denoted by xkfââ x¯, if xk â x¯ withf(xk)â f(x¯).Definition 2.10. Let f : Rn â R be proper and a point x¯ â dom f . For avector v â Rn, we say that v â Rn is a(i) Fenchel subgradient of f at x¯, denoted by v â âFf(x¯), if(âx â Rn) f(x) â¥ f(x¯) + â¨v, xâ x¯â©. (2.1)The set âFf(x¯) is called the Fenchel subdifferential of f at x¯.(ii) proximal subgradient of f at x¯, denoted by v â âpf(x¯), if there existÏ > 0 and δ > 0 such that(â â¥xâ x¯â¥ â¤ δ) f(x) â¥ f(x¯) + â¨v, xâ x¯â© â Ï2 â¥xâ x¯â¥2 . (2.2)The set âpf(x¯) is called the proximal subdifferential of f at x¯.92.2. THE SUBDIFFERENTIAL FAMILY(iii) Fre´chet subgradient of f at x¯, denoted by v â âÌf(x¯), if(âx â Rn) f(x) â¥ f(x¯) + â¨v, xâ x¯â©+ o(â¥xâ x¯â¥). (2.3)Equivalently, v â âÌf(x) if and only iflim infxâx¯f(x)âf(x¯)ââ¨v,xâx¯â©â¥xâx¯â¥ â¥ 0.The set âÌf(x¯) is called the Fre´chet subdifferential of f at x¯.(iv) limiting (Mordukhovich) subgradient of f at x¯, denoted by v â âf(x¯),ifv â Lim supxfââx¯âÌf(x),that is, there exist sequences xkfââ x¯ and vk â âÌf(xk) such that vk âv. The set âf(x¯) is called the limiting (Mordukhovich) subdifferentialof f at x¯.(v) singular (horizon) subgradient of f at x¯, denoted by v â ââf(x¯), ifv â Lim supxfââx¯Î»â0Î» âÌf(x),that is, there exist xkfââ x¯, λk â 0, and vk â âÌf(xk) such that λkvk âv. The set ââf(x¯) is called the singular (horizon) subdifferential of fat x¯.Definition 2.11. Let C â Rn be a nonempty set and let x¯ â C. For avector v â Rn, we say that v is a(i) regular normal vector to C at x¯, denoted by v â NÌC(x¯), if(âx â C) â¨v, xâ x¯â© â¤ o(â¥xâ x¯â¥).The collection of all such vectors of is called the regular normal coneto C at x¯, denoted by NÌC(x¯). Equivalently, v â NÌC(x¯) if and only iflim supxCââx¯â¨v,xâx¯â©â¥xâx¯â¥ â¤ 0.(ii) limiting normal vector to C at x¯, denoted by v â NC(x¯), if there existsxkCââ x¯ and vk â NÌC(xk) such that vk â v. The collection of all suchvectors is called the limiting normal cone to C at x¯, denoted by NC(x¯).102.2. THE SUBDIFFERENTIAL FAMILYRemark 2.12. Let f = δC be the indicator function for a nonempty set C.Then NÌC = âÌf and NC = âf .Below we collect some useful properties of subdifferentials.Proposition 2.13. Let f : Rn â R be proper and a point x¯ â dom f . Thenthe following hold:(i) âFf(x¯) â âpf(x¯) â âÌf(x¯) â âf(x¯).(ii) If f is differentiable at x¯, then âÌf(x¯) = âf(x¯). If in addition f iscontinuously differentiable around x¯, then âÌf(x¯) = âf(x¯) = âf(x¯).(iii) The limiting subdifferential âf is osc with respect to f -attentive con-vergent sequences, that is, for every x â dom f{v â Rnâ£â£â£(âxk fââ x) (âvk â âf(xk)) vk â v} â âf(x).Proof.(i) Clearly (2.1) is a special case of (2.2), therefore âFf(x¯) â âpf(x¯). Forv â âpf(x¯), let Ï be the parameter appeared in (2.2). Thenmin{âÏ2â¥xâx¯â¥2,f(x)âf(x¯)ââ¨v,xâx¯â©}â¥xâx¯â¥ â 0, as xâ x¯,where the limit holds owing to (2.2). Henceo(â¥xâ x¯â¥) = min{âÏ2 â¥xâ x¯â¥2 , f(x)â f(x¯)â â¨v, xâ x¯â©},which satisfies(âx â Rn) f(x) â¥ f(x¯) + â¨v, xâ x¯â©+ o(â¥xâ x¯â¥)â âpf(x¯) â âÌf(x¯).The last inclusion âÌf(x¯) â âf(x¯) holds by choosing a constant sequencexk = x¯ for all k â N.(ii) This is [81, Exercise 8.8].(iii) See [81, Proposition 8.7].Subdifferentials play a central role in characterizing optimality.Fact 2.14 (Fermatâs rule). Let f : Rn â R be proper. If x¯ is a localminimizer of f , then 0 â âÌf(x¯).112.2. THE SUBDIFFERENTIAL FAMILYProof. See [81, Theorem 10.1].Next we recall several calculus rules for subdifferentials.Fact 2.15. Let fi : Rni â R be lsc for i = 1, . . .m, and let f(x1, . . . , xm) =f1(x1) + · · Â·+ fm(xm). Then at x¯ = (x¯1, . . . , x¯m), the following hold:âÌf(x¯) = âÌf1(x¯1)Ã âÌf2(x¯2)Ã · · · Ã âÌfm(x¯m),âf(x¯) = âf1(x¯1)Ã âf2(x¯2)Ã · · · Ã âfm(x¯m).Proof. See [81, Proposition 10.5].Fact 2.16. Let f : Rn â R be continuously differentiable around x¯, at whichg : Rn â R is finite. ThenâÌ(f + g)(x¯) = âf(x¯) + âÌg(x¯), â(f + g)(x¯) = âf(x¯) + âg(x¯).Proof. This is [81, Exercise 8.8].Fact 2.17. Let f1 : Rn â R and f2 : Rn â R be proper lsc functions, andlet x¯ â dom(f1 + f2). ThenâÌ(f1 + f2)(x¯) â âÌf1(x¯) + âÌf2(x¯). (2.4)Suppose in addition that the constraint qualificationââf1(x¯) â© (âââf2(x¯)) = {0} (2.5)holds. Thenâf(x¯) â âf1(x¯) + âf2(x¯). (2.6)Proof. This is [81, Corollary 10.9].Remark 2.18. We delay the discussion regarding a class of functions un-der which (2.5) holds automatically; see Fact 2.70 and the consequentialCorollary 2.71.Fact 2.19 (composition). Suppose that f(x) = g(F (x)) for a proper lscfunction g : Rm â R and a smooth mapping F : Rn â Rm, and let x¯ be apoint where f is finite and the Jacobian JF (x¯) is surjective. ThenâÌf(x¯) = JF (x¯)T âÌg(F (x¯)),âf(x¯) = JF (x¯)T âg(F (x¯)).Proof. See [81, Exercise 10.7].122.3. CONVEXITY AND DUALITY2.3 Convexity and dualityIn this section we recall several useful notions in convex analysis; see [11,76].2.3.1 ConvexityWe start by defining convexity of sets and related notions.Definition 2.20. Let C â Rn be a set. We say that C is convex, if(âx0, x1 â C)(â0 < λ < 1) (1â λ)x0 + λx1 â C.Definition 2.21. For a set C â Rn, the affine hull of C isaff C ={kâi=1Î»ixiâ£â£â£â£â£(âxi â C)(âÎ»i â R)kâi=1Î»i = 1, k â N}.Definition 2.22. Let C â Rn be a set. The relative interior of C, denotedby riC, isriC = {x â aff C |(âÎµ > 0)(x+ εB) â© aff C â C } ,where aff C is the affine hull of C.Fact 2.23. Let C â Rn be a convex set. Then riC Ì¸= â .Proof. This is [76, Theorem 6.2].Lemma 2.24 (accessibility lemma). Let C â Rn be a convex set. Letx0 â riC and let x1 â clC. Then for every 0 â¤ λ < 1, (1âÎ»)x0+Î»x1 â riC.Proof. See [76, Theorem 6.1].Now we turn to convex functions and variations.Definition 2.25. Let f : Rn â R be a function with convex domain. Wesay that f is(i) convex, if (âx0, x1 â dom f) (â0 < λ < 1)f((1â λ)x0 + λx1) â¤ (1â λ)f(x0) + λf(x1).(ii) strictly convex, if (âx0, x1 â dom f) (â0 < λ < 1)x0 Ì¸= x1 â f((1â λ)x0 + λx1) < (1â λ)f(x0) + λf(x1).132.3. CONVEXITY AND DUALITY(iii) Ï-strongly convex for Ï > 0, if f â Ïj is convex.(iv) Ï-hypoconvex (or Ï-weakly convex) for Ï > 0, if f + Ïj is convex.Under convexity, subdifferentials introduced earlier coincide.Fact 2.26. Let f : Rn â R be proper and convex. Then the following hold(i) For every x â dom f ,âf(x) = {v â Rn |(ây â Rn) f(y) â¥ f(x) + â¨v, y â xâ©} = âFf(x).In turn âFf(x) = âpf(x) = âÌf(x) = âf(x).(ii) â =Ì¸ ri dom f â dom âf . Consequently f possesses a continuous affineminorant.Proof. See [81, Proposition 8.12] and [76, Theorem 23.4] for (i) and (ii),respectively.A particularly pleasant feature of convex functions is that they are con-tinuous along line segments.Fact 2.27. Let f : Rn â R be proper lsc and convex. Then for everyx1 â Rnf(x1) = limÎ»â1âf((1â λ)x0 + λx1),for x0 â dom f .Proof. This is [76, Corollary 7.5.1].Theorem 2.28. Let f : Rn â R be proper and convex, and let D be theset of points where f is differentiable. Then D â int dom f , on which thegradient operator âf is continuous.Proof. This is [76, Theorem 25.5].We end this section by introducing convex and closed convex hulls ofsets and of functions.Definition 2.29. Let C â Rn be a set. The convex hull of C, denoted byconvC, is the smallest convex set containing C. The closed convex hull ofC, denoted by convC, is the smallest closed convex set containing C.142.3. CONVEXITY AND DUALITYDefinition 2.30. Let f : Rn â R. The convex hull of f isconv f : Rn â [ââ,â] : x 7â inf {r â R |(x, r) â conv epi f } , (2.7)and the closed convex hull of f isconv f : Rn â [ââ,â] : x 7â inf {r â R |(x, r) â conv epi f } , (2.8)where inf â =â by convention.Proposition 2.31. Let f : Rn â R be proper and minorized by an affinefunction. Then for every x â Rnconv f(x)= sup {h(x) | h is a convex function such that h â¤ f }= sup{n+1âi=1Î±if(xi)â£â£â£â£â£x =n+1âi=1Î±ixi, where xi â dom f andn+1âi=1Î±i = 1}.Proof. See [40, Proposition 2.5.1].Lemma 2.32. Let f : Rn â R be proper and lsc, and let x â dom f . Thenthe following hold:(i) If (conv f)(x) = f(x), then âFf(x) = â(conv f)(x).(ii) If âFf(x) Ì¸= â , then (conv f)(x) = f(x).Proof.(i) Take u â âFf(x). Then the function y 7â â¨u, y â xâ© + f(x) is a convex(linear) minorant of f . Hence(ây â Rn) (conv f)(y) â¥ â¨u, y â xâ©+ f(x) = â¨u, y â xâ©+ (conv f)(x),so u â (conv f)(x). Conversely for u â â(conv f)(x),(ây â Rn) f(y) â¥ (conv f)(y) â¥ â¨u, y â xâ©+ (conv f)(x)= â¨u, y â xâ©+ f(x),so u â âF f(x).(ii) The assumption implies that f is minorized by a convex (linear) func-tion y 7â â¨u, y â xâ© + f(x) at x, hence Proposition 2.31 enforces that(ây â Rn) f(y) â¥ (conv f)(y) â¥ â¨u, y â xâ© + f(x). In particular, f(x) â¥(conv f)(x) â¥ f(x).152.3. CONVEXITY AND DUALITY2.3.2 DualityDefinition 2.33. Let f : Rn â R. The (Fenchel) conjugate of f isfâ : Rn â R : y 7â supxâRn(â¨y, xâ© â f(x)) ,and the biconjugate of f is fââ = (fâ)â.Fact 2.34. Let f : Rn â R. If f has a proper convex hull conv f , then bothfâ and fââ are proper, lsc, and convex, andfââ = conv f.Thus fââ â¤ f .Proof. This is [81, Theorem 11.1].Fact 2.35. Let f : Rn â R be proper. Then f is lsc and convex if and onlyif f = fââ.Proof. See [11, Theorem 13.32].Fact 2.36. Let f : Rn â R. Then the following hold:(i) fâ = (conv f)â = (cl f)â = (conv f)â.(ii) If f is proper, then (âx, y â Rn) fâ(y) + f(x) â¥ â¨x, yâ©.Proof. See [81, Theorem 11.1] and [11, Proposition 13.13] for (i) and (ii),respectively.Fact 2.37 (Fenchel duality). Let f : Rn â R be proper and convex. Thenthe following are equivalent for x, y â Rn:(i) y â âf(x).(ii) fâ(y) + f(x) = â¨x, yâ©.If in addition that f is lsc at x, then the following can be added to theequivalence:(iii) x â âfâ(y).162.3. CONVEXITY AND DUALITYProof. This is [76, Theorem 23.5].Corollary 2.38. Let f : Rn â R be proper, lsc, and convex. Then âfâ =(âf)â1.Now we introduce an important class of functions.Definition 2.39. Let f : Rn â R be proper and convex. We say that f is(i) essentially smooth, if f is differentiable on int dom f Ì¸= â andâ¥âf(xk)â¥ â â,whenever (xk)kâN is a sequence converging to some boundary pointx â bdry dom f .(ii) essentially strictly convex, if f is strictly convex on every convex sub-sets of dom âf .(iii) Legendre, if f is both essentially smooth and essentially strictly convex.Fact 2.40. Let f : Rn â R be proper lsc and convex. Then the followingare equivalent:(i) f is essentially smooth(ii) fâ is essentially strictly convex.(iii) âf is single-valued and dom âf = int dom fProof. See [76, Corollary 26.3.1] for (i) â (ii) and [8, Theorem 5.6] for (i)â (iii).Fact 2.41. Let f : Rn â R be a Legendre function. Then the operatorâf : int dom f â int dom fâ is continuous, one-to-one from int dom f ontoint dom fâ, with inverse (âf)â1 = âfâ.Proof. See [76, Theorem 26.5].172.4. PARAMETRIC MINIMIZATION2.4 Parametric minimizationDefinition 2.42. Let f : Rn â R. We say that f is(i) level-bounded, if for every α â R the level set [f â¤ α] is bounded(possibly empty).(ii) coercive, if limâ¥xâ¥ââ f(x) =â.(iii) 1-coercive, if limâ¥xâ¥ââf(x)â¥xâ¥ =â.Fact 2.43. Let f : Rn â R be proper, lsc, and coercive. Then inf f > ââand argmin f Ì¸= â .Proof. See [11, Theorem 11.9, Proposition 11.12].We shall exchange level-boundedness and coercivity whenever itâs con-venient, owing to the theorem below.Theorem 2.44. A function f : Rn â R is level-bounded if and only if it iscoercive.Proof. Suppose first that f is level bounded and suppose that for some un-bounded sequence xk there is M â R such that f(xk) â¤M for k sufficientlylarge. Then the level set [f â¤ M ] is not bounded, which contradicts tothe assumption. Conversely, suppose that f is coercive and assume that[f â¤ α0] is unbounded for some α0 â R. Taking (xk) â [f â¤ α0] such thatâ¥xkâ¥ â â as k â â yields a contradiction to the assumption that f iscoercive.Proposition 2.45. Let f : Rn â R be 1-coercive. Then the following hold:(i) f + â¨v, Â·â© is 1-coercive for every v â Rn.(ii) If g : Rn â R is proper and convex, then f + g is 1-coercive.Proof.(i) Apply the Cauchy-Schwarz inequality to see thatf(x) + â¨v, xâ© â¥ f(x)â â¥vâ¥ â¥xâ¥ = â¥xâ¥(f(x)â¥xâ¥ â â¥vâ¥)ââ,as â¥xâ¥ â â.182.4. PARAMETRIC MINIMIZATION(ii) In view of Fact 2.26(ii), the function g has a continuous minorant, sayg â¥ â¨v, Â·â© + r for some v â Rn and r â R. Then f + g â¥ f + â¨v, Â·â© + r, towhich (i) implies the desired result.Lemma 2.46. Let f : Rn â R be proper lsc and convex. Then the followingare equivalent:(i) f is 1-coercive.(ii) f â â¨v, Â·â© is coercive for every v â Rn.(iii) dom âfâ = Rn and âfâ maps bounded sets to bounded sets.(iv) f is cofinite, that is, dom fâ = Rn.Proof. See [8, Theorems 3.3â3.4].Fact 2.47. Let f : Rn â R be proper, lsc, and 1-coercive. Then conv(epi f)is closed. Consequently, conv f = conv f .Proof. See [17, Lemma 3.3(i)].The following is an extension of level-boundedness, which will be usefulfor studying parametric minimization and particularly useful in Chapter 5.Definition 2.48. A function f : Rn ÃRm â R with values f(x, u) is level-bounded in x locally uniformly in u, if to each u¯ â Rm and α â R correspondδ > 0 such that âuâu¯+Î´B{x â Rn |f(x, u) â¤ α} is bounded.Theorem 2.49. Considerp(u) = infxâRnf(x, u), P (u) = argminxâRnf(x, u)for a proper and lsc function f : Rn Ã Rm â R with values f(x, u). Thenthe following hold:(i) Let u¯ â Rm be such that the function x 7â f(x, u¯) is proper. If x 7âf(x, u¯) is level-bounded, then p(u¯) is finite.192.4. PARAMETRIC MINIMIZATIONSuppose in addition that f(x, u) is level-bounded in x locally uniformly in u.Then(ii) The value function p is lsc on Rm.(iii) The solution map P satisfies domP = dom p and P (u) is compact forevery u â domP .(iv) If xk â P (uk) for a sequence (uk)kâN converging to some u¯ â dom pin a way such that p(uk) â p(u¯), then (xk)kâN is bounded and all itscluster points lie in P (u¯).(v) If f(x¯, u) is continuous at u¯ for x¯ â P (u¯), then the value function pis continuous at u¯.Proof. We prove only (i) for the sake of completeness, which is a more refinedstatement than [81, Theorem 1.17(a)], and refer readers to [81, Theorem1.17] for the rest. Indeed, the assumption in (i) implies that x 7â f(x, u¯)is coercive in view of Theorem 2.44 and therefore p(u¯) = infx f(x, u¯) =f(x¯, u¯) â R for some x¯ â argminx f(x, u¯) Ì¸= â .2.4.1 Moreau envelopes and proximal operatorsMoreau envelopes and proximal operators play important roles in variationalanalysis and optimization.Definition 2.50. Let f : Rn â R be proper and lsc, and let λ > 0.(i) The Moreau envelope of f with parameter λ > 0 iseλf(x) = infwâRn{f(w) + 12λ â¥xâ wâ¥2}.(ii) The proximal operator of f with parameter λ > 0 isPλf(x) = argminwâRn{f(w) + 12λ â¥xâ wâ¥2}.In general, the Moreau envelope could be ââ everywhere, and simulta-neously the associated proximal operator could be empty-valued.Example 2.51. Let f(x) = â12 â¥xâ¥2. Then for every λ > 1(âx â Rn) eλf(x) = ââ and Pλf(x) = â .202.4. PARAMETRIC MINIMIZATIONProof. It is easy to see thateλf(0) =12 infwâRn{(1Î» â 1) â¥wâ¥2} = ââ.Now fix nonzero x â Rn. By definition for every r â R,eλf(x) = infwâRn{f(w) + 12λ â¥xâ wâ¥2}â¤ f((1 + r)x) + 12λ â¥(1 + r)xâ xâ¥2= ââ¥xâ¥22((1â 1λ)r2 + 2r + 1)â ââ as r ââ,implying that eλf(x) = ââ and Pλf(x) = â .To avoid the situation in Example 2.51, certain regularization should bein place to ensure meaningful eλf and Pλf , giving rise to the following.Definition 2.52 (prox-bounded function). A f : Rn â R is said to be prox-bounded, if there exists λ > 0 such that eλf(x) > ââ for some x â Rn. Thesupremum of all such λ is the threshold of prox-boundedness for f , denotedby λf .Proposition 2.53. Let f : Rn â R be proper and lsc. Then the followingare equivalent:(i) f is prox-bounded.(ii) f + r2 â¥Â·â¥2 is bounded below for some r â R.(iii) lim infâ¥xâ¥ââf(x)â¥xâ¥2 > ââ.Proof. See [81, Exercise 1.24].Moreau envelopes and proximal operators of prox-bounded functions be-have nicely.Fact 2.54. Let f : Rn â R be proper, lsc, and prox-bounded with thresholdλf . Then for every 0 < λ < λf the following hold:(i) dom eλf = domPλf = Rn.(ii) The Moreau envelope eλf(x) is continuous.(iii) The proximal operator Pλf is compact-valued, and outer semicontinu-ous.212.4. PARAMETRIC MINIMIZATIONProof. This is [81, Theorem 1.25].Definition 2.55. For f : Rn â R and λ > 0, the λ-proximal hull of f ishλf(x) = supwâRn{eÎ»f(w)â 12λ â¥xâ wâ¥2}.Proposition 2.56. The following hold for f : Rn â R and λ > 0:(i) hλf â¤ f . If x â ranPλf , then hλf(x) = f(x).(ii) hλf = â eÎ»(â eλf) and eλ(hλf) = eλf .(iii) λeλf = jâ(Î»f+j)â and hλf = (f+Î»â1j)âââÎ»â1j. If in addition f isproper, lsc, and prox-bounded, then for 0 < λ < λf the biconjugate (f+Î»â1j)ââ can be replaced by conv(f+Î»â1j) and domhλf = conv dom f .Proof.(i) Fix x â Rn. Then for every w â Rn, the definition of Moreau envelopeyields f(x) â¥ eλf(w) â 12λ â¥xâ wâ¥2. Hence f(x) â¥ hλf(x). Suppose x âPÎ»f(w) for some w â Rn. Then hλf(x) â¥ eλf(w) â 12λ â¥xâ wâ¥2 = f(x) +12λ â¥xâ wâ¥2 â 12λ â¥xâ wâ¥2 = f(x) â¥ hλf(x), enforcing f(x) = hλf(x).(ii) The first identity is trivial from the definition. To see the second, noticethat (i) implies eλ(hλf) â¤ eλf . The converse holds because for a fixedw â Rn(âx â Rn) hλf(x) + 12λ â¥xâ wâ¥2 â¥ eÎ»f(w)â 12λ â¥xâ wâ¥2+ 12λ â¥xâ wâ¥2= eλf(w),from which taking the infimum over x yieldseλ(hλf)(w) = infxâRn{hÎ»f(x) +12λ â¥xâ wâ¥2}â¥ eλf(w).(iii) See [81, Example 11.26].222.5. MONOTONICITY AND NONEXPANSIVENESS2.5 Monotonicity and nonexpansivenessIn this section, we recall some fundamental facts about monotone and non-expansive operators.Definition 2.57. Let T : Rn â Rn be a set-valued operator. We say thatT is monotone, if(â(x1, u1), (x2, u2) â gphT ) â¨x1 â x2, u1 â u2â© â¥ 0T is said to be maximally monotone, if there is no set-valued operator TË :Rn â Rn such that TË is monotone and gphT â gph TË .Proposition 2.58. Let T : Rn â Rn be a set-valued operator. Then T ismonotone if and only if so is Tâ1.Fact 2.59. Let f : Rn â R be proper and lsc. Then f is convex if and onlyif âf is (maximally) monotone.Proof. This is [81, Theorem 12.17].Theorem 2.60. Let f : Rn â R be proper and lsc, and let λ > 0. ThenPλf : Rn â Rn is monotone, and the following are equivalent:(i) Pλf is maximally monotone.(ii) λf + j is convex.Proof. See [81, Proposition 12.19].Definition 2.61. Let D be a nonempty subset of Rn and let T : D â Rn.We say that T is(i) firmly nonexpansive if(âx, y â D) â¨xâ y, Txâ Tyâ© â¥ â¥Txâ Txâ¥2 . (2.9)(ii) nonexpansive if(âx, y â D) â¥Txâ Txâ¥ â¤ â¥xâ yâ¥ . (2.10)Fact 2.62. Let D be a nonempty subset of Rn and let T : D â Rn. Thenthe following are equivalent:232.6. VARIOUS NOTIONS OF SMOOTHNESS(i) T is firmly nonexpansive.(ii) IdâT is firmly nonexpansive.(iii) 2T â Id is nonexpansive.(iv) (âx, y â D) â¥Txâ Tyâ¥2 + â¥(IdâT )xâ (IdâT )yâ¥2 â¤ â¥xâ yâ¥2.Proof. This is [11, Proposition 4.2].Definition 2.63. Let D be a nonempty subset of Rn and let T : D â Rn.Let β > 0. We say that T is β-cocoercive, if βT is firmly nonexpansive, thatis,(âx, y â D) â¨xâ y, Txâ Tyâ© â¥ β â¥Txâ Txâ¥2 . (2.11)Definition 2.64. Let D be a nonempty subset of Rn, let T : D â Rn benonexpansive, and let 0 < α < 1. Then T is α-averaged, if there exists anonexpansive operator R : D â Rn such that T = (1â α) Id+αR.2.6 Various notions of smoothness2.6.1 Lipschitz smoothnessIn this section we collect important notions of smoothness, starting fromLipschitz smoothness.Definition 2.65. Let f : Rn â R be a differentiable function and let L > 0.We say that f is L-Lipschitz smooth (or Lipschitz differentiable), if(âx, y â Rn) â¥âf(x)ââf(y)â¥ â¤ L â¥xâ yâ¥ .Fact 2.66. Let f : Rn â R be a proper, lsc, and convex function. Then thefollowing are equivalent:(i) Lj â f is a convex function.(ii) f is L-Lipschitz smooth.(iii) fâ is 1/L-strongly convex.(iv) f is differentiable and âf is 1/L-cocoercive.(v) (descent lemma) f is differentiable and(âx, y â Rn) f(y) â¤ f(x) + â¨âf(x), y â xâ©+ L2 â¥xâ yâ¥2 .242.6. VARIOUS NOTIONS OF SMOOTHNESS(vi) (âx, y â Rn) â¨xâ y,âf(x)ââf(y)â© â¤ L â¥xâ yâ¥2 .Proof. See [11, Theorem 18.15].Corollary 2.67 (Baillon-Haddad). Let f : Rn â R be a differentiableconvex function. Then âf is nonexpansive if and only if âf is firmly non-expansive.We prove an extended version of Fact 2.66 in the absence of convexity.Theorem 2.68. The following are equivalent for f : Rn â R and L > 0:(i) Lj + f and Lj â f are convex functions.(ii) (descent lemma) f is continuously differentiable and(âx, y â Rn) |f(y)â f(x)â â¨âf(x), y â xâ©| â¤ L2 â¥xâ yâ¥2 . (2.12)(iii) f is continuously differentiable and(âx, y â Rn) |â¨xâ y,âf(x)ââf(y)â©| â¤ L â¥xâ yâ¥2 . (2.13)(iv) f is differentiable and âf is L-Lipschitz.Proof.â(i)â (ii)â Fact 2.16 imply that âÌ(Â±f) = âÌ(Â±f+LjâLj) = âÌ (Â±f + Lj)âL Id Ì¸= â on Rn, where âÌ (Â±f + Lj) Ì¸= â everywhere by convexity, implyingthat f is differentiable. Hence Theorem 2.28 yields that âf = â(Lj + f)âL Id is continuous owing to the convexity of Lj+f . Moreover, the convexityof f + Lj also implies for every x, y â Rn, L2 â¥yâ¥2 + f(y) â¥ L2 â¥xâ¥2 + f(x) +â¨â(Lj + f)(x), y â xâ©. Equivalently,L2 â¥y â xâ¥2 â¥ âf(y) + f(x) + â¨âf(x), y â xâ©.Similarly the convexity of âf + Lj givesL2 â¥y â xâ¥2 â¥ f(y)â f(x)â â¨âf(x), y â xâ©,completing the proof.â(ii) â (iii)â Use the symmetry of (2.12).252.6. VARIOUS NOTIONS OF SMOOTHNESSâ(iii) â (iv)â Let g = f + Lj. Then (2.13) is equivalent to(âx, y â Rn) 0 â¤ â¨xâ y,âg(x)ââg(y)â© â¤ 2L â¥xâ yâ¥2 ,which means that g is convex and satisfies the condition in Fact 2.66(vi).Apply Fact 2.66 to see that, equivalently, âg is 1/(2L)-cocoercive. Hencefor every x, y â Rn12L â¥âg(x)ââg(y)â¥2 â¤ â¨xâ y,âg(x)ââg(y)â©.Expanding the squares and rearranging yield (iv).â(iv) â (i)â By the Cauchy-Schwarz inequality |â¨âf(x)ââf(y), xâ yâ©| â¤â¥âf(x)ââf(y)â¥ â¥xâ yâ¥ â¤ L â¥xâ yâ¥2 holds for every x, y â Rn. Equiv-alently â¨â(Lj Â± f)(x) â â(Lj Â± f)(y), x â yâ© â¥ 0, entailing (i) throughFact 2.59.2.6.2 SubsmoothnessIn this section we introduce subsmoothness. The following concepts will beneeded soon.Definition 2.69. A function f : Rn â R is(i) strictly continuous at x¯, if x¯ â int dom f and the valuelim supx,xâ²âx¯,x Ì¸=xâ²|f(x)âf(xâ²)|â¥xâxâ²â¥ is finite.(ii) strictly differentiable at x¯ if f(x¯) is finite and there is a vector v, whichwill be the gradient âf(x¯), such thatf(xâ²)âf(x)ââ¨v,xâ²âxâ©â¥xâ²âxâ¥ â 0, as x, xâ² â x¯ with x Ì¸= xâ².Equivalently, (âx, xâ² â Rn) f(xâ²) = f(x) + â¨v, xâ² â xâ©+ o(â¥xâ² â xâ¥).Strict continuity amounts to local Lipschitzness.Fact 2.70. Let f : Rn â R be proper and lsc, and let x¯ â dom f . Then thefollowing are equivalent:(i) f is strictly continuous at x¯.(ii) the mapping x 7â âÌf(x) is locally bounded around x¯.262.6. VARIOUS NOTIONS OF SMOOTHNESS(iii) the mapping x 7â âf(x) is locally bounded around x¯.(iv) ââf(x¯) = 0.(v) f is finite-valued and locally Lipschitz around x¯.When one of the above holds, âf is nonempty and compact.Proof. See [81, Theorem 9.13].Corollary 2.71. Let f, g : Rn â R be proper and lsc, and let x¯ â dom f .If g is strictly continuous at x¯, thenâ(f + g)(x¯) â âf(x¯) + âg(x¯).If in addition that g is strictly differentiable at x¯, then the above inclusionbecomes an equality.Proof. The strict continuity of g at x¯ implies that ââg(x¯) = 0, in whichcase the constraint qualification (2.5) in Fact 2.17 is satisfied, implying theinclusion. Assumed now that g is strictly differentiable at x¯. Then g is differ-entiable at x¯ with â(âg)(x¯) = â(âg)(x¯) = ââg(x¯). In turn appealing theestablished inclusion to f = (f + g)+ (âg), where âg is strictly continuous,yieldsâf(x¯) â â(f + g)(x¯) + â(âg)(x¯) = â(f + g)(x¯)ââg(x¯),completing the proof.Definition 2.72 (subsmoothness). A function f : Rn â R, is said to be(i) lower-C1 on O â Rn, if on some neighborhood V of each x¯ â O, thereis a representationf(x) = maxtâTft(x),in which the functions ft(x) are of class C1 on V and the index setT is a compact topological space such that ft(x) and âft(x) dependscontinuously jointly on (t, x) â T Ã V .(ii) lower-Ck on O â Rn, if such a local representation can be arranged inwhich the functions ft(x) are of the class Ck, with ft(x) and its kthorder partial derivatives depend continuously jointly on (t, x) â T ÃV .272.7. PROX-REGULAR FUNCTIONSFact 2.73 (consequences of subsmoothness). A lower-C1 function f : Rn âR on an open set O is strictly continuous, and thus satisfy properties inFact 2.70.Fact 2.74 (subsmoothness of Moreau envelopes). Let f : Rn â R be proper,lsc, and prox-bounded with threshold λf . Then the following hold for 0 <λ < λf the function âeÎ»f is lower-C2, and therefore strictly continuous.Moreover,â(âeÎ»f) = 1λ (convPλf â Id) , (2.14)â(eÎ»f) â 1λ (IdâPÎ»f) . (2.15)Proof. See [81, Example 10.32].2.7 Prox-regular functionsDefinition 2.75. A function f : Rn â R is prox-regular at x¯ for v¯ â âf(x¯)if x¯ â dom f at which f is lsc, and there exist ε > 0 and Ï â¥ 0 such that(ây â B(x¯; ε)) f(y) â¥ f(x) + â¨v, y â xâ© â Ï2â¥y â xâ¥2, (2.16)for every (x, v) â gph âf â© [B(x¯; Îµ)Ã B(v¯; ε)] with f(x) < f(x¯) + ε.Prox-regularity has significant consequences.Fact 2.76. Let f : Rn â R be proper and lsc. Let v¯ â âf(x¯) be a proximalsubgradient. Then the following are equivalent:(i) f is prox-regular at x¯ for v¯.(ii) âf has an f -attentive localization T around (x¯, v¯) such that (âÏ > 0)T + Ï Id is monotone, where (âÎµ > 0)T (x) ={{v â âf(x) |v â v¯ + εB} , if f(x) < f(x¯) + ε, x â x¯+ ÎµB;â , otherwise.(2.17)Proof. See [81, Theorem 13.36].Fact 2.77. Suppose that f : Rn â R is prox-regular at x¯ for v¯ â âf(x¯), andthat f is prox-bounded. Then there exists λ¯ > 0 such that for all 0 < λ < λ¯there is a neighborhood Uλ of x¯+ λv¯ on which the following hold:282.8. THE KURDYKA- LOJASIEWICZ PROPERTY(i) Pλf is monotone, single-valued and Lipschitz continuous on Uλ, andPλf(x¯+ λv¯) = x¯. Moreover Pλf = (Id+λT )â1 on Uλ.(ii) eλf is differentiable on Uλ with âeÎ»f = Î»â1(IdâPÎ»f) =(λ Id+Tâ1)â1.Here the mapping T is a truncation of âf around (x¯, v¯) taking the form of(2.17) and the neighborhood Uλ can be taken to be ran(Id+λT ).Proof. See [81, Theorem 13.37].2.8 The Kurdyka- Lojasiewicz propertyFor η > 0, we writeKη ={Ï : [0, Î·)â R+â£â£â£â£â£ Ï is increasing, and continuous on [0, η),and differentiable on (0, η) with Ï(0) = 0.},andΦη ={Ï : [0, Î·)â R+â£â£â£â£â£ Ï is increasing, and continuous on [0, η),with Ï(0) = 0.}.Definition 2.78. Let f : Rn â R be proper and lsc, and let x¯ â dom âf .We say that f satisfies the (concave) Kurdyka- Lojasiewicz (KL) property atx¯, if there exist η > 0, a neighborhood of U â x¯, and concave Ï â Kη, suchthatÏâ²(f(x)â f(x¯)) dist(0, âf(x)) â¥ 1, (2.18)for every x â U â© [f(x¯) < f < f(x¯) + η].Definition 2.79.(i) A set E â Rn is called semialgebraic if there exist finitely many poly-nomials gij , hij : Rn â R such thatE =pâj=1qâi=1{x â Rn : gij(x) = 0 and hij(x) < 0}.(ii) A function f : Rn â R is called semialgebraic if its graph is semialge-braic.292.8. THE KURDYKA- LOJASIEWICZ PROPERTYFact 2.80. Let f : Rn â R be a proper and lsc function and let x¯ â dom âf .If f is semialgebraic, then it has the KL property at x¯ with Ï(t) = c · t1âÎ¸for some c > 0 and θ â (0, 1).Proof. This is [20, Corollary 16].Semialgebraic sets and functions are ubiquitous in real-world applica-tions, and we refer the readers to [3, Section 4.3] for a comprehensive col-lection. Here we only mention several examples which will be used later.In this rest of this thesis, the support of a vector x = (x1, . . . , xn) â Rn isdefined to besupp(x) = {J â {1, . . . , n} | xj Ì¸= 0 if j â J and xj = 0 otherwise} ,and we define for r â¥ 0Jr = {J â {1, . . . , n} |card(J) = r} ,where card(J) denotes the cardinality of the set of indices J . The zeroânormâ is â¥Â·â¥0 : Rn â N defined by(âx â Rn) â¥xâ¥0 = card(supp(x)).Example 2.81 (semialgebraic sets). The following are semialgebraic:(i) Finite unions and intersections of semialgebraic sets.(ii) {x â Rn |Ax â¤ b} and {x â Rn |Ax = b} for A â RmÃn and b â Rm.(iii) {x â Rn |â¥xâ¥0 â¤ r} for r â¥ 0.(iv) {x â Rn |â¥xâ¥â â¤ R} for R â¥ 0.Proof.(i) Immediate from the definition.(ii) Suppose that A = [a1, a2, . . . , am] for ai â Rn and that b = (b1, . . . , bm) âRm. Notice that halfspaces {x â Rn |â¨ai, xâ© â¤ bi } are semialgebraic. There-fore so are their intersections {x â Rn |Ax â¤ b}. The same procesure provesthat {x â Rn |Ax = b} is semialgebraic.302.8. THE KURDYKA- LOJASIEWICZ PROPERTY(iii) The definition of â¥Â·â¥0 indicates that for every x â Rnâ¥xâ¥0 â¤ r â (âJ â Jr)(âj â {1, . . . , n} J) xj = 0.â (âJ â Jr)(âj â {1, . . . , n} J) â¨ej , xâ© = 0,where ej denotes the jth basis in Rn. Therefore{x â Rn |â¥xâ¥0 â¤ r} =âJâJrâjâ{1,...,n}J{x â Rn |â¨ej , xâ© = 0} ,completing the proof.(iv) Note that {x â Rn |â¥xâ¥â â¤ R} is a convex polytope. Indeed,â¥xâ¥â â¤ Râ (â1 â¤ i â¤ n) xi â¥ âR and xi â¤ Râ (â1 â¤ i â¤ n) â â¨ei, xâ© â¤ R and â¨ei, xâ© â¤ R,which means once again that the box {x â Rn |â¥xâ¥â â¤ R} is semialgebraicowing to (ii).Example 2.82 (semialgebraic functions). The following functions are semi-algebraic, and therefore satisfy the KL property.(i) The zero ânormâ function â¥Â·â¥0.(ii) Every polynomial function is semialgebraic.Proof.(i) Let J â {1, . . . , n} be a set of indices and define(â1 â¤ j â¤ n) AJj ={{x â Rn |â¨ej , xâ© = 0} , if j /â J ;{x â Rn |â¨ej , xâ© < 0 or â¨ej , xâ© > 0} , if j â J,where ej denotes the jth basis of Rn. Then AJj is semialgebraic, being eithera hyperplane, or the union of half spaces. Similarly, so is AJj Ã {card J},becauseAJj Ã {card J}={(x, r) â Rn Ã R â£â£x â AJj }â {(x, r) â Rn Ã R |r â card J = 0} .Moreover, observe thatâ¥xâ¥0 = r â (âJ â {1, . . . , n}) x ânâj=1AJj and r = card J.312.8. THE KURDYKA- LOJASIEWICZ PROPERTYIt then follows thatgph â¥Â·â¥0 =âJâ{1,...,n}ï£«ï£­ nâj=1AJj Ã {card J}ï£¶ï£¸is semialgebraic being a finite union of semialgebraic sets.(ii) Let f : Rn â R be a polynomial. Thengph f = {(x, r) â Rn Ã R |f(x)â r = 0}is semialgebraic, since (x, r) 7â f(x)âr is again a polynomial on RnÃR.Remark 2.83. The exponential function x 7â ex is not semialgebraic. Nev-ertheless, it is a KL function.Fact 2.84. The following hold:(i) If f, g : Rn â R are semialgebraic, then so is f + g.(ii) If the function g : RnÃRm â R and the set D â Rm are semialgebraic,then so is f(x) = infyâD g(x, y).Proof. See [3, Section 4.3].Lemma 2.85. Let f : Rn â R be proper lsc and let µ â R. Let â¦ â dom âfbe a nonempty compact set on which f(x) = µ for all x â â¦. Supposethat f satisfies the KL property every each point x¯ â â¦. Then there existε > 0, η > 0 and concave Ï â Kη such thatÏâ²(f(x)â µ) dist (0, âf(x)) â¥ 1whenever x â â¦Îµ â© [µ < f < µ+ η], where â¦Îµ = {x â Rn |dist(x,â¦) < ε}.Proof. See [22, Lemma 6].The left derivative of a function Ï : Râ R at x is defined byÏâ²â(x) = limhâ0âÏ(x+h)âÏ(x)h .The following is an extension of the classic KL property; see [93]. Utilizing anondifferentiable extension, it captures the smallest possible desingularizingfunction.322.8. THE KURDYKA- LOJASIEWICZ PROPERTYDefinition 2.86. Let f : Rn â R be proper and lsc. Let x¯ â dom âf andµ â R, and let V â dom âf be a nonempty subset.(i) We say that f has the pointwise generalized concave KL at x¯ â dom âfif there exist a neighborhood U â x¯, η â (0,â] and concave Ï â Φη,such that for all x â U â© [0 < f â f(x¯) < Î·],Ïâ²â(f(x)â f(x¯)) · dist (0, âf(x)) â¥ 1. (2.19)(ii) Suppose that f(x) = µ on V . We say f has the setwise generalizedconcave KL property1 on V if there exist U â V , η â (0,â] andconcave Ï â Φη such that for every x â U â© [0 < f â µ < Î·],Ïâ²â(f(x)â µ) · dist (0, âf(x)) â¥ 1. (2.20)Definition 2.87. Let f : Rn â R be proper and lsc. Let x¯ â dom âfand let U â dom âf be a neighborhood of x¯. Let η â (0,â] and defineh : (0, Î·)â R byh(s) = sup{distâ1(0, âf(x)): x â U â© [0 < fâf(x¯) < η], s â¤ f(x)âf(x¯)}.The exact modulus of the generalized concave KL property of f at x¯ withrespect to U and η is the functionÏË : [0, Î·)â R+ : t 7ââ« t0h(s)ds, ât â (0, η),and ÏË(0) = 0. If U â© [0 < f â f(x¯) < η] = â for given U â x¯ and η > 0, thenwe set the exact modulus with respect to U and η to be ÏË(t) â¡ 0.Proposition 2.88. Let f : Rn â R be proper lsc and let x¯ â dom âf . Let Ube a nonempty neighborhood of x¯ and η â (0,â]. Let Ï â Φη be concave andsuppose that f has the generalized concave KL property at x¯ with respect toU , η and Ï. Then the exact modulus of the generalized concave KL propertyof f at x¯ with respect to U and η, denoted by ÏË, is well-defined, concave andsatisfiesÏË(t) â¤ Ï(t), ât â [0, η).Moreover, the function f has the generalized concave KL property at x¯ withrespect to U , η and ÏË. Consequently, the exact modulus ÏË satisfiesÏË = inf{Ï â Î¦Î·â£â£â£â£â£ Ï is a concave desingularizing functionof f at x¯ with respect to U and η}1For simplicity, we shall omit adjectives âpointwiseâ and âsetwiseâ whenever there isno ambiguity. We also remind readers that one can define similarly generalized concaveKL property around function values, which, however, will not be treated here.332.9. THE BREGMAN DISTANCEProof. This is [93, Lemma 4.4].2.9 The Bregman distanceIn this section, we introduce the Bregman distance, which measures thedifference between a function and its linear approximation, and record usefulproperties of the Bregman distance that will be used in the sequel.Definition 2.89. Let h : Rn â R be differentiable on int domh Ì¸= â . TheBregman distance of h is Dh : Rn Ã Rn â R defined byDh(x, y) ={h(x)â h(y)â â¨âh(y), xâ yâ©, if y â int domh;+â, otherwise. (2.21)Bregman distances can be generated through choosing suitable kernel h.Some popular choices of kernel h are listed below, and we refer the readersto [7, 23] and the references therein for more examples.Example 2.90.(i) Energy h(x) = 12x2 induces the squared Euclidean distance Dh(x, y) =12(xâ y)2 for x, y â R.(ii) The Boltzmann-Shannon entropy(âx â R) h(x) =ï£±ï£´ï£²ï£´ï£³x lnx, if x > 0;0, if x = 0;+â, otherwise,induces the Bregman distance(âx, y â R) Dh(x, y) =ï£±ï£´ï£²ï£´ï£³x (lnxâ ln y)â (xâ y), if x, y > 0;y, if x = 0, y > 0;+â, otherwise,(iii) Burgâs entropy(âx â R) h(x) ={â lnx, if x > 0;+â, otherwise,342.9. THE BREGMAN DISTANCEinduces the Bregman distance(âx, y â R) Dh(x, y) ={ln y â lnx+ xâyy , if x, y > 0;+â, otherwise.Although the above choices are one-dimensional, Bregman distances in Rncan be constructed through separable sums(âx, y â Rn) h(x) =nâi=1hi(xi) and Dh(x, y) =nâi=1Dhi(xi, yi).Fact 2.91 (linearity of the Bregman distance). Let h1, h2 : Rn â R bedifferentiable on an open set C â int domh1 â© int domh2. Then for every(x, y) â Rn Ã C and λ > 0,Dh1±h2(x, y) = Dh1(x, y)±Dh2(x, y) and Dλh1(x, y) = λDh1(x, y).Proof. Immediate by the definition (2.21).Fact 2.92 (three-point identity). Let h : Rn â R be differentiable onint domh Ì¸= â . For x â domh and y, z â int domh,Dh(x, z) = Dh(x, y) + DÏ(y, z) + â¨xâ y,âh(y)ââh(z)â©.Proof. See [32, Lemma 3.1].Now we record some useful properties of the Bregman distance, begin-ning with a useful identity.Lemma 2.93. Let h : Rn â R be a Legendre function. Then(âx, y â int domh) Dh(x, y) = Dhâ(âh(y),âh(x)).Proof. h is Legendre, then so is hâ and therefore hâ is differentiable onranâh. Applying Fact 2.37 to see hâ(âh(x)) = â¨x,âh(x)â© â h(x), whichjustifies the claimed identity after simple algebra.Unlike the Euclidean distance, special care is needed when the Bregmandistance between points vanishes.Lemma 2.94. Let h : Rn â R be a 1-coercive Legendre function. Then thefollowing hold:352.9. THE BREGMAN DISTANCE(i) Dh is continuous on int domhà int domh.(ii) If x â domh and y â int domh, then Dh(x, y) â¥ 0 with Dh(x, y) =0â x = y.(iii) If (xk)kâN and (yk)kâN are sequences in int domh converging to x âint domh and y â int domh respectively, then Dh(xk, yk) â 0 â x =y.(iv) If x â int domh and (yk)kâN is a sequence in int domh such thatDh(x, yk)â 0, then yk â x.Proof. See [10, Proposition 2.2].Lemma 2.95. Let h : Rn â R be a 1-coercive Legendre function withdomh = Rn. Then for bounded sequences (xk)kâN and (yk)kâN,Dh(xk, yk)â 0â xk â yk â 0.Proof. See [9, Example 4.10(ii)].Lemma 2.96. Let h : Rn â R be a 1-coercive Legendre kernel. If either his strongly convex or domh = Rn, then Dh(x, y) is level bounded in y locallyuniformly in x.Proof. It suffices to show for every compact set V â domh and α â RâxâV{y |Dh(x, y) â¤ α} is bounded. (2.22)Assume the opposite that there exist α â R, a compact set V, along withsequences (xk)kâN â V and (yk)kâN such thatDh(xk, yk) â¤ α but â¥ykâ¥ â â as k ââ.Consider the first case where h is strongly convex, say hâÏhj is convex.Thenα â¥ Dh(xk, yk) â¥ Ïh2 â¥xk â ykâ¥2 ââ as k ââ,which is absurd.362.9. THE BREGMAN DISTANCESuppose now that domh = Rn. Then it follows from Lemma 2.46 thathâ is 1-coercive. In turnα â¥ Dh(xk, yk) = h(xk)â h(yk)â â¨âh(yk), xk â ykâ©= h(xk) + hâ(âh(yk))â â¨âh(yk), xkâ©â¥ c+ hâ(âh(yk))â â¥âh(yk)â¥câ²= c+ â¥âh(yk)â¥(hâ(âh(yk))â¥âh(yk)â¥ â câ²),from which we deduce that (âh(yk))kâN must be bounded, otherwise theabove inequality would enforce α â¥ c+ â¥âh(yk)â¥(hâ(âh(yk))â¥âh(yk)â¥ â câ²)ââ ask â â, owing to the 1-coercivity of hâ. Since (âh(yk))kâN â int domhâ isbounded, we conclude that so is (yk)kâN by virtue of Lemma 2.46, complet-ing the proof.37Chapter 3Forward-reflected-backwardsplitting for nonconvexminimization problems3.1 IntroductionConsider the general inclusion problem of maximally monotone operators:find x â Rn such that 0 â A(x) +B(x), (3.1)where A : Rn â Rn and B : Rn â Rn are (maximally) monotone oper-ators with B being Lipschitz continuous. Successful splitting methods forthe inclusion problem (3.1) are numerous; see, e.g., [11, Chapters 26, 28],including the forward-backward method with B being cocoercive [35] andTsengâs method [88].In 2020, Malitsky and Tam [62] proposed a forward-reflected-backward(FRB) splitting method for solving (3.1). Let λ > 0 and JλA = (Id+Î»A)â1be the resolvent of operator λA; see, e.g., [11]. Given initial points x0, xâ1 âRn, the Malitsky-Tam FRB method with a fixed step-size (see [62, Remark2.1]) iteratesxk+1 = JλA(xk â 2λB(xk) + Î»B(xkâ1)), (3.2)with only one forward step computation required per iteration comparedto Tsengâs method. In [62] Malitsky and Tam showed that sequences xkgenerated by (3.2) converge to a solution of (3.1).In this chapter, our goal is to analyze the behavior of the Malitsky-TamFRB splitting method when being applied to the following structured noncon-vex minimization problem:minxâRnF (x) = f(x) + g(x), (P)where f : Rn â R is proper lower semicontinuous (lsc), and g : Rn â R hasa Lipschitz continuous gradient with constant L > 0. Our main results aresummarized below:383.2. A FORWARD-REFLECTED-BACKWARD SPLITTING METHODR1 In Theorem 3.2, without any additional assumption, we show that amerit function (Definition 3.1) associated with problem (P) decreasesalong the iterates generated by FRB, and the distance between con-secutive iterates vanishes eventually.R2 We show in Theorem 3.5 that all cluster points of the sequence gener-ated by FRB are stationary points of (P) and the merit function valuesconverge to the value of the objective function F at a stationary point,provided that the iterates remain bounded.R3 By assuming the merit function satisfies the generalized concave KLproperty, we prove in Theorem 3.8 that the sequence generated by FRBconverges globally to a stationary point of (P) and the gap betweenconsecutive iterates is summable.The rest of this chapter is organized as follows. The FRB scheme for (P)is introduced in Section 3.2. Basic properties of FRB are investigated inSection 3.3. In Section 3.4, we show that FRB converges globally under thegeneralized concave KL property. We analyze the convergence rates of FRBin Section 3.5 and end this chapter by presenting in Section 3.6 applicationsof FRB in solving nonconvex feasibility problem.3.2 A forward-reflected-backward splittingmethodIn this section, we introduce the FRB splitting method in the nonconvex set-ting. For convex minimization problems, the iteration scheme (3.2) specifiestoxk+1 = Pλf(yk â Î»âg(xk)),where yk = xk + Î»(âg(xkâ1) â âg(xk)) is an intermediate variable intro-duced for simplicity. Since proximal mapping Pλf might not be single-valuedwithout convexity of f , we formulate the FRB scheme for solving (P) as fol-lows:Algorithm 1 Forward-reflected-backward splitting method (FRB)Initialization: Pick xâ1, x0 â Rn and real number λ > 0.Iteration For k â N,1. Compute yk = xk + Î»(âg(xkâ1)ââg(xk)).2. Choose xk+1 â Pλf (yk â Î»âg(xk)).393.3. BASIC PROPERTIESSince the sequence generated by FRB starts from xâ1, we denote in therest of this chapterNâ = N âª {â1} .Moreover, it is easy to see thatPλf(yk â Î»âg(xk))=argminxâRn{f(x) + â¨xâ yk,âg(xk)â©+ 12λ â¥xâ ykâ¥2}. (3.3)3.3 Basic propertiesAfter formulating the iterative scheme, we now investigate its basic proper-ties. We begin with a merit function for the FRB splitting method, whichwill play a central role in our analysis.Definition 3.1 (FRB merit function). Let λ > 0. Define the FRB meritfunction H : Rn Ã Rn â R byH(x, y) = f(x) + g(x) +(14λ â L4) â¥xâ yâ¥2 . (3.4)We use the Euclidean norm for RnÃRn, i.e., â¥(x, y)â¥ =ââ¥xâ¥2 + â¥yâ¥2.Theorem 3.2. Let (xk)kâNâ be a sequence generated by FRB and definezk = (xk+1, xk) for k â Nâ. Assume that 0 < λ < min{13L , λf}and inf(f +g) > ââ. Let M1 = 14λ â 3L4 . Then the following hold:(i) For k â N, we haveM1 â¥zk â zkâ1â¥2 â¤ H(zkâ1)âH(zk), (3.5)which means that H(zk) â¤ H(zkâ1). Hence, the sequence(H(zk))kâNâis convergent.(ii)ââk=0â¥zk â zkâ1â¥2 <â. Consequently limkâââ¥zk â zkâ1â¥ = 0.Proof.(i) The sequence (xk)kâNâ is well-defined because λ < λf . The FRB iterativescheme and (3.3) imply thatf(xk+1) â¤ f(xk) + â¨âg(xk), xk â xk+1â©+ 12λ â¥xk â ykâ¥2 â 12λ â¥xk+1 â ykâ¥2 . (3.6)403.3. BASIC PROPERTIESInvoking the descent lemma (recall Fact 2.66) to g, we haveg(xk+1) â¤ g(xk) + â¨âg(xk), xk+1 â xkâ©+ L2 â¥xk+1 â xkâ¥2 . (3.7)Summing (3.6) and (3.7) yieldsf(xk+1) + g(xk+1) â¤ f(xk) + g(xk) + L2 â¥xk+1 â xkâ¥2 + 12λ â¥xk â ykâ¥2â 12λ â¥xk+1 â ykâ¥2= f(xk) + g(xk) +(L2 â 12λ) â¥xk+1 â xkâ¥2+ 1Î»â¨xk â xk+1, xk â ykâ©â¤ f(xk) + g(xk) +(L2 â 12λ) â¥xk+1 â xkâ¥2+ â¥xk â xk+1â¥ â¥âg(xkâ1)ââg(xk)â¥â¤ f(xk) + g(xk) +(Lâ 12λ) â¥xk+1 â xkâ¥2+ L2 â¥xk â xkâ1â¥2 ,where the first equality follows from the identity â¥xk â ykâ¥2 = â¥xk+1 â ykâ¥2ââ¥xk+1 â xkâ¥2 â 2â¨xk+1 â xk, xk â ykâ©, and the second inequality is impliedby Step 1 of FRB, and the Cauchy-Schwarz inequality. Moreover, thethird inequality holds by using the inequality a2 + b2 â¥ 2ab and the Lip-schitz continuity of âg. Recall the product norm identity â¥zk â zkâ1â¥2 =â¥xk+1 â xkâ¥2+â¥xk â xkâ1â¥2. It then follows from the inequality above thatf(xk+1) + g(xk+1)â¤ f(xk) + g(xk) +(Lâ 12Î»)(â¥zk â zkâ1â¥2 â â¥xk â xkâ1â¥2)+ L2 â¥xk â xkâ1â¥2= f(xk) + g(xk) +(Lâ 12λ) â¥zk â zkâ1â¥2 â (L2 â 12λ) â¥xk â xkâ1â¥2 .Adding(14λ â L4) â¥xk+1 â xkâ¥2 to both sides of the above inequality andapplying again the product norm identity, one getsH(zk) = f(xk+1) + g(xk+1) +(14λ â L4) â¥xk+1 â xkâ¥2â¤ f(xk) + g(xk) +(Lâ 12λ) â¥zk â zkâ1â¥2 â (L2 â 12λ) â¥xk â xkâ1â¥2+(14Î» â L4) â¥zk â zkâ1â¥2 â ( 14λ â L4 ) â¥xk â xkâ1â¥2= f(xk) + g(xk) +(14λ â L4) â¥xk â xkâ1â¥2+(3L4 â 14λ) â¥zk â zkâ1â¥2= H(zkâ1)âM1 â¥zk â zkâ1â¥2 ,which proves (3.5). Because M1 > 0, we have H(zk) â¤ H(zkâ1). Theconvergence result then follows from the assumption that inf(f + g) > ââ.413.3. BASIC PROPERTIES(ii) For integer l > 0, summing (3.5) from k = 0 to k = l â 1 yieldslâ1âk=0â¥zk â zkâ1â¥2 â¤Mâ11(H(zâ1)âH(zl)) â¤Mâ11 (H(zâ1)â inf(f + g)).Passing to the limit, one concludes thatââk=0â¥zk â zkâ1â¥2 <â.Next we estimate the lower bound on the gap between iterates. To thisend, a lemma helps.Lemma 3.3. For every (x, y) â Rn Ã Rn, we haveâH(x, y) ={âf(x) +âg(x) + ( 12λ â L2 ) (xâ y)}Ã {( 12λ â L2 ) (y â x)} .Proof. Apply Fact 2.15 and Fact 2.16.Lemma 3.4. Let (xk)kâNâ be a sequence generated by FRB and define zk =(xk+1, xk) for k â Nâ. Let M2 =â2(1Î» + L+â£â£ 1λ â Lâ£â£) and define for k â Npk+1 =1λ(xk â xk+1) +âg(xk+1)â 2âg(xk) +âg(xkâ1),Ak+1 = pk+1 +(12λ â L2)(xk+1 â xk), Bk+1 =(12λ â L2)(xk â xk+1).Then (Ak+1, Bk+1) â âH(zk) and â¥(Ak+1, Bk+1)â¥ â¤M2 â¥zk â zkâ1â¥.Proof. The FRB scheme and (3.3) imply that0 â âf(xk+1) +âg(xk) + 1λ(xk+1 â yk).Equivalently, 1Î»(xkâxk+1)+âg(xkâ1)â2âg(xk) â âf(xk+1), which togetherwith the subdifferential sum rule Fact 2.16 implies thatpk+1 â âf(xk+1) +âg(xk+1).It then follows from Lemma 3.3 with x = xk+1 and y = xk that(Ak+1, Bk+1) â âH(zk),andâ¥(Ak+1, Bk+1)â¥ â¤â¥â¥pk+1 + ( 12λ â L2 ) (xk+1 â xk)â¥â¥+â¥â¥( 12λ â L2)(xk â xk+1)â¥â¥â¤ â¥pk+1â¥+â£â£ 1λ â Lâ£â£ â¥xk+1 â xkâ¥â¤ ( 1λ + L+ â£â£ 1λ â Lâ£â£) â¥xk+1 â xkâ¥+ L â¥xk â xkâ1â¥â¤M2 â¥zk â zkâ1â¥ ,423.3. BASIC PROPERTIESwhere the third inequality follows from the Lipschitz continuity of âg withmodulus L.We now connect the properties of FRB merit function with the actualobjective of (P). Denote by Ï(zâ1) the set of cluster points of (zk)kâNâ .Theorem 3.5. Let (xk)kâNâ be a sequence generated by FRB and definezk = (xk+1, xk) for k â Nâ. Assume that conditions in Lemma 3.2 aresatisfied and (zk)kâNâ is bounded. Suppose that a subsequence (zkl)lâN of(zk)kâNâ converges to some zâ = (xâ, yâ) as l â â. Then the followinghold:(i) limlââH(zkl) = f(xâ) + g(xâ) = F (xâ). In fact, limkââH(zk) = F (xâ).(ii) xâ = yâ and 0 â âH(xâ, yâ), which implies 0 â âF (xâ) = âf(xâ) +âg(xâ).(iii) The set Ï(zâ1) is nonempty, compact and connected, on which theFRB merit function H is finite and constant. Moreover, we havelimkââdist(zk, Ï(zâ1)) = 0.Proof.(i) By the identity (3.3), for every k â Nâf(xk+1) + â¨xk+1 â yk,âg(xk)â©+ 12λ â¥xk+1 â ykâ¥2â¤ f(xâ) + â¨xâ â yk,âg(xk)â©+ 12λ â¥xâ â ykâ¥2 .Combining the identity â¥xâ â ykâ¥2 = â¥xâ â xk+1â¥2+ â¥xk+1 â ykâ¥2+2â¨xââxk+1, xk+1 â ykâ© with the above inequality and using the definition yk =xk + Î»(âg(xkâ1)ââg(xk)), one getsf(xk+1) â¤ f(xâ) + â¨xâ â xk+1,âg(xk)â©+ 12λ â¥xâ â xk+1â¥2+ 1Î»â¨xâ â xk+1, xk+1 â ykâ©â¤ f(xâ) + â¨xâ â xk+1, 2âg(xk)ââg(xkâ1)â©+ 12λ â¥xâ â xk+1â¥2+ 1Î»â¨xâ â xk+1, xk+1 â xkâ©.Replacing k by kl, passing to the limit and applying Lemma 3.2(ii), we havelim suplââf(xkl+1) â¤ f(xâ) â¤ lim inflââf(xkl+1),433.3. BASIC PROPERTIESwhere the last inequality is implied by the lower semicontinuity of f , whichimplies that limlââf(xkl+1) = f(xâ). Hence we conclude thatlimlââH(zkl) = limlââ(f(xkl+1) + g(xkl+1) +(14λ â L4) â¥xkl+1 â xklâ¥2)= f(xâ) + g(xâ),where the last equality holds because of Lemma 3.2(ii). Moreover, Theo-rem 3.2 states that the function value sequence(H(zk))kâNâ converges, fromwhich the rest of the statement readily follows.(ii) By assumption, we have xkl+1 â xâ and xkl â yâ as lââ. Therefore0 â¤ â¥yâ â xââ¥ â¤ â¥yâ â xklâ¥ + â¥xkl â xkl+1â¥ + â¥xkl+1 â xââ¥ â 0 as l â â,which means that xâ = yâ. Combining Lemma 3.2, Lemma 3.4, statement(i)and the outer semicontinuity of z 7â âH(z), one gets0 â âH(xâ, yâ) = âH(xâ, xâ) = {âf(xâ) +âg(xâ)}Ã {0},implying 0 â âf(xâ) +âg(xâ) as desired.(iii) Given that (zk)kâNâ is bounded, it is easy to see thatÏ(zâ1) is a nonempty compact set.By using Lemma 3.2 and a result by Ostrowski [11, Theorem 1.49], the setÏ(zâ1) is connected. Pick zË â Ï(zâ1) and suppose that zkq â zË as q ââ.Then statement(i) implies thatH(zË) = limqââH(zkq) = H(zâ).Finally, by the definition of Ï(zâ1), we have limkââdist(zk, Ï(zâ1)) = 0.Theorem 3.5 requires the sequence (zk)kâNâ to be bounded. The resultbelow provides a sufficient condition to such an assumption.Theorem 3.6. Let (xk)kâNâ be a sequence generated by FRB and definezk = (xk+1, xk) for k â Nâ. Assume that conditions in Lemma 3.2 aresatisfied. If f + g is coercive (or level bounded), then the sequence (xk)kâNâis bounded, so is (zk)kâNâ.Proof. Lemma 3.2(i) implies that we haveH(x1, x0) â¥ f(xk+1) + g(xk+1) +(14λ â L4) â¥xk+1 â xkâ¥2â¥ f(xk+1) + g(xk+1).Suppose that (xk)kâNâ was unbounded. Then we would have a contradictionby the coercivity or level boundedness of f + g.443.4. GLOBAL CONVERGENCE TO STATIONARY POINTS3.4 Global convergence to stationary pointsFollowing basic properties of the FRB method, we now present the mainconvergence result under the generalized concave KL property. For ε > 0and nonempty set â¦ â Rn, we define â¦Îµ = {x â Rn : dist(x,â¦) < ε}.Lemma 3.7. Let f : Rn â R be proper lsc and let µ â R. Let â¦ â dom âfbe a nonempty compact set on which f(x) = µ for all x â â¦. Then thefollowing hold:(i) Suppose that f satisfies the pointwise generalized concave KL propertyat each x â â¦. Then there exist ε > 0, η â (0,â] and Ï â Φη such thatf has the setwise generalized concave KL property on â¦ with respectto U = â¦Îµ, η and Ï.(ii) Let U = â¦Îµ and define h : (0, Î·)â R+ byh(s) = sup{distâ1(0, âf(x)): x â U â© [0 < f âµ < η], s â¤ f(x)âµ}.Then the function ÏË : [0, Î·)â R+,t 7ââ« t0h(s)ds, ât â (0, η),and ÏË(0) = 0, is well-defined and belongs to Φη. The function f hasthe setwise generalized concave KL property on â¦ with respect to U , ηand ÏË. Moreover,ÏË = inf{Ï â Î¦Î·â£â£â£â£â£ Ï is a concave desingularizing function of fon â¦ with respect to U and η}We say ÏË is the exact modulus of the setwise generalized concave KLproperty of f on â¦ with respect to U and η.Proof. See [93, Lemma 4.4].Theorem 3.8 (global convergence of FRB). Let (xk)kâNâ be a sequencegenerated by FRB and define zk = (xk+1, xk) for k â Nâ. Assume that(zk)kâNâ is bounded, inf(f + g) > ââ, and 0 < λ < min{13L , λf}. Supposethat the FRB merit function H has the generalized concave KL property onÏ(zâ1). Then the following hold:453.4. GLOBAL CONVERGENCE TO STATIONARY POINTS(i) The sequence (zk)kâNâ is Cauchy and has finite length. To be specific,there existM > 0, k0 â N, ε > 0 and η â (0,â] such that for i â¥ k0+1ââk=iâ¥zk+1 â zkâ¥ â¤ â¥zi â ziâ1â¥+MÏË (H(zi)âH(zâ)) . (3.8)where ÏË â Φη is the exact modulus associated with the setwise gen-eralized concave KL property of H on Ï(zâ1) with respect to ε andη.(ii) The sequence (xk)kâNâ has finite length and converges to some xâ with0 â âF (xâ).Proof.(i) By the boundedness assumption, assume without loss of generality thatzk â zâ = (xâ, yâ) for some zâ â Rn Ã Rn. Then Theorem 3.5 implies thatxâ = yâ and H(zk) â F (xâ) = H(zâ) as k â â. Recall from Lemma 3.2that we have H(zk+1) â¤ H(zk) for k â Nâ, therefore one needs to considertwo cases.Case 1: Suppose that there exists k0 such that H(zâ) = H(zk0). ThenLemma 3.2(i) implies that zk0+1 = zk0 and xk0+1 = xk0 . The desired resultsthen follows from a simple induction.Case 2: Assume that H(zâ) < H(zk) for every k. By assumption andLemma 3.7, there exist ε > 0 and η â (0,â] such that the FRB meritfunction H has the setwise generalized concave KL property on Ï(zâ1) withrespect to ε > 0 and η > 0 and the associated exact modulus ÏË. On onehand, by the fact that H(zk) â H(zâ), there exists k1 such that zk â [0 <H â H(zâ) < η] for k > k1. On the other hand, Theorem 3.5(iii) impliesthat there exists k2 such that dist(zk, Ï(zâ1)) < ε for all k > k2. Put k0 =max(k1, k2). Then for k > k0, we have zk â Ï(zâ1)Îµ â© [0 < H âH(zâ) < η].Hence for k > k0(ÏË)â²â (H(zk)âH(zâ)) · dist(0, âH(zk)) â¥ 1.Invoking Lemma 3.4 yieldsM2(ÏË)â²â (H(zk)âH(zâ)) â¥zk â zkâ1â¥ â¥ 1. (3.9)For simplicity, define for k > lâk,k+1 = ÏË (H(zk)âH(zâ))â ÏË (H(zk+1)âH(zâ)) .463.4. GLOBAL CONVERGENCE TO STATIONARY POINTSBy the concavity of ÏË and (3.9), one hasâk,k+1 = ÏË (H(zk)âH(zâ))â ÏË (H(zk+1)âH(zâ))â¥ (ÏË)â²â (H(zk)âH(zâ)) · [H(zk)âH(zk+1)]â¥ H(zk)âH(zk+1)M2â¥zkâzkâ1â¥ . (3.10)Applying Lemma 3.2 to (3.10) implies thatâk,k+1 â¥ â¥zk+1âzkâ¥2â¥zkâzkâ1â¥ Â·M1M2.Put M = M2M1 . Note that a2 + b2 â¥ 2ab for a, b â¥ 0. Then the aboveinequality gives2 â¥zk+1 â zkâ¥ â¤ 2âMâk,k+1 â¥zk â zkâ1â¥ â¤Mâk,k+1 + â¥zk â zkâ1â¥ .(3.11)Pick i â¥ k0 + 1. Summing (3.11) from i to an arbitrary j > i yields2jâk=iâ¥zk+1 â zkâ¥ â¤jâk=iâ¥zk â zkâ1â¥+Mjâk=iâk,k+1â¤jâk=iâ¥zk+1 â zkâ¥+ â¥zi â ziâ1â¥+MÏË (H(zi)âH(zâ))âMÏË (H(zj+1)âH(zâ))â¤jâk=iâ¥zk+1 â zkâ¥+ â¥zi â ziâ1â¥+MÏË (H(zi)âH(zâ)) ,where the second inequality is implied by the definition of âk,k+1, implyingjâk=iâ¥zk+1 â zkâ¥ â¤ â¥zi â ziâ1â¥+MÏË (H(zi)âH(zâ)) ,from which (3.8) readily follows. Moreover, one gets from the above inequal-ity that for i â¥ k0 + 1 and j > i,â¥zj â ziâ¥ â¤jâ1âk=iâ¥zk+1 â zkâ¥ â¤ â¥zi â ziâ1â¥+MÏË(H(zi)âH(zâ)).Recall from Theorem 3.5(i)&(ii) thatH(zi)â H(zâ) and from Lemma 3.2(ii)that â¥zi â ziâ1â¥ â 0 as i â â. Passing to the limit, one concludes that(zk)kâNâ is Cauchy.473.5. CONVERGENCE RATES(ii) The statement follows from the definition of (zk)kâNâ and Theorem 3.5(ii).Remark 3.9.(i) Note that iterates distance was shown to be only square-summable inthe original FRB paper [62, Theorem 2.5]. Therefore the finite lengthproperty is even new in the convex setting.(ii) Unlike the usual concave KL convergence analysis, our approach usesthe generalized concave KL property and the associated exact modulusto describe the sharpest upper bound ofââk=â1 â¥zk+1 â zkâ¥. To seethis, note that the usual analysis would yield a similar bound as (3.8)with ÏË replaced by a concave desingularizing function associated withthe concave KL property ofH. Lemma 3.7 states that ÏË is the infimumof all associated concave desingularizing functions. Hence the upperbound (3.8) is the sharpest; see also [93] for a similar sharp result ofthe celebrated PALM algorithm.A key assumption of Theorem 3.8 is the concave KL property of the FRBmerit function H on Ï(zâ1). The class of semialgebraic functions providesrich examples of functions satisfying such an assumption.Corollary 3.10. Let (xk)kâNâ be a sequence generated by FRB. Assume that(xk)kâNâ is bounded, inf(f + g) > ââ, and 0 < λ < min{13L , λf}. Sup-pose further that f and g are both semialgebraic functions. Then (xk)kâNâconverges to some xâ with 0 â âF (xâ) and has the finite length property.Proof. Recall from [3, Section 4.3] that the class of semialgebraic func-tions is closed under summation and notice that the quadratic function(x, y) 7â ( 14λ â L4 ) â¥xâ yâ¥2 is semialgebraic. Then Fact 2.80 implies that theFRB merit function H is concave KL. Applying Theorem 3.8 then completesthe proof.3.5 Convergence ratesAssuming that the FRB merit function admits KL exponent θ â [0, 1), weestablish the following convergence rate result. Our analysis is standardand follows from the usual KL convergence rate methodology; see, e.g., [2,Theorem 2], [22, Remark 6] and [6, Lemma 4]. We provide a proof here forthe sake of completeness.483.5. CONVERGENCE RATESTheorem 3.11 (convergence rate of FRB). Let (xk)kâNâ be a sequencegenerated by FRB and define zk = (xk+1, xk) for k â Nâ. Assume that(zk)kâNâ is bounded, inf(f + g) > ââ, and 0 < λ < min{13L , λf}. LetM,k0 and xâ be those given in Theorem 3.8. Suppose that the FRB meritfunction H has KL exponent θ â [0, 1) at (xâ, xâ). Then the following hold:(i) If θ = 0, then (xk)kâNâ converges to xâ in finite steps.(ii) If θ â (0, 12], then there exist Q1 â (0, 1) and c1 > 0 such thatâ¥xk â xââ¥ â¤ c1Qk1,for k sufficiently large.(iii) If θ â (12 , 1), then there exists Q2 > 0 such thatâ¥xk â xââ¥ â¤ Q2kâ1âÎ¸2Î¸â1 ,for k sufficiently large.Proof. Let zâ = (xâ, xâ). Then Theorem 3.8 shows that zk â zâ as k ââ. Assume without loss of generality that H(zâ) = 0 and H(zk) > 0 forall k. Before proving the desired statements, we will first develop severalinequalities which will be used later. We have shown in Theorem 3.8 thatfor k > k0,1 â¤ (ÏË)â²â (H(zk)) · dist(0, âH(zk)) â¤ Ïâ²â (H(zk)) · dist(0, âH(zk)),which by Lemma 3.4 and our assumption further implies that1 â¤ (1â θ)c (H(zk))âÎ¸ â¥(Ak+1, Bk+1)â¥â¤ (1â θ)cM2 (H(zk))âÎ¸ â¥zk â zkâ1â¥ . (3.12)Furthermore, define for k â NÏk =ââi=kâ¥zi+1 â ziâ¥ ,which is well-defined due to Theorem 3.8. Assume again without loss ofgenerality that Ïkâ1âÏk = â¥zk â zkâ1â¥ < 1 for every k (recall Lemma 3.2).Rearranging (3.12) givesH(zk)1âÎ¸ â¤ [(1â Î¸)cM2]1âÎ¸Î¸ (Ïkâ1 â Ïk)1âÎ¸Î¸ . (3.13)493.5. CONVERGENCE RATESBy (3.8), we have for k > k0Ïk â¤ Ïkâ1 â Ïk + cM(H(zk))1âÎ¸ â¤ Ïkâ1 â Ïk + C(Ïkâ1 â Ïk)1âÎ¸Î¸ , (3.14)where the last inequality follows from (3.13) and C = cM((1 â θ)cM2) 1âÎ¸Î¸ .Clearly â¥xk â xââ¥ â¤ â¥zk â zââ¥ â¤ Ïk. Hence, in order to prove the desiredstatements, it suffices to estimate Ïk.(i) Let θ = 0 and suppose that (zk)kâNâ converges in infinitely many steps.Then (3.12) means that for every k > k01 â¤ cM2 â¥zk â zkâ1â¥ .Passing to the limit and applying Lemma 3.2, one gets 1 â¤ 0, which isabsurd.(ii) If θ â (0, 12], then 1âÎ¸Î¸ â¥ 1 and (Ïkâ1âÏk) 1âÎ¸Î¸ â¤ Ïkâ1âÏk. Hence (3.14)implies that for k > k0Ïk â¤ (C + 1)(Ïkâ1 â Ïk)â Ïk â¤ C+1C+2Ïkâ1 â¤(C+1C+2)kâk0Ïk0 ,from which the desired result readily follows by setting Q1 =C+1C+2 andc1 =(C+1C+2)âk0Ïk0 .(iii) If 12 < θ < 1, then 0 <1âÎ¸Î¸ < 1 and (Ïkâ1 â Ïk)1âÎ¸Î¸ â¥ Ïkâ1 â Ïk. Itfollows from (3.14) that Ïk â¤ (1 + C) (Ïkâ1 â Ïk) 1âÎ¸Î¸ . Define h(t) = tâÎ¸1âÎ¸for t > 0. Then1 â¤ (1 + C) Î¸1âÎ¸ (Ïkâ1 â Ïk)h(Ïk). (3.15)Let R > 1 be a fixed real number. Next we consider two cases for k > k0.If h(Ïk) â¤ Rh(Ïkâ1) then (3.15) implies that1 â¤ R(1 + C) Î¸1âÎ¸ (Ïkâ1 â Ïk)h(Ïkâ1) â¤ R(1 + C)Î¸1âÎ¸â« Ïkâ1Ïkh(t)dt= R(1 + C)Î¸1âÎ¸1â Î¸1â 2Î¸(Ï1â2Î¸1âÎ¸kâ1 â Ï1â2Î¸1âÎ¸k).Set v = 1â2Î¸1âÎ¸ < 0. Then the above inequality can be rewritten asÏvk â Ïvkâ1 â¥ âvR(1 + C)âÎ¸1âÎ¸ . (3.16)503.5. CONVERGENCE RATESIf h(Ïk) > Rh(Ïkâ1) then Ïk <(1R) 1âÎ¸Î¸ Ïkâ1 = qÏkâ1, where q =(1R) 1âÎ¸Î¸ â(0, 1). HenceÏk < qÏkâ1 â Ïvk > qvÏvkâ1 â Ïvk â Ïvkâ1 > (qv â 1)Ïvkâ1 > (qv â 1)Ïvk0 .(3.17)Set c2 = min{(qvâ1)Ïvk0 ,â vR(1+C)â Î¸1âÎ¸}> 0. Combining (3.16) and (3.17)yieldsÏvk â Ïvkâ1 â¥ c2, âk > k0.Summing the above inequality from k0 + 1 to any k > k0, one getsÏvk â Ïvk0 =kâi=k0+1(Ïvi â Ïviâ1) â¥ (k â k0)c2 â Ïvk > kk0+1c2,where the last inequality holds because kâ k0 â¥ kk0+1 for k > k0+1. HenceÏk < k1v(C2k0 + 1) 1v,from which the desired statement follows by setting Q2 =(c2k0+1) 1v.Corollary 3.12. Let (xk)kâNâ be a sequence generated by FRB and supposethat all conditions of Theorem 3.8 are satisfied. Let xâ be as in Theorem 3.8.Assume further that F has KL exponent θ â [12 , 1) at xâ. Then the followinghold:(i) If θ = 12 , then there exist Q1 â (0, 1) and c1 > 0 such thatâ¥xk â xââ¥ â¤ c1Qk1for k sufficiently large.(ii) If θ â (12 , 1), then there exists Q2 > 0 such thatâ¥xk â xââ¥ â¤ Q2kâ1âÎ¸2Î¸â1 ,for k sufficiently large.Proof. The FRB merit function H has KL exponent θ at (xâ, xâ) by ourassumption and [54, Theorem 3.6]. Applying Theorem 3.11 completes theproof.Following similar techniques in [26,27], we now deduce convergence ratesof the merit function value sequence.513.5. CONVERGENCE RATESLemma 3.13. Let (ek)kâN be a monotonically decreasing sequence in R++converging to 0. Assume further that there exist natural numbers k0 â¥ l0 â¥ 1such that for every k â¥ k0ekâl0 â ek â¥ Ce2θk ,where C > 0 is a constant and θ â [0, 1). Then the following hold:(i) If θ = 0, then ek â 0 in finite iterations.(ii) If 0 < θ â¤ 1/2, then there exist C1 > 0 and 0 â¤ Q < 1 such that forevery k â¥ k00 < ek â¤ C1Qk.(iii) If 1/2 < θ < 1, then there exists C2 > 0 such that for every k â¥ k0+l0,0 â¤ ek â¤ C2(k â l0 + 1)â12Î¸â1 .Proof. See [27, Lemma 10].Theorem 3.14 (convergence rate of merit function value). Let (xk)kâNâ be asequence generated by FRB and define zk = (xk+1, xk) for k â Nâ. Assumethat (zk)kâNâ is bounded, inf(f + g) > ââ, and 0 < λ < min{13L , λf}.Let xâ be the limit given in Theorem 3.8(ii). Suppose that the FRB meritfunction H has KL exponent θ â [0, 1) at zâ = (xâ, xâ). Define ek = H(zk)âF (xâ) for k â N. Then (ek)kâN converges to 0 and the following hold:(i) If θ = 0, then (ek)kâN converges in finite steps.(ii) If θ â (0, 12], then there exist cË1 > 0 and QË1 â [0, 1) such that for ksufficiently large,ek â¤ cË1QËk1.(iii) If θ â (12 , 1), then there exists cË2 > 0 such that for k sufficiently large,ek â¤ cË2kâ12Î¸â1 .Proof. We learn from Theorems 3.5 and 3.8 that the sequence (H(zk))kâNis decreasing, H(zk) â H(zâ) = F (xâ) and zk â zâ as k â â. Therefore(ek)kâN converges to 0 and ek â¥ 0 for all k. By the KL exponent assumption,there exist c > 0 and k0 such that for k â¥ k0dist(0, âH(zk)) â¥ (H(zk)âH(zâ))Î¸c(1âÎ¸) = eÎ¸kc(1âÎ¸) .523.6. AN APPLICATION TO NONCONVEX FEASIBILITY PROBLEMSTherefore by using Lemmas 3.2 and 3.4ekâ1 â ek = H(zkâ1)âH(zk) â¥M1 â¥zk â zkâ1â¥2â¥ M1M22dist(0, âH(zk))2 â¥ Ce2θk ,where C = M1c2(1âÎ¸)2M22. Applying Lemma 3.13 completes the proof.Corollary 3.15 (convergence rate of objective function value). Let (xk)kâNâbe a sequence generated by FRB and define zk = (xk+1, xk) for k â Nâ. As-sume that (zk)kâNâ is bounded, inf(f+g) > ââ, and 0 < λ < min{13L , λf}.Let xâ be the limit given in Theorem 3.8(ii). Suppose that the FRB meritfunction H has KL exponent θ â [0, 1) at zâ = (xâ, xâ). Then the followinghold:(i) If θ = 0, then F (xk) converges to F (xâ) in finite steps.(ii) If θ â (0, 12], then there exit c¯1 > 0 and Q¯1 â [0, 1) such that for ksufficiently large|F (xk)â F (xâ)| â¤ c¯1Q¯k1.(iii) If θ â (12 , 1), then there exists c¯2 > 0 such that for k sufficiently large,|F (xk)â F (xâ)| â¤ c¯2kâ1âÎ¸2Î¸â1 .Proof. Observe that|F (xk)â F (xâ)| â¤ |H(xk, xkâ1)â F (xâ)|+â£â£ 14λ â L4â£â£ â¥xk â xkâ1â¥2â¤ |H(xk, xkâ1)â F (xâ)|+â£â£ 12λ â L2â£â£ (â¥xk â xââ¥2 + â¥xkâ1 â xââ¥2).It suffices to apply Theorems 3.11 and 3.14.3.6 An application to nonconvex feasibilityproblems3.6.1 The problemLet ProjC : Rn â Rn be the projection mapping associated with a nonemptyclosed subset C â Rn defined by(âx â Rn) ProjC(x) = argminwâCâ¥xâ wâ¥ . (3.18)533.6. AN APPLICATION TO NONCONVEX FEASIBILITY PROBLEMSWe now turn to an application of FRB by considering the problem offinding x â C â©D,where C,D â Rn are nonempty sets. The above problem can be solved byFRB by a reformulation:minxâRnF (x) = 12 dist2(x,C) + δD(x), (Pâ)in which case FRB specifies to iteratingxk+1 â ProjC((1â λ)yk + λProjC(xk)), (3.19)where yk = xk + Î»(xkâ1 â xk)â Î»(ProjC(xkâ1 â xk)).Corollary 3.16. Consider the problem (Pâ), where C â Rn is a nonempty,closed and convex set and D â Rn is nonempty and closed. Suppose thatC â© D Ì¸= â and at least one of C and D is compact. Assume further thatboth C and D are semialgebraic.Let 0 < λ < 13 and let (xk) be a sequence generated by (3.19). Then thefollowing hold:(i) There exists xâ â Rn such that xk â xâ and 0 â âF (xâ).(ii) Suppose, in addition, thatNC(ProjC(xâ)) â© (âND(xâ)) = {0}. (3.20)Then xâ â C â©D. Moreover, there exist Q1 â (0, 1) and c1 > 0 suchthatâ¥xk â xââ¥ â¤ c1Qk1,for k sufficiently large.Proof.(i) By the compactness assumption, the function f + g is coercive. HenceTheorem 3.6 implies that (xk)kâNâ is bounded. We assume that sets C,Dare semialgebraic, then so are functions f and g; see [3, Section 4.3] and [4,Lemma 2.3]. Moreover, note that λf =â and g admits a Lipschitz contin-uous gradient with constant L = 1; see [11, Corollary 12.31]. The desiredresult then follows from Corollary 3.10.543.6. AN APPLICATION TO NONCONVEX FEASIBILITY PROBLEMS(ii) Taking the fact that 0 â âF (xâ) into account and applying the subdif-ferential sum rule Fact 2.16, one getsxâ â ProjC(xâ) â âND(xâ).The constraint qualification (3.20) then implies that xâ = ProjC(xâ) andconsequently xâ â C. Moreover, the FRB scheme together with the closed-ness of D ensures that xâ â D, hence xâ â C â© D. Consequently, theconstraint qualification (3.20) amounts toNC(xâ) â© (âND(xâ)) = {0}.Then a direct application of [31, Theorem 5] guarantees that F has KLexponent θ = 12 at xâ. The desired result follows immediately from Corol-lary 3.12.In the remainder of this chapter, letC = {x â Rn |Ax = b} and D = {x â Rn |â¥xâ¥0 â¤ r, â¥xâ¥â â¤ R} ,for A â RmÃn and b â Rm and some r,R > 0, which are semialgebraic owingto Example 2.81. Then ProjC(x) = (IdâAâ A)x + Aâ b for every x â Rn,where Aâ denotes the Moore-Penrose inverse of A; see [11, Example 29.17].Then the FRB scheme (3.19) further specifies toxk+1 â ProjD(xk â Î»Aâ A(2xk â xkâ1) + Î»Aâ b), (3.21)where the projection ProjD can be computed by the following,Proposition 3.17. Let z = (z1, . . . , zn) â Rn. For every i, set zËâi =Proj[âR,R](zi), vâi = |zi|2 â |zËâi â zi|2, and let Iâ â {1, . . . , n} be the setof indices corresponding to the r largest elements of vâi , i = 1, . . . , n. Definezâ â Rn by zâi = zËâi if i â Iâ and zâi = 0 otherwise. Then zâ â ProjD(z).Proof. Apply [58, Proposition 3.1] with Ïi = | · âzi|2 and Xi = [âl, l].3.6.2 Numerical experimentsWe shall benchmark FRB against several algorithms for solving (Pâ) withC = {x â Rn |Ax = b} and D = {x â Rn |â¥xâ¥0 â¤ r, â¥xâ¥â â¤ R} .553.6. AN APPLICATION TO NONCONVEX FEASIBILITY PROBLEMSSimilar to [53, Section 5], our problem data is generated through creatingrandom matrices A â RmÃn with entries following the standard Gaussiandistribution. Then we generate a vector xË â Rr randomly with the samedistribution, project it onto the box [âR,R]r, and create a sparse vectorxË â Rn whose r entries are chosen randomly to be the respective valuesof the projection of xË onto [âR,R]r. Finally, we set b = AxË to guaranteeC â©D Ì¸= â .Targets We will compare FRB with the Douglas-Rachford method withfixed step-size (DR) [53] by Li and Pong, inertial Tsengâs method (iT-seng) [24] by Bot¸ and Csetnek, and DR equipped with step-size heuris-tics (DRh) [53] by Li and Pong. These splitting algorithms for nonconvexoptimization problems are known to converge globally to a stationary pointof (P) under the same assumptions considered in Corollary 3.16; see [53, The-orems 1â2, Remark 4, Corollary 1] and [24, Theorem 3.1]. The convergenceof DR and iTseng in our setting are already proved in [53, Proposition 2]and [24, Corollary 3.1], respectively.Oracles We implement FRB with the following specified scheme for theproblem of finding an r-sparse solution of a linear system {x â Rn |Ax = b}xk+1 â ProjD(xk â Î»Aâ A(2xk â xkâ1) + Î»Aâ b)where the step-size λ = 0.9999 · 13 (recall Example 3.16). The inertial typeTsengâs method studied in [24, Scheme (6)] is applied with a step-size Î»â² =0.1316 given by [24, Lemma 3.3] and a fixed inertial term α = 18 :pk+1 â ProjD(xk â Î»â²Aâ (Axk â b) + α(xk â xkâ1)),xk+1 = pk + Î»â²Aâ A(xk â pk).As for DR and DRh, we employ the schemes specified by [53, Scheme (7)]and [53, Section 5] with the exact same step-sizes, respectively.Termination condition All algorithms are initialized at the origin, andwe terminate FRB and iTseng whenmax{ â¥xk+1 â xkâ¥ , â¥xk â xkâ1â¥}max{1, â¥xk+1â¥ , â¥xkâ¥ , â¥xkâ1â¥} < 10â8.We adopt the termination criteria from [53, Section 5] for DR and DRh,where the same tolerance of 10â8 is applied.563.7. CONCLUSIONSize FRB DR iTseng DRhm n iter fvalmin succ iter fvalmin succ iter fvalmin succ iter fvalmin succ300 600 290 5.78001e-14 49 476 1.63979e-13 43 724 3.93939e-13 23 436 1.69111e-30 50300 700 381 7.99399e-14 48 601 1.90854e-13 36 866 4.72653e-13 10 444 1.25125e-30 50300 800 473 1.00221e-13 42 743 2.17886e-13 22 1054 7.52841e-13 7 448 8.06062e-31 50300 900 577 1.20731e-13 34 890 2.57981e-13 21 1226 7.01903e-13 5 453 8.60352e-31 50300 1000 662 1.35447e-13 31 954 3.0211e-13 11 1326 9.27902e-13 1 459 8.77713e-31 50400 600 177 4.89578e-14 50 271 1.12518e-13 50 461 3.09084e-13 45 427 5.72429e-30 50400 700 230 7.51594e-14 50 365 1.74449e-13 50 628 4.5344e-13 35 433 3.50651e-30 50400 800 294 8.46721e-14 50 487 2.0469e-13 47 810 5.37112e-13 26 441 2.05666e-30 50400 900 359 1.03652e-13 49 589 2.39422e-13 41 910 7.90857e-13 18 443 2.05578e-30 50400 1000 431 1.00662e-13 47 712 2.40475e-13 35 1064 7.66054e-13 8 447 1.55654e-30 50500 600 116 4.46448e-14 50 169 8.31312e-14 50 267 2.6074e-13 50 369 2.70253e-29 50500 700 158 5.70864e-14 50 239 1.33359e-13 50 407 3.81561e-13 50 419 1.11142e-29 50500 800 198 8.82954e-14 50 308 1.9478e-13 50 545 5.10359e-13 46 431 7.31337e-30 50500 900 246 9.45053e-14 50 399 2.12284e-13 50 700 5.67092e-13 36 434 5.51452e-30 50500 1000 294 1.25389e-13 50 478 2.76383e-13 47 847 6.55755e-13 17 436 4.44226e-30 50Table 3.1: Comparing FRB to DR, iTseng, and DRh.Results Results of our experiments are presented in Table 3.1. For eachproblem of the size (m,n), we randomly generate 50 instances using thestrategy described above, and report ceilings of the average number of iter-ations (iter), the minimal objective value at termination (fvalmin), and thenumber of successes (succ). Here we say an experiment is âsuccessfulâ, if theobjective function value at termination is less than 10â12, which means thatthe algorithm actually hits a global minimizer rather than just a stationarypoint. We observed the following:- Among algorithms without heuristics (FRB, DR, and iTseng), FRBoutperforms the others in terms of both the number of iterations andsuccesses, and it also has the smallest termination values. Moreover,itâs worth noting that our simulation results align with Malitsky andTamâs observation that FRB converges faster than Tsengâs method ona specific problem [62, Remark 2.8].- DRh has the most number of successes and the best precision at termi-nation (see fvalmin), but tend to be slower on âeasyâ problems (largem).3.7 ConclusionWe established convergence of the Malitsky-Tam FRB splitting algorithmin the nonconvex setting. Under the generalized concave KL property, weshowed that FRB converges globally to a stationary point of (P) and ad-mits finite length property, which is even new in the convex setting. The573.7. CONCLUSIONsharpness of our results is demonstrated by virtue of the exact modulus as-sociated with the generalized concave KL property. We also analyzed bothfunction value and sequential convergence rates of FRB when desingulariz-ing functions associated with the generalized concave KL property have the Lojasiewicz form. Numerical simulations suggest that FRB is a competitivecompared to DR and inertial Tsengâs methods for the problem of findingsparse solutions of linear systems.58Chapter 4A Bregman inertialforward-reflected-backwardmethod for nonconvexminimization4.1 IntroductionConsider the following structured nonconvex problem:minxâRnF (x) = f(x) + g(x), (P)where f : Rn â R is proper lower semicontinuous (lsc), and g : Rn â R hasa Lipschitz continuous gradient with constant L. A standard practice whenanalyzing splitting algorithms for (P) is identifying a merit function of thefollowing formH(x, y) = F (x) + p â¥xâ yâ¥2 for some p â¥ 0. (4.1)See for instance Chapter 3, and [22, 24, 25, 27, 51, 53, 70] and the referencestherein. The requirement p â¥ 0 in (4.1), however, may lead to unpleasantconsequences on algorithm parameters. For example:â¢ The inertial proximal algorithm investigated in [70, Algorithm 5] forsolving (P) has a merit function at the kth iteration (in our notation)Hk(x, y) = F (x) +(1Î»kâ L2 â Î±k2Î»k)â¥xâ yâ¥2 ,where λk > 0 represents the stepsize and αk > 0 is the inertial param-eter at the kth iteration, leading to a restriction on stepsize λ:λk <2(1âÎ±k)L ,see [70, Lemma 4.6]. The above means, the stepsize λk has to be smallwhen the inertial parameter αk is large; and vice versa.594.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCEâ¢ There are only existence results for parameters of inertial Tsengâsmethod [24] for solving (P); see [24, Lemma 3.3].The issues described above are resulted from accepting a fuzzy estimationon parameter p in (4.1) for the convenience of analysis, which neverthelessenforces restrictive conditions on implementing algorithm.This chapter presents a novel methodology to analyze splitting algorithmsfor nonconvex problems by working with a sequence of recursively definedmerit functions of the form (4.1). As a case study, we investigate a Bregmaninertial forward-reflected-backward method (BiFRB) for solving the noncon-vex composite problem (P). Informally, the proposed algorithm computesxk+1 â (âh+Î»kâf)â1(xkâ2Î»kâg(xk)+Î»kâg(xkâ1)+Î±k(xkâxkâ1)), (4.2)for some stepsize λk > 0, inertial parameter αk â [0, 1) and kernel h : Rn âR. In contrast to the literature; see, e.g., [24,70] and the references therein,our methodology untangles the dependence between stepsize and inertialparameter. Our main results are listed below:R1 We obtain an abstract function values convergence result in Theo-rem 4.12 based on Assumption 4.10.R2 When in addition the problem satisfies the KL property, we show thatthe proposed algorithm converges globally to stationary points of (P);see Theorem 4.17.R3 The sought independence between the stepsize and the inertial param-eter is obtained in Proposition 4.20.The rest of this chapter is organized as following: The proposed algorithmand its convergence are studied in Section 4.2 under an abstract assump-tion on algorithm parameters; see Assumption 4.10. Then, in Section 4.3,we present suitable stepsize rules under which the critical Assumption 4.10holds. Subproblem formulae for our algorithm BiFRB are obtained in Sec-tion 4.4.4.2 The BiFRB method and its abstractconvergence4.2.1 The BiFRB methodIn the remainder of this chapter, suppose that the following hold:604.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCEAssumption 4.1.(i) f : Rn â R is proper, lsc and prox-bounded with threshold λf > 0.(ii) g : Rn â R has Lipschitz continuous gradient with constant Lâg.(iii) h : Rn â R is Ï-strongly convex and has Lipschitz continuous gradientwith constant Lâh.(iv) The objective F = f + g is bounded below.The Bregman distance of h : Rn â R is defined to beDh(x, y) = h(x)â h(y)â â¨âh(y), xâ yâ©.The Bregman distance Dh is proportional to square of the Euclideandistance under Assumption 4.1(iii). Such assumption has been used in theliterature to obtain non-Euclidean extension of splitting algorithms; see,e.g., [24, 25].Lemma 4.2. Let h : Rn â R be Ï-strongly convex and have Lipschitzcontinuous gradient with constant Lâh. Then(âx, y â Rn) Ï2 â¥xâ yâ¥2 â¤ Dh(x, y) â¤ Lâh2 â¥xâ yâ¥2 .Proof. See [11, Exercise 17.5, Theorem 18.15].We now propose the Bregman inertial forward-reflected-backward algo-rithm.Algorithm 2 Bregman inertial forward-reflected-backward (BiFRB)Initialization: Pick xâ1, x0 â Rn. Let 0 < λ â¤ λ â¤ λf . Chooseλ â¤ Î»â1 â¤ λ.Iteration: For k â N, choose λ â¤ λk â¤ λ and 0 â¤ αk < 1.1. Compute yk = xk + Î»kâ1(âg(xkâ1)ââg(xk)).2. Choose xk+1 â argminxâRn{f(x) + â¨x â yk,âg(xk) + αkλk (xkâ1 âxk)â©+ 1λkDh(x, yk)}.Remark 4.3. Some comments are in order.614.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCE(i) When the kernel h = â¥Â·â¥2 /2, BiFRB reduces to the inertial forward-reflected-backward (iFRB) method investigated in [62] in the convexsetting. The convergence of iFRB in nonconvex setting, will be coveredby our analysis in the sequel.(ii) When the kernel h = â¥Â·â¥2 /2, αk = 0, and (Î»k)kâN is a constant,BiFRB reduces to FRB investigated in Chapter 3.Recall that the Fenchel conjugate of f : Rn â R is fâ = supyâRn{â¨Â·, yâ©âf(y)} and the Moreau envelope of f : Rn â R with parameter λ > 0 is(âx â Rn) eλf(x) = infyâRn{f(y) + 12λ â¥xâ yâ¥2}.Below is a systemic way to construct kernel h satisfying Assumption 4.1(iii),via Fenchel conjugate and Moreau envelope.Proposition 4.4. Let L, Ï > 0 and let p : Rn â R be proper, lsc andconvex. Define j = â¥Â·â¥2 /2. Then the following hold:(i) p has L-Lipschitz gradient if and only if p is the Moreau envelope withparameter λ = 1/L of a proper, lsc and convex function (pâ â Î»j)â.(ii) If p has L-Lipschitz gradient, then h = p + Ïj is Ï-strongly convexwith (L+ Ï)-Lipschitz gradient.Proof. See [11, Corollary 18.19].Example 4.5. Let h(x) =â1 + â¥xâ¥2 + â¥xâ¥2 /2. Then h satisfies Assump-tion 4.1(iii) with Ï = 1 and Lâh = 2.Proof. Define (âx â Rn) p(x) =â1 + â¥xâ¥2. Then pâ is 1-strongly convex;see, e.g., [14, Section 4.4, Example 5.29]. Applying Proposition 4.4 com-pletes the proof.We also assume the following throughout the chapter, under which theproposed algorithm is well-defined.Assumption 4.6. Suppose that one of the following holds:(i) The kernel h has strong convexity modulus Ï â¥ 1.(ii) The function f has prox-threshold λf =â.624.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCERemark 4.7. The threshold λf =â when f is convex or bounded below;see Proposition 2.53.Proposition 4.8. Let u, v â Rn and suppose that Assumption 4.6 holds.Thenfor 0 < λ < λf , the function x 7â f(x) + â¨xâ u, vâ©+Dh(x, u)/λ is coercive.Consequently, the setargminxâRn{f(x) + â¨xâ u, vâ©+ 1λDh(x, u)}Ì¸= â .Proof. We claim that x 7â f(x) + â¨x, Ïâ© is prox-bounded with λf for anarbitrary Ï â Rn. Indeed,lim infâ¥xâ¥ââf(x)+â¨x,Ïâ©â¥xâ¥2 = lim infâ¥xâ¥ââf(x)â¥xâ¥2 + limâ¥xâ¥âââ¨x,Ïâ©â¥xâ¥2 = lim infâ¥xâ¥ââf(x)â¥xâ¥2 .Thus invoking Proposition 2.53 proves the claim.By our claim, x 7â f(x) + â¨x, v â Ïu/Î»â© is prox-bounded with thresholdλf . Let Î»â² be such that λ < Î»â² < λf . Then Î»â²/Ï < λf under Assumption 4.6.Thereforef(x) + â¨xâ u, vâ©+ 1λDh(x, u)â¥ f(x) + â¨xâ u, vâ©+ Ï2Î» â¥xâ uâ¥2 â Ï2Î»â² â¥xâ¥2 + Ï2Î»â² â¥xâ¥2â¥ Ï2(1Î» â 1Î»â²) â¥xâ¥2 + f(x) + â¨x, v â ÏÎ»uâ©+ Ï2Î»â² â¥xâ¥2 â â¨u, vâ©â¥ Ï2(1Î» â 1Î»â²) â¥xâ¥2 + infxâRn{f(x) + â¨x, v â ÏÎ»uâ©+ Ï2Î»â² â¥xâ¥2}â â¨u, vâ© â â,as â¥xâ¥ â â, where the infimum in the last inequality is finite thanks to prox-boundedness of x 7â f(x) + â¨x, v â Ïu/Î»â©, which completes the proof.4.2.2 BiFRB merit function and its parameter rulesIn the remainder of this chapter, unless specified otherwise, let pâ1 â R anddefine for k â Npk =((LâhâÏ)L2âg2)Î»2kâ1Î»k+ 12(ÏÎ»kâ Lâg)â pkâ1, (4.3)M1,k = pkâ1 â Î±k+ÏLâgÎ»kâ12Î»k â(LâhâÏ)L2âgÎ»2kâ12Î»k, (4.4)where (Î»k)kâNâ is a stepsize sequence of BiFRB; (Î±k)kâN is a sequence ofinertial parameters; and Lâh, Lâg, Ï are parameters specified in Assump-tion 4.1. The following concept plays a central role in our analysis.634.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCEDefinition 4.9. Let p â R. Define Hp : Rn Ã Rn â R by(â(x, y) â Rn Ã Rn) Hp(x, y) = F (x) + p â¥xâ yâ¥2 .Then we say that Hp is a quadratic merit function with parameter p. Let(pk)kâNâ be given by (4.3). Then Hpk is the kth BiFRB merit function. Ifpk â¤ p¯ for some p¯ > 0, then Hp¯ is a dominating BiFRB merit function.Despite the recursion (4.3), we shall see soon that (pk)kâNâ and (M1,k)kâNâare indeed implicit in our analysis. Imposing the following conditions on(pk)kâNâ and (M1,k)kâNâ , one gets abstract convergence of BiFRB which inturn yields less restrictive stepsize rules; see Section 4.3 and in particularRemark 4.21.Assumption 4.10 (general rules of BiFRB merit function parameter). Let(pk)kâNâ and (M1,k)kâN be given as in (4.3) and (4.4). Suppose that thefollowing hold:(i) lim infkââ pk â¥ 0 and lim infkââM1,k > 0.(ii) There exists p¯ such that pk â¤ p¯ for all k.Remark 4.11. Assumption 4.10 generalizes quadratic merit function prop-erties in the present literature; see, e.g., [24, Lemma 3.2], [25, Lemma 6], [69,Assumption (H1)] and [70, Algorithm 5, Lemma 4.6, Proposition 4.7].4.2.3 Abstract function value convergenceIn this section, we study function value convergence of BiFRB under As-sumption 4.10; see Section 4.3 for concrete stepsize rules under which As-sumption 4.10 holds. Since the sequence generated by BiFRB starts fromxâ1, we denote in the rest of this chapterNâ = N âª {â1} .Here is the first main result of this section.Theorem 4.12. Let (xk)kâNâ be a sequence generated by BiFRB and definezk = (xk+1, xk) for k â Nâ. Then the following hold:(i) For all k â N,Hpk(zk) â¤ Hpkâ1(zkâ1)âM1,k â¥zk â zkâ1â¥2 . (4.5)644.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCE(ii) Suppose that Assumption 4.10(i) holds. Then (Hpk(zk))kâN is decreas-ing andââk=0M1,k â¥zk â zkâ1â¥2 < â. Consequently, the sequence(Hpk(zk))kâN is convergent and limkââ â¥zk â zkâ1â¥ = 0.Proof.(i) By using the second iterative step of BiFRB,f(xk+1) â¤ f(xk) + â¨xk â xk+1,âg(xk)â©+ αkλk â¨xk â xk+1, xkâ1 â xkâ©+ 1λk (Dh(xk, yk)âDh(xk+1, yk))â¤ f(xk) + â¨xk â xk+1,âg(xk)â©+ αkλk â¨xk â xk+1, xkâ1 â xkâ©+ Lâh2Î»k â¥xk â ykâ¥2 â Ï2Î»k â¥xk+1 â ykâ¥2 , (4.6)where the last inequality is implied by Lemma 4.2. Invoking Fact 2.66 to gyieldsg(xk+1) â¤ g(xk) + â¨xk+1 â xk,âg(xk)â©+ Lâg2 â¥xk+1 â xkâ¥2 . (4.7)Adding (4.6) and (4.7), one hasF (xk+1) â¤ F (xk) + αkλk â¨xk â xk+1, xkâ1 â xkâ©+Lâh2Î»kâ¥xk â ykâ¥2â Ï2Î»k â¥xk+1 â ykâ¥2 +Lâg2 â¥xk+1 â xkâ¥2â¤ F (xk) +(Î±k2Î»k+Lâg2)â¥xk+1 â xkâ¥2 + αk2λk â¥xkâ1 â xkâ¥2+ Lâh2Î»k â¥xk â ykâ¥2 â Ï2Î»k â¥xk+1 â ykâ¥2 . (4.8)Next we estimateLâh2Î»kâ¥xk â ykâ¥2 â Ï2Î»k â¥xk+1 â ykâ¥2= LâhâÏ2Î»k â¥xk â ykâ¥2 â Ï2Î»k(â¥xk+1 â ykâ¥2 â â¥xk â ykâ¥2)=(LâhâÏ)Î»2kâ12Î»kâ¥âg(xk)ââg(xkâ1)â¥2 â Ï2Î»k â¥xk+1 â xkâ¥2â ÏÎ»kâ1Î»k â¨xk+1 â xk,âg(xk)ââg(xkâ1)â©â¤ (LâhâÏ)L2âgÎ»2kâ12Î»kâ¥xk â xkâ1â¥2 + ÏLâgÎ»kâ1Î»k â¥xk+1 â xkâ¥ â¥xk â xkâ1â¥â Ï2Î»k â¥xk+1 â xkâ¥2â¤ (LâhâÏ)L2âgÎ»2kâ1+ÏLâgÎ»kâ12Î»kâ¥xk â xkâ1â¥2 + Ï(LâgÎ»kâ1â1)2Î»k â¥xk+1 â xkâ¥2 .654.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCEThus, inequality (4.8) further implies thatF (xk+1) â¤ F (xk) +(Ï(LâgÎ»kâ1â1)+Î±k2Î»k+Lâg2)â¥xk+1 â xkâ¥2+(LâhâÏ)L2âgÎ»2kâ1+ÏLâgÎ»kâ1+Î±k2Î»kâ¥xk â xkâ1â¥2 . (4.9)Adding pk â¥xk+1 â xkâ¥2 to the above inequality givesHpk(zk) = F (xk+1) + pk â¥xk+1 â xkâ¥2â¤ F (xk) +(Ï(LâgÎ»kâ1â1)+Î±k2Î»k+Lâg2 + pk)(â¥zk â zkâ1â¥2 â â¥xk â xkâ1â¥2)+(LâhâÏ)L2âgÎ»2kâ1+ÏLâgÎ»kâ1+Î±k2Î»kâ¥xk â xkâ1â¥2= F (xk) + pkâ1 â¥xk â xkâ1â¥2 âM1,k â¥zk â zkâ1â¥2= Hpkâ1(zkâ1)âM1,k â¥zk â zkâ1â¥2 ,where the first inequality follows from the identityâ¥zk â zkâ1â¥2 = â¥xk+1 â xkâ¥2 + â¥xk â xkâ1â¥2 ,and the second equality follows from (4.3) and (4.4).(ii) Assume without loss of generality that pk â¥ 0 and M1,k > 0 for k â N.Thus by statement(i) and Assumption 4.1, the sequence (Hpk(zk))kâN isdecreasing and bounded below, therefore it is convergent. Summing in-equality (4.5) from k = 0 to k = nâ 1, we havenâ1âk=0M1,k â¥zk+1 â zkâ¥2 â¤nâ1âk=0(Hpk(zk)âHpk+1(zk+1))= Hp0(z0)âHpn(zn)â¤ Hp0(z0)â F (xn+1) â¤ Hp0(z0)â infxâRnF (x) <â,which together with the assumption that lim infkââM1,k > 0 implies thatââk=0 â¥zk â zkâ1â¥2 <â.The following lemma is an immediate consequence of subdifferential sumrules Fact 2.15 and Fact 2.16.Lemma 4.13. Let p â R. Then for every (x, y) â Rn Ã Rn,âHp(x, y) = {âf(x) +âg(x) + 2p(xâ y)} Ã {2p(y â x)}.664.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCEThe lemma below provides a lower bound for consecutive iterates gap.Lemma 4.14. Let (xk)kâNâ be a sequence generated by BiFRB, let zk =(xk+1, xk) for k â Nâ, and let p â¥ 0. For k â N, define uk = (âh(yk) ââh(xk+1))/Î»k ââg(xk) + αk(xk â xkâ1)/Î»k,Ak = uk +âg(xk+1) + 2p(xk+1 â xk), Bk = 2p(xk â xk+1).SetM2,k =â2max{LâhÎ»k+ Lâg + 6p,LâhLâgÎ»kâ1+1Î»k},andM2 =â2max{LâhÎ» + Lâg + 6p,LâhLâgÎ»+1Î»}.Then (Ak, Bk) â âHp(zk) andâ¥(Ak, Bk)â¥ â¤M2,k â¥zk â zkâ1â¥ â¤M2 â¥zk â zkâ1â¥ .Proof. Invoking (2.), one has0 â âf(xk+1) +âg(xk) + Î±k(xkâ1âxk)Î»k +âh(xk+1)ââh(yk)Î»k,which implies that uk â âf(xk+1). It then follows from Lemma 4.13 that(Ak, Bk) â âHp(zk). Moreover,â¥Akâ¥ â¤ â¥uk +âg(xk+1)â¥+ 2p â¥xk+1 â xkâ¥â¤â¥â¥â¥ 1λk (âh(yk)ââh(xk+1)) +âg(xk+1)ââg(xk)â¥â¥â¥+ 4p â¥xk+1 â xkâ¥+ αkλk â¥xk â xkâ1â¥â¤ LâhÎ»k â¥yk â xk + xk â xk+1â¥+ (Lâg + 4p) â¥xk+1 â xkâ¥+ 1λk â¥xk â xkâ1â¥â¤ LâhLâgÎ»kâ1+1Î»k â¥xk â xkâ1â¥+(LâhÎ»k+ Lâg + 4p)â¥xk+1 â xkâ¥ .Thenâ¥(Ak, Bk)â¥ â¤ â¥Akâ¥+ â¥Bkâ¥ â¤ â¥Akâ¥+ 2p â¥xk+1 â xkâ¥â¤ LâhLâgÎ»kâ1+1Î»k â¥xk â xkâ1â¥+(LâhÎ»k+ Lâg + 6p)â¥xk+1 â xkâ¥ ,â¤M2,k â¥zk â zkâ1â¥ â¤M2 â¥zk â zkâ1â¥ ,as desired.Having established basic properties of(Hpk(xk+1, xk)), we now show itsconvergence. Denote by Ï(zâ1) the set of all cluster points of (zk)kâNâ .674.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCETheorem 4.15. Let (xk)kâNâ be a sequence generated by BiFRB and de-fine zk = (xk+1, xk) for k â Nâ. Suppose that Assumption 4.10 holds and(zk)kâNâ is bounded. Let (zkl)lâN be a subsequence such that zkl â zâ forsome zâ = (xâ, yâ) as lââ. Then the following hold:(i) We have limlââHpkl (zkl) = F (xâ). In fact, limkââHpk(zk) = F (xâ).Consequently limkââHp¯(zk) = F (xâ), where p¯ is given in Assump-tion 4.10(ii).(ii) The limit point zâ = (xâ, yâ) satisfies xâ = yâ and 0 â âHp¯(xâ, yâ),which implies that 0 â âF (xâ).(iii) The set Ï(zâ1) is nonempty, connected, and compact, on which thefunction Hp¯ is constant F (xâ). Moreover, limkââ dist(zk, Ï(zâ1)) =0.Proof.(i) For every k â N, appealing to the iterative scheme (2.), we havef(xk+1)â¤ f(xâ) + â¨xâ â xk+1,âg(xk)â©+ αkλk â¨xâ â xk+1, xkâ1 â xkâ©+ 1Î»kDh(xâ, yk)â 1λkDh(xk+1, yk)â¤ f(xâ) + â¨xâ â xk+1,âg(xk)â©+ αkλk â¨xâ â xk+1, xkâ1 â xkâ©+ LâhâÏ2Î»k â¥xâ â ykâ¥2 â Ï2Î»k( â¥xk+1 â ykâ¥2 â â¥xâ â ykâ¥2 )= f(xâ) + â¨xâ â xk+1,âg(xk)â©+ αkλk â¨xâ â xk+1, xkâ1 â xkâ©+ LâhâÏ2Î»k â¥xâ â ykâ¥2 â Ï2Î»k( â¥xk+1 â xââ¥+ 2â¨xk+1 â xâ, xâ â ykâ©).(4.10)Recall that xkl â xâ as l â â. Then Lemma 4.12(ii) and the triangleinequality imply that xklâ1 â xâ andâ¥xâ â yklâ1â¥ â¤ â¥xâ â xklâ1â¥+ Î»klâ2Lâg â¥xklâ2 â xklâ1â¥ â 0, as lââ,where the first inequality holds due to Step 1 of BiFRB. Taking the abovelimits into account, replacing the index k + 1 by kl in (4.10) and passing tothe limit, one getslim suplââf(xkl) â¤ f(xâ) â¤ lim inflââf(xkl),684.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCEwhere the last inequality holds thanks to the lower semicontinuity of f , whichimplies that limlââ F (xkl) = F (xâ). Note that the sequence (Hpk(zk))kâNis convergent. Then one concludes thatlimkââHpk(zk) = limlââHpkl (zkl) = limlââ(F (xkl) + pkl â¥xkl â xklâ1â¥2)= F (xâ),wherelimlââpkl â¥xkl â xklâ1â¥2 = 0due to Assumption 4.10(ii) and Lemma 4.12(ii).Finally, it is easy to see that0 â¤ (p¯â pk) â¥xk â xkâ1â¥2 â¤ p¯ â¥xk â xkâ1â¥2 â 0,as k ââ under Assumption 4.10(ii). ThuslimkââHp¯(zk) = limkââ(Hpk(zk) + (p¯â pk) â¥xk â xkâ1â¥2)= F (xâ).(ii) By Lemma 4.12(ii) and the triangle inequality, subsequences (xkl)lâN and(xkl+1)lâN have the same limit. Therefore xâ = yâ. Combining Lemma 4.14with p = p¯, Lemma 4.12(ii), statement(i), and the outer semicontinuity ofz 7â âHp¯(z), we conclude that 0 â âHp¯(zâ), which further implies 0 ââF (xâ).(iii) Apply a similar proof as in [92, Theorem 3.6(iii)].The result below provides a sufficient condition to the critical bounded-ness assumption in Theorem 4.15.Theorem 4.16. Let (xk)kâNâ be a sequence generated by BiFRB and definezk = (xk+1, xk) for k â Nâ. Suppose that Assumption 4.10(i) holds. If f + gis coercive (or level-bounded), then the sequence (xk)kâNâ is bounded, so is(zk)kâNâ.Proof. Assume without loss of generality that M1,k > 0 and pk â¥ 0 fork â N. Then, appealing to Lemma 4.12, one has Hpk(zk) â¤ Hpâ1(zâ1) fork â N. ThereforeF (xk+1) = Hpk(zk)â pk â¥xk+1 â xkâ¥2 â¤ Hpk(zk) â¤ Hpâ1(zâ1).If (xk)kâN was unbounded, then the above inequality would yield a contra-diction due to the coercivity of f + g (or level-boundedness).694.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCE4.2.4 Abstract sequential convergenceIn this section, we establish global sequential convergence of BiFRB to astationary point by using the generalized concave KL property (recall Defi-nition 2.86). For ε > 0 and nonempty set â¦ â Rn, we define â¦Îµ = {x â Rn :dist(x,â¦) < ε}.Theorem 4.17 (global convergence of BiFRB). Let (xk)kâNâ be a sequencegenerated by BiFRB and define zk = (xk+1, xk) for k â Nâ. Suppose thatAssumption 4.10 holds and (zk)kâNâ is bounded. Suppose further that thedominating BiFRB merit function Hp¯ (recall Definition 4.9) has the gener-alized concave KL property on Ï(zâ1). Then the following hold:(i) The sequence (zk)kâNâ is Cauchy and has finite length. To be specific,there exist index k0 â N, ε > 0 and η â (0,â] such that for i â¥ k0+1,ââk=iâ¥zk+1 â zkâ¥ â¤ â¥zi â ziâ1â¥+ CÏË (Hpi(zi)â F (xâ)) , (4.11)whereC =2â2max{LâhÎ» +Lâg+6p¯,LâhLâgÎ»+1Î»}lim infkââM1,k ,and ÏË is the exact modulus associated with the setwise generalized con-cave KL property of Hp¯ on Ï(zâ1) with respect to ε and η.(ii) The sequence (xk)kâNâ has finite length and converges to some xâ with0 â âF (xâ).Proof.(i) By the boundedness assumption, assume without loss of generality thatzk â zâ = (xâ, yâ) â Rn Ã Rn. Then Theorem 4.15 implies that xâ = yâand Hpk(zk) â F (xâ) as k â â. By Lemma 4.12, assume without loss ofgenerality that(Hpk(zk))kâN is decreasing. Consider two cases:Case 1: Suppose that there exists k0 such that F (zâ) = Hpk0 (zk0). ThenLemma 4.12(i) implies that zk0+1 = zk0 and xk0+1 = xk0 . The desired resultsthen follows from a simple induction.Case 2: Assume that F (zâ) < Hpk(zk) for all k. By Theorem 4.15(iii), thefunction Hp¯ is constant F (xâ) on Ï(zâ1). Then Lemma 3.7 yields ε > 0and η > 0 such that Hp¯ has the setwise generalized concave KL property onÏ(zâ1) with respect to ε > 0 and η > 0 and the associated exact modulus704.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCEÏË. Appealing to Theorem 4.15 again, there exists k1 such that zk â [0 <Hp¯ â Hp¯(zâ) < η] for k > k1 and k2 such that dist(zk, Ï(zâ1)) < ε for allk > k2. Put k0 = max(k1, k2). Then for k > k0,(ÏË)â²â (Hpk(zk)â F (xâ)) · dist(0, âHp¯(zk))â¥(ÏË)â²â (Hp¯(zk)â F (xâ)) · dist(0, âHp¯(zk)) â¥ 1,where the first inequality holds because Hpk(zk) â¤ Hp¯(zk) and the last onefollows from the generalized concave KL inequality (recall Definition 2.86).Invoking Lemma 4.14 yieldsM2,k(ÏË)â²â (Hpk(zk)â F (xâ)) · â¥zk â zkâ1â¥ â¥ 1, (4.12)where M2,k =M2,k(p¯). For simplicity, defineâk,k+1 = ÏË (Hpk(zk)âH(zâ))â ÏË(Hpk+1(zk+1)âH(zâ)).By the concavity of ÏË and (4.12), one hasâk,k+1 = ÏË (Hpk(zk)âH(zâ))â ÏË(Hpk+1(zk+1)âH(zâ))â¥ (ÏË)â²â (Hpk(zk)âH(zâ)) · [Hpk(zk)âHpk+1(zk+1)]â¥ Hpk (zk)âHpk+1 (zk+1)M2,kâ¥zkâzkâ1â¥ . (4.13)Applying Lemma 4.12 to (4.13) yieldsâk,k+1 â¥ â¥zk+1âzkâ¥2Ckâ¥zkâzkâ1â¥ ,where Ck =M2,k/M1,k. On one hand, by Assumption 4.10(i), assume with-out loss of generality that M1,k â¥ lim infkââM1,k/2 for all k. On the otherhand, M2,k satisfiesM2,k â¤â2max{LâhÎ» + Lâg + 6p¯,LâhLâgÎ»+1Î»}.Then Ck â¤ C and2 â¥zk+1 â zkâ¥ â¤ 2âCkâk,k+1 â¥zk â zkâ1â¥ â¤ Câk,k+1 + â¥zk â zkâ1â¥ .(4.14)714.2. THE BIFRB METHOD AND ITS ABSTRACT CONVERGENCEPick i â¥ k0 + 1. Summing (4.14) from i to an arbitrary j > i yields2jâk=iâ¥zk+1 â zkâ¥ â¤jâk=iâ¥zk â zkâ1â¥+ Cjâk=iâk,k+1â¤jâk=iâ¥zk+1 â zkâ¥+ â¥zi â ziâ1â¥+ CÏË (Hpi(zi)â F (xâ))â CÏË (Hpj+1(zj+1)âH(zâ))â¤jâk=iâ¥zk+1 â zkâ¥+ â¥zi â ziâ1â¥+ CÏË (Hpi(zi)â F (xâ)) ,implying thatjâk=iâ¥zk+1 â zkâ¥ â¤ â¥zi â ziâ1â¥+ CÏË (Hpi(zi)â F (xâ)) ,from which (4.11) readily follows. Finally, notice thatâ¥zj â ziâ¥ â¤jâk=iâ¥zk+1 â zkâ¥ â¤ â¥zi â ziâ1â¥+ CÏË (Hpi(zi)â F (xâ)) . (4.15)Recall from Theorem 4.15(i) and Lemma 4.12(ii) that Hpi(zi)â F (xâ) andâ¥zi â ziâ1â¥ â 0 as iââ. Thus, passing the index i to infinity, we concludefrom inequality (4.15) that (zk)kâN is Cauchy. Invoking Theorem 4.15(ii)completes the proof.(ii) The desired statement follows from the definition of (zk)kâNâ and The-orem 4.15(ii).The generalized concave KL assumption in Theorem 4.17 can be easilysatisfied by semialgebraic functions.Corollary 4.18. Let (xk)kâNâ be a sequence generated BiFRB. Assume that(xk)kâNâ is bounded and Assumption 4.10 holds. Suppose further that bothf and g are semialgebraic. Then (xk)kâNâ converges to some xâ with 0 ââF (xâ) and has finite length property.Proof. Let p¯ be as in Assumption 4.10. Then (x, y) 7â p¯ â¥xâ yâ¥2 is semial-gebraic. In turn, the dominating BiFRB function Hp¯ (recall Definition 4.9)is semialgebraic by the semialgebraic functions sum rule; see, e.g., [3, Section4.3]. The desired result then follows immediately from Theorem 4.17.724.3. STEPSIZE STRATEGIES4.3 Stepsize strategiesFollowing abstract convergence properties of BiFRB, we now exploit con-ditions such that the important Assumption 4.10 holds. We emphasis thatour choices for stepsizes and inertial parameters are independent in Propo-sitions 4.20, 4.22 below; while parameters pk involved in constructing meritfunctions Hpk are implicit. This improves a bad practice in recent literature,where explicit assumptions are imposed on parameter pk for the convenienceof convergence analysis, resulting in dependent or even implicit choices ofstepsizes or inertia parameters; see [24,25,70].In the remainder of this section, unless specified otherwise,a = (Lâh â Ï)L2âg, b = Ï, and c = Lâg. (4.16)Consequently, the sequence (pk)kâNâ and (M1,k)kâNâ given by (4.3) and (4.4)satisfypk =a2(Î»2kâ1Î»k)+ 12(bÎ»kâ c)â pkâ1, M1,k = pkâ1 â Î±k+bcÎ»kâ12Î»k âaÎ»2kâ12Î»k.(4.17)The lemma below asserts that (pk)kâNâ and (M1,k)kâN are governed bystepsizes and an initial input pâ1, which will be useful soon.Lemma 4.19. Let (Î»k)kâNâ be a sequence of nonzero real numbers and leta, b, c â R. Set pâ1 â R and define (pk)kâNâ as in (4.17). Then for k â N,p2k =a2kâi=1(Î»22iâ1Î»2iâ Î»22iâ2Î»2iâ1)+ b2kâi=1(1Î»2iâ 1Î»2iâ1)+aÎ»2â1+b2Î»0â c2 â pâ1,p2k+1 =a2kâi=0(Î»22iÎ»2i+1â Î»22iâ2Î»2iâ1)+ b2kâi=0(1Î»2i+1â 1λ2i)+ pâ1,where we adopt the conventionâ0i=1 ai = 0 for any sequence (ak)kâN.734.3. STEPSIZE STRATEGIESProof. Let k â N{0}. Thenp2k =a2(Î»22kâ1Î»2k)+ 12(bÎ»2kâ c)â p2kâ1= a2(Î»22kâ1Î»2kâ Î»22kâ2Î»2kâ1)+ b2(1Î»2kâ 1Î»2kâ1)+ p2kâ2= a2kâi=0(Î»22iâ1Î»2iâ Î»22iâ2Î»2iâ1)+ b2kâi=1(1Î»2iâ 1Î»2iâ1)+ p0= a2kâi=1(Î»22iâ1Î»2iâ Î»22iâ2Î»2iâ1)+ b2kâi=1(1Î»2iâ 1Î»2iâ1)+aÎ»2â1+b2Î»0â c2 â pâ1,andp2k+1 =a2(λ22kλ2k+1)+ 12(bÎ»2k+1â c)â p2k= a2(λ22kλ2k+1)+ 12(bÎ»2k+1â c)â a2kâi=1(Î»22iâ1Î»2iâ Î»22iâ2Î»2iâ1)â b2kâi=1(1Î»2iâ 1Î»2iâ1)â aÎ»2â1+b2Î»0+ c2 + pâ1= a2kâi=0(Î»22iÎ»2i+1â Î»22iâ2Î»2iâ1)+ b2kâi=0(1Î»2i+1â 1λ2i)+ pâ1,from which the desired identities hold.4.3.1 Non-Euclidean caseWe shall establish the existence of suitable merit function parameters (pk)and (M1,k) satisfying Assumption 4.10, under which stepsize and inertialparameter are independent. As an internal technical necessity, such exis-tence result does not affect the implementation of the proposed algorithmbecause bounds on both stepsize and inertial parameter are explicit.Proposition 4.20 (fixed stepsize). Let (Î»k)kâNâ be a constant BiFRB se-quence with λk = λ > 0 for all k. Suppose that Ï > 2, (Lâh â Ï)Ï > 1/4and 0 â¤ αk < 1. DefineÎ»â =â(2bc+c)2+4a(bâ2)â2bcâc2a > 0,744.3. STEPSIZE STRATEGIESwhere a, b, c are specified by (4.16). If λ < min{Î»â, (Ï â 1)/[(Ï + 1)Lâg]},then there exists pâ1 such that (pk)kâNâ and (M1,k)kâN generated by (4.3)and (4.4) satisfy Assumption 4.10.Proof. Applying Lemma 4.19 to (4.17) yields that pk = pâ1 if k is odd;pk = aλ/2 + (b/Î»â c) /2â pâ1 if k is even; andM1,k ={pâ1 â αk2λ â bc2 â aλ2 , if k is even,bâÎ±k2Î» â (b+1)c2 â pâ1, if k is odd.Thenmin{pâ1, a2λ+12(bλ â c)â pâ1} â¤ pk â¤ max{pâ1, a2λ+ 12 ( bλ â c)â pâ1},(4.18)lim infkââM1,k â¥ min{pâ1 â 12λ â bc2 â aλ2 , bâ12Î» â (b+1)c2 â pâ1}. (4.19)To furnish Assumption 4.10, we will prove thata2λ+12(bλ â c)> 0, (4.20)12λ +bc2 +aλ2 <bâ12Î» â (b+1)c2 , (4.21)12λ +bc2 +aλ2 <aλ2 +b2λ â c2 . (4.22)Combining inequalities (4.20)â(4.22) yields the existence of some pâ1 suchthat0 < pâ1 < a2λ+12(bλ â c)and 12λ +bc2 +aλ2 < pâ1 <bâ12Î» â (b+1)c2 ,which together with (4.18) and (4.19) entails Assumption 4.10. First, weestablish (4.20). Observe thata2λ+12(bλ â c)> 0â aλ2 â cλ+ b > 0.By assumption, a > 0 and c2 â 4ab = L2âg(1â 4(Lâh â Ï)Ï)< 0. Then thequadratic function λ 7â aλ2 â cλ + b has no root and is therefore strictlypositive, implying (4.20). Next we justify (4.21), which amounts to âaÎ»2 â(2bc+c)Î»+bâ2 > 0. Notice that the discriminant (2bc+c)2+4a(bâ2) > 0and product of roots (b â 2)/(âa) < 0. Then the quadratic function λ 7ââaÎ»2 â (2bc+ c)λ+ bâ 2 has two real roots with opposite signs. In turn,(â 0 < λ < Î»â)â aλ2 â (2bc+ c)λ+ bâ 2 > 0,754.3. STEPSIZE STRATEGIESfurnishing inequality (4.21). Finally, the assumption λ < (Ï â 1)/[(Ï +1)Lâg] = (bâ 1)/[(b+ 1)c] implies (4.22) after rearranging.Remark 4.21. Some comments are in order.(i) (Comparison to known results) Compared to a result of Bot¸ and Cset-nek [24, Lemma 3.3] with explicit merit function parameters but im-plicit stepsize bounds, our stepsize is explicit and independent of in-ertial parameters; see [25, Remark 7] for a similar restrictive resultand [62, Corollary 4.4] for a result with dependent stepsize bound andinertial parameter in the presence of convexity; see also [68, Table 1].(ii) With inertial parameter αk â [0, 1) independent of stepsize λ in Propo-sition 4.20, the Nesterov acceleration scheme is implementable. In-deed, it corresponds to Proposition 4.20 with(âk â Nâ) αk = tkâ1tk+1 ,where (âk â Nâ) tk+1 = (1 +â1 + 4t2k)/2 and tâ1 = 1; see, e.g., [66].(iii) The assumption that (LâhâÏ)Ï > 1/4 is not demanding. For instance,let α, β > 0 be such that αβ > 1/4 and let u : Rn â R be a functionwith 1-Lipschitz gradient. Then h = αu + β â¥Â·â¥2 /2 satisfies (Lâh âÏ)Ï > 1/4.4.3.2 Euclidean caseNevertheless, in contrast to the non-Euclidean case, we shall see that stepsizeand inertial parameter are dependent.Proposition 4.22 (dynamic stepsize). Let (Î»k)kâNâ â [λ, λ] be a iFRBstepsize sequence and let (pk)kâNâ be the sequence generated by (4.17) witha = 0, b = 1 and 0 â¤ αk < α¯ for some α¯ < 1/2. Let (ak)kâN be a sequenceof positive real numbers withââk=0 ak <â and let 0 < ε < (1â2Î±¯)/(3Lâg).Suppose that the following hold:(i) The stepsize lower bound λ = ε and upper bound λ < Îµ/(2Î±¯+3ÎµLâg).(ii) For all k â N, 0 â¤ (1/Î»k)â (1/Î»kâ1) â¤ ak.Then there exists pâ1 such that (pk)kâNâ and (M1,k)kâN generated by (4.3)and (4.4) satisfy Assumption 4.10.764.3. STEPSIZE STRATEGIESProof. First notice that assumption(i) is not contradictory because ε <(1 â 2Î±¯)/(3Lâg) â 2α¯ + 3ÎµLâg < 1 â ε < ε/(2α¯ + 3ÎµLâg). Then, byassumption(i), we haveλ0 <Îµ2Î±¯+3ÎµLâg âLâg2 +α¯2ε <12Î»0â Lâgâ α¯2ε ,which implies that there exits pâ1 such thatLâg2 +α¯2ε < pâ1 <12Î»0â Lâgâ α¯2ε . (4.23)According to Lemma 4.19 and assumption(ii)pâ1 â¤ p2k+1 = 12kâi=0(1Î»2i+1â 1λ2i)+ pâ1 â¤ 12ââk=0ak + pâ1,12Î»0â c2 â pâ1 â¤ p2k = 12kâi=1(1Î»2iâ 1Î»2iâ1)+ 12λ0 â c2 â pâ1â¤ 12ââk=0ak +12Î»0â c2 â pâ1,which together with (4.23) shows that pk is positive and bounded above.By assumption(ii), λk â¤ Î»kâ1 for all k. Consequently, the sequence (λk) isconvergent and limkââ Î»kâ1/Î»k = 1. It follows from the above inequalitiesand (4.23) thatM1,2k+1 = p2k â Î±2k+1+LâgÎ»2k2Î»2k+1 â¥ 12λ0 âLâg2(Î»2kÎ»2k+1+ 1)â α¯2ε â pâ1â 12λ0 â Lâg â α¯2ε â pâ1 > 0,M1,2k = p2kâ1 â Î±2k+LâgÎ»2kâ12Î»2k â¥ pâ1 âLâg2(Î»2kÎ»2k+1)â Î±¯2Îµâ pâ1 â Lâg2 â α¯2ε > 0.Hence lim infkââM1,k > 0.774.4. BREGMAN PROXIMAL SUBPROBLEM FORMULAE4.4 Bregman proximal subproblem formulaeSet α â¥ 0, β > 0. Fix Ï â Rn and λ > 0. In the remainder of this section,leth(x) = Î±â1 + â¥xâ¥2 + β2 â¥xâ¥2 .Define for u â Rnpλ(u) = Î»Ï ââh(u)andTÎ»(u) = argminxâRn{f(x) + â¨xâ u, Ïâ©+ 1λDh(x, u)}(4.24)= argminxâRn{f(x) + â¨x, Ïâ©+ 1Î»h(x)â 1Î»â¨âh(u), xâ©}= argminxâRn{Î»f(x) + â¨x, pÎ»(u)â©+ h(x)}. (4.25)Clearly Assumption 4.6(ii) holds with Ï = β and Lâh = α+β and the Breg-man proximal subproblem (2.) corresponds to (4.24) with Ï = Î±k(xkâ1 âxk)/Î»k +âg(xk), u = yk and λ = λk. Note that (4.24) covers subproblemsin [24, Algorithm 3.1] and [25, Agorithm 1] as well.In this section, we develop formulae of Tλ(u) with the kernel h chosen asabove, supplementing not only BiFRB but also [24, Algorithm 3.1] and [25,Algorithm 1].4.4.1 Formula of l0-constrained problemsLetD = {x â Rn |â¥xâ¥0 â¤ r, â¥xâ¥ â¤ R} ,for integer an r â¤ n and real number R > 0, where (âx â Rn) â¥xâ¥0 =âni=1 | sgn(xi)| with sgn(0) = 0. In this subsection, consider minimizationproblemminxâDg(x),which is (P) with f = δD. Recall that the Hard-threshold of x â Rn withparameter r, denoted by Hr(x), is given by (âi â {1, . . . , n})(Hr(x))i={xi, if |xi| belongs to the r largest among |x1|, . . . , |xn|,0, otherwise.Immediately, Hr(cx) = cHr(x) for c Ì¸= 0. The following lemma will beinstrumental.784.4. BREGMAN PROXIMAL SUBPROBLEM FORMULAELemma 4.23. [61, Proposition 4.3] Given 0 Ì¸= p â Rn and a positive integerr < n, we havemaxxâRn{â¨p, xâ© |â¥xâ¥ = 1, â¥xâ¥0 â¤ r} = â¥Hr(p)â¥ ,with optimal value attained at xâ = â¥Hr(p)â¥â1Hr(p).Following a similar route in [23, Proposition 5.2], we now prove thedesired formula.Proposition 4.24 (subproblem formula). Let f = δD, where D = {x âRn : â¥xâ¥0 â¤ r, â¥xâ¥ â¤ R} for integer an r â¤ n and real number R > 0.Definexâ ={0, if âh(u) = Î»Ï,âtâ â¥â¥Hr(pÎ»(u))â¥â¥â1Hr(pÎ»(u)), if âh(u) Ì¸= Î»Ï, (4.26)where tâ is the unique solution ofmin0â¤tâ¤RÏ(t) = Î±â1 + t2 + β2 t2 â â¥Hr(pÎ»(u)â¥ t, (4.27)which satisfies tâ = R if â¥Hr(pÎ»(u))â¥ â¥ α(1 + R2)â1/2R + βR; otherwisetâ â (0, R) is the unique solution ofα(1 + t2)â1/2t+ Î²tâ â¥Hr(pÎ»(u))â¥ = 0.Then xâ â Tλ(u).Proof. Set p = pλ(u) for simplicity. If âh(u) = Î»Ï, then p = 0 and clearlyTλ(u) = argminxâRn{h(x) : â¥xâ¥0 â¤ r, â¥xâ¥ â¤ R} = 0. Now we considerâh(u) Ì¸= Î»Ï, in which case p Ì¸= 0. Invoking Lemma 4.23, one has for t > 0minxâRn{â¨x, pâ© |â¥xâ¥0 â¤ r, â¥xâ¥ = t} = âmaxxâRn{â¨x/t,âtpâ© |â¥x/tâ¥ â¤ r, â¥x/tâ¥ = 1}= ââ¥Hr(âtp)â¥ = ât â¥Hr(p)â¥ , (4.28)attained atxË = xË(t) = t â¥Hr(âtp)â¥â1Hr(âtp) = ât â¥Hr(p)â¥â1Hr(p). (4.29)It is easy to see that optimal value (4.28) and solution (4.29) still hold whent = 0.794.4. BREGMAN PROXIMAL SUBPROBLEM FORMULAENext we show that xâ â Tλ(u). By (4.25), Tλ(u) is the solution set tominxâRn{â¨x, pâ©+ Î±â1 + â¥xâ¥2 + β2 â¥xâ¥2 |â¥xâ¥0 â¤ r, â¥xâ¥ â¤ R}. (4.30)Consider an arbitrary feasible x of (4.30) and let tx = â¥xâ¥. Then 0 â¤ tx â¤ Randâ¨p, xââ©+ Î±â1 + â¥xââ¥2 + β2 â¥xââ¥2= âtâ â¥Hr(p)â¥+ Î±â1 + (tâ)2 + β2 (tâ)2= Ï(tâ) â¤ Ï(tx) = ââ¥Hr(p)â¥ tx + Î±â1 + t2x +β2 t2xâ¤ â¨p, xâ©+ Î±â1 + â¥xâ¥2 + β2 â¥xâ¥2 ,where the first and last inequalities hold due to (4.27) and (4.28) respectively.Finally, we turn to (4.27). Noticing the strong convexity of Ï and ap-plying optimality condition, one getsâÎ±(1 + t2)â1/2tâ βt+ â¥Hr(p)â¥ = âÏâ²(tâ) â N[0,R](tâ),Sotâ = Râ âÏâ²(R) â N[0,R](R) = [0,â)â Ïâ²(R) â¤ 0â â¥Hr(p)â¥ â¥ α(1 +R2)â1/2R+ Î²R,andtâ = 0â âÏâ²(0) â N[0,R](0) = (ââ, 0]â â¥Hr(p)â¥ â¤ 0â Hr(p) = 0,which never occurs when p Ì¸= 0. If 0 < â¥Hr(p)â¥ < α(1 + R2)â1/2R + βR,then Ïâ²(0) < 0 and Ïâ²(R) > 0, meaning that there exists unique tâ â (0, R)such that Ïâ²(tâ) = 0 due to the strong convexity of Ï.Remark 4.25. When 0 < â¥Hr(pÎ»(u)â¥ < Î±(1+R2)â1/2R+Î²R, tâ in Propo-sition 4.24 is indeed one of the four roots of quartic equation0 =β2t4 â 2β â¥Hr(pÎ»(u))â¥ t3 +(â¥Hr(pÎ»(u))â¥2 + β2 â Î±2)t2â 2β â¥Hr(pÎ»(u))â¥ t+ â¥Hr(pÎ»(u))â¥2Although quartic equations admit closed-form solutions, it is challenging todetermine beforehand the branch of solutions to which tâ belongs, due toparameters α, β and â¥Hr(pÎ»(u))â¥. To find tâ efficiently, one may employnumerical root-finding algorithms. For instance, apply bisection method toidentify a subinterval of [0, R] in which the desired root tâ lies, then appealto Newtonâs method; see, e.g., [30, Chapter 2].804.4. BREGMAN PROXIMAL SUBPROBLEM FORMULAE4.4.2 Formula of problems penalized by a positivelyhomogeneous convex functionRecall that f : Rn â R is positively homogeneous if f(tx) = tf(x) for everyx â Rn and t > 0. In this subsection, we provide subproblem formula forproblems of the formminxâRnf(x) + g(x),where f : Rn â R is convex and positively homogeneous with known proxi-mal mapping, which is a special case of problem (P).The result below generalizes the technique of Bolte-Sabach-Teboulle-Vaisbourd [23, Proposition 5.1]. However one cannot recover their result asour kernel is different.Proposition 4.26. Let f : Rn â R be proper, lsc and convex. Suppose thatf is positively homogeneous. Then the following hold:(i) (âx â dom âf) (ât > 0) âf(x) = âf(tx).(ii) (subproblem formula) (âu â Rn) Tλ(u) = tâPÎ»f(âpÎ»(u)), where tâis the unique root of the equation1â αt(1 + t2 â¥PÎ»f(âpÎ»(u))â¥ )â1/2 â βt = 0.Proof. (i) v â âf(x) â (ây â Rn) f(y) â¥ f(x) + â¨v, y â xâ© â (ây âRn) (ât > 0) f(ty) â¥ f(tx) + â¨v, ty â txâ© â v â âf(tx).(ii) For simplicity, let p = pλ(u). Note that Tλ(p) is single-valued bystrong convexity. Then invoking optimality condition and subdifferentialsum rule yieldsxâ = TÎ»(u)â 0 â Î»âf(xâ) + p+âh(x)= Î»âf(xâ) + p+ [α(1 + â¥xââ¥2)â1/2 + Î²]xâ.Define tâ = [α(1 + â¥xââ¥2)â1/2 + Î²]â1 and v = xâ/tâ. Then tâ > 0 and theabove inclusion together with statement(i) implies that0 â Î»âf(xâ) + p+ v = Î»âf(tâv) + p+ v = Î»âf(v) + v + pâ âp â (Id+Î»âf)(v)â v = (Id+Î»âf)â1(âp) = PÎ»f(âp).In turn, tâ satisfiesv[1â Î±tâ(1 + (tâ)2 â¥vâ¥2)â1/2 â Î²tâ]= v â [α(1 + (tâ)2 â¥vâ¥2)â1/2 + Î²]tâv= v â xâtâ = v â v = 0.814.5. NUMERICAL EXPERIMENTSIf v Ì¸= 0, then clearly 1âÎ±tâ(1+(tâ)2 â¥vâ¥2)â1/2âÎ²tâ = 0 as claimed; if v = 0,then xâ = 0 and tâ = 1/(α+ β), which is consistent with statement(ii). Tosee the uniqueness, let Ï(t) = α(1 + â¥vâ¥2 t2)1/2/ â¥vâ¥2 + Î²t2/2â t for t â R,which is strongly convex. Simple calculus shows that tâ is stationary pointof Ï, thus unique.Remark 4.27. Similarly to Proposition 4.24 and Remark 4.25, one mayfind tâ efficiently by appealing to root-finding algorithms.Equipped with Proposition 4.26, we now turn to concrete problems.Recall that the Soft-threshold of x â Rn with parameter λ is Sλ(x) =Pλ â¥Â·â¥1 (x), which satisfies(âi â {1, . . . , n}) (Sλ(x))i = max(|xi| â λ, 0) sgn(xi),see, e.g., [14, Example 6.8].Corollary 4.28 (l1-penalized problem). Let f = â¥Â·â¥1. Then Tλ(u) =âtâSÎ» (pλ(u)), where tâ is the unique solution of1â αt(1 + t2 â¥SÎ» (pÎ»(u))â¥2)â1/2 â βt = 0.Corollary 4.29 (lâ-penalized problem). Let f = â¥Â·â¥â. Then Tλ(u) = tâv,wherev = pÎ»(u)â Î»Proj(pÎ»(u)/Î»;Bâ¥Â·â¥1),and tâ is the unique solution of 1â αt(1 + t2 â¥vâ¥2)â1/2 â βt = 0.Remark 4.30. See [14, Example 6.33] for computing Proj(Â·;Bâ¥Â·â¥1).4.5 Numerical experimentsIn the remainder, let A â RmÃn, b â Rm, r â N{0}, and R > 0. DefineC = {x â Rn : Ax = b} and D = {x â Rn : â¥xâ¥0 â¤ r, â¥xâ¥ â¤ R}. Clearly Cis semialgebraic. The set D can be represented asD = {x â Rn : â¥xâ¥0 â¤ r} â© {x â Rn : â¥xâ¥2 â¤ R2},which is an intersection of semialgebraic sets;thus semialgebraic. ConsiderminxâD12dist2(x,C), (4.31)which corresponds to (P) with f = δD and g = dist2(Â·, C)/2. In turn, BiFRBconverges to a stationary point of (4.31) owing to Theorem 4.17.824.5. NUMERICAL EXPERIMENTSTargets We shall benchmark BiFRB against the Douglas-Rachford (DR)[53], Forward-reflected-backward (FRB) [92] and inertial Tsengâs (iTseng)[24] methods with R = 1, 1000 and r = âm/5â. These splitting methodsare known to converge globally to a stationary point of (P) under the sameassumptions on F ; see [53, Theorems 1â2, Remark 4, Corollary 1], [92,Theorem 3.9] and [24, Theorem 3.1], respectively.Parameters and termination BiFRB and FRB are implemented withstepsize rules given by Proposition 4.20 and [92, Theorem 3.9] respectively,and terminated whenmax{ â¥xk+1 â xkâ¥ , â¥xk â xkâ1â¥}max{1, â¥xkâ¥ , â¥xkâ1â¥} < 10â10. (4.32)Moreover, BiFRB uses the kernel h(x) = 0.1 Â·â1 + â¥xâ¥2+2.51 Â·â¥xâ¥2 /2. Weapply DR with stepsize and termination condition given by [53, Section 5]with the same tolerance 10â10; while iTseng employs a stepsize given by [24,Lemma 3.3] and termination condition (4.32). As for inertial parameter, weuse α1 = 0.9 for BiFRB and α2 = 0.49 for both of iFRB and iTseng. Finally,each algorithm is initialized at the origin and is equipped with the stepsizeheuristic described in [53, Remark 4].Experiments data and results Simulation results are presented in Ta-ble 4.1 below Our problem data is generated through creating random ma-trices A â RmÃn with entries following the standard Gaussian distribution.For each problem of the size (m,n), we randomly generate 50 instances andreport ceilings of the average number of iterations (iter) and the minimal ob-jective value at termination (fvalmin). As problem (4.31) has optimal valuezero, fvalmin indicates whether an algorithm actually hits a global minimizerrather than simply converging to a stationary point. We observed the fol-lowing:- BiFRB is the most advantageous method on âbadâ problems (smallm and R), but its numerical performance is unstable when the size ofthe problem or the radius R changes. This is due to the limitation ofour stepsize heuristic and the numerical method for solving the BiFRBsubproblems; see Remark 4.31 for a detailed explanation. We delaypossible improvements for future research.- DR tends to have the smallest function value at termination, however,it can converge very slowly.834.6. CONCLUSION- In most cases, iFRB has the smallest number of iterations with fairfunction values at termination.Remark 4.31. Some comments are in order.(i) Although the heuristic described in [53, Remark 4] tends to promotesmaller function values at termination by taking more optimistic step-sizes, we observed that it may cause oscillation in function values andslowly vanishing consecutive iterates gap of BiFRB when m is large.(ii) Recall from Proposition 4.24 that one needs to find roots of quarticequations on interval [0, R] when solving BiFRB subproblems. Theroot-finding strategy described in Remark 4.25 is employed in our ex-periments, which turns out to affect the efficiency of BiFRB. WhenR = 1, the aforementioned method works well on small interval [0, R]with negligible number of failures. In contrast, failures in solvingBiFRB subproblems occur much more frequently when R = 1000,producing inexact solutions to subproblems that eventually lead tothe disadvantageous performance of BiFRB.Table 4.1: Comparing BiFRB and iFRB to FRB, DR and iTseng.Size (R = 1) BiFRB iFRB FRB DR iTsengm n iter fvalmin iter fvalmin iter fvalmin iter fvalmin iter fvalmin100 4000 50 0.03251 93 0.03251 631 0.03929 860 0.02819 1367 0.03149100 5000 52 0.02413 115 0.02413 733 0.02875 684 0.02192 1632 0.02268100 6000 57 0.01369 145 0.01369 911 0.01794 683 0.01253 1970 0.0133200 4000 870 0.2923 39 0.2745 190 0.2769 5466 0.2711 448 0.2746200 5000 1198 0.2367 41 0.2095 231 0.2157 5864 0.2073 546 0.2097200 6000 81 0.2306 43 0.2259 257 0.2292 4199 0.2236 620 0.2264300 4000 885 1.051 32 1.036 105 1.038 5497 1.036 253 1.048300 5000 925 0.7481 34 0.7044 124 0.7071 5673 0.7032 302 0.7051300 6000 1407 0.583 36 0.5546 144 0.555 6173 0.5541 352 0.5566Size (R = 1000) BiFRB iFRB FRB DR iTsengm n iter fvalmin iter fvalmin iter fvalmin iter fvalmin iter fvalmin100 4000 1873 0.006609 1210 0.00365 7376 0.00816 2194 4e-21 10001 0.01671100 5000 574 0.00278 1427 0.002244 8675 1.098e-05 2919 1.626e-20 10001 0.0158100 6000 790 0.003827 1696 0.001261 9394 0.0002952 2344 3.438e-20 10001 0.01066200 4000 5842 2.992 750 2.442e-18 8223 0.007688 1038 1.106e-20 9797 0.1094200 5000 5938 0.6547 859 2.876e-05 7257 0.02581 1242 2.082e-20 9997 0.07125200 6000 5666 0.917 1086 0.0001211 7312 0.006726 1487 9.129e-21 10001 0.07524300 4000 7164 3.322 619 2.196e-18 4620 1.462e-05 753 9.795e-21 7722 0.2417300 5000 5267 2.299 693 2.175e-18 6352 0.02936 880 5.446e-20 9194 0.1702300 6000 5724 5.06 750 4.71e-18 8644 0.1109 1027 2.175e-20 9823 0.14234.6 ConclusionWe proved convergence of BiFRB by using the generalized concave KL prop-erty and a novel convergence framework for decreasing property. Our frame-844.6. CONCLUSIONwork leads to stepsize condition is independent of inertial parameter. Webelieve that our approach could be useful for other splitting algorithms indesigning more implementable stepsize rules in the absence of convexity.Formulae for our Bregman subproblem are provided as well, which supple-ments not only BiFRB but also [24,25].85Chapter 5A mirror inertialforward-reflected-backwardsplitting for minimizationproblems beyond convexityand Lipschitz smoothness5.1 IntroductionConsider the following composite minimization problemminimizexâCÏ(x) = f(x) + g(x), (P)where C â Rn is a nonempty open and convex set with closure C, f : Rn âR = R âª {Â±â} is proper, lower semicontinuous (lsc), and differentiable onC, and g : Rn â R is proper and lsc. For notational brevity, we denoteÏC = Ï+ δC where δX is the indicator function of set X â Rn, namely suchthat δX(x) = 0 if x â X and +â otherwise. By doing so, problem (P) canequivalently be cast as the âunconstrainedâ minimizationminimizexâRnÏC(x).The problem (P) is beyond the scope of traditional first-order methods uti-lizing the Lipschitz continuity of âf and the consequential descent lemma inTheorem 2.68; see Chapters 3 and 4 and [4,51,52,53,71] for such algorithms.To resolve this issue, Lipschitz-like convexity was introduced in the seminalwork [7], furnishing a descent lemma beyond the aforementioned setting.This notion was then referred to as relative smoothness (see Definition 5.5)and has played a central role in extending splitting algorithm to the settingof (P); see, e.g., [23, 38,57,67,84].865.1. INTRODUCTIONThe goal of this chapter is to propose and analyze a Bregman inertialforward-reflected-backward method iâFRB (Algorithm 3) for solving (P).This work differs from the analysis carried out in Chapter 4, which alsodeals with an inertial forward-reflected-backward algorithm using Bregmandistances but is still limited to the Lipschitz smoothness assumption. Thegame changer here is taking the inertial step in the dual space, that is,interpolating application of âh (cf. step 2 of Algorithm 3), whence thename, inspired by [15], mirror inertial forward-reflected-backward splitting(iâFRB). A fundamental tool in our analysis is the iâFRB-envelope (seeDefinition 5.21), which is the value function associated to the parametricminimization of a âmodelâ of (P); see Section 5.4.1, and thus possessesmore favorable properties. The term âenvelopeâ is borrowed from the cel-ebrated Moreau envelope and its relation with the proximal operator; seeDefinition 2.50. Indeed, there has been a re-emerged interest of employingan associated envelope function to study convergence of splitting methods,such as forward-backward splitting [1, 86], Douglas-Rachford splitting andADMM [85,87], alternating minimization algorithm [83], as well as the split-ting scheme of Davis and Yin [55]. The aforementioned works share onecommon theme: regularity properties of the associated envelope functionare used for further enhancement and deeper algorithmic insights.The main results in the chapter are listed below:R1 Favorable properties of the iâFRB-operator and iâFRB-envelope areobtained in Theorems 5.20 and 5.22, respectively.R2 The decreasing property of merit functions are investigated in Theo-rem 5.25.R3 We investigate function value convergence of iâFRB in Theorem 5.32.R4 Global convergence of iâFRB is obtained in Theorem 5.37.The rest of the chapter is structured as follows. In the next sectionwe formally define the problem setting and the proposed mirror inertialforward-reflected-backward algorithm (iâFRB), after providing some pre-liminary material and notational conventions. In Section 5.3 we revisit thenotion of relative smoothness and interpret it as a two-sided relative weakconvexity. After introducing the iâFRB-envelope, these findings are used toconstruct a merit function for the proposed iâFRB; The convergence analysisof iâFRB is carried out in Section 5.5.875.1. INTRODUCTION5.1.1 PreliminariesThroughout this chapter, we will adopt the convention thatâââ =â, 1/0 = +â,+â · 0 = 0. (5.1)For an extended-valued number, the usual arithmetics for real numbers maylead to pathological behavior. For instance when a =â, aâ a =â =Ì¸ 0 =a · (1â1). To avoid this, we adopt the extended arithmetics +Ë and âË of [63,Section 2], defined as addition and subtraction operations for extended-valued functions respectively, as reminders that the usual of arithmetics forreal numbers are unavailable and the conventions (5.1) should be in place.Namely we defineââ âË (ââ) = ââ +Ëâ = +â âËâ = +â +Ë (ââ) =â.Furthermore, we denote by Â±Ë and âË the extended-arithmetic equivalents of± and â, respectively. Notice in particular thata âË b = (âb) âË (âa) (5.2)holds for every extended-real pair (a, b) â RÃ R.The following lemma collects other properties of extended arithmeticsthat will be frequently used throughout.Lemma 5.1. Let Ï : Rn â R be proper, and a, b â R be fixed. Then, thefollowing hold:(i) aÏ Â±Ë bÏ +Ë δE = (a± b)Ï +Ë δE for every E â domÏ.(ii) If a > 0, then aÏ Â±Ë bÏ = (a± b)Ï +Ë Î´domÏ.Proof. In both cases we shall verify the equivalences pointwise at everyx â Rn, analyzing the cases x â domÏ and x /â domÏ separately.If x â domÏ, one has that Ï(x) â R by properness of Ï, hence the ex-tended arithmetic notation is superfluous in both assertions and the claimsare trivially true. If x /â domÏ, then δE(x) = +â in assertion 5.1(i) andsimilarly aÏ(x) =â in assertion 5.1(ii). Therefore, the extended-arithmeticconvention ensures that all expressions evaluate to +â in this case, and thuscoincide.885.2. THE ALGORITHM5.2 The algorithmThroughout the rest of this chapter, h : Rn â R is 1-coercive and LegendrewithC = domâh = int domh.We will consider the following iterative scheme for addressing problem (P).Algorithm 3 Mirror inertial forward-reflected-backward (iâFRB)Initialization Choose xâ1, x0 â C, inertial parameter β â R, andstepsize γ > 0Iteration Iterate for k = 0, 1, . . . until a termination criterion is met(cf. Lemma 5.35)1: Compute yk = âhâ(âh(xk)â Î³(âf(xk)ââf(xkâ1)))2: Choose xk+1 â argminwâRn{g(w)+ â¨w,âf(xk)â Î²Î³(âh(xk)ââh(xkâ1))â©+1Î³ Dh(w,yk)}In the rest of this chapter, unless specified otherwise, we assume thefollowing:Assumption 5.2. The following hold in problem (P):A1 f : Rn â R is Lf,h-smooth relative to h (see Section 5.3).A2 g : Rn â R is proper and lsc.A3 inf ÏC > ââ.A4 For every v â Rn and γ > 0, argmin{γg + hâ â¨v, · â©} â C.A5 For every γ > 0, limâ¥xâ¥ââÎ³g(x)+h(x)â¥xâ¥ =â.Some comments are in order.Remark 5.3 (sufficient conditions for Assumption 5.2.A4). Assumption5.2.A4 excludes the minimizers argmin{γg + hâ â¨v, · â©} from occurring onthe boundary of domh. Assumption 5.2.A4 holds when(i) domh is open, because argmin{γg + hâ â¨v, · â©} â dom g â© domh.895.3. RELATIVE SMOOTHNESS AND WEAK CONVEXITY(ii) g is convex or more generally the constraint qualification (ââg) â©(âââh) = {0} holds on the boundary bdry domh. To see this, takex¯ â argmin{γg + h â â¨v, · â©}. Then optimality condition Fact 2.14ensures0 â â(Î³g + h)(x¯)â v â v â â(Î³g + h)(x¯)â v â Î³âg(x¯) + âh(x¯),where the last implication owes to subdifferential sum rule (recallFact 2.17) and the assumption on either convexity or constraint qualifi-cation. Hence âh(x¯) Ì¸= â . Note that h is a Legendre function, thereforeessentially smooth and equivalently dom âh = C; see Fact 2.40(iii). Assuch one must have x¯ â C.Remark 5.4 (Assumption 5.2.A5 and prox-boundedness). Apparently, As-sumption 5.2.A5 together with lower semicontinuity ensures that minimizersof γg + h â â¨v, · â© do exist for every γ and v. We postpone the proof toLemma 5.10 for the details.5.3 Relative smoothness and weak convexityThe following generalizes the classic Lipschitz smoothness discussed in Chap-ter 2.Definition 5.5 (relative smoothness). We say that a function f : Rn â R issmooth relative to h if dom f â domh and there exists a constant Lf,h â¥ 0such thatLf,hh +Ë f +Ë δC and Lf,hh âË f +Ë δC (5.3)are proper convex functions. We may alternatively say that f is Lf,h-smoothrelative to h to make the smoothness modulus Lf,h explicit.Remark 5.6. The addition of δC in (5.3) serves the purpose of assessingconvexity of Lf,h Â±Ë f only on set C, in line with the original âLipschitz-likeconvexityâ notion of [7, §2.2] as well as the subsequent nonconvex general-ization in [23, Def. 2.2], referred to as âLf,h-smooth adaptabilityâ of thepair (f, h). Differently from those works, however, we do not impose (con-tinuous) differentiability of f on C in the definition, for this property isautomatically guaranteed; see Proposition 5.12 for the details.Notice further that the constant Lf,h may be loose. For instance, if fis convex, then Lh +Ë f is convex for every L â¥ 0, indicating that it is onlyconvexity of Lf,hh âË f +Ë δC that dictates the value of Lf,h. This motivatesus to consider one-sided conditions and treat f and âf separately.905.3. RELATIVE SMOOTHNESS AND WEAK CONVEXITYDefinition 5.7 (relative weak convexity). We say that a function f : Rn âR is weakly convex relative to h if f +Ë δdomh is proper and there exists a(possibly negative) constant Ïf,h â R such that f âË Ïf,hh +Ë δC is a convexfunction. We may alternatively say that f is Ïf,h-weakly convex relative toh to make the weak convexity modulus Ïf,h explicit.In accordance with the Euclidean case, when Ïf,h â¥ 0 we may talk aboutrelative convexity, and relative strong convexity when Ïf,h > 0. Consideringpossibly improper functions in Definition 5.7 allows us to identify relativesmoothness as a two-sided relative weak convexity, as we are about to show,regardless of whether a function has full domain or not. This fact extendsthe well-known equivalence between Lipschitz differentiability and the com-bination of weak convexity and weak concavity in the Euclidean setting; seeLemma 5.13 for the details.Lemma 5.8 (relative smoothness and relative weak convexity). Let f :Rn â R be proper. Then, f is smooth relative to h iff both f and âf areweakly convex relative to h.More precisely, if f is Lf,h-smooth relative to h, then both f and âfare (âLf,h)-relatively weakly convex. Conversely, if f and âf are Ïf,h- andÏâf,h-weakly convex relative to h, respectively, then f (as well as âf) isLf,h-smooth relative to h withLf,h = max{|Ïf,h|, |Ïâf,h|} (5.4)(see (5.7) below for a simplified expression without absolute values).Proof. That relative smoothness implies relative weak convexity with thegiven moduli is straightforward (properness of Â±f +Ë δdomh follows from theinclusion dom f â domh). Suppose that Â±f are ÏÂ±f,h-relatively weaklyconvex. First, observe that properness of Â±f +Ë δdomh implies the inclusiondom f â domh. Then, the convexity of Â±f âË ÏÂ±f,hh +Ë δC implies thatLf,hh Â±Ë f +Ë δC = (âÏÂ±f,h + ÏÂ±f,h + Lf,h)h Â±Ë f +Ë δC= â ÏÂ±f,hh +Ë ÏÂ±f,hh +Ë Lf,hh Â±Ë f +Ë δC=[±f âË ÏÂ±f,hh +Ë δC]+ [(Lf,h + ÏÂ±f,h)h]are proper and convex (since Lf,h + ÏÂ±f,h â¥ 0, cf. (5.4)), where the secondidentity uses Lemma 5.1(i) to distribute the coefficients of h. Appealing toDefinition 5.5, f is Lf,h-smooth relative to h.915.3. RELATIVE SMOOTHNESS AND WEAK CONVEXITYThe relative weak convexity moduli ÏÂ±f,h will be henceforth adoptedwhen referring to Assumption 5.2.A1. It will be convenient to normalizethese quantities into pure numbersp±f,h =ÏÂ±f,hLf,hâ [â1, 1]. (5.5)Notice that Lf,h = 0 only when f is affine on C, and in this case we con-ventionally set p±f,h = 0. The comment below will be instrumental inSection 5.4.6.Remark 5.9. If f and âf are Ïf,h- and Ïâf,h-weakly convex relative to h,respectively, then invoking (5.4) and (5.5) yields thatâ2 â¤ pf,h + pâf,h â¤ 0 and â 1 â {pf,h, pâf,h}, (5.6)where the inclusion holds provided Lf,h Ì¸= 0 (as said above, the case Lf,h = 0amounts to f being affine on C). The second inequality owes to the factthat, by definition, both f â Ïf,hh +Ë δC and âf â Ïâf,hh +Ë δC are convexfunctions, and therefore so is their sum[f âË Ïf,hh]+Ë[âf âË Ïâf,hh] +Ë δC = (âÏf,h â Ïâf,h)h +Ë δC= Lf,h(âpf,h â pâf,h)h +Ë δC ,where we used Lemma 5.1(i) together with the fact that f âË f = δdom f andthat dom f â domh â C. In turn, the inclusion in (5.6) follows from (5.4)and the definition (5.5): indeed, as long as Lf,h Ì¸= 0, (5.5) implies that atleast one among pf,h and pâf,h must attain absolute value of one. If thevalue is 1, then combining inequalities (5.5) and (5.6) entails the desiredinclusion. Thus, whenever f is convex (resp. concave), since one can takepf,h = 0 (resp. pâf,h = 0), by virtue of the inclusion in (5.6) it directlyfollows that pâf,h = â1 (resp. pf,h = â1) must hold.Notice that the condition Ïf,h + Ïâf,h â¤ 0 shown in the above remarkyields a simplification in the expression (5.4), for the absolute values can beresolved toLf,h = max{|Ïf,h|, |Ïâf,h|} = max{âÏf,h,âÏâf,h}. (5.7)We now turn to a lemma that guarantees well definedness of Algorithm 3.Lemma 5.10. Suppose that Assumptions 5.2.A2 & 5.2.A5 hold. Then theset argmin{γg + hâ â¨v, · â©} is nonempty for every v â Rn and γ > 0.925.3. RELATIVE SMOOTHNESS AND WEAK CONVEXITYProof. As â¥xâ¥ â â,Î³g(x) + h(x)â â¨v, xâ© â¥ γg(x) + h(x)â â¥vâ¥ â¥xâ¥= â¥xâ¥(Î³g(x)+h(x)â¥xâ¥ â â¥vâ¥)ââ.Hence γg + h â â¨v, · â© is coercive. The desired result then follows fromFact 2.43.Although will not be used throughout this chapter, we point out analternative condition for nonempty argmin{γg + hâ â¨v, · â©}.Proposition 5.11. Suppose that Assumption 5.2 holds with Assumption5.2.A5 replaced by the weaker conditionlimâ¥xâ¥ââxâbdry domhÎ³g(x)+h(x)â¥xâ¥ =â (5.8)for every v â Rn and γ < 1/Lf,h, and that dom gâ©domh Ì¸= â . Then the setargmin{γg + hâ â¨v, · â©} is nonempty.Proof. Observe thatγg + h+ δC = Î³ÏC âË γf + h+ δC= γ ÏCâ¥inf ÏC+Î³(âf +Ë Lf,hh+ δC)proper and convex+(1â γLf,h)>01-coerciveh .Then it follows that γg+h+δC is 1-coercive by Proposition 2.45. Therefore,for every v â Rn the function γg+ h+ δC ââ¨v, · â© is level bounded owing toTheorem 2.44. To fix a notation, letÏ = γg + hâ â¨v, · â© and ÏË = Ï + δC .Then ÏË is 1-coercive. To show that indeed argminÏ is nonempty, let (xk)kâNbe a minimizing sequence for Ï, namely such that Ï(xk) â inf Ï. Weclaim that (xk)kâN is bounded in domh. Then, passing to a subsequence ifnecessary, we may assume without loss of generality that xk â x¯ for somex¯ and the lower semicontinuity of Ï ensures thatÏ(x¯) â¤ lim infkââÏ(xk) = inf Ï â Ï(x¯) = inf Ï,completing the proof.935.3. RELATIVE SMOOTHNESS AND WEAK CONVEXITYFinally we justify the claim. Without loss of generality, by taking sub-sequence if necessary, suppose to the contrary that â¥xkâ¥ â â. Then thecoercivity of ÏË implies that xk â bdryC for k infinitely often, otherwise wewould have xk â C for all but finitely many k andÏ(xk) = ÏË(xk)ââ, as k âââ inf Ï =ââ Ï â¡ â,which contradicts to assumption that domÏ = dom g â© domh Ì¸= â . Apply(5.8) to arrive at a contradiction.We now discuss âtransitivityâ properties of relative smoothness, begin-ning from continuous differentiability. We point out that the following resultis well-known, however, the proof is included for completeness.Proposition 5.12. Suppose that f is Lf,h-smooth relative to h. Then f iscontinuously differentiable on C = int domh.Proof. By assumption, Lf,hh Â± f are finite convex functions on C andtherefore âÌ(Lf,hh Â± f) are nonempty on C in view of Fact 2.26. The sub-differential sum rule Fact 2.16 yields that (âx â C) âÌ(Lf,hh Â± f)(x) =Lf,hâh(x)+ âÌ(Â±f)(x), implying that âÌ(Â±f)(x) must be nonempty and con-sequently f is differentiable at x. In turn the convexity of Lf,hh + f on Cimplies that âf(x) = â(Lf,hh+ f)(x)â Lf,hâh(x) is continuous for everyx â C by using Theorem 2.28.Next we prove a variation of Theorem 2.68, emphasizing the interplaybetween parameters LF and ÏÂ±F .Proposition 5.13 (Lipschitz smoothness from weak convexity). For everyF : Rn â R, the following are equivalent:(i) There exist ÏÂ±F â R such that both F â ÏF j and âF â ÏâF j areproper, convex and lsc.(ii) F is continuously differentiable on Rn and there exist ÏÂ±F â R suchthat ÏF â¥x1 â x2â¥2 â¤ â¨âF (x1)ââF (x2), x1 â x2â© â¤ âÏâF â¥x1 â x2â¥2.(iii) There exists LF â¥ 0 such that F is LF -smooth relative to j.(iv) There exists LF â¥ 0 such that âF is LF -Lipschitz continuous.In particular, assertions (i) and/or (ii) imply assertions (iii) and (iv) withLF = max{âÏF ,âÏâF }, and conversely (iii) and/or (iv) imply (i) and (ii)with ÏÂ±F = âLF .945.3. RELATIVE SMOOTHNESS AND WEAK CONVEXITYProof.â(i) â (iii)â The equivalence between the statements as well as the relationbetween the constants follows from Lemma 5.8 and (5.7).â(i) â (ii)â Apply Proposition 5.12 to see that F is continuously differen-tiable on int dom j = Rn. Function Ï = F â ÏF j is convex, and thereforeits subgradient âÏ = âF â ÏF Id is monotone. This readily shows the firstinequality in assertion (ii). The second inequality will follow from the sameargument applied to the convex function âF â ÏâF j.â(ii) â (iv)â The assumption implies that |â¨âF (x) â âF (y), x â yâ©| â¤LF â¥xâ yâ¥2 with LF = max{|ÏF |, |ÏâF |}. In turn, simple algebra yields0 â¤ â¨â(F + LF j)(x)ââ(F + LF j)(y), xâ yâ© â¤ 2LF â¥xâ yâ¥2.By virtue of Fact 2.59 and Fact 2.66, function F + LF j is convex with(1/2LF )-cocoercive gradient, namely such thatâ¨â(F + LF j)(x)ââ(F + LF j)(y), xâ yâ©â¥ 12LF â¥â(F + LF j)(x)ââ(F + LF j)(y)â¥2,for all x, y â Rn. Expanding the square and rearranging yields the soughtLipschitz inequality â¥âF (x)ââF (y)â¥2 â¤ L2F â¥xâ yâ¥2.â(iv)â (i)â Applying the Cauchy-Schwarz inequality yields for every x1, x2 âRn|â¨âF (x1)ââF (x2), x1 â x2â©| â¤ LF â¥x1 â x2â¥2 .Equivalently â¨â(LF j Â±F )(x1)ââ(LF j Â± f)(x2), x1 â x2â© â¥ 0, This provesconvexity of LF j Â± F , whence the claim by taking ÏÂ±F = âLF .The above lemma can be used to show that a function which is smoothrelative to h is Lipschitz differentiable whenever h is, as shown next. Theproof hinges on the following more general âtransitivityâ property of relativesmoothness.Lemma 5.14. Let h1, h2 : Rn â R be Legendre kernels, and let f : Rn â R.If f is Lf,h1-smooth relative to h1 and h1 is Lh1,h2-smooth relative to h2, thenf is Lf,h2-smooth relative to h2 with Lf,h2 = Lf,h1Lh1,h2.Proof. By definition, domh2 â domh1 â dom f , and Lf,h1h1 Â±Ë f + δC1 andLh1,h2h2 Â±Ë h1 + δC2 are all proper convex functions, where Ci = int domh1,i = 1, 2. If Lf,h1 = 0, then f is affine on C1 as discussed in Lemma 5.8,955.3. RELATIVE SMOOTHNESS AND WEAK CONVEXITYand the claim is trivially true. Suppose that Lf,h1 > 0, and notice thatnecessarily Lh1,h2 > 0 too holds since h1 is strictly convex. Thus,Lf,h1Lh1,h2h2 Â±Ë f +Ë δC2 = Lf,h1Lh1,h2h2 Â±Ë f + δC2= Lf,h1(Lh1,h2h2 â h1) + (Lf,h1h1 Â± f) + δC2are also convex functions, where the second identity uses Lemma 5.1(i) to-gether with the fact that domh2 â domh1. In fact, they are also propersince the domains include domh2 Ì¸= â . By Definition 5.5, this means that fis Lf,h1Lh1,h2-smooth relative to h2.When h2 = j, and by appealing to the equivalence between Lipschitzdifferentiability and smoothness relative to h2 asserted in Lemma 5.13, thefollowing special case is obtained.Corollary 5.15. Suppose that f is Lf,h-smooth relative to h, and that h isLh-Lipschitz differentiable. Then f is Lipschitz differentiable with modulusLf = Lf,hLh.We conclude the section with a result regarding relative weak convexityand smoothness of linear combinations that will be useful in the next section.Lemma 5.16. Suppose that f is smooth relative to h, and let ÏÂ±f,h be theweak convexity moduli of Â±f relative to h. Then, for every α, β â R thefunction Ï = αf +Ë βh is smooth relative to h withÏÏ,h ={|Î±|Ïf,h + β if α â¥ 0,|Î±|Ïâf,h + β if α < 0,(5.9a)ÏâÏ,h ={|Î±|Ïâf,h â β if α â¥ 0,|Î±|Ïf,h â β if α < 0,(5.9b)and|α|Lf,h + |β| â¥ LÏ,h ={max{âÎ² â Î±Ïf,h, β â Î±Ïâf,h} if α â¥ 0,max{β + Î±Ïf,h,âÎ² + Î±Ïâf,h} if α < 0.(5.9c)Proof. If α = 0 the claim is trivial. If α > 0, then for every Ï â R we haveÏ âË Ïh +Ë δC = α[(f âË Ïf,hh +Ë δC) +Ë Î²âÏ+Î±Ïf,hÎ± h],where Lemma 5.1(i) was used to distribute the coefficients of h. Since theterm in round brackets is (proper and) convex, for every Ï â¤ Î±Ïf,h + β one965.4. ALGORITHMIC ANALYSIS TOOLBOXhas that Ï âË Ïh +Ë δC is convex. Clearly, it is also proper, with domainagreeing with domh. If α < 0, the same arguments can be used via theidentityÏ âË Ïh +Ë δC = âÎ±[(âf âË Ïâf,hh +Ë δC) +Ë Î²âÏâÎ±Ïâf,hâÎ± h].The expression for ÏâÏ,h follows by replacing α and β with âÎ± and âÎ²; inturn, the expression for LÏ,h follows from (5.7).5.4 Algorithmic analysis toolboxIn the literature, convergence analysis for nonconvex splitting algorithmstypically revolves around the identification of a âLyapunov potentialâ, namely,a lower bounded function that decreases its value along the iterates. In thissection we will pursue this direction. To simplify the discussion we introducehË = 1γh âË f and fËÎ² = f âË βγh +Ë δdomh . (5.10)In the remainder of this chapter,(âr â R) [r]â = max(0,âr).We shall see frequently the expression 1/[Ïâf,h]â, which satisfies1[Ïâf,h]â={1âÏâf,h , if Ïâf,h < 0;â, otherwise.Properties of hË and fËÎ² are summarized below.Lemma 5.17. Suppose that Assumption 5.2.A1 holds. The following hold:(i) For every β â R, dom fËÎ² = domh and fËÎ² is smooth relative to h withÏÂ±fËÎ² ,h =Î³ÏÂ±f,hâÎ²Î³ and LfËÎ² ,h =1γ max{β â Î³Ïf,h, âÎ² â Î³Ïâf,h}.(5.11)(ii) For every γ < 1/[Ïâf,h]â the function hË is a Legendre kernel withdom hË = domh.Proof.(i) The domain equality statement is obvious. As a linear combination of fand h, invoke Lemma 5.16 to infer that fËÎ² is smooth relative to h with thegiven moduli.(ii) See [1, Theorem 4.1].975.4. ALGORITHMIC ANALYSIS TOOLBOX5.4.1 Parametric minimization modelAs a first step towards the desired goals, as well as to considerably simplifythe discussion, we begin by observing that the iâFRB-update is the result ofa parametric minimization. To this end, we introduce the âmodelâMh-frbÎ³, β :domhà C Ã C â R defined byMh-frbγ, β (w;x, xâ) = Ï(w) + DhË(w, x) + â¨w â x,âfËÎ²(x)ââfËÎ²(xâ)â©(5.12a)= Ï(w) + DhËâfËÎ² (w, x) + DfËÎ² (w, xâ)âDfËÎ² (x, xâ),(5.12b)where the last equality holds due to the well-known three-point identityFact 2.92. Notice that no extended arithmetics are necessary in the aboveformulae due to the restriction (w, x, xâ) â domhÃCÃC which guaranteesthe finiteness of all quantities involved, except possibly Ï(w). Then, addingconstant terms from the x-update in iâFRB yieldsxk+1 â argminwâRn{g(w)+ â¨w,âf(xk)â Î²Î³(âh(xk)ââh(xkâ1))â©+ 1γ Dh(w,yk)}= argminwâdomh{Ï(w)+ hË(w)+ â¨w,âfËÎ²(xk)+ Î²Î³âh(xkâ1)â 1Î³âh(yk)â©}= argminwâdomh{Ï(w)+ hË(w)+ â¨w,âfËÎ²(xk)ââfËÎ²(xkâ1)ââhË(xk)â©}= argminwâdomhMh-frbÎ³,Î² (w;xk,xkâ1),where the second last equality owes to the relation âh(yk) = âh(xk) âÎ³(âf(xk)ââf(xkâ1)) (recall step 1 of iâFRB), motivating the following:Definition 5.18 (iâFRB-operator). The iâFRB-operator with stepsize γ >0 and inertia parameter β â R is Th-frbγ, β : C Ã C â domh defined byTh-frbγ, β (x, xâ) = argminwâdomhMh-frbÎ³, β (w;x, xâ) (5.13)Remark 5.19. We shall formalize in Theorem 5.20(iii) that Th-frbγ, β mapspairs in C Ã C to subsets of C is a consequence of Assumption 5.2.A4, aswe are about to formalize in Theorem 5.20(iii).It follows that the x-update in iâFRB can be compactly expressed asxk+1 â Th-frbγ, β (xk, xkâ1). (5.14)985.4. ALGORITHMIC ANALYSIS TOOLBOXNote that many modelsMh-frbγ, β can be defined whose marginal minimizationwith respect to the first variable results in the same Th-frbγ, β , and all thesediffer by additive terms which are constant with respect to w. Among these,the one given in (5.12b) reflects the tangency conditionMh-frbγ, β (x;x, xâ) = Ï(x) = ÏC(x),for every x, xâ â C. A consequence of this fact and other basic propertiesare summarized next.Theorem 5.20 (basic properties of the model Mh-frbγ, β and the operatorTh-frbγ, β ). Suppose that Assumption 5.2 holds, and let γ < 1/[Ïâf,h]â andβ â R be fixed. The following hold:(i) Mh-frbγ, β (x;x, xâ) = ÏC(x) for all x, xâ â C.(ii) Mh-frbγ, β (w;x, xâ) is level bounded in w for every (x, xâ) â C Ã C. Infact, it is level bounded in w locally uniformly in (x, xâ).(iii) Th-frbγ, β is outer semicontinuous and compact-valued with domTh-frbγ, β =C Ã C and ranTh-frbγ, β â C. Consequently, Th-frbγ, β is locally bounded,in the sense thatâ(x,xâ)âV Th-frbγ, β (x, xâ) is bounded for every compactsubset V â C Ã C.(iv) âhË(x)ââhË(x¯)ââfËÎ²(x) +âfËÎ²(xâ) â âÌÏ(x¯) for every x, xâ â C andx¯ â Th-frbγ, β (x, xâ).(v) If x â Th-frbγ, β (x, x), then 0 â âÌÏ(x).Proof. We start by observing that Lemma 5.10 ensures that Th-frbγ, β (x, xâ)is nonempty for every x, xâ â C; this follows by considering the expression(5.12a) of the model, by observing that, for every x â C, Ï + DhË( · , x) =g + 1γh â hË(x) â â¨âhË(x), · â xâ©. For the same reason, it then follows fromAssumption 5.2.A4 that Th-frbγ, β (x, xâ) â C.(i) Apparent, by considering w = x in (5.12b).(ii) & (iii) Recall that hË is 1-coercive (recall Lemma 5.17), then so is hË+â¨v, Â·â©for any vector v due to Proposition 2.45. In turn for a fixed (x, xâ) â CÃC,Mh-frbÎ³, β (w;x, xâ) â¥ inf ÏC + hË(w) + â¨w â x,âfËÎ²(x)ââf(xâ)ââhË(x)â©â â as â¥wâ¥ â â,995.4. ALGORITHMIC ANALYSIS TOOLBOXwhich means that w 7â Mh-frbγ, β (w;x, xâ) is coercive and equivalently levelbounded. We prove the uniform level boundedness by claiming that for everycompact set V â C Ã C,limâ¥wâ¥â+â(inf(x,xâ)âVMh-frbÎ³, β (w;x, xâ))= +â. (5.15)In turn for every (x, xâ) â V and α â R,{wâ£â£â£â£ inf(x,xâ)âVMh-frbÎ³, β (w;x, xâ) â¤ α}is bounded, (5.16)It then follows from the obvious inclusion{wâ£â£â£Mh-frbÎ³, β (w;x, xâ) â¤ α} â {w â£â£â£â£ inf(x,xâ)âVMh-frbÎ³, β (w;x, xâ) â¤ α},and Theorem 2.44 that the desired (ii) holds. Now we justify the claim(5.15). Notice that for a fixed w â domh and every (x, xâ) â VMh-frbγ, β (w;x, xâ) = Ï(w) + DhË(w, x) + â¨w â x,âfËÎ²(x)ââfËÎ²(xâ)â©= Ï(w) + hË(w)â hË(x) + â¨âfËÎ²(x)ââfËÎ²(xâ)ââhË(x), w â xâ©â¥ inf ÏC + hË(w)â h(x)â (â¥wâ¥+ â¥xâ¥)â¥â¥â¥âfËÎ²(x)ââfËÎ²(xâ)ââhË(x)â¥â¥â¥ .The convexity of hË (recall Lemma 5.17) enforces âhË to be continuous inview of Theorem 2.28, which together with the continuity of hË and that ofâfËÎ² implies thatinf(x,xâ)âVMh-frbÎ³, β (w;x, xâ) â¥ inf ÏC + hË(w)â Ï1 â¥wâ¥ â Ï2,forÏ1 = sup(x,xâ)âVâ¥â¥â¥âfËÎ²(x)ââfËÎ²(xâ)ââhË(x)â¥â¥â¥ ,andÏ2 = sup(x,xâ)âVh(x) + â¥xâ¥â¥â¥â¥âfËÎ²(x)ââfËÎ²(xâ)ââhË(x)â¥â¥â¥ .Recall that hË is 1-coercive by Lemma 5.17. Thereforeinf(x,xâ)âVMh-frbÎ³, β (w;x, xâ) â¥ â¥wâ¥(hË(w)â¥wâ¥ â Ï1 + inf ÏâÏ2â¥wâ¥)ââ,as â¥wâ¥ â â, which proves (5.15).1005.4. ALGORITHMIC ANALYSIS TOOLBOXNext we prove (iii). Invoke Theorem 2.49 to see domTh-frbγ, β = C Ã Cand that Th-frbγ, β is outer semicontinuous and compact-valued; while therange inclusion follows from Assumption 5.2.A4. To see that Th-frbγ, β is lo-cally bounded, observe that for every (x, xâ) â V and the infimum α =infwMh-frbγ, β (w;x, xâ)w â Th-frbγ, β (x, xâ)â w â{wâ£â£â£Mh-frbÎ³, β (w;x, xâ) â¤ Î±}â{wâ£â£â£â£ inf(x,xâ)âVMh-frbÎ³, β (w;x, xâ) â¤ α},so â(x,xâ)âVTh-frbÎ³, β (x, xâ) â{wâ£â£â£â£ inf(x,xâ)âVMh-frbÎ³, β (w;x, xâ) â¤ α},is bounded by (5.16).(iv) Apply the optimality conditions of x¯ â argminMh-frbγ, β ( · ;x, xâ) to seethat0 â â(Ï+ hË+ â¨Â· â x,âfËÎ²(x)ââfËÎ²(xâ)â©)(x¯).Having x¯ â C by assertion (iii) ensure that hË to be smooth around x¯, andtherefore the calculus rule Fact 2.16 applies and yields0 â âÌÏ(x¯) +âhË(x¯)ââhË(x) +âfËÎ²(x)ââfËÎ²(xâ),completing the proof.(v) The inclusion 0 â âÌÏ(x) follows from assertion (iv).5.4.2 The iâFRB-envelopeHaving defined modelMh-frbγ, β and its solution mapping Th-frbγ, β resulted fromparametric minimization, we now introduce the associated value function,which we name iâFRB-envelope.Definition 5.21 (iâFRB-envelope). The envelope associated to iâFRB withstepsize γ > 0 and inertia β â R is the function Ïh-frbÎ³, β : C ÃC â R definedasÏh-frbÎ³, β (x, xâ) = infwâdomhMh-frbÎ³, β (w;x, xâ). (5.17)Theorem 5.22 (basic properties of Ïh-frbÎ³, β ). Suppose that Assumption 5.2holds. Then, for every γ < 1/[Ïf,h]â and β â R the following hold:1015.4. ALGORITHMIC ANALYSIS TOOLBOX(i) Ïh-frbÎ³, β is real-valued and continuous on C Ã C. In fact, it is lower-C1and consequently locally Lipschitz, provided that f, h â C2(C).(ii) For every x, xâ â C and x¯ â Th-frbγ, β (x, xâ)Ïh-frbÎ³, β (x, xâ) =Mh-frbγ, β (x¯;x, xâ)= Ï(x¯) + DhËâfËÎ² (x¯, x) + DfËÎ² (x¯, xâ)âDfËÎ² (x, xâ).(iii) Ïh-frbÎ³, β (x, xâ) â¤ Ï(x) for every x, xâ â C.Proof.(i) In light of Theorem 5.20(ii), Ïh-frbÎ³, β is real-valued and continuous onC Ã C owing to Theorem 2.49(i)&(v) and the continuity of (x, xâ) 7âMh-frbÎ³, β (w;x, xâ) for every w.Suppose now that f and h are both C2 on C. Then the partial gradientâ(x,xâ)Mh-frbÎ³, β (w, x, xâ)=(âfËÎ²(xâ)ââfËÎ²(x) + (â2fËÎ² ââ2hË)(x)(w â x)â2fËÎ²(xâ)(xâ w))exists and is continuous with respect to all its arguments. Moreover, re-call from Theorem 5.20(iii) that Th-frbγ, β is locally bounded. Then for everynonempty compact set V â C Ã C the set T defined byT =â(x,xâ)âVTh-frbÎ³, β (x, xâ) is compact.In turn for every (x, xâ) â V and x¯ â Th-frbγ, β (x, xâ)Ïh-frbÎ³, β (x, xâ) = infwâdomhMh-frbÎ³, β (w;x, xâ) â¤ infwâTMh-frbÎ³, β (w;x, xâ)â¤Mh-frbÎ³, β (x¯;x, xâ) = Ïh-frbÎ³, β (x, xâ),enforcing the representation Ïh-frbÎ³, β (x, xâ) = infwâT Mh-frbγ, β (w;x, xâ). HenceâÏh-frbÎ³, β is a lower-C1 function in the sense of Definition 2.72, and in partic-ular locally Lipschitz continuous by virtue of Fact 2.73.(ii) & (iii) The identity follows by definition; see (5.17) and (5.13). Theinequality follows by considering w = x in (5.17) and (5.12b).1025.4. ALGORITHMIC ANALYSIS TOOLBOX5.4.3 Establishing a merit functionWe now work towards establishing a merit function for iâFRB, starting fromcomparing the values of Ïh-frbÎ³, β (x¯, x) and Ïh-frbÎ³, β (x, xâ), with x¯ â Th-frbγ, β (x, xâ).Owing to Theorem 5.22(iii), we haveÏh-frbÎ³,Î² (x¯, x) â¤ Ï(x¯) = ÏC(x¯)= Ïh-frbÎ³,Î² (x,xâ)â DhËâfËÎ² (x¯, x)âcomplicating termDfËÎ² (x¯, xâ) + DfËÎ² (x,xâ).(5.18)From here two separate cases can be considered, each yielding surprisinglydifferent results. The watershed lies in whether the âcomplicatingâ term ispositive or not:â¢ One case will result in a very straightforward convergence analysis inthe full generality of Assumption 5.2; see Theorem 5.25(A).â¢ The other will necessitate an additional Lipschitz differentiability re-quirement; see Theorem 5.25(B).To supplement Theorem 5.25, special and worst-case scenarios leading tosimplified statements will be given in Section 5.4.6. We remind thatj = 12â¥ · â¥2 (5.19)is the squared Euclidean norm. In what follows, patterning the normaliza-tion of ÏÂ±f,h into p±f,h detailed in Section 5.3, we also introduce the scaledstepsizeα = γLf,h, (5.20)which as a result of the convergence analysis will be confined in the interval(0, 1).5.4.4 Technical lemmasLemma 5.23. Suppose that Assumption 5.2 holds and let γ > 0 and β â Rbe such that fËÎ² = f âË βγh + δdomh is a convex function. Then, for everyx, xâ â C and x¯ â Th-frbγ, β (x, xâ)(Ïh-frbÎ³, β +DfËÎ²)(x¯, x) â¤(Ïh-frbÎ³, β +DfËÎ²)(x, xâ)â DhËâ2fËÎ² (x¯, x). (5.21)1035.4. ALGORITHMIC ANALYSIS TOOLBOXProof. The claimed inequality follows from (5.18) together with the factthat DfËÎ² â¥ 0.Lemma 5.24. Additionally to Assumption 5.2, suppose that fËÎ² is LfËÎ² -Lipschitz differentiable for some LfËÎ² â¥ 0. Then, for every x, xâ â C andx¯ â Th-frbγ, β (x, xâ) we have(Ïh-frbÎ³, β +DLfËÎ² j)(x¯, x) â¤(Ïh-frbÎ³, β +DLfËÎ² j)(x, xâ)âDhËâ2LfËÎ² j(x¯, x). (5.22)Proof. By the three-point identity Fact 9.11 and using (5.12a) in place of(5.12b), inequality (5.18) can equivalently be written asÏh-frbÎ³, β (x¯, x) â¤ ÏC(x¯)= Ïh-frbÎ³, β (x, xâ)âDhË(x¯, x)â â¨x¯â x,âfËÎ²(x)ââfËÎ²(xâ)â©â¤ Ïh-frbÎ³, β (x, xâ)âDhË(x¯, x) +LfËÎ²2 â¥x¯â xâ¥2 +LfËÎ²2 â¥xâ xââ¥2, (5.23)where the second inequality follows from using Youngâs inequality on theinner product and LfËÎ² -Lipschitz differentiability. Rearranging and using thefact that Dj(x, y) =12â¥xâ yâ¥2 yields the claimed inequality.5.4.5 Identifying merit functionsHere is the main result of the section. The convergence analysis revolvesaround the identification of a constant c > 0 determining a lower boundedmerit functionLh-frbγ, β = Ïh-frbÎ³, β + c2γ Dh+Dξ, (5.24)where ξ is a function yet to be determined.Theorem 5.25. Let α be given by (5.20). Suppose that Assumption 5.2holds and consider one of the following scenarios:(A) either fËÎ² is convex (e.g., when Î±pf,hâÎ² â¥ 0) and β > â(1+3Î±pâf,h)/2,in which caseξ = fËÎ² and c = 1 + 2β + 3Î±pâf,h > 0,1045.4. ALGORITHMIC ANALYSIS TOOLBOX(B) or fËÎ² is LfËÎ² -Lipschitz differentiable, h is Ïh-strongly convex, andc = (1 + Î±pâf,h)â2Î³LfËÎ²Ïh> 0,in which case ξ = LfËÎ²j.Then, for Lh-frbγ, β as in (5.24) the following assertions hold:(i) For every x, xâ â C and x¯ â Th-frbγ, β (x, xâ),Lh-frbÎ³, β (x¯, x) â¤ Lh-frbγ, β (x, xâ)â c2γ Dh(x¯, x)â c2γ Dh(x, xâ) (5.25a)andÏC(x¯) â¤ Lh-frbγ, β (x, xâ)â cγ Dh(x¯, x)â c2γ Dh(x, xâ). (5.25b)(ii) inf Lh-frbγ, β = infC Ï.(iii) If either h is strongly convex or domh = Rn, then Lh-frbγ, β is levelbounded provided that ÏC is.Proof.(i)&(ii) In the setting of Theorem 5.25(A), recall that we set c = +2β +3Î±pâf,h > 0. Then, inequality (5.21) can equivalently be written in termsof Lh-frbγ, β asLh-frbγ, β (x¯, x) â¤ Lh-frbγ, β (x, xâ)â DhËâ2fËÎ²â cγ h(x¯, x)âc2Î³ Dh(x¯, x)â c2γ Dh(x, xâ)â¤ Lh-frbγ, β (x, xâ)â c2γ Dh(x¯, x)â c2γ Dh(x, xâ),where the second inequality owes to the fact that DhËâ2fËÎ²â cγ hâ¥ 0, sincehËâ 2fËÎ² â cγh is convex, havinghËâ 2fËÎ² â cγh = 1+2Î²âcÎ³ hâ 3f ==01+2Î²+3Î±pâf,hâcÎ³ h+ 3(âf â Ïâf,hh),the coefficient of h being null by definition of c, and âfâÏâf,hh being convexby definition of the relative weak convexity modulus Ïâf,h, cf. Definition5.7. This proves (5.25a); inequality (5.25b) follows similarly by observingthat0 â¤ DhËâ2fËÎ²â cγ h = DhËâfËÎ² âDfËÎ² âcÎ³ Dh â¤ DhËâfËÎ² âcÎ³ Dh,1055.4. ALGORITHMIC ANALYSIS TOOLBOXso thatLh-frbγ, β (x, xâ)(ii)= Ï(x¯) +â¥ cγ DhDhËâfËÎ² (x¯, x) +â¥0DfËÎ² (x¯, xâ)+ c2γ Dh(x, xâ) â¥ ÏC(x¯).This concludes the proof of assertions (i) and (ii) in the setting of Theorem5.25(A).Under Theorem 5.25(B), recall that we definec = (1 + Î±pâf,h)â2Î³LfËÎ²Ïh> 0.We will pattern the arguments of the previous case, and observe that in-equality (5.22) can equivalently be written in terms of Lh-frbγ, β asLh-frbγ, β (x¯, x) â¤ Lh-frbγ, β (x, xâ)ââ¥0DhËâ2LfËÎ² jâcÎ³ h(x¯, x)â c2γ Dh(x¯, x)â c2γ Dh(x, xâ).Once again, the fact that DhËâ2LfËÎ² jâcÎ³ hâ¥ 0 owes to the convexity of hË â2LfËÎ²j âcÎ³h on domh, havinghËâ 2LfËÎ²j âcÎ³h =1âcÎ³ hâ f â 2LfËÎ²j==2LfËÎ²/Ïhâ¥01+Î³Ïâf,hâcÎ³ (convexhâ Ïhj) + (convexâ f â Ïâf,hh) +( =01+Î³Ïâf,hâcÎ³ Ïh â 2LfËÎ²)jaltogether proving (5.25a). Similarly, inequality (5.25b) follows from (5.23)together with the fact that DhËâLfËÎ² jâ¥ DhËâ2LfËÎ² j â¥cÎ³ Dh, as shown above,havingÏC(x¯)(5.23)â¤ ( Ïh-frbÎ³, βLh-frbγ, β âD c2γ h+LfËÎ²j)(x, xâ)âDhË(x¯, x)+LfËÎ²2 â¥x¯â xâ¥2 +LfËÎ²2 â¥xâ xââ¥2= Lh-frbγ, β (x, xâ)â c2γ Dh(x, xâ)âDhËâLfËÎ² j(x¯, x)â¤ Lh-frbγ, β (x, xâ)â c2γ Dh(x, xâ)â cγ Dh(x¯, x).(iii) By contraposition, suppose that Lh-frbγ, β is not level bounded, and con-sider an unbounded sequence (xk, xâk )kâN such thatLh-frbγ, β (xk, xâk ) â¤ â,1065.4. ALGORITHMIC ANALYSIS TOOLBOXfor some â â R. Then, it follows from (5.25b) thatinf ÏC +cγ Dh(x¯k, xk) +c2γ Dh(xk, xâk )â¤ ÏC(x¯k) + cγ Dh(x¯k, xk) + c2γ Dh(xk, xâk ) â¤ â,and in particular both (Dh(x¯k, xk))kâN and (Dh(xk, xâk ))kâN are bounded.Moreover, it follows from Lemma 2.96 that if (xâk )kâN is unbounded then sois (xk)kâN, and similarly unboundedness of (xk)kâN implies that of (x¯k)kâN.Since at least one among (xk)kâN and (xâk )kâN is unbounded, it follows that(x¯k)kâN is unbounded. Noticing that this sequence is contained in [ÏC â¤ â],we conclude that ÏC is not level bounded.As clarified in the statement of Theorem 5.25(A), convexity of fËÎ² can beenforced by suitably choosing γ and β without imposing additional require-ments on the problem. However, an unusual yet reasonable condition oninertial parameter β may be necessary.Remark 5.26 (negative inertia). In order to furnish Theorem 5.25(A),we shall see soon that β â¤ 0 may be required; see Section 5.4.6. Suchassumption, although more pessimistic, coincides with a recent conjectureby Dragomir et al. [36, §4.5.3], which states that inertial methods withnonadaptive coefficients fail to converge in the relative smoothness setting,and provides an alternative perspective to the same matter through the lensof the convexity of fËÎ².Unlike Theorem 5.25(A), however, additional assumptions are neededfor the Lipschitz differentiable case of Theorem 5.25(B). This is because therequirement is equivalent to smoothness relative to the Euclidean Bregmankernel j, while Assumption 5.2 prescribes bounds only relative to h.Remark 5.27 (conditions for Theorem 5.25(B)). Under Assumption 5.2,one has that fËÎ² is Lipschitz differentiable with modulus LfËÎ² under either oneof the following conditions:(B1) either âh is Lh-Lipschitz, and LfËÎ² =Lhγ max{Î²âÎ±pf,h,âÎ²âÎ±pâf,h},(B2) or β = 0 and âf is Lf -Lipschitz, in which case LfËÎ² = Lf .Recalling that fËÎ² = f â βγh, the second condition is tautological. In case âhis Lh-Lipschitz, the claim follows from (5.11) together with Corollary 5.15.1075.4. ALGORITHMIC ANALYSIS TOOLBOX5.4.6 Simplified boundsIn this section we provide bounds that only discern whether f is convex,concave, or neither of the above. As discussed in Remark 5.9, these casescan be recovered by suitable combinations of the coefficients p±f,h â {0,±1},and thus lead to easier, though possibly looser, bounds compared to thosein Theorem 5.25. We will also avail ourselves of the estimates of LfËÎ² inRemark 5.27 to discuss the cases in which fËÎ² is Lipschitz differentiable. Tosimplify the exposition, we may provide smaller estimates of the coefficientc in Theorem 5.25, owing to the fact that replacing c with every câ² â (0, c]does not affect the validity of the statement and only causes the inequalities(5.25) to be possibly looser.Without distinguishing between upper and lower relative bounds, when-ever f is Lf,h-smooth relative to h as in Assumption 5.2 one can considerÏÂ±f,h = âLf,h or, equivalently, pf,h = pâf,h = â1. Plugging these valuesinto Theorem 5.25 yields the following.Corollary 5.28 (worst-case bounds). Suppose that Assumption 5.2 holds.All the claims of Theorem 5.25 hold when γ > 0, β â R and c > 0 are suchthat(A) either â1/2 < β < 0 and γ â¤ (1/Lf,h)min{âÎ², (1 + 2β â c)/3}, inwhich case ξ = fËÎ².(B1) or h is Ïh-strongly convex and Lh-Lipschitz differentiable, |β| < Ïh/2Lhand γ â¤ (1/Lf,h)[(Ïh(1 â c) â 2Lh|Î²|)/(Ïh + 2Lh)], in which caseξ = (Lh/γ)(α+ |β|)j.(B2) or h is Ïh-strongly convex, âf is Lf -Lipschitz continuous, β = 0 andγ â¤ Ïh(1â c)/(ÏhLf,h + 2Lf,h), in which case ξ = Lf j.Proof. Setting p±f,h = â1 in Theorem 5.25, one has:(A) The bounds in the statement of Theorem 5.25(A) read 0 < c = 1+2Î²â3Î± and β â¤ âÎ±. Expressed in terms of α = γLf,h, the claimed bounds on γare obtained. In turn, imposing α > 0 results in the claimed bounds on β.(B1) & (B2) The two subcases refer to the corresponding items in Remark5.27. We shall show only the first one, as the second one is a trivial adapta-tion after observing that Lf = Lf,hLh by virtue of Corollary 5.15. The valueof LfËÎ² as in Remark 5.27(B1) reduces to LfËÎ² = (Lh/γ)(α + |β|). Pluggedinto Theorem 5.25(B) yields 0 < c = 1âÎ±â 2(α+ |Î²|)Lh/Ïh, implying that1085.4. ALGORITHMIC ANALYSIS TOOLBOXγ = α/Lf,h is bounded as in assertion (B1). Imposing α > 0 yields also theclaimed bounds on β.When f is convex on C, Ïf,h = 0 can be considered resulting in pf,h = 0and pâf,h = â1.2Corollary 5.29 (bounds when f is convex). Suppose that Assumption 5.2holds and that f is convex. All the claims of Theorem 5.25 remain valid ifγ > 0, β â R and c > 0 are such that(A) either â1/2 < β â¤ 0 and γ â¤ (1/Lf,h)[(1 + 2β â c)/3], in which caseξ = fËÎ².(B1) or h is Ïh-strongly convex and Lh-Lipschitz differentiable,|β| < Ïh2Lh and γ â¤ 1Lf,h min{Ïh(1âc)+2LhÎ²Ïh+2Lh, Ïh(1âc)â2LhÎ²Ïh },in which case ξ = (Lh/γ)max{β, Î±â β}j.(B2) or h is Ïh-strongly convex, âf is Lf -Lipschitz, β = 0, and γ â¤ (1 âc)Ïh/(ÏhLf,h + 2Lf ), in which case ξ = Lf j.Proof. We will pattern the proof of Corollary 5.28, and omit the proof ofassertion (B2) which is an easy adaptation of that of assertion (B1). Settingpf,h = 0 and pâf,h = â1 in Theorem 5.25, one has:(A) The bounds in the statement of Theorem 5.25(A) read 0 < c = 1 +2Î²â 3α and β â¤ 0. This readily yields the bound γLf,h = α â¤ 1+2Î²âc3 afterreplacing c = 1 + 2β â 3α by c â¤ 1 + 2β â 3α with an abuse of notationon c, under which inequalities in the desired Theorem (i) hold with possiblylooser bounds. In turn, the condition α > 0 then constrains β â (â1/2, 0],as claimed.(B1) The value of LfËÎ² in Remark 5.27(B1) reduces toLhγ max{Î²,Î±âÎ²}.Plugged into Theorem 5.25(B) yields 0 < c = (1âÎ±)â2Lh/Ïhmax{Î², Î±âÎ²},and in particular {c â¤ (1â Î±)â 2Î²Lh/Ïh,c â¤ (1â Î±)â 2(Î±â Î²)Lh/Ïh.In terms of γ = α/Lf,h, this results in the bound for γ as in assertion (B1).In turn, imposing α > 0 results in the claimed bounds on β.2This also covers the case in which f is affine on C, although a tighter pâf,h = 0 couldbe considered in this case and improve the range to β â (â1/2, 0] and every γ > 0.1095.5. CONVERGENCE ANALYSISSimilarly, when f is concave (that is, âf is convex) on C, then Ïâf,h = 0can be considered, resulting in pf,h = â1 and pâf,h = 0.Corollary 5.30 (bounds when f is concave). Suppose that Assumption 5.2holds and that f is concave. All the claims of Theorem 5.25 remain valid ifγ > 0, β â R and c > 0 are such that(A) either (câ 1)/2 â¤ β < 0 and γ â¤ âÎ²/Lf,h, in which case ξ = fËÎ².(B1) or h is Ïh-strongly convex and Lh-Lipschitz differentiable,â (1âc)Ïh2Lh â¤ β <Ïh2Lhand γ â¤ 1Lf,hÏh(1âc)â2LhÎ²2Lh,in which case ξ = Lhγ max{α+ Î²,âÎ²}j.(B2) or h is Ïh-strongly convex, f is Lf -Lipschitz differentiable, β = 0 andγ â¤ Ïh(1â c)/(2Lf ), in which case ξ = Lf j.Proof. Set pf,h = â1 and pâf,h = 0 in Theorem 5.25. A similar argumentas in Corollary 5.29 completes the proof:(A) From Theorem 5.25(A) we obtain âÎ± â β â¥ 0 and c = 1 + 2β > 0.Recalling that α = γLf,h must be strictly positive, the bound on γ and onβ as in the statement are obtained.(B1) Remark 5.27(B1) yields the estimate LfËÎ² = Lh/γmax{α + Î²,âÎ²},which plugged into the statement of Theorem 5.25(B) gives 0 < c = 1 â2Lh/Ïhmax{Î±+ Î²,âÎ²}. Therefore,{c â¤ 1 + 2Î²Lh/Ïh,c â¤ 1â 2(α+ Î²)Lh/Ïh,which in terms of γ = α/Lf,h results in the bound on γ and lower boundon β as in assertion (B1). The upper bound on β follows from ensuringÏh â 2Lhβ > 0, which is necessary for the bound γ > 0. Once again, thecase (B2) is an easy adaptation of (B1).5.5 Convergence analysisIn this section we study the behavior of sequences generated by iâFRB.Although some basic convergence results can be derived in the full general-ity of Assumption 5.2, establishing local optimality guarantees of the limitpoint(s) will ultimately require an additional full domain assumption.1105.5. CONVERGENCE ANALYSISAssumption 5.31. The function h has full domain, that is, C = Rn.Assumption 5.31 is standard for nonconvex splitting algorithms in arelative smooth setting. To the best of our knowledge, the question regardingwhether this requirement can be removed remains open; see, e.g., [84] andthe references therein.5.5.1 Function value convergenceWe begin with the convergence of merit function value.Theorem 5.32 (function value convergence of iâFRB). Let (xk)kâN be asequence generated by iâFRB (Algorithm 3) in the setting of Theorem 5.25.Then,(i) It holds thatLh-frbγ, β (xk+1, xk) â¤ Lh-frbγ, β (xk, xkâ1)â c2γ Dh (xk+1, xk)â c2γ Dh (xk, xkâ1) . (5.26)In particular,ââk=0Dh (xk, xkâ1) < +â andLh-frbγ, β (xk, xkâ1)â Ïâ as k â +â,for some Ïâ â¥ inf ÏC .If Assumption 5.31 also holds, then:(ii) If ÏC is level bounded, then (xk)kâN is bounded.(iii) Let â¦ be the set of cluster points of (xk)kâN. Then, Ï is constant on â¦with value Ïâ, and for every xâ â â¦ it holds that xâ â Th-frbγ, β (xâ, xâ)and 0 â âÌÏ(xâ).Proof.(i) Recall from Theorem 5.25 that the inequality (5.26) holds and thatinf Lh-frbγ, β = inf ÏC > ââ, from which convergence of (Lh-frbγ, β (xk, xkâ1))kâNreadily follows. In turn, telescoping (5.26) shows thatâkâNDh(xk, xkâ1) isfinite.(ii) From Theorem 5.32(i), Lh-frbγ, β (xk+1, xk) â¤ Lh-frbγ, β (x0, xâ1) holds for ev-ery k. Then boundedness of (xk)kâN is implied by level boundedness ofLh-frbγ, β ; see Theorem 5.25(iii).1115.5. CONVERGENCE ANALYSIS(iii) Suppose that a subsequence (xkj )jâN converges to a point xâ, then sodo the subsequences (xkj±1)jâN by Theorem 5.32(i) and Lemma 2.94. Sincexkj+1 â Th-frbγ, β (xkj , xkjâ1), by passing to the limit, osc of Th-frbγ, β (recall The-orem 5.20(iii)) implies that xâ â Th-frbγ, β (xâ, xâ). Invoking Theorem 5.20(v)yields the stationarity condition 0 â âÌÏ(xâ). Moreover, by continuity ofLh-frbγ, β one hasÏâ(def)= limkâ+âLh-frbÎ³, β (xk, xkâ1) = Lh-frbγ, β (xâ, xâ) = Ï(xâ),where the last equality follows from Theorem 5.22(ii), owing to the inclusionxâ â Th-frbγ, β (xâ, xâ) (and the fact that DÏ(x, x) = 0 for every differentiablefunction Ï). From the arbitrarity of xâ â â¦ we conclude that Ï â¡ Ïâ onâ¦.Remark 5.33. The full domain assumption on h in Theorem 5.32(ii) isstronger than necessary, but suffices to our purposes. The proof invokes thelevel boundedness of Lh-frbγ, β via Theorem 5.25(iii), which hinges on the prop-erty that whenever (xk)kâN â C is bounded and (xâk )kâN â C is unbounded,(Dh(xk, xâk ))kâN too is unbounded. As such, Theorem 5.32(ii) remains validfor every h, possibly without full domain, as long as the induced Bregmandistance Dh(w, x) is level bounded in x locally uniformly in w.It is now possible to demonstrate the necessity of some of the bounds onthe stepsize that were discussed in Section 5.4.6, by showing that Dh(xk+1, xk)may otherwise fail to vanish. Note that, for β = 0, the following counterex-ample constitutes a tightness certificate for the bound γ < 1/3Lf derivedin [92] in the noninertial Euclidean case.Example 5.34. The bound α = γLf,h < (1 + 2β)/3 is tight even in theEuclidean case. To see this, consider g = δ{±1} and for a fixed L > 0 letf(x) = Lh(x) = L2 x2. Then, one has Lf,h = Ïf,h = L and Ïâf,h = âL. Forγ < 1/L = 1/[Ïâf,h]â, it is easy to see thatTh-frbγ, β (x, xâ) = â sgn(âfËÎ²(x)ââfËÎ²(xâ)ââhË(x))= sgn((1â 2α+ β)x+ (Î±â Î²)xâ)(with sgn 0 = {±1}), where the first equality follows from (5.12a) and (5.13).Let xâ1 = â1, x0 = 1. Suppose that α â¥ (1 + 2β)/3, then ((â1)k)kâN is asequence generated by iâFRB for which Dh(xk+1, xk) â¡ 2 Ì¸â 0.1125.5. CONVERGENCE ANALYSISAs a consequence of Theorem 5.32(i), the condition Dh(xk+1, xk) â¤ εis satisfied in finitely many iterations for every tolerance ε > 0. Whilethis could be used as termination criterion, in the generality of Assump-tions 5.2 and 5.31 there is no guarantee on the relaxed stationarity measuredist(0, âÌÏ(xk+1)), which through Lemma 9.27(iv) can only be estimated asdist(0, âÌÏ(xk+1)) â¤ â¥vk+1â¥with vk+1 = âhË(xk)ââhË(xk+1)ââfËÎ²(xk) +âfËÎ²(xkâ1),(5.27)where hË and fËÎ² are as in (5.10).Lemma 5.35 (termination criteria). Suppose that Assumption 5.31 holds,and let (xk)kâN be a sequence generated by iâFRB (Algorithm 3) in thesetting of Theorem 5.25, and let vk+1 be defined by (5.27). If Ï is levelbounded, then vk+1 â 0. Thus, for every ε > 0 the condition â¥vk+1â¥ â¤ ε issatisfied for all k large enough and guarantees dist(0, âÌÏ(xk+1)) â¤ ε.Proof. Theorem 5.32 implies that (xk)kâN is contained in a compact subset ofC = Rn. Recall from Theorem 5.32(i) that Dh(xk+1, xk)â 0, which impliesthrough the boundedness of (xk)kâN, Assumption 5.31 and the consequentialLemma 2.95 that xk+1 â xk â 0. In turn, vk+1 â 0 holds by uniformcontinuity of âhË and âfËÎ² on the aforementioned compact set.Remark 5.36. An alternative termination criteria can be made when hsatisfies uniform convexity, which is not treated in this thesis but can befound in [96, Lemma 5.4].5.5.2 Global convergenceIn this subsection we work towards the global sequential convergence ofiâFRB under the KL property introduced Definition 2.78. In the remainder,we will make use of the norm ||| · ||| on the product space RnÃRn defined as|||(x, y)||| = â¥xâ¥+ â¥yâ¥.Theorem 5.37 (sequential convergence of iâFRB). Suppose that Assump-tion 5.31 holds, and let (xk)kâN be a sequence generated by iâFRB (Algorithm3) in the setting of Theorem 5.25. Assume in addition the following:A1 Ï is level bounded.A2 f, h are twice continuously differentiable and â2h is positive definite.1135.5. CONVERGENCE ANALYSISA3 Ï, h and f are semialgebraic functions (recall Definition 2.79).Thenââk=0 â¥xk+1 â xkâ¥ < +â and there exists xâ with 0 â âÌÏ(xâ) suchthat xk â xâ as k â +â.Proof. Let (âk â N) zk = (xk+1, xk, xkâ1), and let â¦ be the set of clusterpoints of (zk)kâN. Define for Ï, x, xâ â RnFh-frbγ,β (Ï, x, xâ) =Mh-frbγ, β (Ï;x, xâ) + c2γ Dh(x, xâ) +Dξ(x, xâ).Set (âk â N) δk = Fh-frbγ,β (zk) for simplicity. Then by invoking Theorem5.32,δk = Lh-frbγ, β (xk, xkâ1), δk â Ïâ decreasingly,and dist(xk,â¦)â 0 as k â +â.Assume without loss of generality that (âk â N) δk > Ïâ, otherwise wewould have (âk0 â N) xk0 = xk0+1 due to Theorem 5.32(i), from whichthe desired result readily follows by simple induction. Thus (âk0 â N)(âk â¥ k0) zk â â¦r â© [0 < Fh-frbγ,β â Ïâ < η]. Appealing to Theorem 5.32(iii)and Theorem 5.20(i) yields that Fh-frbγ,β equals to Ïâ on the compact set â¦.Note that Fh-frbγ,β satisfies the KL property under Assumption 5.37.A3; see,e.g., [3, §4.3]. In turn, appealing to Assumption 5.37.A3 and Lemma 2.85implies that there exists a concave Ï â Ψη such that for k â¥ k01 â¤ Ïâ² (δk â Ïâ) dist(0, âFh-frbÎ³,Î² (zk)). (5.28)Define (âk â N)uk = â2hË(xk)(xk â xk+1) +â2fËÎ²(xk)(xk+1 â xk)+ c2Î³(âh(xk)ââh(xkâ1))+âfËÎ²(xkâ1)ââfËÎ²(xk) +âÎ¾(xk)ââÎ¾(xkâ1),vk = â2fËÎ²(xkâ1)(xk â xk+1) + c2Î³â2h(xkâ1)(xkâ1 â xk)+â2Î¾(xkâ1)(xkâ1 â xk).Applying subdifferential calculus to âFh-frbÎ³,Î² (zk) yields thatâFh-frbÎ³,Î² (zk) =(âÏ(xk+1) +âhË(xk+1)ââhË(xk)+âfËÎ²(xk)ââfËÎ²(xkâ1))Ã {uk} Ã {vk},1145.5. CONVERGENCE ANALYSISwhich together with Lemma 9.27(iv) entails that(0, uk, vk) â âFh-frbÎ³,Î² (zk).In turn, Assumption 5.37.A2 implies that there exists M > 0 such thatdist(0, âFh-frbÎ³,Î² (zk))â¤M |||(xk+1 â xk, xk â xkâ1)|||. (5.29)Finally, we show that (xk)kâN is convergent. For simplicity, define(âk, l â N) âk,l = Ï (δk â Ïâ)â Ï (δl â Ïâ) .Then, combining (5.28) and (5.29) yields1 â¤MÏâ²(Î´k â Ïâ)|||(xk+1 â xk, xk â xkâ1)|||â¤ Mâk,k+1Î´kâÎ´k+1 |||(xk+1 â xk, xk â xkâ1)|||â¤ 2Î³Mâk,k+1c(Dh(xk+1,xk)+Dh(xk,xkâ1)) |||(xk+1 â xk, xk â xkâ1)|||â¤ 2Î³Mâk,k+1|||(xk+1âxk,xkâxkâ1)|||cÏâ¥xk+1âxkâ¥2+cÏâ¥xkâxkâ1â¥2â¤ 4Î³Mâk,k+1cÏ|||(xk+1âxk,xkâxkâ1)||| ,where the second inequality is implied by concavity of Ï, the third onefollows from (5.26), and the fourth one holds because Ï > 0 is the strongconvexity modulus of h on a convex compact set that contains all the iterates.Hence,â¥xk â xkâ1â¥ â¤ |||(xk+1 â xk, xk â xkâ1)||| â¤ 4Î³McÏ âk,k+1. (5.30)Summing (5.30) from k = k0 to an arbitrary l â¥ k0 + 1 yields thatlâk=k0â¥xk â xkâ1â¥ â¤ 4Î³McÏ âk0,l â¤ 4Î³McÏ Ï (δk0 â Ïâ)= 4Î³McÏ Ï(Lh-frbÎ³, β (xk0 , xk0â1)â Ïâ),where the second inequality holds as Ï â¥ 0, from which one sees thatââk=0 â¥xk+1 â xkâ¥ is finite as l is arbitrary. A similar procedure showsthat (xk)kâN is Cauchy, which together with Theorem 5.32(iii) entails therest of the statement.Remark 5.38. Compared to the Lipschitz smooth case considered in [92],the twice continuous differentiability assumption in Theorem 5.37 is a tech-nicality for finding upper bound on â¥(uk, vk)â¥, which consists of differenceof gradients, as multiple of â¥xk â xkâ1â¥; see also [1, Thm. 5.7] for a similarassumption. We delay its relaxation for future research.1155.5. CONVERGENCE ANALYSIS5.5.3 Convergence ratesHaving established convergence of iâFRB, we now turn to its rate. Recallthat a function is said to have KL exponent θ â [0, 1) if it satisfies the KLproperty (recall Definition 2.78) and there exists a desingularizing functionof the form Ï(t) = ct1âÎ¸ for some c > 0.Lemma 5.39. Let (ek)kâN be a monotonically decreasing sequence in R++converging to 0. Assume further that there exist natural numbers k0 â¥ l0 â¥ 1such that for every k â¥ k0ekâl0 â ek â¥ Ce2θk ,where C > 0 is a constant and θ â [0, 1). Then the following hold:(i) If θ = 0, then ek â 0 in finite iterations.(ii) If 0 < θ â¤ 1/2, then there exist C1 > 0 and 0 â¤ Q < 1 such that forevery k â¥ k00 < ek â¤ C1Qk.(iii) If 1/2 < θ < 1, then there exists C2 > 0 such that for every k â¥ k0+l0,0 â¤ ek â¤ C2(k â l0 + 1)â12Î¸â1 .Proof. See [27, Lemma 10].Theorem 5.40 (function value and sequential convergence rate). Supposethat all the assumptions in Theorem 5.37 are satisfied, and follow the nota-tion therein. Define (âk â N) ek = Lh-frbγ, β (xk+1, xk)â Ïâ and for Ï, x, xâ âRnFh-frbÎ³,Î² (Ï, x, xâ) =Mh-frbγ, β (Ï;x, xâ) + c2γ Dh(x, xâ) +Dξ(x, xâ).Assume in addition that Fh-frbγ,β has KL exponent θ â [0, 1) at (xâ, xâ, xâ).Then the following hold:(i) If θ = 0, then ek â 0 and xk â xâ after finite steps.(ii) If θ â (0, 1/2], then there exist c1, cË1 > 0 and Q1, QË1 â [0, 1) such thatfor k sufficiently large,ek â¤ cË1QËk1 and â¥xk â xââ¥ â¤ c1Qk1.1165.5. CONVERGENCE ANALYSIS(iii) If θ â (1/2, 1), then there exist c2, cË2 > 0 such that for k sufficientlylarge,ek â¤ cË2kâ12Î¸â1 and â¥xk â xââ¥ â¤ c2kâ1âÎ¸2Î¸â1 .Proof. Assume without loss of generality that Fh-frbγ,β has desingularizingfunction Ï(t) = t1âÎ¸/(1 â θ) and let (âk â N) δk =ââi=k â¥xi+1 â xiâ¥. Weclaim that(âk â¥ k0 + 1) δk â¤ 4Î³M(1âÎ¸)cÏe1âÎ¸k . (5.31)Indeed, summing (5.30) from every k â¥ k0 to l â¥ k + 1 and passing l toinfinity giveÎ´kâ1 =ââi=kâ¥xi â xiâ1â¥ â¤ 4Î³McÏ Ï(Lh-frbÎ³, β (xk, xkâ1)â Ïâ)= 4Î³McÏ Ï (ekâ1) ,from which the desired claim readily follows. It is routine to see that thedesired sequential rate can be implied by those of (ek)kâN through (5.31);see, e.g., [94, Theorem 5.3], therefore it suffices to prove convergence rate of(ek)kâN.Recall from Theorem 5.32(i) that (ek)kâN is a decreasing sequence con-verging to 0. Then invoking the KL exponent assumption yieldseÎ¸kâ1 = [Fh-frbγ,β (xk+1, xk, xkâ1)â Ïâ]Î¸ â¤ dist(0, âFh-frbÎ³,Î² (xk+1, xk, xkâ1)),where the first equality holds due to Theorem 5.22(ii), which together with(5.29) implies thateθk â¤ eÎ¸kâ1 â¤M |||(xk+1 â xk, xk â xkâ1)|||. (5.32)Appealing again to Theorem 5.32(i) givesekâ1 â ek = Lh-frbγ, β (xk, xkâ1)â Lh-frbγ, β (xk+1, xk)â¥ c2γ [Dh(xk+1, xk) + Dh(xk, xkâ1)]â¥ cÏ8Î³ |||(xk+1 â xk, xk â xkâ1)|||2 â¥ cÏ8Î³M2 e2θk ,where the last inequality is implied by (5.32). Then Lemma 5.39 justifiesthe desired rate of (ek)kâN.1175.6. CONCLUSION5.6 ConclusionThis work contributes a mirror inertial forward-reflected-backward splittingalgorithm (BiFRB), extending the forward-reflected-backward method pro-posed in [62] to the nonconvex and relative smooth setting. We have shownthat the proposed algorithms enjoy pleasant properties akin to other split-ting methods in the same setting. However, our methodology deviates fromtradition through the BiFRB-envelope, an envelope function defined on aproduct space that takes inertial terms into account, which, to the best ofour knowledge, is the first of its kind and thus could be instrumental forfuture research. This approach also requires the inertial parameter to benegative, which coincides with a recent result [36] regarding the impossi-bility of accelerated non-Euclidean algorithms under relative smoothness.Thus, it is interesting to see whether an explicit example can be constructedto prove the sharpness of such restrictive assumption. It is also worth ap-plying our technique to other two-stage splitting methods, such as Tsengâsmethod, to obtain similar extensions.118Chapter 6Every proximal operator isthe resolvent of a levelproximal subdifferential6.1 IntroductionRecall that a function f is said to be 1/λ-hypoconvex for λ > 0, if f + Î»â1jis convex wherej = 12 â¥Â·â¥2 .A fundamental fact in variational analysis states that every proximal opera-tor of a hypoconvex function is the resolvent of the associated Mordukhovichlimiting subdifferential.Fact 6.1. Let λ > 0 and let f : Rn â R be proper, lsc and 1/λ-hypoconvex.ThenPλf = (Id+Î»âf)â1 , (6.1)where âf denotes the Mordukhovich limiting subdifferential operator of f ;see Definition 2.10.Proof. By the definition of Pλf (recall Definition 2.50), v â Pλf(x) amountsto(ây â Rn) (f + Î»â1j)(y)â 12Î»â¨y, xâ© â¥ (f + Î»â1j)(v)â 12Î»â¨v, xâ©,implying that v is a minimizer of the convex function y 7â (f + Î»â1j)(y)â12Î»â¨y, xâ©. Then appealing to optimality condition Fact 2.14 yields0 â â(f + Î»â1j)(v)â Î»â1x = âf(v) + Î»â1v â Î»â1xâ x â (Id+Î»âf) (v)â v â (Id+Î»âf)â1 (x),as desired.1196.1. INTRODUCTIONThe resolvent representation (6.1) is the bridge between subdifferentialand proximal operators, and thus plays a crucial role in variational analy-sis and optimization. However, (6.1) fails beyond the usual hypoconvexitysetting.(Id+Î»âf)â1 PλfFigure 6.1: An illustration of Example 6.2.Example 6.2. Let f : Râ R be the one-dimensional zero ânormâ function,given by f(0) = 0 and f(x) = 1 if x Ì¸= 0. Thenâpf(x) = âÌf(x) = âf(x) ={R, if x = 0;{0}, if x Ì¸= 0.Let λ > 0. Then (â|x| < â2Î») Pλf(x) = {0} â {0, x} = (Id+Î»âf)â1 (x);see Figure 6.1 for an illustration.Proof. See the more comprehensive Example 6.13.We see from Example 6.2 that resolvents of all classic subdifferentialsare too big to capture proximal operators, leading to the question below:Which subdifferential completely represents proximal operator?The goal of this chapter is to resolve this question completely. We investigatethe level proximal subdifferential of a functionâa refinement of the classicproximal subdifferential whose resolvent always captures the proximal oper-ator. Our main results are summarized below:R1 We introduce the level proximal subdifferential in Definition 6.4 andshow in Theorem 6.5 that every proximal operator is the resolvent ofa level proximal subdifferential.1206.2. LEVEL PROXIMAL SUBDIFFERENTIALR2 We explore in Proposition 6.7 various fundamental properties of levelproximal subdifferential.R3 The relation between the level proximal subdifferential and classic sub-differentials in variational analysis (recall Section 2.2) is investigatedin Theorem 6.10.We note that the level proximal subdifferential first appeared informallyin [79, Equation (2.13)] by Rockafellar in 2021. Therefore, having claimedprematurely the discovery of level proximal subdifferential in [95], we pointout that the credit belongs to Rockafellar. Nevertheless, throughout thisthesis, we shall adopt the convention in [95] for convenience, since no termi-nology was given in [79].The rest of this chapter is organized as follows. We introduce the levelproximal subdifferential and study its resolvent in Section 6.2, followed byexamples in Section 6.4. Fundamental properties of the level proximal sub-differential, its relationship with classic subdifferentials as well as sum rulesare investigated in Section 6.3.6.2 Level proximal subdifferentialBefore introducing the level proximal subdifferential, we show that the con-verse of Fact 6.1 holds, which shall indicate once more the urgent need for anew subdifferential to represent proximal operators in full generality â thelimiting subdifferential representation Pλf = (Id+Î»âf)â1 works only if theunderlying function f is hypoconvex!Theorem 6.3. Let f : Rn â R be proper, lsc and prox-bounded with thresh-old λf â (0,â]. Then for every λ â (0, λf )Pλf = (Id+Î»âf)â1 â f + 12λ â¥Â·â¥2 is convex. (6.2)Proof. Suppose that Pλf = (Id+Î»âf)â1. Then appealing to Theorem 2.60yields that (Id+Î»âf)â1 is monotone, and so is the set-valued operatorId+Î»âf by Proposition 2.58. In turn â((1/2Î») â¥Â·â¥2 + f)= (1/λ) Id+âf ismonotone, where the first equality owes to Fact 2.16. Invoking Fact 2.59implies the right hand side of (6.2). The converse holds due to Theo-rem 2.60.We are now ready to define the level proximal subdifferential of a func-tion.1216.2. LEVEL PROXIMAL SUBDIFFERENTIALDefinition 6.4 (level proximal subdifferential). Let f : Rn â R be proper,x â dom f and let λ > 0. We say that u â Rn is a λ-level proximalsubgradient of f at x, denoted by u â âÎ»p f(x), if(ây â Rn) f(y) â¥ f(x) + â¨u, y â xâ© â 12λ â¥y â xâ¥2 . (6.3)Amazingly, every proximal operator is the resolvent of a level proximalsubdifferential operator, regardless of hypoconvexity.Theorem 6.5 (resolvent representation of Pλf). Let f : Rn â R be proper,lsc and prox-bounded with threshold λf â (0,â]. Then(â0 < λ < λf ) Pλf =(Id+Î»âÎ»p f)â1. (6.4)Furthermore gph âÎ»p f = T (gphPλf), whereT =[0 IdId /λ â Id /λ]is an invertible linear transformation.Proof. Note that (âx â Rn) Pλ(x) is nonempty thanks to Fact 2.54. Then(âx â Rn) xâ â PÎ»f(x)â(ây â Rn) f(y) + 12λ â¥y â xâ¥2 â¥ f(xâ) + 12λ â¥xâ â xâ¥2â(ây â Rn) f(y) â¥ f(xâ) + 1Î»â¨xâ xâ, y â xââ© â 12λ â¥y â xââ¥2â 1Î»(xâ xâ) â âÎ»p f(xâ)â xâ â(Id+Î»âÎ»p f)â1(x),from which (6.4) and the linear transformation result readily follow.Theorem 6.6. Let f : Rn â R be proper and lsc. For every λ > 0,âÎ»p f = âF(f + Î»â1j)â Î»â1 Id.Proof. u â âÎ»p f(x) amounts to for every y â Rn, f(y) â¥ f(x) + â¨u, y â xâ© â12Î» â¥y â xâ¥2 â f(y) + 12λ â¥yâ¥2 â¥ f(x) + 12λ â¥xâ¥2 + â¨u + 1λx, y â xâ©, which isequivalent to u+ Î»â1x â âF(f + Î»â1j)(x).1226.3. FUNDAMENTAL PROPERTIES6.3 Fundamental propertiesIn this section, we explore fundamental properties of âÎ»p f , and connect itwith the known subdifferentials introduced in Section 2.2. Several funda-mental properties of âÎ»p f are in order.Proposition 6.7. Let f : Rn â R be proper and lsc. Then the followinghold:(i) (âx â dom f) (âÎ» > 0) âÎ»p f(x) is convex and closed.(ii) (â0 < λ1 â¤ λ2) (âx â dom f) âÎ»2p f(x) â âÎ»1p f(x) and âpf(x) =âÎ»>0 âÎ»p f(x).(iii) If (âÎ» > 0) ran âÎ»p f is bounded, then dom âÎ»p f is closed.(iv) (âx â dom f) (âÎ» > 0) âÎ»p f(x) â âpf(x) â âf(x).(v) (âx â dom f) âÎ»>0 âÎ»p f(x) = {v â Rn : (ây â Rn) f(y) â¥ f(x) +â¨v, y â xâ©}, the Fenchel subdifferential of f at x.(vi) If f is differentiable on int dom f Ì¸= â and 1/λ-hypoconvex, then (âx âint dom f) âÎ»p f(x) = {âf(x)}.Proof.(i) Immediate from the definition.(ii) Pick (x, u) â gph âÎ»2p f . Then (ây â Rn) f(y) â¥ f(x) + â¨u, y â xâ© â(1/2Î»2) â¥y â xâ¥2 â¥ f(x)+â¨u, yâxâ©â(1/2Î»1) â¥y â xâ¥2, implying that âÎ»2p f(x) ââÎ»1p f(x). Invoking [81, Proposition 8.46] justifies thatâpf(x) =âÎ»>0âÎ»p f(x).(iii) Let (xk, uk)kâN â gph âÎ»p f be such that xk â x â Rn. Taking subse-quence if necessary, assume without loss of generality that uk â u. Then(ây â Rn) f(y) â¥ lim infkââ(f(xk) + â¨uk, y â xkâ© â 12λ â¥y â xkâ¥2)â¥ f(x) + â¨u, y â xâ© â 12λ â¥y â xâ¥2 ,where the last inequality holds because f is lsc, entailing the desired claim.1236.3. FUNDAMENTAL PROPERTIES(iv) The first inclusion follows from (ii). Then apply [81, Proposition 8.46(e)].(v) For simplicity, denote A = {v â Rn : (ây â Rn) f(y) â¥ f(x)+â¨v, yâxâ©}.First we show thatâÎ»>0 âÎ»p f(x) â A. Assume without loss of generalitythatâÎ»>0 âÎ»p f(x) is nonempty, otherwise it is trivial. Let u ââÎ»>0 âÎ»p f(x).Then(âÎ» > 0)(ây â Rn) f(y) â¥ f(x) + â¨u, y â xâ© â 12λ â¥y â xâ¥2â f(x) + â¨u, y â xâ©,where the limit holds by taking Î»ââ. Next we justify A â âÎ»>0 âÎ»p f(x).Assume without loss of generality again that A is nonempty and pick u â A.Then (âÎ» > 0) (ây â Rn) f(y) â¥ f(x) + â¨u, y â xâ© â¥ f(x) + â¨u, y â xâ© â(1/2Î») â¥y â xâ¥2, completing the proof.(vi) Note that âÎ»p f(x) â âpf(x) â {âf(x)}. Thus it suffices to show thatâf(x) â âÎ»p f(x). By assumption,(ây â Rn) f(y) + 12λ â¥yâ¥2 â¥ f(x) + 12λ â¥xâ¥2 + â¨âf(x) + 1λx, y â xâ©,implying (6.3) after rearrangement.Proposition 6.8. Let u, x â Rn. The following are equivalent:(i) u â âÎ»p f(x).(ii) x â Pλf(x+ λu), i.e.,(ây â Rn) f(y) + 12Î»â¥(x+ Î»u)â yâ¥2 â¥ f(x) + 12Î»â¥(x+ Î»u)â xâ¥2.(iii) u+ x/λ â âFf(x)Proposition 6.9. On every nonempty convex subset of dom âÎ»p f , the func-tion f is 1/λ-hypoconvex. In particular, if dom âÎ»p f = dom f and dom f isconvex, then f is 1/λ-hypoconvex.Proof. Let C be a nonempty convex subset of dom âÎ»p f . At every x â C, byProposition 6.8(iii), the function f +(1/2Î»)â¥ · â¥2 is Fenchel subdifferentiableat x. Since C is convex, we conclude that f + (1/2Î»)â¥ · â¥2 is convex on C.The remaining result is immediate.Theorem 6.10 (subdifferential relationships). Let f : Rn â R be proper,lsc and prox-bounded with threshold λf â (0,â]. Then the following hold:1246.3. FUNDAMENTAL PROPERTIES(i) Let 0 < λ < λf . Then f is 1/λ-hypoconvex â âÎ»p f = âf â âÎ»p f =âpf â âÎ»p f = âÌf .(ii) Suppose that λf = â. Then f is convex â (âÎ» > 0) âÎ»p f = âfâ (âÎ» > 0) âÎ»p f = âpf â (âÎ» > 0) âÎ»p f = âÌf .Proof.(i) Assume that âÎ»p f = âf . Then Theorem 6.5 yields that Pλf = (Id+Î»âf)â1,which is monotone by using Theorem 2.60. In turn Id+Î»âf is mono-tone by [81, Exercise 12.4] and consequently âf is 1/λ-hypomonotone. Thehypoconvexity of f follows immediately; see, e.g., [81, Exercise 12.61]. Tojustify the converse direction, observe that(Id+Î»âÎ»p f)â1= Pλf = (Id+Î»âf)â1by Theorems 6.3 and 6.5, implying that âÎ»p f = âf .We now show that âÎ»p f = âpf implies f being 1/λ-hypoconvex. Note that(Id+Î»âpf)â1 is monotone due to Theorem 6.5 and [81, Exercise 12.4], en-tailing âpf to be 1/λ-hypomonotone. Take arbitrary (x, u), (y, v) â gph âf .Then there exist (xk, uk), (yk, vk) â gph âpf such that (xk, uk)â (x, u) and(yk, vk)â (y, v). In turn,(âk â N) â¨xkâyk, ukâvkâ© â¥ â 1λ â¥xk â ykâ¥2 â â¨xây, uâvâ© â¥ â 1λ â¥xâ yâ¥2 ,which justifies the desired claim by invoking [81, Exercise 12.61] again. Theconverse direction is immediate by using Proposition 6.7(iv). The last equiv-alence regarding âÌf can be proved by a similar argument.(ii) Suppose that f is convex. Then (âÎ» > 0) âf(x) â âÎ»p f(x) â âpf(x) ââf(x), where the first inclusion is implied by (6.3) and the second oneholds due to Proposition 6.7, from which the desired result readily follows.Now suppose that (âÎ» > 0) âÎ»p f = âf . Then (i) implies that for every(x, u), (y, v) â gph âf(âÎ» > 0) â¨xâ y, uâ vâ© â¥ â 1λ â¥xâ yâ¥2 â â¨xâ y, uâ vâ© â¥ 0,where the right hand side holds by taking Î»ââ, which means that âf ismonotone. It follows immediately that f is convex owing to Fact 2.59. Theremainder can be justified by a similar argument as in statement(i).1256.3. FUNDAMENTAL PROPERTIESProposition 6.11 (addition of functions). Let f, g : Rn â R be proper,λ1, λ2 > 0 and let λ3 = (λ1λ2)/(λ1 + λ2). Let x â dom(f + g). Then thefollowing hold:(i) âÎ»1p f(x) + âÎ»2p g(x) â âÎ»3p (f + g)(x).(ii) Suppose that âg is 1/λ2-Lipschitz. ThenâÎ»3p (f + g)(x) â âÎ»Ë3p f(x) +âg(x) = âÎ»Ë3p f(x) + âÎ»2p g(x), (6.5)where Î»Ë3 = (λ2λ3)/(λ2 + λ3). Assume in addition that f is 1/λ1-hypoconvex. ThenâÎ»1p f(x) + âÎ»2p g(x) = âÎ»1p f(x) +âg(x) = âÎ»3p (f + g)(x). (6.6)Proof.(i) Assume without loss of generality that âÎ»1p f(x), âÎ»2p g(x) are nonempty,otherwise the desired inclusion is trivial. Now, pick u â âÎ»1p f(x), v ââÎ»2p g(x). Then adding corresponding level proximal subgradient inequal-ity (6.3) yields that(ây â Rn) (f + g)(y) â¥ (f + g)(x) + â¨u+ v, y â xâ© â 12λ3 â¥y â xâ¥2 ,meaning that u+ v â âÎ»3p (x).(ii) Note that the Lipschitz assumption of âg amounts to (1/2λ2) â¥Â·â¥2 Â± gbeing convex; see, e.g., [14, Lemma 5.7, p. 109]. Then Proposition 6.7(vi)implies that âÎ»2p g(x) = âg(x) and âÎ»2p (âg)(x) = ââg(x). Pick an arbitraryu â âÎ»3p (f + g)(x). Then for every y â Rnf(y) â¥ f(x) + â¨uââg(x), y â xâ© â 12λ3 â¥y â xâ¥2â (g(y)â g(x)â â¨âg(x), y â xâ©)â¥ f(x) + â¨uââg(x), y â xâ© â(12Î»3+ 12Î»2)â¥y â xâ¥2= f(x) + â¨uââg(x), y â xâ© â 12Î»Ë3â¥y â xâ¥2 ,where the second inequality holds because âÎ»2p (âg)(x) = {ââg(x)}, whichcombined with âÎ»2p g(x) = {âg(x)} completes the proof of (6.5). Finally,we turn to (6.6). Note that λ1 > Î»Ë3. Invoking Theorem 6.3 and Proposi-tion 6.7(ii)&(iv), one concludes thatâpf(x) = âÎ»1p f(x) â âÎ»Ë3p f(x) â âpf(x)â âÎ»1p f(x) = âpf(x) = âÎ»Ë3p f(x),which entails (6.6) by taking statement(i) and (6.5) into account.1266.4. EXAMPLES6.4 ExamplesIn this section, we illustrate Theorem 6.5 in a few examples.Example 6.12. Let λ > 0 and f(x) = Î´{â1,1}(x). Then the following hold:(i) âpf(x) = âÌf(x) = âf(x) ={R, if x â {â1, 1};â , otherwise.(ii) âÎ»p f(x) =ï£±ï£´ï£²ï£´ï£³[â1/Î»,â), if x = 1;(ââ, 1/λ], if x = â1;â , otherwise.(iii) Pλf =(Id+Î»âÎ»p f)â1 â (Id+Î»âpf)â1 = (Id+Î»âf)â1, where (âx âR) (Id+Î»âf)â1 (x) = {â1, 1} andPλf(x) ={{â1, 1}, if x = 0;sgn(x), if x Ì¸= 0.Proof.(i) It suffices to show âpf(±1) = R, in which case appealing to the in-clusion âpf â âÌf â âf completes the proof; see Proposition 2.13. ByDefinition 2.10, the following holds trivially for every v â R and Ï > 0(â|xâ 1| < 1) f(x) â¥ f(1) + v(xâ 1)â 12Ï(xâ 1)2,implying that R â âpf(1) and therefore âpf(1) = R as claimed. The sameargument justifies that âpf(â1) = R.(ii) Let v â R. Then v â âÎ»p f(1) â f(â1) â¥ f(1) + 2v â 2/λ â v â¥ â1/Î».Similarly, v â âÎ»p f(â1)â f(1) â¥ f(â1) + 2v â 2/Î»â v â¤ 1/λ.(iii) Observe that Pλf is the projector onto {â1, 1} to see the proximaloperator formula. Resolvent computation is immediate, hence omitted.Example 6.13. Let λ > 0 and f(x) = 1 for x Ì¸= 0 and f(0) = 0. Then thefollowing hold:(i) âpf(x) = âÌf(x) = âf(x) ={R, if x = 0;0, if x Ì¸= 0.1276.4. EXAMPLESFigure 6.2: Resolvents in Example 6.12.(ii) âÎ»p f(x) =ï£±ï£´ï£²ï£´ï£³â , if 0 < |x| < â2Î»;[ââ2/Î»,â2/Î»], if x = 0;0, if |x| â¥ â2Î».(iii) Pλf =(Id+Î»âÎ»p f)â1 â (Id+Î»âpf)â1 = (Id+Î»âf)â1, where (âx âR) (Id+Î»âf)â1 (x) = (Id+Î»âpf)â1 (x) = {0, x}, whereasPλf(x) =ï£±ï£´ï£²ï£´ï£³0, if |x| < â2Î»;{0, x}, if |x| = â2Î»;x, if |x| > â2Î».Proof.(i) Appealing Proposition 2.13 to see âpf(x) = âÌf(x) = âf(x) = {0} atnonzero x. We shall next argue âpf(0) = R, which enforces the same forâÌf(0) and âf(0) due to Proposition 2.13. To check v â âpf(0) for v â R,it suffices to see if there exist δ, Ï > 0 such that (â0 < |x| < δ) f(x) â¥f(0) + vxâ 12Ïx2, which boils down to(â0 < |x| < δ) 12Ïx2 + vx+ 1 â¥ 0.Setting 0 < Ï < 2v2guarantees that the above inequality to hold even glob-ally, completing the proof.1286.4. EXAMPLESFigure 6.3: Resolvents in Example 6.13.(ii) Note that v â âÎ»p f(0) â (ây â R) (1/2λ)y2 â vy + 1 â¥ 0. Then thediscriminant â = v2 â 4(1/2λ) â¤ 0 â |v| â¤ â2/Î», entailing âÎ»p f(0) =[ââ2/Î»,â2/Î»].For x Ì¸= 0 and v â R, define (ây â R) g(y) = gv,x(y) = f(y) â v(y âx) + (1/2λ)(x â y)2 â f(x). Then v â âÎ»p f(x) â (ây â R) g(y) â¥ 0. Letv = 0. Then (ây â R) g(y) = f(y) + (1/2λ)(x â y)2 â 1. In turn (ây â R)g(y) â¥ 0 â (1/2λ)x2 â 1 = g(0) â¥ 0 â |x| â¥ â2Î», which together withProposition 6.7(iv) justified the desired formula.(iii) Note that the proximal formula is one-dimensional hard thresholdingoperator; see, e.g., [14, Example 6.10]. The rest follows from simple resolventcomputation.Example 6.14. Let λ > 0 and f(x) = 0 if x â¤ 0 and f(x) = 1 otherwise.Then the following hold:(i) âpf(x) = âÌf(x) = âf(x) ={0, if x Ì¸= 0;[0,+â), if x = 0.(ii) âÎ»p f(x) =ï£±ï£´ï£²ï£´ï£³[0,â2/Î»], if x = 0;â , if 0 < x < â2Î»;0, otherwise.1296.4. EXAMPLESFigure 6.4: Resolvents in Example 6.14.(iii) Pλf =(Id+Î»âÎ»p f)â1 â (Id+Î»âf)â1, where(Id+Î»âf)â1(x) = (Id+Î»âpf)â1 (x) ={{x, 0}, if x â¥ 0;x, if x < 0,whereasPλf(x) =ï£±ï£´ï£²ï£´ï£³x, if x â¤ 0 or x > â2Î»;0, if 0 < x <â2Î»;{0,â2Î»}, if x = â2Î».Proof.(i) Apparently f is (continuously) differentiable at x Ì¸= 0, in which caseâpf(x) = âÌf(x) = âf(x) = âf(x) = 0.We continue with justifying âpf(0) = [0,â). By Definition 2.10, verifyingv â âpf(0) amounts to finding Ï, δ > 0 such that(â|x| < δ) f(x) â¥ f(0) + vxâ 12Ïx2 = x(v â 12Ïx). (6.7)Clearly there is no such δ > 0 when v < 0. When v â¥ 0, one may al-ways choose Ï > 0 sufficiently small, under which (6.7) holds even globally,completing the proof.Now we turn to âÌf(0) = [0,â). By Proposition 2.13, [0,â) = âpf(0) ââÌf(0), and therefore it suffices to verify v /â âÌf(0) for v < 0. Indeed, for1306.4. EXAMPLESv < 0,lim infxâ0f(x)âf(0)âvx|x| â¤ limxâ0ââvx|x| = v < 0,violating Definition 2.10.Finally, we show that âf(0) = [0,â). Take a sequence xk â 0 and vk ââÌf(xk). Then for every k â N, either âÌf(xk) = 0 (when xk Ì¸= 0) or âÌf(xk) =[0,â) (when xk = 0), which means the set of cluster points of vk is [0,â),demonstrating that âf(0) = [0,â).(ii) Notice that v â âÎ»p f(x)â (ây â R)g(y) â¥ 0, whereg(y) ={12λ(y â x)2 â v(y â x)â f(x), if y â¤ 0;12λ(y â x)2 â v(y â x)â f(x) + 1, if y > 0.Consider x Ì¸= 0, in which case one only needs to check whether 0 â âÎ»p f(x)because Proposition 6.7(iv). If x > 0, then 0 â âÎ»p f(x) amounts to miny>0 g(y) =1 â f(x) â¥ 0,minyâ¤0 g(y) = (1/2λ)x2 â 1 â¥ 0 â x â¥â2Î», implying thatâÎ»p f(x) = â when 0 < x <â2Î»; and âÎ»p f(x) = {0} when x >â2Î». If x < 0,then minyâ¤0 g(y) = g(x) = âf(x) = 0 and infy>0 g(y) = limyâ0+ g(y) =(1/2λ)x2 + 1 > 0, so 0 â âÎ»p f(x).Now let x = 0 and v â¥ 0. Then λv â¥ 0 and infy<0 g(y) = g(0) = 0,minyâ¥0 g(y) = g(λv) = 1 â (λ/2)v2 â¥ 0 â |v| â¤â2/Î». Hence âÎ»p f(0) =[0,â2/Î»].(iii) Resolvents are immediate due to (ii), so we only justifies formula for Pλf .Note that Pλf(x) = argminyâR g(y), where g(y) = f(y) + (1/2λ)(y â x)2satisfiesg(y) ={12λ(y â x)2, if y â¤ 0;12λ(y â x)2 + 1, if y > 0.When x â¤ 0, it is easy to see that argminyâR g(y) = {x}. Now consider x >0, in which case g(0) = minyâ¤0 g(y) = (1/2λ)x2 and g(x) = miny>0 g(y) = 1.Then g(0) < g(x)â x < â2Î» and g(0) = g(x)â x = â2Î», from which thedesired formula readily follow.Example 6.15. Let λ > 0 and f(x) = ââ¥xâ¥ for x â Rn. Then the followinghold:1316.4. EXAMPLES(i) âf(x) ={S, if x = 0;âx/ â¥xâ¥ , if x Ì¸= 0, where S â Rn represents the unitsphere, andâpf(x) = âÌf(x) ={â , if x = 0;âx/ â¥xâ¥ , if x Ì¸= 0.(ii) âÎ»p f(x) ={â , if â¥xâ¥ < Î»;âx/ â¥xâ¥ , if â¥xâ¥ â¥ λ.(iii) Pλf(x) =(Id+Î»âÎ»p f)â1(x) â (Id+Î»âpf)â1 (x) â (Id+Î»âf)â1 (x),where(Id+Î»âpf)â1 (x) ={{x± λProjS(x)}, if â¥xâ¥ < λ,{x+ λProjS(x)}, if â¥xâ¥ â¥ Î»,(Id+Î»âf)â1 (x) ={{x± λProjS(x)}, if â¥xâ¥ â¤ λ,{x+ λProjS(x)}, if â¥xâ¥ > Î».and(âx â Rn) Pλf(x) = x+ λProjS(x).Proof.(i) f is (continuously) differentiable at x Ì¸= 0, therefore âpf(x) = âÌf(x) =âf(x) = âf(x) = âx/ â¥xâ¥; see Proposition 2.13. It follows immediatelythat âf(0) = S.Next we justify âÌf(0) = â , enforcing âpf(0) = â thanks to the inclusionâpf â âÌf ; see Proposition 2.13. Suppose there was v â âÌf(0). When v Ì¸= 0,define xk =vkâ¥vâ¥ . Then0 â¤ lim infxâ0f(x)âf(0)ââ¨v,xâ0â©â¥xâ0â¥ â¤ limkââf(xk)âf(0)ââ¨v,xkâ0â©â¥xkâ0â¥= limkâââ1/kââ¥vâ¥/k1/k = â1â â¥vâ¥ < 0,where the first inequality follows from Definition 2.10, which is absurd.When v = 0,lim infxâ0f(x)âf(0)ââ¨v,xâ0â©â¥xâ0â¥ = â1,which still contradicts to Definition 2.10.1326.4. EXAMPLES(ii) Owing to Proposition 6.7(iv), it suffices to check whether (âx Ì¸= 0)v = âx/ â¥xâ¥ â âÎ»p f(x) holds. In turn,v â âÎ»p f(x)â (ây â Rn) â â¥yâ¥ â¥ ââ¥xâ¥ = â¨v, y â xâ© â (1/2λ) â¥y â xâ¥2â minyâRng(y) â¥ 0,where g(y) = (1/2λ) â¥y â xâ¥2 + â¨v, xâ yâ© â â¥yâ¥+ â¥xâ¥.When 0 < â¥xâ¥ < λ,g(x+ 2λv) = â¥xâ¥ â â¥x+ 2Î»vâ¥ = â¥xâ¥ â | â¥xâ¥ â 2λ| = 2 â¥xâ¥ â 2λ < 0,implying that âÎ»p f(x) = â .Now let â¥xâ¥ â¥ λ. Clearly g is coercive, which means that there existsa minimizer y â Rn of g. We claim that g(y) â¥ 0, entailing the desiredinclusion v â âÎ»p f(x). Indeed, if y = 0, then g(y) = (1/2λ) â¥xâ¥2 â¥ 0. Ify Ì¸= 0, then optimality condition Fact 2.14 yields that0 â âg(y) = (1/λ)(y â x)â v + â(ââ¥Â·â¥)(y),implying y = x + λ(v â d), where d = ây/ â¥yâ¥ = â(ââ¥Â·â¥)(y). Henceg(y) = â¥xâ¥ â â¥x+ λ(v â d)â¥. Recall that d = ây/ â¥yâ¥ and v = âx/ â¥xâ¥.Thenâ¨v, dâ© â¤ 1â Î»â â¥xâ¥ â¤ (Î»â â¥xâ¥)â¨v, dâ©â 2λ2 â 2Î»2â¨v, dâ©+ 2Î»â¨x, v â dâ© â¤ 0â â¥x+ λ(v â d)â¥2 = â¥xâ¥2 + â¥Î»(v â d)â¥2 + 2Î»â¨x, v â dâ© â¤ â¥xâ¥2â g(y) â¥ 0,as desired.(iii) Beginning from showing Pλf(x) = x+ λProjS(x), we note that(ây â Rn) â â¥yâ¥+ 12λ â¥y â xâ¥2 â¥ 12λ â¥yâ¥2 â(1Î» â¥xâ¥+ 1) â¥yâ¥+ 12λ â¥xâ¥2â¥ ââ¥xâ¥ â λ2 .Then (ây â Rn) (âd â ProjS(x))ââ¥x+ Î»dâ¥+ 12λ â¥x+ Î»dâ xâ¥2 = ââ¥xâ¥ â λ2 â¤ ââ¥yâ¥+ 12λ â¥y â xâ¥2 , (6.8)implying x+λProjS(x) â Pλf(x). To justify the converse, take v â Pλf(x).Then optimality condition Fact 2.14 implies that1Î»(xâ v) â â(ââ¥Â·â¥)(v)â â¥v â xâ¥ = λ. (6.9)1336.5. CONCLUSIONIf x = 0, then â¥vâ¥ = λ as desired. Now suppose x Ì¸= 0. We have learnedfrom (6.8) thatââ¥vâ¥+ 12λ â¥v â xâ¥2 = ââ¥xâ¥ â λ2 â â¥vâ¥ â â¥xâ¥ = λ. (6.10)Combing identities (6.9)â(6.10) and equality characterization of the triangleinequality suggests that there exists c â R such that v = cx and|c| â 1 = |câ 1| = Î»â¥xâ¥ â c = 1 + Î»â¥xâ¥ ,implying v = x+ λProjS(x). Next we justify(Id+Î»âÎ»p f)â1= Id+λProjS.Invoking the formula for âÎ»p f yields v â(Id+Î»âÎ»p f)â1(x)â (1/Î»)(xâv) ââÎ»p f(v)â â¥vâ¥ â¥ λ and x = (1â λ/ â¥vâ¥)v. We claim thatâ¥vâ¥ â¥ λ and x =(1â Î»â¥vâ¥)v â v â x+ λProjS(x), (6.11)which entails the desired formula. To establish (6.11), we argue by cases.Suppose first the left hand side holds. If â¥vâ¥ = λ, then x = 0 and clearlyv â λProjS(x). If â¥vâ¥ > λ, then 1âÎ»/ â¥vâ¥ > 0, which implies that x/ â¥xâ¥ =v/ â¥vâ¥ and x = v â λv/ â¥vâ¥ = v â λx/ â¥xâ¥, justifying the right hand sideof (6.11). Now let v â x + λProjS(x). If x = 0 then clearly â¥vâ¥ = λ andx = (1 â λ/ â¥vâ¥)v = 0. If x Ì¸= 0, then v = x + λx/ â¥xâ¥ and consequentlyâ¥vâ¥ = â¥xâ¥ + λ > λ, thus completes the proof of (6.11). Formulas for(Id+Î»âf)â1 and (Id+Î»âpf)â1 can be proved similarly by repeating theabove argument with extra care for the case â¥xâ¥ = λ, thus omitted forsimplicity.6.5 ConclusionWe showed that every proximal operator is the resolvent of a level proximalsubdifferential, investigated fundamental properties of this new subdifferen-tial and its the connection to various classic subdifferentials in the literature.Examples are given to illustrate our results.134Chapter 7Finer properties of levelproximal subdifferential7.1 IntroductionThe level proximal subdifferential, introduced by Rockafellar [79] and in-vestigated by Wang and Wang [95] (see Chapter 6), is a powerful tool tostudy proximal operators. However, it remains poorly understood in theliterature. The goal of this chapter is to investigate systematically the levelproximal subdifferential. The main results in this chapter areR1 We characterize the level proximal subdifferential âÎ»p f in Theorem 7.3,which reveals a surprising connection between âÎ»p f and a âpointwiseâhypoconvexity.R2 Building on R1, we provide in Theorem 7.17 a novel subdifferentialcharacterization of local Lipschitz smoothness through the lens of thelevel proximal subdifferential.R3 We investigate integration of the level proximal subdifferential, whoseâantiderivativeâ turns out to be the proximal hull of a function, gen-eralizing results by Benoist and Daniilidisâ results for Fenchel subdif-ferentials [16].The rest of the chapter is organized as follows. In Section 7.2, we char-acterize the existence of level proximal subdifferential, followed by resultsconcerning the single-valuedness results in Section 7.3. We study new opti-mality conditions in view of the level proximal subdifferential in Section 7.4;while Section 7.6 is devoted to integration of level proximal subdifferential.7.2 Level proximal subdifferentiabilityThe level proximal subdifferential âÎ»p f(x) is a possibly empty-valued selec-tion of the proximal subdifferential âpf(x); see for instance Section 6.4. As1357.2. LEVEL PROXIMAL SUBDIFFERENTIABILITYsuch, a natural question is how and where should âÎ»p f(x) be selected. Inthis section, we address this question completely by characterizing âÎ»p f(x)through convex hulls. We start with an auxiliary result.Lemma 7.1. Let f : Rn â R be proper, lsc and prox-bounded with thresholdλf > 0, and let 0 < λ < λf . Then the following hold:(i) f+Î»â1j is 1-coercive. Consequently conv (f+Î»â1j) = conv(f+Î»â1j).(ii) hλf + Î»â1j = conv(f + Î»â1j) is a proper, lsc and convex function;(iii) hλf(x) = f(x)â conv(f + Î»â1j)(x) = (f + Î»â1j)(x).Proof.(i) Note that the function (âÎ» < λ0 < λf ) f + Î»â10 j is bounded belowbecause f is prox-bounded with threshold λf ; see Proposition 2.53. Thenf + Î»â1j = f + Î»â10 j + (Î»â1 â Î»â10 )j is 1-coercive, implying that conv (f +Î»â1j) = conv(f + Î»â1j); see Fact 2.47.(ii) Apply (i) and Proposition 2.56 to obtain hλf = conv(f + Î»â1j) âÎ»â1j = conv(f + Î»â1j)â Î»â1j.(iii) Again by Proposition 2.56, we have hλf(x) = (f+Î»â1j)ââ(x)âÎ»â1j(x).Then f(x) = hλf(x) = (f + Î»â1j)ââ(x) â Î»â1j(x) â conv(f + Î»â1j)(x) =conv (f + Î»â1j)(x) = (f + Î»â1j)ââ(x) = (f + Î»â1j)(x).To start with, let us demonstrate that the domain of level proximal subdif-ferential is always nonempty for a prox-bounded function.Proposition 7.2. Let f : Rn â R be proper, lsc and prox-bounded withthreshold λf > 0. Then (â0 < λ < λf ) dom âÎ»p f Ì¸= â .Proof. Note that (â0 < λ < λf )ran (Pλf) = ran(Id+Î»âÎ»p f)â1= dom(Id+Î»âÎ»p f)= dom âÎ»p f,where the first equality owes to Theorem 6.5. Appealing to [81, Theorem1.17] yields that domPλf = Rn. Hence dom âÎ»p f = ran (Pλf) Ì¸= â .Proposition 7.2 indicates that âÎ»p f is always meaningful for a prox-bounded function f . However, the following question remains unclear:1367.2. LEVEL PROXIMAL SUBDIFFERENTIABILITYWhere exactly should âÎ»p f(x) be nonempty?We now present the first main result of this section, which fully charac-terize âÎ»p f , unveiling a remarkable connection between the λ-level proximalsubdifferentiability of f and a âpointwiseâ hypoconvexity.Theorem 7.3 (level proximal subdifferentiability at a point). Let f : Rn âR be proper, lsc and prox-bounded with threshold λf > 0. Let x â dom f andlet 0 < λ < λf . Then the following are equivalent:(i) u â âÎ»p f(x).(ii) hλf(x) = f(x) and u â âhÎ»f(x).(iii) conv(f+Î»â1j)(x) = (f+Î»â1j)(x) and u+Î»â1x â â conv(f+Î»â1j)(x).Proof.â(i) â (ii)â Let u â âÎ»p f(x). Then x â Pλf(x+ λu), which implies throughProposition 2.56 that f(x) = hλf(x). Next we show that u â âhÎ»f(x).Define v = u+ x/λ. Then the assumption u â âÎ»p f(x) implies(ây â Rn) f(y) â¥ f(x) + â¨u, y â xâ© â 12λ â¥y â xâ¥2= f(x) + â¨u, y â xâ© â 12λ â¥yâ¥2 + 1Î»â¨x, y â xâ©+ 12λ â¥xâ¥2=(hÎ»f + Î»â1j)(x) + â¨v, y â xâ© â Î»â1j(y)where the last equality owes to f(x) = hλf(x). Equivalently(ây â Rn) f(y) + Î»â1j(y) â¥ (hλf + Î»â1j) (x) + â¨v, y â xâ©,entailing the functiony 7â (hλf + Î»â1j) (x) + â¨v, y â xâ©to be a convex minorant of f + Î»â1j. In turn(ây â Rn) (hλf + Î»â1j) (y) = conv (f + Î»â1j) (y)â¥ (hλf + Î»â1j) (x) + â¨v, y â xâ©,thus v â â(hÎ»f + Î»â1j)(x), and consequently u = v â Î»â1x â â(hÎ»f +Î»â1j)(x)â Î»â1x = âhÎ»f(x).â(ii) â (iii)â Lemma 7.1(iii) gives conv(f + Î»â1j)(x) = (f + Î»â1j)(x).Moreover, dom âhÎ»f = dom â(hÎ»f + Î»â1j) and the identity hλf + Î»â1j =conv(f + Î»â1j) from Lemma 7.1(ii) entail that x â dom â[conv(f + Î»â1j)].1377.2. LEVEL PROXIMAL SUBDIFFERENTIABILITYâ(iii) â (i)â Let u â â[conv(f + Î»â1j)](x). Applying the subgradient in-equality yields that(ây â Rn) (f + Î»â1j)(y) â¥ conv(f + Î»â1j)(y)â¥ conv(f + Î»â1j)(x) + â¨u, y â xâ©= (f + Î»â1j)(x) + â¨u, y â xâ©,implying that u â âF(f + Î»â1j)(x). Put v = uâx/Î». Then v â âÎ»p f(x) byTheorem 6.6.The condition u â âhÎ»f(x) in Theorem 7.3 is essential, as the example belowdemonstrates.Example 7.4. Let λ > 0. Consider the functionf : Râ R : x 7âï£±ï£´ï£²ï£´ï£³x ln(x), if x > 0;0, if x = 0;+â, otherwise.Then f is convex, implying that hλf = f and âÎ»p f = âf . However, one hasâÎ»p f(0) = âf(0) = â .Revisiting the zero ânormâ function in Example 6.13, we now illustratehow Theorem 7.3 makes computing âÎ»p f much easier by converting the prob-lem of solving (6.3) into a simple two-step procedureâfind the convex hullof a function then compute its subgradient.Example 7.5. Let f(x) = 1 for x Ì¸= 0 and f(0) = 0. Then for every λ > 0,conv(f + Î»â1j)(x) ={(2Î»)â1x2 + 1, if |x| â¥ â2Î»;â2Î»â1|x|, otherwise.Obviously conv(f + Î»â1j)(x) < (f + Î»â1j)(x) for 0 < |x| < â2Î», on whichâÎ»p f(x) = â owing to Theorem 7.3. After calculating the subgradient of theconvex function conv(f + Î»â1j)(x) for x = 0 and |x| â¥ â2Î», we invokeTheorem 7.3 again to see thatâÎ»p f(x) =ï£±ï£´ï£²ï£´ï£³â , if 0 < |x| < â2Î»;[ââ2/Î»,â2/Î»], if x = 0;{0}, otherwise.1387.2. LEVEL PROXIMAL SUBDIFFERENTIABILITYWe have seen that âÎ»p f demonstrates its full advantages over classic sub-differentials when f is not hypoconvex, yet Theorem 7.3 reveals a close rela-tion between dom âÎ»p f and a âpointwiseâ hypoconvexity. This subtlety givesrise to the question: How âbigâ can the domain of âÎ»p f be while maintainingits novelty? We approach this by identifying an extreme case with âbigâdom âÎ»p f , in which hypoconvexity is recovered and the distinction betweenâÎ»p f and classic subdifferentials vanishes.Proposition 7.6. Let f : Rn â R be proper, lsc and prox-bounded withthreshold λf > 0, and let 0 < λ < λf . Suppose that dom f is convex. Thenthe following are equivalent:(i) ri dom f â dom âÎ»p f .(ii) f + Î»â1j is convex.(iii) âf = âÎ»p f , where the limiting subdifferential âf can be replaced by âpfor âÌf .Proof.â(i) â (ii)â The assumption along with Theorem 7.3 implies that conv(f +Î»â1j) = f + Î»â1j on ri dom f . To justify the convexity of f + Î»â1j, takex1 â dom f , x0 â ri dom f , and define xµ = (1 â µ)x1 + µx1 for 0 < µ < 1.Then xµ â ri dom f owing to Lemma 2.24 and moreover(f + Î»â1j)(x1) â¤ lim infµâ1â(f + Î»â1j)(xµ)= limµâ1âconv(f + Î»â1j)(xµ) = conv(f + Î»â1j)(x1),where the first inequality owes to the lower semicontinuity of f and the lastholds due to the continuity of conv(f+Î»â1j) on line segments; see Fact 2.27.As such, we conclude that f + Î»â1j coincides with its convex hull on theconvex set dom f = dom(f + Î»â1j), justifying the desired convexity.â(ii)â (i)â Apply Theorem 6.6 to see âÎ»p f(x) = âF(f +Î»â1j)(x)âÎ»â1x forx â ri dom f = ri dom(f + Î»â1j), which is nonempty owing to convexity off + Î»â1j.â(ii) â (iii)â This is Theorem 6.10.The equivalence in Proposition 7.6 collapses when dom f is not convex.1397.3. SINGLE-VALUED LEVEL PROXIMAL SUBDIFFERENTIALExample 7.7. Consider the proper, lsc and prox-bounded function f =Î´{â1,1}. Then dom f = {â1, 1} is not convex and recall Example 6.12 to seethat(âÎ» > 0) âÎ»p f(x) =ï£±ï£´ï£²ï£´ï£³[â1/Î»,â), if x = 1;(ââ, 1/λ], if x = â1;â , otherwise,Clearly â = ri dom f â dom âÎ»p f , but f + λ1j is not convex.7.3 Single-valued level proximal subdifferentialFor a convex function f : Rn â R, it is well-known that f is differentiableat x if and only if âf(x) is single-valued.Fact 7.8. Let f : Rn â R be proper, lsc, and convex. Then f is differen-tiable at x with âf(x) = u if and only if âf(x) is singleton with âf(x) = {u}.Proof. [11, Proposition 17.36].Next we seek a nonconvex counterpart of Fact 7.8, through the lens ofthe level proximal subdifferential. To this end, the following will be useful.Definition 7.9. Let f : Rn â R be proper. We say that {xi}iâI â dom fis called by x â dom(conv f) if(i) |I| â¤ n+ 1;(ii) there exists {Î±i}iâI such thatâiâI αi = 1 and (âi â I) αi > 0;(iii) x =âiâI αixi and (conv f)(x) =âiâI αif(xi).Lemma 7.10. Let f : Rn â R be proper, lsc, and 1-coercive. Let x âconv dom f . Then(i) There always exists {xi}iâI â dom f called by x.(ii) For any {xi}iâI â dom f called by x, â conv f(x) =âiâI âFf(xi).Proof. See [17, Lemma 3.3].Theorem 7.11 (level proximal subdifferential single-valued at a point).Let f : Rn â R be proper, lsc and prox-bounded with threshold λf > 0. Letx â dom âÎ»p f and let 0 < λ < λf . Then the following are equivalent:1407.3. SINGLE-VALUED LEVEL PROXIMAL SUBDIFFERENTIAL(i) âÎ»p f(x) is a singleton.(ii) conv(f + Î»â1j)is differentiable at x, andconv(f + Î»â1j)(x) =(f + Î»â1j)(x).(iii) hλf is differentiable at x, and hλf(x) = f(x), i.e., f is proximal at x.(iv) There exist a unique v â Rn, a finite collection of points {xi}iâI âdom f called by x (for function f + Î»â1j) such that(âi â I) xi â Pλf(v), and x â Pλf(v). (7.1)In particular, v = Î»â conv(f + Î»â1j)(x).Proof.â(i) â (ii)â The Fenchel subdifferential âF(f + Î»â1j)(x) is a singleton byassumption and Theorem 6.6. Theorem 7.3 yields that conv(f + Î»â1j)(x) =(f + Î»â1j)(x). In view of Lemma 2.32, we conclude that â conv(f + Î»â1j)(x)is a singleton, so conv(f + Î»â1j)is differentiable at x.â(ii) â (i)â The convex function conv (f + Î»â1j) being differentiable at ximplies that â conv(f + Î»â1j)(x) is single-valued, so is âF (f + Î»â1j)(x)by Lemma 2.32 and the assumption conv(f + Î»â1j)(x) =(f + Î»â1j)(x).Therefore applying Theorem 6.6 again completes the proof.â(ii)â (iii)â Invoke the identity hλf = conv(f+Î»â1j)âÎ»â1j in Lemma 7.1(ii).â(ii)â (iv)â By Theorem 6.6, applying Lemma 7.10 to conv(f+Î»â1j) yieldsa collection of finite points {xi}iâI â dom f called by x such thatu = â conv(f + Î»â1j)(x) =âiâIâF(f + Î»â1j)(xi) =âiâI[âÎ»p f(xi) + Î»â1xi].Invoking the fact that x calls {xi}iâI and (ii) yieldsx =âiâIÎ±ixi, and (f+Î»â1j)(x) = conv(f+Î»â1j)(x) =âiâIÎ±i(f+Î»â1j)(xi),where (Î±i)iâI â (0, 1) andâiâI αi = 1. Let v = λu. Then(âi â I) u â âÎ»p f(xi) + Î»â1xi â xi â Pλf(v),1417.3. SINGLE-VALUED LEVEL PROXIMAL SUBDIFFERENTIALandf(x) + 12Î»â¥xâ vâ¥2 = f(x) + 12Î»â¥xâ¥2 + 12Î»â¥vâ¥2 â 1Î»â¨v, xâ©=âiâIÎ±i(f(xi) +12Î»â¥xiâ¥2 + 12Î»â¥vâ¥2 â 1Î»â¨v, xâ©)=âiâIÎ±i(f(xi) +12Î»â¥xi â vâ¥2)=âiâIÎ±ieÎ»f(v) = eλf(v),implying that x â Pλf(v). Hence v satisfies (7.1).Next, we argue that v satisfying (7.1) is unique. Suppose there exists adistinct vË satisfying (7.1). Since(âi â I) xi â PÎ»f(vË)â vË â xi + Î»âÎ»p f(xi) = Î»âF(f + Î»â1j)(xi),we have vË/Î» â âiâI âF(f + Î»â1j)(xi) = â conv(f + Î»â1j)(x). Then havingtwo different v/λ, vË/Î» â â conv(f + Î»â1j)(x) contradicts the assumptionthat conv(f + Î»â1j) is differentiable at x.â(iv)â (ii)â First we show that (f+Î»â1j)(x) = conv(f+Î»â1j)(x). Indeed,by the assumptions that x, xi â Pλf(v) and that x calls {xi}iâI , we havef(x) + 12Î»â¥xâ vâ¥2 =âiâIÎ±ieÎ»f(v) =âiâIÎ±i(f(xi) +12Î»â¥xi â vâ¥2)=âiâIÎ±i(f(xi) +12Î»â¥xiâ¥2)+ 12Î»â¥vâ¥2 â 1Î»âiâIÎ±iâ¨xi, vâ©= conv(f + Î»â1j)(x) + 12Î»â¥vâ¥2 â 1Î»â¨x, vâ©,implying that (f + Î»â1j)(x) = conv(f + Î»â1j)(x). Furthermore,(âi â I) xi â PÎ»f(v)â (âi â I) v â Î»âF(f + Î»â1j)(xi)â vλ ââiâIâF(f + Î»â1j)(xi) = â conv(f + Î»â1j)(x),where the last equality holds due to Lemma 7.10. The uniqueness of vimplies that â conv(f + Î»â1j)(x) is a singleton, hence conv(f + Î»â1j) isdifferentiable at x.1427.4. OPTIMALITY CONDITIONS7.4 Optimality conditionsIn this section, we use the level proximal subdifferential to characterize var-ious optimal conditions. Notably, the level proximal subdifferential equiva-lently characterizes the global optimality of a possibly nonconvex function.Proposition 7.12 (optimality condition). Let f : Rn â R be proper lsc,and let x â dom f . Then the following hold:(i) x is a global minimizer of f if and only if (âÎ» > 0) 0 â âÎ»p f(x) if andonly if (âÎ» > 0) x â Pλf(x).(ii) Assume in addition that f is prox-bounded with threshold λf > 0. If xis a local minimizer of f , then (â0 < λ < λf ) (â0 < λ â¤ λ) 0 â âÎ»p f(x).Proof.(i) x is a global minimizer of f if and only if (ây â Rn) f(y) â¥ f(x) â0 â âF f(x) =âÎ»>0 âÎ»p f(x), from which the first equivalence readily follows.The second holds by observing that (âÎ» > 0) 0 â âÎ»p f(x) â (âÎ» > 0) x â(Id+Î»âÎ»p f)(x)â (âÎ» > 0) x â Pλf(x).(ii) By assumption, 0 â âpf(x) =âÎ»>0 âÎ»p f(x), implying that (âÎ» > 0)0 â âÎ»p f(x). Noticing that (â0 < λ < λ) âÎ»p f(x) â âÎ»p f(x) completes theproof.Corollary 7.13 (local minimizer). Let f : Rn â R be proper, lsc and prox-bounded with threshold λf > 0. Let x be a local but not global minimizerof f and define S = {λ > 0| 0 â âÎ»p f(x)}, and λ = supS. Then λ < â,(â0 < λ â¤ λ) 0 â âÎ»p f(x) and (âÎ» > λ) 0 /â âÎ»p f(x).Proof. Suppose the contrary that λ = â. Then we can take λk â S suchthat λk â â as k â â. In turn (âk â N) (â0 < λ < λk) 0 â âÎ»kp f(x),implying that (âÎ» > 0) 0 â âÎ»p f(x), which forces x to be a global minimizerby Proposition 7.12, contradicting to our assumption. Hence λ < â. Nowtake a sequence (Î»k)kâN â S such that λk â λ. Then (âk â N) (ây âRn) f(y) â¥ f(x)â 12λk â¥y â xâ¥2. Passing k to infinity yields 0 â âÎ»p f(x).In terms of Moreau envelopes, we have the following characterizations.1437.5. POINTWISE LIPSCHITZ SMOOTHNESSTheorem 7.14 (connection to Moreau envelope). Let f : Rn â R be proper,lsc and prox-bounded with threshold λf > 0. Then (â0 < λ < λf ) 0 ââÎ»p f(x)â eλf(x) = f(x)â x â Pλf(x).Proof. Suppose first that 0 â âÎ»p f(x). Then(ây â Rn) f(y) + 12Î»â¥y â xâ¥2 â¥ f(x)â eλf(x) â¥ f(x),but it always holds that eλf(x) â¤ f(x) by [81, Example 1.44], hence eλf(x) =f(x). Conversely, assume eλf(x) = f(x). Then (ây â Rn) f(y) + 12Î»â¥y âxâ¥2 â¥ f(x), implying that 0 â âÎ»p f(x). The last equivalence follows fromthe definition of proximal mapping.7.5 Pointwise Lipschitz smoothnessIn this section, we propose the pointwise Lipschitz smoothness for possiblynonconvex functions, which opens the door to study Lipschitz smoothnesson open sets.Definition 7.15. Let f : Rn â R be proper lsc and differentiable at x âdom f and let L > 0. We say that f is L-Lipschitz smooth at x if(ây â Rn) |f(y)â f(x)â â¨âf(x), y â xâ©| â¤ L2 â¥xâ yâ¥2 ,and that f is Lipschitz smooth on U â Rn if f is Lipschitz smooth at everyx â U .The bridge to investigate the pointwise Lipschitz smoothness is a two-sided condition utilizing the level proximal subdifferentials âÎ»p (Â±f).Lemma 7.16. Let U be a nonempty open subset in Rn and let L > 0.Suppose that g : Rn â R satisfies(ây â Rn)(âx â U) 0 â¤ g(y)â g(x)â â¨âg(x), y â xâ© â¤ L â¥y â xâ¥2 . (7.2)Then (2L)â1âg is firmly nonexpansive on U , i.e.,(âx, y â U) 12L â¥âg(y)ââg(x)â¥2 â¤ â¨âg(x)ââg(y), xâ yâ© . (7.3)Proof. Fix x â U , and define Ï : Rn â R : y 7â g(y) â â¨âg(x), yâ©. Then(7.2) implies that Ï attains global minimum at x, i.e., Ï(x) = minzâRn Ï(z).We claim that for every y â U(âz â Rn) Ï(z)â Ï(y)â â¨âÏ(y), z â yâ© â¤ L â¥y â zâ¥2 , (7.4)1447.5. POINTWISE LIPSCHITZ SMOOTHNESSimplying thatÏ(x) = minzâRnÏ(z) â¤ minzâRn[Ï(y) + â¨âÏ(y), z â yâ©+ L â¥z â yâ¥2]â¤ minrâ¥0[Ï(y)â â¥âÏ(y)â¥ r + Lr2] = Ï(y)â â¥âÏ(y)â¥24L , (7.5)where the last equality holds by minimizing the quadratic function r 7âââ¥âÏ(y)â¥2 r + Lr2. To see (7.4), by (7.2), for every y â U we have(âz â Rn) Ï(z)â Ï(y)â â¨âÏ(y), z â yâ©= (g(z)â â¨âg(x), zâ©)â (g(y)â â¨âg(x), yâ©)â â¨âg(y)ââg(x), z â yâ©= g(z)â g(y)â â¨âg(y), z â yâ© â¤ L â¥z â yâ¥2 ,as claimed. Therefore (7.5) implies(ây â U) g(x)â g(y)â â¨âg(x), xâ yâ© = Ï(x)â Ï(y) â¤ ââ¥âg(y)ââg(x)â¥24L ,owing to the identity âÏ(y) = âg(y)ââg(x) and the definition of Ï. Recallthat x â U is arbitrary. Then(âx, y â U) g(x) + â¨âg(x), y â xâ©+ â¥âg(y)ââg(x)â¥24L â¤ g(y). (7.6)By symmetry(âx, y â U) g(y) + â¨âg(y), xâ yâ©+ â¥âg(y)ââg(x)â¥24L â¤ g(x). (7.7)Adding (7.6) and (7.7) yields(âx, y â U) â¨âg(x)ââg(y), y â xâ©+ â¥âg(y)ââg(x)â¥22Lâ¤ 0,which is (7.3).Theorem 7.17 (Lipschitz smooth on an open set). Let f : Rn â R be aproper lsc function. Let U â dom f be a nonempty open convex set andL > 0. Then the following are equivalent:(i) (âx â U) â1/Lp (Â±f) (x) Ì¸= â .(ii) (âx â U) âF(Lj Â± f)(x) Ì¸= â .1457.5. POINTWISE LIPSCHITZ SMOOTHNESS(iii) The function f is differentiable on U , and(ây â Rn)(âx â U) |f(y)â f(x)â â¨âf(x), y â xâ© | â¤ L2 â¥y â xâ¥2 .(7.8)When one of the above holds, âf is L-Lipschitz on U .Proof.â(i) â (ii)â Apply Theorem 6.6.â(ii) â (iii)â For every x â U , the assumption âF(Lj Â± f)(x) Ì¸= â andsubdifferential calculus yieldâÌ(Â±f)(x) = âÌ (Lj Â± f â Lj) = âÌ (Lj Â± f) (x)â Lxâ âF (Lj Â± f) (x)â Lx Ì¸= â , (7.9)entailing f to be differentiable at x. Consequently (7.9) givesâ =Ì¸ âF (Lj Â± f) (x) â Lx±âf(x)â âF (Lj Â± f) (x) = Lx±âf(x),from which we conclude that â1/Lp (Â±f) (x) = âF (Lj Â± f) â Lx = Â±âf(x)by Theorem 6.6. Then(ây â Rn) f(y) â¥ f(x) + â¨âf(x), y â xâ© â L2 â¥y â xâ¥2 , and(ây â Rn) â f(y) â¥ âf(x) + â¨ââf(x), y â xâ© â L2 â¥y â xâ¥2 ,together furnish (7.8).â(iii) â (i)â Inequality (7.8) yields ââf(x) â â1/Lp (âf)(x) and âf(x) ââ1/Lp f(x) for x â U , thus justifies (i).Finally we show that âf is L-Lipschitz on U provided (iii) holds. Defineg = f + Lj. Then (iii) amounts to (ây â Rn)(âx â U) 0 â¤ g(y) â g(x) ââ¨âg(x), y â xâ© â¤ L â¥xâ yâ¥2 , which, by Lemma 7.16, gives that for x, y â U ,â¥âf(x)ââf(y)+L(xây)â¥22L =â¥âg(y)ââg(x)â¥22L â¤ â¨âg(x)ââg(y), xâ yâ©= â¨âf(x)ââf(y) + L(xâ y), xâ yâ©, (7.10)where the inequality holds by (7.3). From (7.10) we deriveâ¥âf(x)ââf(y)â¥2 + 2Lâ¨âf(x)ââf(y), xâ yâ©+ L2 â¥xâ yâ¥2â¤ 2Lâ¨âf(x)ââf(y), xâ yâ©+ 2L2 â¥xâ yâ¥2 ,which reduces to â¥âf(x)ââf(y)â¥2 â¤ L2 â¥xâ yâ¥2; thus(âx, y â U) â¥âf(x)ââf(y)â¥ â¤ L â¥xâ yâ¥ ,as required.1467.6. INTEGRATION ON LEVEL PROXIMAL SUBDIFFERENTIAL7.6 Integration on level proximal subdifferentialMotivated by the fact that the level proximal subdifferential âÎ»p f is a gener-alized derivative, this section resolves around the natural question: Can weestablish results parallel to the classic fundamental theorem of calculus?Lemma 7.18. Let f : Rn â R be proper, lsc, and 1-coercive. Let x0 âdom âFf and define fÌ : Rn â R byfÌ(x) = f(x0) + supkâNsup(xi,xâi )âgph âFf,i=1,2,...,k.{â¨xâk, xâ xkâ©+kâ1âi=0â¨xâi , xi+1 â xiâ©}.Then fÌ = conv f .Proof. This is [5, Proposition 2.7].It turns out that the âantiderivativeâ of âÎ»p f is the proximal hull of f .Theorem 7.19 (integration of level proximal subdifferential). Let f : Rn âR be proper, lsc and prox-bounded with thresholds λf > 0. Let 0 < λ < λfand x0 â dom âÎ»p f . Define fÌ : Rn â R byfÌ(x) = f(x0) + Î»â1j(x0)â Î»â1j(x)+ supkâNsup(xi,xâi )âgph âÎ»p f,i=1,2,...,k.ï£±ï£´ï£´ï£²ï£´ï£´ï£³kâ1âi=0â¨xâi + Î»â1xi, xi+1 â xiâ©+ â¨xâk + Î»â1xk, xâ xkâ©ï£¼ï£´ï£´ï£½ï£´ï£´ï£¾ . (7.11)Then the following hold:(i) fÌ = hλf the proximal hull of f , and eÎ»fÌ = eλf .(ii) If in addition that f + Î»â1j is convex, then fÌ = f .Proof.(i) By Fact 6.6, âF (f+Î»â1j)(x) = âÎ»p f(x)+Î»â1x. Since dom âF (f+Î»â1j) =dom âÎ»p f Ì¸= â by Proposition 7.2, it is possible to find xi â dom âÎ»p f fori = 0, . . . , k so that (7.11) is well defined. Furthermore, xi â dom âF (f +Î»â1j) â xi â dom âÎ»p f , and âF (f + Î»â1j)(xi) = âÎ»p f(xi) + Î»â1xi for i =1477.6. INTEGRATION ON LEVEL PROXIMAL SUBDIFFERENTIAL0, . . . , k. The function f + Î»â1j is 1-coercive (so epi-pointed) and conv(f +Î»â1j) = conv (f+Î»â1j) by Lemma 7.1(i). Employing Lemma 7.18, we have(âx â Rn) conv(f + Î»â1j)(x)= f(x0) + Î»â1j(x0) + supï£±ï£´ï£´ï£²ï£´ï£´ï£³kâ1âi=0â¨xâi + Î»â1xi, xi+1 â xiâ©+ â¨xâk + Î»â1xk, xâ xkâ©ï£¼ï£´ï£´ï£½ï£´ï£´ï£¾ ,where the supremum is taken for k â N, all x1, . . . , xk â dom âF (f + Î»â1j)and all xâ0+Î»â1x0 â âF (f +Î»â1j)(x0), . . . , xâk+Î»â1xk â âF (f +Î»â1j)(xk).It follows that fÌ(x) = conv(f + Î»â1j)(x) â Î»â1j(x) = hλ(x). Moreover,eÎ»fÌ = eλ[hλf ] = eλf by Proposition 2.56.(ii) Immediate from (i).The fundamental theorem of calculus also states that for differentiablefunctions f1, f2 : R â R, f â²1 = f â²2 if and only f1 = f2 + c for some constantc â R. We now turn to its counterpart for prox-bounded functions, wherehaving identical generalized derivatives does not necessarily imply the sameâantiderivativesâ.Lemma 7.20. For f : Rn â R and λ > 0, (Î»eÎ»f)â = (Î»f)â + j.Proof. By the definition of Moreau envelope, for every y â Rn(ây â Rn) (Î»eÎ»f)â(y)= supxâRn{â¨x, yâ© â λeλf(x)}= supxâRn{â¨x, yâ© â infwâRn{Î»f(w) + 12 â¥xâ wâ¥2}}= supxâRn{â¨x, yâ©+ supwâRn{âÎ»f(w)â 12 â¥xâ wâ¥2}}= supwâRn{â¨w, yâ© â λf(w) + supxâRn{â¨xâ w, yâ© â 12 â¥w â xâ¥2}}= supwâRn{â¨w, yâ© â λf(w) +(12 â¥Â·â¥2)â(y)}=(Î»f)â(y) + 12 â¥yâ¥2 ,competing the proof.1487.6. INTEGRATION ON LEVEL PROXIMAL SUBDIFFERENTIALTheorem 7.21. Let fi : Rn â R be proper, lsc and prox-bounded withthreshold λfi > 0 for i = 1, 2, and let 0 < λ < min{λf1 , λf2}. Consider thefollowing statements:(i) âÎ»p f1 = âÎ»p f2.(ii) Pλf1 = Pλf2.(iii) (âc â R) eλf1 = eλf2 + c.(iv) (âc â R) hλf1 = hλf2 + c.(v) (âc â R) convf1 = convf2 + c, provided that conv fi is proper fori = 1, 2.Then (i)â (ii)â (iii)â (iv)â (v).Proof.â(i) â (ii)â âÎ»p f1 = âÎ»p f2 â Id + Î»âÎ»p f1 = Id + Î»âÎ»p f2 â Pλf1 = PÎ»f2.â(ii) â (iii)â This implication can be found in the proof of [90, Proposition3.16(ii)]. Here we include a similar argument for the sake of completeness.Recall from Fact 2.74 that âeÎ»fi is lower-C2 for each i = 1, 2. Therefore,âeÎ»fi is primal lower-nice in the sense of [73, Definition 3.1] and âp(âeÎ»fi) =â(âeÎ»fi) by [73, Proposition 3.5] for i = 1, 2. Furthermore,(âx â Rn) âp(âeÎ»f1)(x) = Î»â1 [convPÎ»f1(x)â x]=Î»â1 [convPÎ»f2(x)â x] = âp(âeÎ»f2)(x),where the first equality follows from Fact 2.74, and the second from ourassumption. In turn [73, Theorem 4.1] implies that there exists a neigh-borhood of x on which eλf1 â eλf2 is a constant, and this holds for everyx â Rn. Since both eλf1, eλf2 are locally Lipschitz on Rn, we deduce that(âc â R) eλf1 = eλf2 + c.â(iii) â (iv)â Suppose first that (iii) holds. Applying the identity hλfi =âeÎ»(âeÎ»fi), we have hλf1 = âeÎ»(âeÎ»f1) = âeÎ»(âeÎ»f2âc) = â[eÎ»(âeÎ»f2)âc] = hλf2 + c. Conversely, use the identity eλf = eλ(hλf); see Proposi-tion 2.56(ii)â(iii) â (v)â This proof follows from [90, Proposition 3.16] with additionaldetails for the sake of completeness. Invoke Lemma 7.20 to see that(Î»f1)â + j = (Î»eÎ»f1)â = (λeλf2 + c)â = (Î»f2)â + j + c,1497.6. INTEGRATION ON LEVEL PROXIMAL SUBDIFFERENTIALimplying that (Î»f1)â = (Î»f2)â + c, to which taking conjugate to both sidesagain and appealing to Fact 2.34 yieldsλ conv f1 = conv(λf1) = conv(λf2) + c = λ conv f2 + c,which completes the proof.The example below shows that the coincidence of closed convex hulls up toa constant does not imply that of level proximal subdifferentials.Example 7.22. Consider the zero ânormâf : Râ R : x 7â{1, if x Ì¸= 0;0, if x = 0,and g = 0. Then the following hold:(i) conv f = conv g = g = 0.(ii) (âÎ» > 0) âÎ»p f Ì¸= âÎ»p g because for x â R,âÎ»p f(x) =ï£±ï£´ï£²ï£´ï£³â , if 0 < |x| < â2Î»;[ââ2/Î»,â2/Î»], if x = 0;0, if |x| â¥ â2Î»,and âÎ»p g(x) = 0.Proof.(i) We claim that conv f = 0. Because conv f â¥ 0, it suffices to showconv f â¤ 0. Indeed,(âx â R) (conv f)(x) â¤ (conv f)(x)â¤ inf0<Î»<1u,vâR{Î»f(u) + (1â λ)f(v) : x = λu+ (1â Î»)v}â¤ inf0<Î»<1(Î»f(Î»â1x) + (1â λ)f(0)) â¤ inf0<λ<1λ = 0.(ii) Apply [95, Example 4.2].The next example shows that Theorem 7.21(iii) does not imply Theo-rem 7.21(i) either.1507.6. INTEGRATION ON LEVEL PROXIMAL SUBDIFFERENTIALExample 7.23. For λ > 0, define f, g : Râ R byf(x) ={1, if x Ì¸= 0;0, if x = 0,and g(x) ={â2Î»â1|x| â Î»â1j(x), if |x| â¤ â2Î»;1, otherwise .Then both f and g are proper, lsc and prox-bounded with thresholds λf =λg =â. Moreover, the following hold:(i) eλf = eλg, because hλf = g.(ii) Pλf Ì¸= Pλg. To be specific,Pλf(x) =ï£±ï£´ï£²ï£´ï£³{0}, if |x| < â2Î»;{0, x}, if |x| = â2Î»;x, otherwise,andPλg(x) =ï£±ï£´ï£´ï£´ï£´ï£²ï£´ï£´ï£´ï£´ï£³{0}, if |x| < â2Î»;[0,â2Î»], if x =â2Î»;[ââ2Î», 0], if x = ââ2Î»;x, otherwise,whereaseλf(x) = eλg(x) =ï£±ï£²ï£³1, if |x| â¥â2Î»;x22Î», if |x| < â2Î».Proof.(i) To see eλf = eλg, by Proposition 2.56, it suffices to show hλf = g. Tothis end, observe that for every x â Rconv (f + Î»â1j)(x) â¤ inf0<µ<1{µ(f + Î»â1j)(µâ1x)}= inf0<µ<1{µ[1 + (2Î»)â1µâ2x2]}= inf0<µ<1{µ+ x22Î»µâ1},where the first inequality holds because x = (1â µ)0 + µ(µâ1x). We claimthatinf0<µ<1{µ+ x22Î»µâ1}= (g + Î»â1j)(x). (7.12)1517.6. INTEGRATION ON LEVEL PROXIMAL SUBDIFFERENTIALFor simplicity, let a = x2/2λ. Then easy calculus shows that |a| â¤ 1 â|x| â¤ â2Î» andinf0<µ<1{µ+ aµâ1} ={2âa, if |a| â¤ 1;1 + a, if |a| > 1,from which (7.12) readily follows. Moreover, the convex function g+Î»â1j isdominated by f+Î»â1j. To see this, note that (g+Î»â1j)(x) = 1+Î»â1j(x) =(f + Î»â1j)(x) for |x| > â2Î» and(â|x| â¤â2Î») (g + Î»â1j)(x) =â2Î»â1|x|â¤ 12(Î»(2Î»â1) + Î»â1x2)= (f + Î»â1j)(x),where the inequality follows from the Fenchel-Young inequality. Altogetherconv (f +Î»â1j) = g+Î»â1j, and therefore Proposition 2.56 entails hλf = g.(ii) The formula of Pλf is well-known; see, e.g., [95, Example 4.2]. Theo-rem 6.6 yieldsâÎ»p g(x) = âF(g + Î»â1j)(x)â Î»â1x=ï£±ï£´ï£²ï£´ï£³[ââ2Î»â1,â2Î»â1], if x = 0;â2Î»â1 sgn(x)â Î»â1x, if 0 < |x| â¤ â2Î»;0, otherwise,(7.13)and moreover(Id+Î»âÎ»p g)(x) =ï£±ï£´ï£²ï£´ï£³[ââ2Î»,â2Î»], if x = 0;â2Î» sgn(x), if 0 < |x| â¤ â2Î»;x, otherwise.The claimed formula of Pλg follows from inverting the above and Theo-rem 6.5, after which simplifying the identity eλg(x) = g(PÎ»g(x))+(PÎ»g(x)âx)2/2Î» completes the proof.1527.7. CONCLUSIONFigure 7.1: An illustration of Example 7.23.7.7 ConclusionThis chapter presents a systematic study of the level proximal subdifferen-tial, including the level proximal subdifferentiability of a function, and thesingle-valuedness of the level proximal subdifferential. We also used the levelproximal subdifferential to study optimality conditions and a pointwise Lip-schitz smoothness. Additionally, convex integration of this subdifferential isinvestigated. These results illustrate the rich connections between the levelproximal subdifferential and various classic notions in variational analysis.153Chapter 8Variational convexity,relative monotonicity, andlocal nonexpansiveness8.1 IntroductionProximal mappings are of central importance in both convex optimization[11,64,71,77,82] and variational analysis [18,33,44,60,72,74,81]. In convexoptimization, proximal mappings are resolvents of subdifferentials of convexfunctions and firmly nonexpansiveâa fundamental property underpins theconvergence of various splitting algorithms. This line of work has beenstudied extensively, and we refer readers to the monographs [11, 82] for in-depth exposition.In nonconvex optimization, a major barrier is that proximal mappingsof nonconvex functions are set-valued in general and no longer resolventsof their subdifferentials; thus no longer firmly nonexpansive. In fact, it isimpossible to obtain firmly nonexpansive proximal operator in the absenceof convexity, given the following for a proper, lsc and prox-bounded functionf : Rn â Rf is convexâ âf is monotoneâ (âÎ» > 0) Pλf is firmly nonexpansive,(8.1)see Theorem 8.2. However, firm nonexpansiveness may appear locally in theabsence of convexity, even on âbadâ functions. But they cannot be simplycharacterized by local convexity.Example 8.1. Let f(x) = 1 for x Ì¸= 0 and f(0) = 0. Then the proximaloperator of fPλf(x) =ï£±ï£´ï£²ï£´ï£³0, if |x| < â2Î»;{0, x}, if |x| = â2Î»;x, if |x| > â2Î»,1548.1. INTRODUCTIONis firmly nonexpansive on (ââ2Î»,â2Î»). However, f is not convex on thesame interval.The goal of this chapter is to characterize variational and variationalstrong convexity, and to explore their calculus rules and applications to non-convex optimization. Our results, in particular, characterize completely localnotions of nonexpansiveness of the proximal operators of nonconvex func-tions. Our analysis hinges on the relative monotonicity of the associatedlevel proximal subdifferential investigated in Chapters 6 and 7. Main re-sults in this chapter include:R1 We characterize the variational convexity of a function in Theorem 8.19by local (firm) nonexpansiveness of the associated proximal opera-tor and by relative monotonicity of the level proximal subdifferential,which is a natural extension of (8.1) or the more comprehensive The-orem 8.2 in the absence of convexity.R2 Variational strong convexity is characterized in Theorem 8.22. In par-ticular, we show that a proximal operator is a Banach contractionlocally if and only if the underlying function is variationally stronglyconvex.R3 We present in Theorem 8.37 that the proximal gradient operator islocally averaged in the presence of variational convexity, and it be-comes a Banach contraction locally when variational strong convexityis satisfied.The rest of this chapter is organized as following: We provide backgroundinformation and prove needed technical lemmas on variational convexity andrelative monotonicity in Sections 8.2 and 8.3. In Sections 8.4 and 8.5, wecharacterize variational convexity and variational strong convexity, followedby their calculus rules in Section 8.6. We end this chapter by exploringalgorithmic applications of our results in Section 8.7.8.1.1 A classic correspondenceTheorem 8.2. Let f : Rn â R be proper, lsc and prox-bounded with thresh-old λf > 0. Then the following are equivalent:(i) f is convex.(ii) âf is (maximally) monotone.1558.2. VARIATIONAL CONVEXITY AND VARIATIONAL STRONG CONVEXITY(iii) (â0 < λ < λf ) the Moreau envelope eλf is convex.(iv) (â0 < λ < λf ) Pλf is firmly nonexpansive.(v) (â0 < λ < λf ) Pλf is averaged.(vi) (â0 < λ < λf ) Pλf is nonexpansive.Proof. See [81, Theorem 12.17] and [79, Theorem 1] for (i) â (ii) and (i) â(iv), respectively. The equivalence (i) â (iii) is established in [90, Theorem3.17]. It is trivial to see that (iv) â (v) â (vi); while the implication (vi)â (i) is proved in [60, Theorem 3.2].8.2 Variational convexity and variational strongconvexityThe following is proposed by Rockafellar in 2019; see [78].Definition 8.3. A lsc function f : Rn â R is called variationally convex atx¯ for v¯ â âf(x¯), if for some convex neighborhood U ÃV of (x¯, v¯) there exista lsc convex function fÌ â¤ f on U and a number ε > 0 such that[Uε Ã V ] â© gph âf = [U Ã V ] â© gph âfÌ andf(x) = fÌ(x) at the common elements,where Uε = {x â U | f(x) < f(x¯) + ε}.Remark 8.4. Variational convexity is a new notion with emerging inter-est. Some comments are in order. In 2019, Rockafellar proposed varia-tional convexity and provided various characterizations through subgradi-ents; see [78]. More recently, Khanh, Mordukhovich, and Phat [47] estab-lished that the variational convexity of a function is equivalent to the as-sociated Moreau envelope being locally convex and the underlying functionbeing prox-regular [47, Theorem 3.2]. In [46] Khanh, Khoa, Mordukhovichand Phat have extended this equivalence to infinite-dimensional spaces. Wesummarize known finite-dimensional characterizations of variational convex-ity in Theorem 8.13.Variational convexity is stronger than prox-regularity.Proposition 8.5. Let f : Rn â R be lsc. If f is variationally convex at x¯for v¯ â âf(x¯), then f is prox-regular at x¯ for v¯1568.2. VARIATIONAL CONVEXITY AND VARIATIONAL STRONG CONVEXITYProof. The convexity of fÌ in Definition 8.3 implies that(ây â Rn) fÌ(y) â¥ fÌ(x) + â¨v, y â xâ© for (x, v) â gph âf â© [Uε Ã V ].But fÌ(x) = f(x) and fÌ â¤ f on U , so(ây â U) f(y) â¥ fÌ(y) â¥ f(x) + â¨v, y â xâ© for (x, v) â gph âf â© [Uε Ã V ],which means that f is prox-regular at x¯ for v¯; see Definition 2.75.We highlight the following special case of variational convexity:Definition 8.6 (variational sufficiency). Let f : Rn â R be a proper andlsc function. We say that f satisfies the variational sufficient condition atx¯ â Rn, if f is variationally convex at x¯ for 0 â âf(x¯).Variational sufficiency guarantees local optimality.Proposition 8.7. If f satisfies the variational sufficient condition at x¯,then x¯ is a local minimizer of f .Proof. The assumption implies f is variationally convex at x¯ for 0 â âf(x¯).Then there exist a convex function fÌ satisfying 0 â âfÌ(x¯) and fÌ â¤ f on aconvex neighborhood U of x¯ with f(x¯) = fÌ(x¯); see Definition 8.3. Hence forevery x â U , f(x¯) = fÌ(x¯) â¤ fÌ(x) â¤ f(x).Remark 8.8. The converse of Proposition 8.7 is not necessarily true. Thefunctionf(x) ={min{(1 + 1n) |x| â 1n(n+1) , 1n} , if (ân â N) 1n+1 â¤ |x| â¤ 1n ,0, if x = 0,has local minimizer x¯ = 0, but variational sufficiency for f fails at x¯; see [47,Remark 2.8(i)]. This is because of Proposition 8.5 and the fact f is not prox-regular at x¯ for v¯ = 0. Indeed, if there were ε, Ï > 0 such that Definition 2.75holds. Then for k â N sufficiently large such that Ï2k(k+1) < 1, yk = 1k+1 ,and xk =1k , and vk = 0 â âf(xk), it holds thatf(yk) =1k+1 <1k â Ï2(1k â 1k+1)2= f (xk) + â¨vk, yk â xkâ© â Ï2 â¥yk â xkâ¥2 ,which in turn violates Definition 2.75.Example 8.9. Let f(x) = 1 for x Ì¸= 0 and f(0) = 0. Then f is variationallyconvex at every x¯ for all v¯ â âf(x¯).1578.2. VARIATIONAL CONVEXITY AND VARIATIONAL STRONG CONVEXITYProof. Recall from Example 6.13 thatâf(x) ={R, if x = 0;{0} , if x Ì¸= 0.We shall see that Definition 8.3 holds in two cases:Case 1 (x¯ = 0). Choose an arbitrary v¯ â R = âf(x¯). Let fÌ(x) = |x| + v¯xand let U be the neighborhood of x¯ on which fÌ(x) â¤ f(x). ThenâfÌ(x) ={[v¯ â 1, v¯ + 1], if x = 0;sgn(x) + v¯, if x Ì¸= 0.and moreover Uε = {0} for 0 < ε < 1. Putting everything together,gph âf â© [Uε Ã (âÎµ+ v¯, ε+ v¯)] = {0} Ã (v¯ â ε, v¯ + ε) = gph âfÌ â© [U Ã V ],and f(x) = fÌ(x) whenever (x, v) â {0} Ã (v¯ â ε, v¯ + ε).Case 2 (x¯ Ì¸= 0). Let v¯ = 0 â âf(x¯). Consider the constant functionfÌ(x) = 1 and the neighborhoods U = (x¯ â |x¯|, x¯ + |x¯|). Then for ε > 0,Uε = U and gph âf â© [Uε Ã (âÎµ, ε)] = U Ã {0} = gph âfÌ â© [U Ã (âÎµ, ε)].Moreover f(x) = fÌ(x) whenever (x, v) â U Ã {0}.Definition 8.10. A lsc function f : Rn â R is called variationally Ï-strongly convex at x¯ for v¯ â âf(x¯) and Ï > 0, if for some convex neighbor-hood U Ã V of (x¯, v¯) there exist a lsc Ï-strongly convex function fÌ â¤ f onU and a number ε > 0 such that[Uε Ã V ] â© gph âf = [U Ã V ] â© gph âfÌ andf(x) = fÌ(x) at the common elements,where Uε = {x â U | f(x) < f(x¯) + ε}.Next, we collect the known characterizations of variational convexitypresented in [78, Theorem 1] and [47, Theorem 3.2]. We provide additionaldetails in the proof while minimizing the repetition of known arguments,starting from recalling epi-convergence.Definition 8.11. Let (fk)kâN be a sequence of functions fk : Rn â R andlet f : Rn â R. We say that fk epi-converges to f , denoted by fk eââ f , ifepi fk â epi f in the sense of Definition 2.6.1588.2. VARIATIONAL CONVEXITY AND VARIATIONAL STRONG CONVEXITYThe following lemma facilitates calculating epi-convergence.Lemma 8.12 (description of epi-convergence). Let fk : Rn â R for k â Nand let f : Rn â R. Then fk eââ f as k ââ if and only if at each x â Rnlim infkââfk(xk) â¥ f(x) for every sequence xk â x,lim supkââfk(xk) â¤ f(x) for some sequence xk â x.Proof. See [81, Proposition 7.2].Theorem 8.13 (characterizations of variational convexity). Let f : Rn â Rbe proper and lsc. Let x¯ â dom f and v¯ â âÌf(x¯). Then the following areequivalent:(i) f is variationally convex at x¯ for v¯.(ii) âf is f -locally monotone around (x¯, v¯), that is, there exists open convexneighborhoods U â x¯, V â v¯, and ε > 0 such thatgph âf â© [Uε Ã V ] is monotone. (8.2)(iii) f is prox-regular at x¯ for v¯, and the Moreau envelope eλf is locallyconvex around x¯+ λv¯ for all λ sufficiently small.(iv) There is a convex neighborhood U Ã V of (x¯, v¯) along with ε > 0 suchthat for (x, v) â gph âf â© [Uε Ã V ](ây â U) f(y) â¥ f(x) + â¨v, y â xâ©, (8.3)where Uε = {x â U |f(x) < f(x¯) + Îµ}.Proof.â(i) â (ii)â Immediate from the convexity of fÌ in Definition 8.3.â(ii) â (iii) â (iv)â See [47, Theorem 3.2].â(iv) â (i)â We follow the same line of proof in [78, Theorem 1]. DefinefÌ : Rn â R byfÌ(y) = sup {f(x) + â¨v, y â xâ© |(x, v) â gph âf â© [Uε Ã V ]} .1598.2. VARIATIONAL CONVEXITY AND VARIATIONAL STRONG CONVEXITYThen by (iv) and the construction, fÌ is lsc and convex with fÌ â¤ f on U ,and moreoverf(x) = f(x) + â¨v, xâ xâ© â¤ fÌ(x)â f(x) = fÌ(x),for (x, v) â gph âf â© [Uε Ã V ]. We prove first that for possibly smaller Uand VC1: (x, v) â gph âf â© [Uε Ã V ] â (x, v) â gph âfÌ â© [U Ã V ] and fÌ(x) =f(x).âââ of C1: Pick (x, v) â gph âf â© [UÎµÃV ]. Then f(x) = f(x)+â¨v, xâxâ© â¤ fÌ(x) â¤ f(x). Moreover for every y â RnfÌ(y) â¥ f(x) + â¨v, y â xâ© = fÌ(x) + â¨v, y â xâ© â (x, v) â gph âfÌ .âââ of C1: Take (x, v) â gph âfÌ â© [U Ã V ] and fÌ(x) = f(x). Recallthat fÌ â¤ f on U . Then(ây â U) f(y) â¥ fÌ(y) â¥ fÌ(x) + â¨v, y â xâ© = f(x) + â¨v, y â xâ©,implying thatlim infyâxf(y)âf(x)ââ¨v,yâxâ©â¥yâxâ¥ â¥ 0â v â âÌf(x) â âf(x).It remains to see that x â Uε. Shrinking U if necessary, assume withoutloss of generality that for x â U , â¥v¯â¥ â¥xâ x¯â¥ < ε and therefore f(x) â¤f(x¯)â â¨v¯, xâ x¯â© â¤ f(x¯) + â¥v¯â¥ â¥xâ x¯â¥ < f(x¯) + ε.With C1, it remains to show, with possibly smaller U and V , that(x, v) â gph âfÌ â© [U Ã V ]â fÌ(x) = f(x), (8.4)which implies gph âf â© [UÎµÃV ] = gph âfÌ â© [U ÃV ] and thereby all require-ments of variational convexity are justified (recall Definition 8.3). To thisend, pick a closed ball C â U of x¯ and define for (x, v) â gph âfÌhx,v(y) = f(y)â fÌ(x)â â¨v, y â xâ©+ 12 â¥y â xâ¥2 + δC(y). (8.5)We claim thatC2: argminhx¯,v¯ = {x¯}, minhx¯,v¯ = 0, andhx,veââ hx¯,v¯ as (x, v) gph âfÌâââââ (x¯, v¯).1608.2. VARIATIONAL CONVEXITY AND VARIATIONAL STRONG CONVEXITYThereby invoking [81, Theorem 7.33] yieldsargminhx,v â {x¯} and minhx,v â 0 as (x, v) gph âfÌâââââ (x¯, v¯). (8.6)Now take xË â argminhx,v. The following hold as (x, v) gph âfÌâââââ (x¯, v¯):â In view of (8.6),f(xË)â f(x)â â¨v, xËâ xâ©+ 12 â¥xâ xâ¥2 â 0â f(xË)â f(x¯).So xË â UÎµ.â Appealing to the optimality condition Fact 2.14 gives 0 â âhx,v(xË) =âf(xË)â v + xËâ xâ vË â âf(xË) with vË = v + xâ xË. Thereby vË â v¯.With the analysis above, we may assume without loss of generality that(x, v) â gph âfÌ â© [U Ã V ]â (xË, vË) â gph âf â© [Uε Ã V ],where xË â argminhx,v and vË = v + x â xË. Apply C1 again to (xË, vË) âgph âf â© [Uε Ã V ] to see that(xË, vË) â gph âfÌ â© [U Ã V ] and fÌ(xË) = f(xË), (8.7)but this means0 â¤ â¨xâ xË, v â vËâ© = â¨xâ xË, xËâ xâ© = ââ¥xâ xËâ¥2 â x = xË,and therefore fÌ(x) = fÌ(xË) = f(xË) = f(x), where the second equality owesto (8.7). Hence, we have proved (8.4) and thus justified by the desired (i).It remains to justify C2. Note that f(y) â¥ fÌ(y) â¥ fÌ(x) + â¨v, y â xâ© fory â C and (x, v) â gph âfÌ . Then the definition (8.5) implies that(ây â Rn) hx,v(y) â¥ 0 with hx,v(y) = 0â y = x and f(x) = fÌ(x).and thereby in view of C1,(x, v) â gph âf â© [Uε Ã V ]â argminhx,v = {x} and minhx,v = 0,which in particular holds at (x, v) = (x¯, v¯). Finally we show the epi-convergence hx,veââ hx¯,v¯ as (x, v) gph âfÌâââââ (x¯, v¯). For every (x, v) â gph âfÌ ,1618.2. VARIATIONAL CONVEXITY AND VARIATIONAL STRONG CONVEXITYthe convexity of fÌ ensures fÌ(x¯) â¥ fÌ(x) + â¨v, x¯ â xâ©. Thereby for everysequence (xk, vk)gph âfÌâââââ (x¯, v¯),fÌ(x¯) â¥ lim supkââfÌ(xk) + â¨vk, x¯â xkâ© = lim supkââfÌ(xk) â¥ lim infkââfÌ(xk) = f(x¯),where the last inequality owes to the lower semicontinuity of fÌ , meaningthatfÌ(x)â fÌ(x¯) whenever (x, v) gph âfÌâââââ (x¯, v¯). (8.8)It is then easy to verify that the conditions in Lemma 8.12 are satisfiedand thus hx,veââ hx¯,v¯ as (x, v) gph âfÌâââââ (x¯, v¯). Indeed, consider an arbitraryy â Rn. Then for every yk â y and every (xk, vk) gph âfÌâââââ (x¯, v¯)lim infkââhxk,vk(yk)= lim infkââf(yk)â fÌ(xk)â â¨vk, yk â xkâ©+ 12 â¥yk â xkâ¥2 + Î´C(yk)â¥ f(y) + Î´C(y)â fÌ(x¯)â â¨v¯, y â x¯â©+ 12 â¥y â x¯â¥2 = hx¯,v¯(y),where the inequality follows from (8.8) and the lower semicontinuity off + δC . Next, for the constant sequence (âk â N) yk = y and every(xk, vk)gph âfÌâââââ (x¯, v¯),lim supkââhxk,vk(yk)= lim supkââf(y)â fÌ(xk)â â¨vk, y â xkâ©+ 12 â¥y â xkâ¥2 + δC(y)= f(y) + Î´C(y)â fÌ(x¯)â â¨v¯, y â x¯â©+ 12 â¥y â x¯â¥2 = hx¯,v¯(y),where the second equality again owes to (8.8).Theorem 8.14 (characterizations of variational strong convexity). Let f :Rn â R be proper and lsc. Let x¯ â dom f and v¯ â âÌf(x¯), and let Ï > 0.Then the following are equivalent:(i) f is variationally Ï-strongly convex at x¯ for v¯.(ii) âf is f -locally Ï-strongly monotone around (x¯, v¯), that is, there existsopen convex neighborhoods U â x¯, V â v¯, and ε > 0 such thatgph âf â© [Uε Ã V ] is Ï-strongly monotone. (8.9)1628.3. RELATIVE MONOTONICITY(iii) f is prox-regular at x¯ for v¯ and the Moreau envelope eλf is locallystrongly convex around x¯+ λv¯ for all λ sufficiently small.(iv) There is a convex neighborhood U Ã V of (x¯, v¯) along with ε > 0 suchthat(ây â U) f(y) â¥ f(x) + â¨v, y â xâ©+ Ï2 â¥y â xâ¥2 , (8.10)whenever (x, v) â gph âf â© [Uε Ã V ],where Uε = {x â U |f(x) < f(x¯) + ε}.Proof. See [78, Theorem 2] and [47, Theorem 4.5].8.3 Relative monotonicityDefinition 8.15. Let A : Rn â Rn be a set-valued operator.(i) A is monotone relative to a set W â Rn Ã Rn if (â(xi, ui) â gphA â©W, i = 1, 2) â¨x1 â x2, u1 â u2â© â¥ 0.(ii) A is maximally monotone relative to a set W â RnÃRn if it is mono-tone relative toW and there exists no operator B : Rn â Rn monotonerelative to W such that gphA â©W â gphB â©W .(iii) We say that A is Ï-monotone for Ï â R relative to W â Rn Ã Rn, if(â(xi, ui) â gphA â©W, i = 1, 2) â¨x1 â x2, u1 â u2â© â¥ Ï â¥u1 â u2â¥2.The following lemma connects local cocoercivity and relative monotonic-ity, which will be useful soon.Lemma 8.16. Let A : Rn â Rn be a set-valued mapping, let λ > 0, andlet â Ì¸= U â dom JλA. Define Wλ = {(u, v) â Rn Ã Rn| u+ λv â U}. Thenfor γ > 0, JλA is γ-cocoercive on U if and only if A is1λ (γ â 1)-monotonerelative to WÎ».Proof.âââ Take (ui, vi) â gphA â©WÎ» and set xi = ui + λvi for i = 1, 2. Thenxi â U and (ui, vi) = (JλA(xi), Î»â1[xi â JλA(xi)]) by Lemma 8.18(i). Thenthe assumption implies thatâ¨u1 â u2, v1 â v2â© = 1Î»â¨JÎ»A(x1)â JλA(x2), (x1 â x2)â (JÎ»A(x1)â JÎ»A(x2))â©â¥ 1λ (γ â 1) â¥JÎ»A(x1)â JÎ»A(x2)â¥2 = 1λ (γ â 1) â¥u1 â u2â¥2 ,as claimed.1638.4. CHARACTERIZING VARIATIONAL CONVEXITYâââ Suppose that x â U â dom JλA. Then x = u + λv for some (u, v) âgphA, consequently (u, Î»â1(xâu)) â gphAâ©WÎ» and u â JλA(x). We claimthat JλA is single-valued. Indeed, let uâ² â JλA(x). Then the same processas above implies (uâ², Î»â1(xâ uâ²)) â gphA â©WÎ», and monotonicity impliesâ¨uâuâ², 1Î»(xâu)â 1Î»(xâuâ²)â© â¥ 1Î»(Î³â1)â¥â¥uâ uâ²â¥â¥2 â γ â¥â¥uâ uâ²â¥â¥ â¤ 0â u = uâ².Next, let ui = JλA(xi) with xi â U for i = 1, 2. Then there exists vi â A(ui)such that xi = ui + λvi and the strong monotonicity of gphA â©WÎ» givesâ¨u1 â u2, v1 â v2â© â¥ 1λ (γ â 1) â¥u1 â u2â¥2.Lemma 8.17. Let A : Rn â Rn be a set-valued mapping, let λ > 0, and letâ =Ì¸ U â dom JλA. Define Wλ = {(u, v) â Rn Ã Rn| u+ λv â U}. Then thefollowing are equivalent:(i) JλA is firmly nonexpansive on U ;(ii) A is monotone relative to Wλ;(iii) A is maximally monotone relative to Wλ.Proof. The equivalence (i) â (ii) follows from applying Lemma 8.16 withÏ = 1, hence we prove only the direction (ii) â (iii). Take (u, v) â Wλthat is monotonically related to gphA â©WÎ». We claim that (u, v) â gphA,which entails the desired maximal monotonicity. Indeed, (u, v) âWÎ» impliesu+ λv â U â dom JλA. Let y â JλA(u+ λv). Then(y, Î»â1(u+ λv â y)) âgphAâ©WÎ». By the assumption that (u, v) is monotonically related to gphAâ©WÎ», we obtainâ¨uâ y, Î»â1(y â u)â© = â¨uâ y, v â Î»â1(u+ λv â y)â© â¥ 0â u = y,enforcing that u â JλA(u+ λv). In turn u+ λv â u+ Î»Auâ v â Au.8.4 Characterizing variational convexityWe shall need a localized Mintyâs parametrization.Lemma 8.18. Let U be a nonempty set in Rn, let λ > 0, and let Wλ ={(u, v) â Rn Ã Rn| u+ λv â U}.1648.4. CHARACTERIZING VARIATIONAL CONVEXITY(i) Suppose that A : Rn â Rn and that JλA is single-valued on U . ThengphA â©WÎ» = {(JλA(x), Î»â1[xâ JλA(x)])| x â U}. (8.11)(ii) Suppose that f : Rn â R is proper, lsc and prox-bounded with thresholdλf , and 0 < λ < λf . If Pλf is single-valued on U , thengph âÎ»p f â©WÎ» = {(Pλf(x), Î»â1[xâ Pλf(x)])| x â U} (8.12)= {(xâ Î»âeÎ»f(x),âeÎ»f(x))| x â U}. (8.13)Proof.(i) Denote the right-hand side of (8.11) by Eλ. We have(u, v) â Eλ â (âx â U) u = JλA(x), v = Î»â1[xâ JÎ»A(x)]â (âx â U) x â u+ λAu, x = u+ Î»vâ (âx â U) v â Au, x = x+ λv â (u, v) â gphA â©WÎ».(ii) Apply (i) with A = âÎ»p f to obtain (8.12). When 0 < λ < λf and Pλf issingle-valued on U , Fact 2.77 implies (âx â U) âeÎ»f(x) = Î»â1[xâPÎ»f(x)].Hence (8.13) holds.Here is the main result of this section which is a generalization of Theo-rem 8.2, unifying and enhancing recent advancements on variational con-vexity by Rockafellar [79, Theorem 1] and by Khanh, Mordukhovich, andPhat [47, Theorem 3.2].Theorem 8.19 (characterizations of variational convexity). Suppose thatf : Rn â R is proper, lsc and prox-bounded with threshold λf > 0, and thatf is prox-regular at x¯ for v¯ â âf(x¯). Then there exists λ¯ > 0 such that thefollowing are equivalent:(i) f is variationally convex at x¯ for v¯.(ii) (â0 < λ < λ¯) âÎ»p f is monotone relative to Wλ.(iii) (â0 < λ < λ¯) âÎ»p f is maximally monotone relative to Wλ.(iv) (â0 < λ < λ¯) Pλf is firmly nonexpansive on Uλ.(v) (â0 < λ < λ¯) (â0 < α < 1) Pλf is α-averaged on Uλ.(vi) (â0 < λ < λ¯) Pλf is nonexpansive on Uλ.1658.4. CHARACTERIZING VARIATIONAL CONVEXITY(vii) (â0 < λ < λ¯) eλf is convex on Uλ.(viii) (â0 < λ < λ¯) eλf is convex on Uλ,(ix) (â0 < λ < λ¯) âeÎ»f is λ-cocoercive on Uλ.where the set Uλ is an open convex neighborhood of x¯+λv¯ andWλ = {(x, v) âRn Ã Rn| x+ λv â Uλ}.Proof. By Fact 2.77, there exists λ¯ > 0 such that to every 0 < λ < λ¯corresponds an open convex neighborhood Uλ of x¯+ λv¯ on which:Pλf is monotone, single-valued and Lipschitz continuous; Pλf(x¯+ λv¯) = x¯;(8.14)and eλf is differentiable withâeÎ»f = Î»â1[IdâPÎ»f ]. (8.15)In particular, (8.14) implies v¯ â âÎ»p f(x¯); see Theorem 6.5.â(i) â (ii)â The assumption implies that there exist convex neighborhoodsU â x¯ and V â v¯ and ε > 0 such thatgph âf â© [Uε Ã V ] is monotone, (8.16)where Uε = {u â U |f(x) < f(x¯) + ε}; see Theorem 8.13. For every 0 < λ <λ¯, (8.14) and (8.15) entail that âeÎ»f(x¯ + λv¯) = v¯. For u sufficiently closeto x¯+ λv¯, (8.15) implies that (IdâÎ»âeÎ»f)(u) = Pλf(u). Henceuâ x¯+ λv¯x = PÎ»f(u)âï£±ï£´ï£²ï£´ï£³(IdâÎ»âeÎ»f)(u) = xâ x¯,âeÎ»f(u) = Î»â1(uâ x)â v¯,f(x)â f(x¯)(8.17)where the first limit follows from the outer semicontinuity of Pλf (recallFact 2.54) and the last holds becausef(x) = eÎ»f(u)â 12λ â¥uâ xâ¥2 â eλf(x¯+ Î»v¯)â 12λ â¥x¯+ λv¯ â x¯â¥2 = f(x¯)as uâ x¯+ λu¯. Owing to (8.17), we may assume without loss of generalitythat for u â UÎ»â¥uâ Î»âeÎ»f(u)â x¯â¥ < ε, â¥âeÎ»f(u)â v¯â¥ < ε, f(uâ Î»âeÎ»f(u)) < f(x¯) + ε.1668.4. CHARACTERIZING VARIATIONAL CONVEXITYRecall that (8.15) impliesuâ Î»âeÎ»f(u) = Pλf(u) = argminxâRn{f(x) + 12λ â¥uâ xâ¥2}.Therefore the optimality condition Fact 2.14 entails0 â âf(uâ Î»âeÎ»f(u)) + 1λ (uâ Î»âeÎ»f(u)â u)ââeÎ»f(u) â âf(uâ Î»âeÎ»f(u)).As such, Lemma 8.18 entailsgph âÎ»p f â©WÎ» = {(uâ Î»âeÎ»f(u),âeÎ»f(u))|u â Uλ} â gph âf â© [Uε Ã V ] .Apply (8.16) to see that gph âÎ»p f â©WÎ» is monotone.â(ii) â (iii) â (iv)â Recall that Pλf = (Id+Î»âÎ»p f)â1; see Theorem 6.5.Then applying Lemma 8.17 completes the proof.â(iv) â (v) â (vi)â Clear.â(vi) â (vii)â The assumption implies that âeÎ»f is monotone on Uλ, soeλf is convex on Uλ in view of [42, B Theorem 4.1.4]. Indeed, by thenonexpansiveness of Pλf on Uλ and the Cauchy-Schwarz inequality,(âx1, x2 â Uλ) â¨PÎ»f(x1)â Pλf(x2), x1 â x2â© â¤ â¥x1 â x2â¥2 . (8.18)Since âeÎ»f = Î»â1[IdâPÎ»f ] on Uλ, it follows from (8.18) that(âx1, x2 â Uλ) â¨x1 â x2,âeÎ»f(x1)ââeÎ»f(x2)â© â¥ 0,hence âeÎ»f is monotone on UÎ».â(vii) â (viii) â (i)â The first implication is trivial. We prove the sec-ond following the same line of proof in [47, Theorem 3.2] for the sake ofcompleteness. The assumption (viii) implies that(âx, y â Uλ) eλf(y) â¥ eλf(x) + â¨âeÎ»f(x), y â xâ©. (8.19)By prox-regularity, there exist γ > 0 and T : Rn â Rn defined byT (x) ={{v â âf(x)| â¥v â v¯â¥ < γ}, if â¥xâ x¯â¥ < γ, f(x) < f(x¯) + Î³,â , otherwise,1678.4. CHARACTERIZING VARIATIONAL CONVEXITYsuch that âeÎ»f =(λ Id+Tâ1)â1on Uλ; see Fact 2.77. Take ε > 0 suffi-ciently small so that ε < γ and x + λv â Uλ for x â U and v â V , whereU := x¯+ εB and V := v¯ + εB. Then v â T (x) for (x, v) â gph âf â© [Uε Ã V ]with Uε := {x â U |f(x) < f(x¯) + ε}, implying thatâeÎ»f(x+ λv) =(λ Id+Tâ1)â1(x+ λv) = v and Pλf(x+ λv) = x. (8.20)Therefore for every y â U and (x, v) â gph âf â© [Uε Ã V ], y + λv â Uλ andf(y) â¥ eλf(y + Î»v)â 12λ â¥y + λv â yâ¥2â¥ eλf(x+ λv) + â¨âeÎ»f(x+ λv), y â xâ© â 12λ â¥x+ λv â xâ¥2= f(x) + â¨v, y â xâ©,where the second inequality owes to (8.19) and the equality is implied by(8.20). Appeal to Theorem 8.13 complete the proof.â(iv) â (ix)â Observe that (ix) amounts to IdâPÎ»f = Î»âeÎ»f is firmlynonexpansive on Uλ, which is equivalent to (iv).Remark 8.20. Some comments are in order.(i) Theorem 8.19 establishes a new correspondence between various classicnotions in convex optimization in the absence of convexity, extendingTheorem 8.2; see Figure 8.1 for the big picture. We primarily highlightthe relation (i) â (iv) â (v) â (vi), which reveals that various no-tions of nonexpansiveness of proximal operators reemerge in nonconvexsetting locally if and only if the underlying function is variationallyconvex, in which case all these notions coincide. This is enabled byusing the level proximal subdifferential and relative monotonicity, sothat the elegant technique in classic setting utilizing resolvents of sub-differentials and the Minty parametrization can be preserved.(ii) Akin to the observation in [47, Remark 3.3], the reemergence of classicnotions in convex optimization investigated in Theorem 8.19 opensthe door for a promising new venue to analyze first-order methods inthe absence of convexityânonconvex but variationally convex problemsreduce to convex ones locally! See Section 8.7 for illustrations.1688.4. CHARACTERIZING VARIATIONAL CONVEXITYWorld If is convexââf is maximally monotoneâeÎ»f is convexâPÎ»f is nonexpansiveâPÎ»f is averagedâPÎ»f is firmly nonexpansiveWorld IIf is variationally convex at (x¯, v¯)ââÎ»p f is maximally monotone relative to WÎ»âeÎ»f is convex on UÎ»âPÎ»f is nonexpansive on UÎ»âPÎ»f is averaged on UÎ»âPÎ»f is firmly nonexpansive on UλFigure 8.1: A correspondence between convex and variationally convex set-tings.For λ > 0 and a subset B â Rn, define the localized proximal mapping by(âx â B) PBλ f(x) = argminyâB{f(y) + 12Î»â¥y â xâ¥2}.Next we investigate the case where a function is variationally convex ata stationary point. We shall see Rockafellarâs recent breakthrough [79, The-orem 3], where he characterized variational convexity at stationary pointsusing localized proximal operator, will be partially recovered.Theorem 8.21. Let f : Rn â R be proper, lsc, and prox-bounded. Supposethat f : Rn â R is variationally convex at (x¯, 0) â gph âf . Then for allλ > 0 sufficiently small there exists an open ball B centered at x¯ on whichPλf = PBλ f and Pλf is firmly nonexpansive on B.Proof. Since f is variationally convex at x¯ for 0 â âf(x¯), it is prox-regular x¯for 0 â âf(x¯) and prox-bounded. Then there exists a ball B1 centered at x¯on which Pλf is single-valued, Lipschitz and Pλf(x¯) = x¯; see, e.g., Fact 2.77.By Theorem 8.19, we can find an open ball B2 centered at x¯ and B2 â B1on which Pλf is firmly nonexpansive. Because(âx â B2) â¥PÎ»f(x)â x¯â¥ = â¥PÎ»f(x)â PÎ»f(x¯)â¥ â¤ â¥xâ x¯â¥,this implies that Pλf(x) â B2 if x â B2. Since Pλf(x) is a global uniqueminimizer ofy 7â f(y) + 12Î»â¥y â xâ¥2,it must be a unique minimizer on B2. With B = B2, we have (âx âB) Pλf(x) = PBλ f(x).1698.5. CHARACTERIZING VARIATIONAL STRONG CONVEXITY8.5 Characterizing variational strong convexityTheorem 8.22. Suppose that f : Rn â R is proper, lsc and prox-boundedwith threshold λf > 0, and that f is prox-regular at x¯ for v¯ â âf(x¯). Thenthere exists λ¯ > 0 such that the following are equivalent for Ï > 0:(i) f is variational Ï-strongly convex at x¯ for v¯ â âf(x¯).(ii) (â0 < λ < λ¯) âÎ»p f is Ï-strongly monotone relative to Wλ.(iii) (â0 < λ < λ¯) Pλf is (1 + Î»Ï)-cocoercive on Uλ.(iv) (â0 < λ < λ¯) Pλf is 11+Î»Ï -Lipschitz on Uλ.(v) (â0 < λ < λ¯) eλf is Ï1+ÏÎ» -strongly convex on Uλ.Here Uλ is an open convex neighborhood of x¯ + λv¯ and Wλ = {(x, v) âRn Ã Rn| x+ λv â Uλ}.Proof. By Fact 2.77, there exists λ¯ > 0 such that to every 0 < λ < λ¯corresponds an open convex neighborhood Uλ of x¯+ λv¯ on which:Pλf is monotone, single-valued and Lipschitz continuous; Pλf(x¯+ λv¯) = x¯;and eλf is differentiable withâeÎ»f = Î»â1[IdâPÎ»f ].â(i) â (ii)â In view of Fact 2.77, prox-regularity implies that there existsγ > 0 such that set-valued operator T : Rn â Rn defined byT (x) ={{v â âf(x) : â¥v â v¯â¥ < γ}, if â¥xâ x¯â¥ < γ and f(x) < f(x¯) + Î³;â , otherwise,satisfies âeÎ»f(x) =(λ Id+Tâ1)â1(x) for x â Uλ, and therefore for x â UÎ»,(âeÎ»f(x), xâ Î»âeÎ»f(x)) â gphT. Invoking Lemma 8.18 yieldsgph âÎ»p f â©WÎ» â gphT (8.21)Shrinking γ if necessary, condition (i) together with Theorem 8.14 enforcesgphT to be Ï-strongly monotone, and hence in particular gph âÎ»p f â©WÎ».â(ii) â (iii)â Apply Lemma 8.16.1708.6. CALCULUS RULES OF VARIATIONAL CONVEXITYâ(iii) â (iv)â For every x1, x2 â Uλ, â¥PÎ»f(x1)â PÎ»f(x2)â¥2 â¤ 11+Î»Ï â¨x1 âx2, PÎ»f(x1)â PÎ»f(x2)â© â¤ 11+Î»Ï â¥PÎ»f(x1)â PÎ»f(x2)â¥ â¥x1 â x2â¥, completingthe proof.â(iv) â (v)â For every x1, x2 â Uλ, â¨âeÎ»f(x1) â âeÎ»f(x2), x1 â x2â© =1λ â¥x1 â x2â¥2 â 1Î»â¨x1 â x2, Pλf(x1) â PÎ»f(x2)â© â¤ 1Î»(1â 11+ÏÎ»)â¥x1 â x2â¥2,where the inequality follows from the Cauchy-Schwarz inequality and (iv).â(v) â (i)â Apply Theorem 8.14.8.6 Calculus rules of variational convexityWe start with a sum rule on variationally convex functions.Theorem 8.23 (sum rule). Let f : Rn â R be proper and lsc and letg : Rn â R be convex and C1 on a neighborhood of x¯. Let x¯ â dom f . If fis variationally convex at x¯ for v¯ â âf(x¯), then f +g is variationally convexat x¯ for v¯ +âg(x¯) â â(f + g)(x¯).Proof. Since f is variationally convex at x¯ for v¯, by Theorem 8.13, thereexist convex neighborhoods U of x¯ and V of v¯, and ε > 0 such that V â v¯ =â(V â v¯),(âx â U)(â(u, v) â gph âf â© (Uε Ã V )) f(x) â¥ f(u) + â¨v, xâ uâ©, (8.22)where Uε := {u â U |f(u) < f(x¯) + 2ε}. Choose a smaller neighborhood U ,if necessary, to make sure that g is convex and C1 on U .Next, because g is C1, we may choose a smaller neighborhood U1 of x¯such that U1 â U ,(âu â U1) |g(u)â g(x¯)| < ε, and âg(u) â âg(x¯) + Vâv¯2 .Forv â â(f + g)(u) = âf(u) +âg(u), (8.23)(u, v) â U1,ε Ã(v¯ +âg(x¯) + Vâv¯2), where (8.24)U1,ε := {u â U1 |(f + g)(u) < (f + g)(x¯) + ε} , (8.25)1718.6. CALCULUS RULES OF VARIATIONAL CONVEXITYone has v ââg(u) â âf(u),v ââg(u) = v â (v¯ +âg(x¯))â (âg(u)ââg(x¯)) + v¯â Vâv¯2 + Vâv¯2 + v¯ = V â v¯ + v¯ = V,andf(u)â f(x¯) = [(f + g)(u)â (f + g)(x¯)]â (g(u)â g(x¯)) < ε+ ε = 2ε. (8.26)Thus, applying (8.22) to (u, v ââg(u)) â gph âf gives(âx â U1) f(x) â¥ f(u) + â¨v ââg(u), xâ uâ©. (8.27)Since g is convex on U1,(âx â U1) g(x) â¥ g(u) + â¨âg(u), xâ uâ©. (8.28)Adding (8.27) and (8.28) yields(âx â U1) f(x) + g(x) â¥ f(u) + g(u) + â¨v, xâ uâ©, (8.29)whenever (u, v) satisfies (8.23)â(8.25). Using Theorem 8.13 again, we con-clude that f + g is variationally convex at x¯ for v¯ +âg(x¯).Theorem 8.24 (separable sum). Let fi : Rni â R be proper and lsc for i =1, . . . ,m, and let f : Rn â R be defined by f(x1, x2, . . . , xm) =âmi=1 fi(xi).If for every i = 1, . . . ,m, the function fi is variationally convex at x¯i forv¯i â âfi(x¯i), then f is variationally convex at x¯ = (x¯1, . . . , x¯m) for v¯ =(v¯1, . . . , v¯m) â âf(x¯).Proof. Invoking Theorem 8.13 yields that to each index i â {1, . . . ,m}corresponds open convex neighborhoods Ui â x¯ and Vi â v¯, and εi > 0 suchthatgph âfi â© [Ui,εi Ã V ] is monotone, (8.30)where Ui,εi = {xi â Ui |fi(xi) < fi(x¯i) + εi }. For each i â {1, . . . ,m}, as-sume without loss of generality that x¯i + εiB â Ui. Define U = x¯+ εB andV = v¯ + εB with ε = min1â¤iâ¤m εi/2. Thenx¯ â U â (â1 â¤ i â¤ m) â¥xi â x¯iâ¥ < εi â (â1 â¤ i â¤ m) xi â Ui, (8.31)v¯ â V â (â1 â¤ i â¤ m) â¥vi â v¯iâ¥ < εi â (â1 â¤ i â¤ m) vi â Vi. (8.32)1728.6. CALCULUS RULES OF VARIATIONAL CONVEXITYIn view of the lower semicontinuity of fi, assume without loss of generalitythat for every 1 â¤ i â¤ mâ¥xi â x¯iâ¥ < εi â fi(x¯i) < fi(xi) + Îµmâ1 .Hence for x â U and f(x) < f(x¯) + Îµ(â1 â¤ i â¤ m) fi(xi) < fi(x¯i) + Îµ+mâj=1j Ì¸=i(fj(x¯j)â fj(xj))< fi(x¯i) + 2ε = fi(x¯i) + min1â¤iâ¤mÎµi â¤ fi(x¯i) + εi, (8.33)Combining (8.31)â(8.33) and the separable sum rule of subdifferential Fact 2.15yields that for every 1 â¤ i â¤ mgph âfâ©[UÎµÃV ] â (gph âf1 â© [U1,ε1 Ã V1])ÃÂ· · Â·Ã(gph âfm â© [Um,εm Ã Vm]) ,where Uε = {x â U |f(x) < f(x¯) + ε}. In view of (8.30), each gph âf â©[Ui,Îµi Ã Vi] is monotone, and therefore so is gph âf â© [Uε Ã V ], from whichTheorem 8.13 indicates that f is variationally convex at x¯ for v¯.Theorem 8.25 (composition). Let g : Rn â R be proper lsc, let A â RmÃnbe surjective, and let f(x) = g(Ax). If g is variationally convex at Ax¯ forv¯ â âg(Ax¯), then f is variationally convex at x¯ for AT v¯ â âf(x¯).Proof. By Theorem 8.13, suppose thatgph âg â© [UAx¯,ε Ã (v¯ + εB)] is monotone, (8.34)for some ε > 0 and UAx¯,ε = {x â Ax¯+ εB |g(x) < g(Ax¯) + ε}. Thenâ¥AxâAx¯â¥ â¤ â¥Aâ¥ â¥xâ x¯â¥ < ε whenever â¥xâ x¯â¥ < â¥Aâ¥â1 ε.Let Ux¯,â¥Aâ¥â1Îµ ={x â x¯+ â¥Aâ¥â1 εB |f(x) < f(x¯) + ε}. Thenx â Ux¯,â¥Aâ¥â1Îµ â Ax â UAx¯,ε. (8.35)The subdifferential composition rule Fact 2.19 implies that to each (x, u) âgph âf corresponds v â âg(Ax) such that u = AT v. Thereforeâ¥â¥uâAT v¯â¥â¥ = â¥â¥AT v âAT v¯â¥â¥ â¥ ÏAT â¥v â v¯â¥ ,1738.6. CALCULUS RULES OF VARIATIONAL CONVEXITYwhere ÏAT denotes the minimal singular value of AT and ÏAT > 0 owing tothe surjectivity of A. Therefore for u â âf(x) and v â âg(Ax) satisfyingu = AT v,â¥v â v¯â¥ < ε whenever â¥â¥uâAT v¯â¥â¥ < ÏAT ε. (8.36)Define V = AT v¯ + ÏAT εB. Combining (8.35) and (8.36) yields that to each(xi, ui) â gph âf â© [Ux¯,ÎµÃV ] corresponds vi â âg(Axi) such that ui = AT vi,(Axi, vi) â gph âg â© [UAx¯,ε Ã (v¯ + εB)]and thereforeâ¨x1 â x2, u1 â u2â© = â¨x1 â x2, AT v1 âAT v2â© = â¨Ax1 âAx2, v1 â v1â© â¥ 0,where the last inequality follows from (8.34), proving thatgph âf â© [Ux¯,ε Ã V ] is monotone.Apply Theorem 8.13 completes the proof.Definition 8.26. Let f, g : Rn â R be proper, lsc, and prox-boundedfunctions with thresholds λf , λg > 0 respectively. The proximal average off, g with parameter 0 < λ < min(λf , λg) and 0 â¤ α â¤ 1 isÏÎ±Î»(f, g) = â eλ (âÎ± eλf â (1â α) eλg) .Lemma 8.27. Let f, g : Rn â R be proper, lsc, and prox-bounded functionswith thresholds λf , λg > 0 respectively. Then the following hold for 0 < λ <min(λf , λg) and 0 â¤ α â¤ 1(i) ÏÎ±Î»(f, g) is prox-regular.(ii) The Moreau envelope of proximal average satisfieseλ (ÏÎ±Î»(f, g)) = α eλf + (1â α) eλg.Theorem 8.28 (proximal average). Let f, g : Rn â R be proper, lsc, andprox-bounded functions. Suppose that both f, g are variationally convex atx¯ for v¯ â âf(x¯) â© âg(x¯). Then there exists λ¯ > 0 sufficiently small suchthat for every 0 < λ < λ¯ and 0 â¤ α â¤ 1 the proximal average ÏÎ±Î»(f, g) isvariationally convex at x¯ for v¯.1748.7. ALGORITHMIC APPLICATIONSProof. The variational convexity of f, g at x¯ for v¯ â âf(x¯) â© âg(x¯) impliesthat Moreau envelopes eλf and eλg are locally convex around x¯ + λv¯ forevery 0 < λ < λ¯. Shrinking λ if necessary, apply Lemma 8.27 and Theo-rem 8.13(iii) to complete the proof.Example 8.29. The zero ânormâ f(x) = â¥xâ¥0 is variationally convex atevery x â Rn for all v â âf(x), owing to Theorem 8.24 and the fact thatf(x) =âni=1 fi(xi), where (â1 â¤ i â¤ n) the function fi : Râ R defined byfi(xi) ={1, if xi Ì¸= 0;0, if xi = 0.is variationally convex at every xi â R for all vi â âfi(xi); see Example 8.9.Example 8.30. Let A â RmÃn and b â Rm. Then the function f : Rn â Rdefined byf(x) = 12 â¥Axâ bâ¥2 + â¥xâ¥0is variationally convex at every stationary point x¯ for 0 â âf(x¯), and there-fore x¯ is in fact a local minimizer in view of Proposition 8.7. Indeed, thefunction g(x) = 12 â¥Axâ bâ¥2 is convex and C1. By Example 8.29, â¥Â·â¥0 isvariationally convex at x¯ for ââg(x¯) â â â¥Â·â¥0 (x¯), to which Theorem 8.23entails the claimed property of f .8.7 Algorithmic applications8.7.1 KrasnoselâskiËÄ±-Mann iterationWe first provide an application of Theorem 8.19, where the celebrated Kras-noselâskiËÄ±-Mann iteration [11, Theorem 5.14] kicks in locally on a nonconvexbut variationally convex problem. To provide proper context, let us recallthe KrasnoselâskiËÄ±-Mann iteration below.Fact 8.31. Let D be a nonempty closed convex set, let T : D â D be anonexpansive operator such that FixT Ì¸= â . Let (µk)kâN be a sequence in[â1, 1] such that ââk=1 µk(1â µk) =â and let x0 â D. Set(âk â N) xk+1 = µkTxk + (1â µk)xk.Then the following hold:1758.7. ALGORITHMIC APPLICATIONS(i) The sequence (xk) is Feje´r monotone with respect to FixT and Txk âxk â 0 as k ââ.(ii) xk converges to xË â FixT .Proof. See [11, Theorem 5.14].Remark 8.32. Fact 8.31 and its variations underpin the convergence ofvarious splitting algorithms in convex optimization, including the Douglas-Rachford algorithm and proximal gradient algorithm, just to name a few.We refer readers to [11] for a detailed exposition.Here is the main result of this section.Corollary 8.33. Let f : Rn â R be proper, lsc, and prox-bounded withthreshold λf > 0. Suppose that f is variationally convex at x¯ for 0 â âf(x¯)and that f is prox-regular around x¯.Let λ > 0 and Uλ be those described in Theorem 8.19. Take x0 suffi-ciently close to x¯, that is, x0 belonging to a closed ball C â Uλ centered atx¯ on which f is prox-regular. Definexk+1 = µkPλf(xk) + (1â µk)xk,where (µk)kâN â [0, 1] andââk=1 µk(1 â µk) = â. Then FixPλf â© C Ì¸= â and the following hold:(i) The sequence (xk)kâN stays in C and is Feje´r monotone with respectto FixPλf â© C. Moreover, PÎ»f(xk)â xk â 0 as k ââ.(ii) xk â xË â C for a local minimizer xË of f .Proof. The assumption on variational convexity implies that f is prox-regular at x¯ for 0 â âf(x¯), thus x¯ â Pλf(x¯) and moreover x¯ â FixPλf â© C.(i) Invoke Theorem 8.19 to see that Pλf is nonexpansive on C, in which caseit is routine to verify the claim. Here we provide full detail following [11,Theorem 5.14] for the sake of completeness. Pick x â FixPλf â© C. Thenfor every k â N,â¥xk+1 â xâ¥2 = â¥µk(PÎ»f(xk)â Pλf(x)) + (1â µk)(xk â x)â¥2= µk â¥PÎ»f(xk)â PÎ»f(x)â¥2 + (1â µk) â¥xk â xâ¥2 â µk(1â µk) â¥PÎ»f(xk)â xkâ¥2â¤ â¥xk â xâ¥2 â µk(1â µk) â¥PÎ»f(xk)â xkâ¥2 ,1768.7. ALGORITHMIC APPLICATIONSwhich justifies the Feje´r monotonicity and entailsââk=1µk(1â µk) â¥PÎ»f(xk)â xkâ¥2 â¤ â¥x0 â xâ¥2 <â.So we must have lim infkââ â¥PÎ»f(xk)â xkâ¥ = 0, otherwise the assumptionon µk would enforceââk=1 µk(1 â µk) â¥PÎ»f(xk)â xkâ¥2 = â. NeverthelessPλf is nonexpansive on C, which contains all (xk)kâN, soâ¥PÎ»f(xk+1)â xk+1â¥= â¥PÎ»f(xk+1)â Pλf(xk) + (1â µk)(PÎ»f(xk)â xk)â¥â¤â¥xk+1 â xkâ¥+ (1â µk) â¥PÎ»f(xk)â xkâ¥ = â¥PÎ»f(xk)â xkâ¥ ,implying that â¥PÎ»f(xk)â xkâ¥ is in fact convergent. Thereby PÎ»f(xk)âxk â0.(ii) The behavior of (xk)kâN established in (i) ensures that all cluster pointsof (xk)kâN lie in FixPλf â© C, entailing that xk â xË â FixPλf â© C; seefor instance [11, Theorem 5.5]. Hence 0 â âf(xË). Shrinking C if necessary,assume without loss of generality f is prox-regular at xË for 0 â âf(xË). ThenTheorem 8.19 is applicable at xË and in turn the convexity of eλf on C yieldsthat f is variationally convex at xË for 0 â âf(xË), and moreover the defini-tion of variational convexity (recall Definition 8.3) entails xË to be a localminimizer of f .8.7.2 Proximal gradient algorithmConsider the structured minimization problemmin {f(x) + g(x) : x â Rn} , (8.37)where f : Rn â R is a proper, lsc, and prox-bounded function; and g :Rn â R is L-Lipschitz smooth for L > 0. The proximal gradient operatorfor λ > 0 is defined byTλ = PÎ»f(IdâÎ»âg). (8.38)The proximal gradient algorithm generates a sequence by iterating Txk+1 â Tλ(xk), âk â N,where x0 â Rn is an arbitrary initial point.When both f, g are convex, the proximal gradient algorithm behavesnicely and it has been investigated extensively in the literature. We referreaders to [11,14] for comprehensive exposition on the topic.1778.7. ALGORITHMIC APPLICATIONSFact 8.34. Let f : Rn â R be proper, lsc, and convex, and let g : Rn â Rbe a convex function whose gradient âg is Lipschitz continuous. Then thefollowing hold for 0 < λ â¤ 1/L:(i) The proximal gradient operator Tλ is 2/3-averaged.(ii) Suppose that argmin(f + g) Ì¸= â and define for an arbitrary x0 â Rnxk+1 = Tλ(xk).Then xk â xâ â argmin f + g.Proof. This is [11, Corollary 27.9].In this section we uncover new properties of the proximal gradient algo-rithm under variational convexity, starting from the following a new inter-pretation of fixed points of the proximal gradient operator Tλ through thelens of level proximal subdifferential.Lemma 8.35. Let f : Rn â R be a proper, lsc, and prox-bounded function;and g : Rn â R be L-Lipschitz smooth for L > 0. With Tλ given by (8.38),we haveFixTλ ={x â Rnâ£â£â£0 â âg(x) + âÎ»p f(x)} â {x â Rn â£â£â£0 â âÎ»Ëp (f + g)(x)} ,where Î»Ë = 1Lâ1+Î»â1 > 0.Proof. Indeed, x â Tλx â x â Pλf(x â Î»âg(x)) â x â Î»âg(x) â(Id+Î»âÎ»p f)(x)â 0 â Î»âg(x)+Î»âÎ»p f(x)â 0 â âg(x)+âÎ»p f(x). To proceed,note that the Lipschitz smoothness on g (recall Theorem 2.68(ii)) impliesthat(ây â Rn) g(y) â¥ g(x) + â¨âg(x), y â xâ© â L2 â¥xâ yâ¥2 . (8.39)Moreover, by the definition of level proximal subdifferential, 0 â âg(x) +âÎ»p f(x) amounts to(ây â Rn) f(y) â¥ f(x)â â¨âg(x), y â xâ© â 12λ â¥xâ yâ¥2 . (8.40)Adding (8.39) and (8.40) yields 0 â âÎ»Ëp (f + g)(x).Lemma 8.35 shows that the proximal gradient method finds precisely xsuch that 0 â âg(x)+âÎ»p f(x). We call such an x as a λ-level p-critical pointof f + g.1788.7. ALGORITHMIC APPLICATIONSLemma 8.36. Let D1, D2 be nonempty subsets of Rn, let T1 : D1 â D2be α1-averaged, and let T2 : D2 â Rn be α2-averaged, where αi â]0, 1[ fori = 1, 2. SetT := T2T1 and α :=Î±1+Î±2â2Î±1Î±21âÎ±1Î±2 .Then α â (0, 1), and T is α-averaged on D1.Proof. See [11, Proposition 4.44]Utilizing Theorems 8.19 and 8.22, we propose a localized proximal gra-dient algorithm, where pleasant behavior of proximal gradient described inFact 8.34 occur locally.Theorem 8.37 (proximal gradient algorithm under variational convexity).Suppose that f : Rn â R is proper, lsc, and prox-bounded, and that g : Rn âR is convex, differentiable, and has a Lipschitz continuous gradient. Let f bevariationally convex at x¯ for ââg(x¯) â âf(x¯). Define Tλ := PÎ»f(IdâÎ»âg).Then x¯ is a local minimizer of f + g, and âÎ»¯ > 0 such that(i) âÎ» â (0, λ¯) there exists a neighborhood Uλ of x¯ on which Tλ : Uλ â Uλ,Tλ is 2/3-averaged on Uλ, and Tλ(x¯) = x¯.(ii) Let x1 â Uλ. Define the sequence (xk)kâN by xk+1 = Txk. Then(xk)kâN converges to xâ â Uλ, a λ-level p-critical point of f + g.(iii) If, in addition, f is variationally strongly convex at x¯ for ââg(x¯) ââf(x¯), then, by choosing a smaller neighborhood Uλ, the sequence givenin (ii) converges linearly to x¯.Proof. Since f is variationally convex at ââg(x¯) â âf(x¯), Theorem 8.23assures that f + g is variationally convex at x¯ for 0 â â(f + g)(x¯). It followsfrom Proposition 8.7 that x¯ is a local minimizer of f + g.(i) By Theorem 8.19, there exists λ¯ such that for 0 < λ < λ¯ the mappingPλf is firmly nonexpansive on a neighborhood of x¯ â Î»âg(x¯), say B(x¯ âÎ»âg(x¯); δ); and Tλ(x¯) = PÎ»f(x¯âÎ»âg(x¯)) = x¯. Because âg is Lipschitz, bychoosing a smaller λ¯, we can make IdâÎ»âg firmly nonexpansive by usingCorollary 2.67 and Fact 2.62. For x â Uλ := B(x¯; δ), we haveâ¥(xâ Î»âg(x))â (x¯â Î»âg(x¯))â¥ â¤ â¥xâ x¯â¥ â¤ δ,so thatâ¥Txâ x¯â¥ = â¥PÎ»f(xâ Î»âg(x))â PÎ»(x¯â Î»âg(x¯)â¥â¤ â¥(xâ Î»âg(x))â (x¯â Î»âg(x¯)â¥ â¤ δ,1798.7. ALGORITHMIC APPLICATIONSbecause IdâÎ»âg is nonexpansive on Rn, and Pλf is nonexpansive on B(x¯âÎ»âg(x¯); δ). Thus, Tλ : Uλ â Uλ. Being a composition of firmly nonexpan-sive (1/2-averaged) mappings, Tλ is 2/3-averaged on Uλ by Lemma 8.36.(ii) Applying [11, Proposition 5.16] to Tλ on Uλ gives that xk â xâ â Uλand xâ = Txâ. It suffices to apply Lemma 8.35.(iii) By Theorem 8.22(iv) âÎ»¯ such that for 0 < λ < λ¯ the mapping Pλf is aBanach contraction on a neighborhood of x¯âÎ»âg(x¯), say B(x¯âÎ»âg(x¯); δ);and and Tλ(x¯) = PÎ»f(x¯âÎ»âg(x¯)) = x¯. Becauseâg is Lipschitz, by choosinga smaller λ¯, we can make IdâÎ»âg firmly nonexpansive on Rn. Then Tλ :Uλ â Uλ is a Banach contraction with Uλ := B(x¯; δ). It follows from theBanach contraction principle [11, Theorem 1.50] that for every x1 â Uλ thesequence (xk)kâN must converges linearly to x¯.Local optimality can be guaranteed if suitable assumption on prox-regularity is in place.Corollary 8.38. Suppose that f : Rn â R is proper, lsc, and prox-bounded,and that g : Rn â R is convex, differentiable, and has a Lipschitz continuousgradient. Let f be variationally convex at x¯ for ââg(x¯) â âf(x¯) and f beprox-regular on a neighborhood of x¯. Define T := PÎ»f(IdâÎ»âg). ThenâÎ»¯ > 0 such that âÎ» â (0, λ¯) there exists a neighborhood Uλ of x¯ on whichfor every x1 â Uλ, the sequence (xk)kâN given by xk+1 = Txk converges toxâ â Uλ, a local minimizer of f + g.Proof. Observe that f+g is prox-bounded because of f being prox-boundedand g being convex. By assumption, we can choose a neighborhood U ofx¯ on which f is prox-regular. Because g is C1+, f + g is prox-regular onU ; see, e.g., [81, Exercise 13.35]. Theorem 8.37 shows that âÎ»¯ such thatâÎ» â (0, λ¯) there exists a neighborhood Uλ of x¯ such that Uλ â U and thatfor every x1 â Uλ, the sequence (xk)kâN given by xk+1 = Txk converges toxâ â Uλ such that ââg(xâ) â âÎ»p f(xâ). Since f + g is variationally convexat x¯ for 0 â â(f + g)(x¯), by Theorem 8.19, using a smaller λ¯ we can alsomake eλ(f + g) convex on Uλ. Now for xâ â Uλ, we have that f + g isprox-regular at xâ for 0 â â(f + g)(xâ) and that eλ(f + g) is convex on Uλ.Using Theorem 8.19(i)â (vii), we deduce that f + g is variationally convexat xâ for 0 â â(f + g)(xâ). Then xâ is a local minimizer for f + g in view ofProposition 8.7.Our results so far have revealed that variationally convex problems re-duce to convex ones locally, but how do we get close enough?1808.7. ALGORITHMIC APPLICATIONSTheorem 8.39. Suppose that f : Rn â R is proper, lsc, and prox-bounded,and that g : Rn â R is convex and L-Lipschitz smooth. Define Tλ :=PÎ»f(IdâÎ»âg) for 0 < λ < 1/L. Pick an arbitrary x0 â Rn and definexk+1 â T (xk) for k â N. Then the following hold:(i) If (xk) is bounded and f + g satisfies the KL property, then xk â xâfor a stationary point xâ of f + g.(ii) If in addition that f is variationally convex at xâ for ââg(xâ) ââf(xâ), then xâ is a local minimizer of f + g.Proof.(i) This is [4, Theorem 5.1].(ii) Apply Theorem 8.23 and Proposition 8.7 to see that xâ is a local mini-mizer.Remark 8.40. As a posterior result, we note that there exists λ¯ > 0 suchthat Tλ to be 2/3-averaged around xâ for 0 < λ < λ¯; see Theorem 8.37.Next we consider the zero-norm regularized least squares problem as aconcrete example. Amazingly, owing to the theory developed earlier, proxi-mal gradient method converges globally to local minimizers when being ap-plied to this problem!Example 8.41. Let A â RmÃn and b â Rm. Choose γ > 0 and considerminxâRnf(x) + g(x), (8.41)where f(x) = Î³â¥xâ¥0, g(x) = 12â¥Ax â bâ¥2. Denote the one-dimensional zeronorm by | · |0. Then(âu â R) Pλγ| · |0(u) =ï£±ï£´ï£²ï£´ï£³u, if |u| â¥ â2Î³Î»;{0, u}, if |u| = â2Î³Î»;0 otherwise,and therefore (âu â Rn) PÎ»Î³â¥ · â¥0(u) = (Pλγ| · |0(u1), · · · , Pλγ| · |0(un)). Foran arbitrary x0 â Rn, the proximal gradient algorithm reads as(âk â N) xk+1 â Tλxk := PÎ»Î³â¥ · â¥0(xk â Î»(AâºAxk âAâºb)),where 0 < λ < 1/â¥Aâ¥2F , square of the Frobenius norm of A. If (xk)kâN isbounded, then xk â xâ for a local minimizer xâ of f + g.1818.8. CONCLUSIONProof. The function f + g is a KL function in view of Example 2.82 andFact 2.84, and f is variationally convex everywhere owing to Example 8.29.Moreover, g is L-Lipschitz smooth with L = â¥Aâ¥2F . Thereby all conditionsof Theorem 8.39 are satisfied, which yields the claimed result.8.8 ConclusionWe characterized the variational and variational strong convexity of a func-tion. These properties are equivalent to various degrees of relative mono-tonicity of the level proximal subdifferential and to the local nonexpansive-ness of the associated proximal operator. As applications, we showed thatthe famous proximal gradient and the KrasnoselâskiËÄ±-Mann methods achievelocal optimality in the presence of variational convexity. Calculus rules ofthe variational convexity are investigated to facilitate verifying and/or con-structing this class of functions efficiently.182Chapter 9The Bregman level proximalsubdifferential9.1 IntroductionIt is well-known that the proximal operator is the resolvent of the Mor-dukhovich (limiting) subdifferential operator âf : Rn â Rn of f when f isconvex, that is,Pλf = (Id+Î»âf)â1 , (9.1)leading to beautiful correspondences among the properties of the proximaloperator Pλf , the subdifferential âf and the function f . For instance, prox-imal operators of convex functions are resolvents of the associated subdiffer-ential operators and therefore firmly nonexpansiveâa fundamental propertythat underpins the convergence of various first-order methods in convex op-timization; see [11].Assume now that Ï : Rn â R is proper, lsc, convex, and differentiableon int domÏ Ì¸= â . The Bregman distance induced by Ï is DÏ : RnÃRn â Rdefined byDÏ(x, y) ={Ï(x)â Ï(y)â â¨âÏ(y), xâ yâ© if y â int domÏ;â otherwise. (9.2)Since the Bregman distance is typically asymmetric, it gives rise to the leftand the right Bregman proximal operators âââproxÏÎ»f : int domÏâ domÏ andâââproxÏÎ»f : domÏâ int domÏ defined respectively by(ây¯ â int domÏ) âââproxÏÎ»f(y¯) = argminxâdomÏ{f(x) + 1λ DÏ(x, y¯)},(âx¯ â domÏ) âââproxÏÎ»f(x¯) = argminyâint domÏ{f(y) + 1λ DÏ(x¯, y)}.In particular, both âââproxÏÎ»f and âââproxÏÎ»f become Pλf when the kernel Ï =12 â¥Â·â¥2. The Bregman proximal operators are of central importance in varia-1839.1. INTRODUCTIONtional analysis and optimization; see for instance [1,7,9,10,15,23,32,45,87,91,96] and the references therein. Below we highlight a few lines of work:â¢ The Chebyshev problem in the sense of the Bregman distance [13].â¢ Bregman iterative methods [9, 10,13].â¢ Mirror descent algorithms for solving constrained optimization prob-lems [15,65].â¢ Optimization problems beyond the traditional Lipschitz smoothnessassumption [1, 7, 23,57,96].It is thus paramount to understand the properties of the Bregman proximaloperators. Significant progress has been made in [12, 13, 45, 49]. However,none of these works have addressed a fundamental question:What is the relation between âââproxÏÎ»f and a subdifferential operator? (9.3)When f is convex, the answer is clear, owing to the following generalizationof (9.1):âââproxÏÎ»f = (âÏ+ Î»âf)â1 â¦ âÏ. (9.4)Such representation fails in the absence of convexity of f . Nevertheless, akinto the Euclidean case, answering (9.3) in full generality shall shed lights onhow properties of the Bregman proximal operators correspond to those ofthe underlying function, and therefore providing deep insights into theseoperators and their applications in numerical optimization.The goal of this chapter is to resolve the fundamental question (9.3)completely by introducing the left and right Bregman level proximal subdif-ferentials and investigating them systematically. Amazingly, their resolventsalways coincide with the Bregman proximal operators, regardless of convex-ity, providing new insights into the Bregman proximal operators. Mainresults of this chapter are summarized below.R1 Theorem 9.29 presents various relations between Bregman proximaloperators and the Bregman level proximal subdifferentials, resolvingthe question (9.3) completely.R2 We characterize the (left) Bregman level proximal subdifferentiabilityin Theorem 9.37.1849.1. INTRODUCTIONR3 In Theorem 9.48, we establish a correspondence between the maximalmonotonicity of the Bregman proximal operator and properties of theBregman level proximal subdifferential. See Fig. 9.1.Euclidean setting (Fact 9.45)f + Î»â1j is convexâPÎ»f is maximally monotoneâPÎ»f is convex-valuedâPÎ»f = (Id+Î»âf)â1Bregman setting (Theorem 9.48)f + Î»â1Ï is convexââââproxÏÎ»f â¦ âÏâ is maximally monotoneââââproxÏÎ»f is convex-valuedââââproxÏÎ»f = (âÏ+ Î»âf)â1 â¦ âÏÏ = jFigure 9.1: A correspondence between various notions in Euclidean andBregman settings in Theorem 9.48.R4 The convexity of the (left) Bregman Moreau envelope and that ofthe underlying function are characterized in Theorems 9.54 and 9.58,respectively. See Fig. 9.2 for an illustration.Euclidean setting (Fact 9.50)f is convexââf is maximally monotoneâPÎ»f is firmly nonexpansiveBregman setting (Theorem 9.54)f is convexââââÏ/Î»p f is maximally monotoneââââproxÏÎ»f â¦ âÏâ is âÏ-firmly nonexpansiveEuclidean setting (Fact 9.50)eλf is convexââeÎ»f is cocoerciveBregman setting (Theorem 9.58)ââenvÏÎ»f â¦ âÏâ is convexââ(ââenvÏÎ»f â¦ âÏâ) is Bregman cocoerciveÏ = jExamples 9.59 and 9.60Ï = jFigure 9.2: A correspondence among various notions in Euclidean and Breg-man settings established in Section 9.6.3.1859.2. PRELIMINARIESThe chapter is organized as the following. We start from introducing thenecessary preliminaries in Sections 9.2 and 9.3. In Section 9.4, we definethe Bregman level proximal subdifferential and investigate its properties,followed by its pointwise and global characterizations in Sections 9.5 and 9.6respectively. Finally, we provided in Section 9.7 an interesting interpretationof R2 in view of a generalized conjugacy in optimal transport.9.2 PreliminariesMotivated by our need to work with extended-real-valued functions definedon X â Rn and their induced operators, rather than those defined on thewhole space Rn, we adapt some convexity and variational analysis notionsalready introduced in Chapter 2 for functions f : X â R and set-valuedoperators T : X â Y , whereX,Y â Rn are assumed nonempty and convex.3Set-valued operators The notation T : X â Y indicates a set-valuedoperator T mapping points x â X to sets T (x) â Y . The graph of T isthe set gphT = {(x, y) â X Ã Y |y â T (x)}, while its (effective) domain isdomT = {x â X |T (x) Ì¸= â } and its range is ranT = âxâX T (x). T is saidto be outer semicontinuous (osc) if gphT is closed relative to X ÃY (in thesense that gphT = C â© (X Ã Y ), where C is a closed subset of Rn Ã Rn).Equivalently, T is osc if(âx â X){y â Y â£â£â (xk, yk)kâN â (x, y) with (xk, yk) â gphT âk} â T (x).T is locally bounded if every x¯ â X admits a neighborhood Nx¯ such thatâxâNx¯â©XT (x) is bounded.The inverse of T is defined as the set-valued mapping Tâ1 : Y â X givenby Tâ1(y) = {x â X |y â T (x)} for every y â Y .Extended-real valued functions The (effective) domain of a functionf : X â R is the set dom f = {x â X |f(x) <â}, and f is said to be properif f Ì¸â¡ â and f > ââ. The epigraph of f isepi f = {(x, α) â X Ã R |f(x) â¤ α} ,3Many definitions given here do not require convexity of either sets X and Y , but thisrestriction suffices to the purposes of the chapter.1869.2. PRELIMINARIESand f is said to be lower semicontinuous (lsc) if f(x) = lim infXâxâ²âx f(xâ²)holds for every x â X. Equivalently, f is lsc if epi f is closed relative toX ÃR. The lower semicontinuous hull of f is the lsc function cl f : X â Rdefined as the pointwise supremum among all lsc functionsX â Rmajorizedby f , pointwise given bycl f(x) = lim infXâxâ²âxf(xâ²).Clearly, f is lsc iff f = cl f . We say that f is coercive (or level bounded) iflim infâ¥xâ¥ââ f(x) =â, and 1-coercive if lim infâ¥xâ¥ââ f(x)/ â¥xâ¥ =â (theselimits being â by default if X is bounded). The Fre´chet subdifferential off is âÌf : X â Rn given byâÌf(x) ={v â Rnâ£â£â£â£lim infXâzâxz Ì¸=xf(z)âf(x)ââ¨v,zâxâ©â¥zâxâ¥ â¥ 0},and we call an element v â âÌf(x) a Fre´chet subgradient of f at x. Thelimiting subdifferential of f is âf : X â Rn given byâf(x) = Lim supXâxâ²âxâÌf(xâ²).Convex analysis The Fenchel (convex) subdifferential of f is âFf : X âRn, whereâFf(x) ={u â Rn â£â£f(xâ²) â¥ f(x) + â¨u, xâ² â xâ© âxâ² â X } .In particular,x¯ â argmin f â 0 â âFf(x¯). (9.5)The convex hull of f , denoted conv f , is the pointwise supremum among allconvex functions X â R majorized by f , while the supremum of all suchfunctions which are additionally lsc is the closed convex hull, denoted conv f .In particular, one always has thatconv f â¤ conv f â¤ f and conv f â¤ cl f â¤ f. (9.6)The convex hull of a set E â Rn, also denoted convE, is the intersectionof all convex subsets of Rn containing E. The notation riE indicates therelative interior of E, namely the interior of E relative to its affine hull.We lastly remind the definition of the Fenchel conjugate of a functionh : Rn â R, as the convex and lsc function hâ : Rn â R given by hâ(y) =supxâRn {â¨x, yâ© â h(x)}, and that of its Fenchel biconjugate hââ = (hâ)â. Wealso remind that [81, Proposition 11.1], for proper h : Rn â R with an affineminorant,conv h = hââ. (9.7)1879.2. PRELIMINARIES9.2.1 Canonical extensions to the whole space RnThere is a natural way to extend set-valued operators X â Y and extended-real-valued functions X â R to the entire space Rn while maintaining theirfundamental properties.Definition 9.1 (canonical extension of a set-valued operator). The canon-ical extension of T : X â Y is TË : Rn â Rn given byTË (x) ={T (x) if x â X;â otherwise.Remark 9.2.(i) Operators T and TË have same graph, (effective) domain, and range,and moreover one is locally bounded iff the other is.(ii) An important difference between T and TË , however, is that TË may failto be osc even when T is. To see this, consider T : (0, 1) â R whereT (x) = {0} for all x â (0, 1). Clearly, on the other hand, when TË isosc then so is T .A similar extension can be defined for extended-real-valued functions.Definition 9.3 (canonical extension of an extended-real-valued function).The canonical extension of f : X â R is fË : Rn â R given byfË(x) ={f(x) if x â X;â otherwise.Remark 9.4.(i) Functions f and fË share epigraph, effective domain, and whether theyare proper, convex.(ii) A primary difference among the f and fË lies in lower semicontinuity.Consider f : (0, 1) â R where f(x) = 0 for all x â (0, 1). Then f islsc while the extension fË is not.(iii) Moreover, note that âFfË , âÌfË , and âfË are the respective canonicalextensions of âFf , âÌf , and âf , in the operator sense of Definition 9.1.Functions f and fË also share the convex hull, but not the closed convexhull owing to the previous item.1889.2. PRELIMINARIESThe following lemma nevertheless clarifies that the hulls of f and itscanonical extension fË coincide on the set X. This simple yet importantobservation allows us to apply known results, originally formulated for func-tions defined on the entire space Rn, within our specific setting.Lemma 9.5. Let f : X â R be proper and let fË : Rn â R be its canonicalextension as in Definition 9.3. Then, the following hold:(i) cl f = (cl fË)â£â£X. In particular, f is lsc iff there exists an lsc functionfË : Rn â R such that f = fË â£â£X.(ii) fËâââ£â£X= (conv fË)â£â£X= conv f â¤ conv f = (conv fË)â£â£X. The inequalityholds as equality whenever f is 1-coercive.(iii) f is convex iff fË is convex. In this case, one has that conv f = cl f .Proof.(i) For x â X one has thatcl fË(x) = lim infxâ²âxfË(xâ²) = lim infXâxâ²âxfË(xâ²) = lim infxâ²âxf(xâ²) = cl f(x),where the second identity follows from the fact that dom fË â X, and thethird one from the fact that fËâ£â£X= f . The claim follows from the arbitrarityof x â X.(ii) The inequality was shown in (9.6); the first identity follows from (9.7)and the last one is obvious. We now show that conv f = (conv fË)â£â£X. Observethat, as a function X â R, (conv fË)â£â£Xis convex and lsc, and (conv fË)â£â£Xâ¤fËâ£â£X= f by definition. As such, (conv fË)â£â£Xâ¤ conv f . To show the converseinequality, let g : X â R be convex, lsc and such that g â¤ f , and letgË : Rn â R be its canonical extension. It follows from assertion (i) thatg = (cl gË)â£â£X. Moreover, since gË is convex (on Rn), cl gË is convex and lscby [11, Corollary 9.10], and globally supports fË . Consequently conv fË â¥ cl gË,resulting inconv fËâ£â£Xâ¥ cl gËâ£â£X= g.Taking the supremum among all such functions g yields the sought inequalityconv fËâ£â£Xâ¥ conv f . When f is 1-coercive, then so is fË . By invoking Fact 2.47we conclude that conv fË = conv fË in this case, and in particular equalityholds on X.1899.2. PRELIMINARIES(iii) The equivalence of convexity of f and of fË is straightforward. Next,suppose that f is convex. The claimed identity is true when X = Rn,see [11, Corollary 9.10], and in particular conv fË = cl fË . Combined with theprevious assertions the claim follows.Fact 9.6. Let f : X â R be proper and x¯ â X be fixed. Consider thefollowing statements:(i) âFf(x¯) Ì¸= â ;(ii) f(x¯) = conv f(x¯) â R;(iii) âFf(x¯) = â(conv f)(x¯).One has (i) â (ii) â (iii).Proof. See [41, Proposition 1.4.3]. Although the cited result is stated forfunctions Rn â R, the validity of Fact 9.6 for f : X â R is guaranteedby Lemma 9.5, because for every x¯ â X one can replace f : X â R by itscanonical extension fË .By combining Fact 9.6 with the inequalities in (9.6) it is apparent that,for every f : X â R proper and x â X, one has thatâFf(x) â âF[cl f ](x) â âF[conv f ](x). (9.8)Lemma 9.7. For every function f : X â R with X â Rn nonempty,one has that inf f = inf cl f = inf conv f and argmin f â argmin cl f âargmin conv f .Proof. That inf conv f â¤ inf cl f â¤ inf f is obvious from (9.6), and equal-ity thus must hold if the latter is ââ. If instead inf f is finite, then theconstant function inf f is convex and lsc and is globally majorized by f ;as such, inf f â¤ conv f and consequently inf f â¤ inf conv f , completing theproof. Finally, the claimed inclusions are apparent from (9.8) and (9.5).9.2.2 Preliminaries on Bregman distancesA function Ï : Rn â R is said to be a distance-generating function (dgf), ifit is proper, lsc, convex, and differentiable on int domÏ Ì¸= â . This definition1909.2. PRELIMINARIESis motivated by the Bregman distance DÏ : Rn Ã Rn â RDÏ(x, y) ={Ï(x)â Ï(y)â â¨âÏ(y), xâ yâ© if y â int domÏ;â otherwise. (9.9)In general, the Bregman distance is asymmetric in the sense that DÏ(x, y) Ì¸=DÏ(y, x). The symmetrized Bregman distance âÏ is defined asâÏ(x, y) = DÏ(x, y) + DÏ(y, x)={â¨âÏ(x)ââÏ(y), xâ yâ© if x, y â int domÏ;â otherwise. (9.10)Additional assumptions on Ï induce favorable properties on DÏ and on re-lated objects, and will be sometimes be needed in our results. Among thesewe mention Legendre functions.Definition 9.8 (Legendre function). A proper lsc convex function Ï : Rn âR is said to be of Legendre type (or simply Legendre) if it is(i) essentially smooth, namely differentiable on int domÏ Ì¸= â and suchthatâ¥â¥âÏ(xk)â¥â¥ââ whenever int domÏ â xk â x â bdry domÏ, and(ii) essentially strictly convex, namely strictly convex on every convex sub-set of dom âÏ.We next recall two fundamental characterizations of essential smoothnessand/or strict convexity.Fact 9.9. For every proper lsc and convex Ï : Rn â R, the following areequivalent:(i) Ï is essentially smooth.(ii) âÏ is (at most) single valued.(iii) Ïâ is essentially strictly convex.Under any of the above equivalent conditions, one also has that dom âÏ =int domÏ and âÏ(x) = {âÏ(x)} for every x â int domÏ.Proof. See [76, Theroems 26.1 and 26.3].1919.3. BREGMAN PROXIMAL MAP AND MOREAU ENVELOPEFact 9.10. A proper lsc convex function Ï : Rn â R is of Legendre typeiff its conjugate Ïâ is. In this situation, one has that âÏ : int domÏ âint domÏâ is a bijection with inverse âÏâ, and the following identity holds:DÏ(x, y) = DÏâ(âÏ(y),âÏ(x)) âx, y â int domÏ.Proof. See [76, Theorem 26.5].The following is a well-known generalization of the parallelogram iden-tity, and whose proof is immediate from the definition (9.9).Fact 9.11 (three-point identity). For dgf Ï : Rn â R, that is Ï is proper,lsc, convex, and differentiable on int domÏ Ì¸= â , x â domÏ and y, z âint domÏ it holds thatDÏ(x, z) = DÏ(x, y) + DÏ(y, z) + â¨xâ y,âÏ(y)ââÏ(z)â©.Proof. See [32, Lemma 3.1].9.3 Bregman proximal map and Moreau envelopeWe will work with a dgf Ï complying with the following minimal require-ments, which will be assumed throughout this chapter without further men-tion. More specific assumptions will be introduced when needed to supportfurther analysis.Assumption 9.12. Function Ï : Rn â R is a distance generating function(dgf), namely, proper, lsc, convex, and differentiable on int domÏ Ì¸= â . Wedenote X = domÏ and Y = int domÏ.The symbols X and Y are introduced for notational conciseness, and willalso be helpful later when highlighting a parallel with the general theory ofΦ-convexity. We begin by introducing well known objects that comprise thebasic toolbox of the chapter. It will be implied without mention that all thesubsequent definitions are relative to the dgf Ï and, when applicable, to aparameter λ > 0.Definition 9.13 (Bregman proximal map). The left (Bregman) proximalmapping of f : X â R is âââproxÏÎ»f : Y â X defined byâââproxÏÎ»f(y¯) = argminxâX{f(x) + 1λ DÏ(x, y¯)}. (9.11a)1929.3. BREGMAN PROXIMAL MAP AND MOREAU ENVELOPESimilarly, the right (Bregman) proximal mapping of g : Y â R is âââproxÏÎ»g :X â Y defined byâââproxÏÎ»g(x¯) = argminyâY{g(y) + 1λ DÏ(x¯, y)}. (9.11b)Definition 9.14 (BregmanâMoreau envelopes). The left (BregmanâMoreau)envelope of f : X â R is ââenvÏÎ»f : Y â [ââ,â] defined byââenvÏÎ»f(y¯) = infxâX{f(x) + 1λ DÏ(x, y¯)}. (9.12a)Similarly, the right (BregmanâMoreau) envelope of g : Y â R is ââenvÏÎ»g :X â [ââ,â] defined byââenvÏÎ»g(x¯) = infyâY{g(y) + 1λ DÏ(x¯, y)}. (9.12b)Remark 9.15. We remark that the definitions of Bregman proximal map-ping also apply to functions f, g : Rn â R defined on all the space Rn.Our explicit emphasis on X and Y is chosen to highlight that âââproxÏÎ»f andâââproxÏÎ»g solely depend on the value of f on X and of g on Y, and will findan independent appeal when framing the discussion under the more generallens of Φ-convexity; see Section 9.7.Definition 9.16 (Bregman proximal hull). The (left) Bregman proximalhull of f : X â R is âââhullÏÎ»f : X â R given byâââhullÏÎ»f = âââenvÏÎ»(âââenvÏÎ»f).9.3.1 Ï-Prox-boundednessWe mention a well-known property which, coupled with 1-coercivity of Ï,constitutes a fundamental requisite for the left proximal mapping and theMoreau envelope to be meaningful objects in practice; see [45, Definition2.3].Definition 9.17 (Ï-prox-boundedness). We say that f : X â R is Ï-prox-bounded if there exists λ > 0 and y â Y such that ââenvÏÎ»f(y) > ââ. Thesupremum λf of all such λ is the Ï-prox-boundedness threshold of f .As formalized in Fact 9.19, the above property is useful in combinationwith 1-coercivity of Ï, as it ensures, among other things, the non-emptinessofâââproxÏÎ»f(x¯) for every sufficiently small λ and x¯ â X . We now turn to provethe properties on Ï needed to justify the statement of Fact 9.19. As we shallsee, the key ingredients in this argument are Ï-prox-boundedness and theconstraint λ < λf .1939.3. BREGMAN PROXIMAL MAP AND MOREAU ENVELOPELemma 9.18. Let f : X â R be proper, lsc, and Ï-prox-bounded. Extendf to fË : Rn â R by setting it to â on Rn X (function fË is proper but notnecessarily lsc). Then,âââproxÏÎ»f(y) =âââproxÏÎ»fË(y) and ââenvÏÎ»f(y) =ââenvÏÎ»fË(y)for every y â Y and λ > 0. Moreover, for every λ â (0, λf ) function fË+Î»â1Ïis lsc (on Rn), and it is also 1-coercive whenever Ï is.Proof. The identities hold trivially, in fact, regardless of properness and/orlower semicontinuity of f , since DÏ(x, y) =â whenever x /â X .We next prove the remaining claims concerning fË + Î»â1Ï. To this end,let λ0 â (λ, λf ) be fixed, and let y0 â Y be such thatââenvÏÎ»0f(y0) > ââ; notethat properness of f : X â R ensures that ââenvÏÎ»0f(y0) â R. To streamlinethe notation, let us denote ÏÎ»,y¯ = fË +1λ DÏ( · , y¯). Since ÏÎ»,y¯ and fË + Î»â1Ïdiffer by a linear term, it suffices to show the claims for ÏÎ»,y¯. For everyx â Rn it holds thatÏÎ»,y¯(x) = fË(x) +1Î»0DÏ(x, y0) +(1λ â 1Î»0)DÏ(x, y0) + 1λ DÏ(y0, y¯)+ 1Î»â¨xâ y0,âÏ(y0)ââÏ(y¯)â©â¥ ââenvÏÎ»0f(y0) + 1λ DÏ(y0, y¯) +(1λ â 1Î»0)DÏ(x, y0)+ 1Î»â¨xâ y0,âÏ(y0)ââÏ(y¯)â©=(1Î» â 1Î»0)Ï(x) + â¨v, xâ©+ c, (9.13)where c = ââenvÏÎ»0f(y0) + 1Î»Ïâ(âÏ(y¯)) â 1Î»0Ïâ(âÏ(y0)) and v = 1Î»0âÏ(y0) â1Î»âÏ(y¯). From this expression it is clear that whenever Ï is 1-coercive, thenso is ÏÎ»,y¯.Regardless of this assumption, we move on to show lower semicontinu-ity. To this end, let α â R and (xk)kâN be a sequence converging to apoint x¯ and such that ÏÎ»,y¯(xk) â¤ α for all k. To arrive to a contradiction,suppose that ÏÎ»,y¯(x¯) > α. Since ÏÎ»,y¯ is lsc on X by assumption, one musthave that x¯ /â X , that is, Ï(x¯) = â, otherwise ÏÎ»,y¯(xk) â¤ α would im-ply ÏÎ»,y¯(x¯) â¤ α. On the other hand, for every k it follows from (9.13) that( 1Î»â 1λ0 )Ï(xk)+â¨v, xkâ©+c â¤ α. The fact that ( 1Î»â 1λ0 )Ï(x¯)+â¨v, x¯â©+c =â > αcontradicts the fact that Ï is lsc (on Rn).Fact 9.19. Suppose that Ï is 1-coercive, and that f : X â R is proper, lsc,and Ï-prox-bounded. Then, for every λ â (0, λf ) the following hold:1949.3. BREGMAN PROXIMAL MAP AND MOREAU ENVELOPE(i) domââenvÏÎ»f = domâââproxÏÎ»f = Y.(ii) ââenvÏÎ»f : Y â R is continuous.(iii) âââproxÏÎ»f : Y â Rn is compact-valued and outer semicontinuous.Proof. Patterning [45, Theorems 2.2 and 2.6, Corollary 2.2], we shall proveonly (i), which is necessary since the cited results assume lsc f : Rn â R.The rest of the statements follow from an identical argument as in [45,Theorems 2.2 and 2.6, Corollary 2.2], thus omitted to avoid repetition ofknown arguments.Let fË : Rn â R be fË(x) = f(x) for x â X and fË(x) = â otherwise.Then Lemma 9.18 ensures thatfË(x) + Î»â1Ï(x) is lsc and 1-coercive. (9.14)DefineÏ(x, y, λ) = fË(x) + Î»â1DÏ(x, y),which is lsc and 1-coercive in x for every y â Y and 0 < λ < λf owing to(9.14). For every y â Y, Ï(x, y, λ) is coercive in x. HenceâââproxÏÎ»fË(y) = argminxâXÏ(x, y, λ) Ì¸= â and ââenvÏÎ»fË(y) = infxâdomÏÏ(x, y, λ) = Ï(x¯, y, λ)for some x¯ â âââproxÏÎ»f(y), which is finite. So domâââproxÏÎ»fË = domââenvÏÎ»fË = Y.Apply Lemma 9.18 again to see that fË can be replaced by f .The following example illustrates the advantages of working with theunconventional f : X â R.Example 9.20. Let Ï : R â R be defined as Ï(x) = â ln(x) for x â X =(0,â) and Ï(x) =â for x â¤ 0. Then, f : X â R given by f(x) = ln(x) isproper, lsc, and Ï-prox-bounded with λf = 1. However, f : X â R cannotbe extended outside X while being proper and lsc, to which results in theseminal paper [45] are not applicable. Although f does not admit a properand lsc extension on R, âââproxÏÎ»f and ââenvÏÎ»f still enjoy the properties statedin Fact 9.19 for every λ â (0, λf ).1959.3. BREGMAN PROXIMAL MAP AND MOREAU ENVELOPELastly, it is important to remark thatââenvÏÎ»f andâââproxÏÎ»f are, respectively,lower and outer semicontinuous only relative to Y. Unless Ï has full domain,the same property is not guaranteed if viewed as objects defined on Rn, andthis comment holds regardless of whether or not f admits lsc extensionson the whole space. In this respect, we point out the imprecise statementsof [45, Theorems 2.2(iii) and 2.6] in which such restrictions are not accountedfor.Example 9.21. Consider a dgf Ï with domÏ â Rn and let f â¡ 0. Viewedas an operator Rn â Rn, one has that âââproxÏÎ»f(y) = {y} if y â Y and isempty otherwise. Since Y = int domÏ is not closed (in Rn), this operatoris not osc in the sense of Definition 2.7. Similarly, ââenvÏÎ»f reduces to theindicator function of set Y, which is continuous on Y, but not lsc on Rn inthe sense of Definition 2.3.9.3.2 Alternative expressionsThe following provides a convenient way of expressing the Bregman proximalmap and Moreau envelope both in terms of convex conjugates and with thecorresponding Euclidean counterparts defined on the entire space Rn.Lemma 9.22. Given f : X â R, extend it to fË : Rn â R by setting it toâ on Rn X . Then, for every λ > 0 the following hold:(i) Î»ââenvÏÎ»f =[Ïââ (Î»fË + Ï)â] â¦âÏ. In particular, when Ï is of Legendretype, Î»ââenvÏÎ»f â¦ âÏâ =[Ïâ â (Î»fË + Ï)â]â£â£int domÏâ.(ii) It holds thatÎ»âââhullÏÎ»f =[((Î»fË + Ï)â + Î´ranâÏ)â]â£â£X â Ï â¤ conv(λf + Ï)â Ïâ¤ conv(λf + Ï)â Ï â¤ λf.When Ï is essentially smooth and 1-coercive, the first inequality holdsas equality, and so does the second one when in addition f is Ï-proxbounded and λ < λf .Proof.1969.3. BREGMAN PROXIMAL MAP AND MOREAU ENVELOPE(i) By direct computation, for every y â Y one hasÎ»ââenvÏÎ»f(y) = infxâX {λf(x) + DÏ(x, y)}= infxâRn{(Î»fË + Ï)(x)â â¨âÏ(y), xâ©}â Ï(y) + â¨âÏ(y), yâ©= â supxâRn{â¨âÏ(y), xâ© â (Î»fË + Ï)(x)}+ Ïâ(âÏ(y))= Ïâ(âÏ(y))â (Î»fË + Ï)â(âÏ(y)),as claimed, where the third equality owes to Fact 2.37. Notice that, sinceâÏ(y) â dom âÏâ â domÏâ, this expression is well determined (albeit pos-sibly ââ). When Ï is a Legendre function, the operator âÏ : int domÏâint domÏâ is one-to-one and onto with âÏâ = (âÏ)â1, from which the finalclaim follows; see Fact 2.41.(ii) By direct computation, for every x â X one hasÎ»âââhullÏÎ»f(x) = â Î»ââenvÏÎ»(âââenvÏÎ»f)(x)= supyâY{Î»ââenvÏÎ»f(y)âDÏ(x, y)}= supyâYinfxâ²âX{Î»f(xâ²) + DÏ(xâ², y)âDÏ(x, y)}= supyâYinfxâ²âRn{(Î»fË + Ï)(xâ²) + â¨âÏ(y), xâ xâ²â©}â Ï(x)= supyâY{â¨âÏ(y), xâ© â supxâ²âRn{â¨âÏ(y), xâ²â© â (Î»fË + Ï)(xâ²)}}â Ï(x)= supyâY{â¨âÏ(y), xâ© â (Î»fË + Ï)â(âÏ(y))}â Ï(x),which is the first identity in the statement. The first inequality in thestatement owes to the fact that((Î»fË + Ï)â + Î´ranâÏ)â â¤ ((Î»fË + Ï)â)â = conv(Î»fË + Ï).Here, the first inequality follows from [11, Proposition 13.16(ii)], and theidentity from Lemma 9.5(ii). When Ï is 1-coercive, ranâÏ = dom âÏâ = Rn,where the first equality follows from Corollary 2.38 and the second owes toLemma 2.46, implying the that((Î»fË + Ï)â + Î´ranâÏ)â=((Î»fË + Ï)â)â.Applying Lemma 9.18, Lemma 9.5(ii) also yields the rest of the claimedstatements.1979.3. BREGMAN PROXIMAL MAP AND MOREAU ENVELOPENeither assumption of essential smoothness or 1-coercivity can be waivedfor the first inequality in Lemma 9.22(ii) to hold as equality, even if f isconvex.Example 9.23 (importance of essential smoothness for the Ï-proximalhull). Let Ï = j + δX where X = [â1, 1], which is 1-coercive but not essen-tially smooth, and let f = δ{0}. For any λ > 0 it is easy to see that thecanonical extension fË = f and ââenvÏÎ»f(y) = 12λy2 andÎ»âââhullÏÎ»f(x) = supyâ(â1,1){12y2 â 12(xâ y)2}= 12(âx2 + 2|x|),which quite differs from conv(Î»fË + Ï)(x) = δ{0}(x).Example 9.24 (importance of 1-coercivity for the Ï-proximal hull). Con-sider Ï = exp for X = R, which is essentially smooth but not 1-coercive.Let f = δ{0}, so that f = fË and for all λ > 0 one has (Î»fË+Ï)â â¡ â1. Then,âââhullÏÎ»f =((Î»fË + Ï)â + Î´ranâÏ)â=((Î»fË + Ï)â + Î´R++)â= 1 + Î´Râ ,which differs from conv(Î»fË + Ï) = 1 + δ{0}.Next we show that the (left) Bregman proximal operators and Moreauenvelopes can be expressed in terms of their Euclidean counterparts.Lemma 9.25 (Euclidean representations). Given f : X â R, extend it tofË : Rn â R by setting it toâ on RnX . Then, for every λ > 0 the followingidentities hold:(i) âââproxÏÎ»f = PÎ»(fË + ÏâjÎ»)â¦ âÏ.(ii) ââenvÏÎ»f =[eÎ»(fË +ÏâjÎ» ) +ÏââjÎ»] â¦ âÏ.If Ï is 1-coercive, then the following also holds(iii)âââhullÏÎ»f =[hÎ»(fË +ÏâjÎ» )â ÏâjÎ»]â£â£X .Proof. By direct computation, for every x â Rn and y â Y one hasDÏ(x, y) = Ï(x)â â¨âÏ(y), xâ©+ â¨âÏ(y), yâ© â Ï(y)= Ï(x)â j(x) + 12 â¥xââÏ(y)â¥2 â j(âÏ(y)) + Ïâ(âÏ(y)).After dividing by λ and adding fË ,fË(x) + 1λ DÏ(x, y) =(fË + ÏâjÎ»)(x) + 12λ â¥xââÏ(y)â¥2 + ÏââjÎ» (âÏ(y)).1989.4. THE BREGMAN LEVEL-PROXIMAL SUBDIFFERENTIALConsidering minimizers and infimum with respect to x â Rn yields theclaimed expressions of âââproxÏÎ»f and ââenvÏÎ»f .When Ï is 1-coercive, the identity hλf = (f+Î»â1j)âââj in Lemma 9.22yields thatâââhullÏÎ»f =[(fË + Î»â1Ï)ââ â Î»â1Ï]â£â£X=[(fË + ÏâjÎ» + Î»â1j)ââ â Î»â1j â ÏâjÎ» ]â£â£X=[hÎ»(fË + ÏâjÎ»)â ÏâjÎ» ]â£â£Xas claimed.9.4 The Bregman level-proximal subdifferentialWe henceforth pattern the analysis for the Euclidean case in Chapter 6 toidentify a âproximalâ subdifferential that enable a resolvent representationof the Bregman proximal map. To begin with, observe that x¯ â âââproxÏÎ»f(y¯)ifff(x) â¥ f(x¯) + 1λ DÏ(x, y¯)â 1λ DÏ(x, x¯), âx â X .As long as x¯ â intX , with the three-point identity (Fact 9.11) the aboveinequality can be expressed asf(x) â¥ f(x¯) + 1Î»â¨âÏ(y¯)ââÏ(x¯), xâ x¯â© â 1λ DÏ(x, x¯), âx â X ,where the vector u¯ = 1λ [âÏ(y¯) â âÏ(x¯)] plays the role of a âsubgradientâof f at x¯. In fact, it is not difficult to see that all vectors u¯ for which suchglobal inequality holds form a subset of the Fre´chet subdifferential of f atx¯; see Theorem 9.29 for the details. A similar computation for the rightproximal map reveals that y¯ â âââproxÏÎ»g(x¯) iffg(y) â¥ g(y¯) + 1Î»â¨x¯â y¯,âÏ(y)ââÏ(y¯)â© â 1λ DÏ(y¯, y), ây â Y,leading to the following definitions.Definition 9.26 (Bregman level-proximal subdifferential). The left (Breg-man) λ-level-proximal subdifferential of f : X â R is âââÏ/Î»p f : X â Rn,where u¯ â âââÏ/Î»p f(x¯) if and only ifx¯ â intX and f(x) â¥ f(x¯) + â¨u¯, xâ x¯â© â 1λ DÏ(x, x¯), âx â X . (9.15)1999.4. THE BREGMAN LEVEL-PROXIMAL SUBDIFFERENTIALSimilarly, the right (Bregman) λ-level-proximal subdifferential of g : Y â RisâââÏ/Î»p g : Y â Rn, where v¯ â âââÏ/Î»p g(y¯) if and only ifg(y) â¥ g(y¯) + â¨v¯,âÏ(y)ââÏ(y¯)â© â 1λ DÏ(y¯, y), ây â Y. (9.16)As mentioned earlier, the left level-proximal subdifferentialâââÏ/Î»p f is aparticular selection of âÌf characterized by an inequality that holds globally,as opposed to in the limit only. A similar claim can be made forâââÏ/Î»p g.In both cases, however, special care has to be taken when Ï is not fulldomain. To ease the following discussion, we list some elementary propertiesof set-valued operators involving restrictions and inverses. For a set-valuedoperator T : X â Y and U â X, we adopt a slight abuse of notation ininterpreting the restriction of T on U as the operator Tâ£â£U: X â Y (asopposed to Tâ£â£U: U â Y ) defined byTâ£â£U(u) ={T (u) if u â U ;â otherwise.Lemma 9.27. For every set-valued operator T : X â Y between two setsX and Y , the following hold:(i) For every U â X, one has that U â© Tâ1 = T â£â£Uâ1.(ii) For every V â Y , one has that (V â© T )â1 = Tâ1â£â£V.Proof. In what follows, let u¯ â U and v¯ â V be fixed.(i) u¯ â U â© Tâ1(v¯)â u¯ â U and v¯ â T (u¯)â v¯ â T â£â£U(u¯)â u¯ â T â£â£Uâ1(v¯).(ii) u¯ â (V â© T )â1(v¯)â v¯ â V and u¯ â Tâ1(v¯)â u¯ â Tâ1â£â£V(v¯).The following known result will be invoked in the proof of the followingtheorem.Fact 9.28. Let h : Rn â R be proper, lsc, and 1-coercive. Then, for everyx â conv domh there exists a collection of points xi â domh and αi > 0,i â I, with âiâI αi = 1 such thatx =âiâIÎ±ixi, conv h(x) =âiâIÎ±ih(xi), and â(conv h)(x) =âiâIâFh(xi).2009.4. THE BREGMAN LEVEL-PROXIMAL SUBDIFFERENTIALProof. See [17, Lemma 3.3 & Theorem 3.6].Here is the first main result of this section, where we present the rela-tion between the left and right Bregman level proximal subdifferential andproximal operators. Our result also features a generalization of [94, Prop.2.12], where the identity convâââproxÏÎ»f â¦âÏâ =âââproxÏÎ»âââhullÏÎ»f â¦âÏâ is establishedunder second-order assumptions.Theorem 9.29. For every λ > 0, f : X â R and g : Y â R the followinghold:(i)âââÏ/Î»p f =[âF(f + Î»â1Ï)]â£â£intX â Î»â1âÏ â âÌf .(ii) âââproxÏÎ»f = âF(Ï+ Î»f)â1 â¦ âÏ.(iii) intX â©âââproxÏÎ»f =(âÏ+ Î»âââÏ/Î»p f)â1 â¦ âÏ.(iv) âââproxÏÎ»g =(Id+Î»âââÏ/Î»p g)â1â£â£X .If Ï is Legendre, then the following also hold:(v)âââÏ/Î»p g = âF(g â¦ âÏâ + Î»â1Ïâ)â Î»â1âÏâ â âÌ[g â¦ âÏâ].(vi) âââproxÏÎ»g = âF(Ïâ + λg â¦ âÏâ)â1.(vii) âââproxÏÎ»f â¦ âÏâ and âââproxÏÎ»g are monotone operators.If, additionally, Ï is 1-coercive, then the following also holds:(viii) convâââproxÏÎ»f â¦ âÏâ =âââproxÏÎ»âââhullÏÎ»f â¦ âÏâ.Proof. We start with the first four claims without extra assumptions on Ï.(i) The claimed identity holds trivially outside intX . Let x¯ â intX be fixed.After expanding DÏ(x, x¯), the inequality in (9.15) can be cast as[f + Î»â1Ï](x) â¥ [f + Î»â1Ï](x¯) + â¨u¯â Î»â1âÏ(x¯), xâ x¯â© âx â X .This shows that u¯ â âââÏ/Î»p f(x¯) iff u¯ â Î»â1âÏ(x¯) â âF[f + Î»â1Ï](x¯), whichafter adding âÏ(x¯) (well defined, since x¯ â intX ) on both sides leads tothe claimed identity. The inclusionâââÏ/Î»p f(x¯) â âÌf(x¯) is trivial if x¯ /â intX ,since the left-hand side is empty in this case. When x¯ â intX , it followsfrom (9.15) together with the fact that Ï is differentiable at x¯ and thereforeDÏ(x, x¯) = o(â¥xâ x¯â¥) as xâ x¯.2019.4. THE BREGMAN LEVEL-PROXIMAL SUBDIFFERENTIAL(ii) Noticing that âââproxÏÎ»f(y¯) = argmin {λf + Ïâ â¨âÏ(y¯), · â©}, the claimfollows from the fact that argmin {Ï â â¨u, · â©} = [âFÏ]â1(u) holds for everyÏ : X â R and u â Rn.(iii) Assertion (ii) in combination with Lemma 9.27(i) implies that intX â©âââproxÏÎ»f =([âF(Ï+ Î»f)]â£â£intX)â1 â¦ âÏ. In turn, the identity in assertion (i)yields the claim.In what follows, suppose that Ï is Legendre.(iv) For every x¯ â X we have y¯ â âââproxÏÎ»g(x¯) iffây â Y g(y) â¥ g(y¯) + 1λ DÏ(x¯, y¯)â 1λ DÏ(x¯, y),which using the three-point identity of Fact 9.11 expands toây â Y g(y) â¥ g(y¯)â 1Î»â¨x¯â y¯,âÏ(y¯)ââÏ(y)â© â 1λ DÏ(y¯, y).This shows thaty¯ â âââproxÏÎ»g(x¯)â x¯ â X and x¯ây¯Î» ââââÏ/Î»p g(y¯)â x¯ â X and y¯ â (Id+Î»âââÏ/Î»p g)â1(x¯).Invoking Lemma 9.27(ii) results in the claimed expression.(v) By appealing to Fact 9.10 and denoting η = âÏ(y) and η¯ = âÏ(y¯), theinequality in (9.16) readsg â¦ âÏâ(Î·) â¥ g â¦ âÏâ(Î·¯) + â¨v¯, η â Î·¯â© â 1λ DÏâ(Î·, η¯) âÎ· â int domÏâ,and the same conclusion follows.(vi) Similarly as in the proof of assertion (ii), the claim follows by observingthatâââproxÏÎ»g(x¯) = argminyâY{Î»g(y) + Ï(y)â â¨âÏ(y), x¯â yâ©}= argminyâY{Î»g(y) + Ïâ(âÏ(y))â â¨âÏ(y), x¯â©}= argminÎ·âint domÏâ{Î»g â¦ âÏâ(Î·) + Ïâ(Î·)â â¨Î·, x¯â©} .(vii) Follows from assertions (ii) and (vi), since âFh is a monotone operatorfor every function h, and as such so is its inverse.2029.4. THE BREGMAN LEVEL-PROXIMAL SUBDIFFERENTIALTo conclude, suppose that Ï is both Legendre and 1-coercive.(viii) In order to simplify the notation, let h =âââhullÏÎ»f . It follows fromLemma 9.22(ii) that λh+ Ï = conv(λf + Ï) is convex, to which [91, Propo-sition 2.10(ii)] entailsâââproxÏÎ»h â¦ âÏâ =[â(Î»h+ Ï)]â1=[â conv(λf + Ï)]â1. (9.17)We proceed to prove the claimed identity:âââ Let x¯ â âââproxÏÎ»f â¦ âÏâ(Î·¯) be fixed, so that η¯ â âF(Î»f + Ï)(x¯) =â conv(λf + Ï)(x¯), where the inclusion follows from assertion (ii) and theidentity, in turn, from Fact 9.6. Equivalently, x¯ â [â conv(λf + Ï)]â1(Î·¯) =âââproxÏÎ»hâ¦âÏâ(Î·¯) by (9.17). From the arbitrarity of x¯ and η¯ we conclude thatconvâââproxÏÎ»f â¦ âÏâ(Î·¯) â convâââproxÏÎ»h â¦ âÏâ(Î·¯).Since h+Î»Ï is convex, [91, Proposition 2.10(ii)] yields thatâââproxÏÎ»hâ¦âÏâ(Î·¯)is convex, hence that the convex hull in the right-hand side of the inclusionabove is superfluous.âââ Consider now x¯ â âââproxÏÎ»h â¦ âÏâ(Î·¯), so that η¯ â â conv(λf + Ï)(x¯) by(9.17). Notice that λf + Ï too is 1-coercive, so that Fact 9.28 ensures thatx¯ is a convex combination of points x¯i, i â I, withη¯ ââiâIâF(Î»f+Ï)(x¯i) â x¯i â[âF(Î»f+Ï)]â1(Î·¯) =âââproxÏÎ»fâ¦âÏâ(Î·¯) âi â I.This show that x¯ â convâââproxÏÎ»f â¦ âÏâ(Î·¯), and the sought inclusion followsfrom the arbitrarity of x¯ and η¯.Remark 9.30. The identity in Theorem 9.29(iii) can equivalently be statedasx¯ â âââproxÏÎ»f(y¯) â© intX â âÏ(y¯)ââÏ(x¯)Î» ââââÏ/Î»p f(x¯).In particular, domâââÏ/Î»p f â intX â© ranâââproxÏÎ»f , and equality holds when Ïis 1-coercive (a y¯ satisfying the inclusion on the right always exists in thiscase).Similarly, the identity in Theorem 9.29(iv) readsy¯ â âââproxÏÎ»g(x¯) â x¯ â X and x¯ây¯Î» ââââÏ/Î»p g(y¯),which implies domâââÏ/Î»p g â ranâââproxÏÎ»g, and with identity holding when Ïhas full domain (that is, X = Rn).2039.4. THE BREGMAN LEVEL-PROXIMAL SUBDIFFERENTIALRemark 9.31. We make some comments on the content of Theorem 9.29.(i) The identity in Theorem 9.29(iii) can equivalently be stated asx¯ â âââproxÏÎ»f(y¯) â© intX â âÏ(y¯)ââÏ(x¯)Î» ââââÏ/Î»p f(x¯).Thus, domâââÏ/Î»p f â intX â© ranâââproxÏÎ»f and equality holds when Ï is1-coercive (a y¯ satisfying the inclusion on the right always exists inthis case).In absence of 1-coercivity of Ï, equality may fail even if f is convexand X is open. Let Ï(x) = â ln(x) and f(x) = â lnx + 2x â 2, andconsider x¯ = 1 and λ = 1. It is easy to see that u¯ := 1 â âââÏ/Î»p f(x¯),havingâ lnx+ 2xâ 2 =0f(1)+1(xâ 1) +xâ1âlnxDÏ(x, 1) âx.Thus, x¯ = 1 â domâââÏ/Î»p f , yet x¯ /â ranâââproxÏf since there exists no y¯such that 0 â â[f + Ï](x¯)ââÏ(y¯) = âÏ(y¯).(ii) The identity in Theorem Theorem 9.29(iv) readsy¯ â âââproxÏÎ»g(x¯) â x¯ â X and x¯ây¯Î» ââââÏ/Î»p g(y¯),which implies domâââÏ/Î»p g â ranâââproxÏÎ»g, and with identity holding whenÏ has full domain (that is, X = Rn). Also note that, unless X = Rn,the restriction to X is fundamental, and under no assumption on Ïsuch as 1-coercivity and/or Legendreness, nor on g such as convexitycan it be waived. To see this, observe that, in general, the inequalityin (9.16) amounts tov¯ â âââÏ/Î»p g(y¯) â y¯ â argminyâY{g(y) + 1λ DÏ(y¯, y)â â¨v¯,âÏ(y)â©}.Let Ï(x) = x lnxâx, which is 1-coercive and Legendre with X = [0,â)and Y = (0,â). Fix λ > 0 and y¯ â Y, and choose any v¯ < âÎ»â1y¯.Observe that v¯ â âââÏ/Î»p g(y¯) for the linear function g(y) = (Î»y¯)â1v¯y.Then, one has thatx¯ = y¯ + λv¯ â (Id+Î»âââÏ/Î»p g)(y¯), that is, y¯ â (Id+Î»âââÏ/Î»p g)â1(x¯),but clearly y¯ /â âââproxÏÎ»g(x¯), since 0 > x¯ /â X .2049.4. THE BREGMAN LEVEL-PROXIMAL SUBDIFFERENTIALThe following generalization of resolvent will be useful; see [29, Definition1.1].Definition 9.32 (warped resolvent). Let X â Rn be a nonempty set and letY â X be a nonempty subset. Let K : Y â X be a single-valued operatorand let T : Y â X be a set-valued operator such that ranK â ran(K + T ).The warped resolvent of T with kernel K is (K + T )â1 â¦K.Theorem 9.29(iii) suggests that the following assumption should be inplace in order for the intersections with intX in the formulas therein besuperfluous.Assumption 9.33. ranâââproxÏÎ»f â int domÏ.Corollary 9.34. For every λ > 0, f : X â R and g : Y â R the followinghold:(i) Warped resolvent representation of âââproxÏÎ»f : if Assumption 9.33 holds,then âââproxÏÎ»f = (âÏ+ Î»âââÏ/Î»p f)â1 â¦ âÏ. (9.18)(ii) Resolvent representation of âââproxÏÎ»g: if X = Rn, thenâââproxÏÎ»g = (Id+Î»âââÏ/Î»p g)â1. (9.19)As showcased in Theorem 9.29(iii) & (iv) the appeal of the newly intro-duced subdifferentials stems from the fact that they enable a full resolventcharacterization of the Bregman proximal maps, in the same way that thesubdifferential does for convex functions. We next show that left and rightlevel-proximal subdifferentials are mutually related in a similar fashion asthe proximal mappings when Ï is of Legendre type.Lemma 9.35. Let Ï be of Legendre type. Then, for every g : Y â R onehasâââÏ/Î»p g =âââÏâ/Î»p (g â¦ âÏâ) â¦ âÏ,while for f : X â R one hasâââÏ/Î»p f ââââÏâ/Î»p (f â¦ âÏâ) â¦ âÏ.A sufficient condition for the inclusion to hold everywhere as equality ishavingf(x) = lim infintXâxâ²âxf(xâ²) âx â X intX . (9.20)2059.5. POINTWISE CHARACTERIZATIONSProof. We begin by proving the claim for f . We haveu¯ â âââÏ/Î»p f(x¯) â f(x) â¥ f(x¯) + â¨u¯, xâ x¯â© â 1λ DÏ(x, x¯) âx â intX(and x¯ â intX )â fË(Î¾) â¥ fË(Î¾¯) + â¨u¯,âÏâ(Î¾)ââÏâ(Î¾¯)â© â 1λ DÏâ(Î¾¯, Î¾)âÎ¾ â int domÏââ u¯ â âââÏâ/Î»p fË(Î¾¯),where the first equivalence follows from Fact 9.10 with fË = f â¦ âÏâ, ξ =âÏ(x), η = âÏ(y), and ξ¯ and η¯ defined similarly. Under the condition in(9.20), the first inequality above can be extended to all x â X (so that theimplication therein can be reversed). Indeed, for every x â X intX thereexists a sequence intX â xk â x such that f(xk) â f(x), and a limitingargument in the inequality yields the claim.We now turn to the claim for g. Denote gË = g â¦ âÏâ and adopting thesame notation for ξ, η, ξ¯, η¯ as above, we havev¯ â âââÏ/Î»p g(y¯) â g(y) â¥ g(y¯) + â¨v¯,âÏ(y)ââÏ(y¯)â© â 1λ DÏ(y¯, y) ây â Yâ gË(Î·) â¥ gË(Î·¯) + â¨v¯, η â Î·¯â© â 1λ DÏâ(Î·, η¯) âÎ· â int domÏââ gË(Î·) â¥ gË(Î·¯) + â¨v¯, η â Î·¯â© â 1λ DÏâ(Î·, η¯) âÎ· â domÏâ,where the second equivalence follows from Fact 9.10, and the last one fromthe fact that dom gË â int domÏâ. By definition, this means that v¯ ââââÏâ/Î»p gË(Î·¯), yielding the claim.Remark 9.36. The boundary qualification (9.20) indicates that f is fullycharacterized by its value onto intX . When f is lsc, this condition canbe expressed as f = cl(fË), where fË : X â R is given by fË(x) = f(x) forx â intX , and â otherwise.9.5 Pointwise characterizationsThe goal of this section is to characterize the Bregman level proximal subdif-ferentiability, beginning with a result concerning the fundamental question:How does one characterizeâââÏ/Î»p f? The definition (9.15) is always an option,but it may be difficult to deal with.We now present equivalent descriptions forâââÏ/Î»p f . Akin to classic subd-ifferentials, which are often characterized by the behavior of minorants with2069.5. POINTWISE CHARACTERIZATIONSmore favorable properties, the hypoconvex minorantâââhullÏÎ»f of f completelydeterminesâââÏ/Î»p f .Theorem 9.37. Suppose that Ï is 1-coercive and Legendre, and let f : X âR be proper, lsc, and Ï-prox-bounded. Then, for every λ â (0, λf ), x¯ â intX ,u¯ â Rn, and denoting y¯ = âÏâ(Î»u¯+âÏ(x¯)) the following are equivalent:(i) u¯ â âââÏ/Î»p f(x¯).(ii) x¯ â âââproxÏÎ»f(y¯).(iii) y¯ â âââproxÏÎ»(âââenvÏÎ»f)(x¯) and x¯ â domâââÏ/Î»p f .(iv) x¯ây¯Î» ââââÏ/Î»p (âââenvÏÎ»f)(y¯) and x¯ â domâââÏ/Î»p f .(v) u¯ â ââââhullÏÎ»f(x¯) andâââhullÏÎ»f(x¯) = f(x¯).(vi) âÏ(y¯) â â conv(λf + Ï)(x¯) and conv(λf + Ï)(x¯) = (λf + Ï)(x¯).Proof. We start by observing that the definition of y¯ can equivalently beexpressed asu¯ = âÏ(y¯)ââÏ(x¯)Î» . (9.21)â(i) â (ii)â Taking into account (9.21), this is Theorem 9.29(iii).â(ii) â (iii)â We start by observing that nonemptyness of âââÏ/Î»p f(x¯) followsby virtue of the equivalence â(i) â (ii)â proven above. We haveâââhullÏÎ»f(x¯) â¤ f(x¯) =ââenvÏÎ»f(y¯)â 1λ DÏ(x¯, y¯)â¤ supyâY{ââenvÏÎ»f(y)â 1λ DÏ(x¯, y)} =âââhullÏÎ»f(x¯).All inequalities thus hold as equalities, and in particulary¯ â argmaxyâY{ââenvÏÎ»f(y)â 1λ DÏ(x¯, y)} = âââproxÏÎ»(âââenvÏÎ»f)(x¯)as claimed.â(iii) â (ii)â We haveâââhullÏÎ»f(x¯) =ââenvÏÎ»(âââenvÏÎ»)(x¯) = âââenvÏÎ»f(y¯) + 1λ DÏ(x¯, y¯). (9.22)2079.5. POINTWISE CHARACTERIZATIONSOn the other hand, by Theorem 9.29(iii) there exists y¯â² â Y such thatx¯ â âââproxÏÎ»f(y¯â²) (specifically, of the form y¯â² = âÏâ(v¯ + Î»âÏ(x¯)) with v¯ ââââÏ/Î»p f(x¯)). Then, for such y¯â² one hasf(x¯) =ââenvÏÎ»f(y¯â²)â 1λ DÏ(x¯, y¯â²) â¤âââhullÏÎ»f(x¯) â¤ f(x¯)implying that f(x¯) =âââhullÏÎ»f(x¯). By combining this identity with (9.22) weobtain that ââenvÏÎ»f(y¯) = f(x¯) + 1λ DÏ(x¯, y¯), that is, that x¯ â âââproxÏÎ»f(y¯).â(iii) â (iv)â This is Theorem 9.29(iv).â(i) â (v)â By virtue of the equivalence â(i)â(ii)â proven above, we havex¯ â âââproxÏÎ»f(y¯). Then,f(x¯) â¥ âââhullÏÎ»f(x¯) = supwâY{ââenvÏÎ»f(w)â 1λ DÏ(x¯, w)}â¥ ââenvÏÎ»f(y¯)â 1λ DÏ(x¯, y¯) = f(x¯),where the first inequality owes to Lemma 9.22(ii). This shows that f(x¯) =âââhullÏÎ»f(x¯). By adding Î»â1Ï to both sides in (9.15), from the inclusion u¯ ââââÏ/Î»p f(x¯) we have thatf + 1Î»Ï â¥ f(x¯) + â¨u¯, · â x¯â© â 1λ DÏ( · , x¯) + 1Î»Ï= f(x¯) + 1Î»Ï(x¯) +1Î»â¨âÏ(y¯), · â x¯â©=âââhullÏÎ»f(x¯) +1Î»Ï(x¯) +1Î»â¨âÏ(y¯), · â x¯â©.By taking the closed convex hull and by invoking of Lemma 9.22(ii) thisresults inâââhullÏÎ»f +1Î»Ï â¥âââhullÏÎ»f(x¯) +1Î»Ï(x¯) +1Î»â¨âÏ(y¯), · â x¯â©,that is,1Î»âÏ(y¯) â âF[âââhullÏÎ»f +1Î»Ï](x¯) = â[âââhullÏÎ»f +1Î»Ï](x¯) = ââââhullÏÎ»f(x¯)+1Î»âÏ(x¯),where the first equality follows from the convexity ofâââhullÏÎ»f +1Î»Ï. Recallingthat Î»â1âÏ(y¯) = u¯+ Î»â1âÏ(x¯), cf. (9.21), the proof follows.â(v) â (vi)â Apply Lemma 9.22(ii).2089.5. POINTWISE CHARACTERIZATIONSâ(vi) â (i)â We have,(âx â X ) (f + Î»â1Ï)(x) â¥ conv(f + Î»â1Ï)(x)â¥ conv(f + Î»â1Ï)(x¯) + â¨u¯+ Î»â1âÏ(x¯), xâ x¯â©= (f + Î»â1Ï)(x¯) + â¨u¯+ Î»â1âÏ(x¯), xâ x¯â©= f(x¯) + â¨u¯, xâ x¯â© â 1λ DÏ(x, x¯) + 1Î»Ï(x),which by (9.15) shows that u¯ â âââÏ/Î»p f(x¯).Equipped with Theorem 9.37, one may calculateâââÏ/Î»p f easily through ageometric approach utilizing conv(λf + Ï).Example 9.38. Let f(x) = xâ1â x2 and Ï(x) = ââ1â x2, and denotex0 = 0. For λ = 1,conv(f + Ï)(x) ={(f + Ï)(x) if x < x0;(f + Ï)â²(x0)(xâ x0) + (f + Ï)(x0) otherwise,which is smaller than (f+Ï)(x) on (x0, 1]. As such, Theorem 9.37(vi) impliesthatâââÏpf(x) ={{âf(x)} , if â 1 < x â¤ x0;â , if x0 â¤ x < 1.â¢fÏf + Ïconv(f + Ï)Figure 9.3: An illustration of Example 9.38.2099.5. POINTWISE CHARACTERIZATIONSIn convex analysis, differentiability of a convex function is closely tiedwith its subdifferential operator being single-valued; see for instance Fact 7.8.Next we explore a parallel result in the absence of convexity through thelens ofâââÏ/Î»p f .Theorem 9.39. Suppose that Ï is 1-coercive and Legendre, and let f :X â R be proper, lsc, and Ï-prox-bounded. Then, for every λ â (0, λf ) andx¯ â intX the following are equivalent:(i)âââÏ/Î»p f(x¯) is a singleton.(ii) conv(f + Î»â1Ï) is differentiable and coincides with f + Î»â1Ï at x¯.(iii)âââhullÏÎ»f is differentiable and coincides with f at x¯.Under any of the above equivalent conditions, one also has thatâ conv(f + Î»â1Ï)(x¯) = u¯+ Î»â1âÏ(x¯) and ââââhullÏÎ»f(x¯) = u¯,where {u¯} =âââÏ/Î»p f(x¯).Proof.â(i) â (ii)â Let âââÏ/Î»p f(x¯) = {u¯}. Then appealing to Theorem 9.37(v) yieldsconv(f + Î»â1Ï)(x¯) = (f + Î»â1Ï)(x¯), and in particularâ conv(f + Î»â1Ï)(x¯) = âF(f + Î»â1Ï)(x¯) ={u¯+ Î»â1âÏ(x¯)} ,where the first identity owes to Fact 9.6 and the second one to Theo-rem 9.29(i). â conv(f + Î»â1Ï)(x¯) is thus a singleton, and hence conv(f +Î»â1Ï) is differentiable at x¯ by Fact 7.8.â(ii) â (iii)â Immediate from Lemma 9.22(ii).â(iii) â (i)â It follows from Lemma 9.22(ii) that conv(f + Î»â1Ï)(x¯) = (f +Î»â1Ï)(x¯), hence that âF(f + Î»â1Ï)(x¯) = â conv(f + Î»â1Ï)(x¯) owing toFact 9.6. Differentiability ofâââhullÏÎ»f at x¯ implies single-valuedness of thesesubdifferentials, hence that ofâââÏ/Î»p f(x¯) by virtue of Theorem 9.29(i).We end this section with a subdifferential characterization of relativesmoothness.Definition 9.40 (relative smoothness). Let f : X â R, let Ï be a Legendrefunction, and let L â¥ 0. We say that f is L-smooth relative to Ï if LÏÂ± fis convex on intX .2109.5. POINTWISE CHARACTERIZATIONSLemma 9.41. Let f : X â R and let x¯ â intX . Then f is differentiable atx¯ with (âx â X ) DLÏÂ±f (x, x¯) â¥ 0 if and only if âââÏ/Lâ1p (Â±f)(x¯) Ì¸= â .Proof. The âonly ifâ direction follows immediately implies that Â±âf(x¯) ââââÏ/Lâ1p (Â±f)(x¯). Now assume âââÏ/Lâ1p (Â±f)(x¯) Ì¸= â . Then âÌ(Â±f)(x¯) Ì¸= â inview of Theorem 9.29(i), which means that f is differentiable at x¯ andmoreoverâ =Ì¸âââÏ/Lâ1p (Â±f)(x¯) â âÌ(Â±f)(x¯) = Â±âf(x¯)ââââÏ/Lâ1p (Â±f)(x¯) = Â±âf(x¯).Hence (âx â X ) DLÏÂ±f (x, x¯) â¥ 0 by using the definition of âââÏ/Lâ1p (Â±f)(x¯),as desired.Theorem 9.42. Let f : X â R, let Ï be a 1-coercive Legendre function,and let L > 0. ThenâââÏ/Lâ1p (Â±f) Ì¸= â on intX implies that f is L-smoothrelative to Ï. This implication can be reversed provided thatf(x) = limintXâxâ²âxf(xâ²) âx â X intX . (9.23)Proof. Suppose first thatâââÏ/Lâ1p (Â±f) Ì¸= â on intX . Then Lemma 9.41implies that (âx, xâ² â intX ) DLÏÂ±f (xâ², x) â¥ 0 and equivalently LÏ Â± f isconvex on intX . Conversely, the relative smoothness enforces that LÏ Â±f +δintX are convex, which combined with the additional assumption (9.23)implies that LÏ Â± f = cl(LÏ Â± f + δintX ) are convex. In turn, to eachx â intX = ri dom(LÏÂ± f) corresponds some u â â(LÏÂ± f)(x) such that(âxâ² â X ) (LÏÂ± f)(xâ²) â¥ (LÏÂ± f)(x) + â¨u, xâ² â xâ©,which amounts to uâ LâÏ(x) â âââÏ/Lâ1p (Â±f)(x).Remark 9.43. The condition (9.23) is necessary. ConsiderÏ(x) ={ââ1â x2, if |x| â¤ 1;â, if |x| > 1,and f : X = [â1, 1] â R defined by f(x) = â1â x2 for |x| < 1 andf(x) = â1 for |x| = 1. Then f is smooth relative to Ï because both ÏÂ±f areconvex on intX = (â1, 1). However, conv(f +Ï)(x) = â1 < 0 = (f +Ï)(x)2119.6. GLOBAL CHARACTERIZATIONSfor all |x| < 1, which means that âââÏ/Î»p f(x) = â for every x â (â1, 1) in viewof Theorem 9.37. This is becauselimXâxâ±1f(x) = 0 Ì¸= f(±1),violating the condition (9.23).9.6 Global characterizationsIn this section, our primarily goal is to investigate how various global proper-ties of a function, a Bregman proximal operator and an envelope correspondto the associatedâââÏ/Î»p f . Along the way, we relax certain known resultsin [45,91].9.6.1 A coincidence theoremWe start with a coincidence result, which is parallel to the fundamentaltheorem of calculus, which states that for differentiable functions f1, f2 :Rn â R, âf1 = âf2 if and only if f1 and f2 differ by a constant. We shall seethat a similar relation holds for having âÎ»p f1 = âÎ»p f2, and it generalizes [59,Theorem 7.1].Theorem 9.44. Let fi : X â R, i = 1, 2 be proper, lsc, and Ï-prox-bounded.Let λ â (0,min {λf1 , λf2}) be fixed, and consider the following assertions:(i) ââenvÏÎ»f1 =ââenvÏÎ»f2 + c for some c â R.(ii)âââhullÏÎ»f1 =âââhullÏÎ»f2 + câ² for some câ² â R.(iii) âââproxÏÎ»f1 =âââproxÏÎ»f2.(iv)âââÏ/Î»p f1 =âââÏ/Î»p f2.Then, (i) â (ii) (in which case c = câ²) and (iii) â (iv) always hold. IfAssumption 9.33 holds for both f1, f2 and the chosen λ, then (iii) â (iv).If Ï is 1-coercive and Legendre, then (iii) â (i).Proof. We start from implications without additional assumptions.â(i) â (ii)â Follows from the fact that, for any c â R and f : X â R,one has c+âââhullÏÎ»f =âââhullÏÎ»(f + c) = âââenvÏÎ»(âââenvÏÎ»(f + c)) and the identityââenvÏÎ»(âââhullÏÎ»f) =ââenvÏÎ»f .2129.6. GLOBAL CHARACTERIZATIONSâ(iii) â (iv)â Obviously intX â© âââproxÏÎ»f1 = intX â© âââproxÏÎ»f2. Apply Theo-rem 9.29(iii) completes the proof.Suppose that Ï is Legendre and 1-coercive.â(iii) â (i)â For i = 1, 2, let hi : Rn â R be defined as fi+Î»â1(Ïâ j) on Xandâ on Rn X , and observe that it is lsc by virtue of Lemma 9.18. It alsofollows from Lemma 9.25 and the assumptions on Ï that âââproxÏÎ»fi â¦ âÏâ =Pλhi, to which Theorem 7.21 yields that the proximal hull of h1 satisfieshλh1 = hλh2 + c for some c â R. By invoking again Lemma 9.25, note thathÎ»hiâ£â£X =âââhullÏÎ»fi+Î»â1(Ïâj)â£â£X , i = 1, 2. Since Ïâj is finite on X = domÏ,the claimed identityâââhullÏÎ»f1 =âââhullÏÎ»f2 + c follows.Suppose now that Assumption 9.33 is satisfied.â(iv) â (iii)â Apply Theorem 9.29(iii) again, in which case the intersectionwith intX therein is superfluous because of Assumption 9.33.9.6.2 Hypoconvexity and monotonicityIn this section, we explore the interplay between âÎ»p f and the relative hypocon-vexity of f , and establish a Bregman extension of Fact 9.45.We say a function f : X â R is λ-hypoconvex for λ > 0 relative to Ï, iff + Î»â1Ï is convex. In particular, we say f : Rn â R is λ-hypoconvex if itis relatively λ-hypoconvex with kernel Ï = j.Fact 9.45. Let f : Rn â R be proper, lsc, and j-prox-bounded with thresholdλf . Then the following are equivalent for 0 < λ < λf :(i) f is λ-hypoconvex.(ii) Pλf is maximally monotone.(iii) Pλf is convex-valued.(iv) Pλf = (Id+Î»âf)â1.Proof. See [81, Prop. 12.19] and [33, Prop. 2.6].We first provide a simple result that will be useful in the proof of thefollowing Theorem 9.48.2139.6. GLOBAL CHARACTERIZATIONSLemma 9.46. Let h : Rn â R be a proper and lsc function. Then, h isconvex iff h â¡ conv h on ri dom conv h.Proof. The âonly ifâ implication is obvious. Suppose now that h â¡ conv hon ri dom conv h. Since h is proper, conv h Ì¸â¡ â and as such domconv his nonempty (and convex), and thus so is ri dom conv h by Fact 2.23. Fixx0 â ri dom conv h and x1 â conv domh â domconv h, and for µ â R definexµ = (1â µ)x0 + µx1. Then,h(x1) â¤ lim infµâ1âh(xµ) = limµâ1âconv h(xµ) = conv h(x1) <â,where the first inequality owes to lower semicontinuity, the last equality fol-lows from Fact 2.27 (even though conv h could, a priori, be improper, thereference still holds since x1 â domconv h), and the last inequality from thefact that x1 â domconv h. This shows that h(x1) = conv h(x1) < â forx1 â conv domh. From the arbitrarity of x1 â conv domh, it follows thatconv domh = domh and that h agrees with a convex function on its convexdomain, and must thus be convex.Lemma 9.47. Let C,D â Rn be convex sets be such that C â D. If C isnot entirely contained in the relative boundary of D, that is, C â D riD,then riC â riD.Proof. See [76, Corollary 6.5.2].What is the relation betweenâââÏ/Î»p f and the function f? The picture isclear at a point x, as Theorem 9.37 provides a connection between nonemptyâââÏ/Î»p f(x) and a pointwise hypoconvexity at x. The following result illustratesa global picture, revealing that domâââÏ/Î»p is âbigâ iff the underlying function fis hypoconvex. We also provide new correspondences, supplementing the oneto maximal monotonicity of proximal operator discovered in [45, Theorem4.2].Theorem 9.48. Suppose that Ï is Legendre and 1-coercive, let f : X â Rbe proper, lsc, and Ï-prox-bounded, and let λ â (0, λf ) be fixed. Considerthe following statements:(i) f is λ-hypoconvex relative to Ï, that is, f + Î»â1Ï is convex.(ii) âââproxÏÎ»f â¦ âÏâ is maximally monotone.2149.6. GLOBAL CHARACTERIZATIONS(iii) âââproxÏÎ»f â¦ âÏâ is convex-valued.(iv) ri conv dom f â© intX â domâââÏ/Î»p f .(v) ri conv dom f â dom âf and âf =âââÏ/Î»p f on intX .(vi) ri conv dom f â dom âf and âââproxÏÎ»f = (âÏ+ Î»âf)â1 â¦ âÏ.One always has (vi) â (i) â (ii) â (iii) â (iv) â (v). The implications(iv) â (v) and (i) â (v) hold when ri conv dom f â intX . If in additionAssumption 9.33 is satisfied, then (vi) â (i).Proof. We first prove the implications without additional assumptions.Throughout this proof, let fË : Rn â R be the canonical extension of fdefined by fË(x) = f(x) for x â X and fË(x) =â otherwise. It is helpful toremind that dom fË = dom f = dom(λf + Ï), and that λf + Ï is 1-coercive,and Î»fË + Ï is proper, lsc, and 1-coercive; see Lemma 9.18.â(i) â (ii)â The assumption together with Lemma 9.18 ensures that Î»fË +Ïis proper lsc (on Rn) and convex. HenceâââproxÏÎ»f â¦ âÏâ =âââproxÏÎ»fË â¦ âÏâ =(â(Î»fË + Ï))â1,where the last equality follows from Theorem 9.29(ii) and the convexity ofÎ»fË+Ï. The operator â(Î»fË+Ï) is maximally monotone in view of Fact 2.59,and therefore so is its inverse.â(ii) â (iii)â Follows from Theorem 9.29(viii) together with the fact thatâââproxÏÎ»f â¦ âÏâ is monotone.â(iii) â (i)â It is suffices to show that Î»fË + Ï is convex. Note thatâF(Î»fË + Ï) =(âââproxÏÎ»fËâ¦ âÏâ)â1=(âââproxÏÎ»âââhullÏÎ»fËâ¦ âÏâ)â1= âF(Î»âââhullÏÎ»fË + Ï) = â conv(Î»fË + Ï),where the first and the third equalities follow from Theorem 9.29(ii); thesecond one owes to Theorem 9.48(iii) and Theorem 9.29(viii). We claimthat âF(Î»fË + Ï) Ì¸= â on ri conv dom f . In turn Î»fË + Ï = conv(Î»fË + Ï)on ri conv dom f and therefore Î»fË + Ï is convex owing to Lemma 9.46. In-deed, notice that dom conv(Î»fË + Ï) = conv dom(Î»fË + Ï) = conv(dom fË â©domÏ) = conv dom f . Therefore ri conv dom f = ri dom conv(Î»fË + Ï) âdom â conv(Î»fË + Ï).2159.6. GLOBAL CHARACTERIZATIONSâ(vi) â (i)â One has that (âÏ+ Î»âf)â1 =âââproxÏÎ»f â¦ âÏâ = âF(Ï+ Î»f)â1,where the last identity follows from Theorem 9.29(ii), overall implying thatâF(Ï+ λf) = âÏ+ Î»âf . Hence âF(Î»f + Ï) Ì¸= â on ri conv dom(λf + Ï) =ri conv dom f â dom âf , which entails through Fact 9.6 that f + 1Î»Ï =conv(f + 1Î»Ï) on ri conv dom(f +1Î»Ï). Invoking Lemma 9.46 concludes theproof.â(i) â (iv)â The implication holds vacuously when ri dom f â© intX = â .Suppose now that ri dom f â© intX Ì¸= â and pick x â ri dom f â© intX . ThenâF(f + Î»â1Ï)(x) Ì¸= â and therefore âââÏ/Î»p f(x) Ì¸= â by Theorem 9.29(i).â(iv) â (v)â Suppose that ri conv dom f â© intX Ì¸= â , otherwise the claimholds vacuously. Then for x â ri conv dom f â© intX , (v) implies âââÏ/Î»p f(x) =âf(x) Ì¸= â .In what follows, suppose that ri conv dom f â intX , and observe thatri conv dom f = ri conv dom(λf + Ï).â(iv) â (v)â The assumptions (iv) and ri conv dom f â intX entailâââÏ/Î»p f = âF(f + Î»â1Ï)â Î»â1âÏ Ì¸= â on ri conv dom f,implying ri conv dom f â dom âf and moreover on ri conv dom fconv(f + Î»â1Ï) = conv(f + Î»â1Ï) = (f + Î»â1Ï)by Fact 9.6 and Lemma 9.5(ii). Apply Lemma 9.46 to see f+Î»â1Ï is convex,and therefore for x â intXâf(x) = â(f + Î»â1Ï)(x)â Î»â1âÏ(x) = âF(f + Î»â1Ï)(x)â Î»â1âÏ(x)= âÎ»p f(x),as desired.â(v)â (i)â Observe that ri conv dom f = ri dom conv f Ì¸= â ; the identity fol-lows from [11, Proposition 9.8(iv)] and [76, Theorem 6.3], and nonemptynessfrom properness of f together with Fact 2.23. Pick x0 â ri conv dom f =ri domconv(λf + Ï) â intX . Then,â =Ì¸ âf(x0) =âââÏ/Î»p f(x0) = âF[f + 1Î»Ï](x0)â 1Î»âÏ(x0),where the second identity again uses Theorem 9.29(i). This shows thatâF[f +1Î»Ï] agrees with â[f +1Î»Ï] on ri dom f and is nonempty there, whichappealing to Fact 9.6 implies that f+ 1Î»Ï = conv(f+1Î»Ï) on ri dom conv(f+1Î»Ï). Invoking Lemma 9.46 concludes the proof.2169.6. GLOBAL CHARACTERIZATIONSTo conclude, let us also assume that Assumption 9.33 holds.â(i) â (vi)â Appealing to Theorem 9.29(ii), one has that x¯ â âââproxÏÎ»f(y¯)iff âÏ(y¯) â âF(Î»f + Ï)(x¯), and such subdifferential equals â(Î»f + Ï)(x¯)by convexity. Since x¯ â intX by Assumption 9.33, it also coincides withâÏ(x¯) + Î»âf(x¯), leading to the sought conclusion.Remark 9.49. Wemake some comments on the conditions in Theorem 9.48:(i) The additional assumption ri conv dom f â intX is unavoidable forthe implication â(v) â (i)â. Consider Ï : R2 â R given byÏ(x) ={x1 lnx1 + x2 lnx2, if x â¥ 0;â, otherwise,and f : X = R2+ â R defined asf(x) ={â1â (1â x1)2 if x2 = 0;â, otherwise,with the convention that 0 ln(0) = 0. One has ri conv dom f = R++ Ã{0} while intX = R++ Ã R++, so that ri conv dom f Ì¸â intX .For every λ > 0, the conditions in Theorem 9.48(v) are met, havingâf =âââÏ/Î»p f = â on intX and ri conv dom f = R++ Ã {0} â R+ Ã{0} = dom âf . However, Theorem 9.48(i) fails, as f + Î»â1Ï is clearlynonconvex.(ii) When X = Rn, the assumption ri conv dom f â dom âf in the state-ments of Theorem 9.48(v) and (vi) is superfluous.Regarding Theorem 9.48(v), having âf =âââÏ/Î»p f everywhere impliesthrough Theorem 9.29(i) that â(f+Î»â1Ï) = âF(f+Î»â1Ï) everywhere,which further entails the monotonicity of â(f+Î»â1Ï) and equivalentlythe convexity of f + Î»â1Ï. Consequently, dom f is convex and âf =â(f + Î»â1Ï)â Î»â1âÏ Ì¸= â on ri dom f = ri conv dom f .Similarly, when X = Rn the conditionâââproxÏÎ»f = (âÏ+Î»âf)â1â¦âÏ inTheorem 9.48(vi) implies through Theorem 9.48(vii) that the operatorâ(Î»f+Ï) is monotone on the whole Rn, whence the convexity of Î»f+Ï.In turn, the inclusion ri conv dom f â dom âf readily follows.(iii) Assumption 9.33 is needed for the implication â(vi) â (i)â. Considerany kernel Ï whose domain is not open, and let f = δ{x¯} for some2179.6. GLOBAL CHARACTERIZATIONSx¯ â X intX . Then f + Î»â1Ï is trivially convex and âââproxÏÎ»f â¡ {x¯}.However, x¯ /â ran(âÏ+ Î»âf)â1 â¦ âÏ because x¯ â X intX .9.6.3 Convexity and nonexpansivenessIn this section, we investigate a non-Euclidean counterpart of the classiccorrespondence:Fact 9.50. Let f : Rn â R be a proper, lsc, and prox-bounded function.Then the following are equivalent:(i) f is convex.(ii) âf is (maximally) monotone.(iii) (âÎ» > 0) eλf is convex.(iv) (âÎ» > 0) Pλf is firmly nonexpansive.(v) (âÎ» > 0) eλf is differentiable and âeÎ»f is cocoercive.Proof. See [81, Theorem 12.17], [79, Theorem 1], and [90, Theorem 3.17].We shall see in this section that the same correspondence does not holdfor Bregman proximal operators and Moreau envelopes, owing to a dualitygap pointed out by Laude et al. [50]. See Fig. 9.2 for a summary of the mainresults in this section.Definition 9.51. Let D â Rn be nonempty. We say that an operatorT : D â int domÏ is âÏ-firmly nonexpansive (âÏ-FNE) ifâ¨T (a)â T (b), aâ bâ© â¥ â¨âÏ(T (a))ââÏ(T (b)), T (a)â T (b)â© âa, b â D.Definition 9.52. A proper lsc function f : X â R is said to be(i) BÏ-strongly convex if f â Ï is convex.(ii) BÏ-smooth if both f + Ï and âf + Ï are convex.Lemma 9.53. Suppose that Ï is Legendre and 1-coercive, and that Assump-tion 9.33 holds for two proper, lsc, and Ï-prox-bounded functions fi : X â R,i = 1, 2. If âââproxÏÎ»f1 = âââproxÏÎ»f2 for some λ â (0,min {λf1 , λf2}), then thereexists c â R such that âââhullÏÎ»f1 =âââhullÏÎ»f2 + c.2189.6. GLOBAL CHARACTERIZATIONSProof. For i = 1, 2, let hi : Rn â R be defined as fi+Î»â1(Ïâj) on X andâon Rn X , and observe that it is lsc by virtue of Lemma 9.18. It also followsfrom Lemma 9.25 and the assumptions on Ï that âââproxÏÎ»fi â¦ âÏâ = Pλhi, towhich [59, Theorem 6.1] yields that hλh1 = hλh2+ c for some c â R. By in-voking again Lemma 9.25, note that hÎ»hiâ£â£X =âââhullÏÎ»fi+Î»â1(Ïâj)â£â£X , i = 1, 2.Since Ïâj is finite on X = domÏ, the claimed identityâââhullÏÎ»f1 =âââhullÏÎ»f2+cfollows.Theorem 9.54. Suppose that Ï is Legendre and 1-coercive, and that As-sumption 9.33 holds for a proper, lsc, and Ï-prox-bounded function f : X âR and λ â (0, λf ). Then the following are equivalent:(i) f is convex.(ii) fâ£â£intX is convex.(iii)âââÏ/Î»p f is monotone.(iv)âââÏ/Î»p f is maximally monotone relative to intX , that is, no monotoneT : intX â Rn such that gphâââÏ/Î»p f â gphT .(v) â¨x1 â x2, η1 â Î·2â© â¥ âÏ(x1, x2) for every ηi â Rn and xi â âââproxÏÎ»f â¦âÏâ(Î·i), i = 1, 2; that is, âââproxÏÎ»f â¦ âÏâ is âÏ-firmly nonexpansive.Any of the above equivalent conditions implies that âââproxÏÎ»f â¦ âÏâ is single-valued and thus maximally monotone.Proof. In the following discussion, observe that because of Assumption 9.33(and Ï-prox-boundedness) it necessarily holds that dom f â© intX Ì¸= â .â(i) â (ii)â and â(iv) â (iii)â Trivial.â(ii) â (iii)â For every x â int domX it follows from Theorem 9.29(i) thatâââÏ/Î»p f(x) = âf(x) = â[fâ£â£intX](x), where the second equality follows fromthe fact that intX is open. Monotonicity of âââÏ/Î»p f then follows from that ofâ[fâ£â£intX].â(iii) â (iv)â Take (x¯, u¯) â intX Ã Rn which is monotonically related togphâââÏ/Î»p f , that is, such that for every (x, u) â gphâââÏ/Î»p fâ¨xâ x¯, uâ u¯â© â¥ 0. (9.24)2199.6. GLOBAL CHARACTERIZATIONSWe claim that u¯ â âââÏ/Î»p f(x¯), which proves the desired maximality. To see theclaim, take xË â âââproxÏÎ»f â¦ âÏâ(Î»u¯+âÏ(x¯)) =âââproxÏÎ»f â¦ âÏâ(Î»u¯+âÏ(x¯)) â©intX , where the equality holds due to Assumption 9.33, to which Theo-rem 9.29(iii) yields u¯ + Î»â1(âÏ(x¯) â âÏ(xË)) â âââÏ/Î»p f(xË) and (9.24) entailsthat â¨xËâ x¯,âÏ(x¯)ââÏ(xË)â© â¥ 0, enforcing x¯ = xË. Invoking Theorem 9.37 tothe inclusion x¯ â âââproxÏÎ»f â¦ âÏâ(Î»u¯+âÏ(x¯)) yields u¯ ââââÏ/Î»p f(x¯), justifyingthe claim.â(iii) â (v)â Fix ηi â Rn and xi â âââproxÏÎ»f â¦ âÏâ(Î·i) for i = 1, 2. Sincexi â intX by Assumption 9.33, it follows from Theorem 9.29(iii) thatÎ»â1 (ηi ââÏ(xi)) â âââÏ/Î»p f(xi), to which the monotonicity of âââÏ/Î»p f impliesthatâ¨x1 â x2, (η1 ââÏ(x1))â (η2 ââÏ(x2))â© â¥ 0.After rearranging, the claimed inequality is obtained. Taking η1 = η2 resultsin âÏ(x1, x2) â¤ 0, implying x1 = x2 (since Ï is essentially strictly convex).This shows that âââproxÏÎ»f â¦ âÏâ is indeed single-valued as required in Defini-tion 9.51, and is thus âÏ-FNE as claimed; maximal monotonicity is thenensured by Theorem 9.48.â(v)â (ii)â The inequality implies thatâââproxÏÎ»f is single-valued, hence Î»f+Ïis convex by virtue of Theorem 9.48. Therefore,âââÏ/Î»p f = âF(f + Î»â1Ï)â£â£intX â Î»â1âÏ = âfâ£â£intX , (9.25)where the first identity owes to Theorem 9.29(i). As an intermediate step,we first show that assertion (iii) holds true. Take ui â âââÏ/Î»p f(xi) for i = 1, 2.Then, Theorem 9.29(iii) entailsxi â âââproxÏÎ»f â¦ âÏâ(Î»ui +âÏ(xi)),from which âÏ-firm nonexpansiveness yieldsâ¨x1 â x2, λu1 +âÏ(x1)â λu2 ââÏ(x2)â© â¥ â¨x1 â x2,âÏ(x1)ââÏ(x2)â©.It follows immediately thatâââÏ/Î»p f is monotone. Combined with (9.25) wededuce that fË = f+δintX is proper (because of Assumption 9.33) and convex(though possibly not lsc).2209.6. GLOBAL CHARACTERIZATIONSâ(v) â (i)â As shown in the item above, f¯ = conv fË is proper and convex,and âââproxÏÎ»f is single-valued. For every y â Y we haveâââproxÏÎ»f(y) = argminxâintX{Î»f(x) + DÏ(x, y)}= argminxâX{Î»fË(x) + DÏ(x, y)}â argminxâX{Î»f¯(x) + DÏ(x, y)}=âââproxÏÎ»f¯(y),where the first identity holds because of Assumption 9.33, the second onebecause fË equals f on intX and â elsewhere, and the third one becauseof Lemma 9.7. On the other hand, both âââproxÏÎ»f(y) and âââproxÏÎ»f¯(y) are sin-gletons (the latter one because f¯ is convex). Therefore, the inclusion holdsas equality. We then infer that f¯ =âââhullÏÎ»f¯ =âââhullÏÎ»f + c = f + c for somec â R, where the first and last identity follow from the fact that f¯ + Î»â1Ïand f + Î»â1Ï are (proper, lsc and) convex, cf. Lemma 9.25, and the secondone from Lemma 9.53. Therefore, f coincides with the convex function f¯âc(in fact, with c = 0).We shall see soon that Assumption 9.33 is essential for the full equiva-lence in Theorem 9.54. Even more, the necessity of Assumption 9.33 show-cases a striking departure from the Euclidean case studied in [59, Theorem4.7]. When this requirement is not met, it can still be shown by the samearguments that intX â©âââproxÏÎ»f â¦âÏâ being âÏ-FNE is tantamount toâââÏ/Î»p fbeing monotone, but not necessarily maximally so, as the following exampledemonstrates.Example 9.55 (nonmaximal monotonicity ofâââÏ/Î»p f). ConsiderÏ(x) ={ââ1â x2, if |x| â¤ 1;â, otherwise,and f : X = [â1, 1]â R defined asf(x) ={â14 â (xâ 12)2, if |xâ 12 | â¤ 12 ;â, otherwise.2219.6. GLOBAL CHARACTERIZATIONSFor λ = 2, Theorem 9.29(i) yields thatâââÏ/Î»p f(x) =[âF(f + Î»â1Ï)]â£â£(â1,1)(x)â Î»â1âÏ(x)={(ââ, 12] , if x = 0;â , otherwise.Apparently,âââÏ/Î»p f is monotone but not maximally so. This is becauseranâââproxÏ2f = {0, 1} Ì¸â (â1, 1) = intX ,and therefore Assumption 9.33 is violated.fÏf + 12Ïconv(f + 12Ï)Figure 9.4: An illustration of Example 9.55.Fact 9.56. Suppose that Ï is Legendre and 1-coercive, and let f : X â R beproper, lsc, and Ï-prox-bounded. Then, for every λ â (0, λf ) the followingare equivalent:(i) λf + Ï is essentially strictly convex.(ii) âââproxÏÎ»f â¦ âÏâ is single-valued.(iii) ââenvÏÎ»f â¦ âÏâ is everywhere differentiable.Proof. See [45, Theorem 4.3].2229.6. GLOBAL CHARACTERIZATIONSCorollary 9.57. In the setting of Fact 9.56, any of the equivalent conditionstherein implies thatâââproxÏÎ»fâ¦âÏâ is continuous andââenvÏÎ»fâ¦âÏâ continuouslydifferentiable with â[ââenvÏÎ»f â¦ âÏâ] = Î»â1(âÏâ ââââproxÏÎ»f).Proof. The assumption in Fact 9.56(ii) implies continuity ofâââproxÏÎ»f through[45, Corollary 2.5]. In turn, the formula for the gradient and the claimabout continuous differentiability follow from the fact that âF(Î»f + Ï)â =âââproxÏÎ»f â¦ âÏâ by Theorem 9.29(ii).Theorem 9.58. Suppose that Ï is Legendre and 1-coercive, and that As-sumption 9.33 holds for a proper, lsc, and Ï-prox-bounded function f : X âR. Define hλ =ââenvÏÎ»f â¦ âÏâ. Then, for every λ â (0, λf ), the following areequivalent:(i) hλ is convex.(ii) hλ is convex and BÏâ-smooth.(iii) (λf + Ï)â is BÏâ-smooth.(iv) â¨x1 â x2, y1 â y2â© â¤ âÏâ(y1, y2) for all yi â Rn and xi â âââproxÏÎ»f â¦âÏâ(yi).(v) hλ is differentiable and satisfies the Bregman cocoercive inequalityλDhλ(ξ, η) â¥ DÏ(âÏâ(Î¾)âÎ»(âhÎ»(Î¾)ââhÎ»(Î·)), âÏâ(Î¾)) âÎ¾, η â Rn.When any one of the above holds,ââenvÏÎ»fâ¦âÏâ and (Î»f+Ï)â are continuouslydifferentiable.Proof.â(i) â (ii)â The left implication is obvious, while the right one from theexpression Ïâ â λhλ = (λf + Ï)â, see Lemma 9.22(i), since the right-handside is convex.â(ii) â (iii)â Again follows from Lemma 9.22(i). Under either assumption,[96, Proposition 3.7] ensures continuous differentiability of both (λf + Ï)âand hλ, and in particular essential strict convexity of λf + Ï by Fact 9.56.â(i)â (iv)â We have already shown that Î»f+Ï is essentially strictly convex,which invoking Corollary 9.57 ensures that hλ is differentiable with Î»âhÎ» =âÏâ ââââproxÏÎ»f â¦ âÏâ. The claimed inequality follows from monotonicity ofâhÎ».2239.6. GLOBAL CHARACTERIZATIONSâ(iv)â (i)â The inequality implies thatâââproxÏÎ»f â¦âÏâ is single-valued, whichimplies that hλ is differentiable by virtue of Fact 9.56 with Î»âhÎ» = âÏâ ââââproxÏÎ»f . The proof concludes by observing that the inequality in assertion(iv) is precisely monotonicity of âÏâ ââââproxÏÎ»f .â(i) â (v)â See [37, Proposition 5.5].â(v) â (i)â Trivial.Having established Theorems 9.54 and 9.58, we now show that, in strik-ing contrast to the Euclidean case, the convexity of ââenvÏÎ»f â¦ âÏâ is notequivalent to the convexity of f . We will explain this gap in Remark 9.61.Example 9.59. Let f(x) = 14(xâ 1)4â 14x4 and Ï(x) = 14x4. ThenââenvÏ1f â¦âÏâ(y) = ây. Clearly ââenvÏ1f â¦ âÏâ is convex, while f is not.Example 9.60. Let f(x) = x and Ï(x) = 13 |x|3. Then ââenvÏ1f â¦ âÏâ(y) =23 |y|32 â 23 |y â 1|32 . Clearly f is convex but ââenvÏ1f â¦ âÏâ is not.Remark 9.61 (a duality gap). In the Euclidean case, the Moreau envelopeis convex if and only if so is the underlying function; see Theorem 8.2. Suchequivalence hinges on the well-known duality between Lipschitz smoothnessand strong convexity; see Fact 2.66(iii). Indeed, λeλf = j â (λf + j)â andthereforeeλf is convex â j â (λf + j)â is convexâ (λf + j)â is 1-Lipschitz smoothâ λf + j is 1-strongly convexâ f is convex,where the second equivalence owes to Fact 2.66. However, Laude, Themelis,and Patrinos developed a conjugacy theory in [50], revealing that the classicduality between strong convexity and Lipschitz smoothness is not inheritedby their Bregman extensions, being BÏ-strong convexity and BÏâ-smoothnessintroduced in Definition 9.52, respectively. More precisely, for a functiong : Rn â Rg is BÏ-strongly convex Ì¸â gâ is BÏâ-smooth.Adopting these jargons, one hasf is convexâ λf + Ï is BÏ-strongly convexÌ¸â (λf + Ï)â is BÏâ-smoothâââenvÏÎ»f â¦ âÏâ is convex.2249.6. GLOBAL CHARACTERIZATIONSThis means, the gap between the convexity ofââenvÏÎ»f â¦âÏâ and that of f owesexactly to the duality gap between BÏ-strong convexity and BÏâ-smoothness.Nevertheless, instead of BÏ-strong convexity, BÏâ-smoothness is the dualof anisotropic strong convexity ; see [50, Theorem 4.3]. We remind that aproper and lsc function f : Rn â R is said to be anisotropically stronglyconvex relative to Ï, or aÏ-strongly convex, if for every (x¯, v¯) â gph âf itholds thatf(x) â¥ f(x¯) + Ï(xâ x¯+âÏâ(v¯))â Ï(âÏâ(v¯)) âx â Rn.Under a full domain assumption, we may thus synopsize the main conditionsin both Theorems 9.54 and 9.58 as follows:Corollary 9.62. Suppose that Ï is Legendre, 1-coercive and with domÏ =Rn, and that Assumption 9.33 holds for a proper, lsc, and Ï-prox-boundedfunction f : Rn â R. Then, for every λ â (0, λf ) the following are equiva-lent:(i) λf + Ï is both BÏ- and aÏ-strongly convex.(ii) Both f and ââenvÏÎ»f â¦ âÏâ are convex.(iii) âÏ(x1, x2) â¤ â¨x1âx2, y1â y2â© â¤ âÏâ(y1, y2) holds for all yi â Rn andxi =âââproxÏÎ»f â¦ âÏâ(yi), i = 1, 2.In the rest of this section, we investigate conditions under whichââenvÏÎ»f â¦âÏâ is convex. The following lemma will be useful soon.Lemma 9.63. Let f : X â R be proper, lsc, and prox-bounded with thresh-old λf . Suppose that f + Ïf j is convex for some Ïf â R. Then for everyλ â (0, λf ) and xi â Pλf(yi), i = 1, 2,â¨x1 â x2, y1 â y2â© â¥ (1â Î»Ïf ) â¥x1 â x2â¥2 .When 1â Î»Ïf > 0, Pλf is single-valued and (1â Î»Ïf )â1-Lipschitz.Proof. For i = 1, 2, applying optimality condition to xi â Pλf(yi) yieldsyiâxiÎ» â âf(xi)â yiâxiÎ» + Ïfxi â â(f + Ïf j)(xi), to which the monotonicityof f +Ïf j entails â¨x1âx2, y1âx1Î» â y2âx2Î» +Ïf (x1âx2)â© â¥ 0â â¨x1âx2, y1ây2â© â¥ (1âÎ»Ïf ) â¥x1 â x2â¥2. Apply the Cauchy-Schwarz inequality to see theclaimed Lipschitz continuity.2259.7. A PERSPECTIVE FROM GENERALIZED CONJUGACYTheorem 9.64. Suppose that Ï is Legendre. Let f : X â R be proper,lsc, and Ï-prox-bounded with threshold λf , and let 0 < λ < λf . Supposethat Ï is L-smooth and that λf +Ï is L-strongly convex. Then ââenvÏÎ»f â¦âÏâis convex. In particular, the assumption holds when Ï is L-smooth, f isµ-strongly convex, and λ â¥ µâ1L.Proof. For simplicity, let q = Î»â1(Ïâj) and h = f+q. For Ïh = Î»â1(1âL),h + Ïhj = f + q + Î»â1(1 â L)j = Î»â1(Î»f + Ï â Lj) is convex owing tothe assumption. Moreover, Lemma 9.25 implies that Pλh =âââproxÏÎ»f â¦ âÏâis well-defined with domPλh = domâââproxÏÎ»f â¦ âÏâ = domâÏâ. Hence allconditions of Lemma 9.63 are satisfied, which yields that Pλh is Lipschitzwith modulus Lâ1. Write gλ = ââenvÏÎ»f â¦ âÏâ. Then Lemma 9.25 impliesthat λgλ = Î»eÎ»h+Ïââ j. We claim that eλh is differentiable, which impliesÎ»âgÎ» = Î»âeÎ»h+âÏââ Id = IdâPÎ»h+âÏââ Id = âÏââPÎ»h, where âÏâis Lâ1-strongly monotone. Indeed, h+ Î»â1j = f + Î»â1Ï is strongly convexand in particular strictly convex, justifying the claim thanks to [90, Theorem3.5]. Altogether, for every yi â domâÏâ and xi = PÎ»h(yi),Î»â¨âgÎ»(y1)ââgÎ»(y2), y1 â y2â©=â¨âÏâ(y1)ââÏâ(y2), y1 â y2â© â â¨x1 â x2, y1 â y2â©â¥Lâ1 â¥y1 â y2â¥2 â Lâ1 â¥y1 â y2â¥2 = 0,justifying the desired convexity of gλ.9.7 A perspective from generalized conjugacyThroughout this section, let X and Y be two sets and let Φ : X Ã Y â Rbe a coupling function. We start from defining Î¦-conjugatesâconcepts inoptimal transport; see for instance [39, Section 2.6.2] and [81, Chapter 11].They provide an interesting perspective to interpret the (Bregman) levelproximal subdifferential investigated in this chapter.Definition 9.65. The (left) Φ-conjugate of f : X â R is fΦ : Y â Rdefined byfΦ(y) = supxâX{Î¦(x, y)â f(x)} .Likewise, the (right) Φ-conjugate of g : Y â R is gΦ : Y â R given bygΦ(x) = supyâY{Î¦(x, y)â g(y)} .2269.7. A PERSPECTIVE FROM GENERALIZED CONJUGACYRemark 9.66. Let λ > 0 and let Ï be a Legendre kernel.(i) When Φ(x, y) = â 1λ DÏ(x, y), the left Φ-conjugate becomes the leftBregman Moreau envelope:fΦ(y) = supxâX{Î¦(x, y)â f(x)} = supxâX{â 1λ DÏ(x, y)â f(x)}= â infxâX{f(x) + 1λ DÏ(x, y)}= âââenvÏÎ»f(y);while the right Φ-conjugate satisfies fΦ(x) = âââenvÏÎ»f(x), which is theright Bregman Moreau envelope.(ii) When Φ(x, y) = â 12λ â¥xâ yâ¥2, both the left and right Φ-conjugatesbecome the Moreau envelope:fΦ(y) = fΦ(y) = supxâX{Î¦(x, y)â f(x)} = supxâX{â 12λ â¥xâ yâ¥2 â f(x)}= â infxâX{f(x) + 12λ â¥xâ yâ¥2}= âââenvÏÎ»f(y).(iii) When Φ(x, y) = â¨x, yâ©, both the left and right Φ-conjugates reduce tothe Fenchel conjugate.Definition 9.67. The (left) Φ-biconjugate of f : X â R isfΦΦ = (fΦ)Φ,namelyfΦΦ(x) = supyâY{Î¦(x, y)â fΦ(y)} .Similarly, the (right) Φ-biconjugate of g : Y â R is gΦΦ = (gΦ)Φ, namelygΦΦ(x) = supxâX{Î¦(x, y)â gΦ(x)} .Remark 9.68. Let λ > 0 and let Ï be a Legendre kernel.(i) When Φ(x, y) = â 1λ DÏ(x, y),fΦΦ = (fΦ)Φ = (âââenvÏÎ»f)Î¦ = âââenvÏÎ»(âââenvÏÎ»f) =âââhullÏÎ»f.(ii) When Φ(x, y) = â 12λ â¥xâ yâ¥2, fΦΦ = hλf .(iii) When Φ(x, y) = â¨x, yâ©, fΦΦ = fΦΦ = fââ.2279.7. A PERSPECTIVE FROM GENERALIZED CONJUGACYDefinition 9.69. We say that y¯ â Y is a (left) Φ-subgradient of f : X â Rat x¯ â dom f , denoted by y¯ â âÎ¦f(x¯), if for every x â Xf(x) â¥ f(x¯) + Φ(x, y¯)â Φ(x¯, y¯).Similarly, x¯ â X is a (right) Φ-subgradient of g : Y â R at y¯ â dom g,denoted by x¯ â âÎ¦g(y¯), if for every y â Yg(y) â¥ g(y¯) + Φ(x¯, y)â Φ(x¯, y¯).Remark 9.70. Let λ > 0 and let Ï be a Legendre kernel.(i) When Φ(x, y) = â 1λ DÏ(x, y), y¯ â âÎ¦f(x¯) amounts to for every x â Xf(x) â¥ f(x¯)â 1λ DÏ(x, y¯) + 1λ DÏ(x¯, y¯)= f(x¯)â 1λ (Ï(x)â Ï(x¯)â â¨âÏ(y¯), xâ x¯â©)= f(x¯) + 1Î»â¨âÏ(y¯)ââÏ(x¯), xâ x¯â© â 1λ DÏ(x, x¯),which means 1λ (âÏ(y¯)ââÏ(x¯)) ââââÏ/Î»p f(x¯).Moreover, the x¯ â âÎ¦g(y¯) is equivalent to(ây â Y) g(y) â¥ g(y¯)â 1λ (DÏ(x¯, y)âDÏ(x¯â y¯))= g(y¯) + 1Î»â¨âÏ(y)ââÏ(y¯), y â y¯â© â 1λ DÏ(y¯, y),which amounts to 1Î»(x¯â y¯) ââââÏ/Î»p g(y¯).(ii) When Φ(x, y) = â 12λ â¥xâ yâ¥2, y¯ â âÎ¦f(x¯) is equivalent to for everyx â Xf(x) â¥ f(x¯)â 12λ â¥xâ y¯â¥2 + 12λ â¥x¯â y¯â¥2= f(x¯) + 1Î»â¨y¯ â x¯, xâ x¯â© â 12λ â¥xâ x¯â¥2 ,being equivalent to 1λ(y¯ â x¯) â âÎ»p f(x¯).(iii) When Φ(x, y) = â¨x, yâ©, y¯ â âÎ¦f(x¯) is equivalent to(âx â X ) f(x) â¥ f(x¯) + â¨xâ x¯, y¯â© â y¯ â âFf(x¯).The following complements [50, Lemma 2.6], where the authors onlyproved the 9.71(i) â 9.71(ii) â 9.71(iii).2289.8. CONCLUSIONTheorem 9.71 (Fenchel Φ-duality). Let f : X â R be proper. For anyx¯ â dom f and y¯ â Y, the following are equivalent:(i) y¯ â âÎ¦f(x¯);(ii) x¯ â âÎ¦fÎ¦(y¯) â© dom âÎ¦f ;(iii) f(x¯) = fΦΦ(x¯) and y¯ â âÎ¦fÎ¦Î¦(x¯).Proof. We prove only the direction 9.71(iii) â 9.71(i) and refer readers to[50, Lemma 2.6] for the rest to minimize the repetition of known arguments.Pick x â dom f . Then the definition of fΦ implies that (ây â Y) fΦ(y) â¥Î¦(x, y)â f(x), entailing f(x) â¥ fΦΦ(x). Hencef(x) â¥ fΦΦ(x) â¥ fΦΦ(x¯) + Φ(x, y¯)â Φ(x¯, y¯) = f(x¯) + Φ(x, y¯)â Φ(x¯, y¯),where the second inequality and the equality hold owing to 9.71(iii), imply-ing the desired 9.71(i).As we specify the coupling function Φ, previous results on Bregman levelproximal subdifferential will be recovered; see Theorem 9.37.Corollary 9.72. Let Ï : X â R be a 1-coercive Legendre kernel and letX = domÏ and Y = int domÏ. Then the following are equivalent for aproper, lsc, and Ï-prox-bounded function f : X â R and λ â (0, λf ):(i) u¯ â âââÏ/Î»p f(x¯).(ii) 1Î»(x¯â y¯) ââââÏ/Î»p (âââenvÏÎ»f)(y¯) and x¯ â domâââÏ/Î»p f , where y¯ = âÏâ(Î»u¯+âÏ(x¯)).(iii) u¯ â ââââhullÏÎ»f(x¯) andâââhullÏÎ»f(x¯) = f(x¯).Proof. Apply Theorem 9.71 to the coupling function Φ(x, y) = â 1λ DÏ(x, y),and refer to Remarks 9.66, 9.68 and 9.70 for specifications of fΦ, fΦΦ, andâÎ¦f, âÎ¦fÎ¦, respectively.9.8 ConclusionWe proposed left and right Bregman proximal subdifferentials and investi-gated their rich connections to various notions in variational analysis. No-tably, Bregman proximal operators are always (warped) resolvents of Breg-man level proximal subdifferentials. This pleasant representation leads new2299.8. CONCLUSIONinsights into the correspondences between properties of the Bregman proxi-mal operators and of subdifferential operators, include hypomonotonicity,convex-valuedness, and more. We also study the Bregman level proxi-mal subdifferentials systematically by characterizing its existence, single-valuedness, and coincidence results.230Chapter 10Conclusion and future work10.1 ConclusionMathematical foundations in convex analysis, monotone operator theory,and variational analysis are presented in Chapter 2.In Chapters 3 to 5, we extended the forward-reflected-backward split-ting method in convex optimization and its variations to nonconvex setting.Our analysis hinges on the Kurdyka- Lojasiewicz propertyâa celebrated reg-ularity in variational analysis. We showed that, among other things, thesenonconvex extensions converge globally to a stationary point of a nonconvexobjective, both with and without Lipschitz smoothness.Subdifferentials and proximal operators are central objects in variationalanalysis and optimization.In Chapters 6 and 7, we investigated the level proximal subdifferen-tial systematically. Proximal operators are always resolvents of level prox-imal subdifferentials. We presented various fundamental properties of thelevel proximal subdifferential, characterized its existence, single-valuedness,and investigated its convex integration and connections to local Lipschitzsmoothness.In Chapter 8, we presented new characterizations of variational andvariational strong convexity, including various nonexpansive-type proper-ties of the proximal operators and relative monotonicity of the level proxi-mal subdifferentials. We also explored calculus rules of variational convexityand explored its algorithmic applications on classical convex optimizationalgorithms. In particular, variational convexity, which together with theKurdyka- Lojasiewicz property, guarantees that proximal gradient methodconverges globally to a local minimizer.In Chapter 9, we introduced and studied the left and right Bregman levelproximal subdifferentials. Their (warped) resolvents coincide with Breg-man proximal operators under a standard range assumption. We investi-gated their properties and characterizations systematically, including theirexistence, single-valuedness, and connections to relative smoothness and toBregman proximal operators, generalizing respective results on level proxi-23110.2. FUTURE WORKmal subdifferentials. New characterizations of Bregman proximal operatorsare obtained.10.2 Future workIn this section, we list several research questions for future work.â¢ Optimistic Gradient Method (OGM) is a celebrated algorithm forsolving minmax problem in modern machine learning applications.It is thus tempting to investigate the convergence of this method innonconvex-concave setting. This task will be facilitated by our work onthe forward-reflected-backward splitting method investigated in Chap-ters 3 to 5, which is nothing else but a special case of OGM.â¢ Consider the proximal operator Pλf : Rn â Rn of a possibly non-convex function. Chapter 8 has revealed that Pλf is locally Lipschitzcontinuous with modulus Ï > 0 for Ï â¤ 1 if and only if the functionf satisfies variational convexity when Ï = 1 and variational strongconvexity when Ï < 1; see Theorem 8.19 and Theorem 8.22. It istherefore tempting to complete the big picture by characterizing thelocal Lipschitz continuity of Pλf with modulus Ï > 1.â¢ Despite the pleasant features and potential in algorithmic applicationsof variational convexity investigated in Chapter 8, the verification ofthis property needs to be conducted on a case-by-case basis. There-fore it is tempting to see which class of functions satisfy variationalconvexity, and thereby facilitating a broader application of variationalconvexity.232Bibliography[1] M. Ahookhosh, A. Themelis, and P. Patrinos. A Bregman forward-backward linesearch algorithm for nonconvex composite optimization:superlinear convergence to nonisolated local minima. SIAM Journal onOptimization, 31(1):653â685, 2021.[2] H. Attouch and J. Bolte. On the convergence of the proximal algorithmfor nonsmooth functions involving analytic features. Mathematical Pro-gramming, 116(1):5â16, 2009.[3] H. Attouch, J. Bolte, P. Redont, and A. Soubeyran. Proximal alternat-ing minimization and projection methods for nonconvex problems: anapproach based on the Kurdyka- Lojasiewicz inequality. Mathematicsof Operations Research, 35(2):438â457, 2010.[4] H. Attouch, J. Bolte, and B. F. Svaiter. Convergence of descent methodsfor semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized Gauss-Seidel methods. Mathemat-ical Programming, 137(1):91â129, 2013.[5] M. Bachir, A. Daniilidis, and J.-P. Penot. Lower subdifferentiabilityand integration. Set-Valued Analysis, 10(1):89â108, 2002.[6] S. Banert and R. I. Bot, . A general double-proximal gradient algo-rithm for d.c. programming. Mathematical Programming, 178(1):301â326, 2019.[7] H. H. Bauschke, J. Bolte, and M. Teboulle. A descent lemma beyondLipschitz gradient continuity: first-order methods revisited and appli-cations. Mathematics of Operations Research, 42(2):330â348, 2017.[8] H. H. Bauschke, J. M. Borwein, and P. L. Combettes. Essential smooth-ness, essential strict convexity, and Legendre functions in Banachspaces. Communications in Contemporary Mathematics, 3(04):615â647,2001.233Bibliography[9] H. H. Bauschke, J. M. Borwein, and P. L. Combettes. Bregman mono-tone optimization algorithms. SIAM Journal on Control and Optimiza-tion, 42(2):596â636, 2003.[10] H. H. Bauschke and P. L. Combettes. Iterating Bregman retractions.SIAM Journal on Optimization, 13(4):1159â1173, 2003.[11] H. H. Bauschke and P. L. Combettes. Convex Analysis and MonotoneOperator Theory in Hilbert Spaces. Springer, Cham, 2017. http://dx.doi.org/10.1007/978-1-4419-9467-7.[12] H. H. Bauschke, M. N. Dao, and S. B. Lindstrom. Regularizingwith Bregman-Moreau envelopes. SIAM Journal on Optimization,28(4):3208â3228, 2018.[13] H. H. Bauschke, X. Wang, J. Ye, and X. Yuan. Bregman distances andChebyshev sets. Journal of Approximation Theory, 159(1):3â25, 2009.[14] A. Beck. First-order Methods in Optimization. SIAM, 2017.[15] A. Beck and M. Teboulle. Mirror descent and nonlinear projected sub-gradient methods for convex optimization. Operations Research Letters,31(3):167â175, 2003.[16] J. Benoist and A. Daniilidis. Integration of Fenchel subdifferentials ofepi-pointed functions. SIAM Journal on Optimization, 12(3):575â582,2002.[17] J. Benoist and J.-B. Hiriart-Urruty. What is the subdifferential of theclosed convex hull of a function? SIAM Journal on MathematicalAnalysis, 27(6):1661â1679, 1996.[18] F. Bernard and L. Thibault. Prox-regular functions in Hilbert spaces.Journal of Mathematical Analysis and Applications, 303(1):1â14, 2005.[19] J. Bolte, A. Daniilidis, and A. Lewis. The Lojasiewicz inequality fornonsmooth subanalytic functions with applications to subgradient dy-namical systems. SIAM Journal on Optimization, 17(4):1205â1223,2007.[20] J. Bolte, A. Daniilidis, A. Lewis, and M. Shiota. Clarke subgradients ofstratifiable functions. SIAM Journal on Optimization, 18(2):556â572,2007.234Bibliography[21] J. Bolte, A. Daniilidis, O. Ley, and L. Mazet. Characterizations of Lojasiewicz inequalities: subgradient flows, talweg, convexity. Transac-tions of the American Mathematical Society, 362(6):3319â3363, 2010.[22] J. Bolte, S. Sabach, and M. Teboulle. Proximal alternating linearizedminimization for nonconvex and nonsmooth problems. MathematicalProgramming, 146(1):459â494, 2014.[23] J. Bolte, S. Sabach, M. Teboulle, and Y. Vaisbourd. First order meth-ods beyond convexity and Lipschitz gradient continuity with applica-tions to quadratic inverse problems. SIAM Journal on Optimization,28(3):2131â2151, 2018.[24] R. I. Bot¸ and E. R. Csetnek. An inertial Tsengâs type proximal algo-rithm for nonsmooth and nonconvex optimization problems. Journalof Optimization Theory and Applications, 171(2):600â616, 2016.[25] R. I. Bot¸, E. R. Csetnek, and S. C. La´szlo´. An inertial forward-backwardalgorithm for the minimization of the sum of two nonconvex functions.EURO Journal on Computational Optimization, 4(1):3â25, 2016.[26] R. I. Bot¸, E. R. Csetnek, and D.-K. Nguyen. A proximal minimizationalgorithm for structured nonconvex and nonsmooth problems. SIAMJournal on Optimization, 29(2):1300â1328, 2019.[27] R. I. Bot¸ and D.-K. Nguyen. The proximal alternating direction methodof multipliers in the nonconvex setting: convergence analysis and rates.Mathematics of Operations Research, 45(2):682â712, 2020.[28] S. P. Boyd and L. Vandenberghe. Convex Optimization. Cambridgeuniversity press, 2004.[29] M. N. Bui and P. L. Combettes. Warped proximal iterations for mono-tone inclusions. Journal of Mathematical Analysis and Applications,491(1):124315, 2020.[30] R. Burden, D. Faires, and A. Burden. Numerical Analysis. CengageLearning, 2014.[31] C. Chen, T. K. Pong, L. Tan, and L. Zeng. A difference-of-convexapproach for split feasibility with applications to matrix factorizationsand outlier detection. Journal of Global Optimization, 78(1):107â136,2020.235Bibliography[32] G. Chen and M. Teboulle. Convergence analysis of a proximal-likeminimization algorithm using Bregman functions. SIAM Journal onOptimization, 3(3):538â543, 1993.[33] J. Chen, X. Wang, and C. Planiden. A proximal average for prox-bounded functions. SIAM Journal on Optimization, 30(2):1366â1390,2020.[34] P. L. Combettes and J.-C. Pesquet. Proximal splitting methods in signalprocessing. Fixed-point algorithms for inverse problems in science andengineering, pages 185â212, 2011.[35] P. L. Combettes and V. R. Wajs. Signal recovery by proximal forward-backward splitting. Multiscale Modeling & Simulation, 4(4):1168â1200,2005.[36] R. Dragomir, A. Taylor, A. dâAspremont, and J. Bolte. Optimal com-plexity and certification of Bregman first-order methods. MathematicalProgramming, 194(1):41â83, 2022.[37] R.-A. Dragomir. Me´thodes de gradient de Bregman pour proble`mes a`re´gularite´ relative. PhD thesis, Universite´ Toulouse, 2021.[38] R.-A. Dragomir, A. dâAspremont, and J. Bolte. Quartic first-ordermethods for low-rank minimization. Journal of Optimization Theoryand Applications, 189:341â363, 2021.[39] A. Figalli and F. Glaudo. An Invitation to Optimal Transport, Wasser-stein Distances, and Gradient Flows. European Mathematical Society,2023.[40] J.-B. Hiriart-Urruty and C. Lemare´chal. Convex Analysis and Mini-mization Algorithms I, volume 305 of Grundlehren Der MathematischenWissenschaften. Springer, 1993.[41] J.-B. Hiriart-Urruty and C. Lemare´chal. Convex Analysis andMinimization Algorithms II: Advanced Theory and Bundle Methods.Grundlehren der mathematischen Wissenschaften. Springer Berlin Hei-delberg, 1996.[42] J.-B. Hiriart-Urruty and C. Lemare´chal. Fundamentals of Convex Anal-ysis. Springer Science & Business Media, 2004.236Bibliography[43] C. Homann. Phase Retrieval Problems in X-ray Physics. Go¨ttingenUniversity Press, 2015.[44] A. Jourani, L. Thibault, and D. Zagrodny. Differential properties of theMoreau envelope. Journal of Functional Analysis, 266(3):1185â1237,2014.[45] C. Kan and W. Song. The Moreau envelope function and proximalmapping in the sense of the Bregman distance. Nonlinear Analysis:Theory, Methods & Applications, 75(3):1385 â 1399, 2012.[46] P. D. Khanh, V. V. H. Khoa, B. S. Mordukhovich, and V. T. Phat. Vari-ational and strong variational convexity in infinite-dimensional varia-tional analysis. SIAM Journal on Optimization, 34(3):2756â2787, 2024.[47] P. D. Khanh, B. S. Mordukhovich, and V. T. Phat. Variational con-vexity of functions and variational sufficiency in optimization. SIAMJournal on Optimization, 33(2):1121â1158, 2023.[48] K. Kurdyka. On gradients of functions definable in o-minimal struc-tures. Annales de lâinstitut Fourier, 48(3):769â783, 1998.[49] E. Laude, P. Ochs, and D. Cremers. Bregman proximal mappings andBregman-Moreau envelopes under relative prox-regularity. Journal ofOptimization Theory and Applications, 184(3):724â761, 2020.[50] E. Laude, A. Themelis, and P. Patrinos. Dualities for non-Euclideansmoothness and strong convexity under the light of generalized conju-gacy. SIAM Journal on Optimization, 33(4):2721â2749, 2023.[51] G. Li, T. Liu, and T. K. Pong. Peaceman-Rachford splitting for a classof nonconvex optimization problems. Computational Optimization andApplications, 68(2):407â436, 2017.[52] G. Li and T. Pong. Global convergence of splitting methods fornonconvex composite optimization. SIAM Journal on Optimization,25(4):2434â2460, 2015.[53] G. Li and T. Pong. Douglas-Rachford splitting for nonconvex optimiza-tion with application to nonconvex feasibility problems. MathematicalProgramming, 159(1):371â401, Sep 2016.237Bibliography[54] G. Li and T. K. Pong. Calculus of the exponent of Kurdyka- Lojasiewiczinequality and its applications to linear convergence of first-order meth-ods. Foundations of Computational Mathematics, 18(5):1199â1232,2018.[55] Y. Liu and W. Yin. An envelope for Davis-Yin splitting and strictsaddle-point avoidance. Journal of Optimization Theory and Applica-tions, 181(2):567â587, 2019.[56] S. Lojasiewicz. A topological property of real analytic subsets. Partialdifferential equations, 117(87-89), 1963.[57] H. Lu, R. Freund, and Y. Nesterov. Relatively smooth convex opti-mization by first-order methods, and applications. SIAM Journal onOptimization, 28(1):333â354, 2018.[58] Z. Lu and Y. Zhang. Sparse approximation via penalty decompositionmethods. SIAM Journal on Optimization, 23(4):2448â2478, 2013.[59] H. Luo, X. Wang, Z. Wang, and X. Yang. Level proximal subdifferen-tial, variational convexity, and pointwise Lipschitz smoothness. arXivpreprint arXiv:2406.00648, 2024.[60] H. Luo, X. Wang, and X. Yang. Various notions of nonexpansivenesscoincide for proximal mappings of functions. SIAM Journal on Opti-mization, 34(1):642â653, March 2024.[61] R. Luss and M. Teboulle. Conditional gradient algorithms for rank-one matrix approximations with a sparsity constraint. SIAM Review,55(1):65â98, 2013.[62] Y. Malitsky and M. K. Tam. A forward-backward splitting method formonotone inclusions without cocoercivity. SIAM Journal on Optimiza-tion, 30(2):1451â1472, 2020.[63] J. Moreau. Fonctionnelles convexes. Se´minaire Jean Leray, (2):1â108,1966.[64] J.-J. Moreau. Proximite´ et dualite´ dans un espace hilbertien. Bulletinde la Socie´te´ Mathe´matique de France, 93:273â299, 1965.[65] A. S. Nemirovskij and D. B. Yudin. Problem Complexity and MethodEfficiency in Optimization. Wiley-Interscience, 1983.238Bibliography[66] Y. Nesterov. A method for solving the convex programming prob-lem with convergence rate O(1/k2). Doklady Akademii Nauk SSSR,269:543â547, 1983.[67] Y. Nesterov. Implementable tensor methods in unconstrained convexoptimization. Mathematical Programming, 186:157â183, 2021.[68] P. Ochs. Local convergence of the heavy-ball method and iPiano fornon-convex optimization. Journal of Optimization Theory and Appli-cations, 177(1):153â180, 2018.[69] P. Ochs. Unifying abstract inexact convergence theorems and blockcoordinate variable metric iPiano. SIAM Journal on Optimization,29(1):541â570, 2019.[70] P. Ochs, Y. Chen, T. Brox, and T. Pock. iPiano: inertial proximal algo-rithm for nonconvex optimization. SIAM Journal on Imaging Sciences,7(2):1388â1419, 2014.[71] N. Parikh and S. Boyd. Proximal algorithms. Foundations and Trendsin Optimization, 1(3):127â239, 2014.[72] J.-P. Penot. Proximal mappings. Journal of Approximation Theory,94(2):203â221, 1998.[73] R. A. Poliquin. Integration of subdifferentials of nonconvex functions.Nonlinear Analysis: Theory, Methods & Applications, 17(4):385â398,1991.[74] R. A. Poliquin and R. T. Rockafellar. Prox-regular functions in vari-ational analysis. Transactions of the American Mathematical Society,348(5):1805â1838, 1996.[75] B. Polyak. Gradient methods for the minimization of functionals (inrussian). Zhurnal Vychislitelânoi Matematiki i Matematicheskoi Fiziki,3(4):643â653, 1963.[76] R. T. Rockafellar. Convex Analysis. Princeton University Press, 1970.[77] R. T. Rockafellar. Monotone operators and the proximal point algo-rithm. SIAM Journal on Control and Optimization, 14(5):877â898,1976.239Bibliography[78] R. T. Rockafellar. Variational convexity and the local monotonicity ofsubgradient mappings. Vietnam Journal of Mathematics, 47(3):547â561, 2019.[79] R. T. Rockafellar. Characterizing firm nonexpansiveness of prox map-pings both locally and globally. Journal of Nonlinear and Convex Anal-ysis, 22(5):887â899, 2021.[80] R. T. Rockafellar. Augmented Lagrangians and hidden convexity insufficient conditions for local optimality. Mathematical Programming,198(1):159â194, 2023.[81] R. T. Rockafellar and R. J. B. Wets. Variational Analysis, volume 317.Springer, Berlin, 1998.[82] E. K. Ryu and W. Yin. Large-scale Convex Optimization: Algorithms &Analyses via Monotone Operators. Cambridge University Press, 2022.[83] L. Stella, A. Themelis, and P. Patrinos. Newton-type alternating min-imization algorithm for convex optimization. IEEE Transactions onAutomatic Control, 2018.[84] M. Teboulle. A simplified view of first order methods for optimization.Mathematical Programming, 170(1):67â96, 2018.[85] A. Themelis and P. Patrinos. Douglas-Rachford splitting and ADMMfor nonconvex optimization: tight convergence results. SIAM Journalon Optimization, 30(1):149â181, 2020.[86] A. Themelis, L. Stella, and P. Patrinos. Forward-backward envelope forthe sum of two nonconvex functions: further properties and nonmono-tone linesearch algorithms. SIAM Journal on Optimization, 28(3):2274â2303, 2018.[87] A. Themelis, L. Stella, and P. Patrinos. Douglas-Rachford splittingand ADMM for nonconvex optimization: accelerated and Newton-typealgorithms. Computational Optimization and Applications, 82:395â440,2022.[88] P. Tseng. A modified forward-backward splitting method for maxi-mal monotone mappings. SIAM Journal on Control and Optimization,38(2):431â446, 2000.240Bibliography[89] S. Wang, C. Ding, Y. Zhang, and X. Zhao. Strong variational sufficiencyfor nonlinear semidefinite programming and its implications. SIAMJournal on Optimization, 33(4):2988â3011, 2023.[90] X. Wang. On Chebyshev functions and Klee functions. Journal ofMathematical Analysis and Applications, 368(1):293â310, 2010.[91] X. Wang and H. H. Bauschke. The Bregman proximal average. SIAMJournal on Optimization, 32(2):1379â1401, 2022.[92] X. Wang and Z. Wang. Malitsky-Tam forward-reflected-backward split-ting method for nonconvex minimization problems. Computational Op-timization and Applications, 82(2):441â463, 2022.[93] X. Wang and Z. Wang. The exact modulus of the generalized concaveKurdyka- Lojasiewicz property. Mathematics of Operations Research,47(4):2765â2783, 2022.[94] X. Wang and Z. Wang. A Bregman inertial forward-reflected-backwardmethod for nonconvex minimization. Journal of Global Optimization,89(2):327â354, 2024.[95] X. Wang and Z. Wang. Every proximal mapping is a resolvent of levelproximal subdifferential. Optimization Letters, 18(5):1237â1252, 2024.[96] Z. Wang, A. Themelis, H. Ou, and X. Wang. A mirror inertial forward-reflected-backward splitting: convergence analysis beyond convexityand Lipschitz smoothness. Journal of Optimization Theory and Ap-plications, 203:1127â1159, 2024.[97] Z. Wang, A. Themelis, and X. Wang. The Bregman level proximalsubdifferential and new chacarizations on Bregman proximal operators.work in progress, 2024.241
Thesis/Dissertation
eng
Mathematics
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Wang, Xianfu
Regularities in variational analysis and their applications in nonconvex optimization
Text