Finding stationary points on bounded-rank matrices: a geometric hurdle and a smooth remedy

We consider the problem of provably finding a stationary point of a smooth function to be minimized on the variety of bounded-rank matrices. This turns out to be unexpectedly delicate. We trace the difficulty back to a geometric obstacle: On a nonsmooth set, there may be sequences of points along which standard measures of stationarity tend to zero, but whose limit points are not stationary. We name such events apocalypses, as they can cause optimization algorithms to converge to non-stationary points. We illustrate this explicitly for an existing optimization algorithm on bounded-rank matrices. To provably find stationary points, we modify a trust-region method on a standard smooth parameterization of the variety. The method relies on the known fact that second-order stationary points on the parameter space map to stationary points on the variety. Our geometric observations and proposed algorithm generalize beyond bounded-rank matrices. We give a geometric characterization of apocalypses on general constraint sets, which implies that Clarke-regular sets do not admit apocalypses. Such sets include smooth manifolds, manifolds with boundaries, and convex sets. Our trust-region method supports parameterization by any complete Riemannian manifold.

An Accelerated First-Order Method for Non-convex Optimization on Manifolds

Nicolas Boumal, Christopher Arnold Criscitiello

We describe the first gradient methods on Riemannian manifolds to achieve accelerated rates in the non-convex case. Under Lipschitz assumptions on the Riemannian gradient and Hessian of the cost function, these methods find approximate first-order critical points faster than regular gradient descent. A randomized version also finds approximate second-order critical points. Both the algorithms and their analyses build extensively on existing work in the Euclidean case. The basic operation consists in running the Euclidean accelerated gradient descent method (appropriately safe-guarded against non-convexity) in the current tangent space, then moving back to the manifold and repeating. This requires lifting the cost function from the manifold to the tangent space, which can be done for example through the Riemannian exponential map. For this approach to succeed, the lifted cost function (called the pullback) must retain certain Lipschitz properties. As a contribution of independent interest, we prove precise claims to that effect, with explicit constants. Those claims are affected by the Riemannian curvature of the manifold, which in turn affects the worst-case complexity bounds for our optimization algorithms.

SPRINGER2022

Trust-region methods based on radial basis functions with application to biomedical imaging

Rodrigue Oeuvray

We have developed a new derivative-free algorithm based on Radial Basis Functions (RBFs). Derivative-free optimization is an active field of research and several algorithms have been proposed recently. Problems of this nature in the industrial setting are quite frequent. The reason is that in a number of applications the optimization process contains simulation packages which are treated as black boxes. The development of our own algorithm was originally motivated by an application in biomedical imaging: the medical image registration problem. The particular characteristics of this problem have incited us to develop a new optimization algorithm based on trust-region methods. However it has been designed to be generic and to be applied to a wide range of problems. The main originality of our approach is the use of RBFs to build the models. In particular we have adapted the existing theory based on quadratic models to our own models and developed new procedures especially designed for models based on RBFs. We have tested our algorithm called BOOSTERS against state-of-the-art methods (UOBYQA, NEWUOA, DFO). On the medical image registration problem, BOOSTERS appears to be the method of choice. The tests on problems from the CUTEr collection show that BOOSTERS is comparable to, but not better than other methods on small problems (size 2-20). It is performing very well for medium size problems (20-80). Moreover, it is able to solve problems of dimension 200, which is considered very large in derivative-free optimization. We have also developed a new class of algorithms combining the robustness of derivative-free algorithms with the faster rate of convergence characterizing Newtonlike-methods. In fact, they define a new class of algorithms lying between derivative-free optimization and quasi-Newton methods. These algorithms are built on the skeleton of our derivative-free algorithm but they can incorporate the gradient when it is available. They can be interpreted as a way of doping derivative-free algorithms with derivatives. If the derivatives are available at each iteration, then our method can be seen as an alternative to quasi-Newton methods. At the opposite, if the derivatives are never evaluated, then the algorithm is totally similar to BOOSTERS. It is a very interesting alternative to existing methods for problems whose objective function is expensive to evaluate and when the derivatives are not available. In this situation, the gradient can be approximated by finite differences and its costs corresponds to n additional function evaluations assuming that Rn is the domain of definition of the objective function. We have compared our method with CFSQP and BTRA, two gradient-based algorithms, and the results show that our doped method performs best. We have also a theoretical analysis of the medical image registration problem based on maximization of mutual information. Most of the current research in this field is concentrated on registration based on nonlinear image transformation. However, little attention has been paid to the theoretical properties of the optimization problem. In our analysis, we focus on the continuity and the differentiability of the objective function. We show in particular that performing a registration without extension of the reference image may lead to discontinuities in the objective function. But we demonstrate that, under some mild assumptions, the function is differentiable almost everywhere. Our analysis is important from an optimization point of view and conditions the choice of a solver. The usual practice is to use generic optimization packages without worrying about the differentiability of the objective function. But the use of gradient-based methods when the objective function is not differentiable may result in poor performance or even in absence of convergence. One of our objectives with this analysis is also that practitioners become aware of these problems and to propose them new algorithms having a potential interest for their applications.

EPFL2005

Optimal control of geodesics in Riemannian manifolds

Roland Rozsnyo

The aim of this dissertation is to solve numerically the following problem, denoted by P : given a Riemannian manifold and two points a and b belonging to that manifold, find a tangent vector T at a, such that expa(T) = b, assuming that T exists. This problem is set under an optimal control formulation, which requires the definition of an objective function and a space of control, the choice of a method for the calculation of the descent direction of that function in the space of control and the use of an optimization algorithm to find its minimum, which corresponds to the solution of the original problem by construction. Several techniques are necessary to be put together, coming from the fields of geometry, numerical analysis and optimization. The first part will concern a recalling of the mathematical context in which this formulation takes place. The general principles of optimal control will also be given. In the second part, we will present an intrinsic formulation of the optimal control problem associated to P, based on Jacobi fields, which will play the role of the so called adjoint state. This derivation leads to necessary optimality conditions. We will illustrate explicitly that formulation by treating the specific case of Riemannian manifolds with constant sectional curvature. Then, we will derive the optimal control problem in coordinates, not only to check the intrinsic formulation but also to reveal how it is hidden behind the expressions in coordinates. Their use reveals some quantities whose interpretation may be given this way. Moreover, we will show that more possibilities exist to chose the cost function and the control space in coordinates. In a second step, an alternative approach will consider the Hamiltonian formulation of geodesics. This is an incursion into symplectic geometry. We will then reformulate the Riemannian optimal control problem in its Hamiltonian version. In the third part, the numerical methods used for solving P will be presented. The discretization imposes the definition of new discrete optimal control problems. The technique shows that the discrete adjoint state equation strongly depends on the numerical scheme used to solve the direct problem. We will give a collection of numerical computations in the specific case of parametric piece of surfaces, where the surface can be defined by one or several Bézier patches, each one corresponding to a chart, which is representative of a Riemannian manifold. We will compare the different numerical approaches. The last but one part will be devoted to the interesting application of wooden roof building, where the structure is made of wooden boards, with geodesic trajectories on the designed piece of surface. The Geos (Geodesic solver) software has been developed for that purpose. After having introduced some specific numerical methods used in the code, we present the Geos application interface (AI) developed as a tool for the conception of such a roof. We then show an existing wooden structure built according to that mean. Finally, we will summarize the results of our research and discuss future possible prospects.