Êtes-vous un étudiant de l'EPFL à la recherche d'un projet de semestre?

Travaillez avec nous sur des projets en science des données et en visualisation, et déployez votre projet sous forme d'application sur GraphSearch.

Découvrez-en plus sur les applications Graph.

Graph
Search

fr|en

Se Connecter

Publication

Convergence without Convexity: Sampling, Optimization, and Games

Ya-Ping Hsieh
EPFL, 2020
Thèse EPFL

Résumé

Many important problems in contemporary machine learning involve solving highly non- convex problems in sampling, optimization, or games. The absence of convexity poses significant challenges to convergence analysis of most training algorithms, and in some cases (such as min-max games) it is not even known whether common training algorithms converge or not. In this thesis, we aim to partially bridge the gap by

Proposing a new sampling framework to transform non-convex problems in to convex ones.
Characterizing the convergent sets of a wide family of popular algorithms for min-max optimization.
Devising provably convergent algorithms for finding mixed Nash Equilibria of infinite-dimensional bi-affine games.

Our theory has several important implications. First, we resolve a decade-old open problem in Bayesian learning via our non-convex sampling framework. Second, our algorithms for bi-affine games apply to the formidably difficult training of generative adversarial networks and robust reinforcement learning, and on both examples we demonstrate promising empirical performance. Finally, our results on min-max optimization lead to a series of negative results for state-of-the-art algorithms, suggesting that one requires fundamentally new tools to advance the theory.

Official source

https://infoscience.epfl.ch/record/280224?ln=fr

À propos de ce résultat

Cette page est générée automatiquement et peut contenir des informations qui ne sont pas correctes, complètes, à jour ou pertinentes par rapport à votre recherche. Il en va de même pour toutes les autres pages de ce site. Veillez à vérifier les informations auprès des sources officielles de l'EPFL.

Concepts associés

Chargement

Publications associées

Chargement

Mean-Field methods for Structured Deep-Learning in Computer Vision

Pierre Bruno Baqué

In recent years, Machine Learning based Computer Vision techniques made impressive progress. These algorithms proved particularly efficient for image classification or detection of isolated objects. From a probabilistic perspective, these methods can predict marginals, over single or multiple variables, independently, with high accuracy. However, in many tasks of practical interest, we need to predict jointly several correlated variables. Practical applications include people detection in crowded scenes, image segmentation, surface reconstruction, 3D pose estimation and others. A large part of the research effort in today's computer-vision community aims at finding task-specific solutions to these problems, while leveraging the power of Deep-Learning based classifiers. In this thesis, we present our journey towards a generic and practical solution based on mean-field (MF) inference. Mean-field is a Statistical Physics-inspired method which has long been used in Computer-Vision as a variational approximation to posterior distributions over complex Conditional Random Fields. Standard mean-field optimization is based on coordinate descent and in many situations can be impractical. We therefore propose a novel proximal gradient-based approach to optimizing the variational objective. It is naturally parallelizable and easy to implement. We prove its convergence, and then demonstrate that, in practice, it yields faster convergence and often finds better optima than more traditional mean-field optimization techniques. Then, we show that we can replace the fully factorized distribution of mean-field by a weighted mixture of such distributions, that similarly minimizes the KL-Divergence to the true posterior. Our extension of the clamping method proposed in previous works allows us to both produce a more descriptive approximation of the true posterior and, inspired by the diverse MAP paradigms, fit a mixture of mean-field approximations. We demonstrate that this positively impacts real-world algorithms that initially relied on mean-fields. One of the important properties of the mean-field inference algorithms is that the closed-form updates are fully differentiable operations. This naturally allows to do parameter learning by simply unrolling multiple iterations of the updates, the so-called back-mean-field algorithm. We derive a novel and efficient structured learning method for multi-modal posterior distribution based on the Multi-Modal Mean-Field approximation, which can be seamlessly combined to modern gradient-based learning methods such as CNNs. Finally, we explore in more details the specific problem of structured learning and prediction for multiple-people detection in crowded scenes. We then present a mean-field based structured deep-learning detection algorithm that provides state of the art results on this dataset.

EPFL2018

Chargement

Fréchet means in Wasserstein space

Yoav Zemel

This work studies the problem of statistical inference for Fréchet means in the Wasserstein space of measures on Euclidean spaces,

\mathcal W_2 ( \mathbb R^d )

. This question arises naturally from the problem of separating amplitude and phase variation in point processes, analogous to a well-known problem in functional data analysis. We formulate the point process version of the problem, show that it is canonically equivalent to that of estimating Fréchet means in

\mathcal W_2 ( \mathbb R d )

, and carry out estimation by means of

M

-estimation. This approach allows to achieve consistency in a genuinely nonparametric framework, even in a sparse sampling regime. For Cox processes on the real line, consistency is supplemented by convergence rates and, in the dense sampling regime,

\sqrt n

-consistency and a central limit theorem. Computation of the Fréchet mean is challenging when the processes are multivariate, in which case our Fréchet mean estimator is only defined implicitly as the minimiser of an optimisation problem. To overcome this difficulty, we propose a steepest descent algorithm that approximates the minimiser, and show that it converges to a local minimum. Our techniques are specific to the Wasserstein space, because Hessian-type arguments that are commonly used for similar convergence proofs do not apply to that space. In addition, we discuss similarities with generalised Procrustes analysis. The key advantage of the algorithm is that it requires only the solution of pairwise transportation problems. The results in the preceding paragraphs require properties of Fréchet means in

\mathcal W_2 ( \mathbb R ^d )

whose theory is developed, supplemented by some new results. We present the tangent bundle and exploit its relation to optimal maps in order to derive differentiability properties of the associated Fréchet functional, obtaining a characterisation of Karcher means. Additionally, we establish a new optimality criterion for local minima and prove a new stability result for the optimal maps that, enhanced with the established consistency of the Fréchet mean estimator, yields consistency of the optimal transportation maps.

EPFL2017

Chargement

In the last decade, deep neural networks have achieved tremendous success in many fields of machine learning.However, they are shown vulnerable against adversarial attacks: well-designed, yet imperceptible, perturbations can make the state-of-the-art deep neural networks output incorrect results.Understanding adversarial attacks and designing algorithms to make deep neural networks robust against these attacks are key steps to building reliable artificial intelligence in real-life applications.In this thesis, we will first formulate the robust learning problem.Based on the notations of empirical robustness and verified robustness, we design new algorithms to achieve both of these types of robustness.Specifically, we investigate the robust learning problem from the optimization perspectives.Compared with classic empirical risk minimization, we show the slow convergence and large generalization gap in robust learning.Our theoretical and numerical analysis indicates that these challenges arise, respectively, from non-smooth loss landscapes and model's fitting hard adversarial instances.Our insights shed some light on designing algorithms for mitigating these challenges.Robust learning has other challenges, such as large model capacity requirements and high computational complexity.To solve the model capacity issue, we combine robust learning with model compression.We design an algorithm to obtain sparse and binary neural networks and make it robust.To decrease the computational complexity, we accelerate the existing adversarial training algorithm and preserve its performance stability.In addition to making models robust, our research provides other benefits.Our methods demonstrate that robust models, compared with non-robust ones, usually utilize input features in a way more similar to the way human beings use them, hence the robust models are more interpretable.To obtain verified robustness, our methods indicate the geometric similarity of the decision boundaries near data points.Our approaches towards reliable artificial intelligence can not only render deep neural networks more robust in safety-critical applications but also make us better aware of how they work.

EPFL2022

Chargement

Mean-Field methods for Structured Deep-Learning in Computer Vision

Pierre Bruno Baqué

EPFL2018

EPFL2022

Fréchet means in Wasserstein space

Yoav Zemel

This work studies the problem of statistical inference for Fréchet means in the Wasserstein space of measures on Euclidean spaces,

\mathcal W_2 ( \mathbb R^d )

\mathcal W_2 ( \mathbb R d )

, and carry out estimation by means of

M

\sqrt n

\mathcal W_2 ( \mathbb R ^d )

EPFL2017