Publications by Thomas Michaelsen Pethick

Training Deep Learning Models with Norm-Constrained LMOs

Volkan Cevher, Kimon Antonakopoulos, Thomas Michaelsen Pethick, Zhenyu Zhu, Wanyun Xie

In this work, we study optimization methods that leverage the linear minimization oracle (lmo) over a norm-ball. We propose a new stochastic family of algorithms that uses the lmo to adapt to the geometry of the problem and, perhaps surprisingly, show that ...

2025

Improving SAM Requires Rethinking its Optimization Formulation

Volkan Cevher, Kimon Antonakopoulos, Thomas Michaelsen Pethick, Wanyun Xie, Fabian Ricardo Latorre Gomez

This paper rethinks Sharpness-Aware Minimization (SAM), which is originally formulated as a zero-sum game where the weights of a network and a bounded perturbation try to minimize/maximize, respectively, the same differentiable loss. We argue that SAM shou ...

2024

SAMPa: Sharpness-aware Minimization Parallelized

Volkan Cevher, Thomas Michaelsen Pethick, Wanyun Xie

Sharpness-aware minimization (SAM) has been shown to improve the generalization of neural networks. However, each SAM update requires sequentially computing two gradients, effectively doubling the per-iteration cost compared to base optimizers like SGD. We ...

2024

Mixed Nash for Robust Federated Learning

Volkan Cevher, Thomas Michaelsen Pethick, Wanyun Xie

We study robust federated learning (FL) within a game theoretic framework to alleviate the server vulnerabilities to even an informed adversary who can tailor training-time attacks (Fang et al., 2020; Xie et al., 2020a; Ozfatura et al., 2022; Rodríguez-Bar ...

2024

Finding Actual Descent Directions For Adversarial Training

Volkan Cevher, Thomas Michaelsen Pethick, Igor Krawczuk, Leello Tadesse Dadi, Fabian Ricardo Latorre Gomez

Adversarial Training using a strong first-order adversary (PGD) is the gold standard for training Deep Neural Networks that are robust to adversarial examples. We show that, contrary to the general understanding of the method, the gradient at an optimal ad ...

2023

Federated Learning under Covariate Shifts with Generalization Guarantees

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Thomas Michaelsen Pethick

This paper addresses intra-client and inter-client covariate shifts in federated learning (FL) with a focus on the overall generalization performance. To handle covariate shifts, we formulate a new global model training paradigm and propose Federated Impor ...

2023

Revisiting adversarial training for the worst-performing class

Volkan Cevher, Grigorios Chrysos, Thomas Michaelsen Pethick

Despite progress in adversarial training (AT), there is a substantial gap between the topperforming and worst-performing classes in many datasets. For example, on CIFAR10, the accuracies for the best and worst classes are 74% and 23%, respectively. We argu ...

2023

Stable Nonconvex-Nonconcave Training via Linear Interpolation

Volkan Cevher, Thomas Michaelsen Pethick, Wanyun Xie

This paper presents a theoretical analysis of linear interpolation as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss lan ...

2023

Escaping limit cycles: Global convergence for constrained nonconvex-nonconcave minimax problems

Volkan Cevher, Thomas Michaelsen Pethick

This paper introduces a new extragradient-type algorithm for a class of nonconvex-nonconcave minimax problems. It is well-known that finding a local solution for general minimax problems is computationally intractable. This observation has recently motivat ...

2022

Subquadratic Overparameterization for Shallow Neural Networks

Armin Eftekhari, Volkan Cevher, Chaehwan Song, Thomas Michaelsen Pethick

Overparameterization refers to the important phenomenon where the width of a neural network is chosen such that learning algorithms can provably attain zero loss in nonconvex training. The existing theory establishes such global convergence using various i ...

2021