Publications de Zhenyu Zhu | EPFL Graph Search

How Gradient Descent Balances Features: A Dynamical Analysis For Two-Layer Neural Networks

This paper investigates the fundamental regression task of learning k neurons (a.k.a. teachers) from Gaussian input, using two-layer ReLU neural networks with width m (a.k.a. students) and m, k = O(1), trained via gradient descent under proper initializati ...

2025

Training Deep Learning Models with Norm-Constrained LMOs

Volkan Cevher, Kimon Antonakopoulos, Thomas Michaelsen Pethick, Zhenyu Zhu, Wanyun Xie

In this work, we study optimization methods that leverage the linear minimization oracle (lmo) over a norm-ball. We propose a new stochastic family of algorithms that uses the lmo to adapt to the geometry of the problem and, perhaps surprisingly, show that ...

2025

Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modeling

Volkan Cevher, Zhenyu Zhu

This paper provides statistical sample complexity bounds for score-matching and its applications in causal discovery. We demonstrate that accurate estimation of the score function is achievable by training a standard deep ReLU neural network using stochast ...

2023

Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks

Volkan Cevher, Fanghui Liu, Zhenyu Zhu, Reza Shokri

We analytically investigate how over-parameterization of models in randomized machine learning algorithms impacts the information leakage about their training data. Specifically, we prove a privacy bound for the KL divergence between model distributions on ...

2023

Benign Overfitting in Deep Neural Networks under Lazy Training

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayesoptimal test error for classification while obtaining (nearly) zero-trai ...

2023

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu, Yongtao Wu

Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamar ...

2022

Controlling the Complexity and Lipschitz Constant improves Polynomial Nets

Volkan Cevher, Grigorios Chrysos, Zhenyu Zhu, Fabian Ricardo Latorre Gomez

While the class of Polynomial Nets demonstrates comparable performance to neural networks (NN), it currently has neither theoretical generalization characterization nor robustness guarantees. To this end, we derive new complexity bounds for the set of Coup ...

2022

Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

We study the average robustness notion in deep neural networks in (selected) wide and narrow, deep and shallow, as well as lazy and non-lazy training settings. We prove that in the under-parameterized setting, width has a negative effect while it improves ...

2022

Generalization Properties of NAS under Activation and Skip Connection Search

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

Neural Architecture Search (NAS) has fostered the automatic discovery of stateof- the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generaliza ...

2022