Publications de Hugo Chao Cui | EPFL Graph Search

A phase transition between positional and semantic learning in a solvable model of dot-product attention

Florent Gérard Krzakala, Lenka Zdeborová, Freya Behrens, Hugo Chao Cui

Many empirical studies have provided evidence for the emergence of algorithmic mechanisms (abilities) in the learning of language models, that lead to qualitative improvements in the model capabilities. Yet, a theoretical characterization of how such mecha ...

2025

Bayes-optimal learning of deep random networks of extensive-width*

Florent Gérard Krzakala, Lenka Zdeborová, Hugo Chao Cui

We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width ...

IOP Publishing Ltd2025

High-dimensional learning of narrow neural networks

Hugo Chao Cui

Recent years have been marked with the fast-pace diversification and increasing ubiquity of machine learning (ML) applications. Yet, a firm theoretical understanding of the surprising efficiency of neural networks (NNs) to learn from high-dimensional data ...

2025

Asymptotics of feature learning in two-layer networks after one gradient-step

Florent Gérard Krzakala, Lenka Zdeborová, Luca Pesce, Yue Lu, Bruno Loureiro, Hugo Chao Cui

In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step. Leveraging the insight from (Ba et al., 2022), we model t ...

2024

Topics in statistical physics of high-dimensional machine learning

Hugo Chao Cui

In the past few years, Machine Learning (ML) techniques have ushered in a paradigm shift, allowing the harnessing of ever more abundant sources of data to automate complex tasks. The technical workhorse behind these important breakthroughs arguably lies in ...

EPFL2024

Deterministic equivalent and error universality of deep random features learning

Hugo Chao Cui

This manuscript considers the problem of learning a random Gaussian network function using a fully connected network with frozen intermediate layers and trainable readout layer. This problem can be seen as a natural generalization of the widely studied ran ...

2024

High-dimensional Asymptotics of Denoising Autoencoders

Lenka Zdeborová, Hugo Chao Cui

We address the problem of denoising data from a Gaussian mixture using a twolayer non-linear autoencoder with tied weights and a skip connection. We consider the high-dimensional limit where the number of training samples and the input dimension jointly te ...

2023

Bayes-optimal Learning of Deep Random Networks of Extensive-width

Florent Gérard Krzakala, Lenka Zdeborová, Hugo Chao Cui

We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width ...

2023

Error scaling laws for kernel classification under source and capacity conditions

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Hugo Chao Cui, Bruno Loureiro

In this manuscript we consider the problem of kernel classification. While worst-case bounds on the decay rate of the prediction error with the number of samples are known for some classifiers, they often fail to accurately describe the learning curves of ...

2023

Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime*

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Hugo Chao Cui

In this manuscript we consider kernel ridge regression (KRR) under the Gaussian design. Exponents for the decay of the excess generalization error of KRR have been reported in various works under the assumption of power-law decay of eigenvalues of the feat ...

2022