Réseau neuronal résiduel

droite|vignette| Forme canonique d'un réseau neuronal résiduel. Une couche ℓ − 1 est ignoré sur l'activation de ℓ − 2. Un réseau neuronal résiduel ( ResNet ) est un réseau neuronal artificiel (ANN). Il s'agit d'une variante du HighwayNet , le premier réseau neuronal à action directe très profond avec des centaines de couches, beaucoup plus profond que les réseaux neuronaux précédents. Les sauts de connexion ou "raccourcis" sont utilisés pour passer par-dessus certaines couches ( les HighwayNets peuvent également avoir des poids pour les saut eux-mêmes, grâce à une matrice de poids supplémentaire pour leurs portes). Les neurones à l'origine d'un saut de connexion sont appelés les neurones "amonts". Les modèles ResNet typiques sont mis en œuvre avec des sauts à double ou triple couche qui contiennent des non-linéarités ( ReLU ) et une normalisation par lots entre les deux. Les modèles avec plusieurs sauts parallèles sont appelés DenseNets. Dans le contexte des réseaux de neurones résid

Model Fusion via Optimal Transport

Martin Jaggi, Sidak Pal Singh

Combining different models is a widely used paradigm in machine learning applications. While the most common approach is to form an ensemble of models and average their individual predictions, this approach is often rendered infeasible by given resource constraints in terms of memory and computation, which grow linearly with the number of models. We present a layer-wise model fusion algorithm for neural networks that utilizes optimal transport to (soft-) align neurons across the models before averaging their associated parameters. We show that this can successfully yield "one-shot" knowledge transfer (i.e, without requiring any retraining) between neural networks trained on heterogeneous non-i.i.d. data. In both i.i.d. and non-i.i.d. settings, we illustrate that our approach significantly outperforms vanilla averaging, as well as how it can serve as an efficient replacement for the ensemble with moderate fine-tuning, for standard convolutional networks (like VGG11), residual networks (like ResNet18), and multi-layer perceptrons on CIFAR10, CIFAR100, and MNIST. Finally, our approach also provides a principled way to combine the parameters of neural networks with different widths, and we explore its application for model compression.

2020

Towards Real-World Super-Resolution using Deep Neural Networks

Ruofan Zhou

Image super-resolution reconstructs a higher-resolution image from the observed low-resolution image. In recent years, machine learning models have been widely employed and deep learning networks have achieved state-of-the-art super-resolution performance. Most of these methods, however, are trained and evaluated on simulated datasets that assume a simple and uniform degradation model. The deep learning models trained on such simulated datasets fail to generalize to practical applications because the actual or real degradation in real-world low-resolution images is much more complex. In this thesis, we propose several approaches to improve the robustness and generalization of deep super-resolution models. The first technique reduces the accumulation of errors in the camera pipeline. We build a deep residual network for learning an end-to-end mapping between raw images and high-resolution images. Our proposed network, trained on high-quality samples, is able to reconstruct in a single step high-quality super-resolved images from low-resolution Bayer mosaics. Extensive experiments show that the proposed method achieves better results than the state-of-the-art techniques, both qualitatively and quantitatively. To resolve the problem of the mismatch between the applied blur kernel in the synthetic dataset and the real-world camera blur, we propose to incorporate blur kernel modeling in the training. We generate the super-resolution training dataset by employing a set of realistic blur kernels estimated from real low-resolution photographs. We build a pool of realistic blur kernels with a generative adversarial network; then, we train a super-resolution network using the low-resolution images constructed with the generated kernels. Our method reconstructs more visually plausible high-resolution images compared to other state-of-the-art methods that rely on a simple degradation model. In order to study the effect of noise on super-resolution, we collect a dataset that contains pairs of noisy low-resolution images and the corresponding high-resolution images by using microscopy. We then benchmark the combinations of denoising methods and super-resolution networks on the dataset. Our experimental results show that the super-resolution networks are sensitive to noise and that the consecutive application of two applications suffers from the accumulation of errors. The benchmark results also suggest that the networks can benefit from joint optimization, hence we use a single network for joint denoising and super-resolution. Our network, trained with a novel texture loss, outperforms any combination of state-of-the-art deep denoising and super-resolution networks. Finally, to take advantage of multi-modal data available in certain applications, we propose a super-resolution system based on the fusion of information from multiple sources. For the application of spectral image super-resolution, we use two downsampled versions of the same image to infer a better high-resolution image for training. We refer to these inputs as a multi-scale modality. As color images are usually taken at a resolution higher than spectral images, we make use of color images as another modality to improve the super-resolution network. We build a pipeline that learns to super-resolve by using multi-scale spectral inputs guided by a color image by combining both modalities. Our proposed method is economical in time and memory consumption yet achieve competitive results.

EPFL2020

Towards Real-World Super-Resolution using Deep Neural Networks

Ruofan Zhou

EPFL2020

Model Fusion via Optimal Transport

Towards Real-World Super-Resolution using Deep Neural Networks

Neural Network Initialization

Towards Real-World Super-Resolution using Deep Neural Networks

Model Fusion via Optimal Transport

Neural Network Initialization