Publication

Reservoir optimization in recurrent neural networks using properties of Kronecker product

Related publications (42)

Graph Chatbot

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Neural tensor contractions and the expressive power of deep neural quantum states

Giuseppe Carleo

We establish a direct connection between general tensor networks and deep feed-forward artificial neural networks. The core of our results is the construction of neural-network layers that efficiently perform tensor contractions and that use commonly adopt ...

AMER PHYSICAL SOC2022

, , , ,

Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamar ...

2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Ludovic Théo Stephan, Bruno Loureiro

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Stop Wasting my FLOPS: Improving the Efficiency of Deep Learning Models

Angelos Katharopoulos

Deep neural networks have completely revolutionized the field of machinelearning by achieving state-of-the-art results on various tasks ranging fromcomputer vision to protein folding. However, their application is hindered bytheir large computational and m ...

EPFL2022

By taking inspiration from the backflow transformation for correlated systems, we introduce a tensor network Ansatz which extends the well-established matrix product state representation of a quantum many-body wave function. This structure provides enough ...

AMER PHYSICAL SOC2022

Advances In Morphological Neural Networks: Training, Pruning And Enforcing Shape Constraints

Nikolaos Dimitriadis

In this paper, we study an emerging class of neural networks, the Morphological Neural networks, from some modern perspectives. Our approach utilizes ideas from tropical geometry and mathematical morphology. First, we state the training of a binary morphol ...

IEEE2021

A unified framework for Hamiltonian deep neural networks

Giancarlo Ferrari Trecate, Clara Lucía Galimberti, Liang Xu

Training deep neural networks (DNNs) can be difficult due to the occurrence of vanishing/explod- ing gradients during weight optimization. To avoid this problem, we propose a class of DNNs stemming from the time discretization of Hamiltonian systems. The t ...

2021

Gradient-based Methods for Deep Model Interpretability

Suraj Srinivas

In this dissertation, we propose gradient-based methods for characterizing model behaviour for the purposes of knowledge transfer and post-hoc model interpretation. Broadly, gradients capture the variation of some output feature of the model upon unit vari ...

EPFL2021

Active Subdivision Surfaces for the Semiautomatic Segmentation of Biomedical Volumes

Michaël Unser, Anaïs Laure Marie-Thérèse Badoual

We present a new family of active surfaces for the semiautomatic segmentation of volumetric objects in 3D biomedical images. We represent our deformable model by a subdivision surface encoded by a small set of control points and generated through a geometr ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2021

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

François Fleuret, Nikolaos Pappas, Angelos Katharopoulos, Apoorv Vyas

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input’s length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a line ...

Idiap2020