iPool--Information-Based Pooling in Hierarchical Graph Neural Networks

Pascal Frossard, Xing Gao, Chenglin Li
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2021
Journal paper

Abstract

With the advent of data science, the analysis of network or graph data has become a very timely research problem. A variety of recent works have been proposed to generalize neural networks to graphs, either from a spectral graph theory or a spatial perspective. The majority of these works, however, focus on adapting the convolution operator to graph representation. At the same time, the pooling operator also plays an important role in distilling multiscale and hierarchical representations, but it has been mostly overlooked so far. In this article, we propose a parameter-free pooling operator, called iPool, that permits to retain the most informative features in arbitrary graphs. With the argument that informative nodes dominantly characterize graph signals, we propose a criterion to evaluate the amount of information of each node given its neighbors and theoretically demonstrate its relationship to neighborhood conditional entropy. This new criterion determines how nodes are selected and coarsened graphs are constructed in the pooling layer. The resulting hierarchical structure yields an effective isomorphism-invariant representation of networked data on arbitrary topologies. The proposed strategy achieves superior or competitive performance in graph classification on a collection of public graph benchmark data sets and superpixel-induced image graph data sets.

Official source

https://infoscience.epfl.ch/record/290942?ln=en

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related concepts

Related publications

Graph Representation Learning with Optimal Transport: Analysis and Applications

Effrosyni Simou

In several machine learning settings, the data of interest are well described by graphs. Examples include data pertaining to transportation networks or social networks. Further, biological data, such as proteins or molecules, lend themselves well to graph-structured descriptions. In such settings, it is desirable to design representation learning algorithms that can take into account the information captured by the underlying structure. This thesis focuses on providing new methods to ingrain geometrical information in end-to-end deep learning architectures for graphs through the use of elements from Optimal Transport theory.First, we consider the setting where the data can be described by a fixed graph. In this setting, each datapoint corresponds to a node of the graph and the edges among nodes signify their pairwise relations. We propose an autoencoder architecture that consists of a linear layer in the encoder and a novel Wasserstein barycentric layer at the decoder. Our proposed barycentric layer takes into account the underlying geometry through the diffusion distance of the graph and provides a way to obtain structure-aware, non-linear interpolations, which lead to directly interpretable node embeddings that are stable to perturbations of the graph structure. Further, the embeddings obtained with our proposed autoencoder achieve competitive or superior results compared to state-of-the-art methods in node classification tasks.Second, we consider the setting where the data consist of multiple graphs. In that setting, the graphs can be of varying size and, therefore, a global pooling operation is needed in order to obtain representations of fixed size and enable end-to-end learning with deep networks. We propose a global pooling operation that optimally preserves the statistical properties of the representations. In order to do so, we take into account the geometry of the representation space and introduce a global pooling layer that minimizes the Wasserstein distance between a graph representation and its pooled counterpart, of fixed size. This is achieved by performing a Wasserstein gradient flow with respect to the pooled representation. Our proposed method demonstrates promising results in the task of graph classification.Overall, in this thesis we provide new methods for incorporating geometrical information in end-to-end deep learning architectures for graph structured data. We believe that our proposals will contribute to the development of algorithms that learn meaningful representations and that take fully into account the geometry of the data under consideration.

EPFL2022

The explosive growth of machine learning in the age of data has led to a new probabilistic and data-driven approach to solving very different types of problems. In this paper we study the feasibility of using such data-driven algorithms to solve classic physical and mathematical problems. In particular, we try to model the solution of an inverse continuum mechanics problem in the context of linear elasticity using deep neural networks. To better address the inverse function, we start first by studying the simplest related task,consisting of a building block of the actual composite problem. By empirically proving the learnability of simpler functions, we aim to draw conclusions with respect to the initial problem.The basic inverse problem that motivates this paper is that of a 2D plate with inclusion under specific loading and boundary conditions. From measurements at static equilibrium,we wish to recover the position of the hole. Although some analytical solutions have been formulated for 3D-infinite solids - most notably Eshelby’s inclusion problems - finite problems with particular geometries, material inhomogeneities, loading and boundary conditions require the use of numerical methods which are most often efficient solutions to the forward problem, the mapping from the parameter space to the measurement/signal space, i.e. in our case computing displacements and stresses knowing the size and position of the inclusion. Using numerical data generated from the well-defined forward problem,we train a neural network to approximate the inverse function relating displacements and stresses to the position of the inclusion. The preliminary results on the 2D-finite problem are promising, but the black-box nature of neural networks is a huge issue when it comes to understanding the solution.For this reason, we study a 3D-infinite continuous isotropic medium with unique concentrated load, for which the Green’s function gives an analytical mathematical expression relating relative position of the point force and the displacements in the solid. After de-riving the expression of the inverse, namely recovering the relative position of the force from the Green’s matrix computed at a given point in the medium, we are able to study the sensitivity of the inverse function. From both the expression of the Green’s function and its inverse, we highlight what issues might arise when training neural networks to solve the inverse problem. As the Green’s function is not bijective, bijection must been forced when training for regression. Moreover, due to displacements growing to infinity as we approach the singularity at zero, the training domain must be constrained to be some distance away from the singularity. As we train a neural network to fit the inverse of the Green’s function, we show that the input parameters should include the least possible redundant information to ensure the most efficient training.We then extend our analysis to two point forces. As more loads are added, bijection is harder to enforce as permutations of forces must be taken into account and more collisions may arise, i.e. multiple specific combinations of forces might yield the same measurements.One obvious solution is to increase the number of nodes where displacements are measured to limit the possibility of collision. Through new experiments, we show again that the best training is achieved for the least possible amount of nodes, as long as the training data generated is indeed bijective. As the medium is elastic, we propose a neural network architecture that matches the composite nature of the inverse problem. We also present another formulation of the problem which is invariant to permutations of the forces,namely multilabel classification, and yields good performance in the two-load case.Finally, we study the composite inverse function for 2, 3, 4 and 5 forces. By comparing training and accuracy for different neural network architectures, we expose the model easiest to train. Moreover, the evolution of the final accuracy as more loads are added indicates that deep-neural networks (DNNs) are not well suited to fit a non-linear mapping from and to a high-dimensional space. The results are more convincing for multilabel classification.

2020

Learning music composition with recurrent neural networks

Florian François Colombo

Throughout this thesis, we are interested in modeling music composition. To do so, we study the association of music theory concepts with the learning capabilities of recurrent neural networks. Especially, we explore numerical formalizations of music so that our models operate on data containing useful information to decide how to combine particular rhythms and notes in specified musical contexts. The first part introduces the Deep Artificial Composer, a generative model of monophonic melodies. We present a network architecture that considers the temporal and instantaneous relationships between the pitch and duration of notes. By combining Irish with klezmer melodies to train the Deep Artificial Composer, we study how it reacts to heterogeneous data. Furthermore, we show how to generate melodies containing Irish or Klezmer folk musicâs characteristics from our probabilistic model of notes. The second part presents BachProp, a model for the composition of polyphonic music. We discover how scores of different styles can be generated from models using neural networks. In particular, we train BachProp with Bachâs chorales and the English folk music corpus Nottingham. Also, we verify that generated compositions share specific properties with the original corpus it processes. Especially, our musical novelty statistics show that BachProp generates musical sequences as innovative as those present in the original corpora. Finally, a large audience appreciated the Ada string quartet performances of BachPropâs compositions. Whereas BachProp operates on any music in the usual MIDI format, the BeethovANN algorithm, introduced in the last part, needs the information contained in scores in the MusicXML format. These digital scores allow us to formalize relevant musical characteristics in order to generate music with BeethovANN. Thus, to create new voices that can be played in accompaniment to Beethoven string quartets, we take into account not only the note sequences, but also the harmonic progression, interactions between voices, and the rhythm of each instrument. Eventually, professional musicians misclassified BeethovANN musical phrases for Beethovenâs original compositions.

EPFL2021