Learning and generation of slow sequences: an application to music composition

Human brains can deal with sequences with temporal dependencies on a broad range of timescales, many of which are several order of magnitude longer than neuronal timescales. Here we introduce an artificial intelligence that learns and produces the complex structure of music, a specific type of slow sequence. Our model employs a separation of fundamental features and multi-layer networks of gated recurrent units. We separate the information contained in monophonic melodies into their rhythm and melody features. The model processes these features in parallel while modelling the relation between them, effectively splitting the joint distribution over note duration and pitch into conditional probabilities. Using such an approach, we were able to automatically learn the temporal dependencies inherent of large corpora of Irish folk songs. We could use the extracted structural rules to generate interesting complete melodies or suggest possible continuations of melody fragments that are coherent with the characteristics of the fragments themselves.

Learning music composition with recurrent neural networks

Florian François Colombo

Throughout this thesis, we are interested in modeling music composition. To do so, we study the association of music theory concepts with the learning capabilities of recurrent neural networks. Especially, we explore numerical formalizations of music so that our models operate on data containing useful information to decide how to combine particular rhythms and notes in specified musical contexts. The first part introduces the Deep Artificial Composer, a generative model of monophonic melodies. We present a network architecture that considers the temporal and instantaneous relationships between the pitch and duration of notes. By combining Irish with klezmer melodies to train the Deep Artificial Composer, we study how it reacts to heterogeneous data. Furthermore, we show how to generate melodies containing Irish or Klezmer folk musicâs characteristics from our probabilistic model of notes. The second part presents BachProp, a model for the composition of polyphonic music. We discover how scores of different styles can be generated from models using neural networks. In particular, we train BachProp with Bachâs chorales and the English folk music corpus Nottingham. Also, we verify that generated compositions share specific properties with the original corpus it processes. Especially, our musical novelty statistics show that BachProp generates musical sequences as innovative as those present in the original corpora. Finally, a large audience appreciated the Ada string quartet performances of BachPropâs compositions. Whereas BachProp operates on any music in the usual MIDI format, the BeethovANN algorithm, introduced in the last part, needs the information contained in scores in the MusicXML format. These digital scores allow us to formalize relevant musical characteristics in order to generate music with BeethovANN. Thus, to create new voices that can be played in accompaniment to Beethoven string quartets, we take into account not only the note sequences, but also the harmonic progression, interactions between voices, and the rhythm of each instrument. Eventually, professional musicians misclassified BeethovANN musical phrases for Beethovenâs original compositions.

EPFL2021

Biologically plausible unsupervised learning in shallow and deep neural networks

Bernd Albert Illing

The way our brain learns to disentangle complex signals into unambiguous concepts is fascinating but remains largely unknown. There is evidence, however, that hierarchical neural representations play a key role in the cortex. This thesis investigates biologically plausible models of unsupervised learning of hierarchical representations as found in the brain and modern computer vision models. We use computational modeling to address three main questions at the intersection of artificial intelligence (AI) and computational neuroscience.The first question is: What are useful neural representations and when are deep hierarchical representations needed? We approach this point with a systematic study of biologically plausible unsupervised feature learning in a shallow 2-layer networks on digit (MNIST) and object (CIFAR10) classification. Surprisingly, random features support high performance, especially for large hidden layers. When combined with localized receptive fields, random feature networks approach the performance of supervised backpropagation on MNIST, but not on CIFAR10. We suggest that future models of biologically plausible learning should outperform such random feature benchmarks on MNIST, or that such models should be evaluated in different ways.The second question is: How can hierarchical representations be learned with mechanisms supported by neuroscientific evidence? We cover this question by proposing a unifying Hebbian model, inspired by common models of V1 simple and complex cells based on unsupervised sparse coding and temporal invariance learning. In shallow 2-layer networks, our model reproduces learning of simple and complex cell receptive fields, as found in V1. In deeper networks, we stack multiple layers of Hebbian learning but find that it does not yield hierarchical representations of increasing usefulness. From this, we hypothesise that standard Hebbian rules are too constrained to build increasingly useful representations, as observed in higher areas of the visual cortex or deep artificial neural networks.The third question is: Can AI inspire learning models that build deep representations and are still biologically plausible? We address this question by proposing a learning rule that takes inspiration from neuroscience and recent advances in self-supervised deep learning. The proposed rule is Hebbian, i.e. only depends on pre- and post-synaptic neuronal activity, but includes additional local factors, namely predictive dendritic input and widely broadcasted modulation factors. Algorithmically, this rule applies self-supervised contrastive predictive learning to a causal, biological setting using saccades. We find that networks trained with this generalised Hebbian rule build deep hierarchical representations of images, speech and video.We see our modeling as a potential starting point for both, new hypotheses, that can be tested experimentally, and novel AI models that could benefit from added biological realism.

EPFL2021

Enactive robot vision

Mototaka Suzuki

The complexity of today's autonomous robots poses a major challenge for Artificial Intelligence. These robots are equipped with sophisticated sensors and mechanical abilities that allow them to enter our homes and interact with humans. For example, today's robots are almost all equipped with vision and several of them can move over rough terrain with wheels or legs. The methods developed so far in Artificial Intelligence, however, are not yet ready to cope with the complexity of the information gathered through the robot sensors and the need for rapid action in partially unknown and dynamic environments. In this thesis, I will argue that the apparent complexity of the environment and of the robot brain can be significantly simplified if perception, behavior, and learning are allowed to co-develop on the same time scale. In doing so, robots become sensitive to, and actively exploit, characteristics of the environment that they can tackle within their own computational and physical constraints. This line of work is grounded on philosophical and psychological research showing that perception is an active process mediated by behavior. However, computational models of active vision are very rare and often rely on architectures that are preprogrammed to detect certain characteristics of the environment. Previous work have shown that complex visual tasks, such as position and size invariant shape recognition as well as navigation, can be tackled with remarkably simple neural architectures generated by a coevolutionary process of active vision and feature selection. Behavioral machines equipped with primitive vision systems and direct pathways between visual and motor neurons were evolved while freely interacting with their environments. I proceed further on this line of investigation and describe the application of this methodology in three situations, namely car driving with an omnidirectional camera, goal-oriented navigation of a humanoid robot, and cooperative tasks by two agents. I will show that these systems develop sensitivity to a number of oriented, retinotopic, visual features – oriented edges, corners, height – and a behavioral repertoire to locate, bring, and keep these features in sensitive regions of the vision system that allow them to accomplish their goals. In a second set of experiments, I will show that active vision can be exploited by the robot to perform anticipatory exploration of the environment in a task that requires landmark-based navigation. Evolved robots exploit an internal expectation system that makes use of active exploration to check for expected events in the environment. I will describe a third set of experiments where, in addition to an evolutionary process, the visual system of the robot can develop receptive fields by means of unsupervised Hebbian learning and show that these receptive fields are significantly affected by the behavior of the system and differ from those predicted by most computational models of visual cortex. Finally, I will show that these robots replicate the performance deficiencies observed in experiments of motor deprivation with kitten when they are exposed to the same type of motor deprivations. Furthermore, the analyses of our robot brains suggest an explanation for the deficiencies observed in kitten that have not yet been fully understood.

EPFL2007

Biologically plausible unsupervised learning in shallow and deep neural networks

Bernd Albert Illing

EPFL2021

Learning music composition with recurrent neural networks

Florian François Colombo

EPFL2021

Enactive robot vision

Mototaka Suzuki

EPFL2007