Learning Vision-based Flight in Drone Swarms by Imitation

Dario Floreano, Julien Jean Denis Nicolas Lecoeur, Fabrizio Schiano, Fabian Maximilian Schilling
2019
Article

Résumé

Decentralized drone swarms deployed today either rely on sharing of positions among agents or detecting swarm members with the help of visual markers. This work proposes an entirely visual approach to coordinate markerless drone swarms based on imitation learning. Each agent is controlled by a small and efficient convolutional neural network that takes raw omnidirectional images as inputs and predicts 3D velocity commands that match those computed by a flocking algorithm. We start training in simulation and propose a simple yet effective unsupervised domain adaptation approach to transfer the learned controller to the real world. We further train the controller with data collected in our motion capture hall. We show that the convolutional neural network trained on the visual inputs of the drone can learn not only robust inter-agent collision avoidance but also cohesion of the swarm in a sample-efficient manner. The neural controller effectively learns to localize other agents in the visual input, which we show by visualizing the regions with the most influence on the motion of an agent. We remove the dependence on sharing positions among swarm members by taking only local visual information into account for control. Our work can therefore be seen as the first step towards a fully decentralized, vision-based swarm without the need for communication or visual markers.

Official source

https://infoscience.epfl.ch/record/269166?ln=fr

À propos de ce résultat

Cette page est générée automatiquement et peut contenir des informations qui ne sont pas correctes, complètes, à jour ou pertinentes par rapport à votre recherche. Il en va de même pour toutes les autres pages de ce site. Veillez à vérifier les informations auprès des sources officielles de l'EPFL.

Concepts associés

Chargement

Publications associées

Chargement

Dario Floreano, Julien Jean Denis Nicolas Lecoeur, Fabrizio Schiano, Fabian Maximilian Schilling
2019
Article

Résumé

Official source

https://infoscience.epfl.ch/record/269166?ln=fr

À propos de ce résultat

Concepts associés (15)

Concepts associés

Chargement

Publications associées (29)

Publications associées

Chargement

Vision-based Drone Flocking in Outdoor Environments

Dario Floreano, Fabrizio Schiano, Fabian Maximilian Schilling

Decentralized deployment of drone swarms usually relies on inter-agent communication or visual markers that are mounted on the vehicles to simplify their mutual detection. This letter proposes a vision-based detection and tracking algorithm that enables groups of drones to navigate without communication or visual markers. We employ a convolutional neural network to detect and localize nearby agents onboard the quadcopters in real-time. Rather than manually labeling a dataset, we automatically annotate images to train the neural network using background subtraction by systematically flying a quadcopter in front of a static camera. We use a multi-agent state tracker to estimate the relative positions and velocities of nearby agents, which are subsequently fed to a flocking algorithm for high-level control. The drones are equipped with multiple cameras to provide omnidirectional visual inputs. The camera setup ensures the safety of the flock by avoiding blind spots regardless of the agent configuration. We evaluate the approach with a group of three real quadcopters that are controlled using the proposed vision-based flocking algorithm. The results show that the drones can safely navigate in an outdoor environment despite substantial background clutter and difficult lighting conditions. The source code, image dataset, and trained detection model are available at https://github.com/lis-epfl/vswarm.

2021

Chargement

Biologically plausible unsupervised learning in shallow and deep neural networks

Bernd Albert Illing

The way our brain learns to disentangle complex signals into unambiguous concepts is fascinating but remains largely unknown. There is evidence, however, that hierarchical neural representations play a key role in the cortex. This thesis investigates biologically plausible models of unsupervised learning of hierarchical representations as found in the brain and modern computer vision models. We use computational modeling to address three main questions at the intersection of artificial intelligence (AI) and computational neuroscience.The first question is: What are useful neural representations and when are deep hierarchical representations needed? We approach this point with a systematic study of biologically plausible unsupervised feature learning in a shallow 2-layer networks on digit (MNIST) and object (CIFAR10) classification. Surprisingly, random features support high performance, especially for large hidden layers. When combined with localized receptive fields, random feature networks approach the performance of supervised backpropagation on MNIST, but not on CIFAR10. We suggest that future models of biologically plausible learning should outperform such random feature benchmarks on MNIST, or that such models should be evaluated in different ways.The second question is: How can hierarchical representations be learned with mechanisms supported by neuroscientific evidence? We cover this question by proposing a unifying Hebbian model, inspired by common models of V1 simple and complex cells based on unsupervised sparse coding and temporal invariance learning. In shallow 2-layer networks, our model reproduces learning of simple and complex cell receptive fields, as found in V1. In deeper networks, we stack multiple layers of Hebbian learning but find that it does not yield hierarchical representations of increasing usefulness. From this, we hypothesise that standard Hebbian rules are too constrained to build increasingly useful representations, as observed in higher areas of the visual cortex or deep artificial neural networks.The third question is: Can AI inspire learning models that build deep representations and are still biologically plausible? We address this question by proposing a learning rule that takes inspiration from neuroscience and recent advances in self-supervised deep learning. The proposed rule is Hebbian, i.e. only depends on pre- and post-synaptic neuronal activity, but includes additional local factors, namely predictive dendritic input and widely broadcasted modulation factors. Algorithmically, this rule applies self-supervised contrastive predictive learning to a causal, biological setting using saccades. We find that networks trained with this generalised Hebbian rule build deep hierarchical representations of images, speech and video.We see our modeling as a potential starting point for both, new hypotheses, that can be tested experimentally, and novel AI models that could benefit from added biological realism.

EPFL2021

Chargement

Aerial robot swarms have the potential to perform time-critical and dangerous tasks such as disaster response without compromising human safety. However, their reliance on external infrastructure such as global positioning for localization and wireless networks for communication is still a limiting factor for many applications. Such infrastructure may not be available everywhere, increasing their chance of collisions in case of signal interruptions and limiting their robustness to failure. Moreover, agent-to-agent wireless communication can suffer from time delays and outages, preventing their scalability to large swarms that fly in dense configurations. Drones should have the autonomy to make their own decisions exclusively based on local sensory information to avoid scalability and robustness issues.To address these limitations, we propose an entirely vision-based approach to swarm control inspired by flocking birds. In particular, we develop methods that enable drones to coordinate their motion by recognizing each other only with visual perception. We propose two distinct strategies that leverage the predictive power of convolutional neural networks: an end-to-end approach based on imitation learning and a modular system based on object detection and tracking. We test the algorithms using agent-based models and physics-based simulations with realistic sensor noise and validate them with a fleet of custom-built quadcopters in controlled indoor and challenging outdoor environments. The drones are equipped with an omnidirectional camera setup to avoid blind spots and onboard computation to process the images in real-time without requiring specialized hardware such as easy-to-recognize visual markers. Extensive simulations and real-world experiments show that vision-based swarms can perform collision-free and cohesive navigation while only relying on local visual information for control. We finally address the scalability of vision-based swarms in terms of group size and density.

EPFL2022

Chargement

Biologically plausible unsupervised learning in shallow and deep neural networks

Bernd Albert Illing

EPFL2021

EPFL2022