Vision-based Drone Flocking in Outdoor Environments

Dario Floreano, Fabrizio Schiano, Fabian Maximilian Schilling
2021
Article

Résumé

Decentralized deployment of drone swarms usually relies on inter-agent communication or visual markers that are mounted on the vehicles to simplify their mutual detection. This letter proposes a vision-based detection and tracking algorithm that enables groups of drones to navigate without communication or visual markers. We employ a convolutional neural network to detect and localize nearby agents onboard the quadcopters in real-time. Rather than manually labeling a dataset, we automatically annotate images to train the neural network using background subtraction by systematically flying a quadcopter in front of a static camera. We use a multi-agent state tracker to estimate the relative positions and velocities of nearby agents, which are subsequently fed to a flocking algorithm for high-level control. The drones are equipped with multiple cameras to provide omnidirectional visual inputs. The camera setup ensures the safety of the flock by avoiding blind spots regardless of the agent configuration. We evaluate the approach with a group of three real quadcopters that are controlled using the proposed vision-based flocking algorithm. The results show that the drones can safely navigate in an outdoor environment despite substantial background clutter and difficult lighting conditions. The source code, image dataset, and trained detection model are available at https://github.com/lis-epfl/vswarm.

Official source

https://infoscience.epfl.ch/record/283488?ln=fr

À propos de ce résultat

Cette page est générée automatiquement et peut contenir des informations qui ne sont pas correctes, complètes, à jour ou pertinentes par rapport à votre recherche. Il en va de même pour toutes les autres pages de ce site. Veillez à vérifier les informations auprès des sources officielles de l'EPFL.

Concepts associés

Chargement

Publications associées

Chargement

Dario Floreano, Fabrizio Schiano, Fabian Maximilian Schilling
2021
Article

Résumé

Official source

https://infoscience.epfl.ch/record/283488?ln=fr

À propos de ce résultat

Concepts associés (20)

Concepts associés

Chargement

Publications associées (71)

Chargement

Afficher plus

Publications associées

Chargement

Publications associées (71)

Tracking and Relative Localization of Drone Swarms With a Vision-Based Headset

Dario Floreano, Giuseppe Loianno, Maxim Pavliv, Fabrizio Schiano

We address the detection, tracking, and relative localization of the agents of a drone swarm from a human perspective using a headset equipped with a single camera and an Inertial Measurement Unit (IMU). We train and deploy a deep neural network detector on image data to detect the drones. A joint probabilistic data association filter resolves the detection problems and couples this information with the headset IMU data to track the agents. In order to estimate the drones’ relative poses in 3D space with respect to the human, we use an additional deep neural network that processes image regions of the drones provided by the tracker. Finally, to speed up the deep neural networks’ training, we introduce an automated labeling process relying on a motion capture system. Several experimental results validate the effectiveness of the proposed approach. The approach is real-time, does not rely on any communication between the human and the drones, and can scale to a large number of agents, often called swarms. It can be used to spatially task a swarm of drones and also employed without a headset for formation control and coordination of terrestrial vehicles.

2021

Chargement

Learning Vision-based Flight in Drone Swarms by Imitation

Dario Floreano, Julien Jean Denis Nicolas Lecoeur, Fabrizio Schiano, Fabian Maximilian Schilling

Decentralized drone swarms deployed today either rely on sharing of positions among agents or detecting swarm members with the help of visual markers. This work proposes an entirely visual approach to coordinate markerless drone swarms based on imitation learning. Each agent is controlled by a small and efficient convolutional neural network that takes raw omnidirectional images as inputs and predicts 3D velocity commands that match those computed by a flocking algorithm. We start training in simulation and propose a simple yet effective unsupervised domain adaptation approach to transfer the learned controller to the real world. We further train the controller with data collected in our motion capture hall. We show that the convolutional neural network trained on the visual inputs of the drone can learn not only robust inter-agent collision avoidance but also cohesion of the swarm in a sample-efficient manner. The neural controller effectively learns to localize other agents in the visual input, which we show by visualizing the regions with the most influence on the motion of an agent. We remove the dependence on sharing positions among swarm members by taking only local visual information into account for control. Our work can therefore be seen as the first step towards a fully decentralized, vision-based swarm without the need for communication or visual markers.

2019

Chargement

Vision-based Flocking in Aerial Robot Swarms

Fabian Maximilian Schilling

Aerial robot swarms have the potential to perform time-critical and dangerous tasks such as disaster response without compromising human safety. However, their reliance on external infrastructure such as global positioning for localization and wireless networks for communication is still a limiting factor for many applications. Such infrastructure may not be available everywhere, increasing their chance of collisions in case of signal interruptions and limiting their robustness to failure. Moreover, agent-to-agent wireless communication can suffer from time delays and outages, preventing their scalability to large swarms that fly in dense configurations. Drones should have the autonomy to make their own decisions exclusively based on local sensory information to avoid scalability and robustness issues.To address these limitations, we propose an entirely vision-based approach to swarm control inspired by flocking birds. In particular, we develop methods that enable drones to coordinate their motion by recognizing each other only with visual perception. We propose two distinct strategies that leverage the predictive power of convolutional neural networks: an end-to-end approach based on imitation learning and a modular system based on object detection and tracking. We test the algorithms using agent-based models and physics-based simulations with realistic sensor noise and validate them with a fleet of custom-built quadcopters in controlled indoor and challenging outdoor environments. The drones are equipped with an omnidirectional camera setup to avoid blind spots and onboard computation to process the images in real-time without requiring specialized hardware such as easy-to-recognize visual markers. Extensive simulations and real-world experiments show that vision-based swarms can perform collision-free and cohesive navigation while only relying on local visual information for control. We finally address the scalability of vision-based swarms in terms of group size and density.

EPFL2022

Chargement

Afficher plus

Vision-based Flocking in Aerial Robot Swarms

Fabian Maximilian Schilling

EPFL2022