Vision-based Flocking in Aerial Robot Swarms

Aerial robot swarms have the potential to perform time-critical and dangerous tasks such as disaster response without compromising human safety. However, their reliance on external infrastructure such as global positioning for localization and wireless networks for communication is still a limiting factor for many applications. Such infrastructure may not be available everywhere, increasing their chance of collisions in case of signal interruptions and limiting their robustness to failure. Moreover, agent-to-agent wireless communication can suffer from time delays and outages, preventing their scalability to large swarms that fly in dense configurations. Drones should have the autonomy to make their own decisions exclusively based on local sensory information to avoid scalability and robustness issues.To address these limitations, we propose an entirely vision-based approach to swarm control inspired by flocking birds. In particular, we develop methods that enable drones to coordinate their motion by recognizing each other only with visual perception. We propose two distinct strategies that leverage the predictive power of convolutional neural networks: an end-to-end approach based on imitation learning and a modular system based on object detection and tracking. We test the algorithms using agent-based models and physics-based simulations with realistic sensor noise and validate them with a fleet of custom-built quadcopters in controlled indoor and challenging outdoor environments. The drones are equipped with an omnidirectional camera setup to avoid blind spots and onboard computation to process the images in real-time without requiring specialized hardware such as easy-to-recognize visual markers. Extensive simulations and real-world experiments show that vision-based swarms can perform collision-free and cohesive navigation while only relying on local visual information for control. We finally address the scalability of vision-based swarms in terms of group size and density.

Vision-based Drone Flocking in Outdoor Environments

Dario Floreano, Fabrizio Schiano, Fabian Maximilian Schilling

Decentralized deployment of drone swarms usually relies on inter-agent communication or visual markers that are mounted on the vehicles to simplify their mutual detection. This letter proposes a vision-based detection and tracking algorithm that enables groups of drones to navigate without communication or visual markers. We employ a convolutional neural network to detect and localize nearby agents onboard the quadcopters in real-time. Rather than manually labeling a dataset, we automatically annotate images to train the neural network using background subtraction by systematically flying a quadcopter in front of a static camera. We use a multi-agent state tracker to estimate the relative positions and velocities of nearby agents, which are subsequently fed to a flocking algorithm for high-level control. The drones are equipped with multiple cameras to provide omnidirectional visual inputs. The camera setup ensures the safety of the flock by avoiding blind spots regardless of the agent configuration. We evaluate the approach with a group of three real quadcopters that are controlled using the proposed vision-based flocking algorithm. The results show that the drones can safely navigate in an outdoor environment despite substantial background clutter and difficult lighting conditions. The source code, image dataset, and trained detection model are available at https://github.com/lis-epfl/vswarm.

2021

Model Predictive Control of Aerial Swarms

Enrica Soria

Aerial robot swarms can have a large socio-economic impact. They can perform time-critical missions faster than a single robot and access dangerous environments without compromising human safety. However, swarm deployment is often limited to free environments where no obstacle interferes with the robots' flight. Their trajectories are computed before the flight in many applications, and the robots are perceptually blind to one another. In these cases, all decisions are taken by a central computer that is informed about all robots' states, thus increasing their chance of failure in case of signal interruptions, limiting their robustness to failure, and preventing their scalability in size. Drones should have the autonomy to make their own decisions based on local information, and they should integrate a safe obstacle avoidance behavior to offer more versatility for their application in real-world environments.We propose a predictive approach to swarm control inspired by flocking birds to address these limitations. In particular, we develop methods that enable drones to coordinate their motion by predicting their future trajectory and coordinating with their neighbors based on local information. We first propose a centralized predictive algorithm that computes the trajectories of the drones in real-time and improves the synchronization and order of the swarm flight compared to current state-of-the-art algorithms. We show in extensive simulations that our algorithm is robust to different obstacle densities, swarm speeds, and inter-agent distances. Then, we formulate a distributed predictive algorithm with the same qualitative advantages as its centralized counterpart but scalable in the swarm size. We also show that our approach is tolerant to a wide range of sensor noise. Finally, we extend and analyze the usage of this algorithm also for pure sensor-based swarms that cannot use explicit communication. We validate the algorithms in extensive simulation and in the real world with a fleet of hand-sized quadcopters in controlled indoor environments with obstacles.

EPFL2022

Learning Vision-based Flight in Drone Swarms by Imitation

Dario Floreano, Julien Jean Denis Nicolas Lecoeur, Fabrizio Schiano, Fabian Maximilian Schilling

Decentralized drone swarms deployed today either rely on sharing of positions among agents or detecting swarm members with the help of visual markers. This work proposes an entirely visual approach to coordinate markerless drone swarms based on imitation learning. Each agent is controlled by a small and efficient convolutional neural network that takes raw omnidirectional images as inputs and predicts 3D velocity commands that match those computed by a flocking algorithm. We start training in simulation and propose a simple yet effective unsupervised domain adaptation approach to transfer the learned controller to the real world. We further train the controller with data collected in our motion capture hall. We show that the convolutional neural network trained on the visual inputs of the drone can learn not only robust inter-agent collision avoidance but also cohesion of the swarm in a sample-efficient manner. The neural controller effectively learns to localize other agents in the visual input, which we show by visualizing the regions with the most influence on the motion of an agent. We remove the dependence on sharing positions among swarm members by taking only local visual information into account for control. Our work can therefore be seen as the first step towards a fully decentralized, vision-based swarm without the need for communication or visual markers.

2019

Model Predictive Control of Aerial Swarms

Enrica Soria

EPFL2022