Time Dependent Image Generation of Plants from Incomplete Sequences with CNN-Transformer

Data imputation of incomplete image sequences is an essential prerequisite for analyzing and monitoring all development stages of plants in precision agriculture. For this purpose, we propose a conditional Wasserstein generative adversarial network TransGrow that combines convolutions for spatial modeling and a transformer for temporal modeling, enabling time-dependent image generation of above-ground plant phenotypes. Thereby, we achieve the following advantages over comparable data imputation approaches: (1) The model is conditioned by an incomplete image sequence of arbitrary length, the input time points, and the requested output time point, allowing multiple growth stages to be generated in a targeted manner; (2) By considering a stochastic component and generating a distribution for each point in time, the uncertainty in plant growth is considered and can be visualized; (3) Besides interpolation, also test-extrapolation can be performed to generate future plant growth stages. Experiments based on two datasets of different complexity levels are presented: Laboratory single plant sequences with Arabidopsis thaliana and agricultural drone image sequences showing crop mixtures. When comparing TransGrow to interpolation in image space, variational, and adversarial autoencoder, it demonstrates significant improvements in image quality, measured by multi-scale structural similarity, peak signal-to-noise ratio, and Frechet inception distance. To our knowledge, TransGrow is the first approach for time- and image-dependent, high-quality generation of plant images based on incomplete sequences.