Contrastive 3D Shape Completion and Reconstruction for Agricultural Robots Using RGB-D Frames

Abstract

Monitoring plants and fruits is important in modern agriculture, with applications ranging from high-throughput phenotyping to autonomous harvesting. Obtaining highly accurate 3D measurements under real agricultural conditions is a challenging task. In this letter, we address the problem of estimating the 3D shape of fruits when only a partial view is available. We propose a pipeline that exploits high-resolution 3D data in the learning phase but only requires a single RGB-D frame to predict the 3D shape of a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">complete</i> fruit during operation. To achieve this, we first learn a latent space of potential fruit appearances that we can decode into an SDF volume. With the pretrained, frozen decoder, we subsequently learn an encoder that can produce meaningful latent vectors from a single RGB-D frame. The experiments presented in this letter suggest that our approach can predict the 3D shape of whole fruits online, needing only 4 ms for inference. We evaluate our approach in controlled environments and illustrate its deployment in greenhouses without modifications.