Towards real-time weed detection and segmentation with lightweight CNN models on edge devices

Abstract

With the resurgence of on-the-go site-specific weed control technologies, research has increasingly focused on real-time weed identification using computer vision and machine learning. While Convolutional Neural Networks (CNNs) have been successfully used for automatic weed detection, many studies prioritize accuracy over inference speed and computational resource demand. The objective of this work was to develop, deploy, and evaluate CNN-based computer vision models on computationally resource-restrained edge devices for real-time weed classification and segmentation. RGB images were collected by UAVs over corn and soybean fields infested with Palmer Amaranth ( Amaranthus palmeri ). A few object detection (OD) and semantic segmentation (SS) models were modified, if necessary, to be lightweight, then trained and deployed on edge devices for real-time weed detection with resized input images in dimensions of 256 × 256 pixels. Specifically, the original backbones of Single-Shot Detector (SSD) and DeepLabv3+ models were replaced by a pre-trained MobileNetV3 backbone and deployed on a Jetson Nano. A YOLOv8n model and a proposed MobileNetV4-Seg model were trained and deployed on a Jetson Orin Nano. All models achieved real-time performance with acceptable accuracies with the data collected in corn and soybean fields. Among them, the MobilenetV4-Seg achieved the best performance, with IoUs of 69.9 % and 76.8 %, F 1 scores of 82.3 % and 86.9 %, for the corn and soybean datasets, respectively, and 44 FPS on the Jetson Orin Nano. Overall, this study demonstrated a use case and feasible workflow for developing and deploying lightweight CNN models on resource-constrained edge devices for on-the-go real-time site-specific applications.