Original papersPerformance of deep learning models for classifying and detecting common weeds in corn and soybean production systems
Introduction
Weeds have been a major concern to crop production due to a negative influence on yield. It is reported to be responsible for reducing 30% of crop yield worldwide (Gao et al., 2018). Potential economic losses from weeds equal more than $25 billion on an annual basis for corn and soybean crops in North America alone (Soltani et al., 2016). A typical weed control strategy involves application of herbicide on over 90% of field crops in the US (Gianessi & Reigner, 2007). In 2012, pesticides accounted for 5% of the total farm expenditure in the US, of which 58% was used for herbicides (Atwood & Paisley-Jones, 2017). In addition, according to the European Food Safety Authority (EFSA) approximately 99% of food products contain some residue from these herbicides (Partel et al., 2019). It is therefore crucial to develop effective and efficient weed control methods to harvest crops with minimum chemical residues. One such solution includes applying herbicide to only specified regions of a field consisting of weeds, i.e., targeted herbicidal application. The success of this management approach would require accurate identification of weeds early on in the production season.
Current weed management practices in corn and soybean rely on uniformly applying appropriate herbicides on the entire field instead on areas with weed presence, leading to economic losses and undesirable environmental impact (Kunz et al., 2018). Automated weed control for targeted herbicidal application using modern technologies such as machine learning and computer vision is gaining interest among the research community (Shamshiri et al., 2018).
Traditional computer vision techniques for weed identification relied on extracting important features such as vein patterns, color, shape, texture, etc. of different weeds. In the study of Li et al. (2013), image processing techniques were used to extract three important features namely shape, texture, and fractal dimensions from multispectral images. Computer vision was then used to identify weeds. Nevertheless, the extraction of important features is time consuming, subject to personal bias, and does not generalize well to unseen data. In order to improve generalization, machine learning techniques such as Support Vector Machines have been used to train computers for automatic weed identification (Ahmed et al., 2012). However, traditional machine learning also requires the extraction of important features from images which is time consuming and subject to bias as each scientist may give importance to different features. Deep learning, a subset of machine learning, provides the advantage of automatic feature extraction through the use of convolutional neural networks (CNN). Deep learning gained a lot of attention after CNNs were successfully used to identify 1000 different classes from 1.2 million images (Krizhevsky et al., 2012). Furthermore, advancements in computational efficiency and increased memory capacity over the last decade have enabled scientists to adapt deep learning networks for finding solutions in several fields including agriculture. Image classification and object detection are two common deep learning techniques that have been used for the identification of weeds.
Image classification methods are focused on identifying one species within an image by providing a softmax output. Use of softmax helps provide a probability of how accurately the class of interest is identified. Image classification has been used for weed identification by Smith et al. (2019), Olsen et al., 2019, Lammie et al., 2019, Espejo-Garcia et al., 2020, and Yu et al. (2019). In the study of Smith et al. (2019), image classification models trained using transfer learning on pre-trained models were able to maintain their performance for weed identification when the dataset size was reduced. VGG19, DenseNet, Xception, and IncRes pre-trained models were used by Espejo-Garcia et al. (2020) to show that transfer learning helped improve weed identification. A new DeepWeeds dataset was created by Olsen et al. (2019) and was used to train InceptionV3 and ResNet50 pre-trained models to classify multiple weed species found in Australian rangeland. Classification accuracy of 95% was reported after the models were trained on Keras framework. The same DeepWeeds dataset was also used by Lammie et al. (2019) with a 6–1 training–testing split. Accuracies of 91.08%, 89.40%, and 94.82% were reported for VGG16, DenseNet, and Wide Residual Netwrok pre-trained models respectively. In the study of Yu et al. (2019), a dataset of 36,000 images was used to train image classification models for identifying multiple weeds where VGGNet was reported as the dominant image classification pre-trained model with F1-scores of greater than 95% for each weed species.
Image classification, however, provides no information regarding the location of the weed and is unable to identify multiple instances or species of weeds within an image. In contrast, object detection, is a common deep learning method which uses bounding boxes to help provide the location of multiple weeds within an image. Object detection is of two types, two-stage and single-stage (Soviany & Ionescu, 2018). In two-stage object detection, Regional Proposal Network (RPN) is used to identify locations of interest and then the objects within those regions are classified. Other common two-stage object detection models such as RCNN (Girshick et al., 2014), Fast-RCNN (Girshick, 2015), Faster RCNN (Ren et al., 2015), and Mask-RCNN (He et al., 2017) also find mention in research literature. Among these, Faster RCNN was used for weed detection using RGB images (Veeranampalayam Sivakumar et al., 2020) while Mask-RCNN was used to detect weeds using multispectral imagery (Osorio et al., 2020). However, two-stage object detection models usually require longer inference times. In single-stage object detection, the algorithm learns about the bounding boxes and the objects within the images simultaneously. You Only Look Once (YOLO) (Redmon et al., 2016) and Single Shot Detector (SSD) (Liu et al., 2016) are two common single-stage object detection models. The third version of YOLO, YOLOv3 was introduced by Redmon & Farhadi (2018), which has become one of the most popular object detection models. YOLOv3 was used for object detection by Espinoza et al., 2020, Yu et al., 2019, Etienne and Saraswat, 2019, Sharpe et al., 2020a, and Gao et al. (2020) in order to identify and locate weeds within images. In the study of Espinoza et al. (2020), YOLOv3 was reported as being capable of detecting weeds faster than Faster-RCNN and with a greater accuracy than SSD. YOLOv3 was also used by Etienne & Saraswat (2019) where YOLOv3 was reported to detect weeds with accuracies of 54% to 98% and was superior to NVIDIA Deep Learning GPU Training System (DIGITS) as YOLOv3 was capable of processing larger bounding boxes. YOLOv3 was the preferred object detection model and provides a combined advantage of speed and accuracy when detecting objects of interest (Qu et al., 2018).
In this study, the performance of image classification and object detection models are reported. The term image classification refers to identification of individual weed species present in an image. Whereas the term object detection includes the task of locating and identifying presence of multiple weed instances and different species within an image. In addition, three pre-trained models for image classification were trained on two reference frameworks, namely Keras and PyTorch, for identifying weeds.
The remainder of the paper is divided into three parts. The next section will discuss the “Materials and Methods” which introduces the dataset used, explains the different deep learning techniques used, i.e., image classification and object detection, and explains different evaluation metrics for assessing results. The results and findings are then reported and discussed in the “Results and Discussion” section. The final evaluations and conclusions are reported in the “Conclusions” section followed by the last section on “Acknowledgements”.
Section snippets
Dataset
A common, annotated imagery dataset with a total of 462 images of four common, early-season weeds in corn and soybean was used in the interest of this study to train deep learning models for image classification and object detection. The dataset comprises of 35 cocklebur (Xanthium strumarium) images, 73 foxtail (Setaria viridis) images, 181 redroot pigweed (Amaranthus retroflexus) images, and 173 giant ragweed (Ambrosia trifida) images. The images for cocklebur, redroot pigweed, and giant
Image classification
The results obtained from training the image classification models are shown in Table 2. The best performance was observed when the VGG16 pre-trained model was trained on the PyTorch framework, which resulted in an overall testing accuracy of 98.90% and a F1-score of 99%. The models also trained 2x to 3x faster on the
PyTorch framework as compared to Keras. Overall, the average testing accuracy for models trained on the Keras deep learning framework was 93.4% and the average F1-score was 88.33%.
Conclusion
This study compared popular deep learning techniques, frameworks, and pre-trained models for weed identification to support development of a SSWM system. Images of four common weed species found in corn and soybean fields were used for training the models. The images consist of weeds in their early growth stages. During testing, image classification helped accurately identify individual weeds within images and PyTorch framework outperformed Keras in terms of speed and accuracy. Object detection
CRediT authorship contribution statement
Aanis Ahmad: Investigation, Software, Writing - original draft. Dharmendra Saraswat: Conceptualization, Supervision, and Arranging funding. Varun Aggarwal: Data curation, Validation, Writing - original draft. Aaron Etienne: Investigation. Benjamin Hancock: Software.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
The research was made possible by the funding provide by Purdue University's SURF program, Wabash Heartland Innovation Network grant number 18024589, Foundation of Food and Agricultural Research grant number 534662, USDA National Institute of Food and Agriculture Hatch project 1012501, and the Department of Agricultural and Biological Engineering at Purdue University.
References (39)
- et al.
Classification of crops and weeds from digital images: A support vector machine approach
Crop Prot.
(2012) - et al.
Towards weeds identification assistance through transfer learning
Comput. Electron. Agric.
(2020) - et al.
Recognising weeds in a maize crop using a random forest machine-learning algorithm and near-infrared snapshot mosaic hyperspectral imagery
Biosyst. Eng.
(2018) - et al.
Development and evaluation of a low-cost and smart technology for precision weed management utilizing artificial intelligence
Comput. Electron. Agric.
(2019) - et al.
Apple detection during different growth stages in orchards using the improved yolo-v3 model
Comput. Electron. Agric.
(2019) - et al.
Deep learning for image-based weed detection in turfgrass
Eur. J. Agron.
(2019) - Atwood, D., & Paisley-Jones, C. (2017). Pesticides industry sales and usage: 2008–2012 market estimates. US...
- et al.
Performance of deep learning architectures and transfer learning for detecting glaucomatous optic neuropathy in fundus photographs
Sci. Rep.
(2018) - Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image...
- Espinoza, M. A. M., Le, C. Z., Raheja, A., & Bhandari, S. (2020). Weed identification and removal using machine...