Original papers
Performance of deep learning models for classifying and detecting common weeds in corn and soybean production systems

https://doi.org/10.1016/j.compag.2021.106081Get rights and content

Highlights

  • Object detection and image classification helped identify weeds in corn and soybean.

  • YOLOv3 was a fast and accurate model for weed localization and identification.

  • VGG16 was the best image classification pre-trained model for weed identification.

  • PyTorch framework yielded faster training and more accurate models than Keras.

  • Deep learning models had difficulty identifying unbalanced classes and grassy weeds.

Abstract

Knowing precise location and having accurate information about weed species is a prerequisite for developing an effective site-specific weed management (SSWM) system. Due to the effectiveness of deep learning techniques for vision-based tasks such as image classification and object detection, its use for discriminating between weeds and crops is gaining acceptance among the agricultural research community. However, limited studies have used deep learning for identifying multiple weeds in a single image and most of the studies have not compared the effectiveness of deep learning based image classification and object detection by using a common, annotated imagery dataset of early season weeds under field conditions. This study addresses the research gap by evaluating comparative performance of three different pre-trained image classification models for classifying weed species and also assesses the performance of an object detection model for locating and identifying weed species. The image classification models were trained on two commonly used deep learning frameworks i.e., Keras and PyTorch, to assess any performance differential due to the choice of framework. An annotated dataset comprising of RGB images of four, early season weeds, found in corn and soybean production system in the Midwest US, namely, cocklebur (Xanthium strumarium), foxtail (Setaria viridis), redroot pigweed (Amaranthus retroflexus), and giant ragweed (Ambrosia trifida) was used in this study. VGG16, ResNet50, and InceptionV3 pre-trained models were used for image classification. The object detection model, based on the You Only Look Once (YOLOv3) library, was trained to locate and identify different weed species within an image. The performance of image classification models was assessed using testing accuracy and F1-score metrics. Average precision (AP) and mean average precision (mAP) were used to assess the performance of the object detection model. The best performing image classification model was VGG16 with an accuracy of 98.90% and an F1-score of 99%. Faster training times and higher accuracies were observed with PyTorch. The detection model helped locate and identify multiple weeds within an image with AP scores of 43.28%, 26.30%, 89.89%, and 57.80% for cocklebur, foxtail, redroot pigweed, and giant ragweed respectively and an overall mAP score of 54.3%. The results suggest that under field conditions, use of pre-trained models for image classification and YOLOv3 for object detection are promising for identifying single and multiple weeds, respectively, given that sufficient data is available. Additionally, unlike image classification, the localization capabilities of object detection are desirable for developing a system for SSWM.

Introduction

Weeds have been a major concern to crop production due to a negative influence on yield. It is reported to be responsible for reducing 30% of crop yield worldwide (Gao et al., 2018). Potential economic losses from weeds equal more than $25 billion on an annual basis for corn and soybean crops in North America alone (Soltani et al., 2016). A typical weed control strategy involves application of herbicide on over 90% of field crops in the US (Gianessi & Reigner, 2007). In 2012, pesticides accounted for 5% of the total farm expenditure in the US, of which 58% was used for herbicides (Atwood & Paisley-Jones, 2017). In addition, according to the European Food Safety Authority (EFSA) approximately 99% of food products contain some residue from these herbicides (Partel et al., 2019). It is therefore crucial to develop effective and efficient weed control methods to harvest crops with minimum chemical residues. One such solution includes applying herbicide to only specified regions of a field consisting of weeds, i.e., targeted herbicidal application. The success of this management approach would require accurate identification of weeds early on in the production season.

Current weed management practices in corn and soybean rely on uniformly applying appropriate herbicides on the entire field instead on areas with weed presence, leading to economic losses and undesirable environmental impact (Kunz et al., 2018). Automated weed control for targeted herbicidal application using modern technologies such as machine learning and computer vision is gaining interest among the research community (Shamshiri et al., 2018).

Traditional computer vision techniques for weed identification relied on extracting important features such as vein patterns, color, shape, texture, etc. of different weeds. In the study of Li et al. (2013), image processing techniques were used to extract three important features namely shape, texture, and fractal dimensions from multispectral images. Computer vision was then used to identify weeds. Nevertheless, the extraction of important features is time consuming, subject to personal bias, and does not generalize well to unseen data. In order to improve generalization, machine learning techniques such as Support Vector Machines have been used to train computers for automatic weed identification (Ahmed et al., 2012). However, traditional machine learning also requires the extraction of important features from images which is time consuming and subject to bias as each scientist may give importance to different features. Deep learning, a subset of machine learning, provides the advantage of automatic feature extraction through the use of convolutional neural networks (CNN). Deep learning gained a lot of attention after CNNs were successfully used to identify 1000 different classes from 1.2 million images (Krizhevsky et al., 2012). Furthermore, advancements in computational efficiency and increased memory capacity over the last decade have enabled scientists to adapt deep learning networks for finding solutions in several fields including agriculture. Image classification and object detection are two common deep learning techniques that have been used for the identification of weeds.

Image classification methods are focused on identifying one species within an image by providing a softmax output. Use of softmax helps provide a probability of how accurately the class of interest is identified. Image classification has been used for weed identification by Smith et al. (2019), Olsen et al., 2019, Lammie et al., 2019, Espejo-Garcia et al., 2020, and Yu et al. (2019). In the study of Smith et al. (2019), image classification models trained using transfer learning on pre-trained models were able to maintain their performance for weed identification when the dataset size was reduced. VGG19, DenseNet, Xception, and IncRes pre-trained models were used by Espejo-Garcia et al. (2020) to show that transfer learning helped improve weed identification. A new DeepWeeds dataset was created by Olsen et al. (2019) and was used to train InceptionV3 and ResNet50 pre-trained models to classify multiple weed species found in Australian rangeland. Classification accuracy of 95% was reported after the models were trained on Keras framework. The same DeepWeeds dataset was also used by Lammie et al. (2019) with a 6–1 training–testing split. Accuracies of 91.08%, 89.40%, and 94.82% were reported for VGG16, DenseNet, and Wide Residual Netwrok pre-trained models respectively. In the study of Yu et al. (2019), a dataset of 36,000 images was used to train image classification models for identifying multiple weeds where VGGNet was reported as the dominant image classification pre-trained model with F1-scores of greater than 95% for each weed species.

Image classification, however, provides no information regarding the location of the weed and is unable to identify multiple instances or species of weeds within an image. In contrast, object detection, is a common deep learning method which uses bounding boxes to help provide the location of multiple weeds within an image. Object detection is of two types, two-stage and single-stage (Soviany & Ionescu, 2018). In two-stage object detection, Regional Proposal Network (RPN) is used to identify locations of interest and then the objects within those regions are classified. Other common two-stage object detection models such as RCNN (Girshick et al., 2014), Fast-RCNN (Girshick, 2015), Faster RCNN (Ren et al., 2015), and Mask-RCNN (He et al., 2017) also find mention in research literature. Among these, Faster RCNN was used for weed detection using RGB images (Veeranampalayam Sivakumar et al., 2020) while Mask-RCNN was used to detect weeds using multispectral imagery (Osorio et al., 2020). However, two-stage object detection models usually require longer inference times. In single-stage object detection, the algorithm learns about the bounding boxes and the objects within the images simultaneously. You Only Look Once (YOLO) (Redmon et al., 2016) and Single Shot Detector (SSD) (Liu et al., 2016) are two common single-stage object detection models. The third version of YOLO, YOLOv3 was introduced by Redmon & Farhadi (2018), which has become one of the most popular object detection models. YOLOv3 was used for object detection by Espinoza et al., 2020, Yu et al., 2019, Etienne and Saraswat, 2019, Sharpe et al., 2020a, and Gao et al. (2020) in order to identify and locate weeds within images. In the study of Espinoza et al. (2020), YOLOv3 was reported as being capable of detecting weeds faster than Faster-RCNN and with a greater accuracy than SSD. YOLOv3 was also used by Etienne & Saraswat (2019) where YOLOv3 was reported to detect weeds with accuracies of 54% to 98% and was superior to NVIDIA Deep Learning GPU Training System (DIGITS) as YOLOv3 was capable of processing larger bounding boxes. YOLOv3 was the preferred object detection model and provides a combined advantage of speed and accuracy when detecting objects of interest (Qu et al., 2018).

In this study, the performance of image classification and object detection models are reported. The term image classification refers to identification of individual weed species present in an image. Whereas the term object detection includes the task of locating and identifying presence of multiple weed instances and different species within an image. In addition, three pre-trained models for image classification were trained on two reference frameworks, namely Keras and PyTorch, for identifying weeds.

The remainder of the paper is divided into three parts. The next section will discuss the “Materials and Methods” which introduces the dataset used, explains the different deep learning techniques used, i.e., image classification and object detection, and explains different evaluation metrics for assessing results. The results and findings are then reported and discussed in the “Results and Discussion” section. The final evaluations and conclusions are reported in the “Conclusions” section followed by the last section on “Acknowledgements”.

Section snippets

Dataset

A common, annotated imagery dataset with a total of 462 images of four common, early-season weeds in corn and soybean was used in the interest of this study to train deep learning models for image classification and object detection. The dataset comprises of 35 cocklebur (Xanthium strumarium) images, 73 foxtail (Setaria viridis) images, 181 redroot pigweed (Amaranthus retroflexus) images, and 173 giant ragweed (Ambrosia trifida) images. The images for cocklebur, redroot pigweed, and giant

Image classification

The results obtained from training the image classification models are shown in Table 2. The best performance was observed when the VGG16 pre-trained model was trained on the PyTorch framework, which resulted in an overall testing accuracy of 98.90% and a F1-score of 99%. The models also trained 2x to 3x faster on the

PyTorch framework as compared to Keras. Overall, the average testing accuracy for models trained on the Keras deep learning framework was 93.4% and the average F1-score was 88.33%.

Conclusion

This study compared popular deep learning techniques, frameworks, and pre-trained models for weed identification to support development of a SSWM system. Images of four common weed species found in corn and soybean fields were used for training the models. The images consist of weeds in their early growth stages. During testing, image classification helped accurately identify individual weeds within images and PyTorch framework outperformed Keras in terms of speed and accuracy. Object detection

CRediT authorship contribution statement

Aanis Ahmad: Investigation, Software, Writing - original draft. Dharmendra Saraswat: Conceptualization, Supervision, and Arranging funding. Varun Aggarwal: Data curation, Validation, Writing - original draft. Aaron Etienne: Investigation. Benjamin Hancock: Software.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The research was made possible by the funding provide by Purdue University's SURF program, Wabash Heartland Innovation Network grant number 18024589, Foundation of Food and Agricultural Research grant number 534662, USDA National Institute of Food and Agriculture Hatch project 1012501, and the Department of Agricultural and Biological Engineering at Purdue University.

References (39)

  • Etienne, A., & Saraswat, D. (2019). Machine learning approaches to automate weed detection by uav based sensors. In...
  • M. Everingham et al.

    The pascal visual object classes (voc) challenge

    Int. J. Comput. Vision

    (2010)
  • J. Gao et al.

    Deep convolutional neural networks for image-based convolvulus sepium detection in sugar beet fields

    Plant Methods

    (2020)
  • L.P. Gianessi et al.

    The value of herbicides in us crop production

    Weed Technol.

    (2007)
  • Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE international conference on computer vision (pp....
  • R. Girshick et al.

    Rich feature hierarchies for accurate object detection and semantic segmentation

  • He, K., Gkioxari, G., Dolĺar, P., & Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international...
  • K. He et al.

    Deep residual learning for image recognition

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks....
  • Cited by (0)

    View full text