Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Zaji, Amirhossein
Doctor of Philosophy - PhD
University of British Columbia
Wheat plant phenotyping is crucial for plant breeding and crop management. Traditional methods, however, are laborious and error prone. Computer vision and Deep Learning (DL) can address these issues by quickly and accurately analyzing wheat plant images, but data collection, annotation, and preprocessing for DL models can be costly and time-consuming. This thesis aims to reduce data preparation time for high-throughput wheat plant phenotyping using DL models. It explores the use of readily available sensors like cellphone RGB cameras for image acquisition and develops more efficient and automated annotation and preprocessing methods. The proposed aug- mentation algorithm reduces the number of training samples needed for accurate DL models, while transfer learning and other techniques are investigated to de- crease the amount of labeled data required. A state-of-the-art framework for localizing and counting wheat spikes using dotted annotation datasets and generation algorithms is proposed and evaluated. This approach is faster than bounding box and polygon annotation methods for wheat spike localization and counting. The computational component employs hybrid UNet architectures, and the proposed models demonstrate improved spike localization and counting compared to previous research studies. This research introduces Automatic Object Level Augmentation (AutoOLA), a novel augmentation algorithm that reduces DL model training sample requirements. AutoOLA augments spikes, leaves, stems, and backgrounds in wheat images separately using their optimized augmentation policies. Results indicate that AutoOLA can improve wheat spike counting by 60%. Stereovision cameras, which are more user-friendly, are employed for plant height measurement. An active learning model is used to annotate images from one camera and transfer them to the other camera’s images to expedite the process. Mask R-CNN is utilized for spike localization, and spike heights are estimated using depth layers. In conclusion, this thesis develops methods to reduce time and effort for apply- ing DL models in plant phenotyping. It investigates image acquisition with accessible sensors and develops more efficient annotation and preprocessing methods, including the proposed augmentation algorithm. By making DL models more accessible and scalable, this research advances plant breeding, genetics, and crop production.
HIGH-THROUGHPUT WHEAT PLANTPHENOTYPING USING DEEP LEARNINGTECHNIQUESbyAmirhossein ZajiB.Sc., Razi University, Kermanshah, IranM.Sc., Bahonar University, Kerman, IranA THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinTHE COLLEGE OF GRADUATE STUDIES(Civil Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)May 2023© Amirhossein Zaji, 2023The following individuals certify that they have read, and recommend to the Col-lege of Graduate Studies for acceptance, the thesis/dissertation entitled:HIGH-THROUGHPUT WHEAT PLANT PHENOTYPING USING DEEP LEARNINGTECHNIQUESsubmitted by Amirhossein Zaji in partial fulfillment of the requirements of thedegree of Doctor of Philosophy.Dr. Zheng Liu, School of EngineeringSupervisorDr. Nicolas Peleato, School of EngineeringSupervisory Committee MemberDr. Pankaj Bhowmik, Aquatic and Crop Resource Development, National ResearchCouncil of CanadaSupervisory Committee MemberDr. Hadi Mohammadi, School of EngineeringUniversity ExaminerDr. Ian Stavness, University of SaskatchewanExternal ExamineriiAbstractWheat plant phenotyping is crucial for plant breeding and crop management. Tra-ditional methods, however, are laborious and error prone. Computer vision andDeep Learning (DL) can address these issues by quickly and accurately analyz-ing wheat plant images, but data collection, annotation, and preprocessing for DLmodels can be costly and time-consuming.This thesis aims to reduce data preparation time for high-throughput wheatplant phenotyping using DL models. It explores the use of readily available sen-sors like cellphone RGB cameras for image acquisition and develops more effi-cient and automated annotation and preprocessing methods. The proposed aug-mentation algorithm reduces the number of training samples needed for accurateDL models, while transfer learning and other techniques are investigated to de-crease the amount of labeled data required.A state-of-the-art framework for localizing and counting wheat spikes usingdotted annotation datasets and generation algorithms is proposed and evaluated.This approach is faster than bounding box and polygon annotation methods forwheat spike localization and counting. The computational component employshybrid UNet architectures, and the proposed models demonstrate improved spikelocalization and counting compared to previous research studies.This research introduces Automatic Object Level Augmentation (AutoOLA),a novel augmentation algorithm that reduces DL model training sample require-ments. AutoOLA augments spikes, leaves, stems, and backgrounds in wheat im-ages separately using their optimized augmentation policies. Results indicate thatAutoOLA can improve wheat spike counting by 60%.iiiStereovision cameras, which are more user-friendly, are employed for plantheight measurement. An active learning model is used to annotate images fromone camera and transfer them to the other camera’s images to expedite the process.Mask R-CNN is utilized for spike localization, and spike heights are estimatedusing depth layers.In conclusion, this thesis develops methods to reduce time and effort for apply-ing DL models in plant phenotyping. It investigates image acquisition with acces-sible sensors and develops more efficient annotation and preprocessing methods,including the proposed augmentation algorithm. By making DL models more ac-cessible and scalable, this research advances plant breeding, genetics, and cropproduction.ivLay SummaryPlant characterization plays a crucial role in managing crops and enhancing theirgrowth. Previous research studies have shown that advanced data analysis tech-niques, such as machine learning and computer vision, can be used to alleviate thecostly and time-consuming nature of conventional plant characterization methods.This research aims to reduce the time and effort needed for preparing datawhen using these computer models in plant analysis. It investigates the use ofaffordable cameras for image collection, automated annotation and preprocessingmethods to streamline data handling, a smart algorithm to minimize the number oftraining samples needed, and a transfer learning technique to reduce the amountof labeled data required for training the models.The study’s ultimate goal is to make plant analysis computer models more ac-cessible and scalable for researchers and practitioners. These models may improvecrop productivity and sustainability by simplifying data preparation, benefiting theagriculture industry.vPrefaceThe present dissertation is premised upon an investigation that was undertaken atthe Okanagan Campus of The University of British Columbia, within the Schoolof Engineering, and under the guidance of Dr. Zheng Liu. I was responsible for themajority of the research activities, which included the comprehensive examinationof relevant literature, the formulation of the research question, the development ofthe relevant algorithms, and the composition of the final manuscript. Dr. ZhengLiu provided invaluable assistance through offering research ideas and providingconstructive feedback to enhance the quality of the work.Chapter 3. Wheat Spike Localization and Counting via Hybrid UNet Architec-tures is based on the following published scholarly articles:Zaji, A., Liu, Z., Xiao, G., Bhowmik, P., Sangha, J. S., & Ruan, Y. (2022).“Wheat spike localization and counting via hybrid UNet architectures.” Comput-ers and Electronics in Agriculture, 203, 107439. https://www.sciencedirect.com/science/article/abs/pii/S0168169922007475Zaji, A., Liu, Z., Xiao, G., Bhowmik, P., Sangha, J. S., & Ruan, Y. (2021, Au-gust). Wheat Spike Counting Using Regression and Localization Approaches. In2021 IEEE International Conference on Imaging Systems and Techniques (IST)(pp. 1-6). IEEE. https://ieeexplore.ieee.org/abstract/document/9651407Chapter 4. AutoOLA: Automatic Object Level Augmentation for Wheat SpikesviCounting is based on the following published scholarly articles:Zaji, A., Liu, Z., Xiao, G., Bhowmik, P., Sangha, J. S., & Ruan, Y. (2023).“AutoOLA: Automatic Object Level Augmentation for Wheat Spikes Counting.”Computers and Electronics in Agriculture. https://www.sciencedirect.com/science/article/pii/S016816992300011X?dgcid=coauthorZaji, A., Liu, Z., Xiao, G., Bhowmik, P., Sangha, J. S., & Ruan, Y. (2022, May).Wheat spikes counting using object-level data augmentation. In 2022 IEEE Inter-national Instrumentation and Measurement Technology Conference (I2MTC) (pp.1-6). IEEE. https://ieeexplore.ieee.org/abstract/document/9806479Chapter 5. Wheat Spikes Height Estimation Using Stereo Cameras is based onthe following scholarly article:Zaji, A., Liu, Z., Xiao, G., Sangha, J. S., & Ruan, Y. (2022). “Wheat spikesheight estimation using stereo cameras.” IEEE Transactions on AgriFood Elec-tronics. (Under Review)Furthermore, we put forth an ontology-based knowledge system as a means ofstudying literature within the domain of wheat plant phenotyping, which is ex-pounded upon in the appendix section of this dissertation. Appendix A. Ontology-based knowledge management for deep learning applications in wheat phenotyp-ing is founded upon the following published scholarly article:Zaji, A., Liu, Z., Xiao, G., Sangha, J. S., & Ruan, Y. (2022). “A survey on deeplearning applications in wheat phenotyping.” Applied Soft Computing, 109761.https://www.sciencedirect.com/science/article/abs/pii/S1568494622008109viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . . 11.2 Research Gaps and objectives . . . . . . . . . . . . . . . . . . . . 41.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.1 Wheat Growth Monitoring . . . . . . . . . . . . . . . . . . . . . 112.1.1 Wheat Spike Localization and Counting . . . . . . . . . . 112.1.2 Wheat Height Estimation . . . . . . . . . . . . . . . . . . 14viii2.2 Data Analytic Methods . . . . . . . . . . . . . . . . . . . . . . . 172.2.1 Data Augmentation Techniques . . . . . . . . . . . . . . 172.2.2 Active Learning in High-Throughput Phenotyping . . . . . 193 Wheat Spike Localization and Counting via Hybrid UNet Architec-tures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2 Wheat Spikes Localization and Counting Methodology . . . . . . 223.2.1 Overview of the proposed approach . . . . . . . . . . . . 233.2.2 DL architectures . . . . . . . . . . . . . . . . . . . . . . 233.2.3 Data augmentation . . . . . . . . . . . . . . . . . . . . . 293.2.4 Density map generation . . . . . . . . . . . . . . . . . . . 303.2.5 Wheat spikes localization and counting . . . . . . . . . . 333.2.6 Data preparation . . . . . . . . . . . . . . . . . . . . . . 353.2.7 Model evaluation . . . . . . . . . . . . . . . . . . . . . . 403.3 Experimental Results and Discussion . . . . . . . . . . . . . . . . 433.3.1 Modeling the benchmark dataset (ACID) . . . . . . . . . 443.3.2 Modeling the in-field dataset (GWHD) . . . . . . . . . . . 523.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584 AutoOLA: Automatic Object Level Augmentation for Wheat SpikesCounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.2 AutoOLA Methodology . . . . . . . . . . . . . . . . . . . . . . . 624.2.1 Objects Extracting and Assembling . . . . . . . . . . . . 634.2.2 Augmentation Policy Structures . . . . . . . . . . . . . . 664.2.3 Augmentation Policy Optimization . . . . . . . . . . . . . 674.2.4 Architecture of the F-CNN and C-CNN Models . . . . . . 734.2.5 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . 754.3 Experimental Results and Discussion . . . . . . . . . . . . . . . . 754.3.1 Dataset Description . . . . . . . . . . . . . . . . . . . . . 76ix4.3.2 Design of the Models . . . . . . . . . . . . . . . . . . . . 774.3.3 Optimization Results . . . . . . . . . . . . . . . . . . . . 784.3.4 Wheat Spikes Prediction . . . . . . . . . . . . . . . . . . 794.3.5 AutoOLA-Based Model in Real-World Situation . . . . . 814.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845 Wheat Spikes Height Estimation Using Stereo Cameras . . . . . . . 925.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925.2 Wheat Spikes Height Estimation Methodology . . . . . . . . . . . 935.2.1 Greenhouse Experiment . . . . . . . . . . . . . . . . . . 945.2.2 Dataset Annotation . . . . . . . . . . . . . . . . . . . . . 955.2.3 Transferring the OAKD labels to D455 . . . . . . . . . . . 975.2.4 Wheat Spikes Localization . . . . . . . . . . . . . . . . . 995.2.5 Wheat Spikes Height Estimation . . . . . . . . . . . . . . 1025.2.6 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . 1045.3 Experimental Results and Discussion . . . . . . . . . . . . . . . . 1065.3.1 Results of Wheat Spikes Localization . . . . . . . . . . . 1075.3.2 Results of Wheat Spikes Height Estimation . . . . . . . . 1085.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1166 Conclusions, Contributions and Recommendations for Future Re-search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1186.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1186.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1196.3 Future Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122Appendix A Ontology-based knowledge management for deep learningapplications in wheat phenotyping . . . . . . . . . . . . . . . . . . 145xList of TablesTable 3.1 Proposed Deep Learning (DL) architectures. . . . . . . . . . . 28Table 3.2 Evaluating the wheat spike localization on the ACID testingdataset using F1 score. . . . . . . . . . . . . . . . . . . . . . . 45Table 3.3 Evaluating the wheat spike counting on the ACID testing datasetusing Root Mean Squared Error (RMSE) and Mean AbsolutePercentage Error (MAPE) indices. . . . . . . . . . . . . . . . . 51Table 3.4 Comparing the in-field wheat spikes localization and count-ing performance of the proposed model with previous researchstudies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57Table 4.1 The search space of magnitude for each transformation typeand the impact ranges of minmag and magmax. . . . . . . . . . 71Table 4.2 The defined ranges for the number of ROIs in the augmentedimages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Table 5.1 The OAKD and D455 cameras’ technical specifications . . . . 95Table 5.2 Wheat spikes localization performance based on Instance Seg-mentation (IS) and Object Detection (OD) models. . . . . . . . 107Table 5.3 Wheat spikes height prediction performance. . . . . . . . . . . 109Table 5.4 The percentage of wheat spikes whose height was indirectlyestimated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112xiList of FiguresFigure 2.1 The original image (left column), spikes annotated (middlecolumn), and spikelets annotated (right column) of ACID dataset. 12Figure 3.1 Overview of the proposed wheat spike localization and count-ing procedure. . . . . . . . . . . . . . . . . . . . . . . . . . . 24Figure 3.2 (a) VGG16-UNet and (b) ResNet50-UNet/ResNeXt-UNet ar-chitectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Figure 3.3 Data augmentation procedure. . . . . . . . . . . . . . . . . . 31Figure 3.4 Difference between CDM and GDM. . . . . . . . . . . . . . . 33Figure 3.5 The proposed wheat spike counting approach based on Non-Maximal Suppression (NMS) and Density Map Estimation(DME) algorithms. . . . . . . . . . . . . . . . . . . . . . . . 35Figure 3.6 Challenges of the ACID dataset wheat localization and count-ing. (a) Problematic annotations. (b) Closed spikes. (c) Oc-clusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Figure 3.7 Resizing and cropping procedures of input images. . . . . . . 38Figure 3.8 A selection of the Global Wheat Head Dataset (GWHD) wheatimages with their associated dot annotations. . . . . . . . . . 39Figure 3.9 An example of TP, False Positive (FP), and FN. . . . . . . . . 42xiiFigure 3.10 Localization performance of Deep Learning (DL) models trainedusing different procedures on the ACID testing dataset. (a)compares the DL models using fixed weights in the featureextraction part (P) and DL models without any fixed weights(S). (b) compares the DL models with Gaussian (G) and Con-stant (C) density maps. . . . . . . . . . . . . . . . . . . . . . 47Figure 3.11 Wheat spike localization performance on the ACID datasetover (a) training and (b) testing procedures for the first 600epochs and different SDT amounts. . . . . . . . . . . . . . . . 48Figure 3.12 Some examples of VGG16-UNet results on the ACID testingdataset. The left column is the cropped 256×256 pixels im-age, the middle column is the corresponding ground truth an-notation using the CDM algorithm, and the right column is theDL model’s output. The red rectangles in this figure illustratesome of the challenging predictions of the used DL model. . . 49Figure 3.13 Augmentation effect on the ACID testing dataset for the con-sidered models with SDT of 30, 20, and 10. . . . . . . . . . . 50Figure 3.14 Results of wheat spikes localization and counting on the GWHDtesting dataset. (a) shows the localization results using F1score. (b) shows the counting results using MAPE. (c) com-pares the model without any additional training on the GWHDdataset to the model that was retrained for 200 epochs using97 samples from the GWHD dataset. . . . . . . . . . . . . . . 53Figure 3.15 Some examples of wheat spikes localization model on theGWHD testing dataset. The left column is the cropped 256×256pixels image, the middle column is the corresponding groundtruth annotation using the CDM algorithm, and the right col-umn is the DL model’s output. The red rectangles in this figureillustrate some of the challenging predictions of the used DLmodel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55xiiiFigure 4.1 Overview of the AutoOLA algorithm. . . . . . . . . . . . . . 63Figure 4.2 Some examples of an in-field image’s spike, leaf, stem, andbackground ROIs. . . . . . . . . . . . . . . . . . . . . . . . . 65Figure 4.3 The structure of the (a) 2aug and (b) 5aug policies. . . . . . . 68Figure 4.4 An example of the employed transformation types with ran-dom magnitudes. . . . . . . . . . . . . . . . . . . . . . . . . 69Figure 4.5 (a) F-CNN and (b) C-CNN architectures. . . . . . . . . . . . 73Figure 4.6 Some examples of the (a) GWHD dataset and (b) ETHZ-1sub-dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . 77Figure 4.7 Some examples of the (a) AugMix, (b) AutoAugment, (c) Cut-Mix, (d) Mixup, (e) RandAugment, (f) Random Erasing, (g)2aug-minmag, (h) 5aug-minmag, (i) 2aug-magmax, and (j)5aug-magmax based augmented images. . . . . . . . . . . . . 86Figure 4.8 Performance of the AutoOLA techniques during the optimiza-tion iterations. . . . . . . . . . . . . . . . . . . . . . . . . . . 87Figure 4.9 Reduced performance of the models according to (a) RMSE,(b) MAE, and (c) MAPE with only one training sample in com-parison with the basic-all model as the benchmark. . . . . . . 88Figure 4.10 Comparing the ground truth number of spikes with the pre-dicted amounts for the testing samples of (a) basic-all, (b)basic-one, (c) 5aug-magmax, (d) 2aug-magmax, (e) 5aug-minmag,(f) 2aug-minmag, (g) AugMix, (h) AutoAugment, (i) CutMix,(j) Mixup, (k) RandAugment, and (l) Random Erasing basedmodels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Figure 4.11 Some examples of the randomly cropped images, the groundtruth DA masks, and the predicted DA masks. . . . . . . . . . 90Figure 4.12 Reduced performance of the basic-one and 5aug-magmax mod-els tested on (a) ETHZ-1 sub-dataset and (b) GWHD datasetin comparison with the basic-all model as the benchmark. . . . 91Figure 5.1 The greenhouse experiment’s setup. . . . . . . . . . . . . . . 94xivFigure 5.2 An example of RGB images produced by (a) OAKD and (b)D455 cameras. . . . . . . . . . . . . . . . . . . . . . . . . . 95Figure 5.3 Wheat spikes (a) bounding box and (b) polygon annotation. . . 96Figure 5.4 An overview of the annotation process for the D455 and OAKDdatasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97Figure 5.5 A Scale-Invariant Feature Transform (SIFT) transformationexample; (a,b) the original OAKD and D455 images with thematched key points, (c) the matched OAKD image within theD455 image, and (d) the transformed D455 image based onthe OAKD image. . . . . . . . . . . . . . . . . . . . . . . . . 98Figure 5.6 An illustration of (a) a cropped OAKD image and (b) a croppedD455 image transformed by SIFT, and (c) an overview of thedeveloped active learning algorithm. . . . . . . . . . . . . . . 100Figure 5.7 The performance of the developed active learning RandomForest (RF) model in predicting the transformations. . . . . . 100Figure 5.8 An example of cropped RGB images using (a) OAKD and (b)D455 cameras. . . . . . . . . . . . . . . . . . . . . . . . . . 102Figure 5.9 OAKD depth layer (a) prior to correction and (b) subsequentto correction. . . . . . . . . . . . . . . . . . . . . . . . . . . 103Figure 5.10 An illustration of (a) OAKD and (b) D455 RGB-D images. . . 104Figure 5.11 The Intersection over Union (IoU) of the OAKD and D455-based Instance Segmentation (IS) and Object Detection (OD)models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108Figure 5.12 Scatter plot of wheat height prediction performance in spikeand plot levels. . . . . . . . . . . . . . . . . . . . . . . . . . 110Figure 5.13 Comparing the ground truth depth estimation of OAKD andD455 cameras with manually measured ground truth. . . . . . 111Figure 5.14 An example of a D455 plot with its problematic depth layer. . 111Figure A.1 Structure of the classes and object properties of the presentedontology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147xvFigure A.2 Sample queries and individuals’ information of Goals, Model Goals,Model Name, and Articles classes. . . . . . . . . . . . . . . . 148Figure A.3 The structure of the data properties used in the presented on-tology and an example of their values for one of the papers ofthe Articles class. . . . . . . . . . . . . . . . . . . . . . . . . 149Figure A.4 The structure of the annotation properties used in the pre-sented ontology and an example of their values for one of thepapers of the Articles class. . . . . . . . . . . . . . . . . . . . 150xviGlossaryR2 coefficient of determination. 105, 106FN False Negative. 40, 41, 51, 104, 105, 113FP False Positive. xii, 40–42, 51, 104, 105, 113MAE Mean Absolute Error. 13, 62, 75, 79, 105, 106, 109MAPA Mean Absolute Percentage Accuracy. 56, 57MAPE Mean Absolute Percentage Error. 22, 43, 49, 50, 54, 56, 58, 59, 62, 75,79, 80, 82, 93, 105, 106, 109MedAE Median Absolute Error. 105, 106, 109RMSE Root Mean Squared Error. 43, 56, 62, 75, 79, 105, 106, 109SDT Successful Distance Threshold. 41–46, 48TN True Negative. 104, 105, 113TP True Positive. 40, 41, 51, 104, 113rRMSE Relative Root Mean Squared Error. 56, 57AutoOLA Automatic Object Level Augmentation. iii, 5, 7, 62, 64, 68, 71–73,75, 76, 78–85, 119, 120xviiC-CNN Child-Convolutional Neural Network. 73, 74, 76, 79, 80, 85CDM Constant Density Map. 23, 32, 43, 44, 46, 52, 58CNN Convolutional Neural Network. 11, 13, 18, 73, 101DA Dot Annotation. 63, 64, 66, 67, 76, 120, 121DL Deep Learning. iii, iv, xi, xiii, 3–5, 7, 11, 12, 16, 17, 19, 22–25, 28, 29,33–36, 38–41, 43, 44, 46–49, 51–59, 62, 66, 74, 76, 78–81, 83–85, 92–97,102, 107, 112, 114–116, 118–121, 145, 147DME Density Map Estimation. xii, 35, 120F-CNN Full-Convolutional Neural Network. 62, 73, 74, 76, 80FCN Fully Convolutional Networks. 11–13, 24, 101GA Genetic Algorithm. 71, 78, 79GAN Generative Adversarial Networks. 17, 58GDM Gaussian Density Map. 23, 32, 44–46, 48, 58GPU Graphics processing units. 2, 16GWHD Global Wheat Head Dataset. xii–xiv, 7, 22, 36, 38–40, 44, 52–57, 59,76, 77, 81, 82HOG Histograms of Oriented Gradients. 3, 98IoU Intersection over Union. xv, 101, 107, 108, 114IS Instance Segmentation. xi, xv, 92, 93, 95–97, 101, 102, 104, 106–108, 111–113, 116xviiiLiDAR Light Detection And Ranging. 14LR Linear Regression. 23, 34, 35, 39, 48, 101Mask R-CNN Mask Region-based Convolutional Neural Networks. 92, 99, 101,116ML Machine Learning. 3, 4NMS Non-Maximal Suppression. xii, 5, 23, 33–35, 40, 41, 43, 46, 48, 51, 58,120OD Object Detection. xi, xv, 92, 95–97, 101, 102, 104, 106–108, 111–113, 116PA Polygon Annotation. 63, 64, 67, 76, 120, 121PLT Pixel-Level Transformation. 64, 67R-CNN Region-based Convolutional Neural Networks. 99, 101RF Random Forest. xv, 13, 93, 99, 100, 116ROI Region of Interest. 62–67, 69, 70, 82–84SIFT Scale-Invariant Feature Transform. xv, 3, 93, 96, 98–100, 102, 116SLT Spatial-Level Transformation. 64, 67SVM Support Vector Machine. 3, 101UAV Unmanned Aerial Vehicle. 15, 121xixAcknowledgementsI would like to express my sincere gratitude to my supervisor, Dr. Zheng Liu, fortheir exceptional guidance, mentorship, and support throughout my Ph.D. journey.From the beginning, Dr. Liu has been a constant source of inspiration, wisdom,and encouragement. Their insightful advice and guidance have been invaluable inshaping my research and in helping me to achieve my goals. Dr. Liu has not onlyserved as a supervisor, but also as a mentor and a friend. Their passion for thefield and dedication to students have been a constant source of inspiration.I am also greatly indebted to my supervisory committee members, Dr. PankajBhowmik and Dr. Nicolas Peleato, for their insightful comments, suggestions, andguidance. Their expertise and support have been essential in shaping my research.I would also like to express my gratitude to Dr. Ian Stavness and Dr. HadiMohammadi for their service as external and university examiners, respectively.Their critical evaluations and feedback have been invaluable in the final stages ofmy thesis.I am also grateful to my supervisors from the National Research Council ofCanada (NRC) and Agriculture and Agri-Food, Dr. Gaozhi Xiao, Dr. Jatinder S.Sangha, and Dr. Yuefeng Ruan, for their valuable input, support, and collaborationthroughout my research. Their expertise and guidance have been instrumental inshaping the direction of my research and in helping me to achieve my goals.I would like to extend my thanks to my colleagues at the Intelligent Sensing,Diagnostics, and Prognostics Research Lab (ISDPRL) for their support, cama-raderie, and encouragement throughout my Ph.D. journey.Lastly, I would like to express my sincerest gratitude to my significant other,xxmother, father, and sisters for their unwavering love and support. Without theirencouragement and support, I would not have been able to complete this journey.I am forever grateful for their constant love and understanding.xxiChapter 1Introduction1.1 Background and MotivationThe global population has risen dramatically over the last few decades. It hasincreased in population from 1.8 billion in 1915 to 7.7 billion in 2019 and is ex-pected to reach around 10 billion by 2050 [1]. This growth significantly increasesfood demand, and crop production is expected to double by 2050 [2]. Thus, cropproduction should be increased by 2.4% per year [3], up from 1.3% at the moment[4]. The situation is exacerbated further for wheat, which is a critical crop withan annual production increase of approximately 1% [5]. To allay concerns aboutfuture wheat shortages, it is critical to boost crop production efficiency throughthe development of optimized smart farming procedures [6] and high-throughputphenotyping techniques [4]. Additionally, by precisely defining pests and diseasesand upgrading crop maintenance techniques, applying advanced computing tech-nologies to various aspects of agriculture can contribute to a nearly 40% reductionin crop product wastage [7].Simultaneously, improvements in farming management and crop yield must beseriously considered. Agricultural procedure optimization is necessary in farmingmanagement in order to achieve the highest quality and quantity with the fewestinputs. To increase crop yields, breeding new cultivars with increased yields and1improved adaptability to harsh environments is necessary. Accurate evaluation ofvarious crop varieties in a high-throughput manner can significantly expedite theprocess of achieving the aforementioned objectives [8].Traditionally, the procedure for evaluating crop varieties and identifying thosewith high yield potential was to (a) examine multiple plots with varying environ-mental and crop conditions and (b) measure the performance of each plot sep-arately. However, this type of crop evaluation is prohibitively expensive, time-consuming, and labor-intensive. Thousands of plots should be planted and cul-tivated in the conventional evaluation procedure to determine their yield. Yieldanalysis entails several steps, including harvesting, packaging, labeling, diseaseand pest analysis, and weight measurement [4]. What complicates crop evalua-tion further is that the entire experiment must be conducted separately for eachplot to avoid blending different experimental results. Additionally, to assess cropvarieties’ durability under a variety of environmental conditions, plots should belocated in a variety of locations, which complicates the experimental procedure[9].To address the aforementioned challenges, optimize crop management, andefficiently investigate complex and unpredictable crop behavior, researchers haveturned to big data in agriculture [10] for both small- and large-scale problems suchas farm and ecosystem management, respectively [11].In recent years, as computing Graphics processing units (GPU) power hasincreased and the cost of related hardware has decreased, computer vision hasemerged as one of the most critical and popular branches of artificial intelligence.Due to the unprecedented use of computer vision, one of the most important datatypes that is being used more frequently is images. In this regard, advancements inphotography technology have aided significantly in the acquisition and collectionof large image datasets that are preserved and stored using various algorithms[12]. Image processing has received considerable attention, particularly in thefield of agriculture. It is capable of resolving a variety of issues in this field,including crop identification and classification, disease diagnosis, pest detection,2and greenhouse monitoring [13, 14].In the field of agriculture, traditional Machine Learning (ML) models havebeen extensively used to solve computer vision problems and extract useful in-formation from images [15, 16]. In most studies conducted on computer visionusing traditional ML algorithms, modeling has been conducted in two steps. Inthe first step, manual methods were used to extract features. At this stage, variousfeature extraction methods have been used to reduce the dimension of the imagedata, as the image data is extremely large and computers lack the computationalpower necessary to process it. For example, Chu et al. [17] used the Histogramsof Oriented Gradients (HOG) method [4] to extract features for crop pest moni-toring, Suh et al. [18] used the Scale-Invariant Feature Transform (SIFT) methodto classify sugar beet and volunteer potato, Kamath et al. [19] used the speeded uprobust features method to detect weed on farmland, and Altinbas and Serif [20]used Haar-like features to detect the defective crops. After reducing the dimensionof data, in the second step, the extracted features were fed as input into one of theclassifier algorithms, such as Support Vector Machine (SVM) [21] or multi-layerperceptron, to classify each pixel of the image separately [22].While traditional methods are simple to apply to a variety of problems andhave undergone significant transformations in recent years, the majority of themare only accurate in images with a low level of complexity and density [23]. Theinability of traditional ML models to process complex data, particularly in theagricultural sector, makes accurate results extremely difficult to achieve. Themajority of images of farming lands and agricultural products depict intertwinedplants with multiple gene sequences. As a result, numerous studies have demon-strated that Deep Learning (DL) methods are significantly more capable of mod-eling and classifying agricultural data, as well as modeling high-density and com-plex images with substantially greater accuracy [14, 24, 25].According to the literature, DL methods have recently attracted the attentionof a large number of researchers for modeling agricultural issues [26]. DL isa term that refers to techniques similar to artificial neural network, in which a3deeper neural network attempts to present a data representation using various hi-erarchical convolutions [27]. Numerous studies have been conducted to date onthe application of DL in a variety of fields. Apart from agriculture, it has beendemonstrated that DL methods can produce highly acceptable results in a varietyof other scientific areas, including biology [28], intelligent vehicles [29], machinetranslation [30], and water resource management [31]. In comparison to conven-tional ML algorithms, DL methods can automatically identify critical data featuresmore quickly and accurately due to their deep structure.1.2 Research Gaps and objectivesPrevious research has contributed significantly to our understanding of wheatplant phenotyping, but there is still a gap in the literature. This part seeks toaddress this deficiency by evaluating the existing body of knowledge criticallyand supplying new insights to advance our understanding. The research gaps aresummarized as follows:• According to the literature, a number of studies utilizing advanced DL mod-els have been conducted with the aim of detecting and counting wheatspikes, which is a critical parameter in yield estimation. However, therehas been a lack of focus on making the proposed models practical for usein real-world applications, particularly with regards to facilitating the datapreparation process. Thus, one of the objectives of this research is to presenta model that can accurately localize and count wheat spikes through the im-plementation of designed pretrained algorithms and a streamlined annota-tion procedure.• The collection of a large number of wheat images and the annotation ofthousands of objects within these images is both cost-prohibitive and time-consuming. Despite the effectiveness of data augmentation in reducing thenumber of necessary training samples, there has been a lack of focus on4the development of augmentation techniques specifically tailored for wheatplant phenotyping.• Continuously measuring wheat plant height is important as it can providevaluable information on the growth and development of the plant, and canhelp to optimize crop management strategies for improved yields. Further-more, having accurate and continuous measurements of plant height canenable early detection of issues such as disease or pests, which in turn canaid in the implementation of timely interventions and prevent significantyield losses. Within the field of plant height estimation, previous studieshave largely relied on the use of advanced sensors that are not widely ac-cessible to many breeders. Additionally, there has been a lack of focus onthe investigation of wheat height at the individual plant level, which is aprimary goal within the realm of precision agriculture.This research seeks to develop high-throughput phenotyping methods for wheatthat are accurate and simple to implement in the field. In order to make DL-basedwheat plant phenotyping more practical and cost-effective, the models presentedin this study require significantly less data preparation than previous methods inthis area. This objective will be attained by achieving the following sub-goals:• Wheat spikes localization and counting: A new method called DME-Non-Maximal Suppression (NMS) has been proposed. This model is able tocombine the localization and regression results in order to improve thecounting accuracy. In addition, this method makes use of dot annotationto speed up the tedious process of annotating data. It also uses transferlearning to achieve a satisfactory level of accuracy with a smaller numberof training samples.• Automatic Object Level Augmentation (AutoOLA): By introducing an ad-vanced augmentation technique, this model seeks to decrease the number5of training samples required. The proposed method comprises two cutting-edge concepts: augmenting each object of the wheat images separately andautomatically designing the augmentation procedure.• Active learning algorithm: By annotating the most informative parts of thedataset, this model aims to reduce the data preparation phase of model-ing wheat plants. This algorithm, developed specifically for wheat plantphenotyping, places the annotator in a smart loop to prevent the user fromannotating unimportant portions of the dataset.• Wheat plant height estimation: By introducing a simple, cost-effective methodfor high throughput measuring the height of wheat plants. This section pro-poses a novel method for measuring plant height using overhead RGB-Dimaging.1.3 Thesis OutlineThe present section serves to offer a comprehensive outline of the correspondingchapters in order to attain the objective of the study. A thorough explanation of thetechniques and procedures implemented in fulfilling the research objectives canbe found within the appropriate chapters. This thesis comprises of six chapters,which include:• Chapter 1. Introduction provides an overview of the background and mo-tivation behind the study. The primary aim of the chapter is to introducethe reader to the research problem being addressed and to provide a ratio-nale for why the study is important. The chapter also presents the specificobjectives of the research and provides an outline of the thesis as a whole,including the structure and organization of the remaining chapters. Overall,chapter 1 serves as an introduction to the research and sets the stage for therest of the thesis.6• Chapter 2. Literature Review presents a comprehensive literature review onthe current state of plant phenotyping methods and their associated chal-lenges. The review highlights the limitations of conventional methods, suchas manual measurements and destructive sampling, and the advancementsin technology that have led to the development of new, non-destructiveand high-throughput methods such as imaging-based, sensor-based, andmolecular-based methods. The literature review also identified challengesrelated to data acquisition, processing, and analysis, as well as the need forefficient and accurate data analysis methods. Furthermore, research gaps inthe field of plant phenotyping have been identified which sets the stage forthe research presented in the rest of the thesis.• Chapter 3. Wheat Spike Localization and Counting via Hybrid UNet Archi-tectures presents a state-of-the-art framework for localizing and countingwheat spikes using dotted annotation datasets in conjunction with Gaus-sian and constant density map generation algorithms. Additionally, it aimsto improve the performance of wheat counting by using hybrid UNet ar-chitectures and a hybrid density map estimation-non-maximal supervisionalgorithm. The proposed models are evaluated using wheat images from theACID and Global Wheat Head Dataset (GWHD) datasets and show signifi-cant improvements in spike localization and counting compared to previousresearch studies. Additionally, the models’ generalizability is tested by us-ing the ACID-based pretrained models to predict the more complex in-fieldGWHD dataset, with an improvement in performance after retraining themodels using the GWHD dataset’s small training sample size.• Chapter 4. AutoOLA: Automatic Object Level Augmentation for Wheat SpikesCounting proposes a novel augmentation algorithm, AutoOLA, that aims toreduce the required number of training samples in DL models for wheatspike counting. This is achieved by decoupling different objects in thewheat images, including spikes, leaves, stems, and backgrounds, and aug-7menting them independently based on their own augmentation policy. Anevolutionary optimization algorithm is also used to automatically design thetypes and magnitudes of transformations for each decoupled object. Theresults of the proposed technique show significant improvement in wheatspike counting performance by up to 60 percent with only one trainable im-age, and a high level of accuracy in predicting the number of wheat spikesin the field.• Chapter 5. Wheat Spikes Height Estimation Using Stereo Cameras intro-duces a cost-effective and easy-to-use method for measuring the height ofwheat plants using stereovision cameras. The study aims to collect datausing two well-known stereovision cameras, OAKD and D455, and thenuse a deep learning model called Mask-RCNN to identify and distinguishthe spikes in the images. Additionally, the study develops a hybrid SIFT-RF-based active learning algorithm to make the process of annotating thecollected datasets easier, and to transfer annotations from one camera to theother. The results of the study indicate that the OAKD camera is more suit-able for wheat plant height estimation due to its higher quality RGB imagesand better matching of mono camera images. The proposed algorithm isable to estimate wheat height with a high degree of accuracy.• Chapter 6. Conclusions, Contributions and Recommendations for FutureResearch presents the contributions made by the study and summarizes thekey findings of the research. Furthermore, the chapter addresses the limita-tions of the study to provide context and insight into the scope and reliabilityof the research. Finally, the chapter concludes by identifying potential ar-eas for future research. Overall, the chapter serves to provide a conciseoverview of the research and its implications for the field.• Appendix A. Ontology-based knowledge management for deep learning ap-plications in wheat phenotyping introduces an ontology-based approach,which was used in the published review paper, to provide a clear and or-8ganized framework for understanding and categorizing the diverse applica-tions of DL in wheat plant phenotyping.9Chapter 2Literature ReviewThe literature review chapter aims to provide an overview of the current stateof research on wheat spikes localization and counting, different augmentationtechniques in plant phenotyping, active learning in agriculture, and wheat spikesheight estimation.Wheat spikes localization and counting is an important task in plant pheno-typing as it can provide valuable information about the growth and developmentof the crop. The ability to accurately and efficiently locate and count wheat spikescan be used to make informed decisions about crop management practices, suchas fertilization, irrigation, and harvesting.Wheat spikes height estimation is an important task in plant phenotyping, as itcan provide valuable information about the growth and development of the crop,which can be used to make informed decisions about crop management practices,such as fertilization, irrigation, and harvesting. In this thesis, we will review thecurrent state of research on these topics and explore how they can be used toimprove the yield of wheat crops.Different augmentation techniques in plant phenotyping, such as rotation, scal-ing, and flipping, can help to increase the diversity and size of the training dataset,which can improve the generalization ability and robustness of the model, andprevent overfitting.10Active learning in agriculture is a relatively new area of research, which aimsto develop algorithms that can learn from the data in an active and interactive way.This can help to improve the accuracy and efficiency of the model, by allowing itto focus on the most informative samples.2.1 Wheat Growth Monitoring2.1.1 Wheat Spike Localization and CountingThe number of spikes / m2 is critical in estimating the wheat yield [32, 33].As with wheat yield and biomass estimation, conventional methods of countingspikes are time-consuming and prone to error. Automatic and precise wheat spikecounting is required for large-scale farm management. Despite the rapid advance-ment of Deep Learning (DL) algorithms and computer vision techniques and theirpromising performance in counting various objects such as animals [34] and vehi-cles [35], counting wheat spikes remains challenging due to their non-rigid nature.In the field of wheat spikes counting, all of the research studies employedground-based RGB images as DL the input source. Pound et al. [36] was one ofthe first to use DL models to count wheat spikes and spikelets. The dataset pre-sented in this article was collected and annotated with the goal of counting wheatspikes and spikelets. This data, called ACID, was subsequently used in additionalstudies [37, 38], which will be discussed in this subsection. Additionally, in or-der to perform wheat identification, the ACID dataset was classified as having anawned phenotype. According to Figure 2.1, each image was created using severalwheat plants and includes detailed annotations regarding the spikes and spikelets’positions. Instead of directly predicting the spikes and spikelets’ locations, thisstudy used heatmap regression [39] to identify areas of high likelihood. Thisarticle’s Convolutional Neural Network (CNN) architecture was inspired by thestacked hourglass network [40], a hybrid of Fully Convolutional Networks (FCN)and residual networks [41]. Additionally, an investigation was conducted in thisstudy to determine the optimal size of the input images. It was found that 512 by11Figure 2.1: The original image (left column), spikes annotated (middle col-umn), and spikelets annotated (right column) of ACID dataset.The images were reproduced using the figures of Pound et al. [36] withpermission of the authors.512 pixels input images perform better for both spikes and spikelets counting with95.91% and 99.66% accuracy, respectively.Alkhudaydi et al. [38] developed a new algorithm called SpikeletFCN to countthe number of spikelets in a wheat image using the ACID [36] and CropQuant [42]datasets. SpikeletFCN is an FCN method for estimating the density of dots in an-notated images. The importance of two distinct aspects of DL in identifying wheatspikelets was investigated in this article: segmenting the spike regions prior to thespikelet counting procedure and using pre-trained DL algorithms. To perform thestudy, four distinct scenarios were defined:• Train SpikeletFCN with the CropQuant dataset without any additional pre-processing.• Remove the background of CropQuant images using ground truth masksand then train the SpikeletFCN with the segmented CropQuant dataset.• Pre-train SpikeletFCN using the ACID dataset and then fine-tune that usingthe CropQuant dataset.12• Pre-train SpikeletFCN using the ACID dataset and then fine-tune that usingsegmented images of the CropQuant dataset.The findings of this study are fascinating because they demonstrated the im-portance of segmentation and pre-training in this field. The study concluded thatby segmenting the task prior to counting, the Mean Absolute Error (MAE) wasreduced by 83.5%. Additionally, it was demonstrated that by utilizing pre-trainedmodels, the MAE was reduced by 84.5%.Alkhudaydi et al. [38] concluded that performing segmentation before count-ing can significantly improve modeling performance. However, one of Alkhu-daydi et al. [38]’s limitations was their manual segmentation procedure, whichinvolved removing the background using ground truth masks. Ma et al. [43] devel-oped a two-stage image segmentation technique to increase the accuracy of spikesegmentation, which was necessary for accurately counting them. A CNN modelis used in the initial stage of this method to perform coarse segmentation. Fol-lowing that, an FCN was applied to improve segmentation accuracy and removenon-ear pixels from the first stage results. In this paper, to apply the two-stagesegmentation, the input data for the CNN model was prepared using the entropyrate superpixel algorithm [44]. Then, the first stage segmentation was performedusing a CNN structure introduced in Ghosal et al. [45]. In the second stage ofsegmentation, using VGG16-FCN, a pixel-wise classification was performed toremove the noisy pixels from the first segmentation stage. The authors of thisstudy compared the two-stage segmentation results with Random Forest (RF) andtwo other algorithms that were presented in Ma et al. [46] and Fernandez-Gallegoet al. [47]. The results showed that with 83.7% accuracy, the proposed methodoutperformed the compared algorithms.In Ma et al. [43], it was concluded that two-stage segmentation could sig-nificantly improve wheat spike identification accuracy. However, the effect oftwo-stage segmentation on counting accuracy was not examined in that paper. Tocover that, the authors of Misra et al. [48] used the same idea of applying a two-stage segmentation on the wheat plant and then calculated the counting accuracy13based on the segmented images. The authors proposed a method called SpikeSeg-Net that was formed from an encoder-decoder-based two-stage segmentation andflood-fill technique [49] that was used in the counting part. This study showedthat by having accurate segmentation (99.91% accuracy), it is possible to achieve95% accuracy in spike counting using simple counting algorithms.2.1.2 Wheat Height EstimationPlant height, which strongly correlates with yield [50], lodging [51], disease resis-tance [52], biomass [53], and harvests index [54], is one of the crucial parametersfor wheat selection in a breeding program. Continuous plant height measurementsthroughout the growing season may aid in identifying various growth phases andselect genotypes with an extended grain filling period that result in a higher yield[55].Traditionally, wheat height was measured manually with a yardstick. Manualcrop height estimation, however, is subject to human bias when identifying plantboundaries [55], which may be exacerbated by adverse weather conditions [56].Additionally, when applied to a large number of wheat plants, this assessment pro-cess is time-consuming, labor-intensive, and expensive [57]. Conventionally, theheight of wheat plants was measured by sampling some from a specific number ofplots [58]. However, in precision agriculture, all wheat spikes must be measuredindividually. Consequently, developing high-throughput phenotyping strategies isrequired for the rapid, accurate, reliable, and continuous measurement of wheatheight.In recent years, various high-throughput techniques have been proposed forautomatically and accurately estimating plants’ height. The height of a plant canbe determined using imaging devices and sensors. Light Detection And Rang-ing (LiDAR) and ultrasonic sensors are the most popular sensors for plant heightestimation, and stereovision cameras are an example of imaging devices.Mounted on a mobile field vehicle, the LiDAR sensors are able to provide ahigh-resolution 3D structure of the canopy [56]. Nevertheless, LiDAR sensors’14performance depends on sensor resolution, geolocations during movement, andsampling frequency [55]. Moreover, they are significantly more expensive thanother plant height estimation devices.As range detectors, ultrasonic sensors are capable of measuring plant heightaccurately [59]. Nevertheless, the adjustment of these sensors’ field of view iscomplicated. A narrow ultrasonic field of view does not cover the plant canopy,whereas a broad ultrasonic field of view reports the height of non-plant objectsoutside the canopy region [55].3D imaging is another technique utilized for plant height estimation [60]. Var-ious imaging techniques, such as stereo triangulation [61], structured light [62],interferometry [63], and time-of-flight [64], can be used to create 3D images. Thetime-of-flight and stereo-triangulation techniques are the most prevalent in esti-mating height and other phenotypic traits [55]. Using these imaging techniques,acquiring RGB-D images containing color (RGB) and depth (D) information ispossible.Using RGB-D images, Jiang et al. [65] analyzed the canopy size of cottonplants. The authors used depth information to segment the canopy point cloudsfrom the ground and weeds and concluded that the extracted traits are able to con-tribute to yield prediction. Vit and Shani [66] compared the Microsoft Kinect V2,Intel D435, Intel SR300, and Orbbec Astra S RGB-D sensors for on-ground close-range outdoor phenotyping and concluded that the Intel D435 performs more ac-curately than the other devices examined. Rueda-Ayala et al. [67] compared Un-manned Aerial Vehicle (UAV)-based technologies and land-based RGB-D imagesfor estimating plant height and biomass. The authors concluded that the RGB-Destimation is more reliable and accurate than UAV-based systems for estimatingplant height and dry biomass. Using RGB-D images, Xu et al. [68] extracted 3Dstructural information from the wheat canopy in order to estimate nitrogen ac-cumulation. The authors concluded that the presented model accurately measuresnitrogen accumulation in the leaf layer and shoot. Using RGB-D images, Liu et al.[69] estimated the plant height and aboveground biomass of Toona Sinensis. The15authors used the UNet DL model to segment plants from complex backgroundsand a linear regression model to estimate plant height from RGB-D images. Theauthors concluded that the presented models produce reliable and promising plantheight prediction results.In recent years, as the computing power of Graphics processing units (GPU)has increased and the cost of related hardware has decreased, computer vision hasbecome one of artificial intelligence’s most important and well-known subfields.Due to the unprecedented use of computer vision, images are one of the mostcritical data types that are being used more frequently. In this regard, techno-logical advancements in photography have greatly facilitated the acquisition andcollection of large image datasets that are preserved and stored using a variety ofalgorithms [12].The literature defines wheat plant height as the distance between the soil sur-face and the top of the spike, excluding awns [58, 70–74]. In order to measurethe height of wheat using high-throughput phenotyping, locating the spikes in afield is one of the first steps. Numerous studies have been conducted on the lo-calization of wheat spikes using advanced DL models, which are discussed inSubsection 2.1.1. Wheat Spike Localization and Counting.One of the most critical priorities of precision agriculture is the capacity todo phenotypic analysis at a high resolution, down to the level of the plant [75].Many agricultural firms, including Cares, John Deere, Iron Ox, and FarmWise,are now developing plant-level insights to guarantee that each plant gets the ap-propriate amounts of sunlight, water, and nutrients. Compared to researchingbroad regions, the plant-level analysis provides several advantages. Breeders, forinstance, may examine just the most severely stressed plants, varieties of plantscan be researched, irrigation management can be altered prior to yield effect, etc.162.2 Data Analytic Methods2.2.1 Data Augmentation TechniquesAccording to Subsection 2.1.1. Wheat Spike Localization and Counting, the pri-mary objective of the research studies that counted wheat spikes was to developmore advanced DL networks to detect the wheat spikes accurately. However, thedata preparation portion of real-world applications remains prohibitively difficult.In order to feed the DL models, it is necessary to capture numerous in-field imagesand annotate thousands of wheat spikes during the data preparation phase.One of the most effective methods for reducing the number of required train-ing samples is to generate synthetic samples using data augmentation [76]. Eventhough appropriate data augmentation techniques can significantly reduce datapreparation efforts, this topic has received less attention. For instance, the dataaugmentation technique, which was introduced in 2012 for classifying the Ima-geNet dataset [77], is still a standard procedure [78]. As another example, almostall of the studies on modeling the MNIST dataset used simple augmentation tech-niques such as scaling, rotation, and elastic distortion [79–82].The techniques for augmentation can be divided into two distinct groups: syn-thesis of new samples and augmenting the existing samples. In the first group, at-tempts are made to synthesize new data by employing DL models. In this method,the augmented data are directly reconstructed using a family of DL algorithmsknown as Generative Adversarial Networks (GAN) [83]. While the use of GANto generate augmented data is promising, its implementation is complex, and theresulting data may deviate from real-world samples [84–89]. The second groupof augmentation techniques is significantly simpler and has a significant impacton training the robust DL models. In this method, augmented images are gener-ated using simple image transformations such as scaling, random cropping, androtation [90]. Recent research has been conducted to augment existing samplesusing more advanced augmentation techniques, such as image erasing, image mix,feature augmentation, and auto augmentation [91].17The idea behind image erasing-based augmentation techniques is to replacethe values of selected image subregions with constant or random values. Thesetechniques can improve the robustness of CNN models and force the model to findrelevant information in the input images while removing important subregions[84, 92–96].The image mix-based augmentation techniques produce an augmented imageby combining two or more images. As neural network models are prone to mem-orizing corrupted labels [97], image mix techniques combine the input features toprevent the model from becoming overconfident about the relationship betweenthe input features and the labels [98–103].Instead of augmenting the input space, feature augmentation techniques aimto transform a learned feature space. Research in this field has demonstrated thatmanipulating the vector representation of input data can result in more realisticaugmented samples [104, 105].While the proposed augmentation techniques are able to improve the diversityof the training dataset, selecting the appropriate transformations from numerousaugmentation approaches is still challenging. In addition, the suitable transfor-mations can vary among different datasets. So, the auto augmentation techniquestry to find the most appropriate augmentation policy and its magnitude for eachspecific dataset automatically [78, 106, 107].The referred studies demonstrated a positive impact of data augmentation onthe robustness of the trained neural network models. However, the majority ofthem perform the augmentation at the image level. This indicates that the bound-aries between different objects and between objects and the background remainunchanged during the augmentation. In addition, image-level augmentation ap-plies the selected augmentation pipeline to all of the image’s objects simultane-ously; it is impossible to perform different augmentation pipelines on each objectin the image.182.2.2 Active Learning in High-Throughput PhenotypingAccording to Subsection 2.1.1. Wheat Spike Localization and Counting, DL-basedmodels are able to locate and count the wheat spikes accurately. However, imple-menting the DL models in real-world problems still poses the fundamental prob-lem of data preparation. In order to develop a robust DL model, it is needed tohave access to a large labeled dataset, which is a time-consuming and expensivetask.One of the effective methods in order to reduce the annotation time and min-imize the supervision amount in the labeling procedure is active learning [108–110]. Active learning [111, 112] is an iterative training method that selects themost efficient samples to be labeled and then used in the training procedure. Whilehybrid DL-based active learning algorithms have been vastly investigated in otherfields of science [113–115], arguably limited studies have applied these techniquesin the field of agriculture phenotyping [116].One of the first studies that employed active learning for high throughputphenotyping is Kumar et al. [117]. This study aimed to learn the distributionof sorghum phenotypes in the field. To perform that, an active learning modelwith the Gaussian process was employed to develop an autonomous system. Theauthors concluded that the developed model outperforms the conventional ap-proaches.Chandra et al. [118] applied an active learning method to alleviate the requiredlabeled data for cereal crop detection. In the proposed active learning approach,the DL model constantly interacts with a human annotator by asking the labelsfor the most informative images instead of all training images. The authors con-cluded that using the proposed active learning algorithm; the annotation costswere reduced by over 50% on wheat spikes and sorghum head detection tasks.To distinguish between crop and weed plants from image data, Sheikh et al.[119] developed an active learning-based DL method to perform the semantic seg-mentation. The authors concluded that their approach could reach higher accuracywith a smaller number of samples compared with random and entropy-based sam-19pling.Nagasubramanian et al. [120] explored the use of active learning in plant phe-notyping on two classification datasets exhaustively and showed that active learn-ing strategies outperform random sampling and cut labeling costs to a reasonabledegree.Rawat et al. [121] performed a broad investigation on active learning applica-tions in plant organ segmentation. This study applied four uncertainty-based ac-tive learning methods to three different natural plant organ segmentation datasets.The authors concluded that the used active learning strategies struggled to out-perform the random sampling technique. The authors of this study believed thatthe reason for the poor performance of their models is the fact that active learningstrategies should be designed specifically for plant phenotyping tasks to operateat the regional level instead of the image level.20Chapter 3Wheat Spike Localization and Count-ing via Hybrid UNet ArchitecturesA version of this chapter has been published in Computers and Electronics inAgriculture as a research paper titled “Wheat spike localization and counting viahybrid UNet architectures”.3.1 OverviewWheat yield is positively correlated with the number of wheat spikes in the field,which is an essential index for plant breeders. To efficiently calculate this in-dex, there is a high demand for precise and automatic plant phenotyping meth-ods that quantify images. As a tool for estimating crop yields, machine visionis rapidly advancing. In the majority of crop phenotyping fields, however, anno-tating thousands of small objects using bounding boxes or polygons is extremelytime- and labor-intensive. This chapter aims to develop a state-of-the-art frame-work for localizing and counting wheat spikes using dotted annotation datasets inconjunction with Gaussian and constant density map generation algorithms. In ad-dition, we developed hybrid UNet architectures as the computational component,including VGG16-UNet, ResNet34-UNet, ResNet50-UNet, and ResNeXt-UNet.Furthermore, we improved the performance of wheat counting by employing a21hybrid density map estimation-non-maximal supervision algorithm. Wheat im-ages from the ACID and Global Wheat Head Dataset (GWHD) datasets are usedto evaluate the proposed models. With F1 score and Mean Absolute Percent-age Error (MAPE) of 0.96 and 1.68%, respectively, the results from the ACIDdataset demonstrate a significant improvement in spike localization and countingcompared to previous research studies. Additionally, to assess the proposed mod-els’ generalizability in real-world modeling scenarios, the ACID-based pretrainedmodels are used to predict the more complex in-field GWHD dataset. With an F1score of 0.57 and a MAPE of 2.302%, the pretrained models were able to localizeand count the wheat spikes in the new dataset. Following that, the ACID-basedpretrained models are retrained for a few epochs using the GWHD dataset’s smalltraining sample size. The results demonstrate a significant improvement in theproposed model’s performance, with an F1 score of 0.91 and a MAPE of 1.56%.3.2 Wheat Spikes Localization and CountingMethodologyThe model for wheat localization and counting developed in this chapter is com-posed of several components. The first subsection provides an overview of theproposed methodology. In the second subsection, we discussed the developedDeep Learning (DL) algorithms. The third subsection introduces the data aug-mentation technique, which is used to increase the input images’ variety and avoidoverfitting. The fourth subsection introduces two density maps that are used inwheat spikes density estimation. After applying the DL algorithms in combina-tion with data augmentation and density estimation, the fifth subsection introducesthe methodologies used to localize and count wheat spikes using the obtained re-sults. The sixth subsection discusses the methods used to prepare the datasets.Finally, the seventh subsection discusses the evaluation metrics used to assess theproposed methods’ localization and counting capabilities.223.2.1 Overview of the proposed approachAn overview of the proposed wheat spike localization and counting frameworkis presented in Figure 3.1. According to this figure, after preparing the dataset,which was discussed in Subsection 3.2.6. Data preparation, four different UNet-based pretrained DL architectures, namely VGG16-UNet, ResNet34-UNet, ResNet50-UNet, and ResNeXt-UNet, are developed. The developed DL architectures’ per-formance is compared with the original UNet, and two recently developed ver-sions of it called UNet++ and UNet3+. Detailed information about the proposedDL architectures is available in Subsection 3.2.2. DL architectures. Besides,to improve the performance of the developed models, data augmentation is de-signed to randomly perform on the input dataset, which is explained in Subsec-tion 3.2.3. Data augmentation. Another critical part of the proposed framework isdensity map generation, which significantly impacts the prediction performance.In this model, we used Gaussian Density Map (GDM) and Constant Density Map(CDM), which are introduced in Subsection 3.2.4. Density map generation. Fi-nally, the wheat spike localization and counting approaches are presented in Sub-section 3.2.5. Wheat spikes localization and counting. According to Figure 3.1,two different methods called DME and Non-Maximal Suppression (NMS) areused in this part. Besides, to estimate the number of spikes in the image, an Lin-ear Regression (LR) algorithm combines DME and NMS results to improve theprediction performance.3.2.2 DL architecturesIn the present chapter, four different UNet-based architectures are developed andapplied for wheat spike localization and counting. In addition, to compare theperformance of the developed architectures, the performance of our models iscompared with that of the original UNet model and two powerful UNet-basedarchitectures entitled UNet++ [122] and UNet3+ [123].Introduced by Ronneberger et al. [124], UNet was firstly used for biomedicalimage segmentation. This model shared the important concept of skip connections23LRDensity map generationDataset preparationStartEndDL model development UNet UNet++UNet3+VGG16 ResNet34 ResNet50 ResNeXtVGG16-UNet ResNet34-UNet ResNet50-UNet ResNeXt-UNetData augmentationSpike countingCDMGDMSpike localizationNMSDME Generating the DM resultsDL: Deep LearningLR: Linear RegressionDM: Density MapGDM: Gaussian Density MapCDM: Constant Density MapDME: Density Map EstimationNMS: Non-Maximal SuppressionFigure 3.1: Overview of the proposed wheat spike localization and countingprocedure.in DL models, which was also introduced in Fully Convolutional Networks (FCN)[125] (Table 3.1). UNet architecture is comprised of an encoder and a decoder.The encoder part (i.e., the first half of the network) can be regarded as the featureextraction part and the decoder part (i.e., the second half of the network) as theupsampling part. The decoder part seeks to project the discriminative featuresonto the pixel space. Discriminative features are the outputs of the encoder part,24which is learned in a lower resolution, and the pixel space is the result of thedecoder part, which is learned in a higher resolution.In the decoding parts of the original UNet, transposed convolution is used forupsampling. However, we found that it is not necessary for our case study, andit merely increases the number of DL parameters without upgrading the modelperformance. Thus, bilinear upsampling was utilized. In addition, valid convolu-tions that avoid using padding on the input image were used in the original UNet,causing size variation of the output matrix in comparison with the input matrix.Accordingly, when using the original UNet, it is required to crop output matricesof the encoding part in order to concatenate them to the decoding part. However,in the proposed UNet, we used the same convolution (a type of convolution inwhich the output image has the same dimension as the input image) to preventdimension reduction subsequent to each convolution.The input image dimension in all the proposed architectures is 256×256 pix-els. After four levels of downsampling in the proposed UNet using max poolingwith kernel size and stride of two, the dimension downsized to 16×16 pixels.Then, by having four levels of bilinear upsampling and concatenation, the dimen-sion of output increases to 256×256 pixels once again. Furthermore, since RGBimages were used, the number of input features is three, which increases to 1024in encoding and reduces to one (the output density map) in decoding.The proposed pretrained network based-UNet algorithms pursue two objec-tives; to reduce the number of trainable parameters by freezing the feature ex-traction part of the network and solely training the upsampling part, and to makebetter the model performance by using deeper network and also some additionalconcepts such as residual learning in ResNet34-UNet and ResNet50-UNet, andcardinality in ResNeXt-UNet models. All the pretrained architectures used in theproposed models have been trained on ImageNet dataset [126]. ImageNet is alarge-scale ontology of images containing over 14 million images with almost 21thousand classes. Although ImageNet lacks the class of images applied in thepresent study, our aim is to realize whether ImageNet-based pretrained networks25L1 L2 L3 L4 128×128×256 64×64×512 64×64×256 64×64×256 64×64×256 64×64×256 64×64×25632×32×1024 16×16×2048 256×256×256 256×256×256 256×256×256 256×256×256 256×256×256128×128×51264×64×102432×32×2048 64×64×102464×64×1024 64×64×1024 64×64×1024 128×128×512128×128×512 128×128×512 128×128×512256×256×64256×256×64256×256×64256×256×64256×256×64 128×128×128 128×128×128 128×128×128 128×128×128 128×128×12832×32×512Conv 3×3ReLUMax poolConcatenationBilinear upsamplingPretrained blockResNet/ResNeXt layerTransposed convBatch normalization32×32×51232×32×51232×32×512 32×32×51216×16×1024(a)(b)Figure 3.2: (a) VGG16-UNet and (b) ResNet50-UNet/ResNeXt-UNet archi-tectures.can enhance wheat localization and counting accuracy or not. Thus, all the devel-oped pretrained-based UNet architectures are trained once with fixed weights inthe encoding part and once without using the pretrained weights from scratch.The proposed VGG16-UNet architecture is presented in Figure 3.2 (a) andTable 3.1. This network can be considered an extended version of the proposedUNet, with more convolutional layers. According to the figure, the dimensionand size of the feature maps are exactly the same as the UNet architecture. Wedid some minor modifications in the original VGG16 network [127] to make itappropriate to be concatenated in the upsampling part of the VGG16-UNet archi-tecture. All VGG16 layers are presented in transparent gray boxes in the figure.26In the original VGG16 architecture, there is a max pooling layer following everytwo convolutional layers. However, the first max pooling layer is eliminated fromthe network as shown in Figure 3.2 (a). In addition, we used the output of the first,second, third, and fifth layers for concatenation purposes since they match the ex-pected number of feature maps, which are 64, 128, 256, and 512, respectively. Theoutput of the VGG16 network (after eliminating the fully connected layers of thenetwork) has 32×32 pixels dimension and 512 feature maps. Though, since wedid not intend to modify the upsampling part of the UNet model in VGG16-UNet,a max pooling layer and two convolutional layers were added to the end of themodel to get 16×16 pixels dimension and 1024 feature maps output. In Table 3.1,the pretrained layers are shown in gray cells, and the rest of the layers are shownin white cells. Since it was intended to freeze the pretrained part of the VGG16-UNet model, it is necessary to have more trainable parameters in the upsamplingpart. Thus, transposed convolution layers were used for upsampling (dark greenlayers in the figure) rather than bilinear upsampling used in the proposed UNetarchitecture.Other pretrained based-UNet architectures utilized ResNet34, ResNet50, pre-sented in He et al. [41], and ResNeXt, presented in Xie et al. [128]. The detailedinformation on the architecture of ResNet34, ResNet50, and ResNeXt-UNet mod-els are presented in Table 3.1. In addition, ResNet50 and ResNeXt-UNet archi-tectures are plotted in Figure 3.2 (b). It should be noted that we did not providethe ResNet34-UNet plot in Figure 3.2 to avoid prolonging the paper. These net-works start with a 7×7 convolution layer with stride 2. Since this convolutionlayer shrinks the input dimension from 256×256 pixels to 128×128 pixels, weeliminated the next max-pooling layer, which is available in the original ResNetand ResNeXt architectures. Prior to presenting the results of the concatenations,a bilinear upsampling was added to all of the residual networks (Figure 3.2 (b)).In this part, transposed convolutional layers were not used since the intention wasnot to add any trainable layer between the pretrained networks.After the fourth encoding layer of ResNet34-UNet, the output has 512 feature27Table 3.1: Proposed DL architectures.UNet VGG16-UNet ResNet34-UNet ResNet50-UNet ResNeXt-UNetconv 7, stride 2[conv3 - 64conv3 - 64] [conv3 - 64conv3 - 64] [conv3 - 64conv3 - 64]×3conv1 - 64conv3 - 64conv1 - 256×3conv1 - 128conv3 - 128, C = 32conv1 - 256×3bilinear upconcatenation 1max poolconv3 - 128conv3 - 128conv3 - 128conv3 - 128max pool [conv3 - 128conv3 - 128]×4conv1 - 128conv3 - 128conv1 - 512×4conv1 - 256conv3 - 256, C = 32conv1 - 512×4bilinear upconcatenation 2max poolconv3 - 256conv3 - 256conv3 - 256conv3 - 256conv3 - 256max pool[conv3 - 256conv3 - 256]×6conv1 - 256conv3 - 256conv1 - 1024×6conv1 - 512conv3 - 512, C = 32conv1 - 1024×6bilinear upconcatenation 3max poolconv3 - 512conv3 - 512conv3 - 512conv3 - 512conv3 - 512max poolconv3 - 512conv3 - 512conv3 - 512[conv3 - 512conv3 - 512]×6conv1 - 512conv3 - 512conv1 - 2048×3conv1 - 1024conv3 - 1024, C = 32conv1 - 2048×3bilinear upconcatenation 4max poolconv3 - 1024conv3 - 1024max poolconv3 - 1024conv3 - 1024max poolconv3 - 1024conv3 - 1024conv3 - nbilinear upconcatconv3 - nconv3 - n×4trans convconcatconv3 - nconv3 - n×4trans convconcatconv3 - nconv3 - n×4trans convconcatconv3 - nconv3 - n×3trans convconcatconv3 - nconv3 - n×3[conv3 - 1][output - 1 - 256×256 pixels]28maps. Then, same as VGG16-UNet, another trainable encoding layer was addedto increase the number of feature maps to 1024 before initiating the decoding partof the network. However, ResNet50 and ResNeXt-UNet models undergo a differ-ent situation. Figure 3.2 (b) and Table 3.1 show that after four times of encodinglayer of these networks, the output has 2048 feature maps. Accordingly, since wedid not aim to deepen the models further, the fifth trainable encoding layer wasnot added to these networks. Obviously, without having the fifth encoding layer,ResNet50 and ResNeXt-UNet models merely possess three levels of decodinglayers.All the networks used in the present chapter terminate by a 3×3 convolu-tion layer to reach a 256×256 pixels output with one feature map, representingthe input image density map. In addition, all the models were trained for 2000epochs using the University of British Columbia (UBC) ARC SOCKEYE as ahigh-performance computing platform. The initial learning rate of the proposednetworks was considered 2.5e-4, reducing by a factor of ten every 400 epochs.Additionally, the Mean Squared Error (MSE) loss function was used in the mod-els. The proposed model codes and the dataset used in the present chapter arepublicly available in our GitHub repository. Notably, while the dataset prepara-tion part of the models was written in MATLAB, all the proposed DL models weredeveloped in Python via the Pytorch package.3.2.3 Data augmentationThe major challenge ahead of counting and localizing the wheat spikes is that thisdataset is not as extensive as the majority of the datasets applied for training theDL models, e.g., the ones proposed in the present research. Further, as articulatedin Pound et al. [36], these datasets incorporate a variety of extremely identicalsamples to the extent that ResNeXt-UNet, as an example of a deep model withover 104.1e6 parameters simply can overfit the training samples. To overcomethe overfitting problem using this dataset, Pound et al. [36] and Hu et al. [37]recommended applying online random data augmentation during training rather29than generating a fixed augmented dataset prior to the training procedure.In the present model, subsequent to random cropping that was debated earlier,various augmentation techniques were randomly applied to the images. Same asthe cropping procedure, since all of the augmentations are done randomly, themodels fail to see exactly similar input images during training. It means that themodel requires more epochs for thorough training. Different data augmentationtechniques used in the modeling are plotted in Figure 3.3. According to this figure,after cropping, the augmentation process is comprised of hue alteration, saturationchange, value change, shifting, scaling, rotation, horizontal flipping, vertical flip-ping, and 90° rotation of the images. All these augmentations are randomly doneon the images with a probability of 0.5. This means that the chance of applyingeach augmentation technique to the original image is 50%. Thus, an image maybe modified using some or all of the specified augmentation techniques.3.2.4 Density map generationWheat localization and counting can be performed via two different approaches.In the first approach, it is required to detect all the ears of wheat through object de-tection [129, 130] or to segment via instance segmentation [43, 48, 131, 132], andto count the detected spikes. Although this approach may present more informa-tion, for example, the form or cultivar class of wheat spikes, it needs prior objectdetection or instance segmentation. In the second approach, built upon densityestimation [37, 38, 133], point-like annotations are applied to the dataset of objectpositions, and a corresponding density map is generated. The approach operatesfaster, especially in terms of annotation [134], and it does not need prior objectdetection or instance segmentation.When the primary objectives are spike localization and counting, the first ap-proach places an additional burden on the model and requires significant time forobject annotation. As a result, the current research employs the second approach.The second approach is based on estimating a density map in order to localize andcount the wheat spikes indirectly. This objective necessitates the generation of a30Hue/Saturation/ValueCrop Shift/Scale/RotateHorizontal flip Vertical flip Rotate 90◦Figure 3.3: Data augmentation procedure.corresponding ground truth density map for each wheat image that will be used asinput to the proposed models. The following steps should be followed in order togenerate the density map:1. Create a zeros matrix with the same dimensions as the input image but onlyone channel. Given that the input image in this study is 256×256×3, thegenerated matrix of zeros is 256×256×1.2. Annotate each wheat spike in the image with a dot.3. Change the value of elements corresponding to dot positions in the createdmatrix of zeros by a fixed amount. In this model, the amount of those el-ements was changed to 100. In this work, this matrix is referred to as rawannotated ground truth.314. To create the ground truth density map, apply a filter to the raw annotatedground truth. For example, convolution with a Gaussian kernel is requiredto generate the GDM.Gaussian filter is highly popular in density map generation for counting pur-poses and has been used in many of the studies cited in the present research. Onedimensional Gaussian function is shown in the following equation.G(x) =1√2πσ2e−x22σ2 (3.1)where σ is the standard deviation of the distribution, which is considered onein our generated density maps. An example of GDM is shown in Figure 3.4. Inthe figure, applying the Gaussian kernel on the raw annotated ground truth turnsevery 1×1 pixel dot annotation into a 9×9 ground truth density map.As mentioned, in the raw annotated ground truth, as a 256×256×1 pixelsimage, all dot annotations are considered 100, and the remaining pixels are zero.Our experiments demonstrated that this magnification of dot annotations helps themodel detect them more straightforwardly. After performing the Gaussian densitymap, the sum of each 9×9 ground truth density map remains 100 with a maximumamount of 15.91 at the center and a minimum amount of 1.79e-6 on the corners.According to Figure 3.4, a novel yet more straightforward procedure for den-sity map generation called CDM was introduced as well. Accordingly, same asthe Gaussian kernel, a 9×9 density map was developed for each dot annotation.However, rather than having various amounts in each pixel, a quantity constant inall 9×9 pixels was considered. Then, since the sum of each density map has to be100, the amount of each pixel can be calculated as follows:100/(9×9) = 1.234 (3.2)CDM algorithm benefits from some advantages, especially for wheat spikelocalization, which is discussed in Section 3.3. Experimental Results and Discus-sion.32GDMCDM1901.230Figure 3.4: Difference between CDM and GDM.3.2.5 Wheat spikes localization and countingAfter generating the ground truth density maps, the proposed DL algorithms’ taskis to regress the density map from the corresponding input wheat images. Thisis done by minimizing the MSE loss function between the output and the groundtruth density maps.Predicting the density maps via DL algorithms, wheat spike positions can beidentified using NMS algorithm [135]. In this approach, all the output density mappixels are examined, and the ones with a greater amount than their four neighborsare considered a wheat spike position. The present model uses the NMS algorithmfor wheat spike localization.Besides the NMS algorithm that can explore the predicted location and wheatspike number, another technique can be employed for wheat spike counting. Asmentioned in Subsection 3.2.4. Density map generation, pixel values in raw dotannotation image are zero except for wheat spike positions in the image, whichare 1×1 pixels with a value of 100. It was also revealed that after generating the33density map according to raw dot annotated images, the sum of each 9×9 pixelsdensity map remains 100. Therefore, regarding an input image containing eightwheat spikes, the sum of the pixels of the ground truth density map is 8×100= 800. In this example, if the sum of the pixels of the output density map aris-ing from DL algorithm is 750, it can be concluded that our DL algorithm has(|750−800|/800)× 100 = 6.25% error. This wheat spike counting procedure iscalled DME in the present model.Evidently, NMS and DME take two different outlooks at wheat spike counting.The results presented in Section 3.3. Experimental Results and Discussion showthat some of the investigated DL algorithms are highly capable in terms of wheatspike localization, which results in high accuracy in wheat spike counting via theNMS algorithm. On the other hand, some algorithms that are sufficiently accuratevia the DME algorithm have a multitude of false positive pixels in the outputdensity map, leading to poor counting results using the NMS algorithm.As shown in Figure 3.5, we proposed both NMS and DME approaches to im-prove the reliability and accuracy of DL models in wheat spike counting. Accord-ing to the figure, when the goal is counting (rather than localization), the datasetis randomly divided into three segments: training dataset, counting dataset andtesting dataset. DL model is trained via training dataset. After training is oper-ated, the counting dataset is applied to the trained DL model to generate NMSand DME results. As mentioned earlier, both NMS and DME algorithms presentsome predictions about the number of wheat spikes in the image through differentcalculation procedures. According to this figure, using the NMS and DME resultsobtained from the counting dataset, we train a LR model. The input variables ofthe LR model are the NMS and DME, which are the outputs of the DL model, andthe objective of the LR model is to estimate ground truth counting results.After training the LR model, images from the test dataset are used as inputsto the trained DL model in order to predict the number of wheat spikes usingNMS and DME algorithms. The resulting NMS and DME were then used asinput variables in the trained LR model to generate the ultimate counting results.34Total datasetTrainTest CountDL modelNMSDMETraining the DL modelNMSDMETraining the LR modelLR modelTest resultsFigure 3.5: The proposed wheat spike counting approach based on NMS andDensity Map Estimation (DME) algorithms.As will discuss in Section 3.3. Experimental Results and Discussion, applying theLR model followed by attaining the results from the DL model can enhance theaccuracy and reliability of our models’ counting results.3.2.6 Data preparationIn the present model, the models are firstly trained and evaluated on a benchmarkdataset called ACID Pound et al. [36]. According to Figure 3.6, the images fromthe ACID dataset were captured under controlled conditions. So, in the next step,the ACID-based trained models are used to predict a more complex in-field dataset35called GWHD, which was published in 2020 [136] and revised in 2021 [137]. Inthis subsection, the preparation procedure of these two datasets are discussed.ACID datasetTo compare the performance of the proposed models with that of the ones testedin prior studies, the ACID dataset was used for training and evaluation purposes.The dataset comprises a set of 534 wheat plant images under glasshouse condi-tions and incorporates the dot annotations for both wheat spikes and spikelets.However, the present research seeks to count and localize spikes (not spikelets).After eliminating the problematic samples without any wheat spikes, 520 images(same as the original study) with 4158 annotated spikes were attained.The challenge of the ACID dataset is that occluded spikes are annotated aswell [37]. Besides, some wheat spikes in the images are excessively adjacent toeach other and overlap most parts of the wheat spike, making it very difficult forthe DL algorithms to detect them separately. On the other hand, since this datasetis simpler than in-field datasets, the expected accuracy of wheat localization andcounting is significantly high.The image annotations and some challenging conditions are shown in Fig-ure 3.6. According to part (a) of this figure, some spikes in the dataset lack anno-tation. Since DL algorithms probably detect and count unannotated spikes, suchweakness can weaken the DL model performance. Also, in some spikes, two dotannotations were applied for one wheat spike, convoluting the model once again.Figure 3.6 (b) shows the cases in which two or more spikes are too close to eachother, which is also a challenging situation for DL algorithms since they tend totake into account all of them as one single spike. Part (c) of Figure 3.6 revealssevere occlusion conditions of some spikes in the dataset. Based on the figure,DL should detect some spikes merely using their tips or awns.The original size of the images in the dataset is 1956×1530 pixels. However,the images must be downsized due to hardware restrictions. The image scalingprocedure applied in this model is shown in Figure 3.7. According to the figure,36(a)(b)(c)Figure 3.6: Challenges of the ACID dataset wheat localization and counting.(a) Problematic annotations. (b) Closed spikes. (c) Occlusion.the images are first resized to 512×512 pixels and then cropped to 256×256 pixelsto be considered as the model input. The image resizing is done prior to croppingto enable the model to see a wider image context following the cropping.According to Figure 3.7, image resizing is preceded before the training pro-cedure, while cropping is done during the training procedure. Accordingly, after37Randomly spike selection Start epoch NCrop imageResizingLoad image MN = 1M = 1M = M + 1Update the DL weightsN = N + 11956 1530´512 512´ 512 512´256 256´Figure 3.7: Resizing and cropping procedures of input images.loading each image to the model, one of the wheat spikes is randomly selected,and a 256×256 pixels cropping is performed centering on the selected spike po-sition. Using this procedure, the dataset varies in each epoch by cropping theresized images centered on random spike selection. As a result, even though eachtraining image is used once, only a small portion of the dataset is discerned by themodel during each epoch. The advantage of this procedure is to reduce overfittingand elevate the variety of the images. However, the disadvantage of image crop-ping according to random spike selection is that the DL model requires a greaternumber of epochs to see the whole dataset and train completely [36].GWHD datasetThe GWHD dataset is a collection of 6500 images from 16 institutions in 12countries. Unlike the ACID dataset that was published with dot annotations, theGWHD dataset was published with bounding boxes. As a result, we had to dotannotate the images of the GWHD dataset to use them in this research. To accom-plish this, we developed a code using Python programming language and OpenCVpackage to assist the experts in transferring the bounding boxes into the dot an-notations. Then, we sampled 193 images from the GWHD dataset and annotatedeach one with a dot at the tip of wheat spikes.Figure 3.8 illustrates a few examples of the sampled GWHD wheat imagesalong with their dot annotations. According to this figure, the GWHD dataset38Figure 3.8: A selection of the GWHD wheat images with their associateddot annotations.is significantly more complex than the ACID dataset due to the abundance ofadjacent and occluded spikes.According to Figure 3.5, the datasets used in the presented models shouldbe divided into three distinct parts: training, counting, and testing. As statedpreviously, the training dataset is used to train the DL models, the counting datasetis used to train the LR models, and the testing dataset is used to evaluate themodels’ performance. It should be noted that, as expected, the testing dataset isnot used to tune the weights or hyperparameters. On a total of 193 annotatedimages, we selected 97 samples for the training set, 46 samples for the countingset, and 46 samples for the testing set.The images in the GWHD dataset were originally 1024×1024 pixels in size.We followed the same procedure for the dataset preparation as we did for the39ACID dataset (Figure 3.7). As a result, the input images were first resized to512×512 pixels and then randomly cropped to 256×256 pixels during training.Furthermore, the GWHD dataset was evaluated using the same quality metricsused to assess the ACID dataset’s localization and counting performance.3.2.7 Model evaluationAfter training DL models and predicting the density maps, NMS was applied tofind the position of each wheat spike. To evaluate the performance of the proposedmodels in wheat spike localization, we used F1 score, which is defined as follows:F1 =2× precision× recall(precision+ recall)(3.3)where:precision =T P(T P+FP)(3.4)recall =T P(T P+FN)(3.5)Here, True Positive (TP), False Positive (FP), and False Negative (FN), repre-sent True Positive, False Positive, and False Negative, respectively. The definitionof TP, FP, and FN specific to our case study is defined as follows:TP: The model predicts that there is a wheat spike in a position, and there is aspike in that position according to the ground truth annotations.FP: The model predicts that there is a wheat spike in a position, but there isnot any spike in that position according to the ground truth annotations.FN: The model does not predict the existence of any spike in a position, butthere is a spike in that position according to the ground truth annotations.According to the presented definitions, we can define the precision and recallas well:precision: Fraction of DL model predictions that have been detected correctly.recall: Fraction of ground truth annotations that have been detected correctly.40According to Equation (3.3), F1 score, which is applied for evaluating thelocalization performance of the models, uses the combination of precision andrecall via a simple harmonic mean (weighted average).Based on the presented equations, TP, FP, and FN should be calculated toestimate the F1 score. To this end, the procedure proposed in Pound et al. [36] wasused. In this approach, it is necessary to define a Successful Distance Threshold(SDT). To calculate TP and FP, this SDT defines a circle around each groundtruth dot annotation. If DL-NMS model predicts wheat spike position within theSDT circle, it is considered TP, and if it predicts wheat spike position exteriorto the SDT circle, it is considered FP. To calculate the FN, SDT circle shouldbe defined around the DL-NMS prediction points. Accordingly, any ground truthannotation that is not within those SDT circles is considered FN. Figure 3.9 showsan example of TP, FP, and FN points in an image. In this figure, ground truthannotations are shown in red color, and DL-NMS predictions are shown in yellowcolor. According to Figure 3.9, all ground truth dot annotations in the proposedmethod are located at the top of wheat spikes. As a result, if the length of awheat spike exceeds the radius of the SDT circle and the position of the DL-NMSprediction is at the bottom of the wheat spike, an FP point is produced.Determining an appropriate SDT amount is challenging. Based on the statedTP, FP, and FN calculation procedures, it can be concluded that higher SDT in-creases the TP, while reduces the FP and FN, leading to higher precision andrecall, and as a result, higher F1 score. On the other hand, with the same logic,reducing the size of SDT results in a lower F1 score. To resolve this problem,previous studies proposed normalized SDT [36, 37]. The authors calculated SDTas the multiplication of a predefined coefficient by wheat spike length and used0.2 and 0.1 as the constant coefficients. However, this procedure has two differentdrawbacks that are not in line with the objectives pursued by this paper:a) According to the previous figures, since ACID dataset images were taken inside view, it is feasible to measure the wheat lengths with respect to the images.However, for most of other wheat datasets such as the ones used in Xiong et al.41FNSDTTPFPTP: True PositiveFP: False PositiveFN: False NegativeSDT: Successful Distance ThresholdFigure 3.9: An example of TP, FP, and FN.[133], Chandra et al. [118], Ma et al. [43], and Yiwen et al. [138], the datasets im-ages were taken in nadir view. Wheat length calculation in these datasets throughimages is not possible, and consequently, normalized SDT is not available.b) As it was discussed in Section 3.2. Wheat Spikes Localization and Count-ing Methodology, when the intention is to count and localize small objects suchas wheat spikes, performing a point-like annotation is much faster than object de-tection or instance segmentation annotation. However, even by having side viewimages, wheat spike length calculation entails other practices, such as image scal-ing and wheat spike bottom annotation.As a result, a fixed number of pixels were utilized in the present evaluationinstead of SDT . It is much faster and also applicable to nadir view datasets (such asthe dataset that is used in Section 3.2.6. GWHD dataset). However, the drawbackof this approach is that it should be adjusted according to each specific dataset.As mentioned, previous studies on the ACID dataset used normalized SDT .The average normalized SDT of all 4158 wheat spikes is 45.95 and 21.97 forconstant coefficients of 0.2 and 0.1, respectively. It was debated that higher SDT42leads to higher F1 score. In other words, if the model can reach to higher F1 scorewith lower amounts of SDT , it would be capable of more precise localization ofwheat spikes in their exact position. In the present study, all the models wereexamined with various SDTs to measure their performance in higher and loweramounts. The SDT amounts that we considered are 30, 25, 20, 15, and 10 pixels.To evaluate the proposed algorithms’ counting performance, same as the lo-calization approach, our models were tested in 100 epochs, and the average per-formance was calculated. Accordingly, Root Mean Squared Error (Root MeanSquared Error (RMSE)) and MAPE statistical indices were utilized as follows:RMSE =√√√√√ N∑i=1(GTi−Oi)2N(3.6)MAPE =N∑i=1∣∣∣(GTi−Oi)/GTi∣∣∣N×100 (3.7)where GTi and Oi represent the Ground Truth and DL counting Output forith image, and N is total image number. As we used 100 epochs to evaluate ourmodels, N can be calculated according to the following equation:N = 100× (number of testing images) (3.8)3.3 Experimental Results and DiscussionThe current research develops and implements four new UNet-based architec-tures, including VGG16-UNet, ResNet34-UNet, ResNet50-UNet, and ResNeXt-UNet, in addition to UNet, UNet++, and UNet3+ architecture, which were pre-sented in the previous studies. Additionally, as discussed in Section 3.2. WheatSpikes Localization and Counting Methodology, the CDM algorithm and hybridNMS-DME are introduced for the purposes of localization and counting the wheat43spikes, respectively. The purpose of this section is to evaluate the proposed method-ology that was illustrated in Figure 3.1. This section is divided into two parts.In the first part, the proposed models are trained and evaluated on the ACIDdataset. According to Subsection 2.1.1. Wheat Spike Localization and Counting,this dataset has been used as a benchmark in a number of studies to determinethe location and count of wheat spikes. After training the proposed models on theACID dataset, the second part will evaluate their capability on an in-field datasetcalled GWHD in order to determine their performance in real-world conditions.3.3.1 Modeling the benchmark dataset (ACID)Wheat spike localization results attained from the DL models are presented inTable 3.2. As it was mentioned previously, in these models, random cropping wasdone on input images during training. As a result, the images presented to themodel in each epoch were diversified. Therefore, to evaluate the performance ofthe models, the average value of 100 epochs of the DL models on the test datasetwas calculated.Table 3.2 presents two different versions of DL models: a) Usage of pretrainedweights in the feature extraction part of the network and only training the up-sampling part, and b) Training of the models from scratch and using pretrainedweights for initialization. Furthermore, Table 3.2 compared the performance ofDL models by using GDM and CDM approaches of density map generation. Ac-cording to the table, the pretrained-CDM-based VGG16-UNet model attained thehighest F1 score for all the considered SDT amounts. This model can localizethe wheat spikes with F1 score higher than 0.94 even with SDT as small as 10.The current study results outperform those of Pound et al. [36] with F1 score of0.89 and 0.83 at SDTs of 45.95 and 21.97, respectively. In addition, our proposedmodel outperforms Hu et al. [37] with F1 score of 0.905 at SDT of 45.95.In order to visualize the effect of using the fixed pretrained weights on featureextraction (i.e., encoding) part of ResNet34-UNet, ResNet50-UNet, ResNeXt-UNet, and VGG16-UNet models, bar plot of their F1 scores in testing dataset44Table 3.2: Evaluating the wheat spike localization on the ACID testingdataset using F1 score.Density map Training Model name SDT30 25 20 15 10GDMScratchUNet 0.919 0.908 0.899 0.883 0.855UNet++ 0.903 0.884 0.871 0.853 0.825UNet3+ 0.918 0.906 0.892 0.873 0.849ResNet34-UNet 0.920 0.905 0.893 0.872 0.845ResNet50-UNet 0.935 0.916 0.900 0.886 0.853ResNeXt-UNet 0.924 0.914 0.903 0.882 0.864VGG16-UNet 0.916 0.897 0.880 0.863 0.834PretrainedResNet34-UNet 0.904 0.885 0.869 0.846 0.806ResNet50-UNet 0.897 0.882 0.859 0.837 0.812ResNeXt-UNet 0.895 0.876 0.854 0.830 0.796VGG16-UNet 0.891 0.879 0.862 0.848 0.819CDMScratchUNet 0.961 0.955 0.949 0.943 0.933UNet++ 0.945 0.937 0.931 0.927 0.921UNet3+ 0.958 0.952 0.945 0.938 0.925PretrainedResNet34-UNet 0.921 0.913 0.907 0.901 0.893ResNet50-UNet 0.935 0.930 0.922 0.912 0.903ResNeXt-UNet 0.923 0.916 0.912 0.903 0.895VGG16-UNet 0.964 0.958 0.952 0.948 0.941was shown in Figure 3.10 (a). In this plot, models with fixed pretrained weightsare shown by P and models that were trained end-to-end from scratch are shownby S. Figure 3.10 (a) plots the results for SDT of 30 and using conventional GDMin generating the density maps. According to the figure, weight freezing in the en-coding part weakens the performance of the models up to 2%. This is predictablesince we trained all the models for 2000 epochs, and even when all model weightsare trained from scratch, the models have sufficient epochs to adjust all the weightsthoroughly. On the other hand, model training from scratch contributes to a dra-matic increase in the number of trainable parameters. For instance, concerningVGG16-UNet, as the optimal model in wheat spike localization, the number of45trainable parameters increases from 34.7e6 for the fixed encoding part to 49.4e6for training the model from scratch.Figure 3.10 (b) reveals the effect of the introduced CDM approach in gener-ating the density map on model performance. The figure comprises the modelsfor SDT of 30. In addition, for all ResNet34, ResNet50, ResNeXt, and VGG16-UNet models, we used fixed pretrained encoding. According to this figure, CDMsignificantly increases the localization performance of networks (around 5%) incontrast to the conventional GDM algorithm. This better performance is due tothe specific way adopted by NMS in wheat spike localization in DL output densitymaps. In the NMS algorithm, a predicted pixel is a wheat spike position if it hasa maximum amount in contrast to its neighbor pixels. According to Figure 3.4,when using GDM, the ground truth density map has only one maximum pixel foreach wheat in the image. As a result, it is difficult for DL algorithms to model ex-actly that specific maximum pixel in the output density map. When CDM is used,on the other hand, all 9×9 pixels in the ground truth density map that correspondsto a dot in the raw annotated ground truth have similar values. As a result, themodels easily recognize and locate them as wheat spikes.Figure 3.11 shows the wheat spike localization accuracy of VGG16-UNet forthe first 600 epochs on both training and testing datasets. Although the model wastrained for 2000 epochs, its performance plateaued around 500 epochs. Addition-ally, as illustrated in Figure 3.11, the models’ performance in training and testingdatasets is very close, indicating that they are not subject to overfitting [122, 139].It also should be noted that the results in Table 3.2 and Figure 3.10 were attainedusing the testing dataset and by the last epoch model, not necessarily the bestmodel during training.According to Table 3.2, the VGG16-UNet model with pretrained encodingpart and CDM density map generation algorithm achieved the best results in com-parison with other models. Some examples of the model are plotted in Figure 3.12.In the figure, some challenging situations handled amazingly by our DL model areshown in red boxes. In the first row, only a few pixels of two wheat spike tips are460.850.900.951.00S P S P S P S PF1ResNet34-UNetResNet50-UNetResNeXt-UNetVGG16-UNet0.850.900.951.00G C G C G C G C G C G C G CF1ResNet34-UNetResNet50-UNetResNeXt-UNetVGG16-UNetUNetUNet++UNet3+(a) (b)Figure 3.10: Localization performance of DL models trained using differentprocedures on the ACID testing dataset. (a) compares the DL modelsusing fixed weights in the feature extraction part (P) and DL mod-els without any fixed weights (S). (b) compares the DL models withGaussian (G) and Constant (C) density maps.visible in the image. Furthermore, two spikes overlap each other, and it is verydifficult to recognize the tip of the lower one. In the second row, one wheat spikeis occluded with a wheat leaf, making it very difficult to detect even with humanvision. In the third row, two spikes overlap each other, and they are occluded bywheat leaves. On the other hand, in this image, there are three spikes in the rightcorner of the image. However, since we annotated the tips of the three spikes,they are not available in the ground truth density map. The right column imagesindicate that the VGG16-UNet model is powerful enough to recognize and detectall of the aforementioned challenges thoroughly.As it was discussed, data augmentation can significantly increase the perfor-mance of DL models. This is even more important when using the ACID dataset.To investigate the effect of data augmentation on the accuracy of the proposed47(a) (b)00.20.40.60.810 200 400 600F1Epochtrain 30train 25train 20train 15train 1000.20.40.60.810 200 400 600F1Epochtest 30test 25test 20test 15test 10Figure 3.11: Wheat spike localization performance on the ACID datasetover (a) training and (b) testing procedures for the first 600 epochsand different SDT amounts.models, all were trained without any augmentation. To this end, the conventionalGDM algorithm was utilized to generate a density map and fixed encoding partfor the pretrained-based UNet models. The results for 30, 20, and 10 SDTs areshown in Figure 3.13. According to the figure, data augmentation significantlyimproves the testing results of the models and prevents overfitting.As it was stated in Section 3.2. Wheat Spikes Localization and Counting Method-ology, the current study also proposed a new algorithm for accurate wheat count-ing in the images. In this approach, NMS results are combined with DME resultsvia an LR model. To this end, as indicated in Figure 3.5, the whole dataset isdivided into three parts: training dataset, counting dataset, and testing dataset. DLmodel is trained using the training dataset, the counting dataset is applied for LRmodel training, and the testing dataset is used for result evaluation.The results of the wheat spike counting models are presented in Table 3.3. Ac-cording to this table, ResNeXt-UNet, which was trained end-to-end from scratch,48Figure 3.12: Some examples of VGG16-UNet results on the ACID testingdataset. The left column is the cropped 256×256 pixels image, themiddle column is the corresponding ground truth annotation usingthe CDM algorithm, and the right column is the DL model’s output.The red rectangles in this figure illustrate some of the challengingpredictions of the used DL model.performs much better than other models with MAPE of 1.68%. In addition, among490.700.750.800.850.900.951.0030 20 10 30 20 10 30 20 10 30 20 10 30 20 10 30 20 10 30 20 10UNet UNet++ UNet3+ ResNet34-UNetResNet50-UNetResNeXt-UNetVGG16-UNetF1With augmentation Without augmentationFigure 3.13: Augmentation effect on the ACID testing dataset for the con-sidered models with SDT of 30, 20, and 10.the models with fixed pretrained encoding, the VGG16-UNet model with MAPEof 3.367 outperforms the other models. As it was discussed in the previous part,the choice of the best network among scratched ResNeXt-UNet and pretrainedVGG16-UNet depends on the available training resources because the trainableparameter number of the former model is 104e6 while the trainable parameternumber of the latter model is only 34e6. This table also shows that UNet3+performs very well with the proposed counting approach with MAPE of 2.55%and a much smaller number of trainable parameters (26.9e6). Using the samedataset, Pound et al. [36] attained mean percentage error of -14.46%, -5.13%, and-4.09% for input resolutions of 256×256 pixels, 384×384 pixels, and 512×512pixels, respectively, which are much higher than the errors of the proposed modelsin the current paper. A comparison of the performance of the pretrained modelswith the models trained from scratch shows that all of the residual-based networks(ResNet34-UNet, ResNet50-UNet, and ResNeXt-UNet) perform better when theytrain end-to-end without any frozen parameter. On the other hand, the VGG16-UNet model performs better when trained with fixed parameters in the encoding50Table 3.3: Evaluating the wheat spike counting on the ACID testing datasetusing Root Mean Squared Error (RMSE) and Mean Absolute Percent-age Error (MAPE) indices.Training Model name RMSE MAPEScratchUNet 12.921 4.122UNet++ 12.008 12.008UNet3+ 8.401 2.555ResNet34-UNet 13.956 4.524ResNet50-UNet 9.917 3.095ResNeXt-UNet 5.936 1.680VGG16-UNet 13.289 4.282PretrainedResNet34-UNet 15.216 4.873ResNet50-UNet 18.549 6.195ResNeXt-UNet 25.445 8.572VGG16-UNet 10.628 3.367part.Comparing Table 3.2 and Table 3.3 demonstrates that the optimal model forwheat spike localization (VGG16-UNet) is different from the optimal model forwheat spike counting (ResNeXt-UNet). This discrepancy can be explained by thefact that wheat spike localization performance is strongly dependent on the NMSalgorithm. In contrast, wheat spike counting performance is dependent on boththe NMS and DME algorithms (Figure 3.5). To perform well in the NMS algo-rithm, the DL model should predict the exact location of the wheat spikes, whichis referred to as TP, and predict as low FP and FN points as possible. However, amodel’s performance in DME is contingent upon its ability to predict the summa-tion value of each density map accurately. It can be seen that the proposed model-ing procedures take a completely different approach to localization and countingthe wheat spikes, and therefore selecting two distinct models for these tasks is notunexpected.513.3.2 Modeling the in-field dataset (GWHD)As it was presented, we achieved exceptional results in localizing and countingwheat spikes by utilizing the ACID dataset. This subsection aims to evaluate theperformance of the presented models in real-world situations using the GWHDdataset. Comparing Figure 3.8 to Figure 3.6 demonstrates that a DL model hasa much easier time localizing and counting wheat spikes in the ACID datasetthan it does in the GWHD dataset. In the GWHD dataset modeling procedure,we used DL models that had been pretrained on ACID datasets. The purposeof using pretrained DL algorithms from the ACID dataset to model the GWHDdataset is to investigate the effect of transfer learning on the performance of wheatspike localization and counting in a new dataset. However, it is worth notingthat, despite its complexity, the GWHD does not properly reflect all in-field wheatspikes situations for two reasons. The first reason is that every wheat spike in theGWHD dataset originates from the same variant. Thus, the wheat spikes seem tobe similar in shape and size, and the second reason is that we used a small portionof this dataset for our research.In addition, we did not test all of the DL algorithms used in the previous sub-section for localization and counting the wheat spikes when modeling the GWHDdataset. Rather than that, we used the most optimal models for the localization andcounting of wheat spikes. According to Table 3.2, the best model for wheat spikeslocalization using the ACID dataset is the VGG16-UNet model with a pretrainedencoding part and the CDM density map generation algorithm, while according toTable 3.3, the best model for wheat spikes counting is the ResNeXt-UNet modelthat was trained end-to-end from scratch. As a result, we used the first and secondmodels for wheat spike localization and counting in the GWHD model, respec-tively.Figure 3.14 provides the results of the GWHD localization and counting. Asexplained previously, we employed transfer learning with pretrained weights onthe ACID dataset to model the GWHD dataset. According to Figure 3.14 (a) and(b), we modeled the GWHD dataset using 21 distinct scenarios. In the first sce-520 10 20 30 40 50 60 70 80 90 100Epochs0123456Absolute percentage errorWithout additional training (Epochs=0)With additional training (Epochs=200, Samples=97)MAPE = 2.302MAPE = 1.56710 30 50 70 97Samples1001502004.0264.44333.8693.4741.7045.1233.3572.8942.2993.3722.3792.2791.5677.481 22.533.544.555.566.57(a)(c)10 30 50 70 97Samples50100150200Mean of F10.69290.80680.81440.86570.86720.84910.85980.86380.85360.87270.87420.88660.89350.89110.87570.89760.89230.89310.90570.9105 0.60.650.70.75F110 30 50 70 97l s050100150200EpochsMean of MAPE2.3022.671..2.5733.209..2.3431.718...2.7961.661...2.8622.337.... 22.533.544.555.56.510 30 50 70 97l s050100150200Epochs 0.5736 0.5742.0.5777 0.5766 0.5727.............. 0.60.650.70.750.80.850.9EpochsEpochs0 4950 . . . ..0.85.0 (b) MAPE2. 02Figure 3.14: Results of wheat spikes localization and counting on theGWHD testing dataset. (a) shows the localization results using F1score. (b) shows the counting results using MAPE. (c) compares themodel without any additional training on the GWHD dataset to themodel that was retrained for 200 epochs using 97 samples from theGWHD dataset.nario, we did not perform any additional training on the DL models trained usingthe ACID dataset. However, in the rest scenarios, the weights of the DL modelswere tuned by performing some additional training. To do that, we retrained themodels for four different epochs of 50, 100, 150, and 200, as well as five different53training sample sizes of 10, 30, 50, 70, and 97. In this figure, epoch 0 indicatesthat we did not train the DL models on the GWHD dataset, and we only used theACID-based pretrained weights in modeling the GWHD dataset. Epoch 50 indi-cates we retrained the ACID-based pretrained weights for only 50 epochs usingthe GWHD dataset. The sample size of 10 indicates that we considered merely10 images from the GWHD dataset in the training set, while the sample size of30 indicates that we considered 30 images from the GWHD dataset in the trainingset.The results of localization using the F1 score are shown in Figure 3.14 (a).According to this figure, the F1 score is 0.5749 when no training on the GWHDdataset is done, and only the ACID-based pretrained weights are used. Accordingto Figure 3.11, the F1 score for the first epoch was zero when we modeled theACID dataset. This demonstrates the importance of transfer learning in wheat lo-calization. On the other hand, as presented in Figure 3.14 (a), the F1 score can beraised to 0.8672 by using only ten images of GWHD in the training dataset andretraining the model for additional 200 epochs. This demonstrates that when wehave a pretrained model on a different dataset, we may achieve excellent perfor-mance by annotating a small number of images from the new dataset and includingthem in the training set. Finally, as expected, Figure 3.14 (a) demonstrates thatretraining the DL model with all training samples and the maximum number ofepochs results in the highest F1 score.The results of the wheat spikes counting using the MAPE quality metric areshown in Figure 3.14 (b). As with the localization results, the best counting re-sult was obtained when 97 training samples were employed, and the pretrainedDL models were retrained for 200 epochs. One point worth noting about Fig-ure 3.14 (b) is that when we utilize a small number of training samples (10 images)and train for a high number of epochs (200 epochs), the DL model may becometrapped in overfitting, dramatically reducing modeling performance in the testingset.Figure 3.14 (c) compares the performance of the scenario of using the DL54Figure 3.15: Some examples of wheat spikes localization model on theGWHD testing dataset. The left column is the cropped 256×256pixels image, the middle column is the corresponding ground truthannotation using the CDM algorithm, and the right column is the DLmodel’s output. The red rectangles in this figure illustrate some ofthe challenging predictions of the used DL model.model that was trained on the ACID dataset without any additional training on the55GWHD dataset with the scenario of using the DL model that was trained on theACID dataset and retrained with 97 samples of GWHD dataset for 200 additionalepochs. According to this graph, while the prediction performance of the modelwith additional training is better in comparison with the model without any addi-tional training, the difference between the MAPE of these models is only 0.735percent. This low difference between the performance of the compared mod-els shows that without any additional training on the GWHD dataset, the modeltrained using the ACID dataset can detect wheat spikes with satisfactory accuracy.Figure 3.15 shows some examples of the localization results from the GWHDdataset. This figure demonstrates that the localization model is sufficiently pow-erful to identify and locate the majority of wheat spikes in a complicated in-fielddataset such as GWHD. Additionally, the red boxes in this image demonstratethat even though some of the wheat spikes in the ground truth do not have a dotannotation (due to a mistake or random cropping technique), the presented modeleffectively spotted them.The proposed models’ performance in localizing and counting wheat spikesin the field is compared to previous research studies in Table 3.4. It is impor-tant to note that each of the compared research studies used a different dataset.However, all of them attempted to localize or count wheat spikes in the field forpractical purposes. Due to the fact that the datasets used in the compared studiesare distinct, it is not possible to compare their performance using scale-dependentstatistical indices such as RMSE. As a result, we compared the performance ofcurrent research to that of previous research that used scale-independent statisti-cal indices such as MAPE to report their findings. Additionally, because some ofthe compared research studies reported model performance in terms of accuracypercentages, we calculated the Mean Absolute Percentage Accuracy (MAPA) as100 minus MAPE [145] and presented it in Table 3.4. Furthermore, some of theresearch studies in Table 3.4 used the Relative Root Mean Squared Error (rRMSE),which is calculated by dividing the RMSE by the average value of the ground truthmeasurement [146]. As a result, we calculated the rRMSE in the table to compare56Table 3.4: Comparing the in-field wheat spikes localization and countingperformance of the proposed model with previous research studies.References Description/Used model Statistical indexMAPA rRMSE F1Current study Pretrained using the ACID datasetwithout any additional training onthe GWHD dataset97.698 2.8459 0.5749Current study Pretrained using the ACID datasetand retrained using 97 GWHD im-ages for 200 epochs98.433 1.8357 0.9105Hasan et al. [129] Faster-R-CNN 93.4 — 0.95Wang et al. [131] FCN-Harris corner detection 97.4 — —Sadeghi-Tehran et al. [132] Linear iterative clustering-CNN — 15.5 —Madec et al. [130] Faster-R-CNN — 6 —Xiong et al. [133] TasselNetv2 91.01 — —Yiwen et al. [140] VGG-localization-based countingloss90.4 — —Ma et al. [43] CNN-FCN — — 0.8177Misra et al. [48] SpikeSegNet 95 — —Yang et al. [141] CBAM-YOLOv4 — — 0.8925Bhagat et al. [142] WheatNet-Lite — — 0.8571Li et al. [143] Faster R-CNN 86.7 — —Ullah et al. [144] EfficientDet-D7 94.03 — —our models’ performance to that of the research studies.Table 3.4 contains two distinct versions of the GWHD modeling depicted inFigure 3.14 (a and b). In the first version, the DL model was fully trained us-ing the ACID dataset without additional training on the GWHD dataset. On theother hand, in the second version, the DL model was retrained using 97 imagesfrom the GWHD dataset for 200 epochs after being fully trained using the ACIDdataset. According to this table, both of the provided models outperformed previ-ous studies in terms of counting wheat spikes (MAPA and rRMSE). Additionally,by retraining the proposed models for 200 epochs with a small number of trainingimages, the model was able to achieve a high level of localization performance(F1 score). Given that the proposed DL models were not fully trained on theGWHD dataset, the results in Table 3.4 demonstrate the proposed method’s high57generalization ability for use in practical applications.3.4 SummaryNumerous studies have demonstrated the efficacy of computer vision-based high-throughput phenotyping for agricultural performance evaluation, yield estimation,and farm monitoring. Despite the fact that DL models are widely regarded as themost effective computer vision approaches in many agricultural disciplines, theyrequire a large amount of well-annotated data, which can be expensive and time-consuming. For the purpose of wheat spike localization and counting, variousdata annotation strategies, such as bounding boxes for object detection, polygonsfor instance segmentation, and point-like annotation, can be implemented. How-ever, point-like annotation can significantly reduce the data preparation time for awheat phenotyping dataset with numerous tiny spikes.The primary objective of this chapter is to develop novel techniques for wheatspike localization and counting by utilizing dot annotation datasets to improve thepreviously proposed algorithms. The present chapter introduced four pretrainedarchitectures based on the UNet algorithm, namely VGG16-UNet, ResNet34-UNet, ResNet50-UNet, and ResNeXt-UNet, and compared their performance tothat of the original UNet and two state-of-the-art architectures, UNet++ and UNet3+.For wheat localization, a novel method of density map development called CDMwas introduced, which outperformed the conventional GDM algorithm by a sig-nificant margin. For wheat spike counting, a novel approach was used to combineNMS, based on localization, and DME, based on density estimation, to achievemore reliable and accurate performance. Using the ACID dataset as a benchmark,the VGG16-UNet model with an F1 score of 0.96 improved wheat spike local-ization by 15% when compared to the stacked hourglass DL model presented inPound et al. [36] and by 6% when compared to the hybrid Generative AdversarialNetworks (GAN)-attention mechanism DL model presented in Hu et al. [37]. Onthe other hand, the ResNeXt-UNet model with an MAPE of 1.68% in counting thewheat spikes outperformed the stacked hourglass DL model presented in Pound58et al. [36] by 143%.The other objective of this chapter is to determine the generalization abilityof the proposed methods in localizing and counting wheat spikes in practical sit-uations. With this aim, the investigated models were first fully trained for 2000epochs on a simple dataset called ACID. Following that, the trained models’ per-formances were evaluated in terms of localization and counting wheat spikes inan in-field dataset called GWHD using no additional training. While predictingwheat spikes in the ACID dataset is significantly easier than in the GWHD dataset,the results indicated that the DL models were capable of localizing and countingwheat spikes in the GWHD dataset with an F1 score of 0.57 and MAPE of 2.3%,respectively. On the other hand, the results indicated that by retraining the DLmodels for 200 epochs and 97 samples from the GWHD dataset, the obtained F1score and MAPE could be improved to 0.91 and 1.56, respectively.The current chapter developed a novel algorithm for localizing and count-ing wheat spikes with high accuracy. It was demonstrated that the proposed al-gorithms exhibit a high degree of generalization when predicting in-field wheatspikes. Thus, in practice, trained models can accurately model unobserved datasets.Additionally, the proposed dot annotation can significantly reduce the time andeffort required for data preparation, particularly in wheat spike modeling, wherethousands of small objects must be annotated.While the current research demonstrates remarkable accuracy in localizingand counting wheat spikes, it does have room for improvement in future research.The first recommendation for future research is to enhance the used augmenta-tion algorithm. Due to the similar shape and size of the majority of wheat spikes,an improved augmentation technique, such as object level augmentation, can fur-ther reduce the number of training samples required to achieve acceptable results.Second, future research can focus on reducing the number of trainable parametersin DL models. The smallest DL model used in this chapter has 26.9e6 trainableparameters. This amount, however, can be reduced by reducing the number oftrainable layers while maintaining acceptable performance. Using shallower DL59models can be extremely beneficial, particularly in practical applications wheretraining run time is a critical parameter.60Chapter 4AutoOLA: Automatic Object Level Aug-mentation for Wheat Spikes Count-ingA version of this chapter has been published in Computers and Electronics inAgriculture as a research paper titled “AutoOLA: Automatic Object Level Aug-mentation for Wheat Spikes Counting”.4.1 OverviewThe ability to count wheat spikes is one of the most important indicators in wheatplant agriculture that correlates positively with yield estimation. Similar to themodels that are presented in Chapter 3. Wheat Spike Localization and Count-ing via Hybrid UNet Architectures, numerous studies have developed machinevision-based high-throughput phenotyping algorithms and demonstrated that thedeveloped models could accurately detect and count wheat spikes. However, thedrawback of previously developed models is their expensive and time-consumingdata preparation process. While capturing images of wheat is relatively simple,annotating thousands of small wheat spikes in the collected images requires anenormous amount of time and effort. This chapter, therefore, proposes a novel61augmentation algorithm, Automatic Object Level Augmentation (AutoOLA), thatcan considerably reduce the required number of training samples in Deep Learn-ing (DL) models. In this algorithm, different objects in the wheat images, in-cluding the spikes, leaves, stems, and backgrounds, are decoupled and augmentedindependently based on their own augmentation policy. The augmented imagesare then generated by randomly assembling the augmented objects. In addition,an evolutionary optimization algorithm is used to automatically design the typesof transformations and their magnitudes for each decoupled object. The resultsdemonstrated that the proposed AutoOLA technique could improve wheat spikecounting performance by up to 60 percent. With only one trainable image, theemployed hybrid AutoOLA-DL model can predict the number of wheat spikes inthe field with Root Mean Squared Error (RMSE), Mean Absolute Error (MAE),and Mean Absolute Percentage Error (MAPE) values of 2.695, 2.085, and 13.817,respectively.4.2 AutoOLA MethodologyIn this approach, after extracting various objects from the input images, the aug-mentation policies are designed and optimized using evolutionary optimizationalgorithms. The optimized augmentation policies are then used to manipulateeach extracted object individually. After that, the augmented dataset is generatedby assembling the manipulated objects into the images. In the final step, the aug-mented dataset is used to train the Full-Convolutional Neural Network (F-CNN)model, and the testing dataset generates prediction results.Figure 4.1 illustrates an overview of the proposed AutoOLA. According to thisfigure, AutoOLA is comprised of four distinct components: extracting and assem-bling the Region of Interests (ROIs), defining the augmentation policy structures,optimizing the augmentation policies, and designing the used DL models. Each ofthese components will be explained in this section. The source code of AutoOLAis available on GitFront 1.1https://gitfront.io/r/user-9708596/A2o5JS7hfWm9/AutoOLA/62Start GA iterationParse the populationStartExtract the ROI imagesExtract the spike ground truth annotations in the ROI imagesSpecify GA parametersGenerate the initial populationSpecify the augmentation typesSpecify the magnitude of augmentationsSpecify the number of elements in the imageEvaluate the cost functionTrain the C-CNNEvaluate the resultsGenerate 10 OLA imagesPerform the crossoverPerform the mutationMerge the population with offspringSort the resultsSelect the first npopresultsCheck the termination criteriaReturn the optimized resultsGenerate 200 OLA imagesTrain the F-CNNReturn the resultsEndFigure 4.1: Overview of the AutoOLA algorithm.4.2.1 Objects Extracting and AssemblingTo count the wheat spikes using the proposed model, we need Dot Annotation(DA), which serves as the ground truth [147]. In the annotation procedure, eachwheat spike is marked with a dot to generate density maps [148]. In addition, thePolygon Annotation (PA) should be performed to extract ROI from the imagesduring object-level augmentation. Both the DA and PA are matrices and containbinary values. In DA, a dot with a value of one indicates the location of a spike,whereas, in PA, pixels with a value of one indicate the presence of an object atthat location. It is clear that DA is only accessible for spike ROIs. In the equationbelow, i represents the i-th ROI in the image.63DAi and PAi ∈ {0,1}width×height×1 (4.1)According to Equation (4.1), PAix,y equals one if the pixel of the input imagebelongs to the i-th object. Since each pixel in an image belongs to only one ofthe extracted objects (spikes, leaves, stems, and backgrounds), ∑ni=1 PAi is alwaysequal to one. The first step of AutoOLA, as shown in Figure 4.1, is to extract ROIsfrom the input images. With the PA masks, extracting the i-th ROI is simple usingthe equation below.ROIi = PAi · Image where ROIi ∈ Rwidth×height×3 (4.2)where 3 represents the RGB images’ red, green, and blue layers. Severalexamples of extracted ROIs are shown in Figure 4.2.As stated previously, the AutoOLA procedure manipulates each image objectindependently prior to generating the final augmented images. It is evident that theposition of pixels within an ROI is altered by Spatial-Level Transformation (SLT)such as flipping, rotating, scaling, and elastic transformation. So that the sametransformations are applied to the PA and DA masks in those instances. However,the PA and DA masks remain unchanged for Pixel-Level Transformation (PLT),such as noise injection and RGB shifting, where the position of pixels does notchange.After defining the structure of augmentation policies (Subsection 4.2.2. Aug-mentation Policy Structures) and optimizing it with the evolutionary algorithm(Subsection 4.2.3. Augmentation Policy Optimization), it is time to produce theaugmented images by assembling the manipulated ROIs. In order to accomplishthis, the number of manipulated ROIs that will be used to assemble an augmentedimage should be determined. Clearly, the utilization of more ROIs results in morecrowded augmented images and vice versa. So, to generate a suitable augmenteddataset, it is essential to determine the appropriate quantity of ROIs for each class(nspike, nlea f , nstem, and nbackground). In AutoOLA, these numbers are determined64StemsBackgroundsLeavesSpikesFigure 4.2: Some examples of an in-field image’s spike, leaf, stem, andbackground ROIs.using the employed optimization algorithm. However, to prevent the augmentedimages from becoming too crowded and reduce the search space size, we defined arange for each of them, which is discussed in Section 4.2.3. Range of optimizationvariables.The background inpainting procedure of each augmented image is performedby randomly selecting nbackground manipulated background ROIs and distributingthem throughout the augmented image. The amount of pixels not covered by themanipulated background ROIs is then determined using the subsequent equation.pl(x,y) =∑nbackgroundk=1 ROIk,l∣∣∣p(x,y) ∀p(x,y) ∈⋃nbackgroundk=1 ∣∣∣ ∀p(x,y) /∈nbackground⋃k=1ROIkbackground (4.3)65where l is the augmented image’s color channel. In the final step, the manip-ulated ROIs of nspike, nlea f , and nstem spikes, leaves, and stems are selected andrandomly inserted into the augmented image. This process is repeated until thedesired number of augmented images has been created.It should be noted that since the DL models used in the current study requiredot annotations in order to count the wheat spikes, the same inserting procedureis performed using the manipulated DA masks.4.2.2 Augmentation Policy StructuresChoosing the appropriate augmentation pipeline can be very challenging. First,there are a variety of transformation types that can be applied to the input imagesto perform the augmentation. Second, the magnitude of the transformation mustbe determined for each dataset by means of a laborious trial-and-error procedure.Therefore, inspired by Cubuk et al. [78], we have defined an augmentation policysearch space that will be optimized using the evolutionary optimization algorithm.In this study, two distinct structures for the policy search space, known astwo transformations augmenter (2aug) and five transformations augmenter (5aug),are designed and evaluated (Figure 4.3. The structure of the (a) 2aug and (b)5aug policies.). Each policy in 2aug is comprised of six sub-policies, the firstfive of which consist of two transformations and the sixth of which performs notransformation on the input ROI. In contrast, each policy in 5aug is composed ofthree sub-policies, the first two of which consist of five transformations and thethird of which performs no transformation on the input ROI. In addition, eachtransformation is associated with a hyperparameter, which is its magnitude [149].After the policy structure has been optimized, each ROI has a chance of be-ing manipulated by one of the sub-policies. Notably, in order to make policiesless stochastic, we disregard the probability of applying sub-policies during theoptimization procedure. Instead, the appropriate probability amounts were deter-mined through trial and error. In 2aug, each of the first five sub-policies has a 16percent chance of being utilized, while there is a 20 percent chance of not being66manipulated. In contrast, there is a 30 percent chance of using each of the firsttwo sub-policies in 5aug, and a 40 percent chance of not being manipulated. Theproposed augmentation policies can be expressed as follows:SP = Policy(P) (4.4)ROIiaug = SP(ROIi,M1)◦ · · · ◦SP(ROIi,Mn) (4.5)PAiaug,DAiaug = SP(PAi,DAi,M1)◦ · · · ◦SP(PAi,DAi,Mn) ∀A ∈ SLT (4.6)where Policy refers to the entire search space structure (2aug or 5aug), P isthe set of sub-policy probabilities ([p1, · · · , pn]), n is the number of sub-policies,which is six for 2aug and three for 5aug, i is the index of input objects, M isthe magnitude of each transformation, and A is the transformation type. Equa-tions (4.5) and (4.6) depict the transformations performed sequentially on the in-put ROIs, PAs, and DAs. In addition, in the case of an SLT, the transformationson the PA and DA masks are applied according to Equation (4.6).In the present study, twenty candidate transformation types are evaluated,some of which belong to the PLT type and others to the SLT type. Figure 4.4is an illustration of a spike ROI and its transformations. It should be noted thatall transformations in the current study are performed using the Albumentationsopen-source library [150].4.2.3 Augmentation Policy OptimizationTo determine the appropriate augmentation policy for counting wheat spikes, itis essential to optimize the policy structures introduced in Subsection 4.2.2. Aug-mentation Policy Structures. According to Figure 4.3, the optimization proceduremust determine ten transformation types and their magnitudes in both the 2aug67Input ROIOutput ROI2aug policyAug. type #1Aug. magnitude #1P=0.16Aug. type #2Aug. magnitude #2Aug. type #3Aug. magnitude #3P=0.16Aug. type #4Aug. magnitude #4Aug. type #5Aug. magnitude #5P=0.16Aug. type #6Aug. magnitude #6Aug. type #7Aug. magnitude #7P=0.16Aug. type #8Aug. magnitude #8Aug. type #9Aug. magnitude #9P=0.16Aug. type #10Aug. magnitude #10P=0.2No augmentationInput ROIOutput ROI5aug policyAug. type #1Aug. magnitude #1P=0.3Aug. type #2Aug. magnitude #2Aug. type #3Aug. magnitude #3Aug. type #4Aug. magnitude #4Aug. type #5Aug. magnitude #5Aug. type #6Aug. magnitude #6P=0.3Aug. type #7Aug. magnitude #7Aug. type #8Aug. magnitude #8Aug. type #9Aug. magnitude #9Aug. type #10Aug. magnitude #10No augmentationP=0.4(a)(b)Figure 4.3: The structure of the (a) 2aug and (b) 5aug policies.and 5aug policy structures. In addition, as stated in Subsection 4.2.1. ObjectsExtracting and Assembling, it is necessary to know the number of manipulatedspikes, leaves, stems, and backgrounds during the AutoOLA assembly process.68Blur MedianBlur ISONoise CLAHE DownscaleEmboss RGBShift HueSaturationValue MultiplicativeNoise RandomBrightnessContrastRandomToneCurve CoarseDropoutOriginalGridDistortion ElasticTransform PerspectivePiecewiseAffine Flip Rotate Scale ShearFigure 4.4: An example of the employed transformation types with randommagnitudes.Range of optimization variablesThe optimization model has a total of 24 dimensions, with the first ten variablespertaining to the transformation types, the second ten variables belonging to thetransformation magnitudes, and the remaining four variables relating to the num-ber of spikes, leaves, stems, and backgrounds ROIs in the augmented image.It should be noted that the first ten optimization variables are categorical (trans-69formation types), and the subsequent fourteen are continuous (transformation mag-nitudes and the number of different ROI classes). During the optimization, how-ever, we consider the range of all variables between zero and one. Consequently,in order to evaluate the outcomes, it is necessary to map the optimization outputsto their actual values.The transformation type for categorical variables is also determined by theoutput values. For instance, if the quantity of the categorical variable falls between0 and 0.05, the transformation type is Blur; if the amount falls between 0.05 and0.1, the transformation type is MedianBlur, and so on.For each specific transformation, we determined an appropriate magnituderange, which is presented in Table 4.1 as the minimum and maximum values. So,for the ten continuous variables related to the magnitudes of the transformationtypes, the optimization outputs will map to fall within the specified ranges.After determining the magnitude of each transformation type using the em-ployed optimization algorithm, two distinct impact ranges are defined and pre-sented in Table 4.1 under the names minmag and magmax. During the process ofmanipulating ROIs, a random value is selected from these ranges to represent theimpact of each augmentation type. We defined these ranges to avoid using a steadymagnitude for the transformations and to increase the diversity of the manipulatedROIs. Table 4.1 reveals, however, that for certain transformation types, includ-ing RGBShift, HueSaturationValue, and ElasticTransform, we do not consider arange and instead use the determined magnitude directly.The number of ROIs in the augmented images is related to the last four con-tinuous variables. Similar to the other continuous variables, we map the optimiza-tion algorithm’s outputs to the defined ranges, which is presented in Table 4.2. Itshould be noted that the ranges presented in Table 4.2 are determined based onthe case study used. If a broader range of ROI values is considered, additionaloptimization iterations are required to reach the optimal values.70Table 4.1: The search space of magnitude for each transformation type andthe impact ranges of minmag and magmax.Transformation type Min value Max value minmag magmaxBlur 3 7 (min, mag) (mag, max)MedianBlur 3 7 (min, mag) (mag, max)ISONoise 0.01 0.2 (min, mag) (mag, max)CLAHE 0 100 magDownscale 0.7 0.99 (min, mag) (mag, max)Emboss 0 1 (min, mag) (mag, max)RGBShift 0 20 magHueSaturationValue 0 30 magMultiplicativeNoise 0 0.3 (1-mag, 1+mag)RandomBrightnessContrast 0 0.2 magRandomToneCurve 0 0.4 magCoarseDropout 4 20 number:[mag] size:[mag/4]GridDistortion 0 0.3 (-mag, mag)ElasticTransform 0 5 magPerspective 0 0.2 (min, mag) (mag, max)PiecewiseAffine 0.01 0.05 (min, mag) (mag, max)Flip -Rotate 0 90 (-mag, mag)Scale 0.5 1.2 (min, mag) (mag, max)Shear 0 30 (-mag, mag)Table 4.2: The defined ranges for the number of ROIs in the augmented im-ages.ROI Min Maxnspike 10 30nlea f 10 60nstem 10 50nbackground 100 200Optimization algorithmThe introduced AutoOLA optimizes the defined policy structures using the Ge-netic Algorithm (GA) [151]. GA is one of the most widely used evolutionaryalgorithms for optimizing a variety of engineering problems [152–154].GA follows a straightforward procedure, as shown in Figure 4.1. The ini-71tial step is to generate a population at random. The subsequent generation iscreated based on the previous generation during the evolution loops. In the evo-lution loops, the crossover operator generates a portion of the next generation’sindividuals. Crossover accepts two parents and produces two offspring who in-herit the properties of the parents. There are numerous methods for crossover. Incontinuous problems, the arithmetic crossover, a subset of the uniform crossover[155, 156], can be used. The arithmetic crossover is expressed as follows:P1 = (p11, p12, · · · , p1n)P2 = (p21, p22, · · · , p2n)}→ O1i = αiP1i+(1−αi)P2iO2i = (1−αi)P1i+αiP2i(4.7)where P1 and P2 are the selected parents from the current generation, while O1and O2 are the offspring produced by the crossover operator. In Equation (4.7),α is a coefficient that specifies the offspring interval. Considering 0⩽ α ⩽ 1, theoffspring always fall between P1 and P2. So that, to increase the exploration of thecrossover operator, we defined −γ ⩽ α ⩽ 1+ γ , where γ is considered 0.1 in thecurrent study.After the crossover, an additional operator known as the mutation is applied tothe offspring. The purpose of mutation is to maintain genetic diversity from onegeneration to the next. Therefore, mutation enables the development of innovativeand revolutionary solutions. This procedure involves randomly altering the valuesof a subset of the offspring’s elements.Following by the crossover and mutation operations, the current populationand the offspring generated are merged and ranked from best to worst. Whenminimizing the cost function is the objective, the best individual is the one withthe lowest output. Finally, the first npop individuals from the sorted list are chosenas the next generation’s population, where npop is the number of individuals inthe population. The evolution loop depicted in Figure 4.1 concludes when thetermination criteria are met.During the AutoOLA optimization process, to evaluate each individual andreport the cost, it is necessary to run a model and compare the actual number of72234555 5 5 444 433 3 3222 291011 1010 10998109 9ReLUConv 3⨉3Batch normalizationBilinear upsamplingMax pool(a)(b)1 → 3 ⨉ 256 ⨉ 256 2 → 64 ⨉ 256 ⨉ 256 3 → 128 ⨉ 128 ⨉ 128 4 → 256 ⨉ 64 ⨉ 64 5 → 512 ⨉ 32 ⨉ 32 6 → 1024 ⨉ 16 ⨉ 16 7 → 1 ⨉ 256 ⨉ 256 8 → 3 ⨉ 128 ⨉ 128 9 → 4 ⨉ 128 ⨉ 128 10 → 8 ⨉ 64 ⨉ 64 11 → 16 ⨉ 32 ⨉ 32 12 → 1 ⨉ 128 ⨉ 128 1 7126Figure 4.5: (a) F-CNN and (b) C-CNN architectures.wheat spikes to the predicted number. To increase the speed of AutoOLA, a childConvolutional Neural Network (CNN) model is defined and implemented in thecost function, as described in Subsection 4.2.4. Architecture of the F-CNN andC-CNN Models.4.2.4 Architecture of the F-CNN and C-CNN ModelsThe presented model contains two distinct Child-Convolutional Neural Network(C-CNN) and F-CNN models. In accordance with Figure 4.1, ten object level aug-mentation images are generated for each generated individual during optimization.The object level augmentation images are then used as training samples for the C-CNN model. Finally, the trained C-CNN is used to model a fixed testing datasetto compute the examined individual’s cost function amount.To train the F-CNN, however, 200 object level augmentation images are gen-erated following the completion of the optimization phase using the optimum aug-73mentation policy. In the final step, the testing dataset is modeled using the trainedF-CNN, and its accuracy in counting the number of wheat spikes is returned.Figure 4.5 depicts the designed architectures for the F-CNN and C-CNN. Bothof the proposed models are designed based on UNet architecture Ronnebergeret al. [124]. According to Figure 4.5, UNet employs skip connections, which arealso employed in fully connected networks [125]. UNet consists of an encoder anda decoder, with the encoder responsible for extracting features and the decoder forprojecting discriminative features onto the pixel space.Two minor distinctions exist between the proposed DL models and the originalUNet:(a) While transposed convolution is used for upsampling in the decoder portionof the original UNet, we discovered that it increases the number of trainableparameters without improving the performance of the predictions. Accord-ing to Figure 4.5, we instead employed bilinear upsampling.(b) In the original UNet, the valid convolution is used. However, since it causessize variation of the output matrix, we used same convolution in F-CNN andC-CNN instead.In the F-CNN procedure, input images are first scaled to 512×512 pixels andthen cropped to 256× 256 pixels at a random location. However, in the C-CNNprocedure, the input images are first scaled to 256×256 pixels and then croppedto 128×128 pixels at a random location in order to reduce the number of trainingparameters.As the last point, the F-CNN model has around 34e6 trainable parameters,whereas the C-CNN model only has 8700 trainable parameters. As a result, the C-CNN model can be trained in a matter of seconds during the optimization process.In addition, the C-CNN training technique is ended if the prediction accuracy ofthe training samples does not improve after five consecutive epochs, while theF-CNN training is fixed at 2000 epochs.744.2.5 Evaluation MetricsThe proposed AutoOLA technique is evaluated and compared to previous aug-mentation techniques using three statistical error indices, including RMSE, MAE,and MAPE. A major benefit of RMSE and MAE is that they report the error onthe same scale as the output. MAPE, on the other hand, presents the error with anon-dimensional index. RMSE, MAE, and MAPE are defined as follows:RMSE =√√√√√ N∑i=1(GTi−Oi)2N(4.8)MAE =N∑i=1|GTi−Oi|N(4.9)MAPE =N∑i=1∣∣∣(GTi−Oi)/GTi∣∣∣N×100 (4.10)where N represents the number of samples, GT represents the ground truth num-ber of wheat spikes, O represents the predicted number of wheat spikes, and irepresents the sample index.4.3 Experimental Results and DiscussionThis study aims to develop an augmentation technique capable of improving pre-diction performance when fewer training samples are available. The purpose ofthis section is to compare the performance of the proposed AutoOLA algorithmto that of other available augmentation techniques by developing twelve distinctmodels.754.3.1 Dataset DescriptionWe utilized ETHZ-1, a sub-dataset of the Global Wheat Head Dataset (GWHD)dataset, to evaluate the proposed AutoOLA and compare it to the performance ofthe previous augmentation techniques. The GWHD dataset was initially releasedin 2020 [136] and revised in 2021 [137]. In this dataset, over 6,500 images havebeen collected from 16 institutions in twelve countries. More than 750 images areincluded in the ETHZ-1 sub-dataset, which was obtained by the ETH Zurich insti-tution in Switzerland. The GWHD dataset was published along with the boundingbox annotations. However, as stated in Subsection 4.2.1. Objects Extracting andAssembling, the images must be annotated with DA to count the number of wheatspikes in the presented modeling. In the DA mask, a dot is provided at the tipof each wheat spike. Therefore, we developed an OpenCV-based application toaid experts in converting available bounding boxes into dot annotations. Addi-tionally, the PA annotation used in AutoOLA was performed using an online toolcalled Make Sense [157].Zaji et al. [158] estimates that approximately 100 training images from theETHZ-1 sub-dataset will be sufficient to achieve appropriate results using a UNetDL model. Therefore, 100 images were selected and annotated for the trainingdataset. Additionally, 50 images were considered for the validation dataset, and 40images were considered for the testing dataset. The training dataset is used to trainthe DL models, the validation dataset is used to assess the performance of C-CNNmodels during the optimization procedure, and the testing dataset is used to assessthe performance of the F-CNN model.A few representative samples from GWHD dataset and ETHZ-1 sub-datasetare presented in Figure 4.6. As depicted in the figure, with numerous adjacentand occluded wheat spikes, predicting the accurate number of wheat spikes ischallenging for DL models. It should also be noted that the original size of theGWHD images is 1024× 1024 pixels, which are resized and cropped during theF-CNN and C-CNN modeling processes (Subsection 4.2.4. Architecture of the F-CNN and C-CNN Models).76(a) GWHD dataset (b) ETHZ-1 subdatasetFigure 4.6: Some examples of the (a) GWHD dataset and (b) ETHZ-1 sub-dataset.Lastly, we tested the proposed algorithms on the entire GWHD dataset and theETHZ-1 sub-dataset to assess their performance in solving real-world problems(Section 4.3. Experimental Results and Discussion).4.3.2 Design of the ModelsAs the first model, we used the entire training dataset and the basic augmentationtechnique that has been used in the vast majority of wheat spike localization andcounting research studies [37, 38, 147, 148]. This model, which is referred to asbasic-all, is used as a benchmark for other comparisons performed in the currentsection. The basic augmentation technique utilized in the basic-all model con-sists of hue\saturation value modification, shifting, scaling, vertical\horizontalflipping, and rotation.For the remaining models developed in this study, the number of training sam-77ples was reduced from 100 to one. So that, we are able to calculate and comparethe performance reduction of DL models that have been developed using differentaugmentation techniques.The second model developed in this section is the basic-one model. Thismodel employs the exact same augmentation technique as the basic-all model.The number of training samples, which is one for the basic-one model, is the onlydifference between basic-one and basic-all models.As described in Subsection 4.2.2. Augmentation Policy Structures, this re-search presents two distinct policy structures, 2aug and 5aug. Alternatively, af-ter determining the magnitude of each transformation type during the optimiza-tion procedure, the minmag and magmax impact ranges are applied per Sec-tion 4.2.3. Range of optimization variables. Based on the utilized policy structuresand impact ranges, we developed four distinct AutoOLA variants and four mod-els with the following names: 2aug-minmag, 5aug-minmag, 2aug-magmax, and5aug-magmax. Similar to the basic-one model, we used a single training sampleto train each AutoOLA-based DL model.To compare the performance of the developed AutoOLA models with previ-ous augmentation techniques, we developed six additional models using AugMix[159], AutoAugment [78], CutMix [100], Mixup [98], RandAugment [107], andRandom Erasing [94]. The AugMix, AutoAugment, RandAugment, and RandomErasing models were also trained using a single training sample in the same man-ner as the basic-one and AutoOLA-based models. However, CutMix and Mixuprequire at least two images to perform augmentation; therefore, a second sourceimage was used in order to create the augmented images.Figure 4.7 illustrates a few examples of augmented images obtained using theproposed AutoOLA variants and other augmentation techniques.4.3.3 Optimization ResultsAs mentioned in Subsection 4.2.3. Augmentation Policy Optimization, AutoOLAuses the GA evolutionary optimization algorithm to tune the number of image78elements and the augmentation policy. In order to reach the optimal amounts,200 iterations of GA are performed. To evaluate each individual during the opti-mization process, the C-CNN is trained on ten augmented images and tested onthe validation dataset. Figure 4.8 depicts the performance of the best individualin each iteration. While the graph demonstrates that the model accuracy is stillincreasing during the GA optimization, we kept the termination threshold at 200iterations as the objective is to find an AutoOLA variant that can be optimized asquickly as possible.4.3.4 Wheat Spikes PredictionFigure 4.9 illustrates the performance of the developed models using the RMSE,MAE, and MAPE metrics. In this figure, the evaluation metrics for the basic-all model are displayed in the upper right corner of each plot. In contrast, theevaluation metrics for the other models are displayed within the bars. This figurealso displays the performance reduction percentage of the models trained with asingle training sample compared to the basic-all model that was trained using all100 training samples. According to Figure 4.9, the basic-all model performs thebest compared to the other eleven models. As stated, the purpose of the basic-allmodel is to serve as a benchmark against which other developed models can becompared. Clearly, the superior model is the one whose performance decreasesthe least when 100 training samples are reduced to one.Using the same augmentation technique and decreasing the number of train-ing samples from 100 to one increases the prediction MAPE by 125 percent, asdepicted in Figure 4.9. In addition, it can be seen that using advanced augmen-tation techniques such as AugMix and Random Erasing has no appreciable effecton wheat spikes number prediction performance. From this graph, it is clear thatcounting the number of wheat spikes in an image is challenging and that the em-ployed DL model should have access to a diverse training dataset.Figure 4.9 demonstrates that the increment error decreases substantially whenAutoOLA-based augmentation techniques are utilized. In addition, comparing the79AutoOLA techniques reveals that employing the magmax impact range improvesthe performance of the models by enhancing the impact of the transformationsused. Moreover, it is evident that the 5aug policy structure performs considerablybetter than the 2aug policy structure. Consequently, the 5aug-magmax modeloutperforms all other used augmentation techniques, with only a 60 percent incre-ment in MAPE index compared to the basic-all model.Note that the results presented in Figure 4.9 are derived from training the F-CNN model with 200 augmented images. In comparison, the results in Figure 4.8are derived from training the C-CNN model with ten augmented images duringthe optimization procedure. It can be observed that the order of the AutoOLAmodels from the best model to the worst model is identical in Figures 4.8 and 4.9.The 5aug-magmax model performs the best, followed by 2aug-magmax, 5aug-minmag, and 2aug-minmag models. This similarity between the results of F-CNNand C-CNN indicates that despite having smaller number of trainable parametersthan F-CNN, C-CNN is reliable enough to appropriately rank the models.As stated in Subsection 4.2.4. Architecture of the F-CNN and C-CNN Models,the F-CNN training procedure involves resizing the input images from 1024×1024 pixels to 512× 512 pixels and then cropping them randomly to 256× 256pixels. Due to the random cropping, the performance of the DL model duringtesting can vary between runs. Therefore, each trained DL model was evalu-ated for 100 epochs, and the average performance of each model is shown inFigure 4.9. As there are 40 images in the testing dataset, the testing procedurepredicts 100× 40 = 4000 cropped images by evaluating the DL model for 100epochs. Using the explained 4000 predicted samples in the testing procedure,we plotted Figure 4.10 with the ground truth number of spikes (the sorted redline) and the predicted amounts (the green line). As expected, it can be seen thatthe predicted amounts closely match the actual amounts in the basic-all model.However, the basic-one model and the models based on the previous augmenta-tion techniques, such as AugMix and RandAugment, cannot accurately predict thenumber of wheat spikes in the samples. Instead, the outputs of those models are80almost the average amount of the ground truth. Compared to other augmentationtechniques, it is clear that AutoOLA considerably improves prediction accuracy.For instance, the outputs of 5aug-magmax closely match the ground truths, andthe results are substantially more precise than those of other augmentation-basedmodels.Figure 4.11 illustrates two examples of wheat spike localization for each of thetwelve models considered. As mentioned, random cropping of the input imageswas performed during the training and testing phases of the developed DL models.Consequently, using the generated results, we could not display the localizationof wheat spikes in the same cropped image for all of the considered models. So,to plot this figure, we altered our algorithm to conduct testing on a predeterminedcropping area. This diagram illustrates that the basic-one model cannot accuratelypredict the majority of wheat spikes in the image. In addition, too many wheatspikes are detected as false positives using previous augmentation methods. WithAutoOLA, however, the accuracy of localization is significantly enhanced.4.3.5 AutoOLA-Based Model in Real-World SituationAs described in Subsection 4.3.1. Dataset Description, we trained and evaluatedthe developed models on a subset of the ETHZ-1 sub-dataset. To assess the perfor-mance of the proposed AutoOLA augmentation technique for solving real-worldproblems, however, it is necessary to test the results on larger datasets. To accom-plish this, we conducted two additional experiments. In the first experiment, thetrained models were evaluated on the entire ETHZ-1 sub-dataset, and in the sec-ond experiment, the trained models were evaluated on the entire GWHD dataset. Itshould be noted that testing models with large datasets like GWHD require exten-sive computational resources. In light of this, not all developed models were eval-uated in these two experiments. Instead, we only considered the basic-all modelas the benchmark, the 5aug-magmax model as the best variant of AutoOLA, andthe basic-one model as one that is trained using a single training image and basicaugmentation.81The outcomes of these two experiments are shown in Figure 4.12. The evalua-tion metrics of the basic-all models are presented at the bottom left of each plot inthis figure. Moreover, the evaluation metrics of the basic-one and 5aug-magmaxmodels are displayed within the bars. Using the ETHZ-1 sub-dataset and thebasic-one model, as shown in Figure 4.12, the MAPE index increased by 91 per-cent. However, using the AutoOLA augmentation technique, the MAPE index in-creased to 27 percent. Similar outcomes were obtained using the GWHD dataset.Using the basic-one model, the MAPE index increased by 34 percent in this exper-iment. However, using the AutoOLA augmentation technique, the MAPE indexincreased to 23 percent.Our findings indicate that the newly-introduced AutoOLA augmentation tech-nique can significantly improve prediction results compared to previous augmen-tation techniques. While various studies have been conducted on the localizationand counting of wheat spikes, less attention has been paid to developing novel aug-mentation techniques to improve prediction performance and reduce data prepara-tion time. A powerful augmentation technique is advantageous because it reducesthe number of training samples required to achieve a desirable outcome. Addition-ally, it reduces the effort required to annotate thousands of tiny spikes in trainingdatasets.The AutoOLA algorithm proposed, which is specifically designed for the lo-calization and counting of wheat spikes, has two advantages. First, it augments each ROI inthe image separately, including spikes, leaves, stems, and backgrounds. Second,it designs the augmentation pipeline for each ROI in accordance with its opti-mal augmentation policy. Using the previous image augmentation techniques, asdepicted in Figure 4.7, a remarkable image augmentation is possible. The issuewith them, however, is that the transformations of different ROIs in the image areidentical. In fact, these methods perform image-level augmentation. AutoOLA,on the other hand, performs augmentation at the ROI level, resulting in diverseaugmented images.The other noteworthy finding is that an effective augmentation technique in82a particular case study may not be effective in another case study. As shown inFigure 4.9, the performance of AugMix is even worse than that of the basic aug-mentation technique.This study presents four variants of AutoOLA derived from two distinct aug-mentation policy structures, 2aug and 5aug, and two distinct impact ranges, min-mag and magmax. Comparing 2aug-minmag to 2aug-magmax and 5aug-minmagto 5aug-magmax demonstrates, as depicted in Figure 4.7, that using the magmaximpact range enhances the impact of each transformation on the input ROIs. Inaddition, according to Figure 4.3 policy, the following are the benefits of the 2augand 5aug policy structures:(a) 2aug has more sub-policies than 5aug. When there are more sub-policies,more augmenting pipelines are available to the input ROIs, which diversifiesthe augmented ROIs.(b) There are more transformations in each sub-policy of 5aug than in 2aug,which diversifies the augmented ROIs from the original ones.Although the presented augmentation technique is able to improve the perfor-mance of wheat spike prediction, this study has some limitations. Below is a listof limitations and suggestions for future research.(a) The purpose of this study was to evaluate and compare the performance ofthe proposed augmentation technique using fewer training samples. As aresult, we used a DL model that was trained using the entire training sam-ples as the benchmark (basic-all model) and calculated the performancereduction of models trained using only one training sample with variousaugmentation techniques. However, it would be interesting to compare thepresented augmentation techniques’ performance with the basic-all model’sperformance using the entire training dataset.(b) AutoOLA is explicitly designed for situations where a small number oftraining samples are desired. Utilizing fewer training samples is advan-83tageous in two cases. First, when we lack access to additional training sam-ples. Second, when reducing data preparation time is the objective. How-ever, applying the current version of AutoOLA to massive training samplesis challenging. In the first step of AutoOLA, different ROIs of the inputimage (spikes, leaves, stems, and backgrounds) must be annotated usingpolygons, which is time-consuming. To be able to apply AutoOLA to largedatasets, it is necessary to develop an instance segmentation technique thatautomatically annotates ROIs.(c) While the AutoOLA augmentation technique was developed specifically forthe localization and counting of wheat spikes, it may improve the perfor-mance of models when used to predict other crops, such as maize. There-fore, evaluating its performance in modeling other case studies is interest-ing.(d) The current version of AutoOLA implements the basic transformation typesshown in Figure 4.4. To increase the diversity of the augmented images.However, other augmentation techniques, such as AugMix and AutoAug-ment, can be used as one of the transformation types in AutoOLA.4.4 SummaryPrevious studies demonstrated that it is possible to accurately count wheat spikesusing computer vision and DL algorithms. However, the main drawback of previ-ously developed models is their labor-intensive and time-consuming data prepa-ration. Data preparation involves gathering the training images and annotatingthousands of small spikes within the collected dataset, which is prohibitivelychallenging in real-world problems. To address this issue, we proposed a novelaugmentation technique known as AutoOLA. In this algorithm, different imageobjects, including spikes, leaves, stems, and backgrounds, are decoupled and ma-nipulated independently. After that, the augmented images are generated by as-sembling the manipulated objects. In order to perform the augmentation on the84decoupled objects, the 5aug and 2aug policy structures were introduced. In addi-tion, to further diversify the augmented images, the minmag and magmax impactranges were presented. Then, an evolutionary optimization algorithm and a smallDL model called C-CNN were developed to optimize the type and magnitude ofthe transformations in the defined augmentation policy structures. The resultsdemonstrated that using AutoOLA considerably improves the counting perfor-mance of wheat spikes. In addition, it was concluded that the 5aug policy struc-ture combined with the magmax impact range is superior to other AutoOLA-basedaugmentation strategies and previous augmentation techniques.85(g) 2aug-minmag (h) 5aug-minmag (i) 2aug-magmax (j) 5aug-magmax(a) AugMix (b) AutoAugment (c) CutMix (d) Mixup (e) RandAugment(f) Random ErasingFigure 4.7: Some examples of the (a) AugMix, (b) AutoAugment, (c) Cut-Mix, (d) Mixup, (e) RandAugment, (f) Random Erasing, (g) 2aug-minmag, (h) 5aug-minmag, (i) 2aug-magmax, and (j) 5aug-magmaxbased augmented images.863.63.94.24.54.80 50 100 150 200RMSEGA iteration number5aug_magmax2aug_magmax5aug_minmag2aug_minmagFigure 4.8: Performance of the AutoOLA techniques during the optimiza-tion iterations.87100%107%85% 82% 78%96%90%83% 78%64% 59%020406080100120Increased RMSE (%)109% 114%85% 88% 84%100% 94%86%73%64%55%020406080100120Increased MAE(%)125%138%107% 108% 104%122%112%94%84%73%60%020406080100120140Basic-augAugMixAutoAugmentCutMixMixupRandAugmentRandom…2aug-minmag5aug-minmag2aug-magmax5aug-magmaxIncreased MAPE(%)Basic-all RMSE = 1.69Basic-all MAE = 1.34Basic-all MAPE = 8.62RMSE=3.39RMSE=3.5RMSE=3.13RMSE=3.08RMSE=3.01RMSE=3.32RMSE=3.21RMSE=3.09RMSE=3.01RMSE=2.78RMSE=2.69MAE=2.8MAE=2.87MAE=2.49MAE=2.52MAE=2.47MAE=2.69MAE=2.59MAE=2.5MAE=2.32MAE=2.19MAE=2.08MAPE=19.41MAPE=20.53MAPE=17.81MAPE=17.89MAPE=17.55MAPE=19.11MAPE=18.3MAPE=16.74MAPE=15.82MAPE=14.91MAPE=13.81Random Erasing(a)(b)(c)oneFigure 4.9: Reduced performance of the models according to (a) RMSE, (b)MAE, and (c) MAPE with only one training sample in comparison withthe basic-all model as the benchmark.88610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage numberBasic-all Basic-one 5aug-magmax2aug-magmax 5aug-minmag 2aug-minmag610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage number610141822260 1000 2000 3000 4000Number of spikesImage numberAugMix AutoAugment CutMixMixup RandAugment Random Erasing(a) (b) (c)(d) (e) (f)(g) (h) (i)(j) (k) (l)Figure 4.10: Comparing the ground truth number of spikes with the pre-dicted amounts for the testing samples of (a) basic-all, (b) basic-one,(c) 5aug-magmax, (d) 2aug-magmax, (e) 5aug-minmag, (f) 2aug-minmag, (g) AugMix, (h) AutoAugment, (i) CutMix, (j) Mixup, (k)RandAugment, and (l) Random Erasing based models.89Image5aug-minmag 2aug-magmax2aug-minmag 5aug-magmaxAugMix AutoAugment CutMix Mixup RandAugmentRandom ErasingGround truth Basic-all Basic-oneImage5aug-minmag 2aug-magmax2aug-minmag 5aug-magmaxAugMix AutoAugment CutMix Mixup RandAugmentRandom ErasingGround truth Basic-all Basic-one(a)(b)Figure 4.11: Some examples of the randomly cropped images, the groundtruth DA masks, and the predicted DA masks.9080%26%81%25%91%27%020406080100Basic-aug5aug-magmaxBasic-aug5aug-magmaxBasic-aug5aug-magmaxRMSE MAE MAPEIncreased error (%)19%9%28%17%34%23%010203040Basic-aug5aug-magmaxBasic-aug5aug-magmaxBasic-aug5aug-magmaxRMSE MAE MAPEIncreased error (%)MAE=3.32MAPE=14.41MAPE=21.62MAE=2.29RMSE=4.2RMSE=2.94MAE=5.1MAPE=65.15MAPE=70.92MAE=4.65RMSE=6.7RMSE=6.12(a) (b)Basic-all RMSE = 2.33Basic-all MAE = 1.84Basic-all MAPE = 11.34Basic-all RMSE = 5.62Basic-all MAE = 3.99Basic-all MAPE = 52.85Basic-oneBasic-oneBasic-oneBasic-oneBasic-oneBasic-oneFigure 4.12: Reduced performance of the basic-one and 5aug-magmax mod-els tested on (a) ETHZ-1 sub-dataset and (b) GWHD dataset in com-parison with the basic-all model as the benchmark.91Chapter 5Wheat Spikes Height Estimation Us-ing Stereo Cameras5.1 OverviewThere is a positive correlation between wheat plant height and lodging, yield, andbiomass. So, in precision agriculture, a high-throughput estimation of the wheatplant’s height in terms of its spikes is essential. This study aims to develop astraightforward, cost-effective method for measuring the height of wheat plantsusing stereovision cameras. To collect the required datasets, we conducted anexperiment in which we collected RGB images along with their depth layer usingtwo renowned stereovision cameras, OAKD and D455. Then, we used a DeepLearning (DL) model called Mask Region-based Convolutional Neural Networks(Mask R-CNN) to localize and distinguish the spikes in the collected images.In this study, we localized the wheat spikes using Object Detection (OD) andInstance Segmentation (IS) models. The advantage of the OD model over theIS model is that its bounding box annotation procedure in the data preparationphase is significantly faster than the IS model’s polygon annotation. However, thedisadvantage of OD is that there are many background pixels in each predictedbounding box, which reduces the performance of height estimation. To facilitate92the annotation process of the collected datasets, we also developed a hybrid Scale-Invariant Feature Transform (SIFT)-Random Forest (RF)-based active learningalgorithm to transfer the annotations of one camera to the other. The results showthat the OAKD camera performs better than the D455 camera for wheat plantheight estimation due to its higher RGB quality and better matching of the monocamera images. Using the OAKD camera and IS model, the algorithm proposed inthis study is able to estimate wheat height with Mean Absolute Percentage Error(MAPE) values of 0.75 and 0.67 percent at the spike and plot levels, respectively.5.2 Wheat Spikes Height Estimation MethodologyThe primary objective of this study is to estimate wheat height using overheadRGB-D imaging. Despite the fact that RGB-D images contain depth informa-tion, they cannot be used directly to measure the height of wheat plants for thefollowing reasons:1. As stated in Subsection 2.1.2. Wheat Height Estimation, wheat height isdefined as the distance between the soil and the wheat spikes. So as tolocalize the wheat spikes, computer vision and DL models must be used.2. The depth is measured using the disparity of stereo cameras. The inabilityto measure depth when both cameras cannot see the same points or whenthe used camera has poor matching capabilities is known as occlusion. Theocclusion also occurs when the lighting conditions cause the points to re-flect differently for each mono camera. In this instance, the stereo cameracannot locate the same points from each mono camera to calculate the dis-parity. The developed depth estimation model should consider the obscuredportions of the depth image.Using stereovision cameras and DL models, we developed an algorithm tomeasure the height of the wheat spikes in this section. In order to develop and eval-uate the presented models, a greenhouse experiment, which is described in Sub-section 5.2.1. Greenhouse Experiment, was conducted. In Subsection 5.2.2. Dataset93Figure 5.1: The greenhouse experiment’s setup.Annotation, we then discussed the annotation process for the stereovision cam-eras utilized in this study. Then, in Subsection 5.2.4. Wheat Spikes Localiza-tion, we described the DL model used to localize the wheat spikes. In Subsec-tion 5.2.5. Wheat Spikes Height Estimation, we then discussed the wheat heightestimation method using the developed DL model. Subsection 5.2.6. EvaluationMetrics introduces the statistical metrics that are used to evaluate the developedmodels.5.2.1 Greenhouse ExperimentIn order to train and evaluate the proposed algorithms, we conducted an experi-ment in a greenhouse. Since this experiment aimed to determine wheat’s height,only one wheat variety (red Spring wheat berries) and the same environmentalconditions were used.Figure 5.1 depicts the experimental setup for the greenhouse experiment. Inthis experiment, sixty pots were used to cultivate the wheat plants. During theimaging phase, we divided the wheat pots into fifteen zones, with four pots con-stituting each zone.94Table 5.1: The OAKD and D455 cameras’ technical specificationsSensor specifications OAKD D455RGB frame resolution 4056×3040 1280×800Depth output resolution 1280×800 1280×720RGB field of view 81°×68.8° 90°×65°Depth field of view 81°×71.8° 87°×58°Depth accuracy < 3% < 2%(b)(a)Figure 5.2: An example of RGB images produced by (a) OAKD and (b)D455 cameras.The imaging was performed with two different stereovision cameras, OAKDand D455, manufactured respectively by OpenCV and Intel. Both cameras uti-lized in the greenhouse experiment produce the overhead RGB and depth layers.Table 5.1 lists some of the most important technical specifications of the usedcameras.The images were taken in five series depicting various stages of growth. Ineach series, two images were captured from each zone using both cameras byrotating the pots by 90 degrees. Figure 5.2 depicts an example of the overheadRGB images captured by the employed cameras.5.2.2 Dataset AnnotationThe majority of studies in the field of wheat spike localization using computervision and DL employ OD or IS [160]. In the OD model, the location of thewheat spikes is specified by a bounding box. The localized objects are then fed95ok, pig••fᵗBg Aman SkamAs•_B← Gtfo*• BEBBTooooo←oÑBgo.B.GG→ABBOTBEBB(a) (b)Figure 5.3: Wheat spikes (a) bounding box and (b) polygon annotation.to a classifier for labeling. In contrast, in the IS model, each pixel of an image isassociated with a specific class. Unlike semantic segmentation, however, multipleobjects of the same class are treated as separate objects [161].According to our experiments, the training time for DL models using the ODand IS approaches is nearly identical. However, the most significant distinctionbetween these two approaches is their data preparation procedure. A DL modelmust have access to the appropriately labeled datasets to be trained and validated.Using the OD method, the images must be labeled by drawing bounding boxesaround the wheat spikes. In contrast, using IS procedure, images are labeled bydrawing a polygon around each spike. Figure 5.3 depicts the distinction betweenthe bounding box and polygon annotation.In the present study, both the OD and IS approaches are employed to local-ize the wheat spikes, and their performance is compared in estimating the wheatheight using the stereovision cameras. We begin by completely annotating theOAKD dataset. After that, we developed a procedure based on SIFT and activelearning algorithms to transfer the labels of the OAKD dataset to the D455 dataset.This method is described in the next subsection.96Labeling the wheat spikes using polygonsTransferring the polygons to bounding boxesOAKD datasetMatching the D455 images to OAKD images using SIFT algorithmModifying the D455 polygon labels using RF-based active learningD455 datasetStartEndTransferring the polygons to bounding boxesFigure 5.4: An overview of the annotation process for the D455 and OAKDdatasets.5.2.3 Transferring the OAKD labels to D455Annotation is one of the most time-consuming and labor-intensive aspects of DLmodeling. In the field of agriculture, where it is necessary to annotate thousandsof small objects in the images, this process is even more laborious. Figure 5.4provides an overview of the annotation method. According to this diagram, theannotation process begins with the polygon labeling of wheat images capturedwith an OAKD camera. As stated, the polygon annotation is utilized for IS mod-eling. Following this, the OAKD dataset’s drawn polygons are converted intobounding boxes for OD modeling. Instead of manually labeling the D455 dataset,we developed an algorithm to annotate it using the OAKD dataset in the next step.Figure 5.2 demonstrates that although the D455 and OAKD cameras are at thesame height level in the greenhouse experiment setup, the captured images havedifferent perspectives. This difference results from the cameras’ different fields ofview and slight differences in their angles toward the wheat plants. Therefore, it isimpossible to transfer the D455 annotations to OAKD without modification. In the97(c) (d)(a) (b)Figure 5.5: A SIFT transformation example; (a,b) the original OAKD andD455 images with the matched key points, (c) the matched OAKDimage within the D455 image, and (d) the transformed D455 imagebased on the OAKD image.first step of transferring the annotations, we matched the D455 and OAKD imagesusing the SIFT algorithm. SIFT, developed by Lowe [162], is a feature detectionalgorithm that identifies the image’s key points and uses them to match the imageto another with comparable key points. Notably, the SIFT key points are scale- androtation-invariant. Therefore, the primary advantage of SIFT over other featuredetection algorithms, such as Histograms of Oriented Gradients (HOG) [163], isthat the detected features are unaffected by the image’s size or orientation.Figure 5.5 depicts an instance of aligning D455 images with OAKD imagesusing SIFT. According to this figure, the key points of the OAKD image arematched with the key points of the D455 image in the first step. Then, basedon the matched key points, a transformation is applied to the D455 image to makeit as similar as possible to the D455 image.Figure 5.6 (a) and (b) depict an example of an OAKD image and a transformed98D455 image, respectively. It is important to note that these images are croppedat the same location. While transferring the D455 image using SIFT helps tomake the images similar, the scale, orientation, and position of the spikes are stillslightly different in each image, as depicted in the figure. Therefore, we couldnot transfer OAKD labels to D455 without losing accuracy. To reduce the timeand effort required to modify all D455 labels, we developed an active learningalgorithm based on RF [164–166] in this step.Figure 5.6 provides an overview of the developed active learning algorithm(c). This algorithm uses RF to predict four distinct actions for each wheat spike inthe OAKD labels repository and transfer them to the D455 labels repository. Thepredicted actions are “move up-down”, “move left-right”, “re-scale”, and “rotationangle”. After predicting the actions of each wheat spike label in this algorithm, thetransformed label is presented to an expert for final modifications and acceptanceof the actions. Accepting the predicted actions adds the modified label to theD455 labels repository and retrains the RF model. This active learning algorithmsignificantly reduces the time and effort required for label transformation, as theemployed machine learning model automatically performs the transformations.Figure 5.7 depicts the results of the developed RF model for predicting thenecessary actions to transfer annotations from OAKD images to D455 images.According to this figure, the developed active learning algorithm nearly convergedafter approximately one thousand label transfers.5.2.4 Wheat Spikes LocalizationAs stated, to estimate the wheat plants’ height, we should first identify the wheatspikes within the RGB images. To accomplish this, we utilized Mask R-CNN[167], one of the latest branches of the Region-based Convolutional Neural Net-works (R-CNN) [168] family. Mask R-CNN was selected because of its popular-ity and high performance in high-throughput phenotyping [169–172] and wheatplant modeling [173–177].Utilizing a selective search algorithm to scan the probable objects [178], exe-99(a) (b)OAK-D labelsPlot the transferred labelModify the transferred labelRe-train the RF modelUse trained RF to predict: 1. Move up-down2. Move left-right3. Re-scale4. Rotation angle Add the modified label to D455 repositoryD455 labels(c)Figure 5.6: An illustration of (a) a cropped OAKD image and (b) a croppedD455 image transformed by SIFT, and (c) an overview of the devel-oped active learning algorithm.051015200 1000 2000 3000 4000 5000 6000RMSELabelsUp-downRotateScaleRight-leftFigure 5.7: The performance of the developed active learning RF model inpredicting the transformations.100cuting a Convolutional Neural Network (CNN) on each of the proposed regions,and transferring CNN outputs to Support Vector Machine (SVM) and Linear Re-gression (LR) models are the three steps that comprise the R-CNN architecture.The SVM is used to classify the region of objects, while the LR is used to con-dense the bounding boxes of the objects. R-CNN demonstrated an improvement inOD. However, this algorithm’s training and testing procedure are time-consumingdue to the extraction of approximately 2000 region proposals per image. Fast R-CNN [179] was proposed as a solution to this issue. Instead of extracting regionproposals, this algorithm feeds the input image directly to a CNN. In addition,a softmax layer is substituted for the SVM to predict the class of each proposedregion. After this, it was suggested that Faster R-CNN [180] would improve theresults using anchor boxes. This algorithm detects an object in the input imageif the Intersection over Union (IoU) between the object and its corresponding an-chor box exceeds a threshold. While Faster R-CNN performed well in OD tasks,it was unable to determine which pixel of the input image represents an object.Thus, Mask R-CNN was proposed to incorporate IS into Faster R-CNN. In MaskR-CNN, OD and semantic segmentation are utilized to perform IS. Mask R-CNNoperated an Fully Convolutional Networks (FCN) [125] along with the Faster R-CNN algorithm for the semantic segmentation phase.We used standard Mask R-CNN to perform OD and IS on wheat images in thisstudy. In addition, we utilized ResNet-50 [41] as the backbone of Mask R-CNNto extract features. Note that the CNN backbone model used was pretrained onthe COCO dataset [181]. COCO is a large-scale dataset containing approximately330,000 images representing 80 distinct classes of objects. Using pretrained CNNmodels, Zaji et al. [147, 158] has demonstrated that it is possible to achieve goodperformance in small wheat plant phenotyping datasets.For training and evaluating the results using Mask R-CNN, 70 percent of thegreenhouse samples were used as training data and the remaining 30 percent astesting data.According to Subsection 5.2.3. Transferring the OAKD labels to D455, we101(b)(a)Figure 5.8: An example of cropped RGB images using (a) OAKD and (b)D455 cameras.rescaled the D455 images using SIFT to make them comparable to the OAKDimages. As shown in Figure 5.2 (a), however, the OAKD images contain a largenumber of unused pixels surrounding the wheat pots without any wheat spikes.Having these unused pixels could cause the DL model to become confused andincrease the running time. In order to feed the greenhouse images into the DLmodel, the unused pixels were cropped. To accomplish this, we cropped the im-ages so that approximately 98 percent of the spikes remained based on the positionof the spikes in the entire dataset. Figure 5.8 illustrates the cropped images.5.2.5 Wheat Spikes Height EstimationAs stated in Subsection 2.1.2. Wheat Height Estimation, the height of a wheatplant is defined as the distance between the soil surface and each spike. Afterpredicting the wheat spikes’ locations using RGB images and the DL model, it istime to estimate their height using the deep layer of the collected images.This study compares the performance of OD and IS in wheat height estima-tion. Therefore, the initial step in measuring the wheat spikes is to connect eachselected wheat spike in the RGB images to its corresponding wheat spike in thedepth layer, which can be performed easily using the predicted bounding boxesand polygons for the OD and IS models, respectively. The height of each wheatspike is then estimated by calculating the mean of all pixels within the predicted102(a) (b)Figure 5.9: OAKD depth layer (a) prior to correction and (b) subsequent tocorrection.bounding boxes or polygons.A few incorrect depth predictions were identified as one of the issues with thedepth cameras used. In some pixels of the depth layer, the depth is incorrectlypredicted to be up to 70,000 mm, according to Figure 5.9 (a). To solve this issue,we used the camera’s height (1,880 mm) as a criterion and set the value of pixelswith greater amounts to zero. In addition, as there are no wheat spikes closerthan 300 mm to the cameras, the quantity of all pixels with a value less than 300mm was set to zero. Figure 5.9 (b) depicts the corrected depth layer after themodifications described.In order to estimate depth, stereovision cameras take two separate images us-ing mono cameras and attempt to match similar points to calculate disparity. Incertain circumstances, however, stereovision cameras cannot match the similarpoints captured by mono cameras, which include the following:• When an occlusion prevents both mono cameras from seeing the same pointin the images.• When the object does not have a specific key point that can match the monocameras’ images (such as a flat white surface).• When the object is too close to the camera, and the distance between thematched points exceeds the acceptable range of the camera.103(a) (b)Figure 5.10: An illustration of (a) OAKD and (b) D455 RGB-D images.When any of the aforementioned issues occur, the OAKD and D455 camerasconvert the corresponding pixel values to zero. In the case of wheat plant depthestimation, occlusion is the primary reason why the depth layer contains zero pix-els. This issue is illustrated in Figure 5.10 for OAKD and D455 cameras. In thisfigure, the RGB layers are combined with the depth layer. There is a dark shadowsurrounding each of the wheat spikes and other objects in the presented images.These dark shadows are connected to the depth layer’s zero pixels. We realizedthat the dark shadows completely conceal the wheat spikes in some instances.Therefore, we are unable to determine the height of these wheat spikes. To solvethis problem, we developed an algorithm that estimates the height of occludedwheat spikes by calculating the average height of the three nearest wheat spikes.5.2.6 Evaluation MetricsSeveral evaluation metrics are applied to various sections of the current study. Inorder to evaluate the wheat spikes localization using OD and IS models, we usedTrue Positive (TP), False Positive (FP), True Negative (TN), False Negative (FN),recall, precision, and F1. The TP, FP, TN, FN are defined as follows:• TP indicates that the model correctly predicted the presence of a spike in aparticular location.104• FP indicates that the model incorrectly predicted the presence of a spike ina particular location.• TN indicates that the model correctly predicted the absence of a spike in aparticular location.• FN indicates that the model incorrectly predicted the absence of a spike ina particular location.Using the defined indices, other important evaluation metrics such as precisionand recall can be defined, as shown in the following equations:Precision =T PT P+FP(5.1)Recall =T PT P+FN(5.2)According to Equation (5.1),precision is the proportion of correctly predictedwheat spikes relative to the total number of predicted wheat spikes, and recall isthe proportion of correctly predicted wheat spikes relative to the total number ofground truth wheat spikes. To measure the balance between precision and recall,a new evaluation metric, F1, is defined, which is calculated using the harmonicmean of these two variables (Equation (5.3)).F1 = 2× Precision×RecallPrecision+Recall(5.3)Furthermore, in order to evaluate the generated active learning algorithm, aswell as the wheat spike height estimation, we used Mean Absolute Error (MAE),Root Mean Squared Error (RMSE), Median Absolute Error (MedAE), MAPE, andcoefficient of determination (R2), which are defined as follows:MAE =1NN∑i=1|gi− pi| (5.4)105RMSE =√1NN∑i=1(gi− pi)2 (5.5)MedAE = median(|g1− p1| , . . . , |gn− pn|) (5.6)MAPE =1NN∑i=1|gi− pi||gi| ×100 (5.7)R2 =N∑i=1(gi− g¯)(pi− p¯)√N∑i=1(gi− g¯)2N∑i=1(pi− p¯)22(5.8)where N is the number of samples, gi is the ground truth of i-th sample and piis its corresponding predicted value. The MAE and RMSE have the advantage ofmeasuring error in the same units as the target variable. In this study, the MedAE isreported because it is resistant to outliers, whereas the MAPE is reported becauseit is sensitive to relative errors and not the global scale of the target variable. R2is yet another measure of model fit; it is the proportion of the variance in thedependent variable.5.3 Experimental Results and DiscussionThis section presents the results of estimating wheat plant height using OD and IS.As mentioned in Section 5.2. Wheat Spikes Height Estimation Methodology, thedepth layer captured by OAKD and D455 cameras serves as the ground truth inthe present study. Therefore, the first step is to estimate the height prediction errorfor wheat spikes using the employed cameras. Due to the impossibility of man-ually measuring the height of approximately 10,000 wheat spikes and comparingthem to the estimated depth of the stereovision cameras used in this experiment,200 wheat spikes were sampled. According to Table 5.1, the OAKD and D455106Table 5.2: Wheat spikes localization performance based on IS and OD mod-els.Camera Type Dataset FN% TP% FP% Recall Precision F1OAKDIS Train 8.206 91.794 5.222 0.918 0.946 0.932IS Test 9.552 90.448 5.882 0.904 0.939 0.921OD Train 4.585 95.415 5.082 0.954 0.949 0.952OD Test 6.233 93.767 5.855 0.938 0.941 0.939D455IS Train 10.522 89.478 6.543 0.895 0.932 0.913IS Test 11.893 88.107 7.120 0.881 0.925 0.903OD Train 6.683 93.317 6.372 0.933 0.936 0.935OD Test 6.958 93.042 7.012 0.930 0.930 0.930cameras have a depth accuracy of less than three and two percent, respectively.According to the 200 manually measured samples in this experiment, the error indepth prediction was less than two percent. Therefore, we considered the mea-sured depth with stereovision cameras to be the ground truth.5.3.1 Results of Wheat Spikes LocalizationConsidering the accuracy of stereovision cameras, the biggest challenge is to accu-rately localize wheat spikes and solve the zero-pixel problem caused by the cam-eras’ inability to match similar points between images captured by mono cameras,as discussed in Subsection 5.2.5. Wheat Spikes Height Estimation.Table 5.2 presents the performance of IS and OD models for wheat spike lo-calization using the OAKD and D455 cameras. According to this table, the local-ization performance of the OD model is superior to the IS model for both OAKDand D455 images. In addition, the localization performance of IS and OD modelsusing OAKD images is superior to that of OAKD images. This table demon-strates that the wheat spikes localization performance of the testing dataset for theOAKD and D455 cameras and the IS and OD models are comparable to that ofthe training dataset, indicating that the employed DL models are not overfitted.The IoU between the ground truth polygon and bounding box annotations andthe IS and OD predictions, respectively, are depicted in Figure 5.11. The IoUis defined as the ratio of overlap area to union area between the ground truthannotations and the predictions. As expected, the IoU of OAKD-based models107Train set Test set0.60.650.70.750.80.850.90 10 20 30 40 50 60IoUImageOAKD - IS modelOAKD - OD modelD455 - IS modelD455 - OD model0.60.650.70.750.80.850.90 5 10 15 20 25 30IoUImageOAKD - IS modelOAKD - OD modelD455 - IS modelD455 - OD modelAvg=0.77Avg=0.76Avg=0.67Avg=0.69Avg=0.76Avg=0.73Avg=0.66Avg=0.65Figure 5.11: The IoU of the OAKD and D455-based IS and OD models.is higher than D455-based models, given that the performance of models usingOAKD images is superior to that of models using D455 images.5.3.2 Results of Wheat Spikes Height EstimationAlthough the ability to precisely locate wheat spikes is required to predict theirheight accurately (Table 5.2), this is not the primary objective of this study. Theprimary aim is to predict the height of wheat spikes accurately.Table 5.3 and Figure 5.12 illustrate the performance of the IS and OD modelsin predicting the height of wheat plants at two different levels of spike and plot. Inorder to estimate the height prediction performance, which is one of the primarygoals of precision agriculture [182], we considered each spike individually at thespikes level. On the other hand, at the plot level, we estimate the performanceof wheat height prediction in each individual plot. While the optimal plot size inwheat farming is approximately 5m×5m [183], we treated each image of wheatplants as a single 0.6m×0.6m plot in this study.It can be seen from Table 5.3 and Figure 5.12 that the performance of themodels in predicting the training and testing datasets is very similar. The results108Table 5.3: Wheat spikes height prediction performance.Predict Camera Type Dataset MAE (mm) MAPE RMSE (mm) MedAE (mm)SpikesOAKDIS Train 7.254 0.663 18.148 2.511IS Test 8.281 0.750 20.833 2.636OD Train 19.603 1.818 33.415 10.522OD Test 16.620 1.523 31.204 7.935D455IS Train 10.237 0.949 30.155 2.237IS Test 9.503 0.909 24.945 2.487OD Train 17.076 1.615 43.219 4.669OD Test 14.461 1.387 32.330 4.567PlotsOAKDIS Train 6.096 0.565 7.316 5.446IS Test 7.209 0.666 8.066 7.635OD Train 8.410 0.781 9.739 8.036OD Test 3.905 0.358 4.698 3.768D455IS Train 6.432 0.617 8.363 4.805IS Test 7.086 0.680 7.884 7.227OD Train 5.528 0.521 10.471 3.872OD Test 4.193 0.402 4.980 3.849also indicate that the performance of wheat height prediction at the plot level issuperior to that at the spikes level.One of the considerable conclusions that can be drawn from Table 5.3 and Fig-ure 5.12 is that the OAKD camera is superior to the D455 camera for predictingwheat height. As previously stated, when we examined the accuracy of the cam-eras using 200 sample spikes, the accuracy of both cameras exceeded 98 percent.Therefore, the difference in wheat height prediction performance between thesetwo cameras must be attributable to factors discussed at the end of this section.During the experiment in the greenhouse, the average height of each wheat plotwas measured manually. We compared the manually measured wheat height ineach plot to the wheat height obtained using the depth layer of the cameras andground truth polygon annotations in Figure 5.13. According to this figure, thedepth prediction performance of both the OAKD and D455 cameras is very closeto the wheat height manually measured. In certain circumstances, however, theresults of the D455 camera are considerably different from those of the measuredground truth and the OAKD camera.To investigate the cause of wheat height estimation outliers using the D455camera, we displayed an example of problematic images in Figure 5.14. Thecamera could not estimate the depth of most of the plot pixels, as depicted in109R² = 0.81670013001900700 1300 1900IS modelOD modelGT GT GT GTOAKD camera D455 cameraTrain set Test set Train set Test setSpikesR² = 0.93770013001900 R² = 0.913 R² = 0.844 R² = 0.857R² = 0.809700 1300 1900R² = 0.713700 1300 1900R² = 0.762700 1300 1900R² = 0.967102010801140 R² = 0.975 R² = 0.947 R² = 0.901IS modelOD modelPlotsR² = 0.9481020108011401020 1080 1140R² = 0.9681020 1080 1140R² = 0.9041020 1080 1140R² = 0.8891020 1080 1140Figure 5.12: Scatter plot of wheat height prediction performance in spikeand plot levels.this figure. As described in Subsection 5.2.5. Wheat Spikes Height Estimation,stereovision cameras must match the two mono cameras’ images to determinedisparity and depth. The inability of the D455 camera to measure the depth insome of the plots is due to its poor performance in matching images captured bymono cameras.As stated in Subsection 5.2.5. Wheat Spikes Height Estimation, when the dark110DepthPlotTrain set Test set100010501100115012000 20 40 60GT measuredGT OAKDGT D455100010501100115012000 10 20 30GT measuredGT OAKDGT D455PlotFigure 5.13: Comparing the ground truth depth estimation of OAKD andD455 cameras with manually measured ground truth.(a) (b)Figure 5.14: An example of a D455 plot with its problematic depth layer.shadow depicted in Figure 5.10 completely covers a wheat spike, the averageheight of the three nearest wheat spikes is determined and reported as the heightof the covered wheat spike. The percentage of covered wheat spikes in each of theutilized datasets is presented in Table 5.4. This table indicates that while the needto indirectly estimate the height of spikes is minimal in OAKD images, it is muchgreater in D455 images due to the inferior matching ability of the D455 cameracompared to the OAKD camera. In addition, as expected, since the covered areaof the polygon annotations is smaller than that of the bounding box annotations,this amount is higher for the IS models compared to the OD models.In the present study, two methods for measuring the height of wheat plants111Table 5.4: The percentage of wheat spikes whose height was indirectly esti-mated.Type Dataset OAKD D455IS Train 0.034 4.030Test 0.119 2.173OD Train 0.016 3.398Test 0.029 1.420using stereovision cameras are proposed. In the first model, we localized thewheat spikes using the bounding box annotation technique and the OD model. Incontrast, in the second model, we used the polygon annotation technique and theIS model. Due to the use of the same DL model for both the OD and IS models, thetraining time for both models is identical. However, each model has advantagesand disadvantages.The primary benefit of the OD model over the IS model is its simple datapreparation. According to our experience, it takes approximately 15 minutes todraw bounding boxes around the wheat spikes for each image in the dataset. Usingpolygons, it takes longer than one hour to fully annotate a single image. Accord-ingly, we only annotated the OAKD camera images and then transferred themto the D455 images using an active learning algorithm that is explained in Sub-section 5.2.2. Dataset Annotation and Subsection 5.2.3. Transferring the OAKDlabels to D455. Obviously, the aforementioned difference in annotation time canhave a significant impact on real-world modeling datasets containing thousandsof images.In contrast, the primary benefit of the IS model over the OD model is its pre-cision in depth estimation. According to Section 5.3. Experimental Results andDiscussion, the following factors contribute to the accuracy of height estimation:1. Accuracy of the used camera.2. Performance of the localization model.3. Type of the localization model.112In our experiment, the depth estimation accuracy of both cameras is greaterthan 98 percent. Thus, there is no significant difference between the first andsecond items. Due to the fact that we used the same model for IS and OD models,their localization performance is comparable, as shown in Table 5.2. This tabledemonstrates that the OD model’s localization performance is slightly superiorto the IS model. However, the third item, the type of localization model, hasthe greatest impact on the height estimation accuracy. Since we calculate theaverage of all pixels in the predicted region of interest, some pixels representingthe image’s background are included in the depth estimation when OD modelsare applied. This issue with the bounding box annotations is exacerbated whenthe model attempts to estimate the depth of curved spikes as the bounding boxexpands and contains more background pixels. This is the primary reason why,according to Table 5.3, the IS model’s height estimation accuracy is considerablyhigher than that of the OD model.The importance of estimating plot-level performance in addition to spike-levelperformance is an additional topic that needs to be addressed in this section. Ac-cording to Table 5.3 and Figure 5.12, we presented the estimation performanceof the models at both the spike and plot levels. This is due to the fact that whenestimating the wheat plant height in spike level, we take into account both the TPand FP predictions. However, we cannot consider the effect of TN and FN onheight estimation at the spike level. In order to resolve this issue, we also reportedthe plot level. It is evident that all TP, FP, TN, and FN are considered at the plotlevel. As shown in Table 5.3 and Figure 5.12, the plot level performance in thisexperiment is higher than the spike level performance, indicating a high amountof TN and a low amount of FN.The final topic that should be covered in this section is the differences in wheatheight prediction performance between the cameras used. According to Table 5.3and Figure 5.12, the performance of the OAKD-based models is significantly su-perior to that of the D455-based models. The explanations for this distinction areas follows:1131. The difference in the quality of RGB images: In the present study, we usedRGB layers as the DL model’s input to localize the wheat spikes. Addition-ally, the annotation quality and quantity of OAKD and D455-based datasetsare identical. Therefore, the difference in wheat spike localization perfor-mance between these two cameras (Table 5.2) is due to the difference in theRGB image quality. This distinction in RGB quality is discernible in Fig-ure 5.11. According to Figure 5.8, a decrease in wheat spikes localizationperformance results in a reduction of IoU, and according to Table 5.3 andFigure 5.12, a decrease in IoU results in a reduction of height estimationperformance.2. The difference in the matching abilities of the cameras: As discussed inSection 5.3. Experimental Results and Discussion, poor matching capabil-ities of the cameras result in larger dark shadows in the depth layers. Thisissue is depicted in Figure 5.14 and as anomalies in Figure 5.13. Further-more, it can result in higher spikes with indirect depth estimation, as shownin Table 5.4.This study presents a novel method for estimating wheat spikes using over-head stereovision cameras. While the stereovision cameras are inexpensive andstraightforward, they are not as accessible as the standard RGB cameras foundon almost all smartphones. The disadvantage of the methods presented in thisstudy is that they require a depth layer to measure the height of wheat plants. Theproposed future research suggestions aim to develop a method for estimating theheight of wheat plants using standard RGB images.1. Estimating the height of wheat using regular overhead RGB cameras: Thismodel’s procedure is very similar to what we introduced in this study. Dueto the absence of depth information in the RGB images, however, an encoder-decoder model such as the UNet architecture should be used to generatedepth maps from the RGB images. Clearly, in order to train the employed114DL model, we still require access to the RGB-D dataset. However, fol-lowing the training procedure, the breeder is no longer required to accessthe depth cameras, as the encoder-decoder model transfers the RGB imagesto the depth map [184]. Scaling the depth map is the only remaining is-sue when estimating wheat plant height using overhead RGB images. Thisalgorithm considers the distance between the RGB camera and the wheatplants in a field. Before estimating the wheat plant height, ArUco markers[185] can be used to scale the depth map to solve this problem.2. Estimating the height of wheat using regular side view RGB cameras: Forfuture research, we propose a method for estimating the height of wheatplants using side view RGB images. It is most convenient to measure theheight of wheat plants by capturing side view RGB images, as there is noneed to mount the camera on a device to capture images from above. Incontrast to the method proposed in this study, we do not use depth layerinformation to estimate the height of wheat. This algorithm’s procedure canbe described as follows:(a) Utilize the method proposed in this study to detect wheat spikes.(b) Utilize ArUco markers to perform the perspective transformation onthe image. Therefore, there is no need to capture side view imagesfrom a specific angle.(c) Utilize the ArUco markers to scale the image and determine groundlevel. Therefore, capturing the images at a specific distance from thewheat plants is unnecessary.(d) Measure the distance from the wheat spikes to the ground and reportthat as the plant height.1155.4 SummaryEstimating the height of the wheat plant is essential for identifying various growthphases and has a strong correlation with yield, lodging, and biomass. Numerousstudies have demonstrated the effectiveness of stereovision cameras for estimat-ing height. However, less emphasis has been placed on using this technology toestimate the wheat height at the spike level, which is crucial for precision agri-culture. This research seeks to develop an algorithm that uses the depth layer ofstereovision cameras to estimate the height of wheat spikes.A greenhouse experiment was conducted in the first step to collect the nec-essary data. In this experiment, the well-known OAKD and D455 stereovisioncameras were utilized. Then, a DL model was utilized to localize the wheat spikesin each image and to distinguish them from one another. To accomplish this, weemployed Mask R-CNN and created two distinct models of OD and IS.To prepare the collected datasets, the OAKD images were annotated withbounding boxes and polygons for use in the OD and IS models, respectively. Wealso developed a method for transferring annotations from OAKD to D455 imagesusing SIFT and an RF-based active learning algorithm.Although the ultimate goal of this study is not to localize the wheat spikes,this section is crucial for estimating the wheat height. The outcomes demonstratedthat the employed DL model could accurately localize OAKD and D455 images.Following the localization of the wheat spikes, the wheat height was estimatedusing the OD and IS models.In general, the proposed method accurately estimates the height of wheat. Thebenefit of using the OD model over the IS model is that bounding box annotationis significantly easier than polygon annotation. However, the results showed thatthe IS model is more accurate than the OD model. This is because bounding boxescontain a large number of background pixels, which hinders the performance ofthe OD model. In addition, the results demonstrated that the OAKD camera per-forms more accurately in comparison with the D455 camera. OAKD’s superiorperformance is a result of its greater accuracy in RGB imaging and fewer zero116pixels in the depth layer.117Chapter 6Conclusions, Contributions and Rec-ommendations for Future Research6.1 ConclusionsThis thesis aimed to investigate and develop strategies to minimize the time andeffort required for data preparation in the application of Deep Learning (DL) mod-els for high-throughput plant phenotyping. Through the proposed research, weaimed to investigate the use of more accessible sensors, such as low-cost cameras,for image acquisition, as well as to develop more efficient and automated methodsfor annotation and preprocessing. Additionally, we proposed an augmentation al-gorithm that can reduce the number of training samples needed for accurate DLmodels. Furthermore, this research explored the use of transfer learning and othertechniques to reduce the amount of labeled data required for training DL models.In the first goal, we developed a state-of-the-art framework for localizingand counting wheat spikes using dotted annotation datasets in conjunction withGaussian and constant density map generation algorithms. We also developedhybrid UNet architectures as the computational component, including VGG16-UNet, ResNet34-UNet, ResNet50-UNet, and ResNeXt-UNet. Furthermore, weimproved the performance of wheat counting by employing a hybrid density map118estimation-non-maximal supervision algorithm. The results from the ACID datasetdemonstrate a significant improvement in spike localization and counting com-pared to previous research studies.In the second goal, we proposed a novel augmentation algorithm, AutomaticObject Level Augmentation (AutoOLA), that can considerably reduce the re-quired number of training samples in DL models. The results demonstrated thatthe proposed AutoOLA technique could improve wheat spike counting perfor-mance by up to 60 percent. With only one trainable image, the employed hybridAutoOLA-DL model can predict the number of wheat spikes in the field with highaccuracy.In the third goal, we aimed to develop a straightforward, cost-effective methodfor measuring the height of wheat plants using stereovision cameras. Our pro-posed method uses a stereo camera system to capture images of wheat plants, andthen employs a DL model to estimate the height of the wheat plants.Overall, the proposed research aimed to improve the data preparation processfor high-throughput plant phenotyping, which can be a major bottleneck for thewidespread adoption of these techniques. By reducing the data preparation timeand effort, this research aims to increase the accessibility and scalability of thesemodels for researchers and practitioners in the field of plant phenotyping.6.2 ContributionsThis study’s primary contribution is making high-throughput wheat phenotypingmore applicable and simple to implement in real-world settings. To accomplishthis, we created precise and reliable DL-based models that can predict the wheatphenotype using dot annotation, which is probably the most straight forward an-notation technique. In addition, we used transfer learning, a developed activelearning algorithm, and an introduced augmentation technique that were designedspecifically for wheat phenotyping in order to reduce the time-consuming andcostly data preparation portion of high-throughput modeling. In the area of wheatplant height estimation, we also sought to develop algorithms that can measure119wheat height using sensors that are readily available and affordable. UtilizingRGB-D imaging from above was the primary focus of wheat height estimation.The following are the principal contributions of this study:• Using dot annotations and a pretrained DL algorithm, a method for locatingand counting wheat spikes is presented. Using the combination of DensityMap Estimation (DME) and Non-Maximal Suppression (NMS), this modelcan accurately count the wheat spikes with fewer training samples.• A novel augmentation algorithm designed specifically for wheat plant phe-notyping is presented. Using the object-level augmentation concept in con-junction with an optimization algorithm, the proposed method can train DLmodels with a high degree of accuracy using only a single training sample.• A novel model is presented for estimating the height of wheat plants usingRGB-D and RGB cameras. The proposed models can estimate plant heightwith less effort and inexpensive equipment. A greenhouse experiment wasperformed to assess the presented algorithms. In addition, an active learningalgorithm is proposed to transfer the annotations of one depth camera tothe other one. The human annotator is placed in an intelligent loop in thisalgorithm to annotate the most informative parts of the training images.6.3 Future StudiesThe limitations concerning this study and the direction for corresponding futurestudies are identified and discussed as follows:• The AutoOLA algorithm has been developed specifically for situations wheretraining samples are scarce. So that the time-consuming data preparationcomponent of this algorithm, which requires Polygon Annotation (PA) andDot Annotation (DA) labels to be combined, was not a significant issue. Ifthe objective is to apply this augmentation technique to a massive dataset,120the data preparation process must be optimized. A trained DL model shouldbe incorporated into the algorithm for future research in order to performautomatic PA labeling. Additionally, an active learning algorithm can beincorporated into the model to reduce DA labeling time.• The dataset used for training and evaluating the wheat plant height estima-tion was collected during the greenhouse experiment. In a future study, theused cameras should be mounted on a Unmanned Aerial Vehicle (UAV) inorder to evaluate the performance of the proposed models in estimating theheight of wheat plants on a large scale.In summary, the future studies have mapped out potential avenues for researchin the field of wheat plant phenotyping. The ideas discussed serve as a springboardfor additional research and discovery. It is hoped that future research will buildupon and contribute to the work presented here. With the continued developmentof technology and the introduction of novel research techniques, the potential forfuture studies is limitless. It is an exciting time to be a part of this field, and thefuture promises many more breakthroughs and discoveries.121Bibliography[1] United Nations. World population prospects 2019: highlights. Departmentof Economic and Social Affairs, Population Division, 2019. → page 1[2] David Tilman, Christian Balzer, Jason Hill, and Belinda L Befort. Globalfood demand and the sustainable intensification of agriculture.Proceedings of the national academy of sciences, 108(50):20260–20264,2011. → page 1[3] Deepak K Ray, Nathaniel D Mueller, Paul C West, and Jonathan A Foley.Yield trends are insufficient to double global crop production by 2050.PloS one, 8(6):e66428, 2013. → page 1[4] Ali Moghimi, Ce Yang, and James A Anderson. Aerial hyperspectralimagery and deep neural networks for high-throughput yield phenotypingin wheat. Computers and Electronics in Agriculture, 172:105299, 2020.→ pages 1, 2, 3[5] Jared Crain, Suchismita Mondal, Jessica Rutkoski, Ravi P Singh, andJesse Poland. Combining high-throughput phenotyping and genomicinformation to increase prediction and selection accuracy in wheatbreeding. The plant genome, 11(1):170043, 2018. → page 1[6] Avinash C Tyagi. Towards a second green revolution. Irrigation andDrainage, 4(65):388–389, 2016. → page 1[7] Abriti Paul, Sourav Ghosh, Amit Kumar Das, Saptarsi Goswami, Sruti DasChoudhury, and Soumya Sen. A review on agricultural advancementbased on computer vision and machine learning. In Emerging technologyin modelling and graphics, pages 567–581. Springer, 2020. → page 1122[8] Yu Jiang and Changying Li. Convolutional neural networks forimage-based high-throughput plant phenotyping: a review. PlantPhenomics, 2020, 2020. → page 2[9] New YorkSmith A Wiley, B Cullis, and A Gilmour. The analysis of cropvariety evaluation data in australia. Aust NZ J Stat, 43:129145Smith,2001. → page 2[10] Keith H Coble, Ashok K Mishra, Shannon Ferrell, and Terry Griffin. Bigdata in agriculture: A challenge for the future. Applied EconomicPerspectives and Policy, 40(1):79–96, 2018. → page 2[11] Andreas Kamilaris and Francesc X Prenafeta-Boldu. Deep learning inagriculture: A survey. Computers and electronics in agriculture, 147:70–90, 2018. → page 2[12] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh,Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, MichaelBernstein, et al. Imagenet large scale visual recognition challenge.International journal of computer vision, 115(3):211–252, 2015. → pages2, 16[13] Jerome Treboux and Dominique Genoud. Improved machine learningmethodology for high precision agriculture. In 2018 Global Internet ofThings Summit (GIoTS), pages 1–6. IEEE, 2018. → page 3[14] Arti Singh, Baskar Ganapathysubramanian, Asheesh Kumar Singh, andSoumik Sarkar. Machine learning for high-throughput stress phenotypingin plants. Trends in plant science, 21(2):110–124, 2016. → page 3[15] F Kurtulmus and H Unal. Discriminating rapeseed varieties usingcomputer vision and machine learning. Expert Systems with Applications,42(4):1880–1891, 2015. → page 3[16] Angel Garcia-Pedrero, Consuelo Gonzalo-Martin, and M Lillo-Saavedra.A machine learning approach for agricultural parcel delineation throughagglomerative segmentation. International journal of remote sensing, 38(7):1809–1819, 2017. → page 3[17] Hongyu Chu, Depei Zhang, Yanhua Shao, Zhiyuan Chang, Yuying Guo,and Ningning Zhang. Using hog descriptors and uav for crop pest123monitoring. In 2018 Chinese Automation Congress (CAC), pages1516–1519. IEEE, 2018. → page 3[18] Hyun K Suh, Jan Willem Hofstee, Joris IJsselmuiden, and Eldert J vanHenten. Sugar beet and volunteer potato classification usingbag-of-visual-words model, scale-invariant feature transform, or speededup robust feature descriptors and crop row information. BiosystemsEngineering, 166:210–226, 2018. → page 3[19] Radhika Kamath, Mamatha Balachanra, and Srikanth Prabhu. Paddy cropand weed classification using color features for computer vision basedprecision agriculture. International Journal of Engineering andTechnology (UAE), 7(4):2909–2916, 2018. → page 3[20] Mehmet Dogan Altinbas and Tacha Serif. Detecting defected crops:Precision agriculture using haar classifiers and uav. In InternationalConference on Mobile Web and Intelligent Information Systems, pages27–40. Springer, 2019. → page 3[21] JGPW Clevers, GWAM Van der Heijden, S Verzakov, and Michael ESchaepman. Estimating grassland biomass using svm band shaving ofhyperspectral data. Photogrammetric Engineering & Remote Sensing, 73(10):1141–1148, 2007. → page 3[22] Camilo Pulido, Leonardo Solaque, and Nelson Velasco. Weed recognitionby svm texture feature classification in outdoor vegetable crop images.Ingenieria e Investigacion, 37(1):68–74, 2017. → page 3[23] Steven W Chen, Shreyas S Shivakumar, Sandeep Dcunha, JnaneshwarDas, Edidiong Okon, Chao Qu, Camillo J Taylor, and Vijay Kumar.Counting apples and oranges with deep learning: A data-driven approach.IEEE Robotics and Automation Letters, 2(2):781–788, 2017. → page 3[24] Sue Han Lee, Chee Seng Chan, Paul Wilkin, and Paolo Remagnino.Deep-plant: Plant identification with convolutional neural networks. In2015 IEEE international conference on image processing (ICIP), pages452–456. IEEE, 2015. → page 3[25] Kentaro Kuwata and Ryosuke Shibasaki. Estimating crop yields with deeplearning and remotely sensed data. In 2015 IEEE International124Geoscience and Remote Sensing Symposium (IGARSS), pages 858–861.IEEE, 2015. → page 3[26] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436–444, 2015. → page 3[27] Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images,speech, and time series. The handbook of brain theory and neuralnetworks, 3361(10):1995, 1995. → page 4[28] Sarah Webb. Deep learning for biology. Nature, 554(7693), 2018. → page4[29] Mohammed Al-Zeyadi, Javier Andreu-Perez, Hani Hagras, Chris Royce,Darren Smith, Piotr Rzonsowski, and Ali Malik. Deep learning towardsintelligent vehicle fault diagnosis. In 2020 International Joint Conferenceon Neural Networks (IJCNN), pages 1–7. IEEE, 2020. → page 4[30] Martin Popel, Marketa Tomkova, Jakub Tomek, Lukasz Kaiser, JakobUszkoreit, Ondrej Bojar, and Zdenek Zabokrtsky. Transforming machinetranslation: a deep learning system reaches news translation qualitycomparable to human professionals. Nature communications, 11(1):1–15,2020. → page 4[31] Muhammed Sit, Bekir Z Demiray, Zhongrun Xiang, Gregory J Ewing,Yusuf Sermet, and Ibrahim Demir. A comprehensive review of deeplearning applications in hydrology and water resources. Water Science andTechnology, 2020. → page 4[32] AJD Pask, Julian Pietragalla, DM Mullan, and MP Reynolds.Physiological breeding II: a field guide to wheat phenotyping. Cimmyt,2012. → page 11[33] Ariel Ferrante, Julia Cartelle, Roxana Savin, and Gustavo A Slafer. Yielddetermination, interplay between major components and yield stability ina traditional and a contemporary wheat across a wide range ofenvironments. Field Crops Research, 203:114–127, 2017. → page 11[34] Carlos Arteta, Victor Lempitsky, and Andrew Zisserman. Counting in thewild. In European conference on computer vision, pages 483–498.Springer, 2016. → page 11125[35] Daniel Onoro-Rubio and Roberto J Lopez-Sastre. Towardsperspective-free object counting with deep learning. In EuropeanConference on Computer Vision, pages 615–629. Springer, 2016. → page11[36] Michael P Pound, Jonathan A Atkinson, Darren M Wells, Tony PPridmore, and Andrew P French. Deep learning for multi-task plantphenotyping. In Proceedings of the IEEE International Conference onComputer Vision Workshops, pages 2055–2063, 2017. → pages11, 12, 29, 35, 38, 41, 44, 50, 58, 59[37] Gensheng Hu, Lidong Qian, Dong Liang, and Mingzhu Wan.Self-adversarial training and attention for multi-task wheat phenotyping.Applied Engineering in Agriculture, 35(6):1009–1014, 2019. → pages11, 29, 30, 36, 41, 44, 58, 77[38] Tahani Alkhudaydi, Ji Zhou, et al. Spikeletfcn: Counting spikelets frominfield wheat crop images using fully convolutional networks. InInternational Conference on Artificial Intelligence and Soft Computing,pages 3–13. Springer, 2019. → pages 11, 12, 13, 30, 77[39] Adrian Bulat and Georgios Tzimiropoulos. Human pose estimation viaconvolutional part heatmap regression. In European Conference onComputer Vision, pages 717–732. Springer, 2016. → page 11[40] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networksfor human pose estimation. In European conference on computer vision,pages 483–499. Springer, 2016. → page 11[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residuallearning for image recognition. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 770–778, 2016. → pages11, 27, 101[42] Ji Zhou, Daniel Reynolds, Danny Websdale, Thomas Le Cornu, OscarGonzalez-Navarro, Clare Lister, Simon Orford, Stephen Laycock, GrahamFinlayson, Tim Stitt, et al. Cropquant: an automated and scalable fieldphenotyping platform for crop monitoring and trait measurements tofacilitate breeding and digital agriculture. BioRxiv, page 161547, 2017. →page 12126[43] Juncheng Ma, Yunxia Li, Keming Du, Feixiang Zheng, Lingxian Zhang,Zhihong Gong, and Weihua Jiao. Segmenting ears of winter wheat atflowering stage using digital images and deep learning. Computers andElectronics in Agriculture, 168:105159, 2020. → pages 13, 30, 42, 57[44] Ming-Yu Liu, Oncel Tuzel, Srikumar Ramalingam, and Rama Chellappa.Entropy rate superpixel segmentation. In CVPR 2011, pages 2097–2104.IEEE, 2011. → page 13[45] Sambuddha Ghosal, David Blystone, Asheesh K Singh, BaskarGanapathysubramanian, Arti Singh, and Soumik Sarkar. An explainabledeep machine vision framework for plant stress phenotyping. Proceedingsof the National Academy of Sciences, 115(18):4613–4618, 2018. → page13[46] Juncheng Ma, Keming Du, Lingxian Zhang, Feixiang Zheng, JinxiangChu, and Zhongfu Sun. A segmentation method for greenhouse vegetablefoliar disease spots images using color information and region growing.Computers and Electronics in Agriculture, 142:110–117, 2017. → page 13[47] Jose A Fernandez-Gallego, Shawn C Kefauver, Nieves Aparicio Gutierrez,Maria Teresa Nieto-Taladriz, and Jose Luis Araus. Wheat ear countingin-field conditions: high throughput and low-cost approach using rgbimages. Plant Methods, 14(1):22, 2018. → page 13[48] Tanuj Misra, Alka Arora, Sudeep Marwaha, Viswanathan Chinnusamy,Atmakuri Ramakrishna Rao, Rajni Jain, Rabi Narayan Sahoo, MrinmoyRay, Sudhir Kumar, Dhandapani Raju, et al. Spikesegnet-a deep learningapproach utilizing encoder-decoder network with hourglass for spikesegmentation and counting in wheat plant from visual imaging. Plantmethods, 16(1):1–20, 2020. → pages 13, 30, 57[49] Anand Asundi and Zhou Wensen. Fast phase-unwrapping algorithm basedon a gray-scale mask and flood fill. Applied optics, 37(23):5416–5420,1998. → page 14[50] Madhav Bhatta, Kent M Eskridge, Devin J Rose, Dipak K Santra,P Stephen Baenziger, and Teshome Regassa. Seeding rate, genotype, andtopdressed nitrogen effects on yield and agronomic characteristics ofwinter wheat. Crop Science, 57(2):951–963, 2017. → page 14127[51] A Navabi, Muzammal Iqbal, K Strenzke, and D Spaner. The relationshipbetween lodging and plant height in a diverse wheat population. CanadianJournal of Plant Science, 86(3):723–726, 2006. → page 14[52] Shuang-Lin Mao, Yu-Ming Wei, Wenguang Cao, Xiu-Jin Lan, Ma Yu,Zheng-Mao Chen, Guo-Yue Chen, and You-Liang Zheng. Confirmation ofthe relationship between plant height and fusarium head blight resistancein wheat (triticum aestivum l.) by qtl meta-analysis. Euphytica, 174(3):343–356, 2010. → page 14[53] Michael Schirrmann, Andre Hamdorf, Andreas Garz, Anton Ustyuzhanin,and Karl-Heinz Dammer. Estimating wheat biomass by combining imageclustering with crop height. Computers and Electronics in Agriculture,121:374–384, 2016. → page 14[54] Christine D Zanke, Jie Ling, Jorg Plieske, Sonja Kollers, Erhard Ebmeyer,Viktor Korzun, Odile Argillier, Gunther Stiewe, Maike Hinze, KerstinNeumann, et al. Whole genome association mapping of plant height inwinter wheat (triticum aestivum l.). PloS one, 9(11):e113287, 2014. →page 14[55] Xu Wang, Daljit Singh, Sandeep Marla, Geoffrey Morris, and JessePoland. Field-based high-throughput phenotyping of plant height insorghum using different sensing technologies. Plant Methods, 14(1):1–16,2018. → pages 14, 15[56] Wenan Yuan, Jiating Li, Madhav Bhatta, Yeyin Shi, P Stephen Baenziger,and Yufeng Ge. Wheat height estimation using lidar in comparison toultrasonic sensor and uas. Sensors, 18(11):3731, 2018. → page 14[57] Yishan Ji, Zhen Chen, Qian Cheng, Rong Liu, Mengwei Li, Xin Yan,Guan Li, Dong Wang, Li Fu, Yu Ma, et al. Estimation of plant height andyield based on uav imagery in faba bean (vicia faba l.). Plant Methods, 18(1):1–13, 2022. → page 14[58] N Budak, PS Baenziger, and KM Eskridge. Effect of replications onmeasuring wheat plant height. Canadian journal of plant science, 75(1):171–173, 1995. → pages 14, 16[59] S Gilbert Rozario and V Vasanthi. Ultrasonic sensor-based canopy heightmeasurement and root depth estimation. In Proceedings of Third128International Conference on Communication, Computing and ElectronicsSystems, pages 1083–1091. Springer, 2022. → page 15[60] Yin Bao, Lie Tang, Srikant Srinivasan, and Patrick S Schnable.Field-based architectural traits characterisation of maize plant usingtime-of-flight 3d imaging. Biosystems Engineering, 178:86–101, 2019. →page 15[61] Ramviyas Parasuraman. Mobility enhancement for elderly. arXiv preprintarXiv:1410.5600, 2014. → page 15[62] Zhuang Zhuo, Cheng Zhu, Chao-Sheng Tang, Hao Xu, Xiangchao Shi,and Vail Mark. 3d characterization of desiccation cracking in clayey soilsusing a structured light scanner. Engineering Geology, 299:106566, 2022.→ page 15[63] Pocholo Miguel De Lara, Trisha Marie Morado, and Rosalie Reyes.Estimation of net absolute sea level change in mogpog, marinduque usingpersistent scatterer interferometric synthetic aperture radar-derivedvertical land motion and tide gauge measurements. PhilippineEngineering Journal, 43(1), 2022. → page 15[64] Xiaowen Jiang, Valerio Cambareri, Gianluca Agresti, Cynthia IfeyinwaUgwu, Adriano Simonetto, Fabien Cardinaux, and Pietro Zanuttigh. Alow memory footprint quantized neural network for depth completion ofvery sparse time-of-flight depth maps. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, pages2687–2696, 2022. → page 15[65] Yu Jiang, Changying Li, Andrew H Paterson, Shangpeng Sun, Rui Xu,and Jon Robertson. Quantitative analysis of cotton canopy size in fieldconditions using a consumer-grade rgb-d camera. Frontiers in plantscience, 8:2233, 2018. → page 15[66] Adar Vit and Guy Shani. Comparing rgb-d sensors for close range outdooragricultural phenotyping. Sensors, 18(12):4413, 2018. → page 15[67] Victor P Rueda-Ayala, Jose M Pena, Mats Hoglind, Jose MBengochea-Guevara, and Dionisio Andujar. Comparing uav-basedtechnologies and rgb-d reconstruction methods for plant height andbiomass monitoring on grass ley. Sensors, 19(3):535, 2019. → page 15129[68] Ke Xu, Jingchao Zhang, Huaimin Li, Weixing Cao, Yan Zhu, XiaopingJiang, and Jun Ni. Spectrum-and rgb-d-based image fusion for theprediction of nitrogen accumulation in wheat. Remote Sensing, 12(24):4040, 2020. → page 15[69] Wenjian Liu, Yanjie Li, Jun Liu, and Jingmin Jiang. Estimation of plantheight and aboveground biomass of toona sinensis under drought stressusing rgb-d imaging. Forests, 12(12):1747, 2021. → page 15[70] Amita Mohan, William F Schillinger, and Kulvinder S Gill. Wheatseedling emergence from deep planting depths and its relationship withcoleoptile length. PLoS One, 8(9):e73314, 2013. → page 16[71] GJ Rebetzke, DG Bonnett, and MP Reynolds. Awns reduce grain numberto increase grain size and harvestable yield in irrigated and rainfed springwheat. Journal of Experimental Botany, 67(9):2573–2586, 2016.[72] Jinhai Cai, Pankaj Kumar, Joshua Chopin, and Stanley J Miklavcic.Land-based crop phenotyping by image analysis: Accurate estimation ofcanopy height distributions using stereo images. PloS one, 13(5):e0196671, 2018.[73] Abdeljalil El Habti, Delphine Fleury, Nathaniel Jewell, Trevor Garnett,and Penny J Tricker. Tolerance of combined drought and heat stress isassociated with transpiration maintenance and water soluble carbohydratesin wheat grains. Frontiers in plant science, 11:568693, 2020.[74] Leonardo Volpato, Francisco Pinto, Lorena Gonzalez-Perez,Iyotirindranath Gilberto Thompson, Aluizio Borem, Matthew Reynolds,Bruno Gerard, Gemma Molero, and Francelino Augusto Rodrigues Jr.High throughput field phenotyping for plant height using uav-based rgbimagery in wheat breeding lines: feasibility and validation. Frontiers inplant science, 12:591587, 2021. → page 16[75] Avital Bechar. Agricultural robotics for precision agriculture tasks:concepts and principles. Innovation in Agricultural Robotics for PrecisionAgriculture, pages 17–30, 2021. → page 16[76] Nour Eldeen Khalifa, Mohamed Loey, and Seyedali Mirjalili. Acomprehensive survey of recent trends in deep learning for digital images130augmentation. Artificial Intelligence Review, pages 1–27, 2021. → page17[77] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenetclassification with deep convolutional neural networks. Communicationsof the ACM, 60(6):84–90, 2017. → page 17[78] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, andQuoc V Le. Autoaugment: Learning augmentation strategies from data. InProceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 113–123, 2019. → pages 17, 18, 66, 78[79] Patrice Y Simard, David Steinkraus, John C Platt, et al. Best practices forconvolutional neural networks applied to visual document analysis. InIcdar, volume 3, 2003. → page 17[80] Dan Ciregan, Ueli Meier, and Jurgen Schmidhuber. Multi-column deepneural networks for image classification. In 2012 IEEE conference oncomputer vision and pattern recognition, pages 3642–3649. IEEE, 2012.[81] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus.Regularization of neural networks using dropconnect. In Internationalconference on machine learning, pages 1058–1066. PMLR, 2013.[82] Ikuro Sato, Hiroki Nishimura, and Kensuke Yokoi. Apac: Augmentedpattern classification with neural networks. arXiv preprintarXiv:1505.03229, 2015. → page 17[83] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.Generative adversarial nets. Advances in neural information processingsystems, 27, 2014. → page 17[84] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Dataaugmentation generative adversarial networks. arXiv preprintarXiv:1711.04340, 2017. → pages 17, 18[85] Giovanni Mariani, Florian Scheidegger, Roxana Istrate, Costas Bekas, andCristiano Malossi. Bagan: Data augmentation with balancing gan. arXivpreprint arXiv:1803.09655, 2018.131[86] Jaehoon Choi, Taekyung Kim, and Changick Kim. Self-ensembling withgan-based data augmentation for domain adaptation in semanticsegmentation. In Proceedings of the IEEE/CVF International Conferenceon Computer Vision, pages 6830–6840, 2019.[87] Abdul Waheed, Muskan Goyal, Deepak Gupta, Ashish Khanna, FadiAl-Turjman, and Placido Rogerio Pinheiro. Covidgan: data augmentationusing auxiliary classifier gan for improved covid-19 detection. IeeeAccess, 8:91916–91923, 2020.[88] Alceu Bissoto, Eduardo Valle, and Sandra Avila. Gan-based dataaugmentation and anonymization for skin-lesion analysis: A criticalreview. In Proceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition, pages 1847–1856, 2021.[89] Fei Ma, Yang Li, Shiguang Ni, Shaolun Huang, and Lin Zhang. Dataaugmentation for audio–visual emotion recognition with an efficientmultimodal conditional gan. Applied Sciences, 12(1):527, 2022. → page17[90] Jakub Nalepa, Michal Marcinkiewicz, and Michal Kawulok. Dataaugmentation for brain-tumor segmentation: a review. Frontiers incomputational neuroscience, page 83, 2019. → page 17[91] Suorong Yang, Weikang Xiao, Mengcheng Zhang, Suhan Guo, Jian Zhao,and Furao Shen. Image data augmentation for deep learning: A survey.arXiv preprint arXiv:2204.08610, 2022. → page 17[92] Terrance DeVries and Graham W Taylor. Improved regularization ofconvolutional neural networks with cutout. arXiv preprintarXiv:1708.04552, 2017. → page 18[93] Krishna Kumar Singh, Hao Yu, Aron Sarmasi, Gautam Pradeep, andYong Jae Lee. Hide-and-seek: A data augmentation technique forweakly-supervised localization and beyond. arXiv preprintarXiv:1811.02545, 2018.[94] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.Random erasing data augmentation. In Proceedings of the AAAIConference on Artificial Intelligence, volume 34, pages 13001–13008,2020. → page 78132[95] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmaskdata augmentation. arXiv preprint arXiv:2001.04086, 2020.[96] Pu Li, Xiangyang Li, and Xiang Long. Fencemask: A data augmentationapproach for pre-extracted image features. arXiv preprintarXiv:2006.07877, 2020. → page 18[97] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and OriolVinyals. Understanding deep learning (still) requires rethinkinggeneralization. Communications of the ACM, 64(3):107–115, 2021. →page 18[98] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.mixup: Beyond empirical risk minimization. arXiv preprintarXiv:1710.09412, 2017. → pages 18, 78[99] Hiroshi Inoue. Data augmentation by pairing samples for imagesclassification. arXiv preprint arXiv:1801.02929, 2018.[100] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, JunsukChoe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strongclassifiers with localizable features. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision, pages 6023–6032, 2019.→ page 78[101] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, IoannisMitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup:Better representations by interpolating hidden states. In InternationalConference on Machine Learning, pages 6438–6447. PMLR, 2019.[102] Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, AdamPrugel-Bennett, and Jonathon Hare. Fmix: Enhancing mixed sample dataaugmentation. arXiv preprint arXiv:2002.12047, 2020.[103] Jiawei Zhang, Yanchun Zhang, and Xiaowei Xu. Objectaug: object-leveldata augmentation for semantic image segmentation. In 2021International Joint Conference on Neural Networks (IJCNN), pages 1–8.IEEE, 2021. → page 18[104] Chia-Wen Kuo, Chih-Yao Ma, Jia-Bin Huang, and Zsolt Kira. Featmatch:Feature-based augmentation for semi-supervised learning. In European133Conference on Computer Vision, pages 479–495. Springer, 2020. → page18[105] Boyi Li, Felix Wu, Ser-Nam Lim, Serge Belongie, and Kilian QWeinberger. On feature normalization and data augmentation. InProceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 12383–12392, 2021. → page 18[106] Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and SungwoongKim. Fast autoaugment. Advances in Neural Information ProcessingSystems, 32, 2019. → page 18[107] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.Randaugment: Practical automated data augmentation with a reducedsearch space. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition Workshops, pages 702–703, 2021. →pages 18, 78[108] Yuejun Guo, Qiang Hu, Maxime Cordy, Mike Papadakis, and YvesLe Traon. Robust active learning: Sample-efficient training of robust deeplearning models. In 2022 IEEE/ACM 1st International Conference on AIEngineering–Software Engineering for AI (CAIN), pages 41–42. IEEE,2022. → page 19[109] Abbas Khan, Ijaz Ul Haq, Tanveer Hussain, Khan Muhammad,Mohammad Hijji, Muhammad Sajjad, Victor Hugo C De Albuquerque,and Sung Wook Baik. Pmal: A proxy model active learning approach forvision based industrial applications. ACM Transactions on MultimidiaComputing Communications and Applications, 2022.[110] Patrick Hemmer, Niklas Kuhl, and Jakob Schoffer. Deal: deep evidentialactive learning for image classification. In Deep Learning Applications,Volume 3, pages 171–192. Springer, 2022. → page 19[111] Hamed H Aghdam, Abel Gonzalez-Garcia, Joost van de Weijer, andAntonio M Lopez. Active learning for deep detection neural networks. InProceedings of the IEEE/CVF International Conference on ComputerVision, pages 3672–3680, 2019. → page 19134[112] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li,Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep activelearning. ACM Computing Surveys (CSUR), 54(9):1–40, 2021. → page 19[113] Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen.Suggestive annotation: A deep active learning framework for biomedicalimage segmentation. In International conference on medical imagecomputing and computer-assisted intervention, pages 399–407. Springer,2017. → page 19[114] Samuel Budd, Emma C Robinson, and Bernhard Kainz. A survey onactive learning and human-in-the-loop deep learning for medical imageanalysis. Medical Image Analysis, 71:102062, 2021.[115] Youcef Nafa, Qun Chen, Zhaoqiang Chen, Xingyu Lu, Haiyang He, TianyiDuan, and Zhanhuai Li. Active deep learning on entity resolution by risksampling. Knowledge-Based Systems, 236:107729, 2022. → page 19[116] Akshay L Chandra, Sai Vikas Desai, Wei Guo, and Vineeth NBalasubramanian. Computer vision with deep learning for plantphenotyping in agriculture: A survey. arXiv preprint arXiv:2006.11391,2020. → page 19[117] Sumit Kumar, Wenhao Luo, George Kantor, and Katia Sycara. Activelearning with gaussian processes for high throughput phenotyping. arXivpreprint arXiv:1901.06803, 2019. → page 19[118] Akshay L Chandra, Sai Vikas Desai, Vineeth N Balasubramanian, SeishiNinomiya, and Wei Guo. Active learning with point supervision forcost-effective panicle detection in cereal crops. Plant Methods, 16(1):1–16, 2020. → pages 19, 42[119] Rasha Sheikh, Andres Milioto, Philipp Lottes, Cyrill Stachniss, MarenBennewitz, and Thomas Schultz. Gradient and log-based active learningfor semantic segmentation of crop and weed for agricultural robots. In2020 IEEE International Conference on Robotics and Automation (ICRA),pages 1350–1356. IEEE, 2020. → page 19[120] Koushik Nagasubramanian, Talukder Jubery, Fateme Fotouhi Ardakani,Seyed Vahid Mirnezami, Asheesh K Singh, Arti Singh, Soumik Sarkar,135and Baskar Ganapathysubramanian. How useful is active learning forimage-based plant phenotyping? The Plant Phenome Journal, 4(1):e20020, 2021. → page 20[121] Shivangana Rawat, Akshay L Chandra, Sai Vikas Desai, Vineeth NBalasubramanian, Seishi Ninomiya, and Wei Guo. How useful isimage-based active learning for plant organ segmentation? PlantPhenomics, 2022, 2022. → page 20[122] Xue Ying. An overview of overfitting and its solutions. In Journal ofPhysics: Conference Series, volume 1168, page 022022. IOP Publishing,2019. → pages 23, 46[123] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, andJianming Liang. Unet++: A nested u-net architecture for medical imagesegmentation. In Deep learning in medical image analysis andmultimodal learning for clinical decision support, pages 3–11. Springer,2018. → page 23[124] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:Convolutional networks for biomedical image segmentation. InInternational Conference on Medical image computing andcomputer-assisted intervention, pages 234–241. Springer, 2015. → pages23, 74[125] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutionalnetworks for semantic segmentation. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages 3431–3440,2015. → pages 24, 74, 101[126] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.Imagenet: A large-scale hierarchical image database. In 2009 IEEEconference on computer vision and pattern recognition, pages 248–255.Ieee, 2009. → page 25[127] Karen Simonyan and Andrew Zisserman. Very deep convolutionalnetworks for large-scale image recognition. arXiv preprintarXiv:1409.1556, 2014. → page 26[128] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He.Aggregated residual transformations for deep neural networks. In136Proceedings of the IEEE conference on computer vision and patternrecognition, pages 1492–1500, 2017. → page 27[129] Md Mehedi Hasan, Joshua P Chopin, Hamid Laga, and Stanley JMiklavcic. Detection and analysis of wheat spikes using convolutionalneural networks. Plant Methods, 14(1):1–13, 2018. → pages 30, 57[130] Simon Madec, Xiuliang Jin, Hao Lu, Benoit De Solan, Shouyang Liu,Florent Duyme, Emmanuelle Heritier, and Frederic Baret. Ear densityestimation from high resolution rgb imagery using deep learningtechnique. Agricultural and forest meteorology, 264:225–234, 2019. →pages 30, 57[131] Daoyong Wang, Yuanyuan Fu, Guijun Yang, Xiaodong Yang, Dong Liang,Chengquan Zhou, Ning Zhang, Hongya Wu, and Dongyan Zhang.Combined use of fcn and harris corner detection for counting wheat ears infield conditions. IEEE Access, 7:178930–178941, 2019. → pages 30, 57[132] Pouria Sadeghi-Tehran, Nicolas Virlet, Eva M Ampe, Piet Reyns, andMalcolm J Hawkesford. Deepcount: in-field automatic quantification ofwheat spikes using simple linear iterative clustering and deepconvolutional neural networks. Frontiers in plant science, 10:1176, 2019.→ pages 30, 57[133] Haipeng Xiong, Zhiguo Cao, Hao Lu, Simon Madec, Liang Liu, andChunhua Shen. Tasselnetv2: in-field counting of wheat spikes withcontext-augmented local regression networks. Plant Methods, 15(1):1–14,2019. → pages 30, 42, 57[134] Weidi Xie, J Alison Noble, and Andrew Zisserman. Microscopy cellcounting and detection with fully convolutional regression networks.Computer methods in biomechanics and biomedical engineering: Imaging& Visualization, 6(3):283–292, 2018. → page 30[135] Irene Cortes, Jorge Beltran, Arturo de la Escalera, and Fernando Garcia.sianms: Non-maximum suppression with siamese networks formulti-camera 3d object detection. In 2020 IEEE Intelligent VehiclesSymposium (IV), pages 933–938. IEEE, 2020. → page 33[136] Etienne David, Simon Madec, Pouria Sadeghi-Tehran, Helge Aasen,Bangyou Zheng, Shouyang Liu, Norbert Kirchgessner, Goro Ishikawa,137Koichi Nagasawa, Minhajul A Badhon, et al. Global wheat head detection(gwhd) dataset: a large and diverse dataset of high-resolution rgb-labelledimages to develop and benchmark wheat head detection methods. PlantPhenomics, 2020, 2020. → pages 36, 76[137] Etienne David, Mario Serouart, Daniel Smith, Simon Madec, KaaviyaVelumani, Shouyang Liu, Xu Wang, Francisco Pinto Espinosa, ShahamehShafiee, Izzat SA Tahir, et al. Global wheat head dataset 2021: morediversity to improve the benchmarking of wheat head localizationmethods. arXiv preprint arXiv:2105.07660, 2021. → pages 36, 76[138] Zhang Yiwen, Shu Baiyi, Xue Ziwei, Wang Yue, and Mu Jiong. Predictionand counting of field wheat based on LC-DcVgg. In Proceedings of the2020 5th International Conference on Mathematics and ArtificialIntelligence, ICMAI 2020, pages 209–213, New York, NY, USA, 2020.Association for Computing Machinery. ISBN 9781450377072. → page 42[139] H Jabbar and Rafiqul Zaman Khan. Methods to avoid over-fitting andunder-fitting in supervised machine learning (comparative study).Computer Science, Communication and Instrumentation Devices, pages163–172, 2015. → page 46[140] Zhang Yiwen, Shu Baiyi, Xue Ziwei, Wang Yue, and Mu Jiong.Prediction and counting of field wheat based on lc-dcvgg. In Proceedingsof the 2020 5th International Conference on Mathematics and ArtificialIntelligence, pages 209–213, 2020. → page 57[141] Baohua Yang, Zhiwei Gao, Yuan Gao, and Yue Zhu. Rapid detection andcounting of wheat ears in the field using yolov4 with attention module.Agronomy, 11(6):1202, 2021. → page 57[142] Sandesh Bhagat, Manesh Kokare, Vineet Haswani, Praful Hambarde, andRavi Kamble. Wheatnet-lite: a novel light weight network for wheat headdetection. In Proceedings of the IEEE/CVF international conference oncomputer vision, pages 1332–1341, 2021. → page 57[143] Lei Li, Muhammad Adeel Hassan, Shurong Yang, Furong Jing, MengjiaoYang, Awais Rasheed, Jiankang Wang, Xianchun Xia, Zhonghu He, andYonggui Xiao. Development of image-based wheat spike counter through138a faster r-cnn algorithm and application for genetic studies. The CropJournal, 2022. → page 57[144] Ehsan Ullah, Mohib Ullah, Muhammad Sajjad, and Faouzi Alaya Cheikh.Deep learning based wheat ears count in robot images for wheatphenotyping. Electronic Imaging, 34:1–6, 2022. → page 57[145] Ivan Lujic, Vincenzo De Maio, and Ivona Brandic. Efficient edge storagemanagement based on near real-time forecasts. In 2017 IEEE 1stInternational Conference on Fog and Edge Computing (ICFEC), pages21–30. IEEE, 2017. → page 56[146] Milan Despotovic, Vladimir Nedic, Danijela Despotovic, and SlobodanCvetanovic. Evaluation of empirical models for predicting monthly meanhorizontal diffuse solar radiation. Renewable and Sustainable EnergyReviews, 56:246–260, 2016. → page 56[147] Amirhossein Zaji, Zheng Liu, Gaozhi Xiao, Pankaj Bhowmik, Jatinder SSangha, and Yuefeng Ruan. Wheat spike counting using regression andlocalization approaches. In 2021 IEEE International Conference onImaging Systems and Techniques (IST), pages 1–6. IEEE, 2021. → pages63, 77, 101[148] Siddharth Srivastava, Swati Bhugra, Brejesh Lall, and Santanu Chaudhury.Drought stress classification using 3d plant models. In Proceedings of theIEEE International Conference on Computer Vision Workshops, pages2046–2054, 2017. → pages 63, 77[149] Baris Ozmen. AutoML for Data Augmentation, 05 2019. URLhttps://github.com/barisozmen/deepaugment. → page 66[150] Alexander Buslaev, Vladimir I Iglovikov, Eugene Khvedchenya, AlexParinov, Mikhail Druzhinin, and Alexandr A Kalinin. Albumentations:fast and flexible image augmentations. Information, 11:125, 2020. →page 67[151] John H Holland. Genetic algorithms. Scientific american, 267(1):66–73,1992. → page 71[152] Aradhita Bhandari, BK Tripathy, Khurram Jawad, Surbhi Bhatia,Mohammad Khalid Imam Rahmani, and Arwa Mashat. Cancer detection139and prediction using genetic algorithms. Computational Intelligence andNeuroscience, 2022, 2022. → page 71[153] Gabriel Mihail Danciu and Alexandru Dinu. Coverage fulfillmentautomation in hardware functional verification using genetic algorithms.Applied Sciences, 12(3):1559, 2022.[154] Mohammad Esrafilian-Najafabadi and Fariborz Haghighat. Impact ofpredictor variables on the performance of future occupancy prediction:Feature selection using genetic algorithms and machine learning. Buildingand Environment, page 109152, 2022. → page 71[155] Stephan Olariu and Albert Y Zomaya. Handbook of bioinspiredalgorithms and applications. CRC Press, 2005. → page 72[156] Fred W Glover and Gary A Kochenberger. Handbook of metaheuristics,volume 57. Springer Science & Business Media, 2006. → page 72[157] Piotr Skalski. Make Sense. https://github.com/SkalskiP/make-sense/,2019. → page 76[158] Amirhossein Zaji, Zheng Liu, Gaozhi Xiao, Pankaj Bhowmik, Jatinder SSangha, and Yuefeng Ruan. Wheat spike localization and counting viahybrid unet architectures. Computers and Electronics in Agriculture, 203:107439, 2022. → pages 76, 101[159] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer,and Balaji Lakshminarayanan. Augmix: A simple data processing methodto improve robustness and uncertainty. arXiv preprint arXiv:1912.02781,2019. → page 78[160] Amirhossein Zaji, Zheng Liu, Gaozhi Xiao, Jatinder S Sangha, andYuefeng Ruan. A survey on deep learning applications in wheatphenotyping. Applied Soft Computing, page 109761, 2022. → page 95[161] Syed Sahil Abbas Zaidi, Mohammad Samar Ansari, Asra Aslam, NadiaKanwal, Mamoona Asghar, and Brian Lee. A survey of modern deeplearning based object detection models. Digital Signal Processing, page103514, 2022. → page 96140[162] G Lowe. Sift-the scale invariant feature transform. Int. J, 2(91-110):2,2004. → page 98[163] Navneet Dalal and Bill Triggs. Histograms of oriented gradients forhuman detection. In 2005 IEEE computer society conference on computervision and pattern recognition (CVPR’05), volume 1, pages 886–893.Ieee, 2005. → page 98[164] Vrushali Y Kulkarni and Pradeep K Sinha. Pruning of random forestclassifiers: A survey and future directions. In 2012 InternationalConference on Data Science & Engineering (ICDSE), pages 64–68. IEEE,2012. → page 99[165] Paulo Angelo Alves Resende and Andre Costa Drummond. A survey ofrandom forest based methods for intrusion detection systems. ACMComputing Surveys (CSUR), 51(3):1–36, 2018.[166] Anjaneyulu Babu Shaik and Sujatha Srinivasan. A brief survey on randomforest ensembles in classification model. In International Conference onInnovative Computing and Communications, pages 253–260. Springer,2019. → page 99[167] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Maskr-cnn. In Proceedings of the IEEE international conference on computervision, pages 2961–2969, 2017. → page 99[168] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Richfeature hierarchies for accurate object detection and semanticsegmentation. In Proceedings of the IEEE conference on computer visionand pattern recognition, pages 580–587, 2014. → page 99[169] Minxi Rong, Zhizheng Wang, Bin Ban, and Xiaoli Guo. Pest identificationand counting of yellow plate in field based on improved mask r-cnn.Discrete Dynamics in Nature and Society, 2022, 2022. → page 99[170] Daichi Minagawa and Jeyeon Kim. Prediction of harvest time of tomatousing mask r-cnn. AgriEngineering, 4(2):356–366, 2022.[171] He Li, Hongtao Shi, Anghong Du, Yilin Mao, Kai Fan, Yu Wang,Yaozong Shen, Shuangshuang Wang, Xiuxiu Xu, Lili Tian, et al.141Symptom recognition of disease and insect damage based on mask r-cnn,wavelet transform, and f-rnet. Frontiers in Plant Science, 13, 2022.[172] Alakananda Mitra, Saraju P Mohanty, and Elias Kougianos. A smartagriculture framework to automatically track the spread of plant diseasesusing mask region-based convolutional neural network. In IFIPInternational Internet of Things Conference, pages 68–85. Springer, 2022.→ page 99[173] Wen-Hao Su, Jiajing Zhang, Ce Yang, Rae Page, Tamas Szinyei, Cory DHirsch, and Brian J Steffenson. Automatic evaluation of wheat resistanceto fusarium head blight using dual mask-rcnn deep learning frameworks incomputer vision. Remote sensing, 13(1):26, 2020. → page 99[174] Elias Ennadifi, Sohaib Laraba, Damien Vincke, Benoit Mercatoris, andBernard Gosselin. Wheat diseases classification and localization usingconvolutional neural networks and gradcam visualization. In 2020International Conference on Intelligent Systems and Computer Vision(ISCV), pages 1–5. IEEE, 2020.[175] Deepak Kumar and Vinay Kukreja. An instance segmentation approachfor wheat yellow rust disease recognition. In 2021 InternationalConference on Decision Aid Sciences and Application (DASA), pages926–931. IEEE, 2021.[176] Yichao Gao, Hetong Wang, Man Li, and Wen-Hao Su. Automatic tandemdual blendmask networks for severity assessment of wheat fusarium headblight. Agriculture, 12(9):1493, 2022.[177] Deepak Kumar and Vinay Kukreja. Quantifying the severity of loose smutin wheat using mrcnn. In 2022 International Conference on Decision AidSciences and Applications (DASA), pages 630–634. IEEE, 2022. → page99[178] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, andArnold WM Smeulders. Selective search for object recognition.International journal of computer vision, 104(2):154–171, 2013. → page99[179] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE internationalconference on computer vision, pages 1440–1448, 2015. → page 101142[180] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn:Towards real-time object detection with region proposal networks. InAdvances in neural information processing systems, pages 91–99, 2015.→ page 101[181] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona,Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco:Common objects in context. In European conference on computer vision,pages 740–755. Springer, 2014. → page 101[182] Egidijus Sarauskis, Marius Kazlauskas, Vilma Naujokiene, IndreBruciene, Dainius Steponavicius, Kestutis Romaneckas, and AlgirdasJasinskas. Variable rate seeding in precision agriculture: Recent advancesand future perspectives. Agriculture, 12(2):305, 2022. → page 108[183] Chantana Sornsiri, Saowanee Pisithpun, and Patchance Kamnirdrat.Optimum plot size and shape for wheat experimental plot. WarasanWichakan Kaset, 1991. → page 108[184] Ibraheem Alhashim and Peter Wonka. High quality monocular depthestimation via transfer learning. arXiv preprint arXiv:1812.11941, 2018.→ page 115[185] Sergio Garrido-Jurado, Rafael Munoz-Salinas, Francisco JoseMadrid-Cuevas, and Manuel Jesus Marin-Jimenez. Automatic generationand detection of highly reliable fiducial markers under occlusion. PatternRecognition, 47(6):2280–2292, 2014. → page 115[186] Thomas R Gruber. The role of common ontology in achieving sharable,reusable knowledge bases. Kr, 91:601–602, 1991. → page 145[187] Steffen Staab and Rudi Studer. Handbook on ontologies. Springer Science& Business Media, 2010. → page 145[188] Raul Navarro-Almanza, Reyes Juarez-Ramirez, Guillermo Licea, andJuan R Castro. Automated ontology extraction from unstructured textsusing deep learning. In Intuitionistic and Type-2 Fuzzy LogicEnhancements in Neural and Optimization Algorithms: Theory andApplications, pages 727–755. Springer, 2020. → page 145143[189] Diana Sousa and Francisco M Couto. Biont: deep learning using multiplebiomedical ontologies for relation extraction. In European Conference onInformation Retrieval, pages 367–374. Springer, 2020.[190] Mark Abraham Magumba, Peter Nabende, and Ernest Mwebaze.Ontology boosted deep learning for disease name extraction from twittermessages. Journal of Big Data, 5(1):31, 2018. → page 145[191] P Sanjeevi, B Siva Kumar, S Prasanna, J Maruthupandi, R Manikandan,and A Baseera. An ontology enabled internet of things framework inintelligent agriculture for preventing post-harvest losses. Complex &Intelligent Systems, pages 1–17, 2020. → page 145[192] D Budaev, A Lada, E Simonova, P Skobelev, V Travin, and O Yalovenko.Conceptual design of smart farming solution for precise agriculture.Manag. App. Complex Syst, 13:309–316, 2019.[193] Quoc Hung Ngo, Nhien-An Le-Khac, and Tahar Kechadi. Ontology basedapproach for precision agriculture. In International Conference onMulti-disciplinary Trends in Artificial Intelligence, pages 175–186.Springer, 2018. → page 145[194] Nataliia Kussul, Mykola Lavreniuk, Sergii Skakun, and Andrii Shelestov.Deep learning classification of land cover and crop types using remotesensing data. IEEE Geoscience and Remote Sensing Letters, 14(5):778–782, 2017. → page 149144Appendix AOntology-based knowledge manage-ment for deep learning applicationsin wheat phenotypingThis chapter introduced an ontology-based knowledge management system forDeep Learning (DL) applications in the wheat phenotyping domain. The de-veloped ontology enables advanced searchability, which significantly facilitatesknowledge workers’ access to unstructured and scattered information.Ontology was previously regarded as a philosophical concept. However, withthe advent of computers, the ontology was adopted as a means of concept shar-ing [186]. Ontology, in general, is defined by four fundamental properties: ex-plicit, formal, shared, and conceptualization [187]. The application of ontologyto any body of knowledge enables the proper recognition of the relationships be-tween all concepts. Due to the formal nature of ontology, computers can easilyrecognize the knowledge and assist us in comprehending it. Numerous articleshave examined ontology’s application in DL-related issues [188–190] as well asin agriculture [191–193].The ontology proposed in this chapter is intended to present gathered knowl-edge in a way that makes it easier and more feasible for other researchers to apply,145Appendixanalyze, and review. The ontology presented here contains several classes, prop-erties, and instances. An Instance refers to a thing or a particular individual ofthe ontology. The term class refers to a variety of ontological concepts, each ofwhich contains multiple individuals belonging to a particular domain. Three dis-tinct types of properties are used in the presented ontology: object properties,data properties, and annotation properties. Object properties are utilized to definethe relationships between individuals and classes. The relationship between anontology object and a data value is defined using data properties and annotationproperties.Figure A.1 illustrates the structure of the classes and object properties pre-sented in this ontology. According to this figure, in the first layer, the ontol-ogy is linked to four classes called Goals, Authors, Models, and Articles withthe has Subclass object property. Then, in the second layer, the Models classhas five subclasses: Preprocessing, Frameworks, Model Goal, Model Name, andModel Type, with Predefined Architecture as a subclass of Model Name.As the name implies, the Authors class contains all authors of articles that havebeen reviewed. Each author is defined as an individual. They are associated withtheir paper via the has Author object property in the Articles class. As a result,one of the benefits of the presented ontology is that it enables easy access to allarticles published in this field by a particular researcher via a simple query.Another class in the first layer of this ontology is Goals, which contains thelist of general objectives pursued by the reviewed articles. It should be noted thateach of these goals is stored as an individual in the Goals class. As previouslystated, one of the benefits of utilizing ontologies in the knowledge managementof a review article is the ability to query the collected information. For instance,we can use the query shown in Figure A.2 to find individuals who belong to theGoals class. The articles reviewed in this review are classified into six categoriesbased on the objective they pursue. Additionally, this figure indicates the numberof articles in each category. As can be seen, the most popular group contains atotal of twelve articles.146Knowledge baseArticlesAuthorsFrameworksGoals ModelsModel_Goal Model_NamePredefined_ArchitectureModel_TypePreprocessingbest_Modelbetter_Thanhas_Subclasshas_Authorhas_Goalhas_Model_Goalhas_Model_Namehas_Model_Typehas_Predefined_Architectureis_A_Kind_Ofpreprocessing_Methodused_FrameworkFigure A.1: Structure of the classes and object properties of the presentedontology.Another class defined in this ontology is Models, which has additional sub-classes. This class is used to organize data pertaining to the DL model used in eachpaper. One of the subclasses of the Models class is Preprocessing, which is usedby researchers to perform preprocessing prior to running the DL model. Anothersubclass of Models is Frameworks, which contains the names of the frameworksused in the modeling procedure. TensorFlow, Torch, and Cafe are three of the mostwell-known frameworks mentioned in the reviewed articles. The Model Goal sub-class of the Models class is used to investigate the overall purpose of modeling.Note that this class is distinct from the Goals class discussed previously. Thearticle’s overall purpose was classified under Goals, but the DL model’s goal ispresented under Model Goal. As illustrated in Figure A.2, the model goals arestored as individuals in the Model Goal class, similar to the Goals class. Anothersubclass of the Models class is Model Name, which, as the name implies, storesthe model names used in the articles studied. Additionally, if the model being usedis associated with a predefined architecture, the model’s name is stored in the Pre-147defined Architecture class, which is a subset of the Model Name class. Figure A.2illustrates several of the Model Name individuals. The last subclass of Modelsis Model Type, which indicates the model’s overall type, such as unsupervised orsupervised learning.Articles is a critical class in this ontology. This class is used to store the arti-cles discussed in this paper. Figure A.2 illustrates several members of this class.As can be seen, each article discussed in this ontology is preceded by a numberfrom the developed ontology. As illustrated in Figure A.1, this class is connectedto other classes via a large number of object properties. The best Model objectproperty, which has the Articles domain and the Models range, displays the ar-ticle’s best model, which produces the most accurate results. The better Thanobject property method compares the best model results to those of other modelsstudied in the reviewed papers and indicates which models are defeated by thearticle’s best model. Additionally, as illustrated in Figure A.1, the Articles classmakes use of the has Author, has Goal, preprocessing Method, used Framework,has Model Goal, has Model Name, has predefined Architecture, and has Model Typeobject properties to connect to the Authors, Goals, Preprocessing, Frameworks,Model Goal, Model Name, Predefined Architecture, and Model Type, respectively.SELECT ?Goals (count(?var) as ?No)WHERE {?var :has_Goal ?Goals.}GROUP BY ?GoalsORDER BY DESC(?No)Goals Nowheat_spike_detection_and_counting12wheat_production_modeling11wheat_content_estimation7wheat_spatial_distribution_extracting7wheat_grain_identification 2wheat_root_modeling 2SELECT ?Model_GoalsWHERE {?Model_Goals rdf:type :Model_Goal.} Model_Goalsclassificationregressionsegmentationlocalizationcountingdetection...SELECT ?Model_NamesWHERE {?Model_Names rdf:type:Model_Name.} Model_Namesmulti_layer_perceptron_MLPrandom_forest_RFmultivariate_linear_regressionconvolutional_neural_network_CNNmultiple_instance_learning_MILfully_convolutional_network_FCN...SELECT ?Paper_NoWHERE {?Paper_No rdf:type :Articles.} Paper_No151163140175152173...Figure A.2: Sample queries and individuals’ information of Goals,Model Goals, Model Name, and Articles classes.148Data properties are another type of property that is used in the ontology pre-sented in this article. Unlike object properties, which establish a connectionbetween classes and individuals, data properties establish a connection betweenan individual and its value. Figure A.3 illustrates the ontology’s data proper-ties and an example of their values for one of the articles under review [194].The first layer of this ontology contains three data properties: code Properties,dataset Properties, and performance Metrics. Given that the majority of the modeland code specifications are covered by object properties in this ontology, the solepurpose of code Properties is to connect the article under review and the codewritten for it (if available). The most comprehensive collection of data propertiesis dataset Properties, which contains and organizes all of the properties of thedata used in the article under review. Finally, performance Metrics is responsiblefor the accuracy of the best model used in the article under review.data_Propertiescode_Propertiesdataset_Propertiesannotation_Typedata_Areadata_Availabledata_Typedataset_Namelink_Of_Datasetnumber_Of_Classesnumber_Of_Samplestrain/test_Descriptionused_Instrumentperformance_Metricslink_Of_Codesdata_Properties Valuesnumber_Of_Classes 11number_Of_Samples 547train/test_Description "The polygons were randomly divided into training set (50%) and validation set (50%). "data_Area "satellite images"dataset_Name "Sentinel-1A satellite"data_Available "false" annotation_Type "unsupervised segmentation"performance_Metrics classification accuracy = 94.6data_Type "Image"dataset_Name "Landsat-8 satellite"SELECT DISTINCT ?data_Properties ?ValuesWHERE {?data_Properties rdf:type owl:DatatypeProperty .:102 ?data_Properties ?Values}Figure A.3: The structure of the data properties used in the presented ontol-ogy and an example of their values for one of the papers of the Articlesclass.Annotation is the last type of property in the ontology. Annotation is used toprovide extra-logical information about a subject that is not used in ontology rea-149soning but is nonetheless helpful and readable by humans. Figure A.4 illustratesthe types of annotations included in this ontology and their associated values forone of the articles reviewed in this study. As shown in the figure, the annotationpublished In is used to record the journal or conference in which the article waspublished. published Year denotes the year of publication, doi denotes the arti-cle’s digital object identifier, link Of Paper denotes the article’s publisher’s webpage link, problem Description denotes the article’s solution to a problem, andtitle, as the name implies, denotes the article’s title.annotation_Propertiespublished_Inpublished_Yeardoiannotation_Properties Valuespublished_In "IEEE Geoscience and Remote Sensing"published_Year 2017doi "10.1109/LGRS.2017.2681128"link_Of_Paper "https://ieeexplore.ieee.org/document/7891032"problem_Description"Using multilevel DL architecture that targets land cover and crop type classification from multitemporal multisource satellite imagery."title "Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data"SELECT DISTINCT ?annotation_Properties ?ValuesWHERE {?annotation_Properties rdf:type owl:AnnotationProperty .:102 ? annotation_Properties ?Values}link_Of_Paperproblem_DescriptiontitleFigure A.4: The structure of the annotation properties used in the presentedontology and an example of their values for one of the papers of theArticles class.150
Thesis/Dissertation
eng
Civil Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Liu, Zheng (Professor of engineering)
High-throughput wheat plant phenotyping using deep learning techniques
Text