Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Zhang, Ran
Doctor of Philosophy - PhD
University of British Columbia
Automated container code reading systems are of great value to logistics management. However, previous container code spotting methods have limited recognition performance in complex outdoor environments of transportation industry. Therefore, this research focuses on designing high-performance shipping container code spotting (including detection and recognition) methods for automated container code reading. First, an adaptive deep learning framework is proposed for horizontal code spotting. Adaptive threshold and average score aggregation (ASA) are designed to remove the noisy text. The detected code regions are adaptively adjusted for code recognition in our framework. Experiments have been conducted on real horizontal code image datasets. The overall recognition accuracy has been improved from 90.17% to 93.33%. Second, a deep learning model is proposed for vertical code spotting. It aligns vertical features of container codes. In addition, it includes two recognition pipelines and an uncertainty and probability-based decision strategy to adaptively choose the final results from two pipelines. Experiments conducted on real vertical code datasets demonstrate that the overall recognition f1-score performance has been improved from 93.5% to 96.5%. Third, a visual and textual information fusion-based zero-shot deep learning framework is proposed to spot placard codes. In this framework, Logarithmic Weighted Cross-Entropy (LWCE) is designed to reduce the effects of imbalanced classes and Logarithmic Weighted Confidence Fusion (LWCF) is proposed to fuse the visual and textual information to handle unseen classes and enhance performance in seen classes. Our proposed framework has been validated on real industrial placard datasets. The overall end-to-end recognition f1-score performance has been improved from 79.97% to 92.77%. Overall, our deep learning-based code spotting methods improve the accuracy of automated container code recognition and facilitate automated container management.
Spotting Shipping Container Codes withDeep Learning MethodsbyRan ZhangB.Eng., Xidian University, 2015M.Eng., Capital Normal University, 2018a thesis submitted in partial fulfillmentof the requirements for the degree ofDOCTOR OF PHILOSOPHYinthe college of graduate studies(Electrical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)December 2022© Ran Zhang, 2022The following individuals certify that they have read, and recommend to theCollege of Graduate Studies for acceptance, the dissertation entitled:SPOTTING SHIPPING CONTAINER CODES WITH DEEP LEARNINGMETHODSsubmitted by Ran Zhang in partial fulfillment of the requirements of thedegree of Doctor of Philosophy.Dr. Zheng Liu, School of Engineering, The University of British ColumbiaSupervisorDr. Richard Klukas, School of Engineering, The University of British ColumbiaSupervisory Committee MemberDr. Lo¨ıc Markley, School of Engineering, The University of British ColumbiaSupervisory Committee MemberDr. Jian Liu, School of Engineering, The University of British ColumbiaUniversity ExaminerDr. Sreeraman Rajan, Department of Systems and Computer Engineering,Carleton UniversityExternal ExamineriiABSTRACTAbstractAutomated container code reading systems are of great value to logisticsmanagement. However, previous container code spotting methods have lim-ited recognition performance in complex outdoor environments of transporta-tion industry. Therefore, this research focuses on designing high-performanceshipping container code spotting (including detection and recognition) meth-ods for automated container code reading.First, an adaptive deep learning framework is proposed for horizontalcode spotting. Adaptive threshold and average score aggregation (ASA) aredesigned to remove the noisy text. The detected code regions are adaptivelyadjusted for code recognition in our framework. Experiments have beenconducted on real horizontal code image datasets. The overall recognitionaccuracy has been improved from 90.17% to 93.33%.Second, a deep learning model is proposed for vertical code spotting. Italigns vertical features of container codes. In addition, it includes two recog-nition pipelines and an uncertainty and probability-based decision strategyto adaptively choose the final results from two pipelines. Experiments con-ducted on real vertical code datasets demonstrate that the overall recognitionf1-score performance has been improved from 93.5% to 96.5%.Third, a visual and textual information fusion-based zero-shot deep learn-ing framework is proposed to spot placard codes. In this framework, Loga-rithmic Weighted Cross-Entropy (LWCE) is designed to reduce the effects ofimbalanced classes and Logarithmic Weighted Confidence Fusion (LWCF) isiiiABSTRACTproposed to fuse the visual and textual information to handle unseen classesand enhance performance in seen classes. Our proposed framework has beenvalidated on real industrial placard datasets. The overall end-to-end recog-nition f1-score performance has been improved from 79.97% to 92.77%.Overall, our deep learning-based code spotting methods improve the ac-curacy of automated container code recognition and facilitate automatedcontainer management.ivLAY SUMMARYLay SummaryShipping containers are the fundamental equipment in international trans-portation, carrying nearly all kinds of materials and products. Containercodes include container identification codes, container type codes, and haz-ardous material placard codes. They provide indispensable information aboutcontainers and their transported freight. Accordingly, automated containercode reading systems are crucial to logistics management and are stronglydesired by transportation industry. However, contemporary methods of de-tecting and recognizing container codes suffer from low accuracy due to awide variety of text arrangements, backgrounds, and outdoor environments.Therefore, this research focuses on spotting, including detecting and rec-ognizing, shipping container codes with deep learning methods to providehigh-performance solutions for the challenge of automated container codereading. The research outcomes will improve the quality of container man-agement, thus contributing to the acceleration of the supply chain and growthof the global economy.vPREFACEPrefaceThis dissertation is written based on my research work in the School ofEngineering, the University of British Columbia, Okanagan Campus, underthe supervision of Dr. Zheng Liu. I accomplished the literature review,method design, experimental validation, data analysis, as well as the writingof this thesis. Dr. Zheng Liu guided me during the finalization of the researchscope and research plan. Dr. Richard Klukas, and Dr. Lo¨ıc Markley offeredvaluable suggestions.The publications that I participated in during the Ph.D. program arelisted as follows:Journal PapersJ1. Ran Zhang, Zhila Bahrami, Teng Wang, and Zheng Liu, “An Adap-tive Deep Learning Framework for Shipping Container Code Localiza-tion and Recognition,” in IEEE Transactions on Instrumentation andMeasurement, vol. 70, pp. 1-13, 2021.J2. Ran Zhang, Zhila Bahrami, and Zheng Liu, “A Vertical Text Spot-ting Model for Trailer and Container Codes,” in IEEE Transactions onInstrumentation and Measurement, vol. 70, pp. 1-13, 2021.J3. Ran Zhang, Zhila Bahrami, Ke Feng, and Zheng Liu, “A Visual andTextual Information Fusion-based Zero-Shot Framework for HazardousMaterial Placard Detection and Recognition,” in IEEE Transactions onArtificial Intelligence (submitted on July 21, 2022).viPREFACEJ4. Zhila Bahrami, Ran Zhang, Teng Wang, and Zheng Liu, “An End-to-End Framework for Shipping Container Corrosion Defect Inspection,”in IEEE Transactions on Instrumentation and Measurement, 2022.J5. Zhila Bahrami, Ran Zhang, Rakiba Rayhana, and Zheng Liu, “AnHRCR-CNN Framework for Automated Security Seal Detection on theShipping Container,” in IEEE Transactions on Instrumentation andMeasurement, vol. 70, pp. 1-13, 2021.J6. Jing Lin, Julia´n A. Ferna´ndez, Rakiba Rayhana, Amirhossei Zaji, RanZhang, Omar E. Herrera, Zheng Liu, and Walter Me´rida, “Predic-tive analytics for building power demand: Day-ahead forecasting andanomaly prediction,” in Energy and Buildings, vol. 255, 2022.Conference PapersC1. Ran Zhang, Zhila Bahrami, and Zheng Liu, “An Effective Method forAutomated Railcar Number Detection and Recognition Based on DeepLearning,” in 2021 Canadian Conference on Electrical and ComputerEngineering (CCECE), Sep. 12 - 17, 2021.C2. Ran Zhang, Junchi Bin, Zheng Liu, and Erik Blasch, “Image transla-tion to enhance IR2VIS image registration,” in Geospatial InformaticsXI, Apr. 14, 2021.C3. Junchi Bin, Ran Zhang, Kasun Hewage, and Zheng Liu, “An IterativeVisible-Infrared Image Fusion Framework Based on Wavelet Autoen-coder”, 10th International Conference on Computer and Communica-tions Management (ICCCM), Okayama University, Japan, Jul. 29-31,2022.C4. Junchi Bin, Ran Zhang, Chengkai Zhang, Xiangrui Liu, Shan Du,Erik Blasch, and Zheng Liu, “Improved Object Detection in ThermalImaging Through Context Enhancement and Information Fusion: AviiPREFACECase Study in Autonomous Driving”, Conference on Neural Informa-tion Processing Systems (NeurIPS) 2021, Workshop on Machine Learn-ing for Autonomous Driving, Dec. 13, 2021.C5. Zhila Bahrami, Ran Zhang, Rakiba Rayhana, Teng Wang, and ZhengLiu, “Optimized Deep Neural Network Architectures with Anchor Boxoptimization for Shipping Container Corrosion Inspection”, in 2020IEEE Symposium Series on Computational Intelligence (SSCI), pp.1328-1333, Dec. 1-4, 2020.C6. Zhila Bahrami, Ran Zhang, Rakiba Rayhana, and Zheng Liu, “DeepLearning-based Framework for Shipping Container Security Seal De-tection,” in 2021 Joint 10th International Conference on Informatics,Electronics & Vision (ICIEV) and 2021 5th International Conferenceon Imaging, Vision & Pattern Recognition (icIVPR), 2021, pp. 1-7Book ChapterB1. Ran Zhang, Junchi Bin, Zheng Liu, and Erik Blasch, “Chapter 13- WGGAN: A Wavelet-Guided Generative Adversarial Network forThermal Image Translation,” in Generative Adversarial Networks forImage-to-ImageTranslation, A. Solanki, A. Nayyar, and M. Naved, Eds.Academic Press, 2021, pp. 313–327.J1 [1], J2 [2], and J3 [3] are our work presented in Chapters 3, 4, and 5,respectively. Part of the work in C1 [4] is included in Chapter 2. Regardingthe relative contributions, I conducted all the method designs, experiments,and manuscript writing of J1 [1], J2 [2], J3 [3], and C1 [4], where other co-authors offered valuable suggestions. In C2 [5] and B1 [6], Junchi Bin andI contributed equally, with the rest of the authors providing advice. In J4[7], J5 [8], J6 [9], C3 [10], C4 [11], C5 [12], and C6 [13], the first authorof each paper made the principal contribution to the research plan design,experiments, and paper writing, while the other co-authors participated inviiiPREFACEthe discussion of the experimental validation, result presentation, and paperwriting.ixTABLE OF CONTENTSTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxvAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . .xxviiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxviii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Research Background and Motivation . . . . . . . . . . . . . . 11.2 Shipping Container Codes . . . . . . . . . . . . . . . . . . . . 21.2.1 Horizontal Container Codes . . . . . . . . . . . . . . . 31.2.2 Vertical Container Codes . . . . . . . . . . . . . . . . . 51.2.3 Hazardous Material Placard Codes . . . . . . . . . . . 71.3 Research Gaps . . . . . . . . . . . . . . . . . . . . . . . . . . . 9xTABLE OF CONTENTS1.3.1 Spotting Horizontal Shipping Container Codes . . . . . 91.3.2 Spotting Vertical Shipping Container Codes . . . . . . 101.3.3 Spotting Hazardous Material Placard Codes . . . . . . 111.4 Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . 111.5 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . 152.1 Computer Vision-Based Scene Text Detection and Recogni-tion Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.1.1 Scene Text Detection . . . . . . . . . . . . . . . . . . . 162.1.2 Scene Text Recognition . . . . . . . . . . . . . . . . . . 172.1.3 Scene Text Spotting . . . . . . . . . . . . . . . . . . . 182.2 Horizontal Shipping Container Code Detection and Recogni-tion Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.2.1 Non-deep Learning Methods . . . . . . . . . . . . . . . 192.2.2 Deep Learning Methods . . . . . . . . . . . . . . . . . 202.3 Vertical Shipping Container Code Detection and RecognitionMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.3.1 Non-Deep Learning Methods . . . . . . . . . . . . . . . 212.3.2 Deep Learning Methods . . . . . . . . . . . . . . . . . 222.4 Hazardous Material Placard Code Detection and RecognitionMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.4.1 Visual Information-based Methods . . . . . . . . . . . 232.4.2 Textual Information-based Methods . . . . . . . . . . . 242.5 Other Automated Code Reading Systems in TransportationIndustry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Spotting Horizontal Shipping Container Codes . . . . . . . 273.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.2 An Adaptive Deep Learning Framework for Horizontal CodeDetection and Recognition . . . . . . . . . . . . . . . . . . . . 28xiTABLE OF CONTENTS3.2.1 Horizontal Shipping Container Code Detection . . . . . 293.2.2 Horizontal Shipping Container Code Recognition . . . 433.3 Experimental Results and Discussion . . . . . . . . . . . . . . 473.3.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . 473.3.2 Network Parameters for Training, and Evaluation Met-rics for Testing . . . . . . . . . . . . . . . . . . . . . . 503.3.3 Model Optimization . . . . . . . . . . . . . . . . . . . 523.3.4 Comparison with the State-of-the-Art Solutions . . . . 593.3.5 Processing speed of code recognition at ports . . . . . . 633.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644 Spotting Vertical Shipping Container Codes . . . . . . . . . 664.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.2 Vertical Shipping Container Code Spotting Model . . . . . . . 684.2.1 Vertical Shipping Container Code Detection . . . . . . 684.2.2 Feature Sharing for Code Detection and Recognition . 704.2.3 Vertical Shipping Container Code Recognition . . . . . 724.3 Experimental Results and Discussion . . . . . . . . . . . . . . 764.3.1 Trailer and Container Code Dataset . . . . . . . . . . . 764.3.2 Model Implementation . . . . . . . . . . . . . . . . . . 774.3.3 Ablation Study . . . . . . . . . . . . . . . . . . . . . . 804.3.4 Comparison with the State of the Art . . . . . . . . . . 874.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 965 Spotting Hazardous Material Placard Codes . . . . . . . . . 985.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 995.2 Proposed Zero-Shot Framework for Placard Code Spotting . . 1015.2.1 Visual Information Extraction . . . . . . . . . . . . . . 1025.2.2 Textual Information Extraction . . . . . . . . . . . . . 1045.2.3 Logarithmic Weighted Confidence Fusion . . . . . . . . 1065.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108xiiTABLE OF CONTENTS5.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . 1085.3.2 Training and Testing Settings . . . . . . . . . . . . . . 1095.3.3 Experimental Results . . . . . . . . . . . . . . . . . . . 1125.3.4 Ablation Studies . . . . . . . . . . . . . . . . . . . . . 1205.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1216 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1236.1 Thesis Summary . . . . . . . . . . . . . . . . . . . . . . . . . 1236.2 Research Contributions . . . . . . . . . . . . . . . . . . . . . . 1246.3 Research Strengths and Potential Applications . . . . . . . . . 1256.4 Research Limitations and Future Work . . . . . . . . . . . . . 126Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128xiiiLIST OF TABLESList of TablesTable 1.1 The meanings of various placards . . . . . . . . . . . . . . . 8Table 3.1 Blocks details in the adopted Resnet . . . . . . . . . . . . 32Table 3.2 Hyperparameters in our framework . . . . . . . . . . . . . . 51Table 3.3 The comparison of different threshold methods on score map 57Table 3.4 Recognition performance comparison of different mergingmethod for overlapped bounding boxes . . . . . . . . . . . 58Table 3.5 Localization performance when IoU threshold is set with 0.75 59Table 3.6 Recognition performance and processing speed . . . . . . . 59Table 4.1 Training cost of each method. . . . . . . . . . . . . . . . . . 80Table 4.2 Recognition performance of lightweight network on differ-ent feature levels. . . . . . . . . . . . . . . . . . . . . . . . 81Table 4.3 Recognition performance by different decision strategy. . . . 81Table 4.4 Recognition performance after using different preprocessingmethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85Table 4.5 Performance comparison with the state of the art. . . . . . 85Table 5.1 Recognized code patterns and their related classes. . . . . . 107Table 5.2 The numbers of training and testing samples on each class . 109Table 5.3 Comparison with other state-of-the-art object detection meth-ods on the whole testing dataset . . . . . . . . . . . . . . . 113xivLIST OF TABLESTable 5.4 Comparison with other state-of-the-art object detection meth-ods on closed and open classes . . . . . . . . . . . . . . . . 113Table 5.5 Performances of our framework using different placard de-tectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120Table 5.6 The performances of our framework using different coderecognizers . . . . . . . . . . . . . . . . . . . . . . . . . . . 120xvLIST OF FIGURESList of FiguresFigure 1.1 The meaning of a container identification code. The firstthree digits are the owner code which must be a sequenceof uppercase letters. The fourth digit indicates which kindof equipment it is. The category identifier is also an up-percase letter. The next six digits are the serial number toidentify the container of that owner. The last is the checkdigit, which is used to validate the previous ten digits.©2021 IEEE. Reprinted with permission from [1]. . . . . 3Figure 1.2 The meaning of a container type code. The first digitindicates the length of the container, which is 40 feet inthis example. The second digit explains the height. In thiscase, it refers to the high cube container of 9 feet and 6inches. The last two digits are the type code. G1 in thisexample means a general-purpose container with passivevents at the upper part of cargo space. ©Ran Zhang,2022. Diagram created by Ran Zhang. . . . . . . . . . . . 4Figure 1.3 An example of horizontal codes on a shipping containerrear door. ©Ran Zhang, 2022. Image provided by Can-scan Inc [14]. Diagram created by Ran Zhang. . . . . . . . 4Figure 1.4 Markings on a shipping container side panel. ©Ran Zhang,2022. Image provided by Canscan Inc [14]. Diagram cre-ated by Ran Zhang. . . . . . . . . . . . . . . . . . . . . . . 6xviLIST OF FIGURESFigure 1.5 Examples of different kinds of placards. ©Ran Zhang,2022. Image provided by Canscan Inc [14]. Diagram cre-ated by Ran Zhang. . . . . . . . . . . . . . . . . . . . . . . 7Figure 1.6 Thesis outline. ©Ran Zhang, 2022. ©Ran Zhang, 2022.Image provided by Canscan Inc [14]. Diagram created byRan Zhang. . . . . . . . . . . . . . . . . . . . . . . . . . . 13Figure 3.1 Structure of proposed localization model. Feature extrac-tion and up-scaling are based on the existing modules ofU-net and Resnet. We define the loss function for bet-ter localization performance. In post-processing, we de-sign the adaptive score aggregation, and average to max-imum suppression range algorithms for adaptive localiza-tion. ©2021 IEEE. Reprinted with permission from [1]. . 30Figure 3.2 An example of code localization. The feature extractionand up-scaling process propose possible code regions. Butthere are noisy text regions that we do not need. OurASA cleans this noisy text. After ASA, code areas haveoverlapped bounding boxes. Therefore, our AMSR mergesthese overlapped proposals and outputs the minimum andmaximum code boundaries. ©2021 IEEE. Reprinted withpermission from [1]. . . . . . . . . . . . . . . . . . . . . . . 31xviiLIST OF FIGURESFigure 3.3 Illustration of text region proposal process. The localiza-tion model generates four distance maps and one scoremap. Regions with higher confidence levels above thethreshold are proposed. The bounding box location is con-structed from distances. In this example, a bounding boxprediction consists of five elements, i.e., one confidencevalue from the generated score map that is higher thanthe threshold, and four distances to the edges of bound-ing boxes {d1, d2, d3, d4}. ©2021 IEEE. Reprinted withpermission from [1]. . . . . . . . . . . . . . . . . . . . . . . 36Figure 3.4 An example of ranked confidence on score map from lowto high. The x-axis is the ranked order and the y-axisindicates the confidence values. The blue line is the rankedconfidence value and the red line is the increase of thecurrent value compared to the previous value. The greenline is the adaptive threshold for the score map. In thisexample, the whole score map has around 3000 pixels andtheir ranked values are shown in (a). The highest 300confidence are shown in (b). ©2021 IEEE. Reprinted withpermission from [1]. . . . . . . . . . . . . . . . . . . . . . . 37Figure 3.5 Minimum border distance db is used to measure whethertwo bounding boxes are close or not. In this figure, dbis the minimum distance of vertical edges. If db is lessthan min(L1,L2), we consider these two bounding boxesare close horizontally. ©2021 IEEE. Reprinted with per-mission from [1]. . . . . . . . . . . . . . . . . . . . . . . . 40Figure 3.6 Strategy for Adaptive Localization. ©2021 IEEE. Reprintedwith permission from [1]. . . . . . . . . . . . . . . . . . . . 41xviiiLIST OF FIGURESFigure 3.7 Different combinations of localization outputs. The 11-digit shipping container codes in localization procedurecan be divided into three parts in figure (a) when the lastcheck digit is far away from other numbers, or two part infigure (b) when they are close. ©2021 IEEE. Reprintedwith permission from [1]. . . . . . . . . . . . . . . . . . . . 44Figure 3.8 The used CRNN structure in our framework. It takes thecropped code regions as inputs and recognizes the text in-side them. ©2021 IEEE. Reprinted with permission from[1]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Figure 3.9 A bidirectional LSTM layer in CRNN. There are two layersof bidirectional LSTM stacked in CRNN. ©2021 IEEE.Reprinted with permission from [1]. . . . . . . . . . . . . . 47Figure 3.10 An illustration about the data collection scene. There arecameras installed at the gate where the trucks go through.The camera specifications, the distance of the truck tothe gate, as well as the gate sizes, are not fixed in differentdata collection sites. It is ensured that the container codescan be captured by cameras. ©Ran Zhang, 2022. Thediagram is created by Ran Zhang. . . . . . . . . . . . . . . 48Figure 3.11 Localization outputs when the score map threshold is setfixed with 0.01. (a) shows the bounding boxes before merg-ing, and (b) shows predicted maximum boundaries afterAMSR. ©2021 IEEE. Reprinted with permission from [1]. 54Figure 3.12 Localization outputs when the score map threshold is setfixed with 0.99. (a) shows the bounding boxes before merg-ing, and (b) shows predicted maximum boundaries afterAMSR. ©2021 IEEE. Reprinted with permission from [1]. 55xixLIST OF FIGURESFigure 3.13 Localization outputs when the score map threshold is adap-tively chosen by our ASA. (a) shows the bounding boxesbefore merging, and (b) shows predicted maximum bound-aries after AMSR. ©2021 IEEE. Reprinted with permis-sion from [1]. . . . . . . . . . . . . . . . . . . . . . . . . . 56Figure 3.14 Two examples of AMSR outputs. The red bounding boxesare the maximum boundaries and the green bounding boxesindicate the minimum ones. The maximum boundaries setcan give a better localization in Figure (a) because min-imum boundaries cannot cover the whole number area,while minimum boundaries set is more suitable for recog-nition in Figure (b) since maximum boundaries include amore noisy background. The rod in Figure (b) might berecognized as number 1 if maximum boundaries are used.©2021 IEEE. Reprinted with permission from [1]. . . . . . 58Figure 3.15 The comparison of localization performance when IoU thresh-old is set with different values. AP stands for AveragePrecision. ©2021 IEEE. Reprinted with permission from[1]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60Figure 3.16 ROC curve and precision-recall curve in individual char-acter classification using different methods. AUC and APvalues are calculated. ©2021 IEEE. Reprinted with per-mission from [1]. . . . . . . . . . . . . . . . . . . . . . . . 62xxLIST OF FIGURESFigure 4.1 The overall structure of our proposed vertical text spottingmodel. FPN stands for Feature Pyramid Network. RPNis Region Proposal Network. In the detection stage, ourmodel incorporates FPN as the feature extraction back-bone and Cascade R-CNN as the code localization net-work. In the recognition stage, it leverages both extractedfeatures by vertical RoIAlign and original images by rotat-ing cropped images. The decision on word level is madebetween our two recognition networks at the end. ©2021IEEE. Reprinted with permission from [2]. . . . . . . . . . 67Figure 4.2 The illustration of feature sharing in our vertical text spot-ting model. NMS stands for Non-Maximum Suppression.RPN is the network that proposes the possible text re-gions. C and B stand for the task of classification andbounding box position prediction, respectively, with a setof increasing thresholds in the Cascade R-CNN. Firstly,the feature pyramid is used by the detection process thatprovides code areas for vertical RoIAlign. Then text fea-tures are vertically aligned for recognition. Finally, a lightweighttext recognizer is conducted after vertical RoIAlign. ©2021IEEE. Reprinted with permission from [2]. . . . . . . . . . 70Figure 4.3 Structure of the lightweight recognition network. CNNand BiLSTM are used to extract sequence features. RNNwith attention predicts the text sequence. ©2021 IEEE.Reprinted with permission from [2]. . . . . . . . . . . . . . 73xxiLIST OF FIGURESFigure 4.4 Qualitative results of our model in spotting vertical codes.(a) and (c) were captured at night. (b) and (d) were takenduring the day. No cropping or object segmentation isused before the images are sent to our model, i.e., ourmodel takes original collected images as inputs. ©2021IEEE. Reprinted with permission from [2]. . . . . . . . . . 79Figure 4.5 A demonstration of our recognition process. ©2021 IEEE.Reprinted with permission from [2]. . . . . . . . . . . . . . 83Figure 4.6 Qualitative results of prepossessing on cropped code re-gions. (a), (f), and (k) are the results processed by Cannyedge detection. (b), (g), and (l) are results of adaptiveGaussian thresholding. (c), (h), and (m) are results ofadaptive mean thresholding. (d), (i), and (n) are resultsof Otsu thresholding. (e), (j) and (o) are the original verti-cal text images. ©2021 IEEE. Reprinted with permissionfrom [2]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86Figure 4.7 AP-IoU curves of different vertical code detection meth-ods. AP stands for average precision. ©2021 IEEE.Reprinted with permission from [2]. . . . . . . . . . . . . . 88Figure 4.8 Visualization of text spotting models’ outputs. (a) (b) areoutputs of MST and MTSv3, respectively. Detection re-sults are shown in red polygons or bounding boxes. Recog-nition results are with red text. Zoom in for better visu-alization. ©2021 IEEE. Reprinted with permission from[2]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89xxiiLIST OF FIGURESFigure 4.9 Visualization of text spotting models’ outputs. (a) (b) areoutputs of ABCNet and ours, respectively. Detection re-sults are shown in red polygons or bounding boxes. Recog-nition results are with red text. Zoom in for better visu-alization. ©2021 IEEE. Reprinted with permission from[2]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Figure 4.10 Visualization of text spotting models’ outputs. (a) (b) areoutputs of MST and MTSv3, respectively. Detection re-sults are shown in red polygons or bounding boxes. Recog-nition results are with red text. Zoom in for better visu-alization. ©2021 IEEE. Reprinted with permission from[2]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91Figure 4.11 Visualization of text spotting models’ outputs. (a) (b)are outputs of ABCNet and MTSv3, respectively. Detec-tion results are shown in red polygons or bounding boxes.Recognition results are with red text. Zoom in for bettervisualization. ©2021 IEEE. Reprinted with permissionfrom [2]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92Figure 5.1 Proposed framework for hazardous material placard codedetection and recognition. Our designed modules includecode-related confidence boosting, logarithmic weighted con-fidence fusion, and code recognition. ©Ran Zhang, 2022.Images provided by Canscan Inc [14]. Diagram created byRan Zhang. . . . . . . . . . . . . . . . . . . . . . . . . . . 102Figure 5.2 The structure of our optimized placard code recognitionmodel. ©Ran Zhang, 2022. Diagram created by RanZhang. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104xxiiiLIST OF FIGURESFigure 5.3 The training losses of our framework. LWCE is our pro-posed logarithmic weighted cross entropy. ©Ran Zhang,2022. Image provided by Canscan Inc [14]. Diagram cre-ated by Ran Zhang. . . . . . . . . . . . . . . . . . . . . . . 110Figure 5.4 Qualitative results. (a) and (b) show results from YOLOV5sand YOLOV5x, respectively. In the result images, the de-tected placards are illustrated with their bounding boxes.©Ran Zhang, 2022. Image provided by Canscan Inc [14].Diagram created by Ran Zhang. . . . . . . . . . . . . . . . 114Figure 5.5 Qualitative results. (a) and (b) show results from Reti-naNet and Faster R-CNN, respectively. In the result im-ages, the detected placards are illustrated with their bound-ing boxes. ©Ran Zhang, 2022. Image provided by Can-scan Inc [14]. Diagram created by Ran Zhang. . . . . . . . 115Figure 5.6 Qualitative results. (a) and (b) show results from Cas-cade R-CNN and ours, respectively. In the result images,the detected placards are illustrated with their boundingboxes. ©Ran Zhang, 2022. Image provided by CanscanInc [14]. Diagram created by Ran Zhang. . . . . . . . . . . 116Figure 5.7 (a) Precision (b) Recall of each class. ©Ran Zhang, 2022.Diagram created by Ran Zhang. . . . . . . . . . . . . . . . 118Figure 5.8 (a) Average precision (b) f1-score of each class. ©RanZhang, 2022. Diagram created by Ran Zhang. . . . . . . . 119xxivGLOSSARYGlossaryALPR Automated Licence Plate RecognitionAMSR Average to Maximum Suppression RangeAP Average PrecisionARNDR Automated Railcar Number Detection and RecognitionASA Adaptive Score AggregationAUC Area under the ROC CurveBiLSTM Bidirectional Long Short Term MemoryCNN Convolutional Neural NetworksCRNN Convolutional Recurrent Neural NetworkCTC Connectionist Temporal ClassificationDG DangerousLQ Limited QuantityFPN Feature Pyramid NetworkGRCNN Gated Recurrent Convolution Neural NetworkGRU Gated Recurrent UnitxxvGLOSSARYLSTM Long Short Term MemoryLWCE Logarithmic Weighted Cross-EntropyLWCF Logarithmic Weighted Confidence FusionmAP mean Average PrecisionMP Marine PollutantMTS Mask Text SpotterNMS Non-Maximum SuppressionOCR Optical Character RecognitionRNN Recurrent Neural NetworksRPN Region Proposal NetworkSVM Support Vector MachinexxviACKNOWLEDGMENTSAcknowledgmentsI sincerely appreciate all those who helped me during the progress of myPh.D. study. First, I would like to truthfully express my superlative grati-tude to my supervisor, Dr. Zheng Liu, who has been continuously encour-aging, guiding, and supporting me during my knowledge acquisition journey.The thoroughgoing guidance that he offered to me not only assisted me inovercoming academic challenges but also inspired me to dive into the fun-damental relationship of results and causes that could be constructive inconquering other obstacles in my life adventure. In addition, I would like totruly thank my supervisory committee members, Dr. Richard Klukas and Dr.Lo¨ıc Markley. They offered extremely encouraging suggestions and acutelypertinent instructions to my research. Besides, I would candidly acknowledgeDr. Jian Liu, my university examiner, and Dr. Sreeranman Rajan, my ex-ternal examiner, for participating in my defense. Also, I would thank all mycolleagues from the Intelligent sensing, Diagnosis and Prognostics ResearchLab (ISDPRL), at the University of British Columbia, Okanagan Campus.Furthermore, I would acknowledge Mitacs for offering me the precious chanceto participate in the project (No. IT12857), which established the connec-tions between industry and academia. I would appreciate our industrialpartner, Canscan Inc., for providing valuable research data and industrialbackgrounds.Ultimately, I would frankly appreciate my family, who has nourished andsupported me all the way.xxviiDEDICATIONDedicationTo my familyxxviiiCHAPTER 1. INTRODUCTIONChapter 1Introduction1.1 Research Background and MotivationShipping containers, conveying freight without unloading and reloading in-side goods when transportation modes shift among ships, trucks and railways[15, 16], are indispensable to intermodal freight transportation in global sup-ply chain. With large cargo capacities and competitively low freight rates,they are widely used to transport nearly all kinds of materials and productsin logistics services for global manufacturing and trading. As the total num-ber of shipping containers all over the world is estimated between 5 millionand 170 million [17], high-quality container management is expected to easeport congestion and accelerate transportation.Container codes, regulated by international standards, are the fundamen-tal information for container management. Generally, there are three kindsof important codes on shipping containers, i.e., identification codes, typecodes, and safety codes. Reading these codes and keeping results in recordsare preliminary operations in container management.In the past, in order to identify containers, port authorities and trans-portation companies had to train employees to manually check and recordimportant code information on site. But the costs of labor are high, and1CHAPTER 1. INTRODUCTIONthe efficiency of human checking is low. Besides, checkers tend to make mis-takes because of fatigue after being involved in repetitive work. Therefore,automated container code reading systems, integrating a variety of computervision-based technologies and computational methods to automatically ob-tain imaging-based information [18], are now universally required by thetransportation industry to accelerate container checking processes and en-hance code recognition accuracy.Currently, most departments of container management have installedcameras to record the traffic, attempting to identify container codes auto-matically. However, due to various markings on shipping container panelsand versatile outdoor environments, it is challenging for contemporary meth-ods to detect and recognize all the codes comprehensively and accurately.The lack of effective computer vision-based algorithms in spotting containercodes critically impedes the development of automated container code read-ing systems and restricts the quality of container management. As a result,low-quality container management could cause a series of logistics issues thatdelay global trading. For instance, mistakes and omissions in reading safetycodes could even lead to wrong operations and serious catastrophes withextraordinary losses in human lives and economics [19].Therefore, this research, motivated by a strong industry desire for high-quality container management, focuses on designing container code spottingalgorithms to provide practical artificial intelligence-based solutions for in-dustrial challenges. The research outcomes are anticipated to ease supplychain disruptions and contribute to global economic growth.1.2 Shipping Container CodesThe three most important codes for containers are container identificationcode (also known as container number), container type code (ISO containertype and size code) [20], and placard code (safety code). The first two kindsof code are printed either horizontally or vertically at certain positions on2CHAPTER 1. INTRODUCTIONCLHU 854179 1Owner codeCategory identifierSerial numberCheck digitFigure 1.1: The meaning of a container identification code. The first threedigits are the owner code which must be a sequence of uppercaseletters. The fourth digit indicates which kind of equipment it is. Thecategory identifier is also an uppercase letter. The next six digits arethe serial number to identify the container of that owner. The lastis the check digit, which is used to validate the previous ten digits.©2021 IEEE. Reprinted with permission from [1].container panels, while the last category is printed on hazardous materialplacards, which can be attached with uncertain rotated angles at flexiblepositions on container panels. During the image or video collection process,these codes are usually recorded together along with adjacent markings andsigns on shipping containers.1.2.1 Horizontal Container CodesThe horizontal container codes that we focus on in this research are the con-tainer identification code and the type code. A container identification codeis the unique identification of a container, while a type code specifies thesize and weight information of a container. According to ISO 6346 [20], thestandard shipping container identification code is an 11-digit alphanumericcharacter sequence. The meaning of the identification code is illustrated withan example of a horizontal code sequence shown in Figure 1.1. It consists ofthe owner code, category identifier, serial number, and check digit. Differ-ently, the container type code is a shorter 4-digit sequence, with an exampleshown in Figure 1.2.The horizontal container identification code and type code, as the fun-damental information of a shipping container, must be printed on the upper3CHAPTER 1. INTRODUCTION45G1LengthHeightTypeFigure 1.2: The meaning of a container type code. The first digit indicatesthe length of the container, which is 40 feet in this example. Thesecond digit explains the height. In this case, it refers to the highcube container of 9 feet and 6 inches. The last two digits are the typecode. G1 in this example means a general-purpose container withpassive vents at the upper part of cargo space. ©Ran Zhang, 2022.Diagram created by Ran Zhang.Container NumberISO ContainerTypeand Size CodeWeight and Capacity informationSafety Approval PlatePlacardContainer Owner or LessorCaution StikerFigure 1.3: An example of horizontal codes on a shipping container reardoor. ©Ran Zhang, 2022. Image provided by Canscan Inc [14].Diagram created by Ran Zhang.4CHAPTER 1. INTRODUCTIONright part of the shipping container back panel. They might also be printedon the sides and the roof of the container, depending on the container ownerand container type. The horizontal code is not ensured to be shown on pan-els other than the backdoor. Therefore, container managers usually focus onthe back of containers to read horizontal codes.An example of horizontal codes on a shipping container rear door is shownin Figure 1.3. As can be seen, there are many other markings and signs withtext that affect the reading of horizontal container codes. Safety approvalplates with very small text are mandatory for all international shipping con-tainers. It is also mandatory that the container weight and capacity informa-tion is displayed on the backdoor. The caution stickers are optional markingsto show the height warning of containers. Other optional signs include con-tainer owner and lessor logos and hazardous material placards. As trucksloaded with shipping containers are usually moving during the data collec-tion process, these markings and signs are usually unavoidably captured inthe raw images. Distinguishing the container codes from other text becomesone of the challenges for automated container code reading systems.1.2.2 Vertical Container CodesApart from the horizontal style, shipping container identification codes andtype codes can also be printed vertically with the same contents in terms ofthe same container. Horizontal codes play the dominant role during trans-portation as they are the main style of character alignment in text instances.However, horizontal codes, in certain conditions, might not be in the viewof cameras as they are usually printed on the front and back panels. Duringtransportation, horizontal codes are sometimes occluded by trucks and arehard to capture by cameras. On the other hand, when vehicles are movingat normal speed on the roads, it is more convenient to collect vertical codes,which are generally printed on the left and right panels of trailers and con-tainers. It is easier to install cameras off the road to catch container sidepanels than to place cameras above the road to record horizontal codes on5CHAPTER 1. INTRODUCTIONPlacards UN Number Container NumberISO Container Type and Size CodeContainer Owner or LessorCaution Stikeric CodeFigure 1.4: Markings on a shipping container side panel. ©Ran Zhang,2022. Image provided by Canscan Inc [14]. Diagram created by RanZhang.front or back panels. Hence, spotting vertical codes is of significant value toautomated container code reading systems, especially when horizontal codesare incomplete or unavailable.An example of vertical codes on a shipping container side panel is shown inFigure 1.4. Similar to the container rear door, there are also many markingsand signs apart from vertical container codes. The container identificationcode (container number), container type code, and InterContainer (ic) codeare mandatory on side panels, where ic code refers to the company organi-zation that has certified the container. Optional markings include containerowner or lessor signs, caution stickers, hazardous material placards, and UNnumbers. Different from horizontal codes on back doors, the vertical code6CHAPTER 1. INTRODUCTION5.21.1 1.1 1.2 1.3 1.42.1 2.1 2.2 2.2 2.33 3 4.1 4.2 4.35.1 6 7 88 9 MP LQ DGFigure 1.5: Examples of different kinds of placards. ©Ran Zhang, 2022.Image provided by Canscan Inc [14]. Diagram created by Ran Zhang.is aligned from top to bottom. Extracting vertical codes from the rest ofthe text is relatively easier than that process for horizontal codes due to thespecial aspect ratios (width to height ratios) and vertical sizes. However, itcould be more difficult for computers to recognize vertical text, as there aremore horizontal text instances in the real world and less attention has beenpaid to the research of vertical text.1.2.3 Hazardous Material Placard CodesWhile container identification codes and type codes provide the containerinformation, hazardous material placard codes give information on dangerous7CHAPTER 1. INTRODUCTIONTable 1.1: The meanings of various placardsClass Meaning1.1, 1.2, 1.3, 1.4 Explosives2.1, 2.2, 2.3 Gases3 Flammable liquids4.1, 4.2, 4.3 Other flammable substances5.1, 5.2 Oxidizing substances & organic peroxides6 Poisonous substances7 Radioactive materials8 Corrosives9 Miscellaneous hazardous materialsMP Marine pollutantDG DangerousLQ Limited quantityfreight inside containers. Automatically detecting and recognizing placardcodes ensures the safety of container transportation.Hazardous materials can be indispensable components of widely usedproducts in global manufacturing and trading. For instance, lithium bat-teries, as a kind of dangerous good, are the key parts of electronic products.Transporting hazardous materials must be paid with extra attention sinceleakage and explosion may cause enormous catastrophes [19], disrupting theglobal supply chain and halting trading. Reading the safety signs correctlycan guide the proper operations and arrangements, accordingly, reducing therisk of accidents. Hazardous material placards are crucial safety informationsigns that are required to be displayed on the shipping containers during thetransportation of hazardous freight [21]. They clearly explain the proper-ties of dangerous goods. As shown in Fig. 1.5, there are various kinds ofplacards. Some of them indicate explosives, and some describe flammableproperties. The detailed meanings of each class are explained in Table 1.1.To read the placards efficiently and prevent human checkers from workinga long time near hazardous materials, the computer vision-based automated8CHAPTER 1. INTRODUCTIONplacard recognition system is seriously in demand for dangerous freight man-agement. It could meaningfully ensure the safety of transportation, as wellas the high efficiency of global supply chain management.There are a few challenges in spotting placard codes. Firstly, the placardscould be placed at any position on a shipping container, while the shippingcontainer number and type code are displayed at relatively fixed positions.Secondly, the total number of placards on the container is not certain. Therecould be no placards or a few of them as the container might transport a fewkinds of dangerous freight together. Thirdly, the text size of placard codesis relatively smaller so they are easier to be occluded or deformed, addingmore difficulties to the recognition process.1.3 Research GapsAlthough the research on text detection, recognition, and spotting has maderapid progress, there are research gaps in spotting container codes, includinghorizontal and vertical shipping container codes, as well as placard codes.1.3.1 Spotting Horizontal Shipping Container CodesThe previous shipping container code detection and recognition methods [22–25] perform effectively in conditions with abundant illumination and clearbackgrounds. However, there are still a few research gaps.• The previous code detection models [26–28] cannot distinguish con-tainer codes from the noisy text in complex backgrounds. In realindustry environments, the distances of the camera to the shippingcontainers, as well as illumination, change greatly. In addition, thereare other text affecting code detection. For instance, license plates oftrucks and caution stickers on the backdoor of shipping containers areconsidered to be noisy text when the task is only detecting the con-tainer number and type code. It is difficult for previous methods todistinguish the container codes from the noisy text in these situations.9CHAPTER 1. INTRODUCTION• The conventional shipping container code detection and recognitionmodels are separate parts. The code detection model outputs the fixedareas of shipping container codes, which cannot be adjusted. Theseareas are cropped and directly used as the inputs of the code recognitionmodel. There is no feedback from the recognition part to the detectionpart. These traditional open-loop frameworks lead to relatively lowaccuracy in the code recognition procedure.1.3.2 Spotting Vertical Shipping Container CodesAlthough the state-of-the-art scene text spotting models [28–30] have madesignificant enhancements in terms of processing arbitrary-oriented scene text,they are deficient in analyzing vertically aligned shipping container codes ineither detection or recognition for the following reasons:• In traditional text spotting models, the features of text regions are notstrictly vertically aligned for recognition, restricting the performanceof vertical code recognition. Conventional text spotting models usuallypool text features from left to right for recognition by the RoIAlignprocess. Even though they could handle curved text, they still assumethe text direction is from left to right of the character. In contrast, thetext direction goes from top to bottom in vertical codes. Pooling textfeatures in the wrong direction would confuse the recognition networks.• The single recognition pipeline-based text spotting model [30] mightnot be applicable to all conditions. The reason is that the featuremaps are usually shared between detection and recognition networksso they are not specially extracted for recognition.• When there are multiple recognition pipelines in a text spotting model[28, 29], it is not optimal to select preferred results only by predictionprobabilities. The recognition output with higher probability might10CHAPTER 1. INTRODUCTIONalso have higher uncertainty, which could be a symbol of inaccurateresults.1.3.3 Spotting Hazardous Material Placard CodesContemporary placard code detection and recognition methods [19, 31–33]show feasible computer vision-based solutions in industrial applications. How-ever, the following restrictions need to be lifted:• Traditional visual information-based methods [19, 31–33] rely heavilyon handcrafted features, including frequency-domain features, edges,and colors. They can only recognize a small number of placard codeclasses, for example, 6 categories in [19].• Prior visual information-based methods, including traditional methodsand deep learning-based methods [34–36], are restricted by the imbal-anced and open class datasets, i.e., they have limited performance onrarely seen classes and cannot recognize unseen classes.• Textual information-based methods [28–30] have difficulties in directlydetecting placard codes, which usually have very small sizes and are eas-ily affected by noise (including other text and complex environments).1.4 Research ObjectivesThe overall objective of this research is to leverage machine vision and deeplearning-based techniques to detect and recognize the important codes onshipping containers so as to improve the efficiency of intermodal freight trans-portation, where the efficiency can be considered in two folds, processingspeed and accuracy. Improving the processing speed would increase effi-ciency. Achieving higher recognition accuracy would also contribute to theimprovement of efficiency as it would reduce the frequency of human checkingand correction, which could lead to a longer waiting time for all trucks.The detailed research objectives are as follows.11CHAPTER 1. INTRODUCTION• This research will build a framework to spot horizontal shipping con-tainer numbers and type codes, with the capability of removing noisytext instances and adaptively fine-tuning predicted code locations dur-ing the recognition process.• This study will design a vertical text spotting model that exploitsthe vertical features of shipping container codes to enhance end-to-endrecognition performance.• This work will spot hazardous material placard codes by fusing thetextual and visual information to overcome the challenge of imbalancedand open class datasets, which are very common in industrial data.1.5 Thesis OutlineThe outline of this thesis is shown in Fig. 1.6. Chapters 3, 4, and 5 presentour designed container code spotting methods for horizontal codes, verticalcodes, and placard codes, respectively. This thesis contains six chapters,which are summarized as follows:• Chapter 1 introduces the research background and motivation, as wellas the the meaning and categories of shipping container codes. Researchgaps and objectives are also expressed in this chapter.• Chapter 2 reviews the state-of-the-art text detection, text recognition,as well as text spotting methods. The contemporary approaches ofspotting horizontal and vertical container codes, and placard codes arealso reviewed. Other automated code reading systems in transportationindustry are introduced as well.• Chapter 3 presents the proposed horizontal code spotting method, i.e.,an adaptive deep learning framework, which consists of two principal12CHAPTER 1. INTRODUCTIONHorizontal codedetection withadaptive stragetyVertical code detection Placard detectionUpper rear veiw camerasSide view camerasSide view camerasHorizontal coderecognitionStandardrecognitionLightweightrecognitionResults selectionVisualinformationextractionTextualinformationextractionFusionVertical codes Placard codesChapter 3 Chapter 4 Chapter 5Horizontal codesImagesChapter 1:introductionChapter 2:literaturereviewChapter 6:conclusionsAutomated container code reading systemsImage collection modulesFigure 1.6: Thesis outline. ©Ran Zhang, 2022. ©Ran Zhang, 2022. Imageprovided by Canscan Inc [14]. Diagram created by Ran Zhang.parts, horizontal code detection and recognition. The adaptive local-ization strategy is designed in the first part to provide adjustable lo-cations for code recognition. Experiments are conducted on horizontalcontainer code datasets to compare the proposed framework with otherstate-of-the-art methods. The experimental results validated that ouradaptive framework achieves best recognition accuracy.• Chapter 4 illustrates the proposed vertical text spotting model for ver-tical container codes. In this model, two recognition pipelines are in-corporated and the results are chosen by our fusion strategy basedon both probabilities and uncertainties. The experimental results on13CHAPTER 1. INTRODUCTIONvertical code datasets showed that our model outperforms other state-of-the-art methods for end-to-end code recognition.• Chapter 5 describes our zero-shot framework for placard code spotting.It fuses visual and textual information with our designed LogarithmicWeighted Confidence Fusion (LWCF) strategy to enhance the perfor-mance in closed classes and initiate the performance in open classes. Inaddition, the proposed Logarithmic Weighted Cross-Entropy (LWCE)loss alleviates the class imbalance in training data. Experiments thathave been conducted on real industrial images demonstrated the supe-riority of our methods.• Chapter 6 draws conclusions and summarizes the whole research. Thestrengths and potential applications for this research are also given. Inaddition, the limitations and future research directions are discussed.14CHAPTER 2. LITERATURE REVIEWChapter 2Literature ReviewThis chapter provides a comprehensive literature review of computer vision-based scene text detection and recognition methods, horizontal and verticalcontainer code detection and recognition approaches, placard code detectionand recognition approaches, as well as other automated code reading systemsin the transportation industry.2.1 Computer Vision-Based Scene TextDetection and Recognition MethodsScene text is the text that appears in natural scenes, for example, ship-ping container codes are the scene text instances captured during intermodalfreight transportation. Detecting and recognizing scene text can be sepa-rated, i.e., scene text detection and recognition, or integrated into scene textspotting. Optical Character Recognition (OCR), also known as text detec-tion and recognition, is the process of detecting text instances and read-ing the text contents. OCR techniques convert text in images to text inmachine-readable formats which can be stored and processed in computers.Automated OCR techniques are superior to human checking, which is ex-pensive and inefficient, especially during transportation when there is a large15CHAPTER 2. LITERATURE REVIEWamount of code detection and recognition tasks with limited processing time.Great improvements have been made in document OCR research [37] wherethe text is surrounded by relatively unified backgrounds, whereas it is morechallenging to conduct OCR on scene text, which is the text that appearsin images captured in various outdoor and indoor environments. The scenetext instances are usually with different arrangements of characters, in vari-ous backgrounds, and under illumination variations. They are captured dayand night with backgrounds in different colours and brightness under diverseweather conditions. Consequently, the text might be blurry or partially oc-cluded by rain, snow, or dirt.Computer vision-based scene text detection and recognition methods pro-vide great value to automated container code recognition systems. Initially,the traditional image processing skills, including filtering [38], contour de-tection, and image binarization were used to extract handcrafted featuresbefore text detection and recognition. Recently, scene text detection andrecognition with the emerging deep learning techniques in computer vision,have attracted more attention in research and provided practical solutions forindustrial applications [39], since deep neural networks have more powerfulfeature extraction abilities than the traditional filters when processing imageswith more variations [40]. A lot of research about detecting and recognizingscene text has been conducted [26–30, 41–44].2.1.1 Scene Text DetectionScene text detection is to find the text instances in images. The output of thescene text detection process can be either bounding boxes or masks on thetext regions. Traditionally, text features, such as stroke width [45] [46] andmaximally stable external regions [47], were leveraged to localize the scenetext. However, these methods can only detect the scene text with regularshapes. Recently, various deep learning models have been proposed to de-tect scene text with arbitrary shapes or orientations. These models providedifferent ways from traditional methods that rely on handcrafted feature ex-16CHAPTER 2. LITERATURE REVIEWtraction. In scene text detection, deep learning methods [26, 48] were firstused to generate quadrilaterals to localize the arbitrary-oriented word. Then,the curved text has also been studied [27, 49]. The connectionist text pro-posal network [50] connects sequential text proposals and detects multi-scaleand multi-lingual text in a single process, which avoids further post filteringor refinement. EAST [26], which is based on the fully convolutional neuralnetwork, is capable of detecting text with different angles. TextBoxes++[48] uses quadrilateral representation to localize the arbitrary-oriented word.PSENet [27] leverages feature pyramid networks [51] to extract features forscene text. With progressive scale expansion, PSENet is able to detect textwith arbitrary shapes. Aiming to achieve real-time processing, a scene textdetection model with differentiable binarization was proposed by Liao [49] tolocalize scene text with the speed of 82 fps and f-measure of 0.79 in MSRA-TD500 dataset [46].2.1.2 Scene Text RecognitionScene text recognition is to read the characters inside text instances. Theoutput of scene text recognition is the machine-readable digital informationof the text content. Traditional OCR systems [52] was dedicated to readingthe text in documents, which is more regularly arranged in orientation andshape than the scene text. Deep learning-based models, which ignore thenoisy background and read text by sequence, perform better in scene textrecognition. Horizontal text without great rotation was first addressed bydeep neural networks [41, 42]. After that, text rectification was used [43, 44]to align the text horizontally in the image for the convenience of recognition.Arbitrary-oriented text recognition has been studied [53] as well. The issueof vertical text recognition has been considered in [54] but it distinguishesvertical and horizontal only by aspect ratio, which is not suitable for rotatedor irregular text. Convolutional Recurrent Neural Network (CRNN) [41] canrecognize text sequences of arbitrary length. Gated Recurrent ConvolutionNeural Network (GRCNN) [42] chooses the context information dynamically17CHAPTER 2. LITERATURE REVIEWby adding a gate to the recurrent convolutional layer to recognize text in nat-ural images. CRNN and GRCNN can only read the text horizontally withoutgreat distortion. Direction encoding mask and selective attention network-based model [54]was proposed to simultaneously recognize horizontal andvertical text but it did not apply to tilted text or curved text. ASTER [44]can rectify the tilted or curved text to be horizontal and then read the con-tent. AON [53] was designed to recognize arbitrary oriented text but it stillneeds to resize the input image to 100×100 square. VGG and ResNet-basedscene text recognition models were compared by Baek et al. [43] and it wasfound that using ResNet [55] in feature extraction [56] [57] gains a higheraccuracy but consumes more time than utilizing VGG [58].2.1.3 Scene Text SpottingText spotting models aim to simultaneously detect and recognize the text,focusing on refining the overall structure design and improving the perfor-mance. The outputs are the positions and content of the text in images.Designing the scene text spotting model is of more practical value than con-centrating on only text detection or recognition models since that industryputs more attention on the overall performance. The text spotting was firststudied in the text with the horizontal direction. Liu et al. [59] proposed anend-to-end text spotting model with convolutional recurrent neural networksto spot horizontal text with 1.1 fps on the image size of 600 × 800. Then,the arbitrarily oriented text spotting was studied. FOTS [60] simultaneouslydetected and recognized the oriented scene text but did not perform wellin curved text as it generated the quadrilateral bounding boxes. Recently,text with arbitrary shapes, especially curved text, was taken into consid-eration [28–30, 61, 62]. MTS [28], an end-to-end trainable neural networkfor text spotting, can handle irregular text, including arbitrarily orientedtext and curved text, with dual recognition pipeline. The unconstrainedend-to-end text spotting model [61] was also designed to spot text in arbi-trary shapes. ABCNet [30] with parameterized Bezier curve was reported18CHAPTER 2. LITERATURE REVIEWto spot the arbitrary-shaped text with the processing speed of 22.8 fps onCOCO-Text [63] dataset. PGNet [62], a single shot text spotter withoutnon-maximum suppression and region proposal operations, runs with 46.7FPS on Total-Text [64] dataset.2.2 Horizontal Shipping Container CodeDetection and Recognition MethodsThe complex and volatile environments become significant challenges whenthe models of scene text detection and recognition, as well as scene textspotting, are applied to intermodal freight transportation. Numerous re-search regarding detecting and recognizing horizontal container codes duringintermodal freight transportation has been conducted. Automated shippingcontainer code detection and recognition frameworks usually consist of twoindividual steps, code localization, and recognition.2.2.1 Non-deep Learning MethodsTraditional non-deep learning methods were first studied. For code localiza-tion, traditional methods use features of strokes, contours, gray levels [65],edges, and histogram information from grayscale images. Filters [22] andmasks [66] are also used to extract the text area features. Code localizationis followed by code recognition. Traditional code recognition methods aredivided into two steps, character isolation, and single character recognition.A commonly used technique in character isolation is connected componentsanalysis [67] which can find separated characters. When it comes to single al-phanumeric character recognition, image features, including histograms, gra-dients, and edge densities generated by the Sobel filter, are taken as inputsof the classifiers. Previously used classifiers include Support Vector Machine(SVM), and adaptive resonance theory network [24]. Single character recog-nition can also be implemented by template matching [38] [68] which can findsimilar patterns with the templates. He et al. [38] use the median filter and19CHAPTER 2. LITERATURE REVIEWadaptive linear filter to select the code lines, and adopt template matchingfor code recognition. Kim et al. [24] use Sobel masking to detect edges andlocalize code, then propose an 8-directional contour tracking mask to extractfeatures for code recognition. As character U is the category identifier forthe freight containers, Kumano et al. [22] search for the location of U tolocalize the container code. Wu et al. [23] design a horizontal high-pass filterand scanline analysis strategy for code localization, and implement verticalprojection on the histogram for character isolation, then use SVM for singlecharacter recognition.Conventional non-deep-learning-based code detection and recognition meth-ods rely heavily on manually designed filters to extract the features of codeareas [22–25, 33, 65, 69]. These filters are restricted by their generalizationability, i.e., they can only be used in images that are captured in specificconditions or scenarios.2.2.2 Deep Learning MethodsDeep learning-based methods do not rely on handcrafted features. Thus, theycan be used in more diverse environments. Since that container codes arespecial kinds of scene text, a lot of deep learning models are introduced fromgeneral text processing to shipping container code localization and recogni-tion. Mei et al. [25] combine template matching and convolutional neuralnetworks for container code character recognition. Verma et al. [70] leveragespatial transformer networks to recognize the single characters. Roeksukrun-grueang et al. use LeNet [71] for code recognition. Liu et al. [72] combineconnectionist text proposal network and maximally stable extremal regionsfor code localization, and use both character segmentation and CRNN forcode recognition.Although contemporary deep learning-based text detection and recogni-tion methods [26–30, 41, 42] are applicable in automated horizontal codereading systems, they are usually designed for general text. In other words,they ignored the specialties of shipping container codes, thus they might be20CHAPTER 2. LITERATURE REVIEWaffected by the noisy text in intermodal freight transportation. Due to thegreat diversity of the text and backgrounds, the previous methods are notable to deal with all of them with both high accuracy and fast speed. There-fore, this research aims to design efficient and robust container code spottingmethods which can be applied seamlessly in versatile industrial environmentsto detect and recognize important codes on shipping containers.2.3 Vertical Shipping Container CodeDetection and Recognition MethodsEven though vertical container codes provide the same information as hor-izontal codes do, vertical container code reading still remains a challengingtopic due to the vertical character alignment, which is a huge difficulty fortext recognition. Similar to the procedures to process horizontal codes, theautomated vertical code reading also consists of two steps, code detection andrecognition. They can be conducted using either non-deep learning-based ordeep learning-based methods.2.3.1 Non-Deep Learning MethodsWhen accomplished by traditional non-deep learning-based methods, verti-cal code detection can reuse the localization methods for horizontal codesby changing the expected aspect ratios of code regions. Thus, selecting thehandcrafted features by traditional image processing techniques, includingthresholding [73], filtering [23], edge detection [74], gray level extraction [65],as well as connected component analysis [23, 70], can also be applied in verti-cal code detection. Regarding the vertical recognition procedure, traditionalnon-deep learning methods usually segment the characters first and classifythem [23, 75]. Previous work of automated container code reading systems[23, 71, 75] claimed that their methods can be applied to both horizontaland vertical codes. However, there was no experiment conducted to validatetheir performance exclusively on vertical codes.21CHAPTER 2. LITERATURE REVIEW2.3.2 Deep Learning MethodsApart from these non-deep learning methods, deep learning-based methodsalso provide solutions for automated vertical code detection and recognition.In terms of the detection procedure, deep learning-based scene text detec-tors [26, 27, 76] and object detectors [35, 77], without any modification, canbe used to detect vertical container codes, which can be treated as text in-stances or objects. When it comes to vertical code recognition, however, thecropped vertical code images are required to be rotated by 90 degrees beforebeing used as the inputs of the deep learning-based scene text recognizers[41, 42, 54]. They treat the characters in rotated images as horizontal symbolsand still read from left to right. With the advances in deep learning tech-niques, the recent arbitrary oriented text spotting methods [28–30] provideintegrated approaches to detect and recognize the vertical codes simultane-ously. ABCNet [30] could align vertical features by fitting them with Beziercurves. MTS [28] and MTSv3 [29] could leverage the attention mechanismto automatically segment and recognize the single characters with the wholeimage as an input. However, they are designed for general text instances,where the horizontal style plays a dominant role in text alignment. Theyconduct either shape transformation [30] or single character spotting [28, 29]to handle vertical text instances. Therefore, more effective approaches areanticipated in this research to directly extract and make good use of thevertical text features.2.4 Hazardous Material Placard CodeDetection and Recognition MethodsHazardous material placards could be placed at any position of the shippingcontainer doors and it is not required to attach them to containers withouthazardous freight. Thus, whether they appear or not and where they stick areuncertain, which challenges the placard detection and recognition methods.22CHAPTER 2. LITERATURE REVIEWPrior placard detection and recognition methods can be divided into vi-sual information-focused methods and textual information-focused methods.2.4.1 Visual Information-based MethodsVisual information-focused methods detect and recognize the whole placards.They can be further categorized into traditional feature extraction-basedmethods and deep learning-based methods.Traditional Feature Extraction-based MethodsTraditional methods extract handcrafted features that are invariant underrotating, scaling and illumination changing. These custom-built featuresinclude Pascal Triangle [78], convex quadrilateral [79], sharpness intensityvariation and rhombus shapes [19]. Before extracting features, commonlyused image pre-processing techniques, including image binarization, Cannyedge detection [19, 74, 79], and contour detection [19, 79], are often applied.After feature extraction, the placard regions could be isolated by settingthresholds on these features. These regions are cropped for the recognition,which is accomplished by the colour recognition in multiple channels [79], orartificial neural network-based classification [19].Due to the design of handcrafted feature extraction, traditional methodscan be only applied to a very limited number of categories, e.g., 6 placardclasses in [19].Deep Learning MethodsDeep learning-based object detection methods can be directly applied toplacard detection and recognition. They usually integrate object detectionand classification into end-to-end trainable neural networks. Generally, theycan be divided into one-stage and multi-stage detectors. Methods in theformer category, e.g., YOLO series [35, 80–82], and RetinaNet [83], conductthe inference with fewer steps, thus, running faster. But they might sufferfrom performance decreases in terms of precision and recall. Methods of23CHAPTER 2. LITERATURE REVIEWthe latter category usually contain multiple stages to refine the detectionresults. For example, Faster R-CNN [34] build two stages in the predictionprocess, and Cascade R-CNN [36] employs more than two stages. As a result,they tend to achieve better results in precision and recall, but fall behind inprocessing speeds.Although these deep learning-based object detection methods have maderemarkable progress in many applications, their performances are affected byimbalanced and open class data.2.4.2 Textual Information-based MethodsTextual information-focused methods detect and recognize the placard codes.They can be implemented by building text spotting models, which detect andrecognize the text sequences in an integrated framework [28–30, 84]. In textspotting models, the first step is to detect text, where some object detectionor instance segmentation methods are applied or modified to isolate the textinstances, treating them as the target objects. For example, the Mask R-CNN [85] is integrated into MTS [28] to generate masks of text instances,while RetinaNet [83] is modified in ABCNet [30] to detect text. The secondstep is to recognize the text inside the isolated regions. Text recognizercould either share the extracted features with the text detector [2, 28–30] orbe separated with the detection part.The limitation of textual information-focused methods is that they re-quire unique printed placard codes, which might not be available in somecategories.2.5 Other Automated Code ReadingSystems in Transportation IndustryText detection and recognition are also widely used in many other automatedcode reading systems in the transportation industry, for instance, AutomatedLicence Plate Recognition (ALPR). Plenty of research has been conducted in24CHAPTER 2. LITERATURE REVIEWthe field of ALPR. In traditional methods, ALPR can be divided into threeconsecutive subtasks, i.e., license plate localization, character segmentation,and single character recognition. Traditional methods designed image filtersor feature extraction manually for license plate localization and charactersegmentation. Edge detection-based methods [86–88] were used to detectthe license plate. Vertical projections of histograms after image binarization[89] were used to find the gap between characters. Vertical and horizontalprojections can also be combined to localize the license plate [90]. The singlecharacter recognition was usually accomplished by classification models inmachine learning, such as artificial neural networks [90] and SVM [91]. Deeplearning-based methods are able to integrate a few subtasks into one model.Zhuang et al. [92] proposed the semantic segmentation-based model to detectcharacters then predict the character sequence. The character isolation andprediction were integrated. Moreover, deep learning has a stronger abilityto handle various backgrounds. Silva et al. [93] proposed an unconstrainedmodel to recognize license plates with oblique views from different regions orcountries. SNIDER [94] integrated text denoising and rectification into theframework. It was constructed on YOLOv3 [80] and showed the real-timeprocessing speed of 37 fps with the accuracy of 0.9308 on their VTLP [94]dataset.Besides ALPR, Automated Railcar Number Detection and Recognition(ARNDR) systems are also of great value to transportation. Railcars are oneof the most important means of transportation on land. They usually carrya large amount of freight at relatively low costs. The speed of railcars onthe railway might go up to 50 mph [95] in Canada. Efficiently detecting andrecognizing the railcar number contributes greatly to logistic management.ARNDR methods using computer vision techniques could accurately andefficiently process the video frames or images that are taken when railcarstravel fast. Contemporary ARNDR methods [96–98] focus on detecting andrecognizing single characters. In [96], the input images are first converted to25CHAPTER 2. LITERATURE REVIEWgray-scale images, then morphological image processing is used to enhancethe images and detect the railcar number regions. It also segments everycharacter and needs an artificial neural network to classify these characters.The image colour information is lost and the morphological image processingcannot be applied to diversified environments of railcars. In [97], Faster R-CNN [34] has been used to detect railcar numbers. Then median filter andbinarization have been used to find the gap between characters, which isdifficult when characters are blurry or get connected by dirt. In [98], theinstance segmentation model, Mask R-CNN [85] is conducted to segmentthe railcar tanker from the background. But there is no need to segmentrailcars from the background for the sake of ARNDR. Although the deeplearning model, namely, improved EAST [99], has been applied to detectrailcar numbers on the word level, it has not conducted research on railcarnumber recognition.26CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESChapter 3Spotting Horizontal ShippingContainer CodesThis chapter presents the research on spotting horizontal shipping containercodes. As horizontal container codes are the mandatory unique identifiers ofshipping containers, recognizing these codes is an essential step to managethe containers and logistics. The conventional code localization methods caneasily be interfered with by varied noises and cannot identify the best regionsfor code recognition. In this chapter, we propose an adaptive deep learningframework for shipping container code localization and recognition. In theframework, the noisy text regions will be removed by an adaptive score ag-gregation (ASA) algorithm. The code region boundaries are identified by theaverage to maximum suppression range (AMSR) algorithm. Thus, the pre-dicted locations can be adjusted within this range to fit the code recognitionmodel to achieve higher accuracy. The experimental results of the compar-ative study with the state-of-the-art models, including EAST [26], PSENet[27], GRCNN [42], and MTS [28], demonstrated the proposed frameworkachieved better localization performance and obtained 93.33% recognitionaccuracy. The processing speed reaches 1.13 fps, which is sufficient to meetthe operational requirements. Thus, the proposed solution will facilitate27CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESthe digital transformation of shipping container management and logistics atports.3.1 OverviewAn adaptive deep learning framework is proposed for horizontal shippingcontainer code localization and recognition. Different from traditional auto-mated container code recognition methods, the predicted locations of con-tainer codes in our adaptive framework can be adjusted to the best posi-tions for the recognition model. ResNet [55] is a widely used backbone fordeep feature extraction and U-Net [100] is a neural network structure thatcould exploit multi-scale features. Therefore, our horizontal code localizationmodel is built based on ResNet and U-Net to extract deep and multi-scaletext features on text instances with various sizes and background. Our recog-nition model is based on CRNN [41], a widely used scene text recognizer. Theinputs of the localization model are images containing the shipping containerbackdoor, and the outputs are ranges of possible code regions. Then, CRNNis used to recognize codes in cropped code areas. After the code recognitionstep, we check whether the recognition results meet the ISO 6346 interna-tional shipping container standard [20]. If the results meet the requirement,the process of recognizing codes in this image ends with valid container codes.Otherwise, the predicted locations of shipping container codes should be ad-justed according to our designed strategy.3.2 An Adaptive Deep Learning Frameworkfor Horizontal Code Detection andRecognitionThe proposed framework consists of two main parts, i.e., horizontal containercode detection and recognition. As our designed adaptive strategy is to adjustdetected bounding boxes of codes, it is included in the code detection part.28CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES3.2.1 Horizontal Shipping Container Code DetectionThe localization model includes the feature extraction process, up-scalingprocess, and post-processing steps. Our designed ASA and AMSR algorithmsare in post-processing steps. The localization model structure is shown inFigure 3.1. One example of the localization process can be seen in Figure 3.2.We adopt Resnet [55] as the backbone in the feature extraction process. Inthis process, the size of feature maps gets smaller and smaller to learn higher-level features. The next process is the up-scaling process when feature mapsget larger and larger. Outputs of a few feature extraction layers are usedas a part of the inputs of the concatenation layers in the up-scaling process.The up-scaling process finally outputs the one score map channel and fourdistance map channels. The score map indicates the confidence, or possibilityof every pixel location being in the shipping container code areas, while thedistance maps measure the distances of every pixel to the boundaries of codeareas.After the up-scaling process, the post-processing is leveraged to get theboundaries of shipping container codes. We propose the ASA algorithm forscore map cleaning to remove low confidence detection, as well as wrong pre-dictions on noisy text. Thus our model can focus only on the most confidentpart, which is the backdoor shipping container code area. After that, AMSRmerges the rest overlapped bounding boxes that have relatively high confi-dence. The outputs of the localization model are the minimum and maximumboundaries of possible locations of the container codes.Feature Extraction and Up-ScalingIn our code localization model, The first step of feature extraction is to resizethe input image according to the requirements of the following layers. Theheight and width of the input images are divisible by 32 as there are onepooling layer and four convolutional layers with a stride of 2 during thefeature extraction process. The parameters, for these layers, are given in29CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESConv(7×7, 2, 64)Pool(3×3, 2, 64)Resnet50 block 1 Resnet50 block 2Resnet50 block 3Resnet50 block 4ResizeBilinear scaling ×2ConcatConv(1×1,1,128)Conv(3×3,1,128)Bilinear scaling ×2ConcatConv(1×1, 1, 64)Conv(3×3, 1, 64)Bilinear scaling ×2ConcatConv(1×1, 1, 32)Conv(3×3, 1, 32)Conv(3×3, 1, 32)Conv(1×1, 1, 5)Adaptive score aggregationAverage to maximum suppression rangePost-processingFeature extractionImageScore map 1 channel distances 4 channelsMinimum andmaximumboundaires of of codes areasUp-scalingExisting modules from Resnet and Unet Our modulesFigure 3.1: Structure of proposed localization model. Feature extractionand up-scaling are based on the existing modules of U-net and Resnet.We define the loss function for better localization performance. Inpost-processing, we design the adaptive score aggregation, and aver-age to maximum suppression range algorithms for adaptive localiza-tion. ©2021 IEEE. Reprinted with permission from [1].30CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESPropose possible code regions Clean noise by our ASAMerge bounding box and calculate range by our AMSRFigure 3.2: An example of code localization. The feature extraction andup-scaling process propose possible code regions. But there are noisytext regions that we do not need. Our ASA cleans this noisy text.After ASA, code areas have overlapped bounding boxes. Therefore,our AMSR merges these overlapped proposals and outputs the mini-mum and maximum code boundaries. ©2021 IEEE. Reprinted withpermission from [1].31CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESTable 3.1: Blocks details in the adopted ResnetBlocksResidual UnitsNumberConvolutional Layersin Each UnitBlock1 3Conv(1×1,1,64)Conv(3×3,1,64)Conv(1×1,stride,256)Block2 4Conv(1×1,1,128)Conv(3×3,1,128)Conv(1×1,stride,512)Block3 6Conv(1×1,1,256)Conv(3×3,1,256)Conv(1×1,stride,1024)Block4 3Conv(1×1,1,512)Conv(3×3,1,512)Conv(1×1,1,2048)Figure 3.1. For instance, conv(7×7,2,64) means that the kernel size is 7×7with stride 2, and the output feature maps dimension is 64. The detailedlayers of Resnet blocks are listed in Table 3.1. We set the stride with 2 atthe last layer of Block 1, Block 2, and Block 3. The strides in other layersare set with 1. The concatenation layers merge two sets of feature maps intoone set.Bi-linear scaling is adopted in the process of up-scaling. The computingprocess is as follows. First, the scale ratios are calculated by(3.1)sx = (w1 − 1)/(w0 − 1)(3.2)sy = (h1 − 1)/(h0 − 1)where sx and sy are the horizontal and vertical scale ratios, respectively.w1, h1 are the width and height of scaled image height. w0, h0 are the originalimage width and height. If w1 = 0, sx would be set to 0. If h1 = 0, sy32CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESwould be set to 0. Second, in order to calculate the new value at (i, j), thecorresponded four pixels (a, b), (c, b), (c, d), (a, d) in original image are foundby following equations:(3.3)xa = ⌊i/sx⌋(3.4)ya = ⌊j/sy⌋(3.5)xb = ⌈i/sx⌉(3.6)yb = ⌈i/sx⌉Finally, the new pixel value Mˆ(i,j) at position (i, j) in the scaled image isdecided by weighting four pixels (xa, ya), (xb, ya), (xb, yb), (xa, yb) in originalimage by following equation:(3.7)Mˆ(i,j) = (1− γx)(1− γy)M(xa,ya) + γx(1− γy)M(xb,ya)+ γxγyM(xb,yb) + (1− γx)γyMxa,ybwhere M(xa,ya),M(xb,ya),M(xb,yb),M(xa,yb) are pixel values at original positions(xa, ya), (xb, ya), (xb, yb), (xa, yb), respectively. The horizontal weights γx andvertical weights γy are computed as follows:(3.8)γx = i/sx − ⌊i/sx⌋(3.9)γy = j/sy − ⌊j/sy⌋As shown in Figure 3.1, the outputs of the up-scaling process are fourdistance channels and one score map channel. In the training process, thescore map values are set to 1 inside the bounding box of the labelled text area,and other locations are set to 0. In the predicting process, the score mapvalues should be less than or equal to 1. Values in the score map indicate theprobabilities, or confidence of the pixels being in text areas. With the scoremap and distance channels, the predicted text areas and their confidence canbe calculated.33CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESIn the training process, the loss function L of the localization network isdefined as the following equations.L = λsLs + Ld (3.10)where L includes score map loss Ls, and distances loss Ld. λs is the weightfactor for Ls and it is set to 0.01, and use the dice coefficient loss functionfor Ls.Ls =1− 2 ·∑(P ⊙ T )∑T +∑P + 10−5(3.11)where P is the matrix of predicted score map, T is the matrix of targetscore map, and ⊙ is the element-wise product. ∑P and ∑T are the sumof all elements in matrix P and T , respectively. When both P and T arewith value 0, i.e., no text in target score map and predicted score map, thedenominator could be prevented to fall to value 0 by adding an extremelysmall value 10−5.Distance loss function Ld consists of two parts as follows.Ld = − ln IoU(A,A∗)− ln proj(A,A∗) (3.12)where A is the predicted code area, and A∗ is the ground truth code area. Onthe right side of this equation, the first part is the loss for the Intersection overUnion (IoU), and the second part is the loss for the projection of predictedarea to the ground truth area in the horizontal direction. The former trainsthe model to predict more overlapping with ground truth code regions, whilethe latter contributes to the output’s horizontal coverage. The IoU andhorizontal projection can be calculated by the following equations.(3.13)IoU(A,A∗) =A ∩ A∗A ∪ A∗=(min(d2, d∗2) + min(d4, d∗4))(min(d1, d∗1) + min(d3, d∗3))(max(d2, d∗2) + max(d4, d∗4))(max(d1, d∗1) + max(d3, d∗3)34CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESproj(A,A∗) =min(d2, d∗2) + min(d4, d∗4)d∗2 + d∗4(3.14)where d1, d2, d3, d4 represent values of four distance channels in predictionresults, that is, distance to the top, right, bottom and left boundaries of thepredicted text area. And d∗1, d∗2, d∗3, d∗4 are the distances in ground truth. Ifone pixel (x, y) is inside a text area [x1, y1, x2, y2], these four distance can becalculated as follows.d1 = |y − y1| (3.15)d2 = |x2− x| (3.16)d3 = |y2− y| (3.17)d4 = |x− x1| (3.18)where (x1, y1), (x2, y2) are the left-top and right-bottom coordinates of theground truth bounding box, respectively. Even though our designs in lossfunction try to make the predicted horizontal coverage a little bit wider thanthe actual text area as we only consider d2 and d4 in equation 3.14, this kindof prediction is more helpful in the following text recognition part since itcan include the codes completely.After the feature extraction and up-scaling, the network proposes possibleshipping container code regions with score map values, i.e., confidence values.The text region proposal process is illustrated in Figure 3.3.Post-Processing I, Adaptive Score Aggregation on Score MapIn the post-processing for shipping container code localization, we proposeASA to remove the low confidence regions as well as noisy areas. ASA cal-culates an adaptive threshold and aggregates confidence in adjacent regions.Different from threshold methods in image segmentation [101] which filter thepixels in images, our adaptive threshold in ASA focuses on the confidencelevels of possible text areas in the score map. Thus, the adaptive threshold35CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESthresholdConfidence levelWidthLengthGeneratescore mapGeneratedistancemapsd1 d2d3d4d1d2d3d4Figure 3.3: Illustration of text region proposal process. The localizationmodel generates four distance maps and one score map. Regions withhigher confidence levels above the threshold are proposed. The bound-ing box location is constructed from distances. In this example, abounding box prediction consists of five elements, i.e., one confidencevalue from the generated score map that is higher than the thresh-old, and four distances to the edges of bounding boxes {d1, d2, d3, d4}.©2021 IEEE. Reprinted with permission from [1].is used to remove the low-confidence text proposals. Next, bounding boxesthat draw the boundaries of text in adjacent areas are aggregated. Mean-while, the scores of bounding boxes are accumulated in this area. At last,the set of bounding boxes that are close to each other and have the highestaggregation scores are selected as the outputs of ASA.If the threshold for score map cleaning is set with a high value, some areaswith codes will be cut off. But if the threshold is low, many false-positivepositions will be added to the final localization outputs. Most score valuesin the score maps are at low levels as shown in Figure 3.4a, and only a smallportion of them are relatively high. Therefore, the threshold in our proposed36CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESThresholdIncrements00.20.40.60.81.0Confidence value0 500 1000 1500 2000 2500 3000The order of rankingRanked Confidence(a) Ranked confidence of whole score map.ThresholdIncrements00.20.40.60.81.0Confidence value0 50 100 150 200 250 300The order of rankingRanked Confidence(b) Ranked highest 300 confidence values.Figure 3.4: An example of ranked confidence on score map from low to high.The x-axis is the ranked order and the y-axis indicates the confidencevalues. The blue line is the ranked confidence value and the red lineis the increase of the current value compared to the previous value.The green line is the adaptive threshold for the score map. In thisexample, the whole score map has around 3000 pixels and their rankedvalues are shown in (a). The highest 300 confidence are shown in (b).©2021 IEEE. Reprinted with permission from [1].37CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESASA is set adaptively according to the distribution of the score map values.ASA consists of two steps, adaptive threshold calculation, and score ag-gregation. In order to calculate the adaptive threshold, scores (or confidencevalues) are ranked from low to high. The score map is a two-dimensionalmatrix corresponding to the confidence of the text areas. We reshape thismatrix into a one-dimensional vector and rank them from low to high. Theseranked scores (confidence values) in acceding order on the score map are de-noted as S⃗ = (s1, s2, s3, ..., sn), and n is the length of this vector, which isequal to the total number of pixels in this score map. The value changes inthe ranked scores can be defined as R⃗ = (r1, r2, r3, ..., rn), where r1 = s1, andrk = sk − sk−1 if 2 ≤ k ≤ n. Then, the index I of the inflection point can befound by the following equation:I = max(argkmax(rk)), 1 ≤ k ≤ n (3.19)where I ∈ [1, n]. If there are multiple values of argkmax(rk), the maximumindex k would be chosen.The threshold h on the score map is set between 0.2 and 0.8 to avoidextremely high or low values. Therefore, the adaptive threshold h can becalculated as follows:h =0.8, if sI ≥ 0.8sI , if 0.2 < sI < 0.80.2, if sI ≤ 0.2(3.20)where sI is the value at the inflection point I on score map. Then, the scoreswhich are larger than or equal to the threshold h are considered as relativelyhigh confidence. They are kept for processing in the next step. Scores thatare less than h are removed before the next process. Figure 3.4 gives anexample of finding the adaptive threshold. The red line indicates the valuechanges compared to the previous score value. The maximum value of rk38CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESis the maximum value of the red line on the y-axis, and the index I of theinflection point is the x-axis value at that point. Finally, we can get thethreshold value from the blue line at index I. Scores of recent areas areaggregated after the threshold. The set of bounding boxes with the highestaggregation score will be selected. The score aggregation process is shown inthe below process.1. Sort the bounding boxes by their left-top coordinates from high to low.2. Select the first bounding box.3. Put the selected bounding box bbj into a new area set ASi. The rangeof this new area set Ri is the same as this bounding box area rj.4. Select the next bounding box. If all bounding boxes are calculated, goto step 6. Otherwise, go to step 5.5. If the selected bounding box bbm is close to one of the existing area setrange Rn, put it into this area set ASn, and expand the area range Rnwhich can cover the newly included bounding boxes. Otherwise, go tostep 3.6. Aggregate the scores of all bounding boxes in every area set. Outputthe area set ASh with the highest aggregation score.The minimum border distance db is used to measure whether two areasor bounding boxes are close or not. In the horizontal direction, db is definedas follows.db = min(d11, d12, d21, d22) (3.21)where dij is the horizontal distance of vertical edge i in one bounding boxto vertical edge j in another bounding box. If db is less than the widthof the smaller area as shown in Figure 3.5, we consider they are close inthe horizontal direction. In the same way, we can judge whether the two39CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESdbL1L2Figure 3.5: Minimum border distance db is used to measure whether twobounding boxes are close or not. In this figure, db is the minimumdistance of vertical edges. If db is less than min(L1,L2), we con-sider these two bounding boxes are close horizontally. ©2021 IEEE.Reprinted with permission from [1].bounding boxes are close in the vertical direction. Only if they are close inboth horizontal and vertical directions, they are considered close in our ASAprocess. The ASA process finally outputs bounding boxes in code regionswith high confidence. The reason why we independently judge distances inhorizontal and vertical directions instead of directly computing the distanceof two bounding boxes is that the latter method cannot easily define whethertwo boxes are close or not. For example, if we compute Euclidean distanceof two boxes, it is hard to find the threshold with a clear physical meaningto judge whether these two boxes are close. In contract, if two directions arejudged independently, we can clearly define this threshold with the minimumvalue of L1 and L2 as shown in Figure 3.5.Post-Processing II, Merging Overlapped Detected Areas by UsingAverage to Maximum Suppression RangeAfter the low confidence and noisy bounding boxes are removed by ASA,there are still some overlapped predictions or bounding boxes that shouldbe merged in the code regions. Non-Maximum Suppression (NMS) is a tra-ditional and commonly used post-processing method to handle overlappedbounding boxes. However, NMS only chooses the most confident detectionresult and discards other overlapped bounding boxes with lower confidence.40CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESMaximum boundaries setMinimum boundaries setMean of maximum andminimum boundaries setSelect most two confidentbounding boxesCRNN recognize codeSelect next bounding boxfrom ranked setNoYesWhether results meet ISO 6346NoYesOther boxes in current set leftChoose other left boundarysets from AMSRRank the bounding boxesby confidence in thisboundary setAverage to MaximumSuppression RangeEnd with validcodesYesNoOther boundary set leftEnd with novalid codesCode RecognitionCode localizationFigure 3.6: Strategy for Adaptive Localization. ©2021 IEEE. Reprintedwith permission from [1].The final bounding boxes of the detected area after NMS might be smallerthan the ground truth. Therefore, we propose the AMSR algorithm to com-pute the minimum and maximum boundaries, which gives the range of pos-sible text areas. Compared with standard NMS, AMSR can provide bettertext localization areas for the next text recognition step. The recognitionmodel can search for suitable regions from AMSR and then get better perfor-mances. Our proposed AMSR in one possible text area includes two boundingboxes and their confidence [Ba, Bm, conf ], where Ba = (xa1, ya1 , xa2, ya2), andBm = (xm1 , ym1 , xm2 , ym2 ). Ba and Bm indicate the minimum boundaries andmaximum boundaries of the AMSR, respectively. We use weighted merge[26]instead of the minimum merge to decide the minimum boundaries. (x1, y1) isthe top-left coordinate of the bounding box and (x2, y2) is the right-bottomcoordinate. The proposed AMSR is shown in algorithm 1.After computing the AMSR in the post-processing part, the localizationmodel gives the minimum and maximum boundaries of container codes with41CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESAlgorithm 1: AMSR1 Function AMSR (B,C);Input : A list of bounding boxes B after score map cleaning, andtheir confidence COutput: minimum and maximum boundaries Ba, Bm, and theirconfidence Cout2 Sort B by confidence from high to low3 while B ̸= ∅ do4 bm0 ← pop out the highest confident bounding box from B5 conf0 ← pop out the highest confidence from C6 ba0 ← bm07 lb ← length(B)8 for j ← 0 to lb do9 if IoU(bm0 , bj) > 0.3 then10 conf0 ← max(conf0, confj)11 ba0 ← conf0 ∗ ba0 + confj ∗ bj12 bm0 ← maxmerge(bm0 , bj)13 remove bj from B14 append ba0, bm0 , conf0 into Ba, Bm, Couttheir confidence.Strategy for Adaptive LocalizationAn adaptive localization strategy is designed in our framework. This strategycan take advantage of the maximum and minimum boundaries from AMSR toprovide multiple choices for the recognition model. In our adaptive strategy,we give three possible boundary sets for AMSR, that is, maximum boundaryset, minimum boundary set, and mean boundary set which is the average ofmaximum and minimum boundaries. More boundary sets can be designedfor the code recognition part. For instance, the mean of maximum boundaryset and mean boundary set. Our adaptive framework is expected to gethigher recognition accuracy provided with more boundary sets. If one of theseboundaries set cannot get the required shipping container codes, our adaptive42CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESframework will automatically choose the next boundary for container coderecognition. In our experiments, we only give three boundary sets to avoidthe possible endless loop. The adjustment strategy of localization can beseen in Figure 3.6There are a few bounding boxes in one boundary set. These boundingboxes outline different parts of the container code. For example, In themaximum boundary set, the bounding boxes outline the maximum regionsof container codes. Then the code recognition model is used to recognize thetext inside these bounding boxes. The results are finally merged into a codesequence.3.2.2 Horizontal Shipping Container CodeRecognitionCode Recognition and Sequence CombinationOur adaptive deep learning framework aims to detect and recognize the 11-digit shipping container code. The outputs of our code localization consistof three bounding boxes if the last check digit number is far away from theprevious numbers as Figure 3.7 (a). But when the check digit is too closeto the number on their left side, the 7-digit numbers will be put into onebounding box in the localization outputs as shown in Figure 3.7 (b).Therefore, the recognition results in cropped areas might be 4 digits, 6 dig-its, 7 digits, or 1 digit according to the localization predictions. These resultsare combined into an 11-digit sequence. In the results combination process,the recognition results in cropped regions are ranked by their localizationconfidence from high to low. Then, the recognition results with the highestand second-highest confidence are combined into a sequence. If this sequencedoes not meet the ISO 6346 standard, the next confident bounding box areais taken into consideration for the combination until the combined sequencemeets the standard. Otherwise, if the recognition results of all boundingboxes in a certain set of boundaries, for instance, maximum boundaries, can-43CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES(a) The 11-digit sequence is in three parts.(b) The 11-digit sequence is in two parts.Figure 3.7: Different combinations of localization outputs. The 11-digitshipping container codes in localization procedure can be divided intothree parts in figure (a) when the last check digit is far away fromother numbers, or two part in figure (b) when they are close. ©2021IEEE. Reprinted with permission from [1].44CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESnot get the results that meet the ISO 6346 standard, the boundary set wouldbe replaced by the next boundary set. In this way, our adaptive frameworkbuilds an interactive process between the code localization and recognitionas shown in Figure 3.6.CRNNCRNN [41] are used to recognize a sequence of text without splitting theinput images into single-character images. Moreover, when we recognizecodes by CRNN, there is no need to remove the rectangle box surroundingthe check digit.CRNN has advantages over other models. GRCNN [42] has gated re-current convolution layers where the gates control the context modulation.Its receptive fields are reduced compared with CRNN [41]. With reducedreceptive fields, GRCNN is able to concentrate on the current single charac-ter and eliminate the effects of other pixels during the recognition process.The performance of GRCNN can be improved if the context is not useful fortext recognition. However, when it comes to container code recognition, thecontext provides plenty of information for the container code recognition asshown in Figure 1.1. Therefore, CRNN with larger receptive fields performsbetter. MTS [28] is an end to end trainable neural network utilizing seman-tic segmentation. But it still needs to separate characters one by one beforecharacter recognition. When the characters are too close to each other, itis hard for characters separation. especially when container codes are visu-ally connected to each other caused by the illumination changing and snowcovering.CRNN model consists of three parts, sequence feature extraction by Con-volutional Neural Networks (CNN), sequence scoring by Recurrent NeuralNetworks (RNN), and sequence translation by Connectionist Temporal Clas-sification (CTC)[102]. The structure of CRNN is shown in Figure 3.8 andFigure 3.9.In CRNN, deep convolutional neural networks are used to learn the char-45CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESFeatures Features Features FeaturesCTC 6 4 2 4 7 2 0Feature extractionby CNNStacked two layers of bidirectional LSTMAggregateFigure 3.8: The used CRNN structure in our framework. It takes thecropped code regions as inputs and recognizes the text inside them.©2021 IEEE. Reprinted with permission from [1].acteristics of the inputs and generate the feature sequence. The RNN partconsists of two stacked Bidirectional Long Short Term Memory (BiLSTM),and every BiLSTM has two layers of Long Short Term Memory (LSTM)[103]. One of them is forward and another is backward. Both layers have 256LSTM units separately. Finally, the concatenation layer is used to combinethe RNN outputs and CTC is utilized to find the most likely alphanumericcharacter sequence, i.e., the predicted results.The training loss function L of CRNN is defined as follows.46CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESLSTM LSTM LSTM LSTM...LSTM LSTM LSTM LSTM...Concat Concat Concat Concat...x1 x2 x3 xny1 y2 y3 ynFigure 3.9: A bidirectional LSTM layer in CRNN. There are two layers ofbidirectional LSTM stacked in CRNN.©2021 IEEE. Reprinted withpermission from [1].L = −∑y∗∈Y ∗,y∈Yln p(y∗|y) (3.22)where Y ∗ is the ground truth label sequences of input images and Y is thecorresponding predicted results.3.3 Experimental Results and Discussion3.3.1 Data CollectionExperimental images have been collected from gates where trucks with ship-ping containers must pass through in the ports. Consequently, the distanceof the truck to the camera was changing during collection and the code sizesin collected images could be different. An illustration of the data collection47CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESFigure 3.10: An illustration about the data collection scene. There arecameras installed at the gate where the trucks go through. Thecamera specifications, the distance of the truck to the gate, as wellas the gate sizes, are not fixed in different data collection sites. Itis ensured that the container codes can be captured by cameras.©Ran Zhang, 2022. The diagram is created by Ran Zhang.scene is shown in 3.10. Different port terminals might have different specifi-cations for gate sizes and distances of cameras to trucks. In our experiment,to capture the horizontal codes, the cameras were put in the back-top direc-tion of shipping containers. To consider as many outdoor environments aspossible, the images were collected under different weather conditions andillumination conditions by conducting the collection process all year roundwhen the port terminal was open, from 6 am to 12 am.In our training and testing data, the 11-digit shipping container codeareas are completely included in the images. The dataset used for code lo-calization contains 3000 images, and the training data ratio is 0.8. Aftercropping the code regions, we have 23418 cropped images for the code recog-nition dataset. The target shipping container should not be too far awayfrom the camera, otherwise, the code would be too small to read. There48CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESmight be some containers which are far away from the camera but are stillin the image. They are considered as background. In order to validateour model in various environments around the shipping container, we takeimages in different illuminations and weather conditions into considerationwhen preparing the dataset. For illuminations, images are collected in boththe daytime with sunlight and the night when extra lighting is added. As forweather conditions, the ports need to keep the recognition system runningall year round. Our dataset needs to cover all 12 months, so we randomlychoose 250 images of shipping containers from every month. All the data areRGB images with the height of 1080 and width of 1920. Another factor takeninto consideration in our dataset is the font size. The trucks with containerswere moving at different speeds when they passed the gate with cameras.And the images in our dataset are randomly selected. So the distances ofshipping containers to the camera are different in our dataset. This leads tovarious font sizes of container codes.We labelled the shipping container code areas with bounding boxes andthe actual text in these areas. The 11-digit shipping container number mightbe included in two or three bounding boxes. The first 4 digits are upper casecharacters, which are usually in one bounding box. The remaining 7 numbersare separated into two parts if the check digit is far away from the other 6digits. Otherwise, these 7 digits are put into the same bounding box. For onebounding box, we give the label of the left-top, right-bottom coordinates, aswell as the text inside this bounding box. The 4-digit container type code isusually below the 11-digit shipping container code. We also label this by onebounding box because the text in this area has a similar size and color tothat of the unique 11-digit shipping container number. This kind of labelingstyle conforms to the localization outputs shown in Figure 3.7.49CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES3.3.2 Network Parameters for Training, andEvaluation Metrics for TestingThe code localization deep neural networks, and CRNN are trained sepa-rately, then they are put together in an interactive process of code local-ization and recognition in the testing, or predicting procedure. Both the11-digit shipping container code and 4-digit type code are put into trainingdata.In the training process of our localization model, the Adam optimizer[104] and decayed learning rate ld are used.(3.23)ld = l0 × 0.94s/10000where l0 is the initial learning rate, which we set it as 0.0001, and s is thecurrent training step. In the training process of CRNN, all the text insideone bounding box is taken as a whole target. There is no need to separateevery single character and number. The text areas are cropped according tothe bounding boxes and used as the input training images for CRNN. Wealso take the Adam optimizer but keep the learning rate steady to be 0.0001.The input image to CRNN should be resized to fit the model. We set theinput height as 32, and keep the aspect ratio when resizing. Different IoUthresholds are used when we are evaluating the localization performance. Ahigher IoU threshold requires the localization model to fit the ground truthbetter. The hyperparameters of our framework are summarized in Table 3.2.To evaluate the performance in the testing process, we set a few metricsfor localization and recognition. From the top-rear direction of a shippingcontainer, we might see the codes on the backdoor and roof simultaneously.We do not consider the localization predictions of codes on the roof, which aremore likely to be corroded or sheltered by snow. Roof codes are not as clearand compact as those on the backdoor and they are not guaranteed to bedisplayed. If the codes on the roof area are considered, the code localizationand recognition models will be affected. So our code localization model is50CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESTable 3.2: Hyperparameters in our frameworkHyperparameter Value/ContentUpper limit of adaptive threshold on score map 0.8Lower limit of adaptive threshold on score map 0.2IoU threshold for merging bounding boxes 0.3Training data ratio 0.8Initial Learning rate for localization model 0.00001Learning rate for recognition model Steady at 0.00001The height of resized inputs of CRNN 32Layers of bidirectional LSTM in CRNN 2IoU threshold used to evaluate performance0.5, 0.55, 0.56,0.60, 0.70, 0.75trained to ignore predictions on the roof of the shipping container.We choose precision, recall, f1-score, and Average Precision (AP) to eval-uate the performance of code localization. In the field of object detection,AP is almost equivalent to Area Under Curve (AUC). We adopt the 11-pointinterpolated AP defined in PASCAL visual object classes challenge [105].And accuracy is used for code recognition. Only when the whole predicted11-digit shipping container code sequence is the same as the ground truth,we consider it correct. In industry, the port managers also require thatthe whole sequence should be correct, otherwise they cannot get the correctunique identification for the shipping container even when only one digit iswrong. The evaluation metrics are defined as follows:(3.24)p =TPTP + FP(3.25)r =TPTP + FN(3.26)f1 = 2p · rp+ r51CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES(3.27)AP =111∑r∈Vrpinterp(r),Vr = {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}(3.28)acc =NcorrectNtotalwhere p stands for precision, r stands for recall, f1 stands for f1-score, AP isaverage precision, acc stands for accuracy. TP is true positive, indicating thatthere prediction successfully detected the code. TN is true negative, whichmeans no predictions on non-text area. FN is false negative, which indicatesthat no prediction is made where there is a code. FP is false positive, i.e.,predicting a box where there is no code. Ncorrect is the number of correctlypredicted sequences, and Ntotal is the number of total testing images. Vr isthe set of r that is used in the calculate of interpolated precision pinterp(r).AP is computed by pinterp(r), which is defined as:(3.29)pinterp(r) = maxrˆ≥r(p(rˆ))where p(rˆ) is the measured precision at recall rˆ. If the confidence thresholdfor the predicted bounding boxes is set to be 1, there will be only falsepredictions and no positive predictions. Therefore, the rˆ would be 0 and p(rˆ)is defined as 1. Consequently, the measured precision is 1 at recall 0. Withthe increase of the threshold, there might be more false positives and lessfalse negatives. rˆ would increase and p(rˆ) would decrease. Ultimately, whenthreshold is 0, rˆ is 1 and p(rˆ) is 0. pinterp(r) is the maximum value of p(rˆ)when rˆ >= r. Finally, AP is calculated using pinterp(r).3.3.3 Model OptimizationOptimization of Threshold Methods for Score Map CleaningScore map gives the confidence, or possibilities, of the code presence in cor-responding positions in the map. Some false positive predictions have high52CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESconfidence because they have similar features to the ground truth shippingcontainer code. Although using a threshold can remove the low confidencepredictions, they cannot discern high confidence distractions. As shown inFigure 3.11, 3.12, and 3.13, if we clean the score map with a fixed threshold,the low threshold method would give some false positive predictions on thebackground, while the high threshold method might not include all the codeareas.In order to validate the performance of our proposed ASA method onthe score map, fixed thresholds in different values are set in comparativeexperiments as shown in Table 3.3. In the fixed threshold method, scores, orconfidence values, which are above the fixed threshold are selected and othersare removed from the score map. Our proposed ASA method only selectsthe set of scores in a specific area with the highest aggregation score afterthe adaptive threshold. Our ASA is followed by AMSR, which outputs theminimum to maximum area range of codes. The performance of thresholdmethods should be evaluated after merging overlapped bounding boxes sincethreshold methods influence the location predictions. Therefore, the mergedbounding boxes are compared with the ground truth bounding boxes bycalculating the precision, recall and f1-score. If the IoU of the predicted areaand the ground truth is not less than the IoU threshold, we consider it tobe true positive. As shown in Table 3.3, our proposed ASA method achievesthe highest precision, f1-score and AP. Although the recall of our method islower, the performance is very close.As the visualization of the score map in Figure 3.3, and ranked confidencevalues in Figure 3.4, the confidence levels are mostly in two ranges, i.e., arelatively high range, which is above 0.8, and a relatively low range whichis below 0.2. p, r, f1, AP stand for precision, recall, f1-score, and averageprecision, respectively. As shown from Table 3.3, the highest precision isobtained by setting the threshold at 0.9, while the best recall is achievedwith a threshold of 0.3. Empirically, most regions with confidence below 0.253CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES(a)(b)Figure 3.11: Localization outputs when the score map threshold is set fixedwith 0.01. (a) shows the bounding boxes before merging, and (b)shows predicted maximum boundaries after AMSR. ©2021 IEEE.Reprinted with permission from [1].54CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES(a)(b)Figure 3.12: Localization outputs when the score map threshold is set fixedwith 0.99. (a) shows the bounding boxes before merging, and (b)shows predicted maximum boundaries after AMSR. ©2021 IEEE.Reprinted with permission from [1].55CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES(a)(b)Figure 3.13: Localization outputs when the score map threshold is adap-tively chosen by our ASA. (a) shows the bounding boxes beforemerging, and (b) shows predicted maximum boundaries after AMSR.©2021 IEEE. Reprinted with permission from [1].56CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESTable 3.3: The comparison of different threshold methods on score mapThreshodMethodp r f1 AP0.1 0.9417 0.9397 0.9406 0.89190.2 0.9433 0.9401 0.9417 0.89230.3 0.9467 0.9414 0.9440 0.89290.4 0.9462 0.9405 0.9433 0.89230.5 0.9453 0.9369 0.9425 0.89150.6 0.9457 0.9369 0.9427 0.89150.7 0.9461 0.9369 0.9429 0.89150.8 0.9474 0.9397 0.9435 0.89150.9 0.9482 0.9401 0.9442 0.8920Our ASA 0.9482 0.9401 0.9442 0.8952are noise or background, and regions with confidence above 0.8 are the text.We set the threshold adaptive between 0.2 and 0.8 for every case. There-fore, different inputs might get different thresholds. Our adaptive thresholdmaintains a part of the text proposals with confidence between 0.2 and 0.8.After merging the overlapped proposals, the code regions get higher confi-dence in our method, which contributes to the higher AP and benefits thecode recognition process.Optimization of Strategies for Merging Overlapped BoundingboxesThe way how the overlapped bounding boxes are merged would affect therecognition model. If the merging method includes a larger area, there mightbe some noisy text or background. If the merging method only gives a verylimited area, some parts of shipping container codes might not be covered.Therefore, we use final recognition accuracy to compare our proposed AMSRmethod to the other three merging methods as shown in Table 3.4. Maximum57CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESTable 3.4: Recognition performance comparison of different mergingmethod for overlapped bounding boxesMerging MethodsOverall RecognitionAccuracyStandard NMS 0.8567Weighted merge 0.8950Maximum merge 0.9067Mean of weighted and maximum merge 0.9133Our AMSR 0.9333(a) (b)Figure 3.14: Two examples of AMSR outputs. The red bounding boxesare the maximum boundaries and the green bounding boxes indi-cate the minimum ones. The maximum boundaries set can give abetter localization in Figure (a) because minimum boundaries can-not cover the whole number area, while minimum boundaries set ismore suitable for recognition in Figure (b) since maximum bound-aries include a more noisy background. The rod in Figure (b) mightbe recognized as number 1 if maximum boundaries are used. ©2021IEEE. Reprinted with permission from [1].merge is to find the union of the overlapped bounding boxes. These mergingmethods are applied in the post-processing steps of the code localization part,followed by the same trained CRNN recognition model.The difference of our AMSR method and previous methods lies in thatprevious localization methods only give fixed bounding boxes, while theAMSR gives the maximum and minimum boundaries of the text area as58CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESTable 3.5: Localization performance when IoU threshold is set with 0.75MethodsLocalization Performancep r f1 APPSENet [27] (2019) 0.6833 0.7763 0.7269 0.5720EAST [26] (2017) 0.9274 0.9119 0.9196 0.8838MTS [28] (2021) 0.9033 0.9230 0.9131 0.8850Ours 0.9482 0.9401 0.9442 0.8952Table 3.6: Recognition performance and processing speedMethods Recognition Accuracy FPSPSENet [27] (2019) + GRCNN [42] (2017) 0.7233 2.16PSENet [27] (2019) + CRNN [41] (2017) 0.7283 2.21EAST [26] (2017) + GRCNN [42] (2017) 0.8033 2.78EAST [26] (2017) + CRNN [41] (2017) 0.7683 3.01MTS [28] (2021) 0.9017 2.40Ours 0.9333 1.13shown in Figure 3.14. Therefore, the predicted bounding boxes of codes inour adaptive framework can be adjusted inside this range. We traverse thepossible bounding boxes using CRNN until it gets the expected recognitionresults. Experimental results show that our proposed AMSR with adjustingstrategy achieves higher final recognition accuracy. AMSR can provide betterlocalization outputs to the CRNN-based recognition model in our adaptiveframework.3.3.4 Comparison with the State-of-the-Art SolutionsFor comparison, PSENet [27] and EAST [26] are trained separately by thesame training data. CRNN and GRCNN [42] are used in code recognition forcropped areas. MTS [28], an end-to-end trained text localization and recog-nition model, is also trained on our dataset for comparison. Experimental59CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES0.5 0.6 0.7 IoUthreshold0.50.60.70.80.91AP(a)PSENet EAST MaskTextSpotter Ours0.5 0.6 0.7 IoU threshold0.70.80.91F1-score(b)0.5 0.6 0.7 IoU threshold0.70.80.91Precision(c)0.5 0.6 0.7 IoU threshold0.80.91Recall(d)Figure 3.15: The comparison of localization performance when IoU thresh-old is set with different values. AP stands for Average Precision.©2021 IEEE. Reprinted with permission from [1].60CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESresults in Figure 3.15, Table 3.5 and Table 3.6 show that our proposed adap-tive deep learning framework achieves the best localization performance andhighest recognition accuracy. p, r, f1, AP stand for precision, recall, f1-score,and average precision, respectively.Figure 3.15 compares the localization performance when the IoU thresh-old is set with different values in the evaluation process. When the IoUthreshold is set with 0.5, more predictions can be considered as true pos-itives. When this threshold increases, the evaluation metrics have higherrequirements for the localization. But when the IoU threshold is higher than0.75, it would be too strict for the localization. Therefore, we evaluatedthe performance with the IoU threshold setting with values between 0.5 and0.75. When the IoU threshold is set at 0.5, EAST, MTS and our frameworkhave similar performance in AP, f1-score, precision and recall. Performanceof PSENet decreases more than other models, which means it tends to havelarger, smaller or slightly deviated predictions. When the IoU threshold in-creases, our framework’s performance degrades less than other models sinceour predictions have more overlapping with the ground truth. This also ben-efits the code recognition process. Having predicted better code location, ourframework archives the higher recognition accuracy.The Area under the ROC Curve (AUC) and Average Precision (AP) forindividual character classification of different methods have been comparedin Figure 3.16. Our method has the best AP and comparable AUC comparedwith other methods. The AUC values for individual character classificationusing different methods are all above 0.99. In our character classificationprocess, the number of total classes is 36, including 10 numbers and 26 up-percase letters. For an individual character, the expected outputs, i.e., thetrue condition, are 1 condition positive and the rest 35 conditions negative.Only 1 condition positive corresponds to the real character label. The finalfalse positive rate of multiclass classification is the average of false positiverates of all classes. In the prediction process, the classifiers with high dis-61CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODES0.0 0.2 0.4 0.6 0.8 1.0False positive Rate0.00.20.40.60.81.0True positive RatePSENet+CRNN (AUC = 0.9920)PSENet+GRCNN (AUC = 0.9935)EAST+CRNN (AUC = 0.9953)EAST+GRCNN (AUC = 0.9973)MaskTextSpotter (AUC = 0.9983)Ours (AUC = 0.9978)(a) ROC curve.0.0 0.2 0.4 0.6 0.8 1.0Recall0.00.20.40.60.81.0PrecisionPSENet+CRNN (AP = 0.9183)PSENet+GRCNN (AP = 0.9229)EAST+CRNN (AP = 0.9581)EAST+GRCNN (AP = 0.9665)MaskTextSpotter (AP = 0.9665)Ours (AP = 0.9704)(b) Precision recall curve.Figure 3.16: ROC curve and precision-recall curve in individual characterclassification using different methods. AUC and AP values are cal-culated. ©2021 IEEE. Reprinted with permission from [1].62CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODEScriminability assign one class with high possibility and the rest 35 with lowconfidence, leading to a low false positive rate and high AUC. The perfor-mance with AUC is not differentiable in this scenario. Therefore, the APis calculated to eliminate the effects of a low false positive rate. In our ex-periments, the performance is compared in both metrics. Noticeably, thedifferences in AUC and AP values for single character classification are notas large as the differences in code recognition accuracy using different meth-ods. The reason is that ports consider code recognition as a successful caseonly if all characters in the whole code sequence of a container are correctlyrecognized.3.3.5 Processing speed of code recognition at portsThe processing speed must be considered when the method is deployed inports. In deep learning models, the computational complexity of a fully con-nected layer is O(ni−1ni), where ni is the number of nodes in ith layer. Thecomputational complexity of a convolutional layer is O(twhf), where t is thetotal number of images, w, h are the width and height of the input image,and f is the number of feature maps in this layer. When the deep learningmodel is built, the number of feature maps f in every convolutional layer andthe neuron nodes in fully connected layers, as well as the number of layers,are fixed. Different layers are stacked to form the deep learning architectureand the data are processed layer by layer. Therefore, the computational com-plexity of the deep convolutional neural networks in the feedforward processis O(twh). All the methods in Table 3.6 are with the complexity of O(twh).Consequently, the processing speed, FPS, is more generally used to measurethe computational complexity for deep learning methods as shown in Table3.6.The processing speeds on testing data have been compared in Table3.6. All the experiments were conducted on a single computer with a singleGeForce GTX 1070 8GB GPU, and an Intel Core i7-8750H CPU @ 2.20GHz.When the ports’ regulations about trucks [106, 107] are taken into con-63CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESsideration, our method satisfies the requirements of processing speed. Thesize of the semi-trailer truck which loads the shipping container should havea minimum length of 9.25 m [108]. The recommended safe distance betweentrucks is 6 m [107]. The speed limit is 10 km/h when the truck passes thegates in ports [106]. Therefore, the minimum time of one truck passing thegate can be estimated as 5.49 seconds if they follow regulations. In our exper-iments, the processing speed of our framework is 1.13 FPS, i.e., 0.88 secondsper frame. When the truck arrives at the gate, the camera on the gate willbe triggered to take photos or record videos. If 5 frames are sampled duringthe truck’s passing, our processing time would be 4.42 seconds, which is lessthan the time that an individual truck spends on passing the gate. If only 1frame is chosen to recognize the container code, our method only takes 0.88seconds to get the results. It is far less than the truck’s passing time, 5.49seconds. As long as the camera captures legible code images, our methodwith 0.88 seconds of processing time could still finish the recognition processeven if the truck travels at 62 km/h, which is over the speed limit. Ourmethod gains practical value by improving the recognition accuracy underthe condition of satisfying the need for processing speed in the industry.3.4 SummaryThis chapter proposes an adaptive deep learning framework for shipping con-tainer code localization and recognition. The deep neural network for local-ization is constructed based on ResNet and U-net. The code recognitionmodel is based on CRNN. In the code localization process, the proposedadaptive score aggregation algorithm selects aggregated areas with high con-fidence. The noisy text, such as license plates on the truck, and weightinformation on the container panel, are effectively removed. Moreover, thedesigned average to maximum suppression range algorithm provides the max-imum and minimum boundaries for code areas. In the proposed framework,the predicted code locations can be adjusted within these boundaries to64CHAPTER 3. SPOTTING HORIZONTAL SHIPPING CONTAINER CODESfind the most suitable positions for the recognition operation, gaining higherchances to perfectly match the code recognition model. Experimental resultsshow that the proposed adaptive framework achieves better accuracy thanthe state-of-the-art solutions for code localization and recognition. The com-putational speed of our framework does meet the operational requirementat ports, where there is not a critical need for real-time processing. Thus,the proposed framework will empower the digitalization of the logistics forshipping containers and contribute to the global transportation and trading.The contributions of this chapter are summarized as follows:• In the adaptive deep learning framework, we propose an ASA algorithmto clean the low confidence and noisy localization proposals. ASA ag-gregates high-confidence backdoor code areas which are clear and com-plete. Thus, the localization model is able to focus on code regions andignore the noisy text instances.• In this framework, a feedback mechanism is designed between the coderecognition model and the code localization model. Different from thetraditional ACCR framework, the localization outputs in our adaptiveframework can be adjusted in a range given by our designed AMSR al-gorithm. Therefore, the recognition model can achieve higher accuracyby choosing more appropriate input images.65CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESChapter 4Spotting Vertical ShippingContainer CodesThis chapter focuses on spotting vertical shipping container codes. Accu-rately detecting and recognizing vertical codes of containers plays a crucialrole in logistic management during freight transportation, especially in theabsence of horizontal codes. Existing research mainly focuses on developingtext spotting models for texts in the direction from left to right, with or with-out rotation, in curved or straight lines. Vertical code spotting has not beenfully explored yet. Thus, in this chapter, we propose a vertical text spottingmodel for vertical codes. Our model adopts a multi-stage detector for betterlocalization performance. The regional text features extracted by the back-bone network are pooled vertically with vertical RoIAlign to conform withthe code direction for the convenience of recognition. In the recognition part,our model integrates two recognition pipelines that take cropped images andextracted features. The recognition results from two pipelines are selectedby the proposed decision strategy considering uncertainties and probabili-ties. Experimental results show that our proposed model outperforms otherstate-of-the-art text spotting models in detecting and recognizing verticalcodes.66CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESFeaturepyarmidLocalizationnetworkStandardrecognitionLightweightrecognitionDecision makingFeaturesOriginal imageCodelocationsVertical RoIAlignCrop & rotateFPNResultsRPNFigure 4.1: The overall structure of our proposed vertical text spottingmodel. FPN stands for Feature Pyramid Network. RPN is RegionProposal Network. In the detection stage, our model incorporatesFPN as the feature extraction backbone and Cascade R-CNN as thecode localization network. In the recognition stage, it leverages bothextracted features by vertical RoIAlign and original images by rotat-ing cropped images. The decision on word level is made between ourtwo recognition networks at the end. ©2021 IEEE. Reprinted withpermission from [2].4.1 OverviewA vertical text spotting model for vertical shipping container codes is pro-posed in this chapter. The overall structure of the model is shown in Fig-ure 4.1. It consists of two main steps, i.e., vertical code detection, andrecognition. In the code detection stage, Feature Pyramid Network (FPN)[51] first extracts features of the whole image in different levels to establishthe feature pyramid. Then Region of Proposal Network (RPN) [34] with thelocalization network of Cascade R-CNN [36] detects the vertical code text in-stances. The localization is refined by setting a set of increasing thresholds inthe Cascade R-CNN. After vertical code detection, the calculated code loca-tions are mapped to both the feature maps and original images. In the coderecognition stage, there are two recognition pipelines starting with differentinputs and ending with their own recognition results. The final recognitionresult of a text instance is adaptively chosen on the word level from one ofthe pipelines by our decision strategy.67CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES4.2 Vertical Shipping Container CodeSpotting ModelOur vertical code spotting model includes vertical code detection and recog-nition processes. They are described in the following subsections.4.2.1 Vertical Shipping Container Code DetectionVertical code detection is to search for the vertical code regions. The outputsare expected to have high IoU values with the ground truth text. If the codelocalization results include too many background pixels, e.g., when the non-text areas are even larger than the text regions, the noise might hamper therecognition. Oppositely, if the detected code areas are much smaller than theactual text regions, the code recognizer, with only partial text areas as inputs,will not be able to decode the correct text sequences. Cascade R-CNN [36]has been proved to refine the object detection results with multiple stages.FPN [51] and ResNet [55] have shown powerful hierarchical feature extractionabilities, which facilitate the processing of detecting and recognizing text invarious sizes. Therefore, we integrate Resnet-based FPN and Cascade R-CNN’s head network into the detector in our text spotting model.Feature Extraction BackboneThe feature extraction backbone is performed by residual blocks in FPN.Following [51], the input images are first feed-forwarded to residual blocks ina bottom-up pathway to get features, denoted as {C2, C3, C4, C5}, which arein the size of 1/4, 1/8, 1/16, and 1/32 of the input image size, respectively.Then the features are unsampled consecutively in a top-down pathway withthe factor of 2 and element-wise added with {C5, C4, C3, C2}, respectively, toget feature maps {P5, P4, P3, P2}. These feature maps construct an ensembleof FPN’s outputs P , i.e., the feature pyramid. In the standard FPN [51], P6is produced by subsampling P5. But P6 is not generated or used in our textspotting model as it aims for larger objects. In our model, P5 is the highest68CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESlevel of features while P2 is the lowest level. Consequently, the receptive fieldin the higher level is larger than the lower level if the same size region ofthe feature is given. Roughly speaking, the backbone provides the featurepyramid for other steps.Region of Proposal NetworkAfter the feature extraction process, we follow [34, 51] to construct the Re-gion of Proposal Network (RPN) to propose the Region of Interest (RoI)for text. We set the anchor size as {322, 642, 1282, 2562} for {P2, P3, P4, P5}.The aspect ratios are customized to {0.1, 0.5, 1.0} in our model to fit thevertical codes’ special property, i.e., its height is usually not less than thewidth. RPN proposes the possible regions for vertical code instances. Theseregions are refined in the Cascade R-CNN head later to remove the proposedinaccurate areas.Code Localization NetworkThe code localization process is to draw bounding boxes around the text.Our vertical text spotting model integrates Cascade R-CNN [36] as the lo-calization network. Three stages with the IoU threshold of {0.5, 0.6, 0.7} areset in the training process, respectively. Every stage consists of two tasks,that is, predicting the width and height of the bounding box and classifyingwhether it is the vertical code text or not. In the training process, the pro-posed regions which have higher IoU with the ground truth are sampled totrain the model. Since there is no ground truth provided during inference inthe testing process, the IoU thresholds are not used. However, the trainedstructure, i.e., the three-stage Cascade R-CNN head network, is kept to re-fine the predictions. The overlapped bounding box predictions are mergedby Non-Maximum Suppression (NMS) in post-processing steps. After codedetection, the localization information of code, i.e., the bounding box lo-cations, is parsed to both feature maps P of FPN and original images forrecognition.69CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESBText locationsand sizesHorizontalRoIAlign(selectedlevel)NMSFeature pyramidRPN (all levels)VerticalRoIAlign(selectedlevel)BCCBCCascade R-CNNFigure 4.2: The illustration of feature sharing in our vertical text spottingmodel. NMS stands for Non-Maximum Suppression. RPN is thenetwork that proposes the possible text regions. C and B stand forthe task of classification and bounding box position prediction, re-spectively, with a set of increasing thresholds in the Cascade R-CNN.Firstly, the feature pyramid is used by the detection process thatprovides code areas for vertical RoIAlign. Then text features are ver-tically aligned for recognition. Finally, a lightweight text recognizeris conducted after vertical RoIAlign. ©2021 IEEE. Reprinted withpermission from [2].4.2.2 Feature Sharing for Code Detection andRecognitionThe feature sharing for detection and recognition in our proposed verticaltext spotting model is accomplished by using two different RoIAlign Meth-ods, i.e., horizontal and vertical RoIAlign. FPN first extracts a pyramid offeature maps on different scales. These features are shared by detection andrecognition networks as inputs. The sharing process is shown in Figure 4.2.Then RPN proposes possible code regions which might not be extremelyaccurate and might include lots of false positives to be refined later. RPNoperates on all levels of features so that it is able to handle various scalesof code. The RoIAlign is following RPN, selecting an appropriate level of70CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESfeatures, and pooling them into a fixed size. In this pooling step, the fea-tures are aligned horizontally for simplicity due to the fact that this poolingis conducted for word-level classification and regression tasks which are notsensitive to the direction of code. After horizontal RoIAlign, Cascade R-CNN’s head network refines the proposals in three stages and NMS removesthe low confident region proposals. The detection part in the text spottingis completed after the NMS. Then, it comes to the recognition part when thedetected code locations and sizes are mapped to the feature pyramid and theinput image. The vertical RoIAlign is conducted at the beginning of coderecognition.Horizontal and Vertical RoIAlignHorizontal RoIAlign can be considered as the standard RoIAlign [85] sincethe relative position is not changed after pooling. Vertical RoIAglin is thetranspose of horizontal RoIAlign. However, the vertical RoIAlign is a sepa-rate and different step from the horizontal RoIAlign. Their differences are asfollows: (a) The purposes are different. Horizontal RoIAlign after RPN aimsat code localization while vertical RoIAlign is for code recognition. (b) Theinputs are changed. Horizontal RoIAlign gets initial possible code regionsfrom RPN, while the vertical RoIAlign in code recognition takes the final de-tection outputs as inputs. (c) The direction sensitivity is different. The textrecognition network is sensitive to the direction of the text so the poolingfeatures should be aligned vertically to facilitate vertical code recognition.However, the horizontal RoIAlign after RPN doesn’t care about the codedirection. (d) The pooler sizes are different. The pooler shape of verticalRoIAlign is set to be adapted to the aspect ratio of vertical code. The coderecognition network needs to decode the text sequence from pooled features.Therefore, the pooler size for recognition is expected to be vertically longerto include more features of characters and numbers. But the code detectionfocuses on drawing the rectangular borders of the text and calculating thedetection confidence, which is not related to the sequence decoding. So the71CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODEShorizontal RoIAlign after RPN can be pooled to any shape as long as it couldcatch the features of the whole code instance. In our model, the pooler sizeis 7×7 for horizontal RoIAglin, 8×32 for vertical RoIAlign.Adaptive Feature Level SelectionIn order to localize and recognize text in different sizes, the feature maps Pof FPN to be used in both horizontal and vertical RoIAglin should be chosenadaptively according to the size of codes. The level n of feature maps isselected according to the following equation:n = clip(⌊n0 + log2(√Ap/sc + 10−8)⌋, nmin, nmax) (4.1)where n0 is the level of feature maps to be selected when the area of code is inthe canonical size of ImageNet s2c , i.e., 2242. nmin and nmax are the minimumand maximum levels of feature maps to be used, respectively. They are setwith 2 and 5 in our model. Ap is the area of RoI in horizontal RoIAlign, andit is the area of detected code in vertical RoIAlign.4.2.3 Vertical Shipping Container Code RecognitionVertical code recognition is to read the code in the detected locations. In thispart, our vertical text spotting model combines two separated pipelines, i.e.,lightweight recognition, and standard recognition to achieve higher recogni-tion accuracy. We simplify the attention-based text recognition model [109][28] to be a lightweight network and incorporate it with code detection bysharing the features which are extracted by the FPN backbone. Additionally,a standard text recognition network runs separately to enhance code recog-nition. The final recognition results are chosen from one of these pipelinesby our decision strategy.Lightweight Text RecognitionAs the name indicates, the lightweight recognition network is lighter (withfewer parameters and layers) than the standard one. The deep feature ex-72CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESBiLSTMVertical featuresyt-1ht-1AttentionRNNythtSteps ...FOutputsY(y1,y2,...,yT)RNN with attentionCNNF Step tFigure 4.3: Structure of the lightweight recognition network. CNN and BiL-STM are used to extract sequence features. RNN with attention pre-dicts the text sequence. ©2021 IEEE. Reprinted with permissionfrom [2].traction has already been conducted in FPN so this lightweight recognitionnetwork focuses on extracting sequence features and decoding text. The net-work structure is shown in Figure 4.3. Vertical features are from verticalRoIAlign which selects a level of features and pools them to a fixed size(8, 32). CNN, consisting of three convolutional neural network layers with astride of 2 in this network, is applied to the vertical features. BidirectionalLSTM (BiLSTM) is used after CNN to encode features F to be the inputsfor Recurrent Neural Network (RNN). F is expanded with BiLSTM’s hiddensize, which is the same as RNN’s hidden layer size. Then, we incorporateRNN with attention to decoding text. The Gated Recurrent Unit (GRU)RNN [110] is implemented instead of the vanilla RNN. The output predic-tions Y (y1, y2, ...yT ) is on word level and yt is the probability distributionof possible characters at step t. T is the maximum length (a defined fixedvalue) of character that our model can recognize and it is equal to the totalprocessing steps of RNN with attention in Figure 4.3. RNN with attentionmechanism computes the current hidden state ht and output yt at step taccording to the following equations:wt = softmax(ϕ(tanh(ht−1 + F ))) (4.2)at = ϕ(concat(wt × F, ψ(yt−1))) (4.3)73CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESot, ht = RNN(at, ht−1) (4.4)yt = softmax(ϕ(ot)) (4.5)where wt is the attention weights and at is the output after applying atten-tion. ot is RNN output. ϕ is the trainable linear transformation layer and ψis the trainable embedding layer which takes the top 1 index of yt−1 as input.In the prediction stage, y0 and h0 are set with 0. yt has the size of (T,M).M is the total number of categories, including 26 letters, 10 numbers andanother one for the end-of-sentence.Standard Text RecognitionA standard recognition network is integrated into our vertical text spottingmodel in case the lightweight recognition network gets low confident outputswhich might result from inaccurate feature extraction. Since the FPN featureextraction backbone is trained for both code detection and lightweight textrecognition networks, the shared feature maps might not always be the bestfor text recognition. Therefore, our model integrates a separate standardscene text recognition model. It starts from the original images and con-ducts the feature extraction and recognition from scratch. Compared withlightweight text recognition, it is not affected by the shared features from thecode detection network. As the standard text recognition network has its ownfeature extraction part, the computational complexity should be taken intoconsideration when designing the model. Therefore, we build a standardtext recognition network that is similar to our lightweight recognition net-work in Figure 4.3. In order to improve the diversity and independence offeature extraction and recognition processes, the standard one differs fromthe lightweight one in the following aspects: (a) CNN takes the original im-age instead of feature maps as input. (b) There are more convolutional layersin CNN. We construct seven convolutional layers in the standard recognitionnetwork. (c) We stack two layers of BiLSTM to establish the feature F inthe standard network. Although the structure of RNN with attention is the74CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESsame as that in Figure 4.3, the standard and lightweight recognition are twoseparate networks, so they have different weights and are trained separately.Decision Strategy for RecognitionThe two separated text recognition networks will run inference on differentinputs to get the recognition results. However, choosing the prediction witha higher probability is not optimal since that higher probability predictionmight not have lower uncertainty. Hence, we propose a decision strategy,decided by the probability c and uncertainty u of the predicted text sequence,to choose results on the word level from one of the network’s outputs. Thehigher probability the predicted text sequence has, the more likely it willbe chosen. Nevertheless, the more uncertain the prediction distribution is,the less possible it will be selected. In our decision strategy for recognition,the selected results are decided by the selection factor Θ which is computedaccording to the following equation:Θ =µ0c1µ1c0(4.6)If Θ < 1, the results from the lightweight recognizer are chosen as the finalrecognition results. Otherwise, results from the standard recognizer are se-lected. µ0, c0 and µ1, c1 are the uncertainty and probability for lightweightand standard recognizer, respectively. In our model, the uncertainty of theprediction is measured by Shannon entropy. Probability c and uncertainty uare computed from the recognition network by the following equations:yk = {p(k,0), p(k,1), ..., p(k,M)} (4.7)c =K∏k=1max(p(k,m)),m ∈ {1, 2, ...,M} (4.8)µ =1KK∑k=1M∑m=1(−p(k,m)) ln(p(k,m)) (4.9)75CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESwhere p(k,m) is the probability that a character in the position k is predictedas the category m. p(k,m) is one item in RNN’s output yk as shown in (4.7).When computing the uncertainty µ, the Shannon entropy is divided by K toeliminate the effect of sequence length. Therefore µ can be considered as theaverage uncertainty on the character level. K is the length of the predictedtext sequence and it might be less than T due to that the end-of-sentencecould appear at any position.4.3 Experimental Results and DiscussionVertical codes are widely used during transportation. They could be theunique identification numbers, e.g., truck trailer’s IDs and shipping containernumbers, or type codes to indicate the specifications. The vertical codes arecrucial to logistic management as they might be the only identification whenother codes are occluded or not available. In this section, experiments areconducted to validate our proposed method.4.3.1 Trailer and Container Code DatasetThe proposed vertical text spotting model is validated on the vertical trailerand container code dataset from industry. Real images instead of syntheticdata are adopted for both training and testing since there is a huge amountof real data in the historical recordings and appropriate synthetic data arenot easy to generate. The total number of images for the dataset is 9265and we randomly select eighty percent for training and the rest for testing.There are mainly three kinds of vertical codes in our dataset, including thevertical ID of the truck trailer, the vertical shipping container number, andthe shipping container type code on the side panel. Images are captured whentrucks pass the gate under different weather and lighting conditions. Usually,the trailers and containers are loaded on the moving trucks when they arecaptured by cameras. Moreover, the backgrounds of these vertical codes differgreatly, depending on the condition and colour of the background objects.76CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESTherefore, the image quality is highly affected by the environment and thespeed of motion. The vertical code text might be blurry, even partiallycovered by snow or dirt.4.3.2 Model ImplementationTraining Loss FunctionIn the proposed model, the loss L of training code detection and lightweightrecognition is the weighted sum of localization loss Lloc and recognition lossLreco:L = Lloc + αLreco (4.10)where Lloc consist of loss of RPN Lrpn and loss of Cascade RCNN detectionLcas:Lloc = β1Lrpn + β2Lcas (4.11)We follow [34] to set Lrpn and [36] to set Lcas. In the training, we set both β1and β2 with 1 to balance the effect of RPN and Cascade R-CNN detectionhead. But α is set with 0.05 as the lightweight recognition network onlytakes features from limited code regions that relatively have less impact ontraining. Recognition loss Lreco is the negative log-likelihood loss, designedfor the recognition networks. It is defined as:Lreco =1TT∑t=1M∑m=1−ε(t,m) ln(p(t,m)) (4.12)where M is the total number of categories of letters and numbers that ourvertical text spotting model can recognize. T is the maximum text sequencelength in our model. If a text sequence length is less than T , the rest of thesequence is padded with the end-of-sentence category. ε(t,m) is the expectedtarget value of tth character in classm while p(t,m) is the predicted probability.We use one-hot encoding [111], which is suitable for categorical data withoutthe ordinal relationship, to process target data before sending them to the77CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODEStraining process. The training loss for the standard recognition network isthe same with Lreco in (4.12).Model Training and TestingAll the experiments are conducted on a desktop computer with the dualGPUs of Nvidia GTX 1080 and the CPU of AMD Ryzen Threadripper1900X (8-core/16-thread). In our vertical text spotting model, there aretwo separate recognition networks. Considering that the lightweight recog-nition network and the code detection network share the features from PFN,they are trained together. For better weight initialization, the FPN back-bone is pre-trained on ImageNet [112] as most of the text spotting modelsdo. The standard recognition network takes the original images as inputs soit is trained separately. In the testing process, these separated two pipelinesrun cooperatively. The code detection network first conducts the inferenceto localize vertical text. Then the code locations are passed to both of therecognition networks. The lightweight recognition network pools the featurefrom FPN while the standard recognition network crops the code regionsin the original image as inputs. The preferred recognition result is chosenadaptively by the decision strategy for recognition at the end.The training cost is shown in Table 4.1. Different methods require differ-ent training time. For existing networks, we follow the instructions in theirpapers to train the neural networks. For our own model, we train it until theperformance stops improving.Fig. 4.4 shows qualitative results of vertical code detection and recogni-tion. Our model could detect and recognize different kinds of vertical codeswhich are captured by cameras in various angles and distances, under diverseenvironments.78CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES(a) (b)(c) (d)Figure 4.4: Qualitative results of our model in spotting vertical codes. (a)and (c) were captured at night. (b) and (d) were taken during theday. No cropping or object segmentation is used before the imagesare sent to our model, i.e., our model takes original collected imagesas inputs. ©2021 IEEE. Reprinted with permission from [2].79CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESTable 4.1: Training cost of each method.Methods Training Time (hours)EAST [26] (2017) 72.06PSENet [27] (2019) 32.98Rosetta [57] (2018) 33.84MTS [28] (2021) 32.91MTSv3 [29] (2020) 30.85ABCNet [30] (2020) 48.15Ours 43.284.3.3 Ablation StudyRecognition Results on Different Feature LevelsIn order to validate the effectiveness of adaptive choosing feature maps forlightweight recognition in (4.1), we pool the single level of feature maps andrun the text sequence prediction separately for comparison. In this part,we would like to exploit how the level of feature maps affects the lightweightrecognition. Therefore, the code detection network is kept the same and onlythe lightweight recognition performance is compared in this experiment. Asmost spotting models do [29, 30, 113], we evaluate the performance by Preci-sion p, Recall r, and f1-score f1. The performance of using different featurelevels is shown in Table 4.2. The adaptive way achieves better performancein recognition than the way of only using the fixed level. {P2, P3, P4, P5}are the feature maps extracted by FPN in different levels. The used levelof feature maps in the adaptive way is decided by the size of text accordingto (4.1). The larger the vertical code is, the higher level of feature mapswill be chosen. So the adaptive way only uses one level for the lightweightrecognition on a single text instance. The chosen level might be changed foranother text instance of a different size even though they are in the sameinput image. Experimental results validated that adaptively choosing the80CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESTable 4.2: Recognition performance of lightweight network on different fea-ture levels.Feature Maps p r f1P2 0.961 0.904 0.932P3 0.777 0.571 0.658P4 0.293 0.008 0.012P5 0.019 0.017 0.017Adaptively Chosen 0.969 0.916 0.942Table 4.3: Recognition performance by different decision strategy.Strategy p r f1Lightweight Recognition Only 0.969 0.916 0.942Standard Recognition Only 0.971 0.936 0.953Recognition Decided by Probabilities 0.973 0.944 0.958Ours 0.973 0.956 0.965feature level from FPN enhances the performance in the lightweight textrecognition.Effects of Decision Strategy on RecognitionOur model integrates two separated text recognizers and chooses results bythe decision strategy. Traditional text spotting models either use one recog-nizer only [30] or simply choose the prediction with higher probability whenthey incorporate two recognition pipelines [28] [29].We implemented different decision strategies and evaluated performancerespectively. Table 4.3 shows the performance of lightweight recognition,standard recognition, recognition decided by probability, and recognition byour decision strategy, respectively. As we would like to design a more effectiveway to select the recognition result, only the recognition part is comparedin this table. The experimental results validated that our decision strategyoutperforms other methods. That is to say, selecting recognition results con-81CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESsidering both uncertainties and probabilities is more effective than the way offocusing on probability only. On the other hand, Table 4.3 indicates that thestandard recognition network gets a higher precision, recall, and f1-score thanthe lightweight recognition network. Standard recognizer takes the originalimages as inputs and establishes its own feature extraction layers, resulting inhigher accuracy. However, the standard recognizer might perform incorrectlyon some of the images. In the experiments, fusing two recognizers improvesthe recognition accuracy no matter which strategy, considering probabilitiesonly or both probabilities and uncertainties, is conducted. Therefore, our twotext recognition networks are complementary to each other and our decisionstrategy for recognition is indispensable and effective.Figure 4.5 depicts an example of conducting our two attention-basedrecognition networks and deciding which recognition result is adopted ona vertical code. Feature maps of the code region are sent to the lightweighttext recognition network while the original image is used in the standardnetwork. These two networks get different results. In our decision strategyfor recognition, the chosen result is decided by the selection factor Θ whichis related to probabilities and uncertainties. In this example, the standardtext recognition network dominates the recognition.A demonstration of our recognition process is shown in Fig. 4.5. Dur-ing the network inference, attention weights are conducted on the extractedfeatures. In this figure, we overlay the attention weights on the image in-stead of features for better visualization. Red areas indicate high attentionweights. Θ is the selection factor in (4.6). Both of the lightweight and stan-dard recognition networks conduct 6 attention steps to finish searching forthe whole text instance. However, they give different recognition results.The recognition results are selected by our decision strategy at the end.Although the visualizations in Figure 4.5 for these two networks are simi-lar, the attention weights, shown in the red regions, are different. Taking thesecond last number as an example, the attention of lightweight recognition82CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESResult 0:645859Result 1:645869Lightweight recognitionStandard recognition 1 2 3 4 5 6Feature mapsImageAttentionsteps:>1, choose result 1Figure 4.5: A demonstration of our recognition process. ©2021 IEEE.Reprinted with permission from [2].83CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESfocuses on the center of the number but the attention of standard recogni-tion concentrates on the lower part. Each of the networks could performthe text recognition task separately. However, the results might be differentbecause of different weights and inputs. Specifically, the differences betweenthese two networks are as follows: (a) They are trained separately. Thelightweight recognition model is trained together with the detection networkwhile the standard recognition network is trained alone. (b) The inputs aredifferent. The inputs of the lightweight text recognition network are thefeatures extracted from the backbone of the feature pyramid network. Butthe inputs of the standard text recognition network are the cropped images.Thus, they are producing different results.Effects of PreprocessingTo explore the effects of the preprocessing operations on the recognition per-formance, some image preprocessing methods, including Canny edge detec-tion, adaptive Gaussian thresholding, adaptive mean thresholding, and Otsuthresholding have been implemented to process the cropped text area, respec-tively. After preprocessing, the text images are sent to our standard scenetext recognition network. Otsu’s thresholding method [114] is implementedfor image segmentation. Adaptive mean thresholding [115] sets the thresh-old with the mean of neighbourhood values minus a constant value, whileadaptive Gaussian thresholding [115] sets the threshold with the Gaussianweighted sum of the neighbourhood values minus a constant value. Empiri-cally, the kernel size for this neighbour is set with 5 and the constant value isset with 2. Canny edge detection [74] is also adopted to preprocess the textimage.The quantitative performance comparison is shown in Table 4.4. Ourstandard text recognition network without image preprocessing steps achievesthe highest precision, recall, and f1-score. The reason is that conventionalimage preprocessing methods usually involve some manual settings, whichrely on experience and can only be applied in limited conditions. In con-84CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESTable 4.4: Recognition performance after using different preprocessingmethodsPreprocessing MethodsRecognition Resultsp r f1Canny Edge Detection 0.681 0.442 0.536Adaptive Gaussian Thresholding 0.841 0.664 0.742Adaptive Mean Thresholding 0.849 0.690 0.761Otsu Thresholding 0.913 0.821 0.864Without Preprocessing 0.971 0.936 0.953Table 4.5: Performance comparison with the state of the art.MethodDetection End-to-End Recognition Speed(FPS)p r f1 p r f1EAST [26] (2017) 0.965 0.887 0.924 - - - 7.19PSENet [27] (2019) 0.707 0.716 0.711 - - - 6.72Rosetta [57] (2018) 0.931 0.993 0.961 - - - 7.21MTS [28] (2021) 0.892 0.975 0.932 0.667 0.728 0.696 2.30MTSv3 [29] (2020) 0.913 0.592 0.719 0.906 0.587 0.713 3.66ABCNet [30] (2020) 0.993 0.931 0.961 0.966 0.905 0.935 3.57Ours 0.989 0.991 0.990 0.973 0.956 0.965 3.22trast, there are trainable convolutional layers in our recognition network.These layers can be trained to adaptively extract features of text instancesin various conditions. On the other hand, characters are usually small ob-jects, which can be easily eroded in noisy regions when image preprocessingmethods are used. Qualitative examples of preprocessing results are shownin Figure 4.6. Erosion can be found in Figure 4.6 (a) (d) (g) (h) (i) (k).Eroded characters would lose information and cannot be recognized.Therefore, we do not use image preprocessing methods in our model.85CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES(a) (b) (c) (d) (e)(f) (g) (h) (i) (j)(k) (l) (m) (n) (o)Figure 4.6: Qualitative results of prepossessing on cropped code regions.(a), (f), and (k) are the results processed by Canny edge detection.(b), (g), and (l) are results of adaptive Gaussian thresholding. (c),(h), and (m) are results of adaptive mean thresholding. (d), (i), and(n) are results of Otsu thresholding. (e), (j) and (o) are the originalvertical text images. ©2021 IEEE. Reprinted with permission from[2].86CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES4.3.4 Comparison with the State of the ArtDetection and Recognition Performance ComparisonAs shown in Table 4.5, our proposed vertical text spotting model outperformsother state-of-the-art text spotting models in both detection and recognitionstages while maintaining a comparable processing speed with other text spot-ting models.In the code detection part, our model integrates FPN and Cascade R-CNN. FPN provides an ensemble of feature maps in different sizes whichadapt to the diversified sizes of text, while Cascade R-CNN refines the bound-ing box localization in multiple stages. Our model achieves the highest f1-score, outperforming Faster R-CNN-based [34] text spotting models, suchas Rosetta [57], MTS [28] and MTSv3 [29], and one stage detector-basedmodel ABCNet [30]. We implemented all of these methods on the samedatasets. Although ABCNet [30] gets slightly higher detection precision thanour model, it falls behind in f1-score. Rosetta [57] gets the same recall withour model but has lower precision and f1-score. As EAST [26] and PSENet[27] are scene text detection models and Rosetta [57] is designed for hor-izontal text spotting, only their detection performance is evaluated in ourexperiments. EAST is a fast and efficient text detector but the accuracy suf-fers from the single-stage architecture and limited receptive fields. PSENetis a segmentation-based scene text detector that relies on expanding opera-tions in post-processing steps. Consequently, their performance is inferior tomultiple stage-based detectors.Evaluation by Average Precision (AP) has been conducted on our verticalcode datasets. To match practical meanings, the IoU threshold is set between0.5 and 0.75 to measure AP. AP-IoU curves for the vertical code detectionare shown in Fig. 4.7. Our method achieves similar detection performancewith ABCNet [30] but performs better when the IoU threshold is set between0.6 and 0.7. AP-IoU curve of Rosetta [57] is slightly lower than ours. The87CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES0.5 0.55 0.6 0.65 0.7 0.75IoU00.20.40.60.81APOursEASTPSENetRosettaMTSv2MTSv3ABCNetFigure 4.7: AP-IoU curves of different vertical code detection methods. APstands for average precision. ©2021 IEEE. Reprinted with permissionfrom [2].AP values of the other four methods always fall behind these three meth-ods. The results demonstrate that our method achieves the best localizationperformance.Some visualization examples of outputs from arbitrary-oriented text spot-ting models and our model are shown in Fig. 4.8, 4.9, 4.10, and 4.11. BothMTS [28] and MTSv3 [29] generate masks for text instances so their detec-tion outputs are polygons. ABCNet [30] applies Bezier curves to generatepolygons to fit the arbitrary-oriented text. Since the vertical codes in ourdatasets are relatively straight, the generated polygons by ABCNet approxi-mate rectangles. Our proposed text spotting model conducts multi-stage textdetection to generate bounding boxes. The results show that our methodcould accurately localize the vertical code regions.In the text recognition part, our model applies both lightweight recogni-tion and standard scene text recognition. Our proposed model could share88CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES(a) MTS(b) MTSv3Figure 4.8: Visualization of text spotting models’ outputs. (a) (b) are out-puts of MST and MTSv3, respectively. Detection results are shownin red polygons or bounding boxes. Recognition results are with redtext. Zoom in for better visualization. ©2021 IEEE. Reprinted withpermission from [2].89CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES(a) ABCNet(b) OursFigure 4.9: Visualization of text spotting models’ outputs. (a) (b) are out-puts of ABCNet and ours, respectively. Detection results are shownin red polygons or bounding boxes. Recognition results are with redtext. Zoom in for better visualization. ©2021 IEEE. Reprinted withpermission from [2].90CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES(a) MTS (b) MTSv3Figure 4.10: Visualization of text spotting models’ outputs. (a) (b) are out-puts of MST and MTSv3, respectively. Detection results are shownin red polygons or bounding boxes. Recognition results are withred text. Zoom in for better visualization. ©2021 IEEE. Reprintedwith permission from [2].91CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODES(a) ABCNet (b) OursFigure 4.11: Visualization of text spotting models’ outputs. (a) (b) areoutputs of ABCNet and MTSv3, respectively. Detection resultsare shown in red polygons or bounding boxes. Recognition resultsare with red text. Zoom in for better visualization. ©2021 IEEE.Reprinted with permission from [2].92CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESthe features between code detection and recognition, as well as extract fea-tures from images for text recognition separately. It obtains more diversevertical features and archives higher recognition accuracy than any otherstate-of-the-art model. ABCNet [30] generates Bezier curves to align textfeatures. However, there is no need to draw curves around vertically alignedtext for recognition because these extra curves might lead to inappropriatefeature alignments. Both MTS [28] and MTSv3 [29] use two text recognitionpipelines and select the results by probabilities only. Their RoIAlign is notspecially designed for vertical codes and one of their recognizers is based oncharacter segmentation. It is hard to segment single characters when theystick together. Our proposed model recognizes the text on the word level sothere is no strict character segmentation process. Moreover, our two recogni-tion networks take different features which are complementary to each other,performing better than any other methods.Comparing Multi-Stage, Two-Stage, and Single-Stage Detectorsin Text SpottingAlthough Cascade R-CNN [36] with FPN [51] has been used widely in objectdetection, it is the first time that it has been integrated into a text spot-ting model. Existing work only applied Cascade R-CNN in text detection[116], which can be considered as a special task of object detection. Pre-vious work has not studied the feature sharing between Cascade R-CNN,FPN, and text recognition networks, especially for vertical text. In contrast,our text spotting model conducts text detection and recognition together.It integrates Cascade R-CNN, FPN, and two text recognition networks viafeature sharing.Applying a set of increasing IoU thresholds in multi-stage detectors couldimprove the performance of text detection. These IoU thresholds are used todistinguish positive and negative samples during the training of the detec-tion network. We follow the multi-stage detector, Cascade R-CNN [36], toconduct increasing IoU thresholds. There are three classification and three93CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESregression tasks with IoU threshold of {0.5, 0.6, 0.7}, separately. Notably,these six tasks are included in one detection network which is integrated intoour proposed text spotting model. For comparison, the IoU threshold is onlyset with one fixed value, for instance, 0.5, in two-stage-based [57] [34] [28] [29]or single-stage-based [30] detectors. Therefore there is only one classificationand one bounding box regression task in the detection network. The detec-tion performance is compared in Table 4.5. Although setting a fixed value ofIoU threshold could lead to higher recall in Rosetta [57] or higher precisionin ABCNet [30], they fall behind in f1-score greatly in terms of detection.Dual Recognition Pipelines and the Selection Strategy in TextSpottingComparing with MaskTextSpotter (MTS) [28] and MaskTextSpotter v3 (MTSv3)[29], which use two recognition pipelines, there are three major improvementsin our methods. Firstly, all their recognition networks only take the extractedfeatures from the backbone. However, our two recognition networks take ex-tracted features and original cropped images as inputs, respectively. Morediverse inputs are used in our method. The standard text recognition net-work that takes the raw images as inputs is trained separately. Thus, it willbe less affected by the detection network and the other recognition network.Secondly, MTS and MTSv3 choose the results only by comparing proba-bilities. But our method considers both uncertainties and probabilities todecide which results of the dual pipeline to be used, leading to better perfor-mance. Thirdly, both MTS and MTSv3 construct two-dimensional attentionweights in their recognition networks. In contrast, our recognition networksapply only one-dimensional attention weights. It is computationally efficientand fits with vertical text. As MTS and MTSv3 are designed for arbitrary-oriented text spotting, two-dimensional attention weights could search forthe characters in any direction of two-dimensional spaces. Our method isspecially designed for vertical text. Therefore, one-dimensional attention iscapable of performing the search vertically. Using one-dimensional attention94CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESweights would also reduce the computational complexity. With these im-provements, our method performs better than MTS and MTSv3 in terms ofvertical text as shown in Table 4.5.The reason why both probabilities and uncertainties are considered whenselecting preferred results is explained as follows. The text recognition net-work consists of three main parts, feature extraction by the CNN and LTSMlayers, character segmentation by attention and RNN, and character recog-nition by the softmax layer. The softmax layer produces probabilities thatare used to select results in MTS and MTSv3. Character recognition canbe taken as a classification task and text recognition is a series of classifi-cation tasks. Generally, a classification task is completed by choosing thecategory with maximum probability. The probability of a result in a textrecognition task, which is also referred to as the confidence of a recognizedword by contemporary methods, is calculated by equation (4.8), i.e., the cu-mulative product of maximum probability for single character classificationtasks. Nevertheless, it ignores the uncertainty of the classification tasks. Ifa classification task has higher uncertainty, the result is supposed to be lesstrusted. The uncertainty of a single character classification is computed byShannon entropy [117]. We use the cumulative product of Shannon entropyfrom all character classification tasks to measure the uncertainty of the pre-dicted results. It is computed by equation (4.9). When choosing recognitionresults from two pipelines, the principle is to choose the one that is mosttrusted. There are two factors to be considered, probabilities and uncer-tainties. Result with higher probability and lower uncertainty are preferred.Mathematically, the decision is made by comparing the ratio ci/ui, where ciand ui stand for confidence and uncertainty for pipeline i, respectively. ciand ui are positive values. If c0/u0 > c1/u1, result from pipeline 0 is chosen,otherwise, result from pipeline 1 is selected. Thus, the selection factor Θ iscomputed by equation (4.6). If Θ > 1, choose pipeline 1, otherwise, choosepipeline 0. The comparison in Table 4.5 shows that our method achieves95CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODEShigher performance than other dual recognition pipeline methods. The abla-tion study results of Table 4.3 shows that the selection which considers bothuncertainties and probabilities performs better than the selection which onlyconsiders probabilities.Results in Table 4.3 are under our ablation study. These results aregenerated by our networks in the following specific settings: (a) Only thelightweight recognition network is enabled. (b) Only the standard recogni-tion is enabled. (c) Two recognition networks are enabled and selected byprobabilities. (d) Two recognition networks are enabled and selected by ourproposed selecting factor Θ. Among these settings, (d) is the best strategy,performing slightly better than (c). When our model with this selection strat-egy is compared with existing text spotting models in Table 4.5, we couldsee great improvements in precision, recall, and f1-score, while it maintainsa comparable computation speed.4.4 SummaryAutomatically reading vertical container codes with the machine vision-basedmeasurement brings great convenience to the monitoring and recording oflogistical information. The state-of-the-art text spotting models, includingMask TextSpotter v2 and v3, and ABCNet, are elaborated on the arbitrary-oriented text and curved text. Although vertical codes can be consideredas a special category of arbitrary-oriented text, their unique property, thetop-to-bottom character alignment, has not been specially treated in textspotting models. In previous research, the text alignment is usually from theleft to the right of characters in the horizontal text, curved text, and rotatedtext. Directly applying general text spotting models in vertical codes leadsto the deficiency in the recognition process. Therefore, this chapter proposesa vertical text spotting model specially for vertical codes. Especially, theproposed text spotting model pools the text features vertically from detectionto recognition by vertical RoIAlign to match the code direction. In this way,96CHAPTER 4. SPOTTING VERTICAL SHIPPING CONTAINER CODESthe recognition network could decode the text sequence from top to bottom.More notably, the proposed vertical text spotting model integrates two textrecognition networks that are complementary to each other.The experimental results show that the proposed vertical text spottingmodel outperforms other state-of-the-art text spotting models in both thedetection and recognition of vertical codes. The end-to-end recognition re-sults demonstrated the superiority of the designed dual recognition pipelineon vertical codes. The proposed vertical text spotting model provides an ef-fective solution for detecting and recognizing vertical codes and will promotethe digitization and automation of the logistic management of trailers andcontainers in the transportation industry.The contributions of our vertical code spotting model are summarized asfollows:• Vertical RoIAlign is proposed to align features vertically for containercodes in our text spotting model which could decode text from top tobottom. Vertical RoIAlign is the crucial part to connect the verticalcode detection and recognition.• The model incorporates two separate text recognition pipelines. Oneis the lightweight recognition network which shares features with thedetection network. Another is a separate standard text recognition net-work that takes the cropped code areas as inputs. These two pipelinesare complementary to each other and perform better than a singlepipeline.• A decision strategy is designed to adaptively choose recognition out-puts. It considers both uncertainties and probabilities of the pre-dicted text sequences. With the decision strategy, the proposed verticaltext spotting model achieves higher accuracy than single recognitionpipeline-based models. This decision strategy is also more effectivethan the probability-based selection strategy.97CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESChapter 5Spotting Hazardous MaterialPlacard CodesThis chapter conducts research on spotting hazardous material placard codes.Automatically detecting and recognizing hazardous material placards usingcomputer vision-based methods ensures safe operations and proper manage-ment of dangerous freight transportation. Deep learning-based object detec-tion methods provide viable and practical solutions to varied applications.However, contemporary deep learning-based methods suffer from imbalancedand open classes, which are very common in real-life data. Imbalanced classesindicate that these classes have different amounts of training samples. Openclasses refer to the classes that are not included in the training dataset butappear in the testing dataset, while closed classes mean they are consideredin both training and testing datasets. Thus, this study draws attention tothis hitherto neglected challenge in real-world applications and proposes adeep learning-based zero-shot framework to detect and recognize the haz-ardous material placards of both imbalanced and open classes. A logarith-mic weighted cross-entropy (LWCE) loss function is proposed to balance theclosed classes during training. In addition, a logarithmic weighted confi-dence fusion (LWCF) strategy is designed to fuse the separately extracted98CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESvisual and textual information. The experiments on real-world transporta-tion data demonstrate the proposed framework’s effectiveness and superiorityover other state-of-the-art methods. Notably, our framework outperforms theprevious method with a remarkable margin of 12.8% in f1-score on the plac-ard dataset. This study solves the imbalanced and open class problem byfusing object visual information and text information, providing a practicalindustrial application of the zero-shot learning concept.Although existing deep learning-based methods detect and recognize haz-ardous material placards effectively on the categories with a huge amount oftraining data, they fall behind in the classes that are rarely seen. Evenworse, they cannot recognize the types of placards that are not included inthe training process. Our proposed framework enhances the performance ofimbalanced classes and opens up the recognition of new classes that havenever appeared in the training dataset. Compared with the previous state-of-the-art method, our framework improved the overall f1-score from 79.97%to 92.77%. As a very practical solution, our framework will promote the au-tomation of hazardous material freight monitoring, consequently, improvingthe safety of transportation, as well as the efficiency of the global supplychain.5.1 OverviewA great number of computer vision-based approaches can be applied to plac-ard detection and recognition. One kind of approach focuses on textual infor-mation of placard codes, i.e., detecting and recognizing the printed placardclass codes directly, by building robust text spotting models [28–30, 84].However, text spotting methods are only applicable to placards with visibleand unique codes, as some of the class codes may not be shown, for instance,placards of class LQ do not print codes, and some types share the sameprinted code, e.g., placards of class 2.1, 2.2 and 2.3 are all printed with thesame number, 2, on their boards. Although these methods obtain textual99CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESinformation, they can be applied only to limited categories.Another kind of approach pays attention to object-level visual informa-tion of placards, i.e., detecting and classifying the whole placards. Theseapproaches can be roughly divided into two categories, traditional featureextraction-based methods [19, 78, 79, 118, 119] and deep learning-basedmethods. The former designs handcrafted features to extract salient fea-tures of placards before the detection and recognition processes. They donot require a large amount of training data but need expertise in analyzingthe unique features, which might be applicable only in certain scenarios. Incontrast, deep learning-based object detection methods [34–36, 83], learningobject features through training on the established datasets, could directlybe applied to detect and recognize the placards under versatile outdoor envi-ronments, without constructing handcrafted features. Thus, deep learning-based solutions are preferred in industrial applications. The comprehensiveand relatively balanced datasets are usually required to train the deep neuralnetworks to prevent generating biased results [120, 121].However, there is a great challenge in real-world applications, i.e., theimbalanced and open class datasets. Generally, the number of containersthat transport hazardous freight at the data collection sites is not sure, andthe placard types on the containers are also uncertain. In the collected data,the numbers of various classes are often strikingly different from each other,with some types rarely seen, or even missing, while the others are excep-tionally abundant, hampering the establishment of the ideal comprehensiveand balanced placard image datasets. Consequently, prior deep learning-based object detection models have limited performances in the unevenlydistributed classes, and more seriously, fail in the open classes.Zero-shot learning [122, 123] is the concept to handle open classes, orforeign classes [124], while keeping the performance in seen classes. Com-pared with few-shot learning [125–128], which focuses on closed classes witha few training samples, typical generalized zero-shot learning methods usu-100CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESally require aligning visual and semantic information properly [123] to detectobjects in both open and closed classes. Nevertheless, no research has beenconducted to establish placards’ visual and semantic information, and nosuch zero-shot framework has been built.Therefore, to address the unavoidable challenge of imbalanced and openclass distribution of real-world placard datasets, we propose a visual andtextual information fusion-based zero-shot framework with the capability ofdetecting and recognizing different kinds of placards, especially for imbal-anced and open categories.In the training stage, the detection network in our framework is trainedwith our proposed LWCE loss to balance classes. In the testing stage, ourframework first detects and classifies the placards to calculate visual con-fidence. Further, the textual confidence is generated by our destined code-related confidence boosting after code detection and recognition. Eventually,our designed LWCF strategy fuses the visual and textual confidence sets toget the final confidence and predicted classes.5.2 Proposed Zero-Shot Framework forPlacard Code SpottingOur visual and textual information fusion-based zero-shot framework to spotplacard code is shown in Fig. 5.1. The input images are first sent to theplacard detection model to get isolated placard regions. Then, the visualconfidence of each class is obtained by the placard classification model trainedwith our LWCE loss. The textual confidence is acquired by our designedcode-related confidence boosting after the code spotting (consisting of codedetection and recognition). The final confidence is computed by our proposedLWCF strategy, which fuses visual and textual confidence. The class withthe highest confidence is selected as the final class.Our framework are explained in 3 parts, i.e., visual information extrac-tion, textual information extraction, and confidence fusion.101CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESVisualconfidenceTextualconfidencePlacarddetectionCodedetectionCode recogitionRecognized code patternsFinalconfidenceand classesDetectedplacardsDetectedcodesInputimagesPlacard classificationLogarithmicweightedconfidence fusionVisual information extractionTextual information extractionCode-related confidence boostingFigure 5.1: Proposed framework for hazardous material placard code de-tection and recognition. Our designed modules include code-relatedconfidence boosting, logarithmic weighted confidence fusion, and coderecognition. ©Ran Zhang, 2022. Images provided by Canscan Inc[14]. Diagram created by Ran Zhang.5.2.1 Visual Information ExtractionVisual information extraction is conducted by placard detection and classi-fication, which generate the confidence of each class. In our framework, theplacard detection model is implemented by Faster R-CNN [34]. For simplic-ity, its details are left out here. In the placard classification task, the featuremaps that are shared with the placard detection model are cropped in theplacard regions before being utilized as inputs.To train the placard classification model, instead of using the standardcross-entropy loss in R-CNN-based methods [34, 36], we propose LWCE lossLlwce, especially for the imbalanced placard datasets. It is defined asLlwce = −wt ln eyt∑Cc=1 eyc(5.1)where t is the target class index. yt is the predicted confidence on class t. Cis the total number of placard classes. yc is the inference confidence on class102CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESc. Explicitly, the logarithmic weight wt is defined aswt = 1− ln(Nt)max {ln(Nc)} + β0, for c ∈ [1, C] (5.2)where Nc and Nt are the numbers of training samples on class c and targetclass t, respectively. If Nc is 0 on a class, we set the value of Nc to be 1 toprevent infinite value of wt. β0 is the bias to define the minimum value ofthe proposed logarithmic weight. Therefore, the range of wt is [β0, 1 + β0].The principal purpose of our proposed LWCE loss is to differentiate thepenalties for imbalanced classes. Generally speaking, we would like the modelto learn more about rarely sampled classes during the training by adding morepenalties for false-negative predictions of these classes. On the contrary, thefalse-negative predictions on rich sampled classes are less punished since theseclasses appear more frequently during the training. In this way, the losses arerelatively more balanced than the standard cross-entropy on the imbalancedtraining datasets.Instead of using Nt/max (Nc) as the variance factor in equation 5.2, wechoose the logarithmic fraction ln(Nt)/max{ln(Nc)} to adjust wt for differenttarget classes. The advantage of the logarithmic fraction is that it makes thewt change smoothly among significantly imbalanced classes, thus, preventingthe divergence of training.In our framework, β0 is set to 0.1 according to the trial and error. The-oretically, setting β0 with an enormous value makes the penalty for imbal-anced classes undifferentiated. If a large β0 is set, the LWCE loss would needto be assigned with a smaller balancing factor when combined with otherlosses, e.g., placard detection loss, during the training, causing the logarith-mic weight meaningless. Oppositely, the small value of β0, for instance, 0,would significantly reduce the penalty on the classes that come with a largenumber of training samples, making it difficult for the training process toconverge.103CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESCroppedcode regionsConverting tograyscale & resizingFeatureextraction byCNNRNN withattentionCode contents & confidencesFigure 5.2: The structure of our optimized placard code recognition model.©Ran Zhang, 2022. Diagram created by Ran Zhang.At the end of the visual information extraction process, the placard clas-sification confidence is taken as the visual confidence, which is one of theinputs to confidence fusion.5.2.2 Textual Information ExtractionTextual information extraction consists of procedures of code detection, coderecognition, and code-related confidence boosting.Code DetectionIn terms of code detection, the vanilla Faster R-CNN model is leveraged. Asplacard regions are remarkably smaller than the whole image, the input sizeof the code detection model is set at 128 pixels. The cropped placard regionsare resized to images with 128 pixels on the shorter side while the aspectratios are kept. The rest of the structure is the same as the standard one[34].Code RecognitionTo efficiently and accurately recognize the detected placard codes, we de-signed an optimized code recognition model, of which the structure is shownin Fig. 5.2. The cropped code areas are first converted to grayscale and re-sized to 32 pixels in height and 64 pixels in width, while the aspect ratiosare maintained with padding. Secondly, the CNN layers are utilized to ex-tract text features. Finally, the commonly used RNN with attention module[43, 109], is leveraged to conduct the text sequence prediction and outputthe code contents and confidence.Compared with the previous state-of-the-art scene text recognition meth-104CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESods, including CRNN [41], GRCNN [42] and ClovaAISTR [43], our placardcode recognition model is specially constructed for unique properties of plac-ard codes. First, the image size and image aspect ratio of our model fit withthe real-world images, while the recognition accuracy is not decreased. Theprevious text recognition methods usually resize the input image to 32× 100pixels, which are too wide for the placard codes as they have two digits max-imumly. Second, the text shape transformations are abandoned in our recog-nition model, as they are previously used to transform the curve text to bestraight, e.g., thin plate transformation in ASTER [44] and ClovaAISTR [43].They are unnecessary because the placard codes are horizontally straight inthe text alignment. Third, the bidirectional LSTM [41, 103], which is widelyleveraged to capture the contextual information of text sequences in othermethods [41–43], is also excluded in our model as the contextual informa-tion are negligible in these short codes. Consequently, our optimized coderecognition model only consists of the image preprocessing steps, the CNN-based feature extraction layers [41], and the RNN with attention-based textprediction module [109]. With a simplified structure and smaller image in-put size, our optimized code recognition model is expected, theoretically, tobe converged easier during the training and run faster during the inference,while enhancing the recognition performances.Code-Related Class Confidence BoostingWe designed the code-related class confidence boosting strategy, which acts asthe final step of textual information extraction to generate textual confidence.In this strategy, the textual confidence of code-related classes is boosted bythe placard code spotting results. The recognized code patterns and theirrelated classes are shown in Table 5.1. For instance, if the recognized codeis 1, the textual confidence of classes 1.1, 1.2 and 1.3 are assigned with thevalue from code spotting confidence, while the textual confidence of otherclasses is set with 0.It is worth mentioning that, in our code recognition datasets, the decimal105CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESpoints are ignored as these points are usually too small and illegible, orheavily affected by noise. Omitting the decimal points does not lead topattern confusion. For example, as there are no other codes defined as 51,ignoring the decimal point in code 5.1 would not lead to confusion withothers. Therefore, the recognized code patterns contain numbers only, asshown in Table 5.1. If no placard code is spotted, the textual confidencewould be empty. If the code recognition model outputs code patterns otherthan those in Table 5.1, these undefined patterns would be ignored beforeconfidence boosting. If multiple codes are detected inside one placard, onlythe longer two-digit codes, which contain more detailed information aboutcode classes, would be adopted to conduct code-related confidence boosting.Mathematically, the textual confidence, −→pt , after code-related class con-fidence boosting, can be expressed as−→pt = [pt1 , ..., pti , ..., ptC ] (5.3)where C is the total number of classes. pti is the textual confidence for classi, defined aspti =qd × qr if i ∈ B0, else (5.4)where i is an index of a class. B is the index set of related classes, whichcan be found in Table 5.1. qd and qr are the code detection and recognitionconfidence, respectively.Eventually, the textual information extraction process outputs the textualconfidence −→pt , which is also the input of confidence fusion.5.2.3 Logarithmic Weighted Confidence FusionAfter computing the visual and textual confidence, it comes to confidencefusion. The visual confidence is for closed classes, while the textual confidenceis for code-related classes, which could be either open or closed classes. These106CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESTable 5.1: Recognized code patterns and their related classes.Code Pattern Related Classes1 1.1, 1.2, 1.311 1.112 1.213 1.314 1.42 2.1, 2.2, 2.33 34 4.1, 4.2, 4.351 5.152 5.26 67 78 89 9two sources of confidence are fused by our designed LWCF strategy.The final fused confidence −→pf is computed by−→pf = −→pv × w +−→pt × (1− w) (5.5)where −→pv and −→pt are the visual and textual confidence, respectively. Thelogarithmic fusion weight, w, balances these two items during the fusionprocess. It is defined asw =max{ln(Ni)}max {ln(Nc)} , for c ∈ [1, C], i ∈ B (5.6)where C is the total number of classes, and B is the index set of classes relatedto recognized codes. On the one hand, our framework, with the proposedfusion weights, relies more on the visual confidence for the classes that havemore training samples. On the other hand, in terms of the classes with fewertraining samples, it puts more belief on textual confidence. In the extreme107CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODEScases of unseen classes, when there is no training sample of specific classesof placards, our framework abandons the visual information and fully truststhe textual information.With our framework, the final confidence and predicted classes are themaximum elements of −→pf and the class name of the maximum elements,respectively.5.3 ExperimentsThe effectiveness of our proposed framework is validated by real-world in-dustrial data. All the experiments are conducted on a computer with twoNvidia GTX 1080 GPU (8 GB graphics memory each), a CPU of AMD RyzenThreadripper 1900X (8 cores and 16 threads), and a 32 GB random accessmemory.5.3.1 DatasetsThere are two datasets involved in our experiments. One is the Synth90kdataset [129], a publicly available synthetic text recognition dataset. Theother is a real-world placard dataset, which was collected by our industrypartner during shipping container transportation.• Synth90k [129] is a public synthetic text recognition dataset. It con-tains 9 million images of alphanumerical text instances and their anno-tations. In our experiment, only the numerical text images are chosenbecause we need to recognize the placard codes that consist of numbersonly. In total, 11264 synthetic images are selected, then mixed with ourreal-world cropped placard code images to train our code recognitionnetwork.• The placard dataset contains 3015 images in total. These images aretaken at ports via the cameras in the fixed positions, while the shippingcontainers attached with placards are transported by trucks. However,108CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESTable 5.2: The numbers of training and testing samples on each classClass Index Class NameNumber inTrainingNumber inTesting1 MP 480 412 3 420 423 LQ 374 414 8 342 415 9 324 416 6 276 417 1.1 231 418 2.2 196 419 4.1 188 4110 2.3 159 4111 7 141 4112 2.1 79 4113 5.2 27 4114 DG 17 4115 4.2 14 4116 4.3 11 4117 5.1 10 4018 1.2 0 4019 1.3 0 4020 1.4 0 40as the trucks are moving in either curved or straight lines, the distancesand angles between the cameras and placards are changing, leading tovarious sizes and shapes of placards in the images. This dataset is usedto train our placard detection and classification models. The placardand code areas are cropped to train the code detection and recognitionmodels.5.3.2 Training and Testing SettingsIn our framework, there are four models to be trained, i.e., placard detection,placard classification, placard code detection, and code recognition.109CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESFeautureextractionPlacad detectionPlacard classificationCode detectionCode recognitionCropping placardsPlacarddetectionlossLWCE lossCodedetectionlossCoderecognitionlossTraining LossesInput imagesCropping codesFigure 5.3: The training losses of our framework. LWCE is our proposedlogarithmic weighted cross entropy. ©Ran Zhang, 2022. Image pro-vided by Canscan Inc [14]. Diagram created by Ran Zhang.Training LossesFig. 5.3 shows the training losses in our framework. The input images arefirst sent to feature extraction networks before the placard detection andclassification. Further, the placards and codes are cropped from the originalinput images for the training of code detection and code recognition networks.The training losses in our framework consist of the following parts: plac-ard detection loss, LWCE loss, code detection loss, and code recognition loss.The first two losses are in the visual information extraction process. Theyare for placard detection and recognition, of which networks are trained to-gether. We balance the placard detection loss and LWCE loss evenly, i.e.,the weights of these two losses are the same during the training process.The last two losses are in the textual information extraction process. Theyare designed for the placard code detection and recognition tasks, which are110CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODEStrained separately. Placard detection and code detection loss can be foundin [34]. The cross entropy loss [43] is used as the code recognition trainingloss.Training Settings in Visual Information ExtractionPlacard detection and classification are in visual information extraction pro-cesses. As the placard detection and classification models share the extractedfeature maps, they are trained together. In the preprocessing steps of inputdata, the shorter side of the input image is set to be 640 pixels, and the as-pect ratios are maintained. According to the collected placard images fromthe industry, the numbers of different placards are different from each other.Some placards appear frequently while others rarely show, resulting in animbalanced dataset. To fully utilize the data and evaluate the performancesfairly, we make the testing dataset balanced and put the rest of the imagesfor training, following the rule in [122]. We randomly select around 40 sam-ples from real-world data of each class for testing. As one shipping containermight be attached with multiple placards, selecting one image for testingmight bring multiple samples. Consequently, the numbers of each placard inour testing dataset are not strictly equal but are relatively balanced with thevariance of two samples. The number of each testing sample falls between40 and 42, while the number of training samples per class ranges from maxi-mumly 480 to minimally 0, as described in Table 5.2. In total, there are 2297originally collected images for training and 718 images for testing. The origi-nal image size is 1080 pixels in height and 1920 pixels in width. These imagesare directly used as the training and testing datasets of placard detection.Training Settings in Textual Information ExtractionPlacard code detection and recognition are in textual information extractionprocesses.The placard regions are cropped to train the placard code detection net-work. If the original images are from the training set, their cropped placard111CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESregions are also kept in the training set. The same rule goes for the testingset of cropped images. The cropped placard images are resized to 128×128in pixels before sending to the placard detection model training. The totalnumber of cropped placard images is 3289 in training and 817 in testing.Regarding the cropped code regions, there are 2368 training images and1594 testing images from real-world environments. Our proposed optimizedcode recognition network is trained on the mixed dataset of cropped placardcode images and numerical synthetic text images. The training and testingimage numbers across code detection and recognition tasks might not bekept the same because one original image might have multiple placards, andsome types of placards, i.e., MP, LQ and GD, do not have numerical placardcodes.Evaluation MetricsThe evaluation is performed on the real-world placard dataset. The widelyused metrics, including precision, recall, mean average precision (mAP), andf1-score, are used to evaluate the performances of end-to-end placard de-tection and recognition. Frame per second (FPS) measures the processingspeeds. These evaluation metrics are chosen for their practical values inindustrial applications. For comparison, the other state-of-the-art objectdetection methods, including Faster R-CNN [34], Cascade R-CNN [36], Reti-naNet [83], YOLOv5s, and YOLOv5x [35], are taken into consideration inour experiments.5.3.3 Experimental ResultsQualitative ResultsThe qualitative results of state-of-the-art object detection methods over asample testing image that contains the open class placard, 1.3, are shown inFig. 5.4, 5.5, and 5.6. Our proposed framework detects and recognizes theplacard correctly, in other words, only true positive predictions are made.Although other methods could detect the placard, they cannot recognize it112CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESTable 5.3: Comparison with other state-of-the-art object detection methodson the whole testing datasetMethod p r f1 mAPSpeed(FPS)YOLOv5s [35] (2021) 0.7833 0.8278 0.7930 0.7853 23.47YOLOv5x [35] (2021) 0.7887 0.8340 0.7997 0.8041 15.24RetinaNet [83] (2017) 0.6745 0.8352 0.7343 0.7968 11.49Faster R-CNN [34] (2015) 0.7704 0.8164 0.7755 0.7818 11.21Cascade RCNN [36] (2018) 0.7946 0.8116 0.7870 0.7777 9.26Ours 0.9527 0.9355 0.9277 0.9284 6.04Table 5.4: Comparison with other state-of-the-art object detection methodson closed and open classesMethod Classes p r f1 mAPYOLOv5s [35] (2021) Closed 0.9215 0.9739 0.9330 0.9239YOLOv5x [35] (2021) Closed 0.9278 0.9812 0.9408 0.9460RetinaNet [83] (2017) Closed 0.7935 0.9826 0.8639 0.9374Faster R-CNN [34] (2015) Closed 0.9064 0.9605 0.9124 0.9197Cascade R-CNN [36] (2018) Closed 0.9348 0.9548 0.9259 0.9150Ours Open 0.9535 0.6355 0.7134 0.6358Ours Closed 0.9526 0.9856 0.9645 0.9800correctly, thus, generating one or more false-positive predictions, even withhigh confidence, e.g., class of 3 with the confidence of 1.0 in the results ofFaster R-CNN and class of 1.1 with the confidence of 0.97 in the predictionsof YOLOv5x.Overall Quantitative ResultsTable 5.3 demonstrates the superiority of our framework’s performance whencompared with other methods on the whole placard testing dataset. Whenit is evaluated on entire placard classes, our proposed framework achievesthe best performances on precision p, recall r, mean average precision mAPand f1-score f1, outperforming other methods with significant margins. Thereasons for the inferior performances of prior deep learning-based methods113CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODES(a) YOLOv5s(b) YOLOv5xFigure 5.4: Qualitative results. (a) and (b) show results from YOLOV5sand YOLOV5x, respectively. In the result images, the detected plac-ards are illustrated with their bounding boxes. ©Ran Zhang, 2022.Image provided by Canscan Inc [14]. Diagram created by Ran Zhang.114CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODES(a) RetinaNet(b) Faster R-CNNFigure 5.5: Qualitative results. (a) and (b) show results from RetinaNetand Faster R-CNN, respectively. In the result images, the detectedplacards are illustrated with their bounding boxes. ©Ran Zhang,2022. Image provided by Canscan Inc [14]. Diagram created by RanZhang.115CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODES(a) Cascade R-CNN(b) OursFigure 5.6: Qualitative results. (a) and (b) show results from Cascade R-CNN and ours, respectively. In the result images, the detected plac-ards are illustrated with their bounding boxes. ©Ran Zhang, 2022.Image provided by Canscan Inc [14]. Diagram created by Ran Zhang.116CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESare twofold, i.e., performance degradation on rarely seen classes, and morecrucially, failure to detect and recognize the open classes, which results ina significant amount of false predictions over open classes. Although theprocessing speed of our framework, 6.04 FPS, is lower than that of othermethods, it substantially meets the industry requirement since a truck needsat least 5.49 seconds to pass the gate.Table 5.4 shows the performances of various methods in closed and openclasses, respectively. The closed and open classes refer to the classes with andwithout training samples, respectively. As existing object detection methods[34–36, 83] cannot recognize the placards of open classes, their end-to-endprecision, recall, f1-score, and mAP are all 0 in our experimental results.Thus, the results of these methods on open classes are not shown in Table 5.4.To sum up, the quantitative results on closed classes demonstrated that ourproposed placard detection and recognition framework obtains the best pre-cision, recall, mAP, and f1-score, compared with other state-of-the-art objectdetection methods. More importantly, our framework can be applied to openclasses with a precision as high as 0.9535. Although the performances of Re-call, mAP, and f1-score in open classes are lower than those in closed classes,they are pioneering since they are in open classes and can be applied toreal-world applications.Quantitative Results Per ClassThe precision, recall, AP, and f1-score per class of different methods arecompared in Fig 5.7 and 5.7, where the training numbers of all classes aresorted from high to low on the x-axis. The results on the open classes (in-dex from 18 to 20) validated that our framework opens up the performanceon unseen classes, compared with other state-of-the-art deep learning-basedmethods. On the rarely seen class of index 17 (with only 10 training samplesshown in Table 5.2), our framework outperforms others by a great margin.In the classes with enough training data, our framework could either keepthe comparable performance or improve it significantly, e.g., for class index117CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODES1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Class Indexes0.00.20.40.60.81.0Precision Per ClassYOLOv5sYOLOv5xRetinaNetFaster R-CNNCascade R-CNNOurs(a)1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Class Indexes0.00.20.40.60.81.0Recall Per ClassYOLOv5sYOLOv5xRetinaNetFaster R-CNNCascade R-CNNOurs(b)Figure 5.7: (a) Precision (b) Recall of each class. ©Ran Zhang, 2022. Di-agram created by Ran Zhang.118CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODES1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Class Indexes0.00.20.40.60.81.0AP Per ClassYOLOv5sYOLOv5xRetinaNetFaster R-CNNCascade R-CNNOurs(a)1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Class Indexes0.00.20.40.60.81.0F1 Per ClassYOLOv5sYOLOv5xRetinaNetFaster R-CNNCascade R-CNNOurs(b)Figure 5.8: (a) Average precision (b) f1-score of each class. ©Ran Zhang,2022. Diagram created by Ran Zhang.119CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESTable 5.5: Performances of our framework using different placard detectorsPlacard Detector p r f1 mAPSpeed(FPS)Cascade R-CNN [36] (2018) 0.9547 0.9279 0.9245 0.9276 5.35Faster R-CNN [34] (2015) 0.9527 0.9355 0.9277 0.9284 6.04Table 5.6: The performances of our framework using different code recog-nizersText Recognizer p r f1 mAPSpeed(FPS)CRNN [41] (2017) 0.9481 0.9241 0.9186 0.9098 5.46GRCNN [42] (2017) 0.9408 0.9054 0.9041 0.8839 5.41CLOVA [43] (2019) 0.9433 0.9204 0.9150 0.9056 4.91Ours 0.9527 0.9355 0.9284 0.9277 6.047.5.3.4 Ablation StudiesThe Effects of Different Placard DetectorsIn our framework, the placard detector can be implemented either by FasterR-CNN [34] or Cascade R-CNN [36]. To compare the difference, they aretrained with the placard classification using our proposed LWCE loss. Therest of the components of our framework, including placard code detectionand recognition models, are kept the same. The evaluated performances overall classes are shown in Table 5.5. When our framework leverages CascadeR-CNN as the placard detector, it gains slightly in precision but sacrificesmarginally in the recall, mAP, f1-score, and processing speed. Overall, theresults are similar when different R-CNN-based detectors are adopted as theplacard detection model in our framework.120CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESThe Effects of Different Code RecognizerTo validate the effectiveness of our proposed optimized code recognitionmodel, the ablation study of using different text recognition models in ourframework is conducted. The rest of the structures in our framework are keptthe same. They are trained separately with the same experimental setup andevaluated on the whole testing dataset of all classes. As shown in Table 5.6,our optimized text recognition model outperforms other state-of-the-art textrecognition models, including CRNN [41], GRCNN [42], and CLOVA [43], onprecision, recall, mAP, and f1-score, when it is equipped as the placard coderecognizer. Our overall framework also runs the fastest when the optimizedrecognizer is applied. The experimental results validate that our proposedrecognizer is effectively optimized for placard code recognition.5.4 SummaryEnsuring the safety of transportation can promote the efficiency of globalsupply chain and support global trading. Hazardous material placards areessential safety signs that clearly indicate the properties of the dangerousfreight inside shipping containers that are transported all over the world.Thus, the automated placard recognition system is strongly desired by thetransportation industry.This chapter proposes a visual and textual information fusion-based zero-shot framework for hazardous material placard detection and recognition inthe context of the imbalanced and open class dataset, which is a commoncharacteristic of real-world data. During the training of the placard clas-sification model, the logarithmic weighted cross entropy loss is proposed tobalance the classes of different numbers. In the inference stage, the visual andtextual information are extracted separately; then, the logarithmic weightedconfidence fusion strategy is designed to fuse the information. The proposedframework aims to enhance the performance of the closed classes in the train-ing set, and more importantly, open up the recognition of open classes. The121CHAPTER 5. SPOTTING HAZARDOUS MATERIAL PLACARD CODESexperimental results demonstrated that our framework outperforms otherstate-of-the-art methods in closed classes and pioneers the performance inopen classes.Overall, our framework provides a practical and viable artificial intelligence-based solution that simultaneously detects and recognizes various kinds ofhazardous material placards. It will significantly enhance the efficiency andsafety of hazardous freight management during transportation. And in abroader view, it will boost the global supply chain, and contribute to theincrease of the global economy.The main contributions of our work of spotting placard codes are asfollows:• Our framework establishes the connection between the visual and tex-tual information of placards, demonstrating the practical values of zero-shot learning in industrial applications.• To the best of our knowledge, our work is the first to tackle the challengeof imbalanced and open classes in placard datasets. With the proposedLWCE and LWCF, our framework eliminates the dependency on bal-anced and comprehensive placard datasets, which are usually difficultto obtain in real-world applications.• We optimize the placard code recognition neural network by designingan appropriate structure according to the unique properties of placardcodes, achieving a faster recognition speed and a higher accuracy thanconventional text recognition models.122CHAPTER 6. CONCLUSIONSChapter 6Conclusions6.1 Thesis SummaryThis thesis draws attention to the automated container code reading sys-tems, where deep learning-based container code spotting methods have beendesigned as the core algorithms.First, in order to establish more effective collaboration between the hor-izontal code detection and recognition procedures, an adaptive frameworkhas been proposed and explained in Chapter 3. Experiments have been con-ducted on real industrial data of container backdoor images, which capturethe horizontal codes. The experimental results have validated the effective-ness and practical values of our framework. Specifically, the predicted codelocations can be adjusted to find the most suitable positions for the recogni-tion procedure. With the maximum and minimum location boundaries, theframework could tolerate minor detection errors. Thus, the code recognitionmodel can traverse between these boundaries, gaining higher chances to per-fectly match the localization model. The recognition accuracy has improvedfrom 90.17%, the prior performance, to 93.33%.Second, in an effort to fully exploit the vertical features, a vertical textspotting model for vertical container codes has been proposed in Chapter 4.123CHAPTER 6. CONCLUSIONSIt aligns features vertically for code recognition so as to decode text from topto bottom. Notably, it incorporates two separate text recognition pipelinesto improve recognition performance. These two recognition pipelines areadaptively chosen by our designed decision strategy based on both probabilityand uncertainty of their predicted text sequences. Experiments carried outon real data of container and trainer images that have the vertical codeshave shown that our method achieves better overall end-to-end recognitionperformance than other state-of-the-art methods while keeping the processingspeed at a comparable level. The results have validated that our model couldbetter utilize features of vertical codes than other methods. The end-to-endf1-score has increased to 96.5%, from the previous best, 93.5%.Finally, with the intention of handling imbalanced and open classes in realdatasets of placards, a zero-shot framework that fuses the textual and visualinformation for placard code spotting has been proposed in Chapter 5. Ourframework has been tested on real placard datasets. Though trained on acombination of the real placard and public synthetic text data, it conforms tothe standards of zero-shot frameworks, i.e., auxiliary data could be appliedto provide prerequisite information. The experimental results demonstratedthat our proposed framework, with an adaptive fusion strategy to take advan-tage of the two sets of information, could alleviate the effects of imbalancedclasses and open up the performance on new classes. The overall end-to-endrecognition f1-score has increased from 79.97% to 92.77%.Overall, the proposed container code spotting methods provide deep learn-ing solutions for the challenges of automatically reading different types ofcontainer codes. The transportation industry will benefit from the researchoutcomes to set up or enhance the digitalized container management.6.2 Research ContributionsThe main contributions of this research are summarized as follows:• A horizontal container code spotting method has been put forward. It124CHAPTER 6. CONCLUSIONSacts as the main role in automatically identifying containers when thehorizontal codes are in the camera views. This method will contributeto the identification of the principle container codes when the containeris viewed from the rear.• A vertical container code spotting method has been proposed. It wouldbecome the first choice to identify containers when horizontal codes arenot available. This method will contribute to the side view containercode identification, thus, increasing the robustness of automated con-tainer code reading systems.• A placard code spotting method has been designed. It can spot all typesof placard codes even with limited training data. It will contribute tothe safety monitoring of container transportation.To sum up, these methods will empower the digitalization of logisticsmanagement and enhance the safety of intermodal container transportation,thus contributing to the acceleration of the global supply chain and interna-tional trading.6.3 Research Strengths and PotentialApplicationsThe strengths of this research are listed as follows:• First is robustness. The proposed horizontal and vertical container codespotting methods provide the algorithms for automated container codereading systems to detect and recognize codes of different styles. Thisis of significant importance to the robustness of the system, especiallyin situations when one of the forms, either horizontal or vertical style, isnot available. In addition, these methods have been trained and testedwith real industry images collected under various outdoor conditions,which also demonstrated the robustness of our methods.125CHAPTER 6. CONCLUSIONS• Second is equipment installation. The proposed methods only requirethe images collected by RGB cameras, which can be easily installed onor near the gates that shipping containers pass through. More conve-niently, the container management departments can reuse surveillancecameras if they have already been installed.• Third is automation. The proposed container code spotting methodsare fully automated without human intervention. They can be runningall the time and save the digital results to either the cloud or localstorage.The potential applications are twofold, direct applications and relatedapplications. Direct applications include automated horizontal and verticalshipping container code reading, as well as automated placard identification.As our proposed methods are trained with real images of container codes,they can be directly applied without any modifications in these tasks. Fur-ther, our research could potentially inspire related applications of automatedtext reading systems in industry. First, the adaptive framework proposedin Chapter 3 could be applied in other applications by changing the cod-ing regulation. For instance, railcar numbers must conform to their codingregulations instead of ISO 6346 [20] for container codes. This strategy canbe applied if the corresponding coding regulations are specified. Second, thevertical text spotting model proposed in Chapter 4 can be applied in spottingother vertical text, for example, vertical street numbers. Third, the placardcode spotting framework in Chapter 5 can be applied to enhance recognitionperformance in other applications by fusing textual and visual information.A typical example is traffic sign text recognition. Our proposed confidencefusion strategy can be applied to fuse confidence from text recognition andvisual object recognition of traffic signs to increase recognition accuracy.6.4 Research Limitations and Future WorkThe limitations of this research are listed as follows:126CHAPTER 6. CONCLUSIONS• First, deep learning-based methods require a very large amount of train-ing data. To build robust automated container code reading systems,the training images are expected to cover different outdoor environ-ments.• Second, our proposed container code spotting methods focus on imagesonly. Video processing techniques have not been studied.Accordingly, future work can be conducted in the following aspects:• First is to combine external publicly available data with private datato reduce the required amount of private data. Although the syntheticdata have been leveraged as the auxiliary data in Chapter 5, they areonly used to train the code recognition model. The use of auxiliary datafor code detection needs to be exploited. In addition, how to utilize theexternal real text data, instead of synthetic data only, is still an openproblem.• Second is to investigate video processing and information fusion acrossdifferent frames. As the trucks are moving during the video stream-ing or image collection process, codes captured in images vary in size.Consequently, the recognized code results could be different amongconsecutive frames even for the same shipping container. In the future,video processing techniques and information fusion strategies acrossframes are expected to be studied to reduce unnecessary informationand improve overall recognition accuracy.127BIBLIOGRAPHYBibliography[1] R. Zhang, Z. Bahrami, T. Wang, and Z. Liu, “An adaptive deeplearning framework for shipping container code localization andrecognition,” IEEE Trans. Instrum. Meas., vol. 70, pp. 1–13, 2021. →pagesviii, xvi, xvii, xviii, xix, xx, 3, 30, 31, 36, 37, 40, 41, 44, 46, 47, 54, 55, 56, 58, 60, 62[2] R. Zhang, Z. Bahrami, and Z. Liu, “A vertical text spotting model fortrailer and container codes,” IEEE Trans. Instrum. Meas., vol. 70,pp. 1–13, 2021. → pagesviii, xxi, xxii, xxiii, 24, 67, 70, 73, 79, 83, 86, 88, 89, 90, 91, 92[3] R. Zhang, Z. Bahrami, K. Feng, and Z. Liu, “A vertical text spottingmodel for trailer and container codes,” IEEE Trans. Arti. Intell.,(submitted on July 21, 2022). → page viii[4] R. Zhang, Z. Bahrami, and Z. Liu, “An effective method forautomated railcar number detection and recognition based on deeplearning,” in 2021 Canadian Conference on Electrical and ComputerEngineering (CCECE), Sep. 12-17 2021. → page viii[5] R. Zhang, J. Bin, Z. Liu, and E. Blasch, “Image translation toenhance ir2vis image registration,” in Geospatial Informatics XI, Apr.14 2021. → page viii[6] R. Zhang, J. Bin, Z. Liu and E. Blasch, “Chapter 13 - WGGAN: AWavelet-Guided Generative Adversarial Network for Thermal ImageTranslation,” in Generative Adversarial Networks for Image-to-ImageTranslation, A. Solanki, A. Nayyar, and M. Naved, Eds. AcademicPress, 2021, pp. 313–327. → page viii128BIBLIOGRAPHY[7] Z. Bahrami, R. Zhang, T. Wang, and Z. Liu, “An end-to-endframework for shipping container corrosion defect inspection,” IEEETrans. Instrum. Meas., 2022. → page viii[8] Z. Bahrami, R. Zhang, R. Rayhana, and Z. Liu, “An hrcr-cnnframework for automated security seal detection on the shippingcontaine,” IEEE Trans. Instrum. Meas., vol. 70, pp. 1–13, 2021. →page viii[9] J. Lin, J. A. Fernandez, R. Rayhana, A. Zaji, R. Zhang, O. E.Herrera, Z. Liu, and W. Merida, “Predictive analytics for buildingpower demand: Day-ahead forecasting and anomaly prediction,”Energy Build., vol. 255, 2022. → page viii[10] J. Bin, R. Zhang, K. Hewag, and Z. Liu, “An iterative visible-infraredimage fusion framework based on wavelet autoencoder,” in 10thInternational Conference on Computer and CommunicationsManagement (ICCCM), Okayama University, Jul. 29-31 2022. →page viii[11] J. Bin, R. Zhang, C. Zhang, X. Liu, S. Du, E. Blasch, and Z. Liu,“Improved object detection in thermal imaging through contextenhancement and information fusion: A case study in autonomousdriving,” in Conference on Neural Information Processing Systems(NeurIPS) 2021, Workshop on Machine Learning for AutonomousDriving, Dec. 13 2021. → page viii[12] Z. Bahrami, R. Zhang, R. Rayhana, T. Wang, and Z. Liu, “Optimizeddeep neural network architectures with anchor box optimization forshipping container corrosion inspection,” in 2020 IEEE SymposiumSeries on Computational Intelligence (SSCI), Dec. 1-4 2020. → pageviii[13] Z. Bahrami, R. Zhang, R. Rayhana, and Z. Liu, “Deep learning-basedframework for shipping container security seal detection,” in 2021Joint 10th International Conference on Informatics, Electronics &Vision (ICIEV) and 2021 5th International Conference on Imaging,Vision & Pattern Recognition (icIVPR), 2021, pp. 1–7. → page viii129BIBLIOGRAPHY[14] Canscan Inc. Images are provided by Canscan Inc. to the author ofthis dissertation, Ran Zhang, who holds the copyrights of theseimages. Accessed on Dec. 7, 2022. [Online]. Available:https://www.linkedin.com/company/canscan/ → pagesxvi, xvii, xxiii, xxiv, 4, 6, 7, 13, 102, 110, 114, 115, 116[15] Intermodal container - Wikipedia. Accessed on May 11, 2020.[Online]. Available:https://en.wikipedia.org/wiki/Intermodal container → page 1[16] Intermodal freight transport - Wikipedia. Accessed on May 11, 2020.[Online]. Available:https://en.wikipedia.org/wiki/Intermodal freight transport → page 1[17] How many shipping containers are there in the world? Accessed onJuly 26, 2022. [Online]. Available:https://www.budgetshippingcontainers.co.uk/info/how-many-shipping-containers-are-there-in-the-world/ → page 1[18] Z. Liu, H. Ukida, K. Niel, and P. Ramuhalli, Eds., Integrated Imagingand Vision Techniques for Industrial Inspection. London: Springer,2015. → page 2[19] S. Gaweesh, M. N. Khan, and M. M. Ahmed, “Development of anovel framework for hazardous materials placard recognition systemto conduct commodity flow studies using artificial intelligence alexnetconvolutional neural network,” Transp. Res. Rec., vol. 2675, no. 11,pp. 1357–1371, 2021. → pages 2, 8, 11, 23, 100[20] ISO 6346:1995 Freight containers - Coding, identification andmarking. Accessed on Feb. 20, 2020. [Online]. Available:https://www.iso.org/standard/20453.html → pages 2, 3, 28, 126[21] T. Canada. (2019) Dangerous goods placards. [Online]. Available:https://tc.canada.ca/en/dangerous-goods/dangerous-goods-placards→ page 8[22] S. Kumano, K. Miyamoto, M. Tamagawa, H. Ikeda, and K. Kan,“Development of a container identification mark recognition system,”Electron. Commun. Jpn. (Part II: Electron.), vol. 87, no. 12, pp.38–50, 2004. → pages 9, 19, 20130BIBLIOGRAPHY[23] W. Wu, Z. Liu, M. Chen, X. Yang, and X. He, “An automated visionsystem for container-code recognition,” Expert Syst. Appl., vol. 39,no. 3, pp. 2842–2855, 2012. → pages 20, 21[24] K. Kim, Y. W. Woo, and H. Yang, “An intelligent system forcontainer image recognition using art2-based self-organizingsupervised learning algorithm,” in Proc. Simulated Evol. and Learn.:6th Int. Conf., Hefei, China, Oct. 15-18, 2006, pp. 897–904. → pages19, 20[25] L. Mei, J. Guo, Q. Liu, and P. Lu, “A novel framework for containercode-character recognition based on deep learning and templatematching,” in Proc. Int. Conf. Ind. Inf. Comput. Technol., Intell.Technol., Ind. Inf. Integr., Wuhan, Hubei, China, Dec. 3-4, 2016, pp.78–82. → pages 9, 20[26] X. Zhou, C. Yao, H. Wen, Y. Wang, S. Zhou, W. He, and J. Liang,“EAST: an efficient and accurate scene text detector,” in Proc.CVPR, Honolulu, HI, USA, Jul. 21-26, 2017, pp. 2642–2651. → pages9, 16, 17, 20, 22, 27, 41, 59, 80, 85, 87[27] W. Wang, E. Xie, X. Li, W. Hou, T. Lu, G. Yu, and S. Shao, “Shaperobust text detection with progressive scale expansion network,” inProc. CVPR, Long Beach, CA, USA, Jun. 16-20, 2019, pp.9336–9345. → pages 17, 22, 27, 59, 80, 85, 87[28] P. Lyu, M. Liao, C. Yao, W. Wu, and X. Bai, “Mask textspotter: Anend-to-end trainable neural network for spotting text with arbitraryshapes,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 2, pp.532–548, 2021. → pages9, 10, 11, 18, 22, 24, 27, 45, 59, 72, 80, 81, 85, 87, 88, 93, 94, 99[29] M. Liao, G. Pang, J. Huang, T. Hassner, and X. Bai, “Masktextspotter v3: Segmentation proposal network for robust scene textspotting,” in Proc. ECCV, A. Vedaldi, H. Bischof, T. Brox, andJ. Frahm, Eds., vol. 12356. Glasgow,UK: Springer, Aug. 23-28, 2020,pp. 706–722. → pages 10, 22, 80, 81, 85, 87, 88, 93, 94[30] Y. Liu, H. Chen, C. Shen, T. He, L. Jin, and L. Wang, “ABCNet:Real-time scene text spotting with adaptive Bezier-curve network,” in131BIBLIOGRAPHYProc. CVPR, Seattle, WA, USA, Jun. 13-19, 2020, pp. 9806–9815. →pages 10, 11, 16, 18, 20, 22, 24, 80, 81, 85, 87, 88, 93, 94, 99[31] A. W. Haddad, S. Huange, M. Boutin, and E. J. Delp, “Detection ofsymmetric shapes on a mobile device with applications to automaticsign interpretation,” in Multimedia on Mobile Devices 2012; andMultimedia Content Access: Algorithms and Systems VI, C. G. M.Snoek, N. Sebe, L. Kennedy, R. Creutzburg, and D. Akopian, Eds.,vol. 8304, International Society for Optics and Photonics. SPIE,2012, pp. 135 – 147. → page 11[32] B. Zhao, A. Parra, and E. J. Delp, “Mobile-based hazmat signdetection and recognition,” in 2013 IEEE Global Conference onSignal and Information Processing, 2013, pp. 735–738.[33] B. Zhao, “Image analysis using visual saliency with applications inhazmat sign detection and recognition,” Ph.D. dissertation, PurdueUniversity, 2014. → pages 11, 20[34] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: towardsreal-time object detection with region proposal networks,” in Proc.NeurIPS, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, andR. Garnett, Eds., Montreal, Quebec, Canada, Dec. 7-12, 2015, pp.91–99. → pages11, 24, 26, 67, 69, 77, 87, 94, 100, 102, 104, 111, 112, 113, 117, 120[35] Ultralytics. (2021) Yolov5. [Online]. Available:https://github.com/ultralytics/yolov5 → pages 22, 23, 112, 113[36] Z. Cai and N. Vasconcelos, “Cascade R-CNN: delving into highquality object detection,” in Proc. CVPR, Salt Lake City, UT, USA,Jun. 18-22, 2018, pp. 6154–6162. → pages11, 24, 67, 68, 69, 77, 93, 100, 102, 112, 113, 117, 120[37] R. Mondal, S. Bhowmik, and R. Sarkar, “tsegGAN: A generativeadversarial network for segmenting touching nontext componentsfrom text ones in handwriting,” IEEE Trans. Instrum. Meas., vol. 70,pp. 1–10, 2021. → page 16[38] Z. He, J. Liu, H. Ma, and P. Li, “A new automatic extraction methodof container identity codes,” Proceedings of the 2003 IEEE132BIBLIOGRAPHYInternational Conference on Intelligent Transportation Systems,vol. 6, no. 1, pp. 72–78, 2003. → pages 16, 19[39] F. Gao, S. Li, H. You, S. Lu, and G. Xiao, “Text spotting for curvedmetal surface: Clustering, fitting, and rectifying,” IEEE Trans.Instrum. Meas., vol. 70, pp. 1–12, 2021. → page 16[40] H. Liang, X. Sun, Y. Sun, and Y. Gao, “Text feature extraction basedon deep learning: a review,” EURASIP J. Wirel. Commun. Netw.,vol. 2017, p. 211, 2017. → page 16[41] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural networkfor image-based sequence recognition and its application to scene textrecognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 11,pp. 2298–2304, 2017. → pages 16, 17, 20, 22, 28, 45, 59, 105, 120, 121[42] J. Wang and X. Hu, “Gated recurrent convolution neural network forOCR,” in Proc. NeurIPS, Long Beach, CA, USA, Dec. 4-9, 2017, pp.335–344. → pages 17, 20, 22, 27, 45, 59, 105, 120, 121[43] J. Baek, G. Kim, J. Lee, S. Park, D. Han, S. Yun, S. J. Oh, andH. Lee, “What is wrong with scene text recognition modelcomparisons? Dataset and model analysis,” in Proc. ICCV, Seoul,Korea (South), Oct. 27 - Nov. 2, 2019, pp. 4714–4722. → pages17, 18, 104, 105, 111, 120, 121[44] B. Shi, M. Yang, X. Wang, P. Lyu, C. Yao, and X. Bai, “ASTER: anattentional scene text recognizer with flexible rectification,” IEEETrans. Pattern Anal. Mach. Intell., vol. 41, no. 9, pp. 2035–2048,2019. → pages 16, 17, 18, 105[45] B. Epshtein, E. Ofek, and Y. Wexler, “Detecting text in naturalscenes with stroke width transform,” in Proc. CVPR, San Francisco,CA, USA, Jun. 13-18, 2010, pp. 2963–2970. → page 16[46] C. Yao, X. Bai, W. Liu, Y. Ma, and Z. Tu, “Detecting texts ofarbitrary orientations in natural images,” in Proc. CVPR, RI, USA,Jun. 16-21, 2012, pp. 1083–1090. → pages 16, 17[47] L. Neumann and J. Matas, “A method for text localization andrecognition in real-world images,” in Proc. ACCV, ser. Lecture Notes133BIBLIOGRAPHYin Computer Science, R. Kimmel, R. Klette, and A. Sugimoto, Eds.,vol. 6494, Queenstown, New Zealand, Nov. 8-12, 2010, pp. 770–783.→ page 16[48] B. S. Minghui Liao and X. Bai, “TextBoxes++: A single-shotoriented scene text detector,” IEEE Transactions on ImageProcessing, vol. 27, no. 8, pp. 3676–3690, 2018. → page 17[49] M. Liao, Z. Wan, C. Yao, K. Chen, and X. Bai, “Real-time scene textdetection with differentiable binarization,” in Proc. AAAI, HiltonNew York Midtown, New York, New York USA, Feb. 7-12, 2020. →page 17[50] Z. Tian, W. Huang, T. He, P. He, and Y. Qiao, “Detecting text innatural image with connectionist text proposal network,” in Proc.ECCV, ser. Lecture Notes in Computer Science, B. Leibe, J. Matas,N. Sebe, and M. Welling, Eds., vol. 9912, Amsterdam, TheNetherlands, Oct. 11-14, 2016, pp. 56–72. → page 17[51] T. Lin, P. Dolla´r, R. B. Girshick, K. He, B. Hariharan, and S. J.Belongie, “Feature pyramid networks for object detection,” in Proc.CVPR, Honolulu, HI, USA, Jul. 21-26, 2017, pp. 936–944. → pages17, 67, 68, 69, 93[52] R. Smith, “An overview of the tesseract OCR engine,” in Proc.ICDAR. Curitiba, Parana´, Brazil: IEEE Computer Society, Sep.23-26, 2007, pp. 629–633. → page 17[53] Z. Cheng, Y. Xu, F. Bai, Y. Niu, S. Pu, and S. Zhou, “AON: towardsarbitrarily-oriented text recognition,” in Proc. CVPR, Salt Lake City,UT, USA, Jun. 18-22, 2018, pp. 5571–5579. → pages 17, 18[54] C. Choi, Y. Yoon, J. Lee, and J. Kim, “Simultaneous recognition ofhorizontal and vertical text in natural images,” in Computer Vision -ACCV 2018 Workshops - 14th Asian Conference on ComputerVision, Perth, Australia, Dec. 2-6, 2018, Revised Selected Papers, ser.Lecture Notes in Computer Science, G. Carneiro and S. You, Eds.,vol. 11367. Springer, 2018, pp. 202–212. → pages 17, 18, 22134BIBLIOGRAPHY[55] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in Proc. CVPR, Las Vegas, NV, USA, Jun. 27-30,2016, pp. 770–778. → pages 18, 28, 29, 68[56] W. Liu, C. Chen, K. K. Wong, Z. Su, and J. Han, “Star-net: Aspatial attention residue network for scene text recognition,” in Proc.the British Machine Vision Conference (BMVC), R. C. Wilson, E. R.Hancock, and W. A. P. Smith, Eds., York, UK, Sep. 19-22, 2016. →page 18[57] F. Borisyuk, A. Gordo, and V. Sivakumar, “Rosetta: Large scalesystem for text detection and recognition in images,” in Proc. 24thACM SIGKDD Int. Conf. on KDD, Y. Guo and F. Farooq, Eds.,London, UK, Aug. 19-23, 2018, pp. 71–79. → pages 18, 80, 85, 87, 94[58] K. Simonyan and A. Zisserman, “Very deep convolutional networksfor large-scale image recognition,” in Proc. 3rd InternationalConference on Learning Representations (ICLR), Y. Bengio andY. LeCun, Eds., San Diego, CA, USA, May 7-9, 2015. → page 18[59] H. Li, P. Wang, and C. Shen, “Towards end-to-end text spotting withconvolutional recurrent neural networks,” in Proc. ICCV, Venice,Italy, Oct. 22-29, 2017, pp. 5248–5256. → page 18[60] X. Liu, D. Liang, S. Yan, D. Chen, Y. Qiao, and J. Yan, “FOTS: fastoriented text spotting with a unified network,” in Proc. CVPR, SaltLake City, UT, USA, Jun. 18-22, 2018, pp. 5676–5685. → page 18[61] S. Qin, A. Bissacco, M. Raptis, Y. Fujii, and Y. Xiao, “Towardsunconstrained end-to-end text spotting,” in Proc. ICCV, Seoul, Korea(South), Oct. 27 - Nov. 2, 2019, pp. 4703–4713. → page 18[62] P. Wang, C. Zhang, F. Qi, S. Liu, X. Zhang, P. Lyu, J. Han, J. Liu,E. Ding, and G. Shi, “Pgnet: Real-time arbitrarily-shaped textspotting with point gathering network,” in Proc. AAAI, VirtualEvent, Feb. 2-9 2021, pp. 2782–2790. → pages 18, 19[63] A. Veit, T. Matera, L. Neumann, J. Matas, and S. J. Belongie,“Coco-text: Dataset and benchmark for text detection andrecognition in natural images,” CoRR, vol. abs/1601.07140, 2016. →page 19135BIBLIOGRAPHY[64] C. K. Ch’ng, C. S. Chan, and C. Liu, “Total-text: Towardsorientation robustness in scene text detection,” Proc. InternationalJournal on Document Analysis and Recognition (IJDAR), vol. 23, pp.31–52, 2020. → page 19[65] M. Goccia, M. Bruzzo, C. Scagliola, and S. G. Dellepiane,“Recognition of container code characters through gray-level featureextraction and gradient-based classifier optimization,” in SeventhInternational Conference on Document Analysis and Recognition,Edinburgh, Scotland, UK, Aug. 3-6, 2003, pp. 973–977. → pages19, 20, 21[66] W. Al-Khawand, S. Kadry, R. Bozzo, and S. Khaled, “8-neighborhoodvariant for a better container code extraction and recognition,”International Journal of Computer Science and Information Security(IJCSIS), vol. 14, pp. 182–186, Apr. 2016. → page 19[67] T. Szabo and G. Horvath, “Finite word length computational effectsof the principal component analysis networks,” IEEE Transactions onInstrumentation and Measurement, vol. 47, no. 5, pp. 1218–1222, Oct.1998. → page 19[68] X. Zhou, Y. Wang, C. Xiao, Q. Zhu, X. Lu, H. Zhang, J. Ge, andH. Zhao, “Automated visual inspection of glass bottle bottom withsaliency detection and template matching,” IEEE Transactions onInstrumentation and Measurement, vol. 68, no. 11, pp. 4253–4267,Nov. 2019. → page 19[69] W. Al-Khawand, S. Kadry, R. Bozzo, and S. Khaled, “8-neighborhoodvariant for a better container code extraction and recognition,” Int. J.Comput. Sci. Inf. Secur., vol. 14, pp. 182–186, Apr. 2016. → page 20[70] A. Verma, M. Sharma, R. Hebbalaguppe, E. Hassan, and L. Vig,“Automatic container code recognition via spatial transformernetworks and connected component region proposals,” in IEEEInternational Conference on Machine Learning and Applications(ICMLA), Anaheim, CA, USA, Dec. 18-20, 2016, pp. 728–733. →pages 20, 21136BIBLIOGRAPHY[71] C. Roeksukrungrueang, T. Kusonthammrat, N. Kunapronsujarit,T. N. Aruwong, and S. Chivapreecha, “An implementation ofautomatic container number recognition system,” in InternationalWorkshop on Advanced Image Technology (IWAIT), Chiang Mai,Thailand, Jan. 7-9, 2018, pp. 1–4. → pages 20, 21[72] Y. Liu, T. Li, L. Jiang, and X. Liang, “Container-code recognitionsystem based on computer vision and deep neural networks,” in AIPConference Proceedings, Xi’an, China, Jan. 20-21, 2018, p. 040118. →page 20[73] C. Hsin-Chen, C. Chih-Kai, H. Fu-Yu, L. Yu-San, Y.-T. Wu, andY.-N. Sun, “A computer vision system for automated container coderecognition,” Lecture Notes in Engineering and Computer Science,vol. 2188, 03 2011. → page 21[74] J. Canny, “A computational approach to edge detection,” IEEETrans. Pattern Anal. Mach. Intell., vol. PAMI-8, no. 6, pp. 679–698,1986. → pages 21, 23, 84[75] Y. Yoon, K.-D. Ban, H. Yoon, and J. Kim, “Automatic containercode recognition from multiple views,” ETRI Journal, vol. 38, no. 4,pp. 767–775, 2016. [Online]. Available:https://onlinelibrary.wiley.com/doi/abs/10.4218/etrij.16.0014.0069 →page 21[76] C. Li, S. Liu, Q. Xia, H. Wang, and H. Chen, “Automatic containercode localization and recognition via an efficient code detector andsequence recognition,” in 2019 IEEE/ASME International Conferenceon Advanced Intelligent Mechatronics (AIM), 2019, pp. 532–537. →page 22[77] W. Zhiming, W. Wuxi, and X. Yuxiang, “Automatic container coderecognition via faster-rcnn,” in 2019 5th International Conference onControl, Automation and Robotics (ICCAR), 2019, pp. 870–874. →page 22[78] A. W. Haddad, S. Huange, M. Boutin, and E. J. Delp, “Detection ofsymmetric shapes on a mobile device with applications to automaticsign interpretation,” in Multimedia on Mobile Devices 2012; and137BIBLIOGRAPHYMultimedia Content Access: Algorithms and Systems VI, C. G. M.Snoek, N. Sebe, L. Kennedy, R. Creutzburg, and D. Akopian, Eds.,vol. 8304, International Society for Optics and Photonics. SPIE,2012, pp. 135 – 147. [Online]. Available:https://doi.org/10.1117/12.908815 → pages 23, 100[79] B. Zhao, A. Parra, and E. J. Delp, “Mobile-based hazmat signdetection and recognition,” in Proc. GlobalSIP, 2013, pp. 735–738. →pages 23, 100[80] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,”CoRR, vol. abs/1804.02767, 2018. → pages 23, 25[81] A. Bochkovskiy, C. Wang, and H. M. Liao, “Yolov4: Optimal speedand accuracy of object detection,” CoRR, vol. abs/2004.10934, 2020.[82] C. Wang, A. Bochkovskiy, and H. M. Liao, “Scaled-yolov4: Scalingcross stage partial network,” in Proc. CVPR, 2021, pp. 13 029–13 038.→ page 23[83] T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dolla´r, “Focal loss fordense object detection,” in Proc. ICCV, 2017, pp. 2999–3007. →pages 23, 24, 100, 112, 113, 117[84] C. Zhang, Y. Tao, K. Du, W. Ding, B. Wang, J. Liu, and W. Wang,“Character-level street view text spotting based on deepmultisegmentation network for smarter autonomous driving,” IEEETrans. Artif. Intell., vol. 3, no. 2, pp. 297–308, 2022. → pages 24, 99[85] K. He, G. Gkioxari, P. Dolla´r, and R. B. Girshick, “Mask R-CNN,” inProc. ICCV, Venice, Italy, Oct. 22-29, 2017, pp. 2980–2988. → pages24, 26, 71[86] V. Kamat and S. Ganesan, “An efficient implementation of theHough transform for detecting vehicle license plates using dsp’s,” in1st IEEE Real-Time Technology and Applications Symposium,Chicago, Illinois, USA, May 15-17, 1995. IEEE Computer Society,1995, pp. 58–59. → page 25[87] H. Bai and C. Liu, “A hybrid license plate extraction method basedon edge statistics and morphology,” in 17th International Conference138BIBLIOGRAPHYon Pattern Recognition, ICPR 2004, Cambridge, UK, August 23-26,2004. IEEE Computer Society, 2004, pp. 831–834.[88] D. Zheng, Y. Zhao, and J. Wang, “An efficient method of licenseplate location,” Pattern Recognit. Lett., vol. 26, no. 15, pp.2431–2438, 2005. → page 25[89] S. Zhang, M. Zhang, and X. Ye, “Car plate character extractionunder complicated environment,” in Proceedings of the IEEEInternational Conference on Systems, Man & Cybernetics: TheHague, Netherlands, 10-13 October 2004. IEEE, 2004, pp.4722–4726. → page 25[90] Y. Huang, C. Chen, Y. Chang, and F. E. Sandnes, “An intelligentstrategy for checking the annual inspection status of motorcyclesbased on license plate recognition,” Expert Syst. Appl., vol. 36, no. 5,pp. 9260–9267, 2009. → page 25[91] J. Kim, “MSER and SVM-based vehicle license plate detection andrecognition system,” in Convergence and Hybrid InformationTechnology - 6th International Conference, ICHIT 2012, Daejeon,Korea, Aug. 23-25, 2012. Proceedings, ser. Communications inComputer and Information Science, G. Lee, D. Howard, D. Slezak,and Y. Hong, Eds., vol. 310. Springer, 2012, pp. 529–535. → page 25[92] J. Zhuang, S. Hou, Z. Wang, and Z. Zha, “Towards human-levellicense plate recognition,” in Computer Vision - ECCV 2018 - 15thEuropean Conference, Munich, Germany, Sep. 8-14, 2018,Proceedings, Part III, ser. Lecture Notes in Computer Science,V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., vol.11207. Springer, 2018, pp. 314–329. → page 25[93] S. M. Silva and C. R. Jung, “License plate detection and recognitionin unconstrained scenarios,” in Computer Vision - ECCV 2018 - 15thEuropean Conference, Munich, Germany, Sep. 8-14, 2018,Proceedings, Part XII, ser. Lecture Notes in Computer Science,V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., vol.11216. Springer, 2018, pp. 593–609. → page 25139BIBLIOGRAPHY[94] Y. Lee, J. Lee, H. Ahn, and M. Jeon, “SNIDER: single noisy imagedenoising and rectification for improving license plate recognition,” in2019 IEEE/CVF International Conference on Computer VisionWorkshops, ICCV Workshops 2019, Seoul, Korea (South), Oct.27-28, 2019. IEEE, 2019, pp. 1017–1026. → page 25[95] Government of Canada. Minister of transport issues revised order toimprove rail safety in the winter. Accessed on Apr. 30, 2021. [Online].Available:https://www.canada.ca/en/transport-canada/news/2020/11/minister-of-transport-issues-revised-order-to-improve-rail-safety-in-the-winter.html → page 25[96] S. Yang, R. Xu, Z. Zhou, Z. Xing, and Y. Zhang, “A rail trainnumber identification algorithm based on image processing,” in Proc.the 4th International Conference on Electrical and InformationTechnologies for Rail Transportation (EITRT), B. Liu, L. Jia, Y. Qin,Z. Liu, L. Diao, and M. An, Eds., Qingdao, China, Oct. 25-27, 2019,pp. 207–214. → page 25[97] X. Zou, Y. Fu, and X. Li, “Image feature recognition of railway truckbased on machine learning,” in Proc. of IEEE 3rd InformationTechnology, Networking, Electronic and Automation ControlConference (ITNEC), Chengdu, China, Mar. 15 - 17, 2019, pp.1549–1555. → page 26[98] A. Zabrodina, S. Frolovaa, and A. P. Glukhova, “Development of anautomated system for recognizing the parameters of a railwaycarriage (railway tanks),” in Proc. Models and Methods forResearching Information Systems in Transport 2020, St. Petersburg,Russia, Dec. 11-12, 2020. → pages 25, 26[99] G. Xiucai and C. Gongli, “Railway freight train number detectionmethod based on deep learning,” in Proc. IEEE 9th JointInternational Information Technology and Artificial IntelligenceConference (ITAIC), vol. 9, Chengdu, China, Dec. 11-13, 2020, pp.927–930. → page 26[100] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutionalnetworks for biomedical image segmentation,” in Medical Image140BIBLIOGRAPHYComputing and Computer-Assisted Intervention (MICCAI), Munich,Germany, Oct. 5-9, 2015, pp. 234–241. → page 28[101] H. Gao, W. Xu, J. Sun, and Y. Tang, “Multilevel thresholding forimage segmentation through an improved quantum-behaved particleswarm algorithm,” IEEE Trans. Instrum. Meas., vol. 59, no. 4, pp.934–946, Apr. 2010. → page 35[102] A. Graves, S. Ferna´ndez, F. J. Gomez, and J. Schmidhuber,“Connectionist temporal classification: labelling unsegmentedsequence data with recurrent neural networks,” in Proc. Int. Conf.Mach. Learn., Pittsburgh, Pennsylvania, USA, Jun. 25-29, 2006, pp.369–376. → page 45[103] S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997. → pages 46, 105[104] D. P. Kingma and J. Ba, “Adam: A method for stochasticoptimization,” in Proc. of 3rd Int. Conf. Learn. Representations,, SanDiego, CA, USA, May 7-9, 2015. → page 50[105] M. Everingham, L. V. Gool, C. K. I. Williams, J. M. Winn, andA. Zisserman, “The PASCAL visual object classes (VOC) challenge,”Int. J. Comput. Vis., vol. 88, no. 2, pp. 303–338, 2010. → page 51[106] Y. Port. Yantian international container terminals - port facilities -infrastructure - port traffic. Accessed on Jun. 19, 2020. [Online].Available:https://www.yict.com.cn/page/port-traffic.html?locale=en US →pages 63, 64[107] International Maritime Organization and Maritime SafetyCommittee, Revised recommendations on the safe transport ofdangerous cargoes and related activities in port areas, 3rd ed.London : IMO, 2007, p. 44. → pages 63, 64[108] Transportation Association of Canada and InterjurisdictionalCommittee on Vehicle Weights and Dimension, Heavy truck weightand dimension regulations for interprovincial operations in Canada :resulting from the federal-provincial-territorial Memorandum of141BIBLIOGRAPHYUnderstanding on Interprovincial Weights and Dimensions. Ottawa: Transportation Association of Canada, 1993, p. 13. → page 64[109] Z. Cheng, F. Bai, Y. Xu, G. Zheng, S. Pu, and S. Zhou, “Focusingattention: Towards accurate text recognition in natural images,” inProc. ICCV, Venice, Italy, Oct. 22-29, 2017, pp. 5086–5094. → pages72, 104, 105[110] K. Cho, B. van Merrienboer, C¸. Gu¨lc¸ehre, D. Bahdanau, F. Bougares,H. Schwenk, and Y. Bengio, “Learning phrase representations usingRNN encoder-decoder for statistical machine translation,” in Proc.EMNLP, A. Moschitti, B. Pang, and W. Daelemans, Eds. Doha,Qatar: ACL, Oct. 25-29, 2014, pp. 1724–1734. → page 73[111] D. Harris and S. Harris, Eds., Digital Design and ComputerArchitecture. Burlington, Massachusetts, US: Morgan Kaufmann,2012. → page 77[112] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, “Imagenet: Alarge-scale hierarchical image database,” in Proc. CVPR, Miami,Florida, USA, Jun. 20-25, 2009, pp. 248–255. → page 78[113] H. Lin, P. Yang, and F. Zhang, “Review of scene text detection andrecognition,” Arch. Comput. Methods Eng., vol. 27, p. 433–454, 2020.→ page 80[114] N. Otsu, “A threshold selection method from gray-level histograms,”IEEE Trans. Syst. Man Cybern. Syst., vol. 9, no. 1, pp. 62–66, 1979.→ page 84[115] R. C. Gonza´lez and R. E. Woods, Digital image processing, 3rdEdition. Pearson Education, 2008. → page 84[116] Y. Zhu, C. Ma, and J. Du, “Rotated cascade R-CNN: A shape robustdetector with coordinate regression,” Pattern Recognit., vol. 96, 2019.→ page 93[117] C. E. Shannon, “A mathematical theory of communication,” The BellSystem Technical Journal, vol. 27, no. 3, pp. 379–423, 1948. → page95142BIBLIOGRAPHY[118] B. Zhao, “Image analysis using visual saliency with applications inhazmat sign detection and recognition,” Ph.D. dissertation, PurdueUniversity, 2014. → page 100[119] T. Zhang, J. Zou, and W. Jia, “Fast and robust road sign detection indriver assistance systems,” Appl. Intell., vol. 48, p. 4113–4127, May2018. → page 100[120] W. Li, J. Chen, J. Cao, C. Ma, J. Wang, X. Cui, and P. Chen,“EID-GAN: Generative adversarial nets for extremely imbalanceddata augmentation,” IEEE Trans. Ind. Informat., pp. 1–10, 2022. →page 100[121] S. Das, S. S. Mullick, and I. Zelinka, “On supervised class-imbalancedlearning: An updated perspective and some key challenges,” IEEETrans. Artif. Intell., pp. 1–1, 2022. → page 100[122] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu,“Large-scale long-tailed recognition in an open world,” in Proc.CVPR, Long Beach, CA, USA, Jun. 16-20 2019, pp. 2537–2546. →pages 100, 111[123] S. Rahman, S. H. Khan, and N. Barnes, “Polarity loss for zero-shotobject detection,” Computing Research Repository, vol.abs/1811.08982, 2018. [Online]. Available:http://arxiv.org/abs/1811.08982 → pages 100, 101[124] T. Wang, Z. Zhang, and K.-L. Tsui, “A deep generative approach forrail foreign object detections via semi-supervised learning,” IEEETrans. Ind. Informat., pp. 1–1, 2022. → page 100[125] J. Wu, S. Liu, D. Huang, and Y. Wang, “Multi-scale positive samplerefinement for few-shot object detection,” in Proc. ECCV, ser.Lecture Notes in Computer Science, A. Vedaldi, H. Bischof, T. Brox,and J. Frahm, Eds., vol. 12361. Glasgow,UK: Springer, Aug. 23-282020, pp. 456–472. → page 100[126] Z. Li, H. Wang, T. Swistek, E. Yu, and H. Wang, “Efficient few-shotclassification via contrastive pre-training on web data,” IEEE Trans.Artif. Intell., pp. 1–1, 2022.143BIBLIOGRAPHY[127] G. Koch, R. Zemel, and R. Salakhutdinov, “Siamese neural networksfor one-shot image recognition,” in Proc. ICML, Lille, France, July 6 -11 2015.[128] J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen,Z. Liu, C. C. Loy, and D. Lin, “Seesaw loss for long-tailed instancesegmentation,” in Proc. CVPR, virtual, Jun. 19-25 2021, pp.9695–9704. → page 100[129] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Readingtext in the wild with convolutional neural networks,” Int. J. Comput.Vis., vol. 116, no. 1, pp. 1–20, 2016. → page 108144
Thesis/Dissertation
eng
Electrical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Liu, Zheng (Professor of engineering)
Spotting shipping container codes with deep learning methods
Text
Still Image