Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Fan, Yanshuo
Master of Applied Science - MASc
University of British Columbia
Lightning strikes are a critical concern for aircraft safety, necessitating the evaluation and maintenance of aircraft surface materials after each lightning event. Non-destructive evaluation/testing/inspection (NDE/NDT/NDI) methods play a vital role in assessing the structural condition of aircraft during maintenance procedures. However, each NDE technique has strengths and limitations when evaluating the stroked material. Therefore, the research in the thesis introduces novel image registration and fusion techniques that aim to overcome the limitations of individual NDE techniques and provide a comprehensive approach to aircraft material assessment. First, a structurally enhanced NDE image registration model is proposed, which can automatically align NDE images with ground-truth images. With the structural enhancement filter, the registration accuracy improves significantly. Compared to existing advanced models, the proposed model achieves reliable and consistent alignment results on multimodal NDE images without training. Then, a two-stage image fusion-translation network (FTnet) is proposed. This network combines an image fusion module with an image translation module to enhance the quality of the fused images. The fusion module employs a novel fusion strategy that yields excellent results despite its simple structure. Moreover, the image translation module optimizes the resulting image representation without requiring additional training. The fused images demonstrate a remarkable enhancement of the image information. FTnet surpasses existing methods without relying on large datasets. The model signifies its substantial advantages in the field of image fusion. The most remarkable advantage of the proposed models is their ability to provide reliable and efficient results in real industries with less reliance on large datasets. This makes these models more practical in the industry, where obtaining large amounts of data is challenging. Thus, the proposed models provide a promising way to improve NDE image results in industrial environments while minimizing data requirements.
REGISTRATION AND FUSION OFAIRCRAFT LIGHTNING DAMAGENDE DATAbyYanshuo FanB.Sc., The University of Post and Telecommunication, 2017A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Electrical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)August 2023© Yanshuo Fan, 2023The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:REGISTRATION AND FUSION OF AIRCRAFT LIGHTNINGDAMAGE NDE DATAsubmitted by Yanshuo Fan in partial fulfilment of the requirements of thedegree of Master of Applied ScienceDr. Zheng Liu, School of EngineeringSupervisorDr. Liwei Wang, School of EngineeringSupervisory Committee MemberDr. Peyman Yousefi, School of EngineeringSupervisory Committee MemberDr. Yang Cao, School of EngineeringUniversity ExamineriiAbstractLightning strikes are a critical concern for aircraft safety, necessitatingthe evaluation and maintenance of aircraft surface materials after each light-ning event. Non-destructive evaluation/testing/inspection (NDE/NDT/NDI)methods play a vital role in assessing the structural condition of aircraft dur-ing maintenance procedures. However, each NDE technique has strengthsand limitations when evaluating the stroked material. Therefore, the re-search in the thesis introduces novel image registration and fusion tech-niques that aim to overcome the limitations of individual NDE techniquesand provide a comprehensive approach to aircraft material assessment.First, a structurally enhanced NDE image registration model is pro-posed, which can automatically align NDE images with ground-truth im-ages. With the structural enhancement filter, the registration accuracy im-proves significantly. Compared to existing advanced models, the proposedmodel achieves reliable and consistent alignment results on multimodal NDEimages without training.Then, a two-stage image fusion-translation network (FTnet) is proposed.This network combines an image fusion module with an image translationmodule to enhance the quality of the fused images. The fusion module em-ploys a novel fusion strategy that yields excellent results despite its simplestructure. Moreover, the image translation module optimizes the result-ing image representation without requiring additional training. The fusedimages demonstrate a remarkable enhancement of the image information.FTnet surpasses existing methods without relying on large datasets. Themodel signifies its substantial advantages in the field of image fusion.The most remarkable advantage of the proposed models is their abilityto provide reliable and efficient results in real industries with less relianceiiion large datasets. This makes these models more practical in the industry,where obtaining large amounts of data is challenging. Thus, the proposedmodels provide a promising way to improve NDE image results in industrialenvironments while minimizing data requirements.ivLay SummaryLightning strikes usually result in severe damage to aircraft compositematerials. Therefore, a precise assessment of the extent of this damage,including its depth and contour characteristics, is crucial while avoidingfurther harm. Although existing non-destructive inspection techniques holdvalue, they may not consistently offer an accurate evaluation of the completedamage extent, especially in complex aircraft structures. To address thislimitation, this research focuses on enhancing the damage depth and con-tour information by combining multiple inspection techniques. The researchresults will provide reliable results in the real-world aviation industries.vPrefaceThis thesis is based on the research work conducted in the School of En-gineering at The University of British Columbia, Okanagan Campus, underthe supervision of Prof. Zheng Liu. Published works are contained in thisthesis.Chapter 4 is going to be submitted for consideration in the form of anIEEE-refereed paper.Chapter 5 is based on the following published paper and used with thepermission of SPIE.• Yanshuo Fan, Rakiba Rayhana, Yue Cao, Catalin Mandache, andZheng Liu ”A multimodal fusion-based autoencoder for non-destructiveevaluation of aircraft structures”, Proc. SPIE 12489, NDE 4.0, Predic-tive Maintenance, Communication, and Energy Systems: The DigitalTransformation of NDE, 124890M (25 April 2023).I am the principal contributor to the work in this thesis. Prof. ZhengLiu provided me with some research ideas and suggestions to improve myresearch works. Dr. Catalin Mandache helped me with data acquisition.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . .xviiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Background and Motivation . . . . . . . . . . . . . . . . . . . 11.2 Research Gaps and Objectives . . . . . . . . . . . . . . . . . 21.3 Thesis Outline and Organization . . . . . . . . . . . . . . . . 5Chapter 2: NDE Techniques for Aircraft Composite Materials 72.1 Aircraft Composite Materials . . . . . . . . . . . . . . . . . . 72.2 NDE Techniques . . . . . . . . . . . . . . . . . . . . . . . . . 102.2.1 Eddy Current Testing . . . . . . . . . . . . . . . . . . 102.2.2 Infrared Pulsed Thermography . . . . . . . . . . . . . 122.2.3 Ultrasonic Testing . . . . . . . . . . . . . . . . . . . . 132.2.4 X-ray Radiography . . . . . . . . . . . . . . . . . . . . 16vii2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Chapter 3: Related Work . . . . . . . . . . . . . . . . . . . . . . 183.1 Characterization of CFRP Panel Damages . . . . . . . . . . . 183.2 Registration Models for Multi-Modal Images . . . . . . . . . 193.3 Data Fusion Methods . . . . . . . . . . . . . . . . . . . . . . 223.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Chapter 4: Structural Enhancement for Multi-Modal NDEImage Registration . . . . . . . . . . . . . . . . . . . 264.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264.2 NDE Image Registration Framework . . . . . . . . . . . . . . 284.2.1 SCB Filter . . . . . . . . . . . . . . . . . . . . . . . . 294.2.2 Affine Registration . . . . . . . . . . . . . . . . . . . . 324.2.3 Performance Evaluations . . . . . . . . . . . . . . . . 364.3 Experimental Results and Analysis . . . . . . . . . . . . . . . 374.3.1 Data Description . . . . . . . . . . . . . . . . . . . . . 374.3.2 Experimental Setup . . . . . . . . . . . . . . . . . . . 384.3.3 Comparative Studies . . . . . . . . . . . . . . . . . . . 384.3.4 Ablation Study . . . . . . . . . . . . . . . . . . . . . . 434.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Chapter 5: Two-Stage NDE Data Fusion-Translation Network 505.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.2 Two-Stage Fusion-Translation Network . . . . . . . . . . . . . 535.2.1 Fusion Module . . . . . . . . . . . . . . . . . . . . . . 535.2.2 Translation Module . . . . . . . . . . . . . . . . . . . 585.3 Experimental Results and Analysis . . . . . . . . . . . . . . . 635.3.1 Data Augmentation and Preparation . . . . . . . . . . 635.3.2 Model Implementation . . . . . . . . . . . . . . . . . . 645.3.3 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . 645.3.4 Comparative Analysis . . . . . . . . . . . . . . . . . . 665.3.5 Discussions . . . . . . . . . . . . . . . . . . . . . . . . 685.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72viiiChapter 6: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 74Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77ixList of TablesTable 2.1 The lightning waveform components, aircraft zone, panelname, and thickness. . . . . . . . . . . . . . . . . . . 8Table 2.2 Lightning current waveforms in the lightning test onthe specimens [11]. . . . . . . . . . . . . . . . . . . . 9Table 4.1 Descriptions of baselines in the image registration ex-periment. . . . . . . . . . . . . . . . . . . . . . . . . . 38Table 4.2 Quantitative comparisons (mean and standard varia-tion) of registering eddy current testing with X-ray. . . 44Table 4.3 Quantitative comparisons (mean and standard varia-tion) of registering infrared thermography with X-ray. 44Table 4.4 Quantitative comparisons (mean and standard varia-tion) of registering ultrasonic testing with X-ray. . . . 45Table 4.5 Quantitative comparisons of registration accuracy (meanand standard variation) with SCB filter applied to dif-ferent images on ECT technique. − means the imagedoes not employ the SCB filter, and√denotes theimage put into the SCB filter before registration. . . . 45Table 4.6 Quantitative comparisons of registration accuracy (meanand standard variation) with different image pyramidlayers. . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Table 5.1 The Architecture of the translation network. ConvDenotes the Convolutional Block. The negative slopeof LeakyReLU is 0.1. . . . . . . . . . . . . . . . . . . . 61xTable 5.2 Quantitative comparison of fused images from differ-ent methods. X-ray and UT images are the groundtruth. The metrics of MI, NCC, PSNR, and RMSEof the results are compared with UT. The contours ofthe fusion results are compared with the contour ofthe X-rays using SSIM, area ratio, and CMM metricsfor evaluation. . . . . . . . . . . . . . . . . . . . . . . . 69Table 5.3 Quantitative comparison of fused images from differ-ent methods. X-ray and UT images are the groundtruth. The metrics of MI, NCC, PSNR, and RMSEof fusion results are compared with UT. The contoursof the fusion results are compared with the contour ofthe X-rays using SSIM, area ratio, and CMM metricsfor evaluation. . . . . . . . . . . . . . . . . . . . . . . . 72xiList of FiguresFigure 1.1 The research overall flowchart of image registrationand data fusion. . . . . . . . . . . . . . . . . . . . . . 5Figure 2.1 The structure of Carbon fiber-reinforced polymer (CFRP)material. . . . . . . . . . . . . . . . . . . . . . . . . . 8Figure 2.2 Laboratory lightning experimental facilities set up.(Image courtesy of NRC) . . . . . . . . . . . . . . . . 10Figure 2.3 Illustration of the panel exposed in the lightning strikecondition. (Image courtesy of NRC) . . . . . . . . . . 11Figure 2.4 Samples of eddy current testing images by calculatingfrom 2D sequences and plotting the magnitudes. . . . 12Figure 2.5 Samples of the infrared thermography image in light-ning damage. The first frame in the recording videois chosen. . . . . . . . . . . . . . . . . . . . . . . . . . 13Figure 2.6 The sample ultrasonic signal of a scanning point. . . . 15Figure 2.7 Samples of the visualized ultrasonic testing signal inlightning damage. . . . . . . . . . . . . . . . . . . . . 15Figure 2.8 Samples of the X-ray images in lightning damage. . . 17Figure 4.1 The architecture of the proposed image registrationmodel. . . . . . . . . . . . . . . . . . . . . . . . . . . 29Figure 4.2 Illustration of edge structure on two samples. Thefirst row shows the two NDE images detected by X-ray. The second row plots the intensity gradient ofthe scan lines. . . . . . . . . . . . . . . . . . . . . . . 30Figure 4.3 The image pyramid used in the registration model. . 33xiiFigure 4.4 Results on three panels of registering eddy currenttesting with X-ray. The registration results on fivemethods are overlapped with the reference image forbetter observation. . . . . . . . . . . . . . . . . . . . . 40Figure 4.5 Results on three panels of registering infrared ther-mography with X-ray. The registration results on fivemethods are overlapped with the reference image forbetter observation. . . . . . . . . . . . . . . . . . . . . 41Figure 4.6 Results on three panels of registering ultrasonic test-ing with X-ray. The registration results on five meth-ods are overlapped with the reference image for betterobservation. . . . . . . . . . . . . . . . . . . . . . . . 42Figure 4.7 Illustration of the edge structure in two samples. Thefirst row shows the image pair of X-ray testing imageswithout and with the SCB filter. The second rowplots the gradient of the scan line. . . . . . . . . . . 46Figure 4.8 Contour mapping measure (CMM) evaluation for dif-ferent parameters. . . . . . . . . . . . . . . . . . . . 48Figure 5.1 The architecture of the proposed two-stage Fusion-Translation network (FTnet). The proposed modelcontains two modules: (i) a two-modality image fu-sion module and (ii) an iterative image-to-image trans-lation module. . . . . . . . . . . . . . . . . . . . . . . 53Figure 5.2 The structure of the proposed image fusion module. . 55Figure 5.3 The structure of the proposed translation module. . 59Figure 5.4 The style reconstructions for the layers Conv1, Conv3,Conv5, Conv9 from the pre-trained VGG-19 network. 62Figure 5.5 The content reconstructions for the layers Conv2, Conv4,Conv8, Conv12 from the pre-trained VGG-19 network. 62Figure 5.6 An example of the processed training dataset. (a) isthe eddy current image, (b) is the infrared image, (c)is the ultrasound image, and (d) is the X-ray image. . 64xiiiFigure 5.7 Example of fusion results by different methods. Nest-Fuse, DIDfuse, DenseFuse, FusionGAN, and RFN-Nest are compared with the proposed FTnet model. 67Figure 5.8 The joint histogram for the fused results of NestFuse,DIDfuse, DenseFuse, FusionGAN, and RFN-Nest arecompared with the FTnet. The color of the figuresrepresents the counts of the combination. . . . . . . . 70Figure 5.9 Example of the fusion results. (a) is the result ofthe fusion network, (b) is the result of the translationnetwork (translating eddy current images), (c) is theresult of the translation network (translating infraredimages), (d) is the fusion result of FTnet, and (e) isthe fusion result of TFnet. . . . . . . . . . . . . . . . 71Figure 5.10 The joint histogram for the fused results of fusionnetwork, translation network on eddy current (ET)images, translation network on infrared (IR) images,FTnet and TFnet. The color of the figures representsthe counts of the combination. . . . . . . . . . . . . . 71xivGlossaryAdaSGD Adaptive stochastic gradient descentAR Area RatioBVID Barely Visible Impact DamageCFRP Carbon Fiber-Reinforced PolymerCMM Contour Matching MeasureCNN Convolutional Neural NetworkECF Expanded Copper FoilECT Eddy Current TestingGGD Generalized Gaussian DistributionIR Infrared Pulsed ThermographyIST Image Style Transfer ModelLoFTR Detector-Free Local Feature Matching with TransformersLSP Lightning Strike ProtectionMI Mutual InformationNCC Normalized Cross-CorrelationNDE Non-Destructive EvaluationxvNDT Non-Destructive TestingNTG Normalized Total GradientPSNR Peak Signal-to-Noise RatioRMSE Root Mean Square ErrorSCB Structure Consistency BoostingSSIM Structural Similarity Index MeasureUT Ultrasonic TestingxviAcknowledgementsFirst, I would like to express my highest gratitude to my supervisor, Dr.Zheng Liu, for his invaluable guidance, mentorship, and support through-out my research journey. His exceptional expertise and insightful guidancemotivated me to be enthusiastic about my research. It is a great honor todo my graduate study under his supervision.I would also like to thank Dr. Liwei Wang and Dr. Peyman Yousefifor their willingness to serve on the advisory committee. I really appreciatetheir valuable time and constructive comments on my thesis.In addition, I would like to extend my heartfelt appreciation to my col-leagues at the Intelligent Sensing, Diagnostic and Prognostic (ISDP) Re-search Lab. Their assistance and technical support have been meaningfulfor me over the last two years. Especially, I would also like to express mythanks to Dr. Rakiba Rayhana, Yue Cao, Junchi Bin, and Jiatong Ling fortheir valuable suggestions and encouragement on my research work.Lastly, I would like to express my deepest appreciation to my family fortheir love, encouragement, and support throughout my academic journey.Their patience, understanding, love, and continuous encouragement havebeen the driving force in my graduate study.xviiChapter 1Introduction1.1 Background and MotivationLightning poses a common threat to aircraft. According to the RoyalCanadian Air Force, lightning strikes occur at a frequency of approximatelyonce every 1,000 to 3,000 flight hours [1], which means that commercialtransport aircraft experience an average of one lightning strike per year [2].Although composite structures are increasingly used in the industry be-cause of their favorable strength-to-weight ratio and elastic properties, theypresent a new challenge. Despite numerous advantages, composite structuresare ineffective in dissipating the high currents generated during a lightningevent. The energy delivered by lightning rapidly impacts the rigid surfaceof the aircraft, leading to structural fragmentation and localized heatingcaused by the resistive effect of the high-current-carrying materials.The primary objective of the Lightning Strike Protection (LSP) tech-nique is to establish a low-impedance pathway on the exterior of the aircraft,allowing most of the lightning current to exit quickly and to avoid causingsignificant damage [3]. This material protects against structural damageand protects sensitive circuits. Typically, an LSP mesh made of copper oraluminum is affixed to the outer surface of carbon fiber-reinforced polymer(CFRP) and coated with layers of paint. This protective technique is basedon the principle of a Faraday cage, aiming to enhance the electrical con-ductivity of each component, such as the paint and CFRP. In this context,ensuring the integrity of the protective mesh is crucial, as any corrosion,fractures, or defects in the metal mesh can result in severe damage. There-fore, it is necessary to assess, maintain, or replace the composite structureand LSP material after each lightning event before further flights.1During maintenance procedures, non-destructive evaluation (NDE) meth-ods, also known as non-destructive testing(NDT), are commonly utilized todetect and assess the condition of CFRP aircraft structures beyond what canbe achieved by visual inspection. NDE allows the evaluation of changes inmaterial properties, such as cracks and internal flaws, without compromis-ing the stability and integrity of the materials [4]. Various NDE techniquesare frequently employed in the industry, including eddy current testing, ca-pacity testing, infrared thermography, pulsed ultrasonic testing, and X-rayradiography. Advancements in instrumentation and technology have signif-icantly improved the speed and quality of data acquisition in modern NDEtesting [5]. However, each NDE technique has unique advantages and limi-tations when evaluating the operational, safety, and performance aspects ofthe inspected material. Consequently, employing multiple NDE techniquescan provide complementary information and improve the reliability and ac-curacy of assessing the structural condition of the aircraft. This requiresthe use of data fusion techniques to obtain refined, accurate, and reliableresults.The objective of this research is to develop a method that combinesmultiple NDE techniques through data registration and fusion methods toenhance the information on damaged aircraft material. The proposed imageregistration and fusion methods aim to address the challenge of insufficientfeatures caused by considering the information from a single NDE techniqueduring the inspection. By leveraging deep learning, the research aims toimprove the accuracy and reliability of the assessment results.1.2 Research Gaps and ObjectivesThe lag in the development of NDE techniques poses a significant chal-lenge in assessing the damage to the aircraft surface exposed to lightningstrikes [6]. The challenge arises due to the large size of aircraft structures,limited access from one side, and the presence of multiple layers of functionalmaterials, making many NDE techniques impractical once the structure isassembled. Additionally, several factors influence the reliable detection of2surface defects, including the specific technique used, the type of specimen,and the defect types.Each NDE technique has unique characteristics and limitations. Forinstance, pulsed ultrasonic testing can obtain depth information but re-quires access to the non-damaged side of the component, and radiography issuitable for detecting density changes but needs access to both sides of theaircraft surface. Furthermore, it has been observed that no single NDE tech-nique can comprehensively characterize the inspected object, leading to un-reliable and uncertain results. The aerospace and aircraft industries demandincreasingly higher levels of accuracy and reliability, further emphasizing theneed for multiple NDE techniques to meet industry requirements. The fu-sion of multiple NDE techniques combines complementary information fromindividual NDE techniques, thus reducing measurement uncertainties andimproving the visualization of damaged areas in specimens. The applicationof advanced algorithms to fuse the techniques can significantly enhance theevaluation results and minimize uncertainties [7]. Additionally, in industrialsettings, there is growing interest in automating the evaluation of NDE datato eliminate human visual evaluation factors [8]. Currently, data fusion isused predominantly to improve statistical calculations, refine the positioningof features of interest in the specimen, and improve the observability of theresults. This approach has shown significant progress in increasing informa-tion completeness and facilitating the decision-making process. However, itis vital to acknowledge the limitations of data fusion in the NDE field, whichare listed below.• There is only a limited dataset for model learning. Implementingfusion algorithms requires a large dataset acquired from multiple NDEmethods. However, obtaining such datasets for aircraft surfaces canbe a time-consuming and expensive process, and until now there areno public datasets available for NDE data.• The inconsistent data acquired from multiple techniques, such as dif-ferences in resolution, data formats, and measurement scales, can leadto incompatibility between different NDE data and cause difficulties3in effectively fusing the information.• The methods of using ECT and IR data to enhance the damage infor-mation have not been fully investigated.• The outputs from existing fusion models are difficult to interpret andfurther analyze.To address the problems and limitations mentioned above, the objectivesof this research are defined as follows:• To develop an image registration model specifically tailored for NDEapplications so that the model can enable accurate alignment and reg-istration of images acquired from different NDE techniques, overcom-ing issues related to data misalignment.• To optimize the dataset to overcome the challenges of a limited datasetsize, including exploring techniques such as data augmentation to ex-pand the dataset and improve the generalizability of the fusion algo-rithm.• To propose an data fusion model that can improve the performance ofintegrating damaged areas in the image while also ensuring robustnessand generalizability across modalities.• To design a data translation module in the fusion model to furtherfacilitate the characterization of lightning damages in the fusion pro-cess.The research objectives will be achieved by completing the followingtasks.• By developing an iterative image registration model so that the modelcan automatically align and register NDE data acquired from differenttechniques. The model should effectively handle challenges such asgeometric transformations and varying imaging conditions, ensuringaccurate registration.4• By preprocessing data, including data cleaning, normalization, andaugmentation, to enhance the quality and variety of the dataset.• By designing a deep learning-based image fusion and translation modelso that the model can integrate multi-modal NDE data, leveragingdeep neural networks and advanced fusion strategies to enhance defectvisibility, reduce noise, and improve overall image quality.• By comparing and validating the results with existing fusion methods,insights are provided into the effectiveness and superiority of the de-veloped model, ensuring its practical applicability and contribution tothe field.Figure 1.1: The research overall flowchart of image registration and datafusion.The research flowchart shown in Figure 1.1 outlines the sequential stepsundertaken in this study. In the data preprocessing stage, the registered dataare further augmented and integrated to feed into the fusion model. Thesetasks aim to contribute to the advancement of NDE image registration andfusion in the field. The objective is to propose a novel and effective approachto integrate NDE data, thus improving the defect area in different modalitieson aircraft surfaces afflicted by lightning damage.1.3 Thesis Outline and OrganizationThe thesis is organized into five chapters. Chapter 1 provides an in-troduction to the research, including background studies, research motiva-tion, research gaps, research objectives, and an overview of the research5methodologies employed to achieve the research objectives. In Chapter 2,the materials utilized in the experiment are introduced and various NDEtechniques employed for detecting the damaged part of the specimen areprovided. Chapter 3 presents a review of related computational works em-ployed for NDE image registration and fusion. It highlights existing researchgaps and suggests potential solutions to address these gaps. In Chapter 4,an iterative model for automated registration of different NDE data is in-troduced. Chapter 5 proposes a deep learning-based framework for fusingdata from various NDE modalities to enhance information regarding thedamaged areas. Finally, Chapter 6 concludes the work and suggests futurework for research.6Chapter 2NDE Techniques for AircraftComposite Materials2.1 Aircraft Composite MaterialsCarbon fiber-reinforced polymer (CFRP) materials are widely used incomposite structures for aircraft. To assess the damage caused by lightningstrikes using the non-destructive evaluation (NDE) technique, sample pan-els made of the same CFRP material as the aircraft material were used inthis research. All panels were equipped with expanded copper foil (ECF)meshes for lightning strike protection (LSP) prior to curing. The CFRPpanels included two types: uni-directional (thin panels denoted as UD) andwoven (thick panels). The ECF meshes consisted of perforated and stretchedcopper sheets with two different surface densities: 195 g/m2 and 73 g/m2,corresponding to approximate wire diameters of 125 µm, and 75 µm, re-spectively [9]. The experimental panels comprised four layers, starting witha top layer of aerospace-grade paint measuring 20 µm in thickness, followedby a protective layer of 50 µm, and the ECF mesh adhered to the CFRPsurface to mimic a section of metal mesh. The structure of the sample panelis depicted in Figure 2.1. A total of six panels were subjected to simu-lated lightning conditions in the proposed experiments. The names of thespecimens and the applied lightning waveform components are presented inTable 2.1 and 2.2.The aircraft zones described in Table 2.1 correspond to three specific re-gions representing different in-flight lightning exposures, as defined by [10].These zones include: Zone 1 - susceptible to the first return stroke, Zone 2- susceptible to swept strokes, and Zone 3 - unlikely to be directly struck7Figure 2.1: The structure of Carbon fiber-reinforced polymer (CFRP) ma-terial.Table 2.1: The lightning waveform components, aircraft zone, panel name,and thickness.Zone Components Panel Name Panel Thickness (mm)1A A, B, C CFRP-UD-LSP-10 3.642A D, B, C CFRP-UD-LSP-12 3.652A D, B, C Thick CFRP-LSP-1 5.652A D, B, C Thick CFRP-LSP-2 5.69CFRP-LSP-2 (prior batch) 3.313N A/5, B, C CFRP-UD-LSP-11 3.56but may experience substantial currents. Zones 1 and 2 are further dividedinto regions A and B according to the probability that the lightning arcremains attached. Region A indicates a low likelihood of arc attachment,while region B indicates a high likelihood of arc attachment. Specifically,Zone 1A refers to the area where the first return stroke is expected duringlightning channel attachment, with a low probability that the arc will remainattached. Zone 2A represents the area of the swept stroke where all surfacesof the aircraft are anticipated to be swept after the return stroke, with alow probability that the arc will remain attached. Zone 3 represents areasother than Zone 1 and Zone 2, where lightning channel attachment is un-8Table 2.2: Lightning current waveforms in the lightning test on the speci-mens [11].Component FeatureA (Initial stroke) The time duration is less than 500µsThe peak amplitude is 200kA± 10%The action integral is 2× 106A2s± 20%B (Intermediate current) The time duration is less than 5µsThe average amplitude of 2kA± 10%The maximum charge transfer is 10C ± 10%C (Continuing current) The time duration is 0.25− 1sThe amplitude is between 200− 800AThe charge transfer is 20C ± 20%D (Subsequent return stroke) The time duration is less than 500µsThe peak amplitude is 100kA± 10%The action integral is 0.25× 106A2s± 20%likely. Additionally, Table 2.2 demonstrates that each lightning componentpossesses unique characteristics that can affect the panels in different ways.Subsequently, several NDE techniques are used to assess the lightning-struckcarbon fiber composite samples.Referring to the works of Kumar et al. [12] and Rajesh et al. [13], lab-oratory testing involves the utilization of large banks of storage capacitorsand creating an arc entry on a test panel to replicate the effects of lightningstrikes. These testing facilities, provided by Lightning Technology, Inc., havethe capacity to produce currents exceeding 200kA and voltages surpassing2.4MV . The setup to conduct lightning strike tests in the laboratory envi-ronment is illustrated in Figure 2.2. Throughout the experiments, lightningstrikes were simulated on test panels of two different thicknesses. The pro-cedure and results are illustrated in Figure 2.3.9(a) Overall setup. (b) CFRP panels on lightning test bench.Figure 2.2: Laboratory lightning experimental facilities set up. (Image cour-tesy of NRC)2.2 NDE Techniques2.2.1 Eddy Current TestingEddy current testing (ECT) is an efficient technique for detecting defects.In ECT, a coil is used to generate a time-varying magnetic field by applyinga time-varying voltage. When a conductive specimen is in close proximityto the coil, eddy currents are induced within the specimen, creating a sec-ondary magnetic field. ECT has been widely employed for inspecting metalsdue to its sensitivity to local variations in electrical conductivity. In recentyears, as new materials have emerged in various industries, researchers haveextended the application of ECT to inspect low-conductivity composite ma-terials. The presence of a conductive protective grid within CFRP panelsmakes ECT an ideal method for assessing the continuity of lightning protec-tion. However, ECT is unable to provide information regarding the depthof damage in the composite material.Eddy current measurements were performed using Jentek sensors to mea-sure embedded lightning protection systems in CFRP panels, as describedin [14]. An arbitrary function generator generated a voltage with a peak-to-peak amplitude of 10.0V , which was then amplified by a power amplifier andapplied to the excitation circuit. The ECT probe was placed 3mm above10(a) The arc entry process on the CFRP sample panel(b) The lightning damage at the strike pointFigure 2.3: Illustration of the panel exposed in the lightning strike condition.(Image courtesy of NRC)the sample within an area measuring 220mm× 240mm, with a step size of0.5mm. At each measurement point, two quantities were recorded. The firstwas the peak-to-peak amplitude of the voltage across the sampling resistorin the excitation circuit, which represents the real component of the eddycurrent signal. The second parameter was the peak-to-peak amplitude of thesensing coil voltage, which represents the imaginary component of the signal.Thus, each panel was captured as a sequence of two-dimensional data points.11(a) Sample 1 (b) Sample 2 (c) Sample 3Figure 2.4: Samples of eddy current testing images by calculating from 2Dsequences and plotting the magnitudes.To calculate the magnitude of the panel, the sum of the squared values ofthe real and imaginary components was calculated. Figure 2.4 illustratesthe examples of converted ECT data sequences presented as images.2.2.2 Infrared Pulsed ThermographyInfrared pulsed thermography (IR) testing is a well-established NDEtechnique. This method utilizes a high-energy pulsed flashing lamp as anexcitation heat source to generate temperature maps of the surface of thetest piece. It is commonly used to investigate and detect discontinuities ordefects in materials. If defects are present in the specimen, the presenceof these discontinuities will result in noticeable abnormalities in the sur-face temperature distribution. Pulsed thermography offers the additionaladvantages of being single-sided and non-contact. These attributes makethis technique particularly suitable for surveying impact damage on aircraftsurfaces. However, it captures an integral signal within the entire thermaldiffusion length range. As a result, IR can only provide two-dimensionalinformation about CFRP composites and cannot quantitatively determinethe depth of the damage.Pulsed thermography employs infrared imaging techniques to generatetemperature maps of the specimen’s surface, allowing for the examinationof local variations in thermal conductivity and volume-specific heat. These12(a) Sample 1 (b) Sample 2 (c) Sample 3Figure 2.5: Samples of the infrared thermography image in lightning dam-age. The first frame in the recording video is chosen.variations can indicate the presence of discontinuities and defects. In thetesting process, a 4.8kJ xenon lamp was used as the excitation source, whilea long-wave infrared camera operating within the 8−12µm wavelength rangewas utilized in passive mode to monitor the surface temperature. Then thetemperature maps at various times were recorded as a video. The mea-surements were conducted on CFRP panels with different LSP schemes.The sample area measured 200 × 240mm2, and the first frame of the in-frared recording video were chosen for further processing, which focused onthe lightning-affected side, yielding high-quality results, as demonstrated inFigure 2.5.2.2.3 Ultrasonic TestingUltrasonic testing (UT) is a technique that involves coupling high-frequencywaves into the material being tested. Discontinuities within the material af-fect the acoustic impedance, resulting in the reflection or transmission ofthese waves. This produces images that reveal damaged features withinthe material. Areas with damage exhibit high attenuation of the acousticenergy, and ultrasonic testing relies on the reflection of the beam at thesediscontinuities. In ultrasonic pulse-echo inspections, a low indication is char-acterized by an area of ultrasonic attenuation that is at least 6dB higher thanthe attenuation observed in adjacent flaw-free areas. In the case of assess-13ing simulated lightning damage in foam-core composite sandwich structuressimilar to those used in unmanned aerial vehicles, air-coupled ultrasonicwaves at low frequencies are employed. This assessment is carried out whenthe waveform components A, B, and C are applied, as described in [15]. Oneadvantage of ultrasonic testing is the ability to provide depth informationregarding the detected flaws. However, it should be noted that this methodrequires testing the backside of the sample, which can be inconvenience inreal industrial applications.In the evaluation of damage to CFRP panels with LSP, a pulsed-echoultrasonic technique was utilized using a frequency of 50MHz. The CFRPpanels consisted of 5 to 6mm thicknesses and six unidirectional plies. TheUT was conducted on the backside of the panels, which corresponded to theside impacted by lightning. The testing area was 220× 240mm2. A 5MHzcontact probe and water coupling were utilized for the UT. The analysis ofthe full-time-based waveform obtained from the UT provided valuable in-formation regarding the extent and depth of the damage. By examining thewaveform, the damage volume can be accurately calculated. This techniquehas been effectively applied in assessing damage in CFRP panels exposed tolightning waveforms, as discussed in [16]. In the experiment, an ultrasonicpulse is emitted into the CFRP panel, and the surface of the panel is scannedin a sliding manner using a contact probe. At each scanning point, the re-ceiver records three types of echoes: the front-side echo, the echo from thedamaged surface, and the back-side echo signal, as illustrated in Figure 2.6.To facilitate registration and fusion with other NDE modalities, UT signalsare converted into grayscale images that contain depth information. Thisconversion is achieved by selecting the maximum amplitude of the echo sig-nal from the damaged surface at each scanning point. By converting thetime values to depth information and combining the data from each pointon the sample panel, the UT samples are represented in Figure 2.7. TheseUT images serve as the ground truth in the subsequent image fusion step.14Figure 2.6: The sample ultrasonic signal of a scanning point.(a) Sample 1 (b) Sample 2 (c) Sample 3Figure 2.7: Samples of the visualized ultrasonic testing signal in lightningdamage.152.2.4 X-ray RadiographyX-ray radiography is an essential technique used in NDE by passing pen-etrating X-ray radiography through a sample. This radiography interactswith the internal structures of the sample and is scattered. Then the re-sulting projection image is recorded on a radiographic plate. The recordedimage provides information about the density of the internal structure ofthe sample and any deformities. In the recorded image, areas with cracksor material loss appear darker due to reduced energy absorption by the testpiece. X-ray radiography is particularly useful for identifying volumetricchanges in materials and effectively visualizing the integrity of copper meshwithin composite panels that have LSP. This technique enables the detec-tion of defects or anomalies in the examined materials, providing valuableinformation for quality assessment and ensuring the structural integrity ofthe tested components.However, a notable limitation of X-ray radiography is the practical chal-lenge of accessing both sides of the object being investigated, as it requirespositioning the X-ray source and the recording media on two sides of the de-tected structure. This issue becomes particularly pronounced in real-worldaircraft scenarios due to the large size of the aircraft, making it difficult toaccess both sides of the damaged surface. Furthermore, the two-dimensionalnature of X-ray radiography limits its effectiveness in accurately detectingthe thickness of the defects. Figure 2.8 showcases examples of X-ray im-ages, which are renowned for their ability to capture precise defect details.These images serve as essential ground truth references for both the imageregistration and fusion steps in the experiment.2.3 SummaryThis chapter delves into the application of carbon fiber-reinforced poly-mer (CFRP) panels in aircraft structures, highlighting their superior strengthand lightweight properties. To evaluate the damaged area in the CFRP pan-els using non-destructive evaluation (NDE) techniques, eddy current testing(ECT) and infrared pulsed thermography (IR) have been selected due to16(a) Sample 1 (b) Sample 2 (c) Sample 3Figure 2.8: Samples of the X-ray images in lightning damage.their ability to provide complementary information in NDE data analysis,along with the advantages of damaged-sided access and non-contact prop-erty. Further, ultrasound and X-ray techniques are employed as groundtruth techniques. Ultrasound provides valuable depth information on thepanel, while X-ray imaging offers precise contour details of the damagedarea. However, the practical application of ultrasound (UT) and X-rayimaging in the aircraft industry is often inconvenient due to their specificrequirements. UT requires access to the back side of the damaged area, whileX-ray imaging necessitates access to both sides of the damaged part. As aresult, these techniques are rarely utilized in aircraft inspection. In Chap-ter 4, the focus shifts to the registration of ECT, IR, and ultrasound imageswith X-ray images, aiming to align different image modalities. This regis-tration process is a crucial process for data preparation prior to applying thefusion model. Moving forward to Chapter 5, the fusion of registered ECTand IR images will be applied to enhance the information on the damagedarea, using ultrasound and X-ray images as reference.17Chapter 3Related WorkThis chapter first presents recent quantified methods of non-destructiveevaluation (NDE) for carbon fiber-reinforced polymer (CFRP) panels. Inthe field of aviation, the large size and structural complexity of aircraft posesignificant challenges for the evaluation of CFRP materials, making the ap-plication of high-precision NDE methods such as X-ray and ultrasonic in-spection infeasible. Therefore, the fusion of multiple NDE techniques thatare easy to implement becomes a viable method for efficiently evaluatingCFRP materials while maintaining high accuracy. Nevertheless, the appli-cation of NDE fusion methods in aerospace is largely unexplored. Hence,this chapter reviews the registration and fusion models for NDE in the gen-eral industry. Furthermore, based on the NDE techniques and consideringthe types of sensors employed in these techniques, the registration and fu-sion methods for NDE are extended to other fields, such as medical imagingand robotics.3.1 Characterization of CFRP Panel DamagesIn the past decade, CFRP materials have been widely applied in theaircraft industry due to their excellent mechanical properties and weight re-duction capabilities. However, aircraft structures were susceptible to variousforms of impact in actual aviation, such as lightning strikes, accidental im-pacts, and hailstones. The impact damages, including surface cracking anddelamination, can pose serious threats to the safety of the aircraft. Conse-quently, numerous studies have been conducted to detect and assess damagein CFRP panels. In the literature [17], air-coupled ultrasound was utilizedto detect non-linear ultrasonic responses caused by pressed damage on the18surface of CFRP panels. In the study conducted by Smaranika et al. [18],vibration and ultrasonic tests were used to accurately locate and determinethe depth of delamination in CFRP composite beams. Another method de-scribed in the literature [19] converted CFRP panels into 2D circuits andcalculated the detour length of electrical routes, which can assess the severityof mechanical fracture damage. Lee et al. [20] explored techniques such asPrincipal Component Analysis and k-Nearest Neighbors to identify varioustypes of damage in CFRP wind turbine blades, including matrix cracking,delamination, debonding, and fiber breakage. The research [21] performeda model reconstruction method for barely visible impact damage (BVID)in CFRP structures using X-ray computed tomography scans, ultrasonic Bscan and C scan. Vibrothermographic and local defect resonance methodswere employed for inspecting CFRPs with BVID [22]. Shreekant et al. [23]combined a vibration-based method and an ultrasonic C-scan method toidentify, locate, and quantify damage in CFRP structures impacted by dropweight. Edge illumination X-ray phase contrast imaging was used to locateand identify different types of defects in CFRP plates [24].Generally, ultrasound and X-ray imaging are effective techniques for de-tecting, locating, and assessing damage in CFRP plates, as they can providedetailed information and easily identify distinct damage. However, when itcomes to aircraft lightning scenarios, the large size of the structure poseschallenges for obtaining recorded images from the back side of the defectarea with ultrasonic testing and accessing both sides of the damaged areawith X-ray. As a result, in this experiment, techniques that allow accessto the damaged side are chosen, including eddy current testing and infraredthermography, and a combination of these techniques is employed to enhancethe information regarding the defect area.3.2 Registration Models for Multi-Modal ImagesData registration plays a crucial role in data fusion, particularly whencombining information from multiple modalities. It is essential to achieveaccurate registration, as compensating for errors becomes challenging when19dealing with highly localized features such as lightning damage on the sur-face of an aircraft. In the context of NDE images, the literature [25] utilizedan affine transformation to register NDE images, including infrared thermog-raphy (IR), impact echo, and ground penetration radar, for concrete bridgedecks. Another study [26] proposed an automated registration method basedon normalized mutual information to align ultrasonic and radiographic in-spections of titanium alloy test coupons. In the research described in [27],spatial registration of ultrasonic testing (UT) and the optical images wasachieved by registering landmarks and affine transformation. However, thenumber of studies specific to NDE is limited, leading to the inclusion ofstate-of-the-art models in the following.The multi-modal image registration models can be coarsely categorizedinto two types. The first type introduces registration models that are de-signed to handle the intensity or gradient in the different modalities’ inputimages. These models allow the original images to be directly inputted intothe registration process, and multi-modal similarity measurements are used.Representative models of this type include mutual information maximiza-tion [28], modality-independent neighborhood descriptor [29], and maxi-mization of similarity of normalized gradient fields [30], which all utilized themulti-modality similarity measurements. Also, the multi-scale gradient ori-entation selection strategy [31] reduced uncertainty locations and estimatesorientations after the transformation. Region-Specific Normalized Cross-Correlation [32] dispersed the computation of normalized cross-correlationinto local patches to handle regional intensity variation. The normalizedtotal gradient [33] used the gradient difference between image pairs, whichbecomes sparser when the pair is aligned. Wang et al. [34] proposed anunsupervised model that learns modality-independent features using bothan affine transform network and a spatial transform network. In summary,these models aim to address the challenges posed by multi-modal imageregistration through various techniques such as information theory, gradientanalysis, and learning-based approaches. However, these methods may en-counter difficulties when dealing with non-linear intensity variance betweendifferent modalities. Many widely used feature- or intensity-based models20can become ineffective, leading to heavy outliers and mismatches. This isoften caused by relying solely on local image information for registration.The other type involves performing transformations or translations intoa common space, thereby improving modality consistency and enabling theuse of common uni-modal measurements. The literature [35] introduced aradiation-variation insensitive feature transform method for feature match-ing in multi-modal image registration. An image-to-image translation and adual-stream registration network were utilized in literature [36] to train withcomputationally efficient similarity metrics for multi-modal clinical datasets.The method was designed to achieve accurate registration by translating be-tween different modalities and incorporating adversarial training. Arar etal. [37] employed a geometry-preserving image-to-image translation networkto map images from different modalities to a common representation space.This allowed the spatial registration to use mono measurements to registernear-infra-red and RGB images. Contrastive Multimodal Image Represen-tations [38] trained one neural network per modality with contrastive lossbased on noise-contrastive estimation. The shared representation amongthe networks reduced the multi-modal problem to a mono-modal one, fa-cilitating registration. Qin et al. [39] proposed a method that disentangledimage pairs into a common latent space and separated the latent appearancespace using an image translation model, which can solve the unknown re-lationship between different modalities by learning a shared representation.These studies highlight various approaches in the second type of multi-modalimage registration, where transformations or translations are employed toimprove consistency in registration. Despite the promising developments,several challenges still remain. In the field of NDE, one major challengeis the scarcity of image data from different modalities, which can hinderthe robust training of translation models. Furthermore, in this experiment,the images of three modalities need to be registered with the ground truthmodality. The registration process needs to consider the unique character-istics and latent spaces of each pair of modality to accurately align andintegrate the information from multiple modalities with the ground truth.The limited availability of multi-modal NDE data poses challenges for21training effective translation models. Without a diverse and extensive dataset,it becomes difficult to capture complex relationships and variations acrossmodalities. To address this problem, the study draws inspiration from Caoet al. [40], who proposed a structure consistency boosting (SCB) transformto enhance the performance of affine transformations in local variance witha Gaussian distribution. This method can enhance the alignment of themulti-modal data, mitigate the effects of variations in latent spaces, andimprove the overall registration performance.3.3 Data Fusion MethodsIn the aircraft industry, precise and rigorous testing of the aircraft surfaceis essential to ensure the safety of the crew and passengers while minimizingmaintenance costs. To achieve this, data fusion techniques are commonlyemployed to overcome the limitations of using single NDE techniques. Bycombining the strengths of different NDE techniques and compensating fortheir individual drawbacks, multi-modal data fusion is crucial in enhancingthe overall inspection process and ensuring the integrity of the aircraft sur-face structure. The weighted average was one of the simplest and fastestmethods for data fusion in CFRP panels [41–43]. However, it may not alwaysyield optimal fusion results due to the challenges inherent in assigning appro-priate weights to each sensor. The limitations of traditional fusion methodshave led to the exploration of more advanced fusion techniques, which canbe categorized into two main types: transform-based and learning-basedmethods.Transform-based methods in data fusion involve decomposing the inputdata into different domains to enable better feature extraction. Specifi-cally, multi-resolution transform theories are the most popular models fortransform-based image fusion. A key function of multi-resolution transformsis to alter the resolution of the image or signal in order to analyze its features,such as edges, textures, or frequencies. There were several multi-resolutiontransform-based methods available, including the Laplacian pyramid [44],wavelet transform [45], steerable pyramid [46], and high-pass filtering [47].22In general, the multi-resolution transform was initially applied to the inputimages. Then, a fusion strategy was applied to the transformed images toobtain a composite multi-resolution representation. Finally, the transformedimages were inverted to generate the fused images. Sasi et al. [48] proposeda method based on a discrete wavelet transform to fuse eddy current imagesat two frequencies in stainless steel cladding tubes. The fusion rule used theRmin value of Daubechies wavelets and biorthogonal wavelets. Bernardo etal. [49] employed an algebraic fusion method to characterize impact damagesin CFRP plates using optical lock-in thermography and optical square-pulseshearography images. The wavelet decomposition method was also utilizedin the fusion of terahertz NDE images of glass fiber-reinforced plastic panels.Using a weighted average fusion rule, the method effectively improved thesignal-to-noise ratio and a contrast ratio of the images [50]. Multi-resolutiontransforms had excellent reconstruction ability, minimizing information lossor redundancy in the images. By decomposing the image into low-frequencyand high-frequency coefficients, they can capture different structural infor-mation. Certainty fusion rules can be employed to extract and highlightvarious types of structural information while preserving image edges andother features, which traditional methods may struggle to achieve.Learning-based data fusion methods, including deep learning and ma-chine learning, have gained significant attention in the field of NDE dueto their ability to automatically learn different feature representations fromraw data. These methods have shown promising results in fusing NDE datato extract detailed information about flaw size, depth, and location. In theliterature [51], a network was developed to fuse global features extracted bya deep neural network and deep features extracted by a convolutional neu-ral network (CNN). The dataset used for training consisted of CFRP panelsinspected by active thermography. Hua et al. [52] utilized a multi-sensor in-formation fusion model to integrate ultrasound images from different anglesof carbon fiber sucker rods. Zhang et al. [53] employed a neural networkto fuse magnetic flux leakage, thermal infrared, and visible light signals forsurface defect detection on wire ropes.While there are fewer studies specifically focused on learning-based NDE23data fusion models, the architectural designs for these models are still largelyempirical. To supplement the lack of relevant studies in NDE, advanced im-age fusion models for infrared and visible images have been explored. Forexample, RFN-Nest [54] utilized a residual block and an autoencoder-basednest architecture to fuse infrared and visible images. Li et al. [55] pro-posed a novel method for fusion of infrared and visible images in whicha nest connection-based network and a channel attention model are devel-oped. The authors in the literature [56] proposed an autoencoder-basedfusion network, namely DIDFuse. The core idea is to decompose an imageinto background and detail feature maps with low- and high-frequency infor-mation, and then the decoder recovers the resulting image. DenseFuse [57]developed a deep learning framework that includes convolutional layers, afusion layer, and a dense block for the fusion of infrared and visible images.Ma et al. [58] introduced an end-to-end fusion model based on GenerativeAdversarial Networks.The utilization of learning-based models in NDE data fusion has demon-strated advantages, particularly in enhancing the pixel values resulting fromfusion operations. However, achieving higher fusion accuracy typically in-volves a trade-off between computational speed and cost. It is crucial tomake concerted efforts toward reducing computational complexity and costwhile simultaneously improving fusion accuracy and speed. Furthermore,the efficacy of fusion schemes is heavily dependent on the quality and typeof the training dataset. In the NDE field, acquiring large datasets can bechallenging due to limited availability. Therefore, addressing the limitationsassociated with dataset size and quality is essential for developing robustand reliable learning-based fusion models in NDE applications.3.4 SummaryThis chapter commences with an overview of the quantification methodsemployed in the non-destructive evaluation (NDE) of carbon fiber-reinforcedplastic (CFRP) panels. While ultrasonic and X-ray techniques are widelyutilized for defect detection and localization in CFRP panels, their practi-24cal application in the aircraft industry is hindered by the large size of thedetected structure. To address this challenge, the fusion of damaged-sideaccess NDE techniques is utilized in this experiment. Then the chapterexplores the existing multi-modal registration and fusion models for NDEdata, including their limitations in terms of applicability and accuracy forassessing lightning damage on CFRP panels. The existing registration mod-els fall short in achieving results compared to manual registration, and thefusion models struggle to retain both detailed information and clear con-tour information similar to UT and X-ray images, respectively. To fill thisgap, state-of-the-art multi-modal image registration and fusion models areexplored, which offer solutions for enhanced multi-modal data analysis anddemonstrate promising outcomes. Given the limited dataset available forthis experiment, both the proposed registration and fusion models are de-signed as lightweight models. For image registration, an iterative learningmodel is employed to align eddy current testing (ECT), infrared thermogra-phy (IR), and ultrasonic testing (UT) images with X-ray images. Further-more, an unsupervised autoencoder model is utilized for fusing ECT and IRdata, leveraging its ability to learn features effectively even in small datasetsand achieve optimal fusion results.25Chapter 4Structural Enhancement forMulti-Modal NDE ImageRegistration4.1 IntroductionMulti-modal non-destructive evaluation (NDE) sensors can collect differ-ent properties and characteristics of damaged areas in carbon fiber-reinforcedpolymer (CFRP) panels to obtain a more comprehensive representation ofthe scene. Nevertheless, it is common for multi-modal sensors to have ex-trinsic parameters with different modalities. These variations may includedifferences in the parameters of the detector lens, the resolution, and theunique features [7]. In such scenarios, rigid image registration plays a crucialrole in ensuring the accurate execution of downstream fusion tasks. Rigidregistration commonly involves six parameters for the operations of rotation,scaling, shearing, and translation to correct for global distortions present inthe images to be registered [59].Multi-modal image registration models typically utilize optimization func-tions to align an input image with a reference image by optimizing the sixaffine parameters. This optimization is achieved by comparing pairs of in-put and reference images using simple similarity metrics such as mutualinformation [60], normalized cross-correlation [61], the sum of squared dif-ferences [62], or sum of absolute differences [63]. However, these gradient-or intensity-based models often fail to generalize across the modalities in theNDE field. Moreover, these models suffer from computational limitationsand encounter difficulties when dealing with reference X-ray images that26exhibit uniform pixel intensities in the foreground and background. Hence,extracting meaningful features for automated registration becomes challeng-ing in such scenarios. To overcome these difficulties, transform-based mod-els, which enhance intensity consistency through mathematical formulations,have emerged as a solution. These methods can achieve high accuracy with-out relying on training data. By utilizing mathematical optimization tech-niques, such as intensity-based similarity metrics, the transformation-basedmodels can effectively align and register multi-modal images.In this chapter, a structurally enhanced image registration model is pro-posed to address the challenges mentioned above. The model is inspiredby the work of Cao et al. [40], which focuses on overcoming the limitationsposed by relatively uniform image features that hinder the registration pro-cess between other modalities and X-ray images. The objective is to developa model that can effectively handle all three input modalities, including eddycurrent testing (ECT), infrared thermography (IR), and ultrasonic testing(UT). A key component of the proposed model is the structural consistencyboosting (SCB) filter, specifically designed to enhance the gradient intensitycorrelation of the reference X-ray images. The inclusion of the SCB filtercan mitigate the difficulties of registering with X-ray images. Then, the reg-istration model utilizes mutual information to determine the optimal affineparameters that capture the differences between the computed projectionand the reference image. This model enables for more accurate image align-ment during the registration process without the need for extensive trainingdata. By combining the SCB filter, and the affine registration, the proposedmodel aims to overcome the challenges associated with multi-modal imageregistration. It offers a promising solution for improving the alignment ofdifferent modalities and addressing the scarcity of available datasets. Insummary, the main contributions of this chapter are as follows.• To address the problem that the reference X-ray images have relativelyindistinct features in the defect area and background, an SCB filteris developed. The SCB filter can enhance the edge structure of X-ray images, leading to a significant improvement in the registrationperformance of multi-modal NDE images with X-ray images.27• The structurally enhanced registration model demonstrates superiorperformance compared to advanced models and manual matching tech-niques, showcasing excellent registration results, particularly for im-ages with different input modalities. Moreover, the proposed modeloffers the advantage of easy integration with subsequent fusion pro-cesses. Thus, the proposed model proves to be highly effective for thepre-processing phase of NDE data fusion.These contributions highlight the advances made in enhancing the reg-istration performance and pre-processing of multi-modal NDE images, ulti-mately improving the overall quality and accuracy of the fusion process.4.2 NDE Image Registration FrameworkThe proposed multi-modal NDE registration model comprises two pri-mary components. First, an SCB filter is utilized to enhance gradient inten-sity correlation while preserving the geometric structure. Second, an affineregistration network is employed. The architecture of the model is depictedin Figure 4.1. The input images comprise the images obtained through ECT,IR, and UT, whereas the reference images are acquired from X-ray imaging.The SCB filter operates by transforming the reference images into a latentspace, thus enhancing the edge structure. Afterwards, both the input imageand the filtered reference image are fed into the affine registration networkfor optimization. In the affine registration process, the input and referenceimages are first placed in the image pyramid to generate a sequence of multi-resolution images at different levels. The affine registration is first applied tothe low-resolution image, and then the affine parameters derived from thislevel initiate the registration process for the subsequent higher-resolutionimage. This iterative procedure continues until the registration reaches theoriginal image. By implementing this coarse-to-fine registration strategy,the robustness of the registration model is enhanced. At each level of theaffine registration, an interpolator is subsequently employed to ensure thatthe image values are represented in integer coordinates. The mutual infor-mation (MI) similarity measure is used to maximize the similarity between28the input and reference images.Figure 4.1: The architecture of the proposed image registration model.4.2.1 SCB FilterIn image processing, the image gradient is an important feature that canreflect the structural information of the image. Figure 4.2 reveals that thereference X-ray image contains significant variations in gradient information,causing the edge structures in the image to exhibit relatively low saliency.The saliency edge structure refers to the presence of pixel gradients with sig-nificant magnitudes in localized regions. The lack of saliency in the imageintensity poses challenges during the alignment process, as many registrationmodels rely on prominent features or edges for accurate alignment. Whenthe image intensity lacks strong saliency, alternative approaches or prepro-cessing steps become necessary to enhance the registration performance.Inspired by the work of Cao et al. [40], an SCB filter is proposed toenhance the saliency of the damaged area in images while preserving struc-tural consistency. To deduce the SCB filter, first is to apply a Gaussian29Figure 4.2: Illustration of edge structure on two samples. The first rowshows the two NDE images detected by X-ray. The second row plots theintensity gradient of the scan lines.smoothing filter, which is defined asG(x, y) =12πσ2e−x2+y22σ2 (4.1)where x is the distance from the origin in the horizontal axis, y is the dis-tance from the origin in the vertical axis, and σ is the standard deviationof the Gaussian distribution. Then the gradient of the smoothed image I iscomputed as▽I =[∂I∂x ,∂I∂y](4.2)30where ∂I∂x = Sx⊗I, and ∂I∂y = Sy⊗I. In the experiment, the Sobel operatoris used, and they are shown asSx =18−1 0 1−2 0 2−1 0 1 , Sy = 18 1 2 10 0 0−1 −2 −1 (4.3)where Sx and Sy are 3×3 kernels which are convolved with the input imageto calculate the horizontal and vertical derivatives, respectively.Hence, the edge structure is characterized by its relatively high saliencyin the surrounding local area. To quantify the saliency of an edge structure,the saliency evaluator is defined asτI =(▽I)2(▽I)2 + f2(I) (4.4)where f(I) is the generalized Gaussian distribution (GGD) computed fromthe intensity of the image I, which can indicate the degree of the relativedegree of structure saliency. If ▽I is much larger than f(I), which meansτI approaches 1, indicating the gradient is very salient. On the other hand,if ▽I is much smaller than f(I) and τI approaches 0, indicates the edgestructure is not prominent.The generalized GGD function is of the formg(x;µ, σ, p) =12Γ(1 + 1/p)A(p, σ)exp(−∣∣∣∣ x− µA(p, σ)∣∣∣∣p), x ∈ Nm (4.5)where µ is the mean value that controls the symmetry axis, p is the shape pa-rameter and controls the decay rate from the central peak, σ is the variance,A(p, σ) is the scaling factor satisfying V ar(x) = σ2. In this experiment, theGGD function can be simplified tof(I;α, β, µ, p) = α · exp(−∣∣∣∣I − µβ∣∣∣∣p) (4.6)Then inserting 4.6 into 4.4SCB(I;α, β, µ, p) =(▽I)2(▽I)2 + α2 · exp(−2∣∣∣ I−µβ ∣∣∣p) (4.7)which yields the explicit form of the SCB filter, where SCB(I) is the en-hanced image I.314.2.2 Affine RegistrationTo align the two NDE images, a rigid transformation was applied toensure the distinct detection characteristics of each technique and to main-tain their rigidity during the registration process. The adoption of the rigidtransformation model followed the application of the SCB filter.Affine registration is a well-established and widely used model to achieverigid registration. It has been employed in various applications [33, 64–66]due to its stability and effectiveness. The affine transformation combinesfour basic transformations: translation, rotation, scale, and shear. Theseoperations enable the transformation of the images while preserving theiroverall shape and structure. The transformation φ is formulated asφa(x) =[φa1(x)φa2(x)]=[a1 a2a4 a5][x1x2]+[a3a6]= Ax+ b (4.8)where A =[a1 a2a4 a5]and b =[a3a6]are the affine transformation matrixand the translation vector, respectively. For easier optimization, the vec-tor a = (a1, a2, a3, a4, a5, a6)T is used as the transformation parameter tobe computed in the optimization. In this experiment, a multi-resolutionstrategy was first used to obtain images at different resolutions, and thenthe MI measurement was employed as the objective function to quantify thedissimilarity between the input and reference images. Adaptive stochasticgradient descent (AdaSGD) was utilized as an optimizer to minimize MIby iteratively adjusting the affine parameters. The reference images in thisthesis were X-ray images, while the input images consisted of data from theECT, IR, and UT modalities. The goal is to optimize the affine transforma-tion to align the input images with the reference X-ray images, leveragingthe MI measurement to maximize the similarity between them. In summary,affine registration has two important components to build the framework:multi-resolution strategy and optimization strategy.32Multi-Resolution StrategyThe usage of the Gaussian pyramid as a multi-resolution strategy hasdemonstrated its effectiveness and efficiency in achieving stable training andavoiding local minima. Throughout the image pyramid process, the inputand reference images are converted into multiple levels of resolution images.Figure 4.3 illustrates the multi-resolution pyramid utilized for registration.The image arrangement follows a decreasing resolution pattern, progressingfrom level 0 of the pyramid, which signifies the original image, to level 3,where the lowest resolution images are represented. Specifically, the origi-nal images at level 0 of the pyramid undergo downsampling and Gaussianblurring with a size of N ×N , resulting in the subsequent upper level imagewith a size of N/2 × N/2. In the registration procedure, the image pyra-mid is executed from the top downward in a coarse-to-fine manner. Theproposed model employs a four-level Gaussian pyramid. This strategy aimsto enhance the capacity of the model to address large-scale transformationsand augmenting the probability of identifying the global optimum withinthe parameter space.Figure 4.3: The image pyramid used in the registration model.33Optimization StrategyThen an objective function is defined to measure the difference betweenthe input image Ip and the reference image Ir with mutual information.Accordingly, the objective function is defined asJMI(Ip, Ir) = H(Ip) +H(Ir)−H(Ip, Ir) (4.9)where H denotes Shannon entropy. For an image Ip with the intensity ofi− 1...M , the probability of occurrence of i is p(i). The Shannon entropy isgiven asH(Ip) =M∑i=1p(i)logp(i) (4.10)For an image Ir with an intensity value of j = 1...N , the probability ofjoint occurrence of (i, j) is p(i, j). Thus the joint entropy of Ip and Ir isH(Ip, Ir) =M∑i=1N∑j=1p(i, j)logp(i, j) (4.11)For each layer of the image pyramid, the registration parameters areupdated using adaptive stochastic gradient descent (AdaSGD), which iter-atively updates the affine parameters a viaak+1 = ak − η√G(ak, ξk) + ϵg(ak, ξk) (4.12)with an arbitrary initial a0 and a fixed step-size η; k denotes the iterationnumber; G(ak, ξk) is the diagonal matrix in which each diagonal element i,i = 1, 2, ..., N is the sum of squares of the gradients with respect to ak upto time step k; ϵ is a smoothing term that avoids division by zero; ξk is arandom variable; g(ak, ξk) denotes a stochastic gradient. Hence, combiningthe equation 4.9, the update function isak+1 = ak − η√G(ak, ξk) + ϵ▽JMI(Ip, Ir) (4.13)where the initial affine parameter vector a0 is set to a0 = (1, 0, 0, 0, 1, 0)T .34Algorithm 1: Algorithm of proposed image registration model.Input: Input image P and reference image RParameters in SCB filter α, β, µ, pStep size ηMaxIterationInitialize affine parameter a0 = (1, 0, 0, 0, 1, 0)T// Apply SCB filter on reference image RSCB(R;α, β, µ, p) = ▽R2▽R2+α2·exp(−2∣∣∣R−µβ ∣∣∣p)while Ip, Ir in Image pyramid (P, SCB(R)) do// Affine Parameter TransferTransfer the affine parameters to the next levelwhile i in MaxIteration do// Calculate objective functionJMI(Ip, Ir) = H(Ip) +H(Ir)−H(Ip, Ir)// Calculate gradient using AdaSGD// Update affine parameterai ← ai−1 − η√G(ai−1,ξi−1)+ϵ▽JMI(Ip, SCB(Ir))endendThe proposed registration algorithm is presented in Algorithm 1. Thealgorithm first sets the parameters for the SCB filter, namely α, β, µ, and p,as well as the parameters for affine registration, including η, MaxIteration,and α0. The reference image R is initially passed through the SCB filter forstructural enhancement. Then, four pair of images are generated by passingthe filtered reference image SCB(R) and the input image P into the im-age pyramid, and these pairs are then inputted into the affine registrationnetwork. The registration process begins with the lowest-resolution image,and the resulting registration is used as the initial estimation for the subse-quent higher-resolution image. This process continues until the registrationreaches the original image.354.2.3 Performance EvaluationsIn order to conduct a comprehensive evaluation of the image registra-tion models, both qualitative and quantitative comparisons are necessary.In this experiment, three evaluation metrics are employed, including thecontour matching measure (CMM) [67], the structural similarity index mea-sure (SSIM) [68], and peak signal-to-noise ratio (PSNR) [69]. Due to theless distinct defect features in X-ray lightning damage images, as shown inFigure 2.8, and the presence of texture-induced noise in the CFRP panels,the accuracy of the pixel level metrics in evaluating the model performanceis compromised. As a result, CMM and SSIM metrics are evaluated by thecontour of the image pairs to offset the inconspicuous X-ray images and thedifference of multi-modalities. All the contours are calculated by the Cannyedge detector automatically.Specifically, CMM measures the contour distance between the input im-age and the reference image. The smaller CMM reflects better registra-tion performance. First, assume that the first and last points on the twoboundaries were matched. A cyclical shift σk of the size k of the boundaryIp = m1m2 · · ·mn is defined by σk(m1m2 · · ·mn) = mk+1 · · ·mnm1 · · ·mk,1 ≤ k ≤ n, and σ0(Ip) = Ip. Therefore, the mapping distance δ([Ip], [Ir]) isdefined asδ([Ip], [Ir]) = min{δ(σk(Ir), σl(Ip))|0 ≤ k ≤ n, 0 ≤ l ≤ m}(4.14)Therefore, CMM is defined as the normalized mapping distance between theboundaries Ip and Ir:CMM(Ip, Ir) =1|T |δ ([Ip], [Ir]) (4.15)where T is the trace corresponding to the optimal mapping sequence, and|T |, the size of the trace, denotes the number of mapped point pairs.SSIM is a perception-based measure that quantifies the similarity be-tween two images by considering image degradation as a perceived changein structural information. It takes into account important perceptual phe-nomena such as luminance masking and contrast masking. Higher SSIM36values indicate a better registration result. The measure between two im-ages (Ip, Ir) of size N ×N isSSIM(Ip, Ir) =(2µpµr + c1)(2σpr + c2)(µ2p + µ2r + c1)(σ2p + σ2r + c2)(4.16)where µp, σp and µr, σr are the mean and standard deviation of image Ipand Ir, respectively; σpr is the covariance of Ip and Ir; c1 and c2 are theempirical constant.PSNR computes the ratio of the number of intensities level in the re-sultant image to the pixels in the reference image. A higher PSNR valuemeans that the image quality is better and the result is more similar to thereference image. PSNR between the image pair (Ip, Ir) is defined asPSNR(Ip, Ir) = 10log10(Max2(Ip)MSE(Ip, Ir))(4.17)where Max(Ip) indicates the maximum pixel gray levels value in image Ip,MSE indicates the mean squared error of the image Ip and Ir.4.3 Experimental Results and Analysis4.3.1 Data DescriptionThe dataset utilized in this experiment was collected from six differentCFRP lightning strike panels, as described in Chapter 2. The data was col-lected using four different NDE techniques: ECT, IR, UT, and X-ray. Eachtechnique presented the data in a unique format. For eddy current testing,the current data were recorded as a two-dimensional array that containedboth the real and imaginary parts of the impedance. The infrared techniqueinvolved recording a video using an infrared probe, and for registration pur-poses, the first frame of the video was used. Ultrasonic testing employedsound waves to assess the depth of the images. The X-ray technique recorded2D images of the panel, against which the other techniques were alignedduring the registration process. This experiment converted all data into 2Dimages for registration. X-ray images served as reference images for aligningECT, IR, and UT image images.374.3.2 Experimental SetupIn this experiment, a computer with a NVIDIA RTX 3080Ti GPU, a12th Intel Core CPU, and 32GB memory was utilized. The proposed affineregistration was implemented using Python 3.9.7 and the ITK toolbox [70],which supports a multi-registration framework. The hyperparameters of theSCB filter were set as α = 0.03, β = 2, µ = 0.5, and p = 0.2. For affineregistration, the initial affine parameter was (1, 0, 0, 0, 1, 0)T , the step sizeη = 1, maximum iteration for each level of the image pyramid was 150.4.3.3 Comparative StudiesIn the evaluation of the proposed model, a comparison is made withseveral state-of-the-art rigid image registration models. These models in-clude SCB (Structure Consistency Boosting) [40], which proposes a struc-ture consistency boosting (SCB) transform to boost the structural similarityof multi-modal images. The sum of squared differences similarity metric isemployed in the registration. NTG (Normalized Total Gradient) [33], whichmeasures the degree of alignment of two images in terms of the difference insparsity between two banded images. LoFTR (Detector-Free Local FeatureMatching with Transformers) [71], which first establishes pixel-wise densematches at a coarse level and later refines them based on the Transformerand the convolutional neural network. In the thesis, the pre-trained check-point in LoFTR is used. Figure 4.1 summarizes the comparative baselinesin the chapter.Table 4.1: Descriptions of baselines in the image registration experiment.Models Rigid Category Pre-TrainedSCB√Transform-based -NTG√Intensity-based -LoFTR√Feature-based√38Qualitative ResultsThe qualitative results of the registration models are presented in Fig-ure 4.4. To facilitate a direct and comprehensive comparison of the registra-tion result, the results employ the overlapping of the reference images andthe result images. In all three examples of ECT images, it can be observedthat the manual registration method and the proposed model yield moreaccurate registration results compared to the other models. The LoFTRmodel shows promising results in the second example, but it falls short ofmeeting the fusion requirements. The SCB model shows slight improvementcompared to the unaligned images, but exhibits severe geometric distortionsin the second sample. On the other hand, the NTG model suffers from sig-nificant registration errors in all the examples, failing to align with any ofthe damaged areas.Figure 4.5 and Figure 4.6 illustrate the qualitative results of registeringIR images and UT images, respectively. Similarly to the example illustratedin Figure 4.4, the comparison to both state-of-the-art models and manualmatching method, the proposed model consistently achieves outstandingalignment results. This indicates that the proposed model successfully meetsthe automatic registration requirements for all three modalities.These qualitative results clearly demonstrate the superior performanceof the proposed model over the other advanced models in all three modali-ties. The accurate alignment achieved by the proposed model ensures thatthe damaged areas are properly registered, enabling effective fusion andsubsequent analysis of the multi-modal NDE data.39Input Reference Manual SCB NTG LoFTR Proposed(ECT) (X-ray)Figure 4.4: Results on three panels of registering eddy current testing with X-ray. The registration results on fivemethods are overlapped with the reference image for better observation.40Input Reference Manual SCB NTG LoFTR Proposed(IR) (X-ray)Figure 4.5: Results on three panels of registering infrared thermography with X-ray. The registration results onfive methods are overlapped with the reference image for better observation.41Input Reference Manual SCB NTG LoFTR Proposed(UT) (X-ray)Figure 4.6: Results on three panels of registering ultrasonic testing with X-ray. The registration results on fivemethods are overlapped with the reference image for better observation.42Quantitative EvaluationQuantitative evaluation of image registration focuses mainly on the con-tour of the damaged area. The evaluation is the average value of six panelsusing ECT images to register with X-ray as the ground truth. The resultsare presented in Table 4.2. Based on the evaluation metrics, the proposedmodel achieves optimal results in terms of the three evaluation metrics.This indicates that the proposed registration results align well with the X-ray images, resulting in accurate registration results. On the other hand, theNTG model shows low mean and variation values for both SSIM and PSNRmetrics. This suggests that the registration results obtained by the NTGmodel fail to align well with the X-ray images. This discrepancy may be at-tributed to the heavy dependence of the NTG model on the image gradient,particularly when the gradient does not exhibit significant differences in theX-ray images. The LoFTR model demonstrates a high PSNR but falls shortof achieving the best results compared to the proposed model. Comparedto the SCB model, the proposed model demonstrates improvements acrossthree evaluation metrics. Importantly, the proposed model surpasses themanual matching model, indicating that it reaches a level of performancethat meets the standards of human expertise.The quantitative results of the registration of infrared and ultrasoundimages are presented in Table 4.3 and Table 4.4, respectively. Similar to thefindings for eddy current images, the proposed model exhibits the highestvalues for both the CMM and SSIM metrics. This confirms the effectivenessand superiority of the proposed model in achieving precise image registrationfor multi-modal NDE data fusion.4.3.4 Ablation StudyEffect of SCB FilterFigure 4.7 illustrates the gradient features present in the original sampleimage. It is observed that the blue scan line exhibits frequent local fluc-tuations and significant changes in gradient, which presents a challenge inaccurately registering the structural features of the identified edges. After43Table 4.2: Quantitative comparisons (mean and standard variation) of reg-istering eddy current testing with X-ray.Model CMM ↓ SSIM ↑ PSNR ↑Unaligned 53.076 ± 17.537 0.721 ± 0.201 9.010 ± 1.800Manual 51.406 ± 16.838 0.705±0.236 9.019 ± 1.753SCB 58.955 ± 17.700 0.687±0.171 9.261 ± 2.325NTG 290.274 ± 139.791 0.735 ± 0.209 6.512 ± 2.327LoFTR 77.511 ± 20.556 0.713 ± 0.153 9.883 ± 1.800Proposed 46.919 ± 14.178 0.750±0.154 12.943 ± 1.609Table 4.3: Quantitative comparisons (mean and standard variation) of reg-istering infrared thermography with X-ray.Model CMM ↓ SSIM ↑ PSNR ↑Unaligned 41.006 ± 15.842 0.751 ± 0.082 11.782 ± 1.266Manual 36.337 ± 16.680 0.755 ± 0.083 12.207 ± 0.975SCB 50.679 ± 12.042 0.697 ± 0.069 8.611 ± 2.426NTG 353.510 ± 51.778 0.816 ± 0.072 6.199 ± 2.256LoFTR 59.702 ± 20.152 0.748 ± 0.055 11.786 ± 1.239Proposed 30.041 ± 12.578 0.783 ± 0.072 14.259 ± 0.438applying the SCB filter, a significant enhancement in edge gradients is ob-served, as demonstrated by the yellow scan line in the figure. Meanwhile,the background and structure within the damaged region remain relativelyunchanged. The effectiveness of the SCB filter is further demonstrated inTable 4.5, which highlights the considerable improvement achieved by em-ploying the SCB filter on the reference images. In the table, − indicatesthat the image does not undergo the SCB filter, while√represents the im-age that is processed with the SCB filter before registration. For instance,in the first row of the table, both the input and reference images are notsubjected to the SCB filter before the registration process. This filter suc-44Table 4.4: Quantitative comparisons (mean and standard variation) of reg-istering ultrasonic testing with X-ray.Model CMM ↓ SSIM ↑ PSNR ↑Unaligned 37.360 ± 15.003 0.729 ± 0.124 16.531 ± 2.680Manual 34.651 ± 16.835 0.743 ± 0.115 16.146 ± 2.495SCB 46.389 ± 9.515 0.654 ± 0.162 13.371 ± 4.861NTG 221.993 ± 167.598 0.735 ± 0.114 9.001 ± 4.653LoFTR 59.811 ± 18.459 0.664 ± 0.081 16.503 ± 2.712Proposed 30.873 ± 12.269 0.744 ± 0.105 16.764 ± 2.002cessfully addresses the difficulty of distinguishing the defected region fromthe background, thereby enhancing the overall registration process. How-ever, when the SCB filter is applied to the input image, the results show aslight drop in performance compared to when the SCB filter is applied tothe reference image only. This outcome arises due to the increased gradientintensity in the input images, which can introduce confusion to the affineregistration model. Consequently, exploring the SCB filter on the referenceimages yields the best overall result.Table 4.5: Quantitative comparisons of registration accuracy (mean andstandard variation) with SCB filter applied to different images on ECTtechnique. − means the image does not employ the SCB filter, and √denotes the image put into the SCB filter before registration.InputImageReferenceImageCMM ↓ SSIM ↑ PSNR ↑- - 42.715 ± 15.221 0.708 ± 106 5.097 ± 2.141√- 38.772 ± 7.930 0.717 ± 0.140 11.308 ± 2.030-√30.873 ± 12.269 0.744 ± 0.105 12.943 ± 1.609√ √37.763 ± 8.004 0.710 ± 0.147 11.237 ± 1.95945Figure 4.7: Illustration of the edge structure in two samples. The first rowshows the image pair of X-ray testing images without and with the SCBfilter. The second row plots the gradient of the scan line.Impacts from Hyper-Parameters on SCB FilterFigure 4.8 illustrates the comparative evaluation of CMM registrationmetrics with different values of the SCB filter hyper-parameters. The smallerCMM means that the contour of the result and reference images are regis-tered well. The parameters of α are varied within [0, 0.5], β within [1, 8],46and p within the interval (0, 1]. The results indicate that small variations inthese parameters do not significantly affect the CMM values, demonstratingthe robustness of the proposed model. However, based on the analysis ofthe figure, the experiment selects the values α = 0.04, β = 4.9, and p = 0.32as they yield the best results for the SCB filter.Impact on Image PyramidTable 4.6 illustrates the results of registration evaluations for differentlayers in the image pyramid. The performance of the proposed registrationmodel is found to be optimal when the number of pyramid layers is setto 4. When the number of layers is less than 4, the downsampled imagemay not be sufficient to capture global information, leading to suboptimalresults. On the other hand, when the number of layers exceeds 4, the modelachieves performance similar to the 4-layer configuration, but at the cost ofincreased computational resources. Therefore, the choice of 4 layers strikesa balance between performance and computational efficiency.Table 4.6: Quantitative comparisons of registration accuracy (mean andstandard variation) with different image pyramid layers.Model CMM ↓ SSIM ↑ PSNR ↑Layer = 1 34.751 ± 12.475 0.722 ± 0.199 12.911 ± 1.607Layer = 2 35.475 ± 11.551 0.719 ± 0.205 12.925 ± 1.613Layer = 3 35.639 ± 11.581 0.731 ± 0.165 12.940 ± 1.614Layer = 4 30.873 ± 12.269 0.744 ± 0.105 12.943 ± 1.609Layer = 5 31.690 ± 12.734 0.742 ± 0.164 12.667 ± 1.624Layer = 6 31.086 ± 12.940 0.741 ± 0.165 12.681 ± 1.6164.4 SummaryIn this chapter, a structurally enhanced registration model is introducedfor multi-modal non-destructive evaluation (NDE) images. To address the47Figure 4.8: Contour mapping measure (CMM) evaluation for different pa-rameters.issue of relatively inconspicuous defect areas in the reference X-ray images,the proposed model employs an SCB filter to enhance the edge structure withgeometric consistency. Then an affine registration network is applied, uti-lizing an image pyramid and adaptive stochastic gradient descent to obtain48the transformation. Extensive experiments conducted on the NDE datasetdemonstrate that the proposed model achieves superior registration resultswithout the requirement of training data. In comparison to other state-of-the-art models that rely on feature matching or intensity comparison, theproposed model emphasizes gradient intensity and structural information,making it more compatible with the NDE dataset. By delivering reliableand consistent alignment results, the proposed model has great potential tobe applied in the NDE field. The ability to accurately align images from dif-ferent modalities facilitates the effectiveness of the model for data fusion andsubsequent analysis, ultimately enhancing the overall inspection and evalua-tion process in the aircraft industry. Nevertheless, the generalizability of theproposed registration model cannot be fully verified due to the limited avail-ability of datasets. If additional multi-modal NDE image datasets becomeaccessible in the future, the model performance can be further evaluatedand enhanced.49Chapter 5Two-Stage NDE DataFusion-Translation Network5.1 IntroductionThe safety of aircrafts can be significantly improved with a better under-standing of the extent of damage and effective repairs. Non-destructive eval-uation (NDE) methods can provide a quantitative assessment of the extentand severity of carbon fiber-reinforced polymer (CFRP) aircraft structuresbeyond visual inspection. With improvements in instrumentation and tech-nology, modern NDE testing has accelerated data acquisition and enhanceddata quality [5]. However, each NDE technique has unique characteristicsand is only sufficient to investigate a fraction of the full range of anomaliesthat can occur in composite materials. Concurrently, using multiple NDEtechniques can provide complementary information and improve the reliabil-ity and accuracy of aircraft structural condition assessment. The increasingavailability and diversity of NDE inspection methods provide new opportu-nities and challenges for data analysis and decision-making by aggregatingdata from multiple sensing modalities. Through the utilization of imagefusion techniques, it becomes possible to merge diverse images of the samescene into a singular fused image. This approach maintains the distinctattributes of individual NDE techniques while concurrently offering richerand more comprehensive information. Consequently, this enhancement fa-cilitates improved perception by both humans and machines. Despite thepotential challenge of dealing with inconsistencies in multi-sensor NDE data,data fusion has proven to be a viable solution [72].Benefiting from the rapid development in deep learning, convolutional50neural networks (CNNs) have demonstrated remarkable performance in im-age fusion tasks [73–78]. However, these CNN-based models are often hin-dered by the extensive training time to achieve optimal results. In prac-tical industrial scenarios, where computational resources are limited, thesecomplex models pose a significant computational burden, especially whendeployed on embedded platforms. To overcome this challenge, autoencodermodels have been utilized, which enable the compression of input imagesinto a lower-dimensional latent space in the encoder part and reconstruc-tion in the decoder part. DenseFuse [57] integrates convolutional layers, afusion layer, and a dense block within the autoencoder framework, estab-lishing connections between each layer and every other layer. NestFuse [55]introduces spatial attention and a channel attention autoencoder to capturethe importance of spatial positions and channels in deep features. SEDR-Fuse [79] proposes a symmetric autoencoder with residual blocks trained tocreate a fixed feature extractor. Nevertheless, these approaches encounter asimilar challenge as CNN models with their complex model structures. Sinceacquiring large-scale NDE datasets is challenging, complex models can besusceptible to overfitting.Recently, significant attention has been paid to image translation models.These models are designed to translate images from one domain to another,in order to improve image quality and imbue the input image with the seman-tic essence of the output image [80]. Generative adversarial networks () haveemerged as a popular approach and have demonstrated impressive results,for example, Pix2pix ([81]), CycleGAN [82], FUNIT [83], and MUNIT [80].These models possess the ability to learn complex data distributions andefficiently generate translated samples. However, it is important to notethat these models typically require large-scale training datasets to achieveoptimal performance. In addition, these models lack translation capabilitiesfrom multiple input domains to one domain.To address the aforementioned problems, this chapter presents a two-stage image fusion model that combines images collected using two NDEtechniques (eddy current testing (ECT) and infrared thermography (IR))to evaluate the damage. The imaging principles of these two sensors are51different, so ECT and IR can capture defect information from different do-mains. Considering that information from different frequencies can betterreveal surface defects, the proposed model incorporates both high-frequencyand low-frequency information. The first stage of the proposed model isa self-supervised autoencoder fusion network with four components: en-coder, high-frequency fusion layer, low-frequency fusion layer, and decoder.The log-Gabor filter is adopted for the high-frequency layer to extract edgeinformation from the damaged parts in both NDE images. A spatial at-tention fusion block is used to extract information about the depth of thelow-frequency fusion layer. The second stage translates the autoencoder net-work’s output into the ultrasound image domain using a pre-trained VGG-19network. Inspired by the model proposed by Gatys et al. [84], the proposedtranslation network can iteratively update the input images without train-ing data. Thus, it further improves the similarity and quality of the fusedimages. Hence, the main contributions of this chapter are summarized asfollows.• A data fusion model is proposed to address the issue of the lack ofaccuracy and reliability in traditional unimodal NDE detection tech-niques, which can learn to extract deep features in an unsupervisedmanner and fuse complementary information from eddy current andinfrared images.• An efficient framework is built to integrate data fusion and imagetranslation modules, which enables the model to be applied to smallsample datasets with precision results.• The proposed model achieves the best performance compared to otherstate-of-the-art models in the NDE dataset. Extensive experimentsare implemented to prove the effectiveness of the proposed model.The proposed model demonstrates reliable results even when applied tolimited sample datasets. This advancement highlights the effectiveness ofthe multi-modal NDE data fusion model in enhancing the identification ofdamaged areas on aircraft surfaces in real industry.525.2 Two-Stage Fusion-Translation NetworkThe flowchart of the proposed two-stage Fusion-Translation network ()is presented in Figure 5.1. The input images Ie and Ir, representing theECT images and the IR images, are first input into the fusion module, thenthe fused output If is passed through the translation module along with theultrasonic testing images Iu to obtain additional information and get theresults Io.Figure 5.1: The architecture of the proposed two-stage Fusion-Translationnetwork (FTnet). The proposed model contains two modules: (i) a two-modality image fusion module and (ii) an iterative image-to-image transla-tion module.5.2.1 Fusion ModuleAlthough deep convolutional neural networks are known to exhibit highperformance in data fusion tasks, they often require large datasets. On theother hand, the proposed fusion module employs an autoencoder architec-ture capable of generating excellent results despite its relative simplicity.Specifically, the proposed fusion module comprises an encoder part, a de-coder part, a high-frequency fusion layer and a low-frequency fusion layer.The structure of the proposed fusion module is shown in Figure 5.2.53Network ArchitectureThe network requires two pairs of ECT and IR input images, denotedIe and Ir, generating the fused images, represented as If . The encoderpart (Conv11, Conv21, Conv31, Conv12, Conv22, Conv23) contains 3 × 3convolutional layers, which are utilized to extract information from inputimages of different scales. The feature maps for each convolutional layerin the encoding network are 32, 64, and 128 channels, respectively. Fur-thermore, the filter size and the stride of convolutional operation are 3 × 3and 1, respectively. To retain more details regarding the damaged part, thepooling layers are not employed. The fusion layers incorporate two distinctfusion strategies, which are presented in Figure 5.2(b) and 5.2(c), to high-light different types of features. Lastly, the decoder part (Conv31, Conv32,Conv33) comprises three 3 × 3 convolutional layers. The output of eachencoder layer is fed to a high-frequency fusion layer and concatenated to thecorresponding layer of the decoder part to compensate for the loss of contourinformation, thereby enabling the reconstruction of the features through astraightforward and efficient decoder part.54(a) The model architecture of the proposed fusion module(b) High-frequency fusion layer(c) Low-frequency fusion layerFigure 5.2: The structure of the proposed image fusion module. 55High-Frequency Fusion LayerThe log-Gabor filter [85] is first employed to extract the high-frequencyinformation from each image. Compared to classical Gabor filters, log-Gaborfilters can yield a relatively uniform coverage of the frequency domain in anoctave-scale multi-resolution scheme. The log-Gabor filters are defined inthe log-polar coordinates of the Fourier domain as Gaussian shifted fromthe origin:G(s,t)(ρ, θ) = exp(−12(ρ− ρsσρ)2)exp(−12(θ − θ(s,t)σθ)2)(5.1)where (ρ, θ) are the log-polar coordinates (in log2 scale), s is the directionalangle, and t indexes the scale, the pair (ρs, θ(s,t)) corresponds to the coordi-nates of the center of the filter, (σρ, σθ) are bandwidths in (ρ, θ) and set as(σρ, σθ) =(1, π8). Before analyzing each image, a spectral whitening filter isused to strike a good balance between the energies of the output coefficients.Subsequently, images only containing edge features are obtained through theuse of the log-Gabor filter. The most prominent edge information is obtainedby selecting the absolute maximum of the two contour features. The high-frequency fusion layers are then applied to each encoder convolutional layer,and the outputs are added to the corresponding convolutional layers in thedecoder section.Low-Frequency Fusion LayerTo facilitate low-frequency fusion, the network uses spatial attention [86]to highlight the depth of the damaged area. The fusion layer is composed ofthree convolutional layers of varying sizes, namely 1×1, 3×3, and 5×5, eachwith different receptive fields to enable the network to capture the damagedpart at multiple scales. Subsequently, the attention map is applied to theinput modality for weight allocation. It is worth noting that the ECT tech-nique is sensitive to local differences, thereby enabling clear identification ofthe damaged part. However, details on the damaged part are not obvious.On the contrary, IR images do not exhibit strong image contrast, leading56to ambiguity at the edge of the damaged part. Nonetheless, they can ad-equately depict detailed information regarding the damage. Therefore, theIR images are fed into the spatial attention matrix, and the ECT imagesare multiplied by it.Loss FunctionThe loss function Ltotal is defined as follows:Ltotal = λLperceptual + Lssim (5.2)where Lperceptual and Lssim indicate the perceptual loss and structure simi-larity (SSIM) loss between the input images and the fused image. λ denotesthe trade-off value between Lperceptual and Lssim.The perceptual loss Lperceptual measures the high-level perceptual andsemantic differences between images [87]. The function is calculated by afeedforward deep convolutional neural network. In this module, the lossnetwork defined as φ is the 16-layer VGG network, which is pre-trained onImageNet. The perceptual loss function is the Euclidean distance betweentwo input features:Lperceptual(If , Ik) =1CjHjWj||φj(If )− φj(Ik)||22 (5.3)where Cj , Hj , Wj represents channel, height and width respectively. Ikdenotes the input ECT and IR images. If denotes the output images. φj(x)is the output of the jth layer of the network φ when processing the image Ik.Using the perceptual loss encourages the output images If to be perceptuallysimilar to the input images, but does not force them to match exactly.Lssim is obtained by the following equation:LSSIM = 1− SSIM(Ik, If )= 1− (2µIkµIf + C1)(2σIkIf + C2)(µ2Ik + µ2If+ C1)(σ2Ik + σ2If+ C1)(5.4)where µIk denotes the mean of input Ik, σ2Ikis the variance of Ik, and σIkIfindicates the covariance of Ik and If . C1 and C2 are small constants to57avoid zero in the denominator. SSIM(·) denotes the measure of structuralsimilarity [68], which measures the distance of the input images and thefused images considering the similarity of luminance, contrast, and structureinformation.5.2.2 Translation ModuleAs the ultrasonic testing (UT) images contain accurate information aboutthe depth of the damaged part, the fused image from the fusion moduleis translated into the UT representation to further enrich the informationabout the damage. However, most advanced translation models use GANs,which require large training datasets. On the other hand, the Style transfermodel (IST) [84] uses VGG-19 to iteratively translate the domain of the in-put image into the domain of the target image and does not require training.Therefore, the translation network is implemented based on the idea of IST.The translation module is illustrated in Figure 5.3. First, the featuresof the input If and UT Iu images are extracted by a VGG-19 network (ϕ).The style loss Lϕ,convstyle of target and input images is on layer Conv9. Also,the content loss Lϕ,convcontent is obtained on layers Conv2 and Conv3. The totalvariation loss is calculated between the input fused image and the targetimage. The total loss Ltotal is a linear combination of the above three losses.The pixel values of the input image are updated iteratively until matchingthe style and content of the UT image.58Figure 5.3: The structure of the proposed translation module.59Network ArchitectureIn the translation network, a pre-trained VGG-19 is adopted as a featureextractor to extract the contents and styles of the image. As the translationaims to enhance the contrast of each contour of the damaged part withoutchanging the edge, the translation network needs to convert the image tothe UT style (high contrast) while keeping the content information (shapeof the damage) unchanged. In order to make it applicable to the task, theVGG-19 network[88] is modified as follows: the ReLu activation functionin each convolutional layer is replaced by LeakyReLU, and average poolingis used instead of maximum pooling. The detailed architecture is shown inTable 5.1.Loss FunctionIn the proposed translation module, two loss functions are defined tomeasure the differences in content and style between images on the networkϕ, and one loss function is used to prevent loss of information. Here, ϕrepresents the VGG-19 network, and the goal of the translation network isto minimize the loss function:Ltotal = αLfeat + βLstyle (5.5)where α, β, and γ are the weighting factors of the content loss, style loss, andinformation loss, respectively. The Ltotal is then used to update the pixelvalues of the input image using the L-BFGS [89] optimization strategy.Figure 5.4 and 5.5 show that the information about the lower layerscan reveal more pixel-level features. Therefore, the content difference iscalculated using the feature maps obtained by Conv2 and Conv4, both ofwhich are acquired from the lower layer of the network and can ensure thatthe detailed content information is not lost. On the other hand, the styleof an image is harder to acquire, as each feature map generated by CNNcontains only one feature of the image. However, the style is a high-levelrepresentation of an image and usually refers to multiple features. Therefore,the Gram matrix is used as the style representation of the image to show60Table 5.1: The Architecture of the translation network. Conv Denotes theConvolutional Block. The negative slope of LeakyReLU is 0.1.Layer Size Stride Channel Input Channel Output ActivationConv 1 3 1 3 64 LeakyReLUConv 2 3 1 64 64 LeakyReLUAverage PoolingConv 3 3 1 64 128 LeakyReLUConv 4 3 1 128 128 LeakyReLUAverage PoolingConv 5 3 1 128 256 LeakyReLUConv 6 3 1 256 256 LeakyReLUConv 7 3 1 256 256 LeakyReLUConv 8 3 1 256 256 LeakyReLUAverage PoolingConv 9 3 1 256 512 LeakyReLUConv 10 3 1 512 512 LeakyReLUConv 11 3 1 512 512 LeakyReLUConv 12 3 1 512 512 LeakyReLUAverage PoolingConv 13 3 1 512 512 LeakyReLUConv 14 3 1 512 512 LeakyReLUConv 15 3 1 512 512 LeakyReLUConv 16 3 1 512 512 LeakyReLUAverage Poolingthe correlation between each feature and then use the correlations to definethe style of the image. Gradient descent is then employed to minimize themean square distance between the Gram matrix of the input and the target61(a) Input (b) Conv1 (c) Conv3 (d) Conv5 (e) Conv9Figure 5.4: The style reconstructions for the layers Conv1, Conv3, Conv5,Conv9 from the pre-trained VGG-19 network.(a) Input (b) Conv2 (c) Conv4 (d) Conv8 (e) Conv12Figure 5.5: The content reconstructions for the layers Conv2, Conv4, Conv8,Conv12 from the pre-trained VGG-19 network.image on the layer Conv9.The total loss function contains style loss, content loss, and total varia-tion loss. For style loss, Lstyle penalizes the difference in style, texture, andpattern, which is defined asLϕ,convstyle (If , Iu) =∑wMSE(If , Iu) (5.6)where w is the weighting factor. ϕi(x) is the feature map of the ith con-volutional layer with the shape of Ci ×Hi ×Wi. MSEi denotes the meansquared error (MSE) between the Gram matrixes of the target image Iuand the input image If in the ith convolutional layer. Specifically, MSEi isdefined asMSEi(If , Iu) =1CiHiWi(G(ϕi(If ))−G(ϕi(Iu)))2 (5.7)where G(·) is the operation to calculate the Gram matrix of the input feature62map and can be obtained by calculating the inner product between the inputfeature maps F and (F )T . The operation can be formulated as follows:G(F ) =1CHWF (F )T (5.8)where (F )T denotes the transpose matrix of the feature F .The content loss function encourages the input and target images tohave a similar feature representation. Let ϕi(x) be the feature map of theith convolutional layer with the shape Ci × Hi × Wi. Thus, the Lfeat isdefined as:Lϕ,convcontent(If , Iu) =1CiHiWi∑(ϕ(If )− ϕ(Iu))2 (5.9)Thus, the loss function of content loss is the linear combination of MSElosses in the 2th and 4th layers in the VGG-19 network. The gradients ofLfeat can be easily computed using back-propagation.5.3 Experimental Results and AnalysisIn this section, the image pre-processing and experimental setup is firstdescribed. Then, qualitative and quantitative comparisons are used to eval-uate the performance of the FTnet.5.3.1 Data Augmentation and PreparationIn the experiment, six samples were taken from six different CFRP pan-els, using the input of eddy current and infrared thermography and ultra-sonic and X-ray images as the ground truth. These four techniques wereregistered by the proposed registration model. Then five samples were ex-panded and used as training datasets, while the other sample was used asthe test dataset. The training dataset was generated by flipping, rotating,cropping, scaling, and adding contrast to the images and consisted of a totalof 2800 images. X-ray and ultrasound images were used as ground truth andcompared with the contour features and pixel depth information about thedamaged part in the fused images. A sample is shown in Figure 5.6.63(a) (b) (c) (d)Figure 5.6: An example of the processed training dataset. (a) is the eddycurrent image, (b) is the infrared image, (c) is the ultrasound image, and(d) is the X-ray image.5.3.2 Model ImplementationIn the FTnet fusion module, the eddy current and infrared images arefed into the proposed fusion network. Adam Optimizer is utilized in themini-batch of size 8, with the learning rate lr = 1 × e−4. The trainingepoch is 5. The trade-off λ = 0.17. In the translation module, the modulematches the style on layer Conv9 and the content on layers Conv2 andConv3, which reveals wl = 1 in those layers and wl = 0 in all other layers.The hyper-paramters of α is 1000, and β = 1. The input images for thetranslation module are the fused images obtained from the fusion module,while the target images are the ultrasound images. The proposed networkis implemented using NVIDIA RTX 3080Ti and PyTorch 1.10.Five typical and state-of-the-art image fusion methods were chosen tocompare with FTnet, including NestFuse [55], DIDfuse [56], DenseFuse [57],FusionGAN [58], RFN-Nest [90]. All these comparison methods were imple-mented based on their publicly available codes, and their parameters wereset by referring to their papers.5.3.3 Evaluation MetricsEight quality metrics are utilized to compare the proposed fusion methodand other existing fusion methods. These are: 1) Mutual information64(MI) [28]; 2) Normalized cross-correlation (NCC) [91]; 3) Peak signal-to-noise ratio (PSNR) [69]; 4) Root mean squared error (RMSE) [92]; 5) Simi-larity of structural index (SSIM) [68]; 6) Area ratio (AR); 7) Contour Map-ping Measure (CMM) [67]. Specifically, MI measures the mutual dependencebetween the two images, the equation follows Equation 4.9. More signifi-cant MI indicate that the fusion method can retain more original imageinformation and features.NCC measures the correlation and similarity of two images, which canbe formulated as:NCC(I1, I2) =∑Mi=1∑Nj=1(I1(i, j)− I¯1)(I2(i, j)− I¯2)MN√∑Mi=1∑Nj=1(I1(i, j)− I¯1)2(I2(i, j)− I¯2)2(5.10)where I1 denotes the model output and I2 denotes the ground truth image;M,N denotes the image height and width. A larger NCC implies that thefused image is more correlated and close to the ground truth image. PSNRcomputes the pixel-level similarity between two images, which can be definedas Equation 4.17. The large PSNR indicates that the result is closer to theground truth.RMSE measures the differences in the pixel value between two images,which is formulated as:RMSE(I1, I2) =√√√√ 1MNM∑i=1N∑j=1(I1(i, j)− I2(i, j))2 (5.11)where I1 denotes the model output and I2 denotes the ground truth image;M,N denotes the image height and width, respectively. The smaller RMSEdenotes that the pixel value of the fused image closely matches the pixelvalue of the ground truth image. The SSIM quantifies the structural simi-larity between two images, which can be defined as Equation 4.16. A largerSSIM means that the contours of the fused image retain more structuralinformation from the ground truth image.The AR compares the ratio of the damaged part in the two images, whichis formulated as:AR(I1, I2) =Damage area of I1Damage area of I2(5.12)65where I1 denotes the model output and I2 denotes the ground truth image.As AR closer to 1 implies a higher similarity between the damaged partin the two images. The CMM measures the contour distance between theinput image and the reference image, which is defined as Equation 4.15. Thesmaller CMM indicate that the contours of the fused image are similar tothose of the ground truth image.5.3.4 Comparative AnalysisQualitative AssessmentThe results in Figure 5.9 show that all six methods work well. However,compared to the proposed fusion model, the other models do not employspecific strategies to fuse the contours of the target object, resulting in thetarget not being well highlighted. As shown in Figure 5.9, the contour of theoutput image for the NestFuse model is not well preserved in the fused im-age. The DIDFuse, DenseFuse, FusionGAN, and RFN-Nest models heavilyrely on the ECT image for the contours of the output images, disregardingthe contour information present in the IR images. Unlike the other models,FTnet integrates contour information from ECT and IR images to highlightdamaged areas more prominently. The results can facilitate the subsequentinspection of CFRP panels, especially in industrial applications. This is themain difference between the proposed and existing fusion models. Anotherdistinguishing feature of FTnet is the inclusion of a translation module thathelps to further retain detailed information. The fused image captures thehigh contrast style of the UT image, resulting in an output with more salientdamaged parts. These features make FTnet a suitable model to facilitatesubsequent inspection of CFRP panels, particularly in industrial applica-tions.The IR image information is well preserved in the proposed model, pro-ducing a high-resolution image with a clear and salient defect area. Then,several objective metrics are chosen to evaluate the fusion performance, asthe few visual differences between NestFuse and the proposed method inhuman sensitivity.66(a) Input eddy (b) Input infrared (c) Ultrasound(d) Xray (e) NestFuse (f) DIDfuse(g) DenseFuse (h) FusionGAN (i) RFN-Nest(j) FTnetFigure 5.7: Example of fusion results by different methods. NestFuse, DID-fuse, DenseFuse, FusionGAN, and RFN-Nest are compared with the pro-posed FTnet model.67Quantitative AssessmentThe values of eight metrics for the fused images obtained by existingmethods and the FTnet two-stage fusion method are shown in Table 5.2.The best values are indicated in bold, and the last four metrics are thecomparisons of the edge images. The results show that the proposed FTnetoutperforms the other existing fusion models with regard to four evaluationmetrics (NCC, PSNR, RMSE, SSIM). In particular, FTnet achieves the bestSSIM values, indicating that FTnet can retain more structural informationand features of the damaged part. The best value of RMSE implies thatFTnet can have the best depth value with UT. The best values of NCCand PSNR indicate that FTnet can produce a more natural fused imagewith less artificial noise. Figure 5.8 shows the joint histogram of the fusedresults in different ultrasound models, which represents the relationship ofintensities in the exact position between two images. The histogram pixelstend to concentrate along the y = x line for the two images with a highdegree of overlap. Thus, the images fused by FTnet have more correlationwith the ultrasound images. These results demonstrate the effectiveness ofthe proposed model in fusing eddy current and infrared images for defectevaluation in CFRP panels.The results presented in Table 5.2 demonstrate that the proposed methodeffectively preserves the finer details of depth in comparison to UT. More-over, it exhibits a contour of the damaged part that closely resembles that ofan X-ray. These findings suggest that the proposed fusion module enhancesthe individual NDE techniques, resulting in improved reliability of defectarea information.5.3.5 DiscussionsThe effectiveness of the model is first discussed by comparing two singlemodules (fusion module and translation module) and two-stage models indifferent orders (fusion first (FTnet) and translation first (TFnet)). Fig-ure 5.9 depicts the single modules and the two-stage models in differentorders, and shows evidence that the fusion network (Figure 5.9(a)) does68Table 5.2: Quantitative comparison of fused images from different methods.X-ray and UT images are the ground truth. The metrics of MI, NCC, PSNR,and RMSE of the results are compared with UT. The contours of the fusionresults are compared with the contour of the X-rays using SSIM, area ratio,and CMM metrics for evaluation.Compared to UT Compared to X-rayMethods MI↑ NCC↑ PSNR↑ RMSE↓ SSIM↑ AR CMM↓Original ECT 0.270 0.876 11.295 0.272 0.384 1.123 62.248Original IR 0.382 0.969 15.539 0.167 0.731 1.104 65.611Densefuse 0.330 0.968 16.890 0.143 0.650 1.137 53.647DIDFuse 0.273 0.674 7.920 0.402 0.415 1.118 66.567Nestfuse 0.361 0.969 17.301 0.136 0.726 1.134 46.322FusionGAN 0.295 0.967 16.886 0.143 0.554 1.132 57.824RFN-Nest 0.334 0.954 15.135 0.175 0.635 1.125 53.889FTnet 0.322 0.978 17.603 0.132 0.732 1.122 49.034not provide too much detail information. On the other hand, the transla-tion network is employed to translate the eddy current and infrared imagesinto the ultrasound domain (Figure 5.9(b) and 5.9(c)), which demonstrateslimited capacity to produce clear contours. Subsequently, the fusion firstmodel (FTnet) and translation first model (TFnet) (Figure 5.9(d), 5.9(e))were implemented, leading to an improvement in the depth and contourinformation. The quantitative comparison of these models is presented inTable 5.3.First, the performance of each module is analyzed. As depicted in thecolumns of the single fusion module and single translation module on in-frared and eddy current images in Table 5.3, the metrics, including MI,NCC, SSIM, and CMM, do not show satisfactory results compared to thoseshown in Table 5.2. On the other hand, the two-stage models achieve muchbetter results, indicating that the fusion of different modalities can providecomplementary information to enhance the detection of defects in CFRPpanels. The results of the qualitative comparison are shown in Table 5.3,indicating that FTnet outperformed TFnet in all metrics, which shows the69Figure 5.8: The joint histogram for the fused results of NestFuse, DIDfuse,DenseFuse, FusionGAN, and RFN-Nest are compared with the FTnet. Thecolor of the figures represents the counts of the combination.importance of the order of the two stages. The FTnet, which fuses themodalities first and then translates the fused image to the target modality,performs better than the TFnet, which translates the individual modali-ties to the target first and then fuses them. Combining the modalities firstcan preserve more relevant information and provide a better foundation forsubsequent translation.70(a) (b) (c) (d) (e)Figure 5.9: Example of the fusion results. (a) is the result of the fusionnetwork, (b) is the result of the translation network (translating eddy currentimages), (c) is the result of the translation network (translating infraredimages), (d) is the fusion result of FTnet, and (e) is the fusion result ofTFnet.Figure 5.10: The joint histogram for the fused results of fusion network,translation network on eddy current (ET) images, translation network oninfrared (IR) images, FTnet and TFnet. The color of the figures representsthe counts of the combination.The superiority of FTnet over TFnet can be attributed to several factors.First, the fusion module in FTnet can effectively integrate the information71Table 5.3: Quantitative comparison of fused images from different methods.X-ray and UT images are the ground truth. The metrics of MI, NCC,PSNR, and RMSE of fusion results are compared with UT. The contours ofthe fusion results are compared with the contour of the X-rays using SSIM,area ratio, and CMM metrics for evaluation.Compare with UT Compare with X-rayMethods MI↑ NCC↑ PSNR↑ RMSE↓ SSIM↑ AR CMM↓Original ECT 0.270 0.876 11.295 0.272 0.384 1.123 62.248Original IR 0.382 0.969 15.539 0.167 0.731 1.104 65.611Fusion network 0.333 0.946 14.843 0.181 0.596 1.101 49.890Translation network on IR 0.305 0.979 17.802 0.115 0.602 1.061 53.472Translation network on ET 0.226 0.968 15.753 0.163 0.726 1.111 50.564FTnet 0.322 0.978 17.603 0.132 0.732 1.122 49.034TFnet 0.292 0.979 17.176 0.138 0.655 1.047 49.941from both eddy current and infrared images to produce a high-quality fusedimage with well-preserved structural and contour information. On the con-trary, TFnet performs image translation on each input image separatelybefore fusing the results, which may result in more quality loss and noise.Second, the translation network in FTnet is applied to the fused image,which contains more relevant information than the individual input images.This allows the translation network to retain more details and improve theoverall image quality. In contrast, TFnet applies the translation network toeach input image separately, which may result in a loss of important infor-mation that is only present in the fused image. The experimental results inTable 5.3 show that FTnet outperforms TFnet in several metrics, includingNCC, PSNR, SSIM, AR, and CMM. This indicates that FTnet can producea more natural fused image with less artificial noise and better structuralinformation, which is crucial for defect evaluation in CFRP panels.5.4 SummaryThis chapter introduces a novel two-stage image fusion architecture thatcombines an image fusion module with an image translation module. The72fusion module incorporates high- and low-frequency fusion layers, enablingthe network to achieve remarkable fusion results despite its simple struc-ture, and the translation module further enhances the resulting image rep-resentation without requiring additional training. Through the translationmodule, the fused images can be translated into the same domain as ul-trasound (UT) images, facilitating the use of single modality metrics forestimation purposes while enhancing the information content. By utilizingthis two-level network structure, the fused images show a high similarityto X-rays and UTs. This enhanced similarity can clearly present details ofthe damaged area, thus facilitating the characterization of the damage. Theproposed method showcases significant advantages over existing algorithmsin terms of quantitative measurements and visual quality, even when trainedon limited datasets. In industrial settings, the ability to train models usingonly a small number of data samples can greatly simplify and accelerate theworkflow. However, a limitation of the proposed model is its two-stage prop-erty. To further improve model efficiency, future improvements can involvecombining the two steps into a one-step model.73Chapter 6ConclusionIn this thesis, novel image registration and fusion models are presented toovercome challenges in the field of non-destructive evaluation (NDE). Thesechallenges include the limited availability of datasets from real industry sce-narios and the limited features provided by individual NDE techniques. Tothe best knowledge of the author, this is the first study that combines fu-sion and translation modules for image fusion in NDE images to enhance thequality of the damaged area by lightning strikes. The main contributions ofthis research can be summarized as follows.• To overcome the challenges of unalignment and multi-resolution datain the NDE field, a structurally enhanced image registration model isproposed. In this model, an SCB filter is first employed to enhancethe features of the damaged area in the ground truth images withstructure consistency. Then a multi-resolution registration model isemployed, allowing for the alignment of images at different scales andaccommodating the diverse resolutions present in NDE data. The pro-posed model achieves the lowest distance error and highest structuresimilarity on all three modalities needed to be registered compared tothree state-of-the-art baselines and manual registering methods, whichdemonstrates superior performance. Thus, manual registration can bereplaced to save more time and get more accurate results.• To tackle the challenge that existing methods for enhancing damageinformation using ECT and IR data have not been fully investigated, adeep learning-based image fusion-translation network (FTnet) is pro-posed to enhance the damaged area of single NDE images by effec-tively integrating their respective features. The FTnet combines an74unsupervised learning fusion module and a pre-trained image transla-tion module. The FTnet achieves excellent results, surpassing existingapproaches in enhancing the information with a small training dataset,which implies that FTnet can generate better results that have moresimilarity with the ultrasound images in the depth information andmore similar to X-ray images in the contour of the damaged part.Hence, FTnet can significantly improve the quality of the single NDEimages and have much explicit information about the damaged partin the lightning stroke surface.To conclude, in terms of multi-modal NDE image registration and fu-sion, this study proposes an automated image registration and fusion modelfor combining the eddy current and infrared thermography images to obtainmore depth information and clearer information about contour information.Additionally, the workload of human resources is largely reduced, as manualalignment of the NDE images is not required. With the proposed networks,experts can focus mainly on fused results for further quantification of thedamaged area on the aircraft surface. From the perspective of the method-ologies, the two proposed networks based on hand-craft strategies and cus-tomized neural networks encourage the model to get excellent results in alimited dataset.The thesis proposes promising methods for multi-modal image registra-tion and fusion, which are accurate and robust. Nonetheless, there are manypossible future extensions for the proposed method. The potential futurestudies for the research are discussed and listed below:• For the image registration model, although it has demonstrated ex-cellent alignment performance with the experiment dataset, its per-formance on other NDE datasets is not performed due to the lack ofpublic datasets. If more public datasets become available, the gen-eralizability of the proposed registration model can be evaluated andimproved.• Additionally, considering the promising results and low running timeof supervised learning in registration models, another extension could75involve exploring deep learning-based models if a larger dataset be-comes accessible.• Regarding the fusion model, which currently operates as a two-stepframework, a potential improvement could involve converting the fu-sion and translation module into a one-step framework. This mayimprove the overall efficiency and simplicity of the fusion process.While the proposed models exhibit robustness and reliability, it is im-portant to acknowledge their limitations and consider potential future plansto further enhance their generality and efficiency.76Bibliography[1] G. Gardiner, “Lightning strike protection for composite structures,”vol. 14, pp. 44–50, 07 2006. → pages 1[2] D. Morgan, C. Hardwick, S. Haigh, and A. Meakins, “The interactionof lightning with aircraft and the challenges of lightning testing,” 122012. → pages 1[3] B. Glushakow, “Effective lightning protection for wind turbine genera-tors,” IEEE Transactions on Energy Conversion, vol. 22, no. 1, pp. 214–222, 2007. → pages 1[4] M. Kucharska and J. Jaskowska-Leman´ska, “Active thermography indiagnostics of timber elements covered with polychrome,” Materials,vol. 14, no. 5, p. 1134, 2021. → pages 2[5] L. Cheng and G. Y. Tian, “Surface crack detection for carbon fiberreinforced plastic (cfrp) materials using pulsed eddy current thermog-raphy,” IEEE Sensors Journal, vol. 11, no. 12, pp. 3261–3268, 2011. →pages 2, 50[6] B. Yilmaz, A. Ba, E. Jasiuniene, H.-K. Bui, and G. Berthiau, “Eval-uation of bonding quality with advanced nondestructive testing (ndt)and data fusion,” Sensors, vol. 20, no. 18, p. 5127, 2020. → pages 2[7] Z. Liu, D. S. Forsyth, J. P. Komorowski, K. Hanasaki, and T. Kirubara-jan, “Survey: State of the art in nde data fusion techniques,” IEEEtransactions on Instrumentation and Measurement, vol. 56, no. 6,pp. 2435–2451, 2007. → pages 3, 2677Bibliography[8] E. Jasiu¯niene˙, B. Yilmaz, D. Smagulova, G. A. Bhat, V. Cice˙nas,E. Zˇukauskas, and L. Mazˇeika, “Non-destructive evaluation of the qual-ity of adhesive joints using ultrasound, x-ray, and feature-based datafusion,” Applied Sciences, vol. 12, no. 24, 2022. → pages 3[9] A. Bigand, C. Espinosa, and J.-M. Bauchire, “Lightning surface ex-plosion impact study on damage generation into composite.” 2019. →pages 7[10] S. Aerospace, “Aircraft lightning zone,” Aerospace Recommended Prac-tice, ARP5414A, 2012. → pages 7[11] A.-. L. Committee et al., “Aircraft lightning environment and relatedtest waveforms,” Society of Automotive Engineers International Tech.Rep. ARP5412B, 2013. → pages x, 9[12] V. Kumar, T. Yokozeki, C. Karch, A. A. Hassen, C. J. Hershey, S. Kim,J. M. Lindahl, A. Barnes, Y. K. Bandari, and V. Kunc, “Factors af-fecting direct lightning strike damage to fiber reinforced composites: Areview,” Composites Part B: Engineering, vol. 183, p. 107688, 2020. →pages 9[13] P. Rajesh, F. Sirois, and D. Therriault, “Damage response of compos-ites coated with conducting materials subjected to emulated lightningstrikes,” Materials & Design, vol. 139, pp. 45–55, 2018. → pages 9[14] G. N. Szatkowski, T. X. Nguyen, S. V. Koppen, J. J. Ely, and J. J.Mielnik, “Electrical characterizations of lightning strike protection tech-niques for composite materials,” in International Conference on Light-ning and Static Electricity, no. LF99-9437, 2009. → pages 10[15] K. Hema and S. Dasgupta, “Non-destructive evaluation of lightningstrike induced damages in sandwich composites for unmanned air vehi-cle structures,” The e-Journal of Nondestructive Testing, vol. 20, no. 6,2015. → pages 1478Bibliography[16] J.-K. Jang, J.-M. Hyun, D.-S. Son, and J.-R. Lee, “Nondestructive andelectromagnetic evaluations of stealth structures damaged by lightningstrike,” Journal of Intelligent Material Systems and Structures, vol. 30,no. 17, pp. 2567–2574, 2019. → pages 14[17] R. L. Watson, D. R. Billson, D. A. Hutchins, and F. Ciampa, “Ther-mosonics: electrostatic transducer based air-coupled ultrasonics andthermography for nde of carbon fibre reinforced polymer materi-als,” in Nondestructive Characterization and Monitoring of AdvancedMaterials, Aerospace, Civil Infrastructure, and Transportation XVI,vol. 12047, pp. 18–30, SPIE, 2022. → pages 18[18] S. Nayak, S. Pattnaik, I. Panigrahi, R. Nayak, S. Parida, et al., “Smalldelamination detection in carbon fibre reinforced polymer compositebeam by ndt and vibration analysis,” Indian Journal of Engineeringand Materials Sciences (IJEMS), vol. 27, no. 3, pp. 789–794, 2021. →pages 19[19] H. D. Roh, S. Y. Oh, and Y.-B. Park, “Self-sensing impact damage inand non-destructive evaluation of carbon fiber-reinforced polymers us-ing electrical resistance and the corresponding electrical route models,”Sensors and Actuators A: Physical, vol. 332, p. 112762, 2021. → pages19[20] I. Y. Lee, H. D. Roh, H. W. Park, and Y.-B. Park, “Advancednon-destructive evaluation of impact damage growth in carbon-fiber-reinforced plastic by electromechanical analysis and machine learningclustering,” Composites Science and Technology, vol. 218, p. 109094,2022. → pages 19[21] A. Katunin, A. Wronkowicz-Katunin, W. Danek, and M. Wylez˙o l,“Modeling of a realistic barely visible impact damage in compositestructures based on ndt techniques and numerical simulations,” Com-posite Structures, vol. 267, p. 113889, 2021. → pages 19[22] J. Segers, S. Hedayatrasa, E. Verboven, G. Poelman, W. Van Paepegem,79Bibliographyand M. Kersemans, “In-plane local defect resonances for efficient vi-brothermography of impacted carbon fiber-reinforced polymers (cfrp),”Ndt & E International, vol. 102, pp. 218–225, 2019. → pages 19[23] S. Patil and D. M. Reddy, “Impact damage assessment in carbon fiberreinforced composite using vibration-based new damage index and ul-trasonic c-scanning method,” in Structures, vol. 28, pp. 638–650, Else-vier, 2020. → pages 19[24] D. Shoukroun, L. Massimi, M. Endrizzi, D. Bate, A. Olivo, andP. Fromme, “Composites impact damage detection and characteriza-tion using ultrasound and x-ray nde techniques,” in Health Monitoringof Structural and Biological Systems XIV, vol. 11381, pp. 27–33, SPIE,2020. → pages 19[25] E. Ichi, F. Jafari, and S. Dorafshan, “Sdnet2021: Annotated ndedataset for subsurface structural defects detection in concrete bridgedecks,” Infrastructures, vol. 7, no. 9, p. 107, 2022. → pages 20[26] P. W. Spaeth, E. L. Frankforter, W. C. Schneck, M. R. Webster, M. J.Godbold, and D. F. Perey, “Automated registration of multi-mode non-destructive evaluation data,” in Nondestructive Characterization andMonitoring of Advanced Materials, Aerospace, Civil Infrastructure, andTransportation XVII, vol. 12487, pp. 116–124, SPIE, 2023. → pages 20[27] S. M. Wallentine and M. D. Uchic, “A study on ground truth datafor impact damaged polymer matrix composites,” in AIP conferenceproceedings, vol. 1949, p. 120002, AIP Publishing LLC, 2018. → pages20[28] A. Kraskov, H. Sto¨gbauer, and P. Grassberger, “Estimating mutualinformation,” Phys. Rev. E, vol. 69, p. 066138, Jun 2004. → pages 20,65[29] M. P. Heinrich, M. Jenkinson, M. Bhushan, T. Matin, F. V. Gleeson,M. Brady, and J. A. Schnabel, “Mind: Modality independent neigh-80Bibliographybourhood descriptor for multi-modal deformable registration,” Medicalimage analysis, vol. 16, no. 7, pp. 1423–1435, 2012. → pages 20[30] E. Haber and J. Modersitzki, “Intensity gradient based registrationand fusion of multi-modal images,” in Medical Image Computingand Computer-Assisted Intervention–MICCAI 2006: 9th InternationalConference, Copenhagen, Denmark, October 1-6, 2006. Proceedings,Part II 9, pp. 726–733, Springer, 2006. → pages 20[31] D. De Nigris, D. L. Collins, and T. Arbel, “Multi-modal image regis-tration based on gradient orientations of minimal uncertainty,” IEEETransactions on Medical Imaging, vol. 31, no. 12, pp. 2343–2354, 2012.→ pages 20[32] X. Shen, L. Xu, Q. Zhang, and J. Jia, “Multi-modal and multi-spectralregistration for natural images,” in Computer Vision–ECCV 2014: 13thEuropean Conference, Zurich, Switzerland, September 6-12, 2014, Pro-ceedings, Part IV 13, pp. 309–324, Springer, 2014. → pages 20[33] S.-J. Chen, H.-L. Shen, C. Li, and J. H. Xin, “Normalized total gradient:A new measure for multispectral image registration,” IEEE Transac-tions on Image Processing, vol. 27, no. 3, pp. 1297–1310, 2017. → pages20, 32, 38[34] C. Wang, G. Yang, and G. Papanastasiou, “Fire: Unsupervised bi-directional inter- and intra-modality registration using deep networks,”in 2021 IEEE 34th International Symposium on Computer-Based Med-ical Systems (CBMS), pp. 510–514, 2021. → pages 20[35] J. Li, Q. Hu, and M. Ai, “Rift: Multi-modal image matching based onradiation-variation insensitive feature transform,” IEEE Transactionson Image Processing, vol. 29, pp. 3296–3310, 2019. → pages 21[36] Z. Xu, J. Luo, J. Yan, R. Pulya, X. Li, W. Wells, and J. Ja-gadeesan, “Adversarial uni-and multi-modal stream networks for multi-modal image registration,” in Medical Image Computing and Computer81BibliographyAssisted Intervention–MICCAI 2020: 23rd International Conference,Lima, Peru, October 4–8, 2020, Proceedings, Part III 23, pp. 222–232,Springer, 2020. → pages 21[37] M. Arar, Y. Ginger, D. Danon, A. H. Bermano, and D. Cohen-Or,“Unsupervised multi-modal image registration via geometry preserv-ing image-to-image translation,” in Proceedings of the IEEE/CVF con-ference on computer vision and pattern recognition, pp. 13410–13419,2020. → pages 21[38] N. Pielawski, E. Wetzer, J. O¨fverstedt, J. Lu, C. Wa¨hlby, J. Lindblad,and N. Sladoje, “Comir: Contrastive multimodal image representationfor registration,” Advances in neural information processing systems,vol. 33, pp. 18433–18444, 2020. → pages 21[39] C. Qin, B. Shi, R. Liao, T. Mansi, D. Rueckert, and A. Kamen, “Unsu-pervised deformable registration for multi-modal images via disentan-gled representations,” in Information Processing in Medical Imaging:26th International Conference, IPMI 2019, Hong Kong, China, June2–7, 2019, Proceedings 26, pp. 249–261, Springer, 2019. → pages 21[40] S.-Y. Cao, H.-L. Shen, S.-J. Chen, and C. Li, “Boosting structure con-sistency for multispectral and multimodal image registration,” IEEETransactions on Image Processing, vol. 29, pp. 5147–5162, 2020. →pages 22, 27, 29, 38[41] A. Maghami, M. Salehi, and M. Khoshdarregi, “Automated vision-based inspection of drilled cfrp composites using multi-light imagingand deep learning,” CIRP Journal of Manufacturing Science and Tech-nology, vol. 35, pp. 441–453, 2021. → pages 22[42] W. Sheng, Y. Liu, and D. So¨ffker, “A novel adaptive boosting algorithmwith distance-based weighted least square support vector machine andfilter factor for carbon fiber reinforced polymer multi-damage classifi-cation,” Structural Health Monitoring, vol. 22, no. 2, pp. 1273–1289,2023. → pages82Bibliography[43] X. Gros, J. Bousigue, and K. Takahashi, “Ndt data fusion at pixellevel,” Ndt & E International, vol. 32, no. 5, pp. 283–292, 1999. →pages 22[44] S. Paris, S. W. Hasinoff, and J. Kautz, “Local laplacian filters: Edge-aware image processing with a laplacian pyramid.,” ACM Trans.Graph., vol. 30, no. 4, p. 68, 2011. → pages 22[45] S. Mallat, “A theory for multiresolution signal decomposition: thewavelet representation,” IEEE Transactions on Pattern Analysis andMachine Intelligence, vol. 11, no. 7, pp. 674–693, 1989. → pages 22[46] E. Simoncelli and W. Freeman, “The steerable pyramid: a flexible ar-chitecture for multi-scale derivative computation,” in Proceedings., In-ternational Conference on Image Processing, vol. 3, pp. 444–447 vol.3,1995. → pages 22[47] Z. Wang, D. Ziou, C. Armenakis, D. Li, and Q. Li, “A comparativeanalysis of image fusion methods,” IEEE Transactions on Geoscienceand Remote Sensing, vol. 43, no. 6, pp. 1391–1402, 2005. → pages 22[48] S. Balakrishnan, M. Cacciola, L. Udpa, B. P. Rao, T. Jayakumar,and B. Raj, “Development of image fusion methodology using discretewavelet transform for eddy current images,” Ndt & E International,vol. 51, pp. 51–57, 2012. → pages 23[49] B. C. de Oliveira, P. Nienheysen, C. R. Baldo, A. A. Goncalves Jr,and R. H. Schmitt, “Improved impact damage characterisation in cfrpsamples using the fusion of optical lock-in thermography and opticalsquare-pulse shearography images,” NDT & E International, vol. 111,p. 102215, 2020. → pages 23[50] Q. Wang, H. Zhou, R. Xia, Q. Liu, and B. Y. Zhao, “Time segmentedimage fusion based multi- depth defects imaging method in compositeswith pulsed terahertz,” IEEE Access, vol. 8, pp. 155529–155537, 2020.→ pages 2383Bibliography[51] G. Vesala, V. Ghali, A. V. Lakshmi, and R. Naik, “Deep and hand-crafted feature fusion for automatic defect detection in quadratic fre-quency modulated thermal wave imaging,” Russian Journal of Nonde-structive Testing, vol. 57, pp. 476–485, 2021. → pages 23[52] C. Hua, S. Chen, B. Du, and G. Xu, “Defect identification method ofcarbon fiber sucker rod based on multi-sensor information fusion andgooglenet-based deep learning model,” Available at SSRN 4149572. →pages 23[53] Z. Zhang and J. Zhang, “Quantitative nondestructive testing of brokenwires for wire rope based on multi-image fusion,” Russian Journal ofNondestructive Testing, vol. 58, no. 7, pp. 643–656, 2022. → pages 23[54] H. Li, X.-J. Wu, and J. Kittler, “Rfn-nest: An end-to-end residualfusion network for infrared and visible images,” Information Fusion,vol. 73, pp. 72–86, 2021. → pages 24[55] H. Li, X.-J. Wu, and T. Durrani, “Nestfuse: An infrared and visible im-age fusion architecture based on nest connection and spatial/channelattention models,” IEEE Transactions on Instrumentation and Mea-surement, vol. 69, no. 12, pp. 9645–9656, 2020. → pages 24, 51, 64[56] Z. Zhao, S. Xu, C. Zhang, J. Liu, J. Zhang, and P. Li, “Didfuse: Deepimage decomposition for infrared and visible image fusion,” in Proceed-ings of the Twenty-Ninth International Joint Conference on ArtificialIntelligence, International Joint Conferences on Artificial IntelligenceOrganization, jul 2020. → pages 24, 64[57] H. Li and X.-J. Wu, “Densefuse: A fusion approach to infrared andvisible images,” IEEE Transactions on Image Processing, vol. 28, no. 5,pp. 2614–2623, 2019. → pages 24, 51, 64[58] J. Ma, W. Yu, P. Liang, C. Li, and J. Jiang, “Fusiongan: A generativeadversarial network for infrared and visible image fusion,” InformationFusion, vol. 48, pp. 11–26, 2019. → pages 24, 6484Bibliography[59] K. Ko¨ser, Affine Registration, pp. 22–25. Boston, MA: Springer US,2014. → pages 26[60] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens,“Multimodality image registration by maximization of mutual informa-tion,” IEEE transactions on Medical Imaging, vol. 16, no. 2, pp. 187–198, 1997. → pages 26[61] J.-C. Yoo and T. H. Han, “Fast normalized cross-correlation,” Circuits,systems and signal processing, vol. 28, pp. 819–843, 2009. → pages 26[62] M. Hisham, S. N. Yaakob, R. Raof, A. A. Nazren, and N. Wafi, “Tem-plate matching using sum of squared difference and normalized crosscorrelation,” in 2015 IEEE student conference on research and devel-opment (SCOReD), pp. 100–104, IEEE, 2015. → pages 26[63] D. Guevorkian, A. Launiainen, P. Liuha, and V. Lappalainen, “Archi-tectures for the sum of absolute differences operation,” in IEEE Work-shop on Signal Processing Systems, pp. 57–62, IEEE, 2002. → pages26[64] E. Chee and Z. Wu, “Airnet: Self-supervised affine registra-tion for 3d medical images using neural networks,” arXiv preprintarXiv:1810.02583, 2018. → pages 32[65] J. Klein and T. Aach, “Multispectral filter wheel cameras: modelingaberrations for filters in front of lens,” in Electronic imaging, 2012. →pages[66] N. Masoumi, Y. Xiao, and H. Rivaz, “Arena: Inter-modality affineregistration using evolutionary strategy,” International journal of com-puter assisted radiology and surgery, vol. 14, pp. 441–450, 2019. →pages 32[67] V. Movahedi and J. H. Elder, “Design and perceptual validation ofperformance measures for salient object segmentation,” in 2010 IEEE85BibliographyComputer Society Conference on Computer Vision and Pattern Recog-nition - Workshops, pp. 49–56, 2010. → pages 36, 65[68] D. Brunet, E. R. Vrscay, and Z. Wang, “On the mathematical prop-erties of the structural similarity index,” IEEE Transactions on ImageProcessing, vol. 21, no. 4, pp. 1488–1499, 2011. → pages 36, 58, 65[69] J. Korhonen and J. You, “Peak signal-to-noise ratio revisited: Is sim-ple beautiful?,” in 2012 Fourth International Workshop on Quality ofMultimedia Experience, pp. 37–38, 2012. → pages 36, 65[70] M. McCormick, X. Liu, J. Jomier, C. Marion, and L. Ibanez, “Itk: en-abling reproducible research and open science,” Frontiers in neuroin-formatics, vol. 8, p. 13, 2014. → pages 38[71] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou, “LoFTR: Detector-freelocal feature matching with transformers,” CVPR, 2021. → pages 38[72] M. Kumar and S. Dass, “A total variation-based algorithm for pixel-level image fusion,” IEEE Transactions on Image Processing, vol. 18,no. 9, pp. 2137–2143, 2009. → pages 50[73] H. Li, X.-J. Wu, and J. Kittler, “Infrared and visible image fusion usinga deep learning framework,” in 2018 24th International Conference onPattern Recognition (ICPR), IEEE, aug 2018. → pages 51[74] S. Xu, Z. Zhao, Y. Wang, C. Zhang, J. Liu, and J. Zhang, “Deepconvolutional sparse coding networks for image fusion,” 2020. → pages[75] Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao, and L. Zhang, “Ifcnn:A general image fusion framework based on convolutional neural net-work,” Information Fusion, vol. 54, pp. 99–118, 2020. → pages[76] R. Liu, Z. Liu, J. Liu, and X. Fan, “Searching a hierarchically aggre-gated fusion architecture for fast multi-modality image fusion,” MM’21, (New York, NY, USA), p. 1600–1608, Association for ComputingMachinery, 2021. → pages86Bibliography[77] R. Rayhana, Y. Jiao, Z. Bahrami, Z. Liu, A. Wu, and X. Kong,“Valve detection for autonomous water pipeline inspection platform,”IEEE/ASME Transactions on Mechatronics, vol. 27, no. 2, pp. 1070–1080, 2021. → pages[78] R. Rayhana, Y. Jiao, Z. Liu, A. Wu, and X. Kong, “Real-time embeddedsystem for valve detection in water pipelines,” Journal of Real-TimeImage Processing, vol. 19, no. 2, pp. 247–259, 2022. → pages 51[79] L. Jian, X. Yang, Z. Liu, G. Jeon, M. Gao, and D. Chisholm, “Sedrfuse:A symmetric encoder–decoder with residual block network for infraredand visible image fusion,” IEEE Transactions on Instrumentation andMeasurement, vol. 70, pp. 1–15, 2021. → pages 51[80] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, “Multimodal unsu-pervised image-to-image translation,” in ECCV, 2018. → pages 51[81] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image trans-lation with conditional adversarial networks,” CVPR, 2017. → pages51[82] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-imagetranslation using cycle-consistent adversarial networks,” in ComputerVision (ICCV), 2017 IEEE International Conference on, 2017.→ pages51[83] K. Saito, K. Saenko, and M.-Y. Liu, “Coco-funit: Few-shot unsuper-vised image translation with a content conditioned style encoder,” arXivpreprint arXiv:2007.07431, 2020. → pages 51[84] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer usingconvolutional neural networks,” in Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), June 2016. →pages 52, 58[85] S. Fischer, F. Sˇroubek, L. Perrinet, R. Redondo, and G. Cristo´bal, “Self-invertible 2d log-gabor wavelets.,” International Journal of ComputerVision, vol. 75, no. 2, 2007. → pages 5687Bibliography[86] S. Chang, Y. Zhang, F. Zhang, X. Zhao, S. Huang, Z. Feng, and Z. Wei,“Spatial attention fusion for obstacle detection using mmwave radar andvision sensor,” Sensors, vol. 20, no. 4, 2020. → pages 56[87] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-timestyle transfer and super-resolution,” in Computer Vision – ECCV 2016(B. Leibe, J. Matas, N. Sebe, and M. Welling, eds.), (Cham), pp. 694–711, Springer International Publishing, 2016. → pages 57[88] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” pp. 1–14, Computational and BiologicalLearning Society, 2015. → pages 60[89] C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal, “Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization,”vol. 23, p. 550–560, dec 1997. → pages 60[90] H. Li, X.-J. Wu, and J. Kittler, “Rfn-nest: An end-to-end residualfusion network for infrared and visible images,” Information Fusion,vol. 73, pp. 72–86, 2021. → pages 64[91] J.-C. Yoo and T. H. Han, “Fast normalized cross-correlation,” vol. 28,p. 819–843, dec 2009. → pages 65[92] T. Chai and R. R. Draxler, “Root mean square error (rmse) or meanabsolute error (mae)? – arguments against avoiding rmse in the liter-ature,” Geoscientific Model Development, vol. 7, no. 3, pp. 1247–1250,2014. → pages 6588
Thesis/Dissertation
eng
Electrical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Liu, Zheng (Professor of engineering)
Registration and fusion of aircraft lightning damage NDE data
Text