Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Liu, Xiangrui
Master of Applied Science - MASc
University of British Columbia
Marine mammals and their ecosystem face significant threats from, for example, military active sonar and marine transportation. To mitigate this harm, early detection and classification of marine mammals are essential. Recent solutions involve spectrogram comparison and machine learning. However, the solutions show weaknesses in efficiency. Therefore, we propose a novel knowledge distillation framework, named XCFSMN, for this problem. We construct a teacher model that fuses the features extracted from an X-vector extractor, a DenseNet, and a Cross-Covariance attended compact Feed-Forward Sequential Memory Network (cFSMN). The teacher model transfers knowledge to a simpler cFSMN model through a temperature-cooling strategy for efficient learning. Compared to multiple convolutional neural network backbones and transformers, the proposed framework achieves state-of-the-art efficiency and performance. The improved model size is approximately 25 times smaller and the inference time is 27 times shorter on average without affecting the model’s accuracy significantly.
Highly Efficient Sound Classificationfor Marine MammalsbyXiangrui LiuB.Sc., The University of British Columbia, 2020A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Electrical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)January 2024© Xiangrui Liu 2024The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, the thesis entitled:Highly Efficient Sound Classification for Marine Mammals submitted byXiangrui Liu in partial fulfillment of the requirements forthe degree of Master of Applied Science.Dr. Julian Cheng, School of EngineeringSupervisorDr. Shan Du, Irving K Barber Faculty of ScienceCo-supervisorDr. Jahangir Hossain, School of EngineeringSupervisory Committee MemberDr. Thomas Johnson, School of EngineeringSupervisory Committee MemberDr. Mahmudur Fatmi, School of EngineeringExternal ExamineriiAbstractMarine mammals and their ecosystem face significant threats from, forexample, military active sonar and marine transportation. To mitigatethis harm, early detection and classification of marine mammals are essen-tial. Recent solutions involve spectrogram comparison and machine learn-ing. However, the solutions show weaknesses in efficiency. Therefore, wepropose a novel knowledge distillation framework, named XCFSMN, forthis problem. We construct a teacher model that fuses the features ex-tracted from an X-vector extractor, a DenseNet, and a Cross-Covariance at-tended compact Feed-Forward Sequential Memory Network (cFSMN). Theteacher model transfers knowledge to a simpler cFSMN model through atemperature-cooling strategy for efficient learning. Compared to multipleconvolutional neural network backbones and transformers, the proposedframework achieves state-of-the-art efficiency and performance. The im-proved model size is approximately 25 times smaller and the inference timeis 27 times shorter on average without affecting the model’s accuracy signif-icantly.iiiLay SummaryMarine mammals, like dolphins and whales, are facing serious threatsfrom things like loud military sonar and maritime traffic. To help protectthem, it is crucial to rapidly detect and identify these animals. Traditionalapproaches involve human inspection using a passive acoustic monitor ortelescope. More recent solutions use spectrogram comparison and automaticclassification through machine learning. However, the existing solutions areinefficient as they may require the presence of specialists; they are susceptibleto weather conditions; and they tend to be slow in response.In alignment with the advancements in machine learning, we propose adeep-learning framework named XCFSMN to address the problem efficientlyand accurately. The framework consists of two neural networks: a powerfulyet computationally expensive teacher model and a smaller, faster studentmodel. The teacher model is complex and powerful but at the expense ofinference speed and size. In other words, it takes time to identify the soundand it may be too big to deploy onto most vessels. Therefore, we constructa smaller student model that learns from the power teacher model. Throughsuch a knowledge transfer process, the student model is able to attain 90%of the knowledge from the teacher model and makes a prediction at a speedthat may be 27 times faster than the teacher model on average. In summary,the framework achieves state-of-the-art efficiency and performance, markinga step forward in protecting these animals and their homes in the ocean.ivPrefaceA list of my conference publications related to this thesis at The Uni-versity of British Columbia is provided in the following.1. Xiangrui Liu, Xiaoou Liu, Shan Du, and Julian Cheng. Hear You SayYou: An Efficient Framework For Marine Mammal Sounds’ Classifica-tion. in Proceedings of the AAAI Conference on Artificial Intelligence.(Accepted)1 Xiangrui Liu and Julian Cheng, A Highly Efficient Marine Mam-mals Classifier Based on a Cross-Covariance Attended Compact Feed-Forward Sequential Memory Network (Student Abstract), in Proceed-ings of the AAAI Conference on Artificial Intelligence, vol. 37, pp.16268–16269vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 1Chapter 2: Background . . . . . . . . . . . . . . . . . . . . . . . 52.1 Threats to Cetacean Populations . . . . . . . . . . . . . . . . 52.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2.1 Acoustic Signal Processing and Representation . . . . 72.2.2 Machine Learning Based Methods . . . . . . . . . . . 82.2.3 Deep Learning Based Methods . . . . . . . . . . . . . 82.2.4 Knowledge Distillation . . . . . . . . . . . . . . . . . . 122.3 Comparison Models . . . . . . . . . . . . . . . . . . . . . . . 132.3.1 AlexNet . . . . . . . . . . . . . . . . . . . . . . . . . . 132.3.2 VGG-16 . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3.3 ResNet . . . . . . . . . . . . . . . . . . . . . . . . . . 16viTABLE OF CONTENTS2.3.4 Data-efficient Image Transformers (DeiT) . . . . . . . 172.3.5 Longformer . . . . . . . . . . . . . . . . . . . . . . . . 222.3.6 FNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Chapter 3: Methodology . . . . . . . . . . . . . . . . . . . . . . 253.1 Overall Structure of The Framework . . . . . . . . . . . . . . 253.2 Feature Extraction and Pre-processing . . . . . . . . . . . . . 263.2.1 FBank Extraction . . . . . . . . . . . . . . . . . . . . 263.2.2 Cepstral Mean and Variance Normalization (CMVN) . 263.3 Teacher Model . . . . . . . . . . . . . . . . . . . . . . . . . . 283.3.1 DenseNet-121 . . . . . . . . . . . . . . . . . . . . . . . 283.3.2 TDNN . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.3.3 Modified Student Model Unit . . . . . . . . . . . . . . 323.4 Student Model (CC-FSMN) . . . . . . . . . . . . . . . . . . . 333.4.1 Compact Feed-Forward Sequential Memory Network . 333.4.2 Cross-Covariance Attention (XCA) . . . . . . . . . . . 353.5 Loss Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Chapter 4: Experimental Setup . . . . . . . . . . . . . . . . . . 394.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.2 Experiment Details . . . . . . . . . . . . . . . . . . . . . . . . 394.3 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . 41Chapter 5: Results Comparison . . . . . . . . . . . . . . . . . . 435.1 Experiment Results . . . . . . . . . . . . . . . . . . . . . . . . 435.2 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . 46Chapter 6: Conclusion and Future Work . . . . . . . . . . . . . 51Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Appendix A: Tables . . . . . . . . . . . . . . . . . . . . . . . . . . 59viiList of TablesTable 2.1 The trend of whale population from 1890 to 2001 . . . 5Table 4.1 Audio length of the five classes in hours. . . . . . . . . 40Table 5.1 Comparative results of XCFSMN and the baselines onefficiency. ‘TT’ denotes the training time in minutesand ‘IT’ denotes the inference time in seconds. . . . . 43Table 5.2 Comparative results of 12-class WMMSD. . . . . . . . 44Table 5.3 Comparative results of 5-class WMMSD. . . . . . . . . 44Table 5.4 Comparative results of 5-class Moby. . . . . . . . . . . 45Table 5.5 Comparative results of 5-class merged. . . . . . . . . . 45Table 5.6 Results of the ablation study on the student model.‘TT’ denotes the training time in minutes and ‘IT’denotes the inference time in seconds. . . . . . . . . . 48Table 5.7 Results of the ablation study on the KD temperaturestrategy. ‘TT’ denotes the training time in minutesand ‘IT’ denotes the inference time in seconds. . . . . 49Table 6.1 TOPSIS score of 5-class Moby data. . . . . . . . . . . 59Table 6.2 TOPSIS score of 5-class Watkins data. . . . . . . . . . 60Table 6.3 TOPSIS score of 5-class merged data. . . . . . . . . . 60Table 6.4 TOPSIS score of 12-class Watkins data. . . . . . . . . 61Table 6.5 Overall TOPSIS score. . . . . . . . . . . . . . . . . . . 61viiiList of FiguresFigure 2.1 Sample convolution of a 7x7 pixel RGB image. . . . . 9Figure 2.2 The architecture of a standard transformer. Adaptedfrom [51]. . . . . . . . . . . . . . . . . . . . . . . . . . 11Figure 2.3 AlexNet architecture. . . . . . . . . . . . . . . . . . . 14Figure 2.4 VGG architecture. . . . . . . . . . . . . . . . . . . . . 15Figure 2.5 The residual block. . . . . . . . . . . . . . . . . . . . 17Figure 2.6 The ResNet-50 architecture. Adapted from [18]. . . . 18Figure 2.7 Deit structure. . . . . . . . . . . . . . . . . . . . . . . 21Figure 2.8 Architecture of a Longformer. . . . . . . . . . . . . . 22Figure 2.9 Demonstration of different attention maps. . . . . . . 23Figure 2.10 Structure of FNet. . . . . . . . . . . . . . . . . . . . . 24Figure 3.1 The overall flow of XCFSMN. . . . . . . . . . . . . . 25Figure 3.2 The flow of FBanks extraction. . . . . . . . . . . . . . 27Figure 3.3 The structure of the teacher model. . . . . . . . . . . 29Figure 3.4 FBank spectrogram of bowhead whale and minke whale. 29Figure 3.5 The structure of a DenseNet. . . . . . . . . . . . . . . 31Figure 3.6 The structure of the teacher model. . . . . . . . . . . 32Figure 3.7 The structure of the student model. . . . . . . . . . . 33Figure 3.8 The structures of FSMN (left) and cFSMN (right)networks. . . . . . . . . . . . . . . . . . . . . . . . . . 34Figure 3.9 The illustration of estimating attention in XCA. . . . 35Figure 5.1 Confusion matrix for student model’s results on com-bined dataset. . . . . . . . . . . . . . . . . . . . . . . 47Figure 5.2 Spectrograms of three whale species. . . . . . . . . . . 48ixLIST OF FIGURESFigure 5.3 Visualization of features using a t-SNE plot for 12-class Watkins experiment. . . . . . . . . . . . . . . . 50xAcknowledgementsI would like to thank my advisor, Professor Julian Cheng, and ProfessorShan Du for their invaluable mentorship, guidance, and support. I wouldalso like to thank my family for their support and encouragement.xiChapter 1IntroductionMarine mammals play an important role in marine ecosystems [6]. Ma-rine mammals contribute to the overall health and stability of the ecosystemin the following ways. The top predators, such as killer whales and certaindolphins play essential roles in regulating their prey’s population. Further,marine mammals can also be indicators of the ecosystem’s health. As preda-tors in the upper level of the food webs, they accumulate some of the highestcontaminants [11]. The accumulated contaminants and pollutants in theirbodies can be studied by scientists to identify potential threats in marineecosystems.However, marine mammals’ population has been declining sharply [48].Besides the loss of habitats due to man-made construction, over-fishing, andhuman activities, such as active sonars and vessels, also cause the marinemammal population to decrease [8]. The sound waves generated by militarysonar can severely injure and even kill marine mammals [39]. Moreover, theactive sonar frequency interferes with the echolocation used by cetaceans forlocalization and hunting. Thus, the protection of marine mammals is muchneeded.One direction to protect marine mammals is to achieve advanced de-tection and classification so vessels can make a detour, and the navy canreschedule their exercises to avoid harming the protected marine mammals.Machine learning has been introduced to detect and classify marine mam-mals and it can be used to tackle the problem in two ways. One approachis to analyze the photos of marine mammals using machine learning mod-els. For example, the Simple Linear Iterative Clustering technique was usedto segment the aerial images of dugongs [32] and then train these imageswith convolutional neural networks (CNNs), obtaining an F1 score of 0.406.1Besides aerial images, an attempt has been made to compare three machine-learning methods using Commerson’s dolphins’ photos [41], and the F1 scoreof the best model is 0.833. Another approach classifies marine mammals us-ing the vocalizations captured by passive acoustic monitors (PAM) since ma-rine mammals produce sounds of different frequencies and patterns. Withthe advantages of the complex networks and weights trained using largedatasets, various CNNs [1, 12, 30, 58] have been used to build marine mam-mals’ sound classifiers to produce accurate predictions.Indeed, machine learning has attracted interest as it can identify marinemammals’ sounds accurately without the presence of specialists. There isstill room for improvement. First, existing works mainly focus on applyingtransfer learning using different pre-trained models. These models are com-monly made up of complex and deep neural networks with the advantageof being accurate but at the expense of efficiency. In other words, theirtraining time and inference time can be shortened. Second, most existingmarine mammals’ sound classification work using deep learning is trainedusing private datasets, so the signal-noise ratio (SNR) and data size areunknown. The only known open-sourced dataset used was the WatkinsMarine Mammals Sound Database (WMMSD)1. The experiment that usedWMMSD [30] only tested the model on the best cuts of the database, whichare the audios with high SNR. The noisy audio cuts were not explored;hence the robustness of the models is unknown. Therefore, our goal is todevelop a computationally friendly and robust framework that recognizesmarine mammals’ acoustic signals efficiently.This thesis proposes a knowledge distillation (KD) marine mammals’sound classification framework that is much more efficient than some well-known CNN-based models and transformers. We introduce a Cross-Covarianceattended compact Feed-Forward Sequential Memory Network (CC-FSMN)that can efficiently learn the acoustic features. On the basis of the CC-FSMN, we build a teacher model that embeds an X-vector extractor anda DenseNet network to better learn the features and increase the accuracyof the system. We further replace the Dropout mechanism in the Cross-1https://cis.whoi.edu/science/B/whalesounds/index.cfm2Covariance Attention (XCA) layer with a DropKey mechanism to improvethe robustness of the model. The relatively large and complex teacher net-work effectively improves the CC-FSMN-based student model’s accuracythrough an effective knowledge transfer. Compared to a pure CC-FSMN,the student model has improved robustness, F1 score, and an identical in-ference time. To verify the generalizability and performance of the system,two datasets are included: WMMSD and the MobySound Database [34]. Wecompare the efficiency and accuracy of multiple popular models, includingAlexNet [30], VGG-16 [58], DenseNet [58], ResNets [1, 18], DeiT III [50],FNet [27], and Longformer [54].We can summarize our contributions as follows:− We propose a Cross-Covariance Attended Compact Feed-Forward Se-quential Memory Network (CC-FSMN) that is excellent in handlingacoustic signal classification. It is the base architecture for the teacherand student model.− We propose a teacher model that fuses features extracted from DensetNet-121, TDNN, and CC-FSMN. Through extensive empirical evaluationsand comparisons, the teacher achieves state-of-the-art performance.− We apply a temperature cooling strategy to the knowledge distillationapproach to mitigate the knowledge gap caused by model size andarchitecture differences between the teacher and the student model,increasing the effectiveness of the process.− Through knowledge distillation, the student model, which is approxi-mately 25 times smaller than the teacher model, is well-trained. Thestudent model has the best inference speed while retaining a very closeaccuracy to the teacher model. The student model’s highly compressedsize makes its deployment on various vessels applicable.The structure of the whole thesis is divided into five chapters. Chap-ter 1 provides an introduction to the thesis, while Chapter 2 details thebackground information, including the threats posed to marine mammals,3existing solutions to these threats, and an introduction of the comparisonmodels. Chapter 3 discusses the structure of the proposed framework in de-tail, including the feature extraction process, the teacher model architecture,the student model architecture, and the loss functions. Chapter 4 presentsthe specific experimental setup for the experiments, details of the datasets,and evaluation metrics used. Chapter 5 presents and discusses the resultsof the models and an ablation study of the proposed framework. Finally,Chapter 6 concludes the thesis and recommends potential future work basedon the results and conclusion from the thesis work.4Chapter 2Background2.1 Threats to Cetacean PopulationsIt is challenging to assess the precise number of whale populations. Theestimation of some whale species shown in Table 2.1 [2] suggests the over-all populations of the cetaceans have dropped sharply. Shockingly, certainspecies have their populations reduced by more than 80%. Adding to theseconcerns, at least 25% of the world’s cetaceans were recently confirmed asendangered and the situation is not optimistic as the status of many othersremains unknown.Table 2.1: The trend of whale population from 1890 to 2001Species 1890 2001 Relative ChangeBlue Whale 340,280 4,727 −99%Bowhead Whale 89,000 9,450 −89%Right Whale 84,100 9,239 −89%Fin Whale 762,400 109,600 −86%Humpback Whale 231,700 42,070 −82%Sei/Bryde’s Whale 392,300 181,490 −54%Gray Whale 24,600 15,936 −35%Minke Whale 637,000 506,900 −20%The decrease in cetacean populations can be traced back centuries. Inthe past, people hunted them for food, warmth, and luxuries such as perfume[2]. The advent of steam-allowed boats marked the beginning of the modernwhaling era. The faster and more efficient fishing vessels led to the expan-sion of species and numbers being caught. Other than commercial whaling,climate change is also a significant factor that has decreased the cetacean52.1. THREATS TO CETACEAN POPULATIONSpopulations. Temperature is a key factor that determines the cetaceans’distribution as the species evolved to live in areas of a certain temperaturerange. Some whale species exhibit a degree of adaptability to shifting cli-mates but most of them remain incapable of doing that. For instance, theendangered vaquitas’ habitat is limited to the warm waters located at theembayment of California. Such an enclosed landscape restricts the vaquitas’movement to find cooler water when climate changes [43]. There are otherindirect impacts of climate change, for example, alterations in availability,locality, and abundance of food. This phenomenon particularly imperilsspecies reliant on specialized feeding habitats. For instance, in the NorthAtlantic, warm water species population is rising due to climate warmingand the effects of the North Atlantic Oscillation [15]. This is detrimentalas the warm water species are not replacing the cold water species in sim-ilar abundance. It is also suggested that there may also be a correlationbetween climate-induced reductions in prey and numbers of starving harborporpoises stranding on the Scottish North Sea coast [22, 26, 31].The deployment of military sonar also poses a significant and pervasivethreat to cetacean populations. There were at least 16 whales of 3 speciesstranded in the northern Bahamas when the US Navy vessels passed bythat area [3]. The Bahamas incident in 2000 was not the only strandingcaused by the navy activities. There are more stranding of whales causedby the navies. Across the Pacific, 11 mass strandings on the central Pacificcoast of Honshu in Japan were reported and the strandings happened in thearea adjacent to a US Navy command base [7]. The Scientific Committee ofthe International Whaling Commission (IWC) suggests that “these whalesdied from acoustic or blast trauma that may have been caused by exposureto naval activities south of Taiwan” [10]. In 2005, another 37 whales werestranded in the Outer Bank, North Carolina after a US Navy vessel usedactive sonar for about 7 minutes [24]. Similarly, more than a hundred long-finned pilot whales were stranded in Australia after the Australian Navyused high-frequency sonar [38]. It was later concluded that the militarysonar deployed by the Navy caused these strandings.Mass strandings represent just one facet of the adverse effects inflicted62.2. RELATED WORKupon cetaceans by active sonar. Other abnormal conditions include, but arenot limited to changes in singing behavior [36, 42], keeping silent [52], andthe appearance of deep-water animals in the shallow water [20, 23]. Thecommon thing to them is the presence of navy and military sonar. It is alsoproven that besides the interference between the cetaceans’ sounds and theactive sonar, the high acoustic energy can directly and physically damagethe cetaceans’ auditory system and even result in fatal consequences.2.2 Related Work2.2.1 Acoustic Signal Processing and RepresentationA common approach for identifying the sounds of marine mammals in-volves studying the characteristics of the interested acoustic signals. Onemethod is to find the spectrogram correlation [33], calculating the similarityvalues between the correlation kernels of the templates’ spectrograms andthe input signals’ spectrograms. Another method compares the raw datawith templates of the species. An example of the approach is the sound-comparative method, which evaluates the template and the interested signalusing a preset threshold value [9].Another approach involves extracting useful information and featuresfrom the spectrograms. For example, a whale detection and classificationsystem that estimates the time variation of the fundamental frequency as apitch track [4]. Then the system extracts the attributes from the pitch trackfor classification using quadratic discriminant function analysis. Besidespreprocessing of acoustic signals, representation learning can significantlyaffect the performance of audio signal classification [53]. A new way to rep-resent spectrograms of marine mammals’ acoustic signals has been proposed[49]. The interpolated and stacked spectrograms are generated by running aShort-time Fourier Transform with various parameters to evaluate the fea-tures using ResNet-50 and VGG-19.72.2. RELATED WORK2.2.2 Machine Learning Based MethodsSeveral works [13, 16, 17, 35] have applied machine learning to tacklethe marine mammal classification problem. For example, a self-organizingfeature map was used to classify the vocalizations of Humpback whales [35].Some approaches involve comparing methods such as a regression tree, astandard fully connected network, and a restricted Boltzmann [13, 17]. In-stead of constructing a complex network, another way to increase the accu-racy of a classifier is to enhance the input data. For instance, spectrogramnoise removal through a median filter, average subtraction, and Gaussiansmoothing kernel to enhance the recorded odontocete whistles and trainedusing a linear classifier [16].2.2.3 Deep Learning Based MethodsWith the advancement of deep learning, transfer learning based on CNNsis applied to classify marine mammals based on acoustic signals. TheCNN’s capability of discriminating spectro-temporal information from spec-trograms makes it an ideal network for processing acoustic information. Ithas been widely used in acoustic classification tasks such as bio-acousticsclassification, environmental sound classification, and underwater sonar im-age classification [12]. The main body of a CNN is usually made up ofConvolution layers, Pooling layers, and fully connected layers. To illustratehow CNN studies the pattern of an image, see Figure ??. The image is dig-italized to be a matrix of numbers to the computer and each pixel’s value isdetermined by the magnitude of the color at that pixel. The convolutionallayers apply a set of filters that slide through the image matrix. At eachposition, the filter computes a weighted sum of the pixel values in its re-ceptive field as shown in Figure 2.1. This process allows the model to learnthe local patterns and features such as edges. It is common to have a Pool-ing layer after the convolutional layer to downsample the features reducingcomputational complexity. At the end of a series of convolutional layers andPooling layers, there will be a few fully connected layers that behave as theconventional neural network that classifies the learned features.82.2. RELATED WORKFigure 2.1: Sample convolution of a 7x7 pixel RGB image.92.2. RELATED WORKBesides its promising performance on visual representation classificationtasks, another reason that drives the research focus to shift to deep learningis that automatic detection and classification system based on deep learningoutperforms human analysis in efficiency. For instance, different fine-tunedand pre-trained CNN backbones are used to build accurate marine mam-mals’ sound classifiers [1, 30, 58]. However, these works mainly focus onlimited whale species and are relatively inefficient. A different and novelapproach turns the classification task into a regression problem. Instead ofa classifier, a regressor is built using YOLO to predict bounding boxes inthe spectrograms [12].Later, the appearance of self-attention mechanisms and transformerschanged deep learning significantly. The first transformer stacks attentionlayers and surpasses the best result of that time [51]. The standard trans-former shown in Figure 2.2 consists of two components, an encoder on theleft and a decoder on the right. The encoder consists of N identical blocksand each block is made up of an attention layer and a Fully Connected Feed-Forward Network with two linear transformations and ReLU activation:FFN(x) = ReLU(W1x + b1)W2 + b2. (2.1)The N Transformer encoders apply the same linear transformations to theinput sequence using different weight and bias parameters. The transformerarchitecture does not capture the position information solely and the intro-duction of positional encoding is required. The positional encoding vectorshave the same dimension as the input embeddings and are generated usingsine and cosine functions of different frequencies. The positional informa-tion is obtained by summing the positional vector and the input embeddings.The decoder has the same number of N as the encoder, and it is made up ofthree sublayers. Different from the encoder which is designed to attend toall features in the input, the decoder only attends to preceding words if oneis doing NLP tasks. The prediction for a word only depends on known out-puts that come before it. On the decoder side, this attention layer receivesthe queries from the previous decoder sublayer and the keys and values from102.2. RELATED WORKthe output of the encoder. Hence, the decoder can attend to all features inthe input sequence. The decoder block ends with a Fully Connected Net-work like the encoder. Furthermore, the three sublayers also have residualconnections around them and a normalization layer. In addition, positionalencodings are also injected into the input embeddings of the decoder in thesame manner as the encoder.Figure 2.2: The architecture of a standard transformer. Adapted from [51].112.2. RELATED WORKThe idea of transformers inspires researchers to develop more transformerarchitectures [21]. For instance, a newly proposed model, DeiT III [50], isa variant of ViT that includes a new data-augmentation procedure thatinvolves Gaussian Blur, solarization, and grayscale, and it obtains a com-petitive performance in image classification. Indeed, CNN-based and trans-former models produce some decent classifiers but they are computationallycostly. To speed up the full attention transformer, Longformer [5] specifiesthe full self-attention matrix making it a more efficient architecture for longsequence data. However, the transformers require large amounts of data andstrong computational power to be effective. FNet [27] is an attention-freetransformer architecture that replaces attention layers with a Fourier mix-ing sublayer with a feed-forward sublayer to increase the efficiency of thetransformer. However, the FNet transformer obtains improved efficiency atthe expense of accuracy.2.2.4 Knowledge DistillationBig models show outstanding accuracy but they are computationallyexpensive and hard to deploy on any kind of vessel. Knowledge distillation(KD) is a model compression technique to obtain an accurate small modelby transferring knowledge from big models [37]. KD has been widely used inface recognition tasks. For instance, ShrinkTeaNet [14] proposes an angulardistillation loss that seeks to minimize the angle between teacher and studentembedding vectors. KD has also been introduced to solve translation tasksand two sequence-level KD have been proven to be useful for translation [25].The performance of the student model may be affected by the gap betweenthe student and the teacher model. To effectively transfer knowledge to asmaller model, an intermediate teacher-assistant model is proposed to bridgethe gap [37]. Another way to improve the knowledge transferring processis to use a grouped KD loss which consists of three parts proposed to filterout knowledge that is unrelated to facial identities [57]. To the best of ourknowledge, KD has not been used in marine mammal recognition.122.3. COMPARISON MODELS2.3 Comparison ModelsThis section will provide an in-depth overview of the models selectedfor the comparative study. By examining the architectures, strengths, andweaknesses of these models, we can lay the foundation for designing an op-timized framework for the rapid and accurate classification of marine mam-mals’ sounds.2.3.1 AlexNetStarting with the commonly used CNNs, AlexNet is more or less thepioneer of CNN backbones. AlexNet is made up of five convolutional layersfollowed by three fully connected layers. The conventional activation func-tion for a neuron’s output f is f(x) = tanh(x) which is also the same asf(x) = (1+e−x)−1. The tanh function is a saturating non-linearity which ismuch slower than non-saturating non-linearity. The fully connected layers inAlexNet have an activation function of Rectified Linear Units (ReLU) thatmitigates the vanishing gradient problem and speeds up training. More-over, Local Response Normalization (LRN) is applied in the early layersto enhance the contrast between different features, improving the model’sgeneralizability. The response-normalized activity bix,y is defined by the fol-lowing expressionbix,y = aix,y/(k + αmin(N−1,i+n/2)∑j=max(0,i−n/2)(ajx,y)2)β. (2.2)The sum runs over n “adjacent” kernel maps at the same spatial position,and N is the total number of kernels in the layer. This sort of responsenormalization implements a form of lateral inhibition inspired by the typefound in real neurons, creating competition for big activities amongst neuronoutputs computed using different kernels. To be more precise, a poolinglayer can be thought of as consisting of a grid of pooling units spaced spixels apart, each summarizing a neighborhood of size z×z centered at thelocation of the pooling unit. The conventional local pooling has s = z132.3. COMPARISON MODELSwhereas, AlexNet adapts s < z to obtain an overlapping pooling.The overall structure of AlexNet is shown in Figure 2.3. The first con-volutional layer filters the 224×224×3 input image with 96 kernels of size11×11×3 with a stride of 4 pixels. The output of the first convolutional layeris response-normalized, pooled, and passed to the second convolutional layerwith filters of 256 kernels and size 5×5×48. The next three convolutionallayers are connected to one another without pooling or normalization. Thethird convolutional layer has 384 kernels of size 3×3× 256. The fourth con-volutional layer has 384 kernels of size 3×3×192, and the fifth convolutionallayer has 256 kernels of size 3×3×192. The fully-connected layers contain4096 neurons each. AlexNet achieved the best performance for The Ima-geNet Large Scale Visual Recognition Challenge (ILSVRC). In conclusion,AlexNet is a breakthrough architecture in the computer vision field and itdrives the development of advanced deep learning architectures and some ofthem will be described later.Figure 2.3: AlexNet architecture.2.3.2 VGG-16VGG-16 is characterized by its architecture which is made up of 13 convo-lutional layers and 3 fully connected layers. VGG-16 uses 3x3 convolutional142.3. COMPARISON MODELSlayers throughout the network and it aims to capture the complex featureswhile restricting the model size. The hierarchy convolutional layers learnsimple features at shallow layers and complex features at deep regions. AMax-Pooling is applied to every few convolutional layers to reduce the spa-tial dimensions so the computational complexity. The fully connected layersat the end of the network compute the class probabilities for the task. Sim-ilar to AlexNet, all the hidden layers contain the ReLU activation function;however, VGG does not contain any LRN. The existence of LRN in VGGwill only increase memory consumption and computation time.The straightforward design makes VGG-16 easy to implement and un-derstand. However, its depth and the use of the small filters make VGG-16computationally demanding and it can be slow in training and inferencespeed. Also, the VGG-16 architecture is large in size compared to othermodels so it may not be feasible to deploy in any kind of environment es-pecially when the computational resources are limited. To sum up, VGGnets have been influential and competitive for many computer vision tasksbut it has limitations that include computational complexity, employmentdifficulty, and overfitting.Figure 2.4: VGG architecture.152.3. COMPARISON MODELS2.3.3 ResNetResNet is the abbreviation for ‘Residual Network’ and it is known for itsstrength in dealing an extreme-depth networks. A significant deep networkusually leads to the gradient vanishing problem and ResNet tackles thisproblem by introducing residual blocks which is also known as the skipconnection. Instead of learning the target output directly, the residual blockslearn the differences between the target output and input. The differencesare added to the input to obtain the output, so the network learns theidentity mapping. To better explain the residual blocks, we let H(x) be anunderlying mapping to be fit by the layers and x be the inputs. Instead ofapproximating the H(x) through the stacked layers, layers approximate aresidual function F (x) := H(x)−x. Therefore, the original function becomesF (x) + x, improving the efficiency of the network. The network may findit difficult to approximate identity mappings through multiple non-linearlayers and the identity mapping may not be optimal but this reformulationstill helps the network to obtain better results. Every few stacked layerscarry residual learning and the building block is defined as:y = F(x, {Wi}) + x, (2.3)where F(x, {Wi}) represents the residual mapping and x is the input, andy is the output. For example, in Figure 2.5, σ denotes the ReLU func-tion and F + x is achieved through a shortcut connection and element-wiseaddition. The shortcut connections in ResNet add no burden to the com-putation. ResNet is designed to mitigate the gradient vanishing problemin such a way that the information is propagated from layer to layer with-out transformations. Furthermore, ResNet is inspired by VGG nets, andits convolutional layers are mostly made up of 3x3 filters and two designrules: the layers have the same number of filters as the output feature mapsize and the number of filters will be doubled if the feature map is halvedto make the time complexity consistent. Hence, ResNet has few filters andlower complexity compared to VGG nets.In addition, ResNet has a few variants that include ResNet-18 and162.3. COMPARISON MODELSFigure 2.5: The residual block.ResNet-50 which are used for the comparison experiments. ResNet-50 hasbeen proposed for marine mammal classification and then we decide to adoptResNet18 to study their differences in efficiency and accuracy.2.3.4 Data-efficient Image Transformers (DeiT)DeiT is a vision transformer that is pre-trained using Imagenet andachieves state-of-the-art performance on different downstream tasks such asfine-grained classification and CIFAR-10. DeiT is a convolution-free trans-former so the attention layers make up the major part of the network. Theattention mechanism is associated with the concept of key and value pairs.A query, q vector is matched with a set of key, k vectors using inner prod-ucts. The products will then be normalized and scaled to obtain the weightsof k. The output of the attention maps is formulated as following:A(K,Q, V ) = Softmax(QKT√dk)V, (2.4)where the Softmax function is applied to each row of the input and√dnormalizes the attention maps. DeiT uses Multi-head self-attention lay-ers (MSA) in the transformer which is defined by considering h attention‘heads’. Each of the heads provides a sequence of size N × d and the se-quences are then arranged into N × dh sequence which is further projected172.3. COMPARISON MODELSFigure 2.6: The ResNet-50 architecture. Adapted from [18].182.3. COMPARISON MODELSby a linear layer into N ×D. There are a few advantages of MSA over theself-attention mechanism. The multiple heads allow the model to capturethe information of the input from different aspects simultaneously. Eachhead learns different patterns and relationships of the features which helpsthe model learn the local and global dependencies effectively. MSA is alsomore robust to variability as different heads can interpret the various rep-resentations of identical information. DeiT’s transformer blocks are builtupon the ViT model with a Feed-Forward Network (FNN) added on top ofthe MSA layer. The FFN consists of two linear layers with a Gaussian ErrorLinear Unit (GeLU) activation function in between. The first layer expandsthe feature dimension from D to 4D and the second layer reverses it. BothMSA and GeLU work as residual operators with the aid of skip connectionsand layer normalization. The class token in Figure 2.7, is a trainable vectorthat is concatenated to the patch tokens at the beginning. The concate-nated vectors are then passed into the transformer layers and projected to alinear layer for prediction. This class token is a concept that originated fromNatural Language Processing (NLP) and the transformer will thus processbatches of (N + 1) tokens of dimension D and only the class vector willbe used for predicting the output. Such architecture restricts the attentionlayers to the spread of information between the patch tokens and the classtokens.DeiT comes with a KD strategy that is specific to the transformer.This strategy involves twisting the teacher’s part, instead of measuring theKullback-Leibler divergence between the teacher’s logits and the student’slogits, the KD loss replaces the teacher’s logits with the teacher’s hard la-bels:LDeiT = 0.5LCE(σ(Zs, y)) + 0.5LCE(σ(Zs), yt), (2.5)where yt = argmaxcZt(c) is the hard label predicted by the teacher model.This change is claimed to be better than the conventional one as it isparameter-free and conceptually simpler.DeiT also introduces a new token, the distillation token. The distillationtoken works in a similar way to the class token as it interacts with patches192.3. COMPARISON MODELSand class tokens through self-attention. The distillation token helps theDeiT student model learn from the teacher model and is complementary tothe class embedding. The class tokens and distillation tokens will convergeto similar vectors but not identical.DeiT III is a proposal for a training recipe for the transformers that wasintroduced for ResNet-50 initially. The recipes include stochastic depth,Layerscale, 3-augmentation, and simple resized crop (SRC). Stochastic depthis a regularization technique for deep neural networks especially for CNNand ResNet. In a training with stochastic depth, a random subset of lay-ers will be skipped for each input. It is similar to the Dropout mechanismwhich randomly drops neurons. The random skipping of layers encouragesthe network to learn to perform stably with different parts of the architec-ture disabled, making the network more robust. LayerScale is introducedto boost the convergence of the transformer network. For each layer, ascaling factor is introduced and the layer learns it to encourage a balancedflow of information and gradients. The use of 3-augmentation is inspired byself-supervised learning and it includes transformations: Grayscale, solar-ization, and Gaussian Blur. In addition to the 3 augmentations, DeiT IIIalso includes color-jitter and horizontal flip. SRC is similar to the croppingtechnique proposed in AlexNet as it resizes the images so the smallest sidematches the training resolution. Then apply a reflect padding on all sidesand a square crop along the x-axis of of the image. SRC can reduce the dis-crepancy in apparent size and aspect ratio and it has a higher chance thatthe actual label is the same as the crop as compared to random cropping. Itis also worth mentioning that DeiT variants DeiT-S and DeiT-Ti are smallerin size as compared to ResNet-50 and ResNet-18.To sum up, DeiT III has proven to be the most powerful transformerup-to-date for the vision-related task that includes image classification andit is only pre-trained on ImageNet-1k which is relatively small compared toother transformers that are pre-trained on multiple datasets. Being the bestmodel in the computer vision field, we include it to verify its performancein identifying marine mammals’ sound spectrograms with small data sizes.202.3. COMPARISON MODELSFigure 2.7: Deit structure.212.3. COMPARISON MODELS2.3.5 LongformerLongformer is a type of transformer that is created to solve the prob-lem when the length of the sequences gets too long and the self-attentionmechanism will require a significant amount of computation power whichmay not be feasible. The self-attention layers in Longformer scale linearlywith the sequence length, so Longformer can handle sequences with variouslengths. To further elaborate on how Longformer works, the Longformersparsifies the full self-attention maps following an ‘attention pattern’ speci-fying pairs of input locations attending to one another. As shown in Figure2.8. Longformer uses dilated sliding windows to determine the attention pat-terns which involves the use of a fixed-size window attention that surroundseach token. Stacking multiple layers of windowed attention to obtain a largereceptive field, the top layers access all input locations and are able to buildrepresentations that incorporate information about the entire input. To fur-ther increase the receptive field without jeopardizing the model’s efficiency,the windows are dilated. Dilation creates gaps between the windows.Figure 2.8: Architecture of a Longformer.222.3. COMPARISON MODELSFigure 2.9: Demonstration of different attention maps.Windowed and dilated attention are not good enough to handle task-specific representations. Therefore, global attention is added to a few pre-selected input locations. The operation of attention is symmetric as a tokenis globally attended to all tokens across the sequence, and all tokens inthe sequence attend to it. Figure 2.9 shows the difference of the attentionmaps. It is convenient to add inductive bias to the attention which is simplerthan existing task-specific approaches that use complex networks to mergeinformation. Recall that the equation of self-attention is 2.4, Longformeruses two sets of projections to compute attention scores of sliding windowattention and global attention. The additional projections allow the modelto be flexible to different types of attention.232.3. COMPARISON MODELS2.3.6 FNetFNet is a small fast transformer encoder architecture that outperformsother transformers in speed. FNet replaces the attention layers in conven-tional transformers with Discrete Fourier Transform (DFT). Given a sequen-tial input {xn}, the DFT layer generates a new representation Xk as a sumof input tokens in the following way:Xk =∑N−zn=0 xne−2πink/N , 0≤k≤N − 1. (2.6)This Fourier sublayer applies a 2D DFT to the embedding input as onealong the sequence dimension, Fseq and the other one alone the hiddendimension,Fh:y = R(Fseq(Fh(x))). (2.7)The simplified representations of the tokens make FNet an efficient trans-former.Figure 2.10: Structure of FNet.24Chapter 3Methodology3.1 Overall Structure of The FrameworkThe overall flow of the structure is shown in Fig 3.2. The frameworkconsists of two networks, the teacher network, and the student network.The teacher network is a deep and complex neural network. Whereas, thestudent network is much smaller. The teacher model will first be trained toobtain state-of-the-art accuracy through the model soup technique. Thenthe pre-trained teacher model will transfer the knowledge to the studentmodel to help it learn the features.Figure 3.1: The overall flow of XCFSMN.253.2. FEATURE EXTRACTION AND PRE-PROCESSING3.2 Feature Extraction and Pre-processing3.2.1 FBank ExtractionIn this thesis, Log Mel-Filter Bank (Fbanks) is the feature used formodel training. Compared to Mel-frequency cepstral coefficients (MFCCS),which is a feature commonly and conventionally used for automatic speechrecognition, Fbanks are particularly well-suited for deep learning models asthey preserve more spectral information. The feature extraction processbegins with pre-emphasis applied to the audio signal. Pre-emphasis involvesthe use of a high-pass filter to balance the spectrum and reduce noise. Next,the continuous acoustic signal is segmented into discrete frames using aHanning Window. The chosen sample rate for this process is 8kHz, witha window size of 512 samples and a window hop of 256 samples for eachsuccessive frame. In the third step of feature extraction, a Fast FourierTransform (FFT) is applied to transform the segmented signal from thetime domain to the frequency domain. The outcome of this transformationis an amplitude spectrum, often referred to as the Mel-Filter Bank, whichcaptures essential frequency components of the audio signal. Finally, thelast step involves the application of a logarithmic function to the Mel-FilterBank, resulting in the Log Mel-Filter Bank or Fbanks. This transformationenhances the discriminative power of the feature while facilitating its use indeep-learning models for tasks such as marine mammal classification.3.2.2 Cepstral Mean and Variance Normalization (CMVN)The application of CMVN to Fbank features offers several advantages.This normalization process serves to standardize the features, resulting in aGaussian distribution with values ranging from −1 to 1. This standardiza-tion has several benefits for the training and performance of the model.Firstly, it minimizes the effect of the amplitude of different frequencycomponents. This is essential because large variations in feature magni-tudes can lead to numerical instability during training, resulting in somefeatures dominating the learning process while others become negligible.263.2. FEATURE EXTRACTION AND PRE-PROCESSINGFigure 3.2: The flow of FBanks extraction.CMVN mitigates this issue by scaling all features to a comparable range.Secondly, normalization accelerates the model’s convergence speed. Wheninput features are numerically close to each other, the optimization processtends to converge more rapidly. This can reduce the time and computationalresources. Lastly, CMVN enhances the model’s robustness to noise in theinput feature. Normalizing the features reduces variations in feature valuescaused by noise, making the model less sensitive to minor perturbations.The CMVN process is achieved through the following equation,Xnormed =X − X¯V ar(X), (3.1)where Xnormed is calculated by each Fbank coefficient minus its mean anddivided by its variance throughout the frames.273.3. TEACHER MODEL3.3 Teacher ModelThe teacher model consists of three integral components: a pre-trainedDenseNet network, a Time Delay Neural Network (TDNN), and a modifiedstudent model unit as shown in Figure 3.3. Each of these components playsa distinct role in the model’s overall functionality.− Pre-trained DenseNet-121: DenseNet-121 is responsible for capturingand encoding the spectral information from the Fbank features, allow-ing the teacher model to learn intricate patterns and representationsfrom the input data efficiently.− TDNN: The TDNN is tasked with extracting the unique acoustic char-acteristics specific to different animals. It is designed to analyze thetemporal aspects of the audio data, capturing the nuances and patternsthat distinguish one species from another. The extracted patterns areknown as the X-vectors.− Modified Student Model Unit: This unit focuses on learning inter-frame context information. It complements the DenseNet-121 andTDNN by enhancing the model’s ability to understand the relation-ships between consecutive frames of audio data.Collectively, this architectural design achieves state-of-the-art accuracyin marine mammal classification tasks. The teacher model transfers theknowledge to the student model, contributing to improved performance andfacilitating efficient learning.3.3.1 DenseNet-121Incorporating a pre-trained CNN into the teacher model is a move aimedat enhancing feature extraction from the Fbank representations. CNNs excelat capturing spectro-temporal patterns, making them a valuable addition tothe model’s architecture. For instance, the drastic visual differences betweenthe FBanks of Minke Whales and Bowhead Whales can be effectively differ-entiated by CNN as shown in Figure 3.4. The selection of DenseNet-121 as283.3. TEACHER MODELFigure 3.3: The structure of the teacher model.Figure 3.4: FBank spectrogram of bowhead whale and minke whale.the CNN backbone for our model is the result of a thorough experimenta-tion process, where various CNN architectures were tested. There are a few293.3. TEACHER MODELcharacteristics and advantages that will be elaborated. In the Dense Blocks,each layer obtains additional information from previous layers and passeson its own feature maps to the next layers. The lth layer has l inputs thatstore feature maps from all previous convolutional layers. Subsequently, theoutput of lth will be passed to all layers after it:xl = Hl[x0, X1, . . ., xl−1]. (3.2)The set [x0, X1, . . ., xl−1] denotes the concatenated feature maps from therespective layer. There will be L(L+1)2 connections in a network of L layers.The dense connectivity pattern requires fewer parameters than the tradi-tional CNNs.ResNet presents such information through additive identity transforma-tion but recent variations of ResNet suggest that many layers are redundantand can be dropped; whereas, DenseNet has narrow layers that only adda small set of feature maps to the network’s knowledge. Hl refers to acomposite function of three operations: batch normalization, ReLU, and aconvolution layer with kernel size 3x3. However, if the size of the featuremaps changes, the operation Hl will fail, and down-sampling the featuresis an essential process in CNNs. To make down-sampling applicable, thenetwork is divided into multiple dense blocks that are connected as shownin Figure 3.5. The layers in between the Dense Blocks are the transition lay-ers which are made up of a batch normalization layer, a 1x1 convolutionallayer, and a 2x2 average pooling layer. Each layer takes many more inputsthan its output, hence, the introduction of a 1x1 convolution layer beforeeach 3x3 convolution layer can reduce the size of feature maps, increasingcomputational efficiency. To further restrict the model’s size, a compressionfactor can be implemented at the transition layers. Assuming a dense blockcontains m feature maps, the transition layer generates θm output where0 < θ ≤ 1 is the compression factor. Hence, when θ = 1, the feature mapsremain the same.To sum up, the densely connected neurons in Dense blocks receive fullfeature maps from the previous layer, reducing the gradient vanishing prob-303.3. TEACHER MODELlem. Such feature reuse nature also optimizes the gradient descent process.Further, the transition blocks between the dense blocks reduce the dimen-sions through convolution and pooling. This design of Dense blocks andtransition blocks effectively compresses the model size and fosters efficientfeature usage.Figure 3.5: The structure of a DenseNet.3.3.2 TDNNThe TDNN serves the role of an embedded X-vector extractor. X-vector[44] is a type of feature embedding that is commonly used in speaker recog-nition tasks such as speaker verification, speaker identification, and speakerdiarization. The X-vector is a fixed-length feature representation of variable-length speech segments, such as utterances, that can capture short-termand long-term variations in speaker characteristics. By doing so, X-vectorsreduce intra-speaker variability and expand inter-speaker variability. Imple-menting X-vectors can help the model differentiate cetaceans that producesounds of similar frequencies.The X-vector embedding model typically consists of five TDNN layers.The TDNN layers extract relevant features from the speech signal. Thesefeature vectors are then subjected to a statistics pooling operation to create aconsolidated and fixed-length X-vector representation. The X-vector is thenused as input to a classifier, such as a support vector machine or neuralnetwork, to perform the speaker recognition task.Conventionally, the X-vector extractor is trained isolated from the main313.4. STUDENT MODEL (CC-FSMN)classifier, we embed the X-vector model in the main model. This integrationnot only enhances efficiency but also reduces the overall model size. Theembedded X-vector is concatenated to the output tensors from the CC-FSMN. This design choice enhances the model’s capacity to differentiateamong cetaceans producing sounds with similar frequencies.Figure 3.6: The structure of the teacher model.3.3.3 Modified Student Model UnitThe Modified Student Model Unit has an identical architecture to thestudent model with a different number of neurons. The cFSMN [55] layerprocesses the contextual information that links up every frame in data and itis commonly used for automatic speech recognition. The XCA layers furtherstrengthen the relationship between the frames. The justification for thisstep is that whales sing songs to communicate and the songs are like theirlanguages. The details will be explained in the next section.323.4. STUDENT MODEL (CC-FSMN)Figure 3.7: The structure of the student model.3.4 Student Model (CC-FSMN)3.4.1 Compact Feed-Forward Sequential Memory NetworkConventionally, long-term dependent data work ideally with RecurrentNeural Networks (RNNs) such as a Long Short Term Memory Network(LSTM). The nature of the network’s backpropagation through time [40]increases the RNNs’ computational complexity. Compared to RecurrentNeural Networks (RNNs), FNN has a simpler structure and learns fasterand easier. Therefore, FNN with the ability to carry context information ispreferred.FSMN [56] is a new type of Feed-Forward Network (FNN) proposed in2015. An illustration of an FSMN layer and the improved cFSMN layeris shown in Fig 3.8. FSMN is an FNN with memory blocks that encodethe context information, allowing the transmission of long-term dependencyin sequential data without requiring recurrent feedback. The calculation ofmemory block is defined as follows:h˜ℓt =N1∑i=0aℓi ⊙ hℓt−i +N2∑j=0cℓj ⊙ hℓt+j , (3.3)where the a term encodes the context information up to previous N1 frames,and the c term encodes context information for future frames and ⊙ denotesmatrix multiplication. The output h˜ℓt in equation (3.4) can be treated as333.4. STUDENT MODEL (CC-FSMN)Figure 3.8: The structures of FSMN (left) and cFSMN (right) networks.a representation of the context information of the frames at time t. As aresult, the context information is fed into the next layer using the outputfrom the previous layer, as shown in the equation below:hℓt+1 = F(W ℓ hℓt + W˜ ℓ h˜ℓt + bℓ ), (3.4)where W and b represent the standard weight matrix and bias vector, andW˜ denotes the weight matrix between the memory block and the next layer.The cFSMN is a variant of FSMN architecture; it encodes the contextinformation similarly to a standard FSMN. However, instead of using thefull output hℓt from the previous layer, cFSMN projects the output to alinear layer. The memory block of cFSMN can be formulated asP˜ ℓt = PℓtN1∑i=0a˜ℓt ⊙ P˜ℓt−i +N2∑j=0c˜ℓt ⊙ P˜ℓt+j . (3.5)The projected output P˜ ℓt in the memory block is then used for context343.4. STUDENT MODEL (CC-FSMN)information encoding. The output of the memory block is demonstrated by:hℓ+1t = F(U ℓP˜ ℓt + bℓ+1). (3.6)To improve the cFSMN layer, we add an InstanceNorm layer after eachcFSMN layer. The normalization process resolves the covariant shift thatmay occur in the network, which helps to speed up the training and stabilizethe gradient descent process.3.4.2 Cross-Covariance Attention (XCA)Figure 3.9: The illustration of estimating attention in XCA.In this framework, XCA is applied to the network to increase the robust-ness of the network and avoid the use of expensive computation of quadraticattention map. XCA is an attention mechanism proposed for an imagertransformer that allows an image classifier to handle images of different res-olutions. We believe XCA can be beneficial in the robustness of our classifieragainst noises.Instead of a full pairwise interaction between the tokens by self-attentionas shown in equation (2.4) and the following:QKT = XWqWTk XT , (3.7)XCA derives the attention map using a Cross-Covariance matrix from the353.4. STUDENT MODEL (CC-FSMN)projections of tokens usingAxc(K,Q) = Softmax(KˆT Qˆτ), (3.8)KTQ = W Tk XTXWq. (3.9)The attention map is calculated by dividing the features into h groups whichare similar to the idea of heads in multi-head attention. Then the calculationof the attention is restricted within the heads. XCA has two advantages:compared to self-attention, the complexity of the attention calculation isreduced by a factor of h, the number of heads. Moreover, this block-diagonalversion is easier to optimize, leading to improved results generally. Theaforementioned advantages make XCA a better attention mechanism thanself-attention for our objectives.Conventionally, the XCA layer contains a dropout [45] operation thatrandomly drops out neurons in a layer to prevent the network from puttingmuch weight on certain features. To avoid overfitting and improve the sys-tem’s robustness, we replace the dropout layer in the XCA with a DropKey[28] layer. Instead of blocking out neurons, DropKey drops the input Keyunits in the attention layer. The dropping of Keys before calculating theattention matrix can penalize weight peaks so regularize the weights. Hence,the DropKey mechanism has improved attention weight regularization, se-cures the patterns, and can avoid the local-bias problem. Practically, Drop-Key is done by randomly generating a mask matrix D, and each element inthe matrix D has a chance of d turning negative infinity. The output of theattention with a DropKey mechanism is formulated as:Axc (K,Q) = Softmax(dj +KˆT Qˆτ), (3.10)and dj ∼ Bernoulli(1− dr).To further decrease the size of the student model, we prune about 30%of the neurons from the cFSMN layer and the MLP layer through the global363.5. LOSS FUNCTIONunstructured pruning (GUP) method. Instead of pruning an entire layer,GUP prunes neurons based on their importance scores, providing bettergranularity. GUP does not improve inference speed but it can compress amodel well, making the model more adaptable to various capacity-restraineddevices.3.5 Loss FunctionThe framework involves two loss functions, a Label Smooth Cross-EntropyLoss, and a Cooling Knowledge Distillation Loss. The Cross-Entropy Lossfunction is commonly used for classification tasks. The model predicts theprobability for each label k. p(k|x) = ezk∑ki=1 ezi, where zi is the unnor-malized log-probabilities. The Cross-Entropy loss function is then definedas CE(i) = −∑Kk=1 log(p(k))q(k), and the q(k) is the ground truth la-bel distribution. The experiments adopt Label smoothing [47], which is atechnique to regularize the model making it more robust by reducing themodel’s confidence. Label smoothing is achieved through the introductionof a smoothing term ϵ to the Cross-Entropy function turning it into a func-tion (1− ϵ) ∗CE(i)+ ϵK . Hence, instead of predicting 1 for the correct classand 0 for others, the model predicts 1− ϵ. The student model’s evaluationpartially depends on the Label Smooth Cross-Entropy Loss.The loss used for knowledge distillation is called Cooling KnowledgeDistillation Loss. It is a strategy we applied to mitigate the knowledgegap between the teacher model and the student model. The conventionalknowledge distillation loss is formulated as:LKD = −T 2i∑cσi(ZtT)σi(ZssT). (3.11)The LKD is the cross-entropy between the teacher and student predictiveprobabilities σ(Zt) and σ(Zs). The T term denotes the temperature hyper-parameter which controls the discrepancy between two distributions. In-creasing the T will increase the information provided for the student model373.5. LOSS FUNCTIONby the teacher model. It is wrongly assumed that a large pre-trained teachermodel can always pair up with a small student model effectively [37]. Ourproposed student model is approximately 18 times smaller than the teachermodel. Hence, we believe the conventional KD loss may not be the bestchoice for this task. Moreover, most KD studies involve two networks ofsimilar architecture but different layers and neurons. However, the pro-posed teacher model consists of a CNN backbone and a TDNN based on thestudent model. Hence, we propose to run a varying temperature strategythat reduces the knowledge gap caused by model size and architecture. TheT is replaced with Tc = Tco(1 − α)n where Tco is the initial temperature,α is the decreasing factor and n is the number of decreases. Starting atTco = 20, the temperature Tc of the KD loss will decrease exponentially asthe training process progresses at a rate of α = 0.1. The criterion for trig-gering the decrease is when the early stopping counter reaches patience of 3.The rationale is explained as follows: a high temperature at the beginningallows the teacher model to transfer a large amount of knowledge to thestudent model. As time passes, the temperature decreases so the studentmodel should learn the features more independently because of the differ-ence between its and the teacher’s architecture. Therefore, the KD processbecomes more effective.38Chapter 4Experimental Setup4.1 DatasetOur study draws upon two distinct datasets to assess the robustness andperformance of the systems. The first dataset, WMMSD contains approx-imately 2,000 audio recordings spanning over seven decades, encompass-ing the vocalizations of more than 60 marine mammal species. WMMSDpresents challenges due to variations in recording equipment and setups overtime. These fluctuations in data collection pose potential issues but also of-fer valuable insights into the system’s adaptability.The second dataset employed is MobySound. The creation of MobySoundwas motivated by the need for automatic recognition tasks. MobySoundholds a key advantage over WMMSD in terms of data consistency. Record-ings within each species-specific sound set were collected using a constantconfiguration, ensuring the feature’s uniformity. Whereas, the variations inthe sound collection configuration in the WMMSD allow us to test our sys-tem’s robustness and push the boundary that one classifier can achieve.The two datasets share five common classes: Finback Whales, HumpbackWhales, Blue Whales, Minke Whales, and Bowhead Whales. The audiolength of the 5 classes is shown in Table 4.1. The sizes of the datasets showa significant difference. More, the classes are highly unbalanced which cangenerally lead to a biased model and poor generalization.4.2 Experiment DetailsThe neural networks were implemented using Pytorch and trained us-ing an RTX A6000 GPU. The choice of an input shape for the features as394.2. EXPERIMENT DETAILSTable 4.1: Audio length of the five classes in hours.Class WMMSD MobySoundFinback Whale 3.90 21.0Humpback Whale 1.98 2.12Blue Whale 0.12 10.20Minke Whale 0.05 3.70Bowhead Whale 0.39 1.10256x256 was driven by the need to accommodate the input requirementsof all the models used in our experiments. Regarding the audio data, asample rate of 2kHz is sufficient to cover the frequencies of the 5 classesmentioned above. However, it is important to note that the soundtracksfrom the WMMSD dataset had relatively short durations, which made itchallenging to create a frame number of 256 using the 2kHz sample rate.Therefore, we chose a sample rate of 8kHz for FBanks extraction.To verify the efficiency and performance of XCFSMN, we made compar-ison experiments involving models as follows: CC-FSMN, AlexNet used in[30], ResNet50 used in [1], VGG-16, DenseNet used in [58], DeiT III [50],Longformer [54], and FNet [27]. In addition to ResNet50, we also run exper-iments on ResNet18, so every network in its smallest version will be testedfor a comprehensive efficiency comparison.The experiments conducted in this study encompass four distinct sets ofdata:− WMMSD 12-Class Classification Task: In this task, the WMMSDdataset was employed for a 12-class classification problem. The datasetwas structured to include the top 12 classes, ordered by sample sizein descending order. This arrangement was chosen as classes beyondthe top 12 had fewer than 100 samples, which was insufficient formeaningful training. The primary objective of this experiment was toassess the models’ capacity to handle highly imbalanced multi-classdata.− 5-Class Classification Tasks: A 5-class classification task for WMMSD404.3. EVALUATION METRICSand MobySound each respectively. These two experiments were per-formed to verify the models’ ability to train data from an independentsingle source.− Combined Dataset Classification Task: The data of 5 common classesfrom WMMSD and MobySound were combined for training and test-ing. This experiment aimed to assess the models’ robustness when con-fronted with the challenges of merging datasets with varying recordingtiming, target classes, and configurations. The task tests the models’ability to perform well in a more complex and diverse dataset setting.In all cases, the datasets were randomly split into training and test sets witha 7 to 3 ratio. It is worth noting that for the 5-class classification tasks, thedataset splitting occurred at the file level before the feature extraction step,maximizing the difference between the training set and test set. To avoidoverfitting, we applied a cross-validation of 5 splits. We split the trainingset into a training set and a validation set with five different random states.The results were then averaged to obtain the final results. All the modelswere fine-tuned for the best comparison outcome.4.3 Evaluation MetricsTo evaluate the efficiency of the models, training time and inferencetime were compared. Training time is the time taken for the model tofinish 300 epochs or it triggers an early stop. Inference timing refers to thetime taken for the model to predict the classes of the fixed-size test set. Ashorter training time or inference time indicates improved efficiency. Also,inference time is more important than training time for the marine mammalclassification tasks.The Macro F1 score is typically used to evaluate the performance of themodels as it takes into account both precision and recall. Instead of usingaccuracy, the F1 score can provide a more accurate measurement of theperformance of the models given the datasets are highly unbalanced. F1414.3. EVALUATION METRICSscore is calculated throughF1 =2× Precision×RecallPrecision+Recall. (4.1)Precision in 4.1 is defined as the percentage of the correct labels out of allthe predicted positives. Whereas, a model with high recall indicates thatthe model can find all the positive cases with some negative cases identifiedas positive. Since the objective is to detect and protect all kinds of marinemammals, a high recall is important.42Chapter 5Results Comparison5.1 Experiment ResultsTable 5.1: Comparative results of XCFSMN and the baselines on efficiency.‘TT’ denotes the training time in minutes and ‘IT’ denotes the inferencetime in seconds.Model # Params (M)12-class WMMSD 5-class WMMSD 5-class MobySound 5-class MergedTT IT TT IT TT IT TT ITAlexNet [30] 57.0 5 0.4 2 0.1 17 0.2 2 0.4ResNet18 [18] 11 5.5 0.9 15 0.2 5 0.6 5 0.8ResNet50 [1, 18] 23.5 13 2.7 15 0.6 7 1.6 6.3 2.2VGG-16 [58] 134.0 33 5.9 10 1.0 15 2.7 17 3.7DenseNet [58] 7.0 27 3.5 3 0.7 18 2.0 31 2.6DeiT III [50] 21.6 32 16.3 6 4.7 52 11.5 23 2.6Longformer [54] 16.9 47 8.4 14 11.0 12 4.6 19 6.6FNet [27] 3.1 4.5 0.7 1.3 1.2 5 3.2 0.33 0.5CC-FSMN [29] 4.5 5 0.32 0.7 0.1 3 0.2 2 0.3Teacher (ours) 81.3 12 10.8 3 1.2 22 10.3 5 4Student (ours) 3.15 5.8 0.32 1 0.1 6.5 0.2 12 0.3The efficiency results are shown in Table 5.1 and the performance resultsare shown in Table 5.2, 5.3, 5.4, 5.5. Out of the 11 tested models, FNet is thesmallest in model size followed by the proposed student model which is onlyapproximately 1.6% larger. CC-FSMN [29] and the FNet [27] show excellenttraining speed in all the experiments. The teacher model takes much longerto train which is expected. The student model is a little slower in trainingspeed. CC-FSMN and the student model have identical inference speedsand are dominant over other models in all experiments. This indicates thatour proposed framework and CC-FSMN achieve state-of-the-art efficiency.The teacher model shows absolute dominant results. The teacher modelhas the best precision, recall, and F1 Score for all the experiments. Throughthe temperature cooling knowledge distillation process, the student model435.1. EXPERIMENT RESULTSTable 5.2: Comparative results of 12-class WMMSD.Model Precision Recall F1AlexNet [30] 0.594 0.735 0.612ResNet18 [18] 0.782 0.846 0.806ResNet50 [1, 18] 0.787 0.850 0.809VGG-16 [58] 0.762 0.842 0.789DenseNet [58] 0.705 0.810 0.735DeiT III [50] 0.569 0.660 0.586Longformer [54] 0.237 0.274 0.215FNet [27] 0.120 0.225 0.120CC-FSMN (ours) 0.785 0.828 0.787Teacher (ours) 0.855 0.858 0.853Student (ours) 0.816 0.837 0.820Table 5.3: Comparative results of 5-class WMMSD.Model Precision Recall F1AlexNet [30] 0.636 0.607 0.607ResNet18 [18] 0.791 0.616 0.646ResNet50 [1, 18] 0.791 0.616 0.646VGG-16 [58] 0.891 0.823 0.783DenseNet [58] 0.842 0.819 0.728DeiT III [50] 0.781 0.704 0.660Longformer [54] 0.472 0.247 0.275FNet [27] 0.318 0.430 0.228CC-FSMN (ours) 0.791 0.616 0.646Teacher (ours) 0.946 0.828 0.827Student (ours) 0.900 0.824 0.791has significant improvements from the CC-FSMN. There is a minimum im-provement of 0.064 in the absolute F1 Score and a maximum of 0.145. De-spite the student model not obtaining as high an F1 Score as the teachermodel, it still outperforms all other models given it has almost the smallestsize.The CNNs have similar performance except for AlexNet which showssignificant drops in 12-class WMMSD and the merged dataset. It is observed445.1. EXPERIMENT RESULTSTable 5.4: Comparative results of 5-class Moby.Model Precision Recall F1AlexNet [30] 0.865 0.884 0.863ResNet18 [18] 0.950 0.873 0.880ResNet50 [1, 18] 0.928 0.845 0.846VGG-16 [58] 0.860 0.833 0.827DenseNet [58] 0.957 0.897 0.911DeiT III [50] 0.892 0.860 0.896Longformer [54] 0.657 0.705 0.645FNet [27] 0.275 0.561 0.283CC-FSMN (ours) 0.970 0.898 0.922Teacher (ours) 0.981 0.991 0.986Student (ours) 0.970 0.977 0.973Table 5.5: Comparative results of 5-class merged.Model Precision Recall F1AlexNet [30] 0.867 0.800 0.782ResNet18 [18] 0.934 0.809 0.840ResNet50 [1, 18] 0.904 0.828 0.842VGG-16 [58] 0.784 0.801 0.790DenseNet [58] 0.916 0.830 0.869DeiT III [50] 0.928 0.740 0.787Longformer [54] 0.668 0.594 0.627FNet [27] 0.261 0.296 0.222CC-FSMN (ours) 0.893 0.817 0.828Teacher (ours) 0.976 0.890 0.925Student (ours) 0.944 0.890 0.910that all the transformer models perform poorly. The poor performance canbe attributed to the small dataset size as the vision transformers requirelarge datasets to demonstrate their strength for image classification tasks[46]. DeiT can keep up in the 5-class experiments but fails for the 12-class because the 12-class WMMSD has a really limited sample size for theminority class. Longformer performs poorly since it was designed for long-sequence languages. More, the replacement of the attention mechanism with455.2. ABLATION STUDYthe Fourier Mixing Layer in FNet restricts the model from capturing long-sequence context information reducing the model’s ability to classify thesignal.To better illustrate the ranking of the models. We performed The Tech-nique for Order of Preference by Similarity to Ideal Solution (TOPSIS). Theweights of precision, recall, and training time are assigned with a value of0.1, and a value of 0.35 is assigned to the F1 Score and inference time sinceidentifying the animals quickly and accurately is prioritized. The averageperformance scores indicate that the student model is the best choice fol-lowed by CC-FSMN and then ResNet18.In addition, we compare the results of the aforementioned photo ap-proaches [32, 41]. The mean F1 score achieved by the model trained usingaerial images in [32] is 0.4059 and classification using Commerson’s dolphins’photos in [41] is 0.883. XCFSMN shows a better F1 score and also it is capa-ble of identifying multiple species. Therefore, marine mammal classificationbased on acoustic signals with XCFSMN is better than the same classifica-tion problem based on image processing with deep learning.Looking at the confusion matrix of a 5-class experiment run with thestudent model in Figure 5.1, the model has difficulty distinguishing thespectrograms of the Finback Whale and the Blue Whale. Perhaps this can beexplained by Figure 5.2, both the Fin Whale and Blue Whale spectrogramscontain a large blank space caused by a sampling rate that was too high.The large blank space caused the meaningful part to be small, making itdifficult for the model to find the differences between them.5.2 Ablation StudyWe conducted an ablation study for the student model shown in Table 5.6by comparing efficiency and performance using the 5-class merged dataset.The first row shows the results for a CC-FSMN [29] and behaves as thebaseline. The addition of Instance Normalization to the output of cFSMNincreases the model’s precision and is able to maintain the recall. ReplacingDropout with DropKey further increases the precision but reduces the recall.465.2. ABLATION STUDYFigure 5.1: Confusion matrix for student model’s results on combineddataset.475.2. ABLATION STUDYFigure 5.2: Spectrograms of three whale species.Table 5.6: Results of the ablation study on the student model. ‘TT’ denotesthe training time in minutes and ‘IT’ denotes the inference time in seconds.SturctureTT IT Precision Recall F1InstanceNorm DropKey KD× × × 2.5 0.32 0.893 0.817 0.828✓ × × 1.8 0.32 0.939 0.814 0.839× ✓ × 1.5 0.38 0.954 0.805 0.844✓ ✓ × 1.7 0.38 0.951 0.817 0.850✓ ✓ ✓ 17 0.33 0.944 0.890 0.910The combination of Instance Normalization and DropKey boosts the studentmodel’s precision without affecting its recall. Finally, the introduction oftemperature-cooling KD increases the recall and F1 Score, achieving state-of-the-art.The second ablation study on the KD temperature is presented in Ta-ble 6.1. The experiments were conducted on different fixed temperaturesranging from 1 to 40 and the best one was selected for comparison. We alsoinclude a temperature-increasing strategy that works in a reverse way to ourstrategy. The results indicate that fixing temperature cannot fully exploitthe benefits of KD. Either increasing or decreasing the temperature param-eter during the KD process allows the student model to acquire knowledgefrom the teacher more effectively. Both warming and cooling strategies havesimilar training times, but the cooling strategy shows improved recall andF1 Score.485.2. ABLATION STUDYTable 5.7: Results of the ablation study on the KD temperature strategy.‘TT’ denotes the training time in minutes and ‘IT’ denotes the inferencetime in seconds.Temperature Strategy TT IT Precision Recall F1Fix T = 10 1.5 0.3 0.918 0.836 0.854Warming 3.5 0.3 0.943 0.859 0.887Cooling 3.5 0.3 0.944 0.890 0.910To provide a more intuitive result, we generated the t-distributed stochas-tic neighbor embedding (t-SNE) to visualize the high-dimensional informa-tion from the neurons of the output layer. Fig 5.3 shows that FNet is unableto differentiate most of the classes. Transformer DeiT can isolate the majorclasses but handles the minorities poorly. The teacher model demonstratesthe best capability to isolate different classes. The student model isolatesclass 0 and class 6 better than CC-FSMN. Therefore, we can claim that theproposed framework can capture minor differences between classes betterand is more robust than CC-FSMN.495.2. ABLATION STUDYFigure 5.3: Visualization of features using a t-SNE plot for 12-class Watkinsexperiment.50Chapter 6Conclusion and Future WorkThis thesis proposed the XCFSMN framework that trains a small andfast student model using a pre-trained teacher model with a temperature-cooling strategy. The framework exhibits superior efficiency and perfor-mance compared to CNN backbones and transformers for marine mammalsound classification. Such an acoustic approach has shown better accuracythan the approach based on images. XCFSMN has significant improvementsin performance and similar efficiency compared to CC-FSMN. Moreover, themodel’s pruning also enhances its deployability across various devices.To further improve the classifier, future work can focus on data prepro-cessing techniques rather than model architecture adjustments. Specifically,adopting a diffusion model [19] to denoise the data and generate balanceddatasets. It is worth mentioning that XCFSMN’s applicability extends be-yond marine mammal sound classification. This framework’s concept andmethodology can be adapted for other acoustic signal classification tasks,such as speech sentiment recognition and speaker identification, offeringversatility and broader potential applications in the field of acoustic sig-nal analysis.51Bibliography[1] Ann N. Allen, Matt Harvey, Lauren Harrell, Aren Jansen, Karlina P.Merkens, Carrie C. Wall, Julie Cattiau, and Erin M. Oleson. A con-volutional neural network for automated detection of humpback whalesong in a diverse, long-term passive acoustic dataset. Frontiers in Ma-rine Science, 8, 3 2021. → pages 2, 3, 10, 40, 43, 44, 45[2] Christensen B. Estimated whale population in the pre-whaling period,and in 2001. Marine mammal populations: reconstructing historicalabundances at the global scale., 2006. → pages 5[3] Claridge D.E. Balcomb, K.C. A mass stranding of cetaceans caused bynaval sonar in the bahamas. Bahamas Journal of Science, 2001. →pages 6[4] Mark F. Baumgartner and Sarah E. Mussoline. A generalized baleenwhale call detection and classification system. The Journal of theAcoustical Society of America, 129:2889–2902, 5 2011. → pages 7[5] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: Thelong-document transformer. arXiv:2004.05150, 4 2020. → pages 12[6] W D Bowen. Role of marine mammals in aquatic ecosystems. MarineEcology Progress Series, 158:267–274, 1997. → pages 1[7] Yamada T. Mead J.G. van Helden A.L. Brownell Jr., R.L. Mass strand-ings of cuvier’s beaked whales in japan: Us naval acoustic link? Sci-entific Committee at the 56th Meeting of the International WhalingCommission, 2004. → pages 652Bibliography[8] Eugene H Buck and Kori Clavert. Active military sonar and marinemammals: Events and references references, 2005. → pages 1[9] Christopher W. Clark, Peter Marler, and Kim Beeman. Quantitativeanalysis of animal vocal phonology: an application to swamp sparrowsong. Ethology, 76:101–115, 1987. → pages 7[10] International Whaling Commission. Report of the standing workinggroup on environmental concerns. Journal of Cetacean Research andManagement, 2005. → pages 6[11] Jean Pierre W. Desforges, Christian Sonne, Milton Levin, UrsulaSiebert, Sylvain De Guise, and Rune Dietz. Immunotoxic effects ofenvironmental pollutants in marine mammals. Environment Interna-tional, 86:126–139, 1 2016. → pages 1[12] Dexin Duan, Lian gang Lu¨, Ying Jiang, Zongwei Liu, Chunmei Yang,Jingsong Guo, and Xiaoyan Wang. Real-time identification of marinemammal calls based on convolutional neural networks. Applied Acous-tics, 192, 4 2022. → pages 2, 8, 10[13] Peter J. Dugan, Aaron N. Rice, Ildar R. Urazghildiiev, and Christo-pher W. Clark. North atlantic right whale acoustic signal processing:Part i. comparison of machine learning recognition algorithms. 2010.→ pages 8[14] Chi Nhan Duong, Khoa Luu, Kha Gia Quach, and Ngan Le. Shrink-teanet: Million-scale lightweight face recognition via shrinking teacher-student networks. 5 2019. → pages 12[15] Licandro P. John A.W.G. Edwards M., Johns D.G. and Stevens D.P.Ecological status report: results from the cpr survey 2005/2006. SAH-FOS Technical Report, 4. → pages 6[16] Douglas Gillespie, Marjolaine Caillat, Jonathan Gordon, and PaulWhite. Automatic detection and classification of odontocete whistles.53BibliographyThe Journal of the Acoustical Society of America, 134:2427–2437, 92013. → pages 8[17] Xanadu C Halkias, Se´bastien Paris, and Herve´ Glotin. Classificationof mysticete sounds using machine learning techniques. The Journal ofthe Acoustical Society of America, 134:3496–505, 2013. → pages 8[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep resid-ual learning for image recognition. Computer Vision and Pattern Recog-nition, 12 2015. → pages ix, 3, 18, 43, 44, 45[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion prob-abilistic models. pages 6840–6851, 2020. → pages 51[20] Rotstein D.S. Harms C.A. Southall B.L. Hohn, A.A. Report on marinemammal unusual mortality event umese0501sp: multi species strandingof short-finned pilot whales (globicephala macrorhynchus) minke whale(balaenoptera acutorostrata) and dwarf sperm whales (kogia sima) innorth carolina. NOAA, 2006. → pages 7[21] Muqi Huang and Lefei Zhang. Atrous pyramid transformer with spec-tral convolution for image inpainting. pages 4674–4683. Association forComputing Machinery (ACM), 10 2022. → pages 12[22] Isaac J.L. Effects of climate change on life history: implications forextinction risk in mammals. Endangered Species Research, 2008. →pages 6[23] M. Kaufman. Sonar used before whales hit shore. Washington Post,2004. → pages 7[24] M. Kaufman. Whale stranding in n.c. followed navy sonar use. Wash-ington Post, 2005. → pages 6[25] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distilla-tion. 6 2016. → pages 1254Bibliography[26] Santos M.B. Pierce G.J. Crick H.Q.P. Learmonth J.A., Macleod C.D.and Robinson R.A. Potential effects of climate change on marine mam-mals. Oceanography and Marine Biology: an Annual Review, 2006. →pages 6[27] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago On-tanon. Fnet: Mixing tokens with fourier transforms. arXiv preprintarXiv:2105.03824, 5 2021. → pages 3, 12, 40, 43, 44, 45[28] Bonan Li, Yinhan Hu, Xuecheng Nie, Congying Han, Xiangjian Jiang,Tiande Guo, and Luoqi Liu. Dropkey. arXiv preprint arXiv:2208.02646,8 2022. → pages 36[29] Xiangrui Liu and Julian Cheng. A highly efficient marine mammalsclassifier based on a cross-covariance attended compact feed-forwardsequential memory network (student abstract). Association for the Ad-vancement of Artificial Intelligence, 2023. → pages 43, 46[30] Tao Lu, Baokun Han, and Fanqianhui Yu. Detection and classificationof marine mammal sounds using alexnet with transfer learning. Eco-logical Informatics, 62, 5 2021. → pages 2, 3, 10, 40, 43, 44, 45[31] Simmonds M. and Isaac S. The impacts of climate change on marinemammals: early signs of significant problems. Oryx, 2007. → pages 6[32] Frederic Maire, Lui Mejias Alvarez, and Amanda Hodgson. Automat-ing marine mammal detection in aerial images captured during wildlifesurveys a deep learning approach. pages 379–385, 2015. → pages 1, 46[33] David K. Mellinger and Christopher W. Clark. Recognizing transientlow-frequency whale sounds by spectrogram correlation. The Journalof the Acoustical Society of America, 107:3518–3529, 6 2000. → pages7[34] David K. Mellinger and Christopher W. Clark. Mobysound: A referencearchive for studying automatic recognition of marine mammal sounds.Applied Acoustics, 67:1226–1242, 11 2006. → pages 355Bibliography[35] Eduardo Mercado and Anthony Kuh. Classification of humpback whalevocalizations using a self-organizing neural network. volume 2, pages1584–1589. IEEE, 1998. → pages 8[36] Biassoni N. Samuels A. Tyack P. Miller, P.J.O. Whale songs lengthenin response to sonar. Nature 405, 2000. → pages 7[37] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Aki-hiro Matsukawa, Hassan Ghasemzadeh, and De Shaw. Improved knowl-edge distillation via teacher assistant. 2019. → pages 12, 38[38] Australian Department of the Environment and Water Resources. Mar-ion bay whale stranding. Australian Government, 2005. → pages 6[39] E. C.M. Parsons. Impacts of navy sonar on whales and dolphins: Nowbeyond a smoking gun? Frontiers in Marine Science, 4:295–306, 92017. → pages 1[40] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficultyof training recurrent neural networks. 11 2013. → pages 33[41] De´bora Pollicelli, Mariano Coscarella, and Claudio Delrieux. Roidetection and segmentation algorithms for marine mammals photo-identification. Ecological Informatics, 56, 3 2020. → pages 2, 46[42] Gordon J.C.D. Rendell, L.E. Vocal response of long-finned pilot whales(globicephala melas) to military sonar in the ligurian sea. Marine Mam-mal Science 15, 1999. → pages 7[43] Mark P. Simmonds and Wendy J. Eliott. Climate change and cetaceans:concerns and recent developments. Journal of the Marine BiologicalAssociation of the United Kingdom, 89, 2009. → pages 6[44] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, andSanjeev Khudanpur. X-vectors: Robust dnn embeddings for speakerrecognition. volume 2018-April, pages 5329–5333. Institute of Electricaland Electronics Engineers Inc., 9 2018. → pages 3156Bibliography[45] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, and RuslanSalakhutdinov. Dropout: A simple way to prevent neural networksfrom overfitting, 2014. → pages 36[46] Robin Strudel, Ricardo Garcia, Ivan Laptev Inria, and Cordelia SchmidInria. Segmenter: Transformer for semantic segmentation. pages 7242–7252, 2021. → pages 45[47] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, and JonathonShlens. Rethinking the inception architecture for computer vision. 2016.→ pages 37[48] Barbara L. Taylor, Melissa Martinez, Tim Gerrodette, Jay Barlow, andYvana N. Hrovat. Lessons from monitoring trends in abundance ofmarine mammals. Marine Mammal Science, 23:157–175, 1 2007. →pages 1[49] Mark Thomas, Bruce Martin, Katie Kowarski, Briand Gaudet, andStan Matwin. Marine mammal species classification using convolutionalneural networks and a novel acoustic representation. 7 2019. → pages7[50] Hugo Touvron, Matthieu Cord, and Herve´ Je´gou. Deit iii: Revenge ofthe vit. pages 516–533, 2022. → pages 3, 12, 40, 43, 44, 45[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LlionJones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attentionis all you need. 6 2017. → pages ix, 10, 11[52] Moore K.E. Tyack P. Watkins, W.A. Investigations of sperm whaleacoustic behaviors in the southeast caribbean. Cetology 49, 1985. →pages 7[53] Kang You, Kele Xu, Boqing Zhu, Ming Feng, Dawei Feng, Bo Liu,Tian Gao, and Bo Ding. Masked modeling-based audio representationfor acm multimedia 2022 computational paralinguistics challenge. pages57Bibliography7060–7064. Association for Computing Machinery (ACM), 10 2022. →pages 7[54] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, LeiZhang, and Jianfeng Gao. Multi-scale vision longformer: A new visiontransformer for high-resolution image encoding. ICCV, 2021. → pages3, 40, 43, 44, 45[55] Shiliang Zhang, Hui Jiang, Shifu Xiong, Si Wei, and Lirong Dai. Com-pact feedforward sequential memory networks for large vocabulary con-tinuous speech recognition. pages 3389–3393, 2016. → pages 32[56] Shiliang Zhang, Cong Liu, Hui Jiang, Si Wei, Lirong Dai, and Yu Hu.Feedforward sequential memory networks: A new structure to learnlong-term dependency. 12 2015. → pages 33[57] Weisong Zhao, Xiangyu Zhu, Kaiwen Guo, Xiao-Yu Zhang, and ZhenLei. Grouped knowledge distillation for deep face recognition. 2023. →pages 12[58] Ming Zhong, Manuel Castellote, Rahul Dodhia, Juan Lavista Ferres,Mandy Keogh, and Arial Brewer. Beluga whale acoustic signal classifi-cation using deep learning neural network models. The Journal of theAcoustical Society of America, 147:1834–1841, 3 2020. → pages 2, 3,10, 40, 43, 44, 4558AppendixAppendix A: TablesTable 6.1: TOPSIS score of 5-class Moby data.Model Norm Precision Norm Recall Norm F1 Norm TT Norm IT S+ S− PAlexNet 0.02987 0.03111 0.10778 0.01022 0.00423 0.01787 0.25696 0.93496ResNet-18 0.03280 0.03072 0.10990 0.00876 0.01269 0.01730 0.25027 0.93534resnet-50 0.03204 0.02973 0.10565 0.01022 0.03383 0.03557 0.22922 0.86567vgg-16 0.02969 0.02931 0.10328 0.04673 0.05709 0.07181 0.19873 0.73458Densenet 0.03304 0.03156 0.11377 0.01752 0.04229 0.04197 0.22210 0.84107Deit3 0.03080 0.03026 0.11190 0.07594 0.24316 0.25016 0.07204 0.22359Longformer 0.02268 0.02481 0.08055 0.01752 0.09727 0.10445 0.16198 0.60797Fnet 0.01274 0.02055 0.04284 0.00292 0.00846 0.08437 0.24580 0.74447CC-FSMN 0.03373 0.03160 0.11515 0.00438 0.00423 0.00876 0.26077 0.96750Teacher 0.03387 0.03487 0.12314 0.03213 0.21779 0.21555 0.09830 0.31321Student 0.03349 0.03438 0.12151 0.00949 0.00423 0.00680 0.26137 0.9746559Appendix A: TablesTable 6.2: TOPSIS score of 5-class Watkins data.Model Norm Precision Norm Recall Norm F1 Norm TT Norm IT S+ S− PAlexNet 0.024624494 0.027516838 0.098392909 0.012133865 0.002890465 0.040280719 0.145914568 0.783664134Resnet18 0.030625746 0.027924831 0.104714694 0.006066933 0.00578093 0.031868124 0.147112969 0.821946983Resnet50 0.030625746 0.027924831 0.104714694 0.010313785 0.017342791 0.035639858 0.136119655 0.792501401vgg16 0.034497522 0.037308662 0.12692199 0.016380718 0.028904651 0.030727925 0.138596742 0.818526587Densenet 0.032561634 0.037127332 0.118006652 0.02790789 0.023123721 0.036934778 0.136833566 0.787448177Deit3 0.030238568 0.031914093 0.106984053 0.036401595 0.135851861 0.140284783 0.064831583 0.316072208Longformer 0.018274781 0.011197132 0.044576689 0.084937056 0.317951164 0.339423652 0.188530174 0.357095952Fnet 0.022920912 0.013599755 0.045225077 0.00182008 0.004335698 0.093024087 0.135986699 0.593800413CC-FSMN 0.030625746 0.027924831 0.104714694 0.004550199 0.002890465 0.03156961 0.149989436 0.82611932Teacher 0.036626998 0.037535325 0.13405426 0.007887012 0.034685582 0.032368767 0.140353445 0.812596383Student 0.034845982 0.037353995 0.128218766 0.006066933 0.002890465 0.00743597 0.161843592 0.956072843Table 6.3: TOPSIS score of 5-class merged data.Model Norm Precision Norm Recall Norm F1 Norm TT Norm IT S+ S− PAlexNet 0.030823731 0.031257854 0.104679055 0.00470354 0.014273829 0.020540188 0.125312884 0.859172056Resnet18 0.033205726 0.031609505 0.112442975 0.013065389 0.028547659 0.02471619 0.118978705 0.827995353Resnet50 0.032139161 0.032351879 0.112710696 0.01646239 0.078506061 0.070559269 0.09975296 0.585706386Vgg16 0.027872901 0.031296926 0.105749941 0.044422323 0.132032921 0.130422158 0.091119008 0.411296057Densenet 0.032565787 0.032430024 0.116324935 0.016985006 0.092779891 0.084053202 0.101673468 0.547436013Deit3 0.032992413 0.028913515 0.105348359 0.06010079 0.092779891 0.103117517 0.081137962 0.440355762Longformer 0.023748849 0.023208957 0.083930649 0.049648479 0.235518184 0.234037302 0.15416721 0.397128847Fnet 0.009279116 0.011565406 0.029717072 0.000783923 0.017842287 0.100455621 0.09557267 0.487545289CC-FSMN 0.031748088 0.031922083 0.110836647 0.006532695 0.010705372 0.014781345 0.130787621 0.898458132Teacher 0.034698918 0.034774363 0.123821133 0.013065389 0.157012123 0.146821319 0.127978533 0.465715436Student 0.033561248 0.034774363 0.121813223 0.031356934 0.010705372 0.03065999 0.131043559 0.81039383560Appendix A: TablesTable 6.4: TOPSIS score of 12-class Watkins data.Model Norm Precision Norm Recall Norm F1 Norm TT Norm IT S+ S− PAlexNet 0.0263835 0.029938657 0.093777835 0.007315667 0.005726721 0.038446595 0.250422254 0.866906401Resnet18 0.034733834 0.034460005 0.122902334 0.008047233 0.013509188 0.011911235 0.254858386 0.955350109Resnet50 0.034955917 0.034622936 0.123359787 0.01931336 0.039499692 0.037780633 0.230780348 0.859321958Vgg16 0.0338455 0.034297074 0.120310101 0.048283401 0.086194496 0.092158006 0.187907458 0.670941199Densenet 0.03131375 0.032993622 0.11207595 0.039358287 0.051834169 0.060562783 0.212730358 0.778396257Deit3 0.025273083 0.026883692 0.089355791 0.046820267 0.239347577 0.241997861 0.075900433 0.238756969Longformer 0.01052675 0.011160805 0.032784121 0.054867501 0.167396465 0.198961079 0.074044276 0.271219133FNet 0.00533 0.009164895 0.018298114 0.0065841 0.01027873 0.119392105 0.232575763 0.66078692CC-FSMN 0.034867084 0.033726813 0.120005132 0.001463133 0.004698848 0.01177573 0.262559213 0.957075355Teacher 0.03797625 0.034948799 0.130069095 0.0175576 0.158586125 0.154278033 0.146976377 0.487881246Student 0.036244 0.034093409 0.125037114 0.008486173 0.004698848 0.005715864 0.26362824 0.978778581Table 6.5: Overall TOPSIS score.Moby 5 Watkins 5 Merged 5 Watkins 12 Averaged Score RankingAlexNet 0.934964074 0.783664134 0.859172056 0.866906401 0.861176666 4Resnet18 0.935344641 0.821946983 0.827995353 0.955350109 0.885159271 3Resnet50 0.865674213 0.792501401 0.585706386 0.859321958 0.775800989 5Vgg16 0.734575863 0.818526587 0.411296057 0.670941199 0.658834926 7Densenet 0.841074588 0.787448177 0.547436013 0.778396257 0.738588759 6Deit3 0.223594315 0.316072208 0.440355762 0.238756969 0.304694814 11Longformer 0.607974055 0.357095952 0.397128847 0.271219133 0.408354497 10FNet 0.744471029 0.593800413 0.487545289 0.66078692 0.621650913 8CC-FSMN 0.967497744 0.82611932 0.898458132 0.957075355 0.912287638 2Teacher 0.31321056 0.812596383 0.465715436 0.487881246 0.519850906 9Student 0.974652323 0.956072843 0.810393835 0.978778581 0.929974395 161
Thesis/Dissertation
eng
Electrical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Cheng, Julian
Du, Shan
Highly efficient sound classification for marine mammals
Text