Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Zhao, Beiliang
Master of Applied Science - MASc
University of British Columbia
Monitoring marine mammals through vocalizations provides scientists with essential insights into their abundance and distribution. Systems that classify these acoustic sensing from less than 10 to over 100000 hertz (Hz) enable precise surveillance of specific species, thereby playing a crucial role in the protection efforts for endangered species. However, current solutions based on traditional machine learning technologies face challenges in classifying a diverse range of marine mammal sounds with high accuracy. This study aims to improve classification accuracy for marine mammal vocalizations by refining the feature vectors from sounds, deploying advanced deep-learning models, and comparing the performance of various deep-learning models. Furthermore, this research utilizes the generalization capability of deep-learning models to process fresh acoustic data from different sources or additional species with high accuracy. The advancements proposed in this study offer a marked improvement in marine mammal surveillance, allowing for more precise species identification based on their vocal patterns.
Marine Mammal Sound ClassificationUsing Deep Learning ModelsbyBeiliang ZhaoB.Eng., Beijing Jiaotong University, 2021B.Eng., University of Wollongong, 2021A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF APPLIED SCIENCEinTHE COLLEGE OF GRADUATE STUDIES(Electrical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)December 2023© Beiliang Zhao, 2023The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Marine Mammal Sound Classification Using Deep LearningModelssubmitted by Beiliang Zhao in partial fulfilment of the requirements ofthe degree of Master of Applied ScienceJulian Cheng, School of EngineeringSupervisorThomas Johnson, School of EngineeringSupervisory Committee MemberAnas Chaaban, School of EngineeringSupervisory Committee MemberQian Chen, School of EngineeringUniversity ExamineriiAbstractMonitoring marine mammals through vocalizations provides scientistswith essential insights into their abundance and distribution. Systems thatclassify these acoustic sensing from less than 10 to over 100000 hertz (Hz)enable precise surveillance of specific species, thereby playing a crucial rolein the protection efforts for endangered species. However, current solutionsbased on traditional machine learning technologies face challenges in classi-fying a diverse range of marine mammal sounds with high accuracy. Thisstudy aims to improve classification accuracy for marine mammal vocaliza-tions by refining the feature vectors from sounds, deploying advanced deep-learning models, and comparing the performance of various deep-learningmodels. Furthermore, this research utilizes the generalization capability ofdeep-learning models to process fresh acoustic data from different sources oradditional species with high accuracy. The advancements proposed in thisstudy offer a marked improvement in marine mammal surveillance, allowingfor more precise species identification based on their vocal patterns.iiiLay SummaryProtecting marine mammals to maintain biodiversity is important tothe earth’s ecosystem. Identifying endangered species of marine mammalsthrough their distinct sounds and then knowing their distribution and abun-dance help researchers establish methods to conserve them. However, manycurrent classification systems based on traditional machine learning modelsstruggle to distinguish among the varied sounds produced by categories en-compassing many marine mammal species with high accuracy. This researchdelves into this challenge, employing advanced deep-learning models to en-hance the vocalization classification accuracy of large-scale marine mammalspecies.ivTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . vList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiGlossary of Notation . . . . . . . . . . . . . . . . . . . . . . . . . xAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . xiiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Motivation and Background . . . . . . . . . . . . . . . . . . . 11.2 Research Gap and Objective . . . . . . . . . . . . . . . . . . . 21.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . 3Chapter 2: Background . . . . . . . . . . . . . . . . . . . . . . . 42.1 Traditional Machine Learning . . . . . . . . . . . . . . . . . 42.2 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2.1 Feature Extraction . . . . . . . . . . . . . . . . . . . . 52.2.2 Feature Refining . . . . . . . . . . . . . . . . . . . . . 82.2.3 Deep-Learning Models . . . . . . . . . . . . . . . . . . 92.3 Background Summary . . . . . . . . . . . . . . . . . . . . . . 15Chapter 3: Methodology . . . . . . . . . . . . . . . . . . . . . . 163.1 Overview of Datasets . . . . . . . . . . . . . . . . . . . . . . . 173.2 Data Pre-processing . . . . . . . . . . . . . . . . . . . . . . . 183.2.1 Data Cleaning . . . . . . . . . . . . . . . . . . . . . . 18vTABLE OF CONTENTS3.2.2 Resampling Rate Selection . . . . . . . . . . . . . . . 213.2.3 Spectrogram Generating and Slicing . . . . . . . . . . 243.2.4 Spectrogram Saving and Further Processing . . . . . . 263.3 Deep-Learning Models Deployment and Training . . . . . . . 273.3.1 Training Environment . . . . . . . . . . . . . . . . . . 273.3.2 The Stratified K-Fold Cross-Validation . . . . . . . . . 273.4 Model Performance Evaluation . . . . . . . . . . . . . . . . . 283.4.1 Transfer Learning . . . . . . . . . . . . . . . . . . . . 303.4.2 Artificial Neural Network (ANN) . . . . . . . . . . . . 303.4.3 Long Short-Term Memory Network (LSTM) . . . . . . 303.4.4 Convolutional Neural Network (CNN) . . . . . . . . . 303.4.5 Audio Spectrogram Transformer (AST) . . . . . . . . 31Chapter 4: Results and Discussion . . . . . . . . . . . . . . . . 354.1 Analyzing the Effect of Spectrogram Size on the CNN Model 354.2 Performance Comparison of Deep-Learning Models . . . . . . 364.3 Evaluating the Generalization Capability of the CNN Model . 404.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44Chapter 5: Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 505.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.2 Limitations and Future Research . . . . . . . . . . . . . . . . 505.2.1 Limited Research in “SpecAugmentation” . . . . . . . 505.2.2 Device-specific Noise . . . . . . . . . . . . . . . . . . . 515.2.3 Avoiding Potential Overfitting . . . . . . . . . . . . . 515.2.4 Implementing Feature Engineering . . . . . . . . . . . 535.2.5 Implementing larger deep-learning models . . . . . . . 535.2.6 Collecting More Data and Testing in Real-world . . . 53Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55viList of TablesTable 3.1 The information about sound clips for each speciesfrom “Best of cuts”. . . . . . . . . . . . . . . . . . . . 19Table 4.2 The effect of the spectrogram size on ResNet50 model(original spectrogram). . . . . . . . . . . . . . . . . . . 35Table 4.3 The effect of the spectrogram size on ResNet50 model(resized spectrogram 224× 224). . . . . . . . . . . . . 35Table 4.4 The effect of the spectrogram size on ResNeXt50 model(resized spectrogram). . . . . . . . . . . . . . . . . . . 36Table 4.5 The effect of the spectrogram size on EfficientNetB0model (resized spectrogram). . . . . . . . . . . . . . . 36Table 4.6 The mean cross-validation results of each model on“Best of cuts”. . . . . . . . . . . . . . . . . . . . . . . 37Table 4.7 The mean cross-validation results of each model on“All cuts”. . . . . . . . . . . . . . . . . . . . . . . . . . 40Table 4.8 The mean cross-validation results of generalization eval-uation in ResNeXt50. . . . . . . . . . . . . . . . . . . . 41Table 4.9 The mean cross-validation results of the ResNeXt50and GhostNet models in the MobySound dataset. . . . 42Table 4.10 The mean cross-validation results of each model onoriginal “Best of cuts”. . . . . . . . . . . . . . . . . . . 45Table 4.11 The classification report of the ResNeXt50 in the 5thcross-validation results on “Best of cuts” without datacleaning. . . . . . . . . . . . . . . . . . . . . . . . . . . 46viiList of FiguresFigure 2.1 The feature extraction algorithm and the correspond-ing classification model in the previous research. . . . 5Figure 2.2 Example of a spectrogram for a clip of marine mam-mal sound. . . . . . . . . . . . . . . . . . . . . . . . . 6Figure 2.3 The generation process of a spectrogram [JJLJ20]. . . 8Figure 2.4 The structure of one hidden layer ANN [ON15]. . . . 10Figure 2.5 The structure of the CNN [AMAZ17]. . . . . . . . . . 11Figure 2.6 The schematic structure of the RNN [LBH15]. . . . . 12Figure 2.7 The schematic structure of the LSTM cell [Ola15]. . . 13Figure 3.1 The overview of the proposed framework. . . . . . . . 17Figure 3.2 Clymene dolphin spectrogram with 22.05kHz and 160kHzsample rate. . . . . . . . . . . . . . . . . . . . . . . . 22Figure 3.3 The process of the spectrogram slicing, saving andfurther processing. . . . . . . . . . . . . . . . . . . . . 25Figure 3.4 Comparison of original and “SpecAugment” spectro-grams from a sound clip of the Clymene dolphin. . . . 26Figure 3.5 The schematic figure for the stratified 5-fold cross-validation [CZZY22]. . . . . . . . . . . . . . . . . . . 28Figure 3.6 The example of a confusion matrix [BBG20]. . . . . . 29Figure 3.7 The overview architecture of the ViT and the layerstructure of the transformer encoder [DBK+20]. . . . 31Figure 3.8 The overview of attention mechanism [VSP+17]. . . . 32Figure 3.9 The size change of query matrix [GG20]. . . . . . . . 33Figure 4.1 The valid accuracy and loss change for ResNeXt50,AST, and GhostNet during the 5-fold cross-validationtraining in the “Best of cuts”. . . . . . . . . . . . . . 39Figure 4.2 The valid accuracy change for ResNeXt50 and Ghost-Net with different pre-trained weights during the 5-fold cross-validation training. . . . . . . . . . . . . . . 43viiiLIST OF FIGURESFigure 4.3 The confusion matrix of the ResNeXt50 in the 5thcross-validation results on “Best of cuts” without datacleaning. . . . . . . . . . . . . . . . . . . . . . . . . . 49Figure 5.1 Example of a spectrogram with unusual noise. . . . . 52ixGlossary of NotationANNs Artificial Neural Networks. 2AST Audio Spectrogram Transformer. 2CART Classification Regression Tree. 4CNNs Convolutional Neural Networks. 1dB Decibels. 6DFT Discrete Fourier Transform. 7FN False Negative. 29FP False Positive. 29GMM Gaussian Mixture Model. 4HMM Hidden Markov Model. 4Hz Hertz. 1kHz Kilohertz. 18LDA Linear Discriminant Analysis. 4LSTMs Long Short-Term Memory Networks. 2MLP Multilayer Perceptron. 9PSD Power Spectral Density. 22RNNs Recurrent Neural Networks. 12STFT Short-Time Fourier Transform. 6SVM Support Vector Machine. 4xGlossary of NotationTN True Negative. 29TP True Positive. 29ViT Vision Transformer. 31xiAcknowledgementsI am grateful to my supervisor Prof. Julian Cheng for his guidance andsupport in this thesis.Besides, I want to thank Dr. Ruowen Bai for his advice in experiments.I would also like to thank my family and friends for their support, en-couragement, and love.ChatGPT-3.5/4 and Grammarly are used to proofread and refine myacademic writing.xiiChapter 1Introduction1.1 Motivation and BackgroundMarine mammals are an important part of the Earth’s ecosystem, hold-ing a unique place in the aquatic food chain and contributing to biodiversity.Regrettably, numerous threats such as habitat loss, hunting, vessel strikes,pollution, and climate change caused by human behaviour compromise theirsurvival [SCC08]. Consequently, many marine mammal species, particularlythose close to human activities, are threatened or endangered.One of the salient aspects of marine mammals is their reliance on vocal-izations, which serve as a primary medium for communication and interac-tion [WW85, DTG09]. The frequency spectrum of marine mammal soundcan span from less than 10 to over 100000 hertz (Hz) [JT03]. Monitoringthese vocalizations presents an invaluable opportunity for research, as thesesounds can be captured irrespective of environmental conditions like lowvisibility underwater or inclement weather on the ocean surface [VGG+18].These unique vocal patterns offer a potential way to identify and differenti-ate marine mammal species.Systems that classify species of marine mammals based on classifyingtheir vocalization enable precise surveillance of endangered species, offeringcomprehensive information on their abundance and distribution [RLL22].The crucial component of this system is the efficient classification algorithm.However, current solutions based on traditional machine learning modelsface challenges in classifying a diverse range of marine mammal sounds withhigh accuracy due to their limited capability of processing large amounts ofdata. The objective of this research is to enhance the classification accuracyand efficiency of marine mammal vocalizations using deep learning. A seriesof methods are proposed to improve the classification accuracy, and thegeneralization ability of the Convolutional Neural Network (CNN) is utilizedto process the new acoustic data from different sources or additional specieswith high accuracy. The generalization ability describes the capability ofthe model to accurately adapt to new and unseen data [Sar96].11.2. RESEARCH GAP AND OBJECTIVE1.2 Research Gap and ObjectiveAlthough several methods based on machine learning are applied to clas-sify the vocalization of marine mammal sounds, research gaps still exist. Themain drawback of existing solutions is the limited accuracy in classifyingvaried sounds produced by categories encompassing many marine mammalspecies. The majority of studies classify fewer than ten marine mammalspecies [Nta17, JBD+19, TMK+20, LHY21], with many focusing solely onthe detection of one specific species [BBW+19, BSC+19, AHH+21, EABP22,SPR+20, KFS+20]. However, several different marine mammal species mayappear in the same area, and the monitor system should have the capabil-ity to classify them. The main reason is that traditional machine learningalgorithms or are still prevalent in this system, which lack the capabilityto process large amounts of data. As a result, the monitor system cannotclassify a diverse range of marine mammal sounds with high accuracy.Based on these identified gaps, the primary research objective is to im-prove the classification accuracy of marine mammal sounds, especially byfollowing approaches:− Employing Advanced Deep-learning Models: Audio Spectro-gram Transformer (AST), a modified version of the Transformer spe-cialized in the sound classification field, is applied in the context ofmarine mammal sound classification, whose classification accuracy sur-passes CNN models in the “AudioSet” audio classification challenge[GCG21]. AST has been applied to classify bat calls with classificationaccuracy exceeding the CNN-based approaches [BVG+22]. This modelhas the potential to achieve higher classification accuracy compared tothe prior CNN-based solution in marine mammal sound classification.− Comprehensive Deep-Learning Model Comparison: A widearray of deep-learning models, including Artificial Neural Networks(ANNs), Convolutional Neural Networks (CNNs), Long Short-TermMemory networks (LSTMs), and AST, within the realm of marinemammal vocalization classification are evaluated to discover the bestperformed deep-learning model by comparing the classification accu-racy.− Optimizing Feature Extraction Parameters from Sound: Theimpact of dimension for the audio spectrogram, a feature extractedfrom marine mammal vocalizations as the input of deep-learning mod-els, is examined on classification outcomes. The specific spectrogram21.3. THESIS ORGANIZATIONdimension generated by the optimal parameters to make the CNNshave the best classification accuracy is discovered by comparing theperformance of CNNs trained by the different dimensions of spectro-grams.− Utilizing the Generalization Ability of CNNs to Process Un-seen Acoustic Data: Evaluating the CNN classifier is capable ofprocessing unseen acoustic data from different sources or additionalspecies with high classification accuracy and fast training speed byutilizing its generalization ability and transfer learning.1.3 Thesis OrganizationThis thesis is structured into five distinct chapters:− Chapter 1 - Introduction: Chapter 1 offers background informa-tion and the motivation for leveraging deep-learning models to classifymarine mammal vocalizations, summarizing the existing limitations incurrent research and laying out the objectives of this study.− Chapter 2 - Background: Chapter 2 reviews the basic knowledgeabout audio features, traditional machine learning, and deep-learningalgorithms employed in prior research for marine mammal vocalizationclassification.− Chapter 3 - Methodology: Chapter 3 presents the research frame-work with the applied methods.− Chapter 4 - Results and Discussion: Chapter 4 illustrates theresearch results and contrasts them with earlier works.− Chapter 5 - Conclusion: Chapter 5 concludes the contributionsand limitations of this research and proposes potential future researchdirections.3Chapter 2BackgroundThe majority of existing solutions for automatic marine mammal soundclassification are based on machine learning, including building mathemat-ical algorithms through traditional machine learning methods and trainingneural networks by deep learning. This chapter reviews the basic knowl-edge of previous approaches applied to classify the vocalization of marinemammals, including traditional machine-learning algorithms, deep-learningmodels, and feature vector refining methods employed to enhance classifica-tion performance.2.1 Traditional Machine LearningThe core objective of machine learning is to build models of data by spe-cific learning algorithms [Zho21]. In this thesis, the model aims to classifythe audio data about the vocalization of marine mammals. Developing anefficient classification model of marine mammal vocalizations requires a suit-able traditional machine-learning algorithm and feature vectors extractedfrom audio data.The frequency feature is a critical type of audio feature and machinelearning input, including cepstral coefficients, Mel-frequency cepstral coeffi-cients, and spectrograms. Traditional machine learning techniques such asGaussian Mixture Model (GMM), Hidden Markov Model (HMM), SupportVector Machine (SVM), Linear Discriminant Analysis (LDA), and Classifi-cation Regression Tree (CART) have been employed to perform classifica-tion corresponding to these features [WWX17, KFS+20, DRUC10, RSH+08,BS09, MUB07, Tra21, BS09]. Figure 2.1 illustrates the feature extractionalgorithms and the corresponding classification models used in existing stud-ies.However, it is essential to note the inherent limitations of traditionalmachine learning algorithms. One of the significant shortcomings is thelow classification accuracy in marine mammal vocalization, especially com-pared to the deep-learning models [KFS+20]. Additionally, these algorithmscannot classify categories encompassing many marine mammal species due42.2. DEEP LEARNINGFigure 2.1: The feature extraction algorithm and the corresponding classi-fication model in the previous research.to the limited capacity for traditional machine learning algorithms in dataprocessing.As the ongoing development of machine learning, the rise of deep learn-ing brought neural networks with mighty feature learning and data pro-cessing abilities that potentially addressed the limitations like low classi-fication accuracy and capacity observed in algorithms built by traditionalmachine learning. The following section introduces the implementation ofdeep-learning models in marine mammal vocalization classifications.2.2 Deep LearningThe process of employing deep-learning models includes feature vectorextraction and deep-learning model training.2.2.1 Feature ExtractionThe raw audio data requires conversion into a more representative formatbefore being processed by the deep-learning model. A general choice is thespectrogram, which translates audio data into visual representations of theaudio signal’s strength at a specific frequency and time, enabling them tobe processed as images [BPRL17]. This transformation bridges the gapbetween image and audio classification techniques [Wys17].Figure 2.2 shows an example of a spectrogram of a clip of marine mammalsound. The vertical axis illustrates the frequency in Hz, whose range is from52.2. DEEP LEARNING0 to half of the sample rate used to load the audio signal. The horizontalaxis shows the time in seconds, whose range is the same as the duration ofthe audio. Each point’s colour on the spectrogram corresponds to the audiosignal’s strength at a specific frequency and time. The maximum amplitudeis scaled to 0 decibels (dB), and the amplitude below −80 dB is clipped to−80 dB [MRL+15].Figure 2.2: Example of a spectrogram for a clip of marine mammal sound.The spectrogram is generated through the Short-Time Fourier Transform(STFT) [PS18], as visualized in Figure 2.3. Initially, multiple overlappedwindow functions are applied to the audio’s time domain. When the samplerate used to load the audio signal is fixed, window lengths, hop lengths,and window types predominantly shape the spectrogram’s sizes and quality.The window length determines the number of frequency bins in the spectro-gram. The number of frequency binsNfb can be calculated byNfb = 1+N/2for a window length N [PS18]. Frequency bins represent the intervals be-tween samples within the frequency domain, and this interval is calculatedby the ratio of sample rate and window length, which means that the widthof the frequency bin is fixed in the same sample rate. Thus, an extendedwindow length results in an increased number of frequency bins in the spec-trogram and leads to the expansion of the spectrogram’s vertical dimension62.2. DEEP LEARNING[MRL+15, JJLJ20]. Meanwhile, the hop length affects the overlap betweeneach window. For a set window length, a shorter hop length increases over-lap, enhancing the spectrogram’s time resolution and thereby expanding thehorizontal dimension of the spectrogram. Different window types exhibitvaried curves, influencing the captured signal segmentation and affectingthe frequency spectrum in the spectrogram [PS18, MRL+15, JJLJ20].After implementing the window functions to the signal, Discrete FourierTransform (DFT) is applied to each signal segment extracted by the windowfunction [JJLJ20]. The DFT can be calculated asYk =N−1∑n=0yne−i 2πnkN , k = 0, ..., N − 1 (2.1)where the data sequence yn is transformed to the data sequence Yk in fre-quency domain [SM+05]. The matrix of complex numbers S is obtained byapplying the DFT to all the signal segments. Each column is each audiosignal segment transformed into the frequency domain. Finally, the powerspectrogram D is calculated by scaling complex numbers into decibels. Thescaling equation isd = 20× log10(|s|/ref) (2.2)where d is each element in the power spectrogram and s is each element inthe matrix of complex numbers. The ref is the reference value, and themaximum absolute value from the matrix is utilized as a reference. Thus,the maximum value of the matrix is scaled to 0 dB. The threshold of thelowest value is set to −80 dB, and the value below −80 dB is clipped to −80dB [Smi13, MRL+15, JJLJ20].72.2. DEEP LEARNINGFigure 2.3: The generation process of a spectrogram [JJLJ20].2.2.2 Feature RefiningRefining the input feature vectors before feeding them to deep-learningmodels is a crucial method to optimize the classification performance of82.2. DEEP LEARNINGdeep-learning models. One approach is to optimize the parameters that af-fect the generation of spectrograms. Parameters, including window type,window length, and hop length, employed to generate spectrograms have anoticeable impact on classification results. For instance, the CNN modellike ResNet50 has shown improved classification accuracy with spectrogramgenerated by specific parameter combination [MIHA22]. The debate be-tween the utility of single-channel versus multi-channel spectrograms re-mains ongoing. The single-channel spectrogram feature outperforms thethree-channel spectrogram that is made by stacking three single-channelspectrograms according to the channel dimension when training the ResNet50model [MIHA22]. However, the efficacy of the three-channel spectrogramswith varied window lengths sometimes surpasses their single-channel coun-terparts when trained CNN-based models such as ResNet50, VGG19, andEfficientNetB0 [TMK+20, WWB+22].2.2.3 Deep-Learning ModelsDeploying deep-learning models improves the classification accuracy ofmarine mammal sounds compared with traditional machine-learning meth-ods since deep-learning models have more capability to process data andlearn the data feature. Three types of deep-learning architectures have beenexplored in the realm of marine mammal sound classification.Artificial Neural Network (ANN): The ANN, commonly known asthe multilayer perceptron (MLP), is inspired by biological neural networksand has been implemented in engineering, which consists of numerous arith-metic units, often termed neurons, that process multiple inputs and outputsin parallel [MS14, Zup94]. Figure 2.4 demonstrates a three-layer ANN. Inthis structure, each circle symbolizes an arithmetic unit, and these units aregrouped into layers. All outputs from one layer’s units are connected to thenext layer’s units, characterizing it as a fully-connected layer [Bur19]. Theinput layer receives the initial data, which then traverses through the hiddenlayer(s) for processing and feature extraction. The mathematical expressionrepresenting each unit isf(x) = g(ω0 +N∑n=1ωnxn)(2.3)where g is the activation function in this unit. The weight matrix ω and biasvector ω0 are the parameters that would be optimized during the training[WBK20]. x is the vector of the input data. Then, the f(x) of each unit92.2. DEEP LEARNINGFigure 2.4: The structure of one hidden layer ANN [ON15].in hidden layers feeds forward to the output layer. Finally, the result vec-tor is obtained from the output layer. For multi-class classification tasks,the softmax activation function is utilized in the output layer to transformthe output vector into a probability distribution across multiple categories[Bur19]. The softmax function is expressed asσ(zi) =ezi∑Nk=1 ezk(2.4)where zi is the input vector, and N is the number of the categories.However, the design of this architecture restricts ANNs to only pro-cess one-dimensional data. ANNs face limitations when processing two-dimensional spectrogram features. Since the flattening procedure beforefeeding spectrograms into ANNs can lead to loss of critical information. Asthe deep-learning architectures have evolved, especially with the develop-ment of CNNs and RNNs, the reliance on ANNs has diminished. Neverthe-less, this model could be a baseline because it is the simplest deep-learningmodel.Convolutional Neural Network (CNN): CNN, an evolution of ANN,102.2. DEEP LEARNINGFigure 2.5: The structure of the CNN [AMAZ17].incorporates convolutional layers specifically designed to extract featuresfrom two-dimensional data [LLY+22]. Due to their proficiency in han-dling two-dimensional data, CNN has become mainstream in image pro-cessing tasks. The foundational architecture of a CNN is depicted in Figure2.5. Input data undergoes feature extraction through various CNN layersand is subsequently down-sampled via pooling layers [I1]. This iterativeprocess results in progressively intricate feature maps of the input. Post-feature extraction, the down-sampled vectors are channelled through severalfully-connected layers for further refinement and transformation into morediscerning vectors. The terminal fully-connected layer, equipped with asoftmax activation function, translates these vectors into class predictions[Bri89]. A notable advantage of CNN over ANN is its ability to main-tain model quality with significantly fewer parameters when processing two-dimensional data, primarily due to the inclusion of convolutional and poolinglayers [Bur19].CNNs have gained significant traction in the context of marine mam-mal sound classification. They are especially noted for their versatility andpotential for enhancement via transfer learning [TMK+20]. ResNet andits variants have been prominently deployed among the various CNN ar-chitectures due to their robust performance in classification tasks [MIHA22,TMK+20]. The current bench mark of “Best of cuts” is reached by ResNet50model with 86.7% F1-score. Additionally, specialized models like VGGish,a derivative of the VGG model tailored for audio classification, have demon-strated superiority over the ResNet model in specific scenarios [HHL+22].For classification tasks with a restricted category range, traditional CNNmodels such as LeNet, AlexNet, and VGG remain in use [TMK+20, SPR+20,112.2. DEEP LEARNINGFigure 2.6: The schematic structure of the RNN [LBH15].LHY21].Recurrent Neural Network (RNN): RNNs are developed to handlesequence data. The unique structure of RNNs, characterized by loops withineach layer, enables them to retain information from previous time steps[Bur19]. Figure 2.6 illustrates the architecture of an RNN layer. The tthRNN unit ingests an input vector xt from a sequence of vectors x and astate vector ht−1 from the preceding time step. The resulting state vectorht and output vector yt can be calculated asht = g1(Wxxt +Whht−1) (2.5)yt = g2(Wyht) (2.6)where Wx, Wh, and Wy are the weight matrix of the RNN layer, which arecalculated from the training data and share the same value in every unit(every time step for the sequence input) [Bur19, LBH15]. In Equation 2.6,g1 is the activation function applied to the linear combination of input xt andht−1 with the weight Wx and Wh to update the state vector. The activationfunction tanh is usually applied, which is shown astanh(z) =ez − e−zez + e−z(2.7)where z is the input vector of the tanh activation function.The activation function g2 is applied to the linear combination of currentstate vector ht with weight of Wy to obtain the output yt. The softmaxfunction is mainly used in the classification task.122.2. DEEP LEARNINGFigure 2.7: The schematic structure of the LSTM cell [Ola15].LSTM, a variant of RNN, is embedded with gated units. This designallows LSTMs to hold long-term input data, addressing the limitation oftraditional RNNs that mainly retain short-term memory from a single pre-vious time step [Bur19, LBH15]. Furthermore, LSTMs can decide whetherto retain or discard information from previous inputs through these gatedunits [Ola15]. A schematic representation of an LSTM cell is provided inFigure 2.7. The initial gate in the LSTM cell is termed the “forget gate”,which is mathematically represented asft = σ(Wf · [yt−1, xt] + bf ) (2.8)where Wf and bf are the parameters of this gate that will be obtained bytraining. xt is the input data vector, and yt−1 is the output vector from theprevious LSTM cell, also called the hidden state vector. σ is the sigmoidactivation function, which equation is shown asσ(z) =11 + e−z(2.9)where z is the input vector of the sigmoid activation function.132.2. DEEP LEARNINGThe output range of the sigmoid function is between 0 and 1. Thus,the output of this gate decides how much information from the previous cellstate Ct−1 will be forgotten [Ola15].Then, the input gate will update the state vector of the current cellCt. We introduce two new intermediate parameters it and c′t that are theoutput values from the sigmoid and tanh activation functions, respectively,to represent this gate. These two parameters can be computed asit = σ(Wi · [yt−1, xt] + bi) (2.10)c′t = tanh(Wc · [yt−1, xt] + bc). (2.11)Since the new state vector Ct is the sum of the old state vector and updatedstate vector computed by the forgot gate and input gate respectively. Theformula of the new state vector is shown asCt = ft × Ct−1 + it × c′t. (2.12)Finally, the output of the LSTM cell yt (also called hidden state vector) iscomputed by the output gate based on the updated cell state vector Ct andinput vector xt, whose formulas are shown asot = σ(Wo · [yt−1, xt] + bo) (2.13)yt = ot × tanh(Ct) (2.14)where ot is the output of the sigmoid function for the linear combination ofthe previous output vector yt−1 and input vector xt with the weight matrixof output gate Wo. Then, the ot is multiplied by the output of the statevector Ct in the tanh function to obtain the output vector yt [Ola15].The LSTM layer retains the chained architecture characteristic of RNNlayers, as depicted in Figure 2.6. Every input vector in the sequence isprocessed by an LSTM cell containing N units within a layer. Here, Ndenotes the dimension of both the cell state vector and the hidden statevector.RNNs and their variants like LSTM have been explored for marine mam-mal sound classification since the spectrogram features from audio signalsbelong to time series sequential data [MSK+21, MV21]. However, in somestudies, CNNs have been shown to outperform LSTM [MV21].142.3. BACKGROUND SUMMARY2.3 Background SummaryThis chapter comprehensively reviews the spectrogram audio feature,traditional machine-learning algorithms, and current deep-learning modelsapplied in marine mammal sound classification. Although current solutionsbased on deep-learning approach have enhanced the classification accuracyof marine mammal vocalizations compared to traditional machine-learningalgorithms, there is still much room for improvement in the classificationaccuracy and capability in classifying categories encompassing a large scaleof marine mammal species.15Chapter 3MethodologyThis chapter provides the methodologies utilized for this research, asillustrated in Figure 3.1. The proposed framework includes three studies:1. Analyzing the Effect of Spectrogram Size on the CNNModel:This research delves into understanding how modifications in the spec-trogram dimensions can influence the classification accuracy of theCNN model.2. Performance Comparison of Deep-Learning Models: The pri-mary focus of this study is to benchmark and assess the efficacy ofvarious deep-learning models in classifying marine mammal vocaliza-tions.3. Evaluating the Generalization Capability of the CNN Model:The objective here is to utilize the generalization ability of the CNNmodel to improve the classification accuracy when confronting acousticdata from additional data sources or species.163.1. OVERVIEW OF DATASETSFigure 3.1: The overview of the proposed framework.In essence, each research follows three core procedures: Data pre-processing,Model training, and Model evaluation.3.1 Overview of DatasetsOur research employs three datasets: the “Best of cuts”, partial of “Allcuts”, and the Mobysound dataset. All sound clips in these datasets aresaved in the waveform audio file format. The first two datasets are from theWatkins Marine Mammal Sound Database, which is introduced as follows.Watkins Marine Mammal Sound Database: This database hasabout 2,000 unique recordings of more than 60 marine mammal species fromthe 1940s to the 2000s, which has three sections: “Best of cuts”, “All cuts”,and “Master tapes”. The “Best of cuts” includes 1694 audio files for 32species with lower noise than other sections. There are about 15,000 audiofiles for 51 species in “All cuts”, including the audio files from the “Best ofcuts” section. Meanwhile, the “Master tapes” section consists of approxi-mately 1,600 tapes that record marine mammal sound [SDA+16, LHY21].The “Best of cuts” and partial of “All cuts” with the same 32 species cate-173.2. DATA PRE-PROCESSINGgories in “Best of cuts” are used in this research. Unless otherwise specific,the “All cuts” dataset represents the partial of “All cuts” with the 32 speciescategories in this thesis. The original sample rate for “Best of cuts” is illus-trated in Table 3.1.MobySound Dataset: This collection contains sound clips of 10 ma-rine mammal species, including nine whale species and the leopard sealrecorded between 1988 and 1999 [MC06]. The audio files’ initial samplerates range from 100 to 8,000 Hz. Notably, compared to the 32 species inthe “Best of cuts” from the Watkins Marine Mammal Sound Database, theMobySound dataset has vocalizations from unique species such as the BlueWhale, Bryde’s Whale, and Gray Whale. The vocalization data from theseadditional species would be utilized to evaluate the generalization capabilityof the CNN model.3.2 Data Pre-processingData pre-processing encompasses four pivotal stages: data cleaning, re-sampling rate selection, spectrogram generation and slicing, and the subse-quent spectrogram saving and further processing.3.2.1 Data CleaningDuring the initial data examination, repeated audio records are identi-fied across various species classes in the “Best of cuts” and “All cuts” in theWatkins Marine Mammal Sound Database. Such repetitions can substan-tially hinder classification accuracy. These repeated clips were duly removedbefore further preprocessing. Table 3.1 provides details like the duration ofthe total sound clips for the original and post-cleaning, the initial samplingrate (The unit of frequency is in kilohertz (kHz)), and the peak frequencyof sound clips for each species in “Best of cuts”. This data cleaning pro-cess is also implemented in “All cuts”. The audio files containing soundsfrom multiple marine mammal species are also cleaned in the MobySounddatabase.183.2. DATA PRE-PROCESSINGTable 3.1: The information about sound clips for each species from “Best ofcuts”.Species Duration (s)(Original)Duration (s)(AfterCleaning)MaxFrequency(kHz)OriginalSampleRate(kHz)SpermWhale5561.65 4257.69 16.74 40-160Narwhal 81.32 81.32 25.97 60.60-81.92PantropicalSpotted Dol-phin118.12 118.12 31.48 40.96-148.10FinbackWhale2476.83 2476.83 0.15 0.60-0.64HumpbackWhale2920.12 2920.12 1.64 5.12-14.90CommonDolphin343.73 343.73 10.42 43.9-81.92Long-FinnedPilot Whale527.97 527.97 10.91 30.00-81.92Weddell Seal 42.26 42.26 5.80 20.48-43.90Ross Seal 162.06 162.06 5.03 16.00-30.00Harp Seal 134.51 134.51 2.57 10.24-43.90SouthernRight Whale46.68 46.68 1.35 5.12BottlenoseDolphin29.80 29.80 24.73 30.00-81.92Rough-ToothedDolphin65.04 65.04 26.45 81.92Short-Finned PilotWhale1768.71 464.74 4.42 30.00-80.00Spinner Dol-phin317.62 317.62 23.50 60.60-81.92Leopard Seal 46.27 46.27 0.74 5.00Continued on next page193.2. DATA PRE-PROCESSINGTable 3.1 – Continued from Previous PageSpecies Duration (s)(Original)Duration (s)(AfterCleaning)MaxFrequency(kHz)OriginalSampleRate(kHz)NorthernRight Whale118.92 118.92 2.36 5.12-20.48Fraser’s Dol-phin507.58 507.58 22.61 60.60-81.92Risso’s Dol-phin172.14 172.14 14.16 30.00-81.92False KillerWhale68.31 68.31 22.64 30.00-81.92Striped Dol-phin143.75 143.75 20.57 60.60-166.6White-sidedDolphin183.04 183.04 28.75 60.60-166.60White-beakedDolphin154.80 154.80 16.81 60.60-80.00Warlus 184.28 184.28 7.07 5.12-40.96MinkeWhale20.04 20.04 0.28 1.28ClymentDolphin162.29 162.29 34.30 41.60-166.60BowheadWhale449.09 229.07 2.71 10.00-10.24Beluga 131.33 131.33 8.33 10.24-81.92AtlanticSpotted Dol-phin153.78 153.78 22.30 43.90-80Melon-headedWhale155.29 155.29 17.21 60.60-81.92Bearded Seal 354.20 134.18 3.07 5.12-30.00203.2. DATA PRE-PROCESSING3.2.2 Resampling Rate SelectionA common sample rate to resample audio from the original audio filesis required to process those audio files in digital devices. This uniformityis particularly critical when deploying a sound classification system in realoceanic environments and in real-time scenarios. In such applications, it’simpractical to work with multiple sample rates. Instead, a single, predeter-mined sample rate must be selected. The Watkins Marine Mammal SoundDatabase, sourced from different devices, inherently features a variety oforiginal sample rates. Table 3.1 shows the range of these rates for eachspecies. The highest identified rate within the “Best of cuts” is 160 kHz.While this might theoretically serve as the ideal resampling rate to retain allthe information from audio files, practical implications suggest otherwise.Figure 3.2 provides spectrograms for Clymene dolphins at two differentrates: 22.05 kHz and 160 kHz. 22.05 kHz is the default sample rate in the“librosa” python audio processing library [MRL+15]. The spectrogram cor-responding to the default sample rate, as observed in Figure 3.2(a), resultsin a loss of high-frequency information beyond approximately 10 kHz. Con-versely, using the 160 kHz rate, as depicted in Figure 3.2(b), though thefeature pattern in this spectrogram is significant, nearly half of the spectro-gram appears blank since there is no valid information for the more than 40kHz frequency from the audio file. This disparity arises due to the differ-ence between half of the resampling rate and the highest sound frequency ofcertain audio files, typically less than half of the original sample rate basedon the Nyquist theorem [Sha49]. We want to focus on the frequency rangewhere there is significant spectral content.213.2. DATA PRE-PROCESSING(a) Clymene dolphin spectrogramwith 22.05kHz sample rate.(b) Clymene dolphin spectrogramwith 160kHz sample rate.Figure 3.2: Clymene dolphin spectrogram with 22.05kHz and 160kHz samplerate.A lower resampling rate is demanded to solve that problem. However,this value could not be excessively diminished in order to acquire a sufficientamount of information from the audio signal. An algorithm to obtain thisbalanced resampling rate is proposed based on signal bandwidth. As shownin Algorithm 1, the frequency of an audio signal encapsulating 99% of totalsignal power can be computed.First, after loading each audio file with the initial sample rate, the powerspectral density (PSD) of each audio file, which presents the audio signalpower versus frequency, is computed by Welch’s method [Wel67]. The re-sults of PSD are two lists of data P and F that are signal power and re-spected frequency respectively. After that, the total power of the audiosignal P sum is computed by adding all the elements in the list P , and theencapsulating power P en is calculated by 0.99×P sum. This value can beadjusted if this algorithm is employed in other datasets to obtain a balancedresampling rate while controlling the loss of information from the signals.The accumulated power P temp and the resulting frequency F E are setto zero initially. Then, the signal power element versus the frequency fromlist P is accumulated from the lowest frequency to the highest frequencyuntil the accumulated power P temp is not smaller than the encapsulatingpower P en. The previous frequency related to the power signal elementis returned as the resulting frequency F E when the current accumulatedsignal power P en is larger than the encapsulating power P en.The maximum frequency of each species could be obtained after usingAlgorithm 1 in all the audio files and comparing the frequency for each audio223.2. DATA PRE-PROCESSINGAlgorithm 1: Find the signal bandwidth defined as the power ofsignal equals to 99 percent of total signal power.Input: Audio FileOutput: F ELoad the audio file and obtain the power spectral density.P ← List of Signal PowerF ← List of Respected FrequencyP sum← The Sum of All the Elements in List PP en← 0.99× P sumP temp← 0F E ← 0for i from 0 to (The Length of List P − 1) doif P temp < P en thenP temp = P temp+ P [i]elseF E = F [i− 1]BreakendendReturn F Efile from one species. After comparing, we obtain the maximum frequency ofeach species, which is presented in Table 3.1, indicating that the maximumfrequency for all categories is below 50 kHz from the “Best of cuts”. Based onthe Nyquist theorem, the minimum sample rate must be twice the maximumfrequency of the signal and a resampling rate of 100 kHz could be utilizedas the trade-off resampling rate to make sure that at least 99 % of the signalpower is contained for each audio signal used in this study [Sha49]. Although100 kHz is still very much larger than needed for many species, according toour result, the classification report in Table 4.11, the classification results forFinback and Minke whales possess exceptionally low original sample ratesof 0.6 and 1.28 kHz are satisfactory.The resampling rate of 100 kHz was also implemented to the “All cuts”dataset since the original sample rate and maximum frequency of eachspecies in sound clips from “All cuts” are similar to “Best of cuts”. Asfor the Mobysound dataset, the initial sample range is from 100 to 8,000Hz, which is not as wide as the “Best of cuts” from 0.6 to 160 kHz. Thus,the highest initial sample rate, 8,000 Hz, is used as the common resampling233.2. DATA PRE-PROCESSINGrate.3.2.3 Spectrogram Generating and SlicingThe specific parameters used for generating and slicing spectrogramstailored to different deep-learning models are presented in this step. Thegeneration process of the spectrogram is already introduced in Chapter 2.Figure 3.3 provides the rest process of the spectrogram slicing, saving andfurther processing in the last two steps.Most of the parameters used in spectrogram generation from the priorwork are adopted in our research [MIHA22], including a ‘Hann’ windowtype and a window length of 1024 for STFT-spectrogram generation. Post-generation, the spectrogram is sliced into 256 time frame segments, sincethese parameters proposed by that work upgraded the benchmark of “Bestof cuts” by boosting the F1-score of ResNet50 to 86.7% [MIHA22]. How-ever, the parameter of hop length is modified to decrease the overlappingrate to 75%. This is because the high overlapping rate increases the train-ing time of the deep-learning model significantly [WWB+22, MIHA22], wewant to save training time since many models need to be trained in our re-search. While the window length 1024 is applied, the produced spectrogramhas 513 frequency bins according to the formula between window lengthand frequency bins that is explained in Chapter 2. Thus, the spectrogramdimension 513× 256 is used as the baseline spectrogram dimension.The larger spectrogram sizes like 1025× 256 or 513× 512 generated bywindow length of 2048 or sliced by time segmentation of 512 respectivelyare utilized to investigate the potential correlation between the spectrogramsize and the classification accuracy of the CNN models. Previous researchposits that an increase in spectrogram size may correlate with enhancedperformance of CNN models [MIHA22]. This hypothesis is grounded in theunderstanding that larger-dimension spectrograms provide higher frequencyresolution and more extensive time information, potentially offering a moredetailed representation of audio signals for classification. Such an increase inresolution might allow CNNs to capture and learn from more subtle featureswithin the audio data, possibly leading to improved model accuracy.The “SpecAugment” data augmentation algorithm is initially designedfor speech recognition by introducing random frequency and time maskingon spectrograms [PCZ+19]. A comparison between an original spectrogramand its “SpecAugment” version from a sound clip of the Clymene dolphin isshowed in Figure 3.4(a) and Figure 3.4(b). This augmentation algorithm hasbeen demonstrated to improve the classification accuracy of deep-learning243.2. DATA PRE-PROCESSINGFigure 3.3: The process of the spectrogram slicing, saving and further pro-cessing.253.2. DATA PRE-PROCESSINGmodels in classifying animal vocalizations or ocean noise containing marinemammal sounds by previous experiments [Pel21, MV21]. This technologyis applied to the spectrogram to examine its potential in enhancing CNNs’generalization capacity for diverse marine mammal sound sources.(a) Original spectrogram (b) Spectrogram with “SpecAug-ment”Figure 3.4: Comparison of original and “SpecAugment” spectrograms froma sound clip of the Clymene dolphin.Regarding the LSTM model, the same spectrogram that trains the CNNmodels is employed, and the original size of the spectrogram is kept. How-ever, specialized models such as VGGish and AST have specific spectrogramsize requirements in the number of frequency bins and time frames. For in-stance, VGGish requires 64 frequency bins and 96 time frames, whereas ASTprefers 128 frequency bins. These dimensions are adjusted before spectro-gram generation, ensuring compatibility with each model’s requirements.3.2.4 Spectrogram Saving and Further ProcessingPost-generation spectrograms are stored as grayscale images and un-dergo additional modifications tailored to individual models and researchrequirements.For ANNs, the two-dimensional spectrogram is flattened into a one-dimensional vector suitable for the input layer. In the case of CNN models,spectrograms are typically resized to 224×224×1 pixels since the preferredinput size is 224× 224 for most CNN models deployed in this research. Thedefault pixel size of the spectrogram is the same as the number of frequencybins and time frames. This resizing procedure scales the spectrogram im-age at the pixel level without affecting the number of frequency bins and263.3. DEEP-LEARNING MODELS DEPLOYMENT AND TRAININGtime frames in the spectrogram. However, the original spectrogram sizeis maintained on certain occasions to investigate if the resizing procedurepotentially impacts CNN model performance.For models like LSTM, VGGish, and AST, the spectrograms with specificdimensions that are explained previously are transposed to meet the models’input specifications, where the first dimension represents time frames, andthe second encapsulates frequency bins.3.3 Deep-Learning Models Deployment andTrainingThis section first presents the training environment, k-fold cross-validation,model performance evaluation, and transfer learning methods are utilized inmodel training. Then, four types of deep-learning models deployed in thisresearch are introduced.3.3.1 Training EnvironmentThe deep-learning model training is implemented through the Pytorchlibrary on an NVIDIA A6000 GPU with 48G memory. Specifically, theANN, CNN and LSTM models are trained by the Adam optimizer with aninitial learning rate of 10−4 and a learning rate decay of 0.1 if the loss is notdecreased in 5 epochs with a weight decay of 10−5, while the AST is trainedby the same optimizer with same learning rate but a weight decay of 5×10−7and beta one of 0.95 [HSC+19, GCG21]. The early stopping mechanism isutilized to save computational resources, terminating the training process ifthe loss is not reduced in ten consecutive epochs.3.3.2 The Stratified K-Fold Cross-ValidationStratified k-fold cross-validation segments the dataset into k subsets or“folds”. Each fold maintains the same ratio of data across every class. Intotal, k models are trained: each trained on k-1 folds and assessed on theremaining fold. Thus, the results from k iterations present a comprehensiveevaluation of the model’s performance [Ary22]. The stratified 5-fold cross-validation method is employed in this research, as represented in Figure 3.5showcasing the 5-fold data segmentation.273.4. MODEL PERFORMANCE EVALUATIONFigure 3.5: The schematic figure for the stratified 5-fold cross-validation[CZZY22].3.4 Model Performance EvaluationFour metrics are utilized to evaluate the performance of deep-learningmodels for each class: Accuracy, Recall, Precision, and F1-score.− Accuracy: Proportion of correctly classified samples relative to allsamples.− Recall: Proportion of true positive samples among all actual positivesamples.− Precision: Proportion of true positive samples among all predictedpositive samples.− F1-score: Harmonic mean of precision and recall.283.4. MODEL PERFORMANCE EVALUATIONThe formulas for each metric areAccuracy =TP + TNTP + FP + TN + FN(2)Recall =TPTP + FN(3)Precision =TPTP + FP(4)F1− score = 2× Precision×RecallPrecision+Recall=TPTP + 12(FP + FN)(5)where True Positive (TP), True Negative (TN), False Positive (FP), andFalse Negative (FN) are values derived from the confusion matrix, a toolvisualizing the classification performance of the model. Figure 3.6 shows anexample of a confusion matrix [KCB20, BBG20, CZZY22, LHY21].Figure 3.6: The example of a confusion matrix [BBG20].Since the stratified 5-fold cross-validation is implemented in model train-ing, the model’s performance is evaluated by the mean value of accuracy,recall, precision, and F1-score for these five tests [PT12, MIHA22].293.4. MODEL PERFORMANCE EVALUATION3.4.1 Transfer LearningTransfer learning enhances the learning trajectory by transferring knowl-edge from a relevant task when learning a new one [OGMS+09]. This methodenhances machine-learning models’ efficacy and diminishes the time anddata needed for training [ZQD+20]. The application of transfer learningis training the deep-learning model from the initial pre-trained weight tosuit the new task. If no layers within the model are frozen, it can be fullyadapted to suit the new task, which is a process often termed “fine-tuning”.This research trains deep-learning models like CNN and AST throughtransfer learning. Specifically, these models are fine-tuned to classify marinemammal vocalizations from specific pre-trained weights.Transfer learning also plays a pivotal role in the third study. Since CNNclassifiers pre-trained on the “Best of cuts” dataset are fine-tuned to classifythe “All cuts” dataset, and classifiers pre-trained by the Watkins MarineMammal Sound Database are further fine-tuned to classify the MobySounddataset in this research.3.4.2 Artificial Neural Network (ANN)The ANN used in this work features a single hidden layer with 512neurons. This ANN serves as a baseline for comparing the performanceof other deep learning models, being the simplest deep model deployed formarine mammal sound classification.3.4.3 Long Short-Term Memory Network (LSTM)The LSTM models, featuring one, two, three and four LSTM layers withhidden sizes of 1024, 2048, or 4096, are used to compare the other differentdeep models. The output layer of the LSTM model is a 32-neuron fully-connected layer designed for predictions across 32 marine mammal species.3.4.4 Convolutional Neural Network (CNN)Researchers have developed many deep-learning models based on CNNarchitecture in recent years. For our experiment, eight advanced CNN ar-chitectures are utilized for marine mammal sound classification:− Notable architectures like ResNet50 and EfficientNetB0, previouslyemployed in related works.303.4. MODEL PERFORMANCE EVALUATION− Their upgraded counterparts, ResNeXt50 and EfficientNetV2 Smallmodel.− Given the potential need for real-time audio classification, this re-search trains and evaluates lightweight CNN architectures such asShuffleNetV2 2.0, MobileNetV3 Large, and GhostNet. These architec-tures offer the advantage of easy deployment on mobile devices with-out requiring significant computational resources. All these CNNs arepre-trained using “ImageNet1K” weights.− The VGGish model, a CNN specializing in audio classification, comespre-trained with “AudioSet” weights.3.4.5 Audio Spectrogram Transformer (AST)The AST is based on the Vision Transformer (ViT) [GCG21]. The Trans-formers are a type of deep-learning model based on the attention mechanisminitially for language translation [VSP+17]. Its application has extended toimage processing since the architecture of ViT was proposed [DBK+20].Figure 3.7 illustrates the overview architecture of the ViT and the layerstructure of the transformer encoder.Figure 3.7: The overview architecture of the ViT and the layer structure ofthe transformer encoder [DBK+20].313.4. MODEL PERFORMANCE EVALUATIONThe ViT divides the input into fixed-size patches upon receiving an im-age input. These patches are subsequently transformed into one-dimensionalsequences and projected to match the vector size required by the transformerencoder, resulting in patch embeddings. Positional data is then added tothese embeddings. After positional embedding, the class embedding is addedto the beginning of the embedded sequence, and then this embedded se-quence combining those embeddings flows into the transformer encoder. Fi-nally, the output vectors from the transformer encoder are fed to MLP andconverted to the predicted class [DBK+20, VSP+17].The transformer encoder has multiple equivalent layers. The layer struc-ture is also shown in Figure 3.7. There are two sub-layers in each layer.The first one has the normalization layer and the multi-head self-attentionblock. The second one is the fully-connected layer with the normalizationlayer. The residual connection is applied to each sub-layer [VSP+17].Figure 3.8: The overview of attention mechanism [VSP+17].The multi-head attention block is the crucial component of the Trans-former, whose inputs are query (Q), key (K), and value (V ). The structureof the multi-head attention block is shown in Figure 3.8. First, the dimen-sions of the normalized embedded sequence are augmented to fit the multi-head structure. Figure 3.9 illustrates the transformation of the Q matrix inthis process. The K and V matrices undergo a similar process. Then, thesematrices are dot products between the input embedded sequence and theinitial weighting matrices for Q, K, and V in each linear layer respectively.323.4. MODEL PERFORMANCE EVALUATIONAfter individual transformations, the Q, K, and V matrices engage with thescaled dot-product attention mechanism. The mathematical representationof this block is shown asAttention(Q,K, V ) = softmax(Q ·KT√Query Size)× V (3.1)where the first step is the dot-product between matrix Q and matrix K[VSP+17]. After that, the scale factor, the square root of the size of thequery, is divided by this result, and the softmax function is implemented.The output of the softmax function is multiplied by the V matrix to obtainthe attention score in one head, and then the attention scores from differentheads are merged in the head’s dimension and projected in a linear layer.The final output is obtained from the multi-head attention block throughthese steps [Dos21, VSP+17]. The mathematical representation of the multi-head attention block can be shown asMulti-head(Q,K, V ) = Concat(H1, H2, ...,Hk) ·W o k = 0, ..., N (3.2)Hi = Attention(Q ·WQi ,K ·WKi , V ·W Vi ) (3.3)where Hi is the attention score from the scaled dot-product attention forthe ith head; W o is the weight matrix in the output linear layer of the multi-head attention block; N is the number of heads; WQi , WKi , and WVi arethe weight matrices for Q, K, and V in each linear layer respectively at thebeginning in the ith head [GG20].Figure 3.9: The size change of query matrix [GG20].333.4. MODEL PERFORMANCE EVALUATIONAST in architecture is close to the ViT but has a distinct input: aspectrogram with specific 128 frequency bins. This spectrogram is dividedinto 16 × 16 patches, and each patch overlaps for six units in both timeand frequency domains [GCG21]. Then, each patch is transformed into a768-dimensional vector in the linear projection layer. After appending a 768-dimensional positional embedding and adding class embedding at the firstof the sequence, the result sequence combining those embeddings is fed intoa transformer encoder with 12 layers. Each multi-head self-attention blockwithin these layers has 12 heads. Finally, a fully-connected layer, activatedby a sigmoid function, turns the output of this transformer encoder intoprediction labels [GCG21].The AST is pre-trained on both “ImageNet1K” and “AudioSet”. Ob-servations indicate that AST does not outperform CNNs when trained onsmaller datasets [DBK+20]. Hence, the AST model undergoes fine-tuningfor this task using its pre-trained weights to classify the marine mammals’spectrograms.34Chapter 4Results and DiscussionThis chapter presents the results of this research and discusses the out-comes compared to the previous studies.4.1 Analyzing the Effect of Spectrogram Size onthe CNN ModelAn evaluation was conducted on the performance of the ResNet50 modelwith different spectrogram input sizes: 513×256, 513×512 and 1025×256.These were sourced from the “Best of cuts” dataset. Table 4.2 shows theResNet50 model’s performance with original spectrogram sizes, while Table4.3 highlights results after resizing the input spectrogram to 224×224 pixels.Table 4.2: The effect of the spectrogram size on ResNet50 model (originalspectrogram).Input Size Accuracy Precision Recall F1-score513× 256 99.85% 99.86% 99.85% 99.85%513× 512 99.79% 99.80% 99.79% 99.78%1025× 256 99.55% 99.57% 99.55% 99.55%Table 4.3: The effect of the spectrogram size on ResNet50 model (resizedspectrogram 224× 224).Input Size Accuracy Precision Recall F1-score513× 256 99.84% 99.85% 99.84% 99.84%513× 512 99.76% 99.77% 99.76% 99.76%1025× 256 99.62% 99.64% 99.62% 99.62%The results in Tables 4.2 and 4.3 indicate a decline in the ResNet50model’s performance with an increase in the spectrogram dimensions interms of frequency bins or time frames. This trend is observed with both354.2. PERFORMANCE COMPARISON OF DEEP-LEARNING MODELSoriginal and resized spectrogram dimensions. The optimal 513×256 spectro-gram size was then chosen for training other CNNs in further experiments.Notably, the performance difference between models trained on original-sized spectrograms and spectrograms that resized to 224 × 224 pixels arefairly small, under 0.1%. Leveraging the resizing procedure conserves GPUmemory and proves nearly as effective as using the original sizes.Table 4.4: The effect of the spectrogram size on ResNeXt50 model (resizedspectrogram).Input Size Accuracy Precision Recall F1-score513× 256 99.90% 99.90% 99.90% 99.90%513× 512 99.76% 99.77% 99.76% 99.76%1025× 256 99.81% 99.82% 99.81% 99.81%Table 4.5: The effect of the spectrogram size on EfficientNetB0 model (re-sized spectrogram).Input Size Accuracy Precision Recall F1-score513× 256 99.85% 99.85% 99.85% 99.85%513× 512 99.70% 99.71% 99.70% 99.70%1025× 256 99.71% 99.72% 99.71% 99.71%The ResNeXt50 and EfficientNetB0 models are also trained by varyingspectrogram dimensions. However, those spectrograms are resized to 224×224 in pixels to ensure computational efficiency. As showcased in Tables 4.4and 4.5, the 513 × 256 spectrogram dimensions consistently provided thebest results for both models.4.2 Performance Comparison of Deep-LearningModelsTable 4.6 presents the average cross-validation results for the “Best ofcuts” dataset across various models, including the ANN baseline, LSTMs,CNNs, and AST.The differences among the four metrics for all models are negligible, be-ing less than 0.1% for most results. Therefore, accuracy is selected as theprimary metric for ease of comparison. However, the exceptionally high364.2. PERFORMANCE COMPARISON OF DEEP-LEARNING MODELSTable 4.6: The mean cross-validation results of each model on “Best of cuts”.Model Parameters Accuracy Precision Recall F1-scoreBaseline ANN 68M 97.24% 97.25% 97.24% 97.19%1-layer (1024) LSTM 6M 98.45% 98.46% 98.45% 98.44%1-layer (2048) LSTM 21M 98.49% 98.50% 98.49% 98.48%1-layer (4096) LSTM 76M 98.54% 98.55% 98.54% 98.53%2-layer (1024) LSTM 21M 98.67% 98.67% 98.67% 98.65%2-layer (4096) LSTM 210M 98.88% 98.90% 98.88% 98.87%2-layer (2048) LSTM 55M 98.95% 98.94% 98.94% 98.93%3-layer (2048) LSTM 88M 98.76% 98.77% 98.76% 98.75%4-layer (2048) LSTM 122M 98.67% 98.68% 98.67% 98.66%ResNet50 24M 99.84% 99.85% 99.84% 99.84%ResNeXt50 23M 99.90% 99.90% 99.90% 99.90%EfficientNetB0 4M 99.85% 99.85% 99.85% 99.85%EfficientNetV2 Small 20M 99.83% 99.83% 99.83% 99.83%MobileNetV3 Large 4M 99.24% 99.27% 99.24% 99.22%ShuffleNetV2 2.0 5M 99.69% 99.70% 99.69% 99.69%GhostNet 4M 99.76% 99.76% 99.76% 99.76%VGGish 55M 99.86% 99.86% 99.86% 99.86%AST 87M 99.80% 99.80% 99.80% 99.80%374.2. PERFORMANCE COMPARISON OF DEEP-LEARNING MODELSaccuracy noted for each model warrants further investigation. We hypothe-size that these unusually elevated results may stem from our data-cleaningprocess. To verify this, a supplementary experiment utilizing the original’Best of cuts’ dataset will be conducted in the subsequent section. It is alsoimportant to note that the potential for overfitting cannot be definitivelyruled out due to the lack of additional data for testing the deep-learningmodels. This concern will be further discussed in the sections on limitationsand future research.Turning our attention back to the results, a comparison of various deep-learning models reveals the following ranking in terms of accuracy, fromhighest to lowest, from highest to lowest: CNNs and AST lead, followed byLSTM, and finally ANN. Among the CNN and AST models, ResNeXt50is the top performer. The VGGish model comes in second, outperformingthe ResNet50. Although AST achieves a respectable accuracy of 99.80%in classifying marine mammal vocalizations, it is outdone by several CNNarchitectures. Notably, it only performed better than the CNN models de-signed for mobile applications, such as MobileNetV3 Large, ShuffleNetV2,and GhostNet. Among these lightweight CNNs, GhostNet stands out withan accuracy of 99.76%. Figure 4.1 illustrates the changes in validation accu-racy and loss for the ResNeXt50, AST, and GhostNet throughout the 5-foldcross-validation training.384.2. PERFORMANCE COMPARISON OF DEEP-LEARNING MODELS(a) The valid accuracy change for theResNeXt50.(b) The valid loss change for theResNeXt50.(c) The valid accuracy change for theAST.(d) The valid loss change for TheAST.(e) The valid accuracy change for theGhostNet.(f) The valid loss change for theGhostNet.Figure 4.1: The valid accuracy and loss change for ResNeXt50, AST, andGhostNet during the 5-fold cross-validation training in the “Best of cuts”.394.3. EVALUATING THE GENERALIZATION CAPABILITY OF THE CNN MODELThe LSTM models, though designed to handle sequences, are under-performing compared to the CNNs and AST on this dataset. The 2-layerLSTM with 2048 hidden units was the best-performing LSTMmodel, achiev-ing an accuracy of 98.95%.Subsequently, ResNeXt50 and GhostNet are trained on a more exten-sive, 32-class “All cuts” dataset to evaluate their performance further. Thisdataset experienced the same data pre-processing procedures and resam-pling rate as the “Best of Cuts” dataset. The results are shown in Table4.7.Table 4.7: The mean cross-validation results of each model on “All cuts”.Model Accuracy Precision Recall F1-scoreResNeXt50 99.50% 99.50% 99.50% 99.50%GhostNet 99.42% 99.43% 99.42% 99.42%The ResNeXt50 and GhostNet demonstrate impressive classification ac-curacy in this dataset, achieving 99.50% and 99.42% respectively.4.3 Evaluating the Generalization Capability ofthe CNN ModelThis research evaluates the convolutional layers’ generalization capabil-ities of the CNN-based model and examines if implementing data augmen-tation in training can improve this generalization ability. By utilizing thisgeneralization capability, a higher classification accuracy could be achievedwhen classifying the new acoustic data from different sources or additionalspecies.For the purposes of potential deployment in environments with con-strained computational resources, the convolutional layers of the top-performingResNext50 model pre-trained on “Best of cuts” were frozen [TMK+20]. Onlythe last two dense layers were re-trained using additional data from the “Allcuts” dataset, following the same preprocessing steps. As the “All cuts”dataset lacks additional data for the harp seal, the new classifier output is31 classes.The performance of this classifier that the convolutional layers are frozenand pre-trained by “Best of cuts” (Condition is called “Frozen & Best ofcuts” in Table 4.8.) is compared with the result of the ResNeXt50 model pre-trained by “Best of cuts” without freezing the convolutional layers (Condi-404.3. EVALUATING THE GENERALIZATION CAPABILITY OF THE CNN MODELtion is “Not Frozen & Best of cuts”.). Besides, those outcomes are comparedwith the performance of the model pre-trained by the augmented datasetand freezing all the convolutional layers (Condition is “ Frozen & Augmented”.) to assess whether the “SpecAugment” data augmentation method couldenhance the generalization ability of CNN. Finally, the ResNeXt50 modelwith the initial “ImageNet1K” weight (Condition is “Not Frozen & Ima-geNet1K”.) is trained to test the impact of the “Best of cuts” pre-trainedweight on the model’s performance with “All cuts”.Table 4.8: The mean cross-validation results of generalization evaluation inResNeXt50.Condition Accuracy Precision Recall F1-score“Frozen & Best of cuts” 97.14% 97.15% 97.14% 97.13%“Frozen & Augmented” 95.92% 95.96% 95.92% 95.90%“Not Frozen & Best of cuts” 98.88% 98.89% 98.88% 98.88%“Not Frozen & ImageNet1K” 98.82% 98.83% 98.82% 98.82%From Table 4.8, the comparison between conditions “Frozen & Best ofcuts” and “Not Frozen & Best of cuts” shows that freezing the convolu-tional layers before training decreases the performance by a maximum of1.74% compared to the model with trainable convolutional layers. Thisgap is acceptable and demonstrates that the CNN model has the capabil-ity to generalize to classify marine mammal sounds from additional sources.However, implementing the “SpecAugment” data augmentation during pre-training diminishes this capability since the classifier performance under“Frozen & Augmented” is worse than the “Frozen & Best of cuts” conditionfrom the above table. Furthermore, a ResNeXt50 with non-frozen layersand pre-trained on “Best of cuts” slightly outperforms one with initial “Im-ageNet1K” weights. This suggests that pre-training on varying acousticsources can enhance the classification accuracy of the CNN model whenfine-tuned to classify unseen acoustic sources.Given the positive influence of the “Best of cuts” pre-training on perfor-mance using the “All cuts” dataset, we aimed to discern if a CNN pre-trainedon the Watkins Marine Mammal Sound Database would similarly boost clas-sification performance on a new acoustic dataset with new data source anddata from new species. Thus, the MobySound dataset with three kinds ofadditional species is used to train CNN models further. The ResNeXt50 andGhostNet models from different pre-trained weights are trained to classifythe MobySound dataset, as shown in Table 4.9.414.3. EVALUATING THE GENERALIZATION CAPABILITY OF THE CNN MODELTable 4.9: The mean cross-validation results of the ResNeXt50 and Ghost-Net models in the MobySound dataset.Model Weight Accuracy Precision Recall F1-scoreResNeXt50 “ImageNet1K” 99.40% 99.40% 99.40% 99.40%ResNeXt50 “Best of cuts” 99.31% 99.31% 99.31% 99.31%ResNeXt50 “All cuts” 99.49% 99.50% 99.49% 99.49%GhostNet “ImageNet1K” 99.34% 99.35% 99.34% 99.33%GhostNet “Best of cuts” 99.61% 99.61% 99.61% 99.61%GhostNet “All cuts” 99.62% 99.62% 99.62% 99.62%As observed from Table 4.9, both ResNeXt50 and GhostNet models pre-trained on “All cuts” show the enhanced performance on the MobySounddataset compared to those pre-trained on “ImageNet1K”. GhostNet’s im-provement is especially notable compared to the ResNeXt. Besides, the“Best of cuts” pre-trained weights enhance GhostNet’s performance signif-icantly but do not positively impact ResNeXt. These results show thatthe generalization ability of GhostNet is superior compared to ResNeXt50.Figure 4.2 depicts the validation accuracy change for ResNext and Ghost-Net with various pre-trained weights during 5-fold cross-validation train-ing, highlighting that models pre-trained with “All cuts” weights requirefewer epochs to complete the training process, indicating the “All cuts” pre-trained weights accelerate the training speed of the classifier training by theMobySound dataset.424.3. EVALUATING THE GENERALIZATION CAPABILITY OF THE CNN MODEL(a) The valid accuracy change forthe ResNeXt50 pre-trained by “Im-ageNet1K”.(b) The valid accuracy change forThe GhostNet pre-trained by “Ima-geNet1K”.(c) The valid accuracy change for theResNeXt50 pre-trained by “Best ofcuts”.(d) The valid accuracy change forthe GhostNet pre-trained by “Best ofcuts”.(e) The valid accuracy change forthe ResNeXt50 pre-trained by “Allcuts”.(f) The valid accuracy change for theGhostNet pre-trained by “All cuts”.Figure 4.2: The valid accuracy change for ResNeXt50 and GhostNet withdifferent pre-trained weights during the 5-fold cross-validation training.434.4. DISCUSSION4.4 DiscussionThe result of this research demonstrates significant advancements com-pared to prior work. Here are the detailed analyses:− Data Pre-processing: The repeated audio data across various speciesare cleaned in the “Best of cuts” and “All cuts” from the Watkins Ma-rine Mammal Sound Database. Besides, an algorithm used to computesignal bandwidth is utilized to compute a balanced sample rate basedon the percentage of total power encapsulating. Based on this algo-rithm, a high sampling rate of 100 kHz is utilized in the “Best of cuts”and “All cuts” by encapsulating at least 99% of the total power of theaudio signal. In contrast, prior work neither performs comprehensivedata cleaning nor utilizes a sampling rate beyond the default samplerate of 22.05 kHz [MIHA22]. This leads to a potential loss of high-frequency information from the acoustic data. As a result, our refineddata preprocessing steps enable the ResNet50 model to achieve an F1-score of 99.84% on “Best of cuts” compared to 86.7% of the F1-scorein this previous work [MIHA22].− Impact of Spectrogram Size on Classification: The researchon the effects of larger spectrogram dimensions provided some crit-ical insights. Notably, increasing the frequency bins to 1025 or ex-tending time segmentation to 512 seemed to hinder the classificationefficacy of the CNN models such as ResNet50, ResNeXt50, and Effi-cientNetB0. This indicates that simply increasing the frequency res-olution or elongating time segmentation in the spectrogram does notnecessarily result in better classification outcomes. These outcomespresent a contradiction to the conjecture proposed in prior research,which suggested that larger spectrogram dimensions may correlatewith improved performance of CNN models [MIHA22]. However, thedata cleaning, higher sampling rate, and lower overlapping rate imple-mented in the “ Best of cuts” dataset in this research may cause thisdifferent result.− Comparative Analysis of Deep Learning Models: This researchdeploys multiple deep-learning architectures and assesses their clas-sification performances in marine mammal sounds. Notably, CNNmodels such as ResNeXt50, VGGish, and EfficientNetB0 demonstratesuperiority over the ResNet50 model, which was previously used forthe 32-class classification in “Best of cuts” [MIHA22]. Additionally,444.4. DISCUSSIONwhile both LSTM and AST are capable of classifying marine mammalvocalizations across various species, AST’s performance lags behindmost of the CNN models. This might be attributed to the “Best ofcuts” dataset’s limited size. Transformers require to be trained by thevast dataset to achieve the best performance, as seen in some studieswhere Transformers do not outperform CNNs on the smaller datasetcontaining 9 million images [DBK+20]. As for LSTM, the best LSTMarchitecture (a 2-layer LSTM with 2048 hidden units) does not reachthe performance benchmarks set by the CNNs and Transformers. CNNmodels ResNeXt50 and GhostNet are evaluated on the more extensiveAll cuts” 32-class dataset. Both of them demonstrate impressive clas-sification accuracy, achieving 99.50% and 99.42% respectively.− Comparison with Prior Works: Compared with previous research,our data pre-processing and utilizing advanced deep-learning modelmethods boost the classification accuracy dramatically. However, thishigh classification accuracy rate may be mainly because of the data-cleaning procedure rather than our methods. In order to show the in-fluence of the high resampling rate and advanced deep-learning modelutilized in this work in quality. A supplementary test is conductedto implement the deep-learning models in the “Best of cuts” datasetwithout the data-cleaning process. Table 4.10 illustrates the perfor-mances of ResNet50 and ResNeXt50 models on “Best of cuts” withoutthe data-cleaning process.Table 4.10: The mean cross-validation results of each model on original“Best of cuts”.Model Accuracy Precision Recall F1-scoreResNet50 88.60% 89.19% 88.60% 88.61%ResNeXt50 89.84% 92.09% 89.84% 90.24%From this table, the ResNet50 model achieved 88.61% of the F1-scorecompared to 86.7% of the F1-score in the precious work also utilizedResNet50, which illustrates the high resampling rate utilized in thiswork can improve the classification performance [MIHA22]. Besides,the best-performed ResNeXt50 model boosts the classification perfor-mance to 89.84% in accuracy and 90.24% in F1-score compared to theprevious benchmark of 87% accuracy in SVM and 84.7% accuracy inVGGish [HHL+22].454.4. DISCUSSIONWe present the confusion matrix as Figure 4.3 and the classificationreport as Table 4.11 as primary tools for assessing our model. Theconfusion matrix offers a detailed representation of the model’s per-formance, showing the actual versus predicted classifications and high-lighting where the model confuses between different classes. Further-more, the classification report provides crucial metrics for each class,including precision, recall, and F1-score. By employing both the con-fusion matrix and the classification report, we aim to gain a nuancedunderstanding of the model’s strengths and weaknesses across all 32classes. This approach allows us to identify specific areas where themodel excels or requires improvement, guiding future enhancementsand adjustments.Table 4.11: The classification report of the ResNeXt50 in the 5th cross-validation results on “Best of cuts” without data cleaning.Species Precision Recall F1-score DataAmountAtlanticSpotted Dol-phin0.9945 1.0000 0.9972 181Beluga 0.6364 0.6037 0.6196 429BottlenoseDolphin1.0000 0.9697 0.9846 33BowheadWhale0.7021 0.7293 0.7154 543ClymeneDolphin0.9984 0.9984 0.9984 191CommonDolphin1.0000 1.0000 1.0000 415False KillerWhale1.0000 1.0000 1.0000 77FinbackWhale1.0000 1.0000 1.0000 3019Fraser’s Dol-phin1.0000 1.0000 1.0000 612Continued on next page464.4. DISCUSSIONTable 4.11 – Continued from Previous PageSpecies Precision Recall F1-score DataAmountRisso’s Dol-phin0.9951 0.9951 0.9951 204Harp Seal 0.9936 0.9811 0.9873 159HumpbackWhale0.9989 0.9997 0.9993 3558Killer Whale 1.0000 0.9760 0.9879 125Leopard Seal 0.9825 1.0000 0.9912 56Long-FinnedPilot Whale0.9891 0.9937 0.9914 637Melon HeadWhale0.9834 0.9727 0.9780 183MinkeWhale1.0000 1.0000 1.0000 22Narwhal 1.0000 1.0000 1.0000 93NorthernRight Whale0.9517 0.9857 0.9684 140PantropicalSpotted Dol-phin0.9710 1.0000 0.9853 134Ross Seal 0.9948 1.0000 0.9974 193Rough-ToothedDolphin1.0000 1.0000 1.0000 74Short-Finned PilotWhale0.5532 0.9052 0.6867 2150SouthernRight Whale0.9091 0.9091 0.9091 55SpermWhale0.9601 0.7660 0.8521 6778Spinner Dol-phin0.9973 0.9973 0.9973 376Striped Dol-phin1.0000 1.0000 1.0000 169Walrus 0.9954 0.9909 0.9932 220Continued on next page474.4. DISCUSSIONTable 4.11 – Continued from Previous PageSpecies Precision Recall F1-score DataAmountWeddell Seal 1.0000 1.0000 1.0000 52White-beakedDolphin1.0000 1.0000 1.0000 183White-sidedDolphin1.0000 1.0000 1.0000 219− CNN Generalization Capabilities: This result highlighted thatthe convolutional layers of the CNN have the capacity to generalize toclassify a broader range of marine mammal acoustic data. This out-come suggests that the CNN model could be re-trained efficiently bytransfer learning when new acoustic sources are added to the database,even when computational resources are limited. Since the convolu-tional layers of CNNs could be frozen in this situation. Moreover, uti-lizing the generalization ability of CNN can not only classify the extraacoustic data from additional species [TMK+20], but also improvethe classification accuracy on the different datasets by pre-trainedweight. In this research, the CNN models GhostNet and ResNeXt50pre-trained on the Watkins Marine Mammal Sound Database achievebetter classification accuracy compared to them using “ImageNet1K”pre-trained weight when they are fine-tuned to classify the Mobysounddataset. Moreover, the method that potentially increases the gener-alization ability of the CNN model is explored. Additionally, we ex-amined the effects of the “SpecAugment” data augmentation methodon the generalization ability of the CNN model. Although our resultsindicate that “SpecAugment” hurt model performance, it is prematureto dismiss this method altogether. The effectiveness of data augmen-tation techniques like “SpecAugment” can vary widely based on thespecific parameters used. Therefore, more detailed research is needed.This concern will be further discussed in the sections on limitationsand future research.484.4. DISCUSSIONFigure 4.3: The confusion matrix of the ResNeXt50 in the 5th cross-validation results on “Best of cuts” without data cleaning.49Chapter 5Conclusion5.1 SummaryThis research demonstrates that deep-learning models of ANN, CNN,LSTM, and AST are capable of classifying a broad range of marine mammalvocalizations, and the ResNeXt50 CNN model has the highest classificationaccuracy, followed by the models of AST, LSTM, and ANN in the “Best ofcuts” dataset with data cleaning from the Watkins Marine Mammal SoundDatabase.The results show that spectrogram dimensions 513 × 256 in terms offrequency bins and time frames make the CNN-based models have the bestclassification accuracy, but exceeding spectrogram size by extending thewindow length or slicing in larger time segmentation compromised CNNperformance. Additionally, resizing the spectrogram in pixels to match CNNinput requirements had a minimal impact on the model’s performance.The outcomes also spotlight CNN models that have the generalizationability to adapt to new datasets, whether it’s additional acoustic data fromknown species or entirely new species sound data. Utilizing this featureand transfer learning, the classification accuracy of the CNN model in thenew marine mammal acoustic dataset could be enhanced by pre-training inother marine mammal sound datasets even with different species or acousticsources.5.2 Limitations and Future ResearchThis section summarizes the limitations of our experiments and estab-lishes future research directions.5.2.1 Limited Research in “SpecAugmentation”As for the third study, we need to conduct detailed experiments to eval-uate the “SpecAugmentation” data augmentation method. The key pointsthat need to be considered are as follows:505.2. LIMITATIONS AND FUTURE RESEARCH− Exploration of Parameters: It is essential to conduct additional ex-periments to identify the optimal parameters for the “SpecAugment”.This includes finding the optimal percentage of the mask rate in bothfrequency and time dimensions. By systematically varying these pa-rameters, we can better understand their effects and potentially un-cover a set of conditions that improve model performance.− Blank Areas: Many spectrograms contain blank areas that do notcontribute useful information and may lead to artificially high classi-fication results. The question of whether to apply masking regions tothese areas is a problem that needs to be explored by further experi-mental exploration.By addressing these considerations, we aim to build a more robust under-standing of how spectrogram augmentation can be effectively implementedin CNN models for marine mammal sound classification.5.2.2 Device-specific NoiseIn our research, we observed that the audio data for various marinespecies were collected using a diverse array of recording devices [WFDH92].This variation introduces a critical variable to our analysis: device-specificnoise. A notable manifestation of this is the horizontal straight lines ob-served in the spectrogram that is shown in Figure 5.1, which likely repre-sent noise inherent to the recording equipment. Such patterns are atypicalof natural sounds, which do not maintain a constant frequency over time.The presence of device noise is a significant concern, as it can artifi-cially inflate the classification accuracy of our models. This misleadingenhancement can compromise the validity of our findings and hinder ourunderstanding of the true characteristics of marine species audio data.5.2.3 Avoiding Potential OverfittingThe choice of a larger spectrogram size (513 × 256) in our study mighthave significantly influenced the results. Specifically, a larger spectrogramsize diminishes the total count of spectrograms available for training andvalidation, thereby potentially elevating the risk of overfitting. To mitigatethis issue, we propose the following possible solutions:− Reduction of Spectrogram Size:By adopting smaller spectrogramdimensions, such as 256 × 256 or 128 × 128, we can significantly in-crease the overall number of spectrograms. This enlargement of the515.2. LIMITATIONS AND FUTURE RESEARCHFigure 5.1: Example of a spectrogram with unusual noise.dataset available for training and validation serves to dilute the riskof overfitting, providing a more robust foundation for model learning.Ensuring that the total number of spectrograms exceeds the product ofthe spectrogram dimensions is critical. This strategy aims to providea sufficiently large dataset for training, which is essential for avoidingpotential overfitting.− Increase in the Number of Spectrograms: Alternatively, am-plifying the total count of spectrograms is a viable strategy. Thiscan be accomplished by enhancing the overlapping rate during spec-trogram generation. More overlapping segments translate to an ex-panded training dataset, which helps in mitigating the risk associatedwith overfitting.− Independent Test Dataset: Beyond these measures, another criti-cal method for preventing overfitting involves segregating a test datasetthat remains entirely distinct from the training and validation sets. Alltrained models must be rigorously evaluated and compared within thisindependent test dataset. This practice provides an unbiased assess-ment of the models and ensures our results are valid and reliable.525.2. LIMITATIONS AND FUTURE RESEARCH5.2.4 Implementing Feature EngineeringFuture research endeavours will focus on the application of advancedfeature engineering techniques to spectrograms, aiming to enhance the clas-sification performance of marine mammal species significantly. A pivotalstrategy within this domain is the extraction of horizontal feature lines fromthe spectrograms, which are then used as distinct feature vectors for speciesdifferentiation. This approach is particularly beneficial as it effectively min-imizes the influence of non-informative blank areas within the spectrogram,thereby potentially increasing classification accuracy. Additionally, it ad-dresses the challenge posed by the various frequency ranges of marine mam-mal sounds from different species, which can sometimes artificially inflateclassification results. Moreover, the implementation of this technique couldrender the issue of resampling rates less consequential. By prioritizing thehorizontal aspects of the spectrogram, our dependency on the vertical axisdiminishes, streamlining the data preprocessing process and potentially lead-ing to more robust and accurate classification outcomes.5.2.5 Implementing larger deep-learning modelsIn this study, we established a one-hidden-layer Artificial Neural Net-work (ANN) as the baseline model for comparative purposes. Addition-ally, we employed Convolutional Neural Network (CNN) models, includingResNet50, ResNeXt50, and the EfficientNetV2 small. Notably, each of thesemodels has larger counterparts with an increased number of convolutionallayers, suggesting a potential for enhanced performance. In future inves-tigations, it will be valuable to test these more extensive models againstthe ”Best of cuts” dataset. By doing so, we can ascertain whether theadditional convolutional layers translate into tangible improvements in clas-sification performance. This exploration potentially set new benchmarks forclassification accuracy in marine mammal sound classification.5.2.6 Collecting More Data and Testing in Real-worldA fundamental limitation of our research is that the trained models havenot been tested in real-world scenarios. Indeed, we have not used othervocalization data from the 32 species of marine mammals to evaluate ourmodel’s robustness. The main reason is that the initial sample rate betweendifferent data sources varies greatly, and it is challenging to find acousticdata with similar initial sample rates. Critically, the configuration of thespectrogram is highly determined by the resampling rate used to load the535.2. LIMITATIONS AND FUTURE RESEARCHaudio file. Even the spectrograms for the same audio with different re-sampling rates during the loading process are totally different. This poses asubstantial limitation when classifying marine mammals’ vocalizations usingspectrogram features and deep-learning models. For comparison, applica-tions like human speech recognition usually employ a standard sample rateof 16 kHz, given that the frequency bandwidth of human vocalizations isconstrained. In contrast, the frequency spectrum of marine mammal vocal-izations can span from less than ten hertz to well over a hundred thousandhertz [JT03], making the determination of a standard sample rate for diversemarine mammals considerably challenging.Based on this limitation, more audio data for marine mammal vocal-ization with the same original sample rate must be collected for furtherinvestigation. another approach is implementing our proposed feature en-gineering to render the issue of resampling rates less consequential. Theproposed classifying approaches are required to be assessed in a realisticocean environment.54Bibliography[AHH+21] Ann N Allen, Matt Harvey, Lauren Harrell, Aren Jansen, Kar-lina P Merkens, Carrie C Wall, Julie Cattiau, and Erin MOleson. A convolutional neural network for automated detec-tion of humpback whale song in a diverse, long-term passiveacoustic dataset. Frontiers in Marine Science, 8:607321, 2021.→ pages 2[AMAZ17] Saad Albawi, Tareq Abed Mohammed, and Saad Al-Zawi. Un-derstanding of a convolutional neural network. In 2017 in-ternational conference on engineering and technology (ICET),pages 1–6. IEEE, 2017. → pages viii, 11[Ary22] Nisha Arya. Why use k-fold cross validation?, 2022. → pages27[BBG20] Peter Bruce, Andrew Bruce, and Peter Gedeck. Practicalstatistics for data scientists: 50+ essential concepts using Rand Python. O’Reilly Media, 2020. → pages viii, 29[BBW+19] Peter C Bermant, Michael M Bronstein, Robert J Wood, ShaneGero, and David F Gruber. Deep machine learning techniquesfor the detection and classification of sperm whale bioacoustics.Scientific reports, 9(1):1–10, 2019. → pages 2[BPRL17] Venkatesh Boddapati, Andrej Petef, Jim Rasmusson, andLars Lundberg. Classifying environmental sounds using imagerecognition networks. Procedia Computer Science, 112:2048–2056, 2017. Knowledge-Based and Intelligent Information En-gineering Systems: Proceedings of the 21st International Con-ference, KES-20176-8 September 2017, Marseille, France. →pages 5[Bri89] John Bridle. Training stochastic model recognition algorithmsas networks can lead to maximum mutual information estima-55Bibliographytion of parameters. Advances in neural information processingsystems, 2, 1989. → pages 11[BS09] Judith C Brown and Paris Smaragdis. Hidden Markov andGaussian mixture models for automatic call classification. TheJournal of the Acoustical Society of America, 125(6):EL221–EL224, 2009. → pages 4[BSC+19] Christian Bergler, Hendrik Schro¨ter, Rachael Xi Cheng, VolkerBarth, Michael Weber, Elmar No¨th, Heribert Hofer, and An-dreas Maier. ORCA-SPOT: An automatic killer whale sounddetection toolkit using deep learning. Scientific reports, 9(1):1–17, 2019. → pages 2[Bur19] Andriy Burkov. The hundred-page machine learning book, vol-ume 1. Andriy Burkov Quebec City, QC, Canada, 2019. →pages 9, 10, 11, 12, 13[BVG+22] Hicham Bellafkir, Markus Vogelbacher, Jannis Gottwald,Markus Mu¨hling, Nikolaus Korfhage, Patrick Lampe, Nico-las Frieß, Thomas Nauss, and Bernd Freisleben. Bat echolo-cation call detection and species recognition by transformerswith self-attention. In Intelligent Systems and Pattern Recog-nition: Second International Conference, ISPR 2022, Ham-mamet, Tunisia, March 24–26, 2022, Revised Selected Papers,pages 189–203. Springer, 2022. → pages 2[CZZY22] Wenyu Cai, Jifeng Zhu, Meiyan Zhang, and Yong Yang. A par-allel classification model for marine mammal sounds based onmulti-dimensional feature extraction and data augmentation.Sensors, 22(19):7443, 2022. → pages viii, 28, 29[DBK+20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, DirkWeissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa De-hghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.An image is worth 16x16 words: Transformers for image recog-nition at scale. arXiv preprint arXiv:2010.11929, 2020. →pages viii, 31, 32, 34, 45[Dos21] Ketan Doshi. Transformers explained visually (part 3): Multi-head attention, deep dive, 2021. → pages 3356Bibliography[DRUC10] Peter J. Dugan, Aaron N. Rice, Ildar R. Urazghildiiev, andChristopher W. Clark. North atlantic right whale acoustic sig-nal processing: Part i. comparison of machine learning recog-nition algorithms. In 2010 IEEE Long Island Systems, Appli-cations and Technology Conference, pages 1–6, 2010. → pages4[DTG09] Kathleen M. Dudzinski, Jeanette A. Thomas, and Justin D.Gregg. Communication in marine mammals. In William F.Perrin, Bernd Wu¨rsig, and J.G.M. Thewissen, editors, Ency-clopedia of Marine Mammals (Second Edition), pages 260–269.Academic Press, London, second edition edition, 2009.→ pages1[EABP22] Christian D Escobar-Amado, Mohsen Badiey, and Sean Pec-knold. Automatic detection and classification of beard sealvocalizations in the northeastern chukchi sea using convolu-tional neural networks. The Journal of the Acoustical Societyof America, 151(1):299–309, 2022. → pages 2[GCG21] Yuan Gong, Yu-An Chung, and James R. Glass. AST: audiospectrogram transformer. CoRR, abs/2104.01778, 2021. →pages 2, 27, 31, 34[GG20] Benyamin Ghojogh and Ali Ghodsi. Attention mechanism,transformers, BERT, and GPT: tutorial and survey. 2020. →pages viii, 33[HHL+22] Masato Hagiwara, Benjamin Hoffman, Jen-Yu Liu, MaddieCusimano, Felix Effenberger, and Katie Zacarian. Beans: Thebenchmark of animal sounds. arXiv preprint arXiv:2210.12300,2022. → pages 11, 45[HSC+19] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen,Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruom-ing Pang, Vijay Vasudevan, et al. Searching for mobilenetv3.In Proceedings of the IEEE/CVF international conference oncomputer vision, pages 1314–1324, 2019. → pages 27[I1] Aras M. Ismael and Abdulkadir S¸engu¨r. Deep learning ap-proaches for covid-19 detection based on chest X-ray images.Expert Systems with Applications, 164:114054, 2021. → pages1157Bibliography[JBD+19] Jia-jia Jiang, Ling-ran Bu, Fa-jie Duan, Xian-quan Wang, WeiLiu, Zhong-bo Sun, and Chun-yue Li. Whistle detection andclassification for whales based on convolutional neural net-works. Applied Acoustics, 150:169–178, 2019. → pages 2[JJLJ20] Hohyub Jeon, Yongchul Jung, Seongjoo Lee, and Yunho Jung.Area-efficient short-time fourier transform processor for time-frequency analysis of non-stationary signals. Applied Sciences,10(20):7208, 2020. → pages viii, 7, 8[JT03] Mark P Johnson and Peter L Tyack. A digital acoustic record-ing tag for measuring the response of wild marine mammals tosound. IEEE journal of oceanic engineering, 28(1):3–12, 2003.→ pages 1, 54[KCB20] Ajay Kulkarni, Deri Chong, and Feras A. Batarseh. 5 - founda-tions of data imbalance and solutions for a data democracy. InFeras A. Batarseh and Ruixin Yang, editors, Data Democracy,pages 83–106. Academic Press, 2020. → pages 29[KFS+20] Oliver S. Kirsebom, Fabio Frazao, Yvan Simard, Nathalie Roy,Stan Matwin, and Samuel Giard. Performance of a deep neu-ral network at detecting north atlantic right whale upcalls.The Journal of the Acoustical Society of America, 147(4):2636–2646, apr 2020. → pages 2, 4[LBH15] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learn-ing. nature, 521(7553):436–444, 2015. → pages viii, 12, 13[LHY21] Tao Lu, Baokun Han, and Fanqianhui Yu. Detection and classi-fication of marine mammal sounds using AlexNet with transferlearning. Ecological Informatics, 62:101277, 2021. → pages 2,12, 17, 29[LLY+22] Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and JunZhou. A survey of convolutional neural networks: Analysis,applications, and prospects. IEEE Transactions on Neural Net-works and Learning Systems, 33(12):6999–7019, 2022. → pages11[MC06] David K. Mellinger and Christopher W. Clark. Mobysound:A reference archive for studying automatic recognition of ma-rine mammal sounds. Applied Acoustics, 67(11):1226–1242,58Bibliography2006. Detection and localization of marine mamals using pas-sive acoustics. → pages 18[MIHA22] Daniel T Murphy, Elias Ioup, Md Tamjidul Hoque, and MahdiAbdelguerfi. Residual learning for marine mammal classifica-tion. IEEE Access, 10:118409–118418, 2022. → pages 9, 11,24, 29, 44, 45[MRL+15] Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, MattMcVicar, Eric Battenberg, and Oriol Nieto. librosa: Audioand music signal analysis in python. In Proceedings of the 14thpython in science conference, volume 8, pages 18–25, 2015. →pages 6, 7, 21[MS14] Manish Mishra and Monika Srivastava. A view of artificialneural network. In 2014 international conference on advancesin engineering & technology research (ICAETR-2014), pages1–3. IEEE, 2014. → pages 9[MSK+21] Shyam Madhusudhana, Yu Shiu, Holger Klinck, Erica Fleish-man, Xiaobai Liu, Eva-Marie Nosal, Tyler Helble, DanielleCholewiak, Douglas Gillespie, Ana Sˇirovic´, et al. Improve auto-matic detection of animal call sequences with temporal context.Journal of the Royal Society Interface, 18(180):20210297, 2021.→ pages 14[MUB07] Suleman Mazhar, Tamaki Ura, and Rajendar Bahl. Vocaliza-tion based individual classification of humpback whales usingsupport vector machine. In OCEANS 2007, pages 1–9. IEEE,2007. → pages 4[MV21] B Mishachandar and S Vairamuthu. Diverse ocean noise clas-sification using deep learning. Applied Acoustics, 181:108141,2021. → pages 14, 26[Nta17] Stavros Ntalampiras. A deep learning framework for classifyingsounds of mysticete whales. In Handbook of Neural Computa-tion, pages 403–415. Elsevier, 2017. → pages 2[OGMS+09] Emilio Soria Olivas, Jos David Mart Guerrero, MarcelinoMartinez-Sober, Jose Rafael Magdalena-Benedito, L Serrano,et al. Handbook of research on machine learning applications59Bibliographyand trends: Algorithms, methods, and techniques: Algorithms,methods, and techniques. IGI global, 2009. → pages 30[Ola15] Christopher Olah. Understanding LSTM Networks, 2015. →pages viii, 13, 14[ON15] Keiron O’Shea and Ryan Nash. An introduction to convolu-tional neural networks. arXiv preprint arXiv:1511.08458, 2015.→ pages viii, 10[PCZ+19] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu,Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: Asimple data augmentation method for automatic speech recog-nition. arXiv preprint arXiv:1904.08779, 2019. → pages 24[Pel21] Thomas Pellegrini. Deep-learning-based central african pri-mate species classification with mixup and specaugment. InInterspeech 2021, 2021. → pages 26[PS18] Antonia Papandreou-Suppappola. Applications in time-frequency signal processing. CRC press, 2018. → pages 6,7[PT12] Swarnalatha Purushotham and BK Tripathy. Evaluation ofclassifier models using stratified tenfold cross validation tech-niques. In Global Trends in Information Systems and SoftwareApplications: 4th International Conference, ObCom 2011, Vel-lore, TN, India, December 9-11, 2011. Proceedings, Part II,pages 680–690. Springer, 2012. → pages 29[RLL22] Esteban N Rodofili, Vincent Lecours, and Michelle LaRue. Re-mote sensing techniques for automated marine mammals de-tection: a review of methods and current challenges. PeerJ,10:e13540, 2022. → pages 1[RSH+08] Marie A Roch, Melissa S Soldevilla, Rhonda Hoenigman,Sean M Wiggins, and John A Hildebrand. Comparison of ma-chine learning techniques for the classification of echolocationclicks from three species of odontocetes. Canadian Acoustics,36(1):41–47, 2008. → pages 4[Sar96] D. Sarkar. Randomness in generalization ability: a source toimprove it. IEEE Transactions on Neural Networks, 7(3):676–685, 1996. → pages 160Bibliography[SCC08] Jan Schipper, Janice S. Chanson, and Federica Chiozza. Thestatus of the world’s land and marine mammals: Diversity,threat, and knowledge. Science, 322(5899):225–230, 2008. →pages 1[SDA+16] Laela Sayigh, Mary Ann Daher, Julie Allen, Helen Gordon,Katherine Joyce, Claire Stuhlmann, and Peter Tyack. TheWatkins marine mammal sound database: an online, freelyaccessible resource. In Proceedings of Meetings on Acoustics4ENAL, volume 27, page 040013. Acoustical Society of Amer-ica, 2016. → pages 17[Sha49] C.E. Shannon. Communication in the presence of noise. Pro-ceedings of the IRE, 37(1):10–21, 1949. → pages 21, 23[SM+05] Petre Stoica, Randolph L Moses, et al. Spectral analysis ofsignals, volume 452. Pearson Prentice Hall Upper Saddle River,NJ, 2005. → pages 7[Smi13] Steven Smith. Digital signal processing: a practical guide forengineers and scientists. Elsevier, 2013. → pages 7[SPR+20] Yu Shiu, KJ Palmer, Marie A Roch, Erica Fleishman, XiaobaiLiu, Eva-Marie Nosal, Tyler Helble, Danielle Cholewiak, Dou-glas Gillespie, and Holger Klinck. Deep neural networks forautomated detection of marine mammal species. Scientific re-ports, 10(1):1–12, 2020. → pages 2, 11[TMK+20] Mark Thomas, Bruce Martin, Katie Kowarski, Briand Gaudet,and Stan Matwin. Marine mammal species classification us-ing convolutional neural networks and a novel acoustic rep-resentation. In Machine Learning and Knowledge Discov-ery in Databases: European Conference, ECML PKDD 2019,Wu¨rzburg, Germany, September 16–20, 2019, Proceedings,Part III, pages 290–305. Springer, 2020. → pages 2, 9, 11,40, 48[Tra21] Marek B Trawicki. Multispecies discrimination of whales(cetaceans) using Hidden Markov Models (HMMS). Ecolog-ical Informatics, 61:101223, 2021. → pages 4[VGG+18] Ursula K. Verfuss, Douglas Gillespie, Jonathan Gordon,Tiago A. Marques, Brianne Miller, Rachael Plunkett, James A.61BibliographyTheriault, Dominic J. Tollit, Daniel P. Zitterbart, PhilippeHubert, and Len Thomas. Comparing methods suitable formonitoring marine mammals in low visibility conditions dur-ing seismic surveys. Marine Pollution Bulletin, 126:1–18, 2018.→ pages 1[VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and IlliaPolosukhin. Attention is all you need. Advances in neural in-formation processing systems, 30, 2017. → pages viii, 31, 32,33[WBK20] Jeremy Watt, Reza Borhani, and Aggelos K Katsaggelos. Ma-chine learning refined: Foundations, algorithms, and applica-tions. Cambridge University Press, 2020. → pages 9[Wel67] P. Welch. The use of fast fourier transform for the estima-tion of power spectra: A method based on time averaging overshort, modified periodograms. IEEE Transactions on Audioand Electroacoustics, 15(2):70–73, 1967. → pages 22[WFDH92] William A Watkins, Kurt Fristrup, Mary Ann Daher, and Ter-rance Howland. SOUND database of marine animal vocaliza-tions: structure and operations. Woods Hole OceanographicInstitution, 1992. → pages 51[WW85] William A. Watkins and Douglas Wartzok. Sensory biophysicsof marine mammals. Marine Mammal Science, 1(3):219–260,1985. → pages 1[WWB+22] Ellen Louise White, Paul White, Jonathan Bull, Denise Risch,Suzanne Beck, and Ewan Edwards. More than a whistle: Auto-mated detection of marine sound sources with a convolutionalneural network. Frontiers in Marine Science, 2022. → pages9, 24[WWX17] Qingcui Wang, Lujun Wang, and Xiaonan Xu. Automatic de-tection and classification of marine mammal tonal calls. In 2017IEEE International Conference on Signal Processing, Commu-nications and Computing (ICSPCC), pages 1–5, 2017. → pages462Bibliography[Wys17] Lonce Wyse. Audio spectrogram representations for pro-cessing with convolutional neural networks. arXiv preprintarXiv:1706.09559, 2017. → pages 5[Zho21] Zhi-Hua Zhou. Machine learning. Springer Nature, 2021. →pages 4[ZQD+20] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi,Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. Acomprehensive survey on transfer learning. Proceedings of theIEEE, 109(1):43–76, 2020. → pages 30[Zup94] Jure Zupan. Introduction to artificial neural network (ANN)methods: what they are and how to use them. Acta ChimicaSlovenica, 41:327–327, 1994. → pages 963
Thesis/Dissertation
eng
Electrical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Cheng, Julian
Marine mammal sound classification using deep learning models
Text