Arts, Faculty of
iSchool (Library, Archival and Information Studies)
DSpace
UBCV
Sullivan, Peter
Master of Library and Information Studies - MLIS
University of British Columbia
Arabic dialect identification (ADI) is an important aspect of the Arabic speech processing pipeline, and in particular dialectal Arabic automatic speech recognition (ASR) models. In this work, we present an overview of corpora and methods applicable to both ADI and dialectal Arabic ASR, then we benchmark two approaches to using pre-trained speech representation models for ADI. Namely, we first employ direct fine-tuning, and then use fixed-representations extracted from pre-trained models as an intermediate step in the ADI process. We train and evaluate our models on the granular ADI-17 Arabic dialect corpus (92% F1 for our fine-tuned HuBERT model), and further probe generalization by evaluating our trained models on coarse-grained ADI-5, (80% F1 for fine-tuned HuBERT).
Representation Learning for ArabicDialect IdentificationbyPeter SullivanB.Sc., (Physics), Massachusetts Institute of Technology, 2008A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF LIBRARY AND INFORMATION STUDIESinThe Faculty of Graduate and Postdoctoral Studies(Library and Information Studies)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)August 2022© Peter Sullivan 2022The following individuals certify that they have read, and recommend to the Faculty ofGraduate and Postdoctoral Studies for acceptance, the thesis entitled:Representation Learning for Arabic Dialect Identificationsubmitted by Peter Sullivan in partial fulfilment of the requirements for the degree ofMaster of Library and Information Studies:Examining Committee:Muhammad Abdul-Mageed, Associate Professor, School of Information, UBCSupervisorLuanne Sinnamon, Associate Professor, School of Information, UBCSupervisory Committee MemberMiikka Silfverberg, Assistant Professor, Department of Linguistics, UBCSupervisory Committee MemberiiAbstractArabic dialect identification (ADI) is an important aspect of the Arabic speech processingpipeline, and in particular dialectal Arabic automatic speech recognition (ASR) models.In this work, we present an overview of corpora and methods applicable to both ADIand dialectal Arabic ASR, then we benchmark two approaches to using pre-trained speechrepresentation models for ADI. Namely, we first employ direct fine-tuning, and then usefixed-representations extracted from pre-trained models as an intermediate step in theADI process. We train and evaluate our models on the granular ADI-17 Arabic dialectcorpus (92% F1 for our fine-tuned HuBERT model), and further probe generalization byevaluating our trained models on coarse-grained ADI-5, (80% F1 for fine-tuned HuBERT).iiiLay SummaryAutomatic dialect identification allows for categorization of spoken audio clips into appro-priate dialect or language. We apply a strategy called transfer learning, to use machinelearning models trained on general audio recognition tasks for our more specific goal ofArabic dialect identification. This strategy allows for faster and more data-efficient devel-opment of dialect identification models, enabling application of the method to languagesand dialects that might have few resources available for creation of such models. Ourgoal through this is to support a virtuous cycle whereby these models first improve datacollection and filtering for under-resourced languages. This additional data can then leadto even better language tools, and finally consumer applications such as spoken languageassistants.ivPrefaceModel development, training, and analysis were conducted by myself at the UBC DeepLearning and Natural Language Processing Lab, using compute resources provided byCompute Canada.Figure 1.1 was originally published in Abdul-Mageed et al. 2020 [3] and has beenreprinted with permission.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1 Spoken Language and Dialect Identification . . . . . . . . . . . . . . . . . . 52.1.1 Fixed Representation Systems . . . . . . . . . . . . . . . . . . . . . 52.1.2 Direct LID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1.3 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.4 Section Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.1.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.2 Automatic Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . 132.2.1 HMM Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.2.2 End-to-End ASR . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.2.3 Self-supervised Models . . . . . . . . . . . . . . . . . . . . . . . . . 162.2.4 Quantization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.2.5 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.2.6 Section Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.2.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.3 Arabic Linguistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23viTable of Contents2.3.1 Phonological Differences . . . . . . . . . . . . . . . . . . . . . . . . 242.3.2 Morphological Differences . . . . . . . . . . . . . . . . . . . . . . . . 252.3.3 Lexical Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252.3.4 Section Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.4 Arabic Speech Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.4.1 Diacretization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.4.2 Dialectal Arabic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282.4.3 Morphological Complexity . . . . . . . . . . . . . . . . . . . . . . . 282.4.4 ASR models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.4.5 ADI models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.4.6 Section Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.4.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.5 An Overview of Arabic Speech Corpora . . . . . . . . . . . . . . . . . . . . 322.5.1 Conversational Telephone Corpora . . . . . . . . . . . . . . . . . . . 322.5.2 Broadcast Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . 342.5.3 Other Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.5.4 Non-transcribed Corpora . . . . . . . . . . . . . . . . . . . . . . . . 392.5.5 Section Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.5.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.2 Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463.3.1 E2E Fine-tuned Pre-trained Speech Representation Models . . . . . 463.3.2 Fixed-Representation Classification . . . . . . . . . . . . . . . . . . 474 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.1 Fixed Representation Classifier . . . . . . . . . . . . . . . . . . . . . . . . . 514.2 Fine-tuned Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.2 Ethical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59viiTable of ContentsAppendixA Model Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78viiiList of Tables2.1 Pipeline of select LID papers . . . . . . . . . . . . . . . . . . . . . . . . . . 102.2 LID metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.3 Overview of speech representation models . . . . . . . . . . . . . . . . . . . 182.4 Breakdown of Arabic GALE speech . . . . . . . . . . . . . . . . . . . . . . . 352.5 Overview of sources for GALE Arabic Speech . . . . . . . . . . . . . . . . . 362.6 Overview of Arabic Speech Corpora. . . . . . . . . . . . . . . . . . . . . . . . 413.1 Breakdown of ADI-5 and ADI-17 Corpora . . . . . . . . . . . . . . . . . . . 453.2 Finetuning hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.3 HuBERT Fixed representation hyperparameters . . . . . . . . . . . . . . . . 484.1 ADI-17 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.2 ADI-5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50A.1 HuBERT Fixed representation hyperparameters . . . . . . . . . . . . . . . . 78A.2 E2E XLS-r-300m finetuning hyperparameters . . . . . . . . . . . . . . . . . 78A.3 E2E HuBERT finetuning hyperparameters . . . . . . . . . . . . . . . . . . . 78ixList of Figures1.1 Overview of Arabic speaking regions . . . . . . . . . . . . . . . . . . . . . . 32.1 i-vector training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 x-vector overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.3 Overview of Self-Supervised Speech Representation Models . . . . . . . . . 172.4 ADI-5 and ADI-17 Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.1 Speech Representation ADI Model Overview . . . . . . . . . . . . . . . . . 494.1 Fixed Representation ADI-17 Confusion Matrix . . . . . . . . . . . . . . . . 524.2 Fixed Representation ADI-5 Confusion Matrix . . . . . . . . . . . . . . . . 534.3 HuBERT ADI-17 Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . 544.4 HuBERT ADI-5 Confusion Matrix . . . . . . . . . . . . . . . . . . . . . . . 55xAcknowledgementsI would like to thank Ganesh Jawahar, Ife Adebara, and Weirui Chen for their feedbackon the manuscript, Luanne Sinnamon, and Miika Silvferberg for their insight, Muham-mad Abdul-Mageed for his mentorship and advising, and finally Sandy Abah for logisticalsupport during my time at UBC.xiDedicationTo my family.xiiChapter 1IntroductionRecognizing language is a fundamental aspect of human linguistic ability, but we take forgranted our ability to discern between languages we are familiar with or ones that we arenot. The idiom ‘it’s Greek to me’ is used in expression of unfamiliarity with somethingbeing said. But in the field of machine learning, natural language processing (NLP) modelsdo not inherently discern between expressions in English or Greek, that is unless they havebeen trained to do so. Granting them this ability, to automatically recognize what lan-guage is being spoken is mainly an area of practical concern, models that are fed languageon which they have not been trained will not hesitate to produce meaningless gibberish asoutput. Customers relying on automated telephone services are familiar with expressionsof inputting a number to select the language of communication, but there are many cir-cumstances where such indications are not available, and this language of communicationmust be inferred. Companies such as Google and Meta may rely on automatically identi-fying the language used in emails or status updates before selecting what language an adshould be delivered in. For a country’s foreign surveillance apparatus, automatic languageidentification might be used to filter what conversations are eavesdropped. Digital mediaarchivists, might benefit from using such tools to identify and generate language metadatawhere none might exist, taking a step to make such records more available to relevantcommunities. In all cases, the vast quantity of digital content necessitates the use of au-tomation. With this turn to automation, simple questions such as “do I understand this?”can no longer be answered easily. Instead, we must rely on dedicated tools to categorizetext or spoken utterances based on language–the act of language identification (LID). It isLID that will be the focus of this thesis, as a foundational tool that is part of a multilin-gual spoken language processing pipeline. More specifically, this work is in the context ofdialectal Arabic processing where lack of data availability necessitates the need for toolsto help identify and build corpora.While LID may be at the first part of a speech processing pipeline, the most importantpart of the pipeline is automatic speech recognition (ASR): the task of enabling machinesto develop some level of ‘understanding’ of human speech by transcribing input utter-ances as text. Advances in both sophistication of these tools as well as integration inconsumer products have led to an ever greater importance of ASR in everyday life. Newerapplications of these tools include voice activated assistants [132], simultaneous automaticcaptioning [28], and automatic subtitling [101]. And these advances have been drivenprimarily by the growth of deep learning methods within the last decade, starting with1Chapter 1. Introductionthe introduction of deep neural networks (DNNs) as part of hidden-Markov model basedASR [37], and then as purely neural “end-to-end” (E2E) models [80].This progress, however, primarily supports English-based applications, and even thenmay still exhibit bias against minority speakers of English [108]. As with machine learningin general, there are blind spots based on what data is available and used to train, whichoften benefits study on so-called “standard” American English. In fact the vast majority ofpublished research in NLP is on English [25]. In speech processing, the existing gap betweenresearch conducted on English, on the one hand, and other languages on the other hand isfurther exacerbated by data scarcity problems. That is, there is difficulty in collecting andannotating datasets for languages other than English due to issues with funding and/orlack of sufficiently trained researchers. This is another area where language and dialectidentification (DID) models are vital. Unfortunately, high-quality models require largequantities of data. In absence of large quantities of data, the important question is: “wherecould one start?”.While larger companies might have the resources to collect data and develop modelsfor languages other than English, these resources are not necessarily released for research.This causes independent researchers and smaller companies to fend for themselves, or workwith the models released from these larger companies directly.In the context of large released models, it is important to draw attention to the naturallanguage processing (NLP) revolution which has occurred in the last few years. Withthe advent of large-scale self-supervised machine learning models such as BidirectionalEncoder Representations from Transformers (BERT) [63], NLP has undergone a significantrevolution with the ability to ameliorate what was once a major bottleneck–availability oflabeled datasets for performing traditional supervised machine learning. With BERT, self-supervised approaches, where training labels can be inferred from raw examples, becamemainstream, leading to a cavalcade 1 of similar models [55, 120, 153, 154].This impact has similarly been felt in the domain of speech processing, with self-supervised models such as wav2vec [21, 22, 162] effectively employed in a number of tasksthrough unlabeled pre-training on raw audio and then fine-tuned by training on morespecific speech processing tasks (e.g., LID/DID and ASR).One benefit of the pre-training plus fine-tuning process, as pioneered by BERT [63], isthat pre-trained models can be used on numerous different fine-tuned tasks. This has thepositive effect of democratizing the machine learning pipeline by making available strongstarting points for model creation. For speech processing, this means ASR models or LIDmodels can be built with a much smaller data requirement.Fine-tuning, however, may be model and task specific, unfortunately this means thatrecipes for fine-tuning on ASR are likely not applicable to LID. Furthermore, existing ASRfine-tuning recipes for models such as wav2vec 2.0 Large [22] require 24 GPUs trained1As of May 2022, the original BERT paper has been cited more than 35, 000 times on Google Scholar.2Chapter 1. IntroductionFigure 1.1: A map of the Arab countries [3]for 320, 000 training steps 2. For a 4 GPU setup of nvidia V100 GPUs, this might takeupwards of 10 days. This is impractical for many researchers. While there has been somework utilizing wav2vec 2.0 for LID [71, 176], there has been no systematic study of fine-tuning hyperparameters, and thus no suggested strategy for using these models in LID. Anopen question is how to then efficiently use these models in an LID context, specificallyidentifying appropriate learning rates and maximum steps that would make these toolspractical for researchers with limited access to compute infrastructure.In this work, we focus on Arabic and aim to (i) address the problem of Arabic dialectidentification (ADI). We also aim at (ii) providing a critical literature review of workon Arabic speech (iii) benchmarking recent speech models on Arabic data and (iv) explorestrategies for fine-tuning speech representation models on LID. Arabic represents a numberof specific challenges to ASR, avoided in a monolingual English setting, but present inmany other languages. The Arab speaking world is geographically vast (see Fig. 1.1)and consists of dozens of regional varieties, in many ways echoing other macro-languagessuch as Chinese and Malay. Linguistically, Arabic is a verb-subject-object (VSO) languagenoted as having a non-concatentative, morphologically rich structure, with a tendency todrop pronouns [72]. This additional linguistic complexity makes it a good challenge casefor NLP methods that might perform well in English but potentially fail to generalize to2A training step is considered one update to the model parameters.3Chapter 1. Introductionother languages.Another more practical difference from English, which demonstrates the additionalchallenge of Arabic, is the prevalence of both vernacular and formal styles of Arabic andcode-mixing of these two forms. For instance, study of code-switching between dialectalArabic (e.g. Egyptian Arabic) and the standardized form, Modern Standard Arabic (MSA),may aid in understanding code-switching as it occurs in other language families such asbetween dialectal Chinese and Mandarin. To accomplish this goal of furthering Arabicspeech research, we aim to thus provide a strong foundation in this work by benchmarkingthe performance of self-supervised audio representation models on spoken ADI. In addition,our goal is to provide an review of available spoken dialectal Arabic corpora and research.To summarize, we aim to:1. Review existing LID/ DID and ASR literature as it pertains to Arabic ASR and ADI.2. Systematically explore speech representation fine-tuning hyperparameters in a DIDcontext.3. Investigate effectiveness of pre-trained speech representation models on the task ofADI.4. Compare pre-trained speech representation models in both an E2E fine-tuned andfixed-representation setting.5. Validate robustness of E2E fine-tuned and fixed-representation models on a moregeneralized dataset.The rest of the thesis is structured as follows: in Chapter 2 we review relevant researchon speech processing methods and Arabic corpora. In Chapter 3, we give an overview ofour experimental methods. We discuss results in Chapter 4, and finally we conclude with adiscussion of implications for LID as well as use of speech representation models in Chapter5.4Chapter 2Background2.1 Spoken Language and Dialect IdentificationSpoken language and dialect identification (LID and DID, respectively), as the terms stateit, are the tasks of identifying which language or dialect is being spoken in a speech ut-terance. Since it is convenient to train speech recognition models on monolingual corpora,knowing what language is being spoken is vital for the right application of the correspond-ing recognition system. Similarly, as speech corpora may contain hundreds [38, 145], tothousands [100] of hours, these tools are also useful for development of conversational orbroadcast audio corpora that may contain multilingual or multidialectal context. LID andDID can be used to automatically check what language is being spoken on a collected utter-ance, and ensure that it is handled appropriately (e.g., using the relevant models in tasksfurther downstream). In this chapter, we give a comprehensive review of available LID andDID systems starting with compressing utterances into a fixed representation, such as ini-vector or i-vector inspired systems, followed by end-to-end (E2E) neural network systems.2.1.1 Fixed Representation SystemsThe goal of an i-vector system (see Fig. 2.1 for high-level diagram) is to take a variablelength input utterance and reduce it to a fixed representation, the i-vector, which canthen be used for classification [58, 59]. While originating in the domain of speaker recogni-tion [59], i-vectors were quickly put to work in LID due to the benefits of using a fixed vectorto represent an utterance [61, 172]. Earlier work in speaker identification used joint factoranalysis [104] to separately model the impact of channel and speaker characteristic creat-ing a combined Gaussian mixture model (GMM) supervector for classification [105]. Thei-vector approach, instead, aims to work on a smaller intermediate3 vector representation(much smaller than the GMM supervector [59]). Equation 2.1, illustrates this formulation:M = m+ Tω (2.1)Where M is the GMM supervector, m the universal background model (UBM), T atotal variability matrix, and ω the i-vector (which is simply a random vector with nor-mal distribution [59]). The UBM can be trained by means of a GMM, taking the means3Hence i-vector52.1. Spoken Language and Dialect IdentificationFigure 2.1: A standard approach to calculating i-vectors (ω) [60], first capture sufficientstatistics and the covariance matrix (Σ) using a universal background model (UBM) trainedvia Gaussian mixture models (GMM), and randomly initializing a total variability matrix(T) which acts as an i-vector extraction matrix. Next, utterances are used to calculate the0th (N) 1st (F) and 2nd (S) order statistics matrices. Then apply expectation maximization(EM) to learn T, ω, and Σ. T and Σ are first frozen and used as part of the expectation(E) training step to calculate the mean and covariance matrices of ω. In the maximization(M) step ω is frozen and used to calculate T and Σ.62.1. Spoken Language and Dialect Identificationof each trained cluster and stacking them together. Meanwhile, the total variability pro-jection matrix can be randomly initialized. In this formulation we can use expectationmaximization [62] (EM) to update the weights of our total variability projection matrix,as well as the mean and covariance matrices of our i-vector based on our observed utter-ances. For inference, we can then use our UBM sufficient statistics and total variabilitymatrix to map an utterance to an i-vector [60]. To summarize the i-vector process, we canthink of separating each utterance into a background component, shared by all utterancesin the training set, and the divergence from this background. The divergence can thenbe viewed as the composition of a low-dimension vector specific to the utterance, and alearned projection matrix, which can be used to create low-dimensional representations forall utterances.While the i-vector calculation is fairly computationally intense, simplifications can bemade to reduce memory and computational cost with mild trade-offs to performance [76].For classification, a number of different techniques can be applied. This includes use oflogistic regression directly to the calculated i-vector [130], k-means [172], support vectormachines [61, 130] or similar methods. For practical applications the Kaldi speech recog-nition toolkit[150] can be used for i-vector extraction.An aspect of the i-vector approach that may impact LID, is the acceptance of speakerand channel information being modeled by the total variability space. While speakerinformation is unlikely to impact LID if datasets have thoughtfully sequestered train splitspeakers from their dev and test splits, channel characteristics might differ greatly betweendifferent regions (for instance with dialectal Arabic corpora [35]), and could impact the finalclassification. Unfortunately, this characteristic is shared by other approaches as well.Related to i-vectors are their DNN derived relatives, x-vectors, which similarly aimto create a fixed representation but go about it by eschewing GMMs and instead usingneural network architectures to learn the embedding [168, 169]. While application of x-vectors to LID simply use them in the same way as i-vectors (i.e., as embeddings deployedfor classification on an open-set classification task, where not all target labels are presentin the training set), they showed promising gains compared to i-vectors [136, 167]. In aclosed-set classification task, where all potential target labels are known, however, opensthe opportunity to do direct classification with the x-vector DNN network [167].2.1.2 Direct LIDWhile in open-set tasks, using an i-vector or x-vector embedding is necessary becausethe potential classification target may not have been seen during training, in closed-setclassification tasks, it is possible to use the x-vector networks directly to make an E2Eprediction. Early attempts to use x-vector networks in an E2E fashion highlighted theimportance of data quantity for training [167], as would be expected of any deep learningbased classification model. There are also similarities, between x-vector networks and E2Emethods in that training for both is essentially the same, and is conducted with cross72.1. Spoken Language and Dialect IdentificationFigure 2.2: In the original x-vector proposal for LID [61], the x-vector embedding is learnedby training a DNN system using cross entropy loss. In this setting the early layers of theDNN have some contextual width, through concatenating the t+ 2, t, and t−2 representa-tions from the first two layers. For x-vector systems, the x-vector itself can be taken as theoutput from one of the layers after the pooling function, or the system predictions fromthe softmax can be used for classification.82.1. Spoken Language and Dialect Identificationentropy loss on the output from the softmax layer [42, 46, 136, 167]. The main differenceis to consider whether the output from the softmax layer of the model is used directly,or if the representation prior to the softmax layer is being passed to another classificationmethod, such as probabilistic linear discriminant analysis [95]. In terms of performance,while early end-to-end LID was driven by desire to deal with the poor performance of i-vector systems on short utterances and met with mixed success [122], since then they havebeen shown to far outperform i-vector and x-vector embeddings approaches [42, 139].Direct LID can be made through a number of different deep learning architectures,the earliest being convolutional neural networks-time delay neural network (CNN-TDNN)architectures [138] which even predate i-vectors. Other architectures include directly us-ing the x-vector DNN network [167], directly using DNNs [121], pure convolutional neu-ral networks (CNNs) [122, 138, 139], recurrent neural networks (RNNs) [77], CNN-RNNmodels [109], attention based models [40, 75, 139], speech representation models (e.g.wav2vec [162]) with classification head [109], and learnable dictionary entry (LDE) [41].We can divide these efforts into two categories of approaches: (1) direct feature extrac-tion and classification via neural network and (2) neural network approximation of GMMi-vectors 4.The first approach (e.g.[40, 75, 109, 122, 139, 167]) follows direct classification usingx-vector networks (and could conceivably be used to thus generate x-vector embeddings).In this formulation, the network is responsible to learn what features are important for theLID task, and after passing through the network probabilities are assigned to each of thepotential classes using a softmax function. The network architecture is flexible in that itmight be a mixture of DNNs, CNNs, or RNNs. In addition, attention [23] may or may notbe used. Different models might compress the time dimension of the input utterance indifferent ways. For instance, for CNN-only networks, it is reasonable to use a max poolinglayer or temporal average pooling (TAP) [42, 122] while RNN based models can simply usethe last timestep of the RNN output [42, 75], with work also exploring using self-attention,a concept borrowed from [23], in the form of self-attention pooling (SAP) [42] (see Table2.1).The second approach [42] attempts to replicate the structure of GMM based i-vectors,through learning soft or hard cluster representations of the input frames and uses thesefor classification. In essence, if the first approach sets out to learn a D dimensional repre-sentation for an utterance, this second approach attempts to learn a D×C representation(where C can be thought of as corresponding to the clusters that one would learn fromapplying k-means or GMM [42]). The advantage of this approach is that it allows for softcluster assignment, and potentially a better representation of utterances.4Joint ASR and LID, such as [192] are beyond the scope of this work92.1. Spoken Language and Dialect IdentificationModel YearFeatureExtractionEncodingTechniqueClassificationCNN [138] 2009 mel-spectrogram CNN + Pooling -i-vector [61] 2011shifted deltacepstral (SDC)GMM SVM (cosine kernel)i-vector [76] 2011 MFCC GMM cosine similarityDNN [121] 2014 PLP DNN Linear LayerRNN [77] 2014 PLP RNN Linear LayerCNN [122] 2015 MFCC-SDC CNN + Pooling Linear LayerRNN [75] 2016 PLP LSTM -x-vector [167] 2018 MFCC, BNFs DNN Gaussian ClassifierTAP [41] 2018 FBank+CNN Temporal Averaging Linear LayerSAP [42] 2018 FBank+CNN Self-Attention Linear LayerRNN [41] 2018 FBank+CNN GRU Linear LayerLDE [41] 2018 FBank+CNN LDE Linear LayerCNN [139] 2019 MFCC+CNN Self-Attention Linear Layerx-vector [109] 2020 - DNN DNNCNN-RNN [109] 2020 CNN LSTM Linear Layerwav2vec [109] 2020 wav2vec - Linear LayerTable 2.1: Chronological list of papers in this section categorized according to their LIDpipeline. Feature extractors take an input utterance and convert it into F × T size vector,where F is our feature dimension and T are the number of frames in our utterance, theencoding technique is how the extracted features are summarized. Blanks are present wherethe original papers are ambiguous or do not mention relevant details.102.1. Spoken Language and Dialect Identification2.1.3 Evaluation MetricsSeveral metrics (see Table 2.2) are used in the literature to evaluate performance of LIDmodels. Accuracy is the percentage of correctly predicted utterances. Precision is ameasure of performance confidence, indicating the ratio between true positives and pre-dicted positives. Recall measures the percentage of true positives that were returned, andthus expresses how exhaustive a model is. Finally, among these basic metrics F1 is used toexpress the combination of precision and recall, and can be calculated on a per-item basis(micro-F1) or a per-class basis (macro-F1), with the former treating all utterances thesame, and the latter calculating F1 for each class before averaging them together. Borrow-ing from biometrics, Equal Error Rate (EER), is also used, and indicates the point onthe receiver operating characteristic (ROC) curve at which false positive and false nega-tive rates are equal. A final metric introduced is Cost as defined by the NIST LanguageRecognition Evaluation (LRE) standard [159], which is an aggregated metric that penalizesfalse positives and missed positives, and seeks to summarize an overall performance of themodel. What metric a study uses might depend on the history of the particular benchmarksets being used.Metric FormulaAccuracy TP+TNTP+FN+TN+FPPrecision TPTP+FPRecall TPTP+FNF-measure (F1)2×Recall×PrecisionRecall+PrecisionEqual Error Rate (EER) ROC curve where FP = FNNIST NRE Cost for LT , LN [159]PFN (LT ) + βPFP (LT , LN ),where β = CFP (1−P (LT ))CFNP (LT )Table 2.2: Metrics used for Language Identification. We use the following terms: truepositive (TP), false positive (FP), true negative (TN), false negative (FN), for the Costfunction, we give the normalized Cost function for a language pair target language (LT )and non-target (LN ) and C is the cost penalty (1 for both FP acceptance and FN missesin the NRE evaluation).112.1. Spoken Language and Dialect Identification2.1.4 Section SummaryThe first breakthrough in LID can be traced to the performance gains from borrowingspeaker recognition models, such as i-vector, and using them in a LID context. Use ofDNNs to extract an x-vector representation improved upon the original i-vector, but sinceLID is often a closed task this has given way to E2E DNN models which can classify directlyusing a softmax function to normalize the outputs into a probability distribution over thepossible classes. Finally, recent advances have come from experimenting with the poolingtechniques used to summarize the input utterance.2.1.5 DiscussionThe close ties between speaker recognition and LID raise potential questions about thereliability of LID models. In some sense the assumptions made in the initial formulationof the i-vector model, that channel and speaker information can be jointly modeled, isstill followed by modern E2E methods, as there is no attempt to extract or compensate forchannel characteristics in modern work. Lack of discussing interpretability of DNN models,with a few exceptions such as [50], raise questions about how these models work. Could itbe that they are very good at picking up on geographic proxy information in the recordingssuch as mains hum, which would encode the frequency of the electric grid of the countryin which it was recorded? Or similarly, reflect a distribution in which microphones arecommon in certain countries, or by use of certain organizations? While i-vector models maycontain such channel information in the i-vector embedding, the i-vector’s small dimensionswould potentially limit the impact of this on the final classification, yet there is no suchbottleneck in an E2E system, potentially exacerbating the impact of channel characteristics.Could a fixed representations model avoid this pitfall by relying on non-channel basedfeature?Finally, though LID models seem ideal for small scale use for instance on a smart phone,many models in this section rely on performance alone as a performance metric, with nonethat attempt to evaluate model size and training time. Little discussion of the cost oftraining these models, reflect a field out of step with major environmental concerns in thecost of training DNN models.To recap our discussion, we reflect on the following questions:1. Is the assumption of total variability space, that channel and speaker informationcan be modeled together, appropriate for LID, where channel characteristics mightbe country (e.g. powergrid) dependant?2. Are fixed representations able to be created ‘channel independent’?3. Why has there seemingly been little effort to investigate efficiency improvements (in-cluding model size and training time) for a component that may need to be embeddedinto small devices or as part of a pipeline?122.2. Automatic Speech Recognition2.2 Automatic Speech RecognitionWhile our study focuses on the task of ADI, the techniques we use build from methodsfrom not only traditional LID and DID but also ASR. ASR, the task of transcribing aspoken utterance into written form. These tasks are closely tied, as one must know whichlanguage or dialect is being spoken prior to application of the appropriate ASR model,failure to do so would result in gibberish. As with our survey of LID, in this section weaim to explore the breadth and history of ASR.ASR can be defined as the task of taking an input speech utterance, x, and mapping itto the transcript of words or characters that it contains, y. Two major approaches to thistask can be delineated: generative models, which aim to model the joint probability of bothutterances and transcripts p(x, y), and discriminative approaches, which aim to predict theconditional probability of a transcript given a speech utterance p(y|x) [98]. Generally hid-den Markov model (HMM) approaches fit in the first category (although discriminativetraining of HMMs is possible), with DNN based E2E models in the second category. Theseapproaches have generally been supervised learning, that is with the transcription, y, beingused during training. However, following the wav2vec architecture [162], there has been anincreasing trend in self-supervised approaches, relying only on unlabeled speech utterances(just x) for pre-training, and then fine-tuning the model in a supervised fashion. In thissubsection, we explore these three main trends in ASR architecture, HMM based gener-ative models, purely supervised end-to-end learning approaches, and self-supervised deeplearning approaches.2.2.1 HMM ModelsThe earliest success in ASR applied HMMs [152]. These models used HMMs to model thesequence of phones in an utterance with Gaussian mixture models (GMMs) to model thecontinuous probability density of emission of a phone at each state [198]. These GMM-HMM models rely on having aligned phonemic transcriptions of the audio training data,such as the Texas Instruments/Massachusetts Institute of Technology (TIMIT) Englishspeech corpus [74].Using an DNN to improve the performance of phone emission probability estimationlead to hybrid DNN-HMM models [179]. Like the GMM-HMM models, they also rely onaligned phonemic transcriptions, however, improve in accuracy over the GMM architec-ture [179]. While these earlier hybrid DNN-HMM methods used multi-Layer perceptrons(MLP) as their DNN [37], later models began switching this out for recurrent neural modelssuch as bidirectional long-short term memory (bLSTM) [81, 91]. For simplicity we referto all hybrid models that use a neural network for acoustic recognition and an HMM forsequencing as DNN-HMM models.Because DNNs are trained using back-propagation, and HMMs are trained using theexpectation maximization (EM) algorithm, they must be trained separately, as opposed132.2. Automatic Speech Recognitionto in an end-to-end fashion. Training separately is potentially problematic in that errorsmade by the acoustic model, might then be propagated to other components of the model,leading to decreased performance.2.2.2 End-to-End ASRBecause of the difficulty of assembling TIMIT style datasets for GMM-HMM or DNN-HMM model training, attempts to move towards automatic alignment of transcriptionsbrought the field to connectionist temporal classification (CTC) [79, 80] models.The CTC algorithm allows text to be aligned purely from transcribed training data, nolonger needing pronunciation models or significant annotation of corpora to be possible.This advancement has particular benefit for languages, such as Arabic, where the precisepronunciation must either be indicated through diacritics or inferred from the orthographicform.While CTC was first introduced in 2006 [79], major advancements in CTC architecturesonly started in the mid 2010s with the introduction of modifications of beam search forCTC [80, 89], and large-scale models such as Deep Speech [88] and Deep Speech 2 [15].These early models illustrated some of the potential benefit of deep learning approaches,the ability to deal well with noisy data [88] as well as ability to scale well with additionaldata [15, 80]. Under a dictionary decoding scenario, Grave’s E2E model went from 69.7%to 21.9% word error rate (WER) when going from 14 to 81 hours of training data, whilea comparison DNN-HMM system only improve from 56.1% to 51.1% [80]. Similarly DeepSpeech 2’s data ablation showed significant gains from training on more data, with aWER of 50.97% on their noisy dev set being brought down to 13.59% by increasing thedata quantity from 120 to 12000 hours [15]. These significant advances from using largequantities of data, are mirrored in the growth of self-supervised models, which will bediscussed at the end of the chapter.While CTC models allow you to align to outputs by over generating tokens and nar-rowing them down by removing duplicates and blank tokens, attention mechanisms canalso be used to perform this alignment task. Attention, first pioneered in machine trans-lation [23], is a method for learning to emphasize certain parts of the input sequence,which can then be used in the decoder of a encoder-decoder architecture. Early effortsof applying attention models to ASR found that it was fairly straightforward to adaptencoder-decoder models to ASR by using the attention scores to directly align the out-put tokens. [47] An interesting aspect of using attention is that it is easy to visualize thealignment of characters using the attention weights, providing an important look into thefunctioning of the model. However, challenges faced by these new models include difficultyon long utterances, as alignment can break down past a certain length [48]. One attemptto solve this was by modifying the attention mechanism to take into account the previousstep’s attention, combined with attention smoothing techniques [48], another approach isthrough the use of a pyramidal bidirectional long short-term memory (LSTM) to reduce the142.2. Automatic Speech Recognitiontemporal resolution thus improving the effect of the attention mechanism [45]. This secondapproach, termed Listen, Attend, and Spell (LAS) poised itself as the first ‘end-to-endmodel’ because it essentially learns a language model during training, alongside acousticand pronunciation models. While LAS made claim to being the first ‘end-to-end model’,the inclusion of an external language model, while significantly improving the performanceof the model, seems to demonstrate that such goals, a fully internalized language model, areunlikely given how little labeled audio training data is generally used. A work-around forthis was discovered with SpecAugment, a data augmentation technique that functions bymasking bands of frequency or time on the input data [146]. Using this data augmentationtechnique enabled scaling up the LAS architecture, training it for longer, and reaching apoint where it could perform equivalently without an external language model [146].A third category of E2E model is the RNN-Transducer [78], which aims to use adouble RNN network consisting of a language model prediction network acting on the labelsequence, and a transcription network acting on the input audio. This network generatesa probability density lattice of all labels at each timestep in the sequence, which can thenbe decoded using a variant of the forward backwards algorithm. Improvements the basicmodel involve replacing exponentiation operations in the original model with feedforwardnetworks, and reducing the memory footprint [114].While CNNs are often only components of ASR models, a collection of models uses themas primary component. These ConvNet models, such as ContextNet[87] and QuartzNet [110],are lightweight alternatives to other models, due to the ability of CNNs to effectively down-sample input representations from layer to layer as well as depthwise separable convolutions,which enable breaking up large standard convolutions into two significantly smaller ones. 5Thus these mainly CNN based approaches have the appeal of being able to perform wellat very small parameters sizes.While many models focus only on one of either CTC or Attention for alignment, dueto their independence it is possible to use them together to improve performance, in otherwords a Hybrid CTC-Attention model [92, 193, 196, 201]. For instance using a sharedencoder that feeds outputs independently to a CTC network as well as to a traditionalattention mechanism, can counteract some of the issues with pure attention based modelssuch as early termination of predictions [193]. Finally, the Transformer model [180], whichemerged from text-based NLP has been adapted for use with speech. The main improve-ment to allow Transformers to work for ASR was using convolutional layers to learn contextinstead of sinusoidal encoding [137]. In addition similar to hybrid CTC-Attention systems,The success of just adding the CTC loss to help with alignment is similarly replicated withspeech Transformer models [102, 103]. Taking inspiration from the Transformer as well asleveraging the efficiency gains of ConvNet models, the Conformer [83] has since become5In detail, for a series of C convolutions with widthK on audio withD input features, you would normallyuse (CKD) parameters, but this can be broken into two steps, first a D separate K × 1 convolutions, onealong each of the D features, and then followed by C different 1×D convolutions that convolve just acrossthe feature dimension, leaving you with D(K + C) << CKD parameters used.152.2. Automatic Speech Recognitionthe standard state of the art model for ASR trained from scratch. This modifies the tra-ditional Transformer architecture by adding a convolutional module and Transformer-XLstyle relative position multihead attention (see [57]). The benefits of this approach includeextremely competitive performance on Librispeech even with extremely small model size(10M parameters compares favourably with the best ConvNet-based approach, and largermodels outperform LAS and other transformer based systems), even without a languagemodel. While it uses an encoder-decoder framework, the chosen decoder is an extremelysimple single layer LSTM.2.2.3 Self-supervised ModelsJust as HMM models were largely abandoned due to difficulty in assembling phoneticallyaligned corpora, the need for larger models trained with even more data has illustratedthe limits of using purely supervised learning approaches. An approach to deal with thispotentially lies in representation learning, whose goal is to learn good ‘representations’(probability distributions of pertinent features) of underlying data patterns in an unsuper-vised way that can then be used to condition a supervised classifier [26]. In a representationlearning framework, models are trained on unlabelled data using a self-supervised objec-tive function, a simple example of this being an input sample being partially corruptedand the model tasked with predicting the corrupted portion. In the extremely success-ful BERT model [63], this corruption entailed masking tokens of an input sentence andthen predicting those masked tokens. Many of the models in this section differ in subtleways in how these models construct their self-supervised objective function, with Table2.3 giving a high level overview of these speech representation models covered. Becausethe representation is task agnostic, the speech representation models in these sections arepotentially just as applicable to LID as they are to ASR. For example, while not LID, manyof these early models evaluate performance on speaker recognition [52, 144]. However, ASRhas been a primary motivator for all of these models and benchmarking on LID/DID isunderdeveloped.Self-supervised objective functions can be broadly categorized into contrastive objec-tives or reconstruction objectives, with the former attempting to discern between correctand incorrect segments that might fit in a masked area of the audio [144], and the latteraiming to directly create an unseen portion of the audio [52]. Finally, a third category ofmasked prediction objectives is possible by using a pseudo-labelling step to assign tokenIDs to timesteps, which are then used as training targets for the model [93].Contrastive models, such as Contrastive Predictive Coding (CPC) [144], aim to learn arepresentation of the data by having the model pick frames from amongst a set of distractorframes. For CPC, this is done in an autoregressive fashion. Autoregressive models use pastobservations to predict future timesteps, that is to say having seen up to timestep xt predictwhich frames correctly occur in timestep xt+k, that is to say k steps in the future, amongsta series of distractors pulled from the same distribution. Instead of the input frames, x,162.2. Automatic Speech RecognitionFigure 2.3: An overview of the three main types of speech representation models for self-supervised learning: Masked Prediction models, of which we use HuBERT[93], Contrastivemodels which we use wav2vec 2.0 as an example [22], and Reconstruction models typifiedby DeCoAR 2.0 [116]. For HuBERT we note that training is iterative, and that the k-means clusters come from a prior iteration of the model (with MFCC features used in thefirst iteration).172.2. Automatic Speech RecognitionModel YearInputFeaturesQuantizationPrimaryNeural NetObjectiveCPC [144] 2018 wav No GRU CAPC [52] 2019 log-mel No LSTM Rwav2vec [162] 2019 wav No CNN Cvq-wav2vec [21] 2019 wav Gumbel-Softmax CNN Cvq-APC [53] 2020 log-mel Gumbel-Softmax GRU RDeCoAR [117] 2020 log-mel No LSTM RMockingjay [118] 2020 log-mel* No Transformer Rwav2vec 2.0 [22] 2020 wav Gumbel-Softmax Transformer CDeCoAR 2.0 [116] 2020 log-mel Gumbel-Softmax Transformer RHuBERT [93] 2021 wav k-means Transformer Pw2v-BERT [54] 2021 log-mel Gumbel-Softmax Conformer P + Cdata2vec [20] 2022 wav No Transformer R†Table 2.3: An overview of select speech representation models, in rough chronologicalorder. Log-mel refers to log-mel spectrograms, which are 80 mel log-mel spectrogramsunless noted. Objective function is either contrastive loss (C), reconstruction loss (R), ormasked prediction (P). *Mockingjay experiments with mel scale spectrograms as well aslinear scale spectrograms as input. †data2vec reconstructs the average hidden state of ateacher model instead of the input frames directly.182.2. Automatic Speech RecognitionCPC uses a lower-dimension representation z for this task, which are the frames after beingpassed through a CNN feature extractor. Using a GRU to summarize prior latent framesinto a context vector c, CPC then uses a bilinear model to estimate the correct latentframe, finally it uses a variant of noise contrastive estimation (NCE) for its loss.L = −EX[logfk(xt+k, ct)∑xj∈X fk(x, ct)](2.2)Equation 2.2, is their formulation for NCE loss, which represents the log-likelihood ofpicking the correct latent frame out of all distractor frames, and Equation 2.3 gives thebilinear model for predicting a frame zt+kfk(xt+k, ct) = exp(zTt+kWkct) (2.3)Other contrastive models follow a similar pattern, with noticeable differences. First wav2vecswitches the GRU of CPC to a CNN based context network and experiments with using thecontext outputs as input to an acoustic model for ASR [162], this is followed by vq-wav2vecwhich adds a quantization scheme (see 2.2.4) based on Gumbel-Softmax [97] and demon-strates success training a BERT model on the quantized outputs [19, 21], finally wav2vec2.0 abandons the autoregressive training in favor of a contrastive loss based on predict-ing the quantized representation of masked segments using a transformer context network[22]. These gradual improvements to the core idea of contrastive training ultimately leadto unheard of performance on the test-other split of Librispeech, with a massively self-trained model hitting 3.1% WER using only 960 hr labeled data and 5.2% WER usingonly 10m of labeled data [197]. The core success of the wav2vec 2.0 model set the stagefor several other models such as HuBERT [93], which also uses the same architecture, andw2v-BERT [54], which uses the same constrastive loss and similar quantization.Just as contrastive models have evolved significant over a short period of time, recon-struction models have followed a similar trajectory, sharing similar ideas with their con-trastive brethren. The first of these models, autoregressive predictive coding (APC) [52],is formulated in a similar way to CPC, except instead of predicting a target latent repre-sentation amongst a set of distractors, it directly aims to generate a future timestep. Thisis done simply with a L1 loss function 6 based on the difference between the predicted andinput frames (see Equation 2.4).L =T−k∑t=1|xt+k − yt| (2.4)In this formulation, y is our predicted frame based on x input, where we are predictingk frames ahead. Following APC, vq-APC [53] adds quantization layers using Gumbel-Softmax, as a way to limit model capacity, and allow examination of the quantized rep-resentations to better understand what phonetic information is learned by the models.6L1 loss is generally used because it is less sensitive to outliers [116]192.2. Automatic Speech RecognitionWhile, APC and vq-APC use an autoregressive task to learn representations, an alter-native approach is used by DeCoAR [117], which adapts ELMo’s bi-directional languagemodeling [148] to apply to speech. In this approach forward and backwards passes of the in-put data can be used to reconstruct a masked portion of the input data. DeCoAR 2.0 [116]improves this by replacing the biLSTM that DeCoAR used with a Transformer and addsa Gumbel-Softmax quantization function just prior to the reconstruction layer. DeCoAR2.0 can be thought of as a mirror of wav2vec 2.0, with the exception of using Filterbankinputs and utilizing a reconstruction loss, with similar performance on semi-supervisedtraining [116]. One similar model to DeCoAR is Mockingjay [118], which instead takes amasked language model (MLM) [see BERT [63]] approach and adapts it to reconstruct-ing the frames of the input. A final reconstruction based model is the recent data2vecarchitecture, which aims to create a unified method for training representation learningmodels regardless of modality [20]. Instead of reconstructing masked inputs like many ofthe other models in this section, data2vec instead aims to use a teacher model to generatelatent representations from passing in unmasked input and then a student model (in realitythe same model put in training mode) to predict the average latent representations of themasked portion of the same input. Because all predictions are done in the latent spaceit is an extremely easy method of performing unsupervised pretraining, with no need forquantization or pseudolabeling to generate useful targets. Finally, the model comparesfavourably to wav2vec 2.0 under a 12-layer model configuration, and equivalent to bothwav2vec 2.0 and HuBERT with 24 layers [20]. While other reconstruction models useL1 loss functions, data2vec uses a smoothed version, which uses L2 loss under a certainthreshold β, surprisingly for speech recognition tasks, using an L2 loss by itself was foundto be reasonable [20].L ={12β (yt − ft(x))2 |yt − ft(x)| ≤ β|yt − ft(x)| − 12β Otherwise(2.5)Here ft(x) is the averaged latent state of the teacher model.Finally, there are models that rather than using contrastive loss functions to pick labelsbetween a set of noise, and reconstruction loss functions to actually generate an inputstate, instead use a quantization method to generate the labels for use in a prediction task.Most notable in this HuBERT [93], which uses a k-means pseudo-labeling process, andw2v-BERT [54], which uses wav2vec 2.0’s quantized states as labels for a MLM task.2.2.4 QuantizationWith the increase in self-supervised speech representation models, quantization is a logicalchoice to allow for greater model control compared to operating on purely continuous data.For instance, in the case of vq-APC [53], adding quantization allows for control over modelcapacity (as the model can only rely on the bottleneck features from the quantization),202.2. Automatic Speech Recognitionwhich can aid in interpretability by allowing direct inspection of alignment of quantizedunits and phonetic elements. More importantly though, is the empirical benefit to perfor-mance of the models in downstream tasks from using quantized units [19]. In most casesself-supervised models opt for using a Gumbel-Softmax approach to quantization, this isa fully differentiable approximation to the argmax 7 function and can be thought of as away to incorporate categorical data as part of a neural network model [97]. An alternativeto the Gumbel-Softmax is to use either a k-means [21, 93] or GMM clustering step [93],with k-means performing roughly equivalent to Gumbel-Softmax when used as part of thevq-wav2vec training. While HuBERT opts to use k-means over GMM for its clusteringstep, GMM is shown to be quite effective when compared to a k-means clustering of thesame number of clusters [93].2.2.5 MetricsUnlike classification tasks like LID, ASR has fewer metrics to consider, namely: WordError Rate (WER), Character Error Rate (CER), and occasionally Phone ErrorRate (PER). WER is measured by totalling the number Levenshtein edit operationsbetween the targets and predictions and dividing by the number of words in the corpus [99].WER = 100× I +D + SN(2.6)For I, insertions, D, deletions, and S, substitutions, in a corpus of N words. CER andPER are similarly calculated at the level of character and phone respectively.For unsupervised training, there is less uniformity, and studies may relay performanceonly on downstream tasks [22, 118, 144], use probing classifiers to quantify informationcontained in layers [52, 53], or in the case of HuBERT, propose new metrics [93]. Themetrics proposed by HuBERT are phone purity, a measure of frame-level phone accuracy,cluster purity, how likely all phones of a certain class are assigned the same clusterlabel, and phone-normalized mutual information, which measure the relative drop inentropy regarding a phone from observing a particular label. Unfortunately, since thesemetrics are all based on using k-means clustering for target generation, of which onlyHuBERT uses, they have limited applicability.Finally, the Zero Resource Speech Benchmark [141],includes a number of metrics specifi-cally to deal with unsupervised representation learning. Unfortunately, the evaluation toolsused for the metrics use English corpora, such as Libri-Light. While this may be justifiedfor demonstrating overall model architecture performance, it seemingly plays into the ex-isting English bias of NLP [25], all in the name of potentially improving performance forlow resource languages. Despite this, ignoring metrics that focus on learning lexical or7argmax is defined as a function that takes another function and returns the argument that max imizesthe input function212.2. Automatic Speech Recognitionsyntactic information, which won’t be useful for evaluating non-English speech representa-tion models, it may still be potentially useful to use ABX [161]. This metric measures amodel’s ability to match a phone triplet, X (ex. ’cat’ k æ t) with its minimal pair amongsttwo options, A (ex. ’bat’ b æ t) and B (ex. ’got’ g A t). Even though the evaluation setcomes from an English corpus, it is still likely to be useful as a rough estimate of phoneperformance for non-English languages.2.2.6 Section SummarySimilar to the progression of LID research, ASR has moved to deep learning methods overthe last decade, with even DNN-HMM models fully replaced by E2E DNN models in recentyears. A new trend in ASR research has come in the form of representation learning, whereself-supervised learning objectives allow unlabeled training of datasets. This new paradigmof these new models can be summarized as follows:1. The ability to train on massive raw audio corpora2. Use of large transformer-like architectures3. Self-supervised pre-training which generalizes well to downstream tasksUltimately, with the success of representation learning, most ASR approaches will need toadopt either a pre-training step, or fine-tune from a pre-trained model to be competitive.2.2.7 DiscussionWe focus mainly on the takeaways from the recent speech representation research, as it ismost applicable to LID.While all of the speech representation research has reported results on ASR, few if anyhave evaluated on tasks such as speaker recognition or LID, leaving a gap in both theoreticalresults and practical know how (which models? how to fine-tune?). Understanding thesegaps, however, could be extremely computationally expensive. Full pre-training of a singleself-supervised model may be cost prohibitive (multiple months of GPU time); fine-tuningis significantly less, but still may take upwards of month of GPU time, when consideringhyperparameter selection of fine-tuning settings. This raises questions about what methodscan help ensure sufficient optimization of models, especially where no fine-tuning recipesexist.I would observe that if pre-training is destined only for the realm of large companies,that have large compute clusters to spare, there is little leeway to investigate the choicesmade by these companies from an external researcher perspective. Choices such as modelsize, pre-training language choice, and pre-training method are now potentially left in thehands of researchers at Meta (if one wishes to use a pre-trained wav2vec model say). Thisleaves many researchers reliant on tech companies to have their best interest at heart, yet222.3. Arabic Linguisticsfor some tasks, this may not be the case. As a concrete example, for low resource speechtranslation truncating a pre-trained model in half can be extremely beneficial [34]. Thiscounter-intuitive result, driven by the idea that the wav2vec pre-training leaves later modellayers learning abstractions of the data that are not useful for practical tasks, implies itcould be generally beneficial. But desire by corporate research teams to make larger-and-larger models, optimized for standard benchmark datasets, may leave blindspots to suchefficiency measures.Similarly, independent researchers must contend with available pre-trained models, hav-ing to pick and chose between different configurations that have been released, leading togaps where no models might exist. For models pre-trained on Arabic, the xls-r-300m modelis the only option released from Facebook AI research, and it unfortunately consists of onlyMSA training alongside multilingual training. Finally, released models may be pre-trainedon vastly different domains, thus raising questions of whether pre-training and fine-tuningdomain mismatch matters?We leave this section with the following questions:1. Which models perform best on LID?2. How to fine-tune and use speech representation models for LID, especially with afixed compute budget?3. How much does multilingual pre-training matter?2.3 Arabic LinguisticsAccording to Ethnologue, there are nearly 350 million speakers of Arabic covering 29different varieties [65], showing not only the importance of building NLP tools that workwell for this large population but also hinting at the challenge of developing these toolsto account for such a variation. Arabic is a Verb-Subject-Object (VSO) language notedas having a non-concatenative, morphologically rich, structure, with a tendency to droppronouns [72].Compared to English phonology, Arabic is differentiated with velarization of some con-sonants, use of pharyngeal, uvular, and velar consonants, but simple vowel structure (threevowels that can occur in short or long form) [158]. Phonological differences are especiallyimportant in exploring code-mixed data, for instance when English words are embeddedin an Arabic utterance, the L2 English pronunciation will reflect differences between voweland consonant inventories of the two languages, to give an example, an open mid-frontunrounded vowel (ε) may instead be pronounced as an open front unrounded vowel (a)and other vowels might similarly map to the closest Arabic counterpart [195].Morphologically, however, Arabic stands apart from many other languages through theuse of a root-pattern system, where words are formed by a consonantal root, often with a232.3. Arabic Linguisticsgeneral sense of meaning, and then given specific meaning through the pattern of vowelsoverlaid on the consonants [158].While a standardized variety, Modern Standard Arabic (MSA), is widely taught andused in many situations such as formal speeches and news programs. The authors of theMGB-2 Broadcast TV corpus estimated that as much as 30% of the audio they collected wasdialectal Arabic (DA) rather than MSA [9], further studies on presumably monodialectalArabic varieties have also shown the prevalence of code-mixing, the intrasentential useof more than one language or dialect [51]. Both of these factors, correct utterance-leveldialect identification and code-mixing pose challenges to collecting and validating Arabicaudio datasets in any given variety, as well as building systems relying on these corpora.There are significant differences between MSA and DA at multiple linguistic levels [112].In terms of quantitative analysis, few studies have aimed to quantifying the linguisticdistance between dialects, with exceptions for studies of lexical distribution [13, 90, 112,173], and pitch [134]. Qualitative discussion of these dialects, however, can illuminatefeatures that might be important for the Arabic speech processing. Thus, in this sectionwe aim to summarize linguistic research of differences between DA and MSA on a set ofdifferent linguistic levels: phonological, morphological, and lexical.2.3.1 Phonological DifferencesWhile DA retains many of the same phonemes of MSA, significant differences may existbetween different dialects. To examine the diversity of DA, we look at the voiceless uvularstop (IPA q), which is represented by the letter qaaf () in MSA. For instance, in San’aniArabic (a dialect of Yemen) it has been replaced by a voiced velar stop g, while in EgyptianArabic, it is restricted to religious texts and MSA words [194] or replaced with a glottalstop P. A study of the pronunciation of qaaf in 18 countries found that while it can takethe form of q, g, or P, the distribution of these does not neatly fit into broader regionalcategories and may in fact vary within country [24].Thus while these pronunciations might be clues to potential dialects (ex: use of Pwould eliminate Gulf Arabic as a possibility), the in-country variation calls into questionthe ability to rely on phonology alone to create ADI tools that work across a country.This may also impact corpora creation, as interviews and broadcast news may only reflecturban dialects of a country, potentially biasing models against speakers from rural or less242.3. Arabic Linguisticsrepresented cities.2.3.2 Morphological DifferencesArabic morphology includes variation in root and stem pattern as well as inclusion of af-fixes [194]. While these variations are often simplifications from MSA, they can also bemore complex innovations of the dialect[84]. For the root-level morphology, set forms ofthe verbs can occur, and may be different, both in terms of form and distribution, betweendialects [194]. For instance, in Sa’ani form III of the verb ‘to do’ is ‘fa¯Qal whereas inEgyptian this is ‘fa¯Qil [194]. Similarly, while Sa’ani and Egyptian both have the samestructure for form IV, this is rare in Egyptian [194]. DA having unique forms, as well asdifferent distributions of forms used, presents challenges for Arabic speech processing asthe multitude of different variations lead to potentially large numbers of out of vocabulary(OOV) words. Because most models require specifying the number of words to considerduring training, the vocabulary, when the number of words considered is too small, or donot cover words in the test set, these would be OOV and lead to potential poor perfor-mance of the model. Morphological analyzers such as Columbia Arabic Language anddIalect Morphological Analyzer (CALIMA) [85] aim to bridge this gap by providing toolsto generate correctly diacritized forms of Egyptian DA, however, not all dialects have suchsupport.From the perspective of non-transcribed Arabic speech processing and ADI, morpho-logical differences between dialects can potentially be picked up on with enough data in adeep learning framework. However, such quantities of data are unlikely, and without seeingenough variations of the structure, it is unlikely to act as a significant signal in and of itself.Use of transcribed corpora with careful attention to unique verb forms, could potentially beused to test ADI models for capacity to understand the morphological structure, howeversuch corpora would have to be created from scratch.2.3.3 Lexical DifferencesLexical differences abound between Arabic dialects [2, 3, 84], and are one of the few areasthat have been used to estimate metrics of similarity between dialects. While choice ofmetric influences the resulting similarity, generally analysis across different metrics andacross several different corpora indicate Levantine dialects, such as Palestinian and Jorda-nian, are closest to MSA, and dialects within the same regional classification are similarto one another [3, 112]. Lexical analysis can also be used to understand the evolution ofthese dialects, for instance many Moroccan Arabic words can be traced back to old Arabicwords which are no longer used in MSA [173], indicating that the history of these dialectsmay have major impact on their similarity or dissimilarity to each other. Finally, attemptsto map lexical similarity to geographic distance provide evidence that proximate dialectsare more similar, regardless of country boundary [13], however, methodical issues, such as252.4. Arabic Speech Processingunexplained exclusion of certain high correlation city pairs, seem to undermine the strengthof such findings.2.3.4 Section SummaryWhile we only examine a sample of the linguistic differences between different varieties ofArabic (and wholly ignoring important aspects such as syntax and semantics), it is evidentthat there is significant variation between these varieties, across a number of linguisticlevels:1. Phonological differences, such as use or absence of glottal stops, may manifest betweendialects as well as within dialects (rural vs. urban).2. Morphology poses significant challenges due to significant number of potential wordformations, and lack of available tools for DA.3. Lexical differences may change more gradually over regional areas.2.3.5 DiscussionUnderstanding differences between dialects is important, as tools designed for MSA oftenfail at processing DA [151, 173], and so a general measure of distance (on different linguisticlevels) could aid research in cross dialect adaption of tools. We emphasize that they do notappear to be convenient aspect that can be used solely to predict dialect due to similaritybetween dialects, and differences within country-level expressions of language (e.g. ruralvs. urban), and thus deep learning approaches seem appropriate as a means of automaticfeature selection of important characteristics. The country-level variation of dialects, alsoraises important points in the corpus creation process as lack of inclusion of all segmentsof a society, could lead to integrating bias into models trained from such corpora.We conclude this section with the following thoughts:1. With such similarity between many dialects, is there risk for LID models over-fittingto non-linguistic material present in the dataset?2. While linguistic differences exist at all the levels we examined, which level of featuresare most important to DNN based models?3. Do datasets sufficiently include variation in dialects covering different cities and rurualareas?2.4 Arabic Speech ProcessingAttempts to process Arabic speech data have been ongoing since the mid 90s, with earlybaseline models being set by BBN technologies’ Byblos ASR system [200]. In the early262.4. Arabic Speech Processing2000s, a John Hopkin’s University workshop on Arabic speech processing illustrated thegreatest challenges when dealing with Arabic: “the predominance of non-diacritized textmaterial, the enormous dialectal variety, and the morphological complexity” [107]. Thusmuch effort in Arabic speech processing has dealt less with pioneering acoustic modelingsystems or architectural changes to the overall system, but rather tackling the problemsinherent in such a diverse and linguistically challenging language. We present in thissection an exploration of Arabic speech processing approaches mainly focusing on theselarge vocabulary continuous speech recognition models and how researchers have attemptedto tackle the major challenges of processing Arabic: diacretization, common inclusion ofDA alongside MSA, and morphology. Following this, we explore specific papers that havetackled Arabic ASR and ADI.2.4.1 DiacretizationEarly research efforts in Arabic ASR focus heavily on how best to approach diacretiza-tion, which is to say whether or not vowelization is marked in text 8. Since GMM-HMMASR models use pronunciation dictionaries to map written text to phonetic representation,ambiguity in pronunciation of written text can lead to significant issues with the overallmodel performance. For Arabic text, diacretized texts are rare and generally not used bynative speakers, while ASR systems trained on non-diacretized text might lack the contextto distinguish between several different forms of a word, conversely romanized texts (forinstance Arabic written in Buckwalter transliteration) require special training to read, andare unfamiliar to native speakers [107]. Since most text is non-diacretized, diacretizationmodels can be used to add it into the training data, while adding it may seem like astraightforward solution to this, depending on the genre and training of the diacretizationmodel, it could potentially hinder results [31]. While this line of inquiry was significantin early Arabic ASR research [143, 171, 181], movement away from GMM-HMM models,which required explicitly modeling all pronunciation, towards E2E ASR models (see [94],which requires no diacretization pre-processing whatsoever), has alleviated the need to usediacretized training text.Finally, despite the amount of effort devoted to this problem in Arabic ASR, for spo-ken LID/DID, diacretization is almost completely irrelevant, as generally there are notranscriptions of the utterances available, and instead phone-based language models canbe used for explicit language modeling [1, 119, 131, 165].This is not to say that research on diacretization is completely useless, we note brieflythat for text-to-speech, a topic outside our set scope, diacretization is vital to disambiguatethe precise pronunciation of utterances [175]. However, it is unlikely to be revelant to ASRor DID research in the future.8Since vowelization and diacretization are directly linked, we choose to use the term diacretization torefer to both.272.4. Arabic Speech Processing2.4.2 Dialectal ArabicFew dialectal Arabic corpora exist (see Section 2.5), and thus building tools for dialectalArabic is hindered by the simple lack of data. Furthermore, tools that might be designedfor dialectal Arabic, for instance to diacretize text, might rely on MSA training datapotentially lessening their effectiveness [31]. Related to this is the fact that several largeMSA corpora containing significant dialectal Arabic or codeswitching [9, 156], implyingthat performance on MSA is also potentially hindered by lack of dialectal Arabic tools anddata. Ultimately the goal though should be to develop models that can tackle both MSAand dialectal Arabic in an ensemble fashion, and work in combining systems to do bothDID and dialect specific ASR show promise [170].In addition to lack of data, a major challenge with dialectal Arabic is the lack ofstandard orthography [11]. One approach to this is to simply collect multiple transcriptionsof each sentence, and utilize a multi-reference (MWER) metric [11]. While, this may bethe only logical way to handle a lack of agreed upon standard, it presents a potentiallysignificant burden in the form of requiring multiple transcriptions on each utterance.In terms of tackling the issue of dialectal Arabic, the MGB challenges has so far intro-duced benchmark sets for Egyptian as well as Moroccan [10, 11], which we discuss furtherin Section 2.5.2. In the original MGB-3 challenge the Aalto System, a combination of us-ing TDNN acoustic model with RNN Language model rescoring and minimum Bayes riskdecoding [166], performs the best with a MWER of 29.3%, while [94] improves on this toreach 27.50% MWER using an E2E transformer that had been pre-trained on MGB-2 andthen fine-tuned on MGB-3 adaptation set for 30 epochs, followed by 5 additional epochs offine-tuning on both the adaptation and development set. Initial results from the MGB-5challenge demonstrated a 37.6% MWER on this corpus using a CNN-TDNN achitecture.Later work by [94] with an E2E transformer architecture improved this to 33.8% (againusing a similar pre-train and finetune procedure as their strategy for MGB-3). Like theirwork for MGB-3 this E2E transformer setting ended up not using a language model, intheir work they found that the LM they used appeared to overcorrect their DA resultsback to MSA spelling. Ultimately, performance on DA remains significantly worse thanMSA due to the small training sizes of available corpora for the few dialects that evenhave transcribed datasets. Better tools to collect and process data, could potentially bethe most important factor for improving DA performance.2.4.3 Morphological ComplexityMorphological complexity, as mentioned in Section 2.3.2, primarily impacts vocabularycoverage of Arabic, as the same root might be associated with many, semantically con-nected, words based on the pattern. For instance for root structure KTB, you could havekataba ‘he wrote’, kitaab ‘book’, and maktaba ‘library’ and many others all connected tosome semantic meaning of writing [107]. In a practical context you would need much more282.4. Arabic Speech Processingdata to learn all possible patterns for a given root, with unseen pattern potentially beingOOV.One simple approach is to use more data, either to train the model directly [32] or for thelanguage model used in decoding [128]. Self-supervised pre-training with self-training, inthe vein of wav2vec 2.0 self training [197], could potentially be leverage to cover this largeramount of data, as it would remove the requirement for fully transcribed datasets. We are,however, unaware of any pretrained speech representation model trained on MSA or anydialectal Arabic variety9. Another possible solution is using subword language models [18,111], which can capture a wider variety of words through composition of subword unitswith the benefit of a smaller total vocabulary size. A final approach, which can be viewedas a more specific application of subword vocabulary, is to decompose words based on theirmorphological composition [4, 64, 66, 177]. For instance the MADAMIRA [147] can betrained on text to generate probabilities of all of the possible diacretization of a given rootword, thereby alleviating the system of needing data covering all possible combinations.2.4.4 ASR modelsASR models used in Arabic ASR research has followed the same trajectory as ASR in gen-eral. Much of the initial work on Arabic ASR used HMM-GMM models, such as the BBNByblos 1997 large vocabulary conversational speech recognition system[200]. This earlywork established a baseline for performance on the CALLHOME Egyptian corpus (61.6%WER) as well as experiment with the use of MFCCs and frame energy (plus its derivative)as features, as well as speaker adaptation models (see description of the CALLHOME cor-pus in Section 2.5.1). This system was then modified further to work real time with theTides OnTap project [32], although with significant assumptions: the use of non-diacretizedtranscripts and assuming that localization did not affect pronunciation across the countriesthey were applying it to. The Tides OnTap system aimed to use a larger corpus, throughthe developers own proprietary dataset collected from Egyptian and Syrian TV and radio,as a way to counter out of vocabulary words.Following the release of the GALE challenge (see description in Section 2.5.2), partic-ipating teams developed a series of ASR models tackling the corpus. IBM in particularreleased semi-yearly models improving on their HMM ASR model system starting by em-bracing an automatic diacretization system, cross-adaption between diacretized and undi-acretized speaker adaptation models, unsupervised training and a larger vocabulary [171],this was followed by experimenting with acoustic feature choice, data selection, GMM vari-ations, and variable rate decoding [160], for their 2009 system they added neural networkacoustic modeling, improvements to their language modelling to incorporate syntactic andmorphological features, and similarly context dependent modeling of diacretization in their9The Huggingface model hosting platform indicates many fine-tuned Arabic models which use a wav2vec2.0 base, however, at the time of this writing no pre-trained models, nor any models for speech classifica-tion. [70]292.4. Arabic Speech Processingacoustic model. In their last release based on GALE, they apply similar incremental im-provements, such as applying morphological decomposition, but most interestingly createa dialectal-dependant decision tree for their acoustic model to allow it to dynamicallyadapt depending on likely dialect of the utterance [128]. Other groups participating inGALE followed a similar incremental approach, with improvement found from modelingdiacretization [143, 181], fusing diacretized and non-diacretized systems [143], collectingmore data and diverse data for training language models [143], and finally discriminativetraining [135, 181]. Following GALE the only other large and publicly available MSA cor-pus is MGB-2, which consists of 1,200 hours of MSA taken from broadcast news and talkshows [9]. Initial results on the corpus rely on GMM-HMM based systems, with acousticmodeling done by a variety of DNN architectures [8, 106, 178], with the QCRI system hav-ing the best overall score in the initial release 14.7% [9], which was subsequently improvedto 13.2% by improving the acoustic model [166]. Switching to E2E architectures howeverreached a new state of the art on this set, with a transformer trained with both CTC andattention reaching 12.5%[94]. This performance gain from switching to E2E is echoed byresults on MGB-3 and MGB-5.2.4.5 ADI modelsEarly ADI research focused on using separately collected CTS corpora for different lan-guages [5, 29, 30], which proved to be problematic due to different channel characteristicsof these corpora [35]. This Borˇil study is interesting both as a reminder to use corporaspecifically collected for the purpose of LID/DID, and also because of the technique theyuse to investigate the channel specific attenuation of LID/DID models by creating a subsetof the corpora just containing silence. While ASR models can be more easily evaluatedby inspection of transcripts, interpretability of LID/DID models is more difficult becausenon-linguistic channel and speaker information might influence the model decision, andthese may need to be evaluated directly for instance using probing classifier [50], or usinga ‘challenge’ dataset, like how the Borˇil study uses non-voiced portions of audio [35].For LID, NIST’s Language Recognition Evaluation (LRE) [142], has been an importantbenchmark in measuring the progress of LID and DID models, and in 2011 incorporatedArabic dialects as part of the challenge [82] with coverage of Iraqi, Leventine, Maghrebi,and MSA. The 2011 LRE challenge consists of binary language pair evaluations, strate-gies for submitted systems usually relied on fused subsystem classifiers generally based oni-vector [1, 119, 131, 165], support vector machine-Gaussian supervector (SVM-GSV) orjoint factor analysis based supervector approaches [1, 119, 131, 165], phone recognitionand language modeling (PPRLM) [1, 119, 131, 165], or Gaussian mixture model-maximummutual information (GMM-MMI) [165]. Aside from the addition of Egyptian to the Ara-bic dialects, the 2015 challenge also moved towards groups of similar language clusters,rather than pairs [129]. Submissions to the 2015 challenge, while still consisting mainly ofensembles of i-vector or other traditional approaches, highlighted the growing importance302.4. Arabic Speech Processingof neural architectures [113], these neural network applications include deep bottleneckfeatures (DBN) [113, 140, 149], DNNs for classification [149, 199], i-vector post process-ing [113], as well as summarization [149]. The most recent challenge in 2017 was muchthe same as 2015, except for removing MSA from the list of Arabic dialects [159]. Mostrelevant from this final year was the introduction of neural embedding x-vectors for theclassification task [167], marking the dominance of neural architectures over traditional,GMM-based, approaches.The ADI-5 and ADI-17 broadcast challenges are another set of important ADI bench-marks, consisting of a coarse grain + MSA classification task (ADI-5) [11] and a fine-graincountry-level classification task (ADI-17) [10] (we describe it further in Section 2.5.4 andgive a detailed breakdown in 3.1). Models used on the ADI-5 set mainly consisted ofi-vector based approaches with the winning entry using a siamese-network of CNNs to fur-ther post-process i-vectors [163] and the second best entry using a generative adversarialnetwork in addition to a traditional Gaussian classifier for their backend classification [39].This trend towards more neural architectures continues with ADI-17, where neural E2E ar-chitectures outperformed conventional i-vector approaches [10] with the best performanceby an E2E ResNet architecture, which used novel pooling layers [41].Reflecting on the advancement of ADI models, shows how the field has mirrored changesin machine learning at large, with a great variety of traditional methods eventually replacedby neural network architectures driven by large quantities of data.2.4.6 Section SummaryWe summarize the current challenges in Arabic speech processing as follows:1. Mixing of MSA and DA in common speech, implies better tools to tackle DA (forfiltering or multidialectal ASR) could greatly benefit MSA as well.2. The complex morphology of Arabic still requires large datasets to fully capture.Additionally, we note the following overall changes in the field of Arabic speech processing:1. Diacretization has been rendered unimportant for ASR due to use of E2E deep learn-ing models.2. For both ASR and ADI, deep learning models have now been fully embraced.2.4.7 DiscussionThere are a number of potential directions that seem to emerge from this examination.With the need for large datasets the most pressing factor impacting model quality, a transferlearning approach using speech representation models (see 2.2.3) could potentially alleviatesome of the data need by starting with a strong baseline for phonological recognition. But312.5. An Overview of Arabic Speech Corporait is still a matter of question if corpora cover all variations of verb form for a given dialect.While having fallen out of favor, use of phone-based language models seem like a potentiallyuseful tool to avoid approaches that might inadvertently include channel information inDID.1. Can speech representation models be used to alleviate training needs of ASR andADI models?2. How well can phone-based models perform compared to E2E models for ADI, andhow robust are they to channel effects?2.5 An Overview of Arabic Speech CorporaWhile other surveys of Arabic Speech Corpora exist, such as [6] or [7], we aim to specificallyexamine how these corpora have or can been used in the context of DA, both for ASR aswell as DID, as the latter can be leveraged to build tools to probe dialect assumptions inthe collection of the former. Similarly, due to the diglossic nature of DA, we also examinewhether there are known instances of codeswitching present in the corpora, which maypose great challenges for both ASR and DID.Most useful to developing ASR and LID/DID systems with deep-learning approachesare continuous speech corpora (as opposed to isolated word corpora), which consist of audio,often from existing sources such as radio or TV broadcast, that have been labeled withutterance-level transcriptions, in this section we will look at several different collectionsof these corpora, notably from telephone recordings as well as collected from broadcastmedia. We note the important aspects of these corpora where known, including: dominantdialect spoken, style of speech (for instance formal or conversational), and topic or genreof speech.2.5.1 Conversational Telephone CorporaOne major genre of large-vocabulary continuous speech corpora used in Arabic speech pro-cessing is conversational telephone speech (CTS) corpora. One downside of these corporais that it is often possible to detect the dialect being spoken simply from the pattern ofsilences and channel information [35], thus limiting there use in ADI without appropriatepre-processing.First of these is the CALLHOME Egyptian Arabic Speech corpus [44], whichconsists of a training set 80 telephone conversations totally 14 hours of Egyptian Arabic,as well as a series of development (3.5 hrs) and evaluation sets (totaling 7 hours). Thiscorpus also contains a significant amount of L2 English, found from a study [195] thatexamined the 1084 utterances that contained L2 English and noted 2231 English wordscompared with 9455 L1 Arabic words. This percentage (19%) of L2 language is fairly322.5. An Overview of Arabic Speech Corporahigh for these code-mixed utterances, but only represents a minor portion (1%) of theoverall 178446 word corpus (train + development) that they examined. Like CALLHOME,CALLFRIEND [43] is a telephone conversation corpus covering a total of 25 hours ofEgyptian Arabic. This set of 60 calls, shares with CALLHOME, the potential inclusion ofL2 English, which may make it of interest to researchers looking at codeswitching.Moving beyond Egyptian corpora, the Levantine Arabic QT Training Data Set5 [126] conversational telephone corpus covers the Levantine Arabic dialect-family, whichcovers the vernacular Arabic spoken in Jordan, Lebanon, Palestine, and Syria. This par-ticular corpus is a compilation of several earlier corpora, the Fisher Levantine TrainingData Set 3, Levantine Arabic QT training data set 4, and associated transcripts. In to-tal it consists of 250 hours of telephone conversation on a wide range of randomly pickedtopics. Due to the limited CTS corpora coverage of different dialects, a series of smallercorpora released from a company based in Australia, Appen, could be beneficial in lightof few options. These Appen corpora cover Levantine, Iraqi, and Gulf Arabic. Sim-ilar, to the QT dataset, the 5 hour Levantine corpus [125] covers a total of 982 speakersspeaking on average 5-6 minutes per side of the conversation. Appen also released corporafor Iraqi Arabic [124], which consists of 474 speakers holding telephone conversations to-taling roughly 50 hours of audio, as well as Gulf Arabic [123], consisting of 46 hours ofconversational Gulf Arabic telephone conversations between 975 participants.Last of the CTS corpora is the Multi-Language Conversational Telephone Speech2011 – Arabic Group corpus, which formed the basis of Language Resource Evaluation(LRE) challenge 2011 Arabic dialects training set. Covering Iraqi (37h), Levantine (41h),and Maghrebi (39h) Arabic this corpus consists of organic conversations up to 15minutesin length. While earlier LREs used CALLHOME and CALLFRIENDs for their Arabicdata, this corpus allowed the 2011 LRE to cover additional Arabic dialects.While we include these corpora for completeness, there are significant downsides to allof the CTS corpora. All CTS corpora used in earlier works may be at risk of confoundingchannel information, which may limit their use in DID. This is because each handsetand telephone system might have particular frequency response patterns as well as otherartifacts, for instance buzzing induced by the mains frequency of the electric grid, whichby itself can be used for geolocation [73]. Patterns of silence and channel information alonewas enough to distinguish some samples in a study of Arabic CTS corpora [35], and whilethis study identifies normalization of the audio as a potential remedy for these artifacts,understanding and compensating for the frequency response of unknown handsets as wellas other channel properties in these telephone systems may not be feasible, thus they havelittle use for DID, except to act as a warning about the dangers of not properly accountingfor confounding recording factors. Furthermore, to estimate the frequency response of themicrophones in a corpus the silent portions of the corpus can be used [35], but use of voiceactivity detection (VAD) algorithms to split audio into short utterances with only spokensections renders this clever method difficult to apply to modern corpora, which are oftencropped to contain little to no silence. This implies a major improvement in investigating332.5. An Overview of Arabic Speech Corporachannel effects could be enabled for future corpora by clipping audio to always ensureseveral seconds of silence for each recording.Another unfortunate downside of these corpora is that their recording at a 8khz samplerate (the rate of telephone audio), renders them incompatible to models such as wav2vec2.0 [22], which were trained on 16khz audio corpora. Taken in consideration with the issuesrelated to CTS channel characteristics, while these corpora may be historically important,aside from examination of code-mixing, it appears they have little value to current research.2.5.2 Broadcast CorporaCompared to CTS, Broadcast corpora often contain more formal (MSA) speech. In termsof topics of conversation, these often focus more on current events and politics [9], however,some corpora have been intentionally constructed to more easily balance between a varietyof programs [10, 11].The Global Autonomous Language Exploitation (GALE) [56] project aimed toacquire large quantity corpora in Arabic and Chinese. Although the original website forthe GALE project has been taken offline, a portion of the corpus has been made availablethrough the Linguistics Data Consortium. The following is a breakdown of the sources ofthe audio in Phase 2, 3, and 4 of the Broadcast News and Broadcast Conversation portionsof this corpus (Phase 1 is omitted as it consists of only text). Table 2.5 gives more detailson the sources used in the different segments of GALE Arabic, while Table 2.4 gives abreakdown of the hours per release. We note that the source breakdown of the corpusshows an extremely broad coverage of countries in the collection process, The BroadcastConversation section [182, 183, 186, 187, 190] of this corpus consists of conversationalspeech (such as interviews or discussions), while the Broadcast News section [184, 185,188, 189, 191] of the corpus consists of standard news reporting. All data was collectedbetween 2006 and 2009.342.5. An Overview of Arabic Speech CorporaNewsConv.Phase 2 Part 1 165 123Phase 2 Part 2 170 128Phase 3 Part 1 132 123Phase 3 Part 2 128 129Phase 4 37 75Total Hours 632 578Table 2.4: Breakdown of Arabic GALE speech by hours per release (with separate releasesfor Broadcast News and Broadcast Conversation) currently available on the LinguisticsData Consortium. For an overview of the sources used in each of the 10 different releasessee Table 2.5352.5.AnOverviewofArabicSpeechCorporaAbuDhabiTV,(UAE)AlAlamNews(Iran)AlArabiya(UAE)AlBaghdadia(Egypt*)AlFayhaa(Iraq)AlHiwar(UK)Alhurra(USA)Aljazeera(Qatar)AlOrdiniyah(Jordan)AlIraqiyah(Iraqi)Al-ManarTV(Lebanon)AlSharqiya(Iraq)BahrainTV(Bahrain)DubaiTV(UAE)KuwaitTV(Kuwait)Leb.Broad.Co.(Lebanon)NileTV(Egypt)OmanTV(Oman)QatarTV(Qatar)RadioSawa(USA)SaudiTV(SaudiArabia)SyriaTV(Syria)TunisianNat’lTV(Tunisia)YemenTV(Yemen)Phase 2 Con. P1 • • • • • • • • •Phase 2 Con. P2 • • • • • • • •Phase 3 Con. P1 • • • • • • • • • •Phase 3 Con. P1 • • • • • • • • • • • • • • • • •Phase 4 Con. • • • • • • • • • •Phase 2 News P1 • • • • • • • • • • •Phase 2 News P2 • • • • • • • • • • •Phase 3 News P1 • • • • • • • • • • • •Phase 3 News P2 • • • • • • • • • • • • • •Phase 4 News • • • • • • • • • • • •Table 2.5: Overview of sources for GALE Arabic Speech. *Al Baghdadia, though based in Egypt has an Iraqi focus.362.5. An Overview of Arabic Speech CorporaIn total this amounts to 632 hours of broadcast news speech, and 578 hours of con-versational speech (see Table 2.4 for breakdown). An important aspect of this corpora isthe inclusion of station information that each phase recorded from (see Table 2.5. Thisinformation, can give a sense of the geographic spread of the corpus, as well as draw at-tention to which cities are represented in the DA sections of the corpus. While there isno estimate of amount of total DA within the corpus, work done in analyzing the GALEPhase 2 Conversational, and News corpora 10 for codeswitched segments found significantportions of DA during the analysis [156]. Including 17 hours of pure DA and finding atotal of 38 hours of codemixed speech. While this effort only investigated one phase of thecorpus, it is clear that this a valuable set for investigating both DA and codemixing, butwould need careful segmentation and models to differentiate between the codemixed andpure DA speech, alongside a DID model to filter out MSA.To address different genres of speech, a series of Multi-Genre Broadcast (MGB)corpora were created, of which several focused on Arabic dialects. The first among theseis the MSA MGB-2 [9] corpus. This large Arabic corpus consists of 1, 200 hours of audiotaken from AlJazeera Arabic TV channel programs. As fitting the name, the programsselected varied between different types of programs available on AlJazeera’s channel. Al-though dominated by Political shows (76%), other genres included Society (9%), Economy(8%), Media (3%), Law (2%), and Science (2%). Similarly, the format of the programdiffers, with the majority of the audio coming from conversational speech with multipleparticipants (63%), one-on-one interviews (19%), and news reporting (18%), The corpusconsists mainly of MSA (which the developers estimate at over 70%), however, DA is alsopresent, leading to potential challenges with using language models trained on MSA in ASRon this dataset [94]. Like GALE, a thoughtful segmentation and filtering of utterances us-ing a DID model could be beneficial to both improve the quality of the MSA portion ofthe corpus, and as source of additional DA.Following the success of MGB2, MGB-3 [11] is a DA corpus consists of multigenreEgyptian Arabic covering a wide range of genres of video, collected from Youtube. Itconsists of 16 hours balanced between the 7 different genres collected (comedy, cooking,family/kids, fashion, drama, sports, and science). Finally, the most recent of the ArabicMGB corpora is MGB-5. This Moroccan Arabic corpus [10] similar to MGB-3 consists of13 hours of transcribed speech from a balanced set of different genres of YouTube video.The categories of videos is the same as MGB-3, with roughly even distribution between thedifferent genres selected. Moroccan is a particularly noteworthy dialect; as mentioned inSection 2.3.3, the dialect has retained many old Arabic words, leading to major challengesin adapting MSA tools to work for it.Aside from the aforementioned broadcast corpora, there are several smaller MSA cor-pora. Namely the NEMLAR Broadcast News Speech Corpus [127], which similarto GALE, is a 40 hour MSA Broadcast News corpus consisting of radio recordings of news10only part 1 of the news corpora was included372.5. An Overview of Arabic Speech Corporaprograms on Medi1, Radio Orient, Radio Monte Carlo, and Radio Television Maroc. Intotal a 259 speakers appear on the corpus. And likewise NetDC Arabic BNSC [49],which is 22.5 hour Broadcast News Speech Corpus consists MSA from Radio Orient (notenon-overlapping time period with NEMLAR), In total the corpus covers 90 distinct speak-ers. Compared to NEMLAR and GALE, the lack of variety in source potentially make theNetDC corpus only useful as supplemental validation. Given the low-ratio of DA to MSAin GALE, and the few geographic sources present in these two corpora, it is likely not aworth the effort to filter them for DA.Overall, broadcast corpora illustrate a possible useful source of DA corpora. Whilespecifically collected DA corpora now exist for a few dialects (Egyptian [11] and Moroc-can [10]), filtering, as in [156], of GALE could lead to potentially 30-40 hours of DA ifperformed on the whole dataset, and MGB-2 could potentially have up to 200 hours basedon their initial estimate of ∼70% MSA [9].2.5.3 Other SourcesWhile broadcast speech and conversational telephone recordings are the largest collectionof resources for DA and MSA, there are a number of other corpora either focusing on readspeech, or particular speech related acts (codeswitching, emotion etc). ArzEn [86], isone of the few Arabic speech corpora designed with the explicit goal of studying Egyptiancodeswitching, consists of 11.4 hours of conversational speech. With roughly 70% of thesentences containing codeswitching.Another read speech corpus, Common Voice [16], contains 87 validated hours of readMSA sentences. This dataset is interesting in that it is massively crowdsourced, allowingfor individuals to add their own readings of sentences, and review and validate other’ssubmissions. The genre of the texts vary, with sentences from Wikipedia forming thebulk of initial prompts, but users are able to submit their own sentences to be included.Unfortunately, while it represents a good model for collecting a corpus, there is currently noeffort being made to extend it to DA. An additional factor, which could potentially impedethis models use for DA is that since prompts are read, lack of standard orthography wouldlead to conflict on how to pronounce certain utterances and could make the collectionprocess contentious as annotators and readers disagree over a written sentence.The speaker ID corpus, King Saud University Arabic Speech Database [14],includes 590 hour and covers a wide range of speaking conditions (quiet as well as noisyrooms, read as well as spontaneous speech) among a group of 269 speakers. The speakersinclude both Saudis, non-Saudi Arabs, as well as a variety of non-Arab L2 Arabic speakers(although no women are present in this last set). Although the purpose of the corpusis primarily in speaker identification, however, the large size and variety of speech mightallow for other uses. Another corpus from KSU is KSUEmotions [133]. This small corpusconsists of 23 speakers reading MSA newswire in a variety of different emotional affects:neutral, anger, sadness, happiness, surprise, and interrogative.382.5. An Overview of Arabic Speech CorporaA number of corpora are available from the European Language Resource Association(ELRA)11 Designed for consumer devices, the Egyptian Arabic Speecon database [68],is part of the Speecon project [96], aiming to develop corpora to use for training speechdriven toys and devices, thus it is most interested in command words and recording condi-tions than necessarily focusing on general speech to be used for ASR. This gender balanceddatabase of MSA spoken by 550 adult and 50 children consists of read and spontaneousspeech uttered in several different environments. The spontaneous speech section consistsof 5 minutes of speech on a given topic. In total 196 hours, although only a portion (ap-proximately 46 hours of session time) is continuous speech. As with other ELRA corpora,cost (€50000 for academic ELRA member) may be a barrier to use for research. Anothersmall MSA corpus available through ELRA is the A-SpeechDB[69]. While this small 20hours Arabic corpus is not gender balanced (70% men, 30% women), the non-repetitionof read prompts could potentially be useful for an independent test set in ASR research.GlobalPhone Arabic[67] This 20 hour read speech corpus, using the Assabah newspaperas source material, consists of 78 speakers (35 male, 43 female) reading approximately 100sentences from the newspaper, in a quiet office environment. Because of the nature of thelarger GlobalPhone project [67], which covers 22 different spoken languages, this corpusrepresents a useful benchmark for Language ID.Overall, these set of corpora look to be less promising than the broadcast corpora forexisting sources of DA, with the exception of the ArzEn corpus.2.5.4 Non-transcribed CorporaDue to the difficulty in building transcribed corpora, especially for DA, where the lack offixed orthography presents practical challenges, raw, non-transcribed, corpora can be usedin the context of self-training, self-supervised learning, or fully unsupervised ASR to bridgethe gap of lacking abundant transcribed corpora. We examine several of these raw corporain this section, including a number of corpora that can be additionally used for ArabicDialect Identification.Two corpora associated with the Arabic MGB challenges include ADI-5 and ADI-17. ADI-5, collected as part of the MGB-3 Challenge [11], is a dialectal identificationcorpus consists of roughly 74 hours of audio segments labeled with broad categories forthe Arabic dialects being spoken (between MSA, Gulf, Levantine, North African, andEgyptian). ADI-17, similar to the ADI-5 is a dialectal Arabic corpus [164], which wasalso created as part of the MGB challenges (MGB-5) and consists of roughly 3, 000 hours ofaudio segments from Youtube programs covering a variety of different genres. The segmentsare split into 17 different dialectal categories, allowing for much finer grain dialectal analysisthan the ADI-5 corpus (see Figure 2.4 for comparison of the coverage of the two sets).Finally there is also, KALAM’DZ[36], which consists of 104 hours of sub-dialectsof Algerian, covering the 8 major Arabic subdialects (High Plains, Hilali-Saharan, Hilali-11http://www.elra.info/en/392.5. An Overview of Arabic Speech CorporaFigure 2.4: Overview of the coverage of ADI-5 and ADI-17. ADI-5 covers 4 coarse regionalvarieties of Arabic plus MSA, these are Gulf (dots), Levantine (horizontal), Egyptian (ver-tical), and North African (diagonal). ADI-17 in contrast is fine-grain country-level coveringthe following regions: Morocco, Algeria, Mauritania, Libya, Egypt, Sudan, Lebanon, Pales-tine, Jordan, Syria, Iraq, Kuwait, Saudi Arabia, Yemen, UAE, and Oman, Qatar402.5. An Overview of Arabic Speech CorporaTellian, Ma’qilian, Pre-Hilalien, Sahel-Tell, Sulaymite, and Algiers-Blanks). Audio is col-lected from Algerian TV and Radio, YouTube as well as podcasts. This authors collectionmethod allowed for 39% clean audio, with the rest containing some noise, either frommusic or outdoors recording conditions. In terms of genre breakdown the majority 60%consists of News and Politics, 19.5% covers People and Blogs, with further topics includingHow-to/Style, Education, Entertainment, and Humor.These corpora could potentially be quite beneficial for DA ASR, both as a pre-trainingstep (see Section 2.2.3), as well as potentially sources of self-training material, where modelscreate pseudo-transcriptions and then use these transcriptions for iterative training andrefinement.Name Date Varieties Source Genre(s) HrsCALLFRIEND 1996-1997 arz Tel Conv. 25CALLHOME 1997 arz Tel Conv. 14/3.5/3.3*Appen Iraqi 2003-2004 ayp, acm Tel Conv. 50Appen Levantine 2004 apc, ajp Tel Conv. 41Appen Gulf 2004 afb Tel Conv. 46Levantine QT 2003-2005 apc, ajp Tel Conv. 250GALE Conv. 2006-2009 arb TV Conv. 251/252/75†GALE News 2006-2009 arb TV News 335/260/37†NRE 2011 2011 acm, ara, qal Tel Conv. 117KSU Speech DB 2014 ara In-Person Read & Spon. 590MGB-2 2015 arb TV Conv & News 1200/10/10MGB-3 2017 arz YouTube Multi-Genre 4.6/4.8/6KSUEmotions 2017 arb, ara In-Person Read newswire 5MGB-5 2019 ary YouTube Multi-Genre 10.2/1.3/1.4ArzEn [86] 2020 arz In Person Interview 5.6/2.9/2.9Table 2.6: Overview of transcribed Arabic continuous speech corpora. For varieties welist the ISO 639-3 code of primary Arabic varieties in the corpus. Dataset hour splits (Train / Dev / Test ) are given for corpora that have standard splits. † For GALE we usethe released phases for splits. *CALLHome evaluation sets eval96 and eval97, althoughmore have been released since.2.5.5 Section SummaryWhile earlier corpora focused on 8khz conversational telephone corpora [43, 44, 123–126],attempts to use them in ADI lead to models picking up on the channel characteristics ofthe telephone lines in the countries at question [35], indicating the need for either uniformcollection process or strong channel normalization. While newer specialized corpora havestarted being collected based on non-ASR/LID/DID tasks, such as emotion detection [133]412.5. An Overview of Arabic Speech Corporaand code-mixed speech [86], but with the growth of deep learning approaches, larger corporaalso became extremely important, driving a move from hand collected corpora to usingexisting broadcast news and programs as sources. These larger corpora demand betterpipelines for segmenting and processing audio chunks, however, this potential can lead toa virtuous cycle, in making MGB-2 the researchers used their existing ASR system [12],but the large size of MGB-2 can support training larger models as well as transfer learningto other dialects (e.g. Egyptian [11] and Moroccan [10]).Ultimately, while there are numerous Arabic corpora we have explored in this section,we conclude that only the broadcast corpora and large untranscribed corpora likely havemuch value to studying DA.2.5.6 DiscussionWith our examination of a number of different datasets, we have found, regrettably fewinstances of opportunity for using them to study DA. And what corpora that exist, mightneed significant efforts to make use of (filtering GALE for just a few hours). This lays aclear takeaway, that there is a need for newly collected DA corpora covering the breadthof speaking styles in a geographical region (see the KALAM’DZ corpus for examples ofthe fine-grained nature potentially needed). This supports our original supposition of theneed for ADI tools to help filter utterances collected in an autonomous fashion. A finalaspect to consider in development of these corpora, is the safeguards that must be put inplace to enable corpora to be used for ADI model training, as shown with CTS corpora,channel characteristics can be a confounding factor. While deep learning models have greatpotential for understanding the patterns and intricacies of language, this also may equallyapply to patterns in frequency and nearly undetectable humming that might be embeddedin audio files. Tools to better analyze the channel characteristics, and ultimately be usedto ensure channel invariant corpora, would go a great ways to instill trust in ADI modelsand the corpora they are trained on.We conclude with the following observations as well as questions:1. ADI-17 and ADI-5 appear to be the only two corpora useful for studying Arabic DID.2. There is a need for specially collected DA corpora, as well as ADI tools to filter thesecorpora.3. Existing labeled corpora are likely sources of subsets of DA in the order of 10s to100s of hours, with significant effort needed.4. DA corpora need to ensure a wide in-country regional distribution, to ensure modelsare able to handle different subdialects.5. How can channel characteristics be better accounted for in corpora creation?42Chapter 3Experimental DesignTo contextualize our experiments further, we start by re-examining our observations gleanedfrom our initial literature survey. In our examination of LID methods (Section 2.1.5), wenote our concerns about channel independence assumptions as well as lack of effort toexamine efficiency of LID models. In our discussion of ASR methods (Section 2.2.7), weidentify the potential of self-supervised pre-trained models for LID, but echoing our ear-lier efficiency concerns note the lack of standards for fine-tuning large pre-trained modelsfor the task of LID, which could require many days of compute time for a thorough gridsearch of fine-tuning hyperparameters. Next we note the possibility of using multilingualpre-trained models, which might aid in processing Arabic speech in the absence of modelspre-trained purely on Arabic. Examining Arabic linguistics briefly (Section 2.3.5) we findthat the task of distinguishing similar dialects should be more difficult than standard LID,noting that there are numerous linguistic distinctions between dialects, as well as significantdifferences within dialects (from city-vs-city as well as city-vs-rural differences). These lin-guistic distinctions also raise questions as to what level of linguistic knowledge ADI modelsoperate on. Looking at existing Arabic speech processing research (Section 2.4.7), leadsus to conclude that using pre-trained models could potentially be useful to overcome datascarcity issues, as well as identify phone-based models as potential measures to combatoverfitting to channel information. Finally, in our examination of Arabic corpora (Section2.5.6), we reiterate that few corpora exist for studying DA, with ADI-17 and ADI-5 beingthe few exceptions for specifically studying ADI.We summarize this context as the following questions:1. How can we build channel invariant models for ADI?2. How well do speech representation models work for ADI?3. What is the best way to efficiently fine-tune speech representation models for ADI?4. Does multilingual training aid in performance on ADI?5. Is phone-level information sufficient for ADI?With these guiding questions, we aim to exploit the recent advances in speech represen-tation research for application to the task of ADI, specifically the wav2vec 2.0 and HuBERTmodels as discussed in section 2.2.3. As mentioned earlier, these pre-trained models al-low faster training on our downstream task, enabling us to experiment with the best way433.1. Dataof using the models specifically in DID. To ensure efficient exploration of the fine-tuninghyperparameter space we use a random search process with fixed number of iterations,capping total compute time at one node-month time for each of the fine-tuned models.We evaluate two broad approaches, a straightforward transfer-learning approach where wefine-tune speech representation models directly on the ADI task. We do so with both mul-tilingual pre-trained model as well as monolingual model to explore whether multilingualpre-training is beneficial. Our second approach is a novel pseudo-transcript classificationapproach which aims to capture phone-level information about an utterance. While thetransfer-learning approach has the potential to bring the full power of the pre-trainedmodel to the ADI task, it may be prone to overfitting to channel characteristics or otherfeatures of our dataset, thus our second approach, could potentially be more applicableacross datasets.3.1 DataWhile we cannot test channel information directly used by models (a subject we discussfurther in our future work, see Section 5.3), we use two different datasets as a proxyto capture channel overfitting. We will use the ADI-17 and ADI-5 corpora as our maindatasets for ADI. As mentioned in section 2.5.4, the ADI-17 and ADI-5 corpora are un-transcribed corpora consisting of multi-genre broadcast recordings from YouTube and AlJazeera respectively. Because they cover similar genres of discussion (albeit with differ-ent distribution), we argue that there should be little domain mismatch between the twocorpora, but that there should be significant channel information mismatch due to therecording conditions (various YouTube channels with potentially many different recordingsetups compared with a single organization recording studio). We take the ADI-17 corpusas our main fine-tuning corpus, and use the non-MSA parts of the ADI-5 test set as oursecondary validation to understand generalizability of models to different recording condi-tions. Because the ADI-17 train set is unbalanced (see Table 3.1), we opt to use macro F1as our primary metric as it weights performance on each dialect equally.3.2 ComparisonsFor our baseline methods we compare performance on the ADI-17 test set with previouslypublished work [10]. The state of the art (SOTA) on the ADI-17 dataset is a ensemblemethod using a ResNet architecture from Duke Kunshan University (DKU) [41], which wewill discuss briefly. The SOTA system is a fusion of 4 ResNet systems trained on differentdata sizes and using different residual block sizes, but each use a global pooling layer anddata augmentation through speed perturbation. While the description of this system isnot precise in the ADI-17 initial results [10], we infer that the global pooling layer is alearnable dictionary encoding (LDE) layer [41], which as discussed in Section 2.1.2, is a443.2. ComparisonsDatasetFine-grainRegionCoarse-grainRegionUtterances(train / dev / test)Hours(train / dev / test)ADI-5(noMSA) - EGY (3093 / 298 / 302) (12.4 / 2 / 2)- GLF (2744 / 264 / 250) (10 / 2 / 2.1)- LEV (2851 / 330 / 334) (10.3 / 2 / 2)- NOR (2954 / 351 / 344) (10.5 / 2 / 2.1)ADI-17Algeria NOR (32262 / 246 / 745) (115.7 / 0.6 / 1.9)Egypt EGY (151052 / 680 / 760) (451.1 / 1.9 / 2.1)Iraq GLF (291123 / 646 / 760) (815.8 / 1.5 / 1.9)Jordan LEV (5514 / 422 / 721) (25.9 / 1.7 / 2.0)Kuwait GLF (32654 / 450 / 760) (108.2 / 1.2 / 2.0)Lebanon LEV (38205 / 409 / 760) (116.8 / 1.3 / 1.9)Libya NOR (35692 / 683 / 760) (127.4 / 2.3 / 2.0)Mauritania NOR (138706 / 219 / 509) (456.4 / 0.5 / 1.3)Morocco NOR (18530 / 397 / 760) (57.8 / 1.1 / 1.9)Oman GLF (27188 / 655 / 760) (58.5 / 1.7 / 1.8)Palestine LEV (39129 / 456 / 760) (121.4 / 1.4 / 2.1)Qatar GLF (18883 / 216 / 760) (47.7 / 0.7 / 2.0)Saudi Arabia GLF (69350 / 393 / 760) (186.1 / 1.2 / 2.1)Sudan EGY (26650 / 929 / 760) (62.3 / 2.0 / 1.7)Syria LEV (47606 / 470 / 760) (119.5 / 1.3 / 2.0)UAE GLF (49486 / 1144 / 760) (108.4 / 2.2 / 1.8)Yemen GLF (21139 / 540 / 760) (53.4 / 1.3 / 1.8)Table 3.1: Comparison of the ADI-5 Test [11] and ADI-17 [164] corpora, we follow theconvention of including Jordanian as a Levantine dialect, though [164] notes ambiguitywith Gulf.453.3. Methodsneural approximate of GMM clustering, where a variable length utterance is summarizedinto a D×C matrix, where D is the model hidden dimension size and C clusters are chosen.In addition to the DKU system we also include the second place results from the ADI-17competition, University of Kent’s CNN-LSTM system which mainly focused on applyingdata augmentation techniques to balance for smaller splits in the training set [10].3.3 Methods3.3.1 E2E Fine-tuned Pre-trained Speech Representation ModelsLarge pre-trained speech representation models, such as wav2vec 2.0 [22] and HuBERT [93],have the advantage of being trained on tens of thousands of hours of data, with the capa-bility of to perform near SOTA on ASR tasks with minimal fine-tuning. These models, dueto their strong encoding of language, naturally present an opportunity to act as startingpoints for building dialect ID systems. Performance of wav2vec 2.0 on language ID [71]demonstrates the promise of this class of models in this regard. We choose (i) the mas-sively multilingual model XLS-R 300m [17] as our example of the wav2vec 2.0 model,and pick (ii) HuBERT Large [93] as our example of the monolingual English HuBERTmodel. While both styles of model are powerful pre-trained speech representation models,their pre-training procedures are significantly different: wav2vec 2.0 uses a quantized con-trastive loss function [22], and HuBERT uses k-means cluster assignments as pseudo-labeltargets [93].We aim to deploy these pretrained models as dialect ID systems by adding a modifiedTAP layers [42]. We first add a projection layer to reduce the dimensionality of our output,then mean pool the projected outputs over the temporal dimension, last this is passedthrough a final classification layer. The models are then fine-tuned by training on theADI17 training dataset. Because these pretrained models are designed for use primarilyin ASR, the finetuning process must be adjusted for LID. We do so by exploring theconfiguration space through a random search [27], which enables for an efficient searchdue to the likelihood of many hyperparameters not mattering significantly. Using a fixedcompute budget for our two fine-tuned models of one node month of compute time (roughly30 iterations of the search) each, we search the following training hyperparameters: BatchSize, the number of samples per each gradient update; Freeze Steps, the number of updateswith only the final prediction layer thawed; Learning Rate, a variable multiplying the size ofthe gradient update; Max Steps, the total number of steps in our training; Sample Duration,the duration of the random sample taken from the source audio; Thaw Depth, which layersof the model to thaw during fine-tuning. Additionally we experiment with applying Layer-Norm and Attention (LNA) fine-tuning, which is a fine-tuning technique that freezes alllayers except for layer norm and multihead self-attention [115]. We summarize the rangesand distributions used for this finetuning exploration in Table 3.2. We adopt the originalwav2vec 2.0 tri-state learning rate schedule, which consists of breaking the total training463.3. Methodssteps into a 10% ramp up to max learning rate, 50% plateu, and 40% cooldown period.Hyperparameter Distribution Type RangeBatch Size Uniform 4× b 75SampleDurationcFreeze Steps Uniform [0, 1000]Learning Rate LogUniform [1× 10−5, 1× 10−2]LNA finetuning Binary [True, False]Max Steps Uniform [20k, 40k]Sample Duration Uniform [4, 18] secondsThaw Depth Uniform [0, 23]Table 3.2: The search space for our random search of finetuning hyperparameters. Weset batch size such that a total of 75 seconds of audio can comfortably fit on each V100GPU, of which we train with 4 at a time. LNA finetuning stands for Layer Norm andAttention finetuning. For LNA finetuning, the entire model is frozen except for LayerNorm, Multi-headed Self-attention, and the final classification layers.3.3.2 Fixed-Representation ClassificationWhile both HuBERT and wav2vec 2.0 create outputs that can act as strong representa-tion of the audio input, dialect ID tasks can be prone to models overfitting to channelinformation or other aspects of the audio that are non-linguistically relevant [35], meaningthat there is potential benefit to putting the data into a bottleneck that excludes thesepieces of information. One approach, which we propose, is to use psuedo-labels generatedfrom k-means clustering of outputs from the HuBERT model. These clusters act as ‘loose’phoneme representations, which when taken together give a pseudo-phonetic transcript ofthe recordings. By training on these pseudo-transcripts, we thus avoid potential issueswith channel information, or other artifacts, tainting the experiment, and can potentiallyallow for training dialect ID models on corpora with vastly different recording conditions.Because different layers of the HuBERT model output different qualities of represen-tations, we select representations from the Base model: we pick the layer with the bestPhone-Purity and PNMI, which occurs as output from layer 10 [93] 12. Our process forgenerating the pseudo-transcriptions is as follows. First, we exclude very short and verylong utterances by selecting ADI-17 train files that are between 5 and 30 seconds long andthen pass these raw files through the HuBERT model, extracting the feature representa-tions from layer 10. Because what level of phonetic information captured is related to thenumber of clusters, we treat this as a hyperparameter in our experiment: using a sub-sample of 10% of the resulting frames, we train 5 different k-means models using a choice12While the main HuBERT paper uses layer 9 for training of the HuBERT Large and XL, their graphsindicate potentially better attributes from using layer 10 with respect to PNMI and Phone-Purity [93]473.3. Methodsfrom the set {200, 400, 600, 800, 1000}. Then, we assign cluster labels to the extractedfeatures for our data. Because HuBERT covers 320 samples of audio per each predictionframe, for 16khz sample rate audio this leaves us with sequences of between 250 and 1500labels. We then compress these sequences into a length normalized representation, essen-tially corresponding to the unigram count of each label divided by the total length of thesequence.For classification, we choose the simplest classification technique: passing the lengthnormalized representations through a single linear projection, a multi-class perceptron [157],and training with cross-entropy loss. While it is outside of the scope of this experiment,this method also gives us the option of directly inspecting model weights and identifyingwhich cluster assignments are used by the model to identify each dialect. Since each labelcorresponds to 50ms of audio, we could then directly inspect with audio analysis software,such as Praat [33]. While this strategy lacks the model capacity of fully finetuning wav2vec2.0 or HuBERT, it requires minimal hardware capabilities and can be trained extremelyquickly, with 1.24GBs of trainable parameters in the HuBERT and wav2vec 2.0 models vs.only 68KBs (or 0.005%) for our linear classifier.Hyperparameter RangeBatch Size [64, 128, 256, 512]k-means Clusters [200, 400, 600, 800, 1000]Learning Rate [1× 10−2, 1× 10−3, 1× 10−4]Table 3.3: Overview of hyperparameter search space for our pseudotranscript classificationmodels.Finally, one can think of this as an approximation to earlier i-vector approaches [41],using HuBERT for feature extraction, k-means for dictionary learning, unigram counts forencoding, and with a perceptron for classification. Similar to i-vector approaches, this alsohas the property of mainly being an unsupervised approach, with training impacting onlythe final classification layer. Unlike the finetuning of HuBERT and wav2vec 2.0, we cando a simple grid search on the small set of hyperparameters that govern the perceptronperformance as show in Table 3.3.483.3. MethodsFigure 3.1: Overview of our speech representation models for ADI. Left: Fixed represen-tation models, which generate features using a pre-trained HuBERT model [93], and thenclusters them using k-means before putting them into a bag of words format. Right: E2Emodel, the output of a HuBERT or wav2vec 2.0 model are projected by a linear layer, andthen mean-pooled, before finally projected into the classification space. Both models usesoftmax to assign probabilities to the outputs. Red boxes indicate portions trained duringfine-tuning.49Chapter 4ResultsWe present our results on the ADI-17 and ADI-5 dialect identification corpora. While theADI-17 corpus shows primary performance of our models, the ADI-5 test set shows uspotential overfitting to ADI-17 (which we surmise measure channel mismatch between thedatasets).ADI-17 Dev ADI-17 TestACC PRE REC F1 ACC PRE REC F1FixedMajority Class 12.77 0.75 5.88 1.33 6.02 0.35 5.88 0.67HuBERT (Ours) 52.03 57.18 55.16 52.07 52.15 57.46 52.63 51.36i-Vector [164] 59.7 - - - 60.3 60.7 60.5 60.6x-Vector [164] 71.0 - - - 72.1 72.1 72.7 72.4E2EE2E Baseline [164] 83.0 - - - 82.0 82.1 83.3 82.69DKU [10] 97.4 - - - 94.9 94.9 94.9 94.9U-Kent [10] 92.3 - - - 91.1 91.1 91.1 91.1HuBERT (Ours) 92.42 92.19 92.72 92.23 92.16 92.35 92.19 92.12xls-r-300m (Ours) 90.88 91.32 90.81 90.77 90.19 90.69 90.21 90.20Table 4.1: Our results on ADI-17.ADI-5 Test (minus MSA)ACC PRE REC F1Majority Class 28 6.99 25 10.92HuBERT Fixed 68.94 71 70.3 67.71HuBERT E2E 80.6 81.58 80.7 80.36XLS-R-300m E2E 76.67 77 77.53 76.2Table 4.2: Our results on ADI-5. For our models, we take our trained ADI-17 models andapply them to the ADI-5 test corpus with MSA samples removed. We do not compareprior submissions to the ADI-5 challenge as these predict on MSA as well.504.1. Fixed Representation Classifier4.1 Fixed Representation ClassifierAs Table 4.1 shows, our pseudotranscript classifier, while modest in comparison to thefine-tuned end-to-end models as well as the i-vector and x-vector models from the originalADI-17 baseline, still significantly outperform our majority class classifier (F1: 51% vs.0.6%). We note the poor performance on Jordanian dialect, which may be reflective of therelative lack of Jordanian samples in the training split of the corpus. The mistakes thisclassifier makes appear to be fairly intuitive, confusing between similar dialects, such as theGulf dialects (KSA, QAT, IRA, KUW, OMA, YEM, UAE) for example (see Figure 4.1).Here we note that more measures of distance between dialects specifically on a phonologicallevel (see Section 2.3.5) would enable a richer discussion of this. But as the pseudotranscriptclassifier acts primarily on a phonological distribution level, it may be that this level is notenough to differentiate dialects that are geographically close, and may share many similarphonological properties. Another aspect of this performance reflect size distribution in thetraining set, where dialects with a smaller number of hours (e.g. QAT) are confused withgeographically close dialects that have more representation (e.g. IRA).To investigate how well this classifier would do at an easier coarse-grained dialect iden-tification (North African, Egyptian, Levantine, and Gulf Arabic) we pool the predictionson ADI-17 and find that under this easier setting the classifier reaches 70% F1. To validatethis performance, we generate predictions on the ADI-5 test set (minus MSA), pooling thecoarse regions as earlier, and find roughly the same performance of 67.7% F1 (see Table4.2 and Figure 4.2) indicating that the model is unlikely to be overfitting to any particularaspect of the ADI-17 corpus.4.2 Fine-tuned ModelsWe evaluate performance of both the XLS-r-300m (wav2vec 2.0) and HuBERT Large mod-els on our ADI-17 test set. While both models perform significantly better than any of thefixed-representation approaches, as well as the E2E baseline from ADI-17, we note thatneither surpasses the Duke Kunshan University ResNet model. To confirm performanceof this model, similarly to the fixed representation model we also use the trained modelsto create predictions on the ADI-5 test set (see Table 4.2). While the fixed representationmodel was able to substantially improve performance on the ADI-5 task (as expected asit has significantly fewer classes), we note the significant drop in both the HuBERT E2E(ADI-17 F1: 92.12, ADI-5 F1: 80.36) and XLS-r-300m E2E (ADI-17 F1: 90.2, ADI-5 F1:76.2) models when compared with their performance on the theoretically harder ADI-17set. This could be indicative of overfitting to the ADI-17 corpora, and raises questionsabout how reliable the corpus may be for training generalizeable models. Results can alsoimply the extent to which E2E models learn narrow representations that accommodatedata on which they are fine-tuned but not as much data with different distributions (e.g.,collected from different periods of times, using different recording methods).514.2. Fine-tuned ModelsFigure 4.1: Confusion Matrix of the HuBERT Fixed representation classifier on the ADI17test set. Normalized over true regions to show percentages.524.2. Fine-tuned ModelsFigure 4.2: The coarse-region aggregated predictions and labels from our trained HuBERTFixed representation classifier on the ADI-5 test set without MSA. This has been normal-ized over each row to show percentages of true regions categorized as the column label.534.2. Fine-tuned ModelsFigure 4.3: Confusion Matrix of the HuBERT ADI classifier on the ADI17 test set, nor-malized over the true classes.544.2. Fine-tuned ModelsFigure 4.4: The coarse-region aggregated prediction percentages of our fine-tuned HuBERTmodel on the ADI-5 test set with MSA removed.554.3. Summary4.3 SummaryWe reiterate our findings:1. Multilingual pre-training does not offer particular benefit for ADI.2. Fixed-representation models underperform E2E models.3. Both HuBERT and wav2vec 2.0 pre-training are strong starting points for buildingADI systems.4. Small iteration random search of fine-tuning hyperparameters is able to optimizepre-trained models close to the state of the art.5. E2E models performance degrades when testing on coarse-grain dataset (potentiallydue to domain or channel mismatch).6. Fixed-representation models appear to perform better on more generalized data,potentially indicating domain or channel invariance or under fitting to initial data.56Chapter 5ConclusionWhile LID/DID tools remain a central part in building large-scale corpora of continuousspeech, building and improving these models will lead to a virtuous cycle whereby largercorpora can further lead to better tools. We demonstrate one aspect of this virtuous cycle,by bringing breakthroughs in ASR research to application in DID.We reiterate the contributions of this work:• We provide a thorough review of existing LID/DID and ASR literature as it pertainsto Arabic ASR and ADI.• We carry out an evaluation efficient fine-tuning of speech representation models in aDID context.• We offer a demonstration of the effectiveness of pre-trained speech representationmodels to the task of ADI.• We also offer an evaluation of pre-trained speech representation models in both anE2E fine-tuned and fixed-representation setting.• We validate the robustness of E2E fine-tuned and fixed-representation models on amore generalized corpus.We devote the rest of this section to discussion of the limitation of our research, potentialethical considerations of DID, and future work.5.1 LimitationsA major limitation with any work in DA is simply the lack of datasets, both transcribedand untranscribed, suitable for either ASR or ADI. While a wider variety of datasets couldenable exploration of different domain and channel conditions, we make due with whatavailable data we have. While we aim to explore the generalizeability of the training dataat our disposal (ADI-17), lack of direct channel information exploration tools limits apotential important avenue for checking this direction.Out of concerns for the environmental impact of training deep learning models, we usea fixed number of iterations for our hyperparameter search, noting that performance ofmodels could potentially be increased by opting for both a larger number of iterations, as575.2. Ethical Considerationswell as a longer running time for each iteration (our choice of 20k-40k step enabled keepingeach iteration under a day of run time, but would be considered quite low compared to fine-tuning these models for ASR). Nevertheless, we seem to have achieved reasonable resultswith a small compute budget and short run time.5.2 Ethical ConsiderationsWhile our primary goal involves improvement of LID models for the benefit of DA corporacreation, LID has many other uses. We single out one potentially problematic use here:LID as a surveillance tool. Use of LID in a surveillance context raises concerns about thepower of both governments and corporations to develop profiles of individuals, this infringesupon common notions of privacy. This misuse could also have a negative impact on thosesurveiled, for instance by enabling the spread of misinformation through ads targeted atminority language communities [174]. However, ultimately we hope the benefit of thesetools in helping enable communities to develop speech technologies outweighs the potentialmisuse.5.3 Future WorkFor future work, we note that the gap in performance between E2E fine-tuned systems ontheir trained corpora vs. similar alternative corpora hints at potential issues related tothe quality of the data (too easy to overfit) or significant weakness in the models abilityto be robust to potentially different domains. Contrasting this to the apparent robustnessof a fixed representation model potentially indicates a line of direction in improving fixedrepresentation models, or using these models in ensemble to counteract the fragility of thepurely E2E model. To further probe the cause of this discrepancy, analysis of the spectralcharacteristics of the ADI-17 corpus might identify channel information that could biasmodels when trained directly on the ADI-17 set with the goal of identifying best practicesfor large-scale collection of dialectal corpora. Addressing this same idea in a tangentialdirection, replication of microphone corpora such as handset TIMIT and Lincoln Labo-ratory handset database [155] for 16khz audio and modern recording equipment (laptopmic, cellphone, and video cameras), could enable creation of channel characteristic detec-tion models that could allow for detection and quantification of these characteristics. Suchchannel ‘embeddings’ might be usable in conjunction with generative adversarial techniquesfor creating corpora processed to minimize influence of channel characteristics.58Bibliography[1] Alberto Abad. 2011. The L2F language recognition system for NIST LRE 2011. InThe NIST Language Recognition Evaluation Workshop.[2] Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. Youtweet what you speak: A city-level dataset of arabic dialects. In Proceedings of theEleventh International Conference on Language Resources and Evaluation (LREC2018).[3] Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, and Lyle Ungar.2020. Toward micro-dialect identification in diaglossic and code-switched environ-ments. In Proceedings of the 2020 Conference on Empirical Methods in NaturalLanguage Processing (EMNLP), pages 5855–5876.[4] Mohamed Afify, Ruhi Sarikaya, Hong-Kwang Jeff Kuo, Laurent Besacier, and YuqingGao. 2006. On the use of morphological analysis for dialectal Arabic speech recogni-tion. In Ninth International Conference on Spoken Language Processing.[5] Murat Akbacak, Dimitra Vergyri, Andreas Stolcke, Nicolas Scheffer, and ArindamMandal. 2011. Effective Arabic dialect classification using diverse phonotactic mod-els. In Twelfth Annual Conference of the International Speech CommunicationAssociation.[6] Fawaz Al-Anzi and Dia AbuZeina. 2018. Literature survey of Arabic speech recog-nition. In 2018 International Conference on Computing Sciences and Engineering(ICCSE), pages 1–6. IEEE.[7] Latifa Al-Sulaiti and Eric Steven Atwell. 2006. The design of a corpus of contempo-rary Arabic. International Journal of Corpus Linguistics, 11(2):135–171.[8] Tuka AlHanai, Wei-Ning Hsu, and James Glass. 2016. Development of the MIT ASRsystem for the 2016 Arabic multi-genre broadcast challenge. In 2016 IEEE SpokenLanguage Technology Workshop (SLT), pages 299–304. IEEE.[9] Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Re-nals, and Yifan Zhang. 2016. The MGB-2 challenge: Arabic multi-dialect broadcastmedia recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT),pages 279–284. IEEE.59Bibliography[10] Ahmed Ali, Suwon Shon, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, JamesGlass, Steve Renals, and Khalid Choukri. 2019. The MGB-5 challenge: Recognitionand dialect identification of dialectal Arabic speech. In 2019 IEEE Automatic SpeechRecognition and Understanding Workshop (ASRU), pages 1026–1033. IEEE.[11] Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challengein the wild: Arabic MGB-3. In 2017 IEEE Automatic Speech Recognition andUnderstanding Workshop (ASRU), pages 316–322. IEEE.[12] Ahmed Ali, Yifan Zhang, Patrick Cardinal, Najim Dahak, Stephan Vogel, and JamesGlass. 2014. A complete KALDI recipe for building Arabic speech recognition sys-tems. In 2014 IEEE spoken language technology workshop (SLT), pages 525–529.IEEE.[13] Abdulkareem Alsudais, Wafa Alotaibi, and Faye Alomary. 2022. Similarities betweenArabic dialects: Investigating geographical proximity. Information Processing &Management, 59(1):102770.[14] Mansour Alsulaiman, Ghulam Muhammad, Bencherif Mohamed Abdelkader, AwaisMahmood, and Zulfiqar Ali. 2014. King Saud University Arabic speech database.https://doi.org/10.35111/vpqe-bz17. LDC2014S02.[15] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, EricBattenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, GuoliangChen, et al. 2016. Deep speech 2: End-to-end speech recognition in english andmandarin. In International conference on machine learning, pages 173–182.[16] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, MichaelHenretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020.Common Voice: A massively-multilingual speech corpus. In Proceedings of the 12thLanguage Resources and Evaluation Conference, pages 4218–4222.[17] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Na-man Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al.2021. XLS-r: Self-supervised cross-lingual speech representation learning at scale.arXiv preprint arXiv:2111.09296.[18] Nguyen Bach, Mohamed Noamany, Ian Lane, and Tanja Schultz. 2007. Handlingoov words in Arabic ASR via flexible morphological constraints. In Eighth AnnualConference of the International Speech Communication Association. Citeseer.[19] Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. 2019. Effectiveness ofself-supervised pre-training for speech recognition. arXiv preprint arXiv:1911.03912.60Bibliography[20] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and MichaelAuli. 2022. Data2vec: A general framework for self-supervised learning in speech,vision and language. arXiv preprint arXiv:2202.03555.[21] Alexei Baevski, Steffen Schneider, and Michael Auli. 2019. vq-wav2vec:Self-supervised learning of discrete speech representations. arXiv preprintarXiv:1910.05453.[22] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.wav2vec 2.0: A framework for self-supervised learning of speech representations.Advances in Neural Information Processing Systems, 33:12449–12460.[23] Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machinetranslation by jointly learning to align and translate. In 3rd International Conferenceon Learning Representations, ICLR 2015.[24] Maher Bahloul. 2007. Linguistic diversity: The qaaf across Arabic dialects.AMSTERDAM STUDIES IN THE THEORY AND HISTORY OF LINGUISTICSCIENCE SERIES 4, 289:247.[25] Emily M. Bender. 2019. The #benderrule: On naming the languages we study andwhy it matters. The Gradient.[26] Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning:A review and new perspectives. IEEE transactions on pattern analysis and machineintelligence, 35(8):1798–1828.[27] James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter op-timization. Journal of machine learning research, 13(2).[28] Larwan Berke, Christopher Caulfield, and Matt Huenerfauth. 2017. Deaf and hard-of-hearing perspectives on imperfect automatic speech recognition for captioningone-on-one meetings. In Proceedings of the 19th International ACM SIGACCESSConference on Computers and Accessibility, pages 155–164.[29] Fadi Biadsy, Julia Hirschberg, and Daniel P. W. Ellis. 2011. Dialect and accentrecognition using phonetic-segmentation supervectors. In Proc. Interspeech 2011,pages 745–748.[30] Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken Arabic dialect iden-tification using phonotactic modeling. In EACL 2009 Workshop on ComputationalApproaches to Semitic Languages, page 53.[31] Fadi Biadsy, Pedro J Moreno, and Martin Jansche. 2012. Google’s cross-dialectArabic voice search. In 2012 IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP), pages 4441–4444. IEEE.61Bibliography[32] Jayadev Billa, Mohamed Noamany, Amit Srivastava, John Makhoul, and FrancisKubala. 2002. Arabic speech and text in Tides OnTap. In Proceedings of the secondinternational conference on Human Language Technology Research, pages 7–11.[33] Paul Boersma. 2006. Praat: doing phonetics by computer. http://www. praat. org/.[34] Marcely Zanon Boito, John Ortega, Hugo Riguidel, Antoine Laurent, Lo¨ıc Barrault,Fethi Bougares, Firas Chaabani, Ha Nguyen, Florentin Barbier, Souhir Gahbiche,et al. 2022. ON-TRAC consortium systems for the IWSLT 2022 dialect and low-resource speech translation tasks. In Proceedings of the 19th International Conferenceon Spoken Language Translation (IWSLT 2022), pages 308–318.[35] Hynek Borˇil, Abhijeet Sangwan, and John HL Hansen. 2012. Arabic dialectidentification-” is the secret in the silence?” and other observations. In ThirteenthAnnual Conference of the International Speech Communication Association.[36] Soumia Bougrine, Aicha Chorana, Abdallah Lakhdari, and Hadda Cherroun. 2017.Toward a web-based speech corpus for Algerian dialectal Arabic varieties. InProceedings of the Third Arabic Natural Language Processing Workshop, pages 138–146.[37] Herve A Bourlard and Nelson Morgan. 2012. Connectionist speech recognition: ahybrid approach, volume 247. Springer Science & Business Media.[38] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: Anopen-source mandarin speech corpus and a speech recognition baseline. In 2017 20thConference of the Oriental Chapter of the International Coordinating Committee onSpeech Databases and Speech I/O Systems and Assessment (O-COCOSDA), pages1–5. IEEE.[39] Ahmet E Bulut, Qian Zhang, Chunlei Zhang, Fahimeh Bahmaninezhad, and John HLHansen. 2017. UTD-CRSS submission for MGB-3 Arabic dialect identification:Front-end and back-end advancements on broadcast speech. In 2017 IEEE AutomaticSpeech Recognition and Understanding Workshop (ASRU), pages 360–367. IEEE.[40] Weicheng Cai, Danwei Cai, Shen Huang, and Ming Li. 2019. Utterance-level end-to-end language identification using attention-based CNN-BLSTM. In ICASSP2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 5991–5995. IEEE.[41] Weicheng Cai, Zexin Cai, Wenbo Liu, Xiaoqi Wang, and Ming Li. 2018. Insightsin-to-end learning scheme for language identification. In 2018 IEEE InternationalConference on Acoustics, Speech and Signal Processing (ICASSP), pages 5209–5213.IEEE.62Bibliography[42] Weicheng Cai, Jinkun Chen, and Ming Li. 2018. Exploring the encoding layer and lossfunction in end-to-end speaker and language recognition system. In Proc. Odyssey2018 The Speaker and Language Recognition Workshop, pages 74–81.[43] Alexandra Canavan and George Zipperlen. 1996. Callfriend Egyptian Arabicldc96s49. Philadelphia: Linguistic Data Consortium.[44] Alexandra Canavan, George Zipperlen, and David Graff. 1997. Callhome EgyptianArabic speech. Linguistic Data Consortium.[45] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attendand spell: A neural network for large vocabulary conversational speech recognition.In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 4960–4964. IEEE.[46] Nanxin Chen, Jesu´s Villalba, and Najim Dehak. 2019. Tied mixture of factor ana-lyzers layer to combine frame level representations in neural speaker embeddings. InINTERSPEECH, pages 2948–2952.[47] Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.End-to-end continuous speech recognition using attention-based recurrent nn: Firstresults. In NIPS 2014 Workshop on Deep Learning, December 2014.[48] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, andYoshua Bengio. 2015. Attention-based models for speech recognition. In Advancesin neural information processing systems, pages 577–585.[49] Khalid Choukri, Mahtab Nikkhou, and Niklas Paulsson. 2004. Network of datacentres (netdc): BNSC-an Arabic broadcast news speech corpus. In LREC.[50] Shammur A Chowdhury, Ahmed M Ali, Suwon Shon, and James R Glass. 2020.What does an end-to-end dialect identification model learn about non-dialectal in-formation? In INTERSPEECH, pages 462–466.[51] Shammur A Chowdhury, Younes Samih, Mohamed Eldesouki, and Ahmed Ali. 2020.Effects of dialectal code-switching on speech modules: A study using Egyptian Arabicbroadcast speech. In INTERSPEECH, pages 2382–2386.[52] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019. An unsupervisedautoregressive model for speech representation learning. In Proc. Interspeech 2019,pages 146–150.[53] Yu-An Chung, Hao Tang, and James Glass. 2020. Vector-Quantized AutoregressivePredictive Coding. In Proc. Interspeech 2020, pages 3760–3764.63Bibliography[54] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang,and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and maskedlanguage modeling for self-supervised speech pre-training. In 2021 IEEE AutomaticSpeech Recognition and Understanding Workshop (ASRU), pages 244–250. IEEE.[55] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019.ELECTRA: Pre-training text encoders as discriminators rather than generators. InInternational Conference on Learning Representations.[56] Jordan Cohen. 2007. The GALE project: A description and an update. In 2007IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages237–237. IEEE.[57] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and RuslanSalakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association forComputational Linguistics, pages 2978–2988.[58] Najim Dehak, Reda Dehak, James R Glass, Douglas A Reynolds, Patrick Kenny,et al. 2010. Cosine similarity scoring without score normalization techniques. InOdyssey, page 15.[59] Najim Dehak, Patrick J Kenny, Re´da Dehak, Pierre Dumouchel, and Pierre Ouellet.2010. Front-end factor analysis for speaker verification. IEEE Transactions on Audio,Speech, and Language Processing, 19(4):788–798.[60] Najim Dehak and Stephen Shum. 2011. Low-dimensional speech representation basedon factor analysis and its applications. Johns Hopkins CLSP Lecture.[61] Najim Dehak, Pedro A Torres-Carrasquillo, Douglas Reynolds, and Reda Dehak.2011. Language recognition via i-vectors and dimensionality reduction. In Twelfthannual conference of the international speech communication association. Citeseer.[62] Arthur P Dempster, Nan M Laird, and Donald B Rubin. 1977. Maximum likelihoodfrom incomplete data via the EM algorithm. Journal of the Royal Statistical Society:Series B (Methodological), 39(1):1–22.[63] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 Conference of the North American Chapter of the Associationfor Computational Linguistics: Human Language Technologies, Volume 1 (Long andShort Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computa-tional Linguistics.64Bibliography[64] Frank Diehl, Mark JF Gales, Marcus Tomalin, and Philip C Woodland. 2012. Mor-phological decomposition in Arabic ASR systems. Computer Speech & Language,26(4):229–243.[65] David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2022. Ethnologue:languages of the world, 25th edition. SIL International, Dallas, TX. http://www.ethnologue.com.[66] Amr El-Desoky, Christian Gollan, David Rybach, Ralf Schlu¨ter, and Hermann Ney.2009. Investigating the use of morphological decomposition and diacritization forimproving Arabic LVCSR. In Tenth annual conference of the international speechcommunication association.[67] European Language Resources Association (ELRA). 2006. GlobalPhone Arabic.https://catalogue.elra.info/en-us/repository/browse/ELRA-S0192/. ELRA-S0192.[68] European Language Resources Association (ELRA). 2010. Egyptian Ara-bic speecon database. https://catalogue.elra.info/en-us/repository/browse/ELRA-S0308/. ELRA-S0308.[69] European Language Resources Association (ELRA). 2011. A-SpeechDB.https://catalogue.elra.info/en-us/repository/browse/ELRA-S0315/. ELRA-S0315.[70] Hugging Face. 2022. Hugging Face - models. https://huggingface.co/models.[71] Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu. 2020. Exploring wav2vec 2.0 onspeaker verification and language identification. arXiv preprint arXiv:2012.06185.[72] Ali Farghaly. 2010. The Arabic language, Arabic linguistics and Arabic computationallinguistics. Arabic Computational Linguistics (edt. Ali Farghaly), San Francisco:Center for the Study of Language and Inf, pages 43–81.[73] Ravi Garg, Adi Hajj-Ahmad, and Min Wu. 2013. Geo-location estimation from elec-trical network frequency signals. In 2013 IEEE International Conference on Acoustics,Speech and Signal Processing, pages 2862–2866. IEEE.[74] John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David SPallett. 1993. DARPA TIMIT acoustic-phonetic continous speech corpus CD-ROM.NIST speech disc 1-1.1. NASA STI/Recon technical report n, 93:27403.[75] Wang Geng, Wenfu Wang, Yuanyuan Zhao, Xinyuan Cai, Bo Xu, Cai Xinyuan,et al. 2016. End-to-end language identification using attention-based recurrent neuralnetworks. In Interspeech, pages 2944–2948.65Bibliography[76] Ondrˇej Glembek, Luka´sˇ Burget, Pavel Mateˇjka, Martin Karafia´t, and PatrickKenny. 2011. Simplification and optimization of i-vector extraction. In 2011 IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP),pages 4516–4519. IEEE.[77] Javier Gonzalez-Dominguez, Ignacio Lopez-Moreno, Has¸im Sak, Joaquin Gonzalez-Rodriguez, and Pedro J. Moreno. 2014. Automatic language identification usinglong short-term memory recurrent neural networks. In Proc. Interspeech 2014, pages2155–2159.[78] Alex Graves. 2012. Sequence transduction with recurrent neural networks. arXivpreprint arXiv:1211.3711.[79] Alex Graves, Santiago Ferna´ndez, Faustino Gomez, and Ju¨rgen Schmidhuber. 2006.Connectionist temporal classification: labelling unsegmented sequence data with re-current neural networks. In Proceedings of the 23rd international conference onMachine learning, pages 369–376.[80] Alex Graves and Navdeep Jaitly. 2014. Towards end-to-end speech recognition withrecurrent neural networks. In International Conference on Machine Learning, pages1764–1772.[81] Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013. Hybrid speechrecognition with deep bidirectional LSTM. In 2013 IEEE workshop on automaticspeech recognition and understanding, pages 273–278. IEEE.[82] Craig S Greenberg, Alvin F Martin, and Mark A Przybocki. 2012. The 2011 NIST lan-guage recognition evaluation. In Thirteenth Annual Conference of the InternationalSpeech Communication Association.[83] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu,Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. 2020. Conformer:Convolution-augmented transformer for speech recognition. Proc. Interspeech 2020,pages 5036–5040.[84] Nizar Habash, Mona Diab, and Owen Rambow. 2012. Conventional orthography fordialectal Arabic. In Proceedings of the Eighth International Conference on LanguageResources and Evaluation (LREC’12), pages 711–718.[85] Nizar Habash, Ramy Eskander, and Abdelati Hawwari. 2012. A morphological an-alyzer for Egyptian Arabic. In Proceedings of the twelfth meeting of the specialinterest group on computational morphology and phonology, pages 1–9.66Bibliography[86] Injy Hamed, Ngoc Thang Vu, and Slim Abdennadher. 2020. ArzEn: A speech corpusfor code-switched Egyptian Arabic-english. In Proceedings of The 12th LanguageResources and Evaluation Conference, pages 4237–4246.[87] Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin,Anmol Gulati, Ruoming Pang, and Yonghui Wu. 2020. ContextNet: Improvingconvolutional neural networks for automatic speech recognition with global context.Proc. Interspeech 2020, pages 3610–3614.[88] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen,Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. 2014. Deepspeech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567.[89] Awni Y Hannun, Andrew L Maas, Daniel Jurafsky, and Andrew Y Ng. 2014. First-pass large vocabulary continuous speech recognition using bi-directional recurrentdnns. arXiv preprint arXiv:1408.2873.[90] Salima Harrat, Karima Meftouh, Mourad Abbas, Salma Jamoussi, Motaz Saad,and Kamel Smaili. 2015. Cross-dialectal Arabic processing. In InternationalConference on Intelligent Text Processing and Computational Linguistics, pages 620–632. Springer.[91] Sepp Hochreiter and Ju¨rgen Schmidhuber. 1997. Long short-term memory. Neuralcomputation, 9(8):1735–1780.[92] Takaaki Hori, Shinji Watanabe, and John R Hershey. 2017. Joint CTC/attention de-coding for end-to-end speech recognition. In Proceedings of the 55th Annual Meetingof the Association for Computational Linguistics (Volume 1: Long Papers), pages518–529.[93] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Rus-lan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-supervisedspeech representation learning by masked prediction of hidden units. IEEE/ACMTransactions on Audio, Speech, and Language Processing, 29:3451–3460.[94] Amir Hussein, Shinji Watanabe, and Ahmed Ali. 2021. Arabic speech recognition byend-to-end, modular systems and human. arXiv preprint arXiv:2101.08454.[95] Sergey Ioffe. 2006. Probabilistic linear discriminant analysis. In European Conferenceon Computer Vision, pages 531–542. Springer.[96] Dorota Iskra, Beate Grosskopf, Krzysztof Marasek, H Heuvel, Frank Diehl, and An-dreas Kiessling. 2002. Speecon-speech databases for consumer devices: Databasespecification and validation. In Third international conference on language resourcesand evaluation (LREC). Paris: ELRA, European Language Resources Association.67Bibliography[97] Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparametrization withgumble-softmax. In International Conference on Learning Representations (ICLR2017). OpenReview. net.[98] Hui Jiang. 2010. Discriminative training of HMMs for automatic speech recognition:A survey. Computer Speech & Language, 24(4):589–608.[99] Daniel Jurafsky and James H Martin. 2019. Speech and language processing. 3rdedn. draft. Online: https://web. stanford. edu/˜ jurafsky/slp3.[100] Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu,Pierre-Emmanuel Mazare´, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert,Christian Fuegen, et al. 2020. Libri-light: A benchmark for ASR with limited orno supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), pages 7669–7673. IEEE.[101] Alina Karakanta, Matteo Negri, and Marco Turchi. 2020. Is 42 the answer toeverything in subtitling-oriented speech translation? In Proceedings of the 17thInternational Conference on Spoken Language Translation, pages 209–219.[102] Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma,Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xi-aofei Wang, et al. 2019. A comparative study on transformer vs RNN in speech appli-cations. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop(ASRU), pages 449–456. IEEE.[103] Shigeki Karita, Nelson Enrique Yalta Soplin, Shinji Watanabe, Marc Delcroix, At-sunori Ogawa, and Tomohiro Nakatani. 2019. Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and LanguageModel Integration. In Proc. Interspeech 2019, pages 1408–1412.[104] Patrick Kenny. 2005. Joint factor analysis of speaker and session variability: Theoryand algorithms. CRIM, Montreal,(Report) CRIM-06/08-13, 14(28-29):2.[105] Patrick Kenny, Gilles Boulianne, Pierre Ouellet, and Pierre Dumouchel. 2007. Jointfactor analysis versus eigenchannels in speaker recognition. IEEE Transactions onAudio, Speech, and Language Processing, 15(4):1435–1447.[106] Sameer Khurana and Ahmed Ali. 2016. QCRI advanced transcription system (qats)for the Arabic multi-dialect broadcast media recognition: MGB-2 challenge. In 2016IEEE Spoken Language Technology Workshop (SLT), pages 292–298. IEEE.[107] Katrin Kirchhoff, Jeff Bilmes, John Henderson, Richard Schwartz, Mohamed Noa-many, Pat Schone, Gang Ji, Sourin Das, Melissa Egan, Feng He, et al. 2002. Novel68Bibliographyspeech recognition models for Arabic. In Johns-Hopkins University summer researchworkshop.[108] Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, ZionMengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020.Racial disparities in automated speech recognition. Proceedings of the NationalAcademy of Sciences, 117(14):7684–7689.[109] Can Korkut, Ali Haznedaroglu, and Levent Arslan. 2020. Comparison of deeplearning methods for spoken language identification. In International Conferenceon Speech and Computer, pages 223–231. Springer.[110] Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev,Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang. 2020. Quartznet: Deep au-tomatic speech recognition with 1d time-channel separable convolutions. In ICASSP2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 6124–6128. IEEE.[111] Hong-Kwang Jeff Kuo, Ebru Arisoy, Lidia Mangu, and George Saon. 2011. Minimumbayes risk discriminative language models for Arabic speech recognition. In 2011IEEE Workshop on Automatic Speech Recognition & Understanding, pages 208–213. IEEE.[112] Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis, and Simon Dobnika.2018. A lexical distance study of Arabic dialects. Procedia computer science, 142:2–13.[113] Kong Aik Lee, Haizhou Li, Li Deng, Ville Hautama¨ki, Wei Rao, Xiong Xiao, AnthonyLarcher, Hanwu Sun, Trung Nguyen, Guangsen Wang, et al. 2016. The 2015 NISTlanguage recognition evaluation: the shared view of i2r, fantastic4 and singams. InINTERSPEECH 2016, volume 2016, pages 3211–3215.[114] Jinyu Li, Rui Zhao, Hu Hu, and Yifan Gong. 2019. Improving RNN transducer mod-eling for end-to-end speech recognition. In 2019 IEEE Automatic Speech Recognitionand Understanding Workshop (ASRU), pages 114–121. IEEE.[115] Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, AlexeiBaevski, Alexis Conneau, and Michael Auli. 2021. Multilingual speech translationfrom efficient finetuning of pretrained models. In Proceedings of the 59th AnnualMeeting of the Association for Computational Linguistics and the 11th InternationalJoint Conference on Natural Language Processing (Volume 1: Long Papers), pages827–838.[116] Shaoshi Ling and Yuzong Liu. 2020. DeCoAR 2.0: Deep contextualized acousticrepresentations with vector quantization. arXiv preprint arXiv:2012.06659.69Bibliography[117] Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. 2020. Deep contex-tualized acoustic representations for semi-supervised speech recognition. In ICASSP2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 6429–6433. IEEE.[118] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. 2020. Mock-ingjay: Unsupervised speech representation learning with deep bidirectional trans-former encoders. In ICASSP 2020-2020 IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), pages 6419–6423. IEEE.[119] Gang Liu, Seyed Omid Sadjadi, Taufiq Hasan, Jun-Won Suh, Chi Zhang, MahnooshMehrabani, Hynek Boril, Abhijeet Sangwan, and John HL Hansen. 2011. UTD-CRSSsystems for NIST language recognition evaluation 2011. In NIST 2011 LanguageRecognition Evaluation Workshop, Atlanta, USA, pages 6–7.[120] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, OmerLevy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A ro-bustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.[121] Ignacio Lopez-Moreno, Javier Gonzalez-Dominguez, Oldrich Plchot, David Martinez,Joaquin Gonzalez-Rodriguez, and Pedro Moreno. 2014. Automatic language iden-tification using deep neural networks. In 2014 IEEE international conference onacoustics, speech and signal processing (ICASSP), pages 5337–5341. IEEE.[122] Alicia Lozano-Diez, Rube´n Zazo Candil, Javier Gonza´lez Domı´nguez, Doroteo TToledano, and Joaquin Gonzalez-Rodriguez. 2015. An end-to-end approach to lan-guage identification in short utterances using convolutional neural networks. InProceedings of the Annual Conference of the International Speech CommunicationAssociation, INTERSPEECH. International Speech and Communication Associa-tion.[123] Appen Pty Ltd. 2006. Gulf Arabic conversational telephone speech.https://doi.org/10.35111/nsvg-dd69. LDC2006S43.[124] Appen Pty Ltd. 2006. Levantine Arabic conversational telephone speech.https://doi.org/10.35111/2dcs-9751. LDC2006S45.[125] Appen Pty Ltd. 2007. Levantine Arabic conversational telephone speech.https://doi.org/10.35111/77zm-gw29. LDC2007S01.[126] Mohamed Maamouri, Tim Buckwalter, David Graff, and Hubert Jin. 2006. Lev-antine Arabic QT training data set 5, speech. https://doi.org/10.35111/3bpn-2713.LDC2006S29.[127] Bente Maegaard. 2004. NEMLAR-an Arabic language resources project. In LREC.70Bibliography[128] Lidia Mangu, Hong-Kwang Kuo, Stephen Chu, Brian Kingsbury, George Saon, HagenSoltau, and Fadi Biadsy. 2011. The IBM 2011 GALE Arabic speech transcriptionsystem. In 2011 IEEE Workshop on Automatic Speech Recognition & Understanding,pages 272–277. IEEE.[129] Alvin F Martin, Craig S Greenberg, John M Howard, De´sire´ Banse´, George R Dod-dington, Jaime Herna´ndez-Cordero, and Lisa P Mason. 2015. NIST language recogni-tion evaluation—plans for 2015. In Sixteenth Annual Conference of the InternationalSpeech Communication Association.[130] David Martinez, Oldrˇich Plchot, Luka´sˇ Burget, Ondrˇej Glembek, and Pavel Mateˇjka.2011. Language recognition in ivectors space. In Twelfth annual conference of theinternational speech communication association.[131] David Martınez, Jesu´s Villalba, Alfonso Ortega, and Eduardo Lleida. 2011. I3Alanguage recognition system description for NIST LRE 2011. NIST LanguageRecognition Evaluation, Atlanta, GE, USA.[132] Rami Matarneh, Svitlana Maksymova, V Lyashenko, and N Belova. 2017. Speechrecognition systems: A comparative review. Journal of Computer Engineering.[133] Ali Hamid Meftah, Mustafa A Qamhan, Yasser Seddiq, Yousef A Alotaibi, andSid Ahmed Selouani. 2021. King Saud University emotions corpus: Construction,analysis, evaluation, and comparison. IEEE Access, 9:54201–54219.[134] Mahnoosh Mehrabani, Hynek Borˇil, and John HL Hansen. 2010. Dialect distanceassessment method based on comparison of pitch pattern statistical models. In 2010IEEE International Conference on Acoustics, Speech and Signal Processing, pages5158–5161. IEEE.[135] Florian Metze, Roger Hsiao, Qin Jin, Udhyakumar Nallasamy, and Tanja Schultz.2010. The 2010 CMU GALE speech-to-text system. In Eleventh Annual Conferenceof the International Speech Communication Association. Citeseer.[136] Xiaoxiao Miao, Ian Vince McLoughlin, and Yonghong Yan. 2019. A new time-frequency attention mechanism for TDNN and CNN-LSTM-TDNN, with applicationto language identification. In INTERSPEECH, pages 4080–4084.[137] Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer. 2019. Transform-ers with convolutional context for ASR. arXiv preprint arXiv:1904.11660.[138] Gregoire Montavon. 2009. Deep learning for spoken language identification. In NIPSWorkshop on deep learning for speech recognition and related applications, pages1–4. Citeseer.71Bibliography[139] Joa˜o Monteiro, Jahangir Alam, and Tiago H Falk. 2019. Residual convolutionalneural network with attentive feature pooling for end-to-end language identificationfrom short-duration speech. Computer Speech & Language, 58:364–376.[140] Raymond WM Ng, Mauro Nicolao, Oscar Saz, Madina Hasan, Bhusan Chettri, Mor-taza Doulaty, Tan Lee, and Thomas Hain. 2016. The Sheffield language recognitionsystem in NIST LRE 2015. In Odyssey, pages 181–187.[141] Tu Anh Nguyen, Maureen de Seyssel, Patricia Roze´, Morgane Rivie`re, EvgenyKharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. 2020. The zeroresource speech benchmark 2021: Metrics and baselines for unsupervised spoken lan-guage modeling. In NeuRIPS Workshop on Self-Supervised Learning for Speech andAudio Processing.[142] NIST. 2022. NIST -language recognition. https://www.nist.gov/itl/iad/mig/language-recognition.[143] Mohamed Noamany, Thomas Schaaf, and Tanja Schultz. 2007. Advances inthe CMU/InterACT Arabic GALE transcription system. In Human LanguageTechnologies 2007: The Conference of the North American Chapter of the Associationfor Computational Linguistics; Companion Volume, Short Papers, pages 129–132.[144] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learningwith contrastive predictive coding. arXiv preprint arXiv:1807.03748.[145] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Lib-rispeech: an ASR corpus based on public domain audio books. In 2015 IEEEinternational conference on acoustics, speech and signal processing (ICASSP), pages5206–5210. IEEE.[146] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin DCubuk, and Quoc V Le. 2019. SpecAugment: A simple data augmentation methodfor automatic speech recognition. Proc. Interspeech 2019, pages 2613–2617.[147] Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Es-kander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014.Madamira: A fast, comprehensive tool for morphological analysis and disambigua-tion of Arabic. In Proceedings of the ninth international conference on languageresources and evaluation (LREC’14), pages 1094–1101.[148] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark,Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations.arXiv preprint arXiv:1802.05365.72Bibliography[149] Oldrich Plchot, Pavel Matejka, Ondrej Glembek, Radek Fer, Ondrej Novotny`, JanPesan, Lukas Burget, Niko Brummer, and Sandro Cumani. 2016. BAT system de-scription for NIST LRE 2015. In Odyssey, pages 166–173.[150] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek,Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al.2011. The Kaldi speech recognition toolkit. In IEEE 2011 workshop on automaticspeech recognition and understanding. IEEE Signal Processing Society.[151] Chatrine Qwaider, Stergios Chatzikyriakidis, and Simon Dobnik. 2019. Can modernstandard Arabic approaches be used for Arabic dialects? sentiment analysis as acase study. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages40–50.[152] Lawrence Rabiner and Biinghwang Juang. 1986. An introduction to hidden Markovmodels. ieee assp magazine, 3(1):4–16.[153] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever,et al. 2019. Language models are unsupervised multitask learners. OpenAI blog,1(8):9.[154] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, MichaelMatena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transferlearning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67.[155] Douglas A Reynolds. 1997. HTIMIT and LLHDB: speech corpora for the study ofhandset transducer effects. In 1997 IEEE International Conference on Acoustics,Speech, and Signal Processing, volume 2, pages 1535–1538. IEEE.[156] Gabrielle Cristina Rivera. 2019. Automatic detection of code-switching in Arabicdialects. Ph.D. thesis, Massachusetts Institute of Technology.[157] Frank Rosenblatt. 1958. The perceptron: a probabilistic model for information stor-age and organization in the brain. Psychological review, 65(6):386.[158] Karin C Ryding. 2014. Arabic: A linguistic introduction. Cambridge UniversityPress.[159] Seyed Omid Sadjadi, Timothee Kheyrkhah, Audrey Tong, Craig S Greenberg, Dou-glas A Reynolds, Elliot Singer, Lisa P Mason, Jaime Hernandez-Cordero, et al. 2018.The 2017 NIST language recognition evaluation. In Odyssey, pages 82–89.[160] George Saon, Hagen Soltau, Upendra Chaudhari, Stephen Chu, Brian Kingsbury,Hong-Kwang Kuo, Lidia Mangu, and Daniel Povey. 2010. The IBM 2008 GALE Ara-bic speech transcription system. In 2010 IEEE International Conference on Acoustics,Speech and Signal Processing, pages 4378–4381. IEEE.73Bibliography[161] Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky,and Emmanuel Dupoux. 2013. Evaluating speech features with the minimal-pair ABXtask: Analysis of the classical MFC/PLP pipeline. In INTERSPEECH 2013: 14thAnnual Conference of the International Speech Communication Association, pages1–5.[162] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec:Unsupervised pre-training for speech recognition. Proc. Interspeech 2019, pages 3465–3469.[163] Suwon Shon, Ahmed Ali, and James Glass. 2017. MIT-QCRI Arabic dialect identi-fication system for the 2017 multi-genre broadcast challenge. In 2017 ieee automaticspeech recognition and understanding workshop (asru), pages 374–380. IEEE.[164] Suwon Shon, Ahmed Ali, Younes Samih, Hamdy Mubarak, and James Glass.2020. ADI17: A fine-grained Arabic dialect identification dataset. In ICASSP2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 8244–8248. IEEE.[165] Elliot Singer, Pedro Torres-Carrasquillo, Douglas A Reynolds, Alan McCree, FredRichardson, Najim Dehak, and Doug Sturim. 2012. The MITLL NIST LRE 2011 lan-guage recognition system. In Odyssey 2012-The Speaker and Language RecognitionWorkshop. Citeseer.[166] Peter Smit, Siva Reddy Gangireddy, Seppo Enarvi, Sami Virpioja, and Mikko Ku-rimo. 2017. Aalto system for the 2017 Arabic multi-genre broadcast challenge. In2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU),pages 338–345. IEEE.[167] David Snyder, Daniel Garcia-Romero, Alan McCree, Gregory Sell, Daniel Povey, andSanjeev Khudanpur. 2018. Spoken language recognition using x-vectors. In Odyssey,pages 105–111.[168] David Snyder, Daniel Garcia-Romero, Daniel Povey, and Sanjeev Khudanpur. 2017.Deep neural network embeddings for text-independent speaker verification. InINTERSPEECH, pages 999–1003.[169] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khu-danpur. 2018. X-vectors: Robust DNN embeddings for speaker recognition. In 2018IEEE international conference on acoustics, speech and signal processing (ICASSP),pages 5329–5333. IEEE.[170] Hagen Soltau, Lidia Mangu, and Fadi Biadsy. 2011. From modern standard Ara-bic to Levantine ASR: Leveraging GALE for dialects. In 2011 IEEE Workshop onAutomatic Speech Recognition & Understanding, pages 266–271. IEEE.74Bibliography[171] Hagen Soltau, George Saon, Brian Kingsbury, Jeff Kuo, Lidia Mangu, Daniel Povey,and Geoffrey Zweig. 2007. The IBM 2006 GALE Arabic ASR system. In 2007 IEEEInternational Conference on Acoustics, Speech and Signal Processing-ICASSP’07,volume 4, pages IV–349. IEEE.[172] Yan Song, Bing Jiang, YeBo Bao, Si Wei, and Li-Rong Dai. 2013. I-vector represen-tation based on bottleneck features for language identification. Electronics Letters,49(24):1569–1570.[173] Ridouane Tachicart, Karim Bouzoubaa, and Hamid Jaafar. 2016. Lexical differ-ences and similarities between Moroccan dialect and Arabic. In 2016 4th IEEEInternational Colloquium on Information Science and Technology (CiSt), pages 331–337. IEEE.[174] Dhanaraj Thakur and DeVan Hankerson Madrigal. 2022. Facts and their discontents:A research agenda for online disinformation, race, and gender. Center for Democracyand Technology.[175] Brian Thompson and Ali Alshehri. 2022. Improving Arabic diacritization by learningto diacritize and translate. In Proceedings of the 19th International Conference onSpoken Language Translation (IWSLT 2022), pages 11–21, Dublin, Ireland (in-personand online). Association for Computational Linguistics.[176] Andros Tjandra, Diptanu Gon Choudhury, Frank Zhang, Kritika Singh, Alexis Con-neau, Alexei Baevski, Assaf Sela, Yatharth Saraf, and Michael Auli. 2022. Improvedlanguage identification through cross-lingual self-supervised learning. In ICASSP2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 6877–6881. IEEE.[177] Marcus Tomalin, Frank Diehl, Mark JF Gales, Junho Park, and Philip C Woodland.2010. Recent improvements to the Cambridge Arabic speech-to-text systems. In 2010IEEE International Conference on Acoustics, Speech and Signal Processing, pages4382–4385. IEEE.[178] Natalia Tomashenko, Ke´vin Vythelingum, Anthony Rousseau, and Yannick Esteve.2016. LIUM ASR systems for the 2016 multi-genre broadcast Arabic challenge. In2016 IEEE Spoken Language Technology Workshop (SLT), pages 285–291. IEEE.[179] Edmondo Trentin and Marco Gori. 2001. A survey of hybrid ANN/HMM models forautomatic speech recognition. Neurocomputing, 37(1-4):91–126.[180] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan NGomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advancesin neural information processing systems, 30.75Bibliography[181] Dimitra Vergyri, Arindam Mandal, Wen Wang, Andreas Stolcke, Jing Zheng, MartinGraciarena, David Rybach, Christian Gollan, Ralf Schlu¨ter, Katrin Kirchhoff, et al.2008. Development of the SRI/Nightingale Arabic ASR system. In Ninth AnnualConference of the International Speech Communication Association.[182] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, and StephanieStrassel. 2013. GALE phase 2 Arabic broadcast conversation speech part 1.https://doi.org/10.35111/j224-tx54. LDC2013S02.[183] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, and StephanieStrassel. 2013. GALE phase 2 Arabic broadcast conversation speech part 2.https://doi.org/10.35111/xges-y891. LDC2013S07.[184] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, andStephanie Strassel. 2014. GALE phase 2 Arabic broadcast news speech part 1.https://doi.org/10.35111/2ryw-sp19. LDC2014S07.[185] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, andStephanie Strassel. 2015. GALE phase 2 Arabic broadcast news speech part 2.https://doi.org/10.35111/2k7p-4w91. LDC2015S01.[186] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, and StephanieStrassel. 2015. GALE phase 3 Arabic broadcast conversation speech part 1.https://doi.org/10.35111/6q8v-v073. LDC2015S11.[187] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, and StephanieStrassel. 2016. GALE phase 3 Arabic broadcast conversation speech part 2.https://doi.org/10.35111/fvxj-ey22. LDC2016S01.[188] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, andStephanie Strassel. 2016. GALE phase 3 Arabic broadcast news speech part 1.https://doi.org/10.35111/atsp-4c63. LDC2016S07.[189] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, andStephanie Strassel. 2017. GALE phase 3 Arabic broadcast news speech part 2.https://doi.org/10.35111/zhec-js91. LDC2017S02.[190] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, andStephanie Strassel. 2017. GALE phase 4 Arabic broadcast conversation speech.https://doi.org/10.35111/mr99-m774. LDC2017S15.[191] Kevin Walker, Christopher Caruso, Kazuaki Maeda, Denise DiPersio, andStephanie Strassel. 2018. GALE phase 4 Arabic broadcast news speech.https://doi.org/10.35111/xkxv-fg96. LDC2018S05.76[192] Shinji Watanabe, Takaaki Hori, and John R Hershey. 2017. Language independentend-to-end architecture for joint language identification and speech recognition. In2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU),pages 265–271. IEEE.[193] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi.2017. Hybrid CTC/attention architecture for end-to-end speech recognition. IEEEJournal of Selected Topics in Signal Processing, 11(8):1240–1253.[194] Janet CE Watson. 2007. The phonology and morphology of Arabic. OUP Oxford.[195] Christopher M White, Sanjeev Khudanpur, and James K Baker. 2008. An investiga-tion of acoustic models for multilingual code-switching. In Ninth Annual Conferenceof the International Speech Communication Association.[196] Zhangyu Xiao, Zhijian Ou, Wei Chu, and Hui Lin. 2018. Hybrid CTC-attentionbased end-to-end speech recognition using subword units. In 2018 11th InternationalSymposium on Chinese Spoken Language Processing (ISCSLP), pages 146–150.IEEE.[197] Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Con-neau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2021. Self-training andpre-training are complementary for speech recognition. In ICASSP 2021-2021 IEEEInternational Conference on Acoustics, Speech and Signal Processing (ICASSP),pages 3030–3034. IEEE.[198] Steve Young. 1996. A review of large-vocabulary continuous-speech. IEEE signalprocessing magazine, 13(5):45.[199] Chengzhu Yu, Chunlei Zhang, Shivesh Ranjan, Qian Zhang, Abhinav Misra, FinnianKelly, and John HL Hansen. 2016. UTD-CRSS system for the NIST 2015 lan-guage recognition i-vector machine learning challenge. In 2016 IEEE InternationalConference on Acoustics, Speech and Signal Processing (ICASSP), pages 5835–5839.IEEE.[200] George Zavaliagkos, J McDonough, D Miller, Amro El-Jaroudi, Jayadev Billa, FredRichardson, K Ma, M Siu, and Herbert Gish. 1998. The BBN Byblos 1997 large vo-cabulary conversational speech recognition system. In Proceedings of the 1998 IEEEInternational Conference on Acoustics, Speech and Signal Processing, ICASSP’98(Cat. No. 98CH36181), volume 2, pages 905–908. IEEE.[201] Long Zhang, Ziping Zhao, Chunmei Ma, Linlin Shan, Huazhi Sun, Lifen Jiang, Shi-wen Deng, and Chang Gao. 2020. End-to-end automatic pronunciation error detec-tion based on improved hybrid CTC/attention architecture. Sensors, 20(7):1809.77Appendix AModel HyperparametersHyperparameter RangeLearning Rate 0.01Batch Size 128K-means Clusters 1000Table A.1: HuBERT Fixed representation hyperparametersHyperparameter Distribution Type RangeLearning Rate LogUniform 0.0012761124440023646Max Steps Uniform 35956Freeze Steps Uniform 960Sample Duration Uniform 8.333333333333334 secondsBatch Size Uniform 18 13Thaw Depth Uniform 4LNA finetuning Binary TrueTable A.2: E2E XLS-r-300m finetuning hyperparametersHyperparameter Distribution Type RangeLearning Rate LogUniform 0.0006Max Steps Uniform 29225Freeze Steps Uniform 192Sample Duration Uniform 4.6875 secondsBatch Size Uniform 16Thaw Depth Uniform 3LNA finetuning Binary FalseTable A.3: E2E HuBERT finetuning hyperparameters78
Thesis/Dissertation
eng
Library and Information Studies
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Abdul-Mageed, Muhammad
Representation learning for Arabic dialect identification
Text