Science, Faculty of
Computer Science, Department of
DSpace
UBCV
González Pizarro, Felipe
Master of Science - MSc
University of British Columbia
Neural topic models can successfully find coherent and diverse topics in textual data. However, they are limited in dealing with multimodal datasets (e.g., images and text). This thesis presents the first systematic and comprehensive evaluation of multimodal topic modeling of documents containing both text and images. In the process, we propose three novel topic modeling solutions and two novel evaluation metrics. Moreover, we focus on one of our models and explore additional techniques to improve the quality of topics, such as incorporating external knowledge. Overall, our evaluation on an unprecedented rich and diverse collection of datasets indicates that all of our models generate coherent and diverse topics. Nevertheless, the extent to which one method outperforms the other depends on the metrics and dataset combinations, which suggests further exploration of combined approaches in the future.
Neural Multimodal Topic Modeling: AComprehensive EvaluationbyFelipe Gonza´lez Pizarro,MSc. in Computer Science,Universidad Te´cnica Federico Santa Mar´ıa, 2021BSc. in Computer Science, Universidad Te´cnica Federico Santa Mar´ıa, 2018A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)August 2023© Felipe Gonza´lez Pizarro, 2023The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:Neural Multimodal Topic Modeling: A Comprehensive Evaluationsubmitted by Felipe Gonza´lez Pizarro in partial fulfillment of the re-quirements for the degree of Master of Science in Computer Science.Examining Committee:Giuseppe Carenini, Professor, Department of Computer Science, UBCSupervisorVered Shwartz, Assistant Professor, Department of Computer Science, UBCSupervisory Committee MemberiiAbstractNeural topic models can successfully find coherent and diverse topics in tex-tual data. However, they are limited in dealing with multimodal datasets(e.g., images and text). This thesis presents the first systematic and compre-hensive evaluation of multimodal topic modeling of documents containingboth text and images. In the process, we propose three novel topic model-ing solutions and two novel evaluation metrics. Moreover, we focus on oneof our models and explore additional techniques to improve the quality oftopics, such as incorporating external knowledge. Overall, our evaluationon an unprecedented rich and diverse collection of datasets indicates thatall of our models generate coherent and diverse topics. Nevertheless, theextent to which one method outperforms the other depends on the metricsand dataset combinations, which suggests further exploration of combinedapproaches in the future.iiiLay SummaryThe constant increase in the volume of data has led to the developmentof various algorithms intended to summarize and understand unstructureddata. Topic modeling is a promising solution to this problem, given thatit aims to extract core themes of topics from large datasets. The majorityof topic modeling algorithms are designed to support textual data only.Our work seeks to adapt current state-of-the-art solutions to multimodalscenarios, where documents are composed of an image and text. We conducta systematic and comprehensive evaluation of our algorithms and proposetwo novel metrics to identify the coherence and diversity of the set of imagesthat represent a topic. Finally, we choose one of our neural multimodalmodels and explore different techniques to improve its performance.ivPrefaceThis thesis is an original work of the author, Felipe Gonza´lez-Pizarro, underthe supervision of Professor Giuseppe Carenini in UBC’s Department ofComputer Science.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.1 Latent Dirichlet Allocation (LDA) . . . . . . . . . . . . . . . 42.2 Neural topic models . . . . . . . . . . . . . . . . . . . . . . . 42.3 Multimodal topic models . . . . . . . . . . . . . . . . . . . . 52.4 Topic models with commonsense knowledge . . . . . . . . . . 63 Our New Multimodal Algorithms . . . . . . . . . . . . . . . 83.1 M-CombinedTM . . . . . . . . . . . . . . . . . . . . . . . . . 83.2 M-ZeroShotTM . . . . . . . . . . . . . . . . . . . . . . . . . 103.3 M-Contrast . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Performance boosting strategies . . . . . . . . . . . . . . . . 134.1 Reconstructing text features . . . . . . . . . . . . . . . . . . 134.2 Incorporating commonsense knowledge . . . . . . . . . . . . 14viTable of Contents5 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.1 Topics’ keyword evaluation . . . . . . . . . . . . . . . . . . . 175.1.1 Topic coherence . . . . . . . . . . . . . . . . . . . . . 175.1.2 Topic diversity . . . . . . . . . . . . . . . . . . . . . . 185.2 Topic’s image evaluation . . . . . . . . . . . . . . . . . . . . 185.2.1 Image Embedding-based Coherence (IEC) . . . . . . 185.2.2 Image Embedding-based Pairwise Similarity (IEPS) . 196 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206.1 MS COCO (Microsoft Common Objects in Context) . . . . . 206.2 VIST (Visual Storytelling Dataset) . . . . . . . . . . . . . . 216.3 T4SA (Twitter for Sentiment Analysis) . . . . . . . . . . . . 216.4 MMHS150K . . . . . . . . . . . . . . . . . . . . . . . . . . . 216.5 HC-4chan (Hateful content on 4chan) . . . . . . . . . . . . . 226.6 MEWA (Multimodal English Wikipedia Articles) . . . . . . 227 Experiment Setup and Further Analysis . . . . . . . . . . . 237.1 Textual Baselines . . . . . . . . . . . . . . . . . . . . . . . . 237.2 Configurations . . . . . . . . . . . . . . . . . . . . . . . . . . 237.3 Topic descriptors . . . . . . . . . . . . . . . . . . . . . . . . . 237.4 Topic Models Overlap . . . . . . . . . . . . . . . . . . . . . . 247.5 Visual learned features . . . . . . . . . . . . . . . . . . . . . 247.6 Reconstructing text features on M-ZeroShotTM . . . . . . . . 247.7 Adding Commonsense Knowledge into M-ZeroShotTM . . . . 258 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268.1 Topic Coherence and Diversity . . . . . . . . . . . . . . . . . 268.2 Topic Overlap And Visual Features . . . . . . . . . . . . . . 288.3 Boosting strategies exploration on M-ZeroShotTM . . . . . . 308.3.1 Reconstructing text features . . . . . . . . . . . . . . 308.3.2 Incorporating commonsense knowledge . . . . . . . . 319 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3310 Conclusions & Future Work . . . . . . . . . . . . . . . . . . . 36Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37viiTable of ContentsAppendixA Topics examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 47B Data preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . 48C Models and Baselines . . . . . . . . . . . . . . . . . . . . . . . 49C.1 LDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49C.2 CombinedTM and ZeroShotTM . . . . . . . . . . . . . . . . 49C.3 M-Contrast . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49C.4 M-CombinedTM and M-ZeroShotTM . . . . . . . . . . . . . 49C.5 CLIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50D Performance of M-ZeroShotTM across different λ values . . . 51E Topic coherence and diversity metrics . . . . . . . . . . . . . 52F Using a Different Text Encoder . . . . . . . . . . . . . . . . . 53G Computing Infrastructure . . . . . . . . . . . . . . . . . . . . 54H Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55viiiList of Tables6.1 Dataset document examples. To save space, we only showthe first terms of the document from MEWA. . . . . . . . . . 206.2 Properties of the datasets used. Vocab size is restricted to2000 words to speed computation . . . . . . . . . . . . . . . . 217.1 Average (M) and standard deviation (SD) of commonsenseinferences generated per document in each dataset. . . . . . . 258.1 Topic’s coherence scores. Average results across all datasetsover 4 number of topics (K = 25, 50, 75, 100), where the re-sults for each K are averaged over 5 random seeds. We useCLIP as a text encoder. We bold the highest scores. . . . . . 268.2 Topic’s coherence scores per dataset. We used the followingabbreviations: NPMI (τ), Cv (ϕ), WE (α), and IEC (γ).Average results over 4 number of topics (K = 25, 50, 75, 100),where the results for each K are averaged over 5 randomseeds. We use CLIP as a text encoder. We bold the highestscores. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278.3 Diversity scores of the top keywords and images. Averageresults over 4 number of topics (K = 25, 50, 75, 100), wherethe results for each K are averaged over 5 random seeds. Weuse CLIP as a text encoder. We bold the best scores. . . . . 278.4 Diversity scores of the top keywords and images. Averageresults over 4 number of topics (K = 25, 50, 75, 100), wherethe results for each K are averaged over 5 random seeds. Weuse CLIP as a text encoder. We bold best scores. . . . . . . 288.5 Performance of M-ZeroShotTM after reconstructing text fea-tures. We explore different ϕ values and text encoders (i.e.,CLIP, SBERT). Average results across datasets over 4 num-ber of topics (K = 25, 50, 75, 100), where the results for eachK are averaged over 5 random seeds. We bold best scores. . . 30ixList of Tables8.6 Performance of M-ZeroShotTM with/without commonsenseknowledge. † indicates the models that used commonsenseinferences. The average results across datasets are reportedfor four topics (K = 25, 50, 75, 100), and each result is aver-aged over five random seeds. . . . . . . . . . . . . . . . . . . . 31A.1 Two samples of topics in M-CombinedTM (M-CTM), M-ZeroShotTM(M-ZTM) and M-Contrast (M-C). . . . . . . . . . . . . . . . 47D.1 Topics’ quality for different values of λ on M-ZeroShotTM.Average results across multiple datasets over 4 number oftopics (K = 25, 50, 75, 100), where the results for each K areaveraged over 5 random seeds. We use CLIP as text encoder.We bold best scores. . . . . . . . . . . . . . . . . . . . . . . . 51F.1 Performance of neural topic models after using a differentcontextualized text encoder (i.e., SBERT). Average resultsacross datasets over 4 number of topics (K = 25, 50, 75, 100),where the results for each K are averaged over 5 randomseeds. We bold best scores. . . . . . . . . . . . . . . . . . . . 53H.1 Time in seconds to complete one epoch. Average resultsacross all datasets over 4 number of topics (K = 25, 50, 75,100), where the results for each K are averaged over 5 ran-dom seeds. We use CLIP as text encoder. We bold the lowesttraining time. . . . . . . . . . . . . . . . . . . . . . . . . . . 55xList of Figures3.1 High-level schema of the architecture for M-CombinedTM. Theloss function is detailed in Equation 3.1. . . . . . . . . . . . . 93.2 High-level schema of the architecture for M-ZeroShotTM. Theloss function is detailed in Equation 3.1. . . . . . . . . . . . . 113.3 M3L-Contrast topic model architecture. It includes language-specific and modality-specific encoders and inference networks.We highlight with dashes (- - -) the components that we re-moved during our adaptation. . . . . . . . . . . . . . . . . . 114.1 High-level schema for the architecture for M-ZeroShotTM, whenits decoder reconstructs BoW, image feature, and text fea-tures simultaneously. The loss function is detailed in Equa-tion 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2 Overview of the pipeline for enhancing document representa-tion by incorporating commonsense knowledge . . . . . . . . 154.3 M-ZeroShotTM with commonsense knowledge (top left): Com-monsense inferences are generated for each document, andthese inferences are encoded using a text encoder. The re-sulting embeddings are then concatenated with the text em-bedding of the original input. . . . . . . . . . . . . . . . . . . 168.1 Topic’s overlap between neural topic models. We evaluateoverlap across all datasets over 4 number of topics (K =25, 50, 75, 100), with 5 random seeds per K. . . . . . . . . . . 298.2 Images generated by using a CLIP-Guided Diffusion Modelover the latent space in M-CombinedTM. The topics from left toright are { go, walk, beach, ride}, {snow, water, lake, trees},{food, eat, delicious, drinks},{looked, cool, look, interesting},{students, speech, gave, meeting}, and {cake, family, birth-day, dinner}. . . . . . . . . . . . . . . . . . . . . . . . . . . . 29xiList of Figures8.3 Images generated by using a CLIP-Guided Diffusion Modelover the latent space in M-ZeroShotTM. The topics from leftto right are {water, lake, sand, beach}, {snow, tree, Christ-mas, white}, {cake, made, candles, birthday},{art, building,glass, amazing, architecture}, {students, graduation, speech,school}, and {family, together, happy, whole}. . . . . . . . . . 29xiiAcknowledgmentsCompleting this MSc. thesis would not have been possible without thecontinuous support and guidance of my supervisor, Giuseppe Carenini. Iam immensely grateful for his mentorship and passionate discussions onvarious research problems.I wish to acknowledge the invaluable contributions of Vered Shwartz,whose constructive comments and suggestions have significantly enhancedthe quality of this thesis. Her mentorship and impact on my academicjourney have been truly appreciated throughout the program.Furthermore, I extend my gratitude to all my colleagues and friends atUBC, especially Sahithya Ravi, Adithya Chinchure, Adam Geller, MarieSalomon, Raymond Li, Patrick Huber, Grigorii Guz, Harshinee Sriram, andeveryone associated with the Natural Language Processing research group.Your collaboration, assistance, and the memorable experiences we sharedwithin and outside UBC made this journey genuinely fulfilling.Moreover, I want to express my heartfelt thanks to my family, JuanGonza´lez, Elizabeth Pizarro, Aracely Gonza´lez, my grandfather, and myaunt for their unwavering encouragement and belief in my abilities. Theirsupport has been a constant source of motivation.I am also grateful to the Palma Family for their kindness and assistancethroughout my studies. Lastly, I want to thank my partner, Ignacio Tampe,for being my pillar of strength, providing unwavering support, and believingin me every step of the way.xiiiChapter 1IntroductionThe vast amount of text that is constantly generated has led to the develop-ment of several algorithms designed to interpret and summarize large sets ofdocuments [39]. A well-known automatic mechanism is topic modeling, arobust approach for extracting core themes or topics from large text corpora.In practice, when topic modeling is applied to a corpus (e.g., a collection ofnews articles), the resulting topic model will include a set of topics. Usu-ally, each topic is represented by a list of related terms (e.g., tropical, storm,hurricane, cyclone, weather, rain) [65]. Domain experts (e.g., journalists,physicians, marketers) can use topic modeling to analyze large documentcollections without reading every document [5, 10, 19, 26].The majority of topic modeling algorithms have been designed to pro-cess textual data only; therefore, their performance is limited in datasetswhere additional information is present in other modalities (e.g., images andvideos). Some work has attempted to address this limitation by expandingwell-known probabilistic [2, 63] or neural topic models [67] to multimodalsettings, especially considering the images contained in the documents. Yet,it is still unclear not only what method works better on which dataset butalso what evaluation metrics are more appropriate in this new multimodalscenario. In this thesis, we address this gap by performing the first system-atic and comprehensive evaluation of multimodal topic modeling of docu-ments containing both text and images. In the process, we make severalcontributions with respect to neural topic modeling algorithms, evaluationmetrics, and datasets.Neural topic modeling algorithms: We have developed three novelneural multimodal topic modeling algorithms by adapting current SOTA so-lutions. First, we extend CombinedTM [3] and ZeroShotTM [4], two neuraltopic models that use pre-trained textual embeddings, to the multimodalM-CombinedTM and M-ZeroShotTM, respectively, which additionally embedimages. In particular, in both adaptations, the Bag-Of-Words (BoW) andthe image features associated with each document are reconstructed dur-ing decoding. Secondly, we present a simplified variant, M-Contrast, de-rived from M3L-Contrast [67], a recent multimodal multilingual neural topic1Chapter 1. Introductionmodel that uses Contrastive Learning to map texts from multiple languagesand images into a shared topic space. Our M-Contrast omits the encoderand inference networks associated with a second language.Performance boosting strategies: Topic models do not always gen-erate understandable or useful results for humans [3, 23]. We focus on oneof our proposed algorithms, M-ZeroShotTM, and assess potential boostingstrategies. In particular, we evaluate if the model’s performance can beboosted by reconstructing text features and incorporating commonsenseknowledge.Metrics: The quality of a given set of topics can be automaticallyassessed primarily based on their coherence and diversity. For topic model-ing that only processes textual data, coherence metrics such as NPMI [33],Cv [47], andWE [17] evaluate the semantic relatedness of the topic keywords;while diversity metrics like TD [15] and I-RBO [3] measure the lexical over-lap between the terms of different topics. However, in a multimodal setting,each topic is represented not only by a set of keywords, but also by a setof images. Yet, there are no automatic metrics to assess the coherence andsegregation of the images representing a topic. In this thesis, we propose twonew metrics, namely Image Embedding-based Coherence (IEC) and ImageEmbedding-based Pairwise Similarity (IEPS), to fill this gap.Datasets: While previous work on multimodal topic modeling has beentested only on a few rather homogeneous datasets, we are the first to proposeand leverage six diverse datasets that vary substantially in terms of thedocument size (ranging from 6 to 2,425 words per document on average), thesource of the documents (e.g., Flickr, Twitter, Wikipedia), the underlyingtask/domain (e.g., Object recognition, Visual Storytelling) as well as in theway the gold-standard data was collected (e.g., crowd-sourcing, automaticclassification).Armed with a complete set of metrics and a diverse collection of datasets,we perform the first systematic and comprehensive evaluation of multi-modal neural topic modeling methods, comparing our three novel proposals,namely M-CombinedTM, M-ZeroShotTM and M-Contrast among themselvesand with respect to SOTA only textual topic modeling methods. Overall ourfindings reveal that the coherence and diversity of the topics’ keywords gen-erated by our proposed multimodal methods are similar to the one of theirunimodal counterparts. However, the topics’ overlap between the modelsgenerated by the different approaches is rather limited. Both these findingssuggest that ensemble solutions should be explored. Regarding the perfor-mance of our multimodal solutions, we find that overall M-ZeroShotTM canidentify more coherent and diverse topics than M-CombinedTM. As for a direct2Chapter 1. Introductioncomparison between M-ZeroShotTM and M-Contrast, none seems to domi-nate the other, with the former generating more diverse textual descriptors,while the latter more coherent and diverse image descriptors. This hints atthe potential benefits of exploring hybrid solutions in the future.This work also explores the effectiveness of additional boosting strategiesover one of our models. We first discover that when M-ZeroShotTM recon-struct Bag-Of-Words, image features, and text features simultaneously, thetopics generated by the model can exhibit higher coherence in terms of key-words and images. This approach can also enhance the diversity of topics’keywords.Additionally, we propose a methodology to incorporate commonsenseknowledge into each input document. Our approach involves using textencoders (i.e., from CLIP, SBERT) to encode the textual component ofthe documents and also a set of commonsense inferences related to everyinput. Our results show that by incorporating this external knowledge andreconstructing its text features, M-ZeroShotTM can identify more coherenttopics.The thesis is organized as follows. We summarize prior work on topicmodeling in Chapter 2. Chapter 3 introduces M-CombinedTM, M-ZeroShotTM and M-Contrast. In Chapter 4, we describe the approaches we usedto assess potential boosting strategies on M-ZeroShotTM. Chapter 5 detailsthe evaluation metrics we used for assessing topics’ descriptors, includingthose proposed for evaluating the coherence and segregation of the imagesrepresenting a topic. Chapter 6 describes six multimodal datasets usefulfor evaluating multimodal topic modeling algorithms. Chapter 7 providesadditional details about our experimental setup (e.g., textual baselines).Chapter 8 reports our findings, while Chapter 9 offers a discussion of ourresults. Finally, Chapter 10 provides our conclusions and future work.3Chapter 2Related WorkThis Chapter provides background information on Latent Dirichlet Alloca-tion (LDA), neural topic models, and multimodal topic models.2.1 Latent Dirichlet Allocation (LDA)Over the years, several topic modeling techniques have been proposed. Themost influential and popular topic modeling algorithm is Latent DirichletAllocation (LDA) [7]. This Bayesian probabilistic generative model summa-rizes a collection of documents as a set of latent topics. These latent topicsare represented by distribution over words, and a mixture of these topicsrepresents the documents. Although, over the years, the usefulness of LDAhas been shown in several domains (e.g., social media analysis [31]), it suf-fers from two key limitations. First, documents are simply represented asBoW. Secondly, it fails to generate interpretable topics in large heavy-tailedvocabularies [15].2.2 Neural topic modelsWith the recent developments of deep neural networks, several Neural TopicModels (NTMs) have been proposed to address probabilistic topic model lim-itations in terms of performance, efficiency, and usability [65]. For instance,Srivastava and Sutton [53] proposed Product-of-Experts LDA (ProdLDA), atopic modeling algorithm that still uses a BoW representation of documentsbut leverages it in a more sophisticated way by combining a variational au-toencoder [8] with a Product of Experts (PoE) approach [25]. ProdLdatakes the Bag-of-Words representation of documents as input to learn twoparameters µ and σ2 of a Dirichlet distribution. Then it samples a continu-ous latent representation from these parameters, passing through a softplus1to obtain a document-topic distribution. Finally, this document-topic dis-tribution is used to reconstruct the Bag-of-Words (BoW) representation of1SoftPlus is a smooth approximation to the ReLU function42.3. Multimodal topic modelsthe input documents. As a result, not only ProdLDA consistently identifiesmore coherent and diverse topics than LDA [51, 53], but it can also processdata more efficiently [53].Despite progress, both LDA and ProdLDA are still limited to BoW doc-ument representations. By ignoring critical syntactic and semantic relation-ships among words, they sometimes fail to identify high-quality topics [3].For instance, Burkhardt and Kramer [12] shows that while ProdLDA learnstopics of relatively high coherence, they are usually redundant, which indi-cates poor diversity of the topics.In order to incorporate semantic relationships into topic models, Dienget al. [15] proposed Embedding Topic Models (ETM), a generative probabilis-tic model that relies on static word embeddings [36] to identify interpretabletopics. Word embeddings capture the semantic relationship between termsin a lower-dimensional vector space, which also has the additional advantageof boosting performance in large heavy-tailed vocabularies.As LDA, the ETM algorithm models each document as a mixture oftopics, and each word is assigned to a particular topic. The differencesbetween these algorithms rely on the per-topic conditional probability of aterm. In ETM, the topics are modeled as points in the embedding space,and the topic-term distributions are proportional to the inner product ofthe topic’s embeddings and each term’s embeddings.Nevertheless, a remaining key shortcoming of ETM is that by relying onstatic embeddings, it does not consider contextual relations among words.This limitation was recently addressed in Bianchi et al. [3] and Bianchiet al. [4] which proposed Contextualized Topic Models (CTM). CTM is afamily of neural topic models based on a variational autoencoder (i.e., Com-binedTM [3], ZeroShotTM [4]), that relies instead on contextual embed-dings (e.g., SBERT [45]), obtaining higher quality topics than all previ-ous approaches. Two of the neural topic models we propose in this thesis,M-CombinedTM and M-ZeroShotTM, are based on this recent work.2.3 Multimodal topic modelsOnly a few topic modeling algorithms have been proposed to process morethan textual data. A notable one is SMMTM [64], an unsupervised proba-bilistic multimodal topic model designed for analyzing short-text multimodalsocial media documents. Yet, a critical limitation of this approach is thatwhile images can be associated with multiple topics, the textual content ofdocuments is assumed to belong to only one topic, which is unrealistic for52.4. Topic models with commonsense knowledgelonger documents.Not surprisingly, there has also been very recent work on leveragingcontextual embeddings for multimodal topic modeling. Zosa and Pivovarova[67] propose M3L-Contrast, which maps texts from multiple languages andimages into a shared topic space. More specifically, the model uses pre-trained image (e.g., by using CLIP [41]), and text embeddings (e.g., byusing SBERT [46]) to abstract the complexities between different languagesand modalities. The third neural topic model we propose in this thesis,M-Contrast, is based on this recent work.2.4 Topic models with commonsense knowledgeBianchi et al. [3] and Bianchi et al. [4] show that adding contextual in-formation to neural topic models can significantly improve the coherenceand diversity of generated topics. Recent work has explored expandingthese contextualized representations by injecting external knowledge, suchas commonsense knowledge [9], to improve the performance of models onNLP downstream tasks. Injecting commonsense knowledge into topic mod-eling algorithms might help to obtain a more semantically meaningful rep-resentation of the input document and, therefore, topics more aligned withcommonsense relations.Commonsense knowledge has been successfully leveraged in various taskssuch as question answering [1, 44], sentiment analysis [20, 43], and dialogue[60]. However, few attempts to incorporate commonsense knowledge intotopic modeling algorithms exist [24, 42, 66].For instance, Rajagopal et al. [42] introduced a commonsense-based topicmodeling algorithm where documents are represented as Bag-Of-Conceptsinstead of the traditional Bag-Of-Words. The model identifies relevant singlelexical items or multi-word expressions (e.g., “get good grade”) from docu-ments and retrieves for each of them related commonsense knowledge itemsfrom a knowledge graph (e.g., ConceptNet [50]). Documents are then repre-sented as the union of these commonsense knowledge items, and clusteringis applied over these representations to find the topics.Recently, Zhu et al. [66] proposed an alternative approach for incorpo-rating commonsense knowledge into topic models. Similar to the previousalgorithm, they also transform documents into a graph, where nodes areconcepts from a knowledge graph (i.e., ConceptNet [50]), and edges indicatecommonsense relationships between concepts. However, instead of a clus-tering algorithm, they use a relational graph neural network to capture the62.4. Topic models with commonsense knowledgerelational information in the graphs and manifold regularization to constraindocument topic distributions.The main limitation of previous approaches is that both algorithms heav-ily rely on the information contained in the commonsense knowledge graph,ignoring other levels of information that might be present in Bag-Of-Wordsor contextualized embeddings document representation. This can heavilyimpact the topic model’s performance when the knowledge graph is incom-plete or does not contain accurate information.While we acknowledge the relevance of incorporating commonsense knowl-edge into topic models, we also understand the benefits of using contextu-alized embeddings to represent the input documents. Thus, we developed amethodology to incorporate commonsense knowledge into a multimodal neu-ral topic model algorithm and assess its usefulness for finding more coherentand diverse topics.7Chapter 3Our New MultimodalAlgorithmsThis Chapter introduces our novel multimodal topic modeling algorithms:M-CombinedTM, M-ZeroShotTM, and M-Contrast .3.1 M-CombinedTMWe propose M-CombinedTM, a novel multimodal topic modeling algorithmbased on CombinedTM [3]. Figure 3.1 shows a high-level schema of thearchitecture of our model. Given a document with a textual and visualcomponent (e.g., an image and its caption), we encode each element usinga modality-specific encoder (e.g., by using CLIP image and text encoders).Then, the textual document representations are projected through a hiddenlayer with the same dimensionality as the vocabulary size. We concatenatethe output of this hidden layer with the BoW representation and visualdocument representation and pass them to an inference network as input.After that, the model samples a latent representation from a Gaussian dis-tribution parameterized by µ and σ2, as is done in related works [4, 53, 67].The crucial difference between our model and CombinedTM is that our de-coder network reconstructs both the BoW and the image features associatedwith each document (bottom right of Figure 3.1). Advantageously, by em-bedding images and reconstructing their features, the model can capturecomplementary information not present in the textual components of thedocument; thus, plausibly identifying topics that are more representative ofthe multimodal corpus.Our per document loss function L now includes three components:L = Eq[w⊤ log (softmax (βθ))]−KL (Q (θ | x) ∥P (θ)) + λ(1− cos(ximg, γθ)) (3.1)where the first term measures the loss associated with the BoW vectorreconstruction of the document (see [53] for more details). The second termcorresponds to the KL loss between the posterior and prior distributions83.1. M-CombinedTMTextEncoderInferencenetworkSampledrepresentationBoW reconstructioni-th textImageEncoderi-th imageLoss computationImage featuresreconstructionBoWHidden layerFigure 3.1: High-level schema of the architecture for M-CombinedTM. Theloss function is detailed in Equation 3.1.for the document embedding x (i.e., the concatenation of the output ofthe hidden layer with the BoW representation and image embedding). Themean µ and variance σ2 of the posterior distributions are estimated in eachinference network; and θ is the sampled topic distribution per documentembedding. Finally, the third term corresponds to the loss associated withthe reconstruction of image features. We calculate the cosine embeddingloss2 between the estimated image features γθ and the actual value ximg.This loss function is used to measure the degree to which two vectors aresimilar or dissimilar and is often used for learning nonlinear embeddings. λis a parameter to explore the trade-off between textual and image losses,but has been kept equal to 1 in the main experiments.2Using MSE as this loss delivers similar performance.93.2. M-ZeroShotTM3.2 M-ZeroShotTMWe also propose M-ZeroShotTM, a novel multimodal topic modeling algo-rithm based on ZeroShotTM [4]. Note that ZeroShotTM acquired its namedue to its language-independent nature and its successful performance inzero-shot cross-lingual tasks. When it receives a contextualized representa-tion of a unseen language as input (e.g., by using multilingual embeddingmodels), the model can still predict the topic distribution of the documents.Figure 3.2 shows the architecture of M-ZeroShotTM. Given a documentwith a textual and visual component (e.g., an image and its caption), weencode each element using a modality-specific encoder (e.g., by using CLIPimage and text encoders). Then, we concatenate those embeddings and passthem to an inference network as input. The key difference with M-Combined-TM is that the inference network does not receive BoW document represen-tations as input. Also, the architecture of M-ZeroShotTM does not includea hidden layer to project the text embeddings to the vocabulary size. No-tice however that the following step is the same in M-ZeroShotTM as inM-CombinedTM. The model samples a latent representation from a Gaus-sian distribution parameterized by µ and σ2. Again, the crucial differencebetween M-ZeroShotTM and ZeroShotTM is that our decoder network recon-structs both the BoW and the image features associated with each document(bottom right of Figure 3.2). Given that the task of the decoder is the sameas in M-CombinedTM, the model utilizes the same loss function detailed inEquation 3.1.3.3 M-ContrastWe propose M-Contrast as a simple adaptation of M3L-Contrast [67], a neu-ral topic modeling method originally designed to find topics in multilingualand multimodal datasets. In M3L-Contrast, each document must containan image and textual content in two languages (e.g., English and German),with the model’s architecture including three encoders and inference net-works (see Figure 3.3), each one processing either a text in one of the twolanguages or an image. In our adaptation, we essentially removed from thearchitecture the encoder and inference network for one of the two languages(dashed in Figure 3.3 i.e., the components for language B).Interestingly, the main difference between M-Contrast with M-Combined-TM and M-ZeroShotTM is the third component of the loss function L. In par-ticular, while M-CombinedTM and M-ZeroShotTM consider the loss associated103.3. M-ContrastTextEncoderInferencenetworkSampledrepresentationBoW reconstructioni-th textImageEncoderi-th imageLoss computationImage featuresreconstructionFigure 3.2: High-level schema of the architecture for M-ZeroShotTM. Theloss function is detailed in Equation 3.1.TextEncoderInferencenetworkSampledrepresentationBoW reconstructioni-th text inlanguage AImageEncoderInferencenetworkSampledrepresentationi-th imageLoss computationTextEncoderInferencenetworkSampledrepresentationBoW reconstructioni-th text inlanguage BFigure 3.3: M3L-Contrast topic model architecture. It includes language-specific and modality-specific encoders and inference networks. We highlightwith dashes (- - -) the components that we removed during our adaptation.with the reconstruction of image features, M-Contrast uses the InfoNCEContrastive Learning loss [37] to align topic distributions sampled from thedifferent modalities of the document (i.e., its textual and visual parts). This113.3. M-ContrastContrastive Learning loss maps similar instances close to each other andkeeps non-related instances apart. Overall, L combines three components:L = Eq[w⊤ log (softmax (βθtxt))]−k∑l=1KL(Q(θl | xl)∥P(θl))−ωk∑a,b=1a̸=blogexp((θa · θb) /τ)∑Nj=1∑kc,d=1 exp((θc · θdj)/τ)(3.2)The first term corresponds to the standard BoW reconstruction loss giventhe textual component of the document. The second term corresponds tothe KL loss between posterior and prior distributions for every componentk (i.e., image, text) of a document. So, it is similar but not the same as inM-CombinedTM and M-ZeroShotTM. Here, the mean µ and variance σ2 of theposterior distributions are estimated in each inference network; x can be atextual or visual embedding, and θ is a sampled topic distribution given atextual or visual component of a document. Finally, the third term is theInfoNCE loss, where (θa ·θb) are positive pairs and (θc ·θd) are negative pairs.Positive terms are aligned components of a document (e.g., an image andits caption). N is the batch size, τ is the temperature, and ω is a parameter(like λ for M-CombinedTM and M-ZeroShotTM) to explore trade-off betweencontrastive and other losses, but has been kept equal to 100 in the mainexperiments as [67].12Chapter 4Performance boostingstrategiesIn the previous Chapter, we presented three novel multimodal topic mod-eling algorithnms: M-CombinedTM, M-ZeroShotTM, and M-Contrast. In thisChapter, we introduce two additional strategies to enhance their perfor-mance. In particular, we assess the effectiveness of reconstructing text fea-tures and incorporating commonsense knowledge in M-ZeroShotTM to findmore coherent and diverse topics.4.1 Reconstructing text featuresThe decoder of our proposed multimodal neural topic modeling algorithm,M-ZeroShotTM, reconstructs image features associated with each document.We anticipate that by doing that, the model will be able to capture comple-mentary information not present in the textual components of the document;and plausibly identify topics more representative of the corpus.Additionally, the decoder is responsible for reconstructing the Bag-Of-Words representation of the document. However, this representation hasa critical limitation as it completely disregards the relationship betweenwords [66], which might affect the quality of the generated topics.As our first strategy, we explore whether adding the reconstructionof text features as an additional task for the decoder can overcome thislimitation and helps the model to capture a richer syntactic and semanticrepresentation between tokens [65]. Figure 4.1 illustrates our adapted ar-chitecture, where the decoder of M-ZeroShotTM is required to reconstructBoW, image features, and text features simultaneously. To accomplish this,we incorporate an extra term into the loss function L defined in Equation3.1:L = Eq[w⊤ log (softmax (βθ))]−KL (Q (θ | x) ∥P (θ))+λ(1− cos(ximg, γθ)) + ϕ(1− cos(xtxt, δθ))(4.1)134.2. Incorporating commonsense knowledgeTextEncoderInferencenetworkSampledrepresentationBoW reconstructioni-th textImageEncoderi-th imageLoss computationImage featuresreconstructionText featuresreconstructionFigure 4.1: High-level schema for the architecture for M-ZeroShotTM, whenits decoder reconstructs BoW, image feature, and text features simultane-ously. The loss function is detailed in Equation 4.1The last term of Equation 4.1 corresponds to the loss associated withthe reconstruction of text features. We calculate the cosine embedding lossbetween the estimated text features δθ and the actual value xtxt. Here,ϕ is a parameter that allows us to explore the weight of the text featuresreconstruction loss and balance it with the other components of the lossfunction.4.2 Incorporating commonsense knowledgeAs our second strategy, we assess the effectiveness of injecting commonsenseknowledge into a multimodal neural topic modeling algorithm to obtaintopics more aligned with commonsense relations. Our approach follows athree-step methodology (see Figure 4.2). Given a textual document, we gen-erate commonsense inferences from it by using a commonsense transformermodel. These inferences are subsequently sorted based on their relevance tothe initial input. Next, we convert each inference into natural language andincorporate them as additional input to our multimodal neural topic model.We describe each step below.144.2. Incorporating commonsense knowledgeGet commonsenseInferences usingCOMET-BART-AI2i-th text Sorting inferences(i.e, by using DeBERTa)Transformcommonsense relationsto natural languageFigure 4.2: Overview of the pipeline for enhancing document representationby incorporating commonsense knowledgeFirstly, we use Kogito3 [30], an open-source commonsense knowledge in-ference tool, to generate commonsense inferences based on the situations,entities, and events described in the text. Given a textual document (e.g.,“there are some children playing a soccer game” ), Kogito identifies heads,which correspond to relevant keywords and phrases of the document (e.g.,nouns and verbs) and matches to plausibly relations defined by Concept-Net [50] and ATOMIC [49] resulting in pairs like {“to play”, “Causes”};{“soccer game”, “AtLocation’}.This process constructs an incomplete knowledge graph of (head, rela-tion) prompts. To complete the graph, we pass it as input to a commonsensetransformer model, which generates the missing information (referred to astails). For instance, the tails generated could be {“to play”, “Causes”,“have fun”}; {“soccer game”, “AtLocation’, “soccer field”}. In our ex-periments, we employed COMET [29] to generate these tails.However, the number of commonsense inferences generated from eachdocument can be very large, and due to the token limitations of contextu-alized embeddings (i.e., 77 tokens for CLIP, and 128 tokens for SBERT),not all of them can be incorporated into the multimodal neural topic model-ing algorithm. Therefore, we use an off-the-shelf commonsense fact-linkingmodel, DeBERTa [18], to compute a relevance score for each generated com-monsense inference given the original input document. Then, we sort thecommonsense inferences based on their relevance.In the next step, we transform the generated commonsense inferencesinto natural text by utilizing the templates defined in Kogito. For instance,the transformed inferences for our given examples could be “Sometimes toplay causes to have fun”, and “You are likely to find a soccer game in a soccerfield”. These sentences are concatenated, and the resulting text is passedto the text encoder of our multimodal neural topic modeling algorithm (seeFigure 4.3).The key difference compared to our previous approach is that the infer-ence network now receives two text embeddings: one for the textual compo-3Kogito documentation is available at https://kogito.readthedocs.io/en/latest/154.2. Incorporating commonsense knowledgeTextEncoderInferencenetworkSampledrepresentationBoW reconstructioni-th textImageEncoderi-th imageLoss computationImage featuresreconstructionText featuresreconstructioni-th commonsenseinferencesFigure 4.3: M-ZeroShotTM with commonsense knowledge (top left): Com-monsense inferences are generated for each document, and these inferencesare encoded using a text encoder. The resulting embeddings are then con-catenated with the text embedding of the original input.nent of the original document and another for the commonsense-generatedinferences. Note that the remaining steps of our model remain unchanged.In our experiments, we assess the performance of M-ZeroShotTM both withand without incorporating commonsense inferences. Furthermore, we assessthe model’s performance by simultaneously reconstructing the text featuresof the original document as well as the text features of the commonsenseinferences.16Chapter 5Metrics5.1 Topics’ keyword evaluationThe quality of topic models is commonly assessed based on their coherenceand diversity. In essence, automatic coherence metrics identify the degree oflexical and semantic relatedness between the terms that describe each topic,while automatic diversity metrics measure the lexical overlap between theterms of different topics. In our evaluation, we apply standard metrics toassess the quality of the textual descriptors of the topics, namely NPMI [33],Cv [47], and WE [17] for coherence and TD [15] and I-RBO [3] for diversity.We describe each of them below.5.1.1 Topic coherenceNPMINormalized Pointwise Mutual Information (NPMI) [33] is one of the mostwell-known automatic coherence metric. It measures how much more likelythe most representative terms of a topic co-occur in the corpus than if theywere independent. NPMI returns a high score when the top N words thatdescribe a topic, summed over all pairs wi and wj , have high joint probabilityP (wj , wi) compared to their marginal probability [27]. The range of NPMI isbetween [-1,1], with a higher value indicating a more coherent topic. Usually,NPMI is calculated by using a sliding window of 10 words to identify co-occurrences.CvCv [47] uses a variation of NPMI to calculate the coherence over a slidingwindow with size 110. It calculates the co-occurrence of a word of a giventopic against all words of the same topic. It ranges between [0,1], where ahigher value suggests more coherent topics.175.2. Topic’s image evaluationWEExternal word embeddings topic coherence (WE)[16] provides an additionalmeasure of how similar the words in a topic are. It is based on word em-beddings [36]. First, it is computed as the average pairwise cosine similarityof the word embeddings of the top 10 words in a topic. Then, the overallaverage of those values for all the topics is reported.5.1.2 Topic diversityTDTopic diversity (TD) [15] is computed as the percentage of unique words inthe top N words of all topics. Its range is between [0,1]. A value near zerosuggests redundant topics, while a value near one suggests more differenttopics. In our experiments, we consider the top 10 words.I-RBOInversed Rank-Biased Overlap (I-RBO) [3] assesses the diversity of topicsby taking into account the overlap and ranking of terms. When comparingtopics, the rank of each term matters [56]; it is not the same if topics sharewords at high ranks as if they do at low ranks. I-RBO is the reciprocal of thestandard RBO [59], and it is computed by considering the top 10 words oftopics. In this metric, two topics that share some of the keywords, althoughat different rankings, are penalized less than two topics that share the samekeywords at the highest ranks. It ranges between [0,1], where a higher valuesuggests more diverse topics.5.2 Topic’s image evaluationIn multimodal topic models, each topic is represented not only by a set ofkeywords, but also by a set of images. Since there are currently no automaticmetrics for evaluating the coherence and diversity of images representingtopics, we propose two new metrics to fill this gap.5.2.1 Image Embedding-based Coherence (IEC)IEC extends WE [17] to evaluate the semantic relatedness between imagesrepresenting a topic. Formally, let T be the set of topics, and It be the set185.2. Topic’s image evaluationof top-10 images in topic t ∈ T , The average pairwise image similarity fortopic t is then:sim(It) =1(|It|2) |It|∑i=1|It|∑j=i+1cosine(ei, ej) (5.1)where cosine(ei, ej) computes relatedness between any two images ei and ejas the cosine similarity between their corresponding embeddings obtainedfrom an image embedding model. Finally, our new metric, IEC, is simplythe average of the topic-level similarity scores across all topics:IEC =1|T |∑t∈Tsim(It) (5.2)Like WE, it ranges between [0,1], where a higher value suggests more coher-ent topics.5.2.2 Image Embedding-based Pairwise Similarity (IEPS)IEPS measures the diversity of a topic model by computing the similaritybetween all the topics and then considering lower scores as a sign of higherdiversity. Formally, by adapting the Word Embedding-Based Pairwise Simi-larity (WEPS) [56] metric (originally designed for text embeddings), we firstdefine the similarity between the top-N images that describe any two topicsti and tj as follows:IEPSpair(ti, tj) =1N2∑v∈ti∑u∈tjcosine(ev, eu) (5.3)where ev, and eu denote the image embeddings associated with imagesv and u respectively.Next, to estimate the overall IEPS score for a topic model with k topics,we compute the pairwise similarities between all pairs of topics and thenaggregate them into a single score. Let S denote the set of all pairs oftopics in the model, i.e., S = (ti, tj) | 1 ≤ i < j ≤ k. Then, we can definethe overall IEPS score as:IEPS =1|S|∑(ti,tj)∈SIEPSpair(ti, tj) (5.4)This similarity metric ranges between [0,1] with a lower score indicatingmore diverse topics.19Chapter 6DatasetsWe propose a new benchmark for multimodal topic modeling comprisingsix diverse datasets. For illustration, Table 6.1 presents a sample documentfrom each dataset, while the key properties of each dataset are shown inTable 6.2. We describe a description of each dataset below.MS COCO VIST T4SA MMHS150K HC-4chan MEWAthere aresome chil-dren playinga soccergameHaving agood timebondingand talking.RT @Am-Blujay:This is whyl won’tinterfere inpeople’s re-lationshipscarol reallysaid f*ckyall I’m ad*ke andI’m hereto save theuniverseYes goyim!Don’t fightit! It’s in-evitableFlightradar24is a Swedishinternet-based ser-vice that...Table 6.1: Dataset document examples. To save space, we only show thefirst terms of the document from MEWA.6.1 MS COCO (Microsoft Common Objects inContext)MS COCO [35] is a popular dataset for image captioning, object detection,and image segmentation tasks, consisting of over 200K images labeled withbounding boxes and category labels for more than 80 object categories, aswell as five captions per image. The images were sourced from Flickr andannotated by crowd workers. For our experiments, we used a randomlyselected subset of 30K multimodal documents (image-caption pairs) to keepthe evaluation computationally feasible.206.2. VIST (Visual Storytelling Dataset)Dataset # Docs Vocab size Avg. Len Domain Data collection SourceMS COCO 30,000 2,000 10.46 Object recognition Tag-matching & crowdsourc-ingFlickrVIST 30,000 2,000 9.99 Visual storytelling Crowdsourcing FlickrT4SA 30,000 2,000 6.99 Sentiment analysis Automatic sentiment classifi-cationTwitterMMHS150K 30,000 2,000 6.03 Hateful content detection Keywords-matching & crowd-sourcingTwitterHC-4chan 17,866 1,993 17.43 Hateful content detection Human annotations & auto-matic detection4chanMEWA 18,592 2,000 2,424.83 Information retrieval Crowdsourcing WikipediaTable 6.2: Properties of the datasets used. Vocab size is restricted to 2000words to speed computation6.2 VIST (Visual Storytelling Dataset)VIST [28] is a dataset of multimodal stories comprising sequences of im-ages with corresponding descriptions. All images come from Flickr, and thetextual stories were crowd-sourced. For our experiments, we again randomlyselected from VIST 30K multimodal documents, each consisting of an imagealong with the description of the portion of the story associated with theimage.6.3 T4SA (Twitter for Sentiment Analysis)T4SA [58] is a large-scaleTwitter dataset designed for SentimentAnalysis.It contains textual and multimodal data obtained through the TwitterAPIand each tweet is automatically annotated with its sentiment polarity. In ourexperiments, we used a random sample of 30K multimodal tweets balancedacross sentiment classes.6.4 MMHS150KMMHS150K [21] is a hate speech dataset consisting of 150K Twitter mul-timodal documents. Each document was then labelled by crowdsourcingwith the particular community that was attacked, such as racist, sexist, orhomophobic. Once again, we selected 30K random documents.216.5. HC-4chan (Hateful content on 4chan)6.5 HC-4chan (Hateful content on 4chan)HC-4chan [22] contains posts, phrases, and images containing hateful anddiscriminatory content. It consists of 21K images, identified as presentingAntisemitic/Islamophobic content by CLIP. We removed near-duplicate im-ages and documents with no text, resulting in a dataset of 17K multimodaldocuments.6.6 MEWA (Multimodal English WikipediaArticles)MEWA [67] comprises English Wikipedia Articles from the WikipediaComparable Corpora4, aligned with images from the Wikipedia-based ImageText dataset (WIT) [52]. Each document consists of a complete EnglishWikipedia Article and its corresponding image. For our analysis, we usedthe publicly available subset of 18.5K documents5.4linguatools.org/tools/corpora/wikipedia-comparable-corpora/5github.com/ezosa/M3L-topic-model/tree/master/data22Chapter 7Experiment Setup andFurther Analysis7.1 Textual BaselinesWe consider LDA [6], ZeroShotTM [4], and CombinedTM [3] as strong tex-tual baselines. For LDA, we use the OCTIS [54] implementation, whereasfor the neural topic models, the author’s original ones6.7.2 ConfigurationsWe encode text and images with OpenAI’s CLIP [40], which captures contentsimilarity across modalities. We train all models with similar hyperparame-ter settings, following [4] and [67] for the choice of specific values. We trainmodels for 100 epochs, computing all the metrics for 25, 50, 75, and 100topics. Results for each metric are averaged over 5 random seeds.7.3 Topic descriptorsVAE-based neural topic models like our M-CombinedTM, M-ZeroShotTM andM-Contrast obtain the representative keywords of each topic from the topic-vocab weight matrix used for reconstructing the BoW. However, M-Contrastdoes not reconstruct image features (see Figure 3.3), so a different approachis needed to obtain the most relevant images per topic. To this end, af-ter model training, we rely on the document-topic distributions associatedwith the input documents and select the images of the N documents withthe highest contribution for each topic. We use this same approach forM-CombinedTM and M-ZeroShotTM to ensure a fair comparison. All our ex-periments are run considering the top 10 words and images per topic.6github.com/MilaNLProc/contextualized-topic-models237.4. Topic Models Overlap7.4 Topic Models OverlapTwo topic models might exhibit similar coherence and diversity, but actuallygenerate different topics. So, as part of our systematic comparison betweenM-CombinedTM, M-ZeroShotTM and M-Contrast, we also explore the overlapbetween the topics they generate. Specifically, given a pair of topic models,we construct a topic similarity matrix based on the most relevant keywordsusing the I-RBO diversity metric [3], ranging in [0-1] with higher scoresindicating more substantial topic overlap. Then, to align the topics betweenmodels, we apply the Hungarian method [32]. In the results, we report themean M and standard deviation SD of the topic overlap between modelsacross multiple datasets, numbers of topics, and random seeds.7.5 Visual learned featuresIn order to reconstruct image embeddings from the input, M-CombinedTMand M-ZeroShotTM use a weight topic-image features matrix. After trainingthe models, analyzing such structure can provide valuable insights on therelevance of specific visual attributes to each topic as well as into the neuralmodel’s limitations. In our experiments, we use a CLIP Guided Diffusionmodel [14]7 to generate an image per topic given the weight topic-imagefeature matrix.7.6 Reconstructing text features on M-ZeroShotTMIn Equation 4.1, the parameter ϕ determines the weight of the text featuresreconstruction loss and its balance with other components in the loss func-tion. In our experiments, we explore various values of ϕ. When ϕ is setto 1, the text features reconstruction loss falls within the range of 0 to 1,considering our utilization of a cosine embedding loss. This value might berelatively small compared to the loss associated with BoW reconstruction.To address this, we examine alternative values for ϕ. In particular, we in-crease ϕ to match the magnitude of the BoW reconstruction loss, double themagnitude of the BoW reconstruction loss, and quadruple the magnitude ofthe BoW reconstruction loss.Furthermore, we present our findings using two different text encoders:one from CLIP [41], which supports a maximum of 77 tokens, and SBERT [45],which has a limit of 128 tokens.7github.com/nerdyrodent/CLIP-Guided-Diffusion247.7. Adding Commonsense Knowledge into M-ZeroShotTM7.7 Adding Commonsense Knowledge intoM-ZeroShotTMTo identify the relevant keywords and phrases in each document, we employthe default head extraction methods provided in Kogito. Specifically, we uti-lize noun phrase extraction, verb phrase extraction, and sentence extractionmethods. These extracted heads are then matched to knowledge relationsusing a Simple Heuristics-based Relation Matching algorithm, which is thedefault approach employed in the Python package.For generating commonsense inferences, we rely on the COMET knowl-edge model based on BART (COMET-BART-AI2), which is available withinKogito. To determine the relevance of each commonsense inference to theinput document, we utilize the Deberta-based linker. We do not remove anycommonsense inferences; instead, we use the relevance score to sort them.When concatenating commonsense inferences, those with a higher relevancescore are prioritized and placed at the beginning.Table 7.1 shows the average number of commonsense inferences gener-ated per document in each datasetMS COCO VIST T4SA MMHS150K HC-4chan MEWAM SD M SD M SD M SD M SD M SD53.6 11.8 47.8 18.2 52.3 21.0 45.8 20.5 52.5 57.3 119.0 22.9Table 7.1: Average (M) and standard deviation (SD) of commonsense in-ferences generated per document in each dataset.In the supplementary materials, we provide more details about our datapreprocessing choices, model and baseline parameters, and the specificationsof our computing infrastructure.25Chapter 8Results8.1 Topic Coherence and DiversityTable 8.1 shows the overall performance of the models in terms of the coher-ence of the topics’ descriptors. For the metrics assessing the textual descrip-tors, as expected LDA performs worse, while M-CombinedTM, M-ZeroShotTMand M-Contrast perform similarly to CombinedTM and ZeroShotTM, in-dicating that processing the images of the corpus does not influence thecoherence of the textual descriptors.Text ImageModels NPMI Cv WE IECLDA -0.14 .39 .15CombinedTM .04 .52 .21ZeroShotTM .03 .51 .22M-Contrast .04 .50 .22 .67M-CombinedTM .03 .51 .21 .54M-ZeroShotTM .03 .51 .22 .56Table 8.1: Topic’s coherence scores. Average results across all datasets over4 number of topics (K = 25, 50, 75, 100), where the results for each K areaveraged over 5 random seeds. We use CLIP as a text encoder. We boldthe highest scores.Regarding multimodal neural topic models, we observe that M-ZeroShot-TM shows a slight advantage over M-CombinedTM in generating topics withmore coherent keywords and images. This is indicated by slightly higherscores in both the WE and IEC metrics overall. Interestingly, we also ob-serve subtle differences for the NPMI and Cv scores between M-ZeroShotTMand M-Contrast. As shown in Table 8.2, depending on the metrics anddataset combinations, one model outperforms the other without a clear win-ner. For instance, M-ZeroShotTM can generate more coherent topics thanM-Contrast in the VIST, T4SA, and MMHS150 datasets, but that is notthe case for the MS COCO and MEWA datasets. With respect to the im-268.1. Topic Coherence and DiversityDataset MS COCO VIST T4SA MMHS150K HC-4chan MEWAText Image Text Image Text Image Text Image Text Image Text ImageModels τ ϕ α γ τ ϕ α γ τ ϕ α γ τ ϕ α γ τ ϕ α γ τ ϕ α γLDA -.10 .37 .14 -.23 .34 .13 -.25 .36 .13 -.17 .31 .16 -.21 .40 .13 .10 .57 .20CombinedTM .13 .55 .22 -.01 .42 .21 .02 .45 .22 .00 .45 .22 -.07 .58 .21 .13 .64 .20ZeroShotTM .12 .54 .23 -.02 .39 .23 .02 .44 .23 .00 .44 .22 -.07 .57 .22 .14 .67 .22M-Contrast .13 .56 .23 .75 -.03 .35 .23 .69 -.02 .38 .20 .59 -.01 .40 .23 .65 -.02 .54 .19 .65 .16 .77 .20 .68M-CombinedTM .13 .55 .22 .65 -.01 .40 .21 .52 .02 .44 .22 .45 .01 .46 .22 .55 -.06 .58 .21 .57 .13 .64 .20 .52M-ZeroShotTM .12 .54 .23 .66 -.01 .39 .23 .56 .01 .44 .23 .47 .00 .45 .22 .56 -.07 .58 .21 .57 .14 .66 .21 .54Table 8.2: Topic’s coherence scores per dataset. We used the followingabbreviations: NPMI (τ), Cv (ϕ), WE (α), and IEC (γ). Average resultsover 4 number of topics (K = 25, 50, 75, 100), where the results for each Kare averaged over 5 random seeds. We use CLIP as a text encoder. We boldthe highest scores.age descriptors, we find that the top relevant images of each topic identifiedby M-Contrast are more related (i.e., higher IEC) than those selected byM-ZeroShotTM overall and across datasets.Table 8.3 and 8.4 report the performance of the models in terms of theirability to generate diverse topics. LDA is the top performer, but it scoredthe lowest for coherence by a wide margin (see Table 8.2 and 8.1).Text ImageModels TD I-RBO IEPSLDA .84 .97CombinedTM .50 .96ZeroShotTM .60 .98M-Contrast .47 .94 .41M-CombinedTM .50 .96 .45M-ZeroShotTM .60 .98 .44Table 8.3: Diversity scores of the top keywords and images. Average resultsover 4 number of topics (K = 25, 50, 75, 100), where the results for each Kare averaged over 5 random seeds. We use CLIP as a text encoder. We boldthe best scores.Once again, we observe that M-ZeroShotTM outperforms M-CombinedTMoverall in terms of generating topics with a more diverse set of keywordsand images. This is indicated by higher scores in the TD, I-RBO, and IEPSmetrics. We also observe that for all the datasets, the keywords generatedby M-ZeroShotTM are significantly more diverse than those generated byM-Contrast. Finally, we find that the images representing the topics aremore diverse (i.e., lower IEPS) in M-Contrast than in M-ZeroShotTM.278.2. Topic Overlap And Visual FeaturesDataset MS COCO VIST T4SA MMHS150KK HC-4chan MEWAText Image Text Image Text Image Text Image Text Image Text ImageModels TD I-RBO IEPS TD I-RBO IEPS TD I-RBO IEPS TD I-RBO IEPS TD I-RBO IEPS TD I-RBO IEPSLDA .75 .98 .93 1.00 .96 1.00 .85 .90 .89 1.00 .65 .96CombinedTM .57 .98 .40 .94 .43 .96 .41 .95 .40 .94 .76 .99ZeroShotTM .67 .99 .60 .99 .56 .98 .50 .97 .50 .96 .77 .99M-Contrast .65 .99 .46 .56 .98 .45 .52 .97 .28 .29 .87 .41 .23 .87 .50 .58 .98 .37M-CombinedTM .56 .98 .47 .40 .92 .48 .42 .96 .35 .42 .95 .46 .41 .94 .54 .77 .99 .39M-ZeroShotTM .68 .99 .47 .60 .99 .48 .56 .98 .34 .50 .97 .45 .50 .96 .53 .77 .99 .38Table 8.4: Diversity scores of the top keywords and images. Average resultsover 4 number of topics (K = 25, 50, 75, 100), where the results for each Kare averaged over 5 random seeds. We use CLIP as a text encoder. We boldbest scores.8.2 Topic Overlap And Visual FeaturesFigure 8.1 shows the topic overlap between CombinedTM, ZeroShotTM,and our proposed multimodal neural topic models. Remarkably, althoughCombinedTM and M-CombinedTM exhibit similar performance in terms ofcoherence and diversity, the generated topics only partially overlap (M =.46; SD =.23). We observe a similar pattern in the topic overlap betweenZeroShotTM and M-ZeroShotTM (M = .50; SD =.23). These findings sug-gest that the multimodal alternatives generate some topics that are similar tothose generated by the unimodal parts, but they also produce unique topics.Moreover, M-Contrast produces topics with significantly less keyword over-lap when compared to ZeroShotTM (M = .22; SD =.16), CombinedTM (M= .19; SD =.15), M-ZeroShotTM (M = .21; SD =.16), and M-CombinedTM(M = .19; SD =.15).By examining the topic-image features matrix from M-CombinedTM andM-ZeroShotTM, we can gain valuable insights regarding the relevance of in-dividual visual attributes to each topic and assess whether there is an align-ment between the learned visual features and topics’ keywords.Figure 8.2 and Figure 8.3 showcase images generated by the CLIP-Guided diffusion model considering the topic-image feature matrix γ fromM-CombinedTM and M-ZeroShotTM, respectively. The topics were extractedfrom the VIST dataset. Promisingly, the generated images seem to alignwell with the topic’s descriptors and capture more abstract and complexconcepts (like happy family).288.2. Topic Overlap And Visual FeaturesZeroShotTMCombined-TMM-ZeroShotTMM-CombinedTM0.49ZeroShotTMCombined-TMM-ZeroShotTMM-CombinedTM0.50 0.46ZeroShotTMCombined-TMM-ZeroShotTMM-CombinedTM0.45 0.46 0.49ZeroShotTMCombined-TMM-ZeroShotTMM-CombinedTM0.22 0.19 0.21 0.19Combined-TMM-ZeroShotTMM-CombinedTMM-ContrastFigure 8.1: Topic’s overlap between neural topic models. We evaluate over-lap across all datasets over 4 number of topics (K = 25, 50, 75, 100), with 5random seeds per K.Figure 8.2: Images generated by using a CLIP-Guided Diffusion Modelover the latent space in M-CombinedTM. The topics from left to right are{ go, walk, beach, ride}, {snow, water, lake, trees}, {food, eat, delicious,drinks},{looked, cool, look, interesting}, {students, speech, gave, meeting},and {cake, family, birthday, dinner}.Figure 8.3: Images generated by using a CLIP-Guided Diffusion Model overthe latent space in M-ZeroShotTM. The topics from left to right are {water,lake, sand, beach}, {snow, tree, Christmas, white}, {cake, made, candles,birthday},{art, building, glass, amazing, architecture}, {students, gradua-tion, speech, school}, and {family, together, happy, whole}.298.3. Boosting strategies exploration on M-ZeroShotTM8.3 Boosting strategies exploration onM-ZeroShotTM8.3.1 Reconstructing text featuresTable 8.5 shows the overall performance of M-ZeroShotTM with and with-out text features reconstruction. The evaluation considers different textencoders (i.e., CLIP and SBERT) and ϕ values.Text Image Text ImageEncoder Settings NPMI Cv WE IEC TD I-RBO IEPSCLIPNo reconstruction .03 .51 .22 .56 .60 .98 .44ϕ = 1.0 .03 .51 .22 .56 .60 .98 .44ϕ = 1x BoW loss .02 .49 .23 .58 .61 .98 .44ϕ = 2x BoW loss .01 .48 .23 .58 .60 .98 .44ϕ = 4x BoW loss .01 .48 .23 .58 .60 .97 .44SBERTNo reconstruction .03 .51 .22 .56 .58 .98 .44ϕ = 1.0 .03 .51 .22 .56 .58 .98 .44ϕ = 1x BoW loss .02 .49 .24 .57 .65 .98 .44ϕ = 2x BoW loss .01 .49 .24 .57 .66 .98 .44ϕ = 4x BoW loss .01 .48 .24 .56 .66 .98 .44Table 8.5: Performance of M-ZeroShotTM after reconstructing text features.We explore different ϕ values and text encoders (i.e., CLIP, SBERT). Av-erage results across datasets over 4 number of topics (K = 25, 50, 75, 100),where the results for each K are averaged over 5 random seeds. We boldbest scores.We observe that when the text feature reconstruction loss is initially ofcomparable magnitude to the BoW reconstruction term (i.e., ϕ = 1X BoWloss), the model exhibits the ability to generate topics with more semanti-cally related keywords, as indicated by higher WE scores. However, there isa decrease in the NPMI, and Cv scores, suggesting that in this approach, theselected keywords for each topic do not appear to co-occur more frequentlyin the corpus.Moreover, reconstructing text features results in a more diverse set oftopics, as the TD metric indicates. This improvement is particularly promi-nent when documents are encoded with SBERT.As anticipated, the reconstruction of text features does not appear toimpact the diversity of the images associated with topics, as suggested bythe IEPS scores. However, an interesting observation is that the text features308.3. Boosting strategies exploration on M-ZeroShotTMreconstruction can lead to topics described by a more coherent set of images,as indicated by higher IEC scores.8.3.2 Incorporating commonsense knowledgeTable 8.6 shows the overall performance of M-ZeroShotTM when common-sense inferences are added to enhance the document representation. Ourevaluation considers different text encoders (i.e., CLIP and SBERT) andexplores the impact of text features reconstruction, which involves both thetext features related to the original document and those related to the com-monsense inferences.Text Image Text ImageEncoder Settings NPMI Cv WE IEC TD I-RBO IEPSCLIPNo reconstruction .03 .51 .22 .56 .60 .98 .44No reconstruction † .03 .51 .22 .56 .60 .98 .44ϕ = 1.0 .03 .51 .22 .56 .60 .98 .44ϕ = 1.0† .03 .51 .22 .56 .60 .98 .44ϕ = 1x BoW loss .02 .49 .23 .58 .61 .98 .44ϕ = 1x BoW loss† .02 .50 .23 .58 .60 .98 .44ϕ = 2x BoW loss .01 .48 .23 .58 .60 .98 .44ϕ = 2x BoW loss† .02 .49 .23 .58 .59 .98 .44ϕ = 4x BoW loss .01 .48 .23 .58 .60 .97 .44ϕ = 4x BoW loss† .01 .49 .23 .57 .59 .97 .44SBERTNo reconstruction .03 .51 .22 .56 .58 .98 .44No reconstruction † .03 .51 .22 .56 .60 .98 .44ϕ = 1.0 .03 .51 .22 .56 .58 .98 .44ϕ = 1.0† .03 .51 .22 .56 .60 .98 .44ϕ = 1.0x BoW loss .02 .49 .24 .57 .65 .98 .44ϕ = 1x BoW loss† .02 .50 .24 .57 .66 .98 .44ϕ = 2x BoW loss .01 .49 .24 .57 .66 .98 .44ϕ = 2x BoW loss† .01 .49 .25 .57 .67 .98 .44ϕ = 4x BoW loss .01 .48 .24 .56 .66 .98 .44ϕ = 4x BoW loss† .01 .48 .25 .57 .66 .98 .44Table 8.6: Performance of M-ZeroShotTM with/without commonsenseknowledge. † indicates the models that used commonsense inferences.The average results across datasets are reported for four topics (K =25, 50, 75, 100), and each result is averaged over five random seeds.Our results indicate that integrating commonsense inferences into themodel could yield topics with more coherent keywords for specific combi-nations of text encoders and ϕ values. For example, when documents areencoded with CLIP and the magnitude of the text feature reconstruction is318.3. Boosting strategies exploration on M-ZeroShotTMat least as large as the BoW reconstruction term, the inclusion of common-sense inferences leads to higher Cv scores (e.g., the Cv score increases from.49 to .50 when ϕ = 1x BoW loss). However, this performance enhancementcomes with a trade-off, as it results in slightly lower topic diversity scores,as indicated by the TD metric (e.g., the TD score decreases from .61 to .60when ϕ = 1x BoW loss).When documents are encoded using SBERT, the inclusion of common-sense inferences results in the model generating topics of equal or higherquality compared to those produced without incorporating commonsenseknowledge. Depending on the ϕ values, incorporating commonsense knowl-edge can lead to topics with more coherent keywords, as demonstrated byimprovements in the Cv, and WE scores (e.g., the WE score increases from.24 to .25 when ϕ = 2x BoW loss). Notably, we observe that when SBERTis employed, incorporating commonsense inferences leads to topics with agreater diversity of keywords, even in scenarios where text feature recon-struction is not performed (e.g., the TD score increases from .58 to .60 whenno text feature reconstruction is performed and commonsense inferences areadded).32Chapter 9DiscussionIn this thesis, we introduce three novel multimodal neural topic modelingalgorithms. To assess the effectiveness of our approaches, we measure thecoherence and diversity of generated topics on datasets consisting of imageand text pairs. To achieve this, we propose two new metrics designed toassess the coherence and diversity of the set of images representing topics.Our experiments lead to several findings; we discuss their implications below.Overall, we identify that M-CombinedTM and M-ZeroShotTM generate top-ics with similar quality to those generated by their unimodal counterparts:CombinedTM and ZeroShotTM, respectively. While the multimodal so-lutions generate some topics that overlap with the ones produced by theunimodal algorithms, they also produce a set of unique topics. This isencouraging, considering that M-CombinedTM and M-ZeroShotTM allow ex-tracting topics from datasets where text and image content coexist and needto be analyzed jointly. Each modality might capture different aspects of in-formation and complement each other [48]. By integrating both modalities,multimodal neural topic models can extract more meaningful and semanticcorpus representations than their monolingual counterparts [57].Our results reveal that, overall, M-ZeroShoTM outperforms M-CombinedTMin terms of topic coherence and diversity. Interestingly, despite receivingas additional input the BoW document representation, M-CombinedTM failsto generate topics of higher quality than M-ZeroShotTM. We attribute thisperformance gap to subtle architectural differences between the two models.Specifically, M-CombinedTM incorporates a hidden layer to project the textualembeddings to the vocabulary size, a component absent in M-ZeroShotTM.As a consequence, M-CombinedTM contains a higher number of parameters,which can make its training more challenging and susceptible to overfitting.Our results also show a superiority of M-Contrast over M-ZeroShotTM interms of image coherence and diversity. Plausibly, this can be attributed totheir different training objectives. M-Contrast incorporates a ContrastiveLearning loss [13] to maximize the similarity between positive pairs (text-image pairs that belong together) while minimizing the similarity betweennegative pairs. Training the model to explicitly differentiate between related33Chapter 9. Discussionand unrelated pairs may stimulate the model to learn more discriminativeimage representations [61], which can better support similarly judgmentsinvolved in creating more coherent and diverse topic models. More specu-latively, the reason why this works for image but not for text descriptorsmay be due to the fact that although pre-trained multimodal representa-tions (e.g., CLIP) map data from different modalities into the same space,embeddings from different modalities are located in separate regions [34]and, therefore could be influenced differently by Contrastive Learning.The effectiveness of our boosting strategies over M-ZeroShotTM high-lights the need for further exploration in enhancing the robustness of neuraltopic models. Our findings indicate that by reconstructing text features,multimodal neural topic models could take advantage of the additional in-formation contained in contextualized embeddings, which is absent in thetraditional BoW document representation. This approach has the potentialto generate topics of higher quality. In the same way, our findings sug-gest that commonsense knowledge can be beneficial for neural topic models,given that they can obtain a better document representation and potentiallygenerate topics that align more closely with commonsense relations.Given the recent success of decoder-only GPT-like systems [38] in somany NLP tasks, at first glance, it may seem surprising that they have notbeen applied yet to the topic modeling task. However, the plain reason isthat they are still severely limited in their input size (currently in the 1000sof tokens) [11] and therefore cannot process the large corpora for whichtopic modeling is actually needed. Extremely recent work (e.g., [62]) mayinspire ideas on how to overcome this limitation in the future. Tellingly, theevaluation framework and baselines presented in this thesis will be criticalin assessing these new solutions.Our proposed models can benefit users in different ways. For instance,with M-Contrast, which incorporates two inference networks (one per modal-ity), users can identify topic distributions from documents available in asingle modality (e.g., a set of images without any accompanying caption).This capability is achieved after training the model on text and image pairs.Our proposed neural multimodal topic models are intended solely forresearch purposes. Any use of these models or their derived artifacts out-side of research contexts should be authorized accordingly. We use datasetspublicly available and from recent literature. Users must be aware of thepotential risks of using topic modeling algorithms. First, topic models mayamplify biases present in the training data. For instance, if the dataset con-tains hateful content (e.g., antisemitic/islamophobic text and images), thegenerating topics can perpetuate those discriminatory practices. Users need34Chapter 9. Discussionto consider not only the biases and limitations of datasets but also the biasesof text and image encoders (e.g., CLIP). Moreover, neural topic models lacktransparency and interpretability. This means it becomes challenging to un-derstand how the model arrives at particular topics. Not considering theselimitations may cause users to make wrong assumptions about the corpus.35Chapter 10Conclusions & Future WorkWe present the first systematic evaluation of neural multimodal topic mod-eling, highlighting the importance of considering multiple non-homogeneousdatasets and a comprehensive set of evaluation metrics. In particular, wecontribute a repository of corpora that vary in document size, source, andunderlying task/domain, along with two novel metrics to assess topic im-age descriptors’ coherence and diversity. We apply the resulting evaluationframework to compare three novel multimodal topic modeling methods thatwe developed by adapting current SOTA architectures. Overall our resultsindicate that ensemble and hybrid solutions should be explored in the fu-ture, for instance by either merging the output of different models or bycombining different components in more complex loss functions. LeveragingGPT-like systems [38] is also a potential direction for future work, but firstthe formidable limitation in their input size [11] must be addressed. In an-other short-term direction we plan to assess whether the topic-image featurematrix γ (Eq.1) can benefit multimodal text classification and documentsimilarity. Furthermore, we intend to evaluate the efficacy of text feature re-construction and incorporation of commonsense knowledge in our other mul-timodal neural topic modeling solutions: M-CombinedTM and M-Contrast.As in any study, ours has limitations that need to be considered. First,we used datasets that are only available in English, which might restrict thegeneralizability of our findings. Secondly, we focus our topics’ evaluation ontheir coherence and diversity. Future work should identify the quality of theresults based on other aspects, such as document coverage (i.e., how welldocuments match their assigned topics) and topic model comprehensiveness(i.e., how thoroughly the model covers the topics appearing in the corpus).These aspects are challenging to assess when ground truth is unavailable.While we focus our experiments on automatic metrics, it is also important todesign and conduct human evaluations to understand the limitations of neu-ral topic models. Finally, future work should explore how hyperparameters(e.g., dropout rate, weight for KL divergence loss) impact neural multimodaltopic models.36Bibliography[1] Lisa Bauer, Yicheng Wang, and Mohit Bansal. Commonsense forgenerative multi-hop question answering tasks. In Proceedings of the2018 Conference on Empirical Methods in Natural Language Process-ing, pages 4220–4230, Brussels, Belgium, October-November 2018. As-sociation for Computational Linguistics. doi: 10.18653/v1/D18-1454.URL https://aclanthology.org/D18-1454.[2] Jingwen Bian, Yang Yang, and Tat-Seng Chua. Multimedia summariza-tion for trending topics in microblogs. In Proceedings of the 22nd ACMInternational Conference on Information and Knowledge Management,CIKM ’13, page 1807–1812, New York, NY, USA, 2013. Association forComputing Machinery. ISBN 9781450322638. doi: 10.1145/2505515.2505652. URL https://doi.org/10.1145/2505515.2505652.[3] Federico Bianchi, Silvia Terragni, and Dirk Hovy. Pre-training is ahot topic: Contextualized document embeddings improve topic co-herence. In Proceedings of the 59th Annual Meeting of the Associa-tion for Computational Linguistics and the 11th International JointConference on Natural Language Processing (Volume 2: Short Pa-pers), pages 759–766, Online, August 2021. Association for Compu-tational Linguistics. doi: 10.18653/v1/2021.acl-short.96. URL https://aclanthology.org/2021.acl-short.96.[4] Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elis-abetta Fersini. Cross-lingual contextualized topic models with zero-shot learning. In Proceedings of the 16th Conference of the Euro-pean Chapter of the Association for Computational Linguistics: MainVolume, pages 1676–1683, Online, April 2021. Association for Com-putational Linguistics. doi: 10.18653/v1/2021.eacl-main.143. URLhttps://aclanthology.org/2021.eacl-main.143.[5] David Blei, Lawrence Carin, and David Dunson. Probabilistic topicmodels. IEEE signal processing magazine, 27(6):55–65, 2010.37Bibliography[6] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichletallocation. J. Mach. Learn. Res., 3(null):993–1022, mar 2003. ISSN1532-4435.[7] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirich-let allocation. Journal of machine Learning research, 3(Jan):993–1022,2003.[8] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational in-ference: A review for statisticians. Journal of the American statisticalAssociation, 112(518):859–877, 2017.[9] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya,Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformersfor automatic knowledge graph construction. In Proceedings of the 57thAnnual Meeting of the Association for Computational Linguistics, pages4762–4779, 2019.[10] Jordan Boyd-Graber, Yuening Hu, David Mimno, et al. Applicationsof topic models. Foundations and Trends® in Information Retrieval,11(2-3):143–296, 2017.[11] Se´bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, JohannesGehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, YuanzhiLi, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro,and Yi Zhang. Sparks of artificial general intelligence: Early experi-ments with gpt-4, 2023.[12] Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smooth-ness in the dirichlet variational autoencoder topic model. J. Mach.Learn. Res., 20(131):1–27, 2019.[13] S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity met-ric discriminatively, with application to face verification. In 2005IEEE Computer Society Conference on Computer Vision and PatternRecognition (CVPR’05), volume 1, pages 539–546 vol. 1, 2005. doi:10.1109/CVPR.2005.202.[14] Prafulla Dhariwal and Alexander Nichol. Diffusion modelsbeat gans on image synthesis. In M. Ranzato, A. Beygelz-imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan,editors, Advances in Neural Information Processing Systems,volume 34, pages 8780–8794. Curran Associates, Inc., 2021.38BibliographyURL https://proceedings.neurips.cc/paper_files/paper/2021/file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf.[15] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling inembedding spaces. Transactions of the Association for ComputationalLinguistics, 8:439–453, 2020.[16] Ran Ding, Ramesh Nallapati, and Bing Xiang. Coherence-aware neu-ral topic modeling. In Proceedings of the 2018 Conference on Empiri-cal Methods in Natural Language Processing, pages 830–836, Brussels,Belgium, October-November 2018. Association for Computational Lin-guistics. doi: 10.18653/v1/D18-1096. URL https://aclanthology.org/D18-1096.[17] Anjie Fang, Craig Macdonald, Iadh Ounis, and Philip Habel. Usingword embedding to evaluate the coherence of topics from twitter data.In Proceedings of the 39th International ACM SIGIR Conference onResearch and Development in Information Retrieval, SIGIR ’16, page1057–1060, New York, NY, USA, 2016. Association for Computing Ma-chinery. ISBN 9781450340694. doi: 10.1145/2911451.2914729. URLhttps://doi.org/10.1145/2911451.2914729.[18] Silin Gao, Jena D. Hwang, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji,and Antoine Bosselut. ComFact: A benchmark for linking contextualcommonsense knowledge. In Findings of the Association for Computa-tional Linguistics: EMNLP 2022, pages 1656–1675, Abu Dhabi, UnitedArab Emirates, December 2022. Association for Computational Lin-guistics. URL https://aclanthology.org/2022.findings-emnlp.120.[19] Amon Ge, Hyeju Jang, Giuseppe Carenini, Kendall Ho, and Young JiLee. Octvis: ontology-based comparison of topic models. In 2019 IEEEVisualization Conference (VIS), pages 66–70. IEEE, 2019.[20] Deepanway Ghosal, Devamanyu Hazarika, Abhinaba Roy, Navonil Ma-jumder, Rada Mihalcea, and Soujanya Poria. KinGDOM: Knowledge-Guided DOMain Adaptation for Sentiment Analysis. In Proceedings ofthe 58th Annual Meeting of the Association for Computational Lin-guistics, pages 3198–3210, Online, July 2020. Association for Com-putational Linguistics. doi: 10.18653/v1/2020.acl-main.292. URLhttps://aclanthology.org/2020.acl-main.292.39Bibliography[21] Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas.Exploring hate speech detection in multimodal publications. In Proceed-ings of the IEEE/CVF Winter Conference on Applications of ComputerVision (WACV), March 2020.[22] Felipe Gonza´lez-Pizarro and Savvas Zannettou. Understanding anddetecting hateful content using contrastive learning. arXiv preprintarXiv:2201.08387, 2022.[23] Ismail Harrando and Raphae¨l Troncy. Discovering interpretable topicsby leveraging common sense knowledge. In Proceedings of the 11thon Knowledge Capture Conference, K-CAP ’21, page 265–268, NewYork, NY, USA, 2021. Association for Computing Machinery. ISBN9781450384575. doi: 10.1145/3460210.3493586. URL https://doi.org/10.1145/3460210.3493586.[24] Ismail Harrando and Raphae¨l Troncy. Discovering interpretable topicsby leveraging common sense knowledge. In Proceedings of the 11th onKnowledge Capture Conference, pages 265–268, 2021.[25] Geoffrey E Hinton. Training products of experts by minimizing con-trastive divergence. Neural computation, 14(8):1771–1800, 2002.[26] Enamul Hoque and Giuseppe Carenini. Interactive topic hierarchy re-vision for exploring a collection of online conversations. InformationVisualization, 18(3):318–338, 2019. doi: 10.1177/1473871618757228.URL https://doi.org/10.1177/1473871618757228.[27] Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov,Jordan Boyd-Graber, and Philip Resnik. Is automated topic modelevaluation broken? the incoherence of coherence. Advances in NeuralInformation Processing Systems, 34:2018–2033, 2021.[28] Ting-Hao Kenneth Huang, Francis Ferraro, Nasrin Mostafazadeh, Is-han Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xi-aodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, DeviParikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. Vi-sual storytelling. In Proceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computational Linguistics:Human Language Technologies, pages 1233–1239, San Diego, Cali-fornia, June 2016. Association for Computational Linguistics. doi:10.18653/v1/N16-1147. URL https://aclanthology.org/N16-1147.40Bibliography[29] Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, KeisukeSakaguchi, Antoine Bosselut, and Yejin Choi. (comet-) atomic 2020: Onsymbolic and neural commonsense knowledge graphs. Proceedings ofthe AAAI Conference on Artificial Intelligence, 35(7):6384–6392, May2021. doi: 10.1609/aaai.v35i7.16792. URL https://ojs.aaai.org/index.php/AAAI/article/view/16792.[30] Mete Ismayilzada and Antoine Bosselut. kogito: A commonsenseknowledge inference toolkit. In Proceedings of the 17th Conferenceof the European Chapter of the Association for Computational Lin-guistics: System Demonstrations, pages 96–104, Dubrovnik, Croatia,May 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.eacl-demo.12.[31] Hyeju Jang, Emily Rempel, David Roth, Giuseppe Carenini,Naveed Zafar Janjua, et al. Tracking covid-19 discourse on twitter innorth america: Infodemiology study using topic modeling and aspect-based sentiment analysis. Journal of medical Internet research, 23(2):e25431, 2021.[32] H. W. Kuhn. The hungarian method for the assignment problem. NavalResearch Logistics Quarterly, 2(1-2):83–97, 1955. doi: https://doi.org/10.1002/nav.3800020109. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.3800020109.[33] Jey Han Lau, David Newman, and Timothy Baldwin. Machine readingtea leaves: Automatically evaluating topic coherence and topic modelquality. In Proceedings of the 14th Conference of the European Chapterof the Association for Computational Linguistics, pages 530–539, 2014.[34] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Ye-ung, and James Y Zou. Mind the gap: Understanding the modal-ity gap in multi-modal contrastive representation learning. InS. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, andA. Oh, editors, Advances in Neural Information Processing Sys-tems, volume 35, pages 17612–17625. Curran Associates, Inc., 2022.URL https://proceedings.neurips.cc/paper_files/paper/2022/file/702f4db7543a7432431df588d57bc7c9-Paper-Conference.pdf.[35] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Per-ona, Deva Ramanan, Piotr Dolla´r, and C Lawrence Zitnick. Microsoftcoco: Common objects in context. In Computer Vision–ECCV 2014:41Bibliography13th European Conference, Zurich, Switzerland, September 6-12, 2014,Proceedings, Part V 13, pages 740–755. Springer, 2014.[36] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and JeffDean. Distributed representations of words and phrases and their com-positionality. Advances in neural information processing systems, 26,2013.[37] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Represen-tation learning with contrastive predictive coding. arXiv preprintarXiv:1807.03748, 2018.[38] OpenAI. Gpt-4 technical report, 2023.[39] Jessica Peter, Steve Szigeti, Ana Jofre, and Sara Diamond. Topicks:Visualizing complex topic models for user comprehension. In 2015 IEEEConference on Visual Analytics Science and Technology (VAST), pages207–208. IEEE, 2015.[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh,Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, PamelaMishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learn-ing transferable visual models from natural language supervision. InMarina Meila and Tong Zhang, editors, Proceedings of the 38th Inter-national Conference on Machine Learning, volume 139 of Proceedings ofMachine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021.URL https://proceedings.mlr.press/v139/radford21a.html.[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh,Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, PamelaMishkin, Jack Clark, et al. Learning transferable visual models fromnatural language supervision. In International Conference on MachineLearning, pages 8748–8763. PMLR, 2021.[42] Dheeraj Rajagopal, Daniel Olsher, Erik Cambria, and Kenneth Kwok.Commonsense-based topic modeling. In Proceedings of the Second In-ternational Workshop on Issues of Sentiment Discovery and OpinionMining, WISDOM ’13, New York, NY, USA, 2013. Association forComputing Machinery. ISBN 9781450323321. doi: 10.1145/2502069.2502075. URL https://doi.org/10.1145/2502069.2502075.42Bibliography[43] Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and VeredShwartz. Vlc-bert: Visual question answering with contextualized com-monsense knowledge. 2021.[44] Sahithya Ravi, Aditya Chinchure, Leonid Sigal, Renjie Liao, and VeredShwartz. Vlc-bert: Visual question answering with contextualized com-monsense knowledge. In Proceedings of the IEEE/CVF Winter Confer-ence on Applications of Computer Vision (WACV), pages 1155–1165,January 2023.[45] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embed-dings using Siamese BERT-networks. In Proceedings of the 2019 Con-ference on Empirical Methods in Natural Language Processing and the9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China, November2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410.[46] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddingsusing siamese bert-networks. In Proceedings of the 2019 Conference onEmpirical Methods in Natural Language Processing and the 9th Inter-national Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, 2019.[47] Michael Ro¨der, Andreas Both, and Alexander Hinneburg. Exploring thespace of topic coherence measures. In Proceedings of the Eighth ACMInternational Conference on Web Search and Data Mining, WSDM ’15,page 399–408, New York, NY, USA, 2015. Association for ComputingMachinery. ISBN 9781450333177. doi: 10.1145/2684822.2685324. URLhttps://doi.org/10.1145/2684822.2685324.[48] Stephen Roller and Sabine Schulte Im Walde. A multimodal lda modelintegrating textual, cognitive and visual modalities. In Proceedings ofthe 2013 Conference on Empirical Methods in Natural Language Pro-cessing, pages 1146–1157, 2013.[49] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula,Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, andYejin Choi. Atomic: An atlas of machine commonsense for if-then rea-soning. In Proceedings of the AAAI conference on artificial intelligence,volume 33, pages 3027–3035, 2019.43Bibliography[50] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5:An open multilingual graph of general knowledge. Proceedings of theAAAI Conference on Artificial Intelligence, 31(1), Feb. 2017. doi:10.1609/aaai.v31i1.11164. URL https://ojs.aaai.org/index.php/AAAI/article/view/11164.[51] Dhanya Sridhar, Hal Daume´ III, and David Blei. Heterogeneous su-pervised topic models. Transactions of the Association for Computa-tional Linguistics, 10:732–745, 2022. doi: 10.1162/tacl a 00487. URLhttps://aclanthology.org/2022.tacl-1.42.[52] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky,and Marc Najork. Wit: Wikipedia-based image text dataset for multi-modal multilingual machine learning. In Proceedings of the 44th Inter-national ACM SIGIR Conference on Research and Development in In-formation Retrieval, SIGIR ’21, page 2443–2449, New York, NY, USA,2021. Association for Computing Machinery. ISBN 9781450380379. doi:10.1145/3404835.3463257. URL https://doi.org/10.1145/3404835.3463257.[53] Akash Srivastava and Charles Sutton. Autoencoding variational infer-ence for topic models. In 5th International Conference on LearningRepresentations, 2017.[54] Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tro-peano, and Antonio Candelieri. OCTIS: Comparing and optimizingtopic models is simple! In Proceedings of the 16th Conference of theEuropean Chapter of the Association for Computational Linguistics:System Demonstrations, pages 263–270, Online, April 2021. Associationfor Computational Linguistics. doi: 10.18653/v1/2021.eacl-demos.31.URL https://aclanthology.org/2021.eacl-demos.31.[55] Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tro-peano, and Antonio Candelieri. Octis: comparing and optimizing topicmodels is simple! In Proceedings of the 16th Conference of the Euro-pean Chapter of the Association for Computational Linguistics: SystemDemonstrations, pages 263–270, 2021.[56] Silvia Terragni, Elisabetta Fersini, and Enza Messina. Wordembedding-based topic similarity measures. In International Confer-ence on Applications of Natural Language to Information Systems,pages 33–45. Springer, 2021.44Bibliography[57] Jing Tian, Yu Huang, Zhi Guo, Xiang Qi, Ziyan Chen, and TingleiHuang. A multi-modal topic model for image annotation using textanalysis. IEEE Signal Processing Letters, 22(7):886–890, 2015. doi:10.1109/LSP.2014.2375341.[58] Lucia Vadicamo, Fabio Carrara, Andrea Cimino, Stefano Cresci, Fe-lice Dell’Orletta, Fabrizio Falchi, and Maurizio Tesconi. Cross-medialearning for image sentiment analysis in the wild. In 2017 IEEE Inter-national Conference on Computer Vision Workshops (ICCVW), pages308–317, 2017. doi: 10.1109/ICCVW.2017.45.[59] William Webber, Alistair Moffat, and Justin Zobel. A similaritymeasure for indefinite rankings. ACM Trans. Inf. Syst., 28(4), nov2010. ISSN 1046-8188. doi: 10.1145/1852102.1852106. URL https://doi.org/10.1145/1852102.1852106.[60] Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas,and Minlie Huang. Augmenting end-to-end dialogue systems with com-monsense knowledge. In Proceedings of the AAAI Conference on Arti-ficial Intelligence, volume 32, 2018.[61] En Yu, Zhuoling Li, and Shoudong Han. Towards discriminative repre-sentation: Multi-view trajectory contrastive learning for online multi-object tracking. In Proceedings of the IEEE/CVF Conference on Com-puter Vision and Pattern Recognition (CVPR), pages 8834–8843, June2022.[62] Lili Yu, Da´niel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettle-moyer, and Mike Lewis. Megabyte: Predicting million-byte sequenceswith multiscale transformers, 2023.[63] Huakui Zhang, Cai Yi, Bingshan Zhu, Haopeng Ren, and Qing Li.Multimodal topic modeling by exploring characteristics of short textsocial media. IEEE Transactions on Multimedia, pages 1–1, 2022. doi:10.1109/TMM.2022.3147064.[64] Huakui Zhang, Cai Yi, Bingshan Zhu, Haopeng Ren, and Qing Li.Multimodal topic modeling by exploring characteristics of short textsocial media. IEEE Transactions on Multimedia, 2022.[65] He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and WrayBuntine. Topic modelling meets deep neural networks: A survey. InZhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint45Conference on Artificial Intelligence, IJCAI-21, pages 4713–4720. In-ternational Joint Conferences on Artificial Intelligence Organization, 82021. Survey Track.[66] Bingshan Zhu, Yi Cai, and Haopeng Ren. Graph neural topic modelwith commonsense knowledge. Information Processing & Management,60(2):103215, 2023. ISSN 0306-4573. doi: https://doi.org/10.1016/j.ipm.2022.103215. URL https://www.sciencedirect.com/science/article/pii/S0306457322003168.[67] Elaine Zosa and Lidia Pivovarova. Multilingual and multimodal topicmodelling with pretrained embeddings. In Proceedings of the 29th In-ternational Conference on Computational Linguistics, pages 4037–4048,2022.46Appendix ATopics examplesFigure A.1 display two samples of topics generated respectively by M-CombinedTM,M-ZeroShotTM and M-Contrast from the MS COCO dataset.Model Topics descriptorsM-CTMsnow, covered, person, skis, ski, mountain, slope, snowy, hill, skiingM-ZTMsnow, person, skis, covered, mountain, ski, snowy, slope, hill, skiingM-Csnow, covered, skis, slope, hill, snowy, ski, mountain, skiing, snowboardM-CTMsky, boat, airplane, water, flying, plane, body, boats, airport, jetM-ZTMwater, boat, body, beach, ocean, river, boats, lake, sandy, shoreM-Cbeach, water, boat, body, ocean, boats, kite, river, kites, sandyTable A.1: Two samples of topics in M-CombinedTM (M-CTM),M-ZeroShotTM (M-ZTM) and M-Contrast (M-C).47Appendix BData preprocessingWe follow the training data preprocessing according to [4]. We removestopwords and restrict the vocabulary size to the top 2,000 most frequentterms to speed up computation. We remove English stopwords by usingthe NLTK library8. We also remove punctuation and digits as suggested byprior work [3].8https://www.nltk.org/48Appendix CModels and BaselinesC.1 LDAWe use the implementation of this model in Octis9 [55]. The parametersthat control their document-topic and word-topic distribution are estimatedduring training, as in prior work [3].C.2 CombinedTM and ZeroShotTMWe use their publicly available implementation with the default parame-ters10. We use Adam optimizer and apply 20% of dropout during training.C.3 M-ContrastWe develop this model by adapting M3L-Contrast [67]. Our code is basedon the author’s original implementation11. We use a batch size of 32. Weset the temperature τ to 0.07 and the contrastive weight ω to 100, as in theoriginal implementation.C.4 M-CombinedTM and M-ZeroShotTMWe develop M-CombinedTM by adapting CombinedTM [3]. Similarly, wedevelop M-ZeroShotTM by adapting ZeroShotTM [4]. Our code is based onthe author’s original implementations. As in the original code, our inferencenetwork structures consist of one fully connected hidden layer followed by asoftplus layer with 100 dimensions.9https://github.com/MIND-Lab/OCTIS10https://github.com/MilaNLProc/contextualized-topic-models11https://github.com/ezosa/M3L-topic-model49C.5. CLIPC.5 CLIPWe report the performance of the neural topic models using CLIP [41].We use the text and image encoder of clip-ViT-B-32, which is available inSBERT’s library12 [46]. This model has a 77-token limit. Documents aretruncated if they overpass that limit.12https://www.sbert.net/50Appendix DPerformance of M-ZeroShotTMacross different λ valuesTable D.1 reports the performance of M-ZeroShotTM for different values ofλ, which controls the relevance assigned to the image feature reconstructionloss (see Equation D.1).L = Eq[w⊤ log (softmax (βθ))]−KL (Q (θ | x) ∥P (θ)) + λ(1− cos(ximg, γθ)) (D.1)Our results indicate that for higher λ values, there is an improvementin the coherence and diversity associated with the images that represent atopic. However, the topics’ most relevant keywords are slightly less coherentand diverse.Coherence DiversityText Image Text ImageModels NPMI Cv WE IEC TD I-RBO IEPSλ = 1.0 .03 .51 .22 .56 .60 .98 .44λ = 1x BoW loss .02 .50 .21 .71 .50 .95 .40λ = 2x BoW loss .02 .49 .21 .73 .47 .94 .40λ = 4x BoW loss .01 .48 .21 .73 .46 .94 .39Table D.1: Topics’ quality for different values of λ on M-ZeroShotTM.Average results across multiple datasets over 4 number of topics (K =25, 50, 75, 100), where the results for each K are averaged over 5 randomseeds. We use CLIP as text encoder. We bold best scores.51Appendix ETopic coherence anddiversity metricsFor the automatic metrics: NPMI, Cv, WE, TD, and I-RBO, we use theirimplementation in Octis [55].52Appendix FUsing a Different TextEncoderWe also compare the performance of the neural topic models by using adifferent text encoder. We use a sentence-transformer model, all-mpnet-base-v2 (SBERT), which is available in the SBERT’s library. Table F.1shows the resulting performance of neural topic models.Coherence DiversityText Image Text ImageModels NPMI Cv WE IEC TD I-RBO IEPSCombinedTM .03 .51 .21 .56 .96ZeroShotTM .04 .52 .23 .58 .98M-Contrast .04 .51 .22 .66 .48 .94 .41M-CombinedTM 03 .50 .21 .54 .57 .97 .45M-ZeroShotTM 03 .51 .22 .56 .58 .98 .44Table F.1: Performance of neural topic models after using a different con-textualized text encoder (i.e., SBERT). Average results across datasets over4 number of topics (K = 25, 50, 75, 100), where the results for each K areaveraged over 5 random seeds. We bold best scores.53Appendix GComputing InfrastructureOur experiments were conducted utilizing an NVidia A100 GPU, 20 GBof memory, and 12 cores of an AMD Milan 7413. While prior work hasdemonstrated that neural topic models can also be executed on less advancedhardware, we opt to use this high-performance computing infrastructure toensure the efficient processing of our data.54Appendix HRuntimePrevious research has shown that vocabulary size significantly affects thecomputational time of VAE-based neural topic models [3]. Therefore, fol-lowing prior work, we limit the maximum number of terms for the Bag-Of-Word reconstructions to 2,000. After removing digits, punctuation, andstopwords, we select the 2,000 most frequent words in the corpus for thispurpose as is done in [3, 4, 67].To compare the computational efficiency of different models, we reportthe time taken in seconds to complete one epoch during training. Table H.1presents the time required for each neural topic model to complete one epoch,except LDA, where we report the average training time. Our findings showthat multimodal neural topic models take approximately 1.5 seconds longerto complete an epoch than the unimodal ZeroShotTM model. This can beattributed to the larger input sizes and the need to analyze two modalitiessimultaneously in the multimodal models. This difference in training timeis expected and reasonable, given the increased complexity of the models.Training time per epochLDA 12.49CombinedTM 7.38ZeroShotTM 7.27M-Contrast 8.70M-CombinedTM 8.62M-ZeroShotTM 8.33Table H.1: Time in seconds to complete one epoch. Average results acrossall datasets over 4 number of topics (K = 25, 50, 75, 100), where the resultsfor each K are averaged over 5 random seeds. We use CLIP as text encoder.We bold the lowest training time.55
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Carenini, Giuseppe
Neural multimodal topic modeling : a comprehensive evaluation
Text