Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Xing, Linzi
Doctor of Philosophy - PhD
University of British Columbia
Topic segmentation, as a fundamental NLP task, has been proposed and systematically studied since the 1980s and received increased attention in recent years due to the surge in big data. It aims to unveil the coarse-grained semantic structure of long unstructured documents by automatically dividing them into shorter, topically coherent segments.The coarse-grained structure provided by topic segmentation has been proven to not only enhance human reading efficiency but also play a vital role in other natural language understanding tasks, such as text summarization, question answering, and dialogue modeling. Before the neural era, early computational models for topic segmentation typically adhered to unsupervised paradigms with lexical cohesion directly derived from the input, yet their performance was notably limited. With the evolution of deep learning and enhanced computational capabilities, neural models have delivered significant progress in performance. Nevertheless, inadequate coherence modeling, in terms of both explicitness and reliability in these neural approaches, prevents them from emerging as more accurate and robust solutions for topic segmentation. Additionally, the growing prevalence of multi-modal data content across social media platforms has heightened the need for topic segmentation to traverse beyond mere text, extending into videos. Motivated by the challenges and needs mentioned above, in this thesis, we direct our efforts towards enhancing neural topic segmentation for two types of documents: text and video. To overcome the inadequate coherence modeling (explicitness and reliability) in neural topic segmenters for text, we propose a series of methods that either more explicitly model coherence patterns or leverage coherence signals encoded in related auxiliary tasks, notably discourse parsing and language modeling. For video content, we explore to extend neural topic segmenters, originally designed for text, into a multi-modal setting which is also robust to the often-encountered drastic variance in video length. A comprehensive set of experimental results indicates that our methods not only effectively enhance the overall performance of neural segmenters for text and video in intra-domain scenarios, but also broaden their applicability to data in other domains.
Versatile Neural Approaches to More Accurate and RobustTopic SegmentationbyLinzi XingB. Mgt, Soochow University, 2016M. Sc, University of Colorado Boulder, 2018A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES(Computer Science)The University of British Columbia(Vancouver)February 2024© Linzi Xing, 2024The following individuals certify that they have read, and recommend to the Facultyof Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Versatile Neural Approaches to More Accurate and Robust Topic Seg-mentationsubmitted by Linzi Xing in partial fulfillment of the requirements for the degree ofDoctor of Philosophy in Computer Science.Examining Committee:Giuseppe Carenini, Professor, Computer Science, UBCSupervisorRaymond Ng, Professor, Computer Science, UBCSupervisory Committee MemberDavid Poole, Professor, Computer Science, UBCUniversity ExaminerMarcin Morzycki, Associate Professor, Department of Linguistics, UBCUniversity ExaminerSujian Li, Associate Professor, Computer Science, UBCExternal ExaminerAdditional Supervisory Committee Members:Laks V.S. Lakshmanan, Professor, Computer Science, UBCSupervisory Committee MemberiiAbstractTopic segmentation, as a fundamental NLP task, has been proposed and systemati-cally studied since the 1980s and received increased attention in recent years due tothe surge in big data. It aims to unveil the coarse-grained semantic structure of longunstructured documents by automatically dividing them into shorter, topically co-herent segments. The coarse-grained structure provided by topic segmentation hasbeen proven to not only enhance human reading efficiency but also play a vital rolein other natural language understanding tasks, such as text summarization, questionanswering, and dialogue modeling.Before the neural era, early computational models for topic segmentation typi-cally adhered to unsupervised paradigms with lexical cohesion directly derived fromthe input, yet their performance was notably limited. With the evolution of deeplearning and enhanced computational capabilities, neural models have delivered sig-nificant progress in performance. Nevertheless, inadequate coherence modeling, interms of both explicitness and reliability in these neural approaches, prevents themfrom emerging as more accurate and robust solutions for topic segmentation. Ad-ditionally, the growing prevalence of multi-modal data content across social mediaplatforms has heightened the need for topic segmentation to traverse beyond meretext, extending into videos.Motivated by the challenges and needs mentioned above, in this thesis, we directour efforts towards enhancing neural topic segmentation for two types of documents:text and video. To overcome the inadequate coherence modeling (explicitness andreliability) in neural topic segmenters for text, we propose a series of methods thateither more explicitly model coherence patterns or leverage coherence signals en-coded in related auxiliary tasks, notably discourse parsing and language modeling.iiiFor video content, we explore to extend neural topic segmenters, originally designedfor text, into a multi-modal setting which is also robust to the often-encountereddrastic variance in video length. A comprehensive set of experimental results in-dicates that our methods not only effectively enhance the overall performance ofneural segmenters for text and video in intra-domain scenarios, but also broadentheir applicability to data in other domains.ivLay SummaryTopic segmentation allows computer programs to break down lengthy documentsinto smaller, distinct topic segments, making their inherent information easier tounderstand and locate. In the past, traditional programs achieved this by pinpointingrepeated keywords/phrases, which was not always accurate. Later on, the adventof deep learning has significantly improved this, yet a performance gap remains inmeeting the demands of real-world applications and products.To bridge this gap, this thesis introduces methods that enhance computer pro-grams’ ability to accurately identify topic shifts by more explicitly and reliably ana-lyzing coherence and informational flow in texts. Experiments show that our meth-ods improve the performance and generality of topic segmenters configured withdeep learning technologies.Additionally, to address the contemporary data landscape, especially in socialmedia platforms rich in video content, we have also expanded our methods to adeptlysegment videos into topic sections, making information in multi-modal formatsmore accessible and intelligible.vPrefaceAll the work presented in this dissertation was conducted at the Natural LanguageProcessing (NLP) research group, Faculty of Computer Science, at the Universityof British Columbia, Point Grey campus, and the Media Intelligence Lab at AdobeResearch.A version of Chapter 2 has been published at AACL-IJCNLP 2020 [188]. Forthis publication, I was the lead investigator, responsible for all major areas of con-cept formation, statement of research questions, data collection, implementation aswell as paper composition. Prof. Giuseppe Carenini was in the role of supervisoryauthor, actively engaging in concept development, discussions, and the compositionof papers. Prof. Brad Hackinen and Prof. Francesco Trebbi were partially responsi-ble for data collection.A version of Chapter 3 have been published at SIGDIAL 2021 [184]. For thispublication, I was the lead investigator, responsible for all major areas of conceptformation, statement of research questions, data collection, implementation as wellas paper composition. Prof. Giuseppe Carenini was in the role of supervisory au-thor, actively engaging in concept development, discussions, and the composition ofpapers.A version of Chapter 4 was published at AAAI-2022 [68], where I served asthe co-lead investigator and first co-author, collaboratively working with Dr. PatrickHuber on research question investigation, concept formation, experiment design andimplementation, as well as manuscript preparation. Prof. Giuseppe Carenini was inthe role of supervisory author, actively engaging in concept development, discus-sions, and the composition of papers.A version of Chapter 5 has been published at CODI workshop at COLING 2022vi[190]. For this publication, I was the lead investigator, responsible for all majorareas of concept formation, statement of research questions, data collection, imple-mentation as well as paper composition. Dr. Patrick Huber joined as a collabora-tor involving in the concept formation and statement of research questions. Prof.Giuseppe Carenini was in the role of supervisory author, actively engaging in con-cept development, discussions, and the composition of papers.A version of Chapter 6 is currently under review and I was the lead investigator,responsible for all major areas of concept formation, statement of research ques-tions, data collection, implementation as well as paper composition. Prof. GiuseppeCarenini was in the role of supervisory author, actively engaging in concept devel-opment, discussions, and the composition of papers.A version of Chapter 7 has been published at MMM 2024 [192]. For this pub-lication, I was the lead investigator, responsible for all major areas of concept for-mation, statement of research questions, data collection, implementation as well aspaper composition. Dr. Quan Tran and Dr. Fabian Caba, Dr. Franck Dernoncourt,Dr. David Seunghyun Yoon, Dr. Zhaowen Wang, Dr. Trung Bui, and Prof. GiuseppeCarenini were in the role of supervisory author, actively engaging in concept devel-opment, discussions, and the composition of papers.Portions of the abstract and introductory text in Chapter 1 are an aggregation ofthe papers aforementioned.viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Technical Approaches . . . . . . . . . . . . . . . . . . . . . . . . . 31.1.1 Concepts in Context: Cohesion and Coherence . . . . . . . 41.1.2 Topic Segmentation Approaches Prior to the Neural Era . . 61.1.3 Topic Segmentation Approaches Emerging in the Neural Era 71.2 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 101.3 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 13I Improving Topic Segmentation with Explicit Coherence Mod-eling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 Better Coherence Modeling for Neural Monologue Topic Segmentation 22viii2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.3 Neural Topic Segmentation Model . . . . . . . . . . . . . . . . . . 262.3.1 Problem Definition of This Work . . . . . . . . . . . . . . 262.3.2 Basic Model: Enhanced Hierarchical Attention Bi-LSTMNetwork (HAN) . . . . . . . . . . . . . . . . . . . . . . . 272.3.3 Auxiliary Task Learning . . . . . . . . . . . . . . . . . . . 292.3.4 Sentence-Level Restricted Self-Attention . . . . . . . . . . 312.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 322.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.4.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . 352.4.3 Evaluation Metric . . . . . . . . . . . . . . . . . . . . . . . 352.4.4 Neural Model Setup . . . . . . . . . . . . . . . . . . . . . 362.5 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 362.5.1 Intra-Domain Evaluation . . . . . . . . . . . . . . . . . . . 362.5.2 Domain Transfer Evaluation . . . . . . . . . . . . . . . . . 362.5.3 Multilingual Evaluation . . . . . . . . . . . . . . . . . . . 382.6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Better Coherence Modeling for Dialogue Topic Segmentation . . . . . 403.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.3.1 Training Data for Coherence Scoring . . . . . . . . . . . . 453.3.2 Utterance-Pair Coherence Scoring Model . . . . . . . . . . 483.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493.4.1 Data for Evaluation . . . . . . . . . . . . . . . . . . . . . . 493.4.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.4.3 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . 513.4.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . 513.4.5 Results and Analysis . . . . . . . . . . . . . . . . . . . . . 523.4.6 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . 543.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56ixII Improving Topic Segmentation with Auxiliary Tasks . . . . . 574 Predicting Above-Sentence Discourse Structure using Distant Super-vision from Topic Segmentation . . . . . . . . . . . . . . . . . . . . . 594.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634.3.1 Topic Segmentation Model . . . . . . . . . . . . . . . . . . 644.3.2 Tree Generation . . . . . . . . . . . . . . . . . . . . . . . . 654.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.4.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.4.3 Experimental Design . . . . . . . . . . . . . . . . . . . . . 694.4.4 Experiments and Results . . . . . . . . . . . . . . . . . . . 704.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745 Improving Topic Segmentation by Injecting Discourse Dependencies . 775.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 775.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 805.3 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 815.3.1 Basic Model: Hierarchical Topic Segmenter . . . . . . . . . 825.3.2 Discourse Graph Modeling . . . . . . . . . . . . . . . . . . 825.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 855.4.2 Experimental Design . . . . . . . . . . . . . . . . . . . . . 865.4.3 Intra-Domain Segment Inference . . . . . . . . . . . . . . . 885.4.4 Domain Transfer Segment Inference . . . . . . . . . . . . . 895.4.5 Efficiency Analysis . . . . . . . . . . . . . . . . . . . . . . 905.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 916 Language Models for Dialogue Topic Segmentation . . . . . . . . . . 936.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 936.2 The New Evaluation Benchmark . . . . . . . . . . . . . . . . . . . 946.2.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . 96x6.2.2 Dataset Statistics and Comparison . . . . . . . . . . . . . . 986.3 Language Models for Dialogue Topic Segmentation . . . . . . . . . 996.3.1 Language Representation Models (LRMs) . . . . . . . . . . 996.3.2 Generative Language Models (GLMs) . . . . . . . . . . . . 1016.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 1046.4.1 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . 1046.4.2 Language Models to Compare . . . . . . . . . . . . . . . . 1056.5 Results and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 1066.6 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110III Beyond Text: Topic Segmentation for Videos . . . . . . . . . 1127 Multi-Modal Video Topic Segmentation with Dual-Contrastive Do-main Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1147.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1147.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1177.3 Neural Video Topic Segmentation . . . . . . . . . . . . . . . . . . 1197.3.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . 1197.3.2 Model Architecture . . . . . . . . . . . . . . . . . . . . . . 1197.4 Long Video Adaptation . . . . . . . . . . . . . . . . . . . . . . . . 1227.4.1 Simple Sliding Window Inference . . . . . . . . . . . . . . 1227.4.2 Dual-Contrastive Adaptation . . . . . . . . . . . . . . . . . 1237.5 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 1257.5.1 Intra-Domain Dataset – YouTube . . . . . . . . . . . . . . 1257.5.2 Datasets for Cross-Domain Inference . . . . . . . . . . . . 1267.5.3 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . 1277.5.4 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . 1287.5.5 Implementation Details . . . . . . . . . . . . . . . . . . . . 1297.6 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 1297.7 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1328 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . 1348.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134xi8.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1398.2.1 More Modalities . . . . . . . . . . . . . . . . . . . . . . . 1398.2.2 Segment Topic Labeling . . . . . . . . . . . . . . . . . . . 1408.2.3 Hierarchical Topic Segmentation . . . . . . . . . . . . . . . 141Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143A Predicting Above-Sentence Discourse Structure using Distant Super-vision from Topic Segmentation . . . . . . . . . . . . . . . . . . . . . 175A.1 Complete Quantitative Results . . . . . . . . . . . . . . . . . . . . 175B Additional Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180xiiList of TablesTable 2.1 A Wikipedia sample article about City Marcus covering threetopics: T1, T2 and T3 . . . . . . . . . . . . . . . . . . . . . . . 23Table 2.2 Pk error score (lower the better) of different sentence encodingstrategies on three datasets (Section 2.4.1). To fit in the table, weshorten Att-BiLSTM to BiLSTM. Results in bold are the bestperformance across the comparisons. . . . . . . . . . . . . . . . 28Table 2.3 Statistics of all the English topic segmentation datasets used inour experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . 33Table 2.4 Statistics of the the WIKI-SECTION data in English(EN), Ger-man(DE) and Chinese(ZH). . . . . . . . . . . . . . . . . . . . . 35Table 2.5 Pk error, results in bold indicate the best performance across allcomparisons. Underlined results indicate the best performance inthe bottom section. † indicates the result is significantly different(p < 0.05) from basic model. . . . . . . . . . . . . . . . . . . . 37Table 2.6 Pk error score on the datasets in three languages (English, Ger-man and Chinese). . . . . . . . . . . . . . . . . . . . . . . . . 38Table 3.1 A dialogue topic segmentation example sampled from Doc2Dial[36]. This dialogue is segmented into three topical-coherent units(utterances in the same color are about the same topic). . . . . . 41Table 3.2 Statistics of the two conversational corpora used for coherencescoring training data generation. . . . . . . . . . . . . . . . . . . 47Table 3.3 Detailed topics covered in two conversational corpora used forcoherence scoring training data generation. . . . . . . . . . . . 47xiiiTable 3.4 Statistics of the three dialogue topic segmentation testing sets formodel evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 50Table 3.5 The experimental results on two English testing sets: DialSeg 711[195] and Doc2Dial [36]. ↑/↓ after the name of metrics indicatesif the higher/lower value means better performance. The bestperformances among the listed methods are in bold. . . . . . . . 53Table 3.6 The experimental results on the Chinese testing set proposed inXu et al. [195]. The best performances among the listed methodsare in bold. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Table 3.7 The average variance of depth scores on three testing sets. High-est values are in bold . . . . . . . . . . . . . . . . . . . . . . . 55Table 4.1 Average topic segmentation performance ± standard deviationover 5 runs, using the Pk error score [14] (lower values indicatebetter performance) on three evaluation corpora. Best perfor-mance per column in bold. . . . . . . . . . . . . . . . . . . . . 65Table 4.2 Statistics of the three training/testing datasets. . . . . . . . . . . 68Table 4.3 Evaluation results using the RST Parseval micro-average preci-sion measure on the RST-DT and GUM dataset. Subscripts in-dicate training dataset. TS = Topic Segmentation Model. * =Average performance over 10 runs. Best performance per sub-table underlined, best performance per column bold. . . . . . . . 71Table 4.4 RST Parseval micro-average precision on paragraph-to-documentlevel for genres in the GUM corpus. Sample(s) in “Vlog” &“Conversation” only contain a single paragraph and are thereforeomitted. RB=Right-Branching, 2S=Two-Stage parser, TS=TopicSegmentation model. . . . . . . . . . . . . . . . . . . . . . . . 73Table 5.1 Key dataset statistics of the discourse treebanks used for retrain-ing the Sent-First discourse parser [205]. . . . . . . . . . . . . . 83Table 5.2 Statistics of the datasets used in intra-domain experiments. . . . 85Table 5.3 Statistics of the datasets used in domain transfer experiments. . . 86xivTable 5.4 Pk (↓) error score on three corpora for intra-domain experiment.Results in bold and underlined indicates the best and second bestperformance across all comparisons. The row in purple is theresults achieved by our proposal. The column in green is the re-sults for RSTDT paragraph break prediction with gold discoursestructures integrated. . . . . . . . . . . . . . . . . . . . . . . . . 89Table 5.5 Pk (↓) error score on four test corpora for domain transfer ex-periment. Results in bold and underlined indicates the best andsecond best performance across all comparisons. The row high-lighted in purple is the results achieved by our proposal. . . . . . 90Table 5.6 The efficiency comparison between our proposal and the methodproposed in Xing et al. [188] on the Wiki-Section corpus. Thesetwo models share the same basic segmentation framework. T-Speed refers the training speed as number of batches processedper second during training stage. I-Speed refers the inferencespeed as number of documents processed per second during in-ference stage. . . . . . . . . . . . . . . . . . . . . . . . . . . . 91Table 6.1 Statistics of dialogue datasets in our evaluation benchmark fordialogue topic segmentation. The sub-table on top contains thedatasets proposed with the annotation of topic segment bound-aries, and sub-table on bottom contains the datasets originallyproposed for other dialogue understanding tasks and re-purposedwith strategies illstrated in Figure 6.1. #Dialogues here showshow many dialogues are contained in the test / dev set, following90%-10% data split. . . . . . . . . . . . . . . . . . . . . . . . . 98Table 6.2 ChatGPT’s prompt templates we applied for the end-to-end prompt-ing and pair-wise prompting approach respectively. Text in redwas only added for one-shot setting. . . . . . . . . . . . . . . . 103Table 6.3 Survey of pre-trained language models included in this study.S2S: Seq2Seq, Tr.: Transformer. . . . . . . . . . . . . . . . . . 105xvTable 6.4 Results on four non-meeting datasets included in our evaluationbenchmark (see §6.2). The solid horizontal line divides the tableinto an upper section for LRMs and a lower section for GLMs.Further, dashed horizontal lines separate models in the upper sec-tion into models w/ or w/o fine-tune on dialogue corpora, andmodels in the lower section based on their parameter size, eithermoderate or large (i.e., ChatGPT). Bolded and underlined resultsindicate the best and the second best performance across all com-parisons. Results highlighted in grey indicate best performancewithin the upper/lower section. #Seg. is the average segmentfrequency of model’s hypothesis. . . . . . . . . . . . . . . . . . 107Table 6.5 Experimental results on the three meeting datasets included inour evaluation benchmark introduced in §6.2. . . . . . . . . . . 109Table 7.1 Statistics of the three corpora used for training and testing in Sec-tion 7.5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125Table 7.2 Results on YouTube for intra-domain evaluation. Bolded andunderlined results indicate the best performance across all com-parisons and within their own sub-section, respectively. * indi-cates the marked approach is in supervised setting. . . . . . . . . 130Table 7.3 Results on BBC and Behance corpora for cross-domain evalu-ation. Bolded and underlined results indicate the best perfor-mance across all comparisons and within their own section. † in-dicates results applied contrastive domain adaptation which aresignificantly different (p < 0.05) from the row Ours (Window) inthe table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131Table A.1 Evaluation results using the RST Parseval micro-average preci-sion measure on the RST-DT dataset. Subscripts indicate trainingdataset. TS=Topic Segmentation Model. Disc-Seg=DiscourseSegmentation Model by Wang et al. [172]. ×=Not feasible com-bination. *=Average performance over 10 runs. Best perfor-mance per sub-table underlined, best performance per columnbold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176xviTable A.2 Evaluation results using the RST Parseval micro-average preci-sion measure on the GUM dataset. Subscripts indicate trainingdataset. TS=Topic Segmentation Model. Disc-Seg=DiscourseSegmentation Model by Wang et al. [172]. ×=Not feasible com-bination. *=Average performance over 10 runs. Best perfor-mance per sub-table underlined, best performance per columnbold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177Table A.3 Evaluation results using the original Parseval micro-average pre-cision measure on the RST-DT dataset. Subscripts indicate train-ing dataset. TS=Topic Segmentation Model. Disc-Seg=DiscourseSegmentation Model by Wang et al. [172]. ×=Not feasible com-bination. *=Average performance over 10 runs. Best perfor-mance per sub-table underlined, best performance per columnbold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178Table A.4 Evaluation results using the original Parseval micro-average pre-cision measure on the GUM dataset. Subscripts indicate trainingdataset. TS=Topic Segmentation Model. Disc-Seg=DiscourseSegmentation Model by Wang et al. [172]. ×=Not feasible com-bination. *=Average performance over 10 runs. Best perfor-mance per sub-table underlined, best performance per columnbold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179xviiList of FiguresFigure 1.1 An example sampled from Wikipedia about City Marcus. Onthe left, this article is flat without any structural information.After going through the process of topic segmentation and seg-ment topic labeling, this article is segmented into three topical-coherent textual sections with section labels and dotted lines in-dicate segment boundaries. Putting segment topic labeling inthe shaded box indicates that it is outside the scope of this thesis. 2Figure 1.2 The overall process of segment boundary inference using Text-Tiling algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 7Figure 1.3 Parallel relatedness between the text type and method paradigmsof neural topic segmentation models. . . . . . . . . . . . . . . . 9Figure 1.4 An example showing two hypotheses (hypothesis-1 and hypothesis-2) with the identical F-score of 0, despite hypothesis-2 havingboundaries that more closely match the reference. Black boxesindicate the hypothesized or reference label about whether asentence is the end of a segment or not. Dashed boxes indicatemissed boundary predictions (false negatives). . . . . . . . . . . 11Figure 1.5 Two hypotheses with an identical F-score of 0, hypothesis-2 isassessed closer to the reference using Pk. . . . . . . . . . . . . 12Figure 1.6 Two hypotheses with an identical Pk of 0.22, hypothesis-2 isassessed closer to the reference using WD and hypothesis-1 isoverly segmented. . . . . . . . . . . . . . . . . . . . . . . . . . 12Figure 1.7 The overview of this thesis with corresponding concepts, re-search questions and manuscripts. . . . . . . . . . . . . . . . . 14xviiiFigure 2.1 The architecture of our basic model. sei is the produced sentenceembedding for sentence Si. . . . . . . . . . . . . . . . . . . . . 27Figure 2.2 The overview of our proposed topic segmentation model withcontext modeling components: restricted self-attention, auxil-iary task module. . . . . . . . . . . . . . . . . . . . . . . . . . 29Figure 3.1 The overview of our proposed dialogue topic segmentation pro-cedure. (a) Fine-tuning the NSP BERT on the training data ofutterance-pair coherence scoring generated from the source di-alogue corpus C . (2) Leveraging the fine-tuned BERT as thecoherence scoring model to predict coherence scores for all theconsecutive utterance pairs in a testing dialogue. TextTiling al-gorithm is further utilized to infer segment boundaries. . . . . . 48Figure 3.2 Behaviors of four TextTiling-based segmenters on an exampledialogue selected from Doc2Dial [36]. The horizonal axis is theindex of intervals in a session, and the vertical axis is the valueof depth score (higher value means more topical unrelated). Thereference and prediction of topic boundaries are marked by blueand red vertical lines respectively. The overlaps of reference andprediction are marked by purple lines. . . . . . . . . . . . . . . 54Figure 4.1 Textual levels in a well-structured document and their influenceon downstream tasks and available gold-label data. . . . . . . . 60Figure 4.2 Overview of our approach to generate high-level discourse struc-tures on sentence-to-document level (S-D) from distant topicsegmentation supervision. . . . . . . . . . . . . . . . . . . . . 64Figure 4.3 Greedy top-down tree generation approach. . . . . . . . . . . . 66Figure 4.4 Problematic CKY aggregation without the use of discounting.All trees receive the same final score. . . . . . . . . . . . . . . 67Figure 4.5 Positive example (wsj 1346) with 100% structural overlap be-tween prediction and RST-DT gold-label annotation. Gold para-graphs pn are indicated by solid lines. Leaves are sentences. . . 75xixFigure 4.6 Random example (wsj 1380) with 80.77% structural overlapbetween prediction (top) and RST-DT gold-label annotation (bot-tom). Gold paragraphs pn are indicated by solid lines. Leavesare sentences. . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Figure 4.7 Negative example (wsj 1365) with 55.56% structural overlapbetween prediction (top) and RST-DT gold-label annotation (bot-tom). Gold paragraphs pn are indicated by solid lines. Leavesare sentences. . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Figure 4.8 Positive example (GUM bio jesperser) with 76.92% structuraloverlap between prediction (left) and GUM gold-label annota-tion (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Figure 4.9 Random example (GUM voyage oakland) with 70.83% struc-tural overlap between prediction (left) and GUM gold-label an-notation (right). . . . . . . . . . . . . . . . . . . . . . . . . . . 76Figure 4.10 Negative example (GUM bio dvorak) with 57.14% structuraloverlap between prediction (left) and GUM gold-label annota-tion (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76Figure 5.1 An example article about Cholinergic Urticaria (CU) sampledfrom the en disease portion of Wiki-Section dataset [3]. Left:discourse dependency structure predicted by the Sent-First dis-course parser [205]. . . . . . . . . . . . . . . . . . . . . . . . . 78Figure 5.2 The overall architecture of our discourse-infused topic segmen-tation model with predicted discourse dependencies integratedby Graph Attention Network (GAT). With a document in theform of a sequence of sentences as input, the hierarchical topicsegmenter encodes sentences into contextualized representations,while the discourse graph modeling component constructs a dis-course dependency graph and performs graph modeling on it.Representations from both components are concatenated for thefinal topic segment boundary inference. . . . . . . . . . . . . . 81xxFigure 6.1 Overview of the process for deriving silver standard dialoguetopic segmentation for three conversation corpora in our evalu-ation benchmark, which were originally proposed for other dia-logue understanding tasks. . . . . . . . . . . . . . . . . . . . . 95Figure 6.2 The loss-based topic segmentation workflow for generative lan-guage models. . . . . . . . . . . . . . . . . . . . . . . . . . . 100Figure 6.3 Overview of the iterative prompting mechanism addressing thediscrepancy of utterance count before and after ChatGPT. . . . . 102Figure 6.4 The token length statistics of the three meeting corpora includedin our benchmark. The red dashed line represents the input to-ken limit of ChatGPT (gpt-3.5-turbo as the backend model).104Figure 7.1 An example livestream video about making a choking victimposter, sampled from the creative livestream platform Behance.The left side of the figure illustrates the video’s timeline togetherwith transcript after topic segmentation, consisting of 8 topical-coherent segments in different colors. Words indicating a seg-ment’s topic are highlighed in the transcript. The right side ofthe figure presents a video frame randomly sampled from eachsegment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115Figure 7.2 Left: The overall architecture of our proposed multi-modal videotopic segmentation model, with four sub-modules (in §7.3.2)coded in different colors. Right: The detailed illustration ofcross-modal attention. . . . . . . . . . . . . . . . . . . . . . . 120Figure 7.3 An illustration of updating cross-modal attention module withdual-contrastive adaptation. . . . . . . . . . . . . . . . . . . . 123Figure 7.4 Histogram of YouTube video corpus category distribution (y-axis: number of videos). . . . . . . . . . . . . . . . . . . . . . 126Figure 7.5 Comparison of segment boundary predictions by different set-tings of our proposal (Ours in Table 7.3) on the Behance exam-ple in Figure 7.1. Our proposed long video adaption strategy(Window + CL-Dual) recalls the most ground truth boundaries. 132xxiAcknowledgementsEmbarking on this Ph.D journey was akin to setting sail in uncharted waters, and likeany intrepid voyager, I was not alone. My family served as the anchor, groundingme with love and support. My supervisor and colleagues were the guiding stars, pro-viding direction and wisdom. And my friends, the steady wind in my sails, offeredcamaraderie and encouragement through every wave and gust.First and foremost, to my research supervisor Prof. Giuseppe Carenini, my deep-est gratitude extends beyond the boundaries of this thesis. He meticulously guidedme through each research project, illuminated the path of discovering valuable ideas,and taught me how to sculpt them into coherent and comprehensive research pieces.His sincerity and candidness have been instrumental, especially in the initial phasesof my Ph.D journey. I still remember our conversations during my first year, wherehe, with utmost care and honesty, highlighted the importance of academic writingand its indispensable role in a successful Ph.D. His words pointed out the directionfor my efforts and provided an inexhaustible source of motivation for my relentlesspursuit. During times when I felt lost and feared lagging behind, his understanding,care, and generous sharing of his own experiences anchored me, reassuring that Iwas on a healthy trajectory. His guidance transcended mere academic mentorship –it stands as a guiding light that has, and will continue to, enlighten my journey inacademia and life.My sincere thanks also go to other members in my supervisory committee, Prof.Raymond Ng and Laks V.S. Lakshmanan. Their extensive knowledge, dedicationto academic excellence, and probing questions challenged me to think more deeplyand critically. Their feedback not only elevated the quality of my thesis but also ex-panded my horizons and honed my analytical abilities. Their consistent support andxxiibelief in my potential played a pivotal role in shaping this thesis. I am profoundlygrateful for the time they invested in reviewing my work and for their invaluablecontributions to my academic journey. Additionally, I am also profoundly thankfulto other members in my examining committee, Prof. David Poole, Marcin Morzy-cki, and Sujian Li. Their expertise and thoughtful feedback were also crucial inrefining my arguments and enhancing the clarity of my thesis.Research is often a collaborative effort, and its success hinges on the combinedstrengths and contributions of many. It is also very true during my Ph.D journey.I’ve been lucky to work alongside some truly dedicated collaborators and wouldlike to express my deep appreciation to Patrick Huber, Wen Xiao, Raymond Li,Brad Hackinen, Quan Tran, and Fabian Caba. From brainstorming ideas to offeringvaluable feedback, their collective efforts played a significant role in the completionof research work throughout my Ph.D journey. Each one brought unique insightsand knowledge, making our collaborated work stronger and more comprehensive.Furthermore, I’m grateful to the entire UBC NLP group for creating a supportive andcollaborative environment, and a big thank you to all the reviewers who voluntarilyassessed my publications, broadening my views and improving my research withtheir invaluable feedback.Within these five years, my family is always the pillars of my strength and thefoundation of my journey. My father, Jianzhong Xing, and my mother, Lin Lin, havebeen unwavering anchors in my life. Even in moments of doubt, when I questionedmy path, they remained their faith in me and were consistently reminding me thatthey would always be my most reliable backup, provided the comfort, courage andlove I needed to forge ahead. A special mention must be made for my grandmother,Wei Tian. Since my middle school years, every dish she prepared and each dayshe spent by my side has shaped the individual I am today. She has always been sosupportive. When I shared my aspirations to travel overseas in pursuit of my dreams,I could always discern a hit of reluctance in her eyes. Yet, she still consistently stoodby every decision I made without falter.Last but not least, I owe a depth of gratitude to all my friends. No matter atwhich phase of my Ph.D journey they stepped in, their willingness to listen to myconcerns, their patience in accompanying me through difficulties, and their generousassistance in helping me achieve career goals have been nothing short of a treasure.xxiiiThese acts of kindness and support are priceless fortunes that I will carry and benefitfrom throughout my entire life. In the midst of storms and the calm alike, theirpresence and unwavering support have been invaluable, ensuring I never felt aloneon this voyage.xxivChapter 1IntroductionThe explosion of big data has led to an overwhelming volume of textual data, fromsocial media posts to academic articles, and from casual conversations to extensivemeeting transcripts. Within such a gigantic textual data landscape, a considerableportion of documents, articles, or dialogue sessions exhibit a rather unstructured andflat nature, often lacking explicit cues or indicators to reflect their internal structureor thematic focus. This not only poses a challenge to natural language understand-ing (NLU) for both humans and machines, but also gives opportunity to a line ofresearch in the field of Natural Language Processing (NLP) on the task of Topic Seg-mentation, which intends to provide computational models that can predict topic-relevant structural information in these otherwise unstructured and flat texts.Typically, the structure of a document can span from the coarse-grained level,such as section and paragraph divisions [6] to a more fine-grained level, which in-cludes complete discourse trees with rhetorical relations between adjacent textualunits [77]. Within this context, topic segmentation specializes in the coarse-grainedstructural aspects, aiming to break down documents covering multiple topics intoshorter, topically-coherent segments (sections). A topical-coherent textual segment(section) is expected to include a set of consecutive sentences which are logicallyconsistent and join together to serve a common topic [155, Chapter 1]. It is worthyto underscore that the definition of a topic can vary according to text volumes andgenres. As for a corpus consisting of numerous documents, a topic is more alignedwith a semantic field, which is a set of words spanning a particular semantic do-1Figure 1.1: An example sampled from Wikipedia about City Marcus. On theleft, this article is flat without any structural information. After goingthrough the process of topic segmentation and segment topic labeling,this article is segmented into three topical-coherent textual sections withsection labels and dotted lines indicate segment boundaries. Putting seg-ment topic labeling in the shaded box indicates that it is outside the scopeof this thesis.main and evincing structured interrelations [81, Chapter 6][185]. In contrast, fora single document, a topic more commonly denotes an issue which is consistentlyargued/discussed, or a distinct facet of a subject matter introduced in the document.For example, topics identified in a news broadcast transcript may correspond to indi-vidual news stories, whereas in a court transcript, the topic shift could be recognizedat where different arguments are presented or pieces of evidence are discussed [155,Chapter 1]. Given that multi-document topic segmentation [121] lies beyond thepurview of this thesis, the term “topic” will henceforth refer more closely to theabove meaning within the context of the single-document case. Moreover, topicsegmentation is often performed as a preceding step for segment topic labeling, an-other essential task that assigns interpretable labels to describe the theme of eachsegment previously identified in the phase of topic segmentation (as illustrated inFigure 1.1). Together, these two tasks can synergistically capture more comprehen-sive coarse-grained document structural information, offering a holistic view of the2document’s thematic organization. Even though topic segmentation and segmenttopic labeling are closely related, this dissertation primarily focuses on topic seg-mentation to maintain a concentrated scope and depth of investigation into this task.We earmark segment topic labeling for future research (see §8.2.2).In a variety of scenarios where the structural knowledge from topic segmentationis beneficial, perhaps the most practical advantage is its ability to enhance humanreading efficiency. With topic labels produced by segment topic labeling, readers canquickly find specific information without wading through lengthy, uninterrupted text[40, 121]. Going further than this, the coarse-grained structural output of topic seg-mentation has also demonstrated considerable value in bolstering the performanceof other critical downstream NLU tasks (e.g., document summarization, questionanswering, dialogue modeling). In the context of document summarization, empir-ical evidence suggests that the integration of section segmentation can help neuralsummarizers better understand the input document through jointly considering bothlocal and global information, and thus significantly boosts their performances par-ticularly on long documents [181]. Similarly, for the task of question answering,retrieving answers from topically consistent segments, which are inherently moreconcise and coherent than the entire document, proves more effective than retriev-ing answers from the whole document [31, 126]. Furthermore, the task of dialoguemodeling can also greatly benefit from topic segmentation, as topic segments minedfrom the dialogue history can serve as better basic units used for response selection,compared to individual dialogue turns [195, 200].1.1 Technical ApproachesGiven that topic segmentation has been a NLP research area for many years, a di-verse array of approaches have been proposed to addressed this task. However,before diving into the introduction of these approaches, it is essential to providethe foundational understanding of two linguistic concepts inherently related to topicsegmentation: cohesion and coherence.31.1.1 Concepts in Context: Cohesion and CoherenceCohesion is a central phenomenon of text “continuity” and can be detected if sen-tences and clauses depend on one another by a variety of linguistic means (i.e.,lexical and grammatical devices) that can be identified at the text surface [156].Consider the following example picked from Stede [156, Chapter 2]:(1) Jim went over to the garage. (2) He opened the door slowly.In the second sentence, the pronoun “he” refers back to “Jim” from the first sentence,implying the presence of cohesion. Based on the definition of cohesion, such signalcan act as the indicator of topic continuity and an “off” in cohesion may indicate achange in topic. Nonetheless, since cohesion does not require deep understandingof the text, it might not consistently align with the semantic/topical relationshipsbetween sentences. Consider another example picked from Stede [156, Chapter 4]:(1) John took a train from Paris to Istanbul. (2) He likes spinach.Even though the second sentence uses the pronoun “he” to refer back to “John” inthe first sentence, indicating a cohesive device of reference, it’s obvious that thesetwo sentences do not share the same topic.Coherence, on the other hand, is typically characterized as a property that out-lines how individual sentences are connected to foster a seamless logical progres-sion within a discourse. Therefore, sentences in a coherent discourse are intricatelylinked to collaboratively convey a unified theme or topic [140]. Different from co-hesion, the concept of coherence, places greater emphasis on the semantic and prag-matic facets of text, focusing on its underlying meaning [156, Chapter 4]. Considerthe following example presented in Wang and Guo [170]:(1) Cognitive Pragmatics Seminar: Wednesday 26th, May 2.00 p.m(2) Place: The Meeting Hall in the Library(3) Deirdre Wilson: Department of Phonetics and Linguistics at UCLAt first glance, the three sentences above may seem fragmented and lack explicit co-hesive devices in between. However, with common knowledge, it becomes apparentthat they are semantically relevant and together convey an announcement for a sem-inar on cognitive pragmatics. Thus, they can be regarded as a coherent discourse as4a whole. In the task of topic segmentation, a topical-coherent textual segment (sec-tion) is also defined as a set of consecutive sentences that are logically consistentand collaboratively address a common topic [155, Chapter 1]. Given the significantconceptual overlap between coherence and the characteristics of segments resultingfrom topic segmentation, we can confidently assume that:Coherence can be a more suitable metric than cohesion for detecting topictransitions in text, and thereby serve as a fundamental pillar for topic segmen-tation methodologies.The research presented in this dissertation is primarily driven by this assumption,addressing the shortcomings of prior technical approaches in their ineffective co-herence modeling to achieve more accurate and robust topic segmentation.In previous literature, approaches to model text coherence can be mainly de-lineated into two categories: theory-driven and data-driven. Prior to the neural era,coherence modeling was more theory-driven, relying on traditional NLP approachesrooted in assumptions about specific linguistic features (e.g., lexical chains and en-tity transitions) to model local and global coherence within text. Early approaches,as proposed by Morris and Hirst [120], utilized lexical chains — sequences of re-lated words that co-occur together frequently — to assess coherence within texts.These approaches are based on the assumption that the presence of a long and stronglexical chain, spanning a significant portion of a text, can distinguish coherent textsfrom incoherent ones. Later on, Barzilay and Lapata [11, 12] proposed to use theentity-grid to track the role transitions of selected entities throughout a text. Thecoherence model they proposed is based on the phenomenon that specific entitytransition patterns can be used as features to distinguish between coherent and inco-herent texts in certain domains. With the rapid progress in deep learning, the preva-lence of data-driven approaches, often leveraging neural models, had emerged forcoherence modeling. Compared to theory-driven approaches, these methods typ-ically require a substantial amount of training instances equipped with coherencelabels obtained either through synthetic permutation [99, 159] or human annotation[29, 92]. Given that data-driven neural coherence models generally outperform pre-vious theory-driven approaches, the coherence modeling strategies discussed in thisdissertation are data-driven and implemented within neural frameworks.51.1.2 Topic Segmentation Approaches Prior to the Neural EraDue to the scarcity of training data, early approaches proposed before the neural eracommonly embraced unsupervised paradigms with a primary emphasis on trackingshifts in lexical distribution, given the foundational premise that changes in vocabu-lary use can signal underlying topic transition. In other words, they largely dependedon text cohesion because of its accessibility in the low-resource scenario. Whilethese early approaches employed different lexical cohesion metrics (e.g., lexicaloverlap [33, 56] and lexical chain [41]) to track vocabulary changes throughout therunning text, they largely shared a common unsupervised segmentation frameworkknown as TextTiling [56]. Given the overall TextTiling framework demonstratedin Figure 1.2, a document in the form of a sequence of sentences {s1,s2, ...,sk}can be broken into k− 1 consecutive sentence pairs. Then a selected lexical cohe-sion metric is applied to all pairs and we get a sequence of lexical cohesion scores{c1,c2, ...,ck−1}, where ci theoretically indicates how cohesive two sentences in theith pair are. Next, instead of directly using the lexical cohesion scores to infer seg-ment boundaries, a sequence of “depth scores” {d p1,d p2, ...,d pk−1} is calculated tomeasure how sharp a lexical valley is by looking at the highest cohesion scores hl(i)and hr(i) on the left and right of interval i: d pi =hl(i)+hr(i)−2ci2 . A higher depth scoreindicates a more noticeable disruption in cohesion, which further suggests that thecorresponding pair of sentences is likely to be in close proximity to a topic segmentboundary. The threshold τ to finally determine segment boundaries is empiricallycomputed from the mean µ and standard deviation σ of depth scores: τ = µ − σ2 .A pair of sentences with the depth score over τ will be identified to have a segmentboundary in between. While features relying on lexical cohesion are easy to ob-tain and thus ensure broader applicability of approaches built on them, they providemerely a superficial representation of topical relatedness within the input document.As a result, the collection of approaches that depend solely on these features fre-quently demonstrates limited effectiveness in practical use.While some other research sought to acquire more semantic-related informationand incorporate it into unsupervised topic segmentation frameworks, the suboptimalreliability of such existing semantic modeling techniques (e.g., LDA for sentences’topical distribution [32, 141] and GloVe word embeddings [46]) hindered these ap-proaches from gaining widespread recognition.6Figure 1.2: The overall process of segment boundary inference using TextTil-ing algorithm.1.1.3 Topic Segmentation Approaches Emerging in the Neural EraMore recently, advancements in deep learning, coupled with improved computa-tional capacities, have paved the way for approaches that leverage neural architec-tures, given their superior accuracy and efficiency in many NLP tasks. Generally,neural approaches for topic segmentation can be categorized based upon two distincttext types - Monologue and Dialogue.Neural Topic Segmentation for MonologueWith the availability of large-scale corpora sourced from Wikipedia, where sectionmarks serve as the ground-truth segment boundaries, most research on monologuetopic segmentation follow the similar strategy to simply interpret text segmentationas a sequence labeling problem, determining whether a sentence represents a seg-ment boundary or not. On top of that, fully supervised neural models were builtto make predictions in a straightforward manner. As one of the earliest endeavorsin this line of research, Wang et al. [166] proposed a simple BiLSTM model to la-bel if a sentence is a segment boundary or not. They demonstrated that along withengineered features based on cue phrases (eg., ‘first of all’, ‘second’), their modelcan achieve marginally better performance than early unsupervised methods. Later,7Badjatiya et al. [7] proposed an attention-based BiLSTM model to classify whethera sentence was a segment boundary or not, by considering the context around it.More recently, hierarchical neural architectures (sentence encoding + sentence con-textualization) have taken the dominant place in supervised topic segmentation formonologue. Koshorek et al. [88] first attempted to introduce a hierarchical neuralsequence labeling model specifically designed for topic segmentation and demon-strated its superiority over selected supervised and unsupervised baselines. Follow-ing this progress, Glavas and Somasundaran [45] and Lukasik et al. [113] aimedto leverage even more advanced hierarchical transformers for topic segmentation,achieving similar success in segmenting text at both the topic level and more finer-grained discourse level. Around the same time, Barrow et al. [10] and Lo et al. [109]extended such hierarchical neural frameworks into the multi-task setting by incorpo-rating segment topic labeling. Through considering two tasks jointly, the accuracyof monologue topic segmentation has been further improved.Despite the remarkable success achieved by such neural solutions, it is worthnoting that the lack of interpretability in the fully supervised training process canraise concerns. These models might not always learn to cluster sentences into top-ically coherent segments based on how coherent they are together. Instead, theymight rely on overly simplistic patterns unique to the specific domain of trainingdata, compromising both robustness and transferability.Neural Topic Segmentation for DialogueIn comparison to monologue documents, dialogues often exhibit greater fragmen-tation and are replete with informal expressions. The discourse relations betweenutterances in dialogues also differ significantly from those in monologue text. Forinstance, dialogue discourse, as encoded in the Segmented Discourse Representa-tion Theory (SDRT) [5], may allow non-projective relations among utterances [97],while monologue discourse typically adheres to the constituent tree structure out-lined in the Rhetorical Structure Theory (RST) [115]. Consequently, directly apply-ing aforementioned neural approaches originally designed for monologues to dia-logues can introduce undesirable noise, resulting in suboptimal performance [44].Owing to the scarcity of labeled training data for dialogue topic segmentation, un-supervised neural approaches, particularly those built upon well-established meth-8Figure 1.3: Parallel relatedness between the text type and method paradigmsof neural topic segmentation models.ods like TextTiling (as elaborated in §1.1.2), remain the dominant choices even inthe neural age. For instance, augmenting TextTiling with word embeddings gener-ated by neural networks (e.g., Word2Vec [119]) can provide a more nuanced se-mantic representation compared to traditional bag-of-words style features [151].Later on, Xu et al. [195] substituted these word embeddings with utterance en-codings produced by BERT [30], as such pretrained neural language models ex-cel at capturing coherence relations at the utterance level. Additionally, to mitigateover-fragmentation in topic segmentation, they modified the TextTiling alogrithm togreedily check all intervals between consecutive utterances, aiming to determine oneboundary at a time where the two resulted segments differ the most. However, thismodification might introduce additional hyper-parameter tuning. Meanwhile, whilesome semi-supervised neural settings have been explored through casting dialoguetopic segmentation as a topic tracking problem [84, 158], with the predefined conver-sation topics as part of the supervisory signal, their reliability on out-of-distributiondata is still rather poor.Based on the literature presented in this section, neural-based computationalmodels for topic segmentation can be categorized along two dimensions: (1) themethod paradigm (supervised or unsupervised) and (2) the text type (monologue9or dialogue). Although these two dimensions appear to be conceptually distinct, inpractice they often converge, primarily due to the need and availability of large-scalelabeled data (as shown in Figure 1.3). Such convergence can be summarized as thefollowing scheme:Monologue topic segmentation is commonly approached as a supervisedlearning task, whereas dialogue topic segmentation is often addressed as anunsupervised learning task.This dissertation also employs such a scheme to structure our exploration of neuralapproaches, aiming to improve the accuracy and robustness of topic segmentation inboth monologues and dialogues.1.2 Evaluation MetricsClassification-based MetricAs topic segmentation can be intuitively formulated as a sequence labeling task ina way of predicting whether each sentence is the end of a segment or not (§1.1.3),standard classification-based evaluation metrics such as F-score can be straight-forwardly applied for evaluation. As illustrated in Figure 1.4, by counting and cate-gorizing matches and mismatches between hypothesized and reference boundaries,we can easily calculate F-score typically defined as the standard armonic mean ofprecision and recall, with higher scores indicating better performance. However, thisevaluation approach has a severe shortcoming, as it fails to differentiate between a“near miss” and a completely misplaced segment boundary. Consider hypothesis-1 and hypothesis-2, produced by two distinct topic segmentation systems in Fig-ure 1.4, they both receive the same F-score of 0 because these two predictions areboth scored with recall = 0. Yet, from a human perspective, hypothesis-2 might bemore favorable than hypothesis-1 since its hypothesized boundaries are considerablycloser to the reference boundaries.Segmentation-based MetricsTo combat the above shortcoming of classification-based evaluation metrics like F-score, two alternative measures, Pk [14] and WinDiff(WD) [130] error scores,10Figure 1.4: An example showing two hypotheses (hypothesis-1 andhypothesis-2) with the identical F-score of 0, despite hypothesis-2 havingboundaries that more closely match the reference. Black boxes indicatethe hypothesized or reference label about whether a sentence is the endof a segment or not. Dashed boxes indicate missed boundary predictions(false negatives).have been introduced to better align with human perceptions on segmentation tasks.Unlike F-score, which primarily evaluates segmentation based on exact bound-ary matches, the Pk error score takes the distance between hypothesized and ref-erence boundaries into consideration. In practice, it slides a k-sized window acrossthe text sentence-by-sentence, assessing if each pair at the window’s two endpointsare correctly placed in the same segment or not. The hypothesized placement (eitherin the same segment or different ones) is compared to the reference, and the seg-mentation system under evaluation is penalized for each disagreement. Figure 1.5demonstrates a snapshot of Pk computing procedure while the window is sliding.S2 and S5, two endpoints of the sliding window, belong to different segments inboth reference and hypothesis-2, but fall within the same segment in hypothesis-1.As a result, a false negative is detected in hypothesis-1 and the performance of itscorresponding segmentation system will be penalized during this time step. Moreformally, Pk error score can be calculated as:Pk(re f ,hyp) =∑n−ki=1 δre f (i,i+k)̸=δhyp(i,i+k)n−kwhere δ (·, ·) is an indicator function which is 1 if sentence i and i+ k are in thesame segment, 0 otherwise. It measures the probability of mismatches between theground truth segments (ref ) and model predictions (hyp) within a sliding window k.As a empirical setting which has been commonly used in previous work, windowsize k is half of the average segment length of ref. Since Pk is a penalty metric, lowerscore indicates better performance.11Figure 1.5: Two hypotheses with an identical F-score of 0, hypothesis-2 isassessed closer to the reference using Pk.Figure 1.6: Two hypotheses with an identical Pk of 0.22, hypothesis-2 is as-sessed closer to the reference using WD and hypothesis-1 is overly seg-mented.Although the segmentation-based Pk provides a more suitable measurement fortopic segmentation than F-score, it may still suffer from the problematic calcula-tion on overly segmented cases. Taking Figure 1.6 as an example, Pk does not pe-nalize either hypothesis-1 or hypothesis-2 in the window of this time step, as inboth cases, S2 and S5 are in different segments, even if hypothesis-1 is overly seg-mented in such window (false positive). In order to tackle such Pk’s limitation, thefix called WinDiff(WD) error score modifies Pk by instead comparing the quantityof hypothesized and reference segmentation boundaries within the window. Moreformally, WD error score can be calculated as:WD(re f ,hyp) = ∑n−ki=1 |bre f (i,i+k)−bhyp(i,i+k)|>0n−kwhere b(·, ·) denotes the number of segment boundaries within the window. Similarto Pk error score, lower WD means better performance. Later on, another evaluationmetric named Pr [43] was proposed to address the inadequacy of WD when evalu-ating extensively long sequences with sparse segment boundary placements. Pr can12be formally computed as:Pr(re f ,hyp) =∑n−ki=1 [bre f (i,i+k)<bhyp(i,i+k)]n−k +∑n−ki=1 [bre f (i,i+k)>bhyp(i,i+k)]∑n−ki=1 [bre f (i,i+k)>0]It evaluates the actual likelihood of a miss (false negative) more intuitively by nor-malizing the number of misses by the number of opportunities for a miss (changen− k to ∑n−ki=1 [bre f (i, i+ k)> 0] in the second term of the WD equation).Even though segmentation-based metrics are typically considered more suit-able for topic segmentation, incorporating different metrics in performance anal-ysis might offer deeper insights about model effectiveness. As for certain topic seg-mentation applications that emphasize exact recall of true boundaries, classification-based metrics (e.g., F-score) might be a more favorable choice.1.3 Thesis OverviewAlthough neural topic segmentation techniques have delivered notable progress andoffer numerous advantages, they still face challenges in accuracy and robustness,forming hurdles to their adoptions in real-world scenarios.The challenge in accuracy often stems from the inherent complexities of naturallanguage and is frequently noticed in failed attempts to use outputs of neural seg-menters to improve other NLP tasks. Taking extractive summarization [181] as anexample, even with neural topic segmentation models trained on akin corpora, inte-grating their predictions might merely bring marginal improvements or even hurt theefficacy of extractive summarizers. Conversely, ground-truth segmentation usuallypresents a better chance to deliver noticeable and consistent performance boost forthe task.Robustness, on the other hand, concerns the model’s ability to consistently per-form well across diverse datasets and conditions. Neural topic segmentation models,despite their sophistication, can be sensitive to changes in input data distribution.When exposed to unfamiliar or out-of-distribution data, their performance can de-grade, leading to inconsistent and unreliable segmentations. Empirical studies inBadjatiya et al. [7] and Xing et al. [188] provide such evidence that when applyingthe neural topic segmenter trained on datasets sampled from Wikipedia to the test-13Figure 1.7: The overview of this thesis with corresponding concepts, researchquestions and manuscripts.ing set of articles with rather different length and organization formats (e.g., clinicalbooks), it normally suffers from a significant performance drop.Given that neural topic segmentation techniques still have room for improvementin both accuracy and robustness, it is crucial to identify and leverage more precisesignals for topic transition prediction. With our assumption presented in §1.1.1,embracing coherence as a cornerstone can potentially elevate the performance ofthese models, making them better suited for diverse real-world applications. Toimprove neural topic segmentation with coherence, two avenues are pursued in thisdissertation: one is emphasizing more explicit coherence modeling during segmentprediction, and the other is providing more reliable coherence signals to segmentprediction.Besides addressing challenges of accuracy and robustness in neural topic seg-mentation techniques for text, it is also crucial to put topic segmentation in thebroader information landscape presented by the explosion of big data. This land-scape covers not only textual data but also various other multi-modal media for-mats, prominently including videos. Exploring how topic segmentation can effec-tively contribute in this context is another essential direction to be pursued in thisdissertation.14An overview of the concepts and publications covered in this dissertation canbe found in Figure 1.7. In order to improve the accuracy and robustness of neuraltopic segmentation, as well as put topic segmentation in the broader data landscapeprominently including videos, we structure studies in this dissertation around threecentral research questions:RQ1: How can coherence be explicitly modeled under neural frameworks to im-prove the accuracy and robustness of topic segmentation?Given the distinct data availability associated with monologue and dialogue topicsegmentation (§1.1.3), to answer this research question, Part I delineates our stud-ies on monologue topic segmentation within a fully supervised neural setting anddialogue topic segmentation within an unsupervised neural setting.Specifically, in Chapter 2, we first investigate effective strategies to enhancecontext-dependent coherence modeling for the supervised neural approach for mono-logue topic segmentation. Our goal is to ensure that (1) coherence patterns betweensentences from the same or different segments should be aptly incorporated in topicboundary inference, and (2) the neural model should be encouraged to make predic-tions more explicitly relying on local contextual coherence. The topic segmenter re-fined by our proposed strategy outperforms state-of-the-art methods at the time whentrained and tested on three datasets with different language complexities. Moreover,we find our strategy also advances the neural topic segmenter’s effectiveness androbustness in both domain-transfer and multilingual scenarios.Further, in Chapter 3, we investigate a notable challenge in dialogue topic seg-mentation where prevailing unsupervised approaches predominantly rely on surfacefeatures to assess topical coherence among utterances. In order to address this chal-lenge, we propose to harness supervisory signals from the utterance-pair coherencescoring task through training a simple yet effective neural utterance-pair coherencemodel with pseudo corpora. Experiments conducted in two different languages (En-glish and Chinese) demonstrate that our proposal outperforms state-of-the-art base-lines dealing with dialogues in accuracy and robustness.Research presented in Part I has already shown that the accuracy and robustnessof neural topic segmentation can be enhanced through explicit coherence modelingbased on signals learned from the objective of coherence relation (score) prediction.15Nevertheless, the scarcity of data with high-quality coherence relation (scoring) la-bels introduces another layer of challenge, keeping the potential of such stream ofapproaches from being fully unleashed. To this matter, one efficacious recipe mightbe taking advantage of extensive knowledge from other relevant auxiliary tasks.These tasks, when trained on their specialized datasets, may inherently grasp andencode coherence signals even if they are primarily designed for different objec-tives. This leads us to the second research question:RQ2: Can we harness learning signals encoded in other related auxiliary tasks toimprove the accuracy and robustness of topic segmentation?In Part II, we particularly explore two auxiliary tasks, discourse parsing and lan-guage modeling, as the media to distantly provide coherence signals to topic seg-mentation.Discourse parsing provides the means to understand and infer the semantic andpragmatic relationships underlying complete documents, inherently aligned withthe local text coherence and further correlated to the inter-sentential topical con-sistency. Recognizing that discourse parsing can be grounded in several theoreticalframeworks (e.g., Rhetorical Structure Theory (RST) [115], Penn Discourse Tree-bank (PDTB)[174], Segmented Discourse Representation Theory (SDRT) [4, 5]),our research specifically employs Rhetorical Structure Theory (RST) as the basis toprovide coherence signals for monologue topic segmentation. This choice is moti-vated by its availability of high-quality and reasonably sized treebanks (comparedwith SDRT [97]), such as RST-DT and GUM, as well as its coverage of the entiretext (compared with PDTB, strictly focusing on local discourse relations betweenlimited text spans [156, Chapter 4]). In order to verify if such an alignment be-tween the output of RST-style discourse parsing and topical consistency (coherence)within the text can be captured computationally, in Chapter 4, we first investigate iftopic segmentation can yield valuable signals that can be explicitly utilized for thegeneration of discourse constituency tree. Through directly employing a greedytop-down aggregation algorithm on soft predictions produced by the neural topicsegmenter, we test this discourse parsing approach on two widely recognized gold-standard discourse treebanks (i.e., RST-DT and GUM). Results showcase strongperformance and suggests that signals of topical coherence derived from topic seg-16mentation can indeed be beneficial for understanding and structuring the narrative ofdocuments, reaffirming the intertwined nature of topic segmentation and discourseparsing. Building upon the validated connection between discourse parsing andtopic segmentation, in Chapter 5, we directly address RQ2 by reversely harnessingthe coherence-related learning signals encoded in discourse parsing to improve theaccuracy and robustness of monologue topic segmentation. More specifically, weintroduce a discourse-aware neural topic segmentation framework. This frameworkincorporates RST-style discourse dependency structures, guiding the model towardsmore interpretable predictions of topic boundaries that emphasize the topical coher-ence between sentences. Our empirical study on English evaluation datasets showsthat injecting RST-style discourse structures to a neural topic segmenter with ourproposed strategy can substantially improve its accuracy and robustness, with littleincrease of model’s complexity.Given the notable success of pre-trained language models (PLMs) across a di-verse range of downstream NLP tasks, some notions of coherence patterns may beimplicitly captured within dense representations of these models even if they arenot explicitly emphasized in the pre-training process [89, 129, 179, 209]. Chan-neling this implicit coherence understanding into topic segmentation may furtherrefine the segmentation accuracy, especially in the data scarcity scenario. Therefore,in Chapter 6, we investigate the effectiveness of unsupervised dialogue topic seg-menters empowered by different types of PLMs on a newly introduced evaluationbenchmark ensembling conversations in multiple domains and with varying com-plexity. As of PLMs, we consider the two key categories explored in previous work,namely language representation models (LRMs) (e.g., BERT [30], RoBERTa [108])and generative language models (GLMs) (e.g., BART [96], GPT-2 [136]). At thetime when we conducted our research, gigantic generative language models alignedwith human instructions were garnering significant interest because of their excep-tional performance across a spectrum of general NLP tasks. Thus, in Chapter 6, weexpand the scope of our study to include this type of PLM and explore prompt-basedmethods by directly prompting one state-of-the-art human-aligned PLM - ChatGPT.Our experiments reveal that generally prompt-based ChatGPT achieves the state-of-the-art accuracy for dialogue topic segmentation, but for those dialogues longer thanits length limit, LRMs are still the best choice.17RQ3: Given the surge in video content, how can we expand and adapt the conceptof topic segmentation to this specific medium?Similar to topic segmentation for text, video topic segmentation can be defined asa multi-modal task aiming to break stretches of videos into smaller segments con-sisting of video frames or clips addressing a common topic. This task holds the po-tential to provide significant benefits in contemporary social settings, serving bothhuman-to-human and human-to-system interactions. It can play a vital role in en-hancing real-time engagement between content creators and their audiences on livestreaming platforms, thereby improving the overall user experience.However, achieving accurate and robust video topic segmentation presents itsown set of challenges. Unlike textual data where semantic patterns are directlyderived from written words, videos are a blend of text (in the form of transcripts)and visual cues. This multi-modal nature underscores the need for another layeron top of text segmentation that can seamlessly integrate both textual and visualsignals in a complementary fashion to predict video topic boundaries effectively.Furthermore, videos also exhibit a wide range of lengths, often varying significantlybased on genres. For instance, a documentary may span hours to delve deeply into atopic, while a short advert or a music video might last only a few minutes but changethemes rapidly. Such variations in length and visual changing speed within videosdemand robust and adaptive approaches.To address the two challenges metioned above, in Part III, we similarly mapvideo topic segmentation as a sequence labeling task due to the availability of textualinput (video transcript) and utilize a hierarchical neural sequence labeling frame-work as the basic architecture of our proposed video segmenter. Then, we extendsuch neural model into the multi-modal setting, with the injection of visual signalsfrom video frames by adding a cross-modal attention network on top of it.Finally, we propose a dual-contrastive learning framework adhering to the un-supervised domain adaptation paradigm, allowing our model to generalize to longervideos with rather different semantics than what it was trained on. Experimentalresults on a short video corpus and two long video corpora demonstrate that ourproposed multi-modal solution, augmented by the dual-contrastive domain adapta-tion strategy, significantly surpasses baseline methods in terms of both accuracy andtransferability (robustness), in both intra- and cross-domain settings.18SummaryIn summary, this thesis aims to improve the accuracy and robustness of neural topicsegmentation across both texts (i.e., monologues and dialogues) and videos. For tex-tual content, we explore two main avenues to better harness coherence informationwithin documents: one involves more explicit coherence modeling (RQ1), and theother utilizes reliable coherence signals distantly provided by auxiliary tasks (RQ2),such as discourse parsing and language modeling. For video content, we introduce amulti-modal neural segmentation approach effectively integrating textual transcriptsand visual frames. Additionally, we propose a domain adaptation strategy to ensureits versatility across video domains of varying lengths (RQ3).Based on our investigation, we have confirmed the importance of coherence intopic segmentation. Coherence can serve as a reliable indicator for detecting topictransitions in text; therefore, enhancing the quality of the coherence signal or refin-ing its utilization can indeed improve the performance and generality of neural topicsegmentation. Concurrently, we have also proposed potential solutions to addressthe challenges posed by the multi-modality characteristic of videos and the drasticlength differences across domains, aiming to expand the task of topic segmenta-tion to fit into a broader data landscape. In addition to the focus of main chaptersin this thesis, we also show that topic segmentation is interconnected with otherNLP-relevant research areas, including semantic parsing of LaTeX documents andthe interpretability analysis of neural models, as evidenced by publications listed inAppendix B.19Part IImproving Topic Segmentationwith Explicit Coherence Modeling20In this part, we explore how can coherence be explicitly modeled under neuralframeworks to improve the accuracy and robustness of topic segmentation (RQ1).Considering the discrepancy in data availability between monologue and dialoguetopic segmentation, our studies in this part are delineated into two branches: mono-logue topic segmentation conducted within a fully supervised neural setting, anddialogue topic segmentation accomplished within an unsupervised neural setting.In Chapter 2, we first focus on a perceived limitation in previously proposed neu-ral approaches for monologue topic segmentation, particularly their inadequacy ineffectively modeling the underlying coherence relations in the context of sentences.To address this issue, we propose to enhance a neural segmenter built upon a hier-archical attention BiLSTM network for sequence labeling, by adding a coherence-related auxiliary task and restricted self-attention. Experimental results conductedin in-domain, domain-transfer, and multilingual settings consistently showcase thesignificant performance improvement achieved in monologue neural topic segmen-tation led by our proposal method.In Chapter 3, we shift our attention to dialogue topic segmentation. Specifically,we first identify one of its major challenges, namely that popular unsupervised ap-proaches for this task primarily rely on surface features, which prove to be inade-quate in assessing the topical coherence among utterances. In order to address thischallenge, we present a simple yet effective strategy to generate “pseudo” trainingcorpora for utterance-pair coherence scoring from large-scale but unlabeled conver-sations in hand. Then, we train a BERT-based neural utterance-pair coherence modelwith the obtained pseudo training corpora. Finally, such model is used to measurethe topical coherence between utterances, acting as the basis of the segmentationinference for dialogue sessions. Experimental results conducted in English and Chi-nese consistently showcase the significant performance improvement achieved indialogue neural topic segmentation by our proposal method, following the unsuper-vised paradigm.21Chapter 2Better Coherence Modeling forNeural Monologue TopicSegmentationThe research work presented in this chapter has been peer-reviewed and publishedat AACL-IJCNLP 2020 [188]. As the principal contributor, I am responsible for re-search question investigation, concept formation, model design and implementation,as well as manuscript preparation. Prof. Giuseppe Carenini served as the supervi-sory author of this work, contributing to research question investigation, conceptformation, manuscript preparation and insightful discussions.2.1 IntroductionTopic segmentation is a fundamental NLP task that has received considerable at-tention in recent years [10, 45, 113]. It can reveal important aspects of a documentsemantic structure by splitting the document into topical-coherent textual units. Tak-ing the Wikipedia article in Table 2.1 as an example, without the section marks, areliable topic segmenter should be able to recover the correct boundaries within thetext and chunk this article into the topical-coherent units T1, T2 and T3.As a wide variety of techniques have been proposed for monologue topic seg-mentation, early unsupervised models mainly exploit word statistic overlaps [41,22Preface:Marcus is a city in Cherokee County, Iowa, United States.[T1] History:S1: The first building in Marcus was erected in 1871.S2: Marcus was incorporated on May 15, 1882.[T2] Geography:S3: Marcus is located at (42.822892, -95.804894).S4: According to the United States Census Bureau, thecity has a total area of 1.54 square miles, all land.[T3] Demographics:S5: As of the census of 2010, there were 1,117 people,494 households, and 310 families residing in the city.... ...Table 2.1: A Wikipedia sample article about City Marcus covering three top-ics: T1, T2 and T356], Bayesian contexts [34] or semantic relatedness graphs [46] to measure the lex-ical or semantic cohesion between the sentences or paragraphs and infer the seg-ment boundaries from them. More recently, framing monologue topic segmenta-tion as neural supervised learning has become a mainstream fashion, because of theremarkable success achieved by neural solutions in most other general NLP tasks[3, 88, 146, 165, 166]. Despite minor architectural differences, most of these neu-ral solutions for topic segmentation adopt Recurrent Neural Network [145] and itsvariants (RNNs) as their main framework. On the one hand, RNNs are appropri-ate because topic segmentation can be modelled as a sequence labeling task whereeach sentence is either the end of a segment or not. On the other hand, this choicemakes these neural models limited in how to model the context. Because some so-phisticated RNNs (eg., LSTM, GRU) are able to preserve long-distance information[106, 146, 172], which can largely help language models. But for topic segmenta-tion, it is critical to supervise the model to concentrate more on the deterministiclocal context, rather than on information that might be only loosely relevant from amore distant context.Taking the Wikipedia article in Table 2.1 as an example (which covers threetopical-coherent units T1, T2 and T3), the prediction of the segment boundarybetween T1 and T2 hardly depends on the content in T3. Bringing in excessive23long-distance signals may cause unnecessary noise and further hurt performance.Moreover, text coherence can also play an essential role in the process of topic seg-mentation [45, 165]. For instance, in Table 2.1, sentences from the same segment(like <S1, S2> or <S3, S4>) are more coherent together (holding higher seman-tic consistency) than sentence pairs across segments (like S2 and S3). Arguably,with a proper way of modeling the coherence between adjacent sentences, a topicsegmenter can be further enhanced.Based on two phenomenon exhibited above, in this work, we propose to enhancea state-of-the-art (SOTA) topic segmenter [88] based on hierarchical attention BiL-STM network to better model the local context of a sentence in two complementaryways. First, we add a coherence-related auxiliary task to make our model learn moreinformative hidden states for all the sentences in a document. More specifically, werefine the objective of our model to encourage smaller coherence for the sentencesfrom different segments and larger coherence for the sentences from the same seg-ment. Secondly, we enhance context modeling by utilizing restricted self-attention[172], which enables our model to pay attention to the local context and make betteruse of the information from the closer neighbors of each sentence (i.e., with respectto a window of explicitly fixed size k).Our empirical results show (1) that our proposed context modeling strategy sig-nificantly improves the performance of the SOTA neural segmenter on three datasets,(2) that the enhanced segmenter is more robust in domain transfer setting when ap-plied to four challenging real-world test sets, sampled differently from the train-ing data, (3) that our context modeling strategy is also effective for the segmenterstrained on other challenging languages (eg., German and Chinese), rather than justEnglish.2.2 Related WorkNeural Topic SegmentationRecently, neural-based supervised methods have been devised for topic segmenta-tion because of their more accurate predictions and greater efficiency. One line ofresearch frames topic segmentation as a sequence labeling problem and builds neuralmodels to predict segment boundaries directly (as discribed in §1.1.3). The work we24present in this chapter can be seen as pushing this line of research even further by en-couraging the model to more explicitly consider contextual coherence, as well as toprefer more information from the neighbor context through restricted self-attention.Another rather different line of works first trains neural models for other tasks,and then uses these models’ outputs to predict boundaries. Wang et al. [165] traineda Convolutional Neural Network (CNN) network to predict the coherence scoresfor text pairs. Sentences in a pair with large cohesion are supposed to belong tothe same segment. However, their “learning to rank” framework asks for the pre-defined number of segments, which limits their model’s applicability in practice.Our selected framework overcomes this constraint by tuning a confidence thresholdduring the training stage. A sentence with the output probability above this thresholdwill be predicted as the end of a segment. Following a very different approach,Arnold et al. [3] introduced a topic embedding layer into a BiLSTM model. Aftertraining their model to predict the sentence topics, the learned topic embeddings canbe utilized for topic segmentation. However, one critical flaw of their method is thatit requires a complicated pre-processing pipeline, which includes topic extractionand synset clustering, whose errors can propagate to the main topic segmentationtask. In contrast, our proposal only requires the plain content of the training datawithout any complex pre-processing.Coherence ModelingEarly works on coherence modeling merely predict the coherence score for doc-uments by tracking the patterns of entities’ grammatical role transition [11, 12].More recently, researchers started modeling the coherence for sentence pairs bytheir semantic similarities and used them for higher level coherence prediction oreven other tasks, including topic segmentation. Wang et al. [165] demonstrated thestrong relation between text-pair coherence modeling and topic segmentation. Theyassumed that (1) a pair of texts from the same document should be ranked morecoherent than a pair of texts from different documents; (2) a pair of texts from thesame segment should be ranked more coherent than a pair of texts from differentsegments of a document. With these assumptions, they created a “quasi” trainingcorpus for text-pair coherence prediction by assigning different coherence scores tothe texts from the same segment, different segments but the same document, and dif-25ferent documents. Then they proposed the corresponding model, and further use thismodel to directly conduct topic segmentation. Following their second assumption,we propose a neural solution in which by injecting a coherence-related auxiliarytask, topic segmentation and sentence level coherence modeling can mutually bene-fit each other.2.3 Neural Topic Segmentation ModelSince RNN-based topic segmenters have shown success with high-quality trainingdata, we adopt a state-of-the-art RNN-based topic segmenter enhanced with atten-tion and BERT embeddings as our basic model. Then, we extend such model tomake better use of the local context, something that cannot be done effectivelywithin the RNN framework [172]. In particular, we add a coherence-related aux-iliary task and a restricted self-attention mechanisms to the basic model, so thatpredictions are more strongly influenced by the coherence between the nearby sen-tences. As a preview of this section, we first define the problem of topic segmen-tation and introduce the basic model. In the next section, we motivate and describeour proposed extensions.2.3.1 Problem Definition of This WorkTopic segmentation is usually framed as a sequence labeling task. More precisely,given a document represented as a sequence of sentences, our model will predictthe binary label for each sentence to indicate if the sentence is the end of a topicalcoherent segment or not. Formally,Given: A document d in the form of a sequence of sentences {s1,s2,s3, ...,sk}.Predict: A sequence of labels assigned to a sequence of sentences {l1, l2, l3, ..., lk−1},where l is a binary label, 1 means the corresponding sentence is the end of a segment,0 means the corresponding sentence is not the end of a segment. We do not predictthe label for the last sentence sk, since it is always the end of the last segment.26Figure 2.1: The architecture of our basic model. sei is the produced sentenceembedding for sentence Si.2.3.2 Basic Model: Enhanced Hierarchical Attention Bi-LSTMNetwork (HAN)Figure 2.1 illustrates the detailed architecture of our basic model comprising the twosteps of sentence encoding and label prediction. Formally, a sentence encoding net-work returns sentence embeddings from pre-trained word embeddings. Then a labelprediction network processes the sentence embeddings generated earlier and outputsthe probabilities to indicate if sentences are the segment boundaries or not. Finally,to convert the numerical probabilities into binary labels, we follow the greedy de-coding strategy in Koshorek et al. [88] by setting a threshold τ . All the sentenceswith their probabilities over τ will be labeled 1, and 0 otherwise. This parameter τis set in the validation stage.For training, we compute the cross-entropy loss between the ground truth labelsY = {y1, ...,yk−1} and our predicted probabilities P = {p1, ..., pk−1} for a documentwith k sentences:L1 =−k−1∑i=1[yi log pi+(1− yi) log(1− pi)] (2.1)27Dataset CHOI RULES SECTION MEANMaxPooling 1.04 7.74 12.62 7.14BiLSTM 0.92 7.47 11.60 6.66BERT 0.93 8.35 12.08 7.12BiLSTM+BERT 0.81 6.90 11.30 6.34Table 2.2: Pk error score (lower the better) of different sentence encodingstrategies on three datasets (Section 2.4.1). To fit in the table, we shortenAtt-BiLSTM to BiLSTM. Results in bold are the best performance acrossthe comparisons.Looking at the details of the architecture in Figure 2.1, our basic model constitutes astrong baseline by extending the segmenter presented in Koshorek et al. [88] in twoways (colored parts); namely, by improving the sentence encoder with an attentionmechanism (orange) and with BERT embeddings (blue).Enhancing Task-Specific Sentence RepresentationsWhile Koshorek et al. [88] applied max-pooling to build sentence embeddings fromsentence encoding network, we applied an attention mechanism [197] to make themodel better capture task-wise sentence semantics. The benefit of this enhancementis verified empirically by the results in Table 2.2. As it can be seen, replacing themax-pooling with the attention based BiLSTM sentence encoder yields better per-formance.Enhancing Generality with BERT EmbeddingsIn order to better deal with unseen text in test data and hence improve the model’sgenerality, we utilize a pre-trained BERT sentence encoder1 which complements oursentence encoding network. The transformer-based BERT model [30] was trainedon multi-billion sentences publicly available on the web for several generic sentence-level semantic tasks, such as Natural Language Inference and Question Answering,which implies that it can arguably capture more general aspects of sentence seman-tics in a reliable way. To combine task-specific information with generic semanticsignals from BERT, we simply concatenate the BERT sentence embeddings with1Open-sourced tool at github.com/hanxiao/bert-as-service was the one weused for BERT embedding inference. For languages other than English, we use their correspondingpre-trained BERT models.28Figure 2.2: The overview of our proposed topic segmentation model with con-text modeling components: restricted self-attention, auxiliary task mod-ule.the sentence embeddings derived from our encoder. Such concatenation then be-comes the input of the next level network (see Figure 2.1). The benefit of injectingBERT embedding is also verified empirically by the results reported in Table 2.2.We can see that concatenating BERT embedding and the output of Att-BiLSTMyields the best performance compared with only BERT embedding or the output ofAtt-BiLSTM.2.3.3 Auxiliary Task LearningIn a well-structured document written by human, the semantic coherence of a pairof sentences from the same segment should tend to be relatively greater than thecoherence of a pair of sentences from different segments. This observation pro-vides us with an alternative way to enable better context modeling by formulatinga coherence-related auxiliary task whose objective can be jointly optimized withour original objective (Equation 2.1). This task thereby is to predict the consecu-tive sentence-pair coherence by using the sentence hidden states generated from the29BiLSTM network. Concurrently minimizing the loss of this task can regulate ourmodel to learn better semantic coherence relation between sentences by reducingthe semantic coherence scores for the sentence pairs across segments and increasingthe semantic coherence scores for the sentence pairs within a segment.To obtain the ground truth for our introduced auxiliary task (sentence-pair coher-ence prediction), we leverage the ground truth of our segmented training set ratherthan requiring external annotations. For a document which contains m sentences,there are m− 1 consecutive sentence pairs. If this document has n segment bound-aries, then among those m− 1 sentence pairs, n sentence pairs are from differentsegments, while the remaining m−n−1 sentence pairs are from the same segment.In order to concurrently minimize the coherence of the sentences from different seg-ments and maximize the coherence of the sentences in the same segment, we give asentence pair < si,si+1 > a coherence label li = 1 if sentences in this pair are fromthe same segment, and li = 0 otherwise. The embeddings ei and ei+1 of adjacent sen-tences pairs < si,si+1 > used for coherence computing are calculated from BiLSTMforward and backward hidden states−→h and←−h , following the equations below:ei = tanh(We(−→hi −−−→hi−1)+be) (2.2)ei+1 = tanh(We(←−−hi+1−←−−hi+2)+be) (2.3)However, notice that instead of using the conventional [−→hi ;←−hi ] as the embedding ofsentence i, here, similarly to Wang and Chang [167], we subtract forward/backwardstates to focus on the semantics of sentences in the current sentence pair. The seman-tic coherence between two sentence embeddings is then computed as the sigmoid oftheir cosine similarity:Cohi = σ(cos(ei,ei+1)) (2.4)We use binary cross-entropy loss to formulate the objective of our auxiliary task.For a document with k sentences, the loss can be calculated as:L2 =−k−1∑i=1,li=1logCohi−k−1∑i=1,li=0log(1−Cohi) (2.5)which penalizes high Coh across segments and low Coh within segments.30Combining Equation 2.1 and 2.5, we form the loss function of our new seg-menter as:Ltotal = αL1+(1−α)L2 (2.6)with the trade-off parameter α tuned in validation stage, topic segmentation and thecoherence-related auxiliary task are jointly optimized. The architecture of the aux-iliary task module and its integration in our segmenter is shown in red in Figure 2.2.2.3.4 Sentence-Level Restricted Self-AttentionThe self-attention mechanism [163] has been widely applied to many sequence la-beling tasks due to its superiority in modeling long-distance dependencies in text.However, when the task mainly requires modelling local context, long-distance de-pendencies will instead introduce noise. Wang et al. [172] noticed this problem fordiscourse segmentation, where the crucial information for a clause-like ElementaryDiscourse Unit (EDU) boundary prediction comes usually only from the adjacentEDUs. Thus, they proposed a word-level restricted self-attention mechanism byadding a fixed size window constraint on the standard self-attention. In essence, thismechanism encourages the model to absorb more information directly from adjacentcontext words within a fixed range of neighborhood. We hypothesize that the similarrestricted dependencies also play a dominant role in topic segmentation due to theirclose relation. Hence, instead of at word-level, we add a sentence-level restrictedself-attention on top of the label prediction network of the basic model, as shown ingreen in Figure 2.2.In particular, once hidden states are obtained for all the sentences of document d,we compute the similarities between the current sentence i and its nearby sentenceswithin a window of size S. For example, the similarity between sentence si and s jwhich is within the window size is computed as:simi, j =Wa[hi;h j;(hi⊙h j)]+ba (2.7)where hi, h j are the hidden state of si and s j. Wa and ba are both attention parameters.[; ] is the concatenation operation and ⊙ is the dot product operation. The attention31weights for all the sentences in the fixed window are:ai, j =esimi, j∑Ss=−S esimi,i+s(2.8)The output for sentence i after the restricted self-attention mechanism is the weightedsum of all the sentence hidden states within the window:ci =S∑s=−Sai,i+shi+s (2.9)where ci denotes the local context embedding of sentence i generated by restrictedself-attention. After getting the local context embeddings for all the sentences, weconcatenate them with the original sentence hidden states and input them to anotherBiLSTM layer (top of Figure 2.2).2.4 Experimental SetupIn order to comprehensively evaluate the effectiveness of our context modeling strat-egy of adding a coherence-related auxiliary task and a restricted self-attention mech-anisms to the basic model, we conduct three sets of experiments for evaluation:(i) Intra Domain: we train and test the models in the same domain, repeatingthis evaluation for three different domains (datasets).(ii) Domain Transfer: we train the models on a large dataset which covers avariety of topics and test them on four challenging real-world datasets.(iii) Multilingual: we train and test our model on three datasets within dif-ferent languages (English, German and Chinese), to assess our proposed strategy’sgenerality within different languages.2.4.1 DatasetsData for Intra-Domain EvaluationHigh quality training dataset for topic segmentation usually satisfies the followingcriteria: (1) large size; (2) cover a variety of topics; (3) contains real documentswith reliable segmentation either from human annotations or already specified in32Dataset CHOI RULES SECTION WIKI-727Kdocuments 920 4,461 21,376 727,746# sent/seg 7.4 7.4 7.2 13.6# seg/doc 10.0 16.6 7.9 3.5real world(a) Statistics of datasets for in-domain evaluation.Dataset WIKI-50 CITIES ELEMENTS CLINICALdocuments 50 100 118 227# sent/seg 13.6 5.2 3.3 28.0# seg/doc 3.5 12.2 6.8 5.0real world(b) Statistics of datasets for domain transfer evaluation.Table 2.3: Statistics of all the English topic segmentation datasets used in ourexperiments.the documents e.g., sections. In order to comprehensively evaluate the effectivenessof our context modeling strategy when dealing with data of different quality, wetrain and test models on the following three datasets:- CHOI [27] whose articles are synthesized artificially by stitching together differentsources (i.e., they were not written as one document by one author). Hence, it doesnot really reflect naturally occurring topic drifts. While the quality of this datasetis low, it is an early but popular benchmark for topic segmentation evaluation. Weinclude this dataset to allow comparison with the previous work.- RULES [15] is a dataset collected from the U.S. Federal Register issues2. WhenU.S. federal agencies make changes to regulations or other policies, they must pub-lish a document called a “Rule” in the Federal Register. The Rule describes whatis being changed and discusses the motivation and legal justification for the action.Since each paragraph in a document discusses one topic, we consider the last sen-tence of each paragraph as a ground truth topic boundary. The discussion paragraphsusually cover diverse topics in formal, technical language that can be hard to findonline, so we deem it as an additional well-labelled dataset for testing topic seg-mentation to complement our other datasets which contain more informal use of the2www.govinfo.gov/33language.- WIKI-SECTION [3] is a newly released dataset which was originally generatedfrom the most recent English and German Wikipedia dumps. To better align withthe purpose of intra-domain experiment, we only select the English samples fortraining and the German samples will be used in the experiments of multilingualevaluation. The English WIKI-SECTION (labeled SECTION in the tables) consistsof Wikipedia articles from domain diseases and cities. We deem this dataset as themost reliable training source among the three datasets. It has the largest size and thetwo domains (cities and diseases) cover news-based samples and scientific-basedsamples respectively.We split CHOI and RULES into 80% training, 10% validation, 10% testing. ForSECTION, we follow Arnold et al. [3] and split it into 70% training, 10% validation,20% testing. Table 2.3(a) contains the statistical details for these three sets.Data for Domain Transfer EvaluationWe pick WIKI-SECTION as our training set in this line of experiments, due to itslargest size and variety of covered topics. Following previous work, we evaluateour model and baselines on four datasets that originate from different source dis-tributions: WIKI-50 [88] which consists of 50 samples randomly generated fromthe latest English Wikipedia dump, with no overlap with training and validationdata. Cities [24] which consists of 100 samples generated from Wikipedia aboutcities. We also ensure that this dataset has no overlap with training and validationdata. Elements [24] which consists of 118 samples generated from Wikipedia aboutchemical elements. Clinical Books [114] which consists of 227 chapters from amedical textbook. Table 2.3(b) gives more detailed statistics for these datasets.Data For Multilingual EvaluationIn order to test the effectiveness of our context modeling strategy across languages,besides the English WIKI-SECTION, we train and test our model on two otherWikipedia datasets in German and Chinese:- SECTION-DE which was released together with English WIKI-SECTION in Arnoldet al. [3]. It also contains articles about cities and diseases. The section marks are34Dataset EN DE ZHdocuments 21,376 12,993 10,000# sent/seg 7.2 6.3 5.1# seg/doc 7.9 7.0 6.4real worldTable 2.4: Statistics of the the WIKI-SECTION data in English(EN), Ger-man(DE) and Chinese(ZH).used as the ground truth labels.- SECTION-ZH which was randomly generated from the Chinese Wikipedia dump3mentioned in Hao and Paul [54]. As before, section marks are also used here asground truth boundaries. The statistical details of these two datasets can be found inTable 2.4.2.4.2 BaselinesThese include two popular unsupervised topic segmentation methods, BayesSeg [34]and GraphSeg [46], as well as the three recently proposed supervised neural models,TextSeg [88] (from which we derive our basic model), Sector [3] and HierarchicalTransformer (labeled Transformer in the tables) [45]. We use the original imple-mentation of BayesSeg, GraphSeg and TextSeg. We reimplement the HierarchicalTransformer ourselves. In Table 2.5b, we adopt the results of BayesSeg, GraphSegand Sector from Arnold et al. [3]4.2.4.3 Evaluation MetricIn order to fairly compare with previous work, We choose to use the standard Pkerror score [14] (detailed introduced in §1.2) as our evaluation metric. Since Pk is apenalty metric, lower score indicates better performance.3https://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/4Arnold et al. [3] reported Sector’s performance on multiple model settings. Here we pick theperformance of the model trained on wikifull to be close to our training setting.352.4.4 Neural Model SetupFollowing Koshorek et al. [88], our initial word embeddings are GoogleNews word2vec(d = 300). We also use word2vec embeddings (d = 300) and Fasttext embeddings(d = 300), which are both derived from Wikipedia corpora for German and Chi-nese respectively. We use the Adam optimizer, setting the learning rate to 0.001 andbatch size to 8. The BiLSTM hidden state size is 256 following Koshorek et al. [88].Model training is done for 10 epochs and performance is monitored over the vali-dation set. We generate BERT sentence embeddings with the pre-trained 12-layermodel released by Google AI (embedding size 768). The window size of restrictedself-attention is 3 and α is 0.8. These were tuned on the validation sets of the datasetswe use.2.5 Results and Discussion2.5.1 Intra-Domain EvaluationTable 2.5a shows the models’ performance on the three datasets, when all supervisedmodels are trained and evaluated on the training and test set from the same domain.To investigate the effectiveness of auxiliary task (AUX) and restricted self-attention(RSA), Table 2.5a also shows the results of individually adding each componentto our basic segmenter. The most important observation from the table is that ourmodel enhanced by context modeling outperforms all the supervised and unsuper-vised baselines with a substantial performance gain. With our context modelingstrategy, the average Pk scores of our model over the three datasets improves on thebest model (TextSeg) among the baselines by 25%. Compared with the basic model,adding AUX or RSA equally gives significant and consistent improvement acrossall three sets. Adding both AUX and RSA results in the biggest improvement by upto 17% on the mean across the three datasets.2.5.2 Domain Transfer EvaluationTable 2.5b compares the performance of the baselines and our model on four chal-lenging real-world test datasets. All supervised models are trained on the trainingset of WIKI-SECTION. One important observation is that our model enhanced by36Dataset CHOI RULES SECTION MEANRandom 49.4 50.6 51.3 50.4BayesSeg 20.8 41.5 39.5 33.9GraphSeg 6.6 39.3 44.9 30.3TextSeg 1.0 7.7 12.6 7.1Sector - - 12.7 -Transformer 4.8 9.6 13.6 9.3Basic Model 0.81 7.0 11.3 6.4+AUX 0.64† 6.1† 10.4† 5.7+RSA 0.72† 6.3† 10.0† 5.7+AUX+RSA 0.54† 5.8† 9.7† 5.3(a) Results for intra-domain evaluationDataset Wiki-50 Cities Elements ClinicalRandom 52.7 47.1 50.1 44.1BayesSeg 49.2 36.2 35.6 57.2GraphSeg 63.6 40.0 49.1 64.6TextSeg 28.5 19.8 43.9 36.6Sector 28.6 33.4 42.8 36.9Transformer 29.3 20.2 45.2 35.6Basic Model 28.7 17.9 43.5 33.8+AUX 27.9 17.0† 41.8† 31.5†+RSA 27.8† 16.8† 42.7 31.9†+AUX+RSA 26.8† 16.1† 39.4† 30.5†(b) Results for domain transfer evaluationTable 2.5: Pk error, results in bold indicate the best performance across allcomparisons. Underlined results indicate the best performance in the bot-tom section. † indicates the result is significantly different (p< 0.05) frombasic model.context modeling outperforms all the baseline methods on three out of four testsets with a substantial performance gap. Admittedly, BayesSeg performs better onElements, possibly because that merely word embedding similarity is sufficient toindicate segment boundaries in this dataset. However, BayesSeg is completely dom-inated by our model on the other test sets. Overall, this indicates that our proposedcontext modeling strategy can not only enhance the model under the intra-domainsetting, but also produce robust models that transfer to other unseen domains. Fur-thermore, we observe that AUX and RSA are both necessary for our model, since37Dataset EN DE ZHRandom 51.3 48.7 52.2Basic Model 11.3 18.2 20.5+AUX 10.4† 17.7 20.5+RSA 10.0† 16.6† 19.8†+AUX+RSA 9.7† 15.9† 20.0†Table 2.6: Pk error score on the datasets in three languages (English, Germanand Chinese).they do not only improve performance individually, but they achieve the best resultswhen synergistically combined.2.5.3 Multilingual EvaluationTable 2.6 shows results for our context modeling strategy across three different lan-guages: English (EN), German (DE) and Chinese (ZH). Remarkably, even our basicmodel without any add-on component outperforms the random baseline by a widemargin. Looking at the gains from AUX and RSA, for German we observe a patternsimilar to English, with our complete context modeling strategy (AUX+RSA) de-livering the strongest gains. However, the performance on Chinese is not as strongas on English and German. Employing RSA still achieves a statistically significant0.7 Pk score drop, but introducing AUX does not help. One possible reason is thatthe sentences in the Chinese Wikipedia pages are relatively short and fragmented.Thus, the semantics of these sentences may be too simple to sufficiently guide thecoherence auxiliary task. In general, when comparing the behavior of our contextmodeling strategy across these three languages, RSA appears to yield stable bene-fits, while the effectiveness of AUX seems to depend more on peculiarities of thedataset in the target language.2.6 ContributionsIn this chapter, we address a serious limitation evident in current neural topic seg-menters for monologue text, namely their inadequacy in effectively modeling theunderlying coherence relations based on context. To this end, we propose a novelneural model that adds a coherence-related auxiliary task and restricted self-attention38on top of a hierarchical BiLSTM attention segmenter to make better use of the un-derlying coherence signals in contextual information. Experimental results of intra-domain inference on three datasets show that our strategy is effective within do-mains. Further, results on four challenging real-world benchmarks demonstrate itseffectiveness in domain transfer settings. Finally, the application to other two lan-guages (German and Chinese) suggests that our strategy has potential in multilingualscenarios. In summary, our proposal in this chapter offers an effective strategy toenhance neural solutions for supervised topic segmentation by explicitly modelingcoherence, thereby partially addressing RQ1 from the monologue perspective.39Chapter 3Better Coherence Modeling forDialogue Topic SegmentationThe research work presented in this chapter has been peer-reviewed and publishedat SIGDIAL 2021 [184]. As the principal contributor, I am responsible for researchquestion investigation, concept formation, model design and implementation, aswell as manuscript preparation. Prof. Giuseppe Carenini served as the supervi-sory author of this work, contributing to research question investigation, conceptformation, manuscript preparation and insightful discussions.3.1 IntroductionDialogue Topic Segmentation (DTS), as a fundamental task of dialogue modeling,has received considerable attention in recent years. DTS aims to reveal the topicstructure of a dialogue by segmenting the dialogue session into its topically coher-ent pieces with the clear boundaries where the topic transitions occur at. Taking anexample given in Table 3.1. Topic transition happens after Turn-4 and Turn-6, wherethe topic is correspondingly switched from “the requirement of the insurance cover-age” to “the information presented on the insurance card”, and then to “the way ofsubmitting the insurance card”. Therefore, we can confidently put topic boundariesat where these two topic transition happens.Different from the monologue topic segmentation (MTS) task (Chapter 2) [88,188], the shortage of labeled dialogue corpora has always been one of the major40Turns Dialogue TextTurn-1: A: For how long should the liability insurancecoverage remain in effect?Turn-2: B: As long as the registration of your vehicleremains valid.Turn-3: A: Does this apply for motorcycles too?Turn-4: B: There are some exceptions for motorcycles.Turn-5: A: Regarding the name on my vehicle registra-tion application and the one on the Insurance Identifica-tion Card, do they need to be the same?Turn-6: B: yes, the names must match in both documents.Turn-7: A: Can I submit copies or faxes of my Insuranceidentification card to the DMV?Turn-8: B: yes, you can. But take into consideration thatthe card will be rejected if the DMV barcode reader cannot scan the barcode.Table 3.1: A dialogue topic segmentation example sampled from Doc2Dial[36]. This dialogue is segmented into three topical-coherent units (utter-ances in the same color are about the same topic).problems for DTS. Collecting annotations about topic shifting between the utter-ances of dialogues is highly expensive and time-consuming. Hence, most of theproposed labeled datasets for DTS are typically used for model evaluation ratherthan training. They are often small in size [195], and those that are artificially gener-ated may not align well with real dialogues, possibly leading to noise [36]. Becauseof the lack of training data, most of previously proposed methods for DTS followthe unsupervised paradigm. The common assumption behind these unsupervisedmethods is that the utterances associated with the same topic should be more coher-ent together than the utterances about different topics [56, 134]. Hence, effectivelymodeling the coherence among utterances becomes the key ingredient of a success-ful DTS model. However, the performances of the prior unsupervised DTS modelsare usually limited since the coherence measurements between utterances are typi-cally based on surface features (eg,. lexical overlap) [34, 56] or word-level seman-tics [151, 195]. Even though these features are easy to extract and thus make modelsmore generally applicable, they can only reflect the coherence between utterancesin a rather shallow way. More recently, there is work attempting to depart from theunsupervised setting by casting DTS as a weakly supervised learning task and uti-41lizing a RL-based neural model as the basic framework [158]. However, while thisapproach has been at least partially successful on goal-oriented dialogues when pro-vided with predefined in-domain topics, it cannot deal effectively with more generalopen-domain dialogues.To alleviate the aforementioned limitations in previous work, in this chapter,we still consider casting DTS as an unsupervised learning task to make it moreapplicable to dialogues from diverse domains and resources. However, instead ofmerely utilizing shallow features for utterance coherence consistency prediction, weleverage the supervised information from the text-pair coherence scoring task (i.e.,measuring the coherence of adjacent textual units [165, 168, 194]), which can moreeffectively capture the deeper semantic (topical) relations between them. Due tothe absence of supervision, we propose a simple yet effective strategy to gener-ate a training corpus for the utterance-pair coherence scoring task, with the pairedcoherent/not- utterance pairs as datapoints. Then, after applying such strategy, weuse the resulting corpus to train an utterance-pair coherence scoring model with therelative ranking objective [98].In practice, we create a training corpus from large conversational datasets1 con-taining real daily communications and covering various topics (proposed in Li et al.[103] and Wang et al. [169]). In particular, all the adjacent utterance pairs are firstlyextracted to form the positive sample set. Then for each positive sample, the cor-responding negative samples are generated by replacing the subsequent turn in thepositive sample with (1) an non-adjacent turn randomly picked from the same dia-logue, and (2) a turn randomly picked from another dialogue talking about anothertopic. Once the training corpus is ready, we re-purpose the Next Sentence Predic-tion (NSP) BERT model [30] as the basic framework of our utterace-pair coherencescoring model. After fine-tuning the pretrained NSP BERT on our automaticallygenerated training corpus with the marginal ranking loss, the resulting model canthen be applied to produce the topical coherence score for all the consecutive utter-ance pairs in any given dialogue. Such scores can finally be used for the inferenceof topic segmentation for that dialogue.We empirically test the popular TextTiling algorithm [56] enhanced by the su-pervisory signal provided by our learned utterance-pair coherence scoring model on1The statistical details of the two corpora we used are presented in Section 3.3.1.42two languages (English and Chinese). The experimental results show that TextTilingenhanced by our proposal outperforms the state-of-the-art (SOTA) unsupervised dia-logue topic segmenters by a substantial margin on the testing sets of both languages.Finally, in a qualitative analysis, by visualizing the segment predictions of the dif-ferent DTS segmenters on a sample dialogue, we show that the effectiveness of ourproposal seems to come from better capturing topical relations and consideration fordialogue flows.3.2 Related WorkDialogue Topic Segmentation (DTS)Similar to the topic segmentation for monologue, dialogue topic segmentation aimsto segment a dialogue session into the topical-coherent units. Therefore, a widevariety of approaches which were originally proposed for monologue topic segmen-tation, have also been widely applied to conversational corpora. However, comparedwith monologue documents, dialogues are generally more fragmented and containmany more informal expressions. The discourse relation between utterances are alsorather different from the monologue text. These distinctive features may introduceundesirable noise and cause limited performance when the supervised approachestrained on monologues is applied. Since the lack of training data still remains a prob-lem for DTS, unsupervised methods, especially the ones extending TextTiling [56],are still the mainstream options. For instance, Song et al. [151] enhanced TextTilingwith word embeddings, which better capture the underlying semantics than bag-of-words style features. Later, Xu et al. [195] replaced word embeddings with BERTas the utterance encoder to produce the input for TextTiling, because pretrained lan-guage models like BERT better capture more utterance-level dependencies. Also, toavoid a too fragmented topic segmentation, they adjusted the TextTiling algorithminto a greedy manner, which however requires more hyper-parameters and greatlylimits the model’s transferability. In contrast, in this work, we adopt the originalTextTiling to minimize the need of hyperparameters and use coherence signals forutterances learned from real-world dialogues to make our proposal more suitable forconversational data.Another line of research explores casting DTS as a topic tracking problem [84,43158], with the predefined conversation topics as part of the supervisory signals. Eventhough they have already achieved SOTA performance on the in-distribution data,their reliability on the out-of-distribution data is still rather poor. In contrast, ourproposal in this work does not require any prior knowledge (i.e., predefined topics)as input, so it is more transferable to out-of-distribution data.Coherence ScoringEarly work Barzilay and Lapata [11, 12] observed the linguistic phenomenon thatparticular patterns of grammatical role transition for entities can effectively revealthe coherence of monologue documents. Hence, they proposed the entity-grid ap-proach by using entity role transitions mined from documents as the features fordocument coherence scoring as particular entity transition patterns are highly in-dicative for coherence assessment. More recently, Cervone and Riccardi [22] pickedup this idea again and attempted to explore the potential of the entity-grid approachon conversational data. Their empirical study further proved that this linguistic phe-nomenon was also suitable for text in the form of dialogue. However, one key limi-tation of the entity-grid model is that by excessively relying on the identification ofentity tokens and their corresponding roles, its performance can be negatively influ-enced by accumulated errors from other NLP pre-processing tasks, such as corefer-ence resolution, which can be particularly noisy in dialogues.In order to resolve this limitation, researchers have explored scoring a documentcoherence by measuring and aggregating the coherence of its adjacent text pairs.Mesgar and Strube [118] proposed a neural coherence model which first identifiedthe salient semantics from adjacent sentence pairs to represent the relations. Thedocument coherence score is then predicted based on the consecutive sentence rela-tion shift. Xu et al. [194] claimed an assumption that a document coherence couldbe calculated as the summation of the coherence scores of all consecutive sentencepairs in this document. Therefore, they devised a discriminative coherence modelonly accepting pairs of sentences as input. They used this model to predict coher-ence scores of all the sentence pairs in a document and the coherence score for thewhole document is calculated as the sum of the coherence scores of all consecutivesentence pairs it contains. Within such line of works, Wang et al. [165] is the first todemonstrate the strong relation between text-pair coherence scoring and monologue44topic segmentation. In particular, they argued that a pair of texts from the samesegment should be ranked more coherent than a pair of texts randomly picked fromdifferent paragraphs. With this assumption, they proposed a CNN-based model topredict text-pair semantic coherence, and further use this model to directly conducttopic segmentation. In this chapter, we investigate how their proposal can be ef-fectively extended to dialogues. Furthermore, we propose a novel method for datageneration and model training, so that DTS and coherence scoring can mutuallybenefit each other.3.3 MethodologyFollowing most of the previous work, we adopt TextTiling [56] (§1.1.2) as the basicalgorithm for DTS to predict segment boundaries for dialogues ((b) in Figure 3.1).In this section, we will describe our novel training data generation strategy and thearchitecture of our new utterance-pair coherence scoring model, which are the twokey contributions of this work.3.3.1 Training Data for Coherence ScoringIn this chapter, We follow previous work [61, 165, 194] to optimize the utterance-pair coherence scoring model (described in Section 3.3.2) with marginal rankingloss. Formally, the coherence scoring model CS receives two utterances (u1,u2)as input and return the coherence score c = CS(u1,u2), which reflects the topicalrelevance of this pair of utterances. Due to the lack of corpora labeled with ground-truth coherence scores, we follow the strategy proposed in Wang et al. [165] totrain CS based on the pairwise ranking with ordering relations of coherence betweenutterance pairs as supervisory signals.In order to create the training data labeled with coherence ordering relations,we make two assumptions about linguistic features of dialogue text: (1) A pair ofadjacent utterances is likely to be more topical coherent than a pair of non-adjacentutterances but still in the same dialogue session. (2) A pair of utterances from thesame dialogue is likely to be more topical coherent than a pair of utterances sam-pled from different dialogue sessions. These two assumptions are the principles tofollow for the generation of training data for utterance-pair coherence scoring. To45formalize the ordering relations, we notate a source dialogue corpus as C and use ukito represent the ith utterance in the dialogue dk ∈C . Then the two ordering relationsbased on the above assumptions can be formulated as:CS(uki ,uki+1)>CS(uki ,ukj),j /∈ {i−1, i, i+1}(3.1)CS(uki ,ukj)>CS(uki ,umj ),k ̸= m(3.2)Since the ranking objective is pairwise, given two utterance pairs, we deem the pairwith higher/lower coherence score as the positive/negative instance. Taking eq. 3.1as an example, (uki ,uki+1) and (uki ,ukj) are positive and negative instance respectively.Since the generality of the obtained coherence scoring model will significantlyimpact the robustness of the overall segmentation system, having a proper sourcedialogue corpus C to generate training data from is a critical step. We believe thatan ideal source corpus should satisfy the following key requirements: (1) having afairly large size; (2) covering as many topics as possible; (3) containing both formaland informal expressions. To test the strength of our proposal in a multilingual set-ting, we select DailyDialog2 [103] and NaturalConv3 [169] for English and Chineserespectively. These two conversational corpora both consist of open-domain con-versations about daily topics. Table 3.2 gives some statistics about them. Differentfrom task-oriented dialogues, open-domain dialogues usually contain more diversetopics and expressions. From Table 3.3, we can see that both corpora cover multipletopics and some topics like Politics, Finance and Tech are supposed to havemore technical language, while other topics like Sports, Entertainment andOrdinary Life should include more casual expressions. Due to the simplifi-cation, next we will only use DailyDialog as our running example source dialoguecorpus C to illustrate the training data generation process for coherence scoring.Given the source corpus DailyDialog, we first collect positive instances by ex-tracting the adjacent utterance pairs which meet the Bi-turn Dialog Flow describedin Li et al. [103]. The utterances in this corpus are labeled with the dialogue acts in-2http://yanran.li/dailydialog.html3https://ai.tencent.com/ailab/nlp/dialogue/46Dataset DailyDialog NaturalConvTotal dialogues 13,118 19,919Language English ChineseAvg. # turns per dialog 7.9 20.1Avg. # tokens per turn 14.6 12.2# covered topics 10 6Table 3.2: Statistics of the two conversational corpora used for coherencescoring training data generation.Dataset TopicsDailyDialog Ordinary Life, School Life, Culture & Education, Attitue & Emo-tion, Relationship, Tourism, Health, Work, Politics, FinanceNaturalConv Sports, Entertainment, Technology, Games, Education, HealthTable 3.3: Detailed topics covered in two conversational corpora used for co-herence scoring training data generation.cluding {Questions, Inform, Directives, Commissives}. Among all thepossible combinations, Questions-Inform and Directives-Commissives are deemedas basic dialogue act flows which happen regularly during conversations. Once posi-tive instancesP = {(si, t+i )|i∈N} have been collected, we adopt negative samplingto construct the negative instance for each positive instance by randomly picking:— t−i : an utterance not adjacent to si but in the same dialogue.— t′−i : an utterance from another dialogue different from si.These utterances will replace t+i in the positive instance to form two negative in-stances: (si, t−i ) and (si, t′−i ), where CS(si, t+i ) > CS(si, t−i ) > CS(si, t′−i ). In orderto further enlarge the margins of coherence relations presented above, we set twoconstraints. Firstly, t−i should be labeled with the dialogue act different from t+i .Secondly, t′−i should be sampled from a dialogue about a topic different from thedialogue which t+i belongs to. Notice that the second corpus NaturalConv does nothave dialogue act labels, so all the instance generation strategies with dialog acts inneed are not applicable. In particular, positive instances for NaturalConv are simplyadjacent utterances and the additional constraint for creating negative instances, in47Figure 3.1: The overview of our proposed dialogue topic segmentation pro-cedure. (a) Fine-tuning the NSP BERT on the training data of utterance-pair coherence scoring generated from the source dialogue corpus C . (2)Leveraging the fine-tuned BERT as the coherence scoring model to pre-dict coherence scores for all the consecutive utterance pairs in a testingdialogue. TextTiling algorithm is further utilized to infer segment bound-aries.which t−i should be labeled with the dialogue act different from t+i , cannot be ap-plied as well. By applying our novel data generation process, we obtain 91,581 and599,148 paired pos/neg samples for DailyDialog and NaturalConv respectively. Wesplit them into training (80%), validation (10%) and testing sets (10%) for furthermodel training and evaluation.3.3.2 Utterance-Pair Coherence Scoring ModelAs illustrated in Figure 3.1(a), we choose the Next Sentence Prediction (NSP) BERT[30] (trained for the Next Sentence Prediction task) as the basic framework of ourutterance-pair coherence scoring model due to the similarity of these two tasks4.They both take a pair of sentences/utterances as input and only a topically relatedsentence should be predicted as the appropriate next sentence. We first initialize4Instead of NSP BERT (a cross-encoder), we could have also modelled such pairwise scoringwith a bi-encoder, which first encodes each utterance independently. We eventually selected the cross-encoder due to the results in thakur-etal-2021-augmented showing that cross-encoders usually outper-form bi-encoders for pairwise sentence scoring.48the model with BERTbase, which was pretrained on multi-billion publicly availabledata. At the fine-tuning stage, we expect the model to learn to discriminate thepositive utterance pairs from their corresponding negative pairs. More specifically,we have the (si, t+i ) and (si, t−i ) as positive and negative instances fed into the modelrespectively in the form of ([CLS]||si||[SEP]||t+/−i ||[SEP]), where || denotes theconcatenation operation for sequences and [CLS], [SEP] are both special tokensin BERT. Following the original NSP BERT training procedure, we also add positionembeddings, segment embeddings and token embeddings of tokens all together toget the comprehensive input for BERT. The NSP BERT is formed by a sequence oftransformer encoder layers, where each layer consists of a self-attentive layer anda skip connection layer. Here we use the contextualized representation of [CLS]as the topic-aware embedding to predict how much the two input utterances arematched in topic. The topical coherence score will be estimated by passing [CLS]representation through another multilayer perceptron (MLP).In order to encourage the model to learn to assign a positive instance (si, t+i )a coherence score c+i higher than its paired negative instance (si, t−i ) score c−i , weminimize the following marginal ranking loss:L =1NN∑i=1max(0,η+ c−i − c+i ) (3.3)where N is the size of the training set, η is the margin hyper-parameter tuned atvalidation set.3.4 ExperimentsIn order to comprehensively evaluate the effectiveness and generality of our pro-posal, we conduct experiments and analysis to quantitatively and qualitatively com-pare it with multiple baselines on three datasets in two languages (English and Chi-nese).3.4.1 Data for Evaluation- DialSeg 711 [195]: a real-world dataset consisting of 711 English dialogues sam-pled from two task-oriented multi-turn dialogue corpora: MultiWOZ [18] and Stan-49Dataset DialSeg 711 Doc2Dial ZYSdocuments 711 4,130 505language English English Chinses# sent/seg 5.6 3.5 6.4# seg/doc 4.9 3.7 4.0real-worldTable 3.4: Statistics of the three dialogue topic segmentation testing sets formodel evaluation.ford Dialog Dataset [35]. Topic segments of this dataset are from manual annotation.- Doc2Dial [36]: This dataset consists of 4,130 synthetic English dialogues betweena user and an assistant from the goal-oriented document-grounded dialogue corpusDoc2Dial. This dataset is generated by first constructing the dialogue flow automati-cally based on the content elements that corresponds to the relations across or withinsampled from text sections of the grounding document. Then crowd workers createthe utterance sequence based on the obtained artificial dialogue flow. Topic seg-ments of this dataset are extracted based on text sections of the grounding documentwhere the utterances’ information comes from.- ZYS [195]: is a real-world Chinese dataset consisting of 505 conversations recordedduring customer service phone calls on banking consultation. Similar to DialSeg 711,gold topic segments of this dataset are manually annotated.More details of the three datasets described above can be found in Table 3.4.3.4.2 BaselinesWe compare our dialogue topic segmenter with following unsupervised baselines:- Random: Given a dialogue with k utterances, we first randomly sample the numberof segment boundaries b ∈ {0, ...,k−1} for this dialogue. Then we determine if anutterance is the end of a segment with the probability bk .- BayesSeg [34]: This method models the words in each topic segment as drawsfrom a multinomial language model associated with the segment. Maximizing theobservation likelihood of the dialogue yields a lexically-cohesive segmentation.- GraphSeg [46]: This method generates a semantic relatedness graph with utter-50ances as nodes. Segments are then predicted by finding the maximal cliques of thegraph.- GreedySeg [195]: This method greedily determines segment boundaries basedon the similarity of adjacent utterances computed from the output of the pretrainedBERT sentence encoder.- TextTiling (TeT) [56]: The detailed description of this method can be found inSection 1.1.2.- TeT + Embedding [151]: TextTiling enhanced by GloVe word embeddings, byapplying word embeddings to compute the semantic coherence for consecutive ut-terance pairs.- TeT + CLS [195]: TextTiling enhanced by the pretrained BERT sentence encoder,by using output embeddings of BERT encoder to compute semantic similarity forconsecutive utterance pairs.- TeT + NSP: TextTiling enhanced by the pretrained BERT for Next Sentence Pre-diction (NSP), by leveraging the output probability to represent the semantic coher-ence for consecutive utterance pairs.3.4.3 Evaluation MetricsWe apply three standard metrics to evaluate the performances of our proposal andbaselines. They are: Pk error score [14], WinDiff (WD) [130] and F1 score (macro).As introduced in Section 1.2, Pk and WD are both calculated based on the overlapbetween ground-truth segments and model’s predictions within a certain size slidingwindow. Since they are both penalty metrics, lower score indicates better perfor-mance. F1 is the standard armonic mean of precision and recall, with higher scoresindicating better performance.3.4.4 Experimental SetupWe fine-tune the utterance-pair coherence scoring model on BERTbase which con-sists of 12 layers and 12 heads in each layer. The hidden dimension of BERTbaseis 768. Training is executed with AdamW [110] as our optimizer and the scheduledlearning rate with warm-up (initial learning rate lr= 2e-5). Model training is done51for 10 epochs with the batch size 16. Model’s performance is monitored over thevalidation set and finally the margin hyper-parameter η in eq. 3.3 is set to 1 from theset of candidates {0.1,0.5,1,2,5}.3.4.5 Results and AnalysisTable 3.5 compares the results of baselines and our proposal on two English dia-logue topic segmentation evaluation benchmarks. The chosen baselines are clusteredinto the top three sub-tables in Table 3.5: random baseline, unsupervised baselinesnot extended from TextTiling and unsupervised baselines extended from TextTiling.Overall, our proposal (full) is the clear winner for both testing sets in all metrics.Another observation is that the set of segmenters TeT + X, which were proved to beeffective for monologue topic segmentation, cannot consistently outperform the ba-sic TextTiling on conversational data. The reason may be that the coherence predic-tion components of such approaches all rely on signals learned from monologue text(eg., GloVe and pretrained BERT). Due to the grammatical and lexical difference,signals learned from monologues tend to introduce unnecessary noise and limit theeffectiveness of unsupervised topic segmemters when applied to dialogues. In con-trast, our coherence scoring model trained on the dataset of coherent/non-coherentutterance pairs automatically generated from dialogues performs better than all com-parisons by a substantial margin. Overall, this validates that by effectively using thetopical relations of utterances in dialogue corpora, the BERT for next sentence pre-diction is able to produce coherence scores reflecting to what extend the two inpututterances are matched in topic.To confirm the benefit of taking dialogue flows and topics into account, we alsoconduct an ablation study by removing either one of these two parts from the trainingdata generation process for coherence scoring. As reported in the bottom sub-tableof Table 3.5, sampling positive/negative utterance pairs (t+i /t−i in Section 3.3.1)without using dialogue flows causes substantial performance drop on both testingsets, while sampling the other negative utterance pair (t′−i in Section 3.3.1) with-out taking dialogue topics into consideration seems to have a smaller impact on thetrained model’s performance. This observation shows that the dialogue flow is amore effective signal than the dialogue topic. One possible explanation is that thereare some basic dialogue flows that are commonly followed and generalize across52Method DialSeg 711 Doc2DialPk ↓ WD ↓ F1 ↑ Pk ↓ WD ↓ F1 ↑Random 52.92 70.04 0.410 55.60 65.29 0.420BayesSeg [34] 30.97 35.60 0.517 46.65 62.13 0.433GraphSeg [46] 43.74 44.76 0.537 51.54 51.59 0.403GreedySeg [195] 50.95 53.85 0.401 50.66 51.56 0.406TextTiling (TeT) [56] 40.44 44.63 0.608 52.02 57.42 0.539TeT + Embedding [151] 39.37 41.27 0.637 53.72 55.73 0.602TeT + CLS [195] 40.49 43.14 0.610 54.34 57.92 0.518TeT + NSP 46.84 48.50 0.512 50.79 54.86 0.550Ours (w/o Dialog Flows) 32.60 37.97 0.750 48.76 50.83 0.636Ours (w/o Dialog Topics) 26.95 28.98 0.761 46.61 48.58 0.657Ours (full) 26.80 28.24 0.776 45.23 47.32 0.660Table 3.5: The experimental results on two English testing sets: DialSeg 711[195] and Doc2Dial [36]. ↑/↓ after the name of metrics indicates if thehigher/lower value means better performance. The best performancesamong the listed methods are in bold.Method Pk ↓ WD ↓ F1 ↑Random 52.79 67.73 0.398GreedySeg 44.12 48.29 0.502TextTiling 45.86 49.31 0.485TeT + Embedding 43.85 45.13 0.510TeT + CLS 43.01 43.60 0.502TeT + NSP 42.59 43.95 0.500Ours 40.99 41.32 0.521Table 3.6: The experimental results on the Chinese testing set proposed in Xuet al. [195]. The best performances among the listed methods are in bold.different types of dialogues, while dialogue topics are more specific and vary muchmore between different dialogue corpora.To further investigate the generality of our proposal for different languages, wetrain a Chinese coherence scoring model on the training data generated from Nat-uralConv (in Section 3.3.1) and use it together with TextTiling to infer segmenta-tion for Chinese dialogues. Table 3.6 exhibits the performances of our method andbaselines on the testing set ZYS. Since the publicly available implementations forBayesSeg and GraphSeg only support English text as input, they are not included53Figure 3.2: Behaviors of four TextTiling-based segmenters on an example di-alogue selected from Doc2Dial [36]. The horizonal axis is the index ofintervals in a session, and the vertical axis is the value of depth score(higher value means more topical unrelated). The reference and predic-tion of topic boundaries are marked by blue and red vertical lines respec-tively. The overlaps of reference and prediction are marked by purplelines.in this comparison. We note that although we observe a pattern similar to En-glish, namely that our method surpasses all the selected baselines, gains seem to besmaller. While this still validates the reliability of our proposal for languages otherthan English, explaining this interlingual difference is left as future work. With aproper open-domain dialogue corpus for a particular language, TextTiling can be en-hanced by the high-quality topical coherence signals in that language captured byour proposal.3.4.6 Case StudyTo more intuitively analyze the performance of our method and of the baselines, asample dialogue is presented in Figure 3.2. First, notice that in models using more54Method DialSeg 711 Doc2Dial ZYSTextTiling 0.122 0.102 0.113TeT + Embedding 0.136 0.125 0.131TeT + CLS 0.166 0.154 0.158Ours 0.366 0.319 0.320Table 3.7: The average variance of depth scores on three testing sets. Highestvalues are in boldadvanced features to compute coherence (line charts from top to bottom), the vari-ation of depth scores (see §3.3) becomes more pronounced, which seem to indicatethe more advanced models learn stronger signals to discriminate topically relatedand unrelated content. In particular, as shown again on the right-top of Figure 3.2,the plain TextTiling, which uses TF-IDF to estimate the coherence for utterancepairs, yields depth scores close to each other. With features carrying more com-plex semantic information, like word embeddings and BERT encoder pretrained onlarge-scale textual data, the difference of depth scores becomes more obvious. Re-markably, our utterance-pair coherence scoring model optimized by marginal rank-ing loss further enlarges the difference. More tellingly, this trend holds in generalfor all three corpora as shown quantitatively in Table 3.7. We can observe that withmore advanced features informing coherence computation, the variation of depthscores becomes more pronounced, which indicates that more advanced models canlearn stronger signals to discriminate topically related and unrelated content. Re-markably, among all the presented methods, our proposal yields the largest averagevariance of depth scores across all three testing corpora.A second key observation is about the benefit of our proposal taking dialogueflows into consideration in the training process. Consider (U7, U8) as an example,the first three segmenters tend to assign relatively high depth score (low coherence)to this utterance pair due to the very little content overlap between them. However,our method manages to assign this pair the minimal depth score. This is becausesuch utterance pair is a Questions-Inform in the Dialog Flow, thus even ifthere is very limited content in common, the two utterances should still very likelybelong to the same topic segment.553.5 ContributionsThe proposal in this chapter addresses a key limitation of unsupervised dialoguetopic segmenters, namely their inability to effectively quantify topical coherenceamong utterances in the dialogue. To this end, we investigate leveraging signalslearned from a neural utterance-pair coherence scoring model based on fine-tuningNSP BERT, such model is used to measure the topical relevance between utterances,acting as the basis of the segmentation inference. Remarkably, even in the absenceof labeled data containing gold-standard coherence scores, we introduce a straight-forward yet highly effective method to automatically construct a pseudo trainingdataset from any unlabeled source dialogue corpus. The experimental results con-ducted on three testing sets in English and Chinese demonstrate that our proposaloutperforms all alternative unsupervised approaches considered in this chapter. Insummary, our proposal in this chapter offers a method to enhance neural solutionsfor unsupervised topic segmentation by explicitly enhancing utterance coherencemodeling, thereby partially addressing RQ1 from the dialogue perspective.56Part IIImproving Topic Segmentationwith Auxiliary Tasks57In this part, we investigate the potential of harnessing learning signals from re-lated auxiliary tasks to enhance the accuracy and robustness of topic segmentation(RQ2). Particularly, we explore two auxiliary tasks, discourse parsing and languagemodeling, as the media to distantly provide coherence signals to topic segmentationfor monologue and dialogue respectively.Chapter 4 and 5 contains the exploration associated with (RST) discourse pars-ing. In Chapter 4, we first explore whether topic segmentation can provide valuabletopic coherence signals that can be explicitly leveraged in generating RST-style dis-course constituency trees. By applying a greedy algorithm to predictions from theneural topic segmenter, we assess such approach for discourse structure predictionon two RST discourse treebanks: RST-DT and GUM. Results highlight the deepconnection between topic coherence and discourse parsing. Based on the observedcontribution from topic segmentation to discourse parsing, in Chapter 5, we directlyaddress RQ2 by examining how coherence signals from RST discourse parsing canbolster monologue topic segmentation. We introduce a discourse-aware segmenta-tion framework that incorporates RST-style discourse structures to infer segmentsbased on the topical coherence between sentences. Empirical tests show this ap-proach enhances accuracy and robustness with minimal complexity increase.Chapter 6 contains the exploration associated with language modeling. The re-markable success of pre-trained language models (PLMs) across diverse NLP tasksimplies their ability to capture underlying coherence patterns, even without explicitemphasis during training. Utilizing these implicit coherence cues in topic segmen-tation could potentially enhance its accuracy, especially when labeled data is scarce.So in Chapter 6, we systematically assess unsupervised dialogue topic segmentersempowered by various PLMs using a new benchmark that spans diverse conversa-tional domains and complexities. We focus on two primary PLM categories previ-ously studied: language representation models (LRMs) (e.g., BERT [30], RoBERTa[108]) and generative language models (GLMs) (e.g., BART [96], GPT-2 [136]).Given the rising prominence of large-scale GLMs aligned with human instructions,our study also includes and evaluates prompt-based methods, particularly with thestate-of-the-art, human-aligned PLM - ChatGPT. Experimental outcomes reveal thatprompt-based ChatGPT leads in dialogue topic segmentation accuracy. However,for dialogues exceeding ChatGPT’s length capacity, LRMs prove superior.58Chapter 4Predicting Above-SentenceDiscourse Structure using DistantSupervision from TopicSegmentationThe research work presented in this chapter has been peer-reviewed and published atAAAI 2022 [68]. I was the first co-author with my collaborator Dr. Patrick Huber.As a co-leader on this work, I contribute to research question investigation, conceptformation, experiment design and implementation, as well as manuscript prepara-tion. Prof. Giuseppe Carenini served as the supervisory author of this work, con-tributing to research question investigation, concept formation, manuscript prepara-tion and insightful discussions.4.1 IntroductionThe Rhetorical Structure Theory (RST) [115] is arguably one of the most popu-lar frameworks to represent the discourse structure of complete documents. Com-pared to other competing discourse theories (e.g., PDTB [133]), RST postulatescomplete discourse trees based on clause-like Elementary Discourse Units (abbrevi-ated: EDUs), which are sentence fragments on an intermediate granularity level be-59Figure 4.1: Textual levels in a well-structured document and their influence ondownstream tasks and available gold-label data.tween words and sentences. With EDUs acting as leaf nodes in RST-style discoursetrees, constituents are formed by aggregating EDUs into sub-trees containing: (1) aprojective tree structure, (2) a nuclearity assignment for each internal node and (3)rhetorical relations between siblings. In this chapter, we focus on “plain” discoursetree structures and leave the exploration of nuclearity and relation classification outof the scope of this chapter.Due to the definition of complete discourse trees in the RST framework, treestructures become deeper as documents grow longer (as compared to the definitionin PDTB, where the aggregation stops above sentence level). Furthermore, the defin-ing factors for the tree aggregation on higher levels diverts considerably from theones on lower levels (e.g., aggregating multiple paragraphs vs. combining EDUs)[77]. For example, suitable features for EDU level tree aggregations (i.e., low-levels)are mostly influenced by local syntactic and semantic signals, while the tree aggre-gation on paragraph level (i.e., high-levels) is likely to follow more global features,such as major topic shifts planned by the author for possibly complex communica-tive goals [156].Researchers working on RST-style discourse parsing take these considerationsinto account by either (1) proposing hard constraints to construct discourse trees ondistinct textual levels by using varying feature sets [74, 80] or (2) as done in recent60work by Wang et al. [171] and Guz and Carenini [50], by proposing soft constraints,encoding sentence- and paragraph-breaks as input features. Furthermore, recentwork on tree-level dependent RST-style discourse parsing mostly focuses on high-level tree structures [77].Ideally, an RST-style discourse parser should achieve good performance on alllevels shown in Figure 4.1. However, as argued in Kobayashi et al. [87], the perfor-mance on high-level constituency tree structures is especially important (green arrowin Figure 4.1) when converting these constituency trees into dependency structures,as typically done for key downstream applications [66, 75, 116, 117, 147, 182].Unfortunately, training samples for these critical high-level discourse structuresare extremely limited (see red arrow in Figure 4.1). Not only does the largest avail-able human-annotated treebank in English just contains 385 documents, but for eachdocument, vastly more training samples are available for structures within sentencesthan for structures connecting sentences and paragraphs, since the number of nodesin a binary tree decreases exponentially from the leaves towards the root.To tackle the data sparsity issue for discourse parsing, previous work has pro-posed to leverage distant supervision from tasks with naturally annotated and abun-dantly available training data, such as sentiment analysis [65] and summarization[182]. While we believe that both these auxiliary tasks capture some structural in-formation, they are plausibly even more aligned with other aspects of discourse, suchas nuclearity for summarization and discourse relations (e.g., evidence, concession)for sentiment. In contrast, this chapter focuses exclusively on high-level tree struc-ture generation, by exploiting signals from the auxiliary task of topic segmentation.Training on the naturally occurring and abundantly available topic segmentation an-notations (sections/paragraphs), we believe that valuable information on major topicshifts, an effective signal indicative for high-level discourse tree structures, can belearned [156].More specifically, we train the top-performing neural topic segmentation modelproposed in Chapter 2 and use the trained model to generate discourse structures,which we evaluate on two popular discourse parsing treebanks. Based on the sequen-tial output of the topic segmentation model, we explore the generation of discoursetrees using (1) a greedy top-down algorithm and (2) an optimal bottom-up CKYdynamic programming approach [81], predicting RST-style tree structures on/above61sentence level.To better understand and properly compare the performance of our discourse treegeneration algorithm with previously published models, as well as a set of baselines,we evaluate all approaches on three partially overlapping discourse tree subsetsfrom: sentence-to-paragraph (S-P), paragraph-to-document (P-D) and sentence-to-document (S-D). In our evaluation, we find that distant supervision from topic seg-mentation achieves promising results on the high-level tree structure generation task,consistently outperforming previous methods with distant supervision on sentence-to-document level and in some cases reaching superior performance compared tosupervised models.4.2 Related WorkRhetorial Structure Theory (RST) and RST-style Discourse ParsingAs one of the main guiding theories for discourse parsing, the RST framework pro-poses complete constituency discourse trees by first splitting a document into El-ementary Discourse Units (EDUs) and subsequently aggregating them into larger(internal) sub-trees. The generated, projective tree structure (also called tree-span)is further augmented with a local importance score (called nuclearity), indicating asub-tree as either a “Nucleus” (of primary significance) or a “Satellite” (of supple-mentary importance). Furthermore, rhetorical relations (e.g., evaluation, attributionand contrast) are assigned between adjacent sub-trees to represent the type of con-nection. In this work, we follow the RST paradigm, generating above-sentence,“plain” constituency discourse trees (without nuclearity and relations) through dis-tant supervision from topic segmentation.Building on top of the RST discourse theory, RST-style Discourse Parsing aimsto automatically infer discourse trees for new and unseen documents. With only afew small RST-style discourse treebanks available in English (e.g., RST-DT [21],Instruction-DT [157] and GUM [199]), most research in supervised discourse pars-ing has been focused on traditional machine learning approaches, such as DPLP[74], CODRA [80] and the Two-Stage parser [171]. More recently, a small num-ber of neural solutions [50, 51, 87, 198] have also been proposed. However, due tothe limited size of available treebanks, supervised models have been shown to not62perform well when transferred across domains [65]. As a result, a stream of unsu-pervised [67, 86, 125] and distantly supervised [64, 65, 182] discourse parsers havebeen proposed to address the data sparsity issue. Typically, distantly supervisedapproaches leverage supervision signals learned from tasks with abundant trainingdata (e.g., sentiment analysis and extractive summarization) to infer discourse struc-tures. Unsupervised approaches, on the other hand, mostly exploit the documenthierarchy and separately aggregate tree structures on different levels based on re-cursively computed dissimilarity scores [86], syntactic knowledge [125] or using anauto-encoder objective [67]. In this chapter, we propose another distantly super-vised approach, computing discourse structures from the task of topic segmentation.We thereby draw inspiration from the findings in Jiang et al. [77], showing that apre-trained topic segmenter can benefit supervised discourse parsers, especially onhigh-levels. However, instead of augmenting a supervised discourse parser with in-formation obtained from a topic segmentation model, we explore the potential ofdirectly inferring high-level discourse structures from the output of topic segmen-tation, bypassing the data sparsity issue of supervised models through the use ofdistant supervision from topic segmentation.4.3 MethodologyThis section describes our new approach to generate high-level discourse struc-tures for arbitrary documents using distant supervision from topic segmentation. Anoverview of our methodology is presented in Figure 4.2. At the top, we visualize anaturally organized document, where words are aggregated into clauses (or EDUs),which are in turn concatenated into sentences. Sentences are then further combinedinto paragraphs1, forming the complete document. Here, we aggregate sentences asthe atomic units to obtain high-level discourse trees (bottom of Figure 4.2). We donot consider the intra-sentence task (EDU-to-sentence), as individually tackled byLin et al. [105] with great success.1We overload the term “paragraph”, also referring to what in other places is called “sections”.63Figure 4.2: Overview of our approach to generate high-level discourse struc-tures on sentence-to-document level (S-D) from distant topic segmenta-tion supervision.4.3.1 Topic Segmentation ModelAs presented in Chapter 2, the task of topic segmentation can be interpreted as asequence labeling problem. Formally, given a document d in the form of a sequenceof sentences {s1,s2, ...,sk}, a topic segmentation model assigns a probability scoreto each sentence si for i in [1, ...,k− 1], generating a sequence of probabilities P ={p(s1), p(s2), ..., p(sk−1)} with p(si) ∈ [0,1] indicating how likely sentence si is theend of a segment. The last sentence sk does not need to be scored, since it is bydefinition the end of the last segment.Based on the set of probability scores P and an additional threshold τ , deter-mined on a held-out development set, topic segmentation models make binary pre-dictions, labeling sentences with a probability larger than τ as the “end of segment”,or “within segment” otherwise. Here, instead of converting the probabilities intodiscrete sentence boundary labels, we utilize the real-valued probability scores asour signal for distant supervision to infer RST-style discourse trees.Since the topic segmentation task is generally applied on sentence level, anydiscourse tree generated based on its output can by definition only cover structures64Model Wiki RST-DT GUMTextTiling [56] 62.5 ± 0.00 44.3 ± 0.00 51.6 ± 0.00BayesSeg [34] 38.6 ± 0.00 37.5 ± 0.00 49.8 ± 0.00GraphSeg [46] 43.2 ± 0.00 58.7 ± 0.00 53.9 ± 0.00Koshorek et al. [88] 27.9 ± 0.12 26.9 ± 0.49 48.0 ± 1.37Xing et al. [188] 25.1 ± 0.07 25.4 ± 0.33 40.8 ± 1.64Table 4.1: Average topic segmentation performance± standard deviation over5 runs, using the Pk error score [14] (lower values indicate better perfor-mance) on three evaluation corpora. Best performance per column in bold.on and above sentence level (i.e., where leaf nodes represent sentences). In orderto ensure the effectiveness of our selected topic segmentation model (proposed inChapter 2), in Table 4.1, we show that it yields superior performance over its directcompetitors (described in Section 4.4.2) not only on Wikipedia corpus, but alsoacross two RST-DT evaluation corpora (described in Section 4.4.1).4.3.2 Tree GenerationTo convert the sequential outputs of the topic segmentation model into tree struc-tures, we follow the intuition that the value at position i in the output of the topicsegmentation model, namely the likelihood of si being the end of a thematically-coherent segment, can be interpreted as the topical distance between si and si+1.A natural way to exploit the output of the topic segmentation model is to cre-ate a binary discourse tree by applying a greedy top-down algorithm where textspans (i.e., sub-trees) are determined recursively by splitting the two sentences withthe largest topical distance. A small-scale example for this approach is shown inFigure 4.3. Here, we first search for the sentence smax =si∈d p(si) with the maxi-mum probability in P, making it our segmentation point. We then segment the se-quence P into two sub-sequences: Pl = {p(s1), p(s2), ..., p(smax)} (left portion) andPr = {p(smax+1), p(smax+2), ..., p(sk)} (right portion). Next, we mark smax as a pre-viously selected segmentation point by setting p(smax) = 0.0. We then recursivelyrepeat this process for the two sub-sequences in a divide-and-conquer style, until allsentence probabilities are set to 0.0. Noticeably, the bottom-up greedy strategy isequivalent to the top-down approach in our case.Besides the greedy approach described above, a commonly used tree-aggregation65Figure 4.3: Greedy top-down tree generation approach.technique to convert a real-valued sequence into a binary tree structure is the optimalCKY dynamic programming algorithm, as previously used for distantly superviseddiscourse parsing in Huber and Carenini [65] and Xiao et al. [182]. However, apply-ing the CKY algorithm to the real-valued topic-break probabilities is problematic,since the output of any topic segmentation model only contains a single sequence.The intuitive way to fill the CKY matrix is to start merging any two consecutiveunits si and si+1 with the score p(si) of the former unit si and assigning the new(merged) span the aggregation score of the latter, here p(si+1). Applying this strat-egy using the dynamic programming approach, every tree-candidate receives thesame likelihood to represent the document at hand, due to the commutativity prop-erty of the subtree aggregation (see Figure 4.4 for an example). In order to addressthis issue, additional hyper-parameters can be introduced, such as an attribute toquantify the benefit of merging low-probability sentences early on, and delayinglikely topic-breaks. We explore this extended version of the CKY approach using aset of fixed discount factors. In preliminary experiments, we found that this adap-tion of the CKY approach, despite being theoretically superior, is in practice inferiorto the greedy approach, showing unstable performance and resulting in heavily bal-anced/imbalanced trees for large/small discount factors, respectively. As a result,we leave the task of finding a more effective discounting function to future workand merely focus on the superior greedy top-down approach (as illustrated in Fig-ure 4.3) in this work.66Figure 4.4: Problematic CKY aggregation without the use of discounting. Alltrees receive the same final score.4.4 Evaluation4.4.1 DatasetsWe use three diverse datasets to train the topic segmentation models. We randomlysample a subset of Wikipedia articles from the Wikipedia dump2, strictly followingthe sampling scheme in Koshorek et al. [88] (from here on called Wiki). Our Wikicorpus consists of 20,000 documents, consistent in size with previously proposedtopic segmentation training corpora, such as the Wiki-Section dataset [3], originallyused in the top-performing model by Xing et al. [188]. In contrast to the popularWiki-Section dataset, which extracts a strictly domain-limited subset of Wikipedia,exclusively covering webpages on cities and diseases, we lift this topic restriction byuniformly sampling from all of Wikipedia. We split our Wiki dataset into training,validation and test sets using the default 80%, 10%, 10% data-split. We further trainthe topic segmentation model on the RST-DT [21] and GUM [199] corpora. WhileRST-DT exclusively contains news articles from the Wall Street Journal dataset, theadditional GUM treebank used in our experiments covers a mixture of 12 differ-ent genres, including interviews, news stories, academic writing and conversations.Since the two discourse corpora do not have any explicit human-annotated topic seg-2https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz267Dataset Wiki RST-DT GUM# of Docs. 20,000 385 150# of Para./Doc. 31.1 9.99 12.3# of Sents./Doc. 144.9 22.5 49.3# of EDUs/Doc. × 56.6 114.2# of EDUs/Para. × 5.67 9.29# of EDUs/Sent. × 2.51 2.32Table 4.2: Statistics of the three training/testing datasets.ment boundaries, we use paragraph breaks contained in the textual representationof the data as topic-shift indicators for the training of topic segmentation models.Please note that we do not use any human-annotated discourse structures during thetraining procedure of the topic segmentation model.The key statistics of all three datasets are presented in Table 4.2.4.4.2 BaselinesGenerally, we compare our distantly supervised model against three sets of base-lines:(1) Simple, but oftentimes competitive structural and random baselines, includingRight-Branching, Left-Branching and Random Trees.(2) Completely supervised models, including the traditional Two-Stage model [171]trained within- and cross-domain, as well as the recently proposed neural SpanBERTapproach [50].(3) Distantly supervised models, exploiting signals from auxiliary tasks with avail-able, large-scale training data. In particular, we compare our results against themodel by Huber and Carenini [65], training the Two-Stage parser on the MEGA-DT discourse treebank3 inferred from sentiment analysis, and the model by Xiaoet al. [182], directly computing discourse structures from an extractive summariza-tion model trained on the CNN-DM dataset.For topic segmentation, we compare the model proposed in Xing et al. [188] againstthree widely applied unsupervised baselines: TextTiling [56], BayesSeg [34] and3www.github.com/nlpat/MEGA-DT68GraphSeg [46], and a competitive supervised model by Koshorek et al. [88] (seeTable 4.1).4.4.3 Experimental DesignGiven that our newly proposed RST-style tree generation strategy can only be ap-plied on and above sentence level, we cannot directly compare our model with re-sults provided in the literature, generally evaluating complete discourse trees fromEDU-to-document level. In order to fairly compare against baselines and previouslyproposed models, and to provide a better understanding of how our proposal per-forms on different textual levels of the document (as depicted in Figures 4.1 and4.2), we design a set of comparisons extending the regular evaluation technique forcomplete discourse trees. More specifically, instead of just computing the micro-average performance for a complete document (i.e., EDU-to-document), we aimto investigate the structural performance on sentence-to-document (S-D), sentence-to-paragraph (S-P) and paragraph-to-document (P-D) level, allowing us to observediscourse tree generation trends on different levels of the document. Therefore,complete (EDU-to-document) gold tree structures need to be trimmed to: (1) onlycontain nodes covering a complete unit of the lower-bound of measure (e.g., sen-tences in S-D/S-P) and (2) do not contain any nodes covering multiple units of theupper-bound of measure (e.g., paragraphs in S-P). Hence, we propose to trim andrestrict discourse trees with the following two strategies:Trimming Trees to the Lower-BoundAssuming we want to trim the discourse tree up to the sentence level (however,similarly applicable for paragraphs), we aim to remove all intra-sentence sub-treesthat do not cover at least a complete sentence. In other words, assuming we have asequence of EDUs: E = {e1,e2, ...em}, we keep a node n spanning ei to e j in the dis-course tree iff sent(ei) ̸= sent(e j) or ei.is beginning(sentk)∧ e j.is end(sentk), withsent(·) returning the sentence assignment of an EDU. Subsequently, we update thenode n= (ei,e j) to n= (sent(ei),sent(e j)). While this approach works for completesentence sub-trees, we follow the additional rules presented in Sporleder and Las-carides [154] for potentially leaky sentence-level trees (about 5% of gold-standardtrees in RST-DT [80]).69Leaky Sub-Tree AggregationFollowing the approach presented in Sporleder and Lascarides [154], we replaceleaky sentences with a single node n = (sent(ei),sent(e j)) and attach it dependingon the largest intra-sentence subtree (e.g. if for a sentence containing 5 EDUs asub-tree containing the first 3 EDUs is attached to the previous sentence and thesub-tree containing the last 2 EDUs is combined with the following sentence, weattach it to the previous sentence). In case of ties we attach the sentence to the rightneighbour, as done in Sporleder and Lascarides [154]. Furthermore, since parts ofthe leaky sentences are propagated into larger sub-trees in the original structure,we recursively update the complete tree to be consistent in regards to the sentenceassignment described above.Restricting Trees to the Upper-BoundRegarding the restriction of trees to selected upper-bounds (here exemplified onparagraph level), we remove any node n covering sentences si to s j iff para(si) ̸=para(s j) with para(·) returning the paragraph assignment of a sentence.4.4.4 Experiments and ResultsTo evaluate the performance of discourse structure prediction across different ap-proaches, we utilize the RST Parseval micro-average precision [117] as the keymetric, consistent with prior literature (e.g., [64, 80]). In practice, this metric mea-sures how effective a discourse parser is in identifying the right structure in the dis-course tree by computing how many constituents (trees) in predicted trees correctlymatch those in their corresponding gold trees4:Precision = ∑ni=0 treepredi∩treegoldi∑ni=0 |treepredi |where treepredi/goldi denotes the set of constituents in the ith predicted or gold dis-course tree.We show the discourse structure prediction performance of each model on thesentence-to-paragraph (S-P), paragraph-to-document (P-D) and finally sentence-to-document (S-D) level in Tables 4.3a and 4.3b for RST-DT and GUM respectively.4A detailed example of such precision calculation can be found in Figure 16 and Table 2 of [80].70Model S-P P-D S-DBaselinesRandom* 77.11 63.90 60.20Right-Branching 73.57 65.50 59.46Left-Branching 72.41 64.07 58.07Supervised RST-style ParsersTwo-StageGUM [171] 88.82 65.63 69.58Two-StageRST-DT [171] 90.64 68.09 72.11SpanBERTRST-DT [50] 90.75 76.03 77.19Distantly Supervised RST-style ParsersXiao et al. [182]CNN/DM 74.23 66.15 59.10Two-StageMEGA-DT [65] 85.00 65.50 66.99Ours – TSRST-DT 84.34 62.52 65.96Ours – TSWiki 83.43 69.78 68.13Ours – TSWiki+RST-DT 83.84 66.54 65.84(a) Results on RST-DTModel S-P P-D S-DBaselinesRandom* 67.53 60.96 57.99Right-Branching 64.15 72.71 59.39Left-Branching 62.07 54.35 51.56Supervised RST-style ParsersTwo-StageRST-DT [171] 74.20 63.29 63.65Two-StageGUM [171] 76.70 72.94 68.38Distantly Supervised RST-style ParsersXiao et al. [182]CNN-DM 67.89 57.80 53.82Two-StageMEGA-DT [65] 73.37 69.88 64.69Ours – TSGUM 72.54 67.60 62.79Ours – TSWiki 76.98 63.53 65.84Ours – TSWiki+GUM 74.48 67.29 64.69(b) Results on GUMTable 4.3: Evaluation results using the RST Parseval micro-average precisionmeasure on the RST-DT and GUM dataset. Subscripts indicate trainingdataset. TS = Topic Segmentation Model. * = Average performance over10 runs. Best performance per sub-table underlined, best performance percolumn bold.Our results are thereby subdivided into sub-tables according to the type of super-vision used. The top sub-tables contain unsupervised baselines generating eitherrandom trees or completely right-/left-branching structures on the evaluated levels.The second set of results (in the center sub-tables) contain supervised discourseparsers, and the bottom sub-tables show distantly supervised models, including ourresults. To evaluate the ability of our proposal to generate domain-independent dis-course structures, we compare the Topic Segmenter (TS) trained on in-domain data(TSRST-DT/GUM5), the out-of-domain Wiki corpus (TSWiki) as well as a “fine-tuned”approach, first trained on the Wiki corpus, and subsequently fine-tuned on RST-DTor GUM (TSWiki+RST-DT/GUM).Not surprisingly, when evaluating discourse structures on the RST-DT dataset(Table 4.3a), supervised models generally outperform unsupervised baselines anddistantly supervised models. Assessing the bottom sub-table in more detail, it be-comes clear that while the Two-Stage parser trained on MEGA-DT achieves the5Please note that even though we use the RST-DT/GUM datasets, we do not use any discoursetree annotation.71best performance on the sentence-to-paragraph level, our model solely trained on theWiki dataset performs best on the paragraph-to-document and sentence-to-documentlevel, showing that (1) obtaining discourse structures from topic segmentation effec-tively supports high-level discourse parsing and (2) general out-of-domain trainingon large-scale data (TSWiki) performs better than models trained or fine-tuned onin-domain data (TSRST-DT and TSWiki+RST-DT respectively). We believe that a possi-ble explanation for the surprising under-performance of the in-domain training de-scribed above could originate from the limited size of the RST-DT dataset, as wellas the mismatch between the “granularity” of segment information in Wikipediaand RST-DT (see Table 4.2), where the average number of sentences per segmentis about twice as large in Wikipedia than RST-DT. Plausibly, the larger segments inWikipedia better support higher-level discourse structures (which cover larger textspans), leading to superior performance, with the RST-DT fine-tuning step (at finergranularity) introducing the mismatch, and therefore not delivering the expectedbenefits.Our evaluation results on the GUM dataset (Table 4.3b) show similar trends tothe evaluation on RST-DT. Our in-domain trained and further fine-tuned models inthe bottom sub-table do not achieve improved performances compared to the out-of-domain TSWiki model. We believe a possible explanation for this phenomenon is themix of domains within the small GUM training portion, resulting in the fine-tuningstep mixing noisy signals from different genres, hence not providing consistent im-provements over Wikipedia. Interestingly however, there are some observationswhich differ from RST-DT: (1) Unlike for RST-DT, our distantly supervised modeltrained on Wiki even outperforms supervised approaches on sentence-to-documentlevel. (2) Right-branching trees outperform random trees, which hasn’t been the caseon the RST-DT dataset and (3) the paragraph-to-document (P-D) level differs fromprevious results, with the right-branching baseline reaching a performance close tothe best supervised model (Two-StageGUM), outperforming the out-of-domain su-pervised model (Two-StageRST-DT) and all distantly supervised approaches. In Ap-pendix A.1, we present more results on additional evaluation levels (e.g., EDU-to-sentence and EDU-to-paragraph) using the RST and original parseval scores.To further investigate the performance on paragraph-to-document level for theGUM corpus, we show a comparison by genre for the surprisingly high perform-72Genre RB 2SGUM 2SRST-DT TSWiki TSGUMTravel guides 78.1 75.0 65.6 53.1 68.8Biographies 75.0 78.6 60.7 78.6 71.4Fiction 80.6 80.6 61.1 61.1 61.1How-to guides 69.4 64.3 59.2 66.3 75.5Academic writing 70.4 81.5 74.1 70.4 63.0News stories 57.4 57.4 61.8 63.2 69.1Political speeches 80.0 85.0 70.0 60.0 55.0Textbooks 78.6 71.4 64.3 57.1 71.4Interviews 78.8 83.3 66.7 60.6 60.6Table 4.4: RST Parseval micro-average precision on paragraph-to-documentlevel for genres in the GUM corpus. Sample(s) in “Vlog” & “Con-versation” only contain a single paragraph and are therefore omit-ted. RB=Right-Branching, 2S=Two-Stage parser, TS=Topic Segmentationmodel.ing right-branching baseline, the two supervised models and our methods basedon Wiki and GUM in Table 4.4. Right-branching trees thereby achieve the bestperformance in 3 out of the 9 genres, including textbooks and fiction. Supervisedmethods perform best on 5 out of the 9 genres, including highly structured domainssuch as academic writing and interviews. Our distantly supervised model trained onWikipedia reaches the best performance on biographies and the topic segmentationmodel trained on GUM achieves the highest score on how-to guides and news arti-cles. Furthermore, as expected, the supervised parser trained on RST-DT (i.e., thenews domain) performs well on the news genre. Overall, while these mixed resultsappear to align well with our intuition on the prevalent structures in certain genres,further research is required to better understand the relationship between modellingdecisions and their impact on different discourse structures across genres.To complement our quantitative evaluation, we show a set of predicted treesand their respective gold-label structures from RST-DT corpus for well capturedabove-sentence discourse (Figure 4.5), randomly selected documents (Figure 4.6)and poorly captured samples (Figure 4.7). Similar results for the GUM corpus areshown in Figures 4.8, 4.9 and 4.10. Overall, inspecting a large number of tree-structures along with their gold-labels, we recognize that the generated trees are73slightly more balanced than the respective gold-label trees (see for example Figures4.6 and 4.7), however generally represent non-trivial structures, oftentimes well-aligned with major topic shifts and high-level discourse.4.5 ContributionsIn this paper, we show that topic segmentation can provide useful signals for gen-erating high-level discourse constituency tree structure. Comparing multiple ag-gregation methods, our proposal using a greedy top-down algorithm performs thebest when evaluated on two renowned gold-standard discourse treebanks, namelyRST-DT and GUM. We present a comprehensive evaluation across different tex-tual levels in documents, shedding light on the strengths and limitations of simplebaselines, prior models, and our novel, distantly supervised approach rooted in topicsegmentation, on sentence, paragraph and document level. We show additional in-sights into our modelling approach through an ablation study, per-genre evaluationsand qualitative tree generation samples.In conclusion, the research presented in this chapter comprehensively exploresthe potential of utilizing distant supervision from topic segmentation to benefit dis-course structure prediction, implying that both topic segmentation and discourseparsing tap into similar coherence features indicating sentence connections within adocument. This exploration will strongly motivate the study in Chapter 5 to explorewhether the synergy between discourse parsing and topic segmentation is bidirec-tional, and how to take discourse parsing as an auxiliary task to refine topic segmen-tation models, thus addressing RQ2 directly.74Figure 4.5: Positive example (wsj 1346) with 100% structural overlap be-tween prediction and RST-DT gold-label annotation. Gold paragraphspn are indicated by solid lines. Leaves are sentences.Figure 4.6: Random example (wsj 1380) with 80.77% structural overlap be-tween prediction (top) and RST-DT gold-label annotation (bottom). Goldparagraphs pn are indicated by solid lines. Leaves are sentences.Figure 4.7: Negative example (wsj 1365) with 55.56% structural overlap be-tween prediction (top) and RST-DT gold-label annotation (bottom). Goldparagraphs pn are indicated by solid lines. Leaves are sentences.75Figure 4.8: Positive example (GUM bio jesperser) with 76.92% structuraloverlap between prediction (left) and GUM gold-label annotation (right).Figure 4.9: Random example (GUM voyage oakland) with 70.83% structuraloverlap between prediction (left) and GUM gold-label annotation (right).Figure 4.10: Negative example (GUM bio dvorak) with 57.14% structuraloverlap between prediction (left) and GUM gold-label annotation(right).76Chapter 5Improving Topic Segmentation byInjecting Discourse DependenciesThe research work presented in this chapter has been peer-reviewed and publishedat CODI workshop at COLING 2022 [190]. I was the principal contributor, with mycollaborator Dr. Patrick Huber. As the principal contributor, I am responsible for re-search question investigation, concept formation, model design and implementation,as well as manuscript preparation. Prof. Giuseppe Carenini served as the supervi-sory author of this work, contributing to research question investigation, conceptformation, manuscript preparation and insightful discussions.5.1 IntroductionThe aim of topic segmentation makes it tightly connected to related research ar-eas aiming to understand the latent structure of long and potentially complex text.Specifically, understanding the semantic and pragmatic underpinnings of a docu-ment can arguably support the task of separating continuous text into topical seg-ments. To this end, discourse analysis and discourse parsing provide the meansto understand and infer the semantic and pragmatic relationships underlying com-plete documents, well aligned with the local text coherence and highly correlated tothe inter-sentential topical consistency, as shown in Louis and Nenkova [111] andMuangkammuen et al. [122]. Therefore, with a variety of discourse linguistic theo-ries proposed in the past, such as the Rhetorical Structure Theory (RST) [115] dis-77Figure 5.1: An example article about Cholinergic Urticaria (CU) sampled fromthe en disease portion of Wiki-Section dataset [3]. Left: discourse depen-dency structure predicted by the Sent-First discourse parser [205].cussed in Chapter 4, the lexicalized discourse framework [174] (underlying PDTB),and the Segmented Discourse Representation Theory (SDRT) [4, 5], we follow theRST framework in this work (1) as we focus on monologue text (as compared todialogue frameworks, such as SDRT) and (2) since RST postulates complete dis-course trees spanning whole documents, directly aligned with the topical structureof complete documents as demonstrated in Chapter 4.In this chapter, we further motivate the synergistic relationship between topicsegmentation and discourse analysis/parsing in Figure 5.1, showing anecdotal evi-dence of the alignment between the document’s topical structure and the respectiveRST-style discourse dependency graph. Starting from a sequence of sentences, thetask of topic segmentation addresses the problem of splitting the given Wikipediaarticle into an ordered set of topical-coherent fragments (here: T1, T2 and T3) bypredicting topical boundaries. As shown in the example, the document discourse78tree is indicative of the topical structure of the document, as discourse dependenciesoccur considerably more often within a topic segment than across topic segments.Given significant influence on a variety of real-world tasks, topic segmenta-tion is an active research area in the field of NLP. As such, modern, neural meth-ods for monologue topic segmentation are proposed by formulating the task as asentence-level sequence labeling problem, trained and evaluated on the large-scaleWikipedia dataset [10, 45, 109, 188]. Despite the significant improvements achievedby neural supervised topic segmentation models, it remains unclear if these topicsegmenters effectively learn to cluster sentences into topical-coherent pieces basedon the (document-level) topical consistency, or solely exploit superficial patterns(e.g., simple linguistic cues) in the training domain.To address this challenge, in this chapter, we propose a more discourse-awareneural topic segmentation model. We thereby inject above-sentence discourse struc-tures into basic topic segmenter to encourage the model to base its topic bound-ary prediction more explicitly on the topical consistency between sentences. Morespecifically, we propose to exploit a discourse dependency parser pre-trained on out-of-domain data to induce inter-sentential discourse dependency trees. Subsequently,we convert the dependency tree into a directed discourse graph with sentences asnodes and discourse dependencies as edges. With the generated discourse graph,a Graph Attention Network (GAT) [164] is used to encode sentences as discourse-contextualized representations by aggregating information from neighboring sen-tence nodes in the graph. Finally, the discourse-infused sentence representations areconcatenated with standard encodings for segment boundary prediction.In our empirical study conducted on multiple English evaluation datasets, weclearly demonstrate that:(i) Injecting discourse structures can substantially improve the performance ofthe basic neural topic segmentation model on three datasets.(ii) Our novel, discourse-enhanced topic segmenter is more robust comparedto the basic neural model in settings that require domain transfer, showing su-perior performance on four challenging real-world test sets, to confirm the im-proved domain-independence.(iii) Even if our proposal has inferior accuracy against a state-of-the-art seg-79menter sharing the same basic architecture, it does achieve significantly betterefficiency assessed by model’s parameter size and speeds for learning and infer-ence, which makes it potentially more favorable in real-world use.5.2 Related WorkDiscourse Analysis and ParsingDiscourse analysis and parsing analyze and generalize the underlying semantic andpragmatic structure of a coherence document (called a discourse). As an importantupstream task in the field of NLP, discourse analysis proposes elaborate frameworksand theories to describe the textual organization of a document. To this end, a vari-ety of popular discourse theories proposed in the past, such as (besides others) theRhetorical Structure Theory (RST) [115] and the lexicalized discourse framework[175] for monologues as well as the Segmented Discourse Representation Theory(SDRT) [4, 5] for dialogues. Among these theories, the RST discourse theory pos-tulates a single, complete discourse tree for monologue documents, while the lexi-calized discourse framework only focuses on local discourse connectives within andbetween adjacent sentences. Focusing on the connection between discourse infor-mation and topic segmentation, we employ the RST discourse theory in this work,most aligned with the requirement to capture topical coherence.Building on human annotated discourse treebanks, a mix of traditional and neu-ral discourse parsers have been proposed over the last decades, with traditional ap-proaches mainly exploiting surface-level features through Support-Vector Machines(SVMs) [58, 74, 171] or Conditional Random Fields (CRFs) [38, 80]. On the otherhand, neural models achieve similar or superior results on RST discourse parsing,with models using either custom architectures [107, 198] or pre-trained LMs (e.g.BERT [205], RoBERTa [52], SpanBERT [50]). In this work, we generate discoursedependency trees from a BERT-based neural dependency parser proposed in Zhouand Feng [205], since: (i) The parser follows the intuition that information, andhence structures, in sentences are oftentimes “self-contained”. Therefore, it predictsthe interactions between EDUs of the same sentence in a first stage and subsequentlypredicts the inter-sentential discourse structures, which aligns well with our objec-tive of sentence-level topic segmenation. (ii) The parser by Zhou and Feng [205]80Figure 5.2: The overall architecture of our discourse-infused topic segmenta-tion model with predicted discourse dependencies integrated by GraphAttention Network (GAT). With a document in the form of a sequenceof sentences as input, the hierarchical topic segmenter encodes sentencesinto contextualized representations, while the discourse graph modelingcomponent constructs a discourse dependency graph and performs graphmodeling on it. Representations from both components are concatenatedfor the final topic segment boundary inference.makes direct prediction of dependency discourse structures, alleviating the potentialerror caused by converting constituency structures into their respective dependencytrees.5.3 MethodologyAs shown in Figure 5.2, our proposed discourse-aware neural topic segmentationmodel comprises two components: the Hierarchical Topic Segmenter and DiscourseGraph Modeling, highlighted in green and red respectively. Discourse Graph Mod-eling further comprises of a Discourse Graph Construction and Graph Modelingcomponent.815.3.1 Basic Model: Hierarchical Topic SegmenterThe basic architecture of our proposal is adopted from the basic model in usedin Chapter 2 (§2.3.2), consisting of two hierarchical layers: First, a sentence en-coder contextualizes individual sentences, followed by the second layer, condition-ing sentences on the complete document. Following the similar settings, we adoptthe attention BiLSTM architecture1 for each layer and enhance the encodings withpre-trained BERT embeddings. Formally, given a document D as a sequence of nsentences, the sentence encoder (bottom component in Figure 5.2) yields the embed-ding for each individual sentence. Based on the obtained encodings, the document-level contextualization layer returns an ordered set of hidden states H = {h1, ...,hn}.Next, a simple multilayer perceptron (MLP) with a final softmax activation servesas a binary topic boundary predictor based on a threshold τ , tuned on the validationset. During training, we optimize the model in accordance to the cross-entropy loss,while at inference time, every sentence (except the last sentence2) with a probability≥ τ is considered as the end of a segment.5.3.2 Discourse Graph ModelingOur goal is to inject inter-sentential discourse dependency structures into the task oftopic segmentation. We believe that the additional, structural information is therebywell aligned with the topical consistency between sentences, hence, suited to guidethe prediction of topic transitions. To integrate the discourse information into the ba-sic model described in §5.3.1, we first generate an above-sentence discourse depen-dency tree T D for the document. Specifically, we utilize the discourse dependencyparsing model proposed in Zhou and Feng [205], reaching state-of-the-art perfor-mance for discourse tree construction and relation type identification in multiplelanguage settings. The “Sent-First” parser [205] further fits the aim of our proposaldue to its two-staged approach, first generating discourse trees within sentences andsubsequently combining sentence-level sub-trees. This hard constraint allows us toexclusively obtain above-sentence discourse structures, avoiding potentially leaky1We also considered Transformer as the backbone of contextualized encoder, but eventually choseBiLSTM for its superior performance.2We remove the last sentence from the sequence for prediction since it is per definition the end ofthe last segment.82Treebank # of doc # sent/doc # edu/docRST-DT 385 22.5 56.6GUM 150 49.3 114.2SciDTB 1,355 5.3 14.1COVID19-DTB 300 7.8 20.0Table 5.1: Key dataset statistics of the discourse treebanks used for retrainingthe Sent-First discourse parser [205].sub-trees [80]. Regarding the discourse relations attached to every head-dependentpair (discourse dependency), we follow the observation in Xu et al. [193], statingthat the agreement between the type of rhetorical relation is usually lower and moreambiguous, to leave them for future work to avoid error propagation.In contrast to the original proposal in Zhou and Feng [205], training and testingtheir dependency discourse parser on one corpus (i.e., SciDTB [196]), we believethat a mixture of several diverse and publicly available discourse treebanks withdifferent document lengths and domains can increase the parser’s robustness on newand unseen genres. Therefore, we retrain the parser on a mixture of RST-DT3 [21],GUM4 [199], SciDTB5 [196] and COVID19-DTB6 [124]. More specifically, wecombine those discourse treebanks and randomly split the aggregated corpus into80% training, 10% validation, 10% test data. The parser retrained on our combinedtraining portion achieves an Unlabeled Attachment Score (UAS) of 58.6 on the testportion. We show additional key dataset statistics for each treebank used in thischapter in Table 5.1.After training the discourse parser to infer a discourse dependency tree T D fordocument D, we convert the tree structure into a discourse graph GD (as a binarymatrix). Formally, we initialize the graph GD as a n× n identity matrix GD = In,n,connecting every node to itself. Afterwards, we fill in the remaining cells by as-signing GD[i][ j] = 1 iff ∃ T D(i → j), with i, j indexing the head and dependantsentences in the document, respectively. Using the binary matrix representation ofGD, we apply the multi-layer Graph Attention Network (GAT) [164] to update sen-3catalog.ldc.upenn.edu/LDC2002T074corpling.uis.georgetown.edu/gum5https://github.com/PKUTANGENT/SciDTB6https://github.com/norikinishida/biomedical-discourse-treebanks83tence encodings following the discourse graph. More specifically, with the discoursegraph matrix GD and the contextualized representations H = {h1, ...,hn} describedin §5.3.1, within each graph attentional layer, we perform self-attention on the sen-tence nodes. Taking the lth layer as an example, we compute the attention coefficientαi, j between sentence nodes i, j as:α li j = so f tmax(eli j) =exp(eli j)∑k∈Ni exp(elik), (5.1)eli j = LeakyReLU(aTl [W lgli||W lglj]) (5.2)where W l and al are learnable parameters for layer l and T is the transposition op-eration. Ni denotes the direct neighborhood of node i in the graph (GD[i][·] = 1).As the node representation input of the first GAT layer (l = 0), g0i = hi ∈ H. Onceattention coefficients are obtained, we compute the intermediate node representationzli for sentence node i at layer l by aggregating information from neighboring nodesas:zli = ∑j∈Niα li jW lglj (5.3)Following the step in Huang et al. [61], we combine the intermediate node represen-tation zli with the input of this layer gli to get the updated node representation gl+1ias the input for the next layer:gl+1i = ELU(gli + zli) (5.4)where ELU denotes an exponential linear unit [28]. With the output gi from the lastlayer of GAT, we concatenate it together with hi and further feed [hi;gi] into thepredictor layer for segment boundary prediction.5.4 ExperimentsIn order to quantitatively evaluate the effectiveness, generality and efficiency of ourproposal, we conduct three sets of experiments to compare our topic segmentationapproach against a variety of baselines and previous models. Namely, we assess theperformance of our model in regards to the Intra-Domain Segment Inference Perfor-84Dataset # of doc # sent/seg # seg/docCHOI 920 7.4 10.0RULES 4,461 7.4 16.0SECTION 21,376 7.2 7.9Table 5.2: Statistics of the datasets used in intra-domain experiments.mance, Domain Transfer Segment Inference Performance, and conduct an additionalEfficiency Analysis.5.4.1 DatasetsIntra-Domain DatasetsFor the set of intra-domain segment inference experiments, we train and test modelswithin the same domain (here: on the same corpus). We thereby choose three diversecorpora (see Table 5.2 for more details) for the intra-domain evaluation:- Choi [27]. This corpus consists of 920 articles artificially generated by randomlycombining passages from the Brown corpus. The datapoints in this dataset are nothuman written, leading us to solely use this corpus for a preliminary performance as-sessment for topic segmentation models in a 80% (train)/10%(dev)/10%(test) data-split.- Rules [15]. This corpus consists of 4,461 documents about regulation discussionpublished in the Federal Register7 by U.S. federal agencies. Since each paragraphis about one particular regulation and all regulations covered by one document areunder the same category, we deem it as a reasonably coherent data source for topicsegmentation evaluation with the paragraph breaks as ground-truth segment bound-aries. We split this dataset into training, validation and test sets with the default80%, 10%, 10% data-split.- Wiki-Section (Section) [3]. This corpus originally contains Wikipedia articles inboth English and German. The English portion of the dataset, which we use for ourintra-domain experiment, consists of around 3.6k articles about diseases and 19.5k7https://www.govinfo.gov/85Dataset # of doc # sent/seg # seg/docWIKI-50 50 13.6 3.5Cities 100 5.2 12.2Elements 118 3.3 6.8Clinical 227 28.0 5.0Table 5.3: Statistics of the datasets used in domain transfer experiments.articles about cities around the world. After the step of filtering out problematic sam-ples with incorrect sentence segmentation detected by mismatched counts betweensentences and labels, the resulted dataset covers 21,376 articles with the highest-level section marks as ground-truth segment boundaries. We follow the setting inArnold et al. [3] by splitting the dataset into 70% training, 10% validation and 20%test data.Domain Transfer DatasetsTo better evaluate models’ robustness in cases where a domain-shift is present (called“domain transfer segment inference”), we apply the topic segmenters trained onWiki-Section to four small corpora heavily deviating from the training corpus (seeTable 5.3 for more details):- Wiki-50 [88] consists of 50 Wikipedia articles randomly sampled from the lat-est English Wikipedia dump. There is no overlap between this dataset and Wiki-Section.- Cities [24] consists of 100 Wikipedia articles about cities. There is no overlapbetween this dataset and Wiki-Section, even the theme of this dataset is close to theportion of city articles in Wiki-Section.- Elements [24] consists of 118 Wikipedia articles on chemical elements.- Clinical [114] consists of 227 chapters in a clinical book. The subsection markswithin each chapter are deemed as ground-truth segment boundaries.5.4.2 Experimental DesignBaselines86We directly compare our proposed discourse-aware topic segmentation model (calledBasic Model + Discourse) with the following unsupervised and supervised base-lines:- BayesSeg [34]: This unsupervised method makes segmentation prediction by sit-uating the lexical cohesion of text in a Bayesian framework. A text span producedby a distinct lexical distribution is recognized as a coherent topic segment.- GraphSeg [46]: This unsupervised method derives semantically coherent seg-ments through reasoning on a semantic relatedness graph construed from greedylemma alignment.- TextSeg [88]: This supervised neural topic segmenter adopts a hierarchical neuralsequence labeling framework with BiLSTM as the main architecture of each layer.The basic model used in this work (described in §5.3.1) is an effective extension ofthis approach.- Sector [3]: This is a supervised neural topic segmenter extended from TextSegby adding an auxiliary layer for sentence topic label prediction. The learned inter-mediate topic embeddings for sentences are directly utilized for segment boundaryinference.- Transformer [45]: This is a supervised neural topic segmenter consisting of twohierarchically connected Transformer networks for sentence encoding and sentencecontextualization respectively.- Basic Model + Context [188]: This is a top-performing neural topic segmenterwhich shares the same basic architecture with our proposal. The approach improvesthe context modeling capacity of the plain basic model by adding an auxiliary co-herence prediction module and restricted self-attention.Evaluation MetricsIn order to fairly compare with previous work, We choose to use the standard Pkerror score8 [14] (detailed introduced in §1.2) as our evaluation metric. Since Pk is a8We also considered windiff [130] as another evaluation metric. Since it was highly correlatedwith Pk, we omit it and only present performance by Pk to better compare with results reported inprevious works.87penalty metric, lower score indicates better performance. We further quantitativelyanalyze models’ efficiency according to two aspects: Model size and model speed,evaluating the count of learnable parameters and batches/documents processed persecond during training/inference, besides Pk measurement.Implementation DetailsFor the hierarchical topic segmenter (our basic model), we adopt the default settingin Xing et al. [188], with GoogleNews word2vec (d = 300) as initial word embed-dings and the contextualized representation of special token [CLS] (d = 768) frombert-base-uncased as initial sentence embeddings. All BiLSTM layers havethe hidden state size = 256. For the discourse graph model component, the numberof GAT layers is set to 2 through validation and the number of heads is set to 4 asin [164]. The input and output dimensions of each layer = 256. Training uses Adamwith lr = 1e−3 and batch size = 8. Early stopping is applied within 10 epoches ofmodel training and the boundary prediction threshold τ is tuned over the validationset of each corpus we use for intra-domain model evaluation.5.4.3 Intra-Domain Segment InferenceWe report our results of the intra-domain segment inference on the Choi, Rules andWiki-Section datasets in Table 5.4. For better performance comparison, the table issubdivided into three sub-tables: random baseline, previously proposed approachesand models build on top of the basic model we use. We observe that the basic modelwithout any additinal components already outperforms alternative supervised andunsupervised segmenters. With the above-sentence discourse dependency informa-tion injected, as proposed in this chapter, the method (named +Discourse) furtherimproves the performance by a notable margin across all three corpora. We furtherfind that our proposed approach does not achieve superior performances comparedto the basic model enhanced with the context modeling strategy (+Context) in Xinget al. [188]. We believe that a possible explanation for this under-performance couldbe the upstream parsing error of the discourse dependency parser applied out-of-domain, oftentimes severly impairing the parsing performance [64]. Therefore, weconduct an additional experiment on RST-DT due to the availability of gold dis-course structures annotated by human for this corpus. With no human-annotated88Dataset Choi Rules Section RSTDTRandom 49.4 50.6 51.3 40.5BayesSeg 20.8 41.5 39.5 37.5GraphSeg 6.6 39.3 44.9 58.7TextSeg 1.0 7.7 12.6 26.9Sector – – 12.7 –Transformer 4.8 9.6 13.6 –Basic Model 0.81 7.0 11.3 26.9+Context 0.54 5.8 9.7 25.4+Discourse 0.59 6.1 10.2 24.8Table 5.4: Pk (↓) error score on three corpora for intra-domain experiment.Results in bold and underlined indicates the best and second best perfor-mance across all comparisons. The row in purple is the results achievedby our proposal. The column in green is the results for RSTDT paragraphbreak prediction with gold discourse structures integrated.topic segment boundaries at hand, we use paragraph breaks contained in RST-DTarticles as the ground-truth for training and testing of topic segmentation models.Our results in Table 5.4 show that the quality of discourse structure is positivelycorrelated with enlarged improvements achieved by our proposal. In this case, theupper bound achieved by integrating gold discourse structures can even outperformthe basic model enhanced by context modeling (+Context).5.4.4 Domain Transfer Segment InferenceTable 5.5 presents the performance of simple baselines, previously proposed mod-els and our new approach on the domain transfer task. Similar to the intra-domainsegment inference, the Basic Model+Context approach still achieves the best per-formance across all testing domains except Elements, in which the unsupervisedBayesSeg performs superior. However, our +Discourse strategy still leads to im-provement over the basic model, and achieves comparable performance to the bestmodel (+Context) on Wiki-50 and Cities. We believe that it gives evidence that in-jecting discourse dependency structures has potential to enhance the generality oftopic segmentation models.89Dataset Wiki-50 Cities Elements ClinicalRandom 52.7 47.1 50.1 44.1BayesSeg 49.2 36.2 35.6 57.2GraphSeg 63.6 40.0 49.1 64.6TextSeg 28.5 19.8 43.9 36.6Sector 28.6 33.4 42.8 36.9Transformer 29.3 20.2 45.2 35.6Basic Model 28.7 17.9 43.5 33.8+Context 26.8 16.1 39.4 30.5+Discourse 26.8 16.9 41.1 31.8Table 5.5: Pk (↓) error score on four test corpora for domain transfer experi-ment. Results in bold and underlined indicates the best and second bestperformance across all comparisons. The row highlighted in purple is theresults achieved by our proposal.5.4.5 Efficiency AnalysisTable 5.6 compares the efficiency of the top two models, comparing our proposedapproach (Basic Model+Discourse) against Basic Model+Context. The experimentsfor these systems were carried out on a Nvidia Telsa V100 16G GPU card. We ob-serve that our strategy of injecting discourse dependency structures can improvemodel’s performance on intra-domain and domain transfer setting, but with less in-crease of model size and loss of speed compared to +Context. More specifically,adding our discourse graph modeling component on top of the basic model intro-duces 65% more learnable parameters while the context modeling components inXing et al. [188] cause a 127% parameter increasing. On the other hand, discoursegraph modeling slightly slows down the speed of model training and inference by21% and 7.7% respectively, while making more complex context modeling signifi-cantly slows down the speed by 78% and 46%. Together with the previous resultsabout model’s effectiveness, we can see that our proposed system would be a betteroption in practical settings where efficiency is critical.Additionally, we conduct the same set of experiments for the model with bothcontext modeling module and our proposed discourse structure integration (BasicModel + Context + Discourse). The performance of this model always falls in be-tween +Context and +Discourse individually, but with the worst efficiency measured90# Params ↓ T-Speed ↑ I-Speed ↑Basic Model 4.82M 6.90 35.58+Context 10.93M 1.49 19.23+Discourse 7.97M 5.44 32.85Table 5.6: The efficiency comparison between our proposal and the methodproposed in Xing et al. [188] on the Wiki-Section corpus. These twomodels share the same basic segmentation framework. T-Speed refers thetraining speed as number of batches processed per second during trainingstage. I-Speed refers the inference speed as number of documents pro-cessed per second during inference stage.by model size and speed.5.5 ContributionsDespite the superior effectiveness achieved by neural supervised topic segmentationmodels, they still suffer from limited robustness and transferability due to their re-liance on domain-specific linguistic cues for predictions, while neglecting the moregeneral and important inter-sentential topical consistency. To address this issue, inthis chapter, we consider discourse parsing as an auxiliary task that offers similarinter-sentential coherence signals useful for topic segmentation, echoing observa-tions from Chapter 4. Specifically, we present a discourse-aware neural topic seg-mentation model with the injection of above-sentence discourse dependency struc-tures derived from a state-of-the-art discourse dependency parser. This design en-courages the model to predict topic boundaries based on inter-sentential coherence,aligning with a more comprehensive and interpretable view of topical consistency.Our empirical study across various settings (intra-domain, domain transfer, and ef-ficiency comparison) shows that injecting above-sentence discourse structures to aneural topic segmenter with our proposed strategy can notably improve its perfor-mances on both intra-domain and out-of-domain data, with little increase of model’scomplexity (i.e., model size and speed degradation).In conclusion, this chapter complements Chapter 4 and thus demonstrates thebidirectional synergy between discourse parsing and topic segmentation. Based onsuch synergy, we encode discourse parsing as an appropriate auxiliary task to en-91hance neural topic segmentation in terms of accuracy and robustness, partially an-swering RQ2.92Chapter 6Language Models for DialogueTopic SegmentationThe research work presented in this chapter is currently under review. As the prin-cipal contributor, I am responsible for research question investigation, concept for-mation, model design and implementation, as well as manuscript preparation. Prof.Giuseppe Carenini served as the supervisory author of this work, contributing toresearch question investigation, concept formation, manuscript preparation and in-sightful discussions.6.1 IntroductionAs described in §1.1.3 of Chatpter 1, neural computational methods for topic seg-mentation are typically categorized according to two principal dimensions: (1) su-pervised or unsupervised, and (2) for monologue or dialogue. These two dimen-sions, while conceptually orthogonal, are rarely so in practice. More specifically,with the emergence of expansive corpora derived from Wikipedia, there has been arapid growth in supervised approaches for monologue topic segmentation [10, 45,109, 188]. However, the focus of this chapter, dialogue topic segmentation, fallsinto a different scenario as the existing labeled datasets are merely sufficient formodel evaluation rather than training due to their limited size or potentially noisyorganization. As a result, most previous methods for dialogue topic segmentationfollow the unsupervised paradigm [34, 41, 42, 56, 149, 184, 195].93The performance of early unsupervised methods for dialogue segmentation areusually limited due to the shallow features used for semantic similarity computingbetween utterances, which directly impact the accuracy of topical boundary predic-tion [34, 41, 56]. Although these features are generally straightforward to extract,they merely reveal the similarity between utterances from the lexical aspect (i.e., thedegree of token overlap) and fall short in fully capturing how much two sentences arematched in topic. Recent studies attempt to address this issue by exploring the po-tential of language models extensively pre-trained on massive data [149, 184, 195].Notably, they use such language models as a more reliable alternative to measurethe topical relevance between utterances. Nevertheless, these studies narrowly focuson a single type of language model and limit their evaluation on specific types ofdialogues from relatively confined domains (e.g., customer service in Xu et al. [195]and multi-party meetings in Solbiati et al. [149]). Consequently, it still remainsunclear whether the proposed approaches can maximally unleash the power of pre-trained language models, and if they can be successfully adapted to more complexdialogues or from a wider range of domains.In this chapter, we systematically investigate the effectiveness of various pre-trained language models in two key categories: Language Representation Models(LRMs) and Generative Language Models (GLMs). More specifically, we assessthese models by integrating them into three distinct dialogue segmentation frame-works that align with their unique features. Finally, to more comprehensively in-spect the robustness of these language models, we also introduce a new evaluationbenchmark for this task, which covers seven dialogue corpora with either initialground-truth segment labels or silver standard that we derived from the metadataattached.6.2 The New Evaluation BenchmarkSince prior works conducted model assessment relying on non-open-sourced datasetswith limited domain scopes and potential noise (e.g., SmartPhone and Clothing inTakanobu et al. [158], DAct in Zhang and Zhou [201] and Switchboard with topictransition labels in Soni et al. [152]), the comprehensive and systematic demonstra-tion of the effectiveness and robustness of dialogue topic segmentation systems still94Figure 6.1: Overview of the process for deriving silver standard dialogue topicsegmentation for three conversation corpora in our evaluation benchmark,which were originally proposed for other dialogue understanding tasks.remains an open problem. In order to address this challenge, in this chapter, weintroduce an evaluation benchmark that encompasses a collection of seven publiclyavailable conversational corpora. While four of these datasets were originally an-notated with topical segment boundaries, the remaining three were initially devisedfor alternative dialogue understanding tasks, but we re-purposed them for dialoguetopic segmentation evaluation through synthesizing topic segment boundary labelsbased on their available metadata.In contrast to the evaluation benchmarks employed in prior research, which onlyfocus on a single real-world application, our new benchmark covers a wider spec-trum of domains and data complexity. It can thereby expand the scope of evaluationto better assess the capability of dialogue topic segmentation systems to handle con-versational sessions varying significantly in key attributes (e.g., length, languagestyle, topic coverage).956.2.1 Data CollectionWe introduce the seven conversational datasets included in our evaluation bench-mark. They are:- DialSeg 711 (Synthetic) [195]. This dataset consists of 711 English dialogues arti-ficially synthesized by stitching together various snippets of chat sessions randomlysampled from two multi-turn dialogue corpora: the MultiWOZ Corpus [18], whichis an information-seeking dialogue corpus, and the Stanford Dialog Dataset [35], anopen-domain conversation corpus. Due to the synthetic creation process, this datasetdoes not fully capture naturally occurring topic drifts that may be present in real-world conversations. While the quality of this dataset is relatively lower comparedto other datasets in our benchmark, it still holds value for the research community,particularly in the preliminary stages of developing and evaluating dialogue topicsegmentation approaches.- Doc2Dial [36]. This dataset consists of 4,130 English dialogues grounded in a setof reference documents posted on four public government service websites, whereeach dialogue was created through a combination of automated and human efforts.In particular, a dialogue flow, defined as a sequence of dialogue scenes, was firstgenerated automatically by selecting content from different sections of a source doc-ument. Next, crowdworkers created the actual utterances to fill the dialogue sessionby mimicking the information-seeking interaction between an assisting agent andan end user, strictly following the previously generated flow. While this dataset iswidely applied for tasks like dialogue act identification [36, 37] and response pre-diction [71, 93], it does not explicitly include labels indicating the topic segmentboundaries. Through leveraging the association between each utterance and its cor-responding section in the grounded document, we are able to implicitly derive silverstandard dialogue topic segments as illustrated in Figure 6.1(a).- TopiOCQA [1]. This dataset was originally proposed for conversational questionanswering and consists of 3,920 English conversations formed with information-seeking questions and free-form answers linked to the knowledge derived fromWikipedia. In particular, each conversation started with a question sampled froma subset of NQ [91], a corpus containing genuine information-seeking questions96asked on Google. Then two annotators assumed the roles of a questioner and ananswerer, maintaining the conversation within the context of information-seeking.The questioner had access only to the metadata (section titles) of Wikipedia docu-ments, while the answerer was granted the full text access of documents. Throughpreserving the hyperlinks between Wikipedia articles, the answerer was able to pro-vide answers that are related to, yet extend beyond the current reference document,which contributed to a more natural conversation flow, encouraging topic drifts dur-ing the session while still maintaining a semantic connection to the topics discussedin previous turns. As every utterance in this dataset is tagged with the Wikipediaarticle from which it was generated, we can re-purpose it for dialogue topic segmen-tation by considering consecutive utterances associated with the same article belongto one topic segment (Figure 6.1(b)).- Topical-Chat [47]. This dataset was initially proposed with 10,784 knowledge-grounded open-domain conversations across eight broad subjects. Notably, it wascreated applying four configurations (A-D1) based on the type of information thatthe two Turkers, acting in the roles of a teacher and a participant, could access.Specifically, the knowledge available to the Turkers was organized in a hierarchicalmanner, including a Washington Post article and various fun facts about the entitiesmentioned in that article. To obtain samples with high-quality labels indicating cleartopic shifts from this dataset, we only consider the conversations created under theconfiguration where both Turkers can only view a set of fun facts associated with thesame article. Consecutive utterances tied to the same fun fact are considered as partof the same topic segment (Figure 6.1(c)). Furthermore, we leverage the additionalconversation rating assigned to each sample to filter out those dialogues not assessedas ”excellent” or ”good”. This procedure yields a subset comprising 1,892 dialoguesthat are well prepared for the evaluation of dialogue topic segmentation.- Product Meetings – AMI [20]. This dataset consists of 137 multi-party meetingsspecifically focused on the product design of a new remote control device, com-prising natural conversations between participants, who played different roles in thedesign team. In addition to high-level summaries, each meeting is also equipped1Due to the page limit, please refer to Gopalakrishnan et al. [47] for more details about configu-ration A-D.97Datasets # Dialogues # Turns # Segs # Turns/Seg. Speakers Category real-worldDialSeg 711 639 / 72 27.2 4.9 5.6 – SyntheticCommittee 30 / 6 207.7 7.1 29.4 Multi-Party MeetingAMI 117 / 20 535.6 4.4 122.1 Multi-Party MeetingICSI 50 / 9 819.0 4.5 180.3 Multi-Party MeetingDoc2Dial 3,718 / 414 13.1 3.7 3.5 Dyadic Info. SeekingTopiOCQA 3,340 / 372 13.0 4.2 5.5 Dyadic Info. SeekingTopical-Chat 1,703 / 189 21.6 3.9 5.5 Dyadic Open-domainTable 6.1: Statistics of dialogue datasets in our evaluation benchmark for dia-logue topic segmentation. The sub-table on top contains the datasets pro-posed with the annotation of topic segment boundaries, and sub-table onbottom contains the datasets originally proposed for other dialogue un-derstanding tasks and re-purposed with strategies illstrated in Figure 6.1.#Dialogues here shows how many dialogues are contained in the test / devset, following 90%-10% data split.with the annotations highlighting the main topics discussed during the meeting, aswell as their relevant text spans. These annotations can be directly used as ground-truth topic segment boundaries.- Academic Meetings – ICSI [72]. This dataset consists of 59 multi-party meet-ings happened during weekly academic gatherings held at Berkeley’s InternationalComputer Science Institute (ICSI). Similar to AMI, each meeting comes with an-notations including its respective summaries and the text spans addressing distinctresearch topics covered in the meeting.- Committee Meetings [204]. This dataset consists of 36 multi-party committeemeetings held at the Welsh Parliament and the Parliament of Canada, covering awide range of issues such as public health and public education. Its procedure oftopic segment annotation is identical as AMI and ICSI datasets.6.2.2 Dataset Statistics and ComparisonThe statistics of the datasets included in our evaluation benchmark are presented inTable 6.1, exhibiting several advantages compared to the ones employed in previousliterature:– More Diverse Complexity: Prior research conduct evaluation solely on either syn-98thetic dialogues, where semantic gaps are too easy to spot, or multi-party dialogueswhich are typically extensively long (dialogue length possibly ranges from ˜200to ˜800 turns) and involve asynchronous interactions, making segmentation an ex-tremely challenging task. Our ensembled benchmark addresses this issue by includ-ing three re-purposed corpora featuring real-world dyadic dialogues with intermedi-ate complexity (see # Turns/Seg.), which effectively bridges the difficulty gap seenin earlier studies.– Multi-purpose Setting: Previous evaluation sets are typically focused on a singletype of data resource, such as AMI, ICSI, and Committee, all of which are gatheredfrom meeting resources. Although covering a variety of fine-grained topics, they arestill likely to be similar in language style and session layout owing to their sharedpurpose. Our ensembled benchmark, however, is more diverse in the dialogue objec-tive, additionally including information-seeking Q&A and open-domain dialogues.We hope this inclusive benchmark offers a platform to evaluate a model’s generalityacross different domains/applications, and further helps the exploration to enhancethe model’s robustness.6.3 Language Models for Dialogue Topic SegmentationGiven the scarcity of ground-truth data, most of existing methods for dialogue topicsegmentation follow the unsupervised paradigm. This scenario gives neural lan-guage models the opportunity to provide valuable insights for this task, as theyare usually pre-trained on vast text resources and shown to reliably capture gen-eral sentence topics in an off-the-shelf manner. Generally, these language modelscan be divided into two key categories: Language Representation Models (LRMs)[30, 108] and Generative Language Models (GLMs) [96, 136]. In this section, wedescribe three standard strategies to utilize these two types of language models fordialogue topic segment inference, namely the two-stage approach for LRMs, andthe loss-based approach, as well as the prompt-based approach for GLMs.6.3.1 Language Representation Models (LRMs)Two-stage Approach. The unsupervised dialogue topic segmenters instantiatedwith language representation models (e.g., BERT [30] and RoBERTa [108]) typ-99Figure 6.2: The loss-based topic segmentation workflow for generative lan-guage models.ically adheres to the strategy which can be decoupled into two stages: semanticsimilarity measuring and segment boundary identification [104, 149, 184, 195]. Inthe first step, language models are utilized as text encoders, converting each ut-terance into a fixed-size embedding. Semantic similarity scores between adjacentutterances are then computed based on the cosine or dot product of these embed-dings. In the next step, the standard TextTiling [56] algorithm is applied to predictsegment boundaries based on the similarity scores obtained from last step.Formally, given a dialogue D = [u1, ...,uk] having k− 1 consecutive utterancepairs, a selected language representation model is first leveraged to encode all pairsand get a sequence of semantic similarity scores S = [s1,s2, ...,sk−1], where si ∈[0,1] reflects to what extend two utterances in the ith pair are matched in topic.Then we convert the list of similarity scores into depth scores which indicate thesharpness of local semantic gaps within the dialogue. When a depth score exceedsa certain threshold τ , we identify the corresponding point as a segmentation point.The threshold τ = µS +λ ·σS, where µS and σS are mean and standard deviation ofthe depth score series. λ is a hyper-parameter tuned on the validation set to controlthe granularity of segmentation.1006.3.2 Generative Language Models (GLMs)Loss-based Approach. As proposed in Feng et al. [39], the topical distance be-tween adjacent utterances can be inferred from the contextual interaction betweenwords learned by generative language models. Then the obtained sequence of topi-cal distances between adjacent utterances can be further used as the basis for topicsegmentation inference. In particular, we formalize the topical distance between ut-terances as the difficulty in predicting the next utterance given its preceding, using agenerative language model pre-trained with text generation objective. The underly-ing assumption is that if an utterance is hard to predict correctly from its precedingcontext by a reliable generative language model, it is likely that there is a seman-tic gap between this utterance and its previous context, which further indicates thisutterance may belong to a new topic [39]. As illustrated in Figure 6.2, a dialogueD = [u1, ...,uk] is initially divided into a series of consecutive utterance pairs [(u1,u2), (u2, u3), ..., (uk−1, uk)], where the first utterance serves as the context and thesecond as the response. The context is passed as input to a generative languagemodel, which generates the response with a series of decoding steps. The meanof the token-wise negative log-likelihood (NLL) computed across all steps betweenthe predicted distribution and the golden target response, is subsequently utilizedfor topic boundary inference, where a response prediction with the loss exceeds acertain threshold tuned on the validation set is considered as the start of a new topicsegment.Prompt-based Approach. With the constantly improving emergent capabilities ofgenerative large language models2 (LLMs) [210], the new state-of-the-art perfor-mance on various fundamental NLP tasks (such as information extraction [176],summarization [183], and machine translation [78]) has been achieved through prompt-ing LLMs in a simple and straightforward manner. However, their potential in di-alogue topic segmentation, remains largely unexplored. As such, we include Chat-GPT3 [127], one of the most powerful generative LLMs currently available, as oneof our study objects in this chapter. Yet, instead of applying the loss-based approach2In this chapter, we refer “large” language models as the ones with similar parameter size asGPT-3.5, due to their emergent abilities observed in Zoph et al. [210].3We utilized the version gpt-3.5-turbo-0301 for the implementation. Therefore, we useGPT-3.5 and ChatGPT interchangeably in this study.101Figure 6.3: Overview of the iterative prompting mechanism addressing thediscrepancy of utterance count before and after ChatGPT.described earlier, which requires open-source accessibility of the model, we explorehow to adapt ChatGPT for dialogue topic segmentation in the prompt-based manner.To achieve this, we primarily formulate dialogue topic segmentation as a ques-tion and instruct ChatGPT to produce the answer in an easy-to-process format byusing empirically tuned prompt templates. In order to maximize ChatGPT’s poten-tial for topic segmentation across a range of dialogue lengths within our evaluationbenchmark introduced in §6.2, we design two types of prompting strategies:– End-to-end Prompting: In this method, ChatGPT receives an instruction consist-ing of three essential components: (1) a conceptual explanation of the segmentationtask, (2) a set of rigid rules restricting ChatGPT to structure its output as we ex-pect, and (3) the dialogue content awaiting to be segmented. More details about theprompt template used in this method can be found in Table 6.2. The direct output ofChatGPT is expected in the same format as the input dialogue, with special marksplaced at where topic transitions happen. Notably, we observed in experiments thatChatGPT might not be able to consistently generate the answer maintaining thesame number of utterances as the original input. To tackle this issue, we introducea mechanism to further prompt ChatGPT to keep regenerating the answer until thediscrepancy in utterance count is resolved. Details of this mechanism can be foundin Figure 6.3.– Pair-wise Prompting: Given the token limit (maximum 4096 tokens) of ChatGPT,102Type Content of PromptEnd-to-end Please apply topic segmentation to the dialogue session provided belowby segmenting this dialogue session into a sequence of topically coher-ent pieces following these rules:1. Please ONLY do segmentation by placing ’********’ between turnswhere a topic transition happens.2. Please DO NOT add any new information (such as topic name, speak-ers) to the segmented dialogue session.3. Please DO NOT merge any adjecent utterances, split any utterancesor modify any utterance in the given dialogue session.4. Please be sure the segmented dialogue session you give MUST haveexactly the same number of utterances as before segmentation.5. Please DO NOT change the order of utterances.Here is a ground-truth topic segmentation example for your reference:Before topic segmentation:... ...After topic segmentation:... ...Here is the dialogue session needed to be segmented. It is from the samedataset as the above given example:... ...Pair-wise Given the following [x] utterances pairs, please give me a list of scoresin the range of [0, 1], indicating how semantically coherent the two ut-terances in each pair together. Please only give the list of scores splittedby ’, ’ without any additional text. Make sure the number of score = [x]:Pair 1: (Utterance 1, Utterance 2)Pair 2: (Utterance 2, Utterance 3)Pair 3: (Utterance 3, Utterance 4)... ...Table 6.2: ChatGPT’s prompt templates we applied for the end-to-end prompt-ing and pair-wise prompting approach respectively. Text in red was onlyadded for one-shot setting.103Figure 6.4: The token length statistics of the three meeting corpora includedin our benchmark. The red dashed line represents the input token limit ofChatGPT (gpt-3.5-turbo as the backend model).the end-to-end prompting strategy might not always be applicable, as some lengthydialogues in the three meeting corpora of our evaluation benchmark cannot fit intoa prompt and be fed to ChatGPT (Figure 6.4). To address this challenge, we intro-duce an alternative prompting strategy in which ChatGPT is instructed to score abatch of successive utterance pairs based on their semantic relatedness. In this way,even lengthy dialogues exceeding the maximum token limit can still be broken intomultiple batches of utterance pairs, each adhering to the length limit. The ”seman-tic similarity scores” provided by ChatGPT for all consecutive utterance pairs in adialogue are then utilized in conjunction with TextTiling, as described in §6.3.1, toinfer topic segment boundaries. More details about the prompt used for this methodcan be found in Table 6.2 as well.6.4 Experimental Setup6.4.1 Evaluation MetricsWe apply four standard metrics to evaluate the performances of our proposal andbaselines. They are: Pk error score [14], WinDiff (WD) [130], Pr [43] and F1 score(macro). As introduced in Section 1.2, Pk, WD and Pr are all calculated based onthe overlap between ground-truth segments and model’s predictions within a certainsize sliding window. Since they are penalty metrics, lower score indicates betterperformance. F1 is the standard armonic mean of precision and recall, with higher104Model Training Size Arch. Param.BERT 16GB text Tr. Encoder (LRM) 110MRoBERTa 160GB text Tr. Encoder (LRM) 125MToD-BERT 100K dialogs Tr. Encoder (LRM) 110MDSE 100K dialogs Tr. Encoder (LRM) 110MConveRT 727M dialogs Tr. Encoder (LRM) 29MDMI ˜7M dialogs Tr. Encoder (LRM) 125MBART 160GB text Tr. S2S (GLM) 140MGPT-2 40GB text Tr. Decoder (GLM) 345MDialoGPT 147M dialogs Tr. Decoder (GLM) 345MGPT-3.5 570GB text Tr. Decoder (GLM) 175BTable 6.3: Survey of pre-trained language models included in this study. S2S:Seq2Seq, Tr.: Transformer.scores indicating better performance.6.4.2 Language Models to CompareWe conduct experiments and analyses on a comprehensive selection of pre-trainedlanguage models. Table 6.3 shows detailed statistics on each model and specifiesif the model is LRM or GLM. All language models included in this study are usedto instantiate their respective unsupervised dialogue topic segmentation frameworksdiscussed in §6.3.Language Representation Models (LRMs) We compare BERTbase [30], RoBERTabase[108] against four recently proposed language representation models sharing a sim-ilar architecture (i.e., Transformer encoder [163]) with BERT and RoBERTa, buttrained on large-scale dialogue corpora. They are: (1) TOD-BERT [177], whichfine-tunes BERT on 9 task-oriented dialogue corpora by optimizing a contrastive re-sponse selection objective via treating an utterance and its dialogue context as pos-itive pair; (2) DSE4 [206], which fine-tunes both BERT and RoBERTa on the sameset of task-oriented dialogue corpora as TOD-BERT, but with a simpler yet moreeffective contrastive learning scheme strictly using consecutive utterances as posi-tive samples; (3) ConveRT [57], which is a Transformer encoder-based model pre-4We chose the dse-bert-base version checkpoint.105trained with a compact dual-encoder mechanism on 727M (input, response) pairsfrom Reddit; (4) DMI [144], which fine-tunes RoBERTa on 7.5˜10% of 727M Red-dit data by minimizing a structure-aware discourse mutual information loss.Generative Language Models (GLMs) We consider four pre-trained models in-cluding: (1) BART [96], a Transformer Seq2Seq language model pre-trained onlarge corpora of generic web-crawled English text. Although not specifically trainedon dialogue corpora, it still shows strong performance on generation tasks such asdialogue summarization [25, 95, 207]; (2) GPT-25 [136], a Transformer decoder-based language model trained on a dataset of 8 million web pages via optimizingthe causal language modeling (CLM) objective; (3) DialoGPT [203], which fur-ther tunes GPT-2 on conversations from Reddit by using the original loss functionsof GPT-2; (4) GPT-3.5 [127], which is obtained through fine-tuning GPT-3 [17]towards better alignment with human feedback. In experiments, we keep the textgeneration temperature = 0.7, as this is the default setting in the ChatGPT web in-terface.6.5 Results and AnalysisIn this section, we present our analysis and empirical findings of language models’effectiveness on unsupervised dialogue topic segmentation, specifically focusing onthe division between meeting and non-meeting dialogues.The Impact of Pre-training on Dialogues. Table 6.4 showcases experimental re-sults for non-meeting corpora that suggest a noticeable performance enhancement inboth LRMs and GLMs pre-trained on dialogue corpora. Generally, all four languagerepresentation models pre-trained on dialogue corpora consistently outperform thetwo vanilla models (BERT and RoBERTa) by a considerable margin across three outof four evaluation sets (DialSeg 711, Doc2Dial, Topic-Chat) in our benchmark, ex-cept TopiOCQA, which contains utterances extensively reusing formal expressionsand jargons from Wikipedia. More specifically, ToD-BERT and DSE, both of whichare trained on an identical collection of nine task-oriented dialogue corpora, consis-tently outperform ConveRT and DMI, both trained on Reddit conversations three5We use medium version of GPT-2 and DialoGPT.106Synthetic Open-DomainDatasets DialSeg 711 (4.9) Topic-Chat (3.9)Model Pk ↓ WD ↓ Pr ↓ F1 ↑ #Seg. Pk ↓ WD ↓ Pr ↓ F1 ↑ #Seg.BERT 40.66 42.89 36.83 59.32 3.8 51.30 54.66 43.05 50.28 3.0RoBERTa 44.43 46.63 37.26 55.64 4.5 51.95 55.63 42.85 50.04 3.2ToD-BERT 21.74 24.58 18.82 82.13 4.4 50.85 54.95 41.22 51.72 3.3DSE 27.41 30.85 23.41 76.97 4.7 49.13 53.70 39.70 53.74 3.4ConveRT 33.70 36.97 28.62 70.27 4.7 50.07 54.56 40.37 53.72 3.5DMI 36.91 39.56 31.56 66.55 4.4 50.88 54.21 41.07 52.65 3.3BART 49.81 52.99 44.89 47.95 4.4 51.15 55.12 45.22 48.95 2.9GPT-2 48.77 51.14 44.60 49.65 3.9 52.32 56.08 45.82 48.07 3.2DialoGPT 48.52 50.81 44.45 49.99 3.8 50.54 54.53 42.61 50.36 3.1GPT-3.5 14.64 17.85 10.79 86.76 5.4 47.61 52.39 39.35 53.55 3.6GPT-3.5 (one-shot) 6.92 7.85 7.04 93.39 3.9 43.36 46.45 40.81 52.31 2.6GPT-3.5 (pair-wise) 31.70 34.89 27.77 70.72 4.9 53.47 58.16 42.79 50.08 3.6Information-SeekingDatasets Doc2Dial (3.7) TopiOCQA (4.2)Model Pk ↓ WD ↓ Pr ↓ F1 ↑ #Seg. Pk ↓ WD ↓ Pr ↓ F1 ↑ #Seg.BERT 50.38 58.82 37.55 50.26 4.4 46.57 49.11 42.32 52.18 3.1RoBERTa 49.07 57.90 36.54 53.16 4.4 47.51 50.10 42.34 50.93 3.4ToD-BERT 47.09 55.14 33.83 57.18 4.6 49.98 51.85 42.19 51.46 3.5DSE 49.30 56.25 34.52 57.11 4.7 50.99 53.39 41.02 53.98 3.9ConveRT 49.06 56.83 34.81 56.40 4.8 53.01 55.15 43.75 51.99 3.9DMI 47.23 55.48 33.92 56.97 4.7 49.41 51.62 41.63 53.89 3.7BART 48.88 50.92 41.24 46.41 1.9 49.58 52.48 43.73 43.83 3.7GPT-2 54.14 63.81 42.35 51.12 5.2 52.28 54.31 46.05 48.45 3.3DialoGPT 51.50 53.44 41.50 52.41 1.9 50.82 52.72 46.49 46.68 3.1GPT-3.5 43.64 47.25 28.86 66.27 3.9 54.05 63.32 37.28 60.30 7.5GPT-3.5 (one-shot) 43.22 46.13 28.41 67.69 3.3 42.71 44.90 36.36 62.45 3.7GPT-3.5 (pair-wise) 50.93 57.10 33.25 58.55 4.8 55.02 60.99 39.64 56.08 6.0Table 6.4: Results on four non-meeting datasets included in our evaluationbenchmark (see §6.2). The solid horizontal line divides the table into anupper section for LRMs and a lower section for GLMs. Further, dashedhorizontal lines separate models in the upper section into models w/ or w/ofine-tune on dialogue corpora, and models in the lower section based ontheir parameter size, either moderate or large (i.e., ChatGPT). Bolded andunderlined results indicate the best and the second best performance acrossall comparisons. Results highlighted in grey indicate best performancewithin the upper/lower section. #Seg. is the average segment frequency ofmodel’s hypothesis.107orders of magnitude larger. A similar pattern is observed among generative languagemodels as well, with DialoGPT consistently outperforming GPT-2, a model witha similar architecture but not fine-tuned on dialogues.Nevertheless, Table 6.5 gives a contrasting story for meeting dialogues suchas ICSI and AMI. First, due to the extreme sparse segmentation of these meetingsamples, both Pk and WinDiff become unreliable as they tend to yield lower errorscores to substantially undersegmented predictions (see DialoGPT on ICSI andBART on AMI). Therefore, our analysis on meeting dialogues is primarily basedon Pr, as it effectively addresses the aforementioned inadequacy of Pk and WinDiff[43]. Interestingly, RoBERTa and GPT-2 in contrast achieve superior performanceon ICSI and AMI compared to their rivals pre-trained on dialogues, which could beplausibly explained by the inherent language formality of ICSI and AMI, with lessfragmented and informal expressions compared to casual daily conversations.Based on the findings above, we would recommend language models pre-trainedon dialogue corpora to instantiate unsupervised dialogue segmenters for the chatsessions with simple structures and colloquial language, and vice versa.LRMs vs. GLMs. As shown in Table 6.4 and 6.5, LRMs generally outperformGLMs following the loss-based segmentation framework (§6.3.2). This trend is par-ticularly evident across the four similarly complex evaluation datasets in Table 6.4,where even the worse performing LRM surpasses the best performing GLM (Chat-GPT is not in this scope and we will discuss it separately later). It is also worthnoting that the average segment frequencies predicted by LRMs align more closelywith the ground-truth ones, as the largest discrepancy in segment frequency (com-puted as the absolute difference between the reference’s #seg. and the hypothesis’s#seg.) for each evaluation set is always attributed to one of the GLMs.Therefore, we would suggest utilizing LRMs over GLMs for unsupervised dia-logue topic segmentation owing to their notable superior performance and segmentscale.The Performance of ChatGPT. As shown in Table 6.4, ChatGPT for dialogue topicsegmentation via end-to-end prompting significantly outperforms all other languagemodels, as well as the pair-wise prompting across the four non-meeting datasets.This demonstrates that (1) by directly prompting ChatGPT with clear task defini-108Datasets ICSI (4.5) AMI (4.4)Model Pk ↓ WD ↓ Pr ↓ F1 ↑ #Seg. Pk ↓ WD ↓ Pr ↓ F1 ↑ #Seg.BERT 43.29 45.90 45.16 50.53 3.0 48.78 60.79 43.12 50.26 9.4RoBERTa 37.55 40.15 40.10 50.94 2.9 44.36 57.10 40.00 51.42 10.3ToD-BERT 42.88 44.90 46.29 50.37 2.3 48.36 57.21 45.28 49.95 7.3DSE 49.15 57.99 43.39 49.98 7.6 47.21 52.50 48.42 49.64 4.7ConveRT 42.38 45.48 48.16 49.95 2.7 48.38 54.71 47.24 49.64 6.3DMI 52.97 65.21 44.35 50.37 7.2 49.12 55.48 47.20 49.76 6.0BART 41.66 44.11 46.37 50.02 2.2 38.63 39.79 48.88 50.03 1.2GPT-2 40.14 41.97 44.24 50.21 2.2 54.84 67.18 45.35 50.15 11.9DialoGPT 39.66 39.89 49.20 49.86 0.5 42.78 45.59 46.52 49.64 3.3GPT-3.5 (pair-wise) 49.14 57.69 43.70 50.42 7.5 49.25 58.19 45.27 50.03 7.8Datasets Committee (7.1)Model Pk ↓ WD ↓ Pr ↓ F1 ↑ #Seg.BERT 51.68 59.43 42.29 49.41 11.9RoBERTa 52.22 59.36 42.37 50.36 11.6ToD-BERT 51.45 58.99 42.33 52.14 12.8DSE 51.23 58.73 41.44 51.48 12.1ConveRT 51.67 60.30 43.04 51.04 12.3DMI 52.15 59.77 43.58 50.72 11.7BART 57.63 74.72 45.60 48.63 22.4GPT-2 45.72 46.31 49.51 48.82 2.3DialoGPT 41.34 43.56 44.58 49.66 4.2GPT-3.5 (pair-wise) 41.38 42.29 44.10 50.35 3.5Table 6.5: Experimental results on the three meeting datasets included in ourevaluation benchmark introduced in §6.2.tion and instructions, it can accurately and consistently identify topic segments indialogues of diverse purposes; (2) it is more effective to utilize ChatGPT to directlyinfer topic segments rather than have it perform the intermediate step to infer pair-wise semantic relatedness between adjacent utterances. The limited effectivenessof prompting ChatGPT in the pair-wise fashion can also be observed in Table 6.5,where it achieves slightly better performance than loss-based topic segmenters in-stantiated by other GLMs, but falls short when compared to LRMs.Furthermore, we also investigate the effect of the in-context learning scheme109by integrating a randomly selected in-domain datapoint with its ground truth intothe prompt and feeding it to ChatGPT. As shown in Table 6.4, incorporating in-context learning with one-shot setting (see GPT-3.5(one-shot)) can further enhanceChatGPT’s performance on three out of four datasets.To summarize, when the target dialogue does not exceed ChatGPT’s input lengthlimit, ChatGPT within an end-to-end prompting paradigm is the SOTA for dialoguetopic segmentation. However, for longer dialogues, LRMs still appear to be the mosteffective solutions.6.6 ContributionsSome previous studies have already sought to use pre-trained large language models(LLMs) as a more reliable source to measure the topical coherence between utter-ances. However, they generally concentrate on a single type of LLMs and restricttheir evaluations to specific dialogue types from relatively niche domains. As a re-sult, questions still remain about the full potential of these approaches in harnessingthe power of pre-trained LLMs and their adaptability to more complex dialogues orbroader domains.To address the above questions, in this chapter, we contribute a systematic com-parison between two types of language models: language representation models(LRMs) and generative language models (GLMs), in the context of dialogue topicsegmentation. In order to make our investigation thorough and comprehensive, wealso introduce an evaluation benchmark for dialogue topic segmentation, which ex-pands upon the domains and complexity featured in previous research. The experi-mental results and analysis show that (1) pre-training language models on dialoguescan particularly benefit topic segmentation for dialogues with relatively simple struc-tures and colloquial language; (2) For language models with comparable moderatesize, LRMs generally outperform GLMs; (3) Prompting ChatGPT in an end-to-endmanner exhibits top performance on dialogues with length within its inputs limit,while for longer dialogues, LRMs are still the best choice.In conclusion, this chapter offers substantial evidence that PLMs can serve assuitable auxiliary tasks, providing reliable coherence signals for unsupervised topicsegmentation, thus partially answering RQ2. Furthermore, we discover that different110types of PLMs behave variably across distinct dialogue text genres. This observationoffers valuable insights, particularly in light of the burgeoning growth of LLMs.111Part IIIBeyond Text: Topic Segmentationfor Videos112Compared to topic segmentation for textual content (monolgues and dialogues),topic segmentation for video can be seen as a multi-modal endeavor that seeks todivide extended portions of videos into shorter segments, each centering around aparticular theme or subject. This task remains under-explored but is undeniablycritical, holding substantial potential benefits in the modern media landscape. It canenhance real-time interactions between video creators and their viewers, especiallyon popular live-streaming platforms (e.g., Live-Youtube, Behance). Therefore, inPart III, we mainly concentrate on expanding the concept of topic segmentation tothe realm of videos (RQ3). More specifically, we aim to develop accurate and robustneural solutions tailored to the unique challenges posed by video content.Drawing from the insights on neural topic segmentation for text (discussed inPart I and II), in Part III, we first frame video topic segmentation as a sequence la-beling challenge, especially when textual data such as video transcripts are available.In practice, we employ a hierarchical neural sequence labeling structure as the coreframework for our proposed video segmenting approach. Building upon this, wemake the model to operate in a multi-modal environment by integrating visual cuesfrom video frame sequences via an additional cross-modal attention mechanism.In order to ensure that our proposed video topic segmentation method remainseffective even for videos with various lengths, we further introduce a dual-contrastivelearning approach. This approach particularly aims to enhance models initiallytrained on shorter videos, making them adaptable and effective for longer videoswhich might display more subtle visual transitions. Through experiments conductedon corpora covering videos with various lengths, our introduced multi-modal ap-proach, coupled with the dual-contrastive learning mechanism, demonstrates a note-worthy improvement over conventional segmentation techniques for videos (i.e.,shot/scene segmentation), showcasing its superior accuracy and adaptability acrossdifferent scenarios.113Chapter 7Multi-Modal Video TopicSegmentation withDual-Contrastive DomainAdaptationThe research work presented in this chapter has been peer-reviewed and publishedat MMM 2024 (30th International Conference on Multimedia Modeling) [192]. Iwas the principle contributor and the work was mainly done during my internship atAdobe. As the principal contributor, I am responsible for research question investi-gation, concept formation, model design and implementation, as well as manuscriptpreparation. Prof. Giuseppe Carenini and Dr. Quan Tran served as the supervi-sory authors of this work, contributing to research question investigation, conceptformation, manuscript preparation and insightful discussions.7.1 IntroductionVideo Topic Segmentation aims to break stretches of videos into smaller segmentsconsisting of video frames or clips consistently addressing a common topic. As anexample given in Figure 7.1, video topic segmentation does the job of segmenting114Figure 7.1: An example livestream video about making a choking victimposter, sampled from the creative livestream platform Behance. The leftside of the figure illustrates the video’s timeline together with transcriptafter topic segmentation, consisting of 8 topical-coherent segments in dif-ferent colors. Words indicating a segment’s topic are highlighed in thetranscript. The right side of the figure presents a video frame randomlysampled from each segment.a creative livestream video (sampled from the livestream platform Behance1) into asequence of topical-coherent video pieces (Seg1 – Seg8), placing a boundary inthe video’s timeline where a topic transition happens. This task can offer substan-tial advantages for both human-to-human and human-to-system interactions in con-temporary social environments, such as facilitating real-time engagement betweenstreamers and their audience on live streaming platforms [40]. In particular, the rel-atively coarse-grained temporal structure of the input video produced by video topicsegmentation is shown to not only make the whole video easier to understand butalso help the audience navigate to their interested content as in Fraser et al. [40].1https://www.behance.net/live115More importantly, it can serve as a upstream task and substantially benefit other keyvideo understanding tasks such as video summarization [100, 135], and query-drivenvideo localization [94, 180].Early computational models proposed to segment videos primarily focused onvideo shot/scene detection by merely exploiting surface features related to the vi-sual modality, such as spatiotemporal features or the colors encoded in video frames[26, 138, 139, 142]. These approaches typically measure the temporal similarityalong a video’s timeline to predict shot/scene boundaries in the input video. Despitethe difference in definition from shot and scene segmentation (discussed in §7.2),the task of video topic segmentation focuses more on topic-related semantics in thevideo, which is not necessarily aligned with the visual changes. As the examplepresented in Figure 7.1, the visual background remains similar in Seg3 – Seg7for a considerable time, even though the streaming topic has drastically changed.Moreover, these previously proposed methods for video segmentation are mostlyfocused on short videos, whose shots/scenes are well-organized following simplepatterns and with visual changes easy to identify [53, 69, 73]. These distinctivecharacteristics of short videos may be explicitly leveraged in the model design orlearned by supervised frameworks, and thus cause these models suffering from poortransferability when applied to more complex video contents which have longer du-rations and less explicit visual changes (e.g., documentaries, instructional livestreamvideos).To address the aforementioned issues, in this chapter, we first propose a sim-ple yet effective multi-modal model for video topic segmentation, which can takeboth the aligned video transcript and frames as input. This ability considerably en-hances the model’s performance, as textual and visual features can work togetherto more comprehensively reflect the input video’s topic-related semantics [9, 70].Similar to the formulation used for text topic segmentation [10, 88, 109, 188], wetreat video topic segmentation as a sequence labeling task and introduce a neuralmodel equipped with a cross-modal attention mechanism going beyond simple fu-sion to effectively integrate textual and visual signals in a complementary manner.We initially conduct intra-domain experiments by training and testing our proposedsegmenter on a newly collected YouTube corpus. This reasonably proportioned cor-pus consists of short videos with diverse topic coverage and high-quality human116labeling. Empirical results show that our approach outperforms a set of baselinevideo segmenters by a substantial margin.To further adapt our proposed model trained on YouTube videos to longer videosususally with more complex and diverse semantic and visual organizations, we pro-pose an unsupervised domain adaptation strategy empowered by sliding windowinference and a dual-contrastive learning scheme. Experimental results on two longvideo corpora demonstrate that the model’s performance (generalizability) can besignificantly improved under the cross-domain scenario, by applying this novel do-main adaptation approach.7.2 Related WorkShot and Scene SegmentationShot and Scene Segmentation are two tasks closely relevant to video topic segmen-tation but with narrower scopes or more limited usage scenarios. A shot is formallydefined as a series of frames captured from the same camera over an uninterruptedperiod of time [148]. Consequently, most previously proposed shot segmentationmethods utilize only visual modality, aiming to automatically group frames in aninput video into a sequence of shots. On the other hand, a scene is defined as a muchmore semantically complex unit than a shot. A scene typically refers to a sequenceof semantically related shots together representing an event defined by elements likeactions, places, characters, which are features merely exist in story-telling narrativevideos such as movies [62]. As a result, previous scene segmentation methods aremostly trained and evaluated only on movies, with the goal of grouping consecutivemovie shots into the scenes by maximizing the scene visual consistency assessed byspatiotemporal features or the color similarity between shots [138, 139, 142]. Sim-ilar but more difficult than shot or scene segmentation, our work focuses on videotopic segmentation, which can be deemed as an extension of text topic segmentationwith “topic” usually defined as a relatively self-contained collection of semanticallyclose information. It is worth emphasizing that the precise definition of a topic ishighly dependent on the context and domain of a video, rather than being arbitrarilylimited to the concept of shots or scenes defined in previous works. For example, ina film context, a “topic” might align with a single movie scene or even span multiple117movie scenes. Conversely, in a streaming scenario, adhering to the scene definitionfor “topic” would result in many videos consisting of just one topic/scene (the sameplace and character over time), even though the author may discuss many distincttopics throughout the streaming. From a machine learning perspective, the more dy-namic nature of shot/scene segmentation favours the visual signal, while in our task,the video topic segmentation relies more heavily on the semantic signals carried bynatural language [9, 70].Therefore, prior approaches devised for shot/scene segmentation solely use vi-sual changes to detect shot/scene boundaries, which is inadequate for our video topicsegmentation. It motivates us to propose a video topic segmenter by extending a texttopic segmentation framework and integrating visual frames as auxiliary signals.Contrastive LearningContrastive Learning algorithms attempt to learn data representations by enlargingthe distance between dissimilar samples and meanwhile minimizing the distancebetween similar samples with contrastive loss functions [161]. In practice, con-trastive learning techniques have gained significant attention and been widely ap-plied to various tasks in the domain of computer vision and natural language pro-cessing. Among these tasks, contrastive learning has particularly shown promisewhen applied to various domain adaptation strategies (e.g., unsupervised adaptation,test-time adaption, on-target adaptation) in either uni-modal or cross-modal settings[23, 82, 83, 85]. Recent works on using contrastive learning for video domain adap-tation mainly focus on transferring models from source to target domain, where bothdomains consist of videos of similar lengths and characteristics [85, 143]. In con-trast, our work attempts to adapt the model trained on short videos, which are com-paratively easier to obtain and label, to the low-resource domain containing longvideos that exhibit more subtle visual changes, by utilizing the semantic overlapswithin a single modality or between two modalities to guide the contrastive learningprocess.1187.3 Neural Video Topic Segmentation7.3.1 Problem DefinitionInspired by recently proposed neural-based supervised approaches for topic segmen-tation for text, we similarly frame video topic segmentation as a sequence labelingtask, with sentences in the input video transcript as the smallest units awaiting labelassignment. More precisely, given the input video sample consisting of (1) the tran-script as a sequence of sentences with offsets indicating the start/end time of eachsentence, and (2) the running video processed as a sequence of video frames witha specific timestamp associated with each frame, our model will predict the binarylabel for each transcript sentence to indicate if the sentence indicates a topic segmentboundary or not. Formally,Given: A video v with its transcript Tv as a sequence of sentences {s1,s2, ...,sn}along with start time offsets {b1,b2, ...,bn} and end time offsets {e1,e2, ...,en}, aswell as the video frames Xv = {x1,x2, ... ,xm}, where a single frame xi has timestampti.Predict: A sequence of labels {l1, l2, ..., ln−1} assigned to the sequence of transcriptsentences, where l is a binary label, 1 means the corresponding sentence overlaps avideo topic segment boundary, 0 otherwise. We do not predict the label for the lastsentence sn, as it is by definition equal to 1, i.e., the end of the last segment.7.3.2 Model ArchitectureFigure 7.2 illustrates the detailed framework of our proposed video topic segmenter,which is similar in architecture to TextSeg [88]. It comprises two hierarchicallylinked encoding layers: one as text encoder for contextualized encoding within asentence (orange in Figure 7.2) and the other for contextualized encoding betweensequence units (green in Figure 7.2). To allow both textual and visual modalitiescontribute complementarily to the model’s prediction, we add a frame encoder mod-ule (blue in Figure 7.2) and a cross-modal attention mechanism (red in Figure 7.2).This design choice extends the framework to the multi-modal setting, accepting nowboth the video transcript and frames as input and making prediction of topic segment119Figure 7.2: Left: The overall architecture of our proposed multi-modal videotopic segmentation model, with four sub-modules (in §7.3.2) coded indifferent colors. Right: The detailed illustration of cross-modal attention.boundaries based on them.The text encoder module Et yields low-level features for sentences in the videotranscript. Different from the proposal in Koshorek et al. [88] using the BiLSTM+ attention as the backbone for text encoder, here we adopt the pre-trained vanillaBERT [30] in accordance to its achieved superiority on text segmentation observedin Xing et al. [188] and Lukasik et al. [113]. Parallel to the text encoder, a frameencoder E f is introduced to extract features for visual signals with the standardpretrained ResNet-182 [55], a member of the ResNet family which is still a popularchoice in recent cutting-edge research [178]. Formally, given a transcript sentence siwith its time interval as [bi,ei] and a set of video frames Xi = {xi1, ...,xim} associatedwith this sentence, where each frame (e.g., the kth frame) in the set has timestampt ik ∈ [bi,ei], we can obtain the sentence representation tri =Et(si) and the set of framerepresentations FRi = { f ri1, ..., f rim}, where f rik = E f (xik).Next, we propose to use a cross-modal attention mechanism to produce a text-2During the preliminary phase of our study, we assessed the performance of various text encoders(including BERT, RoBERTa, ALBERT, S-BERT), and visual encoders (including ViT-B16, ViT-B32,ViT-L14, ResNet-18) in conjunction with the unsupervised X-Tiling framework for video segmenta-tion. It is observed that BERT and ResNet-18 achieved the best effectiveness as text and visual encoderrespectively.120aware visual representation for each sentence, rather than obtaining the visual rep-resentation by naively operating mean-pooling over the set of frame representationscovered by the sentence interval. This design is motivated by the observation thattranscript sentences and video frames are in an one-to-many relation, and framessharing more semantics with the text should be given more attention weights for vi-sual representation generation. Irrelevant frames sharing no or little semantics withthe text may negatively affect the quality of the fused multi-modal representationpassed to the subsequent module [173].In practice, the cross-modal attention module basically adopts the standard scaleddot-product attention function proposed in Vaswani et al. [162]. With the tran-script sentence representation tri and its corresponding frame representation setFRi = { f ri1, ..., f rim} as query and key (value) input respectively, we compute thetext-aware visual representation vri as:vri = AiVi, (7.1)Ai = so f tmax(qiKTi√dk) (7.2)where qi ∈ R1×dk , Ki ∈ Rm×dk , and Vi ∈ Rm×dk denote the query vector, key andvalue matrices generated by passing the sentence representation and frame represen-tations through three parallel feedfoward layers, namely Q-FFN, K-FFN and V-FFNrespectively. More formally, we have qi = Q-FFN(tri), Ki = K-FFN(FRi), Vi = V-FFN(FRi), where K-FFN and V-FNN share the same parameters and thus produceidentical key and value matrices following the multi-modal co-attention mechanismsin Lu et al. [112] and Kumar et al. [90].Then all the obtained text-aware visual representations {vr1, ...,vrn} are con-catenated with the corresponding sentence representations {tr1, ..., trn} and fed intoa BiLSTM layer which performs contextualization and returns hidden states. Next,a multilayer perceptron (MLP) followed by Softmax serves as a topic boundary pre-dictor to make binary predictions regarding the input hidden states according to athreshold τ tuned on the validation set. More specifically, a transcript sentence withits output probability over τ will be labeled 1 and predicted as falling on a segmentboundary. The whole model is optimized by minimizing the cross-entropy loss.121We train and test this model on a newly collected YouTube corpus (introducedin §7.5) and empirically verify its in-domain effectiveness (results are reported inTable 7.2). We leverage it as the source model in the next section to help deliver thedomain adaptation strategy coupled with our proposed multi-modal framework.7.4 Long Video AdaptationWith the prevalence of (live-)streaming platforms, the urge for topically segment-ing videos posted on these platforms has recently arisen. Different from the videoswith careful pre-editing and segment labeling provided by their creators, such asvideos on YouTube, videos on (live-)streaming platforms (e.g., documentaries onBBC and creative livestream videos on Behance) are usually extremely long, withsparse visual changes, and more importantly, time-consuming to obtain segment an-notations. Thus, it is impractical to learn a fully-supervised model on such videos,and the segmentation model described in §7.3.2 (source model) trained on shortYouTube videos (source domain) is expected to experience performance degrada-tion when applied to these long videos (target domain). In order to adapt the sourcemodel to the above-described target domain, we propose to equip it with two strate-gies, namely Sliding Window Inference (§7.4.1) and Dual-Contrastive Adaptation(§7.4.2).7.4.1 Simple Sliding Window InferenceDue to the length discrepancy between videos from the source and target domains,directly applying the source model to the target input taking the full transcript sen-tence sequence as input is observed to cause extremely sparse boundary predictions[45]. Therefore, we propose to calibrate the input length by first breaking the longtarget input into snippets by a sliding window with the size consistent with the sourceinput length (i.e., the average length of the YouTube videos). After applying thesource model on every snippet and aggregating outputs of all snippets, we eventu-ally make binary prediction for each sentence if its aggregated probability exceedsthe threshold τ pre-tuned on the source domain. Formally, given a long video inputfrom the target domain with length = n and a fixed window size = k (n≫ k), we cancreate n− k+1 snippets {S1, ...,Sn−k+1} where each snippet consists of k consecu-122Figure 7.3: An illustration of updating cross-modal attention module withdual-contrastive adaptation.tive sentences by sliding the window over the input with the stride of 1. As a result,each transcript sentence sm can be covered by up to k snippets. Once we apply thesource model to all snippets, we can obtain multiple probability predictions for sm.We then aggregate these predictions associated with sm by taking average:p¯m =1kk∑i=1pSim (7.3)Finally, we predict that sm falls on a segment boundary if p¯m > τ .7.4.2 Dual-Contrastive AdaptationTo transfer the pre-trained source model to the target domain while still preserv-ing its performance on its own domain, we follow a more sophisticated unsuper-vised domain adaptation paradigm, which leverages both labeled source data andunlabeled target data. Specifically, we fix the frame and text encoder (Et and E f )while updating the rest of the model in two steps, where the first step updates themodel on unlabeled target data using two contrastive learning objectives namelyintra-modal contrastive loss and cross-modal contrastive loss, while the second stepupdates the model on source data with the supervised training scheme described in§7.3.2. As the overview of the first step shown in Figure 7.3, given a collectionof the paired sentence representations and their corresponding frame representation123sets {tri,FRi}bi=1 in a training batch (size=b) produced by the text and frame en-coder from the target domain, we need two distinct projection heads to map thesesentence/frame representations into a shared space. Here we use Q-FFN and K-FNN (which shares parameters with V-FNN) in the cross-modal attention to serveas projection heads for textual and visual modality respectively. Thus we have:qi = Q-FFN(tri),Ki = K-FFN(FRi) (7.4)where qi and Ki = {ki1, ...,kim} denote the projected sentence embedding and theset of projected frame embeddings covered by the sentence. As frames covered bythe same sentence are more likely to share similar semantics, we pull the framesattached to the same sentence closer and push the ones from different sentences farapart, by minimizing the intra-modal contrastive loss for visual modality defined as:lintra =−b∑i=1logexp(k˜i · k˜i)/τ∑bj=1 exp(k˜i · k˜ j)/τ(7.5)where k˜i∈R FRi and τ is the hyper-parameter of temperature. To learn the semanticrelation between modalities in the target domain, we first obtain the visual repre-sentation matched with a sentence by averaging all frames covered by the sentence,and then bring semantically close sentence-visual pairs together and push away non-related pairs by minimizing the cross-modal contrastive loss formulated as:lcross =−b∑i=1logexp(qi ·MP(Ki))/τ∑bj=1 exp(qi ·MP(K j))/τ(7.6)where MP denotes Mean-Pooling operation. Then the total dual-contrastive loss canbe formed as:l = lintra+ lcross (7.7)The above dual-contrastive learning phase is followed by the second step. In thisstep, the model is further trained on the labeled source data again to leverage thetarget-domain signals learned by the parameters in cross-modal attention for bound-ary prediction. This step also ensures the model adapted to the target domain stillpreserves some level of effectiveness when applied to videos from the source do-124Dataset YouTube BBC Behance# of Vids. 5,422 11 575# of Tokens/Vid. 1,411 2,868 11,554# of Sents./Vid. 108 216 1,287# of Segs./Vid. 6.7 29.1 5.2# of Tokens/Seg. 209 99 2,229# of Sents./Seg. 16 8 248# of Frames/Sec. 10 4 4Avg. Length 0:09:30 0:45:02 2:07:51Table 7.1: Statistics of the three corpora used for training and testing in Sec-tion 7.5.main.7.5 Experimental SetupIn order to evaluate the effectiveness and generality of our proposed video topic seg-mentation model (in §7.3), we experimentally compare it against multiple baselineson two different settings, namely Intra-Domain Segment Inference and Cross-Domain Segment Inference. We also use cross-domain segment inference as thetestbed for our proposed long video adaptation strategy (in §7.4), which is anotherkey contribution in this work.7.5.1 Intra-Domain Dataset – YouTubeFor intra-domain segment inference, we train and test models with the data from thesame domain (corpus). Due to the lack of large-scale human-annotated dataset inthe field of video segmentation, we collect a novel corpus consisting of 5,422 user-generated videos randomly sampled from YouTube. The video category distributionis reported in Figure 7.4. During the video collecting process, we applied filteringcriteria to eliminate unsuitable samples to ensure the quality of the dataset to con-struct. These criteria include constraints on video length (> 100s), word count (>0.5 word/second on average), chapter durations (all chapters with length > 10s), andsentence length (all sentences with length < 60 tokens) inspired by [19]. Each video125Figure 7.4: Histogram of YouTube video corpus category distribution (y-axis:number of videos).in this corpus is associated with a series of user-defined chapter timestamps indi-cating the beginning of each topic chapter contained in this video. We thereby usethese available chapter beginning timestamps as ground-truth topic segment bound-aries since a topic chapter is by definition a main thematic division within a video.The video transcripts are generated by YouTube ASR with token-level offsets andwe further exploit a top performing punctuation restoration model3 [2] to boost thequality of transcript sentence segmentation. Compared with other existing video un-derstanding corpora such as BBC [9] (for scene segmentation, size = 11) and TVSum[150] (for video summarization, size = 50), our constructed YouTube corpus is (1)larger in size; (2) covering more diverse topics; (3) with reliable segmentation whichhas already been specified by video creators. We split this corpus into train/dev/testportions with size: 5148/134/140.7.5.2 Datasets for Cross-Domain InferenceTo evaluate our proposal’s robustness in cases where a domain-shift is present, weconduct experiments for cross-domain segment inference, in which our proposedsupervised segmenter is initially trained on the YouTube video corpus, and tested ontwo other corpora containing videos significantly longer than YouTube videos:3https://github.com/xashru/punctuation-restoration126BBC Planet Earth4 [9] consists of 11 episodes from the BBC educational TVseries Planet Earth. Topic segment boundaries of the dataset have been manuallyannotated by human experts and transcript sentences are obtained using Whisper5.As the statistics in Table 7.1 indicates, this dataset has longer videos with moresegments than YouTube videos, while each segment covers much fewer sentencesand tokens. We split this dataset into 5 samples for validation and 6 samples fortesting.Behance Livestream consists of 575 videos sampled from the creative livestreamplatform Behance. Livestreams, in general, are very challenging to segment intocoherent sections, since they contain mixes of multi-user dialogues, with visual fea-tures that change little over the video. They are also very long, with hours of contentin each video. Behance livestreams, in particular, contain another layer of challenge,since they are highly specialized in creative tasks such as animation and image editwith unknown entities and intricate visual operations. We believe finding structuresin such noisy environment would be not only a good benchmark for future topic seg-mentation research, but also of great practical value for consumers, who would findit hard to navigate contents in a back-and-forth conversation. Thus, we collect sucha corpus and annotate its videos with topic segments given the help from domainexperts in animation and image edits. For each video, we pay $23.16 for humanannotators to watch and create segments based on the video content. The total costto annotate 575 videos is $13,317. We split this dataset into validation and test setswith 57 (10%), 518 (90%) data-split.7.5.3 BaselinesWe consider compare our proposed methods with the following representative base-lines:Random: Given a video with its transcript consisting of k sentences, we first ran-domly sample the number of segment boundaries b ∈ {0, ...,k− 1} for this video,and then determine if each sentence is a segment boundary with probability bk .BayesSeg [34] originally proposed for text segmentation by predicting segment4http://imagelab.ing.unimore.it5https://openai.com/blog/whisper/127boundaries through modeling the lexical cohesion in a Bayesian context.GraphSeg [46] aims for text segmentation by generating a semantic relatednessgraph with sentences as nodes. Segments are then predicted as the maximal cliquesin graph.Cross-BERT [113] is a supervised text topic segmentation framework represent-ing candidate segment boundaries using their left and right contexts encoded withBERT. These candidates are then processed by a binary classifier to predict if theyare topical segment boundaries. The model is trained on YouTube corpus and di-rectly applied to longer BBC and Behance videos.TransNet [153] adopts stacked Dilated DCNN blocks as its basic framework andtargets video segmentation on shot level. Here we use its publicly available versiontrained on synthesized video corpora off-the-shelf.LGSS [138] is a multi-modal movie scene segmentation framework pre-trained onthe limited-scale MovieScenes corpus. We use its publicly available version withonly visual frames as input.X-Tiling is our implemented extension of TextTiling [56]. Different from the orig-inal TextTiling algorithm which only allows textual embeddings as input, X-Tilingcan accept both textual and visual embeddings and their concatenations as inputto compute semantic coherence and then make segment boundary predictions forvideos. To ensure a fair comparison, the input textual and visual embeddings areproduced by pre-trained BERT and mean-pooling over ResNet-18 respectively.All hyper-parameters required by the above baselines are tuned on the validationportion of the datasets included in this work.7.5.4 Evaluation MetricsWe apply three standard metrics in previous literature to evaluate the performancesof our proposal and baselines. They are:- F 1 , which is the standard armonic mean of the precision and recall, with higherscore values indicating better performance. It measures the exact match betweenground-truth segments and the model’s predictions.- Pr error score [43], which is often deemed more suitable for segmentation tasks,128as it addresses the limitations of F1 by penalizing errors based on the distance be-tween hypothesized and reference boundaries. It also fixes the inadequacies6 of Pk[14] and WindowDiff [130], previously considered as two standard evaluation met-rics for text segmentation. Concretely, Pr is the mean of missing and false alarmprobabilities, calculated based on the overlap between ground-truth segments andmodel’s predictions within a certain size sliding window. Since it is a penalty met-ric, lower score indicates better performance.- mIoU score [208], shortened from the mean Intersection-over-Union. It is calcu-lated by taking average over maximal IoUs of all ground-truth segments to predictedsegments. Higher score indicates better performance.7.5.5 Implementation DetailsFor the multi-modal video topic segmenter proposed in this work, we use the repre-sentation of special token [CLS] (d = 768) produced by bert-base-uncasedas sentence representation and the output of ResNet-18’s avg-pooling layer (d =512) as frame representation. The output dimension of all three feedfoward layersin cross-modal attention is set to 768. BiLSTM is set with layer number = 2 and hid-den size = 256. Model training on YouTube uses Adam as optimizer with lr = 1e−3and batch size = 16. For the dual-contrastive learning setting in long video adapa-tion, we use SGD as optimizer with mini-batch size = 256, lr = 3e−2 and softmaxtemperature = 1e−1. Model training is done with 10 epoches for both initial su-pervised learning and domain adaptation. All experimental results associated withmodel training are averaged over 3 runs. The threshold τ for segment prediction istuned on validation sets of all corpora we use for intra-domain and cross-domainevaluation respectively.7.6 Results and DiscussionIntra-Domain Segment Inference: Table 7.2 reports the performance of the cho-sen baselines and our proposal (NeuralSeg in the table) on the YouTube testing set,6Pk and WindowDiff are more commonly used for text segmentation evaluation in previous worksbut have been shown to problematic for long sequences with sparse boundary placing, it is somethingwe also observed in our experiments (e.g., with Behance). Pr addresses this particular issue and werefer readers to the original paper for full details.129Method Modalities Pr ↓ F1 ↑ mIoU ↑Random – 45.84 48.98 37.76X-Tiling Text 39.94 51.56 50.97BayesSeg Text 40.70 50.24 49.69GraphSeg Text 38.12 51.41 51.73Cross-BERT∗ Text 32.89 60.48 60.00X-Tiling Visual 37.53 52.08 53.24TransNet Visual 40.14 51.01 50.66LGSS Visual 39.77 50.87 51.13X-Tiling Text + Visual 38.78 52.30 51.45NeuralSeg∗ (Ours) Text 31.91 63.23 61.36Visual 50.18 47.59 16.15Text + Visual 30.61 65.29 63.11Table 7.2: Results on YouTube for intra-domain evaluation. Bolded andunderlined results indicate the best performance across all comparisonsand within their own sub-section, respectively. * indicates the marked ap-proach is in supervised setting.while NeuralSeg is trained on the YouTube training set with different input modalitysettings. We observe that even merely with video transcripts (text) as model’s input,our model can already remarkably outperform the best baseline method - Cross-BERT, which is supervised as well. But on the other hand, if we train the model byusing only video frames (visual) as input, the model’s performance is even consid-erably worse than the random baseline, possibly because that too diverse visual andtopic presence in the corpus makes it difficult for the model to learn a meaningfulvisual input-to-prediction projection. However, taking both modalities into accountwith cross-modal attention fusion, the model’s performance can be further enhancedcompared with only textual modality. These results confirm that the visual informa-tion itself may not be sufficient to capture a video’s topic-related semantics under asupervised setting, but fusing it together with the textual information can provide amore clear picture of the video’s underlying topics.Cross-Domain Segment Inference: Table 7.3 compares the performance of thebaselines and our proposed segmenter pre-trained on YouTube w/ or w/o applyingour long video adaptation strategy to two challenging long video corpora. To bet-130Method Modalities BBC BehancePr ↓ F1 ↑ mIoU ↑ Pr ↓ F1 ↑ mIoU ↑Random – 48.29 47.36 30.66 46.27 50.00 36.24X-Tiling [56] Text 41.59 50.72 33.27 45.93 49.95 43.16BayesSeg [34] Text 42.01 51.01 33.79 49.01 48.16 37.55GraphSeg [46] Text 45.11 49.28 30.23 46.76 50.12 40.38Cross-BERT [113] Text 44.63 49.70 31.51 46.15 49.88 41.02X-Tiling [56] Visual 44.48 49.89 33.26 44.56 50.73 41.81TransNet [153] Visual 42.54 49.82 31.12 46.34 49.88 40.54LGSS [138] Visual 42.88 50.02 32.66 45.67 50.12 39.90X-Tiling [56] Text + Visual 43.22 50.44 32.68 45.87 50.01 41.72Ours (Plain) Text + Visual 43.14 54.13 31.23 45.71 51.25 34.33Ours (Window) Text + Visual 40.66 54.92 37.50 43.03 51.83 47.66Ours (Window + CL-Cross) Text + Visual 36.45† 54.91 50.92† 42.72 51.68 48.65†Ours (Window + CL-Intra) Text + Visual 37.14† 55.10 49.22† 42.56 51.84 48.39Ours (Window + CL-Dual) Text + Visual 36.01† 55.75 51.56† 42.25† 51.91 49.35†Table 7.3: Results on BBC and Behance corpora for cross-domain evalua-tion. Bolded and underlined results indicate the best performance acrossall comparisons and within their own section. † indicates results appliedcontrastive domain adaptation which are significantly different (p < 0.05)from the row Ours (Window) in the table.ter investigate the effectiveness of the two contrastive objectives associated withlong video adaptation, Table 7.3 also shows the results by adding each objective(CL-Cross/CL-Intra) individually. The key observation from the table is that ourproposed long video adaptation strategy (Window + CL-Dual) significantly andconsistently improves the performance achieved by the segmenter initially trainedon short YouTube videos (Plain). In more detail, we can observe that utilizingthe sliding window inference strategy (Plain → Window) already grants noticeableperformance gains on both corpora. Furthermore, by updating the model with thecontrastive learning objectives, we further enhance accuracy on the two long videocorpora and the highest improvement is achieved by tuning the model with dual-contrastive losses.To further qualitatively analyze the advantages of the two components (slidingwindow inference and dual-contrastive learning) leveraged in our long video adap-tation strategy, Figure 7.5 visualizes the segment boundary predictions for the Be-hance livestream video in Figure 7.1 in an ablation-study manner. We can see that131Figure 7.5: Comparison of segment boundary predictions by different settingsof our proposal (Ours in Table 7.3) on the Behance example in Figure 7.1.Our proposed long video adaption strategy (Window + CL-Dual) recallsthe most ground truth boundaries.employing sliding window inference considerably alleviates the problem of overlysparse boundary prediction caused by directly applying the model trained on shortervideos. Furthermore, the dual-contrastive learning approach further enhances theprediction quality by preventing the model from generating redundant boundarieswithin a brief duration. Finally, our fully long video adaptation strategy (Window +CL-Dual) optimally aligns with the ground truth, as it recalls the highest number ofground truth boundaries.7.7 ContributionsIn this chapter, we present a multi-modal video topic segmentation model acceptingboth video transcripts and video frames as input to segment a video into a sequenceof topical-coherent pieces. Further, we propose a novel unsupervised domain adap-tation strategy mainly empowered by a dual-contrastive learning framework to gen-eralize our model pre-trained on short videos to longer videos with more complexcontent semantics and subtler visual changes. Experiments on two settings (intra-domain and cross-domain segment inference) show that (1) our system achievesthe state-of-the-art performance on a newly collected YouTube corpus consisting oflarge-scale but short videos; (2) When we apply our long video adaption strategy,the model trained for short videos can be enhanced to achieve better performance132when transferred to two domains comprising long (live-)stream videos, while stillpreserving its effectiveness on the domain it is originally trained on.In conclusion, while previous literature offers similar studies on video segmen-tation, these either focus solely on segmenting videos based on visual changes (shotsegmentation) or specifically target narrowly defined segments in the movie videogenre (scene segmentation). In this chapter, we aim to broaden the scope of videosegmentation, terming it as “video topic segmentation”, leveraging the prevalentavailability of video transcripts and visual frames. Experiments conducted on videosacross various domains and lengths reveal the promising performance of our pro-posed neural solution, addressing RQ3 to a certain extent. We believe that by in-corporating more modalities, the accuracy and robustness of neural video topic seg-mentation can be further enhanced.133Chapter 8Conclusions and Future Work8.1 ConclusionTopic Segmentation is a vibrant research area in Natural Language Processing (NLP)that involves breaking continuous text into smaller segments, each containing oneor more sentences centered around a shared topic. Topic segmentation can serve asa means to reveal the coarse-grained structure of a flat document and play a piv-otal role in enhancing both human and machine Natural Language Understanding(NLU), offering significant benefits to other fundamental NLP tasks (e.g., text sum-marization, information extraction, and question answering)Before the neural era, early unsupervised methods simply assess the cohesionbetween consecutive text units with lexical overlaps, and a significant drop in co-hesion usually signals the presence of a topical segment boundary. While theseapproaches are easy to deploy, the lexical features used only capture surface-levelcoherence between text units, leading to limited performance in real-world appli-cations. Later on, the progress in deep learning, combined with increased com-putational capabilities, has facilitated rapid and noteworthy advancements in topicsegmentation approaches. In the context of monologue topic segmentation, mostneural solutions introduced to date have favored supervised hierarchical RNNs (orTransformers). This preference is rooted in the natural inclination to treat topic seg-mentation as a sequence labeling task, where sentences are labeled as either the endof a segment or not. On the other hand, neural solutions for dialogue topic seg-134mentation predominantly utilize unsupervised neural approaches largely due to itslimited availability of labeled training data. Specifically, methods extended fromwell-established techniques such as TextTiling still remain popular. Empirically,these neural solutions for both monologue and dialogue topic segmentation listedabove have shown considerable improvements in accuracy and robustness. Thisholds true even when dealing with more recent textual resources, which display ahigher level of diversity and complexity.Despite the above-mentioned significant progress and numerous benefits offeredby previous neural topic segmentation techniques, there are still challenges and lim-itations that hinder the development of more accurate and robust neural solutionsfor this task. These challenges and limitations primarily stem from inadequate co-herence modeling in terms of both explicitness and reliability in such neural ap-proaches. Moreover, the rising prevalence of multi-modal data content and thewidespread dissemination of such content on social media platforms have intensi-fied the need to topically segmenting them. In order to tackle these aforementionedconcerns, three research questions have been formulated and outlined in Chapter 1:• RQ1: How can coherence be explicitly modeled under neural frameworks to im-prove the accuracy and robustness of topic segmentation?• RQ2: Can we harness learning signals encoded in other related auxiliary tasks toimprove the accuracy and robustness of topic segmentation?• RQ3: Given the surge in video content, how can we expand and adapt the conceptof topic segmentation to this specific medium?These three research questions have been systematically addressed in the followingthree constituent parts of this thesis.In Part I, we investigate how coherence can be explicitly modeled under neuralframeworks to enhance the accuracy and robustness of topic segmentation (RQ1).Given the differences in data availability between monologue and dialogue topicsegmentation, our research conducted for RQ1 diverges into two main paths: the firstcentered on monologue topic segmentation in a fully supervised neural context, andthe second tackling dialogue topic segmentation in an unsupervised neural context.135Chapter 2 highlights a shortcoming commonly found in earlier neural methodsfor monologue topic segmentation: their inability to effectively make predictionsby accounting for the coherence inherent in the text, which could potentially beaddressed through injecting more explicit coherence modeling. Specifically, wechoose an advanced neural segmenter as our testbed and enhance it to more ex-plicitly consider coherence when inferring topic boundaries, by equipping it witha coherence-related auxiliary task and restricted self-attention. Experiments acrossin-domain, domain-transfer, and multilingual settings consistently demonstrate thatour proposed technique adeptly guides the segmenter to explicitly model coherence,thereby boosting its effectiveness and robustness on monologue text.In Chapter 3, we shift our focus to dialogue topic segmentation, addressing oneof its major challenges: most unsupervised methods heavily rely on less advancedsemantic features (e.g., Word2Vec and vanilla BERT-based text encoders) with lim-ited capability to explicitly gauge topical coherence between utterances. In response,we devise a strategy to generate large-scale “pseudo” training datasets derived fromreal but unlabeled conversations and train a neural utterance-pair coherence scor-ing model on such pseudo data. This model is further responsible for assessingtopical relevance between utterances and collaborates with TextTiling to segmentdialogue sessions. Evaluation conducted in both English and Chinese consistentlyunderscore its improved ability in explicit coherence prediction, thereby enhancingunsupervised neural dialogue topic segmentation.In summary, Part I answers RQ1 by improving explicit coherence modeling inboth supervised and unsupervised settings, for both monologue and dialogue. Ourproposed solutions either directly model coherence patterns within the sequence la-beling neural framework or distill coherence signals from linguistic phenomena tiedto inherent coherence relations in authentic but unlabeled dialogues. Results of ourexperiments validate that our two strategies to explicitly model coherence can sig-nificantly boost the performance of supervised and unsupervised neural topic seg-mentation frameworks in terms of accuracy and robustness.In Part II, we explore the potential of harnessing learning signals from related auxil-iary NLP tasks to enhance the accuracy and robustness of topic segmentation (RQ2).Specifically, we investigate two auxiliary tasks: discourse parsing for monologues136and language modeling for dialogues. These two tasks may serve as indirect sourcesof coherence signals, thus paving the way for more accurate and robust topic seg-mentation.Our motivation to consider RST discourse parsing as an auxiliary task to enhancetopic segmentation comes from the fact that both tasks aim to create a more struc-tured and understandable text representation, considering coherence change as thekey signal to use. Thus, the coherence features learned by each individually couldpotentially be shared to mutually improve the performance in the other task. Sucha bidirectional synergy is explored and validated in Chapters 4 and 5. More specif-ically, in Chapter 4, we first investigate whether topic segmentation can provideuseful signals about topical coherence, and if these signals can be directly utilizedfor the construction of RST-style discourse constituency trees. Through applyinga greedy algorithm to the soft output from the neural topic segmenter presented inChapter 2, we conduct RST discourse tree structure inference on two prominent dis-course treebanks: RST-DT and GUM. Results highlight that the topical coherenceencoded in topic segmentation models can be effectively employed for discourseparsing. Given the partially confirmed coherence-rooted synergy between topic seg-mentation and discourse parsing (topic segmentation → discourse parsing), Chap-ter 5 aims to explore the reverse relationship as whether and how coherence signals,learned through RST discourse parsing, can effectively bolster neural topic segmen-tation for monologue. In order to do so, we direct the neural topic segmenter to bemore discourse-aware, based on the linguistic phenomenon that discourse depen-dencies between sentences often occur more frequently within a topic segment thanacross segments. Specifically, we augment our neural segmentation framework byincorporating RST-style discourse dependecy structures via a graph attention net-work. This enables the model to make predictions that better account for topicalcoherence encoded in the above-sentence-level discourse structure (silver standard).Empirical evaluations demonstrate the feasibility of using discourse parsing to im-prove topic segmentation in terms of accuracy and robustness, with only a minimalincrease in complexity.Our motivation to consider language modeling as a second auxiliary task to en-hance topic segmentation comes from the proven success of pre-trained languagemodels (PLMs) in a range of semantically intensive NLP tasks, hinting that PLMs137are capable of effectively capturing underlying text coherence patterns even withoutbeing explicitly trained for it. Therefore, in Chapter 6, we study the effectivenessof unsupervised dialogue topic segmenters particularly empowered by various typesof PLMs, primarily categorized as Language Representation Models (LRMs) andGenerative Language Models (GLMs). More specifically, we evaluate these mod-els by integrating them into three distinct dialogue segmentation frameworks thatalign with their unique features. To more thoroughly assess the robustness of theselanguage models, we further introduce a new evaluation benchmark including sevendialogue corpora, each with either initial ground-truth segment labels or “silver stan-dard” derived from their metadata attached. Our results show that recently emergedgigantic language models (e.g., GPT-3.5, categorized as a GLM) achieve signifi-cantly superior performance in dialogue topic segmentation, but are constrained bytheir input length limits. In scenarios involving prolonged dialogues, such as hours-long meetings, LRMs are the best choice.In conclusion, Part II answers RQ2 by initially confirming the relationship be-tween topic segmentation and two auxiliary tasks: discourse parsing and languagemodeling. With the sophisticated and intuitive methods we proposed, which lever-age the coherence overlap encoded in both parties, we can further enhance neuraltopic segmentation for both monologue and dialogue.In Part III, we present our attempt to answer RQ3: Given the surge in video con-tent, how can we expand and adapt the concept of topic segmentation to this specificmedium? Specifically, we strive to devise an accurate and robust neural solutiontargeting the unique challenges posed by video content, especially its multi-modalnature and drastic length variance. Given the conceptual parallels between videoand text topic segmentation, we conduct investigation based on insights from Part Iand II. Particularly, we also approach video topic segmentation as a sequence la-beling task, emphasizing the role of video transcripts as a fundamental modality.Building on a hierarchical neural framework, we integrate a cross-modal attentionmechanism to harmoniously fuse textual and visual signals for topic boundary infer-ence. To ensure the model’s adaptability for domains containing prolonged videos,we further introduce a domain adaptation strategy empowered by dual-contrastivelearning. Evaluations across diverse video datasets (e,g., short YouTube videos,138documentaries and live-streaming tutorials) show that our proposal notably outper-forms conventional text and video segmentation methods in terms of both accuracyand robustness.8.2 Future WorkThe research conducted in this thesis revolves around more effective and robust topicsegmentation. Nevertheless, when we consider integrating this task into the broader,contemporary real-world application landscape, we can identify three directions forfuture research that would enhance its relevance:8.2.1 More ModalitiesIn this dissertation, we merely attempt to position topic segmentation within a dual-modal framework, specifically by synergizing textual and visual signals (in Part III).Nevertheless, considering that today’s texts are more likely equipped with a broaderspectrum of information that goes beyond just visual cues, flexibly leveraging thesediverse information sources, such as speech signals and speaker activities (or char-acters [186]), can potentially further enhance the accuracy and robustness of topicsegmentation.Where audio is mostly available together with video, useful speech features maybe extracted and used for segmentation. Such attempts have already been pursuedin prior literature on shot and scene segmentation [62, 138]. Given the similaritiesin the essence of topic segmentation to these two video segmentation tasks, it isreasonable to infer that the insight derived from shot/scene segmentation could berepurposed for text segmentation, as integrating audio features (e.g., prosody, pitchvariation, speaker distinction) can be valuable indication regarding shifts in topicsor emphasis.In addition to audio, other modalities have also been explored for topic segmen-tation, such as tool usage in the context of meetings or live streaming. Back in 2007,Banerjee and Rudnicky [8] have already pioneered the idea of providing meetingparticipants with a note-taking tool, and using their interactions with this tool to im-prove the results derived from a TextTiling-based segmentation approach. In a morerecent study, Fraser et al. [40] explored the potential of application usage logs on a139live streaming platform to refine the video segmentation. Their hypothesis centeredon the notion that tools, commands, and settings engaged over the live streamingcan reveal the type of task a user is working on and highlight moments when theytransition to a different activity (theme).8.2.2 Segment Topic LabelingThis dissertation focuses on topic segmentation, which is only responsible for divid-ing the textual (visual) content into topical-coherent segments. However, this taskis usually closely coupled with segment topic labeling, which is another importanttask to assign appropriate labels to describe the theme of each obtained segment (asdiscussed in Chapter 1). These two tasks are usually together to provide the com-plete coarse-grained structural information for documents. Like segments providedby topic segmentation models have been shown to be useful for downstream tasks[181, 195, 200], there is also evidence indicating that the output of segment topiclabeling models can substantially improve readers’ reading experience from mul-tiple aspects [59]. However, in comparison to topic segmentation, the domain ofsegment topic labeling has received relatively less exploration. One possible reasonfor this might be the difficulty in generating human-understandable and meaningfultopic labels, especially for topics not in the training corpora but should be assignedto segments generated by the proposed topic segmenter.An early unsupervised segment topic labeler used document-wide word fre-quency and co-occurrence to extract keywords from the text and further generatephrases to represent the topic label for each segment [79]. However, the semanticrelatedness between words captured by the document-level word frequency and co-occurrence is not as accurate as the one captured by pretrained language models,since even vanilla BERT was shown to perform better than traditional count-basedmodels on several standard language understanding tasks [30]. More recently, seg-ment topic labeling is modelled jointly with topic segmentation by putting the topiclabeling module on top of the topic segmenter [3, 10, 158, 202]. As proposed in Bar-row et al. [10], topic segmentation is formulated as sequence labeling on sentencelevel and segment topic labeling is further performed upon each predicted segment.The strategy they used for topic label prediction is turning it into a topic classifica-tion task, with section titles in Wikipedia as predefined topic classes. In other words,140the name and number of topics are fixed and the trained topic labeler is only able toassign labels covered in the training corpus. This largely limits the labeling model’sgenerality when transferred to other domains with rather different themes or orga-nization formats. The problem is even more severe for dialogues since to the bestof our knowledge, there is still no sufficient dialogue corpora annotated with topiclabels such as section titles available in Wikipedia for model training.Besides framing segment topic labeling as a topic classification task and jointlymodeling it with topic segmentation, another task close to segment topic labeling is“extreme summarization” [123]. It was originally proposed to create a short (e.g.,one-sentence) summary answering the question “What is the article to summarizeabout?”. Ideally, we think unsupervised approaches of this task can also properlydeal with segment topic labeling by generating an even shorter (e.g., one-phrase)summary answering the question “What is this segment about?”. Even though alarge-scale dataset (XSUM [123]) has been proposed for extreme summarization,current models designed for it still fall short of generating summaries with the de-sired brevity (like one phrase or even one word) expected by segment topic labeling.In order to address the issue of the poor robustness and transferability troublingthe methodologies of segment topic labeling, a potential direction worthy to explorein the future could be framing it as a topic label generation task [16, 132] and takingadvantage of the semantic signals learned by more advanced large language models(LLMs), both open-sourced (e.g., Llama models [160], Mixtral [76]) and closed-sourced (e.g., GPT-3.5/4 [127, 128]), via supervised fine-tuning. Furthermore, wecould also explore to prompt LLMs to simultaneously produce segment boundariesand segment topic labels, rather than handling these two tasks in sequence. Such anintegrated approach could potentially enable two tasks to complement each other,ultimately boosting the overall effectiveness of the topic modeling system for singledocuments.8.2.3 Hierarchical Topic SegmentationIt is undeniable that topics within a document often exhibit a hierarchical structure[48, 115, 131]. This can be seen in how stretches of Wikipedia articles are typi-cally organized: a set of main topics may be further broken down into sub-topics.For instance, the main topic of a city’s demographics might be segmented into sub-141topics by different census years, or the main topic of a city’s cityscape might be seg-mented into urban planning and architecture, as two distinct sub-topics. On the onehand, such a structure would further benefit user retrieval and browsing, allowingthem to adjust the level of granularity as they see fit. On the other hand, produc-ing such a fine-grained structure proves to be an exceedingly challenging task evenfor humans. Studies by Galley et al. [41] and Gruenstein et al. [49] demonstratethat when annotators mark topic shifts in the ICSI Meeting Corpus [72], they oftenachieve significantly higher agreement on coarser-grained top-level topics than onfiner-grained sub-topics. While some prior research has attempted to tackle this taskusing unsupervised hierarchical sentence clustering frameworks (e.g., HierarchicalAgglomerative Clustering) driven by lexical or semantic features of text [13, 33, 60],to the best of our knowledge, neural solutions have not been applied yet, as linearsequence labeling is no longer a fitting formulation for this task. To advance thisline of research, a promising approach might be to formulate hierarchical topic seg-mentation as a combination of discourse tree generation and tree node classification.In the first step, we would employ an advanced neural discourse parser to create adiscourse constituency tree for the input document. Subsequently, we would utilizea classifier to traverse all intermediate nodes of the tree, determining whether thesub-tree rooted at each node constitutes a topic segment and identifying its corre-sponding level.Moreover, another significant concern in this line of research is the current eval-uation strategies applied. Most often, the segmentation prediction for each hierarchylevel is evaluated in isolation using standard Pk or WinDi f f metrics (as describedin section 1.2). However, human perception typically deems errors at higher topicsegmentation levels more severe than those at lower levels, given that mistakes atthe top can cascade down and affect the lower level. Thus, how to weigh errorsaccording to their levels in the tree is a crucial research direction towards a moreprecise evaluation for hierarchical topic segmentation.142Bibliography[1] V. Adlakha, S. Dhuliawala, K. Suleman, H. de Vries, and S. Reddy.TopiOCQA: Open-domain conversational question answering with topicswitching. Transactions of the Association for Computational Linguistics,10:468–483, 2022. doi:10.1162/tacl_a_00471. URLhttps://aclanthology.org/2022.tacl-1.27. → page 96[2] T. Alam, A. Khan, and F. Alam. Punctuation restoration using transformermodels for high-and low-resource languages. In Proceedings of the SixthWorkshop on Noisy User-generated Text (W-NUT 2020), pages 132–142,Online, Nov. 2020. Association for Computational Linguistics.doi:10.18653/v1/2020.wnut-1.18. URLhttps://aclanthology.org/2020.wnut-1.18. → page 126[3] S. Arnold, R. Schneider, P. Cudre´-Mauroux, F. A. Gers, and A. Lo¨ser.SECTOR: A neural model for coherent topic segmentation andclassification. Transactions of the Association for ComputationalLinguistics, 7:169–184, 2019. doi:10.1162/tacl_a_00261. URLhttps://aclanthology.org/Q19-1011. → pagesxx, 23, 25, 34, 35, 67, 78, 85, 86, 87, 140[4] N. Asher. Reference to abstract objects in discourse, volume 50. SpringerScience & Business Media, 1993. URL https://link.springer.com/book/10.1007/978-94-011-1715-9. →pages 16, 78, 80[5] N. Asher, N. M. Asher, and A. Lascarides. Logics of conversation.Cambridge University Press, 2003. URLhttps://www.cambridge.org/ca/academic/subjects/languages-linguistics/computational-linguistics/logics-conversation?format=PB&isbn=9780521659512. → pages8, 16, 78, 80143[6] D. Aumiller, S. Almasian, S. Lackner, and M. Gertz. Structural textsegmentation of legal documents. In Proceedings of the EighteenthInternational Conference on Artificial Intelligence and Law, ICAIL ’21,page 2–11, New York, NY, USA, 2021. Association for ComputingMachinery. ISBN 9781450385268. doi:10.1145/3462757.3466085.URL https://doi.org/10.1145/3462757.3466085. → page 1[7] P. Badjatiya, L. J. Kurisinkel, M. Gupta, and V. Varma. Attention-basedneural text segmentation. In European Conference on Information Retrieval2018, pages 180–193, 2018. → pages 8, 13[8] S. Banerjee and A. I. Rudnicky. Segmenting meetings into agenda items byextracting implicit supervision from human note-taking. In InternationalConference on Intelligent User Interfaces, 2007. URLhttps://api.semanticscholar.org/CorpusID:6727358. → page139[9] L. Baraldi, C. Grana, and R. Cucchiara. A deep siamese network for scenedetection in broadcast videos. In Proceedings of the 23rd ACM InternationalConference on Multimedia, page 1199–1202, 2015.doi:10.1145/2733373.2806316. URLhttps://doi.org/10.1145/2733373.2806316. → pages116, 118, 126, 127[10] J. Barrow, R. Jain, V. Morariu, V. Manjunatha, D. Oard, and P. Resnik. Ajoint model for document segmentation and segment labeling. InProceedings of the 58th Annual Meeting of the Association forComputational Linguistics, pages 313–322, Online, July 2020. Associationfor Computational Linguistics. doi:10.18653/v1/2020.acl-main.29.URL https://aclanthology.org/2020.acl-main.29. → pages8, 22, 79, 93, 116, 140[11] R. Barzilay and M. Lapata. Modeling local coherence: An entity-basedapproach. In Proceedings of the 43rd Annual Meeting of the Association forComputational Linguistics (ACL’05), pages 141–148, Ann Arbor, Michigan,June 2005. Association for Computational Linguistics.doi:10.3115/1219840.1219858. URLhttps://aclanthology.org/P05-1018. → pages 5, 25, 44[12] R. Barzilay and M. Lapata. Modeling local coherence: An entity-basedapproach. Computational Linguistics, 34(1):1–34, 2008.doi:10.1162/coli.2008.34.1.1. URLhttps://aclanthology.org/J08-1001. → pages 5, 25, 44144[13] M. Bayomi and S. Lawless. C-HTS: A concept-based hierarchical textsegmentation approach. In Proceedings of the Eleventh InternationalConference on Language Resources and Evaluation (LREC 2018),Miyazaki, Japan, May 2018. European Language Resources Association(ELRA). URL https://aclanthology.org/L18-1241. → page 142[14] D. Beeferman, A. Berger, and J. Lafferty. Statistical models for textsegmentation. Machine Learning, 34(1):177–210, 1999. → pagesxiv, 10, 35, 51, 65, 87, 104, 129[15] M. Bertrand, M. Bombardini, R. Fisman, B. Hackinen, and F. Trebbi. Hallof mirrors: Corporate philanthropy and strategic advocacy. Technical report,National Bureau of Economic Research, 2018. → pages 33, 85[16] S. Bhatia, J. H. Lau, and T. Baldwin. Automatic labelling of topics withneural embeddings. In Proceedings of COLING 2016, the 26th InternationalConference on Computational Linguistics: Technical Papers, pages953–963, Osaka, Japan, Dec. 2016. The COLING 2016 OrganizingCommittee. URL https://aclanthology.org/C16-1091. → page141[17] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal,A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal,A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M.Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray,B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, andD. Amodei. Language models are few-shot learners. CoRR, abs/2005.14165,2020. URL https://arxiv.org/abs/2005.14165. → page 106[18] P. Budzianowski, T.-H. Wen, B.-H. Tseng, I. Casanueva, S. Ultes,O. Ramadan, and M. Gasˇic´. MultiWOZ - a large-scale multi-domainWizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedingsof the 2018 Conference on Empirical Methods in Natural LanguageProcessing, pages 5016–5026, Brussels, Belgium, Oct.-Nov. 2018.Association for Computational Linguistics.doi:10.18653/v1/D18-1547. URLhttps://aclanthology.org/D18-1547. → pages 49, 96[19] X. Cao, Z. Chen, C. Le, and L. Meng. Multi-modal video chaptergeneration, 2022. → page 125[20] J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain,J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M. Lincoln,145A. Lisowska Masson, I. Mccowan, W. Post, D. Reidsma, and P. Wellner. Theami meeting corpus: A pre-announcement. 07 2005. ISBN978-3-540-32549-9. doi:10.1007/11677482_3. → page 97[21] L. Carlson, M. E. Okurowski, and D. Marcu. RST discourse treebank.Linguistic Data Consortium, University of Pennsylvania, 2002. → pages62, 67, 83[22] A. Cervone and G. Riccardi. Is this dialogue coherent? learning fromdialogue acts and entities. In Proceedings of the 21th Annual Meeting of theSpecial Interest Group on Discourse and Dialogue, pages 162–174, 1stvirtual meeting, July 2020. Association for Computational Linguistics. URLhttps://aclanthology.org/2020.sigdial-1.21. → page 44[23] D. Chen, D. Wang, T. Darrell, and S. Ebrahimi. Contrastive test-timeadaptation. In Proceedings of the 2022 IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition (CVPR), pages 295–305, 2022.doi:10.1109/CVPR52688.2022.00039. → page 118[24] H. Chen, S. Branavan, R. Barzilay, and D. R. Karger. Global models ofdocument structure using latent permutations. In Proceedings of HumanLanguage Technologies: The 2009 Annual Conference of the NorthAmerican Chapter of the Association for Computational Linguistics, pages371–379, Boulder, Colorado, June 2009. Association for ComputationalLinguistics. URL https://aclanthology.org/N09-1042. → pages34, 86[25] J. Chen and D. Yang. Simple conversational data augmentation forsemi-supervised abstractive dialogue summarization. In Proceedings of the2021 Conference on Empirical Methods in Natural Language Processing,pages 6605–6616, Online and Punta Cana, Dominican Republic, Nov. 2021.Association for Computational Linguistics.doi:10.18653/v1/2021.emnlp-main.530. URLhttps://aclanthology.org/2021.emnlp-main.530. → page 106[26] S. Chen, X. Nie, D. D. Fan, D. Zhang, V. Bhat, and R. Hamid. Shotcontrastive self-supervised learning for scene boundary detection. InProceedings of the 2021 IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), pages 9791–9800, 2021.doi:10.1109/CVPR46437.2021.00967. → page 116[27] F. Y. Y. Choi. Advances in domain independent linear text segmentation. In1st Meeting of the North American Chapter of the Association for146Computational Linguistics, 2000. URLhttps://aclanthology.org/A00-2004. → pages 33, 85[28] D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deepnetwork learning by exponential linear units (elus). arXiv: Learning, 2016.→ page 84[29] O. de clercq, V. Hoste, B. Desmet, P. Oosten, M. De Cock, and L. Macken.Using the crowd for readability prediction. Natural Language Engineering,20:293–325, 07 2012. doi:10.1017/S1351324912000344. → page 5[30] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training ofdeep bidirectional transformers for language understanding. In Proceedingsof the 2019 Conference of the North American Chapter of the Associationfor Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June2019. Association for Computational Linguistics.doi:10.18653/v1/N19-1423. URLhttps://aclanthology.org/N19-1423. → pages9, 17, 28, 42, 48, 58, 99, 105, 120, 140[31] D. Diefenbach, V. Lopez, K. Singh, and P. Maret. Core techniques ofquestion answering systems over knowledge bases: a survey. Knowledgeand Information Systems, 55(3):529–569, 2018. → page 3[32] L. Du, W. Buntine, and M. Johnson. Topic segmentation with a structuredtopic model. In Proceedings of the 2013 Conference of the North AmericanChapter of the Association for Computational Linguistics: HumanLanguage Technologies, pages 190–200, Atlanta, Georgia, June 2013.Association for Computational Linguistics. URLhttps://aclanthology.org/N13-1019. → page 6[33] J. Eisenstein. Hierarchical text segmentation from multi-scale lexicalcohesion. In Proceedings of Human Language Technologies: The 2009Annual Conference of the North American Chapter of the Association forComputational Linguistics, pages 353–361, Boulder, Colorado, June 2009.Association for Computational Linguistics. URLhttps://aclanthology.org/N09-1040. → pages 6, 142[34] J. Eisenstein and R. Barzilay. Bayesian unsupervised topic segmentation. InProceedings of the 2008 Conference on Empirical Methods in NaturalLanguage Processing, pages 334–343, Honolulu, Hawaii, Oct. 2008.Association for Computational Linguistics. URL147https://aclanthology.org/D08-1035. → pages23, 35, 41, 50, 53, 65, 68, 87, 93, 94, 127, 131[35] M. Eric, L. Krishnan, F. Charette, and C. D. Manning. Key-value retrievalnetworks for task-oriented dialogue. In Proceedings of the 18th AnnualSIGdial Meeting on Discourse and Dialogue, pages 37–49, Saarbru¨cken,Germany, Aug. 2017. Association for Computational Linguistics.doi:10.18653/v1/W17-5506. URLhttps://aclanthology.org/W17-5506. → pages 50, 96[36] S. Feng, H. Wan, C. Gunasekara, S. Patel, S. Joshi, and L. Lastras. doc2dial:A goal-oriented document-grounded dialogue dataset. In Proceedings of the2020 Conference on Empirical Methods in Natural Language Processing(EMNLP), pages 8118–8128, Online, Nov. 2020. Association forComputational Linguistics. doi:10.18653/v1/2020.emnlp-main.652.URL https://aclanthology.org/2020.emnlp-main.652. → pagesxiii, xiv, xix, 41, 50, 53, 54, 96[37] S. Feng, S. Patel, and H. Wan. DialDoc 2022 shared task: Open-bookdocument-grounded dialogue modeling. In Proceedings of the SecondDialDoc Workshop on Document-grounded Dialogue and ConversationalQuestion Answering, pages 155–160, Dublin, Ireland, May 2022.Association for Computational Linguistics.doi:10.18653/v1/2022.dialdoc-1.18. URLhttps://aclanthology.org/2022.dialdoc-1.18. → page 96[38] V. W. Feng and G. Hirst. A linear-time bottom-up discourse parser withconstraints and post-editing. In Proceedings of the 52nd Annual Meeting ofthe Association for Computational Linguistics (Volume 1: Long Papers),pages 511–521, Baltimore, Maryland, June 2014. Association forComputational Linguistics. doi:10.3115/v1/P14-1048. → page 80[39] X. Feng, X. Feng, L. Qin, B. Qin, and T. Liu. Language model as anannotator: Exploring DialoGPT for dialogue summarization. In Proceedingsof the 59th Annual Meeting of the Association for Computational Linguisticsand the 11th International Joint Conference on Natural LanguageProcessing (Volume 1: Long Papers), pages 1479–1491, Online, Aug. 2021.Association for Computational Linguistics.doi:10.18653/v1/2021.acl-long.117. URLhttps://aclanthology.org/2021.acl-long.117. → page 101[40] C. Fraser, J. Kim, H. Shin, J. Brandt, and M. Dontcheva. Temporalsegmentation of creative live streams. In Proceedings of the 2020 CHI148Conference on Human Factors in Computing Systems, page 1–12, 2020.doi:10.1145/3313831.3376437. → pages 3, 115, 139[41] M. Galley, K. McKeown, E. Fosler-Lussier, and H. Jing. Discoursesegmentation of multi-party conversation. In Proceedings of the 41st AnnualMeeting on Association for Computational Linguistics - Volume 1, pages562–569, 2003. → pages 6, 22, 93, 94, 142[42] H. Gao, R. Wang, T.-E. Lin, Y. Wu, M. Yang, F. Huang, and Y. Li.Unsupervised dialogue topic segmentation with topic-aware utterancerepresentation, 2023. → page 93[43] M. Georgescul, A. Clark, and S. Armstrong. An analysis of quantitativeaspects in the evaluation of thematic segmentation algorithms. InProceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pages144–151, Sydney, Australia, July 2006. Association for ComputationalLinguistics. URL https://aclanthology.org/W06-1320. → pages12, 104, 108, 128[44] R. Ghosh, H. S. Kajal, S. Kamath, D. Shrivastava, S. Basu, andS. Srinivasan. Topic segmentation in the wild: Towards segmentation ofsemi-structured unstructured chats, 2022. → page 8[45] G. Glavas and S. Somasundaran. Two-level transformer and auxiliarycoherence modeling for improved text segmentation. In The Thirty-FourthAAAI Conference on Artificial Intelligence (AAAI-20), pages 2306–2315,2020. → pages 8, 22, 24, 35, 79, 87, 93, 122[46] G. Glavasˇ, F. Nanni, and S. P. Ponzetto. Unsupervised text segmentationusing semantic relatedness graphs. In Proceedings of the Fifth JointConference on Lexical and Computational Semantics, pages 125–130,Berlin, Germany, Aug. 2016. Association for Computational Linguistics.doi:10.18653/v1/S16-2016. URLhttps://aclanthology.org/S16-2016. → pages6, 23, 35, 50, 53, 65, 69, 87, 128, 131[47] K. Gopalakrishnan, B. Hedayatnia, Q. Chen, A. Gottardi, S. Kwatra,A. Venkatesh, R. Gabriel, and D. Hakkani-Tu¨r. Topical-Chat: TowardsKnowledge-Grounded Open-Domain Conversations. In Proc. Interspeech2019, pages 1891–1895, 2019.doi:10.21437/Interspeech.2019-3079. → page 97149[48] B. J. Grosz and C. L. Sidner. Attention, intentions, and the structure ofdiscourse. Computational Linguistics, 12(3):175–204, 1986. URLhttps://aclanthology.org/J86-3001. → page 141[49] A. Gruenstein, J. Niekrasz, and M. Purver. Meeting Structure Annotation,pages 247–274. Springer Netherlands, Dordrecht, 2008. ISBN978-1-4020-6821-8. doi:10.1007/978-1-4020-6821-8_10. URLhttps://doi.org/10.1007/978-1-4020-6821-8_10. → page 142[50] G. Guz and G. Carenini. Coreference for discourse parsing: A neuralapproach. In Proceedings of the First Workshop on ComputationalApproaches to Discourse, pages 160–167, Online, Nov. 2020. Associationfor Computational Linguistics. doi:10.18653/v1/2020.codi-1.17.URL https://aclanthology.org/2020.codi-1.17. → pages61, 62, 68, 71, 80, 176, 178[51] G. Guz and G. Carenini. Towards domain-independent text structuringtrainable on large discourse treebanks. In Proceedings of the Workshop onDiscourse Theories for Text Planning, pages 17–28, Dublin, Ireland, Dec.2020. Association for Computational Linguistics. URLhttps://aclanthology.org/2020.dt4tp-1.4. → page 62[52] G. Guz, P. Huber, and G. Carenini. Unleashing the power of neuraldiscourse parsers - a context and structure aware approach using large scalepretraining. In Proceedings of the 28th International Conference onComputational Linguistics, pages 3794–3805, Barcelona, Spain (Online),Dec. 2020. International Committee on Computational Linguistics.doi:10.18653/v1/2020.coling-main.337. URLhttps://aclanthology.org/2020.coling-main.337. → page 80[53] M. Gygli, H. Grabner, H. Riemenschneider, and L. Van Gool. Creatingsummaries from user videos. In Proceedings of the 2014 EuropeanConference on Computer Vision (ECCV), pages 505–520, 2014.doi:10.1007/978-3-319-10584-0_33. → page 116[54] S. Hao and M. J. Paul. An empirical study on crosslingual transfer inprobabilistic topic models. Computational Linguistics, 46(1):95–134, 2020.→ page 35[55] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for imagerecognition. In Proceedings of the 2016 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), pages 770–778, 2016.doi:10.1109/CVPR.2016.90. → page 120150[56] M. A. Hearst. Text tiling: Segmenting text into multi-paragraph subtopicpassages. Computational Linguistics, 23(1):33–64, 1997. URLhttps://aclanthology.org/J97-1003. → pages6, 23, 41, 42, 43, 45, 51, 53, 65, 68, 93, 94, 100, 128, 131[57] M. Henderson, I. Casanueva, N. Mrksˇic´, P.-H. Su, T.-H. Wen, and I. Vulic´.ConveRT: Efficient and accurate conversational representations fromtransformers. In Findings of the Association for Computational Linguistics:EMNLP 2020, pages 2161–2174, Online, Nov. 2020. Association forComputational Linguistics.doi:10.18653/v1/2020.findings-emnlp.196. URLhttps://aclanthology.org/2020.findings-emnlp.196. → page105[58] H. Hernault, H. Prendinger, M. Ishizuka, et al. Hilda: A discourse parserusing support vector machine classification. Dialogue & Discourse, 1(3),2010. doi:10.5087/dad.2010.003. → page 80[59] E. Hoque and G. Carenini. Multiconvis: A visual text analytics system forexploring a collection of online conversations. In Proceedings of the 21stInternational Conference on Intelligent User Interfaces, IUI ’16, page96–107. Association for Computing Machinery, 2016. → page 140[60] E. Hoque and G. Carenini. Interactive topic hierarchy revision for exploringa collection of online conversations. Information Visualization, 18:318 –338, 2019. URLhttps://api.semanticscholar.org/CorpusID:65326016. → page142[61] L. Huang, Z. Ye, J. Qin, L. Lin, and X. Liang. GRADE: Automaticgraph-enhanced coherence metric for evaluating open-domain dialoguesystems. In Proceedings of the 2020 Conference on Empirical Methods inNatural Language Processing (EMNLP), pages 9230–9240, Online, Nov.2020. Association for Computational Linguistics.doi:10.18653/v1/2020.emnlp-main.742. URLhttps://aclanthology.org/2020.emnlp-main.742. → pages45, 84[62] Q. Huang, Y. Xiong, A. Rao, J. Wang, and D. Lin. Movienet: A holisticdataset for movie understanding. In Proceedings of the EuropeanConference on Computer Vision (ECCV), 2020. → pages 117, 139151[63] X. Huang, L. Xing, F. Dernoncourt, and M. J. Paul. Multilingual Twittercorpus and baselines for evaluating demographic bias in hate speechrecognition. In Proceedings of the Twelfth Language Resources andEvaluation Conference, pages 1440–1448, Marseille, France, May 2020.European Language Resources Association. ISBN 979-10-95546-34-4.URL https://aclanthology.org/2020.lrec-1.180. → page 182[64] P. Huber and G. Carenini. Predicting discourse structure using distantsupervision from sentiment. In Proceedings of the 2019 Conference onEmpirical Methods in Natural Language Processing and the 9thInternational Joint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 2306–2316, Hong Kong, China, Nov. 2019.Association for Computational Linguistics.doi:10.18653/v1/D19-1235. URLhttps://aclanthology.org/D19-1235. → pages 63, 70, 88[65] P. Huber and G. Carenini. MEGA RST discourse treebanks with structureand nuclearity from scalable distant sentiment supervision. In Proceedingsof the 2020 Conference on Empirical Methods in Natural LanguageProcessing (EMNLP), pages 7442–7457, Online, Nov. 2020. Association forComputational Linguistics. doi:10.18653/v1/2020.emnlp-main.603.URL https://aclanthology.org/2020.emnlp-main.603. → pages61, 63, 66, 68, 71, 176, 177, 178, 179[66] P. Huber and G. Carenini. From sentiment annotations to sentimentprediction through discourse augmentation. In Proceedings of the 28thInternational Conference on Computational Linguistics, pages 185–197,Barcelona, Spain (Online), Dec. 2020. International Committee onComputational Linguistics. doi:10.18653/v1/2020.coling-main.16.URL https://aclanthology.org/2020.coling-main.16. → page61[67] P. Huber and G. Carenini. Unsupervised learning of discourse structuresusing a tree autoencoder. arXiv preprint arXiv:2012.09446, 2021. URLhttps://arxiv.org/pdf/2012.09446.pdf. → page 63[68] P. Huber, L. Xing, and G. Carenini. Predicting above-sentence discoursestructure using distant supervision from topic segmentation. In TheThirty-sixth AAAI Conference on Artificial Intelligence (AAAI-22), pages10794–10802, 2021. → pages vi, 59[69] S. Jadon and M. Jasim. Unsupervised video summarization framework usingkeyframe extraction and video skimming. In Proceedings of the 2020 IEEE1525th International Conference on Computing Communication andAutomation (ICCCA), pages 140–145, 2020.doi:10.1109/ICCCA49541.2020.9250764. → page 116[70] N. James, K. Todorov, and C. Hudelot. Combining visual and textualmodalities for multimedia ontology matching. In T. Declerck, M. Granitzer,M. Grzegorzek, M. Romanelli, S. Ru¨ger, and M. Sintek, editors, SemanticMultimedia, pages 95–110, 2011. → pages 116, 118[71] Y. Jang, D. Lee, H. J. Park, T. Kang, H. Lee, H. Bae, and K. Jung.Improving multiple documents grounded goal-oriented dialog systems viadiverse knowledge enhanced pretrained language model. In Proceedings ofthe Second DialDoc Workshop on Document-grounded Dialogue andConversational Question Answering, pages 136–141, Dublin, Ireland, May2022. Association for Computational Linguistics.doi:10.18653/v1/2022.dialdoc-1.15. URLhttps://aclanthology.org/2022.dialdoc-1.15. → page 96[72] A. Janin, D. Baron, J. Edwards, D. Ellis, D. Gelbart, N. Morgan, B. Peskin,T. Pfau, E. Shriberg, A. Stolcke, and C. Wooters. The icsi meeting corpus.In 2003 IEEE International Conference on Acoustics, Speech, and SignalProcessing, 2003. Proceedings. (ICASSP ’03)., volume 1, pages I–I, 2003.doi:10.1109/ICASSP.2003.1198793. → pages 98, 142[73] D. Jayaraman and K. Grauman. Slow and steady feature analysis: Higherorder temporal coherence in video. In Proceeding of the 2016 IEEEConference on Computer Vision and Pattern Recognition (CVPR), pages3852–3861, 2016. doi:10.1109/CVPR.2016.418. → page 116[74] Y. Ji and J. Eisenstein. Representation learning for text-level discourseparsing. In Proceedings of the 52nd Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers), pages 13–24,Baltimore, Maryland, June 2014. Association for Computational Linguistics.doi:10.3115/v1/P14-1002. URLhttps://aclanthology.org/P14-1002. → pages 60, 62, 80[75] Y. Ji and N. A. Smith. Neural discourse structure for text categorization. InProceedings of the 55th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers), pages 996–1005,Vancouver, Canada, July 2017. Association for Computational Linguistics.doi:10.18653/v1/P17-1092. URLhttps://aclanthology.org/P17-1092. → page 61153[76] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford,D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel,G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock,S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril,T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts, 2024. → page 141[77] F. Jiang, Y. Fan, X. Chu, P. Li, Q. Zhu, and F. Kong. Hierarchical macrodiscourse parsing based on topic segmentation. In Proceedings of the AAAIConference on Artificial Intelligence, 2021. → pages 1, 60, 61, 63, 178[78] W. Jiao, W. Wang, J. tse Huang, X. Wang, and Z. Tu. Is chatgpt a goodtranslator? yes with gpt-4 as the engine, 2023. → page 101[79] S. Joty, G. Carenini, and R. T. Ng. Topic segmentation and labeling inasynchronous conversations. J. Artif. Int. Res., 47(1):521–573, 2013. ISSN1076-9757. → page 140[80] S. Joty, G. Carenini, and R. T. Ng. CODRA: A novel discriminativeframework for rhetorical analysis. Computational Linguistics, 41(3):385–435, Sept. 2015. doi:10.1162/COLI_a_00226. URLhttps://aclanthology.org/J15-3002. → pages60, 62, 69, 70, 80, 83[81] D. Jurafsky and J. H. Martin. Speech and language processing, volume 3.Pearson London, 2014. → pages 2, 61[82] G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann. Contrastive adaptationnetwork for unsupervised domain adaptation. In Proceedings of the 2019IEEE Conference on Computer Vision and Pattern Recognition, pages4893–4902, 2019. → page 118[83] G. Kang, L. Jiang, Y. Wei, Y. Yang, and A. G. Hauptmann. Contrastiveadaptation network for single-and multi-source domain adaptation. IEEEtransactions on pattern analysis and machine intelligence, 2020. → page118[84] O. Z. Khan, J.-P. Robichaud, P. A. Crook, and R. Sarikaya. Hypothesesranking and state tracking for a multi-domain dialog system using multipleasr alternates. In INTERSPEECH, page 1810–1814, 2015. → pages 9, 43[85] D. Kim, Y.-H. Tsai, B. Zhuang, X. Yu, S. Sclaroff, K. Saenko, andM. Chandraker. Learning cross-modal contrastive features for video domainadaptation. In Proceedings of the 2021 IEEE/CVF International Conferenceon Computer Vision (ICCV), pages 13598–13607, 2021. → page 118154[86] N. Kobayashi, T. Hirao, K. Nakamura, H. Kamigaito, M. Okumura, andM. Nagata. Split or merge: Which is better for unsupervised RST parsing?In Proceedings of the 2019 Conference on Empirical Methods in NaturalLanguage Processing and the 9th International Joint Conference on NaturalLanguage Processing (EMNLP-IJCNLP), pages 5797–5802, Hong Kong,China, Nov. 2019. Association for Computational Linguistics.doi:10.18653/v1/D19-1587. URLhttps://aclanthology.org/D19-1587. → page 63[87] N. Kobayashi, T. Hirao, H. Kamigaito, M. Okumura, and M. Nagata.Top-down rst parsing utilizing granularity levels in documents. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34,pages 8099–8106, 2020. doi:10.1609/aaai.v34i05.6321. URLhttps://ojs.aaai.org/index.php/AAAI/article/view/6321. →pages 61, 62, 176[88] O. Koshorek, A. Cohen, N. Mor, M. Rotman, and J. Berant. Textsegmentation as a supervised learning task. In Proceedings of the 2018Conference of the North American Chapter of the Association forComputational Linguistics: Human Language Technologies, Volume 2(Short Papers), pages 469–473, New Orleans, Louisiana, June 2018.Association for Computational Linguistics.doi:10.18653/v1/N18-2075. URLhttps://aclanthology.org/N18-2075. → pages8, 23, 24, 27, 28, 34, 35, 36, 40, 65, 67, 69, 86, 87, 116, 119, 120[89] F. Koto, J. H. Lau, and T. Baldwin. Discourse probing of pretrainedlanguage models. In Proceedings of the 2021 Conference of the NorthAmerican Chapter of the Association for Computational Linguistics: HumanLanguage Technologies, pages 3849–3864, Online, June 2021. Associationfor Computational Linguistics.doi:10.18653/v1/2021.naacl-main.301. URLhttps://aclanthology.org/2021.naacl-main.301. → page 17[90] A. Kumar, T. Mittal, and D. Manocha. MCQA: multimodal co-attentionbased network for question answering. CoRR, abs/2004.12238, 2020. URLhttps://arxiv.org/abs/2004.12238. → page 121[91] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti,D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones,M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov.Natural questions: A benchmark for question answering research.155Transactions of the Association for Computational Linguistics, 7:452–466,2019. doi:10.1162/tacl_a_00276. URLhttps://aclanthology.org/Q19-1026. → page 96[92] A. Lai and J. Tetreault. Discourse coherence in the wild: A dataset,evaluation and methods. In Proceedings of the 19th Annual SIGdial Meetingon Discourse and Dialogue, pages 214–223, Melbourne, Australia, July2018. Association for Computational Linguistics.doi:10.18653/v1/W18-5023. URLhttps://aclanthology.org/W18-5023. → page 5[93] T. M. Lai, G. Castellucci, S. Kuzi, H. Ji, and O. Rokhlenko. Externalknowledge acquisition for end-to-end document-oriented dialog systems. InProceedings of the 17th Conference of the European Chapter of theAssociation for Computational Linguistics, pages 3633–3647, Dubrovnik,Croatia, May 2023. Association for Computational Linguistics. URLhttps://aclanthology.org/2023.eacl-main.264. → page 96[94] J. Lei, T. L. Berg, and M. Bansal. Detecting moments and highlights invideos via natural language queries. In Proceedings of the 2021 Advances inNeural Information Processing Systems, pages 11846–11858, 2021. URLhttps://proceedings.neurips.cc/paper/2021/file/62e0973455fd26eb03e91d5741a4a3bb-Paper.pdf. → page 116[95] Y. Lei, F. Zheng, Y. Yan, K. He, and W. Xu. A finer-grain universal dialoguesemantic structures based model for abstractive dialogue summarization. InFindings of the Association for Computational Linguistics: EMNLP 2021,pages 1354–1364, Punta Cana, Dominican Republic, Nov. 2021. Associationfor Computational Linguistics.doi:10.18653/v1/2021.findings-emnlp.117. URLhttps://aclanthology.org/2021.findings-emnlp.117. → page106[96] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy,V. Stoyanov, and L. Zettlemoyer. BART: Denoising sequence-to-sequencepre-training for natural language generation, translation, and comprehension.In Proceedings of the 58th Annual Meeting of the Association forComputational Linguistics, pages 7871–7880, Online, July 2020.Association for Computational Linguistics.doi:10.18653/v1/2020.acl-main.703. URLhttps://aclanthology.org/2020.acl-main.703. → pages17, 58, 99, 106156[97] C. Li, P. Huber, W. Xiao, M. Amblard, C. Braud, and G. Carenini. Discoursestructure extraction from pre-trained and fine-tuned language models indialogues. In A. Vlachos and I. Augenstein, editors, Findings of theAssociation for Computational Linguistics: EACL 2023, pages 2562–2579,Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.doi:10.18653/v1/2023.findings-eacl.194. URLhttps://aclanthology.org/2023.findings-eacl.194. → pages8, 16[98] H. Li. A short introduction to learning to rank. IEICE Transactions onInformation and Systems, E94.D(10):1854–1862, 2011.doi:10.1587/transinf.E94.D.1854. → page 42[99] J. Li and D. Jurafsky. Neural net models of open-domain discoursecoherence. In Proceedings of the 2017 Conference on Empirical Methods inNatural Language Processing, pages 198–209, Copenhagen, Denmark, Sept.2017. Association for Computational Linguistics.doi:10.18653/v1/D17-1019. URLhttps://aclanthology.org/D17-1019. → page 5[100] M. Li, L. Zhang, H. Ji, and R. J. Radke. Keep meeting summaries on topic:Abstractive multi-modal meeting summarization. In Proceedings of the 57thAnnual Meeting of the Association for Computational Linguistics, pages2190–2196, Florence, Italy, July 2019. Association for ComputationalLinguistics. doi:10.18653/v1/P19-1210. URLhttps://aclanthology.org/P19-1210. → page 116[101] R. Li, W. Xiao, L. Xing, L. Wang, G. Murray, and G. Carenini. Humanguided exploitation of interpretable attention patterns in summarization andtopic segmentation. In Proceedings of the 2022 Conference on EmpiricalMethods in Natural Language Processing, pages 10189–10204, Abu Dhabi,United Arab Emirates, Dec. 2022. Association for ComputationalLinguistics. doi:10.18653/v1/2022.emnlp-main.694. URLhttps://aclanthology.org/2022.emnlp-main.694. → page 181[102] R. Li, F. Gonzalez-Pizarro, L. Xing, G. Murray, and G. Carenini.Diversity-aware coherence loss for improving neural topic models. InProceedings of the 61st Annual Meeting of the Association forComputational Linguistics (Volume 2: Short Papers), pages 1710–1722,Toronto, Canada, July 2023. Association for Computational Linguistics.doi:10.18653/v1/2023.acl-short.145. URLhttps://aclanthology.org/2023.acl-short.145. → page 180157[103] Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu. DailyDialog: A manuallylabelled multi-turn dialogue dataset. In Proceedings of the EighthInternational Joint Conference on Natural Language Processing (Volume 1:Long Papers), pages 986–995, Taipei, Taiwan, Nov. 2017. Asian Federationof Natural Language Processing. URLhttps://aclanthology.org/I17-1099. → pages 42, 46[104] X. Liang, J. Li, S. Wu, J. Zeng, Y. Jiang, M. Li, and Z. Li. An efficientcoarse-to-fine facet-aware unsupervised summarization framework based onsemantic blocks. In Proceedings of the 29th International Conference onComputational Linguistics, pages 6415–6425, Gyeongju, Republic of Korea,Oct. 2022. International Committee on Computational Linguistics. URLhttps://aclanthology.org/2022.coling-1.558. → page 100[105] X. Lin, S. Joty, P. Jwalapuram, and M. S. Bari. A unified linear-timeframework for sentence-level discourse parsing. In Proceedings of the 57thAnnual Meeting of the Association for Computational Linguistics, pages4190–4200, Florence, Italy, July 2019. Association for ComputationalLinguistics. doi:10.18653/v1/P19-1410. URLhttps://aclanthology.org/P19-1410. → page 63[106] Z. C. Lipton, J. Berkowitz, and C. Elkan. A critical review of recurrentneural networks for sequence learning. CoRR, abs/1506.00019, 2015. →page 23[107] Y. Liu and M. Lapata. Learning structured text representations. Transactionsof the Association for Computational Linguistics, 6:63–75, 2018.doi:10.1162/tacl\_a\_00005. → page 80[108] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis,L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bertpretraining approach. ArXiv, abs/1907.11692, 2019. → pages17, 58, 99, 105[109] K. Lo, Y. Jin, W. Tan, M. Liu, L. Du, and W. Buntine. Transformer overpre-trained transformer for neural text segmentation with enhanced topiccoherence. In Findings of the Association for Computational Linguistics:EMNLP 2021, pages 3334–3340, Punta Cana, Dominican Republic, Nov.2021. Association for Computational Linguistics.doi:10.18653/v1/2021.findings-emnlp.283. URLhttps://aclanthology.org/2021.findings-emnlp.283. → pages8, 79, 93, 116158[110] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. InICLR, 2019. → page 51[111] A. Louis and A. Nenkova. A coherence model based on syntactic patterns.In Proceedings of the 2012 Joint Conference on Empirical Methods inNatural Language Processing and Computational Natural LanguageLearning, pages 1157–1168, Jeju Island, Korea, July 2012. Association forComputational Linguistics. URLhttps://aclanthology.org/D12-1106. → page 77[112] J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-imageco-attention for visual question answering. In Proceedings of the 30thInternational Conference on Neural Information Processing Systems, page289–297, 2016. → page 121[113] M. Lukasik, B. Dadachev, K. Papineni, and G. Simo˜es. Text segmentationby cross segment attention. In Proceedings of the 2020 Conference onEmpirical Methods in Natural Language Processing (EMNLP), pages4707–4716, Online, Nov. 2020. Association for Computational Linguistics.doi:10.18653/v1/2020.emnlp-main.380. URLhttps://aclanthology.org/2020.emnlp-main.380. → pages8, 22, 120, 128, 131[114] I. Malioutov and R. Barzilay. Minimum cut model for spoken lecturesegmentation. In Proceedings of the 21st International Conference onComputational Linguistics and 44th Annual Meeting of the Association forComputational Linguistics, pages 25–32, Sydney, Australia, July 2006.Association for Computational Linguistics.doi:10.3115/1220175.1220179. URLhttps://aclanthology.org/P06-1004. → pages 34, 86[115] W. C. Mann and S. A. Thompson. Rhetorical structure theory: Toward afunctional theory of text organization. Text-Interdisciplinary Journal for theStudy of Discourse, 8(3):243–281, 1988. → pages 8, 16, 59, 77, 80, 141[116] D. Marcu. Discourse Trees are Good Indicators of Importance in Text.Advances in Automatic Text Summarization, pages 123–136, 1999. ISSN1098-6596. doi:10.1017/CBO9781107415324.004. → page 61[117] D. Marcu. The theory and practice of discourse parsing and summarization.MIT press, 2000. → pages 61, 70159[118] M. Mesgar and M. Strube. A neural local coherence model for text qualityassessment. In Proceedings of the 2018 Conference on Empirical Methodsin Natural Language Processing, pages 4328–4339, Brussels, Belgium,Oct.-Nov. 2018. Association for Computational Linguistics.doi:10.18653/v1/D18-1464. URLhttps://aclanthology.org/D18-1464. → page 44[119] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of wordrepresentations in vector space. In Y. Bengio and Y. LeCun, editors, 1stInternational Conference on Learning Representations, ICLR 2013,Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings,2013. URL http://arxiv.org/abs/1301.3781. → page 9[120] J. Morris and G. Hirst. Lexical cohesion computed by thesaural relations asan indicator of the structure of text. Computational Linguistics, 17(1):21–48,1991. URL https://aclanthology.org/J91-1002. → page 5[121] P. Mota, M. Eskenazi, and L. Coheur. BeamSeg: A joint model formulti-document segmentation and topic identification. In Proceedings of the23rd Conference on Computational Natural Language Learning (CoNLL),pages 582–592, Hong Kong, China, Nov. 2019. Association forComputational Linguistics. doi:10.18653/v1/K19-1054. URLhttps://aclanthology.org/K19-1054. → pages 2, 3[122] P. Muangkammuen, S. Xu, F. Fukumoto, K. Runapongsa Saikaew, and J. Li.A neural local coherence analysis model for clarity text scoring. InProceedings of the 28th International Conference on ComputationalLinguistics, pages 2138–2143, Barcelona, Spain (Online), Dec. 2020.International Committee on Computational Linguistics.doi:10.18653/v1/2020.coling-main.194. URLhttps://aclanthology.org/2020.coling-main.194. → page 77[123] S. Narayan, S. B. Cohen, and M. Lapata. Don’t give me the details, just thesummary! topic-aware convolutional neural networks for extremesummarization. In Proceedings of the 2018 Conference on EmpiricalMethods in Natural Language Processing, pages 1797–1807, Brussels,Belgium, Oct.-Nov. 2018. Association for Computational Linguistics.doi:10.18653/v1/D18-1206. URLhttps://aclanthology.org/D18-1206. → page 141[124] N. Nishida and Y. Matsumoto. Out-of-domain discourse dependency parsingvia bootstrapping: An empirical analysis on its effectiveness and limitation.160Transactions of the Association for Computational Linguistics, 10:127–144,2022. doi:10.1162/tacl_a_00451. URLhttps://aclanthology.org/2022.tacl-1.8. → page 83[125] N. Nishida and H. Nakayama. Unsupervised discourse constituency parsingusing Viterbi EM. Transactions of the Association for ComputationalLinguistics, 8:215–230, 2020. doi:10.1162/tacl_a_00312. URLhttps://aclanthology.org/2020.tacl-1.15. → page 63[126] H. Oh, S. H. Myaeng, and M.-G. Jang. Semantic passage segmentationbased on sentence topics for question answering. Information Sciences, 177(18):3696–3717, 2007. → page 3[127] OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt,2023. → pages 101, 106, 141[128] OpenAI, :, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L.Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila,I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian,J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner,L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks,M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey,C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen,S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W.Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry,N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning,A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P.Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel,T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray,R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He,M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele,B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang,A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan,Łukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan,L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, H. Kirchner, J. Kiros, M. Knight,D. Kokotajlo, Łukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic,G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy,C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue,A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin,K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey,P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz,A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu,161M. Murati, O. Murk, D. Me´ly, A. Nair, R. Nakano, R. Nayak,A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachocki,A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita,A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres,M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. Pong,T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae,A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted,H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry,H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard,T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens,J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P.Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. Thompson, P. Tillet,A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe,A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang,B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng,L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong,L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan,W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang,W. Zhuk, and B. Zoph. Gpt-4 technical report, 2023. → page 141[129] L. Pandia, Y. Cong, and A. Ettinger. Pragmatic competence of pre-trainedlanguage models through the lens of discourse connectives. In Proceedingsof the 25th Conference on Computational Natural Language Learning,pages 367–379, Online, Nov. 2021. Association for ComputationalLinguistics. doi:10.18653/v1/2021.conll-1.29. URLhttps://aclanthology.org/2021.conll-1.29. → page 17[130] L. Pevzner and M. A. Hearst. A critique and improvement of an evaluationmetric for text segmentation. Computational Linguistics, 28(1):19–36, 2002.doi:10.1162/089120102317341756. URLhttps://aclanthology.org/J02-1002. → pages 10, 51, 87, 104, 129[131] L. Polanyi. A formal model of the structure of discourse. Journal ofPragmatics, 12(5):601–638, 1988. ISSN 0378-2166.doi:https://doi.org/10.1016/0378-2166(88)90050-1. URLhttps://www.sciencedirect.com/science/article/pii/0378216688900501. → page 141[132] C. Popa and T. Rebedea. BART-TL: Weakly-supervised topic labelgeneration. In Proceedings of the 16th Conference of the European Chapterof the Association for Computational Linguistics: Main Volume, pages1418–1425, Online, Apr. 2021. Association for Computational Linguistics.162doi:10.18653/v1/2021.eacl-main.121. URLhttps://aclanthology.org/2021.eacl-main.121. → page 141[133] R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, andB. Webber. The Penn Discourse TreeBank 2.0. In Proceedings of the SixthInternational Conference on Language Resources and Evaluation(LREC’08), Marrakech, Morocco, May 2008. European LanguageResources Association (ELRA). URL http://www.lrec-conf.org/proceedings/lrec2008/pdf/754_paper.pdf. → page 59[134] M. Purver, K. P. Ko¨rding, T. L. Griffiths, and J. B. Tenenbaum.Unsupervised topic modelling for multi-party spoken discourse. InProceedings of the 21st International Conference on ComputationalLinguistics and 44th Annual Meeting of the Association for ComputationalLinguistics, pages 17–24, Sydney, Australia, July 2006. Association forComputational Linguistics. doi:10.3115/1220175.1220178. URLhttps://aclanthology.org/P06-1003. → page 41[135] J. Qiu, J. Zhu, M. Xu, F. Dernoncourt, T. Bui, Z. Wang, B. Li, D. Zhao, andH. Jin. Semantics-consistent cross-domain summarization via optimaltransport alignment, 2022. URLhttps://arxiv.org/abs/2210.04722. → page 116[136] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever.Language models are unsupervised multitask learners. 2019. → pages17, 58, 99, 106[137] R. Ramamonjison, T. Yu, L. Xing, M. Mostajabdaveh, X. Li, X. Fu, X. Han,Y. Chen, R. Li, K. Mao, and Y. Zhang. LaTeX2Solver: a hierarchicalsemantic parsing of LaTeX document into code for an assistive optimizationmodeling application. In Proceedings of the 61st Annual Meeting of theAssociation for Computational Linguistics (Volume 3: SystemDemonstrations), pages 471–478, Toronto, Canada, July 2023. Associationfor Computational Linguistics. doi:10.18653/v1/2023.acl-demo.45.URL https://aclanthology.org/2023.acl-demo.45. → page 181[138] A. Rao, L. Xu, Y. Xiong, G. Xu, Q. Huang, B. Zhou, and D. Lin. Alocal-to-global approach to multi-modal movie scene segmentation. InProceedings of the 2020 IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 10146–10155, 2020. → pages116, 117, 128, 131, 139163[139] Z. Rasheed and M. Shah. Scene detection in hollywood movies and tvshows. In Proceedings of the 2003 IEEE Computer Society Conference onComputer Vision and Pattern Recognition, pages II–343, 2003.doi:10.1109/CVPR.2003.1211489. → pages 116, 117[140] T. Reinhart. Conditions for text coherence. Poetics Today, 1(4):161–180,1980. ISSN 03335372, 15275507. URLhttp://www.jstor.org/stable/1771893. → page 4[141] M. Riedl and C. Biemann. TopicTiling: A text segmentation algorithmbased on LDA. In Proceedings of ACL 2012 Student Research Workshop,pages 37–42, Jeju Island, Korea, July 2012. Association for ComputationalLinguistics. URL https://aclanthology.org/W12-3307. → page 6[142] Y. Rui, T. S. Huang, and S. Mehrotra. Exploring video structure beyond theshots. In Proceedings of the IEEE International Conference on MultimediaComputing and Systems, pages 237–240, 1998. → pages 116, 117[143] A. Sahoo, R. Shah, R. Panda, K. Saenko, and A. Das. Contrast and mix:Temporal contrastive video domain adaptation with background mixing.Advances in Neural Information Processing Systems, 34, 2021. → page 118[144] B. Santra, S. Roychowdhury, A. Mandal, V. Gurram, A. Naik, M. Gupta, andP. Goyal. Representation learning for conversational data using discoursemutual information maximization. In Proceedings of the 2022 Conference ofthe North American Chapter of the Association for ComputationalLinguistics: Human Language Technologies, pages 1718–1734, Seattle,United States, July 2022. Association for Computational Linguistics.doi:10.18653/v1/2022.naacl-main.124. URLhttps://aclanthology.org/2022.naacl-main.124. → page 106[145] M. Schuster and K. K. Paliwal. Bidirectional recurrent neural networks.IEEE Transactions on Signal Processing, 45:2673–2681, 1997. → page 23[146] I. Sehikh, D. Fohr, and I. Illina. Topic segmentation in asr transcripts usingbidirectional rnns for change detection. In 2017 IEEE Automatic SpeechRecognition and Understanding Workshop (ASRU), 2017. → page 23[147] V. Shiv and C. Quirk. Novel positional encodings to enable tree-basedtransformers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche´-Buc,E. Fox, and R. Garnett, editors, Advances in Neural Information ProcessingSystems, volume 32. Curran Associates, Inc., 2019. URLhttps://proceedings.neurips.cc/paper/2019/file/6e0917469214d8fbd8c517dcdc6b8dcf-Paper.pdf. → page 61164[148] R. Sklar. Film: An international history of the medium. Thames andHudson, 1990. → page 117[149] A. Solbiati, K. Heffernan, G. Damaskinos, S. Poddar, S. Modi, and J. Cali.Unsupervised topic segmentation of meetings with bert embeddings, 2021.→ pages 93, 94, 100[150] Y. Song, J. Vallmitjana, A. Stent, and A. Jaimes. Tvsum: Summarizing webvideos using titles. In Proceeding of the 2015 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), pages 5179–5187, 2015.doi:10.1109/CVPR.2015.7299154. → page 126[151] Y. Song, L. Mou, R. Yan, L. Yi, Z. Zhu, X. Hu, and M. Zhang. Dialoguesession segmentation by embedding-enhanced texttiling. In INTERSPEECH,page 2706–2710, 2016. → pages 9, 41, 43, 51, 53[152] M. Soni, B. Spillane, L. Muckley, O. Cooney, E. Gilmartin, C. Saam,B. Cowan, and V. Wade. An empirical study of topic transition in dialogue.In Proceedings of the 3rd Workshop on Computational Approaches toDiscourse, pages 92–99, Gyeongju, Republic of Korea and Online, Oct.2022. International Conference on Computational Linguistics. URLhttps://aclanthology.org/2022.codi-1.12. → page 94[153] T. Soucˇek and J. Lokocˇ. Transnet v2: An effective deep network architecturefor fast shot transition detection. arXiv preprint arXiv:2008.04838, 2020. →pages 128, 131[154] C. Sporleder and A. Lascarides. Combining hierarchical clustering andmachine learning to predict high-level discourse structure. In COLING2004: Proceedings of the 20th International Conference on ComputationalLinguistics, pages 43–49, Geneva, Switzerland, aug 23–aug 27 2004.COLING. URL https://aclanthology.org/C04-1007. → pages69, 70[155] M. Stede. Discourse Processing. Morgan Claypool Publishers, 2011. ISBN1608457346. → pages 1, 2, 5[156] M. Stede. Discourse processing. Synthesis Lectures on Human LanguageTechnologies, 4(3):1–165, 2011. → pages 4, 16, 60, 61[157] R. Subba and B. Di Eugenio. An effective discourse parser that uses richlinguistic information. In Proceedings of Human Language Technologies:The 2009 Annual Conference of the North American Chapter of the165Association for Computational Linguistics, pages 566–574, Boulder,Colorado, June 2009. Association for Computational Linguistics. URLhttps://aclanthology.org/N09-1064. → page 62[158] R. Takanobu, M. Huang, Z. Zhao, F. Li, H. Chen, X. Zhu, and L. Nie. Aweakly supervised method for topic segmentation and labeling ingoal-oriented dialogues via reinforcement learning. In Proceedings of theTwenty-Seventh International Joint Conference on Artificial Intelligence,IJCAI-18, pages 4403–4410. International Joint Conferences on ArtificialIntelligence Organization, 7 2018. doi:10.24963/ijcai.2018/612.URL https://doi.org/10.24963/ijcai.2018/612. → pages9, 42, 44, 94, 140[159] D. Tien Nguyen and S. Joty. A neural local coherence model. InProceedings of the 55th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers), pages 1320–1330,Vancouver, Canada, July 2017. Association for Computational Linguistics.doi:10.18653/v1/P17-1121. URLhttps://aclanthology.org/P17-1121. → page 5[160] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix,B. Rozie`re, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin,E. Grave, and G. Lample. Llama: Open and efficient foundation languagemodels, 2023. → page 141[161] A. van den Oord, Y. Li, and O. Vinyals. Representation learning withcontrastive predictive coding. ArXiv, abs/1807.03748, 2018. → page 118[162] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V.Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, andR. Garnett, editors, Advances in Neural Information Processing Systems,volume 30, 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. →page 121[163] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,L. u. Kaiser, and I. Polosukhin. Attention is all you need. In Advances inNeural Information Processing Systems 30, pages 5998–6008. 2017. →pages 31, 105[164] P. Velicˇkovic´, G. Cucurull, A. Casanova, A. Romero, P. Lio`, and Y. Bengio.Graph Attention Networks. International Conference on Learning166Representations, 2018. URLhttps://openreview.net/forum?id=rJXMpikCZ. accepted as poster.→ pages 79, 83, 88[165] L. Wang, S. Li, Y. Lv, and H. Wang. Learning to rank semantic coherencefor topic segmentation. In Proceedings of the 2017 Conference on EmpiricalMethods in Natural Language Processing, pages 1340–1344, Copenhagen,Denmark, Sept. 2017. Association for Computational Linguistics.doi:10.18653/v1/D17-1139. URLhttps://aclanthology.org/D17-1139. → pages23, 24, 25, 42, 44, 45[166] P. Wang, Y. Qian, F. K. Soong, L. He, and H. Zhao. Learning distributedword representations for bidirectional LSTM recurrent neural network. InProceedings of the 2016 Conference of the North American Chapter of theAssociation for Computational Linguistics: Human Language Technologies,pages 527–533, San Diego, California, June 2016. Association forComputational Linguistics. doi:10.18653/v1/N16-1064. URLhttps://aclanthology.org/N16-1064. → pages 7, 23[167] W. Wang and B. Chang. Graph-based dependency parsing with bidirectionalLSTM. In Proceedings of the 54th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers), pages 2306–2315,Berlin, Germany, Aug. 2016. Association for Computational Linguistics.doi:10.18653/v1/P16-1218. URLhttps://aclanthology.org/P16-1218. → page 30[168] W. Wang, S. C. Hoi, and S. Joty. Response selection for multi-partyconversations with dynamic topic tracking. In Proceedings of the 2020Conference on Empirical Methods in Natural Language Processing(EMNLP), pages 6581–6591, Online, Nov. 2020. Association forComputational Linguistics. doi:10.18653/v1/2020.emnlp-main.533.URL https://aclanthology.org/2020.emnlp-main.533. → page42[169] X. Wang, C. Li, J. Zhao, and D. Yu. Naturalconv: A chinese dialogue datasettowards multi-turn topic-driven conversation. Proceedings of the AAAIConference on Artificial Intelligence, 35(16):14006–14014, May 2021. URLhttps://ojs.aaai.org/index.php/AAAI/article/view/17649.→ pages 42, 46[170] Y. Wang and M. Guo. A short analysis of discourse coherence. Journal of167Language Teaching and Research, 5, 03 2014.doi:10.4304/jltr.5.2.460-465. → page 4[171] Y. Wang, S. Li, and H. Wang. A two-stage parsing method for text-leveldiscourse analysis. In Proceedings of the 55th Annual Meeting of theAssociation for Computational Linguistics (Volume 2: Short Papers), pages184–188, Vancouver, Canada, July 2017. Association for ComputationalLinguistics. doi:10.18653/v1/P17-2029. URLhttps://aclanthology.org/P17-2029. → pages61, 62, 68, 71, 80, 176, 177, 178, 179[172] Y. Wang, S. Li, and J. Yang. Toward fast and accurate neural discoursesegmentation. In Proceedings of the 2018 Conference on Empirical Methodsin Natural Language Processing, pages 962–967, Brussels, Belgium,Oct.-Nov. 2018. Association for Computational Linguistics.doi:10.18653/v1/D18-1116. URLhttps://aclanthology.org/D18-1116. → pagesxvi, xvii, 23, 24, 26, 31, 175, 176, 177, 178, 179[173] Z. Wang, Y. Zhong, Y. Miao, L. Ma, and L. Specia. Contrastivevideo-language learning with fine-grained frame sampling. In Proceedingsof the 2nd Conference of the Asia-Pacific Chapter of the Association forComputational Linguistics and the 12th International Joint Conference onNatural Language Processing (Volume 1: Long Papers), pages 694–705,Online only, Nov. 2022. Association for Computational Linguistics. URLhttps://aclanthology.org/2022.aacl-main.53. → page 121[174] B. Webber, M. Stone, A. Joshi, and A. Knott. Anaphora and discoursestructure. Computational Linguistics, 29(4):545–587, 2003.doi:10.1162/089120103322753347. URLhttps://aclanthology.org/J03-4002. → pages 16, 78[175] B. Webber, M. Stone, A. Joshi, and A. Knott. Anaphora and discoursestructure. Computational Linguistics, 29(4):545–587, 2003.doi:10.1162/089120103322753347. → page 80[176] X. Wei, X. Cui, N. Cheng, X. Wang, X. Zhang, S. Huang, P. Xie, J. Xu,Y. Chen, M. Zhang, Y. Jiang, and W. Han. Zero-shot information extractionvia chatting with chatgpt, 2023. → page 101[177] C.-S. Wu, S. C. Hoi, R. Socher, and C. Xiong. TOD-BERT: Pre-trainednatural language understanding for task-oriented dialogue. In Proceedings ofthe 2020 Conference on Empirical Methods in Natural Language Processing168(EMNLP), pages 917–929, Online, Nov. 2020. Association forComputational Linguistics. doi:10.18653/v1/2020.emnlp-main.66.URL https://aclanthology.org/2020.emnlp-main.66. → page105[178] H. Wu, K. Chen, Y. Luo, R. Qiao, B. Ren, H. Liu, W. Xie, and L. Shen.Scene consistency representation learning for video scene segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition (CVPR), pages 14021–14030, June 2022. → page 120[179] Z. Wu, Y. Chen, B. Kao, and Q. Liu. Perturbed masking: Parameter-freeprobing for analyzing and interpreting BERT. In Proceedings of the 58thAnnual Meeting of the Association for Computational Linguistics, pages4166–4176, Online, July 2020. Association for Computational Linguistics.doi:10.18653/v1/2020.acl-main.383. URLhttps://aclanthology.org/2020.acl-main.383. → page 17[180] S. Xiao, L. Chen, S. Zhang, W. Ji, J. Shao, L. Ye, and J. Xiao. Boundaryproposal network for two-stage natural language video localization. InProceedings of the 2021 AAAI Conference on Artificial Intelligence(AAAI-21), 2021. → page 116[181] W. Xiao and G. Carenini. Extractive summarization of long documents bycombining global and local context. In Proceedings of the 2019 Conferenceon Empirical Methods in Natural Language Processing and the 9thInternational Joint Conference on Natural Language Processing(EMNLP-IJCNLP), pages 3011–3021, Hong Kong, China, Nov. 2019.Association for Computational Linguistics.doi:10.18653/v1/D19-1298. URLhttps://aclanthology.org/D19-1298. → pages 3, 13, 140[182] W. Xiao, P. Huber, and G. Carenini. Predicting discourse trees fromtransformer-based neural summarizers. In Proceedings of the 2021Conference of the North American Chapter of the Association forComputational Linguistics: Human Language Technologies, pages4139–4152, Online, June 2021. Association for Computational Linguistics.doi:10.18653/v1/2021.naacl-main.326. URLhttps://aclanthology.org/2021.naacl-main.326. → pages61, 63, 66, 68, 71, 176, 177, 178, 179[183] W. Xiao, Y. Xie, G. Carenini, and P. He. Chatgpt-steered editing instructorfor customization of abstractive summarization, 2023. → page 101169[184] L. Xing and G. Carenini. Improving unsupervised dialogue topicsegmentation with utterance-pair coherence scoring. In Proceedings of the22nd Annual Meeting of the Special Interest Group on Discourse andDialogue, pages 167–177, Singapore and Online, July 2021. Association forComputational Linguistics. URLhttps://aclanthology.org/2021.sigdial-1.18. → pagesvi, 40, 93, 94, 100[185] L. Xing and M. Paul. Diagnosing and improving topic models by analyzingposterior variability. Proceedings of the AAAI Conference on ArtificialIntelligence, 32(1), Apr. 2018. doi:10.1609/aaai.v32i1.12033. URLhttps://ojs.aaai.org/index.php/AAAI/article/view/12033.→ page 2[186] L. Xing and M. J. Paul. Incorporating metadata into content-based userembeddings. In Proceedings of the 3rd Workshop on Noisy User-generatedText, pages 45–49, Copenhagen, Denmark, Sept. 2017. Association forComputational Linguistics. doi:10.18653/v1/W17-4406. URLhttps://aclanthology.org/W17-4406. → page 139[187] L. Xing, M. J. Paul, and G. Carenini. Evaluating topic quality with posteriorvariability. In Proceedings of the 2019 Conference on Empirical Methods inNatural Language Processing and the 9th International Joint Conference onNatural Language Processing (EMNLP-IJCNLP), pages 3471–3477, HongKong, China, Nov. 2019. Association for Computational Linguistics.doi:10.18653/v1/D19-1349. URLhttps://aclanthology.org/D19-1349. → page 182[188] L. Xing, B. Hackinen, G. Carenini, and F. Trebbi. Improving contextmodeling in neural topic segmentation. In Proceedings of the 1st Conferenceof the Asia-Pacific Chapter of the Association for Computational Linguisticsand the 10th International Joint Conference on Natural LanguageProcessing, pages 626–636, Suzhou, China, Dec. 2020. Association forComputational Linguistics. URLhttps://aclanthology.org/2020.aacl-main.63. → pagesvi, xv, 13, 22, 40, 65, 67, 68, 79, 87, 88, 90, 91, 93, 116, 120[189] L. Xing, W. Xiao, and G. Carenini. Demoting the lead bias in newssummarization via alternating adversarial learning. In Proceedings of the59th Annual Meeting of the Association for Computational Linguistics andthe 11th International Joint Conference on Natural Language Processing(Volume 2: Short Papers), pages 948–954, Online, Aug. 2021. Association170for Computational Linguistics.doi:10.18653/v1/2021.acl-short.119. URLhttps://aclanthology.org/2021.acl-short.119. → page 182[190] L. Xing, P. Huber, and G. Carenini. Improving topic segmentation byinjecting discourse dependencies. In Proceedings of the 3rd Workshop onComputational Approaches to Discourse, pages 7–18, Gyeongju, Republicof Korea and Online, Oct. 2022. International Conference on ComputationalLinguistics. URL https://aclanthology.org/2022.codi-1.2. →pages vii, 77[191] L. Xing, B. Hackinen, and G. Carenini. Tracing influence at scale: Acontrastive learning approach to linking public comments and regulatorresponses. In D. Preot,iuc-Pietro, C. Goanta, I. Chalkidis, L. Barrett, G. J.Spanakis, and N. Aletras, editors, Proceedings of the Natural LegalLanguage Processing Workshop 2023, pages 266–274, Singapore, Dec.2023. Association for Computational Linguistics.doi:10.18653/v1/2023.nllp-1.26. URLhttps://aclanthology.org/2023.nllp-1.26. → page 180[192] L. Xing, Q. Tran, F. Caba, F. Dernoncourt, S. Yoon, Z. Wang, T. Bui, andG. Carenini. Multi-modal video topic segmentation with dual-contrastivedomain adaptation. In S. Rudinac, A. Hanjalic, C. Liem, M. Worring,B. Jo´nsson, B. Liu, and Y. Yamakata, editors, MultiMedia Modeling, pages410–424, Cham, 2024. Springer Nature Switzerland. → pages vii, 114[193] J. Xu, Z. Gan, Y. Cheng, and J. Liu. Discourse-aware neural extractive textsummarization. In Proceedings of the 58th Annual Meeting of theAssociation for Computational Linguistics, pages 5021–5031, Online, July2020. Association for Computational Linguistics.doi:10.18653/v1/2020.acl-main.451. URLhttps://aclanthology.org/2020.acl-main.451. → page 83[194] P. Xu, H. Saghir, J. S. Kang, T. Long, A. J. Bose, Y. Cao, and J. C. K.Cheung. A cross-domain transferable neural coherence model. InProceedings of the 57th Annual Meeting of the Association forComputational Linguistics, pages 678–687, Florence, Italy, July 2019.Association for Computational Linguistics.doi:10.18653/v1/P19-1067. URLhttps://aclanthology.org/P19-1067. → pages 42, 44, 45[195] Y. Xu, H. Zhao, and Z. Zhang. Topic-aware multi-turn dialogue modeling.Proceedings of the AAAI Conference on Artificial Intelligence, 35(16):17114176–14184, May 2021. URLhttps://ojs.aaai.org/index.php/AAAI/article/view/17668.→ pages xiv, 3, 9, 41, 43, 49, 50, 51, 53, 93, 94, 96, 100, 140[196] A. Yang and S. Li. SciDTB: Discourse dependency TreeBank for scientificabstracts. In Proceedings of the 56th Annual Meeting of the Association forComputational Linguistics (Volume 2: Short Papers), pages 444–449,Melbourne, Australia, July 2018. Association for Computational Linguistics.doi:10.18653/v1/P18-2071. URLhttps://aclanthology.org/P18-2071. → page 83[197] Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy. Hierarchicalattention networks for document classification. In Proceedings of the 2016Conference of the North American Chapter of the Association forComputational Linguistics: Human Language Technologies, pages1480–1489, San Diego, California, June 2016. Association forComputational Linguistics. doi:10.18653/v1/N16-1174. URLhttps://aclanthology.org/N16-1174. → page 28[198] N. Yu, M. Zhang, and G. Fu. Transition-based neural RST parsing withimplicit syntax features. In Proceedings of the 27th InternationalConference on Computational Linguistics, pages 559–570, Santa Fe, NewMexico, USA, Aug. 2018. Association for Computational Linguistics. URLhttps://aclanthology.org/C18-1047. → pages 62, 80[199] A. Zeldes. The GUM corpus: Creating multilayer resources in theclassroom. Language Resources and Evaluation, 51(3):581–612, 2017.doi:http://dx.doi.org/10.1007/s10579-016-9343-x. → pages62, 67, 83[200] H. Zhang, Y. Lan, L. Pang, H. Chen, Z. Ding, and D. Yin. Modeling topicalrelevance for multi-turn dialogue generation. In Proceedings of theTwenty-Ninth International Joint Conference on Artificial Intelligence,IJCAI-20, pages 3737–3743. International Joint Conferences on ArtificialIntelligence Organization, 2020. → pages 3, 140[201] L. Zhang and Q. Zhou. Topic segmentation for dialogue stream. In 2019Asia-Pacific Signal and Information Processing Association Annual Summitand Conference (APSIPA ASC), pages 1036–1043, 2019.doi:10.1109/APSIPAASC47483.2019.9023126. → page 94[202] R. Zhang, J. Guo, Y. Fan, Y. Lan, and X. Cheng. Outline generation:Understanding the inherent content structure of documents. In Proceedings172of the 42nd International ACM SIGIR Conference on Research andDevelopment in Information Retrieval, SIGIR’19, page 745–754, New York,NY, USA, 2019. Association for Computing Machinery. ISBN9781450361729. doi:10.1145/3331184.3331208. URLhttps://doi.org/10.1145/3331184.3331208. → page 140[203] Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu,and B. Dolan. DIALOGPT : Large-scale generative pre-training forconversational response generation. In Proceedings of the 58th AnnualMeeting of the Association for Computational Linguistics: SystemDemonstrations, pages 270–278, Online, July 2020. Association forComputational Linguistics. doi:10.18653/v1/2020.acl-demos.30.URL https://aclanthology.org/2020.acl-demos.30. → page 106[204] M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. H. Awadallah,A. Celikyilmaz, Y. Liu, X. Qiu, and D. Radev. QMSum: A new benchmarkfor query-based multi-domain meeting summarization. In Proceedings of the2021 Conference of the North American Chapter of the Association forComputational Linguistics: Human Language Technologies, pages5905–5921, Online, June 2021. Association for Computational Linguistics.doi:10.18653/v1/2021.naacl-main.472. URLhttps://aclanthology.org/2021.naacl-main.472. → page 98[205] Y. Zhou and Y. Feng. Improve discourse dependency parsing withcontextualized representations. In Findings of the Association forComputational Linguistics: NAACL 2022, pages 2250–2261, Seattle, UnitedStates, July 2022. Association for Computational Linguistics.doi:10.18653/v1/2022.findings-naacl.173. URLhttps://aclanthology.org/2022.findings-naacl.173. → pagesxiv, xx, 78, 80, 82, 83[206] Z. Zhou, D. Zhang, W. Xiao, N. Dingwall, X. Ma, A. Arnold, and B. Xiang.Learning dialogue representations from consecutive utterances. InProceedings of the 2022 Conference of the North American Chapter of theAssociation for Computational Linguistics: Human Language Technologies,pages 754–768, Seattle, United States, July 2022. Association forComputational Linguistics. doi:10.18653/v1/2022.naacl-main.55.URL https://aclanthology.org/2022.naacl-main.55. → page105[207] C. Zhu, Y. Liu, J. Mei, and M. Zeng. MediaSum: A large-scale mediainterview dataset for dialogue summarization. In Proceedings of the 2021173Conference of the North American Chapter of the Association forComputational Linguistics: Human Language Technologies, pages5927–5934, Online, June 2021. Association for Computational Linguistics.doi:10.18653/v1/2021.naacl-main.474. URLhttps://aclanthology.org/2021.naacl-main.474. → page 106[208] W. Zhu, B. Pang, A. V. Thapliyal, W. Y. Wang, and R. Soricut. End-to-enddense video captioning as sequence generation. In Proceedings of the 29thInternational Conference on Computational Linguistics, pages 5651–5665,Gyeongju, Republic of Korea, Oct. 2022. International Committee onComputational Linguistics. URLhttps://aclanthology.org/2022.coling-1.498. → page 129[209] Z. Zhu, C. Pan, M. Abdalla, and F. Rudzicz. Examining the rhetoricalcapacities of neural language models. In Proceedings of the ThirdBlackboxNLP Workshop on Analyzing and Interpreting Neural Networks forNLP, pages 16–32, Online, Nov. 2020. Association for ComputationalLinguistics. doi:10.18653/v1/2020.blackboxnlp-1.3. URLhttps://aclanthology.org/2020.blackboxnlp-1.3. → page 17[210] B. Zoph, C. Raffel, D. Schuurmans, D. Yogatama, D. Zhou, D. Metzler,E. H. Chi, J. Wei, J. Dean, L. B. Fedus, M. P. Bosma, O. Vinyals, P. Liang,S. Borgeaud, T. B. Hashimoto, and Y. Tay. Emergent abilities of largelanguage models. TMLR, 2022. → page 101174Appendix APredicting Above-SentenceDiscourse Structure using DistantSupervision from TopicSegmentationA.1 Complete Quantitative ResultsWe show the complete quantitative results, including the previously shown Sentence-to-Paragraph (S-P), Paragraph-to-Document (P-D) and Sentence-to-Document (S-D) performance, as well as the additional EDU-to-Sentence (E-S), EDU-to-Paragraph(E-P), EDU-to-Document (E-D) and EDU-to-Sentence-to-Document (E-S-D) sub-tree results here. Tables A.1 and A.2 contain the RST parseval scores on the RST-DT and GUM corpus, respectively. Table A.3 contains the full results of the originalparseval score applied to RST-DT and Table A.4 shows the original parseval per-formance on GUM. Besides the additional performance evaluations also coveringlow-level structures for the baseline models (E-S, E-P, E-D and E-S-D), we show anadditional model, combining the out-of-the-box discourse segmenter by Wang et al.[172] with our above-sentence tree structures generated from topic-segmentation inthe last row of Tables A.1, A.2, A.3 and A.4 (Combined – TSWiki/Disc-Seg). Since we175combine strictly intra-sentence structures from the discourse segmenter with strictlyabove-sentence trees, this approach does not support leaky EDUs, and hence consti-tutes E-S-D structures (as compared to non sentence limited E-D trees).Model E-S S-P P-D E-P S-D E-D E-S-DBaselinesRandom* 70.08 77.11 63.90 64.05 60.20 58.03 70.10Right-Branching 64.12 73.57 65.50 58.41 59.46 54.64 74.37Left-Branching 63.85 72.41 64.07 57.73 58.07 53.73 70.58Supervised RST-style ParsersTop-DownRST-DT [87] × × × × × 86.10 86.40Two-StageGUM [171] 92.57 88.82 65.63 88.62 69.58 81.24 ×Two-StageRST-DT [171] 94.48 90.64 68.09 91.06 72.11 83.90 ×SpanBERTRST-DT [50] 96.27 90.75 76.03 92.87 77.19 87.69 ×Distantly Supervised RST-style ParsersXiao et al. [182]CNN/DM 85.95 74.23 66.15 78.19 59.10 60.68 75.13Two-StageMEGA-DT [65] 89.57 85.00 65.50 84.73 66.99 77.90 ×TSRST-DT × 84.34 62.52 × 65.96 × ×TSWiki × 83.43 69.78 × 68.13 × ×TSWiki+RST-DT × 83.84 66.54 × 65.84 × ×Ablation – TSWiki × 83.51 68.61 × 67.47 × ×Combined – TSWiki/Disc-Seg 87.10 83.43 69.78 81.67 68.13 × 75.56Table A.1: Evaluation results using the RST Parseval micro-average pre-cision measure on the RST-DT dataset. Subscripts indicate trainingdataset. TS=Topic Segmentation Model. Disc-Seg=Discourse Segmen-tation Model by Wang et al. [172]. ×=Not feasible combination. *=Aver-age performance over 10 runs. Best performance per sub-table underlined,best performance per column bold176Model E-S S-P P-D E-P S-D E-D E-S-DBaselinesRandom* 72.16 67.53 60.96 61.60 57.99 57.24 69.93Right-Branching 66.30 64.15 72.71 56.23 59.39 54.71 71.58Left-Branching 65.96 62.07 54.35 55.08 51.56 50.77 64.57Supervised RST-style ParsersTwo-StageRST-DT [171] 93.51 74.20 63.29 82.74 63.65 77.16 ×Two-StageGUM [171] 93.25 76.70 72.94 83.49 68.38 79.04 ×Distantly Supervised RST-style ParsersXiao et al. [182]CNN-DM 88.09 67.89 57.80 76.52 53.82 59.28 72.39Two-StageMEGA-DT [65] 88.78 73.37 69.88 78.35 64.69 73.89 ×TSGUM × 72.54 67.60 × 62.79 × ×TSWiki × 76.98 63.53 × 65.84 × ×TSWiki+GUM × 74.48 67.29 × 64.69 × ×Ablation – TSWiki × 75.94 64.71 × 65.38 × ×Combined – TSWiki/Disc-Seg 88.58 76.98 63.53 79.30 65.84 × 73.94Table A.2: Evaluation results using the RST Parseval micro-average pre-cision measure on the GUM dataset. Subscripts indicate trainingdataset. TS=Topic Segmentation Model. Disc-Seg=Discourse Segmen-tation Model by Wang et al. [172]. ×=Not feasible combination. *=Aver-age performance over 10 runs. Best performance per sub-table underlined,best performance per column bold177Model E-S S-P P-D E-P S-D E-D E-S-DBaselinesRandom* 19.35 22.55 30.61 17.13 21.83 16.06 41.07Right-Branching 3.25 7.27 31.62 4.12 19.21 9.27 48.74Left-Branching 2.53 3.20 28.79 2.56 16.45 7.45 41.16Supervised RST-style ParsersMDParser – TSRST-DT [77] × × 40.52 × × × ×Two-StageGUM [171] 79.97 60.76 31.88 73.78 39.38 62.48 ×Two-StageRST-DT [171] 85.10 67.44 36.76 79.40 44.54 70.97 ×SpanBERTRST-DT [50] 90.16 68.69 52.75 83.83 54.37 75.39 ×Distantly Supervised RST-style ParsersXiao et al. [182]CNN-DM 62.11 9.59 32.09 49.72 18.49 21.36 49.21Two-StageMEGA-DT [65] 71.87 47.38 31.62 64.81 34.21 55.81 ×TSRST-DT × 45.06 28.02 × 32.17 × ×TSWiki × 41.86 41.90 × 36.49 × ×TSWiki+RST-DT × 43.31 34.96 × 31.93 × ×Ablation – TSWiki × 42.15 39.07 × 35.17 × ×Combined – TSWiki/Disc-Seg 65.22 41.86 41.90 57.74 36.49 × 50.95Table A.3: Evaluation results using the original Parseval micro-average pre-cision measure on the RST-DT dataset. Subscripts indicate trainingdataset. TS=Topic Segmentation Model. Disc-Seg=Discourse Segmen-tation Model by Wang et al. [172]. ×=Not feasible combination. *=Aver-age performance over 10 runs. Best performance per sub-table underlined,best performance per column bold178Model E-S S-P P-D E-P S-D E-D E-S-DBaselinesRandom* 18.45 17.83 25.46 15.25 16.71 14.49 43.11Right-Branching 1.28 7.35 46.30 3.39 18.78 9.42 43.15Left-Branching 0.30 1.97 10.19 0.86 3.11 1.54 29.15Supervised RST-style ParsersTwo-StageRST-DT [171] 81.00 33.69 27.78 61.90 27.53 54.33 ×Two-StageGUM [171] 80.22 39.78 46.76 63.56 36.75 58.08 ×Distantly Supervised RST-style ParsersXiao et al. [182]CNN-DM 65.72 7.60 16.75 46.69 7.64 18.56 45.97Two-StageMEGA-DT [65] 67.13 31.36 40.74 52.22 29.49 47.79 ×TSGUM × 29.03 37.50 × 25.58 × ×TSWiki × 40.50 31.02 × 31.68 × ×TSWiki+GUM × 34.05 37.04 × 29.38 × ×Ablation – TSWiki × 37.81 33.33 × 30.76 × ×Combined – TSWiki/Disc-Seg 66.54 40.50 31.02 54.32 31.68 × 47.81Table A.4: Evaluation results using the original Parseval micro-average pre-cision measure on the GUM dataset. Subscripts indicate trainingdataset. TS=Topic Segmentation Model. Disc-Seg=Discourse Segmen-tation Model by Wang et al. [172]. ×=Not feasible combination. *=Aver-age performance over 10 runs. Best performance per sub-table underlined,best performance per column bold179Appendix BAdditional WorksApart from the research presented in this thesis, I have also undertaken additional re-search related to discourse parsing, visualization, summarization, sentiment analysisand legal NLP. A comprehensive list of these studies, along with their titles, venuespublished and abstracts, is presented below (with ∗ indicating the first authorship):• Tracing Influence at Scale: A Contrastive Learning Approach to LinkingPublic Comments and Regulator Responses∗ [191]NLLP Workshop at EMNLP 2023Abstract: U.S. Federal Regulators receive over one million comment letters eachyear from businesses, interest groups, and members of the public, all advocatingfor changes to proposed regulations. These comments are believed to have wide-ranging impacts on public policy. However, measuring the impact of specificcomments is challenging because regulators are required to respond to commentsbut they do not have to specify which comments they are addressing. In this paper,we propose a simple yet effective solution to this problem by using an iterativecontrastive method to train a neural model aiming for matching text from publiccomments to responses written by regulators. We demonstrate that our proposalsubstantially outperforms a set of selected text-matching baselines on a human-annotated test set. Furthermore, it delivers performance comparable to the mostadvanced gigantic language model (i.e., GPT-4), and is more cost-effective whenhandling comments and regulator responses matching in larger scale.• Diversity-Aware Coherence Loss for Improving Neural Topic Models [102]180ACL 2023: ShortAbstract: The standard approach for neural topic modeling uses a variationalautoencoder (VAE) framework that jointly minimizes the KL divergence betweenthe estimated posterior and prior, in addition to the reconstruction loss. Sinceneural topic models are trained by recreating individual input documents, theydo not explicitly capture the coherence between words on the corpus level. Inthis work, we propose a novel diversity-aware coherence loss that encourages themodel to learn corpus-level coherence scores while maintaining high diversitybetween topics. Experimental results on multiple datasets show that our methodsignificantly improves the performance of neural topic models without requiringany pretraining or additional parameters.• LaTeX2Solver: a Hierarchical Semantic Parsing of LaTeX Document intoCode for an Assistive Optimization Modeling Application [137]ACL 2023: DemoAbstract: We demonstrate an interactive system to help operations research (OR)practitioners convert the mathematical formulation of optimization problems fromTeX document format into the solver modeling language. In practice, a manualtranslation is cumbersome and time-consuming. Moreover, it requires an in-depthunderstanding of the problem description and a technical expertise to produce themodeling code. Thus, our proposed system TeX2Solver helps partially automatethis conversion and help the users build optimization models more efficiently. Inthis paper, we describe its interface and the components of the hierarchical parsingsystem. A video demo walk-through is available online at http://bit.ly/3kuOm3x.• Human Guided Exploitation of Interpretable Attention Patterns in Summa-rization and Topic Segmentation [101]EMNLP 2022Abstract: Aiming for a better integration of data-driven and linguistically-inspiredapproaches, we explore whether RST Nuclearity, assigning a binary assessmentof importance between text segments, can be replaced by automatically gener-ated, real-valued scores, in what we call a Weighted-RST framework. In partic-ular, we find that weighted discourse trees from auxiliary tasks can benefit keyNLP downstream applications, compared to nuclearity-centered approaches. We181further show that real-valued importance distributions partially and interestinglyalign with the assessment and uncertainty of human annotators.• Demoting the Lead Bias in News Summarization via Alternating AdversarialLearning ∗ [189]ACL 2021: ShortAbstract: In news articles the lead bias is a common phenomenon that usuallydominates the learning signals for neural extractive summarizers, severely limit-ing their performance on data with different or even no bias. In this paper, weintroduce a novel technique to demote lead bias and make the summarizer focusmore on the content semantics. Experiments on two news corpora with differ-ent degrees of lead bias show that our method can effectively demote the model’slearned lead bias and improve its generality on out-of-distribution data, with littleto no performance loss on in-distribution data.• Multilingual Twitter Corpus and Baselines for Evaluating Demographic Biasin Hate Speech Recognition [63]LREC 2020Abstract: Existing research on fairness evaluation of document classificationmodels mainly uses synthetic monolingual data without ground truth for authordemographic attributes. In this work, we assemble and publish a multilingualTwitter corpus for the task of hate speech detection with inferred four author de-mographic factors: age, country, gender and race/ethnicity. The corpus coversfive languages: English, Italian, Polish, Portuguese and Spanish. We evaluate theinferred demographic labels with a crowdsourcing platform, Figure Eight. To ex-amine factors that can cause biases, we take an empirical analysis of demographicpredictability on the English corpus. We measure the performance of four populardocument classifiers and evaluate the fairness and bias of the baseline classifierson the author-level demographic attributes.• Evaluating Topic Quality with Posterior Variability ∗ [187]EMNLP 2019: ShortAbstract: Probabilistic topic models such as latent Dirichlet allocation (LDA) arepopularly used with Bayesian inference methods such as Gibbs sampling to learnposterior distributions over topic model parameters. We derive a novel measure of182LDA topic quality using the variability of the posterior distributions. Comparedto several existing baselines for automatic topic evaluation, the proposed met-ric achieves state-of-the-art correlations with human judgments of topic qualityin experiments on three corpora. We additionally demonstrate that topic qualityestimation can be further improved using a supervised estimator that combinesmultiple metrics.183
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Carenini, Giuseppe
Versatile neural approaches to more accurate and robust topic segmentation
Text