Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Ge, Amon Dongfang
Master of Science - MSc
University of British Columbia
Topic modeling is a natural language processing (NLP) task that statistically identiﬁes topics from a set of texts. Evaluating results from topic modeling is diﬃcult in context and often requires domain experts. To facilitate evaluation of topic model results within communication between NLP researchers and domain experts, we present a visual comparison framework, OCTVis, to explore results from two topic models mapped against a domain ontology. The design of OCTVis is based on detailed and abstracted data and task models. We support high-level topic model comparison by mapping topics onto ontology concepts and incorporating topic alignment visualizations. For in-depth exploration of the dataset, display of per-document topic distributions and buddy plots allow comparison of topics, texts, and shared keywords at the document level. Case studies with medical domain experts using healthcare texts indicate that our framework enhances qualitative evaluation of topic models and provide a clearer understanding of how topic models can be improved.
OCTVis: Ontology-based Comparisonof Topic ModelsbyAmon Dongfang GeB.Sc., University of British Columbia, 2016A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)April 2019c© Amon Dongfang Ge, 2019The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, a thesisentitled:OCTVis: Ontology-based Comparison of Topic Modelssubmitted by Amon Dongfang Ge in partial fulfillment of the require-ments for the degree of Master of Science in Computer Science.Examining Committee:Giuseppe Carenini, Computer ScienceSupervisorDongwook Yoon, Computer ScienceSupervisory Committee MemberiiAbstractTopic modeling is a natural language processing (NLP) task that statis-tically identifies topics from a set of texts. Evaluating results from topicmodeling is difficult in context and often requires domain experts. To facil-itate evaluation of topic model results within communication between NLPresearchers and domain experts, we present a visual comparison framework,OCTVis, to explore results from two topic models mapped against a domainontology. The design of OCTVis is based on detailed and abstracted dataand task models. We support high-level topic model comparison by mappingtopics onto ontology concepts and incorporating topic alignment visualiza-tions. For in-depth exploration of the dataset, display of per-documenttopic distributions and buddy plots allow comparison of topics, texts, andshared keywords at the document level. Case studies with medical domainexperts using healthcare texts indicate that our framework enhances qual-itative evaluation of topic models and provide a clearer understanding ofhow topic models can be improved.iiiLay SummaryTopic modeling is a statistical approach that summarizes collections of textinto topics being discussed. While topic modeling can be an effective wayof understanding large text collections (often infeasible for a human to readthrough one by one), evaluating and improving these machine results re-quires human input and interaction. Particularly, domain experts can in-form topic modeling results with the help of established domain knowledge,known as ontologies. We present an interactive visualization, OCTVis, tocompare results from topic models within the context of a domain ontology.We evaluate the effectiveness of OCTVis with medical domain experts usingonline patient discussions and a medical ontology. Through ontology-basedtopic model comparison, OCTVis enhances qualitative evaluation of topicmodels and provides a clearer understanding of how topic models can beimproved.ivPrefaceThis thesis is the original work of the author, Amon Ge. All design, experi-ments, and writing were done by Amon Ge with help from Dr. Hyeju Jangand under the supervision of Dr. Giuseppe Carenini. Dr. Giuseppe Careniniand Dr. Dongwook Yoon provided feedback for revising the manuscript.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . x1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1 Visualizing topic models for exploration . . . . . . . . . . . . 32.2 Comparison of topic models . . . . . . . . . . . . . . . . . . 62.3 Ontology visualization . . . . . . . . . . . . . . . . . . . . . . 103 Data and Task Abstractions . . . . . . . . . . . . . . . . . . . 143.1 Data model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.2 Task model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.2.1 Evaluative metrics . . . . . . . . . . . . . . . . . . . . 183.2.2 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.1 Topic alignment . . . . . . . . . . . . . . . . . . . . . . . . . 244.1.1 Encodings . . . . . . . . . . . . . . . . . . . . . . . . 244.1.2 Interactions . . . . . . . . . . . . . . . . . . . . . . . 264.2 Ontology mapping . . . . . . . . . . . . . . . . . . . . . . . . 284.2.1 Encodings . . . . . . . . . . . . . . . . . . . . . . . . 28viTable of Contents4.2.2 Interactions . . . . . . . . . . . . . . . . . . . . . . . 304.3 Document-centred comparison . . . . . . . . . . . . . . . . . 304.3.1 Encodings . . . . . . . . . . . . . . . . . . . . . . . . 304.3.2 Interactions . . . . . . . . . . . . . . . . . . . . . . . 314.4 Key considerations during the design process . . . . . . . . . 314.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . 365 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 385.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406 Discussion and Future Work . . . . . . . . . . . . . . . . . . . 437 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47A Pre-Study Questionnaire . . . . . . . . . . . . . . . . . . . . . 52B Post-Study Questionnaire . . . . . . . . . . . . . . . . . . . . 53C Case Study Sample Dataset . . . . . . . . . . . . . . . . . . . 55D Case Study Script . . . . . . . . . . . . . . . . . . . . . . . . . 56D.1 Introduction and pre-study questionnaire . . . . . . . . . . . 56D.2 User training . . . . . . . . . . . . . . . . . . . . . . . . . . . 57D.3 Main study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58D.4 Post-study questionnaire and debrief . . . . . . . . . . . . . 58viiList of Tables5.1 Case study corpora statistics . . . . . . . . . . . . . . . . . . 385.2 Pre-study and post-study questionnaire responses . . . . . . . 40viiiList of Figures2.1 UTOPIAN from Choo et. al. . . . . . . . . . . . . . . . . . . 32.2 ParallelTopics from Dou et. al. . . . . . . . . . . . . . . . . . 42.3 ThemeDelta from Gad et. al. . . . . . . . . . . . . . . . . . . 42.4 TopicPanorama from Wang et. al. . . . . . . . . . . . . . . . 52.5 MultiConVis from Hoque and Carenini . . . . . . . . . . . . . 52.6 SpecEx from El-Assady et. al. . . . . . . . . . . . . . . . . . . 62.7 Topic model comparison from Alexander and Gleicher . . . . 72.8 Comparison of papers from InfoVis, SciVis, and Siggraphfrom Oelke et. al. . . . . . . . . . . . . . . . . . . . . . . . . . 82.9 Correspondence chart between latent topics and referenceconcepts from Chuang et. al. . . . . . . . . . . . . . . . . . . 92.10 Document relevance feedback view from El-Assady et. al. . . 102.11 (Part I) Schematic examples of ontology visualizations fromDuda´sˇ et. al. . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.12 (Part II) Schematic examples of ontology visualizations fromDuda´sˇ et. al. . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.13 PhenoLines from Glueck et. al. . . . . . . . . . . . . . . . . . 133.1 A part of the UMLS semantic network . . . . . . . . . . . . . 144.1 The OCTVis interface . . . . . . . . . . . . . . . . . . . . . . 254.2 The topic alignment facet . . . . . . . . . . . . . . . . . . . . 264.3 Topic alignment linked highlighting . . . . . . . . . . . . . . . 274.4 The ontology facet . . . . . . . . . . . . . . . . . . . . . . . . 284.5 The document-topic heatmap . . . . . . . . . . . . . . . . . . 294.6 Topic coin clustering mockup . . . . . . . . . . . . . . . . . . 314.7 Whiteboard mockup of topic-centred and document-centredviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.8 Mockup of topic alignment matrices, document-topic matri-ces, and document view . . . . . . . . . . . . . . . . . . . . . 344.9 Colour binning for document keywords . . . . . . . . . . . . . 35ixAcknowledgementsI would like to extend my deepest gratitude to Dr. Giuseppe Carenini forhis mentorship and kindness throughout my studies. This would not havebeen possible without his direction, wisdom, patience, and support.Many thanks go to Dr. Hyeju Jang and Patrick Boutet for their eternalsupport since the start of my graduate studies. Their academic, social, andemotional guidance have gotten me to where I am today.I am very grateful to Dr. Dongwook Yoon for his valuable feedback onthe writing of this thesis.Dr. Richard Lester’s expertise and patience have been ever gracious, andI have learned a great deal from his advice. As well, thanks to Dr. RichardLester, Dr. John-Jose Nunez, Dr. Kendall Ho, and Dr. Young Ji Lee fortheir time and intimate feedback during this project’s case studies.Most importantly, my love and my thanks to my partner, AlexandraKim, and my family, for always believing in me.xChapter 1IntroductionTopic modeling is a natural language processing (NLP) technique that iden-tifies topics in a set of texts, or corpus [6]. This technique is a criticalcomponent in most text analytics pipelines. The resulting topic modelscan provide informative and concise summaries of the content of large cor-pora, which can effectively support their exploration and analysis in diversedomains such as health care [7] and education [35]. However, evaluationof topic modeling results is challenging because no general gold standard isavailable, and automatic evaluation metrics, like topic coherence [39], at bestweakly correlate with human judgments of quality [8, 10]. Thus, human-in-the-loop evaluation can be beneficial for improving the model, as shown inprior work [1, 10, 34].Generally speaking, we need both domain experts and NLP researchersto make human-in-the-loop evaluation effective and efficient. Topic model-ing for a particular text domain like health care requires domain expertise toevaluate whether topic modeling accurately captures the high-level overviewof a corpus. Domain experts can interpret the results of a model given theirdomain knowledge, and NLP researchers can take that feedback to enhancethe model. Arguably, the most critical step in this collaborative processis supporting the comparison of two different topic models. Whenever theNLP experts are faced with critical design choices in developing topic model-ing solutions, they can show domain experts the resulting topic models andinteractively gather their assessments of topic quality. Furthermore, topicmodels often involve large number of items (e.g., topics, phrases, words,documents) and relationships among them. The exploration and assess-ment of such complex models can greatly benefit from visual and interactivesolutions [33].In this thesis, we present a visual comparison framework for exploringresults from topic models, which is designed to improve the communica-tion between domain experts and NLP researchers in formative evaluationof topic models. Our framework support inter-model comparison by incor-porating topic alignment visualizations. In addition, we introduce a novelontology view, which provides a taxonomy of domain specific concepts (i.e.,1Chapter 1. Introductionan ontology) for interpreting/labeling topics in a particular domain. Thedomain specific ontology can situate topic model comparison by mappingexpected and understood high-level topics (“concepts”) in specific text do-mains (e.g., medicine) to the actual results from topic models. For furtherin-depth exploration of the dataset, display of per-document topic distribu-tions and buddy plots allow comparison of topics, texts, and shared keywordsat the document level.Our main contributions can be summarized three folds:1. We introduce OCTVis, an analytic framework that allows for high-level and in-depth exploration and comparison of topic models;2. Our framework provides a visual mapping of ontology concepts to top-ics on models generated from domain specific text, which is intendedto enhance the interpretability of topic modeling results; and3. Our use case studies in the health care domain demonstrate that ourframework enhances qualitative evaluation of topic models and pro-vides a clearer understanding of how topic models can be improved.As a preview of the thesis, we discuss related work in visualizing topicmodels for exploration and for comparison, as well as visualizations of do-main ontologies in Chapter 2. In Chapter 3, we derive data and task mod-els for effective ontology-based topic model comparison. In Chapter 4, wepresent OCTVis, our visual framework, and describe the visual encodingsin the final solution, considerations throughout the design process, as wellas implementation details. In Chapter 5, we present the methodology andresults from conducting case studies with domain experts on OCTVis. Wediscuss our evaluation as well as future work in topic model comparison inChapter 6. Finally, we summarize our contributions in Chapter 7.2Chapter 2Related WorkIn the following sections, we address existing visualization research on topicmodeling, topic model comparison, and situating topic models in their con-text via domain ontologies.2.1 Visualizing topic models for explorationFigure 2.1: UTOPIAN from Choo et. al. [9]: (top) topic clusters; (bottomright) individual documents.Text corpora and tasks for exploring their topic models are rich anddiverse, as is the space of visual analytics designed for them. High-leveltag cloud approaches [37] cluster topics and/or their keywords based onword similarity, displaying them in 2D space with size encoding keyword ortopic strength (e.g. SolarMap [5], UTOPIAN [9] (Fig. 2.1), TopicPanorama[41] (Fig. 2.4)). Visualizations including UTOPIAN [9] (Fig. 2.1), Par-allelTopics [12] (Fig. 2.2), iVisClustering [29], and ConVis [23] allow forin-depth exploration of document-topic distributions and documents, high-lighting document keywords pertaining to particular topics. Visual river32.1. Visualizing topic models for explorationmetaphors seen in solutions including ThemeRiver [22], TIARA [30], Hier-archicalTopics [13], and ThemeDelta [18] (Fig. 2.3), are commonly used totake into account the temporal nature of corpora (especially streaming data)and represent topics as they evolve over time.Figure 2.2: ParallelTopics from Dou et. al. [12]. Of note: (top left) parallelcoordinate view of document-topic distributions; (top right) temporal riverview of topics; (bottom left) word cloud of topic keywords.Figure 2.3: ThemeDelta from Gad et. al. [18] shows topics from Obamacampaign speeches over the course of the US 2012 presidential election.42.1. Visualizing topic models for explorationFigure 2.4: TopicPanorama from Wang et. al. [41] displays topics relatedto Google, Microsoft, and Yahoo. Of note: (a) hierarchical level-of-detailvisualization of topics; (e) tag cloud of topic keywords; (f) documents relatedto the topic.Figure 2.5: MultiConVis from Hoque and Carenini [24]. (left) Topic hierar-chy; (right) Blog conversations with sentiment timelines.52.2. Comparison of topic modelsIn order to address text and topic models at scale, hierarchical topic mod-eling frameworks (e.g. HierarchicalTopics [13], TopicPanorama [41] (Fig.2.4), MultiConVis [24] (Fig. 2.5)) group topics at varying levels of scale.More recently, interactive human-in-the-loop techniques including Multi-ConVisIT [25] and SpecEx [16] (Fig. 2.6) have shown that ongoing humanintervention can dramatically improve the quality of topic models by adding,removing, merging, and splitting topics. Our work builds off of this exist-ing topic modeling work using familiar visual idioms including tag cloudsto display topic keywords and highlighting document keywords, in order toaddress comparison based tasks for exploring topic models with the specificfocus of supporting domain and NLP experts in jointly comparing two topicmodels.Figure 2.6: SpecEx from El-Assady et. al. [16] compares differences betweentwo iterations of a topic model in the Tree-Speculation View.2.2 Comparison of topic modelsThe task framework set by Alexander and Gleicher [1] breaks topic modelcomparison into three main tasks: topic alignment, distance comparison,and timeline comparison (see Figure 2.7). To address the task of topicalignment, the authors introduce bipartite graphs and heatmaps for high-level comparisons, as well as colour field rank comparisons for more in-depthtopic-to-topic keyword comparison. To compare document distances, theyintroduce buddy plots: by holding one document stationary, relative dis-62.2. Comparison of topic modelstances to other documents in the corpus can be encoded with distance forone model and colour hue for the other. Sharp deviations in a smooth gradi-ent indicate outlier documents with varied relative distance between the twomodels. For timeline comparison, asymmetric topic flow diagrams juxtaposetwo river flow visualizations [18, 22, 30] to facilitate comparison. While theauthors introduce a number of useful techniques for high-level topic modelcomparison, our detailed task analysis indicates that intra-document ex-ploration needs more support. Thus, our work extends their framework byadding in-depth document keyword comparison, ontology-based comparison.Furthermore, we add a critical step that was missing in their work. WhileAlexander and Gleicher [1] only showcased their interface on usage scenar-ios, we evaluate ours in several case studies with domain-specific corporaand domain experts.Figure 2.7: Topic model comparison from Alexander and Gleicher [1]. (A):Topic alignment heatmap and bipartite graph. (B): Colour field rank com-parison between two topics. (C): Parallel buddy plots. (D): Asymmetrictopic flow diagram.Other comparative frameworks include Srinivasan et. al. [38], who thor-oughly evaluated comparison tasks (not specifically for topic modeling) onbar charts and established a preference for redundant visual encodings (i.e.juxtaposition + explicit) for comparison. Our framework follows these prin-ciples to juxtapose results from two topic models and redundantly encodetopic alignment with a parallel coordinates graph and a heatmap matrix.Oelke et. al. [34] compare individual topics between supervised classes inone topic model using a 2D Euler diagram to represent class containment (seeFigure 2.8). The authors use a topic coin glyph to display a word cloud of thetop twelve keywords in each topic, as well as encoding topic distinctivenessand characteristicness based on other topics in the same class. While theirmulti-class comparison over topics is highly scalable, it requires well defined,supervised class labels over the corpus to compute topic discrimination. This72.2. Comparison of topic modelsapproach is very different from our framework, which compares results fromtwo topic models and facilitates more interactive exploration of the corpus.Figure 2.8: Comparison of 495 papers of InfoVis, SciVis, and Siggraph fromOelke et. al. [34]. Each circle is a topic, and its location encodes howdiscriminating that topic is for its class (the three conferences).To compare topic model results with reference concepts, Chuang et. al.[10] present a probabilistic topic alignment framework. Reference conceptscan come from domain experts, available metadata, or the outputs of othertopic models. The authors compute alignment and four types of misalign-ments between the latent topics and reference concepts: junk topics (nomatching concept), fused topics (2+ matching concepts), missing concepts(no matching topic), and repeated concepts (2+ matching topics) (see Figure2.9). While this framework is a useful diagnostic tool for tuning LDA topicmodels via topic alignment, it expects a one-to-one correspondence betweentopics and concepts, which may not be the case when mapping topics onto ahierarchical ontology. Moreover, evaluation found discrepancies between ex-perts and computed likelihoods when determining significant topics – manytopics marked as significant were considered meaningless junk by experts.Our solution expands this approach by presenting an interactive visualiza-tion that allows users to explore the results of two models, which need notbe probabilistic. Our mapping of domain-specific ontology onto topics does82.2. Comparison of topic modelsnot assume that the ontology is complete nor that the corpus covers all ofthe ontology, instead allowing open-ended comparison tasks in the contextof the text domain.Figure 2.9: Correspondence chart between latent topics (columns) and ref-erence concepts (rows) from Chuang et. al. [10]. Circle areas show matchinglikelihoods, while bars denote probabilities of concept/topic absence, one-to-one correspondence, and merging.More recent interactive topic modeling frameworks [15, 16] provide anexcellent foundation for comparing the results from a topic model againstnewer, re-parameterized results, while constantly responding to feedbackfrom a human in the loop. El-Assady et. al. [15] facilitate a nuanced, in-depth exploration and comparison of the topic models and the corpus in twoways: comparative bar charts to juxtapose topic modeling parameters fromtwo models side by side, and a document relevance feedback view to comparetopic keywords between the models within certain documents (see 2.10).In order to preview the downstream effect of user changes to the model,SpecEx [16] incorporates several computed topic quality metrics includingseparation, distinctiveness, coherence, and pointwise mutual information.While these metrics may not correspond well with human judgments ofquality [8, 10], they remain useful to estimate certain aspects of topic models.Our work incorporates computed metrics via juxtaposed bar charts as well92.3. Ontology visualizationas supporting in-depth exploration of documents adapted specifically forcomparison tasks between differing topic models.Figure 2.10: The document relevance feedback view from El-Assady et. al.[15] compares topic keywords over document snippets between two iterationsof a topic (old: orange, new: purple, shared: blue). (A) Topic keywords ofthe two topics; (B) Documents in the corpus; (C) Top ten sentences for eachtopic; (D) Percentage of matching documents in the corpus.2.3 Ontology visualizationVisualizations for ontologies – hierarchical knowledge taxonomies for par-ticular text domains – have been explored extensively in previous work.Visual encodings have included the use of indented lists, space-filling tech-niques [14, 27], and node-link graphs [17, 27] (see Figures 2.11 and 2.12 forexamples). To the best our knowledge, however, there is little prior workon visualizing ontologies for topic modeling comparison. The idea of ref-erence concepts introduced by Chuang et. al. [10], and used to validatea single topic model is similar to what is captured by an ontology map-ping. Unlike hierarchical ontologies, which often have high-level conceptsthat map to several topics, they expect an optimal one-to-one alignment.The PhenoLines tool [20] (Fig. 2.13) applies the topology of the domain-specific Human Phenotype Ontology to structure topic model output, but isbuilt for comparing domain classes (“phenotypes”) within subtypes of theirtopic model, instead of for comparing results from differing topic models. Inthis thesis, we use established list and graph visual encodings to map anydomain-specific ontology against two topic model results, allowing generalhigh-level comparison for any text domain with established taxonomies.102.3. Ontology visualizationFigure 2.11: (Part I) Schematic examples of ontology visualizations fromDuda´sˇ et. al. [14]112.3. Ontology visualizationFigure 2.12: (Part II) Schematic examples of ontology visualizations fromDuda´sˇ et. al. [14]122.3.OntologyvisualizationFigure 2.13: PhenoLines from Glueck et. al. [20], a visualization for optimizing topic models that describe diseasesymptoms using the Human Phenotype Ontology.13Chapter 3Data and Task AbstractionsBy following the standard information visualization methodology [32], webase the design of our visual interface, OCTVis, on abstracted data andtask models. The data model describes text corpora, topic models, andontologies, while the task model outlines key comparison tasks to enhancequalitative evaluation and understanding of ontology-based topic models.OrganismOrganism AttributeAnatomical StructureFindingLaboratory or Test Result Sign or SymptomEmbryonic StructureAnatomical AbnormalityCongenital AbnormalityAcquired AbnormalityFully Formed Anatomical Structurepart ofevaluation ofproperty ofisa linksnon-isa relationsFigure 3.1: A part of the UMLS semantic network.3.1 Data modelOCTVis is designed to support the comparison of two different topic modelsfor the same corpus. At the highest level, the data model includes: (i) theset of documents comprising the corpus, (ii) two topic models, and (iii) adomain specific ontology.Text often comprises of many topics. Topic modeling is a key NLP taskthat automatically identifies topics within a particular text document or cor-pus. For instance, a news article about electric vehicles could cover battery143.1. Data modeltechnologies, history of electric vehicles, the current market and competi-tors, government incentives, the environment, and so on. In conversationaltext, like email threads and online discussions, topics can vary drasticallywithin a single thread or discussion, and can also include subjective contentsuch as emotion and sentiment. Identifying topics within a text is crucialfor downstream visual analytic tasks like text comparison and exploration.A topic model dataset can be represented as multiple tables. One tablestores topics (attributes) per document (row), assigning a quantitative valuerepresenting the strength of that topic for that particular document [33]. Forprobabilistic methods like Latent Dirichlet Allocation (LDA) [3], each valuein this table is the specific probability, P (t|d), that the attribute topic t(column) occurs given the document d (row). For each document, we canalso assign topics to subdivisions of text, most commonly per word. Overthe entire corpus, we have a table where each row represents a topic, andeach column represents a word in the corpus, and the value in each cellis a quantitative weight for that word in that topic (in LDA, this is theprobability of a word w occurring given a particular topic t, P (w|t)).Although most topic modeling techniques automatically generate labelsfor the extracted topics (e.g., most likely words from the topic in LDA),there often exist human-annotated labels for topics and subtopics withinparticular lexical domains, known as ontologies. These ontologies are hier-archical networks of categories and relationships that classify terms used ina particular domain. As an example, the Unified Medical Language System(UMLS) Semantic Network [31] is an ontology of terms, hierarchical cat-egories (“concepts”), and relations used in the medical lexicon. Conceptswithin UMLS include “Laboratory or Test Result”, “Sign or Symptom”, and“Finding”, and they are linked with their parent concepts by semantic rela-tions, like part of and is a (i.e., subclass), as shown in Figure 3.1. In ourwork, we use the UMLS Semantic Network as our domain-specific medicalontology.Algorithm 11shows the pseudo-code to derive the data model, assumingthat the two topic models to be compared have been already generated.This process involves the four steps below.First, each model is mapped into the ontology. The step comprises twophases: (i) words in the corpus are mapped into the ontology, (ii) each topicof the topic model is mapped into ontology concepts. The first phase is1All the procedures are here specified for an LDA model with conditional probabilitiesP (t|d) and P (w|t). For non-LDA topic models, corresponding weights connecting topicsto documents and words to topics should be used.153.1. Data modelAlgorithm 1 Pseudocode for derived data: ontology mapping, topic align-ment, an example of a topic metric computation (topic segregation), anddocument distances1: procedure MapOntologyConceptsToTopics2: for each word wj in the corpus do3: assign wj to a concept if a mapping exists4: (based on an ontology labeler, MetaMap for the UMLS in thisthesis)5: for each concept ci do6: for each topic tk (in both models) do7: weight(tk, ci)← 08: for word wj mapped to ci do . see Equation 3.19: weight(tk, ci)← weight(tk, ci) + P (wj |tk)10: procedure ComputeTopicAlignment11: for each topic tj in model A do12: for each topic tk in model B do13: alignment = sim(tj , tk) . see Equation 3.214: procedure ComputeTopicSegregation15: for each topic tj in model X do16: for each topic tk in model X do17: compute sim(tj , tk)18: rawSegregation← mean sim(tj , tk) for all other tk19: segregation← normalized rawSegregation to sum to 1 over modelX20: procedure ComputeDocumentDistances21: for each document di in the corpus do22: for each document dj in the corpus do23: distance(di, dj)← sim(p(t|di), p(t|dj))163.2. Task modeldone by using MetaMap [2], a toolkit that automatically codes text with theUMLS semantic concepts. For instance, the text “I need running clothes”would be coded as “I need running [Daily or Recreational Activity]clothes[Manufactured Object]”. In the second phase, to compute thestrength of a mapping between a topic tk and an ontology concept ci, weuse the equation:weight(tk, ci) =∑wj mapped to ciP (wj |tk), (3.1)where wj is a word in the corpus, P (wj |tk) is the probability of that wordgiven the topic tk.Second, following Alexander and Gleicher [1], we compute a mapping(i.e., alignment) between the two topic models as the cosine similarity be-tween their probability distribution of a word occurring given a particulartopic, p(w|t).sim(tj , tk) =∑i P (wi|tj)P (wi|tk)√∑i P (wi|tj)2√∑i P (wi|tk)2, (3.2)where wi ∈ V is each word in the corpus.Third, we derive quantitative topic metrics per topic, including topic seg-regation (how well separated topics are), topic cohesion, and topic stability.These will be described and justified in the next section.Finally, we compute document distances within each topic model. Foreach pair of documents in the corpus, we compute their relative distanceusing the cosine similarity between their document-topic distributions (seethe ComputeDocumentDistances procedure in Algorithm 1).3.2 Task modelThe high-level goal of our visual solution for comparing topic model resultsis twofold:• first, to assist communication between an NLP researcher and a do-main expert during formative evaluation of topic modeling methods,and• second, to assist an NLP researcher in the development of these noveltechniques.173.2. Task modelThe process of refining this high-level goal into a hierarchy of user ana-lytic tasks to drive the interface design was based on principles from existingliterature on visualizing comparisons [1, 19] and topic modeling analytics[4, 11, 16], as well as an informal collection of user requirements from NLPand domain experts (including the authors).In essence, the interface needs to support the assessment of which of twogiven topic models is a better model for the target corpus. This assessmentcan be decomposed into four key qualitative and quantitative metrics forevaluating how good a topic model is. We first describe these metrics andinformally explain how users can be supported in evaluating those. Next,we provide a more formal task model for the metrics.3.2.1 Evaluative metricsTopic segregation This evaluative metric relies on the intuition that anideal topic model should have different topics with little word or semanticoverlap. While word overlap is easy to compute numerically, semantic over-lap is a more difficult problem. The interface should allow the users to gaugethe semantic distance between topics qualitatively, by looking for duplicatewords, synonyms and similar/related words shared across topics.Topic coherence The quality of a topic model depends on the qualityof each of its topics. Thus, we also consider the intuitive metric of within-topic coherence. Essentially, words within a topic should share some over-all semantic relatedness. A topic with seemingly ”random” or ”out of theblue” words is considered noisy, and having poorer quality than one withmore coherent words. This idea relates to well defined quantitative metrics,including topic coherence[11], normalized pointwise mutual information[4],and topic stability[16].Topic model comprehensiveness An important quality metric for atopic model is how thoroughly it covers the topics appearing in the cor-pus. For instance, suppose a corpus discusses the topics of ‘food therapy’,‘symptom’, and ‘sleep’; topic model A captures only ‘food therapy’, ‘symp-tom’, and other irrelevant words while topic model B captures all threetopics successfully. In this case, model B would be preferred to model Asince it is more comprehensive. This is very difficult to measure without ahuman-centred gold standard because it requires understanding the entirecorpus. Our visual analytics solution aims to help researchers create a men-tal topic model to compare topic model results against, enabling evaluation183.2. Task modelof model comprehensiveness. One approximation for this measure can bederived from mapping the topic model on to the ontology space, and seeingthe topic coverage on the ontology concepts. While the ontology might notaccurately reflect the corpus space, it may help the NLP researcher and theclinician create this mental model to evaluate model comprehensiveness andcompare the comprehensiveness of different models.Topic assignment quality A topic model should also map well bothto ontology concepts and at the document level, in the sense of creatinga meaningful middle level between the two. An ideal topic model is wellsituated both within the context of its text domain (and should map wellto an ontology) and of individual documents (i.e. topics should reflect theactual text in each document). If a topic model has well separated high-level topics that are closely related to the domain ontology, but these topicsdo not clearly convey the actual points of discussion in the documents, themodel is too high-level and not much more useful than the ontology itself.On the other hand, if topics are too fine-grained and describe the documentsin detail without merging similar topics or summarizing key ideas, the modelis not a useful substitute for the documents.3.2.2 TasksWith these evaluative metrics in mind, we provide a hierarchy of user ana-lytic tasks for comparing ontology-based topic models.Evaluating Overall Topic Model Quality (TM): These tasks coverthe topic models overall, allowing for high-level comparison between the twomodels before and after delving into individual topics, ontology mappings,and documents. These comparisons should be considered throughout theexploration of the topic models.• (TM1) Overall, does one model describe the corpus better than theother model?An overall quality comparison may be difficult to conclude, but thisprincipal task prompts further investigation and guides the explorationof the topic models.• (TM2) Is one model more coarse-grained or fine-grained than another?As mentioned earlier, an ideal topic model should be situated betweenthe high-level ontology and the individual documents of the corpus.193.2. Task modelWhether or not a topic model is at the right level of detail is relative(to the other model, to the ontology, and to the corpus), use-casedependent, and drives further comparison tasks.• (TM3) Topic coverage/comprehensiveness/segregation:– (TM3a) Are there topics that might be better divided into subtopics,and are there topics that might be better merged into supertopics?Similar to TM2, certain topics may be too broad or specific, andtopic overlap may suggest potential to merge topics. Proper orimproper topic granularity is an indicator of overall model quality.– (TM3b) How well are topics separated from one another?This task overlaps with the previous task (TM3a) in compar-ing relative topic model quality. Distinct, well-segregated topicssuggest a clearer overall model.Assessing Topic Alignment (TA) A principal concern of topic modelcomparison is alignment between topics across models [1]. The followingtasks address this need.• (TA1) How well do topics match? In particular, which topics are themost similar across models? Which topics are the most different?For instance, in Figure 4.2, we see that topics A2 (keywords ”eatlike carbs don fat sugar think”) and B1 (keywords ”eat like carbsknow don try good”) have the highest pairwise alignment, while A4(”study blood learned”...) and B2 (”juice green recipe”...) are themost dissimilar.• (TA2) Of a highly aligned topic pair,– (TA2a) How are the topics similar, and how are they different?From the above example, we see that both topics cover similarkeywords relating to diet and carbs.– (TA2b) Based on this, which of the two topics better captures atopic actually covered in the underlying corpus?While topic keywords may inform the quality of the topic, theusers may also want to explore documents with these alignedtopics and compare document keywords between them (see tasksDW1-3).203.2. Task modelVerifying Topic Quality (TQ) These tasks involve evaluation of thequality of individual topics relative to one another within a single topicmodel and across topic models. As such, computed quantities includingtopic coherence and segregation may provide estimates for human qualita-tive measures of topic quality as well as provide points of comparison forunderstanding how and why topic models differ.• (TQ1) Do measured metrics agree with human judgment for topic qual-ity?– For example, a topic with high segregation (little semantic over-lap) may be a good quality topic in the context of the corpus if itrepresents a clear and unique topic. On the other hand, a highlysegregated topic may consist of nonsensical words that appear infew documents and be relatively uninformative.• (TQ2) Coherence: How coherent or noisy is a topic?We can use both computed coherence and the topic keywords in atopic to evaluate the relative noisiness of a topic, a marker for topicquality.• (TQ3) Topic coverage/comprehensiveness: How well do individual top-ics cover the corpus?Whether a topic spans the entire corpus or only shows up in a fewdocuments can prompt further investigation.Assessing Ontology Mapping Quality (O): The mapping from topicsto a domain ontology situates each model within the context of the textdomain it belongs to. Here, we outline informative tasks that compare topicmodels taking into account this ontology context.• (O1) Which ontology concepts are the most frequent in a topic?For instance, if a topic is strongly mapped to the UMLS concepts”Pharmacologic Substance” and ”Clinical Drug”, we expect it to coverkeywords and documents relating to drugs and medicine.• (O2) Which concepts are prevalent across both topic models?Concepts covered by both models should suggest alignment of topics(see tasks TA1-2) and lead to similar comparisons of topic qualityacross models.213.2. Task model• (O3) Which concepts are seen differently across topic models, andacross topics, and why?If an ontology concept is strongly mapped to one model’s topics butnot the other model’s topics, it can be a red flag to investigate fur-ther differences among these topics. This may indicate more coherenttopics, more segregated topics, or a missed topic entirely in one model.• (O4) Are there concepts which are more informative than others indescribing the corpus?This task informs the quality of the ontology itself which can helpdevelop better ontology mapping techniques. For example, if usersidentify concepts such as “Biologic Function” that might be more in-formative if broken down into “Physiologic” and “Pathologic Func-tion”, updating our ontology mapping in our framework to reflect thisfeedback will better facilitate ontology-based comparisons.• (O5) Do these concepts make sense in the context of the topic wordsand overall corpus?Ontology mappings need to be related back to the context of the corpusitself. High-level topics that map well to ontology concepts may notcorrespond to clear relationships at the document level, and promptsfurther investigation.Comparing Document Distances (DD) Tasks for comparing docu-ment distances are key to understanding similarities within and across topicmodels [1]. As we saw in Chapter 3.1, given a topic model a distance mea-sure between any two documents can be computed. The following tasksaddress these comparisons in document distances and can be applied glob-ally (how documents cluster differently between overall topic models) andlocally (how documents differ relative to a specific document of interest).• (DD1) How do the topic models cluster documents differently with re-spect to the distances among documents?• (DD2) Are there documents that are similar across the models, anddocuments that are quite different?Exploring the Document Topic Distribution (DT) In-depth explo-ration of documents requires first understanding how topics are distributedacross documents and comparing these distributions across models. These223.2. Task modeltasks can identify key topics and documents of interest in the context ofthe text corpus, facilitating fine-grained comparison. A topic that spansmost documents in the corpus may be less or more cohesive or informativethan an infrequent topic, and topic models may perform differently for thesekinds of topics. One model may have broad, spanning, and overlapping top-ics that cover the corpus thoroughly, while another may have sparse andwell-segregated topics that pick out specific documents while having gaps incoverage.• (DT1) How are topics distributed among documents?• (DT2) Are there topics that are prevalent throughout the corpus?• (DT3) Are there topics that only show up in a few documents?• (DT4) Are topics that are prevalent across the corpus substantiallymore or less informative than infrequent topics?Comparing Document Keywords (DW) At the finest level of compar-ison, we need tasks involving topic keywords at the document level to forman in-depth understanding of each topic. These tasks compare aligned top-ics across models by exploring the keywords captured by each topic withinparticular documents. High quality topics should maintain coherence at thislevel, and the keywords within a document should align well with the overallkeywords of the topic.• (DW1) How well do the keywords in aligned topics match at the doc-ument level?• (DW2) Does one model find keywords that another model misses?• (DW3) Are topics assigned correctly at the document level?23Chapter 4SolutionOur proposed solution, OCTVis (Ontology-based Comparison of Topics),uses two linked views to assist in all the topic comparison tasks capturedby our task model (see Figure 4.1). To support our topic-centred tasks, ahigh-level topic-centred facet presents the topics from two topic models tocompare topic alignment, as well as the mappings from an existing ontol-ogy’s concepts to each topic. For more in-depth, document-centred tasks,a document-centred facet shows topic distributions per document, relativedocument similarities, as well as supporting exploration of each individualdocument.4.1 Topic alignment4.1.1 EncodingsThe topic-centred visualization facet, shown in Figure 4.2, builds on thesimilarity heatmap and parallel-coordinates node-link solution presented byAlexander and Gleicher [1]. Two topic models are juxtaposed side-by-side,using a standard word cloud encoding to show topic keywords, with theirstrength encoded by font size [37]. A parallel-coordinates facet in betweenthe two models shows alignment between topics with colour, line width, andopacity all redundantly encoding the alignment strength. Below, a heatmapalso shows this topic alignment information in a similarity matrix, with thetopics of one model along one dimension and the topics of the other modelalong the other dimension. Fields are coloured by the strength of inter-model alignment between those two topics. Additional computed per-topicmetrics, such as topic coherence or segregation, are encoded with alignedhorizontal bars beside each topic node.244.1.TopicalignmentFigure 4.1: The OCTVis interface. Topic models (1) are juxtaposed side by side, allowing comparison of topicalignment to a domain ontology (2) and of document-topic distributions (3). Parallel buddy plots (4) comparerelative distances between documents across both models. A document’s keywords (5) of a selected pair of topicsare highlighted if they belong to the left model’s topic, right model’s topic, or are shared.254.1. Topic alignmentFigure 4.2: The topic alignment facet: topic keywords of two topics modelsare juxtaposed side-by-side, with model A on the left and model B on theright. The topic alignment node-link graph and heatmap (center) encodestopic alignment strength between pairs of topics across topic models.Showing topic keywords beside topics supports the tasks of evaluatingthe quality of matched topics: users can determine how well two topicsmatch (TA1) based on the computed alignment scores, compare the qualityof the match using the topic keywords (TA2a, TA2b), as well as comparesimilarities and differences across model topics, between computed matchesand non-matches (TA1). As well, users can evaluate individual topic quality(TQ1-3) based on their understanding of the topic given the keywords, com-puted per-topic metrics, and through comparison between matched topics.For instance, users can use the topic keywords along with the computedmetric bars to evaluate whether these measured metrics agree with humanqualititative measures (TQ1).The topic alignment heatmap presents a higher-level overview of thealignment, supporting tasks including how well topics are segregated (TQ3)and which topics are most similar and most different (TA1-2). In combi-nation with the topic alignment graph and topic keywords, this supportstopic quality tasks including whether topics might be better subdivided ormerged (TQ2), and in general whether one model is more coarse-grained orfine-grained than another (TM2).4.1.2 InteractionsUsers can filter the displayed computed metrics beside each topic node tocompare metrics of their interest. For instance, while evaluating the co-264.1. Topic alignmentherence or noisiness of a topic, users can hide all metrics except for topiccoherence.As well, highlighting a heatmap field cross-filters that link in the parallel-coordinates facet, desaturating the other links and linking this highlightedlink and corresponding topic nodes (see Figure 4.3). This supports alignmenttasks that compare topics with high computed alignment (TA2a, TA2b), aswell as quality evaluation at the topic level (TQ1-3) and at the model level(TM1-2).Figure 4.3: Linked highlighting between the topic alignment node-link graphand heatmap.274.2. Ontology mapping4.2 Ontology mapping4.2.1 EncodingsThe ontology mapping replaces the topic alignment parallel-coordinatesgraph and alignment heatmap components with a vertical list of ontologyconcepts (see Figure 4.4). A node-link graph on each side of the list connectseach ontology concept with each topic within that side’s topic model. Sim-ilar to the topic alignment graph, the strength of each alignment betweenontology concept and topic (see Chapter 3.1) are redundantly encoded withcolour, line width, and opacity. Concepts are sorted in descending order ofcombined strength from both topic models, so more frequent concepts arehigher on the list than less frequent concepts. Topic keywords and per-topicmetrics are still presented beside each topic node, as in the alignment view.Figure 4.4: The ontology facet (orange, center) displays ontology conceptsand their mapping to each topic. The weight of the mapping is encoded bythe saturation, opacity, and width of the line.The order of the concept list and the concept mapping supports ontology-related comparisons like finding the most frequent concept per-topic (O1)and finding prevalent concepts within both or one topic model (O2, O3).284.2. Ontology mappingFigure 4.5: The document-topic heatmap (right) shows the distribution oftopics (columns) for each document (rows) in this topic model. Strengthsof topics are encoded by the saturation and lightness of the topic model’scolour (red for the left topic, blue for the right topic). The buddy plot facet[1] (left) compares relative document distances across both models. Foreach document, relative distances of every other document (circles) withinthe adjacent model (blue) are encoded with horizontal position, with closerdocuments to the right. Relative distances for the opposite model (red,not shown) are encoded with lightness, with closer documents lighter thanfarther documents. For instance, most documents, except for one, are ratherclose to d25, according to the blue model, but this is not the case with respectto the other model (most circle are dark). Hovering over a document circlehighlights that document in all the parallel buddy plots, and that particularrow in the document-topic heatmap.294.3. Document-centred comparisonUsing the topic mappings and topic keywords, users can then evaluate theinformativeness of these concepts (O4) as well as whether or not they makesense within the context of the corpus (O5).4.2.2 InteractionsUsers can toggle between the topic alignment facet and the ontology map-ping facet to support different tasks. Currently, OCTVis does not supportontology specific interactions, like showing more or less concepts. Such in-teractions, when appropriate, could be added in future versions (see Chapter6).4.3 Document-centred comparison4.3.1 EncodingsBelow each topic model’s topics, we display a document-topic heatmap rep-resenting the distribution of topics within each document in the corpus,expanding on prior work in document-centred comparison [1] (see Figure4.5). Each row represents a single document, and each column representsa topic from the above model. Within the heatmap, the saturation andlightness of the field encodes the strength of that topic (column) within thatdocument (row). A checkerboard background hint relates the mapping fromeach column to each topic’s keywords (See Figure 4.1 for these backgroundhints).To support the comparison of document distances (DD1, DD2), we adaptthe parallel buddy plot encoding introduced by Alexander and Gleicher [1].On each side of a topic model’s document-topic heatmap, a buddy plot facetcan be toggled to compare relative document distances in both models (seeFigure 4.5). Each line corresponds to a document, and the circles repre-sent every other document’s relative distance to the reference document.Inter-document distances in the adjacent model are encoded by horizontaldistance, while distances in the opposite model are encoded by lightness;if the two models have similar document distances, each buddy plot linewould follow a smooth gradient like the reference line at the top of Figure4.5. Sharp changes in this gradient indicate documents that have differingdistances across the two models [1]. In OCTVis, we chose to use hollowcircles instead of solid disks to see overlap between documents more clearly,and replaced hue encodings with lightness to prevent overloading establishedcolour encodings for the two topic models.304.4. Key considerations during the design process4.3.2 InteractionsTo explore the topic models at the document level, single documents aredisplayed in the centre of the interface, similar to interactive topic model-ing techniques presented by El-Assady et al. [15]. Selecting a row in thedocument-topic heatmap displays that relevant document, in addition toa zoomed view of that document’s topic distribution heatmap across eachmodel (just above the document panel, see Figure 4.1). Clicking a par-ticular field in the zoomed heatmap selects that topic for intra-documentexploration. The user can select one topic from each of the two models andthe strongest keywords for the selected topic are highlighted in the docu-ment view. Keywords belonging to the left topic are displayed in red, whilekeywords from the right topic are displayed in blue. Keywords shared acrossboth topics are displayed in green.4.4 Key considerations during the design processIn this section, we discuss key issues that we considured during the designprocess, including alternative views, layouts, and encodings that were noteventually chosen for the final solution.Figure 4.6: Topic coin clustering mockup. Each coin glyph represents one ormore merged topics, with shared keywords in black and unique keywords inorange and purple corresponding to their respective topic model. Relativepositions of each glyph encodes that topic’s similarity to other topics as wellas distinctiveness across the two models.Initially, we explored a clustering topic coin approach similar to Oelkeet. al. [34] by merging similar topics into coin glyphs and projecting topicdistances in two-dimensional space (see the mockup in Figure 4.6). Within314.4. Key considerations during the design processeach glyph, one or more merged topics would be shown in a tag cloud oftopic keywords, with weight encoded by text size. Shared keywords frommerged topics are shown in black, while words unique to one model’s topicare coloured accordingly (orange for the left model, purple for the rightmodel). The location of the topic glyph, as well as its background colour,would encode relative distinctiveness across models; i.e., a glyph farther tothe left would represent a topic more unique to the left model. Relative coindistances would also encode topic similarity or segregation between pairs oftopics.As we considered this clustering paradigm, we extended this frameworkto address more comparison tasks. Particularly, the need to situate topicsin the context of domain-specific knowledge led to the inclusion of ontologymappings in the interface. As well, we introduced the document view toaddress the unmet need to thoroughly explore document similarities andindividual documents. Figure 4.7 shows the first mockups of these viewsduring one of our brainstorming design sessions (Nov 2018, UBC).Eventually, the coin glyph approach was replaced with a simpler parallelcoordinates graph of topics, topic keywords, and topic alignment, in orderto remove the additional problem of merging aligned topics as well as createmore room for other visual encodings. Figure 4.8 shows a much more highfidelity mockup for topic alignment matrices, document-topic matrices, andthe view for exploring individual documents.For the document view, we initially encoded keyword strength withcolour with bilinear interpolation between the two selected topics (see Fig-ure 4.9). These colour gradients were deemed to be too difficult to perceive,and we experimented with varying interpolation, changing colours, and in-creasing the bin size of keyword weights to reduce the colour possibilities.In the final solution, this was simplified further to three static colours (blueand red for keywords in one model topic, and green for shared keywords),as seen in Figure 4.1 (bottom-center).324.4.KeyconsiderationsduringthedesignprocessFigure 4.7: Whiteboard mockup of topic-centred and document-centred views, snapshot during brainstormingdesign session before finalizing the design of OCTVis (adopted designs, discarded/modified designs). (A)parallel coordinates graph to display ontology mappings; (B) topic alignment matrix; (C) clustering of topic coins;(D) clustering with ontology mapping; (E) document-topic matrices with buddy plots; (F) individual documentview334.4. Key considerations during the design processFigure 4.8: Mockup of topic alignment matrices, document-topic matrices,and document view.344.4. Key considerations during the design processFigure 4.9: Colour binning for document keywords. Top left : bilinear in-terpolation between black and red (vertical, model A keyword weight), andblack and blue (horizontal, model B weight). Top centre: bilinear interpo-lation along diagonals, bottom-left to top-right diagonal blending betweenmodel A (red) to shared (orange) to model B (blue), top-left to bottom-rightdiagonal blending with black based on keyword strength. Top right : same ascentre, reduced to 4×4 colour bins. Bottom: document view with keywordscoloured using top-left binning for topic weights, showing the difficulty inperceiving relative topic weight.354.5. Implementation4.5 ImplementationTopic modeling and ontology mappings were computed with Python’s nltkand sklearn libraries. In this thesis, we used Latent Dirichlet Allocation(LDA) and Non-negative Matrix Factorization (NMF) with 5 and 10 topicseach. In order to annotate topic modeling results with ontology concepts,we use MetaMap [2], a toolkit that automatically codes text with the UMLSsemantic concepts.The OCTVis interface is a web app built with HTML, CSS, and Javascript.We used Sass2 for programmable stylesheets and the D3 (v5) Javascriptlibrary3 for data-driven visualization. We make abundant use of ES6 im-provements to Javascript, including async/await for asynchronous file load-ing, Promise.all for parallel asynchronous calls, and arrow functions forD3 callbacks4. All source code can be found on http://www.github.com/humfuzz/octvis. An online demo of the interface can be found on http://www.cs.ubc.ca/cs-research/lci/research-groups/natural-language-processing/octvis/5.We use the following input to OCTVis:1. Raw text for each document in the corpus,2. Comma-separated values for document-topic distributions from theresults of two topic models, where each row is a document, each columnis a topic, and each field is the weight of that topic for that document,3. A list of unique vocabulary words in the corpus, separated by newlines,4. Comma-separated values for word-topic distributions from the resultsof two topic models, where each row is a topic, each column is a wordin the order of the vocabulary list, and each field is the weight of thattopic for that word, and5. Ontology mappings from each ontology term to vocabulary words,represented with a JSON Object.These files are parsed and processed by load.js, which computes topicalignment, document similarities, per-topic metrics (e.g. segregation), and2https://sass-lang.com/3https://d3js.org/4e.g. (d, i, nodes)=>{... nodes[i]} for binding this5Documents used in case studies are redacted for ethics purposes.364.5. Implementationontology weights for each topic (see Chapter 3.1 for details). All data isrepresented in Javascript Objects and Arrays.main.js renders the OCTVis visualization in a webpage using SVG.Rectangular heatmaps and buddy plots use the standard D3 data join modelto associate SVG selections with Javascript data. We take advantage ofD3’s force-body simulations to generate node-link graphs for topic alignmentand ontology mapping, fixing nodes to computed positions and pausing thesimulation on the first timestep.The document view is a foreignObject div nested inside the SVG, andcontains spans for each word in the document. Each word span is dynami-cally classed based on the selected topics to highlight keywords within thattopic.Facet locations are dynamically adjusted for the number of topics anddocuments in the data, and custom tweaks per dataset can be manuallyspecified in metadata parameters. For instance, the top n keywords (default:1% of the vocabulary size) per topic are highlighted in the document view,and n can be manually adjusted for each dataset.37Chapter 5Case Studies5.1 MethodologyTo evaluate OCTVis on real data, we ran case studies with domain expertson two corpora of comments from online discussion forums: one contain-ing comments from diabetes patients, the other containing comments fromovarian cancer patients.Table 5.1: Case study corpora statistics.diabetesdiscussionsovarian cancerdiscussions# comments 201 56,537# wordspercommentmin 1 1max 524 5,577mean 110 99st. dev. 93 130The topic models were built on corpora with very different sizes (seeTable 5.1). For each corpus, the two topic models being compared were gen-erated with differing methods: one with Latent Dirichlet Allocation (LDA)[3], and the other with Non-negative Matrix Factorization (NMF) [28], bothset to generate five topics. For the ontology mapping we used the UnifiedMedical Language System (UMLS) Semantic Network [31], which describes127 terms6 (”concepts” or ”semantic types”) and 54 relations in the medicallexicon. We manually excluded 24 of these concepts that were too general oruninformative (for instance, Qualitative Concept, Functional Concept, andGroup Attribute) and weighted the mapping from each concept to each wordin the corpus using Equation 3.1. For each topic, we computed its segre-gation from the other topics in the same model using the average cosinesimilarity of pairwise topic vectors. As for the set of documents that were6We used UMLS semantic network 2018AA, the latest version at the time of writing.385.1. Methodologyactually shown in the visualization, in order to keep the case studies man-ageable in term of time, a sample of the comments was selected and shown:34 comments for the diabetes corpus and 27 for the ovarian cancer corpus.We assume, given more time, the domain and NLP experts would considerdifferent and possibly larger samples of the data to perform a more in-depthcomparison of the topic models.Our four domain experts were all clinicians that work with patients di-rectly or facilitate online patient discussions. At the start of the case study,we provided the experts with a definition of topic modeling and ontologies,and asked about whether they thought topic modeling can be informativein understanding a possibly large set of documents, whether they had aclear idea what a ‘good’ topic model should look like, and whether or notontologies are useful in evaluating topic model quality.After this questionnaire, each domain expert was guided through thefeatures of OCTVis on a fictional sample dataset and topics, and asked todemonstrate their understanding by answering questions based on perform-ing sample tasks. If they could not answer those questions or anything elsewas unclear, more training was provided.During the main study, they were presented with one of two datasetsand were free to explore the data using the interface. Two of the expertshad more time and were able to also explore the same corpus with twoadditional models comprising ten topics instead of five, and one of theseexperts was able to give feedback on both corpora as well. Since the goalof our interface involves facilitating communication between domain expertsand NLP researchers, one of our NLP researchers discussed the results of thetopic models and prompted topic model comparison with the domain expertthroughout the main study. We were interested in how the interface, throughtopic model comparison with and without mapping to an ontology, couldimprove this communication and facilitate expert evaluation of the topicmodels. After the study, we asked the domain experts the same questionsat the start to see whether their beliefs about topic models and ontologieschanged, and asked for generic feedback on the interface.We made a conscious decision to exclude the parallel buddy plot facetfrom the case study, as we felt that the visual encodings of the parallelbuddy plots had a steep learning period relative to marginal benefits of thedocument distance information in the medical conversational corpora weconsidered. Since each document was a comment from discussion fora, theywere relatively short (around 20-500 words each) compared to other classesof documents in NLP (for instance, a newspaper article or a journal paper);this meant that relative document distances were drastically different based395.2. Resultson small changes in the topics, and so had very little similarities between thetwo topic models. Without any significant patterns in the buddy plot datato compare document distances across models, we decided not to include thisview in the case study. For corpora with longer documents, for instance, acollection of newspaper articles, the buddy plot view would likely be moreinformative. Testing our interface on such documents is left as future work.5.2 ResultsThe case studies indicate that OCTVis was useful in comparing the twomodels. From the pre-study and post-study questionnaires (see Table 5.2),we found that all the experts reported that their idea of what a ‘good’ topicmodel should be became clearer after using the interface. Their belief thattopic modeling can be informative in understanding a large set of documentswas strong before and remained strong after the study. However, two expertslowered their assessment for the usefulness of ontologies, mainly because theyfound that the provided ontology was too generic.Table 5.2: Pre-study and post-study questionnaire responses. Domain ex-perts were asked how strongly they agreed or disagreed with the followingstatements (on a scale from 1-5, with 0.5 increments, where 1 is stronglydisagree and 5 is strongly agree): (S1) Topic modeling can be informativein understanding a possibly large set of documents. (S2) I have a clear ideaof what a ‘good’ topic model should look like. (S3) Ontologies are useful inevaluating topic model quality. For each expert and question, bold numbersindicate stronger or equal agreement before or after the study.domain expert S1 S2 S3pre post pre post pre poste1 4 4 4 4.5 4 4.5e2 4 5 3 5 4 3e3 4 4 3 4 4 4e4 5 5 4 5 4 2Topic AlignmentAll the domain experts used the topic alignment graph and heatmap to com-pare aligned topics and identify differences between a pair of similar topics.405.2. ResultsGenerally, they found that the NMF model generated more coherent topicsthan the LDA model. For the diabetes corpus, one domain expert found thatthe LDA model was more high level and the NMF model was more detailed.In the ovarian cancer discussions, another expert found that the LDA topicswere more medical focused while the NMF topics were more psychosocial.This led the domain and NLP experts to consider the possibility of com-bining the two models in future work. For instance, speculatively, topicsfrom NMF could be used to refine topics form LDA into more fine-grainedsubtopics. Alternatively, topics from NMF and LDA could be joined in asingle model with broader semantic coverage. In terms of topic segregation,the per-topic segregation metric was found to be relatively uninformativeacross our case studies, and one expert expressed the desire to see morequantitative metrics across both topic models, such as the fraction of theconcepts mapped in the ontology.The two domain experts that were able to see the results of ten-topictopic models in addition to five-topic models found that the interface scaledwell for the increased visual information. One expert initially found usingthe alignment matrix heatmap less intuitive than the node-link graph tocompare topic models with five topics, but with ten topics the node-linkgraph became more cluttered, and the alignment matrix heatmap was muchmore effective for identifying pairs of strongly aligned topics, an indicationthat redundantly displaying such information can be beneficial. Within theten-topic models, both the experts identified within-model topics that werevery similar in both keywords and document distributions and suggestedthat merging these topics would result in more cohesive topics.Ontology MappingWhile two of our experts found the ontology mapping to be a useful high-level overview of the topics discussed, the other two found that the mappingas-is still contained many concepts that were too general or uninformative.Those that found the ontology mapping useful were able to re-evaluate theirunderstanding of topics based on their ontology mappings. In the ovariancancer corpus, one topic with top keywords “chemo just time taxol like didgood” mapped strongly to the ontology concept “Therapeutic or PreventiveProcedure”, and no other topics in either model mapped as strongly to thisconcept. With this information, the two experts that explored this corpuswere able to identify this topic as a cohesive and well-segregated topic aboutcancer treatment that the other model failed to capture.All our domain experts provided useful feedback about how to enhance415.2. Resultsthe utility of the ontology mapping. For instance, one clinician was unclearabout what the ontology term labeled “Finding” represented. Even after weclarified that it referred to medical findings, they believed that it was toogeneral and might be better broken down into subcategories including “testresults” or “symptoms”. All the experts expressed a need to dynamicallyalter or explore the ontology concepts shown, through filtering, searching,or manually adding in categories of interest. Particularly, when the topicmodels missed important and expected topics including ”medication”, theontology view could show this gap in coverage. Furthermore, it was sug-gested that we could use these gaps to seed future topic models to improvethe modeling results.Exploring DocumentsTwo of the clinicians expressed little interest in exploring individual docu-ments, as they were more concerned about the overall topics discussed bythe patients, while others used in-depth exploration of the documents toassess topic quality and improve their understanding of the topics. For thediabetes corpus, the document-topic distribution and highlighted keywordsin individual documents helped confirm labels for topics including “exercise”and “diet”, as well as determining that a “medication” topic was missed byboth models. This was less the case for the ovarian cancer corpus, likelybecause the corpus used to generate the topic models was much larger thanthe sample documents shown in the interface. The domain experts thatcompared topics at the document keyword level were often able to qualifysimilar topics with more nuance - for example, while comparing very similartopics relating to “surgery” in the ovarian cancer corpus, one model’s topichad more cohesive keywords (less irrelevant keywords) in a document thanthe other model and was perceived to capture the topic in the text moreprecisely.42Chapter 6Discussion and Future WorkOverall, our case studies showed that OCTVis enabled domain experts toeffectively compare topic model results and suggest topic modeling improve-ments to NLP experts. There was no clear trend in valuing broad compar-isons against an ontology over fine-grained comparisons against individualdocuments, and our evaluation suggests that exploring both contexts en-hances the evaluation of topic model quality and allows for more subtlecomparisons. The ontology mapping acts as a high-level overview of thecorpus to give the two topic models a backbone to compare against, whileexploring documents in-depth enables detailed validation of topic coherenceand coverage.As mentioned in Chapter 5.1, we excluded the parallel buddy plots fromour case studies due to the limited document distance information theyprovided. Given the short documents in our corpora, relative documentdistances had little similarity across the two topic models, and the buddyplots had no significant patterns to identify unique documents in the dis-tance space. We hope to test the effectiveness of this encoding on corporawith longer documents, where we would expect to see more similarities indocument distances between differing topic models.Given our visual framework, a future application exists for comparingontology-based topic model results on differing corpora, similar to the com-parative framework set by Oelke et. al. [34]. For example, clinicians maybe interested in how discussions differ for patients from one region to an-other, and our interface could be adapted to allow for this comparison. Asthis involves comparison of corpora rather than topic models, a future taskmodel and subsequent implementation would need to reflect this change.Existing work [23, 25, 26] has found human-in-the-loop visual analyt-ics to be extremely beneficial for exploring conversational text data, es-pecially asynchronous conversations including online blogs and discussionsmuch like the corpora in our case studies. A future extension to our com-parison framework involves displaying the conversational structure of thesecorpora alongside our ontology-based comparison tools. Additionally, con-versational text is well suited for hierarchical topic modeling techniques43Chapter 6. Discussion and Future Work[13, 25] that allow users to view high-level topics and drill down for moredetailed subtopics, and these hierarchical topic models would be invaluablefor comparison tasks. Both conversational text analytics and hierarchicaltopic modeling methods would increase the scalability of our solution forcorpora with a large number of documents by aggregating topic modelinginformation on collapsible conversational documents.These human-in-the-loop techniques often involve interactive topic modelrevision [15, 16, 25], and our case studies have shown the need for more in-teractive tools in our comparative framework. Techniques for interactivetopic modeling, including seeding the topic model with keywords and merg-ing/splitting/deleting topics, are all tools our domain experts expressed adesire to use, particularly when applied to the ontology space. Allowingdomain experts to build more relevant ontologies and enhance the existingontology network would enhance the effectiveness of ontology-based compar-ison of topic models. Similar to work in building knowledge graphs [21, 36],we can use this to have custom ontologies built for specific domain needs.With larger numbers of topics, ontology concepts, and documents, thescalability of our interface becomes a concern. Aforementioned interactivetechniques including filtering and drilling down can tackle the issue of scaleon many data dimensions. Topics, documents, and ontologies can be clus-tered hierarchically and allow users to explore the data space at multiplegranularities. For instance, for large ontologies, analysis could focus onmore fine-grained subtrees that are specific for relevant topics (e.g. canceror diabetes sub-ontologies for our corpora) and allow interactive expansionand collapse of these concepts. While the alignment encoding with par-allel coordinates can become cluttered with dozens of topics or concepts,heatmap matrices and other space-filling idioms can scale more effectively;our topic-ontology mappings can be redundantly encoded with these visualencodings as well.In terms of the algorithmic choices that were made on how the datamodel is derived, several future alternatives could be considered. For in-stance, in order to compute topic alignment, we used the generic cosinesimilarity measure (see Equation 3.2). If, for instance, the two topic modelsinvolved were both probabilistic, distances that are specific to probabilitydistributions, like KL-divergence, could be more appropriate [40]. Further-more, in computing the mapping from topics to ontology concepts, we haveassumed that the corpus is deterministically labeled with ontology concepts.However, this might not be the case for the given ontology, for which thelabeling could be probabilistic (i.e., a word in the corpus is assigned to dif-ferent ontology concepts with different probabilities). When this happens,44Chapter 6. Discussion and Future WorkEquation 3.1 should be revised accordingly:weight(tk, ci) =∑wj∈V, ci∈CP (ci|wj)P (wj |tk), (6.1)where we include the probability P (ci|wj) of a concept ci given a word wj ,and sum over all words wj in the corpus V and all concepts ci in the ontologyC.45Chapter 7ConclusionOur visual analytics approach to ontology-based comparison of topic modelsis designed to inform NLP experts and domain experts about the qualityof alternative topic modeling methods situated within the context of a do-main ontology. We expand on prior work in topic modeling visualization byenabling topic comparison at multiple granularities, from high-level domain-specific mappings to ontology concepts to in-depth document keyword ex-ploration. To the best of our knowledge, we are the first to perform an eval-uation of a topic modeling comparison interface with potential users. Suchevaluation, through case studies with clinicians, has shown a clear improve-ment in understanding topic modeling, enabling communication betweendomain experts and NLP experts to evaluate and improve novel topic mod-eling techniques. We hope to extend our work in the future by facilitatingcomparison of different corpora, incorporating more interactive techniquesfor human-in-the-loop analysis, and enhancing comparison of hierarchicaltopic models and conversational text.46Bibliography[1] E. Alexander and M. Gleicher. Task-driven comparison of topic models.Trans. Visualization and Computer Graphics (TVCG), 22(1):320–329,Jan 2016. ↑1, ↑6, ↑7, ↑17, ↑18, ↑20, ↑22, ↑24, ↑29, ↑30[2] Alan R Aronson and Franc¸ois-Michel Lang. An overview of MetaMap:historical perspective and recent advances. Journal American MedicalInformatics Assoc., 17(3):229–236, 2010. ↑17, ↑36[3] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichletallocation. Journal Machine Learning Research, 3:993–1022, Jan 2003.↑15, ↑38[4] G. Bouma. Normalized (pointwise) mutual information in collocationextraction. In Proc. German Society Comp. Ling. & Language Tech.(GSCL), pages 31–40, 2009. ↑18[5] N. Cao, D. Gotz, J. Sun, Y. Lin, and H. Qu. SolarMap: Multifacetedvisual analytics for topic exploration. In IEEE 11th Intl. Conf. DataMining, pages 101–110, Dec 2011. ↑3[6] Giuseppe Carenini, Gabriel Murray, and Raymond Ng. Methods formining and summarizing text conversations. Synthesis Lectures on DataManagement, 3(3):1–130, 2011. ↑1[7] K. Chan, X. Lou, T. Karaletsos, C. Crosbie, S. Gardos, D. Artz, andG. Ratsch. An empirical analysis of topic modeling for mining can-cer clinical notes. In IEEE 13th Intl. Conf. on Data Mining Workshops(ICDMW), pages 56–63, Los Alamitos, CA, USA, dec 2013. IEEE Com-puter Society. ↑1[8] Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish,and David M. Blei. Reading tea leaves: How humans interpret topicmodels. In Neural Information Processing Systems (NIPS), 2009. ↑1,↑947Bibliography[9] J. Choo, C. Lee, C. K. Reddy, and H. Park. UTOPIAN: User-driventopic modeling based on interactive nonnegative matrix factorization.Trans. Visualization and Computer Graphics (TVCG), 19(12):1992–2001, Dec 2013. ↑3[10] Jason Chuang, Sonal Gupta, Christopher D. Manning, and Jeffrey Heer.Topic model diagnostics: Assessing domain relevance via topical align-ment. In Proc. Intl. Conf. Machine Learning (ICML), 2013. ↑1, ↑8, ↑9,↑10[11] E. Talley M. Leenders D. Mimno, H. M. Wallach and A. McCallum.Optimizing semantic coherence in topic models. In Proc. Conf. onEmpirical Methods in Natural Language Processing (EMNLP), pages262–272. Association for Computational Linguistics, 2011. ↑18[12] W. Dou, X. Wang, R. Chang, and W. Ribarsky. ParallelTopics: A prob-abilistic approach to exploring document collections. In IEEE Conf.Visual Analytics Science and Technology (VAST), pages 231–240, Oct2011. ↑3, ↑4[13] W. Dou, L. Yu, X. Wang, Z. Ma, and W. Ribarsky. HierarchicalTopics:Visually exploring large text collections using topic hierarchies. Trans.Visualization and Computer Graphics (TVCG), 19(12):2002–2011, Dec2013. ↑4, ↑6, ↑44[14] Marek Duda´sˇ, Steffen Lohmann, Vojtch Sva´tek, and Dmitry Pavlov.Ontology visualization methods and tools: a survey of the state of theart. The Knowledge Engineering Review, 33, 07 2018. ↑10, ↑11, ↑12[15] M. El-Assady, R. Sevastjanova, F. Sperrle, D. Keim, and C. Collins.Progressive learning of topic modeling parameters: A visual analyt-ics framework. Trans. Visualization and Computer Graphics (TVCG),24(1):382–391, Jan 2018. ↑9, ↑10, ↑31, ↑44[16] M. El-Assady, F. Sperrle, O. Deussen, D. Keim, and C. Collins. Visualanalytics for topic model optimization based on user-steerable specula-tive execution. Trans. Visualization and Computer Graphics (TVCG),25(1):374–384, Jan 2019. ↑6, ↑9, ↑18, ↑44[17] Sean Falconer, R Ian Bull, Lars Grammel, and Margaret-AnneD. Storey. Creating visualizations through ontology mapping. pages688–693, 03 2009. ↑1048Bibliography[18] S. Gad, W. Javed, S. Ghani, N. Elmqvist, T. Ewing, K. N. Hamp-ton, and N. Ramakrishnan. ThemeDelta: Dynamic segmentations overtemporal topic models. Trans. Visualization and Computer Graphics(TVCG), 21(5):672–685, May 2015. ↑4, ↑7[19] M. Gleicher. Considerations for visualizing comparison. Trans. Visu-alization and Computer Graphics (TVCG), 24(1):413–423, Jan 2018.↑18[20] M. Glueck, M. P. Naeini, F. Doshi-Velez, F. Chevalier, A. Khan,D. Wigdor, and M. Brudno. PhenoLines: Phenotype comparison visu-alizations for disease subtyping via topic models. Trans. Visualizationand Computer Graphics (TVCG), 24(1):371–381, Jan 2018. ↑10, ↑13[21] Udo Hahn, Martin Romacker, and Stefan Schulz. Creating knowledgerepositories from biomedical reports: the MEDSYNDIKATE text min-ing system. In Biocomputing 2002, pages 338–349. World Scientific,2001. ↑44[22] S. Havre, B. Hetzler, and L. Nowell. ThemeRiver: visualizing themechanges over time. In IEEE Symposium on Information Visualization2000. INFOVIS 2000. Proceedings, pages 115–123, Oct 2000. ↑4, ↑7[23] E. Hoque and G. Carenini. ConVis: A visual text analytic system forexploring blog conversations. In Proc. 16th Eurographics Conf. Visual-ization (EuroVis), pages 221–230. Eurographics Association, 2014. ↑3,↑43[24] Enamul Hoque and Giuseppe Carenini. MultiConVis: A visual textanalytics system for exploring a collection of online conversations. InProc. 21st Intl. Conf. Intelligent User Interfaces (IUI), pages 96–107,New York, NY, USA, 2016. ACM. ↑5, ↑6[25] Enamul Hoque and Giuseppe Carenini. Interactive topic hierarchy re-vision for exploring a collection of online conversations. InformationVisualization, 2018. ↑6, ↑43, ↑44[26] Indratmo, Julita Vassileva, and Carl Gutwin. Exploring blog archiveswith interactive visualization. In Proceedings of the Working Conf. onAdvanced Visual Interfaces, AVI ’08, pages 39–46, New York, NY, USA,2008. ACM. ↑4349Bibliography[27] Akrivi Katifori, Constantin Halatsis, George Lepouras, Costas Vassi-lakis, and Eugenia Giannopoulou. Ontology visualization methods - asurvey. ACM Computing Surveys, 39:10, Nov 2007. ↑10[28] Daniel D Lee and H Sebastian Seung. Learning the parts of objects bynon-negative matrix factorization. Nature, 401(6755):788, 1999. ↑38[29] Hanseung Lee, Jaeyeon Kihm, Jaegul Choo, John Stasko, and HaesunPark. iVisClustering: An interactive visual document clustering viatopic modeling. Computer Graphics Forum, 31(3pt3):1155–1164, 2012.↑3[30] Shixia Liu, Michelle Zhou, Shimei Pan, Weihong Qian, Weijia Cai, andXiaoxiao Lian. TIARA: Interactive, topic-based visual text summariza-tion and analysis. volume 3, pages 543–552, 01 2009. ↑4, ↑7[31] Alexa T McCray. An upper-level ontology for the biomedical domain.International Journal of Genomics, 4(1):80–84, 2003. ↑15, ↑38[32] T. Munzner. A nested model for visualization design and validation.Trans. Visualization and Computer Graphics (TVCG), 15(6):921–928,Nov 2009. ↑14[33] Tamara Munzner. Visualization analysis and design. AK Peters Visu-alization Series, CRC Press, 2014. ↑1, ↑15[34] D. Oelke, H. Strobelt, C. Rohrdantz, I. Gurevych, and O. Deussen.Comparative exploration of document collections: a visual analyticsapproach. Computer Graphics Forum, 33(3):201–210, Jul 2014. ↑1, ↑7,↑8, ↑31, ↑43[35] Arti Ramesh, Dan Goldwasser, Bert Huang, Hal Daume, and LiseGetoor. Understanding MOOC discussion forums using seeded LDA. InProc. 9th Workshop Innovative Use of NLP for Building EducationalApplications, pages 28–33. Association for Computational Linguistics(ACL), Jun 2014. ↑1[36] David Sa´nchez and Antonio Moreno. Learning non-taxonomic relation-ships from web documents for domain ontology construction. Data &Knowledge Engineering, 64(3):600–623, 2008. ↑44[37] C. Seifert, B. Kump, W. Kienreich, G. Granitzer, and M. Granitzer. Onthe beauty and usability of tag clouds. In 12th Intl. Conf. InformationVisualisation, pages 17–25, Jul 2008. ↑3, ↑2450Bibliography[38] Arjun Srinivasan, Matthew Brehmer, Bongshin Lee, and Steven M.Drucker. What’s the difference?: Evaluating variations of multi-seriesbar charts for visual comparison tasks. In Proc. CHI, pages 304:1–304:12, New York, USA, 2018. ACM. ↑7[39] Keith Stevens, Philip Kegelmeyer, David Andrzejewski, and David But-tler. Exploring topic coherence over many models and many topics. InProc. Joint Conf. on Empirical Methods in Natural Language Process-ing and Computational Natural Language Learning (EMNLP-CoNLL),pages 952–961, Stroudsburg, PA, USA, 2012. Association for Compu-tational Linguistics. ↑1[40] Mark Steyvers and Tom Griffiths. Probabilistic topic models. Handbookof latent semantic analysis, 427(7):424–440, 2007. ↑44[41] X. Wang, S. Liu, J. Liu, J. Chen, J. Zhu, and B. Guo. TopicPanorama:A full picture of relevant topics. Trans. Visualization and ComputerGraphics (TVCG), 22(12):2508–2521, Dec 2016. ↑3, ↑5, ↑651Appendix APre-Study QuestionnaireTopic modeling is the process of extracting the topics covered by a set ofdocuments, or corpus. For instance, a set of documents can cover two topics:animals and food.An ontology is a set of concepts and relationships describing knowledge in aparticular domain. For instance, in the medical domain, two concepts couldbe “flu” and “disease”; they are related by an instance of relation: flu is aninstance of disease.Please rate how strongly you agree or disagree with each of the followingstatements with respect to topic modeling.Topic modeling can be informative in understanding a possiblylarge set of documents.stronglydisagreedisagree neutral agreestronglyagreedon’tknowI have a clear idea of what a good topic model should look like.stronglydisagreedisagree neutral agreestronglyagreedon’tknowOntologies are useful in evaluating topic model quality.stronglydisagreedisagree neutral agreestronglyagreedon’tknow52Appendix BPost-Study QuestionnaireTopic modeling is the process of extracting the topics covered by a set ofdocuments, or corpus. For instance, a set of documents can cover two topics:animals and food.An ontology is a set of concepts and relationships describing knowledge in aparticular domain. For instance, in the medical domain, two concepts couldbe “flu” and “disease”; they are related by an instance of relation: flu is aninstance of disease.Please rate how strongly you agree or disagree with each of the followingstatements with respect to topic modeling.Topic modeling can be informative in understanding a possiblylarge set of documents.stronglydisagreedisagree neutral agreestronglyagreedon’tknowI have a clear idea of what a good topic model should look like.stronglydisagreedisagree neutral agreestronglyagreedon’tknowOntologies are useful in evaluating topic model quality.stronglydisagreedisagree neutral agreestronglyagreedon’tknowPlease answer the following questions.What did you like about the interface? What did you find to beuseful?53Appendix B. Post-Study QuestionnaireWhat did you dislike about the interface? How can the visualiza-tion be improved?What did I learn about the dataset, thanks to the topic modelsand ontology?Which topic model better represents the text?Please justify your findings.How can either topic model be improved?54Appendix CCase Study Sample Dataset[document 1]I love animals. Animals are so great. I love cute pandas and giraffes anddogs and horses and cats and chimpanzees.[document 2]Me too! Animals are so cute.[document 3]Here is my recipe for banana bread. Makes one delicious loaf.2 bananas1 cup sugar1 egg, beaten1 cup flour1. Preheat oven to 350F.2. Mash bananas and add egg, sugar, and flour3. Pour batter into loaf pan. Bake for 1h.4. Remove from oven and serve.[document 4]I love feeding my cats banana bread, they gobble it up!55Appendix DCase Study ScriptD.1 Introduction and pre-study questionnaireToday we’ll be investigating comparison of topic models, particularly thoseseeded with existing ontologies, with the goal of assisting NLP researchersdevelop novel techniques as well as improving communication between NLPresearchers and domain experts when evaluating new methods.Topic modeling is the process of extracting the topics covered by a setof documents, or corpus. For instance, a set of documents can cover twotopics: animals and food.An ontology is a set of concepts and relationships describing knowledgein a particular domain. For instance, in the medical domain, two conceptscould be “flu” and “disease”; “disease” would have a superclass relationshipwith “flu”.[Show example image in pre-study questionnaire.]Thank you for participating in our study. The whole process today willlast about one hour. My name is Amon and Im the designer of the interface,and Hyeju is the natural language processing (or NLP) researcher who willhelp facilitate this case study with you, a domain expert.Do you have any questions about these?Do you have any questions before we get started?Okay, please feel free to ask questions anytime.First you will answer a short pre-study questionnaire. Then, I will walkyou through the interface with some sample data. After that, we will moveto the main portion of the study, which will involve you exploring some ofyour data and comparing topic models with Hyeju. At the end of the study,Ill ask a few more questions to get your feedback.Before we start, please answer a few questions.[Have domain expert fill the pre-study.]56D.2. User trainingD.2 User trainingThank you, now we will demonstrate the features of the topic comparisoninterface. Here is a sample corpus, a dataset of four documents.[Hand paper with 4 sample documents printed out.]Now I will show you the results of two topic models on this dataset.The visual interface presents two topic models side-by-side - the leftmodel, model A, is red, and the right model, model B, is blue. For instance,one of the red topics is about the words “animals cute love...”, and one ofthe red topics is about the words “eggs animals delicious...”. Do you haveany questions about this?Above, the Topic Overview presents the top keywords of each topic,as well as per-topic metrics to estimate different aspects of topic quality.Right now, they are showing topic segregation - the longer the bar, themore distinct that topic is from the other topics within that model.[Show/hide computed per-topic metrics.]Do you have any questions about this?The Topic Alignment facet, in green, consists of a graph between thetopics of both models and a heatmap. Both these components indicate thesimilarity between pairs of topics across models with the “brightness” of thegreen colour.[Highlight alignment heatmap cells to show corresponding linked topics.]Do you have any questions about this?[Toggle the Topic Alignment and Ontology Alignment facets.]The orange Ontology Alignment facet replaces the Topic Alignmentfacet to show the strongest ontology concepts in the corpus, sorted in de-creasing strength combined across both models. The strength of each align-ment between ontology concept and topic is encoded by the colour and widthof the line connecting the two. Do you have any questions about this?Below each model’s topics, the Document Overview shows the dis-tribution of topics within each document. The colour of each cell indicatesthe strength of that topic (column) in that document (row). In the bottomcentre of the interface, the Document View displays the text of a singledocument, as well as the top keywords between a selected topic from eachmodel. Red and blue colours indicate keywords belonging to a single modelsselected topic, and green indicates a keyword shared across both topics.[Click document row in Document Overview to view that document.][Click topic cells in the document heatmap to view keywords for that topicwithin that document.]Do you have any questions about this?57D.3. Main studyNow, we will ask you a few questions about the dataset to check thatyou understand how to navigate the interface. At any time, you can askquestions and clarify your understanding with either of us.[Give the domain expert control, and clarify if they don’t know how todo any task.]Can you identify the most similar topics across the two models?In document 3, what are the two strongest topics in model A and modelB?Between these two topics, what keywords are shared within this docu-ment?In model B, which topic is most prevalent throughout the corpus?If you toggle the Ontology view, which concept is topic B3 most stronglymapped to?Which concept is topic A2 most weakly mapped, but not zero?By examining the ontology mappings, the documents, and the topics,which topic model would you pick as the better topic model to represent thedata?D.3 Main studyNow we will begin the study. You are going to be comparing two topic mod-els on a conversation from an online discussion forum for [diabetes/ovariancancer] patients mapped against a medical ontology, the Unified MedicalLanguage System. This will be an open ended exploration between you andthe NLP researcher. Imagine that Hyeju has come to you with these twotopic models, and needs your domain knowledge to help evaluate and com-pare the results of these two models to improve future models and summa-rize the topics being discussed. Feel free to ask questions and communicatewith each other, and please think out loud so that I can understand yourexperience.D.4 Post-study questionnaire and debriefPlease answer the following post-study questionnaire.[Have domain expert fill the post-study.]Thank you very much again for your participation. Do you have anyother comments or questions?58
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
OCTVis : ontology-based comparison of topic models
Text