Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Liu, Zipeng
Doctor of Philosophy - PhD
University of British Columbia
Visualization is an important tool to analyze data, but there emerge various challenges from complex analytical data and tasks. In this dissertation, I present four projects that were motivated by these challenges, situated in the nested model proposed by Munzner, which consists of four layers to describe the components in visualization: domain, data and task, encoding design, and algorithm. In ADVIEW, to address the challenges of comparing many phylogenetic trees in the domain of biology, I propose a visual encoding to compress a tree representation, design and implement a multi-view interactive tool to handle the multiple levels of detail in a tree collection dataset, ranging from the whole collection, through subsets of trees, individual trees, subtrees, to leaf nodes. In SPRAWLTER, to address the existing visual encoding problems of readability metrics for node-link graphs, I propose two novel metrics to measure a finer-grained clutter and to balance the geometric sparseness and clutter. These metrics recognize different levels of visual saliency such as metanodes and leaf nodes in multi-level graphs. In LOGSEG, to fulfill user demands for chunking actions in the domain of image editing software, I propose a segmentation model for the action logs to serve the demands that require different chunking granularities. For example, smart undo for going back to a previous user task needs a low-level chunking, while managing an overview of milestones needs a high-level one. In CORGIE, to fill the gap in visual qualitative evaluation of graph neural networks (GNNs) in the domain of machine learning, I propose an approach and design a tool to explore correspondences between a graph and its embedding to check how different levels of structures are preserved from the input graph to the output embedding. I also design a new graph layout to reveal how a GNN leverages node neighbors and computes an embedding. I identify a common theme among these projects: multi-level structures. They consist of nesting subsets of data points that are relevant to the analytical tasks. I demonstrate how to exploit them in the visualization if provided in hierarchical data, or to compute them for non-hierarchical data.
Visualizing Multi-level Structures in DatabyZipeng LiuBachelor of Science, Peking University, 2015A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)August 2021© Zipeng Liu, 2021The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Visualizing Multi-level Structures in Datasubmitted by Zipeng Liu in partial fulfillment of the requirements for the degreeof Doctor of Philosophy in Computer Science.Supervisory Committee:Tamara Munzner, Computer ScienceSupervisorGiuseppe Carenini, Computer ScienceSupervisory Committee MemberMichiel van de Panne, Computer ScienceSupervisory Committee MemberExamining Committee Members:William Evans, Computer ScienceUniversity ExaminerSathish Gopalakrishnan, Electrical and Computer EngineeringUniversity ExaminerEnrico Bertini, Computer Science and Engineering, New York University TandonSchool of EngineeringExternal ExamineriiAbstractVisualization is an important tool to analyze data, but there emerge various chal-lenges from complex analytical data and tasks. In this dissertation, I present fourprojects that were motivated by these challenges, situated in the nested model pro-posed by Munzner, which consists of four layers to describe the components invisualization: domain, data and task, encoding design, and algorithm.In ADVIEW, to address the challenges of comparing many phylogenetic trees in thedomain of biology, I propose a visual encoding to compress a tree representation,design and implement a multi-view interactive tool to handle the multiple levelsof detail in a tree collection dataset, ranging from the whole collection, throughsubsets of trees, individual trees, subtrees, to leaf nodes. In SPRAWLTER, to addressthe existing visual encoding problems of readability metrics for node-link graphs,I propose two novel metrics to measure a finer-grained clutter and to balance thegeometric sparseness and clutter. These metrics recognize different levels of visualsaliency such as metanodes and leaf nodes in multi-level graphs. In LOGSEG, tofulfill user demands for chunking actions in the domain of image editing software, Ipropose a segmentation model for the action logs to serve the demands that requiredifferent chunking granularities. For example, smart undo for going back to aprevious user task needs a low-level chunking, while managing an overview ofmilestones needs a high-level one. In CORGIE, to fill the gap in visual qualitativeevaluation of graph neural networks (GNNs) in the domain of machine learning, Ipropose an approach and design a tool to explore correspondences between a graphand its embedding to check how different levels of structures are preserved fromiiithe input graph to the output embedding. I also design a new graph layout to revealhow a GNN leverages node neighbors and computes an embedding.I identify a common theme among these projects: multi-level structures. Theyconsist of nesting subsets of data points that are relevant to the analytical tasks.I demonstrate how to exploit them in the visualization if provided in hierarchicaldata, or to compute them for non-hierarchical data.ivLay SummaryData acquired in many domains in our society today needs to be analyzed by userswith visualization, which produces graphical representations from the data andallows the users to interact with the graphics. For different analytical data andtasks in different domains, there are different challenges to visualize the relevantinformation. I propose to focus on the visualization of multi-level structures, theinternal structures in data which consists of multiple levels, to address some ofthese challenges. I introduce four projects to demonstrate how to exploit multi-level structures from data with hierarchical information, and how to compute themfrom data without hierarchical information, in three different domains includingbiology, software usability, and machine learning.vPrefaceParts of this dissertation have appeared in either peer-reviewed publications or on-going journal submissions.Chapter 2 is based on the following published journal paper, code-named ADVIEW.Z. Liu, S. H. Zhan, and T. Munzner. Aggregated Dendrograms for Visual Compar-ison Between Many Phylogenetic Trees. IEEE Transactions on Visualization andComputer Graphics (TVCG), 26(9):2732–2747, 2020 [112].Chapter 3 is based on the following published journal paper, code-named SPRAWL-TER.Z. Liu, T. Itoh, J. Q. Dawson, and T. Munzner. The Sprawlter Graph ReadabilityMetric: Combining Sprawl and Area-aware Clutter. IEEE Transactions on Visual-ization and Computer Graphics (TVCG), 26(6):2180–2191, 2020 [110].Chapter 4 is based on the following published journal paper, code-named LOGSEG.Z. Liu, Z. Liu, and T. Munzner. Data-driven Multi-level Segmentation of ImageEditing Logs. Proceedings of the ACM Conference on Human Factors in Comput-ing Systems (CHI), pages 1–12, 2020. [111].viChapter 5 is based on the following recent journal submission, code-named CORGIE.Z. Liu, Y. Wang, J. Bernard, and T. Munzner. Visualizing Graph Neural Networkwith CorGIE: Corresponding a Graph to Its Embedding. IEEE Transactions onVisualization and Computer Graphics (TVCG). Under review [113].As the first author of all four papers above, I took a leading role in the researchprojects and contributed in many aspects. I characterized the domain discipline ofthe project: phylogenetics in ADVIEW, image editing software in LOGSEG, neuralnetworks for graph machine learning in CORGIE, which involved meeting and in-terviewing with domain experts and stakeholders, reading and summarizing litera-ture, learning to perform the analytical task myself (e.g. using Photoshop, traininggraph neural networks), etc. In this process, I also abstracted the data and tasksfrom domain-specific language to general visualization language. I took charge inmost of the visual design work, and I am the sole developer to implement themwith web-based tools (HTML, CSS, Javascript). I planned and executed the eval-uation of the proposed tools or algorithms, such as conducting user study sessionswith target users and measuring performances on a computer. I am the main writerof the manuscripts, which involves a few composing and editing passes beforesubmission, and addressing issues raised by reviewers. I also wrote most of thesupplemental documents for these papers, and filmed the demonstration videos forthe proposed tools.My supervisor Tamara Munzner shepherded these projects by providing both high-level guidance and concrete feedback throughout all project life cycles. During thewriting phases, she ensured a proper paper structure and also edited my drafts ingreat detail. In ADVIEW, Shing Hei Zhan is the phylogenetic expert who providedinformation about the domain, feedback to my visual design and the tool. The othercoauthors are visualization researchers who have provided help in various phasesin the projects like applying for funding, connecting to stakeholders, reviewingliterature, designing interfaces, recruiting participants for user studies, editing themanuscripts, and more.viiAll user studies in this dissertation have been approved by the UBC BehaviouralResearch Ethics Board (BREB) with the certificate No. H10-03336.viiiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Four layers of visualization concerns . . . . . . . . . . . . . . . . 21.2 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Multi-level structures . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.5 Evaluation methodology . . . . . . . . . . . . . . . . . . . . . . 82 ADView: Aggregated Dendrograms for Visual Comparison BetweenMany Phylogenetic Trees . . . . . . . . . . . . . . . . . . . . . . . . 102.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13ix2.2.1 General tree visualization . . . . . . . . . . . . . . . . . 132.2.2 Biological tree visualization . . . . . . . . . . . . . . . . 142.2.3 Visual tree comparison . . . . . . . . . . . . . . . . . . . 142.2.4 Task abstractions for trees . . . . . . . . . . . . . . . . . 172.3 Phylogenetic tree data . . . . . . . . . . . . . . . . . . . . . . . . 182.3.1 Phylogenetic tree . . . . . . . . . . . . . . . . . . . . . . 182.3.2 Data abstraction . . . . . . . . . . . . . . . . . . . . . . 192.4 Task abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.5 Aggregated Dendrogram (AD) . . . . . . . . . . . . . . . . . . . 232.5.1 Design requirements . . . . . . . . . . . . . . . . . . . . 232.5.2 Visual design . . . . . . . . . . . . . . . . . . . . . . . . 242.5.3 Visual summary: Cluster AD . . . . . . . . . . . . . . . . 262.5.4 Design evolution . . . . . . . . . . . . . . . . . . . . . . 272.6 ADView interface . . . . . . . . . . . . . . . . . . . . . . . . . . 282.6.1 Interface overview . . . . . . . . . . . . . . . . . . . . . 282.6.2 Reference Tree and AD Views . . . . . . . . . . . . . . . 302.6.3 Tree Distribution View . . . . . . . . . . . . . . . . . . . 312.6.4 Pairwise comparison . . . . . . . . . . . . . . . . . . . . 322.6.5 Auxiliary views . . . . . . . . . . . . . . . . . . . . . . . 342.6.6 View coordination and visual encoding . . . . . . . . . . 352.7 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.7.1 Corresponding branches . . . . . . . . . . . . . . . . . . 372.7.2 Aggregated dendrogram layout . . . . . . . . . . . . . . . 392.7.3 Cluster aggregated dendrogram . . . . . . . . . . . . . . 412.8 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.8.1 Computational performance . . . . . . . . . . . . . . . . 422.8.2 Information density comparison . . . . . . . . . . . . . . 432.8.3 Usage scenario 1: 1KP pilot study . . . . . . . . . . . . . 442.8.4 Formative expert feedback . . . . . . . . . . . . . . . . . 472.8.5 Summative expert interview study . . . . . . . . . . . . . 472.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522.10 Summary and future work . . . . . . . . . . . . . . . . . . . . . 53x3 Sprawlter: Combining Sprawl and Area-aware Clutter for GraphReadability Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.2.1 Graph drawing readability metrics . . . . . . . . . . . . . 593.2.2 Evaluation of graph drawing metrics . . . . . . . . . . . . 623.3 sprawlter metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 623.3.1 Graph layout definitions . . . . . . . . . . . . . . . . . . 633.3.2 Measuring geometric overlap . . . . . . . . . . . . . . . . 633.3.3 Mapping measurements to penalties . . . . . . . . . . . . 643.3.4 Summing penalties . . . . . . . . . . . . . . . . . . . . . 653.3.5 Combining clutter metrics with sprawl . . . . . . . . . . . 663.4 Penalty mapping function . . . . . . . . . . . . . . . . . . . . . . 683.4.1 General requirements . . . . . . . . . . . . . . . . . . . . 683.4.2 Specific choices . . . . . . . . . . . . . . . . . . . . . . . 703.4.3 Formal function instantiations . . . . . . . . . . . . . . . 713.4.4 Parameter space analysis . . . . . . . . . . . . . . . . . . 733.5 Data generation . . . . . . . . . . . . . . . . . . . . . . . . . . . 743.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753.6.1 Analysis approach . . . . . . . . . . . . . . . . . . . . . 763.6.2 Comparative analysis . . . . . . . . . . . . . . . . . . . . 773.6.3 Computational time . . . . . . . . . . . . . . . . . . . . . 843.7 Discussion and Future Work . . . . . . . . . . . . . . . . . . . . 853.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874 LogSeg: Data-driven Multi-level Segmentation of Image Editing Logs 884.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 894.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 924.2.1 Event Sequence Visualization . . . . . . . . . . . . . . . 924.2.2 Workflows and Graphical Histories . . . . . . . . . . . . 924.2.3 Event Log Models . . . . . . . . . . . . . . . . . . . . . 934.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 954.3.1 Data Generation . . . . . . . . . . . . . . . . . . . . . . 95xi4.3.2 Model and Feature Selection . . . . . . . . . . . . . . . . 954.3.3 Evaluation Methods . . . . . . . . . . . . . . . . . . . . 964.4 Data Collection and Characterization . . . . . . . . . . . . . . . . 974.4.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . 974.4.2 Data Characterization . . . . . . . . . . . . . . . . . . . . 984.5 Feature Computation . . . . . . . . . . . . . . . . . . . . . . . . 994.5.1 Command Similarity . . . . . . . . . . . . . . . . . . . . 994.5.2 Layer Similarity . . . . . . . . . . . . . . . . . . . . . . 1004.5.3 Image Diff . . . . . . . . . . . . . . . . . . . . . . . . . 1014.5.4 Working Region Overlap . . . . . . . . . . . . . . . . . . 1024.5.5 Duration . . . . . . . . . . . . . . . . . . . . . . . . . . 1024.6 Segmentation Model . . . . . . . . . . . . . . . . . . . . . . . . 1024.6.1 Low-level Segmentation . . . . . . . . . . . . . . . . . . 1024.6.2 High-level Segmentation . . . . . . . . . . . . . . . . . . 1044.7 Results: Feature Relevance . . . . . . . . . . . . . . . . . . . . . 1064.7.1 Quantitative Analysis . . . . . . . . . . . . . . . . . . . . 1064.7.2 Qualitative Analysis . . . . . . . . . . . . . . . . . . . . 1074.7.3 Result Interpretation and Limitations . . . . . . . . . . . 1084.8 Results: Low-level Segmentation . . . . . . . . . . . . . . . . . . 1104.8.1 Performance Metrics . . . . . . . . . . . . . . . . . . . . 1104.8.2 Task Variance . . . . . . . . . . . . . . . . . . . . . . . . 1104.8.3 Comparison to Previous Work . . . . . . . . . . . . . . . 1124.9 Results: Multi-level Segmentation . . . . . . . . . . . . . . . . . 1134.9.1 Exemplar Retouching Session . . . . . . . . . . . . . . . 1134.9.2 Exemplar Poster Session . . . . . . . . . . . . . . . . . . 1144.10 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1154.10.1 Strengths and Limitations of Model . . . . . . . . . . . . 1154.10.2 Dataset Comparison . . . . . . . . . . . . . . . . . . . . 1164.10.3 Real-world User Behaviour . . . . . . . . . . . . . . . . 1164.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1175 CorGIE: Visualizing Graph Neural Network by Exploring Corre-spondences between a Graph and Its Embedding . . . . . . . . . . . 119xii5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1205.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1235.2.1 Graph neural networks . . . . . . . . . . . . . . . . . . . 1235.2.2 Motivating scenarios . . . . . . . . . . . . . . . . . . . . 1245.3 Data and task abstraction . . . . . . . . . . . . . . . . . . . . . . 1265.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 1275.3.2 Data Space Abstraction . . . . . . . . . . . . . . . . . . . 1285.3.3 Task abstraction . . . . . . . . . . . . . . . . . . . . . . 1315.4 CorGIE design . . . . . . . . . . . . . . . . . . . . . . . . . . . 1325.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 1335.4.2 View design . . . . . . . . . . . . . . . . . . . . . . . . . 1345.4.3 View coordination . . . . . . . . . . . . . . . . . . . . . 1395.4.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . 1405.4.5 Design alternatives . . . . . . . . . . . . . . . . . . . . . 1415.5 K-hop layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1415.5.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 1425.5.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . 1445.5.3 Design alternatives . . . . . . . . . . . . . . . . . . . . . 1455.6 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1465.6.1 Usage scenario 1: Movie Recommendation . . . . . . . . 1475.6.2 Usage scenario 2: Cora . . . . . . . . . . . . . . . . . . . 1505.6.3 Expert study overview . . . . . . . . . . . . . . . . . . . 1545.6.4 Expert session 1: Cora decision boundary . . . . . . . . . 1545.6.5 Expert session P2: USAir . . . . . . . . . . . . . . . . . 1565.6.6 Expert feedback summary . . . . . . . . . . . . . . . . . 1585.7 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1595.7.1 Visualizations for machine learning models . . . . . . . . 1595.7.2 Latent space interpreter . . . . . . . . . . . . . . . . . . . 1605.7.3 GNN interpreter . . . . . . . . . . . . . . . . . . . . . . 1615.7.4 Graph layout . . . . . . . . . . . . . . . . . . . . . . . . 1625.8 Discussion and future work . . . . . . . . . . . . . . . . . . . . . 1625.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163xiii6 Reflection and Conclusions . . . . . . . . . . . . . . . . . . . . . . . 1656.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1656.2 Reflection on multi-level structures . . . . . . . . . . . . . . . . . 1666.3 Reflection on collaboration . . . . . . . . . . . . . . . . . . . . . 1686.4 Reflection on research topic choices . . . . . . . . . . . . . . . . 1696.5 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . 170Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172A Supplemental Material for ADView . . . . . . . . . . . . . . . . . . 192A.1 View coordination details . . . . . . . . . . . . . . . . . . . . . . 192A.2 Cluster AD gradient coloring . . . . . . . . . . . . . . . . . . . . 194A.3 Algorithm details . . . . . . . . . . . . . . . . . . . . . . . . . . 196A.3.1 AD layout function parameters . . . . . . . . . . . . . . . 196A.3.2 Front-end caching for rendering ADs . . . . . . . . . . . 196A.4 Expert Interview Study . . . . . . . . . . . . . . . . . . . . . . . 199A.4.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 199A.4.2 Interview Questions . . . . . . . . . . . . . . . . . . . . 199A.5 Full Screenshots of Usage Scenario 1: 1KP pilot study . . . . . . 202A.5.1 Sister group of land plants . . . . . . . . . . . . . . . . . 202A.5.2 Early diversification of land plants . . . . . . . . . . . . . 207A.6 Usage scenario 2: TreeFam . . . . . . . . . . . . . . . . . . . . . 209A.6.1 PROTOSTOMIA and DEUTEROSTOMIA . . . . . . . . . . 210A.6.2 ECDYSOZOA and LOPHOTROZOA . . . . . . . . . . . . . 215A.7 Screenshots of Information Density Comparison . . . . . . . . . . 218A.8 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220A.8.1 Full Screenshots of Case Study 1 . . . . . . . . . . . . . 220A.8.2 Full Screenshots of Case Study 2 . . . . . . . . . . . . . 220A.8.3 Case Study 3 . . . . . . . . . . . . . . . . . . . . . . . . 222A.9 Screenshots of Dataset Upload . . . . . . . . . . . . . . . . . . . 224B Supplemental Material for Sprawlter . . . . . . . . . . . . . . . . . 228B.1 Parameter analysis . . . . . . . . . . . . . . . . . . . . . . . . . 228B.1.1 Minimum penalty fraction . . . . . . . . . . . . . . . . . 229xivB.1.2 EE curve shape . . . . . . . . . . . . . . . . . . . . . . . 233B.2 Pseudo-code for NE and EE area-aware metric . . . . . . . . . . . 235B.3 List of graph layouts . . . . . . . . . . . . . . . . . . . . . . . . 237B.4 Computational time . . . . . . . . . . . . . . . . . . . . . . . . . 238B.5 Computational pipeline . . . . . . . . . . . . . . . . . . . . . . . 240C Supplemental Material for LogSeg . . . . . . . . . . . . . . . . . . . 242C.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242C.1.1 Commands . . . . . . . . . . . . . . . . . . . . . . . . . 242C.1.2 Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247C.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . 248C.3 Hyperparameter Tuning for SVM . . . . . . . . . . . . . . . . . . 252C.3.1 Window size k . . . . . . . . . . . . . . . . . . . . . . . 252C.3.2 Threshold t . . . . . . . . . . . . . . . . . . . . . . . . . 253C.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255C.4.1 Labeling interface . . . . . . . . . . . . . . . . . . . . . 255C.4.2 Visualization of sessions in test set . . . . . . . . . . . . . 257C.5 Rule-based model . . . . . . . . . . . . . . . . . . . . . . . . . . 266D Supplemental Material for CorGIE . . . . . . . . . . . . . . . . . . 268D.1 Concrete task examples . . . . . . . . . . . . . . . . . . . . . . . 268D.2 Configurable settings, filters and menus . . . . . . . . . . . . . . 269D.3 Alternative visual design for topology space . . . . . . . . . . . . 269D.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275D.4.1 Usage scenario 1: Movie Recommendation . . . . . . . . 275D.4.2 Usage scenario 2: Cora . . . . . . . . . . . . . . . . . . . 280D.4.3 Expert session P1: Cora decision boundary . . . . . . . . 283D.4.4 Expert session P2: USAir . . . . . . . . . . . . . . . . . 285D.5 Semi-structured interview in user study . . . . . . . . . . . . . . 288xvList of TablesTable 4.1 The five kinds of layer relationships . . . . . . . . . . . . . . . 101Table 4.2 Corresponding coefficients to features in SVM model . . . . . 107Table 4.3 The performance metrics of sessions in the three task types . . 112Table 5.1 Computational benchmarks for the K-hop layout . . . . . . . . 144Table 6.1 A summary of contributions of my four projects covering thefour layers in the nested model. . . . . . . . . . . . . . . . . . 170Table A.1 View coordination of visual encoding across all levels of detailand all views . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Table A.2 List of participants and datasets used in the study . . . . . . . . 199Table B.1 Full list of graph layouts . . . . . . . . . . . . . . . . . . . . . 237Table B.2 Computation time of each layout for three metric families andtwo approaches . . . . . . . . . . . . . . . . . . . . . . . . . 239Table C.1 Summary of task type . . . . . . . . . . . . . . . . . . . . . . 251xviList of FiguresFigure 1.1 Munzner’s nested model [124] showing four layers of visual-ization concerns . . . . . . . . . . . . . . . . . . . . . . . . . 2Figure 1.2 Illustrations of multi-level structures . . . . . . . . . . . . . . 5Figure 1.3 Evalaution methods under Munzner’s nested model [124] . . . 8Figure 2.1 The problem space of visual tree comparison for two scalabil-ity dimensions . . . . . . . . . . . . . . . . . . . . . . . . . 15Figure 2.2 Illustrations for the tasks and corresponding branches . . . . . 20Figure 2.3 Visual encoding of an aggregated dendrogram . . . . . . . . . 24Figure 2.4 Illustration of how the AD adapts to available space . . . . . . 26Figure 2.5 Two examples of clustering structurally similar aggregated den-drograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Figure 2.6 Five versions of the aggregated dendrogram design in chrono-logical order of invention . . . . . . . . . . . . . . . . . . . . 29Figure 2.7 Screenshot of ADView comparing a reference tree against atree collection . . . . . . . . . . . . . . . . . . . . . . . . . . 29Figure 2.8 Illustration of the Tree Distribution view . . . . . . . . . . . . 31Figure 2.9 Pairwise comparison between the reference tree and a user-specified target tree from tree collection . . . . . . . . . . . . 33Figure 2.10 Legends for the visual encodings . . . . . . . . . . . . . . . . 36Figure 2.11 Annotated screenshots for a research question in the 1KP pilotstudy: which is the sister group of LP . . . . . . . . . . . . . 45Figure 2.12 Detail views of P1’s findings . . . . . . . . . . . . . . . . . . 49xviiFigure 2.13 Cropped figures of P2’s findings . . . . . . . . . . . . . . . . 50Figure 3.1 Comparison of approaches to measuring node-node crossingin three synthetic 2-level graph layouts . . . . . . . . . . . . . 58Figure 3.2 Increasing amount of overlap between node-node, node-edge,and edge-edge pairs . . . . . . . . . . . . . . . . . . . . . . . 64Figure 3.3 General requirements for area-aware penalty mapping function 69Figure 3.4 Illustration of the penalty mapping function for node-node over-lap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72Figure 3.5 The NN penalty mapping function of the smallest node pairwith different α . . . . . . . . . . . . . . . . . . . . . . . . . 74Figure 3.6 Increasing overlap between red and blue metanodes in a small2-level synthetic graph . . . . . . . . . . . . . . . . . . . . . 78Figure 3.7 Metrics for a single-level synthetic graph with variable nodesizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Figure 3.8 Metrics for large real-world graph layout by FM3 in Tulip . . 80Figure 3.9 Metrics for two large multi-level real-world graph layouts byGrouseFlocks . . . . . . . . . . . . . . . . . . . . . . . . . . 81Figure 3.10 Increasing overlap between blue metanodes and edges in asmall 2-level synthetic graph . . . . . . . . . . . . . . . . . . 82Figure 3.11 Metrics for two large multi-level real-world graph layouts byGrouseFlocks . . . . . . . . . . . . . . . . . . . . . . . . . . 83Figure 3.12 Decreasing crossing angles between edges in a small single-level synthetic graph . . . . . . . . . . . . . . . . . . . . . . 83Figure 4.1 Concrete example of session segmentation . . . . . . . . . . . 90Figure 4.2 Distributions of the five feature values of non-boundary andboundary events . . . . . . . . . . . . . . . . . . . . . . . . . 107Figure 4.3 Session visualizations . . . . . . . . . . . . . . . . . . . . . 111Figure 5.1 Motivation of CorGIE . . . . . . . . . . . . . . . . . . . . . 121Figure 5.2 Data and task abstraction . . . . . . . . . . . . . . . . . . . . 128Figure 5.3 Full screenshot of CorGIE on the Movie dataset . . . . . . . . 133Figure 5.4 K-hop layout algorithm in four steps . . . . . . . . . . . . . . 142xviiiFigure 5.5 Design alternatives to K-hop layout . . . . . . . . . . . . . . 145Figure 5.6 Check overview of the Movie scenario . . . . . . . . . . . . . 148Figure 5.7 Check movie recommendations . . . . . . . . . . . . . . . . 149Figure 5.8 Explore overview of Cora dataset . . . . . . . . . . . . . . . 152Figure 5.9 Find problematic node pairs in the Cora dataset . . . . . . . . 153Figure 5.10 P1 explored the decision boundary in the Cora dataset . . . . 155Figure 5.11 The USAir dataset without focal nodes . . . . . . . . . . . . 156Figure 5.12 Find buggy node pairs . . . . . . . . . . . . . . . . . . . . . 157Figure 5.13 Screenshot of P2 focusing on a buggy node pair . . . . . . . . 157Figure A.1 A duplicate screenshot of ADView for reference of the viewcoordination table . . . . . . . . . . . . . . . . . . . . . . . . 193Figure A.2 Illustration of gradient coloring in a cluster AD . . . . . . . . 194Figure A.3 Illustration of different encoding representation for the samenormal distribution . . . . . . . . . . . . . . . . . . . . . . . 195Figure A.4 Illustration of parameters in an aggregated dendrogram layout 197Figure A.5 Pipeline on the frontend to render the aggregated dendrograms 198Figure A.6 Full screenshot of ADView to explore four clades . . . . . . . 202Figure A.7 Support for the most popular hylothesis (LP + ZYGN) . . . . 203Figure A.8 Pairwise comparison with the conflicting trees . . . . . . . . . 204Figure A.9 Combination of user-specified clades . . . . . . . . . . . . . 205Figure A.10 Exploration of low-support branches . . . . . . . . . . . . . . 206Figure A.11 Screenshot of ADView exploring early lineage of LAND PLANTS208Figure A.12 Previous hypotheses about early diversification of LAND PLANTS208Figure A.14 Screenshot of ADView for exploring PROTOSTOMIA and DEUTEROST-SOMIA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212Figure A.15 Exploration of some discordant trees . . . . . . . . . . . . . . 213Figure A.16 Pairwise comparison to locate potential causes for the discor-dance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214Figure A.17 Screenshot of ADView for exploring ECDYSOZOA and LOPHOTRO-ZOA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216Figure A.18 Cluster ADs showing distributions of ECDYSOZOA . . . . . . 217xixFigure A.19 Screenshot of the tool by Bremm et. al. [19] on the 35-treedataset from their paper . . . . . . . . . . . . . . . . . . . . . 218Figure A.20 Screenshot of ADView with the same dataset at the same screenresolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219Figure A.21 Screenshot of the tool by Bremm et. al. with similarity scorefilter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219Figure A.22 Screenshot of P1 using ADView . . . . . . . . . . . . . . . . 220Figure A.23 Screenshot of P2 using ADView . . . . . . . . . . . . . . . . 221Figure A.24 Screenshot of P3 using ADViewto find the best tree . . . . . . 223Figure A.25 Screenshot of P3 using ADViewto compare the best tree against155 gene trees . . . . . . . . . . . . . . . . . . . . . . . . . . 223Figure A.26 The first step of uploading a dataset . . . . . . . . . . . . . . 225Figure A.27 The second step of uploading a dataset . . . . . . . . . . . . . 226Figure A.28 Data pre-processing by a server . . . . . . . . . . . . . . . . 227Figure B.1 The NN penalty mapping function of a big metanode pair . . . 229Figure B.2 Influence of minimum penalty fraction (α) for node-node over-laps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231Figure B.3 Influence of minimum penalty fraction (α) on node-edge over-laps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231Figure B.4 Influence of minimum penalty fraction (α) for edge-edge cross-ings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232Figure B.5 Two different curve shapes of the EE penalty mapping function 234Figure B.6 Comparison of the linear and quadratic penalty function . . . 234Figure C.1 Embedding space neighborhood of Linear Burn command 244Figure C.2 Embedding space neighborhood of Brush Tool command . 245Figure C.3 Embedding space neighborhood of New Color Fill Layercommand . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246Figure C.4 Layer hierarchy examples . . . . . . . . . . . . . . . . . . . 247Figure C.5 8 collected sessions in the first round . . . . . . . . . . . . . . 249Figure C.6 8 collected sessions in the second round . . . . . . . . . . . . 250Figure C.7 ROC curves for window size k . . . . . . . . . . . . . . . . . 253xxFigure C.8 Performance threshold curve for threshold t . . . . . . . . . . 254Figure C.9 Labeling interface on a scene creation session . . . . . . . . . 256Figure C.10 Analysis of segmentation results for session S12 (upper part) . 258Figure C.11 Analysis of segmentation results for session S12 (lower part) . 259Figure C.12 Analysis of segmentation results for session S13 (upper part) . 261Figure C.13 Analysis of segmentation results for session S13 (lower part) . 262Figure C.14 Analysis of segmentation results for session S14 (upper part) . 264Figure C.15 Analysis of segmentation results for session S14 (lower part) . 265Figure C.16 Full screenshot of a rule-based model on S1 . . . . . . . . . . 267Figure C.17 Partial screenshot of a rule-based model on S3 . . . . . . . . 267Figure D.1 Abstracted versions of concrete tasks for two datasets . . . . . 269Figure D.2 Choices of node colors . . . . . . . . . . . . . . . . . . . . . 270Figure D.3 Highlight nodes with specific properties . . . . . . . . . . . . 270Figure D.4 Search a node by its label . . . . . . . . . . . . . . . . . . . . 271Figure D.5 Latent space with bundled edges . . . . . . . . . . . . . . . . 271Figure D.6 Corgi paw-like menu for selected nodes . . . . . . . . . . . . 272Figure D.7 UpSet style alternative design for K-hop topology . . . . . . . 273Figure D.8 Partial adjacency matrix . . . . . . . . . . . . . . . . . . . . 274Figure D.9 Count overview of neighbor set intersections of focal nodes . 274Figure D.10 Full screenshot of CorGIE on the Movie dataset . . . . . . . . 276Figure D.11 Check overview of the Movie scenario . . . . . . . . . . . . . 277Figure D.12 Check movie recommendations . . . . . . . . . . . . . . . . 278Figure D.13 Explore overview of Cora dataset . . . . . . . . . . . . . . . 281Figure D.14 Find problematic node pairs in the Cora dataset . . . . . . . . 282Figure D.15 P1 explored the decision boundary in the Cora dataset . . . . 283Figure D.16 The USAir dataset without focal nodes . . . . . . . . . . . . 285Figure D.17 Find buggy node pairs . . . . . . . . . . . . . . . . . . . . . 286Figure D.18 Screenshot of P2 focusing on a buggy node pair . . . . . . . . 287xxiAcknowledgmentsFirst of all, I would like to express my deepest appreciation to my supervisorTamara Munzner. I met her for the first time in a remote interview when I wasapplying for UBC back in February 2015, and I was impressed by her kindnessand wisdom. It has been six years since I first boarded the flight to Vancouver, andI am still impressed. I am impressed by her great interest in novel ideas, her atten-tion to detail throughout a research project, and her hard-working towards a highacademic standard, which has greatly influenced my taste of research. She taughtme how to conduct scientific research with rigor, how to reason about visualizationsystematically, and how to think and write critically. She is a coauthor of all fourpapers described in this dissertation, and I enjoy working with her a lot. She is kindand caring to me in daily life, especially for allowing me to work remotely (evenbefore the pandemic) to maintain my long-distance relationship between China andCanada. It is hard to express my gratitude completely with mere words, but there isa Chinese phrase, “ 生命中的贵人 ” (my best translation: life-changing savior).It is said that there could be only three in one’s life. I believe Tamara is one of thethree for me.Many have helped me in my academic journey.I thank my PhD committee, Giuseppe Carenini and Michiel van de Panne, for theirsupport and guidance on the trajectory of my PhD program. I thank the examinersof my dissertation, William Evans, Sathish Gopalakrishnan, and Enrico Bertini, fortheir thoughtful comments, which helped improve this document. I thank XiaoruxxiiYuan, my advisor in my undergraduate study in Peking University, for introducingme to the field of visualization and helping me in job-hunting at the end of my PhDprogram.I have enjoyed the collaboration with my paper coauthors: Tamara Munzner, ShingHei Zhan, Takayuki Itoh, Jessica Dawson, Zhicheng (Leo) Liu, Ju¨rgen Bernard,and Yang Wang. Shing, as the biology expert in ADVIEW, taught me biology andspent plenty of time with me to solve the phylogenetic problem. Takayuki is avisiting professor of our lab in 2018, who has a common research interest with me.We worked on the graph layout problem in SPRAWLTER. Even though I have nevermet Jessica in person, her work on the graph readability directly motivated andguided the SPRAWLTER project. Leo was my mentor when I was an intern in AdobeResearch in the summer of 2018. He guided the LOGSEG project and provided lotsof resources. Ju¨rgen and Yang provided lots of input when we worked together onthe CORGIE project.I have enjoyed interacting with the members of the UBC InfoVis reading group:Dylan Dong, Matthew Brehmer, Anamaria Crisan, Kimberly Dextras-Romagnino,Madison Elliott, Georges Hattab, Shannah Fisher, Steve Kasica, Francis Nguyen,Ju¨rgen Bernard, Mara Solen, and Ben Shneiderman. I enjoyed hanging out withthem. I appreciate their feedback for my research and especially my manuscripts.I thank many others who have provided feedback or inspirations for my research,including Sarah Otto, Sean Graham, Wayne Maddison, Daniel Archambault, DavidAuber, Fabian Beck, Hamish Carr, Donghao Ren, Bowen Yu, Meng Qu, andMuhan Zhang. I thank all anonymous reviewers of my papers.I am grateful for the financial support from Natural Sciences and Engineering Re-search Council of Canada (NSERC), Adobe, Uber, and UBC.I thank the administrative staff of the Computer Science department, especiallyJoyce Poon and Laura Selander, for helping me with the paper work during myPhD program.I thank my families and friends for supporting me in various ways. My cousinsxxiiiand their families who live in Vancouver have kindly invited me to holiday din-ners and offered help since 2015. Michael Chen and Hanchen Hou were my lovelyhousemates for more than four years. Michael was my badminton and snowboard-ing buddy, and my guitar teacher. Chenglong Wang played video games with mewhile gossiping about academics and discussing interesting research ideas. Mylandloard Jack Song and his kind family provided me with cozy and affordablehousing. They kept me sane in my busy PhD life.I thank my families for their unconditional love and support over the years: myfather Peixin Liu, my mother Xueping Lv, and my elder brother Zijian Liu. Theysupported my decision to pursue a PhD degree even when we were under enormousfinancial stress. I thank my uncle Huanlun Liu for his financial and emotionalsupport.Lastly, I thank my girlfriend and best friend, Luyao Zhang, for her love, care, andcompany in this long journey despite the fact that we were in different countriesand thousands of miles apart physically.xxivChapter 1IntroductionIn the digital era today, data, or, digitalized information, is collected with a grow-ing scale in many domains, which also grows the need to analyze data. Data anal-ysis is the name for this ubiquitous process to retrieve useful insights and informdecision-making. It is a broad research topic with many aspects and approaches.In this dissertation, I would like to distinguish between two kinds of approachesfor data analysis: automation and human-in-the-loop. Automation, also known asend-to-end solution, is usually convenient and fast in reaching the analytical goal,but it only works in certain conditions. The data needs to be cleaned or transformedto a dedicated format, the analytical task needs to be defined clearly, usually in themathematical form, and the stakeholders needs to trust the data, automation pro-cess, and the output. For example, to obtain the correlation of two scalar variablesin a clean dataset, a statistician can click a button in software to compute their Pear-son correlation value. However, most real-world problems do not come with cleandata, and they are not always clearly defined. Even in the example of computingcorrelation of two variables, a statistician can understand and trust Pearson’s cor-relation, but a random person on the streets might not understand it, or a professormight not trust the student’s calculation in a quiz. Cases with raw data, fuzzy tasks,or low trust usually require stakeholders to intervene in this process manually, or,require human-in-the-loop analysis.1Figure 1.1: Munzner’s nested model [124] showing four layers of visualiza-tion concerns.Visualization makes graphical representations from data for users and allows themto interact with the graphics. It has become an important tool for human-in-the-loop data analysis. Visualization accelerates the analytical process by reducinghuman cognitive load, helps to obtain a deeper understanding of data by revealinghidden patterns from aggregation, and communicates better to stakeholders withintuitive graphics.1.1 Four layers of visualization concernsMunzner proposed the nested model that contains four layers of visualization con-cerns [124], as shown in Figure 1.1, which I will borrow to describe how to sys-tematically reason about visualization for human-in-the-loop data analysis.In the nested model, we, as visualization researchers, first characterize the domainof the targeted real-world problem. We then abstract the data that need to be ana-lyzed and tasks of stakeholders (users) as design requirements. Based on the designrequirements, we design a suitable visual encoding to map the data to graphicalrepresentations, and interaction to link multiple representations. We finally deviseeffective algorithms to implement the design. The model describes a cascadingrelation between layers: the output of an upstream layer is used by the downstreamlayer, and a downstream layer supports the upstream one. Therefore, it often re-2quires solutions for the current and downstream layers to address the challenges inan upstream layer.1.2 ChallengesI will situate the challenges that we1 face in the reported projects in this model. Ingeneral, the outermost layer is the closest to users, while the innermost one is thefarthest. The motivation of our work is sometimes ignited by domain experts, andsometimes by the need of a visual encoding. We then identify the challenges inthis igniting layer as well as the downstream layers.In project ADVIEW, we target the domain of phylogenetics in biology. The disci-pline of phylogenetics studies the relationships between species and genes, whichare often represented with tree data structures. For the data and task layer, biolo-gists are capable of obtaining hundreds or even thousands of trees, and they needto compare these many trees in order to retrieve biological insights. The tasksof interest involve multiple granularities in a dataset, ranging from the whole treecollection, to a leaf node. For the encoding design layer, previous visual repre-sentations are either not efficient enough in terms of pixel real estate to handle thescale of data, or do not address the granularities required by the tasks. For thealgorithm layer, we need efficient algorithms to deal with the scale of data, that isnot only about the number of trees, but also the number of subtrees and leaf nodes.In project SPRAWLTER, we identify challenges in the visual encoding layer relatedto node-link graphs. A node link graph represents nodes as points and links asconnected lines, and it is the most popular visual representation for graph data. Thepositions of nodes are determined by a graph layout. A good graph layout must bereadable by users, thus making readability an important metric for graph layout.Previous readability metrics do not account for different levels of visual saliencyin graph data. For example, salient metanodes in multi-level graphs and variablenode sizes in single-level graphs are ignored in previous metrics; small nodes that1I will use the pronoun we to reflect collaborative nature of the work in this dissertation unlessspecified otherwise.3are hardly visible (no visual saliency) are not penalized. For the algorithm layer,the remedy for these problems should be able to handle large graphs without toomuch computational overhead compared to previous ones.In project LOGSEG, we work in the domain of image editing software. Typicalimage editing software like Photoshop is powerful but complicated. For the dataand task layer, developers seek to understand how the users use their product, typ-ically by collecting and analyzing user action logs. These logs could be huge, as askilled user is able to perform hundreds of actions in a short period of time. Manyuser tasks require an underlying segmentation of the log sequences, that is, divid-ing the logs into meaningful chunks of user actions. However, the challenges arenot only about the large scale of data, but also the variety of user tasks that relyon segmentation: different tasks require different segmentation granularities. Forexample, smart undo, a handy function allowing users to dial a group of actionsback at once, requires a low-level chunking, while managing an overview of workmilestones requires a high-level chunking. For the algorithm layer, the segmenta-tion method should run fast enough as a backend thread without interfering withthe regular functionalities of the software.In project CORGIE, we identify challenges in the domain of graph machine learn-ing. Graph neural networks (GNNs) are a class of powerful machine learning toolsthat model node relations for making predictions of nodes or links. For the data andtask layer 2, GNNs leverage the topology and node features in an input graph toproduce a node embedding, a common middleware byproduct, and then make pre-dictions using the embedding. To evaluate the model, developers focus heavily onthe final predictions, but they ignore the common data middleware of a GNN, thenode embedding. The missing opportunity to connect the node embedding to its in-put graph makes it difficult to understand whether the GNN has truly learned fromthe input graph as expected, for example, whether different levels of node struc-tures are preserved in the node embedding. For the encoding design layer, previousgraph visual encoding cannot reveal the neighborhood structures for specific nodes2To avoid confusion in this dissertation, we use “task” for analytical user task, and “machinelearning application” or “downstream application” for the prediction.4Figure 1.2: Illustrations of multi-level structures, either explicit in the data orfunctional to the user tasks.of user interest, but information about topological neighbors of nodes is crucial inwhat GNN learns. For the algorithm layer, similarly to the other projects, we needto support graphs with many nodes and edges and ensure a usable interactive speedfor users.1.3 Multi-level structuresFrom the four different problems described above, I identify a common challengethat relates to data in multiple levels of detail, or, multi-level structures. As illus-trated in Figure 1.2, for a dataset consists of many data items, we construct nestingsubsets of data items according to the user tasks, resulting in a hierarchy, or, multi-level structure.For hierarchical data such as trees and multi-level graphs, we can exploit the levelsthat come with it. In the biology project ADVIEW, the relevant levels from top tobottom are the whole tree collection, subsets of trees, an individual tree, a subtree,and a branch or a leaf node. In SPRAWLTER, we aim to take into account themultiple levels of visual saliency in readability metrics of node-link graphs. Multi-level graphs come with a node hierarchy, which we can directly exploit.For non-hierarchical data, we can compute the levels to serve the user tasks. Inthe software usability project LOGSEG, we need to segment the user action logsin multiple levels for different user tasks. In the graph machine learning projectCORGIE, we identify a missing opportunity in the evaluation of GNNs to help usersunderstand the structural preservation of nodes in different levels of detail from the5whole graph to a node neighborhood.Through these projects, I find that multi-level structures are ubiquitous in manydomains. In complex real-world problems, the analytical tasks often require usersto explore the multi-level structures in data, which could be either hierarchical ornon-hierarchical.1.4 ContributionsIn this dissertation, I contribute visualization tools to the target domains in the fourprojects, as well as visual techniques and algorithms to the visualization commu-nity. These contributions exemplify how to visualize the multi-level structures indata for human-in-the-loop data analysis. I summarize the contributions of eachproject as follows.ADView (Chapter 2)• The data and task abstraction for comparing phylogenetic trees.• A new visual encoding, the Aggregated Dendrogram (AD), to compressa dendrogram into a small resolution-aware visual representation showingtopological relationships between subtrees of interest.• An interactive visual tool, ADView, to compare one phylogenetic tree againsta collection of hundreds of other trees using multiple linked views.• An evaluation with multiple methods validating the utility of AD and AD-View.Sprawlter (Chapter 3)• An area-aware approach for three families of clutter metrics: node-node,node-edge, and edge-edge crossings.6• The sprawlter metric to capture the trade-off between sprawl and clutter.• An evaluation of the two metrics by comparing to traditional count-basedmetrics on many graph layouts.LogSeg (Chapter 4)• A multi-level segmentation model for image editing tools, where the low-level segmentation works reasonably well for smart undo, and the adjustablehigh-level one is based on the low-level one.• An analysis of feature relevance to log segmentation, where we find thatcommand, timestamp, and layer are relevant attributes, but image content isnot as useful. We are the first to use layers for segmentation and confirmtheir relevance.• An evaluation of the low-level chunks showing our model generalizes acrossthree image editing scenarios, and a preliminary analysis of the suitability ofhigher-level chunks for these three scenarios.CorGIE (Chapter 5)• The data and task abstraction for visually exploring the correspondences be-tween an input graph and the middleware node embedding, to understandif a GNN model has learned important characteristics of the input graph toconstruct the embedding.• An interactive multi-view interface, CorGIE (Correspondences between aGraph and Its Embedding).• A new graph visual encoding, the K-hop layout, to reveal how a GNN ag-gregates information from node neighborhoods.In Chapter 6, I reflect on the multi-level structures in these projects, and the col-laboration with many coauthors, before concluding this dissertation.7Figure 1.3: A summary of the evaluation methods we used in different layersof Munzner’s nested model [124].1.5 Evaluation methodologyI summarize the evaluation methods used in these projects by layers in Munzner’snested model, as solutions for different layers require different methods. Figure 1.3shows a summary of the evaluation methods I used in the four projects.To evaluate the data and task abstraction, we interview domain experts on whetherthe characteristics of data and the identified tasks actually match their needs andfit their mental model. We review literature that describes the data and tasks inboth the target domain and the visualization field. In SPRAWLTER and LOGSEG,we compute quantitative metrics and inspect results qualitatively. In ADVIEW andCORGIE, after implementing the visualization tools, we demonstrate them to thedomain experts (users) and get direct feedback, we test their utility in demonstrativeusage scenarios ourselves, and we also recruit domain experts to use them on theirown datasets. As these methods require the implemented tools, they also evaluatethe downstream design and algorithm layer. Note that in SPRAWLTER, the users ofnode-link graphs could be anyone with basic visualization training, and thus we donot need external domain experts.8To evaluate the visual encoding and interaction design, we discuss the design ratio-nale, that is, to reason whether the visual marks, channels, and layouts can addressthe data and tasks in the context of the visualization literature. We compare theproposed design against alternatives in terms of the suitability and scalability ofthe data and tasks. The visual design should be suitable for the kinds of data (e.g.,scalar vs. categorical values) and for the tasks (e.g., showing aggregated summariesof interesting data points vs. their identities). The visual design should be scalableto the number of data points and the number of data subsets in the levels of detailrequired by the tasks.To evaluate the algorithm implementation, we focus on time and memory usage.We perform complexity analysis with the canonical big O notation, and run the al-gorithms to measure elapsed timings and memory consumed on my computer. AD-VIEW and CORGIE, as web-based tools, are measured on browsers, while SPRAWL-TER and LOGSEG are measured with Python scripts.9Chapter 2ADView: AggregatedDendrograms for VisualComparison Between ManyPhylogenetic Trees1 We address the visual comparison of multiple phylogenetic trees in evolutionarybiology. Specifically, we compare one reference tree to a collection of dozens tohundreds of other trees. We abstract the domain questions of phylogenetic treecomparison as tasks to look for supporting or conflicting evidence for hypothe-ses that requires inspection of both topological structure and attribute values atdifferent levels of detail in the tree collection. We introduce the new visual en-coding idiom of aggregated dendrograms to concisely summarize the topologicalrelationships between interactively chosen focal subtrees according to biologicallymeaningful criteria, and provide a layout algorithm that automatically adapts to theavailable screen space. We design and implement the ADView system, which rep-1This chapter is a slightly modified version of the following publication: Z. Liu, S. H. Zhan, andT. Munzner. Aggregated Dendrograms for Visual Comparison Between Many Phylogenetic Trees.IEEE Transactions on Visualization and Computer Graphics (TVCG), 26(9):2732–2747, 2020 [112].10resents trees at multiple levels of detail across multiple views: the entire collection,a subset of trees, an individual tree, specific subtrees of interest, and the individ-ual branch level. We benchmark the algorithms developed for ADView, compareits information density to previous work, and demonstrate its utility for quicklygathering evidence about biological hypotheses through usage scenarios with datafrom recently published phylogenetic analysis and case studies of expert use withreal-world data, drawn from a summative interview study.2.1 MotivationBiologists who study phylogeny, which aims to resolve evolutionary relationshipsamong all living and extinct organisms, often find it difficult to make sense of largecollections of phylogenetic trees. These evolutionary trees are typically computa-tionally inferred from living organisms. The ability to gather genetic data quicklyand cheaply has led researchers to generate and analyze more and larger trees [101],including intermediate trees that arise within a computational pipeline for recon-structing a final tree. They frequently need to inspect these trees to consider theirplausibility and meaning with respect to existing domain knowledge. They do notblindly trust automatic computational metrics, especially when dealing with un-known genes and species or when developing new concepts and methods. Currentvisual analysis tools fall short of their need to make comparisons between largenumber of trees at multiple levels of detail.In the visualization community, visual comparison of trees has been studied foryears [61], but existing visualization techniques do not scale to the current needsof phylogenetic researchers. We characterize scalability of visual tree comparisonin terms of three factors: the number of nodes per tree, the number of trees beingcompared, and level of detail at which the internal structure needs to be shown.Different tasks require very different levels of detail. These factors have separablerequirements with respect to screen real estate and human cognitive ability: tech-niques that are suitable for scaling up in one of these may not support the others.For example, a technique for inspecting clusters made of hundreds of trees does not11typically support fine-grained inspection at the subtree level within each one. Mostscalable visual tree comparison work has been in support of comparison betweena very small number of large trees. Approaches such as TreeJuxtaposer [126] areable to show highly detailed local structure, such as all descendants of a specificsubtree. A few systems tackle the inverse problem of handling a large number oftrees, where each tree is shown as a single point [77]. In this case, where a globaldistance measure is used to lay out those points, the available level of detail isan overview of the entire collection of trees. Between the two extremes, there issome work comparing dozens of trees at varying levels of detail, in many domainsranging from phylogenetic trees [19] to natural language processing [191].Modern phylogenetic research takes place at a variety of scales: the number of treescan range from one to over ten thousand trees, and the number of nodes per treeranges from a dozen to millions. The visual comparison of phylogenetic trees alsoencompasses a broad and complex spectrum of tasks. The research questions canrange from assessing the overall uniformity or variance across a large collection oftrees, to the exact placement of a particular leaf node across a small set of trees.Questions may pertain only to topological relationships between or within subtrees,or only to the distribution of attributes associated with nodes or branches, or to thecombination of both. The problem space is huge; we worked with a group ofphylogenetic researchers to understand their specific unmet needs.We present ADView, an interactive system to visually compare one tree againsthundreds of other trees, showing multiple levels of detail across multiple linkedviews. We propose an innovative technique to compress a tree into a concise repre-sentation that we call an aggregated dendrogram (AD), which captures the majortopological relationships between several subtrees that are the focus of user in-terest. Our best-effort layout algorithm yields an AD that can adapt its size andthe amount of presented information according to the available screen space. Wecluster similar ADs to generate a visual summary of topological relationships be-tween chosen subtrees that reveals information across multiple scales for datasetsof hundreds of trees, enabling more detailed comparison than previous tools canachieve. ADView also includes a view showing distributions of trees in terms of12their agreement and conflicts of leaf memberships for the chosen subtrees. It hasan on-demand view for pairwise comparison in full detail, and auxiliary views forattribute values and other information. We show the results of using ADView onreal-world datasets, including usage scenarios based on published datasets and casestudies arising from an interview study with five domain experts.In summary, the contributions of this chapter are:• task and data abstractions for the comparison of phylogenetic trees;• an interactive visual tool to compare one phylogenetic tree against a collec-tion of hundreds of other trees using multiple linked views;• a technique to compress a dendrogram into a small resolution-aware visualrepresentation showing topological relationships between subtrees of inter-est;• validation through algorithm benchmarks, usage scenarios, formative expertfeedback, and a summative expert interview study that provided case studies.2.2 Related workWe discuss the visual representation of trees in general, in the biology domain, themany approaches to tree comparison, and task abstractions for trees and networks.2.2.1 General tree visualizationTree comparison falls within the very active research area of general tree visual-ization. The treevis survey by Schulz [155] enumerates hundreds of general tech-niques to visualize single trees, but they do not suffice for our chosen problem oftree comparison, which is more complex than displaying one tree[57]. The spaceefficiency of many tree representations is studied by McGuffin and Robert [118].13We were inspired by their efforts when designing the aggregated dendrogram. Pre-vious approaches to the compression and simplification of general trees and graphsaccording to their statistical properties [9, 120] do not suffice to support our tar-geted tasks for phylogenetic analysis such as preserving monophyletic groups.The DOITrees approach [71] dramatically simplifies trees and like ours is a fo-cus+context technique that uses elision rather than distortion, but does not supportthese targeted phylogenetic analysis tasks. Pretorius et al. [138] propose a visualencoding for cell lineage visualization, but it is focused on different requirementssuch as emphasizing symmetry.2.2.2 Biological tree visualizationBiologists routinely use domain-specific tools such as FigTree [146], iTOL [103]and Dendroscope [86] to visualize and annotate a tree during analysis, and to gen-erate publication-ready figures, but these tools do not support comparison at all.Biologists who are capable of programming could use Python or R libraries suchas DendroPy [165] and Ape [132]. These libraries are handy for flexible data ma-nipulation such as trimming and computing an additional attribute, but fall far shortof the kinds of visual comparison that ADView provides.2.2.3 Visual tree comparisonWe categorize previous literature related to visual tree comparison by three dimen-sions of scalability: the number of trees, the level of detail at which the data mustbe shown to accomplish the intended task, and the number of nodes per tree. Fig-ure 2.1 illustrates the first two dimensions of this problem space. The vertical axisshows the number of trees partitioned into four categories: pairs, dozens, hundreds,and thousands. The horizontal axis shows the level of detail representing how manylevels of detail a tool can show, in terms of three rough groups: a single point, sim-plified structure, and the full topology. We identify three major groupings: Few inFull, Dozens at Multi-Scale, and Many as Points.14Figure 2.1: The problem space of visual tree comparison for two scalabilitydimensions.The Few in Full category has been heavily studied. An extensive list of these sys-tems appears the survey by Graham and Kennedy [61], and all of the entries inthe InfoVis 2003 Contest on tree comparison fall into this category [136]. Withonly a small number of trees to handle, often just two, the most scalable previoussystems can deal with a massive number of nodes per tree. For example, the Tree-Juxtaposer system [126] supports detailed structural comparison of large trees upto several hundred thousand nodes. A similar system was recently re-implementedby Phylo.io as a web-based tool [151]. Some propose superposition to stack treesvisually, such as the color coding of CandidTree [100] and the explicit encodingof Beck et al. [12]. Several systems proposed drawing lines connecting the samenodes on two trees [41, 79, 176]. Tree comparison is a special case of the more gen-eral problem of graph comparison. For example, Archambault proposes a pairwisecomparison approach of hierarchically coarsening difference graphs [6]. These ap-proaches by definition do not handle the larger collections of trees that we focus onin this work.At the other end of the spectrum, the Many as Points systems handle hundredsor thousands of trees, with no limits on the number of nodes per tree. As ourgroup name suggests, each tree is represented by a single point and thus these sys-15tems provide only a high-level overview of relationships between trees accordingto some kind of similarity metric. Hillis et al. employ dimensionality reduction tomap trees to a 2D space [77], revealing only cluster structure. Strimmer et al. pro-pose Likelihood Mapping to visualize likelihoods for different hypotheses within atriangular coordinate system [163], but this tool is tailored for testing hypothesizedrelationships between four sequences or species families, and does not addressmore general problems. Fangerau et al. [49] compute a signature value for eachtree and arrange them along a Hilbert curve to show structurally similar trees, andHess et al. [76] cluster trees into a hierarchy to make sense of different parametersettings used in tree generation. The general idea of reducing complex structure topoints is also applied to dynamic graph visualization by van den Elzen et al. [171].These approaches cannot show the multiple levels of detail required for our targettasks.The Dozens at Multi-Scale category comprises systems that handle from 2 to roughly100 moderately-sized trees of around 100 nodes per tree, where at least two levelsof detail are provided. Graham et al. propose a multi-view system for showingtrees in different levels of detail using treemaps, node-link diagrams and lists [60],but they handle fewer trees than ADView and only a subset of tasks in Section 2.4.In DAViewer [191], Zhao et al. propose a compact view to aggregate informationalong two directions: from root to leaves, and for nodes across the same tree depth.Although it addresses questions about distributions and errors, it is not suitablefor comparing topological structures at the subtree level and their choice of den-drogram does not scale to collections of hundreds of trees. PhenoBlock [58] usessmall multiples to visualize many patient phenotypes compared to a single refer-ence tree. Their solution takes advantage of the fact that phenotype ontology isfixed across all trees, so it would not work for the dynamic situations we presentin Section 2.4. TreeVersity2 [64] embeds bar charts in trees to visualize changesof attribute values on nodes, while Vehlow et al. [174] and Bach et al. [11] dealwith temporal changes of trees over time. Li et al. [105] proposed a new tree com-pressed representation with the focus on node attributes. All these systems supportdifferent data abstraction than ADView.16Consensus trees [20] are traditionally used by biologists to reason about collec-tions of trees, where the places of agreement are shown as a standard tree and thedisagreements are indicated by collapsing multiple leaves together under a singleparent. Although they can be effective to pinpoint sparse disagreement in only afew areas, when substantial differences exist within a large collection of trees thissimple approach breaks down. The summary graph of multiple domain specificgraphs [97] proposed by Koop et al. has a similar limitation. DensiTree2 [18] useshalf-transparent fuzzy branches to imply uncertainty in regions of difference, butthis visual encoding shares the limitations of conventional consensus trees: fine-grained comparison is impossible when there is too much variation.The previous work from Bremm et al. [19] is the most similar to ADView. Theycompare multiple phylogenetic trees at both global and local scales via topologicalsummaries, for dozens of trees with dozens of leaves per tree. They also proposea new distance measure to capture structural differences between subtrees. Theirapproach for reducing visual complexity of trees is to retain the full path from aselected node all the way up to the root and hide the others. In contrast, our aggre-gated dendrogram method for simplifying trees, which preserves focal subtrees andcarefully chosen local context while hiding more distant upstream nodes, is morecompact and scalable. Moreover, our simplification approach can automaticallyadapt to available screen space. We present a detailed comparison of our resultsto theirs in Section 2.8.2, showing their limitations in handling for our identifiedtasks. In addition, they require all species under study to be present in all trees;in contrast, ADView can handle missing species, a situation faced by our domainexperts that commonly occurs in phylogenetic analysis.2.2.4 Task abstractions for treesGleicher offers some structural thoughts on the general task of visualizing compar-isons [57]. The three papers devoted to task abstractions for networks only covera subset of the tasks that we present in Section 2.4. Lee et al. [99], Pretorius etal. [139] and Kerracher et al. [91] cover tasks such as lookup by name, finding at-17tribute ranges, and general questions about topology inspection. However, they donot address tasks such as finding corresponding branches or conflicting taxa acrossa tree collection, much less the questions very specific to phylogeny such as under-standing monophyletic groups. Several previous papers that propose techniques forphylogenetic tree comparison do present abstractions for more specialized tasks,but none cover all seven of the tasks that we propose and none explicitly identifyall five of the relevant levels of detail that we characterize in the data.2.3 Phylogenetic tree dataWe introduce some basic domain terms that pertain to phylogenetic tree data andpresent an abstract data specification.2.3.1 Phylogenetic treeA phylogenetic tree describes the evolutionary history of living organisms. Leafnodes are called taxa, and are extant species that are being studied. Internal nodesrepresent common ancestors, usually inferred by statistical methods of phyloge-netic reconstruction. Tree collections may arise from multiple reconstruction com-putations with different parameters or assumptions. A subtree is composed of alldescendants beneath a branch; the set of all taxa underneath a specific branch formsa monophyletic group, also called a clade. There is a 1-to-1 correspondence be-tween a branch, an interior node, a subtree, and a monophyletic group.Tree collections also arise when inferring gene trees that represent the differentevolutionary histories of individual genes; the synthesis of these multiple historiesinto a single overarching species tree requires extensive comparison between thesetrees.182.3.2 Data abstractionPhylogenetic researchers conduct comparison with respect to a single special treethat can be interpreted as their main hypothesis, where they are looking for evi-dence to support or invalidate the hypothesis from a collection of other trees. Wedenote the special hypothesized tree as the reference tree and the other trees as thetree collection. Both are required as input data for ADView. Two typical scenariosare either one species tree vs. its associated gene trees, or one species tree vs. mul-tiple species trees generated by different methods or parameters. The leaf nodes inthese trees come with names, but internal nodes are usually not labeled.Each tree branch has three attributes, two original and one derived. The primaryoriginal attribute is a support value associated with the branch, which is generatedas an (un)certainty measure of how confident the reconstruction program is in itsdecision that the underlying leaf nodes should be grouped together. Each branchalso has the secondary original attribute of branch length, which measures a dis-tance from its ancestors related to evolutionary time. Although none of our targettasks directly rely on branch length, these lengths can provide useful context forbiologists and are also sometimes used for sanity checking the data.While the entire group of these trees has substantial overlap in the set of taxa, inmany datasets an individual tree will have missing taxa. For example, in a collec-tion of trees about the evolution of animals, some trees might not have kangarooswhile others might not have turtles. We assume that trees are all rooted and binary.We also assume that every leaf node is unique in a tree; in biological language, thatthere are no paralogs.Many of the tasks presented in Section 2.4 require the derived data of correspond-ing branches: for a specific branch in the reference tree, we must find its counter-part of the most similar branch in each tree of the tree collection, as illustrated inFigure 2.2(b). For example, the monophyletic group exploration task (T5 in Sec-tion 2.4) hinges on whether there is an exact match with similarity exactly equalto 1.0 between a reference tree branch and a corresponding branch, or an inexactmatch where the similarity is strictly less than 1. This computation is similar to19Figure 2.2: Task illustrations. (a) Three topological relationships among sub-trees A, B, and C: A and C are separated, A and C are sisters, A isnested beneath C. (b) Tree1 agrees and Tree2 disagrees with Referenceon the monophyletic taxa group (leaf set) under branch A. Referencetree branch A has the corresponding branches A1 and A2 in Tree1 andTree2 respectively.the best corresponding node (BCN) proposed in TreeJuxtaposer [126]; algorithmdetails are provided in Section 2.7.1.We use corresponding branches to also derive a third attribute, the gene supportfrequency (GSF), on every reference tree branch: the percentage of trees in theentire tree collection with exact matches for that branch. This metric capturesuncertainty information of a subtree in the reference tree with respect to a treecollection, where support values usually fall short; several other biological metrics,such as the internode certainty [159], have similar goals.2.4 Task abstractionThe overarching goal in ADView is to compare a single reference tree to a col-lection of many other trees. We break down this goal into a set of seven specifictasks that were identified through multiple rounds of interviews with four biolo-gists who work across multiple sub-fields of phylogenetic research including plantphylogeny, insect phylogeny, and mathematical modeling. We held weekly meet-ings with one of them, who specializes in plants and is a co-author of this chapter,20to allow frequent checks on the validity of our task abstractions and the utility ofour visual interface during incremental refinement. In addition, we reviewed thebiology literature on phylogenetic reconstruction and analysis.In the language of Kerracher and Kennedy [90], we take a multi-strand approachto task gathering, primarily via interviews with domain experts and secondarilyby deriving from literature. Two potential threats to validity did not apply in ourcase: the domain experts were indeed available, we were able to maintain a focuson task discussions, and skewing tasks towards a single domain is a feature not abug given our goal to target phylogenetics in particular. We addressed the potentialdifficulties in introspection by conducting many rounds of elicitation and followup.The seven tasks are:• T1: Find subtree by taxa names. Biologists are typically familiar with thenames of species in their study; an example is to find the subtree that consistsof mammals.• T2: Find branch / subtree with respect to an attribute range in the referencetree. The two attributes of support value and gene support frequencies areuseful in suggesting interesting areas to investigate.• T3: Pairwise compare one tree in the collection with the reference treein detail. Allowing biologists to inspect a tree in full detail alongside withthe reference tree allows them to make biological judgment calls about theirfindings. For example, they can see how species in a subtree are distributedin a target tree compared to the reference tree: whether there are any outlierspecies that are far away from others or scattered across the whole tree.• T4: Inspect topological relationships between multiple subtrees. Figure 2.2(a)shows three examples of relationships: nested, separated, or sisters. A sub-tree can be nested within another subtree. Otherwise, biologists would liketo know if they are separated or direct siblings, known as sisters. They alsowant to know which subtrees are the closest to each other, and have a roughsense of the topological distance between specific subtrees.21• T5: Explore the monophyletic groups (the complete leaf set of a subtree)in terms of agreement or disagreement between trees, as illustrated in Fig-ure 2.2(b): do the taxa nested beneath a branch in one tree (the three leavesto the right of branch A) all fall beneath the corresponding branch in anothertree? (They do for A1 in Tree 1 but not A2 in Tree 2.) How many and whichtrees agree with exact matches or disagree with inexact matches? Do theyagree with each other? How many alternatives are there?• T6: Investigate the conflicting taxa between disagreeing subtrees. Userswant to know which taxa cause the conflicts, allowing them to notice con-nections with previous findings such as whether a particular family of speciesis known to be hard to resolve.• T7: Assess the corresponding branch attributes to associate their valueswith the (dis)agreement for monophyletic groups. This task is lower pri-ority, but the uncertainty measure on the corresponding branches can helpbiologists to gauge their confidence on their findings. For example, a corre-sponding branch from a conflicting tree with an extremely low support valuemight signal a spurious conflict.The actual workflow of biologists is incremental and iterative, so these tasks canoccur in any order. We provide a few concrete examples of tasks sequences drawnfrom their analysis process, and discuss them further in Section 2.8:• T1 ↔ T2: sanity check on topology and attribute values of the referencetrees, then identify interesting subtrees.• T4↔ T3↔ T7: explore topological relationships, compare hypotheses, andsupport values distribution.• T5 & T6↔ T3↔ T7: check agreement and conflicts of taxa memberships,inspect details of a specific tree, check support values.These tasks led us to identify five relevant levels of detail: 1) tree collection, 2) asubset of trees, 3) individual tree, 4) subtree, and 5) individual branch or leaf node.22These five levels provide a new abstraction that is a more precise articulation ofthe problem space than the three levels illustrated in Figure 2.1, and informed thedesign of the Aggregated Dendrogram.2.5 Aggregated Dendrogram (AD)We propose a novel technique to aggregate tree representation that specifically ad-dresses the difficult problem of topological relationships between multiple sub-trees (T4) given hundreds of trees, and partially addresses the agreement of mono-phyletic groups task (T5). In this section, we indicate the motivation of the aggre-gated dendrogram, present its visual design, explain how similar ones are clustered,and share a glimpse of its evolution with a discussion of how earlier designs fellshort.2.5.1 Design requirementsIn essence, the problem comes down to how to visually compress a tree with spacerequirements. First, the size of the compressed representation should be small tofit in as many trees as possible within a single screen. Second, the informationcontained in the compressed representation should adapt to the available screenspace: given a larger area, it should convey more details than in a smaller one.We determine what should and should not be compressed according to our taskabstraction. Given a rooted tree and several focal subtrees, the critical informationto show visually is the type of relationships between focal subtrees: either nestedor separated and more specifically, (non-)sister relations (T4); also, whether thesubtrees agree with the reference tree or not (T5). It can be helpful to also showdistances between subtrees and more information about conflicting subtrees suchas neighboring clades and placement of the conflicting taxa. Our design target is tosupport up to five focal subtrees, to provide a reasonable tradeoff between powerand comprehensibility. Our investigation found that this number suffices for mostphylogenetic questions. We design for a screen resolution of 1920x1080.23Figure 2.3: Visual encoding of an aggregated dendrogram.2.5.2 Visual designDriven by a combination of the design requirements and inspiration from previouswork [19, 71, 118], we decided to take a non-distorting focus+context approachto compressing a tree, where less important information is elided [125, Chap. 14].Specifically, we aggregate the dendrogram representation of a tree, as the biologistsare most familiar with this form. Figure 2.3 presents the final visual design ofaggregated dendrogram (AD). In Section 2.5.4 we justify this design by discussingthe shortfalls of several alternative designs that were confusing to the biologists;our task abstraction evolved iteratively, as the less successful attempts surfaced ourmisunderstandings by violating previously implicit assumptions of phylogeneticanalysis.A crucial design choice is to have each rectangular block in an AD always representa subtree; that is, a monophyletic group in that tree. We also take care to visuallydistinguish whether each block is an exact match that agrees with a subtree inthe reference tree, or an inexact match indicating disagreement, through differentborder styling.The focal blocks represent the subtree beneath this tree’s branch that correspondsto a selected one from the reference tree. These blocks have thick black borders24and labels and their height is linear to the number of taxa inside, subject to the hardlegibility constraints discussed in Section 2.7.2. Context blocks are rendered asless salient small rectangles with a grey border.All AD block colors are consistent with the reference tree, where the proportions ofcolors indicate the percentage of highlighted taxa in this subtree. Exactly matchingblocks, which have solid borders, are always fully colored because they exclusivelycontain all taxa under the reference tree branch. Inexactly matching blocks, whichhave dashed borders, are sometimes partly colored due to other taxa that have in-truded, as shown in the partial blue block in Figure 2.3. Sometimes color appearsin non-focal blocks due to escaped taxa, namely matched taxa that are found out-side the corresponding subtree, shown by the tiny blue and orange context blocksbetween blocks A and B in Figure 2.4(c).There is only one kind of block that is not a subtree: the oval that can appear atthe very bottom of the AD represents all missing taxa, as shown in Figure 2.3. Itsrounded corners clearly distinguish it from the other rectangular blocks, and itsborder is dashed. The distribution of different highlight colors for taxa within thesame block is shown with vertically layered rows.Unimportant parts of the tree are collapsed, as indicated by the special slash markeron a branch. For example, in Figure 2.3, many evolutionary events along the pathfrom the root to A’s direct ancestor are elided, as we see from the slash crossing thedotted branch. Users can specify a collapse threshold of topological distance fromhighlighted blocks to either their lowest common ancestor or their nesting block.For example, Figure 2.3 was made with the threshold set to 2; block C is more than2 levels deeper than B, so everything between B and C is hidden. This threshold isa soft constraint in the legibility criteria discussed in Section 2.7.2.Our approach depicts the topological backbone relationships faithfully. In Fig-ure 2.3, the following information can be easily interpreted: A and B are parallelbut not sisters (direct siblings); C is nested within B with distance greater than two;A, B and C are far from the root (distance > 2).25Figure 2.4: Illustration of how the AD adapts to available space. These threeADs are based on the same tree and the same four focal subtrees, butwith different size specifications: (a) 40, (b) 80, and (c) 160 pixels.The AD algorithm, which we describe in detail in Section 2.7.2, adapts to avail-able screen space automatically, as shown in Figure 2.4. The algorithm makesa best-effort attempt to fit in as much information as possible, in terms of labelsand context blocks, while keeping visible the critical information such as selectedsubtrees and their topological relationships (as defined in Section 2.5.1).The individual ADs can be sorted by global distance to the reference tree, or bylocal distance with respect to a selected subtree.2.5.3 Visual summary: Cluster ADAlthough the individual ADs are spatially compact, when the tree collection is suf-ficiently large there can still be more than will fit within a single screen. We clusterthe ADs that share the same topological relationships (as defined in task T4) andjust render one representative AD, as illustrated in Figure 2.5. By grouping ADsinto clusters, we obtain a summary of all possible kinds of relationships amongfocal subtrees in the tree collection. The value of preserving individual views thatare topologically equivalent is they might end up highlighted differently depend-ing on what other selections are made, which would provide useful information (asdiscussed in Section 2.6.6).26Figure 2.5: Two examples of clustering structurally similar aggregated den-drograms. (a) Cluster groups 51 trees together where A and B are par-allel. (b) Cluster groups 4 trees together where A is nested within B.We ignore all context blocks while performing clustering, so that differences con-sidered unimportant according to our criteria do not result in scattered or evenspurious clusters. We randomly choose one of the constituent ADs from a clusterAD as a proxy for the entire cluster. The normal solid coloring is changed to agradient to indicate the different proportions of filled color across the blocks in allADs of that cluster. The clustering algorithm is presented in Section 2.7.3, and thegradient coloring is documented in Supplemental Section A.2.2.5.4 Design evolutionWe tried many unsuccessful versions of aggregated dendrograms, four of which areshown in Figure 2.6: remainder, container, fine-grained, and frond; it also showsthe final skeleton layout which was successful. Eliciting feedback about these ver-sions led to more clarity about what underlying biological meaning should be pre-served in terms of both the focal and the context parts of the visual encoding.One design question that required extensive iteration is the semantics for visible27blocks in the diagram. Our first two attempts, the remainder and the container lay-outs, did not adhere to our current semantics that a rectangular block must representa monophyletic group; that is, a complete subtree underneath a branch. We even-tually realized that identifying monophyletic groups was an underlying subtaskfor the larger-scale biological questions that were being studied, so all alternativeswhere rectangles could represent non-monophyletic groups were confusing to thebiologists. We then articulated T5 (monophyletic groups) as an explicit task duringour iterative refinement of the task abstraction, which occurred in parallel with thedesign evolution.The fine-grained layout did not scale past a few chosen subtrees. The frond lay-out also suffered from some scalability problems with excessive requirements forhorizontal nesting, and moreover elided so much context that important questionswere difficult to answer. The final skeleton layout was the result of several monthsof exploration in this very large design space of tradeoffs. Our final design featuresa careful mix of which parameters should be controlled by users, vs. hardwired,vs. adaptively computed.2.6 ADView interfaceWe first give a brief tour of ADView, followed by the details about each of theviews, and then discuss the view coordination in depth with respect to the fivelevels of detail.2.6.1 Interface overviewFigure 2.7 is a screenshot of ADView comparing a reference species tree againsta collection of 68 species trees generated with different phylogenetic reconstruc-tion methods. In the Reference Dendrogram on the left, four subtrees have beenselected by the user to be the current focal groups, labelled A, B, C, and D andhighlighted in blue, orange, green, and red, respectively. In the Tree Distribu-tion view near the top, each row shows how the tree collection can be grouped28Figure 2.6: Five versions of the aggregated dendrogram design in chronolog-ical order of invention. The first four were unsuccessful: remainder,container, fine-grained, and frond. The successful fifth layout, skeleton,built on lessons learned.Figure 2.7: Screenshot of ADView comparing a reference tree (left pane)against a tree collection (right pane).29by agreement or disagreement with respect to each focal subtree. Below that, theAggregated Dendrogram (AD) view consists of cluster aggregated dendrogramsand individual ones, showing topological relationships between the selected focalclades. Currently the user has selected the second cluster AD, highlighting 22 treeswith a brown background, and is hovering on the second segment of row B in theTree Distribution view, highlighting 12 trees with a black border. There are threeauxiliary views showing additional information for the trees.2.6.2 Reference Tree and AD ViewsThe Reference Tree view in the left pane of ADView is always devoted to show-ing the reference tree at full detail. We visualize this important phylogenetic treewith a branching style dendrogram with name labels for all taxa. This format isvery familiar to biologists, intuitively supporting the Taxa Name task (T1). Thehorizontal length of a branch either encodes the branch length attribute, or uniformlengths can be chosen if high variance in branch lengths would lead to visual arti-facts. In the Reference Branch Attribute view at the top of that pane, we show thedistribution of the two main branch attribute values, the support values from rawdata and the derived gene support frequency (GSF). The pink range selection areacontrols which branches that match the chosen attribute range are highlighted inpink in the Reference Dendrogram, in support of the Attribute Range task (T2).The crucial interaction for ADView is to select a subtree in the reference treeand inspect the support and conflicts in the tree collection shown in the Aggre-gated Dendrogram views, in support of the Topological Relationships (T4), Mono-phyletic Groups (T5), and Conflicting Taxa tasks (T6). When the user picks aninteresting subtree / branch, that subtree is labelled and colored in the ReferenceDendrogram, a new highlighted block representing it is added to all of the aggre-gated dendrograms, and a row is added in the Tree Distribution view. ADView alsoallows for selecting a user-specified taxa group to handle cases where the user isinterested in an arbitrary set of taxa that is not a monophyletic group in the refer-ence tree, combining multiple choices using a popup menu rather than selecting a30Figure 2.8: Selecting a segment in the Tree Distribution view shows dots foreach taxon within its group in the reference tree, temporarily in placeon hover (above) or right-aligned for comparison between persistent se-lections (below).single reference-group subtree.We provide a simple layer of automatic control for AD size that takes into accountthe number of trees that need to be drawn in each of the two views, the IndividualAD and Cluster AD, to make intelligent use of the available screen real estate. Thegoal is to set a target size that preserves a reasonable amount of detail, ideallywhile fitting all of the ADs into a single pane without requiring vertical scrolling.When the number of trees is too large to fit even with small individual sizes, theleftover ones that do not fit within the pane are elided by default; the user can showthem with the More button. The user can override this automatic size control byusing a manual slider to indicate a specific desired height and width for all ADs.In both cases, the best-match AD layout algorithm is used to create layouts for thespecified size. This size control mechanism was added late in the design process,after watching users miss opportunities to obtain better overviews.2.6.3 Tree Distribution ViewThe Tree Distribution view at the top is focused on the Monophyletic Group (T5)and Conflicting Taxa (T6). Each row consists of multiple horizontal segmentsthat represent groups of trees in the collection that agree with respect to the focal31subtree, as shown in Figure 2.8. A segment represents the subset of trees whosematching subtrees share the same taxa set with each other. The first segment showsthe subset that agrees with the reference tree, which is marked with a circled Rmarker (R for reference). The following segments, sorted by frequency from left toright, are the conflicting subsets. Each subset indicates one hypothesis of the taxaset of a selected subtree, and thus the full row shows all alternatives of the taxa setthat exist in the tree collection with respect to a specific branch.When the user persistently selects or hovers over a segment, dot markers appearbeside the taxa labels for that subtree in the reference tree dendrogram. In the topexample of Figure 2.8, the user hovers over the middle segment of row B, andwe see from the black dots that the taxa set for this subset of trees does not haveSPIROGYRA SP. In the bottom example, the user has persisted the dot markers forthe middle and right segments in row A, and is now hovering on the left segment.The persistent selection triggers the reference tree labels to move from ragged-right layout to be right-aligned, to allow easy comparison of multiple columns ofdot markers. The middle segment (labelled 1) has two taxa absent compared to thereference tree; that is, their closest subtrees to A do not include the first two taxa.The rightmost segment (labelled 2), which is very narrow because it only containsa few trees, is missing nearly half of the taxa in A.2.6.4 Pairwise comparisonUsers can select a specific tree from the tree collection for Pairwise Comparison(T3) with the reference tree in a butterfly layout, as shown in Figure 2.9. Thismirrored layout, where roots are on the outside and taxa labels are close to eachother, is a familiar idiom for biologists. The highlighting is linked between thetwo trees, allowing users to locate taxa of interest in the target tree. This linkageprovides a very salient way to quickly check whether a colored block selectedfrom one side is contiguous (meaning that there is agreement between them), ordispersed on the other side (showing conflict for that selected subtree). The missingtaxa are explicitly shown, and are integrated with the linked highlight color. This32Figure 2.9: Pairwise comparison between the reference tree (left) and a user-specified target tree from tree collection (right).33feature was added in a late iteration in the design process, after we realized theimportance of providing awareness of specific missing taxa. The ability to inspecttaxa membership from the tree distribution, as described in the previous section, issupported on both sides of the butterfly view.Users can also select a subset of trees and create a consensus tree out of it (withextended majority rule [20]) to pairwise compare with the reference tree. Thisfeature provides a convenient way to quickly get a sense of what the subset of treesroughly looks like without diving into each tree one by one.2.6.5 Auxiliary viewsThere are three auxiliary views on the far right.The Tree List view at the top shows a list of names of trees with linked highlightingfor the tree selection. Sometimes trees are named according to conventions thatreveal useful information such as the data source, method, parameters to generatethe tree, and gene identification numbers.The Tree Similarity view in the middle encodes relative distances between pairsof trees to indicate potential cluster structure, where each dot represents a tree.This view is inspired by many papers in the category Many as Points. Distancesbetween dots correspond to distances between trees as computed by the t-SNEdimensionality reduction technique [172]. The default distance metric is Robinson-Foulds to show global relationships, or the user can choose a the inverse Jaccardmeasure that provides local relationships with respect to a chosen subtree. Thesemeasures are discussed further in Section 2.7.1.The Corresponding Branch Attribute view is aimed at the task of that name (T7)and shows the distribution of attribute values of corresponding branches for a se-lected focal branch / subtree.342.6.6 View coordination and visual encodingVisual indication of the coordination between multiple views is a complex andconstrained problem in ADView because of the many levels of detail we support.Supplemental Table A.1 provides a detailed breakdown of the most basic level ofview coordination: what aspect of data is visually encoded (or not) across all levelsof detail and all views. The size of visual elements varies enormously between theviews: the region subtended by a single tree ranges from a small point in the TreeSimilarity view to a medium-sized cell in the AD view; subtree blocks can be largein the Reference Tree view but small in the AD view.This variability poses a difficult challenge for color coding because distinguishabil-ity depends on size. Nevertheless, given the power of color for showing categoriesand the desirability of color consistency across elements between views [145],we carefully designed a color palette in consideration of these tradeoffs. Thefive medium-saturation colors for the subtree highlight blocks are designed tobe roughly equal luminance so that no focal subtree seems more important thanthe others, sufficiently salient for large regions in the Reference Dendrogram viewand small regions in the AD view, against both the white default backgrounds andbrown selected backgrounds. There is adequate luminance contrast for the blacktext label legibility, while maintaining distinguishability for the small highlightedpink branches that could occur within these blocks or against a white background.We off-loaded some selection indicators to other visual channels, using additionalpoint marks to show taxon selection in the Reference Tree, and border outlinesshowing tree selection for both points in the Tree Similarity view and cells in theAD views. Moreover, shape coding and line type coding is already used extensivelyin the AD design itself.Figure 2.10 shows a closeup of the two legends that appear at the bottom. Tosummarize, linked highlighting is encoded in several ways: five medium-saturationblock colors for subtrees (blue, orange, green, red, purple), both background color(brown) and border outlines (black) to indicate two forms of tree selection, linecolor for branches and region color within attribute range histograms (pink), and35Figure 2.10: Legends summarize the visual encodings. (a) Reference tree.(b) Aggregated dendrograms.additional point marks (black) for taxa.2.7 AlgorithmsWe provide an overview of the ADView architecture, then present specific algo-rithms in detail.We use a browser-server architecture for ADView. The user can submit a datasetusing the Upload page in browser, as shown in Supplemental Figure A.26 and Fig-ure A.27, which sends the raw tree files (in Newick format [1]) to the server. APython asynchronous pre-process worker in the server is then invoked as shown inSupplemental Figure A.28. It parses all trees, computes distances between all treepairs and corresponding branches between each branch in the reference tree andeach tree in the tree collection, stores them into a MongoDB database, and notifiesthe user upon completion. When a client session begins, the browser fetches datafrom a lightweight RESTful API, performs dimensionality reduction to the dis-tance matrix using t-SNE [172], presents the data, and responds to user interactionsuch as updating the ADs when a new branch of interest is selected. We use Re-act+Redux to manage state of data model, view, and user interaction in the browser,and we render almost all visual elements in SVG manually with some help fromD3. The implementation is available as open source under an MIT license from36https://github.com/zipengliu/adview.2.7.1 Corresponding branchesFinding comparable counterparts between the trees, namely corresponding branches,is done in the server when the user uploads a new dataset. The correspondingbranch is the most similar branch in each tree of the tree collection to a specificbranch in the reference tree, analogous to the best corresponding node (BCN) pro-posed in TreeJuxtaposer [126]. An example is shown in Figure 2.2(b).We define the similarity between a branch b in reference tree R and a branch b′ ina tree T of tree collection as the Jaccard index between two leaf nodes sets beneaththe two branches, excluding missing leaves:|b.taxa−T.missing|∩ |b′.taxa−R.missing||b.taxa−T.missing|∪ |b′.taxa−R.missing|where b.taxa and b′.taxa are the leaf node sets under b and b′, R.missing andT.missing are the respective missing leaf node sets. Missing taxa are excludedfor a less biased comparison, following general practice in phylogenetics research.Failure to exclude would lead to a record of both conflicting and missing signals,in contrast to our goal which is to focus only on the conflicting signals.This measure, which captures set memberships, is an intuitive extension to the stan-dard biological convention of Robinson-Foulds distance metrics for trees [150].This set-based measure is simple to compute and is a good match with our taskabstraction and our visual encoding and interaction design choices. We note thatBremm et al. [19] propose a more complex similarity metric that captures bothtopological structure and set membership, which they use to create histograms thatconvey aggregate information about mid-level structure; however, the details aboutwhat is different cannot be seen with this approach because they are aggregatedaway. Our approach communicates a great deal of detailed mid-level topologicalstructure through the ADs, so we only rely on the similarity metric for a first stage37of exploring whether trees agree on what taxa are in a clade.We need to compute the similarity score of every branch pair between the referencetree and tree collection, as shown in algorithm 1. We use a bottom-up approach forthe GetSimilarity() function. Supposing A= b.taxa−T.missing and B= b′.taxa−R.missing, the algorithm bottleneck is to compute |A∩B| when A is fixed. The leafset of an internal node is comprised of the leaf sets of its children. We can thuscompute |A∩B| by summing |A∩B1| and |A∩B2|, where B1 and B2 are the leafsets of B’s left and right children excluding R.missing.Algorithm 1: Corresponding branches to every reference-tree branchInput : reference tree R, tree collection COutput: all corresponding branches corr1 for b ∈ R do2 for T ∈C do3 targetSet← b.taxa−T.missing4 for b′ ∈ T in post order do5 similarity← GetSimilarity(b,R,b′,T )6 if similarity is better then7 corr[b][T ]← b′8 Function GetSimilarity(b, R, b’, T)9 if b’ is a leaf then10 // b′.card: cardinality of |b′.taxa−R.missing|12 b′.card = 0 if b′.taxon ∈ R.missing else 114 b′.intersect = 1 if b′.taxon ∈ targetSet else 015 else16 b′.card = b′.le f t.card+b′.right.card17 b′.intersect = b′.le f t.intersect+b′.right.intersect18 union← |targetSet|+b′.card−b′.intersect19 return b′.intersect/unionAlgorithm analysis: algorithm 1 computes similarity of all branch pairs betweenthe reference tree and tree collection in O(N2 ∗ |C|) time, where N is the numberof leaf nodes in a tree and |C| is the number of trees in the collection. Function38GetSimilarity() is amortized O(1) as the checking statement in line 12 and 14 uti-lizes a hash table with random hash functions. The O(1) similarity calculation is asubstantial improvement compared to the O(logN) TreeJuxtaposer [126] approach.2.7.2 Aggregated dendrogram layoutEvery time a subtree of interest in the reference tree is selected or removed, we up-date the layout of all aggregated dendrograms, which is handled on-the-fly by thebrowser. Given a tree data object, a set of focal subtrees, and the AD parameters,we use a best-effort mechanism to generate an AD layout that adapts to the spec-ified size. The user-specifiable parameters exposed in the interface are the heightand width of the AD, the number of context levels, and whether to apply labelsand colors. Several more algorithm parameters exist, and are illustrated in Supple-mental Figure A.4. The block color is determined at a later step in our renderingpipeline; the layout algorithm described here only computes the position and sizesof blocks and branches.Algorithm 2: Best-effort aggregated dendrogram layoutInput : tree, focalSubtrees, paramsOutput: AD layout or failure1 AD← generateLayout(tree, focalSubtrees, params)2 while not isLegible(AD) do3 params← shrinkParams(params)4 if not params then5 return failureGlyph6 else7 AD← generateLayout(tree, focalSubtrees, params)8 return ADAs shown in algorithm 2, we keep generating tentative layouts with different pa-rameter settings until the latest one passes a legibility test or parameter choicesare exhausted. The function generateLayout() compiles a list of blocks and thenlinks them together with either normal or elided branches by a bottom-up traver-39sal. The coordinates of blocks and branches are then calculated based on the blockhierarchy and relevant AD parameters such as total width and height. The functionisLegible() takes a layout and determines whether the blocks and branches will bevisible if drawn. The hard target is that focal subtrees need enough room to becolored; the soft target is to have enough room for labels as well, as a redundantencoding if space permits. A set of parameters will be “downgraded” from moresprawling to more compact by the function shrinkParams() every time the legi-bility test fails until the lower limit of all parameters is reached. There are twocascading sets of flexible parameters that can be changed: 1) the number of con-text levels, a metric to control how many context blocks to show; 2) inter-blockgaps, branch lengths, and block sizes. The function shrinkParams() first shrinksthe first set until lower limit is hit, then starts to shrink the second set. If legibilityis not achieved, the algorithm returns an error, which causes the interface to showa failure marker to notify users that there is no hope of fitting an AD in such space,indicating that the size parameters should be turned up manually using the UI slid-ers to override the automatic size-control algorithm one layer above this layoutalgorithm.One example of a failure case would be rendering 5 selected nested sub-trees in30x30 pixels. The user could decide whether it would be more appropriate toselect fewer subtrees (for example, if some uncleared selections were left over fromprevious analyses rather than being crucial for the current investigation), or simplyto allocate more space to each AD and accept the tradeoff that all of the individualADs do not fit within a single screen. While our automatic algorithms do quite abit of work on behalf of the user, our strategy is a mixed initiative approach thatrelies on the judgment of the expert user.Algorithm analysis: Once the subtrees are selected by a user, the layout of eachaggregated dendrogram is computed independently. The number of trials per ADdepends on both the characteristics of the input tree itself, the amount of con-text to show, and other size related parameters. Our empirical observation is thatwith the current settings of ADView this number is less than 10 and usually just1 or 2. The computational time of this algorithm is hence that of the function40generateLayout(), which is linear to the number of nodes in a tree. We also limitthe number of active colored subtrees of user selection to five, and therefore thelayout only needs to compute the position of about (colored and non-colored) 20blocks at maximum.2.7.3 Cluster aggregated dendrogramAlgorithm 3: Cluster topologically identical ADsInput : AD layoutsOutput: Cluster ADs1 strings← serialize(AD layouts)2 Cluster ADs← group(sort(strings))3 Cluster ADs← sortByFrequency(Cluster ADs)After generating AD layouts for all trees, we cluster them into groups of sharedtopological relationships among focal subtrees, as shown in algorithm 3. The se-rialization of AD computes a string that looks very similar to the Newick file for-mat [1], which maps topologically identical AD layouts to identical strings. Theuser can choose to further distinguish exact and inexact matches, and sister-groupor separate relationships. For example, the AD in Figure 2.3 is serialized into astring “(A,(C)B))”. If exactness and sister-group relationship is considered, it be-comes “(A−,(C+)B+)/”, where + and − stand for exact and inexact matches,and / and | for non-sisters and sisters. We only consider focal subtrees and ignoreeverything else. After serialization, the clustering problem is transformed into atrivial one: grouping the same strings together.Algorithm analysis: Because serialization only requires a traversal of AD blocks,and the number of blocks is usually very small (less than a dozen), the time com-plexity of algorithm 3 relies on sorting short strings. It is thus O(n logn), where nis the number of ADs.412.8 ResultsWe present empirical benchmarks for the computational performance of the algo-rithms, an informal comparison focusing on information density to the Bremm etal. [19] system, and a usage scenario illustrating several typical workflow steps forADView. We briefly report on formative feedback from ten domain experts duringthe iterative design process, and then present a summative expert interview study onthe utility of ADView including two case studies. The Supplemental Section A.5and Section A.6 includes multiple full screenshots of the first usage scenario, asecond usage scenario with over 1300 gene trees compared to a single species tree.There is also a video to show the look and feel of the system in action.2.8.1 Computational performanceWe deployed the server programs and database to a small server with two 2.20GHz Intel Xeon E5-2650 CPUs and 2G RAM hosted by our university, and runthe client on a Chrome browser on a 2012 Macbook Pro with a 2.50 GHz Intel i5CPU and 8G RAM. For a dataset with 500 trees and about 100 leaf nodes per tree,created by sampling from the public 1KP dataset (as discussed in Section 2.8.3), ittakes roughly 50 seconds to pre-process the data on initial upload and 10 secondsto load in browser at startup. The size of the compressed dataset is about 2MB.The user interaction in the browser is fluid when the number of trees is less thanroughly 500. There is noticeable delay of about 1 to 2 seconds for datasets beyondthis scale when all individual aggregated dendrograms are displayed, but imme-diate responsiveness can be restored by collapsing that pane. We used a cachingtechnique to accelerate the rendering of AD in browser, which is detailed in Sup-plemental Section A.3.2.422.8.2 Information density comparisonWe directly compare ADView to its closest competitor, the tool developed byBremm et al. [19], using their dataset of 33 trees and 32 taxa. An initial impres-sion from ADView is that the trees seem very different from each other: very fewbranches in the reference tree are supported by majority. In the rest of this compar-ison we focus on the visual encoding and interaction scalability.Supplemental Figure A.19 shows their system after adding 12 trees to the rightpanel and selecting one subtree of the reference tree, which is highlighted in pink.All matched nodes are highlighted across the trees, but their system only allowsone focal subtree at a time. Without any manual filtering, many irrelevant parts arevisible and the layout is visually crowded. Supplemental Fig.S21 shows the viewafter we filter out elements with similarity scores below 0.5. Substantial space isrequired for layout because the entire path between the internal nodes and the rootis preserved.Supplemental Figure A.20 shows ADView with the same dataset after we selectthree subtrees in the reference tree. (The paper’s materials do not include enoughinformation to replicate the selected groups, so we randomly chose subtrees tohighlight.) The cluster ADs show the major topological relationships between theblue, orange and green clades (task T4), and the Tree Distribution view shows theagreement and conflicts on each of the selected clades in terms of taxa member-ships (T5). These two views in ADView provide topological information in a morespace-efficient way than the full dendrograms in Figure A.19, and with far moredetail than can be gleaned from their heatmap and histogram views based on simi-larity scores. If the similarity scores are of direct interest, in ADView the user canuse sorting by similarity, see distributions in the histograms, or hover on a specificbranch in the Pairwise Comparison view.From the screenshots, we can conclude two major advantages of ADView overtheir system. First, one of the important tasks, topological relationships betweensubtrees (T4), cannot be supported by the color coding and filtering mechanism ofthe similarity score they propose, while the ADs in ADView are more suitable for43this task. However, a usability disadvantage of our system is that due to the broaderscope of task complexity, it is more difficult to learn. Second, ADView can scale tomuch larger number of trees due to higher spatial efficiency of tree representation.We also note that they do not show taxa names on their interface, which is anessential piece of information for our biologists to interpret the display. However,for datasets such as this one with a small number of trees and small tree sizes, theirsystem has the advantage of showing selectable taxa in all of the small-multipletree views, which fit on a single screen side by side.2.8.3 Usage scenario 1: 1KP pilot studyThe 1000 Plants Consortium (1KP) gathered phylogenetic data for thousands ofplants and published a pilot study about the origin and early diversification of landplants [183]. One of the main research questions is to determine the sister group ofLAND PLANTS (LP for short).The authors picked 103 representative plant species of major groups in STREP-TOPHYTES, and produced 69 species trees by performing 69 different analyses;that is, 69 runs of the computation pipeline, spanning different statistical methodsand parameter settings, different filtering criteria, and different kinds of sequences.They compared these trees with previous hypotheses to the research questions.We took their published data and checked whether their research questions could beanswered quickly with ADView. This usage scenario was created with the domainexpert who is a co-author on this chapter, who has close ties to the researchers whowrote the original paper. We report on one of these questions below, and presenta second one in Supplemental Section A.6. We randomly picked one tree as thereference tree, using the other 68 as the tree collection. We also confirmed that thesame results were reproduced by picking another reference tree, where all but 1 ofthe 68 aggregated dendrograms remained very similar.Correspondence of cluster ADs to known hypotheses: the previous hypothe-ses said land plants (LP) might be the sister of ZYGNEMATOPHYCEAE (ZYGN),44Figure 2.11: Annotated screenshots for a research question in the 1KP pilotstudy: which is the sister group of LP? (a) Previous hypotheses (ex-cerpt from their paper [183]) and the cluster ADs in ADView; arrowsshow hypotheses the cluster ADs support. (b) Consensus tree of clus-ter C3 pairwise compared to the reference. (c) Branch support valuesof trees supporting the LP+CHAR hypothesis.45CHARALES (CHAR) or COLEOCHAETALES (COL), as illustrated in Figure 2.11(a).We selected and highlighted these four clades in the reference tree as our focus.ADView generated the ADs and clustered them into five groups (without consid-ering exactness but with sister-group relationships), which can be regarded as fivehypotheses that this dataset supports. We quickly found out that clusters C1 andC4 support the ZYGN-sister hypothesis, clusters C2 and C5 support CHAR-sister,but cluster C3 does not agree with any of the three (task T4).Support values in C1 + C4 (ZYGN-sister trees): when we checked the distri-bution of support values (T7) by selecting the trees of C1 and C4 and created asub-collection, as shown in Supplemental Figure A.7, there are trees with low sup-port values. We can interpret this display as a counter-evidence to the ZYGN-sisterhypothesis.Conflicts in ZYGN: inspecting the Tree Distribution view in Figure 2.7, we sawthat all 68 trees plus the reference tree agree that LP, CHAR and COL are groupedtogether (monophyletic). However, a small portion of trees disagree with the ref-erence tree about ZYGN, as shown in the hovered-over segment (T5). To investi-gate what triggered the disagreement (T6), we selected those trees by clicking onthat segment, made a consensus tree out of them, and pairwise compared with thereference tree (T3), as shown in Figure 2.11(b). It turned out that a species, SPIR-OGYRA, was not included in ZYGN for these trees; we pairwise compared witheach selected tree to double check (T3). The authors of the pilot study confirmedthat there are complicated underlying issues for this misplacement.Details in C2 + C5 (CHAR-sister trees): in Figure 2.11(c), we further dug intothe support for CHAR-sister hypothesis by combining LP and CHAR into groupZ to see whether trees contain it (T1). Since Z is not a complete subtree in thereference tree we could not directly select it, so we used the popup menu thatsupports selecting an arbitrary set of user-specified taxa to choose the combinationof LP and CHAR. . There were 23 exact-match trees, which is exactly the numberof trees for cluster C2 plus that of cluster C5 (T4). In the Corresponding BranchAttribute view, we checked the black foreground bars in the histogram of supportvalues for corresponding branches of these 23 trees (T7). The circled bars showed46that some trees have very low support for LP+CHAR (biologists consider supportless than 0.5 very low). The Tree List view showed that these trees are mostlygenerated from a kind of unfiltered sequence data that has been called into questionin the domain literature.The workflow description above covers some interesting and representative stepsextracted from the full session, in which further visual analysis was conductedto check the supporting and conflicting evidence for additional hypotheses. Bythe end of the full session, we had found substantial evidence to support the pilotstudy’s conclusion that ZYGN is probably the sister-group of LP, although thereseems to be counter-evidence that merits further inspection.2.8.4 Formative expert feedbackWe obtained informal formative feedback during the iterative design phase fromthe four phylogenetic researchers who worked with us throughout the project. Ata quite mature stage of development we also presented it to six other phylogeneticresearchers via chauffeured demos. Overall we received positive feedback con-firming that our task and data abstractions matched with their biological intuitions,and that the functionality provided by ADView would be helpful to them. We alsogathered suggestions for improvement, many of which informed further iterationof the design.2.8.5 Summative expert interview studyWe conducted an expert interview study with five domain experts; our observationsand their feedback served to confirm the validity of our task abstraction and inter-face design. We describe the participants and procedure, present two case studieshere (with an additonal one in Supplemental Section A.8), and discuss their feed-back.47ParticipantsWe recruited five participants who are actively involved in phylogenetic researchand conducted four study sessions of 90 minutes each. In this chapter, we referthem as P1, P2, P3, P4a and P4b (the latter two were in the same shared session).Four of them come from different labs at our own university, and one from over-seas. Four were researchers focusing on biological questions with different kindsof organisms, three of them PhD students and one principal investigator; one wasan experienced bioinformatician whose daily job is to manage, process, analyzephylogenetic data of his lab. Two had seen a previous version of ADView demon-strated to them a few months before the study took place; one had seen screenshotsaccompanied by a general description. Participants P4a and P4b did not have rel-evant research datasets of their own and used those from the usage scenarios; thethree others used their own data. Supplemental Table A.2 provides detailed partic-ipant characteristics.ProcedureWe requested they send us one or two of their own research datasets a few days be-fore the session and added them to ADView ourselves to ensure they were ready forimmediate use; in some cases we undertook minimal pre-processing effort such asmapping machine-friendly sequence labels into meaningful human-readable names.In the session, we first gave a 20-minute demonstration of the two usage scenariodatasets to illustrate the system’s functionality, and also explained the exact detailsof how their individual datasets were pre-processed. Next, the participants usedADView on their prepared datasets for roughly 50 minutes, where they were askedto think aloud while working. Our primary goal with this study was to assess theutility of the system rather than to improve the software usability, so we encour-aged them to immediately ask us any questions about the meaning of the visualencodings and how to perform certain operations, stepped in if we saw a clear mis-use of the tool or implementation bugs, discussed possible follow-up actions if theywere stuck, and asked questions if we were not able to understand what the partic-48Figure 2.12: Detail views of P1’s findings. (a) Reference tree with four fo-cal subtrees selected to investigate the position of T10 (blue, label A)and a branch highlighted in pink (circled). (b) Support distribution his-togram with very low values selected, driving the pink highlight in thereference view. (c) Cluster ADs showing the top-frequency clusters.ipants were doing. In the final 20 minutes we asked a set of open-ended questions,documented in Supplemental Section A.4.2. We recorded audio and video for theentire session, and took notes manually.Case study 1 findingsP1 compared a species tree against 260 gene trees of plants. We obtained permis-sion to show this dataset with anonymized organism names to protect unpublisheddata.Sanity check on support values: upon loading the dataset, she noticed brancheswith extremely low support values according to the reference tree attribute distribu-tion, shown in Figure 2.12(b). She selected the leftmost bar, which highlighted onebranch in thick pink in the reference tree (T2), shown in Figure 2.12(a). Once shehovered on that branch, which contains all in-group taxa, the branch detail panel49Figure 2.13: Cropped figures of P2’s findings: (a) The selected subtrees ofinterest (blue- BLASTOCYSTIS STRAIN 1, orange - BLASTOCYSTISHOMINIS, green - BLASTOCYSTIS STRAIN 5); (b) different positionsof orange taxon across bootstrapped trees.shows that both its support and length are zero. She clicked on this branch, and theTree Distribution view showed that all trees agree with the reference tree, whichseemed to conflict with the zero support value (T5). After checking a few genetrees in detail using the Pairwise Comparison view (T3), she figured out that thezero-support in-group branch was generated by the rooting process, and dismissedit as not relevant to her research questions.Position of T10 She then highlighted branches with relatively low support values(0.8 to 0.9) in the same fashion (T2), and discovered a specific taxon T10 wasplaced differently across the gene trees. To understand the alternative positions ofT10 within a bigger monophyletic group, she broke it down into several subtrees,visible as the blue, orange, green and red groups (T4) in Figure 2.12(a). The first 4of the 56 cluster ADs are shown in Figure 2.12(c). The majority of the remainingclusters contained only one AD, which suggested the relationships between thefocal subtrees were badly disputed. To investigate the reasons for the disagreement,she looked at the consensus trees of the clusters and pairwise compared severalindividual ADs to each other.50Case study 2 findingsP2 compared a consensus tree (used for the reference tree) against 100 bootstrappedtrees of parasites, and approved our usage of this dataset with unsanitized labels.Sanity check on known strains: to familiarize himself with the interface and in-teraction, he selected a few monophyletic groups of different parasite strains (T1).He was able to quickly confirm his knowledge of which two strains are closer thanothers, and the general branching pattern of strains 1 to 5, by selecting them in thereference tree and then checking the cluster ADs (T4).Unknown outlier (orange): he identified an interesting species, BLASTOCYSTISHOMINIS, shown as the orange group in Figure 2.13(b). He knew that it usuallylives in pigs, but he saw that here it was placed within the group mostly related tohumans (T4). He knew that the orange group branched with the green one becausehe had seen it in the reference consensus tree on the left, but he had not realized thatthere is uncertainty about placement of the orange group until he saw the clusterADs. Figure 2.13(b) shows that only 40 trees out of 100 have the same branchingpattern as the reference tree, whereas there are 29 trees suggesting that the orangeis closer to the blue and that the green branched before the other two. He thenconstructed a sub-collection of each cluster and pairwise compared some trees oneby one within each sub-collection against the consensus tree for more detail (T3).FeedbackThere was general agreement from all participants that many aspects of the designwere successful. The task and data abstractions matched their biological intuitions,as did the results of the corresponding branches algorithm. The overall AD designshowed many trees in a way that was considered quite intuitive, despite being acompletely new visual representation to all participants. They all could see them-selves using ADView in upcoming research projects to analyze the trees generatedby a reconstruction pipeline.51Participants also noted limitations of ADView. Most thought it was complicatedto learn, and they would need more experience using it to fully exploit its power.One minor point of confusion came at startup time when no subtrees are selected,where the ADs consist of only a single blank rectangle. While most participantsagreed that the Tree Distribution view was useful for exploring agreement andconflicts of a selected subtree, it was sufficiently unusual that two participants didnot fully understand its meaning until a discussion in the question period at theend. One participant observed its connection to an existing visualization that showsthe distribution of alternative taxa groupings with a small pie chart next to a treebranch [159, Fig. 2]; she noted that we had “unrolled” a pie into a horizontal bar.Although the main goal of this study was not usability testing, we did make designrefinements after each session to address awkward interactions. Most were minorinteraction improvements, but a notable late-stage addition was to automaticallycompute appropriate sizes for the ADs and pass that request to the layout algorithm,rather then rely on the user to explicitly set sizes through a manual slider.2.9 DiscussionADView can be used by researchers to speed up the overall process of phyloge-netic analysis at multiple stages in a project, especially exploratory analysis rightafter obtaining the trees and also to generate figures to communicate findings nearthe end of projects. We envision a workflow where researchers would typicallyfollow up their initial explorations with more rigorous statistical tools. It is dif-ficult to document the amount of speedup quantitatively, since biology papers donot typically provide any explicit estimates of how long their original analysis pro-cess required, as with our usage scenario source [183]. However, our biologistco-author has extensive experience in analyzing this kind of data. His assessmentis that the analyses in the source paper require multiple dedicated R scripts, whichcould take a few days or even weeks depending on the programming expertise ofthe biologist. With ADView, it takes only a few hours or less to conduct similaranalyses. ADView provides substantial speedups in assessing the quality of data,52process debugging and other sanity checking, identifying interesting areas, gather-ing initial evidence, and generating hypotheses.ADView can handle datasets of hundreds of trees with about 100 leaf nodes per treefrom an algorithmic point of view. The de facto limit on the number of leaf nodes isprimarily due to the design target of fitting the detailed reference tree dendrogramon a single screen without scrolling. There is no hard limit on the number of treesfor our proposed method, but our current implementation on the browser can onlydeal with roughly 500 trees at interactive frame rates before the number of SVGelements would slow down the rendering process. However, for larger datasets, theuser can filter out many trees and analyze smaller subsets of dozens to hundredsof trees, or simply close the Individual AD view and achieve fast response time byrelying solely on the Cluster AD view. Addressing these algorithmic bottlenecksin future work would allow studies to determine the perceptual scalability limits ofour approach.2.10 Summary and future workIn this chapter, we provide task and data abstractions for the problem of one-to-many tree comparison in the domain of phylogeny. We propose a new technique,the aggregated dendrogram, to summarize the topological relationships betweenfocal subtrees via a simple, intuitive, and effective representation. We providean algorithm that adapts to available screen space. We present the design andimplemention of ADView, which provides insight into a tree collection of hundredsof trees at five levels of detail: the full collection, a subset of trees, single tree,subtree, and individual tree branches. We validate our approach with empiricalbenchmarks, direct comparison to the most similar previous system [19], usagescenarios, and an interview study conducted with five domain experts includingcase studies with their findings from using the system.Extending ADView to handle duplicate leaf nodes, which arise from common evo-lutionary events such as gene duplication, would be an excellent and challeng-ing direction for future work. It would also be interesting to see what aspects of53ADView are useful in general tree comparison problems outside of the phylogenydomain.54Chapter 3Sprawlter: Combining Sprawland Area-aware Clutter forGraph Readability Metric1 Graph drawing readability metrics are routinely used to assess and create node-link layouts of network data. Existing readability metrics fall short in three ways.The many count-based metrics such as edge-edge or node-edge crossings simplyprovide integer counts, missing the opportunity to quantify the amount of overlapbetween items, which may vary in size, at a more fine-grained level. Current met-rics focus solely on single-level topological structure, ignoring the possibility ofmulti-level structure such as large and thus highly salient metanodes. Most currentmetrics focus on the measurement of clutter in the form of crossings and overlaps,and do not take into account the trade-off between the clutter and the informationsparsity of the drawing, which we refer to as sprawl. We propose an area-awareapproach to clutter metrics that tracks the extent of geometric overlaps between1This chapter is a slightly modified version of the following publication: Z. Liu, T. Itoh,J. Q. Dawson, and T. Munzner. The Sprawlter Graph Readability Metric: Combining Sprawland Area-aware Clutter. IEEE Transactions on Visualization and Computer Graphics (TVCG),26(6):2180–2191, 2020 [110].55node-node, node-edge, and edge-edge pairs in detail. It handles variable-size nodesand explicitly treats metanodes and leaf nodes uniformly. We call the combinationof a sprawl metric and an area-aware clutter metric a sprawlter metric. We presentan instantiation of the sprawlter metrics featuring a formal and thorough discussionof the crucial component, the penalty mapping function. We implement and vali-date our proposed metrics with extensive computational analysis of graph layouts,considering four layout algorithms and 56 layouts encompassing both real-worlddata and synthetic examples illustrating specific configurations of interest.3.1 MotivationComputing quality metrics is an important and popular quantitative approach toevaluate and generate visualizations [13], especially for node-link diagrams, anintuitive and effective visual representation for relationships between entities in agraph. Graph layout has been studied for many years, with major effort devotedto automatic generation of readable and faithful node-link graph layouts [56, 69].One of the important aspects in making good layouts is to have good metrics tomeasure the readability or faithfulness, which are used by graph layout algorithmseither explicitly or implicitly. In particular, we focus on readability metrics, alsohistorically known as aesthetic criteria. Many readability metrics have been pro-posed in previous papers, such as edge-edge crossings, node-edge crossings, edgebends, and angular resolution. Many empirical studies have been conducted toevaluate their correspondence with human judgements [32, 143]. We note threeproblems with many previous readability metrics.First, they report simple integer counts of detrimental events such as node-edgecrossings. Although counts are straightforward to understand and to compute, theylack the precision to capture how badly an event hinders readability, such as howmuch an edge crosses a node as measured in length, or whether two edges crosswith a glancing angle.Second, they only compute on single-level structure, even in a multi-level graphlayout, which are used in many domains [117], especially for large graphs. A multi-56level graph combines a cluster hierarchy with a base graph, where the originalgraph nodes comprise the lowest leaf level of the hierarchy, and metanodes (alsoknown as clusters or subgraphs) at higher levels contain lower-level nodes. Themetanodes are typically more salient than the leaf nodes because they are bigger insize, but this important multi-level structural information is not taken into accountwhen measuring readability by the previous metrics.Third, they are mainly clutter-only metrics that only penalize crossings and donot fully assess the layout quality. In addition to clutter, information density [125](also known as data-ink ratio [170] or space efficiency [118]) also matters.We use the term geometric sprawl, or sprawl for short, to capture disparity be-tween the size of small features and a potentially large canvas on which they arespread out. This measure is different from topological sparseness: sprawl consid-ers geometric layout information such as coordinate positions and area. There is animportant trade-off between clutter and sprawl: it is possible to eliminate all over-laps (especially for node-node and node-edge overlap) by simply either pushing thenodes away from each other or shrinking the node size until they are completelyseparated, but at the cost of tiny nodes and edges spread out on a huge canvas. Thequantification of sprawl itself is not new; it is the inverse (reciprocal) of metricsrecently called compactness by Kieffer et al. [92] and visualization coverage byDunne et al. [39]. The idea of minimizing the metric of total area is longstand-ing [36]. The novelty of our work lies in the direct capture of this readabilitytrade-off, and our emphasis that it constitutes the most central property of graphlegibility.We illustrate the three problems above in Figure 3.1. All three example layoutsshow two-level synthetic graphs containing four colored metanodes drawn as mini-mum bounding circles of their constituent leaf nodes. The count does not reflect thefact that layout (b) is more cluttered than (a) in terms of both leaf node overlaps andmetanode overlaps, as there are exactly six crossings in both layouts (the integercount problem). Nodes in layout (c) are shrunk and spread out on a larger canvasresulting in fewer overlaps, but the layout suffers from poor space efficiency [118]due to sprawl.57Figure 3.1: Comparison of approaches to measuring node-node crossing inthree synthetic 2-level graph layouts. (a) Some overlap. (b) More over-lap. (c) More sprawl. Our sprawlter metric (T ) is the geometric meanof the sprawl (S) and area-aware clutter metrics (A). Clutter metrics(A and C) are broken down into metanode vs. leaf node pairs. For allmetrics, larger numbers are worse, indicating less readability. The in-creased overlap between leaf nodes and the metanode pair is capturedby the area-aware metric (24.03 > 15.91). The increased sprawl whennodes are shrunk in size and pushed away from each other to avoidoverlap, resulting in a more sparse layout, is captured by the sprawl(37.50 > 8.45). Their mean (T) assesses the trade-off between clutterand sprawl in the geometric configurations.To address the first two problems, we contribute an area-aware approach for threefamilies of clutter metrics: node-node (NN), node-edge (NE), and edge-edge (EE)crossings. It measures the amount of geometric overlaps (NN, NE) or crossingangle (EE) in both single-level and multi-level graph layouts, then computes apenalty for each geometric measurement, and finally sums the individual penaltiesas a NN, NE, or EE clutter metric. We identify general requirements and make spe-cific choices for the crucial step of mapping geometric measurements to penalties.To address the third problem, we contribute the sprawlter metric to capture thetrade-off between sprawl and clutter. It computes the geometric mean of sprawland area-aware clutter for each of the NN, NE, and EE families. We implementthese new metrics, and validate their benefits by comparing quantitative computa-tional results of our approaches to traditional count-based approaches and recentproposals [39], with respect to qualitative assessments of layout pictures.583.2 Related workWe discuss related work on readability metrics and the evaluation of these metrics.3.2.1 Graph drawing readability metricsWe identify two categories of graph drawing metrics: single-purpose and com-pound metrics. The former only quantify one single feature of the layout, whilethe latter explicitly combine or implicitly reflect multiple single-purpose ones.Single-purpose metricsThe single-purpose metrics most relevant to our own work focus on how clearlythe drawn elements of a graph layout can be seen. There are many other typesof metrics that are less relevant, including those related to graph topology suchas average degree of nodes or modularity, those related to faithfulness such asshape-based metrics [45], and those measuring specific aspects of perception andcognition such as symmetry [44, 140], edge bends [143], and edge continuity [182].A recent paper uses many metrics to build a neural network to assess quality ofgraph layout, but does not focus on readability metrics [94].Well over a dozen single-purpose readability metrics have been proposed, withsome evaluated by controlled experiments with human subjects. Early work fromPurchase formally documented several metrics [141], with many subsequent exper-iments that compare these and other metrics to human judgements [32, 44, 182];Dunne and Shneiderman provide a survey of empirical studies on readability met-rics [38]. We categorize these metrics by the three problems we note above: clutter,multi-level, and sprawl.Clutter metrics. Clutter is an important and obvious readability obstacle. The wellknown clutter-related metrics are edge-edge, node-edge, and node-node crossings,which report discrete integer numbers of the crossing events. Some metrics re-59port continuous real numbers, such as angular resolution at nodes [141], whichmeasures the minimum angle of edges that are incident to an individual node, andangular resolution at edges, which measures the angles between crossing edges, orthe total resolution of Argyriou et al. [8] that combines them.The impact of crossing angles, namely the angle at which two edges cross, has beenheavily studied. Although Ware et al. did not find significant relevance betweenhuman performance and average crossing angles in their multi-factor experimentusing global graph layouts as stimuli [182], a later single-factor experiment usingsimple synthetic drawings conducted by Huang et al. showed that the crossingangles have a significant impact on response time [83]. Dunne et al. proposedanother version of edge crossing angle metric, defined as the average deviation ofthe crossing angles from an ideal angle (70◦) [39]. Their metric is 0 for the idealcrossing angle, which ignores the fact that there is still a detrimental crossing eventthat reduces readability (R2 in Section 3.4). Our own edge-edge metric differs withan explicit function that maps angles into penalties instead of using them directly,allowing us to clearly distinguish even ideal crossing angles from the non-crossingcase.Multi-level metrics. Some metrics measure geometric separability, namely howcompactly leaf nodes lie within or are spread between metanodes, including edgelength variability [130, 173], which is also applicable to single-level layouts. How-ever, they do not recognize salience of clutter in a multi-level structure. The manytopological separability metrics, such as modularity, conductance, coverage, per-formance, and so on [5, 47] do not measure geometric configurations. The only truegeometric metrics in the literature that address the multi-level problem are ambigu-ity measures for community structure from Wang et al. [179] and the group overlapmetric from Dunne et al. [39]. Wang’s approach focuses on the ambiguity causedby overlap of metanodes rather than readability, where they considered the geo-metric distribution of leaf nodes within metanodes rather than salience of clutter.It is later used in their follow-up papers to assess readability of clusters [70, 180].Dunne’s proposal for group overlap does compare overlaps between a metanodeand leaf nodes, but not between metanodes, which are more visually salient. Our60approach features a uniform treatment of nodes and metanodes to capture the clut-ter in both single-level and multi-level graph layouts.Sprawl metrics. Several simple sprawl-related metrics have been proposed, in-cluding total drawing area, the maximum edge length, sum of edge lengths, and theaspect ratio of the drawing canvas [36]. The multiplicative inverse of sprawl, themetric that we use, has previously been called compactness by Kieffer et al. [92]and visualization coverage by Dunne et al. [39]. We use the inverse quantity toemphasize the problematic nature of sparsity as analogous to clutter. Kieffer etal. found evidence of its importance in small user-generated graph layouts [92],and Dunne et al. proposed it as a stand-alone separate metric for NN. The noveltyof our work lies in exploring the trade-off between sprawl and clutter, not introduc-ing the sprawl metric itself.Compound metricsIndividual single-purpose metrics can be combined into a compound metric. Suchcompound metrics are often implicitly incorporated into automatic layout algo-rithms, as with physics models that measure a graph layout with an artificial physicsquantity, such as force (GEM [52]), energy (LinLog [130]), or stress (NEATO [53]),as if the nodes and edges are real-world physical objects. For example, force mod-els typically encourage uniform edge lengths and clustering around high-degreenodes, and discourage node-node overlaps. Some proposals have been made to addother types of forces to explicitly incorporate more single-purpose metrics [37, 84].A compound metric explicitly proposed in previous work is a weighted sum ofmultiple single-purpose metrics [85], but justifying the weights remains an openproblem. Our sprawlter metric, where sprawl and clutter is combined, is also anexample of an explicit compound metric. We emphasize the specific trade-off be-tween clutter and sprawl as directly opposite quantities, rather than attempting tocombine very disparate quantities of all possible metrics into a single compoundquantity.613.2.2 Evaluation of graph drawing metricsPreviously proposed metrics have been evaluated either with human subjects ormathematical models, with the goal of obtaining a deeper understanding of the nu-ances and ranking the metrics by perceptual or cognitive impact. One approachis to gather quantitative performance data or qualitative feedback on the stimuligraph layouts from human observers, where the stimuli can be generated by auto-matic computational algorithms or human manual labour. There are many papersfrom earlier years that followed this general approach. Typical examples are fromPurchase et al. [140, 142–144], Ware et al. [182], Huang et al. [82, 83], Kobourovet al. [95]. Another approach is to gather user-generated layouts in order to under-stand their mental model of graph layouts. Van Ham and Rogowitz asked partici-pants to position the nodes in a graph layout and investigated which metrics are use-ful for explaining their layouts [173]. Following this thread, Dwyer et al. compareduser-generated layouts with automatic layouts [44]. A third approach focuses onquantitative models of layouts and relevant tasks. McGuffin et al. evaluated spaceefficiency of visual representations of trees mathematically and provided designguidelines accordingly [118]. Dawson et al. proposed and evaluated a predictivemodel of human behaviour tracing paths through a graph [32]. None of these evalu-ations provide direct guidance on how to address the three problems that we tacklein this work.3.3 sprawlter metricsWe describe the computation of the sprawlter metrics in this section including thegeneral idea, technical details, and how they address the three problems presentedin the introduction. First, for each clutter event, the amount of overlap (area ofnode-node overlap, length of node-edge overlaps, and angle of edge-edge crossing)is measured. Then, the overlap measurement is transitioned into a penalty – anindication of the degree of clutter – with a penalty mapping function. We elaboratefurther on these penalty mapping functions in Section 3.4. Finally, these penaltiesfor each individual clutter event are summed up for node-node overlap (NN), node-62edge overlap (NE), and edge-edge crossing (EE) respectively as the area-awaremetrics. Meanwhile, sprawl is also computed based on geometric properties of theentire layout. The sprawlter metric is a geometric mean combining the sprawl andarea-aware metric.3.3.1 Graph layout definitionsWe define a graph G = (V,E), where V is the set of nodes and E is the set ofedges. The graph G can be either a single-level graph or a multi-level graph; inthe latter case, we simply expand the definition of the nodes v ∈V to be either leafnodes or metanodes. Since we treat both of these cases in a completely uniformway, our algorithm does not need to further consider the hierarchy H that is partof a compound graph, except for a low-level implementation detail (when avoidingcomputation of overlaps between a metanode and its descendants). We do notconsider metaedges, which are analogous to metanodes, but are not commonlyused in current layout algorithms.We denote the geometric measurements by functions of graph elements: the areaof a node area(v), overlapping area of two nodes area(v1,v2), length of an edgelength(e), diameter of a node length(v), and overlapping length of a node-edge pairlength(v,e). The crossing acute angle (in radians) of two edges is angle(e1,e2),and its complement is c angle(e1,e2) = pi/2−angle(e1,e2). When two graph el-ements just begin to touch, we consider them to intersect but with zero overlap:area(v1,v2) = 0, or length(v,e) = 0. When they do not intersect, the measurementfunction is undefined – it is not zero.3.3.2 Measuring geometric overlapWe measure the overlapping area of a node pair, overlapping length of a node-edge pair, and (the complement of) crossing angle of an edge pair if the nodes oredges intersect with each other. Figure 3.2 shows an increasing amount of overlapbetween pairs from left to right, from none, to near-minimum, some, and near-63maximum overlap. Obviously, the degree of clutter corresponds to the amount ofoverlap. In the area case, the degree of clutter corresponds positively to area(v1,v2)or length(v,e). We consider length to be the degenerate version of area, and belowoften use area to mean both quantities. In the angle case, the degree of clutter cor-responds positively to c angle(e1,e2), where a perpendicular angle of 90 degreesis the least cluttered and a glancing angle of 0 degree – where one line segment ison top of the other – is the most cluttered. We thus consider the complement of thecrossing angle rather than the crossing angle itself. In Figure 3.2, the count-basedmetrics are only able to distinguish between the leftmost column with no overlapand the other three columns, but not between those three.None Near-min Some Near-maxAreaLengthAngleFigure 3.2: Increasing amount of overlap between node-node, node-edge,and edge-edge pairs. Count-based metrics can only distinguish no over-lap from the other three cases, but not between them.3.3.3 Mapping measurements to penaltiesWe map the measured amount of overlap to penalties with a penalty mapping func-tion in order to match the measurement to the degree of perceived clutter. Gener-ally, we denote the penalty mapping functions by f (x), where x is the overlappingmeasurement. Specifically, for the three different families, we use f NN(x), f NE(x),and f EE(x). Note that for the EE family, x denotes the complementary angle sothat the valence of f EE(x) matches the others (since an original angle of 0 is theworst case).64We avoid the trivial instantiation of f (x) where geometric measurements are useddirectly as penalties (i.e. f (x) = x), because the measured area / length / angledoes not in general equal the degree of visual clutter. In particular, the distinctionbetween the no overlap and touching cases is crucial (first and second columns inFigure 3.2), but f (0) = 0 would fail. The penalty mapping function is discussedfurther in Section 3.4.3.3.4 Summing penaltiesWe directly sum up penalties of all overlapping pairs of node-node, node-edge,and edge-edge in a graph layout respectively, and refer to the three resulting totalpenalties as area-aware metrics, denoted by ANN , ANE , and AEE , while the count-based metrics are denoted by CNN , CNE , and CEE .These area-aware metrics address the integer-count problem with a more preciseindication of the degree of clutter. They also address the single-level problem forNN and NE, because the area awareness automatically captures the hierarchicalinformation in multi-level graph layouts. Metanodes increase in area and diameterat ascending levels of the hierarchy: higher-level metanodes are larger than theirconstituent metanodes at lower levels, and than their leaf nodes at the lowest level.Hence, summing area or length penalties is analogous to a weighted sum of binarycounts, where larger overlaps are weighted more heavily than smaller ones.We show the pseudocode to compute the node-node area-aware metric in algo-rithm 4; the similar analogs for NE and EE are in Supplemental Section B.2. Thecondition statement in line 1 checks for validity of the node pair, where the func-tion IsAncOrDesc(v1,v2) checks if one node is the ancestor / descendant of theother in the node hierarchy. Function CheckIntersection(v1,v2) returns true if v1and v2 intersect each other.65Algorithm 4: Computation of node-node area-aware metrics.Input : G = (V,E)Output: total penalty ANN(G) and count CNN(G)1 totalPenalty← 02 count← 03 for v1 ∈V do4 for v2 ∈V do5 if v1 6= v2 && !IsAncOrDesc(v1, v2) then6 if CheckIntersection(v1, v2) then7 x← ComputeOverlapArea(v1, v2)8 penalty← PenaltyFunc(x, v1, v2)9 totalPenalty← totalPenalty + penalty10 count← count + 111 return totalPenalty, count3.3.5 Combining clutter metrics with sprawlWhile clutter indicates overlaps and occlusions of nodes and edges, sprawl indi-cates geometric sparsity of graph elements on a drawing canvas, as illustrated inFigure 3.1. The relevant factors contributing to sprawl include total drawing area(typically the axis-aligned minimum bounding box of the entire layout), node ar-eas, edge lengths, and the number of nodes and edges. Spreading out nodes of afixed size on a larger canvas would increase the sprawl, for example.We quantify sprawl by dividing the total drawing area of a graph layout G bythe area occupied by all nodes (excluding overlaps). It is the reciprocal of thecompactness from Kieffer et al. [92] or the visualization coverage from Dunne etal. [39]. Formally:S(G) = area(G)/area(⋃v∈Vv) (3.1)S(G) denotes the sprawl, and area(G) the total drawing area of the graph layout66G. The area of union of all nodes captures the intuition behind sprawl well byaccounting for variation in node sizes. We also considered other alternatives. If weuse the number of nodes, |V |, as the denominator, it would not account for absolutenode sizes: for example, the sprawl of layout (b) and layout (c) in Figure 3.1 wouldbe the same. If we use the smallest node area, min{area(v)|∀v ∈V}, the resultingratio would not gracefully handle relative node size variation: for example, a graphwith 1000 large nodes and one tiny node would be over-penalized.There is a direct trade-off between clutter and sprawl: for a fixed node size, themore spread-out the nodes are, the less clutter there would be and the more sprawlthere would be. However, the interplay between clutter and sprawl is not strictlya zero-sum game: it is very possible to keep clutter low while having perceivablenode sizes (or edge lengths) and reasonable total drawing area. It is common tocombine multiple values by computing a mean of them. In mathematics, thereexists many different versions of means [21] for different purposes. We choose thegeometric mean (i.e. square root of the product of two positive values) because itis capable of normalizing different ranges of the constituent values: e.g. a 10%change in sprawl has the same effect as a 10% change in clutter. It is more suitablethan the commonly-used alternatives of the harmonic mean, arithmetic mean, andquadratic mean. The harmonic mean is usually dominated by the minimum of thetwo values; a typical usage is to combine precision and recall for machine learningmodels. The arithmetic mean requires that the values have identical range to avoidunfair averaging. The quadratic mean is often used to measure errors betweenestimations and ground truths.The sprawlter metric (denoted by T ) for NN is stated as follows, and those of NEand EE have the same form.T NN(G) =√S(G) ·max{ANN(G),1} (3.2)We impose a lower bound for the area-aware component: when the area-awaremetric, or generally, the clutter measurement, is very small, the sprawlter metricshould retain the sprawl-aware information; otherwise sprawl has little influence67on the sprawlter metric in such situations. In contrast, we do not need to imposeany bound for the sprawl as it is always greater than or equal to 1.We note that the general idea of sprawlter is completely independent of area-awareness: an instantiation could be designed for purely count-based metrics.3.4 Penalty mapping functionThe previous section provides an overview of the sprawlter metrics computation,which has the penalty mapping function at its core. We now discuss this crucialand challenging component in detail.The penalty mapping function, f (x), takes a geometric measurement of a singleoverlap event (area or length or complementary angle), and emits a number toindicate the degree of clutter for this event. In short, it is a mapping from overlapto clutter.3.4.1 General requirementsWe identify four important general requirements for the penalty mapping functions,illustrated in Figure 3.3.R1. Increasing penalty. ∀x1 < x2, f (x1) < f (x2). The penalty should increaseas the overlap increases. In Figure 3.3, f (x) for the area-aware approachincreases from a minimum to a maximum penalty, as opposed to a flat line forcount-based approach. This requirement is necessary to distinguish betweenminimum, some, and maximum degree of clutter.R2. Substantial touching penalty. f (0) 0. In order to clearly distinguish theno-overlap case from a very small overlap, we require a substantial minimumpenalty that is incurred as soon as two elements touch. That is, f (0) shouldbe substantially greater than zero, in contrast to f (undefined) that is exactly0.68x: geometric measurementf(x): penalty0minPenaltymaxPenaltymaxMeasurement (M)1for area-aware approachf (x) =? (1)for count-based approachf (x) = 1 (2)Figure 3.3: General requirements for area-aware penalty mapping function,vs. a constant function f (x) = 1 for the count-based approach.R3. Count calibration. minPenalty < 1 < maxPenalty. The sprawlter metricsare intended to be a drop-in replacement to count-based metrics but to pro-vide richer information, and thus the penalties need to bracket, and be di-rectly comparable to, the count-based value of 1. The touching case createsless clutter than what is captured by a simple count; the full overlap casedemands a higher penalty.R4. Local function. f NNv1,v2(x) instead of fNNG (x). We need to have a functionthat is calculated separately for each pair of nodes that can incorporate localknowledge of their individual sizes, rather than an identical global functionthat applies to all node pairs independently of their size. The EE family is aspecial case compared to the NN and NE one. Since all crossing angles sharea universal unit (radians) and are bounded within the range [0,pi/2], a localfunction is equivalent to a global one. In the notation, we use subscripts toindicate whether the function is local or global.In order to calibrate penalties to counts (R3), it is necessary to normalize the ab-solute node areas (and edge lengths) to relative areas. We divide the area of eachnode by the smallest node area in the layout such that the normalized area of thesmallest nodes is 1. If all nodes have the same size, the normalized area of everynode is 1; if node sizes vary, the normalized areas of larger nodes are greater than 1.69Similarly, for lengths, we divide them by the diameter of the smallest node. Thereis no need to normalize angles as they are all in radians. We use normalized areasand lengths below, unless otherwise specified.3.4.2 Specific choicesWe make specific choices to fulfill the general requirements of the penalty mappingfunction, while there exist other alternatives. Our goal is to construct a specificfunction that is simple and easy to understand. Figure 3.4 shows the NN penaltymapping function for the smallest node pair.C1. We choose a power function: f (x) is linear to xγ , where γ denotes the expo-nent in the power function and dictates the curve shape of f (x). In Figure 3.4,we show the curves for three different γ values. We chose γ based on previ-ous empirical work [83, 162]. The longstanding psychophysical power lawproposed by Stevens [162], which is a mapping between physical stimulusand perceived sensation, inspired our choice of γ = 0.7 for node-node over-lap areas and γ = 1.0 for node-edge overlap lengths. More recently, Huanget al. [83] suggest that human performance is correlated to the crossing an-gle with a quadratic term in their preliminary user evaluation, so we use aquadratic function (γ = 2.0) for edge-edge crossing angles. To check theimpact of this choice, we conducted theoretical and computational analysiscomparing a linear EE penalty mapping function and a quadratic one. Wefound that difference between the two is very small except when there aremany glancing crossing angles. The details of this comparison are in Sup-plemental Section B.1.2.C2. We choose a minimum penalty proportional to the node sizes, specificallya fraction of the maximum possible clutter measurement M that would beincurred if they completely overlapped: f (0) = αMγ , where α denotes thefraction, and 0 < α < 1. For example, for the smallest node pair v1,v2,the maximum overlap Mv1,v2 = 1, and thus 0 < fNNv1,v2(0) = α < 1, which iscompatible with the requirements of a substantial touching penalty (R2) and70calibration to count (R3). In a layout with variable node sizes, M wouldbe different for node pairs of different sizes, resulting in a larger touchingpenalty for large pairs and a smaller touching penalty for small pairs, so thelocal function requirement is also met (R4). The touching penalty could begreater than one for sufficiently large nodes. Similarly, the same amount ofoverlap area or length (x) will have a bigger impact ( f (x)) if it happens onbigger nodes than smaller nodes.C3. We choose a maximum penalty proportional to the node sizes, with M asabove: f (M) = βMγ , where β denotes the proportion, and β > 1. For thesame example as above where M = 1, R3 and R4 compatibility also holds,and f (M) = β > 1. Meanwhile, as β > 1>α , the maximum penalty is guar-anteed to be greater than the minimum penalty; combined with the choice ofa power function (C1), f (x) is an increasing penalty (R1).C4. We choose to set the halfway point of measured clutter for the smallest pairto a penalty of 1: f NNv1,v2(0.5) = 1, fNEv,e (0.5) = 1, fEEG (pi/4) = 1; where v1,v2is the smallest node pair so Mv1,v2 = min{area(v1),area(v2)} = 1, and v,eis the smallest node-edge pair so Mv,e = 1. Note that the flat line for count-based metrics lies exactly in the middle of minimum and maximum penaltywhen γ = 1, as shown in the left function in Figure 3.4, but this statementdoes not hold when γ 6= 1. The choice of halfway point is a specific wayof calibrating penalties to counts (R3). It also constrains the relationshipbetween α and β so that only one of them needs to be specified beforehand,improving the simplicity of the function.3.4.3 Formal function instantiationsCombining the general requirements and our specific choices, we can define thepenalty mapping function in a general form:f (x) = (β −α)xγ +αMγ (0≤ x≤M) (3.3)71x: overlapping area of (smallest pair)f(x): penalty010.5count-basedarea-awareγ = 1xf(x)0 10.51γ > 1 0xf(x)0 10.511 0< γ < 1 (8)Figure 3.4: Illustration of the penalty mapping function for node-node over-lap of a pair of the smallest nodes (Mv1,v2 = 1) in the graph layout. Thethree functions show three different choices of curve shape (γ).This general form comes with three parameters, α , β , and γ , which are responsiblefor minimum penalty, maximum penalty, and curve shape respectively. Our choiceof γ values was guided by empirical results from previous work [83, 162], as dis-cussed in C1 above. With the halfway point choice (C4), i.e. f (.5) = 1, we are ableto remove β from the general form. We can also constrain α using the choices forminimum (C2) and maximum (C3).(NN) β = (1− 1.5.7)α+1.5.7≈ 1.625− .625α (0<α<1)(NE) β = 2−α (0<α<1) (3.4)(EE) β =16pi2−3α (0<α< 16−2pi3pi2≈.328)Inserting the known parameters β and γ , we obtain the specific instantiations ofpenalty mapping function for NN, NE, and EE.72f NNv1,v2(x) = (1−α)(2x)0.7+αM0.7v1,v2 (0≤x≤Mv1 ,v2 ) (3.5)f NEv,e (x) = 2(1−α)x+αMv,e (0≤x≤Mv,e) (3.6)f EEG (x) = (16pi2−4α)x2+α pi24(0≤x≤ pi2 ) (3.7)3.4.4 Parameter space analysisWe conducted a mathematical analysis to understand the influence of the unspeci-fied parameter in the formal statement above, namely, the minimum penalty frac-tion α . In addition to the theoretical analysis presented here, we also report ona computational analysis that allows us to suggest practical choices for values inSupplemental Section B.1.1.We only report on the analysis of α in the NN penalty mapping function, since αfor NE and EE functions is very similar. Figure 3.5 shows the NN penalty mappingfunction for the smallest node pair (Mv1,v2 = 1), using 6 different α values withinthe range (0,1): 0.01, 0.2, 0.4, 0.6, 0.8, 0.99. We also include the function plot fora node pair that is 10x larger (Mv1,v2 = 10) in Supplemental Figure B.1, where theshapes and relationships between curves remain the same except that the x and yvalues are 10x larger.First, we observe that as α increases from 0 to 1, the range of f (x) decreases, andwhen α takes the extreme value 0.99, f (x) is approximately a flat line, degener-ating to the count-based metrics case. Mathematically, the range of the penaltymapping function is f (M)− f (0) = (β −α)Mγ = (1−α)(2M)0.7. Semantically,the range represents its ability to distinguish between minimum, some, and max-imum overlap. This ability has negative correlation with α according to both ourobservation in Figure 3.5 and the mathematical representation above. Second, weobserve that as α increases from 0 to 1, the touching penalty f (0) (i.e. the y-intercept of the curves) also increases from 0 to 1 in Figure 3.5. Mathematically,73Figure 3.5: The node-node overlap penalty mapping function of the smallestnode pair (M = 1), with different minimum penalty fraction (α) values.The α controls the minimum penalty (y-intercept) as well as the differ-ence between maximum and minimum (function range).the touching penalty is f (0) = αMγ = αM0.7. Semantically, the touching penaltyrepresents its ability to distinguish no overlap from minimum overlap, which haspositive correlation with α . Third, the changes of the range and minimum of thefunction is linear to the change of α .Therefore, α is the trade-off between the ability to distinguish different amounts ofoverlap and the ability to distinguish no overlap from touching.3.5 Data generationTo validate the sprawlter metrics, we created small synthetic graphs with both man-ual and force-directed layouts, and applied four different layout algorithms to largereal-world graphs, resulting in 56 layouts in total: 38 synthetic and 18 real-worldlayouts.We created 9 small synthetic graphs, with under 50 nodes and 150 edges. Weeither manually positioned the nodes or applied off-the-shelf force-directed layoutalgorithms in Tulip [10] to make 38 layouts for five targeted purposes: problem74demonstration, debugging and sanity check, progression of clutter, variable sprawl,and limitations of our metrics. We varied the amount of clutter between NN, NE,and EE pairs, and the overall layout sparsity by pushing nodes away from eachother or shrinking node sizes.To confirm that the sprawlter metrics work with large real-world graph layouts,we used 4 large real-world graphs with 100-5K nodes and 500-10K edges. Theseare an academic coauthor network benchmarked in the Koala layout algorithm pa-per [87], a different coauthor network from the InfoVis 2004 contest [51] bench-marked in the GrouseFlocks layout paper [7], an email network provided by Leskovecet al. [102], and the add32 benchmark graph from the graph partitioning archive [160].Supplemental Table B.1 provides a complete list. We applied the following lay-out algorithms to the real-world graphs: two single-level layouts available withinTulip [10], namely GEM [52] and FME [62] (the single-level version of FM3 [68]),the two-level Koala layout designed to emphasize cluster structure [87], and theGrouseFlocks multi-level layout [7]. We chose these based on implementationavailability, computation speed, and ability to handle multi-level structure.Our full computational pipeline is documented in detail in Supplemental Sec-tion B.5. In brief, we convert a variety of input graphs into the Tulip format, applylayout algorithms, extract geometries and the node hierarchy, compute the metrics,and finally display and analyze results for comparison. Except for applying thelayout algorithms, our code is written in Python. In the supplemental materials,we include all input files, output files, implementation code, and Python scriptsfor results analysis. Supplemental materials are available at http://www.cs.ubc.ca/labs/imager/tr/2020/sprawlter/sprawlter-supplemental.zip and open source code isavailable at https://github.com/zipengliu/sprawlter-metrics.3.6 ResultsWe conduct a comparative analysis between the sprawlter metrics, count-basedmetrics, and two from Dunne et al. [39], with respect to qualitative assessment oflayout pictures. We report on the analysis approach, and then discuss instructive75examples across all three of the NN, NE, and EE families. For computational time,we show that our approach only costs a constant multiplicative factor to the count-based approach. Finally, we present the computational parameter analysis for thepenalty mapping functions, guided by the theoretical analysis in Section 3.4.4.A large table with pictures and metric values for all 56 layouts is available in thesupplemental materials. Their running times are shown in Table B.2.3.6.1 Analysis approachWe compute the area-aware clutter penalties (A), sprawl (S), sprawlter (T ), andcounts (C) for each graph layout, in order to validate them separately. We alsoderive the average penalty A/C, the ratio between the penalties and counts, tohelp us find representative cases. When the average penalty is below one (A/C <1), the overlap areas, lengths or crossing angles are less cluttered than the countsuggests, which is typically caused by overlapping between small nodes and edges,near-touching overlaps, or near-orthogonal crossings. When the average penalty isabove one (A/C > 1), the overlaps are more cluttered than the count suggests,which is typically caused by overlapping between big nodes and edges, nearly fulloverlaps, or glancing angle crossings.We also compute the two most relevant metrics proposed by Dunne et al. [39](denoted by D), the global node-node overlap and the global crossing angle. Thenode-node overlap metric is the ratio of the area of the union of the node repre-sentations over their total area if drawn independently (perceived readability overbest readability). A point not discussed explicitly in their paper is that this unionformula assumes nodes are at the same level. It is not obvious how to extend itto multi-level layouts, since the union of parent metanodes and their constituentchildren would be an over-count. We thus compute it only on leaf nodes. Theedge-edge crossing angle metric computes deviations from a specific optimal an-gle, a parameter that they set to 70◦ according to the empirical study from Huanget al. [83]. Although Dunne et al. also propose a group overlap metric that involvesmetanodes, it only counts the crossings between metanodes and leaf nodes, but not76between metanodes, so we do not compute it.We compare the five computational results (A,S,T,C,D) to the layout pictures, andmake a subjective qualitative assessment of correspondence between the metricsand quality of layout. For the first four metrics, low numbers are good and highnumbers are bad. For D, 0 means the worst and 1 the best.3.6.2 Comparative analysisWe present several representative cases in different types of graphs and layout al-gorithms, and discuss how the tested metrics reflect the readability of these cases.(We use α = 0.20 for NN, NE, and EE in this section.)Node-node overlapIncreasing metanode overlaps (synthetic). Figure 3.6 shows a small syntheticgraph layout with increasing overlap between the red and blue metanodes (fromleft to right). The area-aware metrics for NN (ANN) also increase (4.27 < 14.36 <30.62), but the counts (CNN) do not (1 = 1 < 12) and neither does Dunne’s NNmetric (1.00 = 1.00 > 0.90). Looking closer at the breakdown by metanodes andnodes, the penalties of metanodes (4.27,14.36,22.05), compared to penalties ofleaf nodes (0,0,11), are the major contribution towards the totals, which matchesour intuition that big metanode overlap should be penalized more heavily than thatof leaf nodes, whereas the breakdown of counts does not match this intuition.Force-directed layouts for variable-size nodes (synthetic) Figure 3.7 shows asingle-level graph using two force directed layout algorithms, namely Davidson-Harel [31] and stress majorization [53]. The nodes have variable node sizes, as canoccur either where node size is mapped to some quantitative attribute, or wherenodes contain variable-length labels. We can see that there are no overlaps in theright layout and a few overlaps between large nodes in the left layout (16.05 > 0),but the right one is much more sparse (1605.28 >> 6.09). Overall, the left has77Node-node overlapNear-min Some Near-maxA 4.27+0= 4.27 14.36+0= 14.36 22.05+8.57= 30.62C 1+0= 1 1+0= 1 1+11= 12D 1.00 1.00 0.90Figure 3.6: Increasing overlap between red and blue metanodes in a small 2-level synthetic graph. Only closeups for the overlaps are shown. A and Care broken down by penalties / count of metanodes and leaf nodes. Thearea-aware approach (A) matches with the visible configuration betterthan count-based (C) and Dunne’s (D) approach.better readability (9.95 < 40.07), because its partial overlaps are less problematicthan the tiny size of the nodes on the right, with uninterpretable size coding. Thedifference of Dunne’s metrics between left and right do not suffice to signal thelarge node overlaps in the left, nor do they address the sparsity problem in the left.Many touching leaf node pairs and high sprawl (real-world). Figure 3.8 showsa large real-world graph from the partition benchmark dataset, with 4960 nodes and9462 edges, using FM3 layout (single level). The most noticeable feature about thislayout is that the nodes are too tiny to be seen and there is huge wasted space on theupper left and bottom right, resulting in a large sprawl (319.28). Despite the manytouching overlaps between the tiny leaf nodes, as shown in the inset on the upperleft, the area-aware penalty is still less than count (213.28 < 670), showing that thepenalties for these small overlaps sum to an appropriate amount that is non-zerobut not disproportionate. Dunne’s metric (almost 1.00) does not reflect the wasted78Node-node overlap and sprawlDavidson-Harel Stress majorizationA 16.25 0C 6 0S 6.09 1,605.28T 9.95 40.07D 0.91 1.00Figure 3.7: A single-level synthetic graph with variable node sizes. The leftlayout has a lower sprawlter metric than the right layout although thereare substantial overlaps between nodes; we argue that qualitatively thereadability is indeed better.space or the touching overlaps.High vs low sprawl (real-world). Figure 3.9 shows an email network layout byGrouseFlocks, with only a few metanodes open and others closed. There are only61 nodes and more than 1000 edges in the left layout, and 88 nodes and more than2000 edges in the right layout. As the overlaps only happen between metanodes,Dunne’s metric (1.00) is not helpful. The left layout is less cluttered than the rightone in terms of both area-aware penalties (133.52 < 184.13) and counts (16 <40), but the left is less efficient in space usage than the right (i.e. higher sprawl,947.59> 219.59), resulting in a higher sprawlter (335.71> 201.08). This exampleand the previous one suggest that both clutter and sprawl are important aspectsof measuring graph layout quality, and the trade-off between the two should beconsidered.79Node-node overlap and sprawlA 213.28C 670A/C 0.32S 319.28T 260.95D 0.99Figure 3.8: Large real-world graph layout by FM3 in Tulip (single-level),with high sprawl and many touching events. The average penalty (A/C)indicates penalties for overlaps are small but non-zero, and the highsprawlter value (T ) indicates that this layout is bad in terms of sprawl(S) and clutter (A).Node-edge overlapIncreasing metanode and edge overlaps (synthetic). Figure 3.10 shows a pro-gression of node-edge overlap on a small synthetic layout. Both the area-awaremetrics (12.11 < 54.46 < 136.72) and counts (6 < 15 < 36) are increasing to in-dicate the increasing overlap between the blue metanode and edges, and also theconstituent leaf nodes and edges, but the increasing average penalty (2.02< 3.63<3.80) indicates that the penalty increases at a faster pace. As in the analysis ofFigure 3.6, the breakdown of penalties by metanodes and leaf nodes shows thatoverlap of metanodes and edges contributes more than that of the leaf nodes. Thearea-aware metric better aligns with our judgement of the layout pictures, because80Node-node overlap and sprawlA 133.52 184.13C 16 40S 947.59 219.59T 355.71 201.08D 1.00 1.00Figure 3.9: Two large multi-level real-world graph layouts by GrouseFlocks(4-level). The left layout has higher sprawl but less overlap than theright one, captured by the sprawl S, and clutter with area-aware ap-proach (A) and count-based one (C).the clutter between nodes and edges deteriorates as the blue metanode moves up-wards.Increasing metanode and edge overlaps (real-world). Figure 3.11 shows thecomparison between area-aware metrics and count-based metrics on a real-worldexample. On the left layout, all leaf nodes are hidden; on the right layout, weopen up two of the orange metanodes, making it more cluttered. For the node-edgeoverlap, there are significantly fewer overlaps on the left layout, which is reflectedby both metrics (4482.20 < 32618.75, 539 < 1650), and by the average penalty(8.32 < 19.77). Notice that there are many edges that fully cross the biggest greenmetanode on the right layout, resulting in an extremely high average penalty. Also,the fact that A >>C for both layouts matches our intuition that overlaps related to81Node-edge overlapNear-min Some Near-maxA 5+7= 12.11 40+15= 54.46 98+39= 136.72C 1+5= 6 5+10= 15 8+28= 36A/C 2.02 3.63 3.80Figure 3.10: Increasing overlap between blue metanodes and edges in a small2-level synthetic graph. Although counts (C) are increasing, the area-aware approach (A) captures the deteriorating situation in the pic-ture more precisely, visible via the increasing ratio of average penalty(A/C).big metanodes are visually salient and should be penalized heavily.Edge-edge crossingDecreasing crossing angles (synthetic). Figure 3.12 shows a progression of de-creasing angles between crossing edges on a small single-level synthetic graph,from orthogonal to near-glancing angles. There are exactly two edge-edge cross-ings on each layout, but the area-aware metric is increasing (0.99 < 1.46 < 3.45 <4.42). In two layouts on the right, the edges are almost on top of each other, whichlooks a lot worse than the two layouts on the left. The count does not indicate suchclutter on this naive example, whereas the area-aware metric is a better match tothe degree of clutter of the edge-edge crossings. Dunne’s metric also goes downas clutter increases (0.73,0.87 > 0.28 > 0.10) (least clutter is reached at 70◦), butreports near-maximum EE readability in the orthogonal case which ignores theexistence of remaining edge-edge crossings.Orthogonal vs glancing angles (real-world). Figure 3.11 shows similar results82NEA 4,482.20 32,618.75C 539 1650A/C 8.32 19.77EEA 703.67 5,936.89C 808 4,154A/C 0.87 1.43D 0.73 0.46Figure 3.11: Two large multi-level real-world graph layouts by GrouseFlocks(4-level). The area-aware NE metric (ANE) captures the property thatthere is less node-edge overlap on the left than the right. The area-aware EE metric (AEE) captures that on the left the crossing anglesbetween edges are closer to orthogonal angles, while on the right theyare closer to glancing angles.Edge-edge crossingOrthogonal Near-half Small Near-glancingA 0.99 1.46 3.45 4.42C 2 2 2 2D 0.73 0.87 0.28 0.10Figure 3.12: Decreasing crossing angles between edges in a small single-level synthetic graph, captured by the area-aware EE metric (A).on real-world examples. From the layout pictures, we can see that the edge-edgecrossings on the left layout are mostly near-orthogonal, while those on the right83layout are mostly small and even near-glancing angles. Not only are there feweredge-edge crossings on the left (808 < 4154), but the clutter caused by crossingangles is also less, which is reflected by the average penalties for EE (0.87 < 1.43).In this case, Dunne’s metric (0.73 > 0.46) is also successful in differentiating thetwo layouts.3.6.3 Computational timeAs we see from algorithm 4, the sprawlter metrics are computed along with thecounts with a computational complexity linear in the number of NN, NE, and EEpairs. Therefore, we expect that the computation of the sprawlter metrics wouldbe in the same scale as that of the count-based metrics, with a constant factorslowdown.For every graph layout measured, we timed the full computation of the sprawltermetrics, and also of the count-based metrics by skipping the code only relevantto sprawlter metrics; that is, operations within the inner condition in algorithm 4are all skipped except the one that increments count (line #10). We used a 2012MacBook Pro with a 2.5 GHz Intel Core i5 CPU and 8GB RAM to run all com-putations, and report the average computational times over 4 runs. The absoluterunning time is related to the number of nodes and edges, and also the number ofcrossings. For example, for a large real-world graph with about 5k nodes and 10kedges (Figure 3.8), it takes 128s, 505s, and 510s to compute the NN, NE, and EEsprawlter metrics respectively, while it takes 92s, 358s, and 372s to compute thecounts.We then compared the two execution times by deriving the ratio between runningtime of sprawlter metrics and counts, which is the equivalent of the slowdown fac-tor. We excluded execution times under 1 second as unreliable estimates. Theresults align with our expectation. The average ratio over all included layouts is1.38 for NN, 1.50 for NE, and 1.85 for EE, with relatively small standard devia-tions. Our findings suggest that the slowdown factor is a constant, which rangesfrom roughly 1 to 3.84Our Python implementation of the sprawlter metrics serves adequately as a proof-of-concept, but there are obvious directions for optimization. Our naive algorithmsimply iterates over every pair of nodes (and edges in NE and EE case). It wouldbe possible to apply space decomposition algorithms to speed up the computation,such as the Bentley-Ottmann line sweeping algorithm [14] to compute all crossingedges.3.7 Discussion and Future WorkOverall, the area-aware approach clearly outperforms the simple count-based ap-proach in the NN, NE, and EE cases, and also the Dunne approach for the NNcase. Our results are somewhat better than Dunne’s EE case: more edges doesimply more crossings, but not necessarily worse crossing angles. The area-awareapproach is more complicated than the others due to a parameterized penalty map-ping function; we argue that the benefit of precision is worth this cost.Our approach does not impose an upper bound for the metrics, in contrast toDunne’s metrics that are within [0,1]. Their metrics are essentially ratios of thecurrent readability and the best achievable readability, where 1 corresponds to themost readable and 0 to the least readable. In particular, they propose normalizingthe EE metric by the O(|V |2) term of all possible edge crossings in a completegraph, but this number is misleading because the density levels of node-link graphdrawings must be far below that ceiling in order to be comprehensible; Melan-con [121] argues that typically |E| < 4|V |. While having a fixed bound is bene-ficial in comparing the same graph with different layouts, it is not for comparingdifferent graphs. Their metrics are ratios relative to the graph itself, comparing theperceived readability against the best it could be, but the same number would havea very different meaning for a small and simple graph than for a large and complexgraph.We argue for the necessity of clutter metrics despite the existence of clutter removalalgorithms [42]. The entire point of our area-aware clutter metrics is to more pre-cisely quantify the clutter that results from overlap. It could be used algorithmically85to determine which layout algorithm to use for a particular graph; for example,whether the computational cost of overlap removal is worth the improvement inthe layout quality. Moreover, overlap removal sometimes results in an increase ofsprawl (especially for large graphs), so the trade-off between sprawl and clutter canhelp determine to what extent we want to remove overlap completely, partially, orjust leave it. Moreover, many layout algorithms used in practice do not guaranteethe total elimination of all overlaps; their approach to accelerating computationmay leave some in place. We would like to see these new metrics incorporated intolayout algorithms, which would be easy to do with optimization-based approaches.One limitation of the sprawlter metric is that only clutter and sprawl are captured.Other readability factors that affect the overall quality of a layout include its abilityto show cluster structure; using spatial proximity to show groupings may incurclutter costs as a direct trade-off, as in the Koala layouts (shown in the supplementaltable of full results).Our general area-aware approach could be extended to cover a wider range of met-ric families as future work. Another interesting direction of future work would bea resolution-aware approach. In the mathematical formulas, pixels can be dividedinto infinitely small sub-pixels. If two nodes just touch each other, the overlappingarea is exactly zero. However, in the practical world, pixels on a screen are dis-crete. An explicit awareness of the display resolution would provide the precisionrequired to exactly assess when there is no longer visible space between two itemsbecause they occupy neighboring pixels.A very rich direction for future work would be to incorporate more aspects of multi-level structure by considering metaedges. Progress may not occur until more layoutalgorithms support them and more psychophysical experimentation has taken placeto untangle the factors that affect edge-edge crossing clutter beyond the crossingangle.863.8 SummaryWe contribute the area-aware sprawlter metric for graph layout to capture importantproperties beyond integer counts of crossings on the leaf level by accounting forthe amount of overlap and beyond clutter by accounting for the geometric sparse-ness of layout. We instantiate and implement the sprawlter metric, and present thetechnical details for the crucial component, the penalty mapping function. Throughour validation with synthetic and real-world layouts, we show that the new metricsucceeds at providing a better quality measure than previous work by taking intoaccount both clutter and sprawl.87Chapter 4LogSeg: Data-driven Multi-levelSegmentation of Image EditingLogs1 Automatic segmentation of logs for creativity tools such as image editing systemscould support smart history navigation to improve their usability and learnability.We propose a multi-level segmentation model that works for many image editingtasks including poster creation, portrait retouching, and special effect creation. Thelowest-level chunks of logged events are computed using a support vector machinemodel and higher-level chunks are built on top of these, at a level of granularitythat can be customized for specific use cases. Our model takes into account fea-tures derived from four event attributes collected in realistically complex Photo-shop sessions with expert users: command, timestamp, image content, and artworklayer. We present a detailed analysis of the relevance of each feature and evaluatethe model using both quantitative performance metrics and qualitative analysis ofsample sessions.1This chapter is a slightly modified version of the following publication: Z. Liu, Z. Liu, and T.Munzner. Data-driven Multi-level Segmentation of Image Editing Logs. Proc. CHI Conf. HumanFactors in Computing Systems, pages 1–12, 2020. ACM. [111].884.1 MotivationAnalyzing and modeling user interaction logs can address challenges posed bycomplex user interfaces in professional creativity tools. A user can issue manyactions in a short period of time, but it could be challenging to keep track of theworking process on a fully loaded user interface. Segmenting action logs intosemantically meaningful pieces could alleviate one’s cognitive load of low-levelinformation. A primary use case of segmentation is smart undo [26]: if we can au-tomatically segment the log into chunks of actions, the user will be able to navigateediting history more easily by undoing or redoing a group of related actions insteadof one action at a time. Other use cases that also rely on accurate identification ofkey decision or transition points in the log data include smart version control [25],automatic generation of tutorials from interaction logs, and recommendation ofalternative design ideas or workflows [28].In this chapter, we investigate the segmentation of image editing logs from AdobePhotoshop (PS). Figure 4.1 shows a concrete example of a segmented image log,introducing key terms used in this chapter. Designers and artists use PS for a vari-ety of tasks, such as composing a poster, retouching a photo, and applying specialeffects to an image. A task is accomplished in a number of steps or subtasks,for example positioning images and tuning colors during poster creation. A sub-task may be further decomposed into smaller subtasks, for example adding textinvolves making text content and applying text color. The hierarchy of subtasks ina complex PS task is not balanced; they may have different granularities. At thelowest level, a subtask is accomplished by a group of events, namely individualoperations triggered through key strokes or mouse clicks. Each event has a numberof attributes, including the command invoked, such as Move, and timestamp.The entire sequence of events logged in the process of accomplishing a PS taskcomprises a session.We define segmentation as grouping consecutive events within a session into ahierarchy of semantically meaningful chunks. A perfect segmentation method willproduce chunks that correspond to the subtasks. The desired subtask granularity89Position imagesTune colorsMake lighting around charactersAdd textAdd more light to charactersCheck resultsEventsSession: a poster creation taskHigh-levelchunks(subtasks)Attributes {layer ID: “15”, command: “Move”, image: , timestamp: “19:30:23”}Make title contentType textEdit and move titleSet title styleMake subtitleApply color to title1st trial2nd trial using clipping maskMake a mask layerAdjust opacityLow-levelchunksFigure 4.1: Concrete example of session segmentation. High-level chunksconsist of low-level chunks, which consist of logged events, which haveattributes. The single expanded chunk shows the non-balanced natureof the hierarchy.depends on the downstream use cases: for example, smart undo and redo willrequire finer granularity, while creating an overview of the whole session can bemore coarse-grained.Despite the wide range of potential applications, many questions about segment-ing image editing logs remain unanswered. First, image editing tools support di-verse tasks such as photo retouching, image composition, special effects creationand user interface design. These different tasks result in very different workflows.However, existing segmentation methods usually are limited to one type of edit-ing task [26, 192], and provide no evidence on whether segmentation models cangeneralize across task types.Secondly, many segmentation approaches (e.g. hierarchical clustering [153], se-90quence progression models [186], infinite hierarchical hidden Markov models [74])assume that command name and timestamp are the only two relevant event at-tributes. Other event attributes such as image content and editing area overlap havebeen included in segmentation models [25, 26]. The relevance of these event at-tributes for segmentation has not been thoroughly analyzed, and the suitability ofother potentially relevant attributes has been ignored.Finally, real-world user behaviors are complex due to individual differences. Col-lecting realistic workflow datasets with rich event attributes and labelling themas ground truth is expensive and time-consuming. As a result, the log datasetscollected in previous work either enforce homogeneous workflows with predefinedsubtasks [26], or are unrealistically clean without user mistakes or digressions [25].It is unclear if certain kinds of user behavior may pose challenges in creating un-ambiguous labels and building robust segmentation models.In this chapter, we take a first step in addressing these problems, targeting smartundo [26] as a primary use case. Our goal is three-fold: build a reasonable model,understand the relevance of features, and investigate the performance of the modelin the context of user interface applications given the complexity of heterogeneoustasks and individual variation in user behavior.We collect log data of realistic workflows from PS expert users for different typesof image editing tasks. For each event in a sequence, we record four attributes(command, layer, image content, and timestamp) that are potentially relevant forsegmentation, and label chunk boundaries in all collected sessions as ground truth.We derive five features from these for use in machine learning (ML) models: com-mand similarity, layer similarity, image diff, working region overlap, and dura-tion. We then develop a two-stage segmentation model: a supervised machinelearning model to predict the boundaries for low-level chunks, and two alterna-tive approaches to construct high-level chunks from the low-level ones – multi-tierthresholds or agglomerative clustering. This two-stage scheme allows for the flex-ible choice of levels to support various downstream use cases.Our primary contributions are: 1) a simple and interpretable low-level segmenta-91tion model that works reasonably well for smart undo, and 2) an analysis of featurerelevance to segmentation, where we find that command, timestamp, and layer arerelevant event attributes, but image content is not as useful. We are the first touse layers for segmentation and confirm their relevance. We validate the modelwith quantitative evidence showing that our model generalizes across three imageediting tasks for low-level chunks. Our secondary contribution is an adjustablehigh-level segmentation model built on top of the low-level one. We conduct a pre-liminary qualitative analysis of the suitability of higher-level chunks for these threetasks. We also provide observations on the complexity of real-world user behaviorand note the challenges it raises for multi-level segmentation.4.2 Related WorkWe briefly introduce the related work on event sequence visualization and graphicalhistories, then focus on computational models of event logs.4.2.1 Event Sequence VisualizationEvent sequences have been studied frequently in the visualization community forapplication domains including e-commerce, program execution traces, and medicalrecords, where researchers focus on visualizing large-scale collections of eventsequences to get data insights such as identifying common patterns or outliers [35,65, 109]. In contrast, for segmenting image editing logs we are dealing with onesequence (session) at a time, so visualizing many sessions at once does not helpwith segmentation.4.2.2 Workflows and Graphical HistoriesResearchers have proposed graphical histories of workflows in domains rangingfrom general scientific analysis [22] to specific biological experiments [115] to thestandalone visualization application Tableau [72, 114]. However, these domain-92specific approaches do not adequately address the interface problems of creativitytools such as image editing.Work targeted at creativity tools addressed the question of how to represent andmanipulate application state to undo and redo in single-user [46] and multi-usercollaborative settings [156]. Other approaches used annotations on the graphicalinterface to inform user interaction history [127, 164], or supported the capture,exploration, and playback of creative document workflow histories [63]. The Deltasystem visually represented and compared a small amount of short workflows [96].Grabler et al. [59] and Chi et al. [27] proposed an automatic generation of multi-media tutorials from logged sessions in GIMP (an alternative to Photoshop). Thefocus of these papers are either on data models or visualization of interaction his-tory. Some of them tried to group interactions into chunks, as we do, but their goalwas to support selective undo that ties to space on the canvas, whereas our goal isto group events into subtasks that might or might not relate to spatial proximity onthe canvas.4.2.3 Event Log ModelsThe previous work on segmentation models for general event logs using approachessuch as dynamic programming [169], hidden Markov models [74], and probabilis-tic generative models [186] does not perform well in our setting of image editinglogs with rich attributes. Moreover, some of them require large-scale data whichwe do not have. The work on non-segmentation models for general logs, such asbias detection in visual analytics logs [177], is more distant from our own; modelstargeted at creativity tool logs are more relevant.Creativity Tool LogsWe take inspiration from many non-segmentation models of logs. Lafreniere etal. studied and characterized large-scale real-world image editing command logs [98],providing us with interesting patterns of command usage in image editing tools.93Adar et al. proposed CommandSpace to model the relationships between com-mands, subtasks, and text descriptions of subtasks [3], using machine learningmodels for natural language. Our command similarity feature is computed in asimilar way. Two other approaches to learning usage semantics from commandlogs solidified our interest in ML approaches: Yang et al. proposed a deep learn-ing model, using command logs, for representation of PS users [187], and Wanget al. proposed a two-step log classifier to recommend frequent patterns [178].Causality, a conceptual model by Nancel and Cockburn [128], models causal rela-tionships between events, which helps us think about structure within a session.The few papers on segmentation models of creativity tool logs are most relevant toour work. Denning et al. proposed to use regular expressions to retrieve a multi-level segmentation interaction logs in a mesh construction application [33, 34],which does not apply in image editing case as there is more variation in our tasks.Chen et al. proposed a non-linear revision control system to model, aggregate, andvisualize image editing logs [25]. However, the non-linear model is not intuitivefor PS users, who can already achieve such non-linear edits with non-destructiveediting techniques [135]. Moreover, their dataset seems unrealistically short andclean. Chen et al. later proposed to use a support vector machine to predict a single-level segmentation for portrait retouching logs [26], which specifically targets onetask with predefined subtasks. They also compared to a baseline from Li et al. thatuses duration for segmentation [107], and argued that duration is not effective. Wewere inspired by this approach to use support vector machines ourselves. Zhao etal. developed the Sketcholution system to get a high-level segmentation of sketch-ing logs using an agglomerative algorithm [192], but it is not clear that the specificsolution generalizes to other tasks. However, we were inspired to try agglomerativeapproaches as well. In this chapter, we are dealing with multiple common PS tasksinstead of one, and we collect more realistic sessions than previous work. The useof layer as a feature is absent from all of this previous work, as is the systematicinvestigation of the relevance of each feature.944.3 ApproachWe provide an overview of our approach and its rationale for data generation,model selection and evaluation methods.4.3.1 Data GenerationBased on literature review and our interaction with Photoshop users, we identifyfour potentially relevant event attributes: command, timestamp, image content, andlayer. Command, timestamp and image content have been used in previous work[25, 26, 192]. Layer has not been considered in previous work, but we introduceit as a potentially important attribute because of our observation that layers areessential in any realistically complex PS tasks.No existing logs captured all four of these attributes, so we created a logging PSplugin to record this information for each event during normal usage sessions. Weconducted user studies to collect data from experts carrying out multiple tasks,capturing these logs and also think-aloud recordings. We used these records tomanually create labels for chunks at multiple levels of granularity. For low-levelchunks, we found good inter-coder agreement between two independent coders.However, for high-level chunks there was little agreement even between humancoders. We decided that only the low-level chunk labels were reasonable to con-sider as ground truth, a decision that in turn affected our choices of models andevaluation strategies. We also concluded that deeper investigation of the relevanceof segmentation attributes was merited.4.3.2 Model and Feature SelectionThe goal of the segmentation model is to predict whether an event e is the startingpoint of a new chunk (similar to Chen et al. [26]). Considering that applicationssuch as smart undo operate on streaming data during interactive sessions, we ruledout building models that required knowledge of both the events that precede and95those that succeed e. Instead, our model makes predictions based on only the eventsthat precede e. We further choose to build on similarity (or distance) measures be-tween events according to their attributes, under the assumption that similar eventsare likely to be in the same chunk.Our finding that only the low-level chunk labels are reliable led us to develop a two-stage segmentation model. For the low-level chunks, we use a supervised approachbuilt on the human labels as ground truth; we apply an unsupervised approach forsegmenting higher-level chunks. We frame the low-level segmentation problem asan instance of binary classification in machine learning (ML), where the classes are1 for the boundary and -1 for the non-boundary. We also conducted early experi-ments with multiple hand-designed rule-based models for low-level segmentation(e.g. segment when a user switches to a different layer), which we documented inSupplemental Section C.5, but found that they were substantially outperformed bya simple and popular ML model, a support vector machine (SVM). Since our focusis to gain a deeper understanding of the segmentation problem and we do not havelarge-scale data for training, we prioritize the interpretability of a simple modelover more complex and powerful ones like random forests or neural networks. Weuse a linear kernel, again for simplicity. During development, we also tried SVMwith a quadratic (RBF) kernel, but since it did not improve performance we stayedwith the simpler approach.The simple SVM model achieves good performance, as we show below in Results,with a small number of features (five) that we derive from the four logged at-tributes: command similarity, layer similarity, image diff, working region overlap,and duration.4.3.3 Evaluation MethodsWith reliable labels on low-level chunks, we are able to perform quantitative eval-uation on the SVM model by analyzing feature relevance and the model’s per-formance on low-level segmentation. The lack of ground truth for the high-levelsegmentation led us to a qualitative-only evaluation: we constructed visualizations96compact enough to show entire sessions with event-level detail, so that we couldinspect the relationship between feature values and compare the predictions withhuman labels.4.4 Data Collection and CharacterizationWe report our data collection procedure and characterize event attributes.4.4.1 Data CollectionOur study with expert users was conducted in two rounds of data collection, earlyand late in the project timeline. The data collected from the first round was usedfor exploratory analysis and model building, while data from the second round wasmainly used for model validation and testing.We compiled a pool of PS tasks (9 in round one and 11 in round two) from popularPS tutorial websites to use in the studies, falling into three types: poster creation,portrait retouching and special effects creation. We recruited 13 PS expert userswith many years of in-depth professional experience (ranging from 2 to over 10years), with 7 participants in each round; one person participated in both rounds.The study was run for each participant via online video conference, lasting aroundone hour in total. We first introduced the participant to the goal of the study, andhelped them install the PS plugin. We showed them the pool of PS tasks, and askedthem to choose one (or two) tasks that they felt most comfortable and competentworking on. We provided all the required source images for the task, and a finalimage for reference (except for portrait retouching task). We asked the participantsto think aloud throughout the session, and recorded the entire video conferencewith screen and audio information.We collected a total of 16 sessions, 8 from each round, where one participant ineach round did two sessions. Broken down by task type, there were 8 sessions forposter creation, 6 for portrait retouching, and 2 for special effect creation. There97were 5718 events logged in total across all sessions, with an average of 357 persession (min: 29, max: 1064), and each session lasted an average of 33 minutes(min: 8, max: 53). Our PS recording plugin saved command names, timestamps,and layer names & IDs in a text file, and the image content was saved as JPEGscreenshots.We manually segmented each session into multi-level chunks by labelling the start-ing and ending events of each chunk. We built an interface showing all the eventattributes in a table and used it in conjunction with the video recordings to performthe labelling, as shown in Supplemental Figure C.9.The labelling process involved subjective judgements about which events shouldbe grouped as a chunk at the lowest level, and whether low-level chunks should begrouped into a higher-level chunks; for example, whether to group “Make title” and“Make subtitle” together in Figure 4.1. To assess inter-coder reliability, two codersindependently labelled the starting events of lowest-level chunks in two sessions(a total of 351 events) and compared their results. We found that the two codersmostly agreed with each other (Cohen’s kappa κ = 0.77), which suggests that thelow-level chunks are reliable. However, among the 14 incongruent events, 11 werecaused by disagreement on granularity (e.g. one coder labeled a chunk of events23-44, while the other labeled two chunks 23-39 and 40-44), indicating that therewould be large inter-coder differences on high-level chunks. We thus chose not tocompare human-generated high-level chunks between coders. This observation ledto our decision to only treat the lowest-level labeled chunks as ground truth. Onecoder then continued to label all the remaining sessions.4.4.2 Data CharacterizationCommands are directives from users. The number of unique commands in PS islarge: there are about 1400 menu items in Photoshop 2019, each correspondingto a command. Each command is associated with a meaningful and interpretablename, and is considered highly relevant to segmentation in previous work [25, 26].98Layers are essential for managing visual objects, achieving complex visual effects,and performing non-destructive editing. They are created, ordered, and possiblyplaced into groups within a hierarchy explicitly by the PS user; three examples areshown in Supplemental Figure C.4. During a session, users activate different layersat different points of time to work on, and the layer hierarchy evolves over time.The final count of layers in each session ranged from 3 to 16; the layer hierarchywas always shallow, with final depth ranging from 1 to 3. We consider layer apotentially relevant signal because a change in active layer could indicate a shift insubtask, and the changes in layer hierarchy reflect the user’s mental models of theartwork composition.The image content of an event refers to the pixel state of the artwork after the op-eration has been carried out. The changes in image content after an event could bea useful indicator of the active working area, and may be relevant to segmentation.The timestamp is used to derive duration. Duration between consecutive eventscould be relevant to segmentation, if rapid-fire events are more likely to occurwithin a chunk, whereas a long pause between two events might signal a boundarybetween chunks.4.5 Feature ComputationWe derive features for our segmentation model from the four event attributes col-lected in the expert user sessions: command, layer, image, and timestamp. The fivefeatures that we derived are specific instances of a similarity (or distance) measurebetween events: command similarity, layer similarity, image diff, working regionoverlap, and duration.4.5.1 Command SimilarityA conventional approach to quantify command similarity is to group commandsaccording to the menu hierarchy in the Photoshop interface [26]. This approach99fails to differentiate commands within the same cluster, and might not capture howcommands are used in reality. An alternative approach, validated in the Com-mandSpace previous work [3], is to learn the semantics of the words using machinelearning tools for large natural language datasets. The rationale is that commandsare analogous to words and a PS session is equivalent to a document, where theproximity of logged commands within a session expresses meaningful semanticsthat capture typical usage patterns. Inspired by this idea, we obtained access to acommand log database internal to Adobe, which only recorded commands for PSevents from opt-in non-enterprise users. We extracted 100 million commands fromthis database and reconstructed 169,387 sessions by grouping according to associ-ated session identifiers. We ran word2vec [122] to obtain a vector representationwith 100 dimensions for each unique command. We then computed the similarityscore between two commands as the cosine similarity between their correspond-ing vector representations. We conducted a sanity check of the learned commandvectors using the Google Embedding Projector [158] and found the results to beplausible, as discussed in Supplemental Section C.1.1.4.5.2 Layer SimilarityWe compute layer similarity using rules that we manually generated based on thecombination of observations of participant sessions and additional conversationswith PS experts about their use of layers. We classify the strength of the relation-ship between two active event layers A and B, into five ranked categories and assignheuristic values accordingly, as shown in Table 4.1. Users frequently operate onthe same layer consecutively for a while before switching to a different one, andwe assign the maximum similarity value of 1 between these identical layers. Theuser often copies a layer to a new one to make repeating objects, edit repetitively,or back up a layer, so we assign an 0.8 similarity value between a duplicate andits source layer. An adjustment layer applies color and tonal adjustments to one ormore main layers without permanently changing pixel values, a common techniquefor non-destructive image editing. Layers stack up on each other in pixel space, andalthough technically all layers below another could be affected we determined that100users often consider only directly above/below pairs, so we assign a measure of0.5 similarity between the two. For layers in the same group, We assign a valueranging from 0 (infinitely distant layers) to 0.5 (direct sibling layers) depending onlayer proximity in the hierarchy: layers separated by d hops have similarity 1/2d−1.Finally, all other layers that differ are assigned with the minimum similarity 0. Wesummarize this description with detailed formulae in Supplemental Section C.1.2and provide annotated examples of layer hierarchies in Supplemental .Relationship Description SimilaritySame layer A = B 1.0Duplicate layer A is a copy of B 0.8Adjustment layer A is an adjustmentlayer of B0.5Grouped layer A and B are locatedin the same layer group<=0.5Other diff. layer none of the above 0.0Table 4.1: The five kinds of relationships between two layers A and B andtheir assigned similarity values. Each relationship is commutative.4.5.3 Image DiffThe image difference (diff) of two events is the percentage of different pixels be-tween the two corresponding images out of the total number of pixels in an im-age. The implied conjecture in previous work is that in an image editing sessionthe amount of change, as quantified using image diff, is relevant to segmentationwhere a larger image diff implies greater difference between two events. Althoughthere exist more advanced image diff scores such as the structural similarity index(SSIM) [181], we do not choose them because they are potentially biased to a par-ticular type of image edits: for example, SSIM is sensitive to structural changesbut not color tone changes.1014.5.4 Working Region OverlapWe also compute a second potentially relevant image-related feature, to assesswhich of the two is a better indicator. The working region overlap measures thedegree of continuance in pixels between two events. It is defined as the percentageof overlapping changed pixels between two events out of the total number of pixelsof an image. We conjecture that two events are similar if they operate on a similarpixel area, and thus that they would be likely to belong to the same chunk. Wecompute it as a “diff of a diff”: count the different pixels between the diff image oftwo events down-sampled by a factor of 32, where a diff image of an event is thedifference image between itself and the previous one. We down-sample to checkfor overlap in coarser regions rather than individual pixels.4.5.5 DurationThe duration is simply the time between consecutive events.4.6 Segmentation ModelOur two-stage model predicts low-level chunk boundaries with an SVM classifier,and has two alternatives for computing higher-level chunks from those results.4.6.1 Low-level SegmentationTo classify events as boundary or non-boundary with an SVM, we must constructa feature vector, partition the data, consider the appropriate point in the precision-recall trade-off space, and tune the hyperparameters.102Feature Vector ConstructionTo determine whether an event is the starting point of a chunk, a reasonable wayis to compare it against its previous events in terms of the computed similarityfeatures. If the difference is big enough, it is likely to belong to a new chunk. Weconsider a window of k previous events for comparison, where the window size kas a hyperparameter that will be tuned. For each previous event in the window, weuse the five similarity (or distance) measures, described in the previous section. Wethen concatenate all measures of comparison to all previous events in the windowinto a feature vector, whose size is k×5.Data PartitionWe split our collected dataset (5718 events in 16 sessions) into training, validation,and test sets. By the second round of data collection we had finalized what featuresand model to use; we set aside 5 of the 8 sessions that second round as the test set,and used the other 11 sessions for training and validation (3 from the second round,and all 8 from the first round). The 5 test set sessions are randomly drawn withstratified sampling for task type: 2 for poster creation, 2 for portrait retouching,and 1 for special effect creation. This procedure results in a total of 1842 events(about 30% of grand total) in the test set. The remaining 3860 events are furthersplit randomly into training (75%) and validation (25%) set.Precision Recall Trade-offOur dataset is unbalanced, with many more non-boundary events than boundaryevents. We consider the trade-off between precision and recall in model trainingand validation, and deliberately trade precision for higher recall. It is importantto correctly identify as many boundaries as possible, and false negatives (bound-aries predicted as non-boundaries) are undesirable as it is not easy for end usersto identify these missed boundaries. On the other hand, false positives (predictingnon-boundaries as boundaries) are less detrimental: although these errors lead to103over-segmentation of the logs, users can quickly dismiss them through an interac-tive interface.Hyperparameter TuningWe train the model on the training set, and measure performance on the validationset to tune these hyperparameters, following common practice in machine learning.Supplemental Section C.3 covers the procedure and results in detail. In brief, thefirst hyperparameter, window size k, is the number of previous events used in thefeature vector. Using receiver operating characteristic (ROC) curves, we find thatwindow size has little influence on the model performance, and thus we choosek = 1 to reduce the size of feature vectors. The second one, threshold t, determinesthe probability value where an event is predicted to be a boundary. We selectt = 0.24 in consideration of the trade-off between precision and recall.4.6.2 High-level SegmentationThe higher-level chunks are built based on the predicted lowest-level chunks.Thehigher-level segmentation should be flexible in granularity, to accommodate dif-ferent downstream use cases. We describe two methods to construct high-levelchunks: multi-tier thresholds in SVM and agglomerative clustering.Method 1: Multi-tier ThresholdsIn the SVM model for lowest-level segmentation, the threshold t can be interpretedas the granularity of segmentation: as t increases, the granularity gets coarser andthere are fewer predicted boundaries (chunks). In that case, we use a thresholdthat achieves the optimal performance on a labeled dataset (the validation set).Intuitively, we can simply use a series of higher thresholds to obtain multiple higherlevels of chunks despite the lack of ground truth. Every higher-level boundary willnecessarily fall along a lower-level boundary, since the same probability scores are104in use.Method 2: Agglomerative ClusteringHigher-level chunks can also be generated from lowest-level ones using the bottom-up approach of agglomerative clustering [192]. This algorithm uses a predefinedsimilarity metric to iteratively find the most similar pair of adjacent chunks andmerges them together, until a single remaining chunk contains the whole session.We define the similarity between two chunks as the average similarity of all pairsof events across the chunks. We compute a weighted sum of the five feature values,where the weights are the linear coefficients of the kernel function in the learnedSVM model, since the coefficients indicate feature importance. The output of theagglomerative algorithm is a binary tree, where each node represents a chunk andthe distance to root can be considered as its granularity. Therefore, we can makea cut through this tree to obtain chunks of the desired granularity, either a straightcut with uniform distance to the root or a more complex shape to capture subtreesof different resolution.ComparisonBoth methods reuse some parts of the low-level SVM model: the multi-tier thresh-old method reuses the probability score directly, whereas the agglomerative methodleverages the linear coefficients in the kernel function.We compare the two methods in terms of computational complexity, potential lim-itations, and usage pattern. For computational complexity, the multi-tier thresholdmethod does not need extra computation: it is O(n), where n is the number ofevents. The agglomerative method has to compute event and chunk similarity iter-atively. Its computational complexity is O(n2), an acceptable cost since a sessioncontains a few hundred events.A potential limitation of the multi-tier threshold method is that the SVM model105for low-level segmentation only leverages minimal context information (one pre-vious event), which has not been validated for the high-level one. In contrast, theagglomerative clustering computes average values over many event pairs as thechunk similarities, making it more robust against single-event outliers. However,the agglomerative method may yield sub-optimal results due to the greedy mergingmechanism [48].Thresholds in the multi-tier method are straightforward to specify but less flexi-ble as one value is used across all levels of the hierarchy. The cluster hierarchyresulting from the agglomerative clustering method provides the flexibility for anadaptive criterion that can vary within a session to select nodes according to thespecific use case needs.4.7 Results: Feature RelevanceThe performance of segmentation model highly depends on the effectiveness of theselected features, and moreover one of our goals is to understand feature relevancefor its own sake. We analyze relevance to low-level segmentation quantitativelyand qualitatively, and the results are consistent. Then we discuss the reason oftheir (ir)relevance and limitations.4.7.1 Quantitative AnalysisThe linear coefficients in the SVM model, listed in Table 4.2, are usually inter-preted as feature importance. The sign of a coefficient indicates whether correla-tion between the feature value and probability of being a boundary is positive ornegative: only duration has a positive correlation, e.g., longer duration indicateshigher probability that the event is at a chunk boundary. The absolute value ofthe coefficient indicates the degree of importance, and we bin these quantities intothree categories: commands as most important, layers and duration as important,and the two image-related features have almost no effect.106(b) Layer similarity(a) Command similarity (e) Image diff(d) Working region overlap(c) Duration#non-boundary events#boundary eventsDifferent distributionsDifferent distributions Different distributions Similar distributionsSimilar distributionsHuge diff.BalancedMany identical cmdBigger avg.Smaller avg.Few identical cmdBigger avg.& smaller var.Smaller avg.& bigger var.Figure 4.2: Distributions of the five feature values of non-boundary (top) andboundary (bottom) events. The X axis represents the value of each fea-ture (bin size = 0.05), and the Y axis the counts. Note that the Y scalesare 10x different between top and bottom because there are significantlymore non-boundary events. We annotate the differences or similaritiesbetween the two distributions for each feature.Feature Coefficient Relevance categoryCommand similarity -2.57 Most importantLayer similarity -1.74 ImportantDuration 1.55Working region overlap -0.18 No effectImage diff -0.07Table 4.2: Corresponding coefficients to features in SVM model.4.7.2 Qualitative AnalysisWe also conduct a qualitative analysis of feature relevance. We visualize the distri-butions of feature values using histograms and compare the distributions betweennon-boundary events and boundary events to see if the difference is salient. Fig-ure 4.2 shows these distributions, with annotations including our assessment ofwhen they differ and thus the feature is relevant. We can see clearly that layersimilarity, command similarity, and duration are relevant, while image diff andworking region overlap are not.1074.7.3 Result Interpretation and LimitationsBelow we discuss the potential causes for feature (ir)relevance and the limitationsof our feature choices.Command SimilarityIt has been shown in the CommandSpace system [3] that the command usage se-mantics can be learned from large-scale of command logs, analogous to text data,and that the usage semantics are relevant to user goals and subtasks.However, we see that except for the maximum similarity when commands are iden-tical, the variance in the remaining values is small; most of the range between 0and 1 is not exploited. A second problem is that a single command in PS suchas BrushTool can support multiple different functions, which is analogous tothe polysemy problem in natural language. Future work could experiment with acontext-aware learning model instead of word2vec, or augment the computation byconsidering the command parameters (such as brush size) that are not captured byour current PS instrumentation plugin. The third potential problem is the informa-tion loss in the computation of cosine similarity between the command vectors.Layer SimilarityOur observations during data collection were that participants keep the layer panelvisible at almost all times as they refer to layers frequently, and are able to explainthe semantics of each layer explicitly. Different subtasks in a task are often per-formed on different layers, particularly when the artwork consists of objects fromdifferent source images.However, we heuristically assigned discrete values as layer similarity for our 5chosen cases, and these values have not been strictly verified. Also, our currentapproach does not fully exploit all information available in the layer hierarchy.108We have not found a good way to use the logged information about layer order,which might have further distinguishing power. Also, some layer attributes are notlogged, such as layer mask, blending options, and transparency. Exploration ofthese issues is an important direction of future work.DurationWe observed during the data collection study that pauses do indicate thinking timewhen switching to a new subtask and when comparing current results to a referenceimage. We also saw clear evidence of short duration values indicating continuousoperations when they are actively involved with a subtask, especially when theyhave a mental map of how the subtasks should be fulfilled.However, we think that the duration from our data collection in a lab setting maybe artificially clean, since participants are doing an assigned task under observa-tion. We conjecture that in real-world usage, pauses may often arise from externalinterruption rather than an internal switch of subtasks. Therefore, we conclude thatthe relevance of duration is debatable.Working Region OverlapOur results contradict previous work from Chen et al. [26] that uses working regionoverlap. We conjecture that the signal from pixels of image content is dominatedby the one from layers, which serve as an abstraction of visual object structure,for identifying subtasks. Another possible reason for our opposite finding could bethe information loss with pixel counts: we use a single number instead of a vectorrepresentation in Chen’s paper.109Image DiffWe conclude that the image diff is too noisy for segmentation. A subtask switch(boundary event) does not always come with large change in pixels; for example,adding text in a small font only changes a small number of pixels. Similarly, largechanges in pixels sometimes can happen within a chunk, for example moving alarge object. We see from the distributions that most events change very few pixels.4.8 Results: Low-level SegmentationWith reliable ground-truth labels for the low-level segmentation, we measure modelperformance on the test set, conduct an preliminary analysis on the variance acrossdifferent tasks, and compare our performance metrics with a previous paper.4.8.1 Performance MetricsWe train the SVM model on the training set, tune hyperparameters on the valida-tion set (window size k = 1, threshold t = 0.24), and measure performance on thetest set, which contains 1842 events. The F2 score, which weighs recall twice asmuch as precision, is 0.76, with a decent recall 0.87 but modest precision 0.51.The numbers are almost identical to the performance metrics on the validation set(provided in Supplemental Section C.3), suggesting that the model does not overfit.4.8.2 Task VarianceWe analyze the variance of model performance across different types of task (postercreation, portrait retouching, and special effect creation) to verify our claim that ourmodel generalizes.We categorize all 16 collected sessions by task type, and conduct two analyses tomeasure model performance. In Analysis 1, we use the chosen threshold t = 0.24110TimeTune colors Make lighting around charactersAdd more lightingCheck resultsPosition images Add text Make watersplashProbability(b) A poster session (378 events)HumanComputed (threshold)Computed (agglomerative)ProbabilityTimeForehead and right eyeTry healing brush Fix right face and ear Fix teeth and lips Fix left backgroundOpen image(a) A retouching session (180 events)Hide part of imagesFigure 4.3: Session visualizations. (a) Retouching. (b) Poster. From bot-tom to top in each session: a scatterplot showing the probability scoreof events in sequential order computed by the SVM model; a multi-level computed chunks using multi-tier thresholds (red); the multi-levelcomputed chunks using the agglomerative algorithm (blue); the multi-level human labels (grey); text description of high-level chunks withhuman-assigned labels. Examples of over-segmentation and missingboundaries are highlighted with dotted rectangles. Thresholds (corre-sponding to red chunks from low to high) are 0.24,0.29 for retouchingand 0.24,0.64 for poster. Distances to root, corresponding to high-levelblue chunks from low to high, are 8,5,3 for retouching and 6 for poster.Figure 4.1 illustrates an expansion of the “add text” chunk in Poster.for all three tasks, and compute the F2, recall and precision scores for each task.In Analysis 2, we tune the threshold hyperparameter for each task. We observedthat different participants organize their sessions at different levels of granularity,which is particularly obvious across different task types, so we need to understandhow the threshold affects performance for different task types.We present the performance metrics in Table 4.3. For both analyses, the differenceof F2 scores between any two task types is low (±0.08). Within each task type,the difference of F2 scores across analysis is also low (±0.03). In Analysis 2,where the threshold is optimized for each task type, we notice that the thresholdfor portrait retouching (0.17) is different than the other two (0.27,0.25), confirmingour observation about the influence of task type on granularity. In conclusion, theSVM model can generalize to the three task types, while further research is needed111to address the nuances in granularity.Task type PostercreationPortraitretouchingSpecial effectcreationData scale #sessions 8 6 2#events 3156 1876 670Analysis 1F2 0.71 0.77 0.79recall 0.85 0.82 0.97precision 0.43 0.63 0.45threshold 0.24Analysis 2F2 0.73 0.80 0.79recall 0.82 0.92 0.96precision 0.49 0.52 0.47threshold 0.27 0.17 0.25Table 4.3: The performance metrics of sessions in the three task types.4.8.3 Comparison to Previous WorkThe closest work to our model is Chen et al.’s method [26]. They also trained anSVM model, targeting smart undo as a primary use case. We deemed it infeasibleto directly compare with their model fairly. It does not make sense to apply theirtrained model on our dataset, because their training dataset (and hence their model)only has portrait retouching data, whereas we covered three tasks in total. Also,a difference in data collection (they have data on brushing area but we do not)induces technical difficulties in re-implementing their model for retraining on ourdataset.Chen et al. chose window size k = 5 for their model and achieved a similar F2score (0.74) with a lower recall (about 0.7) but much higher precision (0.96) ontheir dataset with portrait editing task only. That outcome is surprising, since theydid make similar arguments to our own for valuing recall over precision. We alsonote that our segmentation problem is considerably harder: we deal with threetypes of tasks instead of portrait retouching only, and we did not prescribe a set ofpre-defined subtasks for the participants like they did. We argue that our resultsare reasonable, since we tackle a harder problem. In conclusion, we have achieved112reasonable results for a more difficult problem than theirs.4.9 Results: Multi-level SegmentationWe present the visualizations of two exemplar sessions to show how our computedlow and high-level segmentation aligns with human labels, which are subjectivejudgements from the authors. Here we show two from five sessions from the testset: one portrait retouching session (S10), and one poster creation session (S15).They are representative sessions in terms of both general model performance andfailure cases; Supplemental Section C.4.2 contains the analysis of the other threetest sessions. We also provide high-resolution screenshots for all sessions in sup-plemental materials. We first provide an overview of participant’s workflow (high-level segmentation in the human labels), and then inspect how our computed resultsalign with human labels.4.9.1 Exemplar Retouching SessionThe goal of the first exemplar session (Figure 4.3(a)), with 180 events, is to restorean old photo (a child’s portrait) with cracks and stains on it. At the bottom of thefigure there is a scatterplot showing the probability scores of events being predictedas a boundary in time order, with red dotted lines to show threshold values. Abovethe scatterplot are three versions of segmentation (computed chunks by the multi-tier thresholds SVM in red, computed chunks by the agglomerative algorithm inblue, and human labels in gray). Each colored rectangle represents a chunk, and awhite space (gap) between two rectangles represents a chunk boundary.From the grey human labels, we can see that the participant is working on differentregions in the image roughly one at a time: forehead and right eye, right face andear, teeth and lips, and finally the background. Note that the regions (face, eye,background, etc) are only meaningful for a human user. However, there are alsooutliers subtasks that do not follow this order: e.g, trying out a different commandHealing Brush to fix the photo. In the last step to fix the left background, they113used a more complicated procedure than previous chunks, resulting in a two-levelsegmentation.To compare different versions of segmentation, we can inspect the alignment ofgaps (boundaries) vertically. In general, the gaps in human labels and those in thecomputed results align well. One obvious missing boundary happens around themiddle between “fix right face and ear” and “fix teeth and lips” (highlighted witha dotted rectangle), due to the high similarities between these events (same layerand command, duration small, and little changes in image content).4.9.2 Exemplar Poster SessionThe goal of the second exemplar session (Figure 4.3(b)), which is longer with 378events, is to create a poster featuring two football players. From the human labelsin the figure, we can see that the chunks consist of variant numbers of events, andthe hierarchy of chunks is unbalanced; that is, different high-level chunks havedifferent subtree depths below them. From the high-level human labels and anexpanded part in Figure 4.1, we can see that the participant did multiple trials forsome subtasks, making some mistakes and corrections.For the low-level chunks between human labels and computed results, most gapsare aligned, indicating a good match between the two. There are very few missedboundaries (highlighted): one example happens when the participant finished mak-ing a layer mask and then started adjusting opacity, where they stayed at the samelayer and used commands that are semantically close (Brush Tool vs. MasterOpacity Change). There is some but not excessive over-segmentation: theconsecutive gaps that separates individual events into chunks. For example, it hap-pens in a human-labeled chunk “tune colors of front player”: he actually performedtwo scaling operations of the image in between color tuning operations, resulting inlow similarity between consecutive commands (Free Transform vs. CameraRaw Filter) and thus over-segmentation in the computed results. We will dis-cuss this particular user behaviour in the next section.114For the high-level chunks, although both computed results aggregate most of thesingle-event over-segmentation, neither has a good match with the human labels:about half of all high-level boundaries are missed. As we have already noted, we donot consider that data to represent ground truth; the ambiguity about what granular-ity to code for led to the two human coders had even higher levels of discrepancieswith each other in the sessions they both coded, so that data was not used in the de-velopment of the high-level model. Nevertheless, we inspect the probability scoresof the missing boundaries in the scatterplot, and find that many of them have a highprobability score, but it is hard to draw a horizontal straight line that could serve asa threshold to separate most of them out. This finding indicates that there is somevalue in reusing the results of SVM model in the high-level segmentation, but itneeds further research to improve the performance.4.10 DiscussionWe discuss the strengths and weaknesses of our segmentation model, of the datasetwe collected compared to previously reported ones, and the nuanced characteristicsof real-world user behaviour in image editing revealed by our study.4.10.1 Strengths and Limitations of ModelOur multi-level segmentation model achieves fast computation, due to its simplic-ity and the small number of parameters. The low-level SVM model takes less than1 second to train and validate, and prediction is immediate. The high-level segmen-tation takes at most a few seconds to handle an 1K-event session with a JavaScriptimplementation. This speed makes the model suitable for streaming log data, thehallmark of the downstream interactive application use cases.It is hard to thoroughly evaluate our approach to higher-level segmentation sincewe do not have appropriate labelled data for it, due to the subjectivity of the granu-larity assessment. Ground truth labels and a more rigorous evaluation would allowfurther progress. The complex user behaviours described below do leave consider-115able room for improvement in model performance. Our precision is currently only0.5; although we did prioritize recall, both measures need to be improved.4.10.2 Dataset ComparisonWe now compare our dataset to two other datasets from previous work in moredepth. In the nonlinear revision control paper by Chen et al. [25], most of theirsessions are less realistic than ours, where there is little complex user behavioursuch as continuous refinement, mistakes and corrections, and experimentation. Thescale of their data is also smaller in terms of number of sessions and number ofevents per session: a few sessions with less than a dozen events, and a few sessionsof sketching with hundreds of events.The adaptive history paper by Chen et al. [26] targets only the portrait retouchingtask, with predefined subtasks chosen from a list. The restricted variance in sub-tasks makes their model hard to generalize to other task types, or even retouchingthat does not follow the predefined subtasks.A major difference between our dataset and previous ones is that we capture layerinformation. However, there is still some available data that our plugin architec-ture does not capture, such as command parameters (e.g., brush size) and layerattributes (e.g., transparency). Future work could investigate whether it containsuseful signal to exploit for segmentation.4.10.3 Real-world User BehaviourOur data collection study documented that the real-world usage patterns of cre-ativity support tools are usually messy and complex: users rarely perform preciseand effective actions from the beginning of a session to the end. We characterizeseveral of the complex user behaviours that we observed.First, users make mistakes. Common examples of mistakes are misuse of com-mands, undesired command parameters, or operations on a wrong layer. The user116may or may not correct the mistakes, and corrections may happen immediately af-ter a mistake or later. This situation is challenging as the user usually does not con-sider these corrective actions as a change of subtask, but the model would only seethe a sudden change of events and thus predict them as boundaries. Second, usersinterleave different subtasks together. This situation may happen when the user hasno clue of how to perform a subtask, when the user realizes there is another subtaskthat is more urgent than the current one, or when the user finds a previously donesubtask unsatisfying. This situation is challenging for segmentation as the user’smental model is less organized. Third, users experiment with the tool to achievea desired outcome through trial and error. This situation is common in users withlower levels of expertise, and the resulting session usually contains many undo ac-tions and layer deletions. The challenge in automatically segmenting this kind ofuser behaviour is that the difference between trials can be subtle and hard to de-tect. Fourth, the boundaries between subtasks could be intrinsically fuzzy. Beforeswitching to a new subtask, users often check if results are satisfactory by togglinglayer visibility, or make small adjustments of different components in the artworkwhile thinking about what to do next. These actions result in fuzzy boundariesbetween subtasks that span multiple events, but our model assumes single-eventboundaries.Our data collection study succeeded in revealing a picture of user behaviour that isfar more complex than what previous work had captured, where less attention waspaid to ecological validity. Our model makes a significant step towards addressingthis complexity, but there are many interesting problems to address in future work.4.11 SummaryWe developed a multi-level segmentation model for real-world image editing logsthat works for three image editing tasks. We present evidence for what features arerelevant and irrelevant for the segmentation. Results show that command and layersimilarity are highly relevant, image-related features are not useful despite claimsin previous work, and duration is debatable due to the limitation in data collection.117We also present quantitative and qualitative evaluation of our segmentation model,and show that it performs reasonably well for the challenging problem of segment-ing realistic logs that capture mistakes, experiments, and subtask switching.118Chapter 5CorGIE: Visualizing GraphNeural Network by ExploringCorrespondences between aGraph and Its Embedding1 Graph neural networks (GNNs) are a class of powerful machine learning toolsthat model node relations for making predictions of nodes or links. GNN develop-ers rely on quantitative metrics of the predictions to evaluate a GNN, but similarto many other neural networks, it is difficult for them to understand if the GNNtruly learns characteristics of a graph as expected. We propose an approach to cor-responding an input graph to its node embedding (also known as latent space), acommon component of GNNs that is later used for prediction. We abstract the dataand tasks, and develop an interactive multi-view interface called CorGIE to instan-tiate the abstraction. As the key function in CorGIE, we propose the K-hop graph1This chapter is a slightly modified version of the following publication: Z. Liu, Y. Wang, J.Bernard, and T. Munzner. Visualizing Graph Neural Network with CorGIE: Corresponding a Graphto Its Embedding. IEEE Transactions on Visualization and Computer Graphics (TVCG). Underreview [113].119layout to show topological neighbors in hops and their clustering structure. Toevaluate the functionality and usability of CorGIE, we present how to use CorGIEin two usage scenarios, and conduct a case study with two GNN experts.5.1 MotivationGraph neural networks (GNNs) are machine learning (ML) models that have re-ceived substantial recent attention due to their ability to deal with abstract conceptslike relationships and interactions. GNN models are widely used in downstreamML applications such as fraud detection, traffic modeling, and product recommen-dation, in addition to the classic ML application domains of natural language pro-cessing and computer vision.During training, a GNN model takes in a node-link graph and as output generatesa node embedding; that is, a representation of all discrete nodes in the graph asfixed-dimensional vectors in a continuous latent space. Proximities between em-bedded nodes in the latent space represent meaningful similarities between nodesin the input graph learned during the training. The node embedding leverages boththe topology – connectivity between neighboring nodes – and node features – in-formation associated with each node – of the input graph. (Information associatedwith nodes, typically called node features in the ML literature, is sometimes callednode attributes in the visualization literature.) The node embedding is then avail-able for feeding into downstream ML applications, for example to make predictionsabout nodes and links. Figure 5.1a summarizes the standard GNN pipeline.To achieve optimal performance, model developers must evaluate whether the train-ing has succeeded in producing a GNN model that has in fact learned the importantcharacteristics from the input graph. Model developers often need to determine thestopping criteria for training a GNN model; in other words, to answer the question“are we there yet?” when training models or tuning hyperparameters. In additionto global questions such as when to stop training and tuning, model developersneed to debug and trace errors and sub-optimal states in the trained model. Forexample, they may want to identify and understand the groups of nodes that suffer120Figure 5.1: CorGIE motivation. (a) Standard pipeline for graph neural net-work (GNN) training and usage. (b) Previous approaches to evaluateGNNs mostly focus on the predictions [40, 167, 188], or to separatelyinspect the node embedding; (c) our approach with CorGIE is to directlysupport exploration of correspondences between an input graph and itsnode embedding.from mis-classification the most. In some cases, they also need to compare resultsfrom different model architectures or hyperparameters to choose the best ones forproduction use.The most prevalent current approach for model evaluation is to compute quantita-tive metrics based on the downstream ML applications, such as precision and F1for node classification and Hit@k for link prediction, which are used for cross-validation during training [40]. Some researchers have proposed algorithms tocompute explanations for prediction results in terms of influential nodes, or whatfeatures are important in the determination of a specific node [188]. Another com-mon approach is to conduct qualitative sanity checks, such as manual inspection ofnode or link instances, for example by visualizing with multi-class scatterplots ofthe dimensionally reduced node embedding [167]. Figure 5.1b summarizes theseprevious approaches to evaluation.However, these existing evaluation approaches fall short because they do not suffi-ciently support developers in directly understanding the correspondences betweeninput graphs and their node embeddings. Developers must contend with a GNNmodel as a black box, because it is hard to check whether all important informationis codified in the embedding. Even if they find errors in the downstream predictednode labels, it is hard to even trace these errors back to the upstream GNNs, or torefine the GNNs to avoid them.121Our key idea is to aid GNN evaluation by surfacing the correspondences, or lackthereof, between an input graph and its node embedding produced by a GNN. Fig-ure 5.1c illustrates this approach. Geometrically speaking, a GNN should learnto place graph nodes with similar neighbors and feature values in a nearby loca-tion in the high dimensional latent space. Therefore, we can evaluate a GNN byverifying how well it preserves the characteristics of the input graph in the nodeembedding. For instance, one can explore how a cluster in the node embeddingcorresponds to the similarity of topology and/or node features in the input graph.Conversely, one can explore how close two nodes sharing many topological neigh-bors are represented in the node embedding. To achieve these goals, we presentthree contributions.Our first contribution, presented in Section 5.3.3, is a data and task abstraction forvisually exploring correspondences between an input graph and the derived nodeembedding, to understand if a GNN model has learned important characteristicsof the input graph to construct the embedding. The abstraction is the result ofextensive iterative refinement.To instantiate the abstraction, our second contribution is the design and implemen-tation of an interactive multi-view interface, CorGIE (Corresponding a Graph toIts Embedding). CorGIE is agnostic to specific GNN models, with a “grey-box”approach that ignores the internal model details like neurons and weights and onlyassumes that the GNN aggregates information from neighboring nodes. We de-scribe the design of CorGIE in Section 5.4.To fulfill the important task of showing topological neighbors for nodes of interest,our third contribution is a new visualization technique, the K-hop graph layout.The K-hop layout reveals node neighbors hop by hop, similarly to how a GNN ag-gregates information, and reveals the clustering structures within the node neigh-borhoods. Previous graph representation and layout algorithms do not suffice forthis scenario. We present the details of this technique in Section 5.5.We validate our claims in two ways. We provide two usage scenarios that walkthrough how the CorGIE design supports the task and data abstractions in detail.122We also discuss the results of a study with two GNN experts that provide prelimi-nary evidence of the utility and usability of CorGIE.5.2 BackgroundWe first provide the minimum necessary technical background on GNNs to makethis work self-contained. We then describe two concrete usage scenarios that mo-tivate CorGIE. We defer our discussion of the related work to Section 5.7.5.2.1 Graph neural networksThe widespread encoder-decoder perspective is a useful starting point for under-standing general graph neural networks [23]. The encoder combines topology, andoptionally node features, to produce a node embedding. The decoder takes the nodeembedding to make inferences in downstream ML applications. In the GNN lit-erature, the node embedding is an intermediate representation within the pipelinethat is not necessarily of direct interest outside it. Our perspective is different:we emphasize inspection of the learned embedding as the key to understanding.Moreover, the downstream ML applications are usually considered as an intrinsiccomponent of the GNN pipeline. In contrast, we separate the downstream ML ap-plications from the preceding pipeline so that CorGIE can support multiple down-stream applications. CorGIE’s agnostic approach supports both node classificationthat predicts properties of single nodes, as used for movie tagging or documenttopic labeling), and link prediction that predicts properties of node pairs, as usedfor movie recommendation.A GNN is usually trained using either node labels or node connections dependingon the application. For node labels, training can be supervised by separating thenodes into training, validation, and test sets if all node labels are available, or semi-supervised if only a small number of labels are available. For node connections,training is unsupervised. Two nodes connected by an edge are considered a positivenode pair, whereas a negative node pair has no edge between them; both positive123and negative pairs are sampled during training.A GNN learns from the input graph with neighborhood aggregation (also knownas message passing), where the embedding of a node is computed by traversingto its topological neighbors to collect and aggregate node features from them. AGNN has multiple layers, allowing aggregation to take place hop by hop: each nodeaggregates the output of its neighbors from one layer to the next. The input layertakes the node features, the last layer outputs a high dimensional node embedding,and the k-th layer aggregates from the k-th hop neighbor. The number of layers Kis usually small, with K ≤ 3, indicating that it is a shallow network. GNNs differ inthe aggregation functions, ranging from a simple mean to complicated non-linearfunctions [133]. Our approach takes K as a parameter, but is fully agnostic to thedetails of the aggregation function.5.2.2 Motivating scenariosIn this chapter, we refer to the target users of CorGIE as GNN developers, rangingfrom users who consume pre-trained GNNs to experienced researchers who canpropose new model architectures.Before diving into the details of our approach and tool, we present two represen-tative scenarios of a hypothetical GNN developer, Alice. For each, we introducea dataset, which consists of an input graph and the embedding constructed by thetrained GNN model, and a set of visualization tasks that apply to it. We provideextensive additional examples of concrete tasks in Supplemental Section D.1.Movie: recommendationAlice has a bipartite movie-user graph2, where an edge indicates that a user haswatched or rated a movie. Node features in this graph depend on the node type:the movie node features are budget, popularity, vote average, #cast, #crew; the2Extracted from a Kaggle dataset: https://www.kaggle.com/rounakbanik/the-movies-dataset124user node features are vote average and #votes. The goal is to recommend moviesto users; that is, the downstream ML application is link prediction. Alice trains aGCN (graph convolutional network, a well-known GNN model [93]) with the nodeconnections and node features in the bipartite graph, to produce a 16-dimensionalnode embedding. She chooses the hyperparameter setting of 16 based on her pre-vious experience.To evaluate the training results, Alice wants to understand what the GNN haslearned. Besides the typical quantitative metrics like accuracy, she wants to ex-amine:1. The overall clustering structure of the node embedding, to answer questionslike Are the movies separated from the users? Are there clusters of users?2. Within a user cluster, she is interested in why nodes are grouped together: Dothey watch similar movies; that is, do the movies share first hop neighbors?Do they rate movies with similar scores? Do movies they have watched havesimilar budgets? Similarity in topology and/or feature values indicate theGNN has done a good job in grouping these users.3. Between two user clusters, she wants to see their differences: Do they watchdifferent sets of movies? Do they rate movies differently? Significant differ-ences indicate that the GNN has done a good job in differentiating the twouser groups.4. Instance-level inspection of movie recommendations is Alice’s next under-taking. She wants to spot-check the list of recommended movies for someuser, for example a recommendation of Interstellar for a viewer who hasseen The Matrix and Guardians of the Galaxy. She wants to understand whythe GNN has generated them: Are recommendations determined by sharedviewing patterns, as with collaborative filtering [149]? Are they motivatedby similar node features? Do they capture topic similarity, like science fic-tion?125Cora: paper topic labelingThe Cora dataset is commonly used as a benchmark in the GNN literature [116].It is an academic paper citation graph, where the node features are binary vectorsindicating what words from a 1433 word dictionary exist in the paper representedby a node. The downstream ML application is to classify the papers according to aset of given topics (classes). Alice trains a GAT (graph attention network, anotherwell-known GNN model [175]) with the node labels in the Cora graph, and pro-duces a 7-dimension node embedding because there are 7 classes of paper topics.She thus converts the values for each dimension to probabilities of belonging tothat class, which is a common practice in the GNN literature.Following a similar analysis process for the movie scenario, Alice first exploresthe clustering structure by visualizing shared topological neighbors and similarityof node features from an overview level. Then, she tries to understand the patternof misclassified nodes. She selects groups of these nodes that are either located inthe same area of latent space or have the same predicted label; that is, the sameerror. She inspects the shared neighbors and the words within groups to determinewhether there are commonalities to blame for the misclassification.5.3 Data and task abstractionAfter considering many examples of concrete tasks faced by GNN developers, wegeneralized them to understand the problem at a higher level. We formed the dataand task abstractions iteratively, over several months, through four parallel thrusts:review of the GNN and visual analytics literatures, informal interviews of fourGNN developers within three different organizations, development and use of pro-totypes of the CorGIE interface, and our reflection.1265.3.1 OverviewWe generalize the many concerns faced by GNN developers as they train theirmodels into three driving questions. For memorability, we frame them as questionsthat might be asked on a road trip:• Q1: are we there yet: should we train or tune more?• Q2: are we lost: does it behave as we expect?• Q3: what’s that: what does this exemplar do?The first big-picture question concerns the potential need to keep training the modelfor more epochs, tuning the hyperparameters, or reconsider how they construct thegraph. It would be answered by understanding whether the results are satisfy-ing. The second question, also big-picture, concerns developers’ understandingand trust in whether the GNN learns inherent characteristics from the input graph;that is, does it behave as they expect. These two big-picture questions are partic-ularly difficult to answer with previous evaluation approaches, which rely heavilyon quantitative metrics for the downstream ML application; we aim to provide vi-sual and qualitative evidence and insights to answer these two questions in a moredetailed and thorough way. The third question is more specific, concerning inves-tigating exemplar instances of nodes or links. Some previous approaches like theGNNExplainer [188] do support such single-instance inspections, but we seek tomake them easier and faster.We posit that finding the correspondences among 1) the node positions and clus-tering structure in the latent space, 2) the topological neighbors in the input graph,and 3) the node feature distributions in the input graph, can shed light on thesequestions. We thus discuss three abstract data spaces, as illustrated in Figure 5.2:latent space, topology space, and feature space.127Figure 5.2: Data and task abstraction. Users can specify items within all threedata spaces, and investigate correspondences between the latent spaceand topology or feature spaces. The targets of activity in each space arelisted below it.5.3.2 Data Space AbstractionOur data abstraction features three sub-spaces: the latent space, the topology space,and the feature space. We compute distance metrics in each of these spaces. Wealso incorporate prediction results if available. We follow the order of views in theCorGIE interface to introduce the three spaces.Latent spaceThe latent space is where the node embedding learned by the GNN lives. Eachnode in the input graph has its own high-dimensional vector representation. Thenumber of dimensions is usually less than a thousand, although this number doesnot affect the complexity of the tasks and thus is not a constraint for the CorGIEdesign. CorGIE assumes K≤3, where K is the number of layers in the network.In the latent space, the absolute vector values are usually not directly interpretable.We are interested in the vector similarities and particularly, whether there is a clus-tering structure among the node vectors. For example, in the Cora scenario, arelevant question is whether there are exactly seven clusters of papers in the latentspace, which would match the seven topics in the ground truth.Some GNNs can generate edge embeddings or graph embeddings, but the scope ofthis work is node embedding and others are left for future work.128Topology spaceThe topology space consists of the topological connectivity of one single inputgraph. The input graph could be either homogeneous, with a single node type, orheterogeneous, with multiple node types. For example, there are two node types inthe Movie scenario: user and movie. The design target for CorGIE is to handle up to6 node types, which covers many useful cases for heterogeneous graphs [185]. Forgraph size, CorGIE will accommodate up to 20K nodes, to handle many prevalentbenchmark datasets in the GNN literature [40].In the topology space, we are interested in the topological neighborhoods and theconnections (edges) between node pairs. GNNs aggregate information along edgesto compute the node embedding. We thus derive each node’s neighbor set contain-ing its k-hop neighbors; that is, the set of all of the nodes reachable from a node vwithin k topological hops.We limit our scope to one input graph, leaving the multiple graph case to futurework. We also leave the support of multiple edge types for the future.Feature spaceThe feature space consists of all features (often referred to as attributes in the vi-sualization literature) of the graph nodes. We distinguish between dense and sparsefeatures. Dense features are numeric or boolean features that can be collected andunderstood independently, with interpretable semantic meanings for each one, suchas budget or average vote from the Movie scenario.Sparse features are usually collected together and share a combined semantics,where interpretation typically takes place across the entire set considered as awhole. Typically they are stored as one-hot encoded categorical features like citynames or the dictionary of words from the Cora scenario, where the collection isrepresented by hundreds or thousands of bit vectors.In the feature space, we are interested in the distributions of feature values, which129involves deriving aggregations. We deep-dive into the details of feature aggregationin Section 5.4.2.The design target of CorGIE is to handle up to a dozen dense features and up to 2Ksparse features. Note that we deem edge or graph features out of the scope of thiswork.Distance metricsTo quantify correspondences between spaces, we introduce a distance metric be-tween any pair of nodes within each space, so that scalar distance values acrossdifferent spaces can be compared. In the latent space, we derive the cosine dis-tance of the embedding vectors, which is commonly used in the downstream MLapplications. In the topology space, we derive a topologically oriented distancemeasure, the inverse Jaccard index for the full set of k-hop neighbors of each pairof nodes in the input graph: 1.0−|A∩B|/|A∪B| where A and B are the ≤ k-hopneighbors of the two nodes. It is intuitive and easy to compute, but the flattenedneighbor sets are sometimes an oversimplification because neighbors at differenthops cannot be distinguished from each other. In the feature space, we derive theEuclidean distance of feature vectors (scaled linearly to [0,1]), which is also famil-iar to GNN developers.Prediction results (optional)After training is complete, the GNN can be used for prediction in the downstreamML applications (Figure 5.1). We leverage prediction data by comparing the pre-dicted results to ground truth when available, to derive true/false negatives andpositives. For node classification applications, we compare the true and predictednode labels to derive a label correctness value (correct vs wrong) for each node.For link prediction applications, we obtain the predicted positive and negative nodepairs, and compare these to the edges of the input graph. We derive labels for twointeresting categories of node pairs: false positives, where unconnected node pairs130are predicted as connected (recommended pairs); and false negatives which arenode pairs connected with an edge but predicted as disconnected. We also provideoptions to derive the true positives and negatives if desired. Although the funda-mental data and task abstractions do not rely on these prediction results, they canbe used for filtering.5.3.3 Task abstractionWe identify a unifying task abstraction built around an iterative cycle of two phases,as shown in Figure 5.2: specify items in any of the three data spaces, and corre-spond items between either the latent space and topology space, or the latent spaceand feature space. Notably, our task abstraction does not entail finding correspon-dences between the topology and feature spaces, because that exploration wouldfocus only on characteristics within the input graph and would not provide directinsight into GNN behaviors.The specify step allows users to indicate nodes of interest in any of the spaces,based on the targets visible within each of them. In the latent space, they couldspecify nodes based on relative positions with respect to any visible cluster struc-ture, or with respect to the latent distance distribution. In the topology space, theycould specify nodes based on neighbors, cliques, or the topological distance distri-bution. In the feature space, they could specify nodes of specific feature values. Ifavailable, they can also specify a collection of nodes based on the prediction resultssuch as label correctness or newly predicted links between nodes. Below, we referto a group of specified nodes as a focal group, which has a double meaning: it isthe mental focus within a user’s thoughts at the current step of their data analysisprocess, and it is the focus of actions and computation within the CorGIE interface.The correspond step then allows users to explore to what extent the characteristicsof the specified nodes in one space correspond to those in the other spaces. Forexample, for a tight cluster in the latent space, users could verify whether the nodesin the cluster share many neighbors in the topology space and have similar featuresin the feature space. If so, the GNN has successfully learned how to group these131nodes together. The extent to which these spaces line up with each other is aqualitative judgement call, not a precisely computable metric. The intent of thisabstraction is to support GNN developers in obtaining higher confidence and trustin their models than would be possible from purely quantitative summary metricssuch as accuracy.The abstraction supports the process of specify and correspond as an iterative loop,where exploring correspondences can trigger the user’s interest in specifying othersets of nodes. For instance, the user might start by specifying a cluster in the latentspace, and after checking its correspondences to the topology space would becomecurious to then specify a sub-cluster within it to explore further connections. Mul-tiple cycles of refining and changing the sets of specified nodes based on the visualcorrespondences allow users to connect the dots and find answers all the three ofthe driving questions.This task abstraction encompasses all three of our driving questions. It was devel-oped through considering the commonalities between all of the concrete tasks thatwe analyzed, not only those presented in the motivating scenarios (Section 5.2.2)but also many additional concrete examples summarized in Supplemental Sec-tion D.1. During the abstraction process, we initially considered a much morecomplex set of targets and actions, such as analyzing outliers with respect to acluster, or analyzing one node vs. all other nodes, or whether verifying differencesshould be treated differently than verifying similarities. In the end, we realizedthat almost all of these questions could be framed much more simply, in terms ofspecifying a very small number of groups as a target and then checking on cor-respondences. It was rare to require even three groups: one or two groups weresufficient for almost all analysis questions. Our final design is optimized for up totwo groups, although it is possible to specify several more to handle edge cases.5.4 CorGIE designBased on the data space and task abstraction, we design CorGIE, a multi-view toolthat reveals the correspondence between an input graph and its embedding. We132Figure 5.3: Full screenshot of CorGIE interface on the Movie dataset, withtwo focal groups of user nodes. The views, with names shown in greyboxes, are laid out in four major areas on the screen, shown in blue: (a)the LATENT SPACE VIEW is for 2D node positions in latent space; (c)the NODE FEATURES VIEW is for feature distribution of all and focalnodes; (f & g) the TOPOLOGY VIEWS show local topological neighborsand global topology; (b) the LATENT NEIGHBOR BLOCKS VIEW and the(e) DISTANCE COMPARISON VIEW connect different spaces. The fiftharea is for toggles and menus: (d) the SETTINGS VIEW.describe the design of each view, then the view coordination between them, andfinally the implementation.5.4.1 OverviewFigure 5.3 shows an overview of the CorGIE interface on the movie scenario, whichhas seven main views. The LATENT SPACE VIEW is on the top left. Two views atthe bottom right show the topology space, the GLOBAL TOPOLOGY VIEW and anovel graph layout for a user-chosen subset of nodes in the K-HOP TOPOLOGYVIEW. The feature space is showcased at the top right in NODE FEATURES VIEW.To enable direct and quick correspondence exploration, we have two views in themiddle that can incorporate two or three spaces simultaneously. We introduce theLATENT NEIGHBOR BLOCKS VIEW to connect the latent and topology space, and133the DISTANCE COMPARISON VIEW to connect all three spaces. Finally, there is aSETTINGS view at the bottom left.To support the two steps of specify and correspond in the task abstraction, theinteraction design of CorGIE involves three actions: hover, select, and focus.Node specification occurs through a focus action, and correspondence explorationis triggered through hover and select actions. The focus action creates one or a fewspecial node groups of interest, denoted by foc-0, foc-1, and so on. When hoveredor selected, nodes are then highlighted across views. When nodes are focused,there will be a novel graph layout showing their k-hop topological neighbors.5.4.2 View designWe introduce the visual encoding and design rationale for each view.Latent spaceThe LATENT SPACE VIEW on the upper left of the interface (Figure 5.3a, Fig-ure 5.6a, Figure 5.8a) shows the clustering structure and the relative positions ofnodes in the latent space. We use UMAP [119] to project the latent space to twodimensions and plot the dimensionally reduced nodes as a scatterplot. GNN devel-opers are typically familiar with dimensionality reduction and are thus aware thatloss of information is inevitable in this process.CorGIE supports color coding nodes by their UMAP positions in all views, andin that mode a rainbow background for the entire 2D latent space view providesa salient reference for positions within the latent space, as shown in Figure 5.3.We chose a highly saturated CIELAB colormap (x axis for the variable a, and yaxis for b) that has substantial color exploitation and strong performance for thesynoptic localization task of detecting groups of nodes [15]. The combination ofdimensionality reduction and this colormap shows latent space distances, so thatsimilar nodes receive similar colors whereas dissimilar nodes receive dissimilar134colors.Nodes are shape-coded for heterogenous graphs with multiple node types, with thesame shape preserved across the latent and topology space views. Edges are offby default to avoid clutter but can be shown on demand in this view, as straight orbundled edges.Global topologyWe present graph topology with node-link diagrams in CorGIE. We adopt the D3force-directed layout [17] for the GLOBAL TOPOLOGY VIEW (Figure 5.3g, Fig-ure 5.8b). It visualizes the topology of an entire input graph, and is computed oncein the pre-processing step. As in the LATENT SPACE VIEW, we encode graph nodesas circles for homogenous graphs and glyphs of different shapes for different nodetypes for heterogeneous graphs. We use straight lines for graph edges and curvedlines when edge bundling is enabled.Although this straightforward backstop layout may provide useful insights in smallergraphs, such global views can be extremely cluttered for larger ones. In that case,our custom layout for exploring local neighborhoods of user-specified subsets ofthe graph is crucial, in the k-hop topology view.K-hop topologyThe K-HOP TOPOLOGY VIEW (Figure 5.3f, Figure 5.7b & e, Figure 5.8d, Fig-ure 5.9b, Figure 5.10d & e) shows the topological neighbors of the focal node setsspecified by users. We devise the k-hop layout algorithm to mimic how informa-tion is aggregated in GNNs, with boxes enclosing meaningful groups. On the farleft are the focal nodes, with hop-1 neighbors in the middle, hop-2 neighbors to theright, and hop-3 neighbors on the far right. Within the boxes, nodes are clusteredusing the topological (Jaccard) distance. Full algorithmic details on the layout areprovided in Section 5.5.135Node featuresThe NODE FEATURES VIEW on the top right (Figure 5.3c) shows feature distribu-tions for up to three sets of nodes: all nodes, and each of the two possible sets offocal nodes (foc-0 and foc-1). Dense and sparse features are visualized differentlydue to their scale differences.For dense features, we choose histograms as they are intuitive and well-knownfor visualizing and comparing distributions of scalar values. They also scale wellfor the available display space, to handle the design target of up to a dozen denseattributes. We organize these feature histograms as a matrix. Each column repre-sents one feature, whereas the rows show different collections of nodes. The firstrow shows the full distribution of all nodes; the next row shows the distribution ofthe focal node set foc-0 and underneath that is the set foc-1. For example, in Fig-ure 5.3c, the two focal groups of user nodes have different distribution in averagevote (the second last avr column), but little difference in the number of votes (thelast column). Note that for heterogeneous graphs, different types of nodes can havedifferent features, so focal node sets could end up with some empty histograms.In contrast to dense features, there could be thousands of sparse features, and theirordering is not explicitly meaningful. We designed a two-level custom view tomake effective use of the screen space in a compact area, as shown in Figure 5.8f.The top part is a feature strip overview that aggregates the information in the moredetailed feature matrix part below it. The lower feature matrix is a heatmap con-taining one square cell per feature (e.g., a word train), and its luminance en-codes the count of nodes with that feature (e.g., the number of papers with theword train in the Cora dataset). Conceptually, the heatmap contains a singlevery long line that is simply wrapped to fit into a rectangular aspect ratio. It is nota true 2D matrix, so the absolute position of a cell in terms of a row/column is notmeaningful. The upper feature strip aggregates the information in the heatmapinto a highly compressed pixel-based depiction where many consecutive featuresare combined into the same vertical line, with a luminance representing the max-imum values across the range. The number of features to aggregate is determined136by the available horizontal pixel budget of the view, so that the strip fits within it.As with the dense features, there is a row for all nodes on top, with a row for each ofthe two focal groups below to show partial feature distributions. In addition, thereis a lower diff row derived by subtracting the feature values of the two focal nodegroups. Figure 5.8f shows an example (bottom row) that visualizes differences inthe word distribution of foc-0 and foc-1, where the dark strips indicate that thereare large difference in those word counts.In each row, the feature matrix can be hidden to save space, or expanded to inspectdetail. The luminance values are separately normalized within each row since thevalue ranges could be quite different, so they each require a separate legend.Latent neighbor blocksThe LATENT NEIGHBOR BLOCKS VIEW (Figure 5.3b, Figure 5.8c) overlays topo-logical neighbor distributions on the 2D latent space. The challenge is to show thepositions of each node’s neighbors in the latent space, for each of the nodes. Wedo so with aggregation and nesting. We aggregate by partitioning the space into an8×8 grid, to create a coarser representation of the space with 64 blocks, and mapeach node as belonging to the block that encloses it. Within each block, we nest acomplete copy of the latent space itself, again at a coarse resolution as an 8×8 gridof cells. Blocks that do not contain any nodes are omitted from the drawing.This view is inspired by origin-destination (OD) maps for geo-spatial networks [184],but we show neighbor set distributions rather than geographical movement. Withineach high-level block, we outline the single low-level cell that corresponds to itsblock index with red, like the origin in an OD map. For every other cell in theblock, we encode with luminance the number of neighbors of nodes that fall withinthe red-outlined origin cell, like the destinations in an OD map. If the red-outlinedblock and its surrounding cells are darker than others in all blocks, that pattern in-dicates neighbors in the topology space are still neighbors in the latent space, andthe GNN has successfully preserved neighborhood structure. Figure 5.8c for the137Cora scenario illustrates this case.However, there is a limitation to this visual encoding: it is less informative for abipartite graph, such as the Movie scenario, where movie nodes are only allowed tohave user nodes as their first hop neighbors. In this case, as shown in Figure 5.3b,the pattern of neighbors separated in the latent space does not reflect evidence ofpoor GNN performance.Distance comparisonThe DISTANCE COMPARISON VIEW (Figure 5.3e, Figure 5.7e & f) shows distribu-tions of distances in each space and supports comparison between spaces. In thedata abstraction (Section 5.3.2), we derive one distance metric in each space. Thisview reveals matches and mis-matches of distances between the topology and thelatent space in one tab, and between the feature and the latent space in a secondtab. A match constitutes a positive correlation of the distances in two spaces. Thisview shows pairs of nodes, which may either represent an edge or be disconnectedin the input graph.To present and compare two scalar distributions simultaneously, we combine twohistograms and a gridded scatterplot into one chart, where the x-axis representsthe distance distribution in the latent space, and the y-axis represents that in thetopology or feature space. The number of items to plot in the scatterplots can behuge, so we use a grid-binning approach to avoid over-drawing, and to speed upcomputation we also down-sample the node pairs if there are more than a million.Each tab accommodates multiple charts to show different sets of node pairs sideby side, such as all, within a focal group, between two focal groups, or a user-customized filtered set based on connectivity and link prediction values (whenavailable). The customization options are shown in Supplemental Figure D.17,including a choice between linear and log scale to handle the variance in data.For example, the three bottom histograms in Figure 5.6e show the latent distance138distribution within foc-0, within foc-1, and between the two groups respectively.We observe that the between-group distances are much larger than the within-groupdistances, indicating that the two focal groups are two distant clusters in the latentspace. Figure 5.9a shows a pursuit of problematic node pairs, where brushing andhighlighting the bottom-right area in the scatterplot signals a mis-match with anegative correlation between latent and topology distance.5.4.3 View coordinationTo visually underscore the linkages between the views, we carefully maintain theconsistency of visual encodings for nodes across views including the shape, color,and size. The node color is user configurable in the SETTINGS view (Figure 5.3d),to either the latent space position (the similarity rainbow), a specific node feature(a sequential ramp), node type (distinguishable categorical colors), or node classi-fication labels if available (also categorical). The shape depends on the node typefor heterogeneous graphs, or is a circle in the homogeneous case. The size is alsoglobally configurable in SETTINGS.We also develop user interactions with consistent semantics across all views. InCorGIE, there are three major interactions from light weight to heavy weight:hover, select, and focus. Typically, they are used in order: hover is for quick andtemporary exploration, then select offers a more persistent visual prompt for nodesof interest that are identified with hover, and finally focus stores the currently se-lected nodes into a persistent group. The look and feel of the interaction is shownin the supplemental video.The hover action is triggered when users mouse over objects, such as a node oredge in the TOPOLOGY VIEWS, and a block in the LATENT NEIGHBOR BLOCKSVIEW. On hover, CorGIE reacts with strong visual prompts (Figure 5.7c & e):strokes on activated nodes and edges while desaturating the background with ahalf-transparent mask, black partial distributions on top of the node feature his-tograms, tooltips and node labels.139The select action is triggered by clicking a node/edge or brushing multiple nodes.The visual prompt is highly similar to that for hover (Figure 5.6c & d), but persistseven when users move the cursor away. Users can control whether the neighbors oftarget nodes are highlighted during the hover and select actions, with drop-downmenus in the SETTINGS VIEW.The heavy-weight focus action enables users to find correspondences for one ortwo groups of specified nodes. When a focal group is created or modified manyviews are updated, with changes to the FEATURES and DISTANCE views and thecomputationally intensive operation of creating a new K-HOP TOPOLOGY layout.CorGIE supports several focus actions: create a new focal group, add-to/single-out/remove-from existing groups, and clear groups. These actions enable usersrefine and change the specified nodes, as required by the iterative nature of the tasks(Section 5.3.3). Since only selected nodes can be the target of focus actions, thefocus menu to choose one of these actions appears only after nodes are selected. Itdrops down from the top of the window and looks like a corgi dog’s paw, in keepingwith the system name, as shown in the video and Supplemental Figure D.6. Thefocal nodes are bounded by boxes and labelled with foc-0 or foc-1 in both theFOCAL LAYOUT and LATENT SPACE views.5.4.4 ImplementationCorGIE is implemented using ReactJS 3 and Redux 4 as the frontend scaffold. Wechoose Canvas over SVG in most of the views for browser performance. For agraph with more than a thousand nodes, there would be thousands of DOM ele-ments such that re-rendering SVGs on user interactions would be very expensive.We develop a layering system using the Konva library 5 to add visual elements ontop of a static canvas in order to avoid the expensive re-render whenever possible.As the focus action involves heavy computation that takes seconds and even min-utes, we offload it to multiple Web Workers to keep the application responsive, and3https://reactjs.org/4https://redux.js.org/5https://konvajs.org/140use the Comlink library 6 to communicate between threads. We open-source thecode at https://github.com/zipengliu/CorGIE.5.4.5 Design alternativesDuring the iterative refinement process for the CorGIE interface, we experimentedextensively with different alternatives for exploring the neighbor sets of a singlefocus group, including an Upset-based [104] view for neighbor sets and a partialadjacency matrix with roll-up histograms. We discuss these design alternativesin detail in Supplemental Section D.3. However, the pixel space required wouldbe infeasible for realistic sizes of graphs (e.g. graphs with hundreds of nodes),and there was no obvious extension to comparing neighbor sets between two fo-cus groups. Our final design relies more extensively on interactive exploration toexplore neighbor sets for one or two groups at a time, rather than attempting tovisually encode all of possibilities simultaneously. In the case study sessions, adomain expert suggested that we could use the traditional bipartite layout for theMovie scenario, but we consider the force-directed layout a more general solutionfor all kinds of graphs.5.5 K-hop layoutWe present a new visual encoding technique, the K-hop graph layout, which is usedin CorGIE’s K-HOP TOPOLOGY VIEW (Figure 5.3f, Figure 5.7b & e, Figure 5.8d,Figure 5.9b, Figure 5.10d & e). We present the computational scalability of theK-hop layout in this section, and evaluate its suitability for the tasks in Section 5.6.We also discuss three design alternatives that we considered and their limitations.6https://github.com/GoogleChromeLabs/comlink1411. Divide neighborsfocal group: {d,f}hop-1: {b, e, g, k}hop-2: {a, c, h, l}2. Place boxesfocal hop-2hop-13. Layout within groupdfgkbeachlabcdefklgmji4. Adjust groupdf gkbeachlflip rotate 180 no trans.hFigure 5.4: K-hop layout algorithm in four steps. The optional fifth step isbundle edges (not shown).5.5.1 AlgorithmWe first provide an overview of the layout algorithm, then the technical detail andrationale of each step, and finally we discuss its scalability.The K-hop layout aims to organize topological neighbors of user-specified nodesby the number of hops and their clustering structure. As illustrated in Figure 5.4,it is computed in four main steps: 1) divide relevant nodes into groups, 2) boundnodes within boxes and lay out the boxes, 3) lay out nodes within each box in-dependently, and 4) perform transformations to optimize global readability. Anoptional final step is 5) edge bundling.Divide neighbors. We divide the nodes into groups: user-specified focal groups,all 1st-hop neighbors of focal nodes, all 2nd-hop neighbors of focal nodes, andso on until the Kth hop (where K≤3). The rest of nodes are deemed irrelevantand thus discarded, so this view shows only a subset of the nodes in the globallayout. This division naturally matches the neighborhood aggregation in GNNs(Section 5.2.1), by design. Note that a node can only appear in one group to avoidconfusing duplication, with priority given to the leftmost view in which it appears.Place boxes. We bound the nodes of groups within boxes and place the boxes fromleft to right. For the focal groups on the left, multiple focal nodes are placed fromtop to bottom. This layout mimics how the information flow is typically thoughtabout by GNN developers, as we verified in our interviews with them.Lay out within group. We lay out the nodes within each group independently. Toreveal the clustering structure of nodes, we use a dimensionality reduction tech-nique UMAP [119] to reduce a high-dimensional topological distance matrix of142nodes down to two dimensions. Users can choose between local distance (connec-tions to previous hop only) or global distance (all K hop connections). The layoutfalls back to D3 force-directed layout if there are not enough nodes to run UMAP.Because of the independence between groups, we can parallelize the UMAP pro-cesses for performance improvement. It would be possible to use alternative lay-out algorithms in this step like t-SNE [172]; we choose UMAP for its speed andstrength in revealing global structure [29].Adjust group. We need to fine-tune the layout from previous step since purelylocal layout decisions could endanger the global readability. For example, therecan be many edge-edge crossings in step 3 of Figure 5.4, even though the locallayout within each box has been optimized. We use an approach similar to Pro-crustes analysis [16] to potentially reorient each group. Specifically, we can applysix possible transformations to each group: rotation (0, 90, 180, 270 degrees), andflip (horizontal and vertical). These rigid transformations do not change the rela-tive positions within the group. We find the optimal transformations for each groupwith a simple enumeration algorithm, that is, to enumerate all 6 possible transfor-mations for all groups for a total of 6B combinations, where B is the number ofgroups. The node-repulsion Linlog function [130] is used as the readability metricto find the optimum. Note that only node pairs between boxes are considered inthe measurement, and we sample them randomly to compute an approximation ofthe function for performance speedup.Bundle edges. Finally, to further reveal the pattern of connections, we apply edgebundling to the graph layout to reduce visual clutter. Out of several edge bundlingalgorithms [193], we choose the multilevel agglomerative one [54] for its speedand competitive visual performance. We note that it can introduce distorted edges,which are harder to trace than to unchanged ones, so we allow the user to togglebetween straight and bundled edges.143Times (sec)DatasetN/EtotalNk-hopEk-hopNboxBUMAP(step 3)totalMovie 1K/5K 1088 5000 633 4 4.1 5.5Movie 1K/5K 1088 5000 370 5 2.5 4.7Cora 3K/5K 989 2010 576 3 3.3 3.9Cora 3K/5K 2116 4222 1028 4 8.5 9.6Coauthor 10K/54K 9528 43317 4254 3 47.1 57.6Coauthor 10K/54K 9951 44261 6544 4 110.1 123.9Table 5.1: Computational benchmarks for the K-hop layout. N is #nodes, Eis #edges: approximate for full graph and exact for used within k-hoplayout. N box is maximum #nodes in any box (key dependency), B is#boxes; times in seconds.5.5.2 ScalabilityTo aid the perceptual scalability of our approach, our emphasis is on reducingedge-edge crossings, which would hinder users’ judgement about topology andconnectivity. Our choice to reveal clustering structure through the K-hop layoutconstitutes a trade-off, where there could be substantial node-node overlap in theK-hop layout.For computational scalability, we instrument the code to obtain elapsed (wall-clock) timings, and measure the running times with different interactively chosenfocal groups for three datasets: Movie (N=1K, E=3K) and Cora (N=3K, E=5K)as introduced in Section 5.2.2, and Coauthor (N=10K, E=54K) extracted from alarger graph of 69K nodes [166]. The experiment is conducted in Chrome on a2020 MacBook Pro with 2.3GHz 8-core i9 CPU. We present the results in Ta-ble 5.1. In addition to the total graph size, we provide the number of nodes andedges used within for the k-hop graph layout (the combination of the focal nodesand their neighbors within k hops). We see that sometimes the k-hop focal lay-out incorporates all nodes and edges but sometimes shows only a subset of them,depending on the choice of focal groups.144Figure 5.5: Design alternatives to K-hop layout: (a) D3 force-directed layoutwith strict bounding box; (b) unbounded D3 force-directed layout withrepulsion force between groups; (c) WebCola force-directed layout withconstraints; (d) space-filling curve layout using spiral curve.Graphs with 1K nodes like those in the Movie dataset take a few seconds to com-pute, whereas those with 10K nodes take significantly longer. The UMAP compu-tation (step 3) is the computational bottleneck in our algorithm, as shown the timingnumbers on the two rightmost columns. We note that the times do not depend onthe total number of nodes, but instead, the largest number within a single boundingbox, because we run UMAP processes in parallel across multiple worker threadsfor each box. We thus include the node count within the maximum box in the ta-ble. For example, the time for the last run (with 6544 nodes in the most numerousbox) of 110 seconds doubles that of the previous run (4254 nodes maximum) of 50seconds on the same Coauthor dataset. Compared to the UMAP computation, thetime required for step 1 (divide neighbors), 2 (place boxes), and 4 (adjust group) isnegligible; the optional 5th step (edge bundling) takes about 10−20% of the totaltime.5.5.3 Design alternativesWe tried three design alternatives to the K-HOP TOPOLOGY, which were less ef-fective at showing neighbor hops and clustering structure than our final choice: D3force-directed, WebCola, and space-filling spirals.The D3 force-directed layout is versatile as we can configure the type of strengthof forces. Similarly to the K-hop layout, we use topological distance as the pulling145force between nodes, instead of the usual approach of using the edges to controlthe forces. Figure 5.5 (a) and (b) shows two versions of this layout. The first takeis bounded by boxes like the K-hop layout, but the nodes are pulled towards othergroups and positioned alongside the group boundary as an undesired artifact. Itdoes not show clustering structure within a group. The second take is not boundedstrictly but has a strong repulsion force to separate nodes of different neighborgroups. The separation of hops is not obvious enough, especially for homogeneousgraphs where the nodes are not shape coded.The WebCola layout [43] (Figure 5.5c) allows us to specify positional constraintsbetween nodes, but due to its force-directed nature, it suffers from a similar prob-lem as the D3 version in Figure 5.5a.The space-filling curve layout [123] is one of the fastest layout algorithms. Wechoose a spiral curve as it can separate nodes of different groups from center toperipheral. On a polar coordinate system, we use the topological distance as thedistance on the curve between two consecutive nodes. From Figure 5.5d, we cansee that it has some spatial division between different groups but is even worse thanthe D3 version. It also comes with an artifact that sometimes proximity in the polarcoordinate system does not match the proximity in topology: nodes are placed withdifferent radii but similar angles. Moreover, such a spiral space-filling design doesnot use space effectively, with a lot of unused space in the peripheral area wherethe less important neighbors reside.5.6 ResultsWe evaluate CorGIE in two ways: we use it to address the questions in the twomotivating scenarios, and we recruited two GNN experts to participate in a userstudy to evaluate its utility and usability.We first present our two usage scenarios. We then describe our study design, anddiscuss highlights of two expert user sessions. We then summarize the study par-ticipants’ feedback.146Below, we systematically describe four aspects of CorGIE usage: the visualizationtask being conducted, the visual operation used within the CorGIE interface, thedirect observation that is possible from the CorGIE display, and any resulting in-ference of authors or participants. Here we tag only the visualization task withcolor; in Supplemental Section D.4 we provide a version of the text with all fouraspects color-coded.5.6.1 Usage scenario 1: Movie RecommendationTo obtain an overview of the dataset, we first color the nodes by their node type(user and movie). Figure 5.6a shows that we can see two clusters of movies inblue on the right-hand side and many clusters of users in orange in the rest of thelatent space. After coloring the nodes by the underlying positional colormap inthe latent space, we observe in the GLOBAL TOPOLOGY view (Figure 5.3g) thatmost user nodes (triangles) are laid out around movie nodes (circles). Many nodesthat are connected by only one edge are placed around the periphery of the layout,indicating that there are many users who only watched one movie. In Figure 5.6b,we notice that the peripheral users that are connected to the same movies havesimilar colors: for example, the triangles on the bottom right are all green. We canthus infer that the GNN does a good job in grouping these one-time users reviewingthe same movie.To compare user clusters , we select and focus on two clusters from the LATENTSPACE, one in the green zone (foc-0) on the left, the other in the purple zone (foc-1)on the upper right, as shown in Figure 5.3. In the K-HOP TOPOLOGY (Figure 5.3b),there appears to be three green sub-clusters in foc-0, and the green nodes connect tomany nodes in the hop-1 box, while foc-1 does not show any salient internal struc-ture. To understand the topological difference between the two focal groups, weexplore their connections in the K-HOP TOPOLOGY with hover and select actions.Figure 5.6c and 5.6d show the visual highlights in the K-HOP TOPOLOGY whenselecting the green foc-0 nodes and the purple foc-1 nodes respectively. We cansee that the green nodes in the top foc-0 box connect to all hop-1 neighbors, but the147Figure 5.6: Check overview of the Movie scenario: (a) LATENT SPACE,movie nodes in blue and user nodes in orange; (b) inset of GLOBALTOPOLOGY showing similar peripheral nodes in similar green color(closeup of lower right for screenshot in Figure 5.3). When comparingtwo clusters of user nodes, (c) we select nodes in foc-1 of the K-HOPTOPOLOGY, highlighting themselves and four neighbors in the hop-1box, and also in the (d) GLOBAL TOPOLOGY. (e) We compare distancesbetween topology and latent space, and (f) between feature and latentspace.purple nodes in the lower foc-1 box only connect to four of those neighbors. Weinfer that the GNN has learned the topological difference and thus separates themin the latent space.148Figure 5.7: Check movie recommendations: (a) top 5 recommended moviesfor user 5355; (b) K-HOP TOPOLOGY focusing user 5355 and movieLotR; (c) hover on the movie Suicide Squad that the user has watchedbefore; (d) topological distance between 5355 and LotR is large (0.97);(e) K-HOP TOPOLOGY for user 587 and movie LotR.Besides graph topologies, we also want to check whether distances match be-tween spaces. In the DISTANCE COMPARISON VIEW (Figure 5.6e), we can seethat the within group distances in latent space are small, while the between groupdistances are large, which confirms that we have picked two distant clusters fromthe LATENT SPACE. We notice that the topological distances within the green clus-ter (foc-0) are relatively large considering they belong to the same cluster (leftmostvertical histogram in Figure 5.6e), which actually matches the existence of threesub-clusters in the K-HOP TOPOLOGY. The feature distances in all three chartsof Figure 5.6f are small and similar, indicating that the node features (#votes andaverage vote of a user) cannot distinguish foc-0 and foc-1. We further confirm theineffectiveness of the features by reading the feature distances by picking someother node groups (not shown in figure), which indicates that they are not usefulfeatures, and could perhaps be removed from the dataset.After exploring the clustering structure, we inspect instances of recommendation.In Figure 5.7a we select the user node 5355, and list its top 5 recommended movies.The first one is The Lord of the Rings: the Return of King (LotR). We would like tounderstand why GNN decides to recommend LotR , so we focus this recommen-dation by clicking on it. CorGIE automatically creates two focal groups with theuser 5355 in foc-0 and movie LotR in foc-1 (Figure 5.7b). The K-HOP TOPOLOGYshows that user 5355 watched two movies (in pink) before. When we hover on thetwo pink nodes in the hop-1 box (Figure 5.7c), we find out that each only shares afew users with the recommended movie LotR; for example, Suicide Squad has only1495 hop-1 shared neighbors. We believe that this recommendation is poor, potentiallyindicating that we are not there yet with GNN training. We further confirmed thisproblem by reading the topological 0.05 and latent 0.97 distances between user5355 and movie LotR (Figure 5.7e), which seem to be correlated negatively.We later find a recommendation that makes sense: user 587 and movie LotR. Asshown in Figure 5.7e, the two movies that 587 watched, Inception and The DarkNight, share many users with the recommended movie LotR (many hop-1 neighborsare highlighted when hovering on Inception). Moreover, the topological distance is0.72 in this case, which is relatively small compared to the overall distance distribu-tion (not shown in figure). CorGIE thus shows evidence that this recommendationis well supported.We repeat this process to check other recommendations , and we find many thatdo not make sense. We conclude that this training result is not satisfactory. It couldbe due to the “cold start” problem in such a small dataset, where most users onlywatch 1 or 2 movies. Also, the node features were not very useful: e.g., we knowthat #votes cannot distinguish users effectively. To improve the recommendation,we might try a different model or fix the dataset problems.5.6.2 Usage scenario 2: CoraAs with the Movie scenario, we first check the clustering structure of paper nodes.We color the nodes by the predicted labels to see if the label distribution makessense. In LATENT SPACE (Figure 5.8a), we can see that the different classes of pa-pers are roughly separated to different areas. In GLOBAL TOPOLOGY (Figure 5.8b),although the force-directed layout lacks much structure, we can still see that nearbynodes are in similar colors. In LATENT NEIGHBOR BLOCKS (Figure 5.8c), wecan see that the red-outlined origin and its surrounding cells are darker for all theblocks. All three observations indicate that the GNN has done a good job in clas-sifying papers using the graph topology.Next, we inspect and compare a few clusters . For example, when we compare150(select and focus) the entire red cluster and a left part of the blue one (Figure 5.8d),the K-HOP TOPOLOGY (Figure 5.8e) shows that most nodes of the same colorconnect to each other, and the diff chart in the NODE FEATURES VIEW (Figure 5.8f)signals a considerable amount of different words between the two paper clusters(many visible dark strips on the diff row). These observations reinforce our goodimpression on the training result.As the GNN seems to classify most nodes successfully, we look for problematicnodes and edges . We brush the distance scatterplot to highlight the node pairswith large latent distances but small topological distances (i.e. the bottom rightarea), as shown in Figure 5.9a, and we focus one of these problematic node pairs.In the K-HOP TOPOLOGY (Figure 5.9b), we find through a few rounds of inter-active hover that the two focal nodes only share one 1st-hop neighbor (circled inred), which has many nodes connected in the second hop. As the Jaccard distanceaccounts for multiple hops of neighbors, the large number of shared hop-2 neigh-bors can explain the low topological distance (0.23). We conjecture that the GNNdecides to locate them far from each other due to the large difference in the firsthop. Further investigation showed that the node in foc-0 is mis-classified, whichhints at the limitations of this GNN model to deal with such situations.151Figure 5.8: Explore overview of Cora dataset, with nodes colored by pre-dicted label: (a) LATENT SPACE; (b) GLOBAL TOPOLOGY; (c) LATENTNEIGHBOR BLOCKS with zoomed-in inset showing that the red-outlinedcells are darker than others. Compare two paper clusters: (d) focus onthe red and blue clusters in the LATENT SPACE; (e) K-HOP TOPOLOGY;(f) NODE FEATURES VIEW with inset showing the word count differ-ences between foc-0 and foc-1.152Figure 5.9: Find problematic node pairs in the Cora dataset: (a) select nodepairs with high latent distance but low topological distance; (b) K-HOPTOPOLOGY focusing on a buggy node pair, with only one shared hop-1neighbor (circled) and many shared hop-2 neighbors.1535.6.3 Expert study overviewOne of the two GNN experts is a PhD student (P1), and the other is an industriallab researcher (P2). Both have published multiple GNN papers in top-tier venuesin recent years. We had a meeting with each participant a few months before thestudy when we were iterating on the tasks and design of CorGIE. In this meeting,we discussed the pain points in their workflow of model training and confirmedthat our correspondence approach matches their mental model, but did not showany version of the CorGIE interface to the participants at that time. In the remotestudy session, I introduced and demonstrated a near-final version CorGIE for about40 minutes. Then participants used that version of CorGIE on their own datasets forabout 30 minutes, followed by a semi-structured interview to gather their feedback.The results provide preliminary evidence of the utility of CorGIE in fulfilling thetasks we proposed in Section 5.3.3, as the experts were highly positive about theeffectiveness of CorGIE.5.6.4 Expert session 1: Cora decision boundaryIn this case study, expert P1 first chose to perform quite similar tasks as in ourusage scenario. Then he wanted to check the overview to study misclassification.He colored the nodes by label correctness (Figure 5.10a), where he saw that thered (misclassified) nodes are distributed across different clusters. He used a fil-tered brush selection to highlight and focus the wrongly predicted nodes within themiddle cluster in the LATENT SPACE. He colored the nodes by the predicted labels(Figure 5.10d) and true labels (Figure 5.10e) respectively, so he could understandwhich classes are wrong . It appeared that the GNN predicted these nodes as theorange class similar to their hop-1 neighbors, but the ground truth labels state thatthey belong to multiple different classes (blue, green, red, etc) although the hop-1neighbors are still orange. Further inspection with hover and brush selection showsthat the focal nodes are loosely connected to their hop-1 neighbors, but the hop-1neighbors are tightly connected to the hop-2 ones. Based on the variance in true154Figure 5.10: P1 explored the decision boundary in the Cora dataset: (a) hefocused on the wrongly predicted nodes within a cluster (in the bound-ing box) in the LATENT SPACE; he compared (b) feature distances ofthe entire graph to (c) those within the focused nodes; he inspectedneighbors in the K-HOP TOPOLOGY that are colored by (d) predictedlabel and (e) ground-truth label.labels and the sparsity in topology, P1 inferred that these nodes were sitting at thedecision boundary between two or multiple classes in the graph topology. It washis first time to visually inspect the boundary nodes even though he had used thisdataset for years. He emphasized the importance of understanding the decisionboundary if he needed to improve the GNN model.After exploring the topology, P1 turned to DISTANCE COMPARISON for more infor-mation on the node features, where he found the feature distances within the focalnodes are relatively large compared to the overall distribution (Figure 5.10c&b).He conjectured that the current model performed as he expected, and that it wouldbe unlikely to classify the boundary nodes correctly even if he kept training moreepochs. He also conjectured that the GNN should use the node features in a better155Figure 5.11: The USAir dataset without focal nodes.way to correct these nodes. As he confirmed that the GNN has done a good job formost nodes except some boundary ones, he concluded that there is no need to keeptraining more epochs, and training should be stopped to avoid over-fitting.5.6.5 Expert session P2: USAirWe present how P2 used CorGIE on the dataset USAir from his previous GNNpaper. It is a homogeneous graph showing the air routes (2126 edges) betweenairports in the US (332 nodes). There are no node features in this graph. To obtaina node embedding, we used a GCN [93] to predict links between the airports in anunsupervised setting.After CorGIE loads (Figure 5.11), P2 first checks the overview of the graph. Thenodes are colored by their 2D latent space position. The LATENT SPACE VIEWshows that there are many clusters, and the one in the lower left (green) cornerappears to be very different from the others. The initial global layout shows thatmost of the nodes connect to neighbors with the same colors, and the LATENTNEIGHBOR BLOCKS VIEW shows that the topological neighbors are placed nearby156Figure 5.12: Find buggy node pairs. (left) Popup customization dialog topick the node pairs that are connected in the input graph but predictedas false by the GNN; (middle) the resulting 28 erroneous node pairsshown in the scatterplot of DISTANCE COMPARISON VIEW; (right)the 28 error node pairs in the list on the right side of that view.Figure 5.13: P2 focuses on the first buggy node pair (124-283), and hoverson one of the focal nodes to see its neighbors.(the dark blocks are close to the red outlined block). We can infer that the GNNnode embedding makes sense at a high level.157P2 proceeds to find buggy link predictions by the GNN. With the help of the au-thor, he creates a filter to single out the node pairs that are connected in the inputgraph but are predicted false by the GNN, as shown in Figure 5.12, with the popupwindow that appears after clicking on the wrench icon in the upper right cornerof the DISTANCE COMPARISON VIEW. He then clicks the first buggy node pair124-283 in the list on the right side of that view, which automatically creates twonew focus groups of a single node each and a new k-hop focal layout with thosegroups. He hovers on the focal nodes in the focal layout to check their neighbors(Figure 5.13), and he finds out that they actually share many neighbors in both thefirst hop and the second hop. The topological distance, as shown in the distancecomparison, is 0.19, which is a very low value, which further confirms what hefinds in the focal layout. However, the latent distance is 0.50, which equals thethreshold in GNN that separates true and false predictions. He infers that this erro-neous predicted pair is located in the decision boundary area, making it difficult topredict.5.6.6 Expert feedback summaryWe summarize the participants’ feedback throughout the sessions, including thequestions and discussions while using CorGIE the semi-structured interview at theend. Both participants were impressed after they watched our demonstration andused the tool themselves for the first time.Task abstraction. They confirmed that the two-step task framework (specify andcorrespond) is powerful for evaluating GNNs, especially for the iterative refine-ment of focal nodes, and both thought CorGIE is useful for debugging node em-beddings (P1: “finding decision boundary nodes”, P2: “debug [models] quickly inan intense leaderboard competition in ML research”).CorGIE interface. They stated that most views are straightforward to understandafter the introductory explanation, but the heatmap and heat-strips in the NODEFEATURES VIEW are less intuitive. The three-level interaction is easy to use andsuitable for the tasks. As for the usability and learnability, we observed that both158participants picked up the the main functions very quickly, which the participantsalso verbally confirmed in the interview.However, they criticized the interface as being visually busy/complex, making itdifficult to locate a specific object at first glance. We thus improved the interfacelayouts according to their advice, e.g., we made the focus action menu much moresalient, changing it to the eye-catching final design of a corgi paw dropping downfrom the top. Both expressed the excitement to incorporate CorGIE into their re-search workflow once it is available for deployment.K-hop layout. Both participants consider the K-HOP TOPOLOGY the most usefulfeature in CorGIE, which validates the success of our novel K-hop layout algo-rithm. The representation aligns with how they would think about GNNs.5.7 Related workCorGIE relates to previous research on explainable AI, which uses visualization toexplain machine learning models. In this section we relate and compare CorGIEto explainable visualizations for non-graph models, latent space, and graph neuralnetworks. We also discuss the related work for our new K-hop layout.5.7.1 Visualizations for machine learning modelsMany visualization tools have been developed to explain and evaluate machinelearning models, especially in the recent five years. Readers can refer to manysurvey papers to have a comprehensive understanding on this topic [24, 78]. Mostsuch tools target models for images/videos and text sequences, most of which arenot modeled as graphs. Although the seminal GNN work, GCN [93] stems fromconvolutional neural networks (CNNs) for images (as they both use convolutionoperation), it is not practical to simply retarget tools for CNNs to GNNs. Thekey difference is that pixel neighbors in an image are intrinsically different fromtopological neighbors in a graph. Sequence data like text in natural languages is159even more different. Unlike many visualizations that try to open the “black box”of a neural network and those that keep the “black box” completely opaque, wepropose a “grey box” approach that only leverages a key concept in GNN - neigh-borhood aggregation (Section 5.2.1) - to balance the generalizability and specificityfor GNN models.5.7.2 Latent space interpreterThe study of the latent space, also known as the embedding space, has attractedsubstantial attention, especially in text-based ML practices. Embeddings eliminatethe error-prone process of feature selection and they can be pre-trained for manydifferent downstream applications [189]. As CorGIE supports exploration of thecorrespondences between a latent space and a graph, it directly relates to muchprevious work on latent space visualization.One theme is to reveal local and global structures of one or multiple latent spaces.Ghosh et al. [55] developed VisExPres, an interactive toolkit for user-driven eval-uation of embeddings. Heimerl et al. [73] compare embeddings based on differentquantitative metrics, while Cutura et al. does so with dimensional reduction tech-niques and matrix visualizations [30]. Liu et al. [108] analogize the process of map-ping and comparing semantic dimensions within latent spaces to latent space car-tography. This theme mainly focuses on digging deep inside latent spaces, whichis different from our approach that tries to connect a latent space to the input.Another theme, dimension reduction (DR), is also loosely related as many non-linear DR techniques produce latent spaces [131]. There are many visual tools forDR results: some for presenting the data points [158], and some connect the DRresults back to their semantically meaningful high-dimensional spaces [50, 161].Their original input is usually tabular data, where a data item has multiple features(also known as attributes), but none of them consider graph topology like CorGIE.1605.7.3 GNN interpreterThere has been some effort in the GNN community on GNN interpretation. Mostof the previous work focuses on generating algorithms or models to conduct featureand neighbor analysis [190]. Huang et al. [81] propose GraphLIME to produce afew most representative features in the neighborhood of a node. One of the mostwell-known work is the GNNExplainer [188] by Ying et al., who propose a methodto compute the most important nodes of one or a group of user-specified nodesbased on information theory. Rao et al. [147] follow up on GNNExplainer andpropose xFraud to target at fraud detection specifically. Pope et al. [137] extendexplainability algorithms for CNN like saliency map, class activation mapping, andback-propagation to Graph-CNN. This thread of work does not sufficiently supporthuman-in-the-loop visual exploration, and so falls short in providing an overviewof how well a GNN learns from the input graph and in connecting the exemplarinspections iteratively.Two recent papers visualize graph models. Li et al. [106] propose EmbeddingVisto compare multiple graph embeddings generated from different models. It focuseson how the node metrics (e.g. degree, centrality) are preserved in the embeddings,but does not directly support topological neighborhood exploration nor node fea-tures. As this research project was conducted before the recent trends of graphneural networks that use neighborhood aggregation, their embeddings are not gen-erated by GNNs but other graph models. A paper by Jin et al. [89], who developedGNNVis to diagnose errors in GNN, is the most similar related work to CorGIE.It only targets one downstream ML application, node classification, which limitsits scope of usage. Unlike CorGIE, GNNVis ignores the latent space and takesanother route: it supports finding errors in the classification predictions, and com-paring against two surrogate models to approximate the erroneous component inGNN.1615.7.4 Graph layoutWe discuss the related work for our technical contribution, the K-hop layout. Agraph layout survey by Gibs et al. [56] categorizes graph layouts into three ap-proaches: force-directed layouts, dimensional reduction layouts (e.g., MDS, t-SNE, UMAP), and multi-level graph layouts (see survey by McGee et al. [117]).Our K-hop layout combines the DR and multi-level approaches. Out of dozens oflayout algorithms, ours is most related to those focusing on clusters. IPSep-CoLa[43] by Dwyer et al. is a force-directed layout specializes at separation constraints.Our investigation of its utility for this setting shows that it generates undesired arti-facts (Figure 5.5c). Our K-hop layout is similar to the Group-In-a-Box layout pro-posed by Rodrigues et al. [152] for category-based partitions of social networks. Ituses the space-filling treemap techniques to separate the clusters, whereas our divi-sion into groups is dynamic based on user selections and combines DR techniques.The LinLog layout [130] proposed by Noack is an energy-based model for clusterseparation, and it inspires our readability metric in the adjustment step.5.8 Discussion and future workOn reflection, after the whole process of characterizing the problem, designing theinterface, and evaluating it, we believe that the high-level idea of exploring cor-respondences among input, output, and internal data structures is useful for GNNinterpretation and may be generalizable to other deep neural networks. As a neu-ral network itself is usually not interpretable for many reasons (e.g. non-linearlity,layering), the approach to “open the black box” completely demands substantialcomplexity of interpretation, and demands high expertise from the users. We pre-fer a “grey-box” approach balancing the exposure of internal structure to a nonzerobut minimum level. Though often not exposed to the end users for interpreta-tion, the node embedding is universal to all GNNs. By finding correspondencesbetween it and the input data (graph), we can infer if the GNN has achieved satis-factory results without fiddling around the internal structure of a neural network.We imagine this approach could be generalized for other types of neural networks162and ML applications.We would also like to promote our notion of data spaces. Our data abstractionconsists of the three data spaces – latent, topology, and feature – alongside a taskabstraction that connects them. This mental model of connecting data spaces helpsus design the views: some are dedicated for a single space, while some connectmultiple spaces. It also contributes to the usability and learnability of CorGIE,for which we have preliminary evidence through the feedback from two GNN ex-perts. It would be interesting future work to introduce more data spaces, such as ageospatial space to deal with GNNs that specialize in geospatial data [66].It would be useful future work to handle larger graphs. Due to the K-hop layoutcomputation, the current version of CorGIE cannot guarantee a smooth interactiveuser experience for graphs with more than 20K nodes. Although many popularbenchmark datasets are within this scale, the ML community is moving forwardwith larger datasets, such as those on the OGB platform [80].To further extend CorGIE, we could incorporate the algorithms like GNNExplainer [188]either in the K-HOP TOPOLOGY VIEW implicitly or in a dedicated view explicitly.This idea was also independently brought up by one of the GNN experts. We couldalso support more ways to specify nodes, such as topological statistics. Finally,we could enable the comparison of multiple node embeddings of the same inputgraph, to help evaluate model re-architecting and hyper-parameter tuning. A futurepaper could study how to enable GNN developers to explore the correspondencesbetween multiple input graphs and their node/graph embeddings.5.9 SummaryIn this work, we present a task abstraction for exploring the correspondences be-tween an input graph and the latent space created by a GNN, to understand if GNNhas learned important characteristics from the graph and to find bugs in the la-tent space. Based on this abstraction, we develop an interactive multi-view tool,CorGIE, which is validated through usage scenarios and case studies with GNN ex-163perts. As the most important component in CorGIE, we propose the K-hop graphlayout to reveal how GNNs aggregate information for nodes of interest. Both casestudies and expert studies validate the effectiveness of bringing CorGIE into a GNNmodel development life-cycle. We envision that our novel data and task abstrac-tion, in conjunction with our design rationales and implementation considerations,could serve as a stepping stone for future researches.164Chapter 6Reflection and ConclusionsIn the previous chapters, I introduce four projects with a common theme of visual-izing multi-level structures. In this final chapter, I will first discuss the limitationsof each project, then present my reflections, before concluding my PhD work.6.1 LimitationsIn ADVIEW, our design can handle datasets of hundreds of trees with about 100leaf nodes per tree, which is limited by the task of showing the full detail of areference tree dendrogram on a single screen without scrolling. We do not need toworry about the breadth and depth of a tree, as they are bounded by the numberof nodes: an extreme case could be a ladder-like tree, where there are only left orright descendants for each internal node. For the number of trees, there is no hardlimit, but our current implementation on the browser can only deal with roughly500 trees at interactive frame rates. It is possible to exceed these limits by closingthe Individual AD view as the number of SVG elements is reduced drastically.In SPRAWLTER, we only consider readability metrics. It would be useful to inves-tigate how our metrics can combine with others to make better graph layouts. In165terms of computational complexity, our current implementation only adds a con-stant multiplicative factor to the count-based approach, which makes it almost asapplicable as the baseline.In LOGSEG, the simple SVM model is fast to train and use. It has a small numberof parameters compared to other machine learning models like neural networks.The primary concern for such a model is underfitting. User behavior in softwarecould vary across different people or even different sessions of the same person. Ahuman will make mistakes, which could mislead the model. Users interleave dif-ferent subtasks together, rendering the assumption of modeling user behavior withevent sequences problematic. There could be many unpredictable factors whenusing software, such as interrupting the workflow by a colleague. Our segmenta-tion model might be unlikely to perform correctly in these complicated cases. Forscalability, there are no limits on the number of events in a sequence as this onlinealgorithm can work as the user uses Photoshop. The depth of the segmentationhierarchy depends on the nature of the user’s workflow. Usually it will be a smallnumber as the best hierarchical segmentation reflects the user’s mental model abouthis workflow.In CORGIE, we limit the scope to one input graph and one node embedding only.There could be many future directions: GNNs that take many graphs as input andgenerate graph embeddings, GNNs that output edge embeddings, and comparisonof multiple embeddings. The current implementation of K-hop layout also haslimits on the number of nodes, that is, less than 20K nodes, to maintain a smoothinteractive speed on a browser.6.2 Reflection on multi-level structuresMulti-level structures are the common components in the four research projects.Multi-level structure is a broad and general concept, not a specific technique. Thestructures exist in almost all data types in information visualization: they couldbe either hierarchical or non-hierarchical; they could be tables, trees, graphs, timeseries, text, etc. This concept is applicable to many problem domains. My own166examples include phylogenetics in biology, user engineering in image editing soft-ware, graph neural networks in machine learning. I have also seen many examplesin the literature, such as turbulence in scientific visualization [129] and humanresource management [148]. There are various techniques to extract multi-levelstructures depending on the domain, data and task. For example, one can use ageneral hierarchical clustering algorithm with a distance metric to measure differ-ences between data items, or, simply categorize the items according to a categoricaldata attribute. Sometimes these structures are not even computed with algorithms,but manually specified by users in cases like CORGIE. It is hard for me to imaginean overarching framework for computing multi-level structures in data.The visualization of multi-level structures is not a trivial problem in the designlayer of the nested model. The hierarchical nature dictates the need of differentviews for different levels in order to solve a domain problem that contains multi-level tasks. In ADVIEW, the interface contains views for all levels in a tree collec-tion dataset, and some of the levels (e.g., subsets of trees) are shown in multipleviews (e.g., the tree distribution view and cluster aggregated dendrogram view),which is detailed in Supplemental Section A.1. It is not enough to have multiplestatic views. Following the guidelines for view coordination in the visualization lit-erature, such as Qu et al. [145], we link the views with consistent visual encodingand user interactions. In ADVIEW, the color palette is carefully designed to matchthe semantics, where the five medium-saturation colors (Figure 2.10) for the sub-tree highlight blocks have roughly equal luminance so that no focal subtree seemsmore important than the others. Also, the color palette is consistent between thereference tree and all the aggregated dendrograms so that users can quickly catchtheir connections. In CORGIE, the shape, color, and size of nodes are consistentacross different views.User interaction is also crucial for linking views. It is important to design how theinterface reacts to user actions such as clicking and hovering, usually with changesin the visual encoding. Meanwhile, it is also important and challenging to maintainthe consistency of interface reaction to the same user action on different views. InCORGIE, we design three kinds of user interactions from light-weight to heavy-167weight: hover, select, and focus. The trigger action is consistent across views:mouse over, mouse click, and dedicated buttons. The reacting visual prompts arealso consistent: strong highlights with a half-transparent mask on hovering, similarbut persistent visual highlights on selection, and new graph layouts on focus.6.3 Reflection on collaborationAll contributions in this dissertation are the result of collaborative work. I be-lieve visualization for human-in-the-loop data analysis requires collaboration withstakeholders, especially domain experts (users). In ADVIEW, I collaborated with agroup of phyloegenetic biologists in UBC, who became the users of ADView. InSPRAWLTER, I collaborated with a visualization researcher from another university,who brought new insights to our project. In LOGSEG and CORGIE, I collaboratedwith industrial visualization researchers from Adobe and Uber respectively, whoprovided pointers to Photoshop developers or users, and machine learning devel-opers.Through these collaboration, I realize that there is a spectrum for the possibilityof user collaboration from a tight connection with collaborators to a loose one.In ADVIEW, Shing Hei Zhan, a PhD student in biology, worked with us mainlythrough weekly meetings all along the whole research life cycle. He participatedin iterations of domain characterization, data and task abstraction, design, evalu-ation studies, and paper writing. This tight collaboration is best for ensuring theutility of visualization tools as the collaborating domain expert understands the do-main problem well. However, this could be a rare case, as it requires significantinvestment from the domain experts. There is also a potential danger: the scopeof the project and resulting tool could overfit to one or a small group of people,leaving it less generalizable to a bigger scope and similar usage scenarios.The LOGSEG and CORGIE project lie on the loose side of the collaboration spec-trum, where none of the coauthors are domain experts. I believe this is a more usualcase given the challenges to maintain a long-term tight collaboration. I startedthe LOGSEG project during my industrial internship in Adobe Research, where168Zhicheng (Leo) Liu was my mentor. He brought the resources in Adobe as wewere working on Photoshop, the flagship product of the company. We talked tomanagers, researchers, developers, and users of Photoshop to understand the dataand tasks and define the project scope. After we decided to work on segmenta-tion of user action logs, we recruited Photoshop experts as participants to collecttheir usage data on Photoshop in remote study sessions. During this project, Ilearned to use Photoshop myself through online tutorials and have used it in mydaily life since then. I, as the first author, became an amateur user of our target do-main, which lowered the necessity to collaborate with Photoshop experts closelythroughout the project cycle. Besides, many research artefacts were intuitive to us.For example, we could understand the process of making a poster or touch up ahuman face in Photoshop, even though we might not be able to do it ourselves.In CORGIE, Yang Wang played a similar role as Leo in LOGSEG. We discussedpotential research questions with his manager in Uber Research, before deciding towork on interpretation of graph models. Then we talked to graph machine learningexperts in Uber to understand what graph neural networks are and their painpointsin the evaluation of GNN. We recruited two other experts for tool evaluation afterwe had a stable version of the tool in the late stage. In between, we reviewed manypapers published in the machine learning venues and articles / blog posts by expertsto ensure that what we did matched what the users want and how they think. Wewere able to do this in this project because we have computer science backgroundsand basic knowledge of machine learning, but this approach would not have workedin ADVIEW where a deeper knowledge of phylogenetics was required.6.4 Reflection on research topic choicesThe four projects reported in this dissertation may appear to be very different interms of type and domain. As shown in Table 6.1, ADVIEW and CORGIE coversall layers in the nested model, while the other two do not. ADVIEW, LOGSEG,and CORGIE are problem-driven work to address concrete domain problems, whileSPRAWLTER is a technique-driven work to improve a visual encoding. The four169ADView Sprawlter LogSeg CorGIEDomain Phylogenetics Image editingsoftware Graph machine learningData& task Compare many trees Segment logsExplore correspondences ofinput graph & node embeddingEncodingdesignAggregated Dendrogram;Multi-view tool ADViewReadability metricsfor node-link graph:area-aware; SprawlterK-hop graph layout;multi-view tool CorGIEAlgorithm Find correspondingbranchesFind node and edgecrossingsDetermine chunkboundaries Compute graph layoutTable 6.1: A summary of contributions of my four projects covering the fourlayers in the nested model.projects span three different domains: phylogenetics in biology, user engineeringin software, and graph machine learning. However, there still exists a commontheme, multi-level structures, to connect them.I chose to explore broad topics on purpose from an early stage of my PhD program,with the goal of experiencing and learning as much as possible. This choice hada negative impact on the duration of my PhD, as I had to learn about multiplenew application domains over my time in the program. However, the positiveimpact on the breadth of my research outweighs the negative impact on duration. Istrengthened my ability to learn new knowledge in unfamiliar disciplines, increasedmy experience of collaborating with people, and learned how to conduct differenttypes of visualization research. The diversity in topics and types of research in thisearly stage of my carrer will greatly benefit my future career in academia.6.5 Concluding remarksI present four projects for visualizing multi-level structures in data. I summarizetheir coverage of the four layers of visualization concerns using the nested modelin Table 6.1. To conclude, I contribute novel data and task abstraction, visual tech-niques, and tools to deal with multi-level structures in three domains. To solve themulti-level tasks with complex data of the target domains, the multi-level structuresare either exploited directly or derived from the raw data, which are then visual-ized using multiple views. The abstraction and visual techniques are also benefited170from the multi-level structures compared to previous work.Multi-level structures can be found in many real-world problems, and they arebecoming more common and important due to the growing complexity of dataand tasks. There remain many research opportunities on the usage of multi-levelstructures to solve concrete domain problems.171Bibliography[1] Newick tree format.http://evolution.genetics.washington.edu/phylip/newicktree.html. → pages36, 41[2] A. J. Aberer, D. Krompass, and A. Stamatakis. Pruning rogue taxaimproves phylogenetic accuracy: an efficient algorithm and webservice.Systematic biology, 62(1):162–166, 2012. → page 210[3] E. Adar, M. Dontcheva, and G. Laput. CommandSpace: modeling therelationships between tasks, descriptions and features. In Proc. 27th annualACM Symp. User Interface Software and Technology (UIST), pages167–176, 2014. → pages 94, 100, 108[4] A. M. A. Aguinaldo, J. M. Turbeville, L. S. Linford, M. C. Rivera, J. R.Garey, R. A. Raff, and J. A. Lake. Evidence for a clade of nematodes,arthropods and other moulting animals. Nature, 387(6632):489–493, 1997.→ page 209[5] H. Almeida, D. Guedes, W. Meira, and M. J. Zaki. Is there a best qualitymetric for graph clusters? In Joint European Conf. Machine Learning andKnowledge Discovery in Databases, pages 44–59. Springer, 2011. → page60[6] D. Archambault. Structural differences between two graphs throughhierarchies. In Proc. Graphics Interface, pages 87–94. CanadianInformation Processing Society, 2009. → page 15[7] D. Archambault, T. Munzner, and D. Auber. GrouseFlocks: Steerableexploration of graph hierarchy space. IEEE Trans. Visualization andComputer Graphics (TVCG), 14(4):900–913, 2008. → pages 75, 237, 240172[8] E. N. Argyriou, M. A. Bekos, and A. Symvonis. Maximizing the totalresolution of graphs. The Computer Journal, 56(7):887–900, 2013. →page 60[9] D. Auber. Using strahler numbers for real time visual exploration of hugegraphs. In Proc. Intl. Conf. Computer Vision and Graphics, volume 1,page 3, 2002. → page 14[10] D. Auber et al. TULIP 5. In R. Alhajj and J. Rokne, editors, Encyclopediaof Social Network Analysis and Mining, pages 1–28. Springer, Aug. 2017.doi:10.1007/978-1-4614-7163-9\ 315-1. → pages 74, 75, 240[11] B. Bach, N. Henry-Riche, T. Dwyer, T. Madhyastha, J.-D. Fekete, andT. Grabowski. Small MultiPiles: Piling time to explore temporal patterns indynamic networks. In Computer Graphics Forum (CGF), volume 34, pages31–40, 2015. → page 16[12] F. Beck, F.-J. Wiszniewsky, M. Burch, S. Diehl, and D. Weiskopf.Asymmetric visual hierarchy comparison with nested icicle plots. In JointProc. Intl. Workshop on Euler Diagrams & Graph Visualization inPractice, pages 53–62, 2014. → page 15[13] M. Behrisch et al. Quality metrics for information visualization. InComputer Graphics Forum (CGF), volume 37, pages 625–662. WileyOnline Library, 2018. → page 56[14] J. L. Bentley and T. A. Ottmann. Algorithms for reporting and countinggeometric intersections. IEEE Trans. Computers, (9):643–647, 1979. →page 85[15] J. Bernard, M. Steiger, S. Mittelsta¨dt, S. Thum, D. Keim, andJ. Kohlhammer. A survey and task-based quality assessment of static 2dcolormaps. In Visualization and Data Analysis 2015, volume 9397, page93970M. Intl. Society for Optics and Photonics, 2015. → page 134[16] I. Borg and P. J. Groenen. Modern multidimensional scaling: Theory andapplications. Springer Science & Business Media, 2005. → page 143[17] M. Bostock, V. Ogievetsky, and J. Heer. D3 data-driven documents. IEEETrans. Visualization and Computer Graphics (TVCG), 17(12):2301–2309,2011. → page 135[18] R. Bouckaert and J. Heled. DensiTree 2: seeing trees through the forest.bioRxiv, page 012401, 2014. → page 17173[19] S. Bremm, T. Von Landesberger, M. Heß, T. Schreck, P. Weil, andK. Hamacher. Interactive visual comparison of multiple trees. In IEEEConf. Visual Analytics Science and Technology (VAST), pages 31–40, 2011.→ pages xx, 12, 17, 24, 37, 42, 43, 53, 218, 219[20] D. Bryant. A classification of consensus methods for phylogenetics.DIMACS series in discrete mathematics and theoretical computer science,61:163–184, 2003. → pages 17, 34[21] P. S. Bullen. Handbook of means and their inequalities, volume 560.Springer Science & Business Media, 2013. → page 67[22] S. P. Callahan, J. Freire, E. Santos, C. E. Scheidegger, C. T. Silva, and H. T.Vo. VisTrails: visualization meets data management. In Proc. ACMSIGMOD Intl. Conf. Management of data, pages 745–747, 2006. → page92[23] I. Chami, S. Abu-El-Haija, B. Perozzi, C. Re´, and K. Murphy. Machinelearning on graphs: A model and comprehensive taxonomy. ArXiv,abs/2005.03675, 2020. → page 123[24] A. Chatzimparmpas, R. M. Martins, I. Jusufi, K. Kucher, F. Rossi, andA. Kerren. The state of the art in enhancing trust in machine learningmodels with the use of visualizations. Computer Graphics Forum (CGF),39(3):713–756, 2020. doi:https://doi.org/10.1111/cgf.14034. → page 159[25] H.-T. Chen, L.-Y. Wei, and C.-F. Chang. Nonlinear revision control forimages. In ACM Trans. Graphics (TOG), volume 30, page 105, 2011. →pages 89, 91, 94, 95, 98, 116[26] H.-T. Chen, L.-Y. Wei, B. Hartmann, and M. Agrawala. Data-drivenadaptive history for image editing. In Proc. 20th ACM SIGGRAPH Symp.Interactive 3D Graphics and Games (I3D), pages 103–111, 2016. → pages89, 90, 91, 94, 95, 98, 99, 109, 112, 116[27] P.-Y. Chi, S. Ahn, A. Ren, M. Dontcheva, W. Li, and B. Hartmann. MixT:automatic generation of step-by-step mixed media tutorials. In Proc. 25thannual ACM Symp. User Interface Software and Technology (UIST)), pages93–102, 2012. → page 93[28] A. C. Cloud. Accelerating your creativity - Adobe MAX 2017 - day 1keynote, October 2017. Retrieved September 18, 2019 fromhttps://youtu.be/4j pd1awSac?t=8119. → page 89174[29] A. Coenen and A. Pearce. Understanding UMAP.https://pair-code.github.io/understanding-umap/, 2019. → page 143[30] R. Cutura, M. Aupetit, J.-D. Fekete, and M. Sedlmair. Comparing andexploring high-dimensional data with dimensionality reduction algorithmsand matrix visualizations. In Proc. Intl. Conf. Advanced Visual Interfaces(AVI). ACM, 2020. doi:10.1145/3399715.3399875. → page 160[31] R. Davidson and D. Harel. Drawing graphs nicely using simulatedannealing. ACM Trans. Graphics (TOG), 15(4):301–331, 1996. → page 77[32] J. Q. Dawson, T. Munzner, and J. McGrenere. A search-set model of pathtracing in graphs. Information Visualization, 14(4):308–338, 2015. →pages 56, 59, 62[33] J. D. Denning, W. B. Kerr, and F. Pellacini. MeshFlow: interactivevisualization of mesh construction sequences. In ACM Trans. Graphics(TOG), volume 30, page 66, 2011. → page 94[34] J. D. Denning, V. Tibaldo, and F. Pellacini. 3DFlow: Continuoussummarization of mesh editing workflows. ACM Trans. Graphics (TOG),34(4):140, 2015. → page 94[35] K. Dextras-Romagnino and T. Munzner. Segmentifier: Interactiverefinement of clickstream data. In Computer Graphics Forum (CGF),volume 38, pages 623–634. Wiley Online Library, 2019. → page 92[36] G. Di Battista, P. Eades, R. Tamassia, and I. G. Tollis. Graph drawing:algorithms for the visualization of graphs. Prentice Hall PTR, 1998. →pages 57, 61[37] W. Dong, X. Fu, G. Xu, and Y. Huang. An improved force-directed graphlayout algorithm based on aesthetic criteria. Computing and Visualizationin Science, 16(3):139–149, 2013. → page 61[38] C. Dunne and B. Shneiderman. Improving graph drawing readability byincorporating readability metrics: A software tool for network analysts.University of Maryland, HCIL Tech Report HCIL-2009-13, 2009. → page59[39] C. Dunne, S. I. Ross, B. Shneiderman, and M. Martino. Readability metricfeedback for aiding node-link visualization designers. IBM Journal ofResearch and Development, 59(2/3):14–1, 2015. → pages57, 58, 60, 61, 66, 75, 76, 240175[40] V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson.Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982,2020. → pages 121, 129[41] T. Dwyer and F. Schreiber. Optimal leaf ordering for two and a halfdimensional phylogenetic tree visualisation. In Proc. Australasian Symp.Information Visualization, volume 35, pages 109–115, 2004. → page 15[42] T. Dwyer, K. Marriott, and P. J. Stuckey. Fast node overlap removal. In Intl.Symp. Graph Drawing (GD), pages 153–164. Springer, 2005. → page 85[43] T. Dwyer, Y. Koren, and K. Marriott. IPSep-CoLa: An incrementalprocedure for separation constraint layout of graphs. IEEE Trans.Visualization and Computer Graphics (TVCG), 12(5):821–828, 2006. →pages 146, 162[44] T. Dwyer, B. Lee, D. Fisher, K. I. Quinn, P. Isenberg, G. Robertson, andC. North. A comparison of user-generated and automatic graph layouts.IEEE Trans. Visualization and Computer Graphics (TVCG), 15(6):961–968, 2009. → pages 59, 62[45] P. Eades, S.-H. Hong, K. Klein, and A. Nguyen. Shape-based qualitymetrics for large graph visualization. In Intl. Symp. Graph Drawing (GD),pages 502–514. Springer, 2015. → page 59[46] W. K. Edwards, T. Igarashi, A. LaMarca, and E. D. Mynatt. A temporalmodel for multi-level undo and redo. In Proc. 13th annual ACM Symp.User Interface Software and Technology (UIST), pages 31–40, 2000. →page 93[47] S. Emmons, S. Kobourov, M. Gallant, and K. Bo¨rner. Analysis of networkclustering algorithms and cluster quality metrics at scale. PloS one, 11(7):e0159161, 2016. → page 60[48] B. S. Everitt, S. Landau, M. Leese, and D. Stahl. Cluster Analysis. WileyPublishing, 5th edition, 2011. → page 106[49] J. Fangerau, B. Ho¨ckendorf, B. Rieck, C. Heine, J. Wittbrodt, and H. Leitte.Interactive similarity analysis and error detection in large tree collections.In Visualization in Medicine and Life Sciences III, pages 287–307.Springer, 2016. → page 16176[50] R. Faust, D. Glickenstein, and C. Scheidegger. DimReader: Axis lines thatexplain non-linear projections. IEEE Trans. Visualization and ComputerGraphics (TVCG), 25(1):481–490, 2019.doi:10.1109/TVCG.2018.2865194. → page 160[51] J. D. Fekete, G. Grinstein, and C. Plaisant. IEEE InfoVis 2004 contest: Thehistory of InfoVis. www.cs.umd.edu/hcil/iv04contest, 2004. → page 75[52] A. Frick, A. Ludwig, and H. Mehldau. A fast adaptive layout algorithm forundirected graphs (extended abstract and system demonstration). In Intl.Symp. Graph Drawing (GD), pages 388–403. Springer, 1994. → pages61, 75, 237[53] E. R. Gansner, Y. Koren, and S. North. Graph drawing by stressmajorization. In Intl. Symp. Graph Drawing (GD), pages 239–250.Springer, 2004. → pages 61, 77[54] E. R. Gansner, Y. Hu, S. North, and C. Scheidegger. Multilevelagglomerative edge bundling for visualizing large graphs. In IEEE Symp.Pacific Visualization (PacificVis), pages 187–194, 2011. → page 143[55] A. Ghosh, M. Nashaat, J. Miller, and S. Quader. VisExPreS: A visualinteractive toolkit for user-driven evaluations of embeddings. IEEE Trans.Visualization and Computer Graphics (TVCG), pages 1–1, 2020.doi:10.1109/TVCG.2020.3039106. → page 160[56] H. Gibson, J. Faith, and P. Vickers. A survey of two-dimensional graphlayout techniques for information visualisation. Information Visualization,12(3-4):324–357, 2013. doi:10.1177/1473871612455749. → pages56, 162[57] M. Gleicher. Considerations for visualizing comparison. IEEE Trans.Visualization and Computer Graphics (TVCG), 24(1):413–423, 2018. →pages 13, 17[58] M. Glueck, P. Hamilton, F. Chevalier, S. Breslav, A. Khan, D. Wigdor, andM. Brudno. PhenoBlocks: Phenotype comparison visualizations. IEEETrans. Visualization and Computer Graphic (TVCG), 22(1):101–110, 2016.→ page 16[59] F. Grabler, M. Agrawala, W. Li, M. Dontcheva, and T. Igarashi. Generatingphoto manipulation tutorials by demonstration. In ACM Trans. Graphics(TOG), volume 28, page 66, 2009. → page 93177[60] M. Graham and J. Kennedy. Multiform views of multiple trees. In Intl.Conf. Information Visualisation, pages 252–257, 2008. → page 16[61] M. Graham and J. Kennedy. A survey of multiple tree visualisation.Information Visualization, 9(4):235–252, 2010. → pages 11, 15[62] M. Gronemann. Engineering the Fast-Multipole-Multilevel Method formulticore and SIMD architectures. Master’s thesis, Lehrstuhl fu¨rAlgorithm Engineering, Fakulta¨t fu¨r Informatik, Technische Universita¨tDortmund, 2009. → page 75[63] T. Grossman, J. Matejka, and G. Fitzmaurice. Chronicle: capture,exploration, and playback of document workflow histories. In Proc. 23ndannual ACM Symp. User Interface Software and Technology (UIST), pages143–152, 2010. → page 93[64] J. Guerra-Gomez, M. L. Pack, C. Plaisant, and B. Shneiderman.Visualizing change over time using dynamic hierarchies: TreeVersity2 andthe StemView. IEEE Trans. Visualization and Computer Graphics (TVCG),19(12):2566–2575, 2013. → page 16[65] S. Guo, K. Xu, R. Zhao, D. Gotz, H. Zha, and N. Cao. EventThread: Visualsummarization and stage analysis of event sequence data. IEEE Trans.Visualization and Computer Graphics (TVCG), 24(1):56–65, 2017. →page 92[66] S. Guo, Y. Lin, N. Feng, C. Song, and H. Wan. Attention basedspatial-temporal graph convolutional networks for traffic flow forecasting.In Proc. AAAI Conf. Artificial Intelligence, volume 33, pages 922–929,2019. → page 163[67] S. Hachul and M. Ju¨nger. Drawing large graphs with a potential-field-basedmultilevel algorithm. In Intl. Symp. Graph Drawing (GD), pages 285–295.Springer, 2004. → page 237[68] S. Hachul and M. Ju¨nger. Drawing large graphs with a potential-field-basedmultilevel algorithm. In Intl. Symp. Graph Drawing (GD), pages 285–295.Springer, 2005. → page 75[69] S. Hachul and M. Ju¨nger. An experimental comparison of fast algorithmsfor drawing general large graphs. In Intl. Symp. Graph Drawing (GD),pages 235–250. Springer, 2005. → page 56178[70] H. Haleem, Y. Wang, A. Puri, S. Wadhwa, and H. Qu. Evaluating thereadability of force directed graph layouts: A deep learning approach.IEEE Computer Graphics and Applications, 39(4):40–53, 2019. → page 60[71] J. Heer and S. K. Card. DOITrees revisited: scalable, space-constrainedvisualization of hierarchical data. In Proc. Intl. Working Conf. AdvancedVisual Interfaces (AVI), pages 421–424. ACM, 2004. → pages 14, 24[72] J. Heer, J. Mackinlay, C. Stolte, and M. Agrawala. Graphical histories forvisualization: Supporting analysis, communication, and evaluation. IEEETrans. Visualization and Computer Graphics (TVCG), 14(6):1189–1196,2008. → page 92[73] F. Heimerl, C. Kralj, T. Moller, and M. Gleicher. embcomp: Visualinteractive comparison of vector embeddings. IEEE Trans. Visualizationand Computer Graphics (TVCG), pages 1–1, 2020.doi:10.1109/TVCG.2020.3045918. → page 160[74] K. Heller, Y. W. Teh, and D. Gorur. Infinite hierarchical hidden markovmodels. In Proc. 12th Intl. Conf. Artificial Intelligence and Statistics,volume 5, pages 224–231. PMLR, 2009. → pages 91, 93[75] M. Helmkampf, I. Bruchhaus, and B. Hausdorf. Phylogenomic analyses oflophophorates (brachiopods, phoronids and bryozoans) confirm thelophotrochozoa concept. Proc. Royal Society of London B: BiologicalSciences, 275(1645):1927–1933, 2008. → pages 209, 215[76] M. Hess, S. Bremm, S. Weissgraeber, K. Hamacher, M. Goesele,J. Wiemeyer, and T. von Landesberger. Visual exploration of parameterinfluence on phylogenetic trees. IEEE computer graphics and applications,34(2):48–56, 2014. → page 16[77] D. M. Hillis, T. A. Heath, and K. St John. Analysis and visualization oftree space. Systematic Biology, 54(3):471–482, 2005. → pages 12, 16[78] F. Hohman, M. Kahng, R. Pienta, and D. H. Chau. Visual analytics in deeplearning: An interrogative survey for the next frontiers. IEEE Trans.Visualization and Computer Graphics (TVCG), 25(8):2674–2693, 2019.doi:10.1109/TVCG.2018.2843369. → page 159[79] D. Holten and J. J. Van Wijk. Visual comparison of hierarchicallyorganized data. In Computer Graphics Forum (CGF), pages 759–766,2008. → page 15179[80] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, andJ. Leskovec. Open graph benchmark: Datasets for machine learning ongraphs. arXiv preprint arXiv:2005.00687, 2020. → page 163[81] Q. Huang, M. Yamada, Y. Tian, D. Singh, D. Yin, and Y. Chang.GraphLIME: Local interpretable model explanations for graph neuralnetworks. arXiv preprint arXiv:2001.06216, 2020. → page 161[82] W. Huang, S.-H. Hong, and P. Eades. Layout effects on sociogramperception. In Intl. Symp. Graph Drawing (GD), pages 262–273. Springer,2005. → page 62[83] W. Huang, S.-H. Hong, and P. Eades. Effects of crossing angles. In IEEESymp. Pacific Visualization (PacificVis), pages 41–46, 2008. → pages60, 62, 70, 72, 76, 235[84] W. Huang, P. Eades, S.-H. Hong, and C.-C. Lin. Improving multipleaesthetics produces better graph drawings. Journal of Visual Languages &Computing, 24(4):262–272, 2013. → page 61[85] W. Huang, M. L. Huang, and C.-C. Lin. Evaluating overall quality of graphvisualizations based on aesthetics aggregation. Information Sciences, 330:444–454, 2016. → page 61[86] D. H. Huson and C. Scornavacca. Dendroscope 3: an interactive tool forrooted phylogenetic trees and networks. Systematic biology, 61(6):1061–1067, 2012. → page 14[87] T. Itoh and K. Klein. Key-node-separated graph clustering and layouts forhuman relationship graph visualization. IEEE Computer Graphics andApplications, 35(6):30–40, 2015. → pages 75, 237, 240[88] C. H. Jackson. Displaying uncertainty with shading. The AmericanStatistician, 62(4):340–347, 2008. → page 194[89] Z. Jin, Y. Wang, Q. Wang, Y. Ming, T. Ma, and H. Qu. GNNVis: A visualanalytics approach for prediction error diagnosis of graph neural networks.arXiv preprint arXiv:2011.11048, 2020. → page 161[90] N. Kerracher and J. Kennedy. Constructing and evaluating visualisationtask classifications: Process and considerations. Computer Graphics Forum(CGF), 36(3):47–59, 2017. → page 21180[91] N. Kerracher, J. Kennedy, and K. Chalmers. A task taxonomy for temporalgraph visualisation. IEEE Trans. Visualization and Computer Graphics(TVCG), 21(10):1160–1172, 2015. → page 17[92] S. Kieffer, T. Dwyer, K. Marriott, and M. Wybrow. HOLA: Human-likeorthogonal network layout. IEEE Trans. Visualization and ComputerGraphics (TVCG), 22(1):349–358, 2015. → pages 57, 61, 66[93] T. N. Kipf and M. Welling. Semi-supervised classification with graphconvolutional networks. arXiv preprint arXiv:1609.02907, 2016. → pages125, 156, 159[94] M. Klammler, T. Mchedlidze, and A. Pak. Aesthetic discrimination ofgraph layouts. In Intl. Symp. Graph Drawing (GD), pages 169–184.Springer, 2018. → page 59[95] S. G. Kobourov, S. Pupyrev, and B. Saket. Are crossings important fordrawing large graphs? In Intl. Symp. Graph Drawing (GD), pages234–245. Springer, 2014. → page 62[96] N. Kong, T. Grossman, B. Hartmann, M. Agrawala, and G. Fitzmaurice.Delta: a tool for representing and comparing workflows. In Proc. ACMConf. Human Factors in Computing Systems (CHI), pages 1027–1036,2012. → page 93[97] D. Koop, J. Freire, and C. T. Silva. Visual summaries for graph collections.In IEEE Symp. Pacific Visualization (PacificVis), pages 57–64. IEEE, 2013.→ page 17[98] B. Lafreniere, A. Bunt, J. S. Whissell, C. L. Clarke, and M. Terry.Characterizing large-scale use of a direct manipulation application in thewild. In Proc. Graphics Interface, pages 11–18. Canadian InformationProcessing Society, 2010. → page 93[99] B. Lee, C. Plaisant, C. S. Parr, J.-D. Fekete, and N. Henry. Task taxonomyfor graph visualization. In Proc. AVI workshop on BEyond time and errors:novel evaluation methods for information visualization, pages 1–5. ACM,2006. → page 17[100] B. Lee, G. G. Robertson, M. Czerwinski, and C. S. Parr. CandidTree:visualizing structural uncertainty in similar hierarchies. InformationVisualization, 6(3):233–246, 2007. → page 15181[101] E. M. Lemmon and A. R. Lemmon. High-throughput genomic data insystematics and phylogenetics. Annual Review of Ecology, Evolution, andSystematics, 44(1):99–121, 2013. → page 11[102] J. Leskovec, J. Kleinberg, and C. Faloutsos. Graph evolution: Densificationand shrinking diameters. ACM Trans. Knowledge Discovery from Data(TKDD), 1(1):2, 2007. → pages 75, 237[103] I. Letunic and P. Bork. Interactive tree of life (iTOL) v3: an online tool forthe display and annotation of phylogenetic and other trees. Nucleic acidsresearch, 44(W1):W242–W245, 2016. → page 14[104] A. Lex, N. Gehlenborg, H. Strobelt, R. Vuillemot, and H. Pfister. UpSet:Visualization of intersecting sets. IEEE Trans. Visualization and ComputerGraphics (TVCG), 20(12):1983–1992, 2014.doi:10.1109/TVCG.2014.2346248. → pages 141, 272[105] G. Li, Y. Zhang, Y. Dong, J. Liang, J. Zhang, J. Wang, M. J. McGuffin, andX. Yuan. Barcodetree: Scalable comparison of multiple hierarchies. IEEETrans. Visualization and Computer Graphics (TVCG), 26(1):1022–1032,2019. → page 16[106] Q. Li, K. S. Njotoprawiro, H. Haleem, Q. Chen, C. Yi, and X. Ma.EmbeddingVis: A visual analytics approach to comparative networkembedding inspection. In IEEE Conf. Visual Analytics Science andTechnology (VAST), pages 48–59, 2018. doi:10.1109/VAST.2018.8802454.→ page 161[107] W. Li, J. Matejka, T. Grossman, J. A. Konstan, and G. Fitzmaurice. Designand evaluation of a command recommendation system for softwareapplications. ACM Trans. Computer-Human Interaction (TOCHI), 18(2):6,2011. → page 94[108] Y. Liu, E. Jun, Q. Li, and J. Heer. Latent space cartography: Visual analysisof vector space embeddings. Computer Graphics Forum (CGF), 38:67–78,06 2019. doi:10.1111/cgf.13672. → page 160[109] Z. Liu, B. Kerr, M. Dontcheva, J. Grover, M. Hoffman, and A. Wilson.Coreflow: Extracting and visualizing branching patterns from eventsequences. In Computer Graphics Forum (CGF), volume 36, pages527–538. Wiley Online Library, 2017. → page 92182[110] Z. Liu, T. Itoh, J. Q. Dawson, and T. Munzner. The sprawlter graphreadability metric: Combining sprawl and area-aware clutter. IEEE Trans.Visualization and Computer Graphics (TVCG), 26(6):2180–2191, 2020.ISSN 1941-0506. doi:10.1109/TVCG.2020.2970523. → pages vi, 55[111] Z. Liu, Z. Liu, and T. Munzner. Data-driven multi-level segmentation ofimage editing logs. In Proc. ACM Conf. Human Factors in ComputingSystems (CHI), pages 1–12, New York, NY, USA, 2020. ISBN9781450367080. doi:10.1145/3313831.3376152. → pages vi, 88[112] Z. Liu, S. H. Zhan, and T. Munzner. Aggregated dendrograms for visualcomparison between many phylogenetic trees. IEEE Trans. Visualizationand Computer Graphics (TVCG), 26(9):2732–2747, 2020. → pages vi, 10[113] Z. Liu, Y. Wang, J. Bernard, and T. Munzner. Visualizing graph neuralnetwork with CorGIE: Corresponding a graph to its embedding. IEEETrans. Visualization and Computer Graphics (TVCG), under review. →pages vii, 119[114] M. Loorak, M. Tory, and S. Carpendale. ChangeCatcher: Increasinginter-author awareness for visualization development. In ComputerGraphics Forum (CGF), volume 37, pages 51–62. Wiley Online Library,2018. → page 92[115] E. Maguire, P. Rocca-Serra, S.-A. Sansone, J. Davies, and M. Chen.Taxonomy-based glyph design with a case study on visualizing workflowsof biological experiments. IEEE Trans. Visualization and ComputerGraphics (TVCG), 18(12):2603–2612, 2012. → page 92[116] A. K. McCallum, K. Nigam, J. Rennie, and K. Seymore. Automating theconstruction of internet portals with machine learning. InformationRetrieval, 3(2):127–163, 2000. → page 126[117] F. McGee, M. Ghoniem, G. Melanc¸on, B. Otjacques, and B. Pinaud. Thestate of the art in multilayer network visualization. In Computer GraphicsForum (CGF), volume 38, pages 125–149. Wiley Online Library, 2019. →pages 56, 162[118] M. J. McGuffin and J.-M. Robert. Quantifying the space-efficiency of 2Dgraphical representations of trees. Information Visualization, 9(2):115–140, 2010. → pages 13, 24, 57, 62183[119] L. McInnes, J. Healy, and J. Melville. UMAP: Uniform manifoldapproximation and projection for dimension reduction. arXiv preprintarXiv:1802.03426, 2018. → pages 134, 142[120] M. Mehta, J. Rissanen, and R. Agrawal. MDL-based decision tree pruning.In Intl. Conf. Knowledge Discovery in Databases and Data Mining (KDD),volume 21, pages 216–221, 1995. → page 14[121] G. Melanc¸on. Just how dense are dense graphs in the real world?: Amethodological note. In Proc. AVI Workshop BEyond time and errors:novel evaLuation methods for Information Visualization (BELIV). ACM,2006. → page 85[122] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of wordrepresentations in vector space. arXiv preprint arXiv:1301.3781, 2013. →pages 100, 243[123] C. Muelder and K.-L. Ma. Rapid graph layout using space filling curves.IEEE Trans. Visualization and Computer Graphics (TVCG), 14(6):1301–1308, 2008. → page 146[124] T. Munzner. A nested model for visualization design and validation. IEEETrans. Visualization and Computer Graphics (TVCG), 15(6):921–928,2009. → pages xvii, 2, 8[125] T. Munzner. Visualization Analysis & Design. CRC Press, AK PetersVisualization Series, 2014. → pages 24, 57[126] T. Munzner, F. Guimbretie`re, S. Tasiran, L. Zhang, and Y. Zhou.TreeJuxtaposer: Scalable tree comparison using focus+context withguaranteed visibility. ACM Trans. Graphics, 22(3):453–462, 2003. →pages 12, 15, 20, 37, 39[127] T. Nakamura and T. Igarashi. An application-independent system forvisualizing user operation history. In Proc. 21st annual ACM Symp. UserInterface Software and Technology (UIST), pages 23–32, 2008. → page 93[128] M. Nancel and A. Cockburn. Causality: A conceptual model of interactionhistory. In Proc. ACM Conf. Human Factors in Computing Systems (CHI),pages 1777–1786, 2014. → page 94[129] D. B. Nguyen, R. O. Monico, and G. Chen. A visualization framework formulti-scale coherent structures in taylor-couette turbulence. IEEE Trans.184Visualization and Computer Graphics (TVCG), 27(02):902–912, 2021. →page 167[130] A. Noack. Energy models for graph clustering. Journal of GraphAlgorithms and Applications, 11(2):453–480, 2007. → pages60, 61, 143, 162[131] L. G. Nonato and M. Aupetit. Multidimensional projection for visualanalytics: Linking techniques with distortions, tasks, and layoutenrichment. IEEE Trans. Visualization and Computer Graphics (TVCG),25(8):2650–2673, 2018. → page 160[132] E. Paradis, J. Claude, and K. Strimmer. APE: analyses of phylogenetics andevolution in R language. Bioinformatics, 20:289–290, 2004. → page 14[133] N. Peng, H. Poon, C. Quirk, K. Toutanova, and W.-t. Yih. Cross-sentencen-ary relation extraction with graph lstms. Trans. Association forComputational Linguistics (ACL), 5:101–115, 2017. → page 124[134] H. Philippe, N. Lartillot, and H. Brinkmann. Multigene analyses ofbilaterian animals corroborate the monophyly of Ecdysozoa,Lophotrochozoa, and Protostomia. Molecular biology and evolution, 22(5):1246–1253, 2005. → pages 209, 215[135] A. Photoshop. Techniques for nondestructive editing, November 2018.Retrieved September 18, 2019 fromhttps://helpx.adobe.com/ca/photoshop/using/nondestructive-editing.html.→ page 94[136] C. Plaisant, J. D. Fekete, and G. Grinstein. Promoting insight-basedevaluation of visualizations: From contest to benchmark repository. IEEETrans. Visualization and Computer Graphics (TVCG), 14(1):120–134,2008. → page 15[137] P. E. Pope, S. Kolouri, M. Rostami, C. E. Martin, and H. Hoffmann.Explainability methods for graph convolutional neural networks. InIEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), pages10764–10773, 2019. doi:10.1109/CVPR.2019.01103. → page 161[138] A. J. Pretorius, I. A. Khan, and R. J. Errington. Cell lineage visualisation.In Computer Graphics Forum (CGF), volume 34, pages 21–30, 2015. →page 14185[139] J. Pretorius, H. C. Purchase, and J. T. Stasko. Tasks for multivariatenetwork analysis. In Multivariate network visualization, pages 77–95.Springer, 2014. → page 17[140] H. Purchase. Which aesthetic has the greatest effect on humanunderstanding? In Intl. Symp. Graph Drawing (GD), pages 248–261.Springer, 1997. → pages 59, 62[141] H. C. Purchase. Metrics for graph drawing aesthetics. Journal of VisualLanguages & Computing, 13(5):501–516, 2002. → pages 59, 60[142] H. C. Purchase, R. F. Cohen, and M. James. Validating graph drawingaesthetics. In Intl. Symp. Graph Drawing (GD), pages 435–446. Springer,1995. → page 62[143] H. C. Purchase, D. Carrington, and J.-A. Allder. Empirical evaluation ofaesthetics-based graph layout. Empirical Software Engineering, 7(3):233–255, 2002. → pages 56, 59[144] H. C. Purchase, B. Plimmer, R. Baker, and C. Pilcher. Graph drawingaesthetics in user-sketched graph layouts. In Proc. Eleventh AustralasianConf. User Interface-Volume 106, pages 80–88. Australian ComputerSociety, Inc., 2010. → page 62[145] Z. Qu and J. Hullman. Keeping multiple views consistent: Constraints,validations, and exceptions in visualization authoring. IEEE Trans.Visualization and Computer Graphics (TVCG), 24(1):468–477, 2018. →pages 35, 167[146] A. Rambaut. FigTree. http://tree.bio.ed.ac.uk/software/figtree/. → page 14[147] S. X. Rao, S. Zhang, Z. Han, Z. Zhang, W. Min, Z. Chen, Y. Shan, Y. Zhao,and C. Zhang. xFraud: Explainable fraud transaction detection onheterogeneous graphs. arXiv preprint arXiv:2011.12193, 2020. → page161[148] D. Rees, R. S. Laramee, P. Brookes, T. D’Cruze, G. A. Smith, and A. Miah.Agentvis: Visual analysis of agent behavior withhierarchical glyphs. IEEETrans. Visualization and Computer Graphics (TVCG), 2020. → page 167[149] F. Ricci, L. Rokach, and B. Shapira. Introduction to recommender systemshandbook. In Recommender systems handbook, pages 1–35. Springer,2011. → page 125186[150] D. Robinson and L. Foulds. Comparison of phylogenetic trees.Mathematical Biosciences, 53(1):131 – 147, 1981. ISSN 0025-5564. →page 37[151] O. Robinson, D. Dylus, and C. Dessimoz. Phylo.io: interactive viewingand comparison of large phylogenetic trees on the web. Molecular biologyand evolution, 33(8):2163–2166, 2016. → page 15[152] E. M. Rodrigues, N. Milic-Frayling, M. Smith, B. Shneiderman, andD. Hansen. Group-in-a-box layout for multi-faceted analysis ofcommunities. In IEEE Third Intl. Conf. Privacy, Security, Risk and Trustand IEEE Third Intl. Conf. Social Computing, pages 354–361, 2011. →page 162[153] L. Rokach and O. Maimon. Clustering methods. In Data mining andknowledge discovery handbook, pages 321–352. Springer, 2005. → page90[154] J. Ruan et al. TreeFam: 2008 update. Nucleic acids research, 36(suppl 1):D735–D740, 2007. → page 209[155] H.-J. Schulz. Treevis.net: A tree visualization reference. IEEE ComputerGraphics and Applications, 31(6):11–15, 2011. → page 13[156] T. Seifried, C. Rendl, M. Haller, and S. Scott. Regional undo/redotechniques for large interactive surfaces. In Proc. ACM Conf. HumanFactors in Computing Systems (CHI), pages 2855–2864, 2012. → page 93[157] Shapely contributors. Shapely: manipulation and analysis of geometricobjects. https://shapely.readthedocs.io/en/latest, 2018. [Online; accessed28-March-2019]. → page 240[158] D. Smilkov, N. Thorat, C. Nicholson, E. Reif, F. B. Vie´gas, andM. Wattenberg. Embedding projector: Interactive visualization andinterpretation of embeddings. arXiv preprint arXiv:1611.05469, 2016. →pages 100, 160, 243[159] S. A. Smith, M. J. Moore, J. W. Brown, and Y. Yang. Analysis ofphylogenomic datasets reveals conflict, concordance, and gene duplicationswith examples from animals and plants. BMC Evolutionary Biology, 15(1):150, 2015. → pages 20, 52187[160] A. J. Soper, C. Walshaw, and M. Cross. A combined evolutionary searchand multilevel optimisation approach to graph-partitioning. Journal ofGlobal Optimization, 29(2):225–241, 2004. → pages 75, 237[161] J. Stahnke, M. Do¨rk, B. Mu¨ller, and A. Thom. Probing projections:Interaction techniques for interpreting arrangements and errors ofdimensionality reductions. IEEE Trans. Visualization and ComputerGraphics (TVCG), 22(1):629–638, 2016.doi:10.1109/TVCG.2015.2467717. → page 160[162] S. S. Stevens. On the psychophysical law. Psychological review, 64(3):153,1957. → pages 70, 72[163] K. Strimmer and A. Von Haeseler. Likelihood-mapping: a simple methodto visualize phylogenetic content of a sequence alignment. Proc. NationalAcademy of Sciences, 94(13):6815–6819, 1997. → page 16[164] S. L. Su, S. Paris, F. Aliaga, C. Scull, S. Johnson, and F. Durand.Interactive visual histories for vector graphics. MIT. Retrieved July, 11:2011, 2009. → page 93[165] J. Sukumaran and M. T. Holder. DendroPy: a python library forphylogenetic computing. Bioinformatics, 26(12):1569–1571, 2010. →page 14[166] J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, and Z. Su. Arnetminer:Extraction and mining of academic social networks. In Proc. 14th ACMSIGKDD Intl. Conf. Knowledge Discovery and Data Mining (KDD), page990–998, New York, NY, USA, 2008. doi:10.1145/1401890.1402008. →page 144[167] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei. LINE:Large-scale information network embedding. In WWW. ACM, 2015. →page 121[168] M. J. Telford. The animal tree of life. Science, 339(6121):764–766, 2013.→ page 209[169] E. Terzi and P. Tsaparas. Efficient algorithms for sequence segmentation.In Proc. SIAM Intl. Conf. Data Mining, pages 316–327. SIAM, 2006. →page 93[170] E. R. Tufte. The visual display of quantitative information. Graphics Press,2001. → page 57188[171] S. van den Elzen, D. Holten, J. Blaas, and J. J. van Wijk. Reducingsnapshots to points: A visual analytics approach to dynamic networkexploration. IEEE Trans. Visualization and Computer Graphics (TVCG),22(1):1–10, 2016. → page 16[172] L. Van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal ofmachine learning research, 9(11), 2008. → pages 34, 36, 143, 243[173] F. van Ham and B. Rogowitz. Perceptual organization in user-generatedgraph layouts. IEEE Trans. Visualization and Computer Graphics (TVCG),(6):1333–1339, 2008. → pages 60, 62[174] C. Vehlow, F. Beck, and D. Weiskopf. Visualizing dynamic hierarchies ingraph sequences. IEEE Trans. Visualization and Computer Graphics(TVCG), 22(10):2343–2357, 2016. → page 16[175] P. Velicˇkovic´, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio.Graph attention networks. arXiv preprint arXiv:1710.10903, 2017. → page126[176] B. Venkatachalam, J. Apple, K. St John, and D. Gusfield. Untanglingtanglegrams: Comparing trees by their drawings. IEEE/ACM Trans.Computational Biology and Bioinformatics (TCBB), 7(4):588–597, 2010.→ page 15[177] E. Wall, L. M. Blaha, L. Franklin, and A. Endert. Warning, bias may occur:A proposed approach to detecting cognitive bias in interactive visualanalytics. In IEEE Conf. Visual Analytics Science and Technology (VAST),pages 104–115. IEEE, 2017. → page 93[178] X. Wang, B. Lafreniere, and T. Grossman. Leveragingcommunity-generated videos and command logs to classify andrecommend software workflows. In Proc. ACM Conf. Human Factors inComputing Systems (CHI), page 285, 2018. → page 94[179] Y. Wang, Q. Shen, D. Archambault, Z. Zhou, M. Zhu, S. Yang, and H. Qu.Ambiguityvis: Visualization of ambiguity in graph layouts. IEEE Trans.Visualization and Computer Graphics (TVCG), 22(1):359–368, 2016. →page 60[180] Y. Wang, Z. Jin, Q. Wang, W. Cui, T. Ma, and H. Qu. Deepdrawing: Adeep learning approach to graph drawing. IEEE Trans. Visualization andComputer Graphics (TVCG), 26(1):676–686, 2019. → page 60189[181] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, et al. Image qualityassessment: from error visibility to structural similarity. IEEE Trans. imageprocessing, 13(4):600–612, 2004. → page 101[182] C. Ware, H. Purchase, L. Colpoys, and M. McGill. Cognitivemeasurements of graph aesthetics. Information Visualization, 1(2):103–110, 2002. → pages 59, 60, 62[183] N. J. Wickett et al. Phylotranscriptomic analysis of the origin and earlydiversification of land plants. Proc. National Academy of Sciences, 111(45):E4859–E4868, 2014. → pages 44, 45, 52, 202, 208[184] J. Wood, J. Dykes, and A. Slingsby. Visualisation of origins, destinationsand flows with OD maps. The Cartographic Journal, 47(2):117–129, 2010.→ page 137[185] C. Yang, Y. Xiao, Y. Zhang, Y. Sun, and J. Han. Heterogeneous networkrepresentation learning: A unified framework with survey and benchmark.IEEE Trans. Knowledge and Data Engineering (TKDE), 2020. → page 129[186] J. Yang, J. McAuley, J. Leskovec, P. LePendu, and N. Shah. Findingprogression stages in time-evolving event sequences. In Proc. 23rd Intl.Conf. World Wide Web (WWW), pages 783–794. ACM, 2014. → pages91, 93[187] L. Yang, C. Fang, H. Jin, M. D. Hoffman, and D. Estrin. Personalizingsoftware and web services by integrating unstructured application usagetraces. In Proc. 26th Intl. Conf. World Wide Web Companion (WWW),pages 485–493. Intl. World Wide Web Conferences Steering Committee,2017. → page 94[188] R. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec. Gnnexplainer:Generating explanations for graph neural networks. Advances in neuralinformation processing systems (NeurIPS, 32:9240–9251, 2019. → pages121, 127, 161, 163[189] T. Young, D. Hazarika, S. Poria, and E. Cambria. Recent trends in deeplearning based natural language processing. IEEE Computationalintelligence magazine, 13(3):55–75, 2018. → page 160[190] H. Yuan, H. Yu, S. Gui, and S. Ji. Explainability in graph neural networks:A taxonomic survey. arXiv preprint arXiv:2012.15445, 2020. → page 161190[191] J. Zhao, F. Chevalier, C. Collins, and R. Balakrishnan. Facilitatingdiscourse analysis with interactive visualization. IEEE Trans. Visualizationand Computer Graphics (TVCG), 18(12):2639–2648, 2012. ISSN1077-2626. → pages 12, 16[192] Z. Zhao, W. Benjamin, N. Elmqvist, and K. Ramani. Sketcholution:Interaction histories for sketching. Intl. Journal of Human-ComputerStudies, 82:11–20, 2015. → pages 90, 94, 95, 105[193] H. Zhou, P. Xu, X. Yuan, and H. Qu. Edge bundling in informationvisualization. Tsinghua Science and Technology, 18(2):145–156, 2013. →page 143191Appendix ASupplemental Material forADViewA.1 View coordination detailsTable A.1 documents the view coordination discussed in Section 2.6.6, showingwhich aspect of the data is visually encoded across all five levels of detail foreach of the eight views. The table has six columns since we break out branchesfrom leaves for clarity; both of these are the lowest level of detail. We duplicateFigure 2.7 and put it here for a quick reference showing all of these views, asFigure A.1.192View Level of detail (LoD)Tree collection Subset of treesIndividualtreeSubtreeBranch &its attributesLeaf nodeReferenceDendrogram whole viewcolorbackground line & tooltiptext label &black dotTree Distribution row segment in rowCluster AD whole view one clusterIndividual AD one AD blockline orcollapsedPairwisecomparison consensus treebutterflylayoutcolorbackground line & tooltiptext label &black dotTree Similarityt-SNEscatterplot dotRef. Br. Attr &Corr. Br. Attr.attribute table;histogramTree List text labelTable A.1: View coordination of visual encoding across all levels of detailand all views.Figure A.1: A screenshot of ADView as a reference to Table A.1.193Figure A.2: Illustration of gradient coloring in a cluster AD. The percentagerepresents the proportion of highlighted taxa in a block, and is encodedas a solid color in the individual AD, but gradient color in the resultingcluster AD.A.2 Cluster AD gradient coloringWhen grouping topologically identical individual ADs into a cluster AD, we useone of the AD layouts as a proxy for the cluster AD, but use gradient color toconvey the uncertainty of the proportion of taxa instead of the solid color in anindividual AD. The percentage of the color fill represents the proportion of high-lighted taxa in an AD block, which is fixed for individual AD but typically coversa variable range within a cluster AD. As shown in Figure A.2, block B has differ-ent proportions of orange color in the four individual ADs on the left, which canbe considered as a distribution [55%, 50%, 60%, 70%]. In the cluster AD on theright, the orange in block B has a fuzzy edge, which is perceived as the variance oruncertainty of the color proportion.We encode the complementary cumulative distribution function (c-CDF) of the per-centage of color fill with color saturation to achieve the fuzzy visual effect, whichwas first introduced as density strip by Jackson [88]. We do not choose the prob-ability density function (PDF), which is usually rendered as a histogram, nor thecumulative distribution function (CDF), because both of them could be misleadingwhen mapped to color saturation in our context, as illustrated in Figure A.3.194Figure A.3: Illustration of different encoding representation for the samenormal distribution (mean=50%, standard deviation is small). Colorsaturation is mapped to the value of the probability function shown in-side the block. Probability functions from left to right are the densityfunction, the cumulative distribution function, and the complementarycumulative distribution function.195A.3 Algorithm detailsWe supplement more detail about the aggregated dendrograms generation and ren-dering (Section 2.7) for the purpose of replication.A.3.1 AD layout function parametersFigure A.4 illustrates the parameters used in generating AD layouts. We use a best-effort mechanism to adapt the AD layout to user-specified resolution, described inSection 2.7.2. If an AD layout does not pass our legibility test, we will shrink someof the flexible parameters and re-generate a layout.There are two cascading sets of flexible parameters that can be changed: 1) thenumber of context levels, a metric to control how many context blocks to show; 2)inter-block gaps, branch lengths, block sizes.A.3.2 Front-end caching for rendering ADsThe rendering of aggregated dendrograms is handled by the front end; that is, thebrowser. It is not a trivial process, as shown in Figure A.5, and there might behundreds of ADs to compute. Therefore, to achieve reasonable response time touser interactions, especially for frequent ones, such as hovering over an AD, wecannot afford to run the whole pipeline from start to end.Notice that many frequent interactions do not affect the layout of ADs: for exam-ple, hovering only draws a black border around the AD, and resorting only affectsthe order of ADs. We can cache the intermediate results, namely, the ovals in Fig-ure A.5, so that a user interaction only triggers re-computation of the necessarysteps reusing some of the cached results.We use ReactJS 1 and Redux 2 as our rendering and interaction pipeline, and the1https://reactjs.org/2https://redux.js.org/docs/introduction/196Figure A.4: Illustration of parameters in an aggregated dendrogram layout.memoization functionality provided by the Reselect library 3. The memoizationkeeps an internal mapping between the state of data elements specified by the de-veloper and the computation results such as AD layouts. It detects if the statechanges, that is, if the state object is different from the previous one, and deter-mines to whether reuse stored results or trigger a re-computation. In Figure A.5,selecting a subtree in the reference tree leads to a re-computation of the wholepipeline because it changes the corresponding branches; changing the sorting or-der of AD will reuse the results of filtered AD layouts.3https://github.com/reactjs/reselect197Figure A.5: Pipeline on the frontend to render the aggregated dendrograms.The results in ovals are cached so that certain kinds of frequent userinteraction can take place without triggering re-computation of inter-mediate results that remain useable.198Participant P1 P2 P3 P4a, P4bPosition PhD student Bioinformatician Principal Investigator 2 PhD studentsIn-person interview Yes Yes Skype YesInterviewers ZP,SZ,TM ZP,SZ,TM ZP,SZ ZP,SZPrevious experiencewith ADViewchauffeured demo2 months agochauffeured demo8 months ago screenshots NoneOwn dataset Yes Yes Yes No; use multiple public datasetsType ofcomparisonSpecies treevs gene treesConsensus tree vsbootstrapping treesSpecies tree vsgene treesSpecies tree vs genetrees; Consensus treevs bootstrapping treesNumber of Trees 260 100 71 dozens to hundredsType of organisms Plants Parasites Algae Human pathogens; parasitesTable A.2: Relevant information of participants and datasets used in thestudy.A.4 Expert Interview StudyA.4.1 ParticipantsHere, we summarize the relevant information of all participants and their datasetswe recruited in Table A.2.A.4.2 Interview Questions1. Before today, how long have you been analyzing this dataset and with whattools? Did you generate it yourself or get it from someone else?2. Did you find any interesting biological insights in this dataset?(a) Could you confirm things that you already knew? And how long did ittake to see this compared to other tools?(b) Did you notice anything new in this dataset?3. What capabilities of the tool are useful for your research?4. Is there any functionality that is missing from the tool that would be useful?5. Are there aspects of the tool that are confusing, misleading or awkward?1996. In our corresponding branch matching algorithm, we currently compute thesimilarity metrics between every pair of branches and find the most similarbranch in the tree to the one in the reference tree.(a) Does this computation make sense to you? How closely does it matchyour mental model of how you compare two trees? Is there some otherway of thinking about support and conflicts to a specific clade in thereference tree that is an alternative to this kind of matching?(b) Our corresponding branch matching algorithm assumes that all treeshave a root. We know this assumption breaks in many cases, includingwhen the outgroup taxa are missing or you have a non-monophyleticoutgroup. Here’s an example of a tree where the root is wrong and youcan see that the matching is very messy. Which clade do you think isthe real match to A here? Or do you think that this question doesn’tmake sense and we shouldn’t even try to find a match in this case?(c) Do you think we should differentiate between the case where two cladesare very similar (although not exactly the same) from the case wheretwo clades are very different? At some point is it no longer useful toshow a ”best match” past some cutoff value of “too different”?7. Does the visual design of AD make sense to you and match your mentalmodel of how you think of a phylogenetic tree? The same problem withincorrect rooting affects the visual layout of the ADs. Do you think an ADwith incorrect rooting could mislead you?8. Does the tree distribution view make sense to you and match your mentalmodel of how you think about agreement and conflict between clades re-spectively?The tree distribution view sometimes has a very long tail with many sets thatcontain only a single tree. Is seeing all of these tiny sets helpful in terms ofconveying any interesting information to you, or would it be better to leavethat out? Do you have thoughts about whether it’s conveying informationabout the true biology, or about systematic errors in tree reconstruction, or200whether it could just be an artifact of the matching computation in our ownsoftware?9. Do think this tool (or an improved version of it) could be helpful for otherscenarios such as horizontal gene transfer? What kind of scenarios do youthink would be suitable for this tool?How often would you find a tool like this useful in the work that you do? (Forexample, every day, once a month, once a year. . . ?) When in a project lifecycle might you use something like this - at the beginning of a new project,after a specific phase of it, just before paper writing. . . ?)10. Do you have other comments about the tool?201Figure A.6: Full screenshot of ADView to explore four clades. After se-lecting four focal clades: A (blue): LAND PLANTS (LP), B (orange):ZYGNEMATOPHYCEAE (ZYGN), C (green): CHARALES (CHAR), D(red): COLEOCHAETALES (COL). We compared the cluster ADs withthe previous hypotheses to investigate which one (orange or green orred group) is the sister of LAND PLANTS.A.5 Full Screenshots of Usage Scenario 1: 1KP pilotstudyFigure A.6, Figure A.8, Figure A.9 and Figure A.11 are the full screenshots of theusage scenario in Section 2.8.3. We explored the two research questions in the 1KPpilot study [183] in ADView.A.5.1 Sister group of land plantsFigure A.6, Figure A.8, Figure A.9 are full screenshots to supplement the partialscreenshots in Section 2.8.3.202Figure A.7: Support for the most popular hylothesis (LP + ZYGN). Wechecked the distribution of support values for the most popular hypoth-esis (LP + ZYGN) by selecting the trees that have the LP + ZYGN cladeand created a sub-collection out of them. In the Corresponding Branchview, some trees have low support values for this hypothesis, whichmay render some doubts on whether LP + ZYGN is truly strongly sup-ported by this dataset.203Figure A.8: Pairwise comparison with the conflicting trees. We selected theconflicting segment in the second row (B) in tree distribution (wheretrees shown in brown background) and made a consensus tree, which isbeing compared head-to-head against the reference tree in full details.We presented the markers for the selected trees in the butterfly dendro-grams, which showed an outlier species SPIROGYRA is not includedin the orange (ZYGN) group. Notice that selected trees are highlightedacross multiple views in the interface.204Figure A.9: Combination of user-specified clades. We combined LP andCHAR into a user-specified taxa group. This feature is used for explor-ing hypotheses that are not presented in the reference tree, for examplethat there is no single subtree that consists of LP and CHAR in thisreference tree. The second cluster we selected represent the subset oftrees that group LP and CHAR together (solid border indicates exactmatches). With this, we checked the support values for their corre-sponding branches (the overarching branch above LP + CHAR) in thecorresponding branch attribute view on the bottom right. We found thatthere are a lot of low-support corresponding branches. In other words,some trees are not certain on grouping LP and CHAR together althoughthey appear so.205Figure A.10: Exploration of low-support branches. We noticed that sometrees have low support values (below 0.5), so we selected these treesusing the Corresponding Branch Attribute view, shown in the blackcircle at the bottom right. By displaying the tree names above theADs, we found that most of them are generated with the “Supertree”method, which might be worth investigation later.206A.5.2 Early diversification of land plantsIn Figure A.11, it is easy to find out that the monophylies of the four early lin-eages of LAND PLANTS (HORNWORTS, MOSSES, LIVERWORTS, and VASCULARPLANTS) are strongly supported because almost all trees agree with the referencetree in the Tree Distribution view. The widely accepted hypothesis Lv-basal inFigure A.12, that is, LIVERWORTS is the sister-group of all other LAND PLANTS,is rejected by most of the trees in this dataset. We notice that only two trees (the4th and 6th cluster AD) support Lv-basal.According to the 1KP paper, the widely accepted view that LIVERWORTS, MOSSES,and HORNWORTS are, respectively, successive sister groups to VASCULAR PLANTS,are not recovered in this dataset. In ADView, we can see that there are no such treespresenting this topology: (blue, (orange, (green, red))), which is a direct evidenceto support the statement in the 1KP paper. The first cluster AD is exactly the Hw-basal hypothesis in Figure A.12, and the second cluster AD is compatible with theBryo monophyletic group hypothesis.Biologists can then connect this evidence with their domain knowledge such aswhat substitution models are used to generate these trees and analyze their prosand cons. ADView presents the relevant information to them, in hopes that theyinterpret the visualization with biological judgment.207Figure A.11: Screenshot of ADView exploring early lineage of LANDPLANTS. A (blue): HORNWORTS (Hw); B (orange): MOSSES (Mo);C (green): LIVERWORTS (Lv); D (red): VASCULAR PLANTS (VP).Figure A.12: Previous hypotheses about early diversification of LANDPLANTS. Figure excerpted from the 1KP paper [183].208A.6 Usage scenario 2: TreeFamTreeFam [154] is a database of animal gene trees built from the genome sequencesof representative animal species. The data set spans the entire evolutionary his-tory of the phylum ANIMALIA. In this case, we utilized the TreeFam data todemonstrate how ADView may be used to discover gene trees that are concor-dant with well-established evolutionary relationships. More specifically, we ex-amined and identified gene trees consistent with two widely accepted splits in an-imal taxonomy: PROTOSTOMIA and DEUTEROSTOMIA [168]; ECDYSOZOA andLOPHOTROCHOZOA [4]. Both of these views have received support from multi-gene phylogenetic and phylogenomic studies [75, 134].Since the initial publication in 2006, there were several updates to TreeFam. Here,we used the latest release (4.0), which includes genome sequence data from 108animal species and one outgroup plant species (ARABIDOPSIS THALIANA). Wedownloaded the individual gene trees as well as a species tree that captures ma-jor evolutionary relationships in ANIMALIA from http://treefam.genomics.org.cn/.Because ADView cannot yet handle duplicate genes, we excluded gene trees withduplicate genes. Also, we only included gene trees with at least 20% of the taxarepresented. A final set of 1,317 gene trees was taken as input to ADView.Note that we collapsed the Individual AD view when we were only looking atthe Cluster AD and the Tree Distribution view, because there are more than 500individual ADs. Using only the Cluster AD view led to much better responsivenessto user interaction because there were so many fewer visual elements to handle.Also, this choice kept the full screenshots from being extremely long.209A.6.1 PROTOSTOMIA and DEUTEROSTOMIAFigure A.13 and Figure A.14 illustrate how the user can use ADView to visu-ally confirm that many of the gene trees are consistent with the classical tax-onomic thought that PROTOSTOMIA and DEUTEROSTOMIA are the two majormonophyletic branches of BILATERIAN animals. As illustrated in Figure A.13, byselecting the two clades as A (PROTOSTOMIA) in blue and B (DEUTEROSTSOMIA)in orange, we found subsets of gene trees entirely or partially consistent (i.e., withmissing taxa) with the species tree. The evolutionary relationship between A andB is reflected in the clustered ADs shaded with brown backgrounds. We also ob-served several other clustered ADs in which we did not find any support for RQ3.The first clustered AD (leftmost) contains 751 gene trees having only taxa from B;similarly, the fourth clustered AD contains 77 gene trees having only taxa from A.This situation immediately reveals genes unique to either A or B, due to a biologicalprocess (clade-specific gene gain or loss) or incomplete sampling (imperfect datacollection). The other cluster ADs capture seemingly discordant gene trees; how-ever, further exploration of the gene trees under the Individual Aggregated Den-drogram view and pairwise comparison of gene trees, as shown in Figure A.15 andFigure A.16, revealed that the discordant signals likely resulted from inadequatepost-processing of the gene trees. The TreeFam pipeline did not remove “rogue”taxa, which are outlier taxa spuriously inserted into the wrong clades. TreeFamwas assembled a decade ago, therefore the results of the pipeline do not reflect thebest practices in phylogenomics today. Removal of rogue taxa (e.g., using Rogue-NaRok [2]) should eliminate most of the discordant gene trees.Additionally, we checked whether PROTOSTOMIA and DEUTEROSTOMIA are sis-ter clades; that is, whether A and B form a monophyletic group. By selecting C(Figure A.14), we found most of the gene trees (1,163) to be concordant with themonophyly of PROTOSTOMIA + DEUTERSTOMIA. We further explored the genetrees under the Individual AD view (not shown), and found that the discordantgene trees might be caused by a variety of reasons (e.g., rogue taxa, incompletesampling, or stochastic and systematic errors in phylogenetic methods).210Figure A.13: The major clades PROTOSTOMIA (A) and DEUTEROSTSOMIA(B) are selected. This view provides overall gene tree support for themonophyly of PROTOSTOMIA and that of DEUTEROSTSOMIA. Thesupporting clustered ADs are highlighted with brown backgrounds.The Individaul AD view is collapsed because we were not focusing onany individual trees at this point. Collapsing the Individual AD viewalso results in faster response because there were many fewer elementsto render and keep track of for the browser.211Figure A.14: Screenshot of ADView for exploring PROTOSTOMIA andDEUTEROSTSOMIA. Both PROTOSTOMIA and DEUTEROSTSOMIAare selected as a single group (C). This combination reveals most ofthe gene trees are consistent with the monophyly of BILATERIAN ani-mals (C).212Figure A.15: We selected the discordant trees excluding trees that are miss-ing PROTOSTOMIA or missing DEUTEROSTSOMIA, and created asub-collection, that is, a subset of the tree collection. We wanted tofind out why these trees do not behave as expected.213Figure A.16: Pairwise comparison is helpful in locating the potential causesof the discordance. The orange outlier at the top shows that B hasa rogue gene in the gene tree (right dendrogram) compared to thespecies tree (left dendrogram).214A.6.2 ECDYSOZOA and LOPHOTROZOANext, we performed the same tasks, but we revisited the well-established ideasof ECDYSOZOA and LOPHOTROZOA being the two major monophyletic branchesof PROTOSTOMIA. We selected A (ECDYSOZOA) and B (LOPHOTROZOA), butfound only small clustered ADs (highlighted in brown background) consistent withtheir monophyly, as shown in Figure A.17. We then selected only ECDYSOZOA orLOPHOTROZOA separately. When we selected only ECDYSOZOA (A), we foundstrong support for its monophyly, corroborated by most of the gene trees, as shownin Figure A.18. When we selected only LOPHOTROZOA (not shown), however,we observed no strong support in TreeFam for its monophyly, probably becausethere are too few representative taxa included to yield robust phylogenetic signals.Indeed, later studies involving more LOPHOTROZOA taxa produced strong supportfor its monophyly [75, 134].215Figure A.17: Screenshot of ADView for exploring ECDYSOZOA andLOPHOTROZOA. The major clades ECDYSOZOA (A) andLOPHOTROZOA (B) are selected. The clustered ADs in support oftheir monophylies are highlighted with brown backgrounds.216Figure A.18: Cluster ADs showing distributions of ECDYSOZOA. OnlyECDYSOZOA (A) is selected. The clustered AD, which contains mostof the gene trees, corroborating its monophyly is highlighted with abrown background.217Figure A.19: Screenshot of the tool developed by Bremm et. al. [19], on the35-tree dataset from their paper.A.7 Screenshots of Information Density Comparison218Figure A.20: Screenshot of ADView with the same dataset at the same screenresolution, with three subtrees selected. Information about exactlywhat taxa their domain experts explored is not available from the pa-per, so we randomly picked three subtrees.Figure A.21: Screenshot of the tool developed by Bremm et. al [19]. Ele-ments with similarity score below 0.5 are filtered out.219A.8 Case StudiesA.8.1 Full Screenshots of Case Study 1Figure A.22: P1 was exploring the position of the T10 taxon with regardto a bigger monophyletic group: the direct parent of the four focalsubtrees, namely T10 itself (blue) and three sibling groups (orange,green, red). The position of the blue T10 group varies; there are manycluster ADs with a relatively small number of trees in each. She alsoobserved the same by skimming the individual ADs.A.8.2 Full Screenshots of Case Study 2220Figure A.23: P2 identified an interesting taxon BLASTOCYSTIS HOMINIS,shown as the orange group, that usually lives in pigs but was placedclose to human related strains.221A.8.3 Case Study 3P3 is a frequent collaborator of our domain expert co-author, and they were work-ing together on a dataset comparing 1 species tree against 71 gene trees of 115 redalgaes. During the study, they were able to locate some misbehaved missing taxa inthe reference tree and outliers in the gene tree, but could not continue the analysisfurther due to a missing engineering feature of the interface (selecting a tree by itsname). After we added that feature, they used ADView independently for severalweeks and made some interesting biological discoveries.They first compared four species trees created with different methods and kinds ofsequences, mainly using the Pairwise Comparison view, to identify the one that ismost consistent with the current literature (T3), as shown in Figure A.24. Next,using that best tree among the four as a reference, they compared it against 155gene trees built from amino acid sequences. They sought artifacts, such as unusu-ally long branches, and anomalies, primarily “rogue taxa” where outlier leaves arespuriously inserted into the wrong subtree. They quickly scanned over the genetrees to get some insights about where discrepancies typically arise (T4 and T5).Unsurprisingly, discrepancies seem to be concentrated at deep nodes (e.g., CYANI-DIALES), because in many genes the information needed to infer deep relationshipshas been eroded or lost. They also observed that a three-taxa subclade of POLYSI-PHONIA, has undergone exceptionally high molecular evolution in a gene “rpl21”,as shown in Figure A.25. The conflict between rpl21 gene tree and the referencetree on the POLYSIPHONIA subclade is an alert to be cautious when interpretingsignals from POLYSIPHONIA, well-known for uncertain phylogenetic placement.Their future work is to aggregate the results of the molecular evolution analysisacross the 155 genes, and they plan to check the gene trees in ADView for detailsthat may affect their interpretation of the results.222Figure A.24: P3 and our biologist co-author first compared four species treesinferred from different methods and different kinds of molecular se-quences in order to choose the best one for later analysis.Figure A.25: P3 and our biologist co-author used the best tree from the pre-vious step as the reference tree, and compared it against 155 genetrees. They found the blue group is inconsistent across gene trees. Anexample (gene “rpl21”) is shown in the figure.223A.9 Screenshots of Dataset UploadUsers can upload their own datasets through a dedicated tab in the client browserinterface. There are two steps to upload a dataset: upload the tree files, as shownin Figure A.26, and specify an outgroup if the trees are unrooted, as shown inFigure A.27. The server then performs indexing, tree distance calculation, andcorresponding branch matching, as shown in Figure A.28.Finally, the newly uploaded dataset is added to the list of all uploaded datasets,available through another tab.224Figure A.26: The first step of uploading a dataset: specify the files for areference tree and a collection of trees.225Figure A.27: The second step of uploading a dataset: select an outgroup byeither checking the names, clicking on the branches (as shown here),or paste a text file.226Figure A.28: server is pre-processing data.227Appendix BSupplemental Material forSprawlterB.1 Parameter analysisWe present the analysis for two parameters in the penalty mapping functions thatare defined in Section 3.4.3:f NNv1,v2(x) = (1−α)(2x)0.7+αM0.7v1,v2 (0≤ x≤Mv1,v2) (B.1)f NEv,e (x) = 2(1−α)x+αMv,e (0≤ x≤Mv,e) (B.2)f EEG (x) = (16pi2−4α)x2+α pi24(0≤ x≤ pi2) (B.3)228B.1.1 Minimum penalty fraction: αIn Section 3.4.4, we describe the semantics of the minimum penalty fraction, α ,that appears in Equation B.4. The minimum penalty fraction is a trade-off betweenthe ability to distinguish no overlap from touching overlap and the ability to distin-guish the different amount of overlap.(NN) β = (1− 1.5.7)α+1.5.7≈ 1.625−0.625α (0 < α < 1)(NE) β = 2−α (0 < α < 1) (B.4)(EE) β =16pi2−3α (0 < α < 16−2pi3pi2≈ 0.328)Theoretical analysisWe show the function plot for NN overlap between bigger nodes, as mentioned inSection 3.4.4.0 2 4 6 8 10x: NN overlap012345678f(x): Penalty0.010.200.400.600.800.99Figure B.1: The NN penalty mapping function of a big metanode pair thatis 10x the size of the smallest node (M = 10), with different minimumpenalty fraction (α) values.229Computational analysisWe validate the theoretical analysis and generate practical suggestions with our im-plementation and three groups of synthetic small graph layouts, namely, progression-NN, progression-NE, and progression-EE.We tested the influence of α for NN, NE, and EE separately as it is not necessaryto use the same α for all three cases. We used six different α values within itsvalid range defined in Equation B.4: a near-minimum, a near-maximum, and fourevenly-spaced values in between.The progression-NN has a series of graph layouts with minimum, some, and near-max node-node overlap for either leaf nodes only, metanodes only, or both, asshown in Figure B.2. For each α , we can compare the area-aware metrics betweendifferent layouts to obtain a practical sense of whether the differences in penaltiesare too little, too much, or just enough to distinguish the layouts. The same analysisalso apply to NE (Figure B.3) and EE (Figure B.4).With the extremely small α value 0.01, the near-minimum penalty for the touch-ing overlap between leaf nodes are only 0.5 (average penalty is P/C = 0.50/4 =0.125), which is too little to distinguish it from no overlap at all. With α valuesthat are larger than 0.8, the differences between penalties for touching, some, andnear-max overlap of leaf nodes are too tiny (3.29→ 3.91→ 4.24 when α = 0.80),and differences for metanodes are also too tiny (12.40→ 15.34). Note that thelast difference seems sufficient (15.34→ 27.24) only because the last layout hasoverlaps of both leaf node pairs and metanode pairs. With α values in between, thedifferences between penalties are generally sufficient to tell these layouts apart.The choice of α is subjective to user preference, and they can have different α val-ues for NN, NE, and EE, as long as it is within the range specified by Equation B.4.For simplicity, we use α = 0.20 for all three cases.230Figure B.2: Influence of minimum penalty fraction (α) tested on a small 2-level synthetic layout with increasing node-node overlap of leaf nodepairs and metanode pairs.Figure B.3: Influence of minimum penalty fraction (α) tested on a small 2-level synthetic layout with increasing node-edge overlap of leaf node-edge pairs and metanode-edge pairs.231Figure B.4: Influence of minimum penalty fraction (α) tested on a smallsingle-level synthetic layout with decreasing edge-edge crossing an-gles.232B.1.2 EE curve shape: γFor the EE case, we compare the quadratic function (Equation B.3) against a linearfunction, stated as follows.f EEG (x) = (4pi−2α)x+α pi2(0≤ x≤ pi2), where 0 < α <2pi≈ 0.637(B.5)Theoretical analysisTo separate the influence of α and γ , we compare the EE linear function with thequadratic one using the same α , but repeat the comparison for multiple different αvalues, as shown in Figure B.5. Since the domain of α is different for linear func-tion, (0,0.637), and quadratic function, (0,0.328), we only compare the functionswith valid α values for both, i.e., 0.01, 0.1, 0.2, 0.3.We observe that the basic shapes and relationships of the two functions stay thesame across different α values, despite that the touching penalty and range ofpenalty vary (in the way we described in the previous section). The absolute dif-ference between the two functions decreases as α increases, due to the decrease inthe range of penalty. The relative difference between two functions becomes largeonly when the complementary angle x is nearly maximum, pi/2, i.e., the glancingangle; otherwise, they are very close to each other.Therefore, we can choose either the linear or quadratic function depending on howheavy the glancing angle should be penalized.Computational analysisWe computed the area-aware metrics using both linear and quadratic penalty map-ping functions on all 52 layouts we collected. Figure B.6 shows the penalties com-2330π/4π/2x024f(x)(π/4,1)α=0.010π/4π/2x02f(x)(π/4,1)α=0.100π/4π/2x012f(x)(π/4,1)α=0.200π/4π/2x01f(x)(π/4,1)α=0.30linearquadraticFigure B.5: Two different curve shapes (linear and quadratic) of the EEpenalty mapping function.101103105107log EE penalty (linear)101103105107log EE penalty (quadratic)Figure B.6: EE area-aware metric with linear penalty mapping functionagainst that with the quadratic function, using log scale on both axes.puted with the linear function (x-axis) compared to that with the quadratic function234(y-axis). Each dot represents a graph layout in our dataset. We used log scales forboth axes to de-clutter the dots as there are many small synthetic layouts. We canclearly see that the two penalties are linearly correlated. The Pearson correlationbetween the two penalties is 0.9997, and the slope of the linear regression line is1.17, which indicates that penalties with the quadratic function are slightly bigger.In general, the difference between the two functions are very small.We also sorted the layouts by the difference between the penalties using the twofunctions, and manually inspected a few layouts with the biggest difference. Wefound out that in these layouts, the angles between crossing edges are smaller thanthose in other layouts, and there are many near-glancing angles. This discoveryalso confirms our theoretical analysis: difference only becomes relatively large onsmall crossing angles (i.e., large complementary angles).We chose the quadratic function for our analysis as Huang et al. found that thequadratic trend is more relevant to user performance in their study [83].B.2 Pseudo-code for NE and EE area-aware metric235Algorithm 5: Computation of node-edge area-aware metrics.Input : G = (V,E)Output: total penalty PNE(G) and count CNE(G)1 totalPenalty← 02 count← 03 for v ∈V do4 for e ∈ E do5 if v /∈ e.ends && !IsAncOrDesc(v, e.ends) then6 if CheckIntersection(v, e) then7 x← ComputeOverlapLength(v, e)8 penalty← PenaltyMapFunc(x, v, e)9 totalPenalty← totalPenalty + penalty10 count← count + 111 return totalPenalty, countAlgorithm 6: Computation of edge-edge area-aware metrics.Input : G = (V,E)Output: total penalty PEE(G) and count CEE(G)1 totalPenalty← 02 count← 03 for e1 ∈ E do4 for e2 ∈ E do5 if e1 6= e2 then6 if CheckIntersection(v, e) then7 x← ComputeCrossingAngle(e1, e2)8 penalty← PenaltyMapFunc(x)9 totalPenalty← totalPenalty + penalty10 count← count + 111 return totalPenalty, count236Type Name (source) #Nodes #Edge #Layouts Layoutalgorithm PurposeSyntheticfour-clusters 40 123 10Manualdebuggingproxy shape 19 40 2 exploration of proxy shapesmeta-edges 1 43 114 2 meta-edges overlapmeta-edges 2 42 82 1 meta-edge and node overlaptouching 12 18 2 minimum overlapprogression NN 40 123 7 analysis of minimum penaltyfraction (α) for NNprogression NE 40 123 6 analysis of minimum penaltyfraction (α) for NEprogression EE 8 8 4 analysis of minimum penaltyfraction (α) for EEsingle-levelvariable node-size 31 92 4Davidson-harel,FM3, Linlog,stress marjorizationcheck metrics performance onthis widely-used type of graphReal-worldcoauthor-big [87] 1538 8040 5GEM [52],FM3 [67],Grouseflocks [7],Koala [87]various situation in large layoutcoauthor-small [7] 103 505 4email-eu-core [102] 986 15957 4partition-add32 [160] 4960 9462 5Total 13 56Table B.1: Full list of graph layoutsB.3 List of graph layoutsSee Table B.1.237B.4 Computational timeTable B.2 shows detailed running times on each layout for three metric families(NN, NE, and EE) and two approaches (AS vs. count), averaged over 4 runs. Theslowdown factor is the ratio between running time of AS and count, representinghow much the AS metric is slower than traditional count-based metrics. It is com-puted only if the running time is greater than 1 second, in order to reduce timingerrors of the operating system. The average slowdown at the last row is computedover the valid rows.238NN NE EELayout sprawlter count slowdown sprawlter count slowdown sprawlter count ratiofour-clusters-ne0 0.01 0.01 NA 0.06 0.04 NA 0.10 0.06 NAfour-clusters-ne1 0.01 0.01 NA 0.07 0.05 NA 0.12 0.07 NAfour-clusters-nn0 0.01 0.01 NA 0.06 0.04 NA 0.10 0.06 NAfour-clusters-nn1 0.01 0.01 NA 0.08 0.05 NA 0.10 0.06 NAfour-clusters-nn2 0.01 0.01 NA 0.09 0.06 NA 0.10 0.06 NAfour-clusters-nn3 0.01 0.01 NA 0.07 0.04 NA 0.09 0.06 NAfour-clusters-nn4 0.01 0.01 NA 0.08 0.05 NA 0.10 0.06 NAfour-clusters-sprawl 0.01 0.01 NA 0.06 0.04 NA 0.09 0.06 NAfour-clusters-ee0 0.01 0.01 NA 0.10 0.05 NA 0.18 0.07 NAfour-clusters-ee-glancing 0.01 0.01 NA 0.11 0.05 NA 0.18 0.08 NAspecial-cases-1 0.00 0.00 NA 0.01 0.01 NA 0.01 0.01 NAspecial-cases-2 0.00 0.00 NA 0.01 0.01 NA 0.01 0.01 NAspecial-cases-3 0.01 0.01 NA 0.08 0.04 NA 0.10 0.05 NAspecial-cases-4 0.01 0.01 NA 0.07 0.04 NA 0.09 0.05 NAspecial-cases-5 0.01 0.01 NA 0.05 0.03 NA 0.05 0.03 NAspecial-cases-6 0.00 0.00 NA 0.00 0.00 NA 0.00 0.00 NAspecial-cases-7 0.00 0.00 NA 0.00 0.00 NA 0.00 0.00 NAprogression-nn-none 0.01 0.01 NA 0.06 0.04 NA 0.10 0.06 NAprogression-nn-touch-leaf 0.01 0.01 NA 0.07 0.06 NA 0.10 0.09 NAprogression-nn-touch-meta 0.01 0.01 NA 0.07 0.04 NA 0.10 0.10 NAprogression-nn-some-leaf 0.01 0.01 NA 0.06 0.07 NA 0.10 0.09 NAprogression-nn-some-meta 0.01 0.01 NA 0.08 0.06 NA 0.09 0.08 NAprogression-nn-near-max-leaf 0.01 0.01 NA 0.06 0.05 NA 0.09 0.07 NAprogression-nn-near-max-meta 0.01 0.01 NA 0.09 0.07 NA 0.11 0.10 NAprogression-ne-touch-leaf 0.01 0.01 NA 0.06 0.06 NA 0.09 0.06 NAprogression-ne-touch-meta 0.01 0.01 NA 0.06 0.04 NA 0.09 0.07 NAprogression-ne-some-leaf 0.01 0.01 NA 0.06 0.06 NA 0.09 0.08 NAprogression-ne-some-meta 0.01 0.01 NA 0.07 0.04 NA 0.11 0.06 NAprogression-ne-near-max-leaf 0.01 0.01 NA 0.06 0.04 NA 0.09 0.06 NAprogression-ne-near-max-meta 0.01 0.01 NA 0.08 0.05 NA 0.13 0.08 NAprogression-ee-ortho 0.00 0.00 NA 0.00 0.00 NA 0.00 0.00 NAprogression-ee-half 0.00 0.00 NA 0.00 0.00 NA 0.00 0.00 NAprogression-ee-near-glancing 0.00 0.00 NA 0.00 0.00 NA 0.00 0.00 NAprogression-ee-glancing 0.00 0.00 NA 0.00 0.00 NA 0.00 0.00 NAvaried-davidson-harel 0.01 0.01 NA 0.04 0.02 NA 0.05 0.03 NAvaried-fast-multipole-emb 0.01 0.00 NA 0.04 0.02 NA 0.06 0.03 NAvaried-linlog 0.01 0.00 NA 0.04 0.02 NA 0.05 0.03 NAvaried-stress-majorization 0.01 0.00 NA 0.03 0.02 NA 0.05 0.04 NAcoauthor-grouseflocks-0 0.01 0.01 NA 0.23 0.12 NA 0.45 0.24 NAcoauthor-grouseflocks-1 0.01 0.01 NA 0.28 0.12 NA 0.43 0.22 NAcoauthor-grouseflocks-2 0.02 0.01 NA 0.94 0.42 NA 2.22 1.19 1.87coauthor-gem 13.83 9.46 1.46 139.26 97.92 1.42 420.14 293.56 1.43coauthor-koala 12.02 8.85 1.36 197.67 144.26 1.37 966.99 531.41 1.82snap-email-grouseflocks-1 0.02 0.01 NA 2.03 1.00 NA 43.93 20.65 2.13snap-email-grouseflocks-2 0.03 0.02 NA 6.63 3.23 2.05 198.30 86.78 2.29snap-email-gem 4.97 3.64 1.37 212.53 150.01 1.42 4147.91 2197.89 1.89snap-email-koala 4.93 3.69 1.34 331.11 240.04 1.38 10252.12 4196.38 2.44ivOrigins-grouseflocks-2 0.01 0.01 NA 0.25 0.12 NA 1.33 0.61 NAivOrigins-grouseflocks-4 0.00 0.00 NA 0.04 0.02 NA 0.03 0.02 NAivOrigins-gem 0.05 0.04 NA 0.63 0.48 NA 2.77 1.49 1.86ivOrigins-koala 0.05 0.04 NA 1.09 0.73 NA 6.64 2.99 2.22partition-add32-grouseflocks-0 0.21 0.16 NA 1.31 0.82 NA 1.08 0.78 NApartition-add32-grouseflocks-1 0.25 0.18 NA 2.09 1.38 1.51 1.93 1.40 1.37partition-add32-grouseflocks-5 0.64 0.40 NA 5.84 3.73 1.57 6.24 4.22 1.48partition-add32-fm3 127.99 92.38 1.39 504.51 358.67 1.41 510.11 372.16 1.37partition-add32-koala 130.81 94.17 1.39 773.12 560.11 1.38 1604.56 880.74 1.82average 1.38 1.50 1.85Table B.2: Computation time of each layout for three metric families and twoapproaches.239B.5 Computational pipelineThe full computational pipeline that we have implemented includes multiple datapreparation steps in addition to the sprawlter metrics themselves, plus other metricsfor comparative analysis purposes.1. Convert graph format (Python). We convert a variety of input graph for-mats into Tulip format, as Tulip [10] provides several useful layout algo-rithms.2. Apply layout algorithms. For the synthetic graphs, we manually positionthe nodes to create various degree of clutter and sprawl, and also apply fourdifferent layout algorithms within the version 5.2.1 of Tulip [10] (fast mul-tipole embedder, Davidson-Harel, Linlog, stress majorization). For the real-world graphs, we apply two layout algorithms in Tulip (GEM, FM3); we usethe 2008 version of GrouseFlocks [7], and a version of Koala [87] modifiedto save in Tulip format. The output in all cases is the geometric informationof nodes and edges (position, area, shape, length) in Tulip format.3. Extract geometries and node hierarchy (Python). We then parse the 56layouts in Tulip format, extract the information of geometry and node hier-archy, and store it in JSON format.4. Compute metrics (Python). We implement the sprawlter metrics for theNN (algorithm 4), NE, and EE families and also their respective count-basedmetrics, and also the global readability metrics of node-node overlap andcrossing angle of Dunne et al [39]. We use the Shapely Python package [157]for manipulation of geometry.5. Display and analyze results for comparison. We display the computedmetrics of each layout in a table with HTML and JavaScript, and we writePython scripts to analyze parameters and computational time.GrouseFlocks was developed over ten years ago, and while it still functions as aninteractive exploration tool for multi-level graph layouts, the coloring of metanodes240is no longer correct (possibly due to deprecated dependent libraries). In orderto have easy-to-understand results figures, we exported Tulip-format output filesfrom GrouseFlocks into the current Tulip, saved them in SVG format, and placedthe metanodes with the correct colors back to their original positions in AdobeIllustrator.In the supplemental materials, we include all input files (in JSON format after step3), output files (results of step 4), implementation code, and Python scripts forresults analysis.241Appendix CSupplemental Material forLogSegC.1 FeaturesC.1.1 CommandsWe exported 100 million events which contained 1949 unique commands from thedatabase in May 2018. The command frequencies follow a long-tail distribution:a few commands are frequently used, whereas many commands are rarely used.We inspected the distribution of commands ordered by frequencies as a data qual-ity check, particularly those at the head of the distribution. Examples of the mostfrequent commands are Open, Move, and Crop. The actual number of uniquecommands in the database is larger than the number of menu items in the PS inter-face (about 1400), due to non-standard commands like Photoshop (PS) Actions1,and multilingual versions of the same command. Many of the infrequent ones are1Actions and the Action Panel in Photoshop: https://helpx.adobe.com/photoshop/using/actions-actions-panel.html242the user-defined Actions that are personal rather than universal and thus are notuseful for us.We also note that the command name alone does not always contain enough seman-tic information to fully disambiguate user intention or the task at hand, however.For example, Brush Tool, one of the most frequently used commands, can beused to either paint colors on canvas or edit a layer mask.Grouping these events by session identifier and ordering them by timestamp, wereconstructed 169,387 sessions. We specifically define a session to be a temporalsequence of events triggered by a user when completing a PS task.We used word2vec [122] to learn a vector for each command. We conducted asanity check of the embedding space using the Google Embedding Projector [158]with t-SNE [172] to inspect some of the clusters in the interface, and found that thestructure of this space was indeed plausible. For example, in Figure C.1, the redcircle contains many commands for changing layer blending options.Figure C.2 shows the neighborhood around the Brush Tool command, whichalso has plausible command names such as Eyedropper Tool and MixerBrush Tool.Finally, Figure C.3 shows the neighborhood of the New Color Fill Layercommand, which creates a new adjustment layer. Again, nearby commands such asModify Color Fill Layer and Modify Levels Layer seem to cap-ture reasonable similarity.243Figure C.1: Embedding space neighborhood of Linear Burn command.244Figure C.2: Embedding space neighborhood of Brush Tool command.245Figure C.3: Embedding space neighborhood of New Color FillLayer command.246Figure C.4: Layer hierarchy examples.C.1.2 LayerLayer hierarchy examples from three sessions (S1, S2, S14) are shown in Fig-ure C.4. The annotations on the left show examples of layer relationships and thecorresponding layer similarities.The layer similarity feature computation requires complete information about the247layer hierarchy as it has evolved up to any specific time point, so it is derived usinginformation spread across multiple individual event attributes.The overall equation for layer similarity between two layers A and B: If A and Bare identical layers, the similarity is maximum (1.0); otherwise, it is the sum oflayer similarity for duplicate layers, adjustment layers, and layer group, capped by1.0. Note that in these equations, A and B are commutative.S(A,B) ={1.0 i f A = Bmin(1.0,Sdup(A,B)+Sad j(A,B)+Sgrp(A,B)) i f A 6= B(C.1)The similarity for duplicate layer is computed as follows.Sdup(A,B) ={0.8 i f A duplicates B0 otherwise(C.2)The similarity for adjustment layer is computed as follows.Sad j(A,B) ={0.5 i f A is ad justment layer o f B0 otherwise(C.3)The similarity for layer group is computed as follows. The length of path betweenA and B in the layer hierarchy is denoted as d. Most cases are direct siblings(d = 2), and the layer similarity would be 0.5.Sgrp(A,B) ={12d−1 i f A and B are in the same layer group0 otherwise(C.4)C.2 Data Collection248Figure C.5: 8 collected sessions in the first round.249Figure C.6: 8 collected sessions in the second round.250Task type#Sessions in1st round#Sessions in2nd roundTotalPoster creation 4 4 8Portrait retouching 3 3 6Special effect creation 1 1 2Table C.1: Summary of task type251C.3 Hyperparameter Tuning for SVMWe briefly review how a trained SVM model predicts a label given an event. It usesa kernel function that takes the feature vector as input and gives signed distancefrom the data point to a separating hyperplane as output, which is then convertedto the probability of true prediction. Finally, it compares this probability score to athreshold t (model hyperparameter): if probability is greater than t, then this eventis predicted as boundary.There are two hyperparameters in our model: window size k, the number of previ-ous events used in the feature vector, and threshold t, to determine the probabilityvalue where an event is predicted to be the boundary. We train the model on thetraining set, and measure performance on the validation set to tune these hyperpa-rameters, following common practice in machine learning.C.3.1 Window size kFor each window size k from 1 to 10, we assess model performance after modeltraining by computing and plotting its receiver operating characteristic (ROC)curve, shown in Figure C.7. The ROC curve is drawn by varying the hyperpa-rameter threshold t from 0 to 1 and plotting a point on the curve for each value oft. The x axis is the false positive rate, and the y axis is the true positive rate forthat threshold value. The area under curve (AUC) of a ROC curve indicates theprobability that the model will rank a randomly chosen boundary event higher thana randomly chosen non-boundary event. A common interpretation is that the largerthe area is, the better performance the model has. From qualitative inspection ofFigure C.7, we can see that the 10 curves are closely intertwined with each other,and they are all far above the diagonal curve representing random guesses, indi-cating that they are similarly good classifiers. Quantitatively, we can see that theAUCs are also highly similar (min: 0.938, max: 0.962).We conclude that window size does not influence model performance, and thus wechoose k = 1 to reduce the size of feature vectors.252Figure C.7: ROC curves for window size k.C.3.2 Threshold tAfter choosing the window size, we investigate the performance of the model withk=1 under different threshold values. We plot precision (red line) and recall (blueline) against threshold t (x-axis) in Figure C.8. The trade-off between precisionand recall is easy to spot: as threshold increases, precision increases but recalldrops. Considering the downstream applications of the segmentation model, wefavor recall over precision. That is, we are willing to trade precision for higher re-call because there are many fewer positive labels (boundaries) than negative labels(non-boundaries) in the dataset. It is important to correctly identify as many bound-aries as possible, and false negatives (boundaries predicted as non-boundaries) areundesirable as it is not easy for end users to identify these missed boundaries. Onthe other hand, false positives (predicting non-boundaries as boundaries) are lessdetrimental: although these errors lead to over-segmentation of the logs, users can253Figure C.8: Performance threshold curve for threshold t.quickly dismiss them through an interactive interface. We use the F2 score (thickblack line), which weighs recall twice as much as precision, to quantify this trade-off choice.From Figure C.8, we find a clear winner, t = 0.24, which yields the highest F2score (0.74), as the threshold for our lowest-level segmentation. The correspondingrecall is 0.84, and precision is 0.50.254C.4 ResultsC.4.1 Labeling interfaceFigure C.9 shows the interface used by authors of the CHI paper (Zipeng Liu andZhicheng Liu) when labeling the sessions. It shows the logged attributes (layer,command, image content, duration) and a few derived attributes such as the diffimage.255Figure C.9: Labeling interface on a scene creation session (S1).256C.4.2 Visualization of sessions in test setIn Section 4.8 we show and inspect 2 out of the 5 sessions (S10, S15) in the testset, and here we show the other 3 test sessions (S12, S13, S14). We include each ofthe 16 sessions as high-resolution individual PNG files in supplemental materialsas well.Figure C.10 and Figure C.11 show the visualization of session S12, for a postercreation task. The participant works in a highly organized way, which results inmany levels in the human labels. Most of the boundaries in human labels (gapsbetween grey chunks) for both low- and high-level are also predicted as boundariesin red and blue.There are a few low-level missing boundaries. For example, in the top highlightedbox, participant moved a sphere icon (there are 3 icons in total) using only oneevent, and then he continued adjusting the positions of the three icons. The modelfails to detect this boundary as it thinks the participant is still working on theicons in the icon layer group (high layer similarity) with highly similar commands(Move, FreeTransform).There are many over-segmentations. In the bottom highlight box, the participanttries to put a few layers into a layer group, but they were having trouble locatingthem in the layer panel, resulting in a series of events on unrelated different layersuntil he found the correct ones. In this case, the user is having trouble keepingtrack of the interface, so it would naturally results less structure in the logged data.257Figure C.10: Analysis of segmentation results for session S12 (upper part).258Figure C.11: Analysis of segmentation results for session S12 (lower part).259Figure C.12 and Figure C.13 show session S13, a special effect task to turn a dayimage of a city street into night. The computational results align with the humanlabels very well for this session, but it comes with a lot of over-segmentation. In thehighlighted box, the participant frequently copies and hides a layer as its backup asa way of smart version control: marking the milestone states such that they couldgo back at anytime later. It involves a lot of layer switching and toggling, leadingto many single-event low-level chunks in the computed results.However, in the high-level chunks (red), the model successfully removes the over-segmentation while keeping the true boundaries in the human labels.260Figure C.12: Analysis of segmentation results for session S13 (upper part).261Figure C.13: Analysis of segmentation results for session S13 (lower part).262Figure C.14 and Figure C.15 show session S14, a portrait retouching task to restorean old photo. For this session, there are noticeably less chunks in both computedresults and human labels. It is also noticeable that the precision is high and recallis low (compared to other sessions). The many missing boundaries (highlighted)happen during the first half of the session, where the participants were fixing crackson different areas on the photo (upper part, head and face, bottom left, back to face,and clothes) but on the same layer. Each event only fixes a small amount of pixels,which leads to the model’s failure to capture the change of semantic areas of thephoto. In the second half of the session, where the participants were working on theeyes using multiple copy-and-paste phases, the model is able to find true structuresthat aligned with the human labels.263Figure C.14: Analysis of segmentation results for session S14 (upper part).264Figure C.15: Analysis of segmentation results for session S14 (lower part).265C.5 Rule-based modelIn the early phase of the project, we considered a few rule-based models beforecoming up with the two-stage model described in Section 4.6. We present one ofthem here.In the data collection sessions with Photoshop experts, we observed that when theyswitched to a new layer or use a different command, it is likely that they finisheda subtask and started the next one, especially in poster creation tasks. Therefore,we created a model that segment the event sequence whenever there is a layerswitch or command switch. Figure C.16 illustrates the model for session S1, aposter creation session. Note that the visualization is different from the previousfigures in this document as they were created in the early phase of this project.Each row represents a layer in the session, and the rectangular blocks representlogged events, sorted by timestamps from left to right. The blue triangles are theground-truth labels, while the red vertical lines are the predicted boundaries fromthe rule-based model. We also shows a snapshot image of the current canvas whena “difference score” (computed with layer, command, and image difference) is highfrom the previous event. We can see that there are many cases of over-segmentation(false positives) and missing boundaries (false negatives). Figure C.17 shows partof a portrait creation session, where the user tends to create less layers and morelikely to work on the same layer even if (s)he is doing a different subtask. Theerrors of the rule-based models are higher in such tasks.266Figure C.16: Illustration of a rule-based model on S1, a poster creation ses-sion. The model segments at any events with a layer or commandswitch. Events are sorted by time and shown from left to right. Thetop screenshot is the left side of the whole image, the bottom one isthe right side.Figure C.17: Partial illustration of rule-based model on S3, a portrait retouch-ing session.267Appendix DSupplemental Material forCorGIED.1 Concrete task examplesAs part of our iterative refinement of the task abstraction, we explicitly analyzedexamples of concrete tasks with two datasets. The first restaurant-user dataset isfrom Yelp 1. The second dataset is the Cora academic papers citations graph (sec-ond motivating scenario, described in Section 5.2.2). We structured the abstractedversions of these concrete tasks according to the nature and number of the targets.The cluster-oriented targets are one cluster, two clusters directly compared, multi-ple clusters, and outliers versus existing clusters. The node-oriented targets are onenode vs. all others, two nodes directly compared, and three nodes directly com-pared. This analysis took place during an intermediate stage, before we separatedgraph space into topology space vs. feature space in the data abstraction. It wasa stepping stone towards our simpler high-level task abstraction that treats nodesand clusters in a more unified way, and allowed us to cross-check that the CorGIEdesign would support many of these lower-level tasks.1https://www.yelp.com/dataset268Figure D.1: Abstracted versions of concrete tasks for two datasets, structuredaccording to targets.D.2 Configurable settings, filters and menusThe visual appearance of nodes is user-configurable in the SETTINGS VIEW. Nodescan be colored by their UMAP position (i.e. the rainbow color scheme), labelingand node type, as shown in Figure D.2. To highlight nodes with specific properties,we allow users to brush the nodes of a specific node type or node label, as shownin Figure D.3. Users can also search a specific node by its label or ID, as shown inFigure D.4.D.3 Alternative visual design for topology spaceIn Section 5.4.2, we visualize the topology with node-link diagrams: an initial lay-out for a static global overview, and a focal layout for hops of neighbors of focalnodes and their clustering structure (Section 5.5). Before finalizing this visual de-sign, we tried several alternatives which turned out not suitable for our visualizationtasks.269Figure D.2: Choices of node colors in the (a) Movie dataset and (b) Coradataset.Figure D.3: Highlight nodes with specific properties in the (a) Movie datasetand (b) Cora dataset.270Figure D.4: Search a node by its label in the Movie dataset.Figure D.5: Bundled edges can be shown in the latent space. This demon-strates the bipartite graph in the Movie dataset, with movie nodes inblue and user nodes in orange.271Figure D.6: Corgi paw-like menu for selected nodes. Possible actions includecreation, modification and deletion of focal groups, and show predictedlinks (i.e. false positive node pairs).The high-level goal of topology space views is to show shared neighbors of theuser-specified focal nodes. As we have derived a neighbor set for each node,we considered techniques for set visualization. We thought that the UpSet ap-proach [104] could be suitable for our problem, since it could show the intersec-tions of combinations of multiple sets, specifically the neighbor set intersections ofthe focal nodes.Following this idea, we designed a UpSet style visualization, as shown in Fig-ure D.7. Each row represents the intersection of a specific combination of sets,e.g., intersection of the neighbor set of Avatar and that of Fight Club. The circulardots on the left represent which neighbor sets are in this combination. The columnson the right, with each for a node type, represent the cardinality of the intersection.For example, with four focal movies, the 8th row in Figure D.7 tells us that movie2 and 3 have 1 cast member in common, 1 production company in common, and 4genres in common.It is not enough to show the intersection cardinalities: users also need to know theidentities of the shared neighbors, that is, the intersecting nodes. To visualize theidentities of neighbors, we chose a matrix representation (Figure D.8), where therows are for focal nodes and columns are for their neighbors. It can be viewed as272Figure D.7: UpSet style visualization for neighbor sets of four focal movienodes. The input graph has five node types: movie, cast, productioncompany, genre, and user.a partial adjacency matrix, with the neighbors ordered by their popularity in thefocal nodes. In addition to the matrix, there is a histogram at the bottom to showthe popularity count number. As there could be many neighbors, we compress thematrix columns into an intersection overview (Figure D.9) using the popularity. Atthis point, rather than pursuing this line of attack, we moved on to other approaches.There are a few drawbacks of these three views. First, they suffer from visual scal-ability: the Upset style visualization has a power set scaling factor in the numberof rows; the partial adjacency matrix has many columns; the intersection overviewcan potentially have many rows. Second, they are too sensitive to absolute nodecounts or node order, which might lose the context. Third, it is difficult to extendthem to compare two sets of focal nodes.273Figure D.8: Partial adjacency matrix, alternative visual design for showingneighbor set intersections of focal nodes.Figure D.9: Intersection overview for showing counts of neighbor set inter-sections of focal nodes. This view compresses the columns in Fig-ure D.8. The numbers within matrix cells could of course be visuallyencoded, but instead we moved on to other design approaches.274D.4 ResultsIn Chapter 5, we present two usage scenarios and the session with one of thetwo expert P1. We systematically describe four aspects of CorGIE usage: thevisualization task being conducted, the visual operation used within the CorGIEinterface, the direct observation that is possible from the CorGIE display, and anyresulting inference of authors or participants. In Section 5.6, we only color-codethe visualization task with color, and here we provide a version with all four aspectscolor-coded.We also describe the other expert session with P2 in Section D.4.4.D.4.1 Usage scenario 1: Movie RecommendationTo obtain an overview of the dataset, we first color the nodes by their node type(user and movie). Figure D.11a shows that we can see two clusters of moviesin blue on the right-hand side and many clusters of users in orange in the rest ofthe latent space. After coloring the nodes by the underlying positional colormapin the latent space, we observe in the GLOBAL TOPOLOGY view (Figure D.10g)that most user nodes (triangles) are laid out around movie nodes (circles). Manynodes that are connected by only one edge are placed around the periphery of thelayout, indicating that there are many users who only watched one movie. InFigure D.11b, we notice that the peripheral users that are connected to the samemovies have similar colors: for example, the triangles on the bottom right are allgreen. We can thus infer that the GNN does a good job in grouping these one-timeusers reviewing the same movie.To compare user clusters , we select and focus on two clusters from the LATENTSPACE, one in the green zone (foc-0) on the left, the other in the purple zone (foc-1) on the upper right, as shown in Figure D.10. In the K-HOP TOPOLOGY (Fig-ure D.10b), there appears to be three green sub-clusters in foc-0, and the greennodes connect to many nodes in the hop-1 box, while foc-1 does not show anysalient internal structure. To understand the topological difference between the275Figure D.10: Full screenshot of CorGIE interface on the Movie dataset, withtwo focal groups of user nodes. The views, with names shown in greyboxes, are laid out in four major areas on the screen, shown in blue:(a) the LATENT SPACE VIEW is for 2D node positions in latent space;(c) the NODE FEATURES VIEW is for feature distribution of all andfocal nodes; (f & g) the TOPOLOGY VIEWS show local topologicalneighbors and global topology; (b) the LATENT NEIGHBOR BLOCKSVIEW and the (e) DISTANCE COMPARISON VIEW connect differentspaces. The fifth area is for toggles and menus: (d) the SETTINGSVIEW.two focal groups, we explore their connections in the K-HOP TOPOLOGY withhover and select actions. Figure D.11c and D.11d show the visual highlights inthe K-HOP TOPOLOGY when selecting the green foc-0 nodes and the purple foc-1nodes respectively. We can see that the green nodes in the top foc-0 box connectto all hop-1 neighbors, but the purple nodes in the lower foc-1 box only connectto four of those neighbors. We infer that the GNN has learned the topologicaldifference and thus separates them in the latent space.Besides graph topologies, we also want to check whether distances match be-tween spaces. In the DISTANCE COMPARISON VIEW (Figure D.11e), we can seethat the within group distances in latent space are small, while the between groupdistances are large, which confirms that we have picked two distant clusters fromthe LATENT SPACE. We notice that the topological distances within the green276Figure D.11: Check overview of the Movie scenario: (a) LATENT SPACE,movie nodes in blue and user nodes in orange; (b) inset of GLOBALTOPOLOGY showing similar peripheral nodes in similar green color(closeup of lower right for screenshot in Figure D.10). When com-paring two clusters of user nodes, (c) we select nodes in foc-1 of theK-HOP TOPOLOGY, highlighting themselves and four neighbors in thehop-1 box, and also in the (d) GLOBAL TOPOLOGY. (e) We comparedistances between topology and latent space, and (f) between featureand latent space.cluster (foc-0) are relatively large considering they belong to the same cluster (left-most vertical histogram in Figure D.11e), which actually matches the existenceof three sub-clusters in the K-HOP TOPOLOGY. The feature distances in all threecharts of Figure D.11f are small and similar , indicating that the node features277Figure D.12: Check movie recommendations: (a) top 5 recommendedmovies for user 5355; (b) K-HOP TOPOLOGY focusing user 5355 andmovie LotR; (c) hover on the movie Suicide Squad that the user haswatched before; (d) topological distance between 5355 and LotR islarge (0.97); (e) K-HOP TOPOLOGY for user 587 and movie LotR.(#votes and average vote of a user) cannot distinguish foc-0 and foc-1. We fur-ther confirm the ineffectiveness of the features by reading the feature distancesby picking some other node groups (not shown in figure), which indicates thatthey are not useful features, and could perhaps be removed from the dataset.After exploring the clustering structure, we inspect instances of recommenda-tion. In Figure D.12a we select the user node 5355, and list its top 5 rec-ommended movies. The first one is The Lord of the Rings: the Return of King(LotR). We would like to understand why GNN decides to recommend LotR , sowe focus this recommendation by clicking on it. CorGIE automatically createstwo focal groups with the user 5355 in foc-0 and movie LotR in foc-1 (Figure D.12b).The K-HOP TOPOLOGY shows that user 5355 watched two movies (in pink) be-fore. When we hover on the two pink nodes in the hop-1 box (Figure D.12c), wefind out that each only shares a few users with the recommended movie LotR; forexample, Suicide Squad has only 5 hop-1 shared neighbors. We believe that thisrecommendation is poor, potentially indicating that we are not there yet with GNNtraining. We further confirmed this problem by reading the topological 0.05 andlatent 0.97 distances between user 5355 and movie LotR (Figure D.12e), whichseem to be correlated negatively.We later find a recommendation that makes sense: user 587 and movie LotR. Asshown in Figure D.12e, the two movies that 587 watched, Inception and The DarkNight, share many users with the recommended movie LotR (many hop-1 neigh-278bors are highlighted when hovering on Inception). Moreover, the topologicaldistance is 0.72 in this case, which is relatively small compared to the overall dis-tance distribution (not shown in figure). CorGIE thus shows evidence that thisrecommendation is well supported.We repeat this process to check other recommendations , and we find many that donot make sense. We conclude that this training result is not satisfactory. It couldbe due to the “cold start” problem in such a small dataset, where most users onlywatch 1 or 2 movies. Also, the node features were not very useful: e.g., we knowthat #votes cannot distinguish users effectively. To improve the recommendation,we might try a different model or fix the dataset problems.279D.4.2 Usage scenario 2: CoraAs with the Movie scenario, we first check the clustering structure of paper nodes.We color the nodes by the predicted labels to see if the label distribution makessense. In LATENT SPACE (Figure D.13a), we can see that the different classesof papers are roughly separated to different areas. In GLOBAL TOPOLOGY (Fig-ure D.13b), although the force-directed layout lacks much structure, we can stillsee that nearby nodes are in similar colors. In LATENT NEIGHBOR BLOCKS (Fig-ure D.13c), we can see that the red-outlined origin and its surrounding cells aredarker for all the blocks. All three observations indicate that the GNN has done agood job in classifying papers using the graph topology.Next, we inspect and compare a few clusters . For example, when we compare(select and focus) the entire red cluster and a left part of the blue one (Figure D.13d),the K-HOP TOPOLOGY (Figure D.13e) shows that most nodes of the same colorconnect to each other, and the diff chart in the NODE FEATURES VIEW (Fig-ure D.13f) signals a considerable amount of different words between the two pa-per clusters (many visible dark strips on the diff row). These observations reinforceour good impression on the training result.As the GNN seems to classify most nodes successfully, we look for problematicnodes and edges . We brush the distance scatterplot to highlight the node pairswith large latent distances but small topological distances (i.e. the bottom rightarea), as shown in Figure D.14a, and we focus one of these problematic nodepairs. In the K-HOP TOPOLOGY (Figure D.14b), we find through a few roundsof interactive hover that the two focal nodes only share one 1st-hop neighbor(circled in red), which has many nodes connected in the second hop. As the Jaccarddistance accounts for multiple hops of neighbors, the large number of shared hop-2 neighbors can explain the low topological distance (0.23). We conjecture thatthe GNN decides to locate them far from each other due to the large difference inthe first hop. Further investigation showed that the node in foc-0 is mis-classified,which hints at the limitations of this GNN model to deal with such situations.280Figure D.13: Explore overview of Cora dataset, with nodes colored by pre-dicted label: (a) LATENT SPACE; (b) GLOBAL TOPOLOGY; (c) LA-TENT NEIGHBOR BLOCKS with zoomed-in inset showing that the red-outlined cells are darker than others. Compare two paper clusters: (d)focus on the red and blue clusters in the LATENT SPACE; (e) K-HOPTOPOLOGY; (f) NODE FEATURES VIEW with inset showing the wordcount differences between foc-0 and foc-1.281Figure D.14: Find problematic node pairs in the Cora dataset: (a) select nodepairs with high latent distance but low topological distance; (b) K-HOP TOPOLOGY focusing on a buggy node pair, with only one sharedhop-1 neighbor (circled) and many shared hop-2 neighbors.282Figure D.15: P1 explored the decision boundary in the Cora dataset: (a) hefocused on the wrongly predicted nodes within a cluster (in the bound-ing box) in the LATENT SPACE; he compared (b) feature distances ofthe entire graph to (c) those within the focused nodes; he inspectedneighbors in the K-HOP TOPOLOGY that are colored by (d) predictedlabel and (e) ground-truth label.D.4.3 Expert session P1: Cora decision boundaryIn this case study, expert P1 first chose to perform quite similar tasks as in ourusage scenario. Then he wanted to check the overview to study misclassifica-tion. He colored the nodes by label correctness (Figure D.15a), where he sawthat the red (misclassified) nodes are distributed across different clusters. He useda filtered brush selection to highlight and focus the wrongly predicted nodeswithin the middle cluster in the LATENT SPACE. He colored the nodes by thepredicted labels (Figure D.15d) and true labels (Figure D.15e) respectively, sohe could understand which classes are wrong . It appeared that the GNN pre-dicted these nodes as the orange class similar to their hop-1 neighbors, but theground truth labels state that they belong to multiple different classes (blue, green,283red, etc) although the hop-1 neighbors are still orange. Further inspection withhover and brush selection shows that the focal nodes are loosely connected totheir hop-1 neighbors, but the hop-1 neighbors are tightly connected to the hop-2ones. Based on the variance in true labels and the sparsity in topology, P1 inferredthat these nodes were sitting at the decision boundary between two or multipleclasses. It was his first time to visually inspect the boundary nodes even though hehad used this dataset for years. He emphasized the importance of understandingthe decision boundary if he needed to improve the GNN model.After exploring the topology, P1 turned to DISTANCE COMPARISON for more infor-mation on the node features, where he found the feature distances within the focalnodes are relatively large compared to the overall distribution (Figure D.15c&b).He conjectured that the current model performed as he expected, and it wouldbe unlikely to classify the boundary nodes correctly even if he kept training moreepochs. He also conjectured that the GNN should use the node features in a betterway to correct these nodes. As he confirmed that the GNN has done a good jobfor most nodes except some boundary ones, he concluded that there is no need tokeep training more epochs, and training should be stopped to avoid over-fitting.284Figure D.16: The USAir dataset without focal nodes.D.4.4 Expert session P2: USAirWe present how P2 used CorGIE on the dataset USAir from his previous GNNpaper. It is a homogeneous graph showing the air routes (2126 edges) betweenairports in the US (332 nodes). There are no node features in this graph. To obtaina node embedding, we used a GCN to predict links between the airports in anunsupervised setting.After CorGIE loads (Figure D.16), P2 first checks the overview of the graph. Thenodes are colored by their 2D latent space position. The LATENT SPACE VIEWshows that there are many clusters, and the one in the lower left (green) cor-ner appears to be very different from the others. The initial global layout showsthat most of the nodes connect to neighbors with the same colors, and the LA-TENT NEIGHBOR BLOCKS VIEW shows that the topological neighbors are placednearby (the dark blocks are close to the red outlined block). We can infer that theGNN node embedding makes sense at a high level.P2 proceeds to find buggy link predictions by the GNN. With the help of the au-thor, he creates a filter to single out the node pairs that are connected in the input285Figure D.17: Find buggy node pairs. (left) Popup customization dialog topick the node pairs that are connected in the input graph but predictedas false by the GNN; (middle) the resulting 28 erroneous node pairsshown in the scatterplot of DISTANCE COMPARISON VIEW; (right)the 28 error node pairs in the list on the right side of that view.graph but are predicted false by the GNN, as shown in Figure D.17, with the popupwindow that appears after clicking on the wrench icon in the upper right cornerof the DISTANCE COMPARISON VIEW. He then clicks the first buggy node pair124-283 in the list on the right side of that view, which automatically creates twonew focus groups of a single node each and a new k-hop focal layout with thosegroups. He hovers on the focal nodes in the focal layout to check their neighbors(Figure D.18), and he finds out that they actually share many neighbors in boththe first hop and the second hop. The topological distance, as shown in the dis-tance comparison, is 0.19, which is a very low value, which further confirms whathe finds in the focal layout. However, the latent distance is 0.50, which equals thethreshold in GNN that separates true and false predictions. He infers that this er-roneous predicted pair is located in the decision boundary area, making it difficultto predict.286Figure D.18: P2 focuses on the first buggy node pair (124-283), and hoverson one of the focal nodes to see its neighbors.287D.5 Semi-structured interview in user studyIn the user study with GNN experts, we introduced and demonstrated CorGIE forabout 40 minutes, and then they used CorGIE on their own datasets for about 30minutes. Finally we conducted a semi-structured interview to gather their feed-back. Here are the questions we used in the interview:• What is your general impression of the tool?• What are the most useful features of the tool? Please explain why?• What are features that are less / not useful for you?• What features are missing from CorGIE but will be very helpful?• Are the visual encodings easy to understand?• How do you like the interactions? Do they allow you to do what you want?• Would you like to use the tool in your daily workflow (if there is only mini-mum pre-processing work to load the graph and embeddings in CorGIE)?288
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Munzner, Tamara
Visualizing multi-level structures in data
Text