Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Oppermann, Michael
Doctor of Philosophy - PhD
University of British Columbia
Prototyping is employed in information visualization to understand user needs and to iteratively implement, test, and refine possible solutions. Yet, data is often seen merely as a resource in this process, and toy or synthetic datasets can lead to incorrect data abstractions and less effective visualization designs. In this dissertation, we demonstrate how a data-driven prototyping process based on real-world data can lead to novel contributions in information visualization with high industry relevance. The design study on the Ocupado project describes the process of designing, implementing, and evaluating a suite of novel visualization tools for studying space utilization at scale. We reflect on the prototyping process that included multiple stakeholders and present generalizable design choices for visualizing non-trajectory spatiotemporal data related to indoor regions. The findings from the Ocupado study highlight the need for analyzing data concerning time periods of interest that are known in advance rather than determined on the fly. We provide a detailed characterization of non-contiguous time series slices and propose TimeElide, a domain-agnostic visual analysis tool and design space. Inspired by emerging large-scale visualization collections and the difficulty in finding relevant information, we investigate a content-based approach for visualization recommendation in the VizCommender study. We focus on text-based content that is representative of the subject matter of visualizations and compare different similarity measures. We identify that all existing visualization snippets—compact previews of visualizations in those collections—are characterized by their low information density and fail to help people judge the relevance. The VizSnippets study is the first systematic approach to visualization snippet design. We propose a design framework and computational pipeline for the lossy compression of visual and textual content into representative snippets. A critical reflection on our data-driven prototyping approach, and visualization design studies in general, reveals an alternative avenue for applied visualization projects that begins with real-world data rather than specific stakeholder analysis questions. We introduce the notion of data-first design studies and provide practical guidance.
Data-Driven Prototyping in Information VisualizationbyMichael OppermannB.Sc., University of Vienna, 2014Dipl.-Ing., University of Vienna, 2017a dissertation submitted in partial fulfillmentof the requirements for the degree ofDoctor of Philosophyinthe faculty of graduate and postdoctoral studies(Computer Science)T_heUniversity of British Columbia(Vancouver)August 2021©Michael Oppermann, 2021T_he following individuals certify that they have read, and recommend to the Facultyof Graduate and Postdoctoral Studies for acceptance, the dissertation entitled:Data-Driven Prototyping in Information Visualizationsubmitted byMichael Oppermann in partial fulfillment of the requirements forthe degree of Doctor of Philosophy in Computer Science.Examining Committee:Tamara Munzner, Department of Computer Science, UBCSupervisorJoannaMcGrenere, Department of Computer Science, UBCSupervisory CommitteeMemberGiuseppe Carenini, Department of Computer Science, UBCSupervisory CommitteeMemberIvan Beschastnikh, Department of Computer Science, UBCUniversity ExaminerLuanne Sinnamon, School of Information, UBCUniversity ExaminerChristopher Collins, Ontario Tech UniversityExternal ExamineriiAbstractPrototyping is employed in information visualization to understand user needsand to iteratively implement, test, and refine possible solutions. Yet, data is oftenseenmerely as a resource in this process, and toy or synthetic datasets can lead toincorrect data abstractions and less effective visualization designs.In this dissertation, we demonstrate how a data-driven prototyping process basedon real-world data can lead to novel contributions in information visualizationwithhigh industry relevance.T_he design study on theOcupado project describes the process of designing, imple-menting, and evaluating a suite of novel visualization tools for studying space uti-lization at scale. We reflect on the prototyping process that includedmultiple stake-holders and present generalizable design choices for visualizing non-trajectoryspatiotemporal data related to indoor regions.T_he findings from the Ocupado study highlight the need for analyzing data con-cerning time periods of interest that are known in advance rather than determinedon the fly. We provide a detailed characterization of non-contiguous time seriesslices and propose TimeElide, a domain-agnostic visual analysis tool and designspace.Inspired by emerging large-scale visualization collections and the difficulty in find-ing relevant information, we investigate a content-based approach for visualizationrecommendation in the VizCommender study. We focus on text-based content thatiiiis representative of the subject matter of visualizations and compare differentsimilarity measures.We identify that all existing visualization snippets—compact previews of visualiza-tions in those collections—are characterized by their low information density andfail to help people judge the relevance. T_he VizSnippets study is the first systematicapproach to visualization snippet design. We propose a design framework andcomputational pipeline for the lossy compression of visual and textual content intorepresentative snippets.A critical reflection on our data-driven prototyping approach, and visualizationdesign studies in general, reveals an alternative avenue for applied visualizationprojects that begins with real-world data rather than specific stakeholder analysisquestions. We introduce the notion of data-first design studies and provide practicalguidance.ivLay SummaryAll aspects of our lives become increasingly digitized and produce data at ever-increasing rates, but so does the humandesire tomake sense of and extract valuableinsights from. Information visualization transforms raw data into visual represen-tations to help people explore, analyze, and communicate it. However, the processof creating effective visualizations from complex data is challenging. Prototyp-ing is employed to understand user needs and to iteratively implement, test, andrefine possible solutions. T_he research presented in this dissertation contributestowards improving prototyping in information visualization and promotes the useof real-world data throughout the process. Multiple visualization case studies arediscussed that are driven by data: visualize human occupancy across dozens ofbuildings, visualize time-oriented data with special characteristics, andmake largecollections of visualizations more accessible.vPrefaceChapters of this dissertation have been previously published or are accepted forpublication. Although I am the author of this dissertation and took the lead on theresearch and writing process of all publications, the contributions are the productof years of collaboration with my advisor, TamaraMunzner. To acknowledgemycollaborators’ contributions, I will use the first-person plural in all chapters afterthe introduction.Chapter 2 has been previously published as:M.OppermannandT.Munzner. Ocupado: VisualizingLocation-BasedCounts Over Time Across Buildings. Computer Graphics Forum (Proc. ofEuroVis), 39(3): 127-138, 2020 [201].Tamara Munzner and Stefan Storey, the CEO of Sensible Building Sci-ence, initiated the project and reached out to potential stakeholders.Yann Dubois, an undergrad research assistant, helped during inter-views and began the task abstraction. When I joined UBC, I took overas project lead and implemented all prototypes. Together with Tamaraand Stefan, I conducted a series of additional interviews and demos ofthe Ocupado interfaces. Under Tamara’s supervision, I conducted ex-tensive data and task abstractions, and reflected on the design process.Tamara helped with the writing and editing of the publication.viChapter 3 has been accepted for publication:M. Oppermann, L. Liu, and T. Munzner. TimeElide: Visual Analysisof Non-Contiguous Time Series Slices. Proc. of IEEE VIS Short Papers,2021 [205].T_his research follows up on the Ocupado project where TamaraMun-zner and I derived initial data abstractions. I developed usage sce-narios, characterized tasks, and designed the TimeElide user interface.Together with Tamara, I supervised an undergrad research assistant,Luce Liu, who helped with the implementation and conducted usabil-ity tests. All authors jointly developed the visual encoding design space.Tamara and Luce contributed to the editing process of the publication.Chapter 4 has been previously published as:M.Oppermann, R. Kincaid, and T. Munzner. VizCommender: Com-puting Text-Based Similarity in Visualization Repositories for Content-Based Recommendations. IEEE Transactions on Visualization and Com-puter Graphics (Proc. of VAST), 27(2): 495-505, 2021 [204].T_his project is the result of an industry collaboration with Tableau,where I worked as a research intern for 11 months. Initial ideas for cre-ating a content-based model for visualization recommendations wereborn during conversations with the Tableau Recommender SystemsGroup. I implemented all prototypes, and developed and conductedthe crowdsourced similarity study. Robert Kincaid, a research scientistat Tableau, and TamaraMunzner contributed to the similarity mod-els and user study design. Tamara and Robert also helped during thewriting and editing process of the publication.Chapter 5 has been accepted for publication:M.Oppermann and T.Munzner. VizSnippets: Compressing Visualiza-tion Bundles Into Representative Previews for Browsing VisualizationCollections. IEEE Transactions on Visualization and Computer Graphics(Proc. of VIS), 2021 [202].viiI envisioned this project and identified issues related to existing visu-alization snippets when I conducted the embedded inquiry at Tableau.Tamara Munzner and I jointly conceived the VizSnippets computa-tional pipeline, the top-down design framework, and the visual inspec-tion methodology. I reviewed existing snippets and implemented allalgorithmic buildings blocks and visual inspectors. I performed themajority of the writing for the publication, while Tamara contributedto the editing process.Chapter 6 has been previously published as:M.Oppermann andT.Munzner. Data-First VisualizationDesignStud-ies. Proc. of the IEEEVISWorkshopEvaluationandBeyond -MethodologicalApproaches for Visualization (BELIV), pp. 74-80, 2020 [200].Tamara Munzner and I recognized the need to better characterize thisspecific type of visualization design study. Together, we adapted thedesign study methodology framework and reflected on opportunitiesand risks. I conducted the literature review of previous design stud-ies. Tamara contributed to the writing and editing of the publication.T_his work is heavily influenced bymy earlier project on bike sharingnetworks with TorstenMöller andMichael Sedlmair [203].Sec. 7.3 includes content from a previously published extended abstract:M.Oppermann and T. Munzner. Uncovering Spatiotemporal Dynam-ics from Non-Trajectory Data. Proc. of the IEEE VISWorkshop CityVis,4-6, 2018 [199].I reflected on the ongoing Ocupado project andmy previous bike shar-ing work. Tamara Munzner helped me to frame the implications ofnon-trajectory spatiotemporal data and contributed to the editingprocess.T_he UBC Behavioural Research Ethics Board approved all studies described in thisdissertation: certificate number H10-03336.viiiANote onRelatedWorkChapter 2 to Chapter 6 are presented as they have been published or accepted forpublication with minor stylistic changes. T_he review of the literature relevant toits respective subject matter is included as a subsection in each of these researchchapters. Due to the different types of contributions and only minor overlapsacross chapters, I chose this structure instead of detaching the related work andcombining it into its own chapter.ixTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxivAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Context andMotivation . . . . . . . . . . . . . . . . . . . . . 11.2 Data-Driven Visualization Prototyping . . . . . . . . . . . . . 21.3 Research Trajectory . . . . . . . . . . . . . . . . . . . . . . . 41.3.1 Spatiotemporal Data . . . . . . . . . . . . . . . . . . 41.3.2 Non-Contiguous Time Series Slices . . . . . . . . . . . 61.3.3 Visualization Collections and Bundles . . . . . . . . . . 71.3.4 Data-First Design Studies . . . . . . . . . . . . . . . . 91.4 A Note on Chronology . . . . . . . . . . . . . . . . . . . . . . 9x1.5 A Note on Supplemental Materials . . . . . . . . . . . . . . . 111.6 Summary of Contributions . . . . . . . . . . . . . . . . . . . 112 Ocupado: Visualize Location-Based Counts Over TimeAcross Buildings 162.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.2 Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.3 Domain Goals and Task Abstraction . . . . . . . . . . . . . . . 202.4 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.4.1 Data Acquisition . . . . . . . . . . . . . . . . . . . . 222.4.2 Location-Based Counts . . . . . . . . . . . . . . . . . 232.4.3 Spatial and Temporal Data Granularity . . . . . . . . . 232.5 Ocupado Interfaces . . . . . . . . . . . . . . . . . . . . . . . 252.6 Ocupado Design . . . . . . . . . . . . . . . . . . . . . . . . 392.6.1 View Coordination and Layout . . . . . . . . . . . . . 392.6.2 Visualization Components . . . . . . . . . . . . . . . 412.6.3 Floor Plan Linearization . . . . . . . . . . . . . . . . . 442.6.4 Design Rationale . . . . . . . . . . . . . . . . . . . . 462.7 Analysis Scenarios . . . . . . . . . . . . . . . . . . . . . . . 492.7.1 Data Quality Control . . . . . . . . . . . . . . . . . . 492.7.2 Recent Utilization Patterns . . . . . . . . . . . . . . . 512.7.3 Campus-Wide Situational Awareness . . . . . . . . . . 522.8 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 532.8.1 General Spatiotemporal Data Visualization . . . . . . . 532.8.2 Non-Trajectory Spatiotemporal Data Visualization . . . 542.8.3 Space Usage Visualization . . . . . . . . . . . . . . . . 542.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573 TimeElide: Visual Analysis of Non-Contiguous Time Series Slices . . . 593.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.2 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 613.3 Non-Contiguous Time Series Slices . . . . . . . . . . . . . . . 623.3.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . 62xi3.3.2 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . 633.4 Visual Encoding Design Space . . . . . . . . . . . . . . . . . . 663.4.1 Targeting Different Levels of Detail . . . . . . . . . . . 663.4.2 T_he Sparkbox Technique . . . . . . . . . . . . . . . . 673.4.3 Design Space Analysis Rationale . . . . . . . . . . . . . 683.5 T_he TimeElide Tool . . . . . . . . . . . . . . . . . . . . . . . 713.5.1 Overall Design . . . . . . . . . . . . . . . . . . . . . 713.5.2 Manually Specify or Automatically Detect Slices . . . . . 723.5.3 Implementation . . . . . . . . . . . . . . . . . . . . 753.6 Usage Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . 753.6.1 Soccer Player In-Match Activity . . . . . . . . . . . . . 763.6.2 Bakery Transactions . . . . . . . . . . . . . . . . . . . 763.6.3 Bike Rides . . . . . . . . . . . . . . . . . . . . . . . . 793.6.4 Building Occupancy onWeekend Days . . . . . . . . . 803.7 Discussion and FutureWork . . . . . . . . . . . . . . . . . . 823.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834 VizCommender: Content-Based Recommendations for VisualizationCollections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 844.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 854.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 884.2.1 Recommendation Paradigms . . . . . . . . . . . . . . 884.2.2 Project Context . . . . . . . . . . . . . . . . . . . . . 894.2.3 Process . . . . . . . . . . . . . . . . . . . . . . . . . 904.2.4 Users and Tasks . . . . . . . . . . . . . . . . . . . . . 914.2.5 VisualizationWorkbooks . . . . . . . . . . . . . . . . 924.3 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 944.3.1 Visual Encoding Recommendation . . . . . . . . . . . 944.3.2 Semantic Text Similarity Models and Evaluations . . . . 954.3.3 Evaluation of Recommendation Algorithms . . . . . . . 954.4 VizCommender Proof-Of-Concept . . . . . . . . . . . . . . . 964.4.1 User Interface . . . . . . . . . . . . . . . . . . . . . . 974.4.2 Example Recommendation & Usage . . . . . . . . . . . 98xii4.5 Data Extraction and Feature Engineering . . . . . . . . . . . . 994.5.1 Visualization Text Extraction . . . . . . . . . . . . . . 994.5.2 Leaving Out Visual Encodings . . . . . . . . . . . . . . 1004.5.3 Leaving Out Underlying Data . . . . . . . . . . . . . . 1014.5.4 Data Challenges . . . . . . . . . . . . . . . . . . . . . 1024.5.5 Feature Selection and Data Filtering . . . . . . . . . . . 1044.6 Viz-to-Viz Similarity . . . . . . . . . . . . . . . . . . . . . . 1054.6.1 Pairwise Comparisons . . . . . . . . . . . . . . . . . 1054.6.2 NLPModels . . . . . . . . . . . . . . . . . . . . . . . 1064.7 Crowdsourced Similarity Judgements . . . . . . . . . . . . . . 1094.7.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . 1094.7.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . 1114.7.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . 1124.7.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . 1134.8 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1134.8.1 Human Judgement . . . . . . . . . . . . . . . . . . . 1144.8.2 Model Comparison . . . . . . . . . . . . . . . . . . . 1154.8.3 Prototype Instantiation & Similarity Facets . . . . . . . 1164.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 1184.9.1 Limitations of Tableau Collections . . . . . . . . . . . . 1184.9.2 Near-Duplicates Grouping . . . . . . . . . . . . . . . 1194.9.3 Hybrid and Sequence-Aware Recommender Systems . . 1204.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1205 VizSnippets: Representative Previews for Visualization Collections . . 1215.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 1225.2 Visualization Collections and Snippets . . . . . . . . . . . . . 1245.2.1 Data Characterization and Sources . . . . . . . . . . . 1245.2.2 Visualization Snippet Usage . . . . . . . . . . . . . . . 1265.2.3 Embedded InquiryWithin Tableau . . . . . . . . . . . 1275.2.4 Review of Existing Snippet Designs . . . . . . . . . . . 1285.2.5 Challenges . . . . . . . . . . . . . . . . . . . . . . . 1305.2.6 Design Framework . . . . . . . . . . . . . . . . . . . 132xiii5.2.7 Intended Framework and Pipeline Usage . . . . . . . . 1325.3 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 1335.3.1 Result Snippets . . . . . . . . . . . . . . . . . . . . . 1335.3.2 Visualization Collections . . . . . . . . . . . . . . . . 1355.3.3 Approaches to Visualization Compression . . . . . . . . 1355.4 VizSnippets Computational Pipeline . . . . . . . . . . . . . . 1365.4.1 Visual Inspectors . . . . . . . . . . . . . . . . . . . . 1375.4.2 Page and Snippet Layouts . . . . . . . . . . . . . . . . 1405.4.3 Snippet Images . . . . . . . . . . . . . . . . . . . . . 1405.4.4 Snippet Text . . . . . . . . . . . . . . . . . . . . . . 1455.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1485.5.1 Snippet Text . . . . . . . . . . . . . . . . . . . . . . 1485.5.2 Snippet Images . . . . . . . . . . . . . . . . . . . . . 1515.5.3 Page and Snippet Layout . . . . . . . . . . . . . . . . 1585.5.4 Designer Feedback . . . . . . . . . . . . . . . . . . . 1605.6 Visual InspectionT_hrough Random Sampling . . . . . . . . . . 1615.6.1 Inspectors for Iterative Development . . . . . . . . . . 1615.6.2 Inspector Characteristics . . . . . . . . . . . . . . . . 1625.6.3 Visual Inspector RelatedWork . . . . . . . . . . . . . . 1635.7 Discussion and FutureWork . . . . . . . . . . . . . . . . . . 1645.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1656 Data-First VisualizationDesign Studies . . . . . . . . . . . . . . . . 1676.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 1676.2 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . 1696.3 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . 1716.4 Refined and Extended Framework . . . . . . . . . . . . . . . . 1726.5 Reflections and Recommendations . . . . . . . . . . . . . . . 1756.5.1 Winnowing Tasks and Refining Data . . . . . . . . . . 1756.5.2 Opportunities . . . . . . . . . . . . . . . . . . . . . . 1776.5.3 Risks . . . . . . . . . . . . . . . . . . . . . . . . . . 1796.6 Review of Previous Design Studies . . . . . . . . . . . . . . . 1826.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 185xiv6.7.1 Domain Familiarity and Data Understandability . . . . . 1856.7.2 Data Repurposing . . . . . . . . . . . . . . . . . . . . 1866.7.3 Autobiographical Design Studies . . . . . . . . . . . . 1866.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1876.9 Addendum . . . . . . . . . . . . . . . . . . . . . . . . . . . 1887 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . 1897.1 Reflecting on Visualization Patchwork Prototyping . . . . . . . 1907.2 ReflectingonVisualizationTools forDifferentAudiences andContexts 1947.3 Reflecting on Visualizing Spatial and Temporal Non-Contiguity . 1957.3.1 Spatiotemporal Non-Trajectory Data . . . . . . . . . . 1957.3.2 Spatial Non-Contiguity Related to Indoor Regions . . . . 1987.3.3 Temporal Non-Contiguity . . . . . . . . . . . . . . . . 1997.3.4 Visual Perception Studies . . . . . . . . . . . . . . . . 1997.4 Reflecting on Visualization as a Medium . . . . . . . . . . . . 2007.4.1 Analyzing and Communicating Data . . . . . . . . . . 2007.4.2 Capturing and SharingT_houghts and Data Discoveries . 2027.5 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . 203Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204A Appendix: Ocupado . . . . . . . . . . . . . . . . . . . . . . . . . . . 237A.1 Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . 237A.2 Ocupado Interfaces . . . . . . . . . . . . . . . . . . . . . . . 242B Appendix: TimeElide . . . . . . . . . . . . . . . . . . . . . . . . . . 276B.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276B.2 Analysis Questions and Tasks . . . . . . . . . . . . . . . . . . 277B.3 TimeElide Interface . . . . . . . . . . . . . . . . . . . . . . . 278B.3.1 Global and Chart-Specific Settings . . . . . . . . . . . 279B.3.2 TimeElide Screenshots . . . . . . . . . . . . . . . . . 280C Appendix: VizCommender . . . . . . . . . . . . . . . . . . . . . . . 289C.1 Model Investigation . . . . . . . . . . . . . . . . . . . . . . . 289xvC.1.1 UMAP Projections of Document Representations . . . . 290C.1.2 Keyword Probing . . . . . . . . . . . . . . . . . . . . 294C.2 Crowdsourced Study . . . . . . . . . . . . . . . . . . . . . . 295C.2.1 Experiment Interface . . . . . . . . . . . . . . . . . . 295C.2.2 Demographics and Study Experience . . . . . . . . . . 298C.2.3 Agreement: Model Predictions and Human Judgements . 300C.2.4 Confidence of Model Decisions . . . . . . . . . . . . . 306C.3 Study Analysis Tool . . . . . . . . . . . . . . . . . . . . . . . 308C.4 Proof-of-Concept: VizCommender . . . . . . . . . . . . . . . 311D Appendix: VizSnippets . . . . . . . . . . . . . . . . . . . . . . . . . 315D.1 Review of Existing Visualization Snippet Designs . . . . . . . . 316D.2 Visual Inspectors . . . . . . . . . . . . . . . . . . . . . . . . 318D.2.1 Text Content of Visualization Bundles (I1) . . . . . . . . 318D.2.2 Keyword Similarity and Spell Checking (I2) . . . . . . . 318D.2.3 Pairwise Image Similarity (I3) . . . . . . . . . . . . . . 321D.2.4 Bundle Image Similarity (I4) . . . . . . . . . . . . . . 321D.2.5 Image Colours (I5) . . . . . . . . . . . . . . . . . . . 323D.2.6 Image Selection and Ranking (I6) . . . . . . . . . . . . 325D.2.7 Page and Snippet Layouts (I7) . . . . . . . . . . . . . . 328D.2.8 A/B Comparisons (I8) . . . . . . . . . . . . . . . . . . 331D.3 Labeling Interfaces . . . . . . . . . . . . . . . . . . . . . . . 335D.4 Implementation and Time Estimates . . . . . . . . . . . . . . 338D.5 Design Framework (Size and Layout Choices) . . . . . . . . . . 339E Appendix: Data-First Design Studies . . . . . . . . . . . . . . . . . . 340xviList of TablesTable 1.1 Supplemental materials . . . . . . . . . . . . . . . . . . . 11Table 6.1 16 probable cases of data-first design studies . . . . . . . . . 183Table 7.1 Reflections on four key themes . . . . . . . . . . . . . . . . 189Table 7.2 Spectrum of visualization tools . . . . . . . . . . . . . . . . 195Table A.1 Demos and interviews conducted by us and our industry partner 238Table A.2 Demos and interviews conducted by our industry partner . . 239Table A.3 Match between stakeholders and abstract tasks . . . . . . . 240Table A.4 Match between stakeholder questions and abstract tasks . . . 241Table A.5 Match between Ocupado interfaces and abstract tasks . . . . 242Table B.1 Summary of TimeElide datasets . . . . . . . . . . . . . . . 276Table B.2 Match between domain-specific questions and abstract tasks 278Table B.3 Overview of global and chart-specific settings . . . . . . . . 279Table D.1 Overview of existing visualization snippets . . . . . . . . . . 316Table D.2 Detailed analysis of existing visualization snippets . . . . . . 317Table D.3 Design framework: size and layout choices . . . . . . . . . . 339Table E.1 Review of previous design studies . . . . . . . . . . . . . . 341xviiList of FiguresFigure 1.1 Data-driven prototyping process . . . . . . . . . . . . . . 3Figure 1.2 Timelines of the research projects described in this dissertation 10Figure 1.3 Overview of research chapters and contributions . . . . . . 12Figure 2.1 Spatial and temporal data granularities. . . . . . . . . . . . 25Figure 2.2 Ocupado Sandbox: sparklines, normalized y-scale . . . . . . 26Figure 2.3 Ocupado Sandbox: heatmap, normalized colour-scale . . . . 27Figure 2.4 Comparison between sparklines and heatmaps . . . . . . . 28Figure 2.5 Ocupado Sandbox: building details . . . . . . . . . . . . . 29Figure 2.6 Ocupado Sandbox: campus map small multiples . . . . . . 30Figure 2.7 Ocupado Sandbox: clustering . . . . . . . . . . . . . . . . 31Figure 2.8 Ocupado Sandbox: clustering (smaller campus) . . . . . . . 31Figure 2.9 Ocupado Sandbox: clustering results showing individual items 32Figure 2.10 Ocupado Campus Explorer: region subset; sparklines . . . . 33Figure 2.11 Ocupado Campus Explorer: region subset; superimposed lines 33Figure 2.12 Ocupado Campus Explorer: region subset; confidence bands 34Figure 2.13 Ocupado Campus Explorer: region subset; spatial heatmap . 34Figure 2.14 Ocupado Building Long-Term Interface: typical day profiles . 36Figure 2.15 Ocupado Building Long-Term Interface: typical day profiles (2) 36Figure 2.16 Ocupado Building Recent Interface: overview window . . . . 37Figure 2.17 Ocupado Region Compare interface . . . . . . . . . . . . . 38Figure 2.18 View composition and interrelations of all Ocupado interfaces 39Figure 2.19 Ocupado region detail view . . . . . . . . . . . . . . . . . 41xviiiFigure 2.20 Visualization components for region subset views . . . . . . 42Figure 2.21 Binned time series chart at different zoom levels . . . . . . 44Figure 2.22 Linearized floor-plan ordering for regions . . . . . . . . . . 46Figure 2.23 Example quality issues that we identified . . . . . . . . . . 51Figure 2.24 Ocupado Campus Explorer: filter by building . . . . . . . . 53Figure 3.1 Visual encoding design space . . . . . . . . . . . . . . . . 65Figure 3.2 TimeElide interface . . . . . . . . . . . . . . . . . . . . . 70Figure 3.3 Manually specify desired slices . . . . . . . . . . . . . . . 73Figure 3.4 Automatically detect existing slices . . . . . . . . . . . . . 75Figure 3.5 Assign columns of uploaded CSV file . . . . . . . . . . . . 77Figure 3.6 Analyze bakery transactions using the 2D heatmap . . . . . 77Figure 3.7 Analyze bakery transactions using the stepped area chart . . 78Figure 3.8 Analyze transactions on specific days using the 2D heatmap . 79Figure 3.9 Analyze bike rides using the 2D heatmap . . . . . . . . . . 80Figure 3.10 Analyze bike rides using the box-plot-bars . . . . . . . . . . 80Figure 3.11 Manually specify slices; timeline; alternative visual encodings 82Figure 4.1 Outline of the iterative VizCommender process . . . . . . . 91Figure 4.2 Simplified example feature extraction . . . . . . . . . . . . 94Figure 4.3 VizCommender interface results view . . . . . . . . . . . . 97Figure 4.4 VizCommender interface detail view . . . . . . . . . . . . 98Figure 4.5 Examples illustrate that visual similarity is topic independent 101Figure 4.6 Number of unique words in visualization workbooks . . . . 103Figure 4.7 Interface for semantic text similarity task . . . . . . . . . . 111Figure 4.8 Agreement betweenmodel predictions and human judgements 116Figure 5.1 Examples of existing visualization snippets . . . . . . . . . 129Figure 5.2 Design framework for displaying visualization snippets . . . 132Figure 5.3 VizSnippets computational pipeline . . . . . . . . . . . . . 137Figure 5.4 Overview of visual inspectors . . . . . . . . . . . . . . . . 139Figure 5.5 High-level architecture of the Siamese CNN . . . . . . . . . 143Figure 5.6 Frequency of calendar years in extracted text content . . . . 147Figure 5.7 Tableau Public: cumulative reduction of keywords . . . . . . 150xixFigure 5.8 Spell checking the input text and keywords . . . . . . . . . 150Figure 5.9 Observable: cumulative reduction of keywords . . . . . . . 150Figure 5.10 ROC curve and accuracy for different k-means clusters . . . 152Figure 5.11 ROC curve and prediction accuracy for similarity measures . 154Figure 5.12 Number of images per bundle without applying any filters . 156Figure 5.13 Tableau Public: cumulative reduction of images . . . . . . . 156Figure 5.14 Observable: cumulative reduction of images . . . . . . . . . 156Figure 5.15 Visual inspector to analyze the image selection and ranking . 157Figure 5.16 Average number of image collages for bundles . . . . . . . . 158Figure 5.17 Automatically generated snippets of an Observable notebook 159Figure 5.18 Comparison with existing visualization snippets . . . . . . 160Figure 5.19 Conceptual model for item-level visual inspection . . . . . . 162Figure 6.1 Refined and extended framework for data-first design studies 173Figure 6.2 Conceptual model: data and task perspectives . . . . . . . . 177Figure 7.1 Overview of Ocupado and TimeElide interfaces . . . . . . . 192Figure 7.2 Overview of VizCommender and VizSnippets interfaces . . . 193Figure 7.3 Data-driven prototyping process (reproduced from Introduction)203Figure A.1 Ocupado interfaces based on spatiotemporal data granularities 243Figure A.2 Ocupado Sandbox: sparklines, absolute y-scale . . . . . . . 244Figure A.3 Ocupado Sandbox: sparklines; smaller university campus . . 245Figure A.4 Ocupado Sandbox: heatmap, absolute colour-scale . . . . . 246Figure A.5 Ocupado Sandbox: building details with per-floor device counts 247Figure A.6 Ocupado Sandbox: floor details . . . . . . . . . . . . . . . 248Figure A.7 Ocupado Sandbox: zone details; zoomable timeline . . . . . 249Figure A.8 Ocupado Campus Explorer: overview . . . . . . . . . . . . 250Figure A.9 Ocupado Campus Explorer: control panel for visual encodings 251Figure A.10 Ocupado Campus Explorer: filter by space type . . . . . . . 252Figure A.11 Ocupado Campus Explorer: filter by time range . . . . . . . 253Figure A.12 Ocupado Campus Explorer: filter by discrete time slices . . . 254Figure A.13 Ocupado Campus Explorer: activity patterns . . . . . . . . 255Figure A.14 Ocupado Campus Explorer: textual summary of active filters 256xxFigure A.15 Ocupado Campus Explorer: region subset; sparklines . . . . 257Figure A.16 Ocupado Campus Explorer: region selector . . . . . . . . . 258Figure A.17 Ocupado Campus Explorer: region subset view; box-plot-bars 259Figure A.18 Ocupado Campus Explorer: region subset; superimposed lines 260Figure A.19 Ocupado Campus Explorer: region subset; confidence bands 261Figure A.20 Ocupado Campus Explorer: region subset; confidence bands (2) 262Figure A.21 Ocupado Campus Explorer: zone details . . . . . . . . . . 263Figure A.22 Ocupado Campus Explorer: zone details; second example . . 264Figure A.23 Ocupado Campus Explorer: building details . . . . . . . . . 265Figure A.24 Ocupado Building Long-Term Interface . . . . . . . . . . . 266Figure A.25 Ocupado Building Long-Term Interface: typical day profiles . 267Figure A.26 Ocupado Building Recent Interface: live device counts . . . . 268Figure A.27 Ocupado Building Recent Interface: live device counts (2) . . 269Figure A.28 Ocupado Building Recent Interface: live device counts; per-floor counts over past 12h . . . . . . . . . . . . . . . . . . 270Figure A.29 Ocupado Building Recent Interface: live device counts; per-floor counts over past 12h (2) . . . . . . . . . . . . . . . . 271Figure A.30 Ocupado Building Recent Interface: maximum device counts 272Figure A.31 Ocupado Building Recent Interface: select time window . . . 273Figure A.32 Ocupado Region Compare Interface: select regions . . . . . 274Figure A.33 Ocupado Region Compare Interface: multiple views . . . . . 275Figure B.1 Welcomemodal window: data and slicing options . . . . . . 280Figure B.2 Welcomemodal window: visual encodings . . . . . . . . . 280Figure B.3 TimeElide data source page . . . . . . . . . . . . . . . . . 281Figure B.4 Choose betweenmanual specification and automatic detection 281Figure B.5 Analyze soccer activity using sparkboxes . . . . . . . . . . 282Figure B.6 Analyze soccer activity using the stepped area chart . . . . . 282Figure B.7 Analyze soccer activity using heat stripes . . . . . . . . . . 283Figure B.8 Analyze soccer activity using the 2D heatmap (absolute time) 283Figure B.9 Analyze soccer activity using the 2D heatmap (norm. duration) 284Figure B.10 Analyze occupancy using sparkboxes . . . . . . . . . . . . 284Figure B.11 Analyze occupancy using the stepped area chart . . . . . . . 285xxiFigure B.12 Analyze occupancy using the multi-series line chart . . . . . 285Figure B.13 Analyze occupancy using the bandedmulti-series line chart . 286Figure B.14 Sparkboxes at different zoom levels . . . . . . . . . . . . . 287Figure B.15 Sparkboxes using different color palettes . . . . . . . . . . 288Figure C.1 Tableau Public: UMAP of TF-IDF, Doc2Vec, GloVe vectors . . 290Figure C.2 Tableau Public: UMAP applied to LSI and LDA vectors . . . . 291Figure C.3 Tableau Corporate: UMAP for TF-IDF, Doc2Vec, GloVe vectors 292Figure C.4 Tableau Corporate: UMAP applied to LSI and LDA vectors . . 293Figure C.5 Model comparison via keyword probing . . . . . . . . . . . 294Figure C.6 Study interface: obtaining informed consent . . . . . . . . 295Figure C.7 Study interface: all steps must be completed . . . . . . . . . 296Figure C.8 Study interface: instructions for participants . . . . . . . . 296Figure C.9 Study interface: description of text document structure . . . 297Figure C.10 Study interface: task description . . . . . . . . . . . . . . 297Figure C.11 Study interface: example selection . . . . . . . . . . . . . 298Figure C.12 Study interface: screening test . . . . . . . . . . . . . . . 298Figure C.13 Study: age distribution . . . . . . . . . . . . . . . . . . . 299Figure C.14 Study: gender distribution . . . . . . . . . . . . . . . . . 299Figure C.15 Study: educational background . . . . . . . . . . . . . . . 299Figure C.16 Study: task difficulty . . . . . . . . . . . . . . . . . . . . 300Figure C.17 Study: agreement betweenmodel predictions and humans . 302Figure C.18 Study: agreement for triplets with higher consensus . . . . . 303Figure C.19 Study: agreement for triplets with lower consensus . . . . . 304Figure C.20 Study: within-model agreement . . . . . . . . . . . . . . 305Figure C.21 Study: compare models based on similarity score differences 307Figure C.22 VizCommender analysis tool: participants . . . . . . . . . 308Figure C.23 VizCommender analysis tool: triplets overview . . . . . . . 309Figure C.24 VizCommender analysis tool: triplet preview . . . . . . . . 309Figure C.25 VizCommender analysis tool: sort triplets by predictions . . 310Figure C.26 VizCommender analysis tool: filter triplets by consensus . . 311Figure C.27 Early VizCommender prototype . . . . . . . . . . . . . . . 312Figure C.28 VizCommender overview: random sample . . . . . . . . . 312xxiiFigure C.29 VizCommender overview: results for the search query “medal" 313Figure C.30 VizCommender overview: near-duplicate workbooks . . . . 314Figure D.1 Visual inspector I1: analyze text content . . . . . . . . . . . 318Figure D.2 Visual inspector I2: analyze similar keywords . . . . . . . . 319Figure D.3 Visual inspector I2: other examples . . . . . . . . . . . . . 320Figure D.4 Visual inspector I2: adjust Levenshtein distance . . . . . . . 320Figure D.5 Visual inspector I3: incorrectly classified image pairs . . . . 321Figure D.6 Visual inspector I4: analyze image similarity (1) . . . . . . . 322Figure D.7 Visual inspector I4: analyze image similarity (2) . . . . . . . 322Figure D.8 Visual inspector I5: analyze image colours . . . . . . . . . . 323Figure D.9 Visual inspector I5: higher color diversity . . . . . . . . . . 324Figure D.10 Visual inspector I5: lower color diversity . . . . . . . . . . . 324Figure D.11 Visual inspector I6: analyze image selection . . . . . . . . . 325Figure D.12 Visual inspector I6: compare with initial ranking . . . . . . 326Figure D.13 Visual inspector I6: change display settings . . . . . . . . . 327Figure D.14 Visual inspector I6: smaller bundles . . . . . . . . . . . . . 327Figure D.15 Visual inspector I7: grid layout . . . . . . . . . . . . . . . . 328Figure D.16 Visual inspector I7: strip layout . . . . . . . . . . . . . . . 329Figure D.17 Visual inspector I7: list layout . . . . . . . . . . . . . . . . 329Figure D.18 Visual inspector I7: table layout . . . . . . . . . . . . . . . 330Figure D.19 Visual inspector I7: snippet shown as a large preview . . . . 331Figure D.20 Visual inspector I8: Tableau Online snippets . . . . . . . . . 332Figure D.21 Visual inspector I8: Tableau Public snippets . . . . . . . . . 333Figure D.22 Visual inspector I8: Power BI snippets . . . . . . . . . . . . 334Figure D.23 Visual inspector I8: Qlik snippets . . . . . . . . . . . . . . 334Figure D.24 A/B comparison: Observable and VizSnippets . . . . . . . . 335Figure D.25 Interface for labeling empty images (1) . . . . . . . . . . . 336Figure D.26 Interface for labeling empty images (2) . . . . . . . . . . . 337Figure D.27 Interface for labeling similar and dissimilar image pairs (1) . 337Figure D.28 Interface for labeling similar and dissimilar image pairs (2) . 338xxiiiGlossarybi Business intelligencecbr Content-based recommendationcfr Recommendations based on collaborative filteringhci Human-computer interactionhvac Heating, ventilation, and air conditioninglda Latent Dirichlet allocationlsi Latent semantic indexingmmr Maximal marginal relevancenlp Natural language processingsbs Sensible Building Sciencetf-idf Term frequency–inverse document frequencyxxivAcknowledgmentsFirst and foremost, I would like to thank my supervisor, Tamara Munzner, forher invaluable advice, continuous support, encouragement, and patience over theyears.I am deeply grateful for the guidance and support of my previous advisors at theUniversity of Vienna, Michael Sedlmair and Torsten Möller, who introduced me tothe field of visualization and encouragedme to pursue a PhD.Iwould like to thank themembers ofmy supervisory committee, JoannaMcGrenereandGiuseppeCarenini, for their thoughtful comments and suggestions that helpedme improve the dissertation.I have been fortunate to have had great collaborations over the last four years.T_hanks to Stefan Storey and the team at Sensible Building Science for being alwaysavailable for discussions and for sharing your innovative spirit. T_hanks to YannDubois for helping with the Ocupado project. T_hanks to Robert Kincaid, Mau-reen Stone, Eric Brochu, MyaWarren, and colleagues at Tableau for your supportduring my internship. T_hanks to Hasti Seifi, Karon MacLean, and Katherine J.Kuchenbecker for our collaboration on Haptipedia, which has been an enjoyableside project. T_hanks to Luce Liu for helping with the TimeElide project.My dissertation research was generously supported by Sensible Building Science,Cisco, UBC Campus as a Living Lab, Natural Sciences and Engineering ResearchCouncil of Canada (NSERC), and Tableau Software.xxvA huge thank you goes out to all past and present members, and friends, of theUBC InfoVis group for the feedback and support they provided: Jürgen Bernard,Anamaria Crisan, Kimberly Dextras-Romagnino, Madison Elliott, Shannah Fisher,Steve Kasica, Zipeng Liu, Francis Nguyen, Mara Solen, and Ben Shneiderman.Above all, I would like to thankmy family and friends, near and far, for their con-tinuous support. I am thankful for the great times and adventures with a fantasticgroup of friends, including Sampoorna Biswas, Kimberly Dextras-Romagnino,Johanna Fulda, Hayley Guillou, AdamHutter, Siddhesh Khandelwal, Neil Newman,Nico Ritschel, Anna Scholtz, Giovanni Viviani, and Alistair Wick.Finally, I dedicate this dissertation to my parents, Christa and Peter, whose uncon-ditional love and support made this journey possible.T_hank you.xxvi1 IntroductionIt is vital to remember that information– in the sense of raw data– is notknowledge, that knowledge is not wisdom, and that wisdom is not foresight.But information is the first essential step to all of these.—Arthur C. ClarkeT_his dissertation aims to demonstrate how visualization prototyping with real-world data can lead to novel knowledge contributions with high industry relevance.In this chapter, I provide the context andmotivation by explaining why, in currentpractice, visualization prototyping is not intrinsically driven by real-world data,and conclude with an outline of the contributions.1.1 Context andMotivationData is ubiquitous and has become an integral part of our daily lives. But in rawform it is of limited use. Fully automatic algorithmic solutions can only be appliedto well-defined and well-understood analysis tasks [236], and otherwise humancognitive reasoning is required to make sense of data and derive value [82].Visualization has been established as a powerful tool to explore, analyze, and com-municate data. However, the path from raw data to insights and actionable deci-sions is quite complex. Visualization designers—practitioners and researchers—are faced with a series of potential challenges related to data analysis, visual design,and development, including: complex data characteristics and heterogeneous datasources, multiple stakeholders or target users with different needs, ill-defined1tasks given in complicated domain-specific language, limited screen real estate asa precious commodity, and, above all, usability to help perform the desired tasksefficiently and effectively.To address these challenges, visualization designers use iterative prototyping asa means to explore and validate visualization and interaction techniques in closecontact with the target user groups. Prototyping is essential to move from abstractideas to concrete, tangible solutions but also to understand the needs and desiresof users early on in the design process [112].T_he visualization and human-computer interaction (hci) literature propose a largerepertoire of methods related to human-centered design and prototyping pro-cesses [38]. Creative visualization-opportunities workshops [122] and visualizationawareness [131] complement traditional methods, such as interviews and obser-vations, to actively engage users in the early prototyping phase. T_he developedprototypes range from simple hand-drawn sketches [277] to functional high-fidelityprototypes that are close to the final product.Previous work showed that toy or synthetic data, which is commonly used in proto-typing, can lead to incorrect data abstractions, less effective visualization designs,and other project obstacles [236]. While the importance of real-world data has beenemphasized before [161, 236, 246], it is often considered a resource and not a driverof the design process.T_he aimof this dissertation is to put data in the front and center by examining how adata-drivenvisualizationprototypingprocess can lead tonovel knowledge contributionsrelated to visualization algorithms, techniques, and tools.1.2 Data-DrivenVisualizationPrototypingT_his high-level process, illustrated in Fig. 1.1, is informed by previous processmodels that guide the creation and analysis of visualization systems [172, 192, 236],but is tightly coupled to the availability of datasets right from the beginning. T_heiterative process consists of six main stages that I describe below. T_his dissertationmakes contributions towards all of these stages, as I will summarize in Sec. 1.6.2Figure 1.1: Illustration of the six stages of our data-driven prototyping process.Data and context understanding. At the beginning of the process, it is imperativeto acquire an in-depth understanding of the data and problem context related tothe target domain. T_his stepmay also be referred to as requirement analysis [136, 236].T_he use ofmessy real-world data from the beginning leads to a better understandingof data characteristics and data limitations.Data abstraction.T_he fundamental understanding of data semantics, types, andcardinalities is essential for effective visualization design and to support the trans-ferability of specific results to other domains. Data abstraction does not involvethe generation of data or the invention of new data formats, but rather means thedetailed characterization of a given dataset, to describe what should be visualizedin domain-independent language.Visual encoding ideation.T_his stage focuses on analyzing existing or developingnew visual idioms and interaction mechanisms, for the given data abstraction anddesired tasks. Data can serve as a source of ideas and as a guide to delimit theconsideration space of possible solutions.3Implementation. After considering a broad design space and converging on oneor a small set of possible solutions, the ideas are concretized into tangible pro-totypes [172]. T_he implementation is tightly coupled with the ideate stage anditerative refinement is common. Data, that is of relevance and interest to the targetuser groups, allows us to rapidly generate realistic visual artifacts, which in turnleads to an increased user engagement [161] during the overall prototyping process.Validation. Early data access helps us to inquire about needs and validate visual-ization designs in realistic scenarios and within its context of intended use [113].Besides evaluating the visual and interaction idioms with target user groups, addi-tional methods are employed to validate the efficiency and efficacy of a proposedsolution, such as computational benchmarks [192]. Human visual inspection ofalgorithmic results can complement traditional quantitative analyses, to gain con-fidence and to refine algorithmic model and parameter choices.Reflection.T_he critical reflection on individual processes and results can help tovalidate findings and to provide practical guidance for other researchers [180].Schon defines reflection as “the practice by which professionals become aware oftheir implicit knowledge base and learn from their experience” [232]. T_hese lessonslearned are essential to derive new or refined guidelines that are applicable to otherproblem domains and datasets.We regularly jump back to previous stages to refine initial assumptions and ideas,as the understanding of the stakeholder needs and of the dataset semantics evolvesfrom partial to more complete.1.3 Research TrajectoryBefore diving into the contributions of this dissertation in more detail, I will de-scribe my research trajectory and howmy research objectives evolved over time.1.3.1 SpatiotemporalDataAt the beginning of my PhD program, my aimwas to study how to visually analyzeand communicate spatiotemporal data, particularly in the context of non-expert users.4At this time, I just finishedmymaster’s degree at the University of Vienna and along-term project about the analysis of bike sharing networks [203]. With the BikeSharing Atlas, we illustrated how to leverage distributed, heterogeneous data fromhundreds of cities to reveal interesting insights, not only into patterns of bicycleusage but also into urban commuting behaviours around the world. Visualizingspatiotemporal data has been well-studied but previous approaches generally takea local perspective, such as investigating data from one city, and are often limitedto a specific target group. In this project, we began to investigate how to expandsuch approaches to a more global level and for a broad range of users.When I entered the PhD program, I joined a collaboration between UBC’s InfoVisgroup and the industry partner Sensible Building Science (sbs), and together westarted the Ocupado project. SBS usesWiFi device counts as a proxy to estimatespace occupancy with respect to defineable zones within buildings and feeds thisdata into building automation systems to dynamically adapt the heating, ventila-tion, and air conditioning (hvac) to occupant usage [64]. We conjectured that withsuitable visualizations, this data could be actively useful for decision-making andresource management, beyond its use in fully-automated HVAC control systems.Existing techniques to visualize spatiotemporal data focusmainly onmovements orrely on large-scale geographic maps as their fundamental approach which is inap-propriate for indoor spatial data. Having access to extensive real-world data fromthe UBC campus, which served as a testing ground for this technology, promptedthe following questions:• Who would benefit from seeing this proxy data for human occupancy andwhat tasks can we support?• How can we leverage this data effectively while ensuring strong privacy pro-tections?• What are appropriate visual encoding and interaction techniques for thegiven data characteristics and stakeholder needs?• What guidelines can we derive to inform visualization design dealing withsimilar spatiotemporal data related to indoor environments?5We investigated these questions in a multi-year design study [236], with multiplestakeholders in parallel, which led to the design and implementation of a suite ofvisual decision support tools, and is described in detail in Chapter 2.Although the underlying domains and the tasks related to the Bike Sharing Atlasand Ocupado are very different, the intriguing underlying similarity between theseprojects lies in the data characteristics. Both projects visualize spatiotemporalnon-trajectory data, such as the number of available bikes at a docking stationor the number of people occupying a certain floor, to make the data accessible tostakeholders and to facilitate the analysis of usage patterns. In Chapter 7, I reflecton the general implications of spatiotemporal non-trajectory data.1.3.2 Non-Contiguous TimeSeries SlicesI recognized a strong need from multiple stakeholders across these projects tovisually inspect time periods of interest that are known in advance, such as the uti-lization of meeting rooms onMondays between 9am and 5pm over several months.But relatedwork focusesmostly on one single contiguous range between a start andend time and the analysis through open-ended visual exploration, such as zoomingand panning [66].We identified this special data type as non-contiguous time series slices and startedto investigate possible visual encodings as part of the Ocupado project. T_he goal oftheTimeElideprojectwas togeneralize theapproach for visualizingnon-contiguoustime series slices, which led to the following questions:• What are the unique characteristics of non-contiguous time series slices?• What are the typical analysis tasks?• What are appropriate visual encoding and interaction techniques to supporta broad range of datasets from different domains?T_he result of this prototyping process, guided by different datasets, is a visualencoding design space and an open source visualization tool that instantiates thedesign space. TimeElide allows users to upload their own datasets, slice the data,6and choose between visualization options interactively. T_he process and researchcontributions are described in Chapter 3.1.3.3 Visualization Collections andBundlesMy broad interest in the field of information visualization and the aim to bridgeacademic research and industry needs prompted me to deviate from investigatingpurely spatiotemporal data. I collaborated with the industry partner Tableau, andstarted to explore visualization bundles as a new data type of interest.Visualizationbundles combinemultiple visualizations, dashboards, anddata sources,and are the primary artifacts created by users of visualization services, or businessintelligence (bi) tools, such as Tableau1 or Power BI2. T_hese tools also allow usersto amass increasingly large collections of these bundles and share them within acompany, organization, or wider community.Although the data characteristics changed fundamentally compared tomy previousstudies, I continued the data-driven approach. I first scraped thousands of real-world examples from the sprawling collection of publicly shared bundles on TableauPublic, and by engaging in a long-term collaboration with Tableau Research I gotaccess to even richer data.Despite their widespread use, visualization bundles and related challenges haveonly been studied peripherally, but due to their unique data characteristics, existingtechniques may not always be directly applicable. Visualization bundles have anunusual combinationof characteristics suchas sparse andoften fragmented text, hi-erarchical semi-structured content, both visual and semantic features, and domain-specific expressions. More specifically, I started the VizCommender project andinvestigated content-based visualization recommendations to assist in informationseeking in large-scale collections. T_his process was driven by two key questions thatare tightly connected to the special data characteristics:• Which content features are most informative for comparing visualizationbundles?1Tableau, https://tableau.com2Microsoft Power BI, https://powerbi.microsoft.com7• What techniques can we use for comparing and ranking visualization bun-dles?We analyzed different similarity measures for comparing and ranking visualiza-tions using natural language processing (NLP) techniques, and developed a proof-of-concept pipeline for content-based visualization recommendation, describedin Chapter 4.T_he industry collaborationwith Tableau led to a long-term embedded inquirywhichprovided unique access to multiple domain experts in visualization research, ma-chine learning, andmost importantly customer insight into typical use cases andpain points related tomanaging large visualization collections and finding relevantinformation.I learned that existing visualization snippets, compact summaries of visualizations,yield poor information density and fail to help people judge the relevance of bundles.When interviewing users to evaluate the new visualization recommendationmodelparticipants repeatedly commented on the visual presentation of the recommenda-tions despite the intent to focus on the back-endmodel.Addressing this issue motivated the VizSnippets project, described in Chapter 5,where we investigated the creation of compact, human-readable summaries ofvisualization bundles to facilitate relevance judgement when users browse throughdozens of results. T_he result is a computational pipeline to compress the visualand textual content of bundles into representative previews that is adaptive to aprovided pixel budget and provides high information density.In our VizCommender publication [204], we originally referred to large sets ofbundles as visualization repositories. I now believe the term visualization collectionis a more accurate description and distinguishes them from dataset repositoriesthat do not contain visual encodings but are increasingly used in the visualizationcommunity. I will use the term visualization collection throughout the dissertationto be consistent.81.3.4 Data-First Design StudiesBesides the commonalities in data characteristics, both the Bike Sharing Atlasand the Ocupado project are visualization design studies that roughly follow thedesign studymethodology by Sedlmair et al. [236]. Design studies are character-ized by their highly iterative nature with a tight interplay between the analysisand abstraction of stakeholder needs, the transformation of domain-specific intodomain-agnostic data, and the visualization design.While design studies typically start with domain-expert stakeholders and theirvisualization needs, my projects were data-driven and primarily initiated by ac-quiring an interesting dataset rather than selecting a specific stakeholder. In these,what we now call data-first design studies, the early selection of the data constrainsappropriate choices for stakeholders. For instance, in Ocupado the use of WiFidevice connections as a proxy measure for building occupancy is good enough forsome use cases but excluded others.I reviewed additional design studies from the visualization literature and learnedthat some indicate a data-first ordering without explicitly characterizing this typeof design study. I conjectured that some authors under-emphasized their truetrajectory by bending their explanation closer to the widely recognized designstudy methodology framework.In Chapter 6, I provide a first detailed discussion of the notion of data-first designstudies and the interplay between prototyping with data and the user-centered de-sign process. T_he goal is to provide practical guidance and inspire vivid discussionsabout various flavours of visualization design studies.1.4 ANote onChronologyT_he research projects described in this dissertation spanned extensive periods oftime and were overlapping and interspersed. Fig. 1.2 illustrates the chronologicalorder of the core research periods, important milestones during my PhD program,and other research projects [239, 240] that are not included in this dissertation. T_heorder of proposition differs from the chronological order of the publications as the9research chapters are presented along thematic threads.Figure1.2: Timelinesof the researchprojectsdescribed in thisdissertationandother importantPhDmilestones. Core focus period, Milestone101.5 ANote on SupplementalMaterialsT_he Appendices A-E contain additional figures and further details about datasets,stakeholders, methods, and computational models. T_his material is not essentialto understand the presented research outcomes but may be of interest and helps inreplicating the results. T_he content of each appendix was previously published assupplementarymaterial andaccompanied the respectivepublications. See theURLsin Table 1.1 to access short preview videos and additional data from experimentsthat are not included in the appendix.Chapter Appendix Supplemental material2. Ocupado Appendix A https://osf.io/865sb/3. TimeElide Appendix B https://osf.io/yqvmf/4. VizCommender Appendix C https://osf.io/wypqj/5. VizSnippets Appendix D https://osf.io/uezfk/6. Data-First Design Studies Appendix E https://osf.io/e3p2w/Table 1.1: Supplementalmaterials.1.6 Summary of ContributionsT_his thesis offers several research contributions to the visualization communitywhich are described in detail in Chapter 2 to 6. In this section, I provide an overviewby first summarizing the contributions of each chapter and then categorizingthem into three areas. Fig. 1.3 shows an overview. In Chapter 7, I reflect on theoverall findings and discuss how the contributions are interlinked and span acrossindividual projects.11Figure 1.3:Overview of the research chapters and contributions in the dissertation.OcupadoI contribute the design study on theOcupado project that describes the process ofdesigning, implementing, and evaluating a suite of novel visualization tools forstudying space utilization at scale. A detailed analysis and abstraction of data andtasks, in the domain of facilities management and planning, informed the designprocess. T_he feedback frommultiple stakeholders and the take-up by our industrypartner presents preliminary evidence of Ocupado’s utility. By reflecting on thedesign study, I also contribute a set of generalizable design choices for visualizingnon-trajectory spatiotemporal data related to indoor regions.12TimeElideTimeElide is a novel domain-agnostic tool for visualizing non-contiguous timeseries slices. In addition, I contribute a first detailed characterization of this specialtype of time-oriented data, and a visual encoding design space as a conceptualfoundation of TimeElide. I propose the new sparkbox technique for visualizingcoarse and fine grained temporal structures of time slices within one view. Multipleusage scenarios based on real-world data from different domains illustrate theutility of TimeElide.VizCommenderT_he VizCommender project is a first investigation into content-based visualizationrecommendation and the challenges it entails. I contribute a recommendationpipeline consisting of: textual feature extraction, similarity measures for com-paring and ranking visualizations using NLP techniques, and a user interface forshowing recommendations based on a real-world visualization collection. I alsopresent a detailed analysis of applicable similarity measures that is informed by acrowdsourced study to systematically assess the alignment betweenmodel predic-tions and human judgements.VizSnippetsI contribute the first systematic approach to visualization snippet design compris-ing a top-down design framework and a computational pipeline for automaticallycompressing the visual and textual content of visualization bundles into representa-tive previews. Results based on Tableau workbooks and Observable computationalnotebooks demonstrate a substantial increase in information density compared toexisting visualization snippets. In addition, I promote amethodology for extensivevisual inspection through random sampling to refine algorithm and parameterchoices.13Data-First VisualizationDesign StudiesI introduce the notion of data-first design studies that are triggered by the acqui-sition of real-world data, and thereby reflect on two of my own previous designstudies. I present a review of 64 previous design studies and identify 16 of them asprobable cases that may indicate a data-first ordering. I propose an adaptation ofthe design study methodology framework and discuss opportunities and risks toprovide practical guidance and aid transferability of data-first design processes.T_he contributions in this dissertation can be categorized into five areas (see Fig. 1.1and Fig. 1.3):1. Data abstractions. I provide a detailed characterization and abstractionof location-based counts, non-contiguous time series slices, visualization collections,and semi-structured visualization bundles.2. Observations, inquiries, and reviews. T_he close collaboration with domainexperts or iterative testingwith target users is an integral part of the visualiza-tion prototyping process, particularly for data and context understanding andvalidation. T_he contributions in this dissertation are characterized by long-term industry collaborations and user feedback. I elicited requirements andtasks through a series of interviews and chauffeured demos, from stakehold-ers across multiple domains. I conducted an embedded inquiry at Tableauand learned about the challenges related to visualization recommendationsand visualization snippets. I created a crowdsourced experiment to com-pare human text similarity perception with algorithmic predictions, and Ireviewed 64 published design studies to examine data-first design processes.3. Visual encodings. Similarly to the abstraction of data and tasks, it is im-perative to systematically analyze visual encodings to aid transferability. Icontribute design rationales for visualizing non-trajectory spatiotemporalindoor data, a visual encoding design space and the sparkbox techniquefor visualizing non-contiguous time series slices, and a top-down designframework for generating information-dense visualization snippets.144. Artifacts. Following the data-driven prototyping approach, I implemented21 interfaces that range in complexity, from simple labeling tools for userstudies to web applications with multiple coordinated views and live datastreams. TimeElide is released as an open-source application.5. Methodological guidance. After conducting extensive reflections onmulti-ple projects, I propose a methodology for item-level visual inspection and aframework adaptation for data-first design studies.152 Ocupado: VisualizeLocation-Based CountsOverTimeAcross Buildings1Understanding how spaces in buildings are being used is vital for optimizingspace utilization, for improving resource allocation, and for the design of newfacilities. We present a multi-year design study that resulted in Ocupado, a set ofvisual decision-support tools centered around occupancy data for stakeholders infacilities management and planning. Ocupado uses WiFi devices as a proxy forhuman presence, capturing location-based counts that preserve privacy withouttrajectories. We contribute data and task abstractions for studying space utilizationfor combinations of data granularities in both space and time. In addition, we con-tribute generalizable design choices for visualizing location-based counts relatingto indoor environments. We provide evidence of Ocupado’s utility throughmulti-ple analysis scenarios with real-world data refined through extensive stakeholderfeedback, and discussion of its take-up by our industry partner.1T_his chapter is a slightlymodified version of our paperOcupado: VisualizingLocation-BasedCountsOver Time Across Buildings by Michael Oppermann and Tamara Munzner in Computer Graphics Forum(Proc. EuroVis), 39(3): 127-138, 2020. https://doi.org/10.1111/cgf.13968162.1 MotivationEfficient space utilization is a challenge for many organizations. Monitoring andanalyzing building occupancy over time can lead to valuable insights and data-informed decisions [271, 272]. Newmethods are emerging for implicit and explicitoccupancy sensing [176] and considerable attention has been devoted to using thisdata for automation inbuilding control systems to reduce energyusage [17, 140, 219].Better visual data analysis tools would allow these rich spatiotemporal data sourcesto be leveraged in many new decision-making contexts, but current visual dataanalysis tools do not suffice to support decision-making about indoor space usageover time.Previous attempts to visualize occupancy, and other indoor sensor data, are limitedto very small regions such as single roomsorfloors over relatively short timeperiods.Building informationmanagement systems are tuned for the temporal dynamics ofconstruction, and thus emphasize the 3D structure of a single building that is lessrelevant for occupancy. T_hese techniques are inapplicable to larger scale settingssuch as corporate office parks or universities where data exists for many roomsdistributed across many buildings. Conversely, systems that focus on time seriesdata related to spatial locations typically emphasize geographic maps, a spatialscale too large to capture room-level usage.We conducted a multi-year design study with multiple stakeholders in parallelto design and implement visual decision support tools centered around location-based counts. Our industry partner, Sensible Building Science (SBS), gathers andusesWiFi device counts as a proxy to estimate space occupancy in large-scale de-ployments with hundreds of rooms across dozens of buildings. Our goal was tounderstand the tasks of multiple potential stakeholders in facility planning andoperations, select those whose needs align with the characteristics and inaccura-cies of the occupancy data, and design visualization interfaces to support them.We infer approximate occupancy dynamics from location-based device counts, adatatype that provides strong privacy protection because it prevents the trackingof movements of individual people or device identifiers.17We present three contributions. First, an analysis and abstraction of data andtasks for studying space utilization in the domain of facilities management andplanning. Second, the design and implementation of Ocupado, a feature-richvisualization system that addresses multiple levels of data granularity in bothspace and time. T_hird, a set of generalizable design choices for visualizing non-trajectory spatiotemporal data relating to large-scale indoor environments. Finally,we present preliminary evidence of Ocupado’s utility based on stakeholder feedbackand collaborator take-up.2.2 ProcessT_he Ocupado tool suite was created in a highly iterative process through manyrounds of engagement with multiple stakeholders, with the CEO of our industrypartner SBS as the focal and continuous collaborator. T_he SBS Bridge softwaretracks the number of activeWiFi devices with respect to defineable zones withinbuildings and feeds this recorded data into an algorithm that interfaces with build-ing automation systems to dynamically adapt the heating, ventilation, and airconditioning (HVAC) to occupant usage [64]. We conjectured that with suitablevisualizations, this data could be actively useful for decision-making and resourcemanagement, beyond its previous use in fully-automated HVAC control systems.SBS initially provided us with static example datasets that we investigated throughinitial data sketches [161] created in a rapid prototyping sandbox environment, tosupport our iterative derivation of data abstractions. In parallel, in pursuit of taskabstractions, we conducted nine one-hour informal interviews and brainstormingsessions to elicit domain-specific analysis questions from potential stakeholdersacross eight different domains who might be interested in monitoring and ana-lyzing space usage. T_hey were identified by the SBS CEO, who also attended thesesessions. During this due diligence phase, we evaluated the conformity betweenstakeholder tasks and data affordances following from the choice ofWiFi devicesas a proxy for occupants. We ultimately selected five focus domains for continuedengagement: space planning, building management, custodial services, classroom18management, and data quality control. For the latter, the CEO himself was theintended user, to validate the quality of the data gathered by the Bridge.As we refined our task and data abstractions [192], our experimental sandboxevolved into a suite of four Ocupado interfaces. We also gained access to a livedata stream from a new version of the SBS Bridge. We continued with 12 stake-holder sessions to gain feedback about the evolving tools and abstractions. Sessionswith early (potential) and later (focus) stakeholders interleaved showing Ocupadocapabilities through chauffeured live demos with informal semi-structured dis-cussions of stakeholder needs, followed up by granting access to prototypes fortheir direct use. At least two people of our team attended each session to allowdetailed note-taking of qualitative feedback during the conversation. T_he first in-terview typically provided insights into routine tasks and potential use of Ocupado.Live demos—particularly the analysis of spatial regions stakeholders are familiarwith [161]—led to a deeper engagement and a larger volume of feedback.SBS adopted Ocupado as a front end for their Bridge software, so their CEO fre-quently gave demos to potential customers, sponsors, and partners. In additionto serving as a promoter of Ocupado, he was also an intermediary who relayedinformation to us about potential stakeholder needs and how they aligned or di-verged from the Ocupado prototype capabilities. Other stakeholders also becamepromoters themselves: stakeholders at the Cisco Innovation Centre 2 began to useOcupado for chauffeured on-site demos to showcase data from their own building.We discuss the strengths of the extended reach and the limitations of receiving feed-back indirectly in Sec. 2.9. We summarize the full set of stakeholders interviewedby domains and sectors in Sec. A.1. Ocupado has been deployed at two universitiesand one corporate office in the course of this 2-year long research project. After ourresearch work concluded, continued engineering efforts will be needed to createrobust implementations and ensure long-term use.2Cisco Innovation Centre, https://www.cisco.com/c/m/en_ca/innovationcenter/toronto.html192.3 DomainGoals and TaskAbstractionT_he five domains we target with Ocupado can be collectively referred to as facili-ties planning and operations, where a better understanding of space utilizationsupports stakeholders to optimize processes and allocate resources.One goal of custodial services is to develop smart cleaning schedules to prioritizespaces based on their actual usage instead of following traditional cleaning inter-vals. Wedifferentiate between custodial heads and custodialmanagers as end-users forOcupado. A custodial head is responsible for the day-to-day operations of a smallnumber of buildings. T_hey would benefit from live or recent short-term data to pri-oritize regions and assign custodians at the beginning of shifts. Example questionsthatwere elicited in our interviews areWhich roomsare busynow? orWhichfloorswereheavily occupied in the past 12 hours but are empty now? Custodial managers overseeprocesses across the whole campus, need to balance the workload among hundredsof custodians, make strategic decisions, and communicate them effectively.Stakeholders in the domain of space planning, focused on learning spaces, askmany analysis questions with similarities to custodial managers. In addition,they seek to identify under-capacity usage patterns of facilities. Classroomman-agement operates on long-term cycles at the level of entire terms. Both of thesestakeholder types engage with the spatial scale of the entire campus.Buildingmanagement entails abroadcross-sectionofquestions at a limited spatialscale. A building manager supervises all processes related to one or a few buildingsso that the infrastructure meets the needs of the occupants. Interviews with onestakeholder revealed the strong need for temporal comparisons of space usage.Currently, human observation would be necessary to assess the impact of spaceupgrades or to understand the typical usage of labs; a pain point is the lack of suchdata.We selecteddataquality control as a focus domain becauseWiFi occupancy sensingis a fairly new approach and entails many factors of uncertainty. Besides syntacticquality control checks, SBS seeks to understandmore complex issues, such asWhat20is the minimum size of a zone that can be captured? orHow are adjacent zones affected by alarge number of devices in one room?Aftermany rounds of iterationwith SBS andfive interviewswith focus stakeholders,we analyzed the transcripts and narrowed the extensive list of domain-languagequestions into a smaller core set focused on comparison. We further differentiatedthese comparison questions into four abstract tasks [192]. We provide a detailedmapping from domain language into abstract form in Sec. A.1; to summarize:T1 Confirm assumptions or previous observations.(e.g. do students occupy room in evenings or on weekends)T2 Monitor the current/recent utilization rate.(e.g. which rooms are empty/busy)T3 Communicate space usage and justify decisions.(e.g. space usage improved after renovation)T4 Validate the data (quality control).(e.g. check minimum size of a room that can be captured)Notably, exploratory data analysis is not a target task. Our stakeholders do not needto explore data in their daily routine, andmoreover have no training in data analysis;rather, they need information in a clear and concise form to facilitate their decisionmaking.Our original assumption was that stakeholders could be grouped according totheir abstract tasks, and that we would design different interfaces for each group.However, even after substantial attention to task abstraction we did not find a cleanbreakdown along these lines. Instead, we found that stakeholder concerns could becaptured concisely and precisely in terms of comparison with respect to differentslices of space and time. We discuss these combinations of spatial and temporalgranularity below, since it is most natural to frame them as data abstractions.212.4 DataWe use the number of loggedWiFi devices to estimate space occupancy at the levelof individual rooms across multiple-building environments, such as universitycampuses. Implicitmeasuring based onWiFi logging canbe implemented relativelyeasy on a software level and deployed to thousands of rooms without installingadditional sensors in any building where routers are already installed [17, 176, 206].T_his crucial cost-effectiveness allowed us to explore many possible use cases withstakeholders. T_he use ofWiFi devices as a proxy for human occupancy does havesignificant challenges; Sec. 2.9 discusses their affect on stakeholder selection andour design process.Although our work centered onWiFi device connections, our methods and visualencodings are sensor-agnostic and thus are transferable to other domains andproblem scenarios involving non-trajectory spatiotemporal data.2.4.1 DataAcquisitionSBS developed the Bridge software for counting and aggregating WiFi deviceswhich uses the signal strength from access points to triangulate the x/y-locationof a device. T_his method works even for unconnected phones or laptops becausealmost allWiFi devices continuously broadcast probe requests to find networks thatallow connections. SBS records these device coordinates and counts them basedon predefined zones that can be created by SBS or its clients. Individual deviceidentifiers are immediately discarded so no trajectories for individual devices canbe recovered, and theWiFi triangulation is insufficiently accurate to track smallrooms such as single-person offices so de-identifying people is less likely.At the beginning of this project, SBS shared two static database exports gatheredat two university campuses. T_hese datasets, containing several months of dataand covering dozens of buildings, were our primary resource while creating datasketches and initial visualization prototypes.In the second project phase, we switched to using an API provided by SBS thatsupportednear-real-timequeryingof the loggedWiFi connections,wherebuildings22and zones can be added and removed dynamically at any time. Wemade regularqueries every 5 minutes for two organizations: the UBC campus (the same as oneof the static databases) with 25 buildings over a time period of 8 months, with atotal of 778 zones andmore than 62 million timestamped items logged. T_he secondlive data stream came from the corporate offices of the Cisco Innovation Centre,inside one story of a large skyscraper with 27 zones in total, producing 2.95 millionitems over a time period of 13 months.2.4.2 Location-Based CountsWe record the number of devices per zone at regular intervals. By taking these snap-shots over time, we produce a spatial time series [10] for every zone that capturesfine-grained occupancy patterns. We call this data type location-based counts, andnote its non-trajectory nature. T_hese non-trajectory location-based counts allow usto analyze spatiotemporal dynamics but are fundamentally different to trajectory(movement) data that are frequently used in such analyses [199]. We note that thereare significant and intrinsic privacy advantages for this data type, in contrast to theintrusiveness of the trajectory-based standard approaches. It is notoriously easy tode-anonymize individuals from ostensibly sanitized trajectory data [70].Many tasks are well supported by location-based counts, even though obviouslythe analysis of movement flows is not. We can compare location-based counts atdifferent time resolutions and identify trends, outliers, and repeating patterns.One benefit of this data type is that we can easily aggregate counts of multipleregions to capture local and global variations, as we discuss below.2.4.3 Spatial and TemporalDataGranularityWe decompose spatial regions into hierarchical layers. T_he lowest-level unit is azone, roughly corresponding to the room level, which is customized for the stake-holder context. Examples of zones include research labs, classrooms, hallways, andmultiple-office groups. Each zone consists of one or more complex shapes and islinked to a specific floor and building. Zones can include further attributes, suchas room size or space type.23T_he higher-level layers in the hierarchy are floor, building, and campus. A floorusually contains several zones, has a corresponding floor plan and is associatedto a building. T_hemetadata for a building contains geographic coordinates (lati-tude/longitude), a list of floors, and a building name. Currently, the highest spatiallayer is a campus containing dozens of buildings in a roughly contiguous spatialneighborhood. We use the generic term region to subsume zones, floors, andbuildings.T_he spatial data that pertains to a task includes not only regions but also their spatialcontext (see Fig. 2.1). Tasks have different cardinality: somemay concern all zoneson a floor, others concern a few zones distributed across a campus, or only a singleregion.We decompose time into a short-term or long-term period. We also distinguishtemporal rollup via aggregation from the use of individual measures. For example,a custodial head wants to compare a region’s average daily utilization with thecurrent rate (T2Monitor), while a building manager wants to see individual countsover time to compare long-term trends and patterns (T3 Communicate). In additionto the usual contiguous time ranges, we also support non-contiguous time slicesfor use cases such as a custodial manager who needs to check all morning shifts(6am - 2pm) on weekdays during the last winter term (T1 Confirm).To precisely describe these spatiotemporal data granularities and link them to inter-faces, design decisions, and analysis scenarios, we use the notation for space:timecombinations shown in Fig. 2.1. For example, ZF-All:SA (compare all zones on afloor: short-term aggregated) applies to the scenario A custodial head wants to monitor theaverage utilization of all the rooms on the first floor in the past 6 hours. T_he asterisk (*)symbol indicates a wildcard that can be replaced by any of the available options,such as ZB-* for one, few, or all zones in a building.24Figure 2.1: Spatial and temporal data granularities.2.5 Ocupado InterfacesWe created a sandbox prototyping environment and a suite of four Ocupado inter-faces.We started by implementing the sandbox to rapidly explore and evaluate the de-sign space of visual encodings, in a similar spirit to efforts by Brehmer et al. [34].Screenshots and detailed descriptions of all design iterations are included in thefollowing subsections and in Sec. A.2.After obtaining a more precise understanding of our focus domains and access to alive data stream, we created four interfaces that are based on a shared underlyinginfrastructure but can be deployed independently. We call them interfaces ratherthan dashboards because they are fully interactive systems with multiple subpagesand dynamic data sources [226].We follow common practice by separating front-end interactive visualizations fromback-end data processing. T_he front ends are implemented in JavaScript and useD3 for interaction and rendering. T_he back end, responsible for data preprocessingquerying the PostgreSQL database, and for providing a RESTful API, is based onPython and the web framework Django 3.From the very start, we implemented all visualizations as independent componentsthat can be linked, enhanced, and exchanged. T_his design process supported highly3Django Python web framework, https://djangoproject.com25dynamic prototyping and allowed us to reuse components in different interfacesand to engineer for an easy transition from static to streaming data.Sandbox ExperimentationT_heprototyping sandbox contains a set of loosely coupled visualization componentsthat we created at the beginning of the project. We used two static database exportsthat were gathered by SBS at two university campuses. T_he following screenshots(Figures 2.2 to 2.9), provide a walk-through of these components and the differentvisual encoding techniques we tried and discussed with our stakeholders.Figure 2.2: Sandbox, building overview. Side-by-side views provide a high-level overview:(a) Campus map with geographical locations of all buildings. (b) Temporal overview usessparklines to show aggregated device counts over the course of one year. Y-scales are normal-ized to the maximum device count per building to reveal patterns despite varying occupancyrates between buildings. The first dataset was characterized by a large number of outages andshort recording periods for some of the buildings. Grey stripes explicitly indicatemissing data.26Figure 2.3: Sandbox, building overview. We experimented with an interactive heatmap as analternative visual encoding for the overview visualization. Users can choose between differentdailymetrics that define the color of each cell: average,minimum,maximum, and stddev. Thisscreenshot shows independent scales, normalized to themaximum value of each building.27Figure 2.4: A comparison between sparklines and heatmaps. We tried interactive heatmaps but ruled them out quickly: the higher precisionof the positional vs. the color channel for visual encodingwas indeed crucial in this case, as discussedwith Pathline [181] and studied by Lametal. [142].28Users can select a building in the overview visualization to inspect further detailsor to drill down to specific floors or regions. Fig. 2.5 shows the detail view of onebuilding.Figure2.5: Sandbox, buildingdetails. Thebuildingdetail view contains: (a) geomap that showsthe location of the building; (b) list of floors that can be used to navigate; (c) average day pro-files for different temporal resolutions. The y-axis showsnormalizeddevice counts (%) and thex-axis the hours of the day. Users can, for example, analyze the typical daily utilization onweek-days or during the summer term. A click on one of the smallmultiples opens the results for theselected time period in themain view.We can change the spatial aggregation frombuilding tofloor or zone (see next page).We also experimented with geographical maps to provide a high-level overviewof the distribution of devices on a university campus, as shown in Fig. 2.6. T_hisprototype was rejected because it shows location-based counts averaged over a longtimeperiod andonly fromabird’s eye view. T_his viewwasnot directly relevant to ourstakeholders because their analysis questions are at a much lower data granularitylevel.29Figure 2.6: Sandbox, campusmap smallmultiples. We examined the use of geographicalmapsto show thedistributionofdevices onauniversity campus. Themapsare createdwith theopen-source library leaflet.js that uses kernel density estimation to generate the heatmap overlays.We also clustered regions based on their utilization rates and visualized the resultsin a dedicated sandbox interface, shown in Fig. 2.7 and Fig. 2.8. T_he goal of thisendeavour was to identify regions on a campus that are usually very quiet or onlybusy in the morning based on a clustering approach. During an early project stage,we thought about embedding extensive clustering and prediction models intoOcupado. T_his interface was intended as an internal tool for us and our collaboratorto inspect clustering results that we generated offline because of the large volumeof data (input for clustering: 1 row = 1 “region” day). We abandoned this approachcompletely because of its lack of semantic meaning for our focus domain. Similarto the average day profiles, we lose a lot of nuances by clustering that are neededto answer common stakeholder questions. Nevertheless, the workflowmay be ofinterest for other use cases.30Figure 2.7: Sandbox, clustering interface. In this example, we generated 5 clusters and the usercan choose between absolute device counts or normalized counts based on the maximum ofeach region. (a)Multi-series line chart shows the average of each cluster. We can see thatmostof the days are in the blue cluster. The absolute device count during these days was very low.(b) Stacked bar charts for different regional levels. Each row corresponds to a building, floor orzone. Thewidthofabardenotes thenumberofdaysa region is inoneof the5clusters. Selectionwindows can be dragged across the the stacked bar charts to filter regions. (c) The timeline dis-plays the temporal distribution of each cluster as bars. (d) The heatmap displays the locations(building, floor) of the selected zones.Figure2.8: Sandbox, clustering interfacewithdata froma smaller campus.We selectednormal-ized device counts and filtered the regions with selection windows that are displayed as black-bordered rectangles on top of stacked bar charts.31Figure 2.9: Clustering results with data from a smaller campus. Static visualization that showsindividual cluster items (= individual days).Campus Explorer InterfaceT_he goal for our first interface, after our initial sandbox experimentation, was toimplement a general-purpose tool centered around location-based counts thatwould serve the needs of multiple stakeholders who focus on analysis. Screenshotsof the Campus Explorer Interface are shown in Figures 2.10 to 2.13. Although thisinterface has considerable functionality and a product-like look, it is a technologyprobe [112] andwas intended to be a vehicle for continued iteration on visualizationdesigns and continued task elicitation. T_he interface facilitates cross-building32analyses, provides global filter options that can be applied to all views, and enablesusers to look at the data from different angles and levels of detail (T1-T4).Figure2.10: CampusExplorer Interface, region subset view. (a) Control panel for filter anddisplaysettings. (b) High-level region overview shown as quantitative and categorical data stripes. (c)Regionsubsetviewprovidingmid-leveldetails for regionsselected intheregionoverview;userscan choose between different visual encodings for it. Clicking on one region opens a modalregion detail windowwith low-level details.Figure 2.11: Campus Explorer Interface, region subset view. Many discrete time sessions, such asall weekdays during a long recording period, are displayed as superimposed line charts. Differ-ent temporal filter options help users in narrowing down the search space.33Figure 2.12: Campus Explorer Interface, region subset view. Aggregated sessions are presentedas line chartswith confidencebands to answer questions such asWhendopeople usually leaveonweekday evenings?Figure 2.13: Campus Explorer Interface, region subset view. The spatial heatmap provides acampus-level view on device counts. Each row in the grid represents a building and each col-umn indicates a floor. A cell is composed of one or more subcells, one for each zone, where itssize and color dependson thenumberof devices. Wedecided toencode the samevariablewithtwo channels (size and color) to increase saliency. Users can hover over subcells to seemore de-tails or click on it to open the floor plan. This visualization approach provides a way to locateregions with certain characteristics and to investigate spatial relationships.34T_heCampus Explorer interface is flexible and powerful, as it supports a cross-cut ofdata granularities in space and time (* except [ZC,FC]-All:LI ), but it is quite complex.[ZC,FC]-All:LI is not supported because showing individual long-term counts of allzones would be computationally very expensive and is also not required to supportthe given tasks. Over the course of many interviews and brainstorming sessions,it became clear that some common domain questions would be easier to addresswith a simpler interface, especially for users who monitor only a small subset ofregions and are not trained data analysts.Our first foray into simplifying the sometimes-overwhelming Campus Explorerwas in the form of application presets that provide single-click access to any com-bination of actions that could be accomplished through the interactive interface.One example isSelect all regions that are associatedwith theComputerScience department,rank them in descending order based on the current device count compared to the maximumactivity during the last 12 hours, and show the top 10 as sparklines. (ZB-Few:LI ; Fig. A.13)Building Long-Term InterfaceMany questions concern only zones within a single building, even for stakeholderswho are responsible for regions across a whole campus. To enable a more target-oriented analysis, we created an interface that lets users inspect location-basedcounts in a similar way as in the Campus Explorer but only for a specific building( [ZF,ZB]-*:L* , FB-One:LI , FB-[Few,All]:LA ; Fig. 2.14 and Fig. 2.15).35Figure 2.14: Building Long-Term Interface. (a) Interactive floor plans for selecting regions of inter-est. (b) Per-floor rollups showing average utilization. (c) Each selected zone shown as typicalday profiles or sparklines.Figure2.15: BuildingLong-TermInterface. Typicaldayprofiles formanyzones, automatically sizedto fit onscreen.BuildingRecent InterfaceSome stakeholder questions are focused on live or recent short-term data wheredetailed historical data may be distracting. T_he Building Recent Interface, shown36in Fig. 2.16, explicitly tackles T2Monitor and facilitates a holistic short-term view ofone building ( [ZF,ZB,FB]-*:S* ). In the left sidebar, we also show a direct comparisonto aggregated historical device counts (FB-*:LA , FB-One:LI ).Figure 2.16: Building Recent Interface, overviewwindow. (a) Floor plans sized to all fit within onescreen with zone-level superimposed circle symbols showing live device counts. (b) Per-flooraggregate trend charts showing device counts over past 12 hours and prediction for the next 3hours (red dashed line). (c) Alternative sidebar showing aggregate per-floor usage for typicalday vs. current live data.Region Comparison InterfaceComparing a small subset of regions or time periods in detail is relevant for alltasks except T2 Monitor; stakeholder examples include analyzing the impact ofspace upgrades, comparing summer vs. winter use, and assessing the custodialworkload of different floors (*-[One,Few]:L* ). T_he other interfaces only supportthis task to a limited extent because the faceting approach can hinder direct localcomparison [116]. In the Region Comparison Interface, shown in Fig. 2.17, wedisplay data frommultiple regions in the same space, superimposed.37Figure 2.17: Region Compare Interface. This interface allows users to compare a small subset ofregions or time periods. A recurring task (T6), for instance, to analyze the impact of space up-grades, to compare summer vs. winter use, or to help assess the custodialworkload of differentfloors.382.6 OcupadoDesignWe now describe the individual components of our visualization prototypes, andthen discuss the rationale for visual encodings and interaction techniques withrespect to our data and task abstractions. T_he accompanying video shows the lookand feel of the interactive interfaces (https://osf.io/h5k4r/).2.6.1 ViewCoordination and LayoutFig. 2.18 shows a schematic overview of the sandbox and the four Ocupado in-terfaces, which feature carefully designed linkages betweenmultiple coordinatedviews.Figure2.18: Schematicdiagramsto illustrate theviewcompositionand interrelationsofallOcu-pado interfaces.39T_he sandbox starts with a campus-level overview of all buildings on a single page,with drill-down navigation into the spatial hierarchy where clicking on a regionselects it to see in more detail. Each drilldown transition to a building-level page, afloor-level page, or a zone-level page requires a page reload. T_his static navigationpath between pages impedes transitions between overview and details andmakecomparisonsbetween regionsdifficult. All subsequentOcupado interfaces (CampusExplorer, Building Long-Term, Building-Recent, Region Comparison) address thislimitation by allowing users to switch between views seamlessly.T_heCampusExplorer andBuilding Long-Term interfaces incorporate a two-columnlayout with the region selector (high-level) on the left and the region subset view(mid-level) on the right. T_he region selectors are shown in Fig. 2.10b for the CampusExplorer and in Fig. 2.14a for the Building Long-Term Interface. Selected subsetsare shown in Fig. 2.10c and Fig. 2.14c; different visual encodings for region subsetviews can be selected depending on the task, as we describe in Sec. 2.6.2.T_he Campus Explorer provides a control panel at the top, as shown in in Fig. 2.10a,that allows users to narrow down the results and to customize the display settings.Activitypatternsprovide shortcuts to complex combinationsof actions, asdescribedin Sec. 2.5.A click on a specific region opens a modal window that contains the region details(low-level), as shown inFig. 2.19. All the selections anddisplay settings remainactiveuntil the user returns to the overview visualizations to continue the analysis. T_hismulti-tier approach [5] in the Campus Explorer and Building Long-Term Interfacefacilitates fluent transitions between three levels of data granularity without pagereloads and loss of active selections. T_he Building Recent interface also integratesan on-click detail view, for users who want to inspect a specific region more closelyand beyond the 12 hour short-term time window.40Figure2.19:Regiondetail view. (a) Zoomable binned time series chart. (b) Floor plan for spatialcontext and navigation. (c) Typical day profiles.T_heRegionComparison Interface is an exception and does not provide any overviewor detail pages. Users begin the analysis by choosing regions and time periods ofinterest, similarly as with the Campus Explorer control panel, and the results aredisplayed in multiple visualizations within the same page.2.6.2 Visualization ComponentsWe now discuss the visualization components used in the selectors, the subsetviews, and the detail views.Region SelectorRegion selectors provide a high-level overview and enable users to select a subsetof regions based on their location or utilization. In the Campus Explorer, we use asimplified version of the LineUp visualization [94] that we call data stripes as anabstract sortable representation of many regions, as shown in Fig. 2.10b (*:[S*,LA] ).Depending on the selected region level, a row corresponds to a zone, floor or build-ing and each column encodes a categorical or quantitative attribute, such as average41device count. T_he selection window (black bordered) can be dragged across the rowsto pick a region subset. T_he Building Long-Term Interface uses interactive floorplans as selectors to show the whole building while preserving spatial contiguity,as shown in Fig. 2.14a. Users can click on individual zones or select whole floors.Region Subset ViewWe divide the visual components for analyzing region subsets into two groups, em-phasizing either temporal or spatial aspects. Fig. 2.20 shows all subset componentsand their characteristics.Figure 2.20: Visualization components for region subset views, emphasizing either temporalor spatial aspects, to facilitate comparisons between contiguous regions, such as rooms on afloor, or between a subset of regions that are distributed across a campus.T_hewall of sparklines (see Fig. 2.10c for an example instantiation within the Cam-pus Explorer) shows location-based counts to quickly scan long-term patterns, toidentify data gaps (encoded with grey diagonal stripes), and to compare regions atthe overview level. It works well for contiguous time periods (*:LI ,≤ 30 regions).T_he explicit indication of missing data by the grey stripes was an iterative improve-42ment over our first attempt, which relied on analysts to notice anomalous plateausandmake conjectures [247].Some analysis questions focus on very specific, non-contiguous time slices. A usermaywant to analyze the activity inmultiple regions during Friday evenings. We usebox-plot-bars [16] to visualize these time slices while also retaining the underlyingcontinuous time line. T_he light blue area shows the min-max range, the blue areashows the interquartile range (IQR; between the 25% and 75% percentile), and thedark blue line indicates the average device count (*:LA ,≤ 30 regions).We use confidence-band line charts to visualize aggregated days and time slicesto help analysts reason about typical activity, such asWhen do people usually leave onFriday evenings? (*:LA ,≤ 30 regions) Despite the confidence bands, aggregating alldevice counts hides important details that are crucial for certain domain questions,such as identifying outliers or unusual patterns on one of the days. We added an-other view that supports these local time-slice comparisons by showing individualline charts superimposed, as illustrated in Fig. 2.20d (*:LI ,≤ 30 regions).Users can globally choose between independent and absolute y-scales that areapplied to all temporal visualizations. Independent y-scales emphasize temporalpatterns despite widely differing region sizes and utilization rates. Absolute scalesensure consistent axes and enable direct comparisons of device counts betweenregions.We use floor plans and spatial heatmaps to visualize current device counts andaggregatedmetrics in the spatial context of onebuildingor thewhole campus. Floorplanswith circle symbols are the core component of the Building Recent Interfaceand present an at-a-glance view of all regions within a building ( [ZF,ZB]-*:S* ,≤ 10floors). T_he spatial heatmap is an abstract campus representation, where each rowcorresponds to one building; it is a concise depiction of multiple floor plans thatpreserves spatial contiguity at the building level ( [ZF,ZB,ZC]-*:S* ).43RegionDetail ViewAll the detail views contain a zoomable binned time series chart that shows thedevice count over time at different levels of granularity. As with the line charts, weuse confidence bands to show the min-max interval and interquartile range forthese dynamically sized time windows (temporal bins). Initially, the bands aroundthe average line are wide but the further a user zooms in, the narrower they get,until the dark blue line represents the actual device count and the bands disappear.T_his transition is illustrated in Fig. 2.21.T_his view also includes charts presenting the average daily utilization on weekdaysand weekends as line charts with confidence bands, similarly to Fig. 2.20c.At the floor or zone level, an interactive floor plan is displayed as a side-by-sideview (see Fig. 2.19b). It helps users to navigate between regions and to get a senseof the physical location and structure of a zone. T_he detail view of a building showsinstead all floor plans, similarly to Fig. 2.16 but without circle symbols (Fig. A.23).Figure 2.21: Binned time series chart at different zoom levels. (a) 13 weeks of high frequencydata are aggregated and displayed with confidence bands to avoid visual clutter. (b) Zoomedin to reveal more details, with narrower confidence bands. (c) At maximum zoom, confidencebands disappear and the line represents the original device count.2.6.3 Floor Plan LinearizationWe use a shortest path algorithm to create a linearized order for the per-zonedisplays of sparklines or line charts in the region subset view of the Building Long-44term interface, based on spatial contiguity within the floor plans. Fig. 2.22 showsthe order of spatial regions after applying our algorithm on three different floorplan examples.High-level algorithm:1. In the outer loop, traverse each floor in order from top to bottom floor.2. Within each floor, the start position is at the top left corner (x = 0; y = 0).3. Compute the bounding box for all shapes/zones.4. Calculate the distance to the nearest bounding box centroid (nearest zone).Save the zone ID that is assigned to a shape in an ordered list and label theshape as visited. Repeat this step until all shapes are visited.45Figure2.22:Derive a linearizedfloor-planordering for regionsusing a shortest path algorithm.2.6.4 DesignRationaleWhy data stripes? Data stripes can show multiple attributes of several hundredregions in a compact view (*:[S*,LA] ). Enhanced with an interactive selection win-46dow and ranking options, it serves as an overview in a linked-view. In the sandboxenvironment, we experimented with a geographical building map as a selector butquickly rejected it because it cannot show the rooms and floor levels of the spatialhierarchy.Why juxtaposed sparklines? Sparklines provide a high-level overview of manyregions over extensive time periods within one view (*:LI ). We can display about 30sparklines andmore thanoneyear ofdatawith a sufficient level of detail on standarddisplays. Location-based counts are averaged based on three hour windows toavoid occlusion and latency. We ruled out superposition because distinguishing theidentity of individual lines is too challenging with a large number of regions [116].We tried interactive heatmaps, but ruled them out quickly: the higher precision ofthe positional vs. the color channel for visual encoding was indeed crucial in thiscase, as discussed with Pathline [181] and studied by Lam et al. [142]. See Fig. 2.4for a direct comparison.Whybinnedtimeseries?Weusea temporal binningapproach toavoidvisual clutterand rendering latency due to high-frequency data. In contrast to data aggregationapproaches that lead to compact time series representations [156], we supportdynamic binning through user interaction. When users zoom in, the visible timeframe (bin) gets smaller to reveal more details.Whynot use spatially embedded time series?We investigated spatially embeddedtime series and cartograms, for instance as used byWood et al. [291] in geographicsmall multiples to study bike sharing dynamics, as a viable option to visualizelocation-based counts. We ruled it out because of the indoor spatial data modeland the highly varied region sizes, which would require trade-offs that affect thestatistical or spatial accuracy.Why all floor plans side by side? Stakeholders in facility planning and operationswork with printed floor plans on a regular basis, for example, to assign shifts or tocommunicate renovations, so this depiction of spatial context is both familiar andhighly relevant for many use cases. Whenever possible, we integrate floor plansto capture the spatial context, providing a spatial overview and allowing users tolook up and compare zones. On the right side of Building Recent and the left side47of Building Long-Term, we automatically size the plans so that all floors fit onto asingle screen without scrolling on a standard desktop display. T_his approach workswell for buildings with up to ten floors, which is the limit of our example datasets.Campus-wide analyses and cross-building comparisons ( [ZC,FC,BC]-* ) necessitatea higher level of abstraction, as is provided by the data stripes.Whynot use exploded views (3D buildingmodel)? Exploded views [153] are a com-mon technique to visualizemultiple floors of a building, particularly in architectureand interior design. We ruled out this option because of the myriad limitations of3D visualizations. T_he side-by-side layout of 2D floor plans provides a bird’s eyeview on a building without occlusion.Why circle symbols on top of floor plans? Our stakeholders strongly expected tosee absolute numbers for device counts (T2Monitoring). We did experiment withcolor-coding zones within floor plans as choropleths [297], which suffered fromthe known problems that small zones containing large absolute numbers were notsalient. After several design iterations we chose superimposed circle symbols (sizecoded by radius), augmented with a numeric label, where the symbol can extendbeyond the zone boundaries. T_he circle scale is automatically adjusted to the sizeof the largest zone in the building. T_his choice leaves open the future option ofcolor-coding zones based on tags such as space types.Why order spatial regions according to floor plans (linearization)? When usingdata stripes as the selector in the Campus Explorer, the order of the stripes controlsthe order of the regions in the subset view, so they are completely divorced fromany spatial context and stakeholders found it hard to reason about them (Fig. A.15).In the Building Long-Term Interface, we devise an order for regions in the subsetview that preserves important aspects of spatial contiguity within each buildingby linearizing a traversal of its floor plans, preserving floor-level structure andensuring that spatially nearby zones are close to each other in the display (Sec. 2.6.3).Linear representations are often used to display multivariate data, particularlygraphs, at scale [196, 209]. Decisions between abstraction and spatial contiguitypervademany aspects of the Ocupado design and are primarily driven by scalabilityissues. Our stakeholders strongly signalled their comfort with and preference48for using floor plans many times, but these cannot be the principal medium todisseminate all informationbecause of the very limited spatial extent of each zone atbuilding-level scales. While we can encode one or two numbers with circle symbolsor glyphs, we cannot show the many data points contained within sparklines orline charts. Our linearization approach attempts to combine the benefits of bothin amiddle ground. We use a similar automatic sizing as above to fit all selectedzones in a single screen.Whymodal windows? We use modal windows for region details instead of a fo-cus+context technique [304] to provide different perspectives, such as the floorlayout and the zoomable time line, to not overwhelm users with too many linkedviews. T_he region selector and region subset view are shown side by side and athird layer, with region-level details, would subdivide the page layout even further.Seeing all levels of details within onemulti-view visualization is not required for thegiven tasks. To guarantee fluent transitions, all the selections and display settingsremain active until the user returns to the overview visualizations to continue theanalysis.2.7 Analysis ScenariosWe illustrate the capabilities of Ocupado in three analysis scenarios conductedby the author of this dissertation using real data, based on themes that emergedrepeatedly in sessions with stakeholders.2.7.1 DataQuality ControlT_hemost central task for our SBS collaborator is to identify inaccurate records andto analyze issues that affect data integrity (T4 Validate). In this scenario, an analystuses all four Ocupado interfaces and is able to detect four types of quality issues:Static devices (e.g., printers) that are continuously sending signals. T_his problemwas expected as one of the limitations of usingWiFi counts as a proxy variable. T_heconfidence-band line charts (subset view) and the zoomable timeline (detail) makethe shifted baseline for device counts immediately visible, as shown in Fig. 2.23a.49Missing data records were also expected because the UBC campus is a test en-vironment for the underlying Bridge data collection and preprocessing pipeline.Nevertheless, the duration and number of system outages was an important newinsight surfaced by juxtaposed sparklines, shown in Fig. 2.23b, which provided adense overview of the recording periods and outages of up to 40 buildings on onescreen. T_heanalyst uses thedata stripes in theCampusExplorer to rankzones basedon average device count in order to locate constantly empty regions (ZC-All:R ). Allinterfaces allow live monitoring and ensure early detection of any system failures.Suddendrops are artifacts caused by theWiFi recording. T_he device count drops tozero for a single data point before it is up again at a normal level. T_his unexpectedbehavior is different from missing data values where nothing at all is recorded.T_his type of failure is particularly noticeable in the binned timeline, as shownin Fig. 2.23c. T_he overview visualizations conceal the drops due to the binningover the long recording periods. T_he analyst observes this issue in different zonesacross many buildings. To avoid confusion, new versions of Ocupado interfacesautomatically filter out these sudden drops by default, through the interpolation ofcorrect values [29].Incorrect zone allocations were unexpected and pose a challenge for WiFi occu-pancy sensing. Inaccurate device coordinates and floor bleed-through—a largeevent on one floor results in a rapid increase of devices on the floor directly below orabove— can lead to wrong inferences. Possible causes are triangulation problemsbecause of a small number ofWiFi routers, wall attenuations, and uncalibrated net-works. T_his problem is difficult to debugwithout humanobservation to cross-check.Ocupado supports the analysis by displaying the actual device coordinates on floorplans for debugging. Analysts can immediately see if coordinates are plausible, forexample, if they are within the floor plan boundaries. In the Region ComparisonInterface, the analyst can compare multiple regions to investigate effects such asfloor bleed-through (FB-Few:L* ). Fig. 2.23d illustrates how a rapid increase in oneroom (blue) affected the device count in an adjacent room (green).50Figure 2.23: Example quality issues: (a) static devices add noise and shift the baseline, result-ing in a utilization rate that is always above zero; (b) sparklines uncover data gaps; (c) randomsudden drops for single data points that are filtered out; (d) direct comparison of two regionsconfirms human observation that an event on floor leads to an increase of devices in adjacentfloors.2.7.2 RecentUtilizationPatternsCustodial heads need to assign custodians to regions at the beginning of shifts,a recurring task (T2Monitoring) with difficulties due to absences, one-off events,and other factors that influence the schedule. Custodial managers reported theprioritization of regions as an important task for custodial heads, where the scopeis usually limited to one building at a time. T_he Building Recent Interface can beused to inform these decisions.T_he analyst opens the interface and live activity of all zones in the Forest Sciencebuilding is superimposed on floor plans, as shown in Fig. 2.16a (ZB-All:SI ). Twomajor hotspots in the main floor stand out immediately. T_he left sidebar showsthe device count during the last 12 hours aggregated per floor (FB-All:SI ). A dashedline indicates how the utilization might change during the next three hours basedon averages for that day of the week. T_hey switch to typical day and notes that thedevice count in the first floor is much higher than usual at this time (FB-All:LA ).T_he analyst wants to knowmore about these two hotspots and selectsmean insteadof live in the menu bar (Fig. A.31), and adjusts the time range slider. T_he symbolsoverlaid on floor plans now show the mean value during the selected time window51and are updated automatically (ZB-All:SA ). T_hey ascertain that these two zones gotbusy recently and instead other zones on the same floor were occupied before andare empty now. In this case, custodians can be assigned to the empty zones.2.7.3 Campus-Wide Situational AwarenessCross-building analysis was required by some of our stakeholders and is supportedby the Campus Explorer. For example, the question What is going on around thecampus? (T1 Confirm, [ZC,FC,BC]-*:S* ) has been repeatedly articulated.T_he analyst selects the Campus, live overview activity pattern from the control panel.T_he interface automatically sorts zones based on live activity in descending order,selects the top 70, and visualizes them in a spatial heatmap (ZC-Few:SI ). T_he analystruns this query on the weekend, and they are unsurprised to see high activity inNEST, a student union building. However, the high device counts in DMP aresurprising in a building primarily used for lectures. In the control panel, theyfilter on DMP to exclude all other buildings (Fig. 2.24) and changes the view tosuperimposed line charts (ZB-Few:LI , Fig. 2.20d). Since each line denotes oneday, clear patterns are not discernible due to the long (8-month) recording period.T_he analyst selects only weekends with the discrete time sessions menu, and seesfive outlier days stand out from the other near-zero lines, seeing that occasionalafter-hours use does occur.52Figure 2.24: Campus explorer, control panel. Regions can be filtered by building.2.8 RelatedWorkWe review relevant previous work on spatiotemporal visualizations in general,without trajectories in specific, and space usage.2.8.1 General SpatiotemporalDataVisualizationTechniques to visualize time-oriented data have beenwell studied in recent years [4,35]. Our more specific focus is on spatiotemporal data [9]. T_hemany techniques tovisualize geographical trajectory or origin-destination data include FlowMap Lay-out [214], Space Time Cubes [75, 137], or Flowstrates [31]. In contrast, the concernwith Ocupado is on non-trajectory, event-based data. We facilitate the analysis andcomparison of spatial regions across multiple temporal resolutions, not visualiza-tion of movements per se.Visualization techniques for time series data are often applicable to scenarios thatinvolve spatiotemporal data. Wijk’s approach to visual time series clustering [268]provided inspiration for our superimposed line chart component (see Fig. 2.20d).532.8.2 Non-Trajectory SpatiotemporalDataVisualizationSpatiotemporal datasets are often event-based and consist solely of geo-referencedtimestamped itemswithout trajectory information. Ourdataabstractionof location-based counts is one instance of such data, but in particular comes with guaranteesthat we collect data regularly from the same locations over time.Kim et al. [125] proposed a flow extractionmodel based on kernel density estima-tion to visualize flow patterns without having an explicit notion of movement. Incontrast, we do not attempt to reconstruct flows from the recorded location-basedcounts; our underlying data collection infrastructure was designed expressly topreclude such usage.Many previous systems aggregate and visualize dynamic one-off locations, suchas GPS coordinates that are assigned to social media posts [45] or crime incidentreports [163]. Miranda et al. [184] defined the concept of an urban pulse to capturethe spatiotemporal activity in a city based on geo-located Tweets and Flickr uploads.T_he goal of these systems is to identify and visualize geographical hotspots whileour work is focused on counts from very specific indoor zones.More closely related to our work are systems that visualize data that is collectedregularly at the same spots, such as weather stations [275] or ocean observing sys-tems [21]. Oppermann et al. [203] used linked views to visualize spatiotemporalchanges in bike sharing networks based on station fill levels. T_hese methods havetemporal resolution restricted to predefined periods, and rely on large-scale ge-ographic maps as their fundamental approach which is inappropriate for indoorspatial data. In contrast, our data model has a spatial hierarchy comprised ofbuildings, floors, and rooms, requiring an alternative approach.2.8.3 SpaceUsageVisualizationSeveral studies specifically address the visual analysis of space usage. Ivanov etal. [115] proposed an occupancy detection system based on motion sensors andvideo cameras. In a side-by-side view, users canwatch the video footage andanalyzethe recordedmovements and occupancy. In recent years there has been a growing54interest in visualizations of museum visitor flows [168, 256]. Kuutti et al. [140]proposed a real time activity visualization in addition to a demand-controlledventilation system. Occupancy is estimated via infrared sensors and overlaid on afloorplan to support livemonitoring. Although these systemsdo informabout spaceoccupancy in small regions, their primary mission is the analysis of movements;they do not support in-depth analysis of non-trajectory data.Building management systems have been proposed that visualize occupancy tostudy energy-related behavior [51, 217]. Most closely related to our approach is thework fromVerbree et al. [271] who alsoworkedwith facilitymanagers at a universityto analyze usage patterns based on WiFi signals. T_hey created a visualizationdashboard as part of their study but the proof of concept is based on a 1-weektime frame and a small number of rooms. Ocupado allows users to explore muchlarger-scale environments, with hundreds of zones andmonths of recordings.Our collaboration partner SBS had an existing interactive interface for monitoringone single floor at a time. It did not support comparisons, extensive analyses, orany of the abstract tasks in full.Google’s popular times histogram [93] helps users to estimate wait times. We use asimilar technique in the Building Recent Interface to show per-floor usage for atypical day.2.9 DiscussionWe discuss spatial and temporal data granularity, feedback channels, and proxymeasures.Interfaces by Data Granularity. We implemented four interfaces each targetingdifferent combinations of spatial and temporal data granularities. T_he same stake-holdermight use different interfaces for different tasks but does not switch betweenall four routinely. We originally envisioned interfaces tailored to each focus domain,but the process of data and task abstraction led us to realize that data granularity isa better grouping strategy than stakeholder domain. In particular, constraining thedata dimensions allowed us to significantly reduce the complexity of the Building55Long-Term, Building Recent, and Region Comparison Interface to increase ease ofuse.Combining individual visualization components deemed to be effective into aholistic and deployable system, while maintaining a fluidity of use, is a challengingand often neglected endeavour in visualization design studies (see Pitfalls 24-25 inDesign StudyMethodology [236]). We invested this level of engineering effort toachieve deployable software in hope of observing real-world use by our collaboratorand third-party customers. T_he take-up by SBS, and pilot installations of ourprototypes at UBC, Cisco’s Innovation Centre, and Carleton University showedthe utility of the Ocupado interfaces for different user groups. To study sustainedlong-term use, continued efforts and robust implementations of these interfacesare needed.Multiple Feedback Channels. In an iterative process with multiple parallel stake-holders, comparable with the multiple channels of discourse byWood et al. [292],we discussed domain-specific questions, presented ideas, and gathered feedbackover the course of more than two years. While we personally conducted interviewsprimarily with stakeholders on a university campus, the SBS CEO took the role ofa promoter andmade vigorous efforts to present Ocupado and to assess potentialusage scenarios with external clients. In total, he gave 24 demos to potential stake-holders for whom direct contact with us was not feasible. T_his type of outreachresulted in a new feedback channel. Instead of directly attending or leading demos,we received high-level informal feedback from our industry partner. Although thefeedback was filtered and summarized, it was actively useful in the core designand implementation stages. Despite the potential limitations that an intermediarycould mischaracterize actual needs, we found that the additional feedback channelprovided valuable insights. T_his promoter role may be a useful addition to the otherdesign study methodology roles identified in previous work [236].Closely related is the use of Ocupado by SBS and Cisco’s Innovation Centre toshowcase their back-end data collection pipeline on live data. While the human-in-the-loop occupancy analysis is just one possible use case for their product, theydecided to use Ocupado extensively to demonstrate their technology in a real-world56setting. T_his non-analytical task was unexpected but an additional validation ofthe effectiveness of our visualization design.ProxyMeasures are often required to stand in for variables that cannot be directlymeasured or if the acquisition is prohibitively expensive; the choice to use them ispart of the operationalization process [86]. We conjectured that usingWiFi devices asa proxy for human occupancy would be good enough for some use cases but excludeothers. Initial talks with potential stakeholders confirmed this assumption. T_hemajor threatwith proxymeasures is a lack of correlationwith the variable of interestthat would entice analysts to draw wrong conclusions. Previous studies [17, 206]ascertained thatWiFi counts can estimate occupancy with a relatively high degreeof accuracy and laid the groundwork for Ocupado. In addition to stating that thevisualizations show device counts and not head counts, we deliberately chose notto visualize headcount capacity limits in rooms to remove the temptation of directcomparisons. Nevertheless, we ourselves sometimes fell prey to confusion on thisfront, so it may remain a hazard for users.A useful area for future work would be a dynamic, data-driven estimate of theheadcount-devicecount ratio suitable for a specific context. A single static formulawould not suffice, since the number of loggedWiFi devices per personmay vary dra-matically. In addition, even if we would have sufficient information to approximatethe head count, theWiFi sensingmethodmight be too unreliable and inaccurate inlarge-scale uncontrolled environments. Further data analysis using Ocupado couldinform future studies.2.10 SummaryT_heOcupadovisual decision-support tools showspaceusagepatternswithaprivacy-preserving architecture based on location-based counts to support analysis acrossseveral hundred zones in dozens of buildings. Interviews and feedback frommanystakeholders engaged in facilities planning and operations were incorporated intodata and task abstractions, exposing the need for visualization interfaces thatsupport flexible combinations of data granularities in both space and time. We con-tributed generalizable design rationales to visualize non-trajectory spatiotemporal57data related to indoor environments and discussed the adoption of Ocupado by ourindustry collaborator.583 TimeElide: Visual Analysis ofNon-Contiguous TimeSeriesSlices1We introduce the design and implementation of TimeElide, a visual analysis toolfor the novel data abstraction of non-contiguous time series slices, namely timeintervals that contain a sequence of time-value pairs but are not adjacent to eachother. T_his abstraction is relevant for analysis tasks where time periods of interestare known in advance or inferred from the data, rather than discovered throughopen-ended visual exploration. We present a visual encoding design space as anunderpinning of TimeElide, and the new sparkbox technique for visualizing fineand coarse grained temporal structures within one view. Datasets from differ-ent domains and with varying characteristics guided the development and theiranalysis provides preliminary evidence of TimeElide’s utility.3.1 MotivationTime series data is often visualized in a way that emphasizes the continuous se-mantics of discretized data samples, where there is a single contiguous range ofdata between a start and end time. Visualization systems support this data type1T_his chapter is a slightly modified version of our short paper TimeElide: Visual Analysis of Non-Contiguous Time Series Slices by Michael Oppermann, Luce Liu, and Tamara Munzner, accepted forpublication in Proc. IEEE VIS Short Papers 2021.59through interactive exploration in order to help users in identifying interestingtime periods. We consider a problem scenario that differs from this standardparadigm. We want to support analysis tasks that relate to specific time periodsof interest known in advance rather than determined on the fly, and that pertainto dozens of non-contiguous time intervals rather than just a few selected timeranges. Examples with these temporal data characteristics can be found acrossmany domains: working shifts for job scheduling, public holidays as times of heavymuseum attendance, product sales on specific days, occasional bike rides recordedwith a smartphone, and so on. We refer to the combination of known time intervalsand time series data as non-contiguous time series slices.When working on visualizations for building occupancy [201] we repeatedly experi-enced the need to analyze this type of non-contiguous temporal data. Aforemen-tioned examples from other domains have very similar data characteristics butexisting tools do not support such analysis explicitly.To fill this gap, we contribute the design and implementation of TimeElide, adomain-agnostic tool for visualizing non-contiguous time series slices. We presenta detailed characterization of this data type and a visual encoding design space thatunderlies TimeElide. As a secondary contribution, we propose the new sparkboxtechnique for visualizing coarse and fine grained temporal structures within oneview. We provide preliminary evidence of the system’s utility based on multiplereal-world datasets and four usage scenarios.We present our contributions in this chapter as an application note to the visualiza-tion community. We describe the development of, and the design rationale behindour open-source visualization tool that supports the visual analysis of a specificdata type, relevant for many domains and use cases.Appendix B contains details about the datasets and a series of high-resolutionscreenshots of TimeElide.Online demo https://cs.ubc.ca/group/infovis/time-elideSupp. material https://osf.io/yqvmf/603.2 RelatedWorkVisualizations for temporal data have been well studied and applied to many ap-plications. Aigner et al. [4] characterized time-oriented data types and surveyedvisualization techniques, but did not discuss the combination of time intervals andsequences of time-value pairs explicitly.T_he bulk of the previous work applies to contiguous time series. We broadly clas-sify methods to interactively explore and select time periods of interest into threegroups: navigation and filtering, such as zooming & panning [66]; lensing to delimita period of interest to be put in focus [26, 126, 264]; and overview+detail linked viewsto drill down to smaller temporal regions while maintaining the high-level con-text [105, 253]. T_hese methods are typically limited to one time slice, or a very smallnumber of them. Some exceptions exist, such as ChronoLenses [304], TimeClassi-fier [276], CareCruiser [97], andMultiSegVA [178] which let users select segmentsfor exploratory visual analysis and comparison. Lee et al. [148] introduced a toolthat lets users select time slices of events that are visualized as node-link diagrams.However, all these methods are most suitable for a data-driven exploration; incontrast, our focus is on the investigation of known time periods.Although the term time slice appears in the visualization literature, it has been usedfor different purposes, such as to describe the segments in theT_hemeRiver [100]visualizations where time periods are connected with continuous curves, or tomean the interactive selection of one time period of interest [286].Considering the sparkbox technique in particular, some previous work focuses oninterval or event data that does not have finer-grained internal structure within theinterval [35, 124, 215]. Other approaches have been proposed for visualizingmultipletime series, ranging from handling unevenly-spaced samples [11] to navigating verylarge sets of time series [173]. In contrast, our goal with the sparkbox technique isto provide an adequate level of detail for the internal structure given amoderatenumber of time series. Previous methods leave out the details that our techniqueillustrates, as they are not aimed at our proposed data abstraction.61Szafir et al. [258] used composite charts, composed of line graphs on top of barcharts, in their evaluation of aggregations in time series visualization. T_he authorsaggregated data from continuous intervals while we assume non-contiguous, un-evenly spaced periods. T_hemost related previous work is box-plot-bars [16], whichwe directly build on.3.3 Non-Contiguous TimeSeries SlicesWe characterize data and tasks for non-contiguous time series slices.3.3.1 DataWe used four real-world datasets with different characteristics and cardinalities tovalidate the utility of TimeElide:• Soccer player in-match activity [208]: numbers of events (pass, shot, freekick, duel, ...) of one player, recorded during 22 matches and aggregatedinto 5-minute intervals.• Building occupancy [201]: estimated occupants recorded in multiple univer-sity buildings every 5 minutes, over more than a year.• Bakery sales [227]: number of items sold, logged over several months andaggregated into 15-minute bins.• Bike rides [242]: 20 bike rides recorded with Strava and converted into CSVformat by computing the median speed every 5 minutes.All four datasets are available in the online demo, and further details about themare provided in Sec. B.1.We distinguish between twomain types of non-contiguous time series slices:(1) existing: data is naturally divided into slices since data is not collected duringperiods of non-interest (bike rides, soccer matches),62(2) specified: data is continuously recorded but the analyst manually specifies timeperiods of interest, as illustrated in Fig. 3.2h (building occupancy).Time slices may vary in length but should be within the same order of magnitude,and TimeElide only handles slices less than or equal to 24 hours. A single time slicecan contain dozens to hundreds of data points. A dataset can contain dozens tohundreds of time slices. T_he gap between time slices can be fixed (every Monday 8-10am) or variable (occasional bike rides at different times). T_he variable of interest,such as occupancy, speed, or number of actions, must be quantitative.3.3.2 TasksWe identified five abstract tasks to support the visual analysis of non-contiguoustime series slices:T1 Extrema andAverageswithin and across slicesT2 Trends across all slicesT3 Patternswithin and across slices; outliers or unusual patternsT4 Comparison between pairs of slicesT5 Durations of slices and how they change over timeQuestions related to facilities management initially motivated and guided ourinvestigation. Example questions elicited in previous interviews [201] includeWhen do people usually leave on Friday evenings? Are they coming and going? and Does ithappen often that people occupy the space or are there irregular recurring events?From that domain-specific scenario, we abstracted four high-level tasksmost perti-nent to non-contiguous time series data,T1-T4. To ensure our approachgeneralizes,we then verified that these tasks alignwith questions usersmay askwhen analyzingthe other three datasets. We recognized the need to explore within-slice data (T3),to gain a better understanding of the gist of all slices, and to understand changesrelated to the temporal order (T5). We provide a detailed mapping from domainlanguage into abstract tasks in Sec. B.2.63Finally, we added Task T5 for analyzing and comparing the duration of slices, whichwas not relevant for the occupancy use case but is certainly applicable to otherexamples such as bike rides.64Figure 3.1: Visual encoding design space for non-contiguous time series slices. Supported, Partially supported, Not supported653.4 Visual EncodingDesign SpaceWe introduce a design space for visualizing non-contiguous time series slices. Itincludes our proposed visualization technique, sparkboxes, to support the analysisof within-slice and across-slice patterns simultaneously. Fig. 3.1 shows an overviewof the design space.3.4.1 TargetingDifferent Levels ofDetailT_he design space spans a set of eight visual encodings, that comprise differentlayouts and target different levels of detail (LoD), to support the outlined analysistasks.Non-contiguous slices are essentially individual time series where, in most cases,the order matters. A natural way to visualize these slices is to juxtapose individualline charts along the x-axis (Fig. 3.1b), which leads to a high LoD because the valuesare not aggregated. However, it may be difficult to discern trends (T2), especiallyin cases of high-frequency changes. T_he critical factor is the combination of thenumber of slices and the count of points within each slice.Box-plot-bars [16] are an alternative visual encoding that are a minimalist repre-sentation of Tukey box plots displaying the min-max range, interquartile range,and themedian (or average) value (Fig. 3.1c). Although box-plot-barswere originallyproposed to visualize contiguous high-frequency time series data, they can also beused to show summaries of time slices (T1, T2).Aggregating all values within a slice, such as in a stepped area chart or in heatstripes (Fig. 3.1d-e), facilitates overviews and helps users recognize trends andextrema. However, these compact representations may conceal important nuanceswithin slices.T_he 2D heatmap addresses the trade-off between LoD and scalability (Fig. 3.1f).T_he slices are again aligned along the x-axis but instead of aggregating all valuesinto a single number, each slice is divided into a desired number of vertical bins,66depending on the specified resolution. T_he per-bin aggregated values are visualizedas color-coded cells in the 2D space.T_he superimposedmulti-series line chart promotes local comparisons of within-slicepatterns [116], forusewhen the temporal orderof slices isnot relevant (Fig. 3.1g).However, distinguishing the identity of individual lines is challenging. T_he bandedmulti-series line chart (Fig. 3.1h) is an alternative to avoid the visual clutter in caseof many slices. T_he within-slice values are binned and then aggregated across allslices: min-max range, interquartile range, and the median value. T_he bandedmulti-series chart helps users to better understand typical patterns but prohibitsthe option to look up individual time slices. TimeElide does not support panningand zooming for these two unordered encodings, as it does for all others.Depending on the intended task, usersmaywant to visualize thewithin-slice timingsdifferently for time slices with varying lengths, such as bike rides. We distinguishbetween absolute duration and normalized duration, where the latter case meansthat all slice widths are equal. T_he y-axis in the 2D heatmap and the x-axis in bothmulti-series charts can also denote the absolute time of the day. Fig. 3.2f-g providesa comparison between absolute time and absolute duration for 2D heatmaps. Inthe soccer example, the absolute and normalized duration are equal because playtime is standardized to 90minutes.3.4.2 The Sparkbox TechniqueTo present the coarse and fine grained temporal structure of non-contiguous timeslices simultaneously, we propose sparkboxes—a combination of box-plot-barsand sliced line charts that are superimposed in the same space, as shown in Fig. 3.2candFig. 3.1a. T_hesliced line chart provides ahighLoDbut impedes seeing theoverallstructure at a glance. T_he box-plot-bars are more concise visually and capture thecoarse-grained structure. Sparkboxes combine the advantages of both, providing avery high LoD for dozens of slices.TimeElide supports zoomingandpanningbut sparkboxes canbealsousedas a staticrepresentation without any manual interaction, for example in a print medium.Different choices for the color palette can shift the focus to selectively emphasize67fine or coarse structures in the data in order to support different tasks, such asanalyzing general trends (T2) or within-slice patterns (T3). Our informal scalabilitytesting suggests a minimumwidth of 20 pixels per slice, at our target cardinalityof dozens of points per slice; at smaller sizes, we consider the 2D heatmap to besuperior.3.4.3 Design SpaceAnalysis RationaleHere, we provide further details about the design dimensions and supported tasks.• T_he LoD is very high for sparkboxes and only high for line charts. T_he aggregatedvalues in sparkboxes are considered as additional information and help tobetter grasp the gist and overall trends.• T_he LoD is high for line charts (sliced) and only medium for multi-series line charts.It is easier to compare within-slice patterns in a multi-series line chart but,due to superposition, the information about temporal order of the time slicesgets lost.• Tasks T1, T2, T3 are only partially supported by the bandedmulti-series line chart. Itis easy to recognize extrema and averages but it is impossible to link them toindividual time slices. T_he patterns are aggregated across slices and, there-fore, analyzing how patterns in specific slices change or deviate is impossible.Due to the across-slice aggregation, it is easy to recognize trends of dailypatterns but it is impossible to discern weekly or seasonal trends.• Task T4 (comparison) is only partially supported by the presented visual encodings.Although pairwise comparisons are possible with most visual encodings (Fig3 a-g), accurate comparisons are challenging when slices are juxtaposed, andin case of superposition, distinguishing the identity of individual lines andtheir temporal order is difficult.• Task T5 (duration) is only partially supported by the presented visual encodings. Al-though it is possible to identify slice lengths (durations) in most visual en-codings, an accurate analysis is challenging, and especially in case of manyslices would require an auxiliary view focused on this specific task.68• Line charts, stepped area charts, heat stripes, and multi-series line charts do not ex-hibit a within-slice layout. In our definition, visual encodings only have awithin-slice layout if slices are subdivided into layers or components. T_hewithin-slice data points are not aggregated for some of the encodings, suchas line charts, but have only a single layer or visual mark per slice.69Figure 3.2: TimeElide. (a) Select data source and slicing method, (b) select visualization type, (c) sparkbox visualization to show the numberof per-player events during 22 soccer matches, (d) tooltip for the currently active slice, and (e) global timeline. Alternative visual encodings:(f) 2D heatmap with absolute within-slice timing, and (g) 2D heatmap with normalized within-slice timing. (h) Illustration of extracting non-contiguous time series slices according to given periods of interest (manual slicing).703.5 The TimeElide ToolWe introduce TimeElide, an open-source visual analysis tool, that instantiates adesign space for visualizing non-contiguous time series slices. T_he accompany-ing video and online demo illustrate the look and feel of the interactive interface.Here, we first describe the tool itself, then cover the design space in depth in thesubsequent section.3.5.1 Overall DesignTimeElide incorporates a two-column layout with the sidebar on the left and themain view on the right, as shown in Fig. 3.2. T_he sidebar shows the three steps—select data source, select slicing method, and select visualization type—and allows usersto navigate between those flexibly. T_he data source page or the visualization pageare alternately displayed in the main view while the slicing method can be changeddirectly in the sidebar.T_he user uploads a CSV file or selects an example on the data source page. T_he toolautomatically detects existing slices or the user manually specifies custom slices.After the data source and slicing option have been chosen, TimeElide automaticallyvisualizes the data as sparkboxes or as a 2D heatmap, depending on the number ofslices and the available pixels in the main view.Sparkboxes reveal finer-grained structure within slices, by showing the raw datain the foreground and aggregated data in the background, while the 2D heatmapscales better for a larger number of slices. T_he user can switch between six primaryvisualization types in the sidebar, shown in Fig. 3.2b, and can configure the visualrepresentation using the settings menu above the visualization. Sparkboxes com-bine line charts and box-plot-bars, and therefore we group those three options andusers can show or hide layers.T_he global timeline (Fig. 3.2e) provides additional context by showing where eachslice falls along the continuous rangebetween start andenddate, revealingpotentialirregularity in the timing. Linked highlighting connects the timeline with themainview.71After a series of usability tests, we refined the default settings, added explanationsfor the available visualization types, and provided the option to customize the axistitle for the variable of interest.3.5.2 Manually Specify or AutomaticallyDetect SlicesManual slicing is intended to be used when the user is interested in clearly defined,recurring periods (e.g., everyMonday from3-5pm). Automatic detection of existingslices is appropriate when the user cannot pinpoint exact start and end times ofslices, such as bike rides that take place on different days of the week and vary inlength.Manually SpecifyDesired SlicesTimeElide provides an input form for users to specify slices manually, as shownin Fig. 3.3. Slices can be defined for different days of the week and time periods,but must be shorter than or equal to 24 hours.According to these slice specifications and thefirst and last timestamp in thedataset,we generate a set of empty slice arrays that correspond to actual dates. We theniterate through all data points and assign them to these slice arrays based on theirtimestamps. Some slice arrays may remain empty in case of missing data, which isindicated visually in the visualization.If users want to analyze slices related to specific dates, in contrast to days of theweek, they need to slice the data in a separate preprocessing step and then choosethe option to automatically detect existing slices in TimeElide, as we describe inthe following section.72Figure 3.3:Manually specify desired slices.AutomaticallyDetect Existing SlicesTimeElide can detect slices automatically to support the following two cases:• For some datasets, the slices can be inferred from the data rather than beingspecified by the user. For instance, workouts can occur at different times anddays of the week and vary in lengths. Defining manual slices for those casesis not effective.• Some datasets are generally continuous but users may want to analyze veryspecific slices that are not supported by TimeElide’s manual slicing interface,such as public holidays, and therefore, they remove data outside of theseperiods in a prior preprocessing step outside of TimeElide.We implemented a simple mechanism to detect slices based on the differencesbetween timestamps. When iterating through the data, the key question of interestis if a data point is still in the current slice or in a new slice. To determine anappropriate threshold and group the data points into slices, we use the followingprocess:1. Sort all timestamps.732. Iterate through all data points and determine the differences between con-secutive timestamps (in seconds) and save them in a new list.3. Remove the top and bottom 1% of this list to minimize the influence of out-liers.4. Compute the standard deviation of the filtered list and use it as a threshold.5. Iterate through the original dataset again and use the threshold to group thedata points.T_he threshold and the median within-slice and between-slice distances are dis-played in the left sidebar of the interface. T_he user can also adjust this thresholdmanually.Fig. 3.4 illustrates the differences between timestamps and the detected thresholdfor four different datasets that are naturally divided into slices (bike rides, soccergames) or where the data has been sliced in a preprocessing step (occupancy onFridays 6pm-midnight, opening hours of bakery).74Figure3.4:Automaticallydetectexistingslices. Eachdotcorresponds to thedifferencebetweentwo consecutive timestamps. The black dashed line indicates the detected threshold. The bot-tomand top 1%ofdifferences (greydots) are consideredoutliers andarenotused fordetermin-ing the threshold. If the differences between twodata points is smaller than the threshold, thedata points are in the same slice (blue), otherwise they are in separate slices (orange).3.5.3 ImplementationTimeElide is a serverless web application composed of nested reactive componentsbuilt using the Svelte2 framework. All visualization types are wrapped in reusablecomponents with minimal dependencies. Svelte compiles all components intoefficient imperative code to allow quick updates of the Document Object Modelwhich is particularly suitable for interactive visualizations with many elements.3.6 Usage ScenariosWepresent four exampleusage scenarios to illustrate theuser experience inTimeElide.T_he first three scenarios are based on example data that we acquired online, and2https://svelte.dev75the fourth exemplifies building occupancy questions raised by a previous projectstakeholder.3.6.1 Soccer Player In-MatchActivityA soccer player wants to understand how their performance changed throughoutthe season and during each of the 22matches. T_heplayer uploads the file containingall events and selects the option to automatically detect existing slices (Fig. 3.2a).TimeElide detects 22 matches of 90 minutes each and shows a sparkbox visualiza-tion (Fig. 3.2c). T_he activity varies within eachmatch, but falls into the range of 2-10actions across each 5-minute period for most games. T_hey hover over a sparkbox tosee more details in a popup (Fig. 3.2d). T_hey see they were only substituted in forthe last fewminutes in six of the matches, because the number of actions is zerofor most of the time. On the global timeline (Fig. 3.2e), below the sparkboxes, theynotice that they did not play at all in September, but note that the activity duringthe first match after their return looks normal.T_he soccer player changes the visualization type to 2D heatmap to analyze if thereare any differences between afternoon and evening games (Fig. 3.2f). T_he slices areagain juxtaposed along the x-axis. T_he y-scale is set to absolute time and shows therange between the earliest match start time (1:30pm) and latest end time (9:00pm).T_he number of in-match actions are aggregated to bins and shown as color-codedrectangles. T_hey adjust the number of bins to 30, so each bin lasts 15 minutes.T_hey quickly spot that their coach typically substituted them in at 2:30pm duringafternoonmatches.3.6.2 Bakery TransactionsA bakery manager wants to find out when the store is busiest and when extra helpis most needed. T_hey use the number of transactions as a proxy measure becausethis data has been already recorded over several months.T_he manager uploads the dataset to TimeElide and matches the columns of theCSV file with the two required variables timestamp and value, as shown in Fig. 3.5.76T_hey also rename the variable of interest “# Items sold” in the sidebar, as illustratedin Fig. 3.6-1 to create more descriptive axis titles.Figure 3.5: Assign columns of uploaded CSV file to timestamp and value variables.T_hemanager chooses to automatically detect slices to see each day of the week asa separate slice (Fig. 3.6-2). TimeElide detects 162 slices and visualizes them as a2D heatmap by default. T_he 2D heatmap confirms the manager’s prior assumptionthat most transactions occur between 9am and 5pm (Fig. 3.6-4).Figure 3.6: Annotated screenshot of TimeElide: Analyzing the number of transactions at a bak-ery at different days and times of the week using the 2D heatmap. (1) Rename title of variableof interest to “# Items sold”, (2) select automatic slicingmethod, (3) 2Dheatmapgets suggested,and (4) 2D heatmapwith absolute within-slice timing is displayed in themain view.77T_heynowwant to analyze the average number of transactions on differentweekdaysand if there are any trends or repeating patterns. T_hemanager selects stepped areachart (Fig. 3.7) and hovers over the peak bars to learn that they are all Saturdays.In addition, they observe a weekly pattern with more sales taking place betweenFriday and Sunday compared to the Monday toT_hursday period.Figure 3.7: Annotated screenshot of TimeElide: Analyzing the average number of transactionsat a bakery on different days of theweek using the stepped area chart.T_hemanager changes the slicing method to manually specify desired slices andselects only Fridays, Saturdays, and Sundays (Fig. 3.8-1). T_he goal is to do amorefine-grained analysis for those days to determine busy hours. T_he visualizationtype is changed to 2D heatmap again and the manager adjusts the number of binsto 30, so they can see the average number of transactions in 30-minute intervals(Fig. 3.8-3). T_hey can see that there are often spikes in sales between 10am and 2pm,and decide to hire additional staff for those periods.78Figure 3.8: Annotated screenshot of TimeElide: Analyzing the number of transactions at a bak-ery on Fridays, Saturdays, and Sundays using the 2D heatmap.3.6.3 BikeRidesA hobby athlete tracked their bike rides over the last few months and would liketo analyze the speed during all the workout logs in one place. T_hey concatenatethe downloaded datasets into one CSV file and upload it to TimeElide to get anoverview of all the data.T_he user selects the option for automatically detecting existing slices in the dataset.T_hey also select the 2D heatmap as a visualization type and change the y-scale modeto absolute duration and the number of bins to 15 (Fig. 3.9). At a glance the user cansee thatmost of the rides have been around one hour long, with no distinct upwardsor downwards trend across the months. It appears that the rides earlier in the year(closer to the left side) have been faster on average.To confirm, they choose sparkboxes in the left sidebar and adjust the settings: nor-malize slice widths to hold ride length constant, remove raw data layer and replacemedian with average speed (Fig. 3.10). T_he result is a visualization with box-plot-bars summarizing each bike ride. T_here looks to bemore variation in average speedin the latter months, but the long-term change seems to be negligible overall.79Figure3.9:Annotated screenshotofTimeElide: Analyzing the speedduringbike ridesusing the2D heatmap.Figure 3.10: Annotated screenshot of TimeElide: Analyzing the speed during bike rides usingbox-plot-bars.3.6.4 BuildingOccupancy onWeekendDaysA largeuniversity campus serves as a test bed for anew technology to recordbuildingoccupancy. A custodial analyst, who oversees facilities across the campus, wants toleverage the data to investigate claims that lecture rooms in a building are untidyafter weekends.80T_hey upload a CSV file to TimeElide that contains 5-minute samples of occupancyinformation, spanning a period of several months. T_he tool automatically suggeststhe file should be parsed as two columns with time and occupancy values, andthe analyst confirms the choice. T_hey choose to manually specify slices and selectSaturdays and Sundays as time periods of interest (Fig. 3.11a). TimeElide slices thecontinuous recordings by removing extraneous data outside of the given periods,and visualizes the result as sparkboxes. T_he analyst sees from the global timelinethat the dataset contains weekends between April and December (Fig. 3.11b). Afterthey use the settings menu to collapse slices without any data available, they thenchoose the visually concise box-plot-bars technique (Fig. 3.11c) to showonly aggregatedata. T_hey notice a few weekends with significant activity, and hover over the slicesto learn that most events occurred on Saturdays. In general, the occupancy rate isrelatively low which is expected for a building that is primarily used for lectures.T_hey do not recognize any repeating patterns.T_he analyst selects the visualization typemulti-series line chart, which disaggregatescompletely to show all time slices superimposed (Fig. 3.11d), to examine how theoccupancy typically changes throughout the day. T_hey learn that weekend activityrapidly decreases after 4-5pmandnoevening events takeplace. Sending a custodianfor a check-up in the late afternoon should suffice and the night shifts can remainunchanged.81Figure 3.11: (a)Manually specify desired slices. (b) Global timeline shows the regularity of sliceintervals. Building occupancy on weekend days visualized as (c) box-plot-bars and (d) multi-series line chart with absolute within-slice timing.3.7 Discussion and FutureWorkWe validate the effectiveness of TimeElide and the underlying design space withreal-worlddata fromfourdomains, illustrated through fourusage scenarios inSec. 3.6.Wepresent a descriptive design space as afirst step andhope it inspires refinementsand extensions in the future.Wemake a simplifying assumption in designing the TimeElide user interface byrestricting slices to amaximumof 24 hours. However, the presented design space isapplicable to large-scale slices beyond the 24 hour limit, for example, to analyze theGDP during recession periods over the last century. Our example data is sampled82at regular intervals such as every 5 minutes, but the design space and tool could beadapted to also support irregular intervals.TimeElide focuses on the detailed investigation of time slices belonging to one timeseries, such as occupancy in one building or region over time. Future work couldfurther investigate techniques and extensions of the interface for analyzing andcomparing multiple independent time series at the overview level (e.g., analyzemultiple buildings). Visual encodings with a lower LoD, such as stepped area chartsor heat stripes, highlight the coarse structure of slices in a way that scales to lowerresolutions, and can be used, for example, in a small multiple visualization withdozens of time series.3.8 SummaryIn this chapterwehavepresentedacharacterizationandabstractionofnon-contiguoustime series slices. Visualization systems tend to focus on the unbounded explo-ration of time series rather than the investigation of known time periods of interest.To better support this latter type of analysis, we introduce the open-source toolTimeElide, and a flexible design space including the sparkbox technique that pro-vides the conceptual foundation. All visualization designs were informed by adetailed analysis of data characteristics and alternative visual encodings. Fourscenarios from different domains illustrate how TimeElide facilitates an in-depthanalysis of non-contiguous time series slices.834 VizCommender:Content-BasedRecommendations forVisualization Collections1,2 Cloud-based visualization services have made visual analytics accessible to amuch wider audience than ever before. Systems such as Tableau have started toamass increasingly large collections of analytical knowledge in the form of inter-active visualization workbooks. When shared, these collections can form a visualanalytic knowledge base. However, as the size of a collection increases, so does thedifficulty in finding relevant information. Content-based recommendation (cbr)systems could help analysts in finding andmanaging workbooks relevant to theirinterests. For instance, when a user opens a specific visualization workbook, aCBR system could provide a list of additional workbooks that have been previouslycreated andmay be helpful to the user. Toward this goal, we focus on text-based1T_his chapter is a slightly modified version of our paper VizCommender: Computing Text-BasedSimilarity in Visualization Repositories for Content-Based Recommendations [204] byMichael Oppermann,Robert Kincaid, and Tamara Munzner in IEEE Transactions on Visualization and Computer Graphics(Proc. VAST), 27(2): 495-505, 2021. https://doi.org/10.1109/TVCG.2020.30303872In the VizCommender publication [204] we originally referred to large sets of bundles as visu-alization repositories. We now believe the term visualization collection is a more accurate descriptionand distinguishes them from dataset repositories that do not contain visual encodings but are in-creasingly used in the visualization community. Wewill use the term visualization collection through-out the dissertation to be consistent.84content that is representative of the subject matter of visualizations rather thanthe visual encodings and style. We discuss the challenges associated with creatinga CBR based on visualization specifications and explore more concretely how toimplement the relevancemeasures required using Tableauworkbook specificationsas the source of content data. We also demonstrate what information can be ex-tracted from these visualization specifications and how various natural languageprocessing techniques can be used to compute similarity between workbooks asone way to measure relevance. We report on a crowd-sourced user study to deter-mine if our similaritymeasuremimics human judgement. Finally, we choose latentDirichlet allocation (lda) as a specificmodel and instantiate it in a proof-of-conceptrecommender tool to demonstrate the basic function of our similarity measure.4.1 MotivationAs information visualization and visual analytics have matured, cloud-based visu-alization services have emerged. Tableau, Microsoft Power BI, Looker, and GoogleData Studio are a few such examples. In addition to enabling the sharing of indi-vidual one-off visualizations, users collaboratively build shared repositories thatserve as visual analytic knowledge bases. By providing large-scale community- andorganization-based collaboration services [273], massive collections of visual datarepresentations are created over time, referred to as visualization collections in theremainder of this chapter. T_heprimary artifacts in these collections are visualizationworkbooks (or reports) that bundle a set of visualizations or dashboards regarding aspecific topic or data source. Visualization collections exhibit the same issues of adhoc organization, inconsistent metadata, and scale that are common in the largercontext of the web itself and similar large-scale item collections such as streamingservices [305], web-based shopping portals [169], digital libraries [287], or datalakes that transform into data swamps [32, 99]. T_hus, cloud-based visualizationsystems are also encountering an increasing need for efficient discovery of relevantvisualization artifacts, much like these more well-known platforms.As with all such repositories, many possible approaches can be employed to findrelevant information. Indexed searching [281], faceted navigation [299], or ad-85vanced filter options are commonmethods. In addition to these active queryingmethods, recommender systems [2] are increasingly used to passively assist usersby surfacing relevant content, depending on the individual context.For example, an enterprise user selects a workbook to analyze the recent perfor-mance of one of their company websites. A recommender system can provide a listof relatedworkbooks that have beenpreviously created, such as about the long-termtrends or about the performance of a different website for comparison.A major paradigm for recommender systems is content-based filtering that incor-porates available content features into a domain-specificmodel rather than lookingat generic interactions between users and items, or ratings. Content-based recom-mendation (CBR) systems exist for a wide range of data types, such as songs [266],videos [72], artworks [179], and news articles [133]. For visualization collections,however, content-based recommendation has not been explicitly addressed. Con-tent features are highly subjective and imprecise in nature [301], and thus CBRsystems for new kinds of data require custom feature engineering building uponextensive domain expertise in order to makemeaningful comparisons and rank-ings.Visualization collections, and more specifically visualization workbooks, are basedon a set of static visualization specifications that contain information about the datasource and any data transformations, how the data is presented to the user througha visual encoding, and how users can interact with the visualization. Visualiza-tion specifications have an unusual combination of characteristics such as sparseand often fragmented text, hierarchical semi-structured content, both visual andsemantic features, and domain-specific expressions. Our goal is to extract suit-able content features to perform similarity comparisons between visualizationspecifications that can ultimately provide the basis of content-based recommenda-tions for visualization collections. T_his process is driven by two key questions: (1)Which content features aremost informative for comparing visualization specifica-tions? and (2)What techniques can we use for comparing and ranking visualizationspecifications?86In this chapter, we describe our work towards developing a text-based similaritymeasure suitable for content-based recommendations in visualization collections.We focus the notion of similarity on the subject matter of the visualizations ratherthan the specific visual encoding, with the assumption that similar topics are ofhigh relevance. Initial investigations indicated that chart types and visual stylesare of peripheral interest in the information seeking process.We examine the performance ofmultiple existing natural language processingmod-els within our proposed similarity model, and find that many of them yield robustresults that align with human judgements, despite the challenging characteristicsof the textual data that can be extracted from visualization workbooks.Our contributions are:• Identificationandcharacterizationofunique challenges indesigningcontent-based recommendations for visualization collections.• Designand implementationof theVizCommenderproof-of-conceptpipeline:Textual feature extraction, similarity measures for comparing and rankingvisualizations using natural language processing (NLP) techniques, and auser interface to demonstrate and provide insight into the measure’s utilityfor visualization recommendations.• Analysis of four applicable NLP similarity measures by examining both com-putational comparisons and a user study assessing alignment with humanjudgements of similarity.In our work, we align with the terminology in the recommender system litera-ture [2] where recommendations are proposed for preexisting items of content,namely previously designed visualizations stored in visualization collections asspecifications. In contrast, the term visualization recommendation is often used invisualization researchquite differently, to refer to recommending anew visual encod-ing for a given dataset [270], for examplewith Tableau’s ShowMe [164] orDraco [188].In this chapter, we do not focus on generating new visual encodings.874.2 BackgroundWe briefly discuss general approaches to recommender systems and elucidate ourindustry collaboration, followed by a description of our target users and a datacharacterization.4.2.1 RecommendationParadigmsMany different types of recommender systems have been proposed and are widelydisseminated formanydomains [2]. Wedifferentiate between the twomajor recom-mendation paradigms: collaborative filtering (cfr) and content-based filtering (CBR).CFR analyzes the interaction between users and items and derives recommenda-tions from patterns of similar user behavior or ratings. T_his approach requiresno domain knowledge, allows fast computation, and provides diverse and oftenserendipitous recommendations. However, CFR suffers from the so-called coldstart problemwhen new items or new users emerge and the system does not havesufficient information to make recommendations [144].CBR is the focus of this chapter and is based on finding relevant items based ontheir actual content. Since this method is based on inherent properties of the itemsit does not suffer from the cold start problem of collaborative filtering and canfrequently be more accurate depending on the recommendation needs. In contrasttoCFR, this approach also allows one to identifynear-duplicate itemswhich are highlyrelevant in some cases but need to be excluded in other cases [274]. Approachesto CBR systems have been widely investigated; the challenge is to selecting whatcontent features should be considered and how they are to be compared, which isknown to be highly domain-dependent [162].Ideally, both CFR and CBR approaches are combined into a hybrid system to al-leviate the issues of the other. CBR can make useful recommendations for newsystems while user behavior is monitored. Once sufficient information is collectedthat CFR becomes useful, then serendipity and diversity can be increased in therecommendations.884.2.2 Project ContextT_his work was conducted at Tableau Research. T_his environment provided uniqueaccess to multiple domain experts in visualization research, machine learning, andmost importantly customer insight into typical use cases and pain points relatedto managing large visualization collections and finding relevant information. Wehad over eight months of continuous engagement with product managers, userexperience designers, and multiple machine learning engineers in the TableauRecommender Systems Group. T_he overall approach, feature engineering, andalgorithm choices were informed, encouraged, and reviewed by these experts onan ongoing, iterative basis.Tableau has already deployed CFR systems to present users with relevant datasources in their data preparation workflow and to provide personalized viz rec-ommendations in a dedicated section on Tableau Online. T_he latter application ismost relevant for this work and motivated our investigation of CBR approachesbecause the standard user-item CFR model lacks precision and faces cold startissues. In particular, when viewing a specific viz workbook, the correspondingrecommendations should be related to the reference. Ideally, advantages of bothCFR and CBR are leveraged by combining them in a hybrid model. Although ahybrid system is beyond the scope of this study we hope to inform the developmentof future hybrid systems with our work.T_his collaboration allowedus to get access to twoTableau collectionswith thousandsof hand-crafted visualizations: (1) A sample of 29,521 workbooks (147,241 visualiza-tions) that are publicly shared on Tableau Public, a community platform with morethan twomillion workbooks; (2) An internal Tableau corporate collection with 3,661workbooks (54,715 visualizations). Since Tableau Public can be used by anyone forfree, that community-generated content is inherently messy and contains manymixed-language and incomplete workbooks, as described in Sec. 4.5. We knewfrom our collaborators that the sprawling Tableau Public dataset has very differentusage patterns than collections used in more typical enterprise settings, so we alsoused an internal Tableau company collection in our experimentation and whenobtaining expert feedback on versions of our proof-of-concept recommender. All89data used in the crowd-sourced experiment and the examples shown in this chapterare from the public dataset, for data security reasons.After excluding workbooks with no visual encodings and non-English content, weobtained one Tableau Public collection with 18,820 workbooks and one Tableaucorporate collection with 3,424 workbooks.4.2.3 ProcessWe followed an iterative process, shown in Fig. 4.1, that began by examining datafrom the provided visualization collections and by identifying and synthesizing usecases and recommendation challenges. In parallel to the investigation of appropri-ate models, we implemented a prototype user interface (Sec. 4.4) to analyze modeloutputs in aCBRcontext directly. T_hegoal of VizCommender usagewas to resemblethe browsing experience for other visualization platforms sufficiently to facilitatediscussions and dialogue with project stakeholders, comparable to a technologyprobe [112]. For example, we investigated the use of visual versus topic features,and early demo sessions revealed that visual encodings are not as important asanticipated, as we describe in further detail in Sec. 4.5.2. T_he first analysis phaseprovided sufficient direction and informed the design of the crowdsourced study tosystematically compare the alignment between human judgements and predictionsof various NLP models (see Sec. 4.7). T_he insights gained from informal modelexperiments, the study, and discussions with our collaborators were incorporatedinto the prototype which was continuously refined.90Figure4.1:Outlineofourprocess: (1) Extract andanalyze textual content fromviz specificationsbelonging to two visualization collections. Investigate appropriate content-based recommen-dation models with varying input features and implement initial prototypes to facilitate dis-cussions with collaborators; (2) Crowdsourced study: Sample viz triplets in a semi-automatedprocess, collect human judgements about the semantic text similarity, and run the sameexper-iment with different NLP models; (3) Compare agreement between human judgements andmodel predictions to assessmodel appropriateness. Implement LDA-based similaritymeasurein proof-of-concept pipeline.4.2.4 Users andTasksWe began our project by understanding target users and tasks associated witha recommender system for visualization collections. T_his process was informedby in-depth discussions with our collaborators, who have direct access to end-users and insight into their needs and pain points. Our collaborators in Tableau’sRecommender Systems Group also conducted a user study with a wizard-of-ozrecommendation tool to investigate user needs.T_he target users are explorers who look up their own visualizations or browse acollection for relevant work that has been created by other users from the sameorganization or community. Our investigation revealed that information seeking91or foraging is the core task when browsing visualization collections in enterprisesettings. Information seeking is a well-known visual analytics task [22, 55, 244]. Inthe context of CBR’s this task is highly focused on finding informative workbooksaround some specific analytical question in the user’s mind. For instance, if theanalyst inspects a specific workbook, additional relevant workbooks based on itstopic or data should be automatically provided in-place.Another task that is particularly focused on content features is version detection.T_his task supports quickly grouping of duplicate and near-duplicate items for usersto compare different versions of workbooks and to maintain the organization of acollection.Oneof themain insights of our requirements elicitationprocesswas that a single setof recommendations is not adequate to capture user intent and the different tasksthat should be facilitated by a recommender system: finding related workbooks,finding similar versions of a workbook, and finding workbooks that use similardata. Our three proposed recommendation facets are described in Sec. 4.4.We identified additional tasks that are not the focus of this study. Recommender-assisted authoring is an interesting future possibility where recommendationsare presented in real time while a user is authoring a new workbook. Such rec-ommendations could be used for design inspiration, or onemight even discoverthat a suitable workbook already exists in the collection. A more casual task istopic browsing. In this case a user may be simply browsing through the collectionwithout any specific goal in mind other than to look for interesting workbooksand visualizations. Success in this task requires a more lax sense of relevance inorder to surface serendipitous recommendations that venture further away fromworkbooks in the user’s profile.4.2.5 VisualizationWorkbooksWeuseTableauworkbook specifications as our source of visualization content, sincewehad ready access to large collections of this data type. Bundling visualizations anddata sources into workbooks or reports is a common practice between visualizationservices, such as Looker or Power BI, so our findings are generalizable beyond92Tableau. In the remainder of this dissertation, we will refer to Tableau workbooksas workbooks.Workbooks are stored as XML documents describing all necessary componentsthat are required for loading and displaying the fully-interactive visualizations tousers. Fig. 4.2 shows an example workbook and the corresponding XML excerpt. Aworkbook typically contains linkages to one or more tabular data sources that areeither static datasets, such as CSV data files, or live data streams. T_he viz specifica-tion includes the column names of all data attributes and any data transformationsthat have been applied, but not the raw data itself.Individual visualizations referred to as views are authored within containers calledsheets and can point to one ormore data sources. A sheet can alternatively representa dashboardwhich consists of references to individual sheets and additional layoutinformation to result in a collection of related and possibly coordinated views.A workbook could contain many hundred instances of views and dashboards; inpractice, the majority of the workbooks that we analyzed have less than a dozensheets, and about one third contain only a single view. Our collaborators, whohave insight into other collections, confirmed our perception that workbooks varygreatly in content and complexity.A complex set of XML tags are used to define the specifications for each of these ob-jects. At a high level, one can consider aworkbook specification to be structured verysimilarly to the typical XML-based spreadsheet which consists of multiple sheets,each with various data and style elements, and potentially with cross-referencesbetween sheets. T_he specification of a single view is comparable to the JSON-basedview specification in Vega-Lite [229].93Figure 4.2: Simplified example feature extraction from a Tableau workbook. To illustrate typi-cal features, a highly abbreviated example of theworkbook XML is shown in themiddle. High-lighted text color indicates the corresponding features that are converted into a numeric vectorrepresentation and used to compute text-based similarity. Similar types of text gets extractedfrom the remaining views and dashboards of theworkbook.4.3 RelatedWorkIn this sectionwe discuss relatedwork on visualization recommendation, semantictext similarity models, as well as evaluation methods.4.3.1 Visual EncodingRecommendationHistorically, the visualization community has frequently used the term recommen-dation in the context of recommending one or a set of alternative visual encodingsfor a specific combination of data types. Tableau ShowMe [164], Voyager [289],Draco [188], Data2Vis [78], and VizML [109] are examples of such recommendationengines, where the user inputs a dataset and the system recommends a visualencoding, such as a bar chart or geographic map. Tremendous progress has beenmade to handle increasingly complex combinations of data types, to incorporateperceptual effectiveness scores, and to leverage machine learning models [225].However, much remains to be done until automated visualization design reachesthe sophistication of manually crafted visualizations and dashboards, that includedynamic data streams, data transformations, multi-view interactions, and customannotations which collectively involve countless human design decisions [226]. Incontrast, the long-term goal of our effort is to develop a traditional recommendersystem to surface existing viz workbooks that are related to a reference workbook94selected by a user, that have been created previously and are stored in a collection;and not the recommendation of visual encodings for given datasets.4.3.2 Semantic Text SimilarityModels andEvaluationsSemantic text similarity tasks are common in the field of natural language pro-cessing to improve algorithms for applications such as machine translation, sum-marization, or question answering [3, 207]. However, these tasks are often basedon curated datasets, such as sentence pairs, that do not reflect the special charac-teristics of the text that is embedded in viz workbooks and results might not begenerally applicable.Previous work compared human judgments with model predictions for the task ofsemantic text similarity. Towne et al. [265] compared human judgements with LDApredictions based on multi-paragraph documents. Colucci et al. [63] comparedhuman judgements with TF-IDF in the context of movie recommendations. Incontrast, we compare human judgements with a broader range ofmodels includingword embeddings.4.3.3 Evaluation of RecommendationAlgorithmsMany approaches have been proposed to evaluate recommendation algorithms,both in general and for CBR systems specifically, which in turn dictate the modelselection and refinement process [241].A/B testing allows us to compare algorithms in the real world and is usually themost desired method but is costly and often not feasible, particularly if there aremultiple algorithms to test. Instead, offline metrics are frequently used to evaluatealgorithms [151]. Utilitymetrics, such as coverage, scalability, or computational per-formance, can be evaluated independently while accuracy metrics require groundtruth data that is not always available. Moreover, several studies [129, 135, 174]demonstrated that these standard metrics that are used to evaluate recommendersystems are often not adequate to reflect user expectations and amore user-centricevaluation is necessary.95User studies either assess the users’ satisfaction with recommendations in thecontext of a specific interface directly [80] or exclusively evaluate the underlyingalgorithm. Ourwork focuseson the latter andspecifically about the similaritymodelas the main building block towards content-based visualization recommendations.Several studies investigated how human similarity perception aligns with the algo-rithmic notion of similarity for specific domains [149, 152, 262]. Winecoff et al. [285]collected relative similarity judgements to compare the Jaccard similarity indexwith a newmetric that accounts for psychological factors. In a similar spirit, wealso conducted a crowdsourced two-alternative forced choice study but our taskfocused on the semantic similarity between semi-structured text snippets insteadof the visual appearance of fashion items.More closely related to our approach is the work by Yao &Harper [298] who evalu-ated several CFR and CBRmethods for movie recommendations with similarityratings by users. T_hemain finding was that CBR algorithms outperform CFRmeth-ods. T_he authors asked participants to rate the similarity between pairs of familiarmovies on an absolute scale while we chose to collect relative similarity judgementsthat are more robust and reliable [74, 152]. In addition, they showed the title andmovie poster of familiar movies while our study interface shows all the text toparticipants that is also used for model predictions.4.4 VizCommender Proof-Of-ConceptWe now describe our proof-of-concept pipeline, VizCommender, that we createdto investigate different similarity measures and to get stakeholder feedback aboutvisualization recommendations. We use a collection with 4,698 Tableau Publicworkbooks as input data and extract text elements from the viz specifications thatare informative about the underlying topic. We convert these bag-of-words intonumeric vector representations and compute the similarity between them usingNLP techniques in order to generate workbook recommendations. T_he final versionof the underlying model is informed by the results of our crowdsourced user study:we decided to instantiate a similarity model based on LDA and Jensen-Shannondivergence, as described in Sec. 4.8.3.96We preprocess these similarities and when users browse in the prototype front-end, content-based recommendations are displayed, as described in the followingsection.4.4.1 User InterfaceInitially, the VizCommender interface displays an overview of many workbooks ar-ranged in a scrollable grid view, as shown in Fig. 4.3. Eachworkbook preview consistsof a thumbnail, title, author name, and date. If a workbook contains a dashboard,the screenshot of a dashboard is shown as a thumbnail to make it immediatelyapparent. An auto-suggest search box enables users to findworkbooks by keywordsor author names. We use TF-IDF to extract keywords from viz specifications anddisplay the most important ones across all loaded workbooks as tags at the top ofthe page. By clicking on a tag, users can further drill-down.Clicking a workbook preview opens the quick view sidebar with further details aboutthat workbook, including custom recommendations.Figure 4.3: VizCommender interface that allows users to browse through a visualization collec-tions. Workbook thumbnails are arranged in a grid view. Users can search for content or furtherdrill down by selecting one of the tags at the top. The quick view sidebar on the right providesfurther details including recommendations when aworkbook is selected.97Users can open the interactive detail view from the sidebar or through a double clickon the workbook preview. T_he detail view, shown in Fig. 4.4, contains a panel at thebottom of the screen that can be expanded to reveal workbook recommendationssimilar to the quick view. A tab navigation allows users to explore recommendationsbased on three similarity facets: (1) related workbooks, (2) similar versions, and (3)workbooks with similar data. We provide further details about these similarityfacets in Sec. 4.8.3. T_he goal is to provide groups where users can choose frominstead of creating a single all-purpose list. Additional recommendations can beloaded by horizontally scrolling within the panel.Figure 4.4: Interface detail viewwith recommendations. (a) Interactive Tableau workbook; (b)Expanded recommendation panel at the bottom of the screen showing related workbooks; (c)Tab navigation to switch betweendifferent recommendation types; (d) Alternative recommen-dation panel showingworkbooks that use similar data.4.4.2 ExampleRecommendation&UsageAn example workbook about Olympic medals is selected in Fig. 4.4a. T_he recom-mendation panel in Fig. 4.4b shows two related workbooks that are sufficientlydifferent but similar in topic. T_he first four recommendations from the similardata facet are shown in Fig. 4.4d. One workbook is included in both facets whichillustrates that the models lead to related results although one is trained on alltextual content and the other one is only trained on data column names.98We used VizCommender regularly ourselves and with our collaborators to betterunderstand and debugmodel results. For example, we implemented recommenda-tions for individual sheets and workbooks and only after several chauffeured demosessions we concluded that workbooks are the primary object of interest.VizCommender also helped us to tune threshold values for the similarity facets(see Sec. 4.8.3). We identified imprecise recommendations that have no topic-relationship to the reference which can not be avoided because of the nature of textdata and the relatively small visualization collections but mitigated through higherthresholds.4.5 Data Extraction and Feature EngineeringWe now describe the characteristics of visualization collections and the input fea-tures that we extract and use in our approach, and describe the intrinsic challengesof the data in a CBR context.4.5.1 Visualization Text ExtractionA viz workbook specification contains a number of text objects: workbook name,sheet names, titles, axes labels, captions, annotations, and data source columnnames. All of these objects can contain human readable text that may, but doesnot always, capture meaningful semantic information about the topic of the visu-alization. We use the collection of words and phrases to construct a document inorder to apply NLP techniques to compute similarities and classifications betweendocuments. After preprocessing, we store the data in a PostgreSQL database toease querying and further analyses.For each workbook, we extract and preprocess the text from the XML specification.We remove numbers, punctuation, strings with less than three characters. Weapply a custom stop-word filter to eliminate text such as "Number of Records" thatis auto-generated by Tableau for all data sources. We created a short list of 354generic stop-words by analyzing frequently but non-informative words or phrasesthat occur in Tableau collections. More extensive corpus-specific filters are beyond99the scope of this work; previous work [231] indicates that stop-word removal mightoffer only superficial improvement. We also annotate the text with part-of-speechtagging and lemmatize it usingNLTKWordNet [27].4.5.2 LeavingOutVisual EncodingsIn this work, we chose to focus on basing recommendations on the subject matterof a visualization or workbook and not the visual style. T_hereby, we ignore all visualencoding specifications such asmark types, colors, reference lines, and layout prop-erties. T_his decision was informed by in-depth discussions with our collaboratorsand the prioritization of information seeking. T_his task is highly focused on findinginformative workbooks around a user’s specific analytical questions. For example,while looking at a 2019 report, a CBR systemmight also recommend the related2018 report for comparison and it is not relevant whether the workbook contains ascatter plot or a bar chart.We also verified this expert guidance informally through our prototype. An earlyversion used an adjustable weighted combination of visual and topic features, butwe found that directly using visual encoding features added noise to the modelwhen the task is information seeking in visualization collections. We saw thatsome workbooks are visually perceived similar although they use different dataand address distinct topics, and conversely identical informationmay be visualizedfundamentally differently.Fig. 4.5 shows simplified examples to illustrate some challenges with similarityperception. T_he pie charts in Fig. 4.5a are perceived visually similar although theyuse different data and address distinct topics. In contrast, identical data may bevisualized fundamentally differently, as shown in Fig. 4.5b.100Figure 4.5: Simple examples illustrating that visual similarity is typically topic independent.Although visual styles are not directly incorporated, we do extract chart-visible textelements, such as axis titles or annotations, that may reveal information about theunderlying topic.A few very large and public-facing visualization collections, such as Tableau Publicor Power BI’s Data Stories Gallery, are unusual cases that are focused on entirelydifferent tasks than is typical in an enterprise setting. One such task is search-ing for design inspiration, which would indeed require making visual encodingspecifications first class citizens. However, that task is beyond the scope of thisstudy.4.5.3 LeavingOutUnderlyingDataAll viz specifications inherently also include references to the underlying data beingvisualized, and this data includes additional semantic information that could beincorporated. However, early in the project, we made the fundamental decision tonot access a workbook’s underlying raw data. In enterprise settings, viz workbooks101are often not just built upon simple CSV files and instead are powered by large scaledata warehouses or live data streams. Including all such data content for recom-mendation purposes would quickly lead to computational scalability challenges.Given this concern, we decided the computational cost required to process raw datafor each workbook to be prohibitive for large scale visualization collections, andfor enhanced scalability chose to focus on the more accessible data within the vizspecification itself. Incorporating samples of the raw data offers an interestingavenue for future work that can be informed by our results.4.5.4 Data ChallengesBased on the two previously mentioned sample collections, we identified five chal-lenges in leveraging text content from viz workbooks:C1: Workbooks andNested Visualizations. For many business intelligence applicationssuch as Tableau, the dashboard is the primary visual artifact being created. A dash-board is typically made of multiple related visualizations, that may share morethan one data source. Additionally, it is common for any single visualization tocontain faceted nested visualizations in order to create trellis views or small multi-ples. T_he result is a potentially complex hierarchy of visualization specifications.A given workbook can contain an arbitrary number of such complex dashboards.T_his structural complexity and, in some cases, a mix of topics within a workbook,complicates meaningful comparisons.C2: Very Limited Text.T_he text found in most sheets generally consists of fragmentsof a fewwords, such as titles or labels. Less frequently, a visualization authormightinclude captions or annotations of lengthier text, but not usually something ap-proaching the length of an actual text document. See Fig. 4.6 for a distributionof unique words per workbook. T_his characteristic is somewhat similar to theproblem of classifying tweets [250], but is even more extreme: the title and labeltext may or may not be strictly related, and certainly do not have the same naturallanguage relationships that a typical grammatical sentence would have. An addi-tional problemwith titles and labels is that Tableau provides auto-generated text102Figure 4.6: Number of unique words per workbook with and without data column names. In-corporating column names significantly increases the document size,most notable for the cor-porate visualization collection, and can be a strong signal when comparing the topic similaritybetweenworkbooks.by default which simply re-iterates the field names being visualized and providesno additional semantic information.C3: IncompleteWorkbooks. In both visualization collections, we found a substantialnumber of low quality workbooks that were incomplete, work-in-progress drafts,or even completely empty.C4: Multiple Versions. In both collections we found numerous examples of exact-duplicate or near-identical workbooks. T_his challenge was particularly strikingon Tableau Public which is often the system used for teaching Tableau to studentsand data scientists. T_he result is that there are many near-duplicate workbooksrelated to a particular course curriculum, or on-line design challenge as for exampleMakeoverMonday [139]. A similar problem exists with periodic reports such ascorporate finance reports that may occur quarterly, or annually. Choosing themostrelevant from these sets of nearly identical workbooks is a major challenge andstrongly linked to the individual context. Someone working on a 2020 report mightwant to see a reference to the 2019 report, but someone working on a class projectmay not want to see many other very similar examples of the same use case.103C5: Out-of-Vocabulary Words. Most visualizations have a domain-specific naturereflected in their data and text. T_his domain specificity can make the use of pre-trained NLPmodels less effective since the text is often expressed in proprietarycompany-internal vocabulary or a very domain-specific language. Although collec-tions frequently contain thousands of workbooks, the corpus size is too small totrain customword embeddings that accurately capture word representations [182].Further, column names are often abbreviated or totally cryptic to any user otherthan the database administrator, and such naming enigmas carry over to defaultlabels generated from such field names.4.5.5 Feature Selection andData FilteringIn Tableau collections, a workbook is the primary artifact that combines visualiza-tions targeting an analytical question or to communicate data around a specific sub-ject. Broadly comparable to Excel files, all sheets within a workbook typically focuson the same topic and individual views are building blocks for dashboards. In thiswork we propose only workbook to workbook comparisons, for an achievable scopethat eliminates the need to decompose nested visualizations and cross-referencesto individual views (C1). We found only a small number of workbooks that containa mixture of distinct topics and speculate that these cases primarily occur whenusers learn Tableau and experiment with features and different data sources.Analyzing an entire workbook allowsmore text to be extracted than from a singlesheet and thus helps alleviate someof the issues that arise from feature extraction ofa single visualization (C2, C5). We also use the column names from data sources asadditional text data. Fig. 4.6 shows how the number of unique words per workbookincreases when column names are incorporated. T_his choice includes data thatmayor may not be used in any visualizations, so that the semantic content of the datasource itself canhelp informus about the topic it describes. While this approach canfrequently result in repeated text fragments, it is analogous to human-authored textdocuments with repeated words and phrases. Standard methods for normalizingword frequency can be applied to rigorously and meaningfully account for suchrepetitions.104For recommendationpurposes,we excludeworkbookswithno visualmarks (C3) butwe include mostly-complete workbooks that contain a few partial sheets becausethose could be remnants from a data analysis process that have not been explicitlyremoved. While it would be highly desirable to filter out misleading and deceptivevisualizations from recommendations, automatically assessing the quality of visu-alizations remains one of the major challenges in our research field [127, 175] and isnot the focus of this work.Many of the NLPmodels we use generate poor or unreliable comparisons for verysmall number of features (that is, a too-small bag of words) in one or bothworkbooksbeing compared. Hence, after feature extraction and stop-word filtering, anyworkbook with less than 10 relevant words is also removed from consideration (C3).4.6 Viz-to-Viz SimilarityWe propose to use textual similarity for creating visualization recommendationsand briefly summarize four NLPmodels that we considered.4.6.1 Pairwise ComparisonsIn order to create a ranked list of recommended visualizations or workbooks, weneed somemechanism tomake comparative judgements. In this work, we focuson the use case where a user looks at a reference workbook and wants to discoverother workbooks that lead to additional insights. T_he basis for this computationalmodel can be distilled to comparisons between pairs of items. As we will discussin Sec. 4.9.3, more advancedmodels can build upon this similarity measure andincorporate multiple workbooks and user profiles.We chose to pursue an approach that uses textual features extracted from pairs ofvisualization workbook specifications and computes the similarity between the twosets of features. In line with other CBR systems, our assumption is that similarityis a surrogate for relevance and a crucial building block for a recommender system,even though for a full-fledged instantiationmany other factors such as diversity,serendipity, novelty, and trust must also be taken into account [90].1054.6.2 NLPModelsWe now review a set of methods to compute pairwise similarity scores based on tex-tual features. A broad range of NLP techniques exist that have been widely adoptedfor many different applications in a variety of domains that are also applicable toCBR contexts.We conducted a literature review to assess the suitability based on the special com-bination of data characteristics in visualization collections and narrowed downthe selection to four general models, each composed of a document representationmethod and a distance measure. T_hemodels we use in our experiments are tf-idf,lsi, LDA, and document embeddings Doc2Vec and GloVe, summarized below. Wechose TF-IDF as a simple but powerful baseline model. Our collaborators wereparticularly interested in LSI as they were considering using it in a hybrid rec-ommender system. LDA is popular for topic modeling but also commonly usedto assess text similarity. Recent document embedding approaches surpass thesetraditional methods for many nlp tasks, so we also tested them.In addition to these widely used models, which we chose as an appropriate firststep, many potentially useful alternatives exist that could be explored as futurework, particularly word/document embedding techniques [30, 77, 213] that havereceived significant attention recently. ELMo [213] and BERT [77], for example,can better capture out-of-vocabulary words because embeddings are learned forn-grams but their primary strengths of incorporating the surrounding context ofwords is more applicable to actual text documents with sentence structures thanfor viz specifications.We use cosine similarity, one of the most commonly used distance measures invector space, for all models except LDA. Following previous work [23], we decidedto use Jensen-Shannon divergence [155] for LDA because it is more suitable forprobability distributions.All NLPmodels were implemented in Python based on the powerful and populargensim software package [220].106To determine which of those selected models is a good fit for our problem weconducted an informal investigation of all models followed by a crowdsourced userstudy to systematically assess the alignment betweenmodel predictions and humanjudgements (see Sec. 4.7).Model: TermFrequency–Inverse Document Frequency (TF-IDF)Metric: Cosine SimilarityTF-IDF [222] is an extension of a simple bag-of-words approach that considers aterm’s frequency (TF) and its inverse document frequency (IDF) to promote termsthat are unique across the corpus and to weed out common language. In essence,documents are vectorized by the relative importance of each word and TF-IDFfunctions as a more advanced stop-word filter. After computing a numerical TF-IDF-weighted score for each term, the result is a sparse two-dimensional document-termmatrix. We use cosine similarity to compute the similarity between vectors inthehighdimensional space; other suitable similaritymeasures [1] could also beused.TF-IDF is a simple and efficient algorithmbut does not captureword semantics andonline learning of new documents requires revisiting all the previous documents.Model: Latent Semantic Indexing (LSI)Metric: Cosine SimilarityLSI [71], also referred to as latent semantic analysis (LSA), uses truncated singularvalue decomposition to analyze relationships between terms. T_he assumption isthat words that appear together in a document have a similar meaning. An LSImodel, trained on a TF-IDF weighted term-documentmatrix, produces low dimen-sional document representations that can be compared using cosine similarity. T_henumber of latent dimensions is the most important hyper-parameter and can bedifficult to determine. We test 15, 30, 75, and 150 dimensions.Model: Latent Dirichlet Allocation (LDA)Metric: Jensen-ShannonDivergence (JSD)T_he idea behind LDA [28] is that each document contains a mixture of latent topicsthat are found throughout the corpus and each topic is composed of a set of words.LDA is a probabilistic model that is first trained on bag-of-words of documents and107then computes a probability distribution over k topics for each document. We usethe inverse of JSD to compare the similarity between topic distributions. LDA is themost popular technique for topic modeling but also frequently used for semanticsimilarity tasks. As with LSI, the number of topics must be predefined and anoptimal configuration is difficult to determine. We probe the model with 15, 30, 75,and 150 topics.Model:Document EmbeddingsMetric: Cosine SimilarityWe distinguish between three variations,Doc2Vec, GloVe Pre-Trained (GloVe-Pre),and GloVe Transfer Learning (GloVe-TF).Doc2Vec [147] generalizes theWord2Vec word embeddingmethod to documents.Word2Vec is a neural network that produces numeric representations for eachwordby incorporating the word context in order to capture extensive semantic relation-ships, such as synonyms, antonyms, or analogies. Doc2Vec extends this model tosentences or whole documents to create fixed-length document representations.T_hemain challenge is to train Doc2Vec on a sufficiently large corpus, as is the casewith most machine learningmethods. We use theDistributed BagOfWords (DBOW)approach on a large set of viz specifications and test the model with 100 and 300dimensional feature vectors.GloVe [211] is an alternative to Word2Vec to train word embeddings that uses aco-occurrence matrix of the corpus. Several pre-trainedmodels have beenmadeavailable. ForGloVe-Pre, we use Stanford’smodel [211] trained on 400K vocabulariesfromWikipedia and Gigaword (vectors with 100 and 300 dimensions). In order toget one vector per document, we compute the averageword embedding. T_hismodelcould be further tuned by using more advanced techniques to combine individualword embeddings such asWordMover’s Embedding [295].While many pretrained models based on news or Wikipedia articles have beenreleased, they are often not adequate for domain-specific documents and lackimportant vocabulary. For GloVe-TF, we use the pre-trained GloVe-Pre model andfurther train it on the viz specifications from our corpus, similar to Doc2Vec.108In case of Doc2Vec, we generate five permutations with different word orderings toimprove the performance. T_he strength of these embeddingmodels to learn seman-tics through surrounding words is significantly impacted because the extractedtext from viz specifications resembles an unordered list of tags instead of actualtext documents (C2). See Fig. 4.6 for a distribution of words per workbook.We also used our prototype, VizCommender, to analyze the model output in a CBRcontext directly. T_his subjective inspectionallowedus to see if the recommendationsmake sense for domain experts.4.7 Crowdsourced Similarity JudgementsWe now describe the user study in which we collected human judgements froma semantic text similarity task, in order to use the resulting data to inform theselection of appropriate NLPmodels.4.7.1 MethodTo assess the alignment between human judgements and our algorithmic notionof similarity, we conducted a crowdsourced two-alternative force choice (2AFC)experiment using the Mechanical Turk (MTurk) platform.HumanSimilarity JudgementsParticipants were presented sets of three documents (triplets) and asked to selectone of two alternatives that is most similar to a reference document, as shownin Fig. 4.7. Several studies [74, 152] have shown that triplet matching and rela-tive similarity judgements are more reliable and robust across participants thanpairwise judgements where subjects are asked to rate the perceived similarity ona numerical scale. An early pilot study using pairwise ratings confirmed theselimitations by showing that participants interpret the absolute scale differently, anambiguity that does not exist in relative similarity judgements.109Experimental StimulusAs described earlier, our objective is semantic text similarity and not the visual styleof visualizations. T_herefore, participants were only shown text extracted from vizspecifications with no indication of the layout or chart type (see Fig. 4.7). To makejudgements less challenging and allow crowdsourced participation, we reduced thescale of each document by including only content fromone visualization (sheet) andthe workbook title. Using only these features yields identical results to single-sheetworkbooks, which constitute roughly one third of all workbooks across the corporathat we analyzed.Model Similarity JudgementsAnalogous to collecting human judgements, we ran the same experiment with NLPmodels (TF-IDF, LSI, LDA, Doc2Vec, and GloVe) to determine if the algorithmicnotion of similarity mimics human judgement and if there are significant differ-ences betweenmodels. To imitate the judgement task, models produce predictionsby independently computing the pairwise similarity scores between the referencedocument and one of the alternatives. T_hemodel compares the two scores and thealternative with the higher score is deemed to be more similar.110Figure 4.7: Annotated user interface for the semantic text similarity task. Participants selectone of two alternatives that ismost similar to a reference.4.7.2 DataWe sampled data in a multi-stage process from the corpus of 18,820 Tableau Publicworkbooks.Visualization selection: We randomly selected 4,698 workbooks that were viewedat least twice on public.tableau.com and recognized to use English language. Weextracted textual features from 19,575 visualizations within them and filtered outvisualizations with no visual marks. We also discarded sheets that contain less than10 or more than 200 words, including the workbook title, which resulted in a totalnumber of 15,482 visualizations.Triplet pool generation: We sampled triplets based on TF-IDF and LDA.We usedTF-IDF as a baseline model to calculate pairwise similarity scores (0-1) betweenthe 15,482 visualizations before we randomly assembled the document triplets,under specific conditions: First, we considered only scores between 0.15 and 0.9 toexclude highly dissimilar and nearly identical visualizations. Second, we set the111delta between the two reference-alternative scores within a triplet to aminimum of0.45 to ensure that alternatives are sufficiently different on a syntactic level. Askingparticipants to rate identical or almost identical alternatives would be less insight-ful. T_hird, to avoid a scenario of too many examples revolving around the sametopic, such as customer sales or basketball statistics, we used stratified samplingbased on LDA topics and added each sheet to at most two triplets. All thresholdswere informed through pilot testing and by gradually examining combinations ofsimilarity scores. T_his stage led to a pool of 4,211 triplets.Triplet selection: Based on the available triplets, we randomly sampled 135 can-didates andmanually inspected each for mixed-language content and extensivedomain-specific vocabulary, that could not be filtered automatically. A new tripletwas sampled for each rejection. In total, we examined 169 triplets to get the 135retained.Model training data: All models are trained on 96,439 viz sheet specifications fromthe Tableau Public collection that are not included in the 4,211 sampled triplets.4.7.3 ProcedureWe recruited participants from Amazon’s MTurk. Using an online crowd of work-ers to complete human intelligent tasks (HIT) is a common practice in visualiza-tion [102], and has been used in the past to collect similarity judgements [149, 285].Eligible workers with an approval rate greater than 98% andmore than 5000 ap-provedHITswere redirected to our customwebpage. T_hepageprovided the consentform, a screening task, and detailed instructions, specifically pointing out that deci-sions should be based on semantics and the gist of a document instead of individualword overlaps. T_he emphasis is on correctness instead of speed. T_he screeningtest involved the rating of a triplet where the semantic relatedness between thereference and one of the alternatives is immediately obvious.We divided the 135 triplets into three equal batches of 45 triplets that were eachrated by 25 participants. T_he triplets were shown in random order and the positionof the alternatives was randomized. T_he second and tenth triplet were repeated112at the end again, to assess the intra-rater agreement, which led to 47 requiredjudgements. After finishing all ratings and the post-questionnaire, participantswere provided a unique code to enter onMTurk. We compensated workers witha flat rate honorarium of US$ 4.00 on completion of the study. T_his amount wasdefined based on internal pre-tests indicating that a session would require roughly20-30minutes. A participant who completed the task or was deemed ineligible wasprevented from attempting the task a second time.4.7.4 LimitationsT_he goal of this study was to better understand the appropriateness of various NLPmodels for similarity comparisons. Participants were only shown text extractedfrom viz specifications that is also used for model predictions. Showing visualiza-tions insteadmay have led to different results but our focus is on the subject matterand therefore we excluded the visual encodings to reduce factors influencing thejudgements.We see this semantic text similarity study as a necessary step towards creatinga content-based recommender system and not as a replacement of a summativeevaluation of recommendations. Many other factors play an important role whenevaluating recommendations in an application, such as user interface design, trust,and familiarity, which would be interesting to examine but these aspects were notthe immediate objective of this work.4.8 ResultsWedescribe the results of our experiment anddiscuss how they inform the selectionof an appropriate model for the similarity measure.We implemented an interactive visual analysis tool to both detect suspicious crowd-sourceddata and to better understand the results and the implications for themodelselection, similar in spirit although different in details to Kachkaev et al. [120].Screenshots of this tool are included in Sec. C.3.1134.8.1 Human JudgementA total of 89 participants accepted the MTurk HIT. Of those, 75 (n=75) were deemedeligible and completed the experiment; 5 workers responded incorrectly to thescreening task at the beginning and 8 workers did not complete all required steps.Before the release of the study, we divided the set of 135 triplets randomly into threebatches of 45 triplets each. Once 25 workers completed a batch, we took it offline.We also removed one participant from the analysis who responded implausibly fastand thus reran the batch with a new worker.One of the researchers in our team labeled all 135 triplets beforehand to create agold standard to be used to filter out obviously poorly performing participants. Ouranalysis tool shows the responses of each participant displayed along a timelineand color-coded based on the agreement with the gold standard. We also tracked ifparticipants primarily pick alternatives on the left or right side, indicating a lack ofattention to the intended task, but no participant was excluded from the analysisbased on these twomechanisms.Of the 75 participants, 57.3% were male, 41.3% were female, and 1.3% non-binary.Further details about the demographics, such as age distribution and educationalbackground, are included in Sec. C.2.2.Weuse Fleiss’ Kappaκ [87] that quantifies the agreement between afixednumber oftwo ormore raters in order to comparemodel predictions with human judgements.Fleiss’ Kappa can range from -1 (no agreement) to +1 (perfect agreement) and iscomparable to Cohen’s inter-rater agreement [60] for the case of two raters. T_heinter-rater agreement between all participants (mean Kappa values of all threebatches) is κ = .544, a moderate agreement according to the commonly usedinterpretation of Landis and Koch [146].We duplicated 2 triplets to assess the intra-rater agreement. 56 participants (74.7%)have a perfect agreement, 17 (22.7%) rated 1 triplet differently and 2 reversed theirjudgements for both repetitions. T_he relatively large group of people who changedtheir opinion could be an indicator that, for these triplets, both alternatives arerelated to the reference according to human judgement despite the clear preference114in the baseline TF-IDFmodel. 37 triplets (27.4%) were judged unanimously by allparticipants. One triplet has an equal number of votes for both alternatives. T_here isalso a strong agreement between the participants and the gold standard, althoughthe majority of participants disagreed in 4 cases.In light of this analysis, we classify triplets into two groups based on human consen-sus: 92 high consensus triplets with at least 80% agreement between all participants,and 43 low consensus triplets with less than 80% agreement or disagreement betweenthe gold standard and the majority vote. A detailed examination of low consensustriplets confirmed that these involved non-trivial alternative choices.4.8.2 Model ComparisonWe now compare the models with the human consensus using the majority rule(alternative chosen by themajority of the participants), with results summarized inFig. 4.8. Detailed results for varying model parameters are included in Sec. C.2.3.T_he LDAmodel aligns best with human judgements for all triplets, regardless oflow or high human consensus. Fig. 4.8 shows agreement scores for the LDAmodelwith 150 topics but lower number of topics (30 and 75) achieved comparable results.GloVe-Pre, the pre-trained word embeddingmodel, and GloVe-TF, the extensionthat is further trained on extracted text from viz specifications, made identical pre-dictions. We conjecture that the relatively small number of 96,439 viz specificationsused for online learning is insufficient for GloVe-TF to achieve an additional gain.LSI performed worst, regardless of the number of dimensions.Model predictions align almost perfectly with judgements for triplets with higherhuman consensus (Fig. 4.8b). In contrast, predictions for triplets with lower humanconsensus diverged the most from human judgements, most notably for the newlytrained Doc2Vec model (Fig. 4.8c). T_hese results are an indication that the Doc2Vecembeddings donot capture theunderlying semantics sufficiently andmore trainingdata would be necessary to do so.115In general, the results demonstrate that all models are nearly on par except forDoc2Vec and LSI. T_he result that simpler models, such as LDA or even TF-IDF,are not substantially worse than more complex models confirms that previousfindings [47] apply to this task setting.Although each model can be further tuned andmore sophisticated NLP techniquescan be applied, the results indicate that off-the-shelf models can be leveragedas a baseline despite the challenging data characteristics discussed in Sec. 4.5.4.Although our comparison with human judgement is a reasonable first step, it isonly one signal and other factors, such as performance or explainability of results,should also be considered as next steps when choosing an appropriate model.Figure 4.8: Agreement between model predictions and human judgements for the semantictext similarity task. Each cell shows the Fleiss’ Kappa agreement between two models or be-tween the humanmajority vote and themodel prediction.4.8.3 Prototype Instantiation&Similarity FacetsWe decided to instantiate a similarity model based on LDA and Jensen-Shannondivergence in our final proof-of-concept system. Although other models also hadhigh performance, including document embeddings based on GloVe, the charac-teristics of LDA that led us to choose it are its relative simplicity and its utility forother user interface use cases such as showing topic membership as well. Althoughother approaches are even more appropriate for online learning, it is feasible with116LDA [106] andmedium-sized visualization collections can be re-processed regularly.We discuss further scalability implications in Sec. 4.9.1.Our requirements elicitation process revealed that a single numeric similarity scoreis not adequate for the diverse tasks that should be facilitated by a recommendersystem. We have identified three types of recommendations that we refer to assimilarity facetswhere a text-based similarity model can be applied. All three facetsuse the same underlying model but require a slightly different optimization offeature selection and score thresholds. We tuned these thresholds manually byreviewing example recommendations in the interface (see Sec. 4.4.2).Facet F1: RelatedWorkbooksT_his facet uses all extracted text objects fromworkbook specifications includingcolumn names and computes a pairwise similarity score between 0-1 with the LDA-basedmodel. T_heobjective is tofindworkbooks that are semantically relatedbutnotnear-duplicates (C4). Weultimately chose0.65-0.9 ona scale from0-1. Tuning thesecut-off thresholds leads to a trade-off. By expanding the range of acceptable scores,we increase the diversity of results and the chance for serendipitous discoverieswhile we simultaneously increase the risk of including less relevant workbooks.Facet F2: Similar VersionsIn contrast to F1, users are sometimes specifically interested in seeing differentversions of a workbook (C4), a task that can be expedited by a recommender sys-tem identifying exactly the near-duplicates. T_his facet uses the samemodel andsimilarity scores as F1 but uses a different cut-off threshold that is set very high (≥0.9).Facet F3: SimilarDataIn some scenarios, many workbooks are semantically similar or cover the sametopic. To support more targeted recommendations, we propose a facet that onlytakes columns names as input and ignores other textual features in order to revealworkbooks based on related data sources. Although the underlying data might117not be identical, we hypothesize that a certain degree of overlap between columnnames is an indicator of semantic data similarity. We also use LDA for this facetbut train a separate model with different input data, and choose a high thresholdof≥ 0.9.We surface these similarity facets through our interactive recommender system [101]prototype that allows users to interactively specify their intent while browsing, asdescribed in Sec. 4.4.1 and illustrated in Fig. 4.4.4.9 DiscussionIn this section we discuss limitations, other use cases, and future work.4.9.1 Limitations of Tableau CollectionsOur investigations were based on two Tableau collections, one public and onecorporate. In our crowdsourced experiment with external participants, and theVizCommender interface examples shown in this chapter, we rely on the public onecontaining a sample of workbooks obtained from Tableau Public, because theseworkbooks are publicly available and not restricted by non-disclosure agreements.T_he size of this sampled collection corresponds to typical corporate visualizationcollections with a few thousandworkbooks, althoughwe are aware ofmuch smallerand larger collections in regular use.T_he challenge with a small number of workbooks becomes one of trainingmeaning-ful topics for the LDA-based similarity model. T_he results of our study demonstratethat a GloVe model that is pre-trained on external data represents a viable alterna-tive.T_he challenge of very large visualization collections include scalability problems, al-thoughwenote that the similarities between viz specifications canbepre-computedoffline. One way to address this issue would be to use a two-step process. First,the MinHash [37, 56] algorithm can be applied to compare a large number of vizspecifications quickly. Second, a more precise NLPmodel, such as LDA and Jensen-118Shannon divergence, can be applied to a small set of items that is selected in thefirst step.In addition to the raw size of a collection, for our proposed viz-to-viz similar-ity model to function there must be a large enough number of words within itsworkbooks to discriminate between them or else the recommendations becomeimprecise. In the VizCommender system, workbooks with fewer than 10 words arenot considered for recommendations. Our suggested text-based approach is notadequate for visualization collections that contain a significant number of visual-izations with little or no text. In that case, other techniques, such as collaborativefiltering, could be more appropriate.Although we have focused on Tableau workbook specifications, most visualizationsystems, such as Plotly, MS Power BI, or Looker, have similar textual features fordescribing data visualizations. While feature extraction would depend on thespecification format, the basic processes we outline in this chapter and the findingsof the user study should be applicable to any repository of computer-readable vizspecifications.4.9.2 Near-Duplicates GroupingT_he proposed viz similarity measure can be utilized in other scenarios beyondcontent-based recommendations per se. One such example is the grouping of near-duplicates (challenge C4 in Sec. 4.5.4) when browsing visualization collections. Akeyword search typically results in hundreds of potentially useful results but is oftencluttered because of masses of near-duplicates. To save precious screen real estateand avoid the need to inspect many repetitive workbooks, similar items could begrouped together [118, 278]. One approach would be to show only a single repre-sentative workbook, for example the last modified one, with a symbol to indicatethat it is a group proxy. Users could optionally explore near-duplicates (or differentversions) in a detail view, similar to the facets in the proposed VizCommender.1194.9.3 Hybrid andSequence-AwareRecommender SystemsFor this project we chose to create recommendations based on a single referenceworkbook as an obvious first step, to serve as a base for and inspire future studies.In this scenario, the user visits a workbook detail page and gets recommendationsthat lead to additional insights. T_he query is comparable to the frequently usedsearch-by-example paradigm that requires active user input. Content-based rec-ommender systems are nowadays used in many scenarios as well, and often turninto more sophisticated hybrid systems [40] by incorporating additional user dataand behavioral patterns. For example, the similarity model can take the popularityof workbooks into account when ranking recommendations. Another direction forfuture work would be a sequence-aware model [218] which could record user ses-sions and base recommendations on a collection of relevant workbooks rather thana single query. Further, we speculate that our approachmight prove adaptable toother collections based on annotated, semantically rich, text-based specificationssuch as spreadsheets, electronic circuits, and computer-aided design.4.10 SummaryWe present the concept of a text-based topic similarity measure to be used towardscontent-based recommendations in visualization collections. We investigate fourapplicable NLPmodels and conduct a user study that demonstrates it is possibleto obtain good alignment between human similarity perception and off-the-shelfmodel predictions, despite the special data characteristics and associated issues.An iterative process and continuous engagement with the Tableau RecSys Groupresulted in VizCommender, a proof-of-concept system that uses an LDA-basedmodel to provide recommendations based on a reference viz workbook. While ourprototype is implemented for Tableau workbooks, we believe targeted recommen-dations can significantly facilitate information seeking in all kinds of visualizationcollections that are defined by similar text-based specifications. T_he results of ourcontrolled experiment are generalizable to other short text fragments with similarcharacteristics.1205 VizSnippets: RepresentativePreviews for VisualizationCollections1Visualization collections, accessed by platforms such as Tableau Online or PowerBI, are used by millions of people to share and access diverse analytical knowledgein the form of interactive visualization bundles. Result snippets, compact previewsof these bundles, are presented to users to help them identify relevant contentwhenbrowsing collections. Our engagement with Tableau product teams and review ofexisting snippet designs on five platforms showed us that current practices fail tohelp people judge the relevance of bundles because they include only the title andone image. Users frequently need to undertake the time-consuming endeavour ofopening a bundle within its visualization system to examine its many views anddashboards. In response, we contribute the first systematic approach to visualiza-tion snippet design. We propose a framework for snippet design that addresseseight key challenges that we identify. We present a computational pipeline to com-press the visual and textual content of bundles into representative previews that isadaptive to a provided pixel budget and provides high information density withmultiple images and carefully chosen keywords. We also reflect on the method1T_his chapter is a slightly modified version of our paper VizSnippets: Compressing VisualizationBundles Into Representative Previews for Browsing Visualization Collections by Michael Oppermann andTamara Munzner, accepted for publication at IEEE VIS 2021 (TVCG).121of visual inspection through random sampling to gain confidence in model andparameter choices.5.1 MotivationPeople across organizations and sectors now have access to growing troves of dataand collectively amass large collections of visualization content. Business intelli-gence (BI) tools, such as Tableau2, Power BI3, Qlik4, and Looker5, are used daily bymillions of people. T_he primary artifacts created with these tools are visualizationbundles, that combinemultiple visualizations, dashboards, and data sources. Wenow generalize the concept of a visualization workbook, described in Chapter 4, toother tools and refer to it as visualization bundle. Bundling content to organize,save, and share analysis results is a common practice even beyond BI tools, throughcomputational notebooks such asObservable6 that encompass substantial amountsof diverse visualization content, albeit with different layouts and possibly targetedtowards different associated tasks.When users browse or search collections of visualization bundles, they regularlyneed to choose between multiple results and judge their relevance. To avoid thetime cost of loading and examining in detail these results one by one, bundles aresummarized as compact previews, which we call snippets. However, our reviewof snippets showed that existing snippet designs yield poor information density:they fail to help people judge the relevance of bundles because they include onlyone image and a title, falling far short of communicating the full informationcontent of a bundle that may include multiple views and dashboards. We engagedwith Tableau product groups and end users to determine that this problem hassubstantial real-world impact, and reviewed snippet designs across fivemajor toolsto establish that it is pervasive.2Tableau, https://tableau.com3Microsoft Power BI, https://powerbi.microsoft.com4QlikView and Qlik Sense, https://qlik.com5Looker, https://looker.com6Observable Notebooks, https://observablehq.com122Although guidance exists for snippet design with data types such as text, images,and video, we are the first to systematically approach visualization snippet de-sign. We propose a computational pipeline to compress visualization bundles intorepresentative snippets, providing visualization system designers with a set ofadjustable algorithmic building blocks. It has many controllable features includingthe ability to adapt to a given pixel budget and to create flexible families of layoutswith desired form factors. We also present practical guidance on how visualizationsnippet designers can use the pipeline to support many use cases through concreteexamples. Instead of evaluating specific instances of visualization snippets withend users, our goal in this research endeavour is to provide the conceptual andtechnical underpinnings to snippet designers to facilitate the creation of moreinformative snippets for a variety of usage scenarios.We contribute:• A set of eight key challenges pertaining to visualization snippets, and a top-down design framework for visualization snippets with three levels (page,snippet, images+text) and two categories of design choices (size/number ofelements, visual layout/encoding).• A computational pipeline for the lossy compression of visualization bundlesinto snippets, consisting of a set of algorithmic building blocks to filter, rank,and display visual and textual content in accordance with an available pixelbudget.We validate the pipeline’s algorithmic building blocks on two kinds of visualiza-tions bundles, from Tableau Public and Observable, using quantitative measures,visual inspection, and A/B comparisons. For this purpose, we also labeled severalthousand images to compare human judgements with model predictions.As a secondary contribution, we promote amethodology for extensive visual inspec-tion through random sampling. We reflect on the creation of multiple lightweightvisual inspectors, each tuned for one question or algorithm, that sample a sub-stantial corpus of real-world data to gain confidence in algorithm and parameterchoices.123Inspired by the work of Rogers et al. [223], we include direct links to research arti-facts, included in Appendix D, to transparently provide an abundance of evidencefor our claims and our process.5.2 Visualization Collections and SnippetsWe provide a data characterization and describe the collections we use to assessour work, define snippets in the context of visualization collections, survey currentpractices in snippet design, and discuss our investigation of real-world challengesthrough engagement with Tableau product groups. We identify key challenges andpresent a top-down framework for snippet generation to address them. We finallydiscuss the intended context of use by snippet designers for our computationalpipeline.5.2.1 Data Characterization andSourcesWedefine visualization bundles and collections in general, followed by a descriptionof two sample collections that we acquired.Bundles andCollectionsVisualization bundles combine single views (charts), dashboards combining multi-ple views [226], and one or more datasets.We categorize the content into four groups, while noting that some bundle typescontain only a subset of this information.Bundle meta-data contains the bundle title, author name, date, and usage statistics,such as the number of views and likes.Bundle text is bagsofwords in eachof three categories: 1) textwithin views, includingchart and axis titles, captions, and titles; 2) data column names; 3) longer prose textthat may be attached.124Bundle images are stored as PNG files, each corresponding to a view or dashboard.All images conform to the same 4:3 aspect ratio, and may be cropped or paddedwith white space to fit those dimensions.Bundle specification contains additional layout and chart type information, for ex-ample which views are embedded in a dashboard.We refer to repositories of manually crafted bundles that can be browsed on ded-icated platforms as visualization collections. T_he number of bundles in a collectionranges from hundreds to thousands or more when shared within a company ororganization, or even millions in special cases that are open to larger communities,such as Tableau Public.Data SourcesWe used two collections of real-world visualization bundles to develop and evaluatethe proposed computational pipeline.Tableauworkbooks are awidely used type of visualization bundle, containing all nec-essary components that are required for loading and displaying the fully interactivevisualizations to users.Webuildonourpreviousworkonvisualization recommendation, described inChap-ter 4 [204], where we extracted bundle data from the XML files of a set of 2910hand-crafted workbooks (4908 dashboard images, 10563 view images) randomlysampled from Tableau Public7, a sprawling collection of publicly shared workbooks.An informal visual inspection showed that our corpus covers many different topicsand includes bundles of various complexities.Other business intelligence tools, such as Looker, Qlik, and Power BI, use similarapproaches to bundle visualizations and data sources into workbooks, reports, orapps, so our findings are generalizable beyond Tableau.Observable is a reactive computational notebook for data analysis and visualization,used by a rapidly growing community. A notebook is made up of a series of cells7Tableau Public, https://public.tableau.com125containing JavaScript code, prose, interactive visualizations, and images. T_hisstructure follows the paradigm of literate programming [130] where explanationsin natural language are interspersed with code snippets. Although the use casesrelated toObservable notebooksmaybe very different to those facilitated byBI tools,they exhibit intriguing content similarities; thus we also consider those notebooksas visualization bundles.We implemented a Chrome extension to scrape a diverse sample of 178 publiclyaccessible Observable notebooks. T_he source code of these notebooks could bedownloaded directly from the website, and prose and static images extracted, butwe wanted to capture interactive visualizations in the way they are shown to users.Our tool takes screenshots of Canvas and SVG areas, downloads embedded images,extracts meta-data information, and creates a bag of words from the text found inMarkdown cells. All visualizations are treated as single views because inferring thecomposition of dashboards from source code is non-trivial, and the actual numberof dashboards on Observable appears to be very low (0 in a sample of 40 notebooks).https://osf.io/6kyg3 contains the source code of the scraper and the raw data of ourObservable collection.5.2.2 Visualization SnippetUsageWe define visualization snippets broadly as compact, representative summaries ofvisualization bundles, consisting of visual and textual elements, that help usersdiscriminate between relevant and irrelevant content in visualization collections.Use cases for snippets within visualization platforms include showing search re-sults, supporting faceted browsing, and suggesting recommendations. T_he formfactors vary: many medium-sized snippets can be shown within a large window aswith search results, or a few snippets within a small region of the screen as withrecommendation results, or a single large snippetmay appear upon hover in a largepopup window to preview bundle content with as much detail as possible.We provide an example scenario to illustrate the search use case for snippets. Amarketing analyst searches the company-wide visualization collection to check ifany colleagues have previously created a report on product downloads. T_he col-126lection of several hundred bundles does have some hierarchical organization intofolders, but that structure does not help them scope their search. A query resultsin multiple pages with over a dozen snippets in each. T_he analyst rules out somebundles as irrelevant from the title and single image visible from the snippet, butoften must open the bundle to check whether a bundle contains relevant materials.T_he analyst spends several minutes opening bundles and clicking through each oftheir views, only to realize that they are outdated or concern a different product,before ultimately locating the desired analysis report.In the information retrieval process, snippets concern the relevance prediction step[81]where users predict which items summarized in a results display align with the cur-rent information need before they open the detailed versions [96]. Opening manyvisualization bundles can be a time-consuming and tedious endeavour becauseof substantial load, build, and explore times, especially when visualizations con-tain many views or are connected to live data streams. Snippets serve as previewsto facilitate the relevance prediction step as surrogates that avoid loading entirebundles via visualization platforms.5.2.3 Embedded InquiryWithin TableauT_his work grew out of collaboration with the Recommendation and Search productgroups at Tableau, following up on our previous investigation on content-basedrecommendation models for visualizations reported in Chapter 4. Over a period ofmore than a year, we had conversations regularly with product managers, machinelearning engineers, and UX designers, who relayed end-user feedback and painpoints about many issues to us.T_he key realization that motivated this research project is that the shortcomings ofexisting snippet designs constitute a significant pain point for Tableau end users.Specifically, we have substantial evidence that it is difficult and often impossiblefor users to judge a bundle’s relevance based on the given snippet, without open-ing it. We heard this idea frommultiple people within Tableau. We verified thiscommon internal perception through direct interviews with external target users.T_he author of this dissertation joined several interviews with target users to eval-127uate a new visualization recommendation model; despite the intent to focus onthe back-endmodel, participants repeatedly commented on the presentation ofthe recommendations that were shown as visualization snippets, confirming theproblem of insufficient visible information content.We also observed that visualization authors commonly start from scratch insteadof reusing existing content, even in contexts where re-use would seem to providebenefits; we conjecture that better snippet designmight combat this tendency byhelping authors find relevant content more easily.Finally, we noted thatwhendeveloping newproduct features, product groups eithersimply re-use existing snippet designs or end up making custommodificationsbased on intuition,without systematic reasoning of how snippets could be explicitlydesigned to support platform use cases or user analysis goals.5.2.4 Reviewof Existing SnippetDesignsTo ensure generality of beyond the particular platform of Tableau, we informallysurveyed the content and layout of existing snippet designs used in Tableau On-line/Public, Power BI, Qlik, Looker, and Observable. All existing snippet designsshow only the bundle title and one image thumbnail, and sometimes a fewmeta-data attributes. In many cases these titles are not very descriptive and the singleimage has low information content, as shown in Fig. 5.1, so they fail to supportusers when assessing the relevance of bundles.128Figure 5.1: Examples of existing snippets with low information content in their single imagesand titles: (1) TableauOnline, (2) Tableau Public.None of the snippet approaches that we surveyed incorporate multiple images; inall cases a single thumbnail image of a view or dashboard is selected and croppedautomatically. For example, Observable chooses the first image, SVG, or Canvaselement found in a notebook. Even if a bundle contains useful content, the auto-matically selected images may show incomplete draft views or appear empty whenscaled down to a thumbnail size. Only a few tools allow creators to upload customthumbnails thatmay include illustrations that reflect the bundle context but are notdirectly related to the data [123]. T_he thumbnail size varies substantially betweenplatforms.In terms of text content, the bundle title is always prioritized, although sometimessignificantly truncated. T_he only approaches that include a textual description areTableau Public’s gallery view that presents human-curated bundles and Qlik’s listview. Wenote that creators are frequently unwilling to invest the additional effort ofcrafting a description. None of the existing snippet designs include keywords or anyother textual information extracted from the bundles such as tags, despite taggingfunctionalities in some of the tools. Meta-data attributes such as author name,date, and number of likes, are included in some snippet designs but the selectionof these attributes differs widely across platforms. Although bundle meta-data isoften enhanced with small icons, we consider bundle meta-data as text content inour analysis below.129Snippets are arranged in a uniform grid layout by default in all tools, except TableauPublic which uses a list layout. T_he option to choose between two different layoutsis provided in 4 of the 6 platforms. T_he snippet size is always the same independentof the bundle content, andmost of the pixel space is devoted to the thumbnail.Further details are included in Sec. D.1, and Sec. 5.5.3 presents A/B comparisons be-tween existing product snippet designs and the results of our proposed VizSnippetspipeline.5.2.5 ChallengesOur two-pronged investigation led us to identify eight challenges for snippet gen-eration:C-OneImage: Limited information content of a single image. A single image of one viewor dashboard provides only a very narrow lens on a bundle andmay not accuratelycapture the diverse visual content across multiple views. Blank thumbnail imagesare surprisingly frequent within our sample collections.C-Titles: Limited information content of titles. We learned through interviews and byreviewing the data samples that short and cryptic bundle titles pose a challengeacross all collections. Uninformative titles in similar forms such as “sub-2”, “Project8”, “Final Vis”, and “Untitled” are seen frequently. Vague titles impede the identifi-cation of relevant content for non-authors, and community-generated content isparticularly messy.C-Pixels: Limited pixel space. Pixels are a precious commodity that need to be used aseffectively as possible, on the desktop and especially for mobile. T_he full content ofa bundle can rarely be shown in its entirety, necessitating compression.C-SparseText: Limited textual content.T_he text found in visualization bundles that arecreated with BI tools generally consists of fragments of a few words, such as titlesor labels. Although a visualization author might sometimes include captions orannotations of lengthier text, the amount of available text is far less than in stan-dard text documents [204]. T_his challenge is not directly applicable to Observable130computational notebooks, which commonly include prosewith typical grammaticalsentences similar to text documents, for instance when created as instructionalguides.C-Form: Diversity of form factors. T_hewide variety of use cases for snippets leads todifferent form factor requirements, frommanymedium-sized snippets in a largewindow to support quick comparisons between bundles, to a few small snippetsin a small region using minimal screen real estate, to a full-screen view of a singlelarge snippet to provide a very detailed preview.C-Complex: Diversity of bundle complexity. T_he content in the reviewed collectionsranges from single bar charts to complex bundles with dozens of views and dash-boards. Quantitatively, the bundles in our Tableau Public collection contain 2.19dashboards and 4.11 views on average (the medians are 1 and 3 respectively). Viewsare often just building blocks for dashboards or contain auxiliary information.C-Quality: Diversity of content quality.We found substantial differences in the qualityof information content within bundles: the same bundle might have empty orincomplete drafts in addition to high quality finalized content, or many similarimages alongside quite different ones. When summarizing bundles, high qualityinformative content should be elevated, while incomplete or empty or similar viewsand less informative text should be filtered or downgraded.C-Goals: Diversity of end-user goals.We identified disparate end-user goals that couldbe supported by visualization platforms: collaborate with colleagues, create portfoliosto share work, find out how other people did something, get visual inspiration, look up in-formation, and save all visualizations in one space. T_hese goals may lead to differentsnippet affordances. For example, images and chart types are paramount for vi-sual inspiration, but information lookup might require more emphasis on theunderlying data sources of a bundle.1315.2.6 Design FrameworkTo address these challenges, we propose a top-down design framework for snippetgenerationwith three levels and two categories of choices, shown in Fig. 5.2. T_he toplevel pertains to the layout of a result page, and choices at this level affect options atthemiddle level of snippet and lowest level of images+text (see Sec. D.5 for all possiblecombinations). One category of design choices is the size and number of elements,and theotherpertains to the layout andvisual encoding. T_hecomputational pipelineproposed in Sec. 5.4 fully supports this framework.We consider snippet generation through the lens of lossy compression, where thegoal is to provide asmuch detail as possible within a given pixel budget. Although avery compact snippet that could be used within a grid of many snippets is a highlycompressed representation of a bundle, another use case is a full-screen preview ofa single bundle that could capture a substantial fraction of the salient content withlimited loss.Figure 5.2: Design framework for displaying visualization snippets, with three levels and twocategories of choices. Each blue box corresponds to one snippet and contains visual content(yellow boxes) and textual content (pink boxes).5.2.7 Intended Framework andPipelineUsageWe instantiate this design framework with a computational pipeline that delivers apowerful and flexible infrastructure to visualization snippet designerswhowant to au-132tomatically create snippets with greater information density than existing designs.We consider snippet designers to be a broad group, encompassing not only UXdesigners but others involved in creating or customizing snippets related to visual-ization collections, such as a teamworking on a recommendation systemwantingto extend snippets to add explanations why something gets recommended [261].We anticipate that snippet designers would use this pipeline after they ascertainthe exact requirements for their particular use case. T_hat characterization wouldprobably require targeted user studies, which we leave to those designers ratherthan attempting to conduct any of them ourselves. Instead, we thoroughly validatethe technical underpinnings and performance of the pipeline and its algorithmicbuilding blocks on real-world data, and provide actionable recommendations onhow it can be flexibly adapted for different scenarios.We hope our work encourages the creation of representative snippets beyond singlethumbnails with titles. T_he diversity of snippet form factors (C-Form) and end-usergoals (C-Goals) reveals that different use cases would benefit frommore specializeddesigns.5.3 RelatedWorkWenowdiscuss result snippets for a range of content types, and existing approachesto visualization compression.5.3.1 Result SnippetsOur work is informed by previous studies that evaluated snippet representations.Result snippets have been investigated for a range of scenarios and content types,such as documents [83, 252, 254], source code [73], e-commerce products [190],books [171], music [303], videos [248, 282], images [52], and extensively for web-sites [15, 81, 119, 141, 167, 259] in the context of search engine result pages. Althoughthe relative position of results remains one of themost influential factors for assess-ing their relevance [167], the back-end algorithms to find and rank bundle results133are not the objective of this project; instead, our focus is specifically on the snippetcontent and its representation.Several studies examined website snippets that contain thumbnails only and incombination with text summaries [15, 81, 154], and suggest that thumbnails usedalong with text summaries help in reducing errors when predicting relevance.Capra et al. [41] showed that images provide only a small benefit in judgmentaccuracy compared to text-only snippets. However, they emphasize that image-augmentation helpedmeasurably in several scenarios when textual componentsare poor, which is the case for visualization bundles. Otherwise, the characteristicsof a website are significantly different than visualization bundles.Snippets for videos are particularly relevant for our work. Due to their intrinsicgraphical and temporal nature, videos exacerbate the need for snippets that offeradditional representation facets besides text. Wildemuth et al. [283] concludedthat multi-image compositions or short video sequences are more expressive andfavored by users over single images. Displaying multiple key frames of a video [98,283] is similar in spirit, but different in detail, to the image collage technique thatwe propose in Sec. 5.4.3. Collages are composed of several selected visualizationsand dashboards to provide a bird’s eye view on the bundle content. However, dueto completely different semantics, existing techniques for identifying highlightframes from videos cannot be directly applied to selecting visualization imagesfrom a bundle.Natural language provides great richness in result snippets and is used beyondtextual documents for other types of media, such as videos or photographs. Aug-menting a snippet with a description or keywords has been proven effective inprevious work [57, 85].Recent approaches to automatically generate captions for visualizations are promis-ing but are limited to single views and specific visual features [6, 157, 185]. In con-trast, our goal is to summarize visualization bundles, with potentially dozens ofviews, on a high level. T_he automated creation of snippets for complex visualizationbundles, as we propose in this chapter, has not been studied previously.1345.3.2 Visualization CollectionsManyEyes was a pioneer platform that supported a social style of visualization anddata analysis[273]. Morton et al. [189] analyzed howManyEyes and Tableau Publicwere used but have not discussed result snippets explicitly. In a broader context, weconsider literature surveys such as treevis.net [233] or setviz.net [7] as visualizationcollections as well, and the proposed pipeline can be adapted for those use cases.We distinguish visualization collections from dataset repositories that do not con-tain visual encodings, such as VizNet [110]. Summarizing datasets into snippetsposes its own research questions [53, 68] and is beyond the scope of this project.5.3.3 Approaches toVisualization CompressionIn the visualization literature, many approaches have been discussed to com-press visual encodings into more compact representations, for example, to createoverviews, to support comparisons across many items, or to allow consumptionon mobile devices. T_his body of work spans across visualization techniques, in-cluding trees [160], graphs [14, 300], time series [50], small multiples [36], andinfographics [39, 165]. In recent years, there has been a growing interest in re-sponsive visualizations [107, 294]. In contrast, our goal is different from all theseapproaches. Rather than adapting visual encodings in response to different screenresolutions, we summarize complex visualization bundles containing existing im-ages ofmultiple views and dashboards. T_hemethods discussed in Sec. 5.4 are basedon selecting from a given set of images; we do not propose new altered or derivedvisual encodings.Most closely related is the work on visualization thumbnails. Heer et al. [103]proposed to include thumbnails of single views in graphical histories to supportthe visualization authoring process. Kim et al. [123] surveyed current practices increating visualization thumbnails, specifically for data stories. Our work does notaddress generating new thumbnails, but rather provides computational methodsfor selecting from a set of existing thumbnails to generate snippets.1355.4 VizSnippets Computational PipelineWe propose a computational pipeline to compress visualization bundles into repre-sentative snippets, that is adaptive to given space constraints (C-Pixels and C-Form)and other user-defined preferences. T_he pipeline, illustrated in Fig. 5.3, consists ofmultiple interchangeable algorithmic building blocks with minimal dependencies.T_he extraction (green box at the top) and lossy compression of the bundle content(yellow boxes for images on the left, pink for text on the right) is completed in anoffline preprocessing step. T_he creation and arrangement of the snippets (blue boxat the bottom) is determined at run time.136Figure 5.3: VizSnippets computational pipeline to compress visual and textual content of visu-alization bundles into representative snippets. Visual inspectors I1 to I8 are referenced that weused to gain further confidence inmodel choices and parameter settings.5.4.1 Visual InspectorsWe developed and evaluated the building blocks iteratively by using a suite of eightvisual inspectors, I1 to I8, shown in Fig. 5.4 (and Sec. D.2).T_hese visual inspectors are lightweight user interfaces that allow the inspectionof randomized item-level results, such as extracted keywords, filtered images, orgenerated snippets, instead of presenting aggregated statistics. Each inspector is137targeted at a specific algorithm or analysis question and presents results based onrandom data samples, leveraging an extensive corpus of real-world data.T_his type of extensive visual inspection allowed us to gain further confidence inmodel choices and parameter settings, beyond standard quantitative analyses. Weelucidate how inspectors informed our decisions throughout the following sections,and reflect on this approach as a generalizable method to provide guidance and tofoster transferability in Sec. 5.6.138Figure 5.4: Overview of visual inspectors that we used to analyze models and parameter set-tings. High-resolution screenshots are in Sec. D.2.1395.4.2 Page andSnippet LayoutsWe support five general page layouts that control how snippets are arranged (il-lustrated in Fig. 5.2): the grid layout divides a page into rows and columns, and allsnippets have a uniform size (fixed width, fixed height); the strip layout, also knownas masonry layout, divides the page into equal sized columns (fixed width) withvariable snippet heights; the list layout displays one snippet per row (full width,variable height); the table layout also shows one snippet per row but meta-data at-tributes are divided into separate sortable columns and the snippet height is fixed;the preview layout allocates all pixels of the page to a single large snippet. T_hese fivelayouts support all snippet use cases and form factors (C-Form) that we identified.T_he initial page layout selection has cascading effects on lower levels and dictatescertain design decisions. For example, the page layout dictates if a snippet hasa variable or a fixed size. With the grid layout, we can choose a specific snippetsize and decide what ratio of pixel space should be devoted to visual versus textualcontent (C-Goals). With the strip layout, the snippet height is variable and dependson other parameters related to the text and image content (C-Complex). T_he actualnumber of snippets that are visible on screen depends on the snippet size and thechosen page layout.We provide three options to arrange the visual and textual content within a snippet:a vertical, horizontal, or table layout. We deemed more complex compositionssuch as wrapping text around images to be less important for the use cases weidentified, but these could be added in the future. T_he snippet layout defines thecanvas that will be filled with text and image content. We describe the algorithmsfor compressing and displaying the bundle content in subsequent sections.5.4.3 Snippet ImagesOur investigation confirmed that images are particularly relevant for visualizationsnippets. Besides conveying the visual style and helping users to recognize pre-viously seen content [141, 259], images capture semantic information. T_he charttype can indicate underlying data types or the visualization objective. For instance,maps imply geographic data and pie charts show part-to-whole relationships.140We now describe the algorithmic building blocks for selecting and displaying com-posite images to increase the information content of a snippet beyond what asingle image can convey (C-OneImage). T_he yellow boxes in Fig. 5.3 show theimage-related blocks: filter empty and embedded images, filter nested images,rank images, filter similar images, re-rank images to increase diversity, and displayimage compositions.Filter Empty andEmbedded ImagesIn a first step, we filter empty and nearly empty images based on a dominant colorthresholdCT . We convert each image to a list of RGB pixels (points in 3D space)and use the k-means clustering algorithm to extract the k dominant colors of animage [251], wherek corresponds to the number of cluster centroids that are used toiteratively assign the pixels to clusters. We compute the relative size of each cluster,namely howmany pixels of the original image are assigned to each dominant color,to determine if a single color predominates an image, signalling that it is lessinformative (C-Quality). A dominant color thresholdCT of 98.2% led to the highestaccuracy for the sample visualization collections, with k = 5 colors. Further detailsabout determining these parameters are included in Sec. 5.5.2.For visualization bundles that contain dashboards, we avoid repetition by filteringout images of the embedded views thatwere the building blocks used to create them.Prioritizing dashboards makes wise use of available pixel space (C-Pixels). T_hisprocess is straightforward for Tableau data because the views that are embeddedin a dashboard are defined in the bundle specification. All images extracted fromObservable notebooks are treated as single views.Rank ImagesWe create an initial ranking of all images within a bundle. In many cases, theavailable images do not fit within the allocated pixel budget so some images arehidden or only revealed after user interaction, as we will describe in Sec. 5.4.3.Consequently, the order of images is relevant to increase the explanatory powerduring the first impression. We divide the images into two groups: dashboardimages and single-view images.141Dashboards are prioritized and sorted by the number of embedded views and colordiversity. T_he percentage of the most dominant color that we computed earlieris used as a proxy measure for color diversity. T_hrough visual inspection with I5(see Sec. 5.5.2), we discovered that mostly single-color images are less informativebecause they mostly show a background color.We group all single-view images by chart type, sort the images within each groupby color diversity, and sort all groups by their maximum color diversity. T_hen weiteratively withdraw one image from each group until all groups are empty. All theimages get assigned a consecutive number that is used as an initial score. Whenchart types are not available, such as in Observable collections, the images are onlysorted by color diversity.Filter Similar ImagesBundles frequently include very similar or identical views that should be filtered(C-Quality). We compute pairwise distances between all bundle images and filterout those where the distance is below a given threshold. For determining thedistance between images, we compared three traditional approaches based onhand-crafted visual features, histogram of colors (HoC), histogram of orientedgradients (HoG) [67], and structural similarity index (SSIM) [279], and the morerecent approach of learning visual features and a similarity function through aSiamese convolutional neural network (CNN) [269].Histogramof colours (HoC).We extract the RGB colours from an image and createa feature vector with 512 bins. We compute the similarity between images using theBhattacharyya distance [44] which is commonly used in computer vision to comparehistograms.Histogram of oriented gradients (HoG).T_he HoG feature descriptor focuses onshapes within an image by analyzing the magnitude and direction of edges. T_hehistograms are created by dividing an image into small regions and calculatingthe gradients and orientations of the pixel values. We used the same parametersas in the original paper [67]: resize to=(128,64), orientations=9, pixels per cell=(8, 8), cells142per block=(2, 2). T_he output histogram is of length 3780. Similar to HoC, we use theBhattacharyya to compare two histograms.Structural similarity index (SSIM) [279] is a quality metric that compares the sim-ilarity of two images by extracting luminance, contrast, and structural features.Given a pair of images, SSIM returns a score between 0 and 1.Siamese convolutional neural network (CNN) [269]. T_he architecture of the deeplearning model is shown in Fig. 5.5. It contains two CNNs with identical structuresthat share their learned weights. Each CNN is a stack of convolutional and poolinglayers to learn features from images. T_he CNNs first individually transform theinput data into feature vectors which are then combined using a loss functionto learn the concept of similarity. We chose euclidean distance and contrastiveloss because we labeled similar and dissimilar images pairs earlier; previous workshowed that other loss functions could lead to higher accuracy [48].Figure 5.5:High-level architecture of the Siamese CNN to identify similar images.T_he performance of the CNN, evaluated based on human judgement labels, wassimilar with the best hand-crafted feature approach for this particular task ofidentifying very similar or identical images, so we chose the less complex, unsu-143pervised method (HoG). See further details about the comparison and our labelingapproach Sec. 5.5.2.Re-Rank Images to IncreaseDiversityAfter applying the previous steps, similar images might still be ranked high andclose to each other. T_he chart type specificationmay not be available or cannot bedetected, or views use the same base chart type.We propose a re-ranking step usingmaximalmarginal relevance (mmr) [42] to ensurethat the visual diversity of a bundle is conveyed by a very small number of images.MMR is a greedy, iterative algorithm that linearly interpolates between the originalranking score of an image and the diversity to other, already selected images. T_healgorithm is typically used in diversity-based text retrieval scenarios [42, 197] buthas been also applied to image re-orderings [249].T_heMMR algorithm hinges on the similarity model in Sec. 5.4.3 to accurately com-pare images. First, the top-ranked image is selected based on the initial score andall remaining images are considered as candidates. T_hen, the algorithm iterativelyselects the imagewith the highestMMR score sˆ := λ ·s−(1−λ) ·MaxSim, termi-nating when all candidates have been assigned to the ranking. T_he relative weightbetween the original image score, s, and the maximum similarity,MaxSim, isspecified with a single tunable parameter λ. For example, λ = 1 ranks entirelyby s and λ = 0 selects maximally diverse images irrespective of the initial rank-ing described in Sec. 5.4.3. We found that λ = 0.75 leads to diverse rankings, assuggested in a systematic MMR evaluation on comment diversification [197].Display ImageCompositionsWe propose image collages to show multiple images within a snippet. A collageis composed of equal-sized images that are displayed in a uniform grid layout.T_he number of images depends on the chosen snippet size (see Fig. 5.2): (a) fora variable snippet size, designers specify the number of images and the imagesize, and the overall collage size expands accordingly; (b) for a fixed snippet size,144designers specify the minimum image size and the algorithm tries to fit as manyimages as possible into the collage.T_he construction of the collage layout is based on a grid packing algorithm thatdetermines the optimal number of images iteratively. We first approximate theinitial image height and then tune this value while ensuring the minimum imagesize requirement. When there are more image slots than images available, theimage size gets increased to make optimal use of the given pixel space (C-Pixels).Although text within the image is not legible, the chart types and visual styles canoften be identified.In some cases, the selected images exceed the number of slots within a collage.Besides merely hiding overflow images, one option is to create an image carouselthat consists of multiple collages and allows users to scroll through additionalcontent. T_he supplemental video (https://osf.io/4wz3d/) shows image collages andcarousels in action, and we discuss limitations and opportunities of multi-imagecompositions in Sec. 5.5.4.5.4.4 Snippet TextOur pipeline supports three kinds of text displayed within a snippet: the bundletitle, keywords, andmeta-data. All the BI tools that we reviewed use only the user-specified bundle titles, but our finding that these titles may not accurately reflectthe semantic content of bundles (C-Titles) leads us to amore sophisticated approach.We argue that a small set of carefully chosen keywords (or tags) can provide valuableinsights without consuming too much space.T_he pink boxes in Fig. 5.3 show the process for extracting keywords and display-ing text content in a snippet: preprocess text, generated ordered list of keywords,filter keywords, add calendar years, select meta-data, and display text content.Process Text andExtract KeywordsWedistinguishbetween twodifferent bags ofwords that are extracted fromabundleand used as input data, taking care to glean all information from the limited textualcontent (C-SparseText). First, the visible text includes chart and axis titles, sheet145names, captions, annotations, and descriptions. Second, if available, the columnnames from data sources are used as additional text data and referred to as hiddentext. T_his hidden text can provide a useful signal about the bundle contents [204],to address the problem of limited information content of titles (C-Titles). Both setscan contain human readable text thatmay, but does not always, capturemeaningfulsemantic information. T_he goal of this process is to select and rank keywords thatbest represent the underlying bundle content.We remove punctuation, strings with less than three characters, numbers, andEnglish stopwords. We also lemmatize the text and apply a custom stop-word filterof 349 words we identified by analyzing frequent but non-informative words orphrases in our sample collections.We use the term frequency-inverse document frequency (TF-IDF) model [222] on bothvisible and hidden text to extract keywords. In essence, TF-IDF functions as anadvanced stop-word filter to promote terms that are unique across the collectionand to weed out common language. Up to 30 keywords are extracted for each bagof words; this threshold is relatively high, and inmany cases fewer keywords areavailable and ultimately displayed. T_he numerical TF-IDF-weighted scores of theterms are used for the ranking.We remove keywords that are included in the bundle title to avoid repetitions.Although we lemmatize words to capture variant forms of the same word, verysimilar keywords may still get selected because they are not in the lexical database.To increase the diversity of top-ranked keywords, we compute Levenshtein distancesbetween pairs of words and filter similar ones [221]. T_he Levenshtein distance is ameasure for the number of edit-operations that are needed to transform one wordinto another. A normalized distance of 0 means that strings are identical and 1if strings are completely different. We identified 0.15 as a useful threshold (seeSec. 5.5.1).We give precedence to visible text irrespective of the TF-IDF scores, and computethe distance between those keywords. If the distance between twowords is lower orequal to 0.15, we check the spelling [177] and choose the alternative that is spelledcorrectly. We find that this mechanism significantly improves the quality of key-146words. We then combine the two sets of keywords by iteratively selecting a hiddentext keyword and comparing the distance to already selected keywords until wehave reached the desired threshold.Finally, we attempt to extract calendar years from the removed set of numbers andadd them back to the keywords. We discovered that calendar years are commonlymentioned in visualizations but often not included in the bundle title. We parsethe text for four-digit numbers within the range of 1900 to 2050. We conjecturethat this range will result only in a small number of false positives, based on ourinformal manual inspection (Fig. 5.6). We combine sequences of years into singlestrings, such as 2019-2021 instead of 2019, 2020, 2021.Figure 5.6: This figure shows how frequently calendar years occur in the text content extractedfrom the Tableau Public collection (2910 bundles). A informal inspection of a set of bundlesverified that 4-digit numbers within the 1900-2050 range are typically calendar years.Display Text ElementsT_he core text element is the bundle title specified by the author. T_he pipeline sup-ports title truncation on demand, but we note it requires careful considerationbecause longer titles are more descriptive. T_he titles otherwise remain unchangedbecause removing stop words or numbers would likely confuse authors.T_he pipeline allows snippet designers to manually choose which meta-data at-tributes are relevant for their use case, such as the author, the number of likes, orrecent date of opening.147Adding keywords to snippets follows a similar approach to displaying snippet im-ages, where we distinguish between variable-sized and fixed-sized snippets. T_hepriority of the keywords is predetermined by their order, as described in the previ-ous section.When the snippet size is variable, designers specify the number of keywords thatshould be displayed. In case of fixed-size snippets, we fit as many keywords aspossible based on the individual title length and the meta-data attributes. Morespecifically, we add keywords successively and verify the size of the snippet bound-ing box until the given constraints are reached. T_his approach is computationallyexpensive for many results; for production environments, we suggest to predeter-mine the default number of keywords that should be displayed for specific snippetsizes.In the table layout, the title and the keywords are displayed in one column and themeta-data attributes are split into separate columns. In case of all other layouts,the title, themeta-data, and the keywords are shown as three horizontal rows in thetext area of a snippet. A high-level parameter controls the font size of the snippettext.5.5 ResultsWe conducted substantial testing on the VizSnippets pipeline during and after itscreation, based on the Tableau Public and Observable collections. We now presentquantitative and qualitative results to demonstrate its utility, validate our claims,and discuss trade-offs related to algorithms and parameter choices.5.5.1 Snippet TextWe used visual inspectors (I1, I2) and descriptive statistics to analyze the textualcontent of bundles and snippets.148TitlesWe noted that the titles often provide little informative value, confirming our earlypresumptions. T_he average length of bundle titles in the Tableau Public collectionis 21 characters (median: 18). If we would apply the same text processing steps onTableau titles as we do for other text content, 16% of titles would be empty strings.T_he sample Observable titles are significantly longer and contain 31 characters onaverage (median: 28).Text Content andExtractedKeywordsWe examined the raw text content of Tableau bundles using I1, which indicatedthat the hidden data column names can serve as useful auxiliary data but are oftenless clear than chart-visible text, confirming our choice to select keywords fromvisible text first and referring to hidden text only if room remains. T_he percentageof correctly spelled words is 93% for visible text and 88% for hidden text.As shown in Fig. 5.7, the number of hidden-text keywords, extracted from datacolumn names, is much larger compared to visible-text keywords in our sampleTableau collection. Data column names are often cryptic and less understandablecompared to text that is visible to the audience of a visualization (see Fig. 5.8),which is the reason why we prioritize visible-text keywords. Nevertheless, due tothe limited text in bundles, we also include hidden-text keywords in some cases.For the Levenshtein distance to filter very similar keywords, we determined thethreshold of 0.15 (on a scale from 0-1) through visually inspecting hundreds of key-word pairs using I2. For larger distances, words frequently have differentmeanings,in contrast to smaller differences that are typos or word variations not capturedduring lemmatization. After applying the filtering steps, the average number ofkeywords decreases from 20 to 16 (the median is 13).149Figure 5.7: Cumulative reduction of keywords after applying the filtering steps on both sets ofkeywords that are extracted from each bundle (Tableau Public collection).Figure 5.8: Spell checking the input text and keywords (Tableau Public collection) confirmedour intuition from the visual inspection that visible textmay bemore understandable than hid-den text. The reason for the lower spell check rate of keywords is due to thenatureof TF-IDF thatprioritizes uniquewords.In case of the Observable collection, we extract keywords from the text found inMarkdown cells. Computational notebooks contain significantly more text andtherefore we can avoid extracting keywords from visualization specifications anddatasets.Figure5.9: Cumulative reductionof keywords after applying thefiltering stepson theextractedkeywords from each bundle (Observable collection).Potential Calendar Years: 4-DigitNumbersIn assessing the extraction of calendar years that are not included in the bundletitle, we found that 19.7% of all bundles contain annual details (within the rangeof 1900-2050) and visually verified that these years provide valuable information150when comparing multiple bundles, such as reports covering different time periods.In 67% of cases, none of the extracted years are mentioned in the bundle title.5.5.2 Snippet ImagesT_he average number of extracted images per bundle is 5.3 (the median is 4). Af-ter applying the filtering steps, the average number is only 2.5 (the median is 2),which demonstrates the potential for compressing visual content with minimalinformation loss.Empty Images andColorDiversityTo evaluate the algorithm for filtering empty and nearly empty images using adominant color threshold, wemanually labeled 2600 images using binary labels.We then tested the accuracy for varying color thresholdsCT and different numbersof clusters (k = [3, 5, 7]), where k corresponds to the number of dominant colors.k = 5 and k = 7 led to similar results but the performance of k-means clusteringdecreases with more clusters (see Fig. 5.10). T_he highest accuracy of 93%, withk = 5, came from settingCT to 98.5%.We also used visual inspector I5 to further analyze what type of images are filtered.IfCT is too low, line charts with minimal data ink may get falsely discarded, whilea very highCT may include images that contain mostly white space and provideno informative value. By using this inspector and adjustingCT interactively, weobserved that images with a high color diversity are more informative when scaleddown, compared tomostly single-color images. T_herefore, we incorporate the ratioof the most dominant color in the image ranking.151Figure 5.10: ROC curve and accuracy for different k-means clusters (k=[3,5,7]) based on 2600manually labeled images from the Tableau Publiccollection. k = 5 and k = 7 reach a similar peak performance but k = 5 is significantly faster.152Similar ImagesWe compared four approaches for computing pairwise image similarities. T_hreemethods are based on hand-crafted visual features: histogram of colors (HoC),histogram of oriented gradients (HoG), and structural similarity index (SSIM).T_hefourth method is a Siamese convolutional neural network (CNN) trained on imagepairs to learn deep representations and the similarity function automatically.Wemanually labeled 6019 image pairs to evaluate the algorithms. We constructedpairs by randomly sampling two images from the same bundle, to increase thechance of similar images and because the algorithms are generally only used tocompare images within and not across bundles. Image similarity was determinedsolely based on the visual style and not the underlying semantics. T_he decidingfactor was if two images provide any additional value or if one representative imageis sufficient. 35% of the image pairs were ultimately labeled as similar. We dividedthe pairs into a training and validation set with an 80/20 split for use with the CNN;the other three methods are unsupervised.All methods reach a similar accuracy between 93%-94%. More details about theperformance of the models are shown in Fig. 5.11.153Figure 5.11: ROC curve (Left) and prediction accuracy (Right) for different similarity measures based on 6019 image pairs from the TableauPublic collection. The peak performance for accurately predicting similarity after setting a specific similarity threshold (horizontal axis, Right)ismost relevant for our purpose of determining similar and dissimilar images. We decided to use histogramof gradients because themeasure isunsupervised and reached a high accuracy for a range of similarity thresholds and our example dataset.154We implemented two visual inspectors (I3 and I4) to qualitatively analyze the results.I3 shows random samples of negatively predicted image pairs (false positives, falsenegatives). T_his tool allowed us to better understand when amodel diverges fromthe human annotator. I4 shows a similarity matrix of all images within a bundle tohelp us analyze the magnitude of values apart from the binary classification.T_hequantitative results already showed that allmethodsperformedsimilarly, exceptSSIM has a slightly lower accuracy. T_he visual inspection revealed that all modelsdetect nearly identical images well but grapple with larger differences that thehuman annotator considered irrelevant, such as slightly different layouts, or achart legend that is illegible when the image is rescaled to a thumbnail size.T_he CNN contains only two convolutional layers and was trained on 4815 pairsand learned low-level features. It might learn better high-level concepts by using adeeper architecture anda sufficiently large training set. However, inpractice, ampledata to train a CNN for diverse collections may not be available. For our objectiveof filtering only very similar and identical images within a bundle, conventionalunsupervised methods, such as HoC and HoG, are adequate and can be directlyapplied to new visualization collections. When similar images are not detectedwith less complex models, the MMR algorithmwill ensure that these images arefar apart in the final ranking.CumulativeReduction of ImagesFig. 5.12 shows the distribution of the number of original images per bundle. T_hemedian number of images per bundle is 3 for the Tableau Public collection and 2for the Observable collection.155Figure 5.12:Number of images per bundle without applying any filters.Wefilter empty images, images of views that are embedded in dashboards, and verysimilar images. T_he goal is to eliminate non-informative content and redundanciesto increase the quality of visualization snippets. Fig. 5.13 and Fig. 5.14 show thecumulative reduction of images after applying these filters.Figure 5.13: Cumulative reduction of images after applying the filtering steps on each bundle.We distinguish between dashboard and single-view images in the Tableau Public collection.Figure 5.14: Cumulative reduction of images after applying the filtering steps on each bundle.We consider all images as single views in the Observable collection.ImageRanking andCollagesWe created a visual inspector (I6) that reveals which images are getting filtered atwhich step, and how the original ranking changes after diversifying it based onchart types and image similarity. T_his type of qualitative assessment was crucial in156understanding if combining various building blocks leads to the desired outcomeor if important information gets lost during the image compression.Fig. 5.15 is a screenshot of I6 showing the images of 6 example bundles on a greybackground. Colored crosses indicate why images are filtered. T_he first 2 bundlesare both reduced to a single image because of empty and embedded images. Ingeneral, we can see that most images are filtered because they are embedded in adashboard (blue cross). T_he visual inspection also revealed that, withmore complexdashboards, embedded views may end up very small on screen so keeping them asadditional separate images might be helpful. Nevertheless, we made the trade-offto filter all embedded images because the median number of views per dashboardis only 2 and therefore sufficient details are visible in most cases.Figure 5.15: Inspector I6 to analyze the ranking andwhich images are filtered( incomplete/empty, embedded, and similar views).We analyzed the number of images within a collage in relation to overflow imagesthat are discarded or moved to additional collages in the carousel. After applyingall filters, 87% of all bundles can be fully represented with collages of size 4. 94%of bundles contain 6 or fewer images. Instead of choosing a standard collage size,snippet designers only must specify the minimum image size and all images are157resized or divided into multiple collages accordingly. Hence, the maximal frag-mentation of a collage depends on the overall snippet size, which we will discussbelow.T_he grid packing algorithm displays as many images as possible and resizes themautomatically. Fig. 5.16 shows the averagenumber of collages per bundlewith a vary-ing number of images per collage. If a bundle has more than one collage, we createan image carousel that lets users scroll through additional collages interactively.Figure 5.16: Analyzing the average number of image collages for each bundle by varying themaximumnumber of images per collage.5.5.3 Page andSnippet LayoutWe created the inspectors I7 to analyze different page and snippet layouts, and I8to compare our layouts with existing snippet designs from the commercial toolsTableau, Observable, Power BI, and Qlik. Although these tools prioritize a uniformgrid or list layout, we argue other layouts canbe superior in some scenarios. Fig. 5.17showsmultiple automatically generated snippets of the same Observable notebook.Our pipeline supports trade-offs between compactness and information capacity,where the complexity of a bundle (C-Complexity) is directly linked to the amount ofpixels used (C-Pixels). For example, snippets in a strip layout expand based on theircontent, so bundles with rich visual and textual content take upmore space thanbundles containing only a bar chart. T_hese complexity differences are not apparentin a grid layout. T_he preview layout, with an example snippet shown in Fig. 5.17-6,goes one step further to serve as a minimum-loss representation of a bundle inreturn for larger pixel budget. T_he preview layout can, for instance, be used in a158modal window to provide additional details instead of a time-consuming openingof the fully interactive version of a bundle. T_his use case is not supported in any ofthe reviewed tools, but our results show its promise.Figure 5.17: Automatically generated snippets of the same Observable notebook [13]. (1) Verti-cal snippet layout with image collage of six images, and image carousel to scroll through addi-tional content; (2) Existing visualization snippets in commercial products are limited toa singlethumbnail; (3) Horizontal snippet layout; (4) Small minimum image size allows more imagesto fit in a collage; (5) Table layout; (6) Large preview snippetminimizes information loss.Fig. 5.18 shows a comparison between a snippet of a Tableau workbook generatedwith our pipeline and three alternative designs used in commercial tools. T_hisexample is indicative for many other bundles we have seen in our collection. Itconsists ofdashboardsandsingle views, and thebundle title is vague. T_he three toolsdisplay only one image and a title (see Fig. 5.18-2-4). In contrast, the VizSnippets159pipeline can create an image collage with four images and augment the snippetwith keywords to expose the underlying topic (see Fig. 5.18-1). We see that keywordscan be highly valuable in these cases where the bundle title is non-informative, andour design requires only a minimal increase of the overall snippet size or a slightlysmaller thumbnail. When a user hovers on the snippet, arrows are superimposedand they can scroll through additional image content.Figure 5.18: Snippet comparison: (1) VizSnippets pipeline, (2) Tableau Online, (3) Tableau Pub-lic, and (4) Qlik Sense.5.5.4 Designer FeedbackWe obtained feedback from 6 potential snippet designers within Tableau productand research groups on the final version of the VizSnippets pipeline during aninformal one-hour session.T_hey confirmed the issues related to existing visualization snippets designs andemphasized the importance for increasing the information quality and density.T_he interviewees were highly interested in the overall ability to showmore relevantinformation to users, and saw promise in many specific aspects of the pipeline,including the large preview capability to showmore results without loading thefully-interactive version and the flexibility of generating different types of layoutsfor different tasks.T_his work will inform new snippet designs and they could see using the underlyingalgorithms for future products, although they said they were unlikely to use everysupported layout. In particular, they thought some multi-image layouts may be160too visually busy when showing search results in a consumer product, although theinformation density might be much better.To address this specific issue, designers could include only a few images in onecollage. T_his strategy can result in several collages which are then combined into animage carousel that lets users scroll through additional content without leaving thesnippet. An alternative approach is to show only one image per snippet and replaceit with other images when users point their cursor onto it, similar to interactivevideo thumbnails [283].5.6 Visual Inspection ThroughRandomSamplingWe now reflect on our approach to creating multiple lightweight visual inspectors,to cast it as a general method.5.6.1 Inspectors for IterativeDevelopmentInspectors can support the common trial and error strategy for model or algorithmdevelopment wheremodels and parameters are initially chosen through some com-bination of experience, reading related work, and ideally by running quantitativeanalyses with ground truth data. T_hese algorithmic choices are then iterativelyrefined until results are satisfactory or improvement stagnates.A concrete example is the detection of similar images. Using an algorithm basedon the histogram of gradients, we manually assigned similarity labels to exampleimage pairs, and then computed statistical measures of the performance of thebinary classifier. Precision, recall, and other measures were useful instruments indeterminingmodel appropriateness. However, model results invariably led us tofollow-up questions and error analyses that were amenable to visual inspection.For example, in which cases does the model prediction diverge from the humanannotator? Do those images share any special characteristics? T_hese insights canin turn be used to tweakmodel parameters, such as the size of the cell for whichhistograms are created.1615.6.2 Inspector CharacteristicsInspectors can facilitatediscussions and inquiry aboutneedswithproject stakehold-ers, andmost centrally serve to internally probe and validate algorithmic choicesand parameter settings. We identify four abstract tasks that can be supported byinspectors: 1) see and compare results after applying a model or algorithm on realexamples, 2) see how results change with varying thresholds, 3) see intermediatesteps instead of the final result, and 4) see negative examples (e.g., false positives,false negatives).T_he process, illustrated in Fig. 5.19, is driven by two key ideas. First, randomly loadsamples from a substantial data corpus for item-level inspection to avoid cherry-picking. Second, spin up many lightweight inspectors targeted at specific analysistasks or algorithms, instead of building one feature-rich system, to ensure rapidprototyping and quick insights.Figure 5.19: A conceptual model for extensive, item-level visual inspection through randomsampling.All inspectors shouldhave abutton to loadnewsamples, tohelp the analyst graduallyprogress from a partial to a more complete understanding of model or algorithmbehavior, to judge its effectiveness and limitations. Random sampling guaranteesthat analysts scrutinize a diverse range of results. Inspectors may have controlwidgets, for example to adjust algorithmparameters or to constrain the samples to aspecific data range. Visual inspection through random sampling actively mitigatesthe risk of confirmation bias, but does not provide immunity to it. Analystsmay stillbe enticed to draw premature conclusions after seeing only a small set of examples.162T_he scope of each inspector should be narrowly constrained to support ultra rapidprototyping. T_hese tools are a means to an end, and, preferably, the implementa-tion should not take longer than a few hours or days. T_he design and functionalityof the visual inspectors can vary substantially, but we aimed to reuse componentsin different inspectors and a common database. We implemented web-based in-spectors in JavaScript and D3, with a Python back-end, to provide a maximumof flexibility and because we were familiar with the development stack. Ideally,inspectors provide nearly instantaneous feedback during the interaction. T_hus,model predictions or algorithm output is either preprocessed on all available dataor computed live on the current sample.5.6.3 Visual Inspector RelatedWorkNumerous visual analytics approaches related tomodel explanation [61, 255],modelconstruction [183, 191], andmodel evaluation [8, 54, 138, 302] have been put forward.Our concern here is not to illuminate the model internals [280] or steer the genera-tion of newmodels [43]. Instead, we are using visual methods for the evaluationand refinement of computational models that range from simple algorithms tomore complexmachine learningmodels. T_he goal is insights onmodel effectivenessand limitations that can subsequently inform new parameter andmodel choices,or data iteration [108]. Others have noted that the implementation of feature-richsystems tailored to specific models [158] is not always feasible, particularly whenthe goal is to mainly apply existing models to new domains.We suggest the extensive visual inspection with lightweight tools and real-worlddata as an alternative approach. Similar to the visual parameter space analysis [237],our approachalsodependsona samplingmechanism, butwe randomly sampledatafor item-level inspection to see a model in action while avoiding confirmation bias,instead of systematically sampling the parameter space. Closely related is the visualdiagnosis of binary classifiers byKrause [138] that includes presentingmodel resultsbased on single instances. However, their focus is on instance-level explanationsfor binary classifications while we discuss a high-level visual-inspection strategythat is model agnostic.163Kachkaev et al. [120] provide a single example showing the value of visual inspec-tion with real-world data, in their case survey results; we advocate injecting suchinspectors at many places within a computational system to assess performanceand set parameters.5.7 Discussion and FutureWorkWecreated and validated theVizSnippets computational pipeline primarily throughquantitative and qualitative analysis of two visualization collections, providing sub-stantial evidence that we achieved our goal of dramatically increased informationdensity within snippets. Follow-up benchmarks on additional collections wouldstrengthen our claims of generality. User studies focused on specific use cases,ideally informed by the nuances of specific visualization platform characteristicsand requirements, would shed further light on efficacy.We chose extractive techniques to summarize the content of bundles verbatim. Incontrast, abstractive techniques, such as latentDirichlet allocation (LDA) [28], derivegeneral topics from a corpus and then use topic keywords to describe a document.T_hese high-level concepts may be more understandable to users but the modelsneed to be trained on large corpora of text, and even then, some bundles will getfalsely allocated to a topic and represented by irrelevant keywords. Approachesrelated tonatural languagegeneration [194] also rely onmassive amountsof trainingdata and are not directly applicable to visualization snippets due to the limitedtext fragments. For example, Chen [49] generated abstractive summaries of webpages but ignored those with less than 100 words to ensure a sufficient basis forsummarization. In comparison, our sample Tableau workbooks contain 37 uniquewords on average.In terms of visual content, all our discussed methods are based on a set of screen-shots, with the aim to accurately represent the visual style, but views and dash-boards are scaled down significantly and may become illegible. One option is toredraw smaller versions of charts, which poses its own research problem, becauseof exotic chart types and inaccessible data and chart specifications. Another alter-164native is to analyze the bundle content and summarize it with abstract iconographyor graphics [123].T_he VizCommender study, described in Chapter 4, inspired this project when werealized that participants were not able to accurately judge the relevance of pro-vided recommendations. T_he VizSnippets pipeline and design framework serveas a technical and conceptual foundation to create more informative snippets. Aninteresting direction for future research would be to conduct a human subjectsstudy that combines the VizCommender recommendation model with the newsnippets.5.8 SummaryWe present a computational pipeline to compress visualization bundles, such asTableau workbooks or Observable notebooks, into representative snippets. T_hesesnippets allow users to judge the relevance of bundles for a specific task withoutopening and inspecting them one by one.To gain a better understanding of requirements and challenges, we engaged withTableau teams of potential snippet designers aware of end-user pain points, andreviewed existing snippets across a broad set of five different tools. We found thatexisting designs suffered from poor information density, substantially impairingthe ability of users to make the expected relevance judgements. Snippets have animmense impact on the consumption of visualization collections, and designingthemwith higher information content can significantly improve the workflow ofuse cases including the presentation of search results, navigation through facetedbrowsing, and recommendation suggestions.T_he proposed VizSnippets computational pipeline is responsive to a specified pixelbudget, and can be adapted to many use cases and form factors through a suite ofcontrollable layout families. While the effectiveness of our pipeline was primarilyvalidated through Tableau and Observable bundles, we argue that the suggestedcompression techniques can be applied to many other visualization collections dueto their similar characteristics.165T_he development and evaluation of the pipeline was informed through extensivevisual inspection on random samples of real-world data. We reflect on this methodand provide guidance for other researchers.1666 Data-First VisualizationDesignStudies12We introduce the notion of a data-first design study which is triggered by the ac-quisition of real-world data instead of specific stakeholder analysis questions. Wepropose an adaptation of the design study methodology framework to providepractical guidance and to aid transferability to other data-first design processes.We discuss opportunities and risks by reflecting on two of our own data-first designstudies. We review 64 previous design studies and identify 16 of them as probablecases with characteristics that may indicate a data-first design process in action.6.1 MotivationDesign studies are frequently used to conduct and report problem-driven visual-ization research with domain experts. A design study is characterized by its highlyiterative nature with a tight interplay between the analysis and abstraction of stake-holder needs, the transformation of domain-specific into domain-agnostic data,and the visualization design.1T_his chapter is a slightly modified version of our paper Data-First Visualization Design Studies byMichaelOppermannandTamaraMunzner in Proc.Workshop onEvaluationandBeyond: MethodologicalApproaches for Visualization (BELIV), pp. 74-80, 2020. https://doi.org/10.1109/BELIV51497.2020.000162In our BELIV publication [200], we originally referred to previous design studies that indicatecharacteristics of a data-first ordering as edge cases. We now believe the term probable cases is a moreaccurate description and use it throughout this chapter.167T_he design study methodology (DSM) by Sedlmair et al. [236] provides method-ological guidance on how to conduct this type of applied visualization research,proposing a nine-stage sequence: learn, winnow, cast, discover, design, implement,deploy, reflect, and write.T_he DSM focuses on connecting with collaborators in the form of domain-expertstakeholders at an early winnow stage with suggested criteria for narrowing downthe set of potential collaborators, identifying their roles in the cast stage, and thenundertaking a discover stage focused on problem characterization and task abstrac-tion with the chosen stakeholder. T_he selection of an appropriate stakeholder dic-tates the relevant data, which is then abstracted from domain-specific to domain-agnostic form in the design stage, in parallel with iteratively creating appropriatevisual idioms. We now characterize this DSM approach as a stakeholder-firstordering.Wefind that this framework falls short of capturing thenature of design studies thatare primarily initiated by acquiring an interesting real-world dataset rather thanselecting a specific stakeholder; we call these data-first design studies. In a data-first design study, the early selection of the data constrains appropriate choicesfor stakeholders. Data abstraction is carried out early, and followed by problemcharacterization and abstraction for multiple potential stakeholders. Stakeholdersare chosen based on whether their tasks can be supported by the selected datasource.T_hese non-traditional data-first design studies have not been explicitly reported oranalyzed in the visualization literature. However, in retrospect we realize that theyoccur frequently in different variations, especially in class projects or visualizationdesign competitions [69, 91, 139] but also in research contexts such as our own.We suspect that every visualization researcher has experienced the situation ofstumbling across a dataset that aroused curiosity. In some cases, these serendip-itous discoveries lead to the exploration of interesting design alternatives wherethe results andmethods are shared on social media or documented in blog posts.In other cases, especially with more complex datasets or involving unique datacharacteristics, this process can lead to novel visualization contributions [234].168In this chapter, we reflect on two of our own design studies that we now consider tobe examples of a data-first approach, the Bike Sharing Atlas [203] andOcupado [201],that guided our initial thinking about this alternative methodological approachfor design studies. We follow up with a review of 64 previous design studies wherewe note 16 probable cases that may implicitly indicate a data-first ordering. T_hefull list of 64 studies is provided in Appendix E, and the 16 identified examples areprovided in Table 6.1. While none of these examples can be labelled as data-firstdesign studies with absolute certainty–in this retrospective interpretation andwithout interviewing the authors–25% probable cases suggest a data-first orderingis not at all unusual, although it is often underdocumented.We contribute a first characterization of data-first design studies, namely thosetriggered by data instead of specific stakeholder questions. We propose an adaptedversion of the nine-stage design studymethodology frameworkwherewe introducea new acquire stage, move and rename discover to elicit, and adjust other stagesaccordingly. In addition, we present a review of previous design studies and discussopportunities and risks of these non-traditional design studies. Our goal is toprovide practical guidance and aid transferability of data-first design processes.6.2 RelatedWorkWe discuss the extensive previous work on design studies and visualization processmodels to contextualize our proposed data-first approach. Literature on designstudies emphasizes the need for real-world data and an early data understanding,but a stakeholder-first ordering is generally presumed.In the original DSM [236] the order and timing of the data acquisition is not ad-dressed head on; the closest guidance are the questions “Does real data exist, is itenough, and can I have it?" related to the pitfall of “no real data available (yet)" that isnoted in the winnowing stage where potential stakeholders are being assessed.T_hedesign study litemethodology [257] is an expedited version to fitwithin the timeframe of undergraduate and graduate visualization courses. Recruiting communitypartners is a precondition in this framework. Although highly desirable, access to169many stakeholders with course-relevant visualization needs is not always feasible.In larger class settings, students may not have access to stakeholders but couldbegin with data. T_he data-first methodology would provide a path for some or allsuch student groups to engage with domain experts at a later stage.Munzner’s nested model for visualization design [192] and different types of pro-cess models assume a user with a specific domain problem at the start, whomayact as a data provider as well. T_he activity-centered framework [166] suggests tomandatorily request access to real data, or a data sample of sufficient size, early inthe process. McKenna et al. [172] proposed a framework that links actions directlyto the nested model with the four activities, understand, ideate, make, and deploy,whereby a project can start with any activity. T_he understand activity implies thedata acquisition and task elicitation of the proposed data-first approach.McCurdy et al. [170] suggest that a real-world domain problem is either expressedby domain experts or discovered by design researchers, whereas the latter is closelyrelated to our framework adaptation. T_he human-centered design process by Lloydand Dykes [161] highlights the effectiveness of real and interesting data to engageparticipants in requirement elicitation, which is a key objective in data-first designstudies.Crisan andMunzner [65] proposed a conceptual framework of data reconnaissanceand task wrangling to explore unfamiliar data landscapes. T_his framework putsdata first but focuses on domain experts and how they refine data to support aspecific analysis goal.Goodwin et al. [92] presented a creative design case study in the energy domainthat explores possible visualization usage scenarios for a smart meter dataset,which is closely related to our suggestedworkflowbut discusses primarily creativitytechniques and assumes a set of users at the beginning.In the context of evaluating visualization tools in large companies, Sedlmair etal. [235] distinguish between employee-pull and researcher-push solutions. Althoughthe fundamental concept of pushing ideas from an outside perspective is similar,170these authors regard it as advertisinga specific tool. We insteadpropose to advertisedata and the use of visualizations to solve a domain problem together.6.3 Case StudiesWe introduce two of our own previously published design studies that we nowconsider to be examples of data-first design studies, before we generalize the un-derlyingmethod ina refinedDSMframework inSec. 6.4 anddiscuss its implicationsin Sec. 6.5.T_he Bike Sharing Atlas combines and visualizes distributed data from several hun-dred bike-sharing networks worldwide [203]. T_his project was myMaster’s thesisand is not described in detail in this dissertation, but inspired the concept of data-first design studies. We show that the data produced by bike sharing networksreveal interesting insights, not only into patterns of bicycle usage, but also theunderlying spatiotemporal dynamics of a city [89]. By working with users fromdifferent domains including a bike sharing operator, a city planner, and urbansociologists, we illustrated how interactive visualization can help to open the datathat is produced in our cities to a wider audience.Ocupado is a set of visual decision-support tools centered around occupancy datafor stakeholders in facilities management and planning [201]. We takeWiFi devicecounts as a proxy for human presence, showing how to leverage data that hadpreviously only been used for automation in building control systems in many newways. We interviewed potential stakeholders from different domains and evaluatedthe conformity between tasks and data affordances. In a highly iterative processand with extensive feedback from a set of core stakeholders, we developed anddeployed Ocupado, which was eventually adopted by our industry partner.T_hese two projects were different than regular design studies in the way that weengaged with stakeholders after acquiring the data.We had worked with a bike sharing operator early on to analyze data from Vienna,but the discovery and collection of data frommany station-based bike sharing net-works worldwide [58] triggered additional use cases and aroused interest from new171stakeholders working in different domains. T_he Ocupado project began through acollaboration with a startup3 that estimates occupancy based onWiFi signals forbuilding automation usage, and acted as a data provider. We conjectured that withsuitable visualizations, this data could be actively useful to many additional usergroups for decision-making and resource management, and explicitly sought outthese potential stakeholders. T_he UBC campus served as a test environment forthe data collection and allowed us to present data to potential on-campus stake-holders that is directly relevant to them. We presented example data to externalstakeholders where we had no direct data access yet.In both design studieswe had to considerwhether to embed additional data sourcesinto the mix to support potential stakeholder tasks. For example, for the bike shar-ingproject,we included elevationprofiles because the elevationdifferences betweenstations are an essential factor for the functioning of a bike sharing system, andthe combined data can provide relevant insights. When we worked with custodialmanagers during the Ocupado project, we were asked to add work schedules thatcan be compared to occupancy statistics. We ultimately decided against an integra-tion because these schedules were not relevant for other stakeholders and it wasunclear if and how they facilitate core tasks related to occupancy analysis.6.4 Refined andExtended FrameworkWe propose an alternative, extended version of the DSM framework, generalizingour experience as a guide for future data-first design studies. T_he data-first designstudy stages are shown in Fig. 6.1. Onemajor change is to add an early acquire stagefocused on both obtaining and abstracting data. T_he second change is to movethe discover stage immediately afterwards and rename it elicit to reflect its task-based emphasis: obtaining information about and abstracting tasks. We keep thesame ordering and general function of the winnow and cast stages, though winnowis heavily dependent on the initially acquired data, and nuances differ in the caststage. T_he focus of the design stage is narrowed to cover only idioms, since thedata abstraction occurs earlier in the new acquire phase. T_he subsequent stages3Sensible Building Science: http://sensiblebuildingscience.com172implement and deploy are unchanged, as is the initial learn stage. Reflect and writeare adjusted to reflect and report also on the peculiarities of the data-first process.As with the original DSM framework, jumping backwards to previous stages iscommon and necessary, in particular when adding new stakeholders.Figure 6.1: Refined and extended framework for data-first design studies. A new acquire stageis added for both obtaining and abstracting data. Discover is moved and renamed to elicit, toemphasize the elicitation of tasks from potential stakeholders. Winnow focuses on analyzingthematch between the abstract tasks of these stakeholders and the data abstraction. Nuancesdiffer in the cast and design stages to incorporate the specific characteristics of the data-firstapproach.We use the term visualizer to mean either a visualization researcher or practitioner.T_he adapted framework contains 10 stages:Learn: T_his stage remains unchanged from the original because it concerns thevisualizationknowledge ingeneral and independentof the specificdomainproblem.A solid understanding of the visualization literature is crucial for every design study.Acquire data: A visualizer encounters, collects, generates, or obtains access to adataset. T_he visualizer translates the data description into domain-independentlanguage and begins to develop a data abstraction. Data sketches and descriptivestatistics can provide first insights and guide the development of initial hypothesesabout its underlying semantics.T_hedata-firstDSM framework emphasizes iteration, as does the original DSM, andthus takes into account the idea that further data abstraction refinement will surelyoccur after task abstractions are identified. Relevant questions:What type of data amIworkingwithandwhatare theunderlying semantics? Are thereanydataquality challenges173and is long-term access guaranteed? What is special about this data and who would benefitof seeing and exploring it?Elicit tasks: T_he visualizer seeks out multiple potential stakeholders and elicitsdomain-specific tasks from them that might be relevant for the chosen data ab-straction. T_he visualizer both explains the initial data abstraction and learns aboutunsolved stakeholder needs.T_he visualizer translates the tasks into domain-independent language and con-siders what data is required to address them. Is there amatch between given data andstakeholder need? Is more or other data required to answer domain-specific questions, and ifso, how difficult is it to acquire and integrate?We combine task elicitation and stakeholder outreach into one stage to emphasizethat these steps are highly connected which is in contrast to the original DSM.Winnow:T_hegoal of the data-first DSMwinnowing stage is to assess and prioritizethe set of potential stakeholders according to the match between their abstracttasks and the data abstraction. T_his process may involve multiple rounds of taskand data abstraction, as the understanding of the stakeholder needs and of thedataset semantics evolves frompartial tomore complete. Wediscuss these data andtask perspectives in Sec. 6.5.1. How frequent are their data-relevant tasks? How centralare these tasks to the stakeholder’s primary mission? Howmany people in the organizationdeal with these tasks?Cast: T_he cast stage is similar to the original DSM with the goal to identify col-laborator roles; here we note some previously proposed roles that have particularramifications in the data-first case.We named the promoter role in the Ocupado chapter [201], and call it out here asessential in a data-first design study. T_he visualizer needs to serve in this role toreach out to potential stakeholders to present data and visualization opportunities.In addition, partners or stakeholders may act as promoters themselves when theyundertake technical evangelism to other potential users about the benefits of avisualization system. When this evangelism includes demos of software prototypes,174as it often does, the promoter may be involved as an intermediary in the taskelicitation andwinnowing stages. In this case, the promoters can serve as a conduitto relay information back to the visualizers about stakeholder needs and how theyalign with prototype capabilities. We also identify an example of this role occurringwithin the Overview project [33] by Brehmer et al.T_he data producer and data consumer roles were first proposed by Kerzner et al. [121]within stakeholder-first design studies. T_hese two roles are most certainly held bydifferent people indata-first studies, and therefore require particular consideration.A data provider may need to ensure long-term access to and the functioning of thedata pipeline. T_he visualizer may sometimes take on the data provider role in alater project stage, even if that role was played by somebody else at an early stage.Design: T_he design stage focuses only on visual idioms, whereas it also includesdata abstraction work in the original DSM; in our framework, the latter was carriedout earlier, in the acquire stage.Implement and deploy: Remain unchanged from the original DSM.Reflect andwrite: Data-first design studies invite specific considerations duringthe reflection and writing stages. Howmany potential stakeholders and domains havebeen considered, and ultimately chosen?What are the differences and commonalities amongthe stakeholder tasks? Did the original data suffice or was there a need to integrate data fromadditional sources?6.5 Reflections andRecommendationsWe reflect on the challenges of the winnowing stage, the opportunities of the data-first approach, and its risks.6.5.1 Winnowing Tasks andRefiningDataT_hekey challenge in this non-traditional design study approach is to identify whichstakeholders are appropriate, and thus to identifywhether there is amatch between175potential stakeholder tasks and the available data. To do so, the visualizer mustassess the relevance of the identified task for the given data, and also considerwhether to integrate secondarydata sources. T_hese considerations takeplaceduringthe winnow stage.Fig. 6.2a illustrates a simple conceptual model for reasoning about the conformitybetween data and tasks as aligned axes through a continuumwith peripheral on theleft, and core on the right. At the core are the current data and the high-relevancetasks, with other data and low-relevance tasks being relegated to the periphery.Fig. 6.2b shows several tasks assessed according to the initial data selection arisingfrom the elicit stage, with some determined to bemismatches and some consideredas matches.Fig. 6.2c illustrates the tradeoffs of incorporating additional datasets into thescope of a project during the winnow stage. A possible benefit of adding secondarydata sources could be to support more tasks, and thus to expand the set of eligiblestakeholders. However, the visualizer also needs to ensure that the problem spaceunder consideration maintains sufficient cohesion to have clear system goals anddesign targets. Cohesion can be reasoned about as constraints on the number andvariety of stakeholders and of tasks.176Figure6.2: Simple conceptualmodel illustrating thechallengesoffindingan intersectionof rel-evant tasks and data while keeping the problem space constrained. (a) The data and task axesboth range fromperipheral to core. (b) In the elicit stage, the visualizer examines thematch be-tween the initially acquired abstracted data and stakeholder tasks. (c) In thewinnow stage, thevisualizer can assess the benefits and risks of expanding the scope to additional data to expandthe set of tasks and thus stakeholders.6.5.2 OpportunitiesWe have identified several opportunities of a data-first approach.(1) Capitalizes on possibilities that do not fit traditional DSM process. We con-sider data-first design studies as an alternative way to approach visualizationresearch collaborations, where interesting research questions can emerge throughconversations that unfold later in this process than in the traditional one. Analo-gous to market pull versus technology push dynamics in product innovation strate-gies [20], the originally acquired data is used to push ideas from an outside per-177spective instead of pulling analysis questions from a domain expert. For example,using existingWiFi access points to sense occupancy patterns in hundreds of roomswithout installing additional hardware yielded a completely new opportunity thatOcupado stakeholders had not envisioned before.Launching a data-first design study without a specific stakeholder collaborationcommitment at the beginning of a project is a possible method to approach long-distance work and to engage with domain experts in imperfect circumstances,such as under COVID-19 restrictions. T_he learn and acquire stages are conductedindependently by the visualizer. When reaching out to potential stakeholders, firstideas for solving a specific problem are presented and guide the discussion, insteadof conducting a fully open-ended brainstorming session which can be challengingusing only digital communication tools.(2) Allows very early data sketches and technology probes with real-world data.Launching a design study project without access to real data is a common pit-fall [236]. Synthetic or toy data often results in additional project obstacles anddelays [121, 284], andmay even lead down the wrong path of data abstraction andvisualization design. Access to real-world datasets at the beginning of the projectallows the creation of data sketches [161] and technology probes [112], both to inter-nally guide the development of hypotheses and possible usage scenarios, and toexternally inquire needs and desires of target user groups.(3) Supports gradual expansion of stakeholder set. Multi-channel engagementwith several stakeholders fosters serendipity and enables more impactful visual-ization designs, as demonstrated byWood et al. [292] and confirmed in our owndata-first design studies [201, 203], while providing breadth to a convincing evalu-ation [234]. However, addressing the needs of multiple stakeholders involves therisk of conflicting tasks and an ever-expanding project scope. In data-first designstudies, stakeholders are selected by evaluating the conformity between tasks anddata capabilities. Although additional data could be added, the primary data sourceserves as a guide to navigate the due diligence with stakeholders and to delimit theproblem space.178(4) Encourages progression of course and side projects into publishable results.Data-first design studies may start off during visualization courses or when explor-ing and visualizing data out of curiosity. Characteristics such as complex datasets,unique characteristics, or combinations of different data sources that address high-relevance stakeholder tasksmay suggest a fruitful direction to expand initial effortsto result in interesting and novel knowledge contributions.Previous work showed that it is challenging to teach and conduct traditional designstudies within a regular semester [257], due to the long timelines typically requiredto execute such studies. Some projects may require more time to understand thedomain background and to elicit tasks leaving not enough time for the ideationand implementation stages.T_his issue can be mitigated if the instructor selects a set of stakeholders in advance,as proposed in the design study lite [257], but is not always feasible in practice.In a data-first design study, the data abstraction, ideation, and initial prototyp-ing is front-loaded and thereby presents an alternative model that includes mostcomponents that are typically required in a visualization course project.6.5.3 Risks(1) Hypothesized tasks. In the data-first approach, visualizers begin explorationimmediately after acquiring the initial data, with conjectures about potential stake-holders and usage scenarios. We consider the delayed involvement of potentialdomain experts to be a major risk, because the initial implementation is based onhypothesized rather than verified tasks. Tomitigate this risk, technology probesshould be limited in functionality and open-ended with respect to use to avoidwasted effort [112].(2) Hammer looking for a nail. T_he danger of having data at our fingertips is toconverge on a specific idea prematurely and develop a full-fledged prototype with-out an in-depth engagement of stakeholders and the consideration of the broaderdesign space. In a design study, stakeholder participation needs to be maximized179and many design alternatives must be considered, otherwise stakeholders aremarginalized to design verifiers.(3) No actual users. In the worst case, the search for potential stakeholders maynot succeed. In this case, other types of research contributions such as a novelvisualization technique or algorithmmay be possible, but a successful design studywould not be achievable because it requires working with real users to solve theirreal-world problems. Amajor pitfall would be attempting to validate design deci-sions prematurely based on speculative tasks; care should be taken to avoid thismistake. A less extreme situation is when significant time is spent on promotion,leaving less energy available to direct towards research questions. T_he pitfall in thiscase would be a design study that requires longer than planned before the work issufficiently complete to publish.(4)Mismatchbetweendataandstakeholder tasks. A clearmatch betweendata andstakeholder tasks is not always immediately obvious and a short-cut to data andtask abstractions can lead to a premature commitment to specific stakeholders. Insome cases, proxy measures that stand in for a variable of interest entail a range oflimitations. In other cases, the given data is out of date or needs to be augmentedwith other data sources. A misinterpretation can lead to project detours or to afailed design study.(5) Joy of discovery versus actual needs. T_he presentation of data through visual-izations often sparks interest, particularly when potential stakeholders see theirown data or data similar to theirs. We observed many meetings where domainexperts enthusiastically began to conjecture about the many different ways theymight be able to use the data. T_his process is encouraged to assess the conformitybetween the given data and relevant tasks. However, at these initial meetings ideasare often proposed based on limited information; the challenge for the researcheris to discriminate between initial enthusiasm and substantive user needs. A centralconcern is to understand whether the visualization addresses a core task of thetarget user group, to avoid the pitfall of a task that is only of peripheral relevance.Fellow tool builders and gatekeepers [236], who frequently act as intermediaries in180winnow-stage meetings, might misunderstand specific needs of front-line analysts.T_he pitfall to avoid is premature selection of stakeholders whose actual tasks donot in fact align with the data, who would then abandon a visualization solutionquickly.(6) Data promises. Decoupling the data producer from the consumer may increasethe responsibility for the visualizer who needs to ensure the functioning of the datapipeline throughout all stages of the design study—as opposed to stakeholder-firstdesign studies where data is typically provided by or collected together with thedomain expert. A risk is therefore to have a strong collaboration with stakeholdersbut not deliver on the initial data promises. For example, generating interest bypresenting data from one city, but not being able to follow through by collectingdata from the city that is most relevant to the stakeholder, could be a major barrierfor project success that is independent of the quality of the proposed visualizationsolution.Although in many cases setting up a static database at the beginning is sufficient,for projects that rely on the availability of continuous data streams, external API orsystem level changes can jeopardize the success of a design study. To guaranteelong-term availability and avoid continuous engineering effort, it is imperative tooffload this responsibility.Stakeholders likely depend on data providers even when the research project con-cludes. Visualization researchers may, but do not necessarily, act as data providersin a data-first design study.(7) Closed design approach. Data iteration [108], including the integration of sec-ondary data sources in a visualization system, can be a crucial step to address aspecific domain problem. One major risk in data-first design studies is to keepthe focus too narrow on the initial data without considering the wider problemspace and ensuring extensibility and adaptability from an engineering and designperspective.1816.6 Reviewof PreviousDesign StudiesIn parallel with formalizing our proposal for a refined data-first design studymethodology framework, we conducted a literature review of a diverse set of designstudies. While we had our own direct experience of two projects that started with adataset rather than a specific stakeholder as an existence proof, the ratio of thesestudies in the previous visualization literature was unclear. We were interestedwhether data-first approaches have been explicitly or implicitly described previ-ously, what type of data they have used, and if there are any underlying commonthemes.In total, we selected 64 design studies that were published between 1999 and 2019(full list provided inAppendixE). Weused theworkbyLamet al. [143]who identified39 design studies as a seed and extended this list with additional examples fromSedlmair [234] and by reviewing previous conference proceedings. In addition,we sourced design studies from vispubdata [114] by searching for the term “designstudy" in titles and abstracts. T_hevastmajority of these design studies are publishedat InfoVis with a few examples from VAST and short papers.In our analysiswe identified 16 probable cases (25%) that indicate characteristics of adata-first design process in themotivation or process statements, listed in Table 6.1.An accurate allocation to a stakeholder-first or data-first ordering is not possiblebased on this kind of retrospective interpretation. None of the 16 cases are explicitlydescribed as a data-first design study. Nevertheless, these probable cases providerelevant insights and illustrate that data-first approaches are not at all unusual,although they are often underdocumented.182# Title 1. Author Year Venue Domain/data1 Cluster and Calendar based Visualization of Time Series Data VanWijk 1999 InfoVis Symposium Electricity consumption patterns2 GeneShelf: AWeb-based Visual Interface for Large Gene Expression Time-Series Data Repositories Kim 2009 TVCG Gene chip datasets3 code_swarm: A design study in organic software visualization Ogawa 2009 TVCG Code repositories4 GeneaQuilts: A System for Exploring Large Genealogies Bezerianos 2010 TVCG Genealogy databases5 T_he Streams of Our Lives: Visualizing Listening Histories in Context Baur 2010 TVCG Listening histories6 OpinionSeer: Interactive Visualization of Hotel Customer Feedback Wu 2010 TVCG Hotel customer reviews7 BallotMaps: Detecting Name Bias in Alphabetically Ordered Ballot Papers Wood 2011 TVCG Ballot paper8 SoccerStories: A Kick-off for Visual Soccer Analysis Perin 2013 TVCG Sports9 Understanding Interfirm Relationships in Business Ecosystems with Interactive Visualization Basole 2013 TVCG Interfirm relations10 Creative User-Centered Visualization Design for Energy Analysts andModelers Goodwin 2013 TVCG Electricity consumption11 TenniVis: Visualization for Tennis Match Analysis Polk 2014 TVCG Sports12 Moving beyond sequential design: Reflections on a rich multi-channel approach to data visualization Wood 2014 TVCG Bike sharing data13 Speculative Practices: Utilizing InfoVis to Explore Untapped Literary Collections Hinrichs 2015 TVCG Literary collection14 Visual Mementos: Reflecting Memories with Personal Data T_hudt 2015 TVCG Personal movement data15 Urban pulse: Capturing the rhythm of cities Miranda 2016 TVCG Flickr data (urban)16 Sabrina: Modeling and Visualization of Financial Data over Time with Incremental Domain Knowledge Arleo 2019 VIS Short Paper Financial transaction flowsTable6.1: Inour literature reviewof64design studies,we identified 16probable cases that indicate characteristics of adata-firstdesignprocess.183T_hese potential instances of data-first design studies span a broad range of topicsand data sources, such as music listening histories [19], code repositories [198],electricity consumption [92, 268], financial transaction flows [12], sports [212, 216],social media [184], hotel customer reviews [296], and genealogies [25]. T_he datais mostly acquired from publicly available databases but in some cases scraped orgenerated by researchers.For instance, Miranda et al. [184] collected Flickr activity and tweets to capturespatio-temporal patterns in a city. T_he authors illustrated how interactive visual-izations of this data can help experts specializing in different domains to betterunderstand people’s behaviour in public places. More specifically, they present casestudies that were carried out by architects, urban planners, and human behaviourexperts.T_he BallotMaps [290] design study used electoral data from an open data repositoryto identify name and ordering biases in ballot papers. External domain experts arenot apparent based on the publication but the study demonstrates with real-worldexamples how a visual approach can reveal significant biases that were not evidentfrom statistical analysis alone.Distinguishing between researchers being their own stakeholders and data-firstdesign studies is most difficult, as there is sometimes no clear separation. Re-searchers with extensive expertise in a specific domain problemmay be qualifiedto self-validate a visualization solution in an autobiographical design study, as wediscuss in Sec. 6.7.3. In these 16 probable cases it is often unclear if the researchersstumbled upon data because of a broad interest in a topic or if they had a particularanalysis question themselves. T_he level of domain expertise is sometimes indicated,for example, one of the authors of TenniVis [216] had 35 years of tennis playingexperience, but in most design studies it is not evident.While some authors [18, 212, 296] report on early interviews with domain expertsto elicit requirements, many design study projects include external stakeholdersrather late in the process, for example, to participate in a summative evaluationinstead of influencing the design process. We consider this late involvement oftarget users a major risk (see Sec. 6.5.3), particularly for data-first approaches184when real data is available and simple data sketches progress to more complexsolutions.While this review is an attempt to analyze many different types of design studiesfrom the last two decades, our selection is certainly incomplete as many otherdesign studies have been published. A further limitation is that our judgement isbased on limited information on the design process and additional investigationsare required to better understand implications of data-first approaches that werenot explicitly reported. Interviewing visualization researchers would be a greatopportunity for future work.6.7 DiscussionWediscuss the role of the researcher’s domain expertise, anddesign studieswithoutengaging in collaboration with external domain experts.6.7.1 Domain Familiarity andDataUnderstandabilityT_henature of data-first design studies requires a certain degree of domain familiar-ity and data understandability. T_hus, not all datasets are appropriate for data-firstdesign studies: the selection strongly depends on the personal background of the vi-sualizer as it pertains to the research context. Initial data should be self-explanatoryor well-documented, or the visualizer should have a strong personal interest.Visualizers acquire and investigate data on their own before engaging into conver-sations with potential stakeholders. While specific use cases might not be immedi-ately apparent and talks with different domain experts can lead to serendipitousfindings, visualizers need to be aware of problems faced by domain experts and fore-shadow potential tasks at the beginning of a project. We argue that visualizers needto be task-curious to advance this type of problem-driven research, which is method-ologically very different to traditional design studies where only stakeholders comeforward with tasks.A thorough understanding of the selected data and its limitations is essential.Although this understanding will typically grow over time, a basic knowledge at the185beginning is beneficial. Descriptive statistics and superficial explorations quicklyreach their limits. In both our data-first design studies, we recorded data basedon a live data stream. In the bike sharing project, although we recorded datafrom hundreds of cities, it followed a relatively simple and uniform scheme whichallowed us to detect data limitations without having a deep domain knowledge.In the Ocupado project, we worked closely with our industry partner and dataprovider to better understand data semantics and quality issues.6.7.2 DataRepurposingT_he data-first approach entices visualizers to use available data and apply it to newproblem contexts. For example, proxy variables may be used to answer questionswhen direct measurement is prohibitively expensive [187]. T_he repurposing ofdata [293] provides unique opportunities but entails a range of risks: data collectorsmay not have given consent for their data to be used in a particular way, the originaldata collection process may not be fully transparent and thus the data quality andunderlying assumptions can not be judged reliably, and a lack of correlation withthe variable of interest can ultimately entice analysts to draw wrong conclusions.Visualization researchers are responsible to determine if data is fit for the new useand need to understand all its ethical and analytical implications. As discussedin Sec. 6.5.1, one of the goals in the winnowing phase is to identify if the data isappropriate for a given task or if it would be better served with a different dataset.6.7.3 Autobiographical Design StudiesFollowing the terminology of Neustaedter et al. [195], we define the process of de-signing and evaluating visualizations through self-usage as autobiographical designstudies, that we distinguish from data-first design studies.Although the DSM paper does not use this terminology, which was introduced atroughly the same time as its publication, the idea of it is clearly articulated withinit: “While strong problem-driven work can result from situations where the sameperson holds both of these roles [visualization researchers and domain experts],we do not address this case further here" [236].186Autobiographical design draws on “extensive, genuine usage by those creating orbuilding the system" [195]. Visualization experts act as primary stakeholders anddesign iterations are based on their own experiences. T_his type of first-person re-search [76] usually emerges from an unmet personal need and allows tinkeringwithan idea in a research context. T_he researcher should have or quickly gain extensiveexpertise related to the domain problem in order to validate the effectiveness of avisualization.In contrast, data-first design studies follow the autobiographical design only at thevery beginning. Domain familiarity is important but the process largely focuses onexternal domain experts [288] as target users and their feedback shapes the designdecisions instead of insights from self-usage.6.8 SummaryWe contribute the first characterization of a data-first design study for appliedvisualization projects that begin with an interesting real-world dataset rather thanstakeholder selection. T_hemotivation behind this approach stems from our ownexperience in conducting two data-first design studies. Problem-driven researchmotivated and inspired by data enables opportunities that do not fit in the tradi-tional design study process, although it does introduce risks and tensions. Wepropose an adaptation of the design study methodology framework to providepractical guidance for researchers interested in this alternative way to approachresearch collaborations. A preliminary review of published design studies revealedthat roughly one quarter of them were probable cases which might have had animplicit data-first ordering. We conjecture that some of these researchers mayhave under-emphasized their true trajectory by bending their explanation closerto the previously articulated design study methodology framework. T_here may beeven cases that combine characteristics of both approaches, where some datasetsand some stakeholders are available and get expanded over time. We hope thiswork encourages researchers to be more explicit in documenting and reflecting onvarious flavors of design studies.1876.9 AddendumSince the publication of the manuscript in October 2020, at least two data-firstdesign studies have been carried out by other research labs. T_hey cited our publica-tion [200] and state they follow the processwe outlined. T_hefirst study byKinkeldeyet al. [128] is published in IEEE Computer Graphics and Applications and the secondstudy by Eulzer et al. [84] will be published in CGF (Proc. EuroVis). T_he discussion atBELIV 2020, inspired by our paper, and the immediate take-up by the visualiza-tion community yielded further evidence that data-first studies are not a singularphenomenon. It just has not been articulated before and there was no concreteguidance for researchers.1887 Discussion andConclusionIn this chapter, we reflect on themes, implications, and open research questionsthat span across individual research projects. T_he four key themes are summarizedin Table 7.1 below.Table 7.1: Reflections on four key themes of this dissertation.Within the previous chapters, we discussed the limitations and future directionsof each study individually. Chapter 6 is a reflection on data-first visualization de-sign studies that are initiated by real-world data instead of specific stakeholderquestions. While data-driven prototyping is a general approach that can be flex-ibly applied to different types of visualization projects, a data-first visualizationdesign study is a concrete instantiation and the proposed framework providesmethodological guidance for this process.1897.1 Reflecting onVisualizationPatchworkPrototypingIn this section, we reflect on the prototyping process from a technical perspective.When we collaborate with stakeholders or engage target users in an iterative pro-totyping process, there is often a tension: On one hand, prototypes should berealistic andmimic the system in a potential usage scenario, but on the other hand,prototypes should be produced quickly to get feedback as soon as possible beforespending too much engineering time [79]. T_he HCI literature distinguishes be-tween low-fidelity and high-fidelity prototyping to describe how closely a prototyperesembles the form and function of a final product [224].Previous work argues that low-fidelity prototypes, such as paper sketches or wire-frames, invite more criticism and a broader range of ideas from interviewees [145,263]. T_he roughness indicates being open to suggestion and change, compared tomore polished prototypes that may direct the attention to details in the appearance.T_his process of externalization through sketching can also serve as catalyst for newideas internally [277].Within the context of information visualization, however, it can be difficult tocommunicate complex visualization and interaction techniques with hand-drawnsketches [161]. Domain experts may also be unaware of the wide range of possibili-ties information visualization can offer and its limitations [132, 267]. In addition,we experienced in our own studies that the available meeting time with domainexpert collaborators is typically very limited and need to be used as effectively aspossible. High-fidelity visualization prototypes can show initial results to engagestakeholders and serve as a vehicle for continued task and requirement elicitation.Lloyd andDykes [161] emphasized the crucial importance of visualizing real domaindata for engaging stakeholders in the prototyping phase.Our goal was to create high-fidelity prototypes rapidly but with the flexibility of low-fidelity sketches through aprocess thatwenow refer to as visualizationpatchworkpro-totyping. T_he term patchworkprototypingwas originally defined as “patching togetheropen-source applications as a means of creating high-fidelity prototypes” [117] thatcombines the speed and low cost of paper sketches and the functionality of high-190fidelity prototypes. Floyd et al. [88] extended patchwork prototyping with so-calledwebmash-ups to describe the process of combining publicly available APIs and in-terface components to quickly spin up new applications. A prominent exampleof a web mash-up is to embed Google Maps1 into a prototype and extend it withadditional functionality.We adapted this process as a cornerstone for data-driven prototyping in informa-tion visualization in order to create algorithmic and visualization components thatcan be easily replaced, reused, and glued together.Especially in the Ocupado project, we repeatedly observed that some visualiza-tions or interactions are well received, while others are confusing or do not workas envisioned. We had similar experiences with algorithmic components, in theVizCommender and VizSnippets projects, and iteratively validated and refined thecomputational pipelines. High-fidelity prototypes can lead to a greater degree ofattachment and designersmay becomemore reluctant to abandon ideas, comparedto simpler paper prototypes [161]. Patchwork prototyping based on componentswith minimal dependencies can ease the burden and allows us to change interfacesand algorithmsmore flexibly without discarding whole prototypes.In total, we implemented 21 web-based interactive interfaces using this approach.Fig. 7.1 shows an overview of Ocupado and TimeElide interfaces. Fig. 7.2 showsan overview of VizCommender and VizSnippets interfaces. Although we considerTimeElide as a single interface or tool, it builds upon visualization componentsfrom Ocupado and, throughout the prototyping phase, we replaced and refinedseveral components.T_he patchwork prototyping process has proven to be effective with regard to theprojects described in this dissertation. It allowed us to show initial visualizationsbased on real-world data very early and to respond quickly to new findings andstakeholder needs. However, this prototyping-through-code process, where reusable,independent visualization components are first implemented and then assembledinto web applications, requires considerable technical expertise. Otherwise, theprior arguments fall short and lead to the pitfall of non-rapid prototyping [236].1http://maps.google.com191Existing visualization authoring tools, or prototyping tools in general, do not yetprovide theflexibility and expressivity to prototype these types of interfaceswithoutprogramming, and raises an important issue to be addressed in future research.Visualization tools that allow researchers and designers to augment functionalitythrough code snippets may be a first step.Figure 7.1:Overview of Ocupado (Chapter 2) and TimeElide (Chapter 3) interfaces.192Figure 7.2:Overview of VizCommender (Chapter 4) and VizSnippets (Chapter 5) interfaces.1937.2 Reflecting on Visualization Tools for Different AudiencesandContextsWe reflect on different types of visualization tools by assigning them to a spectrumfrom specialized to generalized. We define visualization tools broadly as interactivesystems that allow users to view or create visualizations through a dedicated userinterface without programming.Visualization tools are often surveyed in terms of their subject matter, such as toolsfor biological networks [210] or for auditing network security [243]. Satyanarayanet al. categorized different types of visualization authoring systems [230]. T_hevisualization literature also commonly distinguishes tools based on their primarygoals, such as presentation of the known and discovery of the unknown [193]. Incontrast, we now look at the different types of visualization tools, that we devel-oped or contributed to in this dissertation, from a target audience and contextperspective.We distinguish between three categories of tools—specialized, flexible, and gener-alized—as shown in Table 7.2. However, we find it also helpful to think of it as acontinuous spectrum from specialized to generalized, as these categories can befuzzy and overlapping.• Specializedvisualizationtools are typically developed in the course of designstudies [236] and in close collaboration with domain experts to help themsolve specific analysis tasks. T_he Ocupado tool suite is an example in thiscategory.• Flexible visualization toolsmay be initially developed as specialized toolsfor a specific domain and are later generalized to support more tasks or datacharacteristics. In other cases, tools may be envisioned and designed to bemore flexible from the beginning, such as TimeElide, which is a domain-independent tool but focuses on very specific data semantics.• Generalized visualization tools are designed for a mass audience and there-fore support a broad range of data characteristics and analysis tasks. Visu-194alization authoring tools, such as Lyra [228] and Data Illustrator [159], areexamples in this category. In particular, commercial BI tools have gained sig-nificant popularity recently andmade visual analysis accessible to a broaderaudience, including non-analysts. T_his widespread use and the ability toaccumulate large-scale visualization collections within an organization leadsto downstream challenges, such as the difficulty in finding and identifyingrelevant visualization content. T_he VizCommender and VizSnippets studiescontribute solutions to address some of these challenges.Table 7.2: Spectrumof visualization tools from specialized to generalized.7.3 ReflectingonVisualizingSpatialandTemporalNon-ContiguityWe reflect on spatial and temporal non-contiguity in the context of visual dataanalysis.7.3.1 SpatiotemporalNon-TrajectoryDataIn Chapter 2, we discussed a characterization and abstraction of spatiotemporalnon-trajectory data, namely data that is collected regularly at the same spots orfrom the same sensors, such as building occupancy in specific regions, meteoro-195logical records from weather stations [275], or station fill levels in bike sharingnetworks. Our own prior work on the Bike Sharing Atlas [203], to visually ana-lyze spatiotemporal non-trajectory data from station-based bike sharing networks,guided our initial thinking and informedkey aspects ofOcupado. In contrast,muchof the previous work on spatiotemporal dynamics has been focused on individualmovements (trajectory) or on origin-destination data.Although the Bike Sharing Atlas concerns large-scale geographic areas and Ocu-pado facilitates the analysis of indoor spatial regions, the underlying data abstrac-tion is very similar. By noting the similarity in the data and the high-level goal tounderstand spatiotemporal dynamics, we now reflect on general implications ofspatiotemporal non-trajectory data in terms of ethics, data preprocessing, tasks,and visual encodings.Implications for EthicsIncreasingly digitized environments producemassive accumulations of data, oftencontaining geo-tagged personal information, that pose risks and raise privacy con-cerns. Especially fine-grained trajectories can disclose sensitive information, forexample, a person’s home, workplace, and daily commuting pattern. Anonymiza-tion and aggregation can only help to a certain degree if the number of trajectoriesis small. Although there are several methods to address this issue, such as spa-tiotemporal generalization [186], we should reconsider if the storage and analysisof trajectories is necessary in the first place to answer certain questions. Some pro-tections and barriers are already in place, such as the EU General Data ProtectionRegulation (GDPR)2, for collecting and processing privacy-sensitive trajectory data,and this trend will likely be reinforced as more organizations become aware of thepotential harms.On the other hand, generalized and location-based counts are often more willinglyshared or can be even accessed in open data libraries.2https://eur-lex.europa.eu/eli/reg/2016/679/oj196Implications forData ProcessingT_he raw data contains the spatial location, a spatial time series, and optional meta-data for each sensor. T_he number of sensors can vary from a few dozens to manythousands. Typically, the sensor states are recorded every few minutes but thiscan be further shortened to generate more detailed time series. For instance, inOcupado we used the API from our industry partner to record occupancy in a5-minute interval from thousands of indoor spatial regions, which can lead tosubstantial amounts of data if it is collected continuously over several months. Wealso binned the recorded data to 15 and 30-minute intervals in a preprocessing stepto improve the load time for interactive visualizations and analysis tasks wheresuch a detailed resolution is not necessary.If sufficient domain-specific knowledge about the data is available, we suspect thatrather than using a uniform interval it can be beneficial to distinguish between timeperiods. For example, in the context of Ocupado, adjusting the recording intervalbetween day and night times can help to reduce the database size and improveprocessing times.An alternative strategy to avoid collecting redundant data is to only save a newrecord in the database when the variable of interest, such as occupancy in a zone,changes since the previous recording.Implications for TasksClearly, the analysis of movement flows is more difficult, if not even impossible,without trajectories or origin-destination data but other tasks are very well sup-ported. T_he task of investigating temporal footprints of locations (sensor level) orwhole regions (i.e., building or district aggregation level) is often in the center ofthe analysis. We can compare sensors at different time resolutions and identifytrends, outliers, and repeating patterns. High-level overviews are necessary butthey are not sufficient. In addition, users frequently want to see the original timeseries for a specific location or compare multiple time series on a detailed level. Abenefit of this data type is, for the task of capturing local and global variations, thatit is much easier to group sensors by regions and to normalize the time series.197Implications for Visual EncodingsNon-trajectory data can be visually encoded with similar techniques that havebeen proposed for visualizing time-oriented data. In particular, small multiples,interactive linked views or superimposed perspectives allow us to capture bothspatial and temporal dimensions. We consider visual encodings for non-trajectorydata fromnon-contiguous indoor regions as a special case that has not been studiedin depth, as we describe in the following section.7.3.2 SpatialNon-Contiguity Related to IndoorRegionsOcupado is a first foray into the visual analysis of sensor data related to non-contiguous indoor regions at scale. Previous attempts are limited to very smallregions such as a few rooms or floors over relatively short time periods. T_hemostcommon visualization techniques for indoor data in general are color-coded floorplans and 3D exploded building views. However, these techniques are inapplicableto large scale settings, such as many rooms distributed across buildings, or foranalyzing more complex data that is linked to spatial regions.In Ocupado, three different visualization approaches have been proven effective:(1) Use high-level overviews with abstract visual encodings for spatial regions andallow users to group, sort, and filter them based on their spatial proximity. (2) Atmid-level scales, provide the spatial context using small floor plans and link thembidirectionally to abstract encodings, such as juxtaposed line charts for showingtime series data. Linearization of floor plans ensures that spatially nearby regionsare close to each other in the display. (3) In detail views, use large floor plans andsuperimpose data, such as size-coded circle symbols and numeric labels. Providecontrol widgets to enable users to interactively update the view.Our stakeholders, fromdifferentdomainsandwithdifferent analysis tasks, stronglysignalled their comfort with and preference for using floor plans many times butthese cannot be the principal medium to disseminate all information because ofpotentially complex data that needs to be embedded, and up to dozens or hundredsof floor plans.198Linking data to indoor spatial regions is important for many tasks, where fullyautomatic solutions are not feasible, and therefore poses a highly relevant directionfor future research. We hope that our research on visualizing indoor spatial timeseries will serve as a base for future studies.7.3.3 TemporalNon-ContiguityTimeElide focuses on the visual analysis of non-contiguous time series slices andgeneralizes some ideas fromOcupado to other domains and datasets that do notexhibit a spatial dimension or when the spatial context is not relevant for the givenquestions.T_he proposed TimeElide design space specifically addresses the detailed investi-gation of one time series that can contain up to hundreds of non-contiguous timeslices, such as the number of passes of a soccer player during a series ofmatches. Incontrast, Ocupado allows to viewmultiple independent time series at the overviewlevel by aggregating the raw data within each time slice. But too much aggregationcan hide important details that are crucial for certain tasks, such as identifyingoutliers or unusual patterns. Managing this trade-off between aggregated dataand fine-grained details remains challenging and presents a vital issue for futurevisualization research. One possible direction is to explore howmachine learningmodels can assist users in visual pattern search [150] andmore broadly during theanalysis of non-contiguous time series at scale.7.3.4 Visual Perception StudiesUnderstanding the graphical perception [59] of visual encodings for low-level analysistasks is critical to effective visualization design. Previous work [24, 104, 116, 134]demonstrated how to conduct such controlled experiments to investigate the effectsof different design choices on graphical perception. T_he findings of this type ofempirical assessment helps to validate theory, replicate previous studies, and toevaluate real-world applications in a controlled setting [102].199In this context, we see ample opportunities for future work, such as comparing theeffectiveness of the proposed sparkbox technique with alternative visual encodings,or systematically evaluating data-enhanced floor plans.7.4 Reflecting onVisualization as aMediumVisualization is a medium that serves different purposes. Here, we reflect onhow the developed visualizations and tools enable exploration, explanation, andcollaboration. We discuss limitations and outline future research directions.7.4.1 Analyzing andCommunicatingDataVisualizations are commonly divided into two different categories according totheir intended use: (1) exploration to allow users to conduct their own analysis aboutthe data (user-driven), and (2) explanation to communicate an author’s narrativeabout the data (author-driven), which includes monitoring.T_hese goals are also reflected in the visualization and interaction design. Butinstead of distinct categories, one can consider visualization design on a spectrumfrom exploration to explanation [260]. On one side, there is a strong tendencyto build powerful tools with many linked views and free-form exploration, andon the other side are narrative visualizations [238] which are increasingly used,especially as an alternative form of storytelling in data-driven journalism. Manyexamples of visualizations fall somewhere in between and have both exploratoryand explanatory characteristics.Increasing the exploratory power of data stories enables users to make their owninquiries and personalize their experience [260]. For instance, the martini glassvisualization structure [238] begins with a tight narrative path, controlled by thevisualization author, and then opens up to allow the user to interactively explorethe data.Increasing the explanatory power of visual analysis tools provides guidance, suchas providing users with a starting point for their own exploration.200T_he challenge for effective visualization design is to find the right balance betweenexplanatory and exploratory characteristics based on the target audience and theirtasks.In the Ocupado project, we made two key observations. First, the elicited tasksconcern theanalysis of occupancydatabut free-formexploratorydata analysis is notthe goal, and moreover our stakeholders have no training in data analysis. Instead,they need information in a clear and concise form to facilitate their decisionmaking.T_he tasks vary and cannot be described precisely enough to be solved by a fullyautomatic solution. Second, although the Ocupado stakeholders are the primaryuser group, they are intermediaries and want to communicate their findings tosuperiors or colleagues. T_his step is often challenging in existing domain-specificvisual analysis tools. It requires users to take screenshots or visualize data withadditional tools to communicate results more clearly.To address these issues, we took some initial steps in Ocupado. We developedmultiple interfaces that target different combinations of spatial and temporal datagranularities, to reduce the complexity of these interfaces and increase ease of use.In addition, we integrated application presets in the Campus Explorer interface toprovide single-click access to commonly used combinations of actions that couldbe accomplished through the interactive interface. All active parameters and filtersare also stored in the URL and allow users to save and share analysis snapshots.T_hemain objective of the TimeElide research prototype is exploratory data analysisof a specific typeof time-orienteddata. T_he tool provides someguidanceby showingexplanations and by selecting smart defaults of visual encodings and settings basedon the data characteristics. However, to be adapted in practice, and to help userscommunicate results, additional features need to be integrated, such as imageexports.In general, we observe a strong need to providemore guidance [62] and to combineexploratory and explanatory elements in interactive visualizations. We consider itas a fundamental direction for future research in information visualization, in orderto make visual data analysis accessible to a broader audience. Specifically in thecontext of Ocupado and TimeElide, we envision the semi-automated generation of201visual data summaries. Visualization tools automatically scan the data and presentpatterns, outliers, trends, and so on, but simultaneously allow users to explore thedata and customize the visual representation [95]. Another direction for futurework is to investigate NLP techniques for the automatic generation of annotationsin interactive visualizations [111, 157].7.4.2 Capturing andSharing Thoughts andDataDiscoveriesVisualization serves also a medium for capturing thoughts and data discoveries.Visualization authors manually filter, aggregate, and join data, choose visual en-codings, highlight data points, and create annotations and captions. All theseoperations implicitly or explicitly capture thoughts and insights.T_he visualization collections described in this dissertation serve as shared knowl-edge bases. Everyone in an organization can access and contribute hand-craftedvisualization content to these collections, and over time amass vast amounts ofinformation. In our research, we learned that visualization collections exhibitthe same issues of ad hoc organization, inconsistent metadata, and scale that arecommon in similar large-scale item collections, that prevent its efficient usage. Werefer to it as the last mile problem in visualization.T_he VizCommender and VizSnippets studies provide encouraging results for rec-ommending and presenting relevant visualization content. T_his research has givenrise to many questions in need of further investigation. For instance, searchingwithin visualization bundles and presenting those results in visual form is a criticalproblem. Mixed-initiative tools are needed to support administrators and usersin decluttering visualization collections. Visualization systems now allow users toadd comments to a chart and have conversations about the content. T_his data canbe highly valuable for improving the quality of recommendations and should bealso incorporated into visualization snippets.2027.5 ConcludingRemarksWe demonstrated through multiple studies how real-world data can guide theprototyping process, help in generating ideas, and lead to new research questions.Our high-level process, that we followed throughout the dissertation, is illustratedin Fig. 7.3 below. After gaining a thorough understanding of the domain-specificdata, we derive a data abstraction, analyze its peculiarities, and ideate possiblesolutions. We converge on a more concrete visualization design and implementit rapidly to get early feedback. Following the validation phase, we reflect on theresults and situate it in the broader context of visualization research. We regularlyjump back to previous stages to refine initial assumptions and ideas. Although datais a driving force, wewant to emphasize that prototyping is strictly human-centeredand it is critical to embed the target users throughout the process.Ever-growing datasets, novel data characteristics, new ways to collect data, andnew audiences who want to understand data will serve as a continuous impulseand inspiration for future visualization research.Figure 7.3: Illustration of the six stages of our data-driven prototyping process (reproducedfromChapter 1 for reference).203Bibliography[1] A. Agarwal andM. Chauhan. Similarity measures used in recommendersystems: A study. Int. Journal of Engineering Technology Science and Research,pages 2394–3386, 2017. → page 107[2] C. C. Aggarwal et al. Recommender Systems, volume 1. Springer, 2016. →pages 86, 87, 88[3] E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez Agirre, R. Mihalcea,G. Rigau Claramunt, and J. Wiebe. Semeval-2016 task 1: Semantic textualsimilarity, monolingual and cross-lingual evaluation. In Proc. ACL Int.Workshop on Semantic Evaluation, pages 497–511, 2016.doi:10.18653/v1/S16-1081. → page 95[4] W. Aigner, S. Miksch, H. Schumann, and C. Tominski. Visualization ofTime-Oriented Data. Springer Science & Business Media, 2011.doi:10.1007/978-0-85729-079-3. → pages 53, 61[5] A. K. Al-Awami, J. Beyer, H. Strobelt, N. Kasthuri, J. W. Lichtman, H. Pfister,andM. Hadwiger. NeuroLines: A subwaymapmetaphor for visualizingnanoscale neuronal connectivity. IEEE Trans. Visualization and ComputerGraphics, 20(12):2369–2378, 2014. doi:10.1109/TVCG.2014.2346312.→ page 40[6] R. A. Al-Zaidy, S. R. Choudhury, and C. L. Giles. Automatic summarygeneration for scientific data charts. In Proc. AAAI Conf. on ArtificialIntelligence, pages 658–663, 2016. → page 134[7] B. Alsallakh, L. Micallef, W. Aigner, H. Hauser, S. Miksch, and P. Rodgers.T_he state-of-the-art of set visualization. Computer Graphics Forum, 35(1):234–260, 2016. doi:10.1111/cgf.12722. → page 135204[8] S. Amershi, M. Chickering, S. M. Drucker, B. Lee, P. Simard, and J. Suh.ModelTracker: Redesigning performance analysis tools for machinelearning. In Proc. ACMSIGCHI Conf. onHuman Factors in Computing Systems(CHI), pages 337–346, 2015. doi:10.1145/2702123.2702509. → page 163[9] G. Andrienko, N. Andrienko, P. Bak, D. Keim, and S.Wrobel. Visual Analyticsof Movement. Springer, 2009. doi:10.1007/978-3-642-37583-5. → page 53[10] N. Andrienko and G. Andrienko. Exploratory Analysis of Spatial and TemporalData: A Systematic Approach. Springer Science & Business Media, 2006.doi:10.1007/3-540-31190-4. → page 23[11] A. Aris, B. Shneiderman, C. Plaisant, G. Shmueli, andW. Jank.Representing unevenly-spaced time series data for visualization andinteractive exploration. In IFIP Conf. onHuman-Computer Interaction, pages835–846. Springer, 2005. doi:10.1007/11555261_66. → page 61[12] A. Arleo, C. Tsigkanos, C. Jia, R. A. Leite, I. Murturi, M. Klaffenböck,S. Dustdar, M.Wimmer, S. Miksch, and J. Sorger. Sabrina: Modeling andvisualization of financial data over time with incremental domainknowledge. In Proc. IEEE Visualization Conference (VIS), pages 51–55, 2019.doi:10.1109/VISUAL.2019.8933598. → page 184[13] Z. Armstrong. Covid-19 viz roundup (Observable notebook).https://observablehq.com/@zanarmstrong/covid-19-viz-roundup, 2021. →page 159[14] D. Auber. Using strahler numbers for real time visual exploration of hugegraphs. In Int. Conf. Computer Vision and Graphics, volume 1, page 3, 2002. →page 135[15] A. Aula, R. M. Khan, Z. Guan, P. Fontes, and P. Hong. A comparison ofvisual and textual page previews in judging the helpfulness of web pages. InProc. Int. Conf. onWorldWideWeb, pages 51–60, 2010.doi:10.1145/1772690.1772697. → pages 133, 134[16] R. Bade, S. Schlechtweg, and S. Miksch. Connecting time-oriented data andinformation to a coherent interactive visualization. In ACMSIGCHI Conf.Human Factors in Computing Systems (CHI), pages 105–112, 2004.doi:10.1145/985692.985706. → pages 43, 62, 66[17] B. Balaji, J. Xu, A. Nwokafor, R. Gupta, and Y. Agarwal. Sentinel: Occupancybased HVAC actuation using existingWiFi infrastructure within commercial205buildings. In ACMConf. Embedded Networked Sensor Systems (SenSys), pages17:1–17:14, 2013. doi:10.1145/2517351.2517370. → pages 17, 22, 57[18] R. C. Basole, T. Clear, M. Hu, H. Mehrotra, and J. Stasko. Understandinginterfirm relationships in business ecosystems with interactive visualization.IEEE Trans. Visualization and Computer Graphics, 19(12):2526–2535, 2013.doi:10.1109/TVCG.2013.209. → page 184[19] D. Baur, F. Seiffert, M. Sedlmair, and S. Boring. T_he streams of our lives:Visualizing listening histories in context. IEEE Trans. Visualization andComputer Graphics, 16(6):1119–1128, 2010. doi:10.1109/TVCG.2010.206. →page 184[20] M. Baxter. Product Design. CRC Press, 1995. → page 177[21] K. Beard, H. Deese, and N. R. Pettigrew. A framework for visualization andexploration of events. Information Visualization, 7(2):133–151, 2008.doi:10.1145/1466620.1466623. → page 54[22] F. Beck, S. Koch, and D.Weiskopf. Visual analysis and dissemination ofscientific literature collections with survis. IEEE Trans. Visualization andComputer Graphics, 22(1):180–189, 2015. doi:10.1109/TVCG.2015.2467757. →page 92[23] T. Beck, F. Böschen, and A. Scherp. What to read next? challenges andpreliminary results in selecting representative documents. In Int. Conf.Database and Expert Systems Applications, pages 230–242, 2018.doi:10.1007/978-3-319-99133-7_19. → page 106[24] A. Bezerianos and P. Isenberg. Perception of visual variables on tiledwall-sized displays for information visualization applications. IEEE Trans.Visualization and Computer Graphics, 18(12):2516–2525, 2012.doi:10.1109/TVCG.2012.251. → page 199[25] A. Bezerianos, P. Dragicevic, J.-D. Fekete, J. Bae, and B.Watson.Geneaquilts: A system for exploring large genealogies. IEEE Trans.Visualization and Computer Graphics, 16(6):1073–1081, 2010.doi:10.1109/TVCG.2010.159. → page 184[26] E. A. Bier, M. C. Stone, K. Pier, W. Buxton, and T. D. DeRose. Toolglass andmagic lenses: T_he see-through interface. In Proc. SIGGRAPH 93, AnnualConference Series, pages 73–80, 1993. → page 61206[27] S. Bird, E. Klein, and E. Loper. Natural language processing with Python:analyzing text with the natural language toolkit. "O’Reilly Media", 2009. → page100[28] D. M. Blei, A. Y. Ng, andM. I. Jordan. Latent dirichlet allocation. Journal ofMachine Learning Research, 3(Jan):993–1022, 2003. → pages 107, 164[29] M. Bögl, P. Filzmoser, T. Gschwandtner, S. Miksch,W. Aigner, A. Rind, andT. Lammarsch. Visually and statistically guided imputation of missingvalues in univariate seasonal time series. In Proc. IEEEConf. Visual AnalyticsScience and Technology (VAST), pages 189–190. IEEE, 2015.doi:10.1109/VAST.2015.7347672. → page 50[30] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov. Enriching word vectorswith subword information. Trans. Assoc. for Computational Linguistics, 5:135–146, 2017. doi:10.1162/tacl_a_00051. → page 106[31] I. Boyandin, E. Bertini, P. Bak, and D. Lalanne. Flowstrates: An approach forvisual exploration of temporal origin-destination data. Computer GraphicsForum, 30(3):971–980, 2011. doi:10.1111/j.1467-8659.2011.01946.x. → page 53[32] W. Brackenbury, R. Liu, M. Mondal, A. J. Elmore, B. Ur, K. Chard, andM. J.Franklin. Draining the data swamp: A similarity-based approach. In Proc.Workshop onHuman-In-the-Loop Data Analytics, pages 1–7, 2018.doi:10.1145/3209900.3209911. → page 85[33] M. Brehmer, S. Ingram, J. Stray, and T. Munzner. Overview: T_he design,adoption, and analysis of a visual document mining tool for investigativejournalists. IEEE Trans. Visualization and Computer Graphics, 20(12):2271–2280, 2014. doi:10.1109/TVCG.2014.2346431. → page 175[34] M. Brehmer, J. Ng, K. Tate, and T. Munzner. Matches, mismatches, andmethods: Multiple-view workflows for energy portfolio analysis. IEEE Trans.Visualization and Computer Graphics, 22(1):449–458, 2016. ISSN 10772626.doi:10.1109/TVCG.2015.2466971. → page 25[35] M. Brehmer, B. Lee, B. Bach, N. H. Riche, and T. Munzner. Timelinesrevisited: A design space and considerations for expressive storytelling.IEEE Trans. Visualization and Computer Graphics, 23(9):2151–2164, 2017.doi:10.1109/TVCG.2016.2614803. → pages 53, 61[36] M. Brehmer, B. Lee, P. Isenberg, and E. K. Choe. A comparative evaluationof animation and small multiples for trend visualization onmobile phones.207IEEE Trans. Visualization and Computer Graphics, 26(1):364–374, 2019.doi:10.1109/TVCG.2019.2934397. → page 135[37] A. Z. Broder. On the resemblance and containment of documents. In Proc.Compression and Complexity of Sequences, pages 21–29, 1997.doi:10.1109/SEQUEN.1997.666900. → page 118[38] M. Buchenau and J. F. Suri. Experience prototyping. In Proc. DesigningInteractive Systems Conference (DIS), page 424–433, 2000.doi:10.1145/347642.347802. → page 2[39] Z. Bylinskii, S. Alsheikh, S. Madan, A. Recasens, K. Zhong, H. Pfister,F. Durand, and A. Oliva. Understanding infographics through textual andvisual tag prediction. arXiv preprint arXiv:1709.09215, 2017. → page 135[40] E. Çano andM.Morisio. Hybrid recommender systems: A systematicliterature review. Intelligent Data Analysis, 21(6):1487–1524, 2017.doi:10.3233/IDA-163209. → page 120[41] R. Capra, J. Arguello, and F. Scholer. Augmenting web search surrogateswith images. In Proc. ACM Int. Conf. Information and KnowledgeManagement,pages 399–408, 2013. doi:10.1145/2505515.2505714. → page 134[42] J. Carbonell and J. Goldstein. T_he use of MMR, diversity-based reranking forreordering documents and producing summaries. In Proc. ACMSIGIR Conf.Research andDevelopment in Information Retrieval, pages 335–336, 1998.doi:10.1145/290941.291025. → page 144[43] D. Cashman, S. R. Humayoun, F. Heimerl, K. Park, S. Das, J. T_hompson,B. Saket, A. Mosca, J. Stasko, A. Endert, et al. A user-based visual analyticsworkflow for exploratory model analysis. Computer Graphics Forum, 38(3):185–199, 2019. doi:10.1111/cgf.13681. → page 163[44] S.-H. Cha. Comprehensive survey on distance/similarity measures betweenprobability density functions. Int. Journal ofMathematicalModels andMethodsin Applied Sciences, 1(4):300–307, 2007. → page 142[45] J. Chae, D.T_hom, H. Bosch, Y. Jang, R. Maciejewski, D. S. Ebert, and T. Ertl.Spatiotemporal social media analytics for abnormal event detection andexamination using seasonal-trend decomposition. In Proc. IEEEConf. VisualAnalytics Science and Technology (VAST), pages 143–152, 2012.doi:10.1109/VAST.2012.6400557. → page 54208[46] J. Chang, J. Boyd-Graber, S. Gerrish, C.Wang, and D. M. Blei. Reading tealeaves: How humans interpret topic models. In Proc. Adv. in NeuralInformation Processing Systems (NeurIPS), page 288–296, 2009.doi:10.5555/2984093.2984126. → page 289[47] J. Chen, P. Yuan, X. Zhou, and X. Tang. Performance comparison of TF*IDF,LDA and paragraph vector for document classification. In Proc. Int. Symp. onKnowledge and Systems Sciences, pages 225–235, 2016.doi:10.1007/978-981-10-2857-1_20. → page 116[48] W. Chen, X. Chen, J. Zhang, and K. Huang. Beyond triplet loss: a deepquadruplet network for person re-identification. In Proc. IEEEConf.Computer Vision and Pattern Recognition, pages 403–412, 2017.doi:10.1109/CVPR.2017.145. → page 143[49] W.-F. Chen, S. Syed, B. Stein, M. Hagen, andM. Potthast. Abstractivesnippet generation. In Proc.T_heWeb Conference, pages 1309–1319, 2020.doi:10.1145/3366423.3380206. → page 164[50] Y. Chen. Visualizing large time-series data on very small screens. In Proc.Eurographics Conf. Visualization (EuroVis), pages 37–41, 2017.doi:10.2312/eurovisshort.20171130. → page 135[51] Y. Chen, X. Liang, T. Hong, and X. Luo. Simulation and visualization ofenergy-related occupant behavior in office buildings. Building Simulation, 10(6):785–798, 2017. doi:10.1007/s12273-017-0355-2. → page 55[52] Z. Chen, M. Cafarella, and E. Adar. Diagramflyer: A search engine fordata-driven diagrams. In Proc. Int. Conf. onWorldWideWeb, pages 183–186,2015. doi:10.1145/2740908.2742831. → page 133[53] G. Cheng, C. Jin, W. Ding, D. Xu, and Y. Qu. Generating illustrativesnippets for open data on the web. In Proc. ACM Int. Conf. onWeb Search andDataMining, pages 151–159, 2017. doi:10.1145/3018661.3018670. → page 135[54] I. K. Choi, N. K. Raveendranath, J. Westerfield, and K. Reda. Visual (dis)confirmation: Validating models and hypotheses with visualizations. In Int.Conf. Information Visualization, pages 116–121, 2019.doi:10.1109/IV-2.2019.00032. → page 163[55] J. Choo, C. Lee, H. Kim, H. Lee, Z. Liu, R. Kannan, C. D. Stolper, J. Stasko,B. L. Drake, and H. Park. Visirr: Visual analytics for information retrievaland recommendation with large-scale document data. In IEEEConf. Visual209Analytics Science and Technology (VAST), pages 243–244, 2014.doi:10.1145/3070616. → page 92[56] O. Chum, J. Philbin, M. Isard, and A. Zisserman. Scalable near identicalimage and shot detection. In Proc. ACM Int. Conf. Image and Video Retrieval,pages 549–556, 2007. doi:10.1145/1282280.1282359. → page 118[57] K. Church, B. Smyth, andM. T. Keane. Evaluating interfaces for intelligentmobile search. In Proc.Workshop onWeb Accessibility (W4A): Building theMobileWeb: Rediscovering Accessibility?, pages 69–78, 2006.doi:10.1145/1133219.1133232. → page 134[58] CityBikes API. https://api.citybik.es, 2016. → page 171[59] W. S. Cleveland and R. McGill. Graphical perception: T_heory,experimentation, and application to the development of graphical methods.Journal of the American statistical association, 79(387):531–554, 1984.doi:10.2307/2288400. → page 199[60] J. Cohen. A coefficient of agreement for nominal scales. Edu. and Psych.Measurement, 20(1):37–46, 1960. doi:10.1177/001316446002000104. → page114[61] D. Collaris and J. J. vanWijk. ExplainExplore: visual exploration of machinelearning explanations. In Proc. IEEE Pacific Visualization Symp., pages 26–35,2020. doi:10.1109/PacificVis48177.2020.7090. → page 163[62] C. Collins, N. Andrienko, T. Schreck, J. Yang, J. Choo, U. Engelke, A. Jena,and T. Dwyer. Guidance in the human–machine analytics process. VisualInformatics, 2(3):166–180, 2018. doi:10.1016/j.visinf.2018.09.003. → page 201[63] L. Colucci, P. Doshi, K.-L. Lee, J. Liang, Y. Lin, I. Vashishtha, J. Zhang, andA. Jude. Evaluating item-item similarity algorithms for movies. In Proc. CHIConf. Extended Abstracts onHuman Factors in Computing Systems, pages2141–2147, 2016. doi:10.1145/2851581.2892362. → page 95[64] L. Corpuz-Bosshart. Innovative software convertsWi-Fi data into energysavings. https://news.ubc.ca/2017/03/30/innovative-software-converts-wi-fi-data-into-energy-savings/, 2017. → pages5, 18210[65] A. Crisan and T. Munzner. Uncovering data landscapes through datareconnaissance and task wrangling. In Proc. IEEE Visualization Conference(VIS), pages 46–50, 2019. doi:10.1109/VISUAL.2019.8933542. → page 170[66] R. Dachselt, M. Frisch, andM.Weiland. FacetZoom: A continuousmulti-scale widget for navigating hierarchical metadata. In ACMSIGCHIConf. Human Factors in Computing Systems (CHI), pages 1353–1356, 2008.doi:10.1145/1357054.1357265. → pages 6, 61[67] N. Dalal and B. Triggs. Histograms of oriented gradients for humandetection. In Proc. IEEEConf. Computer Vision and Pattern Recognition,volume 1, pages 886–893, 2005. doi:10.1109/CVPR.2005.177. → page 142[68] A. Das Sarma, L. Fang, N. Gupta, A. Halevy, H. Lee, F. Wu, R. Xin, and C. Yu.Finding related tables. In Proc. ACMSIGMOD Int. Conf. onManagement ofData, page 817–828, 2012. doi:10.1145/2213836.2213962. → page 135[69] Data is Beautiful. DataViz Battle. https://reddit.com/r/dataisbeautiful, 2020.→ page 168[70] Y.-A. de Montjoye, C. A. Hidalgo, M. Verleysen, and V. D. Blondel. Unique inthe crowd: T_he privacy bounds of humanmobility. Scientific Reports, 3, 2013.doi:10.1038/srep01376. → page 23[71] S. Deerwester, S. T. Dumais, G.W. Furnas, T. K. Landauer, andR. Harshman. Indexing by latent semantic analysis. Journal of the AmericanSociety for Information Science, 41(6):391–407, 1990. → page 107[72] Y. Deldjoo, M. Elahi, P. Cremonesi, F. Garzotto, P. Piazzolla, andM. Quadrana. Content-based video recommendation system based onstylistic visual features. Journal on Data Semantics, 5(2):99–113, 2016.doi:10.1007/s13740-016-0060-9. → page 86[73] R. DeLine, M. Czerwinski, B. Meyers, G. Venolia, S. Drucker, andG. Robertson. Code thumbnails: Using spatial memory to navigate sourcecode. In Proc. IEEE Symp. on Visual Languages andHuman-Centric Computing,pages 11–18, 2006. doi:10.1109/VLHCC.2006.14. → page 133[74] Ç. Demiralp, M. S. Bernstein, and J. Heer. Learning perceptual kernels forvisualization design. IEEE Trans. Visualization and Computer Graphics, 20(12):1933–1942, 2014. doi:10.1109/TVCG.2014.2346978. → pages 96, 109211[75] U. Demšar and K. Virrantaus. Space–time density of trajectories: Exploringspatio-temporal patterns in movement data. Intern. Journal of GeographicalInformation Science, 24(10):1527–1542, 2010. doi:10.1080/13658816.2010.511223.→ page 53[76] A. Desjardins and A. Ball. Revealing tensions in autobiographical design inHCI. In Proc. Designing Interactive Systems Conference (DIS), pages 753–764,2018. doi:10.1145/3196709.3196781. → page 187[77] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training ofdeep bidirectional transformers for language understanding. Proc. Conf.North American Chap. Assoc. for Computational Linguistics: Human LanguageTechnologies (NAACL-HLT), 2019. doi:10.18653/v1/N19-1423. → page 106[78] V. Dibia and Ç. Demiralp. Data2Vis: Automatic generation of datavisualizations using sequence-to-sequence recurrent neural networks. IEEEComputer Graphics and Applications, 39(5):33–46, 2019.doi:10.1109/MCG.2019.2924636. → page 94[79] S. P. Dow, K. Heddleston, and S. R. Klemmer. T_he efficacy of prototypingunder time constraints. In Proc. ACMConf. on Creativity and Cognition, pages165–174, 2009. doi:10.1145/1640233.1640260. → page 190[80] F. Du, S. Malik, G.T_heocharous, and E. Koh. Personalizable and interactivesequence recommender system. In Proc. CHI Conf. Extended Abstracts onHuman Factors in Computing Systems, pages 1–6, 2018.doi:10.1145/3170427.3188506. → page 96[81] S. Dziadosz and R. Chandrasekar. Do thumbnail previews help users makebetter relevance decisions about web search results? In Proc. ACMSIGIRConf. Research andDevelopment in Information Retrieval, pages 365–366, 2002.doi:10.1145/564376.564446. → pages 127, 133, 134[82] A. Endert, M. S. Hossain, N. Ramakrishnan, C. North, P. Fiaux, andC. Andrews. T_he human is the loop: new directions for visual analytics.Journal of Intell. Information Systems, 43(3):411–435, 2014.doi:10.1007/s10844-014-0304-9. → page 1[83] B. Erol, K. Berkner, and S. Joshi. Multimedia thumbnails for documents. InProc. Int. Conf. onMultimedia, pages 231–240, 2006.doi:10.1145/1180639.1180738. → page 133212[84] P. Eulzer, M. Meuschke, C. Klingner, and K. Lawonn. Visualizing carotidblood flow simulations for stroke prevention. Computer Graphics Forum, 40(3), 2021. → page 188[85] K. N. Fachry, J. Kamps, J. Zhang, et al. T_he impact of summaries: Whatmakes a user click. In Proc. Dutch-Belgian Information RetrievalWorkshop,pages 47–54, 2010. → page 134[86] D. Fisher andM.Meyer. Making Data Visual: A Practical Guide to UsingVisualization for Insight. O’Reilly, 2017. → page 57[87] J. L. Fleiss. Measuring nominal scale agreement amongmany raters.Psychol. Bull., 76(5):378, 1971. doi:10.1037/h0031619. → page 114[88] I. R. Floyd, M. C. Jones, D. Rathi, andM. B. Twidale. Webmash-ups andpatchwork prototyping: User-driven technological innovation with web 2.0and open source software. InHawaii Int. Conf. on System Sciences (HICSS),pages 86–86, 2007. doi:10.1109/HICSS.2007.612. → page 191[89] J. E. Froehlich, J. Neumann, and N. Oliver. Sensing and predicting the pulseof the city through shared bicycling. In Int. Joint Conf. on Artificial Intelligence,2009. → page 171[90] M. Ge, C. Delgado-Battenfeld, and D. Jannach. Beyond accuracy: Evaluatingrecommender systems by coverage and serendipity. In Proc. ACMConf.Recommender Systems (RecSys), pages 257–260, 2010.doi:10.1145/1864708.1864761. → page 105[91] GFDRR. VizRisk challenge. https://understandrisk.org/vizrisk/, 2019. →page 168[92] S. Goodwin, J. Dykes, S. Jones, I. Dillingham, G. Dove, A. Duffy, A. Kachkaev,A. Slingsby, and J. Wood. Creative user-centered visualization design forenergy analysts andmodelers. IEEE Trans. Visualization and ComputerGraphics, 19(12):2516–2525, 2013. doi:10.1109/TVCG.2013.145. → pages170, 184[93] Google. Popular times, wait times, and visit duration.https://support.google.com/business/answer/6263531, 2019. → page 55[94] S. Gratzl, A. Lex, N. Gehlenborg, H. Pfister, andM. Streit. Lineup: Visualanalysis of multi-attribute rankings. IEEE Trans. Visualization and ComputerGraphics, 19(12):2277–2286, 2013. doi:10.1109/TVCG.2013.173. → page 41213[95] N. L. Green, G. Carenini, S. Kerpedjiev, J. Mattis, J. D. Moore, and S. F. Roth.AutoBrief: An experimental system for the automatic generation of briefingsin integrated text and information graphics. Int. Journal of Human-ComputerStudies, 61(1):32–70, 2004. doi:10.1016/j.ijhcs.2003.10.007. → page 202[96] S. Greene, G. Marchionini, C. Plaisant, and B. Shneiderman. Previews andoverviews in digital libraries: Designing surrogates to support visualinformation seeking. Journal of the American Society for Information Science, 51(4):380–393, 2000. → page 127[97] T. Gschwandtner, W. Aigner, K. Kaiser, S. Miksch, and A. Seyfang.CareCruiser: Exploring and visualizing plans, events, and effectsinteractively. In IEEE Pacific Visualization Symposium, pages 43–50, 2011.doi:10.1109/PACIFICVIS.2011.5742371. → page 61[98] M. Gygli, Y. Song, and L. Cao. Video2Gif: Automatic generation of animatedgifs from video. In Proc. IEEEConf. Computer Vision and Pattern Recognition,pages 1001–1009, 2016. doi:10.1109/CVPR.2016.114. → page 134[99] R. Hai, S. Geisler, and C. Quix. Constance: An intelligent data lake system.In Proc. Int. Conf.Management of Data, pages 2097–2100, 2016.doi:10.1145/2882903.2899389. → page 85[100] S. Havre, B. Hetzler, and L. Nowell. T_hemeRiver: Visualizing theme changesover time. In Proc. IEEE Symp. on Information Visualization (InfoVis), pages115–123, 2000. doi:10.1109/INFVIS.2000.885098. → page 61[101] C. He, D. Parra, and K. Verbert. Interactive recommender systems: A surveyof the state of the art and future research challenges and opportunities.Expert Systems with Applications, 56:9–27, 2016.doi:10.1016/j.eswa.2016.02.013. → page 118[102] J. Heer andM. Bostock. Crowdsourcing graphical perception: UsingMechanical Turk to assess visualization design. In Proc. ACMSIGCHI Conf.onHuman Factors in Computing Systems (CHI), pages 203––212, 2010.doi:10.1145/1753326.1753357. → pages 112, 199[103] J. Heer, J. Mackinlay, C. Stolte, andM. Agrawala. Graphical histories forvisualization: Supporting analysis, communication, and evaluation. IEEETrans. Visualization and Computer Graphics, 14(6):1189–1196, 2008.doi:10.1109/TVCG.2008.137. → page 135214[104] J. Heer, N. Kong, andM. Agrawala. Sizing the horizon: the effects of chartsize and layering on the graphical perception of time series visualizations.In ACMSIGCHI Conf. Human Factors in Computing Systems (CHI), pages1303–1312, 2009. doi:10.1145/1518701.1518897. → page 199[105] H. Hochheiser and B. Shneiderman. Dynamic query tools for time seriesdata sets: Timebox widgets for interactive exploration. InformationVisualization, 3(1):1–18, 2004. doi:10.1145/993176.993177. → page 61[106] M. Hoffman, F. R. Bach, and D. M. Blei. Online learning for latent dirichletallocation. In Proc. Adv. in Neural Information Processing Systems (NeurIPS),pages 856–864, 2010. → page 117[107] J. Hoffswell, W. Li, and Z. Liu. Techniques for flexible responsivevisualization design. In Proc. ACMSIGCHI Conf. onHuman Factors inComputing Systems (CHI), pages 1–13, 2020. doi:10.1145/3313831.3376777. →page 135[108] F. Hohman, K.Wongsuphasawat, M. B. Kery, and K. Patel. Understandingand visualizing data iteration in machine learning. In ACMSIGCHI Conf.Human Factors in Computing Systems (CHI), pages 1–13, 2020.doi:10.1145/3313831.3376177. → pages 163, 181[109] K. Hu, M. A. Bakker, S. Li, T. Kraska, and C. Hidalgo. VizML: Amachinelearning approach to visualization recommendation. In Proc. ACMSIGCHIConf. Human Factors in Computing Systems (CHI), pages 1–12, 2019.doi:10.1145/3290605.3300358. → page 94[110] K. Hu, S. Gaikwad, M. Hulsebos, M. A. Bakker, E. Zgraggen, C. Hidalgo,T. Kraska, G. Li, A. Satyanarayan, and Ç. Demiralp. VizNet: Towards alarge-scale visualization learning and benchmarking repository. In Proc.ACMSIGCHI Conf. onHuman Factors in Computing Systems (CHI), pages 1–12,2019. doi:10.1145/3290605.3300892. → page 135[111] J. Hullman, N. Diakopoulos, and E. Adar. Contextifier: Automaticgeneration of annotated stock visualizations. In ACMSIGCHI Conf. HumanFactors in Computing Systems (CHI), pages 2707–2716, 2013.doi:10.1145/2470654.2481374. → page 202[112] H. Hutchinson,W.Mackay, B.Westerlund, B. B. Bederson, A. Druin,C. Plaisant, M. Beaudouin-Lafon, S. Conversy, H. Evans, H. Hansen, et al.Technology probes: Inspiring design for and with families. In ACMSIGCHI215Conf. Human Factors in Computing Systems (CHI), pages 17–24. ACM, 2003.doi:10.1145/642611.642616. → pages 2, 32, 90, 178, 179[113] P. Isenberg, T. Zuk, C. Collins, and S. Carpendale. Grounded evaluation ofinformation visualizations. In IEEE VISWorkshop on Beyond Time and Errors -Novel EvaluationMethods for Visualization (BELIV), pages 1–8, 2008. → page 4[114] P. Isenberg, F. Heimerl, S. Koch, T. Isenberg, P. Xu, C. D. Stolper,M. Sedlmair, J. Chen, T. Möller, and J. Stasko. vispubdata.org: A metadatacollection about IEEE Visualization (VIS) publications. IEEE Trans.Visualization and Computer Graphics, 23(9):2199–2206, 2016.doi:10.1109/TVCG.2016.2615308. → page 182[115] Y. A. Ivanov, C. R.Wren, A. Sorokin, and I. Kaur. Visualizing the history ofliving spaces. IEEE Trans. Visualization and Computer Graphics, 13(6):1153–1160, 2007. ISSN 10772626. doi:10.1109/TVCG.2007.70621. → page 54[116] W. Javed, B. McDonnel, and N. Elmqvist. Graphical perception of multipletime series. IEEE Trans. Visualization and Computer Graphics, 16(6):927–934,2010. ISSN 1077-2626. doi:10.1109/TVCG.2010.162. → pages 37, 47, 67, 199[117] M. C. Jones, I. R. Floyd, andM. B. Twidale. Patchwork prototyping withopen source software. Handbook of Research on Open Source Software, page 126,2007. doi:10.4018/978-1-59140-999-1.CH011. → page 190[118] S. Joshi, D. Contractor, K. Ng, P. M. Deshpande, and T. Hampp.Auto-grouping emails for faster e-discovery. Proc. VLDBEndowment, 4(12):1284–1294, 2011. doi:10.14778/3402755.3402762. → page 119[119] S. Kaasten, S. Greenberg, and C. Edwards. How people recognise previouslyseen web pages from titles, urls and thumbnails. In People and ComputersXVI-Memorable Yet Invisible, pages 247–265. Springer, 2002. → page 133[120] A. Kachkaev, J. Wood, and J. Dykes. Glyphs for exploring crowd-sourcedsubjective survey classification. In Computer Graphics Forum, volume 33,pages 311–320, 2014. doi:10.1111/cgf.12387. → pages 113, 164[121] E. Kerzner, L. A. Butler, C. Hansen, andM.Meyer. A shot at visualvulnerability analysis. Computer Graphics Forum, 34(3):391–400, 2015.doi:10.1111/cgf.12651. → pages 175, 178[122] E. Kerzner, S. Goodwin, J. Dykes, S. Jones, andM.Meyer. A framework forcreative visualization-opportunities workshops. IEEE Trans. Visualization216and Computer Graphics, 25(1):748–758, 2018. doi:10.1109/TVCG.2018.2865241.→ page 2[123] H. Kim, J. Oh, Y. Han, S. Ko, M. Brehmer, and B. C. Kwon. T_humbnails fordata stories: A survey of current practices. In IEEE Visualization Conference(VIS), pages 116–120, 2019. doi:10.1109/VISUAL.2019.8933773. → pages129, 135, 165[124] N.W. Kim, S. K. Card, and J. Heer. Tracing genealogical data with TimeNets.In Proc. Intern. Conf. on Advanced Visual Interfaces (AVI), pages 241–248, 2010.doi:10.1145/1842993.1843035. → page 61[125] S. Kim, S. Jeong, I. Woo, Y. Jang, R. Maciejewski, and D. S. Ebert. Data flowanalysis and visualization for spatiotemporal statistical data withouttrajectory information. IEEE Trans. Visualization and Computer Graphics, 24(3):1287–1300, 2018. doi:10.1109/TVCG.2017.2666146. → page 54[126] R. Kincaid. SignalLens: Focus+context applied to electronic time series.IEEE Trans. Visualization and Computer Graphics, 16(6):900–907, 2010.doi:10.1109/TVCG.2010.193. → page 61[127] G. Kindlmann and C. Scheidegger. An algebraic process for visualizationdesign. IEEE Trans. Visualization and Computer Graphics, 20(12):2181–2190,2014. doi:10.1109/TVCG.2014.2346325. → page 105[128] C. Kinkeldey, J.-D. Fekete, T. Blascheck, and P. Isenberg. BitConduite:Exploratory visual analysis of entity activity on the bitcoin network. IEEEComputer Graphics and Applications, 2021. doi:10.1109/MCG.2021.3070303. →page 188[129] B. P. Knijnenburg, M. C.Willemsen, Z. Gantner, H. Soncu, and C. Newell.Explaining the user experience of recommender systems. UserModeling andUser-Adapted Interaction, 22(4-5):441–504, 2012. doi:10.1007/s11257-011-9118-4.→ page 95[130] D. E. Knuth. Literate programming. T_heComputer Journal, 27(2):97–111, 1984.→ page 126[131] L. C. Koh, A. Slingsby, J. Dykes, and T. S. Kam. Developing and applying auser-centered model for the design and implementation of informationvisualization tools. In Proc. Int. Conf. Information Visualisation, pages 90–95,2011. doi:10.1109/IV.2011.32. → page 2217[132] L. C. Koh, A. Slingsby, J. Dykes, and T. S. Kam. Developing and applying auser-centered model for the design and implementation of informationvisualization tools. In Proc. Int. Conf. Information Visualisation, pages 90–95,2011. doi:10.1109/IV.2011.32. → page 190[133] M. Kompan andM. Bieliková. Content-based news recommendation. InProc. Int. Conf. Electronic Commerce andWeb Technologies, pages 61–72, 2010.doi:10.1007/978-3-642-15208-5_6. → page 86[134] N. Kong, J. Heer, andM. Agrawala. Perceptual guidelines for creatingrectangular treemaps. IEEE Trans. Visualization and Computer Graphics, 16(6):990–998, 2010. doi:10.1109/tvcg.2010.186. → page 199[135] J. A. Konstan and J. Riedl. Recommender systems: From algorithms to userexperience. UserModeling and User-Adapted Interaction, 22(1-2):101–123, 2012.doi:10.1007/s11257-011-9112-x. → page 95[136] B. L. Kovitz. Practical software requirements: a manual of content and style.Manning, 1998. → page 3[137] M.-J. Kraak. T_he space-time cube revisited from a geovisualizationperspective. In Proc. Intern. Cartographic Conference, pages 1988–1996, 2003.→ page 53[138] J. Krause, A. Dasgupta, J. Swartz, Y. Aphinyanaphongs, and E. Bertini. Aworkflow for visual diagnostics of binary classifiers using instance-levelexplanations. In Proc. IEEEConference on Visual Analytics Science andTechnology (VAST), pages 162–172, 2017. doi:10.1109/VAST.2017.8585720. →page 163[139] A. Kriebel and E. Murray. #MakeoverMonday: Improving howWe Visualize andAnalyze Data, One Chart at a Time. JohnWiley & Sons, 2018. → pages 103, 168[140] J. Kuutti, P. Saarikko, and R. E. Sepponen. Real time building zoneoccupancy detection and activity visualization utilizing a visitor countingsensor network. In Int. Conf. Remote Engineering and Virtual Instrumentation(REV), pages 219–224, 2014. doi:10.1109/REV.2014.6784260. → pages 17, 55[141] H. Lam and P. Baudisch. Summary thumbnails: readable overviews forsmall screen web browsers. In Proc. ACMSIGCHI Conf. onHuman Factors inComputing Systems (CHI), pages 681–690, 2005. doi:10.1145/1054972.1055066.→ pages 133, 140218[142] H. Lam, T. Munzner, and R. Kincaid. Overview use in multiple visualinformation resolution interfaces. IEEE Trans. Visualization and ComputerGraphics, 13(6):1278–1285, 2007. doi:10.1109/TVCG.2007.70583. → pages28, 47[143] H. Lam, M. Tory, and T. Munzner. Bridging from goals to tasks with designstudy analysis reports. IEEE Trans. Visualization and Computer Graphics, 24(1):435–445, 2017. doi:10.1109/TVCG.2017.2744319. → page 182[144] X. N. Lam, T. Vu, T. D. Le, and A. D. Duong. Approaching the cold startproblem in recommender systems. In Proc. Int. Conf. on UbiquitousInformationManagement and Communication, pages 208–211, 2008.doi:10.1109/ICMLA.2013.199. → page 88[145] J. A. Landay and B. A. Myers. Sketching interfaces: Towardmore humaninterface design. Computer, 34(3):56–64, 2001. doi:10.1109/2.910894. → page190[146] J. R. Landis and G. G. Koch. T_hemeasurement of observer agreement forcategorical data. Biometrics, pages 159–174, 1977. doi:10.2307/2529310. →page 114[147] Q. Le and T. Mikolov. Distributed representations of sentences anddocuments. In Proc. Int. Conf.Machine Learning (ICML), pages 1188–1196,2014. → page 108[148] A. Lee, D. Archambault, andM. Nacenta. Dynamic network plaid: A tool forthe analysis of dynamic networks. In ACMSIGCHI Conf. Human Factors inComputing Systems (CHI), pages 1–14, 2019. doi:10.1145/3290605.3300360. →page 61[149] J. H. Lee. Crowdsourcing music similarity judgments using mechanical turk.In Proc. Intl. Soc. forMusic Information Retrieval Conf. (ISMIR), pages 183–188,2010. → pages 96, 112[150] F. Lekschas, B. Peterson, D. Haehn, E. Ma, N. Gehlenborg, and H. Pfister.Peax: Interactive visual pattern search in sequential data usingunsupervised deep representation learning. Computer Graphics Forum, 39(3):167–179, 2020. doi:10.1101/597518. → page 199[151] M. Levy. Offline evaluation of recommender systems: all pain and no gain?In Proc. Int.Workshop on Reproducibility and Replication in RecommenderSystems Evaluation, pages 1–1, 2013. doi:10.1145/2532508.2532509. → page 95219[152] L. Li, V. L. Malave, A. Song, and A. Yu. Extracting human face similarityjudgments: Pairs or triplets? In Proc.Mtg. Cognitive Science Society (CogSci),2016. doi:10.1167/16.12.719. → pages 96, 109[153] W. Li, M. Agrawala, B. Curless, and D. Salesin. Automated generation ofinteractive 3d exploded view diagrams. ACMTransactions on Graphics (TOG),27(3):101, 2008. doi:10.1145/1360612.1360700. → page 48[154] Z. Li, S. Shi, and L. Zhang. Improving relevance judgment of web searchresults with image excerpts. In Proc. Int. Conf.WorldWideWeb, pages 21–30,2008. doi:10.1145/1367497.1367501. → page 134[155] J. Lin. Divergence measures based on the shannon entropy. IEEE Trans. onInformationT_heory, 37(1):145–151, 1991. doi:10.1109/18.61115. → page 106[156] J. Lin, E. Keogh, S. Lonardi, and B. Chiu. A symbolic representation of timeseries, with implications for streaming algorithms. In Proc. ACMSIGMODWorkshop on Research Issues in DataMining and Knowledge Discovery, DMKD’03, pages 2–11, 2003. doi:10.1145/882082.882086. → page 47[157] C. Liu, L. Xie, Y. Han, X. Yuan, et al. Autocaption: An approach to generatenatural language description from visualization automatically. In Proc. IEEEPacific Visualization Symp., pages 191–195, 2020.doi:10.1109/PacificVis48177.2020.1043. → pages 134, 202[158] S. Liu, X.Wang, M. Liu, and J. Zhu. Towards better analysis of machinelearning models: A visual analytics perspective. Visual Informatics, 1(1):48–56,2017. doi:10.1016/j.visinf.2017.01.006. → page 163[159] Z. Liu, J. T_hompson, A.Wilson, M. Dontcheva, J. Delorey, S. Grigg, B. Kerr,and J. Stasko. Data Illustrator: Augmenting vector design tools with lazydata binding for expressive visualization authoring. In Proc. ACMSIGCHIConf. Human Factors in Computing Systems (CHI), pages 1–13, 2018. → page195[160] Z. Liu, S. H. Zhan, and T. Munzner. Aggregated dendrograms for visualcomparison betweenmany phylogenetic trees. IEEE Trans. Visualization andComputer Graphics, 26(9):2732–2747, 2019. doi:10.1109/TVCG.2019.2898186.→ page 135[161] D. Lloyd and J. Dykes. Human centered approaches in geovisualizationdesign investigating multiple methods through a long term case study. IEEE220Trans. Visualization and Computer Graphics, 17(12):2498–2507, 2011.doi:10.1109/TVCG.2011.209. → pages 2, 4, 18, 19, 170, 178, 190, 191[162] P. Lops, M. De Gemmis, and G. Semeraro. Content-based recommendersystems: State of the art and trends. In Recommender SystemsHandbook,pages 73–105. Springer, 2011. → page 88[163] R. Maciejewski, S. Rudolph, R. Hafen, A. Abusalah, M. Yakout, M. Ouzzani,W. S. Cleveland, S. J. Grannis, and D. S. Ebert. A visual analytics approach tounderstanding spatiotemporal hotspots. IEEE Trans. Visualization andComputer Graphics, 16(2):205–220, 2010. doi:10.1109/TVCG.2009.100.→ page54[164] J. Mackinlay, P. Hanrahan, and C. Stolte. Showme: Automatic presentationfor visual analysis. IEEE Trans. Visualization and Computer Graphics, 13(6):1137–1144, 2007. → pages 87, 94[165] S. Madan, Z. Bylinskii, M. Tancik, A. Recasens, K. Zhong, S. Alsheikh,H. Pfister, A. Oliva, and F. Durand. Synthetically trained icon proposals forparsing and summarizing infographics. arXiv preprint arXiv:1807.10441, 2018.→ page 135[166] G. E. Marai. Activity-centered domain characterization for problem-drivenscientific visualization. IEEE Trans. Visualization and Computer Graphics, 24(1):913–922, 2017. doi:10.1109/TVCG.2017.2744459. → page 170[167] M.-C. Marcos, F. Gavin, and I. Arapakis. Effect of snippets on userexperience in web search. In Proc. Int. Conf. onHuman Computer Interaction,pages 1–8, 2015. doi:10.1145/2829875.2829916. → page 133[168] C. Martella, A. Miraglia, J. Frost, M. Cattani, andM. van Steen. Visualizing,clustering, and predicting the behavior of museum visitors. Pervasive andMobile Computing, 38:430–443, 2017. ISSN 15741192.doi:10.1016/j.pmcj.2016.08.011. → page 55[169] K. Mauge, K. Rohanimanesh, and J.-D. Ruvini. Structuring e-commerceinventory. In Proc. AnnualMeeting of the Association for ComputationalLinguistics: Long Papers-Volume 1, pages 805–814, 2012. → page 85[170] N. McCurdy, J. Dykes, andM.Meyer. Action design research andvisualization design. In IEEE VISWorkshop on Beyond Time and Errors - NovelEvaluationMethods for Visualization (BELIV), pages 10–18, 2016.doi:10.1145/2993901.2993916. → page 170221[171] D. McKay, G. Buchanan, N. Vanderschantz, C. Timpany, S. J. Cunningham,and A. Hinze. Judging a book by its cover: interface elements that affectreader selection of ebooks. In Proc. Australian Computer-Human InteractionConference, pages 381–390, 2012. doi:10.1145/2414536.2414597. → page 133[172] S. McKenna, D. Mazur, J. Agutter, andM. Meyer. Design activity frameworkfor visualization design. IEEE Trans. Visualization and Computer Graphics, 20(12):2191–2200, 2014. doi:10.1109/TVCG.2014.2346331. → pages 2, 4, 170[173] P. McLachlan, T. Munzner, E. Koutsofios, and S. North. LiveRAC: Interactivevisual exploration of systemmanagement time-series data. In ACMSIGCHIConf. Human Factors in Computing Systems (CHI), pages 1483–1492, 2008.doi:10.1145/1357054.1357286. → page 61[174] S. M. McNee, J. Riedl, and J. A. Konstan. Being accurate is not enough: howaccuracy metrics have hurt recommender systems. In Proc. CHI Conf.Extended Abstracts onHuman Factors in Computing Systems, pages 1097–1101,2006. doi:10.1145/1125451.1125659. → page 95[175] A. McNutt, G. Kindlmann, andM. Correll. Surfacing visualization mirages.In Proc. ACMSIGCHI Conf. Human Factors in Computing Systems (CHI), page1–16, 2020. doi:10.1145/3313831.3376420. → page 105[176] R. Melfi, B. Rosenblum, B. Nordman, and K. Christensen. MeasuringBuilding Occupancy Using Existing Network Infrastructure. In Int. GreenComputing Conf. andWorkshops (IGCC), pages 1–8, 2011.doi:10.1109/IGCC.2011.6008560. → pages 17, 22[177] D. Merejkowsky and R. Kelly. PyEnchant spellchecking library.https://pypi.org/project/pyenchant, 2020. → page 146[178] P. Meschenmoser, J. F. Buchmüller, D. Seebacher, M.Wikelski, and D. A.Keim. MultiSegVA: Using visual analytics to segment biologging time seriesonmultiple scales. IEEE Trans. Visualization and Computer Graphics, 27(2):1623–1633, 2020. → page 61[179] P.Messina, V. Dominguez, D. Parra, C. Trattner, and A. Soto. Content-basedartwork recommendation: Integrating painting metadata with neural andmanually-engineered visual features. UserModeling and User-AdaptedInteraction, 29(2):251–290, 2019. doi:10.1007/s11257-018-9206-9. → page 86222[180] M. Meyer and J. Dykes. Reflection on reflection in applied visualizationresearch. IEEEComputer Graphics and Applications, 38(6):9–16, 2018.doi:10.1109/MCG.2018.2874523. → page 4[181] M. D. Meyer, B.Wong, M. P. Styczynski, T. Munzner, and H. Pfister.Pathline: A tool for comparative functional genomics. Comput. Graph. Forum(Proc. EuroVis 2010), 29(3):1043–1052, 2010.doi:10.1111/j.1467-8659.2009.01710.x. → pages 28, 47[182] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributedrepresentations of words and phrases and their compositionality. In Proc.Adv. in Neural Information Processing Systems (NeurIPS), pages 3111–3119, 2013.→ page 104[183] Y. Ming, P. Xu, F. Cheng, H. Qu, and L. Ren. ProtoSteer: Steering deepsequence model with prototypes. IEEE Trans. Visualization and ComputerGraphics, 26(1):238–248, 2019. doi:10.1109/TVCG.2019.2934267. → page 163[184] F. Miranda, H. Doraiswamy, M. Lage, K. Zhao, B. Gonçalves, L. Wilson,M. Hsieh, and C. T. Silva. Urban Pulse: Capturing the rhythm of cities. IEEETrans. Visualization and Computer Graphics, 23(1):791–800, 2017.doi:10.1109/TVCG.2016.2598585. → pages 54, 184[185] V. O. Mittal, J. D. Moore, G. Carenini, and S. Roth. Describing complexcharts in natural language: A caption generation system. ComputationalLinguistics, 24(3):431–467, 1998. → page 134[186] A. Monreale, G. L. Andrienko, N. V. Andrienko, F. Giannotti, D. Pedreschi,S. Rinzivillo, and S.Wrobel. Movement data anonymity throughgeneralization. Trans. on Data Privacy, 3(2):91–121, 2010. → page 196[187] M. R. Montgomery, M. Gragnolati, K. A. Burke, and E. Paredes. Measuringliving standards with proxy variables. Demography, 37(2):155–174, 2000.doi:10.2307/2648118. → page 186[188] D. Moritz, C. Wang, G. L. Nelson, H. Lin, A. M. Smith, B. Howe, and J. Heer.Formalizing visualization design knowledge as constraints: Actionable andextensible models in draco. IEEE Trans. Visualization and Computer Graphics,25(1):438–448, 2018. → pages 87, 94[189] K. Morton, M. Balazinska, D. Grossman, R. Kosara, and J. Mackinlay. Publicdata and visualizations: How are Many Eyes and Tableau Public used for223collaborative analytics? ACMSIGMODRecord, 43(2):17–22, 2014.doi:10.1145/2694413.2694417. → page 135[190] Y. Mu, Q.Wei, G. Chen, and X. Guo. An iterative multi-criteria optimizationof product snippets enhanced by feature extraction from online reviews. InProc. Conf. Data Science and Knowledge Engineering for Sensing Decision Support,pages 545–552, 2018. doi:10.1142/9789813273238_0070. → page 133[191] T. Mühlbacher, H. Piringer, S. Gratzl, M. Sedlmair, andM. Streit. Openingthe black box: Strategies for increased user involvement in existingalgorithm implementations. IEEE Trans. Visualization and Computer Graphics,20(12):1643–1652, 2014. doi:10.1109/TVCG.2014.2346578. → page 163[192] T. Munzner. A nested process model for visualization design and validation.IEEE Trans. Visualization and Computer Graphics, 15(6):921–928, 2009.doi:10.1109/TVCG.2009.111. → pages 2, 4, 19, 21, 170[193] T. Munzner. Visualization analysis and design. CRC Press, 2014.doi:10.1201/b17511. → page 194[194] R. Nallapati, B. Zhou, C. dos Santos, G. Çağlar, and B. Xiang. Abstractivetext summarization using sequence-to-sequence RNNs and beyond. In Proc.SIGNLL Conf. on Comp. Natural Language Learning, pages 280–290, 2016.doi:10.18653/v1/K16-1028. → page 164[195] C. Neustaedter and P. Sengers. Autobiographical design in HCI research:designing and learning through use-it-yourself. In Proc. Designing InteractiveSystems Conference (DIS), pages 514–523, 2012. doi:10.1145/2317956.2318034.→ pages 186, 187[196] C. Nobre, N. Gehlenborg, H. Coon, and A. Lex. Lineage: Visualizingmultivariate clinical data in genealogy graphs. IEEE Trans. Visualization andComputer Graphics, 25(3):1543–1558, 2018. doi:10.1109/TVCG.2018.2811488. →page 48[197] C. G. Northcutt, K. A. Leon, and N. Chen. Comment ranking diversificationin forum discussions. In Proceedings of the Fourth (2017) ACMConference onLearning@ Scale, pages 327–330, 2017. doi:10.1145/3051457.3054016. → page144[198] M. Ogawa and K.-L. Ma. code_swarm: A design study in organic softwarevisualization. IEEE Trans. Visualization and Computer Graphics, 15(6):1097–1104, 2009. doi:10.1109/TVCG.2009.123. → page 184224[199] M. Oppermann and T. Munzner. Uncovering spatiotemporal dynamics fromnon-trajectory data. In Proc. CityVisWorkshop at IEEE VIS, pages 4–6, 2018.https://cityvis.io. → pages viii, 23[200] M. Oppermann and T. Munzner. Data-first visualization design studies. InIEEE VISWorkshop on Evaluation and Beyond -Methodological Approaches toVisualization (BELIV), pages 74–80, 2020.doi:10.1109/BELIV51497.2020.00016. → pages viii, 167, 188[201] M. Oppermann and T. Munzner. Ocupado: Visualizing location-basedcounts over time across buildings. Computer Graphics Forum, 39(3):127–138,2020. doi:10.1111/cgf.13968. → pages vi, 60, 62, 63, 169, 171, 174, 178[202] M. Oppermann and T. Munzner. VizSnippets: Compressing visualizationbundles into representative previews for browsing visualization collections.Accepted, IEEE Transactions on Visualization and Computer Graphics (Proc.of VIS), 2021. → page vii[203] M. Oppermann, T. Möller, andM. Sedlmair. Bike Sharing Atlas: Visualanalysis of bike-sharing networks. Intern. Journal of Transportation (IJT), 6(1):1–14, 2018. doi:10.14257/ijt.2018.6.1.01. → pages viii, 5, 54, 169, 171, 178, 196[204] M. Oppermann, R. Kincaid, and T. Munzner. VizCommender: Computingtext-based similarity in visualization repositories for content-basedrecommendations. IEEE Trans. Visualization and Computer Graphics, 27(2):495–505, 2021. doi:10.1109/TVCG.2020.3030387. → pagesvii, 8, 84, 125, 130, 146[205] M. Oppermann, L. Liu, and T. Munzner. TimeElide: Visual analysis ofnon-contiguous time series slices. Accepted, Proc. IEEE VIS Short Papers,2021. → page vii[206] M. M. Ouf, M. H. Issa, A. Azzouz, and A.-M. Sadick. Effectiveness of usingWiFi technologies to detect and predict building occupancy. Sustainablebuildings, 2:1–10, 2017. doi:10.1051/sbuild/2017005. → pages 22, 57[207] J. O’Shea, Z. Bandar, K. Crockett, and D. McLean. A comparative study oftwo short text semantic similarity measures. In Proc. Int. Symp. on Agent andMulti-Agent Systems: Technologies and Applications, pages 172–181, 2008.doi:10.1007/978-3-540-78582-8_18. → page 95[208] L. Pappalardo, P. Cintia, A. Rossi, E. Massucco, P. Ferragina, D. Pedreschi,and F. Giannotti. A public data set of spatio-temporal match events in soccer225competitions. Nature Scientific Data, 6(1):1–15, 2019.doi:10.1038/s41597-019-0247-7. → page 62[209] C. Partl, S. Gratzl, M. Streit, A. M.Wassermann, H. Pfister, D. Schmalstieg,and A. Lex. Pathfinder: Visual analysis of paths in graphs. Computer GraphicsForum, 35(3):71–80, 2016. doi:10.1111/cgf.12883. → page 48[210] G. A. Pavlopoulos, A.-L. Wegener, and R. Schneider. A survey ofvisualization tools for biological network analysis. BioDataMining, 1(1):1–11,2008. doi:10.1186/1756-0381-1-12. → page 194[211] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for wordrepresentation. In Proc. Conf. EmpiricalMethods in Natural LanguageProcessing (EMNLP), pages 1532–1543, 2014. doi:10.3115/v1/D14-1162. → page108[212] C. Perin, R. Vuillemot, and J.-D. Fekete. SoccerStories: A kick-off for visualsoccer analysis. IEEE Trans. Visualization and Computer Graphics, 19(12):2506–2515, 2013. doi:10.1109/TVCG.2013.192. → page 184[213] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, andL. Zettlemoyer. Deep contextualized word representations. In Proc. Conf.North American Chap. Assoc. for Computational Linguistics: Human LanguageTechnologies (NAACL-HLT), 2018. doi:10.18653/v1/N18-1202. → page 106[214] D. Phan, L. Xiao, R. Yeh, and P. Hanrahan. Flowmap layout. In Proc. IEEESymp. on Information Visualization (InfoVis), pages 219–224, 2005.doi:10.1109/INFVIS.2005.1532150. → page 53[215] C. Plaisant, R. Mushlin, A. Snyder, J. Li, D. Heller, and B. Shneiderman.LifeLines: Using visualization to enhance navigation and analysis of patientrecords. In Proc. AmericanMedical Informatic Association Annual FallSymposium, pages 308–312, 2003. → page 61[216] T. Polk, J. Yang, Y. Hu, and Y. Zhao. Tennivis: Visualization for tennis matchanalysis. IEEE Trans. Visualization and Computer Graphics, 20(12):2339–2348,2014. doi:10.1109/TVCG.2014.2346445. → page 184[217] A. Prouzeau, M. Dharshini, M. Balasubramaniam, J. Henry, N. Hoang, andT. Dwyer. Visual analytics for energy monitoring in the context of buildingmanagement. In Intern. Symp. on Big Data Visual and Immersive Analytics(BDVA), pages 1–9, 2018. doi:10.1109/BDVA.2018.8534026. → page 55226[218] M. Quadrana, P. Cremonesi, and D. Jannach. Sequence-awarerecommender systems. ACMComputing Surveys (CSUR), 51(4):1–36, 2018.doi:10.1145/3190616. → page 120[219] R. Rana, B. Kusy, J. Wall, andW. Hu. Novel activity classification andoccupancy estimation methods for intelligent HVAC (heating, ventilationand air conditioning) systems. Energy, 93:245–255, 2015. ISSN 0360-5442.doi:10.1016/j.energy.2015.09.002. → page 17[220] R. Řehůřek and P. Sojka. Software Framework for Topic Modelling withLarge Corpora. In Proc. LRECWorkshop onNewChallenges for NLP Frameworks,pages 45–50, 2010. http://is.muni.cz/publication/884893/en. → page 106[221] I. Renz, A. Ficzay, and H. Hitzler. Keyword extraction for textcharacterization. Natural language processing and information systems, 2003.→ page 146[222] S. Robertson. Understanding inverse document frequency: On theoreticalarguments for IDF. Journal of Documentation, 2004.doi:10.1108/00220410410560582. → pages 107, 146[223] J. Rogers, A. H. Patton, L. Harmon, A. Lex, andM.Meyer. Insights fromexperiments with rigor in an EvoBio design study. IEEE Trans. Visualizationand Computer Graphics, 2020. doi:10.1109/TVCG.2020.3030405. → page 124[224] J. Rudd, K. Stern, and S. Isensee. Low vs. high-fidelity prototyping debate.Interactions, 3(1):76–85, 1996. doi:10.1145/223500.223514. → page 190[225] B. Saket, D. Moritz, H. Lin, V. Dibia, C. Demiralp, and J. Heer. Beyondheuristics: Learning visualization design. Proc. IEEE VISWkshp. on Creation,Curation, Critique, and Conditioning of Principles and Guidelines in Visualization(VisGuides), 2018. → page 94[226] A. Sarikaya, M. Correll, L. Bartram, M. Tory, and D. Fisher. What do we talkabout when we talk about dashboards? IEEE Trans. Visualization andComputer Graphics, 25(1):682–692, 2019. doi:10.1109/TVCG.2018.2864903. →pages 25, 94, 124[227] S. Sarwar. Transactions from a bakery.https://www.kaggle.com/sulmansarwar/transactions-from-a-bakery, 2018. →page 62227[228] A. Satyanarayan and J. Heer. Lyra: An interactive visualization designenvironment. Computer Graphics Forum, 33(3):351–360, 2014.doi:10.1111/cgf.12391. → page 195[229] A. Satyanarayan, D. Moritz, K.Wongsuphasawat, and J. Heer. Vega-lite: Agrammar of interactive graphics. IEEE Trans. Visualization and ComputerGraphics, 23(1):341–350, 2016. doi:10.1109/TVCG.2016.2599030. → page 93[230] A. Satyanarayan, B. Lee, D. Ren, J. Heer, J. Stasko, J. T_hompson, M. Brehmer,and Z. Liu. Critical reflections on visualization authoring systems. IEEETrans. Visualization and Computer Graphics, 26(1):461–471, 2019.doi:10.1109/TVCG.2019.2934281. → page 194[231] A. Schofield, M. Magnusson, and D. Mimno. Pulling out the stops:Rethinking stopword removal for topic models. In Proc. Conf. EuropeanChapter of the Association for Computational Linguistics, pages 432–436, 2017.→ page 100[232] A. D. Schon. T_heReflective Practitioner: How ProfessionalsT_hink In Action. BasicBooks, 1984. → page 4[233] H.-J. Schulz. Treevis. net: A tree visualization reference. IEEEComputerGraphics and Applications, 31(6):11–15, 2011. doi:10.1109/MCG.2011.103. →page 135[234] M. Sedlmair. Design study contributions come in different guises: Sevenguiding scenarios. In IEEE VISWorkshop on Beyond Time and Errors - NovelEvaluationMethods for Visualization (BELIV), pages 152–161, 2016.doi:10.1145/2993901.2993913. → pages 168, 178, 182[235] M. Sedlmair, P. Isenberg, D. Baur, and A. Butz. Information visualizationevaluation in large companies: Challenges, experiences andrecommendations. Information Visualization, 10(3):248–266, 2011.doi:10.1177/1473871611413099. → page 170[236] M. Sedlmair, M. Meyer, and T. Munzner. Design study methodology:Reflections from the trenches and the stacks. IEEE Trans. Visualization andComputer Graphics, 1(12):2431–2440, 2012. doi:10.1109/TVCG.2012.213. →pages 1, 2, 3, 6, 9, 56, 168, 169, 178, 180, 186, 191, 194[237] M. Sedlmair, C. Heinzl, S. Bruckner, H. Piringer, and T. Möller. Visualparameter space analysis: A conceptual framework. IEEE Trans.228Visualization and Computer Graphics, 20(12):2161–2170, 2014.doi:10.1109/TVCG.2014.2346321. → page 163[238] E. Segel and J. Heer. Narrative visualization: Telling stories with data. IEEETrans. Visualization and Computer Graphics, 16(6):1139–1148, 2010.doi:10.1109/TVCG.2010.179. → page 200[239] H. Seifi, F. Fazlollahi, M. Oppermann, J. A. Sastrillo, J. Ip, A. Agrawal,G. Park, K. J. Kuchenbecker, and K. E. MacLean. Haptipedia: Acceleratinghaptic device discovery to support interaction & engineering design. InACMSIGCHI Conf. Human Factors in Computing Systems (CHI), pages 1–12,2019. doi:10.1145/3290605.3300788. → page 9[240] H. Seifi, M. Oppermann, J. Bullard, K. E. MacLean, and K. J. Kuchenbecker.Capturing experts’ mental models to organize a collection of haptic devices:Affordances outweigh attributes. In ACMSIGCHI Conf. Human Factors inComputing Systems (CHI), page 1–12, 2020. doi:10.1145/3313831.3376395. →page 9[241] G. Shani and A. Gunawardana. Evaluating recommendation systems. InRecommender SystemsHandbook, pages 257–297. Springer, 2011. → page 95[242] S. Sharan. Strava bike rides.https://www.kaggle.com/synapsizeus/strava-bike-rides-from-2020-gpx-files,2020. → page 62[243] H. Shiravi, A. Shiravi, and A. A. Ghorbani. A survey of visualization systemsfor network security. IEEE Trans. Visualization and Computer Graphics, 18(8):1313–1329, 2011. doi:10.1109/TVCG.2011.144. → page 194[244] B. Shneiderman. T_he eyes have it: A task by data type taxonomy forinformation visualizations. In Proc. IEEE Symp. Visual Languages, pages336–343, 1996. doi:10.1109/VL.1996.545307. → page 92[245] C. Sievert and K. Shirley. LDAVis: A method for visualizing and interpretingtopics. In Proc.Workshop on Interactive Language Learning, Visualization, andInterfaces, pages 63–70, 2014. doi:10.3115/v1/W14-3110. → page 289[246] A. Slingsby and J. Dykes. Experiences in involving analysts in visualisationdesign. In IEEE VISWorkshop on Beyond Time and Errors-Novel EvaluationMethods for Visualization (BELIV), pages 1–4, 2012.doi:10.1145/2442576.2442577. → page 2229[247] H. Song and D. A. Szafir. Where’s my data? evaluating visualizations withmissing data. IEEE Trans. Visualization and Computer Graphics, 25(1):914–924,2018. doi:10.1109/TVCG.2018.2864914. → page 43[248] Y. Song, M. Redi, J. Vallmitjana, and A. Jaimes. To click or not to click:Automatic selection of beautiful thumbnails from videos. In Proc. ACM Int.Conf. Information and KnowledgeManagement, pages 659–668, 2016.doi:10.1145/2983323.2983349. → page 133[249] E. Spyromitros-Xioufis, S. Papadopoulos, A. L. Ginsca, A. Popescu,Y. Kompatsiaris, and I. Vlahavas. Improving diversity in image search viasupervised relevance scoring. In Proc. ACM Int. Conf.Multimedia Retrieval,pages 323–330, 2015. doi:10.1145/2671188.2749334. → page 144[250] B. Sriram, D. Fuhry, E. Demir, H. Ferhatosmanoglu, andM. Demirbas.Short text classification in twitter to improve information filtering. In Proc.ACM Int. Conf. on Research andDevelopment in Information Retrieval (SIGIR),pages 841–842, 2010. doi:10.1145/1835449.1835643. → page 102[251] D. Srivastava, R.Wadhvani, andM. Gyanchandani. A review: color featureextraction methods for content based image retrieval. Int. JournalComputational Engineering &Management, 18(3):9–13, 2015. → page 141[252] A. Stoffel, H. Strobelt, O. Deussen, and D. A. Keim. Document thumbnailswith variable text scaling. Computer Graphics Forum, 31(3pt3):1165–1173, 2012.doi:10.1111/j.1467-8659.2012.03109.x. → page 133[253] C. Stolte, R. Bosch, P. Hanrahan, andM. Rosenblum. Visualizingapplication behavior on superscalar processors. In Proc. IEEE Symp. onInformation Visualization (InfoVis), pages 10–17, 1999.doi:10.1109/INFVIS.1999.801852. → page 61[254] H. Strobelt, D. Oelke, C. Rohrdantz, A. Stoffel, D. A. Keim, and O. Deussen.Document cards: A top trumps visualization for documents. IEEE Trans.Visualization and Computer Graphics, 15(6):1145–1152, 2009.doi:10.1109/TVCG.2009.139. → page 133[255] H. Strobelt, S. Gehrmann, H. Pfister, and A. M. Rush. LSTMVis: A tool forvisual analysis of hidden state dynamics in recurrent neural networks. IEEETrans. Visualization and Computer Graphics, 24(1):667–676, 2017.doi:10.1109/TVCG.2017.2744158. → page 163230[256] R. Strohmaier, G. Sprung, A. Nischelwitzer, and S. Schadenbauer. Usingvisitor-flow visualization to improve visitor experience in museums andexhibitions. Museums and theWeb, pages 1–6, 2015. → page 55[257] U. H. Syeda, P. Murali, L. Roe, B. Berkey, andM. A. Borkin. Design study"lite" methodology: Expediting design studies and enabling the synergy ofvisualization pedagogy and social good. In ACMSIGCHI Conf. HumanFactors in Computing Systems (CHI), pages 1–13, 2020.doi:10.1145/3313831.3376829. → pages 169, 179[258] D. Szafir, M. Correll, andM. Gleicher. Task-driven evaluation ofaggregation in time series visualization. In ACMSIGCHI Conf. HumanFactors in Computing Systems (CHI), pages 551–560, 2014.doi:10.1145/2556288.2557200. → page 62[259] J. Teevan, E. Cutrell, D. Fisher, S. M. Drucker, G. Ramos, P. André, andC. Hu. Visual snippets: summarizing web pages for search and revisitation.In Proc. ACMSIGCHI Conf. onHuman Factors in Computing Systems (CHI),pages 2023–2032, 2009. doi:10.1145/1518701.1519008. → pages 133, 140[260] A.T_hudt, J. Walny, T. Gschwandtner, J. Dykes, and J. Stasko. Explorationand explanation in data-driven storytelling. InData-Driven Storytelling,pages 59–83. AK Peters/CRC Press, 2018. → page 200[261] N. Tintarev and J. Masthoff. A survey of explanations in recommendersystems. In IEEE Int. Conf. on Data Engineering, pages 801–810, 2007.doi:10.1109/ICDEW.2007.4401070. → page 133[262] P. Tirilly, X. Mu, C. Huang, I. Xie, W. Jeong, and J. Zhang. Image similarityas assessed by users: A quantitative study. Proc. American Society forInformation Science and Technology, 49(1):1–10, 2012.doi:10.1002/meet.14504901180. → page 96[263] M. Tohidi, W. Buxton, R. Baecker, and A. Sellen. User sketches: a quick,inexpensive, and effective way to elicit more reflective user feedback. In Proc.Nordic Conf. onHuman-Computer Interaction, pages 105–114, 2006.doi:10.1145/1182475.1182487. → page 190[264] C. Tominski, S. Gladisch, U. Kister, R. Dachselt, and H. Schumann. A surveyon interactive lenses in visualization. Proc. EuroVis (STARs), 3(2), 2014.doi:10.2312/eurovisstar.20141172. → page 61231[265] W. B. Towne, C. P. Rosé, and J. D. Herbsleb. Measuring similarity similarly:LDA and human perception. ACMTrans. Intelligent Systems and Technology(TIST), 8(1):1–28, 2016. doi:10.1145/2890510. → page 95[266] A. Vall, M. Dorfer, H. Eghbal-Zadeh, M. Schedl, K. Burjorjee, andG.Widmer. Feature-combination hybrid recommender systems forautomatedmusic playlist continuation. UserModeling and User-AdaptedInteraction, 29(2):527–572, 2019. doi:10.1007/s11257-018-9215-8. → page 86[267] J. J. VanWijk. Bridging the gaps. IEEEComputer Graphics and Applications, 26(6):6–9, 2006. doi:10.1109/MCG.2006.120. → page 190[268] J. J. VanWijk and E. R. Van Selow. Cluster and calendar based visualizationof time series data. In Proc. IEEE Symp. on Information Visualization (InfoVis),pages 4–9, 1999. doi:10.1109/INFVIS.1999.801851. → pages 53, 184[269] R. R. Varior, M. Haloi, and G.Wang. Gated siamese convolutional neuralnetwork architecture for human re-identification. In European Conf. onComputer Vision, pages 791–808, 2016. → pages 142, 143[270] M. Vartak, S. Huang, T. Siddiqui, S. Madden, and A. Parameswaran.Towards visualization recommendation systems. ACMSIGMODRecord, 45(4):34–39, 2017. → page 87[271] E. Verbree, S. van der Spek, E. Kalogianni, R. Sileryte, M. N. Lam, K. Zhou,andM. van der Ham. PassiveWiFi monitoring of the rhythm of the campus.In Proc. AGILE Intern. Conf. Geographic Information Science, 2015. → pages17, 55[272] H. Verma, H. S. Alavi, and D. Lalanne. Studying space use: Bringing HCItools to architectural projects. In ACMSIGCHI Conf. Human Factors inComputing Systems (CHI), pages 3856–3866, 2017.doi:10.1145/3025453.3026055. → page 17[273] F. B. Viegas, M.Wattenberg, F. van Ham, J. Kriss, andM.McKeon.ManyEyes: A site for visualization at internet scale. IEEE Trans. Visualizationand Computer Graphics, 13(6):1121–1128, 2007. doi:10.1109/TVCG.2007.70577.→ pages 85, 135[274] R. Vliegendhart, M. Larson, and J. A. Pouwelse. Discovering userperceptions of semantic similarity in near-duplicate multimedia files. InProc. Int.Workshop on CrowdsourcingWeb Search (CrowdSearch), pages 54–58,2012. → page 88232[275] T. Von Landesberger, S. Bremm, N. Andrienko, G. Andrienko, andM. Tekušová. Visual analytics methods for categoric spatio-temporal data.In Proc. IEEEConf. Visual Analytics Science and Technology (VAST), pages183–192, 2012. doi:10.1109/VAST.2012.6400553. → pages 54, 196[276] J. S. Walker, M.W. Jones, R. S. Laramee, O. R. Bidder, H. J. Williams,R. Scott, E. L. Shepard, and R. P.Wilson. TimeClassifier: a visual analyticsystem for the classification of multi-dimensional time series data. T_heVisual Computer, 31(6):1067–1078, 2015. doi:10.1007/s00371-015-1112-0. →page 61[277] J. Walny, S. Huron, and S. Carpendale. An exploratory study of datasketching for visual representation. Computer Graphics Forum, 34(3):231–240,2015. doi:10.1111/cgf.12635. → pages 2, 190[278] Z.Wang and J. Gemmell. Clean living: Eliminating near-duplicates inlifetime personal storage. Technical report, MSR-TR-2006-30, 2006. →page 119[279] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image qualityassessment: from error visibility to structural similarity. IEEE Trans. ImageProcessing, 13(4):600–612, 2004. doi:10.1109/TIP.2003.819861. → pages142, 143[280] J. Wexler, M. Pushkarna, T. Bolukbasi, M.Wattenberg, F. Viégas, andJ. Wilson. T_heWhat-If Tool: Interactive probing of machine learningmodels.IEEE Trans. Visualization and Computer Graphics, 26(1):56–65, 2019.doi:10.1109/TVCG.2019.2934619. → page 163[281] R. W.White and R. A. Roth. Exploratory search: Beyond the query-responseparadigm. Synthesis Lectures on Information Concepts, Retrieval, and Services, 1(1):1–98, 2009. → page 85[282] B. M.Wildemuth, G. Marchionini, T. Wilkens, M. Yang, G. Geisler,B. Fowler, A. Hughes, and X. Mu. Alternative surrogates for video objects ina digital library: users’ perspectives on their relative usability. In Int. Conf. onT_heory and Practice of Digital Libraries, pages 493–507, 2002. → page 133[283] B. M.Wildemuth, G. Marchionini, X. Fu, J. S. Oh, andM. Yang. T_heusefulness of multimedia surrogates for making relevance judgments aboutdigital video objects. Information Processing &Management, 56(6):102091, 2019.doi:10.1016/j.ipm.2019.102091. → pages 134, 161233[284] K.Williams, A. Bigelow, and K. Isaacs. Visualizing a moving target: Adesign study on task parallel programs in the presence of evolving data andconcerns. IEEE Trans. Visualization and Computer Graphics, 26(1):1118–1128,2019. doi:10.1109/TVCG.2019.2934285. → page 178[285] A. A. Winecoff, F. Brasoveanu, B. Casavant, P. Washabaugh, andM. Graham.Users in the loop: a psychologically-informed approach to similar itemretrieval. In Proc. ACMConf. Recommender Systems (RecSys), pages 52–59,2019. doi:10.1145/3298689.3347047. → pages 96, 112[286] J. A. Wise, J. J. T_homas, K. Pennock, D. Lantrip, M. Pottier, A. Schur, andV. Crow. Visualizing the non-visual: Spatial analysis and interaction withinformation from text documents. In Proc. IEEE Symp. on InformationVisualization (InfoVis), pages 51–58, 1995. doi:10.1109/INFVIS.1995.528686. →page 61[287] I. H.Witten, D. Bainbridge, and D. M. Nichols. How to Build a Digital Library.Morgan Kaufmann, 2009. → page 85[288] Y. L. Wong, K. Madhavan, and N. Elmqvist. Towards characterizing domainexperts as a user group. In IEEE VISWorkshop on Evaluation andBeyond-Methodological Approaches for Visualization, pages 1–10, 2018.doi:10.1109/BELIV.2018.8634026. → page 187[289] K.Wongsuphasawat, Z. Qu, D. Moritz, R. Chang, F. Ouk, A. Anand,J. Mackinlay, B. Howe, and J. Heer. Voyager 2: Augmenting visual analysiswith partial view specifications. In Proc. ACMSIGCHI Conf. Human Factors inComputing Systems (CHI), pages 2648–2659, 2017.doi:10.1145/3025453.3025768. → page 94[290] J. Wood, D. Badawood, J. Dykes, and A. Slingsby. BallotMaps: Detectingname bias in alphabetically ordered ballot papers. IEEE Trans. Visualizationand Computer Graphics, 17(12):2384–2391, 2011. doi:10.1109/TVCG.2011.174. →page 184[291] J. Wood, A. Slingsby, and J. Dykes. Visualizing the dynamics of London’sbicycle-hire scheme. Cartographica: T_he International Journal for GeographicInformation and Geovisualization, 46(4):239–251, 2011.doi:10.3138/carto.46.4.239. → page 47[292] J. Wood, R. Beecham, and J. Dykes. Moving beyond sequential design:Reflections on a rich multi-channel approach to data visualization. IEEE234Trans. Visualization and Computer Graphics, 20(12):2171–2180, 2014.doi:10.1109/TVCG.2014.2346323. → pages 56, 178[293] P. Woodall. T_he data repurposing challenge: new pressures from dataanalytics. Journal of Data and Information Quality (JDIQ), 8(3-4):1–4, 2017.doi:10.1145/3022698. → page 186[294] A.Wu,W. Tong, T. Dwyer, B. Lee, P. Isenberg, and H. Qu. MobileVisFixer:Tailoring web visualizations for mobile phones leveraging an explainablereinforcement learning framework. IEEE Trans. Visualization and ComputerGraphics, 2020. doi:10.1109/TVCG.2020.3030423. → page 135[295] L. Wu, I. E.-H. Yen, K. Xu, F. Xu, A. Balakrishnan, P.-Y. Chen, P. Ravikumar,andM. J. Witbrock. Wordmover’s embedding: FromWord2Vec todocument embedding. In Proc. Conf. EmpiricalMethods in Natural LanguageProcessing (EMNLP), pages 4524–4534, 2018. doi:10.18653/v1/D18-1482. →page 108[296] Y. Wu, F.Wei, S. Liu, N. Au,W. Cui, H. Zhou, and H. Qu. OpinionSeer:interactive visualization of hotel customer feedback. IEEE Trans.Visualization and Computer Graphics, 16(6):1109–1118, 2010.doi:10.1109/TVCG.2010.183. → page 184[297] X. Yang and S. Ergan. Evaluation of visualization techniques for use byfacility operators during monitoring tasks. Automation in Construction, 44:103–118, 2014. doi:10.1016/j.autcon.2014.03.023. → page 48[298] Y. Yao and F. M. Harper. Judging similarity: A user-centric study of relateditem recommendations. In Proc. ACMConf. Recommender Systems (RecSys),pages 288–296, 2018. doi:10.1145/3240323.3240351. → page 96[299] K.-P. Yee, K. Swearingen, K. Li, andM. Hearst. Faceted metadata for imagesearch and browsing. In Proc. ACMSIGCHIConf.Human Factors in ComputingSystems (CHI), pages 401–408, 2003. doi:10.1145/642611.642681. → page 85[300] V. Yoghourdjian, T. Dwyer, K. Klein, K. Marriott, andM.Wybrow. Graphthumbnails: Identifying and comparing multiple graphs at a glance. IEEETrans. Visualization and Computer Graphics, 24(12):3081–3095, 2018.doi:10.1109/TVCG.2018.2790961. → page 135[301] A. Zenebe and A. F. Norcio. Representation, similarity measures andaggregation methods using fuzzy sets for content-based recommender235systems. Fuzzy Sets and Systems, 160(1):76–94, 2009.doi:10.1016/j.fss.2008.03.017. → page 86[302] J. Zhang, Y. Wang, P. Molino, L. Li, and D. S. Ebert. Manifold: Amodel-agnostic framework for interpretation and diagnosis of machinelearning models. IEEE Trans. Visualization and Computer Graphics, 25(1):364–373, 2018. doi:10.1109/TVCG.2018.2864499. → page 163[303] T. Zhang and R. Samadani. Automatic generation of music thumbnails. InProc. Int. Conf. onMultimedia and Expo, pages 228–231, 2007.doi:10.1109/ICME.2007.4284628. → page 133[304] J. Zhao, F. Chevalier, E. Pietriga, and R. Balakrishnan. Exploratory analysisof time-series with ChronoLenses. IEEE Trans. Visualization and ComputerGraphics, 17(12):2422–2431, 2011. doi:10.1109/TVCG.2011.195. → pages 49, 61[305] R. Zhou, S. Khemmarat, L. Gao, J. Wan, and J. Zhang. How YouTube videosare discovered and its impact on video views. Multimedia Tools andApplications, 75(10):6035–6058, 2016. doi:10.1007/s11042-015-3206-0. →page 85236A Appendix: OcupadoT_his appendix contains supplemental material related to the Ocupado project de-scribed in Chapter 2.A.1 StakeholdersWe ultimately selected five focus domains for continued engagement: space plan-ning, building management, custodial services, classroommanagement, and dataquality control. At the beginning of the project, and in pursuit of task abstractions,we reached out to a broader group of potential stakeholders as shown in Table A.1.We learned about the needs of these user groups and the limitations following fromthe choice ofWiFi devices as a proxy for occupants.In addition, the SBS CEO, Stefan Storey, took the role of a promoter and madevigorous efforts to present Ocupado and to assess potential usage scenarios withexternal clients, which are summarized in Table A.2.Table A.3 and Table A.4 provide further details about domain-specific questionsand amapping between stakeholders and the four high-level abstract tasks.237Focus domains (1) Custodial services 6University 1-A 4University 1-B 1Company 1 1(2) Buildingmanagement 3University 1-A 3(3) Space planning 4University 1-A 3University 2 1(4) Classroommanagement 2University 1-A 2(5) Data quality controlSBS Continuous feedbackWinnowing Physical security (University 1-A) 1Risk management (University 1-A) 1Energy management (University 1-B) 3Transportation authority 1Table A.1: Number of demos and informal interviews conducted by us together with SensibleBuilding Science (SBS).238Number of potential customers,sponsors, and partnersUniversities 9Government agencies 6Companies 5Company 1 1Number of demos in each sector Innovation program 3Technology partner 6Facilities planning andmangement 12Energy management 3Table A.2: Number of demos and informal interviews conducted by SBS without our direct in-volvement.239TaskCustodial services BuildingmanagementSpaceplanningClassroommanagementData qualitycontrolCustodial head Custodial managerT1. Confirm observations thathave beenmade—T2. Monitor the current/recentutilization rateT3. Communicate space usageand justify decisions— —T4. Validate the data (qualitycontrol)— — — — —TableA.3:Match between stakeholders and abstract tasks:240TableA.4:Match between stakeholder questions, given in domain-specific language, and abstract tasks:241A.2 Ocupado InterfacesWe started by implementing a sandbox prototyping environment to rapidly exploreand evaluate the design space of visual encodings (see Fig. A.2 to Fig. A.7). Afterobtaining a more precise understanding of our focus domains, we created fourinterfaces that address multiple levels of data granularity in both space and time:campus explorer (Fig. A.8 to Fig. A.23), building long-term interface, building short-terminterface, region compare interface.Table A.5 shows a mapping between Ocupado interfaces and abstract tasks, andFig. A.1 provides a detailed categorization of all interfaces according to spatial andtemporal dimensions.TableA.5:Match between interfaces and abstract tasks:242FigureA.1:CategorizationofOcupado interfacesbasedonspatiotemporaldatagranularities.Weusethenotationfor space:time combinationsshown in Fig. 2.1. For example, ZF-All:SA (compare all zones on afloor : short-termaggregated) applies to the scenarioA custodial headwants tomonitor the average utilization of all the rooms on the first floor in the past 6 hours. The asterisk (*) symbol indicates a wildcard that can be replacedby any of the available options, such as ZB-* for one, few, or all zones in a building.243FigureA.2: Sandbox overview. Users can choose absolute y-scales to compare device counts of different buildings. BC-All:LI244FigureA.3: Sandbox overview. Smaller university campus. BC-All:LI245FigureA.4: Sandbox overview. Heatmapwith absolute scales to facilitate cross-building comparisons. BC-All:LA246FigureA.5: Sandbox, building details. Average per-floor device count onweekdays. One line corresponds to one floor. FB-All:LA247FigureA.6: Sandbox, floor details. (a) interactive floor plan for spatial context and to navigate to specific zones; (b) superimposed average dayprofiles for all zones on this floor. ZF-All:LA248Figure A.7: Sandbox, zone details. Zone detail view includes a zoomable binned timeline to show the absolute device count over time.ZF-One:LI249Figure A.8: Campus explorer. (a) Control panel for filter and display settings; (b) Region selector shown as quantitative and categorical datastripes to provide a high-level overview ZC-All:*A; (c) Region subset view provides mid-level details for selected regions ZC-Few:LI . Users canchoose between different visual encodings for the region subset view. Clicking on one region opens amodal windowwith low-level details.250FigureA.9: Campus explorer, control panel. Users can choose between different visual encoding idioms for the region subset view.251FigureA.10: Campus explorer, control panel. Regions can be filtered by space type.252FigureA.11: Campus explorer, control panel. Time range filter (contiguous)253FigureA.12: Campus explorer, control panel. Advanced temporal filter for discrete time slices.254FigureA.13: Campus explorer, control panel. Activity patterns provideone-click shortcuts to complex combinations of actions. They can controlspatial and temporal filter criteria, the visual encoding of the region subset view, and the selectionwindow in the region overview.255Figure A.14: Campus explorer, control panel. We integrated a small widget that shows a textual summary of all active filters because customselections would not be immediately visible due to themany different options that are hidden in dropdownmenus.256FigureA.15: Campus Explorer, region subset view. Regions are filtered by building (DMP) and displayed as sparklines (ZB-All:LI ).257FigureA.16: Campus Explorer, region selector. Thewidth of the data stripes (ZC-All:*A) is adjustable and users can enable or disable variables.258FigureA.17: Campus Explorer, region subset view. Box-plot-bars for discrete time intervals, such asMondays between 2-8pm. ZB-Few:LA259Figure A.18: Campus Explorer, region subset view. Discrete time intervals visualized as superimposed line charts. A region gets automaticallyhighlighted in both viewswhenwe hover over it. ZC-Few:LI260FigureA.19: Campus Explorer, region subset view. Aggregated sessions are presented as line chartswith confidence bands to answer questionssuch asWhen do people usually leave onweekday evenings? ZC-Few:LA261Figure A.20: Campus explorer, region subset view. Absolute scales ensure consistent axes and enable direct comparisons of device counts be-tween regions. Users canglobally choosebetween independentandabsolutey-scales thatareapplied toall temporal visualizations (sparklines,box-plot-bars, superimposed line charts). ZC-Few:LA262FigureA.21:CampusExplorer, regiondetail view. (a)Zoomablebinnedtimeserieschartwithconfidencebands (ZF-One:LI ). (b) Interactivefloorplan to provide a spatial context and to navigate between close regions. (c) Typical day profiles show aggregated device counts (ZF-One:LA).263FigureA.22: Campus Explorer, region detail view. Example for another zone. ZF-One:L*264Figure A.23: Campus Explorer, region detail view. The detail view of a building is divided into two tabs: the first tab shows all floor plans( [ZB,FB]-All ) and the second tab contains a binned time series chart and typical day profiles, similar to floor or zone detail views.265Figure A.24: Building Long-Term interface. (a) Interactive floor plans for selecting regions of interest (ZB-All ). (b) Per-floor rollups showingaverage utilization (FB-All:LA). (c) Each selected zone shown as sparklines (ZB-Few:LI ).266FigureA.25: Building Long-Term interface. Selected zones are shown as typical day profiles. ZB-Few:LA267FigureA.26: BuildingRecent interface. Floor plans are arranged in a grid layout to fit on screen and live device counts are displayed as superim-posed circle symbols ( [ZF,ZB]-All:S*). The left sidebar shows the aggregated per-floor usage for a typical day compared to the current devicecount (FB-All:LA).268Figure A.27: Building Recent interface. Example for another building with aggregated per-floor usage in the left column (FB-All:LA) and livedevice counts superimposed on floor plans in the right column (ZB-All:SI ).269FigureA.28: BuildingRecent interface. Users can switchbetweenhistogramsand line charts in the left sidebar. This screenshot showsper-flooraggregate trend charts with device counts over past 12 hours (FB-All:SI ) and prediction for the next 3 hours (red dashed line).270Figure A.29: Building Recent Interface. Example for another building with per-floor trend charts in the left column (FB-All:SI ) and live devicecounts in the right column (ZB-All:SI ).271Figure A.30: Building Recent interface. The underlying variable for the glyphs can be changed in the control panel. For example, analysts canchoose to show the maximum device count during the last 12 hours instead of the live device count. Average device count is the third option.[ZF,ZB]-All:SA272Figure A.31: Building Recent interface. The range slider in the top right corner can be used to select a shorter time window; for example, toselect the average device count between 8am - 12pm today. [ZF,ZB]-All:SA273FigureA.32: RegionCompare interface. The visual query builder is the key component that aids comparisons along twodimensions: space andtime. For each query, users can select a building, floor or zone and a specific time period. FB-Few274Figure A.33: Region Compare interface. The results are displayed inmultiple views: (a) Interactive zoomable line chart to show the raw devicecounts over time (ZB-Few:LI ); (b) boxplots to summarize the distribution of counts (ZB-Few:LA); (c) trend charts to facilitate a high-level com-parison (ZB-Few:LA); and (d) line charts with aggregated counts to illustrate the average daily and weekly utilization (ZB-Few:LA). We allowa maximum of five time series or regions to avoid visual clutter. Queries can be added and removed dynamically and all views are updatedaccordingly.275B Appendix: TimeElideT_his appendix contains supplemental material related to the TimeElide projectdescribed in Chapter 3.B.1 DatasetsWe usedmultiple datasets from different domains and with varying characteristicsduring the development of TimeElide, andmade them available in the online demo.T_he datasets are summarized in Table B.1 below.TableB.1: Summary of TimeElide datasets.276T_hree datasets are publicly available:1. Bakery data:https://kaggle.com/sulmansarwar/transactions-from-a-bakery2. Soccer data:https://doi.org/10.1038/s41597-019-0247-73. Bike data:https://kaggle.com/synapsizeus/strava-bike-rides-from-2020-gpx-filesB.2 Analysis Questions andTasksStakeholder questions related to the analysis of building occupancy during specifictime periods motivated the investigation originally. We generalized our approachand included additional datasets and typical analysis questions. A match betweenthese domain-specific questions and abstract tasks (T1-T5) is shown in Table B.2.277TableB.2:Match between domain-specific questions and abstract tasks.Match, Partialmatch, NomatchB.3 TimeElide InterfaceWe provide supplementary information about available settings and a set of high-resolution screenshots of TimeElide.278B.3.1 Global andChart-Specific SettingsTimeElide supports both global and chart-specific settings, as shown in Table B.3.Global settings remain active when the user changes the chart type while chart-specific settings control only the representation of the currently active chart type.T_he default chart settings are specified in JSON format and can be changed interac-tively through the user interface.Table B.3: Overview of global and chart-specific settings that are supported in TimeElide.Supported, Not supported279B.3.2 TimeElide ScreenshotsFigureB.1: TimeElidewelcomemodal window: introduce data and slicing options.FigureB.2: TimeElidewelcomemodal window: introduce visual encodings.280FigureB.3: TimeElide data source page.FigureB.4: TimeElide slicing options: users can choose tomanually specify desired slices or letthe tool automatically detect existing slices.281FigureB.5: TimeElide sparkboxes and tooltip overlay for one slice: soccer player in-match activ-ity.FigureB.6: TimeElide stepped area chart: average soccer player in-match activity.282FigureB.7: TimeElide heat stripes: average soccer player in-match activity.FigureB.8: TimeElide 2D heatmap, absolute time: soccer player in-match activity.283FigureB.9: TimeElide 2D heatmap, normalized duration: soccer player in-match activity.FigureB.10: TimeElide sparkboxes: building occupancy onweekends.284FigureB.11: TimeElide stepped area chart: average building occupancy onweekends.FigureB.12: TimeElidemulti-series line chart: building occupancy onweekends.285FigureB.13: TimeElide bandedmulti-series line chart: building occupancy onweekends.286FigureB.14: Building occupancy onweekends shownas sparkboxes at three different zoom lev-els (a) - (c). TimeElide supports interactivepanningandzooming for all visual encodings exceptfor themulti-series line chart and the bandedmulti-series line chart.287FigureB.15: Sparkboxes: different choices for the color palette can shift the focus to selectivelyemphasize fine or coarse structures in the data in order to support different tasks.288C Appendix: VizCommenderT_his appendix contains supplemental material related to the VizSnippets projectdescribed in Chapter 4.C.1 Model InvestigationIn addition to the crowdsourced study, we informally investigated each NLPmodelbased on the Tableau Public collection (sample of 18,820 workbooks) and internalcorporate collection (3,424 workbooks).After computing a numeric representation of each workbook with the four mod-els and varying hyper-parameters, our first step was to use UMAP dimensionalityreduction to create two-dimensional embeddings. A visual inspection of the scatter-plot projections showed that very similar workbooks are indeed grouped together,but higher level semantic clusters were not obvious and it was not possible to iden-tify whether any of the methods performed particularly well or poorly.For TF-IDF, we compared the representative keywords of each workbook, andfor LDA and LSI, we analyzed the most important keywords describing individualtopics. However, thegenerated topicswerenot always interpretable andmeaningfulbecause documents were often assigned to similar topics although we found themto be clearly distinct; this problem is a well-known issue in NLP [46, 245].289C.1.1 UMAPProjections ofDocumentRepresentationsT_heUMAP projections shown in Fig. C.1 and Fig. C.2 are based on a corpus of 15,482viz specifications (Tableau Public collection).Figure C.1: Tableau Public collection: UMAP applied to TF-IDF and Doc2Vec (100 and 300 di-mensions) andGloVe (100 and 300 dimensions) document vectors.290FigureC.2: Tableau Public collection: UMAPapplied to LSI (15, 30, 75, and 150 dimensions) andLDA (15, 30, 75, and 150 topics) document vectors.291T_heUMAPprojections shown in Fig. C.3 and Fig. C.4 are based on a corpus of 15,482viz specifications (Tableau Corporate collection).Figure C.3: Tableau Corporate collection: UMAP applied to TF-IDF andDoc2Vec (100 and 300dimensions) andGloVe (100 and 300 dimensions) document vectors.292Figure C.4: Tableau Corporate collection: UMAP applied to LSI (15, 30, 75, and 150 dimensions)and LDA (15, 30, 75, and 150 topics) document vectors.293C.1.2 KeywordProbingWe use an approach that we call keyword probing to analyze the dominant topic ofworkbooks to better understand if document representations capture underlyingtopics.For instance, we investigate workbooks from the Tableau Public sample collectionthat contain the term “superstore" (220 out of 18,820 workbooks). Since we knowthat those workbooks are highly related, if they are assigned to a broad range oftopics, it is an indicator that the model does not capture the content well enough.T_he results for the superstore example are shown in Fig. C.5. We try multiple NLPmodels with 15, 30, 75, and 150 topics. LDA is based on topic distributions andwe can choose the topic with the highest probability as the dominant topic. ForLSI (150 dim.), Doc2Vec (100 dim.), and GloVe (pre-trained; 100 dim.), we firstcompute the workbook vectors and then apply k-means clustering to identify themost dominant topic. LDA with 15 and 30 topics perform best because nearly allsuperstoreworkbooks are assigned to the same topic. With a larger number of topics,and across all models, those workbooks are distributed over many topics/clusters.Figure C.5: Model comparison via keyword probing. Distribution of superstoreworkbooks intodifferent topics/clusters. Each green block represents one topic and the height indicates howmany superstoreworkbooks are assigned.294C.2 Crowdsourced StudyTo assess the alignment between human judgements and our algorithmic notionof similarity, we conducted a crowdsourced two-alternative force choice (2AFC)experiment using the Mechanical Turk (MTurk) platform. In this section, we pro-vide further details about the experiment interface, demographics of our studyparticipants, and agreement scores.C.2.1 Experiment InterfaceWeimplementeda customweb-based interface andhosted it ona server providedbyUBC. Participants were recruited on theMTurk platform and redirected to our webpage. Fig. C.6 to Fig. C.12 show screenshots of the interface. After completing allsteps, participants received a code that they entered onMTurk to get compensated.Figure C.6:Obtaining informed consent295Figure C.7: Additional note that all stepsmust be completed.Figure C.8: Interface instructions for participants.296Figure C.9:Description of text document structureFigure C.10: Task description.297Figure C.11: Example selection.Figure C.12: Screening test.C.2.2 Demographics and Study ExperienceParticipants were asked to fill out a post-experiment questionnaire which askedabout their gender, age, educational background, and their experience during thestudy (see Fig. C.13 to Fig. C.16).298Figure C.13: Age distribution. Mean=38;Median=36Figure C.14:Gender distribution.Figure C.15: Educational background.299Figure C.16: Experience ratings on 5-point Likert scale. Task difficulty from 1 (=very easy) to 5(=very difficult). Number of times participants thought both alternatives were equally relatedto the reference (1=Never to 5=Very often). Number of times participants thought none of thealternatives were related to the reference (1=Never to 5=Very often).C.2.3 AgreementBetweenModel Predictions andHuman JudgementsWeuse Fleiss’ Kappaκ to quantify the chance-corrected agreement betweenmodelsand human judgements.300Model abbreviations: D2V= newly trained Doc2Vec model; GloPre = Pre-trainedGloVe model; GloTF = Pre-trained GloVe model that we further trained on viz spec-ifications. See Fig. C.17 to Fig. C.20. Numbers next to the model abbreviationsindicate the number of topics/dimensions.301Figure C.17: Agreement betweenmodel predictions and human judgements for 135 triplets.302Figure C.18: Agreement between model predictions and human judgements for 92 triplets with higher consensus (at least 80% agreementbetween all participants and agreement between themajority vote and the expert’s gold standard)303Figure C.19: Agreement between model predictions and human judgements for 43 triplets with lower consensus (less than 80% agreementbetween participants or disagreement between themajority vote and the expert’s gold standard.)304Figure C.20:Within-model agreement for all 4,211 generated triplets.305C.2.4 Confidence ofModelDecisionsFor the triplet judgement task, models produce predictions by independently com-puting the pairwise similarity scores between the reference document and one ofthe alternatives. T_hemodel compares the two scores and the alternative with thehigher score is deemed to be more similar.T_he delta between two scores is an indicator for the confidence ofmodel predictions.Are both scores very close, and in fact, both alternatives are relevant or is there asignificant difference?We normalized the pairwise similarity scores for each model and compared thedifference. T_he median difference between pairs of similarity scores is slightlyhigher for TF-IDF and all other models are in a similar range, as shown in Fig. C.21.306FigureC.21:Model comparisonbasedonnormalized similarity scoredeltas. Themediandifferencebetweenpairs of similarity scores is slightlyhigher for TF-IDF. All othermodels are in a similar range.307C.3 StudyAnalysis ToolWe implemented an interactive visual analysis tool to both clean the crowdsourceddata and to better understand the results and the implications for the model selec-tion (see Fig. C.22 to Fig. C.26).Figure C.22: Our custom analysis tool provides an overview of all MTurk participants. The re-sponses of each participant are displayed along a timeline, (a) either laid out uniformly or (b)by reaction time, and color-coded based on the agreement with the gold standard or majorityvote.308Figure C.23: A separate view in the analysis tool allows us to compare model predictions withhuman judgements for individual triplets. Each row corresponds to one triplet (out of 135). Thefirst two columns showthe reaction time followedby two columns that summarize thenumberof human votes that each alternative received. The yellow circle inside the blue boxes indicateswhichalternative theexpertannotatorhaspicked. Modelpredictionsareshownusingdiamondsymbols. For each triplet, models produce one similarity score for alternative A and one scorefor alternative B. We have pre-selected 6models which leads to 12 symbols per triplet. Similar-ity scores are between 0-1 and the values are encoded through the x-position. The alternativewith thehigher score is consideredmore similar andshown inblue; otherwise ingold. For theseexample triplets,we can see that nearly allmodels agreewith thehumanmajority that alterna-tiveA ismore similar to the reference. There are only three triplets where onemodel predictedthat alternative B ismore similar (blue diamonds in column B).Figure C.24:Hovering over rows shows a preview of the triplet theway that it was presented toparticipants. The alternative that receivedmore votes is shown on the left.309Figure C.25: Predictions that belong to a model instance can be connected with lines and allcolumns can be sorted to identify patterns. In this screenshot, triplets are sorted based onthe standard deviation of model predictions (column A). All models computed high similarityscores for alternative A and diverging but lower scores for alternative B. The number of humanvotes mostly align with the models but in some cases human consensus is not as clear (e.g.,triplet with 16 participant votes for alternative A and 10 votes for alternative B).310Figure C.26: Triplets can be filtered based on low or high human consensus. Model instancescan be shownor hidden interactively. In this example, the triplets are sorted based on the num-ber of participant votes alternative B received. Triplets with none or low human consensus areshown at the top. LDA andGloVe-TF are selected and all othermodel predictions are hidden.C.4 Proof-of-Concept: VizCommenderFig. C.27 is a screenshot of an earlier version of the VizCommender proof-of-concept interface. Fig. C.28 to Fig. C.30 are screenshots of the final VizCommenderinterface.311Figure C.27: Early VizCommender prototype that incorporated both visual and topic features.The interfaceallowedus to change theweightsof these features (see right column)andobservehow recommendations change. Later prototypes focused on text-based similarity because in-vestigations anddiscussionswith collaborators indicated that specific visual encodings are lessrelevant for information seeking in vis repositories.Figure C.28: VizCommender overview. Random sample of visualizationworkbooks.312Figure C.29: VizCommender overview. Screenshot shows results for the search query “medal".A workbook titled “OlympicGames" is selected and the quick view on the right side providesfurther details and recommendations. The similarity model identified related workbooks andseveral workbooks that use similar data sources.313Figure C.30: VizCommender overview. Five near-duplicate workbooks are shown and one is se-lected. Both similarity facets (“use similar data" and “similar versions") correctly reveal the rela-tionship between the workbooks in the quick view sidebar. Future work could investigate howthe viz-to-viz similarity measure can be used to bundle near-duplicates and to show only onerepresentative in the interface.314D Appendix: VizSnippetsT_his appendix contains supplemental material related to the VizSnippets projectdescribed in Chapter 5.315D.1 Reviewof ExistingVisualization SnippetDesignsTableD.1:Overview of existing visualization snippets that we reviewed.316TableD.2:Detailed analysis of existing visualization snippets.317D.2 Visual InspectorsWe implemented a suite of visual inspectors to facilitate the development andqualitative evaluation of the VizSnippets computational pipeline.D.2.1 Text Content of VisualizationBundles (I1)T_he visual inspector I1 shows extracted text content of visualization bundles andselected TF-IDF keywords. We use colour coding for various types of text attributes,andmany control widgets allow us to drill down to specific attributes. Similar to allother visual inspectors, we can continuously load and inspect new bundle samples.T_his visual inspector helped us to get a better understanding of what content canbe extracted and how readable or cryptic certain attributes are. In addition, weused I1 to compare the extracted keywords with the raw text data.FigureD.1: Example screenshot of the visual inspector I1 to analyze extracted text content andkeywords.D.2.2 Keyword Similarity and Spell Checking (I2)When analyzing extracted TF-IDF keywords, we found that many keywords aresemantically and syntactically very similar and including all variants would take up318precious space without adding value. Although we lemmatize keywords to collapseword variants, many cases remain undetected if they are not in theWordNet dictio-nary or are misspelled. We decided to compute the Levenshtein distance betweenkeyword pairs to filter very similar keywords but were lacking confidence whenchoosing a threshold. We therefore implemented a visual inspector I2 to analyzehundreds of keyword pairs and discern what Levenshtein distance is appropriateto filter semantically similar words. We ultimately chose 0.15 on a range from0 to 1. During this process we also observed that, in a pair of similar keywords,one is often incorrectly spelled, but both keywords may get ranked high by theTF-IDFmodel. We used the PyEnchant spell checking library and extended I2 toindicate correctly spelled words with a green background. Although computingthe similarity between all keywords within a bundle and running a spell checkercan be computationally expensive, we found through visual inspection a significantincrease in keyword quality, and added those steps to the offline keyword extractionpipeline.Figure D.2: Example screenshot of the visual inspector I2 to analyze similar keywords and de-termine an appropriate cut-off threshold.319FigureD.3: A second screenshot of the visual inspector I2 to analyze similar keywords.Figure D.4: A third screenshot of the visual inspector I2. In this example, we select keywordpairs with a larger Levenshtein distance of 0.23 to 0.25 and see that these words are often se-mantically different.320D.2.3 Pairwise Image Similarity (I3)Wemanually labeled 1204 image pairs to compare different similaritymeasures (seeSec. D.3). In addition to quantitative statistics, such as accuracy and precision/re-call, we created the inspector I3 to better understand when a model diverges fromthe human annotator. I3 shows random samples of negatively predicted imagepairs (false positives, false negatives) and allows us to switch between differentsimilarity models.FigureD.5: Example screenshot of the visual inspector I3 to analyze incorrectly classified imagepairs.D.2.4 Bundle Image Similarity (I4)For our purpose of filtering redundancies of visual content in a bundle, we only needto make comparisons between images within and not across bundles. I4 shows asimilarity matrix of all images within a bundle to help us analyze the magnitude ofsimilarity values apart from the binary classification. In this example, similarityscores are computed by comparing colour histograms. We can see that two differentchart types that use the same colour palette are considered more similar comparedto, for example, the tree maps.321FigureD.6: Example screenshot of the visual inspector I4 to analyze the similarity between im-ages within a bundle, based on colour histograms.FigureD.7: A second screenshot of the visual inspector I4 to analyze image similarity.322D.2.5 ImageColours (I5)We extract colours of images and use the ratio of the most dominant colour toidentify empty or nearly empty images that are not informative. T_hrough visualinspection with I5, we further analyzed what type of images are getting filtered. Ifthe dominant colour threshold is too low, line charts withminimal data inkmay getfalsely discarded, while a very high threshold may lead to the inclusion of imagesthat contain mostly white space and provide no informative value.By using this inspector and adjusting the threshold interactively, we also observedthat images with a high colour diversity are more informative when scaled down,compared to mostly single-colour images.Figure D.8: Screenshot of the visual inspector I5 to analyze image colours. The number aboveeach image shows the percentage of the most dominant colour compared to all other pixels.The most dominant colour is above 98% in all examples and indicates that those images arenot informative.323Figure D.9: A second screenshot of the visual inspector I5 to analyze dominant colours of im-ages. We display the extracted dominant colours (e.g., 5 k-means clusters) above each imageand below the percentage label. We found that images with a higher colour diversity, such asshown in this screenshot, aremore informative when used as a thumbnail.Figure D.10: A third screenshot of the visual inspector I5 showing examples with a lower colordiversity and are considered less informative when scaled down substantially.324D.2.6 Image Selection andRanking (I6)We created a visual inspector (I6) that reveals which images are getting filtered atwhich step, and how the original ranking changes after diversifying it based onchart types and image similarity. T_his type of qualitative assessmentwas imperativeto understand if combining various building blocks lead to the desired outcome orif important information gets lost during the image compression.Fig. D.11 is a screenshot of I6 showing the images of 6 example bundles on a greybackground. Coloured crosses with symbols indicate why images are filtered. Var-ious control widgets allow us to change the display settings but also underlyingmodel thresholds interactively. For instance, we can define the minimum distancebetween image pairs before they are considered similar and one of them gets dis-carded.Figure D.11: Example screenshot of the visual inspector I6 to analyze the image selection. Thesymbols indicatewhy imagesarefiltered ( incomplete/empty, embedded, and similarviews). We can see that some bundles contain only two imageswhile other bundles are signifi-cantlymore complex and contain diverse visual content.325Figure D.12: A second screenshot of the visual inspector I6 to analyze the image selection andranking. Here, we enabled an option to compare the initial ranking of bundle images with thenew ranking, that incorporates dashboards, chart types, and colour diversity. The top row ineach grey box shows the original ranking and the bottom row shows the new ranking and thefiltered images.326FigureD.13: A third screenshot of the visual inspector I6. We can change display settings, suchas the thumbnail size, interactively. These random samples demonstrate that some bundlescontain many different views and dashboards, and with our pipeline we are able to remove alot of redundancy and non-informative content.Figure D.14: A fourth screenshot of the visual inspector I6 shows that we are able to compresssome of the visual content even in smaller bundles.327D.2.7 Page andSnippet Layouts (I7)We created the inspector I7 to combine all visual and textual elements and arrangethem as visualization snippets based on the proposed top-down design framework.FigureD.15: Screenshotof thevisual inspector I7: snippetsarearranged inauniformgrid layout.328Figure D.16: Screenshot of the visual inspector I7: snippets are arranged in a strip layout. Theheight of the snippets is variable, depending on the snippet content.FigureD.17: Screenshot of the visual inspector I7: snippets are arranged in a list layout.329FigureD.18: Screenshot of the visual inspector I7: snippets are arranged in a table layout.330Figure D.19: Screenshot of the visual inspector I7: one snippet is shown in the large previewlayout where the goal is to show as much informative content as possible while screening outclutter and redundancy. While this snippet layout consumesmore pixel space comparedmorecompact snippets, it can be used to quickly show further details of a bundle, for example in amodal window, instead of loading the fully interactive version.D.2.8 A/B Comparisons (I8)We recreated existing snippet designs of multiple visualization platforms, TableauOnline, Tableau Public, Power BI, Qlik, and Observable, to facilitate A/B compar-isons with snippets that are generated through the proposed pipeline. T_he visualinspector I8 loads random Tableau or Observable bundles and allows us to choosebetween these snippet layouts.331FigureD.20: Screenshot of the visual inspector I8: TableauOnline snippets. All snippets presentonly a single thumbnail and bundle titles that are heavily truncated. 3 out of 15 random exam-ples show images thatprovideno informative value. The snippets showhowoftenabundlehasbeen viewed or bookmarked but information about the author or creation date is only shownas a tooltip.332FigureD.21: Screenshot of the visual inspector I8: Tableau Public snippets. When browsing bun-dles onTableauPublic, snippets are displayed in a list layoutwith onebundle per row. The snip-pets include the author name and the last modified date, and longer titles are not truncated.However, the snippets show only one image thumbnail and the small image size (78px wide)makes it difficult to recognize any visual encodings. When authors do not add a custom de-scription to a bundle, which is fairly often the case, the bundle title is repeated a second time.333Figure D.22: Screenshot of the visual inspector I8: Power BI snippets. Similar to all other plat-forms, PowerBI snippets showatmostone image. Thumbnails are relatively largealthough theexact size depends on the overall browser window size. Snippets include a substantial amountof white space but, besides the title, no other textual information about the underlying topic isshown.Figure D.23: Screenshot of the visual inspector I8: Qlik snippets. Many snippets are shown si-multaneously because of the small snippet size but the information content hinges on thebun-dle titles which are often not very descriptive.334FigureD.24:A/B comparisonbetweenexistingObservable snippetdesigns and snippets gener-ated with our pipeline. (1) Observable snippets show a single thumbnail, the bundle title, andmeta data. In our simulator, we use coloured symbols with the first letter of the username buton the Observable platform custom avatar images are frequently shown instead. (2) The VizS-nippets designs havemore information content in the same amount of pixel space, withmulti-ple images and informative keywords providing relevant information at high density.D.3 Labeling InterfacesWe evaluated the algorithms for filtering empty and similar images based onman-ually assigned ground truth labels: 2600 binary labels for complete/incompleteimages, and 6019 binary labels for similar/dissimilar image pairs. To facilitatethe process of labeling thousands of images, we created two custom interfacesthat quickly load image samples and support user input via keyboard. We foundthat keyboard input is significantly faster compared to the graphical interface. ZandM keys are used to label items as true and false respectively. T_he user-defined335label is shown as a green or red button to confirm the selection. After pressing theright-arrow key, the label is saved and a new example is shown.Labeling the images took one person roughly eight hours using these interfaces.Figure D.25: Example screenshot of the interface to label empty images. The images are notloaded fully randomly. A control widget at the top lets us specify the percentage of the mostdominant colour. Otherwise, only a very small fraction of imageswould be labeled as empty orincomplete.336FigureD.26: A second screenshot of the labeling interface for empty images.FigureD.27: Example screenshot of the interface to label similar and dissimilar image pairs.337FigureD.28: A second screenshot of the labeling interface for similar image pairs.D.4 Implementation andTimeEstimatesWe implemented all visual inspectors and labeling interfaceswith JavaScript andD3,and use a common Python back-end as an interface between the all the front-endsand the PostgreSQL database.In addition, we used Jupyter notebooks to experiment with algorithmic buildingblocks and to train the CNNmodels.Extracting all the information from bundles, populating the database, and settingup the back-end took about two weeks. Afterwards, we were able to implementmost of the tools within a few hours. I6 (image selection and ranking) and I7 (pageand snippet layouts) are most complex and their development took several days.We also refined these inspectors iteratively and, for example, added control widgetswhen needed.338D.5 Design Framework (Size and Layout Choices)TableD.3: VizSnippets design framework: size and layout choices.339E Appendix: Data-First DesignStudiesTable E.1 included in this appendix provides further details related to the review of64 published design studies, described inChapter 6. We identified 16 probable casesthat may implicitly indicate a data-first ordering (see green fill colour in Table E.1).340Table E.1: Review of 64 previous design studies.341Table E.1: Review of 64 previous design studies (cont.).342Table E.1: Review of 64 previous design studies (cont.).343Table E.1: Review of 64 previous design studies (cont.).344Table E.1: Review of 64 previous design studies (cont.).345Table E.1: Review of 64 previous design studies (cont.).346Table E.1: Review of 64 previous design studies (cont.).347Table E.1: Review of 64 previous design studies (cont.).348Table E.1: Review of 64 previous design studies (cont.).349Table E.1: Review of 64 previous design studies (cont.).350Table E.1: Review of 64 previous design studies (cont.).351Table E.1: Review of 64 previous design studies (cont.).352Table E.1: Review of 64 previous design studies (cont.).353Table E.1: Review of 64 previous design studies (cont.).354
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Munzner, Tamara
Data-driven prototyping in information visualization
Text