UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

A multi-display collaborative urban planning system with a federated architecture Su, Tao 2011

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


24-ubc_2011_fall_su_tao.pdf [ 5.35MB ]
JSON: 24-1.0052096.json
JSON-LD: 24-1.0052096-ld.json
RDF/XML (Pretty): 24-1.0052096-rdf.xml
RDF/JSON: 24-1.0052096-rdf.json
Turtle: 24-1.0052096-turtle.txt
N-Triples: 24-1.0052096-rdf-ntriples.txt
Original Record: 24-1.0052096-source.json
Full Text

Full Text

A Multi-Display Collaborative Urban Planning System with a Federated Architecture  by Tao Su B.Eng., Zhejiang University, 2009  A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF  Master of Science in THE FACULTY OF GRADUATE STUDIES (Computer Science)  THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) September 2011 c Tao Su, 2011  Abstract Neighborhood planning and design is a complex and iterative process that usually involves different stakeholders and groups with diverse interests and goals. These include developers interested in the economic opportunity of development, politicians and administrators interested in public policy values, and neighbors interested in compatibility. These stakeholders work collaboratively to negotiate a mutually acceptable proposal that fulfills many qualitative and quantitative requirements and expectations. Until now, few Planning Support Systems (PSS) were capable of supporting the complexity and interactivity of this collaborative decision making process smoothly through the whole decision making process from the initial planning phase to final completion. In our research, a multi-display, collaborative urban planning system was implemented. The system uses a multi-touch tabletop computer as a central interactive display for designing a neighborhood. Multiple additional projectors can be connected to the tabletop display to present other information, such as 3D rendering of the neighborhood or some quantitative statistics summarizing the plan. Various ways to connect multiple displays together were investigated. The final system uses a distributed, federated architecture to connect the displays through a network. Informal user feedback was gathered from different types of users. In the thesis we discuss various design issues regarding multi-display systems and tabletop urban planning systems and how they were resolved. There are a number of extensions to our new system. One is the ability to support other types of displays and input techniques, such as handheld devices or remote tabletop displays. These and other future research directions are briefly described. ii  Preface All work described in this thesis was performed under the supervision of Prof. Kellogg S. Booth with regular consultation from Prof. Ronald Kellett and Prof. Cynthia Girling from the elementslab in the The School of Architecture and Landscape Architecture (SALA) at The University of British Columbia (UBC). Ronald Kellett was the second reader for this thesis.  iii  Table of Contents Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  ii  Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  iii  Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  iv  List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  viii  List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  ix  Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  xi  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  xii  1  Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1  1.1  Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . .  2  1.2  The Research Setting . . . . . . . . . . . . . . . . . . . . . . . .  3  1.3  Tabletop Displays . . . . . . . . . . . . . . . . . . . . . . . . . .  4  1.4  Wall Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5  1.5  Multi-Display Systems and Federated Architectures . . . . . . . .  6  1.6  How Neighborhood Planning Works . . . . . . . . . . . . . . . .  6  1.7  Glossary of Terms . . . . . . . . . . . . . . . . . . . . . . . . . .  7  2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  10  2.1  Interactive Tabletop Displays . . . . . . . . . . . . . . . . . . . .  10  2.2  Wall Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . .  11  2.3  Multi-Display Systems . . . . . . . . . . . . . . . . . . . . . . .  13  iv  2.4  Planning Support Systems . . . . . . . . . . . . . . . . . . . . .  15  3 The First Iteration of the System Design . . . . . . . . . . . . . . . .  17  3.1  Design Review of the Standalone System . . . . . . . . . . . . .  18  3.1.1  Strengths of the Standalone System . . . . . . . . . . . .  20  3.1.2  Weaknesses and Possible Improvements . . . . . . . . . .  20  3.2  elementsdb . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21  3.3  Desirable Features and Iterative Design Process . . . . . . . . . .  22  3.3.1  Desirable Features . . . . . . . . . . . . . . . . . . . . .  22  3.3.2  Two Iterations of Design and Development . . . . . . . .  25  System Design Details . . . . . . . . . . . . . . . . . . . . . . .  25  3.4.1  Keeping the Standalone System . . . . . . . . . . . . . .  25  3.4.2  Augment the Tabletop Application with Wall Displays . .  26  3.4.3  Federated Architecture . . . . . . . . . . . . . . . . . . .  29  3.4.4  Design Task Selection . . . . . . . . . . . . . . . . . . .  29  3.4.5  Connection with a Building Database . . . . . . . . . . .  30  3.4.6  Running on Various Devices . . . . . . . . . . . . . . . .  31  4 The First Iteration of the System Implementation . . . . . . . . . .  33  3.4  4.1  4.2  4.3  Implementation Review of the Standalone System . . . . . . . . .  33  4.1.1  Hardware Implementation . . . . . . . . . . . . . . . . .  34  4.1.2  Software Implementation . . . . . . . . . . . . . . . . . .  34  Hardware Configuration . . . . . . . . . . . . . . . . . . . . . .  36  4.2.1  Constructing a Multi-Display System . . . . . . . . . . .  36  4.2.2  Network-Based Multi-Display System . . . . . . . . . . .  38  Software and Interaction Implementation . . . . . . . . . . . . .  40  4.3.1  Software Implementation of a Network-based Multi-Display System . . . . . . . . . . . . . . . . . . . . . . . . . . .  4.3.2  40  Three-Dimensional Pattern Viewer on Projected Wall Displays . . . . . . . . . . . . . . . . . . . . . . . . . . . .  44  4.3.3  Controlling the 3D Pattern View from the Table . . . . . .  49  4.3.4  Design Task Selection and Integration with a Database . .  52  4.3.5  Running on Different Devices . . . . . . . . . . . . . . .  54  v  5  The Second Iteration of the System Design . . . . . . . . . . . . . .  55  5.1  Review of the First Iteration Design . . . . . . . . . . . . . . . .  56  5.2  Requirements and Desirable Features . . . . . . . . . . . . . . .  58  5.2.1  Requirements Suggested by elementslab Researchers . . .  58  5.2.2  Other Requirements . . . . . . . . . . . . . . . . . . . .  61  System Design . . . . . . . . . . . . . . . . . . . . . . . . . . .  62  5.3.1  Improving versus Rebuilding . . . . . . . . . . . . . . . .  62  5.3.2  System Architecture . . . . . . . . . . . . . . . . . . . .  62  5.3.3  Communication Design . . . . . . . . . . . . . . . . . .  64  5.3.4  Abstract Data Design . . . . . . . . . . . . . . . . . . . .  65  5.3.5  UI and Interaction Design . . . . . . . . . . . . . . . . .  66  6 The Second Iteration of the System Implementation . . . . . . . . .  68  5.3  6.1  Integrating Google Maps . . . . . . . . . . . . . . . . . . . . . .  68  6.1.1  Embedding Google Maps in a Program . . . . . . . . . .  69  6.1.2  Linear Zooming in Google Maps . . . . . . . . . . . . . .  69  6.1.3  Smooth Zooming and Panning on Google Maps . . . . . .  71  Connection with the Building Database . . . . . . . . . . . . . .  73  6.2.1  The elementsdb API . . . . . . . . . . . . . . . . . . . .  73  6.2.2  Loading the Database . . . . . . . . . . . . . . . . . . . .  73  6.2.3  Additional Resources of the Cases . . . . . . . . . . . . .  74  6.3  Universal Touch Framework . . . . . . . . . . . . . . . . . . . .  75  6.4  Network and Communication . . . . . . . . . . . . . . . . . . . .  76  6.5  Other Improvements . . . . . . . . . . . . . . . . . . . . . . . .  78  6.5.1  UI Improvements . . . . . . . . . . . . . . . . . . . . . .  78  6.5.2  Inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . .  79  6.5.3  Statistics Viewer . . . . . . . . . . . . . . . . . . . . . .  81  7 Discussions and Future Work . . . . . . . . . . . . . . . . . . . . . .  83  6.2  7.1  Collaborative Multi-Display System . . . . . . . . . . . . . . . .  83  7.1.1  Appropriate Location of Information . . . . . . . . . . . .  83  7.1.2  Information Sharing and Coordination among Displays . .  84  7.1.3  Physical Locations of the Displays and Display Identities .  85  vi  7.1.4  Shared Control versus Exclusive Control . . . . . . . . .  86  7.1.5  Other Kinds of Displays . . . . . . . . . . . . . . . . . .  87  7.1.6  User Identification . . . . . . . . . . . . . . . . . . . . .  87  User Interface of the Tabletop Pattern Creator . . . . . . . . . . .  88  7.2.1  Automatic Object Alignment . . . . . . . . . . . . . . . .  89  7.2.2  Case Size . . . . . . . . . . . . . . . . . . . . . . . . . .  90  7.2.3  Control Conflicts . . . . . . . . . . . . . . . . . . . . . .  90  7.2.4  Widgets versus Gestures . . . . . . . . . . . . . . . . . .  90  7.2.5  More Ways to Interact with the Table . . . . . . . . . . .  91  7.2.6  Angle of Bird’s Eye View . . . . . . . . . . . . . . . . .  91  7.2.7  Semantics of Different Types of Views . . . . . . . . . .  92  Connection of Components . . . . . . . . . . . . . . . . . . . . .  93  7.3.1  Connecting to Remote Components . . . . . . . . . . . .  93  7.3.2  Quality of the Architecture . . . . . . . . . . . . . . . . .  94  Usefulness and Effectiveness Evaluation . . . . . . . . . . . . . .  94  8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  96  7.2  7.3  7.4  8.1  Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  96  8.2  Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  97  Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  99  A Algorithms and Code Segments . . . . . . . . . . . . . . . . . . . . . 104 A.1 Discovering and Registering with Central Controller . . . . . . . 104 A.2 An Example of a KML File for Adding a Model . . . . . . . . . . 105 A.3 KML File Requesting Google Earth to Reload the Address Every 2 Seconds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 A.4 An Example of a Design Task Configuration File (Partial) . . . . . 106  vii  List of Tables Table 4.1  CPU utilization of the programs on the tabletop display . . . .  38  Table 4.2  How data of a design task are stored in the standalone system .  53  Table 6.1  Typical interfaces of a map control . . . . . . . . . . . . . . .  69  viii  List of Figures Figure 1.1  Example of cases with a 2D “footprint” representation . . . .  8  Figure 1.2  Example of a pattern . . . . . . . . . . . . . . . . . . . . . .  9  Figure 3.1  Overall system requirements . . . . . . . . . . . . . . . . . .  18  Figure 3.2  Screen shot of the the standalone system . . . . . . . . . . . .  19  Figure 3.3  Cases in elementsdb . . . . . . . . . . . . . . . . . . . . . .  22  Figure 3.4  An example case in elementsdb . . . . . . . . . . . . . . . .  23  Figure 4.1  System configuration – USB hub plus USB to VGA adapters .  38  Figure 4.2  System configuration – via an Ethernet switch . . . . . . . . .  39  Figure 4.3  System configuration – via a WiFi router . . . . . . . . . . .  40  Figure 4.4  Photo of the final system configuration . . . . . . . . . . . . .  41  Figure 4.5  Communication between components in the system . . . . . .  43  Figure 4.6  Screen shot of central controller . . . . . . . . . . . . . . . .  44  Figure 4.7  Screen shot of tabletop pattern designer and 3D pattern viewer  48  Figure 4.8  Real-time update process of 3D pattern viewer . . . . . . . .  49  Figure 4.9  Screen shot of two active widgets sharing one projector . . . .  51  Figure 4.10 Bird’s eye view of a pattern . . . . . . . . . . . . . . . . . . .  51  Figure 4.11 Street view of a pattern . . . . . . . . . . . . . . . . . . . . .  52  Figure 4.12 Top-down view of a pattern . . . . . . . . . . . . . . . . . . .  53  Figure 5.1  Different types of maps . . . . . . . . . . . . . . . . . . . . .  60  Figure 5.2  System architecture . . . . . . . . . . . . . . . . . . . . . . .  63  Figure 5.3  Communication design diagram . . . . . . . . . . . . . . . .  65  Figure 5.4  Abstract data design diagram . . . . . . . . . . . . . . . . . .  66  ix  Figure 6.1  Table pattern designer screen shot . . . . . . . . . . . . . . .  70  Figure 6.2  Zooming issue in maps . . . . . . . . . . . . . . . . . . . . .  72  Figure 6.3  Example of elementsdb API . . . . . . . . . . . . . . . . . .  73  Figure 6.4  Touch framework structure . . . . . . . . . . . . . . . . . . .  76  Figure 6.5  Network diagnostics tool . . . . . . . . . . . . . . . . . . . .  78  Figure 6.6  Comparison of the case bar . . . . . . . . . . . . . . . . . . .  80  Figure 6.7  Example of Google Image Charts . . . . . . . . . . . . . . .  82  Figure 7.1  Semantics of views . . . . . . . . . . . . . . . . . . . . . . .  93  x  Glossary Below is a list of acronyms used in this thesis. For detailed descriptions of these and other terms used in the thesis, please refer to Section 1.7. CIRS  Centre for Interactive Research on Sustainability  FTIR  Frustrated Total Internal Reflection  GIS  Geographic Information System  KML  Keyhole Markup Language  NUI  Natural User Interface  PSS  Planning Support System  SALA  The School of Architecture and Landscape Architecture  TUIO  Tangible User Interface Objects  UBC  The University of British Columbia  xi  Acknowledgments Financial and infrastructure support for this research was provided by the Natural Sciences and Engineering Research Council of Canada, the Networks of Centres of Excellence Program through the Graphics, Animation, and New Media NCE (GRAND), the Canada Foundation for Innovation, and The University of British Columbia. Firstly, I would like to thank my supervisor, Prof. Kellogg S. Booth. It is an honor for me to work with him. The thesis would not have been possible without his help and guidance. I would like to thank Prof. Ronald Kellett and Prof. Cynthia Girling for their collaboration and valuable feedback on our work. Prof. Ronald Kellett also provided valuable suggestions for the writing of this thesis. I would like to thank my friends for their help during my research. Particularly, I would like to thank Cameron Chan for helping me proofread and revise the thesis and Kai Ding for helping me move the equipments many times. Lastly, I would like to thank my parents Lanfeng Su and Ru Cheng for their support during my research. Their emotional and moral support really helped me go through some hardest times during my research. This thesis would not have existed without them.  xii  Chapter 1  Introduction Neighborhood designing and planning is an important yet complicated process. It happens when villages and towns evolve into cities and metropolises and when old neighborhoods turn into new city centers. This process usually involves different stakeholders and groups with diverse interests and goals. People involved in this process might have very diverse educational backgrounds. For example, initial neighborhood plans are designed by professional city planners, and then public residents usually participate in deciding which plan is accepted by most people in a consensus-based study. The government and investors are also involved in this process. The process usually requires coming up with, presenting, and comparing different alternatives of a plan in various ways. Until now, few systems were capable of supporting this smoothly through the whole decision making process from the initial planning phase until completion (see Chapter 2 for details). The research reported in this thesis is about iteratively designing and implementing an interactive sustainable neighborhood planning tool utilizing multi-touch tabletop displays and projected wall displays. The system that was developed is capable of allowing urban designers to design, compare and present neighborhoods in various ways in an open and public setting. All of these actions are accomplished through a Natural User Interface (NUI) on tabletop displays. Through this process, a federated architecture to connect multiple displays together was designed and implemented. Traditionally, neighborhood and regional planners use paper methods to come 1  up with initial plans. They use maps and paper-based building models to mock up an initial plan of a neighborhood. Our current research began by making improvements to an existing application that was developed by Fernquist [21] as part of an on-going joint research project between researchers in architecture, computer science, civil engineering, landscape architecture, and planning. Support was added to the existing system for displaying extra information to augment the planning process using additional wall displays. After reviewing the first extension and collaborating with practitioners from The School of Architecture and Landscape Architecture (SALA) at The University of British Columbia (UBC), the application went through a second round of design and implementation in which more functionality was added based on user feedback. In this thesis, we describe the design and implementation process as well as how critical design decisions were made for the augmented system. Discussion of the feedback gathered from users and possible future formal user studies is also included. The system for neighborhood planning is part of a larger initiative. The longterm goal of our research is to study how to present information on multiple displays, how to control and coordinate the displays, how information can travel between displays, and how such multi-display, multi-user systems can be deployed to support decision making and planning. In the remainder of this chapter we provide an outline of the rest of the thesis in Section 1.1, details of the larger research goals in Section 1.2, and then a brief overview of interactive tabletop displays in Section 1.3 and wall displays in Section 1.4. The federated architecture that connects the displays is introduced in Section 1.5 followed by a summary of how neighborhood planning works in Section 1.6. A glossary of terms used in the thesis is provided in Section 1.7.  1.1  Thesis Organization  The rest of the thesis is organized as follows: Chapter 2 discusses prior work related to the research. Chapter 3 introduces the system design and some important considerations when designing the system. Chapter 4 discusses the implementation of the first iteration of the new system and how different technical considerations were implemented. An assessment of the first iteration and details of the design  2  phase of the second iteration are presented in Chapter 5, with the implementation phase of the second iteration described in Chapter 6. Chapter 7 provides a discussion of the results of the research and possible future work arising from the research. Chapter 8 summarizes the thesis and highlights the main contributions made in the research.  1.2  The Research Setting  The research reported in this thesis is part of a larger investigation of how the characteristics of different types of displays can best be utilized to support specific types of group work. Factors that are of interest include display size (from small handheld personal displays to large wall-sized displays), display orientation (horizontal tabletop displays vs. vertical wall displays), input techniques (singleor multiple-touch direct interaction vs. traditional but less direct mouse-based interaction), and degree of explicit support for collaboration. The particular work reported here looks at a specific type of group work, neighborhood planning. It was conducted by a multi-disciplinary team of architects and computer scientists. In 2010, Fernquist [21] developed and evaluated a collaborative tool for neighborhood planning using tabletop displays. This was the first prototype for the system that was further refined in the current research. Our research extends and enhances that previous implementation. The first step was to augment the tabletop with multiple projected wall displays. A number of alternatives for how to present additional information on the wall screens were proposed. We initially decided to use the wall displays for real-time 3D views of the neighborhood plan that was being designed on the tabletop display. Various design choices were evaluated and compared through this process (see Chapter 3). When implementing these extra views on the large screens, efforts were made to decouple the system into different parts and to connect them through a network. This evolved into a federated display connection architecture. Based on this infrastructure, the system can also connect with other remote devices to augment its functionality, such as iPads, remote displays, or even another touch-table. The ability to connect to additional displays is a crucial piece of functionality for our system to support collaborative decision making. It is the most important improvement in the first interation of our new  3  implementation. Informal user interviews were conducted and we received feedback from our primary users – neighborhood planners at The School of Architecture and Landscape Architecture (SALA) at UBC. Then a second interation of the design and implementation of the system was carried out. In the second iteration, the application was further extended to connect with elementsdb, an urban planning and design database developed by SALA at UBC. Dynamic data can be loaded from the database to the touch table at design time. The purpose and benefits of connecting elementsdb with the touch-table application are discussed in Chapter 3. Other enhancements were made based on user requirements, such as an enhanced user interface and the use of a real map application (such as Google Maps) rather than ad hoc images for the background of the design workspace. The communication infrastructure between the tabletop and the wall displays was further enhanced to better connect to other devices near the table.  1.3  Tabletop Displays  The principal physical workspace of the planning support system is a tabletop display. Tabletop displays are a kind of medium-to-large, interactive display. These displays are usually suitable for small group collaboration or for public presentation [38]. The tabletop display used in our research is a SMART Table [15]. It is capable of sensing 120 touches simultaneously and uses a technology called Frustrated Total Internal Reflection (FTIR) to sense touches [15]. A detailed description of the technology for the table that was used is given in Chapter 4. Tabletop displays are usually bigger than 20 inches and smaller than 50 or 60 inches diagonally. Such size is bigger than ordinary displays and smaller than projected displays. The horizontal orientation of tabletop displays, in addition to their size, makes them suitable for presenting information for a small group of 2 to 4 people at the same time. Also, the interactive nature opens up many possibilities for tabletop display applications that may involve small group collaboration. A more thorough literature review on tabletop displays is provided in Chapter 2. Tabletop displays have a lot of factors that limit their application. A major one is that although tabletop displays have large screen size, with 2 to 4 users work-  4  ing at the same time and all using their fingers as pointing devices, the screen size is still sometimes limited and causes control conflicts [36]. Also, because most tabletop displays use projectors to display images, they inherit the problem of limited resolution from projectors, thus making it hard to display more information in higher resolutions [27]. Because they target a small group, tabletop displays alone are often unable to provide personalized information (like on a handheld device) or public information (like on a projected wall screen). The long-term vision for our research is to augment tabletop display applications and extend them to also use more private, smaller displays as well as more public, larger displays so a range of information can be provided appropriately to the tasks in which collaborating users are engaged. We also wish to study how information should be presented on different displays and how to control and interact with multiple displays. In this research particularly, we focus on augmenting tabletop displays with projected wall displays. An introduction to wall displays is presented in the next section.  1.4  Wall Displays  Large screens are generally used for public meeting or presentation. Their vertical orientation and considerably larger display size provide a good way to present information to a large crowd of people. However, a significant issue is the scale of the screen relative to human scale, which makes it hard for precise manipulation [39]. Another problem is that people usually view projected screens from a much farther distance than other displays, which makes them hard to interact with. Some efforts to make large wall screens more interactive are mentioned in Chapter 2. In our research, a tabletop display was augmented with large screen displays to provide additional information or present information to a larger audience. The content presented on the screen can be controlled on the tabletop display. The system is implemented in a manner that lets it connect to multiple large screens and to possibly other devices as well, including those at remote sites. By combining tabletop displays with wall displays and allowing multiple people to interact with the application, we need to deal with a number of questions about how displays are shared between many users and how applications can best utilize many displays. Prior design considerations on these two types of interac-  5  tions are discussed in Chapter 2.  1.5  Multi-Display Systems and Federated Architectures  While we focus on large screens plus tabletop displays in this thesis, our system is designed to support extensions to many other types of devices. Those devices and displays form a multi-display system. The benefit of using multiple displays is one of the long-term goals of the larger research program of which the research reported here is a part. Using multiple displays can distribute information across them, provide more ways of interacting with displays and facilitate information sharing. Extensive study was carried out regarding how to implement a multi-display system. We started from a semi-federated architecture to connect displays in the first iteration of development, and then moved to a totally federated architecture to support information exchange between displays. Federated architecture is a pattern that supports information exchange among a group of “semi-autonomous decentrally organized” nodes [4]. We argue that a peer-to-peer, decentralized structure is suitable for implementing a multi-display system. Several advantages of using such a structure are presented in later chapters.  1.6  How Neighborhood Planning Works  The goal of our research is to utilize different types of displays to facilitate collaboration in neighborhood planning. In urban planning, a usual task is to redesign an area for future development or for redevelopment due to some changes in the neighborhood (such as a new transit line). These tasks usually involve designing the spatial placement of prototype buildings with different functionality, density and size on an existing road grid. Urban planners use different tools to come up with such plans for a neighborhood. The general way for urban planners to do this is to use an aerial photograph or map as a background, and draw or lay out building markers manually. This works fine if it’s only necessary to review the spatial arrangement of the buildings. But another important task of urban design is to review quantitative performance, goals and targets such as those related to land use density and sustainability of a new neighborhood. That is, some qualities have 6  to be satisfied, such as population/job ratio and building density. Other qualities like visual impact of higher buildings and shadows also need to be considered. The traditional way cannot do all these efficiently. Therefore in the previous phase of this research, Fernquist [21] implemented a computer-aided tabletop neighborhood design tool. The application emulates the traditional planning procedure by allowing users to drag building markers onto a map using an interactive tabletop display. Some qualities described above are also automatically calculated and presented to the users as they make modifications to the plan. User feedback shows this is an effective and promising way of conducting computer-supported neighborhood planning. Our research in this thesis is intended to improve on Fernquist’s earlier system and explore additional possibilities for how neighborhood planning can be done in a more efficient and smart way.  1.7  Glossary of Terms  In addition to those acronyms listed in the Glossary, there are some other terms that are used throughout the thesis. To minimize confusion, a set of frequently used terms are listed below. These terms are specific to this research and to some related projects such as the elementsdb database. These terms may not have the same meaning outside the scope of this thesis. The definitions reflect the meaning of the terms as they evolved through the course of the research reported in this thesis. • Cases can be buildings, open spaces or even roads and intersections. They are the fundamental building blocks of a neighborhood plan. Cases are usually based on real objects and several can be placed together to form a pattern. A case should contain a visual representation, such as a model, an image, or both, as well as facts about the case such as height, size, floor area and so on. The same case can be used to assemble different neighborhood plans. Figure 1.1 shows two different cases, presented using a top-down view. This mimics a current practice in neighborhood planning that uses physical cards, in this case one for a single family house (on the left) and another for a high-rise apartment building (on the right). Each card indicates the “footprint” (physical size) of the case and is annotated with additional in7  formation. Urban planners often use a number of these cards and place them on a map to design an initial neighborhood plan. Our tabletop system uses a similar visual representation for cases, providing continuity of practice with more traditional card-based approaches.  Figure 1.1: Example of cases with 2D “footprint” representation. Left: a 2story single family house that can provide 6 dwellings and 2 jobs. Right: a 10-story mid-rise apartment building that can provide 130 dwellings. Both cases are presented using a 2D “footprint” style, which shows the shape of the case viewed from the top and illustrates their footprints. • Case Sets are a collection of cases. The purpose of case sets is to provide a way of grouping cases. Each case set might include one or more cases. A case set might be created for different reasons. For example, a case set might be created and used for a particular design task (see below) or situation. Only cases in the case set would be used in that design task. • Patterns are multisets of cases (a multiset is a set in which members are allowed to appear more than once) along with their orientations and positions relative to the containing pattern’s orientation and position. Simply speaking, A pattern is a two-dimentional arrangement of some cases (which can have duplicates at various locations in the pattern). A pattern may also contain other sub-patterns. Figure 1.2 shows a pattern that illustrates a neighborhood design for a small area. The pattern is a demonstrative design of a 8  street crossing. Note that a pattern does not necessarily need to be associated with a real geographical location.  Figure 1.2: Example of a pattern. A pattern of a street crossing, showing a composition of family houses and mid-rise buildings [21]. • Design Tasks or Studies define areas where urban planners want to examine planning options. Design tasks can include one or more patterns on the target site, as well as quantitative targets such as population, jobs and density targets. For example, a design task might be to study redevelopment plans of a transit corridor after a new subway line is constructed. This design task may include a few patterns, each placed near one station of the line, and each pattern may contain a few cases and possibly other smaller sub-patterns. • Massing Models / Views are 3D models or images that show a low fidelity visualization of the approximate size and scale of a land use or building. Massing models are useful to quickly visualize a conceptual design of a building or a neighborhood. With these basic terms defined, subsequent chapters will explore in more depth various aspects of the research that was conducted, beginning in the next chapter with a survey of related work.  9  Chapter 2  Related Work This chapter provides an overview of prior work related to this research. Section 2.1 provides a summary of related research on tabletop displays, such as their technologies and application design considerations. Section 2.2 discusses related work on wall displays and large screens, including how people interact with such screens and what information can be presented on them. Section 2.3 discusses prior research on how shared displays work and how information flows between multiple shared displays. An introduction of related work on neighborhood planning support systems is covered in Section 2.4. This section includes recent innovations in neighborhood planning techniques.  2.1  Interactive Tabletop Displays  A tabletop display is the main platform of our research. Since the late 2000s, intensive research has been done on multi-touch, interactive tabletop displays. Among the studies, those focusing on small group collaboration and tabletop application design considerations are of special interest to us. In 2003, Scott et al. [38] discussed design guidelines for interactive, collaborative and co-located groupware on tabletop displays. Eight guidelines are proposed in their paper, including supporting interpersonal interaction, supporting fluid transitions between activities, supporting transitions between personal and group work, supporting transitions between tabletop work and external work, supporting the use  10  of physical objects, providing shared access to physical and digital objects, considering user arrangements, and supporting simultaneous user actions. These are some basic principles that any tabletop system should follow. In the same year, Kruger et al. [34] discussed specific orientation issues on tabletop displays. They reported that while most people know that digital objects on the table should be designed to be viewable from any direction, there are some roles that orientation influences the most, namely comprehension, coordination and communication. They found that an important role of orientation is that it can help inform collaborators about who is interacting with which objects. They also proposed some additional guidelines for orientation and rotation of objects on the table. For example, they argued that free rotation of objects should be supported on all tabletop applications, that user-positioned items must keep their orientation unless repositioned, and that if automatic rotation of objects on the table is supported, then it must be handled carefully. For control conflicts on tabletop displays, Morris et al. [37] conducted a series of user studies and suggested that most users actually prefer replicated controls over shared controls. The users in their study preferred having a separate set of replicated controls on each border of the screen for each user in a collaborative system. There have also been efforts on augmenting a single tabletop display with other devices. For example, Hartmann et al. [25] proposed augmenting a tabletop display with keyboards and mice. Keyboards or mice can be placed on a tabletop display, and they can be recognized and automatically connected by the software. Hartmann et al. also discussed the problem of user location and identity. They proposed to use the keyboard to enter user identity manually when a keyboard is connected.  2.2  Wall Displays  In our research, the multi-display system is primarily composed of tabletop displays and wall displays. Wall displays are large, usually vertically oriented displays. Prior research focuses on cognitive aspects of large screens and how to interact with them.  11  Tan et al. [40] discussed task-specific cognitive aspects of large, projected wall displays. Their results indicate that large screens can increase performance on spatial tasks such as 3D navigation. This implies that our choice of using wall displays to display 3D pattern rendering is a good start. Their study also found that users mostly compare wall-size large screens with movie screens or classroom boards rather than with desktop displays. An experiment about 3D navigation on large screens was conducted by Tyndiuk et al. [41]. Their experiments showed that users with lower visual selective attention tend to benefit more from large screens, and that the performance gain using large screens lies in the tasks being done using 3D interactions. Their study does not reveal strong evidence on which specific kinds of tasks benefit more from large screens. Because most large screen displays do not have direct sensors that make them interactive, and their scale usually makes it hard for users to reach every corner of them, large screens are sometimes hard to interact with. Some research has been done to address this issue. Krahnstoever et al. [33] proposed a system in 2002 to use cameras to capture hand gestures and allow users to interact with large screens. This is one of the early attempts of gesture-based large screen interaction. Khan et al. [31] studied another issue about reaching over to remote objects on wall displays. As mentioned earlier, because the wall displays are too big, users cannot always reach every corner they wish to interact with. So Khan et al. [31] proposed an interface called “frisbee” which contains a “telescope” of another part of the display and also contains controls for that part. Their system is based on a large screen that is capable of direct touch interaction on its surface. Such an on-screen “telescope” provides an easy way to interact with objects that is out of the reach of users. Recently researchers have paid more attention to the use of new sensors. Shoemaker et al. [39] used the concept of virtual shadows to allow people to reach remote areas of a large display. The system creates a virtual shadow on the large screen as if there is a light source behind the user. There is a magnification effect of shadows. The magnified shadow can reach far corners of a screen. Motion capturing is done using Wii remotes, which contain sensors for position and acceleration. Hutama et al. [28] proposed using smart phones to interact with large 12  screens. They proposed a way to use the built-in tilt sensors of smart phones for precise interaction with the large screens. These two techniques are examples of how novel input devices can be integrated with large wall displays.  2.3  Multi-Display Systems  While some researchers continue to work on the multi-touch tables and large displays to provide better interaction and applications on them, there are recent efforts to utilize multiple displays in collaborative systems. For example, some research has been done on moving information between displays and how to control the displays. This section discusses some of that prior work on multi-display systems. What especially interests us is using multiple displays to visualize both 2D and 3D contents that represents different aspects of the same scene. Wigdor et al. [43] discussed a multi-surface system for decision making. The system is centered around a tabletop computer, as is our system. Their system uses a tabletop computer as a central control panel. Some external displays are connected to the system. The tabletop computer shows all of the contents in the system, but in a miniaturized version. Each piece of content on the tabletop display is allocated a color and each additional display also has a color: a red-labeled display would show details of a red item on the table. Wigdor et al. also discussed ways to control which item to show on which display and how to move items between displays. Their work is very valuable to us. However, their system only supports presenting contents on external displays and managing them on a tabletop display. In our system, the problem is more complicated because our system requires interactions other than just controlling external content by actions on the tabletop. Ajaj et al. [18] proposed a system that combines a horizontal 2D view on a tabletop display with a 3D view up on the wall. The task they examined was navigation through the 3D scene on the wall. Ajaj et al. used three kinds of interaction: by direct manipulation, by single mouse interaction, or by a 6-degree-of-freedom joystick. Their system apparatus is very similar to ours, but it uses touch gestures and on-screen slide bars to interact with the tabletop display in order to control the vertical display. They asked users to perform a 3D docking task, which requires 3D  13  object selection, translation and scaling. Their results show that mouse interaction is the fastest way to complete their task, while joysticks are best for experienced users. Tabletop direct interaction seems to be inferior to the other two, but whether this should be attributed to the task itself or to the interaction technique used on the tabletop still needs further study. In a different study, Ajaj et al. [19] looked at how to control 3D views on a large screen from a tabletop display. The system is similar to their system in the previous paper, but only tabletop displays were used to control the large screen. They studied two different ways of navigating. One is moving a camera object on the 2D view on the tabletop display and having the projector show 3D view seen from the position of the camera. The other method is to have a fixed camera in the center of the scene and move and rotate the whole screen on the table to adjust the angle of 3D view on the projector. User studies show both methods have their advantages. Using a movable camera can result in significantly better performance in tasks where “a mental representation of the whole virtual scene is needed.” Using a movable camera can result in less conflict between users. Using a movable 2D scene and a fixed camera, on the other hand, reduces mental rotation to match the 2D and 3D images. Most people prefer movable scene to movable camera. However, the authors did observe that users more easily get disoriented when using a movable scene with fixed camera. Wigdor et al. [44] proposed another system called WeSpace with projected wall displays, laptop computers and tabletop displays. The system is intended for use in small group meetings. A set of frameworks was designed to allow users to easily move windows onto the large screen and manage layouts of virtual desktops on the projected displays, as well as adding mark-up to the applications. The central, shared tabletop display acts as a shared workspace and computer which can be used among all the members of the group. Lee et al. [35] constructed a multi-display system through a network. They connected many different types of displays such as tabletop displays, wall displays and handheld devices connected through a network. Interaction between them is controlled by a central server. The biggest contribution of the paper is that it proposes to use simple “select and point” gestures to connect tabletop displays with additional displays. Touches and pointing gestures are recognized and used to connect 14  devices together, which proves to be simple and effective. However, in their system connections between components are managed by a central “Presence Server”, which requires a central program to administer all of the displays. In our research, our final design is a federated, message-based communication model based on a publisher-consumer model, but Lee et al.’s way of using select and point gestures to pair displays is useful to us.  2.4  Planning Support Systems  Computer supported urban planning systems or Planning Support Systems (PSS) have started to gain attention since the 1990s. An early article by Klosterman [32] illustrates design issues in PSS. Klosterman recommended what a typical should look like and what functions it should have, and emphasized that Geo-  PSS  graphic Information Systems (GIS) should be the most important components in a PSS .  Functions of a PSS besides designing are also discussed, such as communica-  tion function, decision making and politics. Klosterman’s article provides an early vision of PSS. What followed was a series of PSS efforts that try to use tabletop displays for designing and evaluating neighborhoods. Only PSS implementations that are designed to be run on collaborative and interactive tabletop displays are described below. For other general PSS implementations, Fernquist [21] provides a good summary of the systems reported in the literature. In 2002, Ishii et al. [29] from MIT Media Lab described a specific implementation of PSS on tabletop displays. Their system is called “Luminous Table.” It is an augmented reality system that mixes digital tabletop contents with physical objects. The Luminous Table attempts to integrate 2D drawings, 3D models and simulations together. Physical building models can be placed on the table and they will be recognized by the system and then simulations such as shadows and wind flows can be rendered on the table. Users can also put a 2D drawing under the model to provide a background image of the study area. The system provides a very interesting way of implementing PSS that uses mixed-reality. However, the system only uses a single tabletop display. Later, Ben-Joseph et al. [20], also from MIT Media Lab, extended the system to support connecting with additional vertical wall displays. Our system is similar  15  to their system in this way, but our goal is to support more devices besides wall displays. Another difference is that Ben-Joseph et al.’s work uses physical objects to build a neighborhood which, while increasing tangibility, requires a lot of prior work when designing a neighborhood. Our research aims to provide a totally digital solution, with the possibility of supporting tangible objects in the future. Wagner et al. [42] proposed another mixed-reality system that uses both tabletop displays and wall displays, along with physical objects on the table, for urban planning. The difference in their system is that the physical objects are only used as tokens or controls on the tabletop display, instead of being actual buildings as in the system by Ishii et al. [29]. The application itself is more like a presentation tool that only presents previously designed neighborhood plans, without much emphasis on the interactive nature of neighborhood planning. Recent ongoing research by Halatsch et al. [23] produced another system that combines both tabletop displays and projected wall displays, but again the system focuses more on plan visualization, simulation and group discussion, this time with support for video conferencing. Overall, our understanding of prior PSS research based on tabletop computers is that few use real GIS such as recent online, interactive earth and map systems like Google Earth or Microsoft’s Bing Maps. Also, designing of neighborhoods in these systems either is completed on conventional computer and uses tabletop and other displays only as presentation tools, or it requires pre-fabricating necessary objects for use in a mixed-reality system. Few systems employ tabletop displays as well as other auxiliary displays to design, present and evaluate a neighborhood without using complex desktop rendering tools. In the next chapter we describe the system that we implemented, which is based on an earlier system by Fernquist [21] that we used as our starting point.  16  Chapter 3  The First Iteration of the System Design The first step towards making a multi-display planning support system is to understand what are the requirements of city planners and come up with a high-level design of the system. Also it is necessary to know what are the problems that restrain the existing single-display planning system developed by Fernquist [21]. Figure 3.1 shows a diagram that illustrates the requirements and process of urban planners doing neighborhood design and how well the current system can support the process. In this chapter, issues related to requirements gathering and system design are discussed in detail. Issues discussed include: • What are the overall requirements and desirable features of the system? • What features and requirements are selected to be implemented in our research? • Why are these features included in our system? • How could each feature be implemented to meet the overall requirements? We first review the design of the standalone system in Section 3.1. Then elementsdb is introduced in Section 3.2. Desirable features and goals of the system are described in Section 3.3, follow by a detailed system design in Section 3.4. 17  A city planner’s way of designing a neighborhood:  In the existing single-display, multi-user PSS…  1. Designers and stakeholders form a group for collaborative design.  It lacks more screen space and effective ways of presenting various information to other audience.  It lacks rich data sources such as a building database.  2. Design an initial plan of a neighborhood .  large displays  3. Present to other participants. Discuss and evaluate the plan’s quality and review qualitative and quantitative goals. Evaluation can be done through charts and visualization of the plan. 4. Iteratively refine and improve the design. 5. Consulling with others outside the group and further improve the design.  Inside: existing Fernquist’s system – a single-display, multi-user system  elementsdb (building db)  tablets  Outside: desired new system components that tries to meet the requirements of the city planners.  smart phones  other tables  Figure 3.1: A schematic view of the overall system requirements.  3.1  Design Review of the Standalone System  Fernquist [21] implemented a first version of the system in 2010. The application is a computer-aided neighborhood planning support system that runs on a single tabletop multi-touch interactive surface computer. This application is referred to as the “standalone” system in contrast to our multi-display system. The standalone system is designed to simulate urban planners’ paper planning process as mentioned in Section 1.6. It is also designed to be used by multiple people collaboratively at the same time. 18  Figure 3.2: Screen shot of the standalone system (from Fernquist [21]). A: Side bar displaying a list of available cases that can be dragged on to the map. B: Bottom bar showing a list of saved patterns. C: Statistics comparison of saved and current patterns. Center: The map view of the pattern, showing a completed pattern. As shown in Figure 3.2, the center part of the UI is occupied by a map image of the focus area on which a design is carried out. The map can be zoomed and panned by touch gestures (pinch for zooming and drag for panning). On the left of the screen (section A) sits the sidebar listing all available cases. The cases can be dragged onto the map. The sidebar works like an inventory of all available cases that can be used in the study. By dragging multiple cases onto the map and placing them on available or planned slots, urban planners can illustrate an initial neighborhood plan of the area. The cases in the sidebar are sorted by land use types (residential, mixed or commercial), so users can locate a certain type of case quickly. On the bottom of the interface are four icon buttons for saved patterns. An 19  advantage of Planning Support Systems (PSS) is that alternative plans can be saved and compared more easily than using conventional methods [26]. In the standalone system, there are four slots on the bottom of the screen (section B). The first slot shows an overview of the current pattern, and other saved patterns are shown in the other slots. By touching the “Save Pattern” button, the system will save the current workspace to an unoccupied slot. If all the slots are occupied, no more patterns can be saved. Saved patterns can be restored by dragging the pattern thumbnail onto the map. The right side of the interface is occupied by statistics and bar charts (section C). Such statistics might include population and jobs provided by the current pattern, building types ratio, land use ratio, etc. These numbers can be compared with targets of the design task. These statistics are useful quantitative analyses for evaluating the sustainability of a design. They are also updated as the content of the map changes. Statistics of the current pattern and all the alternatives are placed side-by-side for comparison.  3.1.1  Strengths of the Standalone System  The standalone system is advantageous in many ways compared to traditional methods. The participants in a user study of the standalone system conducted by Fernquist generally felt that the system is fun to use. Some also thought that the system “is a good learning tool” through which it can be “quick to see strengths/weaknesses of different strategies and identify gaps/problems before they arise” [21]. The possibility that the output of the system can be saved and later reviewed or further integrated with other applications was also thought to be a strength of the system.  3.1.2  Weaknesses and Possible Improvements  The standalone system is a very good prototype implementation to study multitouch interaction of planning support systems. It is therefore a good foundation for our follow-up studies. But there are a number of possible improvements that were identified by users and noted by Fernquist [21] as future changes or additions to  20  the system. Firstly, informal user feedback shows that the user interface takes too much real estate of the screen. The tabletop display used has a 22-inch screen imaged by a projector. Regular projectors can usually achieve up to 1280 by 1024 pixels, which is not a very high resolution. User feedback indicates that the map should be the centerpiece of the system and it should occupy as much screen real estate as possible. As discussed by Kruger et al. [34], for tabletop displays for small group collaboration, orientation of visual objects is important for participants. In the standalone system, extensive use of text is made, either in menus or to present information about a neighborhood plan or its properties. These require an up-right orientation on the display to be easily read as text. Reducing the use of text or moving text to other types of screens for which orientation will not be an issue are design considerations in a new system. Identified as possible future modifications of the original system, expanding and augmenting the table with more displays and devices can reduce the problem of limited screen real estate and provide more space to convey additional information related to the plan. This is also our major task when implementing this new version. As a Planning Support System, the system needs to provide basic building elements of when users design a neighborhood. The current system has a fixed set of prototype buildings that can be placed onto a map for urban design. Connecting with a complete, separate building database which can provide rich and accurate information for urban planning has been also identified by the users. elementsdb, introduced in the next section, is one such database.  3.2  elementsdb  elementsdb is a database maintained by elementslab at The School of Architecture and Landscape Architecture (SALA) at UBC. The researchers at elementslab are collaborators in this research. elementsdb is a collection of urban land use and buildings with accompanying descriptions and data. Figure 3.3 shows a list of cases in elementsdb. Each case in the database includes related resources, including 3D models built  21  Figure 3.3: Cases in elementsdb. This shows a screenshot of elementsdb that shows a list of cases [3]. using Google SketchUp. Figure 3.4 shows a detailed view of a case. Most cases in elementsdb are modeled after real structures in Vancouver, British Columbia, and other cities in North America. The cases in elementsdb can be used to assemble patterns for a design task.  3.3  Desirable Features and Iterative Design Process  In this section we describe a list of desirable features of the new system and how iterations of development are carried out. These desirable features are selected by consulting with our users – urban planners – and analyzing future directions of the standalone system.  3.3.1  Desirable Features  Because the new system builds upon the design of the standalone system, the overall goal is to keep those good features and improve others to provide a more extensible, interactive and useful system utilizing multiple displays. Below are some detailed features (or goals) that we wish to achieve during our first iteration of design and implementation.  22  Figure 3.4: An example case in elementsdb. This shows screenshots of elementsdb that shows the details of a case. Left: Facts and statistical data about the case. Right: Available resources for the case [3]. G1. A multi-display system and augmented table The first desirable feature identified is that it is necessary to extend the standalone tabletop application with other types of displays or devices. By doing this, screen real estate occupied by some user interface elements can be released and more comprehensive information can be displayed on additional displays. Connecting to one additional screen might be easy, but connecting to multiple screens is not. As described later in this thesis, a distributed and federated universal framework is 23  designed to connect to additional screens and/or devices and to augment the table with other applications. Such additional devices might include external projectors, another tabletop display or handheld devices such as a cell phone or a tablet computer. G2. Displaying additional information of the pattern As expressed in [21], a major future improvement in the application is displaying additional information about the existing neighborhood plan. In our new system, displaying additional information or visualization on an extended display (G1), would be the biggest combined desirable feature. G3. User-substitutable contents Users expressed a desire to be able to replace the contents of the application when necessary. Contents include available cases and focus area in the design task, as well as design targets (target population and jobs). Such contents were difficult to replace in the standalone system. G4. Connecting with a building database The cases in a building database provide useful resources to develop new plans. The researchers at elementslab asked that they be able to connect a building database (such as elementsdb) with the tabletop planning tool so the whole pipeline from producing building prototypes to using them to design a neighborhood plan can be connected. And all these tasks could then be done on computers or tabletop devices. G5. Possible to run on desktops and other devices Users also suggest that it would be good if the tabletop design application could be run on conventional desktop computers or other devices. The reason is that tabletop displays have much less accessibility for students and researchers than other devices, at least until they become commodity products. This is quite challenging because the interaction paradigms of multi-touch tabletop displays and conventional computers are considerably different. But it is also one of our design goals 24  to evaluate the portability of our design on different devices.  3.3.2  Two Iterations of Design and Development  The design and implementation process is intended to be an iterative, user-centered process. Two iterations of designs were carried out, with an informal user feedback and discussion in between. Researching the technologies and determining general requirements started from late October in 2010. Our first implementation finished in April 2011. Informal feedback was collected from various types of users. Implementing the second iteration of the application started in May 2011 and completed in August 2011. Chapter 3 and Chapter 4 discuss the first iteration of design and implementation and Chapter 5 and Chapter 6 discuss the second iteration, with Section 5.1 discussing feedback from the first iteration; it also discusses some of the reasons we pursued two iterations of implementation instead of an in-depth user study and only a single iteration.  3.4  System Design Details  In this section, some of the design choices made to meet each of the goals raised in the last section are described. Our first decision to make was how much of the standalone system should be kept.  3.4.1  Keeping the Standalone System  Based on our desirable features, the first iteration of design and implementation would be done based on the standalone system. Major design decisions illustrated by Fernquist [21] in the standalone system are retained, namely: • Using multi-touch table as the basis for our collaborative system • Using a complete digital tabletop application design instead of physical augmented applications • Keeping the system simple compared to other PSS while having enough key functionality  25  A final list of features to be added to the improved system were selected. How these features could be implemented and how to resolve problems that were encountered will be discussed in Chapter 4. Below is a list of major features / guidelines we tried to realize in our implementation. Section 5.1 and Chapter 7 reflect upon what was achieved.  3.4.2  Augment the Tabletop Application with Wall Displays  The first feature that was chosen to be added to the system was the ability to connect it with multiple wall displays. Some further research was undertaken that reveals design choices that needed consideration. What external display or devices should be connected to the table? The first question in augmenting the table with more displays is what displays to support. As argued in previous chapters and sections, possible candidates include wall displays, handheld devices and other tabletop displays. The main reason for choosing wall displays over handheld or other displays is that applications on wall displays are easier to implement. This is our first step towards a multi-display system. This feature also solves the problem of limited real estate in the current standalone system because an augmented system can display additional information on the auxiliary displays. Though we chose only to support wall displays, we decided that the system should be implemented to be able to eventually connect other devices and other types of displays to the system. As described in Chapter 4, various methods of connecting the tabletop display with external displays were implemented and evaluated. Our final implementation supports connection with any kind of display, ranging from handheld devices to large wall displays. What content is to be displayed on the wall displays? Based on goals G1 and G2 specified in Section 3.3.1, there are various kinds of information that can be displayed on an additional screen. Supporting each of these types of information is future work, but our goal is to design a system that can be easily extended to support all of them easily. 26  Real-time 3D rendering of the plan is one of the most desirable features in the to-do list of the standalone system. Displaying real-time 3D rendering can provide urban designers with a much better view of the plan. With computer-generated 3D rendering, urban planners can get initial assessments of the visual impacts of certain buildings to the entire pattern. Also some virtual reality simulations such as shadow simulation can be visually illustrated using a 3D view. Ishii et al. [29] described how computer-generated virtual reality can help facilitate urban planning. An overall statistical summary of a plan is another key piece of data that can be displayed on a wall display. Such data contain our existing statistics shown on the table, such as population and jobs provided, land use ratio, etc, but may also contain more sophisticated statistics that might be computed by an external process but displayed by the augmented system. Moving some interface elements to the wall to release screen space is another possible content on wall displays. For example, one plan is to add an interface to display detailed information of a case when it is selected, and there is no such space on the table to do this, so displaying it on wall displays might be an option. Of all these possible features, we decided to implement a real-time 3D rendering first, with an architecture capable of being extended to support other contents. Who should be viewing wall displays? Another question is: who should be viewing wall displays? Due to their size, wall displays are good for presenting information to a large group of people when they stand or sit in front of the wall display. So our choice of rendered 3D content should be suitable for display in such a public way. But there do exist situations where in a small group collaboration, a particular collaborator wants to see the 3D rendering of the plan from a certain angle while others who may not be involved in that particular aspect of the planning process want to see the 3D rendering from a different angle. If our wall displays should support such scenarios, the positioning of the wall displays would be crucial. They must be placed to face the target user. Also if some other information should be displayed using wall displays, like statistics, the audience might better be a crowd of people than just one person. In our later discussion of the implementation review, we discuss the possibility of  27  displaying information of personal interest on a personal device such as iPad or handheld device, instead of a wall display. How should wall displays be positioned? Related to the previous point, based on the intended audience of the wall displays and intended use cases the positioning of the wall displays is crucial. If the intention of the wall display is for public viewing, it should be placed facing the public or be in the center of the room. On the other hand, if it is intended to be used by a particular participant in a collaborative process as an extra display, it should be facing that participant and thus be parallel to the edges of the tabletop display. Also a big challenge of using wall display as a personal information presenter in a multi-display setting is that the system has to know who is interacting with the table, who is sitting on which side and who is issuing commands to control the wall displays. These issues are discussed in Chapter 7. In this research, we decided to first use wall displays to present information to a crowd of people and ignore screen direction problems. But later on, we will discuss how it could impact our design on the tabletop if it is indeed necessary to use wall displays to show information to a particular participant within a group of users. How to control the information presented on the wall display? As described in related work, how to control large what information is displayed and where that information appears on a display is always a “hot” research topic. In our system, because there is a central tabletop computer, it is natural that controlling the large displays – at least some level of controlling – should be done on the table. We decided that a simple widget or gesture system should exist on the table and that it should be activated on demand to control the wall displays. However, such controls should be as minimal and as non-intrusive as possible. Summary of design decisions for showing information on additional displays As explained in the above discussion, we decided in our first iteration of implementation that we would implement a real-time 3D rendering component which 28  can present the current plan in 3D. It would be displayed on a wall display which would be connected with the tabletop display in some way. The system should also support multiple displays to be connected and displaying different types of content. The initial intended use of the wall display is for public viewing and orientation of the wall display should not be very important in this scenario as discussed above. Also, controlling of the wall displays would be done on the tabletop display in a minimal and non-intrusive manner. This feature satisfies our goals G1 and G2, as described in Section 3.3.1.  3.4.3  Federated Architecture  When designing the system on which tabletop display and large screens are connected, we decided that instead of a centralized architecture, the multi-display feature of the system will adopt a peer-to-peer, federated architecture. That is, there is no single central program that controls all displays and programs running on those displays. Instead, each screen shows the interface of a single program. That program will retrieve information of its own interest from other programs. Such peer-to-peer architectures have a number of advantages. • Decouples the system to simplify maintenance. • One slow computation is less likely to influence other parts of the system. • It is easier to extend the system without a centralized master program. In our first iteration of implementation, a “semi-federated” architecture was implemented to connect screens. Our implementation includes a central controller that acts as a directory of the whole system. No actual computation is done on the central controller, but it helps screens find each other. Details of connecting screens together are described in the next chapter.  3.4.4  Design Task Selection  The second major feature of the desired new system is to add the ability for users to choose different “design tasks.” The prior version of system essentially hard-coded everything in the senses that the specific data that was used by the system, from 29  the map of the region of interest to all the available cases to be placed on the map, as well as planning targets such as population and job targets were all stored in files specific to the program or embedded in the program itself. As outlined in our goals, our users felt it would be necessary for these data as well as other content to be replacable by users without having to have knowledge of how the data was stored or maintained by the system. For example, the urban planners with whom we work were doing revitalization plans of the Chinatown region of the City of Vancouver in early 2011, but later in the year, they anticipated that they might shift their focus to another study of another region, such as a particular transit corridor. That other region might have a completely different set of available cases, different targets for population and jobs, and of course the base map for the region would be different. Therefore in our new system, it is necessary to remove all hard-coded parts of the program that are associated with these types of data. Instead, a function should be implemented for selecting design tasks, in which all these variables are included. A design task in our application, as described in Section 1.7, includes the region of interest, a background map, its physical geographical location, as well as available cases which can be placed onto the map. A design task also includes a set of design requirements and targets that could be interpreted and displayed by the application. All this information would be stored on the hard drive and in configuration files that can be loaded by the application. This feature satisfies our goal G3 that the content of the design task should be substitutable.  3.4.5  Connection with a Building Database  In the standalone application, images of available cases were derived from manually drawn illustrations (as shown in Figure 1.1). There are a series of problems with this approach. • It takes a large amount of time and effort to design these representations, even for something as simple as the “cards” shown in Figure 1.1. When users want to change to another design task, the contents have to be redesigned (though some should be reusable). • It is even more time consuming if a 3D rendering function is added. Each 30  of the cases has to have a 3D model corresponding to it. And every building model has to be designed, which also takes a lot of effort. Our collaborators proposed that it would be useful to have a building database (such as elementsdb) as the “data source” for the cases available in the planning application. If this could be implemented, it would also be a very useful application and extension of the elementsdb project. It can also solve the problems of producing contents for our planning application. Each available case just needs to be matched to one case in the database. Resources required by the planning application such as a top-down view image or 3D model can be generated from the 3D model of each case, which already exists in the database. Also, because the database contains much more abundant data about each case, more sophisticated statistics can be provided about a pattern. And lastly, at elementslab, there is a group of researchers that maintain the database, including those 3D models. So virtually no time would be spent on designing images and models for the tabletop planning application, because they can all be downloaded from the database. By connecting with a building database, goal G4 can be satisfied. Although connecting with a building database is a very promising feature for the research, a further analysis shows that in our first iteration of development, there is not enough time to do this. So as a compromise, the plan is that a “quasiconnection” with a building database would be implemented and a full connection is left for future iterations. By “quasi-connection”, it means that all the data of the available cases in a design task, including building models, case images and case facts such as stories of the building, population provided, jobs provided etc, would come from the real data from the database, but not through live connection. Those data are then manually converted to a format that the existing application could fully utilize, but without live connection to the database.  3.4.6  Running on Various Devices  The last goal (G5) identified by our user feedback analysis is the ability to run across different platforms. The standalone application runs on SMART Table that supports multi-touch. Users suggested that the application be able to run on desk31  top computers or later possibly other touch tables. Running on desktop computers would be quite different in terms of interaction, because desktop computers only support single point interaction. So we decided our implementation should try to make the interactions possible using a single-touch, while still utilizing the rich interaction provided by multi-touch. The ability to run the application on a wider variety of devices including, other types tabletop displays that might be remote, is discussed as part of the second iteration of development in Chapter 5.  32  Chapter 4  The First Iteration of the System Implementation In this chapter, we discuss how the features proposed in Chapter 3 were implemented. The implementation process is discussed both for hardware (how to connect multiple displays) and software (how to utilize these displays). Many challenges are discussed in this chapter, such as connecting multiple displays in an efficient and scalable way, how to render additional information such as 3D views of a pattern, ways to synchronize views between the tabletop display and the additional displays, and so on. Implementation of new user interactions to control the additional displays is also discussed in this chapter. Section 4.1 reviews the implementation of the current standalone system. The hardware configuration for the new system is discussed in Section 4.2, followed by a high-level summary of the software and interaction implementation in Section 4.3.  4.1  Implementation Review of the Standalone System  Implementation includes two aspects – hardware (what hardware is used and how the system is configured) and software (how the software is developed to support multi-user collaboration).  33  4.1.1  Hardware Implementation  In the standalone implementation [21], the system runs on a multi-touch tabletop computer. No other special devices are used. The tabletop computer used is a SMART Table interactive tabletop display, a commercially available tabletop computer built by SMART Technologies [15]. Under the surface of the SMART Table are a regular computer, a projector and an infrared camera. All computation and software runs on the computer. The projector projects an image onto the screen of the tabletop display through rear-projection. The infrared camera is used to recognize touches. The SMART Table uses a technology called Frustrated Total Internal Reflection (FTIR) [5] to support multi-touch [15]. The technology was first invented by Han [24] in 2005. A layer of total internal reflection material lies under the screen of the table. Infrared light shines from the side of the screen through this layer. As infrared light travels through the total reflection layer, light is mostly contained within the layer; because of internal reflection the light cannot travel outside it. When a finger touches the screen, the total internal reflection layer is disrupted and thus some of the infrared light at the point of touch is refracted downwards. The infrared camera at the bottom of the table captures infrared images of the table surface, so a finger touch appears as a bright spot in the image while the background of the image is relatively dark. The images captured from the camera are constantly fed into the computer in the tabletop, the software running on the computer is responsible for analyzing the stream of images and recognizing bright spots that are likely to be touches. The SMART Table can process up to 120 touches simultaneously [15]. The open source community NUIGroup.com has extensive description and detailed images on how FTIR works. Please refer to [5].  4.1.2  Software Implementation  In the standalone system, the planning support system is implemented according to the design previously described in Section 3.1. The system accepts multi-touch and can support collaboration among a group of people. The system contains a map at the center showing the focus area. The  34  map can be zoomed or panned by pinching and dragging the map with fingers. The system contains a case inventory on the sidebar, from which buildings can be dragged and placed onto the map; a “saved patterns” list that users can save and use to compare different patterns is at the bottom of the screen. On the right of the screen, a list of statistics is provided to evaluate the pattern. The buildings placed on the map scale proportionally with the map as the map zooms in or out. The system’s contents are derived from elementslab. Though the contents are highly similar to the cases in elementsdb, as a first version prototype, the standalone system uses cases adopted from a paper-based presentation of elementsdb and does not try to directly connect with the database. Instead, individual cases are designed manually. The contents and data of the system are either stored on the hard drive (such as images of the cases and the map), in a configuration file (such as statistics of each case) or hard-coded in the program (such as population and job targets). As a planning support system, an important issue is supporting physical size and geo-location of buildings and structures. In order to allow users to design a plan of a neighborhood, the system needs to know, in some way, the physical size of each building and where it is placed. In the standalone system, such data is stored as a ratio relative to the map image. For example, if the map image of the area is 5000 pixels wide and covers 500 meters east to west, then each pixel is 0.1 meters physically. If the building image is 500 pixels in width and is supposed to be 10 meters wide, then the building image’s resolution is 0.02 meters per pixel. So if the building image is to be placed onto the map, the scale ratio would be 0.2 (0.02/0.1), the image has to be shrunk to 20% of its original size in order to have the same physical scale with the map. In the standalone system, every case image has the same resolution and the case/map ratio is hard-coded in the source code. But the standalone system does not contain any geographical or physical information other than relative sizes between the map image and case images. This is a big problem if we want to export the pattern created in the system and use it with other geo-based systems (such as demonstrating the pattern in Google Maps or Google Earth), or if we want to import data from other systems where the relative sizes might differ from those we currently use. The standalone system was implemented using C# and Visual Studio 2008 un35  der the Microsoft Windows operating system. C# was chosen as the implementation language mostly because it is very easy to use for interface prototyping, and because the multi-touch API of SMART Table is provided in C#. The SMART Table SDK provides easy-to-use controls that can respond to touches on the table. In the following sections, we discuss how the first iteration of the new system is implemented.  4.2  Hardware Configuration  In this section we describe implementation issues surrounding the hardware configuration for the new system. The main issue discussed here is how the system can support multiple displays connected to the tabletop display. The term “component” is used throughout this chapter. In our system, components are defined to be standalone programs that can connect together to form a bigger system. For example, the pattern designer on the tabletop display is a component; a 3D pattern viewer program showing 3D renderings of the current pattern on the large screen is another component. These two are separate programs, but they connect in some way. They are relatively independent but visualize some common information. Each component usually occupies a whole display (though it is possible for a display to contain many components). Therefore sometimes when referring to a display, we might also be referring to the component running on it, and vice versa. In our federated architecture, a “component” is a peer or node that connects with other components in a federated fashion. Messages travel between components and each component visualizes part of the information. This mechanism will be describe later, in Section 4.3.1.  4.2.1  Constructing a Multi-Display System  The first step in implementing the new system was to make it connect with additional displays. The first attempt to connect to additional displays was through multiple monitors. The tabletop display itself has a computer in it. So we connected an external monitor to it through its internal secondary VGA port. It was verified that with an additional monitor plugged in, the multi-touch capability still 36  works. This was the foundation for our next steps because if plugging in additional displays could stop multi-touch from working, another way had to be found to do this. In this configuration and for all subsequent configurations, each component is implemented as an independent program. They are connected through web services in a peer-to-peer federated architecture (even though they are sometimes on the same computer, as in this configuration). Details of how components send and receive messages between each other are described in Section 4.3.1. The table does not expose any VGA or DVI port on its exterior. The secondary VGA port is on the computer inside the table, and there is only one, which is limits the number of auxiliary displays we can attach. But there are two USB ports on the side of the table that can be used to connect additional displays. Having only two USB ports is also somewhat limiting if many displays have to be attached to the tabletop. So the next step was to try to use a USB hub and plug in multiple USB-to-VGA adapters. The system’s diagram for this configuration is shown in Figure 4.1. We tried two USB to VGA adapters to verify that the system can support connecting with more than one external display. The system worked; the tabletop can drive two additional projectors with multi-touch still working. However, one limitation of this solution is the USB-VGA adapter can only support 6 adapters connecting to one computer at the same time. Beyond that, the driver cannot support more adapters. This might not be enough if we want to construct a bigger system like a circular meeting room with all walls having projected displays. Nevertheless, we decided to first implement the system using this basic configuration. After the 3D rendering component was implemented and the system was up and running (see Section 4.3), a serious problem appeared. The computation capability of the tabletop computer is very limited. Though it has a dual-core processor and a dedicated 3D video card, CPU usage could still spike to 100% and cause the system to respond very slowly or even freeze. The CPU utilization was measured for each of the programs in the system. Table 4.1 summarizes CPU usage. The 3D rendering component is implemented as a standalone application using Google Earth (see Section 4.3.2 for details) that communicates with the standalone tabletop application. Though it is fairly easy to render buildings in Google Earth, it appears that it easily monopolize the computational power of a whole CPU core 37  USB To VGA Tabletop Display  USB Port  USB To VGA  USB Hub  USB To VGA  Figure 4.1: System configuration – USB hub plus USB to VGA adapters that each connect to a projector Application Touch sensing Pattern designer 3D pattern viewer  CPU Utilization 30% to 35% 10% to 30% up to 50%  Note Stable Depending on user actions Can occupy a whole CPU core  Table 4.1: CPU utilization of the programs on the tabletop display when viewing from street level. There are various discussion threads on the Internet about this issue and people suggest reasons ranging from a video card driver problem to Google Earth compatibility issues. While this issue could not be solved, we think that even if it could be, centralizing all computation on one computer is still a potential risk to future expansion of the system. So we decided to solve this issue once and for all with a more flexible and distributed architecture. We thus tried utilizing more computers and connecting them through Ethernet to reduce the computational load on the tabletop computer. This step is described in the next section.  4.2.2  Network-Based Multi-Display System  Having decided to use a network for data exchange, we investigated how to implement this. Our first trial was to connect the tabletop computer into a switch (through a USB-Ethernet adapter) then connect another computer or some other computers to the switch and connect external displays to them. The 3D rendering  38  program then runs on the secondary computers and communicates through the network with the tabletop computer, getting data for the current pattern and rendering it. This configuration is shown in Figure 4.2  Tabletop Display  USB Port  USB to Ethernet  Switch  Computers  Figure 4.2: System configuration – via an Ethernet switch After deploying the system into an urban design course classroom (see Section 5.1), another problem appeared. In this configuration, there was too much wiring that required considerable amount of work to be done to re-configure the room containing the system. So a different approach was taken using a wireless router to connect all of the components. The tabletop display contains an internal WiFi card that can connect to a wireless router. Our secondary computers can also connect to the router. In this way, the physical configuration of the room can be very clean with almost no wiring (except for power). Also the system has even better extensibility because WiFi routers do not have strict limits on how many devices can be connected to them. A diagram and a photo of the final configuration are shown in Figure 4.3 and Figure 4.4, respectively. This solution can also support extending to different types of displays and input output devices, because WiFi capability is popular on a lot of devices, such as cell phones and tablet computers. For example, an iPad can also be connected into the system through WiFi, as can other tabletop displays. It is greatly superior to just plugging in projectors to the tabletop display (up to the bandwidth limits of WiFi). However, the system still requires different components to connect to the same  39  Tabletop Display  WiFi Router  Computers  Figure 4.3: System configuration – via a WiFi router router. Later, in the second iteration development, another solution with support for remote peers is proposed.  4.3  Software and Interaction Implementation  In this section, we describe how software for the system was implemented, including how to connect multiple displays in the software, how to develop visualizations to show on additional displays, how to synchronize those displays, and other issues.  4.3.1  Software Implementation of a Network-based Multi-Display System  While we had decided to network each component and display, a big issue remained on the software side: how can they find and communicate with each other? As discussed in Section 3.4.3, a federated architecture design was adopted. In every connected computer / display, there are one or more programs that are components of the whole system. They are independent programs that can exchange information with the rest of the system. They only receive information from components that they are interested in, without a centralized manager controlling the whole system. However, as described later in this chapter, there is a “central controller” in the system, but it is only a “directory service” of the system, providing information on where each component is. In the second iteration of the implementation, a 40  Figure 4.4: Photo of the final system configuration. The large image is produced by a projector connected to the computer on the left, which is the secondary computer, connecting to the WiFi router on its left. The tabletop display on the right also connects to the WiFi router.  41  completely federated architecture is implemented. Communication protocols In our new implementation, each component of the system has a common module that provides “services” to other components. Services are data and commands that can be used or invoked by other components. For example, the tabletop application has a service to generate a summary of the current pattern in a certain format (see Section 4.3.2). This summary can then be used by other components such as the 3D rendering program to render the pattern in 3D or by a calculator to provide an automated qualitative summary of the pattern. Each component of the system provides services through an REST HTTP interface [13]. That is, each component that will be providing services has a mini web server built into it. Components that need to request a service would then send an HTTP request to the target component and analyze the response. For example, a tabletop application running on port 9801 on computer might provide a service to return the summary of the current pattern (in some format like XML) at Here the host name and port of the HTTP request indicate the position of the application. /pattern indicates which service is to be invoked and /0 is the argument of the request, meaning that the current pattern’s detail is to be returned (similarly /1 might mean the first saved pattern, /2 means the second, etc). Responses will be returned in a format that can be understood by both the service provider and invoker. Services are based on HTTP web services, meaning that it can even be used by other web applications in the future. Figure 4.5 shows a diagram of a service request and data flowing between components. Note that the diagram is only for demonstration purposes, the code segment in it does not reflect actual code. Component discovery In the above section, it is assumed that each component knows where other components are. But it is not a trivial issue to discover components even when they are connected to the same network. Some existing P2P technologies use broadcasting to discover peers [30], but in our implementation, because HTTP web service  42  Tabletop Application (Pattern Creator)  Request Received  Generate Response  Projector Application (3D Viewer)  <Pattern> <Case id=“11022” heading=“120” longitude=…  Send Response  Request Textual Representation of Current Pattern  Response Received  th Render this Frame  Figure 4.5: Communication between components in the system is used as an internal communication method, such discovery is not easy to implement. In our system, there is a centralized agent that acts like a manager that knows the whole system. Each component, upon starting up, first searches for this central controller in the current network. For example, if the current network’s address space is to, each component will try to connect to the port of the central controller (9200 in our implementation) on every address in this address space. Once a successful connection is made, the component then registers itself with the central controller, sending its own address, port and component identities. After all components have started, if one component wishes to invoke a service from another component, it will first ask the central controller for the address and port of the target component, and then send a request according to the address information returned from the central controller. In order to make this “search and register” paradigm to work, all computers in the network have to have IP addresses that fall into the pre-defined address space. This can be done by changing configurations of the WiFi router’s address pool, or setting IP addresses of each computer manually. Section A.1 is a piece of pseudocode in C++ showing the component discovery process. Figure 4.6 is a screen shot showing the central controller, displaying four registered components, one  43  table application (Pattern Creator) and three external projector applications (Plan Viewer).  Figure 4.6: Screen shot of central controller With this in place, there is a functional system that allows connecting with extended displays (as well as other devices and computers) to the tabletop display, in order to augment and extend its functionality. Next we will describe how to implement the 3D pattern viewer to be displayed on an external projector.  4.3.2  Three-Dimensional Pattern Viewer on Projected Wall Displays  A 3D pattern viewer application is implemented and connected with the table application using the federated infrastructure described in the previous section. The pattern viewer can display near real-time 3D renderings of the pattern and can support controlling the view angle on the tabletop display. Using Google Earth to render 3D view Two ways of rendering a 3D view of a pattern were considered. The first one is writing a 3D rendering program from scratch. Though this method could provide 44  us the most flexibility, it has many problems. • It requires a long time to develop the application. • It requires more time to maintain the system, because the rendering program itself is complicated. • It is hard to use geo-location information in the rendering program. • There is no model/terrain/map of the surrounding area of the design task. • It is hard to extend the application to add more information such as layers on the ground showing additional information. Because of these considerations, using Geographic Information Systems (GIS) like Google Earth becomes a viable solution. Google Earth supports loading and displaying a variety of layers in its 3D scene, including external building models. Below is a list of advantages of using geo-info systems such as Google Earth. • Development time is minimal, software is maintained by another company; we only need to develop certain scripts to load our models and layers into Google Earth. • Google Earth has a module called Google Earth Plugin, which allows embedding a Google Earth rendering window into other applications and websites. • Google Earth itself is essentially location-based; every object in it has latitude, longitude, heading and altitude. It better aligns with the nature of our planning support system. • Google Earth has extensive information about many areas, such as terrain, existing building models, satellite imagery, etc. This information can provide a rich set of background information on the target area for a design task. • Google Earth has a complete scripting and extension system, through which more functionality and visualization can be easily added.  45  However, using Google Earth also has some risks. A lot of data such as building models are copyrighted materials owned by our users (elementslab in our situation). Embedding them into Google Earth is risky because it is hard to guarantee that these models are not uploaded onto Google’s servers. For now, we have not heard of any report that user contents are being uploaded to Google’s servers, but when you transfer your data to Google Earth, theoretically it can do anything it wants with the data. Even cutting off Internet connection after terrain and buildings are loaded will not work, because the program can, if it wants, cache any data passed to it and upload at a later time. On the other hand, another risk is that, while for now there seems to be no explicit terms in Google Earth’s license stating that it cannot be used as part of a program that embeds models in it, we might run into legal issues in the fututre if the application is to be used outside of academia. These issues involving third-party programs need to be considered in any follow-up work. In our implementation, the easiest way to control Google Earth is to have Google Earth Plugin embedded in a web page [6], but our program cannot just be a web page, therefore an application was created with a web browser control in it, displaying a web page that contains the Google Earth Plugin. In this way, it looks like Google Earth is embedded in our program. The host program can then call web browser control methods to invoke JavaScript functions to control Google Earth. Figure 4.7 (right) shows Google Earth embedded in a program. Generating contents for Google Earth The next step is to display useful content in Google Earth. Google Earth uses formatted configuration files called Keyhole Markup Language (KML) files to display contents and extend itself. According to the KML documentation, “KML is an XML notation for expressing geographic annotation and visualization within Internet-based, two-dimensional maps and three-dimensional Earth browsers.... KML is an international standard of the Open Geospatial Consortium” [10], which means it may be used not only in Google Earth or Google Maps, but also in possibly any other geo-info systems that supports the format. Thus using Google Earth and KML format can facilitate changing to another 3D viewer that also supports  46  KML format. To control Google Earth, there are APIs provided by Google Earth Plugin that can load KML files at run-time. These APIs can be invoked by the host application through JavaScript calls to the web page containing the Google Earth Plugin. Section A.2 shows a sample KML code fragment for adding a 3D model to a certain location. The KML file is intended to be self-explanatory. Our task is then to generate such a KML file from the current arrangement of cases on the map and let Google Earth load and display all of the information. The first problem appeared while trying to generate a KML file: there is no geographical location associated with the contents in the standalone system. For example, when performing the task of designing the neighborhood at Knight Street and East 49 Avenue in Vancouver, the only piece of content that implies the geographical location is the satellite imagery of that area. The program behind the scene does not know where the design task is being carried out. So geographic information had to be added to the map. Geographical coordinates of the four corners of the map were added to the configuration file of the design task. Because the map image might not be strictly aligned to the cardinal directions, it is necessary to have the coordinates of all four corners. After the coordinates for each of the four corners are known, the geographical location and heading of each case can be calculated based on its position relative to the map through interpolation algorithms. The Google Earth API only accepts DAE files as 3D models to be loaded into Google Earth. DAE format, whose official name is COLLADA, is “a COLLAborative Design Activity for establishing an interchange file format for interactive 3D applications. [2]” It is an open standard. The format can be loaded in many 3D applications. However, in the building database (elementsdb), there are only Google SketchUp models. DAE models have to be converted manually from these SketchUp models. It is then easy to generate a KML fragment that adds a list of models to Google Earth according to their arrangement on the tabletop. It is only necessary to match each case with a corresponding model file on our model server. This is straightforward, because each case can be assigned a unique ID and models create with those IDs as their names and stored on a server. In this way, Google Earth can render 47  the current pattern in 3D in our rendering program and the image can be projected onto a wall. Figure 4.7 shows a side by side screen shot of the same pattern on Vancouver’s Chinatown, with the table application on the left and the corresponding Google Earth rendering showing on the projected large screen on the right.  Figure 4.7: Screen shot of tabletop pattern designer and 3D pattern viewer  Real-time updating Another issue with using Google Earth to render current pattern’s 3D model is that Google Earth’s contents follow a “load and render” paradigm. You cannot achieve a very high updating frequency by asking Google Earth to load a KML file every few milliseconds. This decreases performance and even causes the process to freeze. However, Google Earth’s KML API does provide a way to automatically update the content periodically. KML has a construct called Network Link, which can allow Google Earth to load another KML file periodically as configured in the primary KML file. Section A.3 is a sample of KML fragment that requests to load another file and refresh every 2 seconds. A KML file containing such network links can be generated that makes Google Earth query the KML generator service, which in turn generates a new KML file that renders the current pattern in 3D according to the current pattern on the table application every 1 or 2 seconds. However a problem is that every time Google Earth refreshes the current KML contents, the models in Google Earth have flicker 48  effects. This is due to the fact that when the contents of KML change during autoupdate, Google Earth removes all current contents and reloads the KML file. To solve this, a caching mechanism was added on the pattern creator side. In the KML generator service, a cache is maintained. If the pattern does not change after the last time a KML is generated and sent to the service invoker, then only an HTTP 304 (not modified) [9] will be sent. This signals to Google Earth that it is unnecessary to refresh its contents. In practice, Google Earth does respect HTTP 304 response and only refreshes the models (with the consequent flicker) when there is actually a change on the table. This greatly reduces the flickering effect. At present, the process is still only near real-time, because the smallest interval of a network link in KML is 1 second. Figure 4.8 shows a diagram illustrating the whole process of near-real time update for a scene.  Google Earth receives the response, refresh the content if necessary and wait for a short period of time.  Tabletop Application notifies Patter Viewer to display current pattern.  Pattern Viewer generates KML file to control embedded Google Earth to refresh repeatedly.  Embedded Google Earth sends requests to Tabletop Application, asking for KML content of the target pattern.  Tabletop Application checks if the target pattern is changed. If so, respond with the KML content, if not, respond with NotModified (304).  Figure 4.8: Real-time update process of 3D pattern viewer  4.3.3  Controlling the 3D Pattern View from the Table  A big disadvantage of the large screens discussed in the introduction is the lack of effective interaction techniques on them. Without additional devices to use as controllers, a number of techniques have been developed that use motion tracking to interact with large screens. This is beyond the scope of our work, primarily because of the cost of installing and maintaining tracking the necessary tracking 49  hardware. Luckily an interactive tabletop display is our primary workspace, so we can use interaction paradigm to control the large screen from the tabletop display. We argued in Chapter 3 that controlling the big screen from the tabletop display should be kept to a minimum and as non-intrusive as possible. We decided to first implement a pop-up widget on the tabletop computer to control the view angle and visualization of the 3D pattern rendering program. The pop-up widget can be activated by holding a finger on the screen for two seconds. This causes the widget to automatically connect to one of the available projectors if any are connected to the system. If there is more than one projector, users can touch the center button to switch between projectors. Because multiple people collaborating around the table is allowed, multiple widgets might be activated to control different projectors. Each pop-up widget has designated a color and the projector program displays that color surrounding the border for that projector, indicating which widget on the table is currently controlling the pattern viewer. Currently the program is designed so that it is possible to share a projector among multiple people (widgets), so they can each gain control of it, but the projector is always updated based on the latest request. Figure 4.9 is a screen shot of the widget UI showing the sharing of one projector with two active widgets/users. Three views of the current pattern are provided for the external displays. The first one is a bird’s eye view. By touching the 3D house icon, the widget is activated to bird’s eye view. A camera circle is shown, indicating the camera position, and the camera is always looking at the center of the widget [19]. The semantics here are that the center of the widget is the focus area, while the camera icon specifies the viewing position of the bird’s eye. In this way, a “fly around” effect can be simulated by dragging the camera icon around the target area. The angle of the bird’s eye is fixed to 30 degrees for now. How to control this degree remains a future work and will be discussed later. Figure 4.10 shows a bird’s eye view of the pattern in Figure 4.9. Note that the orange and blue dual-color frame indicates that the projector is shared by two widgets (orange and blue colored) on the tabletop pattern creator. The second view is a street level view (as shown in Figure 4.11). In this view, the semantics of the widget change, with the center of the widget now representing the position of the camera on the ground while the outer circle/arrow represents the 50  Figure 4.9: Screen shot of two active widgets sharing one projector  Figure 4.10: Bird’s eye view of a pattern  51  direction to look at. The reason for this is that street view is more often used to simulate a person’s view, who is more likely to be standing somewhere and looking around, rather than “running” and “looking” around a focus point. However, the length of the direction arrow has no effect on the final 3D rendering. But the farther away the control arrow is from the center of the widget, the more precise angle it can control.  Figure 4.11: Street view of a pattern The last view is a top-down view (as shown in Figure 4.12), similar to the tabletop display’s view. This view allows the same top-down view of the tabletop display to be shown on the large screen. The arrow here represents the “up” direction of the large display. For example, if the arrow points to south, then on the large display, the pattern is shown with south on the upper edge of the screen. A series of possible HCI research topics can be derived from this part of the implementation, related to controlling and sharing multiple displays. They will be elaborated on in Chapter 7.  4.3.4  Design Task Selection and Integration with a Database  The next feature called for in the system design is the replacement of contents and quasi-connection with a building database. As mentioned in the previous sections, 52  Figure 4.12: Top-down view of a pattern Data Case list Case image Map Case size Geo-location Design targets  Stored A CSV file, containing links to the case images Separate image files in a folder An image file Hard-coded ratio in relation to map size None Hard-coded  Table 4.2: How data of a design task are stored in the standalone system most of the data related to the design task were hard-coded in the standalone program, such as design targets, case/map resolution ratio, etc. In our new system, changes were made so that contents can be easily replaced by loading different design tasks. Table 4.2 provides a list of how different data are stored in the standalone application. The items italicized are not replaceable without modifying the source code. What we did is to include all of these variables for a design task in an XML configuration file but keep others unchanged. Case list, images and the background map are still stored on the hard drive with references to them in the XML configuration file. Section A.4 is a code segment of the XML configuration file.  53  A second design task for the new system was then built based on our modifications. The design task is to study a revitalization plan of Vancouver’s Chinatown area. When building the design task, available cases were extracted from a case set in our building database (elementsdb). The top-down view images were manually converted from the associated SketchUp model for each case. 3D massing models were also exported from the SkecthUp models. These 3D models can be used in our 3D pattern viewer. A sample of the configuration file is shown in Section A.4. Data such as design task name, design targets, resolution of the map and link to the case list index file are stored in the configuration file, as well as the geographical location of the map and design task. Having implemented replaceable contents, a design task selection interface was also needed. It allows the user to choose a design task when the system is starting up.  4.3.5  Running on Different Devices  Because of the built-in compatibility of the SMART Table with a standard mousebased interface, no special implementation is needed for the system to run on conventional desktop computers. The SMART SDK supports using a mouse to emulate touches. All operations on the tabletop application, including activating the pop-up control widget, can be done through single mouse click interaction. However, the SMART SDK does support plugging in multiple mice, which simulates multiple touches on a conventional computer if multiple mice are coordinated.  54  Chapter 5  The Second Iteration of the System Design Following the first iteration of design and implementation, user feedback was gathered. In this chapter, we summarize the user feedback and comments that were gathered after the first iteration of the new system was deployed for testing and we discuss some of the immediate implications of the feedback (additional in-depth analysis is in Chapter 7) and identify areas for later, more broadly-based user studies. We discuss briefly the rationale for choosing to pursue another iteration of design and development to further improve the system while leaving a full user study for later. Similar to the first iteration of design and implementation, desired features and requirements of the second iteration were identified by interviews with users and analysis of the existing software. This process is in this chapter, and thoughts on how to re-design the system and each of its components to meet these requirements are presented. Requirements and desirable features in this iteration of development are described in Section 5.2 and in Section 5.3, the system design and discussion of the design are presented.  55  5.1  Review of the First Iteration Design  After the first iteration of the new system was finished, informal feedback was gathered from our users. We installed the tabletop system in an urban planning classroom to allow students to use the system for real neighborhood design tasks. The system was configured as described in Section 4.2. Though not so many students used the system due to the students being in the middle of a busy project at the time we moved our system into the classroom, we still gathered some useful feedback from the students, especially some possible problems that need more work. The students generally said that “the system is fun to use” and that the statistics displayed on the table were useful. Some thought the case cards might need better design. Regarding the multi-display feature of the system, some of the students thought it was quite interesting to see renderings of the patterns. Both the instructor and the students in the course proposed that utilizing the existing 3D building models in Google Earth and overlapping them with the new patterns designed on the table could provide a direct visual comparison of how the new plan differs from the old one. A quick fix was made to include this feature in our system. We also observed that the pop-up widget is sometimes unintentionally activated and can cause interruption. This may need to be examined more thoroughly in follow-up user studies. Other issues include the user interface taking up too much screen space and making the map area too small. This is a legacy issue derived from the standalone system. But with the use of additional displays, this could be resolved, again an area for follow-up studies. It seemed that most people expect case cards on the table to have physical behaviors such as inertia, but our current system did not support this. After that, we moved the system to a local conference and demonstrated the system in a live poster and demo session. During the three-hour demonstration time, a few dozen people tried out the system. Most of them were researchers or graduate students from universities and companies across Canada. Similar to the classroom field study, the first reaction from most users was positive. Most of them thought the system was “cool” and fun to use and could provide useful information through additional displays to the primary tasks on the tabletop. We  56  made some other valuable observations, including that the control widget might be a little bit big compared to the screen size. Some people got disoriented and did not know where on the projector they were looking. It was hard for them to associate the top image on the table with the 3D rendering on the projector. This disorientation problem requires additional studying about how to combine 2D and 3D contents in a multi-display system. Some users also pointed out that they actually would prefer the 3D rendering to be on the tabletop computer. Switching back and forth between the tabletop display and the large screen display distracted them and caused disorientation. This raises again the question of what contents should best be presented on different screens. This might differ among various use cases. For example, for small group collaboration or personal use, users might prefer that the 3D rendering be co-located on the tabletop display. But in a public demonstration or presentation scenario, they might prefer it on the wall. This issue definitely requires further formal research. It will be discussed in our future work section. Implications of all these issues are reviewed again in Chapter 7 The third assessement was with other researchers and urban planners who were invited to try the system. Problems identified were similar to those summarized above. Our collaborators in SALA proposed some additional in-depth improvements of the system and they wanted to deploy it as part of a much wider field study in a newly constructed “decision theater” in the newly completed Centre for Interactive Research on Sustainability (CIRS) building at UBC. The decision theater will feature many tabletop display computers, a lot of projectors, as well as handheld devices such as iPads. This is a perfect place for us to deploy the system for an in-depth user study. For the next step of the research, we saw two choices: either pick one of the problems identified and do a formal user study or improve the system to prepare it for the CIRS decision theater and leave user studies until later. We decided to improve the system based on new requirements gathered from our collaborators at SALA and postpone user studies until later when the CIRS facility was fully operational. A summary of major improvements identified by our collaborators at SALA, in addition to the issues identified above, is the following.  57  • Replace the map layer of the system with a real map application (such as Google Map). • Provide full integration with a building database. • Display more information on projected screens. • Provide support for personal devices such as the iPad. Details of all these requirements and improvements are presented and discussed in the next sections.  5.2  Requirements and Desirable Features  In the second iteration of development, most important features are proposed by our primary users and collaborators - urban planners in elementslab. Additional desirable features are also identified from the user feedback gathered.  5.2.1  Requirements Suggested by elementslab Researchers  Requirements suggested by researchers associated with elementslab are mainly centered around better support for geo-location and deeper integration with the building database. These requirements can greatly improve the system’s functions as a Geographic Information System (GIS). Use real map systems A first natural extension based on the improved system described in Chapter 3 and Chapter 4 is to have the workspace map replaced by a real map system. Real map systems such as Google Maps and Bing Maps can provide us with the most comprehensive information about a target area for a design task. In our first iteration application, the background map is static and has to be copied from other sources, such as those map systems. The fact that it is only a static image also makes it hard to display additional and dynamic information while doing the design task. The advantages of using a real map application as the design background include: 58  • Using a map application can save the time spent on getting the map images. Creating a high-resolution map image to use in the system is not an easy task. It usually involves capturing and stitching high-resolution images of nearby regions together. • The data of a map application is always up-to-date. Whenever there is a new road or a new building, map applications such as Google Maps will update the contents accordingly. • Images in map applications usually come with different level of details, so at different zooming levels, higher details of a neighborhood can be seen. Maps systems also usually provide different kinds of visualization to the same area. For example, Figure 5.1 shows a satellite map and a street map of the same area in Google Maps. • Using a map application makes it easier to integrate with other GIS applications. The contents displayed in one map system can be converted to some contents displayed in another system, with much less effort, because all GIS applications are geo-location based. • The last important advantage of using a map application is that most map applications support displaying additional layers blended with the existing map. For example, researchers might want to display a projected population density layer over the existing map. This layer of information can be dynamic, depending on the current pattern being designed. Based on this, the first major improvement we decied on is to include interactive map applications as the background of the design task. The map would still be manipulated using pinch and drag gestures to zoom and move. It is desirable to implement this style of interaction so that our system is similar to other map applications on multi-touch surfaces, such as map applications on the iPad and Android devices. Connection with the building database In our first iteration a quasi-connection with the building database (in our system, elementsdb) was implemented. In our second, improved implementation, a live 59  Figure 5.1: Different types of maps. Left: A satellite map. Right: A street map. Image from Google Maps [8]. connection with the database is made possible. To facilitate this, the case inventory bar becomes an interface displaying all cases in the database satisfying certain criteria. For example, users can filter cases by case sets in the database. In this way, if users wish to carry out a new design task, all they need to do is to create a new case set in the database, and then select that case set for use in our tabletop pattern design system. By connecting with a database, the table can display much more detailed information for each case. This would have the added benefit of making the management of cases and case sets the task of administrators of the building database, while users of the Planning Support System (PSS) would simply use these data. All these data entry tasks do 60  not need to be repeated outside the database, so maintenance work will be reduced.  5.2.2  Other Requirements  In additional to deeper integration with other GIS and elementsdb proposed by our collaborators at elementslab, some additional improvements were proposed. These improvements are either natural extensions of the first iteration of development or were identified as problems in the user feedback sessions. Connecting to additional and remote displays using a totally federated architecture As an extension of the first implementation, improving the ability to connect to other displays and devices and extending our multi-display system is another requirement. Our design goal is to eventually support connecting to remote devices through Internet and improve the way the system connects to local devices. Currently, the network needs to be specifically set up in order for components and devices to discover their peers (see Section 4.3.1). In the improved implementation, this needs to be revised. Instant and easy connection is our goal. Our idea is to use a totally federated messaging architecture to connect components of the system. Details of how communication is designed are discussed in Section 5.3.3. Supporting easy extension To integrate with map applications and database sources such as elementsdb, the system needs to be designed to work with these systems. It is important that the system could support easy extensibility for use with other data sources as well. For example, our system is planned to support different types of maps, such as both Google Maps and Bing Maps and possibly static maps, as were used in our previous implementation. Also, the case database should also be exchangeable. The system should be able to easily switch to other data sources, with compatibility for supporting the old database, for example, or new databases.  61  Design at other levels As suggested by the elementslab researchers, making design tasks to focus on different scales of design would also be an improvement. In our previous system, users could only manipulate cases in a pattern, and each design task usually only contained one pattern. That is useful if urban planners want to design a neighborhood from the ground up. But the reality is that oftentimes a new neighborhood can be assembled by putting together various generic patterns. Also, sometimes urban planners might want to move a group of cases as a unit, keeping their relative positions and headings. This is similar to the “grouping” feature in many design and drawing tools. This ability should be included in the new system. Other improvements Other improvements are primarily focused on UI re-design. The UI has to be redesigned so that it takes minimal table space and can support more fluid interactions. Details of how we re-design the UI are presented in part of the next section.  5.3  System Design  Having discussed the requirements and features for the second iteration of development, we need to come up with a system design for how to implement these features.  5.3.1  Improving versus Rebuilding  The first choice is whether it is better to improve the previous system or to rebuild the whole system. We observed that the improvements proposed by our users and ourselves include changing almost everything in the current system, so we decided to design the system from scratch. However, any source code that could be used in the new system should be re-used.  5.3.2  System Architecture  The overall architecture of the pattern creation system was completely re-designed. The system has a highly modular structure. The system contains four parts includ-  62  ing UI, data, communication, and extension. The UI layer contains a map application provider, a touch driver and an UI implementation. The data layer contains a database provider. The communication layer contains different ways to communicate with other components and devices. Lastly, the extension layer provides support to extend the functionality of the system. The system diagram is illustrated in Figure 5.2. Google Maps  UI  Mouse  Bing Maps  SMART Maps  Touch  Static Maps  Surface UI Layer Extension 1  elements db Extension Layer  Data Layer Other Data Source  Extension 2 Comm. Layer Abstraction Layer Implementation Layer  Pattern Designer on Tabletop Display Network  Other Displays  Other Projectors  Stats Viewer  3D Pattern Viewer  Handheld Devices  Other Tables  Figure 5.2: System architecture Following the OOP paradigm and in order to implement easy extensibility, the system is divided into abstraction and realization/implementation layers in each of the modules. In the abstraction level, generic behaviors of the module are defined. For example, each map provider, regardless of how it is realized, should provide basic interfaces for getting the current location and scale of the map. Another 63  example is that for every database of cases, it must contain a list of currently available cases, regardless of how these cases are loaded. Abstract level components are programmed using interface and abstract classes (see Chapter 6). Then each implementation of those abstractions will derive from them. In this way, modules do not need to know how others are implemented; they only need to know their interface so modules can interact with one another. Any instances of one module being passed to another would be of an abstract type. For example, the UI provider never knows if it is dealing with a Google Map or a Bing Map. In this way, it is easier to extend the system, add more functionality, and reduce maintenance hassle by decoupling the modules.  5.3.3  Communication Design  In the last section, a totally federated architecture that supports remote components was proposed. We connected external displays and devices with the tabletop display using a WiFi router in the first iteration of implementation. Components still needed to search for and register with the central controller. This process, though effective, proves to be slow and sometimes troublesome. To satisfy the requirement of connecting to remote devices and connecting to other devices more easily, the “publisher-consumer” design is adopted for communications between components in the second iteration design. Simply speaking, the publisher-consumer design requires one component to publish data to the system over a “channel” and other components subscribing to this channel will then receive the updates automatically. For example, the tabletop pattern designer might publish the textual representation of the current pattern to the channel “/pattern/current” and all subscribers will automatically receive updates and react accordingly. The diagram of this design is shown in Figure 5.3. In this design, a central server is still needed, but we decided that this server is better on the Internet, so remote computers and devices in another city can also connect to our system. Because the publisher-consumer paradigm is a very classic and mature pattern for communication, there are existing systems and protocols for us to use. Such a design has been used in various chat room and instant messaging systems. In the next chapter, we describe the integration of a highly-scalable, high-  64  Subscriber: 3D Pattern Viewer  Subscribes: /pattern/current  Publisher: Table Pattern Designer  Channel: /pattern/current  Message Relay  Subscribes: /pattern/current  Subscriber: Other Subscribers  Subscribes: /pattern/current  Subscriber: Stats Viewer  Figure 5.3: Communication design diagram performance, open-source publisher-consumer communication framework as our communication engine.  5.3.4  Abstract Data Design  As mentioned in our requirements, it is planned to support grouping of cases and patterns. Our system supports a pattern which contains a list of cases and/or other patterns. A design task is a pattern, with geo-location and requirements data. To clarify, in our design and implementation, a case that has been placed onto a map or design task is called a “case instance”. A case instance contains a reference to the underlying case, and a heading and a relative position to its parent pattern. A diagram explaining the relationships between these data objects is shown in Figure 5.4.  65  +location Pattern  Design Task  is a contains is a many  Case Database  Object contains  contains is a  Case Set  contains  + relative pos Case refers to  Case Instance  Figure 5.4: Abstract data design diagram  5.3.5  UI and Interaction Design  After consulting with our users and based on the discussion about UI problems in Section 5.1, design guidelines and improvements were made for an improved user interface. • User interfaces can be retracted (like the Dock in the Macintosh OS/X) to make more room for design workspace. • User interfaces are replicated at each edge of the table to reduce interaction conflict and to solve text direction problems. • More smoothness in the interaction is achieved by adding inertia to objects on the table and also to lists when they are scrolled. • The case inventory / case bar needs to be the prominent UI element except for the central map, which should occupy as much screen real estate as possible. • Relocate some UI elements to extended displays, such as pattern statistics, to free space on the table. • Distinguish between ephemeral and persistent UI elements; some UI elements, such as saved patterns list, should be closed most of the time. 66  These design guidelines led us to the UI implementation described in the next chapter.  67  Chapter 6  The Second Iteration of the System Implementation This chapter discusses how the design in Chapter 5 was implemented. Challenges of the planned features include how to use Google Maps in our system, how to make a live connection with elementsdb, network and communication improvements and so on. The system was implemented according to the overall architectural design described in Section 5.3.2. Below are subsections that discuss how each module in the system was implemented and how we tackled these challenges.  6.1  Integrating Google Maps  First, the skeleton of the whole system and abstract interfaces of each module was built. Then we started to implement a real map provider using Google Maps. Each map provider, including the Google Maps control that was implemented, has a list of interfaces as shown in Table 6.1. The map provider itself should support pinch and dragging actions. There are quite a lot of difficulties when Google Maps provider is implemented. Below is a list of these problems and how they are resolved.  68  Property / Function Center Resolution ViewportSize PixelToLatLng LatLngToPixel  Notes Gets or sets center coordinates of the map Gets or sets average zoom level of the map Size in pixels of the map window Converts screen position to geo-location Converts geo-location to screen position  Table 6.1: Typical interfaces of a map control  6.1.1  Embedding Google Maps in a Program  Google Maps, like most other map applications, is designed for the Web. It is difficult to embed a Google Maps instance into our application on the tabletop. After some research, it was clear that it is possible to do this similar to how the 3D pattern viewer described in Section 4.3.2 was implemented. So the implementation is to embed a web browser in our tabletop application and display a web page that contains Google Maps in it. The Google Maps instance can be controlled by the Google Maps API [11], which is a JavaScript API that can be invoked from our tabletop program through the web browser control. Figure 6.1 is a screen shot of embedding Google Maps in our application. The screen shot also shows the re-designed user interface which is described later in this chapter. The center of the map and the coordinates of the four corners can be retrieved by invoking Google Maps API. These coordinates are required to calculate current center and resolution (meters per screen pixel) of the map. These data can be used by other modules like the UI module to display case instances at their correct screen position and in their correct size.  6.1.2  Linear Zooming in Google Maps  Google Maps, like other map systems, does not support linear zooming. It has discrete zoom levels and the API does not allow zooming between those levels. This problem is solved by zooming the map image in the program if the desired zoom level is in between two levels in Google Maps. A “zoom level” defines how big the area is in viewport and how much detail to display. The lowest zoom level, which is 1, displays the whole world in your window, while higher ones,  69  Figure 6.1: Table pattern designer screen shot such as zoom level 14 or higher, usually display city-level or street-level details. These zoom levels are different across map systems and have to be dealt with specifically on the map system targeted. As an example, if the user wants to zoom to zoom level 16.5 (relative to the lowest zoom level), then Google Maps should zoom to level 16 and then the system zooms the resulting map image so that it represents zoom level 16.5. As for how big the map image of zoom level 16 should be zoomed to represent a zoom level 16.5 image, the scale is calculated based on map resolutions. Map resolution is a number that represents screen pixels per physical meter. This number can be calculated by measuring longitude/latitude difference at the edges of the map and then converting the degree difference to meters. Below is the interpolation algorithm used to calculate a smooth zoom factor. Suppose at level X (where X ∈ N), map resolution (defined as screen pixel per meter) is RX , at zoom level X + 1, map resolution is RX+1 , the desired zoom level is X + p where 0 ≤ p < 1. So the scale factor applying to the image of zoom level X is calculated as  70  f=  RX + p(RX+1 − RX ) RX  To simplify calculation, only the resolution in the latitude direction is used. The reason is that meters per degree of latitude is a constant number regardless position on the Earth. But for longitude, meters for one degree of longitude changes with the latitude. At the far north or south, one degree on longitude measures much shorter than on the equator. Because Google Maps can only get latitude and longitude coordinates of the four corners, but not relative distance in meters between the edges, such a conversion has to be done on our own. Therefore, measuring distance only on latitude is the easiest way. However, a problem with only using latitude is that when the zoom level is very low (very big view area with few details), and at higher latitude regions, longitude resolution differs from latitude resolution a lot. This is because most map applications, including Google Maps, flatten the spherical map image by latitude and longitude grid lines, thus making pole regions look way bigger than they should. But this is not a very big issue because our design tasks all focus at very high zoom level (high detail with smaller region). At such levels, the sphere effect can be ignored. It is unlikely for anyone to do an urban design task for an entire city, much less an entire country. Figure 6.2 is a diagram illustrating this issue.  6.1.3  Smooth Zooming and Panning on Google Maps  Another issue in using Google Maps is that any calls to change Google Maps cannot reflect on the interface immediately. For example a zoom command will cause a “lens zooming” effect which lasts about 100 milliseconds. This effect is not needed because our linear zooming already makes the visualization smooth enough (Section 6.1.2). Also, when changing the center of Google Maps by invoking Google Maps API, it has a few milliseconds delay. This delay combined with the immediate zooming of the map image without needing to invoke Google Maps (when zooming is remained within one zoom level), causes a shaking effect when panning and zooming the map at the same time. To solve this issue, many methods were tested. A secondary Google Maps image is used in the background, working 71  Figure 6.2: Zooming issue in maps. This is a world map (lowest zoom level, lowest detail and largest region), with latitude and longitude lines shown. Most map systems will stretch the map along latitude and longitude lines as in this image. Therefore at low detail zoom levels, resolutions are different at different locations on the map, so Antarctica is considerably bigger (in east-west direction) compared to other continents than it should be. Such sphere effect is small when focusing on a smaller region like city level. Map originally from United Nations Cartographic Section [17]. as a buffer map. When users zoom and pan the map, the foreground map zooms and pans only using image zooming and translating, without invoking any Google Maps API. But without loading a new image at different zoom levels or different locations, this foreground map will eventually become too low in resolution or too small or even out of the viewport. So meanwhile, the background buffer map performs zooming and panning using the Google Maps API by invoking appropriate JavaScript functions. There would be shaky effects on the background map, but they are invisible to the users. The background buffer and the foreground map are switched every few hundred milliseconds, to keep the foreground map image always updated and with the most appropriate scale and resolution. Efforts were made to fine tune and improve the hand-off effects of background and foreground maps. In the end, manipulating Google Maps on the table is now quite smooth and visually well handled. 72  6.2  Connection with the Building Database  In our first iteration, no live connection with the building database (elementsdb) was made. In this iteration, the tabletop design application is connected to the database directly. A derivative of the case database provider which connects to the database and loads the cases was implemented.  6.2.1  The elementsdb API  On the elementsdb side, data is stored in an SQL database. But our system does not directly connect to the SQL database. Instead, the database provides web service APIs to list all cases and retrieve the details of each case. Results are returned in XML format. Figure 6.3 is an example of the database API.  Figure 6.3: Example of elementsdb API. Left: A case viewed in normal elementsdb web site. Right: The same case queried using the API. The returned content is an XML representation of the case details.  6.2.2  Loading the Database  When the tabletop application starts, these APIs are called to load cases into the table. Data from each case, such as floor area, site shape and so on are then loaded 73  and stored for the data objects on the table. Case sets are also loaded. Users on the table can select different case sets to display.  6.2.3  Additional Resources of the Cases  A problem with loading the database is that although the database contains a lot of resources associated with each case, such as 3D models and photos, it does not contain all the needed resources. For example, a top-down massing view of each case is required, which does not exist in the database. Also, our 3D viewer needs DAE format model files, which were manually converted in the first iteration development. To solve this issue, a small program were developed. The program can load a SketchUp model of a case and generate resources needed such as a top massing view, a DAE model, etc. To implement this, the easiest way to manipulate SketchUp files is by using the SketchUp Ruby API [14]. Simply put, SketchUp Ruby API is a Ruby-based scripting system. Programs written using the API can be loaded by SketchUp to perform necessary operations. Most common operations such as switching to top-down view, taking screen shots and exporting to another format can be done using the SketchUp Ruby API. Our program generates SketchUp Ruby API scripts and then executes SketchUp to load the script just generated. A script contains what files to load, what operations to perform and where output files are to be stored. In this way, efficient offline batch processing and resource generation can be done for all cases in the database. However, a problem with this offline batch processing approach is that when a new case is added in the database or a case has been changed, the batch converter has to be re-run in order for the generated resources to reflect the changes. In addition, some resources such as those mentioned above, some data not present in elementsdb are also generated, such as the bounding box of cases, especially for irregular cases. Models in SkecthUp files have their physical size in meters so this information can be used to calculate how big each case should be when placed on a map.  74  6.3  Universal Touch Framework  The tabletop application, as mentioned in earlier chapters, was originally built based on touch APIs for a SMART Table. A big improvement in the second iteration of development was to explore the possibility of making it run on other multitouch tables. Commercially available tabletop displays include the SMART Table, Microsoft Surface, DiamondTouch from Mitsubishi, and many others. There are also extensive tutorials on how to build a low-cost touch table by yourself on some community websites such as NUIGroup.com [5]. Open source drivers are developed by the community to support custom-built tables. So we think that making the tabletop application able to run on various tables will be beneficial to our research. Also, the system is to be deployed in CIRS, but the people working on the decision theatre are still deciding which table to purchase. It is not likely that they will buy something completely based on our choice, so it is better to make the software “cross-table” enabled. After some research, an open-source touch framework called BreezeWPF was eventually selected [1]. The framework can support various tables and provide a unified interface for working with touches. The framework has three layers including a driver layer, touch processing and control containers. The structure of the touch framework is shown in Figure 6.4. The driver layer is responsible for converting table-dependent touches to touches for the framework so that the framework can process them. The framework already comes with the support for Tangible User Interface Objects (TUIO) protocol [16], which is used widely in open-source touch recognition and custom-built touch tables. A SMART Table driver was built by us to support the SMART Table. It has few differences from native APIs. It is very straightforward to develop a driver for the framework. The only thing that needs to be done is to respond to events such as touch down and touch up of the native API that comes with the table you are working on, and then invoke proper methods in the framework to signal to the framework that a touch down or touch up event has occurred. Later more support for other tables can be added, such as for Microsoft Surface. The touch processor receives touch events from the driver layer and maintains a list of active, captured touches. Data from each touch is also calculated, such as  75  Touch Containers responsive to touch messages  Button  Image  Canvas  List  …  Windows Touch Driver  …  Touch Processor  SMART Table Driver  Microsoft Surface Driver  TUIO Driver  Figure 6.4: Touch framework structure speed, acceleration and so on. Control containers are defined in the framework. They are some special controls which ordinary controls such as buttons or lists can be put into. The touch processor looks for and retrieves the control container that is directly under each touch and all actions caused by this touch, such as dragging, zooming, or rotating, will then all target to the control in this control container. BreezeWPF is open source, which means that any part of it can be modified. It is open source under the “GNU Less General Public License” [7], which requires the program that uses this library to release the source code of this library only, along with its direct derivative work. Other parts of the program do not need to be open source.  6.4  Network and Communication  In our first iteration, if some component in the system wished to interact with another component, it had to know the location of that component. This was to be 76  done by asking a central controller, which knew the location of every component in the system. In the second iteration, a complete federated architecture is adopted. As described in the Chapter 5, a “publisher-consumer” model is used. In this model, all components in the system are of equal status, forming a federated network. There is also no central controller or central directory in the network. Each component has its own, always-updated map of the whole system. Every component in the system is both a publisher and a consumer. Each component publishes its updates through some channels. A channel is a topic of interest that can be subscribed to by other components. A consumer can then listen to a channel and react to any updates that have been published. In this model, a server that relays messages is needed. There are a few existing open source systems that support high performance and highly scalable publisher-consumer messaging. A system called RabbitMQ is used in our system [12]. RabbitMQ is an open source project maintained by vmware. The server can be run on a number of platforms including Linux, Windows and Mac systems. RabbitMQ is highly scalable, highly available and has very high performance. Because the message server is on the Internet, it naturally supports remote components connecting to the system. A potential problem is that the performance capability of the central messaging server is limited, but RabbitMQ does support easy scaling by adding more servers (nodes) to offload its workload, and this can be done very easily. In order for the components to know the map of the whole system, a special channel is created to let components broadcast heart-beats to other components. Every component in the system subscribes to this channel so every component will have a local copy of the map of the whole system. In this totally federated way, all components in the system are decoupled from other components. Every component can subscribe to information that is only of its own interest. Such a federated architecture has a number of advantages over a centralized model: high extensibility, ease of maintenance, and separation of the components in the system. Figure 6.5 is a screen shot of the system diagnostics tool that is also a compo77  nent in the system. It shows the network map of the current system. We can see in the screen shot that two 3D pattern viewers, one statistics viewer and one table pattern designer are running in the system. We can also get their self-reported geo-location.  Figure 6.5: Network diagnostics tool  6.5  Other Improvements  Other improvements to the first iteration include UI changes and adding more components and functions to the system.  6.5.1  UI Improvements  As discussed in the last few chapters, one problem with the standalone system is the screen real-estate problem. In this new version, an effort is made to minimize non-essential user interface elements. Another problem with a collaborative and 78  shared interface is control conflicts. So in the second iteration of implementation, our design discussed in the last chapter was implemented. Sidebars become the main UI and now occupy the four edges of the screen. They are retractable, which can release screen space to the center map as much as possible. A case set chooser is also added as described in our design. On the right of the case bar, a few buttons are added. By touching them, users can activate some temporary widgets, such as options, saved patterns list, etc. A screen shot of this is shown in Figure 6.1. In the previous implementation, the case bar displayed cases in their relative original sizes. That is, the size of the cases in the sidebar reflected their actual size relative to each other. However, a problem with this is that the biggest and smallest case in the same case set can differ by 10 or 100 times in terms of size. If relative size between cases is kept, then in order to make the smallest case visible and usable, the big cases have to be extremely big. Our solution is to scale all cases to have the same height (vertical length) on screen and keep their aspect ratio so their width (horizontal length) can vary. In this way, their size on screen does not reflect their actual relative size, so in order to visualize the size of each case, a small scale is drawn at the bottom of the case. The scale indicates 10 meters. So the longer the scale is on the case, the smaller the case is. Figure 6.6 shows a comparison of the case bars. Another minor tweak of the case bar was adding progressive loading of the case images. It appeared that if all images of the cases are loaded immediately when the UI is rendered, it causes the UI to freeze for a very long time. Therefore the images are only loaded when a case becomes visible (scrolls into view) in the case bar.  6.5.2  Inertia  An important feature of direct touch interaction is that it can simulate physical responses of objects. Therefore most modern touch interfaces support such characteristics as inertia. We added inertia support in our new implementation. Inertia is supported when scrolling through a list of available cases, when dragging the map and also when dragging cases onto the map. A small problem with adding inertia on our own when dragging or “throwing”  79  Figure 6.6: Comparison of the case bars. Both screen shots are taken at the same resolution. We can see the old (top) case bar sacrifices a lot of space when displaying very big cases, while the new (bottom) case bar saves space as much as possible by making all cases unified height and displaying a scale at the bottom-left corner.  80  things on the map is that touches tend to be a little bit shaky sometimes because they are not very stable on the table. Sometimes, when you want to align a case to somewhere, and lift off your finger, there might be a small velocity that causes the case to slide a little bit on the map. To solve this issue, the inertia is only active when velocity is greater than a threshold. A better solution might be add constraints to aid alignment.  6.5.3  Statistics Viewer  In our second iteration, a statistics viewer was created to display the current statistical summary of the design task. This component was designed in an effort to move the statistics off the table and free up more screen space for the map. The statistics viewer displays some charts showing facts about the current design task, such as land use ratio. elementslab plans to add a more sophisticated statistics analyzer that takes in a list of cases and calculates some complex figures that can help evaluate the sustainability and quality of the pattern. Our statistics viewer will later use that calculator to compute statistics in the future. The charts in the statistics viewer (shown in Figure 6.7) are generated by the Google Image Charts API. To generate high-quality charts, a series of methods were investigated, including some existing Windows controls that could draw charts. But most of them either rendered too simple an image or were too complex to use. So we ended up using the Google Image Charts service, which is a free HTTP service that allows you to embed all parameters such as chart type, data, titles, colors, etc in the URL to the service. The HTTP response is then a static image of the chart. The API is extremely lightweight, easy to use and renders beautiful charts. All you need is the ability to connect to the Internet. Figure 6.7 is an example of Google Image Charts. We now have our improved, second iteration application up and running. In the next chapter, we summarize and discuss a series of HCI design issues that we encountered during the design and implementation of the final system. Any one of these issues can be expanded and studied later as an independent topic to improve our urban planning system.  81  Figure 6.7: Example of Google Image Charts. The above is the URL that returns the image below. Meaning of each parameter is shown on the left.  82  Chapter 7  Discussions and Future Work Throughout both phases of the project, many design and HCI issues emerged. These issues are discussed in this chapter. Future research directions and possible research topics are also discussed. Discussions are divided into topics including multi-display systems, tabletop application UI design, and system effectiveness evaluation. Strengths and limitations of our current system are also discussed in this chapter.  7.1  Collaborative Multi-Display System  The most significant feature of our system is that it is a collaborative, multi-display system for urban planning. There are a lot of considerations regarding the development of such multi-display systems.  7.1.1  Appropriate Location of Information  The first question is: what content should be presented on which kind of display in the system? In our implementation, 2D designing of a neighborhood is placed on the tabletop display and 3D views of the neighborhood and some statistics are projected onto the wall. However, from some brief user feedback, some users thought that the 3D rendering might better be co-located with the pattern designer on the table. But there is also some prior work that shows large screens are suitable for 3D navigation tasks [40], which implies this particular issue needs more study  83  for our urban planning tool. To make things more complicated, the possibility of including handheld devices in the system also adds some uncertainty about the number and diversity of display locations for the system. Intuitively, information of personal interest should be shown on a handheld device. However, 3D renderings of the pattern might also be appropriately displayed on the handheld devices, if a person wishes to view the pattern individually. It depends on the use case what information should be displayed on which kind of display. Handheld devices might also be more appropriate to control the contents on some non-interactive displays, such as the wall displays. Placing information on an inappropriate display may cause lowered productivity and make the system less useful. In future research, some formal studies are expected to investigate the most appropriate display for a certain piece of information in a certain use case.  7.1.2  Information Sharing and Coordination among Displays  Besides finding the appropriate place for different information, how to coordinate and synchronize it is also an issue. For example, in our current system, a widget is used on the table to control the view angle of the 3D pattern shown on the wall display. While this is effective, some users do expect the tabletop interface and the 3D view to be “synchronized”, which means that when you drag or zoom the map on the table, the 3D view up on the wall should also change to reflect the viewport change on the table. This issue is similar to the one described in [19] which compares 3D navigation between tabletop and wall displays using either a movable camera object on the table or a fixed camera with a movable canvas. A related issue is what information should be synchronized and shared across displays. In our current implementation, only the neighborhood pattern designed on the table is synchronized. But one might expect that their viewports wouldd also be synchronized and shared. With the view angles not synchronized, some users do report they become disoriented after their attention shifts back and forth between the wall display and the tabletop display. If more projectors are added to provide a more cave-like immersive display, should they be synchronized so they form a big display or should each of them  84  be able to show independent information or view angle? One might argue that it depends on different use cases. This problem also requires formal future study. It is possible that for our multidisplay urban planning system, maintaining synchronized view angles between the 2D pattern on the tabletop display and the 3D rendering up on the wall might be preferred by the users. Another possible study might be what information should be shared among displays in our urban planning system. And is it the same for other types of applications such as a multi-display teaching environment?  7.1.3  Physical Locations of the Displays and Display Identities  Most of the time in our system, we are dealing with one or two extra projectors connecting to the system. However, an interesting issue about connecting more of these projectors to the system is the physical location of the projectors. As discussed earlier in this thesis, the system would need to know the location of the projectors if one of the collaborators on the table wishes to display something up on the wall directly facing him or her. The system might also need to know the location of each projector if the user wants to have the ability to visualize information in a cave-like display that is composed of many displays. Another topic related to this is how to identify displays. For example, if a user wants to show some statistics up on the wall, he or she might expect to have some way to choose which projector he or she is going to use. So each display should at least have some identity associated with it. Physical location is one kind of identity. One solution is that the tabletop display can display a diagram of the whole room, including all the connected projectors and where they are. The user only needs to tap on the desired projector to choose where he or she wants to display such statistics. An alternative solution is to assign a number or a color to each of the displays, so when a user chooses a target display, all the affected displays can highlight their identities (a color or a number). The user could then see which one is where and choose the desired color or number accordingly. On the other hand, sometimes one might expect to need to find where some information is currently being displayed. For example, when doing a neighborhood  85  design task, a user might choose to display the 3D rendering of a saved pattern (design alternative) on one of the projectors. But later, as the process progresses, he or she might revisit the saved pattern, so the user needs to find out which projector is displaying it. It is true that some textual name can be displayed on the projector saying for example “Design Alternative #1”, but still it would be best to tell the user which projector is the target projector in the easiest way possible on the table. One way might be to have on each saved pattern icon a small diagram of the whole room with the projector on which this pattern is shown highlighted. This requires the system to know the physical location of each display. Another way, as mentioned above, might be to use color or numeric identities, then users have to search and match the display on their own. A third way might be to provide some visual feedback on the projector. For example in the above scenario, the target projector can “jiggle” up and down a bit to draw the user’s attention. To summarize, display identities and possibly their physical locations could be very important future directions for our multi-display system. We imagine that some sort of a projector server might be involved, which knows the location of each projector when the room is configured. This component might also provide a diagram of the room and ways to control different projectors and move components and programs across projectors.  7.1.4  Shared Control versus Exclusive Control  Another interesting issue with a multi-display and collaborative system is controlling the displays. In a collaborative system, usually the user interface is shared among collaborators, especially on a co-located system such as ours. But an issue is: when one user is controlling a projector or an extra display from the tabletop display, should that user have exclusive rights to control it? In other words, can other people choose to control the same display before that user releases it? As described in our implementation, our system currently allows multiple people to share one projector, and the projector always reflects the latest command. This problem is more difficult considering it is actually hard to determine when a user actually releases the control of a display. For example, when a user chooses to use a projector to show the 3D rendering of the current plan in one view angle,  86  it is actually hard to tell when he or she is done using that projector. He or she might want to view the pattern from a certain angle, but the system cannot know when the user has finished using it. If users are not allowed to share displays, then other users might have to wait and ask the first user if he or she has done with the projector verbally. It might reduce productivity and sometimes causes confusion. Therefore another future research direction is how to share multiple displays between many collaborators. It is possible that different use cases might have different requirements such that some allow sharing the displays while others do not. Also, another research issue is how to determine users’ intentions so that on a “non-sharable” display, the display can be released as soon as a user finishes using it.  7.1.5  Other Kinds of Displays  As mentioned earlier, our system is intended to support handheld devices or tablets such as iPads. There are a whole range of topics that can be investigated, including tablets or handheld devices in our system. For example, one issue is determining the best role for such handheld devices. Handheld devices are convenient for displaying contents of the user’s personal interest. Another issue is that handheld devices can be potential ways to control some of the displays in the system, such as the wall displays. There is some research on how to use handheld devices to control big screens [28], but in our particular application, the urban planning system, there is still work to be done. Also, handheld devices have a few extra input options compared to other displays. How to utilize them can lead to interesting topics. For example most handheld devices have a camera on the back, making it possible to use augmented reality technologies to display more information based on one or more other displays’ contents.  7.1.6  User Identification  The last issue about a collaborative multi-display system is the identification of users. This issue is critical if multiple displays are introduced into a system used by multiple users. The system needs to know, in some way, how to pair users with displays.  87  For example, when a user is working on the tabletop system, and he or she wants to show something on the handheld device on his or her hand, the system needs to know who is using the tabletop and who is using each of the handheld devices. A similar scenario discussed in an earlier chapter is when a user wants to put up something on the wall: the system should be designed so that the information is shown on a wall that is facing the user, by default. In this scenario, the system also needs to know which user is issuing the command and which side of the table he or she is most probably on. An existing tabletop display with such capability is DiamondTouch. It uses electronic signals to identify who is touching the table. The DiamondTouch system requires a special conductive pad on the seat of each user. The pad is connected to the tabletop display so when a user is touching the table, the current can form a closed loop and the table can then tell who (or which pad) is touching the table. Another approach can be a color-coding solution. When a user starts to use the tabletop computer, he or she is assigned a color. Any actions happening on this side of the table will be assumed to be initiated by this color (person). Then the user can connect his or her personal device to this color. This idea is far from complete but it leads to a direction of user identification through color-coding and some additional operations (like manually connecting with a handheld device), when direct user recognition is not possible. In summary, how to identify users around the table, how to use such identities to improve interactions between displays and how to design a system that supports the identification of multiple users and multiple displays are definitely very big future research directions.  7.2  User Interface of the Tabletop Pattern Creator  Issues discussed in this chapter are centered on user interface design issues of tabletop urban planning tools. Tabletop displays are collaborative workspaces for a group of people; there are also some issues with urban planning systems in particular.  88  7.2.1  Automatic Object Alignment  An important issue of the tabletop urban planning system is aligning the cases. This issue is also extensively discussed in the thesis of Fernquist [21]. Because users interact with tabletop displays directly using their fingers, the “fat finger” problem is very serious in our urban planning system because in such systems, users usually expect to have precise alignment of the objects. In practice, on a tabletop display, it is rather hard to align cases on the map. Users usually find it difficult to align cases with the edge of streets or other cases. Direct interaction using the fingers without any additional handles makes such tasks extremely cumbersome. There are some techniques proposed by researchers to solve the fat finger issue. For example Wigdor et al. [45] describe a technique to use one hand as a guideline or a handle and all objects close to it will align with it. For our system in particular, a good solution might be to define street fronts in a design task and align a case to street edges when it comes close to it. Edges of streets can be defined as an array of line segments on the map. Each segment can be defined by the physical coordinates of its two ends. Then whenever a user moves a case, if a case is placed very close to an edge, it will align with the edge. But there are also more issues to consider. For example, a lot of land parcels are not rectangular, and thus neither are the street edges. What if a rectangle case is placed on one corner of such lot? How to define alignment in such a situation is a more complex problem. Therefore a possible future research topic is how objects can be aligned in planning support systems on tabletop displays. This topic is more complicated in nature than aligning arbitrary objects on any generic tabletop display. Fernquist et al. [22] explain how to implement “snapping” to grid lines or to objects for a direct touch display, where there are some issues that do not arise on displays with mouse-based placement. This technique could be adapted to support a variety of alignment rules.  89  7.2.2  Case Size  Another issue discussed in earlier chapters is whether the relative size of the cases should be kept on the case list bar. There are two different solutions in the two iterations of our system. In the first iteration, Fernquist [21]’s implementation was kept. It displays cases by preserving their relative size. But later we discovered that the biggest and smallest cases are of more than ten times difference. So in order to make the smallest case readable, the biggest one was very big, taking too much screen real estate. So another approach was implemented that rendered the cases with a predictable size. The relative sizes of the cases were lost but a scale was added on the corner of each case, to indicate its actual size. A research direction would be to investigate what is better when presenting objects with physical sizes on a tabletop display with restricted screen space. Another alternative to the above solutions is a mixed solution, which displays cases with uniform size (or uniform height and various width), but when one is selected or scrolled into the center, a magnified image showing its actual size would pop up. These designs might have different advantages and problems that would need to be studied further.  7.2.3  Control Conflicts  One issue when multiple users are using the system at the same time is that there exist some control conflicts, just as they exist in most other co-located collaborative systems. For example, one particular issue is that when multiple people are using the table, one might be placing a case onto the map while another person might be dragging the map or zooming it. The first user’s action might be interrupted. A solution could be to lock the map when other people are using it. There are a lot of these issues for a co-located collaborative system. These issues might need to be revisited in future studies.  7.2.4  Widgets versus Gestures  Currently a popup widget is used to control additional projectors. There might be other solutions to this. One possible solution is to use gestures. 90  Using gestures to control additional displays does not introduce any more user interface elements that occupy the screen real estate. For example, a double-tap on a point on the table might mean a fly-to action to a place on a connected projector. The 3D pattern view would fly to the location being tapped and provide a bird’s eye view. Another possibility is using two fingers to represent the location of the camera and a third finger (within proximity to the other two fingers) to control the view direction. In this way, the view angle on the external 3D pattern viewer can be controlled without any additional user interface elements. Gestures are important interaction techniques in touch-based systems, but our system does not make extensive use of them, nor have we done much study on them. So a promising future step is to study how gestures could be used on a tabletop urban planning system and how they can be used to control the system and other displays.  7.2.5  More Ways to Interact with the Table  Besides gestures and on-screen touch-sensitive user interface elements, as mentioned in the last section, additional devices such as a cell phone can also be used to control the system without adding more UI elements. A future step is to consider using physical objects on the table. Some tables can recognize object shape or barcode placed on the table. If these “smart objects” can be used as part of a natural user interface, it would open more possibilities and ways to interact with the system. For example, physical building models might be used to place buildings on the map. Such physical objects can be more naturally interacted with. Another example is that a camera object can be placed on the table to control the view angle of a 3D pattern on the wall. Adding these objects and devices to the interaction paradigm on the table will definitely bring interesting research topics about how to use them effectively and conveniently.  7.2.6  Angle of Bird’s Eye View  On the 3D pattern viewer side, one particular issue with the bird’s eye view is that it is not so easy to control the angle of the bird’s eye view. For now the angle is  91  fixed at 30 degrees. But that is not always what the users want. In most Geographic Information Systems (GIS), to control the position of a camera, there are six degrees of freedom. The x, y and z coordinates of the camera, a heading that indicates the horizontal direction to look at, a tilt indicating the camera’s angle relative to the ground and a roll factor indicating the degrees of rolling relative to the camera’s own axis. In our bird’s eye’s view, on the tabletop display’s 2D interface, the x and y positions of the camera can be easily defined. The target can also be defined so the heading can be calculated. Rolling is presumably always zero because most of the time the sky is supposed to be in the up direction. Tilt could be calculated if the z coordinate of the camera is known, because we can assume that the camera is always looking at a point on the ground. This means that only the z coordinate (altitude) of a camera cannot be determined on the tabletop by the current technique. In some existing GIS applications, for example Google Earth, an automatic way is used to determine the tilt or altitude of a camera. When the camera is far from the view target, the camera’s altitude is high and is viewing the area from a very high angle. As the camera moves lower and towards the target area, both its angle to the ground and altitude get smaller and eventually the camera enters street level view. For our system, some more study needs to be done regarding how to control a 3D scene more easily from the tabletop. Also research needs to be done to study if these automatic view angle control ideas are useful and effective in an urban planning support system.  7.2.7  Semantics of Different Types of Views  Another issue related to the camera as discussed in earlier chapters is the semantics of the control widgets in different modes. In a bird’s eye view mode, the widget is designed so that it can simulate a bird fly around a central point of interest. But in street view mode, the widget is designed so that it simulates a person standing somewhere and looking around. Figure 7.1 shows an example of the difference. Whether these semantics are suitable for the use cases needs further study.  92  Figure 7.1: Semantics of views. Left: Bird’s eye view is looking into the center region from a camera around it. Right: Street view is looking from the center region to the directions around it.  7.3  Connection of Components  Displays and components in the system are connected through a network in a federated fashion. Although initial tests show that this is effective, there are still some issues that require further study.  7.3.1  Connecting to Remote Components  While our federated infrastructure supports distributed components to connect to the system and for connecting remote components to the system, we have not had a chance to test the remote connection functionality in our research. Besides latency issues that might influence a distributed collaborative system, how to increase the sense of presence for the remote peer in our system is also worth studying. The current system is designed to be used in one room, in which users could see all the connected displays as well as other users. If one or more of these displays are separated and placed in another room or even another city, the “sense of presence” 93  (the degree to which remote users feel as if they are part of the local group, and similarly the degree to which the local group feels the remote users are present) become important. We will be connecting with the researchers at the University of Waterloo in the hope of setting up a remote lab using their tabletop displays and then connecting them together over the Internet. It will then be possible for us to study our urban planning system in a physically distributed environment.  7.3.2  Quality of the Architecture  No verification has been made regarding how much data exchange the network connection infrastructure of the system can support. An obvious question is: can such a network support a 30-frames-per-second continuous data stream to render smooth animation? What if the pattern on the table grows too big and needs a long time to transfer?  7.4  Usefulness and Effectiveness Evaluation  One of the most important issues for a future research project would be to assess how useful our system is when it comes to helping planners to design and evaluate urban plans. This is, after all, the whole point of implementing such a system. The current research, and the prior work by Fernquist [21] was predicated on the assumption that the original standalone system and its new version that has been described in this thesis does exactly that. But this needs to be empirically verified. Usefulness or effectiveness can be evaluated in the following aspects: Effectiveness of the extra displays. Are the users actually using the large screens or handheld devices if we add them to the system? How well do they perform with and without a certain kind of display? Effectiveness of the entire multi-display system. Are the users making fewer errors? Do they complete tasks faster? Do they feel an increased sense of accomplishment? Effectiveness of multiple people using the system. Does having multiple displays affect collaborative design? Is group collaboration really improved by using shared, multiple displays? 94  Because the system can be configured to be either a design tool or a public presentation tool, another evaluation step is to ask which task the system supports best. And if support does differ, how could we make it better support the other task(s)?  95  Chapter 8  Conclusions In this research, a multi-display collaborative neighborhood planning system was implemented. Our research is based on the system by Fernquist [21], which is a single-display, interactive, collaborative system for urban planning. Multiple additional displays and components were connected into the system through a federated, distributed architecture. Live connection with a building database (elementsdb) was also added.  8.1  Conclusions  The neighborhood planning system is centered on a multi-touch tabletop computer. Two iterations of the urban planning tool were implemented on a SMART Table. Using the system, a small group of users can collaboratively design a new neighborhood design on the tabletop display. The additional large displays are connected to the central tabletop display computer. These large screens show either 3D rendering of the current neighborhood on the table, or overall statistics of the designed neighborhood. Controlling the large screens is done through the touch table. The system is designed in such a way that other types of displays, such as handheld devices or multiple tabletop displays, can be connected to the system after fairly simple extensions to the current implementation. The implementation of the system was done in iterations, with a brief user  96  feedback gathering step in between. One limitation of the research is that there has not been very much user study activity in this research. The trade off was that the research could either tackle a smaller problem with more extensive user studies or concentrate on a bigger problem but do much less in the way of user studies. After considering our elementslab collaborators’ needs and the fact that we wanted to deploy a better system in the Centre for Interactive Research on Sustainability (CIRS) to gain a wider and more extensive set of users, we decided to focus more on implementing a better system for our long-term goal while delaying some of the user studies until we had a more complete system with which to conduct more extensive studies. The most important contribution of this research is the development of a multidisplay urban planning support system. Though not much user study was done, the system can be a foundation for future research on multi-display planning support systems. Another major contribution is that through analyzing user feedback, a list of issues and possible research directions has been identified. These informal discussions have shed light on a number future research topics that could extend this project. A third contribution is the integration of the technologies we used to connect the displays in the system. A federated, network-based architecture was developed that allows displays to connect with each other, even in remote locations. This approach proved to be effective and easy to install.  8.2  Future Work  Most of the anticipated future work has been summarized in Chapter 7. The most desirable and important future endeavor is doing user studies on the HCI issues identified in Chapter 7. Another future direction is connecting the system with more diverse displays, including handheld devices or another tabletop display. For distributed collaborative systems, a future direction is to set up a remote lab in another city and connect remote displays with our local system. More types of visualizations could be developed and presented on additional displays, such as details of the cases used in a pattern. By integrating more displays into the system, we would continue to explore the utility of multi-display systems, which is a research area of growing interest as the cost of displays comes down and the capabilities of  97  displays go up.  98  Bibliography [1] Breeze Multi-touch Framework. URL http://code.google.com/p/breezemultitouch/. → pages 75  [2] COLLADA. URL http://en.wikipedia.org/wiki/COLLADA. → pages 47 [3] elementsdb. URL http://elementsdb.sala.ubc.ca. → pages 22, 23 [4] Federated Architecture on Wikipedia. URL http://en.wikipedia.org/wiki/Federated Architecture. → pages 6  [5] Frustrated Total Internal Reflection (FTIR). URL http://wiki.nuigroup.com/FTIR. → pages 34, 75 [6] Google Earth Plugin. URL http://www.google.com/earth/explore/products/plugin.html. → pages 46  [7] GNU Lesser General Public License. URL http://www.gnu.org/licenses/lgpl.html. → pages 76 [8] Google Maps. URL http://maps.google.com/. → pages 60 [9] HTTP/1.1 Status Code Definitions. URL http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html. → pages 49  [10] Keyhole Markup Language on Wikipedia. URL http://en.wikipedia.org/wiki/Keyhole Markup Language. → pages 46  [11] Google Maps JavaScript API. URL http://code.google.com/apis/maps/documentation/javascript/. → pages 69  [12] RabbitMQ by vmware. URL http://www.rabbitmq.com/. → pages 77 [13] RESTful Web Services on Wikipedia. URL http://en.wikipedia.org/wiki/Representational State Transfer. → pages 42  99  [14] Google SketchUp Ruby API. URL http://code.google.com/apis/sketchup/. → pages 74 [15] SMART Table interactive learning center. URL http://smarttech.com/table. → pages 4, 34 [16] TUIO - NUI Group Community Wiki. URL http://wiki.nuigroup.com/TUIO. → pages 75 [17] The United Nations Cartographic Section blank map of the world with country names. URL http://www.un.org/Depts/Cartographic/map/profile/world.pdf. → pages 72 [18] R. Ajaj, C. Jacquemin, and F. Vernier. RVDT: a design space for multiple input devices, multipleviews and multiple display surfaces combination. In Proceedings of the 2009 international conference on Multimodal interfaces, ICMI-MLMI ’09, pages 269–276, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-772-1. → pages 13 [19] R. Ajaj, F. Vernier, and C. Jacquemin. Navigation modes for combined table/screen 3D scene rendering. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, ITS ’09, pages 141–148, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-733-2. → pages 14, 50, 84 [20] E. Ben-Joseph, H. Ishii, J. Underkoffler, B. Piper, and L. Yeung. Urban simulation and the luminous planning table bridging the gap between the digital and the tangible. → pages 15, 16 [21] J. Fernquist. A collaborative planning support system for a multi-touch tabletop. Master’s thesis, The University of British Columbia, 2010. → pages 2, 3, 7, 9, 15, 16, 17, 18, 19, 20, 24, 25, 34, 89, 90, 94, 96 [22] J. Fernquist, G. Shoemaker, and K. S. Booth. “oh snap” – helping users align digital objects on touch interfaces. In P. Campos, N. Graham, J. Jorge, N. Nunes, P. Palanque, and M. Winckler, editors, Human-Computer Interaction – INTERACT 2011, volume 6948 of Lecture Notes in Computer Science, pages 338–355. Springer, 2011. ISBN 978-3-642-23764-5. → pages 89 [23] J. Halatsch, A. Kunze, and G. Schmitt. Value lab: a collaborative environment for the planning of future cities. In Computation: The New Realm of Architectural Design, 27th eCAADe Conference Proceedings, pages 279–286, 2009. → pages 16 100  [24] J. Y. Han. Low-cost multi-touch sensing through frustrated total internal reflection. In Proceedings of the 18th annual ACM symposium on User interface software and technology, UIST ’05, pages 115–118, New York, NY, USA, 2005. ACM. ISBN 1-59593-271-2. → pages 34 [25] B. Hartmann, M. R. Morris, H. Benko, and A. D. Wilson. Augmenting interactive tables with mice & keyboards. In Proceedings of the 22nd annual ACM symposium on User interface software and technology, UIST ’09, pages 149–152, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-745-5. → pages 11 [26] K. H.D. A computer-aided strategic approach to decision-making in urban planning: an exploratory case study in thailand. Cities, 15(2):105–119, 1998. → pages 20 [27] C.-H. Hsiao, L.-W. Chan, T.-T. Hu, M.-C. Chen, J. Hsu, and Y.-P. Hung. To move or not to move: a comparison between steerable versus fixed focus region paradigms in multi-resolution tabletop display systems. In Proceedings of the 27th international conference on Human factors in computing systems, CHI ’09, pages 153–162, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-246-7. → pages 5 [28] W. Hutama, P. Song, C.-W. Fu, and W. B. Goh. Distinguishing multiple smart-phone interactions on a multi-touch wall display using tilt correlation. In Proceedings of the 2011 annual conference on Human factors in computing systems, CHI ’11, pages 3315–3318, New York, NY, USA, 2011. ACM. ISBN 978-1-4503-0228-9. → pages 12, 87 [29] H. Ishii, J. Underkoffler, D. Chak, and B. Piper. Augmented urban planning workbench: Overlaying drawings, physical models and digital simulation. pages 203–211, 2002. → pages 15, 16, 27 [30] M. Kelaskar, V. Matossian, P. Mehra, D. Paul, and M. Parashar. A study of discovery mechanisms for peer-to-peer applications. In Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGRID ’02, pages 444–, Washington, DC, USA, 2002. IEEE Computer Society. ISBN 0-7695-1582-7. → pages 42 [31] A. Khan, G. Fitzmaurice, D. Almeida, N. Burtnyk, and G. Kurtenbach. A remote control interface for large displays. In Proceedings of the 17th annual ACM symposium on User interface software and technology, UIST ’04, pages 127–136, New York, NY, USA, 2004. ACM. ISBN 1-58113-957-8. → pages 12 101  [32] R. E. Klosterman. Planning support systems: A new perspective on computer-aided planning. Journal of Planning Education and Research, 17 (1):45–54, Fall 1997. → pages 15 [33] N. Krahnstoever, S. Kettebekov, M. Yeasin, and R. Sharma. A real-time framework for natural multimodal interaction with large screen displays. In Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, ICMI ’02, pages 349–, Washington, DC, USA, 2002. IEEE Computer Society. ISBN 0-7695-1834-6. → pages 12 [34] R. Kruger, S. Carpendale, S. D. Scott, and S. Greenberg. How people use orientation on tables: comprehension, coordination and communication. In Proceedings of the 2003 international ACM SIGGROUP conference on Supporting group work, GROUP ’03, pages 369–378, New York, NY, USA, 2003. ACM. ISBN 1-58113-693-5. → pages 11, 21 [35] H. Lee, H. Jeong, J. Lee, K.-W. Yeom, and J.-H. Park. Gesture-based interface for connection and control of multi-device in a tabletop display environment. In Proceedings of the 13th International Conference on Human-Computer Interaction. Part II: Novel Interaction Methods and Techniques, pages 216–225, Berlin, Heidelberg, 2009. Springer-Verlag. ISBN 978-3-642-02576-1. → pages 14, 15 [36] P. Marshall, E. Hornecker, R. Morris, N. Sheep Dalton, and Y. Rogers. When the fingers do the talking: A study of group participation with varying constraints to a tabletop interface. In Horizontal Interactive Human Computer Systems, 2008. TABLETOP 2008. 3rd IEEE International Workshop on, pages 33 –40, oct. 2008. → pages 5 [37] M. R. Morris, A. Paepcke, T. Winograd, and J. Stamberger. Team-Tag: exploring centralized versus replicated controls for co-located tabletop groupware. In Proceedings of the SIGCHI conference on Human Factors in computing systems, CHI ’06, pages 1273–1282, New York, NY, USA, 2006. ACM. ISBN 1-59593-372-7. → pages 11 [38] S. D. Scott, K. D. Grant, and R. L. Mandryk. System guidelines for co-located, collaborative work on a tabletop display. In Proc. ECSCW 2003, pages 159–178, 2003. → pages 4, 10 [39] G. Shoemaker, A. Tang, and K. S. Booth. Shadow reaching: a new perspective on interaction for large displays. In Proceedings of the 20th annual ACM symposium on User interface software and technology, UIST 102  ’07, pages 53–56, New York, NY, USA, 2007. ACM. ISBN 978-1-59593-679-0. → pages 5, 12 [40] D. S. Tan, D. Gergle, P. Scupelli, and R. Pausch. Physically large displays improve performance on spatial tasks. ACM Trans. Comput.-Hum. Interact., 13:71–99, March 2006. ISSN 1073-0516. → pages 12, 83 [41] F. Tyndiuk, G. Thomas, V. Lespinet-Najib, and C. Schlick. Cognitive comparison of 3D interaction in front of large vs. small displays. In Proceedings of the ACM symposium on Virtual reality software and technology, VRST ’05, pages 117–123, New York, NY, USA, 2005. ACM. ISBN 1-59593-098-1. → pages 12 [42] I. Wagner, M. Basile, L. Ehrenstrasser, V. Maquil, J. Terrin, and M. Wagner. Supporting the formation of communities of practice: Urban planning in the mr-tent. In Proceedings of ALPIS. Sprouts, 2009. → pages 16 [43] D. Wigdor, C. Shen, C. Forlines, and R. Balakrishnan. Table-centric interactive spaces for real-time collaboration. In Proceedings of the working conference on Advanced visual interfaces, AVI ’06, pages 103–107, New York, NY, USA, 2006. ACM. ISBN 1-59593-353-0. → pages 13 [44] D. Wigdor, H. Jiang, C. Forlines, M. Borkin, and C. Shen. Wespace: the design development and deployment of a walk-up and share multi-surface visual collaboration system. In Proceedings of the 27th international conference on Human factors in computing systems, CHI ’09, pages 1237–1246, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-246-7. → pages 14 [45] D. Wigdor, H. Benko, J. Pella, J. Lombardo, and S. Williams. Rock & rails: extending multi-touch interactions with shape gestures to enable precise spatial manipulations. In Proceedings of the 2011 annual conference on Human factors in computing systems, CHI ’11, pages 1581–1590, New York, NY, USA, 2011. ACM. ISBN 978-1-4503-0228-9. → pages 89  103  Appendix A  Algorithms and Code Segments A.1 Discovering and Registering with Central Controller 1 2 3 4 5 6 7 8 9  int k = 0; // size of local network int n = 16; // base address of current network string base = ""; // controller port int port = CONTROLLER_PORT; // result central controller address string controller;  10 11 12 13 14 15 16 17 18 19 20 21  while (true){ // get the k-th address in current network controller = GetKthAddress(base, k); // try to connect and invoke the "register" service // on the controller. If successful, break and the // controller is found if (TryRegisterWithController(controller, port)) break; k = k + 1; // if searched all addresses, repeat again, // because controller might be starting...  104  if (k >= n) k = 0;  22 23 24  }  A.2 An Example of a KML File for Adding a Model 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20  <kml xmlns="http://www.opengis.net/kml/2.2"> <Placemark> <Model id="model_4"> <altitudeMode>relativeToGround</altitudeMode> <Location> <longitude>-105.272774533734</longitude> <latitude>40.009993372683</latitude> <altitude>0</altitude> </Location> <Orientation> <heading>0</heading> <tilt>0</tilt> <roll>0</roll> </Orientation> <Link> <href>http://url.to/the/model.dae</href> </Link> </Model> </Placemark> </kml>  A.3 KML File Requesting Google Earth to Reload the Address Every 2 Seconds 1 2 3 4 5 6  <kml xmlns="http://www.opengis.net/kml/2.2"> <NetworkLink> <name>Current Plan Link</name> <Link> <href>http://localhost:7526/plans/0</href> <refreshMode>onInterval</refreshMode>  105  7 8 9 10  <refreshInterval>2</refreshInterval> </Link> </NetworkLink> </kml>  A.4 An Example of a Design Task Configuration File (Partial) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23  <MapPackage> <PackageName>Chinatown</PackageName> ... <!-- where the map image is stored --> <MapImgString> maps\SS-Map_Table_5k.png </MapImgString> ... <!-- where case list is stored --> <ModelCSVFileLocation> Database.csv </ModelCSVFileLocation> <!-- geographic location information --> <TopLeftLat>49.282294</TopLeftLat> <TopLeftLon>-123.107665</TopLeftLon> <BotRightLat>49.273989</BotRightLat> <BotRightLon>-123.094913</BotRightLon> ... <!-- size of the map --> <MapWidth>2769</MapWidth> <MapHeight>2769</MapHeight> ... </MapPackage>  106  


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items