Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Unlu, Arda Ege
Master of Science - MSc
University of British Columbia
Immersive augmented reality systems allow users to superimpose digital content on their physical environments while immersive virtual reality systems simulate virtual spaces that make use of users' knowledge of the physical world. However, in such novel interactive systems, user unfamiliarity can break the illusion of seamless immersion and hinder adoption. We propose to bridge this gap by introducing familiar interactions and environments into immersive augmented and virtual reality. First, to explore familiar interactions, we propose using smartphones, which are familiar and readily available, as companion devices complementing existing interaction modalities for augmented reality head-mounted displays. Current-generation devices primarily support interaction modalities such as gesture, gaze and voice, which are less familiar as interaction modalities and lack precision and tactility, rendering them fatiguing for extended interactions. We leverage user familiarity with smartphone interactions, coupled with their support for precise, tactile touch input, to unlock a broad range of interaction techniques. For instance, by combining the phone's spatial position and orientation and high-resolution touchscreen within an augmented reality environment, we transform the ordinary smartphone into an interior design palette, a rapier or a bow. We describe a prototype implementation and report on the results of a positional accuracy study. Second, we believe that bringing familiar real world environments into virtual spaces can help create immersive and interactive virtual reality experiences. To explore this, we have chosen the real-world library as an expedient pilot and introduce the Library of Apollo, a virtual-reality experience which aims to bring together the connectedness and navigability of digital collections and the familiar browsing experience of physical libraries. An online deployment of our system, together with user surveys and collected session data showed that our users showed a strong preference for our system for book discovery with 41/45 of users saying they are positively inclined to recommend it to a bookish friend.
Designing Familiar Augmented and Virtual RealityEnvironments and Interactions Through Off-the-ShelfReal-World SolutionsbyArda Ege UnluB.Sc., Bilkent University, 2019A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)August 2021© Arda Ege Unlu, 2021The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Designing Familiar Augmented and Virtual Reality Environments andInteractions Through Off-the-Shelf Real-World Solutionssubmitted by Arda Ege Unlu in partial fulfillment of the requirements for the de-gree of Master of Science in Computer Science.Examining Committee:Robert Xiao, Computer Science, The University of British ColumbiaSupervisorDongwook Yoon, Computer Science, The University of British ColumbiaSupervisory Committee MemberSid Fels, Electrical and Computer Engineering, The University of British ColumbiaSupervisory Committee MemberiiAbstractImmersive augmented reality systems allow users to superimpose digital contenton their physical environments while immersive virtual reality systems simulatevirtual spaces that make use of users’ knowledge of the physical world. However,in such novel interactive systems, user unfamiliarity can break the illusion of seam-less immersion and hinder adoption. We propose to bridge this gap by introducingfamiliar interactions and environments into immersive augmented and virtual real-ity.First, to explore familiar interactions, we propose using smartphones, whichare familiar and readily available, as companion devices complementing exist-ing interaction modalities for augmented reality head-mounted displays. Current-generation devices primarily support interaction modalities such as gesture, gazeand voice, which are less familiar as interaction modalities and lack precision andtactility, rendering them fatiguing for extended interactions. We leverage user fa-miliarity with smartphone interactions, coupled with their support for precise, tac-tile touch input, to unlock a broad range of interaction techniques. For instance, bycombining the phone’s spatial position and orientation and high-resolution touch-screen within an augmented reality environment, we transform the ordinary smart-phone into an interior design palette, a rapier or a bow. We describe a prototypeimplementation and report on the results of a positional accuracy study.Second, we believe that bringing familiar real world environments into vir-tual spaces can help create immersive and interactive virtual reality experiences.To explore this, we have chosen the real-world library as an expedient pilot andintroduce the Library of Apollo, a virtual-reality experience which aims to bringtogether the connectedness and navigability of digital collections and the familiariiibrowsing experience of physical libraries. An online deployment of our system, to-gether with user surveys and collected session data showed that our users showeda strong preference for our system for book discovery with 41/45 of users sayingthey are positively inclined to recommend it to a bookish friend.ivLay SummaryWe studied the use of two different systems that are familiar and much-loved in thereal-world, smartphones and libraries, within the context of virtual and augmentedreality.First, we propose using smartphones, which are familiar and ubiquitous, ascompanion devices for head-mounted augmented reality displays. For example, bycombining the phone’s positional information and high-resolution touchscreen weunlock a broad range of interactions and transform the ordinary smartphone intoan interior design palette, a rapier or a bow.Second, we believe that bringing familiar real-world environments into vir-tual spaces could help create immersive virtual reality experiences. We suggestthat a virtual library can offer the serendipitous browsing experience of a physicallibrary. To explore this, we introduce the Library of Apollo, a virtual-reality expe-rience which aims to bring together the connectedness and navigability of digitalcollections and the familiar browsing experience of physical libraries.vPrefaceThis thesis is submitted in partial fulfilment of the requirements for a Master ofScience degree in Computer Science at The University of British Columbia. Thework presented in this dissertation is the original and independent work of theauthor, performed under the guidance and direction of Prof. Robert Xiao. Thisthesis is organized around two major projects, presented in Chapters 3 and 4, bothof which are designed, implemented and evaluated by the author of this thesis.Research was conducted under the following ethics certificate: H19-02782.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Virtual and Augmented Reality . . . . . . . . . . . . . . . . . . . 11.2 Immersion and Interactivity in AR/VR . . . . . . . . . . . . . . . 21.3 Familiarity in Design . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . 62 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.1 Metaphor and Familiarity in Design . . . . . . . . . . . . . . . . 72.2 Smartphones as Augmented Reality Controllers . . . . . . . . . . 8vii2.3 Bringing the Physical Library into Virtual Reality . . . . . . . . . 102.3.1 Book Browsing and Serendipity in Libraries . . . . . . . . 102.3.2 Book Browsing and Serendipity in the Digital Age . . . . 113 PAIR: Phone as an Augmented Immersive Reality Controller . . . . 133.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.1.1 Marker Detection and Pose Estimation . . . . . . . . . . . 153.1.2 Visual-Inertial Odometry . . . . . . . . . . . . . . . . . . 173.1.3 Coordinate System Alignments and Drift Correction . . . 173.1.4 Transmission of Other Data . . . . . . . . . . . . . . . . 203.2 Interactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.2.1 Phone as an Input Device . . . . . . . . . . . . . . . . . . 203.2.2 6-DOF Input . . . . . . . . . . . . . . . . . . . . . . . . 213.2.3 Touchscreen Input . . . . . . . . . . . . . . . . . . . . . 223.2.4 Intersections of 6-DOF and Touchscreen Input . . . . . . 223.2.5 Phone as an Output Device . . . . . . . . . . . . . . . . . 243.2.6 Display . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.2.7 Tactile Feedback . . . . . . . . . . . . . . . . . . . . . . 243.2.8 Audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.3.1 Interior Design Application . . . . . . . . . . . . . . . . 253.3.2 Games - Archery & Fruit Slicer . . . . . . . . . . . . . . 273.3.3 Celestial Bodies . . . . . . . . . . . . . . . . . . . . . . 283.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.4.1 Conditions and Procedure . . . . . . . . . . . . . . . . . 293.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.5.1 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.5.2 Drift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.5.3 Markerless Tracking . . . . . . . . . . . . . . . . . . . . 344 Library of Apollo: A Virtual Library Experience in yourWeb Browser 354.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . 35viii4.1.1 Designing to Faithfully Recreate Physical Libraries . . . . 364.1.2 Building a Familiar and Expressive Search Bar . . . . . . 374.1.3 Engineering a Smooth Browsing Experience . . . . . . . 384.1.4 Providing Access to Digitized and Physical Books . . . . 404.2 Online Deployment and Survey Results . . . . . . . . . . . . . . 414.2.1 The Front-end and the Cloud Architecture . . . . . . . . . 414.2.2 The Deployed Dataset . . . . . . . . . . . . . . . . . . . 414.2.3 Deployment and Recruitment . . . . . . . . . . . . . . . 414.2.4 Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . 45Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47A Supporting Materials . . . . . . . . . . . . . . . . . . . . . . . . . . 55ixList of TablesTable 3.1 Gestures and activities in the study . . . . . . . . . . . . . . . 30Table 3.2 Results summarized by condition. . . . . . . . . . . . . . . . . 32xList of FiguresFigure 3.1 Applications enabled by our technique. From left to right: a)”make it rain” app where users can catapult virtual currency offthe screen; b) archery game where the user holds the phone likea bow; c) home decorations app which allows users to placevirtual furniture; and d) solar system visualizer with customtouchscreen controls. . . . . . . . . . . . . . . . . . . . . . . 13Figure 3.2 Marker boards are customizable. Different size, scale and place-ments allow application-specific trade offs. From left to right,a full marker bezel, horizontal marker rows, vertical markercolumns, and a half bezel are used for different applications. . 14Figure 3.3 Occlusion resilience of a board of markers, a rectangular whiteand red frame is shown over the phone’s estimated pose throughthe detected markers. (Coloring of the virtual overlay is merelyfor aesthetic purposes, it is not indicatory.) . . . . . . . . . . . 15Figure 3.4 Snapshots from our game bundle. The 6-DOF tracking ofPAIR can handle sword play as well as shooting arrows. . . . 16Figure 3.5 Virtual 3D trackpad. The phone plane is visualized in gray,pan gestures on the phone move the cube on this plane. Thephone plane follows the orientation of the phone. . . . . . . . 18Figure 3.6 Example of the catapult swipe gesture, swiping $100 bills intoexistence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Figure 3.7 Scaling of a sofa through the pinch gesture. . . . . . . . . . . 21Figure 3.8 A simple controller for picking colors. . . . . . . . . . . . . . 22xiFigure 3.9 The user can approach virtual objects with the phone to interactwith them. Here, explosive arrows can be taken out of a quiverlaying on the ground. . . . . . . . . . . . . . . . . . . . . . . 23Figure 3.10 Before and after of an in-air broad swipe that slices a melon inhalf. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Figure 3.11 The devices that are used in the evaluation, a Microsoft HoloLenswith 5 infrared markers attached and an Apple iPhone 11 ProMax with an antenna of 5 infrared markers. . . . . . . . . . . 26Figure 3.12 Custom orbital controllers for the Earth and Moon. . . . . . . 29Figure 4.1 A wide-angle shot of the shelves. . . . . . . . . . . . . . . . 36Figure 4.2 Search results for ‘a tale of two cities’, showing both shelf andbook results. A: Shelf results list LCC classes that are search-hits, while book results are hits for titles, authors and subjectheadings of books. The terms that led to the search-hits arehighlighted. B: Hovering over the ‘Under Shelf’ portion ofbook results expands the shelf hierarchy. . . . . . . . . . . . 37Figure 4.3 A: Clicks on shelved books bring up synoptic book panels. Theelements in the subject heading table are clickable. B: Thesearch-results after the “church architecture” subject headingwas clicked. The hits on subject headings are highlighted withshades of green. . . . . . . . . . . . . . . . . . . . . . . . . . 38Figure 4.4 A: The scroll-view of class hierarchies. Hierarchies can becollapsed by clicking any of the intermediate elements. B: Thecollapsed hierarchies after the click at A. The hierarchies canbe scrolled up and down, or expanded by clicking their tagpanels if they contain child categories. . . . . . . . . . . . . . 39Figure 4.5 A: The “Peek Inside” button on book panels opens an e-readerthat uses OpenLibrary’s digitized book archive. B: The cloudarchitecture of Library of Apollo. . . . . . . . . . . . . . . . 40Figure 4.6 A: The change in reading habits due to COVID-19 pandemic.B: The user perception of the library’s navigability, ease of use,browsing quality and overall quality. . . . . . . . . . . . . . . 43xiiFigure A.1 Part 1 of survey questions for Library of Apollo . . . . . . . . 55Figure A.2 Rest of the Part 1 of survey questions for Library of Apollo . . 56Figure A.3 Part 2 of survey questions for Library of Apollo . . . . . . . . 57Figure A.4 Part 3 of survey questions for Library of Apollo . . . . . . . . 58Figure A.5 Part 4 of survey questions for Library of Apollo . . . . . . . . 59xiiiGlossaryAR Augmented RealityDOF Degree of FreedomLCC Library of Congress ClassificationLCSH Library of Congress Subject HeadingsTCP Transmission Control ProtocolUDP User Datagram ProtocolVR Virtual RealityxivAcknowledgmentsI would like to thank my supervisor Dr. Robert Xiao for his priceless mentoring,guidance and advice. This thesis would not come to be without his immense helpat every stage from ideation to writing. He made it possible for me to creativelycombine innovative and practical technologies, allowed me to take chances at everyturn and enjoy the research process fully for two years and I am most grateful.I would also like to thank my parents for their continuous love, support andsteady encouragement over the years.xvChapter 1Introduction1.1 Virtual and Augmented RealityConventional human-computer interfaces have mostly restricted users to keyboardand mouse, joystick and touchscreen input with visual output on a 2D screen with2D graphics, with limited auditory and haptic feedback [1]. The tantalizing ar-tificial realities that are part computer simulation, part consensual hallucination,promised by Virtual Reality (VR) and Augmented Reality (AR) involve a muchtighter coupling between an extended range of input and output options [1, 2]. BothVR and AR render realistic digital content to create immersive, three-dimensionalworlds and offer many novel interactive possibilities within their simulated cre-ations.AR overlays its renderings on the real world view of the user, often superim-posing content on real world objects and may also make use of the user’s physicalposition, activity and surroundings to update its view or to take in input. On theother hand, VR simulates a realistic environment anew, separate from the user’sreal world surroundings. One can think of the first-person view of T-800 from TheTerminator in which digital data and holograms were overlaid on top of targets asan example of an AR interface, or the vast simulated world in The Matrix or ReadyPlayer One as very realistic renditions of VR.AR/VR technology platforms can be roughly grouped into two categories,firstly the more conventional platforms like desktop computers and smartphones1can enable various AR/VR applications, through 3D graphics, custom controllers,inside-out tracking etc. Secondly, there are Head Mounted Displays (HMD), thatproject stereographic content on the HMD’s screen, presenting computer graphicsdirectly on the user’s view. HMDs often track the user’s head, hands and body touse those as input modalities for interactions with the projected virtual world andto simultaneously adjust the user’s view of the virtual world. VR HMDs may ormay not allow an outside view, while AR HMDs categorically do. Because of thevery immersive and interactive 3D worlds that they enable, AR/VR HMDs havebeen at the forefront of the excitement generated around AR and VR in the lastdecade.1.2 Immersion and Interactivity in AR/VRCompared to more conventional computing interfaces, AR/VR technology plat-forms introduce different interaction paradigms steeped in immersion and interac-tivity. Immersion is the sensation of being in an environment, the mental state ofbeing deeply engaged and involved in said environment often accompanied witha suspension of disbelief. This mental state can be reinforced through physicalmeans [3], which AR/VR achieves by appealing to visual, auditory, haptic andspatial senses for output and bodily movements for input. AR/VR applications inwhich these are synergistically used can create especially immersive experiences.For example, the collaborative bike race exercise games featured on some station-ary bikes, and the rhythm game Beat Saber on VR HMDs in which blocks repre-senting musical beats are sliced with a pair of contrasting-colored sabers are suchexamples of AR and VR respectively.Interactivity refers to the ability of the user to provide input to and get outputfrom the system they are engaging with [3]. Interactivity can come in many forms:it can be as simple as typing on a keyboard and seeing text on the screen, while in aVR environment it usually entails interactions with objects, characters and places,moving and looking around freely, using the head motion and controllers whenusing an HMD or through mouse and keyboard when using a desktop computer.Different AR/VR platforms offer new possibilities for novel interactions, howeverwith the risk of being too unfamiliar and confusing to the point of compromising2the magical immersion of the novel virtual worlds.For a high level of immersion and interactivity, the design of virtual environ-ments and interaction techniques are primarily important, and familiarity signifi-cantly plays into both. In virtual environments familiar places and objects have in-tuitive affordances that can be immediately utilized without any manuals or guides,hence the users can feel much more immersed, expressive and confident. In inter-actions, users have expectations formed through their real world experience andconventional computing interfaces. Any discordance between those can break theflow of interactivity and immersion and hamper usability. In the next subsection1.3, we briefly introduce familiarity and then follow with our contributions of fa-miliar interaction and environment design in AR/VR.1.3 Familiarity in DesignFamiliarity means an intimacy with a thing that makes involvement with it unde-manding or rather straightforward [4]. Similarly, it has been argued that a userinterface is as intuitive as it is familiar. Thus a familiar interface is one in whichalready existing skills can be readily used. Others have shown that familiarity isnot necessarily orthogonal but very much correlated to ease-of-use, attractivenessand productivity of a human-computer interface [5, 6].Familiar interfaces feel intuitive, and users can work through them without ap-parent effort, thought or guidance [5, 7]. Intuitiveness in turn help to achieve asense of flow and harmony which is critical for immersive experiences. To helpachieve this flow through familiarity, user experience designers often make use ofphysical metaphors like folder icons on desktops, page-flip animations on docu-ment readers and shutter-click sounds on digital cameras. Another work recentlyshowed a preference by elders for realistic and photographic app icons over ab-stract and simple ones [8]. As billions of smartphones are actively used everyday[9], their interfaces and interactions have become intensely familiar to almost ev-eryone [10] which can now be exploited by designers to establish familiarity withunfamiliar systems and interfaces.Another avenue where familiarity can be exploited is in the design of virtualenvironments and objects. There is a lot to draw inspiration in the real world and3some are not only familiar to most but also well designed and engineered for cer-tain uses. We believe that these real world environments can be brought into ARand VR worlds, passing through a spectrum of tweaks that ranges from faithfulreplication to transmogrification, to make users feel at home, in places where theycan intuitively move and interact. Others, inspired by the realism and immersionthat VR can provide, have explored ways to bring museums, galleries, exhibitionsand even archaeological sites into virtual worlds [11, 12].1.4 ContributionsIn this work we propose to bring two separate familiar and intuitive systems fromthe real world into the design of virtual environments and interactions, both aimedto be novelties to bolster the interactivity and immersion aspects of VR and AR.First, we propose using smartphones, which are readily available to any user,as companion devices to HMDs to complement existing AR interaction modali-ties. We leverage user familiarity with smartphone interactions, coupled with theirsupport for precise, tactile touch input, to unlock a broad range of interaction tech-niques. For instance, by combining the phone’s spatial position and orientation,touch input and high-resolution screen, the ordinary smartphone can be turned intoan interior design palette, touch-enabled catapult or AR-rendered sword. We de-scribe a prototype implementation of our interaction techniques using an off-the-shelf AR headset and smartphone, named PAIR, demonstrate enabled applications,and report on the results of a positional accuracy study.Current-generation augmented reality HMDs are principally driven by naturalinteraction techniques, such as gaze, gesture or voice, or by custom-designed hand-held controllers (e.g. Magic Leap Control). Natural techniques are convenient (notrequiring additional hardware) and intuitive, but lack precision and speed and canbe fatiguing to use for extended periods. On the other hand, controllers can be veryprecise, often featuring button, touch and orientation inputs, but introduce the ad-ditional cost and complexity of learning a new device. In contrast, smartphones arenow nearly ubiquitous, carry complex and precise sensors and provide comfort-able, expansive touch input (when compared to most AR controllers). Users arealso intuitively familiar with smartphones, and are comfortable using them for ex-4tended periods of time. Thus, since many users possess smartphones, using themas controllers incurs no additional hardware cost, nor imposes a learning curve.Users can simply download an app to begin using their smartphone as a powerful,general-purpose AR controller.Second, we believe that bringing well-known, familiar real world environmentsinto virtual spaces could help create immersive and interactive VR applications.The physical affordances that exist in the real world could also be faithfully repli-cated or transfigured to fit in a virtual environment. To that end, we fleshed outan example application in the figure of a virtual library. We propose virtual li-braries that can offer the browsing experience of real world physical libraries thatallow for fast, easy and high quality serendipitous book discovery. Physical li-braries are good real world targets to bring into virtual environments as they areextremely dense stores of information with a unique browsing experience that isnot yet faithfully replicated by any digital system. To explore this, we introducethe Library of Apollo, a virtual-reality experience which aims to bring together theconnectedness and navigability of digital collections and the familiar browsing ex-perience of physical libraries. Library of Apollo is an infinitely scrollable array ofshelves that has 9.4 million books distributed over 200,000 hierarchical Library ofCongress Classification categories, the most common library classification systemin the U.S. The users can jump between books and categories by search, clicking onthe subject tags on books, and by navigating the classification tree by simple col-lapse and expand options. An online deployment of our system, together with usersurveys and collected session data showed that our users showed a strong positivepreference for our system for book discovery.Currently, online library interfaces are generally search result based and thistakes away from the valuable serendipitous discoveries that are often made inphysical libraries through browsing. Book discoveries made through browsing arefundamental to most readers’ lives. Libraries often play a unique role in this ex-ploratory process. Large collections are classified, organized and laid out throughthe use of continuously-updated, rigorous and systematic classification systems,such as the Library of Congress Classification (LCC) system in the U.S. Under theLCC system, books are uniquely classified into a tree of classifications, and areadditionally tagged with one or more standardized subject headings. These cate-5gorization systems make relationships between books physically explicit, puttingeach book in the context of every other book within the same and nearby shelvesthat have similar books, adjacent in subjects, authorship, time and geography. Thisallows for a natural browsing experience, fast and high quality serendipitous bookdiscoveries. In this work, we make an attempt to bring these much loved physicalplaces into virtual worlds.1.5 Thesis OrganizationThis thesis is organized around two major projects, the PAIR and the Library ofApollo works that are referenced in the previous Section 1.4. In Chapter 2, RelatedWork, outlines concepts relevant to both works, together with the related workon familiarity, intuitiveness and skeumorphism in human-computer interaction anddesign. The presentation of each work in Chapters 3 and 4 starts with their designand implementation sections which is then followed by evaluation and discussionsections. Finally, Chapter 5 concludes the work and discusses future avenues ofexploration.6Chapter 2Related Work2.1 Metaphor and Familiarity in DesignFamiliarity conveys an understanding and intimacy with something that makes fur-ther involvement with that thing undemanding and straightforward [4]. In a similarvein, Raskin [5] argued that a user interface was as intuitive as it was familiar, i.e.as much as it resembled some process or thing already known. Thus a familiaruser interface is one into which existing skills can be readily transferred. For suchinterfaces, phrases like ”this feature is intuitive” are commonly used, implying theinterface works the way the user expected it to and that neither effort, thoughtnor training were necessary [5, 7]. It has been suggested that familiarity stronglyeffects not only perceived ease of use but also attractiveness and productivity inhuman-computer interfaces and interactions[5, 6].In the domain of computer technologies, designers have used metaphor andanalogy to make their systems familiar to new users. Perhaps the most famous ex-ample was made more than 40 years ago when the Xerox Star adopted the ”desk-top” metaphor for the first Graphical User Interface ever built. This desktop ma-chine then contained icons of familiar desk objects like folders, file drawers, bas-kets and printers [13]. Many others [14, 15] have argued that metaphor and analogyare at the very basis of thinking and learning, further supporting the use of familiar,and therefore analogies such as desktops, as a basis of human-computer interfacedesign.7Additionally, the design principle called ‘skeuomorphism’ taps directly into fa-miliarity, it aims to use physical metaphors for user interfaces and emulates phys-ical functions and aesthetics in digital devices. App icons of pen and paper fornotepad applications and the shutter-click sound on smartphone cameras are widelyused skeumorphs. Recently, Cho and Kwon [8] investigated the elders’ preferencefor skeumorphic smartphone app icon designs and found a positive preference forskeumorphic icons over abstract ones, and further a preference of realistic and pho-tographical representations over simple and flat ones.Today, the smartphone is the most ubiquitous computing platform with billionsof active devices in everyday use [9]. This has also made its various interface andinteraction paradigms and popular apps tremendously familiar, some even now feelsecond-nature to a lot of users [10]. Somewhat related to our work, Pokemon Goemerged at the intersection of the novelty of nostalgia and the ubiquity of the smart-phone, bringing together two familiar tech phenomena albeit of different times torapidly ascend to cultural zeitgeist [16].Another familiar physical system though not as high-technology are libraries.Most who have gone through formal education have at least been in some touchwith libraries and experienced the serendipitous book browsing they facilitate.However, most online library interfaces do not even try to replicate the physicalaffordances that can make a similar browsing experience possible. Keeping themuch emphasized [17–20] preference for physical book browsing in mind, we de-cided to bring these much loved and familiar environments into virtual reality.2.2 Smartphones as Augmented Reality ControllersSmartphones and other smart devices have long been proposed as companions toimmersive augmented reality. An early work, the Touring Machine [21], employeda tablet device with a stylus pen to control a head-mounted augmented reality dis-play. Budhiraja [22, 23] performed initial explorations into using a smartphonewith immersive AR, proposing to use the phone for both spatial and touch input.The smartphone has also been explored as a high-resolution display to comple-ment the low-resolution AR environment [24, 25], as an interface to manipulate3D content [26–29], or as part of a multi-display environment [30]. Most recently,8BISHARE [31] combined many of these past explorations into a design space ofbidirectional interactions between a smartphone and an HMD.Notably, all of these past systems employed external tracking in order to obtainthe phone’s spatial position, attaching markers to the phone in order to preciselytrack the phone’s 6Degree of Freedom (DOF) position with external cameras (e.g.Vicon). While external tracking is suitable for prototyping interactions, it does notlend itself to practical deployment. Our goal with PAIR was to explicitly demon-strate the feasibility of using unmodified off-the-shelf devices. In particular, weaim to explore how recent developments in smartphone AR, such as Apple ARKit[32], can be used to precisely track the phone’s 3D rotation and 3D position (6DOF)within the augmented reality environment.Most related to our technical approach are systems which employ fiducialmarkers or visual-inertial odometry to track objects in the environment. Pocket6[33] employs a smartphone as a stand-alone 6DOF controller for desktop comput-ing, using Apple’s ARKit (visual-inertial odometry) to track the device. It demon-strates that ARKit can provide high spatial and positional accuracy.Only a handful of systems have aimed to enable 6DOF interactions betweena smartphone and an HMD without external tracking. Dual-MR [34] appears toemploy tracking on both the phone and the AR HMD to provide 6DOF input, butthe implementation is not described in detail and does not address drift betweenthe tracking systems. TrackCap [35] uses the camera on the phone to sense aspecial fiducial grid attached to the headset, an approach that is limited to a trackingvolume approximately 1 x 1 x 0.6m in size by the phone’s front camera field ofview. By contrast, our approach requires no modification of the headset or phoneand enables a much larger tracking area without positional or rotational limitations.TrackCap includes some experiments integrating the IMU and depth camera, butdo not evaluate stability nor accuracy.Our work is also inspired by explorations of control schemes for mixed andaugmented reality. Natural interactions have been extensively explored [36], in-cluding gaze [37], gesture [38, 39], voice [40], and even touch on the environment[41]. PAIR extends and complements these natural interaction approaches; forinstance, the use of a smartphone (as opposed to e.g. a tablet computer) leavesone hand free for gesture or touch interactions. Alternatively, users could interact9with the system via special hardware, such as gloves [42] or handheld IMU-basedsensors [43, 44]. These systems can provide precise, tactile input, at the cost ofrequiring users to carry around and learn extra hardware in addition to the HMDdisplay.2.3 Bringing the Physical Library into Virtual Reality2.3.1 Book Browsing and Serendipity in LibrariesInformation seeking in physical libraries takes two forms: search of known itemsand browsing. Patrons may begin with a search of a known item, but through undi-rected browsing of nearby items, often discover new and unknown books serendip-itously [45]. Even though patrons could conceivably find interesting items in com-pletely disarranged stacks, libraries aim to ease this fundamental browsing processby cataloguing, classifying and shelving books according to some classificationsystem [46], putting like near like, deemed so through their topics, subtopics, au-thorship, form, place and time.There are different classification systems used for different purposes in differ-ent parts of the world, and as we have focused on large collections as those hostedby research and academic libraries, we decided to use the subject-based, hierarchi-cal LCC [47], currently one of the most widely used library classification systemsin the world [48].LCC divides all knowledge into twenty-one basic classes, each identified by asingle letter. These are then further divided into more specific subclasses, identifiedby two or three letter combinations (class N, Art, has subclasses NA, Architecture;NB, Sculpture etc.). Each subclass includes a loosely hierarchical arrangementof the topics pertinent to the subclass, going from the general to the more spe-cific, often also specifying form, place and time [47, 48]. LCC grows with humanknowledge and is updated regularly by the Library of Congress [48].Library shelves are meticulously organized to encourage serendipity [45, 49]and call numbers themselves are markers that indicate the semantic content ofshelved items [50]. There is some quantified evidence about the neighbour ef-fects created by Dewey Decimal and Library of Congress Classification systems: a10nearby book being loaned increases the probability of a subsequent loan by at least9-fold [51], indicating that browsing is happening regularly and effectively withinlibraries organized according to these classification systems. There is also strongspoken and anecdotal preference for physical library shelf browsing [17, 20] andusers bemoan the lack of opportunity to do so online [17–19].More recently, McKay et. al. detailed the actions taken during library browsingby users, like reading signs, scanning, the number of books, shelves, bays touchedand examined etc., indicating an idiosyncratic browsing behaviour by each user aswell as a high success rate, with 87% of users leaving their browsing session withone or more books [52]. These results suggest a unique and creative character tobrowsing within the general scope of information retrieval.2.3.2 Book Browsing and Serendipity in the Digital AgeEven though libraries have been amongst the earliest adopters of computers, dig-itization and online access [53] and there has been a clearly expressed sentimentfavoring browsing [17–20], there has been little commensurate effort to carry thephysical shelf browsing experience to online libraries. Online library interfacesare almost universally search-result based, and we know the vast majority of usersonly see up to ten search results [54], and only interact with one or two in anydepth for further information [55]. These small numbers are hardly conducive tobrowsing and serendipitous discovery. The recommender systems used by onlinevendors are generally good to discover popular books, popular tastes and genreswhile they often fail to provide novel ranges of selections necessary for browsingand serendipity [56].There is some novel recent work that has been developed to facilitate serendip-itous browsing in libraries [57, 58]. The Bohemian Bookshelf [57] aimed to en-courage serendipity by creating five interlinked visualizations over its collection,based on books’ covers, sizes, keywords, authorships and timelines. This playfulapproach was deployed on a touch kiosk in a library and was received very posi-tively, but the collection size used for implementation and test was limited to only250 books, a number too small to be indicative of performance over larger collec-tions where some of the employed visualizations could become too cluttered and11complex to navigate.The Blended Shelf [58] carried over the physical shelves into very large touchdisplays that offered 3D visualizations of a library collection of 2.1 m books, con-forming to the standard classification used by that library. The 3D shelves weredraggable by swipe, can be searched and reordered and the classifications can benavigated via a breadcrumb-cued input mechanism. However there are no deploy-ments, tests or user studies regarding their implementation.Earlier works from library sciences and information studies include 3D librarysketches [59] and a cylindrical shelf prototype [60] for a small collection of a fewthousand books, that aimed for a spatial organization with visual cues for eas-ier browsing. Using the same digitized collection, [61] experimented with hemi-spherical immersive displays to view digitized shelves simultaneously employinga smaller screen to peek inside digitized books. More recently [62] implemented adigital bookshelf that is displayed on a wall screen on which books are presentedbased on metadata like price, cover colors, ratings etc. and along these adjacen-cies books could be serendipitously browsed. The Search Wall [63] proposed atangible interface for a large display for book browsing that makes heavy use ofphysical, spatial and graphical elements aimed at younger children. Another work[64] proposed a VR system for keyword searching and browsing over Library ofCongress Subject Headings (LCSH), with a prototype limited to Science-relatedheadings, that are shaped into a navigatable 3D tree structure. Pouke et al. [65]implemented a mirror-world virtual copy of a city library, with fantasy layers in-terleaved in between for book recommendations and collaborative work spaces,however there is little focus on book browsing within the stacks. Cook et al. [66]provides a broad analysis for the evolving research library and the benefits of vir-tual reality systems that could help provide lifelike browsing experiences in thenear future, emphasizing the importance of visual and spatial features if smooth,non-fatiguing, serendipitous discoveries are to be achieved.12Chapter 3PAIR: Phone as an AugmentedImmersive Reality Controller3.1 ImplementationIn order to use the phone as a 6DOF controller, the phone and headset must betracked within a uniform coordinate system. We accomplish this by having thephone and headset track themselves in the environment using SLAM [67] andvisual-inertial odometry [32] in their own coordinate system. We align these coor-dinate systems by displaying an unobtrusive grid of fiducial markers [68] aroundthe bezel of the phone, and tracking these fiducial tags using the headset’s frontcamera. This provides the phone’s 6DOF position in the coordinate system of theFigure 3.1: Applications enabled by our technique. From left to right: a)”make it rain” app where users can catapult virtual currency off thescreen; b) archery game where the user holds the phone like a bow;c) home decorations app which allows users to place virtual furniture;and d) solar system visualizer with custom touchscreen controls.13Figure 3.2: Marker boards are customizable. Different size, scale and place-ments allow application-specific trade offs. From left to right, a fullmarker bezel, horizontal marker rows, vertical marker columns, and ahalf bezel are used for different applications.headset. At startup time, the initial detection of this grid provides the initial align-ment of the two coordinate systems.If the phone’s visual-inertial odometry was perfect, the markers would not beneeded after this initial alignment. However, the phone’s own tracking is error-prone and susceptible to drift over time, especially after rapid movements or asthe phone moves to unexplored places in its environment. For this reason, PAIRdoesn’t rely on the initial alignment alone, but rather realigns coordinate systemswhenever the markers become visible, ensuring consistent and healthy tracking.The PAIR prototype is implemented using a first-generation Microsoft HoloLensheadset and an iPhone 11 Pro Max smartphone, with software written in Unity onthe HoloLens and Swift on the iPhone. Our tracking technique makes use of theHoloLens’s built-in inside-out tracking, which uses the built-in infrared and depthcameras and SLAM, and could thereby be adapted to any headset with inside-outtracking. On the phone side, our technique can work with any relatively recent iOSor Android phone equipped with the ARKit or ARCore technologies respectively,which provide the necessary inside-out tracking using the phone’s camera.Our tracking technique, as outlined above, is essentially composed of three14Figure 3.3: Occlusion resilience of a board of markers, a rectangular whiteand red frame is shown over the phone’s estimated pose through thedetected markers. (Coloring of the virtual overlay is merely for aestheticpurposes, it is not indicatory.)distinct steps: marker detection and phone pose estimation on the headset, visual-inertial odometry tracking on the phone, and alignment of the coordinate systems.3.1.1 Marker Detection and Pose EstimationHoloLens includes a front-facing camera which enables applications to see fromthe user’s viewpoint [69]. We run this camera at the highest frame rate (30 framesper second) at a low resolution (896x504, 48 degree horizontal field of view) inorder to minimize latency and reduce image processing load. This low resolutionis still sufficient for our choice of markers and makes it cheaper to run it at a higherframerate.We decided to use the popular ArUco markers [68] as our fiducial markers,as they are visually simple, provide accurate pose estimation, and are supportedby the open-source OpenCV library [70]. An ArUco marker is a synthetic squaremarker composed of a wide black border, which facilitates detection, and an innerbinary matrix which determines its identifier (id)[71]. Rather than displaying asingle marker on the phone, we display a board of ArUco tags around the peripheryof the phone’s display. The relative position between the markers in the board isknown a priori, enabling pose estimation for the entire board even when only asmall subset of its markers are visible [71] and improving the overall accuracy.Occlusion resilience is illustrated with a marker bezel in Figure 3.3.15Figure 3.4: Snapshots from our game bundle. The 6-DOF tracking of PAIRcan handle sword play as well as shooting arrows.The layout of the markers can be customized based on the application, tradingoff between screen space, marker count and size, and aesthetics. Some of themarker configurations we used are shown in Figure 3.2. We kept markers small tominimize their footprint - markers only occupy 40% of the screen even in the mostnumerous configuration (a full bezel all the way around). Markers are between9-10mm in size and are separated by 2mm of white space. Each marker is drawnfrom the DICT_4X4_100 marker dictionary, which consists of 100 markers witha marker size of 4x4 bits.To implement ArUco board detection in our Hololens applications, we used theHololensForCV project with OpenCV 4.1.1. We implemented a number of customclasses to be used from Unity, to detect boards, estimate poses and to convert frommarker coordinate system to Unity’s coordinate system. We run the board detectionloop on a separate thread, and check whether any markers were detected in the maingame loop. The ArUco board detection in total takes around 80 ms on average,from the capture of the latest camera frame to pose estimation. When a board isdetected, we perform some checks on the ARKit data (detailed in Section 3.3), andthen run the coordinate alignment code. For the purpose of only using accurate16pose estimations, we chose to discard any pose estimation done with less than 5markers.3.1.2 Visual-Inertial OdometryMobile AR tools like ARCore or ARKit rely on the ability to create and track acorrespondence between real and virtual spaces. ARKit, which is used to build thePAIR prototype, combines data from the phone’s inertial sensors with computervision analysis of the scene visible to its rear camera. ARKit recognizes notablefeatures in the scene image, tracks differences in the positions of those featuresacross video frames, and compares that information with motion sensing data. Theresult is a high-precision model of the device’s position and motion [32].We use ARKit to track the device position and orientation, specifically, theposition of the rear-facing camera located in the upper-right corner. When ourapplication starts up, we present the standard ARKit setup screen, which promptsthe user to locate nearby horizontal and vertical surfaces for tracking purposes. Itthen establishes both a Transmission Control Protocol (TCP) and User DatagramProtocol (UDP) connection to the HoloLens, over WiFi. ARKit’s reported 6DOFposition and tracking state, updated roughly 60 times per second, is streamed to theHoloLens over UDP (prioritizing low latency over reliable data transfer). We alsoinclude a timestamp to this datagram to enable the HoloLens to detect out-of-ordermessages.3.1.3 Coordinate System Alignments and Drift CorrectionThe coordinate systems of the HoloLens and the phone are initialized arbitrarily,and thus differ in origin and axis orientation. By using the fiducial markers, we canmeasure the phone’s position in both coordinate systems simultaneously, and thusestablish the correspondence between these coordinate systems.Mathematically, orientations (R) are represented as 3x3 rotation matrices orquaternions, while translations (T ) are represented as 3x1 vectors. The transfor-mation from coordinate space A to coordinate space B consists of an (initially un-known) orientation component RAB and a translation component TAB. Suppose thatan object has 3D position xA and 3D orientation θA when viewed in coordinate17Figure 3.5: Virtual 3D trackpad. The phone plane is visualized in gray, pangestures on the phone move the cube on this plane. The phone planefollows the orientation of the phone.space A. Then, we have that θB = RABθA and xB = TAB +(RABxA).Once a point correspondence is obtained from both systems, in our case throughmarker detection on the HoloLens and ARKit on the phone, we can easily estimateR′AB and T′AB by reversing equations 1 and 2: R′AB = θBθ−1A and T′AB = xB−(R′ABxA).For PAIR, system A refers to the coordinate system of the headset, while systemB is that of the phone, so that θA, xA would represent the position of the phoneas detected by marker detection in the HoloLens coordinate system, and θB, xBwould represent the position of the phone as reported by ARKit in the phone’s owncoordinate system.The phone provides a continuous stream of position and orientation data, whichwe can translate into HoloLens coordinates via the computed R′AB and T′AB. How-ever, because the phone primarily relies on visual-inertial odometry, it is prone toaccumulating error (drift), which can be considered to be a continuous change inthe phone’s coordinate system, i.e. a continuous change in the true RAB and TABparameters. Therefore, the estimates R′AB and T′AB should be continually updatedas well. Thus, the HoloLens continually scans for markers, updating the estimated18Figure 3.6: Example of the catapult swipe gesture, swiping $100 bills intoexistence.transforms wherever it receives simultaneous correspondence of marker detectionsand ARKit data, i.e. whenever the ArUco board becomes visible to the HoloLens’sfront camera.Due to latency at all levels (camera shutters, marker detection, ARKit, net-working), it is possible that the detected markers and ARKit data are slightly outof sync temporally. If the phone is moving rapidly, the synchronization error couldresult in a bad estimate of the transform, throwing off the coordinate conversion.To mitigate this, we maintain a small buffer of ARKit and marker detections on theHoloLens, timestamped and synchronized to the head position at the time when theposition was received. If two consecutive ARKit measurements differ by a smallamount (less than 5 cm in position and 10◦in total rotation) and have matchingmarker board detections (more than 5 detected markers within 20 ms of the mea-surements), we update the estimated coordinate correspondences. Otherwise, wekeep the previous correspondences.By employing timestamped data, we ensure that the coordinate transforms willnot be off by more than 6 cm (5 cm per 60 Hz frame, times 20 ms for marker detec-tion) due to latency, with this worst case achieved only during rapid phone motion.19These thresholds also support continual coordinate alignment with phone motionup to 3 m/s (60 frames of 5 cm motion), which is sufficient for most interactiveuses. Tracking remains accurate as long as the marker grid is occasionally visibleto the front camera, and error generally accumulates slowly when the marker gridis not visible. We recommend that interactions which require precise spatial track-ing be designed to make use of the phone touchscreen, which makes it more likelythat the screen will be facing the user (and therefore facing the front camera).3.1.4 Transmission of Other DataSmartphones make exceptionally versatile controllers. Apart from 6-DOF track-ing, they also pack a high-resolution touchscreen, computational resources thatparallel or exceed that of the headset’s, and rich vibrohaptic feedback. In orderto exploit these other resources in PAIR, we establish a TCP connection betweenthe HoloLens and the phone (in parallel with the UDP connection used for sen-sor data), enabling reliable bidirectional communication of touch input, graphicalresources, and feedback commands.3.2 InteractionsWe implemented a range of interaction techniques which extend the input and out-put capabilities of the AR HMD using PAIR, drawing techniques from both theliterature and our own explorations of this design space.3.2.1 Phone as an Input DeviceOur tracking technique enables the smartphone to be used as a precise, flexible6DOF input device, and we explored a range of interaction techniques using thephone purely for 6DOF input. The smartphone can also be used for touch input, sowe borrowed many touchscreen interactions for PAIR: touch buttons and controls,touch gestures such as pinch and zoom and text input via on-screen keyboards.Finally, we explored interactions that blend together data from both 6DOF trackingand touchscreen gestures that we believe to be valuable for AR experiences. Wedemonstrated each category briefly with example applications.20Figure 3.7: Scaling of a sofa through the pinch gesture.3.2.2 6-DOF Input6-DOF tracking is inherently valuable to interact with virtual content. One tech-nique is to lock virtual 3D content to the phone’s position and rotation. For in-stance, the user can see 3D virtual furniture popping up on top of the phone as hebrowses through a furniture catalog (Figure 3.1c).6-DOF tracking can be used for tethered object interactions, e.g. the phone canbe turned into a sword or a bow, as can be seen in Figure 3.4. It can also be used asa pointer, for raycasting or aiming for small scale and fine tuned movements. Forfaster movements, rougher 6-DOF tracking enables fast sweeping motions in theenvironment, e.g. to let the user slice fruit in mid-air (Figure 3.10).As the phone is tracked in the coordinate system of the HMD, it can be usedto interact with virtual objects in its proximity. In Figure 3.9, the user picks upexplosive arrows from a quiver lying on the ground by approaching it with thephone.21Figure 3.8: A simple controller for picking colors.3.2.3 Touchscreen InputTouchscreen interactions have become commonplace due to their ubiquity and in-tuitiveness. They are friendly, expressive, precise and easy to perform. Any touch-screen gesture can be brought into the augmented world to manipulate virtual con-tent. Rotation, pinch, pan, tap, scroll gestures can be seen as immediately valuable.In Figure 3.7, we can see a pinch gesture performed to resize a sofa.As smartphones are programmable devices with considerable computationalresources, they can provide almost any kind of application specific control, re-source or UI for the companion HMD. This can be used to provide a colour palette(Figure 3.8) or a custom circular slider for lunar phases (Figure 3.1d).3.2.4 Intersections of 6-DOF and Touchscreen Input6-DOF data can be blended together with touchscreen gestures for enhanced flex-ibility, aesthetics, control and ergonomics. We show two powerful and intuitivecombinations of 6-DOF and touch data.22Figure 3.9: The user can approach virtual objects with the phone to interactwith them. Here, explosive arrows can be taken out of a quiver layingon the ground.Virtual 3D TrackpadThe touchscreen can provide a trackpad experience for AR for precise positionalmanipulations, but it is a 2D interaction and thus it is inherently limited to movingobjects in a 2D plane.We combine the touchscreen with the 3D orientation of the phone to create avirtual 3D trackpad. The pan gestures on the screen move virtual content on thecurrent plane of the phone. For example, if the phone is horizontally oriented,parallel to the ground, then objects also move on this horizontal plane. When thephone is vertically oriented and facing the user, an up/down pan gesture moves con-tent upwards/downwards, right/left pans moves them to the users right/left. Simi-larly, if the phone is sloped with respect to the ground, objects also move on thissloped plane. We propose this method as a rapid and precise technique to movevirtual content around. Multiple phone-plane oriented pans can additively translatean object to any position in the 3D environment (Figure 3.5).23Catapult SwipesVirtual content can be initialized into mixed reality from the phone. There is aparticularly fun way to do so, by mixing together touchscreen swipes with thephone’s position and orientation.A swipe can essentially instantiate a virtual object that is on the phone screeninto the mixed reality, and give it an initial velocity that is reflective of the swipe’svelocity and direction. This is visualized through swiping $100 bills into existencewith swipes in Figure 3.6. Similarly, one could imagine launching objects from avirtual catapult or slingshot to hit distant targets, or flinging balls out of the phoneto catch virtual creatures.3.2.5 Phone as an Output DeviceSmartphones can provide rich output experiences with their high-res screens, stereospeakers and vibrotactile haptics. We briefly breakdown some of these that webelieve to be most useful to ARHMDs.3.2.6 DisplayHigh resolution content displays are not a staple of typical ARHMD experiences.However, a phone controller can essentially display any content or control of in-terest on its high-res screen. For almost any such use, this would be an aestheticimprovement and in certain applications like those for computer-aided design, itwould also be a functional improvement. We provide a high-resolution furniturecatalog in our furniture app, leveraging the familiar browsing experience of smart-phones (Figure 3.1c).3.2.7 Tactile FeedbackSmartphones are very tangible devices with good programmable haptic engines.Haptic feedback can bring some tactile realism into ARHMD apps which are tra-ditionally missing with gaze and gesture inputs. In our bow & arrow application,the bow string can be gradually drawn to a maximum before release. As the bowstring is drawn more taut, the phone increases its vibration strength to give a sense24Figure 3.10: Before and after of an in-air broad swipe that slices a melon inhalf.of heaviness and power, with a sharp rigid vibration at release to give the sensa-tional feeling of the loosed arrow (Figure 3.4, left).3.2.8 AudioThe phone controller can provide spatial sound effects. A sliced melon can createa splash sound effect on the phone when the user uses the phone as a blade. Thiscould yield arguably more realistic sound experiences than playing sound clips onthe headset (Figure 3.10).3.3 ApplicationsWe built three showcase applications to highlight the use of a combination of in-teractions for different use cases.3.3.1 Interior Design ApplicationThis application combines the familiar browsing experience of smartphones, high-resolution content display, virtual content manipulation through touch input, cus-25Figure 3.11: The devices that are used in the evaluation, a MicrosoftHoloLens with 5 infrared markers attached and an Apple iPhone 11Pro Max with an antenna of 5 infrared markers.tom controls, phone as a pointer to grab objects, and an implementation of ourvirtual 3D trackpad interaction for virtual object placement.Browsing: The starting screen lands the user on a catalog of furniture underdifferent categories as seen in Figure 3.1c. The user can browse through, or pressthe navigation controller to type and search for a specific category. The selectedfurniture pops on top of the phone. The user can move around and the selectedfurniture stays on top of the phone.Virtual 3D Trackpad: The user can use the screen as a trackpad to move thefurniture in 3D, as described in section 4.1.3.Other Touch Interactions: The entire screen is receptive to touch gestures. Panis reserved for the Virtual 3D Trackpad. A two-fingered pinch scales, a two-fingered rotation gesture rotates the furniture around its principal vertical axis, a26long-press recalls the furniture back to the phone, and a double-tap fixes the posi-tion of the furniture in the environment.Pointer Grab: The grab button raycasts along the long edge of the phone intothe AR scene, and grabs and centers the hit furniture on top of the phone. If thereis already an object that the user is playing with, the user can release the furnitureby activating gravity on it, effectively dropping it to the ground, or can choose todelete or change it with another furniture.Color Palette: We also implemented a rectangular color palette which allowsthe user to explore available furniture colours with and visualize them immediatelyon the virtual furniture.3.3.2 Games - Archery & Fruit SlicerWe implemented a collection of games to showcase the robustness of 6-DOF track-ing for varied interactive experiences.Aiming and ShootingAiming requires small scale movements and demands high precision. We use thisapplication to show that PAIR is up to this task. There are three different fixedtargets in front of the user at different distances and angles, with the closes one 5meters away. The user draws the bow using an air-tap gesture on their other hand(recognized by the HoloLens). The user holds down the air tap to progressivelydraw the bow. When the user releases the hold, the arrow is released, firing thearrow with a power proportional to the hold time.Synchronized Screen: The bow is visualized on top of the phone, with the phoneserving as the handle. The phone screen visualizes the handle and it is synced tothe virtual bow. If an arrow is placed on the virtual bow and it is drawn, the phonescreen depicts these changes with animations.Haptic Feedback: As the bow string is drawn more, the phone increases its vi-bration strength to give a sense of heaviness and power, with a sharp rigid vibrationat release to give the feeling of the loosed arrow.Proximity Interaction & Explosives: There is a quiver in the ground with 6explosive arrows. The user can approach this quiver and if the phone comes close27to it, a ”quiver found!” notification screen pops up, with an option to pick up theexplosive arrows. With the explosive arrows, the user can proceed to blow up thetargets!Fruit SlicingSlicing fruits in mid-air with broad sweeping movements is a challenging task formarker-based tracking. We detect rapid motions (more than 5 cm between consec-utive ARKit tracking frames) as the start of a cut motion, visualizing the blade tipas a red dot for the next 2 seconds of motion. Due to our tracking implementation,coordinate realignment will not occur during this motion, ensuring a smooth blademotion based on visual-inertial odometry alone. Furthermore, small movementsare not processed as cuts, making this application a showcase of fast and vigoroustracking on a more coarse scale.Barbarian Sword and RapierWe can also render the sword as a full 3D model, with the phone serving as the hilt.Here, we give the user the option of waving around either a large barbarian swordor a rapier, selectable via the touchscreen.3.3.3 Celestial BodiesThis app allows the user to create, place and control a mini orbital system madeof the Sun, Earth and Moon. The user can place this system in the environmentthrough the Virtual 3D Trackpad technique together with pinch and rotate to scaleand orient it. A double-tap fixes the system in place. After the system is fixed, arotation gesture rotates only the Earth around its own axis.Custom Controllers: A long-tap opens the purpose-made circular lunar orbitslider mapped to lunar phases. The slider here controls the orbital motion of thevirtual Moon. Tapping on the Sun opens the Earth’s Solar orbit slider, which simi-larly controls Earth’s solar orbit.28Figure 3.12: Custom orbital controllers for the Earth and Moon.3.4 EvaluationWe performed an evaluation to assess the 3D positional accuracy of our trackingapproach. We conducted our evaluation in a Vicon-equipped motion tracking labwith a 4.5m×4.5m×2.0m tracking volume. For the study, we attached infrared-reflective markers to our iPhone 11 Pro Max and first-generation HoloLens, asshown in Figure 3.11. We calibrated the Vicon setup to have less than 2mm meanabsolute pose error.3.4.1 Conditions and ProcedureWe evaluated our system on nine different gestures and activities, drawn from ac-tions performed within our sample applications. Each action was performed withvariations in speed, phone position, and ARuco tag configuration. Activities werecontinuous actions: idling (standing still and moving the phone idly within a small50cm3 volume), walking (walking around the space while idly moving the phone),and walking while looking around (continually changing the head pose). Gestureswere discrete actions: crouching and standing repeatedly, swiping the phone acrosseach principal axis (left-right, up-down, in-out), moving the phone in a large ( 1m)circle and moving the phone in a figure-8 shape. As this was a technical evaluationof tracking accuracy, rather than a user evaluation, it was carried out by a memberof the research team.29Table 3.1: Gestures and activities in the studyAction Variations # of recordingsActivities (recorded for 30 seconds)Idle Markers 3Walk Markers 3Walk and Look Around None 1Gestures (recorded 5 times each)Crouching Markers, Speed 6Vertical Swipes Markers, Speed 6Horizontal Swipes Orientation, Speed 6In-and-Out Swipes Orientation, Speed 6Circle Speed 2Figure-8 Speed 2We recorded each activity for a 30 second duration. We repeated every gesture5 times, both at a “regular” speed (approximately 0.85 m/s) and a faster speed(approximately 1.30 m/s). Variations besides speed included the configuration ofmarkers on-screen (three configurations tested: full border, top and bottom rowsonly, and left and right columns only, as shown in Figure 3.2) and the orientationof the device (screen facing the ceiling, the left wall, or right wall, forcing markersto be only visible at certain times, positions and orientations within the gesture).Actions and their variations are summarized in Table 3.1. Not all variations appliedsensibly to all actions.Overall we collected 35 gesture and activity recordings with these differentconditions, each containing five repeats or 30 seconds of motion respectively. Onerecording was dropped due to equipment malfunction. The calibration and datacollection process took over 3 hours, including the time spent following the sanita-tion and prevention procedures as mandated by our institution and regional healthauthority.303.4.2 ResultsDue to clock drift and clock error between the HoloLens, phone and Vicon record-ing setup, and unknown relationship between the HoloLens and Vicon coordinatesystems, we first pre-processed the data by searching for the optimal temporal andspatial alignment between the HoloLens 3D positions as reported by the Vicon,and the HoloLens 3D positions as reported by the headset itself. We were able toidentify the unique optimal alignment (temporal offset and rigid transformation) towithin 3 cm mean absolute positional error in all but three of the 34 gesture record-ings; these three cases were dropped due to apparent HoloLens spatial trackingfailures (mean positional error greater than 7 cm).We computed three measures of error across all of the recorded phone positions(Vicon phone positions, ARKit reported positions, and positions corrected usingthe ARuco tag data):(1) “baseline” ARKit error, which is the error between an optimal rigid-transformalignment of the ARKit-reported positions and the true Vicon-reported phone posi-tions. This approach somewhat unrealistically corrects for drift effects and HoloLenstracking errors, but serves as a lower bound on the error experienced by users.(2) “ARKit-only” error, which estimates the error produced without using theARuco tags. This is computed by optimally aligning the first second of ARKit data- essentially, assuming that the initial alignment of the phone to the HoloLens isperfect - and then applying that initial alignment to the remainder of the ARKitdata.(3) “Final” error, which is the error between the ARuco-corrected positions asreported in real-time by our system (transformed from the HoloLens coordinatesystem to the Vicon coordinate system through the optimal spatial alignment), andthe true Vicon phone positions.A Shapiro-Wilk test performed across each measure did not detect strong non-normality (baseline p = 0.61, ARKit-only p = 0.12, final p = 0.32). Consequently,a one-way ANOVA was performed across the final error measures for each varia-tion to determine significance. Speed was found to be significant (F = 9.5, p =0.005), while marker position and orientation were not found to be significant(F = 0.585, p = 0.57 and F = 0.445, p = 0.65 respectively). Thus, only speed31Table 3.2: Results summarized by condition.Slow and fast refer to speed variations (gestures only); idle and walk refer to walk-ing behaviour (activities only). Results are shown in millimetres.Group n baseline ARKit-only Finalmean stdev mean stdev mean stdevtotal 31 96.58 46.96 192.33 78.92 143.73 45.64slow 13 89.92 28.28 176.50 43.02 136.82 31.73fast 12 132.40 39.32 243.75 72.18 179.11 36.83idle 3 17.50 4.97 43.84 14.43 80.96 5.41walk 3 61.21 22.51 203.78 51.86 94.91 20.00(slow/fast) is broken out in Table 3.2. Activities are also broken out for compari-son purposes. Fast motions produced significantly more error than slow motions,suggesting that latency plays a major role in the observed error.Averaged across all 31 recordings, Final errors were significantly lower thanthe ARKit-only errors, per a paired t-test, t = 4.13, p = 0.0002, indicating that theARuco correction significantly reduced the overall positional error.3.5 Discussion3.5.1 LatencyA major challenge with our current technique is latency. Latency affects the coordi-nate space alignment process, as well as producing visible lag during interactions.In our current system, latency comes from several sources. On the smartphone side,ARKit performs tracking using the camera, meaning that it needs to wait for a cam-era frame (running at 60 Hz), for an average of 8 ms of latency. ARKit itself needsto process the frame and produce a positional estimate; tests with a slow-motioncamera suggest that this latency is approximately 120 ms (including the cameralatency). The ARKit data is then sent over the WiFi network, which has a latencyup to 25 ms in one direction. On the headset side, in parallel, the front cameramust snap a picture of the scene (at 30 Hz, thus averaging 16 ms of latency), thenprocess the image using the ArUco toolkit (averaging 80 ms), and finally merge32the ArUco-detected phone position with an incoming ARKit frame. Although theparallelism between ArUco and ARKit mitigates overall latency, total latency isstill around 150 ms.There are several ways in which latency could be addressed. First, a more opti-mized tracker or faster headset hardware could speed up ArUco marker acquisition.Second, using a different wireless link technology, such as Bluetooth or BLE couldreduce communication delays due to lower latency. Finally, we could leverage pre-cise ”historical” head position, as reported by the HoloLens, to access positioningdata corresponding to ARKit timestamps and thereby mitigate the negative effectsof latency on the coordinate alignment scheme.3.5.2 DriftAs discussed previously, visual-inertial odometry is subject to drift due to error ac-cumulation from IMU integration and inaccuracies from monocular SLAM. Fur-thermore, we observed that the iPhone fell back to purely inertial odometry whenmoved very rapidly, presumably due to severe motion blur on the camera. Thisproduced large accumulated drifts which slowly ”settled” over time as ARKit reac-quired visual tracking. In our current system, therefore, users will not be able tomove the phone at very high speeds while retaining precise 6DOF tracking; ourfruit-slicing demo, for instance, is specifically designed to work with coarse 6DOFtracking. However, our system ”gracefully degrades” in the presence of drift er-rors - it continues to provide precise relative translation and absolute rotation in-put. This is evident in our experimental results, which demonstrated a 5cm aver-age reduction in absolute error, compared to an ARKit-only approach (19.2cm vs.14.4cm). The improvements are even more noticeable over longer tracking periodsdue to accumulating drift; for instance, across the “walking” conditions (30 secondrecordings), our approach resulted in a mean error of 9.4cm vs an error of 20.4cmin the ARKit-only approach.A second problem is increasing misalignment if the markers are not seen for along period of time. This can be mitigated by designing interactions which makeuse of the touchscreen; the user will then turn the touchscreen to face them and theheadset’s front camera.33Future hardware could also significantly reduce the drift issue. For instance, theiPad Pro now ships with LiDAR technology designed specifically for AR interactions[72],which is significantly more precise and stable than monocular SLAM. Such tech-nology is expected to eventually appear in smartphones.3.5.3 Markerless TrackingWe utilized an array of ArUco markers as they are small and easily recognized, canbe put anywhere on the screen and are fast to track. However, they take up valuablescreen real estate. An alternative approach would be to track the content displayedon the phone screen itself, e.g. by allowing the phone to stream its display contentover the TCP connection, or by having the headset locally simulate the phone’sdisplay, and having the headset track that image. We prototyped this approachusing Vuforia Engine’s state-of-the-art image tracking system with static screenimages [73]. The accuracy of the prototype is generally comparable to the ArUcosystem, but the latency is roughly doubled, making it suboptimal for interactiveuse. We thus believe that future improvements in processing speed and trackingalgorithms could eliminate the need for visible on-screen markers.34Chapter 4Library of Apollo: A VirtualLibrary Experience in your WebBrowser4.1 ImplementationWe have regarded the strongly expressed preference for physical shelf browsing[17–20] together with the long cultivated art of library classification systems [47] asthe guiding principles of our design process, and aimed to bring the physical libraryexperience online while making improvements to enhance the connectedness andnavigability of the served online collection.We have wanted from the start to test our implementation widely on as manyusers as possible, but because of the COVID-19 pandemic restrictions we did nothave the opportunity to show and test our application on HMDs, which were notwidely available to potential users and testers. Instead we developed our VR ap-plication for the web using Unity WebGL and deployed and tested it widely in thatway. We made certain tweaks for mouse and keyboard interactions on the browser,and we justify those in the following sections, still we have tried to stay true to the3D graphics and interactions that are natural to VR.35Figure 4.1: A wide-angle shot of the shelves.4.1.1 Designing to Faithfully Recreate Physical LibrariesWe had two conflicting design interests. First, we wanted to design and deploy avirtual library application at scale available to everyone on web browsers. Second,we also wanted this to be as close to a physical library experience as possible.Naı¨vely optimizing for the latter would have meant building an online replica of ahuge library in 3D, that would then be navigated by mouse and keyboard throughsome combination of user motion and teleportation. However, that would havebeen an alien set of interactions to most people, be hard to navigate and possiblyeven clunky.Instead we extracted the most crucial part of the physical book browsing expe-rience from the physical library: the shelves. The user, essentially the camera, isplaced in front of a set of floating shelves, and is allowed 20 degrees of freedomof camera motion left and right, and 10 degrees up and down, achievable with amouse or a trackpad. A wide-angle view of the shelves is seen on Figure 4.1. Theuser can “scroll” the shelves left and right as slow or as fast as they want via theregular scroll gesture on their trackpads, scroll-wheels on mouses or arrow-keys onkeyboards (Video Figure).Books on the virtual shelves are arranged in order according to their LCC clas-sifications, just as they would be in a real library. There are also panels showingLCC hierarchies above shelves indicating where the user is within the library. Thepanels in the center indicate the user’s current location, while the left and rightpanels respectively indicate the previous and next classifications, and thus the con-36Figure 4.2: Search results for ‘a tale of two cities’, showing both shelf andbook results. A: Shelf results list LCC classes that are search-hits, whilebook results are hits for titles, authors and subject headings of books.The terms that led to the search-hits are highlighted. B: Hovering overthe ‘Under Shelf’ portion of book results expands the shelf hierarchy.tent of the shelves that await the user in those directions. A click on the left orright panels scroll the shelves to bring those classes in front of the user. A clickon the center panels opens the LCC classification hierarchy as seen on Figure 4.4.This list can be navigated by up and down scrolling as well as clicks to expand andcollapse it. Clicking on the end portion of these hierarchies brings that class & itsassociated books in front of the user.The user can start typing anytime they want to bring up the search bar, whichcan be used to search over titles, authors, subjects and LCC classifications. If anybook or classification is selected through the search results, the user is transportedto the shelf containing this selected book. The books are also sized accordingto their real dimensions, page numbers and volumes, as seen on Figure 4.1. Wehave not provided custom textures or covers for books, but used generic textures asstoring and serving textures for over nine million books in a fast browsed librarywould have created an extra set of architectural challenges.4.1.2 Building a Familiar and Expressive Search BarSingle input search bars are ubiquitous and users are intimately familiar with them.We utilized the search bar as an entry point into our library by designing it as acatch-all input system that searches over book titles, authors, subject headings ofbooks, and LCC categories. Our search provides a way for users to enter into the37Figure 4.3: A: Clicks on shelved books bring up synoptic book panels. Theelements in the subject heading table are clickable. B: The search-results after the “church architecture” subject heading was clicked. Thehits on subject headings are highlighted with shades of green.hierarchically organized stacks by helping them first with identifying a relevantpool of items they might be interested in.Figure 4.2 shows shelf (class) and book search hits for the query ‘A tale oftwo cities.’ Shelf results show the LCC hierarchy from top to bottom, togetherwith the class numbers and the number of books that are listed under that class.Book results list titles, authors and publishers, together with a table showing thesubject headings listed for each book. To the right of each book result, you can alsosee under which shelf (class) that book can be found, hovering over that sectionexpands the shelf hierarchy as can be seen on the right side of Figure 4.2. The‘under shelf’ portions are always colored to clearly designate shelf hierarchies.The search results are not paginated, and can be “infinitely” scrolled downwardsuntil the end of results.4.1.3 Engineering a Smooth Browsing ExperienceClicking on a search result, both for books and shelves, zooms the user into a setof floating shelves (Figure 4.1). The searched-for book appears on the centre shelfand is briefly highlighted as an indicator of its position. The books on shelves con-form to LCC sorting and appear as they would on a regular library. One other keyfeature we have added is browsing over the LCC classification hierarchy as seenon Figure 4.4. This scroll-view of classification hierarchies is opened by clickingon the centered LCC panels. A listed hierarchy can be expanded or collapsed by38Figure 4.4: A: The scroll-view of class hierarchies. Hierarchies can be col-lapsed by clicking any of the intermediate elements. B: The collapsedhierarchies after the click at A. The hierarchies can be scrolled up anddown, or expanded by clicking their tag panels if they contain child cat-egories.clicking on different parts of the hierarchy. Clicking on any panel within a hierar-chy collapses that hierarchy down to that panel, reducing the number of hierarchiesin the scroll-view. Clicking on the name-tag panel of a hierarchy expands that hier-archy. Clicking on the last panel of a hierarchy transports the user to the beginningof that hierarchical class with the first book of that class highlighted on shelves.Another addition to the browsing experience is through the improved connec-tivity that comes through search over subject headings of books. Book clicks bringup synoptic windows as seen on part 1 of Figure 4.3. The colored subject tablesare filled with LCSH which we have populated for every book from a dataset main-tained by domain experts at the Library of Congress [47]. These LCSH headingsare clickable and trigger a search over the entire dataset’s indexed LCSH fields.Part 2 of Figure 4.3 shows search results for an LCSH field search; notice thatthese books can come from entirely different shelves and classes, often very distantfrom each other, thus providing another way to jump between classes and books.This allows users to transcend the distance imposed by LCC in physical librariesthrough a subject-based search method, which allows for dynamic links betweenbooks. We believe this feature would go a long way to satisfy the expressed desireby users to occasionally see distant books [18].39(a) Digitized books. (b) Cloud architecture.Figure 4.5: A: The “Peek Inside” button on book panels opens an e-readerthat uses OpenLibrary’s digitized book archive. B: The cloud architec-ture of Library of Apollo.4.1.4 Providing Access to Digitized and Physical BooksAnother important feature of real world libraries is in-depth browsing of individualbooks. A library-goer can just pick up any book and read until their curiosity issated. In order to provide a similar experience, when a user clicks on a shelvedbook, we provide a single page overview. Part 1 of Figure 4.3 shows an exampleof this page which displays the book’s title, authorship, publishing house, coloredsubject headings, physical information regarding extent and dimensions, and linksto lookup the same book on Amazon and Goodreads, as well as a Peek Inside buttonthat is enabled when there is a free & digitized version available on OpenLibrary(which houses over 3 million books, or around 30% of the total collection). A sin-gle click on this button opens a new tab with an online reader showing the contentsof the book, as seen on part A of Figure 4.5, providing an in-depth reading expe-rience. The Amazon and Goodreads buttons provide easy access to purchase andsocial reading options, respectively.404.2 Online Deployment and Survey Results4.2.1 The Front-end and the Cloud ArchitectureWe have developed the front-end of our application using Unity WebGL. The com-piled WebAssembly, Javascript and HTML files are hosted on the AWS S3 servicethrough the CDN service of AWS, Cloudfront. The books are sorted according toLCC sorting scheme [47, 48] and clustered into JSON files that contain 500 bookseach, gzipped and stored in S3. These data files are fetched when the user is brows-ing the shelves, and pre-fetched during scrolling when the user is near the end of acluster.The same data that is stored in S3 is indexed on AWS Cloudsearch to powerthe search functionalities of our app. All search requests are performed through theREST API we have developed on AWS API Gateway. Search results are populatedthrough the data returned from Cloudsearch. For analytics, user click-data is alsorecorded on DynamoDB through the same gateway. The search requests to Open-Library’s digitized books dataset are made in a similar fashion. Our architecture issummarized on Figure 4.5.4.2.2 The Deployed DatasetCourtesy of the Library of Congress [74], through their MARC Open-Access pro-gram, we were able to put together 9.4m books, complete with their LCSH andLCC data, distributed over 200k LCC classes. We pre-processed this data to serveour specific needs, and stored and indexed them for use within our virtual libraryapplication as described above. Despite the size of the dataset, the website is verysmooth to use and inexpensive to operate - monthly operational costs are less than$40 USD.4.2.3 Deployment and RecruitmentWe deployed our library at a publicly accessible address, loapollo.com, and seededlinks to the project via Reddit and word-of-mouth. Over a span of two weeks, overtwo hundred unique users visited the site. Each user to the library was assigned arandomly-generated persistent user ID, to track repeated session visits, as well as41a session ID for any given visit. The Library front-end automatically logged clickinteractions with the system in a DynamoDB server for later introspection.Of the 224 unique visitors, 21 (9.4%) visited more than once, with one uservisiting the site a total of five times over the course of two weeks. Users interactedfor an average of 162 seconds, and produced an average of 8 click interactions;notably, however, both distributions are particularly long-tailed. While some usersonly visited for very brief moments (often just a single search, followed presum-ably by some scrolling), 12 users used it for over ten minutes each, and likewise 20users generated over 20 click events each.4.2.4 SurveyWhen users clicked on any book, a survey link was displayed in the upper-rightcorner, ensuring that users interacted with the system to a minimal extent priorto taking the survey. Users who accessed the link were invited to fill out a shortsurvey with two major parts: a first section which asked about their existing (pre-and post-pandemic) book browsing habits, and another section with which askedabout their experiences with the library. Finally, users could provide open-endedfeedback about their experience with the Library. The full survey design can befound in the Supplemental Materials.We received a total of 46 survey responses. Two responses were discarded:one filled out ‘1’s for every single question, and another was a clear duplicatesubmission. Participants reported reading an average of 13.4 books per year (SD15.2; participants reported anywhere from 1 book per year to 80 books per year).In the first part of the survey (Figure 4.6), the participants did not report sig-nificantly changing their reading habits (mean 2.9), but evidently had a signifi-cant shift in book-browsing habits due to the pandemic. Library usage decreasedsignificantly (pre-pandemic mean 2.36, post-pandemic mean 1.54), as did book-shop usage (pre-pandemic mean 2.98, post-pandemic mean 1.45), whereas onlinebrowsing significantly increased (pre-pandemic mean 3.14, post-pandemic mean3.86).In the second part of the survey (Figure 4.6), the users generally expressed astrong preference for the Library, with over 80% (36/44) users being “somewhat42(a) Survey results on reading habits. (b) Survey results on library features.Figure 4.6: A: The change in reading habits due to COVID-19 pandemic. B:The user perception of the library’s navigability, ease of use, browsingquality and overall quality.likely” or more to come back, and over 90% of users (41/44) being “somewhatlikely” or more to recommend this to a friend. Users generally felt that it did repli-cate the feel of real libraries and bookshops to some extent, with over half (25/44)noting that it felt similar or very similar. Similarly, around half of participantsfound it “easier” or “much easier” to find books compared to bookshops, libraries(23/44) and online destinations (21/44), and felt that it contained “more” or “signif-icantly more” new and interesting books compared to bookshops, libraries (23/44)and online destinations (24/44). Overall, survey respondents were very positiveabout the library and its major features.4.3 DiscussionThrough our online deployment and survey, we were able to gather valuable feed-back from readers and book browsers. In the open feedback column, we had severaluseful insights generated by users.User Interface: Three users wished for a mobile version in the feedback, withone noting “Mobile version must be created and [published] on App stores im-mediately”, indicating the modern importance of smartphone-friendly interfaces.Indeed, although our interface does work on some mobile browsers, it does noton many older browsers due to a lack of WebGL support. We expect this limita-tion will improve in the future as newer devices generally do support WebGL. Two43users commented that they would have liked to change the colour scheme: one userremarked that it was too dark, while another user wanted a “night mode” to makeit even darker.Search Performance: Our project primarily focused on book-browsing, ratherthan search, and consequently our search feature is much simpler than e.g. Googleor Amazon’s search functionality. Users commented that this could be improved.Four users noted that search could be improved: providing better search capabili-ties (such as advanced search by author/title/subject fields), improving the presen-tation of results (e.g. moving favorite or recommended books to the top), improv-ing the discoverability of category or tag search, and ordering works by popularityor relevance. Recommendations, in particular, are interesting as they must func-tion differently in a public library setting (with limited or no access to prior userdata) compared with companies like Amazon which have vast access to prior userpreferences via tracking and search history.Browsing Capabilities Users generally praised the browsing and search fea-tures of the Library. One user noted that they were able to find “5 books on musictheory in 5 minutes”, while another noted that “It has everything I am looking for”.Users also had suggestions for improving the experience further: one commentedthat they would have liked even more physicality in the form of different room-s/areas to browse around in, while another suggested a downloadable version fororganizing their own books.44Chapter 5Conclusion and Future WorkIn this thesis, we studied the use of two different systems that are familiar andmuch-loved in the real-world, smartphones and libraries, within the context ofAR/VR interaction and world design. Individually, we developed a system whichenables an unmodified smartphone to act as a flexible controller and companiondevice to an unmodified AR-HMD, and a virtual-reality library implemented asa web application and designed to support library-like browsing and discovery.With the first work, PAIR, we demonstrated the versatility and the ease-of-use of asmartphone as an AR-HMD controller. With the second work, we demonstratedthat bringing well-known, familiar real-world environments into virtual spacesthrough a skeumorphic design can create immersive and interactive VR experi-ences, through the specific example of a virtual library.Our approach in PAIR unlocks a wide range of smartphone capabilities for usein immersive augmented reality, including 6DOF spatial input, precise touch input,high resolution screen, haptic vibration feedback and spatialized audio. We devel-oped a range of interactions and applications made possible by PAIR, demonstrat-ing the immediate feasibility of our approach. The positional tracking accuracystudy also showed that our system is superior to ARKit-only tracking and helpswith drift correction, especially over longer use durations. In the future, with pan-demic restrictions subsided, we would like to conduct user studies to investigatethe user affinity for our approach and interaction design. Another future improve-ment could be testing our system on new LIDAR enabled smartphones which can45perform more accurate inside-out tracking with reduced drift over time.Our Library of Apollo system was deployed publicly and attracted over twohundred visitors and forty four survey responses; our survey results suggested highuser affinity for the book-browsing experience and library capabilities. Throughthe combination of features that we designed, we have built a virtual library whichlends itself to serendipitous browsing and discoveries across a large, connectedbook dataset. Open-ended survey answers include feature requests ranging frommobile app versions to more spatial features like rooms and positional navigationaround shelves. As we wanted to perform public user affinity tests during thetime of COVID-19 we proceeded with a web-deployment that would be easilyaccessible by everyone everywhere but for future work, we would like to test outour system on a VR-HMD, together with different design elements that could bemore suitable for an HMD platform like positional navigation and individual roomsfor categories.For future work, one might want to target the familiar interfaces of conventionalcomputing with familiar places and interactions of the real world for a fusion intoAR and VR. For example, can we merge a Google Search with an art gallery? Thesearch results for image and video could hang on walls, locations could turn intominiature 3D models standing in the room and text results could be sorted into easyto browse shelves. One might want to further the smartphone and AR HMD fusionwith smartwatches. Such combinations could further boost the tracking accuracyand can be combined with gesture and gaze to create interaction techniques thatare natural, precise and intuitive.46Bibliography[1] Alan Wexelblat. Virtual reality: applications and explorations. AcademicPress, 2014. → page 1[2] Frank Biocca and Mark R Levy. Communication in the age of virtual reality.Routledge, 2013. → page 1[3] William R Sherman and Alan B Craig. Understanding virtual reality. SanFrancisco, CA: Morgan Kauffman, 2003. → page 2[4] Phil Turner and G Van De Walle. Familiarity as a basis of universal design.Journal of Gerontechnology, 5(3):150–159, 2006. → pages 3, 7[5] Jef Raskin. Intuitive equals familiar. Communications of the ACM,37(9):17–19, 1994. → pages 3, 7[6] Annemarie Quispel, Alfons Maes, and Joost Schilperoord. Graph and chartaesthetics for experts and laymen in design: The role of familiarity andperceived ease of use. Information Visualization, 15(3):238–252, 2016. →pages 3, 7[7] Simon Lawry, Vesna Popovic, Alethea Blackler, and Helen Thompson. Age,familiarity, and intuitive use: An empirical investigation. Appliedergonomics, 74:74–84, 2019. → pages 3, 7[8] Minji Cho, Soyoung Kwon, Nooree Na, Hyeon-Jeong Suk, and KunPyoLee. The elders preference for skeuomorphism as app icon style. InProceedings of the 33rd Annual ACM Conference Extended Abstracts onHuman Factors in Computing Systems, pages 899–904, 2015. → pages 3, 8[9] Telefonaktiebolaget LM Ericsson. Ericsson mobility visualizer, 2021. →pages 3, 847[10] Antti Oulasvirta, Tye Rattenbury, Lingyi Ma, and Eeva Raita. Habits makesmartphone use more pervasive. Personal and Ubiquitous Computing,16(1):105–114, 2012. → pages 3, 8[11] Fabio Bruno, Stefano Bruno, Giovanna De Sensi, Maria-Laura Luchi,Stefania Mancuso, and Maurizio Muzzupappa. From 3d reconstruction tovirtual reality: A complete methodology for digital archaeologicalexhibition. Journal of Cultural Heritage, 11(1):42–49, 2010. → page 4[12] Rafal Wojciechowski, Krzysztof Walczak, Martin White, and WojciechCellary. Building virtual and augmented reality museum exhibitions. InProceedings of the ninth international conference on 3D Web technology,pages 135–144, 2004. → page 4[13] Jeff Johnson, Teresa L. Roberts, William Verplank, David Canfield Smith,Charles H. Irby, Marian Beard, and Kevin Mackey. The xerox star: Aretrospective. Computer, 22(9):11–26, 1989. → page 7[14] Gilles Fauconnier and Mark Turner. The way we think: Conceptual blendingand the mind’s hidden complexities. Basic Books, 2008. → page 7[15] George Lakoff and Mark Johnson. Metaphors we live by. University ofChicago press, 2008. → page 7[16] Brendan Keogh. Poke´mon go, the novelty of nostalgia, and the ubiquity ofthe smartphone. Mobile Media & Communication, 5(1):38–41, 2017. →page 8[17] Dana McKay. Gotta keep’em separated: Why the single search box may notbe right for libraries. In Proceedings of the 12th Annual Conference of theNew Zealand chapter of the ACM special interest group on computer-humaninteraction, pages 109–112, 2011. → pages 8, 11, 35[18] Annika Hinze, Dana McKay, Nicholas Vanderschantz, Claire Timpany, andSally Jo Cunningham. Book selection behavior in the physical library:implications for ebook collections. In Proceedings of the 12thACM/IEEE-CS joint conference on Digital Libraries, pages 305–314, 2012.→ page 39[19] Stephann Makri, Ann Blandford, Jeremy Gow, Jon Rimmer, ClaireWarwick, and George Buchanan. A library or just another informationresource? a case study of users’ mental models of traditional and digitallibraries. Journal of the American Society for Information Science andTechnology, 58(3):433–445, 2007. → page 1148[20] Hanna Stelmaszewska and Ann Blandford. From physical to digital: a casestudy of computer scientists’ behaviour in physical libraries. InternationalJournal on Digital Libraries, 4(2):82–92, 2004. → pages 8, 11, 35[21] S. Feiner, B. MacIntyre, T. Hollerer, and A. Webster. A touring machine:prototyping 3d mobile augmented reality systems for exploring the urbanenvironment. In Digest of Papers. First International Symposium onWearable Computers, pages 74–81, 1997. → page 8[22] R. Budhiraja, G. A. Lee, and M. Billinghurst. Interaction techniques forhmd-hhd hybrid ar systems. In 2013 IEEE International Symposium onMixed and Augmented Reality (ISMAR), pages 243–244, 2013. → page 8[23] R. Budhiraja, G. A. Lee, and M. Billinghurst. Using a hhd with a hmd formobile ar interaction. In 2013 IEEE International Symposium on Mixed andAugmented Reality (ISMAR), pages 1–6, 2013. → page 8[24] Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg.Multifi: Multi fidelity interaction with displays on and around the body. InProceedings of the 33rd Annual ACM Conference on Human Factors inComputing Systems, CHI ’15, page 3933–3942, New York, NY, USA, 2015.Association for Computing Machinery. → page 8[25] E. Normand and M. J. McGuffin. Enlarging a smartphone with ar to create ahandheld vesad (virtually extended screen-aligned display). In 2018 IEEEInternational Symposium on Mixed and Augmented Reality (ISMAR), pages123–133, 2018. → page 8[26] Tomas Dorta, Gokce Kinayoglu, and Michael Hoffmann. Hyve-3d: A newembodied interface for immersive collaborative 3d sketching. In ACMSIGGRAPH 2014 Studio, SIGGRAPH ’14, New York, NY, USA, 2014.Association for Computing Machinery. → page 8[27] Toma´s Dorta, Gokce Kinayoglu, and Michael Hoffmann. Hyve-3d and the3d cursor: Architectural co-design with freedom in virtual reality.International Journal of Architectural Computing, 14(2):87–102, 2016.[28] A. Millette and M. J. McGuffin. Dualcad: Integrating augmented realitywith a desktop gui and smartphone interaction. In 2016 IEEE InternationalSymposium on Mixed and Augmented Reality (ISMAR-Adjunct), pages21–26, Sep. 2016.49[29] Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel.Tabletinvr: Exploring the design space for using a multi-touch tablet invirtual reality. In Proceedings of the 2019 CHI Conference on HumanFactors in Computing Systems, CHI ’19, New York, NY, USA, 2019.Association for Computing Machinery. → page 8[30] Marcos Serrano, Barrett Ens, Xing-Dong Yang, and Pourang Irani. Gluey:Developing a head-worn display interface to unify the interaction experiencein distributed display environments. In Proceedings of the 17th InternationalConference on Human-Computer Interaction with Mobile Devices andServices, MobileHCI ’15, page 161–171, New York, NY, USA, 2015.Association for Computing Machinery. → page 8[31] Fengyuan Zhu and Tovi Grossman. Bishare: Exploring bidirectionalinteractions between smartphones and head-mounted augmented reality. InProceedings of the 2020 CHI Conference on Human Factors in ComputingSystems, CHI ’20, page 1–14, New York, NY, USA, 2020. Association forComputing Machinery. → page 9[32] Apple Inc. Arkit - augmented reality, 2019. → pages 9, 13, 17[33] Teo Babic, Harald Reiterer, and Michael Haller. Pocket6: A 6dof controllerbased on a simple smartphone application. In Proceedings of the Symposiumon Spatial User Interaction, SUI ’18, page 2–10, New York, NY, USA,2018. Association for Computing Machinery. → page 9[34] Chi-Jung Lee and Hung-Kuo Chu. Dual-mr: Interaction with mixed realityusing smartphones. In Proceedings of the 24th ACM Symposium on VirtualReality Software and Technology, VRST ’18, New York, NY, USA, 2018.Association for Computing Machinery. → page 9[35] Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, DieterSchmalstieg, and Denis Kalkofen. Trackcap: Enabling smartphones for 3dinteraction on mobile head-mounted displays. In Proceedings of the 2019CHI Conference on Human Factors in Computing Systems, CHI ’19, NewYork, NY, USA, 2019. Association for Computing Machinery. → page 9[36] Doug A. Bowman, Ernst Kruijff, Joseph J. LaViola, and Ivan Poupyrev. 3DUser Interfaces: Theory and Practice. Addison Wesley LongmanPublishing Co., Inc., USA, 2004. → page 9[37] Teresa Hirzle, Jan Gugenheimer, Florian Geiselhart, Andreas Bulling, andEnrico Rukzio. A design space for gaze interaction on head-mounted50displays. In Proceedings of the 2019 CHI Conference on Human Factors inComputing Systems, CHI ’19, New York, NY, USA, 2019. Association forComputing Machinery. → page 9[38] Mark Billinghurst, Adrian Clark, and Gun Lee. A survey of augmentedreality. Found. Trends Hum.-Comput. Interact., 8(2–3):73–272, March 2015.→ page 9[39] Thammathip Piumsomboon, Adrian Clark, Mark Billinghurst, and AndyCockburn. User-defined gestures for augmented reality. In CHI ’13Extended Abstracts on Human Factors in Computing Systems, CHI EA ’13,page 955–960, New York, NY, USA, 2013. Association for ComputingMachinery. → page 9[40] Microsoft Corp. Voice input, 2019. → page 9[41] Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, and HrvojeBenko. Mrtouch: Adding touch input to head-mounted mixed reality. IEEETransactions on Visualization and Computer Graphics, 24(4):1653–1660,April 2018. → page 9[42] Bruce H. Thomas and Wayne Piekarski. Glove based user interactiontechniques for augmented reality in an outdoor environment. Virtual Reality,6:167–180, 2002. → page 10[43] T. Ha and W. Woo. Arwand: Phone-based 3d object manipulation inaugmented reality environment. In 2011 International Symposium onUbiquitous Virtual Reality, pages 44–47, 2011. → page 10[44] Juan David Hincapie´-Ramos, Kasim Ozacar, Pourang P. Irani, andYoshifumi Kitamura. Gyrowand: Imu-based raycasting for augmentedreality head-mounted displays. In Proceedings of the 3rd ACM Symposiumon Spatial User Interaction, SUI ’15, page 89–98, New York, NY, USA,2015. Association for Computing Machinery. → page 10[45] Elizabeth B Cooksey. Too important to be left to chance—serendipity andthe digital library. Science & technology libraries, 25(1-2):23–32, 2004. →page 10[46] Jim LeBlanc. Classification and shelflisting as value added: Some remarkson the relative worth and price of predictability, serendipity, and depth ofaccess. Library resources & technical services, 39(3):294–302, 1995. →page 1051[47] Lois Mai Chan, Sheila S Intner, and Jean Weihs. Guide to the Library ofCongress classification. ABC-CLIO, 2016. → pages 10, 35, 39, 41[48] Library of Congress. Library of congress classification.https://www.loc.gov/catdir/cpso/lcc.html. Accessed: 2021-04-06. → pages10, 41[49] Geoffrey C Bowker and Susan Leigh Star. Sorting things out: Classificationand its consequences. MIT press, 2000. → page 10[50] Elaine Svenonius. The intellectual foundation of information organization.MIT press, 2000. → page 10[51] Dana McKay, Wally Smith, and Shanton Chang. Lend me some sugar:Borrowing rates of neighbouring books as evidence for browsing. InIEEE/ACM Joint Conference on Digital Libraries, pages 145–154. IEEE,2014. → page 11[52] Dana McKay, Shanton Chang, and Wally Smith. Manoeuvres in the dark:Design implications of the physical mechanics of library shelf browsing. InProceedings of the 2017 Conference on Conference Human InformationInteraction and Retrieval, pages 47–56, 2017. → page 11[53] Shiao-Feng Su. Dialogue with an opac: How visionary was swanson in1964? The Library Quarterly, 64(2):130–161, 1994. → page 11[54] Amanda Spink, Dietmar Wolfram, Major BJ Jansen, and Tefko Saracevic.Searching the web: The public and their queries. Journal of the Americansociety for information science and technology, 52(3):226–234, 2001. →page 11[55] Micheline Hancock-Beaulieu. Evaluating the impact of an online librarycatalogue on subject searching behaviour at the catalogue and t the shelves.Journal of documentation, 1990. → page 11[56] Jonathan L Herlocker, Joseph A Konstan, Loren G Terveen, and John TRiedl. Evaluating collaborative filtering recommender systems. ACMTransactions on Information Systems (TOIS), 22(1):5–53, 2004. → page 11[57] Alice Thudt, Uta Hinrichs, and Sheelagh Carpendale. The bohemianbookshelf: supporting serendipitous book discoveries through informationvisualization. In Proceedings of the SIGCHI Conference on human factorsin computing systems, pages 1461–1470, 2012. → page 1152[58] Eike Kleiner, Roman Ra¨dle, and Harald Reiterer. Blended shelf:Reality-based presentation and exploration of library collections. In CHI ’13Extended Abstracts on Human Factors in Computing Systems, CHI EA ’13,page 577–582, New York, NY, USA, 2013. Association for ComputingMachinery. → pages 11, 12[59] Pierre Cubaud, Pascal Stokowski, and Alexandre Topol. Binding browsingand reading activities in a 3d digital library. In Proceedings of the 2ndACM/IEEE-CS joint conference on Digital libraries, pages 281–282, 2002.→ page 12[60] Pierre Cubaud, Claire Thiria, and Alexandre Topol. Experimenting a 3dinterface for the access to a digital library. In Proceedings of the third ACMconference on Digital libraries, pages 281–382, 1998. → page 12[61] Rodrigo Almeida, Pierre Cubaud, Je´roˆme Dupire, Ste´phane Natkin, andAlexandre Topol. Experiments towards 3d immersive interaction for digitallibraries. In International Conference on Technologies for E-Learning andDigital Entertainment, pages 1348–1357. Springer, 2006. → page 12[62] Ilhan Aslan, Martin Murer, Florian Primessnig, Christiane Moser, andManfred Tscheligi. The digital bookshelf: decorating with collections ofdigital books. In Proceedings of the 2013 ACM conference on Pervasive andubiquitous computing adjunct publication, pages 777–784, 2013. → page 12[63] Karen Detken, Carlos Martinez, and Andreas Schrader. The search wall:tangible information searching for children in public libraries. InProceedings of the 3rd International Conference on Tangible and EmbeddedInteraction, pages 289–296, 2009. → page 12[64] Charles-Antoine Julien, Catherine Guastavino, and France Bouthillier.Capitalizing on information organization and information visualization for anew-generation catalogue. Library Trends, 61(1):149–161, 2012. → page 12[65] Matti Pouke, Johanna Ylipulli, Ilya Minyaev, Minna Pakanen, PaulaAlavesa, Toni Alatalo, and Timo Ojala. Virtual library: Blending mirror andfantasy layers into a vr interface for a public library. In Proceedings of the17th International Conference on Mobile and Ubiquitous Multimedia, pages227–231, 2018. → page 12[66] Matt Cook. Virtual serendipity: preserving embodied browsing activity inthe 21st century research library. The Journal of Academic Librarianship,44(1):145–149, 2018. → page 1253[67] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira,I. Reid, and J.J. Leonard. Past, present, and future of simultaneouslocalization and mapping: Towards the robust-perception age. IEEETransactions on Robotics, 32(6):1309–1332, 2016. → page 13[68] Sergio Garrido-Jurado, Rafael Mun˜oz-Salinas, Francisco Madrid-Cuevas,and Rafael Medina-Carnicer. Generation of fiducial marker dictionariesusing mixed integer linear programming. Pattern Recognition, 51, 10 2015.→ pages 13, 15[69] Microsoft Corp. Locatable camera, 2019. → page 15[70] G. Bradski. The OpenCV Library. Dr. Dobb’s Journal of Software Tools,2000. → page 15[71] Francisco Romero-Ramirez, Rafael Mun˜oz-Salinas, and RafaelMedina-Carnicer. Speeded up detection of squared fiducial markers. Imageand Vision Computing, 76, 06 2018. → page 15[72] Apple Inc. Visualizing and interacting with a reconstructed scene, 2020. →page 34[73] PTC Inc. Image targets with vuforia engine, 2021. → page 34[74] Library of Congress. Marc open-access, marc distribution services,mdsconnect, 2021. data retrieved from Library of Congress MARCDistribution Services, https://www.loc.gov/cds/products/marcDist.php. →page 4154Appendix ASupporting MaterialsFigure A.1: Part 1 of survey questions for Library of Apollo55Figure A.2: Rest of the Part 1 of survey questions for Library of Apollo56Figure A.3: Part 2 of survey questions for Library of Apollo57Figure A.4: Part 3 of survey questions for Library of Apollo58Figure A.5: Part 4 of survey questions for Library of Apollo59
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Xiao, Robert
Designing familiar augmented and virtual reality environments and interactions through off-the-shelf real-world solutions
Text