Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Building large sets of haptic icons : rhythm as a design parameter, and between-subjects MDS for evaluation Ternes, David Richard 2007

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2007-0610.pdf [ 16.18MB ]
Metadata
JSON: 831-1.0052080.json
JSON-LD: 831-1.0052080-ld.json
RDF/XML (Pretty): 831-1.0052080-rdf.xml
RDF/JSON: 831-1.0052080-rdf.json
Turtle: 831-1.0052080-turtle.txt
N-Triples: 831-1.0052080-rdf-ntriples.txt
Original Record: 831-1.0052080-source.json
Full Text
831-1.0052080-fulltext.txt
Citation
831-1.0052080.ris

Full Text

Building Large Sets of Hap tic Icons: rhythm as a design parameter, and between-subjects MDS for evaluation by David Richard Ternes B . S c , University of British Columbia, 2005 A THESIS S U B M I T T E D IN P A R T I A L F U L F I L L M E N T O F T H E R E Q U I R E M E N T S F O R T H E D E G R E E O F Master of Science in The Faculty of Graduate Studies (Computer Science) The University of British Columbia August 2007 © D a v i d R ichard Ternes 2007 Abstract: Haptic icons (brief, tactile stimuli with associated meanings) are a useful new way to convey information through the modality of touch, but they are difficult to create because of our lack of understanding into what makes good haptic stimuli and how people wi l l perceive them. This thesis aims to enlarge our capabilities to design and evaluate haptic icons, despite these problems. We seek to do this via two overlapping threads of research. In the first thread, we introduce the design parameter of rhythm as a means of extending the expressive capabilities of the simple tactile stimuli used in haptic icons. This allows us to create a set of expressive and perceptually distinguishable haptic stimuli larger by almost an order of magnitude than any previously created. In the second thread of research, we tackle the problem of how to evaluate the perceptual characteristics of such a large set of stimuli with real people. We develop a means of evaluation .that allows us to collect perceived difference data by present each user with only a subset of the total stimulus collection, and then stitch together an aggregate picture of how the stimuli are perceived via data collected from overlapping subsets from different users. To advance these two threads of research, two user studies are run in order to examine how our haptic stimulus set is perceived and to validate our new method of gathering perceptual difference data. One study uses an established but cumbersome technique to study our stimulus set, and finds that haptic rhythms are perceived according to several different aspects of rhythm, and that users can consistently differentiate between haptic stimuli along these aspects. The second study uses our newly developed data collection method to study the same stimulus set, and we find that the new technique produces results that show no significant difference from the established technique, but using a data collection task that is much quicker and less arduous for users to perform. We conclude by recommending the use of our new haptic stimulus set and evaluation technique as a powerful and viable means of extending the use of haptic icons to larger sets. i i Table of Contents Abstract: i i Table of Contents i i i List of Tables vi List of Figures vi i Acknowledgments vi i i Chapter 1: Introduction 1 1.1 Motive 2 1.1.1 Icons vs. Stimuli 4 1.2 Overview and Approach 4 Chapter 2: Related Work 8 2.1 Abstract Tactile Communication 8 2.1.1 Haptic Icons 8 2.1.2 Tactons 9 2.1.3 Other Vibro-tactile Work...". 9 2.2 Multidimensional Scaling 10 2.2.1 Comparison of M D S Results 12 Chapter 3: Creation of Haptic Stimuli 14 3.1 Description of Possible Stimulus Space 14 3.2 Sensory- and Hardware-Specific Limitations on Rhythm Space 16 3.2.1 High-level Limitations 17 3.2.2 Shortest Note 17 3.2.3 Selection of Different Note Types 18 3.3 Description of Stimulus Set 19 3.3.1 Heuristic One: Quarter Notes 20 3.3.2 Heuristic Two: Long Notes 20 3.3.3 Heuristic Three: Long and Quarter Notes 21 3.3.4 Heuristic Four: Substituting Quarter with Eighth Notes 21 3.3.5 Complete Stimulus Set Used 21 3.4 The Space Untested 22 i i i 3.4.1 Unused Rhythms Possible Given Hardware and Sensory Limitations 22 Chapter 4: Subset Data Gathering Methodology for M D S ; 25 4.1 Other Methods for Dealing with Large Set Sizes 26 4.1.1 Incomplete Dissimilarity Matrices 27 4.1.2 Sorting Tasks 28 4.1.3 Per-stimulus Judgment Tasks 29 4.2 Design of Proposed Subset Data Gathering Method 30 4.2.1 Creation of Subsets 31 4.2.2 Robustness and Scalability 37 4.3 Potential Threats to Validity of Method 42 4.3.1 Incomplete Individual Results 42 4.3.2 Subset-relative Judgments 43 4.3.3 Abi l i ty to Discover Overall Perceptual Trends 44 4.4 Pilot Study: Initial Study on Voicecoi l Vibrators 44 4.4.1 Apparatus '. 44 4.4.2. Participants 45 4.4.3 Stimulus Set 45 4.4.4 Procedure 45 4.4.5 Results and Discussion 46 Chapter 5: Methods 49 5.1 Discussion of Hardware Platform 49 5.1.1 Control of Haptic Feedback 50 5.1.2 Baseline Perceptual Data 51 5.1.3 Advantages and Disadvantages of Hardware 52 5.2 M D S sorting program 53 5.2.2 Loading Haptic Feedback 54 Chapter 6: Investigation of Rhythmic Haptic Stimuli (Gold Standard Study) 56 6.1 Purpose and Structure of Study 56 6.2 Full-set M D S study 57 6.2.1 Method 57 6.2.2 Basic Results 59 iv 6.3.1 Frequency 63 6.3.2 Rhythm 65 6.4 Summary 72 Chapter 7: Subset Method Validation Study 74 7.1 Validation Overview 75 7.1.1 Criterion 1: Consistency of Results Obtained from Different Subsets 76 7.1.2 Criteria 2 & 3: Overall Accuracy of Results 77 7.1.3 Strengths and Weaknesses of Validation Process 78 7.2 50-Stimulus Subset M D S Study 79 7.2.1 Method (Study Part One) 79 7.2.2 Results 80 7.2.3 Reasons for Difference in M D S Results 82 7.2.4 Study Part Two: Additional Participants with New Subsets 88 7.3 Validation of Subset Technique 90 7.3.1 Criteria 2 and 3: Reasonableness of Results & Comparison to Go ld Standard90 7.3.2 Consistency of Results: Do Subsets Introduce Too Much Noise? 93 7.4 Reflections on the Design of the Subset Data Gathering Method 96 7.4.1 Observations vs. Subsets 96 7.4.3 "Str iping" of Standard Deviation 100 7.5 Summary 101 Chapter 8: Conclusion 103 8.1 Conclusions on Rhythms for Haptic Icons 104 8.2 Validation of M D S Data Gathering Technique 106 8.3 Future Work 107 References 138 Appendix A : Dissimilarity Matrices 140 Appendix B: Individual M D S Plots 228 Appendix C: Subsets 243 Appendix D: Experiment Materials 246 v List of Tables Table 3.1 110 Table 3.2 I l l Table 3.3 112 Table 6.1 127 Table 7.1 128 vi List of Figures Figure 1.1 109 Figure 4.1 113 Figure 4.2 114 Figure 4.3 115 Figure 5.1 116 Figure 5.2 116 Figure 6.1 117 Figure 6.2 118 Figure 6.3 119 Figure 6.4 120 Figure 6.5 121 Figure 6.6 122 Figure 6.7 123 Figure 6.8 , 124 Figure 6.9 125 Figure 6.10 126 Figure 6.11 : :' ' 127 Figure 7.1 128 Figure 7.2 129 Figure 7.3 130 Figure 7.4 131 Figure 7.5 132 Figure 7.6 133 Figure 7.7 134 Figure 7.8 135 Figure 7.9 136 Figure 7.10 .: 137 Acknowledgments Surprisingly enough, this has been quite the learning experience. I thank deeply everyone who has contributed along the way, helping me learn as much about myself as the topic of my research. First I must give thanks to my supervisor, Karon MacLean, whose guidance and instruction has made my research so rewarding. Your belief and support of my work drove me to always strive for my best. M y second reader Tamara Munzner also aided me greatly in clarifying and strengthening my thesis, for which I am much indebted. I would also like to thank the people at Nokia, particularly Jukka Jalkanen and Patrick Wong, who were generous in providing prototype hardware for my research and were so accommodating to my questions and requests. This thesis would also not have been possible without the help of numerous friends, colleagues and family members. Colleagues and friends Shawn, Steve, Ricardo, Rock, Nels and many others made my life easier and more enjoyable in countless ways. Mario Enriquez, whose work so much of my thesis relies upon, was instrumental in providing me with the tools and knowledge required to tackle all that is M D S and haptic. For her support, advice, and above all friendship, Nicole Arksey played no small role in keeping me sane and (relatively) productive during these last two years. I would also like to thank all my friends outside of school, whose insistence ensured I made it outside of the lab to enjoy myself once and while. Lastly I would like to thank my parents, Joyce and Wi l l i , my brother Dan and his wife Meg , for providing a constant source of strength, encouragement, and positive thought. Without you all it could not have been done. Dave Ternes University of British Columbia, August 2007 vi i i Chapter 1: Introduction Can you touch abstract information? What does it feel like? These questions drive us in our research into haptic communication. We wish to convey information to people from computers enabled with haptic displays, and we wish to do so in the simplest and most transparent fashion possible. To accomplish this we build upon the concept of haptic icons: brief tactile stimuli that have been associated with a meaning. We believe that haptic icons present a new means of displaying information to people that can be discrete, convenient and informative while simultaneously decreasing the dependence on the visual and auditory channels of communication. In our vision, we see haptic icons being integrated into almost any interface in which the visual and auditory channels are already used extensively, where haptic icons could provide information to users without requiring them to visually monitor the interface. We foresee haptic icons integrated into handheld devices, where they can support interaction without the user looking directly at the device, a considerable advantage in busy environments, or social situations in which discretion is required. We see haptic icons as a general-purpose design tool to help ease the flow of information from computer to human. Researchers have worked hard to design haptic icons and test them with users. Across many different haptic devices and technologies, robust design parameters such as frequency, amplitude and waveform have been used to create haptic stimuli that users can easily discern and recognize [19, 6]. Mult iple applications have been created using haptic icons, and have been found to be successful in conveying information in practical work contexts [10, 18]. These promising results are opening up a much larger area of research for work in haptic icons. Yet many challenges still remain, not least of which is the gap in understanding between the design of haptic stimuli and how they wi l l be perceived by users. Because our understanding of the sense of touch is quite primitive when compared to sight or hearing, 1 we are constantly forced to test our stimulus sets with users to determine how their members are perceptually related. Another consequence of this is that we lack insight into what the important perceptual parameters of touch are, such that creating new design parameters is often simply done by guess-and-test. What is required is further sophistication in all aspects of our work on haptic icons: greater sophistication in how they are designed, and greater knowledge in how they wi l l be perceived. This thesis works to f i l l this void, though much more work remains to be done. 1.1 Motive It has been posited that a haptic equivalent of visual icons could be used to increase information flow in an environment where computer interfaces are rich in the visual but poor in the haptic. Fol lowing theory with testing, multiple researchers across the globe have shown promising results in the use of various tactile stimuli to present iconic information. Now it is time to push beyond current capabilities, expand benchmarks outwards and determine how far this concept can go. Presented in this thesis are both an expansion of the methodology for building and evaluating haptic stimuli as well as new, multifaceted design parameters with enough depth to support the creation of an expansive set of expressive, distinct haptic stimuli. The motivation for the use of haptic icons is fairly straightforward. In modern interface design, the visual modality is heavily relied upon. Especially in interfaces such as the cockpit of a plane or the driver's seat of a car, the user's visual field is almost overwhelmed with information. But even in a simpler interface such as that of a cell phone, if it is placed in a busy environment where, either for social or practical reasons, the user cannot spend all his/her time looking at the device, the over-reliance on visual communication creates a bottleneck in information flow from device to user. The haptic modality opens up a new channel between device and user, one that can be constantly in contact with the user without him or her constantly attending to it. This is not to claim that simply moving something from the visual to the haptic domain wi l l necessarily free the attentional resources formerly used to track the visual information: attention and multi-modal perception interact in complex ways that we are still only beginning to 2 understand, and other, cognitive bottlenecks exist in aside from basic perception of stimuli. Nevertheless, as an addition to interfaces already heavily dependant on other modalities, the advantages of using the haptic channel to display information have been consistently shown. For both critical control tasks and social communication, haptic icons can be used as a simple, straightforward means of conveying information to the user through the haptic channel. This simple, one-degree-of-freedom communication makes them, perhaps, the most basic building blocks of abstract haptic communication. Research into haptic icons has generally focused on the use of short, simple vibrotactile stimuli to convey information to users [19, 6]. This focus should make them easy to create, easy to display on a device and easy to evaluate. Unfortunately, this claim has been the goal, but not the reality. In truth, research is still hamstrung by poor haptic displays on which even simple design of stimuli presents serious challenges. Poor displays lead to noisy results which make evaluation difficult, and evaluation of complex human interaction with technology is not a simple task to begin with, as the very existence of the field of Human Computer Interaction (HCI) attests. Nevertheless, we are seeing more technology equipped with more advanced haptic displays every day, especially in handheld devices. If haptic display becomes more common, then the opportunity for new haptic-enabled applications increases sharply, in no small part due to increased user familiarity with the medium. Furthermore, as .discussed above, handheld devices are often operated in busy, demanding environments where a different, discrete modality such as haptics can help make the difference between an easy-to-use, helpful application and one that simply causes the user more stress and frustration. Consequently we are applying our research to the development and application of haptic icons in the real world, in hope that they can solve real-world problems positively and efficiently. 3 We are not there yet. Research into haptic icons has generally been preliminary, involving relatively small numbers of icons used in laboratory environments. Developing stimuli has been done on a per-experiment basis, and has differed widely across researchers (see [27] and [18] as comparisons). What this thesis aims to do is to greatly increase the number of coordinated haptic stimuli that can be used in an experiment or in an application: increase via gross number of stimuli in existence, and increase via easing development. We hope to create the beginning of a general reference set of haptic stimuli that has been evaluated for consistency and distinctness. Moreover, we also aim to create a robust process for the creation and evaluation of large number of haptic stimuli, allowing other researchers to follow in our footsteps. Increasing by roughly an order of magnitude the number of haptic stimuli in a single, coordinated set that are available to researchers, we wi l l be allowing for the creation of larger, more complex applications that can use haptic icons. These applications can then be applied to real-world situations, studied over longer periods of time, and evaluated for their usefulness and usability. Thus the contributions of this thesis are to be viewed as major steps towards a more wide-scale, ecologically valid evaluation of haptic icons. This thesis aims to break haptic icons out of the domain of "toy" research and into the domain of grounded, practical application work. 1.1.1 Icons vs. Stimuli It should be made clear that, because our work deals strictly with the haptic stimuli themselves, and at no point attempts to attach meaning to them, we do not often refer to haptic icons throughout this document, instead haptic stimuli. Only when a stimulus has a meaning associated with it does it become an icon, and this semantic process is not the concern of this work as the stimuli must be designed first, before meaning can be attached to them. The process of assigning meaning is left to the designers of application who wish to use our stimuli to make haptic icons. 1.2 Overview and Approach Two main problems stand in the way of our goal of creating a large, diverse set of haptic stimuli; solving them comprises the bulk of the contributions of this thesis. Both issues 4 stem from the size of the set of stimuli that we wish to create. The first challenge is how to create so many different stimuli which are distinctive and expressive to potential users. This problem is approached here by the systematic and wide-scale use of tactile rhythms, a little-explored parameter for use in haptic stimuli that we find greatly extends the number of perceptually distinct tactile sensations that can be created, even with a duration as short as 2 seconds. The second challenge is to evaluate larger sets of stimuli, when traditional means of evaluation do not scale well due to time and fatigue. Our solution to this problem is a new method of gathering perceptual data that requires users to judge only a manageable subset of the complete stimulus set, with the total, overall perceptual picture stitched together using judgment data from multiple users. Thus this thesis contains two intertwined, interdependent strands of research, one of method and one of design. The design is the creation of a set of 84 haptic stimuli (much larger than the previous standard of 36 set by MacLean and Enriquez [19]), using amplitude, frequency and rhythm. The method is an extension of the data-gathering techniques for multidimensional scaling (MDS) that are used to determine the perceptual characteristics of a set of stimuli. This method has expanded the size of stimulus sets that can be dealt with by a factor of three, enabling 150 stimuli, where 50 was the previous maximum. By running two separate but related studies we are able to both evaluate our stimulus set and validate our new data-gathering method. This duality informs the structure of this thesis: Figure 1.1 lays out the logical structure of the thesis, in terms of the two interlocking research strands. The key to understanding the logic of this thesis is to understand the role played by the two major studies discussed herein, and how they pertain to the two strands of research described. Both studies seek to perform M D S on the same stimulus set; they both follow the basic steps of (a) determining how people group our set of rhythmic haptic stimuli, and then (b) feeding this data into the M D S algorithm, producing a n-dimensional plot of the stimuli where the placement of each stimuli relative to each other stimuli represents how perceptually similar the two were on average judged to be. How the two studies differ is in the method by which perceived similarity ratings were gathered. The first 5 study uses the sorting technique developed by MacLean and Enriquez [19] to gather data. This is a proven method [ 2 0 ] , but it is stretched to its limit by the sheer number of stimuli users are forced to deal with here. The second study uses a modification of the sorting technique that allows a larger number of users to each deal with a smaller subset of stimuli. This method, based on a form of between-subjects analysis, presents the user with a less taxing task that can be completed in a more reasonable timescale, but as an experimental technique the data integration must still be proven to produce valid results. The two studies thus provide two different looks in at the same stimuli set. Since it utilizes a previously validated technique [ 2 0 ] , we use the first study to investigate the perceptual characteristics of our haptic stimulus set. B y completing an in-depth analysis of the M D S plot and examiningprecisely what characteristics of our stimulus set influences how they are perceived, we find strong and interesting effects of tactile rhythm—indicating its strength as a design parameter for use with haptic stimuli. Yet in performing this analysis, we also develop a "gold standard" against which our new data-gathering technique can be compared. Our second study uses the new method to examine the same set of stimuli, and produces results which are both quantitatively and qualitatively similar to the gold standard, thus allowing us to conclude that the technique itself is valid as well as more practical. It might be noted that, if haptic icons are truly to be the touch-based equivalent of visual icons, the process that we are describing seems considerably more complex and involved than what would be expected of a visual icon design process. When needing a new visual icon, one might simply give a graphic designer some requirements as to what information the icon need convey, and then upon receiving the designers best guess at what the icon should look like, one would l ikely simply "eyebal l" the result, ensuring its appropriateness. Even if more detailed user testing was performed, it would l ikely only be a part of a more wider usability analysis, and certainly would never reach the level of complexity and rigor exhibited in our M D S studies. Yet as a technology, as a psychological science, and as a symbolic medium, haptics and vision are by no means on the same playing field—thus comparing relative design processes leaves haptics at a 6 considerable disadvantage. We have been studying and using vision, figuring out different ways of displaying visual information, for far longer, and in far greater detail, than ever has been done for haptics. Relatively primitive displays and lack of knowledge make working with haptic icons considerably more challenging than visual icons. If we were to limit visual icons to abstract expressions only (as haptic stimuli themselves are limited); i f we were to require only 6-pixel visual icons, with no more than a dozen different shades of grey to choose from; if creating each individual visual icon required an extended period of work writing code or mathematically modeling a waveform; if all these limitations were required of visual icon design, then perhaps a fair comparison of methodologies would be possible. As the field stands, our current thorough design process represents our best attempt at overcoming the constraints, both perceptual and technological, that the haptic modality places on icon design. Thus, at the end of this research what we have accomplished is setting up a solid basis from which to extend the expressiveness and diversity of haptic icons. We developed a new design parameter that greatly increases the number of different haptic stimuli that can be created. We also developed a new technique for analyzing these stimuli to ensure their success as useful, informative signals. This creates a toolkit and a process that any new haptic icon developer could use to quickly and easily create a large set of discernable haptic stimuli tailored to his or her needs. 7 Chapter 2: Related Work 2.1 Abstract Tactile Communication Though it could easily be considered a small niche of the research world, work on abstract tactile communication has flourished in the last few years, with groups from several different research labs across the world contributing to a great increase in knowledge about the design and application of haptic icons and informative tactile signals in general. Their results have all been generally positive, indicating a clear ability of people to discern, recognize and use haptic icons or similar types of stimuli in a variety of applications and contexts. 2.1.1 Haptic Icons In the work of MacLean et al, which our own work largely builds upon, the design and use of haptic icons has been study extensively across different platforms and applications. MacLean and Enriquez [19] created haptic icons using a force feedback knob, varying waveform, amplitude and frequency to create a set of 36 haptic stimuli. In order to determine how these icons were actually perceived, they were thoroughly analyzed using Multidimensional Scaling (MDS) , an analysis technique that would prove invaluable for further work on haptic icons, and which is expanded upon in this thesis. They found that their three parameters, with some adjustment, could create an even spread of perceptually distinguishable stimuli. Their results showed both the utility of M D S as a tool for perceptual analysis of haptic sensations, and demonstrated that people can make consistent distinctions between well-designed haptic stimuli. Fol lowing the work of MacLean and Enriquez were several more studies. Chan et al. [10, 11] developed a haptic icon-based protocol for turn-taking in a collaborative environment using a haptic mouse. It showed the efficacy of haptic icons used in cognitively loaded environments to communicate information unobtrusively to users. Luk et al. [18] used a novel piezo-driven skin-stretch display to present users with haptic icons in the context of a handheld device. Luk et al. showed that the haptic icon design paradigm could be 8 applied equally well to new, handheld and non-vibratory platforms. Such robustness is an encouraging result if we wish to continue extending haptic icons in new and interesting directions. 2.1.2 Tactons Coining the term "tactons" to describe their vibratory tactile icons, Brown et al. have also explored somewhat similar terrain, with similarly encouraging results. L ike the work of MacLean et al. they have both developed design parameters for creating informative tactile stimuli and analyzed these stimuli to determine their perceptual nature. In [5], they introduce several different design factors to be used to create tactons, starting with frequency, amplitude and waveform as in [19], but adding to it duration and rhythm, as well the body location at which the stimuli is presented. These ideas were tested in-depth in several later papers. Firstly in [6] they tested icons with three different amplitude-modulated textures and three different rhythms (based on previous audio icon designs) and found strong recognition rates for each of their two parameters (80% and 93% for texture and rhythm respectively). In a still-developing work [7], Brown et al. added in a third parameter, location of stimulus presentation, and found that recognition rates dropped somewhat for three-parameter icons, but could be designed around if needed. In collaboration with a variety of other researchers, both Brown and Brewster continue to expand their research into tactons by examining crossmodal effects [14], various musical techniques such as crescendos [8], and applications in mobile phones [9]. 2.1.3 Other Vibro-tactile Work Research into tactual perception has been done at a more general level as wel l , not just in attempt to create some form of haptic icon. Tan et al. [23] have studied the display of tactual signals, comparing a variety of stimuli to determine the overall level of information that can be transmitted using artificial stimuli. Though their results considering the informational capabilities of tactile stimuli are a positive indication for us, they make no discussion of designing of stimuli for use in practical considerations, nor do they proceed beyond basic tactile waveforms in their stimulus set. Other researchers such as Klatzky and Lederman have performed research into related areas, for example texture perception using a stylus [16], and have gained similarly positive 9 results. Representative of much of the work in this field, both of these works are directed at fairly low-level psychophysical findings, leaving considerable room still to be explored as we bring these tactile stimuli into more and more practical contexts. Other important work on displaying tactile information has been done by van Erp [26], where he specified several design guidelines for communicating through vibro-tactile displays. He discusses several features present in the work of both MacLean and Brown, such as frequency, amplitude, temporal patterns (which are essentially rhythm), and display location. He also points out the dangers of masking and confusion that can occur using spatial and temporal effects. Van Erp has also worked on tactile melodies [27], which is, to date, the most significant analysis of .tactile melodies and rhythms yet published. Van Erp and Spape take 59 real-world melodies and transfer them into the tactile domain. Using M D S along with other statistical methods, they determine two main perceptual characteristics: intrusiveness and tempo. However, these results are limited to a description of very complex and specific musical rhythms, such that direct application to the design of synthetic tactile stimuli from scratch would be difficult. Nevertheless, the results of their experiment show encouraging trends for tactile melodies and rhythms. 2.2 Multidimensional Scaling Multidimensional Scaling (MDS) is a statistical technique that provides quantitative values describing the perceived dissimilarities between a given set of stimuli. The M D S algorithm takes an input of dissimilarity values and calculates a value describing the distance of each stimulus in relation to every other stimulus in a perceptual space. A n appropriate number of dimensions are chosen based on the stress value, a measurement of model fit, where lower stress represents a better fit. Choice of the number of dimensions is determined by the benefits of an additional dimension in reducing the stress level against the loss of interpretability an additional dimension adds. Once an appropriate number of dimensions have been chosen, the data can be mapped visually and analyzed for clustering and trends [3]. 10 The main reason that we use M D S in the study of haptic icons is its ability to pick out trends and grouping of stimuli, when no a priori knowledge exists about how the stimuli wi l l be perceived. The application of M D S to the perceptual domain was pioneered by Roger Shepard at Be l l Laboratories in the early '60s [21] as a general tool for analyzing interstimulus similarity amongst any grouping of stimuli for which the perceived differences were unknown. The technique was quickly picked up by many psychophysicists, being applied to such areas as, for example, musical timbre. Significant new understandings in how timbre was intuitively perceived were brought about by the use of M D S [13], an early positive indicator of its usefulness as an exploratory tool. The haptic modality, compared to modalities such as vision or audition, is still largely lacking in a widely-accepted and intuitive description of its important perceptual characteristics, though the work of Tan [23] and other has taken great strides in gathering the basic perceptual knowledge to needed to gain such intuition. Lacking this intuition, M D S can provide clues to what these characteristic might be, without us knowing beforehand. M D S has thus been used extensively for the study of haptic stimuli, as well as other stimuli in different modalities for which there is a similar lack of intuitive understanding. In addition to the work of MacLean et al. described iri section 2.1, other researchers have successfully used M D S to discover new information about novel stimuli. For example, [15] found a 3D percept map from a set of real tactile surfaces, finding dimensions such as hard/soft and slippery/sticky. Bonebright [2] used M D S to analyze everyday sounds, in hopes of furthering his work into designing informative sound icons. Considerable effort has been applied to analyzing different types of M D S algorithms as well as different means of gathering dissimilarity data for analysis. However, for our purposes the standard SPSS M D S algorithm, A L S C A L , shall be used. Our main research interaction with M D S is in the means of gathering data; this shall be discussed in detail in Chapter 4. 11 2.2.1 Comparison of MDS Results One aspect of M D S algorithmics that does bear mentioning is how best to compare the results of different M D S analyses. Two different M D S output maps based on subjective dissimilarity ratings from different users are almost guaranteed never to be the same, even if the stimulus set is the same. In canonical research into M D S from the psychology and psychophysics fields (such as [22]), it was originally considered that Pearson's r, the product-moment correlation coefficient, was sufficient to measure whether two n-dimensional M D S outputs were statistically correlated. For clarity, note that Pearson's r is unrelated to r2 which is used elsewhere in this document to describe goodness of fit for M D S results. Borg and Leutner [4], with a simple example, proved that the product-moment correlation is in fact inappropriate for use with M D S outputs: Let A and B be two M D S configurations, consisting of NP = 3 points each, with distances d(l,2) = 1, d(2,3) = 2 and d(l,3) = 3 for A , and d(l,2) = 2, d{2,3) = 3, d(l,3) = 4 for B. The P M [Product-Moment] correlation of these distances is r = 1, indicating perfect similarity of A and B. This is false, of course, since the greatest distance in A is three times as great as the smallest, whereas in B it is only twice as long. Hence, A and B do not have the same shape: B forms a triangle, whereas A ' s points lie on a straight line, because they satisfy the equation d(l,2) + d(2,3) = d(l,3). Thus instead, Borg and Leutner proposed the use of the congruence coefficient defined as, [1] c = zZidAidBi/(zZidAl2zZidBl2)m where dxi is the z'-th distance value between stimuli, in configuration X , and the sum is over all pairs of distances in the M D S output map. Due to the tendency of c to cluster close to its upper limit of 1, a transformation was performed, giving us the alienation coefficient K, [2] K= (1 - c 2 ) m 12 This formula was then empirically tested on randomly created dissimilarity matrices, at different dimensions and numbers of points, providing empirically derived constants describing similarity of M D S outputs at a 9 5 % confidence interval. The alienation coefficient thus provides the only statistical measure specifically designed and tested to measure similarity of M D S outputs, and still stands as the state of the art for statistical comparison of M D S results—Borg's own book on M D S confirms this status [2] In our own work we rely on the alienation coefficient to help validate our newly designed method of gathering data for M D S . B y using two different data gathering methods on the same set of stimuli, we produce two M D S results that we hope to be similar. The work of Borg and Leutner gives us a statistical tool to compare the two results, to be used in conjunction with our standard visual interpretation of the M D S map. 13 Chapter 3: Creation of Haptic Stimuli The simple haptic stimulus sets developed by MacLean and Enriquez [19] and built upon in several ensuing publications [11, 18, 12] are based upon varying three parameters: waveform, frequency and amplitude. However, using these three components has only produced a relatively small set of stimuli, when varied using today's tactile display hardware. Whi le these parameter provide a solid basis for building tactile stimuli, we sought to find a means of creating new and interesting stimuli supplementing some of with more complex parameters that might yield better results.. Previous work on haptic stimuli has investigated the application of rhythm [6] and melody [27] to vibrotactile stimuli, with fairly positive results. The approaches taken were, however, quite different, with the work on rhythm using only a handful of very basic rhythms, while the research into melody used a broad range of actual musical melodies transposed into the tactile domain. So while their results point towards a positive use of rhythm and melody in developing expressive tactile signals, their goals were such that they do not cover a broad enough sample of the possible design space to reveal a consistent framework for dealing with these parameters in the tactile domain. Both studies lacked results that could be broadly generalized to all types of tactile rhythms. The initial studies by Brown et al. [6] used only three rhythms, which were too different and too few to establish any clear patterns. Van Erp and Spape [27] used far more stimuli, but because their stimuli were sampled non-systematically from real-world examples of music, they both lack a systematic description of their structure and suffer from the many possible learned musical associations that a participant in the study might have. 3.1 Description of Possible Stimulus Space For our own purposes we felt that it was wrong to assume that the standards that inform normal auditory musical composition would apply to the sense of touch; the skin's sensory capabilities are attuned to different things than the ear, to say nothing of the effects that cognitive aspects of musical appreciation might have. This is not to say an 14 approach attempting to utilize musical capabilities in the haptic domain might not be successful; rather, bringing the musical into the haptic is simply too large an issue to deal with within this thesis. Yet despite our wish to avoid borrowing too heavily from the musical domain, it was nonetheless considered most straightforward to represent rhythm and melody as a sequence of notes of varying length played at set intervals in a bar of "music." For our purposes, we define rhythm as being the repeated, patterned recurrence of some set of variable length beats/notes, and we differentiate this from melody, which is concerned with the different tones that the notes in rhythm are played at. Changing the tones of the notes in a rhythm changes the melody, but not the rhythm; changing the length, number or placement of notes changes the rhythm and would also likely have an effect on the melody. If we first simply look at the number of different combinations of notes that could be used in a rhythm or melody, we are immediately faced with an exponentially increasing set of possibilities. Limiting ourselves only to quarter notes in 4/4 time, we would have 2 4 = 16 different way of arranging notes, with eighth notes 2 8 = 256 variants, with sixteenth notes 2 1 6 = 65536. If we then consider playing melodies (i.e. the tone of each note is different, played at different vibratory frequency) or adding emphasis (i.e. playing different notes at different amplitudes) then the number of possibilities grows even larger. Clearly we needed a means of reducing this huge number of variants down to a manageable handful, in a way that would produce tactile stimuli that were different enough to be perceptually distinguishable and while possessing shared features that would contribute to some natural perceptual groupings to increase learnability. We lacked a clear precedent into what would make a tactile rhythm or melody distinguishable yet also perceptually similar enough to the other stimuli used that some natural grouping would be evident. Thus we were forced to rely heavily on intuition and our own reasoning on how to move forward. 15 V e r y q u i c k l y it became clear to us that, as the above numbers indicate, some a priori design decisions were needed to l i m i t the scope o f this work . O u r first observation was that a l l melodies have a rhy thm, at least i m p l i c i t l y , and this suggested rhy thm was a more fundamental parameter than me lody and therefore should be focused on first. T o el iminate me lody as a confound and keep our search space manageable, we therefore u t i l i zed on ly monotone (non-melodic) rhythms: a l l notes i n a part icular rhy thm were p layed at the same ampli tude and frequency. The part icular ampli tude and frequency leve l at w h i c h a rhy thm is p layed c o u l d s t i l l be varied, meaning that we have a l l owed rhythms that have the same number, type and placement o f notes, but different overa l l frequency and ampli tude levels . It is on ly variat ion o f frequency and ampli tude within a rhy thm that we are choos ing to disregard for the sake o f s impl i c i ty . W e felt that these in i t ia l design choices narrowed the f ie ld o f possible s t imul i d o w n to an area that c o u l d be reasonably approached i n a more rigorous manner. W h a t fo l lows is our analysis o f the tactile rhy thm space, and a descript ion o f how we narrowed d o w n the f i e ld to our f inal selection o f rhythms. 3.2 Sensory- and Hardware-Specific Limitations on Rhythm Space S t i l l without a set precedent on how we might parti t ion the space o f a l l possible tactile rhythms, we set out to study the space as best we cou ld . B y i terat ively creating different haptic rhythms and observing how they felt, we were able to in fo rmal ly develop a set o f rules that we felt tactile rhythms needed to obey i n order to produce diverse yet associable s t imul i . Some o f these rules we bel ieve, based on our o w n testing, to be necessary for creating any tactile rhythm, wh i l e others are more design heuristics that represent our o w n intui t ion on what makes good s t imul i . In a l l cases though, these recommendations are based on our tests on a specific hardware platform (described in more detail i n Chapter 5), and though it is l i k e l y that m u c h o f our w o r k here is broadly generalizable, we cannot remove complete ly the confound o f the specific hardware used. Nevertheless we feel that our extensive informal testing wi th a variety o f different users has lead to consistent h igh- level recommendations. 16 3.2.1 High-level Limitations Two facts were immediately obvious to us as soon as we started creating tactile rhythms. First, there needs to be a gap after each note played in a rhythm in order for the notes to be distinguished as separate. If the individual notes within a rhythm varied by frequency or amplitude, it would be possible to perceptually segment the different notes if the differences were large enough between adjacent notes; however, as we had already decided to limit ourselves to monotone rhythms, gaps in between notes were necessary. The second fact was that unless a rhythm was repeated, it was not perceived as a rhythm, merely a set of isolated vibrations. Furthermore, the more times a rhythm was repeated, the stronger the sense of rhythm became. From our testing we found that four repetitions was a good compromise between having enough repetitions to create a strong sense of rhythm without requiring an overly long total duration for the stimuli.,This observation led us to choose a total stimulus duration of 2 seconds, resulting in a 500 ms duration for each iteration of the rhythm. We felt two seconds to be about the longest a stimulus could last and still be useful in the context of a haptic icon, while the 500 ms duration was long enough to allow for a fair number of different notes to be packed into a rhythm (e.g. 4 bars of 4/4 time played at a brisk tempo, as elaborated below). Though from an auditory musical perspective 500 ms could be considered quite short for a bar of music, we were limited by our need to make the overall signal fairly short and yet still present enough repetitions. However, we did not find this to be too fast a pace to be playing our rhythms at, largely because we were not asking our users to pick out individual notes, just perceive an overall sense of the rhythm. Perception of the rhythms as a whole was still attainable, and benefit of the increased repetitions helped counteract the speed of the overall rhythm. 3.2.2 Shortest Note With our single iteration time of 500 ms established, the next issue was to find the shortest length of note people could consistently perceive. From our informal testing we settled on a sixteenth of the total time (31.25 ms) followed by a break of a similar duration. We could have chosen to make the break shorter in time than the vibration, but 17 in order to make it easier for us to line up where different notes (and breaks) fel l , we chose to keep the on and off time the same. Thus the total time required for the smallest note was 62.5 ms, or exactly one eighth of the 500 ms single iteration time. Consequently the smallest note we use is called an "eighth" note because it takes up exactly one eighth of the time of a single iteration of the rhythm. Having now determined the smallest interval, we built all our rhythms along the basis of 16 consecutive time slots, which can either be on or off. 3.2.3 Selection of Different Note Types We next sought to determine what other note lengths we should use to build up our rhythms. To do.this, we made a series of rhythms containing only one note, with a note length varying from one to fifteen of the sixteen, 31.25 ms time slots (the 16 t h slot was required to introduce a break between rhythm iterations). Of these fifteen possible note sizes, the following observations were made. • The difference between a 31.25 ms vibration and a 62.5 ms vibration is noticeable (i.e., one vs. two consecutive time slots set as "on"), though not overpoweringly so. • The difference between a 62.5 ms vibration and a 93.75 ms vibration (ie, two time slots vs. three time slots) is not consistently noticeable. • This finding holds true for all longer notes: differences of one time slot (31.25 ms) are not noticeable. • However, differences of 62.5 ms are noticeable for all longer notes. These observations resulted in the following five types of notes, also described in Figure 3.1 in terms of the number of time slots they occupy. • Eighth note (62.5 ms total play time: 31.25 ms on, 31.25 ms off) • Quarter note (125 ms total play time: 62.5 ms on, 62.5 ms off) • Half note (250 ms total play time: 187.5 ms on, 62.5 ms off) • Three-quarter note (375 ms total play time: 312.5 ms on, 62.5 ms off) 18 • Whole note (500 ms total play time: 437.5 ms on, 62.5 ms off) We used the larger break time of 62.5 ms (two time slots) rather than 31.25 ms (one time slot) because we felt it gave greater distinction between notes. The 31.25 ms break time is used only for eighth notes, in order to allow us to have two eighth notes played within the same amount of time as one quarter note, which was a feature we used to create several of the different groups of rhythms described in the next section. 3.3 Description of Stimulus Set At this stage in our design process, we had pruned down our selection of possible rhythms considerably through ad hoc and informal testing. However, the rhythm space still remained relatively large. Having done all we could to develop rules describing which types of rhythms not to use, we now had to develop some positive heuristics as to which rhythms we should use. Again, using our intuition along with iterative informal testing, we developed four heuristics, which in turn created five groups of rhythms, each defined according to one or more of these rules. These groupings were designed into the stimuli set from the start, and represent our best attempt at creating a diverse yet logically grouped set of tactile rhythms. We do not claim that this wi l l end up to be the best grouping of rhythms that could be used. However, without a clear precedent into how tactile rhythms might be grouped and perceived, our own intuition, along with continual informal testing, was the best tool we could use. In our study results we discuss how our intuitive groupings were, in part, confirmed: some of our groupings were held out by the study, while other unanticipated perceptual groupings were also found. For clarity we believe it important to specify what groupings we built in to our stimuli beforehand and differentiate them from the post hoc groupings that our studies later revealed. The groupings below represent our initial best guess at how tactile rhythms might be grouped. 19 3.3.1 Heuristic One: Quarter Notes Our first heuristic was to consider all possible rhythms that contain only quarter notes and pauses. This decision was based upon both our initial testing results which found quarter notes to be an easily recognizable duration, as well as the consideration that a straight 4/4 rhythm with notes on every downbeat would l ikely be considered one of the most simple and basic rhythms available. As noted above, there are 16 possible all-quarter note rhythms. However, many of these are perceptually indistinguishable from each other because of the repetition of the rhythms. As a specific example, consider all rhythms containing just one quarter note: the note could occur in any of the four slots, and thus we have four different rhythms. Yet an issue with 'monotone' rhythms (those without varying emphasis and thus a discernible downbeat) is that while looping, there are no indicators of its starting point. Thus all four of the single-note rhythms wi l l feel the same once they have started, as the spaces between the 4 played notes (one in each iteration) wi l l be the same in all cases. Similar situations occur for some rhythms of two and three quarter notes per iteration, and we therefore used just one instance of each of these cases. Thus considering all possible quarter note-only rhythms that are perceptually distinct from each other, we arrive at Group 1, the first five rhythms as indicated in Table 3.2. 3.3.2 Heuristic Two: Long Notes Our second heuristic was to consider all rhythms containing only notes that are longer than quarter notes: i.e. half notes, three-quarter notes, and full notes. This decision was based upon the observed difference in sensation that longer vibrations gave as compared to shorter notes such as the quarter note, and because we felt that having a variety of different note lengths in our rhythms would be prudent i f we wanted to obtain a good cross-section of different types of rhythms. Thus we have categorized quarter notes as being "short" and notes longer than a quarter as being " long." Similar issues of duplication due to repetition were present for this group, narrowing the number of possible rhythms down to four, creating Group 2 in Table 3.2. 20 3.3.3 Heuristic Three: Long and Quarter Notes Our third heuristic was to consider rhythms which contained at least one quarter note, and least one of the longer notes used in Group 2. The goal here was to produce rhythms which had both quick and slow components. Again there were issues of duplication due to repetition, which pruned several rhythms, and the requirement that there be at least one quarter note meant that full notes could not be used. The final set of four possibilities is presented as Group 3 in table 3.2. 3.3.4 Heuristic Four: Substituting Quarter with Eighth Notes Our fourth heuristic actually resulted in two groups of rhythms, as this heuristic was actually a means of modifying two of the groups already described.- Because the number of different rhythms containing only eighth notes is 256 (to say nothing of rhythms containing combinations of eighth notes and other length notes), we felt daunted at the prospect of choosing some reasonable set of rhythms from this space. Consequently, we made the simple choice of taking the rhythms we created in Group 1 and Group 3, and in place of each quarter note (total playing time of 125 ms) we substituted in two eighth notes (playing time 62.5 ms a piece). This gave us a set of four eighth note-only rhythms (Group 4, analogous to Group 1) and four eighth note plus longer note rhythms (Group 5, analogous to Group 3). Group 4 has four rather than the five rhythms in Group 1 because the eighth note analog of rhythm 3 was felt to be very hard to distinguish from the eighth note analog of rhythm 2. 3.3.5 Complete Stimulus Set Used With the 21 rhythms described in Table 3.2, we finally had set of diverse yet associable tactile rhythms. Thankfully we had chosen early on that, though each stimulus must be monotone in terms of the frequency and amplitude of all of the notes played within it, we can create different stimuli by simply playing the same rhythm at a different set level of frequency and amplitude. Keeping in mind our desire to have a large, but not overlarge set of stimuli, we thought it best to have two frequency levels (high and low) and two amplitude levels (high and low) that each of the 21 rhythms could be played at, creating 2 1 x 2 x 2 = 84 different stimuli. By having only two levels of frequency and amplitude, 21 high and low, we hoped to ensure that the differences between each of the four frequency x amplitude levels would be quite strong. Table 3.3 gives the exact value, rhythm-type by amplitude by frequency, that each of the 84 stimuli was given, and can thus be used as a lookup table for all further references to individual stimuli throughout this document. 3.4 The Space Untested While our own selection of 21 different rhythms to use for our stimulus set was guided by a thorough loop of iterative testing, we by no means claim that our choices are the only ones that could have been made. Indeed there are many rhythms that we did not use, and that if studied, may well lead to further insights into tactile rhythms. As important as it is to understand the rhythms that we have chosen to use, it is also important to understand their relationship to the space of rhythms we did not choose. Right away, the choices we made in Section 3.2 sharply decreased the number of possible rhythms that we were working with. First limiting ourselves to only monotone rhythms, and then specifying a time-span that could only allow notes no shorter than an eighth of a bar, we made somewhat arbitary, yet we feel reasonable steps towards a choosing a well defined rhythm space to work within. Many other choices could be made at this level, such as having varying (and slower) tempos, varying amplitude of notes within a rhythm, using melody, using crescendos and many other musical techniques—not even to mention multi-bar musical compositions. A l l these choices could l ikely lead to interesting new developments, but they are left to others to explore. We do believe, however, that many of these choices represent additions of considerable complexity to the rhythm space, such that we feel that in most cases we chose to decrease complexity of the parameters we were working with. 3.4.1 Unused Rhythms Possible Given Hardware and Sensory Limitations Even narrowing down our rhythm space to a more manageable size, we still had more rhythms than we needed for our purposes. The choices that we made to select our final 21 rhythms are outlined above, but it is worth considering the rhythms that we did not choose. It is possible that some of these rhythms might be worth revisiting at a later date, 22 especially if our chosen selection of rhythms does not truly represent an even cross-section of possible rhythms. Given the limitation set out in Section 3.2 for the size and number of notes that can be fit into a rhythm, we can see that for rhythms containing only quarter or longer notes, we have exhausted all possible rhythms that could be used. Rhythm Groups 1, 2 and 3 specify all of the rhythms using only quarter notes, only notes longer than quarter notes, and both quarter and longer notes, respectively. Consequently for these note lengths we can be confident in the coverage of our different rhythms. However, as specified in Section 3.3.4, we were not as thorough in our coverage of rhythms containing eighth notes. There are 256 rhythms containing only eighth notes, and though many of them are likely the same due to repetition, we still only use four. Though these four do have a good level of variance in terms of the number and placement of notes, we neglect many of the more complicated rhythms that could be created, as well as any rhythms with just one eight note separated by'pauses on both sides (this because we were echoing the rhythms in Group 1, replacing one quarter note with two eighth notes). These more complicated rhythms could well have produced interesting, more nuanced results, yet for this initial exploration it was thought best to -start with relatively simple patterns. Moreover, it was noted that the subtle differences in placement of a single eighth note were often very hard to notice perceptually, so we felt that fully examining this space would lead to diminishing returns. Considering the combination of eighth notes with other, longer notes, there are yet more possible combinations that were not used. In rhythm Group 5, we combined eighth notes with longer notes, but many other possibly arrangements remain. Again, we felt that we had a fairly reasonable cross-section of different numbers and placements of eight notes, but by no means exhaustive. Taking, for example, rhythm 18 (containing a two-thirds note followed by two eighths), we could have also made this rhythm using just one eighth note, placing it either in the last or next-to last slot. Whi le one of these rhythms might have produced slightly different results than rhythm 18, it is doubtful it would have been 23 greatly different, given the shortness of the eighth notes. Furthermore, it is almost certain that users would not have been able to discern the difference between the single eight note being in the last or next-to-last slot. This sense of diminishing returns is strong when considering eighth notes, given that they lie almost on the threshold of perceptual distinguishability. One last combination of notes that we did not use at all was combinations of quarter notes and eighth notes. This was largely because, as described in Section 3.2.3, we found the difference between these two note types to be quite perceptually weak. Beyond this, we deemed 21 rhythms to be a fairly large number of rhythms, and so did not want to over-extend our reach. As we wished to test these rhythms at different amplitude and frequency levels, given the importance of these parameters in prior research into tactile stimuli, the number of rhythms we developed would provide us with a large set of stimuli as it was. So we chose not to use all possible rhythms that we could have, mostly for practical reasons. Nevertheless we feel that the majority of the rhythms we did not use would have been perceptually quite difficult to distinguish between, and we feel confident that the rhythms we did select represent the strongest and widest selection we could have reasonably chosen. 24 Chapter 4: Subset Data Gathering Methodology for MDS In order to be able to analyze the large sets of haptic stimuli that we are creating, we need to gather dissimilarity ratings from users that we can then feed into the M D S algorithm. However, collecting judgment data from people on that many stimuli at once is unwieldy and impractical. Our insight is to present users with less than the total stimulus set, and then create a total, aggregate view of the stimulus set by averaging overlapping data from multiple users. If we gather data simply by presenting users with pairs of stimuli and asking them to rate their similarity, it is easy to safely use a subset of possible stimulus pairs in a set, but pairwise comparisons also take far too long to perform and suffer from calibration and drift problems as subjects are being asked to make absolute judgments. Conversely, using a different data gathering method such as asking users to sort the stimuli into different groups based on perceived similarity is a far quicker task, but makes splitting apart the stimulus set much harder. Herein we address these challenges by developing a.means of sorting stimulus sets that allows us to present users with only a subset of the total stimulus set, greatly shortening the total time and effort required of an individual asked to provide perceptual judgments. In practical terms, this brings about a three-fold increase in the number of stimuli that can be examined (from 50 to about 150). This is a significant result for tactile stimuli which are particularly difficult to gather perceptual data from; further, 150 may approach the limits of distinct stimuli that can be displayed and eventually learned given today's tactile display hardware. The challenges imparted by this new data gathering method are of two types. The first is in development of an algorithm for splitting up a stimulus set into subsets that can be sorted individually by users and then successfully stitched back together again to form an aggregate picture. Secondly this method faces several challenges in its experimental validity, there being some potentially confounding effects of judgments gathered from only part of the total stimulus set. As we are proposing a novel method for gathering dissimilarity data for M D S , and several potential problems are clearly extant, a means of 25 validating this method must be developed and applied in a real-world experiment. This validation process is described in Chapter 7, but first, in this chapter we outline the new data gathering method and discuss its strengths and weakness. ,-. ?" 4.1 Other Methods for Dealing with Large Set Sizes One of the limitations of the basic M D S procedure is that it requires a dissimilarity rating for every pair of stimuli involved. A dissimilarity matrix for a stimuli set of size n contains n(n - l ) /2 dissimilarity values (since the dissimilarity ratings are symmetric it is only a half matrix, hence the division by two). Consequently the number of dissimilarity values required increases quadratically with the number of stimuli. As the number of stimuli being compared becomes large, it is an increasingly laborious task to gather all these dissimilarity values. Subject fatigue and loss of calibration quickly become a problem. If we are to study a set of 84 (or more) different haptic stimuli using M D S , then we wi l l need a method of gathering data that overcomes this problem of size. Tsogo et al. performed a review of established data gathering techniques for dealing with oversized sets of stimuli [25], i.e. set sizes which are too large for the acquisition methods available for those data. According to their review, there are two main simplifying approaches available to mitigate this problem: using incomplete dissimilarity matrices, and gathering comparisons via sorting tasks, rather than individual pair-wise comparisons. Both of these methods reduce the length of time and amount of work required from users to gather perceptual data, but both eventually come up against hard limitations as to the total number of stimuli they can handle. In this context "pair-wise comparisons" is the method whereby each possible pairing (disregarding order) of two different stimuli in a set are presented to a user, who is then asked to provide a rating of similarity; a "sorting task" is the method whereby users are presented with the entire stimulus set and ask to "sort" or categorize them into groups according to perceived similarity. A third data gathering method that we would also add to Tsogo's accounting is the use of a pre-determined scale that can be applied individually to each stimuli. In this case, the user is presented with each stimulus individually, and asked to rate it on a pre-determined Likert-type scale, as used by Van Erp, for instance, to study tactile melodies 26 [27]. Since this method requires only one judgment per stimuli, it is only 0(n), while pair-wise comparisons require the full O(n) comparisons. Sorting tasks require a number of comparisons between those two extremes, though the exact number is not f ixed due to its dependence on the individual sorting strategies of each of the users. Nonetheless, these three techniques span the range of data gathering methods for M D S , each with their own strengths and weaknesses, to be discussed below. 4.1.1 Incomplete Dissimilarity Matrices The insight behind incomplete dissimilarity matrices is that it is not always necessary to have difference ratings for all pairs of stimuli from all subjects. Spence and Domoney [22] investigated how incomplete dissimilarity matrices can be dealt with in perceptual M D S methods, when the data comes from a standard pair-wise comparison task. For every dissimilarity value d(i,j), comparing the ith to y'th stimulus, a pair-wise comparison task requires that those two stimuli are presented to a user, and that the user provides a rating of how perceptually similar the two stimuli are. In this case d(i,j) = d(j,i) because judgments are symmetric and the order of stimuli does not matter, requiring n(n - l ) /2 total dissimilarity values. Spence and Domoney make two important claims. First, that for an individual, it is not necessary to have a complete dissimilarity matrix in order to get an accurate result from M D S , though each stimulus must have at least one dissimilarity value (connecting it to at least one other stimulus in the matrix), and it should be possible to move from any one stimuli to any other, by chaining along dissimilarity values (i.e. there are no unconnected islands of stimuli). Second, that since each judgment in a pair-wise comparison task is independent, it is possible to combine multiple incomplete dissimilarity matrices from different users to create an average, complete dissimilarity matrix. The first claim frees us from having to always guarantee that each individual compares every stimulus in the set, while the second claim means that we can combine dissimilarity values from different individuals to make a total, averaged picture—though unfortunately Spence and Domoney's work on incomplete matrices was performed on data from individuals, rather than averaged values, so the combinations of these two claims cannot be directly made.. 27 Most encouraging of Spence and Domoney's results was their finding that even with one-third of the entries in an individual's dissimilarity matrix removed (either at random or in a cyclical pattern), the resulting difference map ( M D S output) varied less than 10% from the map derived from the complete matrix. This result is promising, indicating that we need not be overly concerned with getting every difference rating for a set of stimuli. That there is a certain amount of looseness to how many difference ratings must be gathered, and from whom they must be gathered from, gives us much greater room to devise new data gathering methods of our own. However, i f we wish to gather only particular dissimilarity values in either a cyclical or random fashion as Spence and Domoney recommend, we are required to use pair-wise comparison between each •stimulus, because it is the only technique that allows you to individually pick the exact dissimilarity values you want without getting any other values, unlike other data gathering methods such as sorting. The number of pair-wise comparisons needed for a set increases exponentially with the number of stimuli. Removing one-third of all entries may be a fairly large amount, but it is still one-third of a quadratically increasing amount, meaning that the incomplete dissimilarity matrix method wi l l always eventually run into issues of having too many stimuli to judge in one experimental task. Furthermore, pair-wise comparisons have been found to have almost twice the level subjective fatigue as compared to sorting tasks [1]. So this method is limited both in number of stimuli that can be compared as well as accuracy of ratings given. 4.1.2 Sorting Tasks Whereas Spence and Domoney show that it is not necessary for each participant to compare every stimulus to every other stimulus in the set, the sorting task method seeks to make the act of comparing stimuli much more efficient by having subjects compare all stimuli at once and sort (or categorize) them into discrete groups based on perceived similarity. Dissimilarity matrices can then be created using the number of times that two stimuli occurred in the same group as a measure of their similarity (and inversely, their 28 dissimilarity). Often participants are required to sort the stimuli multiple times into different numbers of groups, in order to give varying levels of resolution. A variant of this method was used by MacLean and Enriquez [19] and analyzed in [20], to determine the perceptual characteristics of a series of haptic stimuli. Their results show that allowing users to sort a large set of stimuli into different numbers of groups provides a strong and robust measurement of the perceived differences between a fairly large set of stimuli, while greatly shortening the overall time of the experiment. Unfortunately, even this method is still limited in the number of stimuli that can be judged. Maclean and Enriquez tested 30 stimuli in their study, a number significantly lower than our own goal, and informally guessed that a maximal reasonable set size by this method and using stimuli of this sort is 40 or 50. 4.1.3 Per-stimulus Judgment Tasks The per-stimulus judgment task is a particular type of data gathering method, that differs largely from the previous two methods in that it provide a confirmatory rather than explanatory description of the stimulus set. Van Erp and Spape [27],' in a study particularly relevant to our own, analyzed 59 different tactile melodies by asking participants to judge the melodies according to 16 different pre-determined criteria such as "cheerful" or "polished," each on its own 5-point Likert scale. This allowed them to get judgments on all 59 melodies within a reasonable time-span and gave them data to which they were able to apply M D S . However, it is clear that they approached the data with a fixed belief about what aspects of the stimuli would be important to peoples' perceptions—specifically the 16 criteria on which they asked participants to judge the stimuli. Though one could, by finding unexpected correlations between parameters, perhaps indirectly discover new features of the stimulus set, it would be difficult to directly discover any completely new and unforeseen perceptual parameters. Whi le this technique may be acceptable when some idea about the nature of the stimuli already exists, in our own case we know so little about the stimuli that we wish to make no assumptions about how people wi l l perceive them going into our experiment, so.as to minimize any bias we might have on the results. 29 4.2 Design of Proposed Subset Data Gathering Method Though the methods described in Section 4.1 have been widely used to handle large sets of stimuli, they still fail to sufficiently reduce the time and effort needed to gather data for the goals defined here. Though the per-stimuli judgment task would be fast enough, it contains too many assumptions about the stimuli for our purposes. Both sorting tasks and incomplete matrices are valid attempts at reducing the number of comparisons needed to get useful M D S data, but again, take too long to be practical. Sorting 84 haptic stimuli takes approximately two hours to perform. Yet it might seem strange that we so quickly came to the limits of the existing methods for dealing with large stimulus sets. The reason this is so is because of the nature of haptic stimuli, especially in our own case. With a two second duration, the comparison of any two stimuli wi l l take at a bare minimum four seconds, and l ikely much more if an individual wishes to feel the stimuli multiple times. Compared to visual stimuli, which can be viewed simultaneously and in the manner of a few milliseconds, it quickly becomes clear why data gathering for our own stimuli is so much more difficult. Add to this the general lack of experience people have with haptic stimuli (compared to aural or visual stimuli) and we are confronted with a situation where the cost of each single comparison is considerably higher for haptics compared to other modalities that M D S is regularly used for. Given this difficulty, we asked whether the sorting task could be combined with incomplete matrices to further cut down on the number of comparisons need to gather perceptual data. This idea forms the basis for our novel M D S data gathering method: using a sorting task on a subset of the total stimuli, and building up an aggregate result by piecing together dissimilarity data from multiple differing subsets. B y using less than the total number of stimuli in a sorting task, we can ensure that a participant wi l l be able to complete their experimental task within a reasonable timescale. However, splitting up the simulus set into subsets creates several difficulties which are discussed throughout the remainder of the chapter. As soon as each participant no longer experiences every stimulus in the set, many potential issues arise of study design (which and how many 30 stimuli should they get) and study validity (can judgments given from only a subset of the stimuli apply to the whole). Through informal testing it was determined that a subset of size 50, with 3 complete sortings into 3 different numbers of groups, would take a participant roughly an hour to complete, and so this became our target subset size. By giving participants different subsets that cover different portions of the set of stimuli, we can gather dissimilarity data about all of the stimuli in the total set. Averaging together the results from the different participants can then give us a total picture of the perceptual space for a given set of stimuli. What this subset method of gathering M D S data does is essentially forego unreasonably long individual experiment session durations by using a larger number of participants to gather the same amount of data. 4.2.1 Creation of Subsets The primary challenge to this new method is in determining how large to make each subset, and how to distribute stimuli amongst subsets in order to ensure each individual's results can be aggregated into the whole to produce accurate overall judgment ratings. Because we wished to avoid biasing results, we chose to create random subsets, giving each random subset to just one participant to judge. We hoped this would minimize data bias due to the way a particular subset was-perceived. However; the creation of random subsets is actually a somewhat more complicated matter if we are concerned with gaining an even coverage of judgments across the entire dissimilarity matrix. Uniform coverage is desirable because it minimizes the number of participants needed in order to achieve a required number of observations for each point in the matrix. To this end, we developed a program that attempts to minimize the number of randomized subsets required to ensure that each value in the dissimilarity matrix has at least the specified number of observations. The algorithm is given the total size of the stimulus set that is to be used, the size of the subsets desired, as well as the minimum number of observations that each point in the dissimilarity matrix needs to have, and produces as many randomized subsets as is required by the given parameters. 31 Unfortunately it is non-trivial to produce a group of subsets which provide only the requisite number of observations to each point, due to the fact that every stimulus that is added into a subset wi l l be compared with all other stimuli in the subset. For example, i f a subset contains stimulus 2, and we are still lacking comparisons of stimulus 2 with 3, and 2 with 4, it is impossible to get those comparisons without also getting a comparison between 3 and 4. If another subset already exists with stimuli 3 and 4, then an overlap between the two subsets is unavoidable. This problem becomes progressively more complicated as the number of subsets increases, requiring more and more comparisons in order to achieve a minimally overlapping set. We have dealt with this problem with the following algorithm. " Descript ion of Subset A lgor i thm Our algorithm contains a two-dimensional array which keeps track of the number of observations (NO_cur) for each value in the dissimilarity matrix, and tries to make sure each value reaches the minimum number of observations (NO) without going over. It does this by continually adding in stimuli to new subsets, trying to bring NO_cur up to NO for each value. Thus at all times a list is kept for each stimulus detailing how many observations are needed against which other stimuli. This list is called a stimulus' free-set (as in, there exists free space to add in new observations), and it is essentially a list of stimuli that this stimulus still needs to be compared with (which means they must appear in the same subset). The algorithm begins by selecting the first stimulus to be placed in a new subset, choosing the stimulus with the largest free-set (i.e. the largest number of stimuli it still needs to be compared against). Thus the most "greedy" stimuli are always dealt with first. Then the following loop begins: • A stimulus is selected that is in the free-sets of all the simuli already in the subset. • If there is not one single stimulus that all the stimuli already in the subset have in their free-set (as is often the case), then the new stimulus is selected according to the following criteria (in decreasing priority): 32 1. The new stimulus should occur in the largest number of free-sets of stimuli already in the subset. If the stimulus is not in an already selected stimulus' free-set, the previously selected stimulus will end up with more than NO observations for that point. By minimizing the number of values in the matrix that receive more than NO observations, we ensure that we get as even coverage of observations as possible. 2. Provided the new stimulus is in as many free-sets as possible, the next check is how much of its own free-set overlaps with the other stimuli's free-sets. The stimuli with the largest amount of overlap is chosen. This will increase likelihood of meeting criteria 1 when more stimuli are selected in the future. 3. The new stimulus should have the largest free-set possible (ie, it should be the most "greedy"). Though ultimately, the greedy stimuli need to be dealt with most urgently, if we only ever grabbed the greediest stimuli without regard to anything else, we might quickly reach a point where it is impossible to add in new stimuli to the subset without creating many values in the matrix with greater than NO observations. • Once the new stimulus is chosen, it is noted which required observations have now been accounted for (and which non-required observations have now been added as well). • The process then repeats itself until the subset has been filled, and then starts again on a new subset until all required observations have been filled. Pseudocode User specified constants: NO - minimum number of observations needed for each value i n d i s s i m i l a r i t y matrix Stimulus_set_size - si z e of the t o t a l stimulus set to be used Subset_size - si z e of subset to be used Main Loop: Variables: MATRIX - 2D array, of Stimulus_set_size, used to keep track of how many 33 observation each value i n the d i s s i m i l a r i t y matrix w i l l have, given the subsets thus s p e c i f i e d NO_temp - number of observations that we wish to ob t a i n i n t h i s i t e r a t i o n of the loop For NO_temp = 1 to NO ' While MATRIX s t i l l has values < NO_temp, do C a l l CreateSubset Add observation to MATRIX caused by new subset Save new subset to f i l e End loop End For CreateSubset: Variables: SUBSET - a r r a y returned c o n t a i n i n g a l l s t i m u l i i n t h i s subset STIMULI - a r r a y of a l l a v a i l a b l e s t i m u l i that could s t i l l be added to SUBSET While SUBSET < Subset_size Do Populate STIMULI w i t h a l l s t i m u l i not i n SUBSET II II F i r s t C r i t e r i o n // For each stimulius i n STIMULI number__of_conflicts = how may values i n MATRIX would be > NO_temp, i f stimulus was added to SUBSET I f number_of_conflicts <• min_conflicts, min_conflicts = number_of_conflicts End For Remove a l l s t i m u l i from STIMULI w i t h number_of_conflicts > min_conflicts I f s i z e of STIMULI i s one, 34 add remaining stimulus to SUBSET, i t e r a t e loop // // Second C r i t e r i o n // For each stimulus i n SUBSET free_list = l i s t of s t i m u l i that s t i l l need to be compared to stimulus i n order to reach NO_temp End For subset_free_list = i n t e r s e c t i o n of a l l s t i m u l i ' s f r e e _ l i s t I f subset_free_list i s empty, s k i p to T h i r d C r i t e r i o n For each stimulus i n STIMULI free_list = l i s t of s t i m u l i that s t i l l need to be compared to stimulus i n order to reach NO__temp overlap_size = s i z e of i n t e r s e c t i o n of free_list and subset_free_list I f overlap_size < mih_overlap, min_overlap = overlap_size End For . ' •"' -Remove a l l s t i m u l i from STIMULI w i t h overlap_size > min_overlap I f s i z e of STIMULI i s one, add remaining stimulus to SUBSET, i t e r a t e loop // // T h i r d C r i t e r i o n // For each stimulus i n STIMULI free_set_size = number of values i n MATRIX along stimulus' row or column that are < NO_temp I f free_set_size < min_free_set_size, min_free_set_size - free_set_size 35 End For Remove a l l s t i m u l i from STIMULI w i t h free_set_size > min_free_set_size I f s i z e of STIMULI i s one add remaining stimulus to SUBSET, i t e r a t e loop E l s e randomly choose stimulus from STIMULI, add to SUBSET, i t e r a t e loop End Loop Problems wi th algor i thm Because this algorithm was not the focus of this thesis, the actual end program written takes one major shortcut for the sake of efficiency and simplicity. In order to find a truly minimal number of subsets, the calculation performed for Criterion 2 should not just check whether the next stimuli wi l l have a maximum amount of overlap1'with the free-sets of the stimuli in the subset, but should also check whether stimuli that are in that overlapping free-set, if chosen, would produce good results. That is to say, it is possible that choosing a stimulus that has a smaller amount of overlap compared to some other stimulus might actually work out better in the long run, because the stimuli that are in that overlap might in fact be better choices than the stimuli in the larger overlapping free-set. That is, choosing these stimuli might not lead as quickly to a point where the only stimuli that can be added in wi l l create overlap points where the number of observations is greater than NO. We realized this error, but had to cut short our development time in order to proceed with the rest of our research. Consequently the subsets created are not necessarily the most mathematically optimal non-overlapping subsets, though for our purposes they do provide reasonable coverage and randomization (see Appendix C for examples of actual subsets used in our studies). One exception to this is the tendency for the distribution of overlapped and non-overlapped points to clump together, as stimuli chosen in earlier sets tend to be used less in the later sets, which are more constrained in which stimuli they can select. A more 36 thorough and mathematically complete algorithm could be developed, but is outside the scope of this work. This effect can be seen clearly in Chapter 7, Sections 7.2 and 7.3, where this subset method is used in a full study and its results are discussed. 4.2.2 Robustness and Scalability As this new data gathering method is designed to accommodate larger sets of stimuli, it is reasonable to ask how large a set this method can handle. The tradeoff that our method offers is that instead of increasing the length of time that a particular individual must spend judging stimuli, an experimenter may simply increase the number of individuals judging stimuli, with the amount of time per individual staying constant. Once an experimenter has figured out how large a stimuli set a person can be reasonably expected to sort within a target time frame (usually an hour), and decided what the minimum number of observations each point in the aggregate dissimilarity matrix should have, then our subset algorithm wi l l be able to provide as many subsets as needed to acquire the requisite number of observations. At this point it is simply an issue of finding enough participants to run through each of the subsets, and then the data wi l l be collected. Thus, theoretically, whatever the set size, it is only an issue of using enough participants in order to gain the necessary data. • . However, in reality there are several concerns in regard to the scalability and robustness of this technique in the face of increasingly large stimuli sets. The first is the size of the subsets used to gather judgments: the smaller the subset, the more participants required to gather data, as well as the greater the potential for disagreement in judgments from different subsets, especially if the superset is large or perceptually complex. Another concern is the number of subsets (and thus participants) that wi l l be required to satisfy the total number of observations specified. Last is the number of overlapping observations required in order to overcome any variability brought about by the large number of different participants contributing to the overall average, as well as any noise brought about by subsets whose small size might create idiosyncratic judgments from participants. 37 Size of Subset Requi red F o r Data Col lect ion One of the first issues an experimenter must deal with when using the subset method is determining the size of the subsets to be used. In order to gather the necessary judgment data as quickly as possible, as large a subset size as possible is desired (the exact benefit, in terms of decreased number of participants, is discussed later below). Consequently, it is suggested that through trial runs with sample participants, the maximum size of stimulus subset that can be sorted within a reasonable timeframe (usually about an hour) be determined for a given hardware and stimulus set combination. However, the maximum size is not the only concern with regard to subset size: there is also the issue of whether there is a minimum size which a subset must be larger than, in order to gain judgments that, when averaged together, wi l l accurately reflect the entire stimulus set. Though finding a time-driven maximum subset size is not overly difficult to determine, the issue of minimum subset size is slightly less clear-cut. Small subset sizes would limit the number of different stimuli a participant was exposed to, giving them a smaller "world view" from which to make their judgments. Whi le their judgments within this world view would be valid, averaging them with other judgments that came from different world views would l ikely cause noise in the data. It is always possible to gather more observations in an attempt to counter the noise, but i f the subsets were different enough it could be that no amount of observations could cause the values to converge in agreement. What would likely determine if a given subset size was too small to produce converging results would be the actual number of underlying perceptual dimensions of the total stimulus set. The more perceptually complex the stimulus set (i.e. the more dimensions it has), the more variability there might be in judgments from different subsets. If the stimuli were only ever perceived as being " A " or " B " then even with very small subset sizes, there would likely be very little disagreement about how each of the stimuli were grouped. It is when there is a wide variety of stimuli that subsets can end up with far more of one type of stimuli than another, and perhaps another type of stimuli not present at all. It is this type of uneven distribution that would cause greater variability between 38 subsets and thus would require greater numbers of observations to counteract. This creates a somewhat paradoxical situation: how can we know how complex the stimuli set is (and thus the size of subset needed), when that is the very thing that M D S is supposed to discover? Thankfully, practical considerations ensure that we rarely come truly face-to-face with this issue. To begin with, as alluded to above, and discussed more thoroughly below, very small subset sizes are quite impractical since they require a huge number of subsets (and thus participants) even to gain a bare minimum number of observations. Secondly, when deciding upon a stimulus set to study, experimenters are rarely without any intuition as to how many perceptual dimensions there might be—it is usually clear roughly how perceptually complex a set of stimuli is. Furthermore, in analysis of perceptual M D S results, it is very rare to deal with a result of dimensionality greater than 4 due to problems of visualization; a quick review of papers involving perceptual M D S finds very few analyses larger than even 3 dimensions. Consequently the level of perceptual complexity that a stimulus set has is often specifically designed in order to ensure its interpretability. This does not mean experimenters could not unwittingly produce a stimulus set too complex for the size of subset they specified, but careful selection of stimulus set, as would be done for any M D S study, wi l l l ikely minimize this danger. As a general rule, an experimenter should ensure that the subset size is large enough such that several stimuli that exhibit any given type of parameter (and any of the particular levels that that parameter might have) be present in any random subset. Though, as discussed above, there is no guarantee that there might be unforeseen characteristics in the stimulus set, as long as each of the various known (or assumed) major parameters has some representation in each subset, then it is l ikely that most important perceptual characteristics wi l l be gathered. The simplest way to achieve this is to have a subset size as close to the superset size as possible without the subset become too large to sort with a reasonable effort. Of course this is generally not possible (since it was too-large supersets that this technique was designed to deal with in the first place), and so, as the subset size decreases and the chances of particular dimensions of the stimulus set being left out of a 39 given subset increase, more randomized subsets (judged by more participants) are needed to counteract the increased level of noise and disagreement. Number of Subsets Requi red F o r Data Col lect ion When each participant is tested on a unique subset, the number of subsets required is equal to the number of participants required. We wi l l discuss here a version of our subset creation algorithm in which each participant is tested on a unique subset, which provides the specified number of observations in the fewest subsets possible. Repeating subsets with multiple participants can also be done if desired (this is, in fact, done in Chapter 7 to help validate the subset method), but is less efficient in terms of number of participants run. In our subset creation algorithm, there are three factors that affect the number of subsets required for an experiment: the size of the total stimuli set (NTT), the size of the subset ( / V S ) , and the minimum number of observations required (NO). In our own practical experience, a bare minimum of five observations per value in the dissimilarity matrix is required for reasonable results, though there are exceptions, as discussed below. Assuming that the size of the subsets is large enough to capture significant characteristics of the stimulus superset, as discussed above, and holding the minimum number of observations constant, it is the ratio between NS and NT that affects the total number of subsets required. The total size of the stimulus set does not have an effect, as the coverage of both subset and total set grows as a quadratic function of their size, thus ensuring that our subset algorithm wi l l produce the same number of subsets for a pairing of NS = 50, NT = 100 as NS = 5, NT = 10; there wi l l simply be a factor of ten fewer stimuli in the subsets produced for the latter as opposed to the former. In Figure 4.1, we show a curve of the number of sets required to obtain at least five observations in each point in a dissimilarity matrix plotted against the NS/NT ratio. As can be seen, the smaller the ratio, the greater the number of subsets required, to the point that any ratio lower than approximately one third wi l l l ikely require far more participants than any experimenter would be wil l ing to run. What this means is that though our new 40 technique is theoretically unbounded, in reality there is a cap on how large a stimulus set can be tested with it. We cannot state exactly what the upper limit on total set size is though, because it is dependant on how many stimuli could be put in a subset. For our purposes, with our target subset size of 50, we would probably not want to have a total stimulus set size of much greater than 150. However, our target size of 50 is at least partly limited by our means of sorting and the nature of the stimuli themselves. Number of Observations Requi red for Data Col lect ion The curve in Figure 4.1 represents, in truth, a lower bound for the number of subsets needed for a given NS/NT ratio, as NO is fixed at five, a value we have found sufficient for our own purposes, but may under other circumstances be insufficient. More observations are needed to deal with noisy data, which can result from two main causes: greater variance within the pool of participants, or toorsmall subsets. Individual differences wi l l always be a factor, but the effect of subset size on the noisiness of the judgment data wi l l vary from stimulus set to stimulus set. As discussed above, the likely determining factor in the size of subset (and thus the number of observations needed) is actually the true underlying dimensionality of the stimulus set. Obviously if NS was equal to NT, then all subsets would be the same and the only cause of variance would be from individual differences. But as NS/NT gets smaller, the difference between individual subsets increases, as they have less chance of overlap. This means that the view each participant has on the stimulus set differs by more and more. We advocate completely randomizing subset selection in order to cover over differences between subsets, so that each aggregate dissimilarity value is built up of values from enough different subsets so as to cover over any large, subset-specific variances. Given this, it would seem clear that smaller subset sizes would require a larger number of observations per dissimilarity value, in order to deal with the higher level of variance. Perceptual judgments from an NS of 2, for example, would l ikely differ hugely, and it may even be that no number of observations would ever create a complete picture of the stimulus set with such a small NS. 41 Thus in each case, the particular combination of NS, NT and NO that wi l l meet an experimenter's needs wi l l have to be decided'individiially. Nevertheless, given a not too diverse set of stimuli, our method can cut the number of stimuli that need to be presented to a user by a third, while still only requiring a very feasible number-of participants to run the experiment. Such a decrease is of great use for our own goals, but also of general use to anyone who wishes to gather perceptual dissimilarity data about a large set of stimuli. 4.3 Potential Threats to Validity of Method The subset method combines two previously validated experimental methods—sorting tasks and incomplete dissimilarity matrices. Joining the two together, however, by no means implies the validity of the combination. Certain characteristics of the sorting method throw into question the validity of results produced by using comparisons collected using anything other than the entire stimulus set. 4.3.1 Incomplete Individual Results Firstly, compared with the straight pair-wise comparison task where it is easy to remove just a single comparison (because each comparison is independent of all others), in the sorting task it is impossible to remove one comparison without removing all of the comparisons of a given stimulus. This is because in a sorting task all stimuli present are compared against all others, so removing one stimulus removes all the comparisons of that stimulus against all the other stimuli present. Thus there is no way to create an M D S plot from just one individual containing all of the stimuli in the set (as in the incomplete but completely connected set advocated by Spence & Domoney [21]), and if each individual is given a different subset, it also means that each individual's M D S plot wi l l involve (at least some) different stimuli. Consequently it wi l l be very hard to compare individual M D S plots directly with each other, as a means of determining how consistent different people were in judging the stimuli set. Comparison of individual results is a useful tool in proving the quality of the averaged results, and the subset method is hurt by not having a direct means of performing this comparison. 42 4.3.2 Subset-relative Judgments Unfortunately, creating average results from a series of incomplete dissimilarity matrices is also problematic due to the inter-dependant nature of all perceptual judgments performed in a sorting task. In a pair-wise comparison task, each comparison is dependant only on the two stimuli being compared (subject learning effects over time are assumed to be negligible). However, in a sorting task, each stimulus is being compared, either explicitly or implicit ly, relative to all other stimuli present, and thus a change of one stimulus could completely change the groupings of all other stimuli. What this means is that an incomplete dissimilarity matrix produced by a sorting task comparison of one subset of stimuli versus a different but overlapping subset of stimuli could produce radically different dissimilarity values even for the stimuli in the intersection of the two subsets. As an example, consider a total stimulus set in which one stimulus was played at ten times the amplitude of any of the other stimuli. If a participant was presented with a subset without the very loud stimulus, the relative amplitude differences of the remaining stimuli would seem more salient. However, i f the very loud stimulus was present in the subset, the participant might now judge all the remaining stimuli to be at the same amplitude level, because the differences between the rest of the stimuli is so small compared to the difference between the very loud stimulus and all the rest. In this way the presence or absence of one stimulus could potential produce very different sorting strategies leading to very different results. Thus any attempt at averaging over all dissimilarity values could be potentially covering over a very noisy set of data, producing averages that essentially reflect no real-world population. However, it is not clear how strong an effect the relative nature of these judgments would have on the resulting dissimilarity matrix; or stated another way, how many repetitions, assembled from many participants, would be required in order to diminish the effect of this noise (as discussed in Section 4.2.2). This potential threat to validity is why we emphasize the randomization of subsets. We believe that cases of the above happening wi l l l ikely be fairly rare, if the stimulus set is well designed. Thus if all participants are presented with a unique, 43 randomized subset, we believe that instances of this problem occurring should, on average, be covered over by the far greater number of reasonable, well-formed subsets. 4.3.3 Ability to Discover Overall Perceptual Trends A n additional problem that arises from averaging together different subsets is that due to overlap points, some values in the dissimilarity matrix wi l l be averaged over observations from a larger number of participants than other values. This might, in essence, make certain dissimilarity values more "trustworthy" than others—that is, less l ikely to contain aberrant, fluke results. One option for dealing with this problem is to use some sort of weighted M D S , with each dissimilarity value weighted according to the number of observations or the standard deviation. However, by choosing to use a method that specifically aims to make the number of observations per value in the dissimilarity matrix as even as possible, we can deal with this issue without resorting to more complicated M D S models. Thus it wi l l be important in experimentation to check the number of observations and/or consistency of the standard deviation of the various values within the dissimilarity matrix to ensure that they are not having an adverse effect on the M D S results, though we wi l l not have a strict mathematical means of analysis for these features. 4.4 Pilot Study: Initial Study on Voicecoil Vibrators A n exploratory pilot study was run to determine the issues involved with both the rhythmic haptic stimuli discussed in Chapter 3, as well as the subset method of data gathering for M D S discussed above. The results of this study were used to inform the more thorough and detailed studies discussed throughout the remainder of this thesis. 4.4.1 Apparatus For this study, vibrotactile stimuli were emitted from the transducers V B W 3 2 Skin Stimulators from Audiological Engineering Corp. M A . The peak frequency transmitted by the device is 250Hz with a usable output range from 100Hz to 800Hz. The transient response of the device is 5ms. The experimental software responsible for presenting the vibratory stimuli was written in V B 6 , which logged results in a .csv format. The experiment was run on a Del l laptop running Windows X P . 44 4.4.2. Participants Thirteen university undergraduate and graduate students (8 females) with ages ranging from 19 to 36 years were recruited for this study. 4.4.3 Stimulus Set V i a brief preliminary psychophysical testing, two frequencies and two amplitude levels were determined for use with the 20 of the 21 rhythms discussed in Chapter 3. We were still in development of our stimulus set at the time, and so did not use rhythm 4 in our set (see Table 3.2 for explanation of rhythm numbers). The two frequency levels were 150Hz and 300Hz, and the amplitudes were defined as the maximum volume output of the vibrators, along with the threshold amplitude level, as determined for each of the frequency levels. This resulted in a set of 80 stimuli, comprised of 20 rhythms x 2 amplitudes x 2 frequencies. 4.4.4 Procedure We generally followed the method of [19]. Participants sorted the entire 80-stimulus rhythmic haptic stimulus set using the apparatus described above. Each participant completed 3 sorting tasks on the same stimulus set. At the beginning of the study, participants were instructed to feel each stimulus by cl icking on to each numbered tile organized at the bottom of the screen, and to group stimuli that felt the same in the same boxes. Participants were also told that they could feel the stimuli as many times as they needed by cl icking on the tile again, and were allowed to change their mind about the groupings by cl icking and dropping the tile in the desired box. In the first sort, participants were told to group stimuli into whatever number of discrete, non-overlapping groups they felt was appropriate to describe the perceived dissimilarity between stimuli. For the remaining two sorting tasks, participants were required to sort the stimuli into a specified number of groups, either 3, 9 or 15. Of these three group numbers, the one closest to the number of groups chosen in the first sorting task was not used, with the remaining two numbers randomly assigned to the second and third sorting tasks. Having three repetitions of the sorting task performed on the same set of stimuli and varying the 45 number of groups that the stimuli are sorted into as we have done has been shown previously in haptic M D S studies to yield good resolution for perceived differences [20]. 4.4.5 Results and Discussion A n average dissimilarity matrix was constructed from the participants' data, and then run through the SPSS A L S C A L algorithm for 1 to 5 dimensions. Graphing the resulting stress values shows no clear elbow indicating a point of diminishing returns in terms of goodness of fit (Figure 4.2); instead both 2 and 3 dimensional results provide reasonable stress levels, while higher dimensions are somewhat decreased in improvement. For the sake of parsimony as well as ease of interpretation, the 2D solution was chosen as the primary solution for analysis, though the 3D solution is given some consideration as a secondary tool for analysis. Several features are immediately evident from visual inspection of the 2-D perceptual map (see Figure 4.3). First is the clear circular arrangement of the stimuli around the center of the graph. According to MacLean and Enriquez [19] this circumplex arrangement is a common result in perceptual M D S studies, including those involving haptic stimuli, resulting from judgments of stimuli as either very similar or very dissimilar according to the frequency and amplitude of the vibrations used, regardless of rhythm. This trend is further emphasized by the projection of the design parameters of amplitude and rhythm onto the perceptual map, as they both neatly bisect the map in nearly orthogonal directions. This projection is done by averaging the location of all the stimuli that have one value of the parameter, plotting the points, and drawing an axis between these points. The length and placement of these axes on the map indicate their overall importance in the perception of the stimuli. This result is consistent with those of MacLean and Enriquez who found frequency and amplitude were both extremely important perceptual features, as well as being highly correlated in terms of perception. The second salient point is the even spread of stimuli around the circumplex.distribution and the general lack of clustering amongst the stimuli. A cluster of stimuli around one position indicates that people perceptually group them together as being related in the 46 visible dimensions. A lack of any such clustering indicates an overall level of distinctiveness and discernability of our stimulus set, such that even stimuli that are perceived as being perceptually similar to each other are nonetheless perceived as distinct and separate sensations. MacLean and Enriquez [19] with a similar circumplex arrangement, nonetheless also showed clear clustering in mid-process results, a result distinctly different from our own. It is noted that in that previous work, iterations were performed for the distinct purpose of designing non-clustered stimuli set with maximum perceptual 'spread', with the M D S result guiding adjustments. In other cases, clustering might be desirable in order to promote ' family' associations of meanings (e.g. Enriquez, Chita & MacLean [12]). Thus we are left with the question of why the M D S plot shows no clear clustering, while other similar studies have. Specifically we wonder what effects rhythm has had on the perception of the stimuli, and whether it has played any role in this lack of clustering. The only clear rhythm effect is the one outlier stimulus, situated far outside the circumplex of stimuli, which is the full 4 quarter note rhythm, stimulus 1, played at the highest frequency and amplitude. That this is the most distinctive of all rhythms is to be expected, as it was the simplest according to our interpretation of rhythm, and it was played at the most clearly discernable frequency and amplitude combination used. Yet the remaining stimuli's marked descent into a cloud of opaque rhythm effects is made all the more frustrating because of the tantalizing promise of this one outlier. We are thus left with ambiguous, noisy results and several possible explanations for this ambiguity. Given that the experimental method used was untested, one obvious possible explanation is that the new data gathering method introduced too much noise into the data. However, since the stimulus set is also unique and untested, it could also be suggested that the results of this study accurately reflect the difficulties people had in perceiving similarities amongst rhythmic haptic stimuli. Yet another issue is that vibrators used to display the stimuli may have lacked sufficient dynamic range and responsiveness to effectively display the more complicated haptic stimuli used here. The 47 difficulty in resolving these possible explanations led us to design and implement the studies described in Chapters 6 and 7, hoping to determine the validity of the experimental method used and the true perceptual nature of the rhythmic haptic stimuli created. 48 Chapter 5: Methods The apparatus used in the pilot study - voicecoil vibrators attached to the sound-card output of a P C - has been used before successfully to create and test haptic stimuli. However, we felt that we may have been approaching the limits of this setup. The voicecoil vibrators were not as precise in their output as might be desired, leading to worries that our more complicated haptic stimuli might not be being displayed in complete detail. Furthermore, the voicecoils were separate from the P C , making them somewhat awkward to naturally introduce into regular application use. Thus when the opportunity arose to use prototype hardware from Nokia based on piezo technology, we gladly took advantage of it. It offered more precise timing and control over feedback, and an embedded platform that could support new haptic applications without any additional peripheral devices. In 5.1, we describe the hardware, a relatively new type of handheld haptic display on which very little haptic icon work has been done previously. In 5.2, we discuss the sorting program that we were required to write, in order to perform data gathering on the new handheld platform. 5.1 Discussion of Hardware Platform The Nokia 770 (Figure 5.1) is a handheld internet tablet, with a large (90x54 mm) high-resolution (800x480) screen, ARM-based processor, and runs a modified version of the Debian Linux distribution. While the 770 is already commercially available, Nokia has added haptic feedback to a prototype model, identified as the 770T (see [16] for details). Though visually identical to the 770, the 770T has a piezo-mounted touchscreen, which allows the screen to be pulsed with small displacements in the axis orthogonal to the screen, giving the sensation of a single "c l ick" when done once, and of a continuous vibration when done repeatedly at tightly spaced intervals. This technique can give quite convincing and satisfying haptic feedback, all within the context of a handheld device. We are much indebted to Nokia for supplying us with several of these devices along with their technical support. What follows is a discussion of the new hardware platform's 49 suitability to our own ends, with regard to creating and analyzing a large set of rhythm-based haptic stimuli. 5.1.1 Control of Haptic Feedback Haptic feedback in the 770T is controlled through the use of feedback scripts, which are compiled into byte-code and sent to the hardware that controls the piezos. The feedback scripts consist of a series of commands for driving the piezos. There are five main commands: charge, discharge, delay, loop and voltage set. The charge command tells the device to begin charging the piezos and can specify the speed (by specifying the resistance of a current l imiting resistor) that the piezos wi l l be charged at. This creates the leading edge of a single "c l ick" motion. The discharge command causes the piezos to discharge, thus creating the trailing edge of a single "c l ick" motion. The delay command is used to specify timing between clicks and between charges and discharges. The loop command is used, as would be expected, to simply specify the number of times a set of commands should be repeated. Lastly the set voltage command sets the overall voltage level to which the piezos wi l l be charged. No more than 255 total commands can be used in any one feedback file. These feedback files, once compiled and loaded into the hardware, can be associated with a given type of G U I widget (for example, a button or a scroll bar) or specific individual widgets, and the feedback wi l l then be played whenever the cl ick event for the specified widget is fired. The 770T hardware only has space for 16 user-defined feedback files to be loaded into the hardware at one time, though multiple widgets can be mapped to the same feedback file. In order to create sustained vibrations which can be used to make up a rhythm, consecutive series of closely spaced clicks had to be placed together to build up what is essentially a square wave playing at a given frequency. These vibrations give us the notes that can be used to make rhythms, while the delay command gives us the off-notes. Thus a single haptic feedback file could be used to make an single haptic stimulus from our rhythm set. 50 5.1.2 Baseline Perceptual Data According to [22] the most perceptually salient parameters to be varied with the piezo touchscreen are the duration of the voltage curve and the speed at which the leading edge of the curve rises. These parameters corresponded roughly to the perceived amplitude or "strength" of the feedback. This claim was confirmed via our own informal user testing; in further agreement with [22], our testing also showed that the height of the voltage curve had an insignificant effect on the perceived strength of sensation, thus it was decided that the default voltage level of 173 V would be used for all feedback. In order to create rhythms, we first needed to determine how to make continuous vibrations that were distinct. This necessitated a small, informal experiment in which users were presented with different combinations of feedback strength and vibration frequency, and asked to order them from strongest to weakest. For amplitude, we used wave durations of 0.5 ms (low) 1 ms, 2 ms, 4 ms and 10 ms (high amplitude) and resistance levels starting at 13.2 (low) and moving up to 1.0 kOhm (high amplitude) in 10 even intervals, thus controlling the sharpness of the leading edge of the voltage curve (curve rise time) as well as the length of the curve. High resistance levels produced low perceptual amplitudes because they decrease current thus slowing curve rise time Frequencies ranged from 150 Hz to 300 Hz , at 50 Hz intervals. From this it was found that there were generally four levels of perceived intensity of signal. Frequencies of 150, 200 and 250 Hz were all perceived essentially the same; they felt very strong and distinct to the touch. Vibrations played at 300 Hz felt much softer. Only the two extremes of voltage curve rise time were distinctively different, but they did tend to dominate the perception of curve duration. Voltage curve durations of greater than 1 ms were found to cause no perceptual differences when occurring within a vibration, while the difference between 1 ms and 0.5 ms was evident, but perceptually it was generally overwhelmed by frequency and curve rise time. Thus for the purposes of creating rhythms, we selected one wave duration (1 ms), two curve rise levels (1.0 and 13.2 kOhm), and Wo frequencies (200 H z and 300 Hz). Thus we have a high and a low 51 amplitude and a high and a low frequency vibration, providing 4 different vibrations that we could use for our rhythms. 5.1.3 Advantages and Disadvantages of Hardware From our initial experiences using the 770T, we observed that compared to the voicecoil vibrators as used in the pilot study, the piezo-driven 770T provided much more crisp and precise feedback. Though it is perhaps not able to produce as strong a sensation, the quick reaction times of the piezos were felt to have delivered a much more distinct feedback with sharp starts and stops, whereas the voicecoils had more noise associated with its edges, creating feedbacks that were not as well defined, feeling "mushier" to the touch. This, coupled with the very precise timing control provided by the feedback scripts, gave us hope that the 770T would have an increased expressive capability, making it easier to distinguish small differences between haptic stimuli, and generally giving greater discriminatory power to our haptic stimulus set. Nevertheless, the 770T did have its drawbacks. As mentioned, amplitude, of feedback given was generally less than the voicecoil vibrators, but in addition to this there was a strong audio component to any feedback given on the 770T due to the vibration of the screen within the casing. This sound required noise-cancelling headphones to be worn at all times during any testing of feedback on the device, with fairly loud white-noise having to be played in order to drown out the sound, which can be fairly intrusive and annoying to users. Another serious drawback was the hardware limitations on the number of commands per feedback file and the number of feedback files that can be active at any given moment. Though these problems could be worked around, they did create difficulties in the development process and somewhat hampered the controllability (and ease of programming) of the overall system. Overall, we felt that the quality of the haptic feedback was well worth the switch to the new device. Furthermore, with an open-source operating system and a large (for its size) graphical display, it was felt that the 770T represented a strong platform on which to 52 develop new haptic applications, and thus was worth choosing as a device to characterize and study. 5.2 MDS sorting program As outlined in Chapter 1, two different M D S studies were required in order to both perceptually characterize our new rhythmic haptic stimuli set as well as validate our new method of gathering perceptual data. Both of these studies require gaining perceptual judgment data from users. Prior to this, we had used a simple P C setup, as in the pilot study. However, using a handheld, Linux-based platform for our studies necessitated a change in the stimuli sorting program that is used to collect perceptual dissimilarity data from the user. For several reasons it was no longer feasible to use the box-sorting technique utilized in our pilot study (and developed by MacLean and Enriquez [18]); the format was too space-intensive to fit on a small screen, and it was felt that having the box-sorting G U I on a desktop P C while having the user still needing to hold the 770 and interact with it using a stylus would be needlessly complicated both from a usability and a technical standpoint. Usability-wise, it would require constant switching between two different tasks on two different platforms, and from a technical side, it would require detailed communication between handheld device and P C , as well as a complete re-write of the box-sorting G U I for Linux rather than the Windows platform. Consequently, we designed a new interface that would allow the stimuli to be sorted using strictly the 770 with no other devices necessary. The main limiting factor in the design of the new interface was screen space. The 800x480, 90 x 54 mm screen does not provide enough space to simultaneously display buttons representing all stimuli as well as boxes which the buttons can be sorted in to. Especially when the user is expected to sort stimuli into a large number of groups, the box and buttons sizes required would be extremely small, something that is definitely a troublesome issue when we are relying on the hand-eye coordination of touch-screen interaction. The possibility of having multiple screens that the user must switch between 53 in order to access all stimuli and boxes was raised, but it was felt that it was important that all stimuli and groups be present and accessible atjall .times, so that no stimuli were neglected and all groups were at the same level of visual saliency. The method thus decided upon was to have a spatially static field of buttons that could be grouped by assigning different colours to the buttons, with each colour representing a different group. Whi le adding colour does bring with it certain limitations, such as inaccessibility to colour-blind users and unavoidable affective responses to certain colours, we felt that this was the simplest technique that would achieve our goals of having all groups and stimuli present in the interface simultaneously. With this interface, users can feel any of the stimuli by pressing on any of the buttons, each associated with one stimulus. The user can then add a button to a group by selecting one of the colours along the bottom and assigning it to the desired button. In order to decrease confusion, an automatic sorting function is provided, which simply places all of the buttons, sorted by colour, at the top of the screen, with the un-coloured placed after it. In order to aid with the sorting task, users were provided with sheets of paper with coloured squares printed on them corresponding to all the grouping colours, where they could write descriptive names for each group if they so desired. This helped users conceptualize and remember the groups they were sorting, as well as providing insight to the experimenters about how users were sorting the stimuli. 5.2.2 Loading Haptic Feedback We were successful in having all stimuli and groups equally accessible at all times both visually and physically, but hardware limitations forced us to introduce load times for playing some of the stimuli. As mention in Section 5.1, the 770 maps specific feedback files to types of G U I widgets, or specifically named widgets, but only provides space for 16 different user-defined feedback files to be loaded at any one time. As a result of this, only 16 buttons can play their particular stimuli immediately after being pressed. Any other stimulus has to be loaded first (a process that takes no more than two seconds), which, in turn, unloads one of the other 16 buttons that already had its feedback loaded. 54 To load an unloaded button, the user simply presses it once to load it, and presses it again to feel it. Loaded buttons are indicated with a " ! " next to their numbers. In order to minimize the amount of loading required by the user, both pre-fetching and a history queue were implemented for feedback loading. Thus eight out of the sixteen feedback slots were used as a history of the last eight buttons the user had pressed, while the remaining eight feedback slots were used to pre-load the nearest buttons next to any newly-loaded button. What this allowed the user to do is move from the top-left down to the bottom-right, having most of the buttons loaded ahead of time for him or her. Furthermore, since the immediate neighbours of any non-loaded button would also be loaded along with it, returning to feel what any given button felt l ike in a colour group would load all of the other buttons in the group provided the buttons have been sorted (see Figure 5.3 for an example). Though this does not remove all loading times, it does greatly decrease the total amount, making the sorting task less frustrating and time consuming for the user. 55 Chapter 6: Investigation of Rhythmic Haptic Stimuli (Gold Standard Study) In Chapter 3 we developed our stimulus set using a novel application of rhythm to tactile stimuli. As always with haptic stimuli, developing them was one challenge, but determining how they were actually perceived by people was another. The intuitive understanding of haptic perception that guided our design process is by no means a guarantor of how the stimuli wi l l be perceived by the broader public. Especially in the case of haptic rhythm, on which so little research has been performed, our knowledge is lacking. Thus in this chapter we seek a clearer picture of the important perceptual characteristics of our stimulus set, and how these relate to the design parameters we used to create the stimuli. 6.1 Purpose and Structure of Study The purpose of this study was to produce a thorough description of the perceptual characteristics of our rhythmic haptic stimulus set through the use of an existing, verified experimental method. In this study, we wished mostly to learn what characteristics of the stimulus set define its perceptual space—that is, the dimensions along which people perceive these stimuli as a group, as opposed to the engineering parameters used to construct them. Moreover, by using a verified method we also aim to produce a "gold standard" result, which our modified, subset method of data gathering can be compared against. To this end, we decided to use the sorting method of data gathering with the full set of 84 haptic stimuli; and in fact, this decision influenced the maximum set size we could test here. Since participants sorted the entire stimulus set, there were no concerns about participants making judgments based on only part of the total set, and thus the resulting aggregate dissimilarity matrix could be taken as a reasonable representation of the average perception of the total stimulus set. Whi le the sorting technique is a validated and widely used technique [19], it is generally used with a much smaller number of stimuli. Sorting a full set of 84 stimuli is a much longer and more involved process, with 56 participant fatigue becoming a major issue. At 2 hours, the experiment time is extremely long; realistically this was the absolute maximum set size that this sorting task can handle, and then only with carefully chosen subjects In order to minimize worries of fatigue, as well as trying to guarantee that our results would indeed constitute a "gold standard," several slightly unusual alterations were made to the study. Primarily, participants were solicited directly, with the express aim of choosing people who were dedicated and trustworthy enough to be vigilant throughout the task, as well as having already had some experience with vibratory tactile stimuli. This would help ensure the quality of the data gathered, with the obvious proviso that it may not reflect entirely accurately the perceptions of the general public. Furthermore, this resulted in all participants, in fact, being acquaintances of the experimenter, another biasing factor. However it was felt that using unmotivated, inexperienced users for a fairly long and arduous study would almost certainly give results too noisy and inconsistent to interpret. Several other allowances were made in an attempt to minimize the strain put on participants running the study, as wi l l be discussed in the "Method" section. We feel that these allowances, while deviating somewhat from the standard experimental method, actually help to guarantee that our results stand up in the face of such a large stimulus set. 6.2 Full-set MDS study As a well established standard for the study of haptic stimuli, M D S studies have been shown to be a great tool for discovering perceptual characteristics of novel sensations [19]. What follows is a description of the first of the two major M D S studies performed in this thesis. It represents our best attempt to create a clear perceptual description of the rhythm-centered stimulus set we made. 6.2.1 Method Six expert participants were solicited directly for the study. Whi le relatively few, the demanding criteria set for the participants made recruitment difficult but meanwhile 57 ensured a smaller amount of higher-quality data. Participants were 5 males and 1 female, all graduate students in computer science, with an age range between 24 and 40. The experiment lasted two hours, but participants were given the choice of breaking the experiment up into two one-hour sessions in order to minimize the fatigue of running the experiment. Of the six, two chose to break the experiment up, and four chose to do it in one two-hour block. Participants were compensated $20 in total for the experiment. Participants sorted the entire 84-stimulus rhythmic haptic stimulus set on the Nokia 770T tactile platform described in Chapter 5, using the program also described there. Each participant completed 3 sorting tasks on the same stimulus set. In the first sort, participants were told to group stimuli into whatever number of discrete, non-overlapping groups they felt was appropriate to describe the perceived dissimilarity between stimuli. For the remaining two sorting tasks, participants were required to sort the stimuli into a specified number of groups, either 3, 9 or 15. Of these three group numbers, the one closest to the number of groups chosen in the first sorting task was not used, with the remaining two numbers randomly assigned to the second and third sorting tasks. Having three repetitions of the sorting task performed on the same set of stimuli and varying the number of groups that the stimuli are sorted into as we have done has been shown previously in haptic M D S studies to yield good resolution for perceived differences [20]. Because of the auditory noise made by the 770T when playing haptic feedback, users wore Bose Quiet Comfort 2 1 acoustic noise cancelling headphones during the experiment. Whi le the normal procedure is to play white noise during testing to drown out the sound, it was felt that for obvious reasons listening to two hours straight of loud white noise would itself be an impediment to making well-reasoned judgments. Consequently, and since participants had already been selected for their trustworthiness and dedication, participants were allowed to listen to music self-chosen according to stated criteria, and told to self-monitor to ensure that no sound from the device could be heard. The criteria were simply that the music was consistently loud enough to mask the noise made by the 1 http://www.bose.com/controller?event=view_product_page_event&product=qc2_headphones_index 58 770T, and that it be emotionally neutral enough that it would not overly affect the participant's mood. The music played by the participants was periodically checked to ensure that it followed these criteria, and no violations were observed. 6.2.2 Basic Results As per the methods of MacLean and Enriquez [19], similarity values were created for each pair of stimuli by taking each instance of co-occurrence in the same sorted group as an indication of perceived similarity, adding a similarity score to that pair proportional to the number of total groups used in that particular sorting task. Thus higher similarity values were given when two stimuli were placed together during the 15-group sorting task than the 3-group sorting task. These total similarity values were then subtracted from 1000 to create a dissimilarity matrix for each subject. These individual dissimilarity matrices, covering the total 84x84 (symmetric) comparisons between stimuli, were then added together and averaged, to create an overall average dissimilarity matrix for the group. Analysis of M D S Plot The resulting dissimilarity matrix (Appendix A ) was then run through the SPSS A L S C A L algorithm for 1 to 6 dimensions. Graphing the resulting stress values (Young's s-stress Formula 1 was used) shows no clear elbow indicating a point of diminishing returns in terms of goodness of fit (Figure 6.1); instead both 2 and 3 dimensional results provide reasonable stress levels, while higher dimensions are somewhat decreased in improvement. Specifically, the 2D solution has an s-stress value of 0.36668 and an r 2 of 0 .47788, while the 3D solution has an s-stress value of 0.26630 and an r 2 of 0.58049. Both s-stress and r values range from 0 to 1, with low s-stress values showing better fit, while low r values show worse fit. These particular s-stress values are both relatively high, indicating the difficulty of fitting the data into the required dimensions; this is l ikely due to the number of stimuli used. Furthermore in both cases the r2 values are fairly low, indicating that around half of the variance in the data set was not accounted for in the M D S model. 59 However, given our own experience with M D S plots of haptic stimuli, such levels of stress and r2 are not necessarily signs that the plots themselves wi l l not yield informative and trustworthy results [20]. Given this, and the diminishing returns of the s-stress plot, for the sake of parsimony as well as ease of interpretation, the 2D solution was chosen as the primary solution for analysis, though the 3D solution is given some consideration as a secondary tool for analysis. Despite the relatively poor s-stress and r2 values, these solutions proved amenable to reasonable interpretation. 2D Results Initial examination of the 2D plot (Figure 6.2) shows a clear circumplex arrangement in the data similar to that found in the pilot. Overall distribution of stimuli along this circumflex is fairly even. A few denser clusters are evident, but only as part of a general trend of dispersed stimuli. At best only three very large clusters could be said to exist, but they are almost too broad to be of any interpretive value. Instead, several grouping trends according to amplitude and rhythm are explored in the next section. In analysis of M D S output, our main goal is to determine how the engineering parameters that were used to create the stimulus set map to the perceptual parameters that participants used to group the stimuli. In our stimulus set, three main engineering parameters were used: amplitude, frequency and rhythm type. Our main method of projecting engineering parameters onto the perceptual space is to average the values of all the stimuli in a given group as defined by some parameter, and treat the resulting point as a centroid representing the overall group that can then be compared against other groups. For example, one group might be all observations for a given amplitude, and another group all observations for another amplitude. Drawing a line between the centroids of two different groups (as done for amplitude and frequency in Figure 6.2) creates an axis from which we can further interpret the data. The length of this line can be interpreted as the strength of the effect of this parameter, because the length of the line is the distance between the two centroids of the two groups, and in M D S maps, distance in layout equates to magnitude of perceptual difference. 60 Thus projection of the design parameters of amplitude and frequency onto the perception space shows a very strong and very clean cut trend of amplitude, but almost no trend of frequency whatsoever (Figure 6.2). The dominance of amplitude in distinguishing stimuli is as expected, and agrees with the previous results in the study of haptic stimuli [19, 18, 20]. However the lack of effect of frequency is a unique result, counter to previous findings, and is discussed in the next section as well. Analys is of Standard Deviat ion Analyzing the standard deviation of the averaged values in the dissimilarity matrix provides encouraging results that broadly agree with the M D S output. The overall average standard deviation for all dissimilarity values for the matrix is 160.02, which, on scores of 0-1000 is fairly large, but not insurmountably so—prior research has shown positive results with comparable levels of SD [19]. More interesting is the distribution of the standard deviation over the dissimilarity matrix (Figure 6.4). Unl ike the pilot study, where the spread of high and low points of standard deviation was, as far as could be distinguished, completely random, there are clearly two different areas of S D , one with a higher overall S D , one with lower overall SD . These can be seen as the rough square of low SD (light coloured squares) in the middle of the half-matrix displayed in Figure 6.4, and the two darker corners (high SD) of the triangle. As stimuli numbered 1-42 were high-amplitude, and those numbered 43-84 low-amplitude (see Table 3.3), the light square corresponds precisely to the area where stimuli with low amplitude are compared against stimuli with high-amplitude (or vice versa), while the darker areas are precisely where stimuli of the same amplitude level (either low or high) are compared against each other. This distribution of standard deviation confirms the large role that amplitude played in how people characterized the stimuli, as evinced by examination of individual M D S results: almost everyone agreed that stimuli of different amplitude levels were indeed different, while there was much more disagreement about the similarity of stimuli of the same amplitude level, differing only by rhythm. Compared to the results of the pilot study, these results contain considerably more structure both in terms of M D S output and 61 distribution of standard deviation. This gives us much more confidence to delve deeper into the M D S results in order to investigate the effect of rhythm on the perception of haptic stimuli (below). Qual i ty of Judgments f rom Part ic ipants Since we are treating the. results of this study as a "gold standard" truth of how our stimulus set is perceived, we need to feel confident that the judgments given us by our participants are trustworthy. The use of only six participants has been shown to produce consistent results (i.e. low between-subjects standard deviation, and similar overall structure in M D S result) for haptic stimuli before [20]. However, the use of six participants with a more tiresome task is a concern. Fatigue was potentially a problem for this experiment, given the large size of stimulus set the participants were being asked to sort. Mitigating this concern, we observed that four out of the six participants elected to perform the experiment in one large 2-hour block, preferring to "get it over with," though by their own reports the task was not overly taxing. Especially with being able to listen to music of their own choosing, most participants reported being fairly comfortable with making perceptual judgments for an extended period of time. Furthermore, we find from analysis of the standard deviation that the level of disagreement between participants was fairly low. Aside from the trend of amplitude described in the above section, the level of SD is within a similar range for all values in the dissimilarity matrix. Whi le we do not have a good threshold for a reasonable absolute value of standard deviation in a task of this type, consistency of the standard deviation values for the averaged dissimilarity matrix helps confirm that participants' results did not suffer from random noise introduced by fatigue. A long with the strength of the M D S results we actually obtained, we feel that our participant pool has been shown to produce trustworthy perceptual judgments. 6.3 Analysis of Frequency and Rhythm The clearest result from our initial data analysis was that of amplitude, neatly bisecting the M D S plot, indicating a strong perceptual role for our stimulus set. The strength of 62 amplitude is in accord with previous research [19], but other results require further examination, such as the absence of the effect of frequency, and the perceptual role played by the different rhythms. In particular, we wi l l show that the frequency and rhythm results are intertwined, and thus discussed here in relation to one another. 6.3.1 Frequency Unlike amplitude, in plotting the centroids from the two frequency ("tonal") levels, we find almost no trend whatsoever (Figure 6.2); the two centroids are extremely close together in the middle of the M D S map. Since the frequency of stimuli did not seem to play a large perceptual role in dividing the overall map, it would be expected that stimuli with the same rhythm and amplitude but different frequencies should occur very closely together, which is largely, but not exactly, the case. Pairs of low and high frequency stimuli with the same rhythm are usually within at least the same quadrant of the map, i f not much closer. For example, in the upper-left quadrant, high-amplitude and high-frequency stimuli 15, 16, and 17 all sit close to their low-frequency equivalents 36, 37, and 38, and in the lower-right quadrant stimuli 50 and 52 are quite near their counterparts 71 and 74. However, i f frequency had absolutely no perceptual salience, then it would have been expected that the pairs of stimuli differing only by frequency would have been exactly co-located, which is almost never the case. The observed difference in placement could be due to noise in the data, but given the consistency of the rest of the results, it would seem odd that there might be pockets of such high noise that coincidently occur between stimuli of different frequencies. In Figure 6.4, there appear to be no distributions of standard deviation for between and within frequency level comparisons, as there are for amplitude (see 6.2.2 for discussion). Consequently it cannot be concluded that the frequency of stimuli had no effect whatsoever, merely that it did not have a consistent effect across all stimuli, and that in total magnitude its effect on distinguishing stimuli was less than that of amplitude, or indeed, certain types of rhythm. 63 Possible Explanat ion for L a c k of Effect This relatively small effect of frequency runs counter to the previous results of MacLean and Enriquez [19], who found that frequency dominated all other parameters, along with amplitude and waveform, among others. One potential explanation for the lack of effect is that the previous results were gained from continuous vibrations (in [19], of 2 second duration), while our results were gained from rhythms involving mostly quite short notes (e.g. a quarter note, the most frequently used, lasted for just 62.5 ms). Potentially, the short duration of the individual vibratory notes did not allow their overall frequency to be clearly perceived: a quarter note would allow just 12.5 repetitions at the lower frequency, 200 Hz (see Section 5.1.2 for frequency values used). Pasquero et al. [20] have shown that performing M D S analysis on a sub-matrix of the total dissimilarity matrix can effectively unfold dimensions that were hidden in the perceptual map for the total stimuli set; essentially, a subsection of the stimuli might have a local dimensionality which is suppressed by a more dominant one that applies to the overall set. Thanks to this fact, we were able to look only at the rhythms containing longer notes, in hope that the longer vibration times would allow people more time to perceive frequency (Figure 6.5). However, when mapped and expanded, the stimuli behaved similarly to the larger group: they fell in a general circumflex arrangement, with amplitude being the largest distinguishing factor and frequency playing a considerably smaller role. Though the axis of frequency was slightly larger than in the full M D S plot, given the small sample size of long note stimuli (16 /84 stimuli contained only half-notes or longer, with an average on-time of 1312.5 ms allowing 26.2.5 cycles of the lower frequency level) it is hard to claim any clear effects. With this explanation eliminated, only two other l ikely possibilities exist: either the two frequency levels were too similar to be consistently distinguished between, or rhythm dominated or masked frequency in a perceptual sense, essentially overriding any judgments that people might have made based on frequency. As outlined in Chapter 5, initial testing was done with different frequency and amplitude levels in order to determine values that would be perceptually quite different. These tests were performed 64 only on continuous vibrations, and not on rhythms, because we wished to avoid the confounding factor of which rhythm or rhythms should be tested. Furthermore, participants could feel each vibration for as long as they wished, furthering differing the sensation from the time-limit rhythms. Nevertheless, the frequency levels selected created sufficient perceptual separation in the continuous vibration test, and the rhythmic stimuli themselves were tested informally to ensure that rhythms at different frequency levels could be distinguished. So despite there being some a priori evidence of perceptual difference between stimuli played at different frequencies, it was seemingly not large enough to create a difference that people noted and used as a grouping criterion when combined with rhythm. Perhaps this lack of difference is because the two frequency levels chosen were dissimilar enough to create a difference, but only at the edge of creating a difference big enough to be perceptually important in classifying. This borderline condition might be an explanation for the inconsistent effect of frequency, but without a more concrete theory, we chose to largely ignore the effect of frequency throughout the remainder of this analysis. In future work, it wi l l be relevant to consider a larger frequency differential than that made possible with the present hardware. 6.3.2 Rhythm In the pilot study no obvious trends over rhythm were discernable. For every possible common-sense grouping applied to the data, there were enough exceptions that flew in the face of the trend that it was impossible to be certain that we were not simply imposing our view of the data onto what was essentially noise. In contrast, in the results from the full-set study the separation on amplitude was much more obvious and pronounced, and some clear clustering was evident, as opposed to almost no clustering in the pilot study results. This gave us greater confidence that our new hardware, as well as evaluation using trusted experts, had in fact tapped into the real perceptual characteristics of our stimuli—which, though displayed on different hardware, were made using the exact same parameters as in the pilot. 65 Analysis of 2D Solut ion As is often the case for analysis of M D S solutions, the 2D map was used as a landmarking tool; its easy-to-conceptualize nature makes distinguishing the broad features of a solution a much more feasible task. Examining the 2D plot of our data (Figure 6.2), there appeared to be two breaks across from each other in the circumflex, and if we drew a line between these two breaks we split the perceptual maps into two halves roughly orthogonal to the two halves created by amplitude. This seemed very encouraging, pointing towards the existence of a second major perceptual axis in the definition of the stimulus set's space, and since frequency had already been ruled out as a potential factor, the next reasonable place to look for a cause of this separation was rhythm. Inspecting the stimuli that fell along the left side of the map and the stimuli that fell along the right, it soon became clear that, barring a handful of exceptions, the stimuli along the left all involved rhythms that were comprised of entirely "short" notes, while the stimuli along the right had rhythms that contained " long" notes (see Figure 6.3). For our purposes, a "short" note was a either a quarter or an eighth of a bar long—thus stimuli from Groups 1 and 4 (see Section 3.3 for explanation) are rhythms containing only short notes. A " long" note is any note that is a half bar or longer; Groups 2, 3 and 5 all contain one or more long note. Plotting centroids for this split of " long" versus "short" note groups, we find an axis roughly as large in size, and orthogonal to, that of amplitude. The fact that such a strong grouping occurred according to pre-defined logical groupings of the rhythms was an encouraging result; these separations were a priori built into the stimulus set based on our intuitive understanding of how rhythms might be perceived. Finding these assumptions confirmed, at least in part, from our experimental results indicates that we are likely seeing evidence of peoples' true perceptual characteristics, rather than chance artifacts of the experimental and data analysis process. That we found evidence of the perception of parameters that we built into the stimulus set might seem to be a self-fulfi l l ing prophesy; however, lacking knowledge of the overall possible space of rhythms, some assumptions and intuitions had to be used. That we have found these 66 assumptions and intuitions confirmed experimentally indicates that we were justified in our original decisions, and nevertheless allowed us to notice some unexpected trends. Analys is of 3D Solut ion Attempting to find a different perspective on rhythm, the 3D M D S solution was examined. Arbitrarily assigning the names X , Y and Z to the three axes produced by the M D S algorithm, it was noted that the X - Y plane (Figure 6.6) was structured according to the same parameters as the 2D solution (though without the circumflex arrangement), with the Y axis differing along amplitude values, and the X axis differing along the " long note" to "short note" rhythms. However, examining the X - Z (Figure 6.7) and Y - Z (Figure 6.8) planes, it became evident that the Z axis was defined according to a different set of criteria. As discussed in Section 3.2, the rhythms were constructed to fall into 5 major groups. Since the grouping of " long notes" and "short notes" already fell along these grouping trends, the 3D data was examined to see where each of these 5 groups was situated. What was discovered was that along the Z axis Group 2, the group containing only " long" notes, appeared at the very extreme end of the axis, with all the other groups spread fairly evenly along the rest of the axis. This seemed to indicate that there might be something special with Group 2, distinguishing it from the rest of the rhythms containing long notes. This makes intuitive sense, as Group 2 contains only long notes, while Groups 3 and 5 contain both long and short notes. This difference manifests itself in an important perceptual characteristic of rhythm that wi l l be discussed further below, namely the feeling of "evenness" or regularity, versus "unevenness" or irregularity of a given rhythm. At a high-level, rhythms that contain only notes of the same length feel even, while rhythms that contain notes of different lengths (or rest notes of different lengths) feel uneven. Yet given only the 3D solution, all that was evident was the different place of the long-note, even rhythm group; there was no clear evidence of evenness similarly affecting the short-note rhythms. 67 Ampli tude-Independent Analysis The fact that in the 3D solution the Y axis accounts for most of the effect of amplitude, presumably allowed the Z axis to account for certain perceptual features of rhythm that could not be fit into the 2D solution. As mentioned above, by taking only one section of the total dissimilarity matrix and analyzing it using M D S , factors that were hidden in the larger solution can appear [20]. Thus if we remove completely the dominant factor of amplitude, the more subtle characteristics that define rhythm have a chance to manifest themselves. To this end, two sub-analyses were performed on the two halves of the stimulus set that had the same amplitude level. Unfolding the data in this way, it became clear that there were actually two perceptual axes involved in the perception of rhythm. As initially noticed in the 2D solution one of the dimensions was the length of the notes in the rhythm (presence of absence of the longest notes). The other dimension, as hinted at in the 3D solution, was the "evenness" of the rhythms. The 2D M D S output for all the high-amplitude stimuli is shown in Figure 6.9. As can be seen, the map can clearly be split into two halves of even and uneven rhythms; a large gap separates the two halves. Here we also see evidence of evenness of rhythms affecting both short and long note rhythms, which was hidden in the 3D solution. The split between the rhythms containing long notes and those containing only short notes is not quite as distinct, but still clearly observable. Initially this result may seem counterintuitive; the length of notes was the major discriminating factor in the 2D solution, so it would seem reasonable to assume that in the unfolded single-amplitude solution it would have the strongest effect. However, the likely cause of this is the fact that the "note length" axis actually contains a range of potential values, and can be somewhat ambiguously defined at certain points, while rhythms can be fairly unambiguously classified as either "even" or "uneven." Note Length Generally speaking, the definition of a "long note" rhythm is any rhythm containing at least a one half, three-quarters or whole note. Seemingly the longest note present in a rhythm defines how it is perceived along this perceptual axis. Rhythms with three-68 quarters or whole notes tend to fall near the ends of the axis, while rhythms containing half notes fall closer towards the middle, and those containing only short notes fall towards the opposite end, all as would be anticipated. Furthermore, if short notes are also present in a rhythm, the more short notes there are relative to long notes, the further towards the middle of the axis the rhythm wi l l fall. This is especially evident for rhythms 11 and 19, which consist of a half note followed by two quarter notes or four eighths respectively, and both of which fall roughly in the middle between the long note and short note groups. This placement only serves to reinforce that note length is indeed the trend that is being displayed here, as the rhythms contain long and short notes in equal measures; their placement directly in the middle of the axis is exactly where one would expect them to be. The trend of note length is not perfectly consistent throughout. Given the position of rhythms 1, 2 and 3 in the map, it may be claimed that the number of notes in a rhythm is also confounded somewhat with overall note length. Seemingly by having multiple short notes, these rhythms have moved towards the center of the axis. Under this explanation, it might actually be more accurate to describe the trend as one of overall time spent with notes playing versus not playing: if we add up the total duration of all notes played in the rhythms near the center of the axis, they come to a similar total, though the number of notes might be quite different for each rhythm. But this description is not strictly true either, as the three rhythms that are the equivalent of rhythms 1, 2, and 3, but with two eighth notes replacing each quarter note (and therefore with the same amount of total playing/not playing time), are placed much farther towards the "short" end of the note length axis. It is sufficient to say, then, that increasing the number of notes present can have an effect of moving rhythms more towards the " long" end of the note length axis, but that effect is not stronger than the overriding effect of the longest note present in the rhythm. Evenness of Rhythm As opposed to the note length axis, the even/uneven perceptual axis is very clearly delineated, with essentially no middle ground between the two groups. This can be felt 69 quite distinctly when the stimuli are actually displayed on the haptic device. Even rhythms have a regular repeating nature in which each part of the rhythm feels the same as every other part, throughout the duration of the stimulus. Uneven rhythms have an irregular, lurching feel to them; even with our monotone, same-amplitude stimuli, there is an emphasized portion of the rhythm and a deemphasized portion, such that the rhythm has an overall two-part structure, with a perceived emphasis on the first part of the rhythm. The most obvious examples of uneven rhythms are those in Groups 3 and 5, which consist of one long note followed by a number of shorter notes. Thus the longer note draws the emphasis, while the smaller notes are deemphasized, creating a skipping, one-two emphasis within the rhythm. B y only looking at the structure of the rhythms (as shown in Table 3.3), it is easy to conceptualize that Groups 2 and 5 might be perceived as uneven given the above description, yet it far less intuitive as to why rhythms 2, 3 and 15 also feel uneven, despite containing only notes of the same length. Yet upon feeling these rhythms, the sensation of unevenness is distinct. What appears to create the feeling of "unevenness" in these stimuli is actually the rest that occurs after the notes; the initial set of notes played thus creates the emphasized portion of the rhythm, with the blank occurring as the deemphasized portion. A caveat to this is that the rhythm must contain more than one note before the rest in order for it to be perceived as uneven. In the case of rhythms like 5, 7 and 8, which all contain only one note and then a rest for the remainder of the bar, the rhythm is seemingly recontextualized into one longer, slower pace rhythm containing a single bar consisting of a note played four times, instead of a bar repeated four times containing one note per bar. What appears to be causing this are the different sizes of the blank periods in the rhythm: in 2, 3 and 14, there are the blank periods that separate each note, as well as a longer rest note at the end of the bar. Thus what appears to define an "uneven" rhythm, in terms of how our subjects have placed them here, is that it either contains notes of two different lengths, or blank periods of two different lengths. The blank space between pairs of eighth notes, however, does not seem to count towards this effect. Consequently rhythms such as 16 and 17 are perceived as even. 70 As a last, confirmatory point, each of the four groups, from the four perceptual quadrants found in the amplitude-independent data (long-even, long-uneven, short-even, short-uneven, as seen in Figure 6.9), were individually run through the M D S algorithm (Figure 6.10). The resulting outputs were fairly similar in layout to the full 2D solution, with amplitude playing the largest defining role, but now with stimuli more spread out. No further insights were gained into how tactile rhythms are perceived. The most significant result of this sub-analysis was the stress and r2 values produced by these solutions (Table 5.1). As can be seen, stress values are lower and r2 values higher than the overall 2D solution. Whi le some of this improvement can be attributed to the smaller number of stimuli, it should be noted that the long-uneven group has a lower stress value that the short-even group, even though it has 8 more stimuli. Consequently, we can see this as further evidence that the stimuli in these groups naturally "fit" together, as it is fairly easy algorithmically for M D S to deal with them. R h y t h m Groupings in 2D Solut ion Returning to the 2D solution, we can see how these two axes manifest themselves when forced to contend with the overriding factor of amplitude (Figure 6.11). Plotting the mid-points of the 4 groups (short-even, short-uneven, long-even, long-uneven), several features can be noted. Firstly, the mid points all fall roughly in a line orthogonal to the line of the axis created by amplitude. Secondly, it is clear that of the two factors, note length has a stronger effect than evenness of rhythm, such that stimuli are grouped first by note length, and then within that group they vary according to evenness of rhythm. However, by introducing unevenness as a criterion, it explains the position of several stimuli whose placement was somewhat counter to the trend using strictly note length. For instance, without considering evenness stimuli 44 and 45 appear to wrongly be positioned with the long-note rhythms, despite consisting only of quarter notes. With evenness considered, it becomes evident that "44 and 45 are uneven rhythms, and are actually positioned in a group containing long-note rhythms as well as uneven rhythms. This result gives further weight to the claim that these two dimensions of rhythm are truly being perceived by people and that this is not a case of over-analysis of the data. 71 6 . 4 Summary In Chapter 3 we created a set of 84 different haptic stimuli by varying three design parameters: amplitude, frequency and rhythm. In this chapter, we set about studying how these stimuli were actually perceived when presented to users. Since the other main challenge this thesis deals with is how to present such a large number of stimuli to users, certain concessions had to be made in order to successfully examine these stimuli. Nevertheless, the results we achieved were extremely encouraging, especially in regards to the effect of rhythm within tactile stimuli. Our analysis showed clearly that different aspects of rhythm could be distinguished, and that coupled with amplitude could create a very wide range of perceptually different haptic stimuli. Our study consisted of asking six expert users to sort all 84 stimuli into groups, using the standard sorting method of data gathering for M D S . Since sorting 84 stimuli using a small handheld display is quite a tiresome task, only devoted and diligent participants were solicited. A n elite subject pool is not always possible to recruit, and nor are its results necessarily reproducible by the general public. Yet despite the challenges in gathering participants, their willingness to accept a more difficult commitment and their prior experience and knowledge in the field made certain that data quality was high, and gave it credence in establishing ground truth. After gathering the subjective perceptual data and running it through the M D S algorithm, we performed analysis, primarily on the 2D solution. We found amplitude to be the strongest perceived differentiating factor, while frequency was almost completely absent from the picture. The strength of amplitude agreed with previous findings [19], but the lack of frequency did not, an effect that can mostly be explained by the strong role of rhythm (which was conversely not present in the earlier analyses). From our analysis, it appears that the two primary characteristics on which our rhythms are distinguished between are the length of the longest note present in a rhythm, and the "evenness" of the rhythm ("even" rhythms only have notes and rests of the same length, "uneven" rhythms have notes or rests of different length). Controll ing for the effect of 72 amplitude, these two criteria are perceived roughly orthogonally, with note length being the slightly more dominant of the two. Though the rhythms that we tested in our stimulus set do not by any means cover the range of all possible rhythms, their simplicity should make these trends quite generalizable. Indeed, the consistency with which these two criteria were used to judge our rhythms is very encouraging, and should prove extremely useful to those wishing to use tactile rhythms in the future. Our results have provided interesting new insights into how tactile stimuli are perceived, and specifically how haptic stimuli can be designed in order to maximize both perceptual differentiability and grouping. Furthermore, using an established, validated technique with committed, expert users has provided us with a clear "gold standard" as to what constitutes ground truth for the human perception of this stimulus set. Thus we now have an empirically derived standard that can be used to compare the results from our as-yet unvalidated novel data gathering method, a goal we pursue in the next chapter. 73 Chapter 7: Subset Method Validation Study In Chapter 6, we examined our haptic stimulus set through the use of the sorting method often used to gather M D S data on haptic stimuli [19] [11] [18]. However, using this method with the full set of 84 stimuli created an extremely arduous task which was demanding even for skilled participants—problems outlined in depth in Chapter 6. In fact, if our total stimuli set had been any larger, we likely could not have performed the full-set study at al l, sorting 84 stimuli at once being about the absolute maximum that could be done by a single participant. Consequently, this makes the full-set study described in Chapter 6 a unique, one-off experiment. With the goal of creating a general, easy-to-use method for evaluating large numbers of haptic stimuli, a less arduous technique must be developed. As proposed in Chapter 4, by combining several existing methods that deal with large stimulus set sizes in M D S , we devised the subset method of data gathering for M D S that allows users to sort only a subset of the total stimulus set, thus greatly shortening experiment times, loosening restrictions on potential participants, and yet still producing a total picture of the perceptual space of a given stimulus set by averaging over a series of overlapping subsets. The cost of this method is in requiring a considerably larger number of participants for a given set size (to obtain sufficient overlap and reduce noise due to between participant variability), as well as the increased complexity of experiment design and analysis. This chapter is concerned with validating the accuracy of this new method, by testing the hypothesis that the subset method of data-gathering for a perceptual M D S analysis can produce results comparable to the normal, full-set sorting method, but with a considerably shorter and less taxing experimental task. Thus, in addition to providing new results about rhythm stimuli, the full-set study also played the role of "gold standard" in this scheme. 74 7.1 Validation Overview As discussed in Chapter 4, our main concern with collecting dissimilarity data from subsets of the total stimuli set is that the specific composition of each subset might affect how participants judge each stimulus. As an analogy, consider a quiet library in which everyone is whispering: if someone were to talk at a normal voice level, they might be considered " loud" compared to everyone else being "quiet." But if someone were then to start yelling, the person talking at a normal level might also be considered "quiet" compared to the " loud" yeller; conversely, the yeller might be considered "very loud" while the normal talker would still be " loud." Thus a subset without the yeller might produce different results across the board compared to a subset with the yeller. This is essentially a question of relative versus absolute judgments, and how great their effect might be on judgments within a subset. A t a slightly higher level, we are also concerned that participants might miss some of the larger patterns existent in the stimuli due to their lack of representation in the particular subset that participants are presented with. Especially if there is a small but highly distinct group of stimuli, there is a definite chance that some subsets might not have any of these stimuli, causing the user to completely miss their existence. Missing these stimuli would, in turn, create problems when averaging together the results from the different users, as different subsets would highlight different aspects of the stimuli, creating noisy averages .that cover over incongruent pieces. On the other hand, it is possible that this wi l l not be an issue, because the particular trends are only noticed when the stimuli that manifest these trends are present in a subset, thus making each subset fit together like a jigsaw, with different subsets providing coverage for the particular trends that are most evident in their stimuli. In fact, this situation could even serve to highlight subtle aspects of the stimuli that might be obscured in a full-set analysis. What we hope is that the relative difference values assigned to stimuli by participants stays at least roughly the same regardless of the composition of the subsets. The presence or absence of particularly " loud" stimuli would thus function in a way similar to a fish-eye lens on the M D S plot—distorting and compacting the positioning of all the stimuli around it, yet keeping their relative positioning. This kind of mild distortion can then be dealt with in a 7 5 convergent manner by averaging together observations collected from different randomly-generated subset comparisons. Our criteria for validation of the subset data-gathering method are thus as follows. Firstly, stimuli that occur together in different subsets multiple times (such that dissimilarity values for that pair of stimuli wi l l come from more that one subset) must still be given comparable dissimilarity ratings by users. "Comparable" in this case wi l l mean that the level of standard deviation between ratings from overlapped stimulus pairs is no larger than the overall level of noise in users' dissimilarity ratings. Secondly, the averaged dissimilarity matrix must produce M D S results that are reasonable and logically sound, standing alone. Thirdly, the results must be similar to the gold-standard study, both qualitatively and quantitatively. Criterion one checks for the effect of subset-relative judgments, while criteria two and three check whether the averaged result in fact reflects the real nature of the stimulus set. Criterion two is thus a lighter version of criterion three, assuming the accuracy of the gold-standard result. 7.1.1 Criterion 1: Consistency of Results Obtained from Different Subsets Inter-subset consistency is checked largely through examination of the uniformity of standard deviations of the averaged dissimilarity matrix elements. B y looking at the standard deviation of values in the matrix where multiple subsets contributed to the dissimilarity rating, and comparing them to those points that have been covered only by a single subset (provided multiple participants judged the subsets, so that standard deviation can be calculated), we can see whether the points of overlap have higher variability compared to the points of non-overlap. There is always some disagreement among different users when it comes to perceptual judgments (and, indeed, users can often disagree with themselves on different repetitions). However, if different subsets do indeed produce highly different results for the same stimulus pairs, then it would be assumed that there would be a much higher level of disagreement (and thus standard deviation) for those areas of overlap, compared to the normal level of noise (disagreement) between ratings given by participants. 76 In order to be able to perform this particular test on the data, a slight variation on the proposed experimental method had to be made. Our original design called for complete randomization of the stimuli in each subset, such that each participant would be presented with a unique subset. Randomization was to be performed in order to minimize any possible effect due to subsets, so that the damage of any particularly unfortunate grouping of stimuli in a subset would be covered over by the bulk of reasonable subsets. It was also done to ensure an even coverage of the dissimilarity matrix with as few participants as necessary. However, if we wished to compare the standard deviation of matrix points that average over multiple different subsets versus points that are averaged only over the same subset, then subsets must be repeated in order to develop a baseline level of noise/standard deviation that is to be expected when different participants are presented with the exact same set of stimuli. In this way we can compare the baseline level of standard deviation from individual difference to the level of standard deviation that occurs from individual differences plus differences due to participants experiencing different stimulus subset. To make this comparison possible, the minimal number of subsets needed to cover the entire dissimilarity matrix with one observation was created, which in our case took 5 subsets. These five subsets were used multiple times, such that each of the five subsets was sorted by several participants, allowing us to determine our baseline level of standard deviation while getting multiple observations per stimulus pair. This baseline level could then be compared against the standard deviation of areas where the five subsets overlapped, giving us a measure of how much comparisons from different subsets disagree with each other compared to the overall level of disagreement. 7.1.2 Criteria 2 & 3: Overall Accuracy of Results Criteria two and three both pertain to the resulting MDS output map: they seek to determine whether the output has real-world traction, and specifically whether it compares favorably to our gold standard. We perform much of the analysis required for these critieria in an ad-hoc method similar to the analysis performed in Chapter 6 on the output of the full-set study. Yet calling our analysis ad-hoc is not meant as a slight to its 77 efficacy. Compute r i zed cluster ing algori thms have yet to consistently attain the results o f detailed human analysis; their lack o f semantic reasoning and "common-sense" appreciation o f the dataset is usual ly their downfa l l . Nevertheless, since this type o f analysis can be considered as quite quali tat ive, a statistical means o f testing s imi la r i ty has been considered as w e l l . A s the accepted statistic for the compar ison o f M D S results, the coefficient o f al ienation, K, is also used to determine statistical s imi lar i ty , us ing the empi r i ca l ly -de r ived values presented by B o r g and Leutner [4] to determine s imi la r i ty at the p = .05 level (as discussed in Sect ion 2.2.1). Howeve r , statistical s ignif icance is not a lways practical s ignif icance, so it is important that our quali tat ive analys is—our assessment o f reasonableness o f result—agrees wi th the statistical measures on the s imi la r i ty o f the results o f the two studies; both measures o f analysis are required i f we are to consider cr i ter ia three to have been met. 7.1.3 Strengths and Weaknesses of Validation Process O v e r a l l , the m a i n weakness o f this val idat ion process is the degree o f bootstrapping i n v o l v e d i n the creation o f the s t imul i and the design o f our studies: we are testing a new study methodology on a new st imulus set. Furthermore, the g o l d standard that we are compar ing the results o f our new study against comes f rom stretching an already established technique potential ly to its breaking point. H o w e v e r , these two studies were carefully designed to m i n i m i z e any c i rcular i ty i n their reasoning and to m i n i m i z e the amount o f bootstrapping. The full-set study uses an already v a l i d technique to gather data, and its ma in weakness is the potential fatigue o f its participants, w h i c h we watched c lose ly for. The subset study uses an unvalidated technique, but its potential weaknesses are i n the logic o f the study itself, and it is , i n fact, designed to m i n i m i z e the issue o f fatigue that troubles the full-set study. So each study's weaknesses are designed to counteract the weaknesses o f the other, wi th the full-set study p rov id ing the so l id ground-truth, produced at a heavy cost, w h i l e the subset study can be compared to this ground-truth wi th data m u c h more easi ly gleaned f rom the user. 78 7.2 50-Stimulus Subset MDS Study Our study using the subset method of data gathering for M D S takes place in two parts. In the first part we ran a study on 15 participants using the subset method, specifically designing our experiment to allow us to analyze several different characteristics of our new methodology in order to test for validity. Whi le this experiment did produce several very important insights into the strengths and weaknesses of our new method, it also failed to produce an M D S plot that was sufficiently similar to our gold standard due to the experimental design decisions we made (specifically, it was derived from non-uniform number of observations across the full-set dissimilarity matrix). Thus in the second part of our study, we ran an additional 7 participants with the specific aim of gaining a better coverage of perceptual data across the entire stimulus set. B y adding these supplementary data points, we were able to increase the quality of the resultant M D S plot such that it was both qualitatively and quantitatively similar to our gold standard. At the same time, we gleaned an important methodological insight, i.e. the importance of uniform coverage. 7.2.1 Method (Study Part One) Fifteen participants, 5 female, 10 male, ages ranging from 22 to 35 were recruited to run this experiment. A l l were graduate students at U B C . The experiment lasted approximately one hour, and participants were compensated $10 for their time. As in the full-set, gold standard study, participants sorted a set of haptic stimuli on the Nokia 770T using the program described in Chapter 5. Each participant sorted a particular stimulus set three times during the experiment session. In the first sort, participants were told to group stimuli into whatever number of discrete, non-overlapping groups they felt was appropriate to describe the perceived dissimilarity between stimuli. For the remaining two sorting tasks, participants were required to sort the stimuli into a specified number of groups, either 3, 9 or 15. Of these three group numbers, the one closest to the number of groups chosen in the first sorting task was not used, with the remaining two numbers randomly assigned to the second and third sorting tasks. Participants wore Bose Quiet Comfort 2 acoustic noise cancelling headphones during the 79 experiment, which played white noise loudly enough to mask the sound made by the haptic feedback on the device. White noise was substituted for music, which was used for the gold-standard study, because (a) the experiment time was shorter (and so the noise less tiresome) and (b) we had recruited 'normal' rather than especially trustworthy participants, and were not comfortable allowing them to self-monitor their own music choice. The use of white noise is a much simpler and more realistic experimental setup. Unl ike the full-set study, participants were not presented with the full 84 haptic stimulus set, but instead with a subset of 50 haptic stimuli. Subsets of size 50 were chosen as the target size as, in initial testing, it was found to be the largest number of stimuli that could be consistently sorted in approximately an hour, using the technique described above. Using the subset algorithm described in Chapter 4, and employing the modifications to our method described in Section 7.1.1, we produced 5 randomly distributed subsets (see Appendix C for specific subsets used) that we could use multiple times in order to test i f judgments differed from subset to subset. Our algorithm ensured that every two stimuli appeared together at least once in one of the subsets, and guaranteed that a dissimilarity value would be present for each combination of stimuli. Whi le our algorithm attempts to minimize the amount of overlapping coverage, certain points^in the dissimilarity matrix are overlapped by as many as four different subsets, though most points are covered by only one or two subsets. This overlap is unfortunate but some amount is unavoidable due to the nature of the sorting task. Each one of the 15 participants performed their sorting task on one (and only one) of the five subsets, meaning that each subset was sorted by three participants, with 3 participants x 5 subsets giving the total 15 participants, as shown in Table 7.1. 7.2.2 Results Dissimilarity values for each participant were calculated in the same manner as described in Section 6.2.2. Fu l l 84x84 symmetric dissimilarity matrices were then created for each participant, containing the dissimilarity values for those stimuli present in the subset they were tested with, and a value of -1 for all stimuli not presented, to mark them as missing. These dissimilarity matrices were then averaged over all users, with only the non-missing 80 values used to create the average value for each point within the matrix (many of the final values were thus averaged over different numbers of individuals). This averaged dissimilarity matrix was then run through the SPSS A L S C A L algorithm for dimensions 1 to 4. The resultant stress values were plotted (see Figure 7.1), and a marked elbow was looked for, but no obvious candidate was forthcoming. The most l ikely candidate for an elbow was the 2D solution, though the overall curve of the graph was fairly even. The 2D solution had S-Stress = 0.38949 and r2 = 0 .23463, while the 3D had S-Stress = 0.28216 and r2 = 0.35597. These stress values are reasonable, though the r values are very low, indicating that a large amount of variability in the data was not accounted for in the solutions. However, this trend occurs across all dimensions, so we were forced to use the data as it was. Therefore, as in our full-set study, for parsimony as well as ease of interpretation, the 2D solution was selected for analysis, and the 3D solution was consulted for clarifying purposes. Upon an initial analysis of the 2D solution, several features were evident (see Figure 7.2). Firstly, the strength of the amplitude axis is still quite evident, which is encouraging if we are concerned about the results being realistic: the subset technique has at least captured this, the strongest trend in the data according to our gold standard. Additionally, essentially no effect of frequency was found, just as in the gold standard. However, applying the rhythm trends as established in our gold standard, we see that the placement of the four groups has shifted. In the gold standard, the- 2D solution was split, orthogonally to the trend of amplitude, first according to the note length of the rhythms, short to long, and then within those two halves, from even rhythms to uneven rhythms. In the 2D solution for the subset study, however, the solution is split first according to evenness of rhythm, and then by note length. The major manifestation of this is that the group of stimuli that has long notes and an even rhythm has shifted over to the far extreme left of the map, pushing the other three groups towards the right. Comparing the subset study M D S output statistically with the gold standard, we find a result contradictory to our visual inspection: the coefficient of alienation, K, is 0.4485, 81 which, at NP = 84, ND = 2, is less than K critical = 0.55, and is significant for p = .05, according to the work of Borg and Leutner [3] and described in detail in Section 2.2.1. This result means that the similarity between the M D S maps of the subset and full-set studies is statistically similar, at a 95% confidence interval. Here we are presented with a case where statistical significance does not seem to agree with our practical analysis. The reason for these troublesome results is discussed in the following section 7.2.3 Reasons for Difference in MDS Results The discrepancy between our statistical and practical analysis was a major concern to us. The placement of the "long-even" group in a different position as opposed to the gold standard study was a potentially fatal result for our new study methodology. This result is the primary reason why we determined that we needed to run additional participants, as described in Section 7.2.4—a choice that would result in a successful validation of our new technique. First, however, we wi l l describe how a limited number of observations caused this group of stimuli to be placed differently, the key insight leading us to run additional participants. Analysis of Standard Deviation As visual inspection and statistical comparison differed in their conclusions, greater importance was placed upon our third means of analysis, comparison of the standard deviation values of the averaged dissimilarity matrix. The average standard deviation of all values in the dissimilarity matrix is 346.27, which is considerably higher than the gold standard's average SD of 160.02, so right away we were presented with a potential explanation for the difference in the two M D S outputs (gold standard and subset study) as being some source of additional noise in the subset data. Next we observed the distribution of the SD values over the dissimilarity matrix for the subset study, and noted a marked difference in the distribution of high and low S D values as compared to the gold standard (see Figure 7.3). The gold standard generally contained high S D values for points in the matrix where two stimuli of the same amplitude level were being compared, and low S D values for points where stimuli of different amplitude 82 levels were compared. B y contrast, the subset study has distinct "stripes" of high SD values running through the dissimilarity matrix. These stripes occur along groups of four or five stimuli, and extend through comparisons with other stimuli of both different and the same amplitude level. No overall trend of between and within amplitude comparisons can be seen. As noted in Figure 7.4, many of these stripes of high SD occur along stimuli from the "long-even" group —the same group whose placement in the M D S map is the primary difference between the full-set and subset study's results. Though there are two other stripes that correspond to stimuli in parts of other groups (the high-amplitude and frequency members of Group 4 from Table 3.2, and the high-amplitude, low-frequency members of Group 5), the long-even group has by far the largest number of stimuli that are part of these high SD stripes. Certainly it can be noted that not all of the stimuli in the long-even group correspond to areas of high standard deviation; indeed several of the stripes are off by one or two stimuli from the actual stimulus groups. However, as we wi l l argue later, this is because such stripes of high S D are the result of a combination of certain hard-to-judge groups with areas that received low numbers of observations, and so this lack of exact correspondence is to be expected. Nevertheless, a high degree of variability would explain why the long-even group appears in a different position in the subset study's results compared to the full-set study. What requires an explanation is the,source of this high degree of disagreement among participants. Comparison of individual M D S plots is not overly fruitful, as most participants saw different sets of stimuli than the others and thus have, by definition, different M D S plots (see Appendix B for individual plots). Consequently we continue to rely on standard deviation as our main method of analysis. Hypothetical Explanations for Divergent Long-Even Group Results There are several possibilities for the observed strips of high standard deviation associated with the long-even group in the subset M D S result. It could be that by a fluke, these stimuli only ever occurred in one of the subsets, thus biasing their results 83 (Hypothesis 1); it could be that this is the result of different subsets producing different results for the stimuli in this group (Hypothesis 2); it could be that there was simply not enough data gathered for the stimuli in this group to gain a statistically reasonable average (Hypothesis 3); or it could be that the long-even group is inherently harder to judge than the other groups (Hypothesis 4). Hypothesis 4 is hard to prove definitively; instead it becomes the default conclusion by elimination if the other explanations are actively disproved. In the following, we wi l l address Hypotheses 1-3. Hypothesis 1: Fluke Distribution of Stimuli As the subsets were randomly created, it would be expected that the stimuli in the long-even group would appear fairly evenly throughout all five different subsets used, and this is indeed the case. A l l five subsets contained between 7 to 12 out of the 16 total stimuli in the long-even group (see Appendix C for subsets used). Thus all of the subsets would have contributed values to describing this group, so Hypotheses 1, the 'f luke' uneven distribution of stimuli, is eliminated. Hypothesis 2: Subset-Relative Judgments We know that long-even note rhythm stimuli occurred at roughly the same frequency in all 5 subsets; however, we do not know if one or more of these subsets had a distribution of stimuli that would cause the judgments in the set to be skewed and/or noisy. If such outlier subsets existed in our study, it could be that they contributed to the long-even group being placed differently in the M D S plot (hypothesis 2). If this were the case, we would notice this most distinctly for dissimilarity values that were averaged using data from different (conflicting) subsets. A simple way of determining whether the averaged dissimilarity values for the long-even group came mostly from overlapping subsets or just from single subsets was to look at the number of values used in the average of each dissimilarity value in the group. If we plot the number of observations for each dissimilarity value in a similar way to the 84 standard deviation values (see Figure 7.5), we can see which values were produced only from a single subset (evaluated by three participants, and shown in dark purple), and which values were produced with data from more than one subset (run by some multiple of three participants (shown as light purple or white cells, i.e. lightest means highest number of both observations and distinct subsets used). B y lining up the columns and rows that contained the long-even group (orange) with the plot of the number of observations, we can see that for the most part, the dissimilarity values for the long-even group have been aggregated from single subsets (dark purple; though the particular subset that has contributed to each value does differ). Given the high standard deviation that the long-even group is correlated with (as illustrated in Figure 7.4), a possible explanation is that the noise was due to judgments for different subsets being distinctly different as a consequence of between-subset variations—yet this appears not to be the case. These levels of high S D seem to be occurring despite the values being averaged from only one subset, so we can claim that Hypothesis 2, noise from conflicting subsets, does not appear to be a convincing explanation for the source of the higher overall noise exhibited in the subset analysis compared to the single set analysis. It should be noted that there is another way that subset-relative judgments could have affected the placement of the long-even group, but it is not an effect that would have produced the distinctive , long-even group associated stripes of high S D that are evident. If there were any subsets that produced judgments for the long-even group which were distinctly different from that of the other subsets (i.e. these stimuli substantially rearranged on the M D S output, as opposed to their relative positions 'stretched' a little), then stitching together the results from these subsets could create a contradictory picture of the entire stimulus set, as evinced by high noise associated primarily with members found in the idiosyncratic subset as opposed to just the problematic stimuli group. However, the stripes of high SD occur across all five different subsets (since the long-even group that corresponds with them appears in all five subsets)—so either all the subsets produced idiosyncratic results for the long-even group, or none of them did. If 85 each subset skewed its judgments consistently, we would not notice a trend of higher S D for the long-even group, a consistent skew should produce similar results for each of the subsets which in turn produce a low SD when aggregated. In fact the only way in which we would be able to determine such a skew would be in comparison to the gold standard. Alternatively, i f the subsets caused judgments to skew unpredictably for each participant (between-participant variations rather than between-subsets, but in a subset-specific way) then we would expect to see levels of high SD across all stimuli ,and not just in the long-even group (Hypothesis 4 already accounts for there being something particular about the long-even group that tends to create noise). Hence it is not the case that our observed noise resulted from the fact that in some instances, dissimilarity values in the long-even group were averaged from multiple-individual evaluations of a single subset, as we can see no evidence that any of the subsets produced 'bad', outlier results. In Section 7.3.2 we further discuss how we found no evidence of subset-relative skewing of judgments overall, but for our current argument pertaining to the long-even group it suffices to say that such subset-relative judgments do not appear to have caused the high SD exhibited by this group. Thus we are left with either Hypothesis 4 (long-even group is inherently hard to judge) or Hypothesis 3 (this group did not receive sufficient observations). Hypothesis 3: Insufficient Observations While one can view the plotted observations (Figure 7.5) as a means of determining how many different subsets contributed to an average value, we can also simply consider the number of observations as a raw value in itself, disregarding how many subsets these observations came from. Performing this mental switch, we notice that the long-even group largely corresponds to areas with the minimum number of observations (three, represented by dark purple). A low n value in the calculation of standard deviation allows outliers to more strongly affect the value, and so the high standard deviation that the long-even group corresponds to could well have been caused by have an n of 3 for many of its values. It is thus possible that outliers from such a small sample of data (as 86 evidenced by the high SD) caused the long-even group to be placed differently in the subset study's M D S output, as compared to the gold standard. Given that the subsets were randomized, we can only conclude that it was simply an unfortunate distribution that caused the long-even group to have so few observations for many of its dissimilarity values. Thus far, our data analysis allows us to make this claim specifically about the long-even group, but we have not yet presented a general analysis of how the number of observations (as well as the number of subsets) can affect the quality of the M D S results. In Section 7.4 we discuss this topic in much greater detail. Nevertheless this analysis is key to understanding why we decided to run additional participants: i.e. we needed to distinguish between Hypothesis 3 (insufficient observations) and Hypothesis 4 (inherent difficulties in judging) to explain the placement of the long-even group. Thus, we chose to obtain additional data to rigorously test Hypothesis 3. To bring the current analysis to closure, we wi l l thus make a forward reference to the conclusions derived from this augmentation of our study in Section 7.3, where we do indeed find that increasing the number of observations (apparently independently of the number of subsets) to a level that is relatively uniform across all stimulus pairs, diminishes the stripes of high SD. That is, low observation numbers seem to correlate to high S D , and this compounded with the fact that the long-even group, by chance, appeared to particularly suffer from receiving a low number of observations. Further, the augmented subset study produced an M D S result in which the long-even group is placed consistently with the gold-standard result. We therefore wi l l conclude that Hypothesis 3 is upheld. Hypothesis 4: Long-Even Group is Inherently Hard to Judge Our results to this point do not allow us either to firmly accept nor refute Hypothesis 4, that the long-even group was inherently harder to judge. Compounded with the lack of observations, this inherent difficulty could well have been an additional source noise. We 87 therefore must conclude that it may also have contributed to the stripes of high standard deviation seen associated with this group in Figure 7.5. Summary of Long-Even Results Overall, these findings are very encouraging: we were initially worried that subsets might produce poor results, but have found instead that they can produce strong results quite similar to the gold standard. A long the way, we discovered the importance of maintaining uniform coverage, of at least 5 observations per point, across the whole dissimilarity matrix. Below we wi l l further disambiguate the role of the subsets themselves in the result (7.3.2). 7.2.4 Study Part Two: Additional Participants with New Subsets The fact that many values in the averaged dissimilarity matrix for the initial subset study only came from a single subset, tested three times on three different participants, was due to a particular choice in the study design of using only 5 subsets for 15 participants. We made this choice so we could gather a baseline level of variance due to individual differences, and compare it against the level of variance between different subsets. However, this choice had the negative side effect that there was a fairly large range in the number of observations that a given value in the aggregate dissimilarity matrix could be averaged over. Values where subsets overlapped were replicated three times, meaning that while some values had as few as three observations, others had as many as twelve. Since it appeared that these values with a low number of observations might be causing the M D S output to differ from the gold standard (Hypothesis 3), it was decided that we should run more participants, using new subsets designed to "f i l l in the gaps" left by the first subsets, thus evening out the number of observations across the entire dissimilarity matrix. Method Seven additional participants were run through the same procedure as described in 7.2.1, but this time each with a unique subset of 50 stimuli designed to even the coverage provided by the first subset study. Participants were all graduate students at U B C , ages 88 ranging from 22 to 29. The seven additional subsets ensured that each point in the aggregate dissimilarity matrix had a minimum of 5 observations (filled from a minimum of 3 different subsets), while the majority of points had between 6-10 observations and some points had as many as 17. Results Adding the dissimilarity matrices produced from the 7 additional participants to those produced from the original 15 participants, we created a new aggregate dissimilarity matrix, which we then ran through the SPSS A L S C A L algorithm as before. Figure 7.6 shows the stress plot for the new M D S solutions, from dimensions one to six. Though the stress curve is very similar to Figure 7.1, again with no marked elbow, there is a significant increase in r2 values for both the 2D and 3D solutions at 0.32848 and 0.47007 respectively, indicating a greater amount of variability in the data has been accounted for in the solution. Graphing the M D S output with the additional participants' data, and applying again the grouping of stimuli from the gold standard, we find a much more encouraging result (see Figure 7.7). The trend of amplitude is just as strong as before; but now the trend of rhythms, from long note to short note with uneven to even nested within, is present in the exact same order as the gold standard (Figure 6.10), though mirrored left-right. As relative, not absolute, position is what is important in M D S plots, mirrored results are equivalent. Mapping an axis along the centroids of all four of these groups creates a line almost perfectly perpendicular to the axis of amplitude, precisely as it does in the gold standard. Additionally, idiosyncratic placements of stimuli such as 44 and 45 (from the short-uneven group) amongst a generally long-uneven cluster are replicated quite similarly to the gold-standard solution. Further qualitative similarities and analysis are described in Section 7.3.1. Two quantitative values also point towards an increase in similarity to the gold standard. The average standard deviation is down from 346.27 to 245.78, which is still higher than the gold standard's value of 160.02, but greatly decreased from the initial subset study, 89 indicating that these results are more internally consistent. Furthermore, the new, lower K value of 0.3534, a roughly 21% decrease from the previous value of 0.4485, is consistent with the theory that the additional participants run have increased the similarity between the subset study's results and the gold standard—though as discussed before and further elaborated in 7.3.1, K cannot be taken as a complete guarantor of similarity. Running additional participants was done in order to address our analysis in Section 7.2.3. The results of this addition back up Hypothesis 3, i.e. that insufficient observations can explain altered placement of the long-even group in Part 1 of this study. 7.3 Validation of Subset Technique We set out to prove the validity of the subset method of data gathering for M D S by ensuring that it met three criteria: that different subsets did not produce significantly different results for the same stimuli; that the resultant M D S plot was reasonable and believable in terms of interpretability; and that the M D S output compared favorably to that of the gold standard. Below, Section 7.3.1 describes how the output of the M D S algorithm is structured and how it compares to the gold standard, thus validating our method in terms of the second and third criteria, given sufficient data as collected in Part Two of this study. After this, Section 7.3.2 describes how the standard deviation of the dissimilarity values shows where discrepancies between individuals arose, disproving the theory that these discrepancies arose from the use of subsets, thus satisfying the first criterion. The analysis of standard deviation (7.3.2) is made easier by first considering the shape of the M D S output, which is why it is discussed second. 7.3.1 Criteria 2 and 3: Reasonableness of Results & Comparison to Gold Standard The initial M D S results of our subset study could be said to have met the second criterion of reasonable and believable results, but failed on the third criterion of similarity to the gold standard. The strength of amplitude and the lack of effect of frequency were as expected, and grouping according to certain aspects of rhythm on an axis perpendicular to that of amplitude were also evident. Without a gold standard referent, we could have 90 concluded that these results in fact represented ground truth as to the perceptual characteristics of the rhythmic haptic stimulus data set. Even with the gold standard, according to our statistic of similarity, K, the two results were similar. However, according to our visual, ad hoc analysis of the M D S outputs, the two trends of rhythm were without a doubt different: the four main groups differed in order of appearance along the rhythm axis, which would lead do different conclusions about which aspects of rhythm were more perceptually important in differentiating between stimuli. Happily, running additional participants closed the gap between the subset study and gold standard, greatly increasing their similarity even to detailed visual inspection. Though rotated roughly 45 degrees clockwise, and mirrored along the rhythm axis (both simply products of random variations within the M D S algorithm itself, and therefore inconsequential), the two M D S maps maintain the same order of grouping along the rhythm axis, and have an even stronger and cleaner separation along the amplitude axis. K was similarly more positive, approaching more closely its ideal value of 0. Thus at a broad level, the subset results did seem to resemble those of the gold standard to a reasonable and practically useful degree (in the absence of other objective measures). Sub-Group Analys is for Higher Resolut ion However, these trends were fairly high level, and so a more detailed analysis was performed in order to determine how well the subset method captured the more nuanced characteristics of the stimulus set. In the analysis of the gold standard, a sub-analysis of all the stimuli With a high .amplitude level was performed in order to examine more closely what effects rhythm had on the stimuli's perception, regardless of amplitude; we performed a similar sub-analysis on the data produced by the subset method, to see if it yielded the same insights. In Figure 7.8, the high-amplitude sub-group is analyzed in isolation and graphed. Apply ing the same groupings as in the gold standard and plotting their axes, we see that they are almost exactly the same length and in similar directions. Indeed the general layout of the two graphs, Figures 7.8 and 6.9, is strikingly similar. The evenness of a 91 given rhythm makes up one axis, going from even in the top right to uneven in the bottom left, while note length makes up the second axis, going from long notes in the top left to short notes in the bottom right. However, there are some differences between the two M D S maps, most notably that the short-even and short-uneven groups are not totally separated, as they are in the gold standard. The two stimuli with a four quarter-note rhythm, and the two stimuli with an eight eighth-note rhythm are situated amongst the short-uneven group, which is counter to their placement in the gold standard. This placement may indicate that, at least for short-note rhythms, the number of notes in a rhythm is sometimes considered as the same thing as the perceived evenness of the rhythm. However, at this level of detail, we are entering into a realm of very precise pronouncements about how very small numbers of stimuli are perceived, from a study that involved the judgment of a very large number of stimuli. Insights at this level are probably better served by studies run on small sections of the data, looking for particular characteristics of individual stimuli. Thus at a secondary level of detail (with amplitude removed) the results from the subset study can be said to be similar to that of the gold standard, though at an even further level of detail, discrepancies begin to appear. This level of correspondence is l ikely greater than we can expect between any two experiments run on the same stimuli, so for our purposes, the M D S results of the subset study and the "gold standard can be said to be both qualitatively and quantitatively similar. Questioning the Statistical Analysis One last question that might be asked is why the K statistic failed to account for the differences in M D S outputs that were observed through our own analysis. A potential explanation for this is due to dependence of K on the distances between each data point on the map, as K is calculated by comparing the distances between each data point in the first map against the same distances in the second map. Since both 2D results were arranged in a circumflex, most points in the M D S map have large distances between them, across the circumflex. Consequently, the similarity of these large distances may have had a large enough effect on the K statistic that the change in position from one side 92 of the circumflex to the other, of a small number of stimuli, created too small a difference to greatly change the overall K value. Since K does not encode the relative importance of any particular stimuli, it could not reflect the significance that the change in position of those particular stimuli had. Indeed, a similar change in position (distance-wise) of a different but similar number of stimuli could likely have produced more-or-less the same ordering of rhythm groups, which we would have then used as an argument for the similarity of the two M D S results. This result only serves to confirm to us the importance of cross-checking conclusions using several means and of looking for practical significance as well as statistical significance; which our results, in the end, have indeed demonstrated. 7.3.2 Consistency of Results: Do Subsets Introduce Too Much Noise? The standard deviation of each averaged dissimilarity value can be used as an indicator of the degree to which different participants disagreed on how dissimilar a pair of stimuli appear: the higher the standard deviation, the higher the disagreement between participants. If all the participants were tested with the same subset, then the level of disagreement can be attributed solely to individual differences, and/or variability in repeated observations by the same individual, in their perception of the given stimuli. If the participants were judging the same stimuli, but in different subsets, then an additional potential source . of disagreement is the relativizing effect of different subsets on perceptual judgments. Thus if we are concerned about whether splitting the stimuli up into subsets wi l l cause too much variability in judgments (Criterion 1 in Section 7.1.1), analysis of standard deviation is where we need to concern ourselves! Noise Due to Subset-Relative Judgments: Between-Subsets Analys is The need to check for this effect drove our initial 3 participant x 5 subset study design: we required replicated data for a small number of unique subsets, as opposed to a larger number of non-replicated unique subsets, with their more complex overlapping pattern. The data from the additional participants is discussed in the next section, but this particular analysis requires the 3x5 structure of the first part of the study. As we had three participants sort each subset in the first round of our study, we can create an average for 93 the dissimilarity values for each subset, and examine the standard deviation of those averaged values to determine the baseline level of noise that comes solely from individual differences. The baseline SD gives us a reference point to compare against the SD of dissimilarity values averaged with data from multiple, overlapping subsets—values indicating the level of noise from individual differences plus the effect of different subsets. If the SD for these overlap values, is substantially greater than the baseline, then we would have strong evidence for there being an effect of subset on the judgments given by participants. • . . This comparison is performed by plotting the SD of all values in the dissimilarity matrix, aside by side with the number of subsets involved in the average of each dissimilarity value, as is shown in Figure 7.9. By comparing back and forth between the two halves of the matrix, various trends can be discerned. Through choice of colouring we highlight that the darker areas of high SD (and specifically the distinct "stripes") generally occur where there are darker areas indicating a single subset - i.e. non-overlap areas; and that the lighter areas of low SD generally occur where there are lighter areas of high numbers of subsets. This result is in contrast to our original concern that increasing the number of subsets in play would increase SD for observations from overlapping subsets, although it is countered by the fact that these points also have more observations overall. Nevertheless, it does appear that dissimilarity values from overlapping subsets converge towards an appropriate value for this stimulus set. If some particular subsets affect peoples' judgments by consistently skewing them a certain way (for all individuals), then we would expect to see the result of overlapped dissimilarity values having higher SD, a result that we did not see. However, if some subsets by chance contain combinations which generate confusion or disagreement and simply make everyones' judgments noisier, then we would expect to see that some subsets exhibit overall levels of noise higher than others. This too, is not evidenced by our data, most strongly by the instances of stripes of high SD. These stripes occur across values from all five of the subsets used originally, and as noted above, generally only have single subsets contributing to each of its dissimilarity values. The consistency of 94 these stripes of SD across all five subsets seems to indicate that no one subset was noisier than the other. The fact that the levels of high SD found in the first round of the study cannot seemingly be attributed to the negative effects of subsets is strong support for the subset method meeting the first criterion of validity (no effect of subset-relative judgments). Evidence from Additional Participants Further evidence for the lack of subset-relative effects is the results from running additional participants, each with their own, unique subset. If each unique subset did tend to produce judgments that were unique for the stimuli contained in the subset, then adding in seven new subsets to a data set already built up of five different subsets should increase the overall noise level in the aggregate matrix. Yet the net effect was to reduce standard deviation and increase the M D S result's similarity to the gold standard. This reduction confirmed our initial hypothesis (Section 4.2) that the best way to counteract any effect of subset, large or small, is to completely randomize the selection of each subset used and to use overall a 'reasonably' large number of subsets relative to the size of the complete stimulus set. In this way, i f any one subset did have a strong relativizing effect on judgments, its effect would be minimized due to its data being mixed in with many other subsets, which should, on average, contain a reasonable cross-section of stimuli. Another way of saying this is that we wished to have as many different subsets as possible, to minimize the effect of each one; randomized subset construction maximizes this effect. B y giving multiple participants the same subset (Part One of this study), we were able to observe this trend, but this technique is not recommended,for regular use: instead complete subset randomization, as originally specified, wi l l minimize overall noise levels in the data, as well as the number of participants needed to obtain a desired number of observations for each data point. In summary, we cannot claim that there wi l l be no effects of subsets; and to some extent, the low impact of subsets observed here could be a function of characteristics of the particular overall stimulus set which we have explored in the present research. Other sets, 95 e.g. those containing small groups of highly salient stimuli, could potentially be more vulnerable to such problems. However, it appears that (a) potential subset effects can be mitigated by using more and randomly created subsets, as opposed to fewer; and (b) the effect of subsets handled in this way are likely to be small, or even negligible (as observed here) in comparison to that of individual differences. Since the problem of individual differences is one that is never going to be removed completely from an experiment, we can assume that the effect of subsets, i f any, wi l l manifest itself very infrequently i f handled properly. Individual differences are a problem that any M D S data gathering technique suffers from, so we feel we can conclude that our new subset technique suffers from no problems worse than those confronted by any other method known to this author. 7.4 Reflections on the Design of the Subset Data Gathering Method With a strong case made for the validity of the subset method of data gathering, we turn next to a reflection on the overall nature of the technique, its strengths, weaknesses and peculiarities. Especially in our analysis of the standard deviation of the dissimilarity matrix, we found many interesting features indicating where our experimental technique succeeded, and where it struggled. As we wish this technique to be taken up by other researchers in the field, we outline here several important features or the subset method that any experimenter who wishes to use it should be aware of. 7.4.1 Observations vs. Subsets In validating our method, we showed that subset-relative judgments did not appear to have a strong negative effect on our results (7.3.2). If judgments did differ from subset to subset, this difference was not evident to us. Furthermore, in our analysis of the long-even group (in Section 7.2.3), we concluded that its improper placement in the M D S plot from the first part of our study was due to it receiving an insufficient number of observations in conjunction, perhaps, with it being an innately difficult group to judge. These two results together seem to indicate that to ensure the quality of M D S results using the subset method, one should concern oneself most with gathering enough 96 observations, and not be too concerned about the effect of subsets; and potentially to even employ more subsets to reduce the skewing influence of 'outlier' stimulus groups. We pursue this argument to its conclusion here. Importance of Having Sufficient Observations A low n value in the calculation of standard deviation allows outliers to more strongly affect the value. Thus analyzing Figure 7.9, we cannot be too surprised to find that having high SD in dissimilarity values seem to consistently occur where there are a low number of observations, though the converse is not always the case. It is important to note that a low number of observations has not in all cases resulted in high standard deviation, especially the noticeable "stripes" of high SD. Referring back to Figure 7.5, we see that these stripes also largely correspond with several different groups of stimuli from particular types of rhythms. Thus a second condition for creating high S D seems to be that the stimuli being averaged are from either the long-even group, or the two additional small subgroups of stimuli that also have high SD. The nature of these two conditions is largely quite encouraging. B y design, the areas of the dissimilarity matrix with the highest number of observations are also the areas with the greatest amount of overlap between subsets. Yet these areas of high overlap in fact generally correspond to values with low standard deviation; and even with the noise-reducing role of increased observations, i f subset overlap was a source of truly discrepant data, we would not expect to see this. What this correspondence seems to indicate is that having observations coming from multiple different subsets has not been a noticeable source of noise in our data—the judgments from different subsets have generally converged. Instead, our major source of noise appears to be simply the effect of individual differences, regardless of subset, causing havoc within an average only when it contains too few observations. Furthermore, adding in "more observations (as was done in the second part of the study) served to decrease overall standard deviations levels (see Figure 7.10) while simultaneously adding in more subsets. Value of Having Many Subsets 97 However, a potential confound to the claim that increased observations brought about better data is that, in adding in seven more participants we also added in seven more subsets. Thus it could be argued that it was adding in subsets and not observations that increased the quality of the data. Though we had previously been concerned that different subsets would produce markedly different judgments, we had also tried to mitigate this problem by creating random subsets, that would, on average, not suffer too heavily from this problem. Thus it could be argued that adding in more subsets simply allowed for the differences between subsets to be covered over more evenly (indeed, a positive role which we originally hypothesized in our argument for subset randomization). . We did not find evidence of the subset-specific effect which the problem subset randomization was meant to mitigate. Furthermore, the only positive effect we would supposedly be gaining from adding in more different subsets would be to counter this effect. It would almost then seem that our entire attempt to create randomized subsets was of no value, given that the main effect that it was meant to counter was not found - at least for this stimulus set. Yet it would be premature to conclude that randomization is unnecessary, for two main reasons. First, we have not shown that there wi l l never be subset-relative effects, merely that they were not evident in the current results. Indeed, it seems intuitive to us that there must be at least some variation due to subsets occurring, i f only at a fairly small level, and that randomization would still be the best means of handling this problem. Thus we would argue that randomization acts as a sort of "safety net" that should help guard against effects of subset, should any manifest themselves; and further, could help to identify when larger subset effects do occur through a simple subset-overlap SD analysis. The second point is efficiency: our subset randomization algorithm also helps to minimize the number of subsets needed in order to have a certain number of observations for all values in the dissimilarity matrix. This helps cut down on the number of participants needed to run a study using the subset method and at the same time optimizes uniformity of number of observations across all stimulus pairs, which is very important 98 given that one of the major tradeoffs of the subset method is the number of participants needed to gather data. In fact, if we had used unique randomized subsets from the start, we would only have required 17 participants to gain the same number of observations (as opposed to 22), and the distribution of observations would have been considerably more even, with a range of 5 to 12 rather than 5 to 17 observations per dissimilarity value. Types of Outl iers and Method to Deal wi th Them Another way viewing the argument put forward above is in terms of outliers. In our analysis of standard deviation, it was noted that there were two main possible sources of outlier data points that could cause noise in the data: individual differences and idiosyncratic subsets. Furthermore it was noted that these two sources produced different types of outliers, and could be dealt with separately using different methods. The common source of outliers in psychological experiments is from individual differences in perceptual, physical or cognitive ability. We found this most evident in the five groups of three participants that each judged the same subset. Despite all experiencing the same subset, they still exhibited a high degree of standard deviation, even more so than where subsets overlapped, though not at a level that their results were entirely different (see Appendix B for individual plots). Nevertheless these differences in opinion must be attributed to individual differences. The standard method of dealing with such individual differences is to ensure the participant pool is representative of the overall population and to use enough participants to gain a representative sample. Whi le we made our best effort to ensure participants were representative, we found that we had initially gathered too few observations for many of the data points. Thus we used increased observations to guard against outliers from individual differences. The second source of outliers was subsets, though we did not find strong evidence of this being a large source of outlier data in the present data set. Nevertheless, we have not disproven that subsets could exist that would greatly skew any judgments given from it. Thus it is useful to consider the fact that the outliers caused by such subsets would occur not at the level of individuals, but at the level of entire subsets. Instead of one individual 99 producing outlier results, one subset could produce outlier results over and over again, each time it was used by a new participant. To guard against this form of outlier, we used unique randomized subsets, so that the cost of any one outlier subset is mitigated by, on average, having many more subsets that do not cause strong subset-specific effects. In essence, adding in more randomized subsets and adding in more observations both help reduce the effect of outliers, but at different levels. More subsets reduces the effect of outlier subsets, more observations reduce the effect of outlier individuals. 7.4.3 "Striping" of Standard Deviation One last particular facet of the standard deviation results that does bear further examination is the "stripes" of high standard deviation in the aggregate dissimilarity matrix. As can be seen in Figure 7.3, the stripes are very distinct and generally occur in groups of four or five consecutive stimuli, though several single-stimulus stripes are also evident. While these stripes were observed to occur due to a combination of a low number of observations with certain groups of stimuli, adding more participants and subsets removed the stripes of extreme high levels of SD. Yet i f we observe the distribution of SD within the new aggregate dissimilarity matrix (Figure 7.10) we find striping again, though at a much lower absolute level of standard deviation. Some of these stripes are in similar places as before, but many of them are not. Furthermore, their correspondence with areas of lower numbers of observations is less marked than before. Thus we are forced to conclude that the striping at least somewhat comes from the algorithm that creates the subsets (see Chapter 4, Section 4.2 for details). The most l ikely explanation is that those stimuli which are placed into subsets made first by the subset algorithm (as is the case for many, but not all, of the stimuli that are part of stripes), end up not being used very often in later subsets, since they have already gotten the requisite number of observations assigned to them via the subset creation algorithm. This in turn causes other stimuli that occur more in the latter subsets to have a much more randomly distributed number of observations, since it becomes harder and harder for the subset algorithm to come up with stimulus pairings that do not create overlap. Thus stimuli used 100 in early subsets wi l l l ikely appear in fewer different subsets, receiving fewer total observations, and thus higher standard deviation. This potential weakness of the subset algorithm was admitted early on, and it could possibly be dealt with through continued iterations on the subset algorithm. However, once lower than a certain threshold of S D , there seemingly appears to be no longer a great effect df the striping on the actual output of the M D S algorithm. Moreover, if we had used completely randomized subsets (rather than re-using the first five) this effect may have been even smaller to begin with. 7.5 Summary With all 2 2 participants run through our subset study, the results we produced were very similar, both qualitatively and quantitatively, to the gold standard which we produced using an established, validated data gathering technique. Furthermore, the addition of more subsets was found to increase rather than decrease the quality of the M D S output. Thus our three criteria for validating our new experimental method were met. 4 The effect of different subsets on judgments made in the sorting task was found to be negligible compared to individual differences in judgments, as evidenced by the relative distribution of standard deviation for within-subset averaged dissimilarities and between-subset dissimilarities. The fact that we saw areas of high SD generally being associated with values that only a single subset contributed too, and that adding in more new subsets resulted in lower SD values, all point towards subsets not having a negative effect on overall agreement within the data. Thus criterion one was met. Criteria two and three were met due to the reasonable and interpretable ;results of running the dissimilarity matrix gathered by our subset method through the M D S algorithm, and its significant similarity to our gold standard. The 2 D M D S output from the subset study exhibited very similar trends of amplitude and rhythm as those found in the gold standard, full-set study. Furthermore, the two outputs were statistically similar in layout according to the alienation coefficient K, the best statistical measure available forjudging similarity of M D S results [4]; although we point out that this statistic must be used with caution in the current context, and paired with other means of analysis. 101 With all criteria met, and with results that are clearly similar to the gold standard, we can feel confident that our new subset method of gathering data for M D S wi l l indeed produce valid results in future studies. Thus validated, we can recommend the method for use with any similarly large stimuli set, as a means of gaining perceptual dissimilarity ratings from users quickly and accurately. 102 Chapter 8: Conclusion We set out in this thesis to accomplish two main goals. Firstly we wished to create a large and diverse set of haptic signals, by using rhythm as a parameter that could potentially increase a set's expressive range; and to ascertain the perceptual dimensions by which users actually categorize these signals. This first goal brought about the second goal, which was to develop a new means of evaluating the perceptual characteristics of such a large set of haptic stimuli. This we accomplished through the development and validation of a novel technique for allocating stimuli to participants, allowing smaller subsets to be tested separately on different users and greatly easing the task of data gathering. In each of these goals we produced contributions to the field. The use of rhythms as a design parameter opened up a huge design space that easily allowed for the creation of a large set of different haptic stimuli. Prior work on haptic rhythms [24] [5] had only shown some evidence of this promise. Furthermore, people responded well to the use of rhythms in haptic stimuli; they were able to discern different aspects of rhythm within the stimuli, and distinguished the stimuli accordingly. Not only was the stimulus set successful, but our new experimental methodology proved valid as well. B y producing results that are comparable both statistically and practically to an established gold standard we showed that breaking the data gathering task into subsets of the total, and then building the overall picture out of the pieces, is indeed a valid way to gather perceptual data about a stimulus set. The particular contributions of each of these successfully met goals are described in the sections below. At a high level, these contributions mean that we have crossed a major hurdle in what we can do with haptic icons. No longer confined to small numbers used in restricted laboratory studies, haptic icons can now be produced and analyzed in set sizes of more broadly practical utility, given human perceptual abilities. Larger scale haptic icon production can ensure that a designer wishing to use haptic icons can find the types of stimuli that he or she wants, can find enough of them, and can know how each of them 103 wil l feel relative to each other. This is a strong step towards mainstreaming the use of haptic icons, and bringing them into the world of practical application development. 8.1 Conclusions on Rhythms for Haptic Icons When we were trying to find a means of enlarging the number of haptic icons that could be made, rhythm intuitively seemed like it might have depth enough to allow this. Our intuition was amply repaid by the results we found. We made a simple first trial at creating haptic stimuli using rhythm, attempting to use as few parameters as possible in order to allow for easier interpretation. Thus we did not even tap all of the potential aspects of rhythm—not to mention melody—that could be used with haptic stimuli. Nevertheless we easily created a set of 84 haptic stimuli that we intuitively believed, and later confirmed, to be distinct. As always with haptic stimuli, designing them was one problem, but determining how people actually perceived them was a second, and in many ways more difficult problem. B y using an established data gathering technique, we were able to build a dissimilarity matrix for all stimuli which could be applied to M D S . Analyzing the resultant output map, we were able to gain great insight into how these rhythm-based stimuli were perceived. Perceptually, our stimuli were distinguished first by their amplitude, though not by frequency. This result counters results of previous studies that found both amplitude and frequency to be of importance [17]. This difference is perhaps because after amplitude, it was the previously untested parameter of rhythm that appeared to be the most important distinguishing feature. Rhythms shortened the overall time people were exposed to the frequencies of the vibrations, the quick succession of relatively short notes overwhelming the effect of frequency. This strong role of rhythm, and the conclusions we were able to make about how haptic rhythms are perceived, is a major contribution of this thesis. Our analysis showed that our haptic rhythms were perceived according to two orthogonal axes. Though these axes may be particular to the types of rhythms we chose to study, we 104 hope that the simplicity of our rhythms should make our insights a solid grounding for more complex instances. The first axis was what we termed "note length," while the second was "evenness" of the rhythm. How a stimulus was perceived along the "note length" axis depended on the longest note present in the rhythm. If a rhythm contained a half note or longer, it would fall on the " long" end of the note-length axis; i f it contained only notes shorter than a half note, then it would fall on the "short" end of the axis. The "evenness" of a rhythm was determined by whether there were notes or rests of differing lengths within a given rhythm. If so, then the rhythm wi l l feel distinctly uneven or unbalanced; conversely i f a rhythm has only the same length notes and rests, it wi l l feel even. This was a consistently reported perception that showed itself clearly in the data, yet was an unexpected perceptual classification which was not part of our original rhythm creation scheme. Nevertheless, these two axes of perception for rhythm appear to be strong and robust. Not only that, but they provide useful tools for future designers to predict how the stimuli they make wi l l be grouped perceptually. Compared to previously created sets of haptic icons, we succeeded in making an icon set larger than any yet created. The next largest, the tactile melodies created in [24], was 53 icons, but these icons were collected at random from a database of real-world musical melodies. More systematic icon sets like those created in the work of MacLean and Enriquez [17] or Brown et. al. [16] are generally even smaller, at 36 and 27, respectively. Thus we have roughly doubled the size of any prior icons sets. Furthermore, the perceptual axes our icon set exhibited—such as "note length" and "unevenness"—have a much larger and more interesting space for growth compared to such prior axes as amplitude or waveform; as they are not,strictly ordinal in nature there is much more room for creative design. The different perceptual axes found for rhythmic haptic stimuli, even when using a fairly simple set of rhythms, already show interesting and novel ways that stimuli can be distinguished. Our stimulus set has shown very promising results, with interesting perceptual features leading to an easily diversified set of rules about perception of haptic rhythms. The detailed insight into how these haptic stimuli are perceived has made them 105 of great use to designers who wish to use haptic icons in their application, or to create more haptic icons themselves. 8.2 Validation of MDS Data Gathering Technique Our new method of gathering perceptual judgment data also proved successful. B y splitting the data gathering task into smaller subsets, we were able to greatly reduce the amount of time it takes for a user to perform a complete set of judgment tasks. This sub-sectioning allows for much more feasible experiment times, and ensures that fatigue is considerably less of an issue for the judgments given. Fatigue in particular is a-very large issue, as other methods of data gathering such as pair-wise comparisons [20] can be heavily affected by drifts in judgment criteria caused by fatigue or loss of attention over time. Our new technique allows experimenters to select whatever size stimulus set they would like, giving them complete control over how hard they wish to push the participants in their study. The only major tradeoff of this technique is that the smaller the subset of stimuli presented to the user, relative to the superset, the more participants wi l l have to be run in order to gain the same amount of data. For practical purposes, we suggest that a subset / superset size ratio less than one third wi l l require an impractical number of participants; in our case, with a subset size of 50, this would allow for a superset of 150, which is almost twice again the size of our current—quite large— stimulus set. Thus we still have considerable room for growth before our data gathering method reaches its capacity. To review the work we accomplished in designing and validating this new method of data gathering, we first proposed a simple study design based on combining two existing methods of gathering data for M D S . B y using the sorting method as used by MacLean and Enriquez [17] among others, as well as the incomplete-set design described by Spence and Domoney [20], we developed the subset method, whereby each participant in the study was presented with a subset of the total stimulus set, with the average dissimilarity matrix being created out of the patchwork coverage of the various overlapping subsets. 106 We then validated our new technique by running a study using the new method on the same rhythmic haptic stimulus set that we had already studied using an established, but more cumbersome technique (the "gold standard"). What we found was extremely encouraging. Using 12 different subsets and 22 participants, we produced M D S results that both visually and statistically highly resemble the M D S results from the gold standard. The same trends of amplitude and rhythm that we found in the gold standard study were also found in the results of the subset study. Furthermore, the statistic of similarity K [3], also showed the two results to be statistically similar at /?=.05. Perhaps even more convincing, was that when presented initially with somewhat unclear results, it was running additional participants with more different subsets that improved the results to a point at which they were clearly similar to the gold standard. It was seen as a potential stumbling point of our method that participants would make their judgments completely relative to the stimuli present in their subset, and thus each set of judgments would be highly dependant on the subset they were from. However, we have shown that by randomizing subset selection, adding in more subsets (along with more participants) actually increases the accuracy of the averaged data. Thus our main concern was allayed, and our resultant M D S output was confirmed as similar to the gold standard. Consequently we feel confident that our new subset data gathering method wi l l produce valid results, and can be used in cases where the size of the stimulus set that needs to be tested is larger than any one user can be reasonably expected to make judgments on in a single sitting. 8.3 Future Work There are two main areas of future work that start where this thesis ends off. The first is a further refinement of the haptic stimulus set using the insights we gained from our studies, and the second is a larger goal of applying these haptic icons in more in-depth applications. 107 Given what we found in our twin studies on the rhythmic haptic stimulus set, there are clear indicators of which design parameters we used were most important to people perceptually, and which were not. Certainly amplitude, length of notes present in rhythm and evenness of rhythm are among the former, while frequency (in the ranges tested, and in the presence of the more salient rhythmic variation) is among the latter. Thus if we wish to improve upon our stimulus set, making each stimulus more distinctive from the others as well as providing logical grouping for the stimuli, we need to take our findings into account and redesign the stimulus set accordingly. This redesigned stimulus set would have to be tested again with users (most l ikely using our new data gathering method), so that we could feel confident that our stimulus set actually exhibited the perceptual characteristics we anticipate. With a well-designed stimulus set in tow, our next task, and the true aim of all the work that has gone into this, would be to apply the stimulus set as haptic icons in an interesting application and test it with users. Specifically, our hope is that with such a large stimulus set, we might be able to study the use of a haptic icon-enabled application over a longer period of time, in order to determine the upper maximum of how many different haptic icons a user can reasonably handle in-an.application. Due to the novel nature of the sensation haptic icons deliver, it is our belief that users have to struggle considerably with overcoming the novelty and unfamiliarity of the feeling before they are able to use haptic icons successfully. Yet i f users were exposed to haptic icons for a longer period of time, to the point at which these sensations became normalized, then we might be able to determine such things as just how prevalent the use of haptic icons can be in an application, how many haptic icons people can learn to use, and how useful haptic icons can be for designing usable interfaces. It is our sincere hope that the work done in this thesis wi l l provide a key piece in answering these questions. 108 Chapter 1; Motivation & Approach Chapter 2: Chapter 3: Design of Rhythmic Haptic Stimulus Set Chapter 4: Subset MDS Data Gathering Method Chapter 5: Methods & Apparatus Chapters: Investigation of Rhythmic Haptic Stimulus Set Chapter 7: Subset Method ValWalion Study Chapter 8: Conclusion Figure 1.1 Logical structure of thesis. Chapters 3 & 6 pertain to the design and evaluation of the rhythmic haptic stimulus set. Chapters 4 & 7 pertain to the experimental design and validation of the new data gathering method. Chapter 5 describes the methods and apparatus that are shared by the studies described in Chapter 6 and 7. 1 0 9 Table 3.1. Note Types Used in Rhythms. Though at its smallest level of granularity there are 16 different slots in which vibration can be played, logically the notes are arranged either according to whole, three-quarter, half, quarter and eighth notes. Each note consists of both the time in which the vibration is played as well as the off-time where no vibration is played that is needed in order for one note to be distinguished from the next. Rest notes are referred to in the same manner as normal notes, except no vibrations are played. R# Note Type 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 Whole Note Three-Quarters Note Hal fNote Ha l fNote Quarter Note Quarter Note Quarter Note Quarter Note Eighth Eighth Eighth Eighth Eighth Eighth Eighth Eighth 110 Table 3.2. I^ythms Used in Stimulus Set. Each row represents one "bar" that is repeated four times over a 2 second interval to make a rhythm. Within each bar, a note is demarcated by a pair of bold black lines. Notes contain both the on-time of the vibration plus the off-time that allows each note to be distinct from the next. Thus within each note there is a grey area indicating a time period where vibrations are playing and a white area indicating no vibrations are playing. This is except for rest notes, which are all white. See Table 3.1 caption for explanation of types of notes. R# I Notes I GROUP 1 111 Table 3.3 Lookup Table for Stimulus Set. Stimulus numbers are used to refer to individual stimuli throughout the remainder of this document. We used a total of 84 stimuli, which consisted of 21 rhythms which varied as described in this chapter, combined with 2 amplitudes and 2 frequencies, distributed as described here. High Amplitude Low Amplitude Rhythm # High Frequency Low Frequency High Frequency Low Frequency 1 1 22 43 64 2 2 /: 23, 44 65 3 '3 24 ' 45 66 4 4 25 46 67 5 5 26 47 68 6 6 27 48 69 7 7 28 49 70 8 8 29 50 71 9 9 30 51 72 10 10 31 52 73 11 11 32 53 74 12 12 33 54 75 13 13 34 55 76 14 14 35 56 77 15 15 36 57 78 16 16 37 58 79 17 17 38 59 80 18 18 39 60 81 19 19 40 61 82 20 20 41 62 83 21 21 42 63 84 112 Number of Subsets Required to Gain Minimum 5 Observations Vs. Subset Ratio 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Subset Ratio (NSINT) Figure 4.1. Graph of subsets versus subset ratio, for a desired minimum of five observations per dissimilarity value. For any ratio of subset size (NS) to total set size (NT), between 0 and 1, our subset creation algorithm attempts to create the minimal number of subsets that w i l l ensure that at least the required number of observations are present for each point in the dissimilarity matrix. The resultant curve is shown above. A s can be seen, anything lower than a ratio of roughly a third requires a number of subsets that is quite unreasonable for practical purposes. 113 Pilot Study, Stress Values 0.1 -• 0 J , , , , 1 2 3 4 5 Dimension Figure 4.2. Stress values for the first five dimensional M D S solutions. No distinct elbow in the curve can be seen 114 Figure 4.3. 2D Map of M D S Results. Trends of frequency and amplitude are as displayed. Green and blue stimuli are high amplitude, grey and orange stimuli are low. Green and orange stimuli are high frequency, blue and grey stimuli are low frequency. Icon numbers are as in Table 3.3. A s can be seen, rhythms do not appeared to be grouped together, and even frequency and amplitude do not appear to have an overly strong grouping effect. 115 Figure 5.1. The Nok ia 770 12! 1! 46! 27! 48! 42! 2! 9! 49! 43! 23! 10! 4! 13! 47 18 45 11 15 37 38 19 31 50 21 14 41 20 35 34 39 25 28 17 29 33 44 40 16 26 22 Colour: System status: 36 1 II II II 1 J 1 1 II 1 Sort 1 II II II II 1 1 F h 7 Figure 5.2. The M D S stimuli sorting program, with 50 stimuli. 24 done 116 Full-Set MDS Stress Values 1 2 3 4 5 6 Dimension Figure 6.1. Stress values for the first six dimensional M D S solutions. No distinct elbow in the curve can be seen. 117 29 2 3 22 1 5 « 26" 38 17 16 46 14 80 58 se Low Frequency 1 3 7 11 6 30 High Amplitude 4 2 1 2 9 / 9 / " 2 7 2 1 / * * » / 4 0 / 1 8 35 s 7 59 / 79 47 / 78 / "77 / 56 / 43 / 5 7 Low Amplitude 66 65 51 7232 High Frequency 50 71 52 74 S3 45 a 7 0 5 5 « 4 7 6 53 W 73 48 60 6| 9 Figure 6.2. 2D M D S output, with all 84 stimuli plotted. Green and blue stimuli are high amplitude, grey and orange stimuli are low. Green and orange stimuli are high frequency, blue and grey stimuli are low frequency. Projected axes are labeled accordingly. See Table 3.3 for a lookup table of individual stimulus numbers, and in particular to identify their rhythm. 118 a 36 38 17 16 High Amplitude « 1 9 2 828 Short Notes _ F 59 79 *7 78 6"77 1*27 2 1 Long Notes 5C Low Amplitude 65 51 7T1 5 5 4 4 7K 45 S 3 7 0 54 7 6 53 " „ 46 60 63,9 Figure 6.3. 2D M D S output, with all 84 stimuli plotted. Projected axes are labeled accordingly. Green stimuli are from Groups 1 and 4, containing only "short" notes; blue stimuli are from Groups 2, 3 and 5, containing " long" notes. See Table 3.3 for a lookup table of individual stimulus numbers and Table 3.2 for a lookup of rhythm groups. 119 Figure 6.4. Distribution of standard deviation values for averaged dissimilarity matrix. Black squares indicate high S D , grey-blue medium-high, grey medium-low, and white squares have lowest SD. 120 Low Amplitude High Frequency Low Frequency High Amplitude , Figure 6.5 M D S plot for stimuli only containing long notes. Green and blue stimuli are high amplitude, grey and orange stimuli are low. Green and orange stimuli are high frequency, blue and grey stimuli are low frequency. Projected axes are labeled accordingly. See Table 3.3 for a lookup table of individual stimulus numbers. 121 40 33 High Amplitude 3G 22 5 25 3 7 Long-. Uneven I! " G7 47 43 Low Amplitude S15 83 70 82 53 73 °" 49 68 „ Figure 6.6 X - Y plane of 3D M D S for all 84 stimuli. X is the horizontal axis, Y the vertical. The green stimuli are from the short-even group, the blue are the short-uneven, the orange the long-even, and the grey are long-uneven (see text for explanation of group names). Projected axes are labeled accordingly. 122 45 65 66 1 56 35 B 46 Figure 6.7 X - Z plane of 3D M D S for all 84 stimuli. X is the horizontal axis, Z the vertical. The green stimuli are from the short-even group, the blue are the short-uneven, the orange the long-even, and the grey are long-uneven 123 Figure 6.8 Y - Z plane of 3D M D S for all 84 stimuli. Y is the horizontal axis, Z the vertical. The green stimuli are from the short-even group, the blue are the short-uneven, the orange the long-even, and the grey are long-uneven. Frequency axis is omitted due to space constraints, but is similar in size to Figures 6.6 and 6.7. 124 6 28 8 2 7 3 « 1 9 Long Notes 33 35 29 4 Even 42 / 32**1 / 10 / 13 / 2 * 1 34 12 / ^ Uneven 2 19 i i 40 ~ — 3 7 . 25 17 Short Notes 1 6 38 5 36 24 23 15 3 Figure 6.9. High-amplitude only M D S sub-analysis. The green stimuli are from the short-even group, the blue are the short-uneven, the orange the long-even, and the grey are long-uneven. Projected axes are labeled accordingly. 1 2 5 • 71 High 2 Low Amplitude Frequency B, ^ » Fluency " ^K** * * * > TD M P u • 77 M High H 1 7 High Amplitude Frequency LOW Fnquncy L w " Amplitude 1 • ID S3 High 6--» Frequency Low Amplitude j I high Frequency 1 j High Amplitude | " SghAmplltude Low 1 3 » „ Frequency Ml fi « • Low Amplitude ^Low Frequency • Figure 6.10 Individual M D S plots for the four rhythm groups. Clockwise from top left: long-even, short-even, short-uneven, long-uneven. Green and blue stimuli are high amplitude, grey and orange stimuli are low. Green and orange stimuli are high frequency, blue and grey stimuli are low frequency.. 126 1 4 17 25 38 29 2 3 22 1 5 36 24 3 11 High Amplitude 7 6 30 4219 2 8 2 8 16 14 46 9 2ft % 4jV» Short-Even 30 / 40 _ 39 58 G8 Short^" — _ Long-18 3 S 6 7 58 Uneven / — - — ^ E v e n * Long-Uneven 79 47 78 5 4 77 50 56 43 j 57 Low Amplitude 74 71 52 66 65 51 7332 63 m 5 5 4 4 TC 45 ft 5 4 53 W 7 3 « 60 6 | g Figure 6.11. 2D M D S output with all stimuli. The green stimuli are from the short-even group, the blue are the short-uneven, the orange the long-even, and the grey are long-uneven. The middle axis is made up of the plotted centroids of the 4 groups, as labeled. We consider this to be the gold standard M D S map for our stimulus set. Table 6.1. S-Stress and r2 va ues for individual groups 2D M D S output Group S-Stress r1 Long-even .23096 .82260 Long-uneven .27128 .73703 Short-even .27397 .71022 Short-uneven .22747 .79722 127 Table 7.1. Mapping of Subse ts to Participants. Subsets are c efined in Appendix C . Subset 1 Subset 2 Subset 3 Subset 4 Subset 5 Part ic ipants 1,6,11 2,7,12 3,8,13 4,9,14 5,10,15 50 Icon Subset MDS Stress Values 1 2 3 4 5 6 Dimensions Figure 7.1. For first round of subset study, stress values for dimension 1 to 6 of the M D S solutions. 128 Figure 7.2. For first round of subset study, 2D M D S output map of results. Grouping is the same as in Chapter 6. Green is the short-even group, blue is short-uneven, orange is long-even, and grey is long-uneven. A s can be noted, the orange long-even group is significantly out of place from the ordering in the gold standard, out on the far right of the map. 129 |Gr.1 Igrj I Gr.3 I Gr.4 |Gr .5 |Gr.1 |Gr .2 [Gr.3 I Gr.4 \Gr.S | G M IGr.2 | G r . 3 | Gr.4 |Gr.5 JGr.1 |Gr .2 IGr.3 I Gr.4 |Gr.5 Figure 7.3. For first round of subset study, distribution of standard deviation values for averaged dissimilarity matrix. Black squares have the highest levels of standard deviation, blue-grey squares the next-highest, light-grey lower stil l, and white the lowest. Note the distinct stripes of darker (higher standard deviation) values, running along various columns and rows. 130 Gr.1 Gr.2 I Gr.3 I Gr.4 Gr.5 Gr.1 Gr.2 I Gr.3 I Gr.4 |Gr.5 |Gr.1 |Gr.2 |Gr.3 I Gr4 |Gr.5 |Gr.1 IGr.2 |Gr.3 I Gr.4 |Gr.5 Figure 7.4. For first round of subset study, distribution of standard deviation values for averaged dissimilarity matrix with stimuli groups marked in opposite half of matrix. Stimulus groups are labeled along the side and top; note that each is spread across 4 places in the matrix, once for each combination of amplitude and frequency. The four orange columns/rows correspond to the "long-even" group. The green columns/rows correspond to stimuli 14-17, members of Group 4 in Table 3.2, played at high amplitude and high frequency. The grey columns/row correspond to stimuli 39-42, members o f Group 5 in Table 3.2, played at high amplitude and low frequency. 131 |Gr.1 |Gr .2 | G r 3 | G r . 4 J G r . S J G r . 1 | G r 2 | G r . 3 I Gr.4 [ G r . 5 |Gr.1 | Gr.2 | G r . 3 | G r . 4 |Gr .S |Gr.1 I Gr.2 ] G r . 3 | Gr .4 |Gf .5 Figure 7.5. For first round of subset study, distribution of number of observation per value of averaged dissimilarity matrix with stimuli groups marked in opposite half of matrix. Stimulus groups are labeled along the side and top. The colour coding for the number of observations is that dark purple values have 3 observations, light purple have 6, white values have 9 or greater. Coding of stimuli groups is the same as in Figure 7.4 132 50 Icon Subset MDS w/Additional Participants: Stress Values 1 2 3 4 5 6 Dimensions Figure7.6. Stress values for dimension 1 to 6 of the M D S solutions for the subset study with additional participants (second round of subset study). 133 Figure 7.7. 2D M D S output map for subset study with additional subjects (second round of subset study). Green is the short-even group, blue is short-uneven, orange is long-even, and grey is long-uneven. 134 7 6 28 27 33 Long Notes 21 ^ S " ^ -1 0 31 9 29 8 30 5 17 26 4 Even 25 3B 37 13 1 2 2« 2 / 41 34 18 39 ;S Uneven i i 40 1 9 32 ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ 16 Short Notes 24 36 3 5 22 2 1 3 1 5 14 Figure 7.8. 2 D M D S map of only high amplitude stimuli (second round of subset study). Green is the short-even group, blue is short-uneven, orange is long-even, and grey is long-uneven. Two perceptual axes are labeled accordingly. 135 Gr.1 |Gr2 | Gr.3 I Gr.4 \Gr.S jsr.1 JGr.2 I Gr.3 | Gr.4 |Gr.5 JGr.1 1 Gr.2 I Gr.3 I Gr.4 I Gr.5 |Gr.1 [Gr.2 I Gr.3 I Gr.4 |Gr.S Figure 7.9. For the first round of the subset study: one half of this matrix shows standard deviation (bottom left triangle) while the other half shows the number of observations (top right triangle). Stimulus groups are labeled along the side and top. Color coding of SD : Black squares have the highest levels of standard deviation, blue-grey squares the next-highest, light-grey lower stil l, and white the lowest.. Colour coding of number of observations: dark purple values have 3 observations, light purple have 6, white values have 9 or greater. It can be seen how the (darker black) stripes of high SD correspond to the (darker purple) areas of low numbers of observations. There are however many areas of low observations that do not correspond to areas of high S D ; thus we claim that both low numbers of observations plus more "diff icult" stimuli are needed to create stripes of high SD. 136 Gr.1 |Gr .2 | Gr.3 I Gr.4 |Gr.5 |Gr.1 |Gr .2 I Gr.3 | Gr.4 |Gr.5 |Gr 1 |G f .2 | Gr.3 | Gr.4 \Gr.5 JGr.1 [Gr.2 | Gr.3 | Gr.4 JQr.S Figure 7.10 Standard deviation and number of observation for subset study with additional participants (second round of study). Format is same as in Figure 7.9. Stimulus groups are labeled along the side and top. Color coding of SD : Black squares have the highest levels of standard deviation, blue-grey squares the next-highest, light-grey lower stil l, and white the lowest. Colour coding of observation values differs slightly, dark purple values have 5 observations, light purple between 6 and 9 observations, white values greater than 9 observations. A s can be seen, overall level of SD is much lower, and overall level of observations is much higher and more evenly spread 137 References [I] Bijmolt, T.H.A, and Wedel, Michel. (1995). The effect of alternative methods of collecting similarity data for Multidimensional Scaling. International Journal of Research in Marketing, 12. pp 363-371. [2] Bonebright, T.L. (2001). Perceptual Structure of Everyday Sounds: A Multidimensional Scaling Approach. Proc. Of the 2001 International Conference on Auditory Display, Espoo, Finland, pp. 73-78. [3] Borg, I. and Groenen, PJ.F. (1997). Modem multidimensional scaling. 2nd edition. New York: Springer. [4] Borg, I. and Leutner, D. (1985). Measuring the Similarity of MDS Configurations. Multivariate Behavioral Research, 20. pp 325-334. [5] Brewster, S. and Brown, L.M (2004). Tactons: Structured Tactile Messages for Non-Visual Information Display. In Proc of5'h Australasian User Interface Conference (AUIC2004), Dunedin. [6] Brown, L.M., Brewster, S.A., and Purchase, H.C. (2005). A First Investigation into the Effectiveness of Tactons. First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environments and Teleoperated Systems (WHC'05). [7] Brown, L.M., Brewster, S.A., and Purchase, H.C. (2006a). Multidimensional Tactons for Non-Visual Information Presentation in Mobile Devices. MobileHCI'06. pp 231-238. [8] Brown, L.M., Brewster, S.A., and Purchase, H.C. (2006b). Tactile Crescendos and Sforzandos: Applying Musical Techniques to Tactile Icon Design. CHI '06 extended abstracts on Human factors in computing systems, Montreal, Canada, pp 610-615. [9] Brown. L.M. and Kaaresoja, T. (2006). Feel Who's Talking: Using Tactons for Mobile Phone Alerts. CHI '06 extended abstracts on Human factors in computing systems, Montreal, Canada, pp 604-609. [10] Chan, A. (2004). Designing Haptic Icons to Support an Urgency-Based Turn-Taking Protocol. M.Sc. Thesis. Univ. of British Columbia. [II] Chan, A., MacLean, K., and McGrenere, J. (2005). Learning and identifying haptic icons under workload. Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2005. WHC 2005. pp 432-439. .': ... " • [12] Enriquez, M., MacLean, K. E., and Chita, C. (2006). Haptic Phonemes: Basic Building Blocks of Haptic Communication, in Proc. of the 8th International Conference on Multimodal Interfaces, ICMI'06, Banff, Alberta, Canada, November 2006. [13] Grey, J. M. (1977). Multidimensional perceptual scaling of musical timbres J. Acoust. Soc. Am., 61:5. pp. 1270-1277. 138 [14] Hoggan, E.E., and Brewster, S.A. (2006). Crossmodal Icons for Information Display. CHI '06 extended abstracts on Human factors in computing yysfem.s,'Montreal, Canada, pp 857-862. [15] Hollins, M, Faldowski, R., Rao, A., and Young, F. (1993). Perceptual dimensions of tactile surface texture: a multidimensional scaling analysis. Perception & Psychophysics, 54(6). Pp. 697-705. [16] Klatzky, R.L., and Lederman, S.J. (1999). Tactile roughness perception with a rigid link interposed between skin and surface. Perception & Psychophysics, 61 (4). pp. 591-607. [17]Laitinen, P., and Maenpaa, J. (2006). Enabling mobile haptic desing: Piezoelectric actuator technology properties in hand held devices. In proc of. IEEE International Workshop on Haptic Audio Visual Environments and their Applications (HAVE2006). [18]Luk, J., Pasquero, J., Little, S., MacLean, K., Levesque, V., and Hayward, V. (2006). A Role for Haptics in Mobile Interaction: Initial Design Using a Handheld Tactile Display Prototype, in Proc. of ACM Conference on Human Factors in Computing Systems (CHI '06), Montreal, Canada, pp. 171-180 [19] MacLean, K., and Enriquez, M. (2003). Perceptual Design of Haptic Icons, in Proc. ofEurohaptics, Dublin, Ireland, pp. 351-363. [20] Pasquero, J., Luk, J., Little, S., and MacLean, K. (2006). Perceptual Analysis of Haptic Icons: an Investigation into the Validity of Cluster Sorted MDS. In Proc of Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS'06). pp. 61-1 A. [21] Shepard, R. N. (1962). The analysis of proximities: Multidimensional scaling with an unknown distance function. Part I. Psychometrika, 27. pp 125-140. [22] Spence, I, and Domoney, D. W. (1974). Single Subject Incomplete Designs for Nonmetric Multidimensional Scaling. Psychometrika, 39 (4). pp. 469-490. [23] Tan, H.Z., Durlach, N.I., Reed, C M . , and Rabinowitz, W.M. (1999). Information transmission with a multifinger tactual display. Perception & Psychophysics, 61 (6). pp. 993-1008. [24]Tikka, V., and Laitinen, P. (2006). Designing Haptic Feedback for Touch Display: Experimental Study of Percieved Intensity and Integrations of Haptic and Audio. HAID 2006. pp 344-360. [25]Tsogo, L., Masson, M.H., and Bardot, A. (2000). Multidimensional Scaling Methods for Many-Object Sets: A Review. Multivariate Behavioural Research, 35 (3). pp. 307-319. [26] Van Erp, J.B.F (2002). Guidelines for the use of vibro-tactile displays in Human Computer Interaction. In Proc. Of Eurohaptics 2002. pp 18-22. [27] Van Erp, J.tf.F, and Spape, M.M.A., (2003). Distilling the Underlying Dimension of Tactile Melodies. In Proc. OfEurohaptics 2003. pp 111-120. 139 Appendix A: Dissimilarity Matrices Proceeding pages display the averaged dissimilarity matrix for the pilot study described in Section 4.4. 140 1 2 3 4 5 6 7 8 9 1 0 2 993 0 3 1000 924 0 4 1000 978 989 0 5 872 966 972 908 0 6 978 1000 1000 983 898 0 7 1000 672 1000 1000 938 971 0 8 1000 1000 1000 950 956 889 893 0 9 973 907 966 871 933 956 950 994 0 10 1000 813 983 622 983 1000 700 951 891 11 978 770 862 674 1000 889 1000 963 848 12 973 938 800 987 989 907 1000 1000 800 13 989 938 926 806 989 1000 815 933 830 14 1000 978 1000 1000 989 1000 956 989 991 15 956 978 947 933 920 1000 1000 1000 920 16 1000 978 1000 972 1000 1000 978 989 965 17 900 933 944 939 858 1000 733 850 858 18 973 844 907 906 917 907 767 1000 1000 19 984 929 956 822 1000 1000 900 969 924 20 1000 973 944 840 833 983 989 947 811 21 971 939 844 973 983 960 956 922 822 22 965 991 830 991 956 898 972 922 960 23 937 989 785 711 956 889 878 1000 968 24 983 948 915 1000 1000 987 666 1000 911 25 870 830 867 956 922 938 585 839 982 26 967 967 889 967 739 839 889 916 933 27 969 924 972 972 807 938 806 911 961 28 989 983 1000 1000 871 933 1000 978 1000 29 839 1000 806 1000 960 1000 1000 972 963 30 900 883 844 978 822 939 883 772 878 31 948 939 630 969 920 939 989 989 944 32 898 987 778 1000 822 928 978 933 880 33 889 889 711 1000 956 828 1000 1000 889 34 937 938 978 933 922 791 889 1000 907 35 952 991 917 974 948 978 928 951 886 36 863 917 963 983 960 960 1000 973 970 37 944 870 926 1000 830 871 889 907 898 38 1000 889 978 1000 704 916 1000 1000 684 39 922 1000 885 1000 839 809 1000 719 917 40 929 983 856 1000 833 889 1000 1000 896 41 858 924 722 1000 807 844 983 1000 861 42 991 989 983 956 978 983 956 933 1000 43 911 978 794 987 972 862 878 811 982 44 889 961 1000 978 972 956 1000 939 972 45 893 989 793 830 1000 900 889 883 937 46 978 970 789 983 960 . . 950 844- 987 857 47 1000 938 922 929 836 972 939 951 906 48 985 941 839 950 702 920 917 878 928 49 943 950 1000 1000 967 950 964 982 993 50 904 • 983 ' 834 . 956 956 896 889 1000 936 51 978 1000 896 750 973 '•• 785 867 989 972 6£6 156 £96 968 ZLL f96 826 006 ££6 08 ZL6 £68 0001 688 686 0001 906 026 688 6Z, PZ6 868 6Z6 Z.86 0001 8Z.6 81^ 6 906 £66 8Z. t>58 818 999 51?9 228 0001 1^ 06 056 9Z.8 LL 8Z.6 PP6 PP6 £86 256 ZP6 996 088 688 9L 0001 868 Z.98 6£6 t/96 8Z.6 8Z,6 8178 6£6 SZ, 266 096 ,006 •'. .258 8Z.6 ... LP6- • 2Z.6 - . -056 686 PL 968 ££6 ; 0001. " . 156 8Z.6 0001 £96 r 088 0001 ZL Z.86 0001 0001 ZL6 PP6 £96 2Z.6 0001 0001 ZL 8 £6 956 688 0001 £86 £96 956 ZL6 £96 IL 608 6£8 99Z. Z.98 0001 £86 228 056 096 OL IZ.8 Z.98, -. 0001. . 0001 Z.86 ' Z.86 P96. • 8PL ' ££6 69 188 m 026 116 6£6 0001 £86 1^ 26 ' 688 89 258 6£6 969 956 006 918 2Z.8 0001 Z,86 Z.9 PP8 658 ZZL ££6 IZ.8 0001 0001 . \P6 8Z.6 99 006 ££6 8Z.8 6£8 858 116 0001 ££6 28Z. S9 166 586 586 0001 0001 686 0001 PLS 926 P9 306 0001 0001 t>88 956 2Z.6 m 8Z.6 008 £9 828 K 8 0001 . £58 £96 968 LP6 996 £86 19 0001 026 116 956 6£6 116 ££6 858 £58 19 ZL6 906 0001 916 9£8 226 ££6 £86 Z.86 09 Z66 196 00Z. 6£Z, Z.I6 058 9£8 686 ££8 6S 688 006 818 996 996 Z.86 228 056 858 8S £68 518 £6Z. Z.98 0001 £86 Z.98 926 LS 658 »V08 L06 6£8 858 166 0001 096 806 9S 0001 ZZ6 9Z6 Z.86 0001 686 8Z.6 116 858 SS 026 Z.86 586 8Z.6 . 916 166 Z.I8 8 £6 858 PS 066 096 0001 056 556 £86 228 95Z, If 6 £S m W.8 8Z.6 916 QLL 0001 858 0001 258 zs 6 8 L 9 S P £ Z I ZL6 68Z. \5L 8Z.8 £86 Z.86 918 036 0001 IS Z66 199 856 0Z.8 386 Z.86 0001 0001 U 6 OS 686 956 8£6 806 0001 858 0Z.6 8Z.6 0001 6P £86 056 Z.176 996 0001 ££6 0001 0001 £86 SP ZL6 ZL6 116 336 338 0001 £86 Z.86 118 LP 896 5P2 .686 IZ.8 £86 Z.86. ^36 106 IIZ. 9P ZZ6 ' 6 8 6 " £86 518 Z.86 £98 , ££Z. - 338 SP 996 0001 6Z6 Z.I6 0001 0001 686 388 0001 PP 0001 ILS Z.06 506 1788 8Z.6 506 Z.£6 8Z.6 ZP 858 0001 0001 £18 118 836 336 8Z.6 956 ZP 0001 116 58Z. 958 0001 0001 518 6£6 0001 IP 688 1^ 06 0001 95Z. 0001 1176 688 Z.86 8Z.6 OP 6Z.8 6PS PS6 Z.£8 816 I7Z.8 658 968 056 6£ 836 ZPL 956 996 0001 0001 688 58Z. Z,86 8£ 618 0001 356 £Z.6 £86 356 196 6176 958 LZ 996 156 0001 Z,£8 Z.86 t-96 956 £98 69Z. 9Z 9£6 PU PL6 £Z.8 £86 ZL6 686 556 Z.06 S£ Z.06 0Z6 088 AW £86 I £8 996 Z.16 m PZ ZZ6 lt>8 386 0Z.8 t796 0001 £68 066 8 £6 zz £58 ££6 0001 998 1^ 6 ££6 336 89Z. £86 zz 386 688 £96 116 £86 398 0001 t*6 0001 IZ 996 ££8 388 PLS 0001 Z.98 Z.96 036 0001 0£ 968 116 688 389 £Z,6 68Z, 0001 8 £6 0001 6Z 688 Z.176 £t76 8Z.6 096 956 356 0001 0001 SZ 008 588 586 S6Z, 0001 686 Z.98 958 £96 LZ 688 508 686 5£8 156 056 ££6 956 Z,£6 9Z Z.£6 Z.98 8t76 588 0001 686 0001 686 686 SZ 0001 116 508 3Z.8 Z.96 688 006 688 0001 PZ £P6 L\6 386 656 968 906 0001 916 336 zz 818 806 368 £96 836 K 6 656 £88 zz 686 0001 186 596 £86 916 9176 656 Z.86 IZ 816 £86 656 8 £6 8Z.6 Z.16 538 686 8W 03 8fr6 ZZ6 858 0001 088 ££6 686 0001 fr86 61 0 8178 838 0001 0001 8Z.6 056 0001 ZL6 81 0 £96 0Z.6 0001 0001 106 ££6 688 LI 0 986 0001 0001 0001 0001 0001 91 0 396 17176 £86 ££6 5178 SI 0 £176 0 W6 Z.38 0 Z.98 918 856 0 0001 Z.08 £Z.6 8Z.6 0 pi zi zi n 01 6 8 L 9 S P z z I 81 LI 91 ST H £1 ZI II 01 10 11 12 13 14 15 16 17 18 52 930 837 796 1000 972 967 941 871 1000 53 987 973 830 933 889 900 989 707 981 54 983 933 876 939 983 756 991 773 867 55 1000 1000 883 938 950 916 978 917 991 56 889 949 898 892 900 944 918 933 830 57 852 989 949 950 1000 841 968 828 911 58 978 1000 939 947 894 839 896 889 983 59 967 835 896 961 889 904 911 922 961 60 1000 987 950 1000 960 966 800 960 983 61 1000 1000 849 898 966 917 933 1000 839 62 862 956 867 966 978 800 806 809 928 63 944 883 828 918 952 966 978 844 933 64 1000 848 1000 985 1000 924 991 867 862 65 987 956 989 1000 1000 853 918 834 944 66 983 947 772 767 904 844 889 956 833 67 944 856 738 867 1000 806 792 805 978 68 792 1000 840 989 978 978 1000 817 919 69 987 884 966 1000 981 694 960 938 924 70 956 972 911 850 987 898 1000 871 1000 71 978 889 844 939 956 793 983 815 924 72 982 1000 875 1000 889 989 924 950 978 73 1000 938 889 906 963 1000 922 831 978 74 990 974 952 973 896 900 975 834 897 75 1000 904 856 789 950 978 933 1000 989 76 973 892 975 970 983 875 968 889 878 77 889 947 946 944 966 841 989 728 733 78 955 870 965 876 861 884 1000 956 893 79 794 1000 804 920 818 811 972 827 983 80 1000 882 874 960 872 993 889 917 994 1 19 20 21 22 23 24 25 26 27 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 0 20 980 0 21 878 928 0 22 917 939 978 0 23 882 964 813 958 0 24 1000 922 904 933 825 0 25 937 970 915 962 987 951 0 26 933 967 944 944 880 933 938 0 27 813 989 960 858 967 933 950 911 0 28 1000 1000 943 907 871 856 833 928 909 29 972 1000 1000 933 876 933 883 1000 978 30 939 933 970 878 934 961 867 861 928 31 1000 993 900 822 883 886 926 917 929 32 893 881 921 991 1000 908 950 1000 900 33 893 918 848 918 917 861 967 844 889 34 1000 978 827 964 911 960 933 907 1000 35 981 965 848 1000 883 1000 978 951 885 36 769 915 989 933 794 943 971 907 949 37 896 856 966 937 1000 924 943 900 924 38 1000 884 1000 956 793 876 717 767 898 39 915 978 858 870 933 933 990 956 952 40 1000 889 933 867 856 839 836 907 973 41 973 939 907 951 889 924 983 965 906 42 1000 942 939 889 876 872 907 964 872 43 918 903 895 987 973 930 924 1000 922 44 971 978 975 867 1000 915 960 806 787 45 774 835 966 1000 978 1000 900 1000 956 46 787 987 966 1000 856 933 981 809 852 47 993 1000 906 1000 906 ' 769 1000 991 704 48 1000 989 987 911 972 711 911 893 911 49 946 989 896 922 806 832 841 849 738 50 911 1000 911 933 884 901 863 963 911 51 989 " 978 906 1000 889 922 926 882 1000 19 20 21 22 23 24 25 26 27 52 867 879 899 948 960 879 973 893 914 53 1000 975 873 956 883 851 900 1000 989 54 1000 972 1000 1000 784 1000 987 1000 1000 55 874 982 951 966 872 1000 844 951 892 56 940 940 914 1000 1000 1000 959 783 978 57 908 902 911 978 839 978 882 989 944 58 925 917 845 972 811 862 907 942 852 59 765 737 871 943 941 927 830 889 911 60 985 920 938 972 983 800 983 961 782 61 772 991 876 989 966 726 811 1000 950 62 1000 844 1000 983 989 831 1000 972 858 63 1000 844 963 767 981 836 875 944 1000 64 993 898 989 922 678 942 858 893 951 65 937 981 959 939 1000 919 902 989 876 66 1000 756 852 844 840 872 961 1000 933 67 920 947 1000 1000 742 893 987 956 787 68 908 978 944 933 822 917 1000 956 944 69 815 IOOO; . • 1000 978 918 987 987 956 929 70 852 1000 966 966 938 900 853 871 841 71 972 1000 966 966 973 867 744 883 983 72 1000 - .896 R 973 929 898 907 833 963 956 73 1000 922 907 . . 1000 978 674 726 . ' • 983 973 74 941 • 982 956 982 987 822 944 ' - 8 7 4 989 75 922 844 827 933 889 1000 933 1000 883 76 990 854 841 805 978 982 952 1000 941 77 1000 733 987 941 917 867 911 778 956 78 930 914 841 987 950 952 956 1000 848 79 938 763 817 1000 956 889 1000 889 983 80 959 1000 915 1000 983 867 818 885 870 146 28 29 30 31 32 33 34 35 36 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 0 29 917 0 30 856 926 0 31 836 811 906 0 32 1000 898 881 931 0 33 960 920 815 956 891 0 34 900 900 978 939 902 870 0 35 933 933 867 911 744 839 995 0 36 885 933 978 1000 978 987 944 783 0 37 983 778 978 1000 898 960 978 943 921 38 898 834 830 844 884 983 778 933 963 39 818 944 878 916 926 922 863 933 978 40 951 938 939 933 1000 794 867. 867 833 41 960 872 822 800 925 951 844 920 822 42 756 911 849 778 763 972 926 983 828 43 778 1000 989 933 1000 916 825 981 896 44 983 . 987 933 941 918 1000 .915, 933 938 45 978 955 987 900 , 969 933 1000 .'• 944 992 46 983 955 933 978 933 966 952 1000 925 47 911 989 1000 1000 911 1000 944 900 966 48 956 939 907 978 961 989 950 1000 960 49 862 922 861 902 882 950 816 902 874 50 906 896 898 859 844 944 885 955 885 51 852 956 900 987 920 922 933 1000 783 28 29 30 31 32 33 34 35 36 52 893 861 882 930 946 950 778 833 973 53 900 950 874 898 737 889 848 895 878 54 778 906 933 939 662 1000 853 785 844 55 789 956 896 839 867 922 929 791 818 56 1000 951 943 896 870 965 908 943 943 57 978 833 937 956 893 896 894 964 989 58 725 889 852 800 756 1000 853 964 966 59 906 978 902 978 989 883 896 987 896 60 987 893 972 920 1000 885 874 987 839 61 1000 956 989 939 963 867 817 983 987 62 920 900 978 1000 939 1000 978 933 1000 63 702 900 966 944 917 817 828 983 983 64 893 989 944 978 911 917 867 852 944 65 1000 862 893 956 943 867 959 989 937 66 922 963 941 963 872 1000 793 922 983 67 952 884 831 1000 806 917 900 871 911 68 1000 916 978 989 906 806 898 889 844 69 947 933 872 978 947 933 956 738 1000 70 800 867 955 748 933 987 840 944 933 71 839 756 839 966 1000 941 844 983 973 72 800 1000 933 867 911 874 729 916 966 73 856 922 807 898 867 966 815 950 893 74 900 1000 911 874 822 989 800 900 978 75 950 898 883 951 911 871 898 1000 920 76 822 947 905 867 870 872 904 948 965 77 1000 1000 941 878 836 967 944 944 895 78 938 849 883 896 852 794 993 989 989 79 1000 950 960 973 960 963 966 880 840 80 978 904 989 848 937 983 975 911 819 148 37 38 39 40 41 42 43 44 45 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 0 38 896 0 39 882 933 0 40 917 956 990 0 41 784 983 836 832 0 42 883 747 800 871 863 0 43 871 946 856 944 876 941 0 44 900 960 917 900 989 960 983 0 45 678 956 761 900 978 942 933 983 0 46 741 1000 884 898 917 900 1000 875 965 47 896 950 878 938 933 - 1000 844 856 911 48 852 983 1000 902 878 1000 871 973 840 49 889 900 844 831 885 982 968 973 970 50 872 838 1000 874 889 848 920 1000 885 51 1000 963 933 1000 966 900 989 1000 1000 149 37 38 39 40 41 42 43 44 45 52 871 908 800 759 938 968 963 867 785 53 872 867 973 734 813 870 947 956 922 54 889 1000 902 878 917 883 989 883 898 55 883 956 700 836 773 796 919 828 1000 56 1000 973 900 906 987 989 984 1000 989 57 815 989 938 872 911 1000 911 952 898 58 748 917 928 947 826 933 822 1000 972 59 920 950 1000 811 835 904 893 942 963 60 950 894 1000 1000 963 900 917 719 874 61 711 1000 966 722 789 983 978 911 916 62 900 967 950 950 872 987 1000 966 1000 63 989 959 1000 861 795 862 1000 904 966 64 889 793 889 778 1000 1000 970 983 911 65 815 989 983 917 1000 915 851 989 989 66 794 1000 963 944 978 906 978 1000 978 67 720 987 767 911 885 989 1000 978 689 68 822 1000 920 889 898 900 941 966 800 69 964 867 889 1000 933. . 938 904 978 822 70 800 . 773' 862 827 805 • 952 1000 889 911 71 883 947 807 844 720 983 966 889 956 72 867 810 833 600 983 933 917 904 806 73 785 941 -., 862 800 .600 928 1000 880 938 74 983 797 963 770 872 956 969' ' 1000 826 75 1000 978 850 991 683 933 991 973 813 76 911 907 950 896 972 959 971 989 974 77 859 989 917 1000 861 989 966 989 867 78 874 930 894 972 947 874 903 1000 1000 79 1000 906 978 867 978 872 939 880 970 80 785 841 950 885 951 978 867 813 936 46 47 48 49 50 51 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 . 42 43 44 45 46 0 47 940 0 48 956 975 0 49 876 956 958 0 50 840 850 800 854 0 51 907 800 963 1000 859 46 47 48 49 50 51 52 53 54 52 969 973 941 841 985 849 0 53 933 985 933 826 755 796 891 0 54 978 991 922 836 920 989 747 937 0 55 926 939 906 794 985 911 973 978 949 56 1000 933 994 982 917 925 889 783 880 57 765 960 956 857 950 1000 926 885 906 58 978 939 994 1000 985 845 893 822 956 59 960 956 982 842 989 889 933 1000 733 60 828 922 1000 871 951 1000 840 956 871 61 933 922 922 726 755 815 871 817 680 62 1000 889 933 900 1000 898 928 933 900 63 850 1000 983 884 978 800 983 983 844 64 902 922 982 915 911 911 970 867 1000 65 698 818 970 876 898 851 773 989 938 66 1000 926 896 867 767 963 785 1000 811 67 972 822 844 852 1000 1000 707 917 917 68 938 959 960 800 911 1000 806 960 956 69 956 963 889 956 956 972 983 972 906 70 1000 918 922 822 800 840 662 889 952 71 956 744 911 933 966 844 955 1000 911 72 944 1000 907 614 1000 830 863 ~ 867 555 73 956 1000 911 785 917 667 933 850 866 74 902 939 944 674 973 819 870 911 872 75 951 965 933 883 716 839 989 628 1000 76 822 791 851 908 978 907 896 845 856 77 911 844 840 941 792 911 908 849 1000 78 987 849 970 918 972 963 937 834 933 79 970 973 939 902 853 972 1000 880 955 80 836 858 848 943 972 946 944 827 924 1 55 56 57 58 59 60 61 62 63 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 153 55 56 57 58 59 60 61 62 63 52 53 54 55 0 56 953 0 57 900 938 0 58 911 783 898 0 59 987 876 972 909 0 60 845 1000 983 889 952 0 61 951 900 939 978 850 870 0 62 1000 1000 1000 920 983 966 967 0 63 1000 1000 933 915 866 966 1000 975 0 64 969 978 772 922 951 1000 950 922 892 65 924 863 956 882 916 711 785 922 915 66 933 889 1000 783 862 966 1000 964 1000 67 983 811 853 1000 967 966 960 987 778 68 956 951 867 844 928 839 963 861 1000 69 878 972 861 898 867 966 947 900 922 70 933 889 967 816 916 800 978 738 852 71 955 1000 939 947" 844 978 983 884 1000 72 966 987 756 796 1000 689 907 640 978 73 844 983 911 756 950 828 978 756 963 74 878 962. . 900 . 847 884 874 1000 978 830 75 956 ' 922 • 1000 948 858 989 1000 ' 1000 1000 76 936 844 796 882' 787 964 .1000 ' . 763 867 77 900 989 956 870 933 966 963 889 916 78 956 982 915 896 956 989 978 816 896 79 822 818 983 889 950 933 1000 822 922 80 956 969 944 946 896 850 911 983 991 1 64 65 66 67 68 69 70 71 72 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 155 64 65 66 67 68 69 70 71 72 52 53 54 55 56 57 58 59 60 61 62 63 64 0 65 884 0 66 867 918 0 67 889 1000 879 0 68 1000 1000 867 994 0 69 960 978 944 852 886 0 70 933 933 960 807 973 854 0 71 978 896 884 904 983 835 959 0 72 806 941 898 978 956 634 674 921 0 73 689 963 839 883 1000 706 918 722 692 74 810 782 1000 807 952 900 939 729 852 75 1000 789 1000 991 972 872 941 333 983 76 940 944 939 947 966 844 785 739 839 77 1000 978 933 889 983 983 983 973 978 78 971 983 1000 1000 966 867 755 911 793 79 880 1000 889 878 978 1000 983 944 1000 80 787 941 911 844 883 813 987 811 872 1 73 74 75 76 77 78 79 80 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 157 73 74 75 76 77 78 79 80 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 0 74 899 0 75 906 - 994 0 76 889 920 973' 0 77 963 1000 963 875 0 78 911 987 886 987 943 0 79 920 907 885 983 956 936 80 940 1000 978 856 960 920 158 Proceeding pages display the averaged dissimilarity matrix for the full-set (gold standard) study described in Chapter 6. 159 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 1 0 745 896 957.28571 957.28571 905.57143 905.57143 905.57143 891.14286 877.28571 649 863.14286 863.14286 839.42857 914.85714 829.42857 896 668.14286 877.28571 877.28571 649 588.28571 872.71429 985.85714 909.85714 957.28571 919.71429 919.71429 909.85714 738.85714 829.85714 696.42857 863.14286 863.14286 696.85714 872.71429 744 896 553.85714 857.71429 682.28571 696.42857 734 1000 952.57143 886.57143 957.28571 795.57143 1000 971.42857 943.14286 0 591.57143 985.85714 843.28571 905.57143 905.57143 648.71429 738.85714 792.14286 773.28571 435.71429 497 985.85714 772.85714 886.57143 985.85714 806.28571 558.57143 611.28571 601.85714 834.85714 687.71429 848 852.71429 900.14286 610.71429 691.42857 938.42857 919.71429 744.71429 649.57143 553.85714 635.42857 985.85714 673.57143 901 985.85714 834.57143 421 549.71429 748.85714 1000 957.28571 724.85714 734.42857 985.85714 952.57143 1000 733.57143 985.85714 0 849 805.71429 743.42857 971.71429 943.14286 805.14286 725.14286 929 734.14286 672.85714 905.57143 768 820.42857 849 828.85714 363.57143 662.57143 607 943.42857 354.42857 678.14286 801.57143 849 805.14286 985.85714 715.85714 985.85714 820.28571 867.71429 734.14286 767.85714 985.85714 697 863.14286 763.28571 867.71429 496.42857 853.57143 825.28571 1000 828.85714 924 • '791 815.28571 952.57143 985.85714 591.85714 985.85714 0 554.42857 791 791 743.57143 853.57143 952.57143 1000 985.85714 938.42857 791.28571 877 650.14286 654.57143 1000 952.57143 938.71429 1000 914.85714 891.42857 792.42857 621.57143 654.57143 985.85714 757.71429 568.85714 882.14286 1000 952.57143 877.14286 985.85714 957.28571 863.14286 550 654.57143 1000 985.85714 938.42857 886.85714 971.42857 952.57143 1000 800.57143 800.57143 1000 985.85714 909.85714 957.28571 0 838.57143 881.85714 725.42857 957.28571 1000 819.28571 843.28571 843.28571 696.14286 696.14286 796.28571 473.71429 957.57143 914.28571 914.28571 857.42857 729.14286 577.42857 654 586.85714 365 938.42857 843.28571 706.42857 985.85714 1000 857.42857 985.85714 757.57143 957.28571 724.71429 553.57143 388 1000 843.28571 985.85714 929.28571 971.42857 1000 1000 753.14286 667.42857 1000 985.85714 957.28571 957.28571 0 449.57143 597.42857 597.14286 706.42857 919.71429 905.57143 834.85714 985.85714 985.85714 971.71429 971.71429 777.14286 834 834 919.71429 858.14286 695.71429 985.85714 971.71429 924.28571 583.28571 407.42857 459 734.42857 849 919.71429 863.14286 749.14286 985.85714 985.85714 938.42857 886 919.71429 985.85714 905.57143 919.71429 1000 857.42857 1000 924.28571 838.57143 833.85714 819.71429 705.71429 848.28571 0 530.57143 763 834 919.71429 905.57143 819.85714 985.85714 900.14286 971.71429 971.71429 877.28571 919.71429 919.71429 834 715.28571 924.28571 985.85714 971.71429 924.28571 549.42857 345.28571 510.85714 482.85714 834 834 692.28571 905.57143 985.85714 985.85714 938.42857 971.71429 919.71429 985.85714 905.57143 777.14286 1000 1000 1000 924.28571.. 924.28571 910.14286 957.28571 985.85714 900.14286 0 834.85714 773 891.14286 734.42857 616 985.85714 985.85714 943.14286 971.71429 848.71429 843.71429 891.14286 748.57143 905.57143 943.14286 957.28571 886 824.71429 682.28571 311.14286 653.42857 810.42857 820.42857 700.85714 743.85714 663.71429 985.85714 957.28571 985.85714 971.71429 891.14286 814.71429 743.85714 805.42857 1000 924 971.42857 971.71429 971.71429 857.42857 985.85714 710.14286 843.28571 1 2 3 4 5 6 7 8 52 938.42857 795.85714 781.14286 1000 1000 857.42857 929.28571 885.71429 53 985.85714 943.14286 957.28571 985.85714 985.85714 985.85714 985.85714 957.28571 54 1000 957.28571 900.71429 952.57143 1000 957.57143 1000 924 55 1000 957.28571 971.42857 1000 952.57143 839.42857 952.57143 900.71429 56 914.85714 1000 1000 971.42857 971.42857 1000 1000 1000 57 938.42857 924 924 1000 1000 1000 1000 971.42857 58 924 952.57143 795.57143 719.57143 719.57143 843.28571 985.85714 985.85714 59 971.42857 957.57143 843.28571 696.57143 744 843.28571 985.85714 938.42857 60 900.14286 971.42857 971.42857 952.57143 1000 957.57143 1000 924 61 985.85714 828.85714 971.42857 952.57143 857.42857 1000 914.28571 781.14286 62 985.85714 885.71429 971.42857 1000 1000 957.57143 1000 885.71429 63 1000 772.28571 885.71429 1000 866.85714 796.14286 809.71429 900.71429 64 862.14286 1000 1000 971.42857 971.42857 1000 1000 1000 65 914.28571 914.85714 971.42857 1000 1000 1000 1000 971.42857 66 985.85714 800.57143 957.28571 938.42857 985.85714 985.85714 915.14286 909.85714 67 971.42857 1000 985.85714 672.14286 719.57143 985.85714 985.85714 938.42857 68 971.42857 914.28571 843.28571 648.85714 648.85714 843.28571 985.85714 900.14286 69 1000 957.57143 985.85714 985.85714 985.85714 862.14286 896 985.85714 70 1000 871.85714 985.85714 985.85714 985.85714 862.14286 957.28571 900.14286 71 919.71429 919.71429 900.14286 985.85714 900.14286 653.71429 877 763 72 871.57143 1000 1000 971.42857 971.42857 862.42857 1000 857.42857 73 952.57143 909.85714 924 1000 1000 1000 1000 971.42857 74 1000 871.57143 971.42857 857.42857 1000 1000 1000 885.71429 75 1000 814.71429 971.42857 1000 857.42857 957.57143 1000 828.85714 76 1000 914.85714 971.42857 1000 1000 1000 914.28571 971.42857 77 829.14286 1000 1000 971.42857 971.42857 1000 1000 1000 78 904.85714 957.28571 885.71429 1000 914.28571 914.28571 1000 971.42857 79 900.71429 929.28571 985.85714 871.57143 957.28571 943.42857 985.85714 985.85714 80 985.85714 900.71429 791.28571 644.85714 606.57143 900.14286 843.28571 909.85714 81 938.42857 924 924 1000 1000 957.57143 938.71429 971.42857 82 1000 957.28571 971.42857 1000 1000 1000 857.42857 971.42857 83 1000 957.28571 828.85714 1000 1000 815 914.28571 971.42857 84 1000 957.28571 971.42857 1000 1000 886.85714 914.28571 900.71429 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 0 834.85714 815.71429 563.28571 625.71429 924 881.57143 882.14286 871.57143 772.42857 682.28571 520.85714 905.57143 891.14286 943.42857 857.42857 853.57143 957.28571 568 748.85714 814.71429 597.71429 692.28571 905.57143 677.57143 801.57143 838.28571 809.71429 867.71429 814.71429 730 771.85714 919.71429 905.57143 924 1000 771.71429 734.14286 772.28571 819.14286 757.57143 667.28571 828.85714 0 582.71429 596.85714 474.28571 1000 857.71429 971.42857 1000 625.14286 711.71429 682.85714 535.85714 919.71429 786.42857 896 1000 1000 863.14286 858.14286 1000 819.85714 141.71429 345.85714 549.42857 241.57143 1000 914.85714 985.85714 1000 664 914.85714 588.28571 521.42857 1000 767 914.85714 1000 1000 1000 1000 795.57143 914.28571 0 375 697 771.85714 611.57143 885.71429 819.28571 720.28571 707.28571 725.28571 725.28571 738.85714 644.57143 776.57143 819.28571 819.28571 905.57143 905.57143 1000 858.14286 582.71429 512 596.85714 549.42857 952.57143 625.42857 805.14286 819.28571 492.57143 720.42857 739.42857 582.71429 866.85714 886.57143 957.28571 957.57143 957.57143 857.42857 952.57143 985.85714 952.57143 0 359.42857 938.42857 778.14286 957.28571 900.14286 648.71429 540.57143 558.57143 596.85714 905.57143 811.14286 943.14286 985.85714 985.85714 738.85714 777.14286 985.85714 872.28571 596.85714 383.57143 401.85714 392.42857 852.71429 792 971.71429 985.85714 563.85714 468.42857 725.28571 596.85714 952.57143 886.57143 871.57143 762.71429 943.42857 914.28571 866.85714 819.28571 938.42857 0 985.85714 801.14286 957.28571 985.85714 639.28571 502.57143 416 449.85714 905.57143 872.42857 881.85714 985.85714 985.85714 738.85714 729.42857 985.85714 834 521.71429 402.14286 497 450.71429 985.85714 900.71429 971.71429 985.85714 493.14286 577.42857 616.57143 678.14286 1000 909.85714 957.28571 762.71429 943.42857 1000 1000 771.85714 900.14286 0 648.71429 905.57143 634.85714 952.57143 1000 1000 1000 563.57143 724.71429 724.71429 696.14286 553.57143 1000 1000 734.42857 952.57143 1000 1000 985.85714 938.42857 511.57143 677.28571 467.85714 634.85714 952.57143 938.42857 985.85714 1000 682.42857 1000 1000 957.28571 957.28571 943.42857 952.57143 971.42857 895.71429 0 905.57143 696.14286 938.42857 663.28571 943.42857 857.71429 776.57143 483.42857 615.42857 696.14286 696.14286 1000 1000 877 866.85714 857.71429 857.71429 929.28571 881.85714 839.14286 321.71429 682 696.14286 853.57143 654.71429 786.71429 943.42857 924 985.85714 985.85714 914.85714 914.85714 1000 952.57143 971.42857 824.14286 0 563.85714 900.71429 971.42857 862.71429 971.42857 872.71429 792.14286 644.57143 650.14286 753.85714 843.28571 985.85714 849 882.14286 971.42857 971.42857 896 957.28571 985.85714 763.85714 593.57143 563.85714 971.42857 871.57143 957.28571 881.57143 733.57143 790.71429 790.71429 901 971.71429 1000 985.85714 985.85714 985.85714 9 52 985.85714 53 871.85714 54 914.28571 55 701 56 909.85714 57 1000 58 943.14286 59 857.42857 60 1000 61 957.57143 62 809.71429 63 929.28571 64 924 65 952.57143 66 914.28571 67 914.85714 68 895.71429 69 757.57143 70 700.42857 71 763 72 828.85714 73 914.28571 74 938.42857 75 866.85714 76 1000 77 924 78 795.57143 79 909.85714 80 1000 81 771.71429 82 952.57143 83 952.57143 84 929.28571 10 772.28571 957.28571 909.85714 886.57143 985.85714 971.42857 843.28571 753.14286 924 838.28571 971.42857 844.14286 1000 914.85714 909.85714 952.57143 843.28571 957.57143 957.57143 919.71429 1000 957.28571 848 957'.28571 829.14286 1000 943.14286 1000 867.42857 971.42857 957.28571 729 800.85714 11 957.28571 914.85714 957.28571 957.28571 938.42857 971.42857 915.14286 985.85714 885.71429 929 924 957.28571 952.57143 824.14286 957.28571 957.57143 938.42857 952.57143 952.57143 872.28571 866.85714 957.28571 800.57143 909.85714 957.28571 866.85714 895.71429 952.57143 957.28571 971.42857 909.85714 909.85714 957.28571 12 971.42857 815 871.57143 871.57143 952.57143 971.42857 929.28571 914.28571 971.42857 786.42857 924 957.28571 952.57143 909.85714 857.42857 957.57143 952.57143 866.85714 952.57143 872.28571 952.57143 871.57143 814.71429 681.28571 957.28571 952.57143 909.85714 952.57143 943.14286 885.71429 909.85714 909.85714 957.28571 13 828.85714 900.71429 909.85714 886.57143 1000 971.42857 1000 952.57143 924 653 971.42857 886.57143 1000 957.28571 895.71429 910.14286 1000 1000 1000 919.71429 1000 814.71429 719.57143 672.14286 871.57143 1000 957.28571 1000 895.71429 828.85714 957.28571 729 658.28571 14 985.85714 800.85714 1000 1000 516.57143 985.85714 971.42857 971.42857 943.42857 943.42857 938.42857 1000 724.85714 809.71429 985.85714 971.42857 924 952.57143 952.57143 952.57143 867.42857 957.57143 910.14286 952.571'43 1000 516.57143 952.57143 695.71429 905.57143 943.42857 910.14286 952.57143 1000 15 1000 929.28571 843.28571 985.85714 924 1000 971.42857 971.42857 1000 871.85714 952.57143 985.85714 924 938.42857 971.71429 929 924 952.57143 952.57143 809.71429 924 985.85714 938.42857 938.42857 '757.57143 924 795.85714 924 891.42857 1000 938.42857 852.71429 757.57143 16 971.42857 909.85714 971.42857 971.42857 1000 828.85714 890.71429 985.85714 743.14286 971.42857 971.42857 971.42857 1000 705 814.71429 890.71429 890.71429 985.85714 985.85714 985.85714 771.71429 828.85714 790.71429 971.42857 924 914.28571 971.42857 915.14286 877 971.42857 790.71429 971.42857 924 1 2 3 4 5 6 7 8 17 18 19 20 21 22 23 24 9 10 11 12 13 14 15 16 17 0 18 914.28571 0 19 1000 862.85714 0 20 952.57143 625.14286 553.85714 0 21 1000 586.85714 616.57143 682.85714 0 22 672.85714 849 919.71429 919.71429 919.71429 0 23 710.57143 758.14286 563.14286 843.28571 786.42857 687 0 24 421.57143 914.85714 800.85714 767 896 762.71429 601 0 25 544.42857 1000 1000 938.71429 952.57143 734.14286 710.57143 682.28571 26 431.28571 1000 1000 1000 1000 686.71429 663.14286 611.57143 27 985.85714 905.57143 724.71429 682.28571 863.14286 872.28571 938.42857 1000 28 985.85714 905.57143 858.14286 905.57143 763 919.71429 985.85714 1000 29 749.71429 1000 914.28571 914.28571 952.57143 914.85714 805.71429 792.42857 30 843.28571 677.28571 905.57143 844.28571 639 919.71429 985.85714 1000 31 1000 767.71429 759.14286 540.28571 488.42857 919.71429 929 753.42857 32 1000 586.85714 641 682.85714 212.42857 919.71429 929 896 33 985.85714 781.85714 606.28571 497.28571 739.42857 763 914.85714 943.14286 34 985.85714 734.42857 687.57143 611.28571 493.14286 905.57143 829.14286 881.85714 35 810.28571 866.85714 1000 1000 1000 601.71429 985.85714 985.85714 36 682.28571 839.14286 819.71429 772.28571 914.85714 606.28571 426.57143 401.85714 37 558.57143 915.14286 985.85714 924.57143 985.85714 473.42857 606.57143 668.14286 38 180 1000 914.28571 724 1000 672.85714 624.85714 278.71429 39 914.28571 311.14286 716.71429 512 483.14286 919.71429 886.57143 896 40 985.85714 909.85714 496.42857 734.14286 629.71429 985.85714 615.85714 848 41 985.85714 601 583 626.28571 454.85714 905.57143 914.85714 881.85714 42 839.42857 586.85714 759.14286 677.85714 483.14286 734.71429 929 735.42857 43 828.85714 952.57143 1000 1000 1000 914.85714 1000 1000 44 1000 711 839.14286 957.28571 957.28571 914.28571 828.85714 914.85714 45 914.28571 871.57143 957.28571 772.28571 867.42857 1000 971.42857 814.71429 46 943.14286 929.28571 776.85714 819.2857.1 1000 839,14286 811.14286 985.85714 47 943.14286 1000 871.85714 771.71429 929.28571 909.85714 796.14286 843.28571 48 914.28571 914.28571 1000 1000 952.57143 943.42857 1000 1000 49 900.14286 866.85714 1000 1000 1000 1000 985.85714 1000 50 957.28571 843.28571 757.71429 805.14286 915.14286 971.42857 843.28571 1000 51 957.28571 952.57143 1000 1000 914.28571 943.14286 985.85714 985.85714 17 52 1000 53 852.71429 54 914.28571 55 914.28571 56 971.42857 57 1000 58 862.14286 59 800.85714 60 1000 61 1000 62 1000 63 1000 64 971.42857 65 1000 66 900.14286 67 719.57143 68 791.42857 69 900.14286 70 985.85714 71 985.85714 72 971.42857 73 914.28571 74 1000 75 914.28571 76 952.57143 77 971.42857 78 1000 79 957.28571 80 834.85714 81 914.28571 82 1000 83 1000 84 952.57143 18 629.71429 871.57143 871.57143 767.85714 938.42857 971.42857 843.28571 757.57143 971.42857 971.42857 881.57143 853.57143 952.57143 806.14286 767.85714 1000 795.57143 866.85714 952.57143 872.28571 952.57143 667.71429 596.71429 577.85714 853.57143 952.57143 895.71429 839.42857 957.28571 743.14286 848.57143 520.71429 711 19 957.28571 914.85714 767 957.28571 985.85714 971.42857 915.14286 938.42857 924 881.57143 971.42857 871.57143 1000 957.28571 909.85714 910.14286 985.85714 1000 1000 691.42857 1000 957.28571 943.14286 957.28571 814.71429 1000 857.42857 1000 824.14286 971.42857 957.28571 957.28571 814.71429 20 772.28571 909.85714 957.28571 814.71429 985.85714 971.42857 985.85714 943.42857 971.42857 971.42857 828.85714 829.14286 1000 914.85714 957.28571 1000 985.85714 957.57143 815 834 1000 814.71429 705.42857 814.71429 867.42857 1000 714.85714 1000 829.14286 686.28571 957.28571 814.71429 767.28571 21 867.42857 957.28571 886.57143 957.28571 985.85714 924 938.42857 943.42857 971.42857 743.14286 971.42857 914.85714 1000 914.85714 957.28571 1000 985.85714 957.57143 957.57143 919.71429 1000 909.85714 848 814.71429 829.14286 1000 943.14286 1000 914.85714 924 957.28571 871.57143 871.57143 22 985.85714 943.42857 1000 952.57143 819.71429 985.85714 971.42857 971.42857 943.42857 943.42857 985.85714 809.71429 914.85714 1000 985.85714 971.42857 971.42857 1000 1000 919.71429 914.85714 957.57143 957.57143 1000 1000 819.71429 1000 900.71429 843.28571 943.42857 815 1000 1000 23 828.85714 914.85714 971.42857 924 1000 971.42857 843.28571 843.28571 971.42857 929 971.42857 838.28571 1000 971.42857 957.28571 943.42857 843.28571 985.85714 985.85714 900.14286 1000 971.42857 971.42857 971.42857 971.42857 1000 885.71429 915.14286 791.28571 971.42857 971.42857 828.85714 971.42857 24 929 895.71429 957.28571 772.28571 914.28571 885.71429 914.28571 929.28571 971.42857 971.42857 786.42857 914.85714 1000 914.85714 900.71429 1000 929.28571 1000 857.42857 1000 1000 957.28571 957.28571 914.85714 867.42857 1000 814.71429 957.57143 792.14286 828.85714 871.57143 914.85714 867.42857 25 26 27 28 29 30 31 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 26 316.14286 0 27 900.14286 852.71429 0 28 900.14286 900.14286 435.71429 0 29 773 607.14286 644.42857 463.71429 0 30 882.14286 985.85714 625.14286 554.42857 748.14286 0 31 952.57143 1000 863.14286 905.57143 952.57143 819.85714 0 32 1000 1000 863.14286 715.28571 1000 639 393.28571 0 33 838.85714 900.14286 653.14286 673.42857 985.85714 787.71429 596.85714 549.42857 34 985.85714 985.85714 877.28571 777.14286 900.14286 872.28571 241.57143 350.57143 35 957.28571 814.71429 1000 1000 814.71429 952.57143 1000 1000 36 682.28571 682.28571 1000 1000 863.14286 952.57143 772.28571 914.85714 37 454.85714 368.28571 952.57143 1000 692 896.28571 985.85714 985.85714 38 544.42857 431.28571 985.85714 985.85714 664 985.85714 857.42857 1000 39 1000 1000 905.57143 905.57143 1000 677.28571 664 483.14286 40 985.85714 985.85714 819.28571 857.42857 985.85714 910.14286 914.85714 772.28571 41 900.14286 900.14286 791.57143 786.57143 985.85714 738.85714 635.71429 407.42857 42 871.85714 801.14286 819.85714 819.85714 886.85714 724.71429 521.42857 340.57143 43 971.42857 971.42857 1000 1000 971.42857 809.71429 1000 1000 44 1000 1000 1000 952.57143 1000 957.57143 957.28571 839.14286 45 952.57143 1000 957.57143 1000 952.57143 1000 724.57143 914.85714 46 943.14286 895.71429 757.71429 985.85714 943.14286 985.85714 1000 1000 47 943.14286 895.71429 938.42857 985.85714 857.42857 985.85714 857.42857 1000 48 952.57143 1000 900.71429 929 952.57143 971.42857 952.57143 1000 49 985.85714 985.85714 957.28571 914.85714 985.85714 909.85714 1000 1000 50 871.57143 871.57143 705.28571 796.14286 957.28571 971.71429 985.85714 938.42857 51 957.28571 957.28571 929.28571 957.57143 957.28571 866.85714 914.28571 914.28571 25 26 27 28 29 30 31 32 52 866.85714 914.28571 857.71429 829.42857 952.57143 915.14286 867.42857 914.85714 53 985.85714 843.28571 1000 1000 843.28571 1000 957.28571 957.28571 54 1000 1000 1000 910.14286 1000 1000 957.28571 909.85714 55 1000 952.57143 952.57143 957.57143 1000 1000 744 957.28571 56 971.42857 971.42857 985.85714 985.85714 971.42857 938.42857 985.85714 985.85714 57 952.57143 1000 857.42857 1000 952.57143 957.57143 924 971.42857 58 814.71429 862.14286 971.71429 971.71429 909.85714 929.28571 938.42857 915.14286 59 957.28571 886.57143 929.28571 924.28571 886.57143 971.71429 943.42857 896 60 1000 1000 786.71429 910.14286 1000 1000 971.42857 924 61 1000 1000 929.28571 809.71429 1000 914.28571 885.71429 695.42857 62 914.28571 914.28571 843.57143 871.85714 1000 952.57143 828.85714 971.42857 63 1000 952.57143 910.14286 1000 914.28571 1000 844.14286 914.85714 64 971.42857 971.42857 1000 1000 971.42857 952.57143 1000 1000 65 1000 857.42857 957.57143 1000 857.42857 952.57143 914.85714 914.85714 66 985.85714 985.85714 857.42857 881.85714 985.85714 929.28571 957.28571 909.85714 67 791.42857 862.14286 985.85714 938.42857 957.28571 843.28571 1000 952.57143 68 776.42857 705.71429 886 886 886.57143 924.28571 985.85714 985.85714 69 985.85714 985.85714 844.14286 957.28571 985.85714 909.85714 957.57143 957.57143 70 900.14286 900.14286 758.42857 871.57143 985.85714 909.85714 815 957.57143 71 985.85714 985.85714 806.28571 834.57143 900.14286 829.57143 919.71429 919.71429 72 971.42857 971.42857 786.71429 957.57143 971.42857 952.57143 1000 1000 73 952.57143 1000 786.71429 1000 952.57143 1000 909.85714 957.28571 74 914.28571 914.28571 829.42857 900.14286 1000 896 848 848 75 1000 1000 929.28571 815 1000 952.57143 957.28571 814.71429 76 1000 ' 1000 886.85714 1000 1000 914.28571 829.14286. ,829.14286 77 971.42857 971.42857 * 1000 1000 971.42857 952.57143 1000 1000 78 1000 1000 985.85714 985.85714 914.28571 896 800.57143 943.14286 79 957.28571 957.28571 985.85714 943.42857 957.28571 938.42857 1000 1000 80 905.57143 834.85714 957.57143 952.57143 749.14286 1000 914.85714 867.42857 81 952.57143 1000 929.28571 957.57143 952.57143 1000 781.14286 971.42857 82 1000 1000 929.28571 1000 1000 952.57143 957.28571 957.28571 83 1000 1000 929.28571 957.57143 1000 866.85714 871.57143 871.57143 84 1000 1000 1000 957.57143 1000 914.28571 800.85714 871.57143 33 34 35 36 37 38 39 40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 582. 35 985. 36 900, 37 910. 38 985, 39 739, 40 41 592, 42 368, 43 44 909, 45 957, 46 805, 47 985 48 957 49 957 50 643 51 943 0 71429 85714 71429 42857 85714 42857 720 14286 57143 1000 85714 28571 14286 85714 57143 57143 71429 42857 0 938.42857 853.28571 971.71429 900.14286 630.71429 710.42857 621.57143 535.57143 952.57143 957.28571 914.85714 985.85714 900.14286 1000 952.57143 1000 938.42857 0 938.42857 672.14286 896 866.85714 867.71429 985.85714 1000 825 1000 834 957.28571 957.28571 857.71429 866.85714 971.42857 715 0 597.42857 539.71429 867.42857 696.85714 900.71429 872.42857 952.57143 871.57143 814.71429 915.14286 843.28571 1000 952.57143 1000 938.42857 0 558.57143 985.85714 971.71429 971.71429 943.42857 971.42857 985.85714 985.85714 839.14286 909*85714 1000 1000 971.42857 957.28571 0 1000 985.85714 985.85714 839.42857 971.42857 1000 857.42857 943.14286 714.85714 1000 985.85714 957.28571 957.28571 0 867.42857 426.57143 483.14286 952:57143 ' 957.28571 871.57143 957.57143 957.57143 771.71429 866.85714 985.85714 952.57143 0 900.71429 914.85714 .786.57143 834.57143 957.28571 724.71429 905.57143 919.71429 872.28571 738.85714 858.14286 33 52 815 53 943.14286 54 867.42857 55 914.85714 56 1000 57 971.42857 58 1000 59 952.57143 60 881.57143 61 924 62 843.28571 63 814.71429 64 1000 65 957.28571 66 825 67 952.57143 68 914.28571 69 1000 70 914.28571 71 877.28571 72 957.57143 73 957.28571 74 776.42857 75 914.85714 76 957.28571 77 1000 78 957.28571 79 957.57143 80 752.85714 81 929 82 814.71429 83 914.85714 84 914.85714 34 929 943.14286 957.28571 886.57143 952.57143 971.42857 1000 957.57143 971.42857 828.85714 924 758.42857 952.57143 867.42857 943.14286 1000 952.57143 910.14286 910.14286 786.57143 952.57143 957.28571 814.71429 767 914.85714 952.57143 824.14286 952.57143 815 971.42857 909.85714 909.85714 886.57143 35 985.85714 715.14286 914.28571 914.28571 549.14286 724.71429 710.28571 624.57143 943.42857 943.42857 938.42857 1000 606.28571 729.42857 639 891.14286 662.85714 866.85714 952.57143 952.57143 867.42857 871.85714 910.14286 866.85714 1000 468.71429 691.42857 662.85714 805.14286 857.71429 910.14286 952.57143 1000 36 971.42857 943.14286 957.28571 814.71429 952.57143 971.42857 1000 1000 971.42857 971.42857 781.14286 957.28571 952.57143 909.85714 943.14286 1000 952.57143 952.57143 809.71429 952.57143 952.57143 957.28571 909.85714 909.85714 957.28571 952.57143 624.42857 881.85714 862.85714 828.85714 909.85714 909.85714 957.28571 37 1000 829.14286 985.85714 938.42857 971.42857 1000 876.28571 971.42857 1000 1000 1000 938.42857 971.42857 843.28571 971.71429 876.28571 876.28571 1000 1000 1000 971.42857 985.85714 985.85714 985.85714 985.85714 971.42857 985.85714 815 891.42857 1000 985.85714 985.85714 985.85714 38 1000 938.42857 1000 857.42857 971.42857 1000 862.14286 886.57143 1000 1000 857.42857 914.28571 971.42857 1000 985.85714 862.14286 791.42857 985.85714 843.28571 900.14286 971.42857 1000 1000 1000 952.57143 971.42857 771.71429 957.28571 749.14286 857.42857 1000 1000 952.57143 39 814.71429 829.14286 871.57143 871.57143 938.42857 971.42857 985.85714 900.14286 971.42857 929 924 957.28571 952.57143 909.85714 871.57143 957.57143 938.42857 866.85714 952.57143 872.28571 952.57143 729 658 681.57143 957.28571 952.57143 895.71429 952.57143 957.28571 743.14286 909.85714 767.28571 814.71429 40 891.14286 862.85714 877 877 872.28571 929 957.57143 1000 805.42857 748.57143 843.71429 877 952.57143 824.14286 943.14286 1000 952.57143 872.28571 872.28571 952.57143 786.57143 877 787.14286 686.71429 877 866.85714 867.42857 952.57143 943.14286 891.14286 829.57143 829.57143 877 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 0 411.57143 1000 909.85714 772.28571 985.85714 985.85714 1000 1000 866.85714 985.85714 0 1000 957.28571 957.28571 1000 1000 1000 1000 900.14286 1000 0 724.71429 805.14286 891.14286 891.14286 849 858.14286 891.14286 829.57143 0 667.28571 919.71429 919.71429 905.57143 905.57143 729.42857 919.71429 0 1000 753.71429 852.71429 838.85714 938.71429 1000 0 535.28571 919.71429 905.57143 696.14286 834.57143 0 919.71429 844.28571 745 877 0 472.85714 639.57143 554.71429 52 843 53 943 54 55 957 56 57 971 58 59 910 60 61 62 .885 63 772 64 65 914 66 753 67 952 68 914 69 957 70 871 71 777 72 73 957 74 776 75 957 76 772 77 78 957 79 80 853 81 971 82 957 83 957 84 814 41 .28571 ,14286 767 ,28571 1000 .42857 1000 .14286 924 924 .71429 .28571 1000 .85714 ,14286 .57143 ,28571 .57143 .85714 ,14286 1000 .28571 .42857 .28571 ,28571 1000 ,28571 1000 ,28571 .42857 ,28571 .28571 .71429 42 871.57143 909.85714 957.28571 957.28571 985.85714 971.42857 985.85714 915.14286 971.42857 971.42857 885.71429 814.71429 1000 957.28571 957.28571 1000 829.42857 1000 914.28571 919.71429 1000 957.28571 762.28571 957.28571 909.85714 1000 943.14286 1000 744 971.42857 814.71429 957.28571 909.85714 43 905.57143 877.28571 905.57143 905.57143 602.14286 971.71429 971.42857 971.42857 763.28571 849 844 905.57143 544.85714 672 1000 814.71429 924 858.14286 858.14286 938.42857 687.28571 863.14286 649 858.14286 905.57143 521.28571 857.42857 767.28571 1000 849 634.85714 858.14286 905.57143 44 706.14286 877 815.42857 759.14286 919.71429 677.14286 744.28571 809.71429 829.57143 591.85714 834.57143 664 985.85714 421 711 938.42857 857.42857 905.57143 905.57143 985.85714 905.57143 801.57143 592.42857 759.14286 759.14286 985.85714 914.85714 943.42857 909.85714 877 383 616.57143 664 45 881.57143 871.57143 796.14286 714.85714 1000 734.42857 872.28571 687.85714 957.28571 862.14286 814.71429 663 905.57143 544.42857 553.57143 905.57143 816 857.71429 739.57143 924.57143 985.85714 810 776.57143 857.42857 900.71429 905.57143 734.42857 905.57143 811.14286 681.28571 667.28571 943.14286 848 46 919.71429 863.14286 919.71429 872.28571 891.14286 1000 711 719.57143 919.71429 877.28571 919.71429 872.28571 885.71429 1000 985.85714 582.85714 814.71429 905.57143 905.57143 985.85714 891.14286 919.71429 919.71429 919.71429 919.71429 971.42857 1000 730.14286 748.14286 919.71429 919.71429 919.71429 919.71429 47 919.71429 682.28571 606.85714 548.57143 891.14286 1000 814.71429 615.85714 738.85714 696.42857 596.28571 605.71429 971.42857 1000 985.85714 591.57143 711 905.57143 701.71429 658.14286 710.28571 919.71429 919.71429 738.85714 919.71429 971.42857 771.71429 862.14286 558.71429 777.14286 919.71429 919.71429 738.85714 48 677.28571 791.57143 777.42857 777.42857 863.14286 924.28571 952.57143 914.28571 735.85714 778.28571 778.28571 905.57143 929.28571 985.85714 914.28571 985.85714 1000 383.28571 530.28571 606.42857 498.14286 : 478.42857 806.57143 706.71429 654 929.28571 1000 943.42857 1000 360.57143 792.42857 611.57143 863.14286 4 9 50 51 52 53 54 55 56 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 - • 49 0 50 664.28571 0 51 734.71429 611 0 1 7 2 49 50 52 738.85714 677.28571 53 763.28571 919.71429 54 673.71429 655.42857 55 777.42857 877.28571 56 872.28571 877 57 943.42857 957.57143 58 985.85714 800.57143 59 838.85714 691.57143 60 863.14286 829.85714 61 905.57143 872.28571 62 815.71429 791.57143 63 905.57143 919.71429 64 938.42857 971.42857 65 938.42857 1000 66 914.28571 952.57143 67 971.71429 909.85714 68 877.14286 653.57143 69 125.85714 768 70 293.14286 621 71 654.42857 602 72 720.57143 611 73 639 919.71429 74 872.28571 819.85714 75 730 877.28571 76 724.71429 919.71429 77 795.57143 971.42857 78 910.14286 943.42857 79 881.85714 914.85714 80 938.71429 891.28571 81 596.57143 877.28571 82 858.14286 919.71429 83 634.85714 734.71429 84 863.14286 877.28571 51 905.57143 844.28571 877.28571 877.28571 587.42857 805.14286 748.28571 790.71429 731.14286 687.85714 745 919.71429 525 952.57143 805.14286 929 743.28571 706.42857 706.42857 601.71429 417.42857 849 801.57143 759.14286 763.28571 729.14286 771.85714 558 805.14286 792.42857 801.57143 673.42857 791.57143 52 0 796 796 753.57143 891.42857 848.57143 795.57143 800.85714 781.85714 877 592.42857 806.28571 985.85714 886.57143 858.28571 1000 757.57143 696.42857 610.71429 1000 905.57143 425.14286 648.71429 611 530.28571 985.85714 896 957.57143 929 411 891.14286 287.57143 706.14286 53 0 372.71429 372.71429 816 971.42857 1000 914.28571 368.85714 564.14286 472.57143 696.14286 896.28571 814.71429 857.42857 776.85714 1000 834 919.71429 819.28571 635.14286 653.71429 834.57143 515.28571 591.85714 957.57143 957.28571 1000 943.14286 667.85714 834.57143 781.85714 648.71429 54 0 316.14286 919.71429 914.85714 1000 805.57143 368.57143 648.71429 416 682 985.85714 943.14286 824.14286 757.71429 938.71429 777.42857 801.85714 516.42857 682.28571 682 877 458.71429 482.57143 985.85714 914.85714 943.42857 848.57143 653.71429 862.85714 725.28571 497 55 0 919.71429 957.28571 1000 914.28571 416 696.14286 231 460.14286 985.85714 839.42857 767.85714 805.14286 1000 819.85714 763 762.71429 682.28571 620.71429 815.71429 355 521.42857 985.85714 814.71429 901 957.28571 511.14286 801.57143 621.57143 465.14286 56 0 719.42857 690.85714 776.57143 801.85714 801.85714 858.14286 919.71429 482.85714 952.57143 819.28571 971.42857 729.14286 872.28571 872.28571 952.57143 725.85714 877.28571 815.71429 872.28571 919.71429 335.85714 757.71429 600.71429 819.28571 863.14286 744.14286 872.28571 919.71429 57 58 59 60 61 62 63 64 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 174 57 58 59 60 61 62 63 64 52 53 54 55 56 57 0 58 563.28571 0 59 738.85714 396.85714 0 60 800.57143 1000 952.57143 0 61 705.42857 1000 952.57143 460.14286 0 62 881.85714 1000 1000 331.14286 611.28571 0 63 862.14286 1000 957.57143 696.14286 601 653.71429 0 64 710.57143 710.28571 710.28571 868 868 924.28571 985.85714 0 65 568.57143 919.71429 877.28571 871.57143 719.57143 867.42857 701.85714 858.14286 66 472.57143 738.85714 605.71429 781.14286 828.85714 929 615.85714 738.85714 67 905.57143 535.57143 687 757.71429 715.28571 805.14286 805.14286 791.28571 68 738.85714 301.71429 222.42857 1000 1000 866.85714 1000 662.85714 69 943.42857 985.85714 857.71429 834.85714 834.85714 787.42857 863.14286 938.42857 70 943.42857 985.85714 882.14286 834.85714 834.85714 558.85714 863.14286 938.42857 71 943.42857 985.85714 924.57143 692 734.42857 644.57143 719.42857 938.42857 72 829.14286 971.42857 .971.42857 265.42857 536.14286 550 724.71429 792 73 767 952.57143 914.28571 526.14286 763.85714 711.14286 801.57143 '• - 901 74 929 943.42857 985.85714 778 778 687.28571 815.71429 910.14286 75 957.28571 1000 914.28571 487.85714 482.85714 398 578.28571 938.42857 76 957.28571 1000 957.57143 568.57143 720.57143 526.14286 716.71429 985.85714 77 710.57143 710.28571 710:28571 843.57143 929.28571 924.28571' 985.85714 449.57143 78 493.42857 682.28571 724.71429 971.42857 971.42857 719.85714: 871.57143 •596.28571 79 724.71429 634.85714 601 943.42857 985.85714 853.57143 943.42857 363.28571 80 790.71429 676.71429 317.28571 924 924 971.42857 686.57143 819.28571 81 895.71429 952.57143 914.28571 612.14286 749.71429 512 877 929.28571 82 633.85714 914.28571 1000 763.85714 526.14286 758.85714 563.85714 853.57143 83 957.28571 857.42857 857.42857 668.71429 720.57143 578.85714 759.14286 938.42857 84 862.14286 1000 1000 653.71429 515.28571 611.28571 412.42857 985.85714 65 66 67 68 69 70 71 72 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 176 65 66 67 68 69 70 71 72 52 53 54 55 56 57 58 59 60 61 62 63 64 65 0 66 678.14286 0 67 905.57143 872.28571 0 68 872.28571 696.42857 639.28571 0 69 896 914.28571 971.71429 938.42857 0 70 896 1000 971.71429 791.42857 241.28571 0 71 938.42857 1000 791 877.14286 687.42857 626.14286 0 72 852171429 857.42857 776.57143 924 692.28571 692.28571 406.85714 0 73 881.85714 667.71429 985.85714 1000 568.28571 654 915.14286 649.85714 74 667.85714 896 1000 852.71429 801.57143 715.85714 881.85714 759.14286 75 792 767.85714 805.14286 952.57143 701.71429 787.42857 644.57143 564.14286 76 797 853.57143 985.85714 1000 611.57143 611.57143 772.57143 834.85714 77 772.42857 738.85714 877 662.85714 795.57143 938.42857 938.42857 767.57143 78 758.85714 696.14286 919.71429 677.28571 910.14286 767.28571 824.42857 952.57143 79 815.71429 696.42857 801.57143 648.71429 924.28571 924.28571 881.85714 867.42857 80 914.85714 715 810 502.28571 957.57143 896.28571 853 1000 81 957.28571 885.71429 985.85714 1000 507 511.42857 872.71429 735.85714 82 416 800.85714 985.85714 952.57143 787.42857 787.42857 867.71429 745 83 792 853.57143 985.85714 809.71429 606.57143 606.57143 825.28571 745 84 744.28571 758.42857 805.14286 1000 905.57143 905.57143 620.14286 682.28571 73 74 75 76 77 78 79 80 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 178 73 74 75 76 77 78 79 80 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 0 74 560 0 75 407.42857 555 0 76 454.85714 745 593.28571 0 77 943.42857 910.14286 938.42857 985.85714 0 . , '' 78 957.28571 853.28571 909.85714 957.28571 691.42857 o 79 985.85714 952.57143 853.57143 943.42857 506.14286 691.42857 0 80 957.28571 957.28571 957.28571 914.85714 819.28571 690.85714 724.14286 0 81 212.14286 635.42857 440.42857 530.28571 929.28571 828.85714 943.42857 971.42857 82 646 474.28571 683.42857 730.85714 896 909.85714 938.42857 814.71429 83 312.28571 555 360.85714 326.71429 938.42857 909.85714 853.57143 957.28571 84 659 673.14286 393.28571 483.14286 985.85714 957.28571 901 957.28571 81 82 83 84 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 180 81 82 83 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 0 82 763.85714 0 83 345.28571 683.42857 0 84 692 706.42857 488.42857 Proceeding pages display the averaged dissimilarity matrix for part one of the subset study described in Section 7.2. 182 1 1 2 3 4 5 6 7 8 1 2 u 687.3333 0 3 906.7778 494 0 4 829.7778 788.6667 886.0833 0 5 967 934 967 769 0 6 967 967 901 967 895.5 0 7 967 934 934 950.5 813 587.5 0 8 810.6667 755.6667 893.1667 827.1667 802 703 868 0 9 884.5 956 908.3333 703 868 835 785.5 835 10 965.44446 929.3333 870.1111 936.1111 967 835 967 656.6667 11 703 791 967 868 983.5 835 884.5 692 12 928.7778 687.3333 873.7778 917.7778 1000 868 1000 783.1667 13 983.5 967 879 945 1000 967 950.5 835 14 466.5 791 886.3333 952.3333 967 967 983.5 1000 15 945 802 809.3333 948.6667 967 857 934 1000 16 549 857 835 802 769 1000 1000 895.5 17 909.6667 876.6667 761.1667 772.1667 835 769 934 725 18 909.6667 794.6667 798.3333 928.7778 967 934 945 689.6667 19 819.3333 848.1111 876.6667 909.6667 868 967 983.5 805.1667 20 884.7778 761.1667 905.3333 822.8333 934 868 884.5 744.6667 21 950.5 967 923 956 857 868 631.5 780 22 578.3333 840.5 912 854.25 967 967 967 862.5 23 857 675.5 829.5 912 901 967 967 827.6667 24 827.1667 591.44446 518.1111 793.1111 730.5 983.5 897.3333 593 25 654.3333 848.1111 871.1667 491.66666 901 637 967 711.6667 26 906.5 824 787.3333 673.6667 307 967 851.5 934 27 967 857 912 945 967 703 538 780 28 967 956 879 952.3333 901 406 406 912 29 967 934 967 934 747 631.5 703 505 30 717.1667 761.1667 876.6667 601.6667 967 868 917.5 794.1667 31 965.44446 896.3333 826.1111 954.44446 934 868 934 882.1667 32 761.1667 574.1667 865.6667 794.1667 967 967 868 761.6667 33 1000 1000 930.3333 919.3333 967 835 884.5 582 34 687.3333 826.1111 827.1667 761.1667 934 835 835 755.6667 35 624.44446 731.3333 965.44446 866.44446 967 824 967 865.6667 36 945 824 849.6667 901 967 857 983.5 967 37 860.6667 527 912 527 769 967 934 796.5 38 852.3333 881.1111 739.1667 689.6667 604 967 983.5 747 39 1000 967 912 1000 967 967 802 983.5 40 835 967 919.3333 937.6667 1000 868 868 967 41 967 967 802 934 950.5 950.5 983.5 934 42 1000 901 901 967 1000 934 670 956 43 670 978 980.75 980.75 967 868 983.5 950.5 44 983.5 703 945 1000 1000 967 978 879 45 930.3333 983.5 835 1000 1000 967 978 923 46 912 961.5 972.5 759.1667 868 934 917.5 950.5 47 967 989 901 934 802 835 785.5 771.3333 48 948.1667 915.1667 931.6667 948.1667 967 703 703 877.44446 49 967 857 1000 967 950.5 736 802 725 50 1000 978 917.5 967 967 934 818.5 769 51 879 950.5 950.5 851.5 967 835 967 967 1 2 3 4 5 6 7 8 53 835 897.3333 983.5 917.5 967 1000 1000 813 54 928.5 956 945 956 1000 901 989 884.5 55 989 1000 969.75 945 1000 967 983.5 873.5 56 952.3333 945 947.75 947.75 1000 868 851.5 884.5 57 362 983.5 1000 934 983.5 934 974.3333 1000 58 1000 902.55554 804.3333 1000 967 967 829.5 901 59 852.3333 925.1111 887.6667 772.1667 1000 934 818.5 893.1667 60 890 952.3333 879 912 868 934 945 798.8333 61 945 707.6667 829.2222 926.6667 967 835 967 917.5 62 928.5 967 970.6667 948.6667 1000 967 978 1000 63 945 961.5 945 961.5 967 1000 1000 875.3333 64 882.6667 663.6667 876.8889 963.3333 1000 967 1000 725 65 904.1667 948.1667 948.1667 964.6667 1000 868 967 976.44446 66 983.5 703 934 989 967 1000 961.5 824 67 1000 862.5 835 983.5 967 895.5 967 857 68 941.3333 923 952.3333 715.55554 967 791 868 879 69 852.3333 870.1111 805.1667 854.6667 736 901 835 730.5 70 967 891.55554 787.8333 983.5 967 670 851.5 785.5 71 929.3333 917.7778 810.6667 926.1667 934 736 653.5 766.6667 72 934 983.5 983.5 950.5 917.5 703 714 736 73 736 868 1000 1000 1000 967 703 1000 74 1000 967 945 945 1000 1000 835 1000 75 758 967 . 1000 -901 ••• 1000. ' '934 1000 .". 1000 76 890 765.8333 815.3333 961.5 934 934 950.5 862.5 77 1000 1000 945 945 967 802 967 857 78 764.3333 932.44446 865.6667 948.1667 1000 1000 901 964.6667 79 882.1667 ,• 730.5 •- 758 937.1667' . 1000 967 „• ' ' ' .967 .888.44446 80 983.5 967 978 "' 824 934 967 967 .846 81 912 952.3333 765.3333 930.3333 934 967 983.5 799.8889 82 901 956 934 1000 1000 934 785.5 967 83 917.5 923 974.3333 952.3333 1000 967 1000 835 84 967 985.3333 879 862.5 1000 934 950.5 934 1 2 3 4 9 10 11 12 13 14 15 16 5 6 7 8 9 0 10 917.5 0 11 791 967 0 12 934 706.6667 868 0 13 956 593 934 505 0 14 941.3333 934 802 1000 934 0 15 978 956 934 1000 934 842.3333 0 16 802 879 857 967 967 802 813 0 17 967 898.6667 802 882.1667 868 868 846 945 18 851.5 530.1667 681 695.1667 780 983.5 967 824 19 1000 670 626 296 923 868 868 857 20 857 717.6667 901 475.66666 813 1000 923 1000 21 923 824 604 708.5 743.3333 989 912 967 22 820.3333 1000 659 952.3333 978 607.6667 897.3333 648 23 835 862.5 857 796.5 1000 945 1000 851.5 24 961.5 931.6667 862.5 893.1667 950.5 961.5 950.5 659 25 824 670 802 560 1000 956 901 604 26 919.3333 983.5 967 967 989 945 915.6667 890 27 923 945 890 967 1000 1000 912 917.5 28 882.6667 912 901 956 1000 974.3333 893.6667 1000 29 868 967 851.5 1000 1000 967 967 934 30 472 931.6667 780 931.6667 934 934 967 824 31 884.5 541.6667 1000 703 719.5 1000 840.5 1000 32 879 909.6667 461 794.1667 934 934 879 912 33 725 626 868 538 772.6667 956 978 890 34 1000 929.3333 593 786.3333 637 967 956 824 35 884.5 976.44446 1000 943.44446 983.5 411.5 807.5 835 36 974.3333 972.5 967 835 923 915.6667 417 945 37 934 970.6667 " 1000 919.3333 1000 945 917.5 912 38 956 929.3333 967 786.3333 967 956 736 670 39 835 1000 736 736 802 967 912 983.5 40 868 917.5 538 967 952.3333 945 853.3333 780 41 868 461 785.5 340 670 967 934 934 42 967 703 736 582 395 868 934 983.5 43 838.6667 978 1000 978 1000 732.3333 989 835 44 989 901 934 741.5 820:3333 967 934 1000 45 989 825.55554 967 840.2222 919.3333 919.3333 • 978- 1000 46 681 941.3333 967 945 923 941.3333 967 802 47 967 967 934 1000 857 1000 1000 967 48 945 .816.1667 .. 967 931.6667. 1000 1000 967. 1000 49 934 967 - 862.5 934 967 868 . 1000 857 50 967 703 1000 967 1000 1000 956 1000 51 706.6667 978 967 934 978 945 985.3333 835 9 10 11 12 13 14 15 16 53 1000 934 593 1000 967 802 923 857 54 917.5 950.5 752.5 917.5 983.5 879 961.5 1000 55 934 708.2222 835 631.2222 794.6667 923 945 945 56 952.3333 923 835 923 875.3333 956 967 1000 57 983.5 1000 917.5 967 950.5 466.5 1000 967 58 769 835 934 1000 967 1000 505 934 59 956 863.3333 967 852.3333 670 956 967 736 60 1000 967 1000 923 967 1000 967 1000 61 950.5 893.6667 703 923 846 967 972.5 945 62 919.3333 906.5 769 967 879 875.3333 959.6667 945 63 923 754.8333 703 732.8333 945 1000 967 983.5 64 967 791 637 794.6667 791 912 950.5 1000 65 967 796.5 868 835 967 912 1000 983.5 66 945 956 967 813 893.6667 956 934 1000 67 1000 1000 879 868 983.5 1000 857 890 68 846 967 967 941.3333 934 983.5 879 802 69 791 830.3333 703 786:3333 1000 956 901 1000 70 967 868 802 1000 1000 1000 956 1000 71 923 698.3333 967 929.3333 1000 956 1000 967 72 884.5 967 917.5 824 967 967 961.5 1000 73 912 1000 868 1000 967 967 1000 934 74 967 945 736 1000 967 967 945 972.5 75 967 868 703 835 868 857 967 1000 76 967 884.5 802 681 835 758 1000 1000 77 1000 857 505 747 747 1000 912 956 78 967 929.3333 1000 929.3333 1000 538 956 967 79 1000 810.6667 890 788.6667 648 901 912 901 80 846 1000 1000 956 864.3333 945 978 835 81 917.5 895.5 967 945 752.5 950.5 983.5 967 82 901 934 736 1000 967 1000 923 1000 83 835 983.5 769 967 934 941.3333 978 835 84 824 791 835 835 901 956 1000 1000 186 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0 18 926.1667 0 19 893.1667 591.44446 0 20 893.1667 848.1111 626 0 21 879 582 758 864.3333 0 22 961.5 952.3333 846 914.75 1000 0 23 952.3333 807.5 824 862.5 736 637 0 24 780 877.8333 892.1111 807.7778 895.5 882.6667 791 0 25 794.1667 928.7778 798.3333 631.5 1000 708.5 895.5 848.1111 26 505 983.5 901 912 1000 908.3333 879 747 27 917.5 725 857 670 967 967 912 857 28 824 928.5 1000 912 761.6667 952.3333 967 961.5 29 703 967 967 934 901 967 901 884.5 30 948.1667 884.7778 909.6667 865.6667 868 818.5 912 807.7778 31 904.1667 612.6667 505 593 752.5 959.6667 917.5 898.6667 32 802 706.1667 673.1667 761.1667 703 818.5 758 576.5 33 857 846 956 395 857 989 835 983.5 34 860.1667 793.1111 804.1111 744.6667 571 730.5 840.5 771.1111 35 964.6667 948.1667 929.3333 899.44446 983.5 765.3333 939.5 865.6667 36 912 983.5 967 923 890 937.6667 868 813 37 862.5 945 890 875.3333 983.5 805.6667 895.5 846 38 461 906.7778 884.7778 810.6667 967 873.5 879 596.6667 39 851.5 703 505 835 901 912 934 967 40 879 664.5 329 787.3333 688.3333 879 890 967 41 901 549 538 307 967 967 934 917.5 42 846 703 835 824 516 1000 879 901 43 983.5 956 1000 983.5 945 765.0833 920.6667 974.3333 44 967 868 967 930.3333 926.6667 989 890 983.5 45 917.5 890 837.3333 849.9167 915.6667 974.0833 909.6667 703 46 1000 923 950.5 950.5 978 858.5833 876.6667 963.3333 47 705.3333 963.3333 974.3333 895.5 824 948.1667 931.6667 825.55554 48 884.7778 915.1667 948.1667 865.6667 967 956 967 948.1667 49 967 912 890 1000 835 967 824 912 50 879 890 989 967 1000 967 967 967 51 1000 989 851.5 909.25 978 903.75 934 1000 17 18 19 20 21 22 23 24 53 835 842.3333 798.3333 917.5 967 818.5 928.5 875.3333 54 818.5 851.5 879 923 917.5 989 950.5 991.75 55 983.5 842.3333 870.3333 772.9167 937.6667 983.5 1000 967 56 917.5 904.6667 840.5 945 791 964.25 945 875.3333 57 967 1000 917.5 1000 983.5 703 1000 1000 58 983.5 748.55554 989 1000 967 964.6667 948.1667 967 59 909.6667 873.7778 928.7778 860.1667 835 939.5 879 914.1111 60 716.3333 952.3333 941.3333 961.5 967 926.1667 909.6667 821.8889 61 917.5 639.3333 1000 912 835 956 967 967 62 1000 802 802 967 857 989 1000 983.5 63 871.6667 796.5 699.8333 721.8333 967 895.5 952.3333 928.5 64 752.5 518.3333 923 868 813 963.3333 923 928.5 65 888.44446 915.1667 769 835 1000 967 926.6667 964.6667 66 945 917.5 967 948.6667 930.3333 978 967 967 67 967 886.3333 912 934 967 1000 758 677.3333 68 945 928.5 890 974.3333 967 921.44446 909.6667 945 69 681 862.7778 895.7778 876.6667 1000 857 912 739.6667 70 703 847.55554 945 983.5 901 967 983.5 989 71 948.1667 844.44446 976.44446 882.1667 868 961.5 967 862.7778 72 692 923 983.5 912 934 950.5 967 978 73 967 835 1000 1000 1000 912 1000 901 74 983.5 571 736 1000 1000 967 1000 1000 75 967 670 736 901 571 1000 1000 1000 76 873.5 744,8889 730.5 796.5 901 961.5 895.5 941.3333 77 884.5 703 736 879 879 1000 983.5 967 78 931.6667 921.44446 965.44446 948.1667 1000 835 917.5 943.44446 79 932.44446 763.5 843.6667 854.6667 . 879 1000 875.33.33 893.1667 80 945 895.5 • .-: 758 974.3333 945 978 .1000 , 967 81 799.8889 871.6667 937.6667 941.3333 961.5 939.7778 837.1111 766.8889 82 901 846 901 1000 967 1000 1000 945 83 1000 620.5 505 967 824 901 835 961.5 84 912 846 879 851.5 1000 961.5 1000 974.3333 25 26 27 28 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 26 626 0 27 967 967 0 28 923 879 692 0 29 736 593 769 637 30 783.1667 934 890 967 31 670 923 692 697.5 32 810.6667 912 945 912 33 868 963.3333 868 912 34 752.7778 1000 791 802 35 764.3333 939.5 967 967 36 956 879 912 879 37 725 769 912 950.5 38 760.1111 494 967 956 39 967 912 895.5 835 40 1000 948.6667 615 827.6667 41 934 934 901 824 42 1000 1000 884.5 967 43 978 945 769 893.6667 44 1000 956 934 978 45 853.8333 989 835 978 46 730.5 875.3333 967 919.3333 47 956 703 868 868 48 931.6667 967 659 769 49 934 967 593 769 50 978 868 901 ' ' • 703 51 934 967 879 901 29 30 31 32 0 967 0 934 931.6667 0 967 645.6667 827.1667 0 967 967 686.5 967 835 662.1667 896.3333 607.1667 967 717.1667 954.44446 849.1667 967 1000 791 780 934 785.5 967 846 901 931.6667 896.3333 747 868 1000 516 648 901 835 923 835 901 884.5 428 703 1000 868 967 730.5 967 884.5 886.3333 917.5 1000 967 967 923 1000 967 869.55554 879 868 884.5 948.6667 917.5 670 950.5 967 837.3333 868 865.6667 854.6667 936.1111 851.5 895.5 1000. . 857 '835 967 835 : •: 983.5 967 705.3333 970.6667 906.5 189 25 26 27 28 29 30 31 32 53 934 1000 857 1000 967 780 967 714 54 901 983.5 967 983.5 917.5 978 983.5 818.5 55 804.3333 945 912 875.3333 1000 950.5 766.8889 1000 56 961.5 945 769 780 901 950.5 897.3333 961.5 57 1000 983.5 967 934 983.5 967 1000 1000 58 989 1000 835 1000 967 934 868 967 59 804.1111 956 802 956 1000 915.1667 731.3333 909.6667 60 952.3333 1000 901 1000 967 983.5 802 831.8333 61 1000 967 560 835 967 884.5 901 967 62 901 1000 780 941.3333 1000 967 912 802 63 815.3333 967 818.5 692 967 835 782.3333 875.3333 64 923 901 967 873.5 1000 950.5 912 923 65 818.5 780 956 901 1000 849.1667 796.5 921.44446 66 1000 945 967 959.6667 967 1000 967 967 67 983.5 983.5 791 983.5 967 945 967 890 68 626 967 1000 983.5 967 851.5 970.6667 934 69 870.1111 560 934 824 967 882.1667 797.3333 697.5 70 923 868 868 868 703 983.5 901 983.5 71 928.7778 824 703 659 835 849.1667 896.3333 948.1667 72 967 851.5 802 703 670 950.5 1000 1000 73 901 1000 873.5 1000 1000 703 945 890 74 1000 901 829.5 1000 1000 1000 1000 851.5 75 1000 1000 967 1000 1000 934 901 736 76 945 1000 934 1000 983.5 967 917.5 780 77 802 901 956 967 802 934 967 752.5 78 954.44446 967 802 967 1000 948.1667 929.3333 964.6667 79 964.6667 1000 868 967 1000 876.6667 931.6667 906.7778 80 1000 978 1000 959.6667 934 604 983.5 967 81 941.3333 983.5 967 1000 934 934 1000 851.2222 82 1000 1000 967 1000 1000 835 802 868 83 956 985.3333 813 985.3333 1000 967 972.5 912 84 930.3333 923 967 923 1000 967 791 917.5 190 33 34 35 36 37 38 39 40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 0 34 1000 0 35 1000 731.3333 0 36 1000 1000 791 0 37 1000 692 842.3333 884.5 0 38 1000 906.7778 896.3333 626 857 0 39 835 604 967 780 1000 857 0 40 875.3333 1000 983.5 919.3333 956 967 912 0 41 340 571 1000 967 967 934 604 967 42 791 868 1000 1000 1000 890 780 890 43 989 983.5 783.6667 930.3333 967 978 923 923 44 930.3333 1000 917.5 835 967 967 835 919.3333 45 886.3333 1000 945 934 978 802 1000 897.3333 46 945 983.5 923 930.3333 882.6667 912 945 897.3333 47 923 912 1000 1000 1000 759.55554 1000 956 48 868 948.1667 948.1667 967 967 876.6667 917.5 835 49 967 857 835 868 934 934 1000 868 50 967 887.8889 967 1000 896.3333 853.3333 857 1000 51 945 978 879 934 910.44446 1000 912 776.3333 33 34 35 36 37 38 39 40 53 901 761.6667 835 967 764.3333 978 736 769 54 917.5 897.3333 934 917.5 948.1667 1000 736 835 55 739.6667 978 967 893.6667 958.1111 1000 802 948.6667 56 952.3333 813 956 945 906.7778 945 923 897.3333 57 1000 934 538 934 934 1000 1000 1000 58 1000 989 1000 505 1000 978 1000 967 59 1000 873.7778 929.3333 923 923 925.1111 967 967 60 1000 974.3333 1000 967 824 777.8889 967 967 61 884.5 967 945 923 914.1111 967 967 758 62 923 1000 917.5 904.6667 956 967 692 783.6667 63 813 763.5 983.5 967 909.6667 890 774.5 835 64 824 923 967 901 950.7778 758 967 813 65 1000 915.1667 904.1667 967 1000 964.6667 917.5 1000 66 970.6667 1000 901 868 950.5 967 802 967 67 950.5 923 824 862.5 934 917.5 967 967 68 983.5 923 897.3333 912 842.3333 923 912 983.5 69 967 928.7778 896.3333 956 857 684.6667 967 1000 70 967 985.3333 967 1000 929.3333 989 1000 1000 71 967 921.44446 929.3333 857 1000 939.7778 1000 1000 72 895.5 978 967 1000 967 846 857 917.5 73 1000 1000 868 967 1000 1000 939.5 1000 74 1000 736 967 912 945 1000 835 945 75 868 736 967 934 967 1000 736 571 76 736 846 934 868 909.6667 890 626 967 77 879 703 1000 780 945 967 769 912 78 967 961.7778 296 901 1000 976.44446 1000 1000 79 879 909.6667 964.6667 615 972.5 882.1667 967 912 80 985.3333 901 851.5 923 758 967 934 974.3333 81 868 875.3333 983.5 983.5 945 733.8889 983.5 923 82 1000 941.3333 901 967 929.3333 989 670 868 83 901 901 917.5 908.3333 983.5 923 879 534.3333 84 703 923 1000 956 929.3333 948.6667 725 1000 192 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 o'-42 868 0 43 1000 967 0 44 1000 582 934 0 45 967 747 888.8333 659 0 46 1000 923 872.3333 860.6667 844.8333 0 47 857 967 948.1667 1000 761.1667 893.1667 0 48 835 912 738.3333 934 917.5 956 868 0 49 983.5 1000 571 692 835 659 571 472 50 1000 857 945 769 917.5 950.5 934 758 51 1000 1000 727.75 890 917.5 788.25 983.5 653.5 193 41 42 43 44 45 46 47 48 53 703 868 666.8333 835 901 983.5 945 765.8333 54 868 868 730.2222 851.5 886.3333 890 978 798.8333 55 593 714 931.25 728.6667 822.4167 925.75 967 840.5 56 934 967 836.1667 798.3333 826.75 914.75 961.5 694.3333 57 1000 1000 516 912 840.5 917.5 950.5 967 58 967 802 931.6667 923 788.6667 843.6667 884.7778 868 59 1000 769 683.3333 967 967 780 868 752.5 60 890 967 865.6667 791 728.1667 865.6667 754.3333 983.5 61 901 1000 868 950.5 923 879 967 598.5 62 1000 1000 809.3333 908.3333 871.6667 879 1000 912 63 538 835 851.5 879 743.8333 967 967 758 64 769 802 . 886.3333 785.5 864.3333 952.3333 967 917.5 65 736 934 879 1000 853.8333 950.5 967 844.44446 66 967 747 945 424.33334 802 853.3333 835 967 67 934 802 917.5 851.5 472 901 983.5 967 68 1000 1000 943.44446 967 877.44446 ' 5^30.6667 863.3333 972.5 69 890'-. 835 978 1000 917.5 829.5 601.8889 865.6667 70 1000 ' 835 895.5 868 934 983.5 868 747 71 802 1000 776.8333 1000 983.5 945 846 483 72 983.5 857 835 983.5 934 901 868 758 73 1000 851.5 813 1000 1000 945 967 774.5 74 736 851.5 912 901 1000 1000 901 769 75 818.5 868 824 901 791 934 1000 967 76 769 725 829.5 835 813 901 950.5 813 77 593 774.5 967 615 879 1000 824 967 78 1000 835 884.5 1000 983.5 983.5 978 948.1667 79 1000 873.5 950.5 714 923 983.5 934 921.44446 80 967 1000 945 897.3333 945 615 857 1000 81 967 884.5 976.44446 670 624.44446 884.7778 721.3333 989 82 703 571 650.3333 967 967 983.5 978 716.3333 83 1000 1000 813 923 901 857 901 681 84 868 560 831.8333 670 802 895.5 923 666.8333 194 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 0 769 538 0 739.1667 195 50 51 52 53 54 55 56 57 53 725 870.1111 722.6667 0 54 901 782.1111 738.1111 618.6667 0 55 868 656.6667 784.3333 755.6667 826.1111 0 56 626 882.1667 836.5833 719.5 824 767.8333 0 57 851.5 983.5 868 769 750.6667 950.5 868 0 58 868 956 983.5 923 941.3333 917.5 950.5 824 59 901 978' 901 847.55554 891.55554 884.5 782.3333 934 60 934 912 934 868 978 917.5 763.5 912 61 802 522.3333 697.7778 654.3333 711.6667 818.7778 859.1111 835 62 835 1000 871.6667 868 483 860.6667 857 774.5 63 934 827.1667 887.6667 717.1667 733.6667 538 612.6667 967 64 967 632.3333 921.44446 863.3333 827.1667 672.1111 767.44446 857 65 1000 1000 884.5 950.5 934 815.3333 956 802 66 725 1000 934 967 818.5 849.6667 758 912 67 895.5 879 961.5 890 875.3333 928.5 851.5 934 68 901 967 915.6667 967 950.5 967 952.3333 934 69 967 890 983.5 1000 989 851.5 879 1000 70 769 683.1111 706.1667 683.1111 646.44446 656.6667 799.6667 763.5 71 505 644.3333 846 843.8889 634.8889 912 837.3333 983.5 72 752.5 576.5 846 879 853.3333 928.5 868 934 73 1000 1000 483 1000 857 890 912 824 74 868 967 637 604 560 736 483 824 75 950.5 967 934 703 582 868 802 879 76 818.5 777.6667 816.1667 816.1667 741.7778 502.66666 810.6667 862.5 77 967 901 1000 703 472 527 703 1000 78 967 908.3333 862.5 941.3333 816.6667 961.5 1000 516 79 758 868 934 846 901 780 950.5 934 80 692 1000 846 967 934 956 820.3333 785.5 81 901 901 945 923 901 945 904.6667 934 82 1000 914.1111 887.6667 662.6667 472 871.1667 785.5 796.5 83 835 1000 692 769 571 919.3333 915.6667 818.5 84 835 686.7778 750.1667 791 703 601.6667 670 868 196 58 59 60 61 62 63 64 65 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 .34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 197 66 67 68 69 70 71 72 73 1 ,2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 199 66 67 68 69 53 54 55 56 57 58 59 60 61 62 63 64 65 66 0 67 884.5 0 68 945 659 0 69 703 884.5 923 0 70 1000 961.5 1000 901 71 802 917.5 967 906.7778 72 950.5 842.3333 934 901 73 967 1000 912 868 74 703 967 967 1000 75 934 950.5 934 1000 76 868 901 912 912 77 736 736 890 857 78 1000 813 967 965.44446 79 835 824 928.5 948.1667 80 787.3333 950.5 516 868 81 983.5 752.5 794.1667 821.8889 82 769 879 967 945 83 923 884.5 923 956 84 1000 835 934 919.3333 70 71 72 73 0 820.3333 0 763.5 686.5 0 824 1000 1000 0 758 967 967 565.5 967 967 917.5 967 785.5 868 895.5 967 802 868 1000 967 860.6667 866.44446 879 659 804.3333 915.1667 769 884.5 835 1000 983.5 934 934 934 967 983.5 723.44446 689.8889 813 758 1000 791 835 714 723.44446 733.8889 912 967 200 74 75 76 77 78 79 80 81 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 201 74 75 76 77 78 79 80 81 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 0 75 670 0 76 769 648 0 77 692 736 736 0 78 659 934 1000 1000 0 79 890 967 688.8333 774.5 849.1667 0 80 934 967 868 890 1000 890 0 81 967 527 862.5 917.5 967 945 813 0 82 758 868 849.1667 835 871.6667 950.5 967 989 83 769 637 835 967 868 967 890 835 84 868 703 563.1667 736 923 967 736 758 82 83 84 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 203 82 83 84 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 0 571 769 0 857 204 Proceeding pages display the averaged dissimilarity matrix for part one of the subset study described in Section 7.2. 205 1 2 3 4 1 • 0 2 792.6 0 3 905.3333 523.8571 0 4 814.5833 818.8571 847.26666 0 5 950.5 960.4 884.5 ' 725 6 960.4 983.5 854.8 894.4 7 960.4 943.4286 938.125 950.5 8 833.25 804.375 847.1429 790.5714 9 868 961.5 917.5 732.7 10 920.2308 937.8 869.5833 943.8333 11 736 829.5 884.5 934 12 928.7778 733.2 873.7778 917.7778 13 975.25 960.4 881.2 950.5 14 457.85715 874.6 887.8 947.2 15 946.375 752.5 828.4 953.8 16 659 912 846 653.5 17 906.7778 880.1429 775.5 804.375 18 895.125 832 781 941.7273 19 860.1667 869.5833 870.375 932.25 20 905.3333 767 889.06665 829.6667 21 943.4286 967 923 956 22 610.6 806.7143 911.1539 857.8461 23 868 645.25 839.7143 910.4286 24 858 584.7273 480.5 . 813.8 25 667.6667 826.3077 903.375 457.875 26 917.5 763.5 808.6 706.3 27 967 912 947.2 967 28 901 961.5 891.1 907.6 29 960.4 917.5 894.4 874.6 30 763.125 808.5 880.1429 644.4286 31 960.8461 918 853.0833 957.5833 32 796.125 620.8571 800 738.7143 33 985.8571 1000 894.4 884.5 34 753 841.53845 781.1429 795.2857 35 638.7273 838.8 959 869.9 36 934 815.2 838 871 37 870.75 478.6 844 451 38 893.1667 877.8333 705.375 721.875 39 1000 967 802 945 40 839.125 967 907.6 943.9 41 967 980.2 785.5 950.5 42 983.5 901 730.5 912 43 652 981.1429 983.5 976.4286 44 987.625 736 940.6 1000 45 947.75 957.5714 861.4 1000 46 884.5 967 978 736.93335 47 980.2 984.7692 915.1429 943.4286 48 961.125 936.375 941.4286 955.5714 49 983.5 914.2 1000 983.5 50 1000 954.3077 929.2857 971.7143 51 901 957.5714 957.5714 872.7143 5 6 7 8 0 887.8 0 864.3333 571 0 861.4 615 670 0 917.5 785.5 802 884.5 917.5 821.8 960.4 717.75 991 838 901 846 1000 851.5 802 713.625 983.5 901 925.75 821.8 950.5 943.4286 978 983.5 967 912 943.4286 980.2 820.8571 924.5714 983.5 868 582 841.6 950.5 711.25 978 835 909.25 730.125 943.4286 983.5 987.625 841.5 950.5 835 901 719.8571 894.4 769 649.375 824 960.4 950.5 967 864.3333 920.8 884.5 829.5 850.2308 769 985.8571 916 645.25 917.5 802 975.25 734.25 318 917.5 872.7143 782.2 967 631.5 543.5 842.3333 884.5 367.5 349.42856 729.4 755.8 580 670 417 962.875 824 875.3333 807.2 901 802 940.6 886.875 861.4 920.8 861.4 788.25 950.5 655.8571 791 587.5 940.6 835 858.5714 767.25 960.4 879 967 909.5 943.4286 863.2857 980.2 912 642.5 912 957.5714 825.1 509.7143 967 987.625 698.875 985.8571 910.4286 884.5 934 934 868 886.8571 960.4 955 910.9 919.3333 901 971.7143 811.4286 802 901 960.4 901 989 962.875 1000 980.2 983.5 927.4 1000 980.2 983.5 934 873.5 960.4 938.125 957.5714 637 901 759.5714 779 980.2 835 - 351.5 915.1539 946 759.1 827.6667 835 894.4 835 844.4286 826.75 881.2 881.2 937.6667 975.25 206 1 2 3 4 5 6 7 8 53 868 914.75 975.25 938.125 985.8571 967 1000 859,75 54 946.375 928 940.6 960.4 1000 915.1429 991 913.375 55 991.75 1000 969.2 956 1000 940.6 987.625 891.5714 56 964.25 952.8571 951.6 958.2 1000 881.2 888.625 901 57 664.5 987.625 1000 945 930.7 950.5 982.2308 1000 58 983.5 926.9167 853.25 950.5 901 983.5 872.125 925.75 59 792.6 940.53845 903.7143 804.7143 1000 967 844.4286 919.875 60 928.5 939.5 909.25 888.625 863.2857 967 958.75 849.125 61 950.5 804.8 836.4 934 983.5 835 983.5 938.125 62 938.7143 983.5 970.6667 948.6667 1000 967 983.5 1000 63 963.3333 967 884.5 909.25 901 1000 1000 906.5 64 918.7692 798.2 899.4167 956 1000 980.2 1000 793.75 65 928.125 948.75 955.5714 969.7143 1000 868 983.5 983.6923 66 985.8571 656.8 940.6 990.1 983.5 985.8571 974.3333 912 67 901 825.5714 863.875 987.625 978 930.3333 946.6923 862.5 68 925.75 934 961 758.2727 960.4 874.6 868 875.3333 69 911.4 902.46155 833 875.4286 722.8 934 858.5714 797.875 70 983.5 918.6667 840.875 987.625 943.4286 681 843.25 839.125 71 964.6667 938.3333 858 944.625 929.2857 714 694.75 825 72 960.4 987.625 987.625 962.875 912 683.2 760.75 868 73 851.5 934 1000 980.2 983.5 967 807.5 1000 74 1000 967 956 972.5 1000 915.1429 868 1000 75 815.2 934 894.4 874.6 1000 925 1000 945 76 926.6667 799.2857 849.125 971.125 956 950.5 967 908.3333 77 835 934 923 890 985.8571 901 917.5 843.25 78 858.6 953.2308 884.8571 955.5714 1000 983.5 915.1429 973.5 79 911.625 769 792.5714 946.1429 1000 980.2 980.2 916.3333 80 987.625 983.5 980.2 821.8 967 983.5 971.7143 907.6 81 877.9 951.7692 771.75 881.75 818.5 872.125 980.2 805.61536 82 841.6 961.9231 915.1429 1000 1000 917.5 759.5714 962.875 83 929.2857 961.5 974.3333 952.3333 1000 868 975.25 884.5 84 980.2 982.2308 896.2857 882.1429 1000 917.5 957.5714 950.5 207 1 9 10 11 12 L 3 4 5 6 7 8 9 0 10 925.75 0 11 783.1429 807.5 0 12 945 706.6667 868 0 13 931.46155 513.25 741.5 591.625 14 931.25 929.2857 802 1000 15 942.25 954.625 967 1000 16 763.5 923 884.5 980.2 17 967 910.44446 • 901 899 18 813 540.375 574.3 734.25 19 985.8571 593 561.5714 557.8 20 871.3 667.25 807.5 475.66666 21 870.53845 721.8571 554.5 772.6667 22 835 990.1 730.5 961 23 851.5 872.125 879 760.75 24 963.3333 924 884.5 866 25 858.5714 818.5 915.1429 736 26 923 938.125 983.5 975.25 27 868 906.5 882.1429 980.2 28 895.5 847.375 950.5 905.125 29 835 901 883 901 30 472 924 798.3333 862.125 31 901 517.6923 857 703 32 808.6 870.375 617.2 795.2857 33 694.75 594.5714 844.4286 611.3333 34 901 799.2 648 871.8 35 901 971.7273 813 953.7273 36 936.53845 967 962.875 857 37 950.5 961.5 1000 907 38 952.8571 948.1667 943.4286 871.8 39 901 857 769 716.2 40 868 826.75 653.5 975.25 41 868 571 766 465.4 42 884.5 692 707.125 729.4 43 844 982 967 980.2 44 992.38464 913.375 950.5 806.125 45 990.1 819.6667 967 840.2222 46 712.9 914.75 983.5 945 47 967 980.2 967 920.8 48 972.5 800.25 967 948.75 49 934 983.5 925 960.4 50 983.5 821.8 967 960.4 51 751 982 967 940.6 13 14 15 16 0 937 0 917.5 862 0 960.4 719.5 808.6 0 917.5 901 887.8 934 665.875 987.625 975.25 912 802 861.4 787.8571 928.5 745.9 990.1 901 846 683.75 983.5 917.5 960.4 980.2 631.5 907.6 769 980.2 956 861.4 863.875 938.125 967 943.4286 775.6 967 953.8 917.5 646.4286 975.25 946 926.38464 914.2 967 967 907.6 940.6 958.75 970 911.1539 980.2 920.8 957.5714 967 896.2857 881.2 934 960.4 874.6 608.125 985.8571 868 901 940.6 874.6 907.6 910.4286 805 954.3077 973 857 498.4 960.4 945 879 987.625 387.66666 855.625 782.2 934 880.6923 376 773.7143 967 952.3333 826.75 725 957.5714 953.8 830.2857 758 709.6 983.5 887.8 961 947.75 901 845.1539 848.2 626 950.5 901 943.4286 445.6 851.5 940.6 943 1000 772 989 901 868 973 950.5 1000 917.5 927.4 980.2 1000 930.7 947.2 970.3 807.5 835 980.2 983.5 950.5 1000 983.5 980.2 1000 967 917.5 983.5 882.1429 1000 901 978 934 980.2 910 985.3333 851.5 208 9 10 11 12 13 14 15 16 53 1000 950.5 797.2857 894.4 971.7143 881.2 967 846 54 945 950.5 802 886.8571 975.25 839.7143 967 901 55 940.6 772.9167 884.5 631.2222 805.3 930.7 950.5 972.5 56 957.1 934 901 923 877.9 960.4 970.3 868 57 989 1000 946 980.2 962.875 600.3333 975.25 971.7143 58 901 917.5 971.7143 1000 943.4286 1000 698.2857 950:5 59 906.5 918 983.5 911.4 802 894.4 950.5 664.5 60 1000 901 985.8571 934 985.8571 1000 985.8571 939.5 61 930.7 894.4 773.7143 934 886.3333 967 979.375 972.5 62 944.1539 919.8571 868 978 909.25 865.25 969.75 887.8 63 961.5 825.55554 818.5 771 956 1000 980.2 987.625 64 975.25 847.6923 802 794.6667 830.875 924.5714 962.875 884.5 65 917.5 847.375 868 876.25 940.6 956 960.4 975.25 66 958.75 962.2857 943.4286 831.3333 895 931.46155 928 934 67 1000 980.2 858.1 782.2 975.25 945 877.4286 846 68 868 975.25 967 907.6 950.5 946.375 896.2857 868 69 879 898.2 851.5 871.8 980.2 953.8 934 967 70 971.7143 884.5 915.1429 1000 1000 1000 981.1429 1000 71 952.8571 717.1667 985.8571 957.6 1000 973.6 1000 983.5 72 814.375 920.8 901 894.4 971.7143 967 971.125 1000 73 858.5714 1000 915.1429 1000 934 901 • 1000 868 74 934 956 814.375 980.2 960.4 901 967 904 75 983.5 881.2 802 796.5 920.8 825.5714 983.5 853.8571 76 983.5 912 857 726.5714 901 854.8 1000 1000 77 1000 879 686.5 689.8 828.4 983.5 947.2 976 78 983.5 957.6 983.5 957.6 1000 557.8 862.5 983.5 79 1000 858 848.2 818.8571 749.2 940.6 907.6 915.1429 80 848.75 1000 1000 967 881.75 955 969.53845 716.2 81 955 907.6 956 952.3333 851.5 960.4 990.1 901 82 884.5 920.8 769 960.4 980.2 1000 961.5 1000 83 855.3077 985.8571 802 956 950.5 947.75 983.5 901 84 912 874.6 884.5 795.4 940.6 874.6 1000 884.5 209 17 18 19 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 0 18 944.625 0 19 917.7778 597.3333 0 20 907.5 797.7273 575.125 0 21 907.6 463.75 763.5 864.3333 22 896.875 934 825.5714 913.6923 23 906.5 793.75 855.625 868 24 752.5 888.2143 893.7273 817.1 25 820.875 946.5833 852.7692 723.625 26 326.8 987.625 943.4286 920.8 27 934 755.8 862.5 802 28 874.6 896.875 943.4286 920.8 29 802 879 983.5 874.6 30 943.44446 851.7273 919.875 870.7143 31 914.1111 577.5 543.5 573.75 32 .. 835 677.4286 606.375 781.1429 33 ' 894.4 760.75 973.6 363.1 34 823.5714 794.7273 828.3333 781.1429 35 965.44446 961.125 957.6 899.6 36 840.5 989 901 898 37 838.3 963.3333 895.5 859 38 365.66666 888.8333 918.8571 858 39 888.625 576.5 543.5 670 40 907.6 711.25 340 808.6 41 934 505 637 444.5 42 872.125 510.5 725 697.5 43 975.25 964 1000 985.8571 44 983.5 888.625 971.7143 927.4 45 938.125 901 865.625 860.13336 46 987.625 937 962.875 927.4 47 747.4286 970 980.75 910.4286 48 913.5833 936.375 961.125 814.1429 49 983.5 941.3333 952.8571 1000 50 896.2857 892 983.5 971.7143 51 938.125 991 872.7143 922.2143 21 22 23 24 0 973 0 769 648 0 909.25 854.8 707.125 0 967 651.1429 847.375 836.5833 983.5 917.5 907.6 726.5714 835 960.4 930.3333 912 763.5 957.1 920.8 952.8571 818.5 967 901 901 818.5 824 825.1 833.7273 660.5714 920.8 913.375 899.25 643.6 814.375 785.5 622.8571 802 983.5 818.5 985.8571 554.5 769 855.625 767.7273 989 749.75 953.8 899.25 909.25 887.25 769 810.25 978 813 808.6 798.3333 950.5 891.5714 896.875 634 920.8 947.2 950.5 960.4 700.25 841.6 914.2 957.5714 861.4 960.4 736 915.1429 505 980.2 896.875 920.8 950.5 768 940.5 979 945 990.1 934 975.25 915.6667 950.6923 922.5714 722.8 978 869.46155 894.2857 967 846 955.5714 936.375 848.2727 983.5 959.6667 977.1539 961.125 881.2 914.2 894.4 924.5714 1000 957.5714 975.25 973 980.2 910.4286 950.5 1000 210 17 18 19 20 21 22 23 24 53 890 832.25 856.2143 863.875 983.5 844.4286 946.375 889 54 863.875 865.6429 892 920.8 938.125 990.1 950.5 960.4 55 987.625 844 890.375 811.73334 937.6667 984.7692 1000 960.4 56 938.125 895 868 916.4 791 941.61536 952.8571 877.9 57 967 1000 945 1000 987.625 754.3333 1000 1000 58 978 811.4167 992.9286 1000 934 969.7143 961.125 973 59 922.5714 869.7273 946.5833 880.1429 917.5 948.1429 896.875 920.7273 60 777.8889 964.25 955.2143 946.375 983.5 936.7143 932.25 854.2727 61 938.125 704.55554 895.5 910.9 846 955 975.25 962.875 62 1000 851.5 829.5 967 901 982 1000 987.625 63 911.1539 835 788.8889 779 980.2 921.625 964.25 934 64 835 626.375 945 851.5 839.7143 934 942.25 934 65 916.3333 886.875 802 858.5714 934 941.3333 941.61536 961.125 66 967 938.125 920.8 953.8 947.75 947.75 983.5 929.2857 67 983.5 906.5 921.625 938.125 975.25 1000 862.5 691 68 952.3333 946.375 934 979 975.25 935.7273 928.7778 952.3333 69 726.5714 887.7273 921.8333 894.2857 983.5 877.4286 921.625 778 70 802 869.1667 964.6429 987.625 950.5 971.7143 987.625 991 71 965.44446 866.8333 984.8571 911.625 934 967 975.25 887.7273 72 815.2 901 987.625 934 913.375 919.3333 934 980.2 73 967 901 1000 1000 1000 861.4 1000 950.5 74 987.625 686.5 763.5 901 940.6 960.4 962.875 980.2 75 980.2 780 785.5 874.6 785.5 983.5 1000 957.5714 76 924.1 782.2727 809.3333 835 940.6 ' 929.875 930.3333 943 77 913.375 785.5 802 923 927.4 914.2 987.625 901 78 941.4286 935.7273 949.3333 955.5714 1000 844.4286 938.125 953.7273 79 949.3333 797.2857 858 875.4286 927.4 1000 906.5 908.4286 80 967 921.625 868 976.9 958.75 980.2 1000 971.7143 81 785.3077 888.3077 938.7143 920.25 976.9 946.5833 872 792.1667 82 915.1429 811 865.25 1000 917.5 1000 962.875 955 83 1000 641.125 681 967 832.46155 910 884.5 971.125 84 924.5714 856 901 872.7143 1000 952.8571 1000 979 211 25 26 27 28 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 26 730.5 0 27 957.5714 940.6 0 28 846 878.1539 531.4 0 29 802 609.5 752.5 554.5 30 787.875 940.6 846 920.8 31 818.5 892.75 780 661.75 32 823.5714 927.4 909.25 828.4 33 901 961 851.5 856 34 806 917.5 747 868 35 858.6 954.625 967 975.25 36 928.5 892 948.1429 892 37 582 752.5 948.1429 950.5 38 811.0769 500.2857 950.5 952.8571 39 985.8571 947.2 907.6 861.4 40 967 926.38464 670 819.7692 41 950.5 950.5 934 829.5 42 985.8571 980.2 851.5 920.8 43 938.125 945 868 893.6667 44 1000 967 967 958.75 45 890.375 990.1 881.2 980.2 46 698.875 887.8 960.4 877.9 47 954.3077 730.5 917.5 917.5 48 948.75 980.2 750.6667 861.4 49 901 934 582 796.5 50 961.9231 813 934 818.5 51 950.5 967 846 901 29 30 31 32 0 901 0 841.6 924 0 920.8 719.7778 808.5 0 740.7143 818.5 717.1429 920.8 785.5 684.75 819 649.1429 967 810.5 953.7273 888.44446 910.4286 983.5 830.875 848.2 895.5 851.5 958.75 820.3333 934 936.375 931.6667 797.875 811.4286 1000 582 698.2857 884.5 881.2 855.625 663.4 881.2 674.125 521.5 762.4 783.1429 821.8 708.5 754.8571 980.2 901 877 929.2857 1000 980.2 962.875 953.8 1000 971.7143 866.4167 896.2857 920.8 901 961.5 929.2857 747 962.875 980.2 860.5714 934 909.5 891 952.0833 901 921.625 862.5 914.2 763.5 962.875 861.4 985.8571 980.2 766.625 976 919.8571 212 25 26 27 28 29 30 31 32 53 954.3077 1000 928.5 929.2857 950.5 835 851.5 785.5 54 917.5 985.8571 983.5 985.8571 929.2857 982 975.25 844.4286 55 853.25 950.5 947.2 868 960.4 957.5714 800.4167 1000 56 934 950.5 841.6 782.2 901 957.5714 829.5 967 57 989 987.625 950.5 950.5 990.1 967 945 1000 58 984.7692 1000 917.5 1000 983.5 950.5 934 975.25 59 818.6923 978 901 978 1000 936.375 838.8 922.5714 60 959.38464 971.7143 950.5 957.5714 983.5 987.625 901 873.875 61 1000 975.25 697.5 876.25 917.5 851.5 901 971.7143 62 950.5 1000 873.5 956 1000 967 924.5714 881.2 63 861.5 980.2 879 815.2 980.2 890 843.8889 865.25 64 928.5 925.75 967 905.125 1000 962.875 868 942.25 65 851.5 868 915.6667 940.6 934 869.9 806.125 941.0833 66 1000 955 967 940 985.8571 983.5 924.5714 980.2 67 975.25 985.8571 895.5 985.8571 978 963.3333 960.4 934 68 604 971.7143 1000 985.8571 980.2 901 936.75 909.25 69 894.8461 730.5 950.5 895.5 884.5 911.625 878.4 740.7143 70 946.6923 901 802 769 769 987.625 868 987.625 71 950.6923 882.1429 719.5 679.4286 835 886.875 865.6667 961.125 72 975.25 851.5 861.4 715.375 752.5 956 835 901 73 943.4286 1000 898 1000 1000 785.5 917.5 917.5 74 1000 940.6 887.8 960.4 971.7143 980.2 868 872.7143 75 950.5 1000 967 1000 970 945 841.6 841.6 76 958.75 1000 950.5 1000 987.625 975.25 842.3333 853.3333 77 844.4286 940.6 963.7 881.2 915.1429 940.6 912 787.8571 78 968.46155 983.5 884.5 983.5 1000 948.75 957.6 969.7143 79 969.7143 1000 901 980.2 1000 917.7778 948.75 930.0833 80 785.5 984.7692 1000 972.0769 967 762.4 987.625 980.2 81 948.1429 910.9 982 1000 921.625 950.5 990.1 880.1667 82 1000 1000 983.5 1000 983.5 876.25 881.2 886.8571 83 978 989 824 972.5 901 868 976.4286 907.6 84 951.7692 961.5 967 928.5 967 962.875 835 929.2857 213 33 34 35 36 37 38 39 40 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 901 0 35 989 838.8 0 36 959.38464 960.4 827.6667 0 37 952.3333 775.6 883.2308 808.6 0 38 980.2 864.0833 937.8 780 769 0 39 725 752.5 980.2 858.5714 945 846 0 40 889 934 962.875 862 954.625 957.5714 947.2 0 41 554.5 663.4 980.2 957.5714 967 886.8571 566.2857 967 42 725 752.5 980.2 924.5714 928.5 796.5 673 854.8 43 991 985.8571 814 939.5 967 967 945 923 44 943 1000 938.125 876.25 978 985.8571 881.2 939.5 45 897.7 1000 950.5 946 982 851.5 983.5 907.6 46 901 985.8571 930.7 934 871 888.625 972.5 907.6 47 934 901 1000 960.4 980.2 811.4167 1000 961.5 48 851.5 961.125 949.1 983.5 980.2 907.5 938.125 901 49 967 894.4 901 929.2857 967 957.5714 1000 917.5 50 980.2 922.38464 960.4 920.8 937.8 890 879 1000 51 955 981.1429 892 917.5 932.8333 929.2857 956 776.3333 214 33 34 35 36 37 38 39 40 53 854.8 821.25 901 983.5 882.1667 985.8571 730.5 901 54 929.2857 907 938.125 839.125 921.44446 1000 821.8 858.5714 55 765.7 910.4286 970.3 913 965.7273 1000 835 953.8 56 957.1 839.7143 960.4 955 923.7273 958.75 912 907.6 57 1000 950.5 703 950.5 957.5714 967 1000 1000 58 1000 967 920.8 538 983.5 964.6429 1000 985.8571 59 940.6 905 957.6 854.8 835 927.3333 934 983.5 60 1000 980,75 1000 967 851.5 786.5 967 985.8571 61 846 901 946 953.8 935.5833 983.5 901 764.875 62 917.5 1000 890 895.5 970.6667 983.5 755.8 802 63 887.8 797.2857 989 983.5 896.3 926.6667 806.125 901 64 849.1429 953.8 973 925.75 963.0833 879 967 859.75 65 934 874.5 932.6 967 970.3 973.5 938.125 1000 66 979.6923 1000 923 855.3077 912 980.2 868 973 67 967 919.8571 829.5 877.9 915.1429 938.125 950.5 971.7143 68 987.625 934 923 919.3333 837.53845 934 939.5 985.8571 69 960.4 935.46155 937.8 934 894.4 694.75 983.5 983.5 70 980.2 989 960.4 1000 964.6667 992.9286 967 957.5714 71 980.2 941.0833 957.6 928.5 1000 961.2857 967 957.5714 72 897.3333 983.5 967 1000 983.5 884.5 879 901 73 950.5 1000 917.5 915.1429 985.8571 983.5 924.1 1000 74 901 868 960.4 • 905.7143 972.5 1000 838 881.2 75 835 835 967 825.5714 879 1000 764.2857 620.5 76 841.6 868 945 934 945.8 926.6667 755.8 980.2 77 890 818.5 980.2 863.2857 923 983.5 820 907.6 78 980.2 973.53845 478.6 940.6 1000 965.8333 1000 950.5 79 927.4 922.5714 895.7778 769 ' 959.6667 911.625 957.5714 947.2 80 961 950.5 888.625 937 731.875 985.8571 901 982.2308 81 877.9 868 980.2 934 844 694.5714 952 934 82 861.4 906.0769 920.8 980.2 957.6 991.75 752.5 829.5 83 851.5 884.5 923 931.25 989 961.5 868 615 84 676.6 946.6923 980.2 894.4 957.6 961.5 692 1000 2 1 5 41 42 43 44 45 46 47 48 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 0 42 693.5714 0 43 1000 983.5 0 44 983.5 729.4 940.6 0 45 967 857 881.1429 607.3 0 46 1000 961.5 883.5 874.6 869.26666 0 47 835 967 955.5714 980.2 795.2857 866 0 48 901 934 791.375 960.4 915.1429 830.2857 863.875 0 49 982 915.1429 676.6 846 846 813 683.2 465.4 50 960.4 895.5 938.7143 841.6 929.2857 957.5714 829.9231 769 51 1000 1000 775.6 901 922.2143 787.8571 985.8571 637 216 41 42 43 44 45 46 47 48 53 830.2857 884.5 714.4286 679.4286 880.375 987.625 950.5 725.375 54 872.7143 901 779.2727 744.25 811.9 901 973 774.875 55 747 807.5 934 719.5 833.73334 920.8 929.2857 863.2857 56 917.5 934 836 782.2 808.6 925.2 967 723.8571 57 1000 1000 574.6667 872.125 846 923 901 950.5 58 943.4286 901 927.2857 882.1429 841.5 870.375 847.5833 888.625 59 1000 884.5 728.5714 940.6 971.7143 811.4286 893.38464 814.375 60 952.8571 983.5 870.7143 783.1429 672.375 853.875 741.5 950.5 61 934 917.5 883 849.6667 894.4 891.1 980.2 612.25 62 . 1000 1000 818.5 881.75 871.6667 879 1000 791 63 752.5 863.875 876.25 697.5 762.5 934 957.5714 766.25 64 868 884.5 877 682.375 807.5 939.5 960.4 925.75 65 841.6 901 868 881.2 813.4286 943.4286 962.875 854.2308 66 983.5 873.5 907 391 739.3 868 901 967 67 945 884.5 934 719.5 496.75 884.5 915.1429 895.5 68 1000 917.5 919.0833 962.875 899.7273 562 799.2 948.6667 69 914.2 901 981.1429 980.2 929.2857 811.4286 579.6923 862.125 70 971.7143 884.5 910.4286 886.8571 950.5 987.625 876.25 637 71 886.8571 967 808.7143 858.5714 925.75 958.75 851.5 513.25 72 967 895.5 810.25 985.8571 909.25 925.75 901 780 73 1000 910.9 802 884.5 934 947.2 967 805.6667 74 844.4286 892 923 868 939.5 895.5 901 773,125 75 891.1 783.1429 854.8 920.8 874.6 960.4 1000 884.5 76 835 716.2 830.875 774.5 760.75 913.375 915.1429 765.3333 77 750.1429 868 895.5 696.4 824 917.5 846 962.875 78 1000 868 844.4286 980.2 985.8571 985.8571 984.7692 936.375 79 1000 891.5714 943.4286 769 934 985.8571 943.4286 932.8333 80 983.5 1000 945 865.25 950.5 633.7 912 1000 81 975.25 898 974.0833 772.3 677.0833 883.3333 730.9231 992.38464 82 821.8 769 643.7143 920.8 971.7143 985.8571 984.7692 713 83 960.4 920.8 821.8 843.25 901 857 950.5 758 84 881.2 747 841.7143 722.8 830.2857 868 916.2308 675.875 217 50 51 52 53 54 55 56 57 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 0 861.4 630.4 0 705.7143 218 50 51 52 53 54 55 56 57 53 769 869.5833 762.2857 0 54 915.1429 785.7273 749.7273 628 0 55 934 677.4286 815.1429 684.75 807.2 0 56 692 870.7143 852.8571 595.75 805.3 710.86664 0 57 868 938.125 835 824 796 967 864.3333 0 58 943.4286 967 985.8571 936.3571 943 938.125 962.875 739.6667 59 841.6 946.6923 858.5714 852.6667 866.2727 901 813.4286 925.75 60 971.7143 851.5 943.4286 879.7857 982 938.125 822.625 930.3333 61 851.5 673.8 716.7273 585.1667 738.375 751.1 787.4 868 62 901 917.5 805.3 901 488.5 860.6667 857 756.625 63 917.5 837.7143 874.5 562.1111 717.75 533.875 602.25 950.5 64 912 759.6 926.7273 557.6667 787.875 641.3333 635.8333 857 65 914.2 1000 802 925.75 888.625 841.7143 901 697.5 66 807.5 881.2 901 901 745.4286 854.8 749.2 860.6667 67 908.3333 896.2857 937.6667 872.125 784 888.625 843.25 954.3077 68 841.6 960.4 912 960.4 956 964 961 950.5 69 920.8 832.46155 985.8571 991.75 982 830.2857 896.2857 1000 70 646.4286 721.0833 696.2857 739.7143 656.7273 717.75 825 842.3333 71 533.2857 683.75 816.1429 800.6429 701.2727 909.25 853.25 989 72 743.3333 583.375 872.125 847.375 868 847.375 773.125 921.3077 73 901 901 565.5 901 862.5 815.2 782.2 824 74 943.4286 851.5 681 527 584.2 543.5 400.5 910.4286 75 841.6 901 861.4 730.5 542.7143 901 821.8 917.5 76 721.3333 . 767 849.75 694.1111 746,7273 519.75 697.125 860.6667 77 938.7143 917.5 983.5 642.5 551.2 604 653.5 938.7143 78 980.2 906.0769 868 956 841 967 1000 562.75 79 854.8 886.8571 943.4286 884.5 901 797.2857 957.5714 901 80 714 983.5 846 971.7143 943.4286 960.4 808.6 839.125 81 962.875 817.2308 958.75 901 917.5 958.75 928.5 937 82 1000 932.9231 903.7143 689.25 487 818.8571 816.1429 797.875 83 901 983.5 683.2 752.5 628.75 919.3333 915.6667 814.375 84 901 714.61536 715.1429 749.75 649 587.8571 688.8571 888.625 219 58 59 60 61 62 63 64 65 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 220 58 59 60 61 53 54 55 56 57 58 0 59 868 0 60 918.8571 906.5 0 61 771.3333 920.8 884.5 0 62 714 851.5 961.5 717.6667 63 945 924.5714 941.3333 709.5 64 787.8333 894.4 879 637 65 925.75 837.375 987.625 839.125 66 815.2 901 716.2 890 67 987.625 957.5714 715.375 840.5 68 918 696.4 832.2 937 69 934 925.3077 695.9167 920.8 70 850.1429 950.5 886.8571 532.5 71 971.7143 840.5 893.9286 813 72 1000 987.625 863.875 881.2 73 796.5 615 967 703 74 813 752.5 967 494 75 983.5 917.5 983.5 637 76 869.55554 910.4286 908.3333 657.625 77 967 884.5 928.5 747 78 824 877.0769 958.75 881.2 79 593.375 800 962.875 851.1429 80 915.1429 384 858.5714 863.875 81 949.5 913.6923 644.0714 970.3 82 906.5 818.3077 925.75 627.6 83 901 846 983.5 655.3333 84 958.75 800.53845 925.75 680.4 62 63 64 65 0 861.4 0 825.5714 668.44446 0 758 781.1667 835 0 815.75 960.4 877.4286 846 913.375 802 788.8 983.5 925.75 941.3333 947.75 934 1000 882.1429 874.6 948.75 884.5 932.44446 769 950.5 884.5 769 785.5 911.625 925.75 835 736 895.5 642.5 838.6667 697.5 820.3333 630.4 632.875 675.5 863.875 697.5 742.6 617.2 829.5 795.4 569.6 519.6667 787.3333 769 719.5 642.5 896.875 835 985.8571 1000 800.25 808.6 958.75 758.375 792.5833 909.25 980.2 929.875 821.8 960.4 908.61536 934 944.1539 598.5 771.7143 858.6 839.125 619.2308 729.4 773.7143 571 736 682.1429 799.2 872.125 221 66 67 68 69 70 71 72 73 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 222 66 67 68 69 53 54 55 56 57 58 59 60 61 62 63 64 65 66 0 67 824 0 68 946.375 730.5 0 69 821.8 886.8571 914.2 0 70 940.6 946.375 1000 901 71 821.8 913.375 960.4 897.0833 72 908.3333 857 940.6 925.75 73 829.5 983.5 877.4286 917.5 74 736 851.5 967 950.5 75 844.4286 901 841.6 1000 76 854.8 871.6667 886.3333 882.1429 77 714 675.5 862.5 928.5 78 980.2 839.7143 980.2 976.0769 79 782.2 874.6 934 955.5714 80 763 844.4286 457.85715 917.5 81 930.7 811.9 846.8 744.6923 82 841.6 896.2857 980.2 961.9231 83 909.25 913.375 905.125 978 84 881.2 858.5714 940.6 913.6923 70 71 72 73 0 637 0 698.875 641.125 0 851.5 840.5 795.4 0 846 950.5 785.5 600.7 884.5 884.5 881.2 868 758 769 818.5 846 901 934 912 897.7 879 891.5833 896.875 796.5 840.875 936.375 841.6 901 915.1429 971.7143 987.625 775.6 957.5714 950.5 930.7 973 743.0833 767.4167 822.625 829.5 967 796.5 863.875 642.5 734.8333 775.6667 896.875 835 223 74 75 76 77 78 79 80 81 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 224 74 75 76 77 78 79 80 81 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 0 75 773.7143 0 76 689.8 612.25 0 77 700 787.8571 604 0 78 813 884.5 985.8571 983.5 0 79 891.5714 960.4 781.55554 721.8571 743.4286 0 80 901 917.5 920.8 874.6 983.5 894.4 0 81 973 756.625 917.5 901 977.1539 958.75 848.2 0 82 692 901 856.5714 769 903.53845 943.4286 934 946.6923 83 696.4 785.5 782.2 901 884.5 802 884.5 901 84 703 697.5 625.5714 785.5 939.0769 971.7143 802 817.2308 225 82 83 84 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 226 82 83 84 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83v 84 0 598.5 748.6923 0 813 227 Appendix B: Individual MDS Plots Low Amplitude 4 3 79 65 73 27 Figure 1. M D S plot for subject 1 of subset study. Axes labeled accordingly. Stimuli numbers as in Table 3.4. Colours are to help distinguish between rhythms. A l l proceeding figures are presented in the same manner, for the next 14 subjects. 228 27 37 4U 22 10 H l 8 h Frequency M | 9 N Amplitude Low Amplitude L o w Frequency 23 36 2E 14 Figure 2. M D S plot for subject 2 of subset study. 229 Low Frequency 18 _ Low K M Frequency High Frequency High Amplitude Figure 3. MDS plot for subject 3 of subset study. 230 M 2 15 70 1 * 43 28 9 S 46 68 High Amplitude 40 77 33 52 4 58 Low \ Frequency i A High 54 ? \ Frequency 50 82 Low Amplitude 81 5 7 61 34 47 79 80 44 56 13 69 71 68 45 6= 5 3 Figure 4. M D S plot for subject 4 o f subset study. 3 7 66 54 38 m Low Amplitude 12 G 3120 33 18 1 5 41 3 Low 71 Frequency W "7 High / Frequency 14 1 60 — — H i g h - Amplitude 7 29 5 22 21 21 40 :" 4 Figure 5. M D S plot for subject 5 of subset study. 232 48 77 50 60 70 83 High Frequency 35 52 3 m * High ^ . . . - — — - Amplitude Low Amplitude 54 81 57 43 78 3TB4 10 Low Frequency 80 1 i 38 68 23 62 67 5875 1 8 Figure 6. M D S plot for subject 6 of subset study. 233 234 38 n 70 41 s 71 80 47 81 Low Amplitude Low Frequency 30 78 66 19 49 A?. at High Frequency ^High Amplitude 52 2 17 82 7 », 56 57 SS 6 18 27 S 1 Figure 8. M D S plot for subject 8 of subset study. 235 7 18 47 28 59 » * S 9 * High Frequency N 20 58 13 70 7 9 G8 46 33 85 57 Low Amplitude High Amplitude 25 4 2G 24 38 1 \ Low Frequency •fc 3E 15 71 43 B6 "56 82 22 14 40 19 Figure 9. M D S plot for subject 9 of subset study. 236 40 61 82 Low i s Amplitud High \ Frequency G 48 71 : 7 28 41E 67 4 • 79 37 5*75 20 3 1 4 1 44 Low \ Frequency ss High Amplitude 24 3E 15 26 65 5 3 56 1 Figure 10. M D S plot for subject 10 of subset study. 237 18 2 f ^ 78 i 1 3S7? High Amplitude High Frequency 57 50 37 Low Frequency Low Amplitude 4 G8 gj . 80 9 67 46 70 43 81 52 Figure 11. M D S plot for subject 11 of subset study. 238 2 3 9 61 '*% 83 H Ai 34 3S 1 5 57 40 1» 82 gh mplitude High ^ Frequency Low j Frequency L o J Ampl 28 7 77 22 56 66 45 tude 5 9 « 38 8 1 5|5 44 79 2 52 43 14 Figure 13. M D S plot for subject 13 of subset study. 240 18 31 3 2«5 High Frequency Low Amplitude Low Frequency High Amplitude '143 73 87 30 35 6 4 3?" Figure 14. M D S plot for subject 14 of subset study. 12 62 52 31 High 54 Frequency 48 " 20 a 2 19 22 34 30 43 37 High Amplitude 4 Low Amplitude so 83 to 18 70 81 68 17 Low Frequency 58 46 35 1 6467 2 5 59 77 38 24 80 4*5 3 Figure 15. M D S plot for subject 15 of subset study. 242 Appendix C: Subsets Table 1. Subsets Used in First Part of Subset Study. Listed are the numbers of all the stimuli used in each of the 5 subsets given to participants in the study described in Section 7.2.1 t1 Subset 2 Subset 3 Subset 4 Subset 5 1 2 2 1 1 3 5 3 3 2 4 6 4 4 3 8 7 7 5 4 9 8 9 6 8 10 11 13 7 10 12 16 14 9 12 13 17 15 10 17 14 18 18 11 18 15 19 19 12 19 16 23 20 13 20 17 24 21 14 22 20 25 22 15 23 21 27 24 18 24 22 29 25 20 25 23 30 26 21 30 26 32 28 22 31 27 34 33 24 32 28 38 34 26 34 31 39 36 28 35 32 41 38 29 37 33 42 40 30 38 35 47 43 31 43 36 48 44 33 45 37 49 45 35 46 39 50 46 36 47 40 52 47 37 48 42 53 50 40 50 43 56 51 41 51 44 57 52 43 52 45 58 53 44 53 46 59 54 45 54 48 62 55 46 55 51 64 56 49 57 54 66 57 51 58 55 68 58 53 59 60 69 59 54 60 61 70 61 55 62 62 71 65 56 63 63 72 66 60 64 64 73 68 61 67 65 74 69 63 68 243 67 75 70 65 69 72 76 71 66 70 73 77 77 67 75 76 78 79 71 77 78 80 80 74 78 79 81 81 75 80 80 83 82 79 81 82 84 83 82 83 Table 2. Subsets Used in Second Part of Subset Study. Listed are the numbers of all the stimuli used in each of the 7 additional subsets given to participants in the second part of the subset study described in Section 7.2.4 6 Subset 7 Subset 8 Subset 9 Subset 10 Subset 11 Subset 12 2 1 1 2 1 5 1 5 3 2 6 2 6 3 6 4 8 7 3 7 4 9 5 9 8 4 8 5 11 6 10 9 5 9 7 12 7 13 11 6 11 9 13 8 15 12 10 12 10 14 10 17 14 11 13 11 15 11 19 16 15 14 13 16 14 21 18 16 15 16 19 16 23 21 18 17 17 21 17 24 22 19 18 18 25 20 25 23 20 19 19 26 22 26 24 25 21 20 27 23 27 25 26 22 24 28 27 28 27 28 23 25 29 29 30 29 29 26 27 33 30 31 30 31 28 31 34 31 32 33 34 29 36 36 32 34 34 38 30 37 38 33 35 35 39 32 38 39 35 37 36 40 33 39 40 36 38 37 41 35 41 41 37 40 39 42 36 42 42 39 44 42 45 37 43 44 41 47 43 46 38 44 47 42 48 47 47 40 45 49 43 50 48 49 41 46 50 45 52 50 50 44 49 52 46 53 51 52 48 51 57 48 57 53 54 49 52 58 49 58 56 55 52 53 59 51 59 58 56 56 54 60 54 61 60 57 57 55 61 55 62 61 58 59 56 65 56 63 64 59 60 57 244 68 62 64 65 63 61 59 69 63 67 66 68 62 60 70 64 68 67 69 64 62 72 65 69 68 70 65 63 73 66 70 71 71 66 66 74 67 72 72 73 69 67 76 71 75 73 74 70 69 77 72 77 74 76 71 70 79 73 78 76 77 74 72 80 74 79 77 79 75 73 81 75 80 80 80 78 75 82 76 81 81 81 79 76 83 78 82 82 83 80 80 84 80 83 83 84 82 84 UBC I rtmgj T H E U N I V E R S I T Y O F B R I T I S H C O L U M B I A You hereby CONSENT to participate in this study and acknowledge RECEIPT of a copy of the consent fbnn: NAME (please print) SIGNATURE DATE If you have any concerns regarding your treatment as a research subject you may contact the Research Subject Information Line in the UBC Office of Research Services at 604-822-8598. Past iff3 803-0470 mised &Q4/2607 248 UBC I T H E U N I V E R S I T Y O F B R I T I S H C O L U M B I A You hereby CONSENT to participate in this study and acknowledge RECEIPT of a copy of the consent form: NAME (please print) SIGNATURE DATE If you have any concerns regarding your treatment as a research subject you niay contact the Research Subject Information Line in the UBC Office of Research Services at 604-822-8598. Pttge3<&3 261-6470 rwisfd&W/m? 251 T H E U N I V E R S I T Y OF B R I T I S H C O L U M B I A You hereby CONSENT to participate in this study and acknowledge RECEIPT of a copy of the consent form: N A M E (please print) SIGNATURE DATE If you have any concerns regarding your treatment as a research subject you may contact the Research Subject Infoimation Line in the UBC Office of Research Services at 604-822-8598. PagtScfS BQ1-M70 ravin* SW2O07 Figure 4. E th ics A p p r o v a l F o r m JgC The University of British Co/umbra Office of Research Services v f^B»^ ' Behavioural Research Ethics Board Suite 102, 6190 Agronomy Road, Vancouver, B.C. VST 123 CERTIFICATE OF APPROVAL- MINIMAL RISK RENEWAL PRINCIPAL INVESTIGATOR: DEPARTMENT: UBC B R E B NUMBER: Karon E. MacLean UBCfSdence/Computer Science H01-80470 INSTrrUTlON(S) WHERE RESEARCH WILL B E CARRIED OUT: Institution 1 Site UBC Point Grey Site Other locations where the research mil be conducted: M/A CO-INVESTIGATOR{S): Ricando Pedrosa Colin Swindells Susan Gerofsky Moorin Fazal David Ternes Viatt Savage-LeBeau Mario Enriquez Steve Yohanan SPONSORING AGENCIES: Innovaion and Science Council of British Columbia - *Physica3 and multimodal user interfaces - usability & psychophysics" Natural Sciences and Engineering Research Council of Canada (NSERC) - T h e design of multi-modal symbolic information displays" - "Orsfl title - Physical user interfaces: Cornrnunication of information and affect" Various Sources P R O J E C T TITLE: QraJI title - Physical yser interfaces: Communication of information and affect EXPIRY DATE O F THIS APPROVAL: June 14, 2008 |APPROVAL DATE: June 14, 2007 The Annual Renewal for Study have been reviewed and the procedures were found to be acceptable on ethical grounds for research involving human subjects. Approval is issued on behalf of the Behavioural Research Ethics Board 255 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0052080/manifest

Comment

Related Items