UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Mother Tongue : a study of participant affect in an interactive installation Vander Zaag, Elizabeth 2007

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-ubc_2007-0628.pdf [ 2.14MB ]
JSON: 831-1.0076887.json
JSON-LD: 831-1.0076887-ld.json
RDF/XML (Pretty): 831-1.0076887-rdf.xml
RDF/JSON: 831-1.0076887-rdf.json
Turtle: 831-1.0076887-turtle.txt
N-Triples: 831-1.0076887-rdf-ntriples.txt
Original Record: 831-1.0076887-source.json
Full Text

Full Text

MOTHER TONGUE: A STUDY OF PARTICIPANT AFFECT IN AN INTERACTIVE INSTALLATION by ELIZABETH VANDER ZAAG B.A., University of Western Ontario, 1973 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in THE FACULTY OF GRADUATE STUDIES (Interdisciplinary Studies) THE UNIVERSITY OF BRITISH COLUMBIA October, 2007 ©Elizabeth Vander Zaag, 2007. Abstract What happens when we can see the sounds of our speech? Is it possible to experience a shift in feeling when speaking the prosodies of other languages? People have an innate visual sense of their language sounds. Mother Tongue mines our visual sense of language to explore our potential to inhabit different cultural and subjective sensibilities through vocalization of different language sounds as prompted by an interactive spectrograph. Participants voiced sounds from 4 language samples, Frisian, Mandarin, Canadian and British English. Using qualitative analysis to understand the experience of interacting with the spectrograph this paper charts a course for understanding participant affect in an interactive installation. 11 Table of Contents Abstract Table of Contents Introduction Hypothesis Research Objectives.... Relevance of the Study Background Theoretical Influences .. Embodiment Mimesis Common Ground Feminism Related Work Research Methodology 1 Qualitative Analysis •• 1 Phenomenology 1 Arts Based Research 1 Artistic Probe Art Practise as Research Reflexivity in Arts Based Research..... Summary of Study Goals Study Design Installation Design Speech Data Real-Time Spectograph Projection '. Study Procedures Ethics Review Participant Selection Location Questionnaire Interview Data Analysis and Results Challenges to Data Analysis Researcher Bracketing Self-Reporting Coding and Categorizing Shift in Physiology 27 Memories, Images and Associations 30 . Performance/Game 31 Influence of Voice Prompts 32 Performance as Source of Emotion 32 Validity of data • 37 Results of the Experience of the Display ." 39 Discussion and Conclusions ... 40 Future Work • 42 Bibliography •• 45 Appendices 49 Appendix A 49 Appendix B , 51 IV I n t r o d u c t i o n This essay will chart a qualitative analysis of participant experience with an interactive voice installation entitled Mother Tongue. The research methodology used is arts based research rooted in the phenomenological tradition of qualitative analysis. Throughout the paper the potential for qualitative analysis in describing participant experience complements the potential for evocative art projects to instigate these experiences. The question that prompted the creation of the interactive work is whether people experience a shift in feeling when producing the sounds of another language. To facilitate speaking an unfamiliar language an interactive display of voice sound waves entitled Mother Tongue is projected onto a screen. Mapping our aural sense to our visual sense is one of the ways that new views can be made possible. While the art work provides an instrument to answer the research question the research will further inform the creation of the art work. Hypothesis The premise of Mother Tongue is that embodiment through physical enactment of different languages may result in shifts in feelings. The hypothesis is that as we engage with an interactive visual voice interface and produce sounds of different languages we experience a different vocal tract activity which can impact our affective and subjective experiences. The framing of this question through a technological interface connects this hypothesis to a wider premise of the mediating influence of digital media on language. This study will look at the aesthetic and affective possibilities with digital mapping of speech to image. 1 Research Objectives The participants in the research relate experiences of emotional shift through speaking the different languages in the Mother Tongue interface or relate that they have no change in affect. The design of the study will affirm the robustness of an arts based enquiry based in qualitative analysis to describe participant experience in interactive media. The goals of the Mother Tongue installation are to facilitate embodiment of different languages to gain greater awareness of the complexities and materiality of mother tongues. There is a tension in the way that the arts-based methodology both informs and mediates the experience. This paper will focus on the analysis of participant experience in interacting with the installation to answer the primary research question. Relevance of the Study The research and artwork described in this paper contribute to the growing body of knowledge about the experience of emotion in digital environments. This study demonstrates the potential of arts based research and qualitative analysis to understanding digital experiences. This artwork is developed within and references an interdisciplinary framework. The work has relevance to and finds sources in the field of Computer Mediated Communication (CMC), Visual Art and Linguistics. For the purposes of this paper C M C encompasses work being done in New Media, Social Computing, Computer Supported Collaborative Work (CSCW) and Human Computer Interface (HCI). This work contributes to the growing affordances in HCI 2 for greater emotional and embodied interactivity (39). In Visual Art and CMC digital experience is evaluated whether for creating better designs or for understanding aesthetic and social impacts of interaction with digital media. Digital environments are creating new places for creative and critical enquiry in the visual arts. The interactive installation Mother Tongue expands the possibilities for visual art. The work references practices and techniques from the field of Phonetics in Linguistics. In the related work section digital interfaces visualizing speech prosody are referenced. I have not found any digital interfaces investigating emotion in the prosody of different languages. Background In a previous art installation entitled Talk Nice participants are coached to raise the the pitch of their voice at the end of phrases to adopt a tentative posture (51). Observation of the participants instigated the hypothesis that when people change the tone of their voice their emotional state shifts. Several participants stood up during the interaction and stated that they could not continue because they were starting to feel unsure of themselves. If changing amplitude and pitch at key points in a phrase could make a person feel differently perhaps speaking the more subtle prosodies of another language could make a person feel differently as well. A phenomenological research project may begin with the personal experience of the researcher (35). My mother's mother tongue is Frisian. My mother tongue is Dutch and my daughter's mother tongue is English. In three generations we have three different languages. 3 In reading letters out loud in Frisian we noted the different sounds of that language. We noted a greater musicality than in the other languages we spoke. My mother and I hypothesized that there are different feelings in Frisian because of the way it sounds. If our hypothesis is true then there is greater poignancy in the continual erosion of languages and dialects in the world today (1.) 4 Theoretical Influences Embodiment Contemporary theories in the fields of neuroscience and philosophy point to how integral physiology is to affect. Cognitive scientists such as Lakoff (26)and Damasio (7) stress the bodily origins of thought and emotion. Lakoff argues that we are limited to thinking what our embodied brains permit (26). Notions of embodiment are central to Mother Tongue and to the philosophical underpinnings of the research methodology, phenomenology. Dourish develops a philosophical approach to understanding HCI by describing interaction with software as embodied interaction. Dourish argues for an approach to digital interaction in "which embodiment is central to and even constitutive of the whole phenomenon" (10, p. 102). Dourish refers to Merleau Ponty with establishing the critical role of the body in linking external and internal experience (34). At the same time cultural theorists such as Hayles (18) and Hansen (16) are identifying embodiment as central to the creation of meaning in interacting with new media work. Both new media and the field of social and tangible computing emphasize the role of the participant's body in interacting with media such as virtual worlds and haptic interfaces. In Mother Tongue visualization of voice sound waves brings to the participant's conscious mind the invisible substrate of their voice and languages. Experiencing an art work means that the viewer should ideally understand its impact as a gain in self-awareness or cultural connection. Subjectivity is that 5 sense or perception of oneself in the experience of the work. Shifting.notions of subjectivity are evolving as the boundaries between ourselves and our technological extensions become more ambiguous (16). Mother Tongue prods the participant into expanded subjectivities through physically producing words in other cultural speech patterns and intonations. Mimesis In Talk Nice it has been asked why I would use an artificial interactive setup to focus on language (47).Tenhaaf suggests that this technological mediation builds into the work an implication that mimicry through the pervasiveness of electronic media plays an important part in the transformation of language (47). Marks states that "mimesis is a form of representation based on getting close enough to the other thing to become it" (31, p.vii) Oviatt brings mimesis into research on speech convergence. In her study of human to human and human to computer speech she found that when the computer voice is low, humans respond more quietly (38). Mimesis facilitates the relationship between embodied human beings and the material domain (16). Mimesis in engaging in interactive artworks may induce emotional states. Ekman postulates that sensory feedback from muscle systems such as facial expression is sufficient to induce a corresponding emotion (12). Ekman et al demonstrate the correlation between affective brain states and facial expressions in their paper aptly entitled Voluntary smiling changes register brain activity (12) Ekman's research confirms the role of mimesis as a basis for interaction with others. Participant experience of their visualized voice will show if the Mother 6 Tongue display facilitates embodiment of different language sounds through mimesis. Common Ground In social computing one of the key theories is Clark's notion of common ground. According to Clark in order for one person to understand another there must be common ground of knowledge between them (3). In a technologically mediated world people are connecting across asynchronous and non co-located places. Gaining a sense of participant presence and having a greater awareness of basic physical attributes fosters greater understanding. Producing the prosodies of another person's language is a way to access that person's reality and to demonstrate a desire for connection. Feminism Semanalysis, a concept developed by Kristeva, focuses on the sounds, rhythms and graphic disposition of language rather than its meaning. Kristeva brings the poetic dimension of the material of language into feminism with the notion of the feminine as the unrepresentable place of the mother( 24). Feminist tenets also influence the methodology of this study. In feminist research approaches the goals are to "establish collaborative and non-exploitive relationships with research participants" (5, p.83). In order to avoid objectification of the participants in this study I share my own language histories and invite the participants into the creation of the art work. The video accompanying the installation 7 attests to the vitality and new perspectives that participant collaboration brings to the final work. The orientation to language acquisition as a matriarchal tradition informs the term "mother tongue". By framing languages of a foreign nature as mother tongues we reduce the distancing associated with different languages. A sense of intimacy is incurred by the notion of mother tongue. The gendered linguistic role of mother in relationship building is inferred (27). Bringing the notion of mother tongue into a technologically mediated environment furthers the feminist stance of this project. The outcome of the study will also be influenced by the positioning of this work through the title and concept of mother tongue. Related Work Artists and researchers use speech visualization strategies to enhance the user's sense of their voice. Levin and Lieberman create voice controlled projections (30).Using parameters of speech such as phonemes and prosodic properties such as pitch and amplitude, they create interactive displays of the user's voice input. They reference the bodily origination of speech by incorporating the speaker's body into large projections of the abstracted representations of their speech. In Pitch Paint the participant's pitch generates abstract paint patterns (30). Synaesthetic mapping between sound and visuals in this work address a similar interest as Mother t Tongue regarding externalization of an invisible part of our lives and the role that technology can play in that. Mother Tongue takes the visual mapping of speech to the goal of facilitating the participant to embody other languages. 8 Speech visualization strategies in the field of social computing are used to transcribe human presence in digital communication environments. The Visiphone visualizes remotely mediated conversation with the goal of showing details of conversational activity (22). Different participants have different colours in the visual display allowing the device to reveal conversational dominance, turn taking and interruption. Visiphone is incorporated in an interactive installation at ISEA 2006 entitled Chit Chat Club (22). The Chit Chat Club expands the telepresence of people in an online cafe by showing animated prosodic features such as word rate and amplitude. In Visual Chitchat participant voice is represented as moving lines and dots. Wild and crazy lines represent yelling, simple flowing lines show calmness. The goals of voice interfaces to develop the aesthetics of phonetics, notions of embodiment, telepresence and emotional cues in social computing are not the goals of most current speech visualization strategies. These strategies are oriented to interventions for language acquisition and for remedial education. Karrahalios' current project combines speech visualization displays to teach autistic children to mimic the patterns on the screen to produce sounds and then words and sentences with displays of social cues such as turn taking in conversation(23). Interventions for the deaf is another focus of speech visualization programs. In 1947 Bell Laboratories set up a program to train a group of deaf adults to read moving spectrographs as a substitute for hearing (17). A current project uses a spectrograph as one of several peripheral sound displays for the deaf. Mechanical sounds are easily identifiable with this because they have regular amplitude and pitch patterns (19). A related project in Japan uses a real time spectrograph which helps deaf people understand speech 9 formants. In this project the detection of formants is integral to speech analysis and forms the most comprehensive component of speech (37). In the Mother Tongue interface formant detection was considered in the design of the interface. Participant analysis will determine the success of that strategy. 1 0 R e s e a r c h M e t h o d o l o g y Qualitative Analysis A UBC study of affect in user interface design employs analysis techniques such as self-reporting by having participants identify their emotional responses on an affect grid. Biofeedback measurements were also used (46).Statistical analysis is not suitable for analyzing user response to Mother Tongue because in this project the interface is a vehicle for enhanced experience of language. Qualitative analysis allows for a more open ended description of participant experience. Marshall and Rossman are drawn to qualitative research "because of the ability to present an answer instead of the one answer" (32, p. 174). As HCI interfaces afford greater social and emotional interaction by incorporating human emotion into the interface in innovative and meaningful ways qualitative analysis of participant experience is valuable to the evaluation of those experiences (9). Phenomenology The phenomenological tradition of qualitative analysis will be used to analyze user experience in Mother Tongue. Creswell states that "the phenomenological approach is individualistic and from individual experience general or universal meanings may be derived" (5, p.53). The phenomenological approach is appropriate for this study since it is the individual user experience that will be analyzed. Concepts of embodiment in the way we take in digital media have a basis in phenomenology through turning action into meaning (10). The precursive 11 preoccupation of Mother Tongue is rooted in the longing for experience outside of a cognitive place and celebrates the pre-cognitive role of prosody in language. The phenomenological tradition is also useful for this study in that the researcher may bring their own question to the study (5). Arts Based Research Arts based methodologies are becoming more prevalent in qualitative analysis. Art practice is seen as a way to raise theoretical questions and enable people to have experiences that bring new insights (45). The role of art in eliciting participant experience is primary in this study. There are two branches to arts based research practices. One is the use of the artistic probe to illicit responses and discover new things about people (41). Another is the goal of art practice as research (48). Mother Tongue is both an example of art practice as research and an example of the use of an art work as a stimulus designed to prompt answers to a research question. Artistic Probe Arts based methodologies often rely on the use of displacement techniques such as vignette technique (41). Photo elicitation has been proven to be a powerful tool for eliciting descriptions of experiences (21). Cultural probes have proven to evoke details of experience that simple questions could not. Gaver et al used cultural probes to garner a wealth of information about participant experience in the design of an interactive public installation for elderly people in their local 12 community(14). In these examples the art work is simply a probe for further information about the experience being investigated, the art work is otherwise not central to the experience. In Mother Tongue the art work instigates the experience but is also central to the experience. Art Practise as Research Sullivan posits his book Art Practise as Research:lnquiry in the Visual Arts on the premise that the imaginative and intellectual work done by artists is a valid form of research (45). The difference to traditional goals of research is that the outcomes of artistic enquiry can generate multiple outcomes instead of finding a more reductive outcome which is often the goal of traditional research (48). Precursors for arts based research originate within art education (11). What distinguishes arts based research is the multiplicity of ways of encountering and representing experience and the different forms of expression that can communicate these phenomena (11). Multiples presentations of research outcomes such as visual art, video, and creative writing each with their own affordances are verified (20). Texts such as A/PJTography: Rendering Self Through Arts Based Enquiry generate an explicit connection between artistic practice and research (20). The process oriented research manifested in the writings and images in A/R/Tography parallels the textual and image work of Daredevil Research which contains juxtapositions of text and image that are simply evocative not explanatory allowing for multiple presentations of data and affords an opportunity for "a coalition between the visual and the verbal that is both critical and supportive". (21, p. 188) 13 I find a parallel between my art process and the qualitative research process. In qualitative analysis the researcher combs through the transcripts of participant interviews for common themes or for metaphors that could point to a larger meaning (32). In creating interfaces one looks for the essential components of an activity to create a user interface that can mimic that activity. In editing video I scan the footage to see and hear it in a new way. The data or video clips which resonate the most are the ones that emerge and become included in the final work. My personal experience initiates and motivates the research but new perspectives emerge from the collaborative input from participants. Moustakas describes a heuristic approach to investigation in which the researcher must first reflect on the personal meaning of the experience and then turn outward to those being interviewed and establish intersubjective validity (36). Arts based research is a process of enquiry that prefers insights emerging from sustained reflections on phenomena - like qualitative analysis. Reflexivity in Arts Based Research The interactive arts are predicated on interaction by the participant with the art work. The reflexive work done by the participants in interacting with the art work is as important as the reflexive practice that occurs in the creation of the work. Hayles describes the role of the participants in a digital project as "reflexivity is the movement whereby that which has been used to generate the system is made, through a changed perspective to become part of the system it generates (18, p. 8). Participant's description of their experience forms the substance of the subsequent 1 4 reflexive analysis of the researcher. The collaborative relationship between researcher and participants in qualitative analysis parallels the collaborations between artist and viewer in new media work. Summary of Study Goals A dilemma arises as to whether the qualitative study is at the root of the creation of this art work or is the art work a probe for the study to answer the research question. This dilemma prompts the constant reiteration of specific goals for both the art work and the study. The goal of the art work is to most effectively allow the participant to reflect on the materiality and complexities of mother tongues and speech visualization. The goal of the study is to interpret participant experience of the interactive installation Mother Tongue and discover if a shift in emotion occurs in performing the prosodies of the different languages. 15 Study Design The study design consists of the creation of the Mother Tongue installation in which participants are coached to repeat phrases spoken in 3 different languages; English, Frisian and Mandarin. Both British English and Canadian English samples are presented. To discover and describe participant experience of the installation and of language prosody a combination of data collection methods are used. A questionnaire precedes interaction. Participants engage with the installation by repeating the sample phrases into a microphone triggering a software spectograph and viewing the resulting display. I observe the participants while they are interacting and note details of emotional reaction. Afterwards I interview participants about their experience. Subsequent data analysis consists of transcriptions, writing analytic memos, and thematic coding. The results are verified through triangulation by comparing results of these different data sources. Installation Design Speech Data Initial research into what the Frisian language looked like on a spectrograph turned up salient features. Samples in a Frisian database supplied by the Fryske Akademy in Friesland were analyzed. These samples were compared to samples recorded in the Speech Lab at UBC by Linda Kaastra, a second generation Frisian speaker. The emergent patterns show frequent and complex vowel transitions. The vowels as traced by formants seldom maintain a steady state and often the F1 and 16 F3 crisscross. Dipthongs and tripthongs in which one vowel turns into two or three are more plentiful in Frisian than in Canadian English. British English and Mandarin were also studied. British English vowels barely shift while Mandarin has rich vowel variations with quite frequent consonant activity. Short neutral sample phrases in these languages were recorded and programmed into interactive prompts using Flash software. The phrases in the 4 languages were produced by 2 males and 2 females. The British phrases were actually from the speech lab's vowel prompts and were included to introduce people to the vowel positions of their mouth. They are also included because for most Canadian English speakers producing British sounds is funny. The Canadian English phrases are as neutral as possible to avoid creating emotion through the story. The Mandarin samples consist of set of two syllable phrases and a set of longer phrases. Real-Time Spectograph To facilitate participant production of the language prompts a real-time software spectrograph which displays voice sound waves was programmed. Existing software spectrograph displays include too much extraneous information such as measurements. The display has to be devoid of other data for this project. Chang Liu hacked into existing PRAAT software and created a windowed but adequate spectrograph display. Because the essential role of the display is formant display he created another program displaying only F1 and F3, some of the key formants used in vowel production. Compared to the richness of the vocal textures in the spectrograph display the formant display seemed like a waste of good screen real 17 estate, so the spectrograph display in textures of black and white became the visualizing interface. Projection The installation consists of two adjacent screens with matching whites. Flash sequences of the audio prompts activate the spectrograph on one laptop which is projected on the entrainment screen. The participant faces the screens and speaks into a Sony microphone which activates a spectrograph on another laptop which is projected on the user screen. The movement of the sound waves becomes continuous through the two screens. Study Procedures Ethics Review Permission to involve humans between the ages of 19 and 84 in this research was granted by the Behavioural Research Ethics Review Board at UBC. Participants agreed to the research by signing consent forms. Participant Selection Seventeen participants were recruited to interact with Mother Tongue. One aspect of phenomenological enquiry is that enough subjects be exposed to the phenomena of interest to make the research robust but not so many that the results become too bulky (5). Interest in the project and availability were the main criteria for 18 participant selection. Slightly more females than males participated. The age range of the participants was between 20 and 84. Participants came from a large variety of language backgrounds. All of the participants were articulate and willing to try out a new thing. Participants were told that they would interact with an interactive artwork entitled Mother Tongue. Some participants were specifically invited and others were found on the campus. Location The Photo Studio of the Visual Arts was made available to set up this installation. The use of one location for the interviews and experience of the installation provides a consistent context for the experience. For people who came from outside the campus the experience was somewhat different than for those who were on the campus due to the location of UBC in the context of the city of Vancouver. The location of the study in the visual arts facility gave credence to my invitation to participants to interact with an art work. Questionnaire In phenomenological research it is useful to find out past experience of the phenomena of interest and the present experience of the phenomena of interest which is gathered in the interview (5). Joining these two narratives can describe individual's essential experience of the phenomena. Participants filled out a questionnaire which primed them to think about prosody in their lives. The questionnaire asked about other languages, mother tongues, childhood language games, interaction with visual speech displays and generally what sounds of 19 languages they enjoyed. An example of purposeful voice prosody change is in speaking to a pet. Participants were asked to select aspects of prosody in speaking to their pets such as word rate, pitch, amplitude. The questionnaire predisposed some participants towards a more emotional response to prosody. One respondent encapsulated the potential of prosody in her voice to create emotion with this story written in the questionnaire before interaction with the work and before the interview. "Do you have memories of these speech games?" "Mostly it was a dramatic scenario where 'speaking the language' (sometimes on pretend phone) I'd let the prosody carry me away Into various emotional states until I yelled (to hang up the phone) or threw myself on the bed weeping" Interview Kvale describes the intersubjectivity inherent in interviews as "inter views" between two people (25, p.45) In this study the intersubjectivity of the interviews is preceded by the intersubjectivity that take place in interaction with the work. Before the interview has begun, through participation in the art project, through physical embodiment, through instigating the piece, the participant and researcher have already begun their inter view. The impact of the researcher/artist on the experience of the participant is a significant feature in the arts based methodology of this study and is openly acknowledged in the interview process. Kvale suggests mapping the interview questions to the research question (25). The primary goal in the interviews was to discover how participants experienced the interactive work in order to discover if they had a shift in emotion. 20 To maintain a neutral stance and to avoid leading questions I asked interviewees open ended questions about how they felt while engaging with the work and afterwards. During the interview I avoided questions resulting in yes or no answers so that the participants give me enough descriptive words about their experience (42). Before the interaction began I would ask them to notice their emotional state. At the beginning of the interview I asked participants to describe any shift in feeling from engaging with the installation. Can you describe any changes in the way that you feel? This open ended question would be followed by more specific questions lead by the response of the participant. I would ask them to elaborate on what that felt like. How did it feel to produce those sounds? If the participant found the question too abstract I would ask to describe feelings from specific language prompts. Rubin and Rubin suggest making the questions relate as directly as possible to the interviewee's experience (42). I encouraged all associations and feelings that came up for the interviewee by following up their initial responses. To make the participant more comfortable I asked questions about their personal language histories. Where applicable I asked them to say a word in their language to hear the sounds. I also asked each person to describe their experience of the graphical display of their voice. I asked what they noticed about their voice from seeing the display in order to trigger memories of the experience and to focus on the materiality of their voices. I also asked questions as a follow up to the questionnaire about their mother tongue and their mother's mother tongue and about prosody in language in general. 21 Interviewees could feel exposed after discussing their emotional state and it is my responsibility to make them feel secure and protected before ending the interview (42). Towards the end of the interview I would recount some of my own language history. Subsequent to the interviews I converted the audio clips of interviews into Quicktime movies, inserted the clips into Microsoft Word documents (44) and transcribed them. 22 Data Analysis and Results Challenges to Data Analysis Challenges to Data Analysis include researcher bracketing and the variations resulting from self-reporting. These challenges are considered when performing the data analysis. Researcher Bracketing The researcher makes a substantial amount of judgement calls while consciously bracketing her/his own presuppositions in order to avoid inappropriate subjective interpretations (15). While writing memos about the transcripts during data analysis I take care not to evaluate or summarize the words of the participants so that it fits into hoped for research results. Instead I look for the essence of what is said and summarize in the margin of the transcripts. I take care to put aside my personal subjectivities and write memo's that reflect exactly what the transcripts relay not my expectations of it (33). Instead of researching an existing phenomena as often happens in the qualitative research taking place in the health and social sciences, an arts based research project poses challenges to researcher bracketing (45). The researcher has instigated and created the work through a process that is not detached from the essential phenomena of the experience. As an artist my personal experience of being a Canadian born English as a second language student motivates me to 23 create a project to look at language mobility and discover how it resonates for many people. Researcher bias makes me more sensitive to sections of the interviews which poignantly portray the difficulties of speaking English. The discomfort which some recent immigrants exhibited with all languages except their native language moved me. One participant used the words "I don't know" in almost every sentence he spoke. Another participant uttered strange "rrr" sounds to demonstrate how he uses fuzzy sounds to disguise pronunciation of words he can't speak. Upon close examination of the transcripts an interesting phenomena emerged. The most rhythmic responses were from the most English challenged. When a strong expression was called for it seemed that rhythm facilitated a flow and tempo in the language. The materiality of language develops the content. "You know sometimes because I don't have the vocabulary, or sometimes like I can't produce the sounds and sometimes I can't even understand them like in English because there are vowels in English that we don't have in ..." Another participant had more challenges to "find a job or something like that, get a fhend" because of his poor English but he became quite confident when he spoke more rhythmically "I can dance, (laughs) I can sing, (laughs) I can smile(laughs)" In these accounts of emotion in language it is necessary to bracket and unbracket my own experience when deciding the importance of this discovery in the context of the whole study. 24 The primary research question in whether there is a shift in emotion during the, interaction with the speech samples and interface in Mother Tongue. The experience of speaking English as a second language emerges as a common theme in the interviews and is included in the video which is more suited to presenting results with specific rhythms and emotional tones and which do not require researcher bracketing. Self-Reporting The primary data for this research is the transcripts of interviews of participant self-reporting. Variability occurs in the density of participant descriptions. In self-reporting about emotions there is a wide range of granularity. When asked how they feel some participants describe their feelings with a great deal of precision which Barrett calls high granularity while others represent their experience more generally in terms of pleasure or displeasure which Barrett calls low granularity (2). After interviewing one subject for a while without hearing any words relating to emotion I broke the cardinal rule of qualitative interviewing and asked a leading question. Question "... / just want to know if there was a change in feeling when you spoke that language" Answer - "well, just like, just like more difficult" Questions - "and how did that make you feel? Were you comfortable?" Answer-"no, no, I wasn't comfortable" 25 Another participant described his experience in speaking Frisian as "not uncomfortable" although during performance of it his arousal was high enough that he laughed out loud. "It actually makes me feel good. ...It sounds a bit funny like Dutch but it isn't - like you probably saw me laughing so well yeah its fun to do - you could say that... it's not uncomfortable." Lower granularity but a shift in affect is evident. Coding and Categorizing Interview transcripts, memos and questionnaire responses comprise the data. I begin by reading the transcripts and writing descriptive words in the margins. In phenomenological analysis it is important to let "the essential, invariant structure of the experience emerge through descriptions of the textures of the experience" (5, p. 55). Codes or labels for these descriptive words are sorted into larger categories (32). The categories and themes themselves shift as the codes identify the actual experience of the work. For example descriptions of mouth and throat sensations go into an emerging category called physiological changes. Sometimes categories merge into themes (32). For example a category of enjoyment of engagement with the work is comprised of codes such as fun and relaxed. Other descriptive words such as concentrate, satisfying, and difficult also discuss the engagement with the interactive work. The theme of interaction flow emerges and contains description of emotions coming out of the interaction itself. Another common thread is stories of childhood, references to films, domestic language in the house which becomes the theme of associations and memories. 26 I develop themes through an iterative process of re-reading the data, considering each of the research questions and considering what stood out in the data (4). The interpretation of the interviews brings another level of analysis to the study. Discarding my primary question I find recurrent themes, reoccurring words and adjectives that link participant responses (32). In referencing participant descriptions to definitions of emotion I begin to see patterns emerge from the data that answer my question but in a way that I had not predicted. There appear to be two ways that emotions shift in interaction with Mother Tongue. One is from the flow of the interaction with the piece and one is from the emotions elicited from the actual production of the words - the embodiment of the language. Both shifts are a direct result of the interactive art work and cannot be separated from the central experience of the work. To examine resulting emotional shifts and reference literature on emotion it is useful to look at these two themes separately. Embodiment of the language through production of the phrases resulted in descriptions of physiological change and mental associations and memories. Performance of the interactive work resulted in a shift in feeling that came from the flow of the interaction which I will develop in a section entitled performance/game. Shift in Physiology In describing how it felt to speak the different languages most participants described shifts in physiological state. Ekman identifies characteristics which 27 distinguish basic emotions from one another (and from other affective phenomena) as having distinctive physiology (13). Participant awareness of body changes revolved around breath and mouth sensations. "Physically I felt differently, my stomach was more pushed in when I let more air out, especially during the longer vowels, I guess you could say more relaxed just because I wasn't keeping it tight" "I feel like in English you are moving your mouth a lot up and down whereas with the Frisian I felt like I was moving kind of side to side" (laughs at her own movements) Many participants expressed pleasure in their physical experience. Valence and arousal have been defined as two basic parameters of emotion (28). A shift in valence is considered a key dimension in affect variability (28). Visceral pleasures are evident in this response: "I liked the way the my mouth was being manipulated and shaped I thought that was kind of nice and refreshing kind ofexcercise.... It made me think of different tastes as well" The goal of this study is not to identify the specific emotion but simply to gauge a shift in emotion. Some participants articulated how their emotions were impacted by the physiological changes that occurred from their interaction. " there was a kind of hyper awareness of the sound of my own voice so in speaking - no English - not know the text - it was somehow.... More rooted in the body, more aware of my own body which for me is the link for the emotional potentially somehow." Others were more specific about the link between a language and a shift in emotion. In speaking the Mandarin one participant felt blocked. 2 8 "There's a feeling with some of those guttural sounds that come more from the back of your throat and the up and downs... it almost made me feel blocked .... Like there's a sound that duh! Like you just stop suddenly and we're not used to that in English I don't know if we have those sound but that STOPPING., it just felt Oh My Gosh... I couldn't be free to communicate something with a sound like that." Another participant described a distinctive physiological difference between speaking Mandarin and Frisian. "My body felt heavier because I was trying to speak in a lower voice and I was less relaxed.. I tried to be more fluid when I spoke the Frisian. " Question - "did that affect your feelings at the time?" Answer - "Absolutely, I... could feel different emotions kicking in - / felt quite, I found the mandarin quite hilarious and I felt more peaceful speaking the woman's language." Some participants reversed the link between body and language saying they had to change body position to speak different languages. To speak the British English for instance one participant said she changed her body position. "you feel sort of... like you have to sit up properly not cross your legs like this, cross your legs at the ankle like the queen does and when you say those sounds it just makes you feel a lot more proper." Another used body position to imitate others. "if I want to imitate someone's speech I have to move like them first don't you do that?" By asking questions about past experiences of prosody the link between physiology and language also came up. " like I remember speaking Spanish when I went to Spain and tried speaking it and feeling kind of excited... that this sounds sort of beautiful... this sounds attractive! And it made me feel that I was more attractive speaking that language it's true!" 29 Several participants volunteered ideas about the transformative role that voice prosody can play in emotional state through the shifting sensations of their body. "Just by slowing down and bringing in that more sensual aspect of... speaking so you feel your body more as opposed to it coming from that automatic place., it's more like a singsong sort of thing, you could relax yourself by doing that." Performance of the sample language phrases are experienced in the body. Participant memories of these sensations provide details of the lived experience of the phenomena of speaking these languages (10) Awareness of body changes is evidence of the shifts in emotion during interaction with Mother Tongue. Memories, Images and Associations Ekman includes the occurrence of distinctive thoughts, memories and images as a distinguishing characteristic of emotion (13). Anecdotes and references in the interviews speak to the associations and memories that different languages provoke in the speaker. People who did not speak Mandarin associated it with Mandarin movies, with working with Mandarin students, and with riding on the bus. Some participants were new to Canadian English and associated Hollywood movies with it. Many of the memories that participants recounted were pleasant. There is a high level of valence in the memories evoked by interaction with the work. For some the sample sounds brought back memories of domesticity. Anecdotal evidence shows that the interviewees had rich common history of mother tongues and second languages and primary associations were pleasant either 30 through cultural associations or through memories of childhood. Interaction with the four languages in the work was sufficient to bring to mind the materiality of incrementally more languages. Taiwanese, Cantonese, Italian, Norwegian, German, Korean, Japanese, Spanish and French were referenced. For the participants these memories contributed to an overall sense of well being and contributed to their motivation to continue interacting with the work. I coded these descriptions into a category of valence, of pleasure with not much motivation. Arousal as a parameter of emotion speaks to the motivation to interact with the work (28). The memories were pleasant enough to continue but it was performance and interest that piqued participant motivation the most. Performance/Game In interpreting data in a phenomenological tradition of qualitative enquiry the goal is to develop a textural description of the interaction experience (5) Participants relay their experience of interacting with Mother Tongue by describing physiological changes, memories and associations. The second stage of data interpretation is to develop a structured description of how the phenomenon was experienced (33). In looking at the "essence" of the experience the interaction with Mother Tongue becomes the core of the experience. Careful reviewing of participant response to the spectrographic speech prompts reveals the significance of the interactive engine to this research. It is impossible to look at the answer to the research question "is there a change in emotion when speaking different languages" without looking at how the experience of speaking these languages is facilitated and 31 mediated by the interactive visualization of the participants' voice and the voice prompts themselves. Influence of Voice Prompts Despite my efforts to create neutral prompts with equal division between male and female and simple phrases, participants responded differently to the male and female recordings of the prompts. Linda's Frisian prompts were appreciated for the feminine quality of her voice. "It made me feel relaxed and just in a kind of lighter state, a playful state and I think also because my voice was more similar to hers, you know a female voice that I could reproduce the prosody especially with the vowels and so on. ..." A persistent theme throughout the interviews was the description of Frisian as "the woman's language" Many of the participants male and female enjoyed the Frisian saying they experience a sense of lightness when speaking it. "It was higher and sounded more friendly and accessible ... and she sounded very kind... the mandarin was a bit intimidating., a bit gruff." Performance as Source of Emotion An important aspect of validating qualitative analysis is to find alternative understandings of the data (32). An emergent description category was the interaction flow and the performative aspect of the interaction with the work. Initially this looked like an alternate outcome. To answer the primary research question about whether a shift in emotion is experienced when performing the prosody of 32 another language it is necessary to recognize the impact of the context of how that emotion is evoked, i.e. the interactive work Mother Tongue. All of the participants expressed a sense of their performance in the interaction. Performing the words in the different languages and comparing personal performance to the sample display became a strong component of the experience. The participants often described the interaction as an "experiment" or "game". The emotions derived from the participants' engagement with "performing" the different language samples underlie all of the experiences of the piece. One of the recurring sets of descriptions is that the experience was "fun", "relaxing", "interesting." Another recurring set of descriptions is the challenge of the interaction with words such as "concentrate", "difficult", "baffling" A large source of the arousal which Lang relates to motivation to continue comes up in the game aspect of the study (28). Trying to get it right, and trying out the different languages and enjoying the visuals of the display were consistent comments about the experience of the work. A shift in emotion resulted from the actual dynamics of interacting with the work. The emotions involved in game play are evident in the emotions involved in interaction with Mother Tongue. The alternate interpretation, looking for a negative case does not apply to this result because under the evolving research in emotion in game playing the experiences described fall into categories defined in game 33 research as emotion. Gaming emotion theories include interest and engagement in their grouping of gaming emotional states (29). To understand whether engagement and interest in the interaction is an emotional state we look at two divergent theories. Ekman categorizes excitement as an emotion that is distinguishable from other emotions (13). However he defines interest as a cognitive state rather than an emotion (13). Tomkins considers interest and excitement to be at the moderate and extreme end of a primary affect (49). Motivation for continuing an action is evident in the engagement of the participants with the interactive work. Lazarro has divided emotion in game play into specific attributes or keys. To discover if there is a shift in emotion in speaking different languages through the interactive work it is useful to compare experiences of Mother Tongue to Lazarro's research as to how emotion is created in game play. For some participants the challenge of performing the words became a source of emotion. When asked about the display and their sense of their own voice several participants noted that they were more preoccupied with getting the patterns to match which subsequently resulted in emotions related to how well they felt they were doing. "/ was more focused on seeing how those visuals were actually matching." "Q: so did you find that you were able to match the patterns?" "Yeah (smiles) I was quite happy -1 was proud" 34 Lazarro describes this emotion as fiero the Italian word for personal triumph which in this case made the person "quite happy" (29). "from where I was sitting it looked close so I kind of had a feeling of happiness in myself that I had more closely approximated the word visually - that was interesting" The challenge of getting it right focused the attention of some participants. "As I got more confident I did look back and go 'oh, hmmm, I wonder if that was the same or I wonder if any of the patterns are similar'" "you are working imitate the sounds, so it feels more like you have to concentrate you like you working to get it right." "I felt slightly anxious when I came in but as we did the exercise I felt fine, I was having a good time and I was trying VERY hard (smiles)" Another motivation for interacting with the work is the emotion created through sensations which Lazarro calls wonder, awe and mystery. Lazarro calls easy fun where the particpant's attention is held by curiosity rather than winning (29). Several participants were intrigued by the shapes their words made. "I was quite intrigued by the shapes that the words made... I was looking at the lines and shapes and wondered how they might work in a painting" Others became intrigued with matching the graphics "I enjoyed hearing my own voice, after I relaxed a bit and let more air out I was just concentrating on how they looked on the screen and how they correlated to what sounds I made more. " . One participant was awed by the image of her voice. We see that the interaction awakens in the participant a sense of curiosity. "...I've never seen my voice, I've seen my heartbeat...(laughs)... I've seen the way I breathe but I've never seen a person's voice" Question- "so how did you find that?" 35 "Well it was really interesting because I've never thought about it. I've never thought about seeing your voice, it was ... yeah... interesting... I liked it." The sheer enjoyment of experiencing the game activities enticed participants to become immersed in the experience. "I liked the fun of the Mandarin .... Sometimes I just play with it myself... when I'm on the skytrain I pretend that I'm Chinese and I.. to myself, I'll say their words... just for the hell of it" "Mandarin was the easiest because I didn't know what it meant at all... it was just like pure sound... it was pretty satisfying (laughs) yeah." In Lazarro's research players report enjoying the changes in their internal state during and after play. For some participants the experience created an escape or "altered state" In Lazarro's description altered states is the way that a game creates emotions inside the player in contrast with their previous emotional state (29). "/ found it kind of relaxing but that could just be the other variables like its relaxing to focus on the experiment rather than moving which is what I'm doing so it made me feel more relaxed just to be here uhm and not worrying about life stuff." " I was more relaxed because I had no idea what the language meant so I entered more into the experiment" The performance of the languages which were not known to the participants provided a deeper experience or a change in inner state for some. "the English was just automatic... doing the other languages seemed so much more evocative" According to Lazarro emotion is created from the actual digital interaction and flow. "it was kind of like a fun game" "when I got to the Mandarin I was trying to make my voice, like I was trying to play the game - to make it match." 36 Participants were stimulated by interaction with the spectrograph, which for Lazarro is the emotions that come from compelling interaction (29). The spectrographic feedback gave people a heightened sense of their voice. "because I wasn't aspirating as much in the beginning and I wouldn't see it register uhm... there was something about that where I felt that I was FAILING the machine... It was sort of like 'why isn't the machine registering my voice'" "I was just interested in reproducing the sounds and I found that maybe coming out of that relaxed state that I was doing pretty well(laughs)" Comparing participant response to Mother Tongue with Lazarro's characterization of gaming emotions there is substantial evidence that participants experienced a shift in emotion. There is shift in feeling through the performance of the prosody of other languages as experienced with the interface. The testimony of physiological changes and memories and associations confirm the shift in feeling in speaking the different languages. Validity of data Triangulation is one method of proving the validity of research results in qualitative analysis. "Triangulation is the act of bringing more than one source of data to corroborate or illuminate the research in question" (32, p.202). The primary data source for this research is the transcripts of interviews of participant self-reporting. Another data source is observations of the participants during the interaction. Participants were also invited to leave comments on their questionnaires after the interaction. An unexpected data source comes from volunteered 37 information from participants often at the end of the interview. The performative interest in the experience is corroborated by triangulating this data with data derived from observation of the participants, questionnaire data and volunteered comments. Observation of participants showed several instances of participants laughing out loud as they spoke the different languages. Emotional arousal is evidenced in this spontaneous laughter. Flow or engagement with the interaction is also evident in my observation that participants did not stop during interaction despite my statement at the beginning that they could stop whenever they liked. Due to a memory leak in the spectrographic display participants were limited in their interaction with the display to 12-15 minutes. The prompts took up 10 minutes of that time which left several minutes for play. My general observation was that no one left the interface voluntarily and they were often disappointed when it crashed. In an interesting follow up activity one participant brought up the example of chanting as a practice in yoga to relax and heal the body. She performed the Ohm yoga chant and keenly watched the dramatic spectrographic display that resulted. Creative engagement with the interface confirms the positive valence that occurred within the participants. Transferability of the research to other situations is another method of verifying the research. Credibility that the ideas posed by Mother Tongue can be relevant to a larger population is confirmed by the diversity of feelings and notions of mother tongue in this study. The variety of mother tongues as divulged by the participants in 38 this research exceeded my preconceived ideas. My personal experience of mother tongues is expanded by the narratives of the participant group in turn suggesting the generalizability of these findings and this research process to similar research questions, or similar situations. Results of the Experience of the Display A wider premise of this study is that technology is mediating our language. In this study we look at the synaesthetic possibilities for digital voice visualization. The study provides rich feedback on the experience of the spectrographic display. In the interview I ask questions about the display in order to bring participants back into the experience and the materiality of their voice. An unexpected amount of data about what people noticed about their voice resulted. Participants noticed their consonants. Several participants noted the spaces between words. Others noticed the tempo of the words by the speed at which the display occurred. The majority of the women participants noticed the feminine attributes of their voice. Some lamented their lack of voice strength in comparison to the male voice prompts. Others noted the position of their female voice as higher on the display than the male voice prompts. Interestingly enough despite all my research and existing research on language acquisition on the importance of formants to the production of language not one participant reported a sense of a match between the display and their vowels. Spacing, tempo, consonants, gendered voice were noticed but not one mention of vowels. A prototype displaying the F1 and F3 formants only was 39 developed for this study. Fortunately I made a purely aesthetic decision to use the whole spectrograph because it reveals more of the organic and textured aspect of the voice. It is simply more beautiful and it turns out to be more useful as well. Essentially the spectrograph served to keep participants interested in producing the sounds because they were getting some feedback of their voice. Participants also liked the display for purely aesthetic reasons. They describe it as "cool", "like clouds or something" and like a "tapestry". The articulate and generous feedback from participants has a positive impact on the creation of the installation of Mother Tongue. Feedback on the gendered imagery of voice in the display is significant and may be integrated into future work. As a language aquisition tool the spectrographic display is not very useful since the untrained user does not disambiguate formant data within the display. As a language embodiment tool the spectrograph provides valuable data partially through its enigmatic and aesthetic quality. Discussion and Conclusions The themes emerging from this research; physiological response, memories and associations and the interactive experience all indicate a shift in feeling within the participant. The answer to my research question as to whether a shift in emotion is experienced in interaction with Mother Tongue is positive. The arts-based interactive engine for this research is more central to the experience of participants 40 than I had expected. As a new media artist I welcome the opportunity to obtain this first hand testimony about the vital role that interactivity plays in creating experience. Feelings about mother tongues are vivid and willingly articulated. In the video that will accompany the installation interviewees reflect on their mother tongues. Interviewees discuss the transformation of language in their lives through merging of their native languages with dominant languages. Technology's impact on language is also reflected upon. We see how intercultural embodiment is experienced. Participants report associations with films, events and images of other cultures during verbalization of the various language prompts. These associations cause a shift in emotion and can open people's minds to other subjectivities and cultures. One participant reported having images in her mind while performing the Frisian and she said she had no knowledge whatsoever of this language and she had no idea where the images came from. The feminist strength of mother tongue is borne out from the statements by participants. Humour is associated with stories of mother tongues through made up words, hybridized phrases, pet names and secret languages. Intimacy in mother tongue is evidenced in domestic memories. Scenes in the kitchen and mealtimes predominate the narrative of these memories. Pleasure with the interactive work is 41 increased due to the context of the mother tongue. Language and mother tongues resonate on many different levels for many people. Participants attempted to mimic the voice and display of the prompt voices as closely as possible even shifting their female voices to be more male like. One woman lowered her voice to imitate the male Mandarin prompts. "My body felt heavier because I was trying to speak in a lower voice and I was less relaxed. Throughout the data analysis and interpretation we see the effort taken by participants to closely match the display and imitate the voice prompts. We see that mimesis as mediated through technology stimulates our untapped abilities to embody different states. This research and art work builds on the rich data that is to be mined when we look at the role that language prosodies play in our social and emotional realities in an interactive installation. Future Work The current Mother Tongue installation does not provide that sense of community which arouses emotion which evolves from a social context (6). Interviewees eagerly voiced samples of their own mother tongues. This finding will be incorporated into the final iteration of the installation. Technical limitations due to 42 producing this work as a student limit the programming of participants' mother tongue samples into the interactive component. Future development of this work will allow participant input into the system to support a more collaborative social environment. Individuals will leave a trace of their language in the system and others will be able to leave a trace of their own interpretation of that word sample. A metaphor of the palimpsest is evoked in this layering of traces of languages. The lines and shapes of the original words are slowly covered by traces of the repeated words until the original spectrographic shapes are mere shadows thereby mirroring the way that language changes and evolves. Including the quotes from the participants brings the voice of the participants into the research paper. Arts based research recognizes the existence of multiple forms of representation. A DVD which brings participant thoughts on their mother tongues in a section entitled Tracing Language, comments on the materiality and affect in language entitled Feeling Language, and the changing nature of language entitled Changing Language will accompany this paper. This DVD will be presented as part a two channel video with one monitor showing the video and another monitor showing the spectrographic display of the participant voices. This two channel display is shown with the projected interactive spectrograph when Mother Tongue is installed in a gallery. The research process and the art making process are intertwined with the final results being distributed primarily through the presentation of the artwork. In the end, further sets of voices enter the conversation 43 and these include the voices and perspectives of those who interact with the final installation^). 44 B i b l i o g r a p h y 1. Abley M. (2003) Spoken Here: Travels among Threatened Languages. New York: Houghton Mifflin Company. 2. Barrett L. F. (2004) Feelings or Words? Understanding the Content in Self-Report Ratings of Experienced Emotion. Journal of Personality and Social Psychology. Aug Vol 87(2) pp 266-281 3. Clark H.H. (1992) Arenas of Language Use, Chicago:University of Chicago Press 4. Coffey A., & Atkinson P. (1996). Making sense of qualitative data: Complementary research strategies. Thousand Oaks, CA: Sage Publications 5. Creswell J . W. (1998) Qualitative Inquiry and Research Design: Choosing among Five Traditions. Thousand Oaks, CA: Sage Publications 6. Czikszentmihalyi M.( 1990) Flow: The psychology of Optimal Experience. New York: Harper & Row Publishers Inc. 7. Damasio A.(2003) Looking for Spinoza: Joy, Sorrow and the Feeling Brain. Florida: Harvest Harcourt Books 8. Doucet, A., Mauthner, N. (1998), Voice, Reflexivity and Relationships in Qualitative Data Analysis, proc. Qualitative Interest Group, UGA, Georgia. retrieved(05/06/05) http://www.coe.uga.edu/quig/proceedings/Quig98 Proceedings/doucet maut hner.htm 9. Dourish P., Finley J. , Sengers P.(2004) Wright P. Reflective HCI - Towards a Critical Technical Practice, Proc.ACM Conf. Human Factors in Computing Systems CHI 2004(Austria) pp. 1727-1728. New York:ACM 10. Dourish P. (2001) Where the Action Is: The Foundations of Embodied Interaction.. Massachusetts :MIT press 11. Eisner E.W. (1991) The enlightened eye: Qualitative inquiry and the enhancement of educational practice. New York: Macmillan Press 12. Ekman P., Donaldson R.J. (1993) Voluntary smiling changes register brain activity. Psychological Science 4, pp.342-345. 13. Ekman P. (1999) Basic Emotions, In T. Dalgleish and M. Power (Eds.) Handbook of Cognition and Emotion, Sussex, UK: John Wiley & Sons, Ltd. 45 14. Gaver B., Dunne T., and Pacenti E. (1999) Cultural Probes. ACM Interactions Magazine, pp.21-29. ACM press 15. Groenewald, T.. (2004) A phenomenological research design illustrated. International Journal of Qualitative Methods, 3(1) Article 4, p.1 - 26, Sage Publications 16. Hansen M. (2000) Embodying Technesis.Technology beyond Writing. Michigan: The University of Michigan Press 17. Harris William. The Poet and the Spectograph Retrieved 06/04/07 from http://community.middleburv.edu/~harris/poet.html 18. Hayles N. Katherine. (1999) How We Became Posthuman: Virtual bodies in cybernetics, literature and informatics^ Chicago: Chicago University Press. 19. Ho-Ching W., Mankoff J. , LandayJ.A. (2003) Can you see what I hear? The Design and Evaluation of a Peripheral Sound Display for the Deaf, CHI Proceedings (Florida) Vol. 5 (1) p. 161 - 169.New York: ACM press 20. Irwin R.L. & deCosson A. (2004) A/r/tography: rendering self through arts-based living inquiry. Vancouver, Canada: Pacific Educational Press. 21. Jipson J.A. & Paley N. (1997) Daredevil research: Re-creating analytical practice. New York: Peter Lang Publishers. 22. Karahalios K., Dobson K. (2006) Chit Chat Club, installation, San Jose, USA, ISEA 2006, 23. Karahalios K. posted 04/17/07 www.enqr.uiuc.edu/archive/index.php 24. Kristeva Julia. (1984) Revolution in Poetic Language, trans. Margaret Waller, New York, Columbia University Press 25. Kvale S. (1996) Interviews: An Introduction to Qualitative Research Interviewing. Thousand Oaks, CA: Sage Publications 26. Lakoff G. Philosophy in the Flesh, retrieved 06/06/06 http://www.edge.org/3rd culture/lakoff/lakoff p4.html 27. Lakoff R.(2004) Language and Woman's Place: Text and Commentaries edited by Mary Bucholtz. New York: Oxford University Press. 28. Lang P.J. (1995) The Emotion Probe, American Psychologist 50(5), pages 372-385. 46 29. Lazzaro N. ( 2004) Why We Play Games: Four Keys to More Emotion Without Story, retrieved 07/07/07 from www.xeodesign.com 30. Levin G., Lieberman Z. (2004) In-situ speech visualization in real-time interactive installation and performance, Proc. 3rd Int'l Symposium on Non-Photorealistic Animation and Rendering, pp 7-14. New York:ACM Press 31. Marks L.U. (2002) Touch: Sensuous Theory and Multisensory Media., Minnesota:University of Minnesota Press. 32. Marshall C , Rossman G.B. (2006) Designing Qualitative Research. Thousand Oaks, CA: Sage Publications 33. Maxwell Joseph A. (1996), Qualitative Research Design: An Interactive Approach. Thousand Oaks, CA: Sage Publications. 34. Merleau-Ponty M. (1962) Phenomenology of Perception trans. Colin Smith. New York: New York Humanities Press 35. Moustakas, C. (1994). Phenomenological research methods. Thousand Oaks, CA: Sage. 36. Moustakas, C. (1990) Heuristic research: Design, methods and applications. Thousand Oaks CA:Sage Publications 37. Nakamuro, Haruki & Sugimot. (2005) The Kannon system at the Risumeikan University - Japan. International Journal of Innovative Computing, Information and Control, Vol. 1,. #3, pp 547-561. ACM press 38.0viatt Sharon, Darves C , Coulston R. (2004) Toward adaptive conversational interfaces: Modeling speech convergence with animated personas. Transactions on computer-human interaction vol: 11 lss:3 p:300-328. ACM press 39. Picard Rosalind. (1997) Affective Computing. London: MIT Press. 40. Radley A.& Taylor, D. (2002) Images of recovery: a photo-elicitation study on the hospital ward. Qualitative Health Research, Vol:13: pp.77-99 41. Rapport, F., Wainwright, P., Elwyn, G., (2005) Of the Edgelands:Broadening the Scope of Qualitative Methodology, Medical Humanities 31(1):37-42 42. Rubin H.J., Rubin I. (2005) Qualitative Interviewing: The Art of Hearing Data. Thousand Oaks, CA: Sage Publications. 47 43.Saarnivaara, M. (2003) Art as Inquiry: The Autopsy of an [Art] Experience, Qualitative Inquiry; 9; p.580. Sage Publications. 44. Spiers J.A. (2004) Tech Tips: Using Video Management/Analysis Technology in Qualitative Research, International Journal of Qualitative Methods 3(1) Retrieved 07/07/07 from http://www.ualberta.ca/~iiqum/backissues/3_1/pdf/speirsvideo.pdf 45. Sullivan, G. (2005) Art Practice as research: Inquiry in the visual arts. Thousand Oaks, CA: Sage Publishers 46. Swindells C , MacLean K., Booth K.S., Meitner M. (2006) A Case study of affect measurement tools for physical user interface design, ACM Proceedings CHI 2006 (Montreal), Vol. 137 p. 243- 250. New York:ACM press 47. Tenhaaf Nell. (2002) Perceptions of Self in Art and Intelligent Agents, In Dautenhan et al (Eds.) Socially Intelligent Agents, p. 235-242. USA: Springer 48. Thompson, C. (2006, July 31). Art practice as research: A review essay. International Journal of Education & the Arts, 7(Review 3). Retrieved 06/06/07 from http://ijea.asu.edu/v7r3/. 49. Tomkins S.S.& McCarter R. (1995) What and Where are the Primary Affects? Some evidence for a theory. In E. Virginia Demo (Ed.) Exploring Affect, The Selected writings of Silvan Tomkins (pp. 325-89) Cambridge Mass: Cambridge University Press. 50. TomkinsS.S. (1984) Affect Theory. In K.R. Scherer and P. Ekman, (Eds.) Approaches to Emotion, pages 163-195. Hillsdale, NJ: Lawrence Erlbaum 51. VanderZaag E., (2000), Talk Nice. Banff Centre Media Co-production. 48 A p p e n d i c e s A p p e n d i x A Questionnaire Before interacting with Mother Tongue display Participants will fill out a Questionnaire Language Background What is your first language? Did your mother speak a different language than you as her first language? Do you speak any other languages fluently? Which one(s) Do you speak any other language at a basic level? Which one(s)? Do you have a favourite language? What is it about this language that you like? Language Play As a child did you: pretend to speak another language with your friend/sibling make up a language with your friend/sibling Do you have any memories of these speech games? Please comment on anything that comes to mind Do you have a pet? What tone of voice do you use with your pet: Your normal voice Your pet voice 49 If you have a special voice for your pet what changes does it entail from your "normal" voice Change in pitch Change in word rate Change in loudness Graphical Display of Voice Have you ever seen your voice visually displayed? For example: levels from a microphone input karaoke interactive art installation other Your State of Mind/Affect Describe how you feel at this moment. Be conscious of your feelings in your body and your mind and try to describe them. 50 The University of British Columbia Office of Research Services Behavioural Research Ethics Board Suite 102, 6190 Agronomy Road, Vancouver, B. C. V6T 1Z3 CERTIFICATE OF APPROVAL - MINIMAL RISK PRINCIPAL INVESTIGATOR: Nancy Nisbet INSTITUTION / DEPARTMENT: UBC/Arts/Art History, Visual Art & Theory UBC BREB NUMBER: H06-03950 INSTITUTION(S) WHERE RESEARCH WILL BE CARRIED OUT: | Institution I Site | UBC Other locations where the research will N/A Point Grey Site ]e conducted: CO-INVESTIGATOR(S): N/A SPONSORING AGENCIES: N/A PROJECT TITLE: Mother Tongue: An exploration of emotion in the prosody of languages CERTIFICATE EXPIRY DATE: April 5, 2008 DOCUMENTS INCLUDED IN THIS APPROVAL: D A T E A P P R O V E D : Apr i l 5, 2007 Document Name | Version | Date | Protocol: Mother Tongue: Research Proposal Consent Forms: Mother Tongue: Consent Form Questionnaire, Questionnaire Cover Letter, Tests: Mother Tongue: Questionnaire Letter of Initial Contact: Mother Tongue- letter of initial contact version 1 February 9, 2007 3 April 4, 2007 1 February 9, 2007 2 April 4, 2007 The application for ethical review and the document(s) listed above have been reviewed and the procedures were found to be acceptable on ethical grounds for research involving human subjects. Approval is issued on behalf of the Behavioural Research Ethics Board and signed electronically by one of the following: Dr. Peter Suedfeld, Chair Appendix B U B C 51 Dr. Jim Rupert, Associate Chair Dr. Arminee Kazanjian, Associate Chair Dr. M. Judith Lynam, Associate Chair Dr. Laurie Ford, Associate Chair 52 


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items