Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

The use of a vee to facilitate knowledge outcomes from an interactive multimedia program Lee, Karen Vickie 1997

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-ubc_1997-0257.pdf [ 4.63MB ]
JSON: 831-1.0064428.json
JSON-LD: 831-1.0064428-ld.json
RDF/XML (Pretty): 831-1.0064428-rdf.xml
RDF/JSON: 831-1.0064428-rdf.json
Turtle: 831-1.0064428-turtle.txt
N-Triples: 831-1.0064428-rdf-ntriples.txt
Original Record: 831-1.0064428-source.json
Full Text

Full Text

THE USE OF A VEE TO FACILITATE KNOWLEDGE OUTCOMES FROM AN INTERACTIVE MULTIMEDIA PROGRAM By Karen Vickie Lee ARCT, The Royal Conservatory of Toronto, 1978 B.Ed., The University of British Columbia, 1984 Dipl.Ed., The University of British Columbia, 1994 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in THE FACULTY OF GRADUATE STUDIES (Department of Curriculum Studies) We accept this paper as conforming to tJjjej^ iTreTi^ standard THE UNIVERSITY OF BRITISH COLUMBIA March 1997 © Karen Vickie Lee, 1997 In p resent ing this thesis in partial fu l f i lment of the requ i remen ts fo r an advanced degree at the Universi ty o f Brit ish C o l u m b i a , I agree that t h e Library shall make it f reely available fo r re ference and s tudy. I f u r t h e r , agree that permiss ion f o r ex tens ive c o p y i n g of this thesis f o r scholar ly pu rposes may be g ran ted by the head o f m y d e p a r t m e n t or by his o r her representat ives. It is u n d e r s t o o d that c o p y i n g o r pub l i ca t i on o f this thesis for f inancial gain shall n o t be a l l o w e d w i t h o u t m y w r i t t e n permiss ion . D e p a r t m e n t o f The Univers i ty of British C o l u m b i a Vancouver , Canada Date DE-6 (2/88) Abstract With the intent of improving music pedagogy, the purpose of this research study was to adapt Gowin's Vee for educators to use to effectively evaluate knowledge outcomes from an interactive multimedia (IMM) program called The Orchestra. The Orchestra, designed for grades 4-7 children, is an IMM computer tutorial designed with the authoring tool Macromedia Director 4.0. The objectives of this study were to determine the reliability of the Vee, to determine the comparative effects of two types of uses of the Vee, and to determine the comparative effects of two groups of subjects who possessed music knowledge and who possessed little music knowledge on the knowledge outcomes from an IMM program. Sixty-two subjects participated in this research study. The sample of subjects, whose ages ranged from 18 to 45 years old, included full-time and part-time teachers and student teachers enrolled in scheduled university music education or technology education classes. This sample represented a diverse group of teachers and student teachers who either possessed some music or some technology knowledge, and also possessed an interest in exploring IMM computer tutorials. None of the subjects had prior experience with the Vee or The Orchestra program. The results indicate that subjects who used the Vee during their investigation scored significandy higher than subjects who used the Vee after their investigation of the IMM program. In terms of objective validity, the Vee may be used to evaluate knowledge outcomes from an IMM program with some caution. Subjects who possessed music knowledge think differently about music than subjects who possessed little music knowledge. Perhaps critically thinking about music is different than critically thinking musically. This suggests critical thinking in music and critical thinking about music are two disparate types of learning. Assessing knowledge outcomes from the Vee requires another form of representation besides written text format since subjects were asked to write about their reflections of the IMM program and were not reflecting musically. ii TABLE OF CONTENTS Abstract.... i i Table of Contents i i i List of Tables v List of Figures vi Acknowledgements vii Chapter I. Introduction, Purpose, and Research Questions 1 Introduction 1 Organicism 1 Constructivism 3 Computer Constructionism 4 Technology .5 Orientations 6 Learning by Creative Design 7 Interactive Mmtimedia 7 Evaluation of Interactive Multimedia 8 Media 11 Music Technology 12 Interactive Multimedia and Education 14 Evaluation Tools 15 Gowin's Vee 17 Purpose 19 Research Questions 19 n. Review of the Literature 20 Introduction 20 Search Procedures 21 Related Research 23 The Stewart Study 23 The Reeves Study 25 The Novak & Gowin Study 26 The Roth & Bowen Study 27 The Vee 28 Summary 29 HT. Methodology 31 iii Sample 31 mstrumentation 32 Data Collection Tool 32 The Teaching Tool 34 Procedures 39 Design and Analysis 40 IV. Results and Interpretations 43 Interjudge Reliabilities 43 Interpretations 45 Observations of Qualitative Aspects of the Data 48 V. Summary and Conclusions 50 Purpose and Research Questions 50 Design and Analysis 50 Results 51 Conclusions ' 52 Recommendations 52 References 54 Appendix A: The Vee 58 Appendix B: The Vee Map Profile Rating Scales 61 Appendix C: Volunteer Consent Form 64 iv LIST OF TABLES 1. Table 3.1: Pictorial Representation of the Research Design 41 2. Table 4.1: Interjudge Reliabilities of Rating Scales for Each Question 44 3. Table 4.2: Means and Standard Deviations 44 4. Table 4.3: ANOVA Summary Table 45 v LIST OF FIGURES 1. Figure 3.1: Graphical User Interface of The Orchestra 36 2. Figure 3.2: The Help Screen 37 3. Figure 3.3: Trumpet Screen 38 4. Figure 3.4: Trumpet Information 38 5. Figure 3.5: Rating Scale of Music Knowledge 40 ACKNOWLEDGEMENTS The researcher gratefully acknowledges Steve Collins, Barbara Leigh, Milton Randall, and Linda Spruston who provided valuable class time for volunteers at The University of British Columbia to participate in this research study. A heartfelt appreciation is expressed to my husband and daughter, Richard and Amber, for their love, friendship, and unending patience during my pursuit of academic discovery and enlightenment; and to my parents, Pearl and Ron for their endless guidance, fortitude and encouragement in all my endeavors. Thanks to my other committee members, Dr. Robert Walker who continues to intellectually challenge me, and Dr. Marv Westrom for his gentle prodding, insightful discussions, and continued support throughout this research study. A special thanks to Dr. Tony Clarke for introducing me to the Vee and for encouraging my fascination with the tool. There are many special friends that assisted in the quest, to them, and to The University of British Columbia, an extended "thanks." Finally, I would like to express immense gratitude to my advisor and mentor, Dr. Peter Gouzouasis, for his salient advice, infinite patience, inspirational wisdom, and alacrity with which he handled my queries. In addition, I am grateful for his wealth of knowledge in music, music education, music psychology, educational research, and philosophical inquiry that has cultivated my scholarly pursuits in academic research. vii Chapter One Introduction, Purpose, and Research Questions Introduction The term senus communis in Cicero's time meant that all the senses, such as seeing, hearing, tasting, smelling, and touch, were translated equally into each other. It was the Latin definition of man in a healthy natural state, when physical and psychic energy were constant and distributed in a balanced way to all sense areas.... Western man thinks with only one part of his brain and starves the rest of it By neglecting ear culture, which is too diffuse for the categorical hierarchies of the left side of the brain, he has locked himself into a position where only linear conceptualization is acceptable.... The constraints of Western logic are tied to our sense of sequential relationships-logic made visual. (McLuhan & Powers, 1989, p.37-39) The senses are spontaneously involved with activity and change that are in continuous transition from one state to another. These transitory states involve the organization of external stimuli that become necessary catalysts for change. This change facilitates further inquiry about knowledge. Thus, knowledge may be actively constructed by the changing minds of people who actively engage in making external artifacts which they reflect upon and share states with others. People construct and reconstruct knowledge in the context of organizing mirinsically meaningful artifacts from their experiences in the world. Constructing knowledge becomes a strategy for educators and researchers as they reflect on the directions for educational reform. One of the main tenets of educational reform is that technology will revolutionize learning by organizing knowledge in unorthodox contexts for learning. Organicism One of the fundamental tenets of organicism entails the representation of objects and l events as ultimately active and changing. Every actual event in the world is a more or less concealed organic process (Pepper, 1942, p.281). An organicist suggests that the progress of knowledge is in the direction of greater and greater inclusiveness, determinativeness, and organicity in the sense defined (Pepper, 1942, p.300). Organicism involves a continuous integrative process always changing in the direction of further integration (Pepper, 1942, p. 291). thus, change is interpreted as integral to development (Gouzouasis, 1993). Overton and Reese (1984) have elaborated on necessary organization in their analysis of the Pepper's organismic research program. In Overton's model of organicism, activity and change are viewed as necessary, free of causal events, and natural to the entity being considered. This is in opposition to the mechanistic research program where activity and change are viewed as accidental and caused by fortuitous or contingent events rooted in physicalistic, materialistic, environmental, and social experience (Overton & Reese, 1984, p.204). Organicism involves the primary task of explanation to entail the discovery of the form, organization, or pattern of activity and change of the entity being considered. By establishing the order of organizational change, the formal explanation of development can be derived. The organismic model asserts that change and development are necessary, intrinsic, irreversible, and directional rather than random. The epistemological assumption is that the knower actively constructs the known (Overton & Reese, 1984). The theme of necessary organization, first elaborated by Plato and Aristotle, was later developed by rationalist philosophers like Kant (Overton & Reese, 1984, p.204). Constructivism, founded on Kantian beliefs, claims reality is constructed by the knower based upon mental activity (Jonassen, 1991, p. 10). Humans are inherently active. Humans, not the environment, are the source of acts (Overton & Reese, 1984, p.206). Since humans are inherently active, knowledge construction may be represented in terms of forms and patterns of activity. An individual constructs knowledge based on prior experience, mental structures, and beliefs used to interpret objects and events. Though constructivism does not preclude the existence of an external reality, it merely outlines that individuals construct their own reality through interpreting perceptual experiences of the external world. Collectively, from an organismic perspective, this is the foundation of the constructivist world view (Gouzouasis, 1996). 2 Constructivism The notion of constructivism arose from comments of the eighteenth century Neapolitan philosopher Giambattista Vico. The term constructivism has appeared amongst philosophers, psychologists, and social science methodologists. As a general descriptor for a loosely coupled family of methodological and philosophical persuasion, the term is best regarded as a sensitizing concept (Blumer, 1954). Constructivist proponents share the goal of understanding the complex world of lived experience from the point of view of those who live it. The world of lived reality and situation-specific meanings that constitute the general object of investigation is thought to be constructed by social actors (Schwandt, 1994). Schwandt (1994) states constructivists as being committed to, "the view that what we take to be objective knowledge and truth is the result of perspective. Knowledge and truth are created, not discovered by mind" (p. 125). Ernst von Glasersfeld (1989, 1991) defines 'radical constructivism' where the mind is an active creator and manipulator of symbols concerned with the nature of knowledge and what it means to know. In von Glasersfeld's view, knowledge is not a particular kind of product (i.e., a representation) that exists independent of the knower, but an activity or process (Schwandt, 1994). He believes this process is best understood in Piagetian terms of adaptation and equilibration (von Glasersfeld, 1989,1991). A knowledge claim is thought to be valid if it is viable or if it provides functional fit, that is, if it works to achieve a goal. Thus, the relationship between knowledge and reality is instrumental, not verificative: To know is "to possess ways and means of acting and thinking that allows one to attain the goals one happens to have chosen" (von Glasersfeld, 1991, p. 16). Schwandt (1994) contrasts radical constructivism from social constructivism. To Schwandt, social constructivism involves the process of knowledge construction but with the attention being outward to the world of intersubjectively shared, social constructions of meaning and knowledge. Kenneth Gergen (1985), acknowledging a debt to the phenomenology of Berger and Schutz, labels this process 'social constructionism' because it more adequately reflects the notion that the world people create in the process of social 3 exchange is a reality sui generis or unique (Schwandt, 1994). The social constructionist approach is predicated on the assumption "the terms by which the world is understood are social artifacts, products of historically situated interchanges among people" (Gergen, 1985, p.267). In an educational context, the role of teaching and other media shifts from one that seeks to maximize the communication of fixed content and/or skills to one in which students are led to experience the knowledge construction process: to construct interpretations, appreciate multiple perspectives, develop and defend their own positions while recognizing other views, and become aware of and able to manipulate the knowledge construction process itself (Knuth & Cunningham, 1993). Computer Constructionism Seymour Papert (1993) has a different interpretation of constructivism which he calls constructionist learning. In a computer context, it refers to how children create programs that peers can manipulate to create their own paths to learning. Papert (1993) argues for an epistemological reversal to more concrete ways of knowing. The word concrete refers to hands-on physical learning (Piaget, 1952). To a Piagetian, "to know is to invent." This means personalizing knowledge, using what one already knows to acquire new knowledge by constructing new internal representations (Druin, 1996, p.42). The computer is a vehicle for Piagetian learning by which children are encouraged to integrate new concepts into their existing repertoires as they manipulate objects defined as figures on an interactive computer display (McNeil, p. 61,1990). Learners are more likely to make new ideas when they are actively engaged in constructing external artifacts. Thus, constructionism involves two intertwined types of construction: 1) the construction of knowledge, and 2) in the context of building personally meaningful artifacts (Kafai & Resnick, 1996, p.l). Some ideologists believe instructionism, a didactic, textbook and worksheet approach to education that usually involves a test-teach-test model, is the road to learning through improved instruction. Thus, if schools are imperfect, then teachers must teach better. In opposition, constructionism denies this possibility of schools improving through better 4 teaching instruction. It does not dismiss the value of instruction but emphasizes the imperative strategies against instructional teaching. Papert (1993) believes teaching should produce the most learning from concrete objects from the least amount of instructional teaching. This behavior cannot be reductionistic in terms of reducing quantity of teaching while leaving other parameters unchanged. The necessary change parallels an African proverb: If a man is hungry you can give him a fish, but it is better to give him a line and teach him to catch fish himself (Papert, 1993, p. 139). Papert's constructionism evolves on the assumption that children succeed by fishing for knowledge they find necessary. Technology Technology may be a catalyst for change. Change seems inevitable with technology at the forefront of imposing newly formed constructs in education. Technology provides a context for experiencing purposeful, meaningful environments where learning can involve higher levels of reasoning not at the end of instruction but as the means to a purposeful end. Technology has become a medium for changing structures, learning environments, and cognitive models of learning. At me Center for Life Long Learning and Design, Eden, Eisenberg, Fischer, & Repenning (1996) "employ the notion of design as a vehicle to engage learners in personally interesting and meaningful activities" (p.40). The objective is not to convey narrow skills or specific information but to facilitate evocative learning situations. Learning situations should not be geared solely to train skills at high speed, but should also allow learners to develop a true passion for their subject resulting from the solution of self-selected problems (Eden, Eisenberg, Fischer & Repenning, 1996, p.40). The impact of computers in education has been minimal for a couple of reasons. First, until recently, technology offered limited computational power (Soloway, 1996). It has only been within the last two to three years that the current generation of CPU's (Intel Pentium, Motorola PowerPC) have offered computing power with functionality. Second, the dominant theory of education focused on knowledge transmission with technology being used as a tool to deliver information (Skinner, 1958). Miller and Seller (1990) outline the transmission orientation to reflect an atomist's world view, where knowledge is broken down into distinct, 5 separate elements. The educational aim was to have students master key skills by experiencing learning in a passive, structured situation. On the other hand, mastery learning is an integrated system of instruction that includes not only procedures for identifying desired learning outcomes and procedures for evaluation, but also (and most important), the process of instruction that will enhance student learning of those outcomes (Ryan & Schmidt, 1979, p.17). Orientations Computers are used to enhance many different orientations of teaching depending on the goals and purposes of the curriculum. At the onset of technology, computers were used to enhance the transmission orientation (Miller & Seller, 1990). This instructional orientation was equated with the limited computational power available. The implicit goal of many instructional strategies espoused by instructional designers appeared to supplant thinking rather than engage or enhance it (Salomon, 1979). Learning strategies of open-ended teaching required more flexibility on the part of the computer (Gardner, 1991; Papert, 1993; Collins, Brown, Newman, 1989). Computation power from high-performance computing and communications now involve three and four digit MIP (millions of instruction per second) machines and high-bandwidth communications which provide a quantum leap in what can be accomplished. With new computational power available, computers are able to advance learning strategies and provide open-ended learning environments for children to learn. Dewey (1916) argued that individuals learn best by engaging in authentic tasks. Miller and Seller (1990) characterize the transaction orientation as to having philosophical roots in experimental pragmatism, particularly in the work of Dewey. According to Dewey, intelligence is developed through the individual's interaction with the social environment, particularly through solving problems (Miller & Seller, 1990, p.65). Learning, in Dewey's view, is not dictated by the teacher, rather, the teacher first attempts to help the students identify problems and then acts as a resource (Miller & Seller, 1990, p.66). Learning becomes an interactive process with understanding and knowledge actively constructed by the learner. 6 Learning Bv Creative Design Learning in an interactive environment can involve creative design opportunities. Learning by design has become more common since the early 1990s. Harel (1991) performed an informal poll to research why computer educators changed from drill-and-practice models to programming languages like Basic, Pascal, or Logo. He discovered that educational researchers believe accessibility to newer software and hardware technologies has facilitated the shift in learning models. Researchers referred to the Macintosh platform and HyperCard authoring tool as the main reason for change. An authoring tool allows designers the opportunity to create their own presentations and interactive experiences (Druin, 1996). They felt these technologies offered classroom teachers and children an easier, quicker path to creating more 'media-rich' classrooms (Harel, 1991). Druin conducted another poll four years later to confirm the paradigm shift. Teachers more frequently combined commercial multimedia software and multimedia authoring tools. Why this shift in 1995? Teachers "now have more Macintoshes in their classrooms," they have "... authoring software that runs on an Apple Ilgs," and they "... feel comfortable with the multimedia tools" (Druin 1995). Interactive Multimedia Engagement in the learning environment implies choice of tasks and goals are user-controlled with programs that are simultaneously self-directed and supportive. Generally, interactivity involves people using various media to communicate art, ideas, and knowledge. The core of effective interactive communication is still a strong message and a clear presentation. Creating with interactive multimedia authoring tools (IMM) requires research, creative design and skillful execution. A crucial variable involved in design is audience choice. Miller and Seller (1990) outline student control to be integral to the transactional orientation. Opportunities to explore and problem solve encourages students to develop new cognitive frameworks. The teacher must allow the child to control a portion of the learning process in order to complete these tasks (Miller & Seller, 1990, p.111). Thus, the learner will actively participate in the construction of meaningful knowledge. Choice in interactive design is 7 necessary for a greater commitment to planning and evaluating IMM. IMM programs can create a transformative learning environment where students may explore synchronous and asynchronous concepts. Miller and Seller (1990) outline the transformation orientation to be rooted in an ecological paradigm that acknowledges the interdependence of phenomena (p. 167). Learning focuses on the integration of the physical, cognitive, affective, and spiritual dimensions; the curriculum tends to center around learning experiences that focus on interdisciplinary activities (Miller & Seller, 1990, p. 167). The transformation orientation asserts students to have as much control as possible over their own learning with knowledge viewed more as process than content (Miller & Seller, 1990). With IMM authoring tools, children are not limited to prepackaged multimedia experiences that someone else designs; they may create their own (Druin, 1996, p. 120). The computer may be considered an expressive medium for children to explore ideas and subject matter through paths that they choose (Druin, 1996, p. 120). As Seymour Papert stated, "the best learning takes place when the learner takes charge" (Papert, 1980). IMM authoring tools have become a platform for communicating and enhancing learning. One may argue that the ultimate aim of IMM is to create a learning environment with a plethora of audio and visual tools in a way that is conducive to learning and long-term memory. Evidence is accumulating from industry and educational institutions that suggest multimedia can indeed transform student learning (Kahn, 1992; Janson, 1992). Bayard-White (1990) quoted (British Audio Visual Society) that people retain 10% of what we read, 20% of what we hear, 30% of what we see, 50% of what we see and hear, 80% of what we say and 90% of what we say and do at the same time (Bayard-White, 1990, p.4). Bayard-White (1990) also argued the essential features of multimedia to be the compelling power of pictures and the television combined with the functionality of computers, which place the control at the hands of the user. Not only are a number of senses brought into play but the user directs the interaction (Hapeshi & Jones, 1992, p.80). Evaluation of Interactive Multimedia To assess whether rMM contexts can provide an educational rich environment for 8 meaningful learning, it is necessary to determine a reliable way to evaluate knowledge outcomes in a computer environment. What is lacking throughout computer learning environments is a reliable way to determine if knowledge has been constructed. Draper suggests we need instruments to measure what is active in terms of both affect and salience (Draper, 1994. p.34). He suggests evaluation tools to assess what influences knowledge, produces the change in thinking, and to determine what is actively prominent when learners are acquiring knowledge. Novak and Gowin (1984) discuss the use of metacognitive tools to help people think about minking while learning how to learn. The term metacognition has been used to describe our knowledge about how we perceive, remember, think, and act - that is, what we know about what we know (Metcalfe & Shimamura, 1994, p.xi). The term was originally developed to characterize change in self-reflection during early development (Brown, 1978; Flavel & Wellman, 1977). Given the widespread problem of inappropriate learning strategies, the use of metacognitive tools can help students understand the nature and role of meaningful learning and move them towards successful patterns of meaningful learning (Novak & Iuli, 1996). One important outcome from IMM programs is to determine whether learners have obtained knowledge from the use of the program. Though the "medium is the message" and will deliver a message, the message should reflect educational value on the part of the learner. Though there are numerous IMM programs developed for learners, programs are imperfect. What appears necessary is having a strategic tool for learners to use while investigating IMM programs. The use of a metacognitive tool allows the learner to personally construct, individually situate, and contextually embed meaningful knowledge. If the learner has acquired knowledge, then the IMM program might be considered influential to the learning process. Educational programs should provide learners with the basis for understanding why and how new knowledge is related to what they already know and give them the affective assurance that they have the capability to use this new knowledge in new contexts (Novak & Gowin, 1995, p. xi). The educational value of any 'object' (test, text, book, lecture, experiment, lab manual, educational event) resides in how well it can help us to realize our power to understand the world we inhabit: it should transform the meaning of our experience so that both we and our world have enriched meaning (Novak & Gowin, 1995, p. 110). 9 IMM authoring tools have been widely used to build on the constructionist theory of learning (Papert, 1993). However, there are concerns about using IMM authoring tools in classroom contexts. An authoring tool allows children to create programs that peers can manipulate to create their own paths to learning. For example, some students create 'buttons' for the sake of learning a computer program while other students create 'buttons' to enhance the animation of cell growth. The difference can be compared to rote memorization of a list of spelling words for a quiz as opposed to learning to spell the words whilst creating a story. This is a common difficulty in any environment that gives children and teachers an open-ended tool such as a programming language like LOGO (Carver 1987; Papert 1985). This difficulty may be overcomed if one assesses both the process and the product from computer constructionist learning environments. The programming language LOGO was developed for computer designers to create something concrete with the computer. LOGO is an environment and a culture made of ideas, things, and people. Not only does this include the computer and the language, but computer-controlled devices like turtles, LEGO motors and lights, as well as integrated facilities to express yourself in words, sounds, pictures, and animation (Druin, 1996, p.l 1). Children can construct their own learning environments. When ideas and actions from computer science intertwine with anthropomorphic thinking, they can become lively tools in a process of dynamic problem solving. This process of trying out ideas, observing, making changes, rethinking, and reconfiguring enhances the natural learning process as children "learn by doing and thinking about what they do" (Papert, 1980). Since the late 1970's, research has integrated programming and design in educational environments for children to support children's constructonist learning (Papert 1980; Druin, 1996; Resnick, 1994). If IMM programs are created with open-ended tools like LOGO, alongside the many different media to present knowledge, IMM programs can become active, stimulating environments for students to engage in. What needs to be determined is whether or not IMM programs contain educational value. The assumption of educational value from IMM programs merits concern, because an IMM program designed for children to construct their own learning environment does not automatically guarantee knowledge acquisition. Papert (1985) outlines a metaphor to create quality leaning environments for children. 10 "Does good wood produce good houses? If I built a house out of wood, and it fell down would this show that wood does not produce good houses? Everyone realizes it is carpenters who use saws, wood, and hammers to produce houses...and the quality of the product depends on their work." An integration of good wood and good carpenters are required to construct enhanced educational environments. IMM is proposed to promote independence, academic skills and teacher usability, while providing an entertaining learning experience for students (Poirier, 1994). However, not all IMM programs will contain 'good wood' for students to acquire knowledge as teachers, parents, product designers, and children will be different carpenters. IMM programs require quality media, skillful presentation, and effective communication to be 'good wood' to facilitate knowledge acquisition. Media Of greatest importance to education is the notion that learning environments are affected by technological innovations in communications media (Gouzouasis, 1995, p. 16). Communications media involve the use of both acoustic and visual media. From the perspective that media are modes of communication that have numerous effects on both conscious and unconscious processes of perception and conception, it is challenging to imagine the cognitive effects of media on both linguistic and musical information acquisition on our lives (Gouzouasis, 1995, p. 15). Children and musicians are living in the context of the Information Age where music education has been strongly influenced by the technological environment of the 20th century. Technology has revolutionized the 20th century where language incorporates such precepts as 'global village' and 'age of information' (McLuhan, 1994). The Delphic aphorism, we are moving out of the age of the visual into the age of the aural and tactile, has merit (Lapham, 1994, p.x). Thus, James Joyce's remark in Finnegans Wake, "what bird has done yesterday, man may do next year" is an observation on the course of technology (McLuhan, 1994). McLuhan's (1994) brilliant and sometimes opaque thoughts about electronic media presented as nonlinear, discontinuous, and intuitive, has enormous impact on education. Protocols of media environments include insurgent technologies that give rise to structures of l l feeling and thought. In the realm of technology, humans are involved in learning and knowing, therefore, if forms of employment can be considered 'paid learning,' then forms of wealth emanate from movement of information. Bork (1987) believes "computers are going to become the dominant delivery system for education. That is, more people will learn, in the future, more things from computers than any other way of learning" (p.3). Music culture has become a phenomenon of mass media. McLuhan stated the content of any medium is always another medium and that, "we become what we behold, that we shape our tools, and thereafter our tools shape us" (Lapham, 1994, p.xi). Media become tools that shape our perception of effective communication. Communication that involves electronic media may occur in the world of cyberspace. The term cyberspace was coined by science-fiction writer William Gibson in his novel Neuromancer and defined as "consensual hallucination" (Appignanesi & Garratt, 1995, p. 128). Cyberspace refers to the perceived reality or space generated by computer software within a computer that simulates a computer mediated, multisensory experience designed to involve our senses in another perceived reality known as virtual reality. This artificial infrastructure, connects millions of users throughout the world in a space where there are a vast amount of information, forums, and discussion groups. The term cyber signifies computer dominance and disembodied experience. Music Technology Cyberspace involves technomusicians where some musicians are 'modemming their parts' through the access of ADSL (asymmetric digital subscriber lines) and ISDN (integrated services digital network) telecommunication line interfaces which do not appear to relate to school music and school music sounds "do not exist outside school" (Walker,1990, p.220). These musical parts are digitized computer files sent to other people over sophisticated telecommunication lines. Walker (1990) believes, "the school experience is likely to be quickly discarded as unrelated to the real world of musical sound" (p.220). Some children are replete by integral awareness of synthesizers, MIDI (musical instrument digital interface), CAI (computer assisted instruction), CD Rom, and the Internet. One only needs to visit a local 12 technology store to see electronic devices and 'books for dummies' that indicate the need for a paradigm shift in music education. While some students are playing ukuleles and recorders with visual, mechanical, sequenced compositions in music classrooms, the same students enter the outside world of tactile, organic, simultaneous improvisation. The former involves transfer of knowledge while the latter involves reconstruction of knowledge. The major difference involves process and argues for reconstructing the tools used for music education. Outside 'wired,' music experiences are demanding the abandonment of inside archaic music classrooms. The notion of being 'wired' often refers to individuals involved with the use of technology. Media constantly evolve and change, and all forms of music are profoundly influenced by various levels of media (Gouzouasis, 1996, p. 17). Gouzouasis notices that academic music professions have been the most apprehensive to validate how phenomena in the evolution of digitized music has become fully manipulatable media. Thus, digitized music can become music knowledge actively manipulated and constructed by the learner. Sound designers believe sound to account for more than half of the experience of using an interactive product (Kristof & Satran, 1995, p.112). Kristof & Satran (1995) further suggest sounds to influence the tone of an interactive product and tremendously influence the users' perceptions of the content. Thus, technomusicians appear to have a salient role in designing and developing IMM programs, and the construction of digitized music and sound effects has become a crucial component in IMM programs. The most pervasive technology of the Information Age involves IMM programs, interactive multimedia (IMM) is defined as a computerized database that allows users to access information in multiple forms, including text, graphics, video and audio (Reeves, 1992, p.47). With IMM authoring tools, a designer can integrate media in a variety of ways. This can include linear 'slide show' presentations, non-linear interactive presentations, branching excursions for side trips to related topics, or launch and return options where students can launch other applications from inside the multimedia program. Computers are now being used for interactive activities that were once noninteractive. This involves a Copernican-like revolution because the program now involves the audience, not the designer, who empowers the learning environment. 13 Interactive Multimedia and Education Multimedia environments are revolutionizing all areas of education by suggesting new perceptions of change, integration, and concept of knowledge. Giardina (1992) states multimedia environments should become the communication medium of the pedagogical intentions of the professor/designer where the learner tests, reflects, and accesses information while trying to interpret, manipulate, and build new knowledge (Giardina, 1992, p.v). Multimedia technology can bring a new sense of 'reality' into the classroom, thus transforming the educational arena and instructional environment (Stewart, 1994, p.2). The old, static equilibrium, in which fixed roles by the teacher and the teaching environment, and the learner, is shifting to a dynamic equilibrium where the nature of information and its processing change, depending on the situation, the learning context and the individual's needs (Giardina, 1992, p.v). The power of the computer lies in its ability to employ a wide range of symbols and to operate on symbolic expressions in powerful ways for these are the capabilities that most closely correspond to human cognitive and social behavior (Kozma & Johnston, 1991, p.12). Giardina (1989,1991) defines interactivity as the integration of physical and cognitive dimensions which become consequences of the different possible contacts shared by the learner and the system when exploring through a network of available information. Interactivity encourages the exchange of relevant information and knowledge between the interactive multimedia learning environment and the learner (Giardina, 1992, p.v). One of the most famous examples of highly interactive learning was 2500 years ago, the method Socrates used in working with students, as least as reported by Plato in the Dialogs (Bork, 1992, p. 7). Though these literary works are different than actual learning contexts, one may hypothesize the concept of interactivity began with 'Socratic' techniques. The essence of interaction in Socratic dialog is that Socrates proceeded entirely by asking questions and not by lecturing. The purpose of Socratic questioning was to stimulate active thinking which would encourage students to discover ideas. It is generally believed that IMM programs encourage learners to question the ideas outlined in the active areas of the interface. By questioning these areas of the interface, learners 14 can actively think while navigating through a large number of different paths in the program. Thus, IMM programs appear to actively engage the mind of learners. Though IMM programs encourage active learning, it is still unclear whether there are educational benefits from the use of IMM programs. Research indicates there are 'no significant differences' between multimedia classrooms and traditional instructional classrooms in terms of achievement (Reeves, 1987). Feedback surveys, interviews, and questionnaires rely on memory and are less valuable than on the spot observations. The longer after the event feedback is provided, the less valuable the feedback suggestions are (Draper, 1996). Draper argues for evaluation with some open-ended questions and personal observations by the evaluator, as well as fixed measures for generating comparative data that can answer specific questions. The root of the term "evaluate" suggests that the function of evaluation is to place a value on the thing being appraised (Lincoln & Guba, 1980, p.61). IMM programs are ubiquitous. IMM programs can be prepackaged commercial software packages programs or programs designed by either teachers or students with authoring tools. Educators need to evaluate which IMM programs and authoring tools have educational value for use in classroom contexts. For example, authoring tools tend to be used to design educational environments with lack of knowledge as to how to best enhance the educational arena. The dynamics between knowledge absorption and the creation of new knowledge needs further investigation. Educators considering IMM programs for classroom contexts are challenged to determine whether students are to absorb knowledge or create new knowledge from IMM programs. Most IMM programs have factual knowledge which educators can test to determine student retention. However, with authoring tools offering open-ended learning environments for student directed learning, consideration to open ended evaluation tools might seem to allow flexibility on the part of the learner to obtain knowledge compatible with his learning needs. Evaluation Tools Research and evaluation should play a critical role in the development of IMM programs. There are many claims about the potential use of integrating IMM technology and 15 its ability to provide opportunities for different modes of learning. IMM programs can be created to serve the purpose and context for which educators can implement to enhance, reinforce, and supplement their curriculum. Up to this point, however, it seems easier to evaluate the materials than to evaluate the learning. While IMM may be technically exciting, it does not automatically lead to more learning or better educational programs. The development of IMM programs involves the use of reliable evaluation tools. Both the iterative process, which involves knowledge, methods, and tools drawn upon as needed (Preece et al, 1994, p.47), and formative evaluation, which occurs throughout the interface development process of the program, (Carroll, Singley, & Rosson, 1992; Dick & Carey, 1978; Scriven, 1967; Williges, 1984), will engage further educational inquiry about the effectiveness of IMM programs. Formative evaluation is considered most important for IMM program development. Scriven (1967) has found formative evaluation to be directed primarily at the improvement of learning while instruction is ongoing, rather than at grading, marking, or ranking, forms of evaluation characterized as 'summative.' Changes to the software product are financially less costly to make, no matter where the product is in the development process (Hix & Hartson 1993; Hourvitz 1994). The computer, especially multimedia workstations, have the potential to enhance learning by creating a more realistic and appealing presentation of virtually any subject matter (Stewart, 1994). Yet, there has been little research with regard to the development of evaluation tools for IMM authoring tools. Educational evaluation of IMM programs is paramount in the assessment of whether these tools can facilitate knowledge outcomes. That educational evaluation concerns the value of education is a point often misunderstood in standard methods of testing, measuring, and evaluating which seek to elicit information from students that exactly mirrors a text or lesson (Novak & Gowin, 1995, p. 111). With the development of more sophisticated technology, learning environments are changing and demand constant evaluation. Perhaps evaluation tools used to facilitate metalearning and metaknowledge would yield different knowledge outcomes from learners instead of using evaluation tools to evaluate achievement scores, on rote, shallow memorization. Children do not get ideas, they make ideas (Kafai & Resnick, 1996, p. 1). Thus, evaluation may involve discovering how children make ideas. Though deep learning is 16 associated with long retention of overall meaning and shallow learning with brief retention spans, deep learning could reflect how the material is structured, rather than learners having to do special work in constructing novel concepts (Draper, 1994, p.34). The quality of learning is the quality of the interaction between learner and material (Svensson, 1984, p.66). Evaluation should involve the characteristics of the sequence of interaction rather than 'how much' effect the IMM program has. Jonassen (1996) suggests evaluation from a constructivist perspective should be less of a reinforcement or behavior control tool and more of a self analysis and metacognitive tool (p.8). Constructivist learning is not supposed to mirror reality, but rather to construct meaningful interpretations (Jonassen, 1996, p.8). Thus, constructivist evaluation involves viewing the construction process of the constructor. Learning emerges from perceptual processes that evolve out of experience and involve higher-order thinking. Constructivist learning environments and tools are intended to engage learners in higher-order, more meaningful learning. Thus, traditional measures of recall cannot be used to assess the outcomes of using these environments (Jonassen, 1996). Hence, constructivist evaluation might consider how the learner collects information, makes hypotheses, and solves relevant problems. Knowledge processes emerge from the learner's interactions. One tool for the constructivist evaluation of emerging knowledge outcomes is the use of an evaluative metacognitive tool. Gowin's Vee Gowin's Vee outlines necessary elements that represent the structure of knowledge and the process of knowledge construction. The Vee outlines how twelve epistemological elements operate in the construction of new meanings for events or objects. Gowin believes every element in the Vee interacts with one another during the construction of knowledge. The Vee is a metacognitive strategy to facilitate meaningful learning by asking students to reorganize new information with knowledge they already possess. The reorganization of new information with prior knowledge involves the evaluation of the sequence of interaction between the material and the learner, thus, examining process as well as product. This metacognitive tool can evaluate 17 the characteristics of the sequence of interaction between the IMM program and the learner. Educational value may be determined by what the learners do with lessons, not by the exact fit between a lesson and its replication on a test (Novak & Gowin, 1995, p.l 11). The Vee illustrates both the complexity of the process of knowledge creation and the relative simplicity of the structure of knowledge as viewed through the heuristic (Novak & Iuli, 1995). Educational evaluation can be improved if we take cognizance of our understanding of how humans create and appraise knowledge and the psychological processes by which they come to understand knowledge (Novak & Gowin, 1995, p. 112). If a student's general educational growth is more important than mastery of specific lesson material, a test that measures general and particularly higher-order educational knowledge and skills, and assesses a student's ability to apply knowledge acquired in school, might be more valuable than one closely geared to instructional experiences (Madaus & Kellaghan, 1992, p. 132). A valid test is sedative instead of stimulative to the teacher's search for better means of instruction and more adequate conceptions of his aims (Schwab, 1967, p.278). Gowin believes the Vee to be more than an assessment of instructional goals. The Vee becomes a metacognitive tool that structures knowledge in a facilitative and reflexive learning environment. Since the goals of IMM should be nothing less than improving the conditions of teaching and learning in education (Reeves, 1992), and instructional goals are merely indicative of mastery of skills, evaluation measures must assess beyond instructional objectives. Gowin's Vee purports to facilitate meaningful learning and construction of knowledge by outlining questions that stimulate learners to be reflexive about the process of learning and about the products they are investigating. Knuth and Cunningham (1993) discuss the importance of designing tools that support thinking in learning environments. Knuth and Cunningham (1993) claim, "reflexivity concerns the ability to consciously be aware of one's own learning, belief systems, actions, and knowledge" (p. 178). The cognitive processes required to engage learning stimulates awareness of thinking skills or metacognition (Flavell, 1976). The Vee purports to stimulate meaningful knowledge by encouraging reflexive thinking. There has been no research regarding the use of Gowin's Vee for evaluating IMM programs. One potential significance of developing the Vee leads to the development of criteria 18 for assessing IMM programs for educational purposes. From these criteria, educators can determine whether an IMM program facilitates knowledge construction and meaningful learning. The other potential significance of developing the Vee encourages further reflexive practices about how constructivist assessment tools can support the development of thinking and increase the probability that educational contexts can be appropriate milieus where learners can be actively engaged in acquiring new, meaningful knowledge. Purpose With the intent of improving the evaluation of IMM, the purpose of this research is to adapt Gowin's Vee for educators to use as an evaluation tool to effectively assess knowledge outcomes from the IMM program called The Orchestra. Research Questions Specifically, the research questions of this study are as follows: What is the objective reliability of the Vee as a metacognitive tool, in its use during and after the investigation of an IMM program? Is there a difference between identified knowledge outcomes measured with the Vee in the IMM context for subjects who possess music knowledge and for those subjects who possess little music knowledge? Is there a difference between identified knowledge outcomes for those subjects who use the Vee during their investigation of an LMM program and for those subjects who use the Vee after their investigation of an IMM program, for subjects who possess music knowledge and those subjects who possess little music knowledge? 2. 19 Chapter Two Review of the Literature Introduction Educators are realizing that interactive multimedia (IMM) programs may be an important catalyst for changing how children learn. Multimedia environments offer teachers valuable tools to provide information geared to multiple learning styles. Teachers, as facilitators, can integrate IMM tutorials to enrich, reinforce or demonstrate abstract theories or techniques. There are several design methods used to create multimedia programs. Some designers create IMM programs based on a transmission mode for instructional strategies. For some teaching contexts, the transmission mode can assist in reinforcing concepts or information. IMM programs can also present opportunities for educators to alter from a transmissive teaching mode to a transformation of knowledge outcomes. The purpose of using IMM programs can vary according to the educational context. For some learners, instructional IMM programs can assist the learning of mastery taught units. According to Bloom, mastery learning can help slow learners to achieve at the same level as others if slow learners are given more time and extra help (Miller & Seller, 1990, p.52). IMM programs have the potential to assist teachers implementing instructional activities and mastery learning conditions. IMM programs can also involve a paradigm shift in learning environments. This paradigm shift involves the learner being in control not only of what is learned, but also how something is learned. Lester Thurow (1995), economist and futurist, comments that new technologies give us the opportunity, if we are up to it, to shape the art of education in the future. This literature review addresses the efficacy of evaluation tools in the assessment of IMM programs that teachers would potentially use in their classrooms. There are two issues addressed in this selected research. First, the issue of how to evaluate the material, and second, the issue of how to evaluate the knowledge they seek to understand. A major concern 20 involved with these issues is the determination of whether IMM programs help students learn about both the structure and production of knowledge. Reeves' (1987) research indicates teacher response to new technology was 'positive' and that students 'showed marked enthusiasm' for using IMM programs. Empirical studies, however, often show 'no significant difference' between multimedia classrooms and traditional instructional classrooms in terms of achievement (Reeves, 1987). An examination of research related to Gowin's Vee encouraged the rationale for the exploration of a reliable evaluation tool. Gowin's Vee focuses the evaluator in an interpretative mode of inquiry. The focus question of the Vee encourages investigative inquiry with active interplay between the conceptual and the methodological activities. The focus question elicits good reflective thinking while exploring the active interplay between what students already know and the new knowledge they seek to understand. The value of Gowin's Vee resides in helping learners understand the process in which humans produce knowledge. Search Procedures A computer search using ERIC database descriptors and dissertation abstracts yielded some research regarding evaluation tools used for computer programs. There have been survey, questionnaires and inventories for evaluating usability of interface design in computer programs (Schneiderman, 1987; SUMI, 1993; Kolb Learning Style Inventory, 1984), as well as automated recordings of the navigational patterns and density of usage of different information units and general students' satisfaction surveys. However, there has been little research with regard to the use of evaluation tools to facilitate knowledge outcomes from the context of IMM programs. Initial attempts were explored with regard to ERIC international, Music Educators Resource Base, SSCI CD-Rom and CD-Rom with old theses and dissertations. Current issues of computer magazines such as Wired, Multimedia, MacUser, MacFormat, and Mac World provided resources to education, technology and related research on IMM design and evaluation. Robert Stewart's dissertation on the development of an IMM tutorial, and Allison Druin's (1996) research on designing multimedia environments for children were 21 primary sources discovered from these searches. Several relevant and useful secondary resources on the Internet had current developments in the field of multimedia. There were web sites on multimedia, evaluation, technology, music education, research and design, and curriculum development with the aid of computers. For example, Macromedia Director has a web site and listserv that were extremely informative about IMM design. Subscribing to newsgroups has also been very resourceful. These newsgroups include:, (user interface engineering), (blurbs on interaction, technology, and society), comp.human-factors, (Educom Review), and The Internet proved to be the most valuable and efficient method to obtain current information in the field of technology. Several journals were also resourceful. The Music Educators Journal (MEJ), Teaching Music, Educational Researcher, Educational Technology, British Journal of Educational Technology, and Journal of Educational Computing Research, were other pertinent journals. Articles from the conference proceedings of the Human Factors in Computing Systems were extremely informative. Through an exhaustive search, the Vee heuristic (Novak & Gowin, 1984), was discovered to be a potentially viable tool to evaluate IMM programs. In terms of the Vee, no literature on the use of the Vee to evaluate IMM programs. Though there is some literature on the use of Vee maps for science investigations (Roth & Bowen, 1993), there is no research on the use of the Vee as an evaluation tool for IMM programs. Moreover, there is no research on the objective reliability of the Vee. Information regarding Vee mapping was discovered from science education resources. The following studies have been selected for extensive review as they provided analogical models that aided the formulation of the theory between theoretical terms and observable variables. The use of a model encouraged the postulation of new rules of correspondence and the application of a theory to new kinds of phenomena (Barbour, 1974). The resources selected for critical analysis are Stewart (1994), Reeves (1987), Novak and Gowin (1995), and Roth and Bowen (1993). 22 Related Research The Stewart Study Stewart (1994) investigated the possibility that a computer enhanced with IMM could improve student learning. He cites research that indicates multimedia has the potential to enhance learning by creating realistic and attractive presentations of any subject matter. In 1994, his research review indicated that IMM tutorials were under utilized. Stewart designed a friendly prototype example of an IMM tutorial for a non-technological curriculum to demonstrate the potential of a multimedia environment. The purpose of Stewart's research was to encourage future educators to utilize technology to the fullest and to prepare students for the changing world of technology. Analysis of formative evaluation data involved using a 'visual analog' scale to elicit a less binary and more natural and free response. This visual analog scale is similar to Likert scales except that 5 distinct points of a continuum by which to rate were altered to a 5 centimeter line anywhere along which a mark could be placed. He argues that the use of a continuum without distinct ratings points allows the respondent greater freedom in expressing a thought, feeling, or position. Stewart used a case study research design. The selection of the design was determined by the JEMM tutorial (Journalism in Education Multimedia Model). The program is particularistic, descriptive, and heuristic in that it illustrates a deeper understanding of how interactive tutorial environments impact education. The subjects included 23 desktop publishing students from San Diego County in southern California. The subjects were administered the JEMM tutorial, pre and post tests, and a standard journalism desktop publishing instrument used by the department. All subjects answered all the questions on the JEMM tests. The pre and post tests were used to investigate the effectiveness of the JEMM tutorial upon learning basic journalism and desktop publishing layout and design skills. Each pair of scores, the pre test and post test scores, were matched for one question for all 23 students who attained either the correct score or the incorrect score. At the 0.05 level of significance, Stewart tested the claim that the JEMM tutorial would 23 result in a significant gain of knowledge. The values from the pre and post test were used. The mean pretest value was 43.19%, with a standard deviation of 3.6. The mean post-test value was 60.03 with a standard deviation of 3.4. A paired t-test was performed since the populations were normally distributed. Due to each paired scores for one person, it was concluded that the values are dependent. The differences computed were the post test minus the pre test. The test statistic t=24.1 was computed with t=1.69, and 3 degrees of freedom, the null hypothesis was rejected. Stewart concluded a statistically significant difference between the pretest and post test and JEMM to be an effective model for teaching newspaper layout and design. Stewart's data collection tool raises concern. These visual analog scales would yield ambiguous results since it would be difficult to identify the scale of measurement of the variables involved. Stewart's results are highly subjective. Rating scales are most beneficial if they are continuous rating scales, in that the values of the variables chosen would have distinct, rich descriptions with lower rating values achieved before one could receive a higher rating value. Stewart did not perform correlation coefficients to determine the reliability of the visual analog scales. Stewart's study reveals how evaluation tools are the key to determining the reliability and validity of the results obtained. Even well designed objective tests correlate with subsequent achievement only in the range of r=.2 to r=.7. After squaring these correlation coefficients to obtain the common variance shared by these measures, one obtains values of only .04 to .49, therefore, one may interpret approximately 50 percent of variation in measures of achievement is predicted by any achievement measure or set of measures given within a few months to a year or two (Novak & Gowin, 1984, p.93). Measuring achievement with objective tests has been problematic (Hoffman, 1962; Gould, 1981). Stewart failed to substantiate his claim of JEMM's effectiveness by measuring achievement with subjective, unreliable visual analog scales. Stewart's knowledge claims are unsubstantiated by his analysis for several reasons. Despite the discovery of significant differences, there are several threats to internal validity when using pretest and post test design. The addition of an observation that occurs before the treatment poses a serious threat. Another serious threat is history. With a lack of comparison 24 group, the researcher cannot be assured other intervening events occurring between the pretest and post test did not cause the changes observed. Another serious constraint involves the possibility of interviewer bias. The researcher was an instructor at the college level in the disciplines of journalism. The researcher was an insider, as opposed to an outsider, to the group of participants and had status within the the social group of participants. A 'blind' test could have eliminated bias if the participants were from other instructor's classes and unaware of the purpose of the research. A 'double-blind' test where the test administrator was also unaware could also eliminate further bias. The Reeves Study Reeves (1987) criticizes media comparison and media replication research as being inadequate as a scientific foundation for instructional design for IMM. He suggests an eclectic approach to research using case studies. Grounded in pragmatic philosophy, Reeves argues to "use whatever works to improve the decisions people make" (Reeves, 1992, p.49). By using formative evaluation, decision making about IMM programs are employed from initial conceptualization through several beta versions of the program. Reeves applies Newman's (1990) idea of formative experimentation as a formative evaluation strategy for IMM. In a formative experiment, the researcher sets a pedagogical goal and finds out what it takes in terms of materials, organization, or changes in the technology to reach the goal (Newman, 1992, p.10). Reeves (1986) suggest the 'no significant differences' problem has plagued research on the effectiveness of innovation in education. He further argues this as the result of failing to describe and measure the unique dimension to characteristics of the innovation. However, no significant differences between groups could indicate that the instruments have low reliability. Reeves does not address the issue of reliability and whether the evaluation tools really measure what they claim to measure to accomplish a specified purpose with a specific group of people. Reeves pragmatist philosophy seems inconsistent with the constructivist world view to which he ascribes. While Reeves argues that knowledge, attitudes, and skills be useful as opposed to inert knowledge, inactive knowledge does not mean knowledge construction has 25 not occurred. If constructivism involves knowledge construction based on knowledge-dependent learning (Glaser, 1984), and that knowledge is a function of both the amount and quality of existing knowledge as well as one's reasoning and other intellectual abilities, knowledge construction can be inactive knowledge that could become active during appropriate contexts. Inert knowledge does not mean that knowledge construction has not occurred nor does useful knowledge indicate that any meaningful learning has occurred. The Novak & Gowin Study Gowin's Vee is a scaffold outlined as an aid to solving a problem or understanding a procedure (Novak & Gowin, 1984, p.55). The 'scaffold' is a metaphor used to describe a temporary, raised framework where necessary elements are outlined to understand the nature of knowledge and knowledge production. Since the primary purpose of educational inquiry is the construction of knowledge, this model or heuristic provides valuable insights into the research processes associated with knowledge construction. Gowin's Vee heuristic is a tool for acquiring knowledge about knowledge and how knowledge is constructed and used. This tool facilitates both metalearning and metaknowledge acquisition (Novak & Gowin, 1984, p.57). The Vee guides students through investigations that are outlined by a focus question. Novak and Gowin (1984) state that the educational value of any object (lecture, text, lab manual, experiment, books, test, educational event) resides in how well the object can help us realize our power to understand the world we inhabit. The educational value should transform the meaning of experience so both the student and their world have enriched meaning. (Novak & Gowin, 1984, p. 110). Educational value yields meanings that can be shared. These socially constructed meanings allow educators to make inferences while enhancing self-understanding. Novak and Gowin call this governance. By grasping meaning, reorganizing meaning through meaningful learning, and shared meanings, educative experience should help us control meaning so that it leads to desirable human effort. In addition to its function as a guide, the Vee heuristic helps students to organize their own individual thinking, make relevant and meaningful actions more productive and efficient, while taking self responsibility for acquiring their own knowledge. Students are able to see the 26 purpose of the materials and understand what was done. One of the most valuable results of using Gowin's Vee is that students come to know the tentative nature of scientific truths. They understand that knowledge claims are indeed claims and not conclusions. Claims are therefore understood to change as new knowledge is discovered (Gurley-Dilger, 1992, p.57). Some research has been done to develop criteria for scoring lab reports (Novak & Gowin, 1984; Gurley-Dilger, 1992; Roth & Bowen, 1993). Though the points assigned for aspects of a Vee are arbitrary, more weight has been given to the more important aspects of the Vee depending on the type of material analyzed, emphasized, skills, focus questions, and relevant principles. Novak and Gowin suggest a simple method for scoring Vees constructed from expository material. This involves assigning points from 0 to 10 for 10 questions that dissect the meaning and values. Novak and Gowin suggest more research into the effects of alternative procedures for scoring Vees. Though they claim to have found high concordance among judges' scores for Vee diagrams, (Novak & Gowin, 1984, p.72) these only appear to have face validity and are not rooted in objectivity. For example, although rating scales are used, no reliability tests have been calculated to determine the efficacy of the metacognitive tool. Novak and Gowins' research begins to fill an important gap in evaluating IMM programs with regard to the assessment of educational value. Their literature not only outlines a way to help students and educators see the meanings of learning materials, but initiates the perceptions of structure and meaning of the knowledge they seek to understand. The Vee is a strategic evaluation tool that facilitates knowledge outcomes for students and teachers to develop shared educational meanings. Emphasis can be placed on the minking side during the use of the Vee. One concern of the Vee is that it is content oriented as opposed to context oriented. To determine the effectiveness of the IMM program, the Vee needs to be modified specifically to the IMM program being evaluated. The Roth and Bowen Study Roth and Bowen (1993) claim Vee maps help a student to organize their thinking, 27 investigate more efficiently, and create guidelines for their own learning. They also suggest students feel better about themselves because they are in control of their own learning. Their research involved grades six through ten students at Appleby College, a preparatory school, working on individual science projects. Teachers evaluated their assignments by scoring Vee maps. Roth and Bowen have suggested a scoring scheme for the Vee but outline these to be arbitrary scores. They suggest a marking scheme that outlines a Vee Map Evaluation Profile by weighting the areas of concept map, data and data transformation, and claims twice as heavily as associated words and investigative activities, yielding 24 points (6,6,6,3, and 3) for a perfect map. They suggest giving a point for overall outlook and presentation, then multiplying a student's score by 4 to yield a percentage score. This chart would monitor a student's progress over a period of a term, or possibly even an entire year. Roth (1992) has used the Vee to conduct dynamic evaluation. He suggests the Vee to be an excellent tool for dynamic assessment as it emphasizes the constructive nature of knowledge. He outlines the use of Vee maps with grades 6 through 12 science students by doing independent investigations. Students frame their own questions, design their own experiments, and collect and interpret their own data. The guided questions in the investigation guide are used first to guide the students through their investigation. Later, the same questions are used to evaluate students. The reflective questions on the Vee map assist students to develop their own style of disciplined inquiry. This tool dynamically assesses students throughout their inquiry by focusing both on the process and the products of their work. Though they claim there to be a good agreement among scorers of Vee maps, the claims are not based on objective facts. Moreover, Roth and Bowen's scoring scheme seems nongeneralizable due to the flexibility of scoring the Vee. The sample of subjects from a preparatory school could also inflate results since their Vee experience could be different than other subjects without Vee experience. The Vee The Vee appears to have potential benefits if adapted to evaluate IMM programs. The evaluation of IMM programs is somewhat analogous to the investigative guide of Roth (1992) 28 where questions are asked, data is collected, and analysis and connections made to concepts. What is lacking is knowledge about whether the Vee provides a consistent measure of knowledge construction and production. To determine if the Vee is an objective facilitation tool of knowledge construction, reliability must be determined. Though establishing a tool to be objectively reliable determines a tool to be objectively valid, the tool can still be rejected on issues of subjective validity. Related literature involves research surrounding the evaluation methods of computer programs. Hedber and Alexander (1994) argue for iterative evaluation. Iteration is a way of ensuring that users can become involved in design and that different kinds of knowledge and expertise can be brought into play as needed (Preece et al, 1994, p.47). The iterative process involves knowledge, methods, and tools drawn upon as they are needed. Practitioners themselves often reveal a capacity for reflection on their intuitive knowing in the midst of action (Schon, 1991, p.viii). While academic efforts required to develop technology based learning programs is at least comparable to the effort required for research projects, academic developers find references to technology based publications in applications for tenure or promotion are rejected or given little importance because of the absence of peer review. Hedber and Alexander (1994) believe if technology is to provide effective and cost effective learning, then the role of iterative evaluation in the development and implementation phases must themselves be evaluated. Most evaluation tools for computer programs have been used to assess usability (Schneiderman 1987; SUMI (Software Usability Measurement Inventory), 1993; Kolb Learning Inventory, 1984). Usability inspection is the generic name for a set of methods based on having evaluators inspect or examine usability-related aspects of a user interface (Nielsen & Mack, 1994, p.1). Evaluation tools for IMM programs need to be developed to determine whether knowledge outcomes have occurred. With the field of IMM rapidly evolving, there is more validity and reliability issues of evaluation tools that need to be addressed. Summary All studies related to IMM programs reveal the importance of developing more effective 29 evaluation tools that would facilitate knowledge outcomes. Determining whether or not IMM is the most appropriate mode of delivery for the particular subject chosen should be done at the needs analysis stage and after a pilot has been developed (McKenna, 1995). Researchers, such as Reeves and Clark, have argued that comparative research methods using quasi-experimental paradigms cannot accurately assess the effectiveness of instructional media or technology (Latchem, 1993). Little research has been conducted on knowledge outcomes in IMM environments partly because most efforts in this field focused on development and because the field is still evolving (Kozma, 1991). The design of the present study attempts to take into consideration both the strengths and weaknesses of the studies reviewed. To achieve an accurate and comprehensive view of knowledge acquisition, questionnaires, inventories, and surveys seem unreliable tools to evaluate the construction and production of knowledge. Furthermore, the efficacy of these evaluation tools has not been comprehensively addressed. However, unlike previous attempts to evaluate IMM programs, the present study has adapted Gowin's Vee as an evaluation tool to facilitate the construction and production of metaknowledge. 30 Chapter Three Methodology Sample Sixty-two subjects participated in the study. The sample consisted of teachers and student teachers, whose ages ranged from 18 to 45 years old, included full and part-time teachers enrolled in scheduled university music or technology classes. Subjects either had some music or some technology knowledge and had an interest in exploring IMM computer tutorials. Specifically, these subjects were enrolled in Computing Studies Education 402, Computing Studies Education 450, Music Education 106, Music Education 320 and Music/Art Education 321 classes. Subjects in Music Education 106 and Music Education 320 classes were in the Teacher Education Program training to become elementary school teachers and possessed varying levels of music knowledge and technology skills. Subjects in Computing Studies Education 402, mostly in the Diploma of Education program, had enrolled in the course to acquire technology knowledge. Subjects in Computing Studies Education 450 possessed previous experience with technology and had a high level of technology knowledge. Subjects in the Music/Art Education 321 class were either music or art education majors learning to integrate technology in the arts. The criteria to determine if a subject had music knowledge was whether or not they had taken private music lessons for more or less than five years. Subjects who possessed more than five years of formal music lessons were considered subjects who possessed music knowledge while subjects who possessed less than five years of formal music lessons were considered subjects who possessed little music knowledge. Though IMM computer tutorials may be used by music teachers as enrichment activities, The Orchestra was designed for teachers who possessed little music knowledge. Many elementary school teachers are asked to teach music in their classes regardless of their knowledge of music. For this reason, it was believed teachers who possessed little music knowledge would gravitate towards integrating IMM computer tutorials in their music classes. 31 Instrumentation Data Collection Tool An adaptation of Gowin's Vee, the Vee (see Appendix A, p.59), was developed to assess knowledge acquisition. Questions under the epistemological elements oudined by the Vee specifically related to the IMM program, The Orchestra. The focus question and the question about the events and/or objects were the two main guiding questions. These two questions focused the structure of knowledge to be constructed from the sequence of interactions from the IMM program. On the right side of the Vee, the methodological side, questions were related to educational values, knowledge claims, learning and knowledge transformations, and technical claims. On the left side of the Vee, the conceptual side, questions were related to world views, theories, and concepts. These questions were structured and intended to provide a facilitative environment for learners to acquire and construct knowledge. It is hypothesized questions from the Vee would alter, enhance, refine, or reconstruct the learner's prior knowledge. The Vee encourages learners to discover the relationship between doing and knowing (Roth & Bowen, 1993, p.28), addressing the active interplay between the methodological side (right side) of the Vee and conceptual side (left side) of the Vee. The Vee consisted of the following nine questions adapted from Roth and Bowen's Vee Map (see Appendix A, p.59) and Westrom's Software Evaluation Form (1995). The questions provided a framework toward meaningful learning and encouraged the need for learners to think about their thinking in order to complete their investigation of the IMM program. This framework encouraged thinking about the characteristics of the sequence of interaction between the IMM material and the learner's prior knowledge. 1. "What is The Orchestra trying to teach me, i.e. what are the educational feature of this program?" This focus question, at the top of the Vee, places an inquiry on the events and objects while attempting to elicit reflective thinking on the part of the learner. 32 2. "How did I structure my investigation?" This question, at the root of the Vee, focused on the events and objects at the foundation of knowledge production. It is crucial learners become acutely aware of the events or objects they are experiencing and about what knowledge is to be constructed (Novak & Gowin, 1995, p.57). The following questions are on the right side, the methodological side, of the Vee. 3. "How can this new knowledge be used in new contexts?" From the observations, learners can begin to outline educational value claims based on the affective or feeling component in knowledge. Educational value claims transform the meaning of experience and provide enriched meaning about the world. Answers to questions such as, "What were your goals for this program? What age range, subject areas would you integrate this program? What principle educational value does this program have in the classroom?" (Westrom, 1995) elicits reflective thinking on the educational value of the program. It is hoped the educational values the users make about the program may assist educators in assessing whether the program can be integrated in classroom practices. By encouraging learners to think about educational value claims, they may reflect on the educational component of the program. This area of the Vee is crucial for teachers to reflect upon. 4. "What did I learn?" From this question, the learner begins to construct knowledge claims about the answer to the focus question. One may hypothesize that constructing new knowledge allows learners to understand the active interplay between new and prior knowledge. Knowledge claims can be new generalizations produced in the context of inquiry. 5. "What is the basis upon which I can claim what I learned?" Learning and knowledge transformations organize observations in forms that learners can use to report knowledge claims. 6. "What technology is required to implement this program?" Technological considerations are important when constructing knowledge about the practical application of the program. It is believed some objective knowledge about the technical requirements assists in assessing the educational value about the program (Westrom, 1995). The following questions are on the left side, the conceptual side, of the Vee. 33 7. "How does this program contribute to my imderstanding of the world? The orchestra? Music?" This question asks the learner to reflect upon the general belief system motivating and guiding the inquiry. This involves the nature and limits of knowledge. 8. "What new music knowledge relates to prior music knowledge?" This question outlines logically related sets of concepts that outline patterns of explanation. Answers will suggest theories or principles that explain the events and the knowledge claims about the events. 9. "What do I know?" This question elicits associated words about the concepts. Concepts are defined as signs or symbols signifying regularities in events and shared socially (Novak & Gowin, 1995, p.56). As Roth and Bowen (1993) have outlined, these guiding questions encourage learners to reflect about the new knowledge they are constructing. Assessment involved the use of the Vee Map Profiles to assess knowledge construction and to measure the efficacy of the Vee. The Vee Map Profiles (see Appendix B, p.62) consisted of a set of 4 point continuous rating scales which have been adapted from Novak and Gowin (1995). The Teaching Tool The Orchestra IMM program was created by the researcher for teachers who possess little music knowledge. Teachers who possess little music knowledge rely heavily on textbook instruction to transmit knowledge about orchestral instruments and the role of the orchestra in developing musicianship. For this reason, it is believed teachers who possess little music knowledge would gravitate toward the integration of The Orchestra program in their music classes. Students may be able construct music knowledge about the orchestra from an interactive multimedia environment that would involve less time than from learning from textbook textbook. The Orchestra program was developed over a period of approximately 180 hours with the multimedia authoring tool Macromedia Director 4.04. This IMM program-is simplistic, self-explanatory, and involves interactivity on the part of the learner. The program presents 34 information about orchestral instruments through multimedia resources that include text, sound, graphics, and animation. The learner can navigate through the program in a modular approach that allows for self-directed learning. The program sequence can be determined by the learner who directs the path of learning through an interactive interface. A sense of control may be developed because the program has been designed to allow the learner to establish and maintain a frame of reference, or working context, for which they can perform actions. Navigating through the program is obvious so the learner can return to a familiar state if leaving the program. Learners can stop and start again without losing their previous work or having to work through cumbersome sequences to get back to where they were. This contextual framework provides stability to encourage learners to return to the program. The visual interface is simple and consistent as learners benefit from easy accessibility and usability (see Figure 3.1). The visual simplicity of the interface is intended to promote clarity by ehminating clutter that does not contribute directly to the content of the program. The visual scheme establishes a hierarchy of tasks that promotes the prominence of certain objects and two-way interactive communication where learners experience rich presentations, interaction capabilities of the interface, and functions without ambiguities. The graphical user interface (GUI) requires the learner to operate the product by clicking on icons (small graphic images). These icons have been designed to be appealing and logical to clarify its function. In addition, the icons have been selected to help reinforce the content. For example, there are small graphic images of instruments on certain icons to indicate what information will be delivered if clicking on that icon. There are help options (see Figure 3.2), relevant information (see Figure 3.3), access to related information (see Figure 3.4), and task-specific decisions. 35 36 The Help Screen Click on any background to access the help screen. General Operation Move the cursor over a music icon and watch it change to a plus sign. The plus sign indicates information is available. $ 3 ^ : 3 ^ 3 $ ! ^ : ^ _J| Figure 3.2: The Help Screen (cropped to omit extraneous detail, greyscale version) 37 Tt urn fjzfi Figure 3.3: Trumpet Screen (cropped to omit extraneous detail, grayscale version) Trumpet # Clear, brilliant sound • A mute in the bell alters sound * Player blows and buzzes with lips Figure 3.4: Trumpet Information (cropped to omit extraneous detail, grayscale version) 38 Objects and their controls are visible and intuitive. There are some real-world representations and natural interactions to provide the interface a familiar look and feel to make the program more intuitive to learn and use. Visual representations provide cues and reminders that help learners to understand roles, relationships, and recognize the objectives of the program. The interface is designed so learners can feel confident in exploring without irreversible consequences. A sense of accomplishment occurs because there are immediate results of actions. Learners are also protected from making errors. There are visual cues, reminders, lists of choices, and other aids that are automatic upon request. Confusion and error have been eliminated as much as possible. By having an interactive interface, learners with varying interests, backgrounds, and prior knowledge, can explore the program at their own pace rather than experience a passive, predetermined program. Student directed learning emancipates the learner to actively access relevant information, construct the knowledge they seek to understand, and explore media they choose to support their learning. Thus, learners are actively involved in creating their own understandings from their experience. The Orchestra is still in the project development stage and requires feedback in the form of formative evaluation from a user audience before further modifications can be made. To the extent there is agreement between what the researcher thinks learners will learn from the IMM program and what knowledge the students have acquired with the Vee, modifications of the IMM program will be done through a formative evaluation process. Procedures A letter asking for volunteers (see Appendix C, p.65) was distributed to Music Education 106, Music Education 320, Computing Studies Education 402, Computing Studies Education 450, and Music/Art Education 321 classes. Upon receiving signed consent forms, subjects were given a short orientation in the computer lab to ensure a minimal technology learning curve. The first group of eleven subjects provided helpful feedback about the Vee. This group was given the Vee during their investigation of the The Orchestra. After fifty minutes, the 39 subjects handed in their Vees to be scored. Several minor revisions of the wording and structure about the Vee was made before further distribution. Design and Analysis A quasi-experiment group design approach was used where subjects were divided into two groups. There were two treatment groups. One group was given the Vee during their investigation of The Orchestra , while the other group was given the Vee after their investigation of The Orchestra.. The decision for whether a group was given the Vee during or after the investigation of the IMM program was random. Some of the classes were divided into several groups. Thus, the use of the Vee was altered equally within the same class with one group using the Vee during their investigation while the other group was given the Vee after their investigation of the IMM program. When the groups were small, the use of the Vee was determined by what was necessary to equal the treatment groups. All the subjects were asked to rate their level of music knowledge (see Figure 3.5). 0 - No music knowledge 1 - Played instrument in elementary school only 2 - Played instrument in secondary school only 3 - Formal music lessons for less than 5 years 4 - Formal music lessons for more than 5 years 5 - Participate (d) in performing musical group Figure 3.5: Rating Scale of Music Knowledge Subjects considered to have little music knowledge were subjects who indicated that they had less than five years of formal music lessons. Formal music lessons were defined as private music lessons outside a school setting. Subjects considered to have music knowledge were subjects who indicated that they had more than five years of formal music lessons. The A effect of the design had two dimensions. One group (Treatment 1) consisted of students that used the Vee during their investigation of the program, and the other group (Treatment 2) of students that used the Vee after their investigation of the program (see Table 3.1). The B effect of the design also had two dimensions. One group consisted of subjects who possessed music knowledge and the other group of subjects who possessed little music knowledge. There were two independent variables in this research study. The use of the Vee during or after the IMM program constitutes the A effect. The level of music knowledge constitutes the dimensions of the B effect. The dependent variables were the Vee Map Profile rating scales. A Effect Music Non Music Vee During n=17 n=15 Vee After n=14 n=16 Table 3.1: Pictorial Representation of the Research Design Subjects were given approximately 45 minutes to individually investigate The Orchestra on Macintosh computers in The MUSES Computer Laboratory for Visual Media, Acoustic Media, and Multimedia at The University of British Columbia. Subjects who used the Vee during their investigation of the IMM program were given the entire 45 minutes to view the program and answer the questions. Subjects who were administered the Vee after their investigation of the IMM program were given 20 minutes to view the IMM program and 25 minutes to answer the questions. There are several presentations of data analysis due to the three research questions. Questions two and three are written as null hypotheses. 1. What is the reliability of the Vee as a metacognitive tool during and after the investigation of an IMM program ? HQ: There will be no difference between the Vee means for subjects who possess 41 music knowledge and for subjects who possess little music knowledge. H©: There will be no difference between the Vee means for subjects who use the Vee during their investigation of the IMM program and for subjects who use the Vee after their investigation of the IMM program. For the first question, interjudge reliability coefficients for the rating scales were calculated by correlating the ratings of the two independent judges for each of the nine . questions of the Vee and the composite scores. For questions 2 and 3, the composite ratings of the two judges were organized into one 2X2 design (treatment of the use of Vee by levels of music knowledge). A two factor analysis of variance for the two-dimensional design was calculated to determine the comparative effects. Interaction effects, main effects, and simple effects were tested at the .01 level of significance. 42 Chapter Four Results and Interpretations Interjudge Reliabilities There was no attrition to the sample size, in that all the subjects who committed to using the Vee while exploring the IMM program completed the Vee. The first group of eleven subjects were given the Vee during their investigation of the IMM program. Several subjects made comments and suggestions about the construct and content validity of the Vee that seemed useful to clarify for the following groups of subjects who were to use the Vee. Thus, although the data collection process had formally begun, the Vee was immediately revised in three areas. First, several subjects were confused as to whether the question, "what theory is implied," meant there was only one theory to be identified. Thus, the question, "what theory is implied," was revised to, "what theories were implied." Second, the two evaluation rating scales, Vee and IMM program were reordered since subjects were confused about what was to be evaluated. Thus, subjects were asked to evaluate the Vee before the IMM program. Third, the space alloted for "addtional comments" was revised since subjects were confused about what the comments were to be about. This section was divided into two sections so subjects could make "additional comments about the Vee," and "additional comments about the program." The revisions were completed to improve the clarity of the Vee and to reduce the time future subjects could possibly spend questioning the researcher on aspects of the Vee. They seemed to be fruitful as the three areas of the Vee did not need further clarification with the subjects who followed. Though those three minor revisions were made to the Vee itself, the evaluation rating scales for the Vee and IMM program and the two "additional comments" sections, which were part of the dependent variables, did not modify the Vee rating scales themselves. It should be noted the first group of subjects required an additional 5 minutes to complete the activity. 43 The following are correlation coefficients calculated using all scores from all the rating scales of the Vee from the two judges. Table 4.1 Inter judge Reliabilities of Rating Scales for each Question Focus question .678 Educational Value .575 Events .583 Knowledge Claims .716 Learning Transformations .717 Technical Claims .648 Associated Words .627 Theories . .643 World View .476 Mean Coefficient .63 As can be seen in Table 4.1, the interjudge reliabilities range from moderately strong to strong in degree of relationship. The mean of the nine correlation coefficients, .63, is also of moderate strength. Thus, the reliabilites indicate there is moderate agreement between the judges. The means and standard deviations for the Vee scores are presented in Table 4.2. Table 4.2. Means and Standard Deviations Treatment 1 (During) Treatment 2(After) Totals n Mean S.D. n Mean S.D. n Mean Music 17 50.41 10.25 14 46.57 11.58 31 48.49 Non-Music 15 55.53 9.58 16 40.81 6.61 31 48.17 Totals 32 52.97 30 43.69 62 48.33 44 The ANOVA summary for the Vee scores is presented in Table 4.3. It can be seen in Table 4.3 that there is no significant interaction. For all effects, a calculated F > 7.08 is required at the .01 level of significance to be statistically significant. Thus, there is a significant A effect (treatment, during/after), which indicates there is a real difference between the mean scores of the treatment groups. It can be seen that the obtained value of 14.42 lies beyond the critical area of .01. Thus, the null hypothesis, for research question #3, that there will be no difference between the Vee means for students who use the Vee during their investigation of the IMM program and for students who use the Vee after their investigation of the IMM program, may be rejected at the .99 level of confidence. Table 4.3. ANOVA Summary Table Source Sum of Square df Mean Square FTest Treatment (during/after) 1334.58 1 1334.58 14.42* Level (Music/Non-Music) 8.5.3 1 8.53 0.09 Treatment x Level 456.35 1 456.35 4.93 Within Error 5367.72 58 92.55 *.oiF = 7.08 Interpretations Though Judge Two, the research assistant, was instructed in the use of the Vee, she used a linear-parts approach to score it. That is, she marked question one for all sixty-two subjects, then procedured to score question two for all sixty-two subjects, etc. In contrast, Judge One used a wholistic technique, in which she scored all the questions for each subject, one Vee at a time. Since the judges had different perspectives of the questions, it is possible these disparate approaches in scoring the Vee deflated the correlation coefficients, thus affecting the level of agreement between judges. In perspective, Gowin's research has provided litde information on scoring procedures and the reliabilities of different scoring techniques. Though Roth and Bowen, (1993) claim they have found, in general, a good agreement between scorers of the Vee, they also claim there are so many alternatives existing 45 for constructing concept maps on a given topic that researcher and teachers must remain flexible in scoring to do justice to a student's learning style and way of expression (p.32). If that is the case, then there may be much subjectivity and little consistency, hence low reliabilities in current Vee assessment practices. Regardless, it seems reasonable to suggest there needs to be further research to determine whether scoring techniques can make a difference in the reliability of the Vee. Another aspect to consider of the scoring procedure is that when Judge One, the researcher scored the Vee, she interpreted consistency within the Vee to mean consistency of exact, literal text inscribed by the subjects. One major focus of rating the Vee was to determine whether the subjects were consistent among their answers in consideration of the various elements of the Vee. For example, when a subject believed the main educational value of the IMM program was to identify the different instrumental sounds within the orchestra, and believed the major knowledge claim to be learning about the clarinet, Judge One interpreted this to be inconsistent for lack of mentioning the concept of sound, while Judge Two interpreted this to be consistent. Perhaps these different interpretations outlines how difficult it can be to assess knowledge acquisition of aural music content from a written text context. Though a higher correlation coefficient would indicate stronger interjudge reliability, it would only be an indicator of the objective validity of the Vee. Since knowledge was personally constructed and contextually embedded by all sixty-two subjects, interpreting knowledge acquisition by written text only would seem to delimit the Vee. Perhaps this suggests the need for other ways of assessing knowledge construction from the Vee. While the rating scales may be one empirical measure of knowledge construction from the Vee, the diverse interpretations between the two judges suggest the need for further research on how to assess the construct validity, that is, to determine whether Gowin's rating scales for the Vee can measure knowledge acquisition. Overall, further research on other forms of representing knowledge from the Vee would determine the efficacy knowledge acquisition. Under each element of the Vee, the final question involved an open ended, higher order thinking question. For example, "what further knowledge could the program suggest, what other educational values could the program suggest?" As learners mature in a discipline, their relevant knowledge becomes more and more complex, and therefore, meaningful learning 46 involves seeking integration with a larger framework of concepts and propositions (Novak & Iuli, 1995). It was hypothesized that these final questions within each section of the Vee would encourage subjects to create further questions after the meanings of prior concepts and propositions might be altered by the incorporation of new concepts and propositions. It was hoped these questions would encourage the proclivity of the participant to seek out new integrative, productive thinking and develop suggestions by critically reflect upon each element of the Vee. It seemed subjects who possessed music knowledge tended to answer the higher order questions with responses like "I already knew everything, I did not learn anything." Often, subjects who possessed music knowledge would either leave the higher order questions blank or write "nothing." Rainbow and Froehlich (1987) state "it is probably safe to say that the professional training of a musician focuses far more on the development of musical-technical skills than on the skills necessary for critical thinking and reasoning (p.25)". While this statement may be a reasonable aspect of explaining the omissions, it appeared these subjects were more concerned about the accuracy of music information presented in the program than about the pedagogical aspects. They seemed more concerned about comparing their own knowledge to the knowledge presented in the IMM program. On the one hand, subjects who possessed little music knowledge appeared to answer the questions requiring critical thinking differently as they tended to make suggestions that involved the pedagogical aspects of the IMM program. They seemed more inclined to suggest how the IMM program could be integrated as a valuable teaching tool in classroom environments. Also, these subjects were more inclined to suggest grade levels, age appropriateness, and directions about the integrativeness of the program with other curricula areas. For example, one component of the IMM program presented cooperative learning to be a benefit from participating in an orchestra. Subjects who possessed little music knowledge appeared to identify this to be a beneficial pedagogical component of the program. On the other hand, subjects who possessed music knowledge appeared to have difficulties with the open-ended questions. Though the researcher explained the questions to be open-ended, many subjects who possessed music knowledge expressed how the questions did not ask for recall of information presented in the IMM program. The open-ended questions appeared to confuse them. Perhaps this might explain why many of these subjects seemed 47 concerned about the accuracy of music information presented in the IMM program. Some of the language used in the open-ended questions could be revised since some subjects required detailed explanations for some of the questions. Observations of Qualitative Aspects of the Data One would expect to find a significant difference between subjects who possessed music knowledge and subjects who possessed little music knowledge. It may be that the Vee used during the investigation of the IMM program could have facilitated subjects who possessed little music knowledge to acquire new knowledge. This may imply the Vee to be a facilitative tool for helping subjects obtain new knowledge outcomes. Conversely, it may be subjects who possessed music knowledge did not acquire new knowledge due to their prior music knowledge. As can be observed in Table 4.2, when the Vee was used after the investigation of the IMM program, subjects who possessed music knowledge scored significantly higher than those who possessed little music knowledge. With no observable difference between the two levels of music knowledge when the Vee was used during their investigation, it may be that acquired knowledge in this instance is more likely to occur when the Vee is used metacognitively, that is, used as a tool to guide reflection when investigating the IMM program. Thus, it may also suggest music knowledge, since it is not something people require on a daily basis, may be more difficult to acquire and maintain without music comprehension over time. This may suggest that one can learn about music but not necessarily comprehend the music itself. It is possible the subjects may have been more motivated by the use of the Vee. It is also possible the instrumentation, The Orchestra IMM program may have effected internal validity. While every attempt was made to eliminate technological problems, unexpected technical errors did occur which prompted immediate troubleshooting. Some subjects were frustrated by the sound quality from some of the computer workstations. Some subjects also seemed impatient with the speed of the IMM program when compared to the speed of programs on other computers in the lab. The speed of the computer was a technological aspect which could not be corrected. Though the researcher made every effort to increase the volume on 48 some of the computer workstations, some subjects would still seem distracted by neighboring computers with higher sound quality. Also, some subjects could also have been more motivated and enthusiastic by the use of the computer. 49 Chapter Five Summary and Conclusions Purpose and Research Questions To assess whether IMM contexts can provide an educational rich environment for meaningful learning, it is necessary to determine the efficacy of evaluation tools used to assess knowledge outcomes. With the intent of improving music pedagogy, the purpose of this research was to develop an understanding of the relationship between the Vee and knowledge outcomes from an IMM program. This study adapted Gowin's Vee for educators to use to effectively evaluate the knowledge outcomes from an IMM program called The Orchestra. The purpose of this study was to determine if there was a difference between the Vee used as a metacognitive tool during the investigation of an IMM program, and the Vee used as an evaluation tool after the investigation of an IMM program. The specific research questions of this study were 1) to determine the reliability of the Vee, 2) to determine the comparative effects of two types of uses of the Vee, and 3) to determine the comparative effects of two groups of subjects who possessed music knowledge and who possessed little music knowledge on the knowledge outcomes from an IMM program. Design and Analysis Sixty-two subjects participated in this research study. The sample of subjects, whose ages ranged from 18 to 45 years old, included full-time and part-time teachers and student teachers enrolled in scheduled university music or technology classes. This sample of subjects represented a diverse group of teachers who either had some music or technology knowledge and had a keen interest in exploring IMM computer tutorials. None of the subjects had prior experience with the Vee or the IMM program. A quasi-experiment group design approach was considered where subjects were 5 0 divided into two quasi-treatment groups. One group was given the Vee during their investigation of The Orchestra while the other group was given the Vee after their investigation of The Orchestra.. The decision for whether a group was given the Vee during or after the investigation of the IMM program was random. Some of the classes were divided into several groups. Thus, the use of the Vee was altered equally within the same class with one group using the Vee during their investigation while the other group was given the Vee after their investigation of the IMM program. When the groups were small, the use of the Vee was determined by what was necessary to equal the sampling groups. All subjects were asked to rate their level of music knowledge. Subjects considered to possess little music knowledge were subjects who indicated they had less than five years of formal music lessons. Formal music lessons were defined as private music lessons outside of school. Subjects considered to possess music knowledge were subjects who indicated they had more than five years of formal music lessons. The A effect of the design involved one group of subjects who used the Vee during their investigation of the program, and one group of subjects who used the Vee after their investigation of the program (See Table 3.1). The B effect of the design involved one group of subjects who possessed music knowledge and one group of subjects who possessed little music knowledge. There were two independent variables in this research study. The use of the Vee during or after the IMM program constituted the A effect. The level of music knowledge constituted the dimensions of the B effect. The Vee rating scales scores constituted the dependent variables . Results There was a significant difference between the use of the Vee in that the mean of the subjects who used the Vee during their investigation was significantly higher than the mean of subjects who used the Vee after their investigation of the IMM program. Conclusions 51 In terms of objective validity, the Vee may be used to evaluate knowledge acquisition from an IMM program with some caution. Persons who possess music knowledge think differently about music than those who possess little music knowledge. Perhaps critically thinking about music is different than critically thinking musically. This suggests critical thinking in music and critical thinking about music are two disparate types of learning. Assessing knowledge acquisition from the Vee requires another form of representation besides written text format since subjects were asked to write about their reflections of the IMM program and were not reflecting musically. Recommendations Further research on assessing constructivist evaluation tools would provide other forms of representing knowledge acquisition. What is regarded as a specific kind of knowledge can be interpreted and known in many different ways. Thus, a particular form of knowing may have as its object many different kinds of knowledge. There needs to be further research to determine whether there is a correlation between kinds of knowledge and forms of knowing, and between process and forms of representation. Prior to the present study, no research with regard to the use of the Vee to assess learning outcomes from an IMM program existed in the literature. Results of this study suggest the need for further replication to corroborate and verify the efficacy of the Vee. There are several suggestions that might be incorporated in future studies. Though the Vee rating scales indicate knowledge construction to occur when used during their investigation of the IMM program, there needs to be further research with regard to the scoring of the Vee. Further refinement and modification of the open ended questions of the Vee might help scorers when interpreting the written answers. Also, further refinement and modification of the open ended questions could clarify several elements of the Vee to subjects. There might be further experimentation with regard to the administration of the Vee. After the subjects were immersed with the computer medium, a switch to a paper and pencil activity seemed to involve a constant shift in their cognitive activity. Perhaps, administering 52 the Vee on the computer medium would promote contiguity of cognitive activity. While it is hypothesized the use of the Vee encourages critical thinking, the limited time in a research milieu could influence subject answers. Some subjects indicated they needed more time to think about the questions but felt rushed by the limited time allotment. Some subjects indicated they wanted more time to explore the IMM program. In a classroom context, perhaps the subjects could be given more time to explore the IMM program and answer the Vee. The activity might even provide possible breaks for subjects to reflect upon the questions of the Vee. For classroom activities that involve young children, teachers might consider eliminating the questions on the conceptual side of the Vee. It would also be beneficial to clarify the elements of the Vee with young children so they understand the differences among the elements of the Vee. Perhaps some definitions in age appropriate language would also clarify the elements of the Vee. With the Vee containing open-ended questions, perhaps more discussion beforehand would greatly relieve subjects expectations of providing recall of information. The Vee rating scales involved interpreting experience from language in written text format. Since knowledge acquisition was personally constructed, contextually embedded, and individually situated, interpreting knowledge acquisition by written text only would seem to delimit the Vee. While the rating scales are one indicator of the efficacy of the Vee, perhaps other plausible alternatives for assessing knowledge acquistion from the Vee might provide another indicator of the efficacy of the Vee. Other plausible methods to determine the efficacy of the Vee might include concentration on developing and transforming meaning through language, the process of construction, and the structure of personal knowledge. Exploring these constructivist assessment techniques might provide other viable interpretations to determine the efficacy of the Vee as well as assisting educators to increase the range of strategies for fostering the process of human knowledge acquisition from IMM programs. 53 References Blumer, H. (1954). What is wrong with social theory? American Sociological Review. vl9, p. 3-10. Bork, A. (1992). Learning in the Twenty-First Century Interactive Multimedia Technology. Giardina, M. (Ed). Interactive Multimedia Learning Environments: Human Factors and Technical Considerations on Design Issues. Germany: NATO ASI Series. Borras, I. & LaFayette, R.C. (1994). Effects of Multimedia Courseware Subtitling on Speaking Performance of College Students of French. The Modern Language Journal. 78, (1). Bosworth, K. & Welsh, T.M. (1993). Design and Evaluation of an Interactive Multimedia Program for Teacher Education. Journal of Technical and Teacher Education. Cates, W.M. (1992). Fifteen Principles for Designing More Effective Instructional HypermeaUa/Multimedia Products. Educational Technology, p. 5-11. Chin, J.P., Diehl, V.A. & Norman, K. (1988). Development of an Instrument Measuring User Satisfaction of the Human Computer Interface. Proceedings of CHI '88. New York: ACM Press. Chou, L., McClintock, R., Moretti, F. & Nix, D.H. (1993). Technology and Education: New Wine in New Bottles Choosing Pasts and imagining Educational Futures. Available: Dettrick, B.W. Constructivist Teaching Strategies. Available at: Draper, S.W. (1996) Observing, Measuring, or Evaluating Courseware (in press) Feb, 1996. Draper, S., McAteer, A.K. & Anderson, A. (1994). Bringing A Constructivist Approach to Bear on Evaluation. Foot et al (Eds), Group and Interactive Learning. Southampton Boston: Computational Mechanics Publications. Druin, A. & Solomon, C. (1996). Designing Multimedia Environments for Children. USA: John Wiley & Sons, Inc. Duffy, T.M. & Jonassen, D.H. (1992). Constructivism and the Technology of Instruction: A Conversation. New Jersey: Lawrence Erlbaum Associates. Eden, H., Eisenberg, M., Fischer, G. & Repenning, A. (April 1996). Making Learning a Part of Life. Communications. 39 (4), p.40-45. Flavell, J.H. (1976). Metacognitive Aspects of Problem Solving. Resnick. L.B..fEd1. The Nature of Intelligence. New Jersey: Lawrence Erlbaum Associates. 54 Franklin, U. (1990). The Real World of Technology. Ontario: House of Anansi Press Limited. Giardina, M. (Ed). £1992). Interactive Multimedia Learning Environments: Human Factors & Technical Considerations on Design Issues. Springer-Verlag: Berlin. Gordon, E.E. (1986). Designing Objective Research in Music Education. Chicago: G.I.A. Publications. Gouzouasis, P. (1996) Organicism and Music Media. Canadian Music Educator. 37 (3). 1996. Gouzouasis, P.J. (1990). An investigation of the comparative effects of two pattern systems and two rhythm pattern systems for playing the guitar. Doctoral Dissertation, Temple University. Dissertation Abstracts International, 51: 2672-A. Gruender, CD. (1996). Constructivism and Learning: A Philosophical Appraisal. Educational Technology. 36 (3), p.21-29. Gurley-Dilger, L. (1992). Gowin's Vee: Linking the Lecture and the Laboratory. The Science Teacher, p. 51-57. Harel, I. (1991). Children Designers: Interdisciplinary Constructions for Learning and Knowing Mathematics in a Computer-Rich School. Norwood, NJ: Ablex. i Hapeshi, K. & Jones, D. (1992). Interactive Multimedia for Instruction: Cognitive Analysis of Role of Audition/Vision. International Journal of HCl. 4 (1), p.79-99. New York: ACM Press. Haugen, H. (1992). Multimedia Learning Environment: An Educational Challenge. Hix, D. & Hartson, H.R. (1993). Developing User Interfaces: Ensuring Usability Through Product and Process. New York: John Wiley and Sons. Holzer, S.M. (1994). From Active Learning. Available at: http://secme.che.ufl.educ/SUCCEED/pubs/innovator/innovator.l.2./succeed3.html. Jeffries, R. & Desurvire, H. (1992). Usability Testing and Heuristic Evaluation: Was there a Contest? SIGCHI Bulletin. 24 (4) p.39-41. New York: ACM Press. Jonassen, D., Mayes, T. & McAleese, R. A Manifesto for a Constructivist Approach to Uses of Technology in Higher Education. Duffy, T.M, Lowyck, J., Jonassen, D.H (Eds) Designing Environments for Constructive Learning. Germany: Springer-Verlag Berlin Heidelberg. Jonassen, D.H. (1991). Objectivism versus Constructivism: Do We Need a New Philosophical Paradigm? ETR&D. 39 (3), p. 5-14. Knuth, R.A. & Cunningham, D.J. (1993). Tools for Constructivism. Duffy, T.M, Lowyck, J., Jonassen, D.H (Eds) Designing Environments for Constructive Learning. Germany: Springer-Verlag Berlin Heidelberg. 55 Kafai, Y. & Resnick, M. (1996). Constructionism in Practice: Designing. Thinking, and Learning in a Digital World. New Jersey: Lawrence Erlbaum Associates. Laurel, B., Oren, T. & Abbe, D. (1990). Issues in Multimedia Interface Design: Media Integration and Interface Agents. Proceedings of CHI '90. p. 133-139. New York: ACMPress. Lehman, J., Carter, C. & Kahle, J. (1985) Concept Mapping, Vee Mapping, and Achievement: Results of a Field study with Black High School Students. Journal to Research in Science Teaching. 22 (7), p. 663-673. Lebow, D. (1993). Constructivist Values for Instructional Systems Design: Five Principles Toward a New Mindset. ETR&D.41. (3). p. 4-16. Metcalfe, J. & Shimamura, A.P. (1994). Metacognition: Knowing about Knowing. Cambridge, Massachusetts: The MTT Press. McLuhan, M. (1994). Understanding Media: The Extensions of Man. Cambridge, Massachusetts: The MTT Press. Miller, J.P. & Seller, W. (19901 Curriculum: Perspectives and Practice. Ontario: Copp Clark Pitman Ltd. Nielsen, J. & Molich, R. (1990). Heuristic Evaluation of User Interfaces. Proceedings of CHI '90. New York: ACMPress. Novak, J.D. & Iuli, RJ. (1996) (in press). Meaningful Learning as the Foundation For Constructivist Epistemology. Novak, J.D. (1993). Human Constructivism: A Unification of Psychological and Epistemological Phenomena in Meaning Making. International Journal of Personal Construct Psychology. 6. 167-193 Novak, J.D. (1990a). Concepts Maps and Vee Diagrams: Two Metacognitive Tools to Facilitate Meaningful Learning. Instructional Science. 19, p.1-25. Novak, J.D., Gowin, D.B. & Johansen, G.T. (1983). The Use of Concept Mapping and Knowledge Vee Mapping with Junior High School Science Students. Science Education. 67. Novak, J.D. (1993). How Do We Learn Our Lesson. Science Teacher. 60 (3). p.50-55. Novak, J.D. & Gowin, D.B. (1995). Learning How to Learn. Cambridge, England: Cambridge University Press. Papert, S. (1980). Mindstorms: Children. Computers and Powerful Ideas. New York: Basic Books, Inc. Papert, S. (1993). The Children's Machine. New York: Basic Books. Park, I. & Hannafin, M. Empirically-Based Guidelines for the Design of Interactive 56 Mdtimedia. Educational Technology Research and Design. 41(3). 63-85. Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S. & Carey, T. (1994) Human-Computer Interaction. Reading, MA: Addison-Wesley. Perlman, G. & Egan, D. (1990). Evaluating Hypermedia Systems. Proceedings of CHI '90. New York: ACM Press. Reeves, T. (1992). Evaluating Interactive Multimedia. Educational Technology. 32 (5), p.47-53. Reiser, R.A. & Kegelmann, H.W. (1994). Evaluating Instructional Software: A Review and Critique of Current Methods. ETR&D. 42(3), p. 63-69. Roth, W.M. & Bowen, M. (1993). The Unfolding Vee. Science Scope. 16 (5), p.28-32. Salomon, G. (1983). The Differential Investment of Mental Effort in Learning from Different Sources. Educational Psychologist. 18(1), p.42-50. Schwandt, T. A. (1994). Constructivist, Interpretivist Approaches to Human Inquiry. Denzin & Lincoln (Eds), The Handbook of Qualitative Research. USA: CA: SAGE Publications. Shackel, B. (1986). Ergonomics in Design for Usability. Proceedings of the Conference of the Second Conference of the BCS. p.44-64. Cambridge: Cambridge University Press. Schumacher, S. & McMillan, J.H. (1993). Research in Education: A Conceptual Introduction. New York: HarperCollins College Publishers. Stewart, R.D. (1994). The Development. Formative and Summative Evaluation of a Computer Multimedia Tutorial: A Case Study. Doctoral Dissertation, Walden University. Solomon, C. (1986). Computer Environments for Children. Cambridge, MA: MIT Press. Stoll, C. (1995). Silicon Snake Oil. New York: Doubleday. Strommen, E.F. (1992). Constructivism, Technology, and the Future of Classroom Learning. Available: http://www.ilt.columbia.educ/k 12/livetext/docs/construct.html. Swanwick, K. (1988). Music. Mind, and Education. New York: Routledge. Trevitt, C. (1995). Interactive Multimedia in University Teaching and Learning: some pointers to help promote discussion of design criteria. Paper presented at the session on package development at the Computers in University Biological Education virtual conference, Jan30-Febl0. C m Liverpool, UK. Available at: Walker, R. (1990). Musical Beliefs: Psychoacoustic. Mythical, and Educational Perspectives. New York: Teachers College Press. Walker, R. (1984). Music Education: Tradition and Innovation. Springfield: Thomas. 57 o o 3 o <D T3 O o c 0 O c (D o' 3 cr X m (D o a. C S uc sed can atlo 3 »>* tt — 3 (0 <B < 3 tt o a> ue o 3 text now w •>> <D a IQ o s a o a o o ta o D) > D O (D 3 a. > H (D < < <D <D O 3 0 00 o> or -iv co ro - * o o o O 0) i" (0 co ro -* O o 3 o <D "D W ro ->• H 3" <D O -« 5' (A o < O o 3 O <0 c 0) z 0) 3 (D CO ro -•• z' -n o o c (0 O c <D (0 O 3 o o tt (D CO ro -»• —I i -r ° a ID © (Q 3 »> 0) O 3 3 O <» i <D a CO CO ro ->• 3 O a. (D O 3' w CO ro -»• CT x m <D o a . 5 5 S 1 2 S 1 1 3 O — — 3 3 5 U 3 » ' 2 < £ 3 0) ra — 8 * • 3 E" 3 ca (D <D 3" O a. o o ta o D) TJ ro cn to o o 3 3 <D 3 to D> CT O < <D <D co ro -»• O o 3 3 (D 3 W 03 CT O c s ® o -2 2= tQ CD O s g. to Q CO 8. s. 0) o o *2 co ro ->• < CD «2 CD X o CD O H 8. to co 03 0) O o < <D CD m < m c to o 3 O <D cn co ro o CO a 3 m < c 0) T I T l T3 t l o i . i . a. 2. TJ 03 o TJ 03 CD TJ CD O CO o' 03_ CQ S O 3 ' 9-o a 3 * 3 3 cz c CO CO o' o' CD CD CO CO CO CO o o 3 3 3 _ 2. C ^ w jo i i 2> CD 3 r j co co o o 3 ^, 2-3 CO 3 tD o CO " CO CD _ 3 - ? 03 0) 3 * < CD CD 0) 5? CO cS » O CD 8 3 Q. § o) 3 . J 03 2 =>• o o — o 0 — £ CO 1 « < 03 f o CD ^ O CO o" 3 o CD Q. CO CD O CO 0) 3 o 3 O CD 03 O Appendix B: The Vee Map Profile Rating Scales Scoring Kev for The Vee 1. Focus Question 1 - No focus question identified. 2 - Focus question identified but inconsistent with EV #3 or Events #2 / #4. 3 - Focus question identified and consistent with EV #3 or Events #2 / #4. 4 - Focus question identified and consistent with EV #3 or Events #2 / #4, with additional suggestions that indicate a link to AW #2 or Theories #1. 2. Qbiects/Events 1 - No events are identified (#2/#3) 2 - Events are identified but inconsistent with FQ. 3 - Events are identified and consistent with FQ. 4 - Events are identified and consistent with FQ, with additional suggestions that indicate new questions (#7). 3. Educational Value Claims 1 - No educational value claims are identified (#3). 2 - Educational value claims are identified but inconsistent with FQ or KC #2. 3 - Educational value claims are identified and consistent with FQ or KC #2 or AW #1/#2. 4. Knowledge Claims 1 - No knowledge claim is identified (#1 / #2). 2 - Knowledge claim is identified but inconsistent with FQ or AW #1 / #2 or LC#l/#2. 3 - Knowledge claim is identified and consistent with FQ or AW #1 / #2 or LC#l/#2. 4 - Knowledge claim is identified and consistent, with additional suggestions that indicate new questions (#3). 61 5. Learning and Knowledge Transformations 1 - No learning/knowledge transformations identified (#1 / #2). 2 - Learning/knowledge transformations identified (#1 / #2) but inconsistent withFQ. 3 - Learning/knowledge transformations identified (#1 / #2) and consistent with FQ. 4 - Learning/knowledge transformations identified and consistent, with additional suggestions that indicate new questions (#3). 6. Technical Claims 1 - No technical claims are identified (#1 - #3). 2 - Technical claims are identified but #4 / #5 inconsistent with FQ. 3 - Technical claims are identified and #4 / #5 consistent with FQ or Events #1 - #5. 4 - Technical claims are identified and consistent, with additional suggestions that indicate new questions (#6). 7. Associated Words (Concepts'! 1 - No concepts are identified (#1 / #2). 2 - Concepts are identified (#1 / #2) but inconsistent with FQ. 3 - Concepts are identified and consistent with FQ. 4 - Concepts are identified and consistent, with additional suggestions that indicate new questions (#3). 8. Theories 1 - No theories are identified (#1). 2 - Theories are identified but inconsistent with FQ or WV. 3 - Theories are identified and consistent with FQ or WV. 4 - Theories are identified and consistent, with additional suggestions that indicate new questions (#2). 9. World View 1 - No world view is identified. 2 - World view is identified but inconsistent with FQ. 3 - World view is identified and consistent with FQ or EV #3. 4. World view is identified and consistent with FQ and EV #3 and KC #2. 


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items